This is a simple question but one that demands extra thought if needed and it's simple what number do you consider a small harem? I believe anything between more than three is a small large harem, as there is no conventional standard for medium-sized harems, just a big or a small one.
So how many members would you consider a small haren three, fourth, or five? I'd say anything past five is pushing it personally.
The ultimate truth is that 3 is indeed the limit.
No matter the preference of the reader.
It can be in term of math/geometry, or in term of feelings but "3" is the ultimate balance.
----Even though some people would say 4-5 are fine, but when it surpass "3" its impossible to deny there is a subtle "surplus/overload" feeling.
Speaking of,
Mushoku can be said to be the ultimate harem and one of the earliest "mainstream work" having a successful popular harem (the WN is quite old and took forever to get anime adaptation bc of its "special content" despite the popularizer of "isekai" slang), accepted even by anti-harem readers.
----- Even though some readers dislike harem, some of them wont deny that its well done and written, unlike the harem series everywhere (years ago) which was the reason they dislike this "genre/tag" (as it was indeed the truth that, in the past, "harem" was a "flag" to "bad quality" with high probability).
They were not vases but unique.
And in term of the pov of author-side,
its undeniable that having less, is far easier/manageable to write than "more".
More means less screentime for all, means lower quality, means likely disliked (and to become vases/pendants), means FAIL.
It become a spiral of doom. A self-eating loop of destruction.