Anything under 3 isn’t harem imo. I would say 3-4 is small. 5-7 is medium. Over 7 is large.
From a literary perspective when it goes beyond 7 it really tends to become much less interesting to me, because it’s just not possible to entertain the MC, the harem members, side characters and a plot...