> I don't care how many people are fawning over one person, if they only reciprocate towards one, than it's not a harem.
No, that's in line with how we commonly define the term. Multiple romantic interests means harem. Doesn't matter if the point of view protagonist ends bedding one of them or...