No, it doesn't unless they're attracted to your MC. If not, then it's just a coincidence they're all women.
If you don't want it to be a Harem, then it isn't. Just build the romance with that one love interest. The rest either get attracted to someone else or no one.