I agree with you. Although, the context of the story matters. For example, if the story is a murder mystery and suddenly the protagonist is up for some Harem action: it will more often than not alienate the readers.
However, if the story is erotica designed specifically to tickle the fancies of...