Meh, i think it's fine. More variety is good.
GB stories where they explore gender identity are like, cool. I'm sure there are many who like them having it a little deeper—simple is fine too, but I don't see why straight people would avoid GB genre just because of that, other than, you know, them not finding what they're looking for.
Literature is always littered with them. The gender identity part is always linked to it. But of course, I won't pretend to be oblivious, Japanese (or other, just less prevalent) gender bender stories are usually devoid of that, and straight people prefer that. But I don't see why the genre is ruined by anything more 'radical' than that kind of works. The genre is popularily connected to them, yes, but what can we do about it? There's an angst novel about a trans(wo)man mc, should it's disallowed from ticking the gender bender genre?
The way I see it, it's, similar to a case of [isekai genre]. It's full of japanese isekai tropes (obviously), and an avid of portal fantasy really can't do anything but to tick that [isekai] box cause there's no other way around it. Are they ruining isekai genre?
As for the sex theme. You just have to find a website or publisher that doesn't cater to it. I assure you, the moment you enter a website nicknamed SmutHub, you have more problem than this supposed sex scenes in muh gender bender stories.
Just an example. JJWX publisher in China. It really really heavily caters to GL stories, and most of the stories are 'innocent' just like what you want in a gender bender. The reason? It pretty much has very light stories with no deeper exploration on homosexuality, cause it's webnovel, write whatever, have fun. They also periodically censor them steamy scenes.