You want some real "fun" with this - look to E. E. "Doc" Smith's "Lensmen" stories - the Lensmen themselves are just elite, well, superheroes in space (they were partial inspiration for the Green Lantern Corps - not the original character, but the Silver Age version - in DC Comics), but the two warring races behind the stories, one evil that I can never remember the name of and one good race, the Arisians, are both from other dimensions, unable to affect each other directly so they contact and empower the natives of this universe; they are, essentially, "reverse Isekai" as they are weaker here and unable to get back home (the bad guys because they destroyed their home, the Arisians because they had to bind themselves to this universe to be able to empower champions to stop the enemy),
There were works like that. If I recall any, I will post the titles on your profile.
I know I have sort of seen two - one where the person tried but wound up on the wrong world, and one where it did work.
Oh, possibly the earliest "true Isekai" stories (but I'm pretty sure they predate the use of the term) would be things like the Harold Shea (started by L. Sprague DeKamp but continued by a few others) novels (also sort of proto GameLit, and definite inspiration for some modern games) and a 1949 novel titled "Silverlock" that has the hero transported after an accident (I think a car crash but it has been 30 years since I read it) to The Commonwealth, a land consisting of several fictional or quasi-historical countries.
I THINK some of the Harold Shea stories are examples of characters "voluntarily Isekai-ing themselves" as they feature people who follow Shea into the fantasy world(s?) he visits trying to find him.