So the idea is that he isn't actually invincible, He's literally just a normal guy like all isekai protagonists.
But maybe if at some point he loses and actually gets sent to the realm, Because he avoided it for so long the realm could be overtaken by the villain and be trying to rise to the...