Regarding Wizards and Armor... Well, first, armor requires fitness that not all wizards have.
Second, you need to get used to actually moving in it.
Third, you might need help actually wearing it.
Fourth, it's expensive.
Fifth, it might actually get in the way of magical gestures as it should restrict your movement to some extent if you aren't used to wearing it (and maybe even if you are used to wearing it? Dunno, never wore armor).
Sixth, in some universes, how well magic is conducted depends not only on the caster, but also on the things that they're wearing. If their gear has a poor magic compatibility, then their whole spellcasting process can be hindered. See Saga Frontier 2 for a good example of this, as it shows Metal as a horrible magic conductor, which forces everyone to fight using weapons of Wood and Stone instead of Steel.
Seventh, it can make for a better narrative to have your casters being fragile. Battlemages are great for power fantasies, but bad for some larger in-universe balance, since you want each role to have their strengths and weaknesses... And let's be frank, a fit, armored mage, with good sword-wielding skills and strong magic is basically what every single caster would become if it was possible.