I like to use descriptors to supplement my world building. Are there squat racks in your character world? Then sure, describe them as gym rats. But if there is no gym or anything of the sort, maybe utilize a comparison to a beast or animal of the region might be a better choice.
Having too much description is certainly something to stay away from as your reader will just get glossy eyed and you'll lose them, but fancy words don't always make the sentence better. Sometimes the simple descriptors are enough because we all understand them and can visualize them quickly. Round, boxy, large, small, jacked, lean. Colors and markings like scars, bruises, and indents. While stacking them up makes it very blocky and drawn out, simple terms remove the need for a reader to either find a dictionary or scratch their head over what you've said as they try to put the pieces together.
No, it's not the prettiest thing nor does it make it seem like you have an extensive vocabulary, but unless you're trying to flex your vocab it's not really something to be stressed over. In addition, some of those other, fancier words all have more specific definitions that might not quite fit what you're trying to describe. It may come across as a better fit because it's a higher "tier" word but someone who knows what the word means will stumble over it should it not actually work in the context you're using it.
Keep a thesaurus handy and throw a few words into it. Go down the rabbit hole a little and that generally does the trick for me. Just don't go too deep or you'll end up further away than the original word.
Edit: Also, just to add; less descriptors might actually take away from the end goal as well. While it may be more efficient, you might accidentally leave out imagery that the reader won't understand. Like for your muscle men. Saying they have muscles is a simple task but taking that extra sentence to describe how ripped they are will put that stronger image in the reader's mind.