Depends on the genre, but in general: both.
Fluff (or the stuff happening in the downtime) is what makes you care about the characters, angst is what creates the stakes and drives the story.
Though that said, SoL or similar genres also work fine with lesser stakes...