That's why you don't listen to concept albums that put at the forefront some sort of narrative, like The Wall or, as you mentioned, Tommy. There's too much filler.
But there are albums that ARE sacred, with a sequence of songs impossible to skip, like In The Aeroplane Over the Sea or Loveless.