I agree the E series are built like bricks, but that is what they are and look like, bricks.
The N series are fashion phones with brains. Think super model with a PhD. You wouldn't put her in a football game. The N series also pioneer design. The N95 has a dual slider, which in IMHO is useless, maybe they could have shaved 2-4mm from the N95 if it was a single slide design.
I have to say I have no build quality issues with my MIF N95. The MBN has the better firmware, IMO. But it wobbled and rattled. The volume button is not even. My $25 dollar E62 could hurt someone.