It's definitely bigger. On the Inspire 4G (or Thunderbolt, or Desire HD...), the grip's slightly strained if you don't have big hands -- mine are strictly average. My real beef is that you get a bigger screen but, right now, no extra resolution. The iPhone 4 lets you wrap your hand more comfortably, and more importantly, fits easily in most pockets.
If possible, wait for the iPhone 5 in June or July. We're looking at a dual-core A5 and, possibly, a larger screen in the same form factor. You're probably going to see some new phones at the CTIA show this week, too. If you had to buy an AT&T Android phone now, I'd get the Atrix, load it with Launcher Pro and enjoy.