I had a Nokia N95 prior to getting the first (and now regarded as pretty crap) Android phone - the T-Mobile G1 / HTC Dream. Newer phones, like the HTC Desire that run Android 2.1 or Android 2.2 are way better.
So let me tell you what differences I noticed, good and bad.
The camera on my G1 is utterly crap compared to my N95. Even the latest and greatest Android phones with 8MP cameras don't seem as good in certain conditions. The Carl Zeiss optics on the Nokia N95/96 is far superior to most Android phones. Similarly, the video recording won't be as good as your Nokia.
After that, everything is better on Android. You'll no doubt find some stuff that appears crap at first, but if that's the case come back here and we'll tell you a bunch of apps that'll make things much better. For example, I was dismayed to find that I couldn't use my own ringtones for SMS tones etc. But you can, you just need an app or put them in the right folder on the SD card. My Nokia (as standard or through an app?) would notify me every X minutes if I had a text message. This was great if I'd been in the shower or something. Android doesn't do this by default, but then I discovered Missed Call, a great app that allows you to configure just about every notification in any way you like and have them repeating as often as you like, even setting the colour of the flashing LED and custom vibrate patterns. It goes further too allowing you to specify custom notifications for individual contacts.
I found the default Messaging app a bit crap. Still I liked it more than Nokia's offering. But then I discovered Handcent that allows you to do a million cool things with text messages and is totally customisable in terms of its look and feel. This is why Android is so good - you can replace anything you don't like with 3rd party alternatives.
Finally, one thing I really missed on my Nokia was having my Calendar entries on my main screen. This has been possible in later versions of Android (since 1.5), but I found an amazing Calender widget called Pure Calendar Widget which allows you to take up as much or as little of your home screen as you like, and display calendar entries in any format you like with as much or little detail as you like, with any "skin" that you like.
Even though I have the first Android phone ever released, there is absolutely no way in hell I'd go back to Nokia. Android is just in a completely different league. Once I'd got to the stage that my G1 did everything my N95 could do, only better, I then discovered that I wasn't even 1% through what my Android phone could do for me. I then started to experiment and found a load of really cool apps.
For a small taster of the kind of really cool apps you can get on Android, have a look at this thread: http://rabroad.com/forums/f7/apps-to-highlight-androids-potential-33883/
I imagine you're getting the picture now... Freedom, customisation, great community support, 60,000 apps to solve all your problems (75% of which are free), and decent hardware to back them up. There are now more free apps on the Android Market than on the Apple App Store.
The only real problem you'll have is knowing what apps exist out there that may improve your experience. There are a lot to discover out there. But this is what we're here for. Many of us here have well over a year and a half of experience with Android, so the community will be happy to recommend decent apps. If there's anything you don't like on your phone, or anything you felt your N96 did better (apart from pictures/vids), or anything you feel "Hey, I wish it could do this....", then just come back and ask. Most of the time there's a good solution.
I personally felt that, until a couple of weeks ago anyway, the HTC Desire was overall the best phone normal people could buy anywhere in the world. I highly doubt you'll regret it, but as with all transitions to different things, there will be a learning curve and you may hate it for the first few days. After that, you'll never look back.