I leave mine on and wash them just like the bike. If I'm still in the cleaning mood, I'll take them off to wash the bike underneath. I haven't found any cleaning product that has caused any issues to the bags. I use my bike to commute everyday, year round and some nasty stuff has gotten on my bags. When I do get a wild hair up my .... I wash the bags on the bike with soap and water, dry them with a leaf blower, then apply a good coat of the HD leather treatment that came with the bike. Wipe it off the next day and the bags still look like new except where the lids rub on the backrest quick releases but thats a seperate issue.
I know I'm not the norm. Like I said, I ride everyday and haven't really "cleaned" the bike since June. It sits out in the sun and rain during the day at work by the chemical plants. But just 10 minutes applying the HD leather treatment, sit over night and when I wipe it off the next day, my bags will look as good as any ones.
Sorry, that was a long post when all I meant to say is, you can treat em like crap and they'll still clean up just fine.