Modesty, formal-wear, and a complimentary hue.
Modesty and formal are self-explanatory.
The color should compliment the bridesmaid's colors, but NOT be an exact match (she's not a bridesmaid!) An off-shade of that color or a neutral color is ideal. For example, if the bridemaids dresses were a Cornflower blue, some possible choices for a Mother's dress could be a navy blue, smokey blue, or a neutral color like beige. The idea is, you don't want your mother's choice of color to be distracting in the photographs.
My friend's mom made this mistake, and every time I look at her family wedding photograph, the first thing that pops out at me in the magenta dress, instead of the Bride and Groom.