I've (usually, not very often) been able to get by with "I can't wake up that early" or, if it's a nighttime service, "I have plans/a date/whatever."
If my mom's there, she says "It's against her religion." Which, while slightly witty, tends to lead to questions like "Do you believe in this god...