The math typically encountered in an introductory astronomy class is relatively mild compared to that encountered in chemistry and certainly compared to an introductory physics class. It may involve plugging numbers into an equation, converting units, and some elementary algebra at most.
If you are interested in astronomy: Try it!
Very often the fact that you are interested in something can help you better understand the aspects that you are less interested in or not very good at. Math is important, and education is all about improving yourself. If you take the class, you will learn something about astronomy, and you may even get better at math---you certainly wont get worse. And that is the very best thing about education!
If you do find yourself having troubles, go to the professor for help early and often. I myself am an astronomy professor, and the students who come for help always do better. The one's who never come for help just suffer and usually don't do well. The professor's job it to teach you. You are paying him/her, so take advantage of that!
Historically, astronomy is one of the main sources of inspiration for human cultures. Every culture has studied the stars, and today we have more reasons than ever to do so!