It's been awhile since I read it, but if the black kids are female, it might be Sula, by Toni Morrison. The two heroines, Nel and Sula start out as children, and there could very well be a part of the novel where they were afraid of a mentally disabled man, but I really can't remember.
Sula was was an Orpah Book Club book in 2002.
If that's not it, here's a list of the books in Oprah's Book Club. Maybe reading the title might jot your memory?
http://www.oprah.com/article/oprahsbookclub/pastselections/20080701_orig_list
