Audi sold just 93,506 cars in the U.S. last year, and doesn't feel that's enough cars to warrant building a Stateside factory.
Audi's head of U.S. sales, Johan de Nysschen, said that they need to move at least 150,000 cars here before they'll consider a U.S. plant.
The only Lexus plant in North America is in Canada.
There are a few Toyota plants in the US, listed here:
http://en.wikipedia.org/wiki/Toyota_Motor_Engineering_&_Manufacturing_North_America