If someone has a wandering eye, which are you supposed to look at when talking to them?

[Malty21]

I find myself looking a the one that is not looking at me.