I would agree with some of the others here with regard to the transmission fluid. It is REALLY important .... and I would bet about 90% of car owners could care less about it .... thus they NEVER service their transmissions (new fluid / filter) ... not ever.
It is never too late to stop smoking ...... and it is never too late to do a trans service. You would need to do a total system flush and also replace the filter. This is a starting point ...... it just may solve your problem ..... if so consider yourself very lucky .... ! (^-^)
by the way .... MAKE SURE you put actual Toyota fluid back in your trans .... as per your factory manual ...... it matters.
I hope it is not something more complicated, for your sake ...
Meanwhile .... you can get more info at the Lexus Owners Club ... just google it .... there is a lot of info there all for free .... what a great web site!!!
hope this helps .....