Any time other than summer. Because temperatures increase as you go lower in elevation, the bottom of the canyon along the river is a hot desert environment even though the rims are up in the cool pines. Temperatures over 110 degrees F at the bottom are not uncommon. In addition, regardless of which trail you take there will be long stretches with little or no shade and the heat reflecting off the rocks can make sections feel like being in a solar oven.
I would avoid trips to the bottom between late-May and early September.
The ideal time to go to the bottom is spring (early April through mid-May) or fall (late September through mid-November). Temperatures will not be bad at the bottom and they will likely not be getting too cold at the top. Be warned however, that this is the most popular time for visitors to Phantom Ranch and so it is one of the hardest times to get reservations.
Winter has become one of my favorite times to go to the bottom. You will need to be prepared for cold temperatures and the possibility of ice or snow on the upper trails, but the bottom is pleasant and this time of year is less crowded. I especially like late November and December when the trees at Phantom Ranch are decked out in fall colors.
Regardless of when you go, remember that spaces are very limited at Phantom Ranch and reservations are typically made a year in advance. You could also backpack down and stay in the campground near Phantom Ranch - you can reserve permits for this only 4 months in advance, but then you will need to carry more stuff.
Have fun! It is a great place.