I may be mistaken, but did Hagrid know about Riddle being Voldemort? When he assumed his pseudonym he completely abandoned his old identity, and after going underground creating his Horcruxes he reappeared with a radically changed appearance, due to ripping his soul apart.
Hagrid might have...