I do find it a bit rude. You could have waited three minutes until he was finished at least. Buuuut, it's nothing to start a huge fight over. Just apologize, explain what you thought would happen (him calling you back), and hopefully he will understand.