As it's his job and he's being paid to do it, then of course we can judge/have opinions. If someone thinks I do a bad job of what I'm paid to do, even if they don't do it themselves, then they've every right to voice their views. By your logic, if I get bad service from a waiter in a restuarant I can't complain because I've never waited a table in my life.
For the record, in my opinion there's far worse the Bloom, but he is completely overrated in the acting and looks department.