Force normal = FgravityOnFirstBox - FgravityHangingBox
So, for 30 N box on other side:
Fnormal = 77 - 30 = 47.0 N
for 60 N box...
Fnormal = 77 - 60 = 17.0 N
for 90 N box...
Fnormal = 77 - 90 = -13 N, so the box is lifted off the ground up to the pulley because the second box weighs more.