i just remove mine and dont worry about the safely remove option. I have never had an issue to date.
Just make sure you arent in the middle of reading/writing anything to it.
if you want to be super careful, then shut off the computer and then remove it.