I've forgotten to rinse my filter before. It won't do any harm to your fish. The only thing you could do would be to clean the entire tank out, but when I did it I didn't have any problems w/ cloudiness, or anything. I think the filter may have actually cleaned the dust particles out.