I would say that not responding to a direct question might be considered rude. You could always say, "I prefer not to talk until we are done, thanks."
You might consider posting this in Games or in Etiquette, since I really am not sure what "crib" is (I am guessing cribbage) and have no idea of the type of environment in which you are playing.
