It has to do with the Eyedropper tool. Whenever I am using the Brush tool, and then I hold the Alt button to select a color, the color gets put on the Secondary color choice and I have to press "X" and switch the colors in order to use the color I selected and it's getting kind of annoying...