A true immune-system mediated allergy should show up, whether it causes sneezing, asthma, rashes or anything else. Some people claim it's possible for allergies to cause fatigue only, some claim otherwise.
For a ragweed allergy, skin test is more reliable than blood test, so most likely either you aren't allergic or you just recently developed it. People with ragweed allergy typically feel symptoms from right about now until first frost.
Milk intolerance, digestive symptoms after eating dairy products, is NOT an allergy. It's just that you don't produce enough of the enzyme that breaks down lactose, which is a major part of dairy. In fact, about half of all adults on the planet (including most of those from certain ethnic groups) are at least somewhat lactose intolerant.
Since you got a positive RAST for milk, you might also have a slight allergy to it. People who are "only" lactose intolerant can generally eat dairy in very small quantities, if you are actually allergic too, you should probably avoid it entirely.