Is there a reason why you had to send in the swabs rather than being tested in a doctor's office or clinic? Aspergillus is everywhere so the chances of cross contamination seem like they would be extremely high (and I'm surprised that any doctor could make a diagnosis of an aspergillus infection in your lungs based on a couple of cotton swabs).
Aspergillus infections are fairly rare and normally occur in people who have a compromised immune system so you may need to get a second opinion from someone who can actually perform a physical exam to confirm that you have an aspergillus infection.
