The reason there is molds in coffee is because it is a dried food item. It gets picked and then it sits outside for days or weeks while it dries. Nuts go through a very similar process but may or may not sit out in the sun. You may not notice the molds either, but they are there. The boiling process should kill some off but its still going to be there.
Coffee is also very acidic. I don’t want to get into a pH debate on the forum but this is another reason you shouldn’t drink it either.