It seems possible that your immune system has more energy to put into attacking wheat when it shows up once in a while since you’re not constantly keeping it busy/downtrodden.
I definitely noticed a physical/contact issue with gluten several months to a year before quitting eating it…I just didn’t realize that was the problem. I started using “natural” shampoo for my crazy dandruff (which turned out to be a gluten-allergy symptom!), and the shampoo I was using contained hydrolyzed wheat protein. It made my scalp SO much worse! Like, painful and insanely itchy and it would end up weeping/bleeding from the combination of being super inflamed and me itching it. I only made the connection somewhat later.
Sugar rage!! I believe it. There’s an article I posted on Orka’s grains thread that discusses gluten-related behavioral/neurological symptoms, too. It’s pretty interesting.