You can’t attribute every symptom that appears to die-off. It’s clear that whatever dietary changes you’ve made have compromised your immunity, and it doesn’t seem uncommon here. Restrictive diets can cause more problems than they solve if you’re not careful. Take any symptoms you get seriously and respond to them accordingly. Hopefully your naturopath can help.