It’s unfortunate (though unsurprising) that your doctors are using your lifestyle choices as a way to try to shame/blame you for your health problems.
My vaginal yeast infections pretty much subsided when I quit eating gluten a few years ago, but prior to that, I noticed a couple of patterns. One, I would get a yeast infection for a couple days before starting menstruating nearly without fail for about a year. It would clear up when I actually started menstruating each month. I assume this is due to hormonal changes. Two, I would somewhat frequently get yeast infections after sex. I think there are a few reasons for this. I’m slightly allergic to latex, and I think the inflammation from this allergy would make me more vulnerable to infection. In addition to being made of latex, a lot of condoms are spermicidal, and it’s been my experience that spermicides contribute to vaginal yeast infections (I secretly call them vagicidal). The last big factor is that other people have yeasty mouths, and people that have been drinking beer or wine have even yeastier mouths. If you’re on the receiving end of oral sex (and as a liberated poly woman, I certainly hope you are!), you can have problems with someone else’s yeast colonizing for a while. It’s definitely happened to me before. And please don’t think I’m trying to slut-shame at all (I’m not, poly is the most accurate way to describe me as well), but I think having sexual encounters with new people presents us with a new set of microbes to deal with that our body has to learn to navigate, and the going can be rough at first if you have other issues compromising your health (like a general candida overgrowth or a bad diet, etc).
Good luck with everything! I hope you have success getting your health back.