I think the doctor is right. A tiny amount of sugar is, well, a tiny amount.
The body itself needs sugar, so even if your sugar intake through food and drinks is absolutely zero, your body will produce sugar from other materials. It has to, as cells use sugar as fuel. Especially brain cells and nerve cells, sugar is their only fuel.
Given the fact that the body produces its own sugar, there can be no harm in adding a tiny amount to that self-produced sugar.
Quitting the antidepressants may or may not be a good idea. It really depends on why you started taking them in the first place. You didn’t mention this. I’ve heard of quite a few people who were talked into quitting their antidepressants and who started feeling really, really bad, so far as to not being able to do anything at all. I’ve read (not personally experienced, thank goodness) of people who committed suicide after having been talked into quitting their antidepressants.
On the other hand, if your antidepressants are “only” a supportive means, or even a “convenience drug”, and if you will not experience any severe problems when you quit – then by all means quit.
The important takeaway is that, without knowing your specific reasons and your specific condition, giving you any advice about the antidepressants is downright risky.