I practically passed out on the morning of the 4th day myself. It takes a while for your body to adjust to using fat as its primary source of energy. Until it does, you’ll suffer from hypoglycemia-style symptoms like weakness, irritability and sugar cravings, which could be quite severe depending how carby your diet used to be. Die-off and caffeine withdrawal only make things worse.
My advice is to eat something with a good amount of calories as late as possible (three hours before bed or so) and to be very careful in the mornings. I usually sit up in bed and let my head adjust for a couple of minutes. After a couple of weeks the cravings largely disappeared for me, and now even if I’m in a shop surrounded by my favourite foods I’m more concerned about the sugar, junk fats and other additives they put in the stuff than actually eating it. It’s like the smokescreen has been lifted.