sugar and the early steps

I know that this question has come up before and I am a bit dense in asking again but here goes.

I am on step two.
I have been working step one and two since mid February. (more thatn 2 months) I had not been eating sugar before RR except for the occasional binge. I was on a fairly low- carb diet before RR.

I started adding sugar back in: chocolate almond milk, jam on toast and a dessert after lunch each day.
Since I have been journalling, I notice that after the dessert I am consistently cranky and sad about 1-2 hours after I eat the sugary desert. I eliminated it yesterday day and felt better. I have not cut out all sugar.
I have decided to eat fruit and continue with sweetened milk.
I know it is suggested to not eliminate sugar at this point but the desserts seems to be having worse effects than they used to, even after meals.
Am I having enough sugar without them?
I have a feeling someone will say- stay on the step you are on and do not worry about the sugar either way but i did read to not eliminate too soon.

Thank you for your responses!!

Copper