Fat, at nine calories per gram, is nothing to worry about but sugar, at four calories per gram, is deleterious to my health? That's not how it works.
If you consume more calories than you burn, regardless of the source of those calories, you will gain weight.
Fat is much more difficult to metabolize. Sugar is easily assimilated into your body.
Best to give up the carbs then you'll be OK.