Given that all heat on Earth, save the minuscule amount derived from the universal background radiation, and from volcanoes, comes from the sun. It is, far and away, the first order variable in the equation. Are you saying that changes in the output of the sun do not impact Earth's temperature?
Let's extrude your argument. Suppose for one month the total output of the sun doubled. What do you suppose the outcome would be here on the planet? Now let's suppose instead that solar flare activity doubled for one month. In which case do you supposed we might find the Earth warmer?
Brightness (lamberts) and Energy (joules) are two different things.