He said they wouldn’t go to Camp David...He did not say that the Congressmen won’t be summoned to the WH, like being called up on the carpet. He doesn’t even have a vote on the matter, this is him trying to order Congress to obey him!
The budget is the sole purview of Congress and the most he has authority to do is to try and veto a budget ge doesn’t like. But he is not a member and he only gets to make input at the Congressional forebearance.
Other than that, he can go suck eggs!