And legally, they should not. Per the IRS’s own publication P15A, there are 20 common law factors, of which sole-customer is just one. The IRS is to consider ALL factors in determining independent contractor or employee status.
Unfortunately, the IRS doesn’t follow its own guidelines and assumes if you meet ANY of the factors you’re an employee. Which is clearly incorrect and illegal.
But you’re right, they’ll come after you anyway and force you to prove otherwise - that you’re not breaking the law. Guilty until proven innocent, in the case of the IRS.
No kidding. They will even freeze access to your bank accounts before you've had a shot a due process. It's pretty hard to defend yourself when the attacking party is free to seize control of the very assets you require to mount a defense.