Driving is not a privilege. We have a common law right to freely move about. There is no constitutional role for the government to restrict movement or require papers to travel.
It is a privilege. Any official State actor can revoke that privilege based upon the agreement made when the license is issued.