It depends on the state. My last congressman faced recall almost as soon as he was in office. It failed but it is an option in several states.
The United States Constitution expressly establishes the exclusive qualifications for congressional office, sets the specific length of terms for Members of the House and for Senators, and places the authority within each House of Congress to judge the elections and qualifications of, and to discipline and remove, its own Members. These provisions of the United States Constitution, with respect to federal officials, have supremacy over State laws and provisions, and State laws in conflict with such constitutional provisions have been found by the courts in the past to be invalid. Although the language of some State recall laws might be broad enough to include Members of Congress, or might even explicitly include such federal officers, such statutes would not appear to be effective in overriding the provisions of the United States Constitution with regard to terms of office, elections and removal of Members.