Not necessarily. When the house next door had its yard cleared after being vacant for over a year the roof rats that had been in their trees paid us a visit. I discovered them when I found sweet potato pieces near the pantry. That hardly equals a crumb strewn dirty house.
The fact is that a mouse can find food just about anywhere.