The SWAT guys are not stupid. They know the drug gangs would shoot back. More to the point, they would be up, awake, and alert at 4am because that's when they do a lot of their business. Even more to the point, if the SWAT people seriously disrupt their business, the drug gangs would make a serious effort to take the war home to the SWAT guys, as they do in Mexico, and the SWAT guys have no interest in starting to play that game.
Every little town has no need for SWAT. Let SWAT be a State Police function, where the local cops have to call the State and justify why they need a SWAT raid.
'Nuff said.