Well, if he is questioning their lack of concern, then it is just a difference of opinion, and after all, Ashcroft and Justice have other things to keep them busy, like HUNTING TERRORISTS.
So, if he is only questioning their lack of concern, what's the big deal?
The Justice Department cannot limit their domestic job to, as you would say, "hunt terrorists." Limiting their duties to hunting for terrorists, while forsaking their sacred duty to protect the civil rights of Americans, would just be another victory for the terrorists.