How would they pass the written test?
I don't see how inability to read words on signs is a significant impairment in terms of safety - stop, yield and danger signs convey their messages through images (like falling rocks) or shape and color (red octagon).
I can see how dyslexics would be more likely to take the wrong exit or miss a detour.
Even if they were more prone to miss safety-related signs, the fact that probably ten million people who know no English are on US roads right now is a greater concern.