Any phone call can be very distracting. A hand-held cell phone call is very distracting, while driving. Glancing at your instrument panel, or adjusting the radio are (usually) less distracting. They don’t require much of your attention.
I find that my window-mounted GPS isn’t distracting, when it’s in voice mode. Sometimes, it removes distractions — e.g. you don’t need to look at every street sign, or building address, when you’re traveling to an unfamiliar destination. You can focus on road and traffic conditions, rather than navigating.
Your point about the levels of distraction is very true. Being able to deal with things that distract are part of the skill of being a good driver. I think laws can go too far trying to regulate distractions. I have noticed that once they start the regulations, they go into the ridiculous. And one careless driver will cause needless regulations that will affect everyone else.