“The fact of the matter is...” is filler. Like the extra fat in the bacon you want lean, or the long stalk on the broccoli crowns they are selling by the pound. These extraneous words and phrases fill up the time so the speaker need not root around for intelligent thoughts; and we pay more attention than they earn.
It’s more than mere filler, it’s a tell. They’re signaling that what they are about to say is actually a matter of dispute.
we pay more attention than they earn.
***Hey, good writing. [insert something pithy here]