The actual term from the Greek is porneia - which is not semantically equivalent to "marital unfaithfulness.
Wideawake is correct—there is a distinction between the Greek moichatai (which more or less equals adultery—not fornication) and the Greek porneia—which would include adultery, fornication, and a whole lot more. Matthew 19:9 has both—the only grounds for putting away a putative life is for porneia, and one who marries a divorce woman is guilty of moichatai.
Under Jewish law, one couldn’t marry anyone one felt like—beginning with close relatives (or a woman that was already married). The best explanation I have heard is that if one has attempted marriage with someone that one couldn’t marry—even if done in good faith by both parties—the resulting state would not be marriage but porenia.
To put it in Catholic terms, if one discovers that there was something present from the beginning that made the marriage impossible, the marriage attempt was nul, and thus the union is capable of being declared nul—thus annulments.