I agree that it breaks the marital covenant. Each partner trades the potential for other sex partners for the assurance that the other partner will do everything he/she can to meet sexual needs.
Fair enough, until one partner decides he/she (usually she) doesn't want to have sex anymore, which means the other partner becomes celibate, which he did not sign up for.
However, I am a little iffy about it being appropriate scriptural grounds for divorce. I think Christ was pretty clear that adultery was the sole ground.
There are a bunch of things a spouse can do that "break the marital covenant." I am wary of elevating all of them to proper Biblical grounds for divorce.
A covenant is just another word for a contract. If one party refuses to fulfill ones obligations, the other party has a legal right to void the contract.
The ending of a marriage due tomthe withholding of sex should only be done after all, efforts have been made to save it. Even so, the person who is refused sex should stay in the marriage until the children are grown. After that, they should feel no guilt in divorcing, and should be free to tell friends and family exactly the reason for ending the marriage.