If annulments were only granted when there was an actual flaw in the marriage, then that would make sense, and you're right that that is the definition of an annulment. But how do you explain wealthy and influential church members like Ted Kennedy getting an annulment after so many years of marriage when it's most convenient? Are flaws in the original marriage easy to find to end one after a couple of decades?
Do you think it's more common for wealthy, public people to get their marriages annuled than everyday folks or do you think it's possible that it might seem that way, cuz they get more attention?
As to finding original flaws after decades long marriages, until a marriage gets into dire trouble, people will tend to hide the flaws from the world. Going public with the flaws may seem to outsiders to always be at a "convenient" time, when in truth it's never convienient.