Do what you want to as long as you can afford it. I subscribe to the Dave Ramsey school of thought. My only debt is my house and I am just a few years away from owning that free and clear.
You and the bank own it now, soon only you and the property tax assessor will own it.