Well, my morals work for me and yours work for you.
I can trust myself to do the right thing, and I can trust you to do the right thing.
But there have been cases where secular authority has gotten more power over the lives of individuals and states than is healthy and has created horrific machines of death.
But also we must recognize that religous authorities have created the same things (witness the perversions of Islam as of late)
So what is the answer?
IMO, we must ensure that the "authorities" whether secular, religious or otherwise do not gain the type of power that causes death and mayhem, and tramples on the rights of others.
You say that you do not wish to recognize the moral authority of those who do not share your faith, but guess what?
I don't want or need your religious based moral authority to live my life.
And I don't wish to foist mine on you.