Nobody tried to “ban Islam”, rather they banned a theocratic legal and political system (Sharia law). How in the world could that conflict with the first amendment?
I can't say for certain, having not seen whatever actual order was handed down, but at the very least, I believe there would be a problem with the fact that the law bans only *a* theocratic legal system.
It doesn't restrict judges' use or implementation of papal law, or Jewish law, or Bahai law, or Hindu law. Judges can still rely on any of those varieties of religious law. Rather, it singles out just one particular type of religious law, Islamic law, and then says that Oklahoma courts have to treat that religious law different from any and all other religious law.
And that definitely puts it in conflict with the First Amendment.