Sometimes people will want religious leaders (Rabbis, for example) to decide disputes. Other times, certain provisions in a contract are stipulated to the decided pursuant to religious law. In CA, this usually ( actually, it’s highly unusual) is between or among Jews. I haven’t seen one involving Muslims. I don’t know what or where their “court” is - I’ve never seen such a proceeding.
They can freely consent to any term or condition they want, or to have any person decide disputes, assuming no criminal law is violated. However, decisions pursuant to all such private arrangements are not enforceable in our court system unless otherwise consistent with our state/federal laws. In other words, they cannot come to the CA court system and seek an order from a state judge implementing the decision of their private arrangement if inconsistent with state law.
Wouldn’t that be arbitration, not court? Most contracts allow for binding arbitration over disputes in conditions of the contract.