Who will make the foreign troops leave? The Serb army? Don't make me laugh.
Assuming that foreign observers do decide to just vanish one day, allowing Serb armed forces to walk in, how will "justice" be restored?
Rounding up and shooting any Kosovars who resist invasion? Patrolling the streets of Kosovo night and day with Serb soldiers and secret police, keeping the Kosovars in line?
Serbs are neither Dhimmis nor quitters
Except, of course, for the three hundred or so years when they were officially dhimmis of the Sultan - and basically quitters.
They will leave on their own. All the Serbs have to do is wait and then reclaim what is theirs.
Your jihadist allies can't kill them all.