The Sign of Peace is completely optional for each priest. Talk to your priest about not interrupting the concentration on the Eucharist that should be going on at that time.
In the earliest centuries of the Church it was called the kiss of peace.
Peace be with you?
Question: One thing at Mass that I find really difficult is the “Sign of Peace”. How come we do this?
Dear Inquirer,
There is no better way to describe the custom of exchanging the greeting of peace at Mass than the words of St. Augustine in the 4th century: “After the Lord’s Prayer, say ‘Peace be with you.’ Christians then embrace one another with a holy kiss. This is the sign of peace.”
In the primitive church at Rome and in the Eastern Church, the kiss of peace was offered after the first part of the Mass and before the Eucharistic Prayer. Early baptismal documents also indicate that the exchange of peace was reserved only for the ‘faithful,’ and so catechumens were dismissed before the Prayer of the Faithful, which was followed by the Kiss of Peace.
In the Western Church the sign of peace was moved quite early to where it is as Augustine described it and where it is today. The Western Church saw a close link between peace and communion—peace with one another before receiving the Prince of Peace.
In the Middle Ages the laity were excluded from the sign of peace and it was then dropped altogether from the Mass; the only remnant of the rite was the action of the priest kissing the altar. Vatican II restored the ancient rite of peace to all who participate at mass.
Custom dictates how the kiss of peace is exchanged in each country: a handshake, an embrace, words of peace, or other actions. In Japan, for example, the celebrant bows deeply to the congregation who in turn bow towards him and then bow respectfully to one another. It is a sign that works well in their culture. How the sign of peace is given will vary, but its meaning remains the same.
http://www.cptryon.org/ask/ask/signpeace.html