That's the way the law is. One person's freedom only goes until it intrudes on another's.
But your freedom from noise, doesn't it intrude on that other persons' freedom to talk? Sounds like a vicious circle of freedoms.