Yes, but who says it's wrong to punch someone in the face just because it's unpleasant?
“Yes, but who says it’s wrong to punch someone in the face just because it’s unpleasant?”
—I would say that (in general) it is wrong to punch someone in the face because we don’t like to be punched in the face and, because of empathy, we generally don’t want others to be punched in the face - and thus the feeling that it’s “wrong” to do so.
And so morality comes primarily from empathy.
Actually, the very fact that you ask the question implies that you already realize this.
One wouldn’t hear a Jew ask a non-Jew “without the Torah how would we know that it is wrong to eat shellfish but not fish?” or a Hindu ask a non-Hindu “without the Hindu scriptures how would we know that it is wrong to kill cows?” as arguments in favor of the Torah or Hindu scripture. In both cases they realize that the ‘nonbeliever’ likely doesn’t share such values, and so the questions wouldn’t make sense.
However, they may ask nonbelievers questions like “how would we know that murder is wrong...” in favor of their religious writings, with the implication being that we need their religious writings to have morality and know right from wrong. But in these cases the questions equally make no sense since they wouldn’t have asked such questions without already assuming that nonbelievers DO have morality and know right from wrong independent of the religious writings. The questioner is appealing to the universal innate empathy we all share.
In other words, the implication of the question contradicts what the questioner must already assume in asking the question.
The question assumes that rules regarding how we treat each other (i.e. morality) is something that we all care deeply about and generally agree on. Why do we all care about morality? Because we all have empathy. That we all care about morality is the reason it exists.