Both objects (you and the target) are at rest, for all practical purposes. You only need to account for distribution. Since the bullet is smaller, it creates higher pressure and can penetrate.
In other words, imagine that two people are in a duel. Person A shoots and hits the barrel of the B's gun. The bullet enters the barrel and stops. Both persons feel exactly the same impact. You can generate additional energy only if you fire rockets.
The gun has a moment of inertia that’s significantly greater than the bullet’s. This accounts for the bullet’s force vector. Managing the recoil has no effect on the bullet.
Regarding the distribution of force, imagine pressing the wrong end of a pin into a cushion.
With a lot of police departments having moved from 9mm to .40, you see LOTS of one shot stops of dogs. In the past amped up angry dogs use to take up to six shots before stopping, probably just bleeding out.