I think Chapter 33:54-55 helps explain 34 a little better:
Now in verse Numbers 34:55 the Lord warns them,
But if you will not drive out the inhabitants of the land from before you; then it shall come to pass, that these which you let remain of them shall be pricks in your eyes, and thorns in your sides, and shall vex you in the land where you’re dwelling. Moreover it shall come to pass, that I shall do unto you, as I thought to do unto them. (Num 33:54-55)
In other words, these people will drag you down and ultimately I will have to destroy you out of the land, even as I destroyed them out of the land.
Sin always has a polluting influence.
So you concede there is a difference between the land inside the Numbers 34 borders versus the land inside the Genesis 15 borders? What is the nature of the difference?