They pretty much set it up that he’s going to at least allow the White Walkers to follow him through the Wall if he passes through it with that mark on him.
In the books there is also the “horn of Joramun” which supposedly can bring down the wall, but they didn’t really introduce that on the show.
In the show, Sam finds a horn in the dragonglass/obsidian cache north of the wall. Ygritte also tells Jon Snow about Mance digging around everywhere, I think it was Season 2 or 3, so I believe that nugget isn’t book-only.