In the Race for the Bomb the Nazis had the immense handicap of their racialist ideology.
A very large number, possibly a majority, of the scientists important in the Manhattan Project were refugees from Nazi Europe. With other policies many of them might have been happy to work for Hitler.
There was also a strong undercurrent in Nazi ideology of disdain for “Jewish science,” which was basically all advanced physics.
OTOH we benefited greatly from Nazi induced German fear of the Reds. The German scientists move to the US as much to get away from Stalin as from Hitler.