I remember from another source Heisenberg and other nuclear physicists had a meeting with Speer about the bomb in 1943 and they decided not to pursue the idea. I think one key was the unresolved issue of slow or fast neutrons. Speer considered that with the war on more immediate return could be expected from other uses of people and resources.
There was also the fact experimental physicists did not particularly like theoretical physicists, because their theories could seldom be proven, but they got prizes like Heisenberg Nobel in 1932. Also, they considered the work “Jew science”.
From Wikipedia:
Hilbert attended a banquet and was seated next to the new Minister of Education, Bernhard Rust. Rust asked whether "the Mathematical Institute really suffered so much because of the departure of the Jews". Hilbert replied, "Suffered? It doesn't exist any longer, does it!"