I suspect you are confusing "manufacturing output" with "factory employment." Manufacturing output has increased almost every year since World War II, but manufacturing employment has not. Constant advances in technology and improvements in productivity have made it possible to manufacture more things with fewer workers over time.
Even China has seen a dramatic decline in factory employment in recent years, despite dramatic increases in factory output over that same period.
In raw numbers, factory employment in the US peaked in the mid 90s, as a percentage yes it declined, but absolute numbers, factor employment untill the late 90s generally tracked with the economy at large.