The chlorination won’t hurt them and can actually be a benefit in larger concentrations because it kills bacteria.
Chlorine also kills the microorganisms and fungi in your soil that have essential symbiotic relationships with your plants, not to mention the benefits they offer your soil by composting. Without those microorganisms your plants will not thrive. Worms don't like chlorine, either. It takes a while, but chlorine residue will build up in the soil over time, especially in potted plants, and kill these essential critters. Chlorine free water is a much better way to go.