So if a comet was to hit Earth it would do a lot more damage than an asteroid due to its much greater speed?
If they are of the same mass and composition, yes. Comets are typically less dense and mostly ice though, which will burn up faster in the atmosphere. But if a comet is rocky enough, then it will do more damage due to its higher speed.