Slow growth accounted for the dramatic loss of seats starting in the 1820s (it was pretty strong growth from 1790-1810, going from 2 to 6 House seats, but it only had 6 from 1813-1823). It lost its 2nd seat back in 1932. We probably should be grateful that it doesn’t have 6 seats today, since 3 or 4 would probably be Leahy Democrats and 1 or 2 would be a liberal RINO and 1 would be a far-left Burlington Progressive.
One also has to take into account how the size of Congress grew through the years, and then stayed stagnant at 435 starting in 1913 IIRC.
If VT had 6 seats today, then it would likely be a Republican state, since only if the state were pro-business and pro-family could it increase its population from its 1830-1930 levels to 4 million.