Philadelphia was the third-largest metro area in the U.S. before the 1950 census. It has been slowly losing ground ever since then. It also has the disadvantage of overlapping with the much larger metro NYC sports market to a degree.
It's gotten to the point where it may not even be considered a "large" sports market anymore. Philadelphia is now the 6th largest city in the U.S. LA, Houston, and Phoenix have past it since 1950. However, the Phildelphia media market is fourth largest in the U.S., ahead of Houston and Phoenix, since its suburbs, including in NJ count in this category:
Ranking the U.S.' largest media markets MLB has 30 teams in 27 cities. Philadelphia is the largest media market to have only one team, but you think the fourth largest media market isn't a "large" sports market. Of course, you have strange ideas about many things...
I know Pennsylvania well. Half the people I know in the Philadelphia metro area are fans of NY/NJ teams.
Before they got Bryce Harper a few years ago, the Phillies were usually in the bottom third of the major leagues in payroll. They never seemed to be in the hunt for the top free agents at any given time -- the Gerritt Cole, the Alex Rodriguez, the Manny Ramirez, the Jason Giambi, the Max Scherzer, etc.
Sometimes size doesn't mean as much in the context of specific sports. Montreal is one of the biggest markets for hockey but failed miserably as a MLB market. St. Louis is a huge baseball market but can't keep an NFL team for its life. And for years, Los Angeles seemed to be a giant market for every sport EXCEPT the NFL.