Paul was speaking of the appointed Days, or Times. They are not fasts, they are Feasts.
I don't see that either but what I do see from Paul's words is that the entire conversation is about food. fasting and eating that which is sacrificed to idols. We know from written history that there were days commanded by he rabbinics for fasting. We also know that often meat sold in the market had been sacrificed to false gods before being brought to the market. So if someone who was new to the Way of the Word of God and chose to not eat the meat in the market for fear it may have been offered up to false gods prior to selling it, then one shouldn't hold it against that person and it is for the stronger in the faith to set aside that strength and not eat the meat lest it make the weak one even weaker.
Regardless, none of it had a lick to do with which day of the week is His Sabbath. There is a truth that can be gleamed from silence and that truth is the fact that Jesus never once had to correct anyone about the day of the Sabbath as the seventh day had never been changed by the dogma of man as was testified by the Word of God through Moses when the mixed multitude came out of Egypt and the record had to be set straight and then written down forever in stone.
Exo 16:4 Then said the LORD unto Moses, Behold, I will rain bread from heaven for you; and the people shall go out and gather a certain rate every day, that I may prove them, whether they will walk in my law, or no. 5 And it shall come to pass, that on the sixth day they shall prepare that which they bring in; and it shall be twice as much as they gather daily.
Exo 16:16 This is the thing which the LORD hath commanded, Gather of it every man according to his eating, an omer for every man, according to the number of your persons; take ye every man for them which are in his tents. 17 And the children of Israel did so, and gathered, some more, some less. 18 And when they did mete it with an omer, he that gathered much had nothing over, and he that gathered little had no lack; they gathered every man according to his eating. 19 And Moses said, Let no man leave of it till the morning. 20 Notwithstanding they hearkened not unto Moses; but some of them left of it until the morning, and it bred worms, and stank: and Moses was wroth with them.
Exo 16:22 And it came to pass, that on the sixth day they gathered twice as much bread, two omers for one man: and all the rulers of the congregation came and told Moses. 23 And he said unto them, This is that which the LORD hath said, To morrow is the rest of the holy sabbath unto the LORD: bake that which ye will bake to day, and seethe that ye will seethe; and that which remaineth over lay up for you to be kept until the morning. 24 And they laid it up till the morning, as Moses bade: and it did not stink, neither was there any worm therein. 25 And Moses said, Eat that to day; for to day is a sabbath unto the LORD: to day ye shall not find it in the field. 26 Six days ye shall gather it; but on the seventh day, which is the sabbath, in it there shall be none. 27 And it came to pass, that there went out some of the people on the seventh day for to gather, and they found none. 28 And the LORD said unto Moses, How long refuse ye to keep my commandments and my laws?