Never ask a question that you haven't looked to see if there are other answers which in fact make more sense than the one you are extending. A couple of observations here:
1) James is quoting Amos, Amos is NOT quoting James. For your scenario to be true AMOS would have to be quoting, cognizant of, or responding to the statement by James in v 14. This is, simply, nonsense. AFTER THESE THINGS refers NOT to the "things" James quoted in v 14, but to the "things" AMOS (remember, that is who James was quoting) had referred to earlier, which was the judgment and dispersion of Israel. If you read Amos 9 (remember, James is quoting Amos, Amos is NOT quoting James re: the statement of Peter about the Gentiles coming to faith), you will find that there is the typical apocalyptic scenario. Israel will be judged (see verses 9-10), but "in that day, or after these things" God will preserve a remnant who will, under His mighty hand, find themselves under the kingship of David and rule over the nations in a golden age of grace and God's favor. In Amos 9, this takes the form of the restoration of the "fallen tabernacle of David" so that "Edom" is subjected to the rule of the covenant people. Amos further explains this by expanding the definition of "edom" to show that it represents all the heathen (Gentiles). That is the clean and clear flow of thought in Amos. James uses the LXX to frame his quote, which might be rendered "after this" (Hebrew is "in that day"). Any time reference from Amos HAS TO refer to AMOS, and not to the words of Peter, as if Amos was commenting on that. That would be just silly. Again, READ AMOS on this, and not just some dispensational commentator, who has an axe to grind.
2) The second reason (not as important as the first, but worth something) this is not a valid construction of James's speech in Acts 15 is the fact that NO ONE PROPOSED THIS KIND OF CONVOLUTED REASONING BEFORE DISPENSATIONALISM CAME ON THE SCENE. I looked up Wesley (John), and he doesn't make this into some silly two stage reference to God saving the Gentiles now and then "promising millenial blessings." Ditto for Matthew Henry, John Calvin, Martin Luther, Toplady, John Gill, JA Alexander (whom Spurgeon termed the best commentary on Acts), JW McGarvey, and other "older commentators. THERE IS NOT ONE MAN WHO MENTIONS THIS KIND OF INTERPRETATION? ALL of the ancients saw this as a fulfilled prophecy and James treating it as a fulfilled (past tense) prophecy. One must ask the question, "why is that?" The answer is quite clear that no one would come up with such a twisted view where (again) one must make Amos refer back to PETER from the mouth of James in order to make the argument. That is, unless you had a vested interest in NOT going there, because you recognize that this will in fact establish what you cannot permit, the horrid "allegorizing" of prophecy. Yet here we have it. It is the clear meaning of the text (READ IT AGAIN before you argue with me about it, please), it is the UNIVERSAL VIEW of men who commented on it before the rise of dispensationalism, and (finally) some of your better modern commentators (FF BRUCE, John Lane and others) have demolished this artifical construct far better than I.
I have great respect for anyone who looks at the Bible, and I DO understand how one's emotional commitments make it difficult to "see the other side" of things. I have had personal commitments to men that absolutely prevented me from shifting my perspective. That happens to all of us. However, there is simply no other explanation for what is happening here. It is clear that James has given us an example in the very first church council, no less, of the fact that apocalyptic passages in the OT were in fact MEANT to look for their fulfillment in the new people of God, and that "literal" passages are in fact interpreted for us by the Holy Spirit to have a figurative fulfillment. This is by no means the only one. The whole New Testament is filled with examples of this claimiing the right to re-interpret the OT prophetic passages and find their fulfillment in the "Israel of God" or the "true circumcision" or the "remnant" or any number of other ways that God says over and over and over and over "ONE PEOPLE, ONE PROMISE, ONE RESPONSE, ONE FAITH, ONE DESTINY."
This is why I cannot be a dispensationalist. It is, in fact, arguing with the Holy Spirit as he says that the fallen tabernacle of David (there is no reference to "throne" here, btw) HAS been raised up, and the worldwide entrance of Gentiles into God's forever family IS the fulfillment of Amos 9:12. Resisting that means, as you have done, making Amos respond to James, which is clearly impossible. Think about it.
"8Behold, the eyes of the Lord GOD are upon the sinful kingdom, and I will destroy it from off the face of the earth; saving that I will not utterly destroy the house of Jacob, saith the LORD. 9For, lo, I will command, and I will sift the house of Israel among all nations, like as corn is sifted in a sieve, yet shall not the least grain fall upon the earth. 10All the sinners of my people shall die by the sword, which say, The evil shall not overtake nor prevent us."
Has this above all been fulfilled yet? or is it being fulfilled now and more in the future. As a matter of fact, while the house of Israel is being sifted through the nations, God is visiting those nations to take out of them a people for his name, just as James said. And "after this" includes both the sifting of Israel through the nations as well as the taking out of those Gentile nations a people for his name. They have been going on simultaneously for centuries.
Furthermore James may be quoting just Amos here, but he is acknowledging that he did his homework and read all the prophets from Isaiah to Jeremiah to Ezekiel to Zechariah to Malachi ..., and they all prophesied that what Peter said was happening to the Gentiles would happen and would continue to happen until it was finally finished, then He would return.
Not just Amos but all the prophets were in agreement with what James and Peter said, prophets like Isaiah:
"For the Lord will have mercy on Jacob, and will yet choose Israel, and set them in their own land, and the sojourners shall be joined with them and they shall cling to the house of Jacob".
And Zechariah: "I will dwell in the midst of thee [O daughter of Zion]. And many nations shall be joined to the Lord in that day, and shall be my people ... and the Lord shall inherit Judah as his portion in the holy land, and choose Jerusalem again".
And Amos: " 11In that day will I raise up the tabernacle of David that is fallen, and close up the breaches thereof; and I will raise up his ruins, and I will build it as in the days of old: 12That they may possess the remnant of Edom [should be men not Edom], and of all the heathen, which are called by my name, saith the LORD that doeth this. 13Behold, the days come, saith the LORD, that the plowman shall overtake the reaper, and the treader of grapes him that soweth seed; and the mountains shall drop sweet wine, and all the hills shall melt. 14And I will bring again the captivity of my people of Israel, and they shall build the waste cities, and inhabit them; and they shall plant vineyards, and drink the wine thereof; they shall also make gardens, and eat the fruit of them. 15And I will plant them upon their land, and they shall no more be pulled up out of their land which I have given them, saith the LORD thy God."