Good try -- but that's not possible.
James substituted the words: "after this I will return" for the words of Amos 9: "in that day", thus meaning that "that day" of Amos 9 would be the day when Jesus "will return".
And let's be clear -- His first coming cannot possibly be called "a return" visit. His second coming will be that day of His return.
The part that you seem to be missing is the fact that James links the calling of the gentiles what was happening in that day with the rebuilding of the tabernacle of David. He explicitly says "And I will set it up; So that the rest of mankind may seek the Lord,"
The "tabernacle of David" needed to be rebuilt "so that" the nations might be called. There is no other reason for James to quote Amos unless this if-then connection is somehow true at that time.
So you can either deny the immediate connection between the "rebuilding" and the calling of the gentiles, or you can face the fact that James didnt have some far off, futurist physical rebuilding in view.