Another reason could be is that perhaps James wasn’t a believer at this point in his life. That too. Of the eleven Apostles remaining, only John even showed up. We have the accounts of Jesus getting grief from his family. A few:
After this, Jesus went around in Galilee. He did not want to go about in Judea because the Jewish leaders there were looking for a way to kill him. But when the Jewish Festival of Tabernacles was near, Jesus’ brothers said to him, “Leave Galilee and go to Judea, so that your disciples there may see the works you do. No one who wants to become a public figure acts in secret. Since you are doing these things, show yourself to the world.” For even his own brothers did not believe in him. (John 7:1-5) Then Jesus went home, and once again a crowd gathered, so that He and His disciples could not even eat. When His family heard about this, they went out to take custody of Him, saying, “He is out of His mind.” (Mark 3:20,21)