From what I can tell, the Proud Boys are white supremacists and Antifa was there to confront them. Not sure where the Patriot Prayer Group fits in.
I think the prayer group had the permit and the Proud Boys ‘attached’ themselves to the group.
Seems there were 3 groups there: Patriot Prayer Group, Proud Boys, and Antifa.
https://www.mercurynews.com/2017/08/23/whos-behind-this-weekends-right-wing-rally-at-crissy-field/