“If he had not married who he did he would have still been an unknown, sorting toilet paper in some no name post.”
His service in the U. S. Armed Forces before the Civil War disproves this statement in and of itself. His private life both before and after his service is more than sufficient evidence to make your statement ridiculous.
RE Lee’s family was disgraced, as Light Horse Harry Lee had died a debtor. RE Lee married a cousin who was thus part of the family, and as disgraced as he.