On June 18, 1793, she married the 9th Governor of Virginia, the widower Henry "Light Horse Harry" Lee, in the mansion's parlor. After Lee retired from the governorship in December 1794, she followed him to his Lee family holdings in northern Virginia. Six children were born to this marriage:
Catharine Mildred Lee, who married Edward Vernon Childe.
Anne Carter Lee's time as a wealthy patrician of northern Virginia would, however, be short. Their principal plantation, Stratford Hall, was relinquished in 1808 by entail to her stepson Henry Lee IV. Her husband had suffered repeated financial reverses, characterized by poor investments, and was forced to declare bankruptcy in 1809 and move to debtor's prison. Anne Carter and "Light Horse Harry" were separated for one year while the husband was under confinement. In 1810, the reunited couple tried to resume life together in Alexandria. Despite the fall in his honor and standing, the former governor "Light Horse Harry" had maintained fervent ties to the Federalist Party, and passionately opposed the decision by the other U.S. political party, the Jeffersonians, to declare war on Great Britain in the War of 1812. In a July 1812 outbreak of political violence in Baltimore, War Hawk rioters raided a Federalist strongpoint and tried to lynch its defenders, including Harry Lee. Anne Carter Lee was forced to learn that her husband had suffered serious physical and psychological wounds in the mob outburst. As life for him in Alexandria had become impossible, Henry Lee was forced to leave his wife and family. Lee emigrated to the South Atlantic coast and the Caribbean in a series of futile attempts to find a place to recuperate from his injuries. He died on the Georgia coast on 25 March 1818, leaving Anne Carter Lee a middle-class widow.
Widowhood
As a widow and the head of her household, Anne Carter Lee brought up her surviving children during her remaining eleven years of life. A small bequest from her Carter family enabled her to maintain the house in modest comfort. She suffered, however, from chronic health conditions herself. Her medical diagnosis, if any, is unknown, and her health challenges were primarily recorded as an additional obstacle to the young life of her fourth son Robert. Born in 1807, in his adolescent years Robert E. Lee assisted his mother to run the household. Although not poor, she was not able to face college tuition bills for Robert, who therefore sought higher education opportunities at the United States Military Academy at West Point. Young Lee did not hesitate to use his mother's Carter family kinship ties to co-sign his West Point application. During young Lee's West Point years, Anne Carter Lee's health further declined. She clung to life until the graduation of her son, and was given a place to live and be nursed at the home of an uncle, William Henry Fitzhugh, the Fairfax County plantation of Ravensworth. Anne Hill Carter Lee died in Ravensworth on 26 July 1829. Her son Robert named one of his daughters, Anne Carter Lee, in honor of his mother.