Through his marriage to Mildred Meriwether in 1741, Thomas Walker acquired the land comprising approximately which would become the site for Castle Hill. In its square hall, the youthful, music-loving Jefferson once played the violin, while the still younger Madison danced. Here in 1781, Walker's wife delayed the British Colonel Banastre Tarleton to give the patriot Jack Jouett time to warn Governor Thomas Jefferson and the Virginia legislators of Tarleton's plan to capture them. In addition to frequent visits by Thomas Jefferson, Castle Hill has entertained other U. S. Presidents and historic figures including George Washington, James Madison, James Monroe, Patrick Henry, Andrew Jackson, Robert E. Lee, James Buchanan, Martin Van Buren, John Tyler, Henry Clay, and Daniel Webster. The Walkers' youngest son, Francis Walker, married Jane Byrd Nelson, the daughter of Governor Thomas Nelson of Yorktown, and inherited Castle Hill. The estate was next inherited by Thomas & Mildred Walker's granddaughter, Judith Page Walker, who married U. S. Senator William Cabell Rives. William Cabell Rives studied law under Thomas Jefferson at Monticello, and was a friend of James Madison. At Castle Hill, Rives wrote a three-volume biography on Madison, entitled The Life and Times of James Madison. A close friend of Dolley Madison, Judith Rives authored the novel Home and the World, in which she wrote of life at Castle Hill as the fictitious “Avonmore.” . Colonel Alfred Landon Rives, son of William and Judith Rives, and chief of engineers to General Robert E. Lee during the American Civil War, inherited Castle Hill. After Rives died in 1903, and his widow, Sarah Catherine MacMurdo Rives, died in 1909, their daughter, Amélie Louise Rives Troubetzkoy, prominent romantic novelist, early feminist, and wife of artist, Russian Prince Pierre Troubetzkoy, inherited the property. At least three of Amélie's novels, Virginia of Virginia, The Quick or the Dead?, and Barbara Dering, are drawn from life at Castle Hill. Amélie and Castle Hill were both featured in a New York Times article published on April 15, 1906. In the early 20th century, Amélie's sister, Mrs. Gertrude Rives Potts, who managed the estate after their father's death, was recognized as the first womanMaster of Foxhounds. While at Castle Hill, Gertrude imported and trained a pack of English Foxhounds, bred and schooled her own horses, organized a hunting staff, and enlisted the consent of neighboring landowners to form a suitable country for the “Castle Hill Hounds.” Castle Hill later became part of the Keswick Hunt Club district.
Architecture
The original clapboard, colonial residence was built by Walker in 1764, with a front porch facing west and six dormer windows. William and Judith Rives added the brick, federal style addition to the home in 1824, which was built by Captain John Perry, one of Thomas Jefferson's master brickmasons. In 1844 the home's columned conservatories were added to each end of Perry's addition by another Jefferson brickmason, William B. Phillips.
Gardens
Castle Hill's formal gardens were featured in an article written by Gertrude Potts published in the Historic Gardens of Virginia. The great box-hedges which stand at the top of the gardens are considered to be some of the tallest and most impressive boxwood in Virginia. The current expanded, multi-terraced and symmetrical formal gardens were designed in 1997 by Landscape architect Rachel M. Lily.
