The second son of Charles FitzRoy, 1st Baron Southampton and Anne Warren, the daughter and co-heir of Peter Warren and a descendant of the Schuyler family, the Van Cortlandt family, and the Delancey family, all from British North America., FitzRoy entered the Army on 27 August 1779 with a commission as cornet in the 11th Dragoons, and in 1782 was made aide-de-camp to General Richard Whyte at Jersey. He was promoted to lieutenant in a newly raised corps, and on 26 March 1783 to the rank of captain-lieutenant in the 96th Regiment of Foot; when the regiment was reduced following the Peace of Paris the same year, he was placed on half-pay. He was made a captain in the 66th Regiment of Foot in 1787, and lieutenant and captain in the 1st Regiment of Foot Guards on 9 July 1788. In 1793 he served in Flanders, including at the siege of Valenciennes and the siege of Dunkirk. On 21 February 1794 he was made captain of a company in his regiment, with the rank of lieutenant-colonel in the Army, and in May he returned to England. FitzRoy was granted brevet rank as colonel on 26 January 1797, and in 1799 commanded the grenadier company of the 1st Foot Guards during the expedition to Holland. On 5 January 1801 he was made an extra equerry to the King. He was later appointed aide-de-camp to the King, and major-general on 25 September 1803. He served on the staff of the Eastern District, and from 1804 of the Western District. On 25 March 1805 he was made colonel of the 25th Regiment of Foot, on 27 August 1809 appointed regular equerry to the King, and on 25 July 1810 he was promoted lieutenant-general. He had resigned his post as an equerry by 11 October 1811, when he was replaced by Brent Spencer. On 21 September 1816 he married Eliza, widow of Clavering Savage; they had no children. He was promoted to general on 19 July 1821. Fitzroy is probably best known for his romance with George III's favourite daughter Princess Amelia. Princess Amelia of the United Kingdom fell in love with Charles Fitzroy, desiring to marry him. Her mother, The Queen,, was told of the affair by a servant, but turned ablind eye. It was hoped that such discretion would prevent the King from discovering the liaison, which may have risked sending him into one of the bouts of mental illness to which he was becoming increasingly prone. Though she never gave up hope of marrying him, Amelia knew she could not legally marry FitzRoy due to the provisions of the Royal Marriages Act passed by her father's Parliament. She would later tell her brother Frederick that she considered herself to be married, taking the initials A. F. R.. Upon the untimely death of Princess Amelia at the age of 27, on 2 November 1810, her will dictated that all of her possessions be given to Charles FitzRoy.