The son of James Caulfeild, 3rd Viscount Charlemont, he was born in Dublin, and succeeded his father as 4th Viscount in 1734. His mother was Elizabeth Bernard, daughter of Francis Bernard, MP and judge of the Court of Common Pleas and Alice Ludlow. After his father's death she remarried Thomas Adderley, and died in childbirth in 1743 at the age of 40, after the birth of her daughter Elizabeth, who later married Major David Ross. The title of Charlemont descended from Sir Toby Caulfeild of Oxfordshire, England, who was given lands in Ireland, and created Baron Charlemont, for his services to King James I in 1620. The 1st Viscount was the 5th Baron, who was advanced in the peerage by Charles II. In 1768 he married Mary Hickman, daughter of Thomas Hickmnan of Brickhill, County Clare. The marriage is said to have been a very happy one. They had two sons, James and Henry. Lord Charlemont was well known for his love of Classical art and culture and spent nine years on the Grand Tour in Italy, Greece, Turkey and Egypt. He returned to Dublin and employed the Scottish architectSir William Chambers to remodel his main residenceMarino House, to design his town houseCharlemont House and the unique Neo-Classical garden pavilion building, the Casino at Marino. , c. 1753-56. Lord Charlemont is historically interesting for his political connection with Henry Flood and Henry Grattan; he was a cultivated man with literary and artistic tastes, and both in Dublin and in London he had considerable social influence. He was the first President of the Royal Irish Academy and was a member of the Royal Dublin Society. He was appointed Custos Rotulorum of County Armagh for life in 1760. For various early services in Ireland he was made an earl in 1763, but he disregarded court favours and cordially joined Grattan in 1780 in the assertion of Irish independence. In 1783 he was made a founding Knight of the Order of St Patrick. He was president of the volunteer convention in Dublin in November 1783, having taken a leading part in the formation of the Irish Volunteers; and he was a strong opponent of the proposals for the Union. His eldest son, who succeeded him, was subsequently created an English Baron. His half-sister, Elizabeth Adderley, was the mother ofMajor-General Robert Ross. The somewhat inaccurate Memoirs of the Political and Private Life of James Caulfield, Earl of Charlemont, Knight of St. Patrick, by Francis Hardy, appeared in 1810.