The land was originally sold by Sir Richard Grosvenor, 4th Bart. to the Commission for Building Fifty New Churches in 1723 as part of his development of the area around Mount Street, to serve as a burial ground for the parish church, St. George's, located in Hanover Square. Beside the burial ground, on South Audley Street, was planned the Grosvenor Chapel, but this was not built until 7 years after the burial ground was laid out, in 1730. To the north of the ground, the site of what is now 103 Mount Street, was the parish's workhouse, wherein the poor were given work, board, and lodgings, built in 1725 and then enlarged in the 1780s to provide room for the parish watchmen's watch-house and the parish office. In the 1850s, Parliament passed a series of Burial Acts by which burial grounds in central London, such as that of St. George's, were to be closed due to health risks. The parish's other burial ground, on Bayswater Road, closed shortly afterwards, and the City of Westminster Cemetery, Hanwell in west London was used instead. As the population of London grew, so did the demand upon the workhouse, and it became so overcrowded that in 1871 the decision was taken to move the workhouse to a new site, spurred on also by plans to widen Mount Street. In 1883, some land for the new workhouse on Buckingham Palace Road was bought from the new Duke of Westminster, Hugh Grosvenor, thricely-great-grand-nephew of the Baronet who originally sold the site of the burial ground, and paupers were transferred both to there and the parish's other workhouse at Little Chelsea on the Fulham Road, with the workhouse finally being demolished in 1886 and converted into more parish offices.
After the demolition of the workhouse, the whole area was somewhat redeveloped; Carlos Place was to be extended to the south of Mount Street over the site of the workhouse's chapel for access to the Church of the Immaculate Conception on Farm Street. The church, built between 1844 and 1849, was in a decorated gothic-style, designed by Joseph John Scoles, and with the altar by A.W.N. Pugin, and is now a Grade II* listed building. However, the new road planned through the disused burial ground was never built. The gardens' footpaths are those originally created in 1889, and the bronze drinking fountain of a rearing horse was designed for the local estate agent Henry Lofts in 1891 by Harold Peto and Sir Ernest George, who was also the designer of 104-111 Mount Street. As of 2005, the fountain is largely removed for restoration. The gravestones had been removed to the gardens' tool shed, but when the shed itself was demolished in 1931 to make way for the Mayfair telephone exchange between the gardens and Farm Street, the City Engineer's office made a copy of the inscriptions, now stored in City of Westminster Archives Centre, along with other records from the Parish. The paths in the gardens are lined with benches, many of which have been donated by the families of US citizens who have enjoyed the gardens whilst based at the nearby US embassy and other US institutions nearby.
Flora and fauna
The garden has a number of London plane trees, the predominant tree throughout central London due to their ability to cope with the capital's formerly heavily polluted air. Other trees, some able to grow only due to the gardens' sheltered spot and warm climate, include an Australian silver wattle and a Canary Islandsdate palm, three dawn redwood from south-east China, and a form of twisted willow, Salix matsudana 'Tortuosa', from northern China, as well as a chusan palm, a common palm in the UK due to its hardiness. Underneath these trees are planted a number of shrubs, most noticeably laurels and hollies, and several camellias and a Fatsia japonica. The gardens provides homes for a number of birds, including great tits, robins, magpies, and blackbirds.