In 1678 a lease was granted on a plot in Soho Square to John Dunton by Richard Frith and William Pym. Dunton built two houses on the site which were later converted into a single dwelling with the address of 21 Soho Square. For the next 90 years the house was lived in by a succession of prominent figures including from 1685, James Grahme, and then briefly in 1691 by his brother Richard Graham, Viscount Preston. Between 1730 and 1734, Manor House was the home of Sir Rowland Winn 4th Bart of Nostell Priory, Yorkshire. During Winn's tenure George Vertue recorded that a "large family picture of Sir Thomas More" was hung there, this was a copy by Rowland Lockey of the painting by Hans Holbein and had been commissioned by the More family in 1592. When Winn left Manor House the painting went to Nostell Priory where it remains. Between 1772 and 1775 21 Soho Square was the location of the Spanish Embassy.
In 1776 the house, known then as The White House, was bought by Thomas Hopper, who, between 1778 and 1801 styled it as an hotel although all contemporary accounts point to its real business being as a high-classmagical brothel. The White House is described as being garishly decorated and had lavish themed rooms including the "Gold Room", "Silver Room" and "Bronze Room", a "Painted Chamber", "Grotto", "Coal Hole" and most famously the "Skeleton Room" which contained a mechanised human skeleton designed to scare the staff and patrons alike. Henry Mayhew called the White House a "notorious place of ill-fame" and wrote:
Some of the apartments, it is said, were furnished in a style of costly luxury; while others were fitted up with springs, traps, and other contrivances, so as to present no appearance other than that of an ordinary room, until the machinery was set in motion. In one room, into which some wretched girl might be introduced, on her drawing a curtain as she would be desired, a skeleton, grinning horribly, was precipitated forward, and caught the terrified creature in his, to all appearance, bony arms. In another chamber the lights grew dim, and then seemed gradually to go out. In a little time some candles, apparently self-ignited, revealed to a horror stricken woman, a black coffin, on the lid of which might be seen, in brass letters, ANNE, or whatever name it had been ascertained the poor wretch was known by. A sofa, in another part of the mansion was made to descend into some place of utter darkness; or, it was alleged, into a room in which was a store of soot or ashes.
In 1838 the house was acquired by Edmund Crosse and Thomas Blackwell of the condiment firm Crosse & Blackwell. They embarked on the first and only major rebuilding/remodelling of the original structure and created the house which we see today of four main storeys of yellow stock brick with a four-window wide façade fronting Soho Square; the front door was previously on Sutton Row as shown in an 1826 watercolour by T. Richardson. They also added a "painted cornice below the third storey and a shallower entablature and blocking above screening the slated mansard roof" as well as cast iron balconies to the first and second floors. Crosse and Blackwell operated their business from the house until 1925 and it is probable that the ground floor had some kind of shop front. Crosse and Blackwell's structure is the house we see today but for the addition of a Portland stone façade to the ground floor by the architect M. W. Watts in 1927–1928.
