Lambton married Belinda Blew-Jones on 10 August 1942. She was the daughter of Douglas Holden Blew-Jones and his wife Violet Hilda Margaret Birkin, sister of Freda Dudley Ward. They had five daughters and one son:
In 1973, Lambton's liaisons with prostitutes were revealed in the Sunday tabloid The News of the World. The husband of one of the prostitutes, Norma Levy, had secretly taken photographs of Lambton in bed with Levy and had attempted to sell the photographs to Fleet Street tabloids. As well, a police search of Lambton's home found a small amount of cannabis. On 22 May, Lambton resigned from both his office and Parliament, which caused a by-election for his seat which was won by Alan Beith for the Liberal Party. Shortly after, the name Jellicoe emerged in connection to a rendezvous for one of Norma's girls at a Somers Town mansion block which had been named Jellicoe House, after the earl's kinsman Basil Jellicoe, the housing reformer and priest from Magdalen College. There was a confusion and Lord Jellicoe, the Lord Privy Seal and Leader of the House of Lords, admitted 'casual affairs' with prostitutes from a Mayfair escort agency but denied knowing Norma Levy. A security inquiry on the prostitution scandal concluded that there had been "nothing in conduct to suggest that the risk of indiscretions on these occasions was other than negligible." Lambton stated that he had never taken his red state boxes of government documents with him when he visited Norma Levy. The security inquiry was held due to fears that the prostitution scandal may have involved an actual or potential breach of national security. When Lambton was interviewed by MI5 officer Charles Elwell, Lambton first claimed that the pressure of his job as a minister was what drove him to procure the prostitutes. Later, Lambton stated that his sense of "the futility of the job" and lack of demanding tasks as a junior minister were reasons he went to prostitutes. Finally, Lambton claimed that his judgment was faulty when he went to the prostitutes due to his obsession with the battle over the use of an aristocratic title that had been used by his father; Lambton claimed that he sought to soothe this obsession by engaging in frantic activities such as gardening and debauchery.
Later years
Following the scandal, Lambton retired, separated from his wife and bought Villa Cetinale, a 17th-century villa in Tuscany, where he lived with Claire Ward, née Baring, mother of actress Rachel Ward and daughter of the cricketer Giles Baring. He never divorced his wife Bindy, who died in 2003. In 1991, he made an extended appearance on the TV discussion programme After Dark, chaired by Baroness Helena Kennedy, alongside Duncan Campbell, Jane Moore, Clare Short, Anthony Howard and others. Despite renouncing his titles, he continued to use his former courtesy title Viscount Lambton, although, since it was now a title that had passed by courtesy to his eldest son, it was argued by Sir Anthony Wagner and others that this was incorrect. Lambton died in hospital in Siena, Italy, on 30 December 2006.