A manor on this site was inherited by John Trussell from his father William in 1481. After John's death in 1499, his daughter, Elizabeth Trussell became the ward of John de Vere, afterwards 15th Earl of Oxford. Elizabeth later became de Vere's second wife. The manor of Bilton was held by this family line for nearly 70 years. In 1574 Edward de Vere, 17th Earl of Oxford, leased it to John, Lord Darcye, and in 1580 he sold it to John Shuckburgh, who immediately leased it to Edward Cordell. When John Shuckburgh died in 1599, the manor was inherited by his eldest son Henry who sold it to Edward Boughton of Lawford Hall in 1610. Boughton rebuilt the manor, creating the central part of the current red brick and sandstone house in about 1623. He intended it for use by his son William. After the death of his father, William was created Baronet Boughton in 1642. The house was generally occupied by junior members of the Boughton family and was sold by Edward Boughton in 1711 to the essayist and poet Joseph Addison, who wrote his book Evidences of Christianity while living there. Following Addison's death in 1719 the house was occupied by Addison family members until it was sold, about 1799, to John Bridgeman Simpson of Babworh Hall, Babworth, a son of Henry Bridgeman, 1st Baron Bradford. A drawing by Turner may depict the hall c.1815. The British census of 1881 records that two of Simpson's unmarried daughters remained in residence. Apart from Addison, other noted residents include the sports writer Charles James Apperley, known as "Nimrod", and Henry Holyoake, who was the rector from 1705 to 1731. During World War I the house was in use as a military hospital. After World War II it was converted into flats.
The building
The original building created by Boughton has been modified and extended several times. It was probably once in a regular H plan, but has since lost its symmetry. Black's picturesque guide to Warwickshire described its appearance in 1857:
The mansion is spacious but irregular. In construction it is of different periods. The oldest and the largest portion bearing marks of the style of architecture common about the time of James I. The remainder of the building consists of a lower range of apartments facing the gardens. This part of the house being of the style which prevailed in the beginning of the eighteenth century may have been erected by Addison himself when preparing the building for the reception of his destined wife.
Entrance gates erected by Addison, bearing his initials and those of his wife Charlotte, Countess of Warwick were moved into the garden in 1825.