The castle was founded by the De Lisle family in the 12th century, and then passed to Sir William de Aldeburgh, following his marriage to Elizabeth de Lisle, heiress of Harewood, who was granted a licence to crenellate in 1366. The rectangular tower house on a steep slope is visible for miles around. The main block of two storeys is flanked by four angle towers, one being a plain entrance tower; the chapel is situated over the portcullis chamber. The lower kitchen wing is of four storeys, with a barrel-vaulted basement containing the well. When the second Baron Aldeburgh died in 1391 without issue, the castle transferred to the Ryther and the Redmayne families, into which his two daughters had married. In 1574, James Ryther and partner William Plompton bought out the Redman family, although Ryther's financial situation must have worsened because he died in London's Fleet Prison in 1595. His son and two daughters sold the castle to Sir William Wentworth of Gawthorpe Hall in 1600 to clear debts; this is probably when Harewood Castle ceased to be a main residence. The castle was last occupied in the 1630s, and in 1656 it was put up for sale as an 'upstanding source of stone and timber'. The Wentworths sold Harewood and Gawthorpe to Sir John Cutler, 1st Baronet, by which time the castle had probably already been partly dismantled though Cutler lived there at the end of his life. At Cutler's death in 1693, it passed to his only surviving daughter, Elizabeth, Countess of Radnor and on her death without heir to Cutler's nephew, unmarried Edmund Boulter MP of Boston and Wimpole Hall then to his nephew John Boulter of Gawthorpe and Westminster, who died unmarried in 1738. His executors sold to Henry Lascelles whose son, Edwin Lascelles, 1st Baron Harewood built Harewood House. Centuries after it had been abandoned, Harewood Castle remained a landmark, the subject of several paintings in the 1797 by J. M. W. Turner.
Present
Recent times saw decay and weathering take their toll, and the unstable castle was placed on the Buildings at Risk Register by English Heritage. Then, around the year 2000, a £1 million rescue plan was drawn up, funded partly through English Heritage and partly by the Harewood Estate. As of 2008, the restoration project, which involved architects, geologists, structural engineers, ecologists and staff of the Harewood Estate, Historic Property Restoration Ltd and English Heritage, was nearing completion. The castle was taken off the Buildings at Risk Register in that year.