Under Margaret Thatcher he was Secretary of State for Employment from May 1979 to 14 September 1981. Thatcher said of their relationship, "we agreed that trade unions had acquired far too many powers and privileges. We also agreed that these must be dealt with one step at a time. But when it came down to specific measures, there was deep disagreement about how fast and how far to move." Prior is believed to have annoyed Thatcher by being too friendly with trade union leaders, with Thatcher writing "He had forged good relations with a number of trade union leaders whose practical value he perhaps overestimated." And during his period in the Cabinet, he is believed to have angered the right wing of his party and the Prime Minister for not pressing far enough with anti-trade union legislation. In the September 1981 cabinet reshuffle Prior was moved from the Employment portfolio to become Secretary of State for Northern Ireland, an office he held until September 1984. At the time of the reshuffle, it was reported that Prior considered following the sacked Ian Gilmour to the back benches to oppose the Thatcher Government's economic policies. However Prior ultimately decided to accept being moved to the Northern Ireland Office after consulting cabinet colleagues William Whitelaw, then Deputy Leader of the Conservative Party, and Francis Pym. This transfer was widely seen as a move by Thatcher to isolate Prior, who disagreed with her on a number of economic issues. The post of Secretary of State for Northern Ireland was seen as a dumping ground to marginalise ministers. However, when Prior resigned, Thatcher revealed that she was going to offer him another Cabinet post during the reshuffle, which would have very likely been a non-economic one.
Later years
In 1986, he collaborated with John Cassels and Pauline Perry to create the Council for Industry and Higher Education, which would become the National Centre for Universities and Business in 2013. He retired from Parliament in 1987, and was created a life peer as Baron Prior, of Brampton in the County of Suffolk on 14 October 1987. He was chairman and later vice-president of the Rural Housing Trust. After his retirement from politics he was much sought after in the world of business: serving as chairman of both GEC and Allders and he had directorships at Barclays, Sainsburys and United Biscuits. Prior was interviewed about the rise of Thatcherism for the 2006 BBCTV documentary series Tory! Tory! Tory! and in 2012 as part of The History of Parliament's oral history project.
In January 1954 Prior married Jane Primrose Gifford Lywood, daughter of Air Vice-Marshal Oswyn George William Gifford Lywood, CB, CBE, a developer of the Typex cypher machines, of a landed gentry family of Woodlands, near Sevenoaks, Kent. They had four children. Prior's oldest son David Prior held the seat of North Norfolk between 1997–2001 and was appointed Parliamentary-Under Secretary of State for NHS Productivity, resulting in his elevation to the peerage in his own right as Baron Prior of Brampton, in May 2015. Prior died on 12 December 2016 at the age of 89. After Prior's death MP Keith Simpson, said of him: "In many ways he was a larger than life figure. He had a ruddy face, he played up to being the farmer. People underestimated him because he didn’t claim to be a Keith Joseph or Enoch Powell parading their intellectualism. But he was somebody who was well-loved by the grassroots and was a decent man who was in politics out of a sense of public service."