Jefferies made more than 300 competitive appearances for Heart of Midlothian. The main highlight of his playing career was playing in the 1976 Scottish Cup Final, which Hearts lost 3–1 to Rangers. He also played in the famous 0-7 New Year Edinburgh Derby defeat against Hibernian. Jefferies eventually left Hearts in 1981, and spent the last two seasons of his career with Berwick Rangers.
Jefferies returned to the Wee Gers as manager in September 1988. Despite a great deal of financial turmoil during that time, he turned the struggling team around to the extent that they set a club record of 21 games unbeaten in the league during season 1988–89.
In August 1995, Jefferies returned to Hearts. He was manager of the Hearts team that won the Scottish Cup in 1998, his greatest success in the game to date.
Bradford City
Jefferies moved south of the border on 20 November 2000 to replace Chris Hutchings as manager of then Premier League side Bradford City. He was given the task of selling players by chairman Geoffrey Richmond and was unable to prevent them from going down. He resigned in December 2001, after a poor start to the season had left Bradford with no hope of a promotion challenge. Former England striker Stan Collymore later described Jefferies as "one of the most useless managers worked under" on Talksport in January 2013. He criticised Jefferies' training regime as "being from 1975".
Kilmarnock
On 28 February 2002, he returned to management back in his native Scotland with Kilmarnock. He kept Kilmarnock in a respectable position despite the necessity of drastically reducing the club's wage bill, reaching the 2007 Scottish League Cup Final. Following Alex McLeish's departure from Rangers at the end of the 2005–06 season, Jefferies was the longest-serving manager in the Scottish Premier League. He left Kilmarnock by "mutual consent" on 11 January 2010.
Return to Heart Midlothian
Jefferies was appointed manager of Hearts for a second time on 29 January 2010, just hours after Csaba László was sacked from the position. Hearts finished third in the SPL in the 2010–11 season, having threatened the dominance of the Old Firm until falling away after February. Jefferies and right-hand man Billy Brown were sacked by Hearts on 1 August 2011, after just two games of the 2011–12 Scottish Premier League season.
Jefferies held talks with Dunfermline Athletic about succeeding Jim McIntyre as their manager and was appointed on 20 March. He was unable to keep the Pars in the top flight and were relegated at the end of the season. The following season in the Scottish First Division the club ran into financial difficulties and were placed in administration in March 2013. This led to a 15-point deduction penalty by the Scottish Football League as well as many first-team players leaving. The effect of the points penalty saw Jefferies' side relegated to the Scottish Second Division through the Scottish First Division play-offs losing to Alloa Athletic in the two-legged final. Jefferies signed a new contract with Dunfermline in December 2013, after the club exited administration. He resigned as manager in December 2014 stating that he felt there was no other option than for him to leave the club. He also stated that Dunfermline was likely to be his last job in management.
In February 2017, Jefferies joined the board of directors at Edinburgh City in a "sporting director" capacity. In September 2019 he suffered a heart attack whilst golfing. In July 2020, the club confirmed that Jeffries had departed in his role as sporting director to return to Heart of Midlothian FC.
Managerial honours
;Gala Fairydean Rovers
Scottish Qualifying Cup South: 1983, 1984, 1985, 1987
