Baker signed for Aston Villa as a 13-year-old in 2004, and rose through the ranks at the club, making his Academy debut in 2007. He was a pivotal figure in the centre of defence alongside Ciaran Clark in the 2007–08 season, as Villa clinched the national title, defeating Manchester City. Baker finished the season with 1 goal from 23 appearances for the Academy team, and 2 goals from 7 appearances for the Reserves, who also tasted success by clinching the Premier Reserve League South. The defender progressed to the first-team in the summer of 2008 by making the squad for the tour of Switzerland, of which Baker played in both matches against FC Wil and FC Zürich. In July 2008, Baker was named on the bench for the second leg of the UEFA Intertoto Cup tie against Odense Boldklub. A week later he was again named on the bench, this time for the trip to Icelandic side Fimleikafélag Hafnarfjarðar in the Second qualifying round of the UEFA Cup. On 23 October 2009, Baker signed for Lincoln City on a month-long loan deal alongside fellow defender Eric Lichaj. Lincoln City manager and former Aston Villa striker Chris Sutton handed Baker his league debut at Sincil Bank on 24 October 2009, in a 0–0 draw with Torquay United. Baker impressed at Lincoln, and his loan deal was eventually extended to the end of the 2009–10 season. Baker was banned from attending Aston Villa's League Cup final against Manchester United at Wembley Stadium on 28 February 2010 after being caught trying to sell his allocation of tickets via the social networking website Facebook. The 18-year-old defender attempted to sell five tickets at a price of £200 each, but later said that he regretted his actions.
2010–11 season
Baker was brought in to the Aston Villa squad for the derby match against Birmingham City on 16 January 2011. He was an unused substitute as Villa drew 1–1. Baker made his starting debut against Wigan Athletic on 25 January 2011 at DW Stadium. On 29 January 2011, his second ever start, Baker gained an assist for the third goal but was then sent off for a dangerous tackle in an FA Cup Fourth Round tie against Blackburn Rovers. On 26 February, Baker made his second league start for Villa against Blackburn Rovers, but went off injured in the first half after a collision with goal keeper Paul Robinson. Villa announced on 2 June 2011 that Baker had signed a new three-year contract with the club.
2011–12 season
On 22 November he joined Millwall on a one-month loan deal. He was assigned the squad number 27 and made his debut four days later, helping Millwall keep a clean sheet against Crystal Palace. He made six league appearances before his loan spell ended in December with a 1–0 win against Portsmouth on 26 December being his last game. He returned to Villa and made his first appearance of the season coming on as a late sub for Richard Dunne against Manchester City on 12 February.
2012–13 season
During the 2012–13 season, Baker established himself as a centre back with great aerial ability. Due to an injury to Ron Vlaar, Baker was partnered with fellow academy graduate Ciaran Clark and impressed during a difficult spell where Villa conceded lots of goals. On occasion he also played at left back in the absence of Joe Bennett. On 9 March, manager Paul Lambert praised his maturity for producing another outstanding performance after slicing the ball into his own net to give Reading a first-half lead. In May 2013 Baker signed a new three-year deal with Aston Villa which would keep him there until 2016.
2013–14 season
Baker made his first appearance against Arsenal, before getting subbed off due to injury in the 17th minute. He missed the next games against Chelsea, Liverpool and Newcastle before making his return to the squad as an unused substitute against Norwich. He made his full return starting at Villa Park in a dramatic 3–2 win over Manchester City, before being dropped to the bench against Hull. After that, Baker and Vlaar were Paul Lambert's preferred defensive partnership, however, due to Vlaar's constant injury problems, Baker was often partnered with fellow academy graduate, Ciaran Clark.
2014–15 season
Although Baker was a regular throughout the 2013–14 season, Baker was an unused substitute against Villa's opening fixture, a 1–0 Victory against Stoke. He made his first appearance of the season as an 82' minute substitution for Aly Cissokho. After injury ruled captain, Ron Vlaar out, Baker made his first start against Liverpool, putting in a great performance alongside Philippe Senderos, keeping a clean sheet in a 1–0 victory. He was set to start against Arsenal, however, like many of his teammates suffered from an illness and was left out of the squad. Baker went on to start against Chelsea, Manchester City and Everton. He wasn't included in the squad against Queens Park Rangers, before starting against Spurs. On 8 November 2014, during the 0–0 draw against West Ham United, Baker picked up a knee injury. After a long lay off, Baker was an unused substitute Leicester before starting against Liverpool on 17 January 2015.
2015–16 season
On 31 July 2015, Baker ended transfer speculation when he signed a new four-year contract with the club. On 1 September 2015, Baker joined Bristol City on a season-long loan.
2016–17 season
Following Aston Villa's relegation from the Premier League, Baker returned to Aston Villa. Under Roberto Di Matteo, Baker often found himself behind the club's new arrivals Tommy Elphick and James Chester, being used as a substitute. Following an injury to Elphick, and the replacement of Matteo for Steve Bruce, Baker formed an impressive partnership with Chester. On 19 November, Baker scored his first goal for Aston Villa against Brighton and Hove Albion, heading in an Albert Adomah delivery. The match ended 1–1.
Bristol City
2017–18 season
On 28 July 2017, Baker joined Bristol City for an undisclosed fee on a four-year deal.
