Kaziboni was signed by Northampton Town while playing for local amateur side Gregory Celtic. Greg was given a squad number in January 2011 and made his Football League debut against Gillingham the same month. On 28 February 2012, Kaziboni was released from his contract at Northampton Town by manager Aidy Boothroyd.
Greg joined Conference North side Corby Town on 16 August 2012. joining a team with a number of links to his previous club, as on the same day former colleagues Nathaniel Wedderburn, Liam Richardson and Tom McGowan all signed, although former Northampton Town defender Andy Holt was forced to retire with a knee injury. Kaziboni went onto make 28 appearances, and scored three goals in what was an extremely disappointing end to the 2012–13 season, as Corby Town finished the season relegated by a single goal, with Histon the fortunate beneficiaries.
In November 2013, Kaziboni signed for Rugby Town, and he made his debut in the club's 3-1 win over Uxbridge. Greg had a mixed season with Rugby Town netting two goals and left at the end of the season to re-join Brackley Town.
Brackley Town
Greg re-signed for Conference North side Brackley Town in 2014. Kaziboni remained with Brackley Town for two seasons, and began the 2017–18 season with the club prior to departing, in total Greg made 46 appearances and scored five goals in the Conference North.
In the summer of 2018, Greg signed for Southern League Premier Central side Banbury United on a one-year contract. Kaziboni made his Southern League Premier Central debut for Banbury United on 11 August 2018, the opening day of the season. The club drew 1-1 in an away fixture with Rushall Olympic. It was confirmed on 24 May 2019 that Kaziboni would leave Banbury United on the expiry of his contract. During his time with the club Kaziboni made 36 appearance and scored 9 goals in the Southern League Premier Central, as well as 3 appearances and 1 goal in the FA Cup, 3 appearances and 2 goals in the FA Trophy as well as an appearance and a goal in the final of the Oxfordshire Senior Cup, which Banbury United lost 4-3 to Oxford City.
On 2 June 2019, Kaziboni signed a one-year contract with Southern League Premier Central side Tamworth. Following an injury lay off from pre-season, he made his debut on 31 August 2019, in an away fixture against Hitchin Town, scoring a brace in a 3-0 victory for the away side. Kaziboni added a third goal to his tally for the club on 14 September 2019, in his third match, a 3-0 victory against Redditch United.
On 30 December 2019, it was announced that Kaziboni had joined Southern League Premier Division Central rivals AFC Rushden & Diamonds on a months loan, in a bid to get some regular first team football. Greg made his debut for AFC Rushden & Diamonds on 1 January 2020 in a Southern League Premier Division Central away fixture with St Ives Town, helping his new club to a 4-3 victory. Kaziboni scored his first goal in an AFC Rushden & Diamonds shirt on 11 January 2020, scoring the equaliser in a Southern League Premier Division Central 3-2 away victory against Leiston.
Banbury United (loan)
On 31 January 2020, Kaziboni rejoined Southern League Premier Division Central side Banbury United on a months loan. Greg made his second debut for the club the following day, as his new side lost narrowly 1-0 at home to Bromsgrove Sporting.
Stratford Town
Greg signed for Southern League Premier Division Central side Stratford Town on 21 June 2020.
