Frazier attended Grand Rapids Christian High School, where he practiced football, basketball and track. In football, he was a three-year starter. As a junior, he was named to the all-conference team. He was a two-way player as a senior, registering 66 tackles, 858 rushing yards and 15 touchdowns, while receiving Division 3/4 All-state, Detroit Free Press All-Area and All-conference honors. In basketball, he helped lead the team to a pair of conference titles and a district title as a senior.
College career
He accepted a football scholarship from Central Michigan University. As a freshman, he appeared in 13 games, collecting 36 tackles, 3 passes defensed, one interception and one forced fumble. He had 11 tackless and one pass defensed against Navy. As a sophomore, he appeared in 12 games, registering 67 tackles, 3 interceptions and 5 passes defensed. He had 11 tackles against the University of Toledo. The next year, he appeared in 13 games, posting 58 tackles, 34 solo tackles, 4 passes defensed and one fumble recovery. He was the regular starter at strong safety as a senior, recording 108 tackles, 74 solo tackles, 4.5 tackles for a loss, 4 passes defensed, one interception and one blocked punt. He had 13 tackles against Oklahoma State University. He made 12 tackles against the University of Toledo. He is a member of Phi Beta Sigma fraternity.
Frazier was selected by the Dallas Cowboys in the sixth round of the 2016 NFL Draft. As a rookie, he played mainly on special teams and was declared inactive in 4 games. He tallied 2 defensive tackles and 4 special teams tackles. In 2017, he tied for the team lead with 10 special teams tackles and began to be used more on the defense. Prior to 2018 preseason, Frazier was placed on non-football-related injured reserve on July 26 due to him having a potential blood disorder. He was tested negative for haemophilia and other blood clotting disorders on August 3, putting him back on the active roster. In 2019, he was a backup at strong safety, collecting four defensive tackles and one special teams tackle in four games. On September 30, he was placed on the injured reserve with a pectoral injury. He was not re-signed after the season.
Miami Dolphins
On April 28, 2020, Frazier signed with the Miami Dolphins.