Caldwell was one of 11 children born in Texas City, near Houston, Texas. He was the son of a longshoreman and mechanic and a homemaker. When he was six, Caldwell witnessed the Texas City disaster in 1947, when a docked ship blew up and 581 people died with thousands injured. Caldwell's family was left unharmed, but he said decades later, "I can still see people flying through the air." When Caldwell was 15, he moved with his sister to Los Angeles, California. He emerged as a late-bloomer player and John Wooden courted him to play for him at UCLA. He ended up at Arizona State instead.
Collegiate career
Caldwell attended John C. Fremont High School in Los Angeles, not playing until his junior year. He then played collegiately at Arizona State University. Caldwell played for Arizona State from 1961 to 1964, setting the Sun Devils career scoring record with 1,515 points. His 929 rebounds, are the second-best total in school history. Caldwell led Arizona State to the NCAA Tournament in each of his three varsity seasons and a 65–18 overall record.
Nicknamed "Pogo Joe" or "Jumping Joe" for his leaping abilities, Caldwell was a guard/forward. In the 1964 NBA draft, Caldwell was the No. 2 overall pick by the Detroit Pistons. Olympic teammate Jim "Bad News" Barnes went No. 1. Caldwell spent the majority of his NBA career with the St. Louis/Atlanta Hawks franchise. After averaging 21.1 points per game for the Atlanta Hawks in the 1969–70 NBA season, Caldwell jumped to the rival ABA, playing for the Carolina Cougars from 1970 to 1974. Caldwell's contract with Carolina called for him to earn $150,000 per year and another $70,000 deferred for five years. A clause called for him to receive $6,600 per month beginning at age 55. Later, the Carolina owner, Tedd Munchak, sued to try to negate the pension. Caldwell was interviewed on 60 Minutes, who ran a segment on the lawsuit. Caldwell prevailed and received his pension payments beginning in 1996. During the 1974–75 ABA season, the Carolina franchise had moved to become the St. Louis Spirits. Spirits' management blamed Caldwell for influencing team star Marvin Barnes to briefly leave the team. Caldwell denied doing this but he was suspended for "activities detrimental to the best interests of professional basketball." Caldwell never played another professional basketball game. He filed various lawsuits, alleging that he was wrongly blacklisted by the ABA and later the NBA. Tedd Munchak, who was suing Caldwell was now Commissioner of the ABA. Caldwell, who was President of the ABA Players Association, had his case go all the way to the Supreme Court. Caldwell averaged 16.1 points, 5.3 rebounds and 3.1 assists in eleven professional seasons. He scored 12,619 combined NBA/ABA career points.
Personal Life
Joe Caldwell is the grandfather of Marvin Bagley III, a power forward for the Sacramento Kings. Bagley's mother is Caldwell's daughter, Tracy Caldwell. Bagley was the No. 2 overall selection in the 2018 NBA draft, the same pick as his grandfather in the 1964 NBA draft. Caldwell attended his grandson's games throughout high school and college.
Honors
Caldwell's jersey #32 was retired by Arizona State University. On November 20, 2010, the ceremony took place before a game against the UAB Blazers.