The castle was originally built on a nearby mountain in 1240 AD by Akiba Shigenobu. Takahashi Muneyasu constructed a castle on the modern site on Mount Gagyū in 1331, though the design of this castle differed from the one that stands on the site now. When Mimura Motochika became the feudal lord of the region, Matsuyama castle was enlarged again and the site extended to cover the entire mountain. With assistance from the Mōri clan, Mimura Motochika conquered the whole Bitchu area and defended it against the Amako clan. Motochika later entered into secret communications with the Oda, and having come to the attention of the Mōri, they forced him from the castle and died in the escape. In 1600, the castle became part of the Bitchū-Matsuyama Domain where Kobori Masatsugu and his son Masakazu came to the area as officers of the Tokugawa shogunate and repaired the castle as part of the efforts to turn Matsuyama into a castle town. In 1617, Ikeda Nagayoshi was transferred here as the new lord of Bitchū-Matsuyama Domain and was followed by Ikeda Nagatsune, who ruled until 1641. The next feudal lord, Mizunoya Katsutaka, rebuilt the donjon, turrets and other gates in addition to building Onegoya house on the southern side of Mount Gagyu where public affairs were administered. The Mizunoya clan ruled here until 1695 where it was transferred again to the Andō clan and the Ishikawa clan in 1711. The tenshu was unusual in that it was only two stories tall, though a larger tenshu along the lines of Himeji Castle's would have been unnecessary as Bitchu Matsuyama Castle was located on a mountain, thus allowing a large field of vision. The lord's palace was constructed at the base of the mountain. Itakura Katsuyoshi became lord in 1744, and eight descendants of his ruled the castle until the Meiji Restoration. After the Edo period had ended, the castle was partly destroyed, but the rest of it was abandoned and slowly fell into disrepair. In 1929, a citizens' group was established and restoration work was begun on the castle. After this work was completed, the Takahashi City authorities repaired the keep's turret, and mud walls. Three parts were saved and still stand today: a short section of wall, the Nijū yagura, and the tenshu.
Today
In recent years, parts of the castle have been reconstructed to augment the parts that remain, all of which have been named Important Cultural Properties in 1950 by the National Government. In 1957 the government began restoring the castle to its original state and was completed by 1960. It is also a popular place to visit because it is the only yamashiro, or mountain castle, to have an original tenshu. The castle is on a mountain and the road up to the summit does not go all the way, so to get to the castle, one must hike up a mountain path. In December of 2018, a local cat named Sanjūrō was installed as the honorary "lord" of the castle. Sanjūrō, named after local samurai Tani Sanjūrō, originally belonged to Megumi Nanba but had run away on July 14, 2018 after torrential rains brought floods and mudslides in the area. The cat was later found living in the castle by one of the workers, who started feeding it. The presence of Sanjūrō, who was first made a provisional mascot before being named as "lord", has helped increase the number of visitors to the castle.