During the Sengoku period, Kakegawa was a fortified settlement of the Imagawa clan. Following the defeat of the Imagawa at the hands of Oda Nobunaga at the Battle of Okehazama, Kakegawa, along with the rest of Tōtōmi Province came under the control of Takeda Shingen. Imagawa loyalist Asahina Yasutomo surrendered Kakegawa to Takeda ally Tokugawa Ieyasu. After the death of Takeda Shingen, Ieyasu took control of all of Tōtōmi Province, and assigned Kakegawa to his retainer Ishikawa Ienari. However, after the Battle of Odawara, Ieyasu was forced to surrender his domains in the Tōkai region to Toyotomi Hideyoshi in exchange for the provinces of the Kantō region. Toyotomi retainer Yamauchi Kazutoyo was assigned Kakegawa, and considerably improved on the structure of the castle and its moat system. After the Battle of Sekigahara, Tokugawa Ieyasu regained control over the Tōkai region, and Yamauchi Kazutoyo surrendered Kakegawa in exchange for Tosa Province in Shikoku. In February 1601, Hisamatsu Sadakatsu, Ieyasu's half-brother, was made daimyō of the new Kakegawa Domain with revenues of 10,000 koku. In April 1607, he was reassigned to the newly created Fushimi Domain, and turned Kakegawa over to his son Sadayuki, who ruled until reassigned to Kuwana Domain in 1617. Andō Naotsugu was daimyō with revenues increased to 28,000 koku from 1617 to 1619, when he was reassigned to Tanabe Domain as advisor to Tokugawa Yorinobu. Rule over Kakegawa was briefly given back to a branch of the Hisamatsu clan with the transfer of Hisamatsu Sadatsuna from Shimotsuma Domain in Shimōsa Province with revenues slightly increased to 30,000 koku. However, he was transferred to Yodo Domain in Yamashiro Province in 1623. The next inhabitant of Kakegawa Castle was Asakura Nobumasa, a close advisor of the ill-fated Tokugawa Tadanaga. After Tadanaga’s forced suicide, Asakura was stripped of his domain and exiled to Kōriyama in Yamato Province. Aoyama Yoshinari, formerly of Hitachi Province was then assigned to Kakegawa, and his revenues were set at 33,000 koku. He was reassigned to Amagasaki Domain in 1635, and replaced by Matsudaira Tadashige, formerly daimyō of Tanaka Domain in Suruga Province. Domain revenues were increased to 40,000 koku, and after his death in 1639, he was succeeded by his son Tadamoto, who was then transferred to Iiyama Domain in Shinano Province only a month later. Honda Tadayoshi, grandson of Honda Tadakatsu was assigned to Kakegawa next, and revenues were increased to 70,000 koku. However, in 1644, Honda Tadayoshi was reassigned to Murakami Domain in Echigo Province. Tanaka Domain again provided a successor, in the form of Matsudaira Tadaharu; however, the Bakufu reduced the rating of Kakegawa Domain back to 25,000 koku. In 1648, he was transferred to Kameyama Domain in Tanba Province. Tanaka Domain once more provided a successor: Hōjō Ujishige, who lasted until his death without heir in 1658. In February the following year, Ii Naoyoshi was brought in from Nishio Domain in neighboring Mikawa Province, and governed Kakegawa until his death in 1672. His branch of the Ii clan continued to rule Kakegawa until 1706. After a brief period under Matsudaira Tadataka until his transfer to Amagasaki Domain, Kakegawa came under control of the generation of the Ogasawara clan from 1711 until their transfer to Tanakura Domain in 1746. Finally, in 1746, Ōta Suketoshi was assigned to Kakegawa from Tatebayashi Domain in Kōzuke Province. The Ōta provided the next seven daimyōs of Kakegawa until the Meiji Restoration, and thus brought about a period of much-needed stability and continuity to government policies. The domain had a population of 3,443 people in 906 households per the 1843 census, the domain maintained its primary residence in Edo at Soto-Sakura, in what is now Kasumigaseki, Tokyo. In February 1869, 7th daimyō, Ōta Sukeyoshi were transferred by the new Meiji government to the short-lived Matsuo Domain in Kazusa Province. The holdings of Kakegawa Domain in Suruga and Tōtōmi were absorbed into the new Shizuoka Domain created for retired ex-shōgunTokugawa Yoshinobu; its holdings in Izu Province were absorbed into Niirayama Prefecture.
As with most domains in the han system, Kakegawa Domain consisted of several discontinuous territories calculated to provide the assigned kokudaka, based on periodic cadastral surveys and projected agricultural yields.
