The Teutonic Knights began building the castle in 1390 on a hill in southeast Bytów. Mikołaj Fellenstein directed the castle's construction works in 1398–1406. The castle was raised on the plan of a rectangle out of stone and brick with dimensions 49 metres by 70 metres. In the corners of the castle, three cylindrical and one square tower was built. The northwestern castle wing was a three-floor building that served as living quarters. The most important rooms were in this building, including the refectory, the chapel and the prosecutor's office. Warehouse space was located on the third floor. The castle's southwest wing housed a kitchen and food storage, and nearby stood a water well. The gate was raised from the northeastern side of the castle. Approaching the gate, there was a moat and drawbridge that served as the entranceway. The four towers in each corner of the castle served as defensive structures, housing weaponry. The Kingdom of Poland under the command of Władysław Jagiełło seized by the castle in 1410.
The Kingdom of Poland took control of the castle during the Thirteen Years' War. In 1466, King Kazimierz Jagiellończyk ceded the castle to the fiefdom, Duchy of Pomerania, then ruled by Eric II, duke of Pomerania. In 1500, the stronghold was surrounded with fortifications and bastions. In the second half of the 16th century, the castle was the residence of the House of Pomerania, which in the years 1560-1570 had their living quarters in the southeastern wing. The Thirty Years' War wrought little damage to the stronghold. Along with the death last member of the House of Pomerania in 1638, the castle was made the residence of the starosta. The Swedish army razed the castle during the Deluge,. In the 19th century the castle was partly rebuilt after Frederick William I of Prussia, elector of Brandenburg, took over. The castle housed a court and a prison. First conservation works were conducted in between the years of 1930 and 1939. The castle was later rebuilt at intervals between the years of 1957 to 1962 years and between the years of 1969 to 1990, when the castle housed the Western Pomeranian Museum, a hotel with a restaurant and a library.
