According to legend, the first castle on the site was built by a Roman hunter who saw the massive sandstone spire on an island in the Sense river. He saw a red deer on the cliff over the river and went to catch it. As he rode after the deer a dragon roared out of a cave, but the hunter quickly killed the dragon. The deer then walked up to the hunter and offered his life to the hunter. The hunter allowed the deer to go free and the deer gave him possession of the area. The hunter than built the first castle on the top of the sandstone spire on the island. A bridge was built over the river and became part of the Roman road from Aventicum. The legend continues that after the Roman Empire collapsed, a Walliser robber took over the old Roman castle as a new hideout. He began to hire local villagers to help him expand the castle toward the east. Initially he acted friendly and kind, but when the workers complained of the work or asked for pay, he murdered them and mixed their blood into the mortar. This, according to legend, is why the mortar on the east side is particularly hard. While the Romans lived in the area, there is no archeological evidence of a Roman or early medieval fortification. Despite the local legends, the first castle on the site may have been a wooden fortification, but the oldest stone walls are from the 11th or 12th century. It was probably built by a Burgundian or Zähringen noble. The castle was first mentioned in 1223 as Grasburc. In the same year, a knight, Otto von Grasburg was mentioned at the castle, followed in 1228 by the knight Kuno von Grasburg. In the 13th century the castle and lands passed to the Kyburgs and then after their family died out in 1263/64 the Habsburgs beat out the Counts of Savoy to inherit it. Under the Habsburgs several Ministerialis families held the castle. In 1310 Henry VII, the King of Germany, pledged the castle and surrounding Herrschaft to Count Amadeus of Savoy to pay debts. The Counts held the estate for over a century, until the remote location and gradual decay forced them to sell the castle and territory to Bern and Fribourg in 1423. The two cities established a condominium or shared rule over the land. The castle served as the residence and administrative center for the vogts that were appointed by alternating cities. In 1575 the increasingly expensive castle was abandoned and the vogt moved to Schwarzenburg Castle. The castle gradually fell into ruin and in 1845 the Canton of Bern sold the ruins to a private owner. In 1894 the city of Bern bought the ruins and began restoring them. By the spring of 1902, the main tower was about ready to collapse. The Canton spent four years repairing and reinforcing the tower. Another project in 1928-31 repaired and restored other parts of the castle ruins. A third project in 1983-84 restored and repaired the ruins further.
