The remains of Fragenstein castle are located in the north-west of the village Zirl on a steep rock. In former times the castle was used as a fortress and as a prison. The dynasty of Fragenstein was mentioned in documents in 1227, 1239 and in 1254. Emperor Maximilan I. used the castle as hunting lodge.
After his death, the castle was let gone to rack and ruin. On June 23, 1703, the "Bavarian Rummel" took place - Bavarian and French troops crossed the inn river and started a fight against the Schützen of Zirl.