The castle hill has already been settled since the 5th millennium BC. The inhab­itants of antique Poetovio built a for­tress and temples there, and in the Early Middle Ages a Slavonic burial place was there.

The medieval fortress was built in the 11th century when the castle and the town were owned by the archdiocese of Salzburg. The archbishops sold the castle to the Lords of Ptuj who estab­lished in their three-hundred-year-old history in Ptuj the Dominican and the Minorite Monastery and built a church for pilgrims on Ptujska Gora. The Tomb of the last Lord of Ptuj, Frederick IX, is built into the castle ground floor. Between 1656 and 1802, the Earls Leslie lived in the castle. The last owners, the Earls of Herberstain, owned the castle between 1873 and 1945; in 1945 it was nationalized and connected to the Ptuj's museum with its rich furniture.

Bipartite medieval fortress had residen­tial and commercial premises on the top of the hill, and on the western plain it consisted of military buildings and be­tween the two there is a Western Tower, the oldest preserved part of the castle. In the 12^th century a group of buildings was situated in the middle of the hill, with residential area and a mighty Konrad's defence tower. In the 14th century the central building got a horseshoe ground plan and consisted of two sto­reys and the castle walls were connected to the town's wall. The look of the castle was substantively changed in the 16th century by Italian architects who con­solidated the town and the castle from the Turks. After the danger from Turks was over, they pulled down the Konrad's tower in the 17th century, enlarged the castle and built stables and a tower on the east plain of the hill. In the 18th cen­tury an administrative building was built along the south wall and a place for stor­ing grains and the outbuilding below.