The old castle of Padberg stood atop the Old Mountain (Alte Hagen) and was constructed by Count Hahold II. He had been ordered by Emperor Otto to build a fortification in the mountains of this territory from which strategic roads could be guarded. The building began in 936 and was completed in 972. Hahold II, who died in 1011, probably had not actually resided in the new castle. It is certain that that his son Hahold III, who died in 1029, lived in the castle as did his descendants through the next centuries.

The Old House was constructed in the old Saxon style with walls encircling the main structure. Walls protected the castle from invaders climbing the mountain. Castles built like this were the easiest to defend, until cannons became the deadly weapons of warfare. In the 17th century canon fire blew apart the Old Castle Padberg. Little remains today.