Talmberk Castle was founded by Hroznata of Úžice from the Moravian Kounic family sometime after 1284. He was the Purgrave of Prague Castle and for his services to King Wenceslas II he was given the territory north of the Sázava River, around Úžice. The first written record of the castle dates back to 1291, when the property was divided among the sons of Hroznata. The castle and the farm were then given to Arnošt, who completed it. From the middle of the 16th century it is mentioned as deserted.
The first mention of the village of Skalice Hor Stříbrný dates back to 1361 under Ctibor of Skalice, who also had the patronage right to the church of St. James in Rovná. It is not documented whether the castle existed earlier (according to archaeological findings, however, there was a settlement on the castle hill as early as the 13th century) or whether it was built under this owner or his son Střížek of Skalice, mentioned in 1376, or Kuneš of Skalice, mentioned in 1377.