The oldest written trace of the castle comes from the pen of Muhammad al-Idrisi, the Muslim geographer who noted it in passing in the twelfth century, when this was Andalusi ground. What stands today is largely a thirteenth-century rebuild, raised on the older Arabian structure after Aragonese troops recovered the Valle de Cofrentes between 1239 and 1243. The builders did not start from nothing. Among the masonry and cut ashlar of the later fortress, sections of rammed earth still survive from those earlier Muslim walls, the two eras pressed together in the same stone, one civilization's foundations carrying the next.

Ayora spent the thirteenth century being traded between crowns like a disputed inheritance. The Treaty of Almizra in 1244, signed between Castile and Aragon, handed the town to the Castilians. In 1281 the Treaty of Campillo, agreed by Alfonso X the Wise of Castile and Peter III the Great of Aragon, returned it to Aragon as war reparation. Finally, in 1305, James II the Just folded Ayora back into the Kingdom of Valencia by the Treaty of Elche. Centuries later a harsher decree fell on the valley: the expulsion of the Moriscos in 1609, which drove out the region's Muslim-descended population after they rose against the crown and left the countryside all but empty of the people who had farmed it.

For much of the late Middle Ages the fortress was a feudal manor, held by the Aragonese admiral Bernardo de Sarria. In 1492 it was bought by Rodrigo Diaz de Vivar y Mendoza, son of Cardinal Mendoza and the first Marquis of Cenete. His daughter, Mencia de Mendoza, Marchioness of Cenete, inherited it; a Renaissance patron born in Jadraque in 1508 and dead in Valencia in 1554, she left her mark on the walls. It was she who commissioned the Puerta Falsa, the so-called False Door, a graceful entrance of reddish limestone ashlars framed by an alfiz and once crowned with her coat of arms. When she died without children the manor passed to her sister Maria and, through marriage, to the Dukes of the Infantado, who held it until feudal titles were abolished in 1837.

The castle's long decline began violently. In 1707, during the War of the Spanish Succession, the troops of Philip V under the Count of Pinto took Ayora by assault, looting and burning both the town and the fortress until its people abandoned it. The wreck never recovered. In 1797 the botanist Antonio Jose Cavanilles, touring the Kingdom of Valencia, recorded it simply as a ruined and destroyed place. What little remained standing was finished off in 1812, when French forces occupied the valley during the Napoleonic wars and reduced the last defensible stone. Since 2006 the site has carried official protection as a Bien de Interes Cultural, guarded now against neglect rather than armies.

Even ruined, the plan is legible. A four-sided keep rises from the middle of the compound, deliberately placed to command the land in every direction. Two enclosures survive: the larger Plaza de Armas, paved with pebbles laid in geometric patterns, and a smaller southern courtyard thought to belong to the earlier palace of the Marchioness. Ramparts once ran roughly 900 meters around the whole, standing six meters high, studded with square towers and one semicircular bastion. Above the False Door, two stone corbels still jut where a machicolation once let defenders drop stones on anyone attacking the gate, a small, practical detail of a fortress built for a border that no longer exists.