The site was first fortified in the eleventh century by Gueldin de Saumur, a Norman seigneur who ringed a high terrace with a wooden palisade overlooking the Indre River. The Comte de Blois later rebuilt it in stone, but by the fifteenth century the castle had fallen to ruin. Jean V de Bueil, a captain-general of Charles VII, purchased it in 1431 and began rebuilding during the 1440s. His son Antoine married Jeanne de Valois in 1462 — the biological daughter of Charles VII and his mistress Agnès Sorel — who brought a dowry of 40,000 golden écus. Even that fortune could not save Antoine from debt, and by 1455 he had sold the château to Jacques d'Espinay, son of a chamberlain to the Duke of Brittany.

The Espinay family transformed Ussé. Jacques began building the chapel, which his son Charles completed in 1612, blending Flamboyant Gothic tracery with the new vocabulary of the Renaissance — an architectural conversation between two centuries happening in a single building. The reconstruction they initiated gave the château the sixteenth- and seventeenth-century appearance it retains today. In the seventeenth century, Louis I de Valentinay, comptroller of the royal household, made the boldest alteration: he demolished the entire north range of buildings to open the interior courtyard to views across the parterre terrace, a design attributed to André Le Nôtre, the legendary landscape architect of Versailles. Valentinay's son-in-law was none other than Vauban, the great military engineer, who visited Ussé on numerous occasions.

Ussé passed through the House of Rohan before being purchased in 1802 by the duc de Duras. In March 1813, the château became the unlikely setting for low-key meetings among Bourbon loyalists — men like the duc de Fitzjames, the prince de Polignac, and the duc de Rochefoucault — who gathered to sound out the possibilities of restoring the Bourbon monarchy. The conspirators chose well; Ussé's remote grandeur offered both privacy and prestige. Later, François-René de Chateaubriand worked on his monumental memoirs, Mémoires d'Outre-Tombe, as a guest of the duchesse Claire de Duras. In 1885 the comtesse de la Rochejaquelein bequeathed Ussé to her great-nephew, the comte de Blacas, and the château remains in the Blacas family today.

Perrault's connection to Ussé gave the château a fame that transcended architecture. In the 1920s, it appeared on a French railroad poster for the Chemin de Fer de Paris à Orléans, its turrets beckoning tourists southward. Disney animators later drew on its silhouette — along with several other Loire châteaux — when designing the castles that would become the company's visual signature. Today, visitors can climb the towers to find mannequin tableaux recreating scenes from the Sleeping Beauty tale, a playful acknowledgment that fiction and stone have been inseparable here for over three hundred years. Ussé was classified as a monument historique in 1931, ensuring that the castle Perrault dreamed into literature would endure in reality.