The Akiyama clan built the first fortification on Kojoyama during the Nanboku-cho period, when Uda District was divided among three rival clans. They chose their ground well. The 100-meter peak commanded the main route between Ise Province and Yamato Province, as well as the principal north-south road connecting Nara with Yoshino. Control of this junction meant control of movement through the heart of ancient Japan. The Akiyama served as vassals first to the Kitabatake clan, then to the Tsutsui clan during the turbulent Sengoku period, fighting against warlords like Matsunaga Hisahide and Oda Nobunaga. Their castle was originally called Akiyama Castle, a name that would not survive the waves of new lords who followed.

In 1585, when Toyotomi Hidenaga entered nearby Koriyama Castle, the Akiyama clan was displaced from Uda. Their castle passed through a succession of Toyotomi generals -- Ito Yoshiyuki, Kato Mitsuyasu, Haneda Masachika, and Taga Hidetane -- each carrying out large-scale renovations. The Toyotomi regime recognized Uda Matsuyama as a key strategic point for controlling Yamato Province, ranking it alongside Yamato Koriyama Castle and Takatori Castle. After the Battle of Sekigahara in 1600, Taga Hidetane was removed for siding with Ishida Mitsunari's Western Army. Fukushima Takaharu, younger brother of the famous general Fukushima Masanori, took over as castellan and poured resources into even more ambitious renovations. He renamed the fortress Matsuyama Castle. Archaeological excavations have since unearthed foundations of stone buildings, the Honmaru Palace, massive stone walls, roof tiles, and ceramics from this period of expansion.

Fukushima Takaharu chose the wrong side one final time. In 1615, the Tokugawa shogunate stripped him of his domain for colluding with Toyotomi Hideyori during the Battle of Osaka. The order came down: the castle must be destroyed. Kobori Enshu -- a man better known today for his tea ceremonies and exquisite gardens -- was assigned to supervise the demolition. His meticulous records, including the newly discovered letter complaining of labor shortages, provide rare documentation of how the shogunate dismantled the military infrastructure of defeated rivals. Stone by stone, the fortress was pulled apart. Oda Nobukatsu was assigned the Uda-Matsuyama Domain afterward, but was expressly forbidden from rebuilding. In 1694, the Oda clan was transferred to Kaibara Domain, and the area became tenryo -- territory under the direct control of the Tokugawa government.

The castle vanished, but its town endured. Without a lord or a fortress, the settlement at the foot of Kojoyama reinvented itself as a merchant town and regional administrative center. Wooden machiya townhouses, whitewashed earthen storehouses, and traditional streetscapes survived the centuries because there was no lord to rebuild, no siege to endure, no modernization drive to replace old with new. The town simply continued, its Edo-period character preserved by the quiet accident of being overlooked. Today, the castle town holds designation as an Important Preservation District for Groups of Traditional Buildings, and the castle ruins above it are protected as a National Historic Site, designated in 2006 and listed as No. 166 on the Continued 100 Fine Castles of Japan. What remains is a place where the walls are gone but the streets remember.