Long before Hojo Soun arrived, the hills around Nirayama carried the weight of political exile. This was the territory of the original Hojo clan during the late Heian period, and the same landscape where Minamoto no Yoritomo, founder of the Kamakura Shogunate, spent his youth banished from the capital. Centuries later, the pattern repeated: Ashikaga Masatomo, brother of Shogun Ashikaga Yoshimasa, was dispatched to govern the Kanto region in the 1460s but found himself refused entry to Kamakura by defiant local lords. Stranded, he constructed the Horigoe Gosho nearby and ordered his vassal Toyama Buzen-no-kami to build a fortification on the Nirayama ridge sometime during the Bunmei era, between 1469 and 1486. The castle began not as a seat of power but as a refuge for the powerless.

Hojo Soun transformed that refuge into a war machine. After seizing Nirayama Castle in 1491, he undertook full-scale construction, turning the narrow hilltop into a formidable stronghold with western-style fortifications and supporting forts on four surrounding hills. From here, Soun conquered Izu Province and pushed deep into Sagami Province. He transferred his official seat to Odawara Castle in 1495, establishing what would become the most famous Hojo fortress, but Nirayama remained his personal residence. He died within its walls in 1519, having built a domain that his descendants would hold for another seven decades. The castle continued as a critical outpost for controlling the Izu Peninsula, its network of satellite forts making it far more than a single hilltop strongpoint.

Nirayama Castle proved its worth in battle more than once. In August 1570, the legendary Takeda Shingen invaded Suruga Province and sent forces against the castle. Hojo Ujinobu, son of Hojo Ujiyasu, held the walls and turned the attackers back. But the castle's most dramatic moment came twenty years later, during Toyotomi Hideyoshi's massive Siege of Odawara in 1590. Hideyoshi dispatched Oda Nobukatsu with a large army to reduce Nirayama while the main force surrounded Odawara. Hojo Ujinobu, hopelessly outnumbered, refused to surrender. For over one hundred days he held the castle, an act of stubborn resistance that ended only when Hideyoshi sent Tokugawa Ieyasu himself to negotiate terms. It was one of the longest holdouts of the entire campaign.

After the Hojo fell, Nirayama Castle passed briefly to Naito Nobunari, a vassal of Tokugawa Ieyasu. But in 1601 Naito was transferred elsewhere, and the castle was simply abandoned. No new lord came to occupy it. Instead, the site passed to the Egawa clan, magistrates who governed Izu from the nearby Nirayama Daikansho on behalf of the Tokugawa shogunate. The fortifications slowly crumbled. Archaeological excavations in 1990 and 1991, conducted during reconstruction of nearby Nirayama High School, uncovered a well, a garden pond, and the remains of what appears to have been a residence. A place called Gozashiki on the high school grounds may mark where samurai residences and government offices once stood, though their exact locations remain unknown.

The castle site sits about ten minutes on foot from Nirayama Station, where walking trails now wind through the earthworks that once channeled attackers into kill zones. Remains of the enclosures and moats are still visible, tracing the outline of a fortress designed to hold against armies many times its garrison's size. The ruins earned designation as a National Historic Site in 2025, formal recognition of a place where exiled princes, self-made warlords, and stubborn defenders all left their mark on the same narrow ridge. Standing among the earthen ramparts, with the Izu Peninsula stretching south toward the sea, it is easy to understand why every generation that found this hilltop tried to hold it.