Takeda Nobuyoshi built this castle. The son of Minamoto no Kiyomitsu of nearby Yato Castle, Nobuyoshi became the progenitor of the Takeda clan, one of the most powerful warrior families in Japanese feudal history. His residence lay just over a kilometer from the castle walls. The fortress took its name from Hakusan Jinja, a Shinto shrine that predated the fortification on this mountain. From these heights, Nobuyoshi could survey the Kofu basin spread below him and the river corridors that served as both trade routes and invasion paths. The choice of location was deliberate: the Kamanashi River on one side and the Shiokawa River on the other formed natural moats that supplemented the castle's engineered defenses.

Hakusan Castle is a yamashiro, a mountain castle, and its builders understood how to make terrain work as a weapon. The fortification extended roughly 150 meters east to west and 180 meters north to south along a ridge running in the same direction. Vertical dry moats cut into the slopes protected the enclosures, while two large moats guarded the ridge behind the western approach. This combination of horizontal and radial moats was a hallmark of Takeda castle engineering, a defensive philosophy the clan would refine across dozens of fortifications throughout Kai Province. Signal towers stood at the tips of ridges descending to the north and south, with beacon fires defending the flanks of the main castle. The ruins of Shinpu Castle, another Takeda stronghold, are visible almost due north across the river.

Little survives in the historical record about the castle's middle centuries, but what is known speaks to turbulent times. Aoki Nobutane completely rebuilt the fortress as part of Kai Province's outer defensive line. After the Takeda clan fell at the Battle of Tenmokuzan, the castle passed to Tokugawa Ieyasu, who recognized its strategic value and used it as a stronghold against the Odawara Hojo clan in the struggle for provincial control. The castle was then assigned to Yamadera Nobumasa before slipping from the historical record entirely during the Kanbun era of the Edo Period, sometime between 1661 and 1673. By then, the long peace of Tokugawa rule had made mountain fortresses obsolete.

What the centuries of warfare could not destroy, the twentieth century nearly did with dynamite and bulldozers. Plans to convert the castle site into a rock quarry surfaced in 1983, threatening to obliterate the remarkably well-preserved ruins. Local conservation groups fought the proposal and won. When the quarry idea resurfaced in 1996, they fought again and won again. In 2001, the Japanese government designated Hakusan Castle a National Historic Site, placing it under permanent protection. Today the ruins preserve the remains of several kuruwa enclosures forming three baileys, along with earthworks, dry moats, vertical moats, and the foundations of gates. The site is accessible by a short walk from the Nabeyamagami bus stop, served by buses from Nirasaki Station on the JR East Chuo Main Line.