The fort's origins trace to a practical crisis. In the 1870s, Iban migration upriver along the Rajang was driving violent clashes with the Orang Ulu peoples who lived in the interior. Rajah Charles Brooke, the second of Sarawak's White Rajahs, built the original Baleh Fort at Nanga Balleh in late 1874, positioning it at the confluence of the Rejang and Baleh rivers to block further Iban movement upstream. In 1877, Brooke nearly drowned there when his boat capsized in the treacherous currents. He abandoned the site the following year and in 1880 built a replacement fort downstream at Kapit. The new structure was constructed entirely from belian, the legendary ironwood of Borneo, with walls thick enough to withstand siege. It would serve as the colonial government's anchor point in a region where headhunting was still practiced.

For 45 years the structure was simply called Kapit Fort. In 1925 it was renamed Fort Sylvia after Rani Sylvia Brooke, wife of the third and last White Rajah, Charles Vyner Brooke. The name stuck even as the fort's purpose shifted with the times. During the 1960s it housed the District Office and District Court House. When Kapit Division was formed in 1973, the fort became the Resident's Office. By the 1990s, the ironwood walls that had once kept out war parties were sheltering filing cabinets and civil servants. The Sarawak state government recognized the building's historical significance in 1997 and handed its care to the Tun Jugah Foundation, which completed conservation and renovation work by December 1998.

The museum that now occupies Fort Sylvia is unexpectedly rich. Beyond the historical photographs of the 1924 peace ceremony and portraits of Iban, Orang Ulu, Malay, and Chinese community leaders, visitors find one of the more remarkable geological exhibits in Borneo. The Amber Gallery, a collaboration between the Tun Jugah Foundation and the Geological Survey Department, displays specimens from one of the world's largest amber deposits, discovered in the Merit-Pila Coal Field along the Batang Rejang. The amber dates to the Miocene epoch, roughly 20 million years old, and contains perfectly preserved insects trapped when the resin was fresh: spiders, ants, mosquitoes, centipedes, wasps, and beetles. Carved by local artist Kojan Kabeng of Punan Bah, polished amber pieces glow in shades of orange, red, yellow, and brown.

Upstairs, the fort preserves something equally ancient in a different way. The Iban weaving gallery displays pieces of pua kumbu, the sacred ikat textiles that hold deep spiritual meaning in Iban culture. A silk piece entitled Kara Jangkit, woven by accomplished weaver Gading ak Mayau from Sungai Kain, hangs alongside cotton works and traditional costumes. The largest exhibit is a mural painting depicting the Iban way of life. Beyond preservation, the Tun Jugah Foundation actively promotes the continuation of weaving through workshops, competitions, and partnerships with organizations like Sarakup Indu Dayak Sarawak. In a fort built to control tribal movement, the culture of the people it once constrained now fills every room.