The Aghlabids, who had wrested control of Ifriqiya from the Abbasid caliphate and built their own independent dynasty in the early ninth century, understood that North Africa's coast required systematic defense. They constructed a network of ten observation towers along the shore at Sfax, linked by a fire-signal system that could relay warnings of approaching threats across dozens of miles. The kasbah was the central node: two high towers for monitoring the sea and communicating with surrounding stations, two mosques for the garrison's spiritual needs, and an underground facility that may have served as an emergency prayer room. Built atop the ruins of an earlier palace south of Mahares, inherited from previous civilizations, the kasbah was as much a statement of authority as a military installation.

As the Aghlabid dynasty matured and the ribat fortification grew, the kasbah evolved from watchtower into the political center of the city. It became the seat of successive governors who administered Sfax under the Fatimids, Zirids, Almohads, and Hafsids, each dynasty inheriting the building along with the territory. The fortress was where power physically resided: whoever held the kasbah held Sfax. At the end of the Hafsid period, the local ruler Abu Abdullah al-Makeni seized the city from the central authority, operating from within these same walls. When the Ottomans arrived, the administrative system shifted. The governor's role changed, and the Aga, the local military commander, became the de facto ruler, responsible for all kasbahs and guards in the city. Ottoman military culture reshaped the building's interior even as its exterior retained its medieval character.

The French conquest of Tunisia in 1881 brought the building's military function full circle. French gendarmes replaced Ottoman soldiers, and the kasbah served as a colonial garrison until Tunisia's independence in 1956. The Tunisian National Guard briefly occupied it before the building was abandoned. For nearly twenty years it sat neglected, its walls deteriorating, its rooms empty. Then, in the 1980s, a restoration project overseen by Zouari transformed the kasbah into a museum of traditional Sfaxian architecture, which has characteristics distinct from other Tunisian cities. The building proved an ideal vessel: its own architectural elements, accumulated over a millennium, constituted much of the exhibit. The museum now houses galleries of building techniques and tools, sectional models of the medina's distinctive fences, and exhibitions of religious architecture from the old city.

The most striking transformation may be the smallest. Where the kasbah's prison once stood, its cells and punishment rooms now house the Art Gallery of Mohammed Al-Vandari, a gallery of fine arts. The upper mosque has been repurposed as an exhibition space for the religious establishments of the old city. Original waqf documents, old metal door knockers, and ancient hinges are displayed alongside architectural models, creating a portrait of a city that built with care and intention. Walking through the kasbah today, you move through time as much as space: from the Aghlabid watchtower foundations to Ottoman military modifications to French colonial alterations to the curatorial choices of the modern museum. Each era is legible in the stone, and none has been erased.