The palace sits on one of the most contested pieces of real estate in North Africa. This hilltop site within Tangier's kasbah has been the seat of political power in the city for centuries. When England held Tangier from 1661 to 1684 -- a possession acquired through Catherine of Braganza's dowry to Charles II -- they built an Upper Castle here. After the English abandoned the city and destroyed their fortifications, the Moroccan sultan Moulay Ismail ordered a new palace built on the ruins. The construction was carried out in the early 18th century by Pasha Ahmad ben Ali al-Rifi, the general of the Jaysh al-Rifi militia and semi-autonomous governor of Tangier. The result was a building that served as both the governor's residence and the sultan's quarters when visiting the city.

The palace was converted into a museum in 1922, beginning a century-long transformation from seat of power to cultural institution. The Museum of Mediterranean Cultures, as it has been known since a comprehensive renovation completed in 2016, occupies the former reception spaces and living quarters. The collection reflects Tangier's position at the intersection of civilizations: Roman mosaics alongside Marinid ceramics, Phoenician artifacts next to Ottoman textiles. The Museum of Antiquities, housed in the former kitchen, contains finds from the ancient Roman cities of Lixus, Cotta, and Volubilis, as well as artifacts from the Tangier region spanning from prehistory through the Middle Ages. The palace garden, visible through ornate doorways, provides a green retreat within the fortified walls.

The palace faces the Mechouar, a formal ceremonial square within the kasbah that once served as the stage for official receptions and military reviews. Buildings flanking the square include a former Bayt al-mal -- an Islamic treasury -- created in the late 18th century, and prisons that held political detainees through various regimes. A former tribunal building also faces the square, its austere facade contrasting with the palace's decorative elaboration. Together, these structures form a compact administrative complex that functioned as a miniature seat of government, reflecting Tangier's historical status as a city that, due to its strategic position at the entrance to the Mediterranean, often operated with a degree of autonomy from the central Moroccan court.

The most striking addition to the museum came in late 2021, when the Kasbah Museum Contemporary Art Space opened in the palace's former prison. The conversion of prison cells into gallery spaces carries its own symbolism in a country still reckoning with its complex political history. The space hosts rotating exhibitions of contemporary Moroccan and international art, adding a living creative dimension to a museum otherwise focused on the deep past. Wooden ceilings carved in the traditional Moroccan style, Roman mosaics set into floors, and ornate doorways frame the visitor's experience throughout the palace. Sultan Hassan I stayed here in 1889, one of the last royal visits before the building's conversion. Today, the kasbah palace belongs to everyone who walks through its doors.