According to local folklore, the mosque was built by Sheikh Karimul Makhdum, a Syrian Arab trader and Sunni Sufi scholar who arrived on Simunul in 1380. If the tradition is accurate, his mosque predates the establishment of the Sultanate of Sulu and represents the earliest institutional foothold of Islam in what would become the Philippines. The original structure would have been built in the indigenous Moro pagoda architectural style, a form that drew on local traditions rather than the domed mosques of the Middle East. Wooden pillars found within the current mosque grounds are theorized to be either remnants of that original 1380 pagoda mosque or from a later wooden structure built centuries after the first was ruined. The distinction matters to scholars; to the community, both possibilities confirm what they already know: this ground has been sacred for a very long time.

The mosque that preceded the current building was largely destroyed in 1941, when Japanese forces occupied the Philippines during World War II. Whether the burning was deliberate targeting or collateral damage in a broader campaign, the loss was devastating to the community. When the mosque was rebuilt in the 1960s, the architecture chosen was the Arabic onion-dome style, a departure from the traditional Moro pagoda design that had characterized the site for centuries. The shift reflected broader currents in the Islamic world, where Middle Eastern architectural forms were becoming standardized symbols of Muslim identity. In recent years, scholars, community leaders, and local residents have advocated for rebuilding the mosque in the traditional Moro pagoda style, arguing that the original form carries cultural significance that the imported dome cannot replicate.

The mosque has been declared a National Historical Landmark by the National Historical Commission and a National Cultural Treasure by the National Museum. On 7 November 2023, the National Historical Commission unveiled a new historical marker at the site. These designations place the mosque alongside the Philippines' most significant heritage sites, yet its location on Simunul Island in Tawi-Tawi, the country's southernmost province, means it remains far from the consciousness of most Filipinos. Tawi-Tawi is closer to the Malaysian state of Sabah than it is to Manila, and reaching Simunul requires a journey that few undertake. The mosque's importance is therefore both universally acknowledged and practically invisible, a national treasure that most of the nation has never seen.

Simunul sits in the Sulu Archipelago, a chain of islands where Islam arrived centuries before Spanish colonizers brought Christianity to the northern Philippines. The province of Tawi-Tawi is 99 percent Muslim, and the Sheik Karimul Makhdum Mosque remains a functioning place of worship, not merely a monument. The four 17th-century pillars stand amid the prayers of people whose faith was shaped by the very tradition those pillars represent. It is an unbroken thread, or something close to one, connecting a Syrian trader's arrival more than six centuries ago to the community that gathers here today. The pillars are at least 400 years old. The faith they support is older still.