The founding legend is specific and vivid. When Nebuchadnezzar II destroyed Solomon's Temple, the Kohanim who were serving at the time escaped Jerusalem and eventually found themselves on Djerba. They brought fragments of the destroyed sanctuary with them, and the synagogue they built around those relics became, in the minds of its community, a direct link to the sole sanctuary of Judaism. Genetic evidence has strengthened the oral tradition: tests for the Cohen modal haplotype show that the vast majority of male Jews on Djerba claiming priestly status share a common ancient male ancestor consistent with priestly lineages across the Jewish diaspora. The island became known among Jews as the Island of the Kohanim. In modern times, the local Jewish men still wear a black band around their pants to signify the destruction of the Temple, carrying the memory of that catastrophe on their bodies.

A second tradition gives the synagogue its name. According to this story, a young girl, a ghriba, an isolated one, lived near the spot where the synagogue stands. She was not accepted by the other people of the area. When she died, her body was found uncorrupted by the Jews of a nearby village. They buried her in a cave that became a pilgrimage site, and the synagogue grew around it. A recess beneath the holy ark still marks the spot where the girl's body is said to have been found. Each year, between Pesach and Shavuot, pilgrims come for festivities that begin on the 14th of Iyar in remembrance of Rabbi Meir Baal HaNess and continue through Lag BaOmer on the 18th of Iyar, honoring Rabbi Shimon bar Yochai. The interior walls are decorated with hand-painted ceramic tiles in green, yellow, blue, brown, and white, resembling qallalin tiles, creating an atmosphere of devotional beauty.

The synagogue's significance has also made it a target. In 1985, during the festival of Simchat Torah, a local policeman assigned to guard the building opened fire on a crowd of celebrating Jews, killing three people, including a child, and wounding fifteen. The attack came in response to Israel's Operation Wooden Leg, which had targeted PLO headquarters in Tunisia. On April 11, 2002, a truck loaded with explosives detonated near the synagogue, killing nineteen people: fourteen German tourists, three Tunisians, and two French nationals. Al-Qaeda claimed responsibility; the bombing was later traced to Khalid Sheikh Mohammed. On May 9, 2023, a member of the Tunisian National Guard shot and killed two Jewish cousins and two Tunisian security guards outside the building before being killed himself. Each attack tested the resilience of a community that has endured for millennia.

Djerba is home to around 1,300 Jews, one of the last surviving Jewish communities in the Arab world. El Ghriba is the center of their spiritual life and a pilgrimage destination for Jews from across North Africa, Europe, and Israel. An independent administration committee, established at the end of the nineteenth century during the French protectorate, oversees the synagogue and organizes the annual pilgrimage, distributing revenues to village elders. The buildings were extensively renovated in the nineteenth century, though the foundations may date to the sixth century BCE. The Tunisian government has extended protection and encouraged Jewish life on the island, particularly since the Arab Spring. The synagogue's existence, in a region where Jewish communities have largely vanished, is itself a statement. El Ghriba persists because the people who tend it have chosen, for twenty-five centuries, not to leave.