When the founder of the Umayyad Caliphate, Mu'awiya I, died in 680, he left the caliphate to his son Yazid. The hereditary succession was not universally accepted, least of all by the old Medinan elite. Two men fled Medina to avoid swearing allegiance to Yazid: Husayn ibn Ali, grandson of the Prophet Muhammad, and Abd Allah ibn al-Zubayr, grandson of the first caliph Abu Bakr and nephew of Aisha. Husayn went to Kufa, where his supporters were preparing a revolt against the Umayyads. On the way he was intercepted. At Karbala in October 680, he was killed, an event that would become central to Shia identity for the next fourteen centuries. That left Ibn al-Zubayr, a Qurayshi aristocrat with deep legitimacy among Medinan traditionalists, as the leading alternative to Umayyad rule. He based himself in Mecca and called himself "the fugitive at the sanctuary".

The Medinan aristocracy, never fond of the Umayyads, regarded Yazid as dissolute and unfit for the caliphate. Their economic interests had been squeezed by Mu'awiya's agricultural projects around the city. They expelled the Umayyad family members, roughly 1,000 of them including the future caliph Marwan ibn al-Hakam, from Medina and denounced their allegiance to Yazid. Yazid responded by sending an army. Muslim ibn Uqba al-Murri, a devout and reluctant commander, led 12,000 Syrian troops to subdue the Hejaz. At the Battle of al-Harra on August 26, 683, the Syrians broke the Medinans' resistance, and then they sacked Medina itself. The plundering became one of the impious acts for which later Muslim tradition would condemn the Umayyads. In Julius Wellhausen's phrase, subsequent tradition remembered Muslim ibn Uqba as "the heathen incarnate," though earlier sources represented him as devout. Muslim fell ill on the march south and died. His lieutenant, Husayn ibn Numayr al-Sakuni, took command for the assault on Mecca.

When the Umayyad army arrived before Mecca in September 683, Ibn al-Zubayr had company. Many refugees from al-Harra had fled south. The commander of the Qurayshite forces at al-Harra, Abd Allah ibn Muti, played a leading role in the defense. The pro-Alid leader Mukhtar al-Thaqafi had arrived. So had a band of Kharijites from central Arabia under Najda ibn Amir al-Hanafi. In the first engagement, Ibn al-Zubayr's forces won. The Umayyads regrouped and, on September 24, placed the city under siege. They set up catapults and began bombarding Mecca with stones. Ibn al-Zubayr established his command post on the grounds of the Grand Mosque itself, a few meters from the Kaaba. The defenders erected a wooden structure covered with mattresses over the Kaaba to protect it from flying stones. It was a reasonable precaution. It did not work as hoped.

On Sunday, October 31, 683, the Kaaba caught fire. The wooden covering burned, and so did the cube beneath it, reducing the most sacred structure in Islam to rubble. The sacred Black Stone cracked into three pieces. The responsibility for the fire is contested in the sources. Many later Muslim accounts blamed the Umayyad besiegers, whose catapults had been throwing burning projectiles for weeks. The historian G.R. Hawting notes that the bombardment figures prominently in subsequent lists of Umayyad crimes. More reliable earlier accounts, however, attribute the fire to a torch carried by one of Ibn al-Zubayr's own followers, caught by the wind and blown onto the building. The destruction of the Kaaba became a scandal across the Muslim world, a catastrophe that neither side could spin in its favor. Whatever the immediate cause, the siege that caused it was the responsibility of the Umayyad command, and that is the way most later tradition remembered it.

Yazid I died on November 11, 683, in circumstances that were sudden and, to the Umayyad court at Damascus, catastrophic. News reached the besiegers on November 26. Husayn ibn Numayr opened negotiations. The Umayyad authority had effectively collapsed in the provinces, and shaky even in Syria. Husayn offered to recognize Ibn al-Zubayr as caliph, provided that Ibn al-Zubayr pardon him and accompany him back to Syria. Ibn al-Zubayr refused the second condition, knowing that a trip to Damascus would place him under the control of the Syrian elites. Husayn left with his army. The siege had lasted 64 days. The Kaaba was in ruins. The Black Stone was in pieces. The Umayyad court in Damascus declared Yazid's sickly young son Mu'awiya II caliph, but Umayyad authority was broken for the moment. Across most of the Muslim world, Ibn al-Zubayr was acknowledged as the rightful caliph.

After the Umayyad retreat, Ibn al-Zubayr began rebuilding the Kaaba. Most Meccans, led by Ibn Abbas, initially left the city fearing divine retribution at the sight of the sacred house torn down. Only when Ibn al-Zubayr himself began demolishing the burned remains did they return to help him. His reconstruction changed the original plan. He built it entirely of stone, where the old Kaaba had been alternating layers of stone and wood, and he gave it two doors, an entrance in the east and an exit in the west. He included the semi-circular hatim wall into the structure proper, bringing the building closer to what Muhammad was reported to have originally intended. He gathered the three fragments of the Black Stone, bound them in a silver frame, and placed them inside. Ibn al-Zubayr would hold Mecca for another nine years as the civil war dragged on. In 692, a second Umayyad siege under Abd al-Malik ibn Marwan retook the city and killed Ibn al-Zubayr. The western gate was walled up, the hatim separated from the building, and the Kaaba reverted to something closer to its pre-Islamic outline. That is the form in which it stands today.