Peisistratos seized power in Athens in the 6th century BC and ruled as tyrant. After his death, his sons Hippias and Hipparchos took over and around 520 BC began a temple meant to outdo every other temple in the Greek world. Their direct rivals were the Heraion of Samos and the second Temple of Artemis at Ephesus, both ranked among the wonders of antiquity. The Athenian project was designed by four architects with names mostly lost to history: Antistates, Callaeschrus, Antimachides, and Phormos. The temple was to be Doric, built of local limestone, on a platform 41 by 108 meters with eight columns across each end and twenty-one along each side. Then the tyranny fell. Hippias was expelled in 510 BC, and the partial foundations sat untouched for 336 years. Aristotle, in his Politics, used the temple as a textbook example of how tyrants kept their populace busy with grand projects to leave them no time to rebel.

In 174 BC, Antiochus IV Epiphanes, the Seleucid king who claimed to be the earthly embodiment of Zeus, decided to revive the project. He hired the Roman architect Decimus Cossutius and rewrote the design entirely. The new plan called for 104 columns, three rows of eight at each end and a double row of twenty along each side. Each column would stand 17 meters high and 2 meters in diameter. The order changed from Doric to Corinthian, marking the first time Corinthian capitals appeared on the exterior of a major temple. The material changed from local limestone to expensive Pentelic marble. Then Antiochus died in 164 BC, and the project stopped again. The temple was perhaps half finished. Then Lucius Cornelius Sulla sacked Athens in 86 BC and shipped some of the unfinished marble columns back to Rome, where they were re-used in the Temple of Jupiter on the Capitoline Hill.

Hadrian, who became Roman emperor in AD 117, was a philhellene who loved Athens. When he visited in AD 124-125, he committed to finishing what tyrants and Seleucids had started. He kept Cossutius's design with little change. The completed temple was dedicated in AD 132, and Hadrian took the title Panhellenios, leader of all Greeks, in commemoration. Inside, a colossal chryselephantine statue of Zeus filled the cella, ivory and gold over a wooden frame. Pausanias, two generations later, wrote that the Zeus statue was exceeded in size only by the colossi at Rhodes and Rome. A second colossal statue, of Hadrian himself, stood behind the temple, and every major Greek city contributed a statue of the emperor to ring the precinct. Hadrian was now the patron of Athens, and the temple proclaimed it. The glory lasted barely a century. Germanic tribes called the Heruli sacked Athens in AD 267 and the temple was badly damaged. It was probably never repaired. A 5th-century earthquake completed the ruin.

Once the building stopped being a temple, it became a quarry. Throughout the Byzantine period, marble blocks were pulled out for churches, houses, walls, plaster. By the time Cyriacus of Ancona visited Athens in 1436, only twenty-one of the original 104 columns were still standing. In 1759, the Ottoman governor of Athens, Mustapha Agha Tzistarakis, used gunpowder to bring down one of the surviving columns. He needed the marble to burn into plaster for the mosque he was building in Monastiraki, which is still standing in the square below the Acropolis. A Greek inscription carved into a surviving column records the date: 27 April 1759. During the Ottoman period, Greeks called the ruin the Palace of Hadrian, while Turks called it the Palace of Belkis, after a folk legend that it had once been home to the Queen of Sheba, supposedly the wife of Solomon.

Excavations began in earnest in 1889 under Francis Penrose of the British School in Athens, the same man who restored parts of the Parthenon. German and Greek archaeologists followed. Today the temple is an open-air monument, part of the unified archaeological precinct of central Athens. The sixteen surviving columns rise 17 meters off their stylobate, taller than a five-story building, capped with their elaborate Corinthian capitals. The seventeenth column from the great storm of 1852 still lies where it fell, broken into stacked drums like a fallen tree. From the Acropolis 500 meters to the northwest, the columns look like a cluster of stone reeds. Stand among them and the scale becomes terrifying. In 2007, a small group of modern Greek pagans held a ceremony to Zeus on the grounds, the first since the 5th century. Whether the god noticed is unrecorded.