Almost no town in the Americas was fought over so persistently. A treaty in 1681 handed Colonia back to Portugal, and over the next hundred years the place passed between the two crowns again and again, captured, ceded, swapped in distant European treaties, and captured once more, changing hands something like ten times in all. Spain held it, then surrendered it under the Treaty of Paris in the 1760s; Portugal returned, then lost it again to Spain under the Treaty of San Ildefonso in 1777. For the soldiers and settlers who actually lived here, the great-power chess game meant a town built, besieged, burned and rebuilt, its loyalty redrawn by men who had never seen it. The competing empires left the place with something no planned colony could fake: two layers of street, two ways of building, fused into one small town.

You can still walk the seam between them. Colonia grew on a low peninsula reaching into the brown river, and the heart of it, the 16-hectare Barrio Historico, was the Portuguese old town, once sealed behind a fortification wall thrown across the neck of the peninsula. Most of that wall came down in 1777, but its line survives, and on the Portuguese side the streets wander in an irregular, organic tangle, the way medieval Portuguese towns did. Step beyond the old wall line and the logic changes completely: there the Spanish laid out their quarter in the rigid checkerboard grid that was their colonial signature across the Americas. One small town, two empires' ideas of order, set side by side.

The Barrio Historico is now a UNESCO World Heritage Site, and it is the kind of place where the past lies just under your feet. Cobblestone lanes laid by the Portuguese in the seventeenth century run down toward the ferry terminal and the river. The wooden drawbridge of the Porton de Campo still guards the old city gate. Near the tree-lined Plaza Mayor stand the stone Basilica of the Holy Sacrament, an eighteenth-century Portuguese house, the ruins of the Convent of San Francisco and the lighthouse that rises beside them, offering a view across the whole peninsula. The most famous lane of all is the Calle de los Suspiros, the Street of Sighs, a steep run of wedge-shaped stones whose name is wrapped in competing legends, of sailors, of a doomed love, of the condemned led to the water. No one knows which story is true, which is exactly why people keep telling them.

For all its violent history, modern Colonia is a place of calm, an easy ferry ride across the estuary from the roar of Buenos Aires. Three ferry lines run the crossing, and visitors step off into a town of roughly 32,000 people where old cars sit parked on streets that look essentially unchanged in two centuries. Beyond the museums and the basilica, there are quiet surprises, including the Plaza de toros Real de San Carlos, a disused bullring on the edge of town, now revived as a stage for performances. Colonia was once a prize that two empires bled for. Today its reward is simpler: it is one of the loveliest small towns on the Rio de la Plata, and it knows it.