Before it was Sæby it was Maristed, named for the monastery of Our Lady, and before that it was a landing place and a fishing camp on land belonging to the Bishop of Børglum. The promotion arrived on 10 August 1524, when King Frederik I granted the settlement market-town rights — the same rights Viborg held — and the local bishop confirmed them the following year. Market rights meant the legal right to trade, and for a town sitting on a coast with no natural harbour that was the entire business plan. It worked, but slowly. There were 670 people here in 1672. By 1890 there were about 1,800, and by 1930, 2,525.

The harbour was Sæby's permanent problem. Sand travels relentlessly along this coast, and the entrance filled almost as fast as it could be cleared, so the town spent centuries paying to maintain what other ports were simply given. What it got in return was flatfish. Sæby's boats worked plaice above all, along with cod and saithe, and in the 1897–98 season they landed over 20,900 dozen plaice — a quarter of a million fish out of one small town. Inland, the Sæby watermill, built as royal property in 1640, ground grain until the 1920s. The town's prosperity was never spectacular, and that is precisely why so much of it survived: nobody here ever had enough money to knock the old streets down and start again.

Walk Algade and the century of each house is usually written on it. Number 3 is Clasens Hotel, a timber-framed building of 1750 that acquired a neoclassical face in the early 1800s and, later, a clientele of writers — Herman Bang stayed here, and so did Gustav Wied. Number 7, Den gamle Dommergaard, is an 1848 merchant's house that lodged Adda Ravnkilde, born in 1862, dead in 1883 at twenty-one, her writing published only after she was gone. Down at Strandgade 2 the old hospital dates from 1675, with a forecourt from 1565, older than the building it fronts. None of this is a reconstructed heritage quarter. It is an ordinary town centre that happens to be three hundred years old and still in daily use.

One Sæby name travelled a great deal further than the rest. Jacob Severin, born here in 1691, was a merchant who held the Danish trade monopoly on Greenland from 1733 to 1749 — sixteen years in which a man from a silting flatfish port on the Kattegat controlled all licensed commerce with an Arctic colony. He died in 1753, a wealthy man. It should be said plainly that the wealth came from a trade the Greenlandic Inuit, whose hunting and labour produced the goods it moved, had no voice in setting the terms of.

At the entrance to the harbour stands a figure six and a quarter metres tall, facing outward. Fruen fra Havet — the Woman from the Sea — is a ship's figurehead scaled up to the height of a small lighthouse, made by the artist Marit Benthe Norheim and installed in 2001. She has become the town's emblem, which is a revealing choice for a place with five centuries of history to draw on. The older answer sits inland at Sæbygaard, a three-winged Renaissance manor with octagonal corner towers and a museum of seventeenth- and eighteenth-century interiors. Sæby's population is 9,012, and since 2007 it has been part of Frederikshavn Municipality rather than the seat of its own.