Sir Henry Griffith built the present hall between 1601 and 1610 after the Crown made him a member of the Council of the North. He worked from designs attributed to Robert Smythson, the Elizabethan architect whose other houses include Hardwick and Wollaton. A surviving Smythson plan shows a square block with bay windows around a small internal courtyard, but the house as built does not match the drawing exactly. Two of the corner bay windows are semicircular instead of five-sided. The west front has been heavily altered. The whole arrangement reads as a first version that picked up changes during construction, either from Smythson rethinking or from Sir Henry pushing back. The semicircular windows are almost certainly Smythson's own. They appear on two of his other plans.

The main facade is built a storey higher than the rest of the house so the second floor could carry a Long Gallery running its full length. Wagon-roofed, richly plastered, the gallery is the showpiece of the interior, and Francis Johnson restored its ceiling in two stages between 1951 and 1974. Inside the gallery and the surrounding rooms much of the original seventeenth-century fitting still stands. Carved woodwork, plaster, alabaster. The alabaster overmantel in the Great Hall depicts the parable of the Ten Virgins, copied from a design by Pieter Bruegel the Elder that Hieronymus Cock printed in Antwerp in the 1560s, the engravings travelling from Flanders to the Yorkshire Wolds and onto a fireplace.

The walled flower garden is laid out with a games motif. A central chessboard is paved in black and white. Other gardens around it are themed by colour and games, draughts, snakes and ladders, hoop toss, each one a self-contained room of plants. The whole estate holds three thousand plant species, including the National Collection of Campanulas. A market garden grows seasonal vegetables. Whimsical statues turn up throughout the grounds. The woodland walk is locally famous for snowdrops in February, when the trees flush white at ground level before the leaves come back. Both the Elizabethan hall and the older Grade I-listed Norman manor house from 1173 next door are open to visitors year-round.

Roger de Stuteville built the first manor house on this site in 1173, the same one that survives as the Norman ruin next to the Elizabethan hall. The Welsh Griffith family inherited the estate in 1457 and held it through to Sir Henry, who built the Elizabethan house. His daughter Frances married Sir Matthew Boynton, the governor of Scarborough Castle, in the 1620s. Her sister Anne is the one who, according to the legend, never quite left. The Boyntons held the place for centuries. When the eleventh baronet died in 1899 it passed through the Wickham line to Marcus Wickham Boynton, who ran a successful stud farm and was High Sheriff of Yorkshire in 1953. He died in 1989 and left the property to a twelve-year-old distant cousin, Simon Cunliffe-Lister, grandson of the politician William Whitelaw. Cunliffe-Lister and his mother now manage it under the Burton Agnes Preservation Trust.