We don't only go to restaurants for the food — but does it ever make sense to go to one where you know in advance the food’s not much cop?

Maybe, if the place itself is special enough — like the Orangery at Kensington Palace. I’ve been going steadily for years, even though the food has never been much more than a token offering.

The exterior, in rich brick with deeply rusticated pilasters, is imposing enough — especially when approached up the avenue of finely topiaried bay trees, smoothly trimmed somehow even over the tops (they are lingams, obviously, and as such a most fitting tribute to Princess Diana, unlike her wretched fountain).

The interior, though, is even better. It’s one long narrow room, white, in the most perfect proportions, with a single entrance in the middle and rounded ends beneath fine arches. On sunny days it’s one of the loveliest spaces in London.

There is no certain documentation for the architect of the Orangery, which was built in 1704 for Queen Anne, but part of its thrill for me is thinking that it is by the English architect I most revere, Nicholas Hawksmoor, then clerk of the works under Wren. Hawksmoor scholar Kerry Downes admits there is only stylistic evidence for it being Hawksmoor’s work but seems certain nonetheless, comparing its handling of space to his achievements at Blenheim and Easton Neston. Strangely, I’m not often invited to either, so it’s the Orangery for me.

Old pictures show the whole room empty, save for a few benches. Now it’s been kitted out with a central island to dispense the food and filled with ironwork chairs and reasonably spaced tables, each adorned with a small citrus in a pot, a nice homage to the building’s original use. However, the overgrown ficuses obstructing any clear view out of either end of the building are a mistake.

English Heritage allows no alterations to the fabric — so the loos are in a hut hidden around the back. And there’s no kitchen — the food is merely dished out. Meals on wheels, then. And not dished out with any great enthusiasm either — the service is lamentably slow and vague.

The Orangery is run by the firm Digby Trout. The company specialises “in the operation of restaurants and cafés within venues where eating may not be the prime reason for our customer’s visit”. You can say that again. It has restaurants at the Ashmolean, Duxford, the Tower, the Natural History Museum, the Science Museum, Stonehenge ...

You could come here and have a cuppa and cakes, if you eat those things. For lunch, the best plan is to order defensively, choosing the simplest dishes, preferably the ones that don’t even need anything as advanced as warming. Among the hot plates spring casserole of lamb with roasted new potatoes (£10.95) was bland and fatty, the potatoes so overlarded with bay they were inedible. A broccoli quiche (£8.95) had the distinctive stodginess that comes from being originally cooked long ago and given time to settle.

But a “sharing plate” of cold cuts, at £17.95 for two, was better — two slices of decent enough cooked English ham, two wedges of fair West-combe cheddar, six slices of three breads (seedy, onion, walnut), cheese straws, a ramekin of average houmus, a few cherry tomatoes, some bouncy piccalilli and a spicy plum chutney. In other words, a poncey ploughman’s. Or a picnic, eaten off a plate, under cover. Heavenly cover, though.

And that’s ample for lunch, really: enough to absorb a glass or two of wine. There’s a short and unremarkable list, beginning at £14.50 for basic brews from southern France. The most expensive white, a Chablis at £25.95, had almost no character. A Rioja Crianza at £16.95 was tasty enough, though — and, in this place, anything nice is doubly delicious.

Is it ridiculous to recommend ever putting yourself into the hands of mass caterers like this when you’re not even a tourist? Perhaps. But I like being in this room so much that there are always days when I’d rather have lunch here than almost anywhere else in town.