Blistering barnacles! It is possible to stay overnight in a converted fisherman's hut on Whitstable seafront. To do such a thing has always been my dream, and so my boyfriend Alex and I drove down from London, collected a key from the Hotel Continental, then navigated our way through the town's narrow streets to the strand, where we let ourselves into our very own hut. It was spotless and spartan and very Peter Grimes, complete with a steep stepladder, a shipshape little bathroom, and no decoration apart from a jaunty lifebelt marked 'Welcome aboard'. Waking up the next morning, we threw open the curtains and saw nothing but shingle, ships, seagulls and sea, glorious sea. The only sound was the wind whipping rigging gently against mast.

Whitstable is a rough diamond. Once an oyster boom town, producing six million bivalves a year for the Victorian market, it suffered a lean period when the industry declined. The post-war years were particularly tough and so a lot of the town's modest original buildings were preserved rather than developed. The result is a beautifully quaint town, untouched by 1960s brutalism and still not overly gentrified (Budgens and The Co-operative are the main supermarkets). It is the kind of place where you can still pick up paperbacks for 50p – we went home with a haul of Penguin Classics.

Perched below the Isle of Sheppey and just above Canterbury, it's an East Coast town without the self-regard of Aldeburgh or the visitor amenities of Southwold. But like those two, it has a special quality of light, the sea low and luminous, the horizon exhilaratingly wide and blue. You can see why artists are attracted to the landscape, and Whitstable boasts ten permanent galleries, from the informal – such as the Beach Hut on Sea Wall – to the well-established: Wendy Croft's Caxton Contemporary on Oxford Street offers a fascinating array of fine art and sculpture and a warm welcome. Complete strangers, we stumbled upon an open view there and were plied with fizzy wine and a tour of the new collection, which included work by Simon Tupper, a colourist in the mode of the late Craigie Atchison, and Jenny Wiggins, who creates compelling canvases using cooperative snails and their slimy trails.

The gallery party was a glimpse of a community many seaside towns have lost. So was the snatch of band practice we overheard emanating from the window of the Sea Cadets' headquarters while taking an early evening stroll. More local bonhomie can be enjoyed at the annual oyster festival which runs from 24 to 30 July – a month with no 'r' in it, but this has been its date since Norman times, apparently, something to do with the low season being the best time for the hard-working townsfolk to take a day off, or the name day of St James of Compostela, the patron saint of scallops, depending on who you ask. Festival highlights include the 'Landing of the oysters', when the 2nd Whitstable Sea Scouts (another boys' group based in the town) haul the shells from the sea to be blessed by the clergy and presented to the mayor. There's also a local custom of grotter-building which is still going strong (a grotter being a hollow mound of sand or mud, decorated with oyster shells), as well as a town-wide parade of walking, talking oysters.

This is, we realised, a wonderfully eccentric place. The most famous restaurant in town is Wheelers Oyster Bar, established in 1856 and long and loudly acclaimed by food critics – and yet it only seats 12 diners, inside what is basically a living room made open to the public, complete with standard lamp, cherished knick-knacks, and a loo at the back that is accessed by going into the street and round the block. The formidable manageress, Delia Fitt, issued brief directions and I stumbled into the wrong garden in the rain; yet our dinner was outstanding, from a superb seasonal menu by chef Mark Stubbs. The look of rapture on Alex's face when he tasted the Guinness-battered oyster said it all. The pricing is scrupulously fair, with a main course – for example, monkfish osso bucco or smoked haddock three ways – costing £17 or £18, a simple fish soup £2.80, and a bowl of samphire, in season, £1. Alcohol is BYO. An unfathomable gem.

If, however, you desire a sea view while you dine, as well as more conventional restroom facilities, you might do better at Pearson's, or The Whitstable Oyster Company's Fish Restaurant. Both overlook the beach and each other, enjoying, presumably, a healthy rivalry that helps explain why they are both so good. Pearson's has an extensive cocktail list and offers the juiciest, creamiest mussels this side of New Zealand, as well as traditionally smoked sprats (like a poor man's sardine). The Whitstable Oyster Company has huge windows, and in the right seats, diners can watch the world and his dog pass by on the beach. Good local walks include, from the West Beach, a section of the Saxon Shore Way footpath, which runs 163 miles from Gravesend to Hastings.

Rollicking home, we bumped into our new friends from the art gallery and were invited to a gig at the Old Neptune pub. Later, our cabin was cosy despite the onshore wind and in the morning the cooked kipper breakfast, served at the Hotel Continental, would have been hearty enough for Barnacle Bill. We left Whitstable reluctantly, driving inland through its urban sprawl. The seaside area is the only place to be in that town, but oh, what a seaside it is to be beside.

Where to stay
A fisherman's hut in high season (April to September) costs £150 per night for a couple; £175 for a family of four. Hotel Continental, 29 Beach Walk, Whitstable (01227 280 280; hotelcontinental.co.uk).

Where to learn
Wheelers' brilliant chef Mark Stubbs runs the occasional Shoreline Cookery School. A day-long course in filleting fresh fish and preparing crustaceans, stock, soup and sauces starts at 8am and finishes with a feast. £140, Wednesdays only. 8 High Street, Whitstable. Call Wheelers on 01227 273 311 to go on the waiting list.

What to buy
Oyster knives can be purchased from the Whitstable Shellfish Company, from basic (£3.50) to beechwood (£7.50). Westmeads Road, Whitstable (01227 282 375).

What to read
Of Human Bondage, W Somerset Maugham's
thinly veiled autobiographical novel, describes 'Blackstable' and the nearby town of 'Tercanbury'. We can't imagine where he means. Maugham's uncle Henry, with whom he lived for an unhappy period after he was orphaned, was vicar of Whitstable.

What to take
A hearty appetite and a sailor's jumper (Bretton stripe anchor top, £20, at oasis-stores.com).

Where to eat
Jojo's, the popular tapas bar, has just reopened in larger premises on Herne Bay Road. Its fresh local cuisine, from canon of lamb to sprats, is carefully sourced (all fish comes from the local 10m trawler Millennia, Whitstable's first Responsible Fishing Scheme vessel) and gets rave reviews. 2 Herne Bay Road, Tankerton (01227 274 591).

What to see
First Venice, now Whitstable… the town's third Contemporary Visual Art Biennale 2010 runs from 19 June to 4 July (left, Mikhail Karikis' Charon the Ferryman collage). It will feature performance work such as Katie Paterson's Black Firework for Dark Skies and Adam Chodzko's Ghost, a custom-built canoe that the artist will be paddling to the local prison hulk graveyard Deadman's Island. Volunteers are invited to join him, one at a time(whitstablebiennale.com).