Gambling baronets, runaway residents, terrified kids and even a perfectly well-run hospital, all play their part in the rich history of Leybourne Grange – but how you remember it probably depends on your age.

Younger residents might only know it as Leybourne Chase, the fresh, new, tree-lined housing development completed in recent years, while those a bit older might still be inclined to think of a spooky derelict site, where security guards patrolled weed-strewn paths in the shadow of a crumbling old manor house.

Not many would remember the manor in its real heyday – you’d have to be well over 150 to recall former owner Sir Joseph Hawley, aka The Lucky Baronet, galloping around the grounds and quaffing champagne and sherry in celebration of his latest Derby winner.

For everyone else, who knew it between 1936 and 1996, the name Leybourne Grange will always conjur the memory of a huge hospital – or "colony for mentally defective persons" – where 'patients' with severe learning difficulties were sent to live, before social attitudes to mental health began to change and "care in the community" rendered the grange and other such hulking facilities redundant.

While it was functioning as a hospital, the grange could cater for 1,200 residents, and my first real awareness of the place was when one of them walked into our garden – our house being about a mile from the hospital – in the late 1980s when I was about nine or 10.

"We were playing on the driveway where dad parked his car," recalled my younger brother Nick, who was with me at the time. "The man walked up, and I froze because he looked scary."

I pushed Nick towards the house and into the kitchen, before the man, who seemed almost a giant to us, pushed his way past my mum at the front door; then he went upstairs – much to the surprise of my dad who was getting changed and wearing only his pants.

Our visitor by contrast was fully clothed, except for the fact he had no shoes, his feet clad only in several pairs of socks.

Fortunately he was happy to be given a tour of the house and garden, but he and my dad made for an odd couple as they strolled the garden together, arm in arm, in their contrasting outfits.

He picked up a rotten apple and my dad persuaded him not to eat it; while my mum phoned the police.

"Then the police turned up and the guy just got straight into the police car," recalled my brother. "Surprised they didn’t think dad was the one they were picking up, walking around in his underwear."

I remember feeling a bit sad thinking about that man being taken off in his socks back to the hospital, and I thought I'd probably try to escape too, if I was in his socks, but it was just the way things were.

If you'd told me at that point I'd end up living down at Leybourne Grange I would have been sadder still, but that's where I am now, and generally happy about it too – a literal stone's throw from the old hospital and only mildly psychologically disturbed about the fact I'm writing this at midnight.

Later on, after the hospital shut down, my brother recalled exploring the grounds around the derelict hospital, which seemed "like something out of a horror film".

And he's not the only one. For a generation that followed, the Grange was a place to go if you wanted to scare yourself to the point of needing psychiatric help; the irony being of course that there was none now to be found.

No nurses on hand, just ex-army security guards who appeared at dusk, ready to use their restraint training and combat skills indiscriminately on any tearaway kids, burglars, ghost hunters or actual ghosts they could find.

Because it was haunted of course, as far as every teenager in and around West Malling and Leybourne was concerned.

Pictures from the time before Taylor Wimpey transformed the site document the rapid decline of the Grange as it fell into disrepair, crumbing into the past. It became a magnet for supernaturalists and urban explorers, but also photographers who saw opportunity to peer into a place that somehow stood stuck in time – or stuck out of time, in a limbo, gateway world between the old and new.

One of them was a young Jamie McGregor Smith, who shot a series at the Grange after he left university – picking up one of his first awards in the process, before going onto embark on a successful international photography career.

Jamie recalls it as an "exciting/terrifying place to document", but some tales from the derelict Grange are so terrifying that the tellers must name anonymous.

"I went there on my bike with two friends when it was derelict and got caught short," recalled one. "I did a poo in the clock tower!"

What The Lucky Baronet would have made of that story – having built the clock tower after winning the 1858 Derby with his horse Beadsman – is anyone's guess.

Or even more disturbing, from another anonymous intruder, who changed the names in the story to protect the guilty.

"The last time was when I took John's dog down there with his younger brother," he recalls. "The dog killed all the chickens in a house just next to the Grange, and it was a house for disabled people.

"They were all in the garden crying at the dead chickens. We ran away but a man chased us on a motorbike, and he held John's brother in his house while I got John's mum.

"She wasn’t at home but an au pair was, and I dragged her down there. She didn’t speak English, but she walked into the house and dragged out John's brother while swearing at the man.

"That was quite the day."

It's all a far cry from when the Grange was tightly-run hospital, and while many say mental health services improved with the closure of such institutions, others say residents received fantastic care at the site.

Jamie Weller recalled how his mother Jennifer Weller founded the riding school – now Leybourne Grange Riding Centre For The Disabled – in 1973.

"At the time this was part of the hospital it started in an old stable that was there it was originally a stud stable for the main house, pre hospital not there now.

"The patron for this to help with raising fund was Jack Warner (who played TV's Dixon of Dock Green).

"The clients were marched over villa by villa – you could hear them coming, calling out for 'Wella' which was how the patients referred to my mum – they loved being there."

He described his mum, who had cerebral palsy, as "a bit of a force", adding: "She rode before she could walk and at the time was one of few if any disabled riding instructors."

David Bellchambers recalled working in the administration section of the hospital between the late 1960s and 1974, before he moved to New Zealand, and said Leybourne Grange was "a good place to work and employed good honest people".

"It had a family atmosphere," he added. "Indeed, several generations of a family would have been employed ranging from porters, drivers to nursing staff.

"The hospital was virtually a self-contained unit and enabled some patients to work on the farm that formed part of Leybourne Grange. It had its own laundry, kitchens to prepare meals and even a piggery.

"Staff employed ranged from doctors, nurses, engineers, porters, seamstresses, drivers to name a few.

"It was a sad day when a decision was made to close the place down. The patients had lived most of their lives there and were to be transferred out into the community and perhaps other institutions. The care they were given would suddenly end; I shudder to think what they thought was happening to them."

"I understand the whole hospital was left abandoned and fell into disrepair and vandalised. It lay like this for some years ignored and forgotten, a black mark in my mind on the Ministry of Health."

Any yet with the redevelopment of Leybourne Grange now almost complete – with 716 new homes and around 1,200 -1,500 new residents – those days of dereliction too are also over, soon to be as forgotten as the hospital before it and days of the Lucky Baronet before that.

Except the Lucky Baronet is not forgotten of course.

Sir Joseph Hawley's coat of arms now gleams anew, with a fresh lick of paint on the facade of the almost fully refurnished manor and the roads around now bear his name along with the Derby winners that netted a huge fortune – Hawley Drive, Beadsman Crescent, Teddington Drive, Bluegown Avenue – although oddly there's no mention of Musjid, the horse that won him the 1859 Derby, netting Hawley £80,000 from the bookies on top of his £5,400 prize money.

That's more than £10 million in today's money, but if he turned up in a time machine with the original winnings, he might not even be able to afford a mortgage on one of the apartments going for between £395,000 to £700,000 in his old manor.

The history of Leybourne Grange stretches back over hundreds of years.

Historic England list the house as being built around 1850 by architect SW Daukes, presumably for Sir Joseph Hawley, but the grange as a stately home dates back long before that.

Writing in 1798, historian Edward Hasted records The Grange as "a seat in this parish" (of Leybourne), with origins dating back to the reign of queen Elizabeth (1533 -1603).

He says the grange was rebuilt in first half of the 18th Century before being bought by James Hawley in 1776, and passed down through the generations of Hawleys.

Sir Joseph was the most famous of them, a renowned thoroughbred race horse owner, who won the Epsom Derby four times with his horses Teddington (1851), Beadsman (1858), Musjid (1859) and Blue Gown (1868).

He used chunks of his winning to improve the estate, building the stone Leybourne Pump in Pump Close, which has the inscription:

“Drink, weary pilgrim,

Drink and Pray,

For Living Waters,

They Only They,

Can Satisfy Earths Sons and Daughters."