Should we be mindful of mindfulness?

At just after 6.15pm in a brightly lit conference room in Oxford, 22 grown men and women are lying on the floor trying hard to focus on their left knee. From across the room a lilting, calm voice has already invited the group to explore their feet and ankles with "gentle curiosity" and is heading up through the body. "When your mind wanders, gently and kindly escort your attention back to your left knee," she tells us.

It's not easy. Lying on a blue plastic mat, dressed in an uncomfortable work suit and open necked shirt, with an air conditioning unit grumbling close by, my mind isn't that keen on being escorted anywhere. Instead, it's wondering if anyone else is struggling to focus. It's wondering how I will be able to recall this experience in enough detail to jot it down afterwards. It's thinking about the Viking occupation of Grimsby, how geckos walk up walls and, most disturbing of all, whether Eric Pickles would float.

According to Marie Johansson, the leader of our session, that's all fine. Even the Eric Pickles bit. This meditation isn't about relaxing, emptying the mind or filling the head with peaceful thoughts. "The intention is to be aware of physical sensations of the body and also simply to notice what the mind does," she says afterwards. "The mind wanders and it entertains itself with all sorts of things. All we are required to do is notice these thoughts. We are not suppressing it or emptying the mind, or making the thoughts go away."

For the growing army of people who have taken part in mindfulness training, these reflective rituals of the 40-minute "body scan" will be all too familiar. The scan plays a key part in helping people to become more mindful – to live more in the moment and to spend less time anticipating stresses, or reliving disasters from the past.

Mindfulness is everywhere at the moment. If you don't know someone who has done a course, downloaded an app or read a book, you will soon. Based on centuries-old Buddhist meditation practices and breathing exercises, it is prescribed to thousands of patients on the NHS each year to help prevent anxiety, depression and stress. Even more pay for private classes believing that they improve the quality of their lives and relationships. And over a million people looking for mindfulness on-the-go have downloaded apps such as Headspace. The mindfulness industry is vast, and growing weekly.

So can an approach so deeply rooted in eastern spiritualism, and which at times comes close to sounding like new age waffle, really work?

Professor Mark Williams thinks so. One of the pioneers of mindfulness-based cognitive therapy in the UK, Williams is a recently retired professor of clinical psychology at Oxford University. With colleagues from Cambridge and Toronto, he developed an eight-week course that is being taught across the world. The course is based on a similar programme developed in the late 1970s by US medical professor Jon Kabat-Zinn to cope with stress.

"It's a preventative treatment – that's what makes it different," says Williams. "People usually seek treatment when they're depressed or anxious, and cognitive therapy is one of the major success stories in treatment. But cognitive therapy is used when people are ill. What we wanted to do was extend this to teach people skills to stay well that they could use before depression threatens."

The idea behind mindfulness is straightforward. Kabat-Zinn calls it "paying attention on purpose, moment by moment, without judging". Practitioners argue that the brain's habit of reliving past stresses and worrying about potential future problems can become an obstacle to mental health.

Mindfulness encourages people to get those critical thoughts about the past and future into perspective so they no longer dominate. Instead, people are given tools to help them become anchored more in the present, and to focus more on the sensations of the world from moment to moment. That is achieved through meditation techniques such as the body scan – a practice where participants are "invited" to focus on the sensations of their own body. Thoughts that pop up during the exercise are acknowledged and "observed kindly" before the mind is refocused back to the sensations of the body.

Other practices focus on breathing and on linking stresses and mental distractions to physical sensations in the body such as tense shoulders, clenched hands or shallow breath. It sounds simple, but it's not. It takes hard work and lots of practice.

"A good example of how it can work is when you're kept awake at night thinking," says Williams. "You toss and turn and you get angry because you can't sleep. The anger doesn't help, but you can't seem to stop it. Mindfulness isn't about suppressing those thoughts, but about enabling you to stand back and observe them as if they were clouds going past in the sky. You see them and you cultivate a sense of kindness towards them."

The best documented benefits are preventing relapses of depression, where it helps people entering the downwards spiral to notice when self-critical thoughts are beginning to arise and to help prevent those negative thoughts from escalating. And it draws attention to small pleasures around people, helping to lift mood.

For those suspicious of therapy and mysticism it can sound vague and woolly. But Williams insists this is a practical, clinically proven approach. And while its origins are in Buddhism, it is completely secular.

The clinical evidence for mindfulness as a way to prevent depression, stress and anxiety appears to be sound. A review of the eight-week course was published in 2011 in Clinical Psychology Review by Jacob Piet and Esben Hougaard of Aarhus University, Denmark.

After looking at six clinical trials involving 593 people, they concluded that mindfulness-based cognitive therapy reduced the risk of relapse for patients with at least three previous incidents of depression by 43% compared with people who received treatment as usual. However, there was no significant benefit for people with fewer than three major incidents.

A review of the research in Clinical Psychology Review last month by researchers at the University of Montreal looked at 209 studies covering 12,145 people. It concluded that mindfulness was an effective treatment for a variety of psychological problems, "and is especially effective for reducing anxiety, depression and stress". Other studies have shown that it is effective for preventing anxiety and mood disorders and may be good for other psychiatric conditions including bipolar disorder.

These are the best of the recent studies – but the published evidence goes back further. In 2004, Nice – the NHS's rationing body – was convinced enough of the benefits that it ruled mindfulness-based cognitive therapy was cost effective. Its most recent advice, updated in 2007, is that it can be prescribed for people with three or more episodes of depression. There is also growing evidence that it's effective for chronic long-term health conditions such as ME.

Victoria Jackson, an Oxford-based publisher, was recommended a course by her GP as a way to ease symptoms of chronic fatigue syndrome.

"Taking a mindfulness course was something I thought long and hard about. I was concerned about the Buddhist origins of mindfulness. I'm an atheist, so the idea of engaging with anything mystical worried me. I'm also a very pragmatic person and anything with the word 'therapy' attached to it makes me uneasy. Moreover, there's been a lot of controversy around the way ME has been regarded as a psychiatric disorder, and I felt concerned that following a mindfulness course would seem to endorse this view about the nature of the illness."

After an eight-week course at the Oxford Centre for Mindfulness in 2012, she noticed an improvement in her health. She now spends 10 to 20 minutes on formal meditation every other day – and snatches informal moments of mindfulness when she can – and says it is something she is consciously trying to apply to her life.

"There's a popular notion that it's a panacea – it's not," she says. "Practising it in a formal or informal way is a constant challenge. The brain doesn't like being still and being focused on something as mundane as your breath. The challenge is to observe your mind wandering, not criticise, and just lead it back."

The changes brought about by mindfulness are difficult for Jackson to quantify.

"It has given me lots more options in my life, but only when I wake up to them. There's a sense that we drift through our lives. Mindfulness gives you an awareness and therefore a choice as often as you choose to bring it to mind – that ability to step out of the situation and evaluate things and make a conscious choice: do I want to pick up my smartphone and distract myself, or choose to see the sunset and notice how it makes me feel?

"But it has worked. Exhaustion can have a cyclical pattern in ME. A lot of people overdo things and become exhausted. Then when they've rested and are feeling less tired, they overdo things again. That's a pattern I fell into. There's been a huge benefit in being more aware of that pattern and the way I feel, and making a conscious choice of how to react and look after myself."

Marie Johansson, who teaches mindfulness at Oxford's Mindfulness Centre, says the approach can also benefit the healthy.

"People often say they notice how much of life passes them by," she says. "Suddenly they are noticing things in nature, in their friendships and neighbours – perhaps they have different relationships with their children and families. They are more present in what they do and they get a sense of appreciating more fully the life they are living."

But there is a danger that the benefits of mindfulness are being overstated, without the clinical data to support them. There are books on applying it to business leadership, to parenting and to weight loss. There are mindfulness exercises for children and guides on living with pain. There is no shortage of courses, books or even smartphone apps being offered to an enthusiastic public – and sometimes little way for people to tell whether they are authentic, quality-controlled and reliable – or on the fringes of new age crankism. Even the experts in mental health can occasionally overstep the mark.

Oxford University and the Mental Health Foundation (MHF) have worked on a 10-session online course, available for £60. On its website, the foundation claims that "the effectiveness of the online course is the subject of a highly significant research paper by Oxford University published in BMJ Open". It adds: "The reported average outcomes for completers of the course show participants enjoying reductions of 58% in anxiety, 57% in depression and 40% in stress."

That is true, but only to a point. The MHF website glosses over an important caveat in the BMJ Open paper. The authors, who include Prof Williams, point out in the paper that the study had no control group, meaning there was nothing to compare the course with. More research is needed.

Williams is acutely aware of the dangers of overclaiming.

"A lot of people think it will cure everything. But we know there is nothing that cures everything. There is some interesting work in psychosis, bipolar disorder and schizophrenia but it's in its early days. There's a lot of hype around mindfulness and we need to be cautious because it doesn't serve our science or patients well if we're overenthusiastic. We have to make sure the science catches up with the enthusiasm."