Fantasy has become big, big news in publishing. Ask any film producer, agent or talent-scout: all they are interested in acquiring at present are the rights to children’s or “cross-over” novels featuring magic. Anne-Louise Fisher, who seeks out new authors here for ten major publishers abroad, including Doubleday US says:

“Publishers are like sheep: they see something successful and they all want to copy it. Perhaps, though, a lot of fiction has been too realistic and gritty. We all need something to escape to and Harry Potter reminded us that it is OK to read escapist literature.”

Those who groan at the idea of Middle-Earth triumphant will not be pleased. Yet as Samuel Johnson famously observed, “Babies do not want to hear about babies; they like to be told of giants and castles, and of somewhat which can stretch and stimulate their little minds.” After an explosion of great fantasy writing in the 1960s and 70s by CS Lewis, Joan Aiken and Alan Garner, new children’s writers such as Judy Blume confronted “issues” of sexuality, divorce and bullying in a realistic framework. Fantasy was relegated to the despised Goosebump horror series, and sales of children’s fiction plunged. That changed, partly as a result of Philip Pullman. “He made the most wonderful acceptance speech when he won the Carnegie Prize for Northern Lights about the importance of story,” says the publisher Jane Nissen, who reprints children’s classics. “I think that was the turning-point. It reminded publishers and librarians what fantasy fiction was all about. I think Harry Potter would never have had the success it had without that speech.”

Now, according to the Ultimate Children’s Book Guide for 8-11 year olds, published by A&C Black next month, children’s best-loved fiction is brimming with wizards, witches, talking beasts and the supernatural. Compiled from 1500 children in 200 schools nationwide, it shows that in children’s preference, fantasy outweighs realism by about 3:1.

“I think the rise and rise of fantasy in children’s fiction is a return, not a new thing,” says Philip Pullman’s publisher, David Fickling. “The modernist and post-modernist rejection of narrative is fundamentally empty. What you’re seeing is a reassertion of the fundamental human delight and desire for story. “

Formerly an editor at Scholastic before founding his own imprint at Random Century, Fickling observes that “in the late 1980s children’s fiction was under fire both from marketing departments and the advent of CD Roms and computer games. We kept being told ‘Children don’t read, they have the attention-span of a gnat, they don’t like long books.’ What Pullman and Rowling proved was that length didn’t matter, what mattered was how much a child wanted to read.”

If many adults find they can’t take novels featuring wizards and quests seriously as a literary form, they should perhaps consider its origins. Lord of the Rings has its roots not only in Tolkien’s love of Beowulf and Norse sagas, but in Malory’s Morte d’Arthur, Spenser’s The Faerie Queene, Shakespeare’s A Midsummer Night’s Dream and The Tempest. Pullman plays with Paradise Lost; Almond with Blake, Rowling with a host of school-stories from Waugh to Wodehouse. Not that a distinguished ancestry is necessarily any guarantee of quality. Christopher Paolini’s much-hyped Eragon (Doubleday £12.99), which is a best-seller in his native America will please only the most dull-witted of Tolkien’s admirers, being wholly derivative, poorly-written and mawkish. It joins other bad fantasy novels by authors such as Robert Jordan and David Gemmell in lacking the moral complexities and thrill of strangeness that good fantasy can bring. John Dickinnson’s The Cup of the World (David Fickling £12.99) is, by contrast, brimful of talent. It concerns a young girl, Phaedra, who gives her heart to a man she has only met in a dream. The consequences, while they involve the use of magic, are fraught with betrayal, politics and passion, and the reader is thrilled and surprised by characters whose adventures and choices cause them to grow or shrink as realistic human beings.

“I wouldn’t make the division between realism and fantasy,” Fickling says. “All good children’s novels are stories to the tips of their fingers, emotionally involving. Lots of bad fantasy does the good stuff down. Fantasy, whether it’s set in another world or another time says, Let’s go far away, and then looks at truths that need to be said. Fantasy can never be “good for you”, but it’s not escapist, it is a form of play. Enid Blyton’s The Far-Away Tree is bad in an adult sense, in that it’s not a coherent, well-worked-out world, but in a play sense it’s good. Adults are so keen to get their children onto the good stuff, the classics, that they forget reading is a form of play, the idling engine of the brain.”

Garth Nix is trenchant that “the things that make a good or bad fantasy novel are usually the same things that make any other kind of novel good or bad. That is, if the novel doesn’t work as a human story without the fantasy, the fantasy elements won’t fix it. Bad fantasy novels have the typical failings of novels that don’t work – like lacklustre stories, mechanical plots, stereotypical characters and pedestrian prose. They compound this by recycling the plots, characters magic and backgrounds of Tolkien without re-inventing them in new and interesting ways. A towering fantasy novel must have a bedrock of the believable beneath it.”

Which brings one to ask, what of the non-fantastical children’s novel? Although Jacqueline Wilson’s tales of the disadvantaged young have a huge following, what is most obviously lacking is the kind of story about ordinary children pioneered by E. Nesbit’s The Treasure-Seekers, Frances Hodgson Burnett’s The Secret Garden or Arthur Ransome’s Swallows & Amazons. There are just as many outstanding realist children’s writers emerging, from Francesca Simon’s tales of sibling rivalry in the Horrid Henry series (Orion £4.99), to the grave and beautifully written Deep Secrets by Berlie Doherty ( Puffin £10.99), or Mark Haddon’s The Curious Incident of the Dog in the Night-time (Fickling £12.99). One kind of fiction read to the exclusion of the rest is as bad for a developing imagination and sensibility as one kind of food, yet it’s clear that the realistic children’s novel gets far less critical attention – not to mention lucrative film deals.

The magic quest seems, as the philosopher Anthony Grayling points out, to be almost hard-wired into our consciousness. “Both quest narrative and fantasy are staple needs of the human psyche, and are as old as any form of civilisation -look at the history of the Odyssey, the Aenied, the various Grail stories – they reprise in all forms of art capable of narrative. Fantasy is an excellent form for focussing light on aspects of human experience without the prosaics of reality getting in the way.”

Fantasy lends itself to Swiftian political satire, as Jonathan Stroud’s The Amulet of Samarkand showed, but it is its heroes and ideals which seem to move us most. Perhaps in these anxious times we yearn for an Aragorn, an Aslan or a Lyra Bellacqua to stir us with their courage and wisdom because it seems so drastically lacking in reality. Cornelia Funke’s remarkable new novel Inkheart (Chicken House £12.99) embodied this yearning in imagining what would happen if characters could step out of the page – although in her story, what emerges are villains, not heroes.

The children’s author Eva Ibbotson, who won the Smarties Gold Prize two years ago for Journey to the River Sea says, “Fantasy presumes solutions (miracles) without working for them. The imagination, however, knows that the solution must be worked for by the protagonist.” In great fiction, the pain of experience is not avoided, and a parallel world is not something you can escape into when this one becomes too difficult, though it returns you to yourself feeling refreshed and encouraged. At the start of a new year, with doubts and fears pressing in on us in the dark days still ahead, this is worth remembering, whatever your age.

The Times, January 2004