Until five years ago, children's publishing lived in a ghetto. Advances were low, marketing budgets non-existent and ambitious editors got out of it as fast as possible. No longer.

"It used to be the case that if you worked in children's publishing you were assumed to have a mental age of twelve," says Marion Lloyd, Associate Publisher of Children's Fiction at Macmillan. "Now there's tremendous interest in the best children's fiction. It has the respect of serious literary commentators and of marketing departments. The perception of its quality has changed, thanks to the Harry Potter phenomenon, and to the success of Philip Pullman who became the first children's author to win the Whitbread this year."

The buzz around the best children's novels has been growing ever since 1997, the year the second Harry Potter caused a 25% jump in Bloomsbury's share value. Now, with stratospheric sales and profits, the vogue for children's fiction is reaping commercial rewards for its authors. This month sees the arrival of yet another star. Twelve years ago, Georgia Byng was a struggling young actress with a small daughter, who turned to writing picture books as a way of expressing her love of drama. Her novel, Molly Moon's Incredible Book of Hypnotism (Macmillan), published this month, has reaped her $1 million, and been bought by the same Warner Bros team that made Harry Potter and the Philosopher's Stone.

The tale of Molly, an unattractive 10-year old abandoned at an orphanage, whose life is tranformed by discovering The Book of Hypnotism, has all the comedy and tension of an instant classic. Molly turns her life around by hypnotising first a dog, then her bullying headmistress, and flies to New York in search of her best friend Rocky. Ambitious and ebullient, she hypnotises her way into a Broadway hit, before confronting the moral implications of her astounding power over adults.

"She's a tremendously original, funny character, who made my hair stand up," says Marion Lloyd. "I'm constantly looking for that magical combination of story-telling, quality writing and strong characters, but the one thing I don't want is me-too wannabees. Agents have learnt not to mention the words Harry Potter when submitting a new author to publishers. We don't want to hear those words."

Yet it is the Harry Potter phenomenon that is single-handedly driving the price of major children's novels, as most privately admit. When JK Rowling was originally signed up to write the first Harry Potter books she received the standard advance of £2,500. Now, following her world-wide fame and fortune, competition to find the next big names in children's fiction means that its authors are increasingly receiving advances that are around ten times that amount. Publishers such as Random Century who turned down Molly Moon are watching its sales nervously: with Macmillan spending an estimated £50,000 on poster and book-shop advertising, its commercial success is guaranteed. Yet what will really make or break the novel's success is word-of-mouth. Children of 6-12 are remarkably sophisticated consumers but they tend not to be fooled, as adults are, by mere fashion and hype.

According to a recent Bookseller report on the children's market, the volume of sales actually fell from 109 million in 2000 to 104 in 2001, with the biggest decrease in books for children of 0-6. It is simply not true that, as some claim, the Harry Potter phenomenon has helped nobody but Harry Potter, Series fiction, such as the Goosebumps horror stories by RL Stine have fallen off, and what is replaced them are the more literary, serious novels by authors children recognise and value. Philip Pullman recently became the first children's author to win the Whitbread Book of the Year prize for The Amber Spyglass, yet it was children's desperation to continue reading quality fiction that lifted sales of Philip Pullman's outstanding trilogy, His Dark Materials into the American and UK best-seller lists. Anyone who read his other novels may well wonder why they took so long to succeed (he has said "they were not so much published as concealed by Penguin") but there is no doubt that Rowling's success has drawn attention to what is effectively a third golden age for children's books.

Georgina Byng is the sister of the Canongate publisher Jamie Byng and daughter of the Earl of Strafford, yet her success is, like that of three other new superstars of children's writing, the product of both inspiration and perspiration. She got the idea for Molly Moon "in three swift steps" when wondering whether she could hypnotise her mother's dog, but in fact she has been writing for younger children for a decade. This is one of the essential differences between children's publishers and adult ones. Where adult editors are prepared to pay £100,000 for an unknown, untried author, children's editors, in Marion Lloyd's words, "don't expect them to write their best work at the beginning of their career. We've got to allow writers the odd weaker book, and nurture them through several before the one that makes the huge impact. We're wary of giving a brand-new author stacks of money, because if the book doesn't succeed, their career is over."

Eva Ibbotson, another Macmillan author, is a case in point. The author of eight children's novels, she won the Smarties Gold Prize this year for Journey to the River Sea, which is also shortlisted for the Carnegie Medal, the Booker of the children's world, which she is hotly tipped to win. Her novels, though always well-received, were not well-known until two years ago. They now sell 200,000 copies a title. Partly this is due to their being rejacketed, but the real impulse is that children themselves wrote in to the Internet bookstore Amazon to say the words publishers have been attempting in vain to attach to other authors - that "if you love Harry Potter, you'll love this." And they did, particularly with regard to The Secret of Platform 13, published three years before Harry Potter and the Philosopher's Stone. The similarities between the two are astounding: Ibbotson's novel describes a door at Platform 13 of King's Cross opening onto a magical world of wizards, ghosts and giants. The hero is a young boy who belongs to this world but who is ignorant of his true nature, and bullied by the grotesquely rich and nasty Trottle family, and made to sleep in the servants' quarters until magic comes to rescue him. Ibbotson would seem to have at least as good a case for claiming plagiarism as the American author currently suing JK Rowling, but unlike her, Ibbotson says she would "like to shake her by the hand. I think we all borrow from each other as writers."

Eva Ibbotson's novels feature witches and ghosts not as scarily powerful but as an endangered species; her thrilling adventures brim with the wit and wonder, and yet it has taken her twenty years to break through. "Each time I finish a book, I honestly expect to be taken away to a lunatic asylum, they're so mad," she confesses, but it's this craziness that makes her irresistible. Three of her novels - Which Witch, The Haunting of Hiram and Dial a Ghost - are now being developed as feature films, the latter by the same Warner Bros. producer who made Harry Potter. Her new book, Journey to the River Sea (Mammoth) is about an orphan who travels to the "green hell" Edwardian Brazil and unexpectedly discovers it to be a paradise. She wrote it when grieving for the death of her husband, and its deep feeling for loneliness make it as remarkable a work of fiction as The Secret Garden. It is, she says, "a plea for children to live big lives, to be allowed to discover their destiny."

This perhaps, is part of the appeal that children's fiction holds for adults as well. Where the literary establishment turned its back on the art of narrative after EM Forster's languid remark that "Yes - oh dear yes - the novel tells a story", a strong plot and pellucid style is essential to children's fiction. Its powerful hold on us has recently been examined in Francis Spufford's best-selling memoir, The Child that Books Built. (Faber £12.99) He describes the second golden age of children's books that occurred in the 1960s as deriving from "a kind of temporary cultural consensus: a consensus both about what children were, and about where we were in history." That consensus has now largely vanished and a new kind of novel has emerged. What children want, and what they are getting isn't the literature of consolation but one that addresses their deepest fears and longings through the medium of an adventure story. As Georgia Byng says, "Bruno Bettelheim pointed out in The Uses of Enchantment that fairy-tales used to be for adults as well as children. I think modern children's novels are providing some of that function. People desperately need the cathartic release that stories can bring. If you hit a spot for children, you hit it for everybody."

This sophistication is reflected in the spin that new children's literature puts on our expectations. Just as Harry Potter posited a magical world existing in parallel with our own, with its own regulations, and Philip Pullman described a universe in which the Devil is on the side of wisdom and God is the force for evil, so the best new authors play with convention. Artemis Fowl is described by its author, Eoin Colfer, as "Die Hard with Fairies". Hugely thrilling and very funny, its premise is that fairies exist underground as a secret race, as technologically advanced as we are. Artemis Fowl, a 12-year-old boy genius and criminal mastermind captures and decodes their hand-book, and so discovers how he can (as he thinks) extort the legendary fairy gold as ransom. Pitted against him is Holly Flint, of the LEPRecon police, her irascible boss and an inventor centaur, Foley. Colfer's publisher, Penguin, launched a microsite featuring a multi-player interactive computer game in which players had to collect thousands of trolls wreaking havoc on the environment, a first in marketing children's literature, and are believed to have spent over £50,000 marketing the hardback and paperback. The first Artemis Fowl novel sold 150,000 in hardback, (230,000 in the US); the second, Artemis Fowl and the Arctic Incident, is due out next month. Film rights have been bought by Miramax, and Colfer too has hit the jackpot.

A 35-year-old Irish primary school teacher, Colfer, like Ibbotson and Byng, had already written several children's books without notable success, and like them is stunned by the change in his fortunes.

"Films definitely had an impact on my writing style. I am a huge movie fan, and the action genre is one of my favourites," he says. "I realised that very few action movies are specifically for kids, even though kids love them. So I decided to fill the vacuum. I grew up reading Irish myths and legends, so I had great fun putting a spin on them. I knew there was no point in regurgitating other people's stories, so instead I gave them a technological makeover."
It was this wit and freshness of approach that made the first Artemis Fowl novel stand out, according the Colfer's editor, Sarah Hughes at Penguin.

"We recognised straight away that this was something different, and sophisticated, which appeals also to adults. What we're starting to realise, following the success of JK Rowling and Philip Pullman, is that children's fiction has always appealed to adults, although we're not overtly trying to capture this market. The sales expectations in the children's market have risen."

Such is certainly the case with William Nicholson, whose Wind on Fire trilogy reaches its climax with Firesong, published by Egmont this month. Nicholson, award-winning screenwriter of Shadowlands and Gladiator, turned to writing a children's novel as a kind of experiment, having tried and failed in his youth to write for adults. His values are the highest, and his approach closer to Philip Pullman's than JK Rowling's. The Wind on Fire trilogy address subjects such as the link between violence and art, the need for willing self-sacrifice and the way love can prove stronger than evil. All of these are wrapped up in linked adventures whose inventiveness and depth of feeling never flags. The Hath family live in a city dominated, even in infancy by exam success: passing or failing affects the colour of clothes, the quality of accommodation and work. This satire on our cultural values has appealed enormously to children, who love the way Kestrel yells "I won't strive harder, I won't reach higher, I won't seek to make tomorrow better than today!" When the twins Kestrel and Bowman defy this, they are exiled into the desert where they acquire supernatural powers that help to save their people, and eventually bring them to a new homeland. The first novel, The Wind Singer, won the Smarties Gold prize in 2000, and sold 100,000 copies so far; its sequel, Slaves of the Mastery, is doing even better in paperback, and the third, Firesong, is published by Egmont this month to huge acclaim.

"My career is studded with failure as well, but everybody forgets that," says Nicholson, 53, dryly. His mother, a Catholic, was taught English by CS Lewis, and despised his books as "drippy, Protty stuff", but he became increasingly interested in children's fiction after writing Shadowlands, the BBC play about Lewis's private life, that launched his career as an Oscar-winning scriptwriter. Now, with three children of his own, William Nicholson like CS Lewis, is examining the deep questions of spirituality and identity. What is particularly good about his novels is that for once they do not exalt the lone child-hero, but show how a family, and a people can have adventures together that change and deepen their innate qualities.

"The glory of writing for children is also the trap: anything goes, but to make it believable you have to rein it back," he says.

Unlike Byng, Ibbotson and Colfer, William Nicholson has so far resisted film offers, although his trilogy, which has been described as "Gladiator for kids", is consummately filmable. He wants children to discover them as books first. Ten years ago, when parents and teachers were in despair as to whether the new generation would ever learn to love books, and children's authors were the poor relations of the adult world, this would have seemed impossible. Now, in the new golden age, it seems that inspiration and perseverance have at last won through.