Prince Harry and six other public figures face paying up to £34.5m to the publisher of the Daily Mail after a High Court judge imposed an exceptional order for legal costs over their failed privacy case.

The seven claimants, also including Sir Elton John and Liz Hurley, must pay an initial £9.54m by Friday of next week to Associated Newspapers Limited (ANL).

The judge ruled that the case was a rare one in which the claimants should have to pay far in excess of the costs that had been agreed before trial, criticising how they presented their claims as “unreasonable to a high degree”.

On 7 July, Judge Mr Justice Nicklin dismissed the claims of unlawful information-gathering, which ANL denied.

The Duke of Sussex and the other six claimants have until 2 October to attempt to launch an appeal.

Those other claimants are Sir Elton John and his husband David Furnish, former Liberal Democrat deputy leader Sir Simon Hughes, racial justice campaigner Baroness Lawrence, and actresses Sadie Frost and Liz Hurley,

They had insurance to cover up to £16.2m of Association Newspapers’ costs in the event of losing the case – but Friday’s ruling means the newspaper group could now try to recover up to the £34.5m that it said it had to spend to defend itself. This means the duke and others could face a bill of some £18m.

Justice Nicklin said: “I regard a claim for costs in excess of £34 million as, on its face, excessive and as giving rise to real concerns as to whether all of the costs now claimed by Associated were reasonably incurred and are reasonable in amount.”

But he added he had “decided not to impose a ceiling on Associated’s recoverable costs”, as it would “be too broad-brushed, would risk unfairness, and would be vulnerable to the charge that it was arbitrary”.

The judge also ruled that ANL’s costs should be determined on an indemnity basis, which is more favourable to the publisher in terms of the amount it can recover, aside from where orders had already been made.

He added it was “striking” that “not a single allegation of serious wrongdoing was voluntarily withdrawn” by the group of seven claimants.

He said: “That matters. Serious allegations of criminality or grave impropriety cannot properly be left hanging over individuals in that way.

“If a party no longer advances such an allegation, or no longer has a proper evidential basis for advancing it, the allegation should be withdrawn clearly and in terms.

“The claimants’ failure to do so was not ordinary forensic judgment. It was not a proper or disciplined way to deal with serious allegations. It was unreasonable to a high degree.”

In a statement released on Friday, the publisher called the ruling “a devastating critique of an attempt to destroy a newspaper and the reputations of its journalists, editors and executives”.

Friday’s decision comes just days after Prince Harry and Meghan announced their plan to return to the UK, with their children Prince Archie, seven, and Princess Lilibet, five, enrolled to start school in September.

During the trial, dozens of people gave evidence including Prince Harry and the other claimants. The Duke of Sussex flew in from the US to attend court in person.

Many current and former Associated Newspapers journalists and executives gave evidence denying illegal activity.

Sir Simon Hughes told the BBC he was “disappointed” by the judgement.

“I am now taking the time necessary to consider both the substantive and the costs judgment and whether to pursue appeals against decisions in either judgment or both,” he added.

In a joint statement following the judge’s ruling last month, Prince Harry and Baroness Lawrence, who became a public figure after her son Stephen was murdered in 1993, said: “We came to court seeking justice and accountability. But we have received neither.”

“It is a complete and obvious whitewash, but sadly not altogether unexpected”, the pair added.

In a civil case such as this, the legal convention is that the losing party must reimburse “reasonable” legal fees incurred by the other side. They also have to settle their own legal costs. Normally these are agreed and capped ahead of trial.

ANL had asked for more than £9.9m as an interim payment ahead of future hearings to decide its legal costs in depth. Only after it won did it become clear it has spent far more than an initial agreed cap of £16m.

Colin Campbell, a former specialist costs judge now with law firm Kain Knight, said the size of that overspend meant the battle was not over quite over yet.

“The order is a significant win for Associated Newspapers,” he said. “It means the publisher will no longer be constrained by the costs budget previously approved by Mr Justice Nicklin.

“Notably, [it] came despite the judge expressly finding that none of the claimants had been dishonest in their evidence.

“However, this is far from game, set and match for Associated Newspapers. The judge was scathing about their projected £34.5m costs bill, branding it exceptionally high and excessive and describing it as a matter of real concern.

“That criticism could prove highly significant when the publisher ultimately seeks to recover its costs.”