'We will lose the centre if we don't get rid of Gordon'

In a clear signal of senior figures' determination to oust Mr Brown, the minister told the Evening Standard that only a third of the Cabinet were convinced that the Prime Minister should lead the party to polling day.

Speculation about Mr Miliband's leadership ambitions continued apace today after he let slip that he had tried to avoid a "Michael Heseltine moment" in his conference speech yesterday.

The Foreign Secretary had tried to heap praise on Mr Brown but he suffered acute embarrassment when he was later overheard in a lift confiding his worries about being compared to the former Tory minister who plotted to oust Margaret Thatcher.

Despite that row, Mr Miliband's allies are making clear that the work of Neil Kinnock, John Smith and Tony Blair in dragging Labour to the centre is at risk if ministers allow Mr Brown to stay in post to 2010.

A Cabinet minister said that if ministers failed to act decisively, they risked activists and unions using the luxury of opposition to elect a Left-wing leader such as John Cruddas.

But if Mr Brown stood down before the election, the discipline of being in government would ensure that the party opted for a more "sensible" figure.

"We can change in government or we can change in opposition. It is much better to change in government. If we get a new leader in opposition, it is much more likely that we could end up with the party choosing someone from the Left," the minister said.

The Cabinet is split into three camps, with one group ready to back Mr Brown to the hilt, another ready to give him until next summer and a third set to act sooner.

Rebel MP Graham Stringer today warned that "it's a question of when, not if " for the party to ask the Prime Minister to step aside. "I think it would be surprising if there hasn't been a challenge by next summer," he told BBC Radio 4's Today programme.

One Cabinet supporter of Mr Miliband even suggested that there need not be a "trigger" - such as a by-election defeat in Glenrothes - for ministers to tell Mr Brown he should go.

"If we wait until next summer, that would be seen as purely selfserving, with MPs only interested in saving their own seats because of the scale of the defeat in the Euro elections. If things are bad now, why do we have to wait until next June to find out that they could be even worse?" the minister said.

The minister added that although Mr Miliband's speech yesterday received a lukewarm reception, it was "an advance" on previous speeches and showed he was making progress.

Today, Mr Miliband stressed privately that reports of his " Heseltine" remark had been "a fuss about nothing".

The gaffe emerged when he was overheard in a hotel lift suggesting that he toned down his conference speech in order to avoid outshining Mr Brown and appearing as ambitious as Mr Heseltine.

The BBC reported that the Foreign Secretary was heard telling an adviser: "I couldn't have gone any further. It would have been a Heseltine moment."

The comment apparently refers to the idea that former premier Margaret Thatcher's demise was hastened by brilliant performances by her most prominent rival for the leadership, Mr Heseltine.