Iain Duncan Smith says he would "lay money" on keeping his job as Conservative Party leader despite reports of plotting against him.

He said he "couldn't care less" about speculation that he might face a leadership challenge, insisting that he had the "full confidence" of his party.

I have no interest in one or two people who may have a different agenda

Iain Duncan Smith

He dismissed the dissenters who were briefing against him as "a few people who ... want things changed".

But former leadership contender David Davis, who was sacked as party chairman, admitted that attacks by his colleagues were "doing harm" to the Conservatives.

Making life difficult

"What concerns me is that if we carry on doing this, and undermining leaders, we are never going to get anywhere," he told BBC's On the Record.

"The people doing the briefing are doing harm to the party - there's no doubt about that. They are making Iain's job incredibly difficult."

Mr Davis, who on Thursday said he would "never" stand against his "friend" Mr Duncan Smith, insisted: "I positively don't want him to stand down. I don't want these attacks on him.

"As certain as I can be, subject to strikes by lightning, I am sure he will lead us into the next election."

Earlier, former leadership challenger Ken Clarke said the speculation was doing the party no good and the Tories should concentrate on developing policies.

Leadership challenge?

Michael Portillo, defeated in the last leadership race, has confirmed there have been "mutterings" among MPs against Mr Duncan Smith.

But as Mr Portillo urged the Tory leader to strike back against any plotters, The Mail on Sunday reported that he wanted 100 Tory MPs to ask him to take over the party before he would consider mounting a challenge against Mr Duncan Smith.

Tory former chairman Lord Tebbit, however, insisted that neither Mr Portillo nor Mr Clarke "is involved in any plot against Iain Duncan Smith".

David Davis: Will 'never' challenge Duncan Smith

"They would not only not do it, but they would be mad to do it," he told BBC's Breakfast with Frost.

But Mr Duncan Smith sought to brush aside talk of leadership challenges by arguing that the "inner workings" of a political party were of little interest to British people.

'Full confidence'

"The reality is there are a few people who have decided that they want things changed or different, but that's not going to happen," he told ITV's Jonathan Dimbleby.

"I just don't think this is an issue the British people are the slightest bit interested in."

Mr Duncan Smith, who has just completed a three-week national tour, said: "The one thing that does not come in any conversation with me is 'oh, tell us all about the ins and outs of the Conservative Party', that's the reality."

He insisted: "My party has full confidence in me.

"I was elected by the membership of this party, not just by the members of Parliament, and I was elected on a basis to go ahead and take this party back to electability.

"I have no interest in one or two people who may have a different agenda.

"The reality is, there will always be some. My answer is, I have to lead the whole of this party forward.

"I am going to do it. There's no question the route we are on is the right route.

Hague and Portillo have offered supportive words

"We have got to get the policies right and we have got to explain to the British people that their concerns are our concerns - their priorities are our priorities."

Asked if a vote of no confidence would be the only instance in which he would stand down, Mr Duncan Smith replied: "I'd lay money on this - none of this is going to happen.

"We are here for the long term which is to get this party back into government, not because I want to be in government, but because I actually think the present government is failing."

Next May's local elections are seen as a key test for Mr Duncan Smith but he warned people not to expect too many gains.