US regulators today launched a full-scale probe into their failings over the Madoff scandal, which is expected to investigate the relationship between one of their top former officials and the niece of Bernard Madoff.

In a devastating admission of failure by his organisation, Securities and Exchange Commission chairman Christopher Cox admitted his watchdog has had "credible and specific allegations" going back nearly a decade against Bernard Madoff, accused of defrauding investors of $50 billion (£33 billion).

The relationship between the niece, Shana Madoff, who was a compliance lawyer at the allegedly fraudulent firm, and Eric Swanson, who worked at the SEC as a senior inspections official, is likely to be a key part of the investigation. Swanson was an assistant director of compliance and examinations at the SEC for 10 years before leaving in 2006. The couple are now married.

Madoff himself was a big and trusted name on Wall Street, who has advised the SEC on regulatory issues.

Speaking of the many red flags that had emerged over Madoff in recent years, Cox said: "I am gravely concerned by the apparent multiple failures over at least a decade to thoroughly investigate these allegations or at any point to seek formal authority to pursue them."

Allegations dating back until at least 1999 "were repeatedly brought to the attention of SEC staff, but were never recommended to the commission for action", he said. He called for an investigation of his agency, saying the way it has handled the Madoff pyramid scheme is "deeply troubling" and asked the agency's inspector general to lead a "full and immediate review" of the past allegations against Madoff.

Madoff, an investment manager and former chairman of Nasdaq, has confessed to operating a $50 billion so-called Ponzi scheme.

Madoff, 70, was arrested on 11 December after he told his sons that Bernard L Madoff Investment Securities was a "giant Ponzi scheme", the SEC said. Clients facing losses range from pension funds to hedge funds and major global banks such as HSBC.

The Republican SEC chairman said the probe by inspector general H. David Kotz will look into the agency's internal policies and whether they should be improved. The investigation will include a review of all SEC contact with "the Madoff family and firm", Cox said.

Bats, the Wall Street firm where Swanson now works, defended him. "Eric Swanson worked at the SEC for 10 years and did not participate in any inquiry of Bernard Madoff Securities or its affiliates while involved in a relationship with Shana," a spokesman said. "Throughout his career, Eric has displayed the highest ethical standards."

The SEC had not inspected Madoff's investment advisory business since he registered the firm with the agency in September 2006.

SEC examiners reviewed Madoff's brokerage business in 2005 after an employee at an investment firm and press reports questioned the validity of his stated returns.

The regulator's enforcement division completed a probe involving Madoff's company last year without bringing a claim.