Nicola Horlick's Bramdean, Arkie Busson's EIM and Man Group were today named as potential targets of legal action in the Bernie Madoff $50 billion (£32 billion) Ponzi scandal.

A leading New York law firm specialising in complex financial cases has formed a Madoff Task Force to collate claims from those who have lost money in the world's biggest fraud. Wolf Haldenstein Adler Freeman & Herz has recently been involved in seeking compensation in the collapse of Bear Stearns, and in chasing Vodafone over its US accounting practices.

The firm said it had been retained by a number of parties, adding: "It is now apparent that funds of funds, hedge funds, investment advisory firms and brokerage houses invested billions of dollars with Madoff, and these investments are now worthless.

"It is becoming increasingly apparent that the funds of funds that entrusted their clients' money with Madoff should have been aware of the numerous 'red flags' that were raised by Madoff's activities. It is also now apparent that these funds of funds failed to perform the necessary due diligence that they were being compensated to perform."

Funds of funds named by Wolf Haldenstein include RMF Investment Management, the fund of funds arm of FTSE 100-listed Man Group; Bramdean Alternatives, an arm of Horlick's investment management firm; and EIM Group, run by Busson, pin-up boy of the Mayfair hedge fund scene and boyfriend of Hollywood star Uma Thurman.

Horlick has insisted the Madoff affair points to "a systemic failure of the regulatory and securities markets regime in the US". Man Group has launched its own investigation.

Busson said it was "practically impossible" that Madoff was acting alone and insisted he was not at fault. "There's only so much due diligence you can do, and in hindsight you always wish you could have done it differently," he said this week.

Auditors of hedge funds involved in the affair are also likely to be targeted. BDO Seidman, a sister firm of London accountants BDO Stoy Hayward, is being sued over its role as auditor of Madoff feeder fund Ascot Partners.

Lawyers are expected to focus on "feeder" funds, which placed their money with the alleged fraudster. Class actions are likely to be complex as Madoff had relationships with a vast global network of asset management firms and fund of funds managers that steered wealthy client assets to him.

Experts say that while thousands of people lost money in former Nasdaq chairman Madoff's fraud, others made gains that they may now have to pay back.

Tamar Frankel, a law professor at Boston University, told the New York Times: "In a Ponzi scheme, not all investors lose. Those who manage to get out in time retain their investments and some of their gains."

However, previous US court rulings in similar cases suggest some of these gains will have to be given up to compensate losers. It is not known how many of Madoff's clients are affected but the structure of Ponzi schemes, which use money from later investors to pay early ones, means many could be ahead.