The counter parties which SEC claims were defrauded are institutional investors whose managers are paid to make their own judgment when investing. If they made the wrong call then they should be charged, not Goldman. No one should sympathize with ACA or IKB.
Goldman was privy to non-public information due to its role as a market-maker. If it did not disclose that Paulson was shorting the housing market to its other clients; its just ethical business.