A question came up on some other forums: I have an ospf domain with 10 routers. I wanted to configure ipsec between two ospf running routers for the sack of security but I came to know that multicast can not be forward over ipsec. Can anyone tell me the reason why we can not do that? And please also state solution for that. If anyone know useful document on this, please tell me.
The solution is easy. GRE over IPSEC.
The reason isn’t so simple. Here’s what I gathered and I’m sticking to it.
The reason is specified in one of the original IPSEC RFC’s: http://tools.ietf.org/html/rfc2401#section-4.7
For multicast traffic, there are multiple destination systems per
multicast group. So some system or person will need to coordinate
among all multicast groups to select an SPI or SPIs on behalf of each
multicast group and then communicate the group’s IPsec information to
all of the legitimate members of that multicast group via mechanisms
not defined here.
The RFC doesn’t say that it can’t be done. It just says that doing so isn’t easy, and they decided to leave the job of actually doing it to someone else. 
One of the big concerns with multicast IPSEC is obviously scaling. It’s something we don’t think about when we’re running point-to-point GRE over IPSEC from our remote sites to our hubs in low-to-no multicast networks.
However, what happens when you start multicasting from your hub site? Each point-to-point connection is encrypted with a separate key, hence each packet of the multicast stream causes the router to perform a key lookup and encryption for each remote site, adding significant overhead through your router. The router has to “encrypt many” and “send many” for each multicast packet; in a network of 1500 peers, that means 1500 encryptions prior to 1500 replications of each multicast packet at the hub router.
Another way of looking at it is that for a single multicast stream, your hub site needs to be able to support the stream’s bandwidth multiplied by the number of peers. Since each multicast packet going to your crypto engine has to be copied first, THEN encrypted, a 1mb stream going to 1500 peers requires that your engine be able to handle 1.5Gbps of encryption throughput. You can’t encrypt the packet first and then send it out to multiple sites, because each site decrypts the packet differently using its own key.
Resolving this requires a re-examination of how IPSEC works. IPSEC is basically point-to-point, with two “connections” on each endpoint: one SA to encrypt outbound traffic, one SA to decrypt inbound traffic. For multicast to scale, IPSEC has to become either point-to-multipoint, or any-to-any. If the IPSEC peers had a common “group,” they could then encrypt a packet with the “group” key just one time, and replicate that packet to the number of peers; we go from “encrypt many, send many” to “encrypt ONCE, send many.” While the packet in my previous examples would still have to be replicated 1500 times, only one encryption would be performed on that packet. This reduces the burden on your crypto engine: it now would only have to encrypt the 1mbps stream, instead of 1500 1mbps streams. All peer sites would have the correct SA to decrypt the stream.
I believe Cisco’s solution to this issue was the introduction of Group Encrypted Transport. See here for more: http://www.cisco.com/en/US/docs/ios/12_4t/12_4t11/htgetvpn.html
Other concerns, including security concerns, are raised in the following IETF draft: http://tools.ietf.org/html/draft-ietf-msec-ipsec-multicast-issues-01
