Remedial Networking 102: How Multicast Works

Did you ever have anything in your life that you just kind of…ignored until it went away? Or at least, ignored it until you forgot about it or pretended it went away? That’s kind of what I did with multicast.

For a long, long time, I would hear people mention multicast, and I would be like, “oh, multicast, yeah, talking to lots of hosts at one time, yeah, that’s cool…” then do my best to not join in that conversation, because I had less than nothing to add. In fact, all I could do was show my ignorance, and THAT’S no fun.

So my shame of not knowing it drove me deeper into my cave. “It’s not like anyone even uses it. I’ve been doing networking for years and its never come up”.

Then I started learning more about IPv6, and multicast would creep into the conversation. And I’d ignore it, and sure enough, the next paragraph wouldn’t mention it anymore, and I could go back to pretending it didn’t exist. But it kept coming up, and one day I slipped.

I thought to myself, “so, I know that multicast is all about sending data to lots of hosts at once, but how could it really do that?” And so I slipped into the inescapable void of my curiosity.

As it turns out, it’s really not all that crazy, and it’s really not that difficult, at least on the surface. There are two sides of the proverbial coin. There’s the theory of how it works, and there’s the practical application. Lets look at the practical side.

A prototypical example use-case would be video streaming. Every multicast application (and by application, here, I mean every solution, such as a NASA video stream and the like) picks or is assigned a particular multicast IP address, then everyone on the internet that wants to watch the stream joins that multicast IP address…and when I say it joins that multicast IP address, I mean that the machine uses a protocol called IGMP to tell its local router that it wants in on the action.

The local router is hopefully configured to know about IGMP so it listens. If it’s configured properly, then it uses a protocol called PIM (or one much like it) to let the next closest router to the video source know that it has someone who wants to watch. That router then sends PIM updates to the router it thinks is closer to the source than it, and so on.

Eventually, you wind up with a tree starting with the root being the video server, and forming the branches are all of the routers who have clients that used IGMP to let them know that they wanted to watch. The video source sends out one packet, to its closest router, and the router remembers which connections joined that group and sends the packets to those routers, which send the packets to the next routers, and the packets eventually wind up at the clients who are watching the stream.

No one machine sends a ton of packets, and no routers have to remember much besides which interfaces belong to which groups. It works out nicely, even though, and I say this with the greatest apology now that you’ve read the preceding paragraphs, almost no one uses it like this.

Sure, there are some internal networks that use it for things like teleconferencing, but I bet yours doesn’t. Except you Googlers and Yahoos out there. Your university probably does, I guess, especially if it’s on that whole Internet2 thing, but the vast majority of you will never have to deal with multicast video streams.

That doesn’t mean that multicast is going away. Actually, with IPv6, it’s getting more airtime.

The biggest difference between IPv4 and IPv6, in my opinion, is the idea that your interface isn’t going to belong to only one layer 3 (IP) network.

You know how, right now, you are probably on a private block, like 192.168.whatever? Well, with IPv6, you’re still going to have a private block, only it’s called link-local. Your machine, and every other machine, will have a link-local network address on their interface. It doesn’t route to the internet at all. When you have one machine that wants to talk to another local machine, it’s going to happen over the link-local interface.

Of course, your computer will also want to talk to the internet, probably, so you’re going to have an IP with the “real” network prefix (which is probably going to be assigned automatically by the IPv6 autonegotiation process, which I’ll write more about later). This “real” network IP will allow you to communicate with the outside world (and it with you, unless you set up your firewall rules correctly). That’s two IP addresses so far. How many more?

Well, there is also going to be the “All Nodes” multicast group, FF01::1 for “node-local” (AKA “all interfaces on the same machine”) and FF02::1 for the link-local (AKA “the local broadcast domain” because routers don’t forward link-local packets) group. Your machine will belong to these multicast groups, because things like ARP are going away with IPv6.

Instead of ARP (and ping and passive IP collision detection), IPv6 includes a protocol called NDP, which provides for all kinds of things…and most of those features come from using multicast addresses.

So what happens is really just a smaller version of above, where the host tells the router that it wants to join a multicast group. Except that in IPv6, we don’t use IGMP anymore, we use MLD, but it works mostly the same way. The router sends us any packets that are sent to that IP address, and things go swimmingly.

If the machine that you’re working on IS the router, well, then it’s going to join any number of IPv6 multicasts…the “All Routers” multicast (ff02::2), maybe the All DHCP Servers group (ff02::1:2), and seriously, a ton of other multicast groups.

The big reason for all of these groups is service discovery. It’s kind of like, in IPv4, if you had a specific private IP block for, say, DNS servers. If you just knew that somewhere in, there would be a DNS server, you could join that network, ping the broadcast, and the DNS server would respond. You could use that to add it to your configuration. It’s kind of like that, but instead of a broadcast, it’s multicast.

If you’re the rare networking wonk who didn’t know multicast, you’ve probably already asked, “yeah, but how does that work at the switch?”, in which case I invite you to read the wiki entry on IP Multicast: Layer 2 Delivery.

So despite my earlier fears, multicast doesn’t operate by magic, just by protocols that I’ve never configured because they didn’t matter to me. But even with the scant knowledge that I just imparted, you can feel a little bit better about needing to use multicast all over the place with IPv6.

Computer Failure Data Repository

I stumbled upon a great resource today that I had no idea existed.

Apparently, USENIX hosts the Computer Failure Data Repository, which is a series of datasets from various installations that includes failure data for various components.

There are around a dozen different data sets, most from HPC clusters, but they’re actively soliciting for additional data:

If you already have your data public on your reference web page so that any one can download it, then all you need to do is to send us a pointer to your reference web page and a brief description of the data.

But otherwise – if you want to make the first release of your data through the CFDR – then the data contribution procedure is as follows:

We need to have a necessary paperwork on file to show that we actually have permission to host this data. You need to sign or find someone at your organization to sign our contributor’s agreement.
If the data contains some sensitive information like user or vendor names, you need to sanitize (anonymize) it. If you don’t have proper sanitization tools, we will try to help you.
Please provide any available documentation or description of the data you are contributing. If no documentation is readily available, it would be helpful to create one in the form of a FAQ with answers to frequently asked questions on the data. You can take a look at the FAQ accompanying the LANL data sets to get an idea of the kind of questions people commonly ask about failure data.
Make your data accessible for us, then we will host it on the CFDR server.

I’m looking forward to going through some of these data sets, just to see what kinds of things I can glean. The idea kind of reminds me of the Public Datasets that Amazon hosts, but more specific to my interests.

It’s not hard to see how crunching these kinds of numbers to get interesting statistics could be an interesting passtime (or maybe I’m just a giant geek).

Do you know of any other resources for long-term statistics on failure data related to IT? What else have I been missing?

Reader Question: Cloud vs Virtual Farm?

I really enjoy getting mail from readers. I know that I sometimes don’t answer in what you might consider a timely fashion, but that’s just because you’re not thinking geologically.

On occasion, though, I do get some downtime to discuss things with my readers, and the other day, I got a good question that I don’t think I’ve written about, so I thought I’d share with everyone else.

Here’s the email:


This might sound like a nooby question, but I will sally forth…

What is the difference between using a Private cloud solution and a Virtual Machine environment?

I am planning to update my test environment, and if possible use a personal Could based solution for testing various software platforms. Based on my limited knowledge of Cloud based platforms, The VM and Cloud based environments are pretty much the same, except when you shutdown an instance of cloud based machine, it loses it configuration.

I have a VM environment right now, but would like to be able to provision new machines at will (depending on hardware restrictions), but would like to keep some of these machines as configured.

Thanks in advance

Here was my answer:


I don’t think it’s a “nooby” question. It’s one that a lot of people struggle with, because there isn’t a clear definition, and the marketing in the industry tries to confuse the matter at every turn.

I’ve thought about it pretty extensively over the past couple of years, and I actually have a definition in my mind of “Cloud” that I think is logical and based on the origination of the term. Remember that the word cloud came from network diagram icons that indicated, for the purposes of that particular diagram, it didn’t matter what specific machines or software filled that particular space, only that there was a resource available that provided a service. At its core, it is a term of abstraction.

In my mind, cloud is still a relative term. Amazon’s AWS cloud is absolutely not a ‘cloud’ to the people who work on the discrete components of it. It *IS* a cloud to you and I, though, because when we interface with it, we see no distinction between the particular constituant machines. We only see the interface to the API that controls the creation and management of the resources that are hosted within the cloud. We don’t know which actual machine or machines inside the Amazon cloud are handling our resources, and we don’t care.

It should also be noted that there are different *types* of Cloud, too, and it all depends on what you want to abstract. If you want to create entire machines and network devices, there is Infrastructure as a Service (IaaS). Sometimes, you just want to write software and not worry about administering the web server that it runs on…then you want a Platform as a Service (PaaS). It’s also possible to just want to rent a service, like hosted Email, in which case you want Software as a Service (SaaS). But since you mentioned building virtual labs, IaaS is what we’re talking about here.

So, this brings us to your question about a private, internal cloud. It IS possible to build an internal cloud such that you can interface with it in an abstracted way, never knowing or caring (other than for debugging purposes, since you’re not only running things on the cloud, you’re running the cloud itself) where the resources are hosted. But for the immediate future, its almost always overkill in a smaller shop.

The Eucalyptus Project is devoted to creating private cloud platforms that can be administered using an abstracted API – in fact, it’s the basis for the new HP Cloud that’s currently in beta.

That being said, is it worth setting up for a lab environment? I guess the answer is, “It depends”. It would be an interesting exercise to build a cloud, just for the sake of doing it, but only you know if you have the number of machines available with the (admittedly low) required resources.

It may be possible to configure your current virtual environment to deploy new images quickly and without much manual intervention, but without knowing more details, I can’t be sure of the implementation. I would, at the very least, research the API for your virtualization platform and check out something like Cobbler.

I hope this answered any questions you have. Please feel free to follow up if you have any other questions. I’m happy to help.



Confusion and misconceptions about what a “cloud” is run rampant because, for a very long time, it was essentially an undefined marketing term. As the idea gels, we get a better idea of what to expect from the service providers.

Also, if you’ll note that I said that running a private internal cloud is overkill for the immediate future. That’s because I suspect more and more software will be developed with the inherent idea of “cloudiness” built-in. I’d be surprised if software like Tomcat didn’t have server abstraction built in during the next couple of releases, where you install Tomcat on a few machines, and then you deploy your application to a central controller, which then deals with the “cloud” instances. There are already commercial services for this, but there’s no reason it shouldn’t be added into the software itself.

Likewise with an abstraction layer above KVM, so you just launch new VMs onto an array of hardware, and you don’t know or care which machine it runs on. This is what Eucalyptus does, of course, but right now, Eucalyptus is exceptional for doing so, where as in a few years, it’s going to be par for the course and built in. More and more administration will be done through APIs, I’m telling you…

Of course, I could be wrong! Let me know what you think in the comments below.