Multicast+Routing

=Setting Up Network Infrastructure for Mac Image Deployment=

Introduction
This guide is intended to help systems and network administrators set up switches and routers to allow Macintosh clients to discover netboot servers on other subnets, and allow ASR servers to multicast images to clients on other subnets. Each of these tasks can be accomplished independently of one another. You can set up a netboot environment to allow a single netboot server to boot clients across multiple subnetworks, and forgo the ASR multicast setup; or you can do the opposite – you can enable multicast ASR, and rely on a netboot server on the local network.

Throughout this document I use Cisco’s IOS commands for enabling multicast routing. However, I will try to explain not only what commands I use for my switches, but what features I’m enabling and why. This will (hopefully) allow readers to look up the features they need for their own brand of switch, rather than leaving them to figure out how to modify my configs to work on their hardware.

Required Knowledge
Before you begin working with these two topics you should have a foundational knowledge of how networks work, and how the NetBoot and ASR processes function in MacOS.

On the network side, a CCNA-level of knowledge should be more than sufficient to set up your network hardware, regardless of whether you are using Cisco gear, or equipment from another vendor (I use a mix of Cisco and 3Com switches and routers). Specifically you should be familiar with:
 * How to set up and connect IP networks using routers and routing switches,
 * The difference between Unicast, Broadcast, and Multicast traffic on a network,
 * Setting up VLANs in your switch or router,
 * Enabling VLAN routing in your switch,
 * Static routing (if you are using routers),
 * Dynamic routing protocols such as EIGRP or OSPF (more on why that’s important later)

Hardware and Software Requirements
If you just want to netboot across subnets, then you can get away with some pretty basic equipment. You’ll need managed switches or routers that can do DHCP packet forwarding. In Cisco land, any of the managed switch portfolio will accommodate this. If you want to route multicast packets across your network, you’ll soon discover the difference between a $600 switch and a $6,000 switch. You will need switches or routers that support multicast routing (obviously) as well as some sort of dynamic discovery protocol to allow the network to discover routes for traffic to take. In these examples, I use Packet Independent Multicast (PIM) routing. Other multicast routing protocols are available, but PIM is (in my experience) the easiest to configure and manage in a small or mid-size network.

Cisco switches and routers will require at least the IP Base software package to handle multicast routing. Some equipment will require the IP Advanced package. Use the Cisco Software Selector to see what software level you will require:
 * @http://tools.cisco.com/ITDIT/CFN/jsp/index.jsp

In your test environment (with a server and one or two computers) you can easily get away with modest equipment. I use a Cisco 3550 series switch with IP Advanced to test my configs. Ebay usually has a large selection of these for a few hundred dollars.

In a production environment, don’t skimp on the switches. 48-port gigabit switches that have enough CPU and the software to run PIM usually run several thousand dollars each. Cisco Catalyst 3560 gigabit switches are usually acceptable for access layer hardware, but I’ve found it cheaper to start buying blade-style switches once I get beyond the 144-port range.

You can also use routers to handle the multicast traffic if you don’t have routing switches. Again, you should be using a relatively new router with enough CPU to handle the workload of distributing the multicast streams.

Network Design


In your site network, you will need to have control of a router or routing switch that is responsible for your network. In the example above, the network 192.168.0.0/20 has been allocated to the switch attached to Router A. The switch is responsible for inter-VLAN routing, as well as routing traffic back to Router A for distribution to the campus network. I would make all my configuration changes on that switch.

Your design could be even simpler. You could have a switch with multiple VLANs to separate broadcast domains (for example, a school with 300 computers would most likely split its network up into several subnetworks to break up the broadcast domain).

You will need to make configuration changes on the router or routing switch that is responsible for your network.

Starting Simple
Before you dive in, start with the simplest possible configuration. You can chose to do this either in a single site with multiple networks (presumably serviced by a single switch or router), or use a low-end managed switch (a Cisco Catalyst 3550 switch running IP Base or similar).



In the above network we have the server on VLAN 10, and the workstation on VLAN 15. We have already enabled routing in our switch, and verified that it works by mounting an AFP share on Workstation B from Server A. Server A has NetBoot enabled, and we’ve verified that it works by testing a client on the same subnet as the server.

Hints:
 * To enable inter-vlan routing on your Cisco switch, issue the ip routing command in global configuration mode.
 * Set up a DHCP server for VLAN 15 on your switch. NetBoot environments don’t allow for static IP address assignment.
 * Enable portfast on your switch ports so the port is in a forwarding state as quickly as possible. See the Spanning Tree section of your manual for more details, and analogous commands for other vendor’s hardware.
 * In the lab environment your Mac server should not have the DHCP service enabled. You could end up assigning IP addresses to clients on the wrong LAN.

Configuring NetBoot
At this point we can pass IP packets between the workstation and the server, but if we open up System Preferences, we discover that we don’t see the NetBoot volumes on Server A.

To accomplish this, we’ll enable an IP helper address on our switch. Below is the configuration for a Cisco switch:

SwitchA# config t SwitchA(config)#int vlan 15 SwitchA(config-if)#ip helper-address 192.168.10.10

It’s that easy! After a few moments, your client should see the NetBoot volumes in the Startup Disk system preferences pane.

The ip helper-address command on an interface tells your switch to listen for DHCP and BOOTP packets. When it sees one it forwards it along to the designated server. If you look at the NetBoot log on your Macintosh server, you should see BootP requests from the client scrolling through. It’s important to note that it’s perfectly permissible to have multiple IP Helper (or DHCP relay) statements on your switch – you could forward the packets to several servers at once. In a production environment you may find that you have to have at least two helper-address statements: one for your NetBoot server and one for your DHCP server.

At this point, you should be able to netboot your client from the server.

Hints:
 * If you’re not using a Cisco switch, look in your manual for “DHCP Relay” or “DHCP Helper.” That’s the feature we’re using here.
 * On some 3Com switches, I’ve found that I have to add an IP helper address for both the Mac server and the DHCP server in the switch.
 * If you set up a DHCP helper, but you still don’t see the NetBoot volume on your client, reset its PRAM (Command-Option-P-R at startup).

Configuring Multicast Routing
This next bit is what most network administrators object to. Enabling multicast routing can be a big leap of faith because it’s not something that many networks deploy in a production environment. However, it’s nothing out of the ordinary. If you’ve ever worked with dynamic routing protocols like OSPF, you should feel right at home.

For a more detailed introduction to the world of multicast routing and more configuration examples than you can shake a UTP cable at, review Cisco’s Multicast Quick-Start Configuration Guide. This covers securing your multicast network (a must in a production environment), and some reasonably exotic situations like how to configure multicast over a satellite link.
 * @http://www.cisco.com/en/US/tech/tk828/technologies_tech_note09186a0080094821.shtml

Cisco recommends using Protocol Independent Multicast Routing (PIM Routing) in most network environments. It automates the task of distributing routes, and facilitates routing multicast packets through equipment that may not be multicast aware.

On our Cisco switch, the configuration is pretty simple. First, enable multicast routing in global configuration mode:

SwitchA(config)#ip multicast-routing

Next, enable PIM on each of the VLAN interfaces:

SwitchA(config)#int vlan 15 SwitchA(config-if)#ip pim sparse-mode SwitchA(config)#int vlan 10 SwitchA(config-if)#ip pim sparse-mode

You should now be able to use Deploy Studio (or the ASR command) to restore a volume.

Keep in mind that Bonjour, Apple’s discovery protocol, is non-routable. You will not see your ASR server in the automatic discovery tools. In my lab, I started up a client on the same LAN as Server A, and copied down the information from the auto discovery process – that way I had it on hand for imaging my client on another network.

From the command line on your client, you could run ASR as follows:

Netboot-client# asr restore \  --source asr://192.168.10.10:7800/Path/To/Image.dmg \  --target /dev/disk0s2 --erase --noverify

Again, if you find that you’re having trouble, image your client on the same LAN as your server first and verify that everything is working correctly before you continue.

Scaling It Up
At this point you should be able to netboot and multicast restore your client from a server on a different VLAN on your test switch (or router). This might be useful if everything in your network is connected through one core switch, but most of us aren’t able to do that. We need to be able to netboot clients in different buildings, and image them efficiently.

DHCP Helpers require no additional configuration. As long as UDP ports 67 and 68 are not blocked in your network, you will be able to discover your netboot server from other networks.

PIM routing, on the other hand, requires some additional configuration. In its simplest form, you can simply enable it on the appropriate interfaces on your adjacent network devices:



In this example, multicast routing would be enabled on each router, and each relavent interface would have PIM enabled:

RouterA(config)#ip multicast-routing RouterA(config)#int fa 0 RouterA(config-if)#ip pim sparse-mode RouterA(config-if)#int fa 1 <span style="font-family: 'Courier New',Courier,monospace;">RouterA(config-if)#ip pim sparse-mode

<span style="font-family: 'Courier New',Courier,monospace;">RouterB(config)#ip multicast-routing <span style="font-family: 'Courier New',Courier,monospace;">RouterB(config)#int fa 0 <span style="font-family: 'Courier New',Courier,monospace;">RouterB(config-if)#ip pim sparse-mode <span style="font-family: 'Courier New',Courier,monospace;">RouterB(config-if)#int fa 1 <span style="font-family: 'Courier New',Courier,monospace;">RouterB(config-if)#ip pim sparse-mode

Just as we did in our test lab with a single switch, we would find that we could route multicast traffic across the two networks.

However, very few organizations have networks made of directly adjacent devices with dedicated media. There is usually some sort of service provider magic going on to form a connection between two sites. Consider the following network:



In this example, we have routing switches, but traffic must cross a service provider’s network. We only have the ability to make configuration changes on the two routing switches.

PIM has a solution for that. You can create a Rendezvous Point (RP), which is the address of a switch or router that is directly connected to the device(s) that generate multicast traffic. Other routers and switches then subscribe to this RP, using your regular IP network.

Let’s say our ASR server is attached to Switch A, and we’re imaging a client attached to Switch B. First, we would globally enable multicast routing, and PIM on the interface for the VLAN 192.168.14.0/24:

<span style="font-family: 'Courier New',Courier,monospace;">SwitchA(config)#ip multicast-routing <span style="font-family: 'Courier New',Courier,monospace;">SwitchA(config)#int vlan 100 <span style="font-family: 'Courier New',Courier,monospace;">SwitchA(config-if)#ip pim sparse-mode

Now we need to tell SwithA that it will serve as a Rendezvous Point for other networks. On our Cisco switch, we make that change in global config mode:

<span style="font-family: 'Courier New',Courier,monospace;">SwitchA(config)#ip pim rp-address 192.168.14.1

The rp-address is any interface on the switch. In this example, I’ve used the gateway address for the locally attached network. However, you may want to choose a loopback interface on your switch or router, so the multicast route continues to function even if the VLAN interface goes down. This is especially practical if you have multiple VLANs on your switches!

On Switch B, you (again) enable multicast routing, and enable PIM on the relevant VLANs. However, your rp-address is the address you declared on Switch A – it’s the parent source for your multicast traffic:

<span style="font-family: 'Courier New',Courier,monospace;">SwitchB(config)#ip multicast-routing <span style="font-family: 'Courier New',Courier,monospace;">SwitchB(config)#int vlan 100 <span style="font-family: 'Courier New',Courier,monospace;">SwitchB(config-if)#ip pim sparse-mode <span style="font-family: 'Courier New',Courier,monospace;">SwitchB(config)#ip pim rp-address 192.168.14.1

Because PIM uses the existing routing information to forward packets between the networks on Switch A and Switch B, the provider equipment does not need to be aware of the multicast nature of the traffic passing between the two LANs.

In continuing with the above example, you would configure additional LANs on your remaining switches and routers to get multicast information from the RP on Switch A.

Hints:


 * Check out Petr Lapukhov’s article for some helpful troubleshooting tips for PIM routing.
 * For more details, look in your switch or router’s configuration guide for the section on multicast routing. For Cisco routers and switches, see the appropriate section of your IOS version's manual. Here's the page for IOS 12.4 T.
 * Pay attention to what multicast address you assign your broadcast server, especially if your network traffic is traversing a service provider’s network. The network 239.0.0.0/8 is reserved for internal use by organizations, and should not interfere with other multicast traffic. @http://en.wikipedia.org/wiki/Multicast_address @http://www.ietf.org/rfc/rfc2365.txt
 * You can restrict which multicast addresses can broadcast traffic into your network with access control lists. The IOS manual and the Cisco Multicast Quick-Start Guide have examples.

Troubleshooting
PIM can work well in a wide variety of environments, but as with anything, the more complexity that gets introduced, the more chances there are of something going wrong. Below are some tips that have served me well, as well as some references to other websites with useful troubleshooting procedures.

First, start with a single switch lab. If you skipped the lab exercise in this document, go back and do it. Set up a test environment with a fresh install of MacOS X Server, MacOS Client, and a switch configured from scratch. Use up-to-date software for everything, and don’t operate any unnecessary devices on your switch.

After you get your basic lab working, use the same code versions that you use in your production environment (for example, if you use older versions of MacOS X server). Then use it on a production LAN switch. Then move the client into a LAN in another building.

If you’re sending data over a service provider’s network and you’re having trouble, set up a lab with the gear that you use on your side of the provider’s equipment. For example, you may have a Cisco 3560 switch connected to a 3Com 5500 switch, with the client on the 5500 and the server on the 3560. Set up a single access link between the two switches to emulate your provider’s connection. If you can make the configuration work correctly in your lab, go to your service contact at the provider.

The more up market ISPs will have no issue supporting your multicast traffic (for example, I use it on a national carrier’s metro Ethernet network through government provided routers with no issues). I can even use multicast routing through a VPN tunnel on my DSL modem (it’s used for music on hold with an IP phone). More “value” oriented ISPs will have difficulty supporting multicast routing simply because it isn’t as commonly used as unicast routing. If you’re using one of these, you may have to resign yourself to having an imaging server in each building. Finally, if you’re using globally routable IP addresses to get traffic between sites (not dedicated circuits), you will probably have problems getting your traffic through the network.

Pay attention to the addresses you use for your imaging server if you’re sending traffic over a service provider’s network. The network 239.0.0.0/8 is reserved for internal use by organizations, and should not interfere with other multicast traffic.

Use network monitoring software like Intermapper to monitor traffic congestion and packet loss on your networks. You may find that 6Mb of traffic is leaving the WAN interface on your router at one site, but only 1Mb is arriving at the receiving site! Netboot and multicast traffic are good ways to stress-test your network!

I found it helpful to use the command line ASR server and client before I stepped it up to Deploy Studio. I was able to watch for errors (use verbose mode) on both ends, and see where things were going wrong.

Use lower data rates for routed multicast traffic than you normally would on a LAN. Your multicast traffic has to traverse the network with all the other traffic your sites generate. If you experience a lot of packet loss, slow things down. Again – test by imaging a single client from your server before you set out to do whole lab.

I found Petr Lapukhov’s article on troubleshooting multicast routing to be extremely helpful. It covers the Cisco commands used to show and prove that multicast routing is working successfully.