A while ago I installed two new Isilon H400 clusters. With any IT infrastructure, consistency and predictability is key to a trouble-free experience in the years to come. Cables should be neatly installed, labeled and predictable. Wiring in the internal network cables, it helps if the nodes 1 through 4 are connected to switch ports 1 through 4 in order, instead of 1,4,2,3. While some might consider this OCD, it’s the attention to detail that makes later troubleshooting easier and faster. Like a colleague said: “If someone pays enough attention to the little details, I can rest assured that he definitely pays attention to the big, important things!”.
So I installed the cluster, configured it, then ran an isi status to verify everything. Imagine my delight when I saw this:
Aaargh!
Where does this strange order of IP addresses come from?
A node is assigned an IP address once you add an interface into subnet0:pool0, the default pool in the default subnet. Which node interface is added to that pool by default when a node joins the cluster, is governed by a rule (default name: rule0). You can adjust this rule, so all new nodes get for example LACP interface 10gige-agg-1 in that pool, instead of the default 10gige-1.
This customer wants to use LACP for link aggregation. So after the installation of the first node, we changed settings but also continued with adding nodes. We also ran into some network issues, causing us to play around with the interfaces a bit more. At some point in time, IP address 112 was taken and then released, but the other nodes already had their assigned IP addresses. Resulting in a gap. This gap isn’t a massive problem: everything will work as intended. However, it’s ugly… and it will survive a reboot.
Fixing it is easy!
You can reassign IP addresses by removing an Isilon node from the network and then readding it. And this is done in the pool configuration. In the example above, node 1 has the correct IP address already. So connect to the Isilon cluster over that IP address, making sure you end up on node 1.
Next, open the SmartConnect settings and navigate through your subnet0 to pool0.
If you click Edit here, you can adjust the pool Interface Members. Then, remove all the interfaces for node 2, 3 and 4. This will remove them from the network and leave just node 1 connected. This is also why you need to check to which node you are connected, else you run the risk of locking yourself out. Click Save/OK, and this page will update.
Next, simply add the correct interfaces to the pool again. Nodes are now assigned a new IP address, and if you do this properly, they should be in order.
Voila!
My thoughts on this Isilon network config
In my humble opinion, attention to detail is important. It exudes a certain level of pride and quality to your installs. And it might come in handy in the 5 years the system will be running, either in day-to-day management or in troubleshooting. So making sure your Isilon node IPs are in order, is one extra step towards predictable operations.
In this specific case, the customer wanted to use LACP for load balancing. And I know LACP isn’t really load balancing, but more link aggregation. In their case, it makes sense because all their infrastructure does it in this way, and they have plenty of experience in configuring LACP port channels.
That said, I’d install new Isilon clusters without LACP. Especially if you license SmartConnect Advanced, this gives you much more control over load balancing and workload segregation. Plus it saves you potentially difficult LACP / VPC configurations on your network infrastructure.
