Adding Isilon Nodes in 60 Seconds

Adding Isilon nodesLast year EMC installed two new three-node Isilon clusters for one of our customers. 13 months later the customer needs more capacity so the time for adding Isilon nodes to the existing clusters has finally come. Good news for me: since the initial install I got certified to install Isilon systems myself so these expansions are all mine! EMC marketing promises an Isilon cluster expansion in 60 seconds; let’s put it to the test!

The customer uses the Isilon clusters for long-term archive storage for their PACS systems: data will have to be stored for 15 years or longer. In essence this means that no data will be thrown away and the systems will keep on filling up for at least the next 15 years.

The clusters that need to be expanded are three-node NL400 Isilon clusters running OneFS 7.0.2.7. 4U nodes, 36 drives each + CPU, RAM and network interfaces. Installing them in the rack is fairly straightforward (read the initial cluster install here): install the rails, slide in the node, insert the drives (24 in the front drive bays of the NL400, 12 in the rear bays) and attach the front panel. When inserting the drives, please check that you’re installing the correct drives for the node: the packages with drives have a label on the outside with the node S/N. Also check that there is no foam trapped in the connector of the drive. Next up, install the Infiniband cabling, external network (make sure the network team has the switchports pre-configured!) and power cables. And that’s it for the physical portion in the datacenter…

Next up lets head to the OneFS management page to perform some pre-checks. Of course you’re not going to expand a cluster that’s not healthy: check the cluster overview for errors. If everything is OK, navigate to the network provisioning rules. Ideally you have a set of provisioning rules that automatically assigns the new nodes interfaces to the correct sub
nets and pools.

Firmware package check prior to adding Isilon nodes.Finally, make sure there’s no firmware package installed on the cluster. This is a lesson in “follow the procedure to the letter”. The procedure to upgrade the firmware on an Isilon cluster clearly states at the end to remove all firmware packages. I thought to be smarter than the procedure: “Oh, I’m adding a node in a short while anyway; lets just keep the package on the cluster so it can automatically update the firmware.” Well.. no. If you have a firmware package installed on the cluster, adding Isilon nodes is not possible since the whole process will halt with an “error downloading new image from cluster” error message. Your LCD front-panel will go blank and you will only see this error in the OneFS GUI. Follow the steps in the screenshot above to remove the firmware package (or search for emc166311 on support.emc.com for the full KB article) prior to adding the Isilon nodes.

Time for the actual expansion: press the little power button on the rear of the server, wait a couple of minutes for the LCD on the front of the server to come alive and use the buttons on the front panel to select and confirm the cluster you want to join. If you’re using dedicated Infiniband switches per cluster and haven’t enabled secure join mode this should just be a single push on the square button. You can make a “beep” sound while you press the button if you like…

After adding Isilon nodes, the new node is visible and healthy. You can see the difference in used capacity.That’s it! The node will download the necessary images and downgrade or upgrade its OneFS version automatically depending on the cluster version. Once everything matches it will join the cluster. It might throw an error about the network interfaces not having an IP address; this should automatically resolve if your network provisioning rules and networking team have set up everything correctly. Do ask them to check all the interfaces came up alright and whether everything looks good from a networking side.The running jobs after adding Isilon nodes. The MultiScan job rebalances capacity across the available nodes. You can adjust the impact to speed it up or down).

The Isilon custer will automatically start to rebalance data across the available cluster nodes with the MultiScan job. This rebalancing happens across the internal Infiniband network so no worries: your external network will not be burdened with TBs of traffic. It might take a long, long time though. You can check the progress on the Operations page, speed it up by adjusting the impact policy for the running job or pause it to allow other jobs to complete first. For example on the secondary cluster (only used in case of DR) I’ve adjusted the default impact from Low to Medium to speed it up a bit…

Back to the 60 seconds claim. Yes, if your new node has the same OneFS version as the cluster, the joining process is over in less than a minute. And as soon as the node is attached to the cluster, the capacity is available for the end-users!  If you include racking and stacking and checking a couple of things in advance, it will take you somewhere close to an hour or two to do it neatly. Not quite 60 seconds, but still immensely fast and easy compared to other systems out there!

  • dynamox

    Good post as usual. I would like to add a little note for folks who are working with clusters that contain SED (self encrypting drives). It takes about 2 hours to format an X400 node with these type of drives, don’t be alarmed if you don’t see the new node immediately. Take a long lunch break and come back in a few hours 🙂

  • Rob Koper

    Nice post, Jon! I’m proud of having such a fine colleague!!!

  • Swapnil

    Hi,

    This is really a nice post indeed. I am doing the same i a couple of days, adding 3 nodes to an existing configured isilon. Can you please help me with a few clarifications from the steps you mentioned:

    1. If you have a firmware package installed on the cluster, adding Isilon nodes is not possible since the whole process will halt

    You mean package installed on the new node right?

    2. We need to uninstall the package from the new node correct? By using isi pkg command

    3. Since the site is remote and i am flying back the same day, what all things i should be carrying with me, so i do not miss anything. (already have the console cables etc.)

    • Jon Klaus

      Hi Swapnil,
      Good luck with the expansion! As for your questions
      1) No, there should not be a firmware package on the existing nodes, so run the commands against your existing cluster nodes. Your new nodes will not have a firmware package on them anyway.
      2) Uninstall any firmware packages on the existing, operational nodes using the isi pkg command.
      3) The serial cable does come in handy if your new node is misbehaving and you want to look on the console what’s going on. Other than that everything is plug and play and mounted in the racks with thumbscrews. So a screwdriver might be useful to tighten screws a bit more. Other than that everything should be included in the install kit.

      Good luck!