Isilon scale-out NAS clusters (or grids) are built out of nodes or servers using (relatively) cheap commodity hardware. Compared to a traditional storage system this has the advantage that with every capacity (=disks) expansion you’re adding to the cluster, the number of CPUs, amount of RAM and network ports grows accordingly. You just add another node or server and poof: more TBs, more speed. The proprietary software OneFS then glues all those nodes together to create one single immense filesystem (up to 20PB with current drive specs). But what’s one trait of commodity hardware? You occasionally need an Isilon firmware upgrade! The LSI disk controller needs a new release once in a while, as do perhaps the front panel or the Infiniband components. This howto explains what to do to make sure you’re running the latest firmwares!
Impact of an Isilon Firmware upgrade
As with most servers, if you apply a firmware upgrade the server or component will have to reboot. During an Isilon firmware upgrade this is no different. Only one node will reboot at a time so your data will always be available to clients. An ideal moment to plan this would be after you’ve upgraded your OneFS version, since nodes have been rebooting for that maintenance anyway.
As with any upgrade: make sure you check the release notes ahead of time and note any fixes, bugs and dependencies. This guide is a condensed howto; if you don’t know what to do exactly, make sure either EMC Tech Support performs the Isilon firmware upgrade or get yourself the specific manual for the nodes and firmware release you’re upgrading to.
Start by downloading the firmware package from the support.emc.com website. The packages will be listed under the downloads section of the Isilon nodes in question, NOT the OneFS section. For example, if you’ve got NL-nodes, navigate to these specific nodes and look for the firmware package there. I’ve downloaded release 9.0.1 to upgrade three NL400 nodes. Upload this to the Isilon system in the /ifs/data/ directory.
Getting started
Once the firmware is safely on the Isilon grid, open an SSH session and install the package with isi pkg install <pathToPkg>. This will unpack the firmware package and place it on all the Isilon nodes. Next, install the firmware with isi firmware update. Both these steps should look like this:
As you can see this system only needs to update the LSI controller. Type yes to confirm you want to upgrade, sit back and enjoy the ride. During the firmware upgrade you will be kept apprised of the status and which node is rebooting. You may have to confirm you want to continue after the first node is upgraded, due to the cluster being in attention state. Just follow the on-screen instructions.
Once the process upgrades and reboots the last node your SSH session will break. No worries, this is normal. Just wait a couple of minutes for the node to come back online, reconnect and run isi firmware status to check if everything went according to plan. Little side note: do NOT run multiple instances of isi firmware status at the same time since according to the manual, this might break things. Also don’t run this command with isi_for_array which basically spawns a process on every node and thus generates multiple instances. Just run it once in your current session and it will display in detail which nodes run what version of firmware:
Upgrade complete! You might want to send a log file bundle to EMC using the isi_gather_info command so these guys and girls know you’ve upgraded. And that’s it!
