Cleaning up and moving Avamar clients

Last year we decommissioned a physical Avamar grid in London because it was both out-of-support and the location was about to close down. The Avamar was however still being used for desktop/laptop (dt/lt) backups. A separate project was taking care of replacing those laptops, but in the meantime we needed to keep the Avamar backup service running.

Avamar capacity utilization decreasing

We did a quick calculation on the required capacity and deployed 4 new Avamar virtual editions in our central VMware farm. After configuring them and connecting them to the Avamar Enterprise Manager dashboard, we were able to move the majority of clients over.

Now, almost a year later, many of those laptops have been replaced and are no longer backed up by these 4 new Avamars. Which clearly shows in the utilization, as you can see. Three out of four systems are <10% utilized. Since these Avamars claim a fair bit of resources from the VMware farm, I set out to consolidate the systems into the first virtual Avamar. Thus, reclaiming 75% of the resources.

Continue reading

Data Domain DD OS upgrade hung on phase 4/4

DD OS upgrade in progressI recently installed a new Data Domain DD6300. Part of the whole installation procedure is to run a DD OS upgrade to bring the system up to the target DD OS release. You can find the target releases over here. While running the upgrade to 6.1.2.20, the Data Domain correctly rebooted as part of the upgrade. Logging back in, the system GUI kept throwing an “Upgrade in progress” popup, blocking everything else in view. There is also an alert that shows “DD OS Upgrade is in progress. The system will not be available for backup and restore operations. The alert will be cleared after the upgrade operation is complete.” Which I guess is NEVER when the upgrade is hung…

Continue reading

Isilon SyncIQ uses incorrect interface: careful with mgmt DNS records!

I’ve installed quite a few new Isilon clusters in 2019. All of them are generation 6 clusters (H400, H500, A200), using the very cool 4-nodes-in-a-chassis hardware. Commonality among all these systems is an 1GbE management port next to the two 10GbE ports. While Isilon uses in-band management, we typically use those UTP ports for management: SRS, HTTP, etc. We assign those interfaces to subnet0:pool0 and make it a static SmartConnect pool. This assigns one IP address to each interface; if you do it right, these should be sequential.

Recent addition to my install procedure is to create some DNS A-records for those management ports. This makes it a bit more human friendly to connect your browser or SSH client to a specific node. In line with the Isilon naming convention, I followed the -# suffix format. So if the cluster is called cluster01, node 1 is cluster01-1, node 2 is cluster01-2, etc. However, it turns out this messes up your SyncIQ replication behavior!

Continue reading

Put all your data on flash with VAST Data

VAST Data logoRoughly 6-7 years ago (around 2012), flash storage became affordable as a performance tier. At least, for the companies I was visiting. It was the typical “flash tier” story: buy 1-2% of flash capacity to speed everything up. All-flash storage systems were still far away into the future for them. They existed, and they were incredibly fast, but they also drove the €/GB price too far up, out of their reach.

However, in the background you could already hear the drums: it is going to be an all-flash future! Not just for performance, but also for capacity/archive storage. In fact, one of those people beating that drum was my colleague Rob. I recall vividly our “not yet!”-discussions…

And it makes sense. Solid-state drives are:

  • More reliable: there are no moving parts in SSDs, and media failures are easier to correct with software/design.
  • Power consumption is very low at rest: there is no little motor to keep platters spinning 24/7.
  • Faster: the number of heads and the rotational speed of the platters limit a hard drive’s performance. Not so with flash!

They are still quite expensive, looking at €/TB. Fortunately, cost is coming down too. The last year or two, all flash arrays have taken flight in general-purpose workloads. Personally, I have not installed a traditional tiered SAN storage system in over a year anymore. Hyper-converged infrastructure: same story, all flash. The development of newer, cheaper types of QLC flash only helps close the gap in €/GB between HDD and SSD. But there is still a 20x gap. And one company we met at Storage Field Day 18 has a pretty solid plan to bridge that gap: VAST Data.

Continue reading

The fast WekaIO file system saves you money!

WekaIO file system logoIt is a fact of IT life: hardware becomes faster and more powerful with every new generation on the market. That absolutely applies to CPUs. A few weeks ago at Intel’s Data Centric Innovation Day in San Francisco, Intel presented their new Intel Xeon scalable processors. These beasts now scale up to 56 cores per socket, with up to 8 sockets per system/motherboard. This incredible amount of compute power enables applications to “do things”, whether it’s analytics, machine learning, or running cloud applications.

One thing in common across all applications is that they don’t want to wait for data. As soon as your %iowait is going up, you are wasting your precious and expensive compute power because the storage subsystem is not fast enough. Fortunately, WekaIO wants to make sure this will not be the case for your applications.

Continue reading