7 posts

Cleaning up and moving Avamar clients

Last year we decommissioned a physical Avamar grid in London because it was both out-of-support and the location was about to close down. The Avamar was however still being used for desktop/laptop (dt/lt) backups. A separate project was taking care of replacing those laptops, but in the meantime we needed to keep the Avamar backup service running.

Avamar capacity utilization decreasing

We did a quick calculation on the required capacity and deployed 4 new Avamar virtual editions in our central VMware farm. After configuring them and connecting them to the Avamar Enterprise Manager dashboard, we were able to move the majority of clients over.

Now, almost a year later, many of those laptops have been replaced and are no longer backed up by these 4 new Avamars. Which clearly shows in the utilization, as you can see. Three out of four systems are <10% utilized. Since these Avamars claim a fair bit of resources from the VMware farm, I set out to consolidate the systems into the first virtual Avamar. Thus, reclaiming 75% of the resources.

Cleaning up old backup clients

First off all, a lot of backup clients no longer exist. To clean up the overview and to get a better sense of the active clients, I started with deleting all clients that haven’t contacted the Avamar systems in the last month. First step is to list all applicable clients. The easiest way to do that is to dive into the CLI and run the following commands:

psql -p 5555 mcdb

SELECT full_domain_name, checkin_ts FROM v_clients where (checkin_ts < ‘2019-05-01’ AND checkin_ts > ‘1970-01-01’) ORDER BY full_domain_name;

You can then exit the psql terminal with \q

Dump this output into Excel or Notepad++ and clean it up until you end up with the client name (e.g. /clients/laptop-1234).

Next, we’re going to delete these specific clients. The CLI command syntax is: mccli client delete –name=/domain/client. Again, you can probably best use Excel or Notepad++ to get the commands sorted. I ‘only’ had 200 clients per Avamar to clean up, so this manual method works reasonably well. If you have thousands of clients to clean up, you might want to make a proper script out of it.

Consolidating clients on the first Avamar

Once all the garbage clients have been cleaned up, the next step is moving clients over to the first Avamar. The EMC Avamar client manager works well here:

Avamar move clients with the client manager

Select a few clients on the old Avamar, and click move. Then, select the recipient Avamar and domain, and click Next. In the next screen, select the recipient group. You’ll have an option whether to replicate the existing backup history. In our case, we’ll replicate the existing backups to ensure we can actually restore something if a user deletes everything tomorrow.

Avamar replicate existing backups

Lastly, insert the password for the repluser on both the source and target Avamar. In our case, these are the same. If you lost this password, you can bypass this by NOT replicating existing backups.

Next, track progress either in the client manager, or in the Avamar client itself. Both should show you a number of replication jobs:

Avamar client move in progress Avamar client

Lastly, check whether the move job actually completed correctly:

Avamar client move results

You will see a few possible outcomes here:

  • The client moved successfully. In that case, no further action is required, other than maybe kicking of a manual backup to check everything.
  • All move tasks except activation completed successfully. In practice however, this client often activated correctly and a test backup succeeds, so this could be a false alarm. Check for registered (not activated) clients though, and retry activation where necessary.
  • Or the client is not on-line and the move doesn’t go through. In that case, the Avamars will not replicate the backup history and the client will remain on the old Avamar. Try again later!

That’s it. Since we cleaned up the old, stale registrations beforehand, any clients that experience problems during the move should pop up in respective client categories. Rinse, repeat and fix issues along the way. Once the old Avamars are empty, decommission them!

My thoughts on Avamar client management

Unfortunately, Avamar does not have a standing “please vacate this Avamar and move to that other one” order. Clients need to be online for these move actions to succeed. With servers, that’s not so much of a problem as they should be online 24/7. Laptops are a bigger problem, as users can have different working hours depending on their geographical location, work schedules and potential holiday absence. In my opinion, this is a major caveat in Avamar functionality.

If you have sufficient time, you could periodically retry these moves over a few weeks. However, there will always be a few clients that can’t be moved because the user is out of office or the laptop is only used for a few hours every month. These clients will have to typically be hunted down by Local IT and migrated manually. Maybe this has been improved in newer versions of Avamar; do let me know if you have better experiences with it!

All in all though: if your clients are online, this process works well. Good luck!

Data Domain migration and retaining your system name and IP addresses

DD3300Several of our Data Domains are end-of-life and need to be replaced with new hardware. In most of the cases it’s a small site with a small Data Domain that only holds roughly 1 month of backups. In these cases we just install a new Data Domain next to it, reconfigure our our backup software, and that’s it. After a month, the old backups have expired and you can switch off the old Data Domain.

For the slightly larger sites, there’s more than one backup client/server writing to the Data Domain. There are Oracle RMAN backups, SQL dumps, etc. Plus the retention of backups on the Data Domain is much, much longer. In these cases you want to perform a proper Data Domain migration which retains the name and IP address of the old Data Domain, so you don’t have to touch all the clients. Here’s how you do that, and a DDBoost gotcha you should be aware of!

Continue reading

Data Domain DD3300: Bye LACP, Hello DDBoost Interface Groups!

DD3300I recently deployed a  new 32TB Data Domain DD3300 system. The initial configuration is easy and familiar enough. Connect to the system via the serial cable, setup iDRAC, and run the initial configuration wizard. Afterwards the rest of the configuration can be performed via SSH and/or the GUI.

While continuing with the configuration, I noticed I could not create an aggregate Ethernet interface. No LACP or Etherchannel! So what if my Ethernet interface goes down, for whatever reason?

Continue reading

Zerto facilitates IT resiliency with a single VM replication platform

Zerto IT ResiliencyBack at Storage Field Day 16 in Boston, Zerto presented their VM replication software. It’s a block level, continuous hypervisor based replication, using a journal to log I/O in a VCR-like fashion. This enables you to rewind to any point in time that’s covered in the journal, and recover your VMs to that exact state. Zerto’s plans are a bit grander than “just VM Replication” though… they aim to cover the complete IT Resiliency market.

Continue reading

Adding Data Domain storage capacity

Data Domain DD2500Increasing the storage capacity of a Data Domain is usually a matter of adding an additional disk shelf to the Data Domain head. Upon  connecting the additional enclosure your new disks will be in an Unknown state. This does not necessarily mean there’s anything wrong with the topology of the system, but just indicates the disks aren’t in use just yet. This is however easily fixed with a couple of CLI commands.

Continue reading

Enable DDBoost on a Data Domain replica

Data Domain DD2500Last week I’ve been implementing two new Data Domain systems for a new customer who’d like to use these systems as backup targets for their existing Veeam 8 environment. Backup would be replicated to the secondary system to guarantee recoverability even if the first system or data center experiences a catastrophic failure. In this case replication will be handled by the Data Domain system itself. You’d like your backup software to be aware of the replicas on the secondary location. This in turn means Veeam should be able to read from the replica, which turned out to be a bit of a configuration challenge. Bring out the CLI!

Continue reading

Avamar Exchange 2013 DAG backup fails – Waiting for VSS

Today I have been troubleshooting Avamar backups of an Exchange 2013 DAG (Database Availability Groups) setup. The Avamar job would start but no data would be backed up and the job would fail within 15 minutes. Initially VSS was suspected to be the culprit but additional troubleshooting revealed some strange behavior of the Avamar client in which the individual plugins would keep waiting for a status message that would never arrive.

The setup is simple: three Windows 2008 R2 SP1 servers, Exchange 2013 Update 5 with Database Availability Groups and Avamar 7.0.2 as the backup software and target. The error that pops up looks like this:

Avamar Exchange 2013 DAG error

Continue reading