Several of our Data Domains are end-of-life and need to be replaced with new hardware. In most of the cases it’s a small site with a small Data Domain that only holds roughly 1 month of backups. In these cases we just install a new Data Domain next to it, reconfigure our our backup software, and that’s it. After a month, the old backups have expired and you can switch off the old Data Domain.

For the slightly larger sites, there’s more than one backup client/server writing to the Data Domain. There are Oracle RMAN backups, SQL dumps, etc. Plus the retention of backups on the Data Domain is much, much longer. In these cases you want to perform a proper Data Domain migration which retains the name and IP address of the old Data Domain, so you don’t have to touch all the clients. Here’s how you do that, and a DDBoost gotcha you should be aware of!

Preparing for the Data Domain migration

The first part is straightforward: install your new Data Domain just like any other system. If your original system is called system01, install the new system as system02. While you’re at it, also reserve a DNS name and some IP addresses for a system03, which you will need during the actual migration itself.

Once the new Data Domain is up and running, you need to copy the data from the old to the new system. You’ve got roughly two easy options to do this:

• A collection replication, which copies the whole Data Domain over to the new system with a single replication. This is the fastest method, both in terms of throughput and configuration. There are a number of caveats in this approach. For one, your DD-OS version should be very close, e.g. 6.1 to 6.1. If your old system is on 6.0 and the new one on 6.1, this is not going to fly. I actually shot myself in the foot with this: our old system was on 6.0, and I upgraded the new DD6300 to 6.1 as part of the installation. “Better make sure it’s current, so we don’t have to upgrade it later”. Woopsie! Fortunately, there’s an alternative, which is…
• MTree replications. This is the approach we took after ruling out the collection replication option. It replicates Mtrees over, which are basically top-level folders on the Data Domain. This does mean you will need to create an individual replication for each MTree on your Data Domain. We only had three Mtrees, so it only took a few minutes more…

You can limit the network throughput that’s used by these replications. In fact, the system we were replacing was already replicating its data to a central unit in an EMEA datacenter. The old Data Domain was configured with a system-wide replication bandwidth throttle of 20Mbit/s. Since it’s a system wide limit, this would interfere with the intra-datacenter replication to the new system. Fortunately, you can overrule this limit by specifying a more specific limit for a single DNS name. In our case, the system wide limit was still 20Mbit/s (so we wouldn’t overload the WAN), with an extra unlimited cap for system02:

replication throttle add destination <host> <sched-spec> <rate>

Lastly, make sure all the settings are copied to the new system as well. That’s: users, SMB shares, NFS exports and DDBoost storage containers. Don’t bother adding the new Data Domain to your Active Directory just yet; its identity will change during the migration, which will void the connection to the AD.

Once everything is in sync, you can move to the actual migration.

The actual Data Domain migration

Once you are ready to cut over to your new system, you need to make sure your old Data Domain is read-only. I did this by disabling all front-end protocols: stop the SMB, NFS and DDBoost services via the GUI.

Next, let the replication run one last time to make sure you copy the last bits of data over. Once that’s in sync, failover your MTree replications to the destination system; this makes them R/W accessible. At this point it also helps to remove your Mtree replications, so the replication database is cleaned up. You could add a step where you try to access your SMB shares on the new system, but we proceeded straight to the identity swap.

Change the system name and IP address of the old system (system01) to the temporary identity, system03. Next, change the identity of your new Data Domain (system02) to the original name/IP addresses: system01. Then change the temporary name of the old system (now system03), to system02. Voila, you’ve swapped identities of the Data Domains!

If your Data Domain uses Active Directory authentication, this is the time to enable it. Next, test your connectivity. If all goes well, you should now be able to access your data again using the old DNS names.

Watch out for DDBoost Storage Units!

One thing to watch out for during this type of Data Domain migration is the DDBoost Storage Units. Using the procedure in this blog post, I prepared all the DDBoost Storage Units and permissions on the replication system beforehand. After the identity swap however, we ran into the issue that we could read from the new Data Domain via DDBoost, but could not write to it. Oracle RMAN would throw an error: sbtbackup: Could not create file <filename> on host <systemname>, error 5034. NetBackup would give a similar kind of error.

It turns out this is related to DDBoost user permissions somehow going belly up during the identity swap. The fix is simple: just remove and add the usernames again, using the following commands:

ddboost storage-unit modify <ddboostSU> user sysadmin
(this changes the DDBoost user to the default sysadmin. You will get a warning that this is not recommended, but just ignore it.)

ddboost storage-unit modify <ddboostSU> user <ddboostuser>
(this changes it back to your proper DDBoost user, which shouldn’t have GUI/CLI login rights)

This solved our problem: RMAN and NetBackup could once again write to the DDBoost storage unit again.

My thoughts on Data Domain migrations

For the really small systems with one backup client and a short backup retention, I wouldn’t really bother with this procedure. Just install a new Data Domain, point your backup software to it, and wait for the old backups to expire.

Similarly, for the really big enterprise class Data Domains (DD990/DD9800 etc), there’s an official headswap procedure. This does require a downtime of several hours and some Dell EMC (Partner) Professional Services, but allows you to transfer your TB/PB’s of data over the internal SAS back-end.

For the systems in the middle though, where you cannot perform a headswap and also don’t want to reconfigure a large number of backup clients to the new destination, this procedure should do the job. Just make sure you write out all your steps and have a second set of eyes to check your steps. Good luck!