VM Hosting Backups and Restores

VM Hosting Backups

By default, all VM Hosting virtual machines receive automatic daily backups to the UBSi – Commvault environment.  These backups are retained for 31 days and are replicated between the University Park and Hershey campuses.  Due to technical constraints, these backup parameters are not individually configurable.

When you create a tenant for a VM Hosting UMG ending in “-vmh” or “-vmhs”, the “Create a New Tenant” workflow will attempt to automatically find any associated VMs and import their existing backups into the tenant.  If your UMG is associated with VM Hosting VMs but does not follow this naming convention, please submit a ticket and we will manually import the VMs for you.

Once your VMs have been imported into your tenant, you will be able to view them by navigating to “Protect” -> “Virtualization” in the left-hand menu.  You can view the backup status for the VM, including the date of the most recent backup.

Clicking on the name of a VM will show additional detail about the VM, including a calendar view of the available “Recovery points”.

Starting a backup manually

Backups occur automatically each night.  However, in some cases you may wish to perform a manual backup (for example, before performing an upgrade or other major change to the VM).

  1. Make sure that you have the correct tenant (“Company”) selected in the upper right corner of the screen.
  2. Navigate to “Protect” -> “Virtualization” in the left-hand menu.
  3. Click on the name of the VM you wish to backup.
  4. Click on the “Back up” link near the upper right of the screen (across from the VM’s name)
  5. Select the type of backup:
    “Incremental” is recommended in most cases.  This will backup only the new changes since the previous backup, and is therefore the fastest to complete.
    “Full” may be helpful in rare cases.  This will backup all of the VM’s data again.  This may take a very long time to complete for a large VM.
    “Synthetic Full” will combine the previous Full and Incremental(s) on the backup server, creating a new virtual ‘Full’.  This DOES NOT read any new data from the VM.  Synthetic Fulls are generated automatically to allow the system to expire outdated backups in a timely manner.
  6. Click “OK” to begin the backup.

Restoring files from a VM backup

You can browse inside VM backup images to extract and restore individual files.  This means that in most cases you do not need to set up an additional file-level backup of your systems; the VM-level backups can serve both roles.

However, our UBSi servers don’t have access to log into your VMs.  So in order to restore files to a system, you must first install the filesystem client, to provide a communication path for the restore to use.  You can install the client in “restore-only” mode if you don’t need file-level backups.  See here for restore-only client installation instructions.

  1. Make sure that you have the correct tenant (“Company”) selected in the upper right corner of the screen.
  2. Navigate to “Protect” -> “Virtualization” in the left-hand menu.
  3. Click on the name of the VM you wish to backup.
  4. Under “Recovery points”, select the date and time of the backup you would like to browse, and click the “RESTORE” button.
  5. Select the “Guest files” restore type.
  6. Wait for the system to process the VM and return a list of files.  This can take some time for a large VM.
  7. Browse to the file(s) you wish to restore, and check the corresponding box(es).
  8. Proceed with one of the options below:
A. Download option

If you want to download a copy of the file to your own computer, click the “Download” button.  The system will begin locating and recalling the requested data.  One the recall is complete your browser will download the file automatically.  If you select multiple files, the download will be a ZIP file.

(Please note that we’re aware of an issue that sometimes prevents downloads of multiple files to fail.  If you experience this issue, try downloading the files one at a time, or use option B.)

B. Restore option

Otherwise, click the “Restore” button to restore the files directly to one of your systems.

Select the desired “Destination client”.  This MUST be a system where you have installed the Commvault filesystem client.

Our UBSi servers are not able to connect directly into your VM’s operating system.  If you are prompted to enter “Virtual machine login” details, you have selected the wrong type of client.  DO NOT enter your operating system credentials at this prompt; they will not work.  Instead, change the “Destination client” to a different entry that does not prompt for these credentials.

Set the “Destination path” by clicking the “Browse” button and browsing to the desired restore directory.

Click the “Submit” button to start the restore job.

Restoring an entire VM

We do not currently provide the option to perform self-service restores of an entire VM.  Please submit a ticket to the VM Hosting staff to request a full VM restore, as you have have in the past.

More details about backups

All VM backups are scheduled to occur daily beginning at 10pm.  Several backup jobs start simultaneously, and each job processes a large number of VMs over time, automatically scheduling the start times and resources for each individual VM backup to optimize the use of the available backup hardware.  As a result, the exact start time for a particular VM might from day to day.

VMs at University Park are backed up to our backup cluster in the University Park Data Center Building, and the data is then replicated to our backup cluster in the University Technology Center Building at Hershey.  VMs at Hershey do the opposite – the data is first backed up to UTC at Hershey, and then replicated to UPDC at University Park.  Replication is scheduled automatically and may begin as soon as 15 minutes after the data is received.  However, our nightly backup jobs ingest a significant amount of data, so it takes some time to complete the replication.  Our goal is to complete replication of all data within a maximum of 24 hours.

Backups are performed in “Crash-consistent” mode.  A full snapshot of the VM is taken automatically, the backup data is collected from that snapshot, and then the snapshot is deleted.  This ensures that the backup captures a single point in time across all files on all disks.  This ensures that the backup is as consistent as possible, even if a file was in the middle of being changed or moved during the backup.

However, VM backups do not capture information stored in the VM’s volatile memory (RAM).  Therefore, a restored VM backup will start out powered off, and will boot up as if it had crashed at the exact second the snapshot was taken.  For many applications this will be sufficient to avoid any significant data loss.  But for more sensitive applications it may be necessary to perform additional application-specific backup procedures to ensure that “in-flight” data is captured appropriately from memory or other sources.

Where possible, backups will use Changed Block Tracking to improve performance.  Changed Block Tracking can’t be enabled for VMs with existing snapshots, so backups for those VMs may be slower than normal.

Objects excluded from backup

All RecoverPoint Target VMs are excluded from backups.  These VMs have a name beginning with “rp.” and ending with “.shadow”.  The Target VM is a duplicate of the Source VM, so there is no value in backing up both copies.

VMs with independent-mode disks are excluded from backups. Snapshots cannot be performed on VMs with independent-mode disks, so it is not possible for us to perform VM-level backups of these systems. This is a rarely-used feature, so it will affect a very small number of VMs. Some clustered applications may require that you share a set of virtual disks between multiple VMs, which may require independent-mode disks. If your VMs must use shared independent-mode disks, you should install the filesystem client inside the VM(s) and perform file-level backups of any important data. If this applies to one of your VMs, you will see its backup jobs failing with the error “Virtual machine [example] is configured with shared virtual disks. Snapshot operations are not supported for shared disks. Use an in-guest agent to protect virtual machines with shared disks.”

A small number of other VMs – mostly internal systems – are manually excluded from backup for technical or business reasons.  VM Hosting staff will notify you if your VM needs to be excluded from backup.

Other backup categories

Enhanced Ransomware Resistance

Backups to UBSi-Commvault using our default retention plans should provide adequate protection against a limited ransomware attack that affects a an individual department.

However, for some systems, Penn State also needs to consider the possibility of a wide-scale attack in which administrative accounts or the UBSi-Commvault service itself could be compromised. While this is very unlikely, some systems are so critical to Penn State that they warrant additional protection.

A small number of VMs, identified as extremely business-critical, are covered by a more expensive backup policy designed to provide better resistance against a ransomware attack that affects the UBSi infrastructure.  Users are not able to select this policy themselves.  If you feel your systems are critical enough to justify the extra expense to protect against this unlikely scenario, you may wish to discuss this possibility with Infrastructure division leadership.