Symmetric IT's Tech Blog


Life's too short for complicated solutions.

Macrium Image Consolidation & Synthetic Backups

Macrium Image Consolidation & Synthetic Backups

We will be looking at Macrium's Synthetic Backup and Consolidation functions to find out how the settings impact actual real world backups.

Synthetic backups refer to the creation of a full backup - instead of physically creating the backup of the source the software will artificially combine and consolidate incremental backups into a full backup. This will reduce the incremental chain length and give you a new baseline full image. Synthetic backups and consolidation have some advantages:

  • Saves Time - Synthetic backups are generally quicker to perform than an actual full backup*
  • Save Storage Space - Creating full synthetic backups reduce storage requirements (merging an incremental into a full will not increase the full by the exact size of the incremental)

*In most cases the time taken for a full synthetic backup is less than taking a manual full. The I/O requirements for a manual backup generally exceeds the requirements of a synthetic backup. Conversely we have observed some cases where the synthetic takes longer and almost as long as a manual full. It's worthwhile to monitor the merging times vs manual fulls to optimize your backup strategy.

PRACTICAL CONSOLIDATION TESTS

For our first scenario we will be using Macrium to run a full backup image with 5 incremental backups. We will show the implications and considerations of using a fixed length incremental chain. The base image is taken of a server with a used hard drive size of 80GB. The resulting full backup on medium compression came to 51GB.


The settings we used to make backup are as in the screenshot below:


Notice the amount of incrementals to keep is set at 5. When the 6th backup is run the following happens: incremental 2 will be merged into incremental 1 forming a new incremental 1(but named 02-02 in explorer). A new 5th incremental will join the end of the chain. Running Backup 7 will merge incremental 3 into the newly formed incremental 1 (02-02), once again a new incremental added to the end.

Visually, this is what happens, illustrating the 6th and 7th incremental backup:


Here is what the backup folder would look like after the 6th backup and first consolidation:


Notice how the first incremental 01-01 is  gone and replaced by 02-02. The base image remains untouched. The first incremental (into which all others will merge) grows linearly according to each individual incremental size.

This is an effective backup plan when you need to have a reference base image with a set incremental chain length which would correspond to your required recovery targets (how far back would you need to be able to recover, in this case 5 days from the last backup or the first day of the original backup). Aspects to take note of:

  1. The base image remains anchored in time, its creation date does not change and remains a reference point if you need to restore data from the original image.
  2. The first incremental grows in linearly in proportion to the individual incremental sizes when they merge.
  3. At some stage the first incremental will exceed the size of the base image.
This what the folder looks like after the 7th backup:


Note that the first incremental (03-03) has now grown in size in proportion to the 2 incrementals that merged to form it.

A situation to consider is when the merged incremental nears the size of the base image. This situation is less than ideal because dealing with such a large incremental when doing a full restore becomes cumbersome and will increase the restore time.

At this stage you can consider doing a Synthetic Full backup. You can enable this by ticking the "Create a Synthetic Full If possible" option:

In this case the first incremental will merge into the base image. The synthetic full took only a few minutes, much faster than performing a normal full.

Here is a visual representation:


This is what the folder looks like afterwards:



The large (400mb) first incremental from before is merged into the base image freeing up disk space. The base image size only increases slightly (less than 400MB) due to data inside the image being overwritten and not appended.

As a result of the synthetic backup the base image date has now moved forward in time. Subsequent backups will keep moving the base image forward in time.

Differential Backups

Next we'll look at how differential backups affect the incremental queue.

We will keep the backup definition as is but add a differential at the end.

Next we run run 5 incrementals. This will merge all the initial 5 incrementals into the base image and create 5 new ones after the differential. The differential will now be directly after the base image:

Notable points:

  • The 5 new incrementals are now dependent on the differential
  • You will not be able to do a Synthetic Full anymore - the differential can't merge into the base images
  • All subsequent incrementals will be merged and consolidated into each other but not the differential
  • Differentials act as 'checkpoints' that can't be moved unless deleted (note that the incremental chain that's dependent on the differential will also be deleted)

In summary, Macrium's synthetic backups and consolidation gives you immense flexibility when designing your backup regime. Backup plans are often a balance between storage space and the required restore history so this may help in optimizing your strategy.

Extra Reading :

Macrium has a standalone consolidation application that can be used to merge backups independently. This useful when archiving backups or when managing backups away from the backup source.

https://knowledgebase.macrium.com/display/KNOW/Standalone+backup+set+consolidation


Outlook Search Broken After Windows 10 1709 Update

Outlook Search Broken After Windows 10 1709 Update

This is an odd one but an easy fix at least. It presents as Outlook instant search not working anymore (ie, it manually searches through the mailbox folder). You can try rebuilding the index, changing the index location, creating a new outlook profile, repair install Office and even a different Windows profile. All will fail.

The  case I had was with Outlook 2007 on Windows 10 1709. More specifically when checking "Tools" then "Options" then "Search Options" you notice that the box under "Index messages in these data files" is empty/blank.


Outlook does find any any pst's to index or search. Upgrading to Office 2010 did resolve the problem but quite often that's not possible, it's not a satisfying solution either.

Adding the following registry key will sort it:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Microsoft\Windows\Windows Search\Preferences]
"{4154494E-BFF9-01B8-00AA-0037D96E0000}"=dword:00000001
"{C0A19454-7F29-1B10-A587-08002B2A2517}"=dword:00000001
"{70fab278-f7af-cd11-9bc8-00aa002fc45a}"=dword:00000001
"{c34f5c97-eb05-bb4b-b199-2a7570ec7cf9}"=dword:00000001
"{0077B49E-E474-CE11-8C5E-00AA004254E2}"=dword:00000001

Copy and paste into a text file, then change the file extension to .reg. Then right click and select Merge.

Comparison of Macrium File Backups To Veeam Agent File Backups

Comparison of Macrium File Backups To Veeam Agent File Backups

We will be comparing the file based backup functionality of Macrium Reflect vs Veeam Agent. All the focus is normally on the image based backups but the file based backups can be useful in certain cases:

  • A NAS that needs to be backed up (can't do an image based backup)
  •  A file share that resides on a partition that also has other data in it that does not need to be backed up
  • Any situation that does not require the full disk to be recovered


HOW

Macrium and Veeam approach file backups differently than traditional backup programs. Traditionally files would be backed up individually to a backup source. The backup destination would contain the actual files as they are in the source side. The next backup would be incremental or differential, ie, only the changed files. This is where it becomes more complex - how do you handle the subsequent backups in the destination location? Some considerations:

  • Do you overwrite files in the destination? This would resemble a mirror like situation or sync.
  • Do you keep versions of changed files? How many versions? How do store these files? Do you rename them with a date to keep track? Do you put them in different folders identified by dates?
  • How long a history do you want to keep?
  • What if there is data loss? How hard is it to restore files from the destination back to the source and recreate the original directory structure?


Macrium and Veeam solve these problems by putting all backed up files into a container, similar to what their image based backups do. Subsequent backups, be it differential or incremental are put in containers as well forming a dependency chain. Once again same as the image based backups.

This simplifies matters immensely. The first backup will contain all your files, the next backups will contain only the changed files so will be much smaller. The key difference comes into play when you mount  or access one of the backup containers - it will show you ALL the files, ie, the full directory structure as it was in that point in time when the backup was done.

This resolves all the issues normally associated with file backups. Point in time restores become easy as well as full restores. Granular recovery of a single file in a specific point in time is trivial now.

Lets investigate Macrium's file backups first:


MACRIUM

Firing up the backup wizard:


This is where you select the source and destination folders. For the first round of testing we will do a a run on an empty folder to discover the storage requirments and efficiency. Note Macrium gives you the "Advanced Options" as well :

Useful options, especially the compression settings (which we leave at default for the tests). 

Password options for protecting the backup archive and Auto Verify Image to ensure the created image is consistent.

Next we get some nice filtering options for the source folder:

Useful to exclude unneeded files based on a mask or wildcard.

Next you need to schedule the backup. Same as the image based side you can select different schemes with retention policies - very flexible. We will run manual test so we just leave all as is.


Next we look at Veeam's backup creation. The backup source selection is simple and clear:



Same as Macrium, Veeam can store on a local drive or network location:



Selecting the destination folder, there are also more options available:

Important to note the option to specify the retention policy. A simpler approach than Macrium but less flexible. The Advanced option gives you Encryption, Compression settings and storage target optimizations which we'll look at now:

Veeam can optimize the compression ratio and deduplication ratio based on target storage, LAN, WAN or LOCAL. Should be interesting to test in a separate review at some stage.


Scheduling is simpler than in Macrium:


An important feature again is the ability to schedule at an event, ie, when a network drive becomes available or when a USB drive is inserted. Some basic Cryptolocker protection is included by ejecting the media after the backup.

It would appear that Veeam only does incremental backups interspersed with full backups whereas Macrium gives you the option between differential and incremental.


First Backup - Macrium an Veeam

The first Macrium backup created this in the destination folder (empty source folder):


A single 691KB file which represents an empty folder backup. This appears to be very efficient


Veeam had the following in the destination folder (empty source folder) :

2 files, one container and 1 meta data file. Quite a difference in terms of storage requirements. 


Next we run a second backup of the empty source to compare the data generated. One would expect a small file or even none at all. Let's have a look:


As one would expect from a folder with no changes - the backup reports no changes and interrupts itself with no changes made.


Lets run Veeam for a second time on an empty folder:



Veeam runs regardless and creates a second container file of 30MB. This is not ideal as this means that every single time a backup job is run a 30MB file at minimum will be created regardless of what happened in your source folder. This is bad in terms of storage requirements.


Changed Files

Next we look at adding a small file to see how Veeam and Macrium determines if a file has changed. There are 2 basic method that backup applications normally use to determine if a file has changed:

  1. Time Stamps - Whenever you change a file the operating system alters the file's "Date Changed" time stamp. The backup application will compare the destination stamp with the source stamp and if different, backup the file.
  2. Archive Bit - A special flag set on each file by the operating system that gets set when a file has changed. When the file is backup up the flag gets unset. This logic can be used to determine what files have been altered since the last backup.

Adding a text file to each source directory, also check the archive bit is set:


Running the next Macrium incremental produces this in the destination:


An expected result, a second container that is slightly larger than the initial. Macrium left the archive bit untouched which means it uses only file date logic.


Next we run the next Veeam backup, it creates another 30MB container for this backup that only added a 0KB text file. Quite inefficient :

Veeam also ignores the archive bit so we can assume that it also uses file altered stamp logic.


Big Files


Next we look at how Macrium and Veeam handle big files. Quite often you need to backup large files such as an Outlook PST file that contains all your email. These files are clumsy because they are normally quite large and change often so can be a pain to backup efficiently.

For testing purposed we will put a 500MB random data file in each source folder. In an ideal world as these files change only the internal changes are backup up (delta backups) which would make them very efficient. This though adds a layer of complexity to keep track of the changed blocks and to reconstruct the file come restore day.

Macrium first, lets run an incremental backup with the large file added:


As expected the backup has grown by +- 500MB, so nice and predictable.


Next we run Veeam's backup with the large file:


As expected, a new container of 530MB. Once again pointing out the storage inefficiency.


Lets change some of the big files's internals to simulate a PST file that has changed. We will hex edit the big file and add some random characters. 

Now we run a Macrium incremental:


The new backup is just less than 1MB. One would have expected the entire 500MB file to have been backup up again but it appears only the internal file changes were backed up. This is very efficient.


Here is Veeam's next backup of the big file with some internal changes:

A new container of the same size as the previous was created. The same big file was just copied. This is what traditional backup applications normally do and is quite slow and inefficient. Consider when you have a 20GB PST file that needs to be backup up daily. The space considerations quickly become a limiting factor.

We can safely assume that Macrium does in file delta backups within its file backup regime.


Restoring Files

An important functionality is to be able to easily restore files from a backup container. Macrium follows the same principle as their image based backups. You can simply double click any backup file which will auto mount the image as a Windows Explorer drive. This makes it easy and intuitive to restore files because the user stays in a familiar environment (Windows Explorer). This makes it easy to browse backup files from any PC with Macrium installed.


Veeam has a custom backup browser from where you have to recover your files. You also can't double click on a backup container, you have to initiate the restore from within the main program itself. This makes it hard to explore the backup files in case the source PC has crashed.



Lets tabulate the key differences:



MACRIUM
VEEAM
Incremental File Backups



Differential File Backups



Advanced Scheduling



Advanced Exclusion Masks



Verify Backups



Storage Friendly Backups



No Files Changed Logic



Advanced Cryptolocker Protection*



Network Backups



In File Delta Backups



Native Backup Image Mounting



Available in Free Version**







*Advanced Cryptolocker Protection - We consider Veeams 'eject mediaafter backup' as a  basic protection system. Macrium has a feature called "Image guardian" that protects all backup locations from unauthorised access which we consider to be superior.

**Available in Free Version - A key difference, Veeam gives you file backup functionality in their free version whereas Macrium only provides this functionality in the paid product.


Even though Macrium's file backups functionality is only available in their paid product (vs  Veeam in their free version) we still award Macrium as the clear winner. The efficiency in terms of storage and delta backup functionality puts it in a different class altogether. The easy restores is just a bonus. 


Feel free to add your opinion in our comments.



Windows 10 Free Upgrade Still Active (April 2018)

Windows 10 Free Upgrade Still Active (April 2018)

You can still upgrade to Windows 10 for free in case you have missed all the other opportunities. The last chance to upgrade was end of December when the free upgrade to users with assistive technologies ran out, or this was the theory.

You can still download the Windows 10 ISO, mount or extract it and install Windows from there. You can use your existing Windows 7 or 8 product key to install and activate Windows. You can do an in place upgrade and keep all your settings or a fresh install.

Download the ISO here:

https://www.microsoft.com/en-us/software-download/windows10ISO

Microsoft has made one ISO for both Home and Professional so that makes it nice and simple.

UPDATE

This no longer seems to work. From our testing using a Win7 key is now rejected. If you have any other means please comment.