Symmetric IT's Tech Blog

Life's too short for complicated solutions.

Macrium Image Consolidation & Synthetic Backups

Macrium Image Consolidation & Synthetic Backups

We will be looking at Macrium's Synthetic Backup and Consolidation functions to find out how the settings impact actual real world backups.

Synthetic backups refer to the creation of a full backup - instead of physically creating the backup of the source the software will artificially combine and consolidate incremental backups into a full backup. This will reduce the incremental chain length and give you a new baseline full image. Synthetic backups and consolidation have some advantages:

  • Saves Time - Synthetic backups are generally quicker to perform than an actual full backup*
  • Save Storage Space - Creating full synthetic backups reduce storage requirements (merging an incremental into a full will not increase the full by the exact size of the incremental)

*In most cases the time taken for a full synthetic backup is less than taking a manual full. The I/O requirements for a manual backup generally exceeds the requirements of a synthetic backup. Conversely we have observed some cases where the synthetic takes longer and almost as long as a manual full. It's worthwhile to monitor the merging times vs manual fulls to optimize your backup strategy.


For our first scenario we will be using Macrium to run a full backup image with 5 incremental backups. We will show the implications and considerations of using a fixed length incremental chain. The base image is taken of a server with a used hard drive size of 80GB. The resulting full backup on medium compression came to 51GB.

The settings we used to make backup are as in the screenshot below:

Notice the amount of incrementals to keep is set at 5. When the 6th backup is run the following happens: incremental 2 will be merged into incremental 1 forming a new incremental 1(but named 02-02 in explorer). A new 5th incremental will join the end of the chain. Running Backup 7 will merge incremental 3 into the newly formed incremental 1 (02-02), once again a new incremental added to the end.

Visually, this is what happens, illustrating the 6th and 7th incremental backup:

Here is what the backup folder would look like after the 6th backup and first consolidation:

Notice how the first incremental 01-01 is  gone and replaced by 02-02. The base image remains untouched. The first incremental (into which all others will merge) grows linearly according to each individual incremental size.

This is an effective backup plan when you need to have a reference base image with a set incremental chain length which would correspond to your required recovery targets (how far back would you need to be able to recover, in this case 5 days from the last backup or the first day of the original backup). Aspects to take note of:

  1. The base image remains anchored in time, its creation date does not change and remains a reference point if you need to restore data from the original image.
  2. The first incremental grows in linearly in proportion to the individual incremental sizes when they merge.
  3. At some stage the first incremental will exceed the size of the base image.
This what the folder looks like after the 7th backup:

Note that the first incremental (03-03) has now grown in size in proportion to the 2 incrementals that merged to form it.

A situation to consider is when the merged incremental nears the size of the base image. This situation is less than ideal because dealing with such a large incremental when doing a full restore becomes cumbersome and will increase the restore time.

At this stage you can consider doing a Synthetic Full backup. You can enable this by ticking the "Create a Synthetic Full If possible" option:

In this case the first incremental will merge into the base image. The synthetic full took only a few minutes, much faster than performing a normal full.

Here is a visual representation:

This is what the folder looks like afterwards:

The large (400mb) first incremental from before is merged into the base image freeing up disk space. The base image size only increases slightly (less than 400MB) due to data inside the image being overwritten and not appended.

As a result of the synthetic backup the base image date has now moved forward in time. Subsequent backups will keep moving the base image forward in time.

Differential Backups

Next we'll look at how differential backups affect the incremental queue.

We will keep the backup definition as is but add a differential at the end.

Next we run run 5 incrementals. This will merge all the initial 5 incrementals into the base image and create 5 new ones after the differential. The differential will now be directly after the base image:

Notable points:

  • The 5 new incrementals are now dependent on the differential
  • You will not be able to do a Synthetic Full anymore - the differential can't merge into the base images
  • All subsequent incrementals will be merged and consolidated into each other but not the differential
  • Differentials act as 'checkpoints' that can't be moved unless deleted (note that the incremental chain that's dependent on the differential will also be deleted)

In summary, Macrium's synthetic backups and consolidation gives you immense flexibility when designing your backup regime. Backup plans are often a balance between storage space and the required restore history so this may help in optimizing your strategy.

Extra Reading :

Macrium has a standalone consolidation application that can be used to merge backups independently. This useful when archiving backups or when managing backups away from the backup source.

Outlook Search Broken After Windows 10 1709 Update

Outlook Search Broken After Windows 10 1709 Update

This is an odd one but an easy fix at least. It presents as Outlook instant search not working anymore (ie, it manually searches through the mailbox folder). You can try rebuilding the index, changing the index location, creating a new outlook profile, repair install Office and even a different Windows profile. All will fail.

The  case I had was with Outlook 2007 on Windows 10 1709. More specifically when checking "Tools" then "Options" then "Search Options" you notice that the box under "Index messages in these data files" is empty/blank.

Outlook does find any any pst's to index or search. Upgrading to Office 2010 did resolve the problem but quite often that's not possible, it's not a satisfying solution either.

Adding the following registry key will sort it:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Microsoft\Windows\Windows Search\Preferences]

Copy and paste into a text file, then change the file extension to .reg. Then right click and select Merge.

Comparison of Macrium File Backups To Veeam Agent File Backups

Comparison of Macrium File Backups To Veeam Agent File Backups

We will be comparing the file based backup functionality of Macrium Reflect vs Veeam Agent. All the focus is normally on the image based backups but the file based backups can be useful in certain cases:

  • A NAS that needs to be backed up (can't do an image based backup)
  •  A file share that resides on a partition that also has other data in it that does not need to be backed up
  • Any situation that does not require the full disk to be recovered


Macrium and Veeam approach file backups differently than traditional backup programs. Traditionally files would be backed up individually to a backup source. The backup destination would contain the actual files as they are in the source side. The next backup would be incremental or differential, ie, only the changed files. This is where it becomes more complex - how do you handle the subsequent backups in the destination location? Some considerations:

  • Do you overwrite files in the destination? This would resemble a mirror like situation or sync.
  • Do you keep versions of changed files? How many versions? How do store these files? Do you rename them with a date to keep track? Do you put them in different folders identified by dates?
  • How long a history do you want to keep?
  • What if there is data loss? How hard is it to restore files from the destination back to the source and recreate the original directory structure?

Macrium and Veeam solve these problems by putting all backed up files into a container, similar to what their image based backups do. Subsequent backups, be it differential or incremental are put in containers as well forming a dependency chain. Once again same as the image based backups.

This simplifies matters immensely. The first backup will contain all your files, the next backups will contain only the changed files so will be much smaller. The key difference comes into play when you mount  or access one of the backup containers - it will show you ALL the files, ie, the full directory structure as it was in that point in time when the backup was done.

This resolves all the issues normally associated with file backups. Point in time restores become easy as well as full restores. Granular recovery of a single file in a specific point in time is trivial now.

Lets investigate Macrium's file backups first:


Firing up the backup wizard:

This is where you select the source and destination folders. For the first round of testing we will do a a run on an empty folder to discover the storage requirments and efficiency. Note Macrium gives you the "Advanced Options" as well :

Useful options, especially the compression settings (which we leave at default for the tests). 

Password options for protecting the backup archive and Auto Verify Image to ensure the created image is consistent.

Next we get some nice filtering options for the source folder:

Useful to exclude unneeded files based on a mask or wildcard.

Next you need to schedule the backup. Same as the image based side you can select different schemes with retention policies - very flexible. We will run manual test so we just leave all as is.

Next we look at Veeam's backup creation. The backup source selection is simple and clear:

Same as Macrium, Veeam can store on a local drive or network location:

Selecting the destination folder, there are also more options available:

Important to note the option to specify the retention policy. A simpler approach than Macrium but less flexible. The Advanced option gives you Encryption, Compression settings and storage target optimizations which we'll look at now:

Veeam can optimize the compression ratio and deduplication ratio based on target storage, LAN, WAN or LOCAL. Should be interesting to test in a separate review at some stage.

Scheduling is simpler than in Macrium:

An important feature again is the ability to schedule at an event, ie, when a network drive becomes available or when a USB drive is inserted. Some basic Cryptolocker protection is included by ejecting the media after the backup.

It would appear that Veeam only does incremental backups interspersed with full backups whereas Macrium gives you the option between differential and incremental.

First Backup - Macrium an Veeam

The first Macrium backup created this in the destination folder (empty source folder):

A single 691KB file which represents an empty folder backup. This appears to be very efficient

Veeam had the following in the destination folder (empty source folder) :

2 files, one container and 1 meta data file. Quite a difference in terms of storage requirements. 

Next we run a second backup of the empty source to compare the data generated. One would expect a small file or even none at all. Let's have a look:

As one would expect from a folder with no changes - the backup reports no changes and interrupts itself with no changes made.

Lets run Veeam for a second time on an empty folder:

Veeam runs regardless and creates a second container file of 30MB. This is not ideal as this means that every single time a backup job is run a 30MB file at minimum will be created regardless of what happened in your source folder. This is bad in terms of storage requirements.

Changed Files

Next we look at adding a small file to see how Veeam and Macrium determines if a file has changed. There are 2 basic method that backup applications normally use to determine if a file has changed:

  1. Time Stamps - Whenever you change a file the operating system alters the file's "Date Changed" time stamp. The backup application will compare the destination stamp with the source stamp and if different, backup the file.
  2. Archive Bit - A special flag set on each file by the operating system that gets set when a file has changed. When the file is backup up the flag gets unset. This logic can be used to determine what files have been altered since the last backup.

Adding a text file to each source directory, also check the archive bit is set:

Running the next Macrium incremental produces this in the destination:

An expected result, a second container that is slightly larger than the initial. Macrium left the archive bit untouched which means it uses only file date logic.

Next we run the next Veeam backup, it creates another 30MB container for this backup that only added a 0KB text file. Quite inefficient :

Veeam also ignores the archive bit so we can assume that it also uses file altered stamp logic.

Big Files

Next we look at how Macrium and Veeam handle big files. Quite often you need to backup large files such as an Outlook PST file that contains all your email. These files are clumsy because they are normally quite large and change often so can be a pain to backup efficiently.

For testing purposed we will put a 500MB random data file in each source folder. In an ideal world as these files change only the internal changes are backup up (delta backups) which would make them very efficient. This though adds a layer of complexity to keep track of the changed blocks and to reconstruct the file come restore day.

Macrium first, lets run an incremental backup with the large file added:

As expected the backup has grown by +- 500MB, so nice and predictable.

Next we run Veeam's backup with the large file:

As expected, a new container of 530MB. Once again pointing out the storage inefficiency.

Lets change some of the big files's internals to simulate a PST file that has changed. We will hex edit the big file and add some random characters. 

Now we run a Macrium incremental:

The new backup is just less than 1MB. One would have expected the entire 500MB file to have been backup up again but it appears only the internal file changes were backed up. This is very efficient.

Here is Veeam's next backup of the big file with some internal changes:

A new container of the same size as the previous was created. The same big file was just copied. This is what traditional backup applications normally do and is quite slow and inefficient. Consider when you have a 20GB PST file that needs to be backup up daily. The space considerations quickly become a limiting factor.

We can safely assume that Macrium does in file delta backups within its file backup regime.

Restoring Files

An important functionality is to be able to easily restore files from a backup container. Macrium follows the same principle as their image based backups. You can simply double click any backup file which will auto mount the image as a Windows Explorer drive. This makes it easy and intuitive to restore files because the user stays in a familiar environment (Windows Explorer). This makes it easy to browse backup files from any PC with Macrium installed.

Veeam has a custom backup browser from where you have to recover your files. You also can't double click on a backup container, you have to initiate the restore from within the main program itself. This makes it hard to explore the backup files in case the source PC has crashed.

Lets tabulate the key differences:

Incremental File Backups

Differential File Backups

Advanced Scheduling

Advanced Exclusion Masks

Verify Backups

Storage Friendly Backups

No Files Changed Logic

Advanced Cryptolocker Protection*

Network Backups

In File Delta Backups

Native Backup Image Mounting

Available in Free Version**

*Advanced Cryptolocker Protection - We consider Veeams 'eject mediaafter backup' as a  basic protection system. Macrium has a feature called "Image guardian" that protects all backup locations from unauthorised access which we consider to be superior.

**Available in Free Version - A key difference, Veeam gives you file backup functionality in their free version whereas Macrium only provides this functionality in the paid product.

Even though Macrium's file backups functionality is only available in their paid product (vs  Veeam in their free version) we still award Macrium as the clear winner. The efficiency in terms of storage and delta backup functionality puts it in a different class altogether. The easy restores is just a bonus. 

Feel free to add your opinion in our comments.

Data Recovery Costs and Backup Best Practices

Data Recovery Costs and Backup Best Practices

Investigating the costs involved when disaster strikes is an undoubted motivation for setting up an iron clad backup regime. Not only does it make you sleep easy at night it also saves you money and potentially your business. Consider that on a long enough timeline all digital storage devices fail, this includes traditional hard rives, solid state drives and portable flash memory.

How do they fail?

In our experience, different storage targets can fail in different ways:

Traditional hard drives are possibly more gracious in their failures. Spindle hard drives quite often give you warning signs that it's nearing its end of life.  These signals can be monitored via S.M.A.R.T data that the hard drive provides. Factors such as spin up errors, bad sectors and reallocated sector counts can be strong indicators that a hard drive needs to be replaced. At Symmetric IT we monitor hard drive S.M.A.R.T data as part of our Managed IT service. If a failure signal is detected a hard drive will be replaced as a matter of policy. Prevention is better than cure.

Often if a hard drive fails it still possible to recover data from it using 3rd party software. As long as the hard drive is still detectable by Windows you have a good probability to get your data back. The worst case scenario is a non responsive hard drive. Quite often data can still be recovered but this is where it moves into the highly qualified expertise area. The hard drive can still be recovered in a clean room using specialized methods. The cost to recover also rises significantly.

A recent quote obtained from a data recovery specialist to recover data from a laptop hard drive that was non responsive:

..Our fees, should you decide to proceed, would be quoted between $249 and $749

So you are looking at a significant amount if the hard drive fails.

Flash memory based storage devices, in our experience, fail less graciously. Solid state hard drives have a finite lifespan, determined by the amount of read and writes to the memory. More often than not they will fail before reaching this theoretical read/write limit. All the SSD failures we have encountered (across different brands) have been fairly sudden with little to no warning. Windows would glitch and then data would be lost, suddenly Windows wont boot or Windows would blue screen unexpectedly with different stop codes. Attempting to read data from the drive would fail.

In these cases data recovery is harder than on a traditional hard drive. Backups, more specifically image based backups of solid state drives is imperative.

USB flash drives also fail, and most we have come across fail suddenly with no warning at all. One day it will be OK, the next time you plug it in it's completely unresponsive.

Consider the following price layout from another data recovery company:

USB flash drives
$125 - 295 + GST
Phone recoveries

$199 - 399 + GST

Hard disk

$350 - $595 + GST

Specialized recoveries

$500 - $1500 + GST

These prices reflect a successful recovery which means there are other aspects to consider as well:

  • What about the data that they can't recover?
  • How much time will you lose recovering this data?
  • What impact will this expenditure and loss of time have on your business?
  • What impact will a loss of data have? (Financial data, Business databases)

Data and information is the lifeblood of a business today.

How do I protect myself or my business?

A structured, monitored backup regime is needed. A simple backup every now and then is not enough. If the day comes a recent backup makes all the difference. Usually the more recent the better, every hour wasted redoing work is time and money lost.

A long history of backups is also important. All the difference can sometimes be made when recovering that one email or spreadsheet you had 5 years ago.

Symmetric IT prides itself on tried and tested backup regimes that can accommodate all disaster recovery scenarios and meet your time to recovery targets (RTO). Having a backup is not enough if it takes 2 days to get up and running again.

Your backup needs 2 or more storage targets onsite, always separate, rotated and securely stored. These backups need to be secure(encrypted) and monitored. You also need secure offsite backups for critical data. Special care needs to be taken to cater for all line of business applications so that they are backed up correctly in a way that allows for a successful restore. A successful backup is one that restores.

Your backups also need to have full and granular restorability. For example, if you need a specific file from a year ago it needs to be easily accessible. If your computer or server fails it needs to be recoverable in its entirety within set time limits; not just the latest version of the server either, quite possibly last week's or even last year.

Your backups also need to be tested, they are only as good as the last time they were tested. Symmetric IT has seen backups that run and verify successfully but when the time came to restore they failed.

In summary, your backups need the following considerations:

Multiple onsite storage targets

Offsite online encrypted backups

Line of business application considerations

Recovery Time Objective (RTO) considerations

Encrypted and stored safely

Rotated to a set schedule

Actively monitored

Sector and file based considerations for backup types

Restorability considerations (full/granular)

Restorability testing

Documented procedures

These are all factors to consider when setting up a backup plan, Symmetric IT can tailor a backup plan for you keep your most valuable digital assets safe.

EDIT: The government has a nice piece on backups and their importance:

Backing up data and devices

If you feel that this is relevant to your business then get in touch with Symmetric IT to align yourself with these best practices.