save disk space on Synology using deduplication

Synology offers deduplication in certain models of their NAS, if you fulfill certain requirements like officially supported disks, RAID1 or 10, supported NAS with at least 16GB memory etc. There are scripts out there to mark your disks as supported and enable the official deduplication version even on unsupported devices and that would have been my preferred way of doing it on my SSD-driven DS620slim – but I am using Raid 5 and so even those scripts will not get this working on my configuration – RAID 5 is not supported for Synology deduplication. So, how to get the Synology NAS to have deduplication on BTRFS if you do not play according Synology rules?

What is bees?

This is where bees kicks in. Bees is a btrfs volume-level block-based deduplication, which works fully transparently to all things running on top of the file system. Bees is running consistently in the background, so there is no scheduled high performance impact after the first full deduplication run on the volume has finished. The only requirements for bees are enough memory to fit in the hash table – about 128MB per TB of unique data – and that the volume or data is not encrypted – as encrypted data cannot be deduplicated at all. Active Backup for Business data is already deduplicated in its storage folder if you did not enable that,

Attention has to be paid to one thing: Do never run any volume defragmentation when you are running bees. Defragmenting the volume copies blocks of data to realign them for reading single files, which is exactly the opposite of what deduplication is doing. There is a button for volume defragmentation at volume level in the Storage Manager on Synology – just don’t use it.

It took me a while to find a working solution of bees for my Synology NAS (x64 CPU – I don’t know if this approach would work with ARM CPUs at all…), but I found one and would like to share how to set it up. For each volume where you want to have bees deduplication running, you need to repeat the setup process – so it is volume-level deduplication and does not deduplicate across different volumes – the less volumes you have, the higher the deduplication ratio willbe. Be aware that data which is by default unique and compressed, cannot be deduplicated. So, if feasible (check your backup strategy), separate that type of data (video/audio/picture libraries) into another volume if it is a rather large amount to not waste disk performance, cpu cycles or system memory on trying to deduplicate that. Data being compressed on shared folders is not a good candidate for deduplication as compression on file level alters the whole file data and part-file deduplication about the same blocks being in different files is becoming more unlikely.

If you are already using snapshots on volume level or create volume level backups of your data, you should know that a block-level deduplication extensively changes your volumes – you will lose a lot of snapshot backups when deduplication runs for the first time and you also risk that the volume fills up due to the large amount of block-level changes. Before putting this in production, remove all snapshots from the volume to be deduplicated and disable snapshots until the full first cycle has finished. Turn it back on afterwards, and for backups done to external disks run a full backup afterwards. As I only implemented this on two Synology NAS systems I am currently managing so far and it is not an official Synology-supported solution, you might want to create a test volume and fill it with a certain amount of data to test your use cases as well as backup/restore and other services you might have running on top first – although I have not encountered a single issue during a full year running this by now. Don’t copy files within file manager for testing – this will just link to the existing file and deduplicate already on copying data there.

I do not take any responsibility/accountability for your data and it is your duty to have proper backups, too.

Mounting the volume for deduplication low-level access

• note down the volume number you want to run bees for (Storage Manager; I am using volume 1 for this example – if you are note using volume 1 you have to replace the numbers accordingly)
• install the Container Manager inside Package Center, if not already done
• as stated before disable snapshots and backups for the volume you want to run bees for, additionally recommended to remove all volume snapshots
• as my NAS is heavily used and I did not want to have any impact on bees, I decided to stop most of the services before running bees for the first time – but that is definitely not required
• enable either terminal or ssh via Control Panel\Connectivity\Terminal & SNMP and connect to the CLI
• create a directory for the btrfs volume to be mounted into, which bees can use to access it later
  • sudo mkdir /mnt_volume1
• now we mount the btrfs volume root to the mount point directory just created
  volume counter starts with 1, but cachedev counter starts with 0 – so the number used behind cachedev_ is one less than the volume number (volume 1 -> cachedev_0, volume 2 -> cachedev_1 ..) – some Synology NAS (like the DS-216 II+) are not showing cachedev entries but are using lvm volumegroups like vg1000-lv instead – just try and test those properly, too
  • sudo mount /dev/mapper/cachedev_0 /mnt_volume1 -o subvol=/
• check that you can see the @syno entry when running the command to list directory content
  • sudo ls -lsa /mnt_volume1

Initial disk space check

If you want to know how much free space you gained, you should of course check the free space first. I am doing this using an SSH connection and running the ‘df’ command as this delivers the free disk space in kiloByte – on the UI I can only see 0.1TB increments and inside the File Explorer 0.01TB increments. As I only run one single volume, the line showing ‘Mounted on’ with /volume one is the interesting one for me – 20.088 teraByte of free space right now.

Creating the bees on Synology deduplication container

• for the expected maximum volume size, calculate the required hash table size (more than needed does not help with performance 🙂 – more details can be found here
  • maximum volume size in TB * 128MB, if files are compressed (also inside!) *2, and if compressed with many text files, code files, database files use *4)
  • examples:
    • 36(TB) *128MB*2 = 9216MB = 9GB
    • 10(TB) *128MB*4 = 5120MB = 5GB
  • this is the amount of memory required for running bees, as the hashtable must fit into memory – so, if you need to increase your memory – do so first (note: my DS620slim officially supports 2x 4GB, but runs just fine with 2x 8GB)
  • as I am somewhere between *2 and *4 maybe, I just increase the value by 1GB for a little buffer
• create the docker container with all required parameters on CLI
  the container source is located at https://github.com/deatheibon/bees-docker
  • sudo docker run -d –privileged -e HASH_TABLE=/mnt_volume1/.beeshome/beeshash.dat -e HASH_TABLE_SIZE=10240M -e OPTIONS=-a -v /mnt_volume1:/mnt –restart unless-stopped –name bees_volume1 deatheibon/bees
  • this will download the image, create and start the container for bees

• open Container Manager, select Containers on the left and your bees container on the right, after you clicked on statistics you should see an increasing RAM usage (hashtable) and a proper CPU usage while bees is running the first deduplication cycle
• when the CPU usage decreased to 0% and the RAM Usage has been stable for some minutes this indicates, that the initial deduplication cycle has finished

adjusting container resources after the initial run completed

• now stop the container within Container Manager and open the settings pane for the container to limit resource usage to make bees a background process
  • enable resource limitation
  • set CPU Priority to low
  • adjust memory limit to the hash table size and add 128MB overhead for the executable
  • save and restart the container
  • check on the General tab after the container started to verify that your settings where taken over (takes a while for them to display after the container was started again)

Ensuring the volume is mounted on each subsequent reboot for deduplication

You remember we mounted the btrfs volume at the CLI manually? We must ensure that this happens at every reboot now automatically and we’ll use the task scheduler for this:

• open the Control Panel, section Services\Task Scheduler
• create a new triggered task with the following settings
  • User: root
  • Event: Boot-up
  • Command: mount /dev/mapper/cachedev_0 /mnt_volume1 -o subvol=/
    same like we used for the manual mount using the CLI just without sudo in front of it as we run it as root during system boot-up

If you want to check if your container is starting up properly on a reboot – just reboot your NAS. And for sake of demonstrating that it is working – check the free space available on the volume. As a final step, you might want to re-enable your volume snapshot schedules.

Checking the results

Now it is 20.253TB – so 169GB space saved – and the more data you add, the more you will save. As backup storage is already deduplicated itself, my savings don’t look crazy as I do not have much duplicate data at all – but I used that on corporate level for more than ten years and savings can get really significant…