Rclone

Rclone is a command line tool that runs on every major OS, including Windows, it’s exclusions are configurable in files either as globs or as regexes and supports multiple backup modes.

💡

rclone can installed via winget (winget install rclone.rclone)

The exclusions can be configured by adding globs or regexes that match the excluded folders in a configuration file (for example:exclusions.config)

💡

Note that exclusions are relative to the backup source, so to exclude directory D:\temp when the backup source is D:\ , temp\** need to be added to the exclusions file.

After configuring exclusions, rclone’s behaviour can by tested by dry-running it:

\>./rclone ls --dry-run D:\ --exclude-from exclusions.config > logs.txt

The files that would be backed-up are logged in logs.txt . By repeating this flow, the exclusions can be finetuned.

In order to create an actual backup, rclone can be executed with the sync command that copies new and modified files and removes deleted files from the backup target.

\>./rclone sync D:\ S:\backups\mypc --exclude-from exclusions.config

By default rclone will ignore any excluded files even if they exist on destination. The --delete-excluded option can be used to deleted the excluded files from destination (useful if a new exclusions has been added).

Another interesting option is the --backup-dir which can specify a directory in destination to move files that would be deleted by the sync operation (and thus keep.

Rclone seems to be pretty fast also, in my case, my backup consists of about ~250K files (without the exclusions) that range from small source-code files to 20GB video files) and rclone needs about 30seconds to check if any actions are needed, when the --delete-excluded flag is not present. Of course, as it is expected, copying the files takes considerably more time depending on the network and disk speed.

My Setup

In my Homelab, I run a primary windows server that hosts all my services and also serves as the primary backup target. This server lives on 10.0.0.20 and one of the services is a local notification server named NTFY. In my script use NTFY to provide feedback about the backup operation


# Find absolute path to exclusions based on the bath of the script file
$scriptPath = Split-Path -Parent -Path $MyInvocation.MyCommand.Definition
$exclusionsFile = Join-Path -Path $scriptPath -ChildPath "exclusions.config"

# Define the log directory path & the log file path
$logDirectory = "D:\temp\rclone-logs"
$logFile = "$logDirectory\rclone-$(Get-Date -Format 'yyyy.MM.dd-HH.mm.ss').log"

$syncSource = "D:\"
$syncDestination = "\\10.0.0.20\Storage\backups\bugs\"

# Create the log directory if it doesn't exist
if (!(Test-Path $logDirectory)) {
    New-Item -Path $logDirectory -ItemType Directory
}

Write-Host "Exclusions file: $exclusionsFile"
write-Host "Log File: $logFile `n"  

rclone sync $syncSource $syncDestination --exclude-from $exclusionsFile --log-file $logFile -v -P 

write-Host "Log File: $logFile `n"  

# Read the log file content & filter out "NOTICE: and "INFO  :" lines.
$filteredLog = (Get-Content $logFile) | Where-Object { $_ -notmatch 'NOTICE:' -and $_ -notmatch 'INFO  :' }

# Create the message content
$messageContent = @{
    topic = "homelab"
    title = "Sarmis' Desktop Backup ($syncDestination)"
    message = ($filteredLog -join "`n") + "`nLog File Path: $logFile"
}

# Send the message to the ntfy server
Invoke-RestMethod -Uri "http://10.0.0.20:82" -Method Post -ContentType "application/json" -Body ($messageContent | ConvertTo-Json)

And this is a typical notification:

Welcome rclone

Rclone

My Setup