Media And Data Management

Last couple of years, having two children, I am facing the problem of both having more photos and videos to manage (and acquired by different devices) and less time infront of my desktop pc (But I’m lucky to be able to work on personal tasks in my work office, at least off hours).

Hardware Setup

My solution is to keep my data (the “live” version) on an external SSD disk. This allows me both to be able to access my personal files wherever I am (home, work, or on my laptop) and also provides a simple way to keep my ‘remote’ backup, located at my work office, easily updated. I also keep a backup at home - that I also use as a read-only version of my data.

At office, I work directly on the external SSD, that way I’m sure that at least my most important projects are on that SSD and thus included in my backup - of course everything work related is also on the office’s GitLab server. Then I use an internal, encrypted disk and SyncBack to maintain my backup - I usually update the backup just before leaving the office - or early in the morning when I’m expecting that it will need some time.

At home I use the HP Proliant Microserver Gen 10 (pictured above) as a home server for storage and media streaming with emby media server. It ’s using a 64GB SSD for OS, 2x4TB (mirrored) for Data, and 1x2TB for donloaded media, it has space for another HDD and quite possible I will update them in the future with a redundant array of 4 larger HDDs. This server is located in the living room, next to the TV.

Software Projects

My workflow for software projects is to create a github repository and clone them on the local drive of my laptop (or desktop) eventually the repository will get cloned to the external SSD and then will be included in my backup - Until then it’s safely stored in github.

Digesting Media

For media files (photos and videos), my workflow is a bit different. After capturing any media, and as soon as I can find some free time I will connect the external SSD to my home server and then proceed to copy all media from my devices to the SSD (I use a temporary inbox folder). Then I will run my backup script and upon completion I will format all sd cards (in their devices). Now I have my data backed up and all devices empty and ready.

For media files from smartphones I plan to used a shared dropbox account and the ‘camera upload’ feature of each phone to automatically download the photos on the server. The I will follow the aboce process for all files and also move them in a subfolder inside the ‘camera uploads’ to mark that these files have been copied to the SSD

Processing

After copying all media in the SSD the photos will be imported in lightroom and processed. For the videos I use a two step process. First videos are lossless cropped using Lossless Cut to remove unwanted content and merged back (using Lossless Cut or FFMPEG) to a single file per device per event. This process is lossless (or in some cases only the audio gets re-encoded from PCM to AAC). I’m currently exploring some options for adding titles and simple transitions effects (without re-encoding).

Accessing

The next step is to export accessible and streamable version of all media - for the photographs I usually manually export the from lightroom and copy them in the emby’s gallery folder. But for video files this would take too much of my time. So I have a simple script in node.js that scans the location where I backup the SSD and finds the videos that are missing from emby’s gallery folder, then it transcodes them to h264 at 6MBit and copied in the gallery folder. I run that script in the server, usually after backing up new video files.