Libraries for the Transient Generation

Having moved house three times in as many years, and many of my peers experiencing the same (or at very least the threat of being uprooted), we become inherently more mobile in response to our surroundings. What was once proudly displayed on bookcases and shelves, is packed away into storage, and we become a little less with every iteration.

Obsolescence factors into it – disc drives have become relatively niche, being eroded from consoles over time, and even being relatively optional on PC’s & Laptops for the last decade. And that doesn’t begin to factor in convenience.

Enter the streaming services.

Outsourced Libraries

Streaming exploded onto the scene, and transformed the way we enjoyed media. Suddenly you had access to every song you could hope for, and had an impressive wealth of film & TV shows on demand. Spread across two primary providers, for a low monthly fee, you had (almost) everything you could want. There were some exceptions, of course, such as Game of Thrones (before everyone collectively groaned in disappointment), where the only solution was a cable subscription, or donning the proverbial eye-patch.

In short, this new medium was an evolution – it changed the landscape for how the media world operated. Suddenly, profit could be earned on an otherwise untapped market. Of course, it wasn’t developed in isolation – at this time, everything was moving to the “as a service” model with a monthly fee attached. In short, this evolution was an erosion of ownership, of permanence; pay for everything, and own nothing.

The Golden Child loses its Luster

Ever taken a fancy to watch a particular show (or movie), or listen to an album/song, only to find that it is no longer available on your streaming services? Songs removed due to licensing/distribution agreements, or even copyright litigation. Movies & TV shows region locked, or excised due to expired deals etc.

I’m old enough to have taken pride in physical media collections – CD’s, DVD’s (and eventually Blu Ray), books, and even partaking in the vinyl record revival. These were proudly displayed, and were a staple of the household, being the source of nightly entertainment. Unfortunately, when space is at a premium, antiquity is often the first casualty. They become surplus to requirements, especially in the digital age where everything is already at our fingertips. Packed up and put into storage, accruing its own cost (either another monthly fee, or grief from parents storing boxes upon boxes of all their children’s interests). It was often easier to simply cut them loose, like cassettes and VHS tapes…relics from a bygone era.

As such, it is all the more frustrating when that very piece of media you wanted to enjoy, and can no longer access, you know you own. Or at least, owned. It can, at times, feel like we are slaves to the curators – several large streaming giants determining what we listen to or watch, and when. And yet, at the same time, we have greater access to a more diverse catalogue than ever before, but that does not speak to quality, leaving you feeling empty. To paraphrase Springsteen; 50+ streaming services, and nothing on.

Cracks Appearing

Although we exist in a centralised age, with several large companies acting as de-facto curators, the streaming space has attracted more parties now that the model has proven viable. It is no longer only Netflix, instead we have Disney+, Amazon Prime, Apple TV, Paramount+, HBO Max, Crunchyroll, Now TV, and so much more. The centralised aggregators have splintered, fanning outward at an irritating rate. What was once an extensive complete library, has been carved up and sectioned off into distinct categories. Now, for access to the same level of content, you have to pay several providers an ever-increasing monthly fee, and even then…what you want to watch may be pulled at the whims of a third party (*cough* Netflix’s rampant cancellations *cough*).

Over time, these cracks in this new ecosystem have grown deeper, joining together to form chasms and exposing the almost forgotten beast…its siren song beckoning…”yo ho…yo ho”. Streaming’s major victory over piracy, convenience, is marred by the re-emergence of inconvenience, prompting many to cancel an ever increasing number of subscriptions, particularly in the wake of draconian enforcement of arbitrary Terms of Service (*cough* looking at you Netflix *cough*).

Become the Curator

Rather than simply accepting fate, and outsourcing your content library to a cabal of third parties (or succumbing to the jolly rancher’s song), there is a real (and legal) solution. Remember that library you once took pride in? Time to dust it off, and make use of it for the modern era, because let’s face it…who wants to get off the couch to pop a disc in? Especially when there’s a better way.

The joy of having your own library is simple – you own it. If a podcaster says something disagreeable, you don’t suddenly lose access to your Neil Young records.

All aboard the Digital Revolution

Digitising your media is nothing new. In fact, it was something we used to do regularly – remember buying physical CD’s to rip to your computer, and load onto your MP3 player? This ubiquitous act was eroded by Apple in the beginning, streamlining the process with iTunes, before eventually become obsolete with the introduction of the streaming services. However, it seems obsolescence may not be permanent itself. We are familiar with the process – transforming physical media into digital for easier access.

Taking your DVD’s, Blu Ray, and CD’s etc, and digitising them centralises your library, and effectively gives you the groundwork to make that library work for you. You can effectively eliminate the recurring fees of the streaming platforms, and return the era of your own curated collection, modified for the modern age. Although you can cut out the monthly costs, this solution is not free.

The Cost of Control

You’re going to need a bigger hard drive. Or many hard drives. You’re also going to need a device to act as a server. Although you could build a dedicated machine, or use a Raspberry Pi (if you could get one for reasonable money), or even a NAS (Network Attached Storage), the most budget friendly option is to repurpose old hardware e.g. an old laptop or PC, and let’s face it…most of us interested in tech have at least one lying around.

Data storage is the primary resource of your digital library, and is the single biggest cost factor. Media libraries are large, and likely to grow, so you will need tens of terabytes (if not more) to store it all. Thankfully the price of hard drives has come down (there was a large spike due to Chia, yet another cryptocurrency, in recent times), but it is still likely to cost you in the region of €100-300+

You don’t have to get fancy – external USB HDD’s will work fine for these purposes. Typically, you’ll want to buy the largest hard drive you can afford to get started.

It is important to note that larger sized external HDD’s are generally SMR (Shingled Magnetic Recording) disks. Traditional HDD’s are CMR/PMR (Concentric or Perpendicular). Confused about the terms? The main thing to note is that SMR drives offer greater storage density per platter (layer), however this comes at the cost of write performance. Think of roof shingles; each block overlaps, so when writing to an overlapping portion, the disk has to read the data, then rewrite it with the new data overlaid. CMR/PMR drives do not have this issue, however as a result they typically need more platters for the same capacity, increasing cost.

SMR drives work fine as distinct units i.e. each drive storing complete files. They should never be used as part of a RAID array.

It will also take time to scan and digitise your library. Although it can be automated (see ARM), it is an inherently manual process, but not a difficult one. Simply download MakeMKV (free for DVD, Blu Ray requires a license (beta versions work for Blu Ray, and have free keys; check here)), pop in your disc, and open it in MakeMKV. Select the tracks you want to save, and click run. It will take minutes for DVD’s, hour(s) for Blu Ray, so in the meantime you can carry on with other tasks. To speed things up, you can have several disc drives on the go, as the impact on the CPU is minimal.

You will notice that each disc has several tracks – there is no way of knowing what each one is until you save it. The only reasonable certainty is that the largest file is the feature, so if you’re only saving the movie and ignoring featurettes, you’re relatively safe with that approach.

For best performance, I recommend saving files to an internal SSD (Solid State Drive) as an ingest point. Once saved, you can view the files, previewing quickly in VLC to see what each file is, and renaming all of the video files. At this stage you can remove whatever is surplus to requirements.

Optional – Re-encoding to save space

The files saved from MakeMKV are uncompressed – true copies of what was on the disc. They are saved as MKV, and can contain several audio tracks, as well as subtitles. If you’re tight on storage space, you can re-encode these files as MP4 with H.264 or H.265 (HEVC). H.264 results in larger file sizes, and has superior support, whereas HEVC is a lot more efficient.

If you want to preserve multiple audio tracks e.g. audio commentaries, you’ll need to save as MKV.

Personally, I don’t recommend compressing your media – preserving the original quality prevents you from having to re-scan your entire library, and considering the time investment, I’d say the trade off is worth it.

Naming Scheme

Depending on which media server you choose, there will be a particular naming scheme employed. Be sure to name your files in accordance with this for best performance.

Copy files to Primary Storage

Once your files are saved, (optionally compressed) and renamed, you can copy them to your primary storage (e.g. External HDD). The primary reason for carrying it out this way is for speed. Your files will write to the internal (ingest) SSD faster, and renaming operations won’t take as long. Once complete, you can copy to the external HDD while scanning the next disc.

Unfortunately, the cost doesn’t stop there…or at least, it shouldn’t. Unless you’re a glutton for punishment, and want to repeat the time-consuming process of scanning your entire library, you’re going to need back ups. Accidents happen, and drive failure is a real thing.

Yes, that means double the cost for hard drives, however it is not all doom and gloom. Starting out, you can buy external HDD’s as required, as you figure out the total storage your library will take up. Once complete, you can then get a large enough drive to back everything up to. In other words, you can stagger the cost (and eliminate the guess work).

Why bother?

If you’re not going to save any money, then what’s the point? Well, put simply, collections were never about saving money, but rather about preservation and access. In other words, saving media. Public libraries have a far greater catalogue of books, yet many of us still buy physical books for ourselves. Why? The public library is certainly cheaper, yet we understand that our collection of books says something about us…a story in itself of where we were, of who we were at a given time. By keeping the physical copy, we keep it in memory – a literal reminder of stories we have enjoyed.

It is the same for music, whether physical, digital, or even analogue. It is a record of what we enjoy (if you pardon the pun). By curating your own collection, you guarantee access at any time, to the content and media that you have grown to love. Streaming is fantastic, but it is inherently nebulous. Trying to remember specific TV shows, or albums is often easier said than done.

In the modern digital world, we are careering closer to George Orwell’s vision of 1984, with media being ousted outright into the memory hole, or simply being revised over and over again. We have cherished children’s classics being re-written to be more inclusive, movies being re-cut and edited within an inch of their life, and TV shows being tweaked after the fact. In other words, if you & I are speaking about a particular movie, how can we be sure we’re talking about the same film? Version control matters.

Version Control for Media

Ok, I know what you’re thinking. It sounds paranoid, but I promise you, it’s not. The original Blade Runner has several distinct versions (workprint, US Theatrical, International Theatrical, director’s cut, and (supposedly) Final Cut). If you’ve watched it, do you even know which version you’ve seen? Or take Star Wars – George Lucas revisited the original trilogy countless times, and actively resisted preservation (more info here). I’m not getting into whether Han shot first (he did), or the Hayden Christensen debacle, just highlighting that kids who watch Star Wars on Disney+ for the first time are watching a different movie than I did (Yub Yub is the only acceptable song to cap off the trilogy).

When we give in to our corporate overlords and cede control of our digital libraries, part of that is sacrificing version control. Now, most people are not going to notice (and of those that do, the majority likely won’t care), but that doesn’t negate its importance. What version of a piece of media do the curators choose? Or more accurately, what version are they provided? The Muppet Christmas Carol sans “When Love is Gone”, Buffy the Vampire Slayer in the awful HD remaster (myriad of issues; aggressive DNR smoothing faces, missing filters, lack of colour correction, crew in frame (due to being shot with 4:3 in mind), more info here), Stargate SG1’s re-cutting of the pilot, to name just a few. The experience of watching these pieces of media is altered. This can also occur with music, being subject to remasters, changing the soundscape, and having you question whether it always sounded that way (hint: it probably didn’t).

In the age of DVD and Blu Ray, films and TV Shows were packaged with an abundance of special features. These featurettes have been lost in time to the streaming services. Sure, you could look up “behind the scenes” or “the making of” on YouTube, but having to think of it, and be discerning of the results runs counter to simply preserving these elements. Being able to enjoy your favourite movies with commentary tracks is a simple pleasure, but moreover its something you can choose to omit because you’re the curator.

You may prefer the versions the streaming services provide, or perhaps not. The point remains that you likely have a preference, and that preference should be protected. Investing in your own library allows you to share a part of yourself, your experiences with others. After all, we are products of our culture, and the collections we foster tell a lot about us as individuals.

Modern Problems require Modern Solutions

Were this the early 2000’s with a single internet connected device in the house, you could simply access the files on the hard drive(s). But it’s not. Things have moved on, and we’re more mobile with several devices each. In short, we need a server.

Home media servers have been around for decades, in a variety of different forms. In short, they allow you to access centrally stored content on other devices (they serve your content library to a client device), which is exactly what you want to achieve. The problem is that the term “home media server” conjures images of clunky specialist solutions, with considerable expense (and maintenance). If you mention it to the non-tech minded people, you will witness their eyes glaze over, assuming it requires specialist knowledge.

Repurposing old Hardware

Something that is important to me is avoiding e-waste wherever possible, and one of the primary methods of this is finding a new purpose for hardware. Have an old laptop lying around? You’ve just found your new media server!

Sure, you could use dedicated specially designed hardware for an optimal setup, but it’s not necessary. All you need is storage (HDD) (external USB HDD will work fine), and a computer to act as a server (preferably with a wired ethernet connection for best performance). You could use your primary machine, however there’s no point if you have another lying around (and let’s face it…most of us do).

Depending on the hardware you’re repurposing, you may need to consider formatting and changing OS to something less demanding i.e. Linux. This will allow you to eliminate the bloat of Windows, and get the most out of your system. Once you have your machine ready, it’s time to select your server software i.e. your ecosystem.

Media Server Software

Whatever the platform (OS) you’re running, there are several software solutions for creating your own media server. You may even have heard of some of them before – Plex, for example, are the hoover of media servers. They have impressive market penetration, and have clients available for virtually every platform (games consoles, Windows, Android, iOS, OSX, or Linux). There is a free tier, however many of the features are pay-walled, and there is growing discontent among users of the direction the company are moving, especially given their roots.

The other two major players in the space are Emby, and Jellyfin (if you’d like to know more about the history, see below)

Way back when, the original XBOX had a program developed for it (requiring modification to the hardware via softmodding, i.e. no disassembly required) called XBMP (an amalgamation of two concurrently developed programs, merged with the source code published after the fact). This would later become XBMC (a fully open sourced project), and would fork into XBMC4Xbox (no prizes for guessing what this was for), with Plex beginning its life as a port of XBMC to OSX. Eventually there was a divergence of interests, so Plex forked from XBMC into its own entity.

Around the same time, Emby (known then as Media Browser) came into being, designed as a Media Browser for Windows Media Center, striking out on its own with the advent of Windows 8 (and the abandonment of Windows Media Centre). XBMC would then rebrand as Kodi (retaining the open source model).

Plex became a proprietary (closed-source) client-server model, whilst Kodi remained firmly in the media centre category (it can technically be used as a server, but it’s not explicitly designed with this use case in mind). Emby was originally open source, however they would follow Plex’s lead in 2018 becoming closed source. This caused Jellyfin to fork from Emby 3.5.2, with development continuing in the Open Source model.

Unlike Plex or Emby, Jellyfin’s feature-set is 100% free, and due to its open nature, plugins can be developed by anyone allowing for unique opportunities to create a server that fits your needs entirely.

Your own Streaming Platform

Jellyfin is 100% open source, freely available, and damn capable. Sure, it lacks some features of its contemporaries, however nothing is locked behind a paywall, and it is highly adaptable. Given the outlay of the drives, free is a good number!

The server component is compatible with Windows, OSX, Linux, and Docker, so installation is straightforward. Whether you elect to install the server natively or via container is down to your use case, and knowledge level. For many, learning about Docker will be surplus to requirements, where the native application will handle everything just fine. If you are installing the server on your main machine, rather than repurposed hardware, then it is worth pursuing the docker route to avoid dependency issues down the line.

Native client applications are available for iOS, Android, Windows, Linux, and OSX, and the web browser will work for XBox & Playstation etc. Your mileage may vary for Smart TV’s, depending on their vintage, so you may need an additional device for best performance e.g. Chrome Cast with Google TV (CCWGTV), NVIDIA Shield, Firestick etc.

By default, Jellyfin is only available on your local network, although you can configure remote access relatively easily. Depending on your Internet Provider, you may not be able to forward ports (required for the standard method of remote access, as well as reverse proxies), but fear not, there is an easy solution available; Tailscale.

This simple solution enables you to access your media server anywhere, and effectively transforms it into your very own Netflix.

Tailscale is a VPN (Virtual Private Network) – it is free to use, however it is important to remember that a VPN has a technical definition, and a colloquial one. This VPN is designed for mesh networking i.e. making remote devices appear as a local to your network, rather than the primary use cases of the likes of ExpressVPN etc.

The setup is fast, and extremely straightforward. Simply register an account, add devices by installing the Tailscale application on each device (including the server), and configure the account settings. You’ll want to make sure you have public DNS servers listed e.g. Cloudflare or Google etc. so that you can still access the internet.

The drawback of Tailscale is that it has to be installed on every client device. This adds a further step to the setup, which can be infuriating, particularly if there is no Tailscale support for your particular device e.g. Smart TV. There are workarounds e.g. Subnet Routers (formerly relay nodes), however it is important to remember that if you’re configuring access for someone else (for example, your parents), then you’re likely going to be the one configuring and maintaining the solution, so the less moving parts the better.

Other solutions such as reverse proxies, or port forwarding, are more straightforward, however they are reliant on your ISP (internet service provider). With traditional port forwarding, there are increased security concerns as you have increased the attack surface of your network (remember the story of LinkedIn being breached due to an employee hosting a website on their home network?), and with reverse proxies there is additional configuration required.

TL;DR

In our nebulous and transient world, we carry with us memories of what we have seen, read, and listened to. These memories matter, and have shaped part of ourselves, and deserve to be protected as is, rather than being subject to re-imaginings after the fact (or to quote Death Cab, relying on a faulty camera in our minds).

By cataloguing and preserving the things that speak to our soul, we ensure access to the media we love. You are no longer beholden to the machinations of mega corporations, and instead you are put back in control. Watch what you want, when you want to watch it, on any device. At the same time, you give new life to an older piece of hardware.

You do not need to know a lot to get up and running, and the great thing about Open Source projects (such as Jellyfin) is there is a vibrant community, so if you’re stuck on something, odds are you’ll find an answer to your problem quickly.

I hope that this article may inspire you to take pride in your library once more, and see it for what it truly is – an extension of yourself.