Backup Solutions

The following is current as of May 2019:

I’ve spent a few days researching software and services for backing up data. My requirements: encrypted backups, deduplication, low cost, compatible with Windows 10 & Linux, and preferably using off-site storage.

Software

I have only considered two competitors: tarsnap and restic. Tarsnap can create keys with different permissions – a server can run backups with no danger of a compromise leading to the destruction of backups, but it is only compatible with tarsnap.com for data storage.

Restic allows you to plug it into any system for storage. I prefer tarsnap’s extra layer of paranoia, but the service costs are where the battle ends for me.

Update: These tools both deduplicate at a block size rather than by file, and both utilize a cache for speeding up backups. Both chunk at a dynamic level, with data blobs/chunks/blocks usually being around 1MB. Tarsnap’s cache is local, but can be restored by scanning the backup server (at a network usage cost of approximately 0.1% the size of the data stored), while restic uses both a local cache and a cache on the destination. Restic also creates checkpoints while uploading backups to reduce duplication caused by interrupted uploads.

Services

  • Tarsnap: $0.25/GB/month (transfer: $0.25/GB)
  • Rsync.net: $0.04/GB/month (min: 200 GB)
  • Amazon S3: f***ing complicated pricing
  • Wasabi: $0.0059/GB/month (no other charges)
  • Backblaze B2: $0.005/GB/month (download: $0.01/GB)
  • Local: Hardware costs + electricity.

Obviously, price is not everything. Rsync.net offers daily snapshots, cheaper per-GB pricing with mass amounts of data storage needed, and additional features. Amazon S3 and Wasabi are designed for application services rather than storage. Backblaze’s B2 is probably the only cloud service (of those I examined) designed for this usage.

Ultimately, cost is my limiting factor. My backups are using restic and local hardware for now, but I plan to move to using Backblaze B2 as I can afford to.

Updates

Since publication, a few have reached out to me recommending alternative services or sharing their choices. I have not compared these as thoroughly as I did my shortlist, but I feel they deserve their own note for anyone pursuing this decision themselves:

  • SpiderOak One Backup: Starts at $0.04/GB for their 150GB plan, goes down to $0.0058/GB with a 5TB plan. I’d probably choose it if I had a bit more money to spend.
  • CrashPlan for Small Business: $10/computer, “unlimited” storage. Haven’t looked at the caveats included.

A Journey into λCalculus

I’m playing around in Minetest, and I have an idea. In order to execute this idea, I’m going to need a simple programming language. Asking Google… implementing a simple programming language

7 lines of code, 3 minutes: Implement a programming language from scratch
Sounds good! Ridiculously simple, fast, and gives us a fully usable language. (I would’ve understood brainfuck better..) Let’s see what this language looks like:

(λ v . e)   anonymous function with argument v and returning e
(f v)       call function f with argument v
(λ a . a)   example: identity function (returns its argument)

And the scary one:
(((λ f . (λ x . (f x))) (λ a . a)) (λ b . b))

Seems okay until I get to understanding that last example. If you’re curious about the steps I went through before I finally figured it out, here are my notes. I’m gonna skip to the good part:

(((λ f . (λ x . (f x))) (λ a . a)) (λ b . b))
two identity functions, let's name them I, & remove excess parenthesis
(λ f . (λ x . (f x))) I I
the syntax is still confusing me, let's make an "F" function
F(x) -> return (λ x . (f x))
F I I                          equivalent to ((F I) I)
substituting the function (identity) into our definition gives us
(λ x . (I x))                  which..is actually the same as the identity function
I(I)                           ..it all comes to returning the identity function

λ x . x

Now, at this point I’ve learned a few small tricks for my understanding, as well as how lambda calculus works in general.

Reduction

Solving a lambda calculus program is made of three (or 2.5) steps called reductions:

  • η-conversion (eta): Replace equivalent functions with simpler forms (λ x . f x) -> f
  • β-reduction (beta): Substitution (λ a . a) x -> x (essentially, THIS is solving it)
  • α-conversion (alpha): Rename conflicting names (λ a . a b) (λ a . a) -> (λ a . a b) (λ c . c)

References

This is where my journey ends for now. I started studying lambda calculus because of a desire to implement a simple programming language, but this will likely not satisfy my needs..at least not in this form. Here are additional resources:

Pocket Dragon (NSFW Fiction)

A short work involving anthropomorphic canid characters and a dragon. Inspired by these images (but not directly based on them).


It was just three of us, but we partied like a frat house. At some point I’d been dared to make myself 5 inches tall instead of 5 feet tall. Yeah, I was a little short for my girls, yet alone a dragon, and that was before the dare.

Well, I say my girls, but really I was theirs. A curse had bound me to Leslie, she was the black canine of indeterminate pedigree. Big floppy ears, a wolfish muzzle, but her voice was all fox. Anise was her best friend, a fennec who had to be gagged when her mate came to visit for a night or two.

So there I was, 5 inches tall thanks to Anise, and very very slowly realizing that the alcohol content in my blood did not shrink with the rest of me. If I were any other species, I’d have been dead already, fortunately dragons are a little harder to poison.

After recovering from a fit of giggles, Leslie looked a little concerned. “You okay there-” a snort, “lil.. little guy?” I’d already passed out, still standing.

The next thing I knew, I was waking up with my snout in a small warm divot on a large silky rug of some- I looked up, seeing the hills of my Leslie’s breasts and her face looking back down at me. The divot was her belly button. I blushed, there was a faint scent of musk in there from when she made me paint her with cum, and she hadn’t washed quite enough to remove it all. I got up onto my knees.

“Hey.. C’mere draggie.” Her hand, slightly larger than me, brushed against my back. The force of it knocked me right back over, dizzying me further.

Before I could react, her hand picked me up and pushed me under the hem of her panties. I immediately noticed that she was in only her panties, and the strong scent of alcohol-fueled arousal.

“C’mon dra.. draggie. Come in.. inside.” She gasped as her clumsy movements brushed my rough exterior against her bowling ball-sized clit. An involuntary movement pushed me harder against it, and my legs and tail were enveloped by sticky, puffy labia.

It was far more pleasurable at this scale than I expected, like being caressed by velvet.. that also happened to be a little too humid. The smell got a long stronger as I felt her open up slightly to accept more. She finished shoving me inside with a moan and I was nearly folded in half in the process.

I took a moment to rest, breathing heavily in a way that made the all-encompassing pink walls around me quiver and further moisten. Thank goodness I didn’t need air for a while, especially at this smaller size, or I could have been in serious danger.

Grasping around – more than a little unsure of where I was with my level of intoxication – I must’ve found a sensitive spot as the walls of my cave trembled and squeezed me tighter. A muffled squeal came from deep within, and the squirt of lubrication that followed tried to drown me.

I don’t remember much after that, the lack of oxygen and booze enough to make me pass out again.

Switches Suck

I don’t like switch statements, in any language. They seem unnecessarily verbose and error-prone, in fact I forgot the break statements in my example below on the first draft. Most of the time, you don’t want the fall-through feature, but you have to remember that extra word for each case to prevent that.

switch (n)
{
    case 1:
    // something useful
    break;
    case 2:
    // another useful option
    break;
    default:
    // nothing matched
}

I also really hate the indenting used in most examples (including my own), as it makes it more difficult to visually parse. I prefer to just create an object with keys based on the possible values, and access what you need directly.

-- we're gonna pretend these are useful functions dependent on a star's type..something to do with heat?
local default = {}           -- used for a unique key
local heat = {
  A = function() end,        -- pretend these are full of useful code
  B = function() end,
  G = function() end,        -- and so on
  [default] = function() end -- default which can't be accidentally chosen
}

-- make sure we don't error, and call default if needed
if heat[star.type] then
  heat[star.type]()
else
  heat[default]()
end

Better Fluid Storage

A while back, I posted a prototype fluid storage system with a mechanic for handling breaches in a pressurized system. I thought I’d be clever by storing fluids as a percentage of a defined volume and pressure. For a “simplified” system, it was quite complicated, and fundamentally flawed.

This time, it is straightforward. Keep track of the amounts of each fluid, a total sum, and volume of the container. Pressure is the sum divided by the volume, and the percent of a fluid is its amount divided by the total sum of all fluids.

tank = {
  volume: 200, sum: 300,
  contents: { hydrogen: 200, oxygen: 100 }
}
pressure = tank.sum / tank.volume -- 1.5
percent_hydrogen = tank.contents.hydrogen / tank.sum -- 0.67

Everything needed from a container can be accessed immediately or with a single calculation involving only two variables.

But what about hull breaches?

Fluids vs Mechanical Classes

I realized that I should define fluid containers very narrowly, all they need care about is a small set of numbers, and have a few functions to modify that state. Enter the Breach class.

Breach(fluidA, fluidB, volume)

Specify which fluid containers are interacting and the volume ( I guess technically it should be area) of the breach. Each update cycle moves the pressure difference multiplied by the volume (area) of the breach from the higher pressure container to the lower pressure container.

What about pumps? I have those, with a “volume” and a “rate” modifier to allow you to adjust how fast the pump works. Pumps only work in one direction, but have a function to reverse them.

Want only one fluid to go through..say, a filter? Made that as well. Valves, so that you can adjust flow rate, filter-pump combos for pushing just the right amount of one fluid, and one-way valves to allow pressure to escape but not allow any blowback.

The Flaws

  • Once pressure is equalized, contents do not mix between fluids.
  • All fluids have the same density. This probably isn’t that hard to fix, but is unneeded for my purposes.
  • All fluids mix. This may or may not be harder to fix depending on how it is approached.
  • Temperature isn’t simulated at all. I would love to have heat transfer and heat affecting density, but these details are not necessary for my current project.

The Code

As of publishing this article, I don’t have a library to give you, but I will update it as soon as I do release it. For now, here is where I have the beginnings of a library. No matter what, I can promise it will be available by the end of April (or upon request).