Introduction

Say you uploaded 200 GB to S3 last night. This morning you want to know one thing: did it arrive intact?

The obvious way to find out is to download it and compare. That means 200 GB of egress, an hour of waiting, and a bill. All to learn a single yes or no.

There is a better way. S3 will checksum a piece of an object for you, on its own hardware, and hand you back the answer in a response header. The bytes never move.

mito is built on that one trick. It asks S3 for thousands of those little checksums at once, computes the matching ones on your side using your CPU's built-in CRC32 instructions, and compares the two lists.

    The obvious way                     What mito does

   ┌──────────────┐                   ┌──────────────┐
   │   S3 object  │                   │   S3 object  │
   │    200 GB    │                   │    200 GB    │
   └──────┬───────┘                   └──────┬───────┘
          │                                  │
          │  200 GB over the wire            │  ~25,600 tiny checksums
          │  ~1 hour                         │  bytes stay in S3
          │  egress charges                  │  seconds to minutes
          ▼                                  ▼
   ┌──────────────┐                   ┌──────────────┐
   │  your disk   │                   │  32-bit CRCs │
   └──────┬───────┘                   └──────┬───────┘
          │                                  │
          └────────► same? ◄─────────────────┘

Same answer. Almost none of the cost. Architecture explains how the trick works.

What follows from that

Once you can compare a remote file without downloading it, everything else falls out.

mito compares by content, in any direction: local to local, local to S3, S3 to S3. And sync can use the same machinery to transfer only the parts of a big file that actually changed, instead of the whole thing.

What people use it for

• Checking that an upload really landed, byte for byte
• Comparing two copies of a dataset that live in different places
• Keeping a bucket in step with a directory, moving only what changed
• Finding out what your S3 API traffic is costing you
• Hunting down failed uploads that are quietly still on your bill

Before you trust it

This is experimental software (0.x). It is a personal project, not operational tooling. It does real work on real buckets, so treat it like anything else that can delete your data: read what it prints before you type --force.

Three things are worth knowing up front.

sync does not checksum to decide what to copy. It looks at file size and modification time, which is fast and cheap. It skips a file only when the sizes match and the destination is newer. That catches most edits, but it trusts two clocks. A restore that sets an old timestamp (tar -x, cp -p, rsync -t) looks older than the copy already there, so it gets skipped. When you need to be sure, use diff, which does checksum.

The no-download trick needs a real S3 feature. It depends on S3 additional checksums. Not every S3-compatible service implements them. See S3-compatible storage.

Interfaces are settled, not frozen. Commands, flags and exit codes are documented and stable enough to script against. But this is still 0.x, and a version bump may change them.

Where to start

New here? Read Install, then Quickstart.

Already have it running and just need an answer? The command pages have the flags. The reference pages explain why things work the way they do. Security and privacy covers what mito reads, what it writes, and what it can destroy.

Disclaimer

MitoSync is a personal project, written to learn hardware checksums, C++ concurrency and the S3 API. It is provided for educational and experimental purposes only, with no warranty of any kind. It is not intended or recommended for use in production or business-critical systems. Evaluate it carefully and run it at your own risk on any system or bucket you rely on.

Licensed under Apache-2.0.