s3tap

s3tap watches what an application does with S3, then judges it. It uses eBPF (a way to run small, safe programs inside the Linux kernel) to record the traffic. From one capture it produces a plain-language health verdict, an optimization advisory or a machine-readable findings stream.

It splits cleanly into two halves:

1. Capture (privileged, uses eBPF): quietly records what an app does with S3: DNS lookups, TCP connects, TLS handshakes, HTTP operations and the kernel's own tcp_sock stats (srtt is the smoothed round-trip time, plus min_rtt, retransmits and cwnd). These come out as versioned public JSONL records.

2. Consumers (unprivileged, pure): read those records and reason about them. Nothing here needs a probe or a privilege, so they compose in a pipe:

   $ s3tap --format jsonl | s3tap doctor
   

The commands at a glance

Design principle: the record is the contract

Every consumer reads the same public records. So you can capture once, then re-judge it offline, diff it against a baseline or feed it to a different tool. doctor is checked against a small reference implementation (a "parity oracle") that works out the same verdicts a second way, so its output stays honest and reproducible.

This site documents the implemented commands and records. Exploratory or not-yet-built ideas are intentionally out of scope here.

Install & capabilities

Requirements

• Linux kernel 5.8 or newer with BTF (BPF Type Format: kernel type information the loader reads to match your exact kernel). The agent uses the BPF ring buffer (bpf_ringbuf, 5.8+), a fast channel that streams events from the kernel to user space. It also relies on CO-RE (Compile Once, Run Everywhere, which lets one build run across different kernels). Optional fields are guarded with bpf_core_field_exists, so a kernel that lacks one still loads. The iov_iter layout and its enum values both shifted across ~5.14 and ~6.4, so s3tap resolves the member names and the enumerators against the kernel it loaded on rather than against its own headers. That keeps DNS-query capture, TLS-SNI capture and the TLS handshake timing working across the whole range instead of only on 6.4+. The 5.8 floor is verified by the kernel matrix (see scripts/kernel-compat/BPF-TESTING.md).
• Build tools: clang, llvm-strip, bpftool (libbpf-dev) and a stable Rust toolchain (see rust-toolchain.toml).

Build

build.rs compiles and embeds the eBPF object, so the agent is self-contained.

$ just bpf-headers      # one-time: vendor the kernel BPF headers
$ just build            # == cargo build --release

The pure consumers (doctor, advise, analyze, scorecard) need no kernel, no probes and no privilege: they read records and nothing else. Their crates carry no eBPF dependency and build anywhere.

The shipped s3tap binary is nonetheless Linux-only, so running s3tap doctor on macOS is not possible today. It links aya unconditionally and its build.rs compiles the eBPF object against a vmlinux.h generated from the build host's own BTF, so the build fails before any consumer code is reached. Judging a capture on a Mac means building the consumer crates into your own tool, not running this one.

Privileges: setup instead of sudo

Loading eBPF needs privilege. Rather than running the whole agent as root, grant the binary its file capabilities once:

$ sudo s3tap setup            # base caps: load probes + attach
$ sudo s3tap setup --uprobes  # also grant the L7 / TLS-plaintext path

setup refuses a binary a local user could rewrite

This catches most people the first time. setup requires the binary and every ancestor directory up to / to be root-owned and not group- or world-writable, on the symlink-resolved path. A build tree (./target/release/s3tap) and the installer's default ~/.local/bin both fail that test, so sudo s3tap setup on either one errors out rather than granting anything. There is no flag to override it.

File capabilities belong to whoever can put bytes in the inode. Capping a binary under a directory you can rename would hand cap_sys_admin to every local user, along with the host-wide decrypted traffic it unlocks. So install to a root-owned path and cap that copy:

$ sudo install -m 0755 ./target/release/s3tap /usr/local/bin/s3tap
$ sudo /usr/local/bin/s3tap setup --uprobes

./setcap.sh is the shell equivalent and applies the identical refusal. It does carry one opt-out for single-user dev boxes, S3TAP_SETCAP_INSECURE=1 ./setcap.sh, which caps the build-tree binary in place after saying out loud what that costs. Running s3tap under sudo every time needs no grant at all.

Capabilities live on the binary file itself, which cargo build replaces, so re-run setup after every rebuild. --uprobes (probes that hook user-space library calls such as OpenSSL) grants more power. It adds cap_sys_admin, a broad capability close to full root. The plaintext path can then see decrypted bytes across the whole host. Opt in on purpose.

If privileges are missing, s3tap offers to self-elevate with sudo. Pass --no-elevate to fail with the normal permission error instead.

Verify

$ sudo s3tap selftest    # loads probes, drives a few real S3 requests, asserts each capability

See setup & selftest for details.

Quickstart

1. Is my path to S3 healthy? (zero-config)

No capture needed. check drives its own probe:

$ s3tap check                    # regional round-trip map + a health check nearby
$ s3tap check my-bucket/my-key   # a verdict on YOUR object (strict exit code)

check is the friendly front-end to doctor --live. It loads eBPF. Without the uprobe caps it only judges the network floor: the fastest round trip the network physically allows. See check.

2. Capture an app, then judge it

Capture the S3 traffic of a workload as JSONL, then pipe it into doctor:

$ sudo s3tap --app python3 --format jsonl > capture.jsonl
# ...run your workload...
$ s3tap doctor --from capture.jsonl

Or straight through in one pipe:

$ sudo s3tap --app python3 --format jsonl | s3tap doctor

doctor exits non-zero when any metric is outside its envelope (⚠). So it drops into CI as a gate. See doctor.

--from wants a capture and it checks. doctor, advise and scorecard refuse an input that parsed to zero s3tap.* records: they print what they actually read and exit 4 (tool failure), rather than judging an empty set. A doctor --json findings file is not a capture. Neither is an empty file, a truncated one, or the wrong log. This matters because the old behaviour was silent: advise --strict over a broken path printed "no advisories" and exited 0, so the gate had been dead for as long as the path was wrong. Exit 1 always means a verdict about your traffic. Exit 4 means fix the command.

A capture with no S3 request in it is not a pass either. If nothing decoded the HTTP layer (a Go or rustls client, or a capture taken without the uprobe caps) doctor reports NO OPERATIONS and exits 2. Only the network path below S3 was judged, so a green verdict there would say something the capture cannot support. A capture whose requests were decoded but never answered, which is what Ctrl-C mid-workload leaves behind, reports NO RESPONSES and exits 2 on the same reasoning.

3. How is the app using S3?

$ s3tap advise --from capture.jsonl

Optimization advice (client churn, missing parallelism, redundant re-fetches, throttling, caching go/no-go), attributed per process. See advise.

4. Machine-readable output

Every consumer speaks NDJSON findings for fleet ingest:

$ s3tap doctor --from capture.jsonl --json     # one s3tap.finding/1 per check + a run roll-up

5. Track a regression

$ s3tap doctor --from after.jsonl --baseline before.jsonl   # gate: fails on a regression

--baseline takes a capture of records, the same JSONL --from reads, not a findings file. Keep a known-good capture around as the reference. If you have no capture to keep, s3tap doctor --live --endpoint <url> --save baseline.jsonl drives its own workload and writes one.

--baseline is checked the same way --from is. A baseline path that opens but parses to zero s3tap.* records is refused with exit 4 rather than accepted as a baseline of nothing. It used to be accepted. An empty one did not even print a note, so a healthy capture passed the gate against a file that was never a capture. The --baseline section of the doctor metrics reference has the detail.

--save creates its file with O_EXCL at mode 0600, so it refuses a path that already exists and never truncates or follows a symlink. Re-running the recipe above over the same filename therefore fails, up front, before any traffic is driven. Delete or rename the old capture first, or save to a new name. There is no --force. The reason is that doctor --live routinely re-execs under sudo, so that write commonly happens as root with a path taken from an argument. Mode 0600 is for the same reason: a capture names your buckets, endpoint IPs, SNI hostnames and key hashes.

Composition

Because the record is the contract, you can capture once, then re-judge it offline, diff it or feed it to a different tool. The capture and the analysis never have to happen at the same time or place.

run: capture traffic

run is the default command (no subcommand). It loads the eBPF probes and passively records an application's S3 traffic as public JSONL records.

$ sudo s3tap --app python3 --format jsonl > capture.jsonl

What it captures

• DNS resolves (via the getaddrinfo uprobe), TCP connects, TLS handshakes.
• HTTP L7 operations (method, status, TTFB, S3 op-class). Needs the uprobe caps.
• Connection close-time tcp_sock stats: srtt, min_rtt, retransmits, cwnd, mss, bytes.
• Periodic in-flight TCP samples: opt in with --sample-interval-ms.

These become s3tap.operation/1, s3tap.connection/2, s3tap.sample/1.

Scoping to an app

Restrict capture so you only see the workload you care about. fork()ed children of a tracked process are followed in-kernel, so pre-fork servers (gunicorn/uWSGI/Spark) are captured:

What --app actually matches

--app matches the exe basename that s3tap reads from /proc/<pid>/exe. A process's comm is deliberately not matched. Any process can rename its own comm to anything (prctl(PR_SET_NAME)) without privilege, without exec, at any time after exec. Matching on it let an unrelated local process walk into a privileged capture. That mattered for two reasons. An admitted process writes its own request line and Host header, so it could inject records with any bucket, verb or status into your capture and flip a verdict or an advise --strict gate. It could also blind the capture, because the in-kernel pid allowlist is capped and spawning enough renamed processes evicts the real target.

The residual limit is worth stating plainly: an exe basename names a real inode and is fixed for the life of the exec, so forging it costs an actual executable at a path the attacker controls. That is a narrower door, not a sealed one. A user who can drop a binary named python3 somewhere they can execute it is still admitted. On an untrusted multi-tenant host use --pid, --exe, --cgroup or --container, none of which a peer process can influence.

One practical note: the basename compared is the one the symlink RESOLVES to. /usr/bin/python3 is commonly a symlink to python3.12. /proc/<pid>/exe gives the resolved target, so a strictly exact match would never fire for --app python3. --app therefore matches the basename exactly OR with a trailing version suffix stripped: python3 matches python3.12, gcc matches gcc-11. The suffix has to start at a - or . and be digits and dots to the end, so --app python still matches nothing and python311 is left alone. If a scope matches no process at startup, s3tap says so rather than sitting quietly and writing an empty capture.

When --app and --exe cannot see the process

Both name-based scopes answer exactly one question: what is /proc/<pid>/exe pointing at? That link is the only identity s3tap will admit on, in the startup scan and at every later exec alike. There is no fallback. The kernel event that reports an exec also carries the execve filename argument, which is the name the process chose to launch itself under. A symlink called python3 pointing at any binary at all is enough to set it, so admitting on it would hand any local user a way into a privileged capture. It is never consulted.

Having no fallback has a cost worth planning around. Reading another user's /proc/<pid>/exe is gated by ptrace_may_access, which wants the same uid or CAP_SYS_PTRACE. s3tap setup deliberately grants neither. It tags the binary with cap_bpf, cap_perfmon and cap_dac_read_search (plus cap_sys_admin under --uprobes). Of those cap_dac_read_search bypasses the DAC check, not the ptrace one. The link is mode 0777, so DAC was never the gate. A capability-tagged s3tap running as you therefore resolves your own processes only. Another user's app is left out of the capture rather than admitted on a name it picked for itself.

s3tap says so when it happens, once per run. The warning names CAP_SYS_PTRACE rather than leaving you to retype the app name. Two ways forward:

• run s3tap as root (sudo) to scope another user's app by name, or
• use --pid, --cgroup or --container, none of which read /proc/<pid>/exe at all.

One case stays quiet by design. A process with no exe link (a kernel thread, a zombie, one that exited mid-scan) is skipped without a word, because its pid could never have captured anything. A hidepid=2 /proc hides the directory outright, which arrives looking exactly like that and is skipped the same way. It still fails closed, it just fails silently.

Output formats

--format selects waterfall (default on a TTY), table, human, or jsonl (default when piped, so scripts get clean machine-readable output). --capture-plaintext records decrypted HTTP via the OpenSSL uprobes. --s3-endpoint points bucket/key decoding at a custom endpoint (MinIO, etc.).

Exit code

A capture exits 0 normally, whether it recorded a thousand records or none. An application that was simply quiet is a legitimate empty capture and not an error.

One shape is different and exits 3: a run that emitted no record at all, whose scope was --app/--exe only, which matched no process at the start of the run or at the end of it. That is a scope which missed rather than an app which was quiet. By exit code alone a script could not otherwise tell those two apart. It is the same 3 that doctor --live returns for a run that captured nothing. It means the same thing: s3tap ran, saw nothing and has no answer to give. The message on stderr names the scope it resolved and points at readlink /proc/<pid>/exe, --pid and --cgroup/--container.

Only the name scopes can end this way. --pid, --cgroup and --container name a target that either exists or fails closed at startup, so an empty capture under them is unambiguous already. The check is also made at BOTH ends of the run, because a process that execs into scope halfway through is a legitimate late match. That is why the startup warning is a prediction rather than a refusal. The closing scan is what turns it into a fact. One gap is worth naming: a matching process that both execs and exits inside the window without moving a byte reads as a miss.

See the CLI reference for every flag.

doctor: health verdict

doctor reads a capture (s3tap.operation/1 + s3tap.connection/2, from --from or stdin) and judges whether each latency span is healthy relative to the connection's RTT floor. The floor is the connection's smoothed round-trip time (srtt_us), falling back to its lowest observed round-trip (min_rtt_us) when no smoothed value was sampled. Smoothed rather than lowest on purpose: it is the basis the ×RTT thresholds are calibrated on, and for think-time (TTFB minus the network round-trip) the round-trip the connection is CURRENTLY seeing is the right thing to subtract, not its best-ever propagation floor. doctor only reads records, so it loads no probes and needs no privileges. It composes in a pipe:

$ s3tap --format jsonl | s3tap doctor
$ s3tap doctor --from capture.jsonl

It exits non-zero when any metric falls outside its healthy range (⚠), so it works as a CI gate.

Exit 1 is always a verdict about your traffic. If s3tap itself could not do the job (a --from path it could not read, an input holding no records, probes that would not load) it exits 4 instead, so a gate can tell "the workload regressed" from "fix the invocation". The full table is in the doctor metrics reference.

--from wants a capture and it checks: an input that parsed to zero s3tap.* records is refused with exit 4, never rendered as a verdict over nothing. --baseline is held to the same standard, so the reference side of a regression gate cannot quietly be empty either.

A capture that holds connections but no S3 operation is a third answer, not a pass. doctor reports NO OPERATIONS and exits 2: the network path was judged, the S3 layer was not. That is what a Go or rustls client looks like, or any capture taken without the uprobe caps.

A capture whose operations were decoded but never answered is the fourth. doctor reports NO RESPONSES and also exits 2. Every request was still in flight when the capture ended, which is what Ctrl-C mid-workload produces, so the reliability rows had nothing to judge either. It is a separate verdict from NO OPERATIONS because the remedy differs: let the workload finish, rather than re-capture with the uprobe caps.

What it judges

• Global rows: DNS cold-resolve, TCP connect, TTFB (new / reused), retransmit rate, HTTP errors.
• S3 per-op-class TTFB: think-time = ttfb − RTT, separating a far server from a slow one.
• Tail latency: p95 / p99 of the TTFB populations (the slowest 5% and 1% of requests, so you see worst-case delays not just the average).
• Connection reuse rate: are you repaying TCP+TLS setup unnecessarily?
• Connection path diagnosis: min_rtt/jitter, send/recv bottleneck, BDP ceiling (the most data that can be in flight at once, which caps throughput), loss shape.
• In-flight time-series: throughput ramp, bufferbloat onset (delay that builds up when buffers overfill), loss timeline (from samples).
• Deployment environment estimate: same-region / cross-region / far.

See the doctor metrics reference for how each is computed.

Modes

--live is what the friendlier check wraps.

What --live says about the capture itself

A capture can be thin for reasons that have nothing to do with your storage path. The record set looks the same either way. So --live reports what the capture actually did before the verdict, on stderr. Read those lines first: they tell you how much the table below them is worth.

The drop warnings matter more than they look. A full ring sheds events under load. Load is exactly when the slow operations happen. A "healthy" median computed after the slow tail was dropped is not a healthy median. Treat any INCOMPLETE line as reason to re-run with lower --concurrency or --requests before believing a percentile.

The zero-record case is separate: if nothing at all was captured there is no report, and --live exits 3.

Only doctor --live and a targeted check print these. The zero-config regional sweep suppresses them, because it drives one short probe per region and summarizes those itself.

Parity

A parity test suite pins doctor's verdicts against a small reference oracle (a second, simple implementation kept as the source of truth). For any input the Rust engine and the oracle must agree on the marks and the run verdict. This keeps the judgments honest and reproducible.

See the CLI reference for every flag.

check: self-driving probe

check is the easy front-end to doctor --live: it drives its own probe and prints a one-line, plain-language verdict. Two modes:

Targeted: "is my path healthy?"

$ s3tap check my-bucket/my-key      # bucket/key, expanded to the S3 endpoint
$ s3tap check https://…/object      # or a full URL

This mode sends keep-alive requests to your object and prints a verdict with a strict exit code, so it drops into CI. Add --triage to also sweep nearby regions and report how much round-trip a closer region would save (with remedies). --auth SigV4-signs the probe for a private bucket. --region targets a specific endpoint. --verbose shows the full report.

--auth and credentials

--auth takes credentials from AWS_ACCESS_KEY_ID plus AWS_SECRET_ACCESS_KEY, else from ~/.aws/credentials under AWS_PROFILE or default. Environment credentials are never gated: they are the caller's own and reading them needs no privilege.

The file read has one guard, worth understanding because it is easy to mistake for a bug. A capability-tagged s3tap can read files the invoking user cannot. HOME is not scrubbed on exec, so HOME=/root s3tap check … --auth would otherwise hand root's 0600 credentials to any local user. s3tap therefore asks whether the REAL user could have read that file, and refuses only when the answer is no. Your own ~/.aws/credentials after the documented sudo s3tap setup install passes, since you can read it yourself. HOME=/root still fails. The check is made against the inode actually opened, not only against the path.

Zero-config: "where am I relative to S3?"

$ s3tap check                       # regional round-trip map + a health check nearby
$ s3tap check --map-only            # just the latency map

With no target, check prints a regional round-trip map, then health-checks a public AWS Open Data object in the nearest covered region (informational exit).

Privileges

check loads eBPF. The L7 rows want the uprobe caps (sudo s3tap setup --uprobes). The map needs only the base caps (sudo s3tap setup). Without them, s3tap offers to run sudo for you. With no uprobes it judges the network floor only (just the raw network timing, not the HTTP layer).

That has a consequence for the targeted mode's exit code. With no uprobe caps there is no S3 operation to judge, so a targeted check reports NO OPERATIONS and exits 2 instead of passing on the strength of the network rows. It is the same refusal as NO BASELINE: a run that judged nothing must not read green in CI. Grant the uprobe caps for a verdict about S3 itself. The zero-config sweep is informational either way and does not gate on this.

Stopping it early

Ctrl-C ends the whole command and reports what it already holds. It is not a kill: the run tears the probes down, notes that it was interrupted along with how many of the requested requests it managed, then prints a verdict over that shorter capture. Read the interruption note before the verdict, since a partial capture is a smaller sample rather than a different workload. A --save target is claimed before any traffic is driven, so an interrupted run does not leave a placeholder file behind blocking the re-run.

That holds for the zero-config regional sweep too, which drives one capture per region. One Ctrl-C ends the sweep rather than skipping to the next region, which is what a sequence of per-capture handlers used to do.

Capture-honesty warnings

A targeted check drives its own workload, so it prints the same capture-honesty lines that doctor --live does, on stderr, above the verdict. They say whether the workload actually completed a request, whether the kernel dropped events (which makes every rate and percentile below them partial) and whether the time budget truncated the run. Read them before the verdict. The zero-config regional sweep suppresses them and summarizes its own probes instead.

See the CLI reference for every flag.

advise: optimization advice

advise reads the same capture as doctor but answers a different question: not "is it healthy?" but "how is the application using S3 and what could be better?", attributed per process. Like doctor, it only reads records, so it loads no probes and needs no privileges.

$ s3tap advise --from capture.jsonl
$ s3tap --format jsonl | s3tap advise

What it surfaces

• Client churn: connections created and thrown away instead of reused.
• Missing parallelism: serial fetches that could overlap.
• Redundant re-fetches: the same object pulled repeatedly.
• Throttling: 503 SlowDown patterns.
• Caching go/no-go: whether a local cache would pay off.

Health vs advice

Health judgments live in doctor. This is optimization advice. By default advice is not a failure, so a clean run and a run full of advisories both exit 0. Pass --strict to exit 1 when any Warn or Advisory finding fired. --json emits s3tap.finding/1 records instead of the human table.

Exit codes

Exit 2 is not a quiet 0. A connection-only capture (a Go or rustls client, or one taken without the uprobe caps) decodes no operation at all. A capture whose operations were never answered has nothing for the checks to draw on either. An empty findings there means "nothing to judge", not "judged and clean". It is not gated by --strict, so a plain advise returns 2 for that capture as well: the population is missing whether or not you asked to gate on advice.

A real judgment still outranks it. An advisory sourced from the connection records can fire on a capture with no judgeable S3 population. That judgment is real, so under --strict the 1 wins. This mirrors doctor, where ATTENTION sits above its own missing-denominator verdicts.

Keep 2 and 4 apart when scripting. Exit 2 says s3tap read a capture and found nothing in it to judge, so the fix is to the capture. Exit 4 says s3tap never got a capture at all, so the fix is to the invocation.

Under --json a capture with no judgeable S3 population also emits one s3tap.finding/1 row, advisor-run, with severity unjudged. It names which of the two denominators was missing, so an ingest that stores NDJSON keeps the reason rather than an unexplained empty stream. It is keyed on the population, not on the exit code, so --strict never changes the set of records written — and the row can therefore sit beside an exit 1, when a connection-sourced advisory fired on a capture that had no operations. No row means the population was fine.

The input must be a capture

advise refuses an input that parsed to zero s3tap.* records. It names what it actually read and exits 4 (tool failure), which is distinct from every verdict code. A findings file is not a capture. Neither is an empty file, a truncated one, or any other log.

This is worth stating plainly because of what it replaced. advise --strict over a broken --from path used to print "nothing to flag" and exit 0, so a CI gate reported a clean run for a file that was never read. A gate that cannot tell "clean" from "no data" is not a gate.

See the CLI reference for every flag.

analyze: deep caching + prefetch report

analyze studies caching for one trace, offline. It then prints a recommendation: should you cache this workload, with which policy, at what size and will prefetching help? It is the deep, opt-in sibling of advise. advise gives a fast object-level yes or no. analyze runs the full retention bake-off (it replays the trace against several cache policies, LRU / ARC / S3-FIFO / OPT, to see which keeps the most useful data) in chunk mode, plus the prefetch tradeoff (prefetch means fetching data before it is asked for). It only reads data. It loads no probes and needs no privilege.

$ s3tap analyze --from capture.jsonl        # deep report (8 MB chunk mode)
$ s3tap --format jsonl | s3tap analyze      # analyse a live capture
$ s3tap analyze --from trace --fast         # retention only, no prefetch pass
$ s3tap analyze --from trace --json         # structured verdict, one JSON object

It auto-detects three trace formats, line by line: s3tap JSONL (s3tap.operation/1 records), NormEvent JSON (the replay format) and IBM COS text lines.

Exit codes

analyze has no --strict: its answer is a cache-suitability verdict, not a health judgment, so a "no-go" is an answer rather than a failure and still exits 0.

Exit 2 is the same code doctor, advise and scorecard return for a capture with no population to judge, and it means the same thing here: s3tap read your file and found nothing to work with, so the fix is to the capture. Reserve 4 for "s3tap never got a capture at all" — the fix there is to the invocation.

Cost levels

The prefetch Adaptive rung runs a per-capacity shadow cache and is minutes-slow on a large trace, so the ladder is tiered:

For a large trace, analyze looks at only the first --max-events ops (default 3,000,000 to match the study, or 0 for the whole trace). The capacity ladder tops out at 4096 chunks (32 GiB). Above that size the verdict is flagged a lower bound.

The verdict

The machine-readable verdict field (--json) is one of four values. It reports the real recommendation. go-latency fires only when prefetching actually hides latency, not just when a sequence looks predictable:

The human banner adds nuance (CACHE IT, CACHE FOR LATENCY, LITTLE TO GAIN, DON'T CACHE, TOO SHORT TO SAY). The retention section names the winning cache (ARC is usually the study's pick, a free upgrade over LRU). It shows how much ARC beats LRU by and how far it sits from the OPT ceiling (OPT is the perfect cache that can see the future, so it marks the best result possible).

advise vs analyze

advise is the quick gate everyone runs: object-level, capped at 500k events, LRU plus Markov prediction, short findings. analyze is the deep report you opt into when the gate says it is worth it: the full policy ladder, chunk by chunk, with a per-trace story. Both use the same decision thresholds from the study. analyze reproduces the report's numbers on the same trace.

See the CLI reference for every flag.

scorecard: observed-SLO scorecard

scorecard reads a capture and reports, for each bucket / s3_op, the latency and reliability it actually saw: request and error counts, the status-code mix and TTFB p50 / p95 / p99. It is the passive analogue of a speed test. The numbers are your own production traffic rather than a synthetic probe. Like doctor and advise it only reads records, so it loads no probes and needs no privileges.

$ s3tap scorecard --from capture.jsonl
$ s3tap --format jsonl | s3tap scorecard

What it shows

• A row per bucket and operation: request count, error count, the status-code mix.
• Latency percentiles: TTFB p50 / p95 / p99, so both the typical case and the tail are visible.
• Reliability findings: gated judgments (for example a 4xx or 503 rate above the floor) raised as s3tap.finding/1 records.

Report, not a gate

A scorecard is a report, so a judged capture exits 0 by default even when a finding fired. Pass --strict to exit 1 when any Warn or Advisory finding fired. --json emits the s3tap.scorecard/1 rows followed by the gated s3tap.finding/1 records instead of the human table, with the same exit-code contract.

Exit 2 is a report that could not be written, not a clean one. A row exists only for a bucket / s3_op group with at least one op carrying an HTTP status, so no rows means nothing was scoreable. Two shapes reach it. One is a capture holding no operation record at all (a Go or rustls client, or one taken without the uprobe caps). The other is a capture whose operations all lack a status, because every request aborted before its response line. The human render tells the two apart. Gating on the operation count alone caught only the first, so a capture of 500 statusless operations printed "none carried an HTTP status" and still exited 0.

It is not gated by --strict: a plain scorecard returns 2 for such a capture as well, because the rows are missing whether or not you asked to gate on findings. A finding that did fire still gates under --strict and exits 1. Note that this is not the precedence doctor and advise have: a scorecard row exists only for a group with at least one statused op, and every finding is derived from a row, so "a finding fired" and "there was nothing to score" cannot both be true here. The --strict branch is a real gate, not a tie-break.

Keep 2 and 4 apart when scripting. Exit 2 means s3tap read a capture and found nothing in it to score, so the fix is to the capture. Exit 4 means it never got a capture, so the fix is to the invocation.

Under --json the exit-2 case also emits one s3tap.finding/1 row, scorecard-run, with severity unjudged. It names which of the two causes applied, so an ingest that stores NDJSON keeps the reason rather than an unexplained empty stream. It appears ONLY for that case, so no row means rows were produced.

Health verdicts live in doctor. This is the descriptive view: what the traffic actually experienced, per bucket and operation.

The input must be a capture

scorecard refuses an input that parsed to zero s3tap.* records. It names what it actually read and exits 4 (tool failure), which is distinct from every verdict code. A findings file is not a capture. Neither is an empty file, a truncated one, or any other log. Rendering a scorecard over nothing would be a report about no traffic, which is not a report.

See the CLI reference for every flag.

setup & selftest

Two supporting commands for getting the capture pipeline running on a host.

setup: grant capabilities without sudo

The programmatic equivalent of setcap.sh: it grants this binary the file capabilities it needs to load its probes (file capabilities are fine-grained permissions attached to a program, so it gets just those powers instead of full root). You then avoid running the whole agent as root. It asks for sudo itself, once.

$ sudo s3tap setup            # base caps
$ sudo s3tap setup --uprobes  # + cap_sys_admin for the L7 / TLS-plaintext path
$ sudo s3tap setup --remove   # revoke the grant

Capabilities live on the binary's inode (the file's on-disk identity), which cargo build replaces. Re-run after every rebuild. See Install & capabilities for the full grant table.

setup refuses a binary that a local user could rewrite: it requires the binary and every ancestor directory up to / to be root-owned and not group- or world-writable. A build tree and a ~/.local/bin install both fail that test, so install to a root-owned path and cap that copy. --remove skips the check, since dropping caps is always safe. Install & capabilities has the reasoning and the dev-box opt-out that setcap.sh carries.

selftest: prove the pipeline

It loads the probes, drives a few real S3 requests and checks which capabilities (DNS / TCP / TLS / HTTP) actually produced a record. It prints a pass/fail table for all four and exits non-zero when TCP, TLS or HTTP failed. Those three are what prove the probes work end to end. This is the fastest way to confirm s3tap works on a new host.

DNS is informational and does not gate the exit code. selftest scopes its capture to s3tap's own pid. The curl driver it spawns is picked up as a forked child in-kernel. Nothing else is ever enrolled, which is the point: an earlier version scoped by app name and so folded any other curl on the host into the run. Under an out-of-process resolver (nscd) the wire query is sent in the daemon's context, outside that scope. No DNS record then appears on an otherwise-healthy pipeline. A DNS row reading FAIL beside a PASS result is that case, not a broken install. The table says so on the row itself.

selftest also prints the capture-honesty lines described under doctor --live before its table, so a run that never completed a request or that lost events to a full ring is not mistaken for a broken probe.

$ sudo s3tap selftest

See the CLI reference for every flag.

doctor metrics

doctor judges latency relative to the network RTT floor (the connection's smoothed round-trip, srtt_us, falling back to its lowest observed one), so a verdict means the same thing whether you are 1 ms or 70 ms from S3. This page explains how each metric is derived.

The RTT floor

Every latency span is judged as a multiple of a floor (in µs). The floor comes from a ladder of sources, most trustworthy first:

1. Per-operation join: each op is matched to its connection by sock_cookie (the join key) and judged against that connection's floor. This keeps a multi-region capture honest. A us-east op (~1 ms) and a cross-region op (~70 ms) get separate floors. They never share a blended floor that fits neither and hides a slow op behind the far one.
2. Per-region median: if the joined connection has no usable floor, doctor uses the median floor of the op's region (found through the join).
3. Pooled global median: the median srtt across the whole capture. This is the single headline number the parity oracle checks.
4. None: with no floor anywhere, the span is marked n/a and never judged. Doctor never reads green on missing data.

A connection's floor prefers srtt (the current smoothed round-trip time, the right value to subtract when isolating think-time, the server's own processing time) and falls back to min_rtt. The sentinel 0 (an unsampled or LRU-evicted socket) and implausibly huge values are dropped per value before the median.

Global rows

The latency rows above (TCP connect, both TTFB rows) are computed over the eligible ops only. An eligible op is non-partial, has status < 400 and is not delimitation: ambiguous (a second request raced the response, so its timing is not cleanly attributable). The reference oracle applies the same gate.

Neither the HTTP errors row nor the S3 status mix uses that population (throttling, server errors, client errors). Both are judged over the answered ops: every operation carrying an http_status, which is what sample.judged publishes for them. The eligibility gate would be nonsense here, because it drops a status ≥ 400 by construction, which is exactly what these rows count. Using the whole capture would be wrong the other way: it dilutes the rate by the ops nobody answered. That is what made doctor --json and scorecard --json report different error rates for one file. When some ops went unanswered the row says so on its own line ("of N ops, only M were answered"). When NONE were, it refuses to publish a rate at all. See NO RESPONSES.

Why cold resolve is never judged

The cold-resolve row is pure telemetry. It is printed with a · and never contributes to the verdict, not even under --strict. There is no honest envelope for it. It cannot be made RTT-relative like every other span on this page: the floor measures the round trip to the S3 endpoint, while the resolver sits on a different path doing recursion, so the floor is not its baseline. A same-region capture (a sub-millisecond floor, a routine 15 ms recursive resolve) would read as 30× RTT and warn on every run. An absolute millisecond ceiling is the other way out and it is exactly the invented number this tool refuses to use: it would fail a clean on-prem or WAN capture whose own path is simply far away. So the number is shown and left unjudged.

S3 per-op-class TTFB: the "money metric"

For each S3 op-class (GetObject, PutObject, …), doctor reports think-time = ttfb − RTT. This is the server-side work with the network time taken out. Each op is judged against its own joined floor. The per-op ratios are then medianed, so a class that spans two regions is still judged fairly. A class warns above 4× RTT.

A would-be-healthy class with fewer than 3 judged ops degrades to insufficient data rather than claiming a confident ✓. One fast op is not proof a class is fine. A clearly-slow class still warns even on one op (a real outlier is worth surfacing).

Tail, reuse, path, time-series

• Tail: p95 (≥ 20 samples) and p99 (≥ 100 samples) of the TTFB populations, judged against a wider envelope than the median line. Like the per-op-class rows, each op is scored against its own floor rather than the pooled one.

  The row carries two numbers, taken over two different orderings: value is the percentile of TTFB — a latency percentile, which is what the label says — and ratio_to_rtt is the percentile of ttfb / its own floor, which is the number the verdict gates on. Ranking the judgment by ratio is the point: on a capture spanning two paths, the slowest op in milliseconds may be the healthiest one relative to the path it took.

  Because they are order statistics over different orderings, they generally describe different operations, and no single floor relates them. The record therefore publishes no baseline_rtt_us for these rows: value / baseline would reconstruct a ratio that contradicts the published one. Use ratio_to_rtt against the threshold, and read value as the latency it says it is.

  A tail row also refuses a ✓ when the ops it could not floor are numerous enough to have occupied the tail themselves — above 1 - p/100 of the population it renders n/a instead, since a percentile computed from the survivors could hide exactly the ops that mattered. A ⚠ still stands: warning from a subset errs in the safe direction.

• Reuse rate: fraction of all non-partial ops on a reused connection (below ~80 % warns). Requires ≥ 5 ops to judge. It is deliberately not restricted to the latency-eligible set — an errored or ambiguous op still tells you whether the client reused a connection — but a partial op does not, and the record schema says so on the field itself: on a partial op the connection facts could not be attributed, so whether it truly reused is unknown rather than false.

• Path diagnosis: advisory rows from the extended tcp_sock fields: propagation floor plus jitter, send or recv bottleneck, the BDP / receive-window ceiling (BDP means bandwidth-delay product, the most data the receive window allows in flight) and loss shape.

• Time-series: from the in-flight sample stream: throughput ramp, bufferbloat onset and a loss timeline. Judged over sample streams, not connections. FYI, not advisory: these rows are marked · and never gate, not even under --strict, which is the difference between them and the path and throughput rows above. They answer "what happened and when" rather than "is this outside its envelope". They are also empty unless the capture was taken with --sample-interval-ms. An advisory row would exit 1 under --strict. These never do.

Verdict & exit code

The run rolls up to one verdict, mapped to an exit code for scripting:

MIXED PATHS is the newest of these and the least obvious. A pooled round-trip median across a 1 ms path and a 200 ms path fits neither, so judging an op against it produces a ratio that is wrong rather than absent — a 300 ms request reading "✓ 3.0×RTT". The floor is therefore withheld from every span that cannot be attributed to one path, and if that leaves nothing judged, the run says so. Two paths are detected either from distinct endpoint regions or from a bimodal round-trip spread, so an unlabelled capture with a near and a far connection is caught too.

A capture can span several paths and still be perfectly judgeable: when each operation joins its own connection, every one is scored against that connection's floor and the verdict is whatever those rows say. MIXED PATHS is reserved for the case where nothing was judged at all.

NO BASELINE is deliberately not 0, because a run that could not judge anything must not read green in CI. Treat 2 as "re-run me", not as a pass. Exit 3 is the --live counterpart: the workload ran but nothing was captured, usually for want of probe caps, so there is no report at all.

NO OPERATIONS is the same refusal for the other missing denominator, which is why it shares code 2. Both mean "nothing was judged", so a script can do the same one thing with either. The report line tells you which, since the remedies differ. The capture holds connections but not a single s3tap.operation/1 record, so the whole S3 half of the report was computed over an empty population. A Go or rustls client produces exactly that shape, as does any capture taken without the uprobe caps, because nothing decodes the HTTP layer. The network rows can be perfectly clean at the same time, which is precisely why this is not CHECKS PASSED: a green line would tell a CI gate that S3 looked fine when no S3 request was ever seen. It sits below ATTENTION in precedence, so a ⚠ from a connection-sourced check (retransmits, path) still wins and is never masked. The fix is to re-capture with --capture-plaintext and the uprobe caps (sudo s3tap setup --uprobes), which is what produces operation records at all. On the machine side the same run publishes as Unjudged rather than Healthy, alongside NO BASELINE, so a fleet ingest can tell "we looked and it was fine" from "there was nothing to look at".

NO RESPONSES is the third missing denominator. Operations WERE decoded here, but not one of them carries an http_status. Every request was still in flight when the capture ended, which is routine at Ctrl-C. The HTTP errors row counts its numerator over answered ops only, so its 0 there is a construction rather than a measurement, yet it used to print "0 / 5 ✓ healthy, all operations 2xx/204" and publish value: 0.0 beside sample.judged: 0, a 0/0 for any consumer computing a rate. That row is now n/a with NO value. It names the shape instead ("none of the N operations in this capture was answered"). The finding publishes Unjudged. --baseline carries the same rule: a metric that was judgeable in the baseline and is not judgeable now is reported as unjudgeable rather than as "unchanged", which is what "HTTP errors 20 → 0 · unchanged" used to claim about a capture that could not judge them.

It gets its own verdict rather than being folded into NO OPERATIONS because the two remedies differ. NO OPERATIONS says "re-capture with --capture-plaintext", which is wrong advice for a capture that decoded every request and simply never saw the responses. The fix there is to let the workload finish, or to capture for longer. Both share exit 2 all the same, since a script can do the same one thing with either. Like NO OPERATIONS it sits below ATTENTION, so a connection-sourced ⚠ still wins. --strict does not move it either.

This shape used to reach CHECKS PASSED with "no timeable operations" on the line. That reading was defensible for the latency rows and wrong for the run: the reliability rows had an empty population too, so a green line told a CI gate that S3 looked fine when no S3 response had ever been seen. It is the same refusal as the two verdicts above.

Exit 1 is a VERDICT and never a tool error. It means s3tap read your capture, judged it and found something outside its envelope. It never means s3tap itself broke. Everything in that second category gets exit 4, reserved for exactly this: an input that could not be read, an input that held no records at all, a --save target that already exists, a baseline file that could not be read OR that parsed to zero records, a kernel below the 5.8 floor or without BTF, probes that would not load and the argument checks s3tap makes itself (--timeout-secs, --requests, --concurrency, --live without --endpoint, --live with --from). Read 1 as "the workload has a problem" and 4 as "fix the invocation". The distinction is what makes exit 1 usable as a CI gate. A gate that cannot tell "your storage path regressed" from "you pointed me at the wrong file" is not a gate. For a while this one could not.

One seam is worth knowing before you script against this. Arguments rejected by the argument parser itself, rather than by s3tap, keep the parser's own exit 2: an unknown flag, an unparseable value, or a --live-only flag such as --save passed without --live. That collides numerically with the two verdicts that use 2. It is deliberate and harmless to read correctly, because the parser writes a usage block to stderr and produces no report at all, so nothing on stdout can be mistaken for a judgment. If your gate distinguishes 2 from 4, distinguish it on whether a report was produced, not on the number alone.

This table judges a capture that exists. The capture command itself exits 0 for an empty capture, with one exception that reuses code 3: an --app/--exe scope that never had a process in it. See run.

--strict widens the exit-1 case. An otherwise-healthy run that raised any advisory exits 1 too. It does not touch the missing-denominator verdicts: NO BASELINE, NO OPERATIONS, NO RESPONSES and MIXED PATHS all stay 2. Nor does it promote an FYI row, which is what separates FYI from advisory. --cost steps outside the mapping entirely: it is informational and always exits 0.

--baseline does not. A missing denominator OUTRANKS the diff, so a current capture that judged nothing exits 2 even under --baseline — the verdicts above really do stay 2 in every mode. Only when the current capture had a population does the code come from the diff below (1 on a regression, 0 otherwise). Keying it on the diff alone made 2 structurally unreachable, so a gate that lost its uprobe caps diffed "NO OPERATIONS → NO OPERATIONS", printed "NO REGRESSION" and exited 0.

Because --cost keeps exit 0 by design, its honesty has to live in the text. A capture that decoded no operation prints "no S3 operation records decoded, so request cost is unknown" and no total at all, rather than a $0.000000 that reads like a measured figure. "Data returned" is counted only for a GET whose body was observed to completion, so a transfer still in flight at Ctrl-C does not contribute its declared content_length as bytes that arrived.

The input must be a capture

doctor, advise and scorecard refuse an input that parsed to zero s3tap.* records. They exit 4 (tool failure) and print what they actually read, instead of rendering a verdict over nothing. A findings file is not a capture. Neither is an empty file, a truncated one, or any other log you happened to point at --from. The error names the count that diagnoses it: a doctor --json findings file lands entirely in the unknown-schema count, a truncated or non-JSONL file in the bad-line count, an empty file in neither.

This closes a real hole rather than a theoretical one. Before the guard, doctor --from findings.jsonl rendered every row as n/a and reported NO BASELINE with capture-tuning advice, so the operator re-captured forever against a file that was never a capture. Worse, advise --strict --from <empty> printed "no advisories" and exited 0. The CI gate had been dead for as long as the path was broken. Nothing said so.

analyze has always refused an empty trace the same way. --baseline now carries the same guard. It used to refuse only a path it could not open, so a file that opened and parsed to zero records was accepted as a baseline of nothing. It is refused with exit 4 too, on the same reasoning: a gate is not a gate if the reference side of the comparison can quietly be empty. See the --baseline section below.

Global rows are pinned to match the parity oracle, a separate reference implementation that recomputes the same checks. The S3-domain, tail, reuse, path and time-series rows are supersets. They can raise the overall verdict but they never change the parity-pinned per-row marks.

--baseline diff

doctor --baseline <capture.jsonl> compares the current run against an earlier one and flags regressions.

The baseline is a capture of records, not a findings file. It is re-analyzed from scratch, so it must hold the same s3tap.operation/1 + s3tap.connection/2 JSONL that --from takes. Feeding it the NDJSON of doctor --json does not work. A s3tap.finding/1 line is not a record, so the whole file parses to zero records and is refused. Two things produce a usable baseline: a normal s3tap --format jsonl capture, or doctor --live --save <file>, which writes the workload it just drove in exactly that format.

A baseline that holds no records is refused, the same way --from is. It opens the file, parses it and, on zero s3tap.* records, reports what it actually read and exits 4 rather than diffing against a baseline of nothing.

That closes the last instance of the pattern the zero-record guard was written for. Before it, the flag refused only a path it could not open. What a zero-record file cost you depended on the file. A doctor --json findings file printed one note on stderr (note: baseline: skipped N unparseable + M unknown-schema line(s)) and then read every current warn as brand new. An empty or truncated file had nothing to skip, so not even that note fired: the baseline held no checks, a healthy current capture raised no new-issue delta and the gate exited 0 against a file that was never a capture. A CI job could sit green for months on a baseline path someone had typo'd.

$ s3tap doctor --live --endpoint https://… --save baseline.jsonl   # record a good run once
$ s3tap doctor --from today.jsonl --baseline baseline.jsonl        # gate on a regression

--save writes a NEW file, mode 0600

--save creates its file with O_CREAT|O_EXCL|O_NOFOLLOW at mode 0600. Two consequences for the recipe above.

It refuses a path that already exists. Re-running the record step over the same filename fails rather than overwriting, so "record a baseline" is a once-per-filename operation. Delete or rename the old file first, or save to a new name. There is no --force. Refusing an existing path is the only version of this with no TOCTOU window, because the existence check and the create are one atomic syscall. The path is claimed up front, beside the rest of the usage validation, so a second run against an existing filename fails before a single packet is sent rather than after driving a full billable workload. If the run then fails before it has a capture, the reservation is released so the re-run is not refused by the empty placeholder it just made. The reason it matters is that doctor --live routinely re-execs itself under sudo, so this write commonly runs as root with the path taken from an argument. The old behaviour followed symlinks and truncated, which let any user who could pre-create the path aim a root-owned truncating write anywhere on the filesystem.

It is owner-readable only. A capture names your buckets, endpoint IPs, SNI hostnames, per-request timings and key hashes. That is a map of your storage. At the old default mode it was readable by every local user on a shared host.

A check that has vanished is treated as loss of signal (it gates as Unjudgeable), never as a green "Resolved". A baseline warn only counts as resolved when it was watched and the current capture can genuinely re-judge it. This stops a smaller or differently shaped capture from hiding a regression.

Public records

s3tap's capture emits stable, public JSONL records: one JSON object per line, each tagged with a versioned schema field. Every consumer reads these, so the record is the contract. A capture taken once can be re-judged offline, diffed or fed to a different tool.

Source of truth: the exact fields, types and (de)serialization live in the s3tap-schema crate. This page is an orientation, not the field-by-field spec. Consult the crate for authoritative shapes and to avoid drift.

The record kinds

Timestamps: emitted_at vs ts_ns

Every record the AGENT ships carries emitted_at: an RFC3339 wall clock (UTC, millisecond precision) for the moment s3tap wrote the record out — not for the traffic it describes. That is ts_ns, which is boot-relative monotonic and therefore comparable only within one host's capture.

emitted_at is the only wall clock in the pipeline, so it is the field a fleet ingest orders and ages records on across hosts. Two properties matter to a consumer:

• One clock read per drained batch, not per record. Every record flushed together carries the same emitted_at, deliberately: reading the clock per record would split one flush into N distinct emit times, which a consumer cannot tell apart from N separate flushes.
• It is absent on a record that never reached the emitter — one built in-process, or in a test — and on the two SECOND-ORDER records. s3tap.finding/1 declares the field but no producer stamps it, so it is omitted from the JSON entirely; s3tap.scorecard/1 has no such field at all. Both are deliberate: those records are derived from a capture rather than observed, and a wall clock in them would also make every golden non-reproducible. Do not order, age or dedupe findings on it — carry your own receive time, or join back to the first-order records the finding was derived from.

s3tap.operation/1

The per-request record. Key fields:

• Identity / join: op_id, sock_cookie (join key to its connection), app, bucket, key_hash. Note there is no region here — the endpoint's region lives on the CONNECTION (endpoint.region on s3tap.connection/2), reachable through the sock_cookie join.
• Timings (ns): tcp_connect_ns, ttfb_ns, download_ns.
• Semantics: verb, http_status, s3_op (op-class), content_length, aws_request_id.
• Eligibility flags: partial, connection_reused, delimitation (clean | ambiguous), dns (a nested resolve block with latency_ns, cache_hit).

Fields whose absent values are easy to misread:

• app.pid is 0 when the process is UNKNOWN, not when the pid is genuinely 0. It happens when s3tap never saw the connect, typically because it attached mid-flight. The close event's own tgid is deliberately not used as a fallback: it is stamped wherever the socket is torn down, routinely in softirq context, so it would name an unrelated task. A consumer that groups by pid should treat 0 as "unattributed" rather than as a real process.
• content_length is null in five distinct cases, all of which mean "we have no trustworthy body length", never "the body was empty": no Content-Length header at all, a Transfer-Encoding header (whose framing supersedes any declared length), duplicate Content-Length headers that disagree, a declared value above 5 TiB (the largest object S3 stores, the bound that keeps this field safe as a plain JSON number), or a response head s3tap never saw in full. It MAY be set when download_ns is null: a HEAD declares a size with no body to time.
• Five connection-scoped fields are placeholders on this record, never measurements. bytes_sent, bytes_recv and retransmits are always 0 here. srtt_us and lifetime_ns are always null. They are read off tcp_sock at CLOSE and are cumulative for the whole connection, while an operation record is emitted the moment its response completes, with the socket still open, so at that instant there is nothing to read. Deferring the record until close is not an option: a pooled connection can outlive the workload. So a 0 here means "not measured on this record", never "no bytes moved" or "no retransmits". Do not aggregate them. Byte volume or loss summed across operation records is zero by construction and reads as a healthy fleet. The real values live on the s3tap.connection/2 record for the same socket: join on sock_cookie. For per-op byte accounting use op_bytes_sent / op_bytes_recv. The fields keep their place at those constants because removing them is a /1 to /2 contract change for fields that carry no information either way. They are the first thing to drop when s3tap.operation next bumps.

s3tap.connection/2

The per-connection record, read off tcp_sock at close. Key fields:

• Identity: sock_cookie, app, endpoint (region, …).
• Floor / quality: srtt_us, min_rtt_us (true propagation floor), rttvar_us, retransmits.
• Volume / window: bytes_sent, bytes_recv, snd_cwnd, mss, delivery-rate.
• Setup: tls (handshake timing), dns, connect_failed.

Fields whose values are easy to misread:

• endpoint.via_vpce and endpoint.cross_region are always false today. s3tap does not yet derive either, so read them as not determined, never as no. A genuinely cross-region capture still reports "cross_region": false. (The advise check that keys on the flag therefore cannot fire on a real capture; it is kept for the record shape.)
• endpoint.region IS real — taken from the SNI, falling back to the observed DNS resolution. It is null only when neither was available.

sock_cookie is the join key between operations and their connection. It is derived from the kernel struct sock * but obscured with a per-run random key at emit time, so shipped records carry only a stable opaque id, never a real pointer.

s3tap.sample/1

A periodic in-flight snapshot (opt in with --sample-interval-ms): sock_cookie, ts_ns, srtt_us, min_rtt_us, snd_cwnd, bytes_recv, delivery-rate. It feeds doctor's time-series section, which is FYI and never gates.

Samples are not purely telemetry. Which part is which matters. They never enter the eligible-op population and never feed the parity-pinned close-time floor, so on a capture where a connection closed they change nothing. But when NO connection closed, which is the normal shape for a long-lived pool, they become the FALLBACK. The round-trip floor comes from their min_rtt/srtt. The retransmit denominator comes from their byte deltas. Every latency verdict in the report is then judged against a number the samples supplied. The baseline row says so on its face (baseline RTT (min_rtt, sampled)). Without --sample-interval-ms that same capture reports NO BASELINE and judges nothing.

s3tap.finding/1

The output contract, emitted by doctor --json, advise --json and scorecard --json: one finding per check or advisory, plus a run roll-up. Each carries a finding_id, domain, severity, verdict, the metric/value/threshold, baseline_rtt_us and ratio_to_rtt for latency checks, a bounded evidence sample of contributing ops and a scope (e.g. the s3_op for per-class rows). This is the format a fleet ingests.

What is stable and what is not

The answer has two parts, because the two halves of this record move at different speeds.

The envelope is a contract, at 0.x already. The field set, their names, their types and their encodings hold. Adding, removing or renaming a field is a schema change and bumps the tag to s3tap.finding/2, exactly as for the first-order records. Parse this shape and expect it to keep working.

The finding vocabulary is not frozen at 0.x. Which finding_ids exist, what a metric is called, what unit it carries, the summary prose and which severity a given condition earns: these move as checks are added and corrected, in a minor release, without a tag bump. The work since 0.7.0 has already renamed serialized_busy_s to serialized_busy_ms, added the advisor-run and scorecard-run rows and narrowed source_schema to name only the stream a finding reads. None of it has shipped yet, which is why the rule is written down now.

So write a consumer that parses the envelope, switches on severity (a closed enum in the envelope) and treats finding_id and metric as data that can gain members. A gate written that way survives a minor upgrade. One that matches on a hardcoded list of ids will need revisiting.

s3tap.scorecard/1

The observed-SLO rollup, emitted by scorecard --json: one row per (bucket, s3_op) group, describing the traffic that group actually saw. Key fields:

• Group key: bucket, s3_op (either may be null, which keeps unclassified traffic visible as its own row instead of dropping it).
• Reliability: ops (the denominator: ops that carried an HTTP status), errors, error_rate, status_counts (an object keyed by the stringified status code).
• Latency: ttfb_p50_ns, ttfb_p95_ns, ttfb_p99_ns plus latency_sample, the number of eligible ops behind them. p95 is null below 20 eligible ops and p99 below 100 — two different floors, because the worst 1% needs more samples to mean anything than the worst 5% does — so a tiny capture's maximum is never sold as a deep-tail estimate.
• Throughput / span: throughput_bytes_per_s (GET groups only, null otherwise) and window, the [ts_start, ts_end] the group's ops fell in.

These rows are descriptive telemetry, never a verdict. Latency has no absolute "good" without a per-group RTT floor, so the percentiles are reported as-is. The judgments computed beside them (the reliability taxonomy and the within-group p99/p50 tail ratio) ride the s3tap.finding/1 rails instead. So scorecard --json writes both kinds of line: the scorecard rows first, then the findings. A consumer that wants only the numbers reads the scorecard rows. A fleet gate reads the findings.

CLI reference

Generated from s3tap --help by book/gen-cli-reference.sh. Do not edit by hand.

s3tap

Observe TCP/S3 connections and operations via eBPF (waterfall/table/jsonl)

Usage: s3tap [OPTIONS] [COMMAND]

Commands:
  selftest   Prove the pipeline works on this host: load the probes, drive a few real S3 requests, and assert each capability (DNS/TCP/TLS/HTTP) produced a record. Prints a pass/fail table and exits non-zero on any failed capability
  doctor     Judge a capture's health: read `s3tap.operation/1` + `s3tap.connection/2` JSONL (stdin or a file) and report whether each latency span is healthy, relative to the connection's RTT floor. A PURE CONSUMER of the public records — no probes, no privilege; composes as `s3tap --format jsonl | s3tap doctor`. Exits non-zero when a metric is outside its envelope (⚠). EXCEPTION: `--live` drives + captures its own workload (loads eBPF, needs caps) instead of reading records
  advise     Optimization advisories over a capture: read `s3tap.operation/1` + `s3tap.connection/2` JSONL (stdin or a file) and report how the application USES S3 — client churn, missing parallelism, redundant re-fetches, throttling, caching go/no-go — attributed per process. A PURE CONSUMER of the public records (no probes, no privilege); composes as `s3tap --format jsonl | s3tap advise`. Health judgments live in `doctor`; this is optimization advice
  scorecard  Observed-SLO scorecard over a capture: read `s3tap.operation/1` JSONL (stdin or a file) and report each `bucket / s3_op` with the latency + reliability it ACTUALLY saw — request/error counts, the status-code mix, and TTFB p50/p95/p99. The passive analogue of a speedtest (your own production numbers, no synthetic probe). A PURE CONSUMER (no probes, no privilege); composes as `s3tap --format jsonl | s3tap scorecard`. `--json` emits `s3tap.scorecard/1` rows plus the gated `s3tap.finding/1` reliability judgments; `--strict` exits non-zero when any judgment fired
  analyze    Deep caching + prefetch report for ONE trace: the offline study, run on your workload. Reads a trace (s3tap JSONL, NormEvent JSON, or an IBM COS line — a file or stdin), runs the FULL retention ladder (LRU / ARC / S3-FIFO / OPT) in chunk mode plus the prefetch tradeoff, and prints a recommendation: cache or not, which policy, what size, and whether prefetching will help. Heavier than `advise` (seconds to minutes) — a PURE CONSUMER (no probes, no privilege). `--fast` runs retention-only (~12× quicker); `--json` emits the structured verdict. Composes as `s3tap --format jsonl | s3tap analyze`
  check      The easy front-end to `doctor --live`: probe an S3 object and print a one-line plain-language verdict. Give it a `bucket/key` (expanded to the S3 endpoint) or a full URL for a verdict on YOUR path (strict exit code). Omit the target for the zero-config check: a regional round-trip map, then a health check against a public AWS Open Data object in the nearest covered region (informational exit; `--map-only` prints just the map). Loads eBPF: the L7 rows want the uprobe caps (`sudo s3tap setup --uprobes`), the map needs only the base caps (`sudo s3tap setup`) — without them s3tap offers sudo itself and, lacking uprobes, judges the network floor only. Full flags: `doctor --live`
  setup      Grant this s3tap binary the file capabilities to load its probes WITHOUT sudo — the programmatic `setcap.sh`. Asks for sudo itself (once); re-run after every rebuild (caps live on the binary inode, which `cargo build` replaces)
  help       Print this message or the help of the given subcommand(s)

Options:
      --no-elevate
          Never self-elevate: when privileges are missing, fail with the normal permission error instead of re-running under sudo

      --format <FORMAT>
          Output format. Default: `waterfall` on an interactive terminal, `jsonl` when piped/redirected (so the machine path stays clean for scripts)

          Possible values:
          - jsonl:     One JSON record per line (s3tap.connection/2 + s3tap.operation/1)
          - human:     A compact human-readable line per record
          - waterfall: A phase-aligned latency timeline per operation
          - table:     One fixed-width row per operation, for scanning a live stream

      --include-loopback
          Include loopback connections (127.0.0.0/8, ::1). They are dropped in-kernel by default as noise — S3 traffic is never loopback. Useful against a local S3-compatible endpoint (MinIO, LocalStack, a test proxy)

      --dump-events
          Print a one-line summary of every raw EVT_* to stderr as it is drained. A diagnostic for confirming a probe FIRES end-to-end (especially the uprobes: getaddrinfo and the OpenSSL read/write family) — independent of whether the event then correlates. Records still go to stdout as usual

      --pid <PID>
          Restrict capture to these process ids (low-level escape hatch)

      --app <NAME>
          Restrict to an app by its EXE BASENAME, e.g. "python3" (resolved from /proc/<pid>/exe). Matched at exec (plus a startup /proc scan), and a tracked process's fork()ed children are followed in-kernel — so a pre-fork server's workers (gunicorn/uWSGI/Spark) are captured even though they never exec (best-effort: needs the sched_process_fork tracepoint; a warning prints if it's unavailable, where --cgroup is the churn-immune fallback). NB the process's own `comm` is NOT matched: any process can rename itself (prctl PR_SET_NAME) and so walk into the capture. A basename is still only as trustworthy as the paths local users can write, so on an untrusted multi-tenant host use --exe (exact path), --cgroup, or --pid, which can't be forged. A run scoped by name alone that never matched a process and captured nothing exits 3, so a script can tell a scope that missed from an app that was quiet.

          A trailing VERSION SUFFIX is stripped before the compare, so --app python3 matches /usr/bin/python3.12 and --app gcc matches gcc-11. That is required rather than a convenience: /proc/<pid>/exe is the symlink-RESOLVED path, so on every mainstream distro the interpreter you name only ever appears under its versioned name. The suffix must start at a `-` or `.` and be digits from there, so python311 and python3-shim stay out.

          Resolving ANOTHER user's process needs CAP_SYS_PTRACE, which `s3tap setup` does not grant (cap_dac_read_search bypasses DAC, not the ptrace check), so a capability-tagged s3tap matches only processes owned by the user running it. It warns once when that happens. Run as root, or scope with --pid, --cgroup or --container.

      --exe <PATH>
          Restrict to an exact executable path. Like --app, matched at exec / the startup scan and followed across fork(), so forked workers are captured. The path is resolved to its absolute form (via /proc/<pid>/exe) at both exec and scan time, so a relative `./server` invocation still matches an absolute --exe. Same exit-3 contract as --app for a scope that never matched anything

      --cgroup <ID>
          Restrict to a cgroup id (as `bpf_get_current_cgroup_id`; see a PROC_EXEC line under --dump-events). Churn-immune — all the cgroup's processes are in scope

      --container <ID|NAME>
          Restrict to a container by a (full) id/name substring of its v2 cgroup path. Best-effort: if it captures nothing, cross-check the resolved id against a PROC_EXEC `cgroup=` line under --dump-events and pass it as --cgroup instead

      --capture-plaintext
          Capture TLS PLAINTEXT (the HTTP request/response heads) via OpenSSL uprobes. Hooks SSL_write/SSL_read plus the OpenSSL 1.1.1+ size_t API, SSL_write_ex/SSL_read_ex, which is what modern clients call. The _ex pair is attached only where libssl exports it, so an older library still works. OFF by default and deliberately so: these are host-wide probes that see DECRYPTED bytes, including AWS SigV4 Authorization headers and `x-amz-security-token` STS tokens — usable credentials — for EVERY process on the host. The default run (connections + SNI) never buffers those. Enable only when you need L7/HTTP semantics and accept that exposure (also requires the uprobe caps: `sudo s3tap setup --uprobes`)

      --s3-endpoint <HOST>
          Treat this host as an S3-compatible endpoint so its bucket is resolved from the request — path-style `<endpoint>/<bucket>/<key>`, or `<bucket>.<endpoint>` virtual-hosted. Repeatable. Opt-in: s3tap otherwise recognizes only AWS hostname patterns and won't guess an arbitrary host is S3 (which could mis-split a non-S3 API's path). Accepts a bare host or a URL — scheme and port are stripped — e.g. `--s3-endpoint gateway.storjshare.io` or `--s3-endpoint https://minio.local:9000`. Only affects bucket/key decoding on the --capture-plaintext path

      --sample-interval-ms [<MS>]
          Emit periodic in-flight TCP samples (`s3tap.sample/1`) every N ms while a connection moves data — the EVOLUTION of cwnd/RTT/throughput/loss the close snapshot can't show. Library-agnostic (kernel TCP); OFF by default and pays ZERO overhead when unset (the probe isn't even attached). With no value it defaults to 100 ms; the minimum is 10 ms. jsonl only. Loud warning if used without a scope flag (--pid/--app/--exe/--cgroup): in TRACK_ALL the probe runs on every host-wide RX softirq

  -h, --help
          Print help (see a summary with '-h')

  -V, --version
          Print version

s3tap selftest

Prove the pipeline works on this host: load the probes, drive a few real S3 requests, and assert each capability (DNS/TCP/TLS/HTTP) produced a record. Prints a pass/fail table and exits non-zero on any failed capability

Usage: s3tap selftest [OPTIONS]

Options:
      --endpoint <ENDPOINT>  S3-style HTTPS endpoint to probe (default: real AWS S3). An unauthenticated request still exercises DNS→TCP→TLS→HTTP, which is all selftest checks [default: https://s3.amazonaws.com]
      --requests <REQUESTS>  Number of requests the driver issues [default: 3]
      --no-elevate           Never self-elevate: when privileges are missing, fail with the normal permission error instead of re-running under sudo
  -h, --help                 Print help

s3tap doctor

Judge a capture's health: read `s3tap.operation/1` + `s3tap.connection/2` JSONL (stdin or a file) and report whether each latency span is healthy, relative to the connection's RTT floor. A PURE CONSUMER of the public records — no probes, no privilege; composes as `s3tap --format jsonl | s3tap doctor`. Exits non-zero when a metric is outside its envelope (⚠). EXCEPTION: `--live` drives + captures its own workload (loads eBPF, needs caps) instead of reading records

Usage: s3tap doctor [OPTIONS]

Options:
      --from <FILE>                  JSONL of public records to judge: a file, or `-`/absent for stdin
      --no-color                     Disable ANSI color (also auto-off when stdout isn't a terminal)
      --json                         Emit `s3tap.finding/1` records (NDJSON, one per check + the run roll-up) instead of the human table — the machine/fleet-ingest format. Still reads from `--from`/stdin and keeps the same exit-code contract
      --baseline <FILE>              Compare the current capture against a baseline JSONL file and print a regression diff instead of the report. Latency checks are compared RTT-relative, so a capture from a different network still compares fairly. Exits 1 if the current capture regressed, 2 if the current capture had nothing to judge, 0 otherwise. A structured diff is future, so this conflicts with `--json` rather than silently writing a human table where NDJSON was asked for
      --strict                       Stricter gate: treat ADVISORY findings (e.g. a GET-throughput drop) as attention too, so they affect the exit code / fail the `--baseline` gate. Off by default — advisory metrics are deliberately unjudged (BDP/window-dependent)
      --cost                         Print an approximate S3 request-cost breakdown (per op-class) instead of the health report. Estimate only (AWS Standard us-east-1 request prices; data egress not included). Informational — always exits 0
      --brief                        Collapse the report to a one-line, plain-language verdict (plus the specific issues to look at on ATTENTION) — the easy read for non-technical users. Same verdict + exit code as the full report, just without the per-span table. Ignored under --json/--cost/--baseline
      --live                         Drive a small keep-alive workload against `--endpoint`, capture it, and report — instead of reading records from `--from`/stdin. Loads eBPF, so it needs the probe caps (`sudo s3tap setup --uprobes` for the L7/operation rows). Composes with `--strict`, and with ONE of `--json` / `--cost` / `--baseline` — those three each replace the report body, so they are mutually exclusive. Exit 3 if it captured nothing
      --no-elevate                   Never self-elevate: when privileges are missing, fail with the normal permission error instead of re-running under sudo
      --endpoint <URL>               (`--live`) The readable S3 endpoint/object the workload hits — REQUIRED under `--live`. An endpoint the workload can't read 2xx from will read unhealthy, by design. Repeatable: with `--rotate`, the requests cycle through all the given objects (one per request) so every fetch is COLD — defeats per-object caching for a cold-fetch measure
      --rotate                       (`--live`) Rotate through the `--endpoint` objects, one per request (round-robin), instead of hammering a single object. Use it to measure COLD-fetch latency: pass at least `--requests` DISTINCT, similar-size objects so none is revisited (and warmed) within the run. Without it, all requests hit the first `--endpoint`
      --requests <REQUESTS>          (`--live`) Number of keep-alive requests to issue (a median + a reuse signal; raise for the p95/p99 tail). Capped at 10000, and lower for a long `--endpoint` URL: the whole sequence is materialized as one curl argv [default: 12]
      --timeout-secs <TIMEOUT_SECS>  (`--live`) Capture budget in seconds. Must be 1..=3600 [default: 15]
      --save <FILE>                  (`--live`) Also write the (cookie-obscured) captured JSONL here — reusable later as a `--baseline`. Created new, mode 0600: the path must not already exist (`--live` often re-execs under sudo, so this write can be root's) and a capture names your buckets, endpoints and SNI
      --auth                         (`--live`) SigV4-sign the workload so a PRIVATE bucket returns 2xx. Creds: AWS_ACCESS_KEY_ID/SECRET (+ AWS_SESSION_TOKEN) env, else ~/.aws/credentials (AWS_PROFILE or `default`). Errors if no creds resolve. The secret is fed to curl off the command line
      --region <REGION>              (`--live --auth`) Region for SigV4 (default: AWS_REGION / ~/.aws/config / us-east-1)
      --s3-endpoint <HOST>           (`--live`) Treat this host as an S3-compatible endpoint so the per-`s3_op` rows resolve a path-style bucket/key (e.g. `--s3-endpoint gateway.storjshare.io`). AWS hosts are recognized natively; repeatable. Without it the global/floor/tail rows still judge, but the S3-domain rows are thin for a non-AWS gateway
      --concurrency <N>              (`--live`) Drive the workload over N parallel connections at once, to measure the path under CONCURRENT load — RTT inflation, retransmits, and the throughput/BDP ceiling only show up under contention, and low reuse only bites when several sockets compete. Each of the N workers runs the full `--requests` sequence on its OWN curl invocation (its own connection), so the doctor sees N connections in flight and total requests = N × `--requests`. Default 1 (serial keep-alive, one connection). Capped at 256 [default: 1]
  -h, --help                         Print help

s3tap advise

Optimization advisories over a capture: read `s3tap.operation/1` + `s3tap.connection/2` JSONL (stdin or a file) and report how the application USES S3 — client churn, missing parallelism, redundant re-fetches, throttling, caching go/no-go — attributed per process. A PURE CONSUMER of the public records (no probes, no privilege); composes as `s3tap --format jsonl | s3tap advise`. Health judgments live in `doctor`; this is optimization advice

Usage: s3tap advise [OPTIONS]

Options:
      --from <FILE>  JSONL of public records to analyze: a file, or `-`/absent for stdin
      --no-color     Disable ANSI color (also auto-off when stdout isn't a terminal)
      --json         Emit `s3tap.finding/1` records (NDJSON, one per advisory) instead of the human table. Same exit-code contract
      --strict       Gate the exit code on advice: exit 1 when any Warn or Advisory finding fired. Off by default — advice is not a failure
      --no-elevate   Never self-elevate: when privileges are missing, fail with the normal permission error instead of re-running under sudo
  -h, --help         Print help

s3tap scorecard

Observed-SLO scorecard over a capture: read `s3tap.operation/1` JSONL (stdin or a file) and report each `bucket / s3_op` with the latency + reliability it ACTUALLY saw — request/error counts, the status-code mix, and TTFB p50/p95/p99. The passive analogue of a speedtest (your own production numbers, no synthetic probe). A PURE CONSUMER (no probes, no privilege); composes as `s3tap --format jsonl | s3tap scorecard`. `--json` emits `s3tap.scorecard/1` rows plus the gated `s3tap.finding/1` reliability judgments; `--strict` exits non-zero when any judgment fired

Usage: s3tap scorecard [OPTIONS]

Options:
      --from <FILE>  JSONL of public records to score: a file, or `-`/absent for stdin
      --no-color     Disable ANSI color (also auto-off when stdout isn't a terminal)
      --json         Emit `s3tap.scorecard/1` rows followed by the gated `s3tap.finding/1` records (NDJSON) instead of the human table. Same exit-code contract
      --strict       Gate the exit code on the scorecard findings: exit 1 when any Warn or Advisory finding fired. Off by default — a scorecard is a report. Independent of exit 2, which a capture with nothing scoreable returns either way
      --no-elevate   Never self-elevate: when privileges are missing, fail with the normal permission error instead of re-running under sudo
  -h, --help         Print help

s3tap analyze

Deep caching + prefetch report for ONE trace: the offline study, run on your workload. Reads a trace (s3tap JSONL, NormEvent JSON, or an IBM COS line — a file or stdin), runs the FULL retention ladder (LRU / ARC / S3-FIFO / OPT) in chunk mode plus the prefetch tradeoff, and prints a recommendation: cache or not, which policy, what size, and whether prefetching will help. Heavier than `advise` (seconds to minutes) — a PURE CONSUMER (no probes, no privilege). `--fast` runs retention-only (~12× quicker); `--json` emits the structured verdict. Composes as `s3tap --format jsonl | s3tap analyze`

Usage: s3tap analyze [OPTIONS]

Options:
      --from <FILE>     The trace to analyze: a file, or `-`/absent for stdin. Auto-detects s3tap JSONL records, NormEvent JSON, or IBM COS text lines
      --fast            Retention-only: skip the prefetch pass entirely (fastest). Answers "which cache, how big" but not "will prefetching help"
      --deep            Run the full, expensive prefetch ladder too (Frequency/Co-occurrence/ Sequential/Adaptive). The Adaptive rung runs a per-size shadow cache and is minutes-to-much-more on a large trace. The default already gives the study's verdict (Markov structure + the shippable self-tuned overlay) fast
      --object          Analyze in object mode (whole objects) instead of the default 8 MB chunk mode. Chunk mode sees within-object/streaming structure; object mode is faster and matches how `advise` sizes
      --max-events <N>  Analyze at most N leading ops (a time slice) so huge traces stay bounded. 0 = the whole trace. Default 3,000,000 (study parity)
      --no-color        Disable ANSI color (also auto-off when stdout isn't a terminal)
      --json            Emit the structured verdict as one JSON object instead of the human report
      --no-elevate      Never self-elevate: when privileges are missing, fail with the normal permission error instead of re-running under sudo
  -h, --help            Print help

s3tap check

The easy front-end to `doctor --live`: probe an S3 object and print a one-line plain-language verdict. Give it a `bucket/key` (expanded to the S3 endpoint) or a full URL for a verdict on YOUR path (strict exit code). Omit the target for the zero-config check: a regional round-trip map, then a health check against a public AWS Open Data object in the nearest covered region (informational exit; `--map-only` prints just the map). Loads eBPF: the L7 rows want the uprobe caps (`sudo s3tap setup --uprobes`), the map needs only the base caps (`sudo s3tap setup`) — without them s3tap offers sudo itself and, lacking uprobes, judges the network floor only. Full flags: `doctor --live`

Usage: s3tap check [OPTIONS] [BUCKET/KEY|URL]

Arguments:
  [BUCKET/KEY|URL]  The S3 object to probe: `bucket/key` (expanded to `bucket.s3[.REGION].amazonaws.com/key`) or a full URL. A bare bucket with no key is rejected — there is no object to GET. OMIT it to run the built-in regional latency probes (a where-am-I-relative-to-S3 read)

Options:
      --region <REGION>      (bucket form) Target a specific region's endpoint, e.g. `eu-west-1`. Default: the global endpoint. The probe does NOT follow redirects, so a bucket OUTSIDE us-east-1 needs this (or a regional URL) — otherwise S3's region redirect reads as unhealthy. Also the SigV4 signing region under `--auth`
      --auth                 SigV4-sign the probe so a PRIVATE bucket returns 2xx (creds from env or `~/.aws`)
      --verbose              Show the full detailed report instead of the one-line summary
      --requests <REQUESTS>  Keep-alive requests to issue (default 12; raise for a p95/p99 tail) [default: 12]
      --triage               (target form) After the health check, sweep the probed regions and report how much round-trip is lower at the nearest one — with the standard remedies. The ms figures are measured (not geo-estimated); "nearest" is nearest of the few probed regions, not every S3 region. Adds the regional sweep time (a few seconds per probed region). Needs a target (the no-target run already sweeps the regions)
      --map-only             (no-target form) Print only the regional round-trip map — skip the follow-up health check against a public test object in the nearest covered region. Needs only the base caps (the map is connection-floor only; the object check drives the L7/uprobe path)
      --no-elevate           Never self-elevate: when privileges are missing, fail with the normal permission error instead of re-running under sudo
  -h, --help                 Print help

s3tap setup

Grant this s3tap binary the file capabilities to load its probes WITHOUT sudo — the programmatic `setcap.sh`. Asks for sudo itself (once); re-run after every rebuild (caps live on the binary inode, which `cargo build` replaces)

Usage: s3tap setup [OPTIONS]

Options:
      --uprobes     Also grant cap_sys_admin — required by the SSL/getaddrinfo uprobe paths (`--capture-plaintext`, `selftest`, the full `check`/`doctor --live` L7 rows). Near-root, and the plaintext path sees decrypted bytes host-wide — opt in deliberately (same posture as `UPROBES=1 ./setcap.sh`)
      --remove      Remove the file-capability grant from this binary instead
      --no-elevate  Never self-elevate: when privileges are missing, fail with the normal permission error instead of re-running under sudo
  -h, --help        Print help