Keyboard shortcuts

Press or to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Observability

Request ids, newline-delimited JSON logs, an access log, a Prometheus text endpoint, W3C trace context, and OTLP export. All of it sits on tracing, which the framework re-exports as arcature::observe::tracing so downstream code targets the pinned version through Arcature.

Nothing here installs itself. There is no global recorder, no global tracer provider, and no subscriber the crate registers on import — a JsonLog sink, a Metrics registry and a Telemetry pipeline are values the application holds and clones, and the subscriber is installed by a call the application makes from main. The rejected alternative is the usual one: a library that grabs the global subscriber when it is linked. It costs the binary the ability to choose, and it costs a test the ability to capture only its own output.

Turning it on

# Logs, request ids, the access log, the Prometheus registry, trace context.
arcature = { version = "0.1", features = ["observe"] }

# The same, plus OTLP span export. `otel` implies `observe`.
arcature = { version = "0.1", features = ["otel"] }
FeatureGives youPulls
observeinstall_logging, JsonLog, RequestId + RequestIdLayer, AccessLogLayer, Metrics + MetricsLayer, TraceContext + TraceContextLayer, redacttracing, tracing-subscriber, uuid
otelTelemetry, TelemetryBuilder, the observe::otel moduleobserve, opentelemetry, opentelemetry_sdk, opentelemetry-otlp, tracing-opentelemetry
Whereobserveotel
framework defaultonoff
framework fullstackonoff
generated applicationonoff, and arc new scaffolds nothing for it

otel = ["observe", ...], so it is an addition and never an alternative. It is an operator opt-in: four crates and the gRPC stack under them enter the graph, and most applications never enable it.

otel adds the OTLP span exporter and nothing else. The Prometheus endpoint belongs to observeobserve::metrics is not gated on otel, as the comment above the feature in Cargo.toml also says — so a default build already has it.

tracing-subscriber is pinned with registry and fmt and nothing else. ansi is off, so colour is unavailable rather than disabled; env-filter is off, so the filter is Targets rather than EnvFilter. json is off too, which is why the JSON formatter in this module is written by hand. That last one is not a saving, it is the design: the layer owns the serialisation, so redaction runs on every field with no way for a caller to opt out of it.

Installing the subscriber

// The first line of the generated application's `run`, before anything that
// might have something to say.
arcature::observe::install_logging(LOG_FILTER).map_err(std::io::Error::other)?;

install_logging(default_filter) -> Result<(), ObserveError>. Call it once, from main, before anything that might log. tracing events go nowhere at all until a subscriber exists: without this call the access log runs on every request and emits into the void, and a job that fails does so without a line anywhere. Nothing errors — the process is quiet, which is the worst way for logging to be broken, because it looks like nothing is happening.

The format is chosen by build profile, not by an environment variable, on the grounds that the shape of a log line is a property of the build:

BuildLayerRedaction
debug (cfg!(debug_assertions))tracing_subscriber::fmt, no ANSI, target shownnone
releaseJsonLog on StderrSinkevery field

Both write to standard error, so a log line never interleaves with what the process writes to standard output.

The redaction column is not a typo, and it is the single most misread thing in this module. install_logging installs JsonLog in release builds only; a debug build gets the fmt layer, which prints every field it is given, verbatim, including one named password. An application that wants redaction while developing composes its own subscriber with JsonLog in it (see the sinks) rather than calling install_logging.

default_filter is used when RUST_LOG is unset. The variable name is in FILTER_ENV, and it is RUST_LOG and not ARCATURE_LOG because every Rust operator already knows the name.

RUST_LOGdefault_filterResult
unset, or blankparsesthe default
set, parsesthe variable
set, does not parseparseswarning on stderr, then the default
unsetdoes not parsewarning on stderr, then info

A typo in a log filter must not stop a process from booting, and an application whose own hard-coded default does not parse has a bug that must not be silent either — hence a warning in both directions and a running process in both directions.

Targets rather than EnvFilter is a deliberate trade. EnvFilter brings a regex engine along for span-field matching that a web application almost never uses; Targets reads the target=level syntax people actually write — info, info,sqlx=warn, my_app=debug — and costs no additional dependency.

install_logging returns Err(ObserveError::Logging) if a global subscriber is already installed. That is a real error rather than a no-op: the second caller’s configuration is being discarded, and the honest response is to say so.

The generated application calls it on the first line of run, with a filter chosen the same way:

BuildLOG_FILTER
debuginfo,<app>=debug,arcature=debug
releaseinfo

What a log line looks like

One JSON object per line, no pretty printing, no trailing state. Every mainstream shipper reads that without configuration.

{"fields":{"client_ip":"203.0.113.9","duration_ms":7,"method":"GET","path":"/dashboard","request_id":"6f1e6f8c-6e5e-4a2b-9a24-6f3a2f0f1c77","status":200},"level":"INFO","message":"GET /dashboard 200 7ms","target":"arcature::observe::access_log","timestamp":"2026-08-23T10:15:04.220Z"}
KeyPresentValue
timestampalwaysRFC 3339 UTC, millisecond precision, always Z
levelalwaysTRACE / DEBUG / INFO / WARN / ERROR
targetalwaysthe emitting module path
messagewhen the event recorded onethe formatted message, lifted out of the fields
spanswhen the event is inside an entered spanspan names, outermost first
fieldswhen there is at least onethe event’s fields, plus inherited span fields

Keys are serialised from a BTreeMap, so they come out in alphabetical order — both at the top level and inside fields. Nothing depends on that; it is what a reader will see.

message is only another field to tracing. Lifting it to a top-level key is what makes the line readable to a human and indexable by everything else.

Span fields are folded into each event inside the span, with a nested span’s field beating its parent’s and the event’s own field beating both. The timestamp is written by hand — Howard Hinnant’s civil_from_days plus a format string — because chrono belongs to the database feature and a logging layer must not drag a database dependency behind it.

JsonLog::without_span_fields() drops the inherited fields and logs only what the event itself recorded. Smaller lines, at the cost of losing whatever the enclosing span was carrying.

Sinks

LogSink is one method, write_line(&self, line: &str), and the line arrives without its terminator. A trait rather than std::io::Write because a sink is shared across threads and must serialise whole lines: a writer that interleaved two events would produce unparseable output.

SinkUse
StderrSinkstandard error, one writeln! under the lock
CaptureSinkkeeps lines in memory; cloning shares the buffer

CaptureSink is how a test asserts on its own output without touching a global:

use arcature::observe::{CaptureSink, JsonLog};
use tracing_subscriber::layer::SubscriberExt as _;

let sink = CaptureSink::new();
let subscriber = tracing_subscriber::registry().with(JsonLog::new(sink.clone()));
tracing::subscriber::with_default(subscriber, || {
    tracing::info!(user = "ada", password = "hunter2", "signed in");
});
assert!(sink.transcript().contains("[redacted]"));

lines() returns the vector, transcript() joins it. A poisoned mutex is recovered from rather than propagated in both sinks: a panic in one log call must not silence every later one.

The request id

RequestId is a validated, low-cardinality identifier. RequestIdLayer resolves it once per request and puts it in request extensions, and every response carries it back as x-request-id — the wire-compatible name, with no X-Arcature-* prefix, so a reverse proxy and a client library both already understand it.

QuestionAnswer
Where fromthe inbound x-request-id header, if it parses; otherwise a fresh UUID v4
CharsetASCII alphanumerics plus - _ . : @ + / =
MaximumMAX_REQUEST_ID_BYTES, 128 bytes
Rejected inputempty, oversized, or a disallowed byte
On rejectiona fresh id is generated — RequestId::from_header never errors
Responsex-request-id, on every response the layer sees

Reusing the upstream value is what makes a trace survive the hop from a reverse proxy. Validating it is what stops the same header from becoming an injection point into every log index downstream: RequestId::parse_str enforces the allow-list, and hostile input is replaced rather than reported, because failing a request over a malformed correlation id would be a denial of service with no upside.

RequestIdError is Empty, TooLarge { size, limit } or InvalidChar, and only the parsing entry points return it.

How it reaches a log line

AccessLogService reads the id out of request extensions and records it as the request_id field on the access line it writes. That is the path, and it is one hop: the id reaches the access line because the access line writes it, not because anything is inherited.

Two consequences worth being exact about:

  • The order matters. RequestIdLayer must run outside AccessLogLayer or the id is not in extensions when the access line is written. The pipeline does this by construction — request id is stage 8 and the access log is stage 9 — and switching the request id off while leaving the log on produces lines with an empty id rather than an error.
  • A handler’s own tracing::info! does carry the request id. AccessLogService attaches its arcature.request span to the inner call with Instrument, so the span is entered exactly while that future is polled and a handler’s own events inherit request_id, method, path and client_ip. Instrument rather than a span.enter() guard: in async code a guard stays entered across every await point, including the ones where the task is parked and another request is running on the thread, which would attribute other requests’ lines to this one. tests/observe_request_span.rs pins both halves — the access line carrying the id, and a handler event inheriting it.

Both layers are off unless asked for:

Application::new()
    .request_id()
    .access_log()

The access log

One tracing event per request, at INFO, from arcature::observe::access_log. The message is "{method} {path} {status} {duration}ms" and the structured fields are:

FieldTypeValue
methodstringthe request method
pathstringuri.path() — the path only
statusnumberthe response status as u16
duration_msnumberwhole milliseconds, truncated
request_idstringthe resolved id, or "" if RequestIdLayer did not run
client_ipstringthe resolved ClientIp, or "" if nothing resolved one

That is the whole line. No request body, no response body, no headers, and no query string.

The query string is the interesting omission. It is the part of a URL that ends up in every proxy log on the way, and applications put credentials in it — an OAuth code, a PKCE code_verifier, a password-reset token. AccessLogService records uri.path() and tests/observe_redaction.rs sends a request whose query string carries a PKCE verifier and asserts it appears in none of the outputs, so uri cannot quietly replace uri.path() one refactor later.

An empty string rather than an absent field is deliberate for both optional values: a reader can tell “not known here” from “this field was never part of this log”.

The client address is a field and never the message. An IP address is personal data in most of the places this will run, and redaction decides per field name — so an address interpolated into the human-readable message would be past the only checkpoint there is. It is written as client_ip and passed through redact::apply("client_ip", ..) on the way, using the same string for both, so adding an address term to the deny-list would withhold it everywhere rather than everywhere-except-the-message. A unit test asserts the name written and the name asked about are the same string.

The layer sits outside the panic catcher, the body limit and the timeout, so a 500, a 413 and a 408 are all logged.

Metrics

Metrics is a registry the application holds and clones. Two Metrics values are two independent registries, which is what lets a test assert on exactly its own counters; cloning shares one, so the handle given to a middleware and the handle given to the /metrics route are the same set of series. There is no global recorder and no macro that reaches for one.

CallDoes
Metrics::new()empty registry, DEFAULT_BUCKETS
Metrics::with_buckets(&[..])same, with your bounds — sorted on the way in, non-finite values dropped
describe_counter/gauge/histogram(name, help)attach # TYPE and # HELP for a name
increment(name, labels, by)add to a counter series
set(name, labels, value)set a gauge series
observe(name, labels, value)record one histogram observation
counter_value(name, labels)Option<u64>, for assertions
render()the whole registry as Prometheus text
response()the same, as an HTTP response with the content type

Recording a series that was never described registers the name with the implied kind and empty help, so a metric is never lost for want of a description. Labels are a BTreeMap internally, so [("a","1"),("b","2")] and [("b","2"),("a","1")] are one series and not two.

The exposition

Prometheus text format, version 0.0.4 — the one every scraper reads, including OpenMetrics parsers, which accept it as a subset. PROMETHEUS_CONTENT_TYPE is text/plain; version=0.0.4; charset=utf-8.

# HELP http_requests_total Total HTTP requests handled.
# TYPE http_requests_total counter
http_requests_total{method="GET",status="200"} 3
# HELP http_request_duration_seconds HTTP request duration in seconds.
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{method="GET",le="0.005"} 1
...
http_request_duration_seconds_bucket{method="GET",le="+Inf"} 3
http_request_duration_seconds_sum{method="GET"} 0.11
http_request_duration_seconds_count{method="GET"} 3

Series are grouped by name and emitted in name order, because the format requires every sample of a name to sit under its one # TYPE line; interleaving names is a parse error, not a style choice. # HELP is written only when help text exists. Whole numbers render without a trailing .0, so a bucket bound of 1.0 is le="1". Label values are escaped for backslash, quote and newline.

tests/observe_prometheus.rs parses the rendered document with a purpose-written validator rather than a contains assertion, and the validator is itself tested against documents that break each rule, because a contains assertion cannot see whether the document around the substring parses — and a scrape a scraper rejects is silent.

The HTTP layer

MetricsLayer::new(metrics) records two series per request:

NameTypeLabelsHelp
http_requests_totalcountermethod, statusTotal HTTP requests handled.
http_request_duration_secondshistogrammethodHTTP request duration in seconds.

MetricsLayer::labelled(metrics, "/users/{id}") adds a route label to both.

DEFAULT_BUCKETS, in seconds: 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0 — the set the Prometheus client libraries ship, bracketing the range a web request lives in from a cached 5 ms answer to a 10 s timeout.

There is deliberately no path label taken from the request URI. A URI is unbounded input and an unbounded label value is how a scrape target runs out of memory. The rejected alternative — label with uri.path() and hope — costs the process its memory the first time a crawler walks a parameterised route. An application that wants a route dimension installs the layer per route or per group and names the template itself, and route is a &'static str precisely so it cannot be a value derived from a request.

Wiring it

The builder installs the layer but never the registry and never the route. ApplicationBuilder::metrics(registry) puts MetricsLayer at stage 9, which is where you want it — see What this module does not do for why stage 21 is not. Nothing in the pipeline constructs a Metrics, and there is no /metrics route unless the application adds one.

Hand-wiring the layer instead puts it among the user layers at 21:

use arcature::observe::{Metrics, MetricsLayer};

let metrics = Metrics::new();
let metrics_for_layer = metrics.clone(); // a clone shares the registry

let router = axum::Router::new()
    .route(
        "/metrics",
        axum::routing::get(move || {
            let metrics = metrics.clone();
            async move { metrics.response() }
        }),
    )
    .layer(MetricsLayer::new(metrics_for_layer));

An ordinary route, so it inherits whatever the application puts in front of it. A metrics endpoint usually wants an allow-list or basic auth, and that belongs to the application: the framework has no way to know which of the two is right, and shipping either one as a default would be wrong for half the deployments and invisible to the other half.

Constants live at arcature::observe::metrics::{DEFAULT_BUCKETS, HTTP_REQUESTS_TOTAL, HTTP_REQUEST_DURATION, PROMETHEUS_CONTENT_TYPE}. Only Metrics, MetricsLayer and MetricsService are re-exported at arcature::observe.

Trace context

TraceContextLayer reads W3C Trace Context off the inbound request, puts a TraceContext in extensions, and opens a span carrying the ids so log lines can be joined to the trace.

Inbound headers are untrusted. A malformed traceparent is discarded and a fresh root started rather than propagated, because half a trace id is worse than none: it silently corrupts every trace it joins.

Ruletraceparent
Lengthexactly 55 bytes, dashes at 2, 35 and 52
Version00 only; anything else, including the reserved ff, is rejected
Hex caselowercase only — accepting both would make two spellings of one id
Trace id16 bytes, all-zero rejected
Parent id8 bytes, all-zero rejected
Flagsone octet; bit 0 is FLAG_SAMPLED
Ruletracestate
Length512 bytes maximum
MembersMAX_TRACESTATE_MEMBERS, 32
Bytesprintable ASCII 0x200x7e only
Without a valid traceparentdropped — it describes a trace this request is not part of
On rejectiondropped; the trace itself still propagates

The member cap and the byte range are not pedantry: tracestate is attacker-controlled text that would otherwise be copied onto every outbound request the service makes. It is stored as the original text rather than a parsed member list, because the only operations performed on it are “carry it forward” and “prepend our own member”, and re-serialising a parsed form risks normalising away something a downstream vendor depends on.

The layer records three fields on an arcature.request span, which it does enter:

FieldMeaning
trace_id32 lowercase hex characters
parent_span_id16 lowercase hex characters
continued_tracetrue if an upstream trace was joined, false if this is a root

continued_trace earns its place: a service at the edge starts roots and one behind a gateway should not, and the difference is otherwise invisible.

It enters that span with span.enter() and holds the guard across the inner call, rather than attaching it to the future with Instrument. The guard is an entry on a thread-local stack and a future that yields does not unwind it, so which of a handler’s own events end up inheriting these fields depends on how the runtime scheduled the task. Correlate on lines that record an id themselves; treat span-field inheritance as a convenience, not a contract.

Nothing is written back onto the response. traceparent is a request header, and echoing it would tell a client the internal trace ids for no benefit.

For a downstream call, context.outbound_headers() returns a HeaderMap with a traceparent whose span id is a fresh child — which is what makes the next hop a child of this one rather than a sibling of the caller — plus the tracestate if one survived validation.

Root ids come from getrandom when the oauth feature has pulled it in, and otherwise from a SplitMix64 mix of the clock, a per-process counter and a stack address. Trace ids need to be unique, not unpredictable — they carry no authority and grant no access — so the fallback is adequate, and this module refuses to require a crypto dependency for a correlation identifier.

OTLP export

With otel, Telemetry is an OTLP-over-gRPC span pipeline the application holds:

use std::time::Duration;
use arcature::observe::{JsonLog, StderrSink, Telemetry};
use tracing_subscriber::layer::SubscriberExt as _;
use tracing_subscriber::util::SubscriberInitExt as _;

let telemetry = Telemetry::builder("checkout")
    .endpoint("http://collector:4317")
    .timeout(Duration::from_secs(5))
    .build()?;

tracing_subscriber::registry()
    .with(JsonLog::new(StderrSink))
    .with(telemetry.tracing_layer())
    .init();

// ... run the application ...

telemetry.shutdown()?;
PointBehaviour
Service namerequired by Telemetry::builder, never defaulted
TransportgRPC over tonic; the grpc-tonic feature is the only one enabled
Signaltraces only — opentelemetry and the SDK are pinned with trace and nothing else
Resource attributesservice.name, and that is all this crate sets
Runtimea Tokio runtime must already be running when build() is called
ShutdownTelemetry::shutdown flushes; dropping the value without it loses whatever is buffered
Samplingnone configured here, and nothing in this crate consults the sampled flag

A deployment reporting as unknown_service is a deployment nobody can find, so the name is a parameter rather than a default, and tests/observe_otlp.rs asserts it actually travels as the service.name resource attribute.

endpoint is worth reading carefully. When it is not called, build() does not call with_endpoint at all, so opentelemetry-otlp’s own default applies — which is the layer that reads OTEL_EXPORTER_OTLP_ENDPOINT. observe::otel::DEFAULT_ENDPOINT is published as http://127.0.0.1:4317, the address a local collector listens on out of the box, but nothing in build() passes it: it is documentation of the address, not a value this builder sends.

The pipeline must be built inside the async entry point and not in a static initialiser, because the batch processor spawns a background task. shutdown is fallible because a collector that is already gone cannot accept the final batch, and that is worth reporting rather than swallowing. A collector that never answers does not panic the application — there is a test that binds a port, drops the listener, and exports at the dead address.

tests/observe_otlp.rs runs a real TraceService gRPC server on a loopback port and decodes the protobuf the exporter wrote, so the assertions are against bytes that crossed a socket: a span arrives, a parent and child share one trace id, a child names its parent’s span id, a three-level nesting arrives as a chain rather than a fan, and a span field arrives as an attribute.

The two id spaces meet, under otel

TraceContextLayer and the OTLP exporter both deal in trace ids, and joining them is the point of a distributed trace.

When otel is on and the request carried a usable traceparent, the layer calls set_parent with a remote SpanContext built from the parsed ids, so the span it exports is part of the caller’s trace rather than the root of a new one. The tracestate travels with it when the exporter accepts it. The traceparent is also still recorded as the trace_id and parent_span_id attributes, which is what correlates a log line.

This did not always hold, and the failure was invisible from the log side. The layer parsed the header and opened a span carrying the ids as fields, and never called set_parent — so tracing-opentelemetry minted a fresh trace id for the exported span. A request arriving with a traceparent started a new trace at this service, and the two halves never met in the backend, while every log line looked correct. an_incoming_traceparent_becomes_the_exported_trace_id in tests/observe_otlp.rs reads the id off the span the collector holds rather than off a log field, because reading the field would pass against the bug.

Two details worth knowing. The parent is only set when the trace was continued: inventing a remote parent for a root would have a backend draw an edge to a span that never existed. And set_parent is called before the span is entered, because it refuses with AlreadyStarted afterwards and would be a silent no-op.

Without otel, none of this compiles in, and the layer is what it always was — correlation in your logs against the ids your upstream used.

Redaction

Logging is the most common way a secret escapes a process, because a log line is written once and then copied everywhere — to a file, to a shipper, to a third-party index, into a support ticket. So the defence is a property of the writer, not a rule the caller remembers: JsonLog asks redact::is_sensitive about every field name it is about to serialise, and a field that matches is written as REDACTED, the fixed marker "[redacted]". A marker rather than an omission, so a reader can tell “this field was withheld” from “this field was never recorded”.

Two mechanisms are at work and they are not the same size. The first is that the framework’s own layers record structured fields and never format a secret into a message string: the access log records method, path, status and duration, and the metric labels are a method, a status and a &'static str. That mechanism holds everywhere. The second is the deny-list, which covers a field an application adds — and it is much narrower than it first looks.

Which sinks it reaches

The deny-list is consulted in exactly two files: json_log.rs, which asks is_sensitive about every field of every event and every span, and access_log.rs, which calls redact::apply on the one client_ip value. metrics.rs, otel.rs and trace_context.rs never call either.

ChannelRedactedNote
JSON log, event fieldyesthe name is matched, the value is dropped
JSON log, span fieldyesredacted when the span is created, so the folded copy is already safe
JSON log, message stringnothe deny-list catches the field, not the sentence
Debug-build console (fmt)noinstall_logging only installs JsonLog in release builds
Metric label valuenoescaped for the exposition format, not redacted
Exported OTLP span attributenotracing_opentelemetry has its own visitor

The last two are pinned by tests that assert the leak, named so nobody mistakes them for tests of a working defence:

  • a_secret_recorded_as_a_span_field_reaches_the_collector_in_full — the same field the JSON layer wrote as [redacted] leaves the process in plaintext, over the wire, to a collector.
  • a_secret_recorded_as_a_metric_label_is_rendered_in_full — a series labelled with a session id publishes it on /metrics.

Both assert today’s behaviour so that closing either gap is a visible change rather than a silent one. If either starts failing because the value is now redacted, the defence has been extended and the test is to be deleted.

Until then: treat a metric label and a span attribute as unredacted channels. Keep the value out of them, or record it as a type whose own Debug renders redacted — the JSON visitor’s fallback formats through Debug, so a secret newtype protects itself in every channel that formats one. A label value is also a series dimension, so a secret used as one is usually an unbounded-cardinality bug as well.

How a name is matched

pub fn is_sensitive(field: &str) -> bool

Three steps, in this order:

  1. Every - and every . becomes _.
  2. Every other character is ASCII-lowercased.
  3. The result is tested for containment of any needle in DENY_LIST.

Substring rather than exact name, on purpose: password, user_password and db.password are all the same mistake, and a deny-list that only catches the spelling someone thought of is not a deny-list. False positives cost a debugging session; false negatives cost a credential.

Separator folding, also on purpose: an HTTP header is x-api-key, an OpenTelemetry attribute is http.request.header.authorization, a struct field is api_key. One secret, three spellings, and a needle written with _ is a substring of only the third. Every needle uses _ and none contains - or ., so folding can only ever match more.

DENY_LIST is public, sorted and lowercase — a unit test asserts the last two — so an application can check its own field names against it:

access_tokenapi_keyapikeyauth
bearerbindbodycache_value
cardcookiecredentialcsrf
cvvid_tokenotppassphrase
passwdpasswordpayloadpin_code
private_keypwdrefresh_tokensecret
session_idsignaturesql_argstoken
verifier

Because the test is containment, ordinary names collide with short needles and are redacted:

Field nameRedacted by
author, authorityauth
wildcard, discard, cardinalitycard
bindingbind
body_bytesbody
token_counttoken

That is the trade the module chose, stated so it is not a surprise at three in the morning when a field reads [redacted] and nothing is wrong.

What is not folded: any separator other than - and .. A space, a slash, a colon and a camelCase word boundary all survive step 1, so a multi-word needle cannot match across them. Non-ASCII case is not folded either — a field name is not expected to contain non-ASCII, and folding Unicode would widen the surface without widening the protection.

redact::apply(field, value) is the borrowing form: it returns REDACTED or the value unchanged, so nothing is copied for a field that is allowed through. is_sensitive and REDACTED are re-exported at arcature::observe; DENY_LIST and apply live at arcature::observe::redact.

The disclosed gap: camelCase multi-word names

tests/observe_redaction.rs pins it by name:

the_deny_list_is_written_in_snake_case_and_therefore_misses_camel_case_spellings

The list is written in snake_case, and camelCase has no separator to fold. privateKey lowercases to privatekey; the needle is private_key; there is no match. The test walks the spellings side by side:

SpellingResultWhy
x-api-key, X-Api-Keyredacted- folds to _, matching api_key
http.request.header.authorizationredacted. folds, and auth matches anyway
apiKeyredactedapikey is on the list as its own needle
accessTokenredactedthe single-word needle token matches
privateKeynot redactedprivatekey does not contain private_key
sessionIdnot redactedsessionid does not contain session_id

The exposure is narrower than the headline. Only multi-word needles are reachable this way, single-word needles catch most camelCase spellings anyway, and Rust field names are snake_case — so what is left is an application that records a JSON body’s keys under the names the client chose. If that is your application, check the names against DENY_LIST in a test of your own, or normalise them before recording.

As with the leak tests: if privateKey starts coming out redacted, the matcher has been widened and that test should be narrowed to whatever spelling is still missed, or deleted.

A list of names cannot see values

The other limit is structural, and it also has a test: a_secret_under_a_field_name_nobody_denied_is_logged_in_full. A field called note carries a password in one handler and a postcode in the next. No list of names can catch that.

Nor can any formatter undo a secret a caller has already interpolated into a message string. tracing::info!("signing in {password}") is past every checkpoint there is by the time the layer sees it. Record fields, not sentences.

The defence for both is the same one the framework applies to itself: its own layers never record a field whose contents they have not chosen. The list of what that means in practice, from the module’s own documentation —

  • Request and response bodies. Method, path, status and duration; never the payload.
  • SQL bind values. A query’s text may be recorded; the parameters bound into it may not, because that is where the row data lives.
  • Cache values. Keys are loggable and are logged; values are not.
  • Credentials of every kind — passwords, password hashes, API keys, bearer tokens, OAuth access and refresh tokens, PKCE verifiers, CSRF state, session identifiers, cookies, and Authorization headers.
  • Email bodies and recipients’ message content. A send is recorded as an event with a message id; the letter is not.
  • Job payloads. A job is recorded by name, queue and attempt count.

— holds for the framework’s own layers in every channel. tests/observe_redaction.rs drives one request carrying a password, a bearer token, a session cookie and a PKCE verifier through the whole stack — request ids, access logging, metrics, trace context — with a log sink, a metrics registry and a live OTLP collector capturing at once, then searches the log transcript, the metrics exposition, the exported span attributes and the exported resource attributes for each secret’s value. It also asserts that each channel captured something, because a harness that silently captured nothing would pass every absence assertion in the file.

Stable span names

Seven &'static str constants at arcature::observe, with is_stable(name) and ALL to iterate them:

ConstantValueOpened by the framework
REQUESTarcature.requestyes — AccessLogService and TraceContextService
DB_QUERYarcature.db.queryno
CACHE_GETarcature.cache.getno
JOB_HANDLEarcature.job.handleno
PAGE_RENDERarcature.page.renderno
EVENT_LISTENERarcature.event.listenerno
SCHEDULE_TICKarcature.schedule.tickno

Six of the seven are reserved names rather than spans anything currently emits: no code in the crate opens them today. They are here so that instrumentation added later, in the framework or in an application, agrees on one spelling instead of inventing a second. arcature.request is opened twice when both layers are installed — once by each — so a spans array can carry the name more than once.

What this module does not do

Install anything on its own. No global subscriber, no global recorder, no global tracer provider. install_logging is a call the binary makes.

Add a /metrics route, or protect one. The registry and the layer are values; routing and access control are the application’s.

Use .metrics(..), not .layer(MetricsLayer::new(..)). Both compile. Only one of them counts a refused request.

A user .layer() lands at stage 21 — inside the body limit (12), the timeout (13), maintenance (14) and the rate limiter (15). A request refused with a 413, a 408, a 503 or a 429 never reaches stage 21, so a counter installed there does not see it. The request total then quietly means “requests that got through admission”, the access log at stage 9 disagrees with it, and the gap is widest under exactly the load an incident is about.

ApplicationBuilder::metrics(registry) installs the same layer at stage 9, beside the access log, where it sees what the access log sees.

let metrics = Metrics::new();

let app = Application::<AppState>::new()
    .routes(routes())
    .rate_limit(RateLimit::per_minute(60))
    .metrics(metrics.clone())   // stage 9: counts the 429s too
    .build();

tests/observe_metrics_stage.rs pins the difference: the same request through the same limit is counted at stage 9 and missed at stage 21, and both placements agree on a request that is served. The second of those asserts the gap on purpose, so that anyone who later moves user layers outside the admission stages learns it from a failing test rather than from a graph.

Use .trace_context() for the same reason. It installs TraceContextLayer at stage 8, beside the request id and outside the admission stages, so the access line for a 429 carries the caller’s trace id. Through .layer(..) it would land at 21 and a refused request would produce log lines with no trace on them at all — and a refused request is one somebody is very likely to go looking for.

tests/observe_trace_stage.rs pins that pair the same way the metrics tests do, including the negative: a user layer is asserted to miss the refusal, so moving user layers later fails a test rather than quietly costing traces.

Redact in debug builds. The fmt layer prints fields verbatim.

Redact metric labels or OTLP span attributes. See the table above.

Sample. TelemetryBuilder sets no sampler and reads no sampling environment variable, so whatever the SDK’s own default does is what happens. TraceParent carries and preserves the sampled flag, but nothing in this crate consults it to decide whether to record or export.

Export metrics or logs over OTLP. Traces only. opentelemetry and the SDK are pinned with the trace feature and nothing else; metrics leave as Prometheus text or not at all.

Register a propagator. The inbound join is done directly with set_parent rather than through a global TextMapPropagator, and outbound headers come from TraceContext::outbound_headers. Nothing installs a global propagator, so a third-party client that expects to find one injects nothing.

Write to a file, rotate, or ship. Both formats go to standard error. The process manager owns the file, and every mainstream one already does this better than a library could.

Throttle or deduplicate log lines. A chatty target is what the filter is for.

Provide error tracking. No Sentry, no crash reporter, no aggregation. An ERROR line with structured fields is what the module produces; turning that into an incident is a shipper’s job.