Signal Calibration: how the confirmation score
behaves after publication

What calibration is, and why it is a harder test than accuracy

Calibration asks whether a stated confidence matches what actually happens. A forecaster is calibrated if, across all the times they said “70%”, the thing happened about 70% of the time. Accuracy asks a cruder question — how often the call was simply right. The two come apart constantly: a forecaster who says “51%” on every coin flip is perfectly calibrated and useless, while one who says “95%” and is right 60% of the time is more accurate than chance and still dangerous, because anyone sizing a position off the 95% is taking far more risk than they think.

For anything you act on, calibration is the property that matters more. Accuracy tells you whether a number is any good on average; calibration tells you whether the number means what it says — and position sizing, stop distance and expected value all run off what the number says.

How to read a reliability diagram

The classic calibration chart — a reliability diagram — puts the predicted probability on the x-axis and the observed frequency on the y-axis, one point per bucket of predictions. A perfectly calibrated forecaster's points sit on the 45° diagonal. Then:

• Points below the diagonal → over-confident. You said 80%, it happened 60% of the time. This is the failure mode that loses money, because it is the one that makes you size up.
• Points above the diagonal → under-confident. You said 60%, it happened 80% of the time. Costly in opportunity, not in drawdown.
• A flat line → the score carries no information. Every bucket resolves at roughly the same rate, so the number moving from 0.5 to 0.7 told you nothing.
• Wide whiskers → you cannot tell yet. Each bucket is an estimate from a finite sample; the 95% interval is the honest width of your ignorance.

Illustration only, not our data: if a model emitted 1,000 forecasts at “90% confident” and 600 of them resolved correctly, that model is 60% accurate and badly over-confident — and a trader sizing on the stated 90% would be running roughly three times the risk they had budgeted for. That gap, not the 60%, is what a calibration chart is built to expose.

Why this page deliberately is not a reliability diagram

A reliability diagram needs a predicted probability on the x-axis. We do not have one. The confirmation engine emits HIGH and MEDIUM — ordinal labels for how strongly the underlying layers agreed — and the regime filter emits a 0–1 score that is a filter threshold, not a probability. Drawing a diagonal against either would invent a claim the system never made. So chart 1 below is a bar chart over labelled buckets, and the only question it asks is the weaker, answerable one: does a higher label correspond to a higher observed hit rate? On the current data the answer is documented below, and it is not the flattering one.

Observed hit rate by score bucket in-sample

HIGH and MEDIUM are categorical confirmation tiers, not predicted probabilities. The regime value is a filter score, also not a probability. This chart asks only whether higher labels or score buckets correspond to different observed hit rates. Whiskers are 95% Wilson intervals; rates are suppressed below the endpoint's minimum sample.

Reading the buckets above, with the numbers the endpoint returned

The chart is live, so it moves. The figures in this section were read from GET /v1/signals/analytics/calibration on 2026-08-26 and are frozen in the text so the reasoning can be checked against what the API said that day. Compare them with the live table above — a difference is expected, and checking it is a legitimate way to audit us.

1. The confirmation tiers are currently inverted

HIGH sits below MEDIUM, and HIGH's interval spans 50% — on this window it is indistinguishable from a coin flip despite nearly 2,500 observations. The label is therefore not doing the job its name implies. We are not going to describe that as “HIGH being selective”: the plain reading is that we do not currently have evidence the HIGH label identifies better calls than the MEDIUM one, and it is published here in that order.

2. The regime score, read as if it were a probability

The regime_flip buckets are the closest thing on this page to a calibration test, because each bucket carries a mean filter score alongside its realized rate. Reading the score as a probability is exactly the mistake the label warns against — and here is what that mistake would cost you:

If you treated 0.59 as “59% likely”, you would have been over-confident by about 11 points; at 0.65, by about 16. On a reliability diagram both points would sit well below the diagonal. Two things follow, and both are worth stating plainly:

• The score is a filter threshold, not a probability, and the gap above is the concrete reason we refuse to relabel it as one.
• Both buckets have negative expectancy and realized rates below 50%. Moving from the lower bucket to the higher one buys 1.5 points of hit rate on a rate that is under a coin flip either way. regime_flip is published as a weak-to-negative signal, it is excluded from the headline win rate for that reason, and this table is why.

The two outer buckets (0.45–0.55 and 0.75–1.00) returned n=0 on this window. They are rendered as empty rows rather than dropped, so a bucket with no data never looks the same as a bucket that was quietly removed.

Why the n here is far larger than the n on the headline stat

A sceptical reader should notice that this page shows n in the thousands while /v1/stats reports a headline sample in the dozens. Both are real; they count different things, and the difference is the single easiest place for a data vendor to mislead.

• This page counts outcome rows. A confirmation call is re-emitted every few minutes while it persists, and each surviving row is scored at three horizons (4h, 12h, 24h). One decision that stays live for hours can therefore contribute dozens of rows — useful for slicing behaviour, worthless as a count of independent evidence.
• The headline counts distinct calls. Consecutive same-symbol, same-direction confirmations are collapsed into one episode; a gap over 12 hours starts a new one. On 2026-08-26 that was n=89 calls over the trailing 30 days at 24h, versus n=13,703 MEDIUM rows over 90 days here.

Quoting the row-level number as a sample size would shrink every confidence interval by roughly an order of magnitude and make each figure look far more settled than it is. So the headline stays on the small denominator even though it is the less impressive one, and the row-level counts stay here, labelled as rows. The full rule is on the methodology page.

Forward holdout since scorer freeze …

The scorer-freeze timestamp is 2026-06-30. Calls generated on or after that timestamp are tracked separately. The API suppresses the overall rate until there are at least 40 outcomes spanning at least 7 calendar days; the HIGH tier also requires 30 observations.

This is stronger evidence than the in-sample window, but it can still represent a narrow market regime. It is a signal-outcome record, not a filled-trade or net-return backtest.

Why a freeze date, and why a calendar-span gate. Freezing means the scoring rules were fixed in code before the data existed, so nothing after the timestamp can have been tuned to it — the difference between a claim and a test. The span gate exists because a minimum sample size alone is not enough: forty outcomes accumulated over two days are forty views of a single market move, not forty independent tests. That requirement was added after a database rotation reset the tables and left a two-day window which would otherwise have been publishable. Below the gate this block reports the word “accruing” and no percentage at all.

The holdout leads with the 24-hour horizon only — the mature outcome — and never blends in or falls back to a shorter horizon that happens to read better. It is counted in distinct calls, not feed rows. The payload also carries an is_distinct_from_insample flag so you can check whether the holdout population still overlaps the in-sample one; the line below reports whichever answer the endpoint gives.

The freeze timestamp is defined in code (SCORER_FROZEN_TS). When the forward sample is still thin, this block reports "accruing" rather than quoting a noisy percentage. When our in-sample walk-forward showed no out-of-sample directional edge in the raw features, we published that too.

Performance by horizon in-sample

Compare observed hit rate, Wilson interval, direction-adjusted mean move, and profit factor at each horizon. A point estimate above 50% is not meaningful when its interval crosses 50%. All figures are pre-fee signal outcomes, not strategy returns. The dashed line marks 50%; bars are hit-rate with 95% CI whiskers.

Three traps in the table above — including one of ours

Figures in this section were read from GET /v1/signals/analytics/edge?type=smart_money_confirm on 2026-08-26 (30-day window, minimum sample 30) and frozen in the text. The live table above will have moved.

Trap 1 — an interval that clears 50% in the wrong direction

At 48h the win rate was 33.7% on n=1,197, interval 31.1%–36.4%, profit factor 1.00. That interval lies entirely below the coin-flip line, so it is not noise — on that window, at that holding period, the signalled direction was wrong about twice as often as it was right. It is on the chart, it is in the table, and it is not annotated away. Note also that a profit factor of exactly 1.00 alongside a 33.7% hit rate is a reminder that hit rate and payoff are different questions: the wins were large enough to offset twice their number in losses, and the expectancy still rounds to +0.008%, which is indistinguishable from nothing.

Trap 2 — the 7-day row, which we do not believe

The 7d row currently reads 89.5% on n=797 with a profit factor of 39.4 and a mean direction-adjusted move of +17.7%. That is the most impressive number anywhere on this site and we are telling you not to use it. Four things are wrong with it as evidence:

• It is roughly forty points out of line with every neighbouring horizon (45.2% at 72h, 33.7% at 48h). Genuine edge does not usually appear only in the last cell of a table.
• A 7-day outcome only exists for signals that survived a full uninterrupted week of the resolver's uptime, so the row over-represents the periods when infrastructure was healthy rather than a random slice of market.
• The book on this window is 89.2% long (measured across the blended 4h–24h horizons; the endpoint does not publish a per-horizon direction split, so we cannot state the 7d row's own share). Over a week, in a rising tape, a mostly-long book wins most of the time with no skill involved whatsoever.
• The drift baseline — the skill-free reference that would tell us how much of this is simply the market moving — is currently reported by the endpoint as not computable (the symbol set is too wide or the price source was unavailable). The payload says so explicitly rather than dropping the caveat.

Read it as a demonstration of why single cells of a table are unsafe, not as a result. It is the number a less careful vendor would put on a billboard.

Trap 3 — direction concentration

Over the blended 4h–24h horizons on the same window the book splits 89.2% long / 10.8% short, and the two sides did not behave alike: longs 47.8% on n=4,831 rows, shorts 57.4% on n=582. When a book is that one-sided, the aggregate hit rate is mostly a measurement of the tape over the window. The honest correction is a matched constant-direction baseline — what a position that simply held the same side would have returned over the identical window — and the endpoint exposes baseline_wr and excess_wr_pts for exactly that purpose. When the baseline cannot be computed, both fields return null and a baseline_note explains why. They are never filled with an estimate.

This is why the site's headline win rate is quoted at a single mature horizon (24h) on distinct calls, rather than as the best number available across six horizons. Six horizons × several signal types × several windows is a large grid, and some cell of it will look significant by chance alone.

What these numbers do — and do not — license you to conclude

The strongest claim the data supports is narrow: a rule fixed in advance has stayed above a coin flip on this venue set over this stretch of market, before costs. That is a great deal weaker than an edge, and we would rather you hold us to the weaker claim.

Our own research supports the caution. A pre-registered cross-sectional ranking strategy cleared its exploration gate at +3.1 basis points per period, then inverted on its one-shot confirmation window to a 0.391 win rate, profit factor 0.33 and −67.6 basis points per period on n=46, with a permutation test at p=0.60. Under the pre-registration's honest-failure clause it was killed without touching the reserved holdout, which remains unused. A leakage-free walk-forward on the raw derivative and whale features found no out-of-sample directional edge at all. Those write-ups exist for the same reason this page does; the methodology page lists them.

Calibration uses the last 90 days of signal_log and signal_outcomes at the 4h, 12h, and 24h blend horizons. The decay table uses the edge endpoint's default 30-day window across 4h through 7d. Analytics are computed on demand and cached for 2 minutes; the outcome worker resolves eligible horizons about every 10 minutes.

• In-sample means the scoring thresholds were tuned within the window shown — treat it as descriptive, not predictive.
• Figures are pre-fee. Taker fees and slippage will reduce every number here; see the per-risk equity curves on the performance page for net-of-cost views.
• Signals were generated live, not backtested — each was published in real time and scored against actual subsequent price.
• Our own long-panel walk-forward found no shippable directional edge out-of-sample in the raw features. The product's value is breadth of real-time aggregation and transparency, not a magic win-rate.
• Past performance does not guarantee future results. Nothing here is financial advice.
• Outcome rows are not independent observations. The n on this page counts rows, and one live call contributes many. The headline sample on /v1/stats collapses those into distinct calls and is far smaller; treat the small number as the honest one.
• The exit is a clock, not a decision. An outcome is stamped at its horizon regardless of the path taken to get there, so a call that went heavily against you before recovering scores identically to one that never did. No stop is modelled.
• Long horizons are survivorship-filtered. A 7-day outcome only exists where the resolver ran uninterrupted for a week, and a horizon resolved more than 30 minutes late is discarded rather than back-filled — so long-horizon rows over-represent healthy-infrastructure periods.
• Suppressed is not zero. Any rate below its minimum sample returns null and renders as “below min n”. An endpoint being down renders as “unavailable”. Neither is ever replaced with a placeholder number.
• We publish no user results. There are no typical returns, no testimonials and no affiliate partners quoted anywhere on this site, because we have nothing honest to put in that slot.

Every chart on this page is a live call you can make yourself: GET /v1/signals/analytics/calibration, GET /v1/signals/analytics/edge, and GET /v1/stats (the forward_holdout block). No key required.