Insights
How to Build a Reproducible Pre-Market Drop-Risk Backtest
Alphanume Team · September 4, 2026
A pre-market drop-risk backtest should answer two separate questions: whether the probabilities describe the observed decline frequencies, and whether a specified trading process could have earned a return after execution and short-sale costs. A table containing both the forecast and its later outcome makes the first question convenient. It also makes leaking the answer into the decision unusually easy.
Alphanume's Pre-Market Drop Risk dataset ranks selected US microcaps by modeled probability of an unadjusted regular-session open-to-close return at or below -5%. This workflow preserves the raw input, grades only known outcomes, and keeps a historical research extract separate from evidence of what a trader actually received before an entry.
Define the event and the information boundary
Keep the target fixed: intraday_return_pct <= -5, with returns expressed in percentage points. An intraday low 5% below the open, a negative close-to-close return, and an eventual gap fill are different events. The field definitions establish which question the served probability addresses.
The model's features are cut at 09:00 ET. Its prob_drop and daily rank use the premarket path, cutoff price, and prior-return context. Point-in-time microcap classification defines eligibility; market_cap is served as context. Outcomes arrive later. The t_0 through t_9 path is cumulative, with a zero at a mark when no prior print exists; do not reinterpret it as ten independent half-hour returns.
The feature cutoff is not proof of API availability at precisely 09:00. The served response has no publication timestamp or model-version field. An execution study needs a recorded receipt time and a rule that places any hypothetical order after that receipt. A historical pull made today cannot establish the minute at which the same row could have been retrieved months ago.
Separate replayed history from a forward observation log
The serving contract describes an early period reconstructed by retrospective walk-forward replay. That replay uses prior training information, but it is not a record of an API response actually published on each earlier morning. Historical signal dates and live publication receipts therefore answer different provenance questions. Establish the production publication boundary from operational evidence before labeling a particular period as live.
Keep retrospective research, live-period historical extracts, and your own saved forward observations separately labeled in the research report. The response does not expose the internal source or publication fields, so a date split alone cannot provide a complete publication audit. Do not describe a combined historical result as independently verified live performance.
Use a future holdout or a saved forward observation period to test a rule chosen after inspecting history. The fact that a model used walk-forward training does not stop a researcher from overfitting rank cuts, price exclusions, or cost assumptions to the returned results.
Start with the returned cohort, not an invented full universe
The dataset publishes a selected list rather than every US microcap with a score. Eligibility includes point-in-time microcap classification and sufficient premarket activity; the published selection also reflects the classifier's signal rule. A ticker missing from the response is not an observed negative outcome and must not be assigned a zero probability.
Retrieve all returned rows over the chosen dates before trying rank and probability cuts. Record returned dates, unique tickers, sub-dollar status, missing market caps, and missing outcomes. An absent date in a historical pull does not by itself tell you whether no candidates qualified or a pipeline failed; the public response lacks the internal run ledger needed to resolve that question.
Predeclare a small number of research variants: for example, the full returned cohort, the top five ranks, and a separate $1-and-up subset. Compare high ranks with lower returned ranks on the same dates. That is a comparison within the published cohort, not a claim about all eligible or all traded stocks.
Pull the full history without turning unknown outcomes into misses
Install requests and pandas, then set ALPHANUME_API_KEY, START_DATE, and END_DATE in your environment. Choose a completed-session end date; this example evaluates the same-day target only. It intentionally requests no probability or rank cut so you can preserve the returned cohort before evaluating a variant.
import json
import os
from datetime import date, datetime, timezone
from pathlib import Path
import pandas as pd
import requests
start = date.fromisoformat(os.environ["START_DATE"]).isoformat()
end = date.fromisoformat(os.environ["END_DATE"]).isoformat()
if start > end:
raise SystemExit("START_DATE must be on or before END_DATE.")
out = Path("premarket-study") / datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%S%fZ")
out.mkdir(parents=True, exist_ok=False)
params = {"date_gte": start, "date_lte": end,
"api_key": os.environ["ALPHANUME_API_KEY"]}
rows, seen, receipts = [], set(), []
with requests.Session() as session:
while True:
try:
response = session.get(
"https://api.alphanume.com/v1/premarket-drop-risk",
params=params, timeout=(10, 60))
except requests.RequestException:
raise SystemExit("Historical pull failed; no result reported.") from None
if not response.ok:
raise SystemExit(f"API returned HTTP {response.status_code}.")
try:
page = response.json()
except ValueError:
raise SystemExit("Invalid JSON response.") from None
if not isinstance(page, dict) or not isinstance(page.get("data"), list):
raise SystemExit("Unexpected data response.")
if not isinstance(page.get("has_more"), bool):
raise SystemExit("Missing pagination flag.")
if not all(isinstance(row, dict) for row in page["data"]):
raise SystemExit("Unexpected row shape.")
filename = f"page-{len(receipts) + 1:03d}.json"
(out / filename).write_text(json.dumps(page, indent=2), encoding="utf-8")
receipts.append({"file": filename,
"retrieved_at_utc": datetime.now(timezone.utc).isoformat()})
rows.extend(page["data"])
if not page["has_more"]:
break
cursor = page.get("next_cursor")
if not isinstance(cursor, dict):
raise SystemExit("More pages indicated without a cursor.")
pair = (cursor.get("date"), cursor.get("rank_for_date"))
if pair[0] is None or pair[1] is None or pair in seen:
raise SystemExit("Invalid or repeated cursor.")
seen.add(pair)
params.update(cursor_date=pair[0], cursor_rank=pair[1])
(out / "manifest.json").write_text(json.dumps({
"start_date": start, "end_date": end, "pages": receipts,
"rows": len(rows), "target": "unadjusted open-to-close return <= -5%"
}, indent=2), encoding="utf-8")
if not rows:
raise SystemExit("No rows returned. Inspect dates/access; no hit rate calculated.")
df = pd.DataFrame(rows)
if df.duplicated(["date", "ticker"]).any():
raise SystemExit("Duplicate observations; inspect raw pages.")
required = ["prob_drop", "intraday_return_pct", "rank_for_date"]
if not all(name in df for name in required):
raise SystemExit("Required grading fields are missing.")
for name in ["prob_drop", "intraday_return_pct"]:
df[name] = pd.to_numeric(df[name], errors="raise")
valid_scores = df["prob_drop"].notna() & df["prob_drop"].between(0, 1)
if not valid_scores.all():
raise SystemExit("Missing or invalid model probabilities.")
df["target"] = pd.Series(pd.NA, index=df.index, dtype="Int64")
known = df["intraday_return_pct"].notna()
df.loc[known, "target"] = (
df.loc[known, "intraday_return_pct"] <= -5
).astype("Int64")
df.sort_values(["date", "rank_for_date", "ticker"]).to_csv(out / "cohort.csv", index=False)
df.loc[~known].to_csv(out / "missing-outcomes.csv", index=False)
graded = df.loc[known].copy()
print({"returned": len(df), "graded": len(graded), "ungraded": int((~known).sum())})
if graded.empty:
raise SystemExit("No known outcomes; no hit rate calculated.")
graded["score_band"] = pd.cut(
graded["prob_drop"], bins=[0, .5, .6, .7, .8, .9, 1],
include_lowest=True, right=True)
table = graded.groupby("score_band", observed=True).agg(
observations=("ticker", "size"),
mean_probability=("prob_drop", "mean"),
event_rate=("target", "mean"),
)
table.to_csv(out / "calibration.csv")
print(table)
The key correction is the nullable target. Comparing a missing floating-point return with -5 can yield False; converting that comparison directly to an integer silently counts an unknown outcome as a non-event. Here, the target begins as pd.NA and is assigned only for rows with a known return.
Every response page and receipt time is retained, and the request follows the documented date/rank cursor pair. A network error, invalid page, or repeated cursor stops the process rather than presenting a partial pull as the full sample. The manifest contains no API credential. The finished cohort includes ungraded rows, with a separate missing-outcome file for investigation.
Interpret the calibration table with its denominator visible
For each predeclared score band, compare mean prob_drop with the fraction of graded observations whose return met the target. A hypothetical mean probability of 0.70 with an event rate of 0.55 is a 15-percentage-point gap in that sample. It does not tell you whether the model will be overconfident next month, nor whether the difference is distinguishable from sampling noise.
The example bands include their right endpoint, so 0.70 belongs to the band ending at 0.70. Keep that convention fixed across runs. Report counts next to rates, and retain the ungraded count outside the rate's denominator. The small script is a descriptive starting table; it does not calculate confidence intervals, daily lift, or a strategy P&L.
For uncertainty estimates, resample dates so observations from the same morning remain together. Report results by date period, price cohort, and company-size group where there are enough observations. Repeated tickers and common market shocks can also create dependence across dates; explain the dependence your chosen uncertainty method does and does not address.
A Brier score can summarize squared probability errors on known outcomes, but it should sit beside the band table and sample counts. A good aggregate score can conceal a weak subgroup. Calibration of the published subset also does not establish discrimination across the full microcap universe, which is not present in this response.
Investigate missing outcomes instead of deleting them quietly
Recent rows can be ungraded because their horizon has not matured. Older rows can remain null when a halt, delisting, or missing required exchange print prevents the outcome calculation. Missingness may therefore be related to the difficulty of trading the stock. Reporting only complete cases without the exception count can hide a material part of the experiment.
For secondary horizons, preserve the documented meanings: return_lead_1d and return_lead_5d refer to one and five trading sessions, while the legacy field named return_lead_30d refers to 20 trading sessions. A matured horizon does not guarantee a valid target print. Never fill these fields with zero to make a chart continuous.
Maintain an exception ledger showing the signal date, ticker, missing field, investigation status, and treatment in the analysis. If you cannot resolve the outcome, keep it unknown and show how much of the cohort it represents. A sensitivity analysis may examine alternative treatments, but those assumptions must remain separate from observed returns.
Add a return study only after specifying executable rules
The classification target starts at the regular-session open. That does not guarantee a trader could receive the full intended size at the official opening price. Specify the order time, entry window, price source, participation assumptions, and what happens when trading is halted or the order is unfilled. The price you can observe after an event is not automatically the price available to an earlier order.
For short research, the SEC's Regulation SHO explanation describes the locate requirement, subject to applicable exceptions. A model score does not establish borrow feasibility. The signal response contains no borrow availability, locate fee, spread, or executable size.
The Short Locates dataset supplies observed quoted fees and availability for tracked hard-to-borrow names from October 8, 2026 onward. It has no earlier backfill. Join only snapshots whose actual snapshot_time_utc preceded the decision, and do not use an afternoon quote for a morning entry. An offered quote is still not a reservation or a broker-confirmed fill; an absent ticker or null price is not free borrow.
Report gross price differences and each modeled cost separately. A locate fee per share differs from an annual borrow rate; the snapshot fee does not include every holding cost. Test plausible slippage and financing assumptions, cap exposure before examining cumulative returns, and keep unresolved execution cases visible. A long-only avoidance rule needs its own benchmark and a defined destination for capital when a candidate is excluded.
Keep enough evidence to reproduce the result
A complete bundle contains the raw pages, receipt manifest, source-code version, dependency versions, exact filters, target definition, cohort, missing-outcome ledger, and final tables. Keep the rule-selection period separate from the period used to assess it. A second researcher should be able to explain both why each row entered the study and when each field became usable.
If the immediate task is simply to prepare today's research list, use the Python premarket watchlist tutorial. This backtest begins where that workflow ends: after preserving an input, it asks what happened later and whether the probability and trading assumptions survived scrutiny.