← Back to feed

Source Rating Methodology

Our Editorial Stance

AlphaCheck believes a strong US military is a net positive for the nation and the world. We also believe the defense acquisition system needs disruption — we're pro-capability and pro-reform, sympathetic to new entrants, innovative approaches, and the defense-tech ecosystem challenging the status quo.

We surface stories from all posture categories because informed readers benefit from understanding how different sources frame defense topics. The posture rating provides context, not censorship.

Who We Write For

We call our target reader the “Squadron Tech Geek” — the person in the unit who actually likes computers instead of merely tolerating them. The one who wired together some solution for the unit back in the vault using only Excel and duct tape because that's all they were allowed to use. The person who figured out how to automate that tedious tasker that came down from higher HQ.

Our coverage sits at the intersection of three vectors:

  • Defense — programs, policy, acquisition, operations
  • Current Tech — what's shipping now in software, hardware, and platforms
  • Future Tech — AI, robotics, autonomy, and how they will reshape defense

If a story lives at the intersection of at least two of these, it belongs on AlphaCheck.


The Military Lean Axis

What It Measures

The Military Lean rating reflects a publication's default editorial stance toward US military capability and the defense industrial base. It answers one question:

When this source covers US defense spending, military programs, or the defense-tech industry, what is their default assumption about whether American military strength is a net positive for the nation and the world?

What It Does NOT Measure

  • Political ideology (left vs. right) — captured separately by the Political Lean axis
  • Factual accuracy — captured separately by the Factuality axis
  • Quality of journalism — a Skeptical source can produce excellent, rigorous reporting
  • Patriotism or loyalty — all positions on this scale can be held in good faith

The Five-Point Scale

Pro-Capability

Default assumption: US military strength is essential. Maintaining technological overmatch matters. Defense spending is generally justified. Critical coverage focuses on "how to do defense better," not "should we do this at all."

Examples: Defense News, Breaking Defense, C4ISRNET, War on the Rocks

Pro-Reform

Accepts the premise of American military strength but emphasizes that the current system — prime contractors, acquisition bureaucracy, Pentagon processes — is broken and needs disruption. Sympathetic to new entrants, nontraditional vendors, and defense-tech startups.

Examples: Defense One, TechCrunch (defense coverage), CNAS, CyberScoop

Neutral

No consistent institutional posture on defense topics. Framing is driven by individual reporters or stories rather than an editorial stance.

Examples: Reuters, Associated Press, Bloomberg Government, Stars and Stripes

Skeptical

Default assumption: defense spending is probably too high and military programs deserve heavy scrutiny. The defense industry's influence on policy is treated as problematic. Core question: "Do we really need this?"

Examples: The Atlantic, ProPublica, Washington Post (defense coverage), Brookings Institution

Anti-Capability

Editorial position: US military power is generally harmful. The defense industrial base is fundamentally corrupt. Coverage consistently frames the military and defense industry as adversarial to the public interest.

Examples: The Intercept, Responsible Statecraft (Quincy Institute), Jacobin


Other Rating Axes

Political Lean

AlphaCheck editorial rating based on sustained analysis of a source's coverage patterns, framing choices, and institutional posture on domestic politics. Every source in our database ships with a published rationale explaining the call — see the source profile for details.

Scale: Left · Lean Left · Center · Lean Right · Right

Factuality

AlphaCheck editorial rating based on a source's use of primary documents, track record of corrections, and whether reporting adds context and verification rather than merely repackaging wire copy.

Scale: Very High · High · Mostly Factual · Mixed · Low

Ownership

Researched and classified by AlphaCheck's editorial team. Captures who owns or funds the publication, which can influence editorial decisions and access.

Categories: Corporate Media · Independent · Think Tank · FFRDC · US Government · Nonprofit · Professional Society · Creator/Newsletter


The Restraint Modifier

An optional secondary label applied to sources and individual voices who support a strong US military but advocate narrow, judicious use of that power. These are the restraint hawks— they sit in an unusual quadrant: Pro-Capability on posture (invest in the force) but measured on employment (don't use it casually).

It's a sparse modifier, not a new axis. Only ~10–15% of sources carry it. The 5-point Military Lean scale stays the primary signal; Restraint adds a second dimension on employment posture without muddying the main rating.

Representative voices

Defense Priorities, the Cato Institute defense shop, Responsible Statecraft; academics like Barry Posen and Stephen Walt.

Why we surface it

Restraint voices offer the sharpest criticism of capability “solutions in search of problems.” Marking them lets readers quickly find the steelmanned case for narrow use without having to memorize which bylines lean which way.

Surfaced as: badge next to the Military Lean on source cards, plus a “Restraint voices only” toggle in the sidebar.


How Ratings Are Assigned

Ratings are AI-drafted and human-approved. When a new source is added, an LLM (Claude, during interactive development) drafts the rating profile — Military Lean, Political Lean, Factuality, plus ownership type, primary beat, and a plain-English rationale — based on a skim of the source's recent RSS output and training-data priors about the publication. A human editor reviews the proposal and approves, modifies, or rejects it before the JSON is committed.

We describe this as “AlphaCheck editorial” because the human approval gate is where editorial responsibility lives — once a rating is merged, AlphaCheck owns it. But we want to be precise about the mechanism: no human is sitting down to read ten articles per source. The drafting is LLM work; the accountability is human.

The draft-and-approve loop:

  1. LLM reads a sample of recent article titles from the source's RSS feed (usually 10–20)
  2. LLM draws on training-data priors (masthead, ownership, reputation, coverage history) and calibrates against sources already in the database
  3. LLM proposes the three ratings plus ownership type, primary beat, and a plain-English rationale
  4. LLM sets a confidence flag (High for major outlets, Medium for niche publications, Low for almost-unknown ones)
  5. Human editor reviews the draft and approves, modifies, or rejects before the JSON is committed

The rationale field is the audit trail — if a rating is ever challenged, the rationale is what we point at. The confidence field is our honest flag for where the rating is weakest; a Low-confidence call on an important source is a signal that it deserves deliberate human re-review.

Ratings reflect sustained coverage patterns, not individual articles. One bad piece doesn't move a rating; a year of consistent framing does. Sources are re-reviewed roughly yearly; when a rating changes, the old rationale is preserved in git history.


What We Don't Use

AlphaCheck does not integrate with, query, scrape, or license data from Media Bias/Fact Check (MBFC), AllSides, Ad Fontes Media, or any RapidAPI wrapper of those services. Earlier planning docs listed such integration as a future task; that task was abandoned and never implemented.

Why not:

  • External services rate only ~15–20 of our 65+ sources. The rest — defense trade press, specialty Substacks, think tanks, government — are too niche to be covered.
  • Our editorial differentiator is Military Lean (Pro-Capability ↔ Anti-Capability). That axis doesn't exist anywhere else.
  • Paid data licenses would buy us an external credential for partial coverage, not better ratings.
  • Our per-source rationale field forces us to explain each call in plain English. That transparency is a stronger editorial posture than “a third party said so.”

Separately from source ratings, our production ingest pipeline uses Gemini / Claude / GPT for article relevance scoring, story clustering, and briefing generation. That automated pipeline never touches the source ratings file; all changes to source ratings pass through the human-approved chat-draft loop described above.


Leaderboards

The Leaderboardspage ranks who’s leading defense-tech coverage across three categories over 7-day, 30-day, and 90-day windows.

What gets ranked

  • Sources — how often a source lands as the “lead” when we group related articles into clusters. Leads on a cluster about a specific news event (a contract announcement, a test flight, a leak) count twice as much as leads on a broader / ongoing story (US-Iran conflict, AI governance debate) that spans multiple angles.
  • VIPs — citation count across titles, descriptions, and cluster briefings. Clicking a row filters the main feed to articles citing that VIP.
  • Podcasts + Newsletters — the same scoring function that picks This Week in Defense Tech: topic overlap (50%) + citation count (30%) + recency (20%). The 7-day board is “TWiDT plus the rest.”

How we read “last 7 / 30 / 90 days”

We use the publish dateof the item. An article counts in the 7-day window if it was published in the last 7 days. A podcast episode or newsletter counts based on when it went out, not when we first saw it. That way the rankings reflect what’s actually happening in the world, not the rhythm of when our pipeline crawls for new content.

Why v1 counts volume, not influence

Today’s rankings reward outlets that publish a lot — a real limitation. A source with modest output but real authority on the beat can get out-ranked by a louder one. The v2 direction (late 2026) fixes this by measuring who cites whom: a source matters more if the outlets already considered authoritative are citing it. That requires several months of article history before the signal is strong enough to rank on, which is why we’re shipping the volume version first while the archive builds up.

What this isn’t

Leaderboards rank who’s getting their coverage read, not whose argument is right. Political Lean, Military Lean, and Factuality badges on the Sources rows are there so readers can see the editorial context of each outlet — they don’t affect ranking position. If a source with a very different posture than ours is leading the coverage, they’ll show up high. That’s the point.

When we show “insufficient data”

Some windows don’t yet have enough history to produce meaningful rankings. Instead of surfacing a misleading “top 3” built from a handful of articles, we show an “insufficient data” banner and the date that window is expected to reopen. For example, the Podcasts + Newsletters 90-day board needs about 60 days of episode and edition history before it starts ranking — so it’s currently empty, with a reopen estimate shown in its place.


Editorial Log

AlphaCheck's ratings aren't static. This log records meaningful changes to how we classify sources, define our rating axes, or surface context in the feed. Most recent first.

We keep this log for three reasons:

  1. Auditability — readers can see what changed and why, not just what the ratings are today.
  2. Intellectual honesty — when we change our mind in public, we show our reasoning.
  3. Discipline — the obligation to write an entry forces us to think harder about the change before making it.

Granular per-source rating deltas live in the git history of our source database. This log captures the editorial narrative — the “why,” not every byte-level “what.”

2026-04-21 — Steelman briefings land in the UI

Contested clusters (Military Lean spread ≥ 3) now carry a clickable ⚡ Debated topic pill that expands to show a steelman of the anti-capability view, generated from the cluster's actual sources anchored to a curated anti-capability seed set (Defense Priorities, Cato defense shop, Responsible Statecraft, Posen, Walt, and peers). The briefing is always an anti-capability steelman — that's the position AlphaCheck's Pro-Cap editorial stance is weakest against, so Rapoport's rule says we steelman thatone, not the in-cluster majority. UI copy makes the framing explicit rather than saying “opposing view” in the abstract.

2026-04-20 — Restraint modifier introduced

Added an optional restraint: true flag to source profiles marking the restraint-hawk quadrant: voices who support strong US military capability but advocate narrow / judicious employment of it. Sparse — only ~10–15% of sources are expected to qualify. Surfaced as a badge on source cards and a “Restraint voices only” pill in the sidebar filter. Initial flags (editorial-validated): War on the Rocks, Hudson Institute, Quincy Institute / Responsible Statecraft. More to come as we work through the roster. This is a modifieron the existing 5-point Military Lean scale, not a new axis — we don't want to muddy the primary signal.

2026-04-18 — Q2 2026 seed audit

Reviewed the Pro-Capability and Anti-Capability seed sets used by clustering and steelman generation as credibility anchors. Changes:

  • Reclassified Dario Amodei (Anthropic CEO) from unrated to Pro-Capability + seed_pole=pro_cap — his public positions on AI export controls and national security framing align with the Pro-Cap anchor set.
  • Added Sundar Pichai (Google CEO) at Pro-Capability + seed_pole=pro_cap for similar reasons.
  • Added Owen West and Paul Scharre to the Pro-Capability analyst anchor set.
  • Caveated Stephen Walt — while he sits in the Anti-Cap seed set as a restraint-realist voice, his framing differs meaningfully from the activist-left anti-cap tradition; the caveat flags this for future prompt tuning.

2026-04-14 — Contested threshold raised

Moved the ⚡ Debated topic threshold from spread ≥ 2 to spread ≥ 3 on the 5-point Military Lean scale. The previous threshold surfaced too many false positives where sources merely skewed Pro-Reform vs. Pro-Capability — a normal, healthy split within our own editorial center of gravity, not a genuine multi-traditions debate. The new threshold reserves the indicator for clusters where sources genuinely span the scale from skepticism to full-throated Pro-Cap.

2026-04-12 — Monthly source audit (April)

First full monthly audit of the source roster. Notable calls:

  • DoD News, DARPA News, NDIA added with fresh ratings.
  • National Defense Magazine added; seed_pole removed on a second pass after realizing the rating alone carries the signal and the seed role wasn't warranted.
  • Broader sweep of rating rationales updated with current reasoning; several “last_reviewed” dates refreshed.

2026-04-08 — Methodology: restraint modifier defined

Added the conceptual definition of the restraint modifier to this methodology page (see The Restraint Modifier section above), ahead of any UI surface. This locked in the editorial idea before we committed to how to display it. Sequencing matters: methodology first, data second, UI third.

Log started 2026-04-21 with backfilled entries for recent editorial decisions. Future entries land here when we make the change, not after the fact.


All three rating axes — Military Lean, Political Lean, and Factuality — are AlphaCheck editorial judgments. Every source in the database includes a published rationale explaining the call. We welcome feedback if you think we've gotten one wrong.