Stratix Ships Compass: Model Selection That Weighs What Your Industry Actually Needs

A team at a hospital picks a model for clinical decision support. A team at a game studio picks one for NPC dialogue. Both teams go to the same leaderboard. They see the same top five. They pick from the same composite score.

The hospital team needs medical reasoning weighted heavily and hallucination risk penalized. The game studio needs creative generation and contextual variety. A single composite built from the same benchmark weights cannot encode both sets of requirements. The leaderboard ranks models well for the task of ranking models. It does not rank them well for the task of deploying into a specific scenario with specific capability demands.

This is not a leaderboard failure. Leaderboards measure what they measure, and the measurements are real. The problem is that model selection has one more step that most teams handle informally: mapping a model’s benchmark profile to the requirements of the scenario where it will actually run. That mapping requires judgment calls about which benchmarks matter, how much each one matters, and whether high scores on certain benchmarks should count against a model in sensitive contexts. Every team deploying a model into production makes those judgment calls. They make them informally, never document them, and lose them the moment the person who made them leaves the project.

Call it the selection gap: the distance between a model’s benchmark profile and a team’s deployment requirements, bridged today by undocumented judgment that varies per team, per project, per quarter. The gap widens every time a new model enters the catalog, every time a team changes its deployment scenario, and every time a regulator asks why this model and not that one. EU AI Act GPAI enforcement activated on August 2, 2026. The Colorado AI Act and Texas TRAIGA introduce per-violation penalties in the US. The selection gap is where the audit trail should be, and few teams have one.

TL;DR

• Compass covers 14 industries and 60 real-world scenarios. Each scenario carries a hand-built scoring rubric that weights 44 benchmarks differently based on what the scenario actually demands.

• The scoring formula is public and auditable: score(model, scenario) = sum of weight(scenario, benchmark) * normalized_score(model, benchmark) across all benchmarks in the rubric.

• 265 weighted entries connect scenarios to benchmarks. Each rubric explains the weighting rationale, including what the rubric can and cannot evidence.

• Six benchmark entries carry negative weights, penalizing models that score high on manipulation or weapons-knowledge benchmarks in sensitive contexts like banking chatbots, marketing copy, and vehicle assistants.

• Four hard rules govern every recommendation: constraints filter but never score, missing data is never imputed as zero, breakdowns reconcile to the total exactly, and no rubric means no ranking.

The Map Between Benchmarks and Deployment Scenarios

The selection gap has a shape. Compass fills it with a taxonomy: 14 industries, 60 scenarios, 44 benchmarks, and 265 weighted entries connecting them.

Think of how a hiring rubric works. A company hiring a backend engineer and a company hiring a designer both draw from the same candidate pool. Both companies see the same resumes. A general “talent score” averaging all skills treats both roles as the same optimization problem. A hiring rubric does not. It assigns explicit weights to the skills that matter for the role, scores candidates against those weights, and produces a ranking that reflects what the role actually demands. The rubric is visible, the weights are auditable, and two hiring managers can disagree about the weights rather than about whether the candidate “seems good.”

Compass does the same thing one layer up. Pick an industry. Pick a scenario. Compass ranks every model in the Stratix catalog using the weighted rubric for that combination. The automotive industry alone breaks into three scenarios: vehicle assistant systems (the AI a driver talks to), ADAS perception interpretation, and connected-vehicle data Q&A. Each carries a different rubric because each demands different capabilities.

A vehicle assistant needs strong instruction following and low scores on manipulation benchmarks. DarkBench carries a negative weight of -0.1 here, penalizing models prone to subtle steering. ADAS perception interpretation weights visual and multimodal benchmarks: image understanding (0.4), MMMU (0.3), and ScreenSpot-Pro (0.3), all proxies for the ability to parse sensor data and scene context. Connected-vehicle data Q&A prioritizes structured data reasoning (BIRD-Critic at 0.3) and instruction following (IFEval at 0.25), because the system fields natural-language queries against telemetry databases.

Three scenarios in one industry, three completely different benchmark profiles. The general leaderboard produces one ranking for all three.

The same logic runs across healthcare and life sciences (six scenarios including clinical decision support, medical documentation, and drug discovery), financial services (six scenarios spanning fraud detection, algorithmic trading, and regulatory compliance), gaming (five scenarios from NPC dialogue to player behavior analysis), and ten other industries. For a team building a general-purpose chatbot with no industry-specific requirements, a general leaderboard works fine. Compass exists for every other team.

The Formula Is Checkable

Compass commits to one formal claim:

score(model, scenario) = Σ weight(scenario, b) × normalized_score(model, b)

normalized_score maps every model’s benchmark result to a 0-1 scale. Compass computes the minimum and maximum across every model that has a result for that benchmark, not just the models surviving scenario constraints. Normalization stays stable regardless of which models hard constraints filter out.

Weights are explicit. In healthcare clinical decision support, medical reasoning (MedQA and GPQA) carries the highest weight. Instruction following (IFEval) is second. Code generation (HumanEval, LiveCodeBench) is weighted lower but still present because clinical decision support systems increasingly involve structured data extraction. Each rubric includes a “Why this rubric” section explaining the weighting rationale, including what the rubric cannot evidence.

Four auditing rules hold across every recommendation:

Hard constraints filter, never score. A model that fails a constraint drops from the ranking entirely. The constraint is a gate, not a gradient.

No imputed zeros. If a model has no result for a weighted benchmark, Compass excludes that benchmark’s contribution for that model. “Not measured” and “scored poorly” are different facts, and conflating them produces false rankings.

Breakdowns reconcile exactly. A team can take any model-scenario breakdown apart, verify each benchmark contribution, and reassemble it to the total composite score with no rounding residual.

No rubric, no ranking. Compass returns nothing for scenarios without a defined rubric rather than guessing.

When Higher Scores Make a Model Worse

Six entries in the taxonomy carry negative weights. These use DarkBench (measuring manipulative tendencies) and WMDP (measuring weapons and mass destruction knowledge) to penalize models that score high in sensitive deployment contexts.

Banking customer service chatbots penalize manipulation (DarkBench at -0.1). Marketing copy generation carries the heaviest penalty (DarkBench at -1), treating manipulative tendency as a near-disqualifier for that scenario. Vehicle assistant systems and wealth advisor copilots each carry a DarkBench weight of -0.1. Retail product recommendation systems apply DarkBench at -1. Technology ISV code review penalizes weapons-knowledge scores (WMDP at -0.08).

A high manipulation score on a general leaderboard is informational. On a banking chatbot rubric, that same score actively reduces the model’s composite. The negative weight encodes the distinction so the ranking reflects it automatically.

What Compass Does Not Solve

Compass weights MedQA for healthcare reasoning and GPQA for graduate-level scientific knowledge, but no benchmark directly measures whether a model handles a specific hospital’s formulary interactions correctly. Each rubric’s rationale says explicitly what the rubric can and cannot evidence. Compass narrows the field and documents the reasoning. It does not replace domain-specific evaluation against your own data and your own edge cases.

A second limitation: the rubrics are versioned (compass-seed-2026-07) and represent one team’s judgment about benchmark relevance. Different domain experts would assign different weights. Compass makes that judgment auditable rather than invisible.

Closing the Selection Gap

When a regulator or an internal audit asks why this model and not that one, the answer is typically “it was the best available at the time,” with no evidence trail showing which benchmarks were considered, how they were weighted, or what tradeoffs were accepted.

Compass tracks nine regulatory frameworks across its industry mappings (SOC 2, ISO 27001, GDPR, HIPAA, PCI DSS, FedRAMP, EU AI Act, NIST AI RMF, and ISO 42001) as metadata, not scoring dimensions, so teams can see which compliance requirements apply alongside the model rankings. Every weight, every benchmark contribution, and every rationale is visible alongside the model rankings.

Pick the industry, pick the scenario, get a ranked list with every weight and every benchmark contribution shown. Disagree with a weight? The rubric tells you why it was set and what it cannot evidence. Compare models within a scenario and see exactly which benchmark contributions drive the ranking.

Compass works with the benchmarks available today, documents what they can and cannot measure, and gives teams a defensible starting point instead of an undocumented judgment call.

Compass is live in Stratix. Explore the full taxonomy, run scenario rankings, and export the breakdown for any model at stratix.layerlens.ai.

Frequently Asked Questions

How does Compass handle models that are missing benchmark results?

It excludes that benchmark’s contribution for that model rather than treating missing data as a zero. A model with no MedQA result does not get penalized on healthcare rubrics for lacking medical reasoning data. It gets scored on the benchmarks it does have, and the rubric notes which benchmarks were excluded.

Can I disagree with the weights in a rubric?

Yes, and the rubric is designed to make disagreement productive. Each rubric includes a “Why this rubric” section explaining why each benchmark was weighted the way it was, what the rubric cannot evidence, and whether each benchmark is a direct measure or a proxy. Two teams can look at the same rubric and reach different conclusions about whether the weights fit their deployment.

Does Compass replace domain-specific evaluation?

No. Compass narrows the field from dozens of models to a ranked shortlist scored against your scenario’s requirements. The shortlist still needs evaluation against your own data, your own edge cases, and your own deployment constraints. Compass is the starting point, not the finish line.

How are the rubrics versioned?

The current taxonomy is compass-seed-2026-07. Rubrics are versioned so that teams can reference which version produced a recommendation, and so that changes to benchmark weights over time are traceable.