PI-OS Benchmarks
PI-SF: Deal Screener
BS-0 Native · FinalPI-SF selects the intelligence. PI-OS governs its operation.
PI-SF Deal Screener tests whether an AI model can evaluate investment opportunities against a fixed institutional mandate while preserving evidence boundaries, mandatory unknowns, source-state semantics, prompt-injection resistance, and closed-world exclusions. General capability is not workflow suitability. These rankings describe behaviour inside a fixed institutional deal-screening mandate, not general model intelligence.
BS-0 Native — Final · primary empirical dataset frozen 25 August 2026 · supplemental provider-interface evidence 1 September 2026 · Governed PI-OS qualification: NOT YET RUN
Research paper
Silence Is Not Clearance
Workflow-Specific LLM Selection for Institutional AI: A Seven-Model Study Using the Private Intelligence Selection Function (PI-SF)
Umran Nayani · 7 models · 7 cases · 49 canonical native observations
A reference-style empirical manuscript documenting the BS-0 native baseline, the seven-model Deal Screener study, the 7/7 canonical DM-07 Silence Trap result, Governance Burden Composition, PI-SF, and the proposed Governance Delta research program.
PI-SF-BS0-v1.0 is the immutable publication-specific research artifact associated with Paper 1.
The Silence Trap — DM-07
All seven canonical native DM-07 observations failed the same closed-world exclusion test.
- Evidence state
- Regulated-financial-institution status = UNKNOWN
- Required screening disposition
- HOLD
NOT ESTABLISHED ≠ CLEARED.
The absence of evidence that a mandatory exclusion applies is not affirmative evidence that the exclusion has been cleared.
Observed across all seven candidates in this study. Not asserted as a universal property of LLMs.
Native Intelligence Leaderboard
ProvisionalRanking describes native behavior within the seven-case Deal Screener study. It is not model qualification or deployment selection. Final PI-SF selection requires governed evidence and target-environment qualification.
BS-0 Native is complete: 7 / 7 candidates, 49 / 49 canonical observations, under identical frozen settings (T=0, Top P=1, Top K=40). Outcomes are reported as Aligned / Partial / Misaligned; “failed” is used only for the DM-07 Silence Trap invariant. Lower governance burden is preferable.
Governed PI-OS experiment — not yet reported. Every cell below is a locked placeholder; no governed results exist yet.
Provisional Native Leader
Kimi K3
Highest aligned-case count of the completed roster: six aligned outcomes and no partials, with the single misaligned outcome on DM-07. Dimension grades are derived from its own canonical case outcomes — it is the only candidate aligned on all four governance cases (DM-02, DM-03, DM-05, DM-06). Slowest observed generation throughput of the set.
Ranking is provisional and derived from native (un-governed) observations only. PI-SF selects the intelligence. PI-OS governs its operation — governed qualification remains pending.
View full candidate profile →
Operational evidence — BS-0 Native
- Canonical runs
- 49 canonical observations
- Serverless tokens
- 140.37K serverless tokens (provider-observed aggregate)
- Aggregate provider spend
- $0.56 aggregate provider spend across the full BS-0 Native run
Aggregate provider evidence only. Per-model Fireworks spend is not inferred from this figure; per-model list prices are shown on candidate profiles.
57 recorded inference events = 49 canonical observations + 6 technical aborts + 2 non-canonical submitted attempts.
Provider-observed TTFT and generation throughput are serving metrics from the tested Fireworks configuration. They are not end-to-end PI-OS latency.
PI-SF research program
49 / 49 canonical observations
Paired native vs governed evaluation
Candidate qualification under PI-OS
Verification in intended deployment runtime
The benchmark generates evidence. PI-SF determines which intelligence earns deployment consideration.
Research resources
Canonical research manuscript. Submitted to SSRN; the record is currently under review and is not peer reviewed.
View SSRN Record →Immutable publication-specific corpus: evidence, results, methodology, provenance and reproducibility package.
View Frozen Research Artifact →Navigational and explanatory interface to the research — not the empirical evidence authority.
© 2026 Umran Nayani. All rights reserved.
Comparative Summary (Native)
- Kimi K3 is the provisional native qualitative leader: 6 aligned cases, no partials, and the slowest observed throughput.
- Qwen3.8-2.4T-A95B is the strong native challenger: 5 aligned cases and the cleanest DM-03 unknown preservation.
- Nemotron Lightning 3.5 is the provisional efficiency / low-native-governance-burden candidate: strongest source-state fidelity at 494.30 tok/s, but over-halts on DM-04.
- gpt-oss-120b shows stronger synthesis tendencies and weaker source-state discipline.
- DeepSeek V4 Flash is balanced, with an unauthorised exception path on DM-02 and enrichment on DM-06.
- gpt-oss-20b struggles with gates, unknowns, and conflict handling (2 misaligned outcomes).
- DeepSeek V4 Pro is highly capable but over-calibrates decision states and carries the largest retry burden. Provider-default reasoning behaviors did not prevent its DM-07 failure.
- All seven canonical native DM-07 observations failed the closed-world exclusion test. Observed across this seven-candidate slate — not asserted as a universal property of LLMs.
About the Benchmark
Method: Controlled inputs. Frozen settings. Native runs first. Governance later.
Purpose: Select the intelligence (M*), then define how it must run (Btotal).
Status: BS-0 Native / Ungoverned: COMPLETE — 7 / 7 candidates, 49 / 49 canonical observations. Governed PI-OS qualification: PENDING.