Coframe Analytics Data Platform — Design Document

Status: v1.0 Author: reeeneeee Spec authority: Coframe Core Manual (the Manual). Where this document and the Manual disagree, the Manual wins.

Changes from v0.6.7 (v1.0 publication pass):

All Manual cross-references reconciled to Manual v0.7.7's chapter structure (the prior Tier-1 organizational pass renumbered chapters 3–12 as 2–11 and consolidated the Pro-feature deferrals into §1.5; this document was not previously reconciled). The MCP capability surface in §6.2 is brought into alignment with Manual §11.3's authoritative capability list. The license decision is closed (§0.2, §12). The straightforward §12 open questions are resolved. Internal section-number annotations in the §2.3 module-layout block (which referred to old Manual chapter numbers that no longer exist) are removed; module names remain self-documenting. Older changelog entries are pruned and preserved in git history.

---

0. Scope and intent

This document specifies the engineering design of the open-source v1.0 release of the coframe basic platform: how the components are laid out as Python packages, how they depend on each other, what their public surfaces are, and in what order they get built.

It is not a re-specification of the framework. The Manual specifies what the framework does. This document specifies how that gets organized into Python packages, modules, and tests.

0.1 v1.0 component set

Five packages, one repository:

1. coframe-core — the grammar-layer engine. Includes Frame-QL parsing and semantic analysis (coframe.ql), AC name_map and operator/mapper customization, identifier translation during resolution, and the natural-language query layer (coframe.dialogue).
2. coframe-connect — thin interface package: Backend protocol (execution), source-binding types, entry-point conventions for backend discovery. No authoring protocol.
3. coframe-polars — execution backend using Polars, plus a complete authoring toolchain in coframe_polars.author (discovery, profiling, FD-candidate testing, QM extraction, LLM-assisted naming, review CLI, AC YAML emission).
4. coframe-duckdb — execution backend using DuckDB, plus a complete authoring toolchain in coframe_duckdb.author with the same scope as the polars version, internally adapted to DuckDB.
5. coframe-mcp — Model Context Protocol server. Backend-blind: depends only on coframe-core; loads backends by name via entry-point discovery. Exposes Frame-QL execution and natural-language querying via coframe.dialogue.

0.2 Decisions locked in this document

• Language: Python 3.11+.
• Repo: monorepo, uv workspace.
• AC catalog format: YAML on disk; Pydantic models in memory.
• Public API: fluent ac.query("...") style. AC must be backend-bound to be queryable.
• Backend execution contract: resolved Frame-QL AST with physical column names (translated from AC name_map) and logical system operator/mapper names (after AC customizations are expanded). See §2.4 and §2.8.
• Backend binding: one execution backend per AC. Multi-source = multiple ACs.
• AC name_map: single global mapping per AC (one logical name → one physical name across the whole AC). Integrity-checked. See §2.8.
• Local operator/mapper customization: declarative-only in v1. No Python in ACs. Code-bearing extensions deferred to Coframe Pro. See §2.9.
• Authoring: lives entirely inside each backend (coframe_polars.author, coframe_duckdb.author). Independent per backend. No AuthoringSupport protocol; no shared library; no cross-backend interface. Each backend ships a complete authoring toolchain. See §4 and §5.
• Natural-language query (NLQ): new submodule coframe.dialogue inside core. Logical-only, no data access, vendor-independent LLM client. See §7.
• MCP layering: coframe-mcp depends only on coframe-core. Backends discovered via entry points declared in coframe-connect. MCP exposes both Frame-QL execution and nl_query. See §6.
• Per-DNA-edge value attestation: enabled by default per Manual §7.6.8. Configuration lives in the AC YAML's attestation: block; defaults apply when absent; opt-out requires explicit attestation: enabled: false. The Backend protocol gains attest_dna_edge (see §3.3); the resolution pipeline runs attestation as part of data-dependent integrity (I3–I6 are extended by I10 — attestation pass; see §2.5). MCP query results propagate coherence_posture and edge-level annotations (see §6.2).
• Slowly Changing Attributes (SCA): out of scope for v1.0. Deferred to Coframe Pro per Manual §1.5. The (entity, family, operator) foundation accommodates SCA via multi-entity anchoring with a slow-time-dimension component (E(a, S) = {d, t}); the v1.0 AC-attribute model assumes |E| = 1 for AC-attributes, which Pro generalizes to |E| ≥ 1. Engineering implication for Core implementation: the integrity machinery's |E| = 1 check on AC-attributes (Phase 1, I0–I2 area) should be cleanly factorable from the column-trichotomy classification logic, so that Pro's relaxation can land additively without restructuring Core's integrity pipeline. ACs containing time-varying attributes that need to be modeled via event-time anchoring (rather than as SCAs) are fully supported in Core. See Manual §2.4, §2.5, §2.6.
• Generalized functional grammar layer: out of scope for v1.0. Deferred to Coframe Pro per Manual §1.5. Coframe Core supports function-derived dimensional columns (operator catalog functions like MONTH_OF, BUCKET, SUBSTR) and function-derived metrics (Frame-QL inline expressions) as first-class participants in the FD-DAG and family genealogy. These operate in the deductive verification regime (verified by construction through function semantics) alongside the empirical verification regime (verified by data attestation through DQ). Pro lifts this duality from Core's special-case treatment to the framework's primary architectural framing: explicit naming of the two regimes throughout the spec, support for user-defined deterministic functions with declared properties, and richer verification-status reporting that distinguishes I3-by-attestation from I3-by-construction. Engineering implication for Core implementation: the coframe.operators catalog interface should be cleanly factorable so user-defined-operator extensions in Pro can land additively. The verification-level computation (§2.10) already produces correct results regardless of regime mix; no Core-side engineering changes needed for this deferral. See Manual §1.2.1, §2.8.5, §1.5.
• Recursive (self-referential) hierarchies: out of scope for v1.0. Deferred to Coframe Pro per Manual §1.5. Core's FD-DAG models functional dependencies among distinct entity-sets; self-referential hierarchies (employee-manager, parent-part bills-of-materials, message-thread reply structures) require recursive query traversal that Frame-QL v1.0 does not provide. Pro extends Frame-QL with recursive query primitives (descendant-aggregation, ancestor traversal, depth-of-hierarchy computation). Engineering implication for Core: no recursive-CTE support in the data-API protocol or in Frame-QL's grammar; ACs whose source data has recursive structure must pre-flatten or use closure-table modeling before Coframe ingestion. See Manual §1.5, §2.8.
• License: Apache 2.0 for all five Coframe Core packages. This is the dominant license for open-source data infrastructure (dbt-core, DuckDB, Polars, Apache Arrow, Apache Parquet) and provides the patent-grant protections that mature enterprise adopters expect. Coframe Pro is separately licensed.

---

0.3 Naming and identifier conventions

This document refers to the project's components in three registers — product names, package names, and import paths. They share underlying meaning but differ in typography and context. The conventions below apply throughout this document and are intended to carry into the codebase, documentation, and downstream artifacts.

Product names (in prose, marketing, manuals, conversation):

• Coframe — the framework as a whole. The grammar-layer thesis, the structural commitments, the verification regime. Edition-independent.
• Coframe Core — the open-source edition specified by this design document and the Coframe Core Manual. Capitalized, two words, no hyphen.
• Coframe Pro — the commercial edition extending Coframe Core. Capitalized, two words, no hyphen. Specified separately in the (forthcoming) Coframe Pro Manual.

Python package names (in pyproject.toml, pip install commands, dependency declarations):

• coframe-core — the foundational package, distributed via PyPI. Lowercase, hyphenated. The package implementing what the product "Coframe Core" provides.
• coframe-connect — the backend protocol package. Same conventions.
• coframe-polars, coframe-duckdb — the reference execution backends.
• coframe-mcp — the MCP server package.

The lowercase-hyphenated form follows Python ecosystem conventions for distribution names (PEP 503). It is independent of product-name capitalization. The intentional homonymy between "Coframe Core" the product and coframe-core the package is informative: both refer to the same foundational artifact in different registers. Readers in prose context parse "Coframe Core" as the product; readers seeing coframe-core (backticks, lowercase) parse it as the package. Capitalization and code formatting do the disambiguation.

This pattern follows the precedent of dbt-core / dbt the product, kafka-clients / Apache Kafka, postgresql / PostgreSQL, and similar mature ecosystems. It is robust in practice; same-word-different-register naming is a feature of how software products and their distribution mechanisms relate.

Import paths (in Python source code, after package installation):

• import coframe — the top-level namespace. Provides the unified API regardless of which coframe-* packages are installed.
• from coframe.types import ColumnSpec — submodule access via the coframe namespace.
• from coframe.connect import Backend — backend-protocol surface (provided by the coframe-connect package, exposed under the coframe.connect namespace).
• from coframe.attestation import VerificationStatus — attestation surface.

The coframe namespace package convention (PEP 420 implicit namespace packages, or PEP 561 explicit if preferred) lets the platform's distribution be modular at the package level (coframe-core, coframe-connect, etc., installable independently) while presenting users with a unified coframe.* import surface. A user installing pip install coframe-core gets coframe.types, coframe.ql, coframe.attestation, etc. Installing coframe-polars adds the polars backend, registered via the coframe.backends entry-point group; the user does not write import coframe_polars — they configure their AC with backend: polars and coframe-mcp discovers and loads the backend at runtime.

Three-register summary:

Register	Form	Example use
Product	Coframe Core	"Coframe Core supports Polars and DuckDB backends."
Package	coframe-core	pip install coframe-core (in command voice)
Module	coframe.types	from coframe.types import ColumnSpec (in source)

The three forms travel naturally with their contexts. Documentation written in prose uses the Product form; install instructions use the Package form; code examples use the Module form. A reader's context parses the appropriate referent without conscious effort.

Why we don't rename the package. A reasonable question is whether coframe-core should be renamed (e.g., to coframe-base) to avoid the same-letters homonymy with the product name. The answer is no: the homonymy is benign (different registers, different contexts), follows established Python ecosystem precedent (dbt-core, pytest-core), and the meaning of "core" is the same in both registers (the foundational, central artifact). Renaming the package would weaken the self-documenting package layout (where each coframe-* package's name describes its function) at the cost of solving a non-problem. The conventions specified in this section make the homonymy explicit; that is sufficient.

---

1. Architecture overview

1.1 Component dependency graph

                       ┌─────────────────┐
                       │   coframe-mcp   │
                       └────────┬────────┘
                                │  (depends only on coframe-core;
                                │   discovers backends via entry points)
                                ▼
                       ┌─────────────────────────────┐
                       │      coframe-core           │
                       │  + coframe.ql               │
                       │  + coframe.dialogue         │
                       │  (logical-only)             │
                       └─────────┬───────────────────┘
                                 ▲
                                 │
                       ┌─────────┴───────────┐
                       │   coframe-connect   │
                       │   Backend protocol +│
                       │   discovery         │
                       └─────────┬───────────┘
                                 ▲
                ┌────────────────┼────────────────┐
                │                                 │
        ┌───────┴──────────────────┐    ┌─────────┴────────────────┐
        │   coframe-polars         │    │   coframe-duckdb         │
        │   + coframe_polars.author│    │   + coframe_duckdb.author│
        │   (complete authoring)   │    │   (complete authoring)   │
        │   registers via          │    │   registers via          │
        │   entry points           │    │   entry points           │
        └──────────────────────────┘    └──────────────────────────┘

1.2 Conceptual roles

• coframe-core owns the AC: loading, validating (I0–I9), resolving Frame-QL queries, producing resolved ASTs. Resolution does name-translation and customization-expansion. coframe.dialogue provides natural-language → Frame-QL translation as a separate, logical-only AI surface — entirely independent of any backend.

• coframe-connect defines only the Backend execution protocol, source-binding types, and entry-point conventions. It is small. It carries no authoring concept.

• coframe-polars / coframe-duckdb are complete units of work. Each ships:

• An execution backend implementing Backend.
• An authoring toolchain (<backend>.author) — discovery, profiling, FD-candidate testing, QM extraction, LLM-assisted naming, review CLI, AC YAML emission. This is a self-contained CLI a data engineer runs to bootstrap an AC against the backend's data sources. It speaks physical names internally where the data lives, optionally consults an LLM for naming proposals (the LLM sees physical names — that's fine, it's inside the backend's authoring scope), and emits a final AC YAML with the logical surface populated.
• A pyproject.toml entry-point registration for execution.

• The two backends' authoring code is independent. They may converge in v1.x; in v1, each is its own thing.

• coframe-mcp wraps a bound AC behind an MCP server. Backend-blind by design. Exposes Frame-QL execution and natural-language querying via coframe.dialogue.

1.3 The two AI roles, separated

The platform has two distinct AI/LLM surfaces; v0.5 makes them architecturally distinct.

Role	Where it lives	Sees physical names?	Sees data?
NL → Frame-QL (query elicitation)	coframe.dialogue (in core)	No	No
Authoring assistance (naming, categorization)	<backend>.author (in each backend)	Yes	Yes

• coframe.dialogue is the user-facing LLM surface. It speaks logical names because that's the AC's user-facing vocabulary. It has no data access because translation is a vocabulary task, not a data task. It is the only LLM surface available to MCP clients.

• Authoring assistance is the internal LLM surface. It exists inside each backend's authoring toolchain and helps with naming, categorization, and proposal generation during AC bootstrap. It speaks physical names because that's the data the backend has, and it has data access because authoring fundamentally involves looking at the data. It is invoked only by the data engineer running coframe-author (or equivalent) against a specific backend.

Two LLM surfaces, two purposes, two privilege levels. Neither pretends to be the other.

1.4 What runs where, at runtime — query path

1. AC is constructed and bound at startup, carrying its name_map and any local operator/mapper customizations.
2. User submits one of:
3. A Frame-QL query string — direct path.
4. A natural-language utterance — coframe.dialogue translates to Frame-QL first; the rest of the path is identical.
5. coframe.ql parses to a raw AST.
6. coframe.resolution resolves: schema selection, identity matching, MTI, dubious-query detection, WITH-block desugaring, ratio expansion, customization expansion, column-name translation.
7. Output: a resolved AST (physical column names, logical system operator/mapper names).
8. The bound backend walks the AST and produces a result.
9. Result returned in the requested format.

1.5 What runs where, at AC-bootstrap time — authoring path

1. Data engineer runs the backend's authoring CLI: coframe-polars-author or coframe-duckdb-author (or a unified entry point that dispatches to one based on a --backend flag — see §4 / §5).
2. The CLI calls into the backend's .author submodule.
3. The submodule discovers sources, profiles columns, tests FD candidates, optionally consults an LLM for naming proposals (the LLM here works inside the backend's scope and sees physical names).
4. The submodule's review CLI walks the engineer through proposals one at a time.
5. The submodule emits an AC YAML and a provenance sidecar.
6. After review, integrity validation runs against the produced AC (this can use the same backend).

The whole authoring lifecycle lives inside one backend. No cross-package coordination is required.

---

2. coframe-core

2.1 Responsibilities

• Load and validate AC catalogs (YAML → Pydantic), including name_map and customizations.
• Maintain self-registry, FD-DAG, system operator/mapper registries, ColumnSpecs (logical names only), lightweight ratios, missing-value policies, quasi-metadata, the AC's effective operator registry (system + customizations).
• Run integrity checks I0–I9 (where I9 = name_map consistency).
• Parse Frame-QL.
• Resolve Frame-QL queries with customization expansion and column-name translation.
• Emit a resolved AST carrying physical column names and logical system operator/mapper names.
• Translate natural-language utterances to Frame-QL via coframe.dialogue.
• Surface diagnostics for parse errors, dubious queries, integrity violations.

2.2 Public API (Python)

from coframe import AC

# Load
ac = AC.load("retail.yaml", source_bindings={"sales": FileSource("sales.parquet"), ...})
# or:
ac = AC.load("retail.yaml", bindings_file="retail.bindings.yaml")
# or AC YAML can declare bindings inline:
ac = AC.load("retail.yaml")

ac.validate()                                   # data-free integrity (I0/I1/I2/I7/I8/I9)

# Bind a backend (entry-point lookup)
ac_p = ac.with_backend("polars")
# or programmatically:
from coframe_polars import PolarsBackend
ac_p = ac.with_backend(PolarsBackend(...))

ac_p.validate()                                 # full integrity, including I3-I6

# Resolve only — needs bindings (for column translation), not a backend
resolved = ac.resolve("SUM(revenue) BY (region, year)")

# Resolve + execute
df = ac_p.query("SUM(revenue) BY (region, year)").to_polars()

Natural-language query (logical-only):

from coframe.dialogue import translate

frame_ql = translate(ac, "what was last quarter's revenue by region?", llm="claude")
# → "SUM(revenue) BY region WHERE quarter == 'Q4-2025'"
df = ac_p.query(frame_ql).to_polars()

# or as a one-liner:
df = ac_p.ask("what was last quarter's revenue by region?", llm="claude").to_polars()

ask(...) is sugar over translate(...) + query(...). It returns a result and records the original utterance and generated Frame-QL on the result for provenance.

Authoring is run from the backend's CLI, not via core's public API:

coframe-polars-author bootstrap --sources ./warehouse/sales/ --output sales_ac.yaml
# or:
coframe-duckdb-author bootstrap --sources postgres://... --output sales_ac.yaml

(Per §12, the two backends ship independent CLIs in v1.0. A unified coframe-author --backend X is a v1.x candidate if cross-backend convergence emerges.)

2.3 Internal module layout

coframe/
├── __init__.py              # re-exports AC, version
├── types.py                 # DataType, Category, Epoch, Grain, simple value types
├── errors.py                # exception hierarchy + diagnostic objects
├── selves.py                # SelfRegistry, Self
├── combination.py           # combination laws, monoid declarations
├── operators.py             # SYSTEM operator registry; basic catalog (Manual §10.4 reducers)
├── mappers.py               # SYSTEM mapper catalog (Manual §10.5 functions)
├── customization/           # AC-LEVEL declarative customizations
│   ├── __init__.py
│   ├── overrides.py
│   ├── compositions.py
│   ├── aliases.py
│   ├── restrictions.py
│   └── effective.py         # SystemRegistry + AC customizations → EffectiveRegistry
├── fd_dag.py                # FDDAG, edge sources, path consistency (Manual §2.8)
├── column_spec.py           # ColumnSpec (logical name only) (Manual Chapter 3)
├── name_map.py              # AC-level name_map: logical name → physical name
├── ratios.py                # lightweight ratios
├── missing.py               # missing-value policy
├── quasi_metadata.py        # shared QM schema, cache, refresh interface (Manual §6.3, §7.5)
├── integrity.py             # I0–I9 (Manual §2.10, Chapter 7)
├── ac.py                    # AC top-level: composition; binding + effective-registry mgmt
├── catalog/                 # YAML loading
│   ├── __init__.py
│   ├── schema.py            # Pydantic models (incl. name_map, customizations)
│   ├── loader.py
│   └── bindings.py
├── ql/                      # Frame-QL: Manual Chapter 8, Appendix A (BNF)
│   ├── __init__.py
│   ├── lexer.py
│   ├── parser.py            # produces RawAST
│   ├── ast.py               # RawAST and ResolvedAST node classes
│   ├── semantics.py         # name resolution against effective registry
│   └── pretty.py
├── resolution/              # Rungs 0–9 + customization expansion + translation (Manual Chapter 9)
│   ├── __init__.py
│   ├── single_schema.py     # Rungs 0–6
│   ├── cross_schema.py      # Rung 7, four-rule filter
│   ├── mti.py               # MTI theorem
│   ├── holistic.py          # Rung 4
│   ├── type_changing.py     # Rung 5
│   ├── epochs.py            # combination-law epochs
│   ├── with_block.py        # Rung 9, WITH
│   ├── ratios.py            # lightweight-ratio expansion
│   ├── dubious.py           # dubious-query detection
│   ├── customization.py     # alias/composition expansion
│   └── translation.py       # column-name translation (logical → physical via name_map)
├── attestation/             # I10: per-DNA-edge value attestation (Manual §7.6.8)
│   ├── __init__.py
│   ├── config.py            # attestation config dataclasses + defaults
│   ├── plan.py              # edge enumeration + attestability classification
│   ├── runner.py            # invokes Backend.attest_dna_edge per edge
│   └── status.py            # verification-status aggregation + propagation
├── plan/
│   ├── __init__.py
│   └── resolver.py          # parse → resolve → expand → translate orchestrator
└── dialogue/                # natural-language → Frame-QL (this document §7)
    ├── __init__.py
    ├── translate.py         # main translate(ac, utterance, llm) entry point
    ├── llm.py               # vendor-independent LLM client
    ├── prompts/             # versioned prompt templates
    │   ├── system.md        # describes the framework + Frame-QL grammar to the LLM
    │   ├── ac_summary.py    # renders an AC's logical surface for the LLM context
    │   └── examples.yaml    # few-shot examples
    ├── validators.py        # validate generated Frame-QL: parse + semantic check + dubious
    └── provenance.py        # NlqProvenance: utterance, generated FQL, model+version, timestamp

The dialogue/ subdirectory is the platform-design v0.5 addition. Its single responsibility is "translate user words into Frame-QL." It speaks only logical names because the AC's surface is in logical names. It has no backend dependency.

2.4 The resolved AST

This is the execution contract. Same as v0.4. Backends touch only coframe.ql.ast.ResolvedQuery and the node types reachable from it.

A resolved AST node carries:

• For each column term: physical column name from name_map.
• For operators and mappers: logical system names; AC customizations have been expanded.
• WITH-blocks desugared.
• Filters tagged with where they apply.
• Lightweight ratios expanded inline.
• Missing-value policies explicit.
• FD-DAG projection paths for grain transitions explicit.

Backends don't see customizations. Backends don't translate column names. Both translation passes happen in core during resolution.

2.5 Integrity checking

I0 → I1 → I2 → I3 → I4 → I5/I6 → I7 → I8 → I9 → I10.

I9 (name_map consistency): - Every logical name referenced by any ColumnSpec has an entry in the AC's name_map. - The name_map is injective. - Customization aliases and compositions reference only registered logical operators. - Behavior when a name_map entry exists for a logical name not referenced by any ColumnSpec — warning vs. error: settle Phase 1.

I7 expanded: AC's effective operator registry is well-formed (no conflicts, no alias cycles, compositions reference registered operators).

I10 (per-DNA-edge attestation, Manual §7.6.8): for each attestable DNA edge in the AC's metric genealogy, the predecessor's data aggregated via the family's ip_reducer at the successor's anchor agrees with the successor's observed values within the configured tolerance, on shared keys, scoped to the intersection of the two schemas' declared scopes. Honors operator catalog missing-value semantics for the predecessor's M signature. Failure-mode-driven response: hard causes I10 violation; soft produces an advisory recorded in the verification status; tolerated produces a recorded acceptance per declared edge. When the AC opts out (attestation.enabled: false), I10 is skipped and the AC's verification status records attestation: disabled. Implementation: coframe.resolution.attestation orchestrates per-edge planning; the backend's attest_dna_edge operation (per coframe-connect §3.3) executes per-edge verification.

Data-dependent checks (I3–I6, I10) require a bound backend.

2.6 Quasi-metadata

Per Manual §6.3 (introspection), §7.5 (quasi-metadata fetch). Schema in coframe.quasi_metadata. QM records keyed by physical column names (data has physical names). Translation between logical and physical happens via the AC's name_map.

QM is consumed by: - coframe.integrity for data-dependent integrity checks (I3–I6). - The backend's authoring submodule for proposal evidence.

The QM schema is part of v1.0 stable surface.

2.7 What's not in coframe-core

• No execution.
• No data sources.
• No backend imports.
• No backend-native operator/mapper names.
• No code-bearing operator/mapper extensions (Coframe Pro).
• No authoring orchestration. Authoring is fully in backends.
• LLM SDK imports lazy in coframe.dialogue.

2.8 Logical-vs-physical naming — AC-level name_map

(Same content as v0.4 §2.8. Single AC-level name_map. Translation runs as a resolution pass. Operators stay logical in the AST.)

The asymmetry:

	What core knows	What's in the resolved AST	Who translates to engine-native form
Column names	Logical name + AC-level name_map	Physical name	Core (during resolution)
Operator names	System registry + AC customizations	System logical name (post-expansion)	Backend (at execute time)
Mapper names	System registry + AC customizations	System logical name (post-expansion)	Backend (at execute time)

2.9 Local operator and mapper customization

(Same content as v0.4 §2.9. Four declarative patterns — combination-law overrides, named compositions, aliases, restrictions. No Python in ACs. Customizations are expanded by resolution/customization.py before the AST reaches the backend.)

2.10 AC Verification Levels

coframe-core computes each AC's verification level (per Manual §7.13) as a deterministic function of the AC's grounding map — the structure that documents how each of the AC's structural commitments is verified. Levels are reported through the public API and propagated by coframe-mcp on query results.

The grounding map. Per Manual §7.13.4, the level definitions are formulated in terms of grounded structural commitments, where grounding admits two sources: empirical (data-attested through DQ Phase 3) and deductive (verified-by-construction through operator catalog semantics). Each commitment in the AC is grounded by at least one source (or, for metric coherence commitments, transparently tolerated with rationale). Phase 4 of the build phasing (resolution and integrity) absorbs the grounding-map computation: as the resolver walks the FD-DAG and metric genealogy, each edge is annotated with its grounding source(s) based on its declared op, the operator catalog's properties, and the integrity-condition results from Phases 1–3.

Concretely, the resolver classifies each FD-edge and metric DNA edge into one of:

• data_attested — the edge is verified by I3 (FD-edges) or I10 (metric edges) attestation; the predecessor is materialized data and the cross-check passed.
• verified_by_construction — the edge is established by a deterministic operator catalog function applied to a predecessor; the operator catalog's declared properties (partition-invariance, identity-preservation, type signature) make the edge true by construction.
• mixed_cross_checked — the edge is both materialized data-side and derivable function-side; I3 or I10 attestation cross-checks that the materialized values agree with the function output.
• tolerated — only valid for metric DNA edges; the edge is declared in attestation.tolerated_edges with rationale.
• unattestable — the edge requires data-attestation grounding (no function-derivation path) but the predecessor is not in the AC or the operator is not partition-invariant; recorded for transparency.
• failed — data-attestation was attempted and the cross-check failed beyond tolerance; not tolerated; blocks the level.

The grounding map is a deterministic output of the resolver: same AC + same DQ inputs produce the same grounding map.

Computation. coframe.attestation.status.compute_level(ac, grounding_map, integrity_results) -> Level returns one of Level.A, Level.AA, or Level.AAA. The implementation:

• Returns Level.AAA if all I0–I9 pass, every dimensional structural commitment in the grounding map is grounded (data_attested, verified_by_construction, or mixed_cross_checked), every metric coherence commitment is grounded (any of those four plus tolerated), and no commitment is failed or unattestable-and-required.
• Returns Level.AA if all I0–I9 pass, every dimensional structural commitment is grounded, but at least one metric coherence commitment is ungrounded (e.g., attestation disabled when data-attested grounding was needed; an unattestable edge required for AAA's commitment coverage; a failed-not-tolerated edge).
• Returns Level.A if I0–I9 pass but at least one dimensional structural commitment is ungrounded.
• Returns None (no level) if any of I0–I9 fail; the AC cannot validate.

The function is pure: same grounding map and integrity results in, same level out. Determinism enables level-based testing and reproducibility across runs. Crucially, the function correctly handles ACs composed primarily or entirely of function-derived structure: when no commitment requires data-attestation grounding, the AC reaches AAA via deductive grounding alone (the absence of I10 attestation runs is not a deficiency).

Public API.

from typing import Literal
from pydantic import BaseModel

class GroundingSummary(BaseModel):
    """Per-commitment grounding statistics for the AC."""
    fd_edges: dict[str, int]  # keys: data_attested, verified_by_construction,
                              #       mixed_cross_checked, unattestable, failed
    metric_coherence: dict[str, int]  # additionally: tolerated

class VerificationStatus(BaseModel):
    level: Literal["A", "AA", "AAA"] | None
    naming_consistency: Literal["verified", "asserted"]
    attestation: AttestationStatus  # enabled, failure_mode, edge counts, etc.
    tolerated_edges: list[ToleratedEdge]
    grounding_summary: GroundingSummary
    integrity_results: IntegrityResults
    coherence_posture: Literal[
        "unconditional_within_scope",
        "conditional_within_scope",
        "asserted_not_verified",
    ]

class AC:
    @property
    def verification_status(self) -> VerificationStatus | None:
        """Returns the AC's verification status, or None if DQ has not been run."""

The verification_status is populated when DQ runs (Phase 3 completes); it persists with the AC catalog and is recomputed only when DQ re-runs (Manual §7.10).

MCP propagation. coframe-mcp surfaces the level field by default in coherence_posture per Manual §11.7.3. Clients requesting include_grounding: true get the full GroundingSummary in the response. The default-compact behavior is the right tradeoff for most consumers (level is the headline; grounding is informational); sophisticated consumers (AI agents reasoning about per-commitment confidence) can opt in to the richer payload.

Stability. Levels are informational in v1.0 and become stable surface in v1.x. The level taxonomy (the names A / AA / AAA, the level-from-grounding-map function, the public API shape including GroundingSummary) is documented and exposed at v1.0; v1.x will lock the taxonomy as a semver-protected commitment. v1.0 deployments that report or branch on levels should expect that v1.x may refine specific edge cases (e.g., how partially-attestable genealogies map to AAA vs. AA, whether the grounding-source breakdown should expose finer-grained classifications) based on field experience. The high-level commitment — three monotonically-stronger levels with grounding-source agnosticism — is firm.

Why not stable at v1.0. The level definitions and grounding-aware computation follow naturally from the framework's empirical/deductive duality, but field experience may surface calibration questions: how to handle ACs with very large unattestable-edge counts; whether tolerated_edges should affect level differently in regulatory contexts; whether the grounding-source taxonomy needs additional categories (e.g., distinguishing user-defined function grounding from built-in catalog grounding once Coframe Pro lands). Holding stable-surface status until v1.x lets us observe authored ACs and adjust before locking semver protection.

---

3. coframe-connect

3.1 Responsibilities

• Define the Backend protocol implemented by execution backends.
• Define source-binding types (SourceSpec, SchemaBinding).
• Define entry-point conventions for backend discovery (coframe.backends).
• Provide shared utilities for walking the resolved AST.
• Define error types backends raise.

Not in coframe-connect (v0.5 simplification):

• No AuthoringSupport protocol. Authoring is a per-backend internal concern.
• No coframe.authoring_backends entry-point group.

This makes coframe-connect materially smaller than in v0.4 — it is now exclusively the execution contract.

3.2 Why a separate package

• Semantic clarity. Community contributors writing a new backend depend on coframe-connect only.
• Versioning. The execution interface stabilizes on its own cadence.
• Test isolation. Conformance tests live with the protocol.

3.3 The Backend protocol

from coframe.ql.ast import ResolvedQuery, FDProjection
from coframe.connect import Backend, BackendData, SchemaBinding, SourceSpec

class Backend(Protocol):
    name: str

    def bind_schema(self, schema_metadata, source: SourceSpec) -> SchemaBinding: ...

    def read_columns(
        self,
        binding: SchemaBinding,
        physical_columns: list[str],
        filters: list[Filter],
    ) -> BackendData: ...

    def apply_mapper(
        self,
        data: BackendData,
        mapper_name: str,         # SYSTEM logical name
        args: dict,
    ) -> BackendData: ...

    def apply_reducer(
        self,
        data: BackendData,
        reducer_name: str,        # SYSTEM logical name
        args: dict,
        group_by: list[str],
        combination_law: str | None = None,   # AC override case
    ) -> BackendData: ...

    def fd_project(self, data: BackendData, projection: FDProjection) -> BackendData: ...
    def join_on_grain(self, left, right, on_dimensions) -> BackendData: ...
    def execute(self, query: ResolvedQuery) -> BackendData: ...
    def to_polars(self, data: BackendData) -> "pl.DataFrame": ...
    def to_arrow(self, data: BackendData) -> "pa.Table": ...

    # Quasi-metadata computation (consumed by core integrity I3–I6 and backend authoring)
    def compute_qm(self, ac, refresh: bool = False) -> "QuasiMetadata": ...

    # Per-DNA-edge value attestation (consumed by core integrity I10; Manual §7.6.8)
    def attest_dna_edge(
        self,
        predecessor_spec: "EdgeColumnSpec",   # schema, physical name, anchor, signature M
        successor_spec: "EdgeColumnSpec",
        op_name: str,                         # SYSTEM logical name of the family ip_reducer
        scope_filter: "ScopeFilter | None",
        epsilon: "EpsilonConfig",
        sampling: "SamplingConfig | None" = None,
    ) -> "AttestationResult": ...

Notes:

• read_columns accepts physical names directly.
• apply_* accept logical system names; customizations have been expanded by core.
• combination_law is an explicit parameter on apply_reducer — backends use the overridden law when provided.
• compute_qm is the QM computation entry point. Same backend method serves both integrity checks (called by core) and authoring (called by the backend's own .author submodule). One QM source of truth per backend.
• attest_dna_edge is the per-edge attestation entry point. The backend computes the predecessor's aggregation honoring the operator catalog's missing-value semantics for the predecessor's M signature (not naive SQL), applies the scope filter, and returns an AttestationResult per Manual §6.4.10. Backends that cannot honor full operator-catalog missing-value semantics declare partial attestation support via the capabilities mechanism (§3.3.x); the runner records the limitation in the verification status.

3.4 Source binding

SourceSpec = FileSource(path, format) | TableSource(connection, name) | InMemorySource(data)

@dataclass
class SchemaBinding:
    schema_name: str
    source: SourceSpec
    backend_handle: Any

No column overrides on the binding. The AC's name_map is the only column-translation source.

3.5 Error types

BackendError (with subclasses BackendDataError, BackendCapabilityError, BackendOperationalError).

3.6 Backend discovery via entry points

coframe-connect declares the entry-point group coframe.backends. Each backend package registers itself:

[project.entry-points."coframe.backends"]
polars = "coframe_polars:PolarsBackend"

Hosts (coframe-mcp, ac.with_backend("polars")) discover backends by name:

from coframe.connect.discovery import load_backend
backend_cls = load_backend("polars")

The entry-point group name is part of v1.0 stable surface.

3.7 Operator and mapper translation

Each execution backend ships translation tables for system operators in its package:

• coframe_polars/operators.py, coframe_polars/mappers.py
• coframe_duckdb/operators.py, coframe_duckdb/mappers.py

Because customizations are expanded by core before reaching the backend, these tables only need to cover system operators and mappers.

---

4. coframe-polars

4.1 Responsibilities

Implement Backend using Polars. Best for in-memory analytics on data fitting in RAM.

Ship a complete, self-contained authoring toolchain in coframe_polars.author.

4.2 Operator and mapper translation tables

System operators and mappers only — AC customizations are expanded by core.

• coframe_polars/operators.py — Manual §10.4 reducer catalog in polars.
• coframe_polars/mappers.py — Manual §10.5 function catalog in polars.

apply_reducer uses the optional combination_law parameter when present (AC override).

4.3 Execution: Backend implementation

• BackendData = polars.LazyFrame.
• bind_schema materializes a LazyFrame: pl.scan_csv, pl.scan_parquet, pl.LazyFrame(data).
• read_columns(binding, physical_columns, filters) → lf.select(physical_columns).filter(...).
• apply_mapper, apply_reducer consult translation tables.
• fd_project joins relevant FD-DAG reference table, then .group_by(...).agg(...).
• join_on_grain is .join(..., how="inner", on=on_dimensions).
• execute walks the resolved AST top-down, producing a single LazyFrame, then collects.
• compute_qm(ac) produces a QuasiMetadata for the AC's bound schemas, used by core integrity and by the backend's authoring.
• attest_dna_edge(...) (Manual §6.4.10) computes the predecessor aggregation by composing read_columns + apply_mapper (for missing-value treatment per operator catalog and the predecessor's M signature) + apply_reducer + fd_project, then full-outer-joins against the successor projection on E_self, and reports per-key deltas. Polars' in-memory model supports full attestation up to RAM-bounded fact tables; for larger tables, the runner falls back to sampling per attestation.sampling_* config. Polars backend declares full operator-catalog missing-value support via the capabilities mechanism.

4.4 Authoring: coframe_polars.author (complete, independent toolchain)

This is a full sub-application living inside the polars package. It is not a thin shim; it is the polars-native answer to "how do I bootstrap an AC?"

Module structure:

coframe_polars/author/
├── __init__.py
├── cli.py                  # `coframe-polars-author` entry point
├── discover.py             # source enumeration (Parquet/CSV/JSON walks)
├── profile.py              # column profiling: distinct counts, types, samples
├── fd_candidates.py        # FD-candidate testing (exhaustive, in-memory)
├── coarsening.py           # common-coarsening checks across sources
├── propose.py              # proposal generation (deterministic + LLM-assisted)
│   #   includes self proposals, FD-edge proposals, ColumnSpec proposals,
│   #   ratio proposals, name_map proposals
├── llm.py                  # LLM client (may share patterns with coframe.dialogue.llm
│   #   but does not import from it; intentional independence)
├── prompts/                # prompt templates for naming, categorization, ratios
├── review.py               # review CLI workflow
├── emit.py                 # AC YAML serialization
└── provenance.py           # provenance sidecar emission

Workflow:

1. CLI: coframe-polars-author bootstrap --sources ./warehouse/sales/ --output sales_ac.yaml.
2. discover enumerates sources.
3. profile produces column profiles in physical-name space.
4. fd_candidates tests every plausible dimension pair, exhaustively (in-memory polars allows confidence 1.0).
5. propose constructs proposals. For data-derivable parts (FD edges, distinct-counts, type-based categorization), proposals are deterministic. For naming and category disambiguation, LLM is consulted — the LLM sees physical names and sample values; this is fine because the entire authoring scope is internal to the backend.
6. review walks the engineer through proposals one at a time. Localized, structurally evidenced, accept/reject/edit, partial-AC-resumable.
7. emit produces:
8. The AC YAML (with name_map, selves, schemas, ColumnSpecs, ratios, customizations if any).
9. A provenance sidecar (with LLM model+version, evidence, confirmation timestamps, physical-name witnesses).
10. After review, the toolchain optionally runs full integrity validation against the produced AC.

Polars authoring advantages: - Exhaustive FD verification (confidence 1.0) on in-memory data. - Fast iteration during interactive review. - Rich evidence display (Polars' DataFrame repr is engineer-friendly).

Independence note. The polars authoring toolchain is not built from a shared library. Patterns (proposal data shapes, review CLI ergonomics, prompt structure) may converge with the duckdb version over time, but in v1 they are independent codebases. This is a deliberate v1 cost; convergence is a v1.x decision once both are working.

4.5 Capability declaration

The polars backend's supported operators/mappers are exactly the keys of its translation tables. v1 target: full Manual basic catalog.

4.6 Entry-point registration

[project.entry-points."coframe.backends"]
polars = "coframe_polars:PolarsBackend"

[project.scripts]
coframe-polars-author = "coframe_polars.author.cli:main"

Note: only one entry-point group used (coframe.backends). No separate authoring-backend group; authoring isn't a discoverable plugin in v1, it's a CLI shipped with the backend.

4.7 Notable limits

• Cannot exceed available RAM.
• No persistence.

---

5. coframe-duckdb

5.1 Responsibilities

Implement Backend using DuckDB. Best for persistent single-node analytics, larger-than-memory data, SQL-table-backed sources.

Ship a complete, independent authoring toolchain in coframe_duckdb.author.

5.2 Operator and mapper translation tables

• coframe_duckdb/operators.py — system operator names → DuckDB SQL aggregate constructions.
• coframe_duckdb/mappers.py — same for mappers.

5.3 Execution: Backend implementation

• BackendData = duckdb.DuckDBPyRelation. Relation-based composition.
• bind_schema: connection.from_csv(...), from_parquet(...), connection.table(name).
• apply_mapper, apply_reducer consult tables, build relation-API expressions.
• fd_project: join + GROUP BY.
• execute produces a final relation; materialization via .df() / .arrow() / .pl().
• compute_qm(ac) produces a QuasiMetadata for the AC's bound schemas.
• attest_dna_edge(...) (Manual §6.4.10) is implemented via DuckDB relations: the predecessor's aggregation is composed from apply_mapper (operator-catalog missing-value treatment) + apply_reducer + fd_project, joined against the successor projection on E_self via FULL OUTER JOIN, with deltas computed in SQL. Spill-to-disk supports full attestation on tables larger than RAM. The sampling fallback uses DuckDB's stratified sampling (TABLESAMPLE BERNOULLI within strata defined by E_self), reporting confidence per the framework's runner. The duckdb backend declares full operator-catalog missing-value support via the capabilities mechanism. Persistent connection mode caches attested-edge status in a system table so that re-attestation runs only against changed schemas (per Manual §7.10's differential re-attestation).

5.4 Authoring: coframe_duckdb.author (complete, independent toolchain)

Module structure mirrors the polars version (§4.4) but uses DuckDB SQL throughout.

coframe_duckdb/author/
├── __init__.py
├── cli.py                  # `coframe-duckdb-author` entry point
├── discover.py             # file walks + SQL-table enumeration
├── profile.py              # SQL-based profiling
├── fd_candidates.py        # FD-candidate testing: exhaustive when in-budget,
│                            # sampled with confidence < 1.0 when not
├── coarsening.py           # SQL-based coarsening checks
├── propose.py              # proposal generation
├── llm.py                  # LLM client (independent from polars version's)
├── prompts/                # prompt templates
├── review.py               # review CLI
├── emit.py                 # AC YAML serialization
└── provenance.py           # provenance sidecar emission

Workflow is the same shape as polars (§4.4), but adapted to DuckDB's strengths and constraints:

• Discovery includes enumerating SQL tables in the connection's catalog.
• Profiling uses single SQL passes per table.
• FD-candidate testing on tables exceeding a configurable memory budget falls back to stratified sampling, surfacing "confidence < 1.0" in the proposal evidence; the engineer chooses whether to accept based on confidence level.
• Review CLI may paginate evidence (data is persistent and potentially large).

DuckDB authoring advantages: - Handles tables larger than RAM via spill-to-disk. - Native SQL-table sources (Postgres, MySQL, etc. via DuckDB scanner extensions). - Persistent caching of QM in the database file; subsequent integrity checks reuse it.

5.5 Connection management

Optional duckdb.DuckDBPyConnection. Otherwise opens an in-memory connection. Long-running deployments (notably MCP) pass a persistent connection.

5.6 Entry-point registration

[project.entry-points."coframe.backends"]
duckdb = "coframe_duckdb:DuckDBBackend"

[project.scripts]
coframe-duckdb-author = "coframe_duckdb.author.cli:main"

5.7 Advantages over polars backend

• Larger-than-memory data via spill-to-disk.
• Direct SQL-table sources without a Polars round-trip.
• Persistent caching of intermediates and QM.
• Wider source ecosystem.

---

6. coframe-mcp

6.1 Responsibilities

Wrap a bound AC behind an MCP server (per Manual Chapter 11), exposing capabilities to LLM clients.

coframe-mcp depends only on coframe-core. Backends loaded at startup via entry-point discovery. Backend-blind — a third-party backend deploys under coframe-mcp without source changes.

6.2 MCP capabilities exposed

Per Manual §11.3 (the authoritative capability surface), with the platform's natural-language query capability:

AC discovery (Manual §11.3.1): - list_acs() — enumerate the ACs the server exposes. - describe_ac(ac_name) — name, description, scope summary (number of schemas, families, dimensions), and naming-function status.

Family-level reasoning (Manual §11.3.2): - list_families(ac_name) — enumerate families with family-name and brief root description. - describe_family(ac_name, family_name) — family-root description, root's op and partition_invariance, ip_reducer, anchors at which the family's columns are observable, structural-relations summary.

Genealogy navigation (Manual §11.3.3): - list_genealogy(ac_name) — AC-wide metric genealogy: family-root, siblings, cousins per family. - describe_column(ac_name, schema_name, column_name) — ColumnSpec details, derived properties, verification status.

FD-DAG navigation (Manual §11.3.4): - list_schemas(ac_name), describe_schema(ac_name, schema_name) — schema enumeration and per-schema details. - list_fd_edges(ac_name) — enumerate FD-DAG edges with channel and attestation status. - describe_fd_edge(ac_name, source, target) — per-edge details.

Dimension value reasoning (Manual §11.3.5): - get_dimension_values(ac_name, dimension) — universe-wide value-set for an AC-dimension. - get_dimension_coverage(ac_name, dimension, schema_name) — coverage map for a schema's coverage of a dimension.

Operator catalog (Manual §11.3.6): - list_operators() — enumerate the catalog operators. - describe_operator(op_name) — operator-catalog entry: type, partition_invariance, identity_preserving flag, default naming, missing-value behavior.

Query operations (Manual §11.3.7): - resolve_query(ac_name, frame_ql) — parse + resolve only; returns the resolution plan or a structured diagnostic. Useful for LLM exploration without execution cost. - execute_query(ac_name, frame_ql) — parse, resolve, and execute. Returns result data plus annotations. - nl_query(ac_name, utterance) — natural-language → Frame-QL via coframe.dialogue. Returns the generated Frame-QL plus its execution result and an NlqProvenance record. The MCP client (Claude Desktop, etc.) can show the user the generated Frame-QL for inspection. Available when the server is configured with an LLM for the dialogue (see §6.3).

Lightweight ratios (Manual §11.3 surface): - list_lightweight_ratios(ac_name) — enumerate registered ratio columns in the AC.

DQ inspection (Manual §11.3.8): - get_dq_status(ac_name) — DQ status: violations, advisories, integrity status, verification level. - get_dq_advisories(ac_name) — current advisories. - validate_ac(ac_name) — explicit revalidation entry point; returns the AC's verification status including I10 attestation results and the computed level.

All MCP responses speak in logical names (matching the AC's user-facing surface). Physical names never appear in MCP responses. ACs are user-facing surfaces; their logical vocabulary applies on every user-facing path.

Query results from execute_query and nl_query carry a coherence_posture field summarizing the AC's verification level (A / AA / AAA per Manual §7.13), attestation configuration, naming-consistency status, and the per-edge status of edges consulted in the query's resolution (per Manual §11.7.3). Failed-edge advisories propagate as coherence-warning annotations; opt-out propagates as coherence-asserted-not-verified (capping the level at AA). AI-agent clients can branch on these fields to assess result trust without making a separate validate_ac call.

The level field within coherence_posture is informational in v1.0 and stable surface in v1.x; clients consuming the level should expect it to carry forward unchanged for v1.x but may want to design for forward-compatibility around the specific edge-case calibrations.

coframe-mcp does not expose authoring in v1. Authoring runs via the backend's CLI on the data engineer's workstation, not over MCP.

6.3 Implementation

Server config example:

acs:
  - name: retail
    catalog:        /etc/coframe/retail.yaml
    bindings_file:  /etc/coframe/retail.bindings.yaml
    backend:        duckdb
    backend_options:
      database:     /var/coframe/retail.duckdb

dialogue:
  llm: claude
  api_key_env: ANTHROPIC_API_KEY

coframe-mcp reads backend: duckdb, calls load_backend("duckdb"), instantiates with backend_options, binds to the AC. Source has zero references to "polars" or "duckdb."

The dialogue: config block configures the LLM that backs nl_query. If absent, nl_query is disabled (the capability is reported as unavailable; execute_query continues to work). This makes the LLM dependency optional at the MCP layer.

6.4 Authentication and access control

API key (simple), OAuth2 (enterprise) via standard MCP transport. Per-AC access policies in server config. Per-method ACLs basic in v1.

6.5 Performance

One backend connection per AC, pooled. QM pre-loaded at startup. Result caching (short TTL for execute, longer for inspection). Streaming for large outputs. nl_query adds an LLM call on top of execute; cache nl_query results by (ac, utterance) for repeated identical queries.

---

7. coframe.dialogue — natural-language query layer

7.1 Architectural principle

coframe.dialogue is the user-facing LLM surface of the platform. Its single role: translate a natural-language utterance into a Frame-QL string against a specific AC.

It speaks only logical names because the AC's user-facing surface is in logical names. It has no data access because translation is a vocabulary task, not a data task. It exists in coframe-core because: - Its dependencies are: the AC's logical surface, the Frame-QL grammar, an LLM client. All three are in core. - It has no need for a backend. - Putting it in core makes it available to anything that uses core: programmatic Python users, the coframe-mcp server, future tooling.

7.2 Translation contract

from coframe.dialogue import translate, NlqProvenance

result = translate(
    ac,                          # bound AC
    "what was last quarter's revenue by region?",
    llm="claude",                # claude | gpt | gemini | local
)
# returns:
#   result.frame_ql:    "SUM(revenue) BY region WHERE quarter == 'Q4-2025'"
#   result.provenance:  NlqProvenance(utterance, frame_ql, llm_model, llm_version, timestamp)
#   result.alternatives: list[str]  # optional, additional candidate translations
#   result.diagnostics: list[Diagnostic]  # warnings, ambiguity notes

The function: 1. Loads the AC's logical surface (selves, schemas, ratios, customizations) into a structured prompt context. 2. Invokes the LLM with: a system prompt describing the framework + Frame-QL grammar; few-shot examples; the AC summary; the user utterance. 3. Parses the LLM's output as Frame-QL using coframe.ql.parser. 4. Runs semantic analysis against the AC. 5. Runs dubious-query detection. 6. If parse fails, tries up to N retries with the parse error fed back to the LLM as a correction prompt. 7. If the result is dubious, surfaces a diagnostic but still returns the candidate so the caller can decide.

The function does not execute the query. It returns Frame-QL; the caller decides whether to execute, show to user, edit, etc.

7.3 Vendor independence

coframe.dialogue.llm supports Claude (default), GPT, Gemini, and local LLMs via OpenAI-compatible interfaces. Switching is configuration. The same prompt templates work across vendors; vendor-specific adapters handle API quirks.

7.4 Prompt structure

The prompts/ subdirectory holds versioned templates:

• system.md — describes the platform's vocabulary: what selves, dimensions, metrics, attributes, ratios, schemas, and grain are. Describes Frame-QL syntax. Describes dubious-query detection. Tells the LLM to produce Frame-QL only, no commentary.
• ac_summary.py — given an AC, render a concise textual summary of its logical surface for the LLM context: selves with categories and combination laws, schemas with grain, lightweight ratios. Excludes anything physical.
• examples.yaml — few-shot examples: utterance → Frame-QL pairs spanning each Rung (0–9), aliases, ratios, WITH-blocks. Updated as new patterns are encountered.

Prompts are intentionally not v1.0 stable surface — they will evolve. The interface (translate(ac, utterance, llm) and NlqProvenance) is stable.

7.5 Validation and safety

Generated Frame-QL is always parsed and semantically validated before being returned. The LLM cannot produce a Frame-QL string that references nonexistent selves, malformed grammar, or undefined operators — those are caught at validation and fed back as correction prompts (or surfaced as failure).

The LLM cannot bypass dubious-query detection: even if the LLM produces "valid" Frame-QL, if resolution flags it as dubious, the diagnostic is surfaced. The user (or the calling agent) decides whether to proceed.

7.6 Provenance

NlqProvenance records on every NLQ result: - The original utterance. - The generated Frame-QL. - LLM model + version. - Timestamp. - Number of retries (if any). - Whether the query was flagged dubious.

In ac.ask(...), this provenance is attached to the QueryResult. In coframe-mcp's nl_query response, it's included in the JSON. Useful for audit trails and for building "show me what you understood" UX.

7.7 What's not in coframe.dialogue

• No data access. None.
• No authoring. Not its job.
• No multi-turn conversation state in v1 (single utterance → single Frame-QL). Multi-turn ("did you mean fiscal or calendar?") is a v1.x extension.
• No result-explanation in v1 (NL output describing query results). Also v1.x.

---

8. Repository layout

coframe/
├── README.md
├── pyproject.toml                 # workspace root, uv config
├── uv.lock
├── CLAUDE.md
├── docs/
│   ├── design/
│   ├── manual/
│   └── article/
├── packages/
│   ├── coframe-core/
│   │   ├── pyproject.toml
│   │   ├── src/coframe/...        # the module tree from §2.3
│   │   ├── tests/
│   │   └── CLAUDE.md
│   ├── coframe-connect/
│   │   ├── pyproject.toml
│   │   ├── src/coframe/connect/...
│   │   ├── tests/
│   │   └── CLAUDE.md
│   ├── coframe-polars/
│   │   ├── pyproject.toml
│   │   ├── src/coframe_polars/
│   │   │   ├── __init__.py
│   │   │   ├── backend.py
│   │   │   ├── operators.py
│   │   │   ├── mappers.py
│   │   │   └── author/
│   │   │       └── ... (see §4.4)
│   │   ├── tests/
│   │   └── CLAUDE.md
│   ├── coframe-duckdb/
│   │   ├── pyproject.toml
│   │   ├── src/coframe_duckdb/
│   │   │   ├── __init__.py
│   │   │   ├── backend.py
│   │   │   ├── operators.py
│   │   │   ├── mappers.py
│   │   │   └── author/
│   │   │       └── ... (see §5.4)
│   │   ├── tests/
│   │   └── CLAUDE.md
│   └── coframe-mcp/
│       ├── pyproject.toml         # depends on coframe-core only
│       ├── src/coframe_mcp/...
│       ├── tests/
│       └── CLAUDE.md
├── tests/
│   ├── reference_suite/           # cross-backend execution conformance
│   │   ├── retail_ac/
│   │   ├── budget_actuals_ac/
│   │   └── queries.yaml
│   └── conformance/
│       └── backend/               # Backend protocol contract
└── tools/
    ├── ac_validate.py
    └── ql_repl.py

tools/coframe_author.py is gone — authoring lives inside each backend now, with its own per-backend CLI script (coframe-polars-author, coframe-duckdb-author) registered via [project.scripts] in each backend's pyproject.toml.

Namespace: coframe-core and coframe-connect share the coframe top-level (coframe, coframe.connect). Backends use underscore form (coframe_polars, coframe_duckdb).

---

9. Testing strategy

9.1 Reference test suite

Manual's Appendix C retail AC + a budget-vs-actuals AC (defined in tests/reference_suite/budget_actuals_ac/, since the Manual does not currently include a budget-vs-actuals worked example) + 200–400 Frame-QL queries with expected results. Same suite runs against coframe-polars and coframe-duckdb; identical results required.

Coverage: each Rung (0–9), each operator/mapper, WITH-block patterns, cross-schema MTI, lightweight ratios, dubious-query detection, edge cases, AC customizations.

9.2 Unit tests per module

Each module in §2.3 has its own tests/unit/test_<module>.py.

9.3 Property-based tests

Combination-law associativity per registered monoid; FD-DAG path consistency; MTI invariance; round-trip parser/pretty-printer; name_map consistency under random AC generation.

9.4 Conformance tests

tests/conformance/backend/ — Backend protocol contract.

The conformance suite includes attestation contract tests: synthetic AC with known DNA edges and seeded coherence violations; assert each backend's attest_dna_edge returns the expected passed | failed_with_deltas | unattestable status with correctly-localized disagreements; verify operator-catalog missing-value semantics are honored (mean-substitution under MCAR-effective produces different results than naive SQL skip; the conformance suite asserts the framework's defined behavior, not SQL defaults).

(No authoring conformance tests — authoring is per-backend, with no cross-backend interface to conform to.)

9.5 Attestation tests

tests/attestation/ — exercises the full attestation pipeline.

• Plan correctness. For a synthetic AC with N edges of known attestability, assert the planner correctly classifies attestable / unattestable-no-predecessor / unattestable-non-partition-invariant edges.
• Per-edge correctness. For seeded coherence-violation AC variants (insert known deltas in pre-aggregated tables), assert the runner reports the correct delta count, delta magnitudes, and top-N disagreements.
• Failure-mode behavior. Same seeded violations under each failure_mode: hard produces validation failure; soft produces advisories with results queryable; tolerated requires explicit tolerated_edges declarations and produces no annotations on tolerated edges.
• Sampling correctness. For large synthetic fact tables exceeding the configurable threshold, assert sampling produces correct status with the stated confidence level.
• Missing-value semantics. For a column with declared MCAR signature, assert attestation honors mean-substitution (not naive SQL skip) when computing the predecessor aggregation.
• Opt-out behavior. With attestation.enabled: false, assert validation skips I10, the verification status records the opt-out, and query results carry coherence-asserted-not-verified annotations.
• Differential re-attestation. Run attestation, change one schema's data, re-run; assert only edges involving the changed schema are re-attested, others reuse cached status.

9.6 NLQ tests (coframe.dialogue)

• Translation correctness. A curated set of (utterance, expected Frame-QL) pairs spanning each Rung. The LLM is run in a pinned mode; expected matches are checked modulo formatting.
• Validation enforcement. A test where the LLM is mocked to produce invalid Frame-QL; assert the validator catches it and (a) retries or (b) surfaces a clear failure.
• Logical-only assertion. Capture all prompts sent during dialogue tests; assert no physical column names appear.
• Vendor parity. A subset of translation tests run against multiple LLM vendors (Claude, GPT, Gemini); divergence above a threshold surfaces as a flake-tagged warning, not a hard failure.

9.7 Translation pass tests

(Same as v0.4: round-trip column-translation, customization-expansion ordering.)

9.8 Backend authoring tests (per-backend)

Each backend's authoring code has its own test suite, in its own tests/ directory. There is no cross-backend authoring conformance suite. The bar:

• End-to-end: run bootstrap against synthetic source data with known AC structure → verify produced AC matches expected modulo naming variation.
• Determinism: with the LLM stubbed/replayed, the same input produces the same output.
• Resume: run a partial review, save state, reload, complete review — verify AC is identical to a single-shot run.

These suites are independent. A change to coframe_polars.author does not require updating coframe_duckdb.author or vice versa.

9.9 What we deliberately do not test in v1

Performance regression beyond Phase 8 quantitative targets (per §11); concurrency under load; backend optimizer behavior; LLM proposal quality across vendors (for authoring).

---

10. Public API stability

Pre-1.0: everything is unstable.

At 1.0, stable surfaces (semver-governed):

• AC YAML format (including name_map schema, customization syntax, and the attestation: block).
• Frame-QL grammar.
• coframe.connect.Backend protocol (including attest_dna_edge signature, EdgeColumnSpec, ScopeFilter, EpsilonConfig, SamplingConfig, AttestationResult shapes).
• coframe.quasi_metadata schema.
• coframe.attestation verification-status shape and the per-edge result shape used in MCP responses (coherence_posture field per Manual §11.7.3).
• AC class public API.
• coframe.dialogue.translate signature and NlqProvenance shape.
• Per-backend authoring CLI surfaces (coframe-polars-author, coframe-duckdb-author — the documented argument shape, not the internal modules).
• Entry-point group name (coframe.backends).

At 1.0, informational surfaces (documented and exposed but not yet semver-governed; will become stable in v1.x):

• AC Verification Levels (A / AA / AAA per Manual §7.13). The level taxonomy, the level-from-grounding-map function, and the public VerificationStatus.level shape (including GroundingSummary) are documented and reported. v1.x will lock the taxonomy as a semver-protected commitment after field experience informs any edge-case calibrations.

Internal modules — resolution internals, parser internals, dialogue prompts, authoring internals per backend, attestation planner internals — are not stable.

---

11. Build phasing

Phase 0 — Skeleton (1 week)

• Monorepo with uv workspace.
• Five package skeletons.
• Repo CI (ruff, pyright, pytest).
• CLAUDE.md files.
• Manual and article in docs/.

Exit: uv sync && pytest runs cleanly.

Phase 1 — Core foundations (3–4 weeks)

coframe-core modules excluding ql/, resolution/, dialogue/:

• types, errors, selves, combination, operators (system), mappers (system), customization/*, fd_dag, column_spec, name_map, ratios, missing, quasi_metadata, integrity (data-free I0/I1/I2/I7/I8/I9), ac, catalog.
• attestation/config.py — Pydantic models for the AC YAML's attestation: block; defaults; opt-out enforcement (explicit enabled: false required).
• attestation/plan.py — edge enumeration from the AC's metric genealogy; attestability classification (attestable / unattestable-no-predecessor / unattestable-non-partition-invariant).

Exit: load Appendix C retail AC YAML (with name_map, a customization or two, and an attestation: block); data-free integrity clean; the planner correctly classifies the retail AC's edges.

Phase 2 — Frame-QL parser & semantic analysis (2–3 weeks)

• coframe.ql.lexer, parser, ast, semantics, pretty.
• Conform to Appendix A BNF.
• Round-trip parser/pretty-printer.

Exit: parse every example query in Manual Chapter 8 (Frame-QL, including the Rung-by-rung examples in §8.8); pretty-prints to equivalent input.

Phase 3 — Resolution + customization expansion + translation (4–5 weeks)

• All resolution/ modules including customization and translation passes.
• Resolved AST node classes finalized.
• plan/resolver.py orchestrator.

Exit: resolve all reference-suite queries (without execution) into resolved ASTs; reject the dubious subset with correct diagnostics.

Phase 4 — coframe-connect + coframe-polars (5–6 weeks)

This phase now includes polars authoring (since authoring lives in the backend), per-DNA-edge attestation runner support, and AC verification level computation.

• coframe-connect: Backend protocol (including attest_dna_edge), WalkerBackend mixin, entry-point discovery, conformance suite (including attestation contract tests per §9.4).
• coframe-core: attestation/runner.py and attestation/status.py — orchestrates per-edge attestation calls via the bound backend; aggregates per-edge results into AC-level verification status; computes VerificationStatus.level (A / AA / AAA per Manual §7.13 and platform-design §2.10); threads I10 into the integrity pipeline.
• coframe-polars execution: BackendData = LazyFrame, system operator/mapper translation tables, execute walker, compute_qm, attest_dna_edge (full attestation up to RAM-bounded sizes; sampling fallback per config), entry-point registration.
• Wire data-dependent integrity checks (I3–I6, I10) using polars backend.
• coframe_polars.author: complete authoring toolchain (discover, profile, fd_candidates, propose, llm, prompts, review, emit, provenance, cli).
• Per-backend authoring tests; attestation-correctness tests on synthetic seeded-violation ACs (per §9.5); level-computation tests covering A / AA / AAA / no-level paths and the level-from-integrity-results determinism property.

Exit: - Reference suite green against coframe-polars. - I10 attestation correctly identifies seeded coherence violations in synthetic ACs; failure-mode behaviors (hard/soft/tolerated) verified. - Retail AC's attestable edges complete attestation in under 5 minutes on the reference dataset; level computation correctly classifies the retail AC as AAA. - Synthetic seeded-violation ACs correctly classify at lower levels (failed I6 → A; opt-out → AA; failed attestation edge → AA). - coframe-polars-author bootstrap reproduces the retail AC from raw Parquet sources in under 30 minutes of engineer review time, matching Appendix C modulo naming.

Phase 5 — coframe-duckdb (4–5 weeks)

Mirror of Phase 4 with DuckDB.

• Execution backend (including attest_dna_edge with spill-to-disk for full attestation on tables exceeding RAM, and DuckDB-native stratified sampling for the size-budget fallback).
• Persistent-connection mode caches attested-edge status in a system table; differential re-attestation runs only against changed schemas.
• coframe_duckdb.author: complete authoring toolchain, independently structured. Patterns may rhyme with the polars version but no shared code in v1.
• Per-backend authoring tests; attestation-correctness tests including the sampling-fallback path.

Exit: - Reference suite green against coframe-duckdb; cross-backend results identical. - I10 attestation results identical between polars and duckdb backends on the retail AC (this is a real cross-backend correctness check — both backends must produce the same delta detection on the same seeded violations). - Retail AC's attestable edges complete full attestation in under 5 minutes on the reference dataset; sampled attestation under 30 seconds. - coframe-duckdb-author bootstrap reproduces the retail AC with persistent backend.

Phase 6 — coframe.dialogue (1–2 weeks)

• coframe.dialogue.translate and the NlqProvenance type.
• LLM client with vendor switching.
• Prompt templates (system, ac_summary, examples).
• Validators using coframe.ql.parser and the resolution pipeline.
• NLQ test suite (§9.6).

Exit: - Translate the curated test suite of utterances against the retail AC with high accuracy (target: 90%+ valid; 80%+ matching expected within tolerance) across at least Claude. - Logical-only assertion test passes (no physical names in any prompt).

Phase 7 — coframe-mcp (2–3 weeks)

• MCP server skeleton (depends only on coframe-core).
• All execution capabilities from §6.2.
• nl_query capability backed by coframe.dialogue.
• coherence_posture and edge-level annotation propagation on execute_query and nl_query results (per Manual §11.7.3), including the AC's verification level (A / AA / AAA) and naming-consistency status.
• validate_ac capability surfaces the AC's verification status including I10 attestation results and the computed level.
• Configuration loader using entry-point backend discovery.
• Integration tested against Claude Desktop and Claude API.

Exit: - Claude Desktop connects via stdio, lists ACs, describes selves, executes Frame-QL queries, and translates+executes natural-language queries against the retail AC. - coherence_posture correctly propagates on query results, including the AC's level, for default-configured (AAA), opt-out-configured (AA), and seeded-violation-configured (mixed) ACs. - Same coframe-mcp source successfully serves polars-backed and duckdb-backed configurations.

Phase 8 — Hardening, docs, release (3–4 weeks)

• Performance pass with quantitative targets:
• Query resolution (parse + four-rule filter + plan, no execution): P50 under 50 ms for retail AC reference suite.
• End-to-end query (parse → resolve → execute → return) for in-cache queries: P50 under 200 ms for retail AC reference suite on duckdb backend.
• Full I10 attestation for the retail AC: under 5 minutes total on duckdb backend with reference dataset (~50M-row transactions table).
• Sampled I10 attestation for the retail AC: under 30 seconds.
• QM cold-load for retail AC: under 10 seconds.
• Verification level computation from cached integrity results: P50 under 1 ms (level computation is a pure function over the integrity-result map; should be effectively free, but the budget makes the commitment explicit).
• Documentation: API docs, getting-started, AC authoring tutorial (per backend), MCP deployment guide, AC customization guide, NLQ usage guide, attestation operational guide (covering failure modes, configuration patterns, debugging coherence advisories, the tolerated_edges declaration, opt-out semantics), verification levels guide (explaining A / AA / AAA semantics, what each level promises consumers, when to opt out, when to use tolerated_edges, how levels propagate to MCP results — written for both AC authors and AC consumers), working with time-varying data guide (Core's approach to time-varying attributes: when to model time-variance as event-time-anchored events, when to use snapshot-fact-table patterns, how to recognize when an AC genuinely needs SCA semantics that Coframe Pro provides — written for AC authors evaluating whether their data fits Core's scope), working with hierarchical data guide (modeling patterns for dimension hierarchies in Coframe: linear chains, multi-path hierarchies, diamond hierarchies, ragged hierarchies; when to use data-attested FDs versus function-derived FDs; recognizing when an AC has recursive/self-referential structure that requires Coframe Pro — written for AC authors coming from Kimball-tradition backgrounds who need help mapping their existing schemas to Coframe's FD-DAG framing).
• 3–5 worked-example ACs at varying verification levels: one at AAA (the retail AC's reference state), one at AA via opt-out, one at AA via seeded coherence violations, one with intentionally-seeded tolerated_edges declarations to demonstrate the AAA-with-tolerance pattern.
• Pre-release versioning, changelog, release process. Levels documented as informational v1.0, stable v1.x in release notes.
• Public 1.0 release.

Total: ~24–30 weeks. Phases 4 and 5 are now larger (each absorbs its backend's authoring and per-DNA-edge attestation); Phase 4 also lands level computation. Phase 6 is much smaller (only coframe.dialogue); Phase 7 gains the nl_query capability and coherence_posture propagation including the level field; Phase 8 carries quantitative performance targets including attestation runtime budgets and the verification-levels guide as a documentation deliverable.

---

12. Open questions and deferred decisions

Resolved at v1.0 publication:

• License. Apache 2.0 (per §0.2).
• AC catalog format versioning. Top-level coframe_version: "1.0" field in every AC YAML. The field is required in v1.0 ACs; absence is a load-time error (with diagnostic suggesting the version stamp). Future minor-version bumps within v1.x are read-compatible; major-version bumps require explicit migration tooling to be shipped alongside the bump.
• Diagnostic format. Two-form: structured (Pydantic Diagnostic model with code, severity, location, message, hints) for programmatic consumption (MCP responses, CI tooling, IDE integrations), human-rendered string form derived from the structured form for terminal display. The structured form is the source of truth; the human form is a deterministic projection.
• Per-backend authoring CLI naming. Independent CLIs: coframe-polars-author and coframe-duckdb-author. Authoring is per-backend (no shared library in v1); two CLIs preserve that boundary cleanly. A unified coframe-author --backend X is a v1.x candidate if convergence emerges.

Deferred to later phases (settle in the indicated phase):

• Quasi-metadata schema versioning. Independent semver from the catalog format. Settle Phase 1.
• Binding override file format. Settle Phase 1.
• I9 strictness. Behavior when a name_map entry exists for a logical name that no ColumnSpec references — warning vs. error vs. silent. Settle Phase 1.
• Customization-syntax niceties. Settle Phase 1.
• Plan caching key. Settle Phase 3.
• DuckDB relation API vs. SQL strings. v1 commits to relation API; revisit Phase 5.
• Authoring confidence threshold for sampled FD checks. Settle Phase 5 (duckdb).
• Provenance file format. Sidecar JSON vs. YAML comments. Settle Phase 4 (polars sets the precedent; duckdb follows in Phase 5).
• Dialogue LLM retry policy. Max retries on parse/semantic failures, prompt-engineering for corrections. Settle Phase 6.
• Multi-turn dialogue. v1 is single-turn; multi-turn is v1.x.
• MCP transport. stdio default; HTTP optional. Confirm Phase 7.
• Attestation default failure mode for v1.0 release. Manual specifies soft as the default; this is calibrated for first-time adoption against real warehouses. Open question: should the AC bootstrap CLI (per backend) default to writing failure_mode: soft explicitly into emitted YAML, or rely on absence-implies-default? Recommendation: explicit, for visibility. Settle Phase 4.
• Attestation epsilon defaults per data type. 1e-9 relative for numeric/float; 0 absolute for integer/decimal — but decimal types in DuckDB have configurable precision and may need a non-zero relative tolerance under some configurations. Settle Phase 5.
• Attestation cache invalidation policy. When does a previously-attested edge become stale? On any change to either schema's data; on changes to the AC YAML's relevant ColumnSpecs; on epsilon changes. Settle Phase 4 (polars baseline) and Phase 5 (duckdb's persistent cache adds complexity).
• Attestation parallelism. Edges within an AC can be attested in parallel; backends must declare their concurrency model. Settle Phase 4/5.

---

13. What this document is not

• Not a marketing document.
• Not a user guide. The Manual is the spec; user-facing guides come in Phase 8.
• Not a final architecture. Major design refinements will surface as new docs in docs/design/.

---

14. Forward-looking — what this v1 enables for Coframe Pro

• Code-bearing operator extensions will register via a coframe.operators entry-point group. AC customizations will reference them as if they were system operators. v1's customization mechanism doesn't change.
• Per-schema name overrides can be added as opt-in extensions to name_map if ever needed.
• Multi-turn dialogue, result explanation, query refinement all build on the existing coframe.dialogue surface — the LLM client, prompt scaffolding, and provenance type are reusable.
• More sophisticated authoring (multi-pass, schema evolution, web UI) can build on each backend's authoring toolchain. Convergence between the two backends' authoring code may happen in v1.x, possibly via a shared backend-side library extracted then rather than designed up front.
• Distributed and streaming backends implement the same Backend protocol; entry-point discovery loads them; coframe-mcp serves them without source change.
• Per-AC backend variants (different schemas in one AC bound to different backends) can extend with_backend to a with_backends mapping; the resolved-AST contract doesn't change.
• Attestation extensions in Coframe Pro build on Coframe Core's per-DNA-edge attestation foundation: federated-edge attestation across multi-backend ACs (the attest_dna_edge protocol generalizes when predecessor and successor live in different backends; the runner orchestrates the cross-backend computation); attestation-driven sensitivity analysis (when an edge has a partial-coverage advisory, Coframe Pro's sensitivity machinery can produce bounded estimates rather than point estimates for queries depending on that edge); incremental attestation with finer-grained invalidation (cache attested-edge status with content hashes; invalidate only on actual data changes); importance sampling and stratification by anchor distribution for very large fact tables. Coframe Core's attestation surface — the Backend protocol method, the AC YAML configuration block, the verification-status shape, the MCP coherence_posture field — is the stable foundation Coframe Pro builds on.

The v1 foundation is intentionally minimal but extensible. Coframe Pro extends; it does not replace.