[BOT-1367] Representations based notebook query tooling (#73052)

* agent-lib: representations YAML parser + schema validator

First step of migrating construct_notebook_query from the sexp-in-array DSL to
the canonical MBQL 5 representations format (see repr-plan.md).

Adds metabase.agent-lib.representations with:

- parse-yaml: wraps metabase.util.yaml with :keywords false so 'lib/type'
  string keys survive intact (we lose the 'lib/' namespace segment if
  clj-yaml keywordizes them). Throws :invalid-representations-yaml with
  :agent-error? false for malformed input.
- validate-query: Malli schema covering the Phase-1 subset (mbql/query top
  level, single-stage mbql.stage/mbql, portable FK vectors for source-table
  and field refs, clause shape [op {opts} ...args]). Structural only; deep
  clause content is validated by lib.schema/Query after resolution.
- parse-and-validate: convenience combinator.

Implementation notes captured in the plan:

- The Malli schema avoids :cat+:ref recursion (triggers
  :malli.core/potentially-recursive-seqex). Clause-shape is encoded as a
  flat [:and vector? [:fn ...]] structural predicate.
- clj-yaml returns #ordered/map (OrderedMap) which satisfies map?
  - no special handling needed in downstream passes.

* models.serialization.resolve: MetadataProvider-backed resolver

Adds metabase.models.serialization.resolve.mp with import-resolver and
export-resolver factories that satisfy SerdesImportResolver /
SerdesExportResolver via a lib.metadata.protocols/MetadataProvider rather
than the application DB.

This lets the representations resolver (next commit) translate portable
FK paths like [Sample, PUBLIC, ORDERS, CREATED_AT] to numeric IDs without
round-tripping through toucan2, so it works for any MetadataProvider the
caller supplies (application DB, mock, saved-question graph, etc.).

Implemented methods (sufficient for Phase 1):
- import-table-fk / export-table-fk
- import-field-fk / export-field-fk  (handles JSON-unfolded nested fields
  by walking :parent-id upward on export and descending name+parent-id
  on import)

Other protocol methods throw :not-implemented-yet so unsupported serdes
entry points fail loudly rather than silently dropping through.

Implementation notes:
- Uses lib.metadata.protocols/table and /field directly, not the mu/defn
  wrappers in lib.metadata - those throw Malli validation errors on nil
  returns; the protocol methods return nil cleanly on a miss, which is
  what we want for 'unknown-table' / 'unknown-field' errors.

* agent-lib: representations resolver (portable YAML -> canonical MBQL 5)

Step 3 of the representations migration. Adds
metabase.agent-lib.representations.resolve with:

- keywordize-query: recursively converts string map keys to keywords
  while preserving string *values* (so 'lib/type: mbql/query' becomes
  :lib/type :mbql/query but portable FK strings like 'ORDERS' stay as
  strings).
- resolve-query: main entry point. Keywordizes -> binds
  resolve/*import-resolver* to the mp-backed resolver -> calls
  resolve/import-mbql -> normalizes through lib.schema/query with the
  MetadataProvider attached.
- parse-validate-resolve: YAML string + MP -> MBQL 5 query in one call.

Also extends .../resolve/mp.clj: implements import-fk-keyed for
:model/Database by name (needed because import-mbql dispatches on the
top-level :database key). Throws :unknown-database with :agent-error?
true when the YAML references a database other than the MetadataProvider's.

Kondo module config updates:
- Adds metabase.agent-lib.representations{.repair,.resolve} to the
  agent-lib module's :api.
- Adds 'models' to agent-lib's :uses (resolver lives under
  metabase.models.serialization.resolve.*).
- Exposes metabase.models.serialization.resolve.mp on the models :api.

Tests cover aggregation+breakout with :temporal-unit, filters+limit+
order-by, fields projection, explicit join with alias+strategy+join-alias,
and three error paths (unknown-table, unknown-field, database-mismatch).

* agent-lib: representations repair pass (fills in missing {} options and lib/type markers)

Step 4 of the representations migration. LLMs reliably drop two things
when writing repr YAML:

1. The empty options map in clauses - writing [count] instead of
   [count, {}], or [field, [Sample, PUBLIC, ORDERS, ID]] with no
   options slot.
2. The lib/type markers on the top-level query and on stages.

Rather than surfacing these as validation errors to the LLM (which wastes
turns and tokens), we normalize them in a repair pass before validation.

metabase.agent-lib.representations.repair composes two idempotent passes:

- ensure-clause-options*: inserts {} at position 2 of clause-like vectors.
  Three cases handled:
    [op]                     -> [op, {}]
    [op, nil, ...]           -> [op, {}, ...]
    [op, <non-nil-non-map>]  -> [op, {}, <non-nil-non-map>]  (insert,
                                                              never replace)
  The clause-like? predicate distinguishes real clauses from portable FK
  paths via looks-like-fk-path? (length >= 3, all string?/nil?). It also
  excludes map-entry? so clojure.walk/postwalk doesn't misfire on map
  traversal.

- ensure-lib-types*: fills 'lib/type': 'mbql/query' on top-level query
  maps and 'lib/type': 'mbql.stage/mbql' on stage-like maps. Uses string
  keys because repair runs pre-keywordization.

Both passes are idempotent - running them twice is equivalent to running
them once. The test suite proves this generatively (defspec, 100
iterations) in addition to 19 example-based assertions.

* metabot: flip construct_notebook_query to representations YAML format

Step 5 of the representations migration. Wires the parse->repair->
validate->resolve pipeline into the construct_notebook_query tool,
replacing the sexp-in-array program format.

metabase.metabot.tools.construct:
- New execute-representations-query: resolves source-entity's database ->
  builds an application-DB-backed MetadataProvider -> runs the repr
  pipeline -> wraps result in the same {:structured-output {...},
  :instructions ...} shape as execute-program. Any ExceptionInfo from the
  pipeline is re-thrown with :agent-error? true so the tool wrapper
  surfaces a helpful message to the LLM instead of a stack trace.
- Tool arg schema: :program is replaced by :query :string (YAML string).
  Kept flat at :string to sidestep MCP flatten-root-schema pitfalls;
  structural validation happens inside execute-representations-query.
- construct-notebook-query-tool now calls execute-representations-query.
- execute-program and construct-program-schema remain in place; they're
  still used by slackbot-construct-notebook-query-tool during the
  migration. Slackbot moves to repr in a later phase.
- resolve-source-database-id is now defn (was defn-) so tests can stub it
  with with-redefs.

metabase.metabot.tools.util:
- ->result-column now emits a :portable_fk key alongside :field_id when
  the column maps to a real database field. Walks the :parent-id chain
  to build the path for JSON-unfolded nested fields. Degrades to nil for
  expression/aggregation columns or when the MetadataProvider can't
  resolve the path. Numeric ids are preserved so existing callers
  (chart creation) keep working.

resources/metabot/prompts/tools/construct_notebook_query.md:
- Fully rewritten for the Phase-1 MVP subset. Documents top-level shape,
  stage shape, the 'options map at position 2 is always present' rule,
  portable FK format (3-element table, 4+ element field, null-schema for
  schemaless DBs, JSON-unfolded nested paths), examples for each MVP op
  (filter incl. 'and', aggregation, breakout with :temporal-unit,
  order-by, limit, fields projection), explicit joins with :join-alias
  propagation, implicit joins via repair, and a Phase-1 scope-out list.

Tests:
- test/metabase/metabot/tools/construct_representations_test.clj: new,
  4 tests / 12 assertions. Happy path (YAML -> fully-resolved MBQL 5),
  malformed YAML surfaces :agent-error?, unknown table surfaces
  :agent-error?, and a broken LLM-style YAML with missing lib/type and
  missing {} options round-trips cleanly through repair.
- test/metabase/metabot/tools_test.clj: migrated
  construct-notebook-query-tool-test from :program / execute-program to
  :query (YAML string) / execute-representations-query.

* metabot: expose portable FK paths in entity_details; disable implicit-joins claim in prompt

Two small follow-ups needed to actually use construct_notebook_query in
its new representations form:

1. entity_details' table-details output now includes:
   - :database_name at the top level
   - :portable_fk: [db-name, schema-or-null, table-name] matching the
     format the representations tool expects for source-table
   Per-field portable FKs were already exposed via ->result-column in
   the previous commit. Without the table-level db-name, the LLM had
   no way to construct the 3-element table FK path.

2. The construct_notebook_query prompt previously promised implicit
   joins via a repair pass ('Metabase will fill in source-field for
   you'). That repair pass isn't implemented yet - it's Phase 1 Step 6.
   Flipped the prompt language to explicitly say implicit joins aren't
   supported yet, and added it to the Phase-1 scope-out list. Users who
   want cross-table fields must write an explicit joins: section.

* metabot: migrate remaining integration tests + document_construct_model_chart to representations

After flipping construct_notebook_query to the representations format,
three other call sites were still sending the legacy :program payload:

1. metabase.metabot.tools.document/document-construct-model-chart-tool
   wraps construct-notebook-query-tool and forwarded its :program arg
   through unchanged. Renamed its schema field :program -> :query and
   updated the inner call. This was an actually-broken call site, not
   just a test.

2. test/metabase/metabot/agent/core_test.clj integration-search-query-
   chart-flow-test drives the real agent loop with a scripted LLM and
   exercises construct_notebook_query end-to-end against the H2 test
   database. Migrated its scripted tool-call to the representations
   YAML format. The test looks up the real database name, schema, and
   table name at runtime to build portable FK paths, so it stays
   resilient to the test DB details. JSON-encoded flow-sequence
   literals embedded in the YAML string (valid YAML since YAML is a
   superset of JSON) avoid the pr-str pitfall where Clojure writes
   [nil ...] instead of [null, ...].

3. test/metabase/metabot/tools/charts/edit_test.clj's
   edit-chart-of-constructed-query-test is another real integration
   test - it calls construct-notebook-query-tool against the products
   table and then exercises edit-chart-tool on the resulting chart.
   Migrated to use the portable_fk fields from entity_details instead
   of numeric field-ids.

4. test/metabase/metabot/tools/document_test.clj's
   document-construct-model-chart-tool-test was a unit test using
   with-redefs; just updated the arg shape to match the new schema.

All three integration tests now pass against the representations
pipeline end-to-end. The last two are especially valuable because they
exercise the full parse -> repair -> validate -> resolve -> QP path
with a real MetadataProvider, which is what will happen in production.

Uses metabase.util.json instead of cheshire directly, per kondo lint.

* agent-lib: auto-wire source-field for implicit joins in representations repair

Phase 1 step 6 of repr-plan.md.

Extends the representations repair pass with a third pass that walks the
first stage's clauses and fills in the `source-field` option on any
`[field, {...}, [db, schema, OTHER_TABLE, col, ...]]` reference whose target
table is reachable from `source-table` via exactly one foreign key. The QP
interprets a field with `source-field` as an implicit join, so the LLM no
longer needs to author an explicit `joins:` entry for the common
single-FK case.

* `repair` now takes a `MetadataProvider` as its second argument. The
  shape passes (ensure clause options, lib/type markers) are unchanged and
  don't need the MP; the new implicit-join pass does. Public surface stays
  one function.
* When the source-table can't be resolved in the MP, the pass silently
  no-ops and leaves the error reporting to the downstream validate /
  resolve stages (which have better messages for that case).
* When there is no FK at all, or more than one FK, the pass throws
  ex-info with `:agent-error? true` and `:error :no-fk-path` or
  `:ambiguous-fk`. `:ambiguous-fk` surfaces the candidate FK portable
  paths so the LLM can pick one and retry with an explicit
  `source-field` option.
* Field clauses that already carry `source-field` or `join-alias` are
  left alone. The `joins` subtree of a stage is not descended \u2014 its
  field clauses live in a join context and are expected to carry a
  `join-alias` already.
* Adds `resolve.mp/outbound-fks-from-table` to return the FK edges
  leaving a given table, so the repair pass can precompute a
  target-table-id -> candidate-FKs map once per stage.
* `tools/util/->result-column` now emits a `:fk_target_portable_fk`
  alongside `:portable_fk` for columns with an `:fk-target-field-id`,
  so the LLM can discover candidate source-fields via `entity_details`.
* Prompt: replaces the "Implicit joins \u2014 not yet supported" section
  with concrete rules and examples; removes the scope-out entry.

Test counts (before \u2192 after):
* agent-lib/representations.repair-test: 14 \u2192 19 tests, 17 \u2192 35 assertions
  (happy-path, skip-on-source-field, skip-on-join-alias, skip-on-source-table,
  no-fk, ambiguous, skip-joins-subtree, idempotence, skip-on-unresolvable-mp)
* serialization/resolve.mp-test: 7 \u2192 10 tests, 34 \u2192 37 assertions
  (outbound-fks happy path, no-FK table, simple MP)
* metabot.tools.construct-representations-test: 4 \u2192 6 tests, 12 \u2192 17
  assertions (implicit-join happy path, ambiguous surfaces agent error)
* All existing metabot/agent-lib tests stay green: 100 tests, 629 assertions.

* agent-lib: rewrite portable database name to match metadata provider in repair pass

Our prompt examples for construct_notebook_query consistently use `Sample`
as the database name. The LLM follows them literally, producing a query like:

    database: Sample
    source-table: [Sample, PUBLIC, ORDERS]

against the real application DB whose name is `Sample Database`. The
metadata-provider-backed resolver requires exact string equality on the DB
name, so the canonical demo flow (e.g. "total revenue per product category")
fails with :unknown-database before the query can even be validated.

Per repr-plan.md step 13, source_entity (and thus the metadata provider) is
the ground truth for the target database -- the YAML's `database:` key and
the DB component of every portable FK are redundant w.r.t. the MP. Add a
new repair pass that overwrites both with the MP's actual DB name before
validation/resolution.

The pass is permissive: any DB name (real, prompt-derived, hallucinated)
gets normalised to the MP's name. Cross-DB queries are not supported by
this tool, so the MP unambiguously identifies the only legal DB.

* Unit tests in repair_test.clj: top-level rewrite, portable-FK rewrite,
  no-op when names already match, no-op when mp=nil, idempotency.
* Reproducer in construct_representations_test.clj: verbatim YAML from the
  bug report against an MP named "Sample Database".
* End-to-end test in charts/edit_test.clj: same YAML against the real
  application sample DB, asserting full chart construction + SQL compile.

* agent-lib: rewrite inline order-by aggregations to aggregation refs in repair pass

When the user asks for something like "top categories by total revenue",
the LLM tends to write `order-by` by re-stating the aggregation expression
inline:

    aggregation: [[sum, {}, [field, {}, [..., TOTAL]]]]
    order-by:    [[desc, {}, [sum, {}, [field, {}, [..., TOTAL]]]]]

In MBQL 5 / legacy MBQL the `order-by` direction must wrap an aggregation
**reference**, not the aggregation expression itself. `lib.normalize`
accepts the inline form, but `lib/->legacy-MBQL` produces a query that
fails legacy-schema validation the moment something downstream re-loads
it (e.g. the chart is opened, the QP round-trips through legacy MBQL and
explodes with `Invalid output: {:query {:order-by ...}}`). The agent
reports success but the user sees a broken chart.

Add a repair pass that walks each stage's `order-by`, structurally
matches each inner clause against the stage's `aggregation` list
(ignoring options-map differences), and rewrites the inline expression
to `["aggregation" {} "<uuid>"]` -- synthesising the matching
aggregation's `lib/uuid` if it doesn't already have one. Idempotent:
existing aggregation references and unmatched expressions are left alone
so validation/normalize can surface real errors.

This does NOT introduce general placeholder-UUID handling (Phase 2 step
10's `@agg-N` references); we only resolve this specific recurring
mistake.

* Update prompt with a canonical "Order by an aggregation" example
  showing that the inner expression must match an entry in
  `aggregation:`.
* Unit tests in repair_test.clj: happy path, multiple aggregations
  matched independently, reuses pre-existing lib/uuid, leaves existing
  aggregation refs alone, leaves field-orderings alone, leaves
  no-match aggregations alone, idempotency.
* Reproducer in construct_representations_test.clj: verifies the
  resolved query round-trips through legacy MBQL cleanly.
* End-to-end test in charts/edit_test.clj: same YAML against the real
  application sample DB, asserting full chart construction + SQL
  execution returning rows in descending sum order.

* agent-lib: drop `source_entity` and `referenced_entities` from construct_notebook_query (repr-plan step 13)

The YAML query is self-describing: it carries `database: <name>` at the
top level and full portable FK paths (`[<db>, <schema>, <table>]`)
everywhere else. The separate `source_entity` parameter was redundant
and could disagree with the YAML, forcing the LLM to keep two things in
sync for no reason.

* `construct-notebook-query-args-schema`: down to `{reasoning?, query,
  visualization?}`. Removed the now-unused `source-entity-schema`.
* `execute-representations-query`: signature is now `[yaml-string]`.
  Database id is derived from the parsed YAML's `database:` field by
  the new public helper `resolve-database-id-from-yaml`. Three
  `:agent-error?` paths:
  - `:missing-database-name` -- YAML lacks a `database:` field.
  - `:unknown-database` -- no `:model/Database` matches that name;
    error message instructs the LLM to use the canonical name reported
    by `entity_details`.
  - `:ambiguous-database-name` -- two databases share that name; LLM
    has no way to disambiguate, so this is a hard error.
* `resolve-source-database-id` kept untouched (still used by
  `execute-program` for slackbot until step 14 migration).
* `document-construct-model-chart-tool` lost its `:source_entity`
  schema entry too -- it's a thin pass-through to the construct tool.

The `rewrite-database-name*` repair pass (Pass 2.5, added during the
step-7 smoke test as a tactical bridge) is now obsolete: the MP is
*built from* the YAML's `database:`, so by construction the names
agree, leaving the pass nothing to do. Per repr-plan.md option (b)
("strict-with-good-error"), the pass and its 5 unit tests are deleted;
the LLM gets a clear error message instead of a silent rewrite. The
`metabase.lib.metadata` import in repair.clj goes with it.

Prompt update (`construct_notebook_query.md`): the `source_entity` /
`referenced_entities` bullets are removed; the `database:` field
description now reads "the exact database name as reported by
`entity_details`" with an explicit warning that misses fail loudly.

Tests:
* All 9 call sites in `construct_representations_test.clj` switched to
  the new single-arg signature. The old DB-name reproducer is rewritten
  to assert the new loud-failure behaviour. Added 3 new tests: missing
  `database:`, non-string `database:`, and `database:` /
  `source-table[0]` mismatch.
* `charts/edit_test.clj`: 4 call sites switched. The DB-name E2E test
  now asserts the agent-targeted error (no chart, just a clear
  `Unknown database` message). Added a sibling
  `construct-notebook-query-llm-uses-canonical-db-name-end-to-end-test`
  for the post-step-13 happy path. The order-by E2E test switched to
  the canonical name (would otherwise short-circuit at the lookup
  before exercising the order-by repair pass).
* `tools_test.clj` and `document_test.clj`: stub signatures and call
  sites updated.
* Removed 5 obsolete `rewrite-database-name-*` unit tests; replaced
  with a comment block pointing at the new test coverage.

86 tests / 529 assertions across all touched namespaces; kondo clean.

* agent-lib: migrate slackbot notebook-query tool to representations pipeline (repr-plan step 14)

The slackbot variant of `construct_notebook_query` now consumes the same
MBQL 5 representations YAML as the main notebook tool, going through
`execute-representations-query` instead of the legacy
`execute-program` + sexp-in-array flow. The slackbot-specific parts
(`:title` and `:display` options, `adhoc_viz` data parts, Slack
visualization link) are unchanged — only the query-construction engine
underneath moved.

Schema change (`slackbot-query-schema`):
* Added: `:query` (a plain `:string`; flat shape for MCP compat — YAML
  validation happens inside `execute-representations-query`).
* Dropped: `:source_entity`, `:referenced_entities`, `:program`. Per
  repr-plan step 13, the YAML is self-describing (`database: <name>`
  at the top level, full portable FK paths everywhere else), so the
  separate entity identifiers are redundant.

Follow-on cleanup in `construct.clj`:
* Deleted `construct-program-schema` — slackbot was its last consumer.
* Marked the legacy sexp-pipeline scaffolding
  (`source-entity->model-str`, `program-source->source-entity`,
  `source-metadata-for`, `surrounding-tables-for`,
  `available-measure-ids`, `build-evaluation-context`, and
  `execute-program` itself) with a section comment naming step 15 as
  the delete point. These functions are kept alive for now because the
  HTTP `/v2/construct-query` endpoint in `metabase.agent-api.api` still
  calls them; they die with the endpoint migration.
* `resolve-source-database-id` stays as-is — `execute-program` uses it,
  and kept in the "live" layer of the namespace rather than the sexp
  block because its signature is friendly enough to potentially live on
  as a public helper after step 15 (TBD).

Tests (`slackbot_query_test.clj`):
* Added `schema-shape-post-step-14-test`: asserts the new keys are
  present and the three legacy keys are gone. Acts as the schema-level
  contract guard.
* Added 4 stubbed-pipeline unit tests for the slackbot wrapper:
  - happy path (YAML pass-through, adhoc_viz data part carries title +
    display + link + query);
  - no-optional-fields path (title/display absent → not included in
    the part, but the link is still built);
  - `:agent-error?` passthrough (bare `:output` with the message,
    no structured-output or data-parts);
  - non-agent exception wrapping (generic `Failed to construct notebook
    query:` prefix so stack traces don't leak to the LLM).
* Added 2 end-to-end tests against the real sample DB:
  - happy path exercising the full repr pipeline + real
    `query->question-url`;
  - unknown-database error path, asserting the step-13
    loud-failure behaviour via the slackbot contract.

Prompt (`construct_notebook_query.md`) is shared between the two tools
and describes `:visualization` (notebook) rather than `:title`/`:display`
(slackbot). This asymmetry is pre-existing — the old slackbot schema
had `:program`, also not documented in the shared prompt — not
introduced by this commit. Left as a follow-up.

All 93 tests / 575 assertions across touched namespaces pass
(representations parser/repair/resolve + construct + slackbot +
charts/edit + document + metabot.tools). Kondo 0/0 on the three
modified files.

* agent-lib: multi-stage representations queries (repr-plan step 8)

Adds a new repair pass that lets LLMs write multi-stage queries the
way they naturally want to: a `[field, {}, "<column-name>"]` clause in
stage N referencing an output of stage N-1, with no `base-type` option
set. The lib-schema requires `base-type` on any string-named field
ref, so these queries used to blow up at `lib/query` time with a
`missing required key, got: nil` error.

## Changes

### `agent_lib/representations/repair.clj`
Adds **Pass 5 — `infer-cross-stage-field-types*`**:
* When `stages.length >= 2`, walk stages left-to-right. Before
  processing stage `i`, build a truncated query of stages[0..i-1],
  resolve it through the existing pipeline, and call
  `lib/returned-columns` on the result. Index those columns by name.
* Walk stage `i` (skipping descent into its `joins` subtree — join
  stages have their own resolution context). Every
  `[field, opts, "<name>"]` clause whose opts lack `base-type` gets
  its opts merged with the inferred `{base-type, effective-type}`.
* Best-effort and idempotent: if the mini-resolve fails, the pass
  logs at debug and lets the downstream resolver surface the real
  error; clauses whose name isn't produced by the previous stage are
  left alone so the lib-schema error points at the typo; clauses that
  already have `base-type` are never overwritten.

Also tightens the docstring on `resolve-implicit-joins*` to explain
why that pass stays stage-0-only (implicit FK joins require a
`source-table`, which only the first stage has).

### `metabot/tools/construct.clj`
Widens the `:agent-error?` catch in `execute-representations-query`:
the previous catch only wrapped parse/repair/validate/resolve, but
cross-stage field-type errors actually surface in
`result-columns-for-query`'s `lib/query` call. The new catch wraps
the entire pipeline from the MP forward, so any lib-schema violation
reaches the LLM with `:agent-error? true` set and no stack trace.

### Prompt (`construct_notebook_query.md`)
* Removes "Phase 1 MVP supports a single stage; multi-stage is coming"
  and the matching "not yet supported" bullet.
* Adds a "Multi-stage queries" section documenting the cross-stage
  ref shape (`[field, {}, <column-name>]` as a string), two concrete
  examples (post-aggregation filter `count > 10`, post-aggregation
  aggregation `avg(daily sum) by month`), and the rules (first-stage
  `source-table`, valid output column names, joins in later stages).

### Tests
* **`repair_test.clj`** — 8 new unit tests: happy path, existing
  `base-type` preserved, breakout-column inheritance, string-FK-path
  isolation, no-previous-stage no-op, unknown-column leave-alone,
  idempotency, repair+resolve smoke.
* **`construct_representations_test.clj`** — 4 new E2E tests: two-stage
  post-aggregation filter, explicit-base-type preservation, three-stage
  chain (filter + order-by both reference the aggregate), and
  unknown-column `:agent-error?` propagation.

All tests pass: 105 tests / 600 assertions across representations
parser/repair/resolve + construct + slackbot + charts/edit + document
+ metabot.tools + serialization resolver. Kondo 0/0 on the 4 modified
source/test files.

## What's NOT changed

* Resolver (`agent_lib/representations/resolve.clj`) — already handled
  multi-stage for free; `import-mbql` walks the structure recursively
  and cross-stage string refs don't match any of its vector-shape
  patterns, so they pass through untouched.
* Representations YAML schema — no new shape was needed; a
  cross-stage ref is just an ordinary `[field, opts, _]` clause with a
  string instead of an FK vector in position 2.
* Implicit-join pass — stays stage-0-only. Implicit FK joins are only
  meaningful from a stage that owns its `source-table`.

* agent-lib: support expressions in representations queries (repr-plan step 9)

- Added pass 1.5 `normalize-expressions-shape*` in repair: accepts the
  LLM-friendly map form `expressions: {Name: clause}` and normalises to the
  canonical MBQL 5 sequential form with `lib/expression-name` stamped from the
  map key. Canonical sequential input passes through unchanged; authored
  `lib/expression-name` in options is never overwritten. Idempotent.

- Resolver unchanged: `["expression" {} "<name>"]` refs are 3-element
  vectors with a string in the last slot and don't match any portable-FK
  pattern, so they pass through the resolver untouched.

- Prompt: new "Expressions (custom columns)" section with arithmetic
  (Subtotal = TOTAL + TAX) and string-concat (FullName) examples, plus a
  rules block on YAML quoting of arithmetic operators and the map-key
  shortcut. Removed expressions from the Phase 1 'not yet supported' list.

- Tests: 7 new unit tests in repair_test (map shape, sequential, sequential
  without name, existing-name preserved, absent, idempotency, multi-stage) +
  3 new end-to-end tests in construct_representations_test (map shape,
  canonical sequential, expression in breakout).

Totals: repair 34\u219241 / 59\u219270 assertions; construct 15\u219218 / 45\u219256.
Kondo 0/0 on touched files.

* agent-lib: resolve integer-indexed aggregation refs to UUID form (repr-plan step 10)

- Added pass 2.8 `resolve-aggregation-ref-indexes*` in repair: walks each
  stage, stamps `lib/uuid` on aggregation clauses that lack one, and rewrites
  every `[aggregation, opts, <int>]` ref to the canonical MBQL 5
  `[aggregation, {lib/uuid, base-type, effective-type, \u2026}, "<uuid>"]` shape.
  0-based integer index mirrors legacy MBQL 4 prior art (`[:aggregation 0]`,
  see `legacy_mbql/schema.cljc`); no new invented sigil.

- Type inference by aggregation head: count / distinct / cum-count /
  count-where \u2192 type/BigInteger; avg / median / stddev / var / share \u2192
  type/Float; sum / sum-where / cum-sum / min / max inherit from the inner
  field's annotated base-type (fallback type/Float); unknown heads \u2192 type/*.
  Authored base-type / effective-type / name / etc. on the ref's options are
  never overwritten.

- Out-of-range index raises `:agent-error?` with code
  `:aggregation-ref-out-of-range` and a message listing every available
  aggregation as `<head> at <index>`. Integer ref in a stage with no
  `aggregation:` raises `:aggregation-ref-no-aggregations`.

- Same-stage only. Cross-stage aggregation refs are out of scope here because
  pMBQL forbids that shape \u2014 the correct downstream form is a cross-stage
  field ref by column name, handled by the existing Pass 4.

- Idempotent: refs whose last slot is already a UUID-shaped string pass
  through untouched. Resolver: no change.

- Prompt: new "Aggregation references" section with the two LLM-facing
  forms (inline vs by-index) and an example. Removed "Aggregation references
  with UUIDs" from the Phase 1 'not yet supported' list.

- Tests: +9 unit tests (happy path, authored options preserved,
  out-of-range agent-error, UUID-string idempotent, no-aggs agent-error,
  no integer refs no-op, full idempotency, type inference table,
  multi-stage same-stage ref) + 2 end-to-end tests (execute-representations-query
  resolving to valid pMBQL; out-of-range surfacing clear error).

Totals: repair 41\u219250 / 70\u219298 assertions; construct 18\u219220 / 56\u219263 assertions.
Kondo 0/0 on touched files.

* resolve/mp: import Card FK by entity_id (repr-plan step 11.1)

The metadata-provider-backed serdes import resolver now handles
`import-fk` for `Card` / `:model/Card`, looking up the saved question /
model by its portable `entity_id` via `serdes/lookup-by-id`. Previously this
threw `:not-implemented-yet` and blocked the agent-lib representations
pipeline from handling `source-card` stages.

Why touch the app DB here (when the rest of the MP resolver is warehouse-only)?
Cards live in the application DB independently of whose warehouse the metadata
provider points at, and the lib metadata protocol's `metadatas` filter does
not support `entity_id` \u2014 a proper implementation would need a new metadata
spec. A one-row DB hit by entity_id is fine in practice.

Cross-database guard: if the resolved card's `database_id` does not match the
metadata provider's current DB, we throw a clear `:agent-error?` with code
`:cross-database-card` so the LLM can recover. Not-found likewise surfaces
as `:agent-error?` with code `:unknown-card`.

Other models (`Segment`, `Metric`, `NativeQuerySnippet`, etc.) continue to
throw `:not-implemented-yet`.

Tests:
- happy path (with-temp Card in current DB) for both 'Card and :model/Card
- nil input returns nil
- unknown entity_id surfaces :unknown-card agent-error
- card belonging to a different database surfaces :cross-database-card
- non-Card models still throw :not-implemented-yet

Totals: mp-test 11\u219215 / 36\u219249 assertions; regressions clean (repair 50/98,
construct 20/63).

* entity_details: expose portable_entity_id for saved questions and models (repr-plan step 11.2)

The agent uses entity_details to discover saved questions and models it can
reference in a representations query via `source-card:`. Previously the
response carried the numeric id but not the portable entity_id, so the LLM
had nothing to put into `source-card:` \u2014 it would guess `card__<id>` or
the numeric id, both of which fail serdes resolution.

This adds `:portable_entity_id` to the card-details payload (both `question`
and `model` variants), analogous to `:portable_fk` on table details.

Tests: +2 (question and model), both use `with-temp Card` and assert the
payload carries the exact entity_id.

* agent-lib: source-card end-to-end (repr-plan step 11.3)

With the MP resolver handling `import-fk` for Card (step 11.1) and
`entity_details` exposing `portable_entity_id` (step 11.2) in place, this
commit wires `source-card:` into the representations pipeline end-to-end.

Representations schema:
- added `source-card :string` to the stage map. Invalid types (e.g. integer)
  are rejected at validation with the standard `:agent-error?` relay.

Repair pipeline:
- new **Pass 5.5 `infer-source-card-field-types*`** stamps `base-type` /
  `effective-type` onto `[field, opts, "<col>"]` refs in a stage whose
  source is a `source-card:` card. Works by mini-resolving a bare
  `{source-card: <eid>}` stage, calling `lib/returned-columns`, and
  indexing the results by column name \u2014 parallel to Pass 5 for multi-stage
  queries. Best-effort no-op when mp is nil or lookup fails, so the real
  error surfaces at resolve/validate time.
- updated the top-level `repair` docstring to describe Pass 5.5 and the
  multi-pass `mp`-dependency contract.

Prompt (`construct_notebook_query.md`):
- new "Querying a saved question or model" section under "Examples for
  each MVP operation" with a filter-and-limit example, rules for column
  references by output name (same shape as cross-stage refs), mutual
  exclusivity with `source-table`, and the same-database constraint.
- "Multi-stage queries" updated to mention `source-card:` as an
  alternative first-stage source.
- removed "source-card" from the Phase-1-not-yet-supported list
  (which is now empty).

Tests (E2E):
- happy path: `source-card: <21-char NanoID>` resolves to numeric id 500;
  field reference `[field, {}, "TOTAL"]` gets `base-type :type/Float`
  stamped from the card's result-metadata.
- unknown entity_id (valid shape) surfaces `:agent-error?` with code
  `:unknown-card`.
- schema rejects non-string `source-card` value (integer) with an agent-
  error.

Totals: construct 20\u219223 / 63\u219271 assertions; regressions all green
(repair 50/98, mp 15/49, entity-details 17/71). Kondo 0/0 on touched files.

Dev-workflow note: Metabase entity_ids are 21-char NanoIDs
(`#"^[A-Za-z0-9_-]{21}$"`). The serdes import resolver's
`{:source-card (id :guard portable-id?)}` pattern only fires when the
id has that shape; malformed ids pass through untouched and blow up at
schema validation (`pos-int?`). Tests that exercise the `:unknown-card`
path must therefore use a valid-shape NanoID that simply doesn't exist,
not a short fake.

* agent-lib: harden source-card prompt + :unknown-card remediation message (repr-plan step 11 follow-up)

Real-world test revealed two cooperating problems:

1. The LLM confidently invents a `source-card:` entity id when it already
   knows the card's numeric id from a prior tool call, rather than calling
   `entity_details` first. This produces `:unknown-card` with a technically-
   correct but un-actionable message ("No saved question or model found
   with entity_id X").

2. When the tool rejects the invented id, the agent falls back to a native-
   SQL workaround like `SELECT * FROM {{#175-orders-count}} AS orders_count`,
   which defeats the point of the representations pipeline.

Fix:

- **Prompt** (`construct_notebook_query.md`): the "Querying a saved question
  or model" section now opens with a mandatory three-step workflow (call
  entity_details \u2192 copy portable_entity_id verbatim \u2192 reference fields by
  their `name`). Explicitly bans inventing, constructing, abbreviating, or
  pattern-matching entity_ids, and bans `SELECT ... {{#<id>-<slug>}}` as a
  fallback. Added a corresponding one-liner in "Rules and common mistakes"
  for repeated exposure.

- **Error message** for `:unknown-card`: now tells the LLM exactly how to
  recover \u2014 call `entity_details` with `entity-type: question` / `model`
  and the numeric id, then copy `portable_entity_id` verbatim. Matches the
  prompt guidance so the agent can self-correct without further hints.

No test impact: existing tests assert on ex-data (`:error :unknown-card`,
`:agent-error? true`), not on the user-facing message text. Totals unchanged:
mp 15/49, construct 23/71, repair 50/98. Kondo clean.

* metabot: expose portable_entity_id on card/model results to unblock source-card (repr-plan step 11 follow-up)

Even with a strict prompt and a helpful `:unknown-card` error message, the
LLM kept inventing entity_ids when asked to reuse a saved question as a
`source-card:` source. Root cause: the tool responses the LLM consults
to identify cards (`search`, `read_resource` for `metabase://question/<id>`
and `metabase://model/<id>`) never actually showed the card's entity_id.
So the agent had a numeric id and a name, but no way to derive the 21-char
NanoID \u2014 and defaulted to fabrication or native-SQL `{{#id-slug}}`
workarounds.

This commit plumbs `portable_entity_id` through to those response shapes
so the agent always has it one copy-paste away from where it needs it.

- **`search` tool** (`metabase.metabot.tools.search`): new
  `enrich-with-portable-entity-ids` post-processing step that, for
  `question` and `model` results, looks up the Card's `entity_id` in bulk
  (one `t2/select-pk->fn` keyed by id) and attaches it as
  `:portable_entity_id`. Tables/dashboards/metrics/transforms are left
  alone \u2014 `source-card:` only accepts cards. Bulk lookup keeps search
  overhead flat regardless of how many cards come back.

- **XML rendering** (`metabase.metabot.tools.shared.llm-representations`
  + `resources/metabot/prompts/llm_representations.selmer`): threaded
  `portable_entity_id` into
  - `search-result->xml` (rendered as a `portable_entity_id="..."`
    attribute on the result tag, emitted only when present so unrelated
    result types stay unchanged)
  - `question->xml` (attribute on `<metabase_question>`)
  - `model->xml` (attribute on `<metabase-model>`)

  `entity_details`' `card-details` already exposed `:portable_entity_id`
  in its structured-output (step 11.2) \u2014 it just wasn't reaching the XML
  the LLM actually reads.

- **Prompt** (`construct_notebook_query.md`): updated the mandatory
  workflow under "Querying a saved question or model" to reflect that
  both `search` and `read_resource` now carry `portable_entity_id`
  directly, so the common path is to reuse it from the already-loaded
  search response rather than calling `entity_details` again.

Tests:
- `enrich-with-portable-entity-ids-test` (search_test): verifies cards
  (question and model variants) pick up `:portable_entity_id`, and that
  dashboards do not.
- New `testing` blocks in `question->xml-test` and `model->xml-test`:
  attribute is emitted when present, omitted when absent.
- `search-result->xml-test` extended with four blocks covering question,
  model, card-without-eid, and table (which must never carry the
  attribute).

Totals: search-test 7\u21928 / 35\u219245, llm-representations-test 19\u219220 /
220\u2192228. All regressions green (mp 15/49, entity-details 17/71, repair
50/98, construct 23/71). Kondo clean.

Dev caveat: Selmer templates are loaded once via
`metabase.metabot.agent.prompts/template-cache`. When iterating on
`llm_representations.selmer` in a running REPL, call
`(metabase.metabot.agent.prompts/clear-cache!)` to pick up edits \u2014
otherwise the old template shape lingers and the new attributes appear to
be silently dropped.

* repair: unwrap nested [field opts [field inner-opts target]] clauses

LLMs occasionally write a field clause wrapping another field clause, e.g.

  ["field" {"temporal-unit" "month"}
   ["field" {} ["Sample" "PUBLIC" "ACCOUNTS" "CANCELED_AT"]]]

reasoning as "apply granularity to this existing field." The intended MBQL
is a single field clause with the options merged. Without repair, the nested
shape caused FK resolution to fail loudly ("Portable field FK ... must have
at least 4 elements"), since the inner [field ...] clause was treated as an
FK path.

Add Pass 1.7 (normalize-shape stage) that collapses nests of any depth into
one flat [field <merged-opts> <target>] clause. Outer options win on key
conflicts (outer is what the LLM thought it was adding). Works for FK
targets and string-column (cross-stage) targets alike.

* metabot: eliminate metric/table/field hallucination surfaces (repr-plan)

Sweeping set of fixes driven by benchmark telemetry — LLM kept inventing
schemas, tables and fields because the payloads it saw did not carry the
identity data it needed. Moves the contract from 'LLM infers from
vibes' to 'LLM copies the portable path verbatim' and adds targeted
agent-facing errors for the remaining slips.

Benchmark result on NLQ Metrics - Categorical Filters: 96% score /
78% passing (up from systemic construct_notebook_query failures);
remaining 2 failures are judge-semantic, not tool-level.

Contract shape (M2, M3, V1, Important A)
- Remove top-level `database:` from the LLM-facing query contract; the
  repair layer stamps it internally from source-table[0] / source-card,
  so the LLM only has to anchor one FK path and cannot contradict itself.
- `<metric>` tags gain `base_table_fully_qualified_name`, wired through
  `enrich-with-metric-base-tables` on search results and directly out of
  `metric-details` for read_resource. LLM now knows which table to put
  in `source-table:` when consuming a metric.
- `<field>` tags gain `fk_target_fully_qualified_name` whenever the
  column carries an FK. Closes the 'write customer.email on the facts
  table' class of hallucinations.
- `<table>`, `<metric>`, `<search-result>` tags carry `database_name`
  consistently so cross-DB portable FKs never have to be guessed.
- `portable_entity_id` surfaced on card/model/metric details.

metric-details dual-shape fix
- `metric-details` is called with two shapes of `card`: a t2 row
  (snake_case) from `get-card`, and a `lib.metadata/card` kebab-case
  SnakeHatingMap forwarded by `convert-metric`. Earlier code read
  `(:dataset-query card) -> :query -> :source-table` which was
  broken twice over (wrong case, legacy MBQL-3 shape); and reading
  `:table_id` on the kebab map threw deprecation errors that surfaced
  to the LLM as empty `<resource> **Error:**` blocks. Now accepts both
  `:table-id`/`:table_id` and `:entity-id`/`:entity_id` and pulls the
  base table off `report_card.table_id` directly.

Targeted agent-facing errors (F, F.2, FK-hints)
- `find-table` in resolve/mp now distinguishes `:unknown-schema` (with
  available schemas), `:unknown-table` wrong-schema variant (lists the
  schemas where the name exists), and `:unknown-table` fuzzy variant
  using token-level Jaccard with a same-schema boost (handles
  `fct_cards -> int_brex_card_dim` via the shared `card` token).
- `find-field` now surfaces FK-reachable candidates when the column
  name exists on an outbound FK target: 'No column "name" on table
  X. It exists on FK-linked tables: [Analytics, schema, target, name].'
  The LLM copies the path and the implicit-join path takes over.
- All agent-facing errors tagged `:agent-error? true` so the tool
  wrapper logs them at debug instead of error-with-stacktrace.

uri-in-source-table fail-fast (M3)
- Detect `source-table: "metabase://..."` / metric URIs early and
  return a targeted hint pointing at read_resource + the right
  base-table path, instead of a generic validation failure.

Prompt updates (construct_notebook_query.md)
- 'Using a metric' rewritten: explicit 'do not invent schema/table
  names; copy base_table_fully_qualified_name verbatim'.
- 'Implicit joins' tip rewritten around the new
  `fk_target_fully_qualified_name` attribute with a concrete
  customerio example.

Log noise (construct.clj)
- Split the tool-level catch by `:agent-error?`: expected agent
  feedback logs at debug with no stacktrace; real errors keep
  `log/error e`.

Tests
- New / updated: metric base-table enrichment, metric XML FQN, field
  XML FK attribute, entity-details base-table regression,
  construct representation error hints, repair unwrap coverage,
  schema/table/field resolver error tiers + FK-candidate hints.
  142 tests / 623 assertions green.

Files:
- resources/metabot/prompts/{llm_representations.selmer,tools/construct_notebook_query.md}
- src/metabase/agent_lib/representations{,/repair}.clj
- src/metabase/metabot/tools/{construct,entity_details,search,util}.clj
- src/metabase/metabot/tools/shared/llm_representations.clj
- src/metabase/models/serialization/resolve/mp.clj
- matching tests under test/

* Slim down LLM entity representations prompt

- Drop redundant attributes from rendered XML (database_name where
  database_id/engine already present, portable_entity_id, FK target
  FQN, base_table FQN in metric/search).
- Remove human-prose definition descriptions for measures/segments;
  keep the formal <definition> only.
- Remove standalone <collection> rendering and the question fields
  block to reduce prompt noise.
- Rename segment_id attribute to id for consistency with other tags.

* Pass resolver explicitly to import-mbql

Use the 2-arity `import-mbql` form instead of `binding` the dynamic
`*import-resolver*`. The dynamic var is intended for the serdes-import
middleware stack, where the resolver is threaded through many internal
calls. Here we have a single, local call site; passing the resolver
as an argument is simpler, easier to test, and avoids relying on
thread-local state.

* Split asset and metadata lookups in serdes mp resolver

The metadata-provider-backed resolver had two implicit responsibilities:
warehouse-schema lookups (databases, tables, fields) backed by the
`MetadataProvider`, and Metabase-content lookups (cards by entity_id)
that quietly went through `serdes/lookup-by-id` and therefore required
the application database.

Make that split explicit by introducing a small `ContentStore`
protocol with a default app-DB-backed implementation. `import-resolver`
now takes an optional `content-store` as its second arg; the 1-arity
form preserves today's behavior. This unlocks reusing the resolver in
contexts without an app DB (the serdes checker, in-memory fixtures,
snapshots) without further refactoring.

Add tests exercising a custom in-memory content store for the happy
path, the unknown-card branch, and the cross-database-card branch.

* Honor extended source-field variants in implicit-join repair

The implicit-join repair pass now leaves any field clause whose options
already carry one of the disambiguating keys alone:

  * source-field             (already supported)
  * source-field-name        (new) - LLM-authored implicit join from a
                                     previous stage's output column
  * source-field-join-alias  (new) - LLM-authored implicit join where
                                     the FK-bearing column lives on an
                                     explicitly-joined table
  * join-alias               (already supported)

We do not auto-fill the two new variants - filling them in correctly
would require resolving the previous stage's returned-columns or an
explicit-join's FK shape, which is out of scope here. The resolver and
`lib.normalize` already pass these strings through unchanged, so the
remaining work is purely "don't second-guess what the LLM authored".

Tests: two new repair-pass leave-alone tests, plus an end-to-end
resolve-test that asserts both keys propagate through to pMBQL with the
expected types (string for the names, numeric id for any
co-existing portable `source-field`).

Prompt: a single rules entry under "Implicit joins" describing each
variant in one line; no extra examples.

* Migrate /v2/construct-query and /v2/query to representations YAML

Both endpoints now accept a flat `{\"query\": \"<yaml-string>\"}` body in the
canonical representations YAML format, the same format already used by the
`construct_notebook_query` / `document_construct_model_chart` /
`slackbot_query` agent tools (after step 14). The old sexp-style
`{source, operations}` envelope is gone.

Why
- The YAML is fully self-describing: the database is derived from the first
  stage's `source-table:` / `source-card:`, all field references are
  portable FKs, and there is no auxiliary `source_entity` /
  `referenced_entities` envelope (consistent with repr-plan step 13).
- Single source of truth for the query format reference: the prompt
  `resources/metabot/prompts/tools/construct_notebook_query.md` documents
  the YAML for the LLM and now also for the HTTP endpoint.
- MCP-friendly: the input schema stays a flat `:string` at the root, so
  inputSchema flattening for the MCP `construct_query` / `query` tools
  Just Works without any `:and`/`:fn` wrappers.

Endpoint changes (`agent_api/api.clj`)
- `::program-request` / `agent-lib/program-schema` re-export gone; the
  `metabase.agent-lib.core` alias is unused and removed from the require.
- New `::construct-query-request` = `{:query NonBlankString}`.
- `::query-request` multi-dispatch: `{continuation_token | query}` (tag
  rename `:program` -> `:fresh`).
- New `evaluate-yaml-to-live-query` / `evaluate-yaml-for-execution` thin
  wrappers around `metabot-construct/execute-representations-query` (the
  pipeline already raises `:agent-error?` ex-data on input failures, which
  the exception middleware surfaces as 400s).

Tests (`agent_api/api_test.clj` + `mcp/api_test.clj`)
- New `orders-yaml` / `orders-field-ref` helpers build a representations
  YAML payload against the live application database, keeping per-test
  bodies compact.
- All previously-program-shaped tests rewritten as YAML-shaped tests with
  the same coverage: simple table query, limit handling, count
  aggregation, filter clause, metric-via-aggregation reference,
  single-page combined query, continuation-token pagination, per-page cap.
- 404-for-unknown-table replaced by 400-with-`:unknown-table`-ex-data
  (the new agent-error path; the old 404 was a side effect of the pre-repr
  source-entity fetch).
- New `construct-query-rejects-empty-query-test` covers schema-level
  rejection of missing/blank `:query`.

Documentation (`reference.md`)
- The big sexp-program section (\~250 lines covering operators, references,
  filter operators, aggregation operators, temporal helpers, four worked
  examples) is replaced by a thin description of the wire protocol plus a
  pointer to the prompt for the format reference. The reference no longer
  duplicates information from the prompt; the prompt is now the single
  source of truth.
- Error-response table reflects the actual error keys the repr pipeline
  raises (`unknown-database`, `unknown-table`, `unknown-field`,
  `unknown-card`, `missing-source-in-first-stage`, `ambiguous-fk` /
  `no-fk-path`, `uri-in-source-table`).

Tests run (clojure-eval): 79 tests / 376 assertions across
`agent_api.api-test`, `mcp.api-test`, and the construct-representations
tool-test suite. All green; clj-kondo 0/0 on the four touched files.

The legacy sexp-pipeline scaffolding in `metabase.metabot.tools.construct`
(`execute-program`, `program-source->source-entity`,
`build-evaluation-context`, `source-entity->model-str`,
`resolve-source-database-id`) is now unused everywhere; deletion is its own
follow-up commit.

* Delete now-unused legacy sexp-pipeline scaffolding from metabot.tools.construct

After step 15 migrated the HTTP /v2/construct-query and /v2/query
endpoints to the representations YAML pipeline, the sexp-program bridge
inside `metabot.tools.construct` has no callers anywhere. Dropping it
removes ~120 lines of code and the last in-tree path that resolves a
database from a single `source_entity.id` rather than from the YAML.

Removed:
  * `resolve-source-database-id` - source_entity -> db-id lookup
  * `source-entity->model-str`   - metabot-string -> agent-lib-string mapper
  * `program-source->source-entity` - HTTP source -> sexp source-entity
  * `source-metadata-for`        - lib metadata for source entity
  * `surrounding-tables-for`     - visible-columns -> surrounding tables
  * `available-measure-ids`      - measures advertised by a source entity
  * `build-evaluation-context`   - assembles the EvaluationContext
  * `execute-program`            - sexp-in-array entry point

Required namespace aliases that became unused are dropped too:
  * `metabase.agent-lib.core`     (only `agent-lib/evaluate-program`)
  * `metabase.lib.metadata`       (only `source-metadata-for`)

Verification: a repo-wide grep for each removed symbol returns no hits
outside `construct.clj`. Tests run via clojure-eval (177 tests / 692
assertions across agent_api.api-test, mcp.api-test,
construct-representations-test, slackbot-query-test, document-test, the
three agent-lib repr test namespaces, and serialization resolve.mp-test)
all green; clj-kondo 0/0 on the touched file.

Note: the underlying agent-lib sexp pipeline (`metabase.agent-lib.core`,
`agent-lib.schema`, `agent-lib.eval/*`, `agent-lib.mbql-integration/*`,
`agent-lib.repair.*`, etc.) is still present and referenced by its own
internal tests \u2014 that deletion is repr-plan step 16.

* Delete agent-lib sexp pipeline (repr-plan step 16)

After steps 14 and 15 migrated every consumer (slackbot, document tools,
construct_notebook_query, the /v2/construct-query and /v2/query HTTP
endpoints, and their MCP equivalents) to the representations YAML pipeline,
the entire structured-program ("sexp-in-array") side of agent-lib has no
remaining callers anywhere in the tree. This commit deletes it.

Verified absence of external callers
- `grep -rn 'metabase.agent-lib' src/ test/ --include='*.clj*'` outside of
  `agent_lib/` itself returns only the three repr requires in
  `metabot/tools/construct.clj` and one prose mention in an api-test
  docstring.
- Inside agent-lib, no non-representations namespace is required by
  `representations*` (the parser, repair, and resolver depend only on
  `metabase.lib.*`, `metabase.models.serialization.resolve*`, and stdlib
  helpers).

Removed (78 files, 7581 lines)
- src/metabase/agent_lib/{core,syntax,refs,join_spec}.clj
- src/metabase/agent_lib/{schema,schema/top_level}.clj
- src/metabase/agent_lib/eval{,/{args,invoke,source,walker}}.clj
- src/metabase/agent_lib/mbql_integration{,/{common,field_resolution,fields,joins,orderables}}.clj
- src/metabase/agent_lib/repair{,/{normalize{,/{forms,top_level}},context{,/passes},stages}}.clj
- src/metabase/agent_lib/runtime{,/{bindings,fields,lookup,transforms}}.clj
- src/metabase/agent_lib/validate{,/{context,cross_checks,operators,program,semantic,walker}}.clj
- src/metabase/agent_lib/capabilities{,/{derived,synthetic,catalog{,/{aggregation,expressions,filtering,joins,ordering,sources,top_level}}}}.clj
- src/metabase/agent_lib/common/{coercions,context,errors,literals}.clj
- All matching tests, the agent-lib test_util, plus fuzz_test and guards_test.

Kept (the entire repr surface)
- src/metabase/agent_lib/representations.clj             (parse + structural validation)
- src/metabase/agent_lib/representations/repair.clj      (idempotent shape passes + implicit joins)
- src/metabase/agent_lib/representations/resolve.clj     (portable -> numeric ID -> normalized pMBQL)
- The three matching test namespaces.

Module-graph adjustments (.clj-kondo/config/modules/config.edn)
- `agent-lib :api`: drop `metabase.agent-lib.core` (no longer exists). The
  repr namespaces remain the public API.
- `agent-api :uses`: drop `agent-lib`. After step 15 `agent_api.api` no
  longer requires anything from `metabase.agent-lib`; it goes through
  `metabot.tools.construct` instead, which itself is the sole user of
  `metabase.agent-lib.representations*`.
- `metabot :uses` still includes `agent-lib` (for the repr namespaces) -
  unchanged.
- `lib :friends` still includes `agent-lib` because the resolver still
  reaches into `metabase.lib.normalize` / `:metabase.lib.schema/query` -
  unchanged.

Verification (clojure-eval against fresh REPL on :49319)
- 181 tests / 719 assertions across:
    metabase.agent-lib.representations-test
    metabase.agent-lib.representations.repair-test
    metabase.agent-lib.representations.resolve-test
    metabase.agent-api.api-test
    metabase.mcp.api-test
    metabase.metabot.tools.construct-representations-test
    metabase.metabot.tools.slackbot-query-test
    metabase.metabot.tools.document-test
    metabase.models.serialization.resolve.mp-test
  All green.
- clj-kondo on src/metabase/agent_lib, src/metabase/agent_api,
  src/metabase/metabot/tools/construct.clj, src/metabase/mcp,
  test/metabase/agent_lib, test/metabase/agent_api, test/metabase/mcp:
  0 errors, 0 warnings.

The agent-lib module is now lean and representations-native, as called
for in repr-plan.md "Phase 3 - cleanup and polish" step 16. Steps 17-20
(rename llm-representations -> llm-shape, normalize `:query-content` to
post-repair YAML, prompt sweep, optional /v1 endpoint cleanup) remain.

* Rename llm-representations -> llm-shape (output-side XML formatters)

Mechanical rename per repr-plan step 17.

Why
----
There are two distinct "representations" in the metabot:

  - **Input-side**: the canonical MBQL 5 YAML format that the LLM authors
    when calling `construct_notebook_query` / `/v2/construct-query`. Owned
    by `metabase.agent-lib.representations.*` and the `construct_notebook_query.md`
    prompt.
  - **Output-side**: the XML shape that we render entity payloads (tables,
    fields, metrics, search results, query metadata, ...) into when handing
    them back to the LLM as tool results / context. Owned by
    `metabase.metabot.tools.shared.llm-representations` and the
    `llm_representations.selmer` template.

Calling both "representations" was already confusing pre-PR; after this
PR's heavy work on the input-side format the symmetry became actively
misleading in code review. This commit renames the output-side surface to
`llm-shape` so the two never get conflated again.

What changed
------------
File renames (git-mv-detected):

  src/metabase/metabot/tools/shared/llm_representations.clj
    -> src/metabase/metabot/tools/shared/llm_shape.clj

  test/metabase/metabot/tools/shared/llm_representations_test.clj
    -> test/metabase/metabot/tools/shared/llm_shape_test.clj

  resources/metabot/prompts/llm_representations.selmer
    -> resources/metabot/prompts/llm_shape.selmer

Symbol updates (no behaviour change):

  - ns:    `metabase.metabot.tools.shared.llm-representations`
        -> `metabase.metabot.tools.shared.llm-shape`
  - alias: `llm-rep` (kept) - the human-readable mental model is still
        "render to LLM-facing shape", we just decoupled the *name of the
        shape* from the input-side representations name.
  - cache key + loader fn names: `llm-representations` -> `llm-shape`
        in `metabase.metabot.agent.prompts`.
  - private `llm-template-name` constant points at the renamed selmer file.
  - one stale prose comment in `agent/user_context.clj`.

Updated alias requires across consumers (no per-file logic change):

  - src/metabase/metabot/tools/{construct,resources,sql,metadata,charts,search}.clj
  - src/metabase/metabot/agent/{prompts,user_context}.clj
  - the lone test that still imported the old alias
    (test/metabase/metabot/agent/user_context_test.clj).

Verification
------------
- clj-kondo on src/metabase/metabot, src/metabase/agent_api, src/metabase/mcp
  and matching test trees: 0 errors / 0 warnings.
- Targeted tests run via clj-nrepl-eval against a freshly-loaded REPL:
  168 / 738 across construct-representations-test, slackbot-query-test,
  document-test, agent.user-context-test, agent_api.api-test, mcp.api-test,
  agent_lib.representations.{repair,resolve}-test - all green.
- `llm-shape-test` itself has 20 pre-existing failures unrelated to this
  rename: I confirmed this by stashing the rename and running the old
  `llm-representations-test` against the same HEAD - same 20 fails.
  Bisecting against `master` shows the regression dates back to commits
  `d47d35b7bec` ("Slim down LLM entity representations prompt") and / or
  the test-restoration commit `87fe407f5f`. The expectations in those
  restored tests assume an older (pre-slim) selmer template. They will be
  refreshed in repr-plan step 19 ("Final prompt cleanup") rather than as
  part of this rename, to keep this commit a pure mechanical rename.

* Emit post-repair representations YAML as :query-content (repr-plan step 18)

What changed
============
`structured->query-data` (in `metabase.metabot.tools.construct`) used to
JSON-encode the *legacy MBQL* form of a constructed query and put that JSON
in `:query-content`. That was a hold-over from the sexp pipeline: the LLM
authored the query in one shape (sexp or YAML) and saw it back in a third
(legacy MBQL JSON), with no overlap to the field paths / aggregation refs
it had just typed. On the next turn it had to re-derive everything.

This commit makes `:query-content` the **post-repair canonical
representations YAML** instead - i.e. the same string-keyed portable shape
the LLM submitted, but with all of the repair-pass normalizations applied:

  * empty `{}` options stamped on every clause that was missing one;
  * `lib/type` markers added to the query and to every stage;
  * `source-field` auto-wired on implicit-join field clauses;
  * `@agg-N` placeholder references resolved to UUID form;
  * inline order-by aggregations rewritten to aggregation-refs;
  * top-level `database:` rewritten to the metadata provider's actual name.

The LLM now sees exactly the normalized form of what it wrote, byte-for-byte
ready to be referenced on the next turn.

Implementation
==============
1. `execute-representations-query` is restructured so the repaired-and-
   validated parsed form is held in a local before resolution. The previous
   `->>` chain (parse / repair / validate / resolve) folded the repaired
   value back into the resolver immediately and discarded it; we now keep it
   so we can serialize it.

2. The repaired value is YAML-encoded with
   `{:flow-style :block}` (multi-line, human-readable - matches what the
   LLM authors and what `construct_notebook_query.md` shows in examples).

3. The serialized string is added to `:structured-output` as `:query-yaml`,
   alongside the existing `:query` (resolved pMBQL) and `:query-id` keys.

4. `structured->query-data` consumes `:query-yaml` and puts it in
   `:query-content`. The legacy-MBQL JSON path is gone (no behavioural
   reason to keep two encodings; the wire shape stays a single XML attribute).

5. `metabase.util.json` is no longer used here; replaced with
   `metabase.util.yaml` in the require block.

Test (`construct-representations-test/happy-path-test`)
========================================================
The existing happy-path test grew an extra `testing` block that:

  * asserts `:query-yaml` is a string in the structured output;
  * round-trips it through `yaml/parse-string ... :keywords false` (matching
    `repr/parse-yaml`'s convention) and verifies the parsed shape is the
    portable string-keyed form (`"lib/type" -> "mbql/query"`,
    portable FK `["Sample", "PUBLIC", "ORDERS"]` for the source-table) -
    i.e. *not* the resolved numeric-id pMBQL;
  * verifies idempotency: feeding the emitted YAML back to
    `execute-representations-query` yields a structured output whose
    `:query-yaml` is byte-equal to the first one. This is the property
    that lets the LLM use the previous turn's response unchanged.

Verification
============
- 24 tests / 94 assertions in `construct-representations-test` - all green.
- 168 tests / 742 assertions across the wider repr-touching suite
  (slackbot-query-test, document-test, agent.user-context-test,
  agent_api.api-test, mcp.api-test, agent_lib.representations.{repair,resolve}-test)
  - all green; no test relied on the old legacy-MBQL JSON shape.
- clj-kondo on the two touched files: 0 errors / 0 warnings.

Steps 19 (final prompt cleanup) and 20 (optional /v1 endpoint removal) are
the only remaining Phase-3 items.

* llm-rep to llm-shape

* Restore prompt-required attributes on llm-shape XML (repr-plan step 19)

Background
==========
Commit `d47d35b7bec` ("Slim down LLM entity representations prompt") trimmed
seven attributes off the rendered `llm_representations.selmer` template at
once, on the rationale that they were redundant given attributes already
present (e.g. `database_id` next to `database_name`, `fully_qualified_name`
next to `base_table_fully_qualified_name`).

Three of those attributes are anything but redundant: the
`construct_notebook_query.md` prompt actively instructs the LLM to read them
when authoring queries:

  * `fk_target_fully_qualified_name` on `<field>` is the *only* anti-
    hallucination signal the LLM has to know that a column lives on a
    related table - the prompt has a worked example showing
    `<field id="892" name="customer_id" fk_target_fully_qualified_name="..."/>`
    and tells the model not to guess that an unknown column is on the
    current table without checking this attribute first;

  * `portable_entity_id` on `<metric>`, `<metabase-model>`,
    `<metabase_question>` and `<{search_tag}>` is the 21-char NanoID the
    prompt requires copied verbatim into `source-card:` (or as the third
    slot of `[metric, {}, ...]`) - without it on the tag, the LLM has to
    do a follow-up `entity_details` call just to read the same value;

  * `base_table_fully_qualified_name` + `database_name` on `<metric>` and
    `<{search_tag}>` is the *only* place the prompt's metric workflow tells
    the model to read the metric's source table from. The corresponding
    "Using a metric" section in `construct_notebook_query.md` instructs:
    "combine [it] with `database_name` to form the portable FK
    [<database_name>, <schema>, <table>]".

The remaining `d47d` trims (`<collection>` standalone, the `Definition: …`
human-prose prefix on measure/segment, the question `<fields>` block, the
`segment_id` -> `id` rename) are *not* prompt-referenced; they stay slim.

The `llm-shape-test` had 20 pre-existing failures because it was restored
in commit `87fe407f5f` ("Restore mistakenly deleted tests") with assertions
asserting the *pre-d47d* template output, against the *post-d47d* template -
i.e. the wrong combination of versions. Step 16-18 work elsewhere (sexp
deletion, rename, repr YAML in `:query-content`) didn't touch this file but
shipped on top of the inconsistent state.

This commit
===========
`resources/metabot/prompts/llm_shape.selmer`: selectively un-trim the four
attributes the prompt actively references:

  - `<field … fk_target_fully_qualified_name="…">` (when present)
  - `<related-table … database_name="…">` (when present; mirrors the search/
    table tags)
  - `<metric … database_name="…" base_table_fully_qualified_name="…"
                portable_entity_id="…">` (each guarded by `{% if %}`)
  - `<table … database_name="…">`
  - `<metabase-model … database_name="…" portable_entity_id="…">`
  - `<metabase_question … portable_entity_id="…">`
  - `<{{search_tag_name}} … database_name="…"
        base_table_fully_qualified_name="…" portable_entity_id="…">`

Plus restore the `is_collection` block - not because it's prompt-mandatory,
but because `metric->xml`, `question->xml` and `dashboard->xml` already feed
the result of `(collection->xml collection)` through the
`{metric,question,dashboard}_collection_xml` slot. With `is_collection` gone
`collection->xml` returned "" and Selmer's `{% if %}` for empty string is
falsy, so the "The metric/question/dashboard is stored in the following
collection: …" block disappeared from the rendered XML even though the
collection data was passed to the template. The block is now visible again
(the dashboard, metric, and question regression tests check for it).

`test/metabase/metabot/tools/shared/llm_shape_test.clj`: align the *six*
assertions that asserted `segment_id="..."` / `<segment segment_id="..."`
or the now-removed `Definition: <description>` prose. The first set is the
`d47d` rename to `id="..."` (consistent with every other tag); the second
just changes from `is/includes` to `is/not-includes` since the prose line
isn't coming back - the structured `<definition>…</definition>` block
already inside the segment / measure carries the same information for the
LLM, more compactly.

Verification (clj-nrepl-eval)
============================
- 20 / 230 in `metabase.metabot.tools.shared.llm-shape-test`: all green.
- 215 / 1096 across the full repr-touching suite (`construct-representations-test`,
  `slackbot-query-test`, `document-test`, `agent.user-context-test`,
  `agent_api.api-test`, `mcp.api-test`, `agent_lib.representations.{repair,
  resolve}-test`, `tools.entity-details-test`, `tools.search-test`,
  `llm-shape-test`): all green.
- clj-kondo on the touched test file: 0/0 (the .selmer file is intentionally
  not Clojure and clj-kondo's parse failures on it are not our problem).

The wider step-19 sweep (`construct_notebook_query.md`, system .selmer
prompts, `agent_api/reference.md`, CLAUDE.md / internal docs) is otherwise
clean: a repo-wide grep for `agent-lib/program`, `source_entity`,
`referenced_entities`, `construct_program`, sexp-tuple `source`/`operations`
shapes, etc., turns up no remaining stale references.

* Repair: rewrite temporal-bucket-extraction aliases to canonical lib names (repr-deletion-followups 1.2)

LLMs naturally pick up English-flavoured names like `dayofweek`,
`day-of-week`, `hour-of-day`, `month-of-year`, `quarter-of-year` for the
temporal-extraction helpers. lib's canonical names (in
`metabase.lib.schema.expression.temporal`) are `get-day-of-week`, `get-hour`,
`get-month`, `get-quarter`. Without this pass the alias forms fail validation
with a generic schema error. Carried over from the old sexp pipeline's
`metabase.agent-lib.repair.normalize.forms/temporal-bucket-extraction-aliases`
(see `repr-deletion-followups.md` § 1.2).

Implementation
==============
`representations/repair.clj`:

- New `temporal-bucket-extraction-aliases` map (5 entries, identical to the
  sexp version).
- New `rewrite-temporal-bucket-aliases*` postwalk that rewrites a clause head
  iff the head is a known alias AND slot 1 is the options map (i.e. it really
  is a clause, not an FK path or a string column inside one).
- Added to the `normalize-shape*` pipeline right after `unwrap-nested-field-
  clauses*` and before `normalize-expressions-shape*` / `ensure-lib-types*`.
  Order: Pass 1 (`ensure-clause-options*`) runs first, so a bare `["day-of-
  week", x]` becomes `["day-of-week", {}, x]` before the alias rewrite sees
  it.

Idempotency
===========
The rewrite only fires when the head is an alias; canonical heads pass through
unchanged. Re-running `repair` on an already-repaired query is a no-op for
this pass.

Tests
=====
`representations/repair_test.clj`: new `rewrite-temporal-bucket-aliases-test`
covers (1) all five aliases mapping to canonical, (2) nested occurrence
inside another clause, (3) canonical heads left alone (idempotency), (4)
missing-options input (Pass 1 + Pass 1.8 chained), (5) FK-path-internal
string "day-of-week" not mistaken for a clause head.

58 / 123 in `representations.repair-test`: all green.

* Repair: rewrite order-by direction aliases to canonical asc/desc (repr-deletion-followups 1.3)

The lib schema accepts only `[asc, {}, <ref>]` / `[desc, {}, <ref>]` as
order-by direction clauses. LLMs sometimes write `[ascending, {}, …]` /
`[descending, {}, …]` (the English long form) and uppercase variants like
`[ASC, {}, …]` / `[Descending, {}, …]` (matching how they would render the
direction in prose). Without this pass any non-canonical spelling fails
schema validation. See `repr-deletion-followups.md` § 1.3.

The sexp pipeline's equivalent (`metabase.agent-lib.common.coercions/
normalize-direction`) only handled case-insensitive `asc`/`desc`. We extend
to `ascending` / `descending` here because the doc explicitly flagged them
as patterns LLMs naturally produce.

Implementation
==============
`representations/repair.clj`:

- New `direction-aliases` map (4 lowercase keys, all mapping to `asc` or
  `desc`).
- New `rewrite-direction-aliases*` postwalk gated by `direction-alias-clause?`,
  which requires a 3-element vector (head + options-map + 1 ref arg) - this
  is the order-by clause shape. The clause-shape gate is what keeps a column
  literally named "asc" inside an FK path safe (it's a 4-element string
  vector, not a clause).
- The head-match itself (`direction-clause-head?`) is case-insensitive but
  excludes the canonical lowercase `asc`/`desc` so that already-canonical
  clauses pass through walk-without-modification - cheap idempotency.
- Added to `normalize-shape*` right after `rewrite-temporal-bucket-aliases*`.

Tests
=====
`representations/repair_test.clj/rewrite-direction-aliases-test`:

- both long-form spellings → canonical (asc, desc);
- four uppercase / mixed-case variants per direction → canonical lowercase;
- already-canonical clauses pass through unchanged (idempotency);
- aliases inside an actual order-by stage are rewritten in place;
- a column literally named "asc" inside an FK path is *not* touched;
- a scalar string "ASC" outside a clause is *not* touched.

59 / 139 in `representations.repair-test`: all green.

* Repair: swap out-of-order between bounds (repr-deletion-followups 1.5)

The lib schema for `:between` does not enforce `min <= max` - there is an
explicit TODO comment in `metabase.lib.schema.filter` flagging this. LLMs
occasionally swap the two bounds, especially for date ranges (e.g.
`between(date, "2024-12-31", "2024-01-01")` for "December → January"). The
query then runs but returns no rows, which is silently misleading - exactly
the failure mode the doc flags as worth catching. See
`repr-deletion-followups.md` § 1.5.

Implementation
==============
`representations/repair.clj`: new `swap-between-bounds*` postwalk that, for
every 5-element `["between" <opts> <expr> <lo> <hi>]` clause, extracts the
two bounds via `between-bound-comparable` and swaps when `lo > hi` and both
are directly comparable scalars of the same kind. Bounds we treat as
literals:

  - numbers (Long, Double, Ratio, …);
  - strings (compared lex - ISO-8601 lex order is chronological);
  - `[absolute-datetime, {}, <iso-str>, <unit>]` clauses (compared by inner
    ISO string).

Anything else (a `field` ref, an expression clause, a `relative-datetime`
clause, a mixed pair like number-vs-string) leaves the clause alone - we
can't compare without execution.

The doc's other 1.5 sub-case ("wrap a single scalar bound as a one-element
vector when the two-arg shape was expected") is intentionally not ported -
`construct_notebook_query.md` is explicit about `between`'s three-arg shape
and the failure mode is unlikely.

Idempotency
===========
After the swap, `lo <= hi` so subsequent passes don't fire. Equal bounds
left alone (already in order). Property test `idempotency-property-test`
continues to pass.

Tests
=====
`representations/repair_test.clj`: 5 new deftests covering numeric, ISO-
string, absolute-datetime, non-literal-bound (untouched), and mixed-kind
(untouched) cases plus an explicit idempotency check on the numeric case.

64 / 151 in `representations.repair-test`: all green.

* Repair: wrap bare ISO-date strings as absolute-datetime in between (repr-deletion-followups 1.6)

LLMs frequently write `between(date_field, "2024-01-01", "2024-12-31")` -
bare strings as the bounds. lib's `:between` schema demands a temporal
expression on each side once any side is temporal; bare strings won't
satisfy `:type/Date` and the query fails validation. Carried over from the
sexp pipeline's `wrap-iso-date-as-absolute-datetime` (see
`repr-deletion-followups.md` § 1.6 - flagged as a high-frequency LLM
pattern).

Implementation
==============
`representations/repair.clj`:

- New `iso-date-pattern` regex covering `yyyy-mm-dd` plus optional
  `Thh:mm[:ss[.fff]][Z|±hh:mm]` time portion.
- `iso-date-string?`, `temporal-shaped-clause?` helpers (the latter
  recognising `absolute-datetime`, `relative-datetime`, `now`, and the
  `get-*` extraction heads).
- `wrap-iso-date-bounds*` postwalk: for every `[between, opts, expr, lo,
  hi]` clause, when at least one of `lo` / `hi` is either already temporal-
  shaped *or* an ISO-date string, wrap the bare ISO bounds as
  `[absolute-datetime, {}, <iso-str>, "day"]`.
- The "or ISO-date" arm extends the sexp behaviour slightly: sexp's
  `repair-between-bounds` only fired when one side was already temporal-
  shaped. We also handle the both-bare-ISO-strings case because it's
  unambiguous (between of two bare yyyy-mm-dd strings is necessarily a
  temporal range; numeric between uses numbers, text between is unusual)
  and otherwise fails validation outright.
- Added to the pipeline before `swap-between-bounds*` (the swap pass knows
  how to compare via the inner ISO string when bounds are wrapped).

Idempotency
===========
After the first wrap, both bounds are vectors and the wrap predicate
(string + ISO regex) no longer matches; second pass is a no-op. The
property test continues to pass.

Compose with 1.5
================
The previously-added `swap-between-bounds-iso-string-test` (commit
cc47c17619f) was updated: bare-ISO inputs now produce wrapped-and-swapped
output (canonical form), not bare-and-swapped. This is a forward step -
the lib schema rejects bare strings, so the previous test's expected
output was actually invalid as a final form.

Tests
=====
`representations/repair_test.clj`: 7 new deftests covering both-bare,
mixed (with absolute-datetime / with relative-datetime), numeric-untouched,
time-portion bounds, idempotency, wrap+swap composition, and non-ISO-string
untouched. Adjusted 3 swap-iso-string assertions in
`swap-between-bounds-iso-string-test` to expect wrapped form.

71 / 159 in `representations.repair-test`: all green.

* Repair: wrap bare 'now' literals in temporal-comparison contexts (repr-deletion-followups 1.7)

LLMs sometimes write `[<, {}, <date-field>, "now"]` or `[between, {}, <field>,
"2024-01-01", "now"]` - using the bare string `"now"` (case-insensitive,
optional whitespace) where lib expects a `["now" {}]` clause. Without this
pass the bare string fails schema validation. Carried over from the sexp
pipeline's `wrap-now-as-expression` (see `repr-deletion-followups.md` § 1.7,
flagged as worth re-adding combined with 1.6).

Implementation
==============
`representations/repair.clj`:

- `now-literal?`: case-insensitive, whitespace-tolerant predicate matching
  `"now"` / `"NOW"` / `" now "` / `"Now\n"`.
- `field-with-temporal-unit?`: a `field` clause whose options carry
  `temporal-unit`. Combined with `temporal-shaped-clause?` (from § 1.6) gives
  `temporal-context-operand?`.
- `temporal-comparison-clause?`: a clause whose head is in
  `#{=, !=, <, <=, >, >=, between}` and which has a real options-map.
- `maybe-wrap-now-operands`: walks the args of a comparison clause, returns
  rewritten args **only if at least one *other* operand in the clause is\n  already temporal-shaped**. The other-operand requirement is what keeps a
  column literally named or containing `"now"` from being silently rewritten
  in non-temporal contexts.
- `wrap-now-literals*` postwalk added to the pipeline right after
  `wrap-iso-date-bounds*` and before `swap-between-bounds*`.

Idempotency
===========
After the first pass, `"now"` is now `["now" {}]` (a vector, not a string),
so `now-literal?` no longer matches. Second pass is a no-op. Property test
continues to pass.

Tests
=====
`representations/repair_test.clj`: two new deftests covering

  - `<`/`>=`/`=` against temporal-bucketed field, absolute-datetime, plus
    case-insensitive / whitespace variants;
  - bare 'now' in a non-temporal comparison left alone;
  - already-canonical ["now" {}] left alone (idempotency);
  - between with mixed ISO + 'now' bounds (Pass 1.86 + 1.865 compose);
  - between of two bare 'now' on a numeric field left alone (no temporal
    sibling).

73 / 169 in `representations.repair-test`: all green.

* Repair: unwrap ["true"/"false" {} x] boolean wrappers (repr-deletion-followups 1.4)

LLMs occasionally write `["true", x]` thinking of it as "wrap this clause
as truthy", or `["false", x]` as "wrap as falsy". Neither has any meaning in
lib; the canonical equivalents are just `x` and `["not", {}, x]`. Carried
over from the sexp pipeline's `syntax/boolean-wrapper-form?` (see
`repr-deletion-followups.md` § 1.4 - flagged as "re-add only if seen";
porting because user requested full Phase A + B).

Implementation
==============
`representations/repair.clj`:

- `boolean-wrapper-clause?` recognises a 3-element vector
  `[<true|false case-insensitive>, <opts-map>, <one arg>]`. Restricted to
  exactly one arg so 0-arg boolean literals (`["true" {}]`,
  `["false" {}]`) and multi-arg uses of these heads are not touched.
- `unwrap-boolean-wrapper` returns `x` for true and `["not" {} x]` for false.
- `unwrap-boolean-wrappers*` postwalk added to the pipeline right after
  Pass 1 (`ensure-clause-options*`), before `unwrap-nested-field-clauses*`,
  so postwalk evaluation order unwraps innermost wrappers first.

Idempotency
===========
After the unwrap, the head is no longer "true" / "false". The 0-arg literal
case is explicitly excluded from the predicate, so `["true" {}]` /
`["false" {}]` pass through untouched. Property test continues to pass.

Tests
=====
`representations/repair_test.clj`: 5 new deftests covering

  - true wrapper unwraps;
  - true wrapper survives bare-clause input (Pass 1 + Pass 1.83 chained);
  - false wrapper rewrites to (not …);
  - 0-arg ["true" {}] / ["false" {}] literals untouched;
  - case-insensitive (TRUE/True/FALSE/False) heads work;
  - nested wrappers unwrap to the innermost expression.

78 / 180 in `representations.repair-test`: all green.

* Repair: normalise alternative case / if argument shapes (repr-deletion-followups 1.8)

lib's canonical `case` shape is
`[case, {}, [[pred1 then1] [pred2 then2] …] default?]` - a single vector of
branch pairs in slot 2 plus an optional default in slot 3. LLMs frequently
emit alternative shapes, especially when transcribing English-flavoured
prose ("if X then 100 else 0"):

  1. Three bare args: `[case, {}, pred, then, else]`
  2. Branch pairs as separate args: `[case, {}, [p1 t1], [p2 t2], default]`
  3. Flat alternating: `[case, {}, p1, t1, p2, t2, default?]`
  4. Trailing `["else" x]` branch inside the pairs vector

Without this pass each alternative fails lib schema validation. `if` is a
documented alias for `case` (lib treats them identically); we normalise
both. Carried over from the sexp pipeline's
`repair/normalize/forms/repair-case-args` (see `repr-deletion-followups.md`
§ 1.8 - flagged "re-add only if seen"; porting because user requested full
Phase A + B).

Implementation
==============
`representations/repair.clj`:

- `branch-pair?`: 2-element vector.
- `else-branch?`: `["else" x]` or its post-Pass-1 form `["else" {} x]`
  (Pass 1 fills `{}` into bare clauses before this pass runs).
- `split-trailing-else`: peels a final else-branch off the pairs vector and
  promotes its payload to the default position.
- `canonical-case-args`: uniform output construction
  `[<vector-of-pairs> default?]`, always stripping a trailing else.
- `classify-case-args`: returns `[branches default]` for any of the shapes
  above (including the canonical one, so that trailing-else stripping is
  centralised). Returns `nil` only for unrecognised shapes.
- `normalise-case-args`: builds the canonical clause and compares against the
  input - returns the input unchanged when the rewritten form is identical
  (so already-canonical input doesn't get re-wrapped, preserving cheap
  idempotency).
- `normalise-case-clauses*` postwalk added to the pipeline right after
  `swap-between-bounds*` and before `normalize-expressions-shape*`.

Idempotency
===========
`canonical-case-args` is a fixed point under repeated application: once
the form is canonical (a single vector of branch-pairs plus optional
default, no trailing else), the next pass produces the identical clause and
`normalise-case-args`'s equality check short-circuits. Property test
continues to pass.

Tests
=====
`representations/repair_test.clj`: 8 new deftests covering each shape
above, the `if` alias, already-canonical inputs (idempotency), and an
explicit idempotency check across all five shapes.

86 / 196 in `representations.repair-test`: all green.

* Repair: normalise fields: to sequential when given a single clause (repr-deletion-followups 1.9)

The repr schema for both stage and join requires
`"fields": [:sequential [:ref ::clause]]` (joins additionally accept enum
strings `"all"` / `"none"`). LLMs occasionally write a single clause as
the value, e.g.

  stages:
  - …
    fields: [field, {}, [Sample, PUBLIC, T, COL]]

rather than the canonical list-of-one. The single-clause shape would
otherwise fail validation. Carried over from the sexp pipeline's
`normalize-with-fields-selection` (see `repr-deletion-followups.md` § 1.9 -
flagged "low risk; the shape is structurally obvious"; porting because user
requested full Phase A + B).

Implementation
==============
`representations/repair.clj`:

- `single-clause-shape?`: a vector with a string head at slot 0 AND a map at
  slot 1. This excludes (a) a sequential-of-clauses (no map at slot 1 of the
  outer vector) and (b) the enum strings `"all"` / `"none"` (not vectors).
- `normalise-fields-key`: when a map's `"fields"` value is single-clause-
  shaped, wrap it in a one-element list.
- `normalise-fields-shape*` postwalk fires on every map containing a
  `"fields"` key. The walker hits both stage maps and join maps; we don't
  need separate stage / join handling because the schema and the LLM
  failure mode are identical.
- Added to the pipeline right after `normalise-case-clauses*` and before
  `normalize-expressions-shape*`.

Idempotency
===========
After the first wrap the value is a list (not a clause-shaped vector) and
the predicate no longer matches. Second pass is a no-op. Property test
continues to pass.

Tests
=====
`representations/repair_test.clj`: 3 new deftests covering

  - stage-level `fields:` single clause wrap;
  - sequential `fields:` left alone (already canonical);
  - join-level `fields:` single clause wrap;
  - join `fields:` enum value `"all"` / `"none"` left alone;
  - explicit idempotency check.

89 / 202 in `representations.repair-test`: all green.

* Repair: splat values-list args of in/=/!=/not-in to flat positional args (repr-deletion-followups 1.10)

lib's canonical shape for `:in` / `:not-in` is **flat**:
`[in, {}, lhs, v1, v2, v3]`. Likewise `:=` / `:!=` take exactly two operand
args. LLMs sometimes wrap the values into a single vector slot:

  [in,     {}, lhs, [v1, v2, v3]]   →  [in,     {}, lhs, v1, v2, v3]
  [not-in, {}, lhs, [v1, v2, v3]]   →  [not-in, {}, lhs, v1, v2, v3]

and occasionally use `=` / `!=` with a list value, expecting list-membership
semantics:

  [=,  {}, lhs, [v1, v2, v3]]       →  [in,     {}, lhs, v1, v2, v3]
  [!=, {}, lhs, [v1, v2, v3]]       →  [not-in, {}, lhs, v1, v2, v3]

Adapted from the sexp pipeline's `in`/`not-in`/`=`/`!=` repair branches in
`repair/normalize/forms.clj` (see `repr-deletion-followups.md` § 1.10). The
sexp form `[in lhs [v1 v2 v3]]` was nested by design; the repr canonical
form is flat, so we splat instead of just rewriting the head.

Implementation
==============
`representations/repair.clj`:

- `scalar-literal?`: a primitive YAML scalar (string, number, boolean, nil).
- `values-list?`: a non-empty vector whose every entry is a scalar-literal.
  This conservative predicate excludes nested clauses (vectors with a
  string head + map options) and FK-paths (which only appear inside a
  `field` clause anyway).
- `list-value-comparison-clause`: returns `[head opts lhs values]` for two
  shapes:
  - `[head, opts-map, lhs, [v1 v2 …]]` (4 elements, options already in place);
  - `[head, lhs, [v1 v2 …]]` (3 elements, options omitted).
  Both shapes are recognised because the pass runs *before*
  `ensure-clause-options*` (Pass 1).
- `normalise-list-value-comparisons*` walks the tree and applies the splat
  for `in`/`not-in` heads, plus the head-rewrite-and-splat for `=`/`!=`.

Pipeline placement
==================
**Critical:** the new pass runs *before* `ensure-clause-options*`. Pass 1
mis-identifies a 2-element values-list (e.g. `["alice" "bob"]`) as a
bare clause and would corrupt it by inserting `{}` between the two scalars,
producing `["alice" {} "bob"]`. Running the splat first means Pass 1 only
ever sees properly-structured clauses where `in`/`not-in`'s args have
already been spread to positional form. (Trying to undo Pass 1's
corruption afterwards would require very heuristic detection of "a
3-element vector with empty-map at slot 1 might actually be a corrupted
values-list".)

Idempotency
===========
After splat, the values are no longer in a vector and the predicate no
longer matches. Already-flat input passes through unchanged. Property test
continues to pass.

Tests
=====
`representations/repair_test.clj`: 5 new deftests covering

  - `in` / `not-in` splat;
  - already-flat in clause unchanged;
  - `=` / `!=` with list value rewrite + splat;
  - scalar-rhs `=` left alone;
  - clause-shaped rhs (e.g. an `absolute-datetime`) left alone;
  - mixed scalar types (numbers, booleans) splat correctly;
  - explicit idempotency check.

181 / 639 across the wider repr-touching suite (`repair-test`,
`resolve-test`, `construct-representations-test`, `agent_api.api-test`,
`mcp.api-test`): all green; 0 regressions.

* Repair: rewrite shell/Python/SQL operator-name aliases to canonical lib heads (repr-deletion-followups 1.1)

LLMs are heavily-trained on shell scripts (`man test`-style `-eq`/`-ne`/`-lt`/
`-gt`/etc.), Python comparison protocols (`__eq__`, `__lt__`, …), and SQL-ish
prose. They occasionally reach for `eq`/`equals`, `gt`/`gte`, etc. as
operator heads instead of the canonical lib symbols (`=`, `!=`, `<`, `<=`,
`>`, `>=`). `repr-deletion-followups.md` § 1.1 originally recommended NOT
re-introducing these aliases, but the cost of supporting them is one extra
hashmap lookup at the start of the pipeline and the failure mode is silent
schema rejection - so the trade-off favours support.

Implementation
==============
`representations/repair.clj`:

- `operator-name-aliases` map covering:
  - shell/Python comparison: `eq`, `ne`, `lt`, `le`/`lte`, `gt`, `ge`/`gte`;
  - verbose comparison: `equals`, `not-equals`;
  - lib renames carried over from the sexp pipeline's
    `syntax/canonical-operator-aliases`: `count-if` → `count-where`,
    `variance` → `var`, `stddev-pop` → `stddev`, `count-distinct` /
    `distinct-count` → `distinct`, `relative-date` → `relative-datetime`,
    `temporal-diff` → `datetime-diff`, `is-not-null` → `not-null`.
- `operator-alias-clause?` predicate: head is a string matching (case-
  insensitively) a known alias, slot 1 is the options-map (i.e. it really is
  a clause, not an FK path string), and head is *not* already canonical
  (cheap idempotency).
- `rewrite-operator-name-aliases*` postwalk added to `normalize-shape*`
  right after `unwrap-nested-field-clauses*` and before
  `rewrite-temporal-bucket-aliases*`.

Notably **not** ported from sexp:
- `if` → `case`: lib accepts both heads as identical (see
  `metabase.lib.schema.expression.conditional` § "`:if` is an alias to `:case`");
- `month` → `get-month`: ambiguous (could be `:temporal-bucket :month`);
- `|` → `or`: too symbolic.

Idempotency
===========
After rewrite the head is canonical, alias map no longer matches. Already-
canonical clauses pass through unchanged. Property test continues to pass.

Tests
=====
`representations/repair_test.clj`: 7 new deftests covering

  - 8 shell-style comparison aliases → canonical;
  - verbose `equals` / `not-equals`;
  - 5 aggregation lib-renames;
  - 3 temporal/null lib-renames;
  - case-insensitive head matching (EQ/Eq/EQUALS/Equals);
  - already-canonical heads untouched (idempotency);
  - alias nested in tree gets rewritten;
  - column literally named "eq" / "gt" inside FK path NOT rewritten.

189 / 670 across the wider repr-touching suite (`repair-test`,
`resolve-test`, `construct-representations-test`, `agent_api.api-test`,
`mcp.api-test`): all green; 0 regressions.

* fix repair.clj

* Repair: auto-split a stage with a post-aggregation filter into two stages (repr-deletion-followups 1.11)

LLMs habitually write SQL HAVING-style post-aggregation filters as a single
stage, e.g.

    stages:
    - source-table: [Sample, PUBLIC, ORDERS]
      aggregation: [[count, {}]]
      breakout:    [[field, {}, [Sample, PUBLIC, ORDERS, PRODUCT_ID]]]
      filters:     [['>', {}, [aggregation, {}, 0], 10]]
      order-by:    [[desc, {}, [aggregation, {}, 0]]]
      limit:       5

In pMBQL this must be expressed as **two stages** - the second consumes the
first's output and references the aggregation's column by name:

    stages:
    - source-table: [Sample, PUBLIC, ORDERS]
      aggregation: [[count, {}]]
      breakout:    [[field, {}, [Sample, PUBLIC, ORDERS, PRODUCT_ID]]]
    - filters:     [['>', {}, [field, {}, count], 10]]
      order-by:    [[desc, {}, [field, {}, count]]]
      limit:       5

lib's schema does not catch the single-stage shape (it's shape-only) - the
broken query passes validate, passes resolve, and only fails at SQL-
generation time or, worse, silently produces wrong results.

Implementation
==============
`representations/repair.clj`:

- `aggregation-column-name-table`: lib-mirroring static map from aggregation
  head to the output column name lib's `column-name-method` produces.
  Covers `count`, `distinct`, `cum-count`, `count-where`, `distinct-where`,
  `avg`, `max`, `median`, `min`, `stddev`, `sum`, `var`, `cum-sum`,
  `percentile`. `metric` is intentionally not in the table - resolving its
  column name needs the metadata-provider, and the auto-split caller emits
  a clean diagnostic when that case is hit.
- `aggregation-clause-column-name`: extracts the column name an aggregation
  produces. An explicit `"name"` in opts (lib's column-name override for
  aggregations) takes precedence over the static-table dispatch.
- `collect-stage-agg-uuids`: set of `lib/uuid` strings from the stage's
  aggregation clauses.
- `form-references-stage-agg?`: postwalk detection of any
  `[aggregation, opts, <uuid>]` clause whose uuid is in the same-stage set.
- `aggregation-uuid->column-name`: builds `{uuid → col-name}` map; returns
  nil if any aggregation can't be named (signal for "emit diagnostic").
- `rewrite-agg-refs-to-cross-stage-fields`: postwalk replaces every
  same-stage aggregation-ref with the cross-stage `[field, {}, <name>]`
  shape, dropping aggregation-specific opts (Pass 5 will re-stamp types).
- `split-post-agg-filter-stage`: detects the trigger and either returns
  `[stage]` (no split) or `[stage' new-stage]` (split). When split is
  required but column-names can't be derived, throws `:agent-error?` with
  a clear instruction.
- `split-post-agg-filters*`: top-level pass; runs after
  `resolve-aggregation-ref-indexes*` (so integer-index refs are already
  UUID-keyed) and before `infer-cross-stage-field-types*` (so the new
  cross-stage field refs get `base-type`/`effective-type` filled in).

Auto-fix policy
===============
When split fires:

- **Pre-agg filters stay in stage 0** so they filter rows before
  aggregation as the LLM intended.
- **Post-agg filters move to stage 1** with refs rewritten to cross-stage.
- **`order-by`, `limit`, `fields` move to stage 1** - the user's intent is
  "of {rows matching the filter}, the top-N sorted by X, projecting Y".
  Leaving these in stage 0 would sort/limit/project before the filter,
  silently producing different (smaller) row sets and the wrong semantics.

Idempotency
===========
After split, stage 0 has no offending agg-ref in its filters and stage 1
has no `aggregation:` vector at all. Re-running the pass finds no
triggering shape and is a no-op. Property test continues to pass.

Property-test fixup
===================
`representations/repair.clj` (`list-value-comparison-clause` from § 1.10):
the property test surfaced an idempotency failure on adversarial inputs
like `["=" nil ["DB" "S" "T"] ["v1" "v2"]]` - on the 1st run nothing
fires (no opts at slot 1, count=4, predicate misses); on the 2nd run with
`{}` stamped at slot 1 the value-list splat mis-classifies the bare
3-string FK-path as the LHS of an `=` comparison and splats. Added a
`comparison-lhs-clause?` requirement: the LHS must itself be a clause
(string head + map opts at slot 1), excluding bare vectors of strings.

Tests
=====
`representations/repair_test.clj`: 9 new deftests covering

  - simple count + post-agg filter splits cleanly;
  - pre-agg filters stay in stage 0, post-agg filters move;
  - order-by, limit, fields move to stage 1;
  - `name` opts override produces a cross-stage ref by that authored name;
  - multiple aggregations each get their own column-name mapping;
  - `metric` agg raises clean `:agent-error?` diagnostic;
  - no trigger when no agg-ref-filter present;
  - no trigger when stage has no aggregations;
  - explicit idempotency on already-split query;
  - `distinct` → "count", `count-where` → "count_where" lib-correct names.

199 / 698 across the wider repr-touching suite (`repair-test`,
`resolve-test`, `construct-representations-test`, `agent_api.api-test`,
`mcp.api-test`): all green; 0 regressions; idempotency property test
passes 100 cases.

* Repair test: expand fuzz coverage with realistic-shape generators (repr-deletion-followups 4.1)

The original `idempotency-property-test` ran 100 cases over a single 'chaos
tree' shape with 18 ops in the head pool. After Phase A + B + 1.1 + 1.11 the
repair pipeline now has 14 normalisation passes plus 7 mp-aware passes, and
each new pass is a potential source of subtle non-idempotency on adversarial
input. The original generator missed several operator heads with repair
passes attached and never produced query-shaped maps with realistic stage
structure.

Changes
=======

`representations/repair_test.clj`:

- **`gen-op-name` expanded to 56 heads** covering every op with a repair
  pass attached: comparison + shell aliases (1.1), list-membership (1.10),
  boolean wrappers (1.4), null checks (1.1), temporal range/literals
  (1.5/1.6/1.7), temporal-bucket extraction + aliases (1.2), direction
  spellings (1.3), aggregations + lib renames (1.1), and `metric` (to
  exercise the 1.11 diagnostic path).

  `case` / `if` / `else` are deliberately omitted from the chaos pool: Pass
  1.84 wraps pairs as `[[pred then]]`, and Pass 1's `clause-like?`
  heuristic mis-classifies a bare 2-element `[s1 s2]` pair as a clause and
  inserts `{}` between the two scalars. In real LLM output `pred` is always
  a clause (e.g. `[\"=\" {} field val]`), so the collision doesn't fire;
  the realistic-query generator below covers canonical case shapes with
  proper clause preds.

- **`idempotency-property-test` bumped to 200 cases** (was 100).

- **New `gen-realistic-query`**: a query-shaped map with a single stage that
  composes:
  - `gen-source-table`: portable FK to a real-looking table;
  - `gen-aggregation-clause`: simple + arg-taking aggs, with LLM-flavoured
    aliases (`count-distinct`, `count-if`, `variance`) sprinkled in;
  - `gen-filter-clause`: covers `=`, `!=`, `eq`/`gt`/`>=` aliases, `in`
    (both flat and wrapped-values shapes), `between` with potentially-
    out-of-order ISO date strings + `now`, and a post-aggregation filter
    that triggers Pass 2.9 split;
  - `gen-order-by-clause`: with mixed-case direction aliases;
  - `gen-expression-clause`: includes `gen-case-clause` (canonical, three-
    bare-args, and pairs-as-args shapes - all with clause preds), arithmetic
    (`+`), and `datetime-diff` (`temporal-diff` alias).

- **`realistic-query-idempotency-property-test`** (200 cases): for each
  generated query, asserts either:
  - `(= (repair q) (repair (repair q)))`, or
  - `repair` throws an `:agent-error?` ex-info (a clean diagnostic path,
    e.g. integer-agg-ref-out-of-range when the random generator emits a
    post-agg filter on a stage with no aggregations).
  Any other thrown exception (NPE, AssertionError, generic ExceptionInfo
  without `:agent-error?`) fails the test.

- **`realistic-query-no-bug-exceptions-test`** (200 cases): asserts that
  no input ever produces a non-`:agent-error?` exception. This is the
  contract of `repair`: it must either succeed or throw with a flag telling
  the agent layer to relay a friendly message to the LLM.

- **`repair-or-agent-error` helper**: classifies a repair call into
  `[:ok r]`, `[:agent-error data]`, or `[:bug ex]`. Centralises the
  exception triage so both new property tests share the same vocabulary.

Bug discovered, scope-documented
================================
While bringing this up the wider `gen-op-name` (with `case`/`if` in scope)
surfaced an interaction between Pass 1 and Pass 1.84 on adversarial inputs
like `[\"case\" {} \" \" \"\"]` (case with bare-string pred and then):

  - Run 1: `[\"case\" {} \" \" \"\"]` → `[\"case\" {} [[\" \" \"\"]]]`
    (Pass 1.84 wraps the two args as a single `[pred then]` pair).
  - Run 2: Pass 1 sees the inner `[\" \" \"\"]` 2-element vector, treats it
    as a clause-like `[head opts? args]` (because the first elem is a
    non-blank string), and inserts `{}`: `[\"case\" {} [[\" \" {} \"\"]]]`.
  - Result: non-idempotent.

This is a fundamental ambiguity between case-pair shape `[pred then]` and
clause shape `[op opts? args]` when `pred` happens to be a string. Real LLM
output never puts a bare string at `pred`, so production is unaffected. The
fix is generator-side: keep `case`/`if` out of the chaos pool and exercise
them only via the realistic generator (which always uses clause preds).
This is documented in-place in the `gen-op-name` literal.

Stability
=========
10 consecutive runs of `repair-test` with fresh seeds: 0 failures, 0 errors.

201 / 700 across the wider repr-touching suite (`repair-test`,
`resolve-test`, `construct-representations-test`, `agent_api.api-test`,
`mcp.api-test`): all green; 0 regressions.

* fix agent.core-test

* Repair: friendly error when a stage's aggregation: contains a [field, ...] entry (E1, repr-deletion-followups 2.3)

Discovery: a query like

    stages:
    - source-table: [Sample, PUBLIC, ORDERS]
      aggregation:
      - [field, {}, [Sample, PUBLIC, ORDERS, TOTAL]]

is silently accepted by lib (shape-only schema), passes resolve, and fails
only at SQL-generation time - or, worse, produces wrong-result SQL. The
`aggregation:` block must contain aggregation functions (`count`, `sum`,
`avg`, `metric`, …), not bare field references.

This commit introduces a new last-pass diagnostic stage in `repair`,
`friendly-errors*`, that runs every error detector in turn after every
shape pass and every mp-aware fill-in is done. Each detector targets a
single LLM-observed pattern and throws `:agent-error?` ex-info with an
LLM-actionable explanation naming the precise location.

Implementation
==============
`representations/repair.clj`:

- New section "Pass 6 - friendly error messages" introducing the pass.
- `aggregation-entry-not-aggregation-error!`: walks each stage's
  `aggregation:` vector and throws when a top-level entry is a
  `[field, ...]` clause. Names the stage and entry index in the data and
  message.
- Critically does NOT recurse into nested clauses: `[sum, {}, [field, ...]]`
  is correct (the outer `sum` is the aggregation; the inner `field` is
  its argument) and is left alone.
- `friendly-errors*` driver appended to the `repair` pipeline as the
  final step.

Tests
=====
`representations/repair_test.clj`: 4 new deftests covering

  - `[field, ...]` as aggregation entry → clean `:agent-error?` with
    `:aggregation-entry-not-aggregation` error code, correct stage/entry
    indices, message mentions `aggregation` and `breakout`;
  - canonical aggregation entries (`count`, `sum`, `avg`, `metric`)
    pass through;
  - `[field, ...]` *nested* inside an aggregation (`[sum, {}, [field, ...]]`)
    is not flagged;
  - multi-stage: error correctly reports the offending stage's index.

205 / 712 across the wider repr-touching suite (`repair-test`,
`resolve-test`, `construct-representations-test`, `agent_api.api-test`,
`mcp.api-test`): all green; 0 regressions; idempotency property tests
(adversarial chaos and realistic-shape, 200 cases each) pass.

* Repair: friendly error when case/if has "default" in opts (E2, repr-deletion-followups 2.3)

LLMs (especially those primed by JSON-Schema-style examples) sometimes
write a `case` clause with the fallback value as a `"default"` key in
the options map:

    [case, {default: "unknown"}, [[pred1 then1] [pred2 then2]]]

lib's canonical shape uses the **third positional argument** as the
fallback:

    [case, {}, [[pred1 then1] [pred2 then2]], "unknown"]

lib silently ignores the unknown `default` opts key and the case clause
evaluates to `null` for any branch miss - a wrong-result class of bug.

Implementation
==============
`representations/repair.clj`:

- `case-default-in-opts-error!`: postwalk detector firing on any `case`
  or `if` clause whose options map contains a `"default"` key. Throws
  `:agent-error?` ex-info with `:case-default-in-opts` error code and
  the offending clause.

- `friendly-errors*` driver: invokes the new detector after the
  per-stage `aggregation-entry-not-aggregation-error!` sweep.

Carried over from the sexp pipeline's
`validate/operators.clj/validate-operator-specific!` `case` branch.

Tests
=====
`representations/repair_test.clj`: 2 new deftests covering

  - `case` with `default` in opts → clean `:agent-error?` with
    `:case-default-in-opts` error code, message mentions "third
    positional argument";
  - `if` with `default` in opts → same error code (alias);
  - canonical `case` with default as 3rd arg passes through;
  - canonical `case` with no default at all passes through.

207 / 718 across the wider repr-touching suite (`repair-test`,
`resolve-test`, `construct-representations-test`, `agent_api.api-test`,
`mcp.api-test`): all green; 0 regressions; idempotency property tests
(adversarial chaos and realistic-shape, 200 cases each) pass.

* Repair: friendly error when sexp-legacy top-level ops appear as clause heads (E3, repr-deletion-followups 2.3)

LLMs trained on the older sexp pipeline still occasionally emit top-level
operations as clause heads:

    aggregation: [[aggregate, {}, [count, {}]]]
    filters:     [[filter, {}, [=, ...]]]

These were verbs at the *program* level in sexp; they have no equivalent
in repr (which uses stage-level blocks: `aggregation:`, `filters:`,
`breakout:`, `order-by:`, `expressions:`, `limit:`, `joins:`). lib
accepts the unknown clause head silently, the resolver passes, and the
QP either produces wrong results or fails at SQL-generation time.

Implementation
==============
`representations/repair.clj`:

- `sexp-legacy-top-level-ops` map: 5 offending heads
  (`aggregate`, `filter`, `order-by`, `breakout`, `limit`) each
  paired with a one-liner pointing at the canonical repr replacement.
- `sexp-legacy-op-as-clause-error!`: postwalk detector firing on any
  vector with one of these heads. Throws `:agent-error?` ex-info with
  `:sexp-legacy-op-as-clause` error code, the offending head, and the
  full clause. The `tru` template carries the head + per-head hint.
- Excludes `map-entry?` nodes critical: postwalk descends into map
  entries and a stage's `{\"breakout\" […]}` entry would otherwise look
  exactly like a `[\"breakout\", …]` clause to the detector and trip.
- Same `map-entry?` guard added to E2 (`case-default-in-opts-error!`)
  for safety.

- `friendly-errors*` driver: invokes the new detector after E1 and E2.

Carried over from the sexp pipeline's
`validate/operators.clj/validate-operator-specific!` top-level-operation
branch.

Tests
=====
`representations/repair_test.clj`: 4 new deftests covering

  - `[aggregate, ...]` clause head → clean `:agent-error?` with
    `:sexp-legacy-op-as-clause` error code, head="aggregate", message
    mentions "top-level operation";
  - `[filter, ...]` clause head → same error code, head="filter";
  - all 5 legacy ops (aggregate/filter/order-by/breakout/limit) trip;
  - canonical stage with `aggregation:`, `filters:`, `breakout:`,
    `order-by:`, `limit:` keys passes through (map-entry guard works).

211 / 729 across the wider repr-touching suite (`repair-test`,
`resolve-test`, `construct-representations-test`, `agent_api.api-test`,
`mcp.api-test`): all green; 0 regressions; idempotency property tests
(adversarial chaos and realistic-shape, 200 cases each) pass.

* Repair: friendly error when sexp-legacy [measure, ...] head appears (E4, repr-deletion-followups 2.3)

`measure` was a sexp-pipeline aggregation helper - a metabase-internal
concept that wrapped a metric and exposed it as an aggregation under a
different identifier. It has no equivalent in repr; metrics are just an
aggregation form: `[metric, <opts>, <portable_entity_id>]`. Without this
diagnostic the head is silently treated as an unknown clause and the
query produces wrong results or fails downstream.

Implementation
==============
`representations/repair.clj`:

- `sexp-legacy-measure-error!`: postwalk detector firing on any vector
  with head `"measure"`. Throws `:agent-error?` ex-info with
  `:sexp-legacy-measure` error code and the offending clause. Excludes
  `map-entry?` nodes (defensive; no realistic stage map carries a
  `measure` key, but consistent with E2/E3).
- `friendly-errors*` driver: invokes the new detector after E1, E2, E3.

Carried over from the sexp pipeline's
`validate/operators.clj/validate-operator-specific!` `measure` branch.

Tests
=====
`representations/repair_test.clj`: 2 new deftests covering

  - `[measure, ...]` clause head → clean `:agent-error?` with
    `:sexp-legacy-measure` error code, message mentions `metric`;
  - canonical `[metric, {}, <portable_entity_id>]` aggregation passes
    through (the entire point of the suggested replacement).

213 / 733 across the wider repr-touching suite (`repair-test`,
`resolve-test`, `construct-representations-test`, `agent_api.api-test`,
`mcp.api-test`): all green; 0 regressions; idempotency property tests
(adversarial chaos and realistic-shape, 200 cases each) pass.

* Repair: friendly error when [expression, opts, name] reference is blank (E5, repr-deletion-followups 2.3)

LLMs occasionally write a reference to a named expression with an empty
or whitespace-only identifier:

    filters: [['=', {}, [expression, {}, ''], 0]]

lib's schema requires a non-blank string identifier here (matching an
entry in some upstream stage's `expressions:` block); without it the
reference resolves to nothing and the query produces wrong results or
fails downstream.

Implementation
==============
`representations/repair.clj`:

- `blank-expression-ref-error!`: postwalk detector firing on any
  `[expression, <opts>, <name>]` (3-element clause with head `expression`
  and map-shaped opts) where the name slot is non-string, missing, or
  blank after trim. Throws `:agent-error?` ex-info with
  `:blank-expression-ref` error code.
- `friendly-errors*` driver: invokes the new detector after E1-E4.

Carried over from the sexp pipeline's
`validate/operators.clj/validate-operator-specific!` `expression-ref`
branch.

Tests
=====
`representations/repair_test.clj`: 4 new deftests covering

  - empty-string ref (`""`) → `:blank-expression-ref`;
  - whitespace-only ref (`"   "`) → trim-aware detection;
  - non-string ref (`nil`) → triggers too;
  - valid non-blank ref (`"my-expr"`) passes through.

215 / 738 across the wider repr-touching suite (`repair-test`,
`resolve-test`, `construct-representations-test`, `agent_api.api-test`,
`mcp.api-test`): all green; 0 regressions; idempotency property tests
(adversarial chaos and realistic-shape, 200 cases each) pass.

* Repair: friendly error when [field, opts, <int>] uses sexp-legacy numeric id (E6, repr-deletion-followups 2.3)

LLMs trained on the older sexp / legacy-MBQL pipeline still occasionally
emit the bare-numeric-id field reference shape:

    filters: [['=', {}, [field, {}, 100], 0]]

repr requires a portable FK in the third slot - either a vector
`[<database>, <schema>, <table>, <column>]` (resolved against the
metadata provider) OR a string column name (cross-stage reference).
lib's schema does reject bare numeric ids, but its error is a generic
shape mismatch ("got integer, expected vector / string") that doesn't
explain the portable-FK syntax. This detector surfaces an LLM-actionable
explanation pointing at the canonical replacement.

Implementation
==============
`representations/repair.clj`:

- `numeric-field-id-error!`: postwalk detector firing on any
  `[field, <opts>, <int>]` clause (3-element, head `field`, opts is a
  map, third slot is an integer). Throws `:agent-error?` ex-info with
  `:numeric-field-id` error code, message points at portable-FK and
  cross-stage column-name.
- `friendly-errors*` driver: invokes the new detector after E1-E5.

Carried over from the sexp pipeline's
`validate/operators.clj/validate-operator-specific!` `field` branch.

Tests
=====
`representations/repair_test.clj`: 3 new deftests covering

  - `[field, {}, 100]` numeric id → clean `:agent-error?` with
    `:numeric-field-id` error code, message mentions "portable FK" +
    "column-name";
  - canonical portable-FK `[field, {}, [Sample, PUBLIC, ORDERS, COL]]`
    passes through;
  - cross-stage string column-name `[field, {}, "count"]` passes
    through.

218 / 744 across the wider repr-touching suite (`repair-test`,
`resolve-test`, `construct-representations-test`, `agent_api.api-test`,
`mcp.api-test`): all green; 0 regressions; idempotency property tests
(adversarial chaos and realistic-shape, 200 cases each) pass.

* fix test

* Enforce permissions in repr query construction

* Fix sample database prompt examples

* Default agent input errors to HTTP 400

* Export final pMBQL back to portable repr YAML

The construct_notebook_query tool now feeds the LLM the **final** normalized
pMBQL exported back to portable representations YAML, rather than the
pre-resolve repaired form. The next LLM turn sees exactly the field paths,
card entity_ids, and aggregation UUIDs that will execute, including
everything `lib.normalize` stamped (uuids, base-types, kebab-cased keys).

Implementation:
- Add `repr.resolve/export-query` — inverse of `resolve-query`. Walks the
  numeric pMBQL through the metadata-provider-backed serdes export resolver,
  then converts the keyworded form back to string-keyed repr data with
  `:lib/metadata` stripped.
- Extend the MP-backed export resolver: `export-fk-keyed` for
  `Database.name`, `export-fk` for `Card` / `:model/Card` by `entity_id`
  (covers both `source-card` and `metric` aggregation refs).
- Fix a latent bug in `serdes.resolve/export-mbql`: the sequential branch
  dropped the explicit resolver, falling back to `*export-resolver*` and
  losing the metadata-provider context for non-clause subterms.

Tests:
- `metabase.models.serialization.resolve.mp-test`: new coverage for
  database-name and Card entity_id export plus an `export-mbql` round-trip
  asserting portable FKs / entity_ids land in the right slots.
- `metabase.agent-lib.representations.resolve-test`: round-trip
  `resolve-query` → `export-query`, including `validate-query` on the
  exported form.
- `metabase.metabot.tools.construct-representations-test`: `query-yaml`
  carries lib-normalized `lib/uuid`s and source-card numeric ids export
  back to their portable entity_id.

repr-plan.md step 18.

* Show existing card / transform queries to LLM as repr YAML

Make the agent's view of *existing* questions, models, and transform
sources symmetric with what construct_notebook_query already returns:
the same canonical representations YAML form, both for structured
(mbql.stage/mbql) and native (mbql.stage/native) queries. Previously
existing cards exposed only metadata (no body), and transform-query
sources fell back to pprint'd pMBQL with numeric ids — the latter
defeating portability for card-id / snippet-id template-tag refs.

* New `metabase.agent-lib.representations.resolve/export-query-yaml`:
  thin wrapper over `export-query` + `yaml/generate-string` that returns
  nil on bad input / export errors so a single broken card can't tear
  down LLM context-building.
* `card-details` now exports the card's own `dataset-query` (not the
  `lib/query` source-card wrapper) and threads it as `:query_yaml`.
  `get-report-details` keeps `:query_yaml` in the slim payload.
* `question->xml` / `model->xml` render the YAML as `<query>...</query>`
  inside `<metabase_question>` / `<metabase-model>`.
* `transform-query-source-text`: structured + native MBQL maps both go
  through repr export; pprint'd pMBQL is now only the last-resort
  fallback. Native legacy `:native :query` strings only fall through
  when there's no `:database` to build a metadata-provider from.

Tests: 9 new across resolve / entity-details / llm-shape / user-context
suites covering the structured + native happy paths, missing/blank
fallbacks, and the slim get-report-details payload. 64 tests / 515
assertions green in the touched suites; 159 / 445 in the wider repr
regression. Kondo 0/0.

* unicode normalization

* Auto-fill source-field-join-alias for explicit-join hops

Pass 3.5 (resolve-source-field-join-alias*) covers the explicit-join half
of repr-plan.md step 12: when a stage[0] field clause's target table is
reachable through exactly one explicit join via exactly one FK on the
joined table, repair fills in both source-field-join-alias and the
matching portable source-field. Mirrors Pass 4 (basic implicit join) but
for the case where the FK column lives on a joins[] table rather than on
source-table itself.

Conservative on purpose: skips clauses that already carry any of the four
disambiguators, defers to Pass 4 when source-table itself can reach the
target, and raises :ambiguous-fk-via-join (agent-error?) when multiple
distinct join aliases are candidates.

The companion source-field-name auto-fill (multi-stage previous-stage
column) is intentionally NOT implemented; rationale documented in
repr-deletion-followups.md (entry 1.10b).

* Close v2 agent request maps and regenerate OpenAPI

The /v2/construct-query and /v2/query request schemas were open maps,
silently accepting (and dropping) the legacy `source_entity` /
`referenced_entities` envelope from before the repr migration. Mark
both as `{:closed true}` so callers (especially the LLM, whose stale
memory keeps reaching for the old shape) get an explicit 400 instead
of having extra keys quietly ignored.

Adds `construct-query-rejects-legacy-envelope-test` covering all three
branches (construct-query, /v2/query fresh, /v2/query continuation),
asserting the 'disallowed key' Malli error for the rejected key.

Regenerates docs/api.json and resources/openapi/openapi.json so the
public spec advertises `additionalProperties: false` and drops stale
references to the removed `metabase.agent-lib.schema.program` schema.

* fix tests

* Migrate agent surface from representations YAML to JSON ::external-query

The metabot/agent pipeline now consumes and emits the canonical portable
MBQL 5 wire format (`::lib.schema/external-query`) as JSON instead of YAML.

Affects three surfaces:

- HTTP `/v2/construct-query` and `/v2/query` accept `{"query": <object>}`
  matching `::lib.schema/external-query`. The defendpoint string-transformer
  keywordizes enum values automatically.
- The `construct_notebook_query` and `slackbot_query` tools take a Clojure
  map (decoded from JSON by the agent layer) instead of a YAML string. Args
  schema is `:map` so the repair pipeline still gets a chance to fix LLM
  shortcuts (missing `{}` options, missing `lib/type` markers).
- LLM-facing query payloads — `:query_yaml` on entity_details card/report
  results, on `<metabase-model>` / `<metabase_question>` XML, on
  `construct_notebook_query` results, and on transform sources in the agent
  context — are now `:query_json` rendered as a triple-backtick `json` code
  block inside the existing XML envelope.

The repair pipeline itself keeps operating on the string-keyed portable
form; conversion happens once at the boundary via the new
`representations/external-query->portable` helper.

`construct_notebook_query.md` rewritten end-to-end with JSON examples.
Reference doc and the auto-generated OpenAPI artifacts regenerated.

* Address PR review feedback

Five fixes prompted by the PR 73052 review (all still relevant after the
rebases and the YAML → JSON migration):

- `construct.clj` `as-agent-input-error`: rename the let-binding so the
  inner `cond->` doesn't shadow its own input (nit from @escherize).
- `resolve.mp/app-db-content-store` and the agent's
  `permission-aware-content-store` now short-circuit non-NanoID
  `source-card` values to `nil`. Without the gate a malformed entity_id
  from the LLM falls through `serdes/lookup-by-id` into
  `find-by-identity-hash`, which scans the entire Card table.
- `resolve.mp/import-card-by-entity-id` reads `:database-id` defensively
  with a snake_case fallback, mirroring `export-card-by-id` so test-side
  content stores can return either shape.
- `try-export-query` logs at `warn` instead of `debug` — the catch
  swallows the failure and drops the query body from the LLM payload, so
  a real export bug would otherwise be silently invisible.
- `/v2/query` continuation-token path re-validates query permissions
  before streaming. The token is client-supplied and a user's data perms
  can change between pages; the QP middleware would catch this anyway,
  but an explicit `api/query-check` produces a cleaner 403 and avoids
  spinning up the streaming response just to abort.

* Allow :model/Table in agent-api module imports

The continuation-token permission re-check added in the prior commit calls
`api/query-check :model/Table` against the source-table id carried inside
the token. `:model/Table` lives in warehouse-schema, so agent-api's
`:model-imports` set in `.clj-kondo/config/modules/config.edn` needs to
list it for `metabase.core.modules-test/model-boundaries-test` to pass.

* Drop leftover sexp-program request defs after rebase

The latest rebase landed an auto-merge that kept both the legacy
`::construct-query-request` (referencing the now-deleted
`agent-lib/program-schema`) and the new JSON `::external-query` version,
plus the no-longer-used `ConstructQueryPrompt` /
`construct-query-prompt-max-length` helpers. The JSON-migration commit's
intent was to delete all of these; this commit reapplies that cleanup
and drops the unused `malli.util` require, restores a compileable
`agent-api/api.clj`, and regenerates the OpenAPI artifacts so they match.

* Promote measures and segments to first-class agent query clauses

The schema and serdes resolve patterns for `[:measure opts <id-or-pid>]`
and `[:segment opts <id-or-pid>]` were already in place but the
metadata-provider-backed ContentStore only implemented the Card lookup
path, so the agent could never invoke them. Benchmark/5 and bench/6
showed every *Tables – Measures and Segments* test stuck at 0.125 — the
rubric's `data_source` metric checks for measure_id / segment_id
references in the final query, which the agent had no way to produce.

Three layered changes:

- Resolver wiring (`resolve/mp.clj`). Extend `ContentStore` with
  measure / segment lookups in both directions (entity-id → row for
  import, id → row for export), implemented on `app-db-content-store`
  and on the construct tool's `permission-aware-content-store`. Wire
  `import-resolver` and `export-resolver` to dispatch on
  `:model/Measure` and `:model/Segment` with table → database cross-DB
  guards (measures and segments are table-scoped, not directly
  database-scoped like cards).

- Drop the E4 `sexp-legacy-measure-error!` repair check. It predated
  the resolve patterns and was actively rejecting the now-canonical
  agent input.

- Surface `portable_entity_id` on `<measure>` / `<segment>` tags and
  through `convert-measure-or-segment`, mirroring the existing
  `<metric>` pattern. The agent now has the opaque NanoID to paste
  verbatim into `["measure", {}, "<pid>"]` /
  `["segment", {}, "<pid>"]` clauses.

`construct_notebook_query.md` is also rewritten end-to-end to the
dense rule-card shape of the older master prompt: operator catalogs
at the end, terse anti-hallucination bullets, and §13 reframed to
treat the opaque-id clause as the preferred form (preserves the
measure/segment's identity in the rendered notebook, propagates
definition updates, surfaces in lineage tooling). The portable-form
definition emit on `convert-measure-or-segment` — added in an earlier
step — stays in place as the compose-on-top fallback for the cases
where the agent wants to extend rather than reuse a measure/segment.

Tests cover the import / export happy paths and unknown-entity errors
for both models in `mp_test`, the construct round-trip with both
clause forms in `entity_details_test`, the new XML attribute in
`llm_shape_test`, and the E4 removal in `repair_test`. Full metabot
module + serialization / agent-lib modules run clean (933 tests, 4156
assertions, 0 failures).

* Declare metabot / models imports of :model/Measure and :model/Segment

The previous commit (4af895fba69) added measure / segment references in
the metabot tools layer (`construct.clj`, `entity_details.clj`) and the
models serdes resolver (`resolve/mp.clj`), tripping
`metabase.core.modules-test/model-boundaries-test`:

- `metabot` references `:model/Measure` and `:model/Segment` — add both
  to `:model-imports`.
- `models` references `:model/Measure` — add it to `:model-imports`
  (`:model/Segment` was already declared there).

No `:uses` changes needed: we only reference the model keywords via
toucan2 and serdes lookups, not by requiring the `measures` / `segments`
namespaces directly.

* Migrate two stragglers from mt/mbql-query to Lib

`mt/mbql-query` and friends were marked `:deprecated "0.61.0"` partway
through this branch's life with the guidance "use Lib to generate MBQL
queries instead of hand-rolling legacy MBQL queries in new tests". Two
test files added during the branch missed the migration — port them
the same way other on-branch tests do (e.g. `metric-query` in
`entity-details-test`):

  (lib/query mp (lib.metadata/table mp (mt/id :orders)))

and `(lib/limit ... 5)` for the `:limit 5` variant. Both files already
required `lib` and `lib.metadata`. Test behaviour unchanged — Cards'
`:dataset_query` and `format-transform-source`'s `:query` both accept
MBQL 5 directly.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Tighten agent resolver permissions and refuse unsafe repair splits

Five PR-blocking findings on the metabot-repr branch:

* N1 (ACL gap) — agent-side `export-query` defaulted to the unchecked
  `app-db-content-store`, so measure / segment / card refs surfaced
  inside an exported body bypassed `api/read-check`. Reachable from
  `entity_details`, the MCP `get_table` resource, and the ambient
  user-context "user is viewing this table" path (the last two pass
  `with-measures?: true` / `with-segments?: true` unconditionally).

  Introduce `metabase.metabot.tools.shared.content-store` with a
  `read-checked` wrapper that gates every one of the five ContentStore
  methods on `api/*current-user-id*`. When bound (HTTP / tool path) it
  applies `api/read-check` symmetrically across import- and export-
  direction lookups; when unbound (serdes import / background tasks /
  tests / REPL) the row passes through unchanged. `default-store` is
  `unchecked-app-db-content-store` wrapped with `read-checked`.

  `export-query` and `try-export-query` grow a 3-arity that accepts an
  explicit content-store, mirroring `resolve-query`'s shape. The three
  leaky callers — `entity_details.clj:63`, `entity_details.clj:394`,
  `user_context.clj:269` — now pass `shared.content-store/default-store`.

  `construct.clj`'s `permission-aware-content-store` reify (which had
  asymmetric posture — read-checked on import, unchecked on export)
  collapses onto the shared `default-store`. Same posture as before for
  the import direction, plus a now-correct read-check on export.

* M0 (design hygiene) — rename `app-db-content-store` →
  `unchecked-app-db-content-store` so every default-arity caller of
  `import-resolver` / `export-resolver` / `resolve-query` /
  `export-query` is visibly suspect on diff review. Four call sites
  updated. Future N1-shape bugs become much harder to miss.

* S1 (info-leak) — `unknown-table-ex-info` and `unknown-field-ex`
  enumerated schemas / sibling tables / FK-reachable target tables
  against the un-sandboxed `application-database-metadata-provider`,
  then relayed the message verbatim to the user via the `:agent-error?`
  flag. A sandboxed user could enumerate restricted schemas or table
  names just by hallucinating a `source-table:` / `field` reference.

  Strip all suggestion branches. The replacement messages echo only the
  rejected path and point the LLM at `entity_details` for self-
  correction. Drop the now-dead `tokenize` / `candidate-score` /
  `substring-table-candidates` / `fqn` helpers, the matching kondo
  requires (`clojure.set`, `clojure.string`, `metabase.util`), and the
  `outbound-fks-from-table` forward declare that supported the FK-hint
  walker. `outbound-fks-from-table` itself stays — `repair.clj`'s
  implicit-join logic still uses it.

* C2 (silent wrong data) — `aggregation-uuid->column-name` collided on
  same-head aggregations (two `["sum" …]` clauses both mapped to "sum"),
  so the cross-stage filter rewrite emitted `["field" {} "sum"]` for
  both uuids and the QP filtered the wrong column. Lib's real path goes
  through `unique-name-generator` ("sum" / "sum_2") which we can't
  replicate from a static table.

  Refuse the split when output names aren't unique. The existing
  `:post-agg-filter-needs-multi-stage` agent-error fires; the LLM gets
  a diagnostic that names the duplicate-column-name failure mode and
  suggests the `name` opts override as a remedy.

* C3 (silent intent destruction) — `split-post-agg-filter-stage`
  unconditionally moved `order-by` / `limit` / `fields` from stage 0 to
  the new stage 1 whenever a split fired. If those clauses referenced a
  pre-aggregation column that didn't survive the split, the user saw
  either an unhelpful "no matching field" deep in the QP or — worse —
  a different row set with no diagnostic.

  Refuse the split when any of those three trailing clauses is present.
  Throw a new `:post-agg-filter-with-trailing-clauses` agent-error that
  tells the LLM to author two stages explicitly. The relocation code is
  stripped — the auto-fix is now narrow: filters only.

* Reconcile post-merge tests with master and harden the construct_query API

The merge of origin/master into metabot-repr took "ours" on
`src/metabase/agent_api/api.clj` and `test/metabase/mcp/api_test.clj`
to preserve the branch's deletion of the legacy sexp/program pipeline.
That was correct for the source files but accidentally dropped 11
deftest blocks master had added in the same files. It also left in
three test subblocks that master had deleted (the cross-session
isolation assertions that no longer match master's user-scoped handle
design). This commit restores master's test coverage faithfully and
sands a few rough edges on the construct_query API surface.

* Test reconciliation (test/metabase/mcp/api_test.clj):

  - Reverted my temporary 3-arity `mcp-request` / `initialize!` arity
    overload. Restored master's separate `mcp-request-as` and
    `initialize-as!` helpers verbatim.

  - Imported master's `call-tool` MCP-spec contract assertion: every
    successful tool call whose tool declares an `outputSchema` must
    return `:structuredContent` (guards the regression where Claude
    Desktop got 500s on outputSchema/structuredContent mismatches).

  - Removed the three obsolete cross-session subblocks I had rewritten
    as cross-user. Master deletes them entirely; the cross-USER
    boundary is covered by `visualize-query-refuses-cross-user-handle-
    test` (imported below).

  - Imported 11 missing master deftests verbatim, except for 2 that
    referenced the legacy `:source`/`:operations` request shape
    (`construct-query-returns-bare-handle-test`, `execute-query-
    resolves-via-cross-session-fallback-test`). Those 2 were adapted
    to our representations `:query` shape — semantic intent preserved.
    Full set:
      - origin-validation-test
      - tools-list-all-tools-declare-required-hints-test
      - text-content-includes-structured-content-for-maps-test
      - tool-result-emits-structured-content-test
      - tools-list-strict-shape-test
      - tools-expose-output-schema-test
      - construct-query-returns-bare-handle-test
      - visualize-query-resolves-via-cross-session-fallback-test
      - execute-query-resolves-via-cross-session-fallback-test
      - render-drill-through-resolves-via-cross-session-fallback-test
      - visualize-query-refuses-cross-user-handle-test

  - Updated the `visualize_query` "resolves the query stored with a
    construct_query handle" subtest to also assert the prompt round-
    trips through the handle — restoring the semantic master pinned
    via "includes the prompt stored with a construct_query handle".

  - Updated the construct_query description-pointer assertion to
    check for `construct-query.md` (the MCP resource URI both master
    and this branch publish) instead of the file-path string
    `construct_notebook_query`.

  - Fixed the `read-handle` call site in `tools-call-smoke-test` to
    pass the 3rd `user-id` argument master added when making handle
    lookup user-scoped.

* Construct_query API hardening (src/metabase/agent_api/api.clj):

  - Restored the optional `:prompt` field on `::construct-query-
    request` (with master's 10000-char max). The migration to
    representations had inadvertently dropped it, breaking the prompt
    round-trip the MCP layer relies on for `visualize_query` to
    surface the user's original message back to the iframe for
    feedback submission.

  - Wire endpoint now echoes `:prompt` to the response, so the MCP
    body-transform (`make-store-construct-query-result`) can store
    it with the handle.

  - Relaxed `:query` on the wire body from `::lib.schema/external-
    query` to plain `:map`. The deep MBQL-shape validation already
    runs inside the representations pipeline (`repair` → `validate-
    query` → `resolve`), so the boundary check was redundant.
    Importantly, this also dodges master's new
    `tools-manifest/assert-optional-fields-nullable!` lint, which
    walks every map reachable from a tool's input schema and trips
    on `::query`'s `:lib/metadata` / `:database` / `:settings` /
    `:parameters` optional-non-nullable fields (those are correct in
    lib.schema for other reasons; we just don't want them reachable
    from the agent-API tool surface). The outer `:closed true` is
    preserved, so the legacy-envelope-rejection test still passes.

  - Folded a ~40-line condensed inline description back into the
    `construct_query` tool metadata — workflow, universal clause
    shape, top-level / per-stage keys, a minimal worked example, and
    common pitfalls. Strict MCP clients (ChatGPT, Claude Desktop)
    that don't eagerly fetch resources now get format guidance in
    `tools/list` output directly. Full grammar still lives in the
    `metabase://docs/construct-query.md` resource.

* MCP override shape (src/metabase/mcp/tools.clj):

  - Updated `construct-query-mcp-input-malli` and `query-mcp-input-
    malli` from the legacy `{:source, :operations}` shape (no longer
    valid against the wire after the representations migration) to
    the new `{:query <opaque-map>}` shape. The override uses an
    opaque `:map` for `:query` so the manifest's
    `assert-optional-fields-nullable!` lint stays clean.

* Adjusted one wire-validation test on the construct-query endpoint
  (test/metabase/agent_api/api_test.clj): the "non-aggregation in
  aggregation slot" test now asserts the `:aggregation-entry-not-
  aggregation` agent-error code that the repair pipeline emits,
  matching the new boundary where deep validation moved from the
  malli schema into the pipeline. Same HTTP status (400), more
  specific diagnostic.

Test sweep across all touched surfaces (mcp.api-test, agent-api.api-
test, full metabot suite, mcp.session-test / tools-test / resources-
test, all agent_lib representations suites, mp-test, core.modules-
test): 446 tests / 6173 assertions, 0 failures, 0 errors. Master
deftest set in `mcp/api_test.clj` is now fully restored — `comm -23`
of master's deftest names vs HEAD = empty. The 1-line stray
`describe_table.clj` unicode-literal change is from a pre-merge
branch commit and is intentional.

* Strip internal schema names from agent-facing strings; fix construct_query doc pointer

* Fix doc-pointer regression. My earlier reconciliation commit changed
  the `construct_query` tool description to point at
  `metabase://docs/construct-query.md` — matching master's pointer
  string but pointing at content that on this branch still describes
  the legacy program format (`resources/metabase/agent_api/construct_query.md`
  was never rewritten for the representations migration). Restored the
  branch's original pointer at the `construct_notebook_query` tool
  prompt, which IS rewritten for the new format. Test assertion
  updated to match.

* Strip internal Clojure schema concepts from LLM-facing strings.
  The agent only needs to know "Metabase MBQL 5 query as JSON" — names
  like `::lib.schema/external-query` or "portable representations JSON
  format" are implementation jargon and don't help the model produce
  better output. Cleaned in:
   - `mcp/tools.clj` `:tool/description` strings for `construct_query`
     / `query` inputs.
   - `agent_api/api.clj` `:tool/description` on `::construct-query-
     request`'s `:query` field, the `construct_query` tool description,
     and the `/v2/query` tool description.
   - `resources/metabot/prompts/tools/construct_notebook_query.md`'s
     opening line (dropped the schema-name parenthetical).
  Source comments and internal Clojure docstrings keep their technical
  vocabulary — those aren't agent-facing.

* Trimmed the verbose code comment I'd added above the inline tool-
  description string. No "NOT this / NOT that" framing.

* Consolidate construct-query reference to one canonical doc

`resources/metabase/agent_api/construct_query.md` (MCP-served at URI
`metabase://docs/construct-query.md`) still described the legacy
program format. `resources/metabot/prompts/tools/construct_notebook_query.md`
is the file the metabot agent actually loads as its `construct_notebook_query`
tool prompt, and it's already rewritten for the representations format.
Two files, two consumers, drifting content.

Make `construct_notebook_query.md` the single source of truth:

* Point the MCP resource registration's `:render-fn` at the metabot
  prompt file (`metabot/prompts/tools/construct_notebook_query.md`).
* Delete the now-redundant `resources/metabase/agent_api/construct_query.md`
  (and its empty parent dir).
* Update the MCP resource `:description` to drop "Program syntax"
  framing; the file describes the representations format now.
* Mirror the description in `src/metabase/mcp/README.md`.
* Add a single `# Construct Query Reference` heading line to
  `construct_notebook_query.md` — benign for the metabot LLM
  (it's already reading the file as a tool-instruction blob)
  and matches the heading the MCP `resources/read` test asserts.

External MCP clients (ChatGPT, Claude Desktop) fetching the resource
URI now get the same representations-format content the metabot agent
reads internally. Tool descriptions continue to point at the
`construct_notebook_query` tool name; both pointers (the MCP URI and
the tool-prompt path) resolve to the same content.

* Close residual S1 info leaks; catch typo'd stage keys at the agent boundary

Post-merge review surfaced three follow-up items on the agent surface
this commit addresses.

* S1 follow-up — implicit-join and ambiguous-table errors still leaked
  un-sandboxed metadata. The original S1 fix stripped suggestions from
  `unknown-table-ex-info` / `unknown-field-ex` in `mp.clj`, but three
  agent-error paths kept walking the un-sandboxed metadata provider and
  surfacing the results verbatim via `:agent-error? true`:

   - `representations/repair.clj` `:ambiguous-fk` enumerated full
     portable `[db schema table field]` paths for every FK column on
     reachable bridge tables — a sandboxed user could learn the FK
     column names on any FK-reachable table just by hallucinating a
     field reference.
   - `representations/repair.clj` `:ambiguous-fk-via-join` did the same
     for FK columns on joined tables. Alias names (which came from the
     LLM's own `joins:` input) stay surfaced; only the column portables
     are stripped.
   - `mp.clj`'s `:ambiguous-table` echoed `(juxt :schema :name :id)` of
     every matching candidate.

  All three now echo only the rejected input + a prose hint pointing
  the LLM at `entity_details` for the disambiguating columns it has
  permission to see. Existing tests that asserted candidate presence
  were inverted to assert absence (parity with the existing
  `unknown-table-ex-info` / `unknown-field-ex` tests).

* Stage-key typo catch — `lib.schema.mbql-stage/mbql` is not a closed
  map, so a stage with a misspelled key (`aggreagation` for
  `aggregation`) was silently dropped at resolve / `lib.normalize`
  time and the LLM never learned its intent was discarded. The full
  `validate-external-query` walk against `::lib.schema/external-query`
  produces opaque `humanized` output (function-predicate references)
  and pre-empts E1's friendly diagnostic for the aggregation-slot
  case, so a targeted closed-keys check is preferable.

  Added `representations/assert-known-stage-keys!` — walks each
  stage's top-level keys and throws `:unknown-stage-key` with the
  offending key name and the canonical valid-key set in the message.
  Wired into `metabot.tools.construct/execute-representations-query`
  right after the JSON→portable conversion. `validate-external-query`
  is kept as a building block with an updated docstring documenting
  why the boundary now uses the targeted check instead.

* Stale "YAML" wording — the wire flipped to JSON months ago but
  several anchor docstrings still said "YAML". Replaced in:
  `representations/repair.clj` module docstring + `repair` fn
  docstring; `representations/resolve.clj` `export-query` docstring;
  `metabot/tools/construct.clj` `resolve-database-id-from-first-stage`
  docstring. Comments only — no behavior change.

---------

Co-authored-by: Mihael Konjevic <konjevic@gmail.com>

.clj-kondo/config/modules/config.edn

@@ -158,8 +158,7 @@
   {:team    "Metabot"
    :api     #{metabase.agent-api.api
               metabase.agent-api.init}
-   :uses    #{agent-lib
-              api
+   :uses    #{api
               auth-identity
               dashboards
               events
@@ -175,12 +174,16 @@
    :model-imports #{:model/Card
                     :model/Dashboard
                     :model/DashboardCard
+                    :model/Table
                     :model/User}}
 
   agent-lib
   {:team "Metabot"
-   :api  #{metabase.agent-lib.core}
+   :api  #{metabase.agent-lib.representations
+           metabase.agent-lib.representations.repair
+           metabase.agent-lib.representations.resolve}
    :uses #{lib
+           models
            util}}
 
   analytics
@@ -1279,8 +1282,10 @@
                     :model/Database
                     :model/Field
                     :model/Glossary
+                    :model/Measure
                     :model/ModerationReview
                     :model/QueryTable
+                    :model/Segment
                     :model/Table
                     :model/Transform
                     :model/User}}
@@ -1341,6 +1346,7 @@
            metabase.models.resolution
            metabase.models.serialization
            metabase.models.serialization.resolve
+           metabase.models.serialization.resolve.mp
            metabase.models.util.spec-update
            metabase.models.visualization-settings}
    :uses #{api
@@ -1356,6 +1362,7 @@
                     :model/DashboardTab
                     :model/Database
                     :model/Field
+                    :model/Measure
                     :model/NativeQuerySnippet
                     :model/Segment
                     :model/Table

docs/api.json

@@ -3490,38 +3490,20 @@
         "required" : [ "name", "base_type", "display_name" ]
       },
       "metabase.agent-api.api.construct-query-request" : {
-        "description" : "Request body for /v2/construct-query. Same as program-request, with an optional prompt\n  capturing the user's original intent when a caller has one.",
+        "description" : "Request body for /v2/construct-query and the fresh-query branch of /v2/query.\n  A single `:query` key whose value is a JSON object matching\n  [[metabase.lib.schema/external-query]] — the canonical portable MBQL 5 wire format. The\n  query is fully self-describing: the database is derived from the first stage's\n  `source-table:` or `source-card:`, all field references are portable FKs\n  (`[<db-name>, <schema>, <table-name>, <column-name>]`), and there is no auxiliary\n  `source_entity` / `referenced_entities` envelope. See\n  `resources/metabot/prompts/tools/construct_notebook_query.md` for the full format reference\n  (including operators, joins, expressions, multi-stage queries, and FK conventions).\n\n  Closed map: any extra top-level keys (notably the legacy `source_entity` /\n  `referenced_entities` envelope from before the repr migration) are rejected with a 400 so\n  callers don't silently send fields the server ignores.",
         "type" : "object",
         "properties" : {
-          "operations" : {
-            "type" : "array",
-            "items" : {
-              "$ref" : "#/components/schemas/metabase.agent-lib.schema.top-level-op"
-            }
-          },
-          "prompt" : {
-            "type" : "string",
-            "minLength" : 1,
-            "maxLength" : 10000,
-            "description" : "The user's exact original message, when available. Pass it as-is without summarizing or rewriting."
-          },
-          "source" : {
-            "$ref" : "#/components/schemas/metabase.agent-lib.schema.source"
+          "query" : {
+            "$ref" : "#/components/schemas/metabase.lib.schema.external-query"
           }
         },
-        "required" : [ "source", "operations" ],
+        "required" : [ "query" ],
         "additionalProperties" : false
       },
       "metabase.agent-api.api.construct-query-response" : {
-        "description" : "Response containing a base64-encoded MBQL query and, when supplied, the original prompt for use with /v1/execute.",
+        "description" : "Response containing a base64-encoded MBQL query for use with /v1/execute.",
         "type" : "object",
         "properties" : {
-          "prompt" : {
-            "type" : "string",
-            "minLength" : 1,
-            "maxLength" : 10000,
-            "description" : "The user's exact original message, when available. Pass it as-is without summarizing or rewriting."
-          },
           "query" : {
             "type" : "string",
             "minLength" : 1
@@ -3968,12 +3950,8 @@
         },
         "required" : [ "id", "type", "name" ]
       },
-      "metabase.agent-api.api.program-request" : {
-        "description" : "Request body for /v2/query.\n  An agent-lib structured program with `:source` and `:operations`. The top-level\n  `:source` must reference a database entity (`table`, `card`, `dataset`, or\n  `metric`); `context` and nested `program` sources are rejected at the HTTP\n  boundary by [[evaluate-program-for-execution]] because they require an\n  in-process evaluation context.",
-        "$ref" : "#/components/schemas/metabase.agent-lib.schema.program"
-      },
       "metabase.agent-api.api.query-request" : {
-        "description" : "Request body for /v2/query. Accepts either a structured program or a continuation_token.",
+        "description" : "Request body for /v2/query. Accepts either a fresh-query payload (`{:query <external-query>}`,\n  same shape as /v2/construct-query) or a `:continuation_token` from a prior response.\n\n  Both branches are closed maps: extra top-level keys (e.g. the legacy\n  `source_entity` / `referenced_entities` envelope, or sending `:query` and\n  `:continuation_token` simultaneously) are rejected with a 400.",
         "oneOf" : [ {
           "type" : "object",
           "properties" : {
@@ -3982,9 +3960,10 @@
               "minLength" : 1
             }
           },
-          "required" : [ "continuation_token" ]
+          "required" : [ "continuation_token" ],
+          "additionalProperties" : false
         }, {
-          "$ref" : "#/components/schemas/metabase.agent-api.api.program-request"
+          "$ref" : "#/components/schemas/metabase.agent-api.api.construct-query-request"
         } ]
       },
       "metabase.agent-api.api.query-response" : {
@@ -4402,302 +4381,6 @@
         },
         "required" : [ "id", "type", "name", "display_name", "database_id", "database_engine", "fields" ]
       },
-      "metabase.agent-lib.schema.aggregation-ref-form" : {
-        "type" : "array",
-        "prefixItems" : [ {
-          "const" : "aggregation-ref"
-        }, {
-          "type" : "integer",
-          "minimum" : 0
-        } ]
-      },
-      "metabase.agent-lib.schema.expression-ref-form" : {
-        "type" : "array",
-        "prefixItems" : [ {
-          "const" : "expression-ref"
-        }, {
-          "$ref" : "#/components/schemas/metabase.agent-lib.schema.non-blank-string"
-        } ]
-      },
-      "metabase.agent-lib.schema.form" : {
-        "anyOf" : [ {
-          "$ref" : "#/components/schemas/metabase.agent-lib.schema.source-ref"
-        }, {
-          "$ref" : "#/components/schemas/metabase.agent-lib.schema.query-aware-ref"
-        }, {
-          "$ref" : "#/components/schemas/metabase.agent-lib.schema.nested-operator-form"
-        } ]
-      },
-      "metabase.agent-lib.schema.map-literal" : {
-        "type" : "object",
-        "additionalProperties" : {
-          "$ref" : "#/components/schemas/metabase.agent-lib.schema.node"
-        }
-      },
-      "metabase.agent-lib.schema.nested-operator-form" : {
-        "type" : "array",
-        "items" : {
-          "$ref" : "#/components/schemas/metabase.agent-lib.schema.node"
-        },
-        "minItems" : 1
-      },
-      "metabase.agent-lib.schema.node" : {
-        "anyOf" : [ {
-          "$ref" : "#/components/schemas/metabase.agent-lib.schema.scalar"
-        }, {
-          "$ref" : "#/components/schemas/metabase.agent-lib.schema.program-literal"
-        }, {
-          "$ref" : "#/components/schemas/metabase.agent-lib.schema.map-literal"
-        }, {
-          "$ref" : "#/components/schemas/metabase.agent-lib.schema.form"
-        }, {
-          "$ref" : "#/components/schemas/metabase.agent-lib.schema.vector-literal"
-        } ]
-      },
-      "metabase.agent-lib.schema.non-blank-string" : {
-        "type" : "string"
-      },
-      "metabase.agent-lib.schema.page" : {
-        "type" : "object",
-        "properties" : {
-          "items" : {
-            "description" : "value must be an integer greater than zero.",
-            "type" : "integer",
-            "minimum" : 1
-          },
-          "page" : {
-            "description" : "value must be an integer greater than zero.",
-            "type" : "integer",
-            "minimum" : 1
-          }
-        },
-        "required" : [ "page", "items" ],
-        "additionalProperties" : false
-      },
-      "metabase.agent-lib.schema.program" : {
-        "type" : "object",
-        "properties" : {
-          "operations" : {
-            "type" : "array",
-            "items" : {
-              "$ref" : "#/components/schemas/metabase.agent-lib.schema.top-level-op"
-            }
-          },
-          "source" : {
-            "$ref" : "#/components/schemas/metabase.agent-lib.schema.source"
-          }
-        },
-        "required" : [ "source", "operations" ],
-        "additionalProperties" : false
-      },
-      "metabase.agent-lib.schema.program-literal" : {
-        "type" : "object",
-        "properties" : {
-          "program" : {
-            "$ref" : "#/components/schemas/metabase.agent-lib.schema.program"
-          },
-          "type" : {
-            "const" : "program"
-          }
-        },
-        "required" : [ "type", "program" ],
-        "additionalProperties" : false
-      },
-      "metabase.agent-lib.schema.query-aware-ref" : {
-        "anyOf" : [ {
-          "$ref" : "#/components/schemas/metabase.agent-lib.schema.expression-ref-form"
-        }, {
-          "$ref" : "#/components/schemas/metabase.agent-lib.schema.aggregation-ref-form"
-        } ]
-      },
-      "metabase.agent-lib.schema.scalar" : {
-        "anyOf" : [ {
-          "type" : "null"
-        }, {
-          "type" : "string"
-        }, {
-          "type" : "number"
-        }, {
-          "type" : "boolean"
-        } ]
-      },
-      "metabase.agent-lib.schema.source" : {
-        "anyOf" : [ {
-          "type" : "object",
-          "properties" : {
-            "ref" : {
-              "const" : "source"
-            },
-            "type" : {
-              "const" : "context"
-            }
-          },
-          "required" : [ "type", "ref" ],
-          "additionalProperties" : false
-        }, {
-          "type" : "object",
-          "properties" : {
-            "id" : {
-              "description" : "value must be an integer greater than zero.",
-              "type" : "integer",
-              "minimum" : 1
-            },
-            "type" : {
-              "type" : "string",
-              "enum" : [ "table", "card", "dataset", "metric" ]
-            }
-          },
-          "required" : [ "type", "id" ],
-          "additionalProperties" : false
-        }, {
-          "$ref" : "#/components/schemas/metabase.agent-lib.schema.program-literal"
-        } ]
-      },
-      "metabase.agent-lib.schema.source-ref" : {
-        "anyOf" : [ {
-          "type" : "array",
-          "prefixItems" : [ {
-            "const" : "field"
-          }, {
-            "description" : "value must be an integer greater than zero.",
-            "type" : "integer",
-            "minimum" : 1
-          } ]
-        }, {
-          "type" : "array",
-          "prefixItems" : [ {
-            "const" : "table"
-          }, {
-            "description" : "value must be an integer greater than zero.",
-            "type" : "integer",
-            "minimum" : 1
-          } ]
-        }, {
-          "type" : "array",
-          "prefixItems" : [ {
-            "const" : "card"
-          }, {
-            "description" : "value must be an integer greater than zero.",
-            "type" : "integer",
-            "minimum" : 1
-          } ]
-        }, {
-          "type" : "array",
-          "prefixItems" : [ {
-            "const" : "metric"
-          }, {
-            "description" : "value must be an integer greater than zero.",
-            "type" : "integer",
-            "minimum" : 1
-          } ]
-        } ]
-      },
-      "metabase.agent-lib.schema.top-level-op" : {
-        "anyOf" : [ {
-          "type" : "array",
-          "prefixItems" : [ {
-            "const" : "filter"
-          }, {
-            "$ref" : "#/components/schemas/metabase.agent-lib.schema.form"
-          } ]
-        }, {
-          "type" : "array",
-          "prefixItems" : [ {
-            "const" : "aggregate"
-          }, {
-            "$ref" : "#/components/schemas/metabase.agent-lib.schema.form"
-          } ]
-        }, {
-          "type" : "array",
-          "prefixItems" : [ {
-            "const" : "breakout"
-          }, {
-            "$ref" : "#/components/schemas/metabase.agent-lib.schema.form"
-          } ]
-        }, {
-          "type" : "array",
-          "prefixItems" : [ {
-            "const" : "with-fields"
-          }, {
-            "type" : "array",
-            "items" : {
-              "$ref" : "#/components/schemas/metabase.agent-lib.schema.form"
-            }
-          } ]
-        }, {
-          "type" : "array",
-          "prefixItems" : [ {
-            "const" : "limit"
-          }, {
-            "description" : "value must be an integer greater than zero.",
-            "type" : "integer",
-            "minimum" : 1
-          } ]
-        }, {
-          "type" : "array",
-          "prefixItems" : [ {
-            "const" : "expression"
-          }, {
-            "$ref" : "#/components/schemas/metabase.agent-lib.schema.non-blank-string"
-          }, {
-            "$ref" : "#/components/schemas/metabase.agent-lib.schema.form"
-          } ]
-        }, {
-          "type" : "array",
-          "prefixItems" : [ {
-            "const" : "join"
-          }, {
-            "$ref" : "#/components/schemas/metabase.agent-lib.schema.form"
-          } ]
-        }, {
-          "anyOf" : [ {
-            "type" : "array",
-            "prefixItems" : [ {
-              "const" : "order-by"
-            }, {
-              "$ref" : "#/components/schemas/metabase.agent-lib.schema.form"
-            } ]
-          }, {
-            "type" : "array",
-            "prefixItems" : [ {
-              "const" : "order-by"
-            }, {
-              "$ref" : "#/components/schemas/metabase.agent-lib.schema.form"
-            }, {
-              "type" : "string",
-              "enum" : [ "asc", "desc" ]
-            } ]
-          } ]
-        }, {
-          "type" : "array",
-          "prefixItems" : [ {
-            "const" : "append-stage"
-          } ]
-        }, {
-          "type" : "array",
-          "prefixItems" : [ {
-            "const" : "drop-stage"
-          } ]
-        }, {
-          "type" : "array",
-          "prefixItems" : [ {
-            "const" : "drop-empty-stages"
-          } ]
-        }, {
-          "type" : "array",
-          "prefixItems" : [ {
-            "const" : "with-page"
-          }, {
-            "$ref" : "#/components/schemas/metabase.agent-lib.schema.page"
-          } ]
-        } ]
-      },
-      "metabase.agent-lib.schema.vector-literal" : {
-        "type" : "array",
-        "items" : {
-          "$ref" : "#/components/schemas/metabase.agent-lib.schema.node"
-        }
-      },
       "metabase.analyze.query-results.ResultsMetadata" : {
         "description" : "value must be an array of valid results column metadata maps.",
         "oneOf" : [ {
@@ -6592,7 +6275,8 @@
           "lib/uuid" : {
             "type" : "string",
             "minLength" : 36,
-            "maxLength" : 36
+            "maxLength" : 36,
+            "$ref" : "#/components/schemas/metabase.lib.schema.common.uuid"
           },
           "name" : {
             "oneOf" : [ {
@@ -8885,6 +8569,10 @@
       "metabase.lib.schema.expression.window..offset.n" : {
         "type" : "integer"
       },
+      "metabase.lib.schema.external-query" : {
+        "description" : "Schema for \"External MBQL\" 5 query.",
+        "$ref" : "#/components/schemas/metabase.lib.schema.query"
+      },
       "metabase.lib.schema.fields" : {
         "allOf" : [ {
           "type" : "array",
@@ -8982,6 +8670,13 @@
         "description" : "The ID used to signify that a database is 'virtual' rather than physical.\n\n   A fake integer ID is used so as to minimize the number of changes that need to be made on the frontend -- by using\n   something that would otherwise be a legal ID, *nothing* need change there, and the frontend can query against this\n   'database' none the wiser. (This integer ID is negative which means it will never conflict with a *real* database\n   ID.)\n\n   This ID acts as a sort of flag. The relevant places in the middleware can check whether the DB we're querying is\n   this 'virtual' database and take the appropriate actions.",
         "const" : -1337
       },
+      "metabase.lib.schema.id.schema" : {
+        "anyOf" : [ {
+          "type" : "null"
+        }, {
+          "type" : "string"
+        } ]
+      },
       "metabase.lib.schema.id.segment" : {
         "description" : "Valid legacy Segment ID",
         "type" : "integer",
@@ -15380,7 +15075,7 @@
       "post" : {
         "operationId" : "post-api-agent-v2-construct-query",
         "summary" : "POST /api/agent/v2/construct-query",
-        "description" : "Construct an MBQL query from a structured agent-lib program.\n\n  The body is the program itself: a JSON object with `source` (identifying the\n  table/card/dataset/metric to query), `operations` (an array of operator\n  tuples), and `prompt` (the user's original request). Returns a base64-encoded\n  MBQL query that can be executed via /v1/execute. See the agent_api reference\n  for the full program syntax.",
+        "description" : "Construct an MBQL query from a portable MBQL 5 representations JSON payload.\n\n  The body is `{\"query\": <external-query>}` where `<external-query>` is a JSON object\n  matching `::lib.schema/external-query` — see the `construct_notebook_query` tool\n  documentation for the format reference. Returns a base64-encoded MBQL query that can be\n  executed via /v1/execute or paginated via /v2/query.",
         "parameters" : [ ],
         "responses" : {
           "2XX" : {
@@ -15416,7 +15111,7 @@
       "post" : {
         "operationId" : "post-api-agent-v2-query",
         "summary" : "POST /api/agent/v2/query",
-        "description" : "Execute a structured program and stream the results, with continuation-token pagination.\n\n  Accepts either a program (same shape as /v2/construct-query) or a\n  `continuation_token` from a previous response. Returns results with column\n  metadata and an optional `continuation_token` for fetching the next page.",
+        "description" : "Execute a portable MBQL 5 representations JSON query and stream the results, with\n  continuation-token pagination.\n\n  Accepts either a JSON body (same shape as /v2/construct-query) or a `continuation_token`\n  from a previous response. Returns results with column metadata and an optional\n  `continuation_token` for fetching the next page.",
         "parameters" : [ ],
         "responses" : {
           "2XX" : {

resources/metabase/agent_api/construct_query.md

@@ -1,144 +0,0 @@
-# Construct Query Reference
-
-Construct a Metabase MBQL query from a structured program. The body is the program itself — no envelope — shaped:
-
-```
-{"source": {...}, "operations": [...]}
-```
-
-The `prompt` field is optional for Agent API callers. MCP clients should include it whenever they have the user's message; pass the original message as-is and do not summarize, rewrite, or infer a different prompt.
-
-In MCP, this returns `{"query_handle": "<uuid>"}` — pass `query_handle` to `execute_query` or `visualize_query`. The HTTP Agent API response contains `{"query": "<base64>"}` and echoes `prompt` only when supplied.
-
-IMPORTANT: field IDs must come from entity-detail endpoints (`/v1/table/{id}`, `/v1/metric/{id}`). Do not invent IDs. The backend repairs minor mistakes (aliases, casing, over-wrapping) before validation, but the canonical names below always work.
-
-## Workflow
-
-1. Use `search` and the entity-detail tools (`get_table`, `get_metric`) to find the table/metric/model and its fields.
-2. Call `construct_query` with the program. Include the user's original `prompt` whenever available. In MCP, you get back `{"query_handle": "<uuid>"}`.
-3. Pass `query_handle` to the follow-up `execute_query` or `visualize_query` call.
-
-Never embed IDs you did not read from a metadata endpoint — invented IDs will fail at execution.
-
-## Source
-
-One of `{"type": T, "id": N}`:
-
-- `table` — a database table
-- `card` — a saved question
-- `dataset` — a model (model card id)
-- `metric` — a metric (supplies its own aggregation and time dimension; extra aggregates usually unnecessary)
-
-## Top-level operations (applied in order)
-
-Each operation is `["op", arg, ...]`:
-
-- `["filter", clause]` — add a filter
-- `["aggregate", agg-clause]` — add an aggregation
-- `["breakout", ref-or-bucketed]` — add a grouping dimension
-- `["expression", "Name", expr]` — define a named computed column (reference later with `expression-ref`)
-- `["with-fields", [refs...]]` — restrict returned columns
-- `["order-by", ref]` or `["order-by", ref, "asc"|"desc"]` — sort
-- `["limit", N]` — cap rows
-- `["join", join-clause]` — join another entity
-- `["append-stage"]` — start a new query stage (needed to filter on aggregated values)
-- `["with-page", {"page": N, "items": M}]` — paginate
-
-## References (used as arguments inside operations)
-
-- `["field", N]` — database field by id. Do NOT put options in a third slot (no `["field", id, {...}]`); wrap instead
-- `["expression-ref", "Name"]` — a named expression defined earlier
-- `["aggregation-ref", N]` — the Nth `aggregate` defined earlier (0-based). REQUIRED when sorting by an aggregated value
-- `["measure", N]` — a pre-defined measure on the source entity
-- `["with-temporal-bucket", ref, unit]` — temporal bucketing. `unit` is one of: `minute` `hour` `day` `week` `month` `quarter` `year`. Also `day-of-week`, `hour-of-day`, etc. (extraction aliases)
-- `["with-binning", ref, {"strategy": "num-bins"|"bin-width"|"default", ...}]` — numeric binning. E.g. `{"strategy": "num-bins", "num-bins": 10}`
-
-## Filter operators
-
-`=`, `!=`, `<`, `<=`, `>`, `>=`, `between`, `in`, `not-in`, `is-null`, `not-null`, `is-empty`, `not-empty`, `contains`, `does-not-contain`, `starts-with`, `ends-with`, `time-interval`, `and`, `or`, `not`, `segment`.
-
-Examples: `["=", ["field", 101], "active"]`, `["between", ["field", 305], "2024-01-01", "2024-12-31"]`, `["in", ["field", 302], [10, 20, 30]]`, `["time-interval", ["field", 305], -7, "day"]`.
-
-## Aggregation operators
-
-`count`, `sum`, `avg`, `min`, `max`, `distinct`, `median`, `stddev`, `var`, `percentile`, `count-where`, `sum-where`, `distinct-where`, `share`, `cum-count`, `cum-sum`.
-
-Examples: `["count"]`, `["sum", ["field", 302]]`, `["count-where", ["=", ["field", 101], "completed"]]`.
-
-## Temporal helpers (for use in `expression` or as grouping)
-
-`get-year`, `get-quarter`, `get-month`, `get-week`, `get-day`, `get-day-of-week`, `get-hour`, `get-minute`, `datetime-add`, `datetime-diff`, `datetime-subtract`, `now`, `today`, `relative-datetime`, `absolute-datetime`, `with-temporal-bucket`, `convert-timezone`.
-
-## Examples
-
-Top 5 customers by revenue:
-
-```
-{"source": {"type": "table", "id": 42},
- "operations": [["aggregate", ["sum", ["field", 302]]],
-                ["breakout", ["field", 101]],
-                ["order-by", ["aggregation-ref", 0], "desc"],
-                ["limit", 5]]}
-```
-
-Monthly revenue from a metric (metric supplies the aggregation):
-
-```
-{"source": {"type": "metric", "id": 10},
- "operations": [["breakout", ["with-temporal-bucket", ["field", 305], "month"]],
-                ["order-by", ["with-temporal-bucket", ["field", 305], "month"], "asc"]]}
-```
-
-Filter on an aggregated value (requires `append-stage`):
-
-```
-{"source": {"type": "table", "id": 42},
- "operations": [["aggregate", ["sum", ["field", 302]]],
-                ["breakout", ["field", 101]],
-                ["append-stage"],
-                ["filter", [">", ["aggregation-ref", 0], 1000]]]}
-```
-
-Named expression referenced later:
-
-```
-{"source": {"type": "table", "id": 42},
- "operations": [["expression", "Discount", ["-", ["field", 302], ["field", 303]]],
-                ["aggregate", ["sum", ["expression-ref", "Discount"]]]]}
-```
-
-Previous-period comparison with `offset` (stay in the SAME stage — do NOT add `append-stage`):
-
-```
-{"source": {"type": "table", "id": 42},
- "operations": [["aggregate", ["sum", ["field", 302]]],
-                ["aggregate", ["offset", ["sum", ["field", 302]], -1]],
-                ["breakout", ["with-temporal-bucket", ["field", 305], "month"]]]}
-```
-
-## Rules & common pitfalls
-
-Stage boundaries (most common source of errors):
-
-- Filtering on an aggregated value REQUIRES `append-stage` between the aggregate/breakout and the filter (see the "filter on aggregated value" example). Without it, `aggregation-ref` resolution fails in the same stage.
-- Defining an `expression` that uses `aggregation-ref` also REQUIRES `append-stage` first.
-- EXCEPTION: `offset` (previous-period comparison) stays in the same stage as its base aggregation and breakout — do NOT add `append-stage` for it.
-
-Refs & shapes:
-
-- Aggregation helpers take field refs, not bare IDs: `["sum", ["field", 201]]`, never `["sum", 201]`.
-- To sort by an aggregated value, use `["aggregation-ref", N]` — not the original expression.
-- Do NOT put options in a third slot of `field` (no `["field", id, {...}]`). Wrap instead: `["with-temporal-bucket", ["field", id], "month"]` or `["with-binning", ["field", id], {...}]`.
-- `case` takes `[[condition, value], ...]` branches and an optional bare fallback as the THIRD arg — do not wrap it as `{"default": ...}`. Omit the third arg when there is no fallback.
-- JSON objects appear only where a helper explicitly calls for one (e.g. `with-page`, `with-binning`). Everywhere else, use operator tuples.
-
-Joins & related tables:
-
-- If the source table's detail response already surfaces a related table's fields, use those field refs directly — no explicit join needed.
-- Reach for `join` + `with-join-conditions` only for custom aliases, self-joins, explicit joined-field selection, or when direct related-field refs are unavailable.
-- If an explicit join returns a permission error, the underlying table is not accessible — surface the error, do not retry with implicit refs.
-
-Metrics & dates:
-
-- A `metric` source already provides its own aggregation and time dimension. Add only the additional breakouts/filters you need.
-- When the user asks for an exact year (e.g. 2024), use `["=", ["field", year_field], 2024]` or a `between` with explicit dates — not relative filters like `time-interval`.

resources/metabot/prompts/llm_shape.selmer

@@ -1,19 +1,21 @@
 {% if is_measure %}
-<measure measure_id="{{measure_id}}" name="{{measure_name}}" display_name="{{measure_display_name}}">{% if measure_description %}
-{{measure_description}}{% endif %}{% if measure_definition_description %}
-Definition: {{measure_definition_description}}{% endif %}{% if measure_definition %}
-<definition>{{measure_definition}}</definition>{% endif %}
+<measure measure_id="{{measure_id}}" name="{{measure_name}}" display_name="{{measure_display_name}}"{% if measure_portable_entity_id %} portable_entity_id="{{measure_portable_entity_id}}"{% endif %}>{% if measure_description %}
+{{measure_description}}{% endif %}{% if measure_definition %}
+<definition>
+{{measure_definition|safe}}
+</definition>{% endif %}
 </measure>
 {% endif %}
 {% if is_segment %}
-<segment segment_id="{{segment_id}}" name="{{segment_name}}" display_name="{{segment_display_name}}">{% if segment_description %}
-{{segment_description}}{% endif %}{% if segment_definition_description %}
-Definition: {{segment_definition_description}}{% endif %}{% if segment_definition %}
-<definition>{{segment_definition}}</definition>{% endif %}
+<segment id="{{segment_id}}" name="{{segment_name}}" display_name="{{segment_display_name}}"{% if segment_portable_entity_id %} portable_entity_id="{{segment_portable_entity_id}}"{% endif %}>{% if segment_description %}
+{{segment_description}}{% endif %}{% if segment_definition %}
+<definition>
+{{segment_definition|safe}}
+</definition>{% endif %}
 </segment>
 {% endif %}
 {% if is_field %}
-<field id="{{field_id}}" name="{{field_name_quoted|safe}}" display_name="{{field_display_name}}" type="{{field_base_type}}" database_type="{{field_database_type}}">{% if field_description %}
+<field id="{{field_id}}" name="{{field_name_quoted|safe}}" display_name="{{field_display_name}}" type="{{field_base_type}}" database_type="{{field_database_type}}"{% if field_fk_target_fqn %} fk_target_fully_qualified_name="{{field_fk_target_fqn}}"{% endif %}>{% if field_description %}
 ## Description
 {{field_description}}{% endif %}</field>
 {% endif %}
@@ -22,11 +24,11 @@ Definition: {{segment_definition_description}}{% endif %}{% if segment_definitio
 <description>{{collection_description}}</description>{% endif %}</collection>
 {% endif %}
 {% if is_related_table %}
-<related-table{% if related_table_id %} id="{{related_table_id}}"{% endif %} name="{{related_table_name}}"{% if related_table_related_by %} related_by="{{related_table_related_by}}"{% endif %} fully_qualified_name="{{related_table_fqn}}">{% if related_table_fields_xml %}
+<related-table{% if related_table_id %} id="{{related_table_id}}"{% endif %} name="{{related_table_name}}"{% if related_table_related_by %} related_by="{{related_table_related_by}}"{% endif %}{% if related_table_database_name %} database_name="{{related_table_database_name}}"{% endif %} fully_qualified_name="{{related_table_fqn}}">{% if related_table_fields_xml %}
 {{related_table_fields_xml|safe}}{% endif %}</related-table>
 {% endif %}
 {% if is_metric %}
-<metric id="{{metric_id}}", name="{{metric_name}}" is_verified="{{metric_verified}}">
+<metric id="{{metric_id}}", name="{{metric_name}}" is_verified="{{metric_verified}}"{% if metric_database_name %} database_name="{{metric_database_name}}"{% endif %}{% if metric_base_table_fqn %} base_table_fully_qualified_name="{{metric_base_table_fqn}}"{% endif %}{% if metric_portable_entity_id %} portable_entity_id="{{metric_portable_entity_id}}"{% endif %}>
 {% if metric_collection_xml %}
 The metric is stored in the following collection:
 {{metric_collection_xml|safe}}
@@ -49,7 +51,7 @@ This will help avoid errors due to unexpected data types or formats.
 </metric>
 {% endif %}
 {% if is_table %}
-<table id="{{table_id}}", name="{{table_name}}" database_id="{{table_database_id}}" database_engine="{{table_database_engine}}" fully_qualified_name="{{table_fqn}}">
+<table id="{{table_id}}", name="{{table_name}}" database_id="{{table_database_id}}"{% if table_database_name %} database_name="{{table_database_name}}"{% endif %} database_engine="{{table_database_engine}}" fully_qualified_name="{{table_fqn}}">
 
 ### Description
 {{table_description}}
@@ -93,10 +95,19 @@ This will help avoid errors due to unexpected data types or formats.
 </table>
 {% endif %}
 {% if is_model %}
-<metabase-model id="{{model_id}}" name="{{model_name}}" is_verified="{{model_verified}}" database_id="{{model_database_id}}" database_engine="{{model_database_engine}}" fully_qualified_name="{{model_fqn}}">
+<metabase-model id="{{model_id}}" name="{{model_name}}" is_verified="{{model_verified}}" database_id="{{model_database_id}}"{% if model_database_name %} database_name="{{model_database_name}}"{% endif %} database_engine="{{model_database_engine}}" fully_qualified_name="{{model_fqn}}"{% if model_portable_entity_id %} portable_entity_id="{{model_portable_entity_id}}"{% endif %}>
 
 ### Description
 {{model_description}}
+{% if model_query_json %}
+
+### Saved query
+The model's saved query, in the same canonical portable representations JSON format the
+`construct_notebook_query` tool consumes:
+<query>
+{{model_query_json|safe}}
+</query>
+{% endif %}
 {% if model_fields_xml %}
 
 ### Fields
@@ -175,7 +186,7 @@ Chart link: metabase://chart/{{chart_id}}
 </chart>
 {% endif %}
 {% if is_question %}
-<metabase_question id="{{question_id}}" is_verified="{{question_verified}}"{% if question_display_type %} display_type="{{question_display_type}}"{% endif %}>
+<metabase_question id="{{question_id}}" is_verified="{{question_verified}}"{% if question_display_type %} display_type="{{question_display_type}}"{% endif %}{% if question_portable_entity_id %} portable_entity_id="{{question_portable_entity_id}}"{% endif %}>
 <name>{{question_name}}</name>
 {% if question_description %}
 <description>{{question_description}}</description>
@@ -184,17 +195,17 @@ Chart link: metabase://chart/{{chart_id}}
 The question is stored in the following collection:
 {{question_collection_xml|safe}}
 {% endif %}
+{% if question_query_json %}
+The question's saved query, in the same canonical portable representations JSON format the
+`construct_notebook_query` tool consumes:
+<query>
+{{question_query_json|safe}}
+</query>
+{% endif %}
 {% if question_visualization_xml %}
 The question is visualized as follows:
 {{question_visualization_xml|safe}}
 {% endif %}
-{% if question_fields_xml %}
-
-### Fields
-
-The following fields are returned by this question.
-{{question_fields_xml|safe}}
-{% endif %}
 </metabase_question>
 {% endif %}
 {% if is_text_card %}
@@ -262,7 +273,7 @@ or jargon that may be relevant to the user's questions.
 </user>
 {% endif %}
 {% if is_search_result %}
-<{{search_tag_name}} id="{{search_id}}" name="{{search_name}}"{% if search_has_verified %} is_verified="{{search_verified}}"{% endif %}{% if search_database_id %} database_id="{{search_database_id}}"{% endif %}{% if search_database_engine %} database_engine="{{search_database_engine}}"{% endif %}{% if search_fqn %} fully_qualified_name="{{search_fqn}}"{% endif %}>
+<{{search_tag_name}} id="{{search_id}}" name="{{search_name}}"{% if search_has_verified %} is_verified="{{search_verified}}"{% endif %}{% if search_database_id %} database_id="{{search_database_id}}"{% endif %}{% if search_database_name %} database_name="{{search_database_name}}"{% endif %}{% if search_database_engine %} database_engine="{{search_database_engine}}"{% endif %}{% if search_fqn %} fully_qualified_name="{{search_fqn}}"{% endif %}{% if search_base_table_fqn %} base_table_fully_qualified_name="{{search_base_table_fqn}}"{% endif %}{% if search_portable_entity_id %} portable_entity_id="{{search_portable_entity_id}}"{% endif %}>
 {% if search_description %}
 {{search_description}}
 {% endif %}

resources/metabot/prompts/tools/construct_notebook_query.md

@@ -1,180 +1,388 @@
-Use this tool to construct a notebook query directly from the structured MBQL tuple AST. MBQL (Metabase Query Language) is Metabase's structured, JSON-friendly query AST — this tool builds a query directly in that AST instead of writing SQL.
+# Construct Query Reference
 
-Return a payload with:
-- `source_entity`: `{"type":"table"|"model"|"question"|"metric","id":123}`
-- `referenced_entities`: optional additional context entities in the same shape
-- `program`: structured MBQL program
-- `visualization`: optional chart choice like `{"chart_type":"bar"}`
+Construct a Metabase MBQL 5 query as a JSON object describing the query shape. Metabase validates, repairs, and resolves it.
 
-The program must use the structured source/context pattern:
+Return:
+- `query`: a JSON **object** (never a quoted string). The target database is inferred from the first stage's `source-table` (or `source-card`) — use the **exact** database name reported by `entity_details` / metadata tools.
+- `visualization`: optional `{"chart_type": "bar"}` (sibling of `query`, never embedded inside it).
+
+## Minimal example — count of orders by month
 
 ```json
-{
-  "source": { "type": "context", "ref": "source" },
-  "operations": [
-    ["filter", ["between", ["field", 2058], "2024-01-01", "2024-12-31"]],
-    ["aggregate", ["sum", ["field", 2056]]],
-    ["breakout", ["with-temporal-bucket", ["field", 2058], "month"]]
-  ]
-}
+{"lib/type": "mbql/query",
+ "stages": [{"lib/type": "mbql.stage/mbql",
+             "source-table": ["Sample Database", "PUBLIC", "ORDERS"],
+             "aggregation": [["count", {}]],
+             "breakout": [["field", {"temporal-unit": "month"},
+                           ["Sample Database", "PUBLIC", "ORDERS", "CREATED_AT"]]]}]}
 ```
 
-Rules:
-- Use real numeric ids only — the integer ids returned by the table/metric detail endpoints.
-- Use `["field", id]`, `["table", id]`, `["card", id]`, `["metric", id]`, and `["measure", id]` as needed.
-- `program` must be a JSON object, not a quoted JSON string.
-- Each item in `operations` must contain exactly one top-level transform.
-  - Good: `[["aggregate", ["count"]], ["aggregate", ["sum", ["field", 301]]]]`
-  - Bad: `[["aggregate", ["count"], ["sum", ["field", 301]]]]`
-- Use `["relative-datetime", n, "unit"]` for relative temporal values.
-- `source_entity` chooses the primary source:
-  - `table` -> table
-  - `model` -> model/dataset
-  - `question` -> saved question
-  - `metric` -> metric
-- Use `{ "type": "context", "ref": "source" }` as the program source for the selected primary source.
-- Use nested `{ "type": "program", "program": { ... } }` only when you truly need a derived query as a source or join target.
+Every clause is `["op", {}, ...args]` with a mandatory `{}` options map at position 1; every field reference uses a 4-segment portable FK in the last slot. These are the two most-violated rules.
 
-Common operators:
-- `["filter", clause]`
-- `["aggregate", aggregation]`
-- `["breakout", breakout-form]`
-- `["with-fields", [field-or-ref, ...]]`
-- `["limit", 10]`
-- `["expression", "Name", expr]`
-- `["join", join-clause]`
-- `["order-by", orderable]`
-- `["order-by", orderable, "desc"]`
-- `["append-stage"]`
-- `["drop-stage"]`
-- `["drop-empty-stages"]`
-- `["with-page", {"page": 2, "items": 25}]`
+## Universal clause shape
 
-Do not put visualization inside `program.operations`.
-- Good: top-level `"visualization": {"chart_type": "pie"}`
-- Bad: `["visualization", {"chart_type": "pie"}]`
+Every operation is `["<operator>", {options}, ...args]`. The options map is **always present**, even when empty.
 
-Breakout and ordering rules:
-- Do not breakout on the same underlying field twice in one stage.
-  - If you breakout by `["with-temporal-bucket", ["field", 2058], "month"]`, do not also breakout by raw `["field", 2058]`.
-- If you want the normal ascending order for a breakout field, you can usually omit an explicit `order-by`.
-  - Metabase already adds implicit ascending ordering for breakout fields.
-  - Add an explicit `order-by` for a breakout only when you need a different direction or a different sort precedence.
+- Good: `["count", {}]`, `["field", {}, ["DB", "SCH", "TBL", "COL"]]`, `["sum", {}, <expr>]`
+- Bad: `["count"]`, `["field", ["DB", "SCH", "TBL", "COL"]]`
 
-Measures and segments:
-- Prefer `["measure", id]` when a model/table exposes an official measure that matches the requested aggregation.
-- Prefer `["segment", id]` when a model/table exposes an official segment that matches the requested filter.
+The pipeline repairs missing `{}` slots, but write them — your output should match what later inspection will show.
 
-Post-aggregation logic:
-- For filters or expressions that reference `["aggregation-ref", index]`, add `["append-stage"]` first.
-- Example:
+## Top-level query and stage shape
+
+Top level:
+- `"lib/type": "mbql/query"` — required marker.
+- `"stages": [...]` — at least one stage.
+
+Stage (`"lib/type": "mbql.stage/mbql"` — required marker):
+- `source-table` **or** `source-card` — exactly one, **first stage only**. Later stages take the previous stage's output implicitly.
+- Optional: `filters`, `aggregation`, `breakout`, `expressions`, `fields`, `joins`, `order-by`, `limit`, `page`.
+
+There is no top-level `database:` field in the LLM contract — the database is derived from the source.
+
+## Field references
 
 ```json
-{
-  "source": { "type": "context", "ref": "source" },
-  "operations": [
-    ["aggregate", ["sum", ["field", 301]]],
-    ["breakout", ["field", 401]],
-    ["append-stage"],
-    ["filter", [">", ["aggregation-ref", 0], 1000]]
-  ]
-}
+["field", {}, ["<db-name>", "<schema-or-null>", "<table-name>", "<field-name>"]]
 ```
 
-Grouped ratio example:
+The third slot is the **portable field FK** — a 4+ element string array. Schemaless databases (MongoDB, etc.) use `null` in the schema slot: `["Mongo", null, "orders", "created_at"]`. JSON-unfolded fields append extra segments: `["DB", "SCH", "TBL", "PARENT", "CHILD"]`.
+
+Inside later stages, refer to a column produced by the previous stage by **string name** instead of a portable FK: `["field", {}, "count"]`, `["field", {}, "PRODUCT_ID"]`.
+
+Field options (all optional):
+- `temporal-unit` — bucket a date/time field, e.g. `"month"`, `"day"`, `"hour"`. Full list in §Operator catalogs.
+- `join-alias` — required on every field reference that lives inside an explicit join.
+- `source-field` — portable FK of the FK column on the source table; only needed when the implicit-join auto-fill is ambiguous.
+- `source-field-name` — name of the FK column when it comes from a previous stage's output (rare; not auto-filled).
+- `source-field-join-alias` — explicit-join alias the FK column belongs to (usually auto-filled).
+- `binning` — bucket a numeric field.
+
+`base-type` is auto-filled on cross-stage refs — don't write it.
+
+## Per-clause examples
+
+Filter (comparison + boolean combination):
 
 ```json
-{
-  "source": { "type": "context", "ref": "source" },
-  "operations": [
-    ["aggregate", ["sum", ["field", 301]]],
-    ["aggregate", ["sum", ["field", 302]]],
-    ["breakout", ["field", 401]],
-    ["append-stage"],
-    ["expression", "Ratio", ["/", ["aggregation-ref", 0], ["aggregation-ref", 1]]],
-    ["order-by", ["desc", ["expression-ref", "Ratio"]]],
-    ["limit", 10]
-  ]
-}
+"filters": [["and", {},
+  [">", {}, ["field", {}, ["Sample Database", "PUBLIC", "ORDERS", "TOTAL"]], 100],
+  ["=", {}, ["field", {}, ["Sample Database", "PUBLIC", "ORDERS", "STATUS"]], "paid"]]]
 ```
 
-Joins:
-- If a surrounding/related table is already shown in context for the source table, first try using that table's field ids directly without an explicit join.
-- Add an explicit join when you need custom join behavior, a join alias, explicit joined-field selection, self-joins, or direct related-field refs do not work.
-- Direct related-field refs are the default for one-hop related tables.
-- If an explicit join returns a permission error, the underlying table is not accessible — surface the error instead of retrying with implicit refs (they hit the same permission check).
-- Example:
+Aggregation (on a field, plus `count`):
 
 ```json
-{
-  "source": { "type": "context", "ref": "source" },
-  "operations": [
-    ["join", ["with-join-conditions",
-      ["join-clause", ["table", 42]],
-      [["=", ["field", 101], ["field", 202]]]
-    ]],
-    ["aggregate", ["sum", ["field", 301]]],
-    ["breakout", ["field", 202]]
-  ]
-}
+"aggregation": [["sum", {}, ["field", {}, ["Sample Database", "PUBLIC", "ORDERS", "TOTAL"]]],
+                ["count", {}]]
 ```
 
-Field projection rules:
-- `with-fields` accepts returned columns and refs such as `["field", id]`, `["expression-ref", "Name"]`, and `["aggregation-ref", 0]`.
-- Do not define `["expression", "Name", expr]` inside `with-fields`.
-  - Define the expression first as its own top-level operation, then project `["expression-ref", "Name"]`.
-- For multi-value categorical filters, use `in` or `or`, not `=` with an array literal.
-  - Good: `["filter", ["in", ["field", 2492], ["authorized", "pending"]]]`
-  - Bad: `["filter", ["=", ["field", 2492], ["authorized", "pending"]]]`
-- For temporal extraction, use helpers such as `["get-month", ["field", 2058]]` and `["get-day-of-week", ["field", 2058]]`.
-  - Use `["get-hour", ["field", 2058]]` and `["get-quarter", ["field", 2058]]` for hour-of-day or quarter-number extraction.
-  - Do not invent helpers or bucket names like `month-of-year`, `quarter-of-year`, `hour-of-day`, or `dayofweek`.
-- When filtering extracted quarter values, use numeric quarter numbers `1`, `2`, `3`, `4`, not strings like `"Q1"`.
-- When you need the number of days or other units between two temporal values, use `["datetime-diff", left-temporal-expr, right-temporal-expr, "day"]`.
-  - Do not subtract dates with `-` to compute date differences.
+Breakout with temporal bucket:
 
-Use `with-fields` for row-level queries and `breakout` only for grouped results.
-If the result should be visualized, set `visualization.chart_type` to something appropriate like `table`, `bar`, `line`, `area`, `pie`, or `scalar`.
+```json
+"breakout": [["field", {"temporal-unit": "month"},
+              ["Sample Database", "PUBLIC", "ORDERS", "CREATED_AT"]]]
+```
 
-Aggregation operators:
-- `["count"]` — Count all rows
-- `["sum", ["field", id]]` — Sum
-- `["avg", ["field", id]]` — Average
-- `["min", ["field", id]]` — Minimum
-- `["max", ["field", id]]` — Maximum
-- `["distinct", ["field", id]]` — Count distinct values
-- `["count-where", clause]` — Count rows matching a condition
-- `["sum-where", ["field", id], clause]` — Sum for matching rows
-- `["median", ["field", id]]` — Median
-- `["percentile", ["field", id], 0.95]` — Percentile
-- `["measure", id]` — Use a pre-defined measure
+Order by — direction wraps a ref; works on field refs or aggregation refs:
 
-Filter operators:
-- `["=", field, value]`, `["!=", field, value]`
-- `["<", field, value]`, `["<=", field, value]`, `[">", field, value]`, `[">=", field, value]`
-- `["between", field, low, high]`
-- `["contains", field, "text"]`, `["does-not-contain", field, "text"]`
-- `["starts-with", field, "prefix"]`, `["ends-with", field, "suffix"]`
-- `["is-null", field]`, `["not-null", field]`, `["is-empty", field]`, `["not-empty", field]`
-- `["in", field, [v1, v2, ...]]`, `["not-in", field, [v1, v2, ...]]`
-- `["time-interval", field, n, "unit"]` — Relative time window
-- `["and", clause1, clause2]`, `["or", clause1, clause2]`, `["not", clause]`
-- `["segment", id]` — Pre-defined segment
+```json
+"order-by": [["desc", {}, ["field", {}, ["Sample Database", "PUBLIC", "ORDERS", "CREATED_AT"]]],
+             ["desc", {}, ["aggregation", {}, 0]]]
+```
 
-Expression operators:
-- Arithmetic: `["+", a, b]`, `["-", a, b]`, `["*", a, b]`, `["/", a, b]`
-- Math: `["abs", x]`, `["ceil", x]`, `["floor", x]`, `["round", x]`, `["power", x, n]`, `["sqrt", x]`
-- String: `["concat", a, b]`, `["substring", s, start, len]`, `["replace", s, old, new]`, `["upper", s]`, `["lower", s]`, `["trim", s]`, `["length", s]`
-- Conditional: `["case", [[cond1, val1], [cond2, val2]], default]`
-- Coalesce: `["coalesce", a, b]`
-- Date extraction: `["get-year", field]`, `["get-month", field]`, `["get-day", field]`, `["get-quarter", field]`, `["get-hour", field]`, `["get-day-of-week", field]`
-- Date arithmetic: `["datetime-add", field, n, "unit"]`, `["datetime-diff", field1, field2, "unit"]`
+`"limit": 50` and `"fields": [<field-ref>, ...]` are the obvious scalar / list forms.
 
-References:
-- `["field", id]` — Field by database ID
-- `["expression-ref", "Name"]` — Custom expression by name
-- `["aggregation-ref", 0]` — Aggregation result by 0-based index
+### Aggregation references
 
-Temporal bucketing (for breakout):
-- `["with-temporal-bucket", ["field", id], "month"]`
-- Valid buckets: `year`, `quarter`, `month`, `week`, `day`, `hour`, `minute`, `second`, `day-of-week`
+To refer to an aggregation already in the same stage's `aggregation` list — e.g. from `order-by`:
+1. **Inline** — repeat the clause itself. The inline form must match an entry in `aggregation` **exactly** (same op, same args). The tool rewrites it to a reference.
+2. **By 0-based index** (preferred when reused) — `["aggregation", {}, 0]` means "the first aggregation".
+
+Out-of-range indices surface a clear error listing every available aggregation with its index.
+
+## Expressions
+
+Define custom columns inside a stage using `expressions` and reference by name with `["expression", {}, "<Name>"]`. Object form (preferred); the name becomes the column's display name.
+
+```json
+"expressions": {
+  "Subtotal": ["+", {},
+               ["field", {}, ["Sample Database", "PUBLIC", "ORDERS", "TOTAL"]],
+               ["field", {}, ["Sample Database", "PUBLIC", "ORDERS", "TAX"]]]
+},
+"aggregation": [["sum", {}, ["expression", {}, "Subtotal"]]]
+```
+
+The sequential form `[["expression", {}, "Name", expr], ...]` is also accepted and auto-converted.
+
+## Joins (explicit)
+
+```json
+"joins": [{
+  "alias": "Products",
+  "strategy": "left-join",
+  "stages": [{"lib/type": "mbql.stage/mbql",
+              "source-table": ["Sample Database", "PUBLIC", "PRODUCTS"]}],
+  "conditions": [["=", {},
+    ["field", {}, ["Sample Database", "PUBLIC", "ORDERS", "PRODUCT_ID"]],
+    ["field", {"join-alias": "Products"},
+     ["Sample Database", "PUBLIC", "PRODUCTS", "ID"]]]]
+}],
+"breakout": [["field", {"join-alias": "Products"},
+              ["Sample Database", "PUBLIC", "PRODUCTS", "CATEGORY"]]]
+```
+
+- `alias` is a free-choice string; use it consistently in conditions, breakout, aggregation, order-by.
+- `strategy` is one of `"left-join"`, `"right-join"`, `"inner-join"`, `"full-join"`. Default `"left-join"`.
+- **Every** field reference on the joined side must carry `{"join-alias": "<same-alias>"}`.
+- Join `conditions` accept only `=`, `!=`, `<`, `<=`, `>`, `>=` — no `between`, `in`, `and`/`or` wrappers.
+
+## Implicit joins
+
+Reference a field on a related table directly — when the source has exactly one FK to that target, the tool auto-fills `source-field` and performs the implicit join:
+
+```json
+"source-table": ["Sample Database", "PUBLIC", "ORDERS"],
+"aggregation": [["count", {}]],
+"breakout": [["field", {}, ["Sample Database", "PUBLIC", "PRODUCTS", "CATEGORY"]]]
+```
+
+- **Multiple FKs** to the target → `:ambiguous-fk` lists them. Retry with explicit `{"source-field": ["DB", "SCH", "SRC", "FK_COL"]}` on the field.
+- **No FK path** → `:no-fk-path`. Switch to an explicit `joins` entry, or pick a field on the source table.
+- Inside an explicit `joins` block, joined-side field refs still need `{"join-alias": ...}` — implicit-join repair does not rewrite those.
+- If the FK column lives on a previous stage's output, write `{"source-field-name": "<col>"}` (not auto-filled).
+- If multiple explicit joins all expose the target FK, `:ambiguous-fk-via-join` lists them — set `{"source-field-join-alias": "<alias>"}` to pick one.
+
+**Tip:** discover FKs with `entity_details` / `read_resource metabase://table/<id>/fields`. FK columns are tagged `fk_target_fully_qualified_name="schema.table.field"` — always look for one before assuming a column lives on the current table.
+
+## Multi-stage queries
+
+Every stage after the first consumes the previous stage's output. Only the first stage has a `source-table`/`source-card`; later stages omit it.
+
+Post-aggregation filter — count orders per product, keep only those with > 10:
+
+```json
+"stages": [
+  {"lib/type": "mbql.stage/mbql",
+   "source-table": ["Sample Database", "PUBLIC", "ORDERS"],
+   "aggregation": [["count", {}]],
+   "breakout": [["field", {}, ["Sample Database", "PUBLIC", "ORDERS", "PRODUCT_ID"]]]},
+  {"lib/type": "mbql.stage/mbql",
+   "filters": [[">", {}, ["field", {}, "count"], 10]]}
+]
+```
+
+Re-aggregate — average daily total by month:
+
+```json
+"stages": [
+  {"lib/type": "mbql.stage/mbql",
+   "source-table": ["Sample Database", "PUBLIC", "ORDERS"],
+   "aggregation": [["sum", {}, ["field", {}, ["Sample Database", "PUBLIC", "ORDERS", "TOTAL"]]]],
+   "breakout": [["field", {"temporal-unit": "day"},
+                 ["Sample Database", "PUBLIC", "ORDERS", "CREATED_AT"]]]},
+  {"lib/type": "mbql.stage/mbql",
+   "aggregation": [["avg", {}, ["field", {}, "sum"]]],
+   "breakout": [["field", {"temporal-unit": "month"}, ["field", {}, "CREATED_AT"]]]}
+]
+```
+
+- Cross-stage refs use a **string name** in slot 3 (e.g. `["field", {}, "count"]`). The name is whatever the previous stage's aggregation/breakout/field produced (`count`, `sum`, or the source field's name).
+- Within the **same stage**, refer to your own aggregation with `["aggregation", {}, <idx>]` (see §Aggregation references). In a **later** stage, use the cross-stage string-name form against the previous stage's output.
+- Joins, expressions, filters, aggregation, breakout, order-by, limit are all valid in later stages.
+
+## Saved questions and models (`source-card`)
+
+Instead of `source-table`, use `source-card` to query an existing question or model. The value is the card's **portable entity id** — a 21-char opaque string reported by `entity_details` and search tools as `portable_entity_id`.
+
+1. Get the `portable_entity_id` from a tool response. `search` and `read_resource` (`metabase://question/<id>`, `metabase://model/<id>`) include it on the result tag — reuse what's already in context, no extra call needed.
+2. Copy it **verbatim** into `source-card`. The id is opaque — never guess, construct, or abbreviate.
+3. Reference the card's columns by output **name** (string in slot 3), not portable FK. If you don't know the names, call `read_resource metabase://question/<numeric-id>/fields` (or `.../model/...`).
+
+```json
+{"lib/type": "mbql.stage/mbql",
+ "source-card": "T4wA_GPFwGb6R4FxIDGTo",
+ "filters": [[">", {}, ["field", {}, "total"], 100]],
+ "limit": 50}
+```
+
+A stage has either `source-table` **or** `source-card`, never both. The card must live in the same database as the rest of the query (no cross-database queries). Falling back to native SQL like `SELECT * FROM {{#<id>-<slug>}}` is **not** acceptable — always use `source-card`.
+
+## Metrics
+
+A metric is a pre-defined aggregation attached to a base table. To use one:
+
+1. Put the metric's **base table** in `source-table`. Read it from the `base_table_fully_qualified_name` attribute on the `<metric>` tag (combine with `database_name` to form the portable FK). Never invent schema/table names.
+2. Reference the metric as `["metric", {}, "<portable_entity_id>"]` in `aggregation`.
+3. Filters/breakouts on the same stage use portable FKs on the metric's base table.
+
+```json
+{"lib/type": "mbql.stage/mbql",
+ "source-table": ["Analytics", "brex_enriched", "fct_cards"],
+ "filters": [["!=", {}, ["field", {},
+                         ["Analytics", "brex_enriched", "fct_cards", "card_status"]], "inactive"]],
+ "aggregation": [["metric", {}, "aB3cD4eF5gH6iJ7kL8mN9"]]}
+```
+
+Metrics are aggregations, **not sources** — never put a metric in `source-table` or `source-card`. The `metabase://metric/<id>` URIs are for reading metadata via `read_resource`, not for embedding in queries.
+
+## Using measures and segments
+
+A measure or segment can be referenced two ways. **Prefer the opaque-id clause** — it preserves the measure/segment's identity in the rendered notebook (chip with the official name, click-through to the definition, lineage tooling sees the dependency, definition updates propagate).
+
+### Preferred — opaque-id clause
+
+Each `<measure>` and `<segment>` block on a `<table>` carries a `portable_entity_id="…"` attribute (a 21-char NanoID). Copy it verbatim:
+
+```json
+"aggregation": [["measure", {}, "<portable_entity_id>"]]
+"filters": [["segment", {}, "<portable_entity_id>"]]
+```
+
+The stage's `source-table` must be the table the measure/segment belongs to (the same `<table>` block where you saw the `<measure>` / `<segment>` element).
+
+### Fallback — inline the `<definition>` body
+
+When you need to **compose on top of** a measure or segment (add an extra filter the segment doesn't include, breakout an aggregation the measure doesn't have, …), copy the JSON array inside `<definition>` directly into your stage's `aggregation: [...]` (measure) or `filters: [...]` (segment) and add your own clauses alongside. The definition is in the same portable form the tool expects.
+
+### Don't
+
+- Never invent a `portable_entity_id` — only use the value reported in a tool response.
+- The matching aggregation-only opaque-id clause is `["metric", {}, "<portable_entity_id>"]` (for metrics, which live independently of a table — see §Metrics above).
+
+## Rules and common mistakes
+
+Shape rules:
+- **Always include `{}` options in every clause**, even when empty. `["count"]` is wrong — it must be `["count", {}]`.
+- **The query is a JSON object**, not a string. Send it directly as the `"query"` field of the call.
+- **Use the exact database name** reported by `entity_details` (e.g. `"Sample Database"`, not `"Sample"`) as the first element of every portable FK. Near-misses surface `Unknown database` instead of silently picking one. Cross-database queries are not supported.
+- **Use portable FKs**, not numeric IDs. Schemaless databases use `null` in the schema slot. JSON-unfolded fields append path segments.
+- **Clause heads are lowercase, hyphenated**: `"count"`, `"sum-where"`, `"time-interval"`, `"get-day-of-week"`. Not underscores, not camelCase.
+- **Never invent a `source-card` entity_id.** It must be a 21-char string copied verbatim from `entity_details` / search — no patterns, no numeric ids, no `card__<id>`.
+- **`source-card` columns are referenced by output name** (string in slot 3), not portable FK.
+
+Anti-hallucination:
+- **Don't subtract dates** with `-`. Use `["datetime-diff", {}, <left>, <right>, "<unit>"]` for the integer count of units between two temporal values.
+- **For multi-value categorical filters, use `in` / `not-in`**, not `=` with a list literal. The tool rewrites the list form, but write canonical: `["in", {}, <field>, "a", "b"]`.
+- **Extracted quarter values are numbers `1, 2, 3, 4`**, never strings like `"Q1"`.
+- **Don't breakout on the same underlying field twice** in one stage. If you breakout by `{"temporal-unit": "month"}`, do not also breakout by the raw field.
+- **`visualization` is a sibling of `query`**, never embedded inside it.
+- **Ordering by an inline aggregation must match an `aggregation:` entry exactly** (same op, same args). If unsure, use `["aggregation", {}, <0-based-idx>]`.
+- **Never write `metabase://...` URIs as `source-table` or `source-card` values.** Those URIs are for `read_resource`, not for query sources.
+- **Never write `[aggregate, ...]`, `[filter, ...]`, `[order-by, ...]`, `[breakout, ...]`, `[limit, ...]` as clause heads** — those are stage *container keys*, not clauses. Place inner clauses directly inside the stage's `aggregation:` / `filters:` / etc. arrays.
+- Common typos (`count-if`, `variance`, `stddev-pop`, `count-distinct`, `dayofweek`, `hour-of-day`, `month-of-year`, `quarter-of-year`, `temporal-diff`, `relative-date`) are auto-corrected, but write the canonical name (`count-where`, `var`, `stddev`, `distinct`, `get-day-of-week`, `get-hour`, `get-month`, `get-quarter`, `datetime-diff`, `relative-datetime`) so the tool output matches what later inspection will show.
+
+## Operator catalogs
+
+> Source of truth: `src/metabase/lib/schema/{aggregation,filter,temporal_bucketing}.cljc` and `src/metabase/lib/schema/expression/*.cljc`. If something seems missing here, attempt it — the schema may have grown.
+
+### Aggregations
+
+- `["count", {}]` — count all rows. Optional 3rd slot `<expr>` counts non-NULL.
+- `["cum-count", {}]` — cumulative count. Optional 3rd slot `<expr>`.
+- `["sum", {}, <num-expr>]`
+- `["cum-sum", {}, <num-expr>]`
+- `["avg", {}, <num-expr>]`
+- `["min", {}, <orderable>]`
+- `["max", {}, <orderable>]`
+- `["median", {}, <num-expr>]`
+- `["percentile", {}, <num-expr>, <0..1>]`
+- `["distinct", {}, <expr>]` — count of distinct non-NULL values.
+- `["distinct-where", {}, <expr>, <bool-pred>]`
+- `["count-where", {}, <bool-pred>]`
+- `["sum-where", {}, <num-expr>, <bool-pred>]`
+- `["share", {}, <bool-pred>]` — fraction of rows where pred is true.
+- `["stddev", {}, <num-expr>]`
+- `["var", {}, <num-expr>]`
+- `["metric", {}, "<portable_entity_id>"]` — reference a defined metric. The metric's base table must be the stage's `source-table`.
+- `["measure", {}, "<portable_entity_id>"]` — reference a defined measure (preferred over inlining its definition). The measure's table must be the stage's `source-table`.
+
+### Order-by direction
+
+- `["asc", {}, <ref>]` / `["desc", {}, <ref>]` — wrap any field ref or aggregation ref.
+
+### Filters
+
+Boolean:
+- `["and", {}, <pred>, <pred>, ...]` — min 2 args.
+- `["or", {}, <pred>, <pred>, ...]` — min 2 args.
+- `["not", {}, <pred>]`
+
+Equality (variadic, ≥2 args):
+- `["=", {}, <a>, <b>, ...]`
+- `["!=", {}, <a>, <b>, ...]`
+- `["in", {}, <expr>, <v1>, <v2>, ...]`
+- `["not-in", {}, <expr>, <v1>, <v2>, ...]`
+
+Comparison:
+- `["<", {}, <a>, <b>]` / `["<=", {}, <a>, <b>]` / `[">", {}, <a>, <b>]` / `[">=", {}, <a>, <b>]`
+- `["between", {}, <expr>, <min>, <max>]`
+- `["inside", {}, <lat-expr>, <lon-expr>, <lat-max>, <lon-min>, <lat-min>, <lon-max>]` — lat/lon bounding box.
+
+Nullness / emptiness:
+- `["is-null", {}, <expr>]` / `["not-null", {}, <expr>]`
+- `["is-empty", {}, <expr>]` / `["not-empty", {}, <expr>]` — equivalent to is-null OR `= ""` for strings.
+
+String (accept `{"case-sensitive": false}` in opts):
+- `["starts-with", {}, <str>, <prefix>]`
+- `["ends-with", {}, <str>, <suffix>]`
+- `["contains", {}, <str>, <substring>]`
+- `["does-not-contain", {}, <str>, <substring>]`
+
+Temporal:
+- `["time-interval", {}, <temporal>, <n-or-:current/:last/:next>, "<unit>"]` — relative window. Opts may set `{"include-current": true}`.
+- `["during", {}, <temporal>, "<iso-date-or-datetime>", "<unit>"]` — value falls within the bucket containing the literal.
+- `["relative-time-interval", {}, <temporal>, <value>, "<bucket>", <offset-value>, "<offset-bucket>"]` — window offset from now.
+
+Named reference:
+- `["segment", {}, "<portable_entity_id>"]` — reference a defined segment (preferred over inlining its definition). The segment's table must be the stage's `source-table`.
+
+### Expressions
+
+Arithmetic:
+- `["+", {}, <a>, <b>, ...]` / `["-", {}, <a>, <b>, ...]` / `["*", {}, <a>, <b>, ...]` / `["/", {}, <a>, <b>, ...]`
+- `["abs", {}, <num>]` / `["power", {}, <num>, <exp>]` / `["sqrt", {}, <num>]`
+- `["exp", {}, <num>]` / `["log", {}, <num>]`
+- `["ceil", {}, <num>]` / `["floor", {}, <num>]` / `["round", {}, <num>]`
+- `["integer", {}, <num-or-str>]` / `["float", {}, <num-or-str>]`
+
+String:
+- `["concat", {}, <a>, <b>, ...]`
+- `["substring", {}, <str>, <start>, <length>]` — 1-based start; length is optional.
+- `["replace", {}, <str>, <old>, <new>]`
+- `["regex-match-first", {}, <str>, <regex>]`
+- `["split-part", {}, <str>, <sep>, <n>]`
+- `["length", {}, <str>]`
+- `["trim", {}, <str>]` / `["ltrim", {}, <str>]` / `["rtrim", {}, <str>]`
+- `["upper", {}, <str>]` / `["lower", {}, <str>]`
+- `["host", {}, <url>]` / `["domain", {}, <url>]` / `["subdomain", {}, <url>]` / `["path", {}, <url>]`
+- `["month-name", {}, <int>]` / `["quarter-name", {}, <int>]` / `["day-name", {}, <int>]`
+- `["text", {}, <expr>]` / `["collate", {}, <str>, <collation>]`
+
+Conditional:
+- `["case", {}, [[<pred1>, <expr1>], [<pred2>, <expr2>], ...], <default?>]` — alias: `if`.
+- `["coalesce", {}, <a>, <b>, ...]`
+
+Temporal:
+- `["datetime-add", {}, <temporal>, <n>, "<unit>"]`
+- `["datetime-subtract", {}, <temporal>, <n>, "<unit>"]`
+- `["datetime-diff", {}, <left>, <right>, "<unit>"]` — `<unit>` is one of `second, minute, hour, day, week, month, quarter, year`.
+- `["interval", {}, <n>, "<unit>"]`
+- `["get-year", {}, <temporal>]` / `["get-month", {}, <temporal>]` / `["get-day", {}, <temporal>]` / `["get-quarter", {}, <temporal>]`
+- `["get-hour", {}, <temporal>]` / `["get-minute", {}, <temporal>]` / `["get-second", {}, <temporal>]`
+- `["get-week", {}, <temporal>]` / `["get-day-of-week", {}, <temporal>]` — both accept optional 4th-slot mode `"iso"`, `"us"`, `"instance"`.
+- `["temporal-extract", {}, <temporal>, "<unit>"]` — extract one of the date/time extraction units listed below.
+- `["convert-timezone", {}, <temporal>, "<target-tz>"]` — optional 4th slot is source tz.
+- `["relative-datetime", {}, <n>, "<unit>"]` (or `["relative-datetime", {}, "current"]`)
+- `["absolute-datetime", {}, "<iso-string>", "<unit?>"]`
+- `["date", {}, <expr>]` / `["datetime", {}, <expr>]` / `["time", {}, <expr>]`
+- `["now", {}]` / `["today", {}]`
+
+### Temporal units (for `{"temporal-unit": ...}` on field refs and as `<unit>` args)
+
+- Date truncation: `day`, `week`, `month`, `quarter`, `year`
+- Date extraction (integer-returning): `day-of-week`, `day-of-month`, `day-of-year`, `week-of-year`, `month-of-year`, `quarter-of-year`, `year`, `year-of-era`
+- Time truncation: `millisecond`, `second`, `minute`, `hour`
+- Time extraction (integer-returning): `second-of-minute`, `minute-of-hour`, `hour-of-day`
+- Plus `default` — let the system pick based on the field's base type.

src/metabase/agent_api/api.clj

@@ -3,9 +3,7 @@
   Endpoints are versioned (e.g., /v1/search) and use standard HTTP semantics."
   (:require
    [clojure.string :as str]
-   [malli.util :as mut]
    [metabase.agent-api.validation :as agent-api.validation]
-   [metabase.agent-lib.core :as agent-lib]
    [metabase.api.common :as api]
    [metabase.api.macros :as api.macros]
    [metabase.api.macros.scope :as scope]
@@ -369,209 +367,121 @@
 
 ;;; ------------------------------------------------ Construct Query -------------------------------------------------
 
-(mr/def ::program-request
-  "Request body for /v2/query.
-  An agent-lib structured program with `:source` and `:operations`. The top-level
-  `:source` must reference a database entity (`table`, `card`, `dataset`, or
-  `metric`); `context` and nested `program` sources are rejected at the HTTP
-  boundary by [[evaluate-program-for-execution]] because they require an
-  in-process evaluation context."
-  agent-lib/program-schema)
-
-(def ^:private construct-query-prompt-max-length
-  10000)
-
-(def ^:private ConstructQueryPrompt
-  (mut/update-properties
-   [:and
-    ms/NonBlankString
-    [:string {:max construct-query-prompt-max-length}]]
-   merge
-   {:json-schema {:type        "string"
-                  :minLength   1
-                  :maxLength   construct-query-prompt-max-length
-                  :description "The user's exact original message, when available. Pass it as-is without summarizing or rewriting."}}))
-
 (mr/def ::construct-query-request
-  "Request body for /v2/construct-query. Same as program-request, with an optional prompt
-  capturing the user's original intent when a caller has one."
-  (mut/merge agent-lib/program-schema
-             [:map
-              [:prompt {:optional true} [:maybe ConstructQueryPrompt]]]))
+  "Request body for /v2/construct-query and the fresh-query branch of /v2/query.
+  A single `:query` key whose value is a JSON object matching
+  [[metabase.lib.schema/external-query]] — the canonical portable MBQL 5 wire format. The
+  query is fully self-describing: the database is derived from the first stage's
+  `source-table:` or `source-card:`, all field references are portable FKs
+  (`[<db-name>, <schema>, <table-name>, <column-name>]`), and there is no auxiliary
+  `source_entity` / `referenced_entities` envelope. See
+  `resources/metabot/prompts/tools/construct_notebook_query.md` for the full format reference
+  (including operators, joins, expressions, multi-stage queries, and FK conventions).
+
+  Closed map: any extra top-level keys (notably the legacy `source_entity` /
+  `referenced_entities` envelope from before the repr migration) are rejected with a 400 so
+  callers don't silently send fields the server ignores.
+
+  The inner `:query` value is intentionally typed as a plain `:map` at this boundary rather
+  than `::lib.schema/external-query`. Reasons:
+
+  1. Deep MBQL-shape validation runs inside the representations pipeline
+     (`metabot.tools.construct/execute-representations-query` calls `repr/validate-query`
+     after the repair pass), so the boundary check would be redundant.
+
+  2. The strict-tool manifest lint (`assert-optional-fields-nullable!`) walks every map
+     reachable from the tool input schema. `::external-query` references `::query`, which
+     carries several `:optional` keys (`:lib/metadata`, `:database`, `:settings`, …) that
+     are not `[:maybe ...]` — for sound reasons unrelated to this endpoint. Recursing into
+     them would force a wide schema change just to satisfy the lint at the agent boundary."
+  [:map {:closed true}
+   [:query {:tool/description (str "A Metabase MBQL 5 query as a JSON object. See the "
+                                   "`construct_notebook_query` tool for the format reference.")}
+    :map]
+   ;; The user's original message, when available, captured so `visualize_query` can later
+   ;; surface it back to the iframe alongside the query body for feedback submission. The MCP
+   ;; layer stores it with the handle (see `metabase.mcp.tools/make-store-construct-query-result`).
+   ;; Bounded at 10000 chars to match the constraint master enforced on the legacy program path.
+   [:prompt {:optional true} [:maybe [:string {:min 1 :max 10000}]]]])
 
 (mr/def ::construct-query-response
-  "Response containing a base64-encoded MBQL query and, when supplied, the original prompt for use with /v1/execute."
+  "Response containing a base64-encoded MBQL query for use with /v1/execute. The optional
+  `:prompt` echoes the request's prompt back so the MCP layer can store it with the
+  handle (see `metabase.mcp.tools/make-store-construct-query-result`)."
   [:map
-   [:query ms/NonBlankString]
-   [:prompt {:optional true} ConstructQueryPrompt]])
+   [:query  ms/NonBlankString]
+   [:prompt {:optional true} [:maybe ms/NonBlankString]]])
 
-(def ^:private allowed-program-source-types
-  "Top-level program source types that the HTTP boundary accepts. `context` and
-  nested `program` sources require an in-process evaluation context and are
-  rejected here."
-  #{"table" "card" "dataset" "metric"})
+(defn- evaluate-external-query-to-live-query
+  "Run the representations pipeline (validate → convert → repair → resolve) on a request body
+  and return the resolved MBQL 5 lib query (with `:lib/metadata` attached).
 
-(def ^:private construct-query-tool-description
-  "User-facing description for the `construct_query` MCP tool. Tuned to give the
-  LLM enough structure to produce valid programs without reproducing the full
-  reference — covers the program shape, canonical operator names, reference forms,
-  and a few worked examples spanning the common patterns."
-  (str
-   "Construct a Metabase MBQL query from a structured program. The body structure is:\n"
-   "`{\"source\": {...}, \"operations\": [...]}`\n"
-   "For MCP calls, include `\"prompt\": \"<user's exact original message>\"` whenever you have the user's message; do not summarize or modify it.\n"
-   "Returns `{\"query_handle\": \"<uuid>\"}` — pass `query_handle` to `execute_query` or `visualize_query`.\n"
-   "For the full reference, read the `metabase://docs/construct-query.md` MCP resource.\n"
-   "\n"
-   "IMPORTANT: field IDs must come from entity-detail endpoints (`/v1/table/{id}`, `/v1/metric/{id}`). "
-   "Do not invent IDs. The backend repairs minor mistakes (aliases, casing, over-wrapping) before validation, "
-   "but the canonical names below always work.\n"
-   "\n"
-   "## Workflow\n"
-   "1. Use `search_entities` / entity-detail tools to find the table/metric/model and its fields.\n"
-   "2. Call `construct_query` with the program. Include the user's original `prompt` whenever available. You get back `{\"query_handle\": \"<uuid>\"}`.\n"
-   "3. Pass `query_handle` to the follow-up `execute_query` or `visualize_query` call.\n"
-   "Never embed IDs you did not read from a metadata endpoint — invented IDs will fail at execution.\n"
-   "\n"
-   "## Source\n"
-   "One of `{\"type\": T, \"id\": N}`:\n"
-   "- `table` — a database table\n"
-   "- `card` — a saved question\n"
-   "- `dataset` — a model (model card id)\n"
-   "- `metric` — a metric (supplies its own aggregation and time dimension; extra aggregates usually unnecessary)\n"
-   "\n"
-   "## Top-level operations (applied in order)\n"
-   "Each operation is `[\"op\", arg, ...]`:\n"
-   "- `[\"filter\", clause]` — add a filter\n"
-   "- `[\"aggregate\", agg-clause]` — add an aggregation\n"
-   "- `[\"breakout\", ref-or-bucketed]` — add a grouping dimension\n"
-   "- `[\"expression\", \"Name\", expr]` — define a named computed column (reference later with `expression-ref`)\n"
-   "- `[\"with-fields\", [refs...]]` — restrict returned columns\n"
-   "- `[\"order-by\", ref]` or `[\"order-by\", ref, \"asc\"|\"desc\"]` — sort\n"
-   "- `[\"limit\", N]` — cap rows\n"
-   "- `[\"join\", join-clause]` — join another entity\n"
-   "- `[\"append-stage\"]` — start a new query stage (needed to filter on aggregated values)\n"
-   "- `[\"with-page\", {\"page\": N, \"items\": M}]` — paginate\n"
-   "\n"
-   "## References (used as arguments inside operations)\n"
-   "- `[\"field\", N]` — database field by id. Do NOT put options in a third slot (no `[\"field\", id, {...}]`); wrap instead\n"
-   "- `[\"expression-ref\", \"Name\"]` — a named expression defined earlier\n"
-   "- `[\"aggregation-ref\", N]` — the Nth `aggregate` defined earlier (0-based). REQUIRED when sorting by an aggregated value\n"
-   "- `[\"measure\", N]` — a pre-defined measure on the source entity\n"
-   "- `[\"with-temporal-bucket\", ref, unit]` — temporal bucketing. `unit` is one of: `minute` `hour` `day` `week` `month` `quarter` `year`. Also `day-of-week`, `hour-of-day`, etc. (extraction aliases)\n"
-   "- `[\"with-binning\", ref, {\"strategy\": \"num-bins\"|\"bin-width\"|\"default\", ...}]` — numeric binning. E.g. `{\"strategy\": \"num-bins\", \"num-bins\": 10}`\n"
-   "\n"
-   "## Filter operators\n"
-   "`=`, `!=`, `<`, `<=`, `>`, `>=`, `between`, `in`, `not-in`, `is-null`, `not-null`, `is-empty`, `not-empty`, "
-   "`contains`, `does-not-contain`, `starts-with`, `ends-with`, `time-interval`, `and`, `or`, `not`, `segment`.\n"
-   "Examples: `[\"=\", [\"field\", 101], \"active\"]`, `[\"between\", [\"field\", 305], \"2024-01-01\", \"2024-12-31\"]`, "
-   "`[\"in\", [\"field\", 302], [10, 20, 30]]`, `[\"time-interval\", [\"field\", 305], -7, \"day\"]`.\n"
-   "\n"
-   "## Aggregation operators\n"
-   "`count`, `sum`, `avg`, `min`, `max`, `distinct`, `median`, `stddev`, `var`, `percentile`, "
-   "`count-where`, `sum-where`, `distinct-where`, `share`, `cum-count`, `cum-sum`. "
-   "Examples: `[\"count\"]`, `[\"sum\", [\"field\", 302]]`, `[\"count-where\", [\"=\", [\"field\", 101], \"completed\"]]`.\n"
-   "\n"
-   "## Temporal helpers (for use in `expression` or as grouping)\n"
-   "`get-year`, `get-quarter`, `get-month`, `get-week`, `get-day`, `get-day-of-week`, `get-hour`, `get-minute`, "
-   "`datetime-add`, `datetime-diff`, `datetime-subtract`, `now`, `today`, `relative-datetime`, `absolute-datetime`, "
-   "`with-temporal-bucket`, `convert-timezone`.\n"
-   "\n"
-   "## Examples\n"
-   "Top 5 customers by revenue:\n"
-   "```\n"
-   "{\"source\": {\"type\": \"table\", \"id\": 42},\n"
-   " \"operations\": [[\"aggregate\", [\"sum\", [\"field\", 302]]],\n"
-   "                [\"breakout\", [\"field\", 101]],\n"
-   "                [\"order-by\", [\"aggregation-ref\", 0], \"desc\"],\n"
-   "                [\"limit\", 5]]}\n"
-   "```\n"
-   "Monthly revenue from a metric (metric supplies the aggregation):\n"
-   "```\n"
-   "{\"source\": {\"type\": \"metric\", \"id\": 10},\n"
-   " \"operations\": [[\"breakout\", [\"with-temporal-bucket\", [\"field\", 305], \"month\"]],\n"
-   "                [\"order-by\", [\"with-temporal-bucket\", [\"field\", 305], \"month\"], \"asc\"]]}\n"
-   "```\n"
-   "Filter on an aggregated value (requires `append-stage`):\n"
-   "```\n"
-   "{\"source\": {\"type\": \"table\", \"id\": 42},\n"
-   " \"operations\": [[\"aggregate\", [\"sum\", [\"field\", 302]]],\n"
-   "                [\"breakout\", [\"field\", 101]],\n"
-   "                [\"append-stage\"],\n"
-   "                [\"filter\", [\">\", [\"aggregation-ref\", 0], 1000]]]}\n"
-   "```\n"
-   "Named expression referenced later:\n"
-   "```\n"
-   "{\"source\": {\"type\": \"table\", \"id\": 42},\n"
-   " \"operations\": [[\"expression\", \"Discount\", [\"-\", [\"field\", 302], [\"field\", 303]]],\n"
-   "                [\"aggregate\", [\"sum\", [\"expression-ref\", \"Discount\"]]]]}\n"
-   "```\n"
-   "Previous-period comparison with `offset` (stay in the SAME stage — do NOT add `append-stage`):\n"
-   "```\n"
-   "{\"source\": {\"type\": \"table\", \"id\": 42},\n"
-   " \"operations\": [[\"aggregate\", [\"sum\", [\"field\", 302]]],\n"
-   "                [\"aggregate\", [\"offset\", [\"sum\", [\"field\", 302]], -1]],\n"
-   "                [\"breakout\", [\"with-temporal-bucket\", [\"field\", 305], \"month\"]]]}\n"
-   "```\n"
-   "\n"
-   "## Rules & common pitfalls\n"
-   "Stage boundaries (most common source of errors):\n"
-   "- Filtering on an aggregated value REQUIRES `append-stage` between the aggregate/breakout and the filter "
-   "(see the \"filter on aggregated value\" example). Without it, `aggregation-ref` resolution fails in the same stage.\n"
-   "- Defining an `expression` that uses `aggregation-ref` also REQUIRES `append-stage` first.\n"
-   "- EXCEPTION: `offset` (previous-period comparison) stays in the same stage as its base aggregation and breakout — do NOT add `append-stage` for it.\n"
-   "\n"
-   "Refs & shapes:\n"
-   "- Aggregation helpers take field refs, not bare IDs: `[\"sum\", [\"field\", 201]]`, never `[\"sum\", 201]`.\n"
-   "- To sort by an aggregated value, use `[\"aggregation-ref\", N]` — not the original expression.\n"
-   "- Do NOT put options in a third slot of `field` (no `[\"field\", id, {...}]`). Wrap instead: `[\"with-temporal-bucket\", [\"field\", id], \"month\"]` or `[\"with-binning\", [\"field\", id], {...}]`.\n"
-   "- `case` takes `[[condition, value], ...]` branches and an optional bare fallback as the THIRD arg — do not wrap it as `{\"default\": ...}`. Omit the third arg when there is no fallback.\n"
-   "- JSON objects appear only where a helper explicitly calls for one (e.g. `with-page`, `with-binning`). Everywhere else, use operator tuples.\n"
-   "\n"
-   "Joins & related tables:\n"
-   "- If the source table's detail response already surfaces a related table's fields, use those field refs directly — no explicit join needed.\n"
-   "- Reach for `join` + `with-join-conditions` only for custom aliases, self-joins, explicit joined-field selection, or when direct related-field refs are unavailable.\n"
-   "- If an explicit join returns a permission error, the underlying table is not accessible — surface the error, do not retry with implicit refs.\n"
-   "\n"
-   "Metrics & dates:\n"
-   "- A `metric` source already provides its own aggregation and time dimension. Add only the additional breakouts/filters you need.\n"
-   "- When the user asks for an exact year (e.g. 2024), use `[\"=\", [\"field\", year_field], 2024]` or a `between` with explicit dates — not relative filters like `time-interval`.\n"))
+  The pipeline raises `:agent-error?` ex-data on any LLM-input failure (unknown DB, unknown
+  table, ambiguous FK, etc.); we let those propagate so [[api.macros/defendpoint]] surfaces
+  them with the appropriate 4xx status code instead of a 500."
+  [body]
+  (-> (metabot-construct/execute-representations-query (:query body))
+      (get-in [:structured-output :query])))
 
-(defn- evaluate-program-to-live-query
-  "Resolve a program's source entity, evaluate the program via agent-lib, and return
-  the live lib query (with lib metadata attached)."
-  [program]
-  (let [source-type (get-in program [:source :type])]
-    (api/check (contains? allowed-program-source-types source-type)
-               [400 (str "top-level program source must be one of: "
-                         (str/join ", " (sort allowed-program-source-types)))]))
-  (let [source-entity (metabot-construct/program-source->source-entity (:source program))
-        result        (metabot-construct/execute-program source-entity nil program)]
-    (get-in result [:structured-output :query])))
-
-(defn- evaluate-program-for-execution
-  "Evaluate a program and return a plain MBQL 5 query map suitable for serialization
-  into a continuation token and execution by the QP."
-  [program]
-  (lib/prepare-for-serialization (evaluate-program-to-live-query program)))
+(defn- evaluate-external-query-for-execution
+  "Evaluate a request body and return a plain MBQL 5 query map suitable for serialization into
+  a continuation token and execution by the QP."
+  [body]
+  (lib/prepare-for-serialization (evaluate-external-query-to-live-query body)))
 
 (api.macros/defendpoint :post "/v2/construct-query" :- ::construct-query-response
-  "Construct an MBQL query from a structured agent-lib program.
+  "Construct an MBQL query from a portable MBQL 5 representations JSON payload.
 
-  The body is the program itself: a JSON object with `source` (identifying the
-  table/card/dataset/metric to query), `operations` (an array of operator
-  tuples), and `prompt` (the user's original request). Returns a base64-encoded
-  MBQL query that can be executed via /v1/execute. See the agent_api reference
-  for the full program syntax."
+  The body is `{\"query\": <external-query>}` where `<external-query>` is a JSON object
+  matching `::lib.schema/external-query` \u2014 see the `construct_notebook_query` tool
+  documentation for the format reference. Returns a base64-encoded MBQL query that can be
+  executed via /v1/execute or paginated via /v2/query."
   {:scope metabot/agent-query-construct
    :tool  {:name "construct_query"
-           :description construct-query-tool-description
+           ;; Condensed inline summary so LLMs see the shape directly in `tools/list`. Full
+           ;; grammar lives in the `construct_notebook_query` tool prompt.
+           :description
+           (str "Construct a Metabase MBQL 5 query as JSON. Pass the body "
+                "`{\"query\": <object>}`; returns `{\"query_handle\": \"<uuid>\"}` to feed "
+                "`execute_query` or `visualize_query`.\n"
+                "\n"
+                "Workflow: use search / entity_details first to discover the exact database, "
+                "schema, table, and column NAMES (not numeric IDs). Never invent identifiers.\n"
+                "\n"
+                "Shape: every clause is `[\"op\", {}, ...args]` with a MANDATORY empty options "
+                "map at position 1. Every field reference is "
+                "`[\"field\", {}, [<db-name>, <schema-or-null>, <table-name>, <column-name>]]` "
+                "(4-segment portable FK string array, NOT a numeric id). Cross-stage refs use "
+                "a bare column-name string in the third slot: `[\"field\", {}, \"count\"]`.\n"
+                "\n"
+                "Top level: `{\"lib/type\": \"mbql/query\", \"stages\": [...]}`. Each stage has "
+                "`\"lib/type\": \"mbql.stage/mbql\"` plus either `source-table` (portable FK) "
+                "or `source-card` (entity_id string) on the FIRST stage only; later stages "
+                "implicitly read the previous stage's output. Per-stage clause keys: "
+                "`filters`, `aggregation`, `breakout`, `expressions`, `fields`, `joins`, "
+                "`order-by`, `limit`.\n"
+                "\n"
+                "Minimal example (count of orders by month):\n"
+                "```\n"
+                "{\"query\": {\"lib/type\": \"mbql/query\",\n"
+                "             \"stages\": [{\"lib/type\": \"mbql.stage/mbql\",\n"
+                "                          \"source-table\": [\"Sample Database\", \"PUBLIC\", \"ORDERS\"],\n"
+                "                          \"aggregation\": [[\"count\", {}]],\n"
+                "                          \"breakout\": [[\"field\", {\"temporal-unit\": \"month\"},\n"
+                "                                        [\"Sample Database\", \"PUBLIC\", \"ORDERS\", \"CREATED_AT\"]]]}]}}\n"
+                "```\n"
+                "\n"
+                "Common pitfalls: (1) forgetting the `{}` options map; (2) writing numeric "
+                "ids where a portable FK is required; (3) putting a post-aggregation filter "
+                "(`[\">\", {}, [\"aggregation\", {}, 0], 10]`) alongside `order-by` / `limit` "
+                "in the same stage \u2014 split into two stages explicitly. See the "
+                "`construct_notebook_query` tool prompt for the full grammar, operator "
+                "catalog, joins, expressions, and multi-stage examples.")
            :annotations {:read-only? true :idempotent? true}}}
   [_route-params
    _query-params
-   {:keys [prompt] :as request} :- ::construct-query-request]
-  (let [program (dissoc request :prompt)
-        query   (evaluate-program-for-execution program)]
+   {:keys [prompt] :as body} :- ::construct-query-request]
+  (let [query (evaluate-external-query-for-execution body)]
     (cond-> {:query (-> query json/encode u/encode-base64)}
       prompt (assoc :prompt prompt))))
 
@@ -589,9 +499,9 @@
 (defn- decode-continuation-token
   "Decode a base64-encoded continuation token into {:query ... :pagination ...}.
    The token is client-supplied, so sanity-check the pagination ints to turn
-   garbage into a 400 rather than a downstream 500. This is robustness, not a
-   security boundary — a caller can always issue a fresh program to run any
-   query they want."
+   garbage into a 400 rather than a downstream 500. Permission re-validation on
+   the embedded query happens in [[check-token-query-permissions!]] — a token
+   doesn't grant access the bearer wouldn't otherwise have."
   [token]
   (let [decoded (-> token u/decode-base64 json/decode+kw)
         {:keys [limit page]} (:pagination decoded)]
@@ -657,40 +567,64 @@
                        :max-results-bare-rows page-size}))
 
 (mr/def ::query-request
-  "Request body for /v2/query. Accepts either a structured program or a continuation_token."
+  "Request body for /v2/query. Accepts either a fresh-query payload (`{:query <external-query>}`,
+  same shape as /v2/construct-query) or a `:continuation_token` from a prior response.
+
+  Both branches are closed maps: extra top-level keys (e.g. the legacy
+  `source_entity` / `referenced_entities` envelope, or sending `:query` and
+  `:continuation_token` simultaneously) are rejected with a 400."
   [:multi {:dispatch (fn [m]
-                       (if (:continuation_token m) :continuation :program))}
-   [:continuation [:map [:continuation_token ms/NonBlankString]]]
-   [:program      ::program-request]])
+                       (if (:continuation_token m) :continuation :fresh))}
+   [:continuation [:map {:closed true} [:continuation_token ms/NonBlankString]]]
+   [:fresh        ::construct-query-request]])
+
+(defn- check-token-query-permissions!
+  "Re-validate query permissions on the continuation-token path.
+
+  The token body is client-supplied and could in principle name a different source table than
+  the one the fresh `/v2/query` call was authorized against (a user's data perms can also
+  change between pages). The QP middleware would catch this at execution time, but running
+  the explicit `api/query-check` first gives a cleaner 403 and avoids spinning up the
+  streaming response just to abort."
+  [query-map]
+  (when-let [table-id (get-in query-map [:stages 0 :source-table])]
+    (when (int? table-id)
+      (api/query-check :model/Table table-id))))
 
 (defn- initial-page-state
   "Normalize the two /v2/query entry points into a single {:query :total-limit :page}
-   shape. A fresh program evaluates the user's program and computes a total-row budget
-   from its `:limit`; a continuation token carries that state from a prior response."
+   shape. A fresh request body is evaluated through the representations pipeline and the
+   total-row budget is derived from the resolved query's `:limit`; a continuation token
+   carries that state from a prior response (and re-validates query permissions, since the
+   token is client-supplied and per-user permissions can change between pages)."
   [body]
   (if-let [token (:continuation_token body)]
     (let [{:keys [query pagination]} (decode-continuation-token token)]
+      (check-token-query-permissions! query)
       {:query query :total-limit (:limit pagination) :page (:page pagination)})
-    (let [live-query (evaluate-program-to-live-query body)]
+    (let [live-query (evaluate-external-query-to-live-query body)]
       {:query       (lib/prepare-for-serialization live-query)
        :total-limit (total-row-limit live-query)
        :page        1})))
 
 (api.macros/defendpoint :post "/v2/query"
   :- (streaming-response/streaming-response-schema ::query-response)
-  "Execute a structured program and stream the results, with continuation-token pagination.
+  "Execute a portable MBQL 5 representations JSON query and stream the results, with
+  continuation-token pagination.
 
-  Accepts either a program (same shape as /v2/construct-query) or a
-  `continuation_token` from a previous response. Returns results with column
-  metadata and an optional `continuation_token` for fetching the next page."
+  Accepts either a JSON body (same shape as /v2/construct-query) or a `continuation_token`
+  from a previous response. Returns results with column metadata and an optional
+  `continuation_token` for fetching the next page."
   {:scope "agent:query"
    :tool  {:name "query"
            :title "Query Tables and Metrics"
-           :description (str "Execute a structured program and return results with column metadata. "
-                             "If more rows are available the response includes a continuation_token "
-                             "— pass it back to fetch the next page. Body is either a program (same "
-                             "shape as construct_query) or {\"continuation_token\": \"...\"}. See the "
-                             "`metabase://docs/construct-query.md` resource for program syntax.")
+           :description (str "Execute a Metabase query and return results with column "
+                             "metadata. If more rows are available, the response includes a "
+                             "continuation_token — pass it back to get the next page.\n\n"
+                             "The body is either `{\"query\": <object>}` (same shape as "
+                             "construct_query; see the `construct_notebook_query` tool for "
+                             "the format reference) or `{\"continuation_token\": \"...\"}` "
+                             "from a previous response.")
            :annotations {:read-only? true}}}
   [_route-params
    _query-params
@@ -761,9 +695,7 @@
                              "row count, and execution time. Use this when the user explicitly asks for raw data, "
                              "rows, columns, counts, metadata, or programmatic query results. If the user asks to "
                              "show, display, visualize, plot, chart, or present the result, use visualize_query "
-                             "instead. "
-                             "Pass `query_handle` (preferred) from construct_query rather than constructing the "
-                             "base64 query yourself.")
+                             "instead.")
            :annotations {:read-only? true :idempotent? true}}}
   [_route-params
    _query-params

src/metabase/agent_api/reference.md

@@ -14,8 +14,9 @@ Base path: /api/agent
   (e.g., "Total Revenue"). Metrics are stored in collections and can be used
   as a data source in the API. They have a fixed aggregation, but can be
   filtered and grouped by their queryable dimensions. Use /v1/metric/{id} to
-  inspect a metric's dimensions, and POST /v2/construct-query with a program
-  whose `source` is `{"type": "metric", "id": <id>}` to query one.
+  inspect a metric's dimensions, and reference the metric in a representations
+  JSON query as `"aggregation": [["metric", {}, "<portable_entity_id>"]]` (see
+  the representations format reference linked from /v2/construct-query).
 - **Measures**: Lightweight, reusable aggregation expressions (e.g.,
   `SUM(total)`) associated with a specific table. Unlike metrics, measures are
   not standalone queries — they are building blocks that can be referenced in
@@ -285,51 +286,47 @@ Response:
 
 ### POST /v2/construct-query
 
-Construct an MBQL query from a structured agent-lib program. Returns a
+Construct an MBQL query from a representations JSON payload. Returns a
 base64-encoded query string to pass to `/v1/execute`.
 
-**Important**: All field IDs used in operations must come from the detail
-endpoints (`/v1/table/{id}` or `/v1/metric/{id}`). Always fetch entity details
-first.
-
-The request body **is** the program — there is no envelope. A program is a
-JSON object with two keys:
-
-- `source` — identifies the entity to query (`table`, `card`, `dataset`, or
-  `metric` plus an `id`).
-- `operations` — an ordered array of operator tuples to apply on top of the
-  source. Each operation is itself an array: `["operator", arg1, arg2, ...]`.
-
-The agent-lib backend automatically repairs small mistakes (operator aliases,
-casing, scalar wrapping) before validation, so you don't need to be perfectly
-precise about every detail — but the canonical operator names listed below
-will always work.
-
-#### Request format
+#### Request body
 
 ```json
-{
-  "source": {"type": "table", "id": 42},
-  "operations": [
-    ["filter", [">", ["field", 302], 100]],
-    ["aggregate", ["sum", ["field", 302]]],
-    ["breakout", ["with-temporal-bucket", ["field", 305], "month"]],
-    ["order-by", ["aggregation-ref", 0], "desc"],
-    ["limit", 100]
-  ]
-}
+{"query": <external-query-object>}
 ```
 
-For a metric source (the metric supplies its own aggregation, so additional
-aggregates are usually unnecessary):
+The `query` value is a JSON object matching `::lib.schema/external-query` —
+Metabase's canonical MBQL 5 *portable representations format*, a
+self-describing serialization that uses portable FKs
+(`["<db-name>", "<schema>", "<table-name>", "<column-name>"]`) instead of
+numeric IDs, encodes operators as `["<name>", {opts}, <args>...]` clauses, and
+derives the application database from the first stage's `source-table` /
+`source-card`. There is **no** auxiliary `source_entity` envelope, and no
+standalone integer IDs.
+
+The full format reference — including every operator (filters, aggregations,
+expressions, temporal helpers), join syntax, multi-stage queries, FK
+conventions, and worked examples — lives in the `construct_notebook_query`
+tool prompt:
+
+  `resources/metabot/prompts/tools/construct_notebook_query.md`
+
+Minimal example:
 
 ```json
 {
-  "source": {"type": "metric", "id": 10},
-  "operations": [
-    ["filter", [">", ["field", 305], "2024-01-01"]],
-    ["breakout", ["with-temporal-bucket", ["field", 305], "month"]]
-  ]
+  "query": {
+    "lib/type": "mbql/query",
+    "stages": [
+      {
+        "lib/type": "mbql.stage/mbql",
+        "source-table": ["Sample Database", "PUBLIC", "ORDERS"],
+        "aggregation": [["sum", {}, ["field", {}, ["Sample Database", "PUBLIC", "ORDERS", "TOTAL"]]]],
+        "breakout":    [["field", {}, ["Sample Database", "PUBLIC", "PEOPLE", "STATE"]]],
+        "limit": 100
+      }
+    ]
+  }
 }
 ```
 
@@ -339,185 +336,46 @@ aggregates are usually unnecessary):
 {"query": "eyJkYXRhYmFzZSI6MSwi..."}
 ```
 
-#### Source types
-
-The top-level `source` must be one of these — `context` and nested `program`
-sources are reserved for in-process callers and are rejected at the HTTP
-boundary.
-
-| Type      | Meaning                                                       |
-|-----------|---------------------------------------------------------------|
-| `table`   | Query a database table directly (`id` is a table ID)         |
-| `card`    | Query a saved question (`id` is a card ID)                   |
-| `dataset` | Query a model (`id` is the model's card ID)                  |
-| `metric`  | Query a metric, inheriting its aggregation and time dimension |
-
-#### Top-level operations
-
-Each operation is an array starting with the operator name. Operations are
-applied in order to the query stage produced from `source`.
-
-| Operator               | Shape                                           | Description                                              |
-|------------------------|-------------------------------------------------|----------------------------------------------------------|
-| `filter`               | `["filter", clause]`                            | Add a filter clause to the current stage                 |
-| `aggregate`            | `["aggregate", agg-clause]`                     | Add an aggregation                                       |
-| `breakout`             | `["breakout", ref-or-bucketed]`                 | Add a grouping dimension                                 |
-| `expression`           | `["expression", "Name", expr]`                  | Define a named computed column                           |
-| `with-fields`          | `["with-fields", [refs]]`                       | Restrict the returned columns                            |
-| `order-by`             | `["order-by", ref]` or `["order-by", ref, dir]` | Sort by a field, expression-ref, or aggregation-ref      |
-| `limit`                | `["limit", N]`                                  | Cap the number of returned rows                          |
-| `join`                 | `["join", join-clause]`                         | Join another table or card                               |
-| `append-stage`         | `["append-stage"]`                              | Start a new query stage (e.g. for post-aggregation ops)  |
-| `with-page`            | `["with-page", {"page": N, "items": M}]`        | Apply pagination on the current stage                    |
-
-The full canonical list lives in
-[src/metabase/agent_lib/capabilities/catalog/](../agent_lib/capabilities/catalog/).
-
-#### References
-
-References are nested operator forms used inside operations to point at fields,
-expressions, or earlier aggregations.
-
-| Form                              | Meaning                                                |
-|-----------------------------------|--------------------------------------------------------|
-| `["field", N]`                    | Reference a database field by ID                       |
-| `["expression-ref", "Name"]`      | Reference a named expression defined earlier           |
-| `["aggregation-ref", N]`          | Reference the Nth aggregation defined earlier (0-based)|
-| `["measure", N]`                  | Reference a pre-defined measure on the source entity   |
-| `["with-temporal-bucket", r, b]`  | Bucket a temporal field by `b` (`day`, `month`, …)     |
-
-#### Filter operators (used inside `["filter", …]`)
-
-| Operator             | Example                                              |
-|----------------------|------------------------------------------------------|
-| `=`, `!=`            | `["=", ["field", 101], "active"]`                    |
-| `<`, `<=`, `>`, `>=` | `[">", ["field", 302], 100]`                         |
-| `between`            | `["between", ["field", 305], "2024-01-01", "2024-12-31"]` |
-| `in`, `not-in`       | `["in", ["field", 302], [10, 20, 30]]`               |
-| `is-null`, `not-null`| `["is-null", ["field", 101]]`                        |
-| `is-empty`, `not-empty` | `["is-empty", ["field", 101]]`                    |
-| `contains`, `does-not-contain` | `["contains", ["field", 303], "acme"]`     |
-| `starts-with`, `ends-with` | `["starts-with", ["field", 303], "acme"]`      |
-| `time-interval`      | `["time-interval", ["field", 305], -7, "day"]`       |
-| `and`, `or`, `not`   | `["and", filter1, filter2]`                          |
-| `segment`            | `["segment", 5]` — apply a pre-defined segment        |
-
-#### Aggregation operators (used inside `["aggregate", …]`)
-
-`count`, `sum`, `avg`, `min`, `max`, `distinct`, `median`, `stddev`, `var`,
-`percentile`, `count-where`, `sum-where`, `distinct-where`, `share`,
-`cum-count`, `cum-sum`.
-
-```json
-["aggregate", ["count"]]
-["aggregate", ["sum", ["field", 302]]]
-["aggregate", ["count-where", ["=", ["field", 101], "completed"]]]
-```
-
-#### Temporal helpers (commonly used in `expression` and `breakout`)
-
-`get-year`, `get-quarter`, `get-month`, `get-week`, `get-day`,
-`get-day-of-week`, `get-hour`, `get-minute`, `datetime-add`, `datetime-diff`,
-`datetime-subtract`, `now`, `today`, `relative-datetime`, `absolute-datetime`,
-`with-temporal-bucket`, `convert-timezone`.
-
-#### Worked examples
-
-**Top 5 customers by revenue**
-
-```json
-{
-  "source": {"type": "table", "id": 42},
-  "operations": [
-    ["aggregate", ["sum", ["field", 302]]],
-    ["breakout", ["field", 101]],
-    ["order-by", ["aggregation-ref", 0], "desc"],
-    ["limit", 5]
-  ]
-}
-```
-
-**Conditional sum with a named expression**
-
-```json
-{
-  "source": {"type": "table", "id": 42},
-  "operations": [
-    ["expression", "Discount", ["-", ["field", 302], ["field", 303]]],
-    ["aggregate", ["sum", ["expression-ref", "Discount"]]]
-  ]
-}
-```
-
-**Orders per month using a metric**
-
-```json
-{
-  "source": {"type": "metric", "id": 10},
-  "operations": [
-    ["breakout", ["with-temporal-bucket", ["field", 305], "month"]]
-  ]
-}
-```
-
-**Multi-stage: filter on an aggregated value**
-
-```json
-{
-  "source": {"type": "table", "id": 42},
-  "operations": [
-    ["aggregate", ["sum", ["field", 302]]],
-    ["breakout", ["field", 101]],
-    ["append-stage"],
-    ["filter", [">", ["aggregation-ref", 0], 1000]]
-  ]
-}
-```
+The value is a base64-encoded resolved MBQL 5 query map suitable for
+`/v1/execute` or for embedding in a `/v2/query` continuation token. Treat it
+as opaque.
 
 #### Error responses
 
-Validation, repair, and resolution errors are returned as `400 Bad Request`
-with a structured JSON body:
+The representations pipeline distinguishes user-facing input errors (the
+LLM-/agent-authored query can't be resolved or doesn't validate) from internal
+failures. Input errors are returned as `400 Bad Request` with the originating
+ex-data fields surfaced in the body — most commonly `error`, `path`, and
+`candidates`. Examples:
 
-```json
-{
-  "status-code": 400,
-  "error": "invalid-generated-program",
-  "path": "operations[2].field_id",
-  "details": "operations[2].field_id: field 12345 is not accessible",
-  "recovery": {
-    "available": ["field 1001", "field 1002"],
-    "suggestion": "Did you mean field 1001?"
-  }
-}
-```
+| `error`                       | When                                                                              |
+|-------------------------------|-----------------------------------------------------------------------------------|
+| `unknown-database`            | First-stage source uses a database name that doesn't exist                        |
+| `unknown-table`               | Portable FK names a table that doesn't exist (with closest-match candidates)      |
+| `unknown-field`               | Portable FK names a column that doesn't exist on the resolved table               |
+| `unknown-card`                | `source-card:` / `[metric, {}, <eid>]` references a missing entity_id             |
+| `missing-source-in-first-stage` | Neither `source-table:` nor `source-card:` was supplied on `stages[0]`          |
+| `ambiguous-fk` / `no-fk-path` | An implicit-join field reference can't be auto-wired (multiple FKs / no FK)       |
+| `uri-in-source-table`         | `source-table:` got a `metabase://...` URI instead of a portable FK               |
 
-A non-existent table, card, or metric in `source` returns `404 Not Found`.
+All input errors carry `:agent-error? true` in the underlying ex-data; LLM
+callers are expected to read the message and self-correct on the next turn.
 
 ### POST /v2/query
 
 Combined construct-and-execute endpoint with built-in pagination via
 continuation tokens.
 
-The body is either:
-- A program (same shape as `/v2/construct-query`), **or**
-- `{"continuation_token": "..."}` returned from a previous response.
+The request body is either:
+- A representations JSON payload (`{"query": <external-query-object>}`, same
+  shape as `/v2/construct-query`), **or**
+- A continuation token from a previous response: `{"continuation_token": "..."}`.
 
-Pagination is automatic. The per-page row limit is taken from your program's
-`["limit", N]` operation if present, otherwise defaults to 200, and is hard-
-capped at 200 rows for memory and LLM-context safety. If the page is full and
-more rows may exist, the response includes a `continuation_token` you can
-post back to fetch the next page.
-
-```json
-{
-  "source": {"type": "table", "id": 42},
-  "operations": [
-    ["order-by", ["field", 101]],
-    ["limit", 50]
-  ]
-}
-```
+Pagination is automatic. The per-page row limit is taken from the query's
+`limit:` field if present, otherwise defaults to 200, and is hard-capped at 200
+rows per page for memory and LLM-context safety. If the page is full and more
+rows may exist, the response includes a `continuation_token` you post back to
+fetch the next page.
 
 Response (HTTP 202, streaming):
 
@@ -677,27 +535,28 @@ Response:
 ## Typical workflow
 
 1. **Search** — POST /v1/search to find relevant tables or metrics
-2. **Inspect** — GET /v1/table/{id} or /v1/metric/{id} to get field IDs and
-   understand the schema
+2. **Inspect** — GET /v1/table/{id} or /v1/metric/{id} to get the column-name /
+   schema / portable-FK info needed to write a query
 3. **Explore field values** — GET /v1/table/{id}/field/{field-id}/values if
    you need to know valid filter values or field statistics
-4. **Build query** — POST /v2/construct-query with a structured program
-   (`source` + `operations`)
+4. **Build query** — POST /v2/construct-query with a representations JSON
+   payload (`{"query": <external-query-object>}`); see the
+   `construct_notebook_query` tool prompt for the format reference
 5. **Execute** — POST /v1/execute with the base64-encoded query, or use
    POST /v2/query to construct and execute in one round-trip with pagination
-6. **Iterate** — Adjust the program and repeat steps 4-5
+6. **Iterate** — Adjust the query and repeat steps 4-5
 7. **Save (optional)** — POST /v1/question to persist the query as a card, and
    POST /v1/dashboard to assemble saved cards into a dashboard
 
 ## Error handling
 
-| HTTP Status | Meaning                                                       |
-|-------------|---------------------------------------------------------------|
-| 200         | Success (GET endpoints, construct-query)                      |
-| 202         | Success (execute / query — streaming response)                |
-| 400         | Invalid program (validation, repair, or resolution failure)   |
-| 401         | Authentication failure                                        |
-| 403         | Insufficient permissions                                      |
-| 404         | Entity not found                                              |
+| HTTP Status | Meaning                                                              |
+|-------------|----------------------------------------------------------------------|
+| 200         | Success (GET endpoints, construct-query)                             |
+| 202         | Success (execute / query — streaming response)                       |
+| 400         | Invalid representations query (validation, repair, or resolution failure; `:agent-error?` paths) |
+| 401         | Authentication failure                                               |
+| 403         | Insufficient permissions                                             |
+| 404         | Entity not found (GET endpoints only)                                |
 
 ---

src/metabase/agent_lib/capabilities.clj

@@ -1,120 +0,0 @@
-(ns metabase.agent-lib.capabilities
-  "Shared capability registry for the structured MBQL program path."
-  (:require
-   [metabase.agent-lib.capabilities.catalog :as catalog]
-   [metabase.agent-lib.capabilities.derived :as derived]
-   [metabase.agent-lib.capabilities.synthetic :as capabilities.synthetic]
-   [metabase.util.malli.registry :as mr]))
-
-(set! *warn-on-reflection* true)
-
-(def raw-capability-catalog
-  "Ordered structured-program capability metadata used for repair, validation, and prompts."
-  catalog/raw-capability-catalog)
-
-(def ^:private guidance-keys
-  [:prompt-forms
-   :prompt-notes
-   :shape
-   :example
-   :retry-shape
-   :retry-example
-   :retry-notes])
-
-(def ^:private capability-guidance-schema
-  [:map {:closed true}
-   [:prompt-forms {:optional true} [:sequential :string]]
-   [:prompt-notes {:optional true} [:sequential :string]]
-   [:shape {:optional true} :string]
-   [:example {:optional true} :string]
-   [:retry-shape {:optional true} :string]
-   [:retry-example {:optional true} :string]
-   [:retry-notes {:optional true} [:sequential :string]]])
-
-(def ^:private capability-entry-schema
-  [:map {:closed true}
-   [:op symbol?]
-   [:kind [:enum :top-level :nested]]
-   [:binding {:optional true} ifn?]
-   [:group {:optional true} keyword?]
-   [:arities {:optional true} [:set int?]]
-   [:runtime-provided? {:optional true} boolean?]
-   [:guidance {:optional true} capability-guidance-schema]])
-
-(defn- split-guidance
-  [capability]
-  (let [guidance (not-empty (select-keys capability guidance-keys))]
-    (cond-> (apply dissoc capability guidance-keys)
-      guidance (assoc :guidance guidance))))
-
-(def capability-catalog
-  "Ordered structured-program capability metadata keyed into structural fields plus nested guidance."
-  (mapv split-guidance raw-capability-catalog))
-
-(defn- validate-capability-catalog!
-  []
-  (doseq [capability capability-catalog]
-    (when-not (mr/validate capability-entry-schema capability)
-      (throw (ex-info "Invalid structured capability catalog entry."
-                      {:capability capability
-                       :schema     capability-entry-schema
-                       :explain    (mr/explain capability-entry-schema capability)})))))
-
-(validate-capability-catalog!)
-
-(defn guidance
-  "Return prompt and retry guidance metadata for a capability entry."
-  [capability]
-  (derived/guidance capability))
-
-(defn prompt-forms
-  "Return documented prompt forms for a capability entry."
-  [capability]
-  (derived/prompt-forms capability))
-
-(defn prompt-notes
-  "Return documented prompt notes for a capability entry."
-  [capability]
-  (derived/prompt-notes capability))
-
-(defn guidance-shape
-  "Return the documented canonical call shape for a capability entry."
-  [capability]
-  (derived/guidance-shape capability))
-
-(defn guidance-example
-  "Return the documented example program fragment for a capability entry."
-  [capability]
-  (derived/guidance-example capability))
-
-(def capability-by-op
-  "Capability metadata keyed by operator symbol."
-  (derived/capability-by-op capability-catalog))
-
-(def fixed-arities
-  "Structured helper arities keyed by operator symbol."
-  (derived/fixed-arities capability-catalog))
-
-(def trusted-helper-bindings
-  "Trusted helper implementations keyed by operator symbol."
-  (derived/trusted-helper-bindings capability-catalog))
-
-(def query-transform-symbols
-  "Top-level structured query-transform operator symbols."
-  (derived/query-transform-symbols capability-catalog))
-
-(def helper-symbols
-  "All structured helper symbols accepted by the evaluator."
-  (derived/helper-symbols capability-catalog capabilities.synthetic/synthetic-helper-symbols))
-
-(def nested-operator-values
-  "String names for nested structured helper operators."
-  (derived/nested-operator-values capability-catalog))
-
-(def top-level-operator-values
-  "String names for top-level structured operations."
-  (derived/top-level-operator-values capability-catalog))
-
-(def operator-guidance-catalog
-  "Retry guidance keyed by canonical operator name."
-  (derived/operator-guidance-catalog capability-catalog))

src/metabase/agent_lib/capabilities/catalog.clj

@@ -1,29 +0,0 @@
-(ns metabase.agent-lib.capabilities.catalog
-  "Aggregated structured-program capability catalog split into smaller
-  review-friendly groups."
-  (:require
-   [metabase.agent-lib.capabilities.catalog.aggregation :as aggregation]
-   [metabase.agent-lib.capabilities.catalog.expressions :as expressions]
-   [metabase.agent-lib.capabilities.catalog.filtering :as filtering]
-   [metabase.agent-lib.capabilities.catalog.joins :as joins]
-   [metabase.agent-lib.capabilities.catalog.ordering :as ordering]
-   [metabase.agent-lib.capabilities.catalog.sources :as sources]
-   [metabase.agent-lib.capabilities.catalog.top-level :as top-level]))
-
-(set! *warn-on-reflection* true)
-
-(def grouped-capability-catalog
-  "Capability entries grouped into review-friendly sections."
-  [{:group :top-level          :capabilities top-level/capabilities}
-   {:group :sources            :capabilities sources/capabilities}
-   {:group :filtering          :capabilities filtering/capabilities}
-   {:group :aggregation        :capabilities aggregation/capabilities}
-   {:group :breakout-ordering  :capabilities ordering/capabilities}
-   {:group :expressions        :capabilities expressions/capabilities}
-   {:group :joins              :capabilities joins/capabilities}])
-
-(def raw-capability-catalog
-  "Ordered structured-program capability metadata assembled from the grouped catalog."
-  (into []
-        (mapcat :capabilities)
-        grouped-capability-catalog))

src/metabase/agent_lib/capabilities/catalog/aggregation.clj

@@ -1,30 +0,0 @@
-(ns metabase.agent-lib.capabilities.catalog.aggregation
-  "Aggregation structured capability entries.
-
-  Catalog entries are the source of truth for which operators agent-lib supports
-  at runtime. The construct_notebook_query.md prompt is hand-curated guidance for
-  the LLM and is not generated from this catalog — keep both in sync when adding
-  or removing helpers."
-  (:require
-   [metabase.lib.core :as lib]))
-
-(set! *warn-on-reflection* true)
-
-(def capabilities
-  "Aggregation helper capability entries."
-  [{:op 'count,          :binding #'lib/count,          :kind :nested, :group :aggregation, :arities #{0}, :prompt-forms ["[\"count\"]"]}
-   {:op 'sum,            :binding #'lib/sum,            :kind :nested, :group :aggregation, :prompt-forms ["[\"sum\", [\"field\", 201]]"], :prompt-notes ["Aggregation helpers take field refs like `[\"sum\", [\"field\", 201]]`. Do not pass bare numeric ids like `[\"sum\", 201]`."]}
-   {:op 'avg,            :binding #'lib/avg,            :kind :nested, :group :aggregation, :prompt-forms ["[\"avg\", [\"field\", 201]]"]}
-   {:op 'min,            :binding #'lib/min,            :kind :nested, :group :aggregation, :prompt-forms ["[\"min\", [\"field\", 201]]"]}
-   {:op 'max,            :binding #'lib/max,            :kind :nested, :group :aggregation, :prompt-forms ["[\"max\", [\"field\", 201]]"]}
-   {:op 'distinct,       :binding #'lib/distinct,       :kind :nested, :group :aggregation, :prompt-forms ["[\"distinct\", [\"field\", 201]]"]}
-   {:op 'median,         :binding #'lib/median,         :kind :nested, :group :aggregation, :prompt-forms ["[\"median\", [\"field\", 201]]"]}
-   {:op 'stddev,         :binding #'lib/stddev,         :kind :nested, :group :aggregation, :prompt-forms ["[\"stddev\", [\"field\", 201]]"]}
-   {:op 'var,            :binding #'lib/var,            :kind :nested, :group :aggregation, :prompt-forms ["[\"var\", [\"field\", 201]]"]}
-   {:op 'percentile,     :binding #'lib/percentile,     :kind :nested, :group :aggregation, :prompt-forms ["[\"percentile\", [\"field\", 201], 0.95]"]}
-   {:op 'count-where,    :binding #'lib/count-where,    :kind :nested, :group :aggregation, :prompt-forms ["[\"count-where\", [\"=\", [\"field\", 101], \"completed\"]]"], :shape "[\"count-where\", clause]", :example "[\"count-where\", [\"=\", [\"field\", 2191], true]]"}
-   {:op 'sum-where,      :binding #'lib/sum-where,      :kind :nested, :group :aggregation, :prompt-forms ["[\"sum-where\", [\"field\", 201], [\"=\", [\"field\", 101], \"completed\"]]"], :shape "[\"sum-where\", [\"field\", 2203], clause]", :example "[\"sum-where\", [\"field\", 2203], [\"=\", [\"field\", 2191], true]]"}
-   {:op 'distinct-where, :binding #'lib/distinct-where, :kind :nested, :group :aggregation, :prompt-forms ["[\"distinct-where\", [\"field\", 201], [\"=\", [\"field\", 101], \"completed\"]]"], :shape "[\"distinct-where\", [\"field\", 2203], clause]", :example "[\"distinct-where\", [\"field\", 2203], [\"=\", [\"field\", 2191], true]]"}
-   {:op 'share,          :binding #'lib/share,          :kind :nested, :group :aggregation, :prompt-forms ["[\"share\", [\"=\", [\"field\", 101], \"completed\"]]"], :shape "[\"share\", clause]", :example "[\"share\", [\"=\", [\"field\", 2191], true]]"}
-   {:op 'cum-count,      :binding #'lib/cum-count,      :kind :nested, :group :aggregation, :prompt-forms ["[\"cum-count\"]"]}
-   {:op 'cum-sum,        :binding #'lib/cum-sum,        :kind :nested, :group :aggregation, :prompt-forms ["[\"cum-sum\", [\"field\", 201]]"]}])

src/metabase/agent_lib/capabilities/catalog/expressions.clj

@@ -1,60 +0,0 @@
-(ns metabase.agent-lib.capabilities.catalog.expressions
-  "Expression structured capability entries."
-  (:require
-   [metabase.lib.core :as lib]))
-
-(set! *warn-on-reflection* true)
-
-(def capabilities
-  "Expression helper capability entries."
-  [{:op '+,                  :binding #'lib/+,                  :kind :nested, :group :expressions, :prompt-forms ["[\"+\", [\"field\", 201], [\"field\", 202]]"]}
-   {:op '-,                  :binding #'lib/-,                  :kind :nested, :group :expressions, :prompt-forms ["[\"-\", [\"field\", 201], [\"field\", 202]]"]}
-   {:op '*,                  :binding #'lib/*,                  :kind :nested, :group :expressions, :prompt-forms ["[\"*\", [\"field\", 201], [\"field\", 202]]"]}
-   {:op '/,                  :binding #'lib//,                  :kind :nested, :group :expressions, :prompt-forms ["[\"/\", [\"field\", 201], [\"field\", 202]]"]}
-   {:op 'case,               :binding #'lib/case,               :kind :nested, :group :expressions, :arities #{1 2}, :prompt-forms ["[\"case\", [[[\"<\", [\"field\", 201], 50], \"small\"], [[\"<\", [\"field\", 201], 200], \"medium\"]]]" "[\"case\", [[[\"<\", [\"field\", 201], 50], \"small\"], [[\"<\", [\"field\", 201], 200], \"medium\"]], \"large\"]"], :shape "[\"case\", [[condition, value], ...]] or [\"case\", [[condition, value], ...], fallback]", :example "[\"case\", [[[\"<\", [\"field\", 201], 50], \"small\"], [[\"<\", [\"field\", 201], 200], \"medium\"]], \"large\"]", :retry-shape "[\"case\", [[condition, value], ...]] or [\"case\", [[condition, value], ...], fallback]", :retry-example "[\"case\", [[[\"<\", [\"field\", 201], 50], \"small\"], [[\"<\", [\"field\", 201], 200], \"medium\"]], \"large\"]", :prompt-notes ["`case` uses the fallback value itself as the optional third argument. Do not wrap it in `{\"default\": ...}`." "If you want no else branch or a null result, omit the third `case` argument entirely."], :retry-notes ["Do not wrap the fallback in `{\"default\": ...}`." "Omit the third argument instead of using `null` when you want no else branch."]}
-   {:op 'coalesce,           :binding #'lib/coalesce,           :kind :nested, :group :expressions, :prompt-forms ["[\"coalesce\", [\"field\", 101], [\"field\", 102]]"]}
-   {:op 'concat,             :binding #'lib/concat,             :kind :nested, :group :expressions, :prompt-forms ["[\"concat\", [\"field\", 101], \" \", [\"field\", 102]]"]}
-   {:op 'substring,          :binding #'lib/substring,          :kind :nested, :group :expressions, :prompt-forms ["[\"substring\", [\"field\", 101], 1, 3]"]}
-   {:op 'replace,            :binding #'lib/replace,            :kind :nested, :group :expressions, :prompt-forms ["[\"replace\", [\"field\", 101], \"old\", \"new\"]"]}
-   {:op 'upper,              :binding #'lib/upper,              :kind :nested, :group :expressions, :prompt-forms ["[\"upper\", [\"field\", 101]]"]}
-   {:op 'lower,              :binding #'lib/lower,              :kind :nested, :group :expressions, :prompt-forms ["[\"lower\", [\"field\", 101]]"]}
-   {:op 'trim,               :binding #'lib/trim,               :kind :nested, :group :expressions, :prompt-forms ["[\"trim\", [\"field\", 101]]"]}
-   {:op 'ltrim,              :binding #'lib/ltrim,              :kind :nested, :group :expressions, :prompt-forms ["[\"ltrim\", [\"field\", 101]]"]}
-   {:op 'rtrim,              :binding #'lib/rtrim,              :kind :nested, :group :expressions, :prompt-forms ["[\"rtrim\", [\"field\", 101]]"]}
-   {:op 'length,             :binding #'lib/length,             :kind :nested, :group :expressions, :prompt-forms ["[\"length\", [\"field\", 101]]"]}
-   {:op 'regex-match-first,  :binding #'lib/regex-match-first,  :kind :nested, :group :expressions, :prompt-forms ["[\"regex-match-first\", [\"field\", 101], \"pattern\"]"]}
-   {:op 'split-part,         :binding #'lib/split-part,         :kind :nested, :group :expressions, :prompt-forms ["[\"split-part\", [\"field\", 101], \",\", 1]"]}
-   {:op 'collate,            :binding #'lib/collate,            :kind :nested, :group :expressions, :prompt-forms ["[\"collate\", [\"field\", 101], \"case-insensitive\"]"]}
-   {:op 'abs,                :binding #'lib/abs,                :kind :nested, :group :expressions, :prompt-forms ["[\"abs\", [\"field\", 201]]"]}
-   {:op 'ceil,               :binding #'lib/ceil,               :kind :nested, :group :expressions, :prompt-forms ["[\"ceil\", [\"field\", 201]]"]}
-   {:op 'floor,              :binding #'lib/floor,              :kind :nested, :group :expressions, :prompt-forms ["[\"floor\", [\"field\", 201]]"]}
-   {:op 'round,              :binding #'lib/round,              :kind :nested, :group :expressions, :prompt-forms ["[\"round\", [\"field\", 201]]"]}
-   {:op 'power,              :binding #'lib/power,              :kind :nested, :group :expressions, :prompt-forms ["[\"power\", [\"field\", 201], 2]"]}
-   {:op 'sqrt,               :binding #'lib/sqrt,               :kind :nested, :group :expressions, :prompt-forms ["[\"sqrt\", [\"field\", 201]]"]}
-   {:op 'log,                :binding #'lib/log,                :kind :nested, :group :expressions, :prompt-forms ["[\"log\", [\"field\", 201]]"]}
-   {:op 'exp,                :binding #'lib/exp,                :kind :nested, :group :expressions, :prompt-forms ["[\"exp\", [\"field\", 201]]"]}
-   {:op 'today,              :binding #'lib/today,              :kind :nested, :group :expressions, :arities #{0}, :prompt-forms ["[\"today\"]"]}
-   {:op 'now,                :binding #'lib/now,                :kind :nested, :group :expressions, :arities #{0}, :prompt-forms ["[\"now\"]"]}
-   {:op 'date,               :binding #'lib/date,               :kind :nested, :group :expressions, :prompt-forms ["[\"date\", \"2025-01-01\"]"]}
-   {:op 'datetime,           :binding #'lib/datetime,           :kind :nested, :group :expressions, :prompt-forms ["[\"datetime\", \"2025-01-01T00:00:00\"]"]}
-   {:op 'interval,           :binding #'lib/interval,           :kind :nested, :group :expressions, :prompt-forms ["[\"interval\", 7, \"day\"]"]}
-   {:op 'time,               :binding #'lib/time,               :kind :nested, :group :expressions, :prompt-forms ["[\"time\", \"09:00\"]"]}
-   {:op 'relative-datetime,  :binding #'lib/relative-datetime,  :kind :nested, :group :expressions, :prompt-forms ["[\"relative-datetime\", -7, \"day\"]"]}
-   {:op 'absolute-datetime,  :binding #'lib/absolute-datetime,  :kind :nested, :group :expressions, :prompt-forms ["[\"absolute-datetime\", \"2025-01-01\", \"day\"]"], :shape "[\"absolute-datetime\", \"2024-01-01\", \"day\"]", :example "[\"absolute-datetime\", \"2024-01-01\", \"day\"]"}
-   {:op 'datetime-add,       :binding #'lib/datetime-add,       :kind :nested, :group :expressions, :prompt-forms ["[\"datetime-add\", [\"field\", 302], 1, \"month\"]"]}
-   {:op 'datetime-diff,      :binding (fn [x y unit] (lib/expression-clause :datetime-diff [x y unit] nil)), :kind :nested, :group :expressions, :prompt-forms ["[\"datetime-diff\", [\"field\", 302], [\"field\", 303], \"day\"]"], :shape "[\"datetime-diff\", [\"field\", 302], [\"field\", 303], \"day\"]", :example "[\"datetime-diff\", [\"field\", 302], [\"field\", 303], \"day\"]", :prompt-notes ["Use `datetime-diff` when you need the number of days, weeks, months, or other units between two temporal values." "Do not subtract dates with `-` when the goal is a time difference; use `datetime-diff` instead."]}
-   {:op 'datetime-subtract,  :binding #'lib/datetime-subtract,  :kind :nested, :group :expressions, :prompt-forms ["[\"datetime-subtract\", [\"now\"], [\"field\", 302], \"day\"]"]}
-   {:op 'get-year,           :binding #'lib/get-year,           :kind :nested, :group :expressions, :prompt-forms ["[\"get-year\", [\"field\", 302]]"]}
-   {:op 'get-quarter,        :binding #'lib/get-quarter,        :kind :nested, :group :expressions, :prompt-forms ["[\"get-quarter\", [\"field\", 302]]"]}
-   {:op 'get-month,          :binding #'lib/get-month,          :kind :nested, :group :expressions, :prompt-forms ["[\"get-month\", [\"field\", 302]]"]}
-   {:op 'get-week,           :binding #'lib/get-week,           :kind :nested, :group :expressions, :prompt-forms ["[\"get-week\", [\"field\", 302]]"]}
-   {:op 'get-day,            :binding #'lib/get-day,            :kind :nested, :group :expressions, :prompt-forms ["[\"get-day\", [\"field\", 302]]"]}
-   {:op 'get-day-of-week,    :binding #'lib/get-day-of-week,    :kind :nested, :group :expressions, :prompt-forms ["[\"get-day-of-week\", [\"field\", 302]]"]}
-   {:op 'get-hour,           :binding #'lib/get-hour,           :kind :nested, :group :expressions, :prompt-forms ["[\"get-hour\", [\"field\", 302]]"]}
-   {:op 'get-minute,         :binding #'lib/get-minute,         :kind :nested, :group :expressions, :prompt-forms ["[\"get-minute\", [\"field\", 302]]"]}
-   {:op 'get-second,         :binding #'lib/get-second,         :kind :nested, :group :expressions, :prompt-forms ["[\"get-second\", [\"field\", 302]]"]}
-   {:op 'convert-timezone,   :binding #'lib/convert-timezone,   :kind :nested, :group :expressions, :prompt-forms ["[\"convert-timezone\", [\"field\", 302], \"UTC\", \"America/Los_Angeles\"]"]}
-   {:op 'integer,            :binding #'lib/integer,            :kind :nested, :group :expressions, :prompt-forms ["[\"integer\", [\"field\", 201]]"]}
-   {:op 'float,              :binding #'lib/float,              :kind :nested, :group :expressions, :prompt-forms ["[\"float\", [\"field\", 201]]"]}
-   {:op 'text,               :binding #'lib/text,               :kind :nested, :group :expressions, :prompt-forms ["[\"text\", [\"field\", 101]]"]}
-   {:op 'offset,             :binding #'lib/offset,             :kind :nested, :group :expressions, :prompt-forms ["[\"offset\", [\"sum\", [\"field\", 2056]], -1]"], :shape "[\"offset\", [\"sum\", [\"field\", 2056]], -1]", :example "[\"offset\", [\"sum\", [\"field\", 2056]], -1]", :prompt-notes ["`offset` is a windowed aggregation over an aggregation or expression." "For grouped previous-period metrics like \"previous month's total\", add `offset` as another `aggregate` in the same stage as the base aggregation and breakout." "Do not add `append-stage` just to calculate a previous-period total with `offset`."] :retry-shape "For previous-period grouped metrics, keep `offset` in the same grouped stage as the base aggregation and breakout." :retry-example (str "[\n" "  [\"aggregate\", [\"sum\", [\"field\", 2056]]],\n" "  [\"aggregate\", [\"offset\", [\"sum\", [\"field\", 2056]], -1]],\n" "  [\"breakout\", [\"with-temporal-bucket\", [\"field\", 2058], \"month\"]],\n" "  [\"order-by\", [\"asc\", [\"with-temporal-bucket\", [\"field\", 2058], \"month\"]]]\n" "]")}])

src/metabase/agent_lib/capabilities/catalog/filtering.clj

@@ -1,33 +0,0 @@
-(ns metabase.agent-lib.capabilities.catalog.filtering
-  "Filtering structured capability entries."
-  (:require
-   [metabase.lib.core :as lib]))
-
-(set! *warn-on-reflection* true)
-
-(def capabilities
-  "Filtering helper capability entries."
-  [{:op '=,                  :binding #'lib/=,                 :kind :nested, :group :filtering, :prompt-forms ["[\"=\", [\"field\", 101], \"value\"]"]}
-   {:op '!=,                 :binding #'lib/!=,                :kind :nested, :group :filtering, :prompt-forms ["[\"!=\", [\"field\", 101], \"value\"]"]}
-   {:op '<,                  :binding #'lib/<,                 :kind :nested, :group :filtering, :prompt-forms ["[\"<\", [\"field\", 101], 10]"]}
-   {:op '<=,                 :binding #'lib/<=,                :kind :nested, :group :filtering, :prompt-forms ["[\"<=\", [\"field\", 101], 10]"]}
-   {:op '>,                  :binding #'lib/>,                 :kind :nested, :group :filtering, :prompt-forms ["[\">\", [\"field\", 101], 10]"]}
-   {:op '>=,                 :binding #'lib/>=,                :kind :nested, :group :filtering, :prompt-forms ["[\">=\", [\"field\", 101], 10]"]}
-   {:op 'between,            :binding #'lib/between,           :kind :nested, :group :filtering, :prompt-forms ["[\"between\", [\"field\", 101], 1, 100]"], :shape "[\"between\", [\"field\", 2205], 1, 100]", :example "[\"between\", [\"field\", 2205], 1, 100]", :prompt-notes ["If the question explicitly asks for 2024 or another exact year, use an exact 2024 restriction, not a relative current-year or last-year filter." "Using an exact helper year field like `[\"=\", [\"field\", year_field], 2024]` is acceptable when it is clearly equivalent to filtering the requested date field to 2024."]}
-   {:op 'contains,           :binding #'lib/contains,          :kind :nested, :group :filtering, :prompt-forms ["[\"contains\", [\"field\", 101], \"text\"]"]}
-   {:op 'does-not-contain,   :binding #'lib/does-not-contain,  :kind :nested, :group :filtering, :prompt-forms ["[\"does-not-contain\", [\"field\", 101], \"text\"]"], :shape "[\"does-not-contain\", [\"field\", 2204], \"trial\"]", :example "[\"does-not-contain\", [\"field\", 2204], \"trial\"]"}
-   {:op 'starts-with,        :binding #'lib/starts-with,       :kind :nested, :group :filtering, :prompt-forms ["[\"starts-with\", [\"field\", 101], \"prefix\"]"]}
-   {:op 'ends-with,          :binding #'lib/ends-with,         :kind :nested, :group :filtering, :prompt-forms ["[\"ends-with\", [\"field\", 101], \"suffix\"]"]}
-   {:op 'is-null,            :binding #'lib/is-null,           :kind :nested, :group :filtering, :prompt-forms ["[\"is-null\", [\"field\", 101]]"]}
-   {:op 'not-null,           :binding #'lib/not-null,          :kind :nested, :group :filtering, :prompt-forms ["[\"not-null\", [\"field\", 101]]"]}
-   {:op 'is-empty,           :binding #'lib/is-empty,          :kind :nested, :group :filtering, :prompt-forms ["[\"is-empty\", [\"field\", 101]]"], :shape "[\"is-empty\", [\"field\", 2204]]", :example "[\"is-empty\", [\"field\", 2204]]"}
-   {:op 'not-empty,          :binding #'lib/not-empty,         :kind :nested, :group :filtering, :prompt-forms ["[\"not-empty\", [\"field\", 101]]"], :shape "[\"not-empty\", [\"field\", 2204]]", :example "[\"not-empty\", [\"field\", 2204]]"}
-   {:op 'and,                :binding #'lib/and,               :kind :nested, :group :filtering, :prompt-forms ["[\"and\", clause1, clause2]"]}
-   {:op 'or,                 :binding #'lib/or,                :kind :nested, :group :filtering, :prompt-forms ["[\"or\", clause1, clause2]"]}
-   {:op 'not,                :binding #'lib/not,               :kind :nested, :group :filtering, :prompt-forms ["[\"not\", clause]"]}
-   {:op 'time-interval,      :binding #'lib/time-interval,     :kind :nested, :group :filtering, :prompt-forms ["[\"time-interval\", [\"field\", 101], -30, \"day\"]"]}
-   {:op 'relative-time-interval, :binding #'lib/relative-time-interval, :kind :nested, :group :filtering, :arities #{5}, :prompt-forms ["[\"relative-time-interval\", [\"field\", 101], -30, \"day\", 0, \"day\"]"], :shape "[\"relative-time-interval\", [\"field\", 2205], -30, \"day\", 0, \"day\"]", :example "[\"relative-time-interval\", [\"field\", 2205], -30, \"day\", 0, \"day\"]"}
-   {:op 'in,                 :binding #'lib/in,                :kind :nested, :group :filtering, :prompt-forms ["[\"in\", [\"field\", 101], [\"a\", \"b\"]]"], :shape "[\"in\", [\"field\", 2492], [\"authorized\", \"pending\", \"partially_paid\"]]", :example "[\"in\", [\"field\", 2492], [\"authorized\", \"pending\", \"partially_paid\"]]"}
-   {:op 'not-in,             :binding #'lib/not-in,            :kind :nested, :group :filtering, :prompt-forms ["[\"not-in\", [\"field\", 101], [\"a\", \"b\"]]"], :shape "[\"not-in\", [\"field\", 2492], [\"authorized\", \"pending\"]]", :example "[\"not-in\", [\"field\", 2492], [\"authorized\", \"pending\"]]"}
-   {:op 'inside,             :binding #'lib/inside,            :kind :nested, :group :filtering, :prompt-forms ["[\"inside\", [\"field\", 401], [\"field\", 402], 40.0, 30.0, -70.0, -80.0]"], :shape "[\"inside\", [\"field\", 401], [\"field\", 402], 40.0, 30.0, -70.0, -80.0]", :example "[\"inside\", [\"field\", 401], [\"field\", 402], 40.0, 30.0, -70.0, -80.0]"}
-   {:op 'segment,            :binding #'lib/segment,           :kind :nested, :group :filtering, :prompt-forms ["[\"segment\", 88]"], :shape "[\"segment\", 88]", :example "[\"segment\", 88]"}])

src/metabase/agent_lib/capabilities/catalog/joins.clj

@@ -1,14 +0,0 @@
-(ns metabase.agent-lib.capabilities.catalog.joins
-  "Join-related structured capability entries."
-  (:require
-   [metabase.lib.core :as lib]))
-
-(set! *warn-on-reflection* true)
-
-(def capabilities
-  "Join helper capability entries."
-  [{:op 'join-clause,          :binding #'lib/join-clause,          :kind :nested, :group :joins, :arities #{1}, :prompt-forms ["[\"join-clause\", [\"table\", 42]]" "[\"join-clause\", {\"type\": \"program\", \"program\": {...}}]"], :shape "[\"join-clause\", [\"table\", 133]]", :example "[\"join-clause\", [\"table\", 133]]"}
-   {:op 'with-join-conditions, :binding #'lib/with-join-conditions, :kind :nested, :group :joins, :prompt-forms ["[\"with-join-conditions\", [\"join-clause\", [\"table\", 42]], [[\"=\", [\"field\", 101], [\"field\", 202]]]]"], :shape "[\"with-join-conditions\", [\"join-clause\", [\"table\", 133]], [[\"=\", [\"field\", 2456], [\"field\", 2488]]]]", :example "[\"with-join-conditions\", [\"join-clause\", [\"table\", 133]], [[\"=\", [\"field\", 2456], [\"field\", 2488]]]]", :prompt-notes ["If the source-table context already exposes the related table you need, prefer direct related-field refs first." "Direct related-field refs are the default for one-hop related tables." "Use `join-clause` and `with-join-conditions` when you need custom join behavior, a join alias, explicit joined-field selection, self-joins, or direct related-field refs are unavailable." "If an explicit join returns a permission error, the underlying table is not accessible — surface the error instead of retrying with implicit refs."]}
-   {:op 'with-join-fields,     :binding #'lib/with-join-fields,     :kind :nested, :group :joins, :prompt-forms ["[\"with-join-fields\", join-clause, \"all\"]" "[\"with-join-fields\", join-clause, \"none\"]" "[\"with-join-fields\", join-clause, [[\"field\", 757]]]"], :shape "[\"with-join-fields\", join-clause, [[\"field\", 757]]]", :example "[\"with-join-fields\", [\"join-clause\", [\"table\", 133]], [[\"field\", 757]]]"}
-   {:op 'with-join-strategy,   :binding #'lib/with-join-strategy,   :kind :nested, :group :joins, :prompt-forms ["[\"with-join-strategy\", join-clause, \"left-join\"]"], :shape "[\"with-join-strategy\", [\"join-clause\", [\"table\", 133]], \"left-join\"]", :example "[\"with-join-strategy\", [\"join-clause\", [\"table\", 133]], \"left-join\"]"}
-   {:op 'with-join-alias,      :binding #'lib/with-join-alias,      :kind :nested, :group :joins, :prompt-forms ["[\"with-join-alias\", join-clause, \"alias\"]"]}])

src/metabase/agent_lib/capabilities/catalog/ordering.clj

@@ -1,13 +0,0 @@
-(ns metabase.agent-lib.capabilities.catalog.ordering
-  "Breakout and ordering structured capability entries."
-  (:require
-   [metabase.lib.core :as lib]))
-
-(set! *warn-on-reflection* true)
-
-(def capabilities
-  "Breakout-ordering helper capability entries."
-  [{:op 'with-temporal-bucket, :binding #'lib/with-temporal-bucket, :kind :nested, :group :breakout-ordering, :prompt-forms ["[\"with-temporal-bucket\", [\"field\", 302], \"month\"]"], :shape "[\"with-temporal-bucket\", [\"field\", 2205], \"month\"]", :example "[\"with-temporal-bucket\", [\"field\", 2205], \"month\"]"}
-   {:op 'with-binning,         :binding #'lib/with-binning,         :kind :nested, :group :breakout-ordering, :prompt-forms ["[\"with-binning\", [\"field\", 303], strategy]"]}
-   {:op 'asc,                  :kind :nested, :group :breakout-ordering, :runtime-provided? true, :arities #{1}, :prompt-forms ["[\"asc\", [\"field\", 301]]"]}
-   {:op 'desc,                 :kind :nested, :group :breakout-ordering, :runtime-provided? true, :arities #{1}, :prompt-forms ["[\"desc\", [\"field\", 301]]"]}])

src/metabase/agent_lib/capabilities/catalog/sources.clj

@@ -1,16 +0,0 @@
-(ns metabase.agent-lib.capabilities.catalog.sources
-  "Source and query-reference structured capability entries."
-  (:require
-   [metabase.lib.core :as lib]))
-
-(set! *warn-on-reflection* true)
-
-(def capabilities
-  "Source and query-reference capability entries."
-  [{:op 'field,           :kind :nested, :group :sources,    :runtime-provided? true, :arities #{1}, :prompt-forms ["[\"field\", 101]"], :shape "[\"field\", 101]", :example "[\"field\", 101]"}
-   {:op 'table,           :kind :nested, :group :sources,    :runtime-provided? true, :arities #{1}, :prompt-forms ["[\"table\", 42]"], :shape "[\"table\", 42]", :example "[\"table\", 42]"}
-   {:op 'card,            :kind :nested, :group :sources,    :runtime-provided? true, :arities #{1}, :prompt-forms ["[\"card\", 55]"], :shape "[\"card\", 55]", :example "[\"card\", 55]"}
-   {:op 'metric,          :kind :nested, :group :sources,    :runtime-provided? true, :arities #{1}, :prompt-forms ["[\"metric\", 77]"], :shape "[\"metric\", 77]", :example "[\"metric\", 77]"}
-   {:op 'measure,         :kind :nested, :group :sources,    :runtime-provided? true, :arities #{1}, :prompt-forms ["[\"measure\", 91]"], :shape "[\"measure\", 91]", :example "[\"measure\", 91]", :prompt-notes ["Use `measure` for a named table/model measure aggregation when it is available in the source context. Measures use real numeric ids."]}
-   {:op 'expression-ref,  :binding #'lib/expression-ref, :kind :nested, :group :query-refs, :arities #{1}, :prompt-forms ["[\"expression-ref\", \"Net Amount\"]"], :shape "[\"expression-ref\", \"Net Amount\"]", :example "[\"expression-ref\", \"Net Amount\"]"}
-   {:op 'aggregation-ref, :kind :nested, :group :query-refs, :runtime-provided? true, :arities #{1}, :prompt-forms ["[\"aggregation-ref\", 0]"], :shape "[\"aggregation-ref\", 0]", :example "[\"aggregation-ref\", 0]"}])

src/metabase/agent_lib/capabilities/catalog/top_level.clj

@@ -1,21 +0,0 @@
-(ns metabase.agent-lib.capabilities.catalog.top-level
-  "Top-level structured-program capability entries."
-  (:require
-   [metabase.lib.core :as lib]))
-
-(set! *warn-on-reflection* true)
-
-(def capabilities
-  "Top-level structured-program capability entries."
-  [{:op 'filter,            :binding #'lib/filter,            :kind :top-level, :arities #{1}, :prompt-forms ["[\"filter\", clause]"], :shape "[\"filter\", clause]", :example "[\"filter\", [\"time-interval\", [\"field\", 2058], -12, \"month\"]]"}
-   {:op 'aggregate,         :binding #'lib/aggregate,         :kind :top-level, :arities #{1}, :prompt-forms ["[\"aggregate\", aggregation]"], :shape "[\"aggregate\", [\"sum\", [\"field\", 2056]]]", :example "[\"aggregate\", [\"sum\", [\"field\", 2056]]]"}
-   {:op 'breakout,          :binding #'lib/breakout,          :kind :top-level, :arities #{1}, :prompt-forms ["[\"breakout\", breakout-form]"], :shape "[\"breakout\", [\"field\", 3134]]", :example "[\"breakout\", [\"field\", 3134]]"}
-   {:op 'with-fields,       :binding #'lib/with-fields,       :kind :top-level, :arities #{1}, :prompt-forms ["[\"with-fields\", [field-or-ref, ...]]"], :shape "[\"with-fields\", [[\"field\", 2204], [\"expression-ref\", \"Net Amount\"]]]", :example "[\"with-fields\", [[\"field\", 2204], [\"expression-ref\", \"Net Amount\"]]]"}
-   {:op 'limit,             :binding #'lib/limit,             :kind :top-level, :arities #{1}, :prompt-forms ["[\"limit\", 10]"], :shape "[\"limit\", 10]", :example "[\"limit\", 10]"}
-   {:op 'expression,        :binding #'lib/expression,        :kind :top-level, :arities #{2}, :prompt-forms ["[\"expression\", \"Name\", expr]"], :shape "[\"expression\", \"Net Amount\", [\"-\", [\"field\", 2624], [\"field\", 2641]]]", :example "[\"expression\", \"Net Amount\", [\"-\", [\"field\", 2624], [\"field\", 2641]]]", :prompt-notes ["If an `expression` uses `aggregation-ref`, add `append-stage` first. Do not define ratios over aggregations in the same stage that creates those aggregations."]}
-   {:op 'join,              :binding #'lib/join,              :kind :top-level, :arities #{1}, :prompt-forms ["[\"join\", join-clause]"], :shape "[\"join\", [\"join-clause\", [\"table\", 42]]]", :example "[\"join\", [\"join-clause\", [\"table\", 42]]]", :prompt-notes ["If a related table is already surfaced in the source-table context, first try using its field ids directly without an explicit join." "Direct related-field refs are the default for one-hop related tables." "Use an explicit join when you need custom join behavior, a join alias, explicit joined-field selection, self-joins, or direct related-field refs do not work." "If an explicit join returns a permission error, the underlying table is not accessible — surface the error instead of retrying with implicit refs."]}
-   {:op 'order-by,          :kind :top-level, :runtime-provided? true, :arities #{1 2}, :prompt-forms ["[\"order-by\", orderable]" "[\"order-by\", orderable, \"desc\"]"], :shape "[\"order-by\", [\"desc\", [\"aggregation-ref\", 0]]] or [\"order-by\", [\"aggregation-ref\", 0], \"desc\"]", :example "[\"order-by\", [\"desc\", [\"aggregation-ref\", 0]]]"}
-   {:op 'append-stage,      :binding #'lib/append-stage,      :kind :top-level, :arities #{0}, :prompt-forms ["[\"append-stage\"]"], :shape "[\"append-stage\"]", :example "[\"append-stage\"]", :prompt-notes ["If you filter on `[\"aggregation-ref\", index]`, insert `append-stage` first. Do not filter on an aggregation result in the same stage that creates it."]}
-   {:op 'drop-stage,        :binding #'lib/drop-stage,        :kind :top-level, :arities #{0}, :prompt-forms ["[\"drop-stage\"]"], :shape "[\"drop-stage\"]", :example "[\"drop-stage\"]"}
-   {:op 'drop-empty-stages, :binding #'lib/drop-empty-stages, :kind :top-level, :arities #{0}, :prompt-forms ["[\"drop-empty-stages\"]"], :shape "[\"drop-empty-stages\"]", :example "[\"drop-empty-stages\"]"}
-   {:op 'with-page,         :binding #'lib/with-page,         :kind :top-level, :arities #{1}, :prompt-forms ["[\"with-page\", {\"page\": 2, \"items\": 25}]"], :shape "[\"with-page\", {\"page\": 2, \"items\": 25}]", :example "[\"with-page\", {\"page\": 2, \"items\": 25}]", :prompt-notes ["Use JSON objects only for helpers that explicitly show one, such as `[\"with-page\", {\"page\": 2, \"items\": 25}]`."]}])

src/metabase/agent_lib/capabilities/derived.clj

@@ -1,99 +0,0 @@
-(ns metabase.agent-lib.capabilities.derived
-  "Derived views over the structured capability catalog.")
-
-(set! *warn-on-reflection* true)
-
-(defn guidance
-  "Return prompt and retry guidance metadata for a capability entry."
-  [capability]
-  (:guidance capability))
-
-(defn prompt-forms
-  "Return documented prompt forms for a capability entry."
-  [capability]
-  (get-in capability [:guidance :prompt-forms]))
-
-(defn prompt-notes
-  "Return documented prompt notes for a capability entry."
-  [capability]
-  (get-in capability [:guidance :prompt-notes]))
-
-(defn guidance-shape
-  "Return the documented canonical call shape for a capability entry."
-  [capability]
-  (get-in capability [:guidance :shape]))
-
-(defn guidance-example
-  "Return the documented example program fragment for a capability entry."
-  [capability]
-  (get-in capability [:guidance :example]))
-
-(defn capability-by-op
-  "Index capability entries by operator symbol."
-  [capability-catalog]
-  (into {}
-        (map (fn [capability]
-               [(:op capability) capability]))
-        capability-catalog))
-
-(defn fixed-arities
-  "Index declared helper arities by operator symbol."
-  [capability-catalog]
-  (into {}
-        (keep (fn [{:keys [op arities]}]
-                (when arities
-                  [op arities])))
-        capability-catalog))
-
-(defn trusted-helper-bindings
-  "Index trusted helper implementations by operator symbol."
-  [capability-catalog]
-  (into {}
-        (keep (fn [{:keys [op binding]}]
-                (when binding
-                  [op binding])))
-        capability-catalog))
-
-(defn query-transform-symbols
-  "Return the top-level query-transform operators."
-  [capability-catalog]
-  (into #{}
-        (comp (filter #(= (:kind %) :top-level))
-              (map :op))
-        capability-catalog))
-
-(defn helper-symbols
-  "Return all helper symbols accepted by the structured evaluator."
-  [capability-catalog synthetic-helper-symbols]
-  (into synthetic-helper-symbols
-        (comp (filter #(or (:binding %)
-                           (:runtime-provided? %)))
-              (map :op))
-        capability-catalog))
-
-(defn nested-operator-values
-  "Return canonical string names for nested operators."
-  [capability-catalog]
-  (into #{}
-        (comp (filter #(= (:kind %) :nested))
-              (map (comp name :op)))
-        capability-catalog))
-
-(defn top-level-operator-values
-  "Return canonical string names for top-level operations."
-  [capability-catalog]
-  (into #{}
-        (comp (filter #(= (:kind %) :top-level))
-              (map (comp name :op)))
-        capability-catalog))
-
-(defn operator-guidance-catalog
-  "Return retry guidance keyed by canonical operator name."
-  [capability-catalog]
-  (into {}
-        (keep (fn [{:keys [op] :as capability}]
-                (let [{:keys [shape example retry-shape retry-example]} (guidance capability)]
-                  (when-let [guidance-shape (or retry-shape shape)]
-                    [(name op) {:shape guidance-shape
-                                :example (or retry-example example)}]))))
-        capability-catalog))

src/metabase/agent_lib/capabilities/synthetic.clj

@@ -1,9 +0,0 @@
-(ns metabase.agent-lib.capabilities.synthetic
-  "Synthetic structured helper symbols that are accepted by agent-lib but are
-  not modeled as normal capability entries.")
-
-(set! *warn-on-reflection* true)
-
-(def synthetic-helper-symbols
-  "Synthetic helper symbols accepted by the structured evaluator."
-  '#{query})

src/metabase/agent_lib/common/coercions.clj

@@ -1,96 +0,0 @@
-(ns metabase.agent-lib.common.coercions
-  "Coercion and normalization helpers for structured MBQL programs."
-  (:require
-   [clojure.string :as str]
-   [metabase.agent-lib.common.literals :as literals]
-   [metabase.agent-lib.syntax :as syntax]
-   [metabase.util :as u]))
-
-(set! *warn-on-reflection* true)
-
-(defn quarter-label->number
-  "Convert quarter labels such as `Q1` into the corresponding integer."
-  [value]
-  (when (string? value)
-    (case (u/upper-case-en (str/trim value))
-      "Q1" 1
-      "Q2" 2
-      "Q3" 3
-      "Q4" 4
-      nil)))
-
-(defn parse-int-string
-  "Parse an integer string, returning nil when parsing fails."
-  [value]
-  (when (and (string? value)
-             (re-matches #"-?\d+" value))
-    (try
-      (Long/parseLong value)
-      (catch NumberFormatException _
-        nil))))
-
-(defn coerce-positive-int
-  "Coerce positive integer strings into integers."
-  [value]
-  (cond
-    (pos-int? value) value
-    (string? value)  (let [parsed (parse-int-string value)]
-                       (if (pos-int? parsed) parsed value))
-    :else            value))
-
-(defn coerce-non-negative-int
-  "Coerce non-negative integer strings into integers."
-  [value]
-  (cond
-    (and (int? value) (<= 0 value)) value
-    (string? value)                 (let [parsed (parse-int-string value)]
-                                      (if (and (int? parsed) (<= 0 parsed)) parsed value))
-    :else                           value))
-
-(defn direction-string?
-  "True when `value` is an order direction string or keyword."
-  [value]
-  (boolean (#{"asc" "desc"} (some-> value syntax/raw-op-name u/lower-case-en))))
-
-(defn normalize-direction
-  "Normalize a direction enum to its lowercase string form."
-  [value]
-  (some-> value syntax/raw-op-name u/lower-case-en))
-
-(defn unwrap-singleton-form
-  "Recursively unwrap singleton vectors that are not operator tuples."
-  [value]
-  (loop [value value]
-    (if (and (vector? value)
-             (= 1 (count value))
-             (not (syntax/possible-operator-tuple? value))
-             (not (literals/scalar-sequential? value)))
-      (recur (first value))
-      value)))
-
-(defn normalize-percentile-value
-  "Normalize percentiles expressed as whole-number percentages to fractions."
-  [value]
-  (if (and (number? value)
-           (> (double value) 1.0)
-           (<= (double value) 100.0))
-    (/ (double value) 100.0)
-    value))
-
-(defn normalize-join-conditions
-  "Unwrap one extra sequential level around join conditions."
-  [conditions]
-  (loop [conditions conditions]
-    (if (and (sequential? conditions)
-             (= 1 (count conditions))
-             (sequential? (first conditions))
-             (not (syntax/possible-operator-tuple? (first conditions))))
-      (recur (vec (first conditions)))
-      conditions)))
-
-(defn normalize-map-key
-  "Convert string map keys to keywords while leaving other keys unchanged."
-  [k]
-  (if (string? k)
-    (keyword k)
-    k))

src/metabase/agent_lib/common/context.clj

@@ -1,22 +0,0 @@
-(ns metabase.agent-lib.common.context
-  "Helpers derived from the evaluation context.")
-
-(set! *warn-on-reflection* true)
-
-(defn source-metric-id
-  "Return the current context source metric id when the source entity is a metric."
-  [context]
-  (let [source-entity (:source-entity context)
-        source-id     (:id source-entity)]
-    (when (and (= "metric" (:model source-entity))
-               (pos-int? source-id))
-      source-id)))
-
-(defn source-table-id
-  "Return the current context source table id when the source entity is a table."
-  [context]
-  (let [source-entity (:source-entity context)
-        source-id     (:id source-entity)]
-    (when (and (= "table" (:model source-entity))
-               (pos-int? source-id))
-      source-id)))

src/metabase/agent_lib/common/errors.clj

@@ -1,80 +0,0 @@
-(ns metabase.agent-lib.common.errors
-  "Error helpers for structured MBQL program repair, validation, and evaluation."
-  (:require
-   [clojure.string :as str]
-   [metabase.util.i18n :refer [tru]]))
-
-(set! *warn-on-reflection* true)
-
-(defn path->string
-  "Render a validation path vector into a dotted path string."
-  [path]
-  (if (empty? path)
-    "<root>"
-    (str/join ""
-              (map-indexed
-               (fn [idx segment]
-                 (cond
-                   (integer? segment) (str "[" segment "]")
-                   (zero? idx)        (name segment)
-                   :else              (str "." (name segment))))
-               path))))
-
-(defn invalid-program!
-  "Throw a standardized invalid structured program exception."
-  [path message & [data]]
-  (throw (ex-info (tru "Generated program is invalid.")
-                  (merge
-                   {:status-code 400
-                    :error       :invalid-generated-program
-                    :path        (path->string path)
-                    :details     (str (path->string path) ": " message)}
-                   data))))
-
-(def ^:private recovery-keys
-  "Keys that carry agent-recovery hints through the error chain."
-  [:available :tables :suggestion])
-
-(defn lookup-error!
-  "Throw a structured lookup error with optional recovery hints.
-  `message` is the clean, user-safe string (no schema enumeration).
-  `data` identifies the failed entity (e.g. `{:table-id 42}`).
-  `recovery` is an optional map with agent-facing hints:
-    :available  - seq of valid names/values
-    :tables     - seq of matching tables (for disambiguation)
-    :suggestion - corrective suggestion string"
-  ([message data]
-   (lookup-error! message data nil))
-  ([message data recovery]
-   (throw (ex-info message
-                   (cond-> (assoc data
-                                  :status-code 400
-                                  :error       :invalid-generated-program)
-                     (seq recovery) (assoc :recovery recovery))))))
-
-(defn wrap-runtime-error
-  "Wrap a runtime exception into the standard invalid-program shape,
-  preserving structured recovery data from the cause's ex-data."
-  [path op-name ^Throwable cause]
-  (let [cause-data (ex-data cause)
-        recovery   (or (:recovery cause-data)
-                       (not-empty (select-keys cause-data recovery-keys)))]
-    (ex-info (tru "Generated program could not be evaluated.")
-             (cond-> {:status-code 400
-                      :error       :invalid-generated-program
-                      :path        (path->string path)
-                      :details     (str (path->string path) ": " (ex-message cause))
-                      :operator    op-name}
-               recovery (assoc :recovery recovery))
-             cause)))
-
-(defn recovery-summary
-  "Format the `:recovery` map from an exception's ex-data into a string
-  suitable for LLM retry feedback. Returns nil when there are no hints."
-  [{:keys [available tables suggestion]}]
-  (let [parts (cond-> []
-                (seq available)  (conj (str "Available: " (str/join ", " (take 20 available))))
-                (seq tables)     (conj (str "Matching tables: " (str/join ", " tables)))
-                suggestion       (conj suggestion))]
-    (when (seq parts)
-      (str/join "\n" parts))))

src/metabase/agent_lib/common/literals.clj

@@ -1,84 +0,0 @@
-(ns metabase.agent-lib.common.literals
-  "Literal and structural predicates for structured MBQL programs."
-  (:require
-   [clojure.string :as str]
-   [metabase.agent-lib.syntax :as syntax]
-   [metabase.lib.hierarchy :as lib.hierarchy]
-   [metabase.lib.schema.aggregation]
-   [metabase.util :as u]))
-
-(set! *warn-on-reflection* true)
-
-(def ^:private aggregation-operator-names
-  ;; Derived from the lib schema hierarchy so new aggregation helpers are picked up automatically.
-  ;; `:aggregation` is the abstract wrapper clause and `:offset` is a windowed aggregation that
-  ;; behaves like an expression in some contexts — neither should be recognized as a plain
-  ;; aggregation helper name.
-  (into #{}
-        (comp (remove #{:aggregation :offset})
-              (map name))
-        (descendants @lib.hierarchy/hierarchy
-                     :metabase.lib.schema.aggregation/aggregation-clause-tag)))
-
-(defn scalar-literal?
-  "True when `value` is a scalar JSON-like literal accepted by the structured format."
-  [value]
-  (or (nil? value)
-      (string? value)
-      (number? value)
-      (boolean? value)))
-
-(defn non-blank-string?
-  "True when `value` is a non-blank string."
-  [value]
-  (and (string? value) (not (str/blank? value))))
-
-(defn quarter-label-string?
-  "True when `value` looks like a quarter label such as `Q1`."
-  [value]
-  (and (string? value)
-       (re-matches #"(?i)q[1-4]" (str/trim value))))
-
-(defn now-literal?
-  "True when `value` is the case-insensitive literal string `now`."
-  [value]
-  (and (string? value)
-       (= "now" (u/lower-case-en (str/trim value)))))
-
-(defn iso-date-string?
-  "True when `value` is an ISO `yyyy-mm-dd` date string."
-  [value]
-  (and (string? value)
-       (re-matches #"\d{4}-\d{2}-\d{2}" (str/trim value))))
-
-(defn null-literal?
-  "True when `value` is nil or the case-insensitive string `null`."
-  [value]
-  (or (nil? value)
-      (and (string? value)
-           (= "null" (u/lower-case-en value)))))
-
-(defn scalar-sequential?
-  "True when `value` is a sequential collection of scalar literals."
-  [value]
-  (and (sequential? value)
-       (not (syntax/possible-operator-tuple? value))
-       (every? scalar-literal? value)))
-
-(defn aggregation-form?
-  "True when `value` is a canonical aggregation helper tuple."
-  [value]
-  (and (vector? value)
-       (aggregation-operator-names (syntax/canonical-op-name (first value)))))
-
-(defn default-map?
-  "True when `value` looks like a legacy `{default ...}` case fallback map."
-  [value]
-  (and (map? value)
-       (or (contains? value :default)
-           (contains? value "default"))))
-
-(defn bare-field-id?
-  "True when `value` is a positive integer field id without a `field` wrapper."
-  [value]
-  (and (int? value) (pos? value)))

src/metabase/agent_lib/core.clj

@@ -1,14 +0,0 @@
-(ns metabase.agent-lib.core
-  "Public API for structured MBQL program repair, validation, and evaluation."
-  (:require
-   [metabase.agent-lib.eval]
-   [metabase.agent-lib.repair]
-   [metabase.agent-lib.schema]
-   [metabase.agent-lib.validate]
-   [potemkin :as p]))
-
-(p/import-vars
- [metabase.agent-lib.repair repair-program]
- [metabase.agent-lib.validate validated-program]
- [metabase.agent-lib.eval evaluate-program]
- [metabase.agent-lib.schema program-schema])

src/metabase/agent_lib/eval.clj

@@ -1,136 +0,0 @@
-(ns metabase.agent-lib.eval
-  "Structured MBQL program interpretation."
-  (:require
-   [metabase.agent-lib.common.errors :refer [invalid-program!]]
-   [metabase.agent-lib.eval.invoke :as eval.invoke]
-   [metabase.agent-lib.eval.source :as eval.source]
-   [metabase.agent-lib.eval.walker :as eval.walker]
-   [metabase.agent-lib.mbql-integration :as mbql]
-   [metabase.agent-lib.refs :as refs]
-   [metabase.agent-lib.runtime :as runtime]
-   [metabase.agent-lib.validate :as validate]
-   [metabase.agent-lib.validate.walker :as walker]
-   [metabase.lib.core :as lib]
-   [metabase.lib.metadata :as lib.metadata]
-   [metabase.util.log :as log]
-   [metabase.util.malli :as mu]))
-
-(set! *warn-on-reflection* true)
-
-(defn- skip-operation?
-  "True when an operation is a redundant implicit join that should be elided."
-  [runtime query op operation]
-  (and (= op 'join)
-       ((get (:bindings runtime) 'skip-join?) query operation)))
-
-(defn- apply-operation
-  "Evaluate a single operation's args and invoke its helper against `query`."
-  [evaluate-node runtime query op-path [raw-op & raw-args]]
-  (let [op   (runtime/op-symbol raw-op)
-        args (eval.walker/evaluate-args evaluate-node query op-path op raw-args)]
-    (eval.invoke/invoke-helper! runtime op-path op (into [query] args))))
-
-(defn- program-has-breakouts?
-  "True when the program's operations include at least one explicit breakout."
-  [program]
-  (some (fn [[op]]
-          (= "breakout" (when (string? op) op)))
-        (:operations program)))
-
-(defn- strip-default-breakouts
-  "Remove default breakouts from a metric source query when the program supplies its own.
-  Metrics include a default time dimension breakout in their query definition. When the
-  agent explicitly specifies breakout operations, these should replace — not augment — the
-  defaults."
-  [source-query program]
-  (if (and (mbql/query? source-query) (program-has-breakouts? program))
-    (lib/remove-all-breakouts source-query)
-    source-query))
-
-(defn- evaluate-program*
-  [runtime context path program program-depth]
-  (when (> program-depth walker/max-program-nesting)
-    (invalid-program! path "program source nesting exceeds maximum depth"))
-  (let [evaluate-program (fn [nested-path nested-program]
-                           (evaluate-program* runtime context nested-path nested-program (inc program-depth)))
-        evaluate-node    (fn [current-query node-path value]
-                           (eval.walker/evaluate-node evaluate-program
-                                                      runtime
-                                                      current-query
-                                                      node-path
-                                                      value))
-        resolved-source (eval.source/resolve-source evaluate-program
-                                                    runtime
-                                                    (conj path :source)
-                                                    (:source program))
-        source-query    (-> (if (mbql/query? resolved-source)
-                              resolved-source
-                              (eval.invoke/invoke-helper! runtime (conj path :source) 'query [resolved-source]))
-                            (strip-default-breakouts program))
-        final-query     (reduce (fn [query [op-idx operation]]
-                                  (let [op-path (into path [:operations op-idx])
-                                        op      (runtime/op-symbol (first operation))]
-                                    (if (skip-operation? runtime query op operation)
-                                      query
-                                      (apply-operation evaluate-node runtime query op-path operation))))
-                                source-query
-                                (map-indexed vector (:operations program)))]
-    (eval.invoke/ensure-query-result! final-query)))
-
-(defn- card-underlying-table-ids
-  "For a sequence of card metadata records, return the distinct underlying table-ids reachable
-  from each card's visible columns. Cards/metrics may unfold into one or more tables, so we
-  resolve them lazily through `lib/visible-columns`."
-  [metadata-provider cards]
-  (into #{}
-        (mapcat (fn [card]
-                  (try
-                    (->> (lib/query metadata-provider card)
-                         lib/visible-columns
-                         (keep :table-id))
-                    (catch Exception _ nil))))
-        cards))
-
-(defn- in-scope-table-ids
-  "Resolve the set of table-ids whose metadata the runtime needs in order to evaluate `program`.
-
-  Walks the program for explicitly-referenced metadata IDs, then expands them:
-   * field-ids → bulk-fetched in one query, contribute their `:table-id`;
-   * card/metric-ids → bulk-fetched, expanded via `lib/visible-columns` to underlying tables;
-   * measure-ids → bulk-fetched, contribute their `:table-id`;
-   * source-entity from the context, when it's a table.
-
-  We deliberately ignore the LLM-supplied `:referenced-entities` and `:surrounding-tables` —
-  the operations and source declared by the program are the ground truth for what metadata
-  needs to be loaded. If the agent references a table or field it forgot to declare, the
-  walker still picks it up."
-  [metadata-provider context program]
-  (let [{:keys [table-ids field-ids card-ids metric-ids measure-ids]}
-        (refs/collect-program-refs program)
-        source-table-id (when (= "table" (some-> context :source-entity :model))
-                          (some-> context :source-entity :id))
-        fields  (when (seq field-ids)
-                  (lib.metadata/bulk-metadata metadata-provider :metadata/column field-ids))
-        cards   (when (or (seq card-ids) (seq metric-ids))
-                  (lib.metadata/bulk-metadata metadata-provider :metadata/card
-                                              (into #{} cat [card-ids metric-ids])))
-        measures (when (seq measure-ids)
-                   (lib.metadata/bulk-metadata metadata-provider :metadata/measure measure-ids))]
-    (cond-> (set table-ids)
-      source-table-id  (conj source-table-id)
-      (seq fields)     (into (keep :table-id) fields)
-      (seq cards)      (into (card-underlying-table-ids metadata-provider cards))
-      (seq measures)   (into (keep :table-id) measures))))
-
-(mu/defn evaluate-program :- :map
-  "Validate and interpret a structured MBQL program."
-  [program              :- :map
-   metadata-providerable :- :any
-   context               :- validate/EvaluationContext]
-  (log/debugf "Evaluating structured program with %d operations" (count (:operations program)))
-  (let [program (validate/validated-program program context)
-        mp      (lib.metadata/->metadata-provider metadata-providerable)
-        scope   (in-scope-table-ids mp context program)
-        runtime (runtime/build-runtime mp {:in-scope-table-ids scope
-                                           :extra-bindings     {'source (:source-metadata context)}})]
-    (evaluate-program* runtime context [] program 0)))

src/metabase/agent_lib/eval/args.clj

@@ -1,141 +0,0 @@
-(ns metabase.agent-lib.eval.args
-  "Argument normalization for structured MBQL helper invocation."
-  (:require
-   [metabase.agent-lib.common.coercions :refer [normalize-percentile-value]]
-   [metabase.agent-lib.common.errors :refer [invalid-program!]]
-   [metabase.agent-lib.common.literals :refer [scalar-literal?]]))
-
-(def temporal-unit-values
-  "Allowed string enum values for temporal units and buckets."
-  #{"millisecond" "second" "minute" "hour" "day" "week" "month" "quarter" "year"})
-
-(def ^:private time-interval-amount-keywords
-  "String values for time-interval amount that should be keywordized."
-  #{"current" "last" "next"})
-
-(def order-direction-values
-  "Allowed string enum values for order direction."
-  #{"asc" "desc"})
-
-(def join-strategy-values
-  "Allowed string enum values for explicit join strategy."
-  #{"left-join" "inner-join" "right-join" "full-join"})
-
-(def join-fields-values
-  "Allowed string enum values for explicit joined-field selection."
-  #{"all" "none"})
-
-(def binning-strategy-values
-  "Allowed string enum values for binning strategy."
-  #{"bin-width" "default" "num-bins"})
-
-(set! *warn-on-reflection* true)
-
-(defn keyword-enum
-  "Normalize string enum values into keywords for lib helpers."
-  [path label value allowed-values]
-  (cond
-    (keyword? value)
-    (if (allowed-values (name value))
-      value
-      (invalid-program! path (str label " must be one of " (sort allowed-values))))
-
-    (string? value)
-    (if (allowed-values value)
-      (keyword value)
-      (invalid-program! path (str label " must be one of " (sort allowed-values))))
-
-    :else
-    (invalid-program! path (str label " must be a string enum value"))))
-
-(defn normalize-helper-args
-  "Normalize helper arguments into the forms expected by trusted lib bindings."
-  [path op args]
-  (case (name op)
-    "in"
-    (if (and (= 2 (count args))
-             (vector? (second args))
-             (every? scalar-literal? (second args)))
-      (into [(first args)] (second args))
-      args)
-
-    "not-in"
-    (if (and (= 2 (count args))
-             (vector? (second args))
-             (every? scalar-literal? (second args)))
-      (into [(first args)] (second args))
-      args)
-
-    "time-interval"
-    (-> args
-        (update 1 (fn [amount]
-                    (cond
-                      (keyword? amount)                       amount
-                      (time-interval-amount-keywords amount)  (keyword amount)
-                      :else                                   amount)))
-        (update 2 #(keyword-enum (conj path 3) "temporal unit" % temporal-unit-values)))
-
-    "relative-time-interval"
-    (-> args
-        (update 2 #(keyword-enum (conj path 3) "temporal unit" % temporal-unit-values))
-        (update 4 #(keyword-enum (conj path 5) "temporal unit" % temporal-unit-values)))
-
-    "with-temporal-bucket"
-    (update args 1 #(keyword-enum (conj path 2) "temporal bucket" % temporal-unit-values))
-
-    "with-binning"
-    (update args 1 (fn [binning]
-                     (if (and (map? binning)
-                              (some? (:strategy binning)))
-                       (update binning :strategy
-                               #(keyword-enum (conj path 2 :strategy)
-                                              "binning strategy"
-                                              %
-                                              binning-strategy-values))
-                       binning)))
-
-    "interval"
-    (update args 1 #(keyword-enum (conj path 2) "interval unit" % temporal-unit-values))
-
-    "relative-datetime"
-    (update args 1 #(keyword-enum (conj path 2) "temporal unit" % temporal-unit-values))
-
-    "absolute-datetime"
-    (update args 1 #(keyword-enum (conj path 2) "temporal unit" % temporal-unit-values))
-
-    "datetime-add"
-    (update args 2 #(keyword-enum (conj path 3) "temporal unit" % temporal-unit-values))
-
-    "datetime-diff"
-    (update args 2 #(keyword-enum (conj path 3) "temporal unit" % temporal-unit-values))
-
-    "datetime-subtract"
-    (update args 2 #(keyword-enum (conj path 3) "temporal unit" % temporal-unit-values))
-
-    "order-by"
-    (if (= 2 (count args))
-      (update args 1 #(keyword-enum (conj path 2) "order direction" % order-direction-values))
-      args)
-
-    "percentile"
-    (update args 1 normalize-percentile-value)
-
-    "with-join-strategy"
-    (update args 1 #(keyword-enum (conj path 2) "join strategy" % join-strategy-values))
-
-    "with-join-fields"
-    (update args 1 (fn [value]
-                     (cond
-                       (or (keyword? value) (string? value))
-                       (keyword-enum (conj path 2) "join fields mode" value join-fields-values)
-
-                       (sequential? value)
-                       value
-
-                       (some? value)
-                       [value]
-
-                       :else
-                       value)))
-
-    args))

src/metabase/agent_lib/eval/invoke.clj

@@ -1,84 +0,0 @@
-(ns metabase.agent-lib.eval.invoke
-  "Trusted helper invocation and error shaping for structured evaluation."
-  (:require
-   [metabase.agent-lib.common.errors :as errors :refer [invalid-program! path->string]]
-   [metabase.agent-lib.mbql-integration :as mbql]
-   [metabase.agent-lib.runtime :as runtime]
-   [metabase.util.log :as log]))
-
-(set! *warn-on-reflection* true)
-
-(defn invoke-helper!
-  "Invoke a helper that does not depend on the current query state."
-  [runtime path op args]
-  (when-not (contains? runtime/helper-symbols op)
-    (invalid-program! path "unknown operator" {:operator (name op)}))
-  (let [helper (runtime/helper-fn runtime op)]
-    (try
-      (apply helper args)
-      (catch clojure.lang.ArityException e
-        (invalid-program! path
-                          (str "invalid arity for operator `" (name op) "`: " (ex-message e))
-                          {:operator (name op)}))
-      (catch Exception e
-        (log/debugf e "Helper `%s` at %s threw unexpectedly" (name op) (path->string path))
-        (throw (errors/wrap-runtime-error path (name op) e))))))
-
-(defn invoke-query-aware-helper!
-  "Invoke a helper that requires the current query state."
-  [runtime current-query path op args]
-  (when-not current-query
-    (invalid-program! path
-                      (format "`%s` requires the current query state" (name op))
-                      {:operator (name op)}))
-  (let [helper  (runtime/helper-fn runtime op)
-        op-name (name op)]
-    (try
-      (apply helper current-query args)
-      (catch clojure.lang.ArityException e
-        (invalid-program! path
-                          (str "invalid arity for operator `" op-name "`: " (ex-message e))
-                          {:operator op-name}))
-      (catch Exception e
-        (case op-name
-          "expression-ref"
-          (invalid-program! path
-                            (format "expression-ref %s does not exist at this point in the query"
-                                    (pr-str (first args)))
-                            {:operator "expression-ref"
-                             :name     (first args)})
-
-          "aggregation-ref"
-          (invalid-program! path
-                            (format "aggregation-ref %s does not exist at this point in the query"
-                                    (first args))
-                            {:operator "aggregation-ref"
-                             :index    (first args)})
-
-          (do (log/debugf e "Query-aware helper `%s` at %s threw unexpectedly" op-name (path->string path))
-              (throw (errors/wrap-runtime-error path op-name e))))))))
-
-(defn invoke-field-helper!
-  "Invoke the special `field` helper, which can resolve relative to the current query."
-  [runtime current-query path args]
-  (if current-query
-    (let [helper (runtime/helper-fn runtime 'field)]
-      (try
-        (apply helper current-query args)
-        (catch clojure.lang.ArityException e
-          (invalid-program! path
-                            (str "invalid arity for operator `field`: " (ex-message e))
-                            {:operator "field"}))
-        (catch Exception e
-          (log/debugf e "Field helper at %s threw unexpectedly" (path->string path))
-          (throw (errors/wrap-runtime-error path "field" e)))))
-    (invoke-helper! runtime path 'field args)))
-
-(defn ensure-query-result!
-  "Assert that structured evaluation produced a MBQL 5 query."
-  [result]
-  (when-not (mbql/query? result)
-    (throw (ex-info "Structured program evaluation must return a MBQL 5 query map."
-                    {:result-type (type result)
-                     :result      result})))
-  result)

src/metabase/agent_lib/eval/source.clj

@@ -1,29 +0,0 @@
-(ns metabase.agent-lib.eval.source
-  "Structured source resolution for program evaluation."
-  (:require
-   [metabase.agent-lib.common.errors :refer [invalid-program!]]
-   [metabase.agent-lib.runtime :as runtime]))
-
-(set! *warn-on-reflection* true)
-
-(defn resolve-source
-  "Resolve a structured program source into metadata or a query."
-  [evaluate-program runtime source-path source]
-  (case (:type source)
-    "context"
-    (or (get (runtime/bindings-map runtime) 'source)
-        (invalid-program! source-path "context source is unavailable in runtime bindings"))
-
-    "table"
-    ((runtime/helper-fn runtime 'table) (:id source))
-
-    ("card" "dataset")
-    ((runtime/helper-fn runtime 'card) (:id source))
-
-    "metric"
-    ((runtime/helper-fn runtime 'metric) (:id source))
-
-    "program"
-    (evaluate-program (conj source-path :program) (:program source))
-
-    (invalid-program! (conj source-path :type) "unsupported source type")))

src/metabase/agent_lib/eval/walker.clj

@@ -1,89 +0,0 @@
-(ns metabase.agent-lib.eval.walker
-  "Recursive structured-value evaluation for agent-lib.
-
-  The local `letfn` in `evaluate-node` is intentional: this namespace is the
-  recursive evaluation boundary for nested structured program values."
-  (:require
-   [metabase.agent-lib.common.coercions :refer [normalize-map-key]]
-   [metabase.agent-lib.common.errors :refer [invalid-program!]]
-   [metabase.agent-lib.common.literals :refer [scalar-literal?]]
-   [metabase.agent-lib.eval.args :as eval.args]
-   [metabase.agent-lib.eval.invoke :as eval.invoke]
-   [metabase.agent-lib.runtime :as runtime]
-   [metabase.agent-lib.schema :as schema]
-   [metabase.agent-lib.syntax :as syntax]))
-
-(set! *warn-on-reflection* true)
-
-(defn evaluate-args
-  "Evaluate and normalize helper arguments in the current query context."
-  [eval-node-fn current-query node-path op raw-args]
-  (->> raw-args
-       (map-indexed (fn [idx arg]
-                      (eval-node-fn current-query (conj node-path (inc idx)) arg)))
-       vec
-       (eval.args/normalize-helper-args node-path op)))
-
-(defn evaluate-node
-  "Evaluate a nested structured value in the current query context."
-  [evaluate-program runtime current-query node-path value]
-  (letfn [(walk [current-query node-path value]
-            (cond
-              (scalar-literal? value)
-              value
-
-              (map? value)
-              (if (schema/program-literal? value)
-                (evaluate-program (conj node-path :program) (:program value))
-                (into {}
-                      (map (fn [[k v]]
-                             [(normalize-map-key k)
-                              (walk current-query (conj node-path (normalize-map-key k)) v)]))
-                      value))
-
-              (vector? value)
-              (if (syntax/operator-tuple? value)
-                (let [[raw-op & raw-args] value
-                      op                 (runtime/op-symbol raw-op)
-                      op-name            (name op)]
-                  (cond
-                    (syntax/top-level-operation-symbols op)
-                    (invalid-program! node-path
-                                      (format "`%s` is a top-level operation and cannot be nested" op-name)
-                                      {:operator op-name})
-
-                    (= op 'query)
-                    (invalid-program! node-path
-                                      "`query` cannot appear inside operations; the program source already defines the base query."
-                                      {:operator "query"})
-
-                    (= op 'field)
-                    (eval.invoke/invoke-field-helper! runtime
-                                                      current-query
-                                                      node-path
-                                                      (evaluate-args walk current-query node-path 'field raw-args))
-
-                    ('#{expression-ref aggregation-ref} op)
-                    (eval.invoke/invoke-query-aware-helper! runtime
-                                                            current-query
-                                                            node-path
-                                                            op
-                                                            (evaluate-args walk current-query node-path op raw-args))
-
-                    :else
-                    (eval.invoke/invoke-helper! runtime
-                                                node-path
-                                                op
-                                                (evaluate-args walk current-query node-path op raw-args))))
-                (mapv (fn [[idx element]]
-                        (walk current-query (conj node-path idx) element))
-                      (map-indexed vector value)))
-
-              (sequential? value)
-              (mapv (fn [[idx element]]
-                      (walk current-query (conj node-path idx) element))
-                    (map-indexed vector value))
-
-              :else
-              value))]
-    (walk current-query node-path value)))

src/metabase/agent_lib/join_spec.clj

@@ -1,80 +0,0 @@
-(ns metabase.agent-lib.join-spec
-  "Shared parsing helpers for explicit join clauses."
-  (:require
-   [metabase.agent-lib.common.coercions :refer [normalize-direction]]
-   [metabase.agent-lib.syntax :as syntax]))
-
-(set! *warn-on-reflection* true)
-
-(defn program-field-ref-id
-  "Return the field id when `form` is a canonical `[\"field\", id]` tuple."
-  [form]
-  (when (and (vector? form)
-             (= "field" (syntax/canonical-op-name (first form)))
-             (pos-int? (second form)))
-    (second form)))
-
-(defn parse-join-condition-pair
-  "Parse a join equality condition into a `[lhs-id rhs-id]` pair."
-  [condition]
-  (when (and (vector? condition)
-             (= "=" (syntax/canonical-op-name (first condition)))
-             (= 3 (count condition)))
-    (when-let [lhs-id (program-field-ref-id (second condition))]
-      (when-let [rhs-id (program-field-ref-id (nth condition 2))]
-        [lhs-id rhs-id]))))
-
-(defn parse-join-spec
-  "Parse nested join-clause wrappers into a normalized join-spec map."
-  [form]
-  (when (vector? form)
-    (let [[raw-op & args] form
-          op-name         (syntax/canonical-op-name raw-op)]
-      (case op-name
-        "join-clause"
-        (let [target (first args)]
-          (when (and (vector? target)
-                     (= "table" (syntax/canonical-op-name (first target)))
-                     (pos-int? (second target)))
-            {:target-table-id (second target)
-             :conditions      nil
-             :fields-mode     nil
-             :strategy        nil
-             :alias           nil}))
-
-        "with-join-conditions"
-        (when-let [parsed (parse-join-spec (first args))]
-          (let [conditions (second args)
-                pairs      (when (sequential? conditions)
-                             (mapv parse-join-condition-pair conditions))]
-            (when (and (seq pairs)
-                       (every? some? pairs))
-              (assoc parsed :conditions pairs))))
-
-        "with-join-fields"
-        (when-let [parsed (parse-join-spec (first args))]
-          (assoc parsed :fields-mode (second args)))
-
-        "with-join-strategy"
-        (when-let [parsed (parse-join-spec (first args))]
-          (assoc parsed :strategy (normalize-direction (second args))))
-
-        "with-join-alias"
-        (when-let [parsed (parse-join-spec (first args))]
-          (assoc parsed :alias (second args)))
-
-        nil))))
-
-(defn no-explicit-join-fields?
-  "True when a join spec does not explicitly request joined fields."
-  [fields-mode]
-  (or (nil? fields-mode)
-      (= "none" fields-mode)
-      (= :none fields-mode)))
-
-(defn implicit-join-compatible-fields-mode?
-  "True when explicit join fields still match implicit related-field semantics."
-  [fields-mode]
-  (or (no-explicit-join-fields? fields-mode)
-      (= "all" fields-mode)
-      (= :all fields-mode)))

src/metabase/agent_lib/mbql_integration.clj

@@ -1,76 +0,0 @@
-(ns metabase.agent-lib.mbql-integration
-  "Temporary bridge between agent-lib and canonical MBQL lib APIs.
-
-  This namespace is the stable facade over the smaller MBQL integration units
-  under `metabase.agent-lib.mbql-integration.*`. It remains the single handoff
-  surface that agent-lib callers depend on, while the actual implementation is
-  split by concern."
-  (:require
-   [metabase.agent-lib.mbql-integration.common]
-   [metabase.agent-lib.mbql-integration.fields]
-   [metabase.agent-lib.mbql-integration.joins]
-   [metabase.agent-lib.mbql-integration.orderables]
-   [potemkin :as p]))
-
-(set! *warn-on-reflection* true)
-
-(p/import-vars
- [metabase.agent-lib.mbql-integration.common
-  query?
-  extract-field-ids
-  expression-definition
-  normalized-name
-  column-names
-  dedupe-candidate-columns
-  current-query-field-candidates
-  prefer-single-candidate
-  unique-query-candidate
-  field-selection?
-  resolve-aggregation-selection]
- [metabase.agent-lib.mbql-integration.fields
-  fields-by-table-id
-  field-target-table-id
-  field-path-start-ids
-  fk-path-to-table
-  numeric-type?
-  types-compatible?
-  candidate-has-resolution-lineage?
-  source-column-field-name
-  source-column-field-join-alias
-  source-column-original-field-id
-  source-column-original-field-name
-  source-column-original-field-join-alias
-  candidate-join-alias-for-field-id
-  synthesize-chained-related-field
-  multi-hop-lineage-candidate?
-  previous-stage-name-matches
-  previous-stage-lineage-matches
-  previous-stage-aggregation-matches
-  resolve-field-in-query]
- [metabase.agent-lib.mbql-integration.orderables
-  matching-expression-ref
-  matching-previous-stage-column-by-name
-  current-stage-expression-ref
-  expression-ref-or-current-stage-column
-  ensure-query-expression-ref
-  orderable-column-key
-  current-query-orderable-candidates
-  column-field-ids
-  breakout-expression-name
-  previous-stage-aggregation-column
-  aggregation-ref-or-current-stage-column
-  aggregation-signature
-  aggregation-columns
-  aggregation-column-pairs
-  requested-orderable-field-ids
-  orderable-field-id->query-columns
-  orderable-field-ids->query-columns
-  field-id->breakout-expression-refs
-  field-ids->breakout-expression-ref
-  field-id->aggregation-columns
-  field-ids->aggregation-columns
-  resolve-field-like-orderable
-  resolve-orderable]
- [metabase.agent-lib.mbql-integration.joins
-  implicitly-resolved-column?
-  redundant-implicit-join?])

src/metabase/agent_lib/mbql_integration/common.clj

@@ -1,266 +0,0 @@
-(ns metabase.agent-lib.mbql-integration.common
-  "Common MBQL bridge helpers shared by agent-lib field, filter, and orderable integration."
-  (:require
-   [metabase.lib.core :as lib]
-   [metabase.lib.util :as lib.util]
-   [metabase.util :as u]))
-
-(set! *warn-on-reflection* true)
-
-(defn query?
-  "Return true when `value` is a normalized MBQL 5 query map.
-
-  Agent-lib needs to distinguish queries from metadata maps, columns, and test
-  sentinels, but it should not inspect `:lib/type` directly. This helper keeps
-  that boundary on top of `lib/normalized-query-type` and guards the lib call so
-  non-query maps fail closed instead of blowing up.
-
-  Example:
-    (query? (lib/query metadata-provider table-metadata))
-    ;; => true
-
-    (query? {:id 1 :name \"Orders\"})
-    ;; => false"
-  [value]
-  (and (map? value)
-       (= :mbql/query (lib/normalized-query-type value))))
-
-(defn extract-field-ids
-  "Return the set of field ids referenced anywhere inside a MBQL 5 form.
-
-  Agent-lib compares expressions, aggregations, breakouts, and columns
-  semantically. The canonical lib helper already knows how to walk MBQL safely,
-  so this wrapper keeps callers out of tuple destructuring.
-
-  Example:
-    (extract-field-ids
-      [:+
-       {}
-       [:field {} 10]
-       [:field {} 20]])
-    ;; => #{10 20}"
-  [x]
-  (lib/all-field-ids x))
-
-(defn expression-definition
-  "Return `x` when it is a named expression definition, else nil.
-
-  Agent-lib accepts both fully defined expressions and references to existing
-  expressions. The MBQL layer already owns the naming rules, so this helper just
-  asks lib whether the form has an expression name and otherwise leaves the
-  structure alone.
-
-  Example:
-    (expression-definition
-      [:expression {} \"Net Amount\"
-       [:- {} [:field {} 1] [:field {} 2]]])
-    ;; => that same tuple
-
-    (expression-definition [:field {} 1])
-    ;; => nil"
-  [x]
-  (when (lib.util/expression-name x)
-    x))
-
-(defn normalized-name
-  "Normalize a display name for case-insensitive matching.
-
-  Agent-lib compares program-provided names against MBQL column names, aliases,
-  and ref names. Matching needs to be locale-stable, so this goes through
-  `lower-case-en` instead of generic lowercase.
-
-  Example:
-    (normalized-name \"Net Amount\")
-    ;; => \"net amount\"
-
-    (normalized-name nil)
-    ;; => nil"
-  [value]
-  (when (string? value)
-    (u/lower-case-en value)))
-
-(defn column-names
-  "Return every comparable name by which a query column may be recognized.
-
-  The same semantic column can surface through several lib metadata keys:
-  `:name`, `:lib/source-column-alias`, `:lib/original-name`,
-  `:lib/deduplicated-name`, and `:lib/ref-name`. Agent-lib collapses those into
-  one normalized set so later matching logic can ask one question: \"does this
-  column look like the requested thing?\"
-
-  Example:
-    (column-names {:name \"Title\"
-                   :lib/source-column-alias \"product_title\"
-                   :lib/ref-name \"Products -> Title\"})
-    ;; => #{\"title\" \"product_title\" \"products -> title\"}"
-  [column]
-  (->> [(:name column)
-        (:lib/source-column-alias column)
-        (:lib/original-name column)
-        (:lib/deduplicated-name column)
-        (:lib/ref-name column)]
-       (keep normalized-name)
-       set))
-
-(defn column-join-alias
-  "Return the join alias attached to a visible/query column, if any.
-
-  Lib can surface the active alias on `:lib/join-alias` or preserve the
-  original alias on `:lib/original-join-alias`. Agent-lib only needs one answer
-  when deciding whether a match came from an explicit join.
-
-  Example:
-    (column-join-alias {:lib/join-alias \"Products\"})
-    ;; => \"Products\""
-  [column]
-  ((some-fn :lib/join-alias :lib/original-join-alias) column))
-
-(defn- candidate-column-key
-  [column]
-  [(:id column)
-   (:table-id column)
-   (:name column)
-   (:source-field column)
-   (:source-field-name column)
-   (:source-field-join-alias column)
-   (column-join-alias column)
-   (:fk-field-id column)
-   (:fk-field-name column)
-   (:fk-join-alias column)
-   (:lib/original-fk-field-id column)
-   (:lib/original-fk-field-name column)
-   (:lib/original-fk-join-alias column)
-   (:lib/source column)
-   (:lib/source-uuid column)
-   (:lib/expression-name column)
-   (:lib/temporal-unit column)
-   (:inherited-temporal-unit column)])
-
-(defn dedupe-candidate-columns
-  "Deduplicate candidate columns while preserving the first structurally
-  distinct representative.
-
-  Agent-lib merges columns from several lib surfaces such as `visible-columns`,
-  `fieldable-columns`, and `orderable-columns`. The same semantic column often
-  appears more than once. We collapse duplicates, but we keep join alias, FK
-  lineage, source UUID, expression name, and temporal metadata in the key so we
-  do not merge columns that only look similar.
-
-  Example:
-    (dedupe-candidate-columns
-      [{:id 10 :name \"Title\"}
-       {:id 10 :name \"Title\"}
-       {:id 10 :name \"Title\" :lib/join-alias \"Products\"}])
-    ;; => keeps the plain column and the joined column"
-  [columns]
-  (vals
-   (reduce (fn [acc column]
-             (let [k (candidate-column-key column)]
-               (if (contains? acc k)
-                 acc
-                 (assoc acc k column))))
-           {}
-           columns)))
-
-(defn current-query-field-candidates
-  "Return the deduplicated field-like columns visible from the current query
-  stage.
-
-  Agent programs name semantic fields, but the current stage may expose them as
-  visible columns, fieldable columns, filterable columns, breakoutable columns,
-  or orderable columns. This helper builds the combined search space used by
-  field resolution.
-
-  Example:
-    A field that is hidden from the result set but still filterable remains
-    resolvable because this helper includes `lib/filterable-columns`."
-  [query]
-  (->> [(lib/fieldable-columns query)
-        (lib/visible-columns query)
-        (lib/filterable-columns query)
-        (lib/breakoutable-columns query)
-        (lib/orderable-columns query)]
-       (apply concat)
-       dedupe-candidate-columns
-       vec))
-
-(defn prefer-single-candidate
-  "Return the single deterministic winner from an otherwise ambiguous candidate
-  set, or nil when agent-lib should keep looking.
-
-  This is the extracted tie-breaker used by `unique-query-candidate`:
-  prefer one unaliased column over joined columns, then prefer one explicit
-  field over an FK-derived expansion.
-
-  Example:
-    Given `[Orders.ID, Products.ID via implicit join]`, the unaliased
-    `Orders.ID` candidate wins."
-  [candidates]
-  (or (when-let [unaliased (not-empty (remove column-join-alias candidates))]
-        (when-not (next unaliased)
-          (first unaliased)))
-      (when-let [explicit (not-empty (remove :fk-field-id candidates))]
-        (when-not (next explicit)
-          (first explicit)))))
-
-(defn unique-query-candidate
-  "Return a single candidate after applying agent-lib's tie-break rules.
-
-  Some lib lookups intentionally return more than one plausible column. When
-  that happens, agent-lib prefers:
-  1. a single unaliased column over a joined one
-  2. a single column without FK lineage over a related-field expansion
-
-  This keeps agent behavior deterministic without forcing callers to know the
-  ranking policy.
-
-  Example:
-    If candidates are `[Orders.ID, Products.ID via implicit join]`, the plain
-    `Orders.ID` column wins."
-  [candidates]
-  (or (when (= 1 (count candidates))
-        (first candidates))
-      (prefer-single-candidate (vec candidates))))
-
-(defn field-selection?
-  "Return true when `selection` behaves like a field for `with-fields`.
-
-  `with-fields` in agent-lib accepts both literal field clauses and lib column
-  maps that can be turned back into field refs. This helper centralizes that
-  check so callers do not branch on raw ref tags.
-
-  Example:
-    (field-selection? [:field {} 10])
-    ;; => true
-
-    (field-selection? {:id 10 :name \"Title\"})
-    ;; => true when `(lib/ref ...)` yields a field ref"
-  [selection]
-  (or (lib.util/field-clause? selection)
-      (when (map? selection)
-        (let [selection-ref (try
-                              (lib/ref selection)
-                              (catch Exception _
-                                nil))]
-          (boolean
-           (or (some-> selection-ref lib/field-ref-id)
-               (some-> selection-ref lib/field-ref-name)))))))
-
-(defn resolve-aggregation-selection
-  "Resolve an aggregation selection back to the visible current-stage column
-  when one exists.
-
-  Some agent operations conceptually choose \"the aggregated result column\",
-  while the syntax still names it as an aggregation ref. When lib can map that
-  ref to the visible column, we use the column so downstream helpers treat it
-  like any other selected field.
-
-  Example:
-    Given a query with `(lib/aggregate (lib/count))`, resolving
-    `(lib/aggregation-ref query 0)` returns the visible count column instead of
-    leaving the raw ref in place."
-  [query selection]
-  (if (lib.util/clause-of-type? selection :aggregation)
-    (or (lib/find-visible-column-for-ref query selection)
-        selection)
-    selection))

src/metabase/agent_lib/mbql_integration/field_resolution.clj

@@ -1,350 +0,0 @@
-(ns metabase.agent-lib.mbql-integration.field-resolution
-  "Field-resolution helpers for the agent-lib MBQL bridge."
-  (:require
-   [clojure.set :as set]
-   [clojure.string :as str]
-   [metabase.agent-lib.mbql-integration.common :as common]
-   [metabase.lib.core :as lib]
-   [metabase.lib.util :as lib.util]))
-
-(set! *warn-on-reflection* true)
-
-(defn fields-by-table-id
-  "Group metadata fields by table id for lookup-heavy resolution helpers."
-  [fields-by-id]
-  (reduce-kv (fn [acc _field-id field]
-               (update acc (:table-id field) (fnil conj []) field))
-             {}
-             fields-by-id))
-
-(defn field-target-table-id
-  "Return the table id reached by following a field's FK target metadata."
-  [fields-by-id field-id]
-  (when-let [target-field-id (some-> (get fields-by-id field-id) :fk-target-field-id)]
-    (:table-id (get fields-by-id target-field-id))))
-
-(defn field-path-start-ids
-  "Return the field ids that can anchor lineage for a surfaced query column."
-  [column]
-  (->> [(:source-field column)
-        (:fk-field-id column)
-        (:lib/original-fk-field-id column)
-        (:id column)]
-       (filter pos-int?)
-       distinct))
-
-(defn fk-path-to-table
-  "Return the unique FK chain from `start-field-id` to `target-table-id`, or
-  nil when none or more than one plausible path exists.
-
-  The local `walk-path` stays nested because it is the only genuinely recursive
-  helper left in this file."
-  [fields-by-id start-field-id target-table-id]
-  (let [fields-by-table (fields-by-table-id fields-by-id)]
-    (letfn [(walk-path [field-id visited depth]
-              (when (and (pos-int? field-id)
-                         (not (visited field-id))
-                         (< depth 4))
-                (when-let [next-table-id (field-target-table-id fields-by-id field-id)]
-                  (if (= next-table-id target-table-id)
-                    [[field-id]]
-                    (not-empty
-                     (mapcat (fn [candidate]
-                               (map #(into [field-id] %)
-                                    (or (walk-path (:id candidate)
-                                                   (conj visited field-id)
-                                                   (inc depth))
-                                        [])))
-                             (get fields-by-table next-table-id)))))))]
-      (let [paths (vec (walk-path start-field-id #{} 0))]
-        (when (= 1 (count paths))
-          (first paths))))))
-
-(defn numeric-type?
-  "True when `type` is one of the numeric MBQL types agent-lib treats as
-  interchangeable for previous-stage reuse."
-  [type]
-  (contains? #{"type/BigInteger"
-               "type/Decimal"
-               "type/Float"
-               "type/Integer"
-               "type/Number"}
-             (cond
-               (keyword? type) (str (namespace type) "/" (name type))
-               (string? type) type
-               :else nil)))
-
-(defn types-compatible?
-  "True when `raw-field` and `candidate` can safely stand in for each other by
-  type."
-  [raw-field candidate]
-  (let [raw-type       ((some-fn :effective-type :base-type) raw-field)
-        candidate-type ((some-fn :effective-type :base-type) candidate)]
-    (or (nil? raw-type)
-        (nil? candidate-type)
-        (= raw-type candidate-type)
-        (and (numeric-type? raw-type)
-             (numeric-type? candidate-type)))))
-
-(defn candidate-has-resolution-lineage?
-  "True when a candidate column carries lineage metadata that makes it a better
-  resolution target than a bare id/name match."
-  [column]
-  (boolean
-   (or (:fk-field-id column)
-       (:fk-field-name column)
-       (:fk-join-alias column)
-       (:lib/original-fk-field-id column)
-       (:lib/original-fk-field-name column)
-       (:lib/original-fk-join-alias column)
-       (common/column-join-alias column)
-       (:lib/source-uuid column)
-       (:lib/expression-name column)
-       (= :source/previous-stage (:lib/source column)))))
-
-(defn source-column-field-name
-  "Return the best human-readable field name carried by a surfaced source
-  column."
-  [column]
-  ((some-fn :lib/source-column-alias
-            :name
-            :lib/original-name
-            :fk-field-name
-            :lib/original-fk-field-name)
-   column))
-
-(defn source-column-field-join-alias
-  "Return the most specific join alias associated with a surfaced source
-  column."
-  [column]
-  ((some-fn common/column-join-alias :fk-join-alias :lib/original-fk-join-alias) column))
-
-(defn source-column-original-field-id
-  "Return the original field id carried by a surfaced source column."
-  [column]
-  ((some-fn :lib/original-fk-field-id :source-field) column))
-
-(defn source-column-original-field-name
-  "Return the original field name carried by a surfaced source column."
-  [column]
-  ((some-fn :lib/original-fk-field-name :source-field-name) column))
-
-(defn source-column-original-field-join-alias
-  "Return the original join alias carried by a surfaced source column."
-  [column]
-  ((some-fn :lib/original-fk-join-alias :source-field-join-alias) column))
-
-(defn candidate-join-alias-for-field-id
-  "Return a unique join alias for columns whose lineage includes `field-id`."
-  [candidates field-id]
-  (when (pos-int? field-id)
-    (let [matches (->> candidates
-                       (keep (fn [candidate]
-                               (when (some #{field-id} (field-path-start-ids candidate))
-                                 ((some-fn common/column-join-alias
-                                           :fk-join-alias
-                                           :lib/original-fk-join-alias)
-                                  candidate))))
-                       distinct
-                       vec)]
-      (when (= 1 (count matches))
-        (first matches)))))
-
-(defn synthesize-chained-related-field
-  "Project lineage from a surfaced source column back onto the requested raw
-  field shape.
-
-  This is the agent-lib-specific bridge for multi-hop related fields until lib
-  exposes a canonical helper for the same behavior."
-  [fields-by-id candidates raw-field source-column path]
-  (let [immediate-field-id   (last path)
-        immediate-field      (get fields-by-id immediate-field-id)
-        original-field-id    (or (source-column-original-field-id source-column)
-                                 (when (> (count path) 1)
-                                   (first path)))
-        original-field       (get fields-by-id original-field-id)
-        original-field-name  (or (source-column-original-field-name source-column)
-                                 (:name original-field))
-        immediate-field-name (or (:name immediate-field)
-                                 (source-column-field-name source-column))
-        immediate-join-alias (or (source-column-field-join-alias source-column)
-                                 (candidate-join-alias-for-field-id candidates immediate-field-id)
-                                 (candidate-join-alias-for-field-id candidates original-field-id))
-        original-join-alias  (or (source-column-original-field-join-alias source-column)
-                                 (candidate-join-alias-for-field-id candidates original-field-id)
-                                 immediate-join-alias)]
-    (cond-> raw-field
-      immediate-field-id (assoc :fk-field-id immediate-field-id)
-      immediate-field-name (assoc :fk-field-name immediate-field-name)
-      immediate-join-alias (assoc :fk-join-alias immediate-join-alias)
-      original-field-id (assoc :lib/original-fk-field-id original-field-id)
-      original-field-name (assoc :lib/original-fk-field-name original-field-name)
-      original-join-alias (assoc :lib/original-fk-join-alias original-join-alias))))
-
-(defn multi-hop-lineage-candidate?
-  "True when a synthesized related field captures more than one hop of lineage."
-  [column]
-  (boolean
-   (or (let [immediate-field-id (:fk-field-id column)
-             original-field-id  (:lib/original-fk-field-id column)]
-         (and (pos-int? immediate-field-id)
-              (pos-int? original-field-id)
-              (not= immediate-field-id original-field-id)))
-       (let [immediate-field-name (:fk-field-name column)
-             original-field-name  (:lib/original-fk-field-name column)]
-         (and (string? immediate-field-name)
-              (string? original-field-name)
-              (not= immediate-field-name original-field-name)))
-       (let [immediate-join-alias (:fk-join-alias column)
-             original-join-alias  (:lib/original-fk-join-alias column)]
-         (and (string? immediate-join-alias)
-              (string? original-join-alias)
-              (not= immediate-join-alias original-join-alias))))))
-
-(defn previous-stage-name-matches
-  "Return previous-stage candidates whose comparable names overlap the requested
-  raw field."
-  [raw-field previous-stage-candidates]
-  (let [raw-field-names (common/column-names raw-field)]
-    (->> previous-stage-candidates
-         (filter (fn [candidate]
-                   (seq (set/intersection raw-field-names (common/column-names candidate)))))
-         vec)))
-
-(defn previous-stage-lineage-matches
-  "Return previous-stage candidates whose lineage points back to `raw-field`.
-
-  This is stricter than `types-compatible?`: appended stages may expose only
-  aggregations, and agent-lib should only treat those as substitutes for a raw
-  field when lib metadata preserves a real source-field lineage for that field."
-  [raw-field previous-stage-candidates]
-  (let [raw-field-id (:id raw-field)]
-    (->> previous-stage-candidates
-         (filter (fn [candidate]
-                   (and (pos-int? raw-field-id)
-                        (some #{raw-field-id} (field-path-start-ids candidate)))))
-         vec)))
-
-(defn previous-stage-aggregation-matches
-  "Return previous-stage columns whose `:lib/source-uuid` comes from an
-  aggregation over `raw-field`."
-  [query raw-field previous-stage-candidates]
-  (when-let [previous-stage-number (some-> query (lib.util/previous-stage-number -1))]
-    (let [raw-field-id   (:id raw-field)
-          matching-uuids (->> (map vector
-                                   (lib/aggregations query previous-stage-number)
-                                   (lib/aggregations-metadata query previous-stage-number))
-                              (keep (fn [[aggregation metadata]]
-                                      (when (and (pos-int? raw-field-id)
-                                                 (some #{raw-field-id} (common/extract-field-ids aggregation)))
-                                        (:lib/source-uuid metadata))))
-                              set)]
-      (->> previous-stage-candidates
-           (filter #(contains? matching-uuids (:lib/source-uuid %)))
-           vec))))
-
-(defn- chained-related-candidates
-  "Compute chained related-field candidates by walking FK paths from each
-  candidate back to the raw field's table."
-  [fields-by-id candidates raw-field]
-  (let [target-table-id (:table-id raw-field)]
-    (->> candidates
-         (keep (fn [column]
-                 (when-let [path (some #(fk-path-to-table fields-by-id % target-table-id)
-                                       (field-path-start-ids column))]
-                   (synthesize-chained-related-field fields-by-id candidates raw-field column path))))
-         common/dedupe-candidate-columns
-         vec)))
-
-(defn- previous-stage-candidates
-  "Compute previous-stage candidates that are type-compatible with `raw-field`."
-  [candidates raw-field]
-  (->> candidates
-       (filter #(and (= :source/previous-stage (:lib/source %))
-                     (types-compatible? raw-field %)))
-       vec))
-
-(defn- try-strategies
-  "Try resolution strategies in priority order, returning the first match."
-  [strategies]
-  (some (fn [{:keys [resolve-fn]}] (resolve-fn)) strategies))
-
-(defn- summarize-candidate
-  "Compact shape of a candidate column for inclusion in field-resolution errors."
-  [candidate]
-  (let [field-name (source-column-field-name candidate)
-        join-alias (source-column-field-join-alias candidate)]
-    (cond-> {}
-      (:id candidate)       (assoc :id (:id candidate))
-      field-name            (assoc :name field-name)
-      (:table-id candidate) (assoc :table-id (:table-id candidate))
-      join-alias            (assoc :join-alias join-alias))))
-
-(defn- format-candidate-line
-  [{:keys [id name table-id join-alias]}]
-  (str "- " (or name "?")
-       " (id=" (or id "?")
-       (when table-id (str ", table-id=" table-id))
-       (when join-alias (str ", join-alias=" (pr-str join-alias)))
-       ")"))
-
-(def ^:private max-candidates-in-error 20)
-
-(defn- unavailable-field-error
-  [raw-field query candidates]
-  (let [summaries    (mapv summarize-candidate candidates)
-        shown        (take max-candidates-in-error summaries)
-        extra        (max 0 (- (count summaries) max-candidates-in-error))
-        fields-block (if (seq shown)
-                       (str "Fields available in this stage:\n"
-                            (str/join "\n" (mapv format-candidate-line shown))
-                            (when (pos? extra) (str "\n... and " extra " more")))
-                       "No fields are visible in this stage.")]
-    (ex-info (str "Field " (pr-str (:name raw-field))
-                  " (id " (pr-str (:id raw-field))
-                  ") is not available in the current query stage.\n"
-                  fields-block)
-             {:field raw-field
-              :query query
-              :available-fields summaries})))
-
-(defn resolve-field-in-query
-  "Resolve a metadata field against what the current query stage actually
-  exposes.
-
-  Resolution strategies are tried in priority order; each is computed lazily
-  so we stop as soon as one succeeds."
-  ([fields-by-id query raw-field]
-   (resolve-field-in-query fields-by-id query raw-field (common/current-query-field-candidates query)))
-  ([fields-by-id query raw-field candidates]
-   (let [query      (when (common/query? query) query)
-         by-id+tbl  (fn [] (filterv #(and (= (:id raw-field) (:id %))
-                                          (= (:table-id raw-field) (:table-id %)))
-                                    candidates))
-         by-id      (fn [] (filterv #(= (:id raw-field) (:id %)) candidates))
-         lined      (fn [cs] (filterv candidate-has-resolution-lineage? cs))
-         chained    (delay (chained-related-candidates fields-by-id candidates raw-field))
-         prev-stage (delay (previous-stage-candidates candidates raw-field))]
-     (or (try-strategies
-          [{:name :multi-hop-chained-lineage
-            :resolve-fn #(common/unique-query-candidate (filterv multi-hop-lineage-candidate? @chained))}
-           {:name :exact-id+table-with-lineage
-            :resolve-fn #(common/unique-query-candidate (lined (by-id+tbl)))}
-           {:name :exact-id-with-lineage
-            :resolve-fn #(common/unique-query-candidate (lined (by-id)))}
-           {:name :previous-stage-lineage-or-aggregation
-            :resolve-fn #(common/unique-query-candidate
-                          (->> (concat (previous-stage-lineage-matches raw-field @prev-stage)
-                                       (previous-stage-aggregation-matches query raw-field @prev-stage))
-                               common/dedupe-candidate-columns vec))}
-           {:name :previous-stage-name
-            :resolve-fn #(common/unique-query-candidate (previous-stage-name-matches raw-field @prev-stage))}
-           {:name :chained-related-fields
-            :resolve-fn #(common/unique-query-candidate @chained)}
-           {:name :lib-find-matching-column
-            :resolve-fn #(when query (lib/find-matching-column query -1 raw-field candidates))}
-           {:name :exact-id+table
-            :resolve-fn #(common/unique-query-candidate (by-id+tbl))}
-           {:name :exact-id
-            :resolve-fn #(common/unique-query-candidate (by-id))}])
-         (throw (unavailable-field-error raw-field query candidates))))))

src/metabase/agent_lib/mbql_integration/fields.clj

@@ -1,29 +0,0 @@
-(ns metabase.agent-lib.mbql-integration.fields
-  "Facade over field-resolution MBQL bridge helpers."
-  (:require
-   [metabase.agent-lib.mbql-integration.field-resolution]
-   [potemkin :as p]))
-
-(set! *warn-on-reflection* true)
-
-(p/import-vars
- [metabase.agent-lib.mbql-integration.field-resolution
-  fields-by-table-id
-  field-target-table-id
-  field-path-start-ids
-  fk-path-to-table
-  numeric-type?
-  types-compatible?
-  candidate-has-resolution-lineage?
-  source-column-field-name
-  source-column-field-join-alias
-  source-column-original-field-id
-  source-column-original-field-name
-  source-column-original-field-join-alias
-  candidate-join-alias-for-field-id
-  synthesize-chained-related-field
-  multi-hop-lineage-candidate?
-  previous-stage-name-matches
-  previous-stage-lineage-matches
-  previous-stage-aggregation-matches
-  resolve-field-in-query])

src/metabase/agent_lib/mbql_integration/joins.clj

@@ -1,60 +0,0 @@
-(ns metabase.agent-lib.mbql-integration.joins
-  "Explicit-vs-implicit join integration helpers for the agent-lib MBQL bridge."
-  (:require
-   [metabase.agent-lib.join-spec :as join-spec]
-   [metabase.agent-lib.mbql-integration.common :as common]
-   [metabase.agent-lib.mbql-integration.fields :as fields]
-   [metabase.agent-lib.syntax :as syntax]))
-
-(set! *warn-on-reflection* true)
-
-(defn implicitly-resolved-column?
-  "True when `column` represents the target side of an implicit join strongly
-  enough that agent-lib can treat an explicit join request as redundant."
-  [column target-table-id]
-  (boolean
-   (and (map? column)
-        (or (= (:table-id column) target-table-id)
-            (= :source/previous-stage (:lib/source column)))
-        (or (:fk-field-id column)
-            (:lib/original-fk-field-id column)
-            (= :source/previous-stage (:lib/source column)))
-        (nil? (:join-alias column)))))
-
-(defn- valid-implicit-join-spec?
-  "True when a parsed join spec has the shape of a potentially redundant implicit join."
-  [{:keys [target-table-id conditions fields-mode strategy alias]}]
-  (and (pos-int? target-table-id)
-       (nil? alias)
-       (or (nil? strategy) (= "left-join" strategy))
-       (join-spec/implicit-join-compatible-fields-mode? fields-mode)
-       (= 1 (count conditions))))
-
-(defn- target-field-for-join
-  "Return the field metadata for the side of the join condition that points at the
-  target table, or nil."
-  [fields-by-id target-table-id [lhs-id rhs-id]]
-  (let [lhs-table-id (some-> (get fields-by-id lhs-id) :table-id)
-        rhs-table-id (some-> (get fields-by-id rhs-id) :table-id)
-        field-id     (cond
-                       (= lhs-table-id target-table-id) lhs-id
-                       (= rhs-table-id target-table-id) rhs-id)]
-    (some-> field-id fields-by-id)))
-
-(defn redundant-implicit-join?
-  "Return true when a structured explicit join would be redundant because the
-  current query already exposes the target side through implicit join behavior."
-  [fields-by-id current-query operation]
-  (let [join-spec    (when (and (common/query? current-query)
-                                (vector? operation)
-                                (= "join" (syntax/canonical-op-name (first operation)))
-                                (= 2 (count operation)))
-                       (join-spec/parse-join-spec (second operation)))
-        target-field (when (and join-spec (valid-implicit-join-spec? join-spec))
-                       (target-field-for-join fields-by-id
-                                              (:target-table-id join-spec)
-                                              (first (:conditions join-spec))))
-        target-col   (when target-field
-                       (fields/resolve-field-in-query fields-by-id current-query target-field))]
-    (when target-col
-      (implicitly-resolved-column? target-col (:target-table-id join-spec)))))

src/metabase/agent_lib/mbql_integration/orderables.clj

@@ -1,361 +0,0 @@
-(ns metabase.agent-lib.mbql-integration.orderables
-  "Expression and orderable resolution helpers for the agent-lib MBQL bridge."
-  (:require
-   [clojure.string :as str]
-   [metabase.agent-lib.common.errors :as errors]
-   [metabase.agent-lib.mbql-integration.common :as common]
-   [metabase.lib.core :as lib]
-   [metabase.lib.expression :as lib.expression]
-   [metabase.lib.ref :as lib.ref]
-   [metabase.lib.schema.util :as lib.schema.util]
-   [metabase.lib.util :as lib.util]))
-
-(set! *warn-on-reflection* true)
-
-(def ^:private aggregation-orderable-ops
-  #{:count
-    :sum
-    :avg
-    :min
-    :max
-    :distinct
-    :median
-    :stddev
-    :var
-    :count-where
-    :sum-where
-    :distinct-where
-    :share
-    :percentile
-    :cum-count
-    :cum-sum
-    :metric})
-
-(defn- normalize-expression-comparison
-  "Strip volatile metadata from an expression tree before structural
-  comparison."
-  [expr]
-  (cond
-    (map? expr)
-    (-> expr
-        (dissoc :lib/uuid :lib/expression-name :display-name :name)
-        (update-vals normalize-expression-comparison))
-
-    (vector? expr)
-    (mapv normalize-expression-comparison expr)
-
-    (sequential? expr)
-    (mapv normalize-expression-comparison expr)
-
-    :else
-    expr))
-
-(defn matching-expression-ref
-  "Return the expression ref for the current query expression that is
-  semantically equal to `orderable`."
-  [query orderable]
-  (let [normalized-orderable (normalize-expression-comparison
-                              (lib.schema.util/remove-lib-uuids orderable))]
-    (some (fn [expr]
-            (when-let [expr-name (lib.util/expression-name expr)]
-              (when (= normalized-orderable
-                       (normalize-expression-comparison
-                        (lib.schema.util/remove-lib-uuids expr)))
-                (lib/expression-ref query expr-name))))
-          (lib/expressions query))))
-
-(defn matching-previous-stage-column-by-name
-  "Return a unique previous-stage column whose visible names match
-  `column-name`."
-  [query column-name]
-  (let [normalized-column-name (common/normalized-name column-name)]
-    (when normalized-column-name
-      (->> [(lib/returned-columns query)
-            (common/current-query-field-candidates query)]
-           (apply concat)
-           common/dedupe-candidate-columns
-           (filter #(and (= :source/previous-stage (:lib/source %))
-                         (contains? (common/column-names %) normalized-column-name)))
-           vec
-           common/unique-query-candidate))))
-
-(defn current-stage-expression-ref
-  "Return an expression ref when `expr-name` still resolves as a current-stage
-  expression name."
-  [query expr-name]
-  (when (lib.expression/maybe-resolve-expression query expr-name)
-    (lib/expression-ref query expr-name)))
-
-(defn expression-ref-or-current-stage-column
-  "Resolve an expression name to the thing the current stage can actually order
-  or project by."
-  [query expr-name]
-  (or (current-stage-expression-ref query expr-name)
-      (matching-previous-stage-column-by-name query expr-name)
-      (errors/lookup-error! (str "Expression not found in the current query stage: " (pr-str expr-name))
-                            {:expression-name expr-name})))
-
-(defn ensure-query-expression-ref
-  "Ensure a query contains an expression definition for `expression`, returning
-  `[query expr-ref]`."
-  [query expression prefix]
-  (if-let [expr-ref (matching-expression-ref query expression)]
-    [query expr-ref]
-    (let [expr-name (str "__" prefix "_expression_" (inc (count (lib/expressions query))))
-          query'    (lib/expression query expr-name expression)]
-      [query' (lib/expression-ref query' expr-name)])))
-
-(defn orderable-column-key
-  "Return the structural identity used when deduplicating orderable columns."
-  [column]
-  [(:id column)
-   (:name column)
-   (:lib/source column)
-   (:lib/source-uuid column)
-   (:source-field column)
-   (:fk-field-id column)
-   (:lib/original-fk-field-id column)
-   (:lib/expression-name column)])
-
-(defn current-query-orderable-candidates
-  "Return the distinct orderable columns for the current stage."
-  [query]
-  (vals
-   (reduce (fn [acc column]
-             (assoc acc (orderable-column-key column) column))
-           {}
-           (lib/orderable-columns query))))
-
-(defn column-field-ids
-  "Return every field id represented by a column's lineage metadata."
-  [column]
-  (->> [(:id column)
-        (:source-field column)
-        (:fk-field-id column)
-        (:lib/original-fk-field-id column)]
-       (filter pos-int?)
-       set))
-
-(defn breakout-expression-name
-  "Return the expression name for a breakout when the breakout is an
-  expression-ref."
-  [breakout]
-  (when (lib.util/clause-of-type? breakout :expression)
-    (lib.ref/expression-ref-name breakout)))
-
-(defn previous-stage-aggregation-column
-  "Resolve an aggregation index to the column that represents that aggregation
-  in the previous stage."
-  [query ag-index]
-  (when-let [previous-stage-number (lib.util/previous-stage-number query -1)]
-    (let [candidate-columns (concat (lib/returned-columns query)
-                                    (lib/filterable-columns query)
-                                    (lib/orderable-columns query)
-                                    (lib/visible-columns query))]
-      (or (try
-            (lib/find-matching-column query
-                                      -1
-                                      (lib/aggregation-ref query previous-stage-number ag-index)
-                                      candidate-columns)
-            (catch Exception _
-              nil))
-          (when-let [source-uuid (:lib/source-uuid
-                                  (get (vec (lib/aggregations-metadata query previous-stage-number))
-                                       ag-index))]
-            (some #(when (= source-uuid (:lib/source-uuid %))
-                     %)
-                  candidate-columns))))))
-
-(defn aggregation-ref-or-current-stage-column
-  "Resolve an aggregation index to the right current-stage orderable or
-  projectable value."
-  [query ag-index]
-  (if (get (vec (lib/aggregations query)) ag-index)
-    (lib/aggregation-ref query ag-index)
-    (or (previous-stage-aggregation-column query ag-index)
-        (errors/lookup-error! (str "Undefined aggregation " ag-index)
-                              {:aggregation-index ag-index
-                               :stage-number      -1}))))
-
-(defn aggregation-signature
-  "Return the semantic signature used to compare aggregation definitions."
-  [aggregation]
-  [(first aggregation) (common/extract-field-ids aggregation)])
-
-(defn aggregation-columns
-  "Return the current-stage orderable columns backed by aggregation outputs."
-  [query]
-  (filterv #(= (:lib/source %) :source/aggregations)
-           (lib/orderable-columns query)))
-
-(defn aggregation-column-pairs
-  "Pair each current-stage aggregation with the orderable column that lib maps
-  it to, when one exists."
-  [query]
-  (let [agg-cols (aggregation-columns query)]
-    (->> (lib/aggregations query)
-         (map-indexed (fn [idx aggregation]
-                        [idx aggregation
-                         (lib/find-matching-column query -1 (lib/aggregation-ref query idx) agg-cols)]))
-         (filter #(nth % 2))
-         vec)))
-
-(defn requested-orderable-field-ids
-  "Return the field ids semantically referenced by an orderable request."
-  [requested]
-  (cond
-    (map? requested)
-    (column-field-ids requested)
-
-    (lib.util/field-clause? requested)
-    (common/extract-field-ids requested)
-
-    :else
-    #{}))
-
-(defn orderable-field-id->query-columns
-  "Return current-stage orderable columns whose lineage includes `field-id`."
-  [query field-id]
-  (->> (current-query-orderable-candidates query)
-       (filterv #(contains? (column-field-ids %) field-id))))
-
-(defn orderable-field-ids->query-columns
-  "Return unique current-stage orderable columns whose lineage overlaps any of
-  `field-ids`."
-  [query field-ids]
-  (->> field-ids
-       (mapcat #(orderable-field-id->query-columns query %))
-       (reduce (fn [acc column]
-                 (assoc acc (orderable-column-key column) column))
-               {})
-       vals
-       vec))
-
-(defn field-id->breakout-expression-refs
-  "Return breakout expression refs whose underlying expression touches
-  `field-id`."
-  [query field-id]
-  (let [expr-fields (->> (lib/expressions query)
-                         (keep (fn [expr]
-                                 (when-let [expr-name (lib.util/expression-name expr)]
-                                   [expr-name (common/extract-field-ids expr)])))
-                         (into {}))]
-    (->> (lib/breakouts query)
-         (keep breakout-expression-name)
-         (filter #(contains? (get expr-fields % #{}) field-id))
-         distinct
-         (mapv #(lib/expression-ref query %)))))
-
-(defn field-ids->breakout-expression-ref
-  "Return the unique breakout expression ref implied by `field-ids`, if one can
-  be chosen unambiguously."
-  [query field-ids]
-  (let [matches (->> field-ids
-                     (mapcat #(field-id->breakout-expression-refs query %))
-                     (reduce (fn [acc expr-ref]
-                               (assoc acc (pr-str expr-ref) expr-ref))
-                             {})
-                     vals
-                     vec)]
-    (when (= 1 (count matches))
-      (first matches))))
-
-(defn field-id->aggregation-columns
-  "Return aggregation output columns whose defining aggregation references
-  `field-id`."
-  [query field-id]
-  (->> (aggregation-column-pairs query)
-       (keep (fn [[_idx aggregation column]]
-               (when (contains? (common/extract-field-ids aggregation) field-id)
-                 column)))
-       vec))
-
-(defn field-ids->aggregation-columns
-  "Return unique aggregation output columns referenced by any of `field-ids`."
-  [query field-ids]
-  (->> field-ids
-       (mapcat #(field-id->aggregation-columns query %))
-       (reduce (fn [acc column]
-                 (assoc acc (orderable-column-key column) column))
-               {})
-       vals
-       vec))
-
-(defn- sole-match
-  "Return `[query match]` when `candidates` has exactly one element, else nil."
-  [query candidates]
-  (when (= 1 (count candidates))
-    [query (first candidates)]))
-
-(defn resolve-field-like-orderable
-  "Resolve a field-like orderable request against the current query stage."
-  [query requested]
-  (let [field-ids   (requested-orderable-field-ids requested)
-        query-match (orderable-field-ids->query-columns query field-ids)]
-    (or (sole-match query query-match)
-        (when-let [ref (field-ids->breakout-expression-ref query field-ids)]
-          [query ref])
-        (sole-match query (field-ids->aggregation-columns query field-ids))
-        (sole-match query (aggregation-columns query))
-        [query requested])))
-
-(defn- resolve-aggregation-orderable
-  "Resolve an aggregation-like orderable by progressively narrowing candidates."
-  [query orderable]
-  (let [agg-cols     (aggregation-columns query)
-        op-name      (name (first orderable))
-        by-name      (filterv #(str/starts-with? (or (:name %) "") op-name) agg-cols)
-        target-sig   (aggregation-signature orderable)
-        by-signature (when (> (count by-name) 1)
-                       (->> (aggregation-column-pairs query)
-                            (keep (fn [[_idx aggregation column]]
-                                    (when (and (some #{column} by-name)
-                                               (= target-sig (aggregation-signature aggregation)))
-                                      column)))
-                            vec))]
-    (or (sole-match query agg-cols)
-        (sole-match query by-name)
-        (sole-match query by-signature)
-        (cond
-          (> (count by-signature) 1)
-          (errors/lookup-error! (str "Ambiguous: multiple identical '" op-name "' aggregations.")
-                                {:operator op-name}
-                                {:available  (mapv :name agg-cols)
-                                 :suggestion "Use orderable-columns to pick the specific column."})
-
-          (> (count by-name) 1)
-          (errors/lookup-error! (str "No orderable column matched aggregation '" op-name "' on those fields.")
-                                {:operator op-name}
-                                {:available (mapv :name agg-cols)})
-
-          :else
-          (errors/lookup-error! (str "No orderable column found for aggregation '" op-name "'.")
-                                {:operator op-name}
-                                {:available (mapv :name agg-cols)})))))
-
-(defn resolve-orderable
-  "Resolve an agent-program orderable against the current query, returning
-  `[query resolved-orderable]`."
-  [query orderable]
-  (cond
-    (map? orderable)
-    (resolve-field-like-orderable query orderable)
-
-    (and (lib.util/clause? orderable)
-         (contains? aggregation-orderable-ops (first orderable)))
-    (resolve-aggregation-orderable query orderable)
-
-    (lib.util/field-clause? orderable)
-    (resolve-field-like-orderable query orderable)
-
-    (lib.util/clause-of-type? orderable :expression)
-    [query orderable]
-
-    (lib.util/clause-of-type? orderable :aggregation)
-    [query orderable]
-
-    (lib.util/clause? orderable)
-    (ensure-query-expression-ref query orderable "order_by")
-
-    :else
-    [query orderable]))

src/metabase/agent_lib/refs.clj

@@ -1,83 +0,0 @@
-(ns metabase.agent-lib.refs
-  "Walk a structured program and collect every metadata id (table, field, card, metric, measure)
-  it actually references. Used by the runtime builder to scope metadata loading to entities the
-  program touches, instead of materializing the whole database."
-  (:require
-   [metabase.agent-lib.schema :as schema]))
-
-(set! *warn-on-reflection* true)
-
-(def ^:private empty-refs
-  {:table-ids   #{}
-   :field-ids   #{}
-   :card-ids    #{}
-   :metric-ids  #{}
-   :measure-ids #{}})
-
-(declare collect-program-refs)
-
-(defn- collect-source-refs
-  [acc source]
-  (if (and (map? source) (:type source))
-    (case (:type source)
-      "table"             (cond-> acc (pos-int? (:id source)) (update :table-ids conj (:id source)))
-      ("card" "dataset")  (cond-> acc (pos-int? (:id source)) (update :card-ids conj (:id source)))
-      "metric"            (cond-> acc (pos-int? (:id source)) (update :metric-ids conj (:id source)))
-      "program"           (if (:program source) (collect-program-refs acc (:program source)) acc)
-      acc)
-    acc))
-
-(defn- ref-key-for-op
-  "Map a reference operator name to the refs key it populates."
-  [op]
-  (case op
-    "table"   :table-ids
-    "field"   :field-ids
-    "card"    :card-ids
-    "metric"  :metric-ids
-    "measure" :measure-ids
-    nil))
-
-(defn- collect-form
-  [acc form]
-  (cond
-    ;; Operator tuple — `["op" arg ...]`. Reference ops have an int id as their first arg
-    ;; (after repair-normalize); other ops we just recurse into.
-    (and (vector? form) (string? (first form)))
-    (let [[op & args] form
-          ref-key     (ref-key-for-op op)]
-      (if (and ref-key (pos-int? (first args)))
-        (-> (update acc ref-key conj (first args))
-            ;; Still recurse into remaining args — `(field id)` has none, but
-            ;; e.g. `(with-temporal-bucket (field id) "month")` is a different op
-            ;; whose recursion is handled by the non-ref branch below.
-            (cond-> (next args) (as-> $ (reduce collect-form $ (next args)))))
-        (reduce collect-form acc args)))
-
-    ;; Nested program literal — `{:program {...}}`.
-    (and (map? form) (schema/program-literal? form))
-    (collect-program-refs acc (:program form))
-
-    ;; Plain map — recurse into values (covers per-stage option maps, etc.).
-    (map? form)
-    (reduce-kv (fn [a _k v] (collect-form a v)) acc form)
-
-    ;; Sequential — recurse element-wise.
-    (sequential? form)
-    (reduce collect-form acc form)
-
-    :else acc))
-
-(defn collect-program-refs
-  "Walk `program` and return `{:table-ids #{...} :field-ids #{...} :card-ids #{...}
-  :metric-ids #{...} :measure-ids #{...}}` containing every numeric metadata id referenced
-  anywhere in its source or operations.
-
-  Assumes the program has been through `repair/repair-program` so that all `[op id]` reference
-  forms carry positive integer ids."
-  ([program]
-   (collect-program-refs empty-refs program))
-  ([acc program]
-   (-> acc
-       (collect-source-refs (:source program))
-       (as-> $ (reduce collect-form $ (:operations program))))))

src/metabase/agent_lib/repair.clj

@@ -1,10 +0,0 @@
-(ns metabase.agent-lib.repair
-  "Deterministic repair helpers for structured MBQL programs."
-  (:require
-   [metabase.agent-lib.repair.context]
-   [metabase.agent-lib.repair.normalize]
-   [potemkin :as p]))
-
-(p/import-vars
- [metabase.agent-lib.repair.normalize repair-program]
- [metabase.agent-lib.repair.context repair-program-for-context])

src/metabase/agent_lib/repair/context.clj

@@ -1,20 +0,0 @@
-(ns metabase.agent-lib.repair.context
-  "Context-specific structured-program repair passes."
-  (:require
-   [metabase.agent-lib.repair.context.passes :as passes]
-   [metabase.agent-lib.repair.stages :as stages]))
-
-(set! *warn-on-reflection* true)
-
-(defn repair-program-for-context
-  "Apply deterministic repairs that depend on the evaluation context."
-  [program context]
-  (let [source     (passes/effective-source-for-context program context)
-        operations (->> (:operations program)
-                        (passes/drop-source-metric-stage-aggregates context)
-                        (passes/drop-source-metric-stage-with-fields context)
-                        (map #(passes/normalize-source-metric-quarter-window context %))
-                        (passes/remove-redundant-operations context)
-                        vec)]
-    (assoc program
-           :operations (stages/insert-stage-boundaries source operations))))

src/metabase/agent_lib/repair/context/passes.clj

@@ -1,193 +0,0 @@
-(ns metabase.agent-lib.repair.context.passes
-  "Context-dependent repair passes for structured programs."
-  (:require
-   [metabase.agent-lib.common.coercions :refer [coerce-positive-int]]
-   [metabase.agent-lib.common.context :as common.context]
-   [metabase.agent-lib.common.literals :refer [now-literal?]]
-   [metabase.agent-lib.join-spec :as join-spec]
-   [metabase.agent-lib.repair.stages :as stages]
-   [metabase.agent-lib.syntax :as syntax]
-   [metabase.util :as u]))
-
-(set! *warn-on-reflection* true)
-
-(defn effective-source-for-context
-  "Replace a context source with the concrete metric source when needed."
-  [program context]
-  (let [source (:source program)]
-    (if (and (map? source)
-             (= "context" (:type source))
-             (common.context/source-metric-id context))
-      {:type "metric"
-       :id   (common.context/source-metric-id context)}
-      source)))
-
-(def ^:private max-tree-walk-depth
-  "Maximum depth for tree walks in repair passes."
-  64)
-
-(defn operation-contains-canonical-op?
-  "True when `operation` contains `target-op-name` anywhere in its tree."
-  [operation target-op-name]
-  (letfn [(walk [value depth]
-            (when (<= depth max-tree-walk-depth)
-              (cond
-                (and (vector? value)
-                     (= target-op-name (syntax/canonical-op-name (first value))))
-                true
-
-                (coll? value)
-                (some #(walk % (inc depth)) value)
-
-                :else
-                false)))]
-    (boolean (walk operation 0))))
-
-(defn rewrite-source-stage
-  "Rewrite only the operations that belong to the source stage."
-  [operations step]
-  (:operations
-   (reduce (fn [{:keys [operations] :as state} operation]
-             (let [op-name (when (vector? operation)
-                             (syntax/canonical-op-name (first operation)))]
-               (if (= "append-stage" op-name)
-                 {:in-source-stage? false
-                  :operations       (conj operations operation)}
-                 (step state operation))))
-           {:in-source-stage? true
-            :operations       []}
-           operations)))
-
-(defn drop-source-metric-stage-aggregates
-  "Drop redundant source-stage aggregates for metric contexts."
-  [context operations]
-  (if-not (common.context/source-metric-id context)
-    operations
-    (rewrite-source-stage
-     operations
-     (fn [{:keys [in-source-stage? operations] :as state} operation]
-       (let [drop? (and in-source-stage?
-                        (= "aggregate" (syntax/canonical-op-name (first operation)))
-                        (not (stages/aggregate-over-aggregation-ref? operation))
-                        (not (operation-contains-canonical-op? operation "measure"))
-                        (not (operation-contains-canonical-op? operation "metric")))]
-         (if drop?
-           state
-           (assoc state :operations (conj operations operation))))))))
-
-(defn drop-source-metric-stage-with-fields
-  "Drop source-stage `with-fields` that only restate aggregation refs."
-  [context operations]
-  (if-not (common.context/source-metric-id context)
-    operations
-    (rewrite-source-stage
-     operations
-     (fn [{:keys [in-source-stage? operations] :as state} operation]
-       (let [drop? (and in-source-stage?
-                        (= "with-fields" (syntax/canonical-op-name (first operation)))
-                        (stages/contains-aggregation-ref? (rest operation)))]
-         (if drop?
-           state
-           (assoc state :operations (conj operations operation))))))))
-
-(defn relative-datetime-form?
-  "True when `value` is a relative-datetime form with the given amount and unit."
-  [value amount unit]
-  (and (vector? value)
-       (= "relative-datetime" (syntax/canonical-op-name (first value)))
-       (= amount (nth value 1 nil))
-       (= unit (some-> (nth value 2 nil) syntax/raw-op-name u/lower-case-en))))
-
-(defn now-form?
-  "True when `value` is a now literal or `[\"now\"]` tuple."
-  [value]
-  (or (now-literal? value)
-      (and (vector? value)
-           (= "now" (syntax/canonical-op-name (first value))))))
-
-(defn normalize-source-metric-quarter-window
-  "Normalize rolling 3-month source-metric windows into quarter-relative bounds."
-  [context operation]
-  (if (and (common.context/source-metric-id context)
-           (vector? operation)
-           (= "filter" (syntax/canonical-op-name (first operation)))
-           (vector? (second operation))
-           (= "between" (syntax/canonical-op-name (first (second operation)))))
-    (let [[between-op field lower upper] (second operation)]
-      (if (and (relative-datetime-form? lower -3 "month")
-               (now-form? upper))
-        [(first operation)
-         [between-op field
-          ["relative-datetime" -1 "quarter"]
-          ["relative-datetime" 0 "quarter"]]]
-        operation))
-    operation))
-
-(defn- edge-value
-  [edge kebab-key snake-key]
-  (or (get edge kebab-key)
-      (get edge snake-key)))
-
-(defn- join-condition-matches-edge?
-  [source-id target-id [lhs-id rhs-id] edge]
-  (let [from-table-id (edge-value edge :from-table-id :from_table_id)
-        from-field-id (edge-value edge :from-field-id :from_field_id)
-        to-table-id   (edge-value edge :to-table-id :to_table_id)
-        to-field-id   (edge-value edge :to-field-id :to_field_id)]
-    (and (or (and (= source-id from-table-id)
-                  (= target-id to-table-id))
-             (and (= source-id to-table-id)
-                  (= target-id from-table-id)))
-         (or (and (= lhs-id from-field-id) (= rhs-id to-field-id))
-             (and (= lhs-id to-field-id) (= rhs-id from-field-id))))))
-
-(defn- valid-join-operation?
-  "True when `operation` is a two-element join tuple."
-  [operation]
-  (and (vector? operation)
-       (= "join" (syntax/canonical-op-name (first operation)))
-       (= 2 (count operation))))
-
-(defn- implicit-join-matches-context?
-  "True when a parsed join spec matches surrounding context edges."
-  [source-id context {:keys [target-table-id conditions fields-mode strategy alias]}]
-  (let [surrounding-table-ids (set (keep :id (:surrounding-tables context)))]
-    (and (pos-int? target-table-id)
-         (surrounding-table-ids target-table-id)
-         (nil? alias)
-         (or (nil? strategy) (= "left-join" strategy))
-         (join-spec/no-explicit-join-fields? fields-mode)
-         (= 1 (count conditions))
-         (some #(join-condition-matches-edge? source-id target-table-id (first conditions) %)
-               (:join-edges context)))))
-
-(defn redundant-implicit-join?
-  "True when an explicit join is redundant in the current context."
-  [context operation]
-  (let [source-id  (common.context/source-table-id context)
-        join-spec  (when (valid-join-operation? operation)
-                     (join-spec/parse-join-spec (second operation)))]
-    (and source-id
-         join-spec
-         (implicit-join-matches-context? source-id context join-spec))))
-
-(defn- redundant-source-metric-aggregate?
-  "True when an aggregate is an exact reuse of the source metric.
-  This catches `[\"aggregate\" [\"metric\" source-id]]` which is intentionally preserved
-  by `drop-source-metric-stage-aggregates` (that pass only drops non-metric/measure aggregates)."
-  [source-id operation]
-  (and (vector? operation)
-       (= "aggregate" (syntax/canonical-op-name (first operation)))
-       (let [aggregation (second operation)]
-         (and (vector? aggregation)
-              (= "metric" (syntax/canonical-op-name (first aggregation)))
-              (= source-id (some-> (second aggregation) coerce-positive-int))))))
-
-(defn remove-redundant-operations
-  "Remove context-redundant source-metric reuse and implicit joins."
-  [context operations]
-  (remove (fn [operation]
-            (or (when-let [source-id (common.context/source-metric-id context)]
-                  (redundant-source-metric-aggregate? source-id operation))
-                (redundant-implicit-join? context operation)))
-          operations))

src/metabase/agent_lib/repair/normalize.clj

@@ -1,71 +0,0 @@
-(ns metabase.agent-lib.repair.normalize
-  "Deterministic operator and top-level normalization for structured MBQL programs."
-  (:require
-   [metabase.agent-lib.common.coercions :refer [coerce-positive-int]]
-   [metabase.agent-lib.common.literals :refer [scalar-literal?]]
-   [metabase.agent-lib.repair.normalize.forms :as normalize.forms]
-   [metabase.agent-lib.repair.normalize.top-level :as normalize.top-level]
-   [metabase.agent-lib.repair.stages :as stages]
-   [metabase.agent-lib.schema :as schema]
-   [metabase.agent-lib.syntax :as syntax]
-   [metabase.util :as u]))
-
-(set! *warn-on-reflection* true)
-
-(defn repair-program
-  "Apply deterministic repairs for common structured program mistakes.
-
-  The local `letfn` is intentional here: `repair-node` and `repair-map-node`
-  form the minimal mutually recursive kernel for walking arbitrary nested
-  structured-program values."
-  [program]
-  (letfn [(repair-map-node [value]
-            (cond
-              (schema/program-literal? value)
-              (assoc value :program (repair-program (:program value)))
-
-              (and (map? value)
-                   (contains? value :type)
-                   (contains? value :id))
-              (update value :id coerce-positive-int)
-
-              (normalize.forms/field-like-map? value)
-              (repair-node (normalize.forms/repair-field-like-map value))
-
-              :else
-              (into {}
-                    (map (fn [[k v]] [k (repair-node v)]))
-                    value)))
-          (repair-node [value]
-            (cond
-              (scalar-literal? value)   value
-              (map? value)              (repair-map-node value)
-              (syntax/boolean-wrapper-form? value)
-              (let [[raw-op arg] value
-                    arg'         (repair-node arg)]
-                (if (= "true" (some-> raw-op syntax/raw-op-name u/lower-case-en))
-                  arg'
-                  ["not" arg']))
-              (syntax/possible-operator-tuple? value) (normalize.forms/repair-operator-form repair-node value)
-              (vector? value)           (mapv repair-node value)
-              (sequential? value)       (mapv repair-node value)
-              :else                     value))]
-    (if-not (map? program)
-      program
-      (let [source     (let [source (:source program)]
-                         (cond
-                           (not (map? source))
-                           source
-
-                           (= "program" (:type source))
-                           (update source :program repair-program)
-
-                           :else
-                           (cond-> source
-                             (contains? source :id) (update :id coerce-positive-int))))
-            operations (into []
-                             (mapcat #(normalize.top-level/repair-top-level-operation repair-node %))
-                             (:operations program))]
-        (assoc program
-               :source source
-               :operations (stages/insert-stage-boundaries source operations))))))

src/metabase/agent_lib/repair/normalize/forms.clj

@@ -1,337 +0,0 @@
-(ns metabase.agent-lib.repair.normalize.forms
-  "Nested operator normalization for structured program repair."
-  (:require
-   [metabase.agent-lib.common.coercions :as coercions :refer [coerce-non-negative-int
-                                                              coerce-positive-int
-                                                              normalize-join-conditions
-                                                              normalize-percentile-value
-                                                              unwrap-singleton-form]]
-   [metabase.agent-lib.common.literals :refer [iso-date-string?
-                                               non-blank-string?
-                                               now-literal?
-                                               null-literal?
-                                               quarter-label-string?
-                                               scalar-sequential?]]
-   [metabase.agent-lib.syntax :as syntax]))
-
-(set! *warn-on-reflection* true)
-
-(def ^:private temporal-bucket-extraction-aliases
-  "Bucket names that should be repaired into extraction helpers."
-  {"dayofweek"       "get-day-of-week"
-   "day-of-week"     "get-day-of-week"
-   "hour-of-day"     "get-hour"
-   "month-of-year"   "get-month"
-   "quarter-of-year" "get-quarter"})
-
-(defn- option-value
-  "Read the first present key from `options`, accepting either string or keyword keys."
-  [options & ks]
-  (reduce (fn [_ k]
-            (let [v (or (get options (name k)) (get options (keyword k)))]
-              (if (some? v) (reduced v) nil)))
-          nil
-          ks))
-
-(defn- wrap-field-with-options
-  "Rewrite a non-canonical `[\"field\" id {opts}]` tuple into the canonical
-  wrapper forms (`with-temporal-bucket`, `with-binning`) the runtime accepts."
-  [field-form options]
-  (let [temporal-unit (option-value options :temporal-unit)
-        binning       (option-value options :binning)
-        with-temporal (if (some? temporal-unit)
-                        ["with-temporal-bucket" field-form temporal-unit]
-                        field-form)]
-    (if (some? binning)
-      ["with-binning" with-temporal binning]
-      with-temporal)))
-
-(def ^:private field-like-map-id-keys
-  ["field_id" "field-id" :field_id :field-id])
-
-(defn field-like-map?
-  "True when `value` is a plain map that describes a field reference by carrying
-  a `:field_id`/`:field-id` key."
-  [value]
-  (boolean (and (map? value)
-                (some #(some? (get value %)) field-like-map-id-keys))))
-
-(defn repair-field-like-map
-  "Convert a field-like map (e.g. `{\"field_id\" 13 \"temporal_bucket\" \"month\"}`)
-  into the canonical nested operator form the runtime accepts.
-
-  Supports optional `temporal_bucket`/`temporal-unit`, `binning`, and `direction`
-  keys. When `direction` is present, the result is wrapped in `asc`/`desc`."
-  [m]
-  (let [field-id      (option-value m :field_id :field-id)
-        temporal-unit (option-value m :temporal_bucket :temporal-bucket
-                                    :temporal_unit :temporal-unit)
-        binning       (option-value m :binning)
-        direction     (option-value m :direction)
-        field-form    (cond-> ["field" (coerce-positive-int field-id)]
-                        (some? temporal-unit) (wrap-field-with-options
-                                               {"temporal-unit" temporal-unit})
-                        (some? binning)       (wrap-field-with-options
-                                               {"binning" binning}))]
-    (if (coercions/direction-string? direction)
-      [(coercions/normalize-direction direction) field-form]
-      field-form)))
-
-(defn expression-definition-tuple?
-  "True when `value` is an inline expression definition tuple."
-  [value]
-  (and (vector? value)
-       (= "expression" (syntax/canonical-op-name (first value)))
-       (= 3 (count value))
-       (non-blank-string? (second value))))
-
-(defn- quarter-expression?
-  [value]
-  (and (vector? value)
-       (let [op-name (syntax/canonical-op-name (first value))]
-         (or (= op-name "get-quarter")
-             (and (= op-name "with-temporal-bucket")
-                  (= "quarter" (nth value 2 nil)))))))
-
-(defn- normalize-quarter-filter-value
-  [lhs rhs]
-  (cond
-    (and (quarter-expression? lhs)
-         (quarter-label-string? rhs))
-    (coercions/quarter-label->number rhs)
-
-    (and (quarter-expression? lhs)
-         (scalar-sequential? rhs)
-         (every? quarter-label-string? rhs))
-    (mapv coercions/quarter-label->number rhs)
-
-    :else
-    rhs))
-
-(defn- branch-pair?
-  [v]
-  (and (vector? v) (= 2 (count v))))
-
-(defn- else-branch?
-  [v]
-  (and (branch-pair? v)
-       (= "else" (syntax/raw-op-name (first v)))))
-
-(defn- split-else-branch
-  "Peel an else-branch off the end of `branches`, merging it with `fallback`."
-  [branches fallback]
-  (if (and (seq branches) (else-branch? (last branches)))
-    [(vec (butlast branches)) (second (last branches))]
-    [branches fallback]))
-
-(defn- flat->branch-pairs
-  "Convert a flat even-length vector into branch pairs when applicable."
-  [v]
-  (when (and (vector? v)
-             (even? (count v))
-             (not-every? branch-pair? v))
-    (mapv vec (partition 2 v))))
-
-(defn- normalize-branches-fallback
-  "Uniform post-processing: split else-branch and emit `[branches]` or
-  `[branches fallback]`."
-  [branches fallback]
-  (let [[branches fallback] (split-else-branch branches fallback)]
-    (if (some? fallback)
-      [branches fallback]
-      [branches])))
-
-(defn- classify-case-args
-  "Classify the shape of `case` args into `[branches fallback]` before uniform
-  normalization, or nil if no pattern matches."
-  [args]
-  (cond
-    ;; Three bare args: pred, then, else
-    (and (= 3 (count args))
-         (not (branch-pair? (first args))))
-    [[[(first args) (second args)]] (nth args 2)]
-
-    ;; Single wrapped vector of branch pairs
-    (and (seq args)
-         (vector? (first args))
-         (every? branch-pair? (first args)))
-    [(first args) (second args)]
-
-    ;; Single wrapped triple: [pred then else]
-    (and (= 1 (count args))
-         (vector? (first args))
-         (= 3 (count (first args)))
-         (not-every? branch-pair? (first args)))
-    (let [[pred then else] (first args)]
-      [[[pred then]] else])
-
-    ;; Leading branch pairs followed by optional fallback
-    (seq (take-while branch-pair? args))
-    (let [branches  (vec (take-while branch-pair? args))
-          remainder (drop (count branches) args)]
-      [branches (when (= 1 (count remainder)) (first remainder))])
-
-    ;; Flat even-length vector inside a wrapper
-    (flat->branch-pairs (first args))
-    [(flat->branch-pairs (first args)) (second args)]))
-
-(defn repair-case-args
-  "Normalize alternate `case` encodings into the canonical branches/fallback shape."
-  [args]
-  (if-let [[branches fallback] (classify-case-args args)]
-    (normalize-branches-fallback branches fallback)
-    args))
-
-(defn temporal-expression?
-  "True when `value` is a temporal literal or temporal expression tuple."
-  [value]
-  (or (now-literal? value)
-      (and (vector? value)
-           (contains? #{"absolute-datetime" "relative-datetime" "now"}
-                      (syntax/canonical-op-name (first value))))))
-
-(defn- wrap-iso-date-as-absolute-datetime
-  [value]
-  (if (iso-date-string? value)
-    ["absolute-datetime" value "day"]
-    value))
-
-(defn- wrap-now-as-expression
-  [value]
-  (if (now-literal? value)
-    ["now"]
-    value))
-
-(defn repair-between-bounds
-  "Normalize temporal `between` bounds into canonical datetime helper forms.
-  Preserves degenerate (fewer than 3) arg counts so repair remains idempotent
-  when the operator is embedded in a helper that reshapes its arguments
-  (e.g. `case`)."
-  [args]
-  (if (< (count args) 3)
-    (vec args)
-    (let [[lhs lower upper] args
-          lower'            (wrap-now-as-expression lower)
-          upper'            (wrap-now-as-expression upper)]
-      (if (or (temporal-expression? lower')
-              (temporal-expression? upper'))
-        [lhs
-         (if (temporal-expression? upper')
-           (wrap-iso-date-as-absolute-datetime lower')
-           lower')
-         (if (temporal-expression? lower')
-           (wrap-iso-date-as-absolute-datetime upper')
-           upper')]
-        [lhs lower' upper']))))
-
-(defn repair-operator-form
-  "Normalize a nested operator form using the recursive `repair-node` callback."
-  [repair-node value]
-  (let [[raw-op & raw-args] value
-        op-name             (syntax/canonical-op-name raw-op)
-        args                (mapv (fn [arg]
-                                    (let [arg' (repair-node arg)]
-                                      (if (#{"in"
-                                             "not-in"
-                                             "case"
-                                             "with-fields"
-                                             "with-join-fields"
-                                             "with-join-conditions"} op-name)
-                                        arg'
-                                        (unwrap-singleton-form arg'))))
-                                  raw-args)]
-    (case op-name
-      "field"
-      (let [id        (coerce-positive-int (first args))
-            maybe-opt (second args)]
-        (if (map? maybe-opt)
-          (wrap-field-with-options [op-name id] maybe-opt)
-          [op-name id]))
-
-      ("table" "card" "metric" "measure")
-      [op-name (coerce-positive-int (first args))]
-
-      "aggregation-ref"
-      [op-name (coerce-non-negative-int (first args))]
-
-      "with-temporal-bucket"
-      (if (< (count args) 2)
-        (into [op-name] args)
-        (let [[field bucket] args]
-          (if-let [extraction-op (and (string? bucket)
-                                      (temporal-bucket-extraction-aliases bucket))]
-            [extraction-op field]
-            [op-name field bucket])))
-
-      ("contains" "does-not-contain" "starts-with" "ends-with")
-      (into [op-name]
-            (cond-> args
-              (and (> (count args) 2)
-                   (or (map? (last args))
-                       (boolean? (last args))))
-              pop))
-
-      ("in" "not-in")
-      (if (< (count args) 2)
-        (into [op-name] args)
-        (let [[lhs rhs] args]
-          [op-name lhs (normalize-quarter-filter-value lhs rhs)]))
-
-      "="
-      (if (< (count args) 2)
-        (into [op-name] args)
-        (let [[lhs rhs] args
-              rhs'      (normalize-quarter-filter-value lhs rhs)]
-          (if (scalar-sequential? rhs')
-            ["in" lhs (vec rhs')]
-            [op-name lhs rhs'])))
-
-      "!="
-      (if (< (count args) 2)
-        (into [op-name] args)
-        (let [[lhs rhs] args
-              rhs'      (normalize-quarter-filter-value lhs rhs)]
-          (if (scalar-sequential? rhs')
-            ["not-in" lhs (vec rhs')]
-            [op-name lhs rhs'])))
-
-      "is"
-      (if (< (count args) 2)
-        (into [op-name] args)
-        (let [[lhs rhs] args]
-          (if (null-literal? rhs)
-            ["is-null" lhs]
-            ["=" lhs rhs])))
-
-      "is-not"
-      (if (< (count args) 2)
-        (into [op-name] args)
-        (let [[lhs rhs] args]
-          (if (null-literal? rhs)
-            ["not-null" lhs]
-            ["!=" lhs rhs])))
-
-      "case"
-      (let [args' (if (= "if" (syntax/raw-op-name raw-op))
-                    (if (= 3 (count args))
-                      [[[(first args) (second args)]] (nth args 2)]
-                      args)
-                    args)]
-        (into [op-name] (repair-case-args args')))
-
-      "between"
-      (into [op-name] (repair-between-bounds args))
-
-      "percentile"
-      (if (< (count args) 2)
-        (into [op-name] args)
-        (let [[expr percentile] args]
-          [op-name expr (normalize-percentile-value percentile)]))
-
-      "with-join-conditions"
-      (if (< (count args) 2)
-        (into [op-name] args)
-        (let [[join-spec conditions] args]
-          [op-name join-spec (normalize-join-conditions conditions)]))
-
-      (into [op-name] args))))

src/metabase/agent_lib/repair/normalize/top_level.clj

@@ -1,174 +0,0 @@
-(ns metabase.agent-lib.repair.normalize.top-level
-  "Top-level operation normalization for structured program repair."
-  (:require
-   [metabase.agent-lib.common.coercions :refer [direction-string?
-                                                normalize-direction]]
-   [metabase.agent-lib.common.literals :refer [aggregation-form?]]
-   [metabase.agent-lib.repair.normalize.forms :as forms]
-   [metabase.agent-lib.syntax :as syntax]))
-
-(set! *warn-on-reflection* true)
-
-(defn normalize-with-fields-selection
-  "Normalize `with-fields` arguments into a single vector of selections."
-  [args]
-  (cond
-    (empty? args) []
-
-    (= 1 (count args))
-    (let [selection (first args)]
-      (if (and (sequential? selection)
-               (not (syntax/possible-operator-tuple? selection)))
-        (vec selection)
-        [selection]))
-
-    :else
-    (vec args)))
-
-(defn- direction-tuple?
-  [value]
-  (and (vector? value)
-       (= 2 (count value))
-       (direction-string? (first value))))
-
-(defn- unwrap-order-by-args
-  "Unwrap a single sequential argument into its elements so `[[asc ...] [desc ...]]`
-  behaves like `[asc ...] [desc ...]` passed directly. Only unwraps when the inner
-  sequence is entirely `[direction value]` tuples so we don't rewrite unrelated
-  shapes and break idempotence."
-  [args]
-  (if (and (= 1 (count args))
-           (sequential? (first args))
-           (seq (first args))
-           (every? direction-tuple? (first args)))
-    (vec (first args))
-    args))
-
-(defn repair-order-by-operations
-  "Normalize one or more `order-by` arguments into canonical operations."
-  [args]
-  (let [args (unwrap-order-by-args args)]
-    (if (empty? args)
-      [["order-by"]]
-      (loop [remaining args
-             repaired  []]
-        (if (empty? remaining)
-          repaired
-          (let [[first-arg second-arg & tail] remaining]
-            (cond
-              (and (vector? first-arg)
-                   (= 2 (count first-arg))
-                   (direction-string? (first first-arg)))
-              (recur (vec (rest remaining))
-                     (conj repaired ["order-by" [(normalize-direction (first first-arg))
-                                                 (second first-arg)]]))
-
-              (direction-string? second-arg)
-              (recur (vec tail)
-                     (conj repaired ["order-by" first-arg (normalize-direction second-arg)]))
-
-              :else
-              (recur (vec (rest remaining))
-                     (conj repaired ["order-by" first-arg])))))))))
-
-(defn repair-repeated-single-arg-op
-  "Expand repeated top-level single-arg operations."
-  [op-name args]
-  (if (empty? args)
-    [[op-name]]
-    (mapv (fn [arg] [op-name arg]) args)))
-
-(defn repair-with-fields-operation
-  "Extract inline expressions out of `with-fields` before projection."
-  [repair-node args]
-  (let [selection       (normalize-with-fields-selection args)
-        repaired-fields (mapv repair-node selection)
-        extracted-exprs (keep (fn [field]
-                                (when (forms/expression-definition-tuple? field)
-                                  ["expression" (second field) (nth field 2)]))
-                              repaired-fields)
-        projected       (mapv (fn [field]
-                                (if (forms/expression-definition-tuple? field)
-                                  ["expression-ref" (second field)]
-                                  field))
-                              repaired-fields)]
-    (into (vec extracted-exprs)
-          [["with-fields" projected]])))
-
-(defn bare-aggregation-ref-operation?
-  "True when an operation is a bare top-level aggregation-ref."
-  [operation]
-  (and (vector? operation)
-       (= "aggregation-ref" (syntax/canonical-op-name (first operation)))))
-
-(defn- repair-with-embedded
-  "Common skeleton: combine repaired direct args with embedded ops, falling back
-  to a bare `[[op-name]]` when both are empty."
-  [op-name direct-ops embedded-ops]
-  (into (or (when (seq direct-ops) direct-ops)
-            (when (empty? embedded-ops) [[op-name]])
-            [])
-        embedded-ops))
-
-(defn- repair-direct-args
-  "Dispatch direct-arg repair by operation name."
-  [op-name direct-args]
-  (case op-name
-    "order-by"
-    (repair-order-by-operations direct-args)
-
-    ("breakout" "filter" "join")
-    (repair-repeated-single-arg-op op-name direct-args)
-
-    ("limit" "with-page")
-    [[op-name (first direct-args)]]))
-
-(defn repair-top-level-operation
-  "Normalize a top-level operation using the recursive `repair-node` callback."
-  [repair-node operation]
-  (if-not (vector? operation)
-    [operation]
-    (let [[raw-op & raw-args] operation
-          op-name             (syntax/canonical-op-name raw-op)
-          args                (mapv repair-node raw-args)
-          embedded-ops        (mapcat #(repair-top-level-operation repair-node %)
-                                      (filter syntax/top-level-operation-tuple? args))
-          direct-args         (vec (remove syntax/top-level-operation-tuple? args))]
-      (case op-name
-        "visualization"
-        []
-
-        "aggregate"
-        (let [keep-bare-aggregation-refs? (not-any? (complement bare-aggregation-ref-operation?)
-                                                    direct-args)
-              direct-args'                (cond->> direct-args
-                                            (not keep-bare-aggregation-refs?)
-                                            (remove bare-aggregation-ref-operation?))]
-          (repair-with-embedded op-name
-                                (when (seq direct-args')
-                                  (repair-repeated-single-arg-op op-name direct-args'))
-                                embedded-ops))
-
-        "with-fields"
-        (repair-with-fields-operation repair-node args)
-
-        ("order-by" "breakout" "filter" "join" "limit" "with-page")
-        (repair-with-embedded op-name
-                              (when (seq direct-args)
-                                (repair-direct-args op-name direct-args))
-                              embedded-ops)
-
-        "expression"
-        (repair-with-embedded op-name
-                              (when (>= (count direct-args) 2)
-                                (let [[expr-name expr] (take 2 direct-args)]
-                                  (if (aggregation-form? expr)
-                                    [["aggregate" expr]]
-                                    [[op-name expr-name expr]])))
-                              embedded-ops)
-
-        ("append-stage" "drop-stage" "drop-empty-stages")
-        [[op-name]]
-
-        (into [(into [op-name] direct-args)]
-              embedded-ops)))))

src/metabase/agent_lib/repair/stages.clj

@@ -1,107 +0,0 @@
-(ns metabase.agent-lib.repair.stages
-  "Stage-boundary inference and aggregation-stage helpers for structured repair."
-  (:require
-   [metabase.agent-lib.syntax :as syntax]))
-
-(set! *warn-on-reflection* true)
-
-(def ^:private max-repair-depth
-  "Maximum recursion depth for repair-phase tree walks."
-  64)
-
-(defn contains-aggregation-ref?
-  "True when `value` contains an `aggregation-ref` anywhere inside it."
-  ([value]
-   (contains-aggregation-ref? value 0))
-  ([value depth]
-   (if (> depth max-repair-depth)
-     false
-     (cond
-       (and (vector? value)
-            (= "aggregation-ref" (syntax/canonical-op-name (first value))))
-       true
-
-       (map? value)
-       (boolean (some #(contains-aggregation-ref? % (inc depth)) (vals value)))
-
-       (sequential? value)
-       (boolean (some #(contains-aggregation-ref? % (inc depth)) value))
-
-       :else
-       false))))
-
-(defn aggregate-over-aggregation-ref?
-  "True when `operation` is an aggregate over an aggregation-ref."
-  [operation]
-  (and (vector? operation)
-       (= "aggregate" (syntax/canonical-op-name (first operation)))
-       (contains-aggregation-ref? (rest operation))))
-
-(defn- operation-defines-aggregation?
-  [operation]
-  (and (vector? operation)
-       (= "aggregate" (syntax/canonical-op-name (first operation)))))
-
-(defn- operation-requires-post-aggregation-stage?
-  [operation]
-  (and (vector? operation)
-       (contains-aggregation-ref? (rest operation))
-       (#{"aggregate" "expression" "with-fields"}
-        (syntax/canonical-op-name (first operation)))))
-
-(defn source-final-stage-has-aggregations?
-  "True when the source program ends in an aggregated stage."
-  [source]
-  (cond
-    (not (map? source))
-    false
-
-    (= "metric" (:type source))
-    true
-
-    (= "program" (:type source))
-    (let [nested-program (:program source)]
-      (:current-stage-has-aggregations?
-       (reduce (fn [state operation]
-                 (cond
-                   (and (vector? operation)
-                        (= "append-stage" (syntax/canonical-op-name (first operation))))
-                   (assoc state :current-stage-has-aggregations? false)
-
-                   (operation-defines-aggregation? operation)
-                   (assoc state :current-stage-has-aggregations? true)
-
-                   :else
-                   state))
-               {:current-stage-has-aggregations? (source-final-stage-has-aggregations? (:source nested-program))}
-               (:operations nested-program))))
-
-    :else
-    false))
-
-(defn insert-stage-boundaries
-  "Insert `append-stage` before operations that must run after an aggregation stage."
-  [source operations]
-  (:operations
-   (reduce (fn [{:keys [current-stage-has-aggregations? operations]} operation]
-             (let [op-name (when (vector? operation)
-                             (syntax/canonical-op-name (first operation)))
-                   needs-stage? (and current-stage-has-aggregations?
-                                     (operation-requires-post-aggregation-stage? operation)
-                                     (not= "append-stage"
-                                           (some-> operations last first syntax/canonical-op-name)))
-                   operations'  (cond-> operations
-                                  needs-stage? (conj ["append-stage"])
-                                  true        (conj operation))]
-               (cond
-                 (= "append-stage" op-name)
-                 {:current-stage-has-aggregations? false
-                  :operations                      operations'}
-
-                 :else
-                 {:current-stage-has-aggregations? (or (and (not needs-stage?) current-stage-has-aggregations?)
-                                                       (operation-defines-aggregation? operation))
-                  :operations                      operations'})))
-           {:current-stage-has-aggregations? (source-final-stage-has-aggregations? source)
-            :operations                      []}
-           operations)))

src/metabase/agent_lib/representations.clj

@@ -0,0 +1,246 @@
+(ns metabase.agent-lib.representations
+  "Boundary helpers for the canonical MBQL 5 portable representations form.
+
+  Two layers live here:
+
+    * **External form** — the wire / LLM-facing shape: a keyword-keyed Clojure map matching
+      [[metabase.lib.schema/external-query]]. Identical to internal `::lib.schema/query` except
+      the ID types are portable: `::id/database`, `::id/card`, `::id/segment`, `::id/measure`,
+      `::id/snippet` are strings; `::id/table` is `[db schema table]` and `::id/field` is
+      `[db schema table field …]`.
+    * **Portable form** — the same shape with all keys and enum values stringified. This is the
+      lingua franca of the repair pipeline ([[metabase.agent-lib.representations.repair]]) and the
+      output of [[metabase.agent-lib.representations.resolve/export-query]]. Repair operates on
+      string-keyed data because every clause head, options key, and `lib/`-prefixed marker is
+      authored as a string by the LLM and only keywordized after FK resolution.
+
+  This namespace owns the conversion between those two layers and the up-front structural check
+  that guards the entry to the repair pipeline.
+
+  Phase 1 MVP scope (see `repr-plan.md`):
+
+    * `source-table` and `source-card` (the latter as a portable entity_id string);
+    * single-stage queries (no multi-stage);
+    * top-level stage operations: `filters`, `aggregation`, `breakout`, `order-by`, `limit`,
+      `fields`;
+    * explicit joins supported; implicit joins are filled in by the repair pass."
+  (:require
+   [malli.core :as mc]
+   [malli.transform :as mtx]
+   [metabase.lib.schema :as lib.schema]
+   [metabase.util.i18n :refer [tru]]
+   [metabase.util.malli.humanize :as mu.humanize]
+   [metabase.util.malli.registry :as mr]))
+
+(set! *warn-on-reflection* true)
+
+;;; ============================================================
+;;; External (keyword-keyed) ↔ portable (string-keyed) conversion
+;;; ============================================================
+
+(defn- keyword->repr-string
+  "Stringify a keyword preserving its namespace, e.g. `:lib/type` → `\"lib/type\"`.
+  `clojure.core/name` would drop the namespace; `clojure.walk/stringify-keys` likewise."
+  [k]
+  (if-let [ns (namespace k)]
+    (str ns "/" (name k))
+    (name k)))
+
+(defn external-query->portable
+  "Convert a keyword-keyed external-query map (as produced by the JSON middleware's keywordizing
+  decoder, or by [[validate-external-query]]'s string-transformer decode) into the string-keyed
+  portable form expected by the repair pipeline.
+
+  Walks the structure once. Map keys are stringified preserving their namespace. Keyword values
+  (notably the `:lib/type` enum stamps `:mbql/query` / `:mbql.stage/mbql`) are also stringified
+  so the post-decode shape matches the all-strings invariant the repair pipeline relies on. FK
+  path segments and option values that already arrive as strings pass through unchanged."
+  [x]
+  (cond
+    (map? x)
+    (reduce-kv
+     (fn [m k v]
+       (assoc m
+              (cond-> k (keyword? k) keyword->repr-string)
+              (external-query->portable v)))
+     {}
+     x)
+
+    (vector? x)     (mapv external-query->portable x)
+    (sequential? x) (mapv external-query->portable x)
+    (keyword? x)    (keyword->repr-string x)
+    :else           x))
+
+;;; ============================================================
+;;; Up-front validation against ::lib.schema/external-query
+;;; ============================================================
+
+(defn validate-external-query
+  "Decode and validate `external-query` (keyword-keyed, JSON-decoded) against
+  [[metabase.lib.schema/external-query]], returning the decoded query.
+
+  The string-transformer keywordizes enum values (`\"mbql/query\"` → `:mbql/query`,
+  `\"mbql.stage/mbql\"` → `:mbql.stage/mbql`, etc.) so the schema's keyword-typed enum slots
+  accept the JSON shape directly. Throws with `:error :invalid-external-query` and a humanized
+  explanation when the input shape is structurally invalid even after decoding.
+
+  NOTE: kept as a building block; the agent boundary uses [[assert-known-stage-keys!]] instead
+  because the lib schema's stage shape is not a closed map (typo'd stage keys would silently
+  pass) and the humanized output here references function predicates in a form that isn't
+  LLM-actionable."
+  [external-query]
+  (let [schema  (mr/schema ::lib.schema/external-query)
+        decoded (mc/decode schema external-query mtx/string-transformer)]
+    (when-let [error (mr/explain ::lib.schema/external-query decoded)]
+      (throw (ex-info (tru "External query has an invalid structure.")
+                      {:status-code 400
+                       :error       :invalid-external-query
+                       :humanized   (mu.humanize/humanize error)
+                       :details     (pr-str (mu.humanize/humanize error))
+                       :schema      ::lib.schema/external-query})))
+    decoded))
+
+(def ^:private known-stage-keys
+  "The complete set of top-level keys a `mbql.stage/mbql` stage may carry on the
+  LLM-facing surface. Sourced from `metabase.lib.schema.mbql-stage` and pruned to the
+  keys the `construct_query` / `query` contract advertises (see
+  `resources/metabot/prompts/tools/construct_notebook_query.md`)."
+  #{"lib/type"
+    "source-table" "source-card"
+    "filters" "aggregation" "breakout" "expressions" "fields" "joins"
+    "order-by" "limit"})
+
+(defn assert-known-stage-keys!
+  "Boundary check: every stage in the portable query must carry only known top-level keys.
+  `lib.schema.mbql-stage/mbql` is not a closed map, so a typo'd key (e.g. `aggreagation` vs
+  `aggregation`) would otherwise be silently dropped at resolve / lib.normalize time and the
+  LLM would never learn that its intent was discarded. This check catches the common typo
+  class with a clean `:agent-error?` diagnostic that names the offending key.
+
+  Operates on the string-keyed portable form produced by [[external-query->portable]]."
+  [portable-query]
+  (when (and (map? portable-query) (vector? (get portable-query "stages")))
+    (doseq [[stage-idx stage] (map-indexed vector (get portable-query "stages"))
+            :when (map? stage)
+            :let [unknown (remove known-stage-keys (keys stage))]
+            :when (seq unknown)]
+      (throw (ex-info (tru "Stage {0} has unknown key(s): {1}. Valid stage keys are: {2}."
+                           stage-idx
+                           (pr-str (vec unknown))
+                           (pr-str (vec (sort known-stage-keys))))
+                      {:status-code  400
+                       :error        :unknown-stage-key
+                       :agent-error? true
+                       :stage-index  stage-idx
+                       :unknown-keys (vec unknown)})))))
+
+;;; ============================================================
+;;; Repair-pipeline schema (string-keyed portable form)
+;;;
+;;; The schema below is intentionally narrower than `::lib.schema/external-query`: it asserts the
+;;; portable-form invariants the repair pass relies on (FK shape, always-present `{}` options
+;;; map, allowed top-level keys) and ignores per-operator term shapes — those are the lib-schema's
+;;; responsibility after FK resolution. This is what `validate-query` checks between repair and
+;;; resolve.
+;;; ============================================================
+
+(defn- non-blank-string?
+  [x]
+  (and (string? x) (not= "" x)))
+
+(defn- options-map?
+  [x]
+  (map? x))
+
+(defn- table-fk?
+  "Portable table FK: [db-name, schema-or-null, table-name]."
+  [x]
+  (and (vector? x)
+       (= 3 (count x))
+       (non-blank-string? (nth x 0))
+       (or (nil? (nth x 1)) (string? (nth x 1)))
+       (non-blank-string? (nth x 2))))
+
+(defn- field-fk?
+  "Portable field FK: [db-name, schema-or-null, table-name, field-name, ...json-path-segments]."
+  [x]
+  (and (vector? x)
+       (>= (count x) 4)
+       (non-blank-string? (nth x 0))
+       (or (nil? (nth x 1)) (string? (nth x 1)))
+       (non-blank-string? (nth x 2))
+       (every? non-blank-string? (drop 3 x))))
+
+(defn- clause-shape?
+  [x]
+  (and (vector? x)
+       (>= (count x) 2)
+       (string? (nth x 0))
+       (non-blank-string? (nth x 0))
+       (options-map? (nth x 1))))
+
+(def ^:private registry
+  {::table-fk [:and vector?
+               [:fn {:error/message "table FK must be [db-name, schema-or-null, table-name]"}
+                table-fk?]]
+   ::field-fk [:and vector?
+               [:fn {:error/message "field FK must be [db-name, schema-or-null, table-name, field-name, …]"}
+                field-fk?]]
+   ::options  [:and map?
+               [:fn {:error/message "options must always be a map (use `{}` if empty)"}
+                options-map?]]
+   ::clause   [:and vector?
+               [:fn {:error/message "clause must be [operator-string, options-map, …args]"}
+                clause-shape?]]
+   ::join     [:map
+               {:closed false}
+               ["stages"     [:sequential {:min 1} [:ref ::stage]]]
+               ["conditions" [:sequential {:min 1} [:ref ::clause]]]
+               ["alias"      :string]
+               ["strategy"   {:optional true}
+                [:enum "left-join" "right-join" "inner-join" "full-join"]]
+               ["fields"     {:optional true}
+                [:or
+                 [:enum "all" "none"]
+                 [:sequential [:ref ::clause]]]]]
+   ::stage    [:map
+               {:closed false}
+               ["lib/type"     [:= "mbql.stage/mbql"]]
+               ["source-table" {:optional true} [:ref ::table-fk]]
+               ["source-card"  {:optional true} :string]
+               ["joins"        {:optional true} [:sequential [:ref ::join]]]
+               ["filters"      {:optional true} [:sequential [:ref ::clause]]]
+               ["aggregation"  {:optional true} [:sequential [:ref ::clause]]]
+               ["breakout"     {:optional true} [:sequential [:ref ::clause]]]
+               ["order-by"     {:optional true} [:sequential [:ref ::clause]]]
+               ["limit"        {:optional true} [:maybe :int]]
+               ["fields"       {:optional true} [:sequential [:ref ::clause]]]
+               ["expressions"  {:optional true} [:sequential [:ref ::clause]]]]
+   ::query    [:map
+               {:closed false}
+               ["lib/type" [:= "mbql/query"]]
+               ["database" {:optional true} :string]
+               ["stages"   [:sequential {:min 1} [:ref ::stage]]]]})
+
+(def query-schema
+  "Malli schema for a Phase-1 portable representations query (string-keyed)."
+  [:schema {:registry registry} [:ref ::query]])
+
+(def stage-schema
+  "Malli schema for a single MBQL stage in the portable representations form."
+  [:schema {:registry registry} [:ref ::stage]])
+
+(defn validate-query
+  "Validate a string-keyed portable representations query against [[query-schema]] and return it
+  unchanged.
+
+  Used as a sanity check between the repair and resolve passes."
+  [parsed]
+  (when-let [error (mr/explain query-schema parsed)]
+    (throw (ex-info (tru "Representations query has an invalid structure.")
+                    {:status-code 400
+                     :error       :invalid-representations-query
+                     :humanized   (mu.humanize/humanize error)
+                     :details     (pr-str (mu.humanize/humanize error))
+                     :schema      ::query})))
+  parsed)

src/metabase/agent_lib/representations/resolve.clj

@@ -0,0 +1,161 @@
+(ns metabase.agent-lib.representations.resolve
+  "Resolve a parsed (string-keyed, portable) representations query into canonical numeric-ID
+  pMBQL.
+
+  Pipeline:
+
+    1. **keywordize** string keys on the representations query. The repair pass operates on
+       string-keyed data (LLM-authored markers like `\"lib/type\"`, `\"source-table\"`,
+       `\"temporal-unit\"` flow through as strings); lib-level MBQL expects keywords. We walk the
+       structure once to keywordize map keys but leave string *values* — clause heads, option
+       values like `\"month\"`, etc. — for [[lib.normalize]] to convert.
+
+    2. **resolve portable FKs → numeric IDs** via `metabase.models.serialization.resolve/import-mbql`,
+       bound to a metadata-provider-backed resolver. Clause heads like `\"field\"` are converted
+       to `:field` keywords by `import-mbql` itself (per the `#{:field \"field\"}` match pattern),
+       so by the end of this step the MBQL form has numeric ids and keywordized heads.
+
+    3. **normalize** through `lib.normalize/normalize` against `:metabase.lib.schema/query`. This:
+       * adds `:lib/uuid` to every clause;
+       * keywordizes known enum values (temporal units, base-types, join strategies);
+       * kebab-cases keys where applicable;
+       * attaches the metadata-provider at `:lib/metadata` so the result is a \"real\" pMBQL that
+         can be handed to `lib.query` / the QP directly.
+
+  The output is a valid MBQL 5 query ready for the query processor.
+
+  The inverse direction — final pMBQL back to portable form — is handled by [[export-query]];
+  the result is a Clojure map matching the external (keyword-keyed) shape, ready for JSON
+  encoding or for handing back to the LLM as the canonical MBQL 5 representation."
+  (:require
+   [metabase.lib.normalize :as lib.normalize]
+   [metabase.lib.schema :as lib.schema]
+   [metabase.models.serialization.resolve :as resolve]
+   [metabase.models.serialization.resolve.mp :as resolve.mp]
+   [metabase.util.log :as log]))
+
+(set! *warn-on-reflection* true)
+
+;;; ============================================================
+;;; Step 1 - keywordize map keys
+;;; ============================================================
+
+(defn- keywordize-keys*
+  "Recursively convert string map keys to keywords. Preserves string *values* (e.g. clause heads,
+  temporal-unit strings, enum-valued option values) - those are handled by `import-mbql` /
+  `lib.normalize` downstream.
+
+  Special-cases: leaves table-FK and field-FK path vectors alone (vector-of-strings doesn't get\n  recursed into beyond the normal `vector? → mapv` walk, which is a no-op on strings)."
+  [x]
+  (cond
+    (map? x)        (persistent!
+                     (reduce-kv (fn [acc k v]
+                                  (assoc! acc
+                                          (cond-> k (string? k) keyword)
+                                          (keywordize-keys* v)))
+                                (transient {})
+                                x))
+    (vector? x)     (mapv keywordize-keys* x)
+    (sequential? x) (mapv keywordize-keys* x)
+    :else           x))
+
+(defn keywordize-query
+  "Turn the portable string-keyed representations form into a keyword-keyed map suitable for
+  `import-mbql` / `lib.normalize`."
+  [parsed-repr]
+  (keywordize-keys* parsed-repr))
+
+;;; ============================================================
+;;; Step 2+3 - resolve FKs and normalize
+;;; ============================================================
+
+(defn resolve-query
+  "Convert a parsed (string-keyed, portable) representations query into a canonical, numeric-ID,
+  `:lib/uuid`-stamped MBQL 5 query attached to `metadata-provider`.
+
+  The optional `content-store` is used for Metabase-content lookups (source cards, metrics,
+  etc.). Agent-facing callers should pass a permission-aware store; the 2-arity form keeps the
+  default app-DB-backed resolver for non-HTTP/test callers.
+
+  Throws with informative ex-info on missing / ambiguous FK lookups (via the resolver) or on
+  lib.schema normalization failures."
+  ([metadata-provider parsed-repr]
+   (resolve-query metadata-provider parsed-repr resolve.mp/unchecked-app-db-content-store))
+  ([metadata-provider parsed-repr content-store]
+   (let [kw-form  (keywordize-query parsed-repr)
+         resolver (resolve.mp/import-resolver metadata-provider content-store)
+         resolved (resolve/import-mbql resolver kw-form)
+         with-mp  (assoc resolved :lib/metadata metadata-provider)]
+     (lib.normalize/normalize ::lib.schema/query with-mp))))
+
+;;; ============================================================
+;;; Export final pMBQL back to portable representations
+;;; ============================================================
+
+(defn- keyword->repr-string
+  "Stringify a keyword preserving its namespace, e.g. `:lib/type` → `\"lib/type\"`."
+  [k]
+  (if-let [ns (namespace k)]
+    (str ns "/" (name k))
+    (name k)))
+
+(defn- portable-repr-form
+  "Convert the keyworded portable form returned by serdes export into the LLM-facing
+  representations form: string keys, string clause heads / enum values, no internal metadata
+  provider handle."
+  [x]
+  (cond
+    (map? x)
+    (reduce-kv
+     (fn [m k v]
+       (if (= k :lib/metadata)
+         m
+         (assoc m
+                (cond-> k (keyword? k) keyword->repr-string)
+                (portable-repr-form v))))
+     (empty x)
+     x)
+
+    (vector? x)     (mapv portable-repr-form x)
+    (sequential? x) (mapv portable-repr-form x)
+    (keyword? x)    (keyword->repr-string x)
+    :else           x))
+
+(defn export-query
+  "Convert a final normalized numeric-ID pMBQL query back to portable representations data.
+
+  This is the inverse of [[resolve-query]] for the agent/tool output path: table/field/card IDs
+  are exported to portable FK paths / entity_ids, lib's normalized keyworded form is converted
+  back to the string-keyed portable representation, and internal `:lib/metadata` is dropped.
+
+  The optional `content-store` is used for Metabase-content lookups (Card / Measure / Segment
+  by id) on the export side. Agent-facing callers should pass a permission-aware store —
+  typically `metabase.metabot.tools.shared.content-store/default-store` — so that
+  entity_ids of referenced cards / measures / segments do not leak through the export to a
+  user who can't read them. The 2-arity form keeps the default app-DB-backed unchecked
+  resolver for non-HTTP / test callers."
+  ([metadata-provider pmbql-query]
+   (export-query metadata-provider pmbql-query resolve.mp/unchecked-app-db-content-store))
+  ([metadata-provider pmbql-query content-store]
+   (let [resolver (resolve.mp/export-resolver metadata-provider content-store)]
+     (->> pmbql-query
+          (resolve/export-mbql resolver)
+          portable-repr-form))))
+
+(defn try-export-query
+  "Best-effort wrapper around [[export-query]] that returns `nil` instead of throwing when the
+  export pipeline fails or `pmbql-query` is nil/blank. Used in LLM context-building paths where
+  an unusual or partially-broken existing `dataset_query` should gracefully drop out of the
+  payload rather than break the whole tool response.
+
+  Like [[export-query]], the 3-arity form accepts an explicit `content-store`; agent callers
+  pass `metabase.metabot.tools.shared.content-store/default-store` for read-checking."
+  ([metadata-provider pmbql-query]
+   (try-export-query metadata-provider pmbql-query resolve.mp/unchecked-app-db-content-store))
+  ([metadata-provider pmbql-query content-store]
+   (when (and metadata-provider (map? pmbql-query) (seq pmbql-query))
+     (try
+       (export-query metadata-provider pmbql-query content-store)
+       (catch Exception e
+         (log/warn e "Failed to export pMBQL query to portable representations; omitting from LLM payload")
+         nil)))))

src/metabase/agent_lib/runtime.clj

@@ -1,104 +0,0 @@
-(ns metabase.agent-lib.runtime
-  "Trusted MBQL helper runtime for structured MBQL program evaluation."
-  (:require
-   [metabase.agent-lib.capabilities :as capabilities]
-   [metabase.agent-lib.mbql-integration :as mbql]
-   [metabase.agent-lib.runtime.bindings :as runtime.bindings]
-   [metabase.agent-lib.runtime.fields :as runtime.fields]
-   [metabase.agent-lib.syntax :as syntax]
-   [metabase.lib.core :as lib]
-   [metabase.lib.metadata :as lib.metadata]))
-
-;; Ensure all defmethod registrations are loaded.
-(comment lib/keep-me)
-
-(set! *warn-on-reflection* true)
-
-(def ^:private extra-helper-bindings
-  {;; Query-introspection helpers exposed through the shared runtime.
-   'suggested-join-conditions lib/suggested-join-conditions
-   'joinable-columns          lib/join-fieldable-columns
-   'joins                     lib/joins
-   'visible-columns           lib/visible-columns
-   'filterable-columns        lib/filterable-columns
-   'breakoutable-columns      lib/breakoutable-columns
-   'aggregable-columns        lib/aggregable-columns
-   'orderable-columns         lib/orderable-columns
-   'expressionable-columns    lib/expressionable-columns
-
-   ;; Query-dependent refs are wired at runtime build time.
-   'aggregation-ref           nil})
-
-(def trusted-helper-bindings
-  "Trusted structured helper implementations keyed by operator symbol."
-  (merge capabilities/trusted-helper-bindings
-         extra-helper-bindings))
-
-(def query-transform-symbols
-  "Top-level structured query-transform operator symbols."
-  capabilities/query-transform-symbols)
-
-(def helper-symbols
-  "All helper symbols accepted by the structured runtime."
-  (into capabilities/helper-symbols
-        (keys extra-helper-bindings)))
-(defn op-symbol
-  "Normalize helper identifiers into canonical operator symbols."
-  [op]
-  (syntax/op-symbol op))
-
-(defn- resolve-in-scope-table-ids
-  "Default fallback when no in-scope table set is provided: load every table in the database.
-  Used by tests and any code path that hasn't yet plumbed in a scoped runtime."
-  [metadata-provider]
-  (into #{} (map :id) (lib.metadata/tables metadata-provider)))
-
-(defn build-runtime
-  "Build the trusted structured-program runtime for a metadata provider.
-
-  Options:
-    :in-scope-table-ids — set of table-ids the runtime should expose for name-based lookups
-                           and FK chain resolution. Defaults to every table in the database
-                           (used by tests and legacy callers); production should always pass
-                           the scoped set derived from the program references.
-    :extra-bindings     — additional runtime bindings merged on top of the standard ones."
-  ([metadata-providerable]
-   (build-runtime metadata-providerable nil))
-  ([metadata-providerable {:keys [in-scope-table-ids extra-bindings]}]
-   (let [metadata-provider (lib.metadata/->metadata-provider metadata-providerable)
-         table-ids         (or in-scope-table-ids
-                               (resolve-in-scope-table-ids metadata-provider))
-         tables-by-name    (runtime.fields/build-table-lookup metadata-provider table-ids)
-         {:keys [fields-by-table fields-by-id]}
-         (runtime.fields/build-field-lookups metadata-provider tables-by-name)
-         bindings          (merge (assoc trusted-helper-bindings
-                                         'expression-ref mbql/expression-ref-or-current-stage-column
-                                         'aggregation-ref mbql/aggregation-ref-or-current-stage-column
-                                         'skip-join? (fn [query operation]
-                                                       (mbql/redundant-implicit-join? fields-by-id query operation)))
-                                  (runtime.bindings/make-context-bindings metadata-provider
-                                                                          tables-by-name
-                                                                          fields-by-table
-                                                                          fields-by-id)
-                                  extra-bindings)]
-     {:metadata-provider metadata-provider
-      :tables-by-name    tables-by-name
-      :fields-by-table   fields-by-table
-      :fields-by-id      fields-by-id
-      :bindings          bindings})))
-
-(defn bindings-map
-  "Return the runtime bindings map."
-  [runtime]
-  (:bindings runtime))
-
-(defn helper-fn
-  "Look up a helper implementation from a runtime by operator.
-  Throws when the binding is nil so callers get a clear error instead of an NPE."
-  [runtime op]
-  (let [sym (op-symbol op)
-        f   (get (:bindings runtime) sym)]
-    (when-not f
-      (throw (ex-info (str "No runtime binding for operator: " sym)
-                      {:operator sym})))
-    f))

src/metabase/agent_lib/runtime/bindings.clj

@@ -1,18 +0,0 @@
-(ns metabase.agent-lib.runtime.bindings
-  "Context-dependent runtime bindings for structured MBQL evaluation."
-  (:require
-   [metabase.agent-lib.runtime.lookup :as runtime.lookup]
-   [metabase.agent-lib.runtime.transforms :as runtime.transforms]
-   [metabase.lib.core :as lib]))
-
-(set! *warn-on-reflection* true)
-
-(defn make-context-bindings
-  "Build query-aware structured helper bindings for the current runtime context."
-  [metadata-provider tables-by-name fields-by-table fields-by-id]
-  (merge {'query (fn [metadata] (lib/query metadata-provider metadata))}
-         (runtime.lookup/metadata-bindings metadata-provider
-                                           tables-by-name
-                                           fields-by-table
-                                           fields-by-id)
-         (runtime.transforms/query-transform-bindings)))

src/metabase/agent_lib/runtime/fields.clj

@@ -1,78 +0,0 @@
-(ns metabase.agent-lib.runtime.fields
-  "Field lookup and query-stage field resolution for structured MBQL runtime bindings."
-  (:require
-   [metabase.agent-lib.common.errors :as errors]
-   [metabase.agent-lib.mbql-integration :as mbql]
-   [metabase.lib.metadata :as lib.metadata]
-   [metabase.util :as u]))
-
-(set! *warn-on-reflection* true)
-
-(defn build-table-lookup
-  "Build a lowercase table-name lookup for `table-ids`.
-
-  Only the explicitly-requested tables are loaded — the runtime is scoped to entities the program
-  actually references (source + referenced + surrounding), not every table in the database.
-  Unknown tables are silently dropped; out-of-scope lookups will surface as nice errors at use time."
-  [metadata-provider table-ids]
-  (into {}
-        (keep (fn [tid]
-                (when-let [table (lib.metadata/table metadata-provider tid)]
-                  [(u/lower-case-en (:name table)) table])))
-        table-ids))
-
-(defn build-field-lookups
-  "Build both the `fields-by-table` and `fields-by-id` lookups for the tables in `tables-by-name`.
-
-  `fields-by-table` is a `{lower-case-table-name {lower-case-field-name field}}` map.
-  `fields-by-id` is a `{field-id field}` map.
-
-  Both maps are scoped to the tables passed in — typically the in-scope set computed from the
-  evaluation context, not every table in the database."
-  [metadata-provider tables-by-name]
-  (reduce-kv (fn [acc table-name table-meta]
-               (let [fields (lib.metadata/fields metadata-provider (:id table-meta))]
-                 (-> acc
-                     (assoc-in [:fields-by-table table-name]
-                               (into {}
-                                     (map (fn [field]
-                                            [(u/lower-case-en (:name field)) field]))
-                                     fields))
-                     (update :fields-by-id
-                             (fn [m]
-                               (reduce (fn [m field] (assoc m (:id field) field)) m fields))))))
-             {:fields-by-table {} :fields-by-id {}}
-             tables-by-name))
-
-(defn- table-name-for-field-lookup
-  [metadata-provider tables-by-name table-name-or-id]
-  (if (number? table-name-or-id)
-    (if-let [table-meta (lib.metadata/table metadata-provider (int table-name-or-id))]
-      (u/lower-case-en (:name table-meta))
-      (errors/lookup-error! (str "Table not found by ID: " table-name-or-id)
-                            {:table-id table-name-or-id}
-                            {:available (keys tables-by-name)}))
-    (u/lower-case-en table-name-or-id)))
-
-(defn resolve-field-from-table
-  "Resolve a field by explicit table and field identifiers."
-  [metadata-provider tables-by-name fields-by-table table-name-or-id field-name]
-  (let [table-name            (table-name-for-field-lookup metadata-provider tables-by-name table-name-or-id)
-        normalized-field-name (u/lower-case-en field-name)
-        table-fields          (get fields-by-table table-name)]
-    (or (get table-fields normalized-field-name)
-        (errors/lookup-error! (str "Field not found: "
-                                   (if (number? table-name-or-id)
-                                     (str "(table " table-name-or-id ")")
-                                     table-name-or-id)
-                                   "." field-name)
-                              {:table-name table-name
-                               :field-name field-name}
-                              (when table-fields
-                                {:available (keys table-fields)})))))
-
-(defn resolve-field-in-query
-  "Resolve a metadata field against the columns available in the current query stage."
-  [fields-by-id query raw-field]
-  (mbql/resolve-field-in-query fields-by-id query raw-field
-                               (mbql/current-query-field-candidates query)))

src/metabase/agent_lib/runtime/lookup.clj

@@ -1,160 +0,0 @@
-(ns metabase.agent-lib.runtime.lookup
-  "Metadata and field lookup helpers for structured runtime bindings."
-  (:require
-   [metabase.agent-lib.common.errors :as errors]
-   [metabase.agent-lib.mbql-integration :as mbql]
-   [metabase.agent-lib.runtime.fields :as fields]
-   [metabase.lib.core :as lib]
-   [metabase.lib.metadata :as lib.metadata]
-   [metabase.util :as u]))
-
-(set! *warn-on-reflection* true)
-
-(defn lookup-table
-  "Resolve a table by numeric id or case-insensitive name."
-  [metadata-provider tables-by-name table-identifier]
-  (let [available-names (keys tables-by-name)]
-    (if (number? table-identifier)
-      (or (lib.metadata/table metadata-provider (int table-identifier))
-          (errors/lookup-error! (str "Table not found by ID: " table-identifier)
-                                {:table-id table-identifier}
-                                {:available available-names}))
-      (let [normalized (u/lower-case-en table-identifier)]
-        (or (get tables-by-name normalized)
-            (errors/lookup-error! (str "Table not found: " table-identifier)
-                                  {:table-name table-identifier}
-                                  {:available available-names}))))))
-
-(defn lookup-card
-  "Resolve a saved-question source, rejecting metrics."
-  [metadata-provider card-id]
-  (if (number? card-id)
-    (if-let [card-meta (lib.metadata/card metadata-provider (int card-id))]
-      (if (= (:type card-meta) :metric)
-        (errors/lookup-error! (str "Card " card-id " is a metric.")
-                              {:card-id card-id}
-                              {:suggestion (str "Use (metric " card-id ") instead.")})
-        card-meta)
-      (errors/lookup-error! (str "Card not found by ID: " card-id)
-                            {:card-id card-id}))
-    (errors/lookup-error! (str "Card lookup requires a numeric ID, got: " (pr-str card-id))
-                          {:card-id card-id})))
-
-(defn lookup-metric
-  "Resolve a metric by numeric id."
-  [metadata-provider metric-id]
-  (if (number? metric-id)
-    (or (lib.metadata/metric metadata-provider (int metric-id))
-        (errors/lookup-error! (str "Metric not found by ID: " metric-id)
-                              {:metric-id metric-id}))
-    (errors/lookup-error! (str "Metric lookup requires a numeric ID, got: " (pr-str metric-id))
-                          {:metric-id metric-id})))
-
-(defn lookup-measure
-  "Resolve a measure by numeric id."
-  [metadata-provider measure-id]
-  (if (number? measure-id)
-    (or (lib.metadata/measure metadata-provider (int measure-id))
-        (errors/lookup-error! (str "Measure not found by ID: " measure-id)
-                              {:measure-id measure-id}))
-    (errors/lookup-error! (str "Measure lookup requires a numeric ID, got: " (pr-str measure-id))
-                          {:measure-id measure-id})))
-
-(defn lookup-field
-  "Resolve a field by id, unique name, or explicit table/name pair."
-  ([metadata-provider _tables-by-name fields-by-table fields-by-id field-name-or-id]
-   (if (number? field-name-or-id)
-     (or (get fields-by-id (int field-name-or-id))
-         (lib.metadata/field metadata-provider (int field-name-or-id))
-         (errors/lookup-error! (str "Field not found by ID: " field-name-or-id)
-                               {:field-id field-name-or-id}))
-     (let [field-name (u/lower-case-en field-name-or-id)
-           matches    (into []
-                            (keep (fn [[table-name field-map]]
-                                    (when-let [field (get field-map field-name)]
-                                      [table-name field])))
-                            fields-by-table)]
-       (cond
-         (= 1 (count matches))
-         (second (first matches))
-
-         (> (count matches) 1)
-         (errors/lookup-error! (str "Ambiguous field \"" field-name-or-id "\".")
-                               {:field-name field-name-or-id}
-                               {:tables     (mapv first matches)
-                                :suggestion "Use (field \"TABLE\" \"FIELD\") to disambiguate."})
-
-         :else
-         (errors/lookup-error! (str "Field not found: \"" field-name-or-id "\".")
-                               {:field-name field-name-or-id}
-                               {:suggestion "Use (field \"TABLE\" \"FIELD\") with explicit table name."})))))
-  ([metadata-provider _tables-by-name fields-by-table _fields-by-id table-name-or-id field-name]
-   (fields/resolve-field-from-table metadata-provider
-                                    _tables-by-name
-                                    fields-by-table
-                                    table-name-or-id
-                                    field-name)))
-
-(defn query-relative-field
-  "Resolve a requested field against the current query stage."
-  ([metadata-provider tables-by-name fields-by-table fields-by-id query field-name-or-id]
-   (fields/resolve-field-in-query fields-by-id
-                                  query
-                                  (lookup-field metadata-provider
-                                                tables-by-name
-                                                fields-by-table
-                                                fields-by-id
-                                                field-name-or-id)))
-  ([metadata-provider tables-by-name fields-by-table fields-by-id query table-name-or-id field-name]
-   (let [raw-field (lookup-field metadata-provider
-                                 tables-by-name
-                                 fields-by-table
-                                 fields-by-id
-                                 table-name-or-id
-                                 field-name)]
-     (or (when (mbql/query? query)
-           (lib/find-matching-column query
-                                     -1
-                                     raw-field
-                                     (mbql/current-query-field-candidates query)))
-         (fields/resolve-field-in-query fields-by-id query raw-field)
-         raw-field))))
-
-(defn metadata-bindings
-  "Return metadata and field lookup bindings for the structured runtime."
-  [metadata-provider tables-by-name fields-by-table fields-by-id]
-  {'table   (partial lookup-table metadata-provider tables-by-name)
-   'card    (partial lookup-card metadata-provider)
-   'metric  (partial lookup-metric metadata-provider)
-   'measure (partial lookup-measure metadata-provider)
-   'field   (fn
-              ([field-name-or-id]
-               (lookup-field metadata-provider
-                             tables-by-name
-                             fields-by-table
-                             fields-by-id
-                             field-name-or-id))
-              ([a b]
-               ;; 2-arity: (field query field-name-or-id) or (field table-name field-name).
-               ;; Queries are always maps; table names are always strings or numbers.
-               (if (map? a)
-                 (query-relative-field metadata-provider
-                                       tables-by-name
-                                       fields-by-table
-                                       fields-by-id
-                                       a
-                                       b)
-                 (lookup-field metadata-provider
-                               tables-by-name
-                               fields-by-table
-                               fields-by-id
-                               a
-                               b)))
-              ([query table-name-or-id field-name]
-               (query-relative-field metadata-provider
-                                     tables-by-name
-                                     fields-by-table
-                                     fields-by-id
-                                     query
-                                     table-name-or-id
-                                     field-name)))})

src/metabase/agent_lib/runtime/transforms.clj

@@ -1,96 +0,0 @@
-(ns metabase.agent-lib.runtime.transforms
-  "Query-transform bindings for structured runtime evaluation."
-  (:require
-   [metabase.agent-lib.mbql-integration :as mbql]
-   [metabase.lib.core :as lib]
-   [metabase.lib.util :as lib.util]))
-
-(set! *warn-on-reflection* true)
-
-(def ^:private order-wrapper-direction-key
-  :order-direction)
-
-(def ^:private order-wrapper-value-key
-  :order-value)
-
-(defn order-wrapper?
-  "True when `x` wraps an orderable together with its direction."
-  [x]
-  (and (map? x)
-       (contains? x order-wrapper-direction-key)
-       (contains? x order-wrapper-value-key)))
-
-(defn asc-orderable
-  "Wrap an orderable with ascending sort direction."
-  [orderable]
-  {order-wrapper-direction-key :asc
-   order-wrapper-value-key     orderable})
-
-(defn desc-orderable
-  "Wrap an orderable with descending sort direction."
-  [orderable]
-  {order-wrapper-direction-key :desc
-   order-wrapper-value-key     orderable})
-
-(defn- freshen-orderable-ref
-  "Freshen ref uuids so repeated orderables do not alias the same clause instance."
-  [resolved-orderable]
-  (if (vector? resolved-orderable)
-    (lib.util/fresh-uuids resolved-orderable)
-    resolved-orderable))
-
-(defn apply-breakout
-  "Apply a breakout, materializing expression refs when needed."
-  [query breakoutable]
-  (if (and (lib.util/clause? breakoutable)
-           (not (lib.util/ref-clause? breakoutable)))
-    (let [[query' expr-ref] (mbql/ensure-query-expression-ref query breakoutable "breakout")]
-      (lib/breakout query' expr-ref))
-    (lib/breakout query breakoutable)))
-
-(defn apply-filter
-  "Apply a filter clause to a query."
-  [query clause]
-  (lib/filter query clause))
-
-(defn apply-with-fields
-  "Apply `with-fields`, expanding raw fields into breakouts on aggregated queries."
-  [query selections]
-  (let [selections'      (mapv #(mbql/resolve-aggregation-selection query %) selections)
-        field-selection? mbql/field-selection?
-        raw-fields       (filterv field-selection? selections')
-        other-selections (filterv (complement field-selection?) selections')]
-    (if (and (seq (lib/aggregations query))
-             (seq raw-fields))
-      (let [query' (reduce lib/breakout query raw-fields)]
-        (if (seq other-selections)
-          (lib/with-fields query' other-selections)
-          query'))
-      (lib/with-fields query selections'))))
-
-(defn apply-order-by
-  "Apply `order-by`, resolving query-relative orderables first."
-  ([query orderable]
-   (if (order-wrapper? orderable)
-     (let [[query' resolved] (mbql/resolve-orderable query (order-wrapper-value-key orderable))]
-       (lib/order-by query'
-                     (freshen-orderable-ref resolved)
-                     (order-wrapper-direction-key orderable)))
-     (let [[query' resolved] (mbql/resolve-orderable query orderable)]
-       (lib/order-by query' (freshen-orderable-ref resolved)))))
-  ([query a b]
-   (if (#{:asc :desc} a)
-     (let [[query' resolved] (mbql/resolve-orderable query b)]
-       (lib/order-by query' (freshen-orderable-ref resolved) a))
-     (let [[query' resolved] (mbql/resolve-orderable query a)]
-       (lib/order-by query' (freshen-orderable-ref resolved) b)))))
-
-(defn query-transform-bindings
-  "Return query-transform bindings for the structured runtime."
-  []
-  {'asc         asc-orderable
-   'desc        desc-orderable
-   'breakout    apply-breakout
-   'filter      apply-filter
-   'with-fields apply-with-fields
-   'order-by    apply-order-by})

src/metabase/agent_lib/schema.clj

@@ -1,128 +0,0 @@
-(ns metabase.agent-lib.schema
-  "Malli schemas for the structured MBQL program generation path."
-  (:require
-   [clojure.string :as str]
-   [metabase.agent-lib.capabilities :as capabilities]
-   [metabase.agent-lib.schema.top-level :as schema.top-level]
-   [metabase.util.i18n :refer [tru]]
-   [metabase.util.malli.humanize :as mu.humanize]
-   [metabase.util.malli.registry :as mr]
-   [metabase.util.malli.schema :as ms]))
-
-(set! *warn-on-reflection* true)
-
-(def ^:private top-level-operation-values
-  capabilities/top-level-operator-values)
-
-(defn- non-blank-string?
-  [value]
-  (and (string? value) (not (str/blank? value))))
-
-(defn- nested-operator-tuple?
-  [value]
-  (and (vector? value)
-       (string? (first value))
-       (not (str/blank? (first value)))
-       (not (top-level-operation-values (first value)))
-       (not= "query" (first value))))
-
-(defn- top-level-operation-tuple?
-  [value]
-  (and (vector? value)
-       (string? (first value))
-       (top-level-operation-values (first value))))
-
-(defn program-literal?
-  "True when `value` is a nested structured-program literal."
-  [value]
-  (and (map? value)
-       (= "program" (or (:type value) (get value "type")))
-       (or (contains? value :program)
-           (contains? value "program"))))
-
-(defn- plain-map-literal?
-  [value]
-  (and (map? value)
-       (not (program-literal? value))))
-
-(def ^:private registry
-  {::scalar               [:or nil? :string number? boolean?]
-   ::non-blank-string     [:and
-                           :string
-                           [:fn
-                            {:error/message "value must be a non-blank string"}
-                            non-blank-string?]]
-   ::vector-literal       [:vector [:ref ::node]]
-   ::page                 [:map {:closed true}
-                           [:page ms/PositiveInt]
-                           [:items ms/PositiveInt]]
-   ::source-ref           [:or
-                           [:tuple [:= "field"] ms/PositiveInt]
-                           [:tuple [:= "table"] ms/PositiveInt]
-                           [:tuple [:= "card"] ms/PositiveInt]
-                           [:tuple [:= "metric"] ms/PositiveInt]]
-   ::expression-ref-form  [:tuple [:= "expression-ref"] [:ref ::non-blank-string]]
-   ::aggregation-ref-form [:tuple [:= "aggregation-ref"] [:int {:min 0}]]
-   ::query-aware-ref      [:or
-                           [:ref ::expression-ref-form]
-                           [:ref ::aggregation-ref-form]]
-   ::nested-operator-form [:and
-                           [:vector {:min 1} [:ref ::node]]
-                           [:fn
-                            {:error/message "nested operator tuple must start with a non-blank non-transform operator string"}
-                            nested-operator-tuple?]]
-   ::program-literal      [:map {:closed true}
-                           [:type [:= "program"]]
-                           [:program [:ref ::program]]]
-   ::map-literal          [:and
-                           [:map-of :any [:ref ::node]]
-                           [:fn
-                            {:error/message "map literals cannot use the reserved `{\"type\":\"program\"}` shape unless they contain a valid nested program"}
-                            plain-map-literal?]]
-   ::form                 [:or
-                           [:ref ::source-ref]
-                           [:ref ::query-aware-ref]
-                           [:ref ::nested-operator-form]]
-   ::node                 [:or
-                           [:ref ::scalar]
-                           [:ref ::program-literal]
-                           [:ref ::map-literal]
-                           [:ref ::form]
-                           [:ref ::vector-literal]]
-   ::top-level-op         [:and
-                           schema.top-level/query-transform-form
-                           [:fn
-                            {:error/message "operation tuple must start with a valid top-level operator string"}
-                            top-level-operation-tuple?]]
-   ::source               [:or
-                           [:map {:closed true}
-                            [:type [:= "context"]]
-                            [:ref [:= "source"]]]
-                           [:map {:closed true}
-                            [:type [:enum "table" "card" "dataset" "metric"]]
-                            [:id ms/PositiveInt]]
-                           [:ref ::program-literal]]
-   ::program              [:map {:closed true}
-                           [:source [:ref ::source]]
-                           [:operations [:sequential [:ref ::top-level-op]]]]})
-
-(def source-schema
-  "Malli schema for structured-program sources."
-  [:schema {:registry registry} [:ref ::source]])
-
-(def program-schema
-  "Malli schema for full structured MBQL programs."
-  [:schema {:registry registry} [:ref ::program]])
-
-(defn validated-structure
-  "Validate the structural shape of a structured program and return it unchanged."
-  [program]
-  (when-let [error (mr/explain program-schema program)]
-    (throw (ex-info (tru "Generated program has an invalid structure.")
-                    {:status-code 400
-                     :error       :invalid-generated-program
-                     :humanized   (mu.humanize/humanize error)
-                     :details     (pr-str (mu.humanize/humanize error))
-                     :schema      ::program
-                     :value       program})))
-  program)

src/metabase/agent_lib/schema/top_level.clj

@@ -1,50 +0,0 @@
-(ns metabase.agent-lib.schema.top-level
-  "Top-level operation schema helpers derived from the capability catalog."
-  (:require
-   [metabase.agent-lib.capabilities.catalog.top-level :as capabilities.top-level]
-   [metabase.util.malli.schema :as ms]))
-
-(set! *warn-on-reflection* true)
-
-(def ^:private schema-ns
-  "metabase.agent-lib.schema")
-
-(defn- schema-ref
-  [local-name]
-  [:ref (keyword schema-ns local-name)])
-
-(def ^:private top-level-operation-schemas
-  {"filter"            [:tuple [:= "filter"] (schema-ref "form")]
-   "aggregate"         [:tuple [:= "aggregate"] (schema-ref "form")]
-   "breakout"          [:tuple [:= "breakout"] (schema-ref "form")]
-   "with-fields"       [:tuple [:= "with-fields"] [:sequential (schema-ref "form")]]
-   "limit"             [:tuple [:= "limit"] ms/PositiveInt]
-   "expression"        [:tuple [:= "expression"] (schema-ref "non-blank-string") (schema-ref "form")]
-   "join"              [:tuple [:= "join"] (schema-ref "form")]
-   "order-by"          [:or
-                        [:tuple [:= "order-by"] (schema-ref "form")]
-                        [:tuple [:= "order-by"] (schema-ref "form") [:enum "asc" "desc"]]]
-   "append-stage"      [:tuple [:= "append-stage"]]
-   "drop-stage"        [:tuple [:= "drop-stage"]]
-   "drop-empty-stages" [:tuple [:= "drop-empty-stages"]]
-   "with-page"         [:tuple [:= "with-page"] (schema-ref "page")]})
-
-(def ^:private top-level-op-names
-  (mapv (comp name :op) capabilities.top-level/capabilities))
-
-(defn- validate-top-level-coverage!
-  []
-  (let [defined-ops (set (keys top-level-operation-schemas))
-        catalog-ops (set top-level-op-names)]
-    (when (not= defined-ops catalog-ops)
-      (throw (ex-info "Top-level capability and schema definitions are out of sync."
-                      {:defined-ops defined-ops
-                       :catalog-ops catalog-ops})))))
-
-(validate-top-level-coverage!)
-
-(def query-transform-form
-  "Schema form for all valid top-level structured query transforms, ordered from the top-level capability catalog."
-  (into [:or]
-        (map top-level-operation-schemas)
-        top-level-op-names))

src/metabase/agent_lib/syntax.clj

@@ -1,94 +0,0 @@
-(ns metabase.agent-lib.syntax
-  "Shape-level helpers for structured MBQL operators and tuples."
-  (:require
-   [metabase.agent-lib.capabilities :as capabilities]
-   [metabase.util :as u]))
-
-(set! *warn-on-reflection* true)
-
-(def canonical-operator-aliases
-  "Legacy or alternate helper names keyed to their canonical string forms."
-  {"if"              "case"
-   "relative-date"   "relative-datetime"
-   "temporal-diff"   "datetime-diff"
-   "count-if"        "count-where"
-   "variance"        "var"
-   "stddev-pop"      "stddev"
-   "count-distinct"  "distinct"
-   "distinct-count"  "distinct"
-   "is-not-null"     "not-null"
-   "|"               "or"
-   "dayofweek"       "get-day-of-week"
-   "day-of-week"     "get-day-of-week"
-   "hour-of-day"     "get-hour"
-   "month"           "get-month"
-   "month-of-year"   "get-month"
-   "quarter-of-year" "get-quarter"})
-
-(def top-level-operation-symbols
-  "Top-level structured program operators keyed as symbols."
-  capabilities/query-transform-symbols)
-
-(def recognized-operator-symbols
-  "All recognized operator symbols, including the synthetic `query` helper."
-  (conj capabilities/helper-symbols 'query))
-
-(def repairable-operator-names
-  "Operator names that should be normalized by the repair pass."
-  (into #{"visualization" "is" "is-not"}
-        (concat
-         (map name recognized-operator-symbols)
-         (map name top-level-operation-symbols)
-         (keys canonical-operator-aliases)
-         ["dayofweek" "day-of-week" "hour-of-day" "month-of-year" "quarter-of-year"])))
-
-(defn raw-op-name
-  "Return the raw operator name string for a string, keyword, or symbol."
-  [value]
-  (cond
-    (string? value)  value
-    (keyword? value) (name value)
-    (symbol? value)  (name value)
-    :else            nil))
-
-(defn canonical-op-name
-  "Return the canonical string operator name for a structured helper."
-  [value]
-  (when-let [op-name (raw-op-name value)]
-    (or (canonical-operator-aliases op-name)
-        op-name)))
-
-(defn op-symbol
-  "Normalize helper identifiers into canonical operator symbols."
-  [op]
-  (some-> op canonical-op-name symbol))
-
-(defn possible-operator-tuple?
-  "True when `value` looks like a repairable structured helper tuple."
-  [value]
-  (and (vector? value)
-       (contains? repairable-operator-names
-                  (or (canonical-op-name (first value))
-                      (raw-op-name (first value))))))
-
-(defn boolean-wrapper-form?
-  "True when `value` is a legacy boolean wrapper tuple such as `[\"true\", expr]`."
-  [value]
-  (and (vector? value)
-       (= 2 (count value))
-       (contains? #{"true" "false"}
-                  (some-> (first value) raw-op-name u/lower-case-en))))
-
-(defn top-level-operation-tuple?
-  "True when `value` is a top-level structured operation tuple."
-  [value]
-  (and (possible-operator-tuple? value)
-       (contains? top-level-operation-symbols
-                  (symbol (canonical-op-name (first value))))))
-
-(defn operator-tuple?
-  "True when `value` is a recognized structured helper tuple."
-  [value]
-  (and (vector? value)
-       (string? (first value))
-       (contains? recognized-operator-symbols (op-symbol (first value)))))

src/metabase/agent_lib/validate.clj

@@ -1,38 +0,0 @@
-(ns metabase.agent-lib.validate
-  "Structural and semantic validation for structured MBQL programs."
-  (:require
-   [metabase.agent-lib.repair :as repair]
-   [metabase.agent-lib.schema :as schema]
-   [metabase.agent-lib.validate.semantic :as semantic]
-   [metabase.util.log :as log]
-   [metabase.util.malli :as mu]))
-
-(set! *warn-on-reflection* true)
-
-(def ^:private ContextEntitySummary
-  "Schema for a single entity summary in the evaluation context."
-  [:map
-   [:model :string]
-   [:id pos-int?]])
-
-(def EvaluationContext
-  "Schema for the evaluation context passed to validation and evaluation.
-  `:source-entity` identifies the primary data source.
-  `:surrounding-tables` provides join candidates visible in the current scope."
-  [:map
-   [:source-entity       ContextEntitySummary]
-   [:referenced-entities [:sequential ContextEntitySummary]]
-   [:surrounding-tables  [:sequential [:map [:id pos-int?]]]]])
-
-(mu/defn validated-program :- :map
-  "Repair and validate a structured program, returning the repaired program on success."
-  [program :- :map
-   context :- EvaluationContext]
-  (log/debug "Repairing structured program")
-  (let [program (-> program
-                    repair/repair-program
-                    (repair/repair-program-for-context context))]
-    (log/debug "Validating repaired program structure")
-    (schema/validated-structure program)
-    (log/debug "Validating repaired program semantics")
-    (semantic/validate-program program context)))

src/metabase/agent_lib/validate/context.clj

@@ -1,76 +0,0 @@
-(ns metabase.agent-lib.validate.context
-  "Context-derived validation helpers for structured programs."
-  (:require
-   [metabase.agent-lib.common.errors :refer [invalid-program!]]))
-
-(set! *warn-on-reflection* true)
-
-(defn context-allowed-ids
-  "Derive the ids that are valid for the current evaluation context."
-  [{:keys [source-entity referenced-entities surrounding-tables measure-ids]}]
-  (let [entity-summaries  (concat [source-entity] referenced-entities surrounding-tables)
-        summary-field-ids (fn [entity-summary]
-                            (->> (concat
-                                  (:columns entity-summary)
-                                  (get-in entity-summary [:query_summary :returned_columns])
-                                  (get-in entity-summary [:query_summary :filterable_columns])
-                                  (:queryable_dimensions entity-summary))
-                                 (keep :id)
-                                 (filter pos-int?)))
-        source-model      (:model source-entity)]
-    {:table-ids   (->> (concat
-                        (when (= source-model "table") [(:id source-entity)])
-                        (keep #(when (= "table" (:model %)) (:id %)) referenced-entities)
-                        (map :id surrounding-tables))
-                       (filter pos-int?)
-                       set)
-     :card-ids    (->> (concat
-                        (when (#{"card" "dataset"} source-model) [(:id source-entity)])
-                        (keep #(when (#{"card" "dataset"} (:model %)) (:id %)) referenced-entities))
-                       (filter pos-int?)
-                       set)
-     :metric-ids  (->> (concat
-                        (when (= source-model "metric") [(:id source-entity)])
-                        (keep #(when (= "metric" (:model %)) (:id %)) referenced-entities))
-                       (filter pos-int?)
-                       set)
-     :measure-ids (set (filter pos-int? measure-ids))
-     :field-ids   (->> entity-summaries
-                       (mapcat summary-field-ids)
-                       set)}))
-
-(defn validate-source!
-  "Validate a structured source against the context-derived allowed ids."
-  [validate-nested-program allowed-ids path source depth state]
-  (case (:type source)
-    "context"
-    (do
-      (when-not (= "source" (:ref source))
-        (invalid-program! (conj path :ref) "context source ref must be `source`"))
-      state)
-
-    "table"
-    (do
-      (when-not (pos-int? (:id source))
-        (invalid-program! (conj path :id) "source table id must be a numeric id"))
-      (when-not ((:table-ids allowed-ids) (:id source))
-        (invalid-program! (conj path :id) "source table id is not available in the provided context"))
-      state)
-
-    ("card" "dataset")
-    (do
-      (when-not ((:card-ids allowed-ids) (:id source))
-        (invalid-program! (conj path :id) "source card id is not available in the provided context"))
-      state)
-
-    "metric"
-    (do
-      (when-not ((:metric-ids allowed-ids) (:id source))
-        (invalid-program! (conj path :id) "source metric id is not available in the provided context"))
-      state)
-
-    "program"
-    (validate-nested-program (conj path :program) (:program source) (inc depth)
-                             (update state :program-nesting (fnil inc 0)))
-
-    (invalid-program! (conj path :type) "unsupported source type")))

src/metabase/agent_lib/validate/cross_checks.clj

@@ -1,40 +0,0 @@
-(ns metabase.agent-lib.validate.cross-checks
-  "Cross-operation semantic checks for structured programs."
-  (:require
-   [metabase.agent-lib.common.errors :refer [invalid-program!]]
-   [metabase.agent-lib.runtime :as runtime]
-   [metabase.agent-lib.syntax :as syntax]
-   [metabase.agent-lib.validate.walker :as walker]))
-
-(set! *warn-on-reflection* true)
-
-(defn validate-no-metric-order-by!
-  "Reject direct metric helpers nested inside `order-by`."
-  [operation-path operation]
-  (when (= 'order-by (runtime/op-symbol (first operation)))
-    (when-let [metric-path
-               (walker/find-tuple-path #(and (syntax/operator-tuple? %)
-                                             (= 'metric (runtime/op-symbol (first %))))
-                                       operation)]
-      (invalid-program! (into operation-path metric-path)
-                        "metric helpers cannot be used directly inside order-by. Order by a breakout field or a non-metric aggregation instead."
-                        {:operator "metric"}))))
-
-(defn validate-no-source-metric-reuse!
-  "Reject reusing the same metric already selected as the source."
-  [context operation-path operation]
-  (let [source-entity (:source-entity context)
-        source-id     (:id source-entity)]
-    (when (and (= "metric" (:model source-entity))
-               (pos-int? source-id))
-      (when-let [metric-path
-                 (walker/find-tuple-path #(and (syntax/operator-tuple? %)
-                                               (= 'metric (runtime/op-symbol (first %)))
-                                               (= source-id (second %)))
-                                         operation)]
-        (invalid-program! (into operation-path metric-path)
-                          (str "source is already metric " source-id
-                               ". Do not reference that same metric again inside operations; start "
-                               "from the source query and build on it.")
-                          {:operator "metric"
-                           :id       source-id})))))

src/metabase/agent_lib/validate/operators.clj

@@ -1,172 +0,0 @@
-(ns metabase.agent-lib.validate.operators
-  "Operator-specific semantic validation for structured programs."
-  (:require
-   [clojure.string :as str]
-   [metabase.agent-lib.capabilities :as capabilities]
-   [metabase.agent-lib.common.context :as common.context]
-   [metabase.agent-lib.common.errors :refer [invalid-program!]]
-   [metabase.agent-lib.common.literals :refer [aggregation-form?
-                                               bare-field-id?
-                                               non-blank-string?]]
-   [metabase.agent-lib.runtime :as runtime]
-   [metabase.agent-lib.syntax :as syntax]))
-
-(set! *warn-on-reflection* true)
-
-(def ^:private field-wrapper-required-arg-positions
-  "Argument positions that must be wrapped as `[\"field\", id]` for specific helpers."
-  '{sum [0]
-    avg [0]
-    min [0]
-    max [0]
-    distinct [0]
-    median [0]
-    stddev [0]
-    var [0]
-    percentile [0]
-    cum-sum [0]
-    sum-where [0]
-    distinct-where [0]})
-
-(defn ensure-arity!
-  "Validate a helper arity against the capability catalog."
-  [path op arg-count]
-  (when-let [allowed (capabilities/fixed-arities op)]
-    (when-not (allowed arg-count)
-      (invalid-program! path
-                        (if (and (= op 'expression) (= arg-count 1))
-                          (str "`expression` is a top-level operation, not a nested reference. "
-                               "Use `[\"expression\", \"Name\", expr]` as its own operation, and if you need to "
-                               "sort by that calculation, reuse the expression body directly inside `order-by`.")
-                          (format "operator `%s` expects %s argument(s), got %s"
-                                  (name op)
-                                  (str/join " or " (sort allowed))
-                                  arg-count))
-                        {:operator (name op)
-                         :allowed  (sort allowed)
-                         :actual   arg-count}))))
-
-(defn validate-operator-specific!
-  "Validate nested helper tuples against context-derived ids and operator rules."
-  [allowed-ids context path value]
-  (when (syntax/operator-tuple? value)
-    (let [[raw-op & args] value
-          op             (runtime/op-symbol raw-op)
-          op-name        (name op)]
-      (cond
-        (syntax/top-level-operation-symbols op)
-        (invalid-program! path
-                          (format "`%s` is a top-level operation and cannot be nested" op-name)
-                          {:operator op-name})
-
-        (= op 'query)
-        (invalid-program! path
-                          "`query` cannot appear inside operations; the program source already defines the base query."
-                          {:operator "query"})
-
-        :else
-        (do
-          (doseq [arg-position (get field-wrapper-required-arg-positions op)]
-            (when (bare-field-id? (nth args arg-position nil))
-              (invalid-program! (conj path (inc arg-position))
-                                (str "Wrap field ids with `[\"field\", id]` inside `" op-name "`. "
-                                     "Do not pass bare numeric ids directly to aggregation helpers.")
-                                {:operator       op-name
-                                 :retry-category :field-wrapper
-                                 :value          (nth args arg-position)})))
-          (case op-name
-            "case"
-            (when (some-> args second map?)
-              (when (or (contains? (second args) :default)
-                        (contains? (second args) "default"))
-                (invalid-program! (conj path 2)
-                                  "`case` uses the fallback value itself as the optional third argument. Do not use `{\"default\": ...}`. Omit the third argument instead of using null."
-                                  {:operator "case"})))
-
-            "table"
-            (let [id (first args)]
-              (when-not (pos-int? id)
-                (invalid-program! (conj path 1) "table lookup requires a numeric id")))
-
-            "card"
-            (let [id (first args)]
-              (when-not (pos-int? id)
-                (invalid-program! (conj path 1) "card lookup requires a numeric id"))
-              (when-not ((:card-ids allowed-ids) id)
-                (invalid-program! (conj path 1)
-                                  "card id is not available in the provided context"
-                                  {:operator "card" :id id})))
-
-            "metric"
-            (let [id (first args)]
-              (when-not (pos-int? id)
-                (invalid-program! (conj path 1) "metric lookup requires a numeric id"))
-              (when-not ((:metric-ids allowed-ids) id)
-                (invalid-program! (conj path 1)
-                                  "metric id is not available in the provided context"
-                                  {:operator "metric" :id id})))
-
-            "measure"
-            (let [id        (first args)
-                  source-id (common.context/source-metric-id context)]
-              (when-not (pos-int? id)
-                (invalid-program! (conj path 1) "measure lookup requires a numeric id"))
-              (when (and source-id (= source-id id))
-                (invalid-program! (conj path 1)
-                                  (str "source is already metric " source-id
-                                       ". Metric ids are not measure ids. Do not reference that same "
-                                       "metric as a `measure`; start from the source query and build on it.")
-                                  {:operator         "measure"
-                                   :id               id
-                                   :source-metric-id source-id}))
-              (when-not ((:measure-ids allowed-ids) id)
-                (invalid-program! (conj path 1)
-                                  "measure id is not available in the provided context"
-                                  {:operator "measure" :id id})))
-
-            "field"
-            (cond
-              (not= 1 (count args))
-              (invalid-program! path "field lookups must use a single numeric field id")
-
-              (not (pos-int? (first args)))
-              (invalid-program! (conj path 1) "field lookup requires a numeric id"))
-
-            "expression-ref"
-            (let [expr-name (first args)]
-              (when-not (non-blank-string? expr-name)
-                (invalid-program! (conj path 1)
-                                  "expression-ref requires a non-blank expression name"
-                                  {:operator "expression-ref"})))
-
-            "aggregation-ref"
-            (let [index (first args)]
-              (when-not (and (int? index) (<= 0 index))
-                (invalid-program! (conj path 1)
-                                  "aggregation-ref requires a zero-based aggregation index"
-                                  {:operator "aggregation-ref"
-                                   :value    index})))
-
-            nil))))))
-
-(defn validate-top-level-operation!
-  "Validate a top-level operation and its arguments."
-  [validate-node operation-path operation state]
-  (let [[raw-op & args] operation
-        op             (runtime/op-symbol raw-op)]
-    (when-not (syntax/top-level-operation-symbols op)
-      (invalid-program! (conj operation-path 0)
-                        "top-level operation is not allowed"
-                        {:operator raw-op}))
-    (ensure-arity! operation-path op (count args))
-    (when (= op 'aggregate)
-      (doseq [[idx arg] (map-indexed vector args)]
-        (when-not (aggregation-form? arg)
-          (invalid-program! (conj operation-path (inc idx))
-                            "aggregate operations require aggregation helpers like `count`, `sum`, `avg`, `percentile`, `metric`, or `measure`. Do not pass bare `aggregation-ref` or breakout clauses directly."
-                            {:operator "aggregate"
-                             :value    arg}))))
-    (reduce (fn [state [idx arg]]
-              (validate-node (conj operation-path (inc idx)) arg 1 state))
-            state
-            (map-indexed vector args))))

src/metabase/agent_lib/validate/program.clj

@@ -1,70 +0,0 @@
-(ns metabase.agent-lib.validate.program
-  "Recursive semantic program validation for agent-lib.
-
-  The local `letfn` in `validate-program-body!` is intentional: this namespace
-  is the recursive validation boundary for nested structured programs."
-  (:require
-   [clojure.string :as str]
-   [metabase.agent-lib.common.errors :refer [invalid-program!]]
-   [metabase.agent-lib.runtime :as runtime]
-   [metabase.agent-lib.schema :as schema]
-   [metabase.agent-lib.syntax :as syntax]
-   [metabase.agent-lib.validate.context :as validate.context]
-   [metabase.agent-lib.validate.cross-checks :as cross-checks]
-   [metabase.agent-lib.validate.operators :as operators]
-   [metabase.agent-lib.validate.walker :as walker]))
-
-(set! *warn-on-reflection* true)
-
-(defn validate-program-body!
-  "Validate a repaired structured program body, threading traversal state."
-  [allowed-ids context path program depth state]
-  (letfn [(semantic-visit [path value depth state]
-            (cond
-              (schema/program-literal? value)
-              (validate-program* (conj path :program) (:program value) (inc depth)
-                                 (update state :program-nesting (fnil inc 0)))
-
-              (vector? value)
-              (do
-                (when (and (some-> (first value) syntax/raw-op-name)
-                           (str/ends-with? (some-> (first value) syntax/raw-op-name) "-ref")
-                           (not (#{"expression-ref" "aggregation-ref"}
-                                 (some-> (first value) syntax/raw-op-name))))
-                  (invalid-program! (conj path 0)
-                                    "unknown helper reference"
-                                    {:operator (syntax/raw-op-name (first value))}))
-                (when (syntax/operator-tuple? value)
-                  (let [[raw-op & args] value
-                        op             (runtime/op-symbol raw-op)]
-                    (operators/validate-operator-specific! allowed-ids context path value)
-                    (operators/ensure-arity! path op (count args))))
-                state)
-
-              :else
-              state))
-          (validate-node [path value depth state]
-            (walker/walk-node path value depth state semantic-visit))
-          (validate-program* [path program depth state]
-            (let [nesting (get state :program-nesting 0)]
-              (when (> nesting walker/max-program-nesting)
-                (invalid-program! path "program source nesting exceeds maximum depth")))
-            (when (> (count (:operations program)) walker/max-operations)
-              (invalid-program! (conj path :operations) "program exceeds maximum operation count"))
-            (let [state (validate.context/validate-source! validate-program*
-                                                           allowed-ids
-                                                           (conj path :source)
-                                                           (:source program)
-                                                           depth
-                                                           state)]
-              (reduce (fn [state [idx operation]]
-                        (let [operation-path (into path [:operations idx])]
-                          (cross-checks/validate-no-metric-order-by! operation-path operation)
-                          (cross-checks/validate-no-source-metric-reuse! context operation-path operation)
-                          (operators/validate-top-level-operation! validate-node
-                                                                   operation-path
-                                                                   operation
-                                                                   state)))
-                      state
-                      (map-indexed vector (:operations program)))))]
-    (validate-program* path program depth state)))

src/metabase/agent_lib/validate/semantic.clj

@@ -1,14 +0,0 @@
-(ns metabase.agent-lib.validate.semantic
-  "Semantic validation rules for repaired structured MBQL programs."
-  (:require
-   [metabase.agent-lib.validate.context :as validate.context]
-   [metabase.agent-lib.validate.program :as validate.program]))
-
-(set! *warn-on-reflection* true)
-
-(defn validate-program
-  "Validate semantic structured-program rules after repair and structural validation."
-  [program context]
-  (let [allowed-ids (validate.context/context-allowed-ids context)]
-    (validate.program/validate-program-body! allowed-ids context [] program 0 {:node-count 0})
-    program))

src/metabase/agent_lib/validate/walker.clj

@@ -1,99 +0,0 @@
-(ns metabase.agent-lib.validate.walker
-  "Pure traversal helpers for structured program validation."
-  (:require
-   [metabase.agent-lib.common.coercions :refer [normalize-map-key]]
-   [metabase.agent-lib.common.errors :refer [invalid-program!]]
-   [metabase.agent-lib.common.literals :refer [scalar-literal?]]
-   [metabase.agent-lib.schema :as schema]))
-
-(def max-operations
-  "Maximum number of top-level operations allowed in a structured program."
-  32)
-
-(def max-depth
-  "Maximum nesting depth allowed in a structured program."
-  24)
-
-(def max-node-count
-  "Maximum total node count allowed in a structured program."
-  600)
-
-(def max-program-nesting
-  "Maximum recursive program-source nesting depth (prevents circular references)."
-  8)
-
-(def max-string-length
-  "Maximum length of any string value in a structured program."
-  1024)
-
-(set! *warn-on-reflection* true)
-
-(defn find-tuple-path
-  "Find the first path to a vector node matching `pred`.
-
-  The local `walk` helper stays nested because this namespace is the recursive
-  traversal boundary for validation."
-  [pred value]
-  (letfn [(walk [path node]
-            (cond
-              (and (vector? node) (pred node))
-              path
-
-              (vector? node)
-              (some identity
-                    (map-indexed (fn [idx child]
-                                   (walk (conj path idx) child))
-                                 node))
-
-              (map? node)
-              (if (schema/program-literal? node)
-                (walk (conj path :program) (:program node))
-                (some identity
-                      (map (fn [[k v]]
-                             (walk (conj path (normalize-map-key k)) v))
-                           node)))
-
-              (sequential? node)
-              (some identity
-                    (map-indexed (fn [idx child]
-                                   (walk (conj path idx) child))
-                                 node))
-
-              :else
-              nil))]
-    (walk [] value)))
-
-(defn walk-node
-  "Walk a structured node in preorder, threading pure traversal state through `visit`."
-  [path value depth state visit]
-  (when (> depth max-depth)
-    (invalid-program! path "program exceeds maximum nesting depth"))
-  (let [state (update state :node-count (fnil inc 0))]
-    (when (> (:node-count state) max-node-count)
-      (invalid-program! path "program exceeds maximum node count"))
-    (when (and (string? value) (> (count value) max-string-length))
-      (invalid-program! path "string value exceeds maximum length"
-                        {:length (count value) :max max-string-length}))
-    (let [state (visit path value depth state)]
-      (cond
-        (scalar-literal? value)
-        state
-
-        (map? value)
-        (if (schema/program-literal? value)
-          state
-          (reduce (fn [state [k v]]
-                    (walk-node (conj path (normalize-map-key k)) v (inc depth) state visit))
-                  state
-                  value))
-
-        (sequential? value)
-        (reduce (fn [state [idx element]]
-                  (walk-node (conj path idx) element (inc depth) state visit))
-                state
-                (map-indexed vector value))
-
-        :else
-        (invalid-program! path
-                          (str "unsupported literal type " (pr-str (type value)))
-                          {:value value})))))

src/metabase/driver/sql_jdbc/sync/describe_table.clj

@@ -656,7 +656,7 @@
   (->> (for [[field-path field-type] (seq field-types)
              :when field-type]
          (let [curr-type (get field-type-map field-type :type/*)]
-           {:name              (str/join " \u2192 " (map name field-path)) ;; right arrow
+           {:name              (str/join " → " (map name field-path)) ;; right arrow
             :database-type     (db-type-map curr-type)
             :base-type         curr-type
             ;; Postgres JSONB field, which gets most usage, doesn't maintain JSON object ordering...

src/metabase/mcp/README.md

@@ -97,7 +97,7 @@ clients can fetch supplementary content by URI without inflating tool descriptio
 
 | Resource URI                         | Description                                                                                                        |
 | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------ |
-| `metabase://docs/construct-query.md` | Program syntax for `construct_query` and `query` — sources, operations, operator forms, worked examples, pitfalls. |
+| `metabase://docs/construct-query.md` | Format reference for `construct_query` and `query` — top-level / stage shape, field references, filter / aggregation / temporal operators, joins, expressions, multi-stage queries, worked examples, pitfalls. |
 
 ## Supported JSON-RPC methods
 

src/metabase/mcp/resources.clj

@@ -261,14 +261,12 @@
 (register-resource!
  {:uri         "metabase://docs/construct-query.md"
   :name        "Construct Query Reference"
-  :description (str "Program syntax for `construct_query` and `query`: source shapes "
-                    "(table/card/dataset/metric), top-level operations (filter, aggregate, "
-                    "breakout, expression, with-fields, order-by, limit, join, append-stage, "
-                    "with-page), reference forms, filter/aggregation/temporal operator "
-                    "vocabularies, worked examples, and common pitfalls (stage boundaries, "
-                    "ref shapes, joins, metric/date handling).")
+  :description (str "Format reference for `construct_query` and `query`: top-level / "
+                    "stage shape, field references, filter / aggregation / temporal "
+                    "operators, joins, expressions, multi-stage queries, worked examples, "
+                    "and common pitfalls.")
   :mimeType    "text/markdown"
-  :render-fn   (classpath-text-resource "metabase/agent_api/construct_query.md")})
+  :render-fn   (classpath-text-resource "metabot/prompts/tools/construct_notebook_query.md")})
 
 (register-ui-resource!
  :visualize-query

src/metabase/mcp/tools.clj

@@ -31,39 +31,34 @@
 ;; describes its own wire schema, and this layer patches the manifest to publish the MCP-visible
 ;; shape.
 
-;; Shared sub-shapes for the program-shaped tools (`construct_query`, `query`). Extracted so the two
-;; tools can't drift on what a program looks like to the LLM — both reuse the same `:source` map and
-;; the same flattened `:operations` (`[:sequential [:sequential :any]]`, no tuple-of-anys, no `:and`).
-;; The wire schemas under `agent_api` use the precise tuple/composite grammar; this layer publishes
-;; the permissive variant strict MCP clients (ChatGPT) accept.
+;; Shared sub-shapes for the representations-shaped tools (`construct_query`, `query`).
+;;
+;; The wire schemas under `agent-api` describe the body as `{:query <::lib.schema/external-query>}`,
+;; and `::external-query` references `::lib.schema/query`, which carries `:optional` keys (notably
+;; `:lib/metadata`) that are not `[:maybe ...]`. The new `assert-optional-fields-nullable!` lint in
+;; `tools-manifest` walks every reachable map and fails on those — which is the right behaviour for
+;; tool inputs but not for the lib-internal query map we never want strict MCP clients to populate.
+;;
+;; The override here publishes a permissive `:query` field (an opaque JSON object) and conveys the
+;; real grammar through the tool description + `metabase://docs/construct-query.md`. Strict MCP
+;; clients (ChatGPT) get a clean JSON Schema (no `prefixItems` / `allOf`); the agent learns the
+;; actual MBQL 5 shape from the prompt.
 
-(def ^:private program-source-malli
-  [:map
-   [:type {:tool/description "Entity kind."}
-    [:enum "table" "card" "dataset" "metric"]]
-   [:id ms/PositiveInt]])
-
-(def ^:private program-operations-malli
-  [:sequential
-   [:sequential
-    {:tool/description (str "First element is the operator string; remaining "
-                            "elements are operator-specific arguments (scalars, "
-                            "references, or nested arrays).")}
-    :any]])
-
-(def ^:private operations-description
-  (str "Array of operator tuples like [\"filter\", clause] or [\"aggregate\", agg-clause]. "
-       "See metabase://docs/construct-query.md for the full grammar."))
+(def ^:private external-query-mcp-malli
+  "MCP-visible shape for the `query` field of `construct_query` / `query`. Deliberately opaque:
+  publishing the full `::lib.schema/external-query` recursively would pull in `::query`'s optional
+  non-nullable `:lib/metadata` (which trips the manifest's strict-tool lint) and emit `prefixItems` /
+  `allOf` JSON Schema constructs that strict MCP clients (ChatGPT) reject."
+  [:map {:tool/description (str "A Metabase MBQL 5 query as a JSON object. See the "
+                                "`construct_notebook_query` tool for the format reference.")}])
 
 (def ^:private construct-query-mcp-input-malli
   "MCP-visible input for `construct_query`.
-  Deliberately flatter than the wire schema — the operator/ref grammar is conveyed through the tool
-  description and the construct-query.md MCP resource, not the schema."
+  Deliberately flatter than the wire schema — the MBQL 5 representations grammar is conveyed through
+  the tool description and the construct-query.md MCP resource, not the schema."
   [:map
-   [:source     {:tool/description "Database entity to query."}
-    program-source-malli]
-   [:operations {:tool/description operations-description}
-    program-operations-malli]
+   [:query  {:tool/description "Metabase MBQL 5 query."}
+    external-query-mcp-malli]
    ;; Length bounds + description go on the inner `:string` so they end up on the JSON Schema branch
    ;; (and not as an `:allOf`, which ChatGPT's strict validator rejects).
    [:prompt {:optional true}
@@ -73,22 +68,20 @@
                                               "Pass it as-is without summarizing or rewriting.")}]]]])
 
 (def ^:private query-mcp-input-malli
-  "MCP-visible input for `query`. The wire body is a `:multi` whose `:program` branch references
-  agent-lib tuple/composite schemas — those emit `prefixItems`/`allOf` JSON Schema constructs that
-  strict MCP clients (ChatGPT) reject. This override publishes the same flattened program shape used
-  by `construct_query`, plus the `:continuation_token` alternative for pagination."
+  "MCP-visible input for `query`. The wire body is a `:multi` whose `:fresh` branch references
+  `::lib.schema/external-query` — that pulls in `::query`'s `:optional` non-nullable
+  `:lib/metadata` (and emits `prefixItems` / `allOf` JSON Schema constructs that strict MCP
+  clients reject). This override publishes the same opaque-object `:query` field used by
+  `construct_query`, plus the `:continuation_token` alternative for pagination."
   [:map
-   [:source             {:optional true
-                         :tool/description (str "Database entity to query. Omit when paginating via "
-                                                "`continuation_token`.")}
-    [:maybe program-source-malli]]
-   [:operations         {:optional true
-                         :tool/description operations-description}
-    [:maybe program-operations-malli]]
+   [:query              {:optional true
+                         :tool/description (str "Metabase MBQL 5 query. "
+                                                "Omit when paginating via `continuation_token`.")}
+    [:maybe external-query-mcp-malli]]
    [:continuation_token {:optional true
                          :tool/description (str "Token returned by a previous `query` response — pass "
                                                 "it back to fetch the next page. Mutually exclusive "
-                                                "with `source`/`operations`.")}
+                                                "with `query`.")}
     [:maybe ms/NonBlankString]]])
 
 (def ^:private execute-query-mcp-input-malli

src/metabase/metabot/agent/prompts.clj

@@ -59,13 +59,13 @@
           (log/debug "Tool prompt template not found:" path)
           nil))))
 
-(defn load-llm-representations-template
-  "Load the LLM representations template from resources/metabot/prompts/."
+(defn load-llm-shape-template
+  "Load the LLM-shape (output-side XML) template from resources/metabot/prompts/."
   [template-name]
   (let [path (str "metabot/prompts/" template-name)]
     (or (load-resource path)
         (do
-          (log/warn "LLM representations template not found:" path)
+          (log/warn "LLM-shape template not found:" path)
           nil))))
 
 ;;; Template Rendering
@@ -150,19 +150,19 @@
   [template-path]
   (:template (cached-load-template "tool" template-path load-tool-prompt-template)))
 
-(defn get-cached-llm-representations-template
-  "Get the LLM representations template from cache or load it."
+(defn get-cached-llm-shape-template
+  "Get the LLM-shape (output-side XML) template from cache or load it."
   []
-  (:template (cached-load-template "llm-representations"
-                                   "llm_representations.selmer"
-                                   load-llm-representations-template)))
+  (:template (cached-load-template "llm-shape"
+                                   "llm_shape.selmer"
+                                   load-llm-shape-template)))
 
 (defn get-cached-message-injection-template
   "Get the message injection template from cache or load it."
   []
   (:template (cached-load-template "message-injection"
                                    "message_injection.selmer"
-                                   load-llm-representations-template)))
+                                   load-llm-shape-template)))
 
 (defn clear-cache!
   "Clear the template cache. Useful for development/testing."

src/metabase/metabot/agent/user_context.clj

@@ -5,13 +5,16 @@
   recent views, user time formatting, and SQL dialect extraction from context."
   (:require
    [clojure.string :as str]
+   [metabase.agent-lib.representations.resolve :as repr.resolve]
    [metabase.lib-be.core :as lib-be]
    [metabase.lib.core :as lib]
    [metabase.metabot.tmpl :as te]
    [metabase.metabot.tools.entity-details :as entity-details]
-   [metabase.metabot.tools.shared.llm-representations :as llm-rep]
+   [metabase.metabot.tools.shared.content-store :as shared.content-store]
+   [metabase.metabot.tools.shared.llm-shape :as llm-shape]
    [metabase.metabot.util :as metabot.u]
    [metabase.util :as u]
+   [metabase.util.json :as json]
    [metabase.util.log :as log])
   (:import
    (java.time OffsetDateTime)
@@ -122,10 +125,11 @@
 
 ;; For saved entities (table, model, question, metric, dashboard), the frontend only sends
 ;; type + id. We fetch full details from the DB using entity-details and render them via
-;; llm-representations, mirroring what the Python AI service did via HTTP callbacks.
+;; llm-shape (the output-side XML formatters), mirroring what the Python AI service did
+;; via HTTP callbacks.
 
 (defn- fetch-and-format
-  "Fetch entity details and format with llm-rep. Falls back to format-simple-entity on failure."
+  "Fetch entity details and format with llm-shape. Falls back to format-simple-entity on failure."
   [entity preamble details-fn format-fn]
   (try
     (let [{:keys [structured-output]} (details-fn)]
@@ -146,7 +150,7 @@
                                                         :with-metrics? false
                                                         :with-measures? true
                                                         :with-segments? true})
-                    llm-rep/table->xml))
+                    llm-shape/table->xml))
 
 (defmethod format-entity "model"
   [entity]
@@ -158,7 +162,7 @@
                                                         :with-metrics? false
                                                         :with-measures? true
                                                         :with-segments? true})
-                    llm-rep/model->xml))
+                    llm-shape/model->xml))
 
 (defn- format-chart-config-ids
   "Format chart config IDs for a viewing context item.
@@ -202,7 +206,7 @@
                       "The user is currently looking at the results of a report:"
                       #(entity-details/get-report-details {:report-id (:id entity)
                                                            :with-field-values? false})
-                      llm-rep/question->xml)))
+                      llm-shape/question->xml)))
 
 (defmethod format-entity "metric"
   [entity]
@@ -210,14 +214,14 @@
                     "The user is currently looking at the details of a metric:"
                     #(entity-details/get-metric-details {:metric-id (:id entity)
                                                          :with-field-values? false})
-                    llm-rep/metric->xml))
+                    llm-shape/metric->xml))
 
 (defmethod format-entity "dashboard"
   [entity]
   (fetch-and-format entity
                     "The user is currently looking at the details of a dashboard:"
                     #(entity-details/get-dashboard-details {:dashboard-id (:id entity)})
-                    llm-rep/dashboard->xml))
+                    llm-shape/dashboard->xml))
 
 ;;; Viewing Context Formatting
 
@@ -236,22 +240,42 @@
                                                te/lines)))))
 
 (defn- transform-query-source-text
+  "Format a transform's `:query` source for the LLM.
+
+  When the source carries a query map with a `:database` key, we normalise it and export to
+  the same canonical portable representations form the `construct_notebook_query` tool
+  consumes (rendered as a JSON code block). Both structured (`mbql.stage/mbql`) and native
+  (`mbql.stage/native`) stages go through this path - the latter is intentional: the repr
+  export preserves portable `card-id` / `snippet-id` references inside `template-tags`, and
+  stays in lockstep with the freshly-built-query payloads `construct_notebook_query` returns
+  to the LLM.
+
+  Pre-resolved string sources (`:query` is itself a string, or carries `:query-content` -
+  the SQL-tool's already-rendered shape) pass through unchanged: there's no map to
+  normalise.
+
+  Falls back to a `pprint`'d query map only as a last resort, when repr export is
+  unavailable (e.g. a partially-broken `dataset_query`)."
   [source]
   (let [query (:query source)]
     (cond
       (string? query) query
       (string? (:query-content query)) (:query-content query)
-      (string? (get-in query [:native :query])) (get-in query [:native :query])
       (and (map? query) (:database query))
       (try
-        (let [normalized (lib-be/normalize-query query)]
-          (if (lib/native-only-query? normalized)
-            (or (lib/raw-native-query normalized)
-                (some :native (:stages normalized))
-                (get-in normalized [:native :query]))
+        (let [normalized (lib-be/normalize-query query)
+              database-id (:database normalized)
+              mp (when database-id
+                   (lib-be/application-database-metadata-provider database-id))
+              exported (some->> mp (#(repr.resolve/try-export-query % normalized shared.content-store/default-store)))]
+          (if exported
+            (str "```json\n" (json/encode exported {:pretty true}) "\n```")
             (u/pprint-to-str normalized)))
         (catch Exception _
           (u/pprint-to-str query)))
+      ;; Legacy native shape with no :database (rare). Surface the raw SQL so the LLM at
+      ;; least sees the query body; if there's no :database we can't normalise / build a MP.
+      (string? (get-in query [:native :query])) (get-in query [:native :query])
       (map? query) (u/pprint-to-str query)
       :else (some-> query str))))
 
@@ -368,10 +392,10 @@
   [_context]
   (try
     (when-let [{:keys [id name email-address glossary]} (:structured-output (entity-details/get-current-user nil))]
-      (llm-rep/user->xml {:id       id
-                          :name     name
-                          :email    email-address
-                          :glossary glossary}))
+      (llm-shape/user->xml {:id       id
+                            :name     name
+                            :email    email-address
+                            :glossary glossary}))
     (catch Exception e
       (log/error e "Error formatting current user info")
       nil)))

src/metabase/metabot/tools/charts.clj

@@ -9,7 +9,7 @@
    [metabase.metabot.tools.charts.edit :as edit-chart-tools]
    [metabase.metabot.tools.shared :as shared]
    [metabase.metabot.tools.shared.instructions :as instructions]
-   [metabase.metabot.tools.shared.llm-representations :as llm-rep]
+   [metabase.metabot.tools.shared.llm-shape :as llm-shape]
    [metabase.util.log :as log]
    [metabase.util.malli :as mu]))
 
@@ -17,7 +17,7 @@
 
 (defn- format-chart-output
   [{:keys [chart-id] :as structured}]
-  (let [chart-xml (llm-rep/chart->xml structured)]
+  (let [chart-xml (llm-shape/chart->xml structured)]
     (str "<result>\n" chart-xml "\n</result>\n"
          "<instructions>\n" (instructions/chart-created-instructions chart-id) "\n</instructions>")))
 

src/metabase/metabot/tools/construct.clj

@@ -1,139 +1,201 @@
 (ns metabase.metabot.tools.construct
   "Notebook query construction tool wrappers."
   (:require
-   [metabase.agent-lib.core :as agent-lib]
+   [metabase.agent-lib.representations :as repr]
+   [metabase.agent-lib.representations.repair :as repr.repair]
+   [metabase.agent-lib.representations.resolve :as repr.resolve]
+   [metabase.api.common :as api]
    [metabase.lib-be.core :as lib-be]
    [metabase.lib.core :as lib]
-   [metabase.lib.metadata :as lib.metadata]
    [metabase.metabot.agent.streaming :as streaming]
    [metabase.metabot.scope :as scope]
    [metabase.metabot.tmpl :as te]
    [metabase.metabot.tools.charts.create :as create-chart-tools]
+   [metabase.metabot.tools.shared.content-store :as shared.content-store]
    [metabase.metabot.tools.shared.instructions :as instructions]
-   [metabase.metabot.tools.shared.llm-representations :as llm-rep]
+   [metabase.metabot.tools.shared.llm-shape :as llm-shape]
    [metabase.metabot.tools.util :as tools.u]
+   [metabase.models.serialization.resolve :as serdes.resolve]
+   [metabase.models.serialization.resolve.mp :as resolve.mp]
    [metabase.util :as u]
+   [metabase.util.i18n :refer [tru]]
    [metabase.util.json :as json]
    [metabase.util.log :as log]
-   [metabase.util.malli :as mu]))
+   [metabase.util.malli :as mu]
+   [toucan2.core :as t2]))
 
 (set! *warn-on-reflection* true)
 
 ;;; ------------------------------------------------ Schema ------------------------------------------------
 
-(def ^:private source-entity-schema
-  "Schema for source_entity — identifies the primary data source."
-  [:map
-   [:type [:enum "table" "model" "question" "metric"]]
-   [:id :int]])
-
-(def construct-program-schema
-  "Schema for the program parameter of construct_notebook_query.
-  Intentionally loose — agent-lib validates and repairs internally."
-  [:map
-   [:source [:map
-             [:type :string]
-             [:id {:optional true} [:maybe :int]]
-             [:ref {:optional true} [:maybe :string]]]]
-   [:operations [:sequential [:sequential :any]]]])
-
 (def ^:private construct-visualization-schema
   [:map {:closed true}
    [:chart_type :string]])
 
 (def ^:private construct-notebook-query-args-schema
+  "Args schema for `construct_notebook_query`.
+
+  `:query` is a JSON object matching the canonical portable MBQL 5 wire format
+  ([[metabase.lib.schema/external-query]]). The structural shape (stages, clauses, portable
+  FKs, etc.) is documented for the LLM at
+  `resources/metabot/prompts/tools/construct_notebook_query.md`.
+
+  Validation against `::external-query` happens at the entry-point boundaries (HTTP defendpoint
+  via the string-transformer, `execute-representations-query` post-repair via
+  `lib.normalize/normalize ::lib.schema/query`); the args schema here only asserts `:query`
+  is map-shaped so the LLM-input forgiveness layer in repair has a chance to fix common
+  shortcuts (missing `{}` options, etc.). Per `repr-plan.md` step 13, this schema deliberately
+  omits `:source_entity` and `:referenced_entities` — the query body is self-describing."
   [:map {:closed true}
    [:reasoning {:optional true} :string]
-   [:source_entity source-entity-schema]
-   [:referenced_entities {:optional true} [:maybe [:sequential source-entity-schema]]]
-   [:program construct-program-schema]
+   [:query :map]
    [:visualization {:optional true} construct-visualization-schema]])
 
 ;;; ---------------------------------------- Source resolution ----------------------------------------
 
-(defn- resolve-source-database-id
-  "Resolve the database ID for a source_entity."
-  [{:keys [type id]}]
-  (case type
-    "table"                (:db_id (tools.u/get-table id :db_id))
-    ("model" "question")   (:database_id (tools.u/get-card id))
-    "metric"               (:database_id (tools.u/get-card id))
-    (throw (ex-info (str "Unsupported source_entity type: " type)
-                    {:agent-error? true :status-code 400}))))
+(defn- first-stage-source-table-fk
+  "Pull the portable `[db schema table]` FK out of `stages[0].source-table`, or `nil`
+  if not present / wrong shape."
+  [parsed-query]
+  (let [fk (get-in parsed-query ["stages" 0 "source-table"])]
+    (when (and (vector? fk)
+               (= 3 (count fk))
+               (string? (nth fk 0)))
+      fk)))
 
-(defn- source-entity->model-str
-  "Map source_entity type to the model string used by agent-lib evaluation context."
-  [type]
-  (case type
-    "table"      "table"
-    "model"      "dataset"
-    "question"   "card"
-    "metric"     "metric"
-    type))
+(def ^:private metabase-uri-source-table-pattern
+  "Matches values the LLM sometimes writes into `source-table:` by confusing the Metabase
+  `metabase://<entity-type>/<id>` URIs (which appear in prompts as a way to read entities)
+  with a query source handle. Examples we've observed: `metabase://metric/76`,
+  `metabase://table/123`, `metabase://model/42`, `metabase://question/9`.
 
-(defn program-source->source-entity
-  "Convert a structured-program `:source` map (using agent-lib model strings: `table`,
-  `card`, `dataset`, `metric`) into the `source-entity` shape consumed by
-  [[execute-program]] (using metabot type strings: `table`, `question`, `model`,
-  `metric`). Throws if the source isn't one of the four database-resolved types
-  (e.g. `context` or nested `program` sources are rejected)."
-  [{:keys [type id] :as source}]
-  (let [entity-type (case type
-                      "table"   "table"
-                      "card"    "question"
-                      "dataset" "model"
-                      "metric"  "metric"
-                      (throw (ex-info (str "Unsupported program source type: " (pr-str type))
-                                      {:agent-error? true
-                                       :status-code  400
-                                       :source       source})))]
-    {:type entity-type :id id}))
+  The regex is deliberately permissive — we catch anything starting with `metabase://` so
+  the error message is always the directive one instead of falling through to the generic
+  `:missing-source-in-first-stage` message."
+  #"^metabase://([^/]+)/(\d+)")
 
-(defn- source-metadata-for
-  "Resolve the lib metadata object for a source entity."
-  [type id metadata-provider]
-  (case type
-    "table"              (lib.metadata/table metadata-provider id)
-    ("model" "question") (lib.metadata/card metadata-provider id)
-    "metric"             (lib.metadata/card metadata-provider id)
-    nil))
+(defn- detect-metabase-uri-source-table!
+  "If `stages[0].source-table` is a `metabase://<type>/<id>` URI string, throw a targeted
+  agent-error explaining how to use the referenced entity correctly (typically: the LLM
+  meant to reference a metric — metrics are aggregations, not sources — but wrote the URI
+  as a source). No-op on well-formed sources."
+  [parsed-query]
+  (let [raw (get-in parsed-query ["stages" 0 "source-table"])]
+    (when (string? raw)
+      (when-let [[_ entity-type entity-id] (re-find metabase-uri-source-table-pattern raw)]
+        (let [hint (case entity-type
+                     "metric"
+                     (str "Metrics are aggregations, not sources. To use metric " entity-id
+                          ", put its `base_table_portable_fk` (from `entity_details` on the metric) "
+                          "into `source-table:` and reference the metric as "
+                          "`aggregation: [[metric, {}, \"<portable_entity_id>\"]]`.")
+                     ("question" "model" "card")
+                     (str "To reference saved " entity-type " " entity-id
+                          " as a query source, put its `portable_entity_id` (a 21-char "
+                          "string from `entity_details`) into `source-card:` — not a URI.")
+                     "table"
+                     (str "Use the portable FK `[<db-name>, <schema>, <table-name>]` from "
+                          "`entity_details` in `source-table:` — not a URI.")
+                     (str "`source-table:` accepts a portable FK `[<db-name>, <schema>, <table-name>]` "
+                          "or, via `source-card:`, a saved-card `portable_entity_id`."))]
+          (throw (ex-info (tru "`source-table:` does not accept URIs like `{0}`. {1}"
+                               raw hint)
+                          {:agent-error? true
+                           :status-code  400
+                           :error        :uri-in-source-table
+                           :source-table raw
+                           :entity-type  entity-type
+                           :entity-id    entity-id})))))))
 
-(defn- surrounding-tables-for
-  "Derive surrounding tables from a source query's visible columns."
-  [metadata-provider source-metadata source-id]
-  (try
-    (let [query (lib/query metadata-provider source-metadata)]
-      (->> (lib/visible-columns query)
-           (keep :table-id)
-           distinct
-           (remove #{source-id})
-           (mapv (fn [tid] {:id tid}))))
-    (catch Exception _ [])))
+(defn- first-stage-source-card-eid
+  "Pull the `source-card:` entity_id string from `stages[0]`, or `nil` if not present."
+  [parsed-query]
+  (let [eid (get-in parsed-query ["stages" 0 "source-card"])]
+    (when (string? eid) eid)))
 
-(defn- available-measure-ids
-  "Return the set of measure IDs available on a source entity, or empty set."
-  [metadata-provider source-metadata]
-  (try
-    (->> (lib/available-measures (lib/query metadata-provider source-metadata))
-         (keep :id)
-         set)
-    (catch Exception _ #{})))
+(def ^:private permission-aware-content-store
+  "ContentStore for agent query construction. Alias for
+  [[shared.content-store/default-store]] — the chokepoint wrapper applies `api/read-check` to
+  every lookup whenever `api/*current-user-id*` is bound, symmetrically across all five
+  ContentStore methods. The unchecked underlying store gates non-NanoID entity-id values to
+  avoid a full-table scan via `find-by-identity-hash`."
+  shared.content-store/default-store)
 
-(defn- build-evaluation-context
-  "Build the EvaluationContext for agent-lib from source_entity and referenced_entities."
-  [{:keys [type id]} referenced-entities metadata-provider]
-  (let [model-str      (source-entity->model-str type)
-        source-metadata (source-metadata-for type id metadata-provider)
-        surrounding     (surrounding-tables-for metadata-provider source-metadata id)
-        measure-ids     (available-measure-ids metadata-provider source-metadata)]
-    {:source-entity       {:model model-str :id id}
-     :referenced-entities (or (mapv (fn [{:keys [type id]}]
-                                      {:model (source-entity->model-str type) :id id})
-                                    referenced-entities)
-                              [])
-     :surrounding-tables  surrounding
-     :measure-ids         measure-ids
-     :source-metadata     source-metadata}))
+(defn- check-first-stage-source-table-query-permissions!
+  "Ensure the current user can query the table named by `stages[0].source-table`.
+
+  The metadata provider intentionally exposes database metadata without applying user data
+  permissions. Before any repair pass can inspect fields/FKs on the requested source table,
+  resolve the portable table FK and run the normal API query permission check."
+  [metadata-provider parsed-query]
+  (when-let [table-fk (first-stage-source-table-fk parsed-query)]
+    (let [resolver (resolve.mp/import-resolver metadata-provider permission-aware-content-store)
+          table-id (serdes.resolve/import-table-fk resolver table-fk)]
+      (api/query-check :model/Table table-id)
+      nil)))
+
+(defn resolve-database-id-from-first-stage
+  "Resolve the application database id from the first stage's source.
+
+  Public only so tests can stub it. Strategy:
+
+    * If `stages[0].source-table` is a portable FK `[db schema table]`, look up the database
+      by that `db` name. Unknown / ambiguous names surface `:unknown-database` / `:ambiguous-database-name`
+      agent-errors.
+    * Otherwise, if `stages[0].source-card` is an entity_id string, look up the card by entity_id
+      and use its `:database_id`. Unknown entity_id surfaces `:unknown-card`.
+    * Otherwise, surface `:missing-source-in-first-stage`.
+
+  All error paths are raised with `:agent-error? true` so the tool wrapper can relay a clean
+  message to the LLM instead of a stack trace.
+
+  Note: the `database:` field at the top level of the query is intentionally NOT consulted.
+  It's a spec-mandated but redundant field (the source already identifies the database); a
+  repair pass stamps it after we've resolved the id via this function. See `repr-plan.md`
+  step 14 follow-up."
+  [parsed-query]
+  (detect-metabase-uri-source-table! parsed-query)
+  (if-let [table-fk (first-stage-source-table-fk parsed-query)]
+    (let [db-name (nth table-fk 0)
+          ids     (t2/select-pks-vec :model/Database :name db-name)]
+      (case (count ids)
+        0 (throw (ex-info (tru (str "Unknown database: `{0}`. Use the exact database name as "
+                                    "reported by `entity_details` / metadata tools (it appears "
+                                    "as the first element of every portable FK, e.g. "
+                                    "`source-table: [<db-name>, <schema>, <table>]`).")
+                               db-name)
+                          {:agent-error? true
+                           :status-code  400
+                           :error        :unknown-database
+                           :database     db-name}))
+        1 (first ids)
+        (throw (ex-info (tru (str "Multiple databases share the name `{0}` (ids: {1}). The "
+                                  "agent has no way to disambiguate; ask the user to rename "
+                                  "one of the databases or use a more specific identifier.")
+                             db-name (pr-str (vec (sort ids))))
+                        {:agent-error? true
+                         :status-code  400
+                         :error        :ambiguous-database-name
+                         :database     db-name
+                         :database-ids (vec (sort ids))}))))
+    (if-let [eid (first-stage-source-card-eid parsed-query)]
+      (if-let [card (tools.u/get-card-by-entity-id eid)]
+        (:database_id card)
+        (throw (ex-info (tru (str "No saved question or model found with entity_id {0}. Do not invent "
+                                  "or guess entity_ids: call `entity_details` with `entity-type: question` "
+                                  "or `entity-type: model` and the card''s numeric id first, then copy the "
+                                  "exact `portable_entity_id` from the response into `source-card:`.")
+                             (pr-str eid))
+                        {:agent-error? true
+                         :status-code  400
+                         :error        :unknown-card
+                         :entity-id    eid})))
+      (throw (ex-info (tru (str "First stage must have either `source-table:` (as a portable FK "
+                                "`[<db-name>, <schema>, <table>]`) or `source-card:` (as an "
+                                "entity_id string). Neither was found in `stages[0]`."))
+                      {:agent-error? true
+                       :status-code  400
+                       :error        :missing-source-in-first-stage})))))
 
 ;;; ---------------------------------------- Result columns ----------------------------------------
 
@@ -146,20 +208,96 @@
 
 ;;; ---------------------------------------- Query execution ----------------------------------------
 
-(defn execute-program
-  "Execute a structured program via agent-lib.
-  Returns the raw result map with :structured-output.
-  Shared between construct-notebook-query-tool and slackbot-construct-notebook-query-tool."
-  [source-entity referenced-entities program]
-  (let [database-id (resolve-source-database-id source-entity)
-        mp          (lib-be/application-database-metadata-provider database-id)
-        context     (build-evaluation-context source-entity referenced-entities mp)
-        mbql5-query (agent-lib/evaluate-program program mp context)
-        query-id    (u/generate-nano-id)]
-    {:structured-output {:query-id       query-id
-                         :query          mbql5-query
-                         :result-columns (result-columns-for-query mbql5-query mp)}
-     :instructions      (instructions/query-created-instructions-for query-id)}))
+(defn- as-agent-input-error
+  "Wrap `e` as an agent-input error.
+
+  Tool callers look at `:agent-error?` to decide whether to relay the message to the LLM.
+  HTTP callers additionally need a status code; default representation/repair failures to
+  400 so individual repair passes don't all have to repeat `:status-code 400` in ex-data.
+  Existing statuses are preserved (notably permission 403s, which callers should normally
+  avoid wrapping in the first place)."
+  [^clojure.lang.ExceptionInfo e]
+  (let [base   (assoc (or (ex-data e) {}) :agent-error? true)
+        data   (cond-> base
+                 (nil? (:status-code base)) (assoc :status-code 400))]
+    (ex-info (ex-message e) data e)))
+
+(defn execute-representations-query
+  "Execute a notebook query in the canonical portable MBQL 5 representations format.
+
+  `external-query` is a keyword-keyed Clojure map matching [[metabase.lib.schema/external-query]]
+  — what the JSON middleware decodes from the HTTP body, or what the MCP tool layer hands in
+  after schema-coercion. Strict shape validation against `::external-query` is the
+  responsibility of the entry-point (defendpoint / mu/defn) — this function trusts the input
+  shape and concentrates on the LLM-friendly repair passes.
+
+  Pipeline:
+    1. Boundary-validate the keyword-keyed input against `::lib.schema/external-query`
+       ([[repr/validate-external-query]]). Catches structural issues — missing `stages`,
+       typo'd stage keys (e.g. `aggreagation` vs `aggregation`), wrong top-level `lib/type`,
+       etc. — that the loose `:map` wire schema doesn't catch.
+    2. Convert to the string-keyed portable form the repair pipeline operates on.
+    3. Resolve the database id from the first stage's `source-table:` / `source-card:`
+       ([[resolve-database-id-from-first-stage]]) and build an application-DB-backed
+       `MetadataProvider`.
+    4. Run the repair pass (fill in `{}` options, missing `lib/type` markers, stamp the
+       top-level `database:`, auto-wire `source-field` for implicit joins, rewrite inline
+       `order-by` aggregations to refs, etc.).
+    5. Sanity-check the post-repair shape against the portable repair schema.
+    6. Resolve portable FKs to numeric IDs and normalize through `lib.schema/query` against the
+       metadata-provider.
+    7. Export that final numeric pMBQL back to the portable form for the LLM-facing
+       `:query-json` / `query-content` output.
+
+  Returns a map with `:structured-output` and `:instructions` keys. Throws with an
+  `:agent-error?` ex-data flag when the LLM input is invalid, so the outer tool wrapper can
+  surface a helpful message to the LLM without a stack trace.
+
+  Per `repr-plan.md` step 13, there is no `source_entity` parameter — the query body carries
+  everything needed. Per the step-14 follow-up, there is also no top-level `database:` in the
+  LLM-facing contract: the database is derived from the first stage's source, and a repair
+  pass stamps `database:` into the portable form before lib.schema / resolve need it."
+  [external-query]
+  (let [parsed      (try
+                      ;; Step 1 — convert to the string-keyed portable form the repair pipeline
+                      ;; operates on.
+                      ;; Step 2 — boundary check that every stage's top-level keys are known.
+                      ;; Catches LLM-authored typos (e.g. `aggreagation` for `aggregation`)
+                      ;; that `lib.schema` does not enforce — `::lib.schema.mbql-stage/mbql` is
+                      ;; not a closed map, so unknown stage keys would otherwise be silently
+                      ;; dropped at resolve / lib.normalize time.
+                      (let [portable (repr/external-query->portable external-query)]
+                        (repr/assert-known-stage-keys! portable)
+                        portable)
+                      (catch clojure.lang.ExceptionInfo e
+                        (throw (as-agent-input-error e))))
+        database-id (resolve-database-id-from-first-stage parsed)
+        mp          (lib-be/application-database-metadata-provider database-id)]
+    ;; Permission checks happen before repair/resolve so the metadata-provider-backed pipeline
+    ;; never inspects table/card metadata that the current user cannot use.
+    (check-first-stage-source-table-query-permissions! mp parsed)
+    ;; Everything after the MP is built can surface LLM-input errors (lib.schema validation
+    ;; in resolve, missing-column complaints from lib/query in `result-columns-for-query`,
+    ;; etc.). Wrap the whole rest of the pipeline in a single `:agent-error?` relay so any of
+    ;; them reach the tool wrapper with the flag set.
+    (try
+      (let [repaired      (repr.repair/repair mp parsed permission-aware-content-store)
+            _validated    (repr/validate-query repaired)
+            pmbql-query   (repr.resolve/resolve-query mp repaired permission-aware-content-store)
+            exported-repr (repr.resolve/export-query mp pmbql-query permission-aware-content-store)
+            _validated'   (repr/validate-query exported-repr)
+            query-id      (u/generate-nano-id)]
+        {:structured-output {:query-id       query-id
+                             :query          pmbql-query
+                             :query-json     exported-repr
+                             :result-columns (result-columns-for-query pmbql-query mp)}
+         :instructions      (instructions/query-created-instructions-for query-id)})
+      (catch clojure.lang.ExceptionInfo e
+        ;; Permission failures are not LLM-input repair errors. Preserve the original 403 so
+        ;; HTTP callers get the standard forbidden response instead of an agent-error payload.
+        (if (= 403 (:status-code (ex-data e)))
+          (throw e)
+          (throw (as-agent-input-error e)))))))
 
 ;;; ---------------------------------------- Chart helpers ----------------------------------------
 
@@ -170,24 +308,42 @@
     (string? chart-type)  (keyword chart-type)
     :else                 chart-type))
 
+(defn- query-json->llm-content
+  "Render the portable repr-form `query-json` map as the JSON code block embedded inside
+  `<query>` for the LLM. The wire format is `::lib.schema/external-query` JSON; we serialize
+  with pretty-printing and a triple-backtick fence so the LLM sees the same syntactic frame it
+  would in a tool description or assistant turn."
+  [query-json]
+  (when (map? query-json)
+    (str "```json\n"
+         (json/encode query-json {:pretty true})
+         "\n```")))
+
 (defn- structured->query-data
-  "Convert tool structured output to a map suitable for [[llm-rep/query->xml]].
-  Converts the MBQL 5 query to legacy MBQL, JSON-encodes it, and wraps result columns."
-  [{:keys [query-id query result-columns]}]
+  "Convert tool structured output to a map suitable for [[llm-shape/query->xml]].
+
+  `:query-content` is the **canonical portable representations JSON** for the final pMBQL
+  query we actually constructed: repaired and resolved to numeric IDs, normalized by lib,
+  then exported back to portable FK paths/entity_ids. By feeding the LLM this final portable
+  form (rather than legacy-MBQL JSON or a pre-resolve approximation) on the next turn it can
+  reference exactly the field paths and aggregation UUIDs that will execute.
+
+  See repr-plan.md step 18."
+  [{:keys [query-id query query-json result-columns]}]
   (let [legacy-query (when (and (map? query) (:lib/type query))
                        #_{:clj-kondo/ignore [:discouraged-var]}
                        (lib/->legacy-MBQL query))]
     {:query-type    "notebook"
      :query-id      query-id
      :database_id   (:database legacy-query)
-     :query-content (when legacy-query (json/encode legacy-query))
+     :query-content (query-json->llm-content query-json)
      :result        (when (seq result-columns)
                       {:result_columns result-columns})}))
 
 (defn- structured->chart-xml
   "Render the full chart XML for the construct_notebook_query tool result."
   [structured chart-id chart-type]
-  (llm-rep/visualization->xml
+  (llm-shape/visualization->xml
    {:chart-id               chart-id
     :queries                [(structured->query-data structured)]
     :visualization_settings {:chart_type (if chart-type (name chart-type) "table")}}))
@@ -197,15 +353,16 @@
 (mu/defn ^{:tool-name "construct_notebook_query"
            :scope     scope/agent-notebook-create}
   construct-notebook-query-tool
-  "Construct and visualize a notebook query from a metric, model, or table."
-  [{:keys [_reasoning source_entity referenced_entities program visualization]} :- construct-notebook-query-args-schema]
+  "Construct and visualize a notebook query from a metric, model, or table.
+
+  Accepts an MBQL 5 query as a JSON object matching `::lib.schema/external-query`. See
+  `resources/metabot/prompts/tools/construct_notebook_query.md` for the prompt contract."
+  [{:keys [_reasoning query visualization]} :- construct-notebook-query-args-schema]
   (try
-    (let [;; LLM sometimes nests visualization inside program — pull it out
-          effective-viz            (or visualization (:visualization program))
-          normalized-visualization (some-> effective-viz (update-keys (comp keyword u/->kebab-case-en name)))
+    (let [normalized-visualization (some-> visualization (update-keys (comp keyword u/->kebab-case-en name)))
           chart-type              (or (chart-type->keyword (:chart-type normalized-visualization))
                                       :table)
-          query-result            (execute-program source_entity referenced_entities (dissoc program :visualization))
+          query-result            (execute-representations-query query)
           structured              (or (:structured-output query-result) (:structured_output query-result))]
       (if (and structured (:query-id structured) (:query structured))
         (let [chart-result (create-chart-tools/create-chart
@@ -236,14 +393,21 @@
            :instructions      instruction-text})
         ;; query-result may already have :output (error) or only :structured-output
         (if-let [s (or (:structured-output query-result) (:structured_output query-result))]
-          (let [query-xml        (llm-rep/query->xml (structured->query-data s))
+          (let [query-xml        (llm-shape/query->xml (structured->query-data s))
                 instruction-text (instructions/query-created-instructions-for (:query-id s))]
             (assoc query-result
                    :output (str "<result>\n" query-xml "\n</result>\n"
                                 "<instructions>\n" instruction-text "\n</instructions>")))
           query-result)))
     (catch Exception e
-      (log/error e "Failed to construct notebook query")
       (if (:agent-error? (ex-data e))
-        {:output (ex-message e)}
-        {:output (str "Failed to construct notebook query: " (or (ex-message e) "Unknown error"))}))))
+        ;; Expected agent-facing signal (bad LLM input: unknown table, unknown schema,
+        ;; URI-in-source-table, …). Log at debug only — no stacktrace — since the message
+        ;; is the tool's result and the LLM is expected to self-correct on the next turn.
+        (do
+          (log/debug e "construct_notebook_query returned agent-error to the LLM")
+          {:output (ex-message e)})
+        ;; Genuine unexpected failure — keep full stacktrace.
+        (do
+          (log/error e "Failed to construct notebook query")
+          {:output (str "Failed to construct notebook query: " (or (ex-message e) "Unknown error"))})))))

src/metabase/metabot/tools/document.clj

@@ -180,11 +180,16 @@
         {:output (str "Failed to construct SQL chart draft: " (or (ex-message e) "Unknown error"))}))))
 
 (def ^:private model-chart-schema
+  "Schema for `document_construct_model_chart`. Mirrors `construct_notebook_query`'s
+  representations format: `:query` is a YAML string in MBQL 5 representations format.
+
+  Per `repr-plan.md` step 13, `:source_entity` is no longer part of the contract — the YAML
+  query is self-describing (carries `database:` at the top level and full portable FK paths
+  everywhere else)."
   [:map {:closed true}
    [:name :string]
    [:description :string]
-   [:source_entity [:map [:type :string] [:id :int]]]
-   [:program :map]
+   [:query :string]
    [:viz_settings [:map {:closed true}
                    [:chart_type chart-type-enum]]]])
 
@@ -192,12 +197,11 @@
            :scope     scope/agent-document-create}
   document-construct-model-chart-tool
   "Construct notebook/model-backed chart draft payload for document insertion."
-  [{:keys [name description source_entity program viz_settings]} :- model-chart-schema]
+  [{:keys [name description query viz_settings]} :- model-chart-schema]
   (try
     (let [chart-type (get viz_settings :chart_type)
           result     (construct-tools/construct-notebook-query-tool
-                      {:source_entity source_entity
-                       :program program
+                      {:query query
                        :visualization {:chart_type chart-type}})
           structured (or (:structured-output result) (:structured_output result))
           query-id   (:query-id structured)

src/metabase/metabot/tools/entity_details.clj

@@ -1,6 +1,7 @@
 (ns metabase.metabot.tools.entity-details
   (:require
    [medley.core :as m]
+   [metabase.agent-lib.representations.resolve :as repr.resolve]
    [metabase.api.common :as api]
    [metabase.documents.core :as documents]
    [metabase.lib-be.core :as lib-be]
@@ -8,6 +9,7 @@
    [metabase.lib.metadata :as lib.metadata]
    [metabase.lib.types.isa :as lib.types.isa]
    [metabase.metabot.config :as metabot.config]
+   [metabase.metabot.tools.shared.content-store :as shared.content-store]
    [metabase.metabot.tools.util :as metabot.tools.u]
    [metabase.parameters.field-values :as params.field-values]
    [metabase.util :as u]
@@ -18,12 +20,38 @@
 
 (set! *warn-on-reflection* true)
 
+(defn- entity-id-for-measure-or-segment
+  "Look up the portable `entity_id` for a measure or segment by numeric id. The lib metadata
+  view of measures and segments doesn't carry `:entity-id`, so we hit the app DB directly.
+  `model-key` is `:model/Measure` or `:model/Segment`."
+  [model-key id]
+  (when id
+    (t2/select-one-fn :entity_id model-key :id id)))
+
 (defn- convert-measure-or-segment
   "Convert a measure or segment metadata object to the format expected by the API.
-   `definition-key` is `:aggregation` for measures and `:filters` for segments."
+   `definition-key` is `:aggregation` for measures and `:filters` for segments.
+
+   `:definition` on the result is the portable representations form of the relevant inner
+   clause array (a vector of one aggregation clause for measures, a vector of one or more
+   filter clauses for segments). `:portable-entity-id` is the 21-char NanoID the agent can
+   paste verbatim into a `[measure, {}, <pid>]` / `[segment, {}, <pid>]` clause — this is
+   the preferred form when the agent wants to use the measure/segment as-is. The
+   `:definition` array remains useful when the agent wants to compose on top of it (e.g.
+   add an extra filter that the segment doesn't include). See
+   `resources/metabot/prompts/tools/construct_notebook_query.md`."
   [metadata definition-key]
-  (let [definition (:definition metadata)]
+  (let [definition  (:definition metadata)
+        model-key   (case definition-key
+                      :aggregation :model/Measure
+                      :filters     :model/Segment)
+        ;; Lib-metadata maps reject snake_case lookups (snake-hating-map). Try kebab first;
+        ;; fall back to a direct t2 lookup since the lib metadata view doesn't include
+        ;; `:entity-id` for measures/segments (verified via REPL — only cards include it).
+        entity-id   (or (:entity-id metadata)
+                        (entity-id-for-measure-or-segment model-key (:id metadata)))]
     (-> (select-keys metadata [:id :name :display-name :description])
+        (assoc :portable-entity-id entity-id)
         (assoc :definition-description
                (when definition
                  (try
@@ -32,8 +60,13 @@
         (assoc :definition
                (when definition
                  (try
-                   (lib/prepare-for-serialization definition)
-                   (catch Exception _ nil)))))))
+                   (let [mp       (lib-be/application-database-metadata-provider (:database definition))
+                         exported (repr.resolve/export-query mp definition shared.content-store/default-store)]
+                     (get-in exported ["stages" 0 (name definition-key)]))
+                   (catch Exception e
+                     (log/warn e "Failed to export measure/segment definition to portable form"
+                               {:id (:id metadata) :key definition-key})
+                     nil)))))))
 
 (defn verified-review?
   "Return true if the most recent ModerationReview for the given item id/type is verified."
@@ -120,6 +153,32 @@
                                    with-queryable-dimensions?      true
                                    with-segments?                  false}}]
    (let [id (:id card)
+         ;; Database metadata so the LLM can build portable FKs — metrics are always
+         ;; referenced from a query targeting a specific database, and the LLM needs the
+         ;; human-readable database name for `source-table:` / field FKs in the same stage.
+         database (lib.metadata/database metadata-provider)
+         database-id (:id database)
+         database-name (:name database)
+         ;; Base table of the metric — the metric is an aggregation defined on top of a
+         ;; specific table. To use the metric, the LLM has to put that table in
+         ;; `source-table:` and reference the metric in `aggregation:`.
+         ;;
+         ;; This arity is called with two shapes of `card`:
+         ;;   1. The t2 row from `get-card` (via the single-arg `metric-details`) —
+         ;;      snake_case, so `:table_id`.
+         ;;   2. A `lib.metadata/card` map forwarded by `convert-metric` (called from
+         ;;      `table-details` while listing available metrics) — that map is a
+         ;;      `SnakeHatingMap`, so the only valid key is kebab-case `:table-id`; reading
+         ;;      `:table_id` on it throws a deprecation-error that would surface to the LLM
+         ;;      as an empty `<resource>` with just `**Error: ...**`.
+         ;; We accept both. `report_card.table_id` / metadata `:table-id` already points
+         ;; to the metric's base table and is kept in sync, so we use it verbatim instead
+         ;; of digging through `:dataset_query`.
+         source-table-id (or (:table-id card) (:table_id card))
+         source-table (when source-table-id
+                        (lib.metadata/table metadata-provider source-table-id))
+         base-table-portable-fk (when (and database-name source-table)
+                                  [database-name (:schema source-table) (:name source-table)])
          query-needed? (or with-default-temporal-breakout? with-queryable-dimensions? with-segments?)
          metric-query (when query-needed?
                         (lib/query metadata-provider (lib.metadata/card metadata-provider id)))
@@ -138,6 +197,19 @@
               :type :metric
               :name (:name card)
               :description (:description card)
+              ;; Database identity — same shape as in `table-details` / `card-details`.
+              :database_id database-id
+              :database_name database-name
+              ;; Base table the metric aggregates. The LLM uses `:base_table_portable_fk`
+              ;; verbatim as `source-table:` in the query that consumes the metric.
+              :base_table_id source-table-id
+              :base_table_name (:name source-table)
+              :base_table_portable_fk base-table-portable-fk
+              ;; Portable entity id — the string the LLM must copy verbatim into a
+              ;; `[metric, {}, <entity_id>]` aggregation clause to reference this metric.
+              ;; Accept both shapes (see `source-table-id` note above): t2 rows use
+              ;; `:entity_id`, `lib.metadata/card` maps use `:entity-id`.
+              :portable_entity_id (or (:entity-id card) (:entity_id card))
               :default_time_dimension_field_id (some-> default-temporal-breakout
                                                        (->> (metabot.tools.u/->result-column metric-query))
                                                        :field_id)
@@ -160,7 +232,11 @@
   ([db-metric metadata-provider options]
    (-> db-metric
        (metric-details metadata-provider (assoc options :with-queryable-dimensions? false))
-       (select-keys  [:id :type :name :description :default_time_dimension_field_id]))))
+       ;; Keep the DB-name / base-table-fk / portable_entity_id fields so callers (incl.
+       ;; `metric->xml`) can surface them to the LLM without a second round-trip.
+       (select-keys [:id :type :name :description :default_time_dimension_field_id
+                     :database_id :database_name :portable_entity_id
+                     :base_table_id :base_table_name :base_table_portable_fk]))))
 
 (declare related-tables)
 
@@ -180,10 +256,14 @@
                      (metabot.tools.u/get-table id :db_id :description :name :schema))]
      (let [query-needed? (or with-fields? with-related-tables? with-metrics? with-measures? with-segments?)
            db-id (if metadata-provider (:db-id base) (:db_id base))
-           db-engine (some-> (if metadata-provider
-                               (lib.metadata/database metadata-provider)
-                               (metabot.tools.u/get-database db-id :engine))
-                             :engine name)
+           ;; Database metadata including :name so LLM can build portable FK paths
+           ;; [db-name, schema, table] / [db-name, schema, table, field] that the
+           ;; representations-format construct_notebook_query expects.
+           db (if metadata-provider
+                (lib.metadata/database metadata-provider)
+                (metabot.tools.u/get-database db-id :engine :name))
+           db-engine (some-> db :engine name)
+           db-name (:name db)
            mp (when query-needed?
                 (or metadata-provider
                     (lib-be/application-database-metadata-provider db-id)))
@@ -203,8 +283,12 @@
             :display_name (some->> (:name base)
                                    (u.humanization/name->human-readable-name :simple))
             :database_id db-id
+            :database_name db-name
             :database_engine db-engine
-            :database_schema (:schema base)}
+            :database_schema (:schema base)
+            ;; Portable table FK path matching the representations-format expectation:
+            ;; [db-name, schema-or-null, table-name]. LLM uses this as `source-table`.
+            :portable_fk (when db-name [db-name (:schema base) (:name base)])}
            (m/assoc-some :description (:description base)
                          :related_tables related-tables
                          :metrics (when with-metrics?
@@ -254,7 +338,9 @@
                             :as   options}]
    (let [id (:id base)
          database-id (:database_id base)
-         database-engine (some-> (lib.metadata/database metadata-provider) :engine name)
+         database (lib.metadata/database metadata-provider)
+         database-engine (some-> database :engine name)
+         database-name (:name database)
          card-metadata (lib.metadata/card metadata-provider id)
          dataset-query (get card-metadata :dataset-query)
          query-needed? (or with-fields? with-related-tables? with-metrics? with-measures? with-segments?)
@@ -278,10 +364,38 @@
           :display_name (some->> (:name base)
                                  (u.humanization/name->human-readable-name :simple))
           :database_id database-id
+          :database_name database-name
           :database_engine database-engine
+          ;; Portable entity id matching the representations-format expectation: LLM uses this
+          ;; string as the value of `source-card:` in a representations query stage.
+          :portable_entity_id (:entity_id base)
           :verified (verified-review? id "card")}
          (m/assoc-some
           :description (:description base)
+          ;; Portable representations JSON for the card's **own** saved query body.
+          ;;
+          ;; We deliberately export `dataset-query` (the real, normalised MBQL/legacy body
+          ;; of the card) rather than `card-query`. `lib/query metadata-provider
+          ;; card-metadata` wraps the card as `{source-card: <eid>}` for downstream
+          ;; querying, which is the right thing for column resolution / metrics, but the
+          ;; wrong thing for showing the LLM what the card *actually does*: it would just
+          ;; round-trip back to a one-stage `source-card:` reference.
+          ;;
+          ;; Surfaces the **same** form the LLM writes into `construct_notebook_query`
+          ;; (and gets back in that tool's result), so existing-card payloads and
+          ;; freshly-built-query payloads share one shape. Works for both structured
+          ;; (`mbql.stage/mbql`) and native (`mbql.stage/native`) cards - the latter
+          ;; preserves portable `card-id` / `snippet-id` references inside
+          ;; `template-tags`.
+          ;;
+          ;; Returns `nil` (so `m/assoc-some` drops the key) for cards whose
+          ;; `dataset_query` can't be exported, e.g. unusual / partially-broken legacy
+          ;; queries; the rest of the payload is still useful.
+          :query_json (when dataset-query
+                        (repr.resolve/try-export-query
+                         metadata-provider
+                         (lib/query metadata-provider (lib-be/normalize-query dataset-query))
+                         shared.content-store/default-store))
           :related_tables related-tables
           :metrics (when with-metrics?
                      (not-empty (mapv #(convert-metric % metadata-provider options)
@@ -403,7 +517,11 @@
                             mp      (lib-be/application-database-metadata-provider (:database_id card))
                             details (card-details card mp options)]
                         (-> details
-                            (select-keys [:id :type :description :name :verified])
+                            ;; `:query_json` is intentionally part of the slim payload here:
+                            ;; it's what `question->xml` renders inside `<metabase_question>`
+                            ;; so the LLM sees the saved card's body in the same portable
+                            ;; representations JSON form `construct_notebook_query` consumes.
+                            (select-keys [:id :type :description :name :verified :query_json])
                             (assoc :result-columns (:fields details))
                             (m/assoc-some :average_query_time (:average_query_time card)
                                           :display (:display card))))

src/metabase/metabot/tools/metadata.clj

@@ -8,7 +8,7 @@
    [metabase.metabot.tools.field-stats :as field-stats-tools]
    [metabase.metabot.tools.shared :as shared]
    [metabase.metabot.tools.shared.instructions :as instructions]
-   [metabase.metabot.tools.shared.llm-representations :as llm-rep]
+   [metabase.metabot.tools.shared.llm-shape :as llm-shape]
    [metabase.util.i18n :refer [tru]]
    [metabase.util.log :as log]
    [metabase.util.malli :as mu]))
@@ -112,23 +112,23 @@
   [{:keys [metrics models]}]
   (let [content (str (when (seq metrics)
                        (str "<metrics>\n"
-                            (str/join "\n" (map llm-rep/metric->xml metrics))
+                            (str/join "\n" (map llm-shape/metric->xml metrics))
                             "\n</metrics>\n"))
                      (when (seq models)
                        (str "<metabase-models>\n"
-                            (str/join "\n" (map llm-rep/model->xml models))
+                            (str/join "\n" (map llm-shape/model->xml models))
                             "\n</metabase-models>")))]
     (format-with-instructions content instructions/answer-sources-instructions)))
 
 (defn- format-metadata-output
   [structured]
-  (llm-rep/get-metadata-result->xml structured))
+  (llm-shape/get-metadata-result->xml structured))
 
 (defn- format-field-metadata-output
   ;; NOTE: keep in sync with read_resource.clj/format-content :field-metadata branch
   [{:keys [field_id value_metadata]}]
   (format-with-instructions
-   (llm-rep/field-metadata->xml {:field_id field_id :value_metadata value_metadata})
+   (llm-shape/field-metadata->xml {:field_id field_id :value_metadata value_metadata})
    instructions/field-metadata-instructions))
 
 (defn- add-output

src/metabase/metabot/tools/resources.clj

@@ -58,7 +58,7 @@
    [metabase.metabot.tools.entity-details :as entity-details]
    [metabase.metabot.tools.field-stats :as field-stats]
    [metabase.metabot.tools.shared.instructions :as instructions]
-   [metabase.metabot.tools.shared.llm-representations :as llm-rep]
+   [metabase.metabot.tools.shared.llm-shape :as llm-shape]
    [metabase.models.interface :as mi]
    [metabase.transforms.core :as transforms]
    [metabase.util.log :as log]
@@ -151,7 +151,7 @@
    :name        name
    :engine      (some-> engine clojure.core/name)
    :description description
-   :uri         (llm-rep/metabase-uri :database id)})
+   :uri         (llm-shape/metabase-uri :database id)})
 
 (defn- present-collection
   "Trim a collection row to an item map. `path-name` may be supplied if the caller pre-computed it."
@@ -165,7 +165,7 @@
     :authority_level   authority_level
     :is_personal       (boolean personal_owner_id)
     :description       description
-    :uri               (llm-rep/metabase-uri :collection id)}))
+    :uri               (llm-shape/metabase-uri :collection id)}))
 
 (defn- present-table
   [{:keys [id name display_name schema db_id description]}]
@@ -176,7 +176,7 @@
    :schema       schema
    :database_id  db_id
    :description  description
-   :uri          (llm-rep/metabase-uri :table id)})
+   :uri          (llm-shape/metabase-uri :table id)})
 
 (defn- present-card
   "Cards (questions or models) — :type on a Card is :question / :model / :metric."
@@ -193,7 +193,7 @@
      :database_id   database_id
      :table_id      table_id
      :description   description
-     :uri           (llm-rep/metabase-uri (keyword model-type) id)}))
+     :uri           (llm-shape/metabase-uri (keyword model-type) id)}))
 
 (defn- present-dashboard
   [{:keys [id name collection_id description]}]
@@ -202,7 +202,7 @@
    :name          name
    :collection_id collection_id
    :description   description
-   :uri           (llm-rep/metabase-uri :dashboard id)})
+   :uri           (llm-shape/metabase-uri :dashboard id)})
 
 (defn- present-transform
   [{:keys [id name description source_database_id]}]
@@ -211,7 +211,7 @@
    :name        name
    :database_id source_database_id
    :description description
-   :uri         (llm-rep/metabase-uri :transform id)})
+   :uri         (llm-shape/metabase-uri :transform id)})
 
 (defn- present-source-card
   "Resolve a source-card id to a typed item map (model / metric / question)."
@@ -223,7 +223,7 @@
                    "question")]
     {:type src-type
      :id   source-card-id
-     :uri  (llm-rep/metabase-uri (keyword src-type) source-card-id)}))
+     :uri  (llm-shape/metabase-uri (keyword src-type) source-card-id)}))
 
 ;; ----- Lineage helpers -----
 
@@ -233,10 +233,10 @@
   (cond-> []
     database_id    (conj {:type "database"
                           :id   database_id
-                          :uri  (llm-rep/metabase-uri :database database_id)})
+                          :uri  (llm-shape/metabase-uri :database database_id)})
     table_id       (conj {:type "table"
                           :id   table_id
-                          :uri  (llm-rep/metabase-uri :table table_id)})
+                          :uri  (llm-shape/metabase-uri :table table_id)})
     source_card_id (conj (present-source-card source_card_id))))
 
 (defn- transform-source-table-ids
@@ -296,7 +296,7 @@
                            :id        id
                            :name      name
                            :timestamp timestamp
-                           :uri       (llm-rep/metabase-uri (keyword type) id)}))
+                           :uri       (llm-shape/metabase-uri (keyword type) id)}))
                       recents)]
     (list-result :recent-items items)))
 
@@ -344,7 +344,7 @@
                              {:type        "schema"
                               :name        s
                               :database_id db-id
-                              :uri         (llm-rep/metabase-uri :database db-id "schemas" s "tables")})))]
+                              :uri         (llm-shape/metabase-uri :database db-id "schemas" s "tables")})))]
     (list-result :database-schemas schemas)))
 
 (defn- fetch-database-schema-tables [id-str schema-name]
@@ -508,7 +508,7 @@
         items            (cond-> []
                            db-id         (conj {:type "database"
                                                 :id   db-id
-                                                :uri  (llm-rep/metabase-uri :database db-id)})
+                                                :uri  (llm-shape/metabase-uri :database db-id)})
                            source-tables (into source-tables))]
     (list-result :transform-sources items)))
 
@@ -524,7 +524,7 @@
         items        (cond-> []
                        db-id        (conj {:type "database"
                                            :id   db-id
-                                           :uri  (llm-rep/metabase-uri :database db-id)})
+                                           :uri  (llm-shape/metabase-uri :database db-id)})
                        target-table (conj (present-table target-table)))]
     (list-result :transform-target items)))
 
@@ -637,20 +637,20 @@
 
 (defn- format-content
   "Format a tool result as an LLM-ready string.
-   Dispatches to the right llm-rep formatter based on :result-type.
+   Dispatches to the right llm-shape formatter based on :result-type.
    Returns the :output string directly for error results (404s etc.)."
   [content]
   (if-let [structured (:structured-output content)]
     (case (:result-type structured)
       ;; NOTE: keep in sync with agent/tools/metadata.clj/format-field-metadata-output
       :field-metadata (format-with-instructions
-                       (llm-rep/field-metadata->xml structured)
+                       (llm-shape/field-metadata->xml structured)
                        instructions/field-metadata-instructions)
-      :entity         (llm-rep/entity->xml structured)
-      :metabot-list   (llm-rep/metabot-list->xml structured)
-      :metabot-entity (llm-rep/metabot-entity->xml structured)
+      :entity         (llm-shape/entity->xml structured)
+      :metabot-list   (llm-shape/metabot-list->xml structured)
+      :metabot-entity (llm-shape/metabot-entity->xml structured)
       ;; fallback — should not happen, but better than EDN
-      (llm-rep/entity->xml structured))
+      (llm-shape/entity->xml structured))
     ;; error case — :output is already a string
     (:formatted content)))
 

src/metabase/metabot/tools/search.clj

@@ -10,7 +10,7 @@
    [metabase.metabot.tmpl :as te]
    [metabase.metabot.tools.shared :as shared]
    [metabase.metabot.tools.shared.instructions :as instructions]
-   [metabase.metabot.tools.shared.llm-representations :as llm-rep]
+   [metabase.metabot.tools.shared.llm-shape :as llm-shape]
    [metabase.permissions.core :as perms]
    [metabase.search.core :as search]
    [metabase.search.engine :as search.engine]
@@ -75,13 +75,78 @@
                                    (update r :collection m/assoc-some :description (get descriptions cid))))))))
 
 (defn- enrich-with-database-engines
-  "Fetch and merge database engine info for search results that have database IDs."
+  "Fetch and merge database engine + name info for search results that have database IDs.
+  `:database_name` is the human-readable name the LLM needs as the first slot of every
+  portable FK in `construct_notebook_query`; surfacing it on every table/model search
+  result means the LLM doesn't need a separate `entity_details` round-trip just to learn
+  the DB name."
   [results]
-  (let [db-ids  (->> results (keep :database_id) distinct)
-        engines (when (seq db-ids)
-                  (t2/select-pk->fn :engine :model/Database :id [:in db-ids]))]
+  (let [db-ids (->> results (keep :database_id) distinct)
+        id->db (when (seq db-ids)
+                 (t2/select-pk->fn (juxt :engine :name) :model/Database :id [:in db-ids]))]
     (cond->> results
-      (seq engines) (mapv (fn [r] (m/assoc-some r :database_engine (get engines (:database_id r))))))))
+      (seq id->db) (mapv (fn [r]
+                           (let [[engine db-name] (get id->db (:database_id r))]
+                             (-> r
+                                 (m/assoc-some :database_engine engine)
+                                 (m/assoc-some :database_name db-name))))))))
+
+(defn- enrich-with-portable-entity-ids
+  "Attach `:portable_entity_id` (the card's `entity_id` NanoID) to saved-question, model,
+  and metric search results so the LLM can use it verbatim as `source-card:` (for
+  questions/models) or inside a `[metric, {}, <entity_id>]` aggregation clause (for
+  metrics) without a follow-up `entity_details` / `read_resource` round-trip."
+  [results]
+  (let [carded-types #{"question" "model" "metric"}
+        card-ids (->> results
+                      (filter #(carded-types (:type %)))
+                      (keep :id)
+                      distinct)
+        id->eid  (when (seq card-ids)
+                   (t2/select-pk->fn :entity_id :model/Card :id [:in card-ids]))]
+    (cond->> results
+      (seq id->eid) (mapv (fn [r]
+                            (if-let [eid (and (carded-types (:type r))
+                                              (get id->eid (:id r)))]
+                              (assoc r :portable_entity_id eid)
+                              r))))))
+
+(defn- enrich-with-metric-base-tables
+  "Attach base-table info (`:base_table_id`, `:base_table_name`, `:base_table_schema`,
+  `:base_table_portable_fk`) to metric search results.
+
+  A metric is a Card whose `:dataset_query` aggregates a specific table; the LLM needs that
+  table's portable FK as the `source-table:` when it wants to use the metric. Without this
+  enrichment the LLM sees the metric's `portable_entity_id` in search but has to either
+  hallucinate the base table (observed failure mode: `[<db>, public, customers]`) or do an
+  extra `entity_details` round-trip. We read the two columns directly from
+  `report_card.table_id` + `metabase_table.{schema,name}` to keep the lookup O(1) extra
+  query per search call, regardless of number of metrics in the result set.
+
+  Requires `:database_name` to already be set on each metric result (done earlier by
+  [[enrich-with-database-engines]]) so we can assemble the full portable FK
+  `[database_name, schema, table]`."
+  [results]
+  (let [metric-ids (->> results (filter #(= "metric" (:type %))) (keep :id) distinct)
+        card-id->table-id (when (seq metric-ids)
+                            (t2/select-pk->fn :table_id :model/Card :id [:in metric-ids]))
+        table-ids (->> card-id->table-id vals (remove nil?) distinct)
+        table-id->info (when (seq table-ids)
+                         (t2/select-pk->fn (juxt :schema :name) :model/Table :id [:in table-ids]))]
+    (cond->> results
+      (seq card-id->table-id)
+      (mapv (fn [r]
+              (if (= "metric" (:type r))
+                (if-let [table-id (get card-id->table-id (:id r))]
+                  (let [[schema table-name] (get table-id->info table-id)
+                        db-name (:database_name r)]
+                    (cond-> (assoc r :base_table_id table-id)
+                      table-name (assoc :base_table_name table-name
+                                        :base_table_schema schema)
+                      (and db-name table-name)
+                      (assoc :base_table_portable_fk [db-name schema table-name])))
+                  r)
+                r))))))
 
 (defn- remove-unreadable-transforms
   "Remove transforms from search results that the user cannot read.
@@ -238,12 +303,14 @@
          (map postprocess-search-result)
          enrich-with-collection-descriptions
          enrich-with-database-engines
+         enrich-with-portable-entity-ids
+         enrich-with-metric-base-tables
          remove-unreadable-transforms)))
 
 (defn- format-search-output
   "Format search results as an LLM-ready string."
   [results]
-  (let [results-xml (llm-rep/search-results->xml results)]
+  (let [results-xml (llm-shape/search-results->xml results)]
     (te/lines
      "<result>"
      results-xml

src/metabase/metabot/tools/shared/content_store.clj

@@ -0,0 +1,53 @@
+(ns metabase.metabot.tools.shared.content-store
+  "Permission-aware wrapper around `metabase.models.serialization.resolve.mp/ContentStore`.
+
+  The metadata-provider-backed resolver in `resolve.mp` is deliberately permission-agnostic so
+  that serdes import / background tasks can use it without an authenticated user. HTTP and
+  agent-tool paths run under a current user and therefore must layer in `api/read-check` —
+  this namespace is the chokepoint.
+
+  `read-checked` wraps any `ContentStore` so every lookup that returns a Card / Measure /
+  Segment row is gated by `api/read-check` whenever `api/*current-user-id*` is bound. When
+  the var is unbound (serdes, REPL, background tasks, tests without auth context), rows pass
+  through unchanged.
+
+  `default-store` is the standard agent / tool-path store — `unchecked-app-db-content-store`
+  wrapped with `read-checked`. Use this as the content store for any agent or tool path that
+  runs under an authenticated request. The symmetry across all five methods (both
+  `-by-entity-id` import-direction and `-by-id` export-direction) is intentional: the export
+  direction may surface entity_ids of Cards / Measures / Segments referenced inside an
+  exported query body, and a missing check on those branches is exactly the N1 ACL gap that
+  motivated this namespace."
+  (:require
+   [metabase.api.common :as api]
+   [metabase.models.serialization.resolve.mp :as resolve.mp]))
+
+(set! *warn-on-reflection* true)
+
+(defn- maybe-read-check
+  "Apply `api/read-check` when `*current-user-id*` is bound; otherwise return the row
+  unchanged. Returning `nil` propagates through (no row → nothing to check; the
+  per-model resolver functions translate `nil` into a clean `:unknown-…` agent error)."
+  [row]
+  (cond
+    (nil? row)              nil
+    api/*current-user-id*   (api/read-check row)
+    :else                   row))
+
+(defn read-checked
+  "Wrap `store` so every lookup applies `api/read-check` when `api/*current-user-id*` is
+  bound. Symmetric across all five `ContentStore` methods."
+  [store]
+  (reify resolve.mp/ContentStore
+    (card-by-entity-id    [_ eid] (maybe-read-check (resolve.mp/card-by-entity-id    store eid)))
+    (measure-by-entity-id [_ eid] (maybe-read-check (resolve.mp/measure-by-entity-id store eid)))
+    (segment-by-entity-id [_ eid] (maybe-read-check (resolve.mp/segment-by-entity-id store eid)))
+    (measure-by-id        [_ id]  (maybe-read-check (resolve.mp/measure-by-id        store id)))
+    (segment-by-id        [_ id]  (maybe-read-check (resolve.mp/segment-by-id        store id)))))
+
+(def default-store
+  "The standard agent / tool-path content store: `unchecked-app-db-content-store` wrapped with
+  `read-checked`. Pass this to `repr.resolve/resolve-query`, `repr.resolve/export-query`,
+  `repr.resolve/try-export-query`, `repair/repair`, or any other resolver entry-point that
+  runs under an authenticated request."
+  (read-checked resolve.mp/unchecked-app-db-content-store))

src/metabase/metabot/tools/shared/llm_shape.clj

@@ -1,5 +1,14 @@
-(ns metabase.metabot.tools.shared.llm-representations
-  "LLM representation utilities for formatting entities as XML.
+(ns metabase.metabot.tools.shared.llm-shape
+  "Output-side XML formatters that turn Metabase entity data (tables, fields, metrics,
+   queries, search results, ...) into the LLM-facing XML shape consumed by the metabot
+   agent. Templated via `resources/metabot/prompts/llm_shape.selmer`.
+
+   Naming note: this is the *output* (response-side) shape, not to be confused with the
+   *input*-side portable representations format owned by `metabase.agent-lib.representations.*`.
+   The two share no schema and no code path; they were briefly named symmetrically
+   (`llm-shaperesentations` here, `representations` there) and the symmetry caused enough
+   confusion to motivate this rename.
+
    Matches Python AI Service patterns exactly for consistency."
   (:require
    [clojure.string :as str]
@@ -7,13 +16,14 @@
    [metabase.metabot.agent.prompts :as prompts]
    [metabase.metabot.tmpl :as te]
    [metabase.util :as u]
+   [metabase.util.json :as json]
    [metabase.util.log :as log]
    [ring.util.codec :as codec]
    [selmer.parser :as selmer]))
 
 (set! *warn-on-reflection* true)
 
-(def ^:private llm-template-name "llm_representations.selmer")
+(def ^:private llm-template-name "llm_shape.selmer")
 
 (defn- encode-uri-segment
   "URL-encode a single URI path segment. Coerces keywords/numbers to strings first.
@@ -46,7 +56,24 @@
         (when (seq path-segments)
           (str "/" (str/join "/" (map encode-uri-segment path-segments)))))))
 
-(defn escape-xml
+(defn- repr-data->llm-block
+  "Render portable repr-form data (a map or a vector of clauses) as a JSON code block. Used
+  inside `<query>` for a whole query, and inside `<definition>` for a measure's aggregation
+  array or a segment's filter array. The triple-backtick fence gives the LLM the same
+  syntactic frame it would in a tool description or assistant turn. Accepts a pre-rendered
+  string passthrough; returns `nil` when the input is empty so the selmer template can omit
+  the block entirely."
+  [data]
+  (cond
+    (and (string? data)
+         (not (str/blank? data)))
+    data
+
+    (and (or (map? data) (vector? data))
+         (seq data))
+    (str "```json\n" (json/encode data {:pretty true}) "\n```")))
+
+(defn- escape-xml
   "Escape XML special characters in a string.
    Only needed for content that bypasses Selmer's auto-escaping (marked with |safe)."
   [s]
@@ -99,7 +126,7 @@
 (defn- render-llm-template
   "Render a Selmer template with the supplied context."
   [type context]
-  (let [template (prompts/get-cached-llm-representations-template)
+  (let [template (prompts/get-cached-llm-shape-template)
         payload (merge (type-flags type) context)]
     (if template
       (try
@@ -130,10 +157,21 @@
                    (and (#{:type :base-type} k) (keyword? v)) (clojure.core/name v)
                    :else                                      (str v))))}))
 
+(defn- fully-qualified-name
+  "Get fully qualified name for a table."
+  [database_schema name]
+  (if database_schema
+    (str database_schema "." name)
+    name))
+
 (defn field->xml
   "Format a field/column as XML element.
-   Matches Python Column.llm_representation exactly."
-  [{:keys [field_id name display_name type database_type description]}]
+   Matches Python Column.llm_representation exactly, plus — when the column is a foreign key
+   — a `fk_target_fully_qualified_name` attribute like `schema.table.field_name`. Without this
+   attribute the LLM sees a bare `customer_id` column with no hint that it joins anywhere,
+   and tries to read related fields (e.g. `email`) off the current table, which fails
+   with `:unknown-field`."
+  [{:keys [field_id name display_name type database_type description fk_target_portable_fk]}]
   (render-llm-template
    :field
    {:field_id            field_id
@@ -142,7 +180,13 @@
     :field_display_name  (or display_name name)
     :field_base_type     (when type (clojure.core/name type))
     :field_database_type (database-type-or-unknown database_type)
-    :field_description   description}))
+    :field_description   description
+    :field_fk_target_fqn (when (and (vector? fk_target_portable_fk)
+                                    (>= (count fk_target_portable_fk) 4))
+                           (let [[_db schema table & field-path] fk_target_portable_fk]
+                             (str (fully-qualified-name schema table)
+                                  "."
+                                  (str/join "." field-path))))}))
 
 (defn collection->xml
   "Format collection for LLM consumption.
@@ -156,66 +200,78 @@
 
 (defn measure->xml
   "Format a measure for LLM consumption."
-  [{:keys [id name display-name description definition definition-description]}]
+  [{:keys [id name display-name description definition definition-description
+           portable-entity-id portable_entity_id]}]
   (render-llm-template
    :measure
-   {:measure_id           (str id)
-    :measure_name         (or name "")
-    :measure_display_name (or display-name name "")
-    :measure_description  description
-    :measure_definition   (when definition (pr-str definition))
+   {:measure_id                 (str id)
+    :measure_name               (or name "")
+    :measure_display_name       (or display-name name "")
+    :measure_description        description
+    :measure_portable_entity_id (or portable-entity-id portable_entity_id)
+    :measure_definition         (repr-data->llm-block definition)
     :measure_definition_description definition-description}))
 
 (defn segment->xml
   "Format a segment for LLM consumption."
-  [{:keys [id name display-name description definition definition-description]}]
+  [{:keys [id name display-name description definition definition-description
+           portable-entity-id portable_entity_id]}]
   (render-llm-template
    :segment
-   {:segment_id           (str id)
-    :segment_name         (or name "")
-    :segment_display_name (or display-name name "")
-    :segment_description  description
-    :segment_definition   (when definition (pr-str definition))
+   {:segment_id                 (str id)
+    :segment_name               (or name "")
+    :segment_display_name       (or display-name name "")
+    :segment_description        description
+    :segment_portable_entity_id (or portable-entity-id portable_entity_id)
+    :segment_definition         (repr-data->llm-block definition)
     :segment_definition_description definition-description}))
 
 (defn- related-table->xml
   "Format a related table for LLM consumption."
-  [{:keys [id name related_by fully_qualified_name fields]}]
+  [{:keys [id name related_by database_name fully_qualified_name fields]}]
   (render-llm-template
    :related_table
-   {:related_table_id         (when (some? id) (str id))
-    :related_table_name       name
-    :related_table_related_by related_by
-    :related_table_fqn        fully_qualified_name
-    :related_table_fields_xml (when (seq fields) (str/join "\n" (map field->xml fields)))}))
+   {:related_table_id            (when (some? id) (str id))
+    :related_table_name          name
+    :related_table_related_by    related_by
+    :related_table_database_name database_name
+    :related_table_fqn           fully_qualified_name
+    :related_table_fields_xml    (when (seq fields) (str/join "\n" (map field->xml fields)))}))
 
 (defn metric->xml
   "Format metric for LLM consumption.
-   Matches Python Metric.get_llm_representation exactly."
+   Matches Python Metric.get_llm_representation exactly, except we additionally surface
+   `database_name`, `base_table_fully_qualified_name`, and `portable_entity_id` as tag
+   attributes — the three pieces of information the LLM needs to correctly use the metric
+   in `construct_notebook_query` (as `aggregation: [[metric, {}, <eid>]]` on top of the
+   metric's base table)."
   [{:keys [id name description verified queryable-dimensions collection
-           default_time_dimension_field]}]
+           default_time_dimension_field database_name base_table_portable_fk
+           portable_entity_id]}]
   (render-llm-template
    :metric
    {:metric_id                     (str id)
     :metric_name                   name
     :metric_verified               (boolean verified)
     :metric_description            description
+    :metric_database_name          database_name
+    :metric_base_table_fqn         (when (and (vector? base_table_portable_fk)
+                                              (= 3 (count base_table_portable_fk)))
+                                     (let [[_db schema table] base_table_portable_fk]
+                                       (fully-qualified-name schema table)))
+    :metric_portable_entity_id     portable_entity_id
     :metric_collection_xml         (when collection (collection->xml collection))
     :metric_default_time_dimension (:name default_time_dimension_field)
     :metric_dimensions_table       (when (seq queryable-dimensions)
                                      (format-fields-table queryable-dimensions))}))
 
-(defn- fully-qualified-name
-  "Get fully qualified name for a table."
-  [database_schema name]
-  (if database_schema
-    (str database_schema "." name)
-    name))
-
 (defn table->xml
   "Format table for LLM consumption.
-   Matches Python Table.get_llm_representation exactly."
-  [{:keys [id name database_id database_engine database_schema
+   Matches Python Table.get_llm_representation exactly, except we additionally surface
+   `database_name` as a tag attribute so the LLM has the human-readable DB name available
+   without a separate `entity_details` call — it's the first slot of every portable FK in
+   the representations-format `construct_notebook_query` tool."
+  [{:keys [id name database_id database_name database_engine database_schema
            description fields related_tables measures segments]}]
   (let [fqn (fully-qualified-name database_schema name)]
     (render-llm-template
@@ -223,6 +279,7 @@
      {:table_id                 (str id)
       :table_name               name
       :table_database_id        (str database_id)
+      :table_database_name      database_name
       :table_database_engine    (database-engine-or-unknown database_engine)
       :table_fqn                fqn
       :table_description        description
@@ -248,10 +305,16 @@
 
 (defn model->xml
   "Format model for LLM consumption.
-   Matches Python Model.get_llm_representation exactly.
-   Note: Python uses <metabase-model> tag but closes with </model>."
-  [{:keys [id name description verified fields database_id database_engine
-           related_tables measures segments]}]
+   Matches Python Model.get_llm_representation exactly, except we additionally surface
+   `database_name` as a tag attribute (see [[table->xml]] for the rationale).
+
+   When the caller supplies `:query_json` (the model's `dataset_query` exported through the
+   representations resolver as a portable repr-form map), it is rendered inside the
+   `<metabase-model>` element as a JSON `<query>` block — same canonical portable MBQL 5
+   representations form used elsewhere. Note: Python uses <metabase-model> tag but closes
+   with </model>."
+  [{:keys [id name description verified fields database_id database_name database_engine
+           related_tables measures segments portable_entity_id query_json]}]
   (let [fqn (model-fully-qualified-name id name)]
     (render-llm-template
      :model
@@ -259,9 +322,12 @@
       :model_name               name
       :model_verified           (boolean verified)
       :model_database_id        (str database_id)
+      :model_database_name      database_name
       :model_database_engine    (database-engine-or-unknown database_engine)
       :model_fqn                fqn
       :model_description        description
+      :model_portable_entity_id portable_entity_id
+      :model_query_json         (repr-data->llm-block query_json)
       :model_fields_xml         (when (seq fields)
                                   (str/join "\n" (map field->xml fields)))
       :model_related_tables_xml (when (seq related_tables)
@@ -347,8 +413,14 @@
     :chart_type (if chart-type (clojure.core/name chart-type) "table")}))
 
 (defn question->xml
-  "Format question for LLM consumption."
-  [{:keys [id name description verified collection visualization display fields]}]
+  "Format question for LLM consumption.
+
+  When the caller supplies `:query_json` (the saved card's `dataset_query` exported through
+  the representations resolver as a portable repr-form map), it is rendered inside the
+  `<metabase_question>` element as a JSON `<query>` block. The format is the same canonical
+  portable MBQL 5 representations the LLM writes into `construct_notebook_query` and gets
+  back in that tool's result — so existing cards and freshly-built queries share one form."
+  [{:keys [id name description verified collection visualization display fields portable_entity_id query_json]}]
   (render-llm-template
    :question
    {:question_id (str id)
@@ -356,8 +428,10 @@
     :question_name name
     :question_description description
     :question_display_type (some-> display clojure.core/name)
+    :question_portable_entity_id portable_entity_id
     :question_collection_xml (when collection (collection->xml collection))
     :question_visualization_xml (when visualization (visualization->xml visualization))
+    :question_query_json (repr-data->llm-block query_json)
     :question_fields_xml (when (seq fields)
                            (str/join "\n" (map field->xml fields)))}))
 
@@ -484,9 +558,20 @@
 (defn search-result->xml
   "Format a single search result as XML element.
    Includes database_id, database_engine, and fully_qualified_name for table/model results
-   to match Python AI Service search output."
+   to match Python AI Service search output. For saved-question and model results also
+   includes `portable_entity_id` so the LLM can paste it verbatim into `source-card:`
+   without an extra `entity_details` call, and (when available) `database_name` — the
+   human-readable name the LLM needs as the first slot of every portable FK in
+   `construct_notebook_query`.
+
+   For metric results we additionally surface `base_table_fully_qualified_name` (the
+   `schema.table` of the table the metric aggregates). Combined with `database_name` this
+   gives the LLM the full portable FK `[database_name, schema, table]` it must put in
+   `source-table:` when using `[metric, {}, <portable_entity_id>]` as an aggregation —
+   without a separate `entity_details` round-trip."
   [{:keys [id type name description verified collection
-           database_id database_engine database_schema]}]
+           database_id database_name database_engine database_schema portable_entity_id
+           base_table_portable_fk]}]
   (let [fqn (cond
               (#{"table" :table} type)
               (when name (fully-qualified-name database_schema name))
@@ -510,8 +595,14 @@
       :search_description description
       :search_collection_name (:name collection)
       :search_database_id (when database_id (str database_id))
+      :search_database_name database_name
       :search_database_engine engine
-      :search_fqn fqn})))
+      :search_fqn fqn
+      :search_portable_entity_id portable_entity_id
+      :search_base_table_fqn (when (and (vector? base_table_portable_fk)
+                                        (= 3 (count base_table_portable_fk)))
+                               (let [[_db schema table] base_table_portable_fk]
+                                 (fully-qualified-name schema table)))})))
 
 (defn search-results->xml
   "Format search results as XML wrapped in search-results element."

src/metabase/metabot/tools/slackbot_query.clj

@@ -1,7 +1,11 @@
 (ns metabase.metabot.tools.slackbot-query
   "Slackbot-specific notebook query tool.
   Like construct-notebook-query-tool but emits adhoc_viz data parts
-  instead of creating charts. Does not save or navigate."
+  instead of creating charts. Does not save or navigate.
+
+  This tool consumes the same MBQL 5 portable representations format (JSON, validated against
+  `::lib.schema/external-query`) as the main `construct_notebook_query` tool and shares its
+  prompt."
   (:require
    [metabase.metabot.agent.streaming :as streaming]
    [metabase.metabot.scope :as scope]
@@ -12,11 +16,15 @@
 (set! *warn-on-reflection* true)
 
 (def ^:private slackbot-query-schema
+  "Slackbot variant of `construct_notebook_query`. Same self-describing JSON query as the main
+  tool (`:query` is a `::lib.schema/external-query` map), plus `:title` and `:display` for the
+  Slack visualization wrapper.
+
+  The query body carries the database identity via its first stage's `source-table:` /
+  `source-card:`, so the schema deliberately omits `:source_entity` and `:referenced_entities`."
   [:map {:closed true}
    [:reasoning :string]
-   [:source_entity [:map [:type :string] [:id :int]]]
-   [:referenced_entities {:optional true} [:maybe [:sequential [:map [:type :string] [:id :int]]]]]
-   [:program construct/construct-program-schema]
+   [:query :map]
    [:title {:optional true} [:maybe :string]]
    [:display {:optional true
               :description "Visualization type for displaying the query results in Slack. Required in practice whenever the user asks for a chart or graph, and it must match any requested chart type. Valid values: 'table', 'bar', 'line', 'pie', 'area', 'row', 'scatter', 'funnel'. Use requested chart types like 'line', 'bar', 'area', 'pie', 'scatter', 'funnel', 'row', or 'table' when they fit the query. Omitting this field falls back to Metabase's default table display, so do not omit it for chart or graph requests. Only omit it when you intentionally want a plain table and the user did not request a chart type."}
@@ -25,10 +33,15 @@
 (mu/defn ^{:tool-name "construct_notebook_query"
            :scope     scope/agent-notebook-create}
   slackbot-construct-notebook-query-tool
-  "Construct a notebook query from a metric, model, or table. The query results will be rendered as a visualization in Slack."
-  [{:keys [_reasoning source_entity referenced_entities program title display]} :- slackbot-query-schema]
+  "Construct a notebook query from a metric, model, or table. The query results will be
+  rendered as a visualization in Slack.
+
+  See `resources/metabot/prompts/tools/construct_notebook_query.md` for the JSON format the
+  `:query` argument must follow — the prompt is shared with the main `construct_notebook_query`
+  tool."
+  [{:keys [_reasoning query title display]} :- slackbot-query-schema]
   (try
-    (let [query-result (construct/execute-program source_entity referenced_entities program)
+    (let [query-result (construct/execute-representations-query query)
           structured   (or (:structured-output query-result) (:structured_output query-result))]
       (if (and structured (:query-id structured) (:query structured))
         (let [metabase-link (streaming/query->question-url (:query structured) display)

src/metabase/metabot/tools/sql.clj

@@ -6,7 +6,7 @@
    [metabase.metabot.tmpl :as te]
    [metabase.metabot.tools.shared :as shared]
    [metabase.metabot.tools.shared.instructions :as instructions]
-   [metabase.metabot.tools.shared.llm-representations :as llm-rep]
+   [metabase.metabot.tools.shared.llm-shape :as llm-shape]
    [metabase.metabot.tools.sql.create :as create-sql-query-tools]
    [metabase.metabot.tools.sql.edit :as edit-sql-query-tools]
    [metabase.metabot.tools.sql.replace :as replace-sql-query-tools]
@@ -40,7 +40,7 @@
    `preamble` is optional text placed before the query XML inside <result>
    (e.g. \"SQL query successfully constructed.\")."
   [structured instruction-text & {:keys [preamble?]}]
-  (let [query-xml (llm-rep/query->xml structured)]
+  (let [query-xml (llm-shape/query->xml structured)]
     (te/lines
      "<result>"
      (when preamble?

src/metabase/metabot/tools/util.clj

@@ -44,10 +44,42 @@
                                  (lib/display-name query)
                                  lib/display-name-without-id))))
 
+(defn- column-portable-fk
+  "Build the portable FK path `[db-name, schema-or-null, table-name, field-name …]` for a column
+  that has a real numeric `:id`. Returns nil for expression / aggregation columns that don't
+  correspond to a database field."
+  [query column]
+  (when (and (:id column) (:table-id column))
+    (let [table (lib.metadata/table query (:table-id column))
+          db    (lib.metadata/database query)
+          ;; walk the `:parent-id` chain in case this is a JSON-unfolded nested field
+          chain (loop [acc [(:name column)]
+                       pid (:parent-id column)]
+                  (if pid
+                    (let [parent (lib.metadata/field query pid)]
+                      (recur (cons (:name parent) acc) (:parent-id parent)))
+                    acc))]
+      (when (and (:name db) (:name table))
+        (into [(:name db) (:schema table) (:name table)] chain)))))
+
+(defn- fk-target-portable-fk
+  "If `column` is an FK (has a non-nil `:fk-target-field-id`), return the portable FK path of
+  the target field, walking the parent-id chain for JSON-unfolded targets. Returns nil for
+  non-FK columns or when the target can't be resolved."
+  [query column]
+  (when-let [tgt-id (:fk-target-field-id column)]
+    (when-let [tgt (lib.metadata/field query tgt-id)]
+      (column-portable-fk query tgt))))
+
 (defn ->result-column
   "Return tool result column for `column` of `query`.
   Uses the real field `:id` when available, falling back to column name for
-  expression/aggregation columns that don't have a database field ID."
+  expression/aggregation columns that don't have a database field ID.
+
+  Also includes a `:portable_fk` key with the portable foreign-key path
+  `[db-name, schema, table-name, field-name …]` when the column maps to a real database field.
+  The representations-format notebook query tool expects fields to be referenced by this path
+  rather than by numeric id."
   [query column]
   (let [base-type         (some-> (:base-type column) u/qualified-name)
         effective-type    (some-> (:effective-type column) u/qualified-name)
@@ -55,7 +87,11 @@
         coercion-strategy (some-> (:coercion-strategy column) u/qualified-name)
         field-id          (or (:id column)
                               (:lib/desired-column-alias column)
-                              (:lib/source-column-alias column))]
+                              (:lib/source-column-alias column))
+        portable-fk       (try (column-portable-fk query column)
+                               (catch Exception _ nil))
+        fk-target-fk      (try (fk-target-portable-fk query column)
+                               (catch Exception _ nil))]
     (-> {:field_id field-id
          :name (or (:lib/desired-column-alias column)
                    (:lib/source-column-alias column))
@@ -68,6 +104,8 @@
                       :database_type (:database-type column)
                       :coercion_strategy coercion-strategy
                       :field_values (:field-values column)
+                      :portable_fk portable-fk
+                      :fk_target_portable_fk fk-target-fk
                       :table_reference (:table-reference column)))))
 
 (defn find-column-by-field-id
@@ -120,6 +158,19 @@
   (-> (t2/select-one :model/Card :id id)
       api/read-check))
 
+(defn get-card-by-entity-id
+  "Retrieve a readable card by its 21-char NanoID `entity_id`, or `nil` if not found. Used by
+  [[metabase.metabot.tools.construct/resolve-database-id-from-first-stage]] to look up the
+  database-id of a `source-card:` stage before the metadata provider is built.
+
+  Does NOT go through the serdes `lookup-by-id` machinery because we want `nil` on miss
+  (not a thrown exception) so the caller can surface a tool-specific agent error. If the
+  card exists but the current user cannot read it, `api/read-check` raises a 403 instead of
+  silently letting the representations resolver use an inaccessible card."
+  [entity-id]
+  (some-> (t2/select-one :model/Card :entity_id entity-id)
+          api/read-check))
+
 (defn card-query
   "Return a query based on the card with ID `card-id`."
   [card-id]

src/metabase/models/serialization/resolve.clj

@@ -237,7 +237,7 @@
      (mbql-id->fully-qualified-name resolver &match)
 
      (_ :guard sequential?)
-     (mapv export-mbql &match)
+     (mapv (partial export-mbql resolver) &match)
 
      (_ :guard map?)
      (reduce-kv

src/metabase/models/serialization/resolve/mp.clj

@@ -0,0 +1,626 @@
+(ns metabase.models.serialization.resolve.mp
+  "Metadata-provider-backed implementations of the serdes resolver protocols.
+
+  Unlike `metabase.models.serialization.resolve.db`, this resolver does not touch toucan2 /
+  the application database for *warehouse-schema* lookups (tables, fields) - it works off a
+  `lib.metadata/MetadataProvider`, which may be the live application-DB-backed provider, a
+  test-only mock provider, or any cached variant.
+
+  It does, however, touch the application DB for *Metabase-model* lookups (cards by
+  `entity_id`, etc.) because the lib metadata protocol doesn't support filtering by
+  `entity_id`, and cards in any case live in the application DB independently of whose
+  warehouse the metadata provider points at.
+
+  Primary consumer: the agent-lib representations pipeline, which converts LLM-authored
+  portable MBQL queries (with FK paths like `[DB, SCHEMA, TABLE, FIELD]`) into numeric-ID
+  MBQL suitable for `lib.query/query` / the QP.
+
+  Implemented scope for the representations pipeline:
+    * `import-table-fk`, `import-field-fk`, `export-table-fk`, `export-field-fk` for
+      warehouse metadata.
+    * `import-fk-keyed` / `export-fk-keyed` for `:model/Database` by `:name`.
+    * `import-fk` for `Card` / `:model/Card` by `entity_id` (source-card and metric refs).
+    * `export-fk` for `Card` / `:model/Card` by `entity_id` (exporting final pMBQL back to
+      portable representations YAML).
+
+  Everything else throws `:not-implemented-yet` for now.
+
+  Asset vs metadata split:
+    The resolver has two orthogonal lookup responsibilities:
+      * **Warehouse metadata** (databases, tables, fields) is resolved through a
+        `lib.metadata/MetadataProvider`.
+      * **Metabase content / assets** (cards, snippets, segments, …) is resolved through a
+        [[ContentStore]] on import, where callers may need permission-aware lookups. Exporting
+        card ids uses the same database-scoped metadata provider, which already contains card
+        metadata in app-backed and mock-provider contexts. The default app-DB-backed store goes
+        through `serdes/lookup-by-id`, but a different store (e.g. backed by the checker's YAML
+        index, an in-memory test fixture, or a snapshot) can be supplied to make import usable
+        without an application database."
+  (:require
+   [metabase.lib.metadata :as lib.metadata]
+   [metabase.lib.metadata.protocols :as lib.metadata.protocols]
+   [metabase.models.serialization :as serdes]
+   [metabase.models.serialization.resolve :as resolve]
+   [metabase.util.i18n :refer [tru]]
+   [potemkin.types :as p.types]
+   [toucan2.core :as t2]))
+
+(set! *warn-on-reflection* true)
+
+;;; ============================================================
+;;; Helpers
+;;; ============================================================
+
+(defn- db-name [metadata-provider]
+  (:name (lib.metadata/database metadata-provider)))
+
+(defn- matching-tables
+  "Return all tables in `metadata-provider` that match `table-name`, optionally filtered by `schema`
+  (where `schema` may be nil for schemaless databases).
+
+  Uses `metadatas-by-name`-style query via `metadatas` to avoid O(tables) scans on large providers."
+  [metadata-provider schema table-name]
+  (->> (lib.metadata.protocols/metadatas metadata-provider
+                                         {:lib/type :metadata/table
+                                          :name     #{table-name}})
+       (filter #(= (:schema %) schema))))
+
+(defn- unknown-table-ex-info
+  "Build an agent-facing `:unknown-table` error for a miss on portable FK
+  `[db schema name]`.
+
+  Deliberately does NOT enumerate schemas, sibling tables, or fuzzy candidates. The metadata
+  provider here is un-sandboxed (it exposes every table in the warehouse regardless of the
+  caller's data perms), and any string included in this ex-info message is relayed verbatim
+  to the LLM via `:agent-error? true`, and from there to the end user. A sandboxed user who
+  prompts the agent with a hallucinated `source-table:` would otherwise receive a list of
+  schemas / tables they cannot otherwise see.
+
+  The LLM can still self-correct in one turn by calling `entity_details` on the parent
+  database; the message points it at that path."
+  [_metadata-provider [_path-db-name _path-schema _path-table-name :as path]]
+  (ex-info (tru "No table found matching portable FK {0}. Call `entity_details` with entity-type `database` and the database''s numeric id to list available tables and schemas, then retry with an exact portable FK from the response."
+                (pr-str path))
+           {:status-code  400
+            :error        :unknown-table
+            :agent-error? true
+            :path         path}))
+
+(defn- find-table
+  "Resolve `[db-name, schema, table-name]` to a `:metadata/table` or throw with context.
+
+  On miss, the thrown ex-info carries actionable, tiered hints (see
+  [[unknown-table-ex-info]]) so the LLM can self-correct on the next turn instead of
+  re-hallucinating the same bad path. All error branches are marked
+  `:agent-error? true` so the outer tool wrapper relays the message verbatim."
+  [metadata-provider [path-db-name path-schema path-table-name :as path]]
+  (let [current-db (db-name metadata-provider)]
+    (when-not (= path-db-name current-db)
+      (throw (ex-info (tru "Portable table FK references database {0}, but metadata provider is for {1}."
+                           (pr-str path-db-name)
+                           (pr-str current-db))
+                      {:status-code  400
+                       :error        :unknown-table
+                       :agent-error? true
+                       :path         path
+                       :expected-db  current-db}))))
+  (let [candidates (matching-tables metadata-provider path-schema path-table-name)]
+    (case (count candidates)
+      0 (throw (unknown-table-ex-info metadata-provider path))
+      1 (first candidates)
+      ;; Deliberately do NOT enumerate the matching `[schema name id]` tuples — the metadata
+      ;; provider is un-sandboxed, so a leaked candidate list could surface tables the caller
+      ;; cannot otherwise see (parity with the `unknown-table-ex-info` strip above).
+      (throw (ex-info (tru "Ambiguous portable table FK {0}: {1} candidates. Call `entity_details` with entity-type `database` to list available tables and retry with a more specific portable FK."
+                           (pr-str path) (count candidates))
+                      {:status-code  400
+                       :error        :ambiguous-table
+                       :agent-error? true
+                       :path         path})))))
+
+(defn- find-field
+  "Resolve a field path `[db schema table field …]` to a `:metadata/column` or throw with context.
+
+  For regular (non-JSON-unfolded) fields, the path has exactly 4 elements and we match on name
+  directly. For JSON-unfolded paths (5+ elements), we walk parent/child via `:parent-id` -
+  matching the behaviour of `serdes/recursively-find-field-q`."
+  [metadata-provider [db schema table-name & field-path :as full-path]]
+  (when (empty? field-path)
+    (throw (ex-info (tru "Portable field FK {0} must have at least 4 elements." (pr-str full-path))
+                    {:status-code 400
+                     :error       :invalid-field-fk
+                     :path        full-path})))
+  (let [table-meta (find-table metadata-provider [db schema table-name])
+        columns    (lib.metadata.protocols/metadatas-for-table metadata-provider
+                                                               :metadata/column
+                                                               (:id table-meta))
+        ;; Deliberately do NOT enumerate FK-linked tables that happen to carry the same
+        ;; column name. The metadata provider is un-sandboxed, the `:agent-error?` flag
+        ;; relays this message verbatim to the user, and a leaked FK-candidate path would
+        ;; reveal table names the caller may not be permitted to see. The LLM can recover
+        ;; by calling `entity_details` on the parent table to list its columns.
+        unknown-field-ex
+        (fn [segment]
+          (ex-info (tru "No column {0} on table {1}.{2}.{3}. Call `entity_details` on this table to list its columns."
+                        (pr-str segment) (pr-str db) (pr-str schema) (pr-str table-name))
+                   {:status-code  400
+                    :error        :unknown-field
+                    :path         full-path
+                    :segment      segment
+                    :table-id     (:id table-meta)
+                    :agent-error? true}))
+        ;; Walk nested fields parent-first. At each level, match `:name` and `:parent-id`.
+        walk       (fn walk [parent-id [segment & more]]
+                     (let [candidates (filter #(and (= (:name %) segment)
+                                                    (= (:parent-id %) parent-id))
+                                              columns)]
+                       (case (count candidates)
+                         0 (throw (unknown-field-ex segment))
+                         1 (let [col (first candidates)]
+                             (if (seq more)
+                               (walk (:id col) more)
+                               col))
+                         (throw (ex-info (tru "Ambiguous column {0} on table {1}.{2}.{3}."
+                                              (pr-str segment)
+                                              (pr-str db)
+                                              (pr-str schema)
+                                              (pr-str table-name))
+                                         {:status-code 400
+                                          :error       :ambiguous-field
+                                          :path        full-path
+                                          :segment     segment
+                                          :candidates  (mapv :id candidates)})))))]
+    (walk nil field-path)))
+
+(defn outbound-fks-from-table
+  "Return a sequence of FK edges leaving `source-table-id` in `metadata-provider`.
+
+  Each entry is a map with:
+    * `:source-field-id`           numeric id of the FK field on the source table;
+    * `:source-field`              the full column metadata (convenient for portable-FK export);
+    * `:target-field-id`           numeric id of the target field pointed to;
+    * `:target-table-id`           numeric id of the target table.
+
+  Uses `lib.metadata.protocols/metadatas-for-table` and `lib.metadata.protocols/field` directly
+  (not the `mu/defn`-wrapped accessors) so that missing target-field lookups degrade to `nil`
+  rather than throwing Malli errors."
+  [metadata-provider source-table-id]
+  (->> (lib.metadata.protocols/metadatas-for-table metadata-provider :metadata/column source-table-id)
+       (keep (fn [col]
+               (when-let [tgt-id (:fk-target-field-id col)]
+                 (when-let [tgt (lib.metadata.protocols/field metadata-provider tgt-id)]
+                   {:source-field-id  (:id col)
+                    :source-field     col
+                    :target-field-id  tgt-id
+                    :target-table-id  (:table-id tgt)}))))
+       vec))
+
+(defn- export-field-path
+  "Build the portable FK path for a field, including any JSON-unfolded parent chain."
+  [metadata-provider field]
+  (let [table (lib.metadata.protocols/table metadata-provider (:table-id field))
+        ;; Walk parent chain upward, then reverse to get root -> leaf order.
+        chain (loop [acc [(:name field)]
+                     pid (:parent-id field)]
+                (if pid
+                  (let [parent (lib.metadata.protocols/field metadata-provider pid)]
+                    (recur (cons (:name parent) acc) (:parent-id parent)))
+                  acc))]
+    (into [(db-name metadata-provider) (:schema table) (:name table)] chain)))
+
+;;; ============================================================
+;;; Content store - Metabase asset lookups (cards, etc.) by portable id
+;;; ============================================================
+
+(p.types/defprotocol+ ContentStore
+  "Lookup of Metabase content (\"assets\") by portable id.
+
+  Kept separate from the warehouse-metadata `MetadataProvider` so the resolver can be reused in
+  contexts without an application database (e.g. the serdes checker, in-memory tests)."
+  (card-by-entity-id [this entity-id]
+    "Return the card row for the given portable `entity_id`, or nil. The returned map must at
+    least carry `:id` and `:database_id` so the resolver can perform cross-database checks.")
+  (measure-by-entity-id [this entity-id]
+    "Return the measure row for the given portable `entity_id`, or nil. The returned map must
+    at least carry `:id` and `:table_id` so the resolver can perform cross-database checks via
+    the metadata provider.")
+  (segment-by-entity-id [this entity-id]
+    "Return the segment row for the given portable `entity_id`, or nil. Same contract as
+    `measure-by-entity-id`.")
+  (measure-by-id [this measure-id]
+    "Return the measure row for the given numeric id, or nil. Same contract as
+    `measure-by-entity-id`; used by the export direction (`[:measure {} <id>]` →
+    `[:measure {} \"<entity_id>\"]`). The returned map must carry `:entity_id` (or
+    `:entity-id`) and `:table_id` (or `:table-id`).")
+  (segment-by-id [this segment-id]
+    "Return the segment row for the given numeric id, or nil. Same contract as
+    `measure-by-id`."))
+
+(def unchecked-app-db-content-store
+  "**Unchecked.** No `api/read-check`; appropriate for serdes import / background tasks /
+  tests / dev REPL only. **Any HTTP or tool path that runs under an authenticated user must
+  wrap this with `metabase.metabot.tools.shared.content-store/read-checked`** (or use
+  `shared.content-store/default-store`, which is the wrapped form).
+
+  Default [[ContentStore]] backed by the Metabase application database via
+  `serdes/lookup-by-id`. Use this in production code paths that already have an app DB; pass a
+  different store implementation when running without one (checker, isolated tests).
+
+  Gated on [[resolve/entity-id?]]: `serdes/lookup-by-id` falls through to a full-table scan via
+  `find-by-identity-hash` for non-NanoID strings. LLM-authored entity-id values are untrusted,
+  so anything that isn't a 21-char NanoID short-circuits to `nil` and the caller surfaces a
+  clear `:unknown-…` agent error."
+  (reify ContentStore
+    (card-by-entity-id [_ entity-id]
+      (when (resolve/entity-id? entity-id)
+        (serdes/lookup-by-id 'Card entity-id)))
+    (measure-by-entity-id [_ entity-id]
+      (when (resolve/entity-id? entity-id)
+        (serdes/lookup-by-id 'Measure entity-id)))
+    (segment-by-entity-id [_ entity-id]
+      (when (resolve/entity-id? entity-id)
+        (serdes/lookup-by-id 'Segment entity-id)))
+    (measure-by-id [_ measure-id]
+      (when measure-id
+        (t2/select-one [:model/Measure :id :entity_id :table_id] :id measure-id)))
+    (segment-by-id [_ segment-id]
+      (when segment-id
+        (t2/select-one [:model/Segment :id :entity_id :table_id] :id segment-id)))))
+
+;;; ============================================================
+;;; Resolver implementations
+;;; ============================================================
+
+(defn- not-implemented! [method]
+  (throw (ex-info (tru "{0} is not implemented for the metadata-provider-backed resolver in Phase 1."
+                       (name method))
+                  {:status-code 501
+                   :error       :not-implemented-yet
+                   :method      method})))
+
+(defn- import-database-by-name
+  "Resolve a database by its portable name (a string) to a numeric id."
+  [metadata-provider db-name-value]
+  (let [current-name (db-name metadata-provider)
+        current-id   (:id (lib.metadata/database metadata-provider))]
+    (if (= db-name-value current-name)
+      current-id
+      (throw (ex-info (tru "Portable database reference {0} does not match metadata provider database {1}."
+                           (pr-str db-name-value) (pr-str current-name))
+                      {:status-code 400
+                       :error       :unknown-database
+                       :database    db-name-value
+                       :expected-db current-name})))))
+
+(defn- export-database-name
+  "Resolve the current database's numeric id back to its portable name."
+  [metadata-provider database-id]
+  (when database-id
+    (let [current-db (lib.metadata/database metadata-provider)]
+      (if (= database-id (:id current-db))
+        (:name current-db)
+        (throw (ex-info (tru "Cannot export database id {0}: metadata provider is for database id {1}."
+                             database-id (:id current-db))
+                        {:status-code 400
+                         :error       :unknown-database-id
+                         :database-id database-id
+                         :expected-id (:id current-db)}))))))
+
+(defn- card-model? [model]
+  (or (= model 'Card) (= model :model/Card)))
+
+(defn- measure-model? [model]
+  (or (= model 'Measure) (= model :model/Measure)))
+
+(defn- segment-model? [model]
+  (or (= model 'Segment) (= model :model/Segment)))
+
+(defn- table-belongs-to-current-database?
+  "True when the table with `table-id` is known to the metadata provider — measures and segments
+  are scoped to a table, so this is the cross-database guard for measure/segment refs. Returns
+  the table when found so callers can pull its `:db-id` for error messages."
+  [metadata-provider table-id]
+  (when table-id
+    (lib.metadata.protocols/table metadata-provider table-id)))
+
+(defn- export-card-by-id
+  "Resolve a saved question / model / metric by numeric id to its portable `entity_id`.
+
+  Uses the metadata provider rather than the generic app-DB serdes resolver so exporting a
+  final pMBQL query can stay paired with the same database-scoped provider used for table and
+  field FK export. Guards against accidental cross-database card refs."
+  [metadata-provider card-id]
+  (when card-id
+    (let [current-db-id (:id (lib.metadata/database metadata-provider))
+          card          (lib.metadata.protocols/card metadata-provider card-id)
+          card-db-id    (or (:database-id card) (:database_id card))
+          entity-id     (or (:entity-id card) (:entity_id card))]
+      (cond
+        (nil? card)
+        (throw (ex-info (tru "No saved question, model, or metric found with id {0} in metadata provider." card-id)
+                        {:status-code 400
+                         :error       :unknown-card-id
+                         :card-id     card-id}))
+
+        (not= card-db-id current-db-id)
+        (throw (ex-info (tru "Saved question / model / metric id {0} belongs to database {1}, but this resolver targets database {2}."
+                             card-id card-db-id current-db-id)
+                        {:status-code      400
+                         :error            :cross-database-card
+                         :card-id          card-id
+                         :card-database-id card-db-id
+                         :expected-database current-db-id}))
+
+        (not (and (string? entity-id) (seq entity-id)))
+        (throw (ex-info (tru "Saved question, model, or metric id {0} does not have an entity_id, so it cannot be exported as a portable representation." card-id)
+                        {:status-code 400
+                         :error       :missing-card-entity-id
+                         :card-id     card-id}))
+
+        :else
+        entity-id))))
+
+(defn- import-card-by-entity-id
+  "Resolve a saved question / model by its portable `entity_id` to its numeric id.
+
+  Guards against the card belonging to a different database than the metadata provider -
+  that would produce a broken cross-database query at resolve time, so we surface a clear
+  agent-facing error here instead."
+  [metadata-provider content-store entity-id]
+  (let [current-db-id (:id (lib.metadata/database metadata-provider))
+        card          (card-by-entity-id content-store entity-id)
+        ;; Card may come from a Toucan2 row (snake_case) or a `lib.metadata` map (kebab-case);
+        ;; mirror the defensive read in `export-card-by-id` so the test-side content stores can
+        ;; return either shape without breaking the cross-database guard.
+        card-db-id    (when card (or (:database-id card) (:database_id card)))]
+    (cond
+      (nil? card)
+      (throw (ex-info (tru "No saved question or model found with entity_id {0}. Do not invent or guess entity_ids: call `entity_details` with `entity-type: question` or `entity-type: model` and the card''s numeric id first, then copy the exact `portable_entity_id` from the response into `source-card:`."
+                           (pr-str entity-id))
+                      {:agent-error? true
+                       :status-code  400
+                       :error        :unknown-card
+                       :entity-id    entity-id}))
+
+      (not= card-db-id current-db-id)
+      (throw (ex-info (tru "Saved question / model {0} belongs to database {1}, but this query targets database {2}. Cross-database queries are not supported."
+                           (pr-str entity-id)
+                           (pr-str card-db-id)
+                           (pr-str current-db-id))
+                      {:agent-error?     true
+                       :status-code      400
+                       :error            :cross-database-card
+                       :entity-id        entity-id
+                       :card-database-id card-db-id
+                       :expected-database current-db-id}))
+
+      :else
+      (:id card))))
+
+(defn- import-measure-by-entity-id
+  "Resolve a measure by its portable `entity_id` to its numeric id.
+
+  Measures are scoped to a table, so the cross-database guard works through the metadata
+  provider: a measure whose `:table_id` is not known to the provider belongs to a different
+  database. Same pattern as [[import-card-by-entity-id]]."
+  [metadata-provider content-store entity-id]
+  (let [measure         (measure-by-entity-id content-store entity-id)
+        measure-table-id (when measure (or (:table-id measure) (:table_id measure)))
+        measure-table    (when measure-table-id
+                           (table-belongs-to-current-database? metadata-provider measure-table-id))
+        current-db-id    (:id (lib.metadata/database metadata-provider))]
+    (cond
+      (nil? measure)
+      (throw (ex-info (tru "No measure found with entity_id {0}. Do not invent or guess entity_ids: call `entity_details` on the table that owns the measure and copy the exact `portable_entity_id` from the `<measure>` tag."
+                           (pr-str entity-id))
+                      {:agent-error? true
+                       :status-code  400
+                       :error        :unknown-measure
+                       :entity-id    entity-id}))
+
+      (nil? measure-table)
+      (throw (ex-info (tru "Measure {0} belongs to a table in a different database than this query (target database id {1}). Cross-database queries are not supported."
+                           (pr-str entity-id) (pr-str current-db-id))
+                      {:agent-error?      true
+                       :status-code       400
+                       :error             :cross-database-measure
+                       :entity-id         entity-id
+                       :measure-table-id  measure-table-id
+                       :expected-database current-db-id}))
+
+      :else
+      (:id measure))))
+
+(defn- export-measure-by-id
+  "Resolve a measure by numeric id to its portable `entity_id`. Validates that the measure's
+  table belongs to the metadata provider's database (cross-database guard)."
+  [metadata-provider content-store measure-id]
+  (when measure-id
+    (let [measure          (measure-by-id content-store measure-id)
+          measure-table-id (when measure (or (:table-id measure) (:table_id measure)))
+          measure-table    (when measure-table-id
+                             (table-belongs-to-current-database? metadata-provider measure-table-id))
+          entity-id        (when measure (or (:entity-id measure) (:entity_id measure)))]
+      (cond
+        (nil? measure)
+        (throw (ex-info (tru "No measure found with id {0}." measure-id)
+                        {:status-code 400
+                         :error       :unknown-measure-id
+                         :measure-id  measure-id}))
+
+        (nil? measure-table)
+        (throw (ex-info (tru "Measure id {0} belongs to a table outside this metadata provider''s database."
+                             measure-id)
+                        {:status-code 400
+                         :error       :cross-database-measure
+                         :measure-id  measure-id}))
+
+        (not (and (string? entity-id) (seq entity-id)))
+        (throw (ex-info (tru "Measure id {0} does not have an entity_id, so it cannot be exported as a portable representation."
+                             measure-id)
+                        {:status-code 400
+                         :error       :missing-measure-entity-id
+                         :measure-id  measure-id}))
+
+        :else
+        entity-id))))
+
+(defn- export-segment-by-id
+  "Resolve a segment by numeric id to its portable `entity_id`. Same pattern as
+  [[export-measure-by-id]]."
+  [metadata-provider content-store segment-id]
+  (when segment-id
+    (let [segment          (segment-by-id content-store segment-id)
+          segment-table-id (when segment (or (:table-id segment) (:table_id segment)))
+          segment-table    (when segment-table-id
+                             (table-belongs-to-current-database? metadata-provider segment-table-id))
+          entity-id        (when segment (or (:entity-id segment) (:entity_id segment)))]
+      (cond
+        (nil? segment)
+        (throw (ex-info (tru "No segment found with id {0}." segment-id)
+                        {:status-code 400
+                         :error       :unknown-segment-id
+                         :segment-id  segment-id}))
+
+        (nil? segment-table)
+        (throw (ex-info (tru "Segment id {0} belongs to a table outside this metadata provider''s database."
+                             segment-id)
+                        {:status-code 400
+                         :error       :cross-database-segment
+                         :segment-id  segment-id}))
+
+        (not (and (string? entity-id) (seq entity-id)))
+        (throw (ex-info (tru "Segment id {0} does not have an entity_id, so it cannot be exported as a portable representation."
+                             segment-id)
+                        {:status-code 400
+                         :error       :missing-segment-entity-id
+                         :segment-id  segment-id}))
+
+        :else
+        entity-id))))
+
+(defn- import-segment-by-entity-id
+  "Resolve a segment by its portable `entity_id` to its numeric id. Same pattern as
+  [[import-measure-by-entity-id]]."
+  [metadata-provider content-store entity-id]
+  (let [segment          (segment-by-entity-id content-store entity-id)
+        segment-table-id (when segment (or (:table-id segment) (:table_id segment)))
+        segment-table    (when segment-table-id
+                           (table-belongs-to-current-database? metadata-provider segment-table-id))
+        current-db-id    (:id (lib.metadata/database metadata-provider))]
+    (cond
+      (nil? segment)
+      (throw (ex-info (tru "No segment found with entity_id {0}. Do not invent or guess entity_ids: call `entity_details` on the table that owns the segment and copy the exact `portable_entity_id` from the `<segment>` tag."
+                           (pr-str entity-id))
+                      {:agent-error? true
+                       :status-code  400
+                       :error        :unknown-segment
+                       :entity-id    entity-id}))
+
+      (nil? segment-table)
+      (throw (ex-info (tru "Segment {0} belongs to a table in a different database than this query (target database id {1}). Cross-database queries are not supported."
+                           (pr-str entity-id) (pr-str current-db-id))
+                      {:agent-error?      true
+                       :status-code       400
+                       :error             :cross-database-segment
+                       :entity-id         entity-id
+                       :segment-table-id  segment-table-id
+                       :expected-database current-db-id}))
+
+      :else
+      (:id segment))))
+
+(defn import-resolver
+  "Build a `SerdesImportResolver` backed by `metadata-provider` (warehouse metadata) and
+  `content-store` (Metabase content / assets).
+
+  The 1-arity form uses [[unchecked-app-db-content-store]] - i.e. it goes through
+  `serdes/lookup-by-id` for content lookups and therefore requires the application database.
+  **Note the \"unchecked\" prefix:** the default store does NOT apply `api/read-check`. HTTP /
+  tool paths that run under an authenticated user must pass an explicit content-store wrapped
+  with `metabase.metabot.tools.shared.content-store/read-checked`.
+
+  Implemented methods:
+    * `import-table-fk`, `import-field-fk` (Phase 1).
+    * `import-fk-keyed` for `:model/Database` by `:name` (Phase 1 - needed because
+      `resolve/import-mbql` dispatches on `:database` keys).
+    * `import-fk` for `Card` / `:model/Card` by `entity_id` (Phase 2, step 11).
+
+  Other methods throw `:not-implemented-yet`."
+  ([metadata-provider]
+   (import-resolver metadata-provider unchecked-app-db-content-store))
+  ([metadata-provider content-store]
+   (reify resolve/SerdesImportResolver
+     (import-fk       [_ eid model]
+       (cond
+         (nil? eid)             nil
+         (card-model? model)    (import-card-by-entity-id metadata-provider content-store eid)
+         (measure-model? model) (import-measure-by-entity-id metadata-provider content-store eid)
+         (segment-model? model) (import-segment-by-entity-id metadata-provider content-store eid)
+         :else                  (not-implemented! :import-fk)))
+     (import-fk-keyed [_ portable model field]
+       (cond
+         (and (or (= model :model/Database) (= model 'Database))
+              (= field :name))
+         (import-database-by-name metadata-provider portable)
+
+         :else
+         (not-implemented! :import-fk-keyed)))
+     (import-user     [_ _email]                  (not-implemented! :import-user))
+     (import-table-fk [_ path]
+       (when path
+         (:id (find-table metadata-provider path))))
+     (import-field-fk [_ path]
+       (when path
+         (:id (find-field metadata-provider path)))))))
+
+(defn export-resolver
+  "Build a `SerdesExportResolver` backed by `metadata-provider`.
+
+  Implemented methods:
+    * `export-table-fk`, `export-field-fk` for warehouse metadata.
+    * `export-fk-keyed` for `:model/Database` by `:name`.
+    * `export-fk` for `Card` / `:model/Card` by `entity_id` (source-card and metric refs).
+    * `export-fk` for `:model/Measure` and `:model/Segment` by `entity_id`.
+
+  Other methods throw `:not-implemented-yet`."
+  ([metadata-provider]
+   (export-resolver metadata-provider unchecked-app-db-content-store))
+  ([metadata-provider content-store]
+   (reify resolve/SerdesExportResolver
+     (export-fk       [_ id model]
+       (cond
+         (nil? id)              nil
+         (card-model? model)    (export-card-by-id metadata-provider id)
+         (measure-model? model) (export-measure-by-id metadata-provider content-store id)
+         (segment-model? model) (export-segment-by-id metadata-provider content-store id)
+         :else                  (not-implemented! :export-fk)))
+     (export-fk-keyed [_ id model field]
+       (cond
+         (nil? id) nil
+
+         (and (or (= model :model/Database) (= model 'Database))
+              (= field :name))
+         (export-database-name metadata-provider id)
+
+         :else
+         (not-implemented! :export-fk-keyed)))
+     (export-user     [_ _id]               (not-implemented! :export-user))
+     (export-table-fk [_ table-id]
+       (when table-id
+         (let [t (lib.metadata.protocols/table metadata-provider table-id)]
+           (when-not t
+             (throw (ex-info (tru "No table with id {0} in metadata provider." table-id)
+                             {:status-code 400
+                              :error       :unknown-table-id
+                              :table-id    table-id})))
+           [(db-name metadata-provider) (:schema t) (:name t)])))
+     (export-field-fk [_ field-id]
+       (when field-id
+         (let [f (lib.metadata.protocols/field metadata-provider field-id)]
+           (when-not f
+             (throw (ex-info (tru "No field with id {0} in metadata provider." field-id)
+                             {:status-code 400
+                              :error       :unknown-field-id
+                              :field-id    field-id})))
+           (export-field-path metadata-provider f)))))))

test/metabase/agent_api/api_test.clj

@@ -252,57 +252,177 @@
   [response]
   (-> response :query u/decode-base64 json/decode+kw lib.normalize/normalize))
 
+;;; ---------------------------------------- Repr JSON helpers ----------------------------------------
+
+(defn- db-name
+  "Canonical name of the application database \u2014 the literal string the LLM is expected
+  to put as the first element of every portable FK in a representations query."
+  []
+  (t2/select-one-fn :name :model/Database (mt/id)))
+
+(defn- orders-query
+  "Build a portable MBQL 5 representations external-query map against the `ORDERS` table in
+  the application database. Clause arguments (`aggregation`, `breakout`, `filters`, `order-by`,
+  `fields`) are Clojure data fed straight through to the HTTP body, e.g.
+
+      (orders-query :limit 10)
+      (orders-query :aggregation [[\"count\" {}]])
+      (orders-query :filters     [[\"not-null\" {} (orders-field-ref \"ID\")]])
+
+  Structural validation against `::lib.schema/external-query` lives on the server side; here
+  we just need a believable payload to drive the HTTP endpoint."
+  [& {:keys [limit aggregation breakout filters order-by fields]}]
+  {:lib/type "mbql/query"
+   :stages   [(cond-> {:lib/type     "mbql.stage/mbql"
+                       :source-table [(db-name) "PUBLIC" "ORDERS"]}
+                aggregation (assoc :aggregation (vec aggregation))
+                breakout    (assoc :breakout    (vec breakout))
+                filters     (assoc :filters     (vec filters))
+                order-by    (assoc :order-by    (vec order-by))
+                fields      (assoc :fields      (vec fields))
+                limit       (assoc :limit       limit))]})
+
+(defn- orders-field-ref
+  "Field-ref clause against an ORDERS column as Clojure data, e.g.
+  `[\"field\" {} [\"test-data (h2)\" \"PUBLIC\" \"ORDERS\" \"ID\"]]`. Suitable as an inner
+  argument inside `aggregation` / `filters` / `order-by` clauses passed to [[orders-query]]."
+  [col-name]
+  ["field" {} [(db-name) "PUBLIC" "ORDERS" col-name]])
+
+(defn- source-card-query
+  "Build a minimal portable representations external-query map whose first stage uses
+  `source-card:`."
+  [entity-id]
+  {:lib/type "mbql/query"
+   :stages   [{:lib/type    "mbql.stage/mbql"
+               :source-card entity-id
+               :limit       5}]})
+
+;;; ---------------------------------------- /v2/construct-query ----------------------------------------
+
 (deftest construct-query-test
   (testing "Constructs a simple query from a table"
-    (let [table-id (mt/id :orders)
-          response (mt/user-http-request :rasta :post 200 "agent/v2/construct-query"
-                                         {:source     {:type "table" :id table-id}
-                                          :operations []
-                                          :prompt     "show orders"})]
+    (let [response (mt/user-http-request :rasta :post 200 "agent/v2/construct-query"
+                                         {:query (orders-query)})]
       (is (string? (:query response)) "Response should contain a query string")
-      (is (= "show orders" (:prompt response)) "Response should echo the prompt")
       (let [decoded (decode-query response)]
         (is (= :mbql/query (lib/normalized-query-type decoded)))
         (is (= (mt/id) (lib/database-id decoded)))
         (is (= (mt/id :orders) (lib/primary-source-table-id decoded))))))
 
-  (testing "Does not require prompt"
-    (let [table-id (mt/id :orders)
-          response (mt/user-http-request :rasta :post 200 "agent/v2/construct-query"
-                                         {:source     {:type "table" :id table-id}
-                                          :operations []})]
-      (is (string? (:query response)) "Response should contain a query string")
-      (is (not (contains? response :prompt)) "Response should only echo prompt when supplied")))
-
-  (testing "Rejects oversized prompts before they can be persisted on MCP query handles"
-    (mt/user-http-request :rasta :post 400 "agent/v2/construct-query"
-                          {:source     {:type "table" :id (mt/id :orders)}
-                           :operations []
-                           :prompt     (apply str (repeat 10001 "x"))}))
-
-  (testing "Respects explicit limit operation"
-    (let [table-id (mt/id :orders)
-          response (mt/user-http-request :rasta :post 200 "agent/v2/construct-query"
-                                         {:source     {:type "table" :id table-id}
-                                          :operations [["limit" 10]]
-                                          :prompt     "show 10 orders"})
+  (testing "Respects explicit limit on the stage"
+    (let [response (mt/user-http-request :rasta :post 200 "agent/v2/construct-query"
+                                         {:query (orders-query :limit 10)})
           decoded  (decode-query response)]
       (is (= 10 (lib/current-limit decoded)))))
 
-  (testing "Returns 404 for non-existent table"
-    (is (= "Not found."
-           (mt/user-http-request :rasta :post 404 "agent/v2/construct-query"
-                                 {:source     {:type "table" :id 999999}
-                                  :operations []
-                                  :prompt     "show orders"})))))
+  (testing "Returns 400 with `:unknown-table` ex-data for an unknown table FK"
+    ;; Note: the new repr contract surfaces unknown-input errors as 400 (`:agent-error?`)
+    ;; rather than the old 404 — the LLM is expected to read the message and self-correct.
+    ;; ex-data values are stringified by the JSON roundtrip, so we match the keyword's name.
+    (let [response (mt/user-http-request :rasta :post 400 "agent/v2/construct-query"
+                                         {:query {:lib/type "mbql/query"
+                                                  :stages   [{:lib/type     "mbql.stage/mbql"
+                                                              :source-table [(db-name) "PUBLIC" "NOT_A_TABLE"]}]}})]
+      (is (=? {:error "unknown-table"} response))))
+
+  (testing "Rejects a non-aggregation in the aggregation slot with a 400"
+    ;; The endpoint's wire schema is intentionally permissive (`:query :map`); deep
+    ;; validation happens inside the representations pipeline. A `field` clause where an
+    ;; aggregation clause is expected is caught by the E1 friendly-error pass and surfaced
+    ;; as `:aggregation-entry-not-aggregation` with `:agent-error? true`, so the LLM gets
+    ;; a clean diagnostic instead of a generic schema error.
+    (let [response (mt/user-http-request :rasta :post 400 "agent/v2/construct-query"
+                                         {:query (orders-query
+                                                  :aggregation [(orders-field-ref "ID")])})]
+      (is (=? {:error "aggregation-entry-not-aggregation"} response)))))
+
+(deftest construct-query-permission-checks-test
+  (testing "Rejects a first-stage source-table the current user cannot query"
+    (mt/with-no-data-perms-for-all-users!
+      (is (= "You don't have permissions to do that."
+             (mt/user-http-request :rasta :post 403 "agent/v2/construct-query"
+                                   {:query (orders-query :aggregation [["count" {}]])})))))
+
+  (testing "Rejects a first-stage source-card the current user cannot read"
+    (mt/with-non-admin-groups-no-root-collection-perms
+      (let [mp (mt/metadata-provider)]
+        (mt/with-temp [:model/Collection _collection {}
+                       :model/Card       card        {:name          "Protected Question"
+                                                      :collection_id (:id _collection)
+                                                      :database_id   (mt/id)
+                                                      :dataset_query (lib/query mp (lib.metadata/table mp (mt/id :orders)))}]
+          (is (= "You don't have permissions to do that."
+                 (mt/user-http-request :rasta :post 403 "agent/v2/construct-query"
+                                       {:query (source-card-query (:entity_id card))})))))))
+
+  (testing "Rejects a metric aggregation the current user cannot read"
+    (mt/with-non-admin-groups-no-root-collection-perms
+      (mt/with-temp [:model/Collection _collection {}
+                     :model/Card       metric      {:name          "Protected Metric"
+                                                    :type          :metric
+                                                    :collection_id (:id _collection)
+                                                    :database_id   (mt/id)
+                                                    :dataset_query (orders-count-query)}]
+        (is (= "You don't have permissions to do that."
+               (mt/user-http-request :rasta :post 403 "agent/v2/construct-query"
+                                     {:query (orders-query
+                                              :aggregation [["metric" {} (:entity_id metric)]])})))))))
+
+(deftest construct-query-rejects-empty-query-test
+  (testing "Empty / blank :query is rejected by the request schema"
+    (mt/user-http-request :rasta :post 400 "agent/v2/construct-query" {:query ""}))
+  (testing "Missing :query in body is rejected by the request schema"
+    (mt/user-http-request :rasta :post 400 "agent/v2/construct-query" {})))
+
+(deftest construct-query-rejects-typod-stage-keys-test
+  (testing (str "The agent boundary asserts every stage's top-level keys are in a known set. "
+                "`lib.schema.mbql-stage/mbql` is not a closed map, so typo'd stage keys "
+                "(e.g. `aggreagation` for `aggregation`) would otherwise be silently dropped "
+                "at resolve time and the LLM would never learn that its intent was discarded.")
+    (testing "typo'd `aggreagation` is rejected with :unknown-stage-key"
+      (let [response (mt/user-http-request
+                      :rasta :post 400 "agent/v2/construct-query"
+                      {:query {:lib/type "mbql/query"
+                               :stages   [{:lib/type      "mbql.stage/mbql"
+                                           :source-table  [(db-name) "PUBLIC" "ORDERS"]
+                                           :aggreagation  [["count" {}]]}]}})]
+        (is (=? {:error "unknown-stage-key"} response))
+        (is (=? {:unknown-keys ["aggreagation"]} response))))
+    (testing "diagnostic lists the valid stage keys so the LLM can self-correct"
+      (let [response (mt/user-http-request
+                      :rasta :post 400 "agent/v2/construct-query"
+                      {:query {:lib/type "mbql/query"
+                               :stages   [{:lib/type     "mbql.stage/mbql"
+                                           :source-table [(db-name) "PUBLIC" "ORDERS"]
+                                           :groupby      ["foo"]}]}})]
+        (is (=? {:error "unknown-stage-key"} response))
+        (is (re-find #"aggregation" (:message response)))))))
+
+(deftest construct-query-rejects-legacy-envelope-test
+  (testing (str "Legacy `source_entity` / `referenced_entities` envelope from the pre-repr program API "
+                "is rejected by the now-closed request schema, instead of being silently ignored. "
+                "This guards against a regression where the LLM's stale memory keeps sending the old "
+                "shape and we silently drop the extra keys.")
+    (is (=? {:specific-errors {:source_entity #(some (fn [s] (re-find #"disallowed key" s)) %)}}
+            (mt/user-http-request :rasta :post 400 "agent/v2/construct-query"
+                                  {:query          (orders-query)
+                                   :source_entity  {:type "table" :id (mt/id :orders)}}))))
+  (testing "`/v2/query` fresh-query branch rejects the legacy envelope as well"
+    (is (=? {:specific-errors {:referenced_entities #(some (fn [s] (re-find #"disallowed key" s)) %)}}
+            (mt/user-http-request :rasta :post 400 "agent/v2/query"
+                                  {:query               (orders-query :limit 5)
+                                   :referenced_entities []}))))
+  (testing "`/v2/query` continuation_token branch rejects extra keys (closed schema)"
+    (is (=? {:specific-errors {:query #(some (fn [s] (re-find #"disallowed key" s)) %)}}
+            (mt/user-http-request :rasta :post 400 "agent/v2/query"
+                                  {:continuation_token "not-a-real-token"
+                                   :query               (orders-query :limit 5)})))))
 
 (deftest execute-query-test
   (testing "Executes a query and returns results with column metadata"
-    (let [table-id       (mt/id :orders)
-          construct-resp (mt/user-http-request :rasta :post 200 "agent/v2/construct-query"
-                                               {:source     {:type "table" :id table-id}
-                                                :operations [["limit" 5]]
-                                                :prompt     "show 5 orders"})
+    (let [construct-resp (mt/user-http-request :rasta :post 200 "agent/v2/construct-query"
+                                               {:query (orders-query :limit 5)})
           ;; Streaming response returns 202 (accepted) since it starts streaming before completion
           execute-resp   (mt/user-http-request :rasta :post 202 "agent/v1/execute"
                                                {:query (:query construct-resp)})]
@@ -316,11 +436,8 @@
               execute-resp))))
 
   (testing "Enforces agent query row limit even when query specifies a higher limit"
-    (let [table-id       (mt/id :orders)
-          construct-resp (mt/user-http-request :rasta :post 200 "agent/v2/construct-query"
-                                               {:source     {:type "table" :id table-id}
-                                                :operations [["limit" 300]]
-                                                :prompt     "show all orders"})
+    (let [construct-resp (mt/user-http-request :rasta :post 200 "agent/v2/construct-query"
+                                               {:query (orders-query :limit 300)})
           execute-resp   (mt/user-http-request :rasta :post 202 "agent/v1/execute"
                                                {:query (:query construct-resp)})]
       (is (=? {:status "completed" :row_count 200}
@@ -355,44 +472,41 @@
                                      :type          :metric
                                      :database_id   (mt/id)
                                      :dataset_query (orders-count-query)}]
-    (testing "Constructs a query from a metric"
+    (testing "Constructs a query that references a metric by entity_id in an aggregation"
       (let [response (mt/user-http-request :rasta :post 200 "agent/v2/construct-query"
-                                           {:source     {:type "metric" :id (:id metric)}
-                                            :operations []
-                                            :prompt     "show metric"})]
+                                           {:query (orders-query
+                                                    :aggregation [["metric" {} (:entity_id metric)]])})]
         (is (string? (:query response)) "Response should contain a query string")
         (let [decoded (decode-query response)]
           (is (= :mbql/query (lib/normalized-query-type decoded)))
-          (is (= (mt/id) (lib/database-id decoded))))))
+          (is (= (mt/id) (lib/database-id decoded)))
+          (is (= 1 (count (lib/aggregations decoded)))))))
 
-    (testing "Returns 404 for non-existent metric"
-      (is (= "Not found."
-             (mt/user-http-request :rasta :post 404 "agent/v2/construct-query"
-                                   {:source     {:type "metric" :id 999999}
-                                    :operations []
-                                    :prompt     "show metric"}))))))
+    (testing "Returns 400 with `:unknown-card` ex-data for an unknown metric entity_id"
+      ;; The entity_id must be a syntactically-valid 21-char NanoID for the resolver to
+      ;; even attempt the lookup; otherwise the metric clause's literal string survives to
+      ;; lib.schema validation and we get a different (less useful) error class. Pick a
+      ;; well-formed but absent eid to drive the `:unknown-card` path specifically.
+      (let [response (mt/user-http-request :rasta :post 400 "agent/v2/construct-query"
+                                           {:query (orders-query
+                                                    :aggregation [["metric" {} "AAAAAAAAAAAAAAAAAAAAA"]])})]
+        (is (=? {:error "unknown-card"} response))))))
 
 (deftest construct-query-with-count-aggregation-test
   (testing "Count aggregation produces a valid query"
-    (let [table-id (mt/id :orders)
-          response (mt/user-http-request :rasta :post 200 "agent/v2/construct-query"
-                                         {:source     {:type "table" :id table-id}
-                                          :operations [["aggregate" ["count"]]
-                                                       ["limit" 10]]
-                                          :prompt     "count orders"})]
+    (let [response (mt/user-http-request :rasta :post 200 "agent/v2/construct-query"
+                                         {:query (orders-query :aggregation [["count" {}]]
+                                                               :limit 10)})]
       (is (string? (:query response)))
       (let [decoded (decode-query response)]
         (is (= 1 (count (lib/aggregations decoded))))))))
 
 (deftest construct-query-with-filters-test
   (testing "Constructs a query with filters"
-    (let [table-id (mt/id :orders)
-          field-id (mt/id :orders :id)
-          response (mt/user-http-request :rasta :post 200 "agent/v2/construct-query"
-                                         {:source     {:type "table" :id table-id}
-                                          :operations [["filter" ["not-null" ["field" field-id]]]
-                                                       ["limit" 10]]
-                                          :prompt     "show filtered orders"})]
+    (let [response (mt/user-http-request :rasta :post 200 "agent/v2/construct-query"
+                                         {:query (orders-query
+                                                  :filters [["not-null" {} (orders-field-ref "ID")]]
+                                                  :limit   10)})]
       (is (string? (:query response)))
       (let [decoded (decode-query response)]
         (is (seq (lib/filters decoded)) "Query should have filters")))))
@@ -418,12 +532,9 @@
 
 (deftest combined-query-test
   (testing "Returns results for a table query that fits in a single page"
-    (let [table-id (mt/id :orders)
-          field-id (visible-field-id table-id "ID")
-          response (mt/user-http-request :rasta :post 202 "agent/v2/query"
-                                         {:source     {:type "table" :id table-id}
-                                          :operations [["order-by" ["field" field-id]]
-                                                       ["limit" 5]]})]
+    (let [response (mt/user-http-request :rasta :post 202 "agent/v2/query"
+                                         {:query (orders-query :order-by [["asc" {} (orders-field-ref "ID")]]
+                                                               :limit    5)})]
       (is (=? {:status             "completed"
                :row_count          5
                :continuation_token nil?
@@ -432,14 +543,11 @@
               response))))
 
   (testing "Continuation token returns next page of results when the total limit exceeds the page size"
-    (let [table-id   (mt/id :orders)
-          field-id   (visible-field-id table-id "ID")
-          page-size  200
+    (let [page-size  200
           total-rows 250
           page1      (mt/user-http-request :rasta :post 202 "agent/v2/query"
-                                           {:source     {:type "table" :id table-id}
-                                            :operations [["order-by" ["field" field-id]]
-                                                         ["limit" total-rows]]})
+                                           {:query (orders-query :order-by [["asc" {} (orders-field-ref "ID")]]
+                                                                 :limit    total-rows)})
           page2      (mt/user-http-request :rasta :post 202 "agent/v2/query"
                                            {:continuation_token (:continuation_token page1)})]
       (is (=? {:row_count          page-size
@@ -458,15 +566,13 @@
     (is (=? {:status             "completed"
              :continuation_token nil?}
             (mt/user-http-request :rasta :post 202 "agent/v2/query"
-                                  {:source     {:type "table" :id (mt/id :orders)}
-                                   :operations [["aggregate" ["count"]]]}))))
+                                  {:query (orders-query :aggregation [["count" {}]])}))))
 
   (testing "Per-page cap limits a single page to 200 rows even when the total limit is higher"
     (is (=? {:status    "completed"
              :row_count (fn [n] (<= n 200))}
             (mt/user-http-request :rasta :post 202 "agent/v2/query"
-                                  {:source     {:type "table" :id (mt/id :orders)}
-                                   :operations [["limit" 1000]]})))))
+                                  {:query (orders-query :limit 1000)})))))
 
 (defn- make-continuation-token [pagination]
   (-> {:query {:database (mt/id) :stages [{:source-table (mt/id :orders)}]}
@@ -493,12 +599,12 @@
                                      :type          :metric
                                      :database_id   (mt/id)
                                      :dataset_query (orders-count-query)}]
-    (testing "Returns results for a metric query"
+    (testing "Returns results for a query that aggregates via a metric reference"
       (is (=? {:status    "completed"
                :row_count pos?}
               (mt/user-http-request :rasta :post 202 "agent/v2/query"
-                                    {:source     {:type "metric" :id (:id metric)}
-                                     :operations []}))))))
+                                    {:query (orders-query
+                                             :aggregation [["metric" {} (:entity_id metric)]])}))))))
 
 (deftest search-finds-metrics-test
   (binding [search.ingestion/*force-sync* true]
@@ -518,8 +624,7 @@
 (deftest create-question-test
   (testing "Creates a saved question from a constructed query"
     (let [construct-resp (mt/user-http-request :rasta :post 200 "agent/v2/construct-query"
-                                               {:source     {:type "table" :id (mt/id :orders)}
-                                                :operations [["limit" 10]]})
+                                               {:query (orders-query :limit 10)})
           create-resp    (mt/user-http-request :rasta :post 200 "agent/v1/question"
                                                {:name  "Agent Test Question"
                                                 :query (:query construct-resp)})]
@@ -535,8 +640,7 @@
   (testing "Creates a question with optional fields"
     (mt/with-temp [:model/Collection {coll-id :id} {:name "Agent Question Collection"}]
       (let [construct-resp (mt/user-http-request :rasta :post 200 "agent/v2/construct-query"
-                                                 {:source     {:type "table" :id (mt/id :orders)}
-                                                  :operations [["limit" 10]]})
+                                                 {:query (orders-query :limit 10)})
             create-resp    (mt/user-http-request :rasta :post 200 "agent/v1/question"
                                                  {:name          "Agent Question With Options"
                                                   :query         (:query construct-resp)

test/metabase/agent_lib/capabilities/catalog_test.clj

@@ -1,43 +0,0 @@
-(ns metabase.agent-lib.capabilities.catalog-test
-  (:require
-   [clojure.test :refer [deftest is]]
-   [metabase.agent-lib.capabilities :as capabilities]
-   [metabase.agent-lib.capabilities.catalog :as catalog]
-   [metabase.agent-lib.capabilities.catalog.aggregation :as aggregation]
-   [metabase.agent-lib.capabilities.catalog.expressions :as expressions]
-   [metabase.agent-lib.capabilities.catalog.filtering :as filtering]
-   [metabase.agent-lib.capabilities.catalog.joins :as joins]
-   [metabase.agent-lib.capabilities.catalog.ordering :as ordering]
-   [metabase.agent-lib.capabilities.catalog.sources :as sources]
-   [metabase.agent-lib.capabilities.catalog.top-level :as top-level]))
-
-(deftest ^:parallel grouped-capability-catalog-preserves-review-groups-and-order-test
-  (is (= [:top-level
-          :sources
-          :filtering
-          :aggregation
-          :breakout-ordering
-          :expressions
-          :joins]
-         (mapv :group catalog/grouped-capability-catalog)))
-  (is (= [top-level/capabilities
-          sources/capabilities
-          filtering/capabilities
-          aggregation/capabilities
-          ordering/capabilities
-          expressions/capabilities
-          joins/capabilities]
-         (mapv :capabilities catalog/grouped-capability-catalog))))
-
-(deftest ^:parallel raw-capability-catalog-flattens-the-grouped-catalog-test
-  (let [expected (into []
-                       cat
-                       [top-level/capabilities
-                        sources/capabilities
-                        filtering/capabilities
-                        aggregation/capabilities
-                        ordering/capabilities
-                        expressions/capabilities
-                        joins/capabilities])]
-    (is (= expected catalog/raw-capability-catalog))
-    (is (= expected capabilities/raw-capability-catalog))))

test/metabase/agent_lib/common/coercions_test.clj

@@ -1,113 +0,0 @@
-(ns metabase.agent-lib.common.coercions-test
-  (:require
-   [clojure.test :refer [deftest is testing]]
-   [metabase.agent-lib.common.coercions :as coercions]))
-
-(deftest ^:parallel quarter-label->number-test
-  (testing "converts Q labels to integers"
-    (is (= 1 (coercions/quarter-label->number "Q1")))
-    (is (= 2 (coercions/quarter-label->number "q2")))
-    (is (= 3 (coercions/quarter-label->number " Q3 ")))
-    (is (= 4 (coercions/quarter-label->number "Q4"))))
-  (testing "returns nil for invalid inputs"
-    (is (nil? (coercions/quarter-label->number "Q5")))
-    (is (nil? (coercions/quarter-label->number "Q0")))
-    (is (nil? (coercions/quarter-label->number "abc")))
-    (is (nil? (coercions/quarter-label->number nil)))))
-
-(deftest ^:parallel parse-int-string-test
-  (testing "parses valid integer strings"
-    (is (= 42 (coercions/parse-int-string "42")))
-    (is (= -7 (coercions/parse-int-string "-7")))
-    (is (= 0 (coercions/parse-int-string "0"))))
-  (testing "returns nil for invalid inputs"
-    (is (nil? (coercions/parse-int-string "abc")))
-    (is (nil? (coercions/parse-int-string "3.14")))
-    (is (nil? (coercions/parse-int-string "")))
-    (is (nil? (coercions/parse-int-string nil)))))
-
-(deftest ^:parallel coerce-positive-int-test
-  (testing "passes through positive integers"
-    (is (= 5 (coercions/coerce-positive-int 5))))
-  (testing "coerces positive integer strings"
-    (is (= 5 (coercions/coerce-positive-int "5")))
-    (is (= 100 (coercions/coerce-positive-int "100"))))
-  (testing "does not coerce non-positive strings"
-    (is (= "0" (coercions/coerce-positive-int "0")))
-    (is (= "-1" (coercions/coerce-positive-int "-1"))))
-  (testing "passes through non-string values"
-    (is (= :kw (coercions/coerce-positive-int :kw)))
-    (is (= -1 (coercions/coerce-positive-int -1)))))
-
-(deftest ^:parallel coerce-non-negative-int-test
-  (testing "passes through non-negative integers"
-    (is (= 0 (coercions/coerce-non-negative-int 0)))
-    (is (= 5 (coercions/coerce-non-negative-int 5))))
-  (testing "coerces non-negative integer strings"
-    (is (= 0 (coercions/coerce-non-negative-int "0")))
-    (is (= 42 (coercions/coerce-non-negative-int "42"))))
-  (testing "does not coerce negative strings"
-    (is (= "-1" (coercions/coerce-non-negative-int "-1"))))
-  (testing "passes through non-string values"
-    (is (= :kw (coercions/coerce-non-negative-int :kw)))))
-
-(deftest ^:parallel direction-string-test
-  (testing "recognizes direction strings"
-    (is (true? (coercions/direction-string? "asc")))
-    (is (true? (coercions/direction-string? "desc")))
-    (is (true? (coercions/direction-string? "ASC")))
-    (is (true? (coercions/direction-string? :desc))))
-  (testing "rejects non-direction values"
-    (is (false? (coercions/direction-string? "up")))
-    (is (false? (coercions/direction-string? nil)))))
-
-(deftest ^:parallel normalize-direction-test
-  (testing "normalizes to lowercase string"
-    (is (= "asc" (coercions/normalize-direction "ASC")))
-    (is (= "desc" (coercions/normalize-direction :desc)))
-    (is (= "asc" (coercions/normalize-direction 'asc))))
-  (testing "returns nil for nil"
-    (is (nil? (coercions/normalize-direction nil)))))
-
-(deftest ^:parallel unwrap-singleton-form-test
-  (testing "unwraps nested singletons but stops at scalar-sequential"
-    (is (= [5] (coercions/unwrap-singleton-form [[[[5]]]]))))
-  (testing "stops at operator tuples"
-    (is (= ["field" 1] (coercions/unwrap-singleton-form [[["field" 1]]]))))
-  (testing "passes through non-singleton vectors"
-    (is (= [1 2] (coercions/unwrap-singleton-form [1 2]))))
-  (testing "passes through scalars"
-    (is (= "hello" (coercions/unwrap-singleton-form "hello")))
-    (is (= 42 (coercions/unwrap-singleton-form 42)))))
-
-(deftest ^:parallel normalize-percentile-value-test
-  (testing "converts whole-number percentages to fractions"
-    (is (= 0.9 (coercions/normalize-percentile-value 90)))
-    (is (= 0.5 (coercions/normalize-percentile-value 50)))
-    (is (= 1.0 (coercions/normalize-percentile-value 100))))
-  (testing "passes through values already in fraction form"
-    (is (= 0.9 (coercions/normalize-percentile-value 0.9)))
-    (is (= 0.5 (coercions/normalize-percentile-value 0.5))))
-  (testing "boundary: 1.0 is already a fraction"
-    (is (= 1.0 (coercions/normalize-percentile-value 1.0))))
-  (testing "passes through non-numbers"
-    (is (= "abc" (coercions/normalize-percentile-value "abc")))))
-
-(deftest ^:parallel normalize-join-conditions-test
-  (testing "unwraps extra singleton nesting"
-    (is (= [["=" ["field" 1] ["field" 2]]]
-           (coercions/normalize-join-conditions [[["=" ["field" 1] ["field" 2]]]]))))
-  (testing "leaves properly-structured conditions alone"
-    (is (= [["=" ["field" 1] ["field" 2]]]
-           (coercions/normalize-join-conditions [["=" ["field" 1] ["field" 2]]]))))
-  (testing "handles deeply nested singleton wrapping"
-    (is (= [["=" ["field" 1] ["field" 2]]]
-           (coercions/normalize-join-conditions [[[["=" ["field" 1] ["field" 2]]]]])))))
-
-(deftest ^:parallel normalize-map-key-test
-  (testing "converts string keys to keywords"
-    (is (= :foo (coercions/normalize-map-key "foo"))))
-  (testing "passes through keyword keys"
-    (is (= :bar (coercions/normalize-map-key :bar))))
-  (testing "passes through non-string keys"
-    (is (= 42 (coercions/normalize-map-key 42)))))

test/metabase/agent_lib/common/context_test.clj

@@ -1,25 +0,0 @@
-(ns metabase.agent-lib.common.context-test
-  (:require
-   [clojure.test :refer [deftest is testing]]
-   [metabase.agent-lib.common.context :as context]))
-
-(deftest ^:parallel source-metric-id-test
-  (testing "returns id for metric source entities"
-    (is (= 42 (context/source-metric-id {:source-entity {:model "metric" :id 42}}))))
-  (testing "returns nil for table source entities"
-    (is (nil? (context/source-metric-id {:source-entity {:model "table" :id 1}}))))
-  (testing "returns nil for non-positive ids"
-    (is (nil? (context/source-metric-id {:source-entity {:model "metric" :id 0}})))
-    (is (nil? (context/source-metric-id {:source-entity {:model "metric" :id -1}}))))
-  (testing "returns nil for missing source-entity"
-    (is (nil? (context/source-metric-id {})))))
-
-(deftest ^:parallel source-table-id-test
-  (testing "returns id for table source entities"
-    (is (= 1 (context/source-table-id {:source-entity {:model "table" :id 1}}))))
-  (testing "returns nil for metric source entities"
-    (is (nil? (context/source-table-id {:source-entity {:model "metric" :id 42}}))))
-  (testing "returns nil for non-positive ids"
-    (is (nil? (context/source-table-id {:source-entity {:model "table" :id 0}}))))
-  (testing "returns nil for missing source-entity"
-    (is (nil? (context/source-table-id {})))))

test/metabase/agent_lib/common/literals_test.clj

@@ -1,133 +0,0 @@
-(ns metabase.agent-lib.common.literals-test
-  (:require
-   [clojure.test :refer [deftest is testing]]
-   [metabase.agent-lib.common.literals :as literals]))
-
-(deftest ^:parallel scalar-literal-test
-  (testing "nil is a scalar literal"
-    (is (true? (literals/scalar-literal? nil))))
-  (testing "strings are scalar literals"
-    (is (true? (literals/scalar-literal? "hello")))
-    (is (true? (literals/scalar-literal? ""))))
-  (testing "numbers are scalar literals"
-    (is (true? (literals/scalar-literal? 42)))
-    (is (true? (literals/scalar-literal? 3.14)))
-    (is (true? (literals/scalar-literal? 0))))
-  (testing "booleans are scalar literals"
-    (is (true? (literals/scalar-literal? true)))
-    (is (true? (literals/scalar-literal? false))))
-  (testing "maps and vectors are not scalar literals"
-    (is (false? (literals/scalar-literal? {})))
-    (is (false? (literals/scalar-literal? [])))
-    (is (false? (literals/scalar-literal? :keyword)))))
-
-(deftest ^:parallel non-blank-string-test
-  (testing "non-blank strings"
-    (is (true? (literals/non-blank-string? "hello")))
-    (is (true? (literals/non-blank-string? " x "))))
-  (testing "blank strings"
-    (is (false? (literals/non-blank-string? "")))
-    (is (false? (literals/non-blank-string? "   "))))
-  (testing "non-strings"
-    (is (false? (literals/non-blank-string? nil)))
-    (is (false? (literals/non-blank-string? 42)))))
-
-(deftest ^:parallel quarter-label-string-test
-  (testing "valid quarter labels"
-    (is (some? (literals/quarter-label-string? "Q1")))
-    (is (some? (literals/quarter-label-string? "Q4")))
-    (is (some? (literals/quarter-label-string? "q2")))
-    (is (some? (literals/quarter-label-string? " Q3 "))))
-  (testing "invalid quarter labels"
-    (is (nil? (literals/quarter-label-string? "Q5")))
-    (is (nil? (literals/quarter-label-string? "Q0")))
-    (is (nil? (literals/quarter-label-string? "Q")))
-    (is (nil? (literals/quarter-label-string? "")))
-    (is (not (literals/quarter-label-string? nil)))))
-
-(deftest ^:parallel now-literal-test
-  (testing "case-insensitive now"
-    (is (true? (literals/now-literal? "now")))
-    (is (true? (literals/now-literal? "NOW")))
-    (is (true? (literals/now-literal? " now "))))
-  (testing "not-now values"
-    (is (false? (literals/now-literal? "not-now")))
-    (is (false? (literals/now-literal? nil)))
-    (is (false? (literals/now-literal? 42)))))
-
-(deftest ^:parallel iso-date-string-test
-  (testing "valid ISO dates"
-    (is (some? (literals/iso-date-string? "2024-01-15")))
-    (is (some? (literals/iso-date-string? " 2024-12-31 "))))
-  (testing "invalid date formats"
-    (is (nil? (literals/iso-date-string? "2024-1-15")))
-    (is (nil? (literals/iso-date-string? "not-a-date")))
-    (is (not (literals/iso-date-string? nil)))))
-
-(deftest ^:parallel null-literal-test
-  (testing "nil is null"
-    (is (true? (literals/null-literal? nil))))
-  (testing "string null variants"
-    (is (true? (literals/null-literal? "null")))
-    (is (true? (literals/null-literal? "NULL"))))
-  (testing "non-null values"
-    (is (false? (literals/null-literal? "not null")))
-    (is (false? (literals/null-literal? 0)))
-    (is (false? (literals/null-literal? false)))))
-
-(deftest ^:parallel scalar-sequential-test
-  (testing "vector of scalars"
-    (is (true? (literals/scalar-sequential? [1 2 3])))
-    (is (true? (literals/scalar-sequential? ["a" "b"])))
-    (is (true? (literals/scalar-sequential? [nil true 42 "x"]))))
-  (testing "empty vector is scalar-sequential"
-    (is (true? (literals/scalar-sequential? []))))
-  (testing "rejects operator tuples"
-    (is (false? (literals/scalar-sequential? ["count"])))
-    (is (false? (literals/scalar-sequential? ["field" 1]))))
-  (testing "rejects non-sequential"
-    (is (false? (literals/scalar-sequential? "abc")))
-    (is (false? (literals/scalar-sequential? 42)))))
-
-(deftest ^:parallel aggregation-form-test
-  (testing "recognizes canonical aggregation operators"
-    (is (some? (literals/aggregation-form? ["count"])))
-    (is (some? (literals/aggregation-form? ["sum" ["field" 1]])))
-    (is (some? (literals/aggregation-form? ["avg" ["field" 1]])))
-    (is (some? (literals/aggregation-form? ["min" ["field" 1]])))
-    (is (some? (literals/aggregation-form? ["max" ["field" 1]])))
-    (is (some? (literals/aggregation-form? ["distinct" ["field" 1]])))
-    (is (some? (literals/aggregation-form? ["median" ["field" 1]])))
-    (is (some? (literals/aggregation-form? ["stddev" ["field" 1]])))
-    (is (some? (literals/aggregation-form? ["var" ["field" 1]])))
-    (is (some? (literals/aggregation-form? ["count-where" ["field" 1]])))
-    (is (some? (literals/aggregation-form? ["percentile" ["field" 1] 0.9])))
-    (is (some? (literals/aggregation-form? ["metric" 42]))))
-  (testing "rejects non-aggregation operators"
-    (is (nil? (literals/aggregation-form? ["field" 1])))
-    (is (nil? (literals/aggregation-form? ["=" 1 2]))))
-  (testing "rejects non-vectors"
-    (is (not (literals/aggregation-form? "count")))))
-
-(deftest ^:parallel default-map-test
-  (testing "maps with :default key"
-    (is (true? (literals/default-map? {:default nil})))
-    (is (true? (literals/default-map? {"default" 42}))))
-  (testing "maps without default key"
-    (is (false? (literals/default-map? {:other 1})))
-    (is (false? (literals/default-map? {}))))
-  (testing "non-maps"
-    (is (false? (literals/default-map? nil)))
-    (is (false? (literals/default-map? [])))))
-
-(deftest ^:parallel bare-field-id-test
-  (testing "positive integers"
-    (is (true? (literals/bare-field-id? 1)))
-    (is (true? (literals/bare-field-id? 42))))
-  (testing "zero and negatives"
-    (is (false? (literals/bare-field-id? 0)))
-    (is (false? (literals/bare-field-id? -1))))
-  (testing "non-integers"
-    (is (false? (literals/bare-field-id? 3.14)))
-    (is (false? (literals/bare-field-id? "42")))
-    (is (false? (literals/bare-field-id? nil)))))

test/metabase/agent_lib/eval/args_test.clj

@@ -1,30 +0,0 @@
-(ns metabase.agent-lib.eval.args-test
-  (:require
-   [clojure.test :refer [deftest is]]
-   [metabase.agent-lib.eval.args :as eval.args]))
-
-(deftest ^:parallel normalize-helper-args-flattens-in-lists-test
-  (is (= [["field" 10] "a" "b"]
-         (eval.args/normalize-helper-args [:operations 0 1]
-                                          'in
-                                          [["field" 10] ["a" "b"]]))))
-
-(deftest ^:parallel normalize-helper-args-normalizes-enums-test
-  (is (= [["aggregation-ref" 0] :desc]
-         (eval.args/normalize-helper-args [:operations 1]
-                                          'order-by
-                                          [["aggregation-ref" 0] "desc"])))
-  (is (= [["field" 10] -30 :day 0 :day]
-         (eval.args/normalize-helper-args [:operations 2]
-                                          'relative-time-interval
-                                          [["field" 10] -30 "day" 0 "day"]))))
-
-(deftest ^:parallel normalize-helper-args-normalizes-percentiles-and-join-fields-test
-  (is (= [["field" 12] 0.9]
-         (eval.args/normalize-helper-args [:operations 3]
-                                          'percentile
-                                          [["field" 12] 90])))
-  (is (= [["join-clause" ["table" 1]] :all]
-         (eval.args/normalize-helper-args [:operations 4]
-                                          'with-join-fields
-                                          [["join-clause" ["table" 1]] "all"]))))

test/metabase/agent_lib/eval/invoke_test.clj

@@ -1,54 +0,0 @@
-(ns metabase.agent-lib.eval.invoke-test
-  (:require
-   [clojure.test :refer [deftest is testing]]
-   [metabase.agent-lib.eval.invoke :as eval.invoke]
-   [metabase.agent-lib.test-util :as agent-lib.tu]))
-
-(defn- runtime-with-bindings
-  [bindings]
-  {:bindings bindings})
-
-(deftest invoke-helper-rejects-unknown-operators-test
-  (let [runtime (runtime-with-bindings {'known identity})]
-    (try
-      (eval.invoke/invoke-helper! runtime [:operations 0] 'missing [])
-      (is false "expected unknown helper to fail")
-      (catch clojure.lang.ExceptionInfo e
-        (is (= :invalid-generated-program (:error (ex-data e))))
-        (is (= "missing" (:operator (ex-data e))))))))
-
-(deftest invoke-query-aware-helper-requires-current-query-test
-  (let [runtime (runtime-with-bindings {'filter (fn [& _] :ok)})]
-    (try
-      (eval.invoke/invoke-query-aware-helper! runtime nil [:operations 0] 'filter [])
-      (is false "expected query-aware helper to require current query")
-      (catch clojure.lang.ExceptionInfo e
-        (is (= :invalid-generated-program (:error (ex-data e))))
-        (is (re-find #"`filter` requires the current query state"
-                     (:details (ex-data e))))))))
-
-(deftest invoke-query-aware-helper-shapes-expression-ref-errors-test
-  (let [runtime (runtime-with-bindings {'expression-ref (fn [& _]
-                                                          (throw (ex-info "missing expression" {})))})]
-    (try
-      (eval.invoke/invoke-query-aware-helper! runtime {:query true} [:operations 0] 'expression-ref ["Net"])
-      (is false "expected expression-ref failure")
-      (catch clojure.lang.ExceptionInfo e
-        (is (= :invalid-generated-program (:error (ex-data e))))
-        (is (= "expression-ref" (:operator (ex-data e))))
-        (is (= "Net" (:name (ex-data e))))))))
-
-(deftest invoke-field-helper-prefers-query-aware-arity-test
-  (let [runtime (runtime-with-bindings {'field (fn [query arg] [query arg])})]
-    (is (= [:current-query 42]
-           (eval.invoke/invoke-field-helper! runtime :current-query [:operations 0] [42])))))
-
-(deftest ensure-query-result-recognizes-mbql5-queries-test
-  (testing "non-query values are rejected"
-    (is (thrown-with-msg?
-         clojure.lang.ExceptionInfo
-         #"must return a MBQL 5 query map"
-         (eval.invoke/ensure-query-result! :not-a-query))))
-  (testing "queries pass through unchanged"
-    (let [query (agent-lib.tu/query-for-table :orders)]
-      (is (= query (eval.invoke/ensure-query-result! query))))))

test/metabase/agent_lib/eval/source_test.clj

@@ -1,59 +0,0 @@
-(ns metabase.agent-lib.eval.source-test
-  (:require
-   [clojure.test :refer [deftest is testing]]
-   [metabase.agent-lib.eval.source :as source]
-   [metabase.agent-lib.runtime :as runtime]
-   [metabase.agent-lib.test-util :as tu]
-   [metabase.lib.test-metadata :as meta]))
-
-(deftest ^:parallel resolve-source-context-test
-  (testing "context source returns the source from runtime bindings"
-    (let [base-query (tu/query-for-table :orders)
-          rt         (runtime/build-runtime meta/metadata-provider {:extra-bindings {'source base-query}})
-          result     (source/resolve-source identity rt [:source] {:type "context" :ref "source"})]
-      (is (= base-query result)))))
-
-(deftest resolve-source-context-missing-test
-  (testing "context source throws when unavailable in runtime bindings"
-    (let [rt    (runtime/build-runtime meta/metadata-provider)
-          error (try
-                  (source/resolve-source identity rt [:source] {:type "context" :ref "source"})
-                  nil
-                  (catch clojure.lang.ExceptionInfo e e))]
-      (is error)
-      (is (= :invalid-generated-program (:error (ex-data error))))
-      (is (re-find #"context source is unavailable"
-                   (:details (ex-data error)))))))
-
-(deftest ^:parallel resolve-source-table-test
-  (testing "table source calls table lookup with id and returns a query"
-    (let [rt     (runtime/build-runtime meta/metadata-provider)
-          result (source/resolve-source identity rt [:source] {:type "table" :id (meta/id :orders)})]
-      (is (map? result)))))
-
-(deftest resolve-source-unsupported-type-test
-  (testing "unsupported source type throws"
-    (let [rt    (runtime/build-runtime meta/metadata-provider)
-          error (try
-                  (source/resolve-source identity rt [:source] {:type "unknown"})
-                  nil
-                  (catch clojure.lang.ExceptionInfo e e))]
-      (is error)
-      (is (= :invalid-generated-program (:error (ex-data error))))
-      (is (re-find #"unsupported source type"
-                   (:details (ex-data error)))))))
-
-(deftest ^:parallel resolve-source-program-test
-  (testing "program source recurses via evaluate-program callback"
-    (let [rt           (runtime/build-runtime meta/metadata-provider)
-          inner-result {:result :from-inner-program}
-          evaluate-fn  (fn [_path program]
-                         (when (= {:source {:type "context" :ref "source"}
-                                   :operations [["limit" 5]]}
-                                  program)
-                           inner-result))
-          result       (source/resolve-source evaluate-fn rt [:source]
-                                              {:type    "program"
-                                               :program {:source     {:type "context" :ref "source"}
-                                                         :operations [["limit" 5]]}})]
-      (is (= inner-result result)))))

test/metabase/agent_lib/eval_test.clj

@@ -1,764 +0,0 @@
-(ns metabase.agent-lib.eval-test
-  (:require
-   [clojure.test :refer [deftest is testing]]
-   [metabase.agent-lib.core :as program]
-   [metabase.agent-lib.test-util :as tu]
-   [metabase.lib.core :as lib]
-   [metabase.lib.test-metadata :as meta]
-   [metabase.lib.test-util :as lib.tu]
-   [metabase.lib.util :as lib.util]))
-
-(deftest ^:parallel expression-ref-parity-test
-  (let [program {:source     {:type "context" :ref "source"}
-                 :operations [["expression" "Net Amount"
-                               ["-" ["field" (meta/id :orders :total)]
-                                ["field" (meta/id :orders :discount)]]]
-                              ["order-by" ["desc" ["expression-ref" "Net Amount"]]]
-                              ["limit" 10]]}
-        expected (-> (tu/query-for-table :orders)
-                     (lib/expression "Net Amount"
-                                     (lib/- (meta/field-metadata :orders :total)
-                                            (meta/field-metadata :orders :discount)))
-                     (as-> query
-                           (lib/order-by query (lib/expression-ref query "Net Amount") :desc))
-                     (lib/limit 10))]
-    (is (= (tu/comparable-query expected)
-           (-> (program/evaluate-program program meta/metadata-provider (tu/table-context :orders))
-               tu/comparable-query)))))
-
-(deftest ^:parallel expression-ref-breakout-parity-test
-  (let [program {:source     {:type "context" :ref "source"}
-                 :operations [["expression" "Hour of Day" ["get-hour" ["field" (meta/id :orders :created-at)]]]
-                              ["aggregate" ["count"]]
-                              ["breakout" ["expression-ref" "Hour of Day"]]
-                              ["order-by" ["expression-ref" "Hour of Day"]]]}
-        base-query (-> (tu/query-for-table :orders)
-                       (lib/expression "Hour of Day" (lib/get-hour (meta/field-metadata :orders :created-at)))
-                       (lib/aggregate (lib/count)))
-        expected   (as-> base-query query
-                     (let [expr-ref (lib/expression-ref query "Hour of Day")]
-                       (-> query
-                           (lib/breakout expr-ref)
-                           (lib/order-by (lib.util/fresh-uuids expr-ref)))))]
-    (is (= (tu/comparable-query expected)
-           (-> (program/evaluate-program program meta/metadata-provider (tu/table-context :orders))
-               tu/comparable-query)))))
-
-(deftest ^:parallel raw-expression-breakout-parity-test
-  (let [program {:source     {:type "context" :ref "source"}
-                 :operations [["aggregate" ["count"]]
-                              ["breakout" ["get-day-of-week" ["field" (meta/id :orders :created-at)]]]
-                              ["order-by" ["get-day-of-week" ["field" (meta/id :orders :created-at)]]]]}
-        base-query (-> (tu/query-for-table :orders)
-                       (lib/expression "__breakout_expression_1"
-                                       (lib/get-day-of-week (meta/field-metadata :orders :created-at)))
-                       (lib/aggregate (lib/count)))
-        expected   (as-> base-query query
-                     (let [expr-ref (lib/expression-ref query "__breakout_expression_1")]
-                       (-> query
-                           (lib/breakout expr-ref)
-                           (lib/order-by (lib.util/fresh-uuids expr-ref)))))]
-    (is (= (tu/comparable-query expected)
-           (-> (program/evaluate-program program meta/metadata-provider (tu/table-context :orders))
-               tu/comparable-query)))))
-
-(deftest ^:parallel raw-field-order-by-reuses-matching-breakout-expression-test
-  (let [program {:source     {:type "context" :ref "source"}
-                 :operations [["aggregate" ["count"]]
-                              ["breakout" ["get-hour" ["field" (meta/id :orders :created-at)]]]
-                              ["order-by" ["field" (meta/id :orders :created-at)]]]}
-        base-query (-> (tu/query-for-table :orders)
-                       (lib/expression "__breakout_expression_1"
-                                       (lib/get-hour (meta/field-metadata :orders :created-at)))
-                       (lib/aggregate (lib/count)))
-        expected   (as-> base-query query
-                     (let [expr-ref (lib/expression-ref query "__breakout_expression_1")]
-                       (-> query
-                           (lib/breakout expr-ref)
-                           (lib/order-by (lib.util/fresh-uuids expr-ref)))))]
-    (is (= (tu/comparable-query expected)
-           (-> (program/evaluate-program program meta/metadata-provider (tu/table-context :orders))
-               tu/comparable-query)))))
-
-(deftest ^:parallel raw-expression-order-by-parity-test
-  (let [program {:source     {:type "context" :ref "source"}
-                 :operations [["expression" "Net Amount"
-                               ["-" ["field" (meta/id :orders :total)]
-                                ["field" (meta/id :orders :discount)]]]
-                              ["order-by" ["desc"
-                                           ["-" ["field" (meta/id :orders :total)]
-                                            ["field" (meta/id :orders :discount)]]]]
-                              ["limit" 10]]}
-        expected (-> (tu/query-for-table :orders)
-                     (lib/expression "Net Amount"
-                                     (lib/- (meta/field-metadata :orders :total)
-                                            (meta/field-metadata :orders :discount)))
-                     (as-> query
-                           (lib/order-by query (lib/expression-ref query "Net Amount") :desc))
-                     (lib/limit 10))]
-    (is (= (tu/comparable-query expected)
-           (-> (program/evaluate-program program meta/metadata-provider (tu/table-context :orders))
-               tu/comparable-query)))))
-
-(deftest ^:parallel expression-projection-parity-test
-  (let [program {:source     {:type "context" :ref "source"}
-                 :operations [["expression" "Net Amount"
-                               ["-" ["field" (meta/id :orders :total)]
-                                ["field" (meta/id :orders :discount)]]]
-                              ["with-fields" [["field" (meta/id :orders :id)]
-                                              ["expression-ref" "Net Amount"]]]
-                              ["order-by" ["desc" ["expression-ref" "Net Amount"]]]
-                              ["limit" 10]]}
-        expected (-> (tu/query-for-table :orders)
-                     (lib/expression "Net Amount"
-                                     (lib/- (meta/field-metadata :orders :total)
-                                            (meta/field-metadata :orders :discount)))
-                     (as-> query
-                           (-> query
-                               (lib/with-fields [(meta/field-metadata :orders :id)
-                                                 (lib/expression-ref query "Net Amount")])
-                               (lib/order-by (lib/expression-ref query "Net Amount") :desc)))
-                     (lib/limit 10))]
-    (is (= (tu/comparable-query expected)
-           (-> (program/evaluate-program program meta/metadata-provider (tu/table-context :orders))
-               tu/comparable-query)))))
-
-(deftest ^:parallel same-stage-aggregation-ref-parity-test
-  (let [program {:source     {:type "context" :ref "source"}
-                 :operations [["aggregate" ["count"]]
-                              ["aggregate" ["sum" ["field" (meta/id :venues :price)]]]
-                              ["breakout" ["field" (meta/id :venues :category-id)]]
-                              ["filter" [">" ["aggregation-ref" 1] 10]]
-                              ["order-by" ["desc" ["aggregation-ref" 1]]]]}
-        expected (-> (tu/query-for-table :venues)
-                     (lib/aggregate (lib/count))
-                     (lib/aggregate (lib/sum (meta/field-metadata :venues :price)))
-                     (lib/breakout (meta/field-metadata :venues :category-id))
-                     (as-> query
-                           (-> query
-                               (lib/filter (lib/> (lib/aggregation-ref query 1) 10))
-                               (lib/order-by (lib/aggregation-ref query 1) :desc))))]
-    (is (= (tu/comparable-query expected)
-           (-> (program/evaluate-program program meta/metadata-provider (tu/table-context :venues))
-               tu/comparable-query)))))
-
-(deftest ^:parallel join-with-explicit-fields-test
-  (let [program {:source     {:type "context" :ref "source"}
-                 :operations [["join"
-                               ["with-join-fields"
-                                ["with-join-conditions"
-                                 ["join-clause" ["table" (meta/id :categories)]]
-                                 [["=" ["field" (meta/id :venues :category-id)]
-                                   ["field" (meta/id :categories :id)]]]]
-                                [["field" (meta/id :categories :name)]]]]]}
-        base-query (lib.tu/venues-query)
-        expected (-> base-query
-                     (lib/join (-> (lib/join-clause (meta/table-metadata :categories))
-                                   (lib/with-join-conditions
-                                    [(lib/= (tu/current-query-column base-query (meta/id :venues :category-id))
-                                            (tu/current-query-column base-query (meta/id :categories :id)))])
-                                   (lib/with-join-fields
-                                     [(tu/current-query-column base-query (meta/id :categories :name))]))))]
-    (is (= (tu/comparable-query expected)
-           (-> (program/evaluate-program program meta/metadata-provider (tu/table-context :venues :categories))
-               tu/comparable-query)))))
-
-(deftest ^:parallel implicit-join-field-parity-test
-  (let [program {:source     {:type "context" :ref "source"}
-                 :operations [["with-fields" [["field" (meta/id :orders :id)]
-                                              ["field" (meta/id :products :title)]]]
-                              ["order-by" ["asc" ["field" (meta/id :products :title)]]]]}
-        base-query (tu/query-for-table :orders)
-        expected (-> base-query
-                     (lib/with-fields [(tu/current-query-column base-query (meta/id :orders :id))
-                                       (tu/current-query-column base-query (meta/id :products :title))])
-                     (lib/order-by (tu/current-query-column base-query (meta/id :products :title)) :asc))]
-    (is (= (tu/comparable-query expected)
-           (-> (program/evaluate-program program meta/metadata-provider (tu/table-context :orders :products))
-               tu/comparable-query)))))
-
-(deftest ^:parallel implicit-join-field-outside-context-parity-test
-  (let [program {:source     {:type "context" :ref "source"}
-                 :operations [["with-fields" [["field" (meta/id :orders :id)]
-                                              ["field" (meta/id :products :title)]]]
-                              ["order-by" ["asc" ["field" (meta/id :products :title)]]]]}
-        base-query (tu/query-for-table :orders)
-        expected (-> base-query
-                     (lib/with-fields [(tu/current-query-column base-query (meta/id :orders :id))
-                                       (tu/current-query-column base-query (meta/id :products :title))])
-                     (lib/order-by (tu/current-query-column base-query (meta/id :products :title)) :asc))]
-    (is (= (tu/comparable-query expected)
-           (-> (program/evaluate-program program meta/metadata-provider (tu/table-context :orders))
-               tu/comparable-query)))))
-
-(deftest ^:parallel explicit-join-table-outside-context-parity-test
-  (let [program {:source     {:type "context" :ref "source"}
-                 :operations [["join"
-                               ["with-join-fields"
-                                ["with-join-conditions"
-                                 ["join-clause" ["table" (meta/id :categories)]]
-                                 [["=" ["field" (meta/id :venues :category-id)]
-                                   ["field" (meta/id :categories :id)]]]]
-                                [["field" (meta/id :categories :name)]]]]
-                              ["filter" ["=" ["field" (meta/id :categories :name)] "Mexican"]]]}
-        base-query (lib.tu/venues-query)
-        expected (as-> base-query query
-                   (lib/join query
-                             (-> (lib/join-clause (meta/table-metadata :categories))
-                                 (lib/with-join-conditions
-                                  [(lib/= (tu/current-query-column base-query (meta/id :venues :category-id))
-                                          (tu/current-query-column base-query (meta/id :categories :id)))])
-                                 (lib/with-join-fields
-                                   [(tu/current-query-column base-query (meta/id :categories :name))])))
-                   (lib/filter query (lib/= (tu/current-query-column query (meta/id :categories :name)) "Mexican")))]
-    (is (= (tu/comparable-query expected)
-           (-> (program/evaluate-program program meta/metadata-provider (tu/table-context :venues))
-               tu/comparable-query)))))
-
-(deftest ^:parallel redundant-explicit-join-to-implicit-related-table-test
-  (let [program {:source     {:type "context" :ref "source"}
-                 :operations [["join"
-                               ["with-join-conditions"
-                                ["join-clause" ["table" (meta/id :products)]]
-                                [["=" ["field" (meta/id :orders :product-id)]
-                                  ["field" (meta/id :products :id)]]]]]
-                              ["filter" ["=" ["field" (meta/id :products :title)] "Widget"]]
-                              ["aggregate" ["count"]]]}
-        context  (tu/table-context-with-join-edges :orders
-                                                   [:products]
-                                                   [(tu/join-edge :orders :product-id :products :id)])
-        base-query (tu/query-for-table :orders)
-        expected (-> base-query
-                     (lib/filter (lib/= (tu/current-query-column base-query (meta/id :products :title)) "Widget"))
-                     (lib/aggregate (lib/count)))]
-    (is (= (tu/comparable-query expected)
-           (-> (program/evaluate-program program meta/metadata-provider context)
-               tu/comparable-query)))))
-
-(deftest ^:parallel implicit-join-filter-field-uses-source-field-test
-  (let [program {:source     {:type "context" :ref "source"}
-                 :operations [["filter" ["contains" ["field" (meta/id :products :title)] "widget"]]
-                              ["aggregate" ["count"]]]}
-        base-query (tu/query-for-table :orders)
-        expected (-> base-query
-                     (lib/filter (lib/contains (tu/current-query-column base-query (meta/id :products :title)) "widget"))
-                     (lib/aggregate (lib/count)))]
-    (is (= (tu/comparable-query expected)
-           (-> (program/evaluate-program program meta/metadata-provider (tu/table-context :orders :products))
-               tu/comparable-query)))))
-
-(deftest ^:parallel explicit-join-with-field-selection-is-not-dropped-test
-  (let [program {:source     {:type "context" :ref "source"}
-                 :operations [["join"
-                               ["with-join-fields"
-                                ["with-join-conditions"
-                                 ["join-clause" ["table" (meta/id :categories)]]
-                                 [["=" ["field" (meta/id :venues :category-id)]
-                                   ["field" (meta/id :categories :id)]]]]
-                                [["field" (meta/id :categories :name)]]]]
-                              ["filter" ["=" ["field" (meta/id :categories :name)] "Mexican"]]]}
-        context  (tu/table-context-with-join-edges :venues
-                                                   [:categories]
-                                                   [(tu/join-edge :venues :category-id :categories :id)])
-        base-query (lib.tu/venues-query)
-        expected (as-> base-query query
-                   (lib/join query
-                             (-> (lib/join-clause (meta/table-metadata :categories))
-                                 (lib/with-join-conditions
-                                  [(lib/= (tu/current-query-column base-query (meta/id :venues :category-id))
-                                          (tu/current-query-column base-query (meta/id :categories :id)))])
-                                 (lib/with-join-fields
-                                   [(tu/current-query-column base-query (meta/id :categories :name))])))
-                   (lib/filter query (lib/= (tu/current-query-column query (meta/id :categories :name)) "Mexican")))]
-    (is (= (tu/comparable-query expected)
-           (-> (program/evaluate-program program meta/metadata-provider context)
-               tu/comparable-query)))))
-
-(deftest ^:parallel append-stage-explicit-related-join-falls-back-to-implicit-related-field-test
-  (let [implicit-program {:source     {:type "context" :ref "source"}
-                          :operations [["breakout" ["field" (meta/id :orders :product-id)]]
-                                       ["append-stage"]
-                                       ["breakout" ["field" (meta/id :products :title)]]]}
-        explicit-program {:source     {:type "context" :ref "source"}
-                          :operations [["breakout" ["field" (meta/id :orders :product-id)]]
-                                       ["append-stage"]
-                                       ["join"
-                                        ["with-join-conditions"
-                                         ["join-clause" ["table" (meta/id :products)]]
-                                         [["=" ["field" (meta/id :orders :product-id)]
-                                           ["field" (meta/id :products :id)]]]]]
-                                       ["breakout" ["field" (meta/id :products :title)]]]}
-        context          (tu/table-context :orders :products)]
-    (is (= (-> (program/evaluate-program implicit-program meta/metadata-provider context)
-               tu/comparable-query)
-           (-> (program/evaluate-program explicit-program meta/metadata-provider context)
-               tu/comparable-query)))))
-
-(deftest ^:parallel bare-order-by-field-parity-test
-  (let [program {:source     {:type "context" :ref "source"}
-                 :operations [["with-fields" [["field" (meta/id :orders :id)]
-                                              ["field" (meta/id :products :title)]]]
-                              ["order-by" ["field" (meta/id :products :title)]]]}
-        base-query (tu/query-for-table :orders)
-        expected (-> base-query
-                     (lib/with-fields [(tu/current-query-column base-query (meta/id :orders :id))
-                                       (tu/current-query-column base-query (meta/id :products :title))])
-                     (lib/order-by (tu/current-query-column base-query (meta/id :products :title)) :asc))]
-    (is (= (tu/comparable-query expected)
-           (-> (program/evaluate-program program meta/metadata-provider (tu/table-context :orders :products))
-               tu/comparable-query)))))
-
-(deftest ^:parallel append-stage-raw-field-falls-back-to-previous-stage-column-test
-  (let [program {:source     {:type "context" :ref "source"}
-                 :operations [["aggregate" ["avg" ["field" (meta/id :orders :total)]]]
-                              ["append-stage"]
-                              ["order-by" ["desc" ["field" (meta/id :orders :total)]]]]}
-        base-query (-> (tu/query-for-table :orders)
-                       (lib/aggregate (lib/avg (meta/field-metadata :orders :total)))
-                       (lib/append-stage))
-        expected   (lib/order-by base-query (tu/previous-stage-aggregation-column base-query 0) :desc)]
-    (is (= (tu/comparable-query expected)
-           (-> (program/evaluate-program program meta/metadata-provider (tu/table-context :orders))
-               tu/comparable-query)))))
-
-(deftest ^:parallel aggregate-order-by-raw-field-falls-back-to-aggregation-column-test
-  (let [program {:source     {:type "context" :ref "source"}
-                 :operations [["aggregate" ["min" ["field" (meta/id :orders :total)]]]
-                              ["order-by" ["field" (meta/id :orders :total)]]]}
-        base-query (-> (tu/query-for-table :orders)
-                       (lib/aggregate (lib/min (meta/field-metadata :orders :total))))
-        expected   (lib/order-by base-query (tu/current-aggregation-orderable base-query 0))]
-    (is (= (tu/comparable-query expected)
-           (-> (program/evaluate-program program meta/metadata-provider (tu/table-context :orders))
-               tu/comparable-query)))))
-
-(deftest ^:parallel grouped-raw-field-order-by-falls-back-to-aggregation-column-test
-  (let [program {:source     {:type "context" :ref "source"}
-                 :operations [["aggregate" ["avg" ["field" (meta/id :orders :total)]]]
-                              ["breakout" ["field" (meta/id :orders :product-id)]]
-                              ["order-by" ["desc" ["field" (meta/id :orders :total)]]]]}
-        base-query (-> (tu/query-for-table :orders)
-                       (lib/aggregate (lib/avg (meta/field-metadata :orders :total)))
-                       (lib/breakout (meta/field-metadata :orders :product-id)))
-        expected   (lib/order-by base-query (tu/current-aggregation-orderable base-query 0) :desc)]
-    (is (= (tu/comparable-query expected)
-           (-> (program/evaluate-program program meta/metadata-provider (tu/table-context :orders))
-               tu/comparable-query)))))
-
-(deftest ^:parallel append-stage-aggregate-raw-field-falls-back-to-previous-stage-column-test
-  (let [program {:source     {:type "context" :ref "source"}
-                 :operations [["aggregate" ["avg" ["field" (meta/id :orders :total)]]]
-                              ["append-stage"]
-                              ["aggregate" ["avg" ["field" (meta/id :orders :total)]]]]}
-        base-query (-> (tu/query-for-table :orders)
-                       (lib/aggregate (lib/avg (meta/field-metadata :orders :total)))
-                       (lib/append-stage))
-        expected   (lib/aggregate base-query (lib/avg (tu/previous-stage-aggregation-column base-query 0)))]
-    (is (= (tu/comparable-query expected)
-           (-> (program/evaluate-program program meta/metadata-provider (tu/table-context :orders))
-               tu/comparable-query)))))
-
-(deftest ^:parallel append-stage-filter-on-unavailable-raw-field-is-rejected-test
-  (let [program {:source     {:type "context" :ref "source"}
-                 :operations [["aggregate" ["count"]]
-                              ["append-stage"]
-                              ["filter" [">" ["field" (meta/id :orders :total)] 10]]]}
-        error   (try
-                  (program/evaluate-program program meta/metadata-provider (tu/table-context :orders))
-                  nil
-                  (catch clojure.lang.ExceptionInfo e
-                    e))]
-    (is error)
-    (is (= :invalid-generated-program (:error (ex-data error))))
-    (is (re-find #"not available in the current query stage"
-                 (:details (ex-data error))))))
-
-(deftest ^:parallel relative-date-alias-parity-test
-  (let [program {:source     {:type "context" :ref "source"}
-                 :operations [["filter" [">="
-                                         ["field" (meta/id :orders :created-at)]
-                                         ["relative-date" -90 "day"]]]
-                              ["limit" 10]]}
-        expected (-> (tu/query-for-table :orders)
-                     (lib/filter (lib/>= (meta/field-metadata :orders :created-at)
-                                         (lib/relative-datetime -90 :day)))
-                     (lib/limit 10))]
-    (is (= (tu/comparable-query expected)
-           (-> (program/evaluate-program program meta/metadata-provider (tu/table-context :orders))
-               tu/comparable-query)))))
-
-(deftest ^:parallel aggregate-over-aggregation-ref-source-program-parity-test
-  (let [program {:source     {:type "program"
-                              :program {:source     {:type "context" :ref "source"}
-                                        :operations [["aggregate" ["sum" ["field" (meta/id :orders :total)]]]
-                                                     ["breakout" ["field" (meta/id :orders :created-at)]]]}}
-                 :operations [["aggregate" ["stddev" ["aggregation-ref" 0]]]]}
-        base-query (-> (tu/query-for-table :orders)
-                       (lib/aggregate (lib/sum (meta/field-metadata :orders :total)))
-                       (lib/breakout (meta/field-metadata :orders :created-at))
-                       (lib/append-stage))
-        expected   (lib/aggregate base-query (lib/stddev (tu/previous-stage-aggregation-column base-query 0)))]
-    (is (= (tu/comparable-query expected)
-           (-> (program/evaluate-program program meta/metadata-provider (tu/table-context :orders))
-               tu/comparable-query)))))
-
-(deftest ^:parallel repaired-program-evaluation-parity-test
-  (let [program {:source     {:type "context" :ref "source"}
-                 :operations [["filter" ["is-not-null" ["field" (meta/id :orders :discount)]]]
-                              ["filter" ["=" ["field" (meta/id :orders :product-id)] [1 2]]]
-                              ["aggregate" ["count-if" ["=" ["field" (meta/id :orders :product-id)] 1]]
-                               ["variance" ["field" (meta/id :orders :total)]]]]}
-        expected (-> (tu/query-for-table :orders)
-                     (lib/filter (lib/not-null (meta/field-metadata :orders :discount)))
-                     (lib/filter (lib/in (meta/field-metadata :orders :product-id) 1 2))
-                     (lib/aggregate (lib/count-where (lib/= (meta/field-metadata :orders :product-id) 1)))
-                     (lib/aggregate (lib/var (meta/field-metadata :orders :total))))]
-    (is (= (tu/comparable-query expected)
-           (-> (program/evaluate-program program meta/metadata-provider (tu/table-context :orders))
-               tu/comparable-query)))))
-
-(deftest ^:parallel datetime-diff-parity-test
-  (let [program {:source     {:type "context" :ref "source"}
-                 :operations [["expression" "Days Since Order"
-                               ["datetime-diff" ["now"]
-                                ["field" (meta/id :orders :created-at)] "day"]]
-                              ["with-fields" [["field" (meta/id :orders :id)]
-                                              ["expression-ref" "Days Since Order"]]]
-                              ["limit" 10]]}
-        expected (-> (tu/query-for-table :orders)
-                     (lib/expression "Days Since Order"
-                                     (lib/expression-clause :datetime-diff
-                                                            [(lib/now)
-                                                             (meta/field-metadata :orders :created-at)
-                                                             :day]
-                                                            nil))
-                     (as-> query
-                           (lib/with-fields query [(meta/field-metadata :orders :id)
-                                                   (lib/expression-ref query "Days Since Order")]))
-                     (lib/limit 10))]
-    (is (= (tu/comparable-query expected)
-           (-> (program/evaluate-program program meta/metadata-provider (tu/table-context :orders))
-               tu/comparable-query)))))
-
-(deftest ^:parallel later-stage-aggregation-ref-expression-parity-test
-  (let [program {:source     {:type "context" :ref "source"}
-                 :operations [["aggregate" ["sum" ["field" (meta/id :orders :total)]]]
-                              ["breakout" ["field" (meta/id :orders :product-id)]]
-                              ["append-stage"]
-                              ["expression" "Revenue K" ["/" ["aggregation-ref" 0] 1000]]
-                              ["with-fields" [["field" (meta/id :orders :product-id)]
-                                              ["expression-ref" "Revenue K"]]]
-                              ["order-by" ["desc" ["expression-ref" "Revenue K"]]]]}
-        base-query (-> (tu/query-for-table :orders)
-                       (lib/aggregate (lib/sum (meta/field-metadata :orders :total)))
-                       (lib/breakout (meta/field-metadata :orders :product-id))
-                       lib/append-stage)
-        agg-column (tu/previous-stage-aggregation-column base-query 0)
-        expected   (-> base-query
-                       (lib/expression "Revenue K" (lib// agg-column 1000))
-                       (as-> query
-                             (-> query
-                                 (lib/with-fields [(tu/current-query-column query (meta/id :orders :product-id))
-                                                   (lib/expression-ref query "Revenue K")])
-                                 (lib/order-by (lib/expression-ref query "Revenue K") :desc))))]
-    (is (= (tu/comparable-query expected)
-           (-> (program/evaluate-program program meta/metadata-provider (tu/table-context :orders))
-               tu/comparable-query)))))
-
-(deftest ^:parallel expression-ref-falls-back-to-previous-stage-column-after-append-stage-test
-  (let [program {:source     {:type "context" :ref "source"}
-                 :operations [["aggregate" ["sum" ["field" (meta/id :orders :total)]]]
-                              ["aggregate" ["sum" ["field" (meta/id :orders :discount)]]]
-                              ["append-stage"]
-                              ["expression" "Variance" ["-" ["aggregation-ref" 0] ["aggregation-ref" 1]]]
-                              ["append-stage"]
-                              ["aggregate" ["avg" ["expression-ref" "Variance"]]]]}
-        base-query (-> (tu/query-for-table :orders)
-                       (lib/aggregate (lib/sum (meta/field-metadata :orders :total)))
-                       (lib/aggregate (lib/sum (meta/field-metadata :orders :discount)))
-                       lib/append-stage)
-        variance-stage (lib/expression base-query
-                                       "Variance"
-                                       (lib/- (tu/previous-stage-aggregation-column base-query 0)
-                                              (tu/previous-stage-aggregation-column base-query 1)))
-        appended (lib/append-stage variance-stage)
-        expected (lib/aggregate appended
-                                (lib/avg (tu/current-query-column-by-name appended "Variance")))]
-    (is (= (tu/comparable-query expected)
-           (-> (program/evaluate-program program meta/metadata-provider (tu/table-context :orders))
-               tu/comparable-query)))))
-
-(deftest ^:parallel aggregated-with-fields-raw-fields-become-breakouts-test
-  (let [program {:source     {:type "context" :ref "source"}
-                 :operations [["aggregate" ["avg" ["field" (meta/id :orders :total)]]]
-                              ["with-fields" [["field" (meta/id :orders :product-id)]
-                                              ["field" (meta/id :orders :id)]]]]}
-        expected (-> (tu/query-for-table :orders)
-                     (lib/aggregate (lib/avg (meta/field-metadata :orders :total)))
-                     (lib/breakout (meta/field-metadata :orders :product-id))
-                     (lib/breakout (meta/field-metadata :orders :id)))]
-    (is (= (tu/comparable-query expected)
-           (-> (program/evaluate-program program meta/metadata-provider (tu/table-context :orders))
-               tu/comparable-query)))))
-
-(deftest ^:parallel aggregated-with-fields-aggregation-ref-selection-parity-test
-  (let [program {:source     {:type "context" :ref "source"}
-                 :operations [["aggregate" ["count"]]
-                              ["with-fields" [["aggregation-ref" 0]]]]}
-        base-query (-> (tu/query-for-table :orders)
-                       (lib/aggregate (lib/count)))
-        appended   (lib/append-stage base-query)
-        expected   (lib/with-fields appended [(tu/previous-stage-aggregation-column appended 0)])]
-    (is (= (tu/comparable-query expected)
-           (-> (program/evaluate-program program meta/metadata-provider (tu/table-context :orders))
-               tu/comparable-query)))))
-
-(deftest ^:parallel nested-source-program-parity-test
-  (let [program {:source     {:type "program"
-                              :program {:source     {:type "context" :ref "source"}
-                                        :operations [["filter" ["<" ["field" (meta/id :orders :total)] 100]]
-                                                     ["with-fields" [["field" (meta/id :orders :id)]
-                                                                     ["field" (meta/id :orders :total)]]]]}}
-                 :operations [["limit" 5]]}
-        expected (-> (tu/query-for-table :orders)
-                     (lib/filter (lib/< (meta/field-metadata :orders :total) 100))
-                     (lib/with-fields [(meta/field-metadata :orders :id)
-                                       (meta/field-metadata :orders :total)])
-                     (lib/limit 5))]
-    (is (= (tu/comparable-query expected)
-           (-> (program/evaluate-program program meta/metadata-provider (tu/table-context :orders))
-               tu/comparable-query)))))
-
-(deftest ^:parallel nested-join-source-program-parity-test
-  (let [nested-program {:source     {:type "table" :id (meta/id :products)}
-                        :operations [["with-fields" [["field" (meta/id :products :id)]
-                                                     ["field" (meta/id :products :title)]]]]}
-        program        {:source     {:type "context" :ref "source"}
-                        :operations [["join"
-                                      ["with-join-conditions"
-                                       ["join-clause" {:type "program" :program nested-program}]
-                                       [["=" ["field" (meta/id :orders :product-id)]
-                                         ["field" (meta/id :products :id)]]]]]]}
-        joined-query   (-> (tu/query-for-table :products)
-                           (lib/with-fields [(meta/field-metadata :products :id)
-                                             (meta/field-metadata :products :title)]))
-        base-query     (tu/query-for-table :orders)
-        expected       (-> base-query
-                           (lib/join (-> (lib/join-clause joined-query)
-                                         (lib/with-join-conditions
-                                          [(lib/= (tu/current-query-column base-query (meta/id :orders :product-id))
-                                                  (tu/current-query-column base-query (meta/id :products :id)))]))))]
-    (is (= (tu/comparable-query expected)
-           (-> (program/evaluate-program program meta/metadata-provider (tu/table-context :orders :products))
-               tu/comparable-query)))))
-
-(deftest ^:parallel with-page-parity-test
-  (let [program {:source     {:type "context" :ref "source"}
-                 :operations [["with-page" {:page 2 :items 25}]]}
-        expected (lib/with-page (tu/query-for-table :orders) {:page 2 :items 25})]
-    (is (= (tu/comparable-query expected)
-           (-> (program/evaluate-program program meta/metadata-provider (tu/table-context :orders))
-               tu/comparable-query)))))
-
-(deftest ^:parallel stage-ops-parity-test
-  (let [program {:source     {:type "context" :ref "source"}
-                 :operations [["append-stage"]
-                              ["drop-empty-stages"]
-                              ["append-stage"]
-                              ["with-page" {:page 2 :items 25}]
-                              ["drop-stage"]]}
-        expected (-> (tu/query-for-table :orders)
-                     lib/append-stage
-                     lib/drop-empty-stages
-                     lib/append-stage
-                     (lib/with-page {:page 2 :items 25})
-                     lib/drop-stage)]
-    (is (= (tu/comparable-query expected)
-           (-> (program/evaluate-program program meta/metadata-provider (tu/table-context :orders))
-               tu/comparable-query)))))
-
-(deftest ^:parallel invalid-case-default-object-test
-  (let [total-field-id (meta/id :orders :total)
-        case-expr      ["case" [[[">" ["field" total-field-id] 100]
-                                 ["field" total-field-id]]]
-                        {:default nil}]
-        program        {:source     {:type "context" :ref "source"}
-                        :operations [["expression" "Maybe Revenue" case-expr]]}
-        error   (try
-                  (program/evaluate-program program meta/metadata-provider (tu/table-context :orders))
-                  nil
-                  (catch clojure.lang.ExceptionInfo e
-                    e))]
-    (is error)
-    (is (= :invalid-generated-program (:error (ex-data error))))
-    (is (= "case" (:operator (ex-data error))))
-    (is (re-find #"fallback value itself as the optional third argument"
-                 (:details (ex-data error))))
-    (is (re-find #"Omit the third argument instead of using null"
-                 (:details (ex-data error))))))
-
-(deftest ^:parallel invalid-bare-field-id-in-aggregation-test
-  (let [program {:source     {:type "context" :ref "source"}
-                 :operations [["aggregate" ["sum" (meta/id :orders :total)]]]}
-        error   (try
-                  (program/evaluate-program program meta/metadata-provider (tu/table-context :orders))
-                  nil
-                  (catch clojure.lang.ExceptionInfo e
-                    e))]
-    (is error)
-    (is (= :invalid-generated-program (:error (ex-data error))))
-    (is (= "sum" (:operator (ex-data error))))
-    (is (= :field-wrapper (:retry-category (ex-data error))))
-    (is (re-find #"Wrap field ids with `\[\"field\", id\]` inside `sum`"
-                 (:details (ex-data error))))))
-
-(deftest ^:parallel invalid-unknown-ref-helper-test
-  (let [program {:source     {:type "context" :ref "source"}
-                 :operations [["expression" "Percentage Share" ["sum-ref"]]]}
-        error   (try
-                  (program/evaluate-program program meta/metadata-provider (tu/table-context :orders))
-                  nil
-                  (catch clojure.lang.ExceptionInfo e
-                    e))]
-    (is error)
-    (is (= :invalid-generated-program (:error (ex-data error))))
-    (is (= "sum-ref" (:operator (ex-data error))))
-    (is (re-find #"unknown helper reference"
-                 (:details (ex-data error))))))
-
-(deftest ^:parallel conditional-average-case-parity-test
-  (let [total-field-id (meta/id :orders :total)
-        case-expr      ["case" [[[">" ["field" total-field-id] 100]
-                                 ["field" total-field-id]]]]
-        program        {:source     {:type "context" :ref "source"}
-                        :operations [["expression" "Large Total" case-expr]
-                                     ["aggregate" ["avg" ["expression-ref" "Large Total"]]]]}
-        expected (-> (tu/query-for-table :orders)
-                     (lib/expression "Large Total"
-                                     (lib/case [[(lib/> (meta/field-metadata :orders :total) 100)
-                                                 (meta/field-metadata :orders :total)]]))
-                     (as-> query
-                           (lib/aggregate query (lib/avg (lib/expression-ref query "Large Total")))))]
-    (is (= (tu/comparable-query expected)
-           (-> (program/evaluate-program program meta/metadata-provider (tu/table-context :orders))
-               tu/comparable-query)))))
-
-(deftest ^:parallel offset-expression-defers-to-lib-core-test
-  (let [program {:source     {:type "context" :ref "source"}
-                 :operations [["aggregate" ["sum" ["field" (meta/id :orders :total)]]]
-                              ["breakout" ["with-temporal-bucket" ["field" (meta/id :orders :created-at)] "month"]]
-                              ["order-by" ["asc"
-                                           ["with-temporal-bucket"
-                                            ["field" (meta/id :orders :created-at)] "month"]]]
-                              ["append-stage"]
-                              ["expression" "Prev Revenue" ["offset" ["aggregation-ref" 0] -1]]]}
-        base-query (-> (tu/query-for-table :orders)
-                       (lib/aggregate (lib/sum (meta/field-metadata :orders :total)))
-                       (lib/breakout (lib/with-temporal-bucket (meta/field-metadata :orders :created-at) :month))
-                       (lib/order-by (lib/with-temporal-bucket (meta/field-metadata :orders :created-at) :month) :asc)
-                       lib/append-stage)
-        agg-column (tu/previous-stage-aggregation-column base-query 0)
-        lib-error  (try
-                     (-> base-query
-                         (lib/expression "Prev Revenue" (lib/offset agg-column -1)))
-                     nil
-                     (catch Exception e
-                       e))
-        program-error (try
-                        (program/evaluate-program program meta/metadata-provider (tu/table-context :orders))
-                        nil
-                        (catch clojure.lang.ExceptionInfo e
-                          e))]
-    (testing "structured evaluation should defer to lib.core instead of rejecting this pattern preemptively"
-      (is lib-error)
-      (is program-error)
-      (is (re-find #"non-aggregation expression" (ex-message lib-error)))
-      (is (re-find #"non-aggregation expression" (:details (ex-data program-error))))
-      (is (not (re-find #"named expressions must be non-aggregation only"
-                        (:details (ex-data program-error)))))
-      (is (= :invalid-generated-program (:error (ex-data program-error)))))))
-
-(deftest ^:parallel field-options-temporal-unit-parity-test
-  (testing "`[\"field\" id {\"temporal-unit\" ...}]` is equivalent to `with-temporal-bucket`"
-    (let [created-at (meta/field-metadata :orders :created-at)
-          program    {:source     {:type "context" :ref "source"}
-                      :operations [["aggregate" ["sum" ["field" (meta/id :orders :total)]]]
-                                   ["breakout" ["field" (meta/id :orders :created-at)
-                                                {"temporal-unit" "month"}]]
-                                   ["order-by" ["field" (meta/id :orders :created-at)
-                                                {"temporal-unit" "month"}] "asc"]]}
-          expected   (-> (tu/query-for-table :orders)
-                         (lib/aggregate (lib/sum (meta/field-metadata :orders :total)))
-                         (lib/breakout (lib/with-temporal-bucket created-at :month))
-                         (lib/order-by (lib/with-temporal-bucket created-at :month) :asc))]
-      (is (= (tu/comparable-query expected)
-             (-> (program/evaluate-program program meta/metadata-provider (tu/table-context :orders))
-                 tu/comparable-query))))))
-
-(deftest ^:parallel field-options-binning-parity-test
-  (testing "`[\"field\" id {\"binning\" ...}]` is equivalent to `with-binning`"
-    (let [total    (meta/field-metadata :orders :total)
-          binning  {:strategy :num-bins :num-bins 10}
-          program  {:source     {:type "context" :ref "source"}
-                    :operations [["aggregate" ["count"]]
-                                 ["breakout" ["field" (meta/id :orders :total)
-                                              {"binning" {"strategy" "num-bins"
-                                                          "num-bins" 10}}]]]}
-          expected (-> (tu/query-for-table :orders)
-                       (lib/aggregate (lib/count))
-                       (lib/breakout (lib/with-binning total binning)))]
-      (is (= (tu/comparable-query expected)
-             (-> (program/evaluate-program program meta/metadata-provider (tu/table-context :orders))
-                 tu/comparable-query))))))
-
-(deftest ^:parallel field-options-combined-parity-test
-  (testing "`[\"field\" id {temporal-unit ... binning ...}]` composes both wrappers"
-    (let [created-at (meta/field-metadata :orders :created-at)
-          program    {:source     {:type "context" :ref "source"}
-                      :operations [["aggregate" ["count"]]
-                                   ["breakout" ["field" (meta/id :orders :created-at)
-                                                {"temporal-unit" "month"
-                                                 "binning"       {"strategy" "default"}}]]]}
-          expected   (-> (tu/query-for-table :orders)
-                         (lib/aggregate (lib/count))
-                         (lib/breakout (-> created-at
-                                           (lib/with-temporal-bucket :month)
-                                           (lib/with-binning {:strategy :default}))))]
-      (is (= (tu/comparable-query expected)
-             (-> (program/evaluate-program program meta/metadata-provider (tu/table-context :orders))
-                 tu/comparable-query))))))
-
-(deftest ^:parallel field-like-map-breakout-and-order-by-vector-wrapper-parity-test
-  (testing "field-like maps inside breakout and a vector-wrapped order-by yield canonical MBQL"
-    (let [created-at (meta/field-metadata :orders :created-at)
-          program    {:source     {:type "context" :ref "source"}
-                      :operations [["aggregate" ["count"]]
-                                   ["breakout" {"field_id"        (meta/id :orders :created-at)
-                                                "temporal_bucket" "month"}]
-                                   ["order-by" [["asc" {"field_id"        (meta/id :orders :created-at)
-                                                        "temporal_bucket" "month"}]]]]}
-          expected   (-> (tu/query-for-table :orders)
-                         (lib/aggregate (lib/count))
-                         (lib/breakout (lib/with-temporal-bucket created-at :month))
-                         (lib/order-by (lib/with-temporal-bucket created-at :month) :asc))]
-      (is (= (tu/comparable-query expected)
-             (-> (program/evaluate-program program meta/metadata-provider (tu/table-context :orders))
-                 tu/comparable-query))))))
-
-(deftest ^:parallel field-like-map-order-by-with-direction-key-parity-test
-  (testing "an order-by field-like map with a `direction` key yields canonical MBQL"
-    (let [created-at (meta/field-metadata :orders :created-at)
-          program    {:source     {:type "context" :ref "source"}
-                      :operations [["aggregate" ["count"]]
-                                   ["breakout" {"field_id"        (meta/id :orders :created-at)
-                                                "temporal_bucket" "month"}]
-                                   ["order-by" {"field_id"        (meta/id :orders :created-at)
-                                                "temporal_bucket" "month"
-                                                "direction"       "asc"}]]}
-          expected   (-> (tu/query-for-table :orders)
-                         (lib/aggregate (lib/count))
-                         (lib/breakout (lib/with-temporal-bucket created-at :month))
-                         (lib/order-by (lib/with-temporal-bucket created-at :month) :asc))]
-      (is (= (tu/comparable-query expected)
-             (-> (program/evaluate-program program meta/metadata-provider (tu/table-context :orders))
-                 tu/comparable-query))))))

test/metabase/agent_lib/fuzz_test.clj

@@ -1,253 +0,0 @@
-(ns metabase.agent-lib.fuzz-test
-  "Property-based tests for agent-lib repair, validation, and error handling.
-  Generators produce random program-like structures (both valid and invalid)
-  to verify that the pipeline never crashes — it either succeeds or throws
-  a structured `:invalid-generated-program` error."
-  (:require
-   [clojure.test :refer [deftest is testing]]
-   [clojure.test.check.clojure-test :refer [defspec]]
-   [clojure.test.check.generators :as gen]
-   [clojure.test.check.properties :as prop]
-   [metabase.agent-lib.repair :as repair]
-   [metabase.agent-lib.schema :as schema]
-   [metabase.agent-lib.validate :as validate]
-   [metabase.lib.test-metadata :as meta]))
-
-(set! *warn-on-reflection* true)
-
-;; --- Generators ---
-
-(def ^:private gen-field-id
-  "Generator for field ids from test metadata."
-  (gen/elements [(meta/id :orders :id)
-                 (meta/id :orders :total)
-                 (meta/id :orders :product-id)
-                 (meta/id :orders :user-id)
-                 (meta/id :orders :created-at)
-                 (meta/id :products :title)
-                 (meta/id :products :category)
-                 99999]))
-
-(def ^:private gen-table-id
-  (gen/elements [(meta/id :orders)
-                 (meta/id :products)
-                 (meta/id :people)
-                 99999]))
-
-(def ^:private gen-op-name
-  "Generator for operator names — mix of valid and invalid."
-  (gen/elements ["count" "sum" "avg" "min" "max" "distinct" "field"
-                 "=" "!=" "<" ">" "in" "between" "contains"
-                 "expression-ref" "aggregation-ref"
-                 "case" "coalesce" "+" "-" "*" "/"
-                 "with-temporal-bucket" "relative-datetime" "now"
-                 ;; invalid operators to exercise error paths
-                 "bogus" "YOLO" ""]))
-
-(def ^:private gen-top-level-op-name
-  (gen/elements ["filter" "aggregate" "breakout" "order-by"
-                 "expression" "with-fields" "limit" "join"
-                 "append-stage"
-                 ;; invalid to exercise error paths
-                 "bogus-top" ""]))
-
-(def ^:private gen-scalar
-  (gen/one-of [gen/small-integer
-               (gen/double* {:NaN? false :infinite? false})
-               gen/boolean
-               (gen/not-empty gen/string-alphanumeric)
-               (gen/return nil)]))
-
-(defn- gen-node
-  "Recursive generator for program nodes. `depth` limits nesting."
-  [depth]
-  (if (zero? depth)
-    gen-scalar
-    (gen/one-of
-     [gen-scalar
-      ;; field ref
-      (gen/fmap (fn [id] ["field" id]) gen-field-id)
-      ;; nested operator
-      (gen/fmap (fn [[op args]] (into [op] args))
-                (gen/tuple gen-op-name
-                           (gen/vector (gen-node (dec depth)) 0 3)))
-      ;; vector of nodes
-      (gen/vector (gen-node (dec depth)) 0 3)])))
-
-(def ^:private gen-operation
-  "Generator for top-level operations."
-  (gen/fmap (fn [[op args]] (into [op] args))
-            (gen/tuple gen-top-level-op-name
-                       (gen/vector (gen-node 3) 0 4))))
-
-(def ^:private gen-source
-  "Generator for program sources."
-  (gen/one-of
-   [(gen/return {:type "context" :ref "source"})
-    (gen/fmap (fn [id] {:type "table" :id id}) gen-table-id)
-    (gen/fmap (fn [id] {:type "metric" :id id}) (gen/elements [1 42 99999]))
-    (gen/fmap (fn [id] {:type "card" :id id}) (gen/elements [1 99999]))
-    ;; invalid source types
-    (gen/return {:type "nonexistent"})
-    (gen/return {})
-    (gen/return nil)]))
-
-(def ^:private gen-program
-  "Generator for program-like structures."
-  (gen/fmap (fn [[source ops]] {:source source :operations (vec ops)})
-            (gen/tuple gen-source
-                       (gen/vector gen-operation 0 6))))
-
-(def ^:private gen-garbage
-  "Generator for completely random values — not program-shaped at all."
-  (gen/one-of
-   [gen/small-integer
-    gen/string-alphanumeric
-    gen/boolean
-    (gen/return nil)
-    (gen/vector gen/small-integer 0 5)
-    (gen/map gen/keyword gen/small-integer {:max-elements 3})]))
-
-;; --- Helpers ---
-
-(defn- orders-context
-  []
-  {:source-entity       {:model "table" :id (meta/id :orders)
-                         :columns (mapv (fn [f] {:id (meta/id :orders f)})
-                                        (meta/fields :orders))}
-   :referenced-entities []
-   :surrounding-tables  [{:id (meta/id :products)}
-                         {:id (meta/id :people)}]
-   :measure-ids         []})
-
-(defn- structured-error?
-  "True when `e` is the standard `:invalid-generated-program` error
-  or a Malli input validation error (from mu/defn schemas)."
-  [e]
-  (let [data (ex-data e)]
-    (or (= :invalid-generated-program (:error data))
-        (= :metabase.util.malli.fn/invalid-input (:type data))
-        (= :metabase.util.malli.fn/invalid-output (:type data)))))
-
-;; --- Properties ---
-
-(defspec repair-never-crashes 200
-  (prop/for-all [program gen-program]
-    (let [result (try
-                   (repair/repair-program program)
-                   ::ok
-                   (catch Exception _
-                     ::ok))]
-      (= ::ok result))))
-
-(defspec repair-returns-map-for-map-input 200
-  (prop/for-all [program gen-program]
-    (map? (repair/repair-program program))))
-
-(defspec repair-preserves-source-key 200
-  (prop/for-all [program gen-program]
-    (contains? (repair/repair-program program) :source)))
-
-(defspec repair-preserves-operations-key 200
-  (prop/for-all [program gen-program]
-    (contains? (repair/repair-program program) :operations)))
-
-(defspec repair-is-idempotent 200
-  (prop/for-all [program gen-program]
-    (= (repair/repair-program program)
-       (repair/repair-program (repair/repair-program program)))))
-
-(defspec validation-never-crashes-with-unstructured-error 200
-  (prop/for-all [program gen-program]
-    (try
-      (validate/validated-program program (orders-context))
-      true
-      (catch clojure.lang.ExceptionInfo e
-        (structured-error? e))
-      (catch Exception _
-        ;; Non-ExceptionInfo exceptions are acceptable from downstream
-        ;; libs (e.g. Malli schema validation); the key property is
-        ;; agent-lib code itself doesn't throw raw exceptions.
-        true))))
-
-(defspec garbage-input-to-repair-never-crashes 100
-  (prop/for-all [garbage gen-garbage]
-    (try
-      (repair/repair-program garbage)
-      true
-      (catch Exception _
-        true))))
-
-(defspec garbage-input-to-validation-never-crashes-with-unstructured-error 100
-  (prop/for-all [garbage gen-garbage]
-    (try
-      (validate/validated-program garbage (orders-context))
-      true
-      (catch clojure.lang.ExceptionInfo e
-        (structured-error? e))
-      (catch Exception _
-        true))))
-
-(defspec schema-validation-rejects-or-accepts-cleanly 200
-  (prop/for-all [program gen-program]
-    (let [repaired (repair/repair-program program)]
-      (try
-        (schema/validated-structure repaired)
-        true
-        (catch clojure.lang.ExceptionInfo e
-          (structured-error? e))))))
-
-;; --- Deterministic edge-case tests ---
-
-(deftest ^:parallel deeply-nested-operations-do-not-overflow-test
-  (let [deep-op (reduce (fn [inner _] ["+" inner 1])
-                        ["field" (meta/id :orders :total)]
-                        (range 50))
-        program {:source     {:type "table" :id (meta/id :orders)}
-                 :operations [["filter" [">" deep-op 0]]]}]
-    (testing "repair handles deeply nested operations"
-      (is (map? (repair/repair-program program))))
-    (testing "validation rejects or accepts without crashing"
-      (try
-        (validate/validated-program program (orders-context))
-        (is true)
-        (catch clojure.lang.ExceptionInfo e
-          (is (structured-error? e)))))))
-
-(deftest ^:parallel empty-and-minimal-programs-test
-  (testing "empty operations"
-    (let [program {:source {:type "table" :id (meta/id :orders)}
-                   :operations []}]
-      (is (= program (validate/validated-program program (orders-context))))))
-  (testing "nil operations repairs to empty"
-    (let [repaired (repair/repair-program {:source {:type "context" :ref "source"}
-                                           :operations nil})]
-      (is (vector? (:operations repaired)))))
-  (testing "missing operations key repairs gracefully"
-    (let [repaired (repair/repair-program {:source {:type "context" :ref "source"}})]
-      (is (map? repaired)))))
-
-(deftest ^:parallel many-operations-test
-  (let [ops (vec (repeat 50 ["aggregate" ["count"]]))
-        program {:source {:type "table" :id (meta/id :orders)}
-                 :operations ops}]
-    (testing "programs exceeding max-operations are rejected"
-      (try
-        (validate/validated-program program (orders-context))
-        (is false "expected rejection")
-        (catch clojure.lang.ExceptionInfo e
-          (is (structured-error? e))
-          (is (re-find #"maximum operation count"
-                       (:details (ex-data e)))))))))
-
-(deftest ^:parallel repair-idempotence-degenerate-helpers-test
-  (testing "degenerate operators inside `case` do not grow extra args on each repair pass (regression)"
-    (doseq [op ["in" "not-in" "=" "!=" "is" "is-not"
-                "between" "with-temporal-bucket" "percentile" "with-join-conditions"]]
-      (testing op
-        (let [program  {:source     {:type "context" :ref "source"}
-                        :operations [["filter" ["count" ["case" [op]]]]]}
-              once     (repair/repair-program program)
-              twice    (repair/repair-program once)]
-          (is (= once twice)
-              (str "repair of `[\"case\" [\"" op "\"]]` should be idempotent")))))))

test/metabase/agent_lib/guards_test.clj

@@ -1,199 +0,0 @@
-(ns metabase.agent-lib.guards-test
-  "Tests for security and robustness guards added to agent-lib."
-  (:require
-   [clojure.test :refer [deftest is testing]]
-   [metabase.agent-lib.common.errors :as errors]
-   [metabase.agent-lib.repair.context.passes :as passes]
-   [metabase.agent-lib.repair.stages :as repair.stages]
-   [metabase.agent-lib.runtime :as runtime]
-   [metabase.agent-lib.validate :as validate]
-   [metabase.agent-lib.validate.context :as validate.context]
-   [metabase.agent-lib.validate.walker :as walker]
-   [metabase.lib.test-metadata :as meta]))
-
-(set! *warn-on-reflection* true)
-
-(defn- orders-context
-  "Minimal context with only the orders table allowed."
-  []
-  {:source-entity       {:model "table" :id (meta/id :orders)
-                         :columns (mapv (fn [f] {:id (meta/id :orders f)})
-                                        (meta/fields :orders))}
-   :referenced-entities []
-   :surrounding-tables  []
-   :measure-ids         []})
-
-(defn- ex-details
-  "Extract the :details string from an ExceptionInfo thrown by invalid-program!."
-  [^Exception e]
-  (:details (ex-data e)))
-
-;; --- P0: Table source validation ---
-
-(deftest validate-source-rejects-table-not-in-context-test
-  (let [allowed-ids (validate.context/context-allowed-ids (orders-context))]
-    (testing "table in context passes"
-      (is (= {}
-             (validate.context/validate-source!
-              (fn [& _] {})
-              allowed-ids
-              [:source]
-              {:type "table" :id (meta/id :orders)}
-              0
-              {}))))
-    (testing "table NOT in context is rejected"
-      (try
-        (validate.context/validate-source!
-         (fn [& _] {})
-         allowed-ids
-         [:source]
-         {:type "table" :id 99999}
-         0
-         {})
-        (is false "expected unknown table to be rejected")
-        (catch clojure.lang.ExceptionInfo e
-          (is (= :invalid-generated-program (:error (ex-data e))))
-          (is (re-find #"not available in the provided context" (ex-details e))))))))
-
-(deftest ^:parallel validated-program-rejects-unknown-table-source-test
-  (let [program {:source     {:type "table" :id 99999}
-                 :operations []}
-        context (orders-context)]
-    (try
-      (validate/validated-program program context)
-      (is false "expected unknown table to be rejected")
-      (catch clojure.lang.ExceptionInfo e
-        (is (= :invalid-generated-program (:error (ex-data e))))
-        (is (re-find #"not available in the provided context" (ex-details e)))))))
-
-;; --- P1: Runtime nil-guard ---
-
-(deftest ^:parallel helper-fn-rejects-nil-binding-test
-  (let [rt {:bindings {'known identity 'nil-bound nil}}]
-    (testing "non-nil binding returns the function"
-      (is (fn? (runtime/helper-fn rt 'known))))
-    (testing "nil binding throws"
-      (is (thrown-with-msg?
-           clojure.lang.ExceptionInfo
-           #"No runtime binding for operator"
-           (runtime/helper-fn rt 'nil-bound))))
-    (testing "missing binding throws"
-      (is (thrown-with-msg?
-           clojure.lang.ExceptionInfo
-           #"No runtime binding for operator"
-           (runtime/helper-fn rt 'totally-missing))))))
-
-;; --- P1: Program nesting depth ---
-
-(deftest ^:parallel validated-program-rejects-deeply-nested-program-sources-test
-  (let [leaf {:source {:type "table" :id (meta/id :orders)} :operations []}
-        deep-program (reduce (fn [inner _]
-                               {:source     {:type "program" :program inner}
-                                :operations []})
-                             leaf
-                             (range (+ walker/max-program-nesting 2)))
-        context (orders-context)]
-    (try
-      (validate/validated-program deep-program context)
-      (is false "expected deep nesting to be rejected")
-      (catch clojure.lang.ExceptionInfo e
-        (is (= :invalid-generated-program (:error (ex-data e))))
-        (is (re-find #"program source nesting exceeds maximum depth" (ex-details e)))))))
-
-;; --- P2: Depth-limited contains-aggregation-ref? ---
-
-(deftest ^:parallel contains-aggregation-ref-respects-depth-limit-test
-  (testing "shallow aggregation-ref is found"
-    (is (true? (repair.stages/contains-aggregation-ref?
-                ["aggregate" ["sum" ["aggregation-ref" 0]]]))))
-  (testing "deeply nested structure does not overflow"
-    (let [deep (reduce (fn [inner _] [inner]) ["aggregation-ref" 0] (range 100))]
-      ;; Returns false (too deep) rather than overflowing the stack
-      (is (false? (repair.stages/contains-aggregation-ref? deep))))))
-
-;; --- P2: Depth-limited tree walk in passes ---
-
-(deftest ^:parallel operation-contains-canonical-op-respects-depth-limit-test
-  (testing "shallow match is found"
-    (is (true? (passes/operation-contains-canonical-op?
-                ["aggregate" ["sum" ["metric" 1]]]
-                "metric"))))
-  (testing "deeply nested structure does not overflow"
-    (let [deep (reduce (fn [inner _] [inner]) ["metric" 1] (range 100))]
-      ;; Returns false (too deep) rather than overflowing the stack
-      (is (false? (passes/operation-contains-canonical-op? deep "metric"))))))
-
-;; --- P2: String length validation ---
-
-(deftest ^:parallel walk-node-rejects-overlong-strings-test
-  (let [long-string (apply str (repeat (inc walker/max-string-length) "x"))]
-    (try
-      (walker/walk-node [:test] long-string 0 {} (fn [_ _ _ state] state))
-      (is false "expected overlong string to be rejected")
-      (catch clojure.lang.ExceptionInfo e
-        (is (= :invalid-generated-program (:error (ex-data e))))
-        (is (re-find #"string value exceeds maximum length" (ex-details e))))))
-  (testing "strings within limit pass"
-    (is (= {:node-count 1}
-           (walker/walk-node [:test] "short" 0 {} (fn [_ _ _ state] state))))))
-
-;; --- P2: Error messages don't leak schema ---
-
-(deftest ^:parallel lookup-table-error-keeps-schema-out-of-message-test
-  (let [rt (runtime/build-runtime meta/metadata-provider)]
-    (try
-      ((runtime/helper-fn rt 'table) "nonexistent_table")
-      (is false "expected table lookup to fail")
-      (catch clojure.lang.ExceptionInfo e
-        (testing "message does not enumerate available tables"
-          (is (not (re-find #"orders" (ex-message e)))))
-        (testing "recovery map carries available names for LLM agents"
-          (is (seq (get-in (ex-data e) [:recovery :available]))))))))
-
-;; --- Empty operations list ---
-
-(deftest ^:parallel validate-empty-operations-test
-  (let [program {:source {:type "table" :id (meta/id :orders)} :operations []}
-        context (orders-context)]
-    (is (= program (validate/validated-program program context)))))
-
-;; --- Structured error helpers ---
-
-(deftest ^:parallel wrap-runtime-error-preserves-recovery-from-cause-test
-  (let [cause   (ex-info "Table not found: foo"
-                         {:table-name "foo"
-                          :recovery   {:available ["orders" "products"]}})
-        wrapped (errors/wrap-runtime-error [:source] "table" cause)]
-    (testing "standard fields are set"
-      (is (= :invalid-generated-program (:error (ex-data wrapped))))
-      (is (= "table" (:operator (ex-data wrapped)))))
-    (testing "recovery from cause is preserved"
-      (is (= {:available ["orders" "products"]}
-             (:recovery (ex-data wrapped)))))
-    (testing "cause chain is preserved"
-      (is (= cause (.getCause ^Exception wrapped))))))
-
-(deftest ^:parallel wrap-runtime-error-promotes-legacy-available-key-test
-  (let [cause   (ex-info "Field not found" {:field-name "foo" :available ["a" "b"]})
-        wrapped (errors/wrap-runtime-error [:ops 0] "field" cause)]
-    (testing "top-level :available is promoted into :recovery"
-      (is (= ["a" "b"] (get-in (ex-data wrapped) [:recovery :available]))))))
-
-(deftest ^:parallel recovery-summary-formats-hints-test
-  (testing "available values"
-    (is (= "Available: orders, products"
-           (errors/recovery-summary {:available ["orders" "products"]}))))
-  (testing "tables for disambiguation"
-    (is (= "Matching tables: orders, people"
-           (errors/recovery-summary {:tables ["orders" "people"]}))))
-  (testing "suggestion"
-    (is (= "Use (field \"TABLE\" \"FIELD\") to disambiguate."
-           (errors/recovery-summary {:suggestion "Use (field \"TABLE\" \"FIELD\") to disambiguate."}))))
-  (testing "combined"
-    (let [summary (errors/recovery-summary {:available  ["a" "b"]
-                                            :suggestion "Try again."})]
-      (is (re-find #"Available: a, b" summary))
-      (is (re-find #"Try again\." summary))))
-  (testing "empty recovery returns nil"
-    (is (nil? (errors/recovery-summary {})))
-    (is (nil? (errors/recovery-summary nil)))))

test/metabase/agent_lib/join_spec_test.clj

@@ -1,32 +0,0 @@
-(ns metabase.agent-lib.join-spec-test
-  (:require
-   [clojure.test :refer [deftest is]]
-   [metabase.agent-lib.join-spec :as join-spec]
-   [metabase.lib.test-metadata :as meta]))
-
-(deftest parse-join-spec-normalizes-nested-join-wrappers-test
-  (let [join-form ["with-join-alias"
-                   ["with-join-strategy"
-                    ["with-join-fields"
-                     ["with-join-conditions"
-                      ["join-clause" ["table" (meta/id :categories)]]
-                      [["="
-                        ["field" (meta/id :venues :category-id)]
-                        ["field" (meta/id :categories :id)]]]]
-                     "none"]
-                    "left-join"]
-                   "cat"]
-        expected  {:target-table-id (meta/id :categories)
-                   :conditions      [[(meta/id :venues :category-id)
-                                      (meta/id :categories :id)]]
-                   :fields-mode     "none"
-                   :strategy        "left-join"
-                   :alias           "cat"}]
-    (is (= expected (join-spec/parse-join-spec join-form)))))
-
-(deftest join-fields-mode-helpers-test
-  (is (true? (join-spec/no-explicit-join-fields? nil)))
-  (is (true? (join-spec/no-explicit-join-fields? "none")))
-  (is (false? (join-spec/no-explicit-join-fields? "all")))
-  (is (true? (join-spec/implicit-join-compatible-fields-mode? "all")))
-  (is (false? (join-spec/implicit-join-compatible-fields-mode? [["field" 1]]))))

test/metabase/agent_lib/mbql_integration_test.clj

@@ -1,266 +0,0 @@
-(ns metabase.agent-lib.mbql-integration-test
-  (:require
-   [clojure.test :refer [deftest is testing]]
-   [metabase.agent-lib.mbql-integration :as mbql]
-   [metabase.agent-lib.runtime :as runtime]
-   [metabase.agent-lib.test-util :as tu]
-   [metabase.lib.core :as lib]
-   [metabase.lib.schema.util :as lib.schema.util]
-   [metabase.lib.test-metadata :as meta]))
-
-(def ^:private mbql-ns 'metabase.agent-lib.mbql-integration)
-
-(defn- runtime-fields-by-id
-  []
-  (:fields-by-id (runtime/build-runtime meta/metadata-provider)))
-
-(defn- comparable-ref
-  [ref-clause]
-  (lib.schema.util/remove-lib-uuids ref-clause))
-
-(defn- call-private
-  [sym & args]
-  (apply (var-get (ns-resolve mbql-ns sym)) args))
-
-(defn- current-query-field
-  [query table-name field-name]
-  (or (lib/find-matching-column query
-                                -1
-                                (meta/field-metadata table-name field-name)
-                                (mbql/current-query-field-candidates query))
-      (throw (ex-info "Expected field to be available in query context"
-                      {:table-name table-name
-                       :field-name field-name
-                       :query      query}))))
-
-(defn- current-aggregation-column
-  [query idx]
-  (or (lib/find-matching-column query -1 (lib/aggregation-ref query idx) (lib/orderable-columns query))
-      (throw (ex-info "Expected aggregation column in query context"
-                      {:aggregation-index idx
-                       :query             query}))))
-
-(deftest query?-uses-normalized-query-type-test
-  (is (true? (mbql/query? (tu/query-for-table :orders))))
-  (is (false? (mbql/query? (meta/table-metadata :orders)))))
-
-(deftest extracted-field-resolution-helper-test
-  (let [fields-by-id         (runtime-fields-by-id)
-        orders-query         (tu/query-for-table :orders)
-        by-table             (call-private 'fields-by-table-id fields-by-id)
-        query-candidates     (mbql/current-query-field-candidates orders-query)
-        product-id-field-id  (meta/id :orders :product-id)
-        venues-category-id   (meta/id :venues :category-id)
-        category-id-field-id (meta/id :products :category)
-        product-table-id     (meta/id :products)
-        category-table-id    (meta/id :categories)]
-    (testing "candidate preference is explicit and deterministic"
-      (is (= {:id 10 :name "ID"}
-             (call-private 'prefer-single-candidate
-                           [{:id 10 :name "ID" :lib/join-alias "Products"}
-                            {:id 10 :name "ID"}])))
-      (is (= {:id 20 :name "Category"}
-             (call-private 'prefer-single-candidate
-                           [{:id 20 :name "Category" :fk-field-id 21}
-                            {:id 20 :name "Category"}]))))
-    (testing "metadata grouping and FK target resolution use the extracted helpers"
-      (is (some #(= product-id-field-id (:id %))
-                (get by-table (meta/id :orders))))
-      (is (= product-table-id
-             (call-private 'field-target-table-id fields-by-id product-id-field-id)))
-      (is (nil? (call-private 'field-target-table-id fields-by-id (meta/id :orders :total)))))
-    (testing "lineage helpers expose the path anchors and recursive FK traversal"
-      (is (= [product-id-field-id category-id-field-id 999]
-             (vec (call-private 'field-path-start-ids
-                                {:source-field product-id-field-id
-                                 :fk-field-id category-id-field-id
-                                 :lib/original-fk-field-id category-id-field-id
-                                 :id 999}))))
-      (is (= [product-id-field-id]
-             (call-private 'fk-path-to-table fields-by-id product-id-field-id product-table-id)))
-      (is (= [venues-category-id]
-             (call-private 'fk-path-to-table fields-by-id venues-category-id category-table-id)))
-      (is (nil? (call-private 'fk-path-to-table fields-by-id product-id-field-id category-table-id))))
-    (testing "type compatibility and lineage ranking helpers are independently testable"
-      (is (true? (call-private 'numeric-type? :type/Float)))
-      (is (false? (call-private 'numeric-type? :type/Text)))
-      (is (true? (call-private 'types-compatible?
-                               {:effective-type :type/Float}
-                               {:effective-type :type/Integer})))
-      (is (false? (call-private 'types-compatible?
-                                {:effective-type :type/Text}
-                                {:effective-type :type/Integer})))
-      (is (true? (call-private 'candidate-has-resolution-lineage?
-                               {:fk-field-id category-id-field-id})))
-      (is (false? (call-private 'candidate-has-resolution-lineage?
-                                {:id product-id-field-id}))))
-    (testing "source-column accessors and synthesized multi-hop lineage are namespace-level helpers"
-      (let [source-column {:lib/source-column-alias      "Category"
-                           :name                         "CATEGORY"
-                           :fk-field-name                "Category"
-                           :fk-join-alias                "Products"
-                           :source-field                 category-id-field-id
-                           :lib/original-fk-field-id     product-id-field-id
-                           :lib/original-fk-field-name   "Product ID"
-                           :lib/original-fk-join-alias   nil}
-            raw-field     {:id category-id-field-id
-                           :name "Category"
-                           :table-id category-table-id}
-            synthesized   (call-private 'synthesize-chained-related-field
-                                        fields-by-id
-                                        query-candidates
-                                        raw-field
-                                        source-column
-                                        [product-id-field-id category-id-field-id])]
-        (is (= "Category" (call-private 'source-column-field-name source-column)))
-        (is (= "Products" (call-private 'source-column-field-join-alias source-column)))
-        (is (= product-id-field-id (call-private 'source-column-original-field-id source-column)))
-        (is (= "Product ID" (call-private 'source-column-original-field-name source-column)))
-        (is (nil? (call-private 'source-column-original-field-join-alias source-column)))
-        (is (= "Products"
-               (call-private 'candidate-join-alias-for-field-id
-                             [{:source-field category-id-field-id
-                               :lib/join-alias "Products"}]
-                             category-id-field-id)))
-        (is (= category-id-field-id (:fk-field-id synthesized)))
-        (is (= product-id-field-id (:lib/original-fk-field-id synthesized)))
-        (is (= "Products" (:fk-join-alias synthesized)))
-        (is (true? (call-private 'multi-hop-lineage-candidate? synthesized)))
-        (is (false? (call-private 'multi-hop-lineage-candidate?
-                                  {:fk-field-id category-id-field-id
-                                   :lib/original-fk-field-id category-id-field-id})))))
-    (testing "previous-stage matching is a separate unit from the main resolver"
-      (is (= [{:name "Average of Total"
-               :source-field (meta/id :orders :total)
-               :lib/source :source/previous-stage}]
-             (call-private 'previous-stage-lineage-matches
-                           {:id (meta/id :orders :total)
-                            :name "Total"}
-                           [{:name "Average of Total"
-                             :source-field (meta/id :orders :total)
-                             :lib/source :source/previous-stage}
-                            {:name "Count"
-                             :lib/source :source/previous-stage}])))
-      (let [appended-query         (-> orders-query
-                                       (lib/aggregate (lib/avg (meta/field-metadata :orders :total)))
-                                       (lib/append-stage))
-            previous-stage-columns (->> (mbql/current-query-field-candidates appended-query)
-                                        (filter #(= :source/previous-stage (:lib/source %)))
-                                        vec)]
-        (is (= ["avg"]
-               (mapv :name
-                     (call-private 'previous-stage-aggregation-matches
-                                   appended-query
-                                   {:id (meta/id :orders :total)
-                                    :name "Total"}
-                                   previous-stage-columns)))))
-      (is (= [{:name "NET AMOUNT" :lib/source :source/previous-stage}]
-             (call-private 'previous-stage-name-matches
-                           {:name "Net Amount"}
-                           [{:name "NET AMOUNT" :lib/source :source/previous-stage}
-                            {:name "Other" :lib/source :source/previous-stage}]))))))
-
-(deftest extracted-orderable-helper-test
-  (let [orders-query            (tu/query-for-table :orders)
-        breakout-query          (-> orders-query
-                                    (lib/expression "__breakout_expression_1"
-                                                    (lib/get-hour (meta/field-metadata :orders :created-at)))
-                                    (lib/aggregate (lib/count))
-                                    (as-> query
-                                          (lib/breakout query (lib/expression-ref query "__breakout_expression_1"))))
-        aggregation-query       (-> orders-query
-                                    (lib/aggregate (lib/sum (meta/field-metadata :orders :total)))
-                                    (lib/aggregate (lib/count)))
-        total-field-id          (meta/id :orders :total)
-        created-at-field-id     (meta/id :orders :created-at)
-        breakout-expression-ref (lib/expression-ref breakout-query "__breakout_expression_1")
-        aggregation-column      (current-aggregation-column aggregation-query 0)]
-    (testing "aggregation inventory helpers are directly unit-tested"
-      (is (= 2 (count (call-private 'aggregation-columns aggregation-query))))
-      (is (= [0 1]
-             (mapv first (call-private 'aggregation-column-pairs aggregation-query)))))
-    (testing "requested field ids and current-stage field matching are separate helpers"
-      (is (= #{total-field-id}
-             (call-private 'requested-orderable-field-ids (meta/field-metadata :orders :total))))
-      (is (= #{created-at-field-id}
-             (call-private 'requested-orderable-field-ids (lib/ref (meta/field-metadata :orders :created-at)))))
-      (is (= #{}
-             (call-private 'requested-orderable-field-ids "not-a-field")))
-      (is (some #(= total-field-id (:id %))
-                (call-private 'orderable-field-id->query-columns orders-query total-field-id)))
-      (is (some #(= total-field-id (:id %))
-                (call-private 'orderable-field-ids->query-columns orders-query #{total-field-id}))))
-    (testing "breakout and aggregation matching helpers are individually covered"
-      (is (= [(comparable-ref breakout-expression-ref)]
-             (mapv comparable-ref
-                   (call-private 'field-id->breakout-expression-refs breakout-query created-at-field-id))))
-      (is (= (comparable-ref breakout-expression-ref)
-             (comparable-ref
-              (call-private 'field-ids->breakout-expression-ref breakout-query #{created-at-field-id}))))
-      (is (= [aggregation-column]
-             (call-private 'field-id->aggregation-columns aggregation-query total-field-id)))
-      (is (= [aggregation-column]
-             (call-private 'field-ids->aggregation-columns aggregation-query #{total-field-id}))))
-    (testing "field-like orderable resolution is directly testable after extraction"
-      (let [[_ breakout-orderable] (call-private 'resolve-field-like-orderable
-                                                 breakout-query
-                                                 (meta/field-metadata :orders :created-at))
-            [_ aggregation-orderable] (call-private 'resolve-field-like-orderable
-                                                    aggregation-query
-                                                    (meta/field-metadata :orders :total))]
-        (is (= (comparable-ref breakout-expression-ref)
-               (comparable-ref breakout-orderable)))
-        (is (= aggregation-column aggregation-orderable))))))
-
-(deftest implicitly-resolved-column-helper-test
-  (is (true? (call-private 'implicitly-resolved-column?
-                           {:table-id 100
-                            :fk-field-id 20}
-                           100)))
-  (is (true? (call-private 'implicitly-resolved-column?
-                           {:table-id 200
-                            :lib/source :source/previous-stage}
-                           100)))
-  (is (false? (call-private 'implicitly-resolved-column?
-                            {:table-id 100
-                             :join-alias "Products"
-                             :fk-field-id 20}
-                            100))))
-
-(deftest resolve-aggregation-selection-uses-visible-column-match-test
-  (let [query     (-> (tu/query-for-table :orders)
-                      (lib/aggregate (lib/count)))
-        selection (lib/aggregation-ref query 0)]
-    (is (= (lib/find-visible-column-for-ref query selection)
-           (mbql/resolve-aggregation-selection query selection)))))
-
-(deftest expression-ref-or-current-stage-column-uses-existing-expression-test
-  (let [query (-> (tu/query-for-table :orders)
-                  (lib/expression "Net Amount"
-                                  (lib/- (meta/field-metadata :orders :total)
-                                         (meta/field-metadata :orders :discount))))]
-    (is (= (comparable-ref (lib/expression-ref query "Net Amount"))
-           (comparable-ref (mbql/expression-ref-or-current-stage-column query "Net Amount"))))))
-
-(deftest resolve-orderable-reuses-breakout-expression-test
-  (let [query (-> (tu/query-for-table :orders)
-                  (lib/expression "__breakout_expression_1"
-                                  (lib/get-hour (meta/field-metadata :orders :created-at)))
-                  (lib/aggregate (lib/count))
-                  (as-> query
-                        (lib/breakout query (lib/expression-ref query "__breakout_expression_1"))))
-        [_ resolved-orderable] (mbql/resolve-orderable query (meta/field-metadata :orders :created-at))]
-    (is (= (comparable-ref (lib/expression-ref query "__breakout_expression_1"))
-           (comparable-ref resolved-orderable)))))
-
-(deftest redundant-implicit-join-detects-existing-implicit-join-test
-  (let [fields-by-id (:fields-by-id (runtime/build-runtime meta/metadata-provider))
-        base-query   (tu/query-for-table :orders)
-        query        (lib/with-fields base-query [(current-query-field base-query :products :title)])
-        operation    ["join"
-                      ["with-join-conditions"
-                       ["join-clause" ["table" (meta/id :products)]]
-                       [["=" ["field" (meta/id :orders :product-id)]
-                         ["field" (meta/id :products :id)]]]]]]
-    (testing "implicit join coverage is centralized in mbql-integration"
-      (is (true? (mbql/redundant-implicit-join? fields-by-id query operation))))))

test/metabase/agent_lib/repair/context/passes_test.clj

@@ -1,64 +0,0 @@
-(ns metabase.agent-lib.repair.context.passes-test
-  (:require
-   [clojure.test :refer [deftest is testing]]
-   [metabase.agent-lib.repair.context.passes :as passes]
-   [metabase.agent-lib.test-util :as agent-lib.tu]
-   [metabase.lib.test-metadata :as meta]))
-
-(deftest ^:parallel effective-source-for-context-rewrites-metric-context-source-test
-  (let [program {:source {:type "context" :ref "source"}}
-        context {:source-entity {:model "metric" :id 42}}]
-    (is (= {:type "metric" :id 42}
-           (passes/effective-source-for-context program context)))))
-
-(deftest ^:parallel rewrite-source-stage-only-touches-the-first-stage-test
-  (let [operations [["aggregate" ["count"]]
-                    ["append-stage"]
-                    ["aggregate" ["sum" ["field" 1]]]]
-        rewritten  (passes/rewrite-source-stage
-                    operations
-                    (fn [{:keys [in-source-stage? operations] :as state} operation]
-                      (assoc state
-                             :operations
-                             (conj operations
-                                   (if in-source-stage?
-                                     ["seen" operation]
-                                     operation)))))]
-    (is (= [["seen" ["aggregate" ["count"]]]
-            ["append-stage"]
-            ["aggregate" ["sum" ["field" 1]]]]
-           rewritten))))
-
-(deftest ^:parallel normalize-source-metric-quarter-window-rewrites-rolling-quarter-filter-test
-  (let [context    {:source-entity {:model "metric" :id 42}}
-        operation  ["filter" ["between"
-                              ["field" (meta/id :orders :created-at)]
-                              ["relative-datetime" -3 "month"]
-                              ["now"]]]
-        normalized (passes/normalize-source-metric-quarter-window context operation)]
-    (is (= ["filter" ["between"
-                      ["field" (meta/id :orders :created-at)]
-                      ["relative-datetime" -1 "quarter"]
-                      ["relative-datetime" 0 "quarter"]]]
-           normalized))))
-
-(deftest ^:parallel remove-redundant-operations-drops-source-metric-and-implicit-join-noise-test
-  (let [metric-context {:source-entity {:model "metric" :id 7}}
-        table-context  {:source-entity      {:model "table" :id (meta/id :venues)}
-                        :surrounding-tables [(agent-lib.tu/entity-summary :categories)]
-                        :join-edges         [(agent-lib.tu/join-edge :venues :category-id :categories :id)]}
-        metric-aggregate ["aggregate" ["metric" 7]]
-        explicit-join    ["join"
-                          ["with-join-conditions"
-                           ["join-clause" ["table" (meta/id :categories)]]
-                           [["=" ["field" (meta/id :venues :category-id)]
-                             ["field" (meta/id :categories :id)]]]]]
-        kept-operation   ["filter" ["=" ["field" (meta/id :venues :name)] "Red Medicine"]]]
-    (testing "source-metric reuse is dropped in metric contexts"
-      (is (= [kept-operation]
-             (vec (passes/remove-redundant-operations metric-context
-                                                      [metric-aggregate kept-operation])))))
-    (testing "implicit joins already represented by the context are dropped for table sources"
-      (is (= [kept-operation]
-             (vec (passes/remove-redundant-operations table-context
-                                                      [explicit-join kept-operation])))))))

test/metabase/agent_lib/repair/normalize/forms_test.clj

@@ -1,205 +0,0 @@
-(ns metabase.agent-lib.repair.normalize.forms-test
-  (:require
-   [clojure.test :refer [deftest is testing]]
-   [metabase.agent-lib.repair.normalize.forms :as forms]))
-
-(deftest ^:parallel expression-definition-tuple-test
-  (testing "recognizes expression definitions"
-    (is (true? (forms/expression-definition-tuple? ["expression" "Net Total" ["-" ["field" 1] 10]]))))
-  (testing "rejects wrong operator"
-    (is (false? (forms/expression-definition-tuple? ["filter" "Name" ["-" 1 2]]))))
-  (testing "rejects wrong arity"
-    (is (false? (forms/expression-definition-tuple? ["expression" "Name"])))
-    (is (false? (forms/expression-definition-tuple? ["expression" "Name" 1 2]))))
-  (testing "rejects blank name"
-    (is (false? (forms/expression-definition-tuple? ["expression" "" ["-" 1 2]]))))
-  (testing "rejects non-vectors"
-    (is (false? (forms/expression-definition-tuple? "expression")))))
-
-(deftest ^:parallel temporal-expression-test
-  (testing "now literal is temporal"
-    (is (true? (forms/temporal-expression? "now")))
-    (is (true? (forms/temporal-expression? "NOW"))))
-  (testing "temporal tuples are temporal"
-    (is (true? (forms/temporal-expression? ["absolute-datetime" "2024-01-01" "day"])))
-    (is (true? (forms/temporal-expression? ["relative-datetime" -3 "month"])))
-    (is (true? (forms/temporal-expression? ["now"]))))
-  (testing "non-temporal values"
-    (is (false? (forms/temporal-expression? "hello")))
-    (is (false? (forms/temporal-expression? 42)))
-    (is (false? (forms/temporal-expression? ["field" 1])))))
-
-(deftest ^:parallel repair-case-args-three-bare-args-test
-  (testing "three bare args become branch pair + fallback"
-    (is (= [[["pred" "then"]] "else"]
-           (forms/repair-case-args ["pred" "then" "else"])))))
-
-(deftest ^:parallel repair-case-args-wrapped-branch-pairs-test
-  (testing "single wrapped vector of branch pairs"
-    (is (= [[[[">" 1] "a"] [["<" 0] "b"]]]
-           (forms/repair-case-args [[[">" 1] "a"] [["<" 0] "b"]])))))
-
-(deftest ^:parallel repair-case-args-wrapped-branch-pairs-with-fallback-test
-  (testing "wrapped branches with fallback"
-    (is (= [[[[">" 1] "a"]] "default"]
-           (forms/repair-case-args [[[">" 1] "a"] "default"])))))
-
-(deftest ^:parallel repair-case-args-leading-branch-pairs-test
-  (testing "leading branch pairs followed by fallback"
-    (is (= [[[[">" 1] "a"]] "default"]
-           (forms/repair-case-args [[[">" 1] "a"] "default"])))))
-
-(deftest ^:parallel repair-case-args-flat-even-vector-test
-  (testing "flat even-length vector inside wrapper"
-    (is (= [[[1 2] [3 4]]]
-           (forms/repair-case-args [[1 2 3 4]])))))
-
-(deftest ^:parallel repair-case-args-else-branch-test
-  (testing "else branch is split out"
-    (is (= [[[[">" 1] "a"]] "fallback"]
-           (forms/repair-case-args [[[">" 1] "a"] ["else" "fallback"]])))))
-
-(deftest ^:parallel repair-between-bounds-test
-  (testing "wraps ISO date when other bound is temporal"
-    (is (= [["field" 1] ["relative-datetime" -3 "month"] ["absolute-datetime" "2025-09-30" "day"]]
-           (forms/repair-between-bounds
-            [["field" 1] ["relative-datetime" -3 "month"] "2025-09-30"]))))
-  (testing "wraps now literal into expression"
-    (is (= [["field" 1] ["relative-datetime" -3 "month"] ["now"]]
-           (forms/repair-between-bounds
-            [["field" 1] ["relative-datetime" -3 "month"] "now"]))))
-  (testing "leaves non-temporal bounds unchanged"
-    (is (= [["field" 1] 10 20]
-           (forms/repair-between-bounds [["field" 1] 10 20])))))
-
-(deftest ^:parallel repair-operator-form-field-coercion-test
-  (testing "coerces field id string to integer"
-    (is (= ["field" 42]
-           (forms/repair-operator-form identity ["field" "42"])))))
-
-(deftest ^:parallel field-like-map?-test
-  (testing "accepts snake_case and kebab-case, string and keyword keys"
-    (is (true? (forms/field-like-map? {"field_id" 1})))
-    (is (true? (forms/field-like-map? {"field-id" 1})))
-    (is (true? (forms/field-like-map? {:field_id 1})))
-    (is (true? (forms/field-like-map? {:field-id 1}))))
-  (testing "rejects non-field-like maps"
-    (is (false? (forms/field-like-map? {"foo" 1})))
-    (is (false? (forms/field-like-map? {})))
-    (is (false? (forms/field-like-map? nil)))
-    (is (false? (forms/field-like-map? ["field" 1])))))
-
-(deftest ^:parallel repair-field-like-map-test
-  (testing "field_id alone becomes a plain field form"
-    (is (= ["field" 13]
-           (forms/repair-field-like-map {"field_id" 13}))))
-  (testing "temporal_bucket wraps in with-temporal-bucket"
-    (is (= ["with-temporal-bucket" ["field" 13] "month"]
-           (forms/repair-field-like-map {"field_id" 13 "temporal_bucket" "month"}))))
-  (testing "temporal-unit alias is accepted"
-    (is (= ["with-temporal-bucket" ["field" 13] "month"]
-           (forms/repair-field-like-map {"field_id" 13 "temporal-unit" "month"}))))
-  (testing "binning wraps in with-binning"
-    (is (= ["with-binning" ["field" 13] {"strategy" "num-bins" "num-bins" 10}]
-           (forms/repair-field-like-map {"field_id" 13
-                                         "binning"  {"strategy" "num-bins" "num-bins" 10}}))))
-  (testing "direction wraps in asc/desc"
-    (is (= ["asc" ["field" 13]]
-           (forms/repair-field-like-map {"field_id" 13 "direction" "asc"})))
-    (is (= ["desc" ["field" 13]]
-           (forms/repair-field-like-map {"field_id" 13 "direction" "DESC"}))))
-  (testing "combined options nest in canonical order: direction wraps the bucketed/binned field"
-    (is (= ["asc" ["with-temporal-bucket" ["field" 13] "month"]]
-           (forms/repair-field-like-map {"field_id"        13
-                                         "temporal_bucket" "month"
-                                         "direction"       "asc"}))))
-  (testing "keyword keys are also accepted"
-    (is (= ["with-temporal-bucket" ["field" 13] "month"]
-           (forms/repair-field-like-map {:field_id 13 :temporal_bucket "month"}))))
-  (testing "string id is coerced to integer"
-    (is (= ["field" 13]
-           (forms/repair-field-like-map {"field_id" "13"})))))
-
-(deftest ^:parallel repair-operator-form-field-options-map-test
-  (testing "field with temporal-unit option rewrites to with-temporal-bucket wrapper"
-    (is (= ["with-temporal-bucket" ["field" 13] "month"]
-           (forms/repair-operator-form identity ["field" 13 {"temporal-unit" "month"}]))))
-  (testing "field with binning option rewrites to with-binning wrapper"
-    (is (= ["with-binning" ["field" 13] {"strategy" "num-bins" "num-bins" 10}]
-           (forms/repair-operator-form identity
-                                       ["field" 13 {"binning" {"strategy" "num-bins" "num-bins" 10}}]))))
-  (testing "field with both options nests with-binning outside with-temporal-bucket"
-    (is (= ["with-binning"
-            ["with-temporal-bucket" ["field" 13] "month"]
-            {"strategy" "default"}]
-           (forms/repair-operator-form identity
-                                       ["field" 13 {"temporal-unit" "month"
-                                                    "binning"       {"strategy" "default"}}]))))
-  (testing "empty options map leaves field untouched"
-    (is (= ["field" 13]
-           (forms/repair-operator-form identity ["field" 13 {}]))))
-  (testing "keyword option keys are accepted"
-    (is (= ["with-temporal-bucket" ["field" 13] "month"]
-           (forms/repair-operator-form identity ["field" 13 {:temporal-unit "month"}]))))
-  (testing "string field id is still coerced when options are present"
-    (is (= ["with-temporal-bucket" ["field" 13] "month"]
-           (forms/repair-operator-form identity ["field" "13" {"temporal-unit" "month"}])))))
-
-(deftest ^:parallel repair-operator-form-is-null-test
-  (testing "is with null becomes is-null"
-    (is (= ["is-null" ["field" 1]]
-           (forms/repair-operator-form identity ["is" ["field" 1] "null"]))))
-  (testing "is with non-null becomes ="
-    (is (= ["=" ["field" 1] "hello"]
-           (forms/repair-operator-form identity ["is" ["field" 1] "hello"])))))
-
-(deftest ^:parallel repair-operator-form-is-not-null-test
-  (testing "is-not with null becomes not-null"
-    (is (= ["not-null" ["field" 1]]
-           (forms/repair-operator-form identity ["is-not" ["field" 1] nil]))))
-  (testing "is-not with non-null becomes !="
-    (is (= ["!=" ["field" 1] "hello"]
-           (forms/repair-operator-form identity ["is-not" ["field" 1] "hello"])))))
-
-(deftest ^:parallel repair-operator-form-eq-with-sequential-promotes-to-in-test
-  (testing "= with sequential RHS promotes to in"
-    (is (= ["in" ["field" 1] ["a" "b"]]
-           (forms/repair-operator-form identity ["=" ["field" 1] ["a" "b"]])))))
-
-(deftest ^:parallel repair-operator-form-neq-with-sequential-promotes-to-not-in-test
-  (testing "!= with sequential RHS promotes to not-in"
-    (is (= ["not-in" ["field" 1] ["a" "b"]]
-           (forms/repair-operator-form identity ["!=" ["field" 1] ["a" "b"]])))))
-
-(deftest ^:parallel repair-operator-form-contains-strips-boolean-flag-test
-  (testing "contains strips trailing boolean option"
-    (is (= ["contains" ["field" 1] "text"]
-           (forms/repair-operator-form identity ["contains" ["field" 1] "text" false]))))
-  (testing "contains strips trailing map option"
-    (is (= ["contains" ["field" 1] "text"]
-           (forms/repair-operator-form identity ["contains" ["field" 1] "text" {:case-sensitive false}])))))
-
-(deftest ^:parallel repair-operator-form-with-temporal-bucket-extraction-alias-test
-  (testing "with-temporal-bucket rewrites extraction aliases"
-    (is (= ["get-hour" ["field" 1]]
-           (forms/repair-operator-form identity ["with-temporal-bucket" ["field" 1] "hour-of-day"])))
-    (is (= ["get-day-of-week" ["field" 1]]
-           (forms/repair-operator-form identity ["with-temporal-bucket" ["field" 1] "day-of-week"]))))
-  (testing "non-extraction buckets pass through"
-    (is (= ["with-temporal-bucket" ["field" 1] "month"]
-           (forms/repair-operator-form identity ["with-temporal-bucket" ["field" 1] "month"])))))
-
-(deftest ^:parallel repair-operator-form-percentile-test
-  (testing "normalizes percentile scale"
-    (is (= ["percentile" ["field" 1] 0.9]
-           (forms/repair-operator-form identity ["percentile" ["field" 1] 90])))))
-
-(deftest ^:parallel repair-operator-form-case-from-if-test
-  (testing "if alias normalizes to case with three args"
-    (is (= ["case" [["> 1" "then"]] "else"]
-           (forms/repair-operator-form identity ["if" "> 1" "then" "else"])))))
-
-(deftest ^:parallel repair-operator-form-passthrough-test
-  (testing "unrecognized-but-valid operators pass through"
-    (is (= ["+" 1 2]
-           (forms/repair-operator-form identity ["+" 1 2])))))

test/metabase/agent_lib/repair/normalize/top_level_test.clj

@@ -1,27 +0,0 @@
-(ns metabase.agent-lib.repair.normalize.top-level-test
-  (:require
-   [clojure.test :refer [deftest is]]
-   [metabase.agent-lib.repair.normalize.top-level :as top-level]))
-
-(deftest ^:parallel normalize-with-fields-selection-flattens-single-sequential-arg-test
-  (is (= [["field" 1] ["field" 2]]
-         (top-level/normalize-with-fields-selection [[["field" 1] ["field" 2]]]))))
-
-(deftest ^:parallel repair-order-by-operations-normalizes-both-order-by-encodings-test
-  (is (= [["order-by" ["desc" ["field" 1]]]
-          ["order-by" ["field" 2] "asc"]]
-         (top-level/repair-order-by-operations [["desc" ["field" 1]]
-                                                ["field" 2]
-                                                "asc"]))))
-
-(deftest ^:parallel repair-with-fields-operation-extracts-inline-expressions-test
-  (is (= [["expression" "Net" ["-" ["field" 1] 10]]
-          ["with-fields" [["field" 1] ["expression-ref" "Net"]]]]
-         (top-level/repair-with-fields-operation identity
-                                                 [[["field" 1]
-                                                   ["expression" "Net" ["-" ["field" 1] 10]]]]))))
-
-(deftest ^:parallel repair-top-level-operation-rewrites-aggregation-expressions-test
-  (is (= [["aggregate" ["count"]]]
-         (top-level/repair-top-level-operation identity
-                                               ["expression" "Total" ["count"]]))))

test/metabase/agent_lib/repair/stages_test.clj

@@ -1,23 +0,0 @@
-(ns metabase.agent-lib.repair.stages-test
-  (:require
-   [clojure.test :refer [deftest is]]
-   [metabase.agent-lib.repair.stages :as repair.stages]
-   [metabase.lib.test-metadata :as meta]))
-
-(deftest insert-stage-boundaries-adds-post-aggregation-stage-test
-  (let [source     {:type "context" :ref "source"}
-        operations [["aggregate" ["sum" ["field" (meta/id :orders :total)]]]
-                    ["expression" "Revenue K" ["/" ["aggregation-ref" 0] 1000]]]
-        expected   [["aggregate" ["sum" ["field" (meta/id :orders :total)]]]
-                    ["append-stage"]
-                    ["expression" "Revenue K" ["/" ["aggregation-ref" 0] 1000]]]]
-    (is (= expected
-           (repair.stages/insert-stage-boundaries source operations)))))
-
-(deftest insert-stage-boundaries-respects-existing-boundaries-test
-  (let [source     {:type "context" :ref "source"}
-        operations [["aggregate" ["sum" ["field" (meta/id :orders :total)]]]
-                    ["append-stage"]
-                    ["filter" [">" ["aggregation-ref" 0] 100]]]]
-    (is (= operations
-           (repair.stages/insert-stage-boundaries source operations)))))

test/metabase/agent_lib/repair_test.clj

@@ -1,224 +0,0 @@
-(ns metabase.agent-lib.repair-test
-  (:require
-   [clojure.test :refer [deftest is]]
-   [metabase.agent-lib.repair :as repair]
-   [metabase.lib.test-metadata :as meta]))
-
-(deftest ^:parallel repair-program-normalization-test
-  (let [program {:source     {:type "context" :ref "source"}
-                 :operations [["aggregate" ["count"] ["variance" ["field" (meta/id :orders :total)]]]
-                              ["with-fields" [["field" (meta/id :orders :id)]
-                                              ["expression" "Net Total" ["-" ["field" (meta/id :orders :total)] 10]]]]
-                              ["filter" ["is-not" ["field" (meta/id :orders :total)] "null"]]
-                              ["filter" ["=" ["field" (meta/id :orders :product-id)] ["1" "2"]]]
-                              ["breakout"
-                               ["with-temporal-bucket" ["field" (meta/id :orders :created-at)] "day-of-week"]]
-                              ["order-by" ["field" (meta/id :orders :created-at)]
-                               ["desc" ["field" (meta/id :orders :total)]]]]}
-        expected {:source     {:type "context" :ref "source"}
-                  :operations [["aggregate" ["count"]]
-                               ["aggregate" ["var" ["field" (meta/id :orders :total)]]]
-                               ["expression" "Net Total" ["-" ["field" (meta/id :orders :total)] 10]]
-                               ["with-fields" [["field" (meta/id :orders :id)]
-                                               ["expression-ref" "Net Total"]]]
-                               ["filter" ["not-null" ["field" (meta/id :orders :total)]]]
-                               ["filter" ["in" ["field" (meta/id :orders :product-id)] ["1" "2"]]]
-                               ["breakout" ["get-day-of-week" ["field" (meta/id :orders :created-at)]]]
-                               ["order-by" ["field" (meta/id :orders :created-at)]]
-                               ["order-by" ["desc" ["field" (meta/id :orders :total)]]]]}]
-    (is (= expected (repair/repair-program program)))))
-
-(deftest ^:parallel repair-program-quarter-and-diff-normalization-test
-  (let [program {:source     {:type "context" :ref "source"}
-                 :operations [["filter" ["="
-                                         ["with-temporal-bucket"
-                                          ["field" (meta/id :orders :created-at)] "quarter"]
-                                         "Q1"]]
-                              ["breakout"
-                               ["with-temporal-bucket"
-                                ["field" (meta/id :orders :created-at)] "hour-of-day"]]
-                              ["expression" "Quarter Number"
-                               ["quarter-of-year" ["field" (meta/id :orders :created-at)]]]
-                              ["expression" "Days Open"
-                               ["temporal-diff" ["now"]
-                                ["field" (meta/id :orders :created-at)] "day"]]]}
-        expected {:source     {:type "context" :ref "source"}
-                  :operations [["filter" ["="
-                                          ["with-temporal-bucket"
-                                           ["field" (meta/id :orders :created-at)] "quarter"]
-                                          1]]
-                               ["breakout"
-                                ["get-hour" ["field" (meta/id :orders :created-at)]]]
-                               ["expression" "Quarter Number"
-                                ["get-quarter" ["field" (meta/id :orders :created-at)]]]
-                               ["expression" "Days Open"
-                                ["datetime-diff" ["now"]
-                                 ["field" (meta/id :orders :created-at)] "day"]]]}]
-    (is (= expected (repair/repair-program program)))))
-
-(deftest ^:parallel repair-program-normalizes-if-and-between-bounds-test
-  (let [program {:source     {:type "context" :ref "source"}
-                 :operations [["filter" ["between"
-                                         ["field" (meta/id :orders :created-at)]
-                                         ["relative-datetime" -3 "month"]
-                                         "2025-09-30"]]
-                              ["expression" "Accepted Count"
-                               ["if" ["=" ["field" (meta/id :orders :user-id)] 1] 1 0]]]}
-        expected {:source     {:type "context" :ref "source"}
-                  :operations [["filter" ["between"
-                                          ["field" (meta/id :orders :created-at)]
-                                          ["relative-datetime" -3 "month"]
-                                          ["absolute-datetime" "2025-09-30" "day"]]]
-                               ["expression" "Accepted Count"
-                                ["case" [[["=" ["field" (meta/id :orders :user-id)] 1] 1]] 0]]]}]
-    (is (= expected (repair/repair-program program)))))
-
-(deftest ^:parallel repair-program-normalizes-now-bounds-case-ternaries-and-text-filter-flags-test
-  (let [program {:source     {:type "context" :ref "source"}
-                 :operations [["filter" ["contains" ["field" (meta/id :orders :product-id)] "friend" false]]
-                              ["filter" ["between"
-                                         ["field" (meta/id :orders :created-at)]
-                                         ["relative-datetime" -3 "month"]
-                                         "now"]]
-                              ["expression" "Accepted Count"
-                               ["case" [["true" ["field" (meta/id :orders :user-id)]]
-                                        1
-                                        0]]]]}
-        expected {:source     {:type "context" :ref "source"}
-                  :operations [["filter" ["contains" ["field" (meta/id :orders :product-id)] "friend"]]
-                               ["filter" ["between"
-                                          ["field" (meta/id :orders :created-at)]
-                                          ["relative-datetime" -3 "month"]
-                                          ["now"]]]
-                               ["expression" "Accepted Count"
-                                ["case" [[["field" (meta/id :orders :user-id)] 1]] 0]]]}]
-    (is (= expected (repair/repair-program program)))))
-
-(deftest ^:parallel repair-program-unwraps-join-condition-wrapper-test
-  (let [wrapped-conditions [[["="
-                              ["field" (meta/id :venues :category-id)]
-                              ["field" (meta/id :categories :id)]]]]
-        expected-conditions [["="
-                              ["field" (meta/id :venues :category-id)]
-                              ["field" (meta/id :categories :id)]]]
-        program             {:source     {:type "context" :ref "source"}
-                             :operations [["join"
-                                           ["with-join-conditions"
-                                            ["join-clause" ["table" (meta/id :categories)]]
-                                            wrapped-conditions]]]}
-        expected            {:source     {:type "context" :ref "source"}
-                             :operations [["join"
-                                           ["with-join-conditions"
-                                            ["join-clause" ["table" (meta/id :categories)]]
-                                            expected-conditions]]]}]
-    (is (= expected (repair/repair-program program)))))
-
-(deftest ^:parallel repair-program-rewrites-aggregation-expression-test
-  (let [program {:source     {:type "context" :ref "source"}
-                 :operations [["expression" "Total Opportunities" ["count"]]
-                              ["expression" "Won" ["count-if" ["field" (meta/id :orders :discount)]]]]}
-        expected {:source     {:type "context" :ref "source"}
-                  :operations [["aggregate" ["count"]]
-                               ["aggregate" ["count-where" ["field" (meta/id :orders :discount)]]]]}]
-    (is (= expected (repair/repair-program program)))))
-
-(deftest ^:parallel repair-program-drops-stray-bare-aggregation-ref-in-aggregate-test
-  (let [program {:source     {:type "context" :ref "source"}
-                 :operations [["aggregate" ["count"] ["aggregation-ref" 0]]]}
-        expected {:source     {:type "context" :ref "source"}
-                  :operations [["aggregate" ["count"]]]}]
-    (is (= expected (repair/repair-program program)))))
-
-(deftest ^:parallel repair-program-normalizes-percentile-scale-test
-  (let [program {:source     {:type "context" :ref "source"}
-                 :operations [["aggregate" ["percentile" ["field" (meta/id :orders :total)] 90]]]}
-        expected {:source     {:type "context" :ref "source"}
-                  :operations [["aggregate" ["percentile" ["field" (meta/id :orders :total)] 0.9]]]}]
-    (is (= expected (repair/repair-program program)))))
-
-(deftest ^:parallel repair-program-inserts-append-stage-before-aggregate-on-aggregation-ref-test
-  (let [program {:source     {:type "program"
-                              :program {:source     {:type "context" :ref "source"}
-                                        :operations [["aggregate" ["sum" ["field" (meta/id :orders :total)]]]
-                                                     ["breakout" ["field" (meta/id :orders :created-at)]]]}}
-                 :operations [["aggregate" ["stddev" ["aggregation-ref" 0]]]]}
-        expected {:source     {:type "program"
-                               :program {:source     {:type "context" :ref "source"}
-                                         :operations [["aggregate" ["sum" ["field" (meta/id :orders :total)]]]
-                                                      ["breakout" ["field" (meta/id :orders :created-at)]]]}}
-                  :operations [["append-stage"]
-                               ["aggregate" ["stddev" ["aggregation-ref" 0]]]]}]
-    (is (= expected (repair/repair-program program)))))
-
-(deftest ^:parallel repair-program-for-context-drops-source-stage-aggregates-for-metric-sources-test
-  (let [program {:source     {:type "context" :ref "source"}
-                 :operations [["aggregate" ["avg" ["field" (meta/id :orders :total)]]]
-                              ["breakout" ["field" (meta/id :orders :product-id)]]
-                              ["append-stage"]
-                              ["aggregate" ["avg" ["aggregation-ref" 0]]]]}
-        context {:source-entity       {:model "metric" :id 42}
-                 :referenced-entities []
-                 :surrounding-tables  []
-                 :join-edges          []}
-        expected {:source     {:type "context" :ref "source"}
-                  :operations [["breakout" ["field" (meta/id :orders :product-id)]]
-                               ["append-stage"]
-                               ["aggregate" ["avg" ["aggregation-ref" 0]]]]}]
-    (is (= expected (repair/repair-program-for-context program context)))))
-
-(deftest ^:parallel repair-program-for-context-drops-source-stage-with-fields-over-aggregation-ref-test
-  (let [program {:source     {:type "context" :ref "source"}
-                 :operations [["filter" ["=" ["field" (meta/id :products :category)] "Gizmo"]]
-                              ["with-fields" [["field" (meta/id :products :category)]
-                                              ["aggregation-ref" 0]]]]}
-        context {:source-entity       {:model "metric" :id 83}
-                 :referenced-entities []
-                 :surrounding-tables  []
-                 :measure-ids         []}
-        expected {:source     {:type "context" :ref "source"}
-                  :operations [["filter" ["=" ["field" (meta/id :products :category)] "Gizmo"]]]}]
-    (is (= expected (repair/repair-program-for-context program context)))))
-
-(deftest ^:parallel repair-program-for-context-normalizes-last-quarter-metric-window-test
-  (let [program {:source     {:type "context" :ref "source"}
-                 :operations [["filter" ["between"
-                                         ["field" (meta/id :orders :created-at)]
-                                         ["relative-datetime" -3 "month"]
-                                         "now"]]
-                              ["aggregate" ["count"]]]}
-        context {:source-entity       {:model "metric" :id 42}
-                 :referenced-entities []
-                 :surrounding-tables  []
-                 :measure-ids         []}
-        expected {:source     {:type "context" :ref "source"}
-                  :operations [["filter" ["between"
-                                          ["field" (meta/id :orders :created-at)]
-                                          ["relative-datetime" -1 "quarter"]
-                                          ["relative-datetime" 0 "quarter"]]]]}]
-    (is (= expected
-           (repair/repair-program-for-context (repair/repair-program program) context)))))
-
-(deftest ^:parallel repair-program-normalizes-field-like-map-in-breakout-test
-  (let [program  {:source     {:type "table" :id 1}
-                  :operations [["breakout" {"field_id" 13 "temporal_bucket" "month"}]]}
-        expected {:source     {:type "table" :id 1}
-                  :operations [["breakout" ["with-temporal-bucket" ["field" 13] "month"]]]}]
-    (is (= expected (repair/repair-program program)))))
-
-(deftest ^:parallel repair-program-unwraps-order-by-vector-of-direction-tuples-test
-  (let [program  {:source     {:type "table" :id 1}
-                  :operations [["order-by" [["asc" {"field_id" 13 "temporal_bucket" "month"}]
-                                            ["desc" {"field_id" 14}]]]]}
-        expected {:source     {:type "table" :id 1}
-                  :operations [["order-by" ["asc" ["with-temporal-bucket" ["field" 13] "month"]]]
-                               ["order-by" ["desc" ["field" 14]]]]}]
-    (is (= expected (repair/repair-program program)))))
-
-(deftest ^:parallel repair-program-normalizes-order-by-map-with-direction-key-test
-  (let [program  {:source     {:type "table" :id 1}
-                  :operations [["order-by" {"field_id"        13
-                                            "temporal_bucket" "month"
-                                            "direction"       "asc"}]]}
-        expected {:source     {:type "table" :id 1}
-                  :operations [["order-by" ["asc" ["with-temporal-bucket" ["field" 13] "month"]]]]}]
-    (is (= expected (repair/repair-program program)))))