Compare commits

...

214 Commits

Author SHA1 Message Date
Kyle Doherty
16bbf677a5 Rename nav sections, flatten Library, move Metabot into the page header
- Nav section switcher tabs are now "Library" (repository icon) and
  "Playground" (beaker icon); browse page copy updated to match.
- Library rail content drops its collapsible heading; folders render directly
  (section keeps its "Library" accessible name).
- Metabot button moves from the Playground rail into the app page header,
  between search and the app switcher (hidden in iframes), and reserves its
  32px slot while permissions load so the header controls don't shift.
2026-09-28 15:36:15 -04:00
Kyle Doherty
e1d1d31a1a Polish main-app metric page header and app page header
- App header breadcrumbs render without icons (CollectionBreadcrumbs showIcons),
  match the page's gutter/background via a published page frame
  (setPageFrame replaces setPageBackground), and align vertically with the
  nav rail's Official/Unofficial switcher.
- Metric header: no metric icon; main-app about page title uses h1 size
  (PaneHeader titles gain a size option).
- Main-app metric pages render title and description read-only;
  Data Studio metric pages opt into inline editing (isInlineEditable).
2026-09-28 13:31:54 -04:00
Kyle Doherty
7b91252075 Trim official metric page and drop remote-sync collection styling
Metric page (main app):
- Pills under the chart reuse the Breakdowns logic: first 4 dimensions,
  "Show more" adds 4, curated default stays visible. Breakdowns panel removed.
- Definition panel collapsed by default (DetailPanel gains collapsible /
  defaultCollapsed).
- Dimensions and History panels shown only in Data Studio
  (showManagementPanels).

Remote sync:
- Collections no longer swap to the synced_collection icon; they show their
  own icon (CollectionCaption now honors collection.icon). Removes
  isSyncedCollection plugin, REMOTE_SYNC_COLLECTION, and the isTenantUser
  icon plumbing.
- Removes unsynced-changes dots from all nav collection trees, including the
  CollectionsNavTree plugin and isCollectionDirty. Data Studio tab badges kept.
2026-09-28 13:11:34 -04:00
Kyle Doherty
8dfb8c5b67 Merge remote-tracking branch 'origin/master' into nav-split-prototype
# Conflicts:
#	frontend/src/metabase/browse/models/BrowseModels.tsx
#	frontend/src/metabase/search/components/DropdownSidebarFilter/DropdownSidebarFilter.tsx
2026-09-28 11:09:02 -04:00
Eric Normand
e8f85e28e0 GHY-3274: Initial sync should not attempt to create field values (#82963)
* GHY-3274: Skip the FieldValues scan on the initial sync of a Database

FieldValues are only created on demand, by the /values endpoints. A new Database has none. But the initial sync (sync-database!, which the database-create event runs) still ran the field-values phase. That phase did per-Field app-DB work for every Field: a count and a delete of expired advanced FieldValues, and an exists check before clearing each non-list Field. On a new Database none of that work can find anything.

sync-database! now reads the initial_sync_status of the Database from the app DB when it starts. When the initial sync is not complete, a :full sync runs the metadata and analyze phases and skips the field-values phase. The status comes from the app DB and not from the map that the caller passes, because the metadata phase marks the Database complete, and callers can hold a map from before the first sync. Later syncs, and the scheduled field-values task, run the scan as before.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* GHY-3274: Read initial_sync_status from the database map, not the app DB

sync-database! fetched the Database from the app DB to read its initial_sync_status. Production callers do not need that: the database-create event and advanced-config pass the Database they just created, the sample-data code creates or selects the Database first, and the notify API loads it per request. Only tests passed a map from before the first sync to a later sync. So sync-database! now reads the status from its argument, and saves one app-DB query per sync.

Three places in the tests re-synced with a stale with-temp map and expected the later sync to refresh FieldValues: one-off-dbs/insert-rows-and-sync!, scanning-test, and sync-and-find-values! in the warehouse-schema FieldValues tests. They now select the Database from the app DB before each later sync.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* GHY-3274: Stub the unique task list in the task filtering e2e test

The "filtering should work" test in tools.cy.spec.ts opens the task list with task=field values scanning, and checks that the "Filter by task" picker shows that value. The picker is a Select whose options come from GET /api/task/unique-tasks, and a Select shows an empty input when its value is not an option. The test stubbed GET /api/task but not unique-tasks, so it depended on real task history.

That real task history had a "field values scanning" row only because the initial sync of the Sample Database ran the FieldValues scan. This branch skips that scan on an initial sync, so the name is no longer in the list, and the picker was empty. The test now stubs unique-tasks with the task it filters on, in the same way that it already stubs the task rows.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 15:02:54 +00:00
Denis Berezin
63cbcc92e3 Render the move-questions-into-dashboards modal without a second overlay (#83023) 2026-09-28 19:46:21 +05:00
Denis Berezin
16ad7b75f2 Keep query column order for new table columns (#82476) (#82972)
* Keep query column order for new table columns (#82476)

* Fix CI
2026-09-28 19:45:55 +05:00
Nemanja Glumac
eedda27c89 Cover Mongo convert-to-native on a unit test level (#83011)
* Move Mongo convert-to-native coverage out of e2e

The Mongo e2e test asserted on a specific product row, but the converted
pipeline has no $sort, so row order depends on the Mongo container's
natural order and the row often fell outside the rendered grid.

Replace it with backend tests (no default $limit, nested query over a
converted native card runs, re-converts with its collection and without
Bson wrappers, and the re-converted query runs) and Jest tests for the
Mongo-specific labels and the collection handoff on convert.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Keep the positive collection case next to the negative one

The "no collection" case only proves something if the same fixture can
produce a collection; without the database feature flag, native extras
are silently dropped and the negative assertion passes vacuously.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 16:44:17 +02:00
Nemanja Glumac
1a0ff4340b Replace flaky metric-aggregation e2e test with a unit test (#82964)
* Replace flaky metric-aggregation e2e test with a unit test

The CodeMirror value() e2e helper read the editor text once, so a read
that landed before the initial format settled failed without retrying.
It is now a Cypress query that re-reads on every assertion retry.

The notebook e2e test is replaced by an AggregateStep unit test. Writing
it exposed a blur during the initial format overwriting the formatted
source with the stale empty one; blur is now ignored while formatting.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Read CodeMirror text through invoke queries instead of a custom command

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 15:32:38 +02:00
Nemanja Glumac
0bdad48e91 Command palette e2e spec simplification (#82905)
* Move command palette shortcut coverage down to RTL

Split the shortcuts monolith into a navigation test and a focus-suppression
test, and drop the e2e assertions a unit test can make deterministically:
shortcut modal toggling, c f / c d, the [ navbar toggle, admin tab digits,
the mobile search button, palette action presence, and the snowplow counts.

New unit specs drive the real shortcuts through kbar and assert the router,
the redux modal state and the tracking call. Admin settings links per role
were already covered by the PaletteResults suite, so those two e2e tests go
away as duplicates.

AdminNavbar's spec binds digits against a fixture tab list; the real list's
order, which those digits index into, is pinned next to getAdminPaths.

Dropped Meta+[ outright: toggle-navbar binds "[", and the modifier trigger
is $mod+. handled outside kbar, so the press matched nothing and the
assertion after it passed on the sidebar's pre-existing state.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Fold the documents-in-recents palette test into the main one

The test was kept separate because it needed the bleeding-edge token, which
enabled enough extra features to disturb the main palette test. Documents
moved to OSS: the token activation and the /api/ee/document endpoint went
with it, so the separation costs a cold app boot and buys nothing.

The document is created before the dashboard so the dashboard stays the most
recent entry and the existing selection assertion holds.

Also pin the personal-collection name on the read path in Clojure, which is
the one thing the e2e assertion covers that no backend test did.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Fold the API-order palette test into the main one

Both tests are "real search results render correctly in the palette" and
each paid its own cold app boot. The order check moves into the query block
of the main test, right after the other search assertions.

The searchData alias is registered immediately before its query rather than
at the top of the test: cy.get yields the last matching request, and the
earlier searches in the merged test would shadow it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Anchor the shortcut navigation test on rendered pages

cy.location resolves when a route transition starts, not when it finishes, so
chaining chords on URL assertions alone lets the next chord land mid-mount
and be dropped. Each hop now waits for content on the destination page.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Sync the dashboard bookmark shortcut on its API writes

The o shortcut decides between add and remove from the refetched bookmark
list, so pressing it twice in a row can toggle against stale state and leave
the dashboard bookmarked. Each press now waits for its write, and the
positive assertion between them proves the list refetched before the second.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Set up the palette permissions test through the API

The test needs a non-admin with settings, monitoring, table-metadata and
database-management access; how those grants get made is not what it covers.
Driving the two permission graphs directly replaces a cold boot into the
admin permissions screen, ten UI interactions and two save round-trips.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Wait for the bookmark toggle before the dashboard shortcut

visitDashboard resolves on the dashcard query responses, not on render, so
the o keystroke could land before DashboardBookmark mounted and registered
the shortcut — no request, no toggle. Anchor on the button that component
renders, which is the same thing that owns the shortcut.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Wait for the app bar before opening the palette after sign-in

The palette registers its toggle only once its effect sees the logged-in
user, and the home greeting renders independently of the app bar, so the
ctrl+k could land while the palette was still disabled and be swallowed.
Anchor on the search button, which the same shell renders.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Tighten restricted-shortcut and admin-path filtering tests

Assert the pathname stays put right after a restricted shortcut, and cover
the admin paths reducer dropping paths for a non-admin.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Mount the admin app reducer at its typed store path in tests

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-28 15:31:35 +02:00
Nemanja Glumac
9b193b0937 Replace flaky help text docs link e2e test with a unit test (#82954)
* Replace flaky help text docs link e2e test with a unit test

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Assert the help text docs link click is not cancelled

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 15:14:41 +02:00
Nemanja Glumac
1139be54dd Replace upload-permission e2e tests with unit coverage (#82973)
* Replace upload-permission e2e tests with unit coverage

The "permissions" e2e block restored the postgres-12 snapshot only to
assert upload button visibility, which is covered by backend
can_upload tests and CollectionHeader unit tests. Add the missing
unit case: button hidden when uploads are enabled but canUpload is
false.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Add a missing backend coverage

* Make upload-permission unit tests assert the right cause

The sandboxed can_upload test enabled uploads on the original database
while with-gtaps-for-user! queries a temporary copy, so can_upload was
false because uploads were disabled. Enable uploads inside the gtaps
block and sanity-check with an admin on the same database.

Add CollectionContent cases driving the upload button from the
database's can_upload response.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 15:13:18 +02:00
Nemanja Glumac
32176e152a Keep the Google Drive modal closed until the connection status loads (#83009)
While the connection query was loading, its missing status was read as
"not-connected", so connected admins briefly saw the connect modal. The
modal now renders nothing until the status arrives.

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 15:12:16 +02:00
Chris Vandoodewaard
1d5c08b71d make number of versions we patch nightly configurable (#82766)
* make number of versions we patch nightly configurable

* if 0 should still patch current version
2026-09-28 08:51:21 -04:00
Anna Tomka
bc75a88888 GDGT-2458: Accordion (#82562)
* test(ui): add Accordion Storybook story and Loki filter

* feat(ui): restyle Accordion from Figma

* fix(Accordion): keep theme selectors one class deep so call-site overrides still win

* refactor: drop Accordion call-site overrides now covered by the theme

* test(ui): add Loki reference for Accordion Overview story

* fix(Accordion): hide the default chevron from assistive tech

* test: update DataComplexityCards snapshot for the Accordion separated default

* fix(RemoteSyncDependencyModal): drop overrides the new Accordion theme makes redundant, keep control transparent

* fix(RemoteSyncDependencyModal): let the Accordion theme space the rows

* fix(Accordion): keep icon grey on disabled hover, honour the radius prop

* test(ui): add disabled hover row to the Accordion Overview story

* Update Loki Snapshots

---------

Co-authored-by: Metabase Automation <github-automation@metabase.com>
2026-09-28 10:20:07 +00:00
Voytek Pituła
cd6bcc23e0 [BOT-2138] Centralize shared logic of metabot LLM provider adapters (#82165)
* Refactor metabot LLM provider adapters: centralize shared logic in `adapter.clj` while simplifying  implementations.

* Clarify error message for unsupported adapter models

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* Rework auth, add some types

* drop span cusotmization

* drop extract and headers from CatalogOpts

* fix counts

* revert adapter/context-window-fn

* more tests for headers merging

* normalize-known-model fix

* doc fix

* more strict catalog name resolution

* simplify

* google coverage v1

* registry v1

* registry v2

* simplify

* align bedroc, drop dead ai-proxy?

* fix docstring

* improve tests

* rename listing to model-listing

* rename ai-proxy?

* rename keywords

* improve catalog error handling

* drop redundant content type

* move google auth inside request span

* route vllm's lsit-models through request!

* Update src/metabase/metabot/self/registry.clj

Co-authored-by: Chris Truter <crisptrutski@users.noreply.github.com>

* remove history from docs and comments

* Update src/metabase/metabot/self/registry.clj

Co-authored-by: Chris Truter <crisptrutski@users.noreply.github.com>

* fix var refs in optional/required

* refactor model context-window handling to use registry function

* migrate span attributes to nested :attributes map in adapter and update tests

* standardize function schemas with `mu/defn` and introduce `ResolvedRef` for adapter inputs

* remove redundant comment block in adapter test

* adjust schemas for closed maps enforcment

* reject unsupported AI proxy usage in Azure adapter and expand test coverage

* refactor registry-test to dynamically derive providers and improve adapter validation

* bring back old logging behavior

* stringify `:family` span attribute across adapters and update tests

* fix vllm test name and bedrock reference

* standardize `list-models` function schemas across adapters with `mu/defn` and update related tests

* add schema conformity test for `:supported-models` across adapters

* fix Auth schema

* Get back old tru errors

* Bake-in content-type header

* Expand `ModelListing` schema to include `learned-config` with additional closed map fields for connection reasoning.

* Remove `^:parallel` marker from test to disable parallel execution

* Update src/metabase/metabot/self/claude.clj

Co-authored-by: Chris Truter <crisptrutski@users.noreply.github.com>

* Rename `:learned-config` to `:connection-info`

---------

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Chris Truter <crisptrutski@users.noreply.github.com>
2026-09-28 11:07:48 +02:00
Fraser
381ab6c1a0 Small store batch: QueryModalType, metabot re-export (#79347)
Move QueryModalType and the metabot slice type out of shared redux

Co-authored-by: Romeo Van Snick <romeo@romeovansnick.be>
2026-09-28 21:31:09 +13:00
YB
176d0682e2 Document where action runs are logged (#82742)
document where action runs are logged
2026-09-28 13:18:46 +08:00
Fraser
daa0c256a6 Build Storybook with --test for Loki screenshot runs (#83003) 2026-09-28 11:16:08 +13:00
Romeo Van Snick
2df3f6daa0 Avoid modulejs warning in stylelint (#82988)
* Avoid modulejs warning in stylelint

* Fix eslint errors in stylelint plugin after .mjs rename

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-25 18:21:41 -04:00
Eric Normand
503f96cb55 GHY-4651: SendMeteringEvents does full table scan of query_execution (#83034)
GHY-4651: Let the metering query use the query_execution started_at index

SendMeteringEvents runs every 15 minutes and on every boot. It counts yesterday's query executions with WHERE CAST(started_at AS DATE) = CAST(? AS DATE). Casting the column hides it from idx_query_execution_started_at, so the app DB scans the whole query_execution table. On busy instances that table holds millions of rows.

The query now filters with a half-open range, started_at >= yesterday 00:00 UTC AND started_at < today 00:00 UTC. The DB can answer that range from the index. The db function becomes query-execution-statistics-between, and query-execution-last-utc-day computes the UTC day bounds. The docstring already said UTC day; the old cast used the DB session time zone instead.

The new test in token-check-test runs metering-stats, captures the SQL it sends for query_execution, runs EXPLAIN on the H2 app DB, and checks that the plan uses the started_at index. A second test pins the day boundaries: yesterday 00:00Z is in, today 00:00Z is out.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-25 16:09:44 -05:00
Andrei Simionescu
f2e185369f Always show the Metabot illustrations toggle (#82946)
* Always show the Metabot illustrations toggle
* Only mention the chat sidebar in the illustrations label and docs
* Test the illustrations toggle with the default icon
2026-09-25 21:30:29 +01:00
Eric Normand
9db952d568 GHY-4323: related-tables lookup does a permission query per table and an uncapped IN (#82962)
GHY-4323: Cap related-table ids before the lookup in get_fields

related-tables-by-requested-table collected every FK target of every requested table into one IN clause, then applied the 50-table cap to the result. A table with many FK columns made the lookup ask for tables that the response then dropped. The cap now applies to each table's targets before the lookup, so the IN holds at most 50 ids per requested table.

The :target hydration in the metadata fetch already drops unreadable targets, so only inactive or unbrowsable targets can now shrink the surfaced list below 50.

The other half of the ticket, one permission query per related table, is already fixed. Since 6908f26b76, field/readable-fields-only primes the table perms cache for every FK target while it hydrates :target, and the related tables are exactly those targets. The new API test covers it: as a non-admin with per-table grants and 60 related tables, the call succeeds under the N+1 guard, and it fails with that prime removed.

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-25 13:18:51 -05:00
Chris Truter
9e27bdf31f Take host-part credentials off the pgvector URL (#83010)
* Take host-part credentials off the pgvector URL too

Credentials written before the host (user:pass@host) stayed on the URL.
pgjdbc reads that form as part of the hostname, so the password showed up
in its "unknown host" error and its DEBUG line.

- Move them into the credentials map, whatever the scheme, splitting at
  the host part's last @, with %-escapes decoded and + kept as is. Setting
  one both there and as a param fails at startup.
- Refuse any other unencoded @, which is what an unencoded / or ? in the
  password leaves behind, with a message that quotes none of the URL.
- Rethrow a malformed %-escape without the decoder's message, which quotes
  part of the value.

* Accept an unencoded @ in pgvector URL query values

pgjdbc decodes query values and gives @ no special meaning, so Azure's
documented user=name@server must keep working. Only a host part with no
path after it, the shape an unencoded ? in a password leaves, still
refuses an @ in the query.
2026-09-25 16:07:23 +00:00
Chris Truter
20f3f6cbdc Keep pgvector credentials out of logs (#82930) 2026-09-25 15:22:14 +00:00
Nemanja Glumac
2f3fe9904e Stop ErrorOverview recovery test from racing the search debounce (#83005)
Stop ErrorOverview recovery test from racing the search debounce

Fake timers skip the 500ms debounce so it no longer eats into findBy's
1s budget, and the lookup is scoped to the row so TreeTable's transient
off-screen measurement copy of the cell can't produce a duplicate match.

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-25 12:14:18 +02:00
Nemanja Glumac
57a08becb3 Cover pivot subtotal date localization with a Jest test (#82947)
Pivot a month-bucketed date breakout under a German locale and assert the
subtotal label keeps the localized month. Formatting the value a second time
re-parses "Mai 2026" and yields "Januar 2026", which is the regression the
assertion pins.

Drop the Cypress reproduction: it rendered a pivot of Product Title x month x
User ID to check the same label, and was racing the render.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-25 08:25:36 +02:00
Andrei Chernykh
fc5cf5c469 Bump @snowplow/browser-tracker to 4.10.2 (#82955)
* Bump @snowplow/browser-tracker to 4.10.2

* Fix SDK analytics spec for batched Snowplow requests
2026-09-24 14:18:19 -07:00
bryan
c03c02a9a1 Lint values, not call style, and write the value-binding rubric down (#82686)
* Lint values, not call style, and write the value-binding rubric down

Two changes to the value-binding sweep (GHY-4477), both correcting guidance that
earlier namespace PRs were written against.

Stop asking for query maps. The linter's `lint-kv-args!` check asked every
`:column value` call to be rewritten as a query map, on the premise that a query
map is the shape the HugSQL extraction in step 2 reads off. That premise does not
hold. Both extraction spikes show it: the glossary spike (#82530) extracted
cleanly *from* kv-args, and the SSO spike (#82587) still needed substantial
rewrites *from* query maps -- a `SELECT 1 ... LIMIT 1` reshape and empty-collection
sentinels. Extraction is semantic. It turns conditional clauses into flags, picks
portable SQL, and discards the HoneySQL shape entirely (`h2x/like-substring`'s
`[:escape ...]` form becomes plain `ESCAPE '!'` text). A query map is not closer
to the target, just a different thing to throw away.

Converting also introduced a bug class. Toucan applies a model's `deftransforms`
`:in` fn only in kv-arg position -- it hooks `query/apply-kv-arg`, and a raw
`{:where ...}` map never passes through there (toucan2 calls HoneySQL "an outlet
to bypass type transforms"). So moving a keyword value into a where map drops the
transform, and HoneySQL formats a keyword in a value slot as an *identifier*:

    (t2/select :model/SearchIndexMetadata :engine :appdb)
    ;; => ["... WHERE ENGINE = ?"  "appdb"]          transform ran

    (t2/select :model/SearchIndexMetadata {:where [:= :engine :appdb]})
    ;; => ["... WHERE ENGINE = \"APPDB\""]           transform skipped, broken

That shipped twice on sweep branches before being caught (`sync/db.clj`'s
`:semantic_type :type/Name`, `collections/db.clj`'s `:type :api-key`). It compiles,
runs, and returns wrong rows. Nothing static can check the hand-written
post-transform value, so the check that produced these conversions is removed
rather than narrowed.

Lint kv-arg values instead. Removing the style check on its own would have left a
real gap: `value-nodes` only understood query-map clause shapes, so a value passed
as a kv-arg pair was never examined for a missing marker. `lint-unmarked-values!`
now walks kv-arg pairs too, starting after the `:model/...` keyword rather than at
a fixed offset (several fns take an argument before the model, e.g.
`(t2/select-one-fn :value :model/Setting :key k)`). Inside a kv-arg the column
comes from the pair key, so every argument of an operator form is a value --
`:id [:in ids]` checks `ids` -- unlike a query-map clause where the first argument
is the column.

Net effect on findings: call style is no longer reported; an unmarked value is
reported whichever style it was written in. Verified against
`settings/db.clj` (kv-arg style, opted in temporarily): both `setting-key` reads
are now flagged, and the `value` in `update!`'s changes map is correctly not.
`content_translation/db.clj`, the merged worked example, stays at zero.

Write the rubric down. docs/developers-guide/value-binding-rubric.md records every
decision the sweep needs, so the remaining namespace PRs stop re-litigating them in
review. Seven ordered rules, each with an example; the two that were previously
implicit are now explicit -- never mark a literal (it cannot carry a request value,
and in step 2 it becomes a HugSQL hole every caller fills with the same constant),
and leave a possibly-empty collection unmarked (Toucan rewrites `[:in col []]` to
`false` inside the step the marker wraps). It also records the `t2/update!` shapes
that work, what `value_guard` refuses for you, and the known limits the lint cannot
see -- so "zero findings" is replaced by "every finding explained by a rule".

* Frame the call-style rules as scaffolding that HugSQL extraction retires

A HugSQL query fn takes one map and has no override surface, so the
kv-arg-vs-query-map question -- and the transform-position trap that comes with
it -- does not exist for an extracted namespace. Say so, rather than leaving
these rules to read as permanent house style.

* See value slots inside conditionally built clauses

value-nodes stopped at a list node, so any value a when/if/cond-> built was
invisible to the unmarked-value check -- a false negative in a security lint,
and a common shape: permissions/db.clj and collections/db.clj both assemble
where clauses that way.

    (t2/select :model/Setting {:where [:and (when k [:= :key k]) [:= :o o]]})
    ;; o was flagged, k was not

Walking list-node children finds it. This surfaces real unbound values in four
namespaces (dependencies 12->17, sync 7->10, collections 5->7, search 5->6);
they are bound in the namespace PRs stacked on this branch.

* Correct the transform section: the call style breaks it, not the marker

The section implied that marking a value on a transformed column is what
produces the identifier comparison. It is not. value_guard lifts the
[:auto/param column v] 3-arity from inside apply-kv-arg, so a MARKED kv-arg
still runs the transform -- verified against :model/SearchIndexMetadata, the
bare and marked kv-arg forms compile to byte-identical SQL.

Only the where-map form skips the transform. So a marker on a transformed
column in kv-arg position is redundant rather than dangerous, and the reason to
leave it off is that the transform already binds the value.

Caught by a reviewer that compiled all three forms rather than taking the
section at its word.

* Rewrite the transform and written-value rules as single explanations

The transform correction was bolted onto the end of the section, so it stated
the rule, then contradicted its own framing, and showed the same example twice.
Rewritten as one block that shows all three forms together -- kv-arg bare,
kv-arg marked, query map -- so the variable is visible: the call style is what
takes a value off the transform path, not the marker. Adds the converse case a
sweeper actually hits: a call that was ALREADY a query map on master must carry
the post-transform value written out.

Rule 1's example showed a t2/update! CONDITIONS map while the prose was about
the changes map, which is the confusion the rule exists to prevent. Replaced
with a real settings/db.clj call where the condition is marked and the changes
map is bare, plus an explicit note that only the trailing changes map is exempt.

Also corrects the stated mechanism. The reason not to mark a written value is
that the column's `:in` transform runs on the marker and persists it as data --
mi/transform-encrypted-json is an :in transform, so an encrypted column
encrypts the marker. The earlier text attributed this to a
define-before-insert hook on :model/Setting, which does not exist; Setting's
encryption is application code plus a column transform.

* Refuse a marker outside a value slot, and see two more shapes

Three findings from review, all verified against a running app DB.

The guard was silent on the one case it most needed to catch. `auto-param`
lifts a marker wherever it sits, so one written in a `:select`, `:from`,
`:order-by` or `:group-by` clause became a bound parameter in a slot HoneySQL
wanted an identifier for -- compiling to the literal identifier `PARAM` and
discarding the value, with nothing downstream able to notice:

    (t2/query {:select [[:auto/param "name"]] :from [:setting]})
    ;; => ["SELECT \"PARAM\" AS \"P3R1MO8JB3WD0J\" FROM \"SETTING\""]

`check-marker-placement!` now refuses that as `::marker-outside-value-slot`. It
stops at a nested query map, because a subquery's own clauses are checked on
their own terms -- `t2/exists?` wraps the query in `:select [[[:exists {...}]]]`
whose inner `:where` is a real value slot, and scanning the outer `:select`
wholesale rejected it. A join's ON condition is likewise a value slot, so only
the table halves of a join clause are checked.

The lint could not see a conditions map. `(t2/update! model {:col v} changes)`
filters on `v`, but it is not a `:where` clause, so neither walker descended it
-- and the rubric recommends that exact shape, so a namespace written that way
passed vacuously. Now walked for `update!`, `update-or-insert!` and `delete!`,
taking only the map immediately after the model (a later map is the changes map,
exempt by rule 1). An operator form there holds only values, since the column
comes from the map key.

The lint could not see into a subquery. The operator branch returned its
arguments without descending, so `[:in :id {:select ... :where [:= :z z]}]` left
`z` unchecked. It now keeps each value and descends into it; descending alone
would have dropped the plain-token case.

Rubric corrections from the same review, each verified rather than reasoned:
rule 2's failure mode (marking a column keeps the comparison and binds the
column NAME, it does not drop it), rule 5 (`:not-in` gets no empty-collection
rewrite, so an empty one emits invalid `NOT IN ()` marked or not; rule 4 wins
for collections of ids; a non-empty proof must be local), rule 4 (`long`
truncates a non-integral number rather than throwing), rule 1 (a written value
is not the injection shape this project targets, rather than never structural),
rule 7 (booleans from outside are values too), the transform mechanism
(`apply-kv-arg` maps the transform over an operator form's tail, which is why a
marked kv-arg keeps it -- nothing to do with `value_guard`), and a new
false-negatives subsection, because the rubric previously described the lint as
only over-flagging.

* Scan subqueries for misplaced markers, and allow one inside an expression

Two holes in check-marker-placement!, both verified by compiling the form.

It only scanned the top-level map, so the silent case survived one level down:

    {:select [[[:exists ^:allow-subquery
                {:select [[:auto/param "name"]] :from [:setting]}] :e]]}
    ;; => SELECT EXISTS (SELECT "PARAM" AS "PACGBLW31WM3R" FROM "SETTING")

No throw, value dropped -- exactly what the check exists to prevent. The
docstring, the test, and the rubric all claimed a subquery was "checked on its
own terms", which was not implemented; nothing re-ran the clause scan on a
nested map. warehouse_schema/db.clj:317 already uses ^:allow-subquery with
:exists, so the shape is live. It now walks every nested map and scans each
one's clauses.

It also refused legitimate value slots. A comparison inside an identifier
clause is a real value slot, and both of these threw at compile:

    {:select [[[:= :engine [:auto/param "h2"]] :is_match]]}
    {:order-by [[[:case [:= :a [:auto/param 1]] 1 :else 2] :asc]]}

So a converted namespace would break the moment it grew a computed projection
or a CASE sort key. `:join` already needed a carve-out for its ON condition;
this is the same need, and the file already recognizes the shape via
`operator-form?`, so the scan descends past a keyword-headed form.

Pre-existing failures in metabase.app-db.custom-migrations-test are unchanged:
24 on origin/ghy-4477-rubric-foundation, 24 with these fixes.

* Classify identifier slots positionally, and close four lint blind spots

Findings from four independent reviews of this PR, each verified by compiling
the form or by linting a probe rather than by reasoning.

The guard missed its own target case. `marker-in-identifier-position?` decided
what was a value by head shape, but HoneySQL tells `[expr alias]` from
`[:= a v]` POSITIONALLY. So `[:t [:auto/param "al"]]` read as though `:t`
headed an operator form, and the alias slot -- the shape the check exists to
catch -- passed:

    {:from [[:t [:auto/param "al"]]]}    => "SELECT * FROM t AS param k"
    {:select [[:a [:auto/param "al"]]]}  => "SELECT a AS param k FROM t"

Now an entry is destructured as `[expr & aliases]`: index 0 may be an
expression and is descended, everything after it is a name and is refused.

`:order-by` and `:group-by` are no longer refused. HoneySQL binds a param in
both (`ORDER BY ?`), so a marker there is a pointless no-op, not a dropped
value -- and the test asserted the throw as correct, pinning a premise HoneySQL
contradicts. Refusing it would have turned a harmless mistake into a
production exception.

`:cross-join` no longer takes the alternating path. It has no ON condition, so
`take-nth 2` skipped every odd-indexed table; a marker there was missed.
Only the four join clauses that actually pair a table with a condition use it.
Added `:using`, `:create-table`, `:drop-table`, `:truncate`, `:partition-by`,
`:window`, `:select-top`.

The scan now short-circuits on `contains-marker?`. It runs on every app-DB
query at compile, and almost none carry a marker. 18.0us -> 13.7us on a
join-heavy query with no marker.

Four lint blind spots, all with live call sites:

  `[:!= :col v]` / `[:<> :col v]` were unchecked -- `value-operators` listed
  only `not=`. collections/db.clj:283 passes a request `user-id` through `:!=`
  in a namespace a stacked PR reports as converted. Added `!=`, `<>`,
  `is-distinct-from`, `is-not-distinct-from`, `not-between`.

  A value inside a function-call form -- `[:= :k [:lower v]]`, which compiles
  to `LOWER(?)` -- was dropped by the `:else` branch, whose token children
  return nothing on their own. Live in audit_app/db.clj. Same fix reaches a
  literal collection: `[:in :k [a b]]` now checks both elements.

  A two-arity `t2/update!` had its CHANGES map read as conditions. Toucan's
  arglist ends in the changes map, so a lone trailing map is changes --
  remote_sync/db.clj's mark-all-rsos-synced! was flagged for a written value,
  contradicting rubric rule 1. `delete!` has no changes map, so its lone map
  is still conditions.

  A query map passed where conditions go was walked twice, reporting every
  value twice. Live in mcp/db.clj, metabot/db.clj, testing_api/db.clj.

Findings are now deduped by source position, which also covers a multi-arg
operator in a kv-arg: `:d [:between lo hi]` reported `hi` twice, because
`value-nodes` misreads the vector as a clause while `kv-arg-value-nodes` reads
it correctly. Deduping is robust where keeping the walkers disjoint is not.

Also: the marker payload is no longer descended (a marked value was reported as
though bare once the `:else` branch started yielding children), `ex-data`
carries the query so a dynamically-built one can be traced, and `args` is no
longer shadowed inside the conditions-map walker.

* Bring the rubric's guard and false-negative sections up to date

The guard section described the head-shape classifier and claimed :order-by /
:group-by were refused; both changed. The false-negative list still carried the
conditions-map, function-call-form, literal-collection, operator-alias and
subquery shapes as open, which are now covered -- and was missing the two that
remain open, a positional pk and a non-literal model.

* Drop the unused clojure.string require and the bang from check-marker-placement

Two clj-kondo failures on the previous push, both mine.

`clojure.string` became unused when the join carve-out stopped matching clause
names by suffix.

`check-marker-placement!` only throws -- it mutates nothing -- but the bang made
`:metabase/validate-deftest` read every call in a `^:parallel` test as a
destructive operation. Renamed rather than whitelisted, since the name was
wrong: the convention marks state change, and this is a predicate that refuses.

* Don't ask for a marker where the guard refuses one

`value-nodes` walked every clause of a query map, and the `:else` vector branch
yields each child as a candidate value. Together those reported symbols sitting
in clauses that name columns and tables, and the message always says the same
thing: write it as `[:auto/param x]`. In a `:select` or a `:from` that advice is
not merely noise -- `check-marker-placement` in this same PR throws
`::marker-outside-value-slot` when you follow it, so the two halves of the
mechanism contradicted each other.

A query map's clause values are now split. A clause that names things
contributes only what is nested inside it: a subquery's own clauses, and the
arguments of a keyword-headed comparison. Everything else is unchanged.

Measured over `src/metabase` and `enterprise/backend/src`, with the linter
forced on for every namespace: 1981 warnings to 1963. Eighteen removed, none
added. All eighteen are identifier slots -- `output-table-id`,
`table-wildcard`, `sort-direction`, `order-column`, `id-column`, `cast-form`,
and a `union-query` alias. Every value-carrying shape survives: a computed
projection, a `CASE` sort key, a subquery nested in `:from`, a subquery in an
`:in`, a plain where clause, a kv-arg.

`:order-by` and `:group-by` are in the set even though the guard deliberately
allows a marker there. A marker in `:order-by` compiles to `ORDER BY ? ASC`,
which loses the ordering, so asking for one is still wrong advice.

Two related shapes stay open, both false positives that ask for a marker in a
position that holds no value:

- A clause built with `assoc` or `cond->` is not a map in the source text, so
  the walker still reads it as a plain vector. `dependencies/db.clj:492` keeps
  its three `sort-direction` warnings.
- A binding vector inside a `let` or a `for` is walked like a collection
  literal, so the binding names are reported.

Neither is urgent: the linter is opt-in per namespace, the noise surfaces one
namespace at a time during a conversion that reads every warning, and the
rubric's gate is already that a rule or a limit explains each one.

* Allow a marker at index 0 of an expression entry

In :select, :select-distinct, :select-top, :returning and :partition-by,
index 0 of an [expr alias] entry is an expression, and HoneySQL binds a
param there (SELECT ? AS a). The guard refused it as an identifier.

The table clauses also bind a param at index 0 (FROM ? AS a), but a
table cannot be a parameter, so the guard still refuses a marker there.

* Refuse a marker that is the whole identifier clause value

When the value of a clause was itself a marker, as in :from [:auto/param "t"],
the guard split it into :auto/param and the payload. Neither is a marker, so
the guard passed it, and it compiled to FROM param, k with the value dropped.
Treat a marker value as a single entry, ahead of the join branch too.

* Remove the value-binding rubric from the repo

A reviewer asked that it not live in the repo.

* Give the lint the clause context it was guessing at

Two fixes, both places where the lint's judgement of a slot did not match
`value-guard`'s.

**Split the expression-entry clauses.** `identifier-clauses` was one flat set,
so the lint stayed silent about every position in `:select`, `:returning` and
friends. The guard no longer agrees: its `expression-entry-clauses` treats index
0 of an `[expr alias]` entry as a value slot, and `{:select [[[:auto/param "a"]
:k]]}` compiles to `SELECT ? AS "K"` with the value bound. The silence there was
a false negative -- a request value in a computed projection reached the
statement text with nothing to report it.

Index 0 of an entry is now a value slot. Everything after it is a name, and a
bare entry is a name, which matches what the guard refuses:

    {:select [[:auto/param "a"]]}        refused   (bare entry)
    {:select [[:key [:auto/param "a"]]]} refused   (alias)
    {:from   [[[:auto/param "t"] :x]]}   refused   (table)
    {:select [[[:auto/param "a"] :k]]}   binds     SELECT ? AS "K"

**Carry clause context through `assoc`.** A clause written outside a map literal
reached the walker through the list branch, which keeps no record of anything,
so the name-slot check never ran. The lint then asked for a marker on a column
name and on a sort direction. Both fixes it named break the query, and one
breaks it in silence: marking the direction throws a ClassCastException, and
marking the column compiles to `ORDER BY ?` and loses the ordering with nothing
at runtime to catch it.

A clause key inside an `assoc` call now routes its value the way a key in a map
literal does. `cond->` is covered because the walk descends into the nested
`assoc`.

Measured over `src/metabase` and `enterprise/backend/src`, with the linter
forced on for every namespace: 1963 warnings to 1954.

Fifteen removed, all of them the `assoc` case: `sort-direction` and
`sort-column` in task_history/db.clj:174, cache/db.clj:102 and
dependencies/db.clj:492, `column` in queries/db.clj:53, models/db.clj:152 and
:171 and documents/db.clj:58, and `table-level-case` in permissions/db.clj:89.

Six added, all of them expression entries the flat set had been hiding:
`output-table-id` in data_studio/db.clj (three sites), `cast-form` in
app_db/db.clj:20, `domain-expr` in internal_stats/db.clj:206 and
`admin-group-id` in permissions/db.clj:681. Each holds a column reference today,
so each is a rule 2 finding to record rather than a value to mark -- but the
lint can no longer tell, which is the point.

Still open, and not addressed here: a binding vector inside a `fn` or a `let` is
walked like a literal collection, so its names are reported; `:order-by` and
`:group-by` are absent from the guard's set, where a bare marker entry drops the
value; and the guard's DDL keys are still missing from the lint's set.

* Give the rubric comments their own reasons

The rubric left the repo, so five comments cite a file a reader cannot open.
"Rubric rule 1" was a useful short form while it was here; now it asks the
reader to go looking in the git history.

Each one now states the reason instead of pointing at it. A written value is
never marked because the column's `:in` transform runs on the marker itself and
stores it as data, and the conditions-map shape is checked because the sweep
leaves it in place.

Comments only. No behaviour changes.

---------

Co-authored-by: Eric Normand <ericwnormand@gmail.com>
2026-09-24 21:05:55 +00:00
Kevin Kelleher
af8ca49a25 Prevent showing stale exploration summary documents (#82442)
avoid fetching stale document while a save is ongoing
2026-09-24 15:32:23 -04:00
Kevin Kelleher
cfb8cbd6f9 Hide exploration error pages for unrequested query variants (#82450)
filter out failed variant pages
2026-09-24 15:32:11 -04:00
Eric Normand
3eff0fac3d GHY-4213: Remote Sync: importing a GUI model before the target's database schema sync completes stores untyped columns (#82945)
GHY-4213: Infer untyped model metadata after a metadata sync

A GUI model that is deserialized (for example by a Remote Sync import) before the target has synced its tables cannot infer its result metadata. The import then stores only the exported model overrides, with base_type type/* and no field ids, and nothing revisits it. Filters on those columns offer only "Is empty" and "Not empty".

After each metadata sync of a database, look at the unarchived MBQL models on that database. For each model whose stored metadata has no known base type, infer the metadata again, keeping the model overrides. A model that still cannot be inferred is left as it is. The update goes straight to the app DB, so it publishes no card-update event and does not mark remote-synced content dirty.

Tests: an API-level test imports a GUI model through POST /api/ee/remote-sync/import before its table is synced, runs POST /api/database/:id/sync_schema, and checks the column types in GET /api/card/:id. Contract tests cover untyped-metadata? and backfill-untyped-model-metadata!.

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-24 14:16:53 -05:00
metamben
b6f625ed4b Use jdbc-type for SQL Server user defined types as fallback (#82860)
Closes #62335
Closes QUE2-143
2026-09-24 17:49:26 +00:00
Eric Normand
bb9482f0fd GHY-4399: Remote Sync doesn't clean up nested content when a parent collection is archived, so orphaned files accumulate in git forever (#82876)
* GHY-4399: Cascade collection archive to nested content in the remote sync ledger

Archiving a collection archives its whole subtree in bulk SQL and publishes one collection-update event for the top collection only. The remote sync collection handler marked only that collection as delete, so the next push removed its own file and left every nested collection, card, and other item in the repo for good.

The handler now brings the ledger rows of the subtree in line with each entity archived state when the collection is archived, or when its own row was a pending delete (it was unarchived before a push). Rows of archived entities become delete. Delete rows of entities no longer archived become update, so restoring a folder before a push does not delete its files.

subtree-expr matched descendants by the prefix /<id>/, which is correct only for root collections. It now uses the location the children have (location + id + /), which is the same string for roots and is also correct for nested collections.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* GHY-4399: Cascade collection archive to transforms in the remote sync ledger

The cascade marked a subtree row as delete only when the entity itself was archived. Transforms have no archived column, so a transform in an archived Transforms folder stayed synced, and the next incremental push left its file on the remote. This is the first repro in the ticket.

A row now also counts as archived when the collection it is in is archived. Unarchiving restores it to update when neither the entity nor its collection is archived and the entity still exists.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-24 11:59:46 -05:00
Andrei Chernykh
8317274709 Migrate AutocompleteInput dropdown off Emotion (#82948) 2026-09-24 09:46:29 -07:00
Sébastien
c796ec01ec Handle both list and search default-value pickers in embedding e2e test (#82727) 2026-09-24 12:35:20 -04:00
Eric Normand
fa7362a1e4 GHY-4554: Mark untrusted data in MCP v2 tool results with an unforgeable boundary (#82879)
* GHY-4554: Mark untrusted data in MCP v2 tool results with an unforgeable boundary

Tool results put instance data (card names, warehouse rows, column comments) next to server prose, with nothing to mark where the data stops. A planted instruction in a name could pass for an instruction from Metabase.

message/data holds data from outside the server. render writes it as JSON between <data boundary="..."> and </data boundary="..."> lines that carry a random 32-hex-digit value, which the data cannot predict and so cannot close early. The initialize instructions explain the convention in one line.

message/json-text encodes JSON and then writes invisible, line-breaking, private-use, and unassigned code points (Cc, Cf, Zl, Zp, Co, Cs, Cn) as \uXXXX escapes. Decoded values do not change. The transport encodes JSON-RPC responses with it, so structuredContent is escaped on the wire too.

Every (message/raw (json/encode ...)) site and common/success-content now use message/data. Tool tests read the text through test-util/strip-data-boundary.

* GHY-4554: Make question_write and transform_write text-only

Both tools sent their payload as structuredContent as well as in the text block. Claude Code shows the model structuredContent in place of the text block, so the model saw the payload with no data boundary and with invisible characters raw. No client reads their structuredContent, so they now answer text-only.

* GHY-4554: Say to keep untrusted content out of structuredContent

A client may show the model structuredContent in place of the text block, and there the data has no boundary and its invisible characters decode raw.

* GHY-4554: Put glossary lookup results inside a data boundary

Glossary terms and definitions are user-written free text. The glossary() listing already put them inside the per-response data boundary, but glossary(term) quoted them into server prose. The lookup now returns its matches in the same data envelope as the listing, so a planted closing tag or instruction reads as data.

* GHY-4554: Make strip-data-boundary strict

The test helper returned text with no data boundary unchanged, so a tool that stopped wrapping its result still passed its suite. It now throws, and the suites that read error results through the same helper read those raw. With the renderer changed to emit bare JSON, every MCP v2 tool suite now fails.

* GHY-4554: Extend the never-follow rule to quoted values

Server prose quotes every value it did not write, so a planted instruction can reach the model inside a quoted value as well as inside a data boundary. The initialize instructions now name both.

* GHY-4554: Name the sliced table by id in the get_fields paging line

The paging line put the warehouse-controlled table name in server prose, outside the data boundary, and printed null when a fields projection dropped the name and id. It now names the table by the id from the source row and gives the exact next call, so the name stays inside the data boundary and the agent always knows which table to page.
2026-09-23 21:21:22 +00:00
Eric Normand
361fc91d80 GHY-4636: MB CLI doesn't have flag for expected branch, git-sync import fails with expected_branch: missing required key, received: nil (#82874)
GHY-4636: Let remote-sync import run without expected_branch

The mb CLI calls POST /api/ee/remote-sync/import with only a branch. GHY-4019 made expected_branch a required key, so every CLI import failed schema validation with 'expected_branch: missing required key'.

expected_branch is a compare-and-swap guard against a stale browser tab. A headless client has no cached view that can go stale, so the key is now optional. The endpoint runs the guard only when the client sends the key. The web UI still sends it, so its behavior does not change.

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-23 14:49:22 -05:00
Nemanja Glumac
f4ab93af92 Extract the backend profiling trigger into a single entry point (#82854)
* Give on-demand test profiling a single entry point

backend.yml, app-db.yml, drivers.yml and semantic-search.yml each carried their
own workflow_dispatch block, their own PROFILE_TESTS/ONLY_TESTS env block, and a
selector input, all to support the same thing: run one leg of one suite on
demand, under a profiler, optionally narrowed to named tests.

profile.backend.yml now owns the only workflow_dispatch and calls the four
unchanged. They declare nothing for this feature: inside a called workflow
`github.event` is the caller's payload, so they read `target` and `tests` off
`github.event.inputs` directly. On a normal CI run those are absent and render
empty, so every gate's `== ''` clause holds and each suite behaves as before.

One `target` dropdown rather than a suite plus a leg keeps the gates at their
existing two clauses, and disambiguates `postgres`, which names a leg in three
suites.

Also fixes two ways a profiled run produced no flamegraph:

- test-driver built `:profile-tests` into BASE, but BASE is used only on the
  ONLY_TESTS branch. Profiling without a test selection ran unprofiled while
  still reporting profiled=true. The alias now goes on ALIASES too, which the
  other three branches use.
- The multi-JVM path runs each JVM from a sibling directory, so the profiler's
  relative output dir resolves per JVM. merge_outputs copied back junit and
  logs but not the flamegraphs.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Collect profile output from multi-JVM test runs

collect-profile predates the multi-JVM test path and reads only the layout a
single-JVM run produces. A partitioned run nests its junit XML one directory per
partition under target/junit, and merges one logs/test-log-p<index>.json per
JVM, so the non-recursive `target/junit/*.xml` glob and the exact
`logs/test-log.json` path matched nothing: an empty wall-time summary, and an
artifact holding the flamegraphs alone.

The two summary greps become `grep -r --include='*.xml'`, which reads both
layouts without needing globstar, and the upload paths become globs matching
test-driver's own failure-log upload.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Simplify inputs

* Use `test` namespace

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-23 18:40:49 +00:00
Timofey Kachalov
6c4b0182c1 Fixed custom actions from the mapQuestionClickActions plugin not appearing in EditableDashboard click action popover (#82832) 2026-09-23 22:33:12 +04:00
Phoomparin Mano
0694a11c90 Fix typed schema action template tag handling (#82488)
Co-authored-by: Timofei <timofei@metabase.com>
2026-09-23 19:41:59 +04:00
Egor Grushin
fa20179ff6 Restyle Button to the Figma spec and migrate call sites (#82236) 2026-09-23 16:59:13 +02:00
Riley Thompson
cdc7f386af Compile mbql stages to CTEs for postgres (#81914)
* initial commit

* Use CTEs for every multi-stage Postgres query

Drop the "standalone statement" gate and the native/persisted-stage and
join-source fallbacks. Postgres accepts WITH everywhere we embed a compiled
query (JOIN parens, CTE bodies, card-reference subqueries, CREATE TABLE AS,
INSERT INTO), so the only remaining condition is the driver opting in and
the query having more than one stage.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix changelog

* simplify with mapv

* comments

* Build CTE tests with lib and assert results only

Rewrite the four multi-stage tests added on this branch to build their
queries with lib/query and a metadata provider instead of mt/mbql-query,
and drop the compiled-SQL string assertions so they only check that each
query runs on Postgres and returns the expected rows.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* trim comments

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-23 14:09:50 +00:00
github-automation-metabase
d994a365f7 Auto-update documentation for master (#82878)
Update generated docs (master)

Co-authored-by: github-actions <github-actions@github.com>
2026-09-23 09:40:25 -04:00
Phoomparin Mano
d7b9a71698 Fix including library sub-collection throws an error in typed schemas (#82513)
Co-authored-by: Timofei <timofei@metabase.com>
2026-09-23 17:34:22 +04:00
YB
12a22c0392 [GDGT-3229] read ClickHouse indexes from SHOW CREATE TABLE instead of system tables (#82573)
ClickHouse denies SELECT on system.data_skipping_indices, so index verification failed for every request (the index itself was created). fetch-table-indexes now reads one SHOW CREATE TABLE and parses its INDEX and ORDER BY lines. A failed warehouse read now carries the driver's message on the request and as warehouse_error on the indexes list, with an alert on the indexes page.
2026-09-23 21:14:23 +08:00
Ryan Laurie
6ec07f7518 Pin to clickhouse 26.8 (#82868)
pin to clickhouse 26.8
2026-09-22 22:28:11 +00:00
Fraser
aa9c3797e8 Move the embed setup modal state into the embedding module (#81656)
Co-authored-by: Romeo Van Snick <romeo@romeovansnick.be>
2026-09-22 18:27:38 -04:00
Eric Normand
8ce3936116 GHY-4642: Renaming a collection and exporting via Remote Sync doesn't update repo paths for content in renamed collection (#82865)
Remote sync: full export when a synced collection is renamed or moved (GHY-4642)

A collection path is also the directory of its contents. The incremental export wrote the renamed collection file and deleted the old one, but left every descendant file under the old directory, and kept their stale file_path values. Later incremental pushes did not repair it.

An update to a Collection whose stored path differs from its new path now falls back to a full export, which rewrites every path and records each file_path. This matches the import side, which already does a full import for any Collection change. In-place collection edits that keep the path stay incremental.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-22 14:11:38 -05:00
Oleksandr Yakushev
5bcd89d33f [pivot] Optimize BE+FE pivot generation (#82811) 2026-09-22 21:22:53 +03:00
Eric Normand
916ee6554c SEC-1208: Login throttle failures crash with fk_login_history_session_id FK violation, causing self-reinforcing lockout (#82859)
SEC-1208: test that a concurrent session delete cannot fail a login

Add an API test for POST /api/session. Between the session insert and the login_history insert, another connection deletes the user's sessions, as a password change does. The test logs in 11 times like this. Each login must return 200. Then a normal login must work, and GET /api/login-history/current must show all 12 logins.

The bug is already fixed. Commit 4a7f73029a (#80718, backported to 61 as #80962, first in v0.61.20.1) writes the session row and the login_history row in one transaction. The reporter ran v0.61.1, which does not have it. When that transaction is removed, this test fails as the ticket describes: each login returns 401 with an fk_login_history_session_id violation, and after 10 of them the throttle returns "Too many attempts" for the correct password.

The test does not use mt/with-temp for the user. with-temp binds a transaction connection, and the test client conveys it into the request, so the delete could not see the login's rows. For the same reason, the delete runs on a plain Thread, not a future.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-22 11:33:28 -05:00
Kyle Doherty
401167cc00 Merge branch 'master' into nav-split-prototype 2026-09-22 06:20:55 -04:00
github-automation-metabase
fff70175e0 Update translations 2026-09-22 (#82845)
* Update translation sources 2026-09-22

* Update translations

---------

Co-authored-by: github-actions <github-actions@github.com>
2026-09-22 06:21:08 +00:00
Alexander Polyankin
74216b3098 Transform tests batch 2 (#82770) 2026-09-21 16:17:38 -04:00
Kyle Doherty
2947b47c23 Select Library tables in the Official rail
A published table has no page of its own — the rail links it as the
ad-hoc question that reads it, /question#<serialized card> — so there was
no card id to key a selection off and the row never highlighted.

Recover the table from the serialized card in the hash, load it, and
select it. Its collection also feeds the section seed, so landing on or
reloading a table URL stays on Official, matching metrics.

The hash is user-controlled, so its shape is checked rather than
asserted; anything unrecognised just leaves the rail alone.
2026-09-21 15:41:57 -04:00
Kyle Doherty
abb710334f Keep Library content out of the open rail
Opening a metric pushed it into the Unofficial rail's open list. The
Library is a curated reference you navigate to and read, not a piece of
work you open, and it already has its own home in the Official rail.

Skip any card or dashboard whose collection is part of the Library.
Dashboards cannot live there today, but the rule is about the Library
rather than about metrics, so it is applied to both.
2026-09-21 15:36:48 -04:00
Kyle Doherty
fed0dbfd58 Read the metric route's card id from the right param
The metric routes declare their param as :cardId, but the navbar read
params.slug for every card path. On /metric/:cardId that was undefined,
so the card was never fetched — which meant no collection to derive the
nav section from and nothing to select, leaving a reload on Unofficial
with nothing highlighted.

With the card loaded the rest of the chain already worked: the seed
resolves to Official off the library collection, the selection resolves
to metric-<id>, and the tree auto-expands the ancestors of the selected
node, so the library collection opens with the metric highlighted.
2026-09-21 15:34:36 -04:00
Kyle Doherty
edd44ef851 Seed the nav section from what is open, and select metric pages
Reloading an official metric dropped the rail to Unofficial: the section
override lives in redux, so a refresh wiped it and the fallback keyed off
the pathname, which says nothing about authority. /metric/... is not
/browse, so it resolved to Unofficial.

Derive a seed from the collection of whatever is open, taking Official
when that collection is marked official or is part of the Library.
Precedence is now explicit choice, then the open entity, then the URL.

getSelectedItems ignored metric paths entirely, so nothing was selected
on a metric page. Handle them alongside questions and models, and carry
the search model on the selection so the Official rail, whose rows are
keyed model-id, can highlight the open entity rather than only its
containing collection.
2026-09-21 15:27:25 -04:00
Kyle Doherty
1cd4d62061 Make the Official/Unofficial switcher a view-only toggle
Switching sections navigated to whatever you last had open in the target
half, so flipping the rail threw away the page you were on. The rail is
a view of the sidebar, not a destination; toggling it should only change
what the sidebar shows.

Drop the navigate call. That makes hrefFor, the per-section last-path
map and the effect feeding it dead, so they go too. The URL still seeds
the initial section for a deep link.
2026-09-21 15:18:50 -04:00
github-automation-metabase
0509e1a5e2 Tighten ratchets (#82829)
Co-authored-by: github-actions <github-actions@github.com>
2026-09-21 18:27:18 +00:00
Oleksandr Yakushev
81736f712a [kondo] Squelch missing-docstring linter for Potemkin-imported vars (#82812) 2026-09-21 21:21:36 +03:00
Michael Eby
812e9b2254 add IP allowlist page to nav (#82774)
* add IP allowlist page to nav

* fix nav and metabase ip addresses label
2026-09-21 12:40:46 -04:00
Timofey Kachalov
d0ce86c19e Fixed InteractiveQuestion and StaticQuestion flashing the "run your code" empty state instead of a loader while card.query is still unresolved (#82693) 2026-09-21 18:48:24 +04:00
Jeff Bruemmer
72e2e9b234 docs - ai providers updates (#82376) 2026-09-20 14:33:58 +00:00
Kyle Doherty
426c9e6406 Use the resolved collection icon in the main navbar
buildCollectionTree already derives each node's icon via
getCollectionIcon, which is where a collection's own icon is honoured.
SidebarCollectionLink then re-derived it from the tree node, whose icon
field is the resolved IconData object rather than the stored name, so a
custom icon never survived to the Official and Library nav sections and
every folder fell back to its type default.

Read the resolved icon off the node instead. The tenant override moves
with it: MainNavbarContainer already passes isTenantUser into the tree
builder, and the one remaining caller that did not is unreachable for
tenant users.
2026-09-20 09:26:20 -04:00
Kyle Doherty
feb7aa645c Allow arbitrary Library folders with custom icons
Library folders were limited to the two seeded Data and Metrics
collections. The backend already inherited the library type onto
children, so subfolders worked, but nothing was exposed in Data Studio
and the Library root rejected everything but those two sections.

Top-level folders reuse the existing `library` type rather than adding
a new one: the root is told apart from user folders by is_library_root,
which is keyed off the hard-coded entity_ids and already existed. The
content rules become: the root holds folders only, a user folder holds
tables, metrics and nested folders, Data and Metrics are unchanged.

Collections gain a nullable icon column so a folder can carry a chosen
icon; when it is null the icon is still derived from the type.

canPlaceEntityInCollection now takes the destination collection instead
of a bare type string. The rule depends on root-vs-folder, and those
share a type, so the string alone would have made the Library root a
valid table-publish target.

The frontend move rule disagreed with the backend in both directions,
permitting a Data subfolder onto the root and blocking user folders from
nesting. It now mirrors check-library-update.
2026-09-20 09:19:20 -04:00
github-automation-metabase
93742a629c [chore] Update E2E test timings with recent data (#82804)
update e2e timings 20260920-062425

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-09-20 07:03:45 +00:00
Nathan Voxland
b485c0f7fa Improve *-allowed-network validation (#82519) 2026-09-20 01:32:39 -05:00
Nathan Voxland
76979e6167 Limit unauthenticated request body sizes (#82572) 2026-09-19 15:32:34 -05:00
Nathan Voxland
da7629bba6 Serve no values for a filter whose widget is an Input box (#82535) 2026-09-19 15:32:12 -05:00
Nathan Voxland
280c1e7be2 Improve indexed-entity handling in xrays (#82699) 2026-09-19 14:50:48 -05:00
Aleksandr Lesnenko
3d924036db bind dev Docker ports, socket REPL, and rspack dev server to loopback only (#82707) 2026-09-19 10:27:04 +07:00
Chris Truter
883e944cce Preserve Slack attachment names in thread history when text is missing (#82631) 2026-09-19 02:48:34 +00:00
Tyler Plude
9d1c919f6b Move glossary entity_id migration from v65 to v64 for backport (#82776)
* move glossary entity_id migration from v65 to v64 for backport

* reset Liquibase changelog cache in glossary precondition test

* Scope glossary precondition test column check to the current schema
2026-09-18 18:06:53 -07:00
Tyler Plude
b08a2f0e65 [GHY-4593] Make remote sync robust against restarts (#82553)
* add remote_sync_task.last_heartbeat_at and a per-task heartbeat thread

* always close a remote sync task row when its worker exits

* close this JVM's running remote sync tasks on graceful shutdown

* reap stale remote sync tasks at boot and when the current-task endpoint observes one

* keep a committed remote sync version as the sync base when its task is cancelled or fails afterwards

* return the initiating user on remote sync task responses

* show interrupted, cancelled, and quiet states in the remote sync progress modal

* report remote sync import progress through every phase with one throttled reporter

* use with-dynamic-fn-redefs for the heartbeat touch-task! redefs in remote_sync_task_test

* log the ids of stale remote sync tasks when the reaper closes them

* fix migration

* materialize data apps inside the import's version transaction

* discover a running or stuck sync from a fresh tab

* drop the unreachable sync interrupted view
2026-09-18 15:43:48 -07:00
John Swanson
22b1a393dc Read c3p0 pool stats from the pool objects instead of JMX (#82760)
* Read c3p0 pool stats from the pool objects instead of JMX

The Prometheus c3p0 collector enumerated every pool MBean and read its
attributes over JMX. A JMX read locks the MBean and then the pool, while
pool construction locks the pool and then the MBean
(swaldman/c3p0#95). The collector guarded against this by holding the
warehouse pool guard, but the application, Quartz and semantic-search
pools are built outside that guard, so a scrape overlapping one of those
constructions deadlocked and then held the guard forever: a fresh pod
passed its health checks but never completed a warehouse query.

`connection-pool-info` now enumerates c3p0's own registry
(`C3P0Registry/getPooledDataSources`, a copy made under the registry
lock) and reads each `PoolBackedDataSource` with the default-user
getters the MBean used. No JMX, no `locking`. A pool closed mid-scrape
throws and is skipped. Metric families, labels and the bug-reporting
endpoint's output shape are unchanged.

Warehouse pools of a non-default connection type get a name suffix
(`-write`, `-admin`, `-transform`) so two pools of one warehouse no
longer share a `database` label and overwrite each other.

The regression test scrapes on one thread while another builds pools
through the app-db path and a third builds a warehouse pool through the
guarded path; on the old collector it deadlocks within a second.

* Disable c3p0 JMX MBean registration

The Prometheus collector no longer reads pool MBeans, but any other
in-process JMX reader (the Datadog agent's JMXFetch, JConsole, a JMX
exporter) can still deadlock a pool construction the same way
(swaldman/c3p0#95). Set the c3p0 management coordinator to the null
one in resources/c3p0.properties so no pool or registry bean is
registered. c3p0 reads the key once through its layered config, with
system properties last, so an operator can restore the beans with
-Dcom.mchange.v2.c3p0.management.ManagementCoordinator=...ActiveManagementCoordinator.

Verified with a standalone harness: with the property, 37534 pool
creations under a concurrent JMX reader over 30 s with no deadlock and
no c3p0 beans; with the -D override, the beans return and the harness
deadlocks in 0.5 s, as it does without the property.
2026-09-18 14:44:18 -07:00
appleby
f58c520891 Require channel write permission to send to a channel by id (#82608) 2026-09-18 14:15:33 -07:00
Nathan Voxland
f2364b1d27 Format the blocked table's ID as a plain number in error message (#82772)
Don't format ids in error messages as if they were normal numbers
2026-09-18 20:35:31 +00:00
github-automation-metabase
39bd87ede3 Tighten ratchets (#82777)
Co-authored-by: github-actions <github-actions@github.com>
2026-09-18 20:20:06 +00:00
Tyler Plude
56232fc3f8 [GHY-4600] Improve memory allocation of remote sync (#82625)
* move the serdes :field normalize schema into a var

* extract the library once on a merging pull or push

* stream the export preflight extraction instead of realizing it

* index each merge side once for the merge and its casualties

* fix test

* share one registered :field normalize schema between serdes and resolve

* test the merging pull and push extract the library once
2026-09-18 13:14:18 -07:00
Michael Eby
61b6635f13 docs: IP allowlist (#82761)
* docs: IP allowlist

* edits from jeff
2026-09-18 15:16:45 -04:00
Alexander Polyankin
86b1f4bca4 Transform tests (#82495) 2026-09-18 14:50:16 -04:00
Eric Normand
ac1f4cdc00 Add a glossary tool to the MCP server (#82696)
* Let a v2 MCP tool describe itself from the instance's own data

deftool gains an optional :description-suffix, a fn of no arguments whose
result is appended to the tool description on every tools/list. The static
part stays in the manifest cache and only the suffix is recomputed, so a
suffix that reads the app DB doesn't drag the schema conversion along with it.

tools-hash skips it on purpose. The keepalive polls that hash every 30 seconds
on every open stream, so covering descriptions would buy a change notification
for a suffix at the price of its query per stream per tick. A suffix reaches a
client when it next lists tools instead.

* Add a glossary tool to the MCP server

glossary() returns every term with its definition; glossary(term) returns
one, matched case-insensitively, since a model echoes a term as it read it
rather than as it happens to be stored.

The term names ride the tool's own description rather than waiting behind the
call. A model only looks a word up when something tells it the word is worth
looking up, and the terms that most need a company definition are the ones
that read as ordinary English -- account, active user, churn -- which a model
believes it already understands. Metabot gets this for free by pasting the
whole glossary into every message it sends, because it owns the prompt; here
the description is the only channel that reaches the model unprompted.

Definitions stay behind the call, and the term list is capped at 100. Nothing
bounds a glossary -- no cap on entries, and a definition is TEXT -- and a
description long enough for a client to truncate costs more than its tail is
worth.

A client may cache a description for the life of its session, so a term added
mid-session shows up when that client next connects.

* Keep glossary out of the driver-test trigger set

mcp now uses glossary, and mcp's transitive dependents are effectively the
whole graph because some modules declare :uses :any. That made a glossary
change reach driver, query-processor and transforms, so every glossary edit
would have run the full driver suite. mcp is already excused from this for the
same reason; glossary joins it rather than raising max-allowed-count, since
none of it is driver logic.

Also fixes two whitespace-linter errors: a literal U+2028 landed in the
glossary test where an escape sequence was meant. The separator is now built
with char, which is what the test was trying to say anyway.

* Reject a :description-suffix that only takes one argument

ifn? was the wrong predicate. Keywords, maps, sets and vectors satisfy it but
are callable only with one argument, so they passed registration and then threw
ArityException at the first tools/list -- the deferred failure the guard exists
to prevent. Worse, the accepted registration sat in the shared registry, so the
throw came from listing any tool at all, not from the one that was wrong.

Vars stay allowed: one holding a fn of no arguments is callable, and :handler
is already registered as a var.

* Point the MCP server instructions at the glossary

The initialize instructions are the only channel that reaches the model before
any tool call, so this is where the glossary gets announced.

The line is unconditional on purpose. An instruction to look a term up when you
don't recognize it never fires, because the words that most need a company
definition are the ones that read as ordinary English -- account, active user,
churn -- which a model believes it already understands. So it asserts that
these terms matter and to fetch them before answering, with no qualifier about
which words. A test pins that: it fails if the line goes missing, and it fails
if the line grows a "don't recognize" style qualifier.

Static, so the instructions stay a compile-time constant. Whether an instance
has any terms at all is reported by the tool's own description instead.

* Instruct rather than list, and page the glossary listing

Three fixes from review.

The description no longer names the terms. The old cap was on the number of
terms, but the thing being protected is the length of a description clients
truncate -- Claude Code at 2048 characters. Seventy ordinary-length terms
produced 2444 characters with no pointer to the rest, so the client cut the
tail and the model saw a partial list it had every reason to read as the whole
glossary. That is worse than no list: a model believing the glossary is covered
never calls. So the description now carries the count and an unconditional
instruction to call before answering, which is inherently bounded and asks the
model to notice nothing.

glossary() is paged. It rendered every row, term and definition, into one
block, with definition a TEXT column that nothing bounds at any layer. Now
limit and offset through the usual list envelope, defaulting to 50 and matching
browse_collection -- the closest analogue, a plain unranked listing of rows
carrying prose.

glossary(term) returns every match. term is unique case-sensitively, so "ARR"
and "arr" are two entries with two meanings, and the lookup is deliberately
case-insensitive because a model echoes a term as it read it. Taking the first
match silently picked one and gave no sign the other existed.

* Make the glossary tool description static again

The suffix reported how many terms an instance defines. The only thing that
bought was letting a model skip one call on an instance with no glossary --
a call that is cheap and self-correcting, since it returns "no terms are
defined" and the model moves on. That does not pay for a public extension
point on deftool, its load-time validation, and a query of every glossary row
on every tools/list.

So the instruction moves into the static description and the machinery goes.
registry.clj is byte-identical to its state before this branch; every change
it had carried was in service of the suffix, including the list-tools split
that existed only so tools-hash could avoid computing one.

The instruction itself is unchanged in substance and still unconditional: call
glossary() before answering any question about this instance's data, with no
qualifier about which words, because the words that most need a company
definition read as ordinary English and the question will not say which ones
are defined.

* Simplify description

* Fixes

* Read the glossary through its module API, not its db namespace

The MCP tool required metabase.glossary.db, which meant exporting another
module's raw Toucan namespace just to read from it. metabase.glossary.core is
the module's stated API namespace and says so in its own docstring; it only
lacked a read, which is why db had to be exported at all.

So core gains entries, and db leaves the module's :api. Nothing outside the
module reaches into it now.

* Match the glossary tests to the simpler unknown-term error

The unknown-term error no longer names the terms that do exist, so the two
tests asserting it did were red. They now assert what it says: the term the
caller asked for.

The third was the cleaning guard, which had been riding on that error naming a
stored term. Retargeted onto a successful lookup, where stored text still
reaches prose the model reads as the server speaking. It now covers the stored
definition as well as the term -- definitions had no cleaning coverage before,
and the definition is the longer field and the one a payload would fit in.

The listing is deliberately not covered: list-content JSON-encodes its envelope
and wraps it raw, so it is bounded by JSON escaping rather than by clean, and
asserting otherwise would have been false.
2026-09-18 13:14:17 -05:00
Bartosz Rybacki
0e83852557 GDGT-3137: accept emoji up to 10 code points as a comment reaction (#82652)
* GDGT-3137: accept emoji up to 10 code points as a comment reaction

The emoji schema counted UTF-16 units, so the England flag (7 code
points, 14 units) got a 400. Postgres and MySQL count varchar(10) in
code points and already fit every picker emoji, so only the schema and
the H2 column change. The MySQL collation set by v63 stays untouched.
2026-09-18 20:04:39 +02:00
Nathan Voxland
61d0fa5d0f Check permissions on nested cards (#82600) 2026-09-18 13:02:41 -05:00
Kevin Kelleher
504b8ea9ee Enable explorations (#82594)
* Revert "Disable explorations (#82230)"

This reverts commit c505a9a785.

* add back test improvements

* Restore up-front profile validation in the agent endpoint

The "Disable explorations" PR (#82230) also fixed an unrelated bug: a
request naming a nonexistent profile was not rejected up front. The SSE
stream opened, the turn was persisted, and the conversation-title LLM
call was made before anything failed. Reverting that PR here dropped
the fix along with the explorations changes.

This re-applies only the validation: `profile-registered?` in the
profiles registry, and a 400 "Unknown profile" check in
`streaming-request` right after the profile id is resolved and before
`start-turn!` persists anything or the title job runs. Explorations
stay registered.

---------

Co-authored-by: John Swanson <john.swanson@metabase.com>
2026-09-18 13:12:20 -04:00
Mahatthana (Kelvin) Nomsawadi
61ce133268 test: Fix failed split permissions modal test (#82757)
test: drop the wall clock from the split modal test

The test gave the metadata request a 100ms fetch-mock delay and then
asserted the loader was still showing. On a loaded CI runner the
render plus the awaits burn through that budget, the loader is gone
and getByTestId throws. Resolve the metadata response explicitly
instead, so the loading window does not depend on timing.

Fixes EMB-2405


Claude-Session: https://claude.ai/code/session_01GuH5ewGP1DzQs5jwNpTgWc

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 16:54:18 +01:00
github-automation-metabase
f24251fb4e Auto-update documentation for master (#82710)
Update generated docs (master)

Co-authored-by: github-actions <github-actions@github.com>
2026-09-18 11:51:01 -04:00
Timofey Kachalov
7994ad35d8 Add data-apps scope to BE endpoints to prevent calling unmarked endpoints from inside a data app (#79382) 2026-09-18 19:34:58 +04:00
Timofey Kachalov
eb845c2ea6 Add version field support to data_app.yaml (#82557) 2026-09-18 18:57:25 +04:00
John Beers
3d8a3901a9 docs: update URL configuration (#82744)
[docs] Add theme keyword
2026-09-18 09:11:12 -04:00
Oleksandr Yakushev
16bd231ba7 [spec] Prevent t2.transformed/in-transforms from throwing on :model/ModelIndexValue (#79634) 2026-09-18 13:00:59 +00:00
Maksym Yakubych
89da5b2bc6 Fix unreadable low-rating data complexity badge (#82665) 2026-09-18 12:00:54 +02:00
Romeo Van Snick
91f5798258 Make SQL editor typing responsive again (DEV-3545) (#82724)
* Stop reconfiguring the SQL editor on every keystroke (DEV-3545)

* Apply native editor edits outside the keystroke (DEV-3545)

* Defer editor edits with Mantine's debounced callback
2026-09-18 19:58:46 +10:00
Jakub Chodorowicz
e49c2e60af Gdgt 3238 release sdk v2 canary1 (#82733)
release v2 canary.1
2026-09-18 11:05:32 +02:00
Mahatthana (Kelvin) Nomsawadi
496db401a1 fix: keep permissions tabs visible while data loads (#82633)
* fix: keep permissions tabs visible while data loads

The Data tab returned a bare spinner until databases, groups and
permissions loaded, so the tabs above it vanished on slow networks.
The page layout now always renders and the spinner shows in its
content area, like the Collections tab.

Fixes EMB-2392

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dg4bU1ueEbNhR2AYfm96T8

* fix: open the split permissions modal after load

The Data tab now renders its layout while data loads, so the one-time
split permissions modal could open over the spinner. Dismissing it
there turns the user setting off before the table it explains shows.
Only allow the modal once loading finishes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TuekFYoRQBgQoySZe6xB3k

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 08:05:46 +00:00
Kevin Kelleher
188c8f412f Fix failing Loki test caused by #82684 (#82713)
fix loki failure
2026-09-17 20:14:55 -07:00
Aleksandr Lesnenko
62831f5e4d infer db from query in transform_write mcp tool (#82461)
* infer db from query in transform_write mcp tool

* dedupe

* review

* fix
2026-09-17 23:20:26 +00:00
Romeo Van Snick
d5d6eba4c5 Move ErrorBoundary into common (#82624)
Claude-Session: https://claude.ai/code/session_01FeHJzstcAE34DTu4G6btjx

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 09:16:47 +10:00
Romeo Van Snick
a3b46fdd3c Return a question from the card hook, not a builder (#82602)
* Return a question from useQuestionFromCard instead of a builder

* Stop exporting the provider factory selector, which has no callers

* Memoise the connected-table link url
2026-09-18 09:16:37 +10:00
Romeo Van Snick
61c0e2915e Stop thirteen sites reaching through question.metadata() (#82552)
* Read the model's table, the list's entity type and the native editor's table from their own requests

* Take the native preview's provider from the store and the metabot table names from Lib

* Let the question carry its own metadata into derived questions and steps

* Take the pristine table URL, the native editor's databases and the last-run card off the Metadata object

* Mock the databases endpoint in the inline SQL prompt spec
2026-09-18 09:16:23 +10:00
Eric Normand
5701125c34 Per-scope checkboxes on the OAuth consent page (GHY-4555) (#82527)
* List every MCP v2 tool regardless of token scopes (GHY-4543)

tools/list filtered out tools the token lacked scope for, so a client
holding a narrow grant never saw the write, SQL or delivery tools and
had nothing to attempt or step up for. Scope stays enforced at
tools/call; the disabled-tools and client-extension filters still apply.

With scopes out of the listing, list-tools and tools-hash drop their
token-scopes parameter, and the transport's tools-hash-fn becomes
zero-arity.

* Explain MCP v2 scope denials to the model (GHY-4543)

Clients replace a scope denial with their own text ("token expired",
"Insufficient scope", "Tool execution failed"), so the model tells the
user their login expired. Server instructions now say that failure is a
missing permission, and how the user reconnects to grant it. Every tool
description ends with a static sentence naming its required permission by
its consent-screen label, identical for every token: Claude Code caches the
first description it loads for the whole session.

* Answer MCP v2 scope denials with 403 insufficient_scope (GHY-4543)

A scope denial was an HTTP 200 carrying a JSON-RPC error or an isError
result, so clients never stepped up: Claude Code only saves a step-up
scope, and mcp-remote only starts one, on a real 403 with an
insufficient_scope WWW-Authenticate challenge. The registry gate and the
four in-handler scope checks now mark their refusal, and the transport
answers a single (non-batched) message with 403 plus a challenge whose
scope is the held v2 scopes plus the required one.

* Keep a DCR client's registration scope from narrowing its step-up (GHY-4543)

Claude Code registers with the scope it read from the protected-resource
metadata, then steps up on the same client_id with the wider scope from a
403 insufficient_scope challenge. oidc-provider validates a requested scope
against the client's registered scopes, so that step-up was refused with a
400 invalid_request. Once scopes_supported is narrowed to the baseline, no
Claude Code user could reach write, SQL, or delivery.

A dynamically registered client's ceiling is now its registered scopes plus
the default grant scopes, computed on read. Registering can widen the
ceiling (a client that registers for mb:full keeps it) but never narrows
it. RFC 8707 resource narrowing is unchanged.

* Declare each MCP v2 tool's OAuth scope in securitySchemes (GHY-4543)

ChatGPT only steps up for scopes a tool declares: without per-tool
securitySchemes it re-authorizes for its narrow login scope in a loop,
and with them it re-authorizes once for held plus declared scopes.
The scheme is derived from the tool's static :scope in the cached
manifest, so it is the same for every token.

* Advertise only the baseline MCP v2 scopes to new clients (GHY-4543)

The protected-resource metadata `scopes_supported` and the 401 challenge
`scope` now list only `agent:content:read agent:resource:read`, so users
who never need write, SQL, or delivery never grant them. Claude Code and
the Claude connectors take their first-login scope from the resource
metadata; ChatGPT takes it from the challenge. All of them reach the rest
through the 403 `insufficient_scope` step-up and per-tool
`securitySchemes`.

What the resource accepts is unchanged: RFC 8707 narrowing still keeps all
six v2 scopes, and the authorization-server metadata stays wide because
Codex requests that list on every login and cannot step up.

* List every MCP v2 resource regardless of token scopes (GHY-4543)

resources/list now matches tools/list: it is token-independent, so a client can
see the resources it could step up for, and a client that pre-fetches the MCP
Apps shells before stepping up to agent:query:run does not cache them as
missing. resources/read keeps its scope gate.

* Answer MCP v2 resources/read scope denials with 403 insufficient_scope (GHY-4543)

resources/list now shows every resource, so answering a denied read with
"Resource not found" hides nothing and leaves a client with no step-up
signal. A client that pre-fetches an MCP Apps shell with a baseline token
now gets the same challenge tools/call sends, naming agent:query:run.
Unknown URIs stay -32602; inside a batch the denial is an in-band -32600.

* Lead MCP v2 tool descriptions with their required permission (GHY-4543)

Clients truncate long tool descriptions (Claude Code at 2048 characters),
so the appended "Requires the ... permission" sentence was cut from
document_write and execute_query. The model needs that sentence to name
the missing permission after a scope denial, so it now opens every
description, and the server instructions say it starts the description.

* Serve every MCP v2 resource on resources/read regardless of token scopes (GHY-4543)

Claude Desktop reads an MCP Apps shell alongside the tool call. When that
read was refused with 403 insufficient_scope, Desktop never stepped up after
the tool call's own 403; with the shell served, it stepped up automatically
and retried. The shell carries no data and, for a token without the shell's
scope, no UI credential: read-resource now withholds the credential instead
of refusing the read. Chart data stays gated by the tool call,
refresh_ui_credential, and the UI-surface route costs.

* Restore the scope check on MCP v2 data resource reads (GHY-4543)

Only the MCP Apps UI shells need to be served to any token: a refused shell
read stopped Claude Desktop stepping up after the tool call's 403. The fields
catalog carries data and has no such constraint, so reading it without
agent:resource:read again gets the 403 insufficient_scope challenge (in-band
-32600 inside a batch). UI shells stay unscoped, minting a credential only for
a token holding their scope.

* Fit MCP v2 tool descriptions under Claude Code's 2048-character limit (GHY-4543)

Claude Code truncates tool descriptions at 2048 characters, so document_write and
execute_query lost their closing guidance, including execute_query's pointer away
from execute_sql for filtered-dashboard cards. Condense both, and get_content for
margin, deferring detail to learn() topics and argument schemas. A test now holds
every listed description to the limit.

Also correct docstrings and comments left stale by the resources/read and
tools/list scope changes and by credential delivery through refresh_ui_credential.

* Spell out the scope union in the MCP v2 challenge docstring (GHY-4543)

The whitespace linter rejects the confusable Unicode character U+222A.

* Add agent:query:run to the MCP v2 baseline scopes (GHY-4543)

Claude Desktop retries a tool after a step-up over a session that does not
declare MCP Apps support, so a stepped-up visualize_query is refused and the
first chart after a step-up never embeds. Querying and charting therefore
must not require a step-up. The protected-resource metadata, the 401
challenge and default-ask-scopes now advertise agent:content:read
agent:query:run agent:resource:read; writes, raw SQL and delivery still step
up.

* Point MCP v2 chart requests at visualize_query in the server instructions (GHY-4543)

The routing line named only execute_query, and clients like Codex load tool descriptions on demand, so chart requests could miss visualize_query.

* Mint only the scopes chosen on the OAuth consent decision (GHY-4555)

The signed scope stays the offered set. The user's choice arrives in a
separate, repeatable granted_scope field and may only narrow it: a choice
outside the offer is rejected as params_tampered, and an approval that
leaves nothing granted is a 400. Offered MCP baseline scopes are always
granted. The token response already echoes the code's scope, so it now
names exactly what was granted.

* Add per-scope checkboxes to the OAuth consent page (GHY-4555)

Each offered scope is a granted_scope checkbox inside the consent form.
The six MCP v2 scopes are listed least to most harmful, with any other
scope after them in request order. The MCP baseline scopes are ticked,
disabled, and marked as always granted; every other scope, mb:full
included, starts unticked. The full-access warning now sits in the
mb:full row.

* Gate native query handles in transform_write on agent:sql:run and the kill switch (GHY-4543)

A query handle is not proof the SQL gates were spent: /api/embed-mcp/drills
stores a native query under agent:query:run alone. transform_write now runs
the same scope and mcp-execute-sql-enabled checks on a handle that resolves
to native SQL as it does on an inline native definition, matching
question_write.

* Tighten resource registration, step-up scopes, and the DCR ceiling (GHY-4543)

- register-resource! takes a closed map and strips :ui?, so a data resource
  can never be served to a token without its scope.
- step-up-scopes matches held scopes instead of looking them up, so a token
  holding a wildcard such as agent:content:* keeps the surface scopes it
  covers when a client replaces its grant with the challenged scope. It now
  takes the single required scope both callers pass.
- The default grant ceiling is applied only while dynamic registration is
  enabled, so turning registration (or MCP) off stops widening existing
  dynamic clients. The merge is now a pure function with unit tests, and an
  HTTP test pins that static clients are never widened. The legacy-client
  migration test runs with registration disabled, where the stored snapshot
  alone decides, so it exercises the migration again.

* Debounce the OAuth consent decision and require a ticked scope (GHY-4568)

The consent page gets an inline script, nonce'd and opted into a
script-src nonce so a production CSP allows it. On submit it shows a
spinner on the clicked button and disables both, carrying the decision
in a hidden input since a disabled button is not submitted, so a second
click can no longer race the first. The buttons come back when the page
is restored from the bfcache. It also keeps Authorize disabled while no
scope checkbox is ticked (GHY-4555).

* Correct MCP v2 tool descriptions, the documents skill, and server instructions (GHY-4543)

- document_write lists the valid entity-link models again.
- The documents skill said edits store new_str literally; the tool parses it
  as Markdown, as edit-replacement-parses-markdown-test pins.
- execute_query no longer claims to save cards (its handles save as cards),
  and names the previous-stage field ref shape instead of an ambiguous
  column-name string.
- The server instructions attribute the rewritten permission failure to the
  client and cover resource reads, which get the same 403.

* Tell the MCP v2 model which scopes the connection holds (GHY-4555)

The initialize instructions are now built per request from the calling token's scopes. A scoped token is told which v2 surface permissions it has and lacks, by consent-screen label; an unrestricted caller gets no list. The instructions also stop claiming the consent screen has no per-permission choices: a permission the connection lacks starts unticked, so the model warns the user to tick it.

* Note the unticked consent checkbox in the MCP v2 insufficient_scope description (GHY-4555)

Tool and data-resource 403 challenges now end their error_description with a note that the permission starts unticked on the consent screen and the user must tick it. The README describes the new consent screen and the per-connection instructions.

* Rename the agent:resource:read consent label to "Read MCP resources" (GHY-4555)

* Tighten the MCP v2 scope instructions (GHY-4555)

The general paragraph states the scope-failure guidance once; the per-connection paragraph carries only the held and missing permissions and how far to trust that list.

* Pin the MCP v2 scope-denial and UI-credential contracts in tests (GHY-4543)

Closes gaps the review found, where removing a check would stay green:
- question_write and subscription_write in-handler denials carry
  :insufficient-scope at the registry, like alert_write's.
- A denied tools/call sent as a notification is a bare 202, no challenge.
- Wildcard and unrestricted tokens get the shell credential; the fields
  catalog mints none, over the registry and over HTTP.
- tool-and-resource-scopes-agree-test asserts the credential is realized,
  since any token now reads a shell.
- advertised-scopes-are-distinct-test covers the advertised baseline, which
  is a vector and can repeat, and its docstring says so.

* Clean up MCP v2 scope-denial helpers, header quoting, and docs (GHY-4543)

- Share one scope-denial message and detail builder between tool calls and
  resource reads, one step-up challenge helper, and a common
  throw-insufficient-scope! for the four in-handler scope checks.
- quoted-string replaces " and \ instead of backslash-escaping them, since
  RFC 6750 excludes both from scope and error_description values; the 401
  challenge now quotes through it too.
- Trim docstrings to their contract and move the reasons into comments.
- The unrestricted UI credential comes from any unrestricted session (cookie,
  API key, or an mb:full token), not only cookie sessions.
- Note that the DCR ceiling applies only while registration is enabled, and
  fix stale comments, test names and strings, long lines, order-sensitive
  scope assertions, and hard-coded consent labels in tests.

* List scope IDs only in the MCP v2 instructions, have the model make the call once the user agrees, and cap instructions at 2048 characters (GHY-4555)

* Pre-tick consent scopes the same app already holds (GHY-4555)

A step-up asks for the held scopes plus the new one, and every non-baseline box started unticked, so ticking only the new scope silently dropped a held one. A scope that a live access or refresh token of the same app holds now starts ticked but stays enabled. The same app is the same client_id, the exact same https redirect, or a loopback redirect with the same host, path and client name on any port, matched against the redirect this request delivers the code to.

* Narrow the same app's tokens when a held scope is unticked (GHY-4555)

Unticking a pre-ticked scope now removes exactly that scope string from the same app's other live access and refresh tokens, so the choice holds across windows. Only scopes offered on the screen and held by the app are removed; nothing is ever added, and a token is revoked only when no scope is left. This runs at the approve decision, after the code is issued, because the authorization code row has no field to carry the declined scopes to the token exchange without a migration.

* Tell the MCP v2 model that reconnecting before a refused call won't offer the permission (GHY-4555)

* Always widen dynamic OAuth clients to the MCP scopes (GHY-4543)

Claude Code registers with the baseline scopes and steps up on the same
client_id. With dynamic registration turned off, that step-up /authorize
was answered with a raw 400. A dynamic client now always gains the six v2
MCP scopes on read, each still behind consent; the remaining agent-API
ceiling is added only while registration is enabled. Registration itself
stays blocked.

* Close the MCP v2 scope test gaps (GHY-4543)

Cover the transform_write update path's native gates, the HTTP 403
insufficient_scope challenge question_write and transform_write send from
inside their handlers, saving a drill handle minted through
/api/embed-mcp/drills, and the closed resource registration schema. Give
the locale test a positive control, merge the two tools/list
token-independence tests into one that compares the whole payload, and
drop the permission-sentence truncation test the fit and starts-with
tests already cover.

* Tidy MCP v2 scope wording and naming (GHY-4543)

Rename registry/insufficient-scope to insufficient-scope-detail so it
does not read like the transport fn of the same name. Drop the server
instructions clause telling the model to read a resource's scope off an
error clients hide. Document agent:resource:read and the handler-internal
scope checks in the module README, correct the stale comments about UI
resources gating on agent:query:run, and wrap the lines this branch left
over 120 columns.

* Let the test fallback template win over a local frontend build (GHY-4543)

The MCP Apps shell renders from the built `embed-mcp.html` when one exists,
so a worktree that has run a frontend build shadowed the inline fallback the
credential tests install. The production template has embedded no
`uiCredential` since #81041, so those tests failed locally and passed on CI,
where backend-only runs produce no build.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014QT1v8on6aPT2UeHCojGj8

* Tell the MCP v2 model a permission can be removed mid-session (GHY-4555)

* Drop cross-connection scope narrowing from the OAuth consent decision (GHY-4555)

Unticking a scope the same app already held used to strip it from that app's
other live access and refresh tokens. Every verified defect in this PR lived in
that path:

- the authorization code and the approved event are persisted before the
  narrowing runs, and both sit inside one `try`, so a narrowing failure answers
  400 while a redeemable code exists and the old tokens keep the scope;
- dynamic client registration is open, so an attacker can register a client
  carrying another app's https redirect -- or a loopback redirect with a copied
  `client_name`, since the port is ignored -- and strip or revoke the real app's
  tokens after phishing a single Authorize click;
- a pending authorization code is not narrowed, so redeeming one mints a
  just-removed scope straight back.

No comparable provider does this: GitHub, Google, Microsoft and Slack all leave
an existing grant alone when a new authorization asks for less. Taking a granted
permission away deserves its own flow, with its own confirmation and audit
trail; a follow-up ticket covers it.

What stays is the pre-tick: the consent page still ticks a scope the same app
already holds, so a step-up does not silently drop it, and the same-app matching
that finds those tokens is unchanged. A test now pins that an untick leaves the
app's other live tokens -- access and refresh -- exactly as they were.

Also drops the two db.clj scope-update helpers nothing else used, and corrects
two claims the removal falsifies: the MCP `initialize` instructions no longer
tell the model that another window can untick a permission mid-session (a
re-authorization without it, or a revocation, is how it goes away now), and the
MCP README says the connection's scopes are listed by scope ID rather than by
consent-screen label.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Store a repeated OAuth scope once in the granted token (GHY-4555)

`scope=agent:content:write agent:content:write` was split without
deduplication, and the decision filters the grant from that list, so the
duplicate reached the token row and the token response's `scope`. The consent
page already deduplicated for display, so nothing on screen hinted at it.

Deduplicate in `scope-tokens` instead, which is the one place both the page and
the decision read the requested scopes from.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Pre-tick the OAuth consent page from the app's newest live grant (GHY-4555)

The held set was the union of every live token the app had for the user. A
refresh token outlives several authorizations, so the union let an old, wider
grant keep setting the defaults: day 1 grants content:write and sql:run with a
30-day refresh token, day 5 the user re-authorizes and leaves sql:run unticked,
and on day 10 the union pre-ticks sql:run again from the day-1 token -- silently
undoing the day-5 decision every time they re-authorize.

Take the newest live grant instead, so the user's latest choice is the default.
A grant's access and refresh rows are written by one token request and share a
`created_at`, so the newest timestamp names a grant rather than a row: every row
carrying it speaks for that grant, which is what keeps the newest grant readable
once its access token has expired and only its refresh token is left. Erring
narrow is safe now that an untick only shapes the token being minted.

Also records why the loopback key ignores the port, and what that costs, next to
the branch that decides it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Drop consent-page pre-ticking of already-held scopes (GHY-4555)

Every non-baseline scope now renders unticked, whatever the user already
holds. Pre-ticking needed to identify the connection being stepped up,
and the consent page cannot:

- the step-up challenge expands a wildcard grant like `agent:content:*`
  into concrete scopes, but the held check matched exact strings, so
  `content:write` started unticked and a step-up dropped it;
- refresh rotation writes new rows with a current `created_at`, so an
  older broad grant refreshed after a newer narrow one looked newest and
  re-ticked a scope the user had dropped;
- the newest app-wide grant is not necessarily the connection stepping
  up: window A holds content:write, window B authorizes later without
  it, then A steps up and gets B's defaults;
- prechecking across registrations is a consent boundary. Registration
  is unauthenticated, so a local app copying the client name and
  loopback callback path, or a client registering another app's https
  redirect, got those scopes from one generic Authorize click instead of
  the user selecting them.

Doing it properly needs grant identity carried into the authorize flow,
the redirect recorded on tokens, and wildcard-aware matching. GHY-4627
tracks that.

Removes `metabase.oauth-server.held-scopes` and the `db` helpers that
existed only for it. A test pins the removal: with live access and
refresh tokens holding `agent:content:write`, the consent page still
offers it unticked.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Correct two MCP scope claims in the README (GHY-4555)

Per-tool `securitySchemes` is draft SEP-1488 (modelcontextprotocol issue
1488), supported by ChatGPT. It is not part of MCP 2025-03-26, the
version this server reports, nor of the final 2026-07-28 tools spec, so
other clients learn a missing scope from the 403 instead. The comment
above `default-ask-scopes` implied it was generally available too.

Automatic scope step-up is likewise defined in MCP 2026-07-28, so
clients are not required to implement it against this server: a scope
left unticked can trigger another step-up in clients that support it,
while others need manual reauthorization.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Drop the per-connection permission list from the MCP v2 instructions (GHY-4555)

The list caused the failure the "make the call anyway" clause was added to
fix: with it in context the model sometimes refused a write without calling
the tool, so no 403 came back, the client saved no step-up scope, and the
user's re-authorization offered only the baseline — the permission never
appeared as a checkbox. Without the list and without that clause, the model
calls the tool, reads the 403, names the permission and gives the reconnect
steps.

So the `initialize` instructions are one static string again: general
scope-failure guidance, plus one sentence that every optional permission on
the consent screen starts unticked, so the user re-ticks the ones they want
to keep. `transport.clj` goes back to a plain `:instructions` option.

* Tell the MCP v2 model the tool list is not a permission grant (GHY-4555)

Every tool is listed whatever the token holds, but a scope-filtered list is
the conventional design and the protocol has no field to signal ours. Asked
what a baseline connection could do, the model read the roster as a grant and
answered that it held every agent scope, reasoning that the server filters
tools by granted scope -- true of Metabase until the filter was removed.

Only a failed call reveals a missing permission, so say that.

* Fix counterfactual docstrings and pin the offered scope non-blank (GHY-4555)

Two docstrings stated the behavior the dedup prevents as though it were the
behavior it has, so a reader chasing the invariant learned the wrong one.
Both now say "would be stored twice".

scope-to-grant gains a postcondition. The decision endpoint answers a nil
grant with 403 params_tampered, which is only the right status while the sole
way to reach it is a choice naming an unoffered scope. That holds because
scope-to-grant throws rather than returning a blank offer; the postcondition
turns a future relaxation into a failing assertion instead of an honest
approval reported as tampering.

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 16:34:41 -05:00
Kyle Doherty
5349fb75d7 clean up crumbs 2026-09-17 16:30:13 -04:00
Ryan Laurie
ecb8fe82b2 Merge automated PRs with the automation user's token so CI runs (#82701)
GitHub starts no workflow run for a push made with `GITHUB_TOKEN`, so
enabling auto-merge with it lands backports on release branches, and
translation updates on master, with no CI at all. Approving still uses
the other token, since a token cannot approve its own pull request.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-09-17 20:19:51 +00:00
Kevin Kelleher
40ebe0bb9f Fix static viz Row Chart failures on remapped columns (#82684)
* make remapping idempotent

* copy the existing remapping so it doesn't get mutated
2026-09-17 19:55:42 +00:00
Eric Normand
0700179dad GHY-4542: Reject unregistered and empty OAuth scopes; fail closed on scope-less bearer tokens (#82498)
* Reject unregistered scopes in OAuth dynamic client registration

POST /oauth/register is unauthenticated and stored any scope string the
client sent. A client's registered scopes are the ceiling /oauth/authorize
checks requests against, so a self-nominated `*` or `agent:*` could later
be requested and granted, and scope-matches? treats both as wildcards.

Registration now answers 400 invalid_client_metadata, naming the offending
scopes, when any requested scope is not a registered scope. An omitted
scope still gets the default ceiling, and `mb:full` is still accepted.

* Reject unregistered scopes at OAuth authorize

Clients that registered a wildcard such as `*` before registration
validated scopes can still request it, since /oauth/authorize only checks
a request against the client's own registration. /authorize now answers
400 invalid_scope when any requested scope is not a registered scope. The
check runs before resource narrowing, which would otherwise drop the
scope when `resource` is sent and keep it when it is not.

Both endpoints now use one error_description that points the client at
the authorization server metadata listing scopes_supported, instead of
echoing the scopes it sent.

Clients registered on v0.60-v0.61 still hold agent:database:read,
agent:table:read, and agent:metric:read, which are no longer registered;
requesting them now fails. The authorize test fixtures move from the
unregistered "profile" to agent:content:read.

* Reject OAuth authorize requests without a scope

A request to /oauth/authorize with no scope rendered a consent screen
listing no permissions and minted a token with no scopes. It now answers
400 invalid_scope whether scope is absent, empty, or whitespace. A blank
scope is dropped before the provider validates the request, so an unknown
client or mismatched redirect URI is still reported first.

The error_description names no request input and points at the
authorization server metadata listing scopes_supported.

The OAuth authorizations E2E spec drove consent without a scope; it now
requests agent:content:read.

* Reject an empty scope in OAuth dynamic client registration

A registration that sent scope as null, an empty string, or whitespace
was stored with no scopes, and such a client can never authorize now
that /oauth/authorize requires a scope. /oauth/register answers 400
invalid_client_metadata instead, with an error_description pointing at
scopes_supported.

Omitting scope still registers the client with the default ceiling, which
is what MCP clients that register without a scope rely on.

* Fail closed on OAuth bearer requests without token scopes

Nil :token-scopes means scope-unaware auth, and sessions, API keys,
the scope middleware, the agent API, and the MCP transport all let it
through as unrestricted. For an OAuth access token, only an empty set
surviving every hop kept a scope-less token from becoming unrestricted.

The session middleware no longer authenticates an access token with no
scopes, and marks requests it does authenticate with an OAuth token as
:authenticated-via-oauth?. Behind it, enforce-scope and
ensure-scopes-checked answer 403, the agent API stops defaulting nil
scopes to unrestricted, and the MCP transport answers its invalid_token
401 for an OAuth-authenticated request with no scopes.

The marker is set by the session middleware rather than inferred from an
Authorization: Bearer header, which the agent API JWT path also uses and
which a session request may carry alongside its session. Sessions, API
keys, JWT, and the MCP Apps UI credential never set it.

* Redirect OAuth authorize errors to the client per RFC 6749

GET /oauth/authorize answered every validation failure with a 400 JSON
body, which strands the user on a raw error in their browser tab and
leaves the client waiting for a callback that never arrives.

The endpoint now looks up the client and checks the redirect_uri is an
exact match for one it registered before validating anything else. If
either is missing or invalid, it keeps the non-redirect 400, as RFC 6749
section 4.1.2.1 requires. Every other failure redirects to that
registered redirect_uri with error, error_description, state, and the
RFC 9207 iss parameter, built by the library's denial redirect builder.

Error codes: invalid_scope for unregistered, missing, fully narrowed,
and unregistered-for-the-client scopes (the last was mislabeled
invalid_request); unsupported_response_type for a response_type the
client did not register; invalid_target for an invalid resource
indicator; invalid_request for PKCE problems and malformed parameters.

* Send RFC 6750 bearer challenges on scope-denial 403s and agent API 401s

RFC 6750 section 3 requires a resource server to answer a bearer token
that does not grant access with a WWW-Authenticate challenge. The scope
middleware's 403s, including the fail-closed ones for OAuth requests
without scopes, sent only a JSON body, so an OAuth client could not tell
it should re-authorize for more scope.

enforce-scope and ensure-scopes-checked now add
  Bearer error="insufficient_scope", scope="...", error_description="..."
with scope naming the endpoint's required scope, and omitted in
ensure-scopes-checked, where the endpoint declares none. The format and
value sanitizing match the MCP transport's insufficient_scope challenge.
The JSON bodies are unchanged.

The agent API's 401 for a bearer token that does not authenticate now
carries Bearer error="invalid_token".

* Answer malformed OAuth resource indicators with client errors, not 500s

A resource value that is not syntactically a URI, such as
"http://bad uri", made oidc-provider's resource check throw a
URISyntaxException rather than an ex-info. Nothing caught it, so
/oauth/authorize, /oauth/authorize/decision, and /oauth/token all
answered a client input error with a 500.

Each endpoint now checks that every resource indicator parses before
handing the request to the library:

- /oauth/authorize: once the client and redirect_uri are validated, an
  RFC 8707 invalid_target error redirect. Otherwise the existing
  no-redirect 400.
- /oauth/authorize/decision: the existing 400 invalid_request, with no
  redirect, since its parameters are untrusted until the signature
  verifies.
- /oauth/token: a 400 invalid_target.

Only URISyntaxException is caught, and only while parsing the resource
values.

* Send a plain Bearer challenge on agent API 401s without a bearer token

RFC 7235 requires a WWW-Authenticate challenge on every 401. The agent
API's missing_authorization 401 (no credentials) and
invalid_authorization_format 401 (a non-Bearer Authorization scheme)
sent only a JSON body.

Both now carry WWW-Authenticate: Bearer. There is no error attribute,
because RFC 6750 section 3.1 says a challenge to a request with no
bearer token should not include an error code. Status codes and bodies
are unchanged.

* Keep OAuth authorize errors non-redirecting

Delivering /oauth/authorize errors as a redirect to the client's
registered redirect_uri made Metabase a zero-click open redirector.
Dynamic client registration is unauthenticated, so anyone can register
a client whose redirect_uri points at their own site, and a single
/oauth/authorize link on the trusted Metabase host would then send any
logged-in user there with no consent step. RFC 9700 section 4.11 says to
answer with an error rather than redirect when the redirect URI is not
trusted.

Every error from GET /oauth/authorize is a 400 JSON body rendered in the
user's browser again, with no Location header. The login redirect for
unauthenticated users is unchanged.

The error codes stay accurate: a scope outside the client's registered
set is invalid_scope, an unregistered response_type is
unsupported_response_type, and an invalid resource indicator, including
one that is not syntactically a URI, is invalid_target. Malformed
resource indicators still cannot reach the 500 they used to.

* Drop unregistered scopes at /oauth/authorize instead of refusing

Registration and authorization see the same problem from different
sides. An error from /oauth/register reaches the client program at setup
time, where it can act on it, so that endpoint keeps rejecting
unregistered scopes outright. An error from /oauth/authorize lands in a
user's browser tab as JSON while the client waits for a callback, and it
also breaks step-up for a client that still holds a scope shipped in
v0.60-v0.61 and since removed from the registry.

/oauth/authorize now filters the requested scopes down to the registered
ones and carries on with what survives, which RFC 6749 section 3.3
allows. Filtering runs before resource narrowing, so a request behaves
the same with and without a resource indicator, and the consent page and
the minted token carry exactly the surviving scopes. Least privilege is
unchanged: an unregistered scope, including a wildcard a client
registered before registration validated scopes, still can never be
granted.

A request in which nothing survives the filter is refused with
invalid_scope and its own description, distinct from the one for a
request narrowed away by its resource indicator. A missing, empty, or
whitespace-only scope is unchanged.

* Enforce the scope rules where the authorization code is issued

The HMAC over the consent form is keyed by the CSRF token, which the
consent page prints for the user, so the user can recompute it over
whatever scope they like. It proves no third party tampered with the
form; it does not prove the scope was ever validated.
/oauth/authorize/decision is what issues the code, so it now applies the
scope rules itself instead of trusting the signature.

Both endpoints call one scope-to-grant function: refuse a missing or
blank scope, drop the scopes that are not registered, refuse when none
survive, and grant exactly what is left. A denial grants nothing and
skips it. /oauth/authorize behaves as before; a refusal at the decision
endpoint keeps its existing 400 JSON shape and never redirects.

A correctly signed form asking for a scope the consent screen never
showed was previously honored: a re-signed wildcard minted a token
carrying it, and a re-signed blank scope minted one with no scopes at
all.

* Tidy the OAuth scope and challenge code

Follows the style review. No behavior change except one described below.

- narrow-scope-to-resource's docstring described a contract it no longer
  has: that nil means both "nothing requested" and "nothing survived",
  that the caller may drop the parameter, and that narrowing can never
  turn a valid authorization into a rejected one. It returns nil when no
  requested scope survives, and the caller refuses.
- One check-resource-parseable! replaces the same pre-check written three
  times with three different ex-data shapes, and the note about why it
  exists sits on unparseable-resource! once instead of at each call site.
- invalid_target now always carries the description saying what a valid
  resource indicator looks like. It used to depend on where the error
  came from: the pre-check said so, while a relative indicator or one
  with a fragment fell back to the generic description. This is the one
  behavior change, and the authorize test that pinned the generic
  description is updated.
- The three inline invalid_client_metadata 400s in /register go through
  one helper.
- The error_description strings and their helpers move out of the
  module's core namespace, where they were public with a single caller,
  into the api namespace as private defs grouped together.
- The MCP transport's two invalid_token 401s were identical; they share
  one response now.
- The agent API's error-response takes the WWW-Authenticate challenge as
  an argument, which is what all three of its callers were adding
  afterwards.
- Dropped an inaccurate comment about parameter order matching the MCP
  transport's challenge, some rationale that belongs in line comments
  rather than docstrings, and a double negative.

* Tighten the OAuth scope and challenge tests

Follows the style review.

- quoted-string is now exercised directly with a quote, a backslash and
  non-ASCII input. The charset assertion it replaces only ever saw
  constants, so it could not fail.
- The agent API's invalid_token 401 test moves to the agent API tests,
  where the behavior lives, and its no-credentials case goes: that is
  already covered there.
- Deleted bearer-token-without-scopes-gets-no-tool-access-test. Its 401
  comes from the transport's invalid-bearer branch, because the session
  middleware declines a scope-less token before the transport sees it, so
  it passed with or without the fail-closed branch and duplicated
  bearer-bridge-refuses-token-without-scopes-test. The branch itself is
  covered by oauth-request-without-token-scopes-is-refused-by-the-
  transport-test, which invokes the handler directly.
- The DCR unregistered-scope test compares the whole error_description
  instead of a substring plus assertions that cannot fail.
- Scope-denial tests assert the exact error code each case produces
  rather than accepting either.
- A test description stated its own history; it states the behavior now.
- Wrapped lines over 120 characters.

* Declare :authenticated-via-oauth? in the request schemas

Master closed ::current-user-info and added a closed ::request schema
enumerating every key the middleware stack may attach. This branch marks
a request the OAuth bearer bridge authenticated with
:authenticated-via-oauth?, which both schemas then rejected as an extra
key: the bridge threw on its return value, so no OAuth bearer request
could authenticate.

* Describe only our own resource indicators as malformed

The error_description helper answered every invalid_target that carried no
description of its own with the one describing a malformed resource indicator.
oidc-provider raises that code without a description for a different condition:
a refresh whose resource is outside the original grant. A client sending a
well-formed absolute URI was told its URI was malformed.

The endpoint's own check now carries its description in the exception, and the
helper falls back to the endpoint's generic description for anything else. That
check also validates what it claims to describe, so a relative indicator or one
with a fragment is answered by it rather than by the library.

* Answer 401 when no JWT provider is registered

The agent API's stateless-JWT fallback dispatches auth-identity/authenticate
on :provider/jwt. That provider ships only in EE, so on OSS nothing derives
the keyword into the auth-identity hierarchy and the dispatch throws
UnsupportedOperationException. Every bearer token the OAuth bridge declines
falls through to it, so the agent API answered 500 with a stack trace instead
of the RFC 6750 invalid_token challenge it owes the request.

Treat an unregistered provider the way a disabled one is treated: the same
jwt_not_configured result, which enforce-authentication already renders as a
401 carrying the invalid_token challenge.

Also stop bearer-bridge-refuses-token-without-scopes-test from holding
with-temp's rollback-only transaction open across its four HTTP round-trips.
The request runs on a conveyed connectable, so anything in its path that
commits implicitly invalidates the savepoint and the scope exit throws
instead of rolling back.

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 13:57:02 -05:00
Nathan Voxland
e96a5d884d Improve OSS include_editable_data_model logic (#82451) 2026-09-17 13:40:38 -05:00
Nathan Voxland
587b659ee7 Don't include archived cards in embedded dashboards (#82529) 2026-09-17 13:39:39 -05:00
Kyle Doherty
5073c20eb1 Metabot conversations show up in the Open list
Starting a conversation with Metabot now puts it in the rail's Open list, next
to the questions and dashboards you have open, so the sidebar panel's thread is
reachable again after you navigate away from it.

- `getStartedConversations` lists the conversations with at least one message,
  in creation order. Empty ones are skipped — an agent always holds a
  conversation, so listing those would show a row for Metabot merely existing —
  and the SQL-suggestion and explorations agents are excluded, since they answer
  in place and have no conversation page to link to.
- Rows link to /metabot/conversation/:id. The title is generated after the first
  answer, so a row starts as "New conversation" and renames itself in place when
  the title lands.
- The selector holds its result identity while ids and titles are unchanged, so
  a streaming answer does not re-fire the effect on every token.
2026-09-17 13:19:43 -04:00
Kyle Doherty
9d6d4c656e Raw data becomes a drawer with a warehouse tree
Data gets the same treatment as collections: one rail row, no section heading,
and a drawer holding the thing itself rather than a link to a page about it.

- The row reads Raw data and opens a drawer with the warehouse as a tree:
  databases, their schemas, and the tables in them. Every row links to the
  browse page it stands for, so the tree is a faster route to the same
  destinations, and any node can be expanded in place instead.
- Children are fetched only when a node is expanded, so opening the drawer costs
  one database list and nothing else.
- Tables with virtual ids (cards in the Saved Questions database) are left out:
  they have no /table/:slug route to link to.
- The Add data action moves from the old section header into the drawer header.
  `BrowseNavSection` and its spec are gone, along with the
  `expand-browse-in-nav` setting it read.
2026-09-17 13:08:57 -04:00
Andrei Simionescu
4b6744c1f4 Apply the chat system prompt to Metabot in Slack (#82639) 2026-09-17 18:06:54 +01:00
Kyle Doherty
2f02a29249 Data only in Unofficial, Getting Started collapsed, Trash in the drawer
- The Data section (Databases) moves inside the Unofficial branch. Official is
  curated content; browsing the warehouse belongs to the creation half.
- Getting Started starts collapsed, so the rail opens on the user's own work
  rather than on onboarding.
- Trash moves to the bottom of the collections drawer, where the things it holds
  were deleted from, instead of being a permanent rail row.
2026-09-17 13:02:22 -04:00
Andrei Simionescu
4c5146d536 Don't feed Metabot queries the user has no permission to run (#80083)
* BOT-1956: Fix permission-check gaps in transform/query export

Card/measure/segment export lookups routed through the loud, audit-logging
read-check even for routine filtering, one query-export call site defaulted
to the unchecked store, and a withheld query left a dangling header instead
of saying so.

* BOT-1956: Audit the permission check on client-supplied query ids

Ids arriving inline in the viewing context are a real access attempt, so the database gate
and the card ids inside the query both keep the audited read-check while stored-query export
stays quiet. export-query-for-llm now takes an explicit content store, so every call site
shows which one it picked.

* Don't audit permission refusals the caller discards (#79969)

* BOT-1956: Resolve -1337 to the real database and gate transform sources

The virtual id passed the database gate as a 404 while the export then
resolved it to the source card's real database, so the gate now resolves
first and fails closed on ids it cannot. Transform sources get the same
gate, queries and charts seeded from the viewing context keep the audit
trail when a read_resource denial withholds them, and the withheld
message no longer blames the database when a card inside the query was
the refusal.

* BOT-1956: Drop the em dashes from two docstrings

* BOT-1997: Don't feed Metabot queries the user has no permission to run

* BOT-1997: Shorten the comments on the Metabot permission checks

* BOT-1997: Close the two remaining fail-open paths in the query export

Exports now defer to the query processor's can-run-query?, the same
check the save path uses, instead of a database-wide can-query?
shortcut plus SQL analysis. For native stages that means the
database-level query-builder-and-native rule with no parsing: the
analyzer was best-effort and could return an incomplete table set with
no completeness signal, so a clean analysis never proved the query was
authorized, and unsupported drivers withheld queries from fully
authorized users. Deferring to the QP contract also keeps queries
sourced from saved questions working; those authorize through the
card's collection, not through database query permission. The
name-resolution checks (queryable tables, sandbox-visible fields,
readable cards) stay on top of the run check.

The same check now also covers conversation state. read_resource used
to export a stored query after only a database read-check, so a user
who could read the database's metadata but not run the query could
recover resolved table and field names through metabase://query/{id}
and metabase://chart/{id}. Both fetchers now share the viewing-context
formatter's exportable-query? check.

* BOT-1956: correct three stale comments

The withheld-message docstring claimed a deleted database is withheld too, when it
deliberately renders. The content-store test docstring counted three pinned properties
while listing four, and the repair comment called every pass best-effort when only the
mini-resolve lookups are.

* BOT-1997: authorize conversation queries by run permission alone

The query and chart resources read-checked the raw :database before the run-permission
check, which broke the two cases the canonical check exists for: a collection-only user
running a saved-question query got a 403 from a database they never touch, and a legacy
-1337 database id 404ed before normalization could resolve it through the source card.
The read-check is gone and the viewing-context predicate now lets normalization resolve
-1337 instead of gating on the raw id.

The source-card tests also stop granting manage-table-metadata, which had made the
database readable and hidden exactly these cases; they now run with no database
permission at all, over both the real and the -1337 id, alongside a new refusal test
for a card in an unreadable collection.

* BOT-1997: Hide transform source queries the user can't run

Reading a transform is authorized per database, but rendering its source resolves
table and field ids to names, so the details tool and the transform resource now
drop the query unless the user could run it. The viewing-context path also stops
normalizing twice and logs a query it can't permission-check at debug, instead of
letting the calculator log an error on every prompt.

* BOT-1956: make the query export gates table-granular

* BOT-1956: fail closed when a native query can't be analyzed

* BOT-1956: Fold the two query database gates into one with an audited? flag

* BOT-1997: Calculate perms inside the try, assert the 403 text in tests

* BOT-1956: Check exports with can-run-query? instead of parsing native SQL

* Build the saved-question test card with Lib instead of the deprecated mbql-query

* Use metabot.db/database-exists? instead of a direct Toucan call

* BOT-1956: Name can-run-query?'s flag for the errors it rethrows

* BOT-1956: Suppress repair's refusal auditing at its call site

* BOT-1956: Keep :client-ids out of the conversation state

The ids describe the viewing context of the request being served rather than the conversation, and final-state-part was handing them back to the client as if they were state.

* BOT-1956: Gate a query export with one Database read and one normalization

* BOT-1956: Audit a chart's refusal against the query it was built from

* BOT-1956: Make run permission the whole gate on a query export

Reading the database was checked on top of it, which withheld a saved question from a user who can read the card and run it but has no grant on the database behind it. Dropping that check also leaves one Database lookup where there were two, so a deletion landing between them can no longer escape as a 404.

* BOT-1997: Render a query we could not permission-check rather than hiding it

A query that will not normalize, or whose permissions will not calculate, was reading as a refusal and disappearing from the transform output and the conversation resources, admins included. It now comes back without a metadata provider and renders through the paths that resolve no ids to names, which is what master did with its EDN fallback.

* BOT-1997: Cover the cases the export gate decides on a transform source

* BOT-1997: cljfmt

* BOT-1997: Make the unresolved fallback actually withhold names

A query that will not parse never reached the fallback, because normalization degrades to an empty map unless it is asked to be strict, so those queries still read as refusals and disappeared. Transform rendering then threw away the checked query and its provider and resolved the source a second time through an unrestricted provider, which put resolved names back in front of the model for a query whose table or sandbox lookup had thrown. Native SQL needed its own answer either way, since its table and column names are already spelled out in the text and having no provider hides nothing, so it stays out until the check has actually run. Rendering now happens where the check's result is still in hand, and an export failure falls back to the query the caller passed rather than the normalized form.

* BOT-1956: Keep a client query's provenance for as long as the query lives

Carried over from the review on #79877, which merged before this was picked up. The ids of queries and charts the client sent were request-local, so a tool could persist one of those queries and a later turn read it back after it had left the viewing context, by which point it looked tool-written and a refusal to show it went unaudited. They live in conversation state again, declared in the state schema this time, and each turn adds to what earlier turns recorded.

* BOT-1997: Tell a check that was never needed from one that could not be made

Both came back as a query with no metadata provider, which conflated two different situations. A transform whose source database has been deleted has nothing behind it to resolve, and an admin is supposed to still read that source while repairing it, but it was being withheld like a query whose permissions had thrown. Going the other way, native SQL was only held back when the source had a single stage, because the check for it was the helper that extracts a complete query's SQL and deliberately gives up on multi-stage queries, so a native stage under an MBQL stage still printed its SQL in the unresolved fallback. The two now come back distinctly, and native is recognised by looking for a native stage anywhere in the query.

* BOT-1956: Take client ids back in the shape the app DB returns them

JSON has no sets, so state read back out of metabot_message carries the ids as a vector, and the agent entry point validates its input before memory initialization turns it back into a set. Every conversation with a stored turn therefore failed to start its next one under dev and test instrumentation. The state schema accepts either shape now and normalization settles it, and the test covers the round trip rather than handing a set straight in.

* Hide native SQL from the viewing context when perms can't be checked

* Use with-dynamic-fn-redefs in the sandboxed field transform test

* Hide native SQL from read_resource when perms can't be checked

* Treat an unresolvable database id as an unchecked query

---------

Co-authored-by: Chris Truter <crisptrutski@users.noreply.github.com>
2026-09-17 18:01:18 +01:00
Kyle Doherty
3b9a1df5ac Collections move to a slide-out drawer; the rail lists what is open
The Unofficial rail stops being a place to browse collections and becomes a
place to see what you are working on, matching the prototype's explore rail.

- Collections are now one row that slides a drawer out from the right edge of
  the rail, floating over the content: the tree, other users' personal
  collections and the create action all move into it. The drawer closes on
  Escape, on clicking outside the rail, and on navigating — picking something is
  the drawer's whole job, so it gets out of the way afterwards. The
  `expand-collections-in-nav` setting is no longer read.
- Opening a question, model, metric or dashboard adds it to an Open list in the
  rail, so several things can be held at once and switched between. Closing a
  row only unlists it; the entity is untouched and reopening restores the row in
  place. State is page-session only (`openNavItems` in the app slice), like the
  section switcher.
- Models and Metrics are gone from Data, which is now just Databases. The browse
  list pages stay routed; they are reached from content rather than from a
  permanent nav row.
2026-09-17 12:40:22 -04:00
Kyle Doherty
7e9ffc83ce Remove the top app bar; rail is always shown, header owns search and account
The app bar is gone. In its place the content area gets its own header —
where you are on the left, search and the account switcher on the right —
matching the prototype, where the rail runs the full height of the window and
each page carries its own header.

- New `AppPageHeader`, rendered once above the scrolling content for every
  non-admin page, holding the collection breadcrumbs, the search button and the
  app switcher. `getIsAppBarVisible` becomes `getIsPageHeaderVisible` and keeps
  gating admin / Data Studio / Monitor / fullscreen / embedding.
- The New button moves to a footer pinned over the bottom of the rail, and the
  Metabot button and git-sync controls move into the rail, all inside the
  Unofficial half: that is the creation section, and the prototype's rail footer
  belongs to creation too.
- The logo, the sidebar toggle and the whole open/close/collapse-by-pathname
  machinery are deleted: `isNavbarOpen`, its three actions, `getIsNavbarOpen`,
  `isNavbarOpenForPathname`, `useCloseNavbarOnMount`, and every caller that
  closed the rail on mount or on entering a sidebar. The rail no longer changes
  with the entity you are looking at.
- Consumers that positioned themselves around a collapsible rail now assume it
  is always there: the bulk-action toast is always offset by the rail width, the
  detail view always subtracts it from the viewport, and the query builder
  header always uses its narrow-viewport layout.

Question lineage and the detail-view row nav went with the app bar; the
prototype puts that context in each page's own header rather than in app chrome.
`SearchBar` (the inline field used only by interactive embedding's app bar) is
left in the tree but is no longer reachable.
2026-09-17 12:29:50 -04:00
Chris Truter
e28ae43d1d Various Slackbot upload improvements (#82216)
* Tidy up slackbot CSV upload handling

Report a file that cannot be staged on disk as one failed file rather than
letting the IOException escape to the async handler, which only logs and leaves
the Slack user with no reply at all.

Take the Slack client from the caller instead of re-reading the app token per
file, fetch the uploads Database once, build the upload target in kebab-case,
and express pre-flight failures as system messages so the API namespace stops
hand-rolling its own.

* Trim the upload docstrings and comments to their contracts

Drop the note about the upload module's private `allowed-extensions`: nothing here
relies on that set, so it would go stale silently. The `csv-file?` test asserts
`txt` is not uploadable instead, which fails if anyone widens the set.

* Polish Slackbot upload handling

* Test attachment-only upload failures

* Clarify Slackbot upload return contracts

* Regenerate the modules config after dropping the Database lookup

* Tighten the upload docstrings and comments

* Close the response stream when a redirect Location will not resolve

URI.resolve throws IllegalArgumentException on a malformed Location, and redirect-target runs
in the let binding that computes target, before the cond reaches either .close branch. A Slack
file host answering with a malformed Location therefore leaked the response stream.

Returning nil instead folds the case into the existing "Unexpected response" branch, which
closes the body.

* Cover remote attachments end to end, and say nothing when there is nothing to say

A message carrying only a file stored outside Slack had no route-level coverage: the unit
tests reached remote-file? and the history message, but nothing exercised the reply path
that replaced master's refused-files-message.

A file_share event may also carry an empty :files vector, which with blank text left
handle-message-file-share posting an empty Slack message. It now posts nothing.

* Name an unsupported file type, and stop suggesting a retry for failed uploads

Retrying a failed upload fails the same way. The 415 from create-csv-upload! is the one error safe and useful to
explain, so say the contents don't look like a CSV or TSV.

* Word the unsupported file type message to cover the filename too

create-csv-upload! also answers 415 for an extension it doesn't accept, such as data.CSV, so the message can't
blame the contents.

* Return only the upload history, and describe the size limit as the bot's own

* Show the upload module's own 4xx errors to the user

* Port the upload failure message cases from #81202 to upload-file!

* Count every attached file in the Slack upload counter, under what became of it

Size rejections, unsupported and remote files, and unavailable uploads were never counted, so the error label missed the most common outcomes. Also stop formatting the model id with %d in the success log, where a nil id would have turned an upload into a reported failure.

* Point the user at the server logs when an upload fails for an unexplained reason

Everything that reaches the generic message needs an admin to read the logs, as the #81202 wording said.
2026-09-17 16:20:55 +00:00
Andrei Simionescu
d37f1a9dd3 Record which provider served each LLM call (#82280) 2026-09-17 17:19:48 +01:00
Ryan Laurie
1a32584bb8 Improve App DB Test Sharding (#82538) 2026-09-17 10:02:46 -06:00
Kyle Doherty
35bf6ca092 Official/Unofficial nav split, prototype metric & model detail pages
Ports the load-bearing parts of the consumption/creation split prototype.

Nav
- Two-segment Official/Unofficial switcher at the top of the main rail,
  backed by a `navSection` app-state slice (page-session, not persisted).
  Section is seeded from the URL for deep links, then user-driven, and
  remembers the last path per section for the life of the page.
- Official renders the Library and official-collection trees with their
  *items* hanging off each collection; Unofficial keeps Home, Bookmarks,
  the collections tree, data apps and Trash. The Data links render in both.
- Official membership is the item's own collection being marked official or
  the item living in the Library. Authority level does not inherit in
  Metabase, so neither does this. There is no authority-level search
  parameter, so the official half is filtered client-side.

Browse
- /browse/metrics and /browse/models get the prototype shell (icon + title,
  dot-separated meta, description paragraph) via a shared BrowsePageLayout,
  with the table inside a bordered panel and a header button that flips the
  section. No tabs - the nav switcher is the control.
- Both tables gain a non-sortable Owner column. Model rows now open the
  model detail page rather than the query builder.

Metric page
- The pill-tab bar is gone; the About route is the whole read view: headline
  figure with prior-period and year-over-year deltas, chart, dimension pills,
  then Definition / Breakdowns / Dimensions / Used by / History panels beside
  a sticky fact rail. The six routes stay so Data Studio's re-hosting keeps
  working, reachable through a Back to metric link.
- Used by embeds the existing EE dependency graph rather than a port of the
  prototype's own graph.

Model page
- New /model/:slug/detail with Fields / Definition / Used by / History and
  the same fact rail; the index no longer redirects to actions.

Avatars
- UserAvatar draws a deterministic DiceBear glyphs tile from a stable seed
  instead of rendering initials on a brand-coloured div, so the same person
  carries the same avatar everywhere. The bg colour plumbing it replaces is
  removed, along with the now-dead colour helpers and userColor props.

Notes
- TrendInfo/AboutVisualization are deleted: the headline card now owns the
  figure and its deltas, and printing both would have doubled the number.
- The metric dataset-value helpers moved to common/metrics so the metrics
  feature can use them without importing from browse.
- ModelHistory/ModelUsedBy are local to the models feature because module
  boundaries forbid models -> query_builder imports.
2026-09-17 11:59:28 -04:00
Chris Vandoodewaard
d3ce9de3cc Bump backend deps for open security alerts; define core-provided deps in one place (#82518)
* Bump backend deps for open Snyk alerts; define core-provided deps once

Security fixes:
- athena: netty 4.1.137.Final -> 4.1.138.Final (CVE-2026-89044, HTTP request
  smuggling in netty-codec-http). Stays on the 4.1 line because the Athena JDBC
  driver's async AWS SDK path is not binary-compatible with 4.2.x.
- snowflake: pin zstd-jni 1.5.7-16. snowflake-jdbc-thin 4.0.2 pulls 1.5.6-5,
  which has eight open out-of-bounds read alerts, all fixed in 1.5.7-14.

t-digest (CVE-2026-87822, CVE-2026-87962) is left alone deliberately: Snyk lists
no fixed version, everything from 3.1 up is affected, and it cannot be excluded
because kixi.stats.core requires kixi.stats.digest, which imports TDigest at the
namespace level. Both CVEs are in MergingDigest.fromBytes and need an
attacker-supplied serialized digest; we only ever build digests in-process and
never deserialize one, so we are not exposed. Those two alerts need a dismissal.

Per review feedback on #82261: driver uberjars are stripped of anything core
already ships (build.plugin-uberjar/prune-provided-libs!), so a driver
re-pinning a core-provided library has no effect on what loads, still lands the
artifact in that driver's generated pom.xml for Snyk to alert on, and adds
another place to bump. Replaced 22 such declarations across 7 drivers with
:exclusions on the transitive dep that drags them in, so the version is defined
only in the root deps.edn.

Verified by resolving every driver basis before and after and diffing the
effective shipped set (basis minus core-provided libs): only the two intended
security bumps move, every other driver is byte-for-byte identical. All 8
touched drivers build and pass verify-driver. Regenerated poms confirm all 15
core-provided artifacts are now declared in the root pom.xml and in zero driver
poms.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Update modules/drivers/clickhouse/deps.edn

Co-authored-by: Chris Truter <crisptrutski@users.noreply.github.com>

* Exclude httpcore5 and bcutil-jdk18on to close scan-only version mismatches

Per review on #82518:

- httpcore5: databricks-jdbc-thin 3.4.1 declares 5.3.6 and libthrift 0.24.0
  declares 5.2, while the root deps.edn pins 5.4.3. Only httpcore5-h2 was
  excluded, so httpcore5 still landed in the generated databricks pom.xml --
  and excluding it from the driver alone would have left libthrift's 5.2 in the
  standalone POM for scanners to flag. Excluded from both, matching what
  clickhouse and hive-like already do.

- bcutil-jdk18on: databricks-jdbc-thin declares 1.84 and snowflake-jdbc-thin
  4.0.2 declares 1.82, but both resolve to 1.85 transitively via the root's
  bcpkix-jdk18on 1.85. Same scan-only mismatch as bcpkix/bcprov, which were
  already excluded; added bcutil alongside them in both drivers.

No change to what either driver ships -- these artifacts are core-provided and
already stripped from the driver uberjars by prune-provided-libs!.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Truter <crisptrutski@users.noreply.github.com>
2026-09-17 11:43:38 -04:00
technomancy
249390c4b6 Make mage start-db print a command to connect a client where known. (#82621)
Sometimes you want to start a DB and connect your client directly to
mess around, but you haven't memorized the exact command. Why not make
mage give you a lil hint?
2026-09-17 08:38:48 -07:00
Eric Normand
efbd247394 Ignore FKs from inactive tables when checking source replacement (#82618)
Sync deactivates a dropped table but leaves its fields active, so an FK
field on a dropped or archived table kept reporting
:incompatible-implicit-joins and blocked find and replace on the table
it targets. The incoming-FK check now requires the FK field's table to
be active too.

Fixes #82615
2026-09-17 15:11:05 +00:00
github-automation-metabase
bfc9d9aa91 Auto-update documentation for master (#82604)
Update generated docs (master)

Co-authored-by: github-actions <github-actions@github.com>
2026-09-17 09:34:23 -04:00
Will Fry
8658d420d2 GDGT-3236 Move field sorting loader to end of row to avoid layout shift (#82645)
Move field sorting loader to end of row to avoid layout shift

The loader rendered before the sorting controls, pushing them right while
the field order request was in flight. Rendering it last keeps the
controls in place.

Fixes GDGT-3236

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 13:24:30 +00:00
github-automation-metabase
1a4afe2350 Tighten ratchets (#82655)
Co-authored-by: github-actions <github-actions@github.com>
2026-09-17 13:14:41 +00:00
Mahatthana (Kelvin) Nomsawadi
d53a909a92 Revert "Integration branch for Embedding settings reorganization (#80336)" (#82570)
* Revert "Integration branch for Embedding settings reorganization (#80336)"

This reverts commit b9d217cbb4.

Also drop enable_embedding_modular from the closed stats snapshot
schema. #82447 (Require typed, closed argument schemas in mu/defn)
added that line after #80336, so git revert leaves it behind while
removing the key from the snapshot.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014sqWaxii4kHFrYmW9c45zZ

* fix: keep writing embedding_theme.is_default

The is_default migration stays after the revert, since it may already
have run and migrations cannot be removed, so the column still exists.

- Restore its serdes :skip entry and the theme API key-set
  expectations for endpoints that return the whole row.
- Keep marking seeded Light/Dark themes as default, so the flag stays
  meaningful when the embedding hub work is picked up again.
- Keep returning it from the theme API, including the list endpoint,
  as #80336 did.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014sqWaxii4kHFrYmW9c45zZ

* refactor: keep the settings-components module

#80336 moved the settings page primitives into the shared
settings-components module. That move is not part of what is being
reverted, and later master work builds on it (#82389, #82462), so keep
it: restore the module, its lint entries, and point imports back at it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014sqWaxii4kHFrYmW9c45zZ

* refactor: keep the setup guide rename

#79514 renamed the embedding hub checklist code to setup guide, which is
what the UI already calls it. It is a rename only: the API path, setting
names and persisted state keep their embedding-hub names. Re-apply it on
top of the revert so the name stays free for the embedding hub section
when that work is picked up again.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014sqWaxii4kHFrYmW9c45zZ

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 14:08:13 +01:00
Chris Vandoodewaard
9daae78940 stop set-milestone from exploding on first CI run on new release branch (#82576) 2026-09-17 08:33:54 -04:00
Stas Gavrylov
c881cfaa4e GDGT-2793: Stabilize Python incremental transform test (#82423)
* GDGT-2793: Stabilize Python incremental transform test

* GDGT-2793: Ground Python transform timeouts

* GDGT-2793: Address PR comments
2026-09-17 14:32:10 +03:00
github-automation-metabase
b4f8c8e8aa chore(sdk): bump SDK version on master to 0.64.0-alpha.4 (#82641)
chore(sdk): bump SDK version to 0.64.0-alpha.4 (alpha)
2026-09-17 13:52:53 +04:00
YB
b826490105 [GDGT-2870] fix non-admins unable to create alerts on OSS jars with a paid token installed (#82555)
gate notification permissions on enable-advanced-permissions? not has-feature?
2026-09-17 17:45:03 +08:00
Stas Gavrylov
499bc1160b Consistent typography (part 1): Text/Title on home, collection, Metrics, Data, and Models (#81888)
* Use Title for EmptyState heading and keep heading role

* Use h1 for the Metrics page title

* Use h1 for the Data page title

* Use h1 for collection name and Title/Text in collection empty state

* Fix EmptyState spec to use the illustrationElement prop

* Add hidden h1 and use Text for home greeting and captions

* Use h1 for the Models page title and fix heading levels

- Page title: Title order={2} -> order={1} (parity with Metrics/Data)
- Empty state title: order={3} -> order={2} (avoids h1 -> h3 gap)
- Explanation banner: Title order={6} -> Text size="md" lh="sm" fw={700}
  (banner copy is not a heading; removes h1 -> h6 skip, style-parity swap)

* Remove heading level unit tests

Drops the getByRole heading-level assertions added across the branch
(8 tests in 7 spec files, incl. the new EmptyState spec). Heading
semantics remain covered by the component changes themselves; e2e/a11y
audits own regression coverage.

* Address Copilot comments

* Update HomeCaption to use div

* Fix e2e test broken by HomeCaption's new wrapper element

HomeCaption now wraps its children in a Text component, which added
an extra DOM level. The custom-viz e2e test relied on .parent() from
the caption text landing on the section's shared ancestor with the
recent-item links, which no longer holds.

Scope the assertion via a stable data-testid instead of DOM depth, so
it doesn't re-break on unrelated markup changes.

* GDGT-3181: Address PR comments
2026-09-17 11:57:29 +03:00
Phoomparin Mano
64e7f5cfc8 Whitelist product analytics ingestion origins for data apps (#82483)
whitelist product analytics ingestion origins for data apps
2026-09-17 16:54:49 +08:00
Mahatthana (Kelvin) Nomsawadi
0cf4271e33 feat: warn when the SDK host app runs React 18 (#82432)
* feat: add a host React major version helper to the SDK

Reads the major from the React the host app hands to the bundle, so
both the React 18 deprecation warning and the upcoming unsupported
React version guard read the same source.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UtrkKRneS6NAuJmdYXEHHi

* feat: warn when the SDK host app runs React 18

The SDK will require React 19 in an upcoming release. Surface that
through the existing usage-problem banner and its console message when
the host app renders the SDK with React 18 on localhost or in the host
app's development mode. Production hosts get the same styled console
warning without the banner, and it ignores allowConsoleLog because
losing React 18 support is a breaking change for them. The iframe embed
is excluded since its React version is Metabase's own.

The new warning has the lowest precedence so license errors, the
API-key evaluation warning and the missing "exp" warning still show
when present.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UtrkKRneS6NAuJmdYXEHHi
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RkRuMTMz2vB8zeicbr9nvq

* fix: log SDK usage problems found after mount

The usage problem is computed into the store in an effect, so on the
first render the hook still reads null. The log-once effect ran with
that null, logged nothing and marked itself done, so the real problem
that arrived a render later was never logged.

Only log once init has finished and a problem exists. Before init
finishes, settings still hold placeholder values, so a problem
computed then can be the wrong one, such as the licensed API key
warning on an unlicensed instance.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RkRuMTMz2vB8zeicbr9nvq

* fix: show the React 18 warning before other problems

API-key users on localhost only ever saw the API key warning, so they
would not learn about the React 18 deprecation until they moved to SSO
or production. The React 18 case now comes first in the usage problem
match, right after the MCP app opt-out.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: keep usage errors above the React 18 warning

Putting the React 18 warning first hid the license, disabled-SDK and
non-localhost API key errors on React 18 dev hosts, and let SDK
components render where those errors would stop them. Errors now come
before every warning, and React 18 stays first among the warnings.

The API key error gains `isLocalhost: false` so it can sit above the
localhost API key warnings without catching them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KsyRP238wpM1hKnPg2VRh2

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-17 09:48:42 +01:00
Tyler Plude
69b65837bb [GHY-4523] Add remote sync support for glossary (#82390)
* add entity_id to glossary and key its serialization on it

* route glossary writes through glossary.core

* track glossary entries in remote sync when the Library is synced

* export and import glossary entries through remote sync

* tweaks

* lock glossary edits on read-only instances and backfill ledger rows on upgrade

* fe support

* tests

* resolve term-keyed glossary files to local entity ids during remote sync imports

* fix tests

* e2e adjustments

* fix test

* resolve term-keyed glossary paths without a shape check

* resolve same-term glossary files by term in the pre-load conflict check; fix migration

* Backfill glossary tracking per entry, not all-or-nothing

* fix test order
2026-09-16 22:35:14 -07:00
Romeo Van Snick
6636c786e2 Read the reference database pages from their own requests (#82481)
* Read the reference database pages from their own requests

* Remove the entity selectors from the reference store

* Pass the entity the reference update actions read
2026-09-17 13:34:01 +10:00
Chris Truter
63b1899372 Tidy up some Search comments (#82623)
* Tidy the comments and docstrings from the search index metadata fix

The sql-errors registries carried a diary block about the change instead of a note per code.
Put each vendor's state above its entry, move the driver-alignment note next to the set it
guards, and put the vendor-code rationale on the cond branch that needs it.

Elsewhere: restore a dropped blank line, move the IF EXISTS rationale into the fn body,
and rewrap the comments and docstrings that broke mid-sentence.

* Name MariaDB alongside MySQL in the sql-errors comments

* Restore SQL error registry maintenance context
2026-09-17 01:21:29 +00:00
Romeo Van Snick
0a441968d1 Make getMetadata private to the metadata store (#82551) 2026-09-17 10:28:20 +10:00
Romeo Van Snick
b4513dcdca Move the card copy form next to the other copy forms (#82549)
* Move the card copy form next to the other copy forms

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FeHJzstcAE34DTu4G6btjx

* Point the side-effect ledger at the card copy form's new path

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FeHJzstcAE34DTu4G6btjx

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 10:28:00 +10:00
Chris Truter
1f1643d5b8 Fix search index metadata deletion guard (#80309)
* Fix search index metadata deletion guard

Delete only the pending SearchIndexMetadata row rather than whichever row happens to match the coordinate, and
state that contract in the docstring.

The cleanups the fix leaned on, in the same change:

- Write the index DDL as Honey SQL maps instead of threading `sql.helpers`, which earns its place only when
  clauses are added conditionally.
- Identify a missing index table by SQL state rather than exception class, covering every state our app databases
  report, and name those states and the vendor error codes instead of spelling out literals at each call site.
- Take the connection as an argument rather than encoding it in an `-on-conn!` name suffix.
- Let the index poll survive an index that has not been built yet.
- Move rationale out of the docstrings into comments: a docstring is the caller's contract.
- Skip `with-model-cleanup`'s reindex when no search-relevant rows were deleted, and give `multiple-limits-test`
  its own search data so it stops depending on another test's rows.

* State which safe-batch-upsert! failures throw and which are skipped

The docstring claimed failed retries propagate; only a missing table
after refresh, a missing table on the retry, and interruption do. Any
other failure is logged and the batch is skipped.

* Classify app-db SQL errors centrally instead of matching states in search

app-db.sql-errors keeps the named SQLSTATE and vendor-code registries and
classifies an exception's cause chain into an error class keyword
(:table-not-found, :duplicate-key) that call sites dispatch on. The
search module's private copy of the missing-table states goes away, and
with it the comment about keeping them aligned with the drivers; the
duplicate-key entries are ready for their caller in the lease work
stacked above. MySQL's catch-all integrity-constraint state is
disambiguated by the vendor code, so no db-type parameter is needed.

* Relax the reset pre-delete to any non-active index metadata

delete-pending-index! becomes delete-non-active-index!: a reset now also
clears retired metadata rows for the coordinate, never active ones. The
lifecycle test covers deleting retired and pending rows while the active
row is refused.

* Count every cleanup deletion and reindex when any rows were removed

The search-hooks filter had a blind spot: raw deletes cascade (a user's
personal collection), removing search-relevant rows even when the listed
model has none. Count the total with a reduce and rebuild whenever
something was deleted; only an empty cleanup skips.

* Address search index review feedback
2026-09-16 20:13:38 -04:00
Eric Normand
95368200b1 Least-privilege MCP v2 scopes with step-up (GHY-4543) (#82452)
* List every MCP v2 tool regardless of token scopes (GHY-4543)

tools/list filtered out tools the token lacked scope for, so a client
holding a narrow grant never saw the write, SQL or delivery tools and
had nothing to attempt or step up for. Scope stays enforced at
tools/call; the disabled-tools and client-extension filters still apply.

With scopes out of the listing, list-tools and tools-hash drop their
token-scopes parameter, and the transport's tools-hash-fn becomes
zero-arity.

* Explain MCP v2 scope denials to the model (GHY-4543)

Clients replace a scope denial with their own text ("token expired",
"Insufficient scope", "Tool execution failed"), so the model tells the
user their login expired. Server instructions now say that failure is a
missing permission, and how the user reconnects to grant it. Every tool
description ends with a static sentence naming its required permission by
its consent-screen label, identical for every token: Claude Code caches the
first description it loads for the whole session.

* Answer MCP v2 scope denials with 403 insufficient_scope (GHY-4543)

A scope denial was an HTTP 200 carrying a JSON-RPC error or an isError
result, so clients never stepped up: Claude Code only saves a step-up
scope, and mcp-remote only starts one, on a real 403 with an
insufficient_scope WWW-Authenticate challenge. The registry gate and the
four in-handler scope checks now mark their refusal, and the transport
answers a single (non-batched) message with 403 plus a challenge whose
scope is the held v2 scopes plus the required one.

* Keep a DCR client's registration scope from narrowing its step-up (GHY-4543)

Claude Code registers with the scope it read from the protected-resource
metadata, then steps up on the same client_id with the wider scope from a
403 insufficient_scope challenge. oidc-provider validates a requested scope
against the client's registered scopes, so that step-up was refused with a
400 invalid_request. Once scopes_supported is narrowed to the baseline, no
Claude Code user could reach write, SQL, or delivery.

A dynamically registered client's ceiling is now its registered scopes plus
the default grant scopes, computed on read. Registering can widen the
ceiling (a client that registers for mb:full keeps it) but never narrows
it. RFC 8707 resource narrowing is unchanged.

* Declare each MCP v2 tool's OAuth scope in securitySchemes (GHY-4543)

ChatGPT only steps up for scopes a tool declares: without per-tool
securitySchemes it re-authorizes for its narrow login scope in a loop,
and with them it re-authorizes once for held plus declared scopes.
The scheme is derived from the tool's static :scope in the cached
manifest, so it is the same for every token.

* Advertise only the baseline MCP v2 scopes to new clients (GHY-4543)

The protected-resource metadata `scopes_supported` and the 401 challenge
`scope` now list only `agent:content:read agent:resource:read`, so users
who never need write, SQL, or delivery never grant them. Claude Code and
the Claude connectors take their first-login scope from the resource
metadata; ChatGPT takes it from the challenge. All of them reach the rest
through the 403 `insufficient_scope` step-up and per-tool
`securitySchemes`.

What the resource accepts is unchanged: RFC 8707 narrowing still keeps all
six v2 scopes, and the authorization-server metadata stays wide because
Codex requests that list on every login and cannot step up.

* List every MCP v2 resource regardless of token scopes (GHY-4543)

resources/list now matches tools/list: it is token-independent, so a client can
see the resources it could step up for, and a client that pre-fetches the MCP
Apps shells before stepping up to agent:query:run does not cache them as
missing. resources/read keeps its scope gate.

* Answer MCP v2 resources/read scope denials with 403 insufficient_scope (GHY-4543)

resources/list now shows every resource, so answering a denied read with
"Resource not found" hides nothing and leaves a client with no step-up
signal. A client that pre-fetches an MCP Apps shell with a baseline token
now gets the same challenge tools/call sends, naming agent:query:run.
Unknown URIs stay -32602; inside a batch the denial is an in-band -32600.

* Lead MCP v2 tool descriptions with their required permission (GHY-4543)

Clients truncate long tool descriptions (Claude Code at 2048 characters),
so the appended "Requires the ... permission" sentence was cut from
document_write and execute_query. The model needs that sentence to name
the missing permission after a scope denial, so it now opens every
description, and the server instructions say it starts the description.

* Serve every MCP v2 resource on resources/read regardless of token scopes (GHY-4543)

Claude Desktop reads an MCP Apps shell alongside the tool call. When that
read was refused with 403 insufficient_scope, Desktop never stepped up after
the tool call's own 403; with the shell served, it stepped up automatically
and retried. The shell carries no data and, for a token without the shell's
scope, no UI credential: read-resource now withholds the credential instead
of refusing the read. Chart data stays gated by the tool call,
refresh_ui_credential, and the UI-surface route costs.

* Restore the scope check on MCP v2 data resource reads (GHY-4543)

Only the MCP Apps UI shells need to be served to any token: a refused shell
read stopped Claude Desktop stepping up after the tool call's 403. The fields
catalog carries data and has no such constraint, so reading it without
agent:resource:read again gets the 403 insufficient_scope challenge (in-band
-32600 inside a batch). UI shells stay unscoped, minting a credential only for
a token holding their scope.

* Fit MCP v2 tool descriptions under Claude Code's 2048-character limit (GHY-4543)

Claude Code truncates tool descriptions at 2048 characters, so document_write and
execute_query lost their closing guidance, including execute_query's pointer away
from execute_sql for filtered-dashboard cards. Condense both, and get_content for
margin, deferring detail to learn() topics and argument schemas. A test now holds
every listed description to the limit.

Also correct docstrings and comments left stale by the resources/read and
tools/list scope changes and by credential delivery through refresh_ui_credential.

* Spell out the scope union in the MCP v2 challenge docstring (GHY-4543)

The whitespace linter rejects the confusable Unicode character U+222A.

* Add agent:query:run to the MCP v2 baseline scopes (GHY-4543)

Claude Desktop retries a tool after a step-up over a session that does not
declare MCP Apps support, so a stepped-up visualize_query is refused and the
first chart after a step-up never embeds. Querying and charting therefore
must not require a step-up. The protected-resource metadata, the 401
challenge and default-ask-scopes now advertise agent:content:read
agent:query:run agent:resource:read; writes, raw SQL and delivery still step
up.

* Point MCP v2 chart requests at visualize_query in the server instructions (GHY-4543)

The routing line named only execute_query, and clients like Codex load tool descriptions on demand, so chart requests could miss visualize_query.

* Gate native query handles in transform_write on agent:sql:run and the kill switch (GHY-4543)

A query handle is not proof the SQL gates were spent: /api/embed-mcp/drills
stores a native query under agent:query:run alone. transform_write now runs
the same scope and mcp-execute-sql-enabled checks on a handle that resolves
to native SQL as it does on an inline native definition, matching
question_write.

* Tighten resource registration, step-up scopes, and the DCR ceiling (GHY-4543)

- register-resource! takes a closed map and strips :ui?, so a data resource
  can never be served to a token without its scope.
- step-up-scopes matches held scopes instead of looking them up, so a token
  holding a wildcard such as agent:content:* keeps the surface scopes it
  covers when a client replaces its grant with the challenged scope. It now
  takes the single required scope both callers pass.
- The default grant ceiling is applied only while dynamic registration is
  enabled, so turning registration (or MCP) off stops widening existing
  dynamic clients. The merge is now a pure function with unit tests, and an
  HTTP test pins that static clients are never widened. The legacy-client
  migration test runs with registration disabled, where the stored snapshot
  alone decides, so it exercises the migration again.

* Correct MCP v2 tool descriptions, the documents skill, and server instructions (GHY-4543)

- document_write lists the valid entity-link models again.
- The documents skill said edits store new_str literally; the tool parses it
  as Markdown, as edit-replacement-parses-markdown-test pins.
- execute_query no longer claims to save cards (its handles save as cards),
  and names the previous-stage field ref shape instead of an ambiguous
  column-name string.
- The server instructions attribute the rewritten permission failure to the
  client and cover resource reads, which get the same 403.

* Pin the MCP v2 scope-denial and UI-credential contracts in tests (GHY-4543)

Closes gaps the review found, where removing a check would stay green:
- question_write and subscription_write in-handler denials carry
  :insufficient-scope at the registry, like alert_write's.
- A denied tools/call sent as a notification is a bare 202, no challenge.
- Wildcard and unrestricted tokens get the shell credential; the fields
  catalog mints none, over the registry and over HTTP.
- tool-and-resource-scopes-agree-test asserts the credential is realized,
  since any token now reads a shell.
- advertised-scopes-are-distinct-test covers the advertised baseline, which
  is a vector and can repeat, and its docstring says so.

* Clean up MCP v2 scope-denial helpers, header quoting, and docs (GHY-4543)

- Share one scope-denial message and detail builder between tool calls and
  resource reads, one step-up challenge helper, and a common
  throw-insufficient-scope! for the four in-handler scope checks.
- quoted-string replaces " and \ instead of backslash-escaping them, since
  RFC 6750 excludes both from scope and error_description values; the 401
  challenge now quotes through it too.
- Trim docstrings to their contract and move the reasons into comments.
- The unrestricted UI credential comes from any unrestricted session (cookie,
  API key, or an mb:full token), not only cookie sessions.
- Note that the DCR ceiling applies only while registration is enabled, and
  fix stale comments, test names and strings, long lines, order-sensitive
  scope assertions, and hard-coded consent labels in tests.

* Always widen dynamic OAuth clients to the MCP scopes (GHY-4543)

Claude Code registers with the baseline scopes and steps up on the same
client_id. With dynamic registration turned off, that step-up /authorize
was answered with a raw 400. A dynamic client now always gains the six v2
MCP scopes on read, each still behind consent; the remaining agent-API
ceiling is added only while registration is enabled. Registration itself
stays blocked.

* Close the MCP v2 scope test gaps (GHY-4543)

Cover the transform_write update path's native gates, the HTTP 403
insufficient_scope challenge question_write and transform_write send from
inside their handlers, saving a drill handle minted through
/api/embed-mcp/drills, and the closed resource registration schema. Give
the locale test a positive control, merge the two tools/list
token-independence tests into one that compares the whole payload, and
drop the permission-sentence truncation test the fit and starts-with
tests already cover.

* Tidy MCP v2 scope wording and naming (GHY-4543)

Rename registry/insufficient-scope to insufficient-scope-detail so it
does not read like the transport fn of the same name. Drop the server
instructions clause telling the model to read a resource's scope off an
error clients hide. Document agent:resource:read and the handler-internal
scope checks in the module README, correct the stale comments about UI
resources gating on agent:query:run, and wrap the lines this branch left
over 120 columns.

* Let the test fallback template win over a local frontend build (GHY-4543)

The MCP Apps shell renders from the built `embed-mcp.html` when one exists,
so a worktree that has run a frontend build shadowed the inline fallback the
credential tests install. The production template has embedded no
`uiCredential` since #81041, so those tests failed locally and passed on CI,
where backend-only runs produce no build.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014QT1v8on6aPT2UeHCojGj8

* Gate the MCP grant-ceiling widening on the MCP kill switch (GHY-4543)

A dynamically registered client gained the MCP surface scopes on every read of
its client record, unconditionally. Only the wider `default-grant-scopes`
ceiling was gated, on `oauth-server-dynamic-registration-enabled`.

`agent:resource:read` is one of those MCP scopes and is also the declared scope
of `POST /api/agent/v1/read-resource`, which a separate lever gates. So with MCP
disabled and the agent API enabled, a dynamic client could authorize for a scope
its registration never included and spend it at a surface still serving.

Read `mcp-enabled?` in the effectful client-store wrapper and pass no MCP scopes
down when it is off; `widen-to-grant-ceiling` stays pure. While MCP is enabled
the existing behavior is unchanged: the MCP scopes are still added whatever the
registration setting says, so a client registered with the baseline can step up.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014QT1v8on6aPT2UeHCojGj8

* Cover the built MCP embed template's no-credential property (GHY-4543)

Serving MCP Apps UI shells to any token holder rests on the built
embed-mcp.html discarding the credential it is handed (since #81041) — the
iframe fetches its own through `refresh_ui_credential`. Nothing asserted that,
so a frontend change reintroducing a `uiCredential` placeholder would land
silently now that the outer scope gate on shell reads is gone.

Add `metabase.mcp.ui-resource-test` rendering the built template with a
credential and asserting the output carries neither the credential nor a
`uiCredential` placeholder. Backend-only CI runs have no frontend build, so the
test logs a skip instead of failing when the resource is absent.

Also gate the test fallback template's precedence over the built one on
`is-test?`, so the reorder can only ever apply in a test process.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014QT1v8on6aPT2UeHCojGj8

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 18:21:34 -05:00
Romeo Van Snick
2602fec3cd Record how many bytes of CSS a page loads (#82541)
Claude-Session: https://claude.ai/code/session_01171Lkmug15JzrcN2aMBLMk

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 06:38:33 +10:00
Kevin Kelleher
b9d8328b19 Adds 'Upgrade now' banner (#82214)
* initial upgrade now banner

* remove security center integration

* fix compareVersions types

* automation, cleanup, tests, fetch version-info.json every hour

* Change CheckForNewVersions poll interval to every 6 hours

https://github.com/metabase/metabase/pull/82214#discussion_r3993430218

---------

Co-authored-by: Mike Appleby <86076+appleby@users.noreply.github.com>
2026-09-16 20:27:32 +00:00
bryan
7d94211ecd Fix two closed-schema regressions from #82447 (#82581)
* Accept localized names in CollectionInstance

The sidebar fails to load: any call passing the Trash collection through
`CollectionInstance` throws.

    Invalid input: ["an instance of the root Collection, got: (toucan2.instance/instance
    :model/Collection {... :name #metabase.util.i18n.UserLocalizedString{:format-string
    "Trash" ...} :type "trash" ...})"]

`trash-collection` sets `:name` with `deferred-tru`, yielding a
`UserLocalizedString` rather than a `String`. #82447 added `CollectionInstance`
with `[:name :string]`, which that value cannot satisfy, so the Trash
collection matches neither branch of `[:or RootCollection CollectionInstance]`.
Malli reports only the first branch's `:error/message`, which is why the error
misleadingly names the root Collection.

Widen `:name` to accept localized strings via the existing
`ms/LocalizedString`. The deferral is deliberate: `trash-collection` is
memoized across requests, so eager-rendering with `str` would freeze the name
in whichever locale filled the cache (#59662). `:slug` and `:description` are
left alone; they are not localized.

Reproduced and verified against a running instance: `shared-tenant-collection?`,
`children-location` and `descendant-ids` no longer throw on the Trash
collection, and `GET /api/collection/tree`, `/api/collection`,
`/api/collection/:trash-id` and `/api/collection/:trash-id/items` all return
200 with the name rendering as "Trash".

* Reopen the cols schema in the static-viz color picker

Rendering a table with conditional formatting throws:

    Invalid input: {:cols [{:display_name ["disallowed key, got: \"Category\""],
                            :base_type ["disallowed key, got: :type/Text\""]} ...]}
    fn-name: cell-background-colors

`QueryResults` describes the value handed to the shared color-picking JS. Its
docstring says it is "a pretty loose schema, more as a safety net" that "just
needs column names from `:cols`" — callers pass whole column metadata maps and
only `:name` is ever read. #82447 closed the inner `:cols` map, so every caller
passing `:display_name` or `:base_type` now fails validation.

Mark that map `::mr/deliberately-open` to restore the documented intent. The
outer map stays closed; the extra keys only ever appear on columns.
2026-09-16 20:23:32 +00:00
Chris Truter
7c771f3c33 Add GPT-6 Astra support (#82499)
* Add GPT-6 Astra to the OpenAI, Azure and Bedrock model pickers

The OpenAI allowlist stopped at the 5.6 family, and the reasoning/temperature
check keyed off a gpt-5 prefix, so a forced request would have sent
temperature, which Astra rejects. Widen the check to GPT-5 and GPT-6 and add
the model to the three adapters that sell it directly.

OpenRouter is left out: its allowlist needs a live reasoning probe per model.

* Correct Bedrock context window comment
2026-09-16 20:17:17 +00:00
Romeo Van Snick
3c3979838a Move the SDK storybook instance URL out of the storybook config (#82544)
Claude-Session: https://claude.ai/code/session_01FeHJzstcAE34DTu4G6btjx

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 06:06:06 +10:00
Eric Normand
49a7152e97 Give flat-table-content typed argument schemas so body.clj loads (#82591)
* Give flat-table-content typed argument schemas so body.clj loads

#82447 made mu/defn throw at macroexpansion when an argument's schema does
not declare its shape. #82277 landed flat-table-content with an unschema'd
card argument and an unschema'd open map, so requiring
metabase.channel.render.body now fails to load.

Use the same argument schemas as its sibling prep-for-html-rendering, which
receives the same two values from `render :table`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014QT1v8on6aPT2UeHCojGj8

* Narrow simple-pivot-content's color-data cols to :name

The second collision between #82447 and #82277: #82447 closed the cols
entries of js.color's QueryResults to exactly {:name :string}, and #82277's
simple-pivot-content hands cell-background-colors full cols, so every
simple-pivot-* test errors with :metabase.util.malli.fn/invalid-input.

Narrow the cols the way the flat-table path already does. The colors are
unaffected: the JS reads cols only to build a name->index map
(makeCellBackgroundGetter -> getColumnIndexesByName).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014QT1v8on6aPT2UeHCojGj8

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 21:32:54 +02:00
Timofey Kachalov
a367aac5d8 Require defineQuery and defineAction exports in data-app hooks (#82489) 2026-09-16 20:19:27 +04:00
Maksym Yakubych
f02360567a Make Metabot metric mentions clickable (#82348) 2026-09-16 16:12:47 +00:00
Eric Normand
6318ae1d86 Build MCP v2 agent-facing text with msg so interpolated values are quoted and escaped (#82437)
* Add a kondo lint for unquoted interpolation into agent-facing prose

Teaching errors, paging hints, and steering lines read to an LLM as the
server speaking. A warehouse table name, driver error, or card name
interpolated into that prose unquoted can carry newlines and text that
pose as server-authored instructions.

`:metabase/unquoted-prose-interpolation` flags `format` %s arguments,
`tru`-family {n} arguments, and non-literal arguments of a `str` call
over a prose literal, unless they are literals, `pr-str` calls, or a
`str/join` over `(map pr-str ...)`. It is enabled only for
`metabase.mcp.v2.*` and `metabase.agent-lib.*` source namespaces.

The rule currently reports 243 findings; a follow-up tunes out the
noise and fixes the real sites.

* Scope the prose-interpolation lint to MCP v2 and the MCP transport

* Add the MCP v2 message builder and lint its lines

`metabase.mcp.v2.message` builds agent-facing prose: `msg` takes a
vector of line format strings plus arguments, and `render` joins the
lines with newlines and interpolates the arguments, cleaning each one
unless it is `raw` or a nested message. `clean` quotes with `pr-str`
and escapes invisible, line-breaking, and quote-like characters as
\uXXXX. `render` never throws: a malformed message, a format failure,
or anything that isn't a message renders with every part cleaned.

`:metabase/agent-message-lines` checks `msg` calls: the lines must be a
literal vector of string literals with no line breaks, `%n`, or
control characters, and the arguments must match the specifiers.

* Render MCP v2 messages at the tool-content and JSON-RPC exits

Teaching errors thrown with a message render it into the exception
message and carry the message in ex-data, so the caller-facing error
paths hand it on to be rendered. error-content, success-content, and
jsonrpc-error render messages; plain strings pass through unchanged for
now. message? is now public.

* Lint that MCP v2 exits receive msg-built text

Add :metabase/agent-message-exit, enabled for agent-prose-namespaces. It flags the message argument of throw-teaching-error, error-content, jsonrpc-error, and success-content when it is syntactically text (a string literal, a string-building call, or a threading, branching, or body form yielding one), and ex-info calls whose data literal carries a 4xx :status-code or an error-code key.

Point the unquoted-prose-interpolation messages at msg instead of pr-str.

* Scope the agent-prose lints in their hooks instead of with :off switches

The prose-interpolation and agent-message-exit linters are :warning globally; their hooks run only in non-test metabase.mcp.v2.* and metabase.mcp.transport namespaces. agent-message-lines runs in any non-test file. This keeps config.edn free of new :off switches for the kondo ratchet.

* Build shared MCP v2 prose with msg

Convert the shared response helpers, the transport's JSON-RPC errors, the
registry's pre-dispatch errors, and the API's method and resource errors to
msg. Registry errors now carry a message, humanized schema detail cleans its
paths and expectations, and the query pipeline's rewrapped schema failure
keeps the original message as a cleaned argument. Adds message-ex-info and
list-message to common.

* Build shared MCP v2 write, resolve, and skills prose with msg

Convert the _write dispatch, id resolution, recipient, projection, skills,
and recovery-hint text. Caller-supplied values (ids, clear names, methods,
recipients) are cleaned; server-declared names stay bare. The skills
catalog, pack, and reference texts are now messages, and the recovery
hints render msg-built English text for the shared pipeline.

* Build MCP v2 bookmark and UI credential prose with msg

* Build MCP v2 alert and subscription prose with msg

* Build MCP v2 transform prose with msg

* Build MCP v2 collection, duplicate, content, and document prose with msg

Card types, include sections, and document snippets are now cleaned in
teaching errors; the tools' own argument keys and type names stay bare.
The default copy name stays a plain content name, and get_content passes
its payload to success-content for encoding.

* Build MCP v2 metric, segment, measure, and question prose with msg

Normalizer and pipeline exception text, card types, template tag names,
column names, and schema summaries are cleaned in teaching errors. The
definition-shape and metric shape-rule sentences are messages nested into
the errors that end with them, so resolve-external-query receives a
message hint. The native-SQL scope refusal goes through
throw-teaching-error.

* Build MCP v2 parameter-values and dashboard-op prose with msg

Stored parameter ids and names, template-tag names and types, and text-card
placeholders are cleaned in teaching errors, listed with list-message. The
get_parameter_values steering lines are messages nested after the JSON
payload, including the shared truncation line. op-error! takes a message
detail; op and argument names the server declares stay bare.

* Build search, learn, and visualize MCP v2 prose with msg

* Build query and dashboard MCP v2 prose with msg

* Build browse MCP v2 prose with msg

* Name the op in dashboard op errors instead of in their detail

op-error! now takes the op name separately and renders `op <idx> (<op-name>): <detail>`, nesting the
detail message. Callers that still passed strings (the unknown-card check, add_link, and the unknown-op
fallback) rendered doubly quoted; the unknown-op fallback cleans the caller's unmatched op name itself.

* Clean plain strings whole at every MCP v2 exit

error-content, success-content, and jsonrpc-error now render everything
through message/render, so a plain string (a string teaching error, or an
upstream exception message surfaced by ->mcp-error-content or
caller-safe-error-message) reaches the client quoted and escaped instead
of passing through. Only msg values render as prose; "Internal error" is
now a msg so it stays unquoted.

* Scope MCP v2 prose linters by config and retire unquoted-prose-interpolation

Remove the format/str/tru interpolation checks now that msg cleans values at
runtime, restoring the i18n hooks to strict-apostrophes. msg lines may be split
with str over literals, and %s can't take width, precision, or flags, nor %S.
The linters are enabled for MCP transport and v2 source namespaces through an
ns-group instead of in-hook namespace checks.

* Harden MCP v2 message rendering, truncation, and rewrapped errors

clean escapes only double-quote look-alikes, keeping single quotes such as
U+2019 so names survive being copied back. render treats %S and %s with
flags, width, or precision as malformed, and renders a fixed sentence when
an argument can't be printed. ellipsize shortens string arguments inside
their quotes, falling back to message/truncate, which cuts a rendering
without leaving a quoted value open. An exception rewrapped with new text
surfaces that text instead of its stale ex-data message.

* Send MCP v2 recovery hints, throttle, and extension errors as prose

Recovery hints are now messages nested after the quoted pipeline text,
added by the v2 query entry point instead of appended by the shared
pipeline. The throttle refusal and missing-extensions error are
server-written messages rather than quoted strings, and the default
duplicate name is localized again with tru.

* Describe recovery-hint contract now that MCP v2 attaches its own hints

* Surface rewrapped exception text in MCP v2 query and definition errors

Use common/exception-message instead of queries' private copy, which
preferred the stale ex-data message over a rewrapped exception's new text.

* Restore plain MCP code restructured only for the retired prose lint

Inline the SSE framing and native summary prefix, return the WWW-Authenticate
builder and schema descriptions to plain str/format, and restore master's text
for developer-only load-time and 500 errors. Agent-facing output is unchanged.

* Tighten MCP v2 message docstrings to their own contracts

* Give MCP v2 message fns malli schemas

* Wrap long MCP v2 msg lines and build the learn catalog as one message

* GHY-4544: Widen agent-message lint to all MCP source and message-taking helpers

Rename hooks.metabase.prose-interpolation to hooks.metabase.agent-message.
Check msg lines in every non-test metabase.mcp namespace; keep the exit
check on transport and v2, whose 4xx errors are MCP exits. Flag malformed
format specifiers, treat cond/condp/case results as text, and check the
messages passed to op-error!, check-execute-sql-enabled!, ellipsize, and
list-content's :empty-hint. Split lint-msg into helpers.

* Never lengthen a quoted value when truncating a message

A value one character over its budget kept its full text and still gained an ellipsis, rendering longer than the uncut message. Cut values and whole values are now measured the same way, and the cut search never considers the whole value.

* Harden message cleaning and share render's formatting with truncation

clean binds the print vars that affect quoting, so a caller's *print-readably* false can't strip the quotes off a value, and is now private. More double-quote look-alikes are escaped. render and truncation share one formatting helper, and truncation helpers are named for what they return.

* Render schema failures as readable detail and add schemas to common message helpers

An invalid-input schema failure now names its problems with humanize-detail instead of printing the humanized data as one quoted value. message-ex-info, exception-message, list-message, and ellipsize are mu/defns, the message helpers move ahead of the teaching errors that use them, and pure helpers are named for what they return.

* Omit a missing exception message from MCP v2 errors and keep the original as cause

A nil exception message rendered as null (or an empty quoted string) in the
query pipeline rewraps and the definition, metric, transform, and inline
question normalization errors. Those messages now leave that text out.
with-schema-detail and with-recovery-hint rethrow with the original exception
as the cause, so logged traces show the pipeline's throw site.

* Unquote get_content item errors and name alert schedule fields readably

A get_content item's error is a JSON string value, so a plain upstream
message goes in as it is instead of quoted and escaped twice. alert_write's
uncompilable-schedule error lists the schedule's fields with their values
cleaned, instead of a double-printed map.

* Tidy MCP v2 message construction and naming

Send the throttle library's refusal as its own text, nest raw values
directly, inline duplicate's copy-name, rename query-failed and browse's
message locals, replace search's empty filler messages with a list join,
trim two Metabot docstrings to their contract, and mark two pure
dashboard_ops tests parallel.

* Judge threading forms by their result and check :empty-hint through browse's paged-list-content

The exit check read the threaded value of -> and ->> rather than what the form
returns. ->, ->>, some->, and some->> now take text-ness from their last step,
cond-> and cond->> from the initial value or any step, as-> from its last form,
and doto from its target; ex-message counts as building a string. The
:empty-hint check is also registered for browse's paged-list-content, which
passes its options map to list-content.

* Keep a truncated MCP v2 message's pieces so truncating it again closes its quotes

A truncation is now a message of the pieces it kept: server text as raw text and
a cleaned value as itself or its cut excerpt, so a later truncation cuts each
piece as the kind it is. Also registers the message and raw schemas, shares one
surrogate-safe string-prefix, merges the argument dispatch, and renders keywords
as their names.

* Wrap long lines in the MCP v2 common tests

* Pass server text to MCP v2 helpers as messages and name missing identifiers

Helpers that interpolated a string parameter through message/raw on the strength of a
comment about their call sites now take a message: alert and subscription scope and
schedule checks, parameter lookup, segment/measure name checks, duplicate's trash check,
alert id checks, and dashboard op errors. Rendered text is unchanged, except that
dashboard_write's unknown-card refusal now quotes the op name as the caller sent it.

A missing tool name, method, op, or dashcard/tab/parameter id now gets its own sentence
instead of rendering as null.

* Keep surrogate pairs whole when cutting document_write error snippets

* Wrap MCP v2 test lines longer than 120 characters

* Cut document_write error snippets with the shared surrogate-safe string-prefix

* Pass single-line server text to MCP v2 messages as raw

A message with one line and no arguments that is only ever interpolated into another message renders the same as raw text, so those call sites now pass (message/raw "..."). Messages that reach an exit, render on their own, or are returned as a top-level message keep msg. Helper docstrings now describe those parameters as server text.

* Inline one-use MCP v2 message helpers

require-hour! in alert_write and subscription_write and catalog-line in skills existed only to keep message-wrapped lines short. With raw text at the call sites they read as direct calls again.

* Allow a plain subject for the MCP v2 execute_sql kill-switch check

The helper interpolates its subject into its own msg, so a string literal is cleaned like any value.

* Minimize raw in MCP v2 prose

Identifiers interpolated into agent-facing messages are now cleaned rather than
passed as raw; raw moves to where server text is defined. Leftover call-site
comments are removed, lists of names render via list-message/lines-message, and
test expectations are updated to match.

* Quote field names in MCP v2 errors and clean remaining identifiers

Field, argument, and tool names in fixed agent-facing msg lines use double quotes
instead of backticks, matching names interpolated through cleaning. Code snippets
keep their backticks. common.clj cleans the not-found model name, the schema-failure
function name, select-fields type names and nearest catalog paths, and the
truncation-line param instead of passing them raw. alert_write's ignored-field error
says "An" before an hourly schedule, as subscription_write does. Test expectations
updated to match.

* Close MCP v2 message arg schemas for mu/defn's shape check and name not-found types as server text

* Allow only %s, %d, and %% in MCP v2 msg lines and govern enterprise MCP namespaces

Match the runtime's allowlist: a `msg` line may use `%s` with no flags but `<`, `%d` with flags and
width but no precision, and a bare `%%`; any other conversion would alter or hide a cleaned value.
Flag private-use characters too, since the truncation markers delimit arguments with them, and cap
the argument index at nine digits so an absurd one reports as malformed instead of overflowing.

The two MCP ns-groups now also match `metabase-enterprise.mcp.*`, which they left ungoverned.

* Bound MCP v2 message printing to 100 elements and 10 levels

An unbounded sequence argument made render hang and a multi-million-element vector rendered
megabytes of text, since printed pinned *print-length* and *print-level* to nil.

* Take only strings as MCP v2 raw message text

A non-string raw was interpolated into the formatted template instead of being kept as a text
piece, which could collide with the private-use truncation markers. Message lines holding
private-use characters are now malformed for the same reason.

* Allowlist MCP v2 message format conversions and widen quote look-alikes

Only %%, a plain %s, and a %d without precision format; every other conversion — %b, %h, %c,
%x, %f, %tY and friends, which reinterpret an already-cleaned argument — makes the message
malformed, so it renders fully cleaned. The double-quote look-alike set gains the ornaments,
ditto mark, and prime family found by sweeping the Unicode blocks it draws from.

* Clean get_content per-item error text instead of trusting JSON quoting

JSON escapes control characters and double quotes but leaves category-Cf
characters (a bidi override, a zero-width space) intact, so a plain-string
caller-facing error reached the model raw. Render it like every other agent
message, which quotes and \uXXXX-escapes the whole string.

* Wrap MCP v2 message lines the branch pushed past 120 columns

Text is unchanged: each line splits into a `str` of the same pieces, or moves
a word to the continuation line.

* Flag msg lines whose argument reference String/format can't resolve

The kondo hook counted a leading %< and an explicit 0$ as consuming no
argument, so a msg using either linted clean while the runtime's format
threw and fell back to rendering every part quoted.
2026-09-16 15:53:05 +00:00
Bartosz Rybacki
f3d1360240 GDGT-2751 Apply the Table viz's pivot toggle in subscriptions (#82277)
* GDGT-2751 Apply the Table viz's pivot toggle in subscriptions

A Table card with the "Pivot table" toggle on arrives flat in a
dashboard subscription or alert. The toggle is a browser-only
transform; the three settings reach the static renderer, which
never read them.

Port the browser's pivot into `render :table`, including its sort
rule, and draw the grid with the existing pivot renderer. The grid
is built from every row and is not cut at the attachment row limit,
like a Pivot Table card. The server-side dashboard PDF gives such a
card the pivot treatment too, so it no longer prints the source row
count under the grid. CSV and XLSX attachments stay flat; they are
a separate code path.

3 functions needed to be moved. A pure move, no changes: 
`setting-value`, `pivot-cell->str` and `pivot->hiccup` go from 
the pivot section to a section of their own under the 
`render` multimethod.
2026-09-16 17:37:44 +02:00
Sloan Sparger
efe0882aca [Metabot] Remove transform_codegen profile (#82208) 2026-09-16 09:17:13 -05:00
Sébastien
f1a48e2b5c SDK entity navigation drops the click behavior filter when the target is a GUI (MBQL) question (#82409)
* Fix SDK entity navigation dropping click behavior filters on GUI questions

* Give ad-hoc question targets their own click behavior type

* Cover the GUI click behavior target with an e2e test

* Fix the internal-navigation dashcard setup clobbering itself
2026-09-16 14:51:54 +02:00
Chris Vandoodewaard
9b59d17d10 notify release managers if auto release is skipped (#82528) 2026-09-16 08:03:13 -04:00
Will Fry
b084ade539 GDGT-3224: migrate TabRow to CSS modules to fix gradient theme issue (#82482)
* GDGT: migrate TabRow to CSS modules to fix gradient theme issue

* Update Loki Snapshots

* increase scroll gradient size

* Update Loki Snapshots

* Add dark theme story for narrow dashboard with many tabs

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* Add placeholder Loki reference for dark theme narrow tabs story

Copy of the light theme snapshot so the loki-update workflow detects a
diff and replaces it with the CI-generated image.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* Update Loki Snapshots

* move left/right styles into existing classes

---------

Co-authored-by: Metabase Automation <github-automation@metabase.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-09-16 11:01:26 +00:00
Nemanja Glumac
2980fd7f29 Stub floating-ui positioning in unit tests (#82493) 2026-09-16 12:53:55 +02:00
Will Fry
6faa93e40c GDGT-3230 fix lack of highlight in CardEmbedNode title input (#82520) 2026-09-16 11:49:58 +01:00
Romeo Van Snick
d3db1a1c53 Read three single-entity lookups from their own requests (#82550)
Read the segment's table, the list's table and the editor's database from their own requests
2026-09-16 19:39:20 +10:00
Timofey Kachalov
82d075c73c Expose a DateRangePopover in the data-app SDK namespace (#82291) 2026-09-16 11:40:43 +04:00
Romeo Van Snick
9f00ac4fc0 Dissolve the hoc module into its four consumers (#82547)
Claude-Session: https://claude.ai/code/session_01FeHJzstcAE34DTu4G6btjx

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 15:18:19 +10:00
Romeo Van Snick
4b5f24e0a9 Read the reference segment pages from their own requests (#82475)
* Read the reference segment pages from their own requests

* Remove the segment selectors from the reference store
2026-09-16 13:59:59 +10:00
Fraser
46541523bc Tier documents and require explicit shared-tier coverage (#82542)
* Tier documents and require explicit shared-tier coverage

* Check shared tier coverage directly in the unit spec
2026-09-16 15:44:38 +12:00
Fraser
1b0ff2917f Preserve the visual-test required check for Loki (#82539) 2026-09-16 15:44:27 +12:00
bryan
4a3d32ae0a Bind inline values in app-DB queries with [:auto/param ...] (#82373)
* Require app-DB value slots to hold bound parameters

HoneySQL resolves a map in a value slot toward SQL structure rather than
data, so a value can reach the database as syntax instead of as a bound
parameter. `[:param ...]` is the only form that always binds.

Adds `metabase.app-db.value-guard`:

  - `auto-param` rewrites inline `[:param* v]` markers into HoneySQL's
    `[:param :kN]` plus the params map, so callers do not keep a separate
    map in sync.
  - `long*`/`longs` coerce ids, which covers the common lookup without
    param noise and rejects anything that is not a number.
  - `assert-values-wrapped!` walks the compiled query at the Toucan
    compile step and throws when a value slot holds something that could
    become SQL.

The walk is clause-aware: it descends only into `:where`/`:having`/`:set`/
`:values`/`:join-by` and knows which argument of each operator is a value,
so column and operator positions are left alone. An operator that is not
classified fails closed.

This is an allowlist and lives apart from `honeysql-guard`, which is a
blocklist of known shapes and is backported to older versions.

Two forms needed handling beyond plain scalars and params: h2x wrappers,
which are checked by their payload rather than trusted by their wrapper,
and HoneySQL's `:%fn.col` call shorthand, which is a closed enumerated set
rather than a prefix test so that a constructed keyword cannot pass.

`strict-modules` is empty for now. A module joins it once its call sites
carry params or coercions, at which point plain scalars are rejected there
too.

* Accept java.time values in value slots

Toucan's before-insert hooks add created_at/updated_at as ZonedDateTime,
which clojure.core/inst? does not cover, so every insert was rejected.
All java.time.temporal.Temporal values bind as parameters.

* Check nested values, report the offending leaf, defer installation

Three changes, all from measuring the check against a real module.

A dev-authored marker on a subquery asserts that its SQL is dev-authored.
It says nothing about the values inside it, which are just as reachable
from a request, so the check now descends into a marked subquery instead
of passing it wholesale. `honeysql-guard` also descends here, but with a
blocklist, so a nested unwrapped value passes it today.

When a nested value was rejected the error named the enclosing subquery
rather than the value, which pointed debugging at the wrong place. The
nested check now throws on the leaf.

The check is no longer installed at the Toucan compile step. Against the
`collections` module it errored 91 of 148 tests, and none of those was an
unparameterized user value -- they were correlated subqueries, a UNION
used as an operand, and SQL function calls in `:set`, all of which are
real SQL that happens to sit in a value-bearing clause. Installing it
globally is therefore not the right first step; per-namespace opt-in is,
and that opt-in has to key off the calling namespace rather than the
model, since every Toucan model keyword is `:model/Something` and carries
no module information.

* Adopt the value check in the content-translation queries

Gives the check one real caller instead of leaving it unused. `locale`
reaches `translations-for-locale` from a request query parameter, so it is
coerced where it enters the query and the query is checked before it runs.

Adoption is an ordinary function call at the site it protects -- no compile
hook, no ambient state -- so the next namespace adopts independently and
nothing changes for namespaces that have not.

Adds `str*` alongside the id coercions. It asserts a value is already a
string, which rejects the maps and keyword vectors HoneySQL would compile as
SQL while leaving a legitimate string untouched.

`checked` deliberately does not demand bound parameters. A coercion returns
a plain scalar, indistinguishable at runtime from an unwrapped one, so that
stronger property cannot be asserted from the compiled form.

Also records a blocker found while writing this: `[:param ...]` needs its
values in a separate `:params` map, and `honeysql-guard` rejects any unmarked
nested map, so that map never reaches HoneySQL. `[:param]` is therefore
unusable through Toucan until `honeysql-guard` learns about `:params`, which
is a change to a backported namespace and is left alone here.

* Scope the value check to namespaces that have adopted it

Runs the check inside Toucan's compile step again, but only for queries
issued from a namespace listed in `enforcing-namespace-prefixes`. A query
from anywhere else compiles exactly as before, so a namespace adopts on its
own schedule and nothing has to be switched on everywhere at once.

The alternative was a `checked` call at each query site, which only works as
long as everyone remembers to write it. Putting it in the pipeline means an
adopting namespace cannot forget.

Scoping by namespace rather than by model because a model is shared: opting
in `:model/Card` would opt in collections, search, dashboards and
serialization together, which is the big-bang this is trying to avoid. A
`db.clj` is the unit a team actually cleans up, and a prefix entry covers a
whole module once it is ready.

`enforcing-caller?` reads the call stack rather than an ambient binding. The
check runs several frames below the caller inside Toucan, so a dynamic var
would have to be threaded through Toucan internals and would not survive a
thread hand-off; the stack already carries the answer. Measured at ~4us per
query against a DB round trip of hundreds, and only paid at the compile step.

`enforcing-namespace-prefixes` is public so a clj-kondo hook can read the
same set and check adopted namespaces at author time.

Tests prove both directions: a query from the adopting namespace is rejected,
and the same query from a non-adopting namespace is not. A bare keyword in a
value slot is the discriminator, since `honeysql-guard` permits it and this
check does not. `collections` stays at the baseline 1958 assertions, 0 errors.

* Bind values inline with [:auto/param ...]

Makes the design doc's ergonomic half real. A caller writes the value where
it belongs and `bound` lifts it into the params map HoneySQL wants, so the
two never have to be kept in sync:

    (bound [query {:where [:= :locale [:auto/param locale]]}]
      (t2/select :model/ContentTranslation query))

Two things had to be fixed for this to reach the database.

`honeysql-guard` rejected the `:params` map as an unmarked nested map. It
now treats `:params` like `:set` and `:values` -- keys that hold values
rather than nested queries -- so its contents are still checked. Nothing in
`:params` can become SQL regardless of shape, since HoneySQL binds it as a
parameter, so checking it catches a value JDBC would later refuse rather
than an injection.

HoneySQL also takes params as a format option rather than a query clause, so
`bound` passes them through `toucan2.honeysql2/*options*`.

The marker is `:auto/param` rather than a bare keyword so it cannot collide
with a HoneySQL operator, and an unresolved one now throws: HoneySQL would
otherwise compile `[:auto/param v]` into a `PARAM(?)` function call, so
forgetting `bound` would produce wrong SQL instead of an error.

* Lift [:auto/param ...] in the pipeline instead of at the call site

A caller now writes the marker inline and nothing else:

    (t2/select :model/ContentTranslation
               {:where    [:= :locale [:auto/param locale]]
                :order-by [:msgid]})

The compile step lifts every marker into the params map and hands it to
HoneySQL, so the query is an ordinary Toucan query and the wrapper macro is
gone. One less thing to remember, and a query that forgets the wrapper is no
longer possible to write.

The method is `:around` rather than `:before` because it has to replace the
query it passes on, not only inspect it.

An unresolved marker still throws. It now means the query reached the
database by some route other than the compile step, since anything through
the pipeline gets lifted.

* Pin what an unlifted marker does, and correct why a hostile value is refused

HoneySQL does not recognise `:auto/param`, so a marker handed straight to
`sql/format` compiles as a call to a function named PARAM over its argument.
The database then refuses the statement -- `Function "PARAM" not found` --
so it is broken rather than dangerous, but it is worth pinning, since the
marker only does its job on the path that lifts it.

Also corrects a test comment that credited JDBC with refusing a hostile
non-scalar. Through Toucan it never reaches the driver: `honeysql-guard`
rejects the unmarked map at the compile step first.

* Check every clause that can hold a value, not a list of five

The walk inspected `:where`, `:having`, `:set`, `:values` and `:join-by`.
HoneySQL has about 92 clauses, so that missed real ones -- notably join ON
conditions, which are the most used clause family in the codebase (~650
sites across `:join`, `:left-join` and friends), plus `:do-update-set` and
`:on-duplicate-key-update`.

Worse than the omissions was the direction. Unknown *operators* failed
closed, but an unlisted *clause* was skipped silently, so a clause nobody
had classified went unchecked. The walk now lists the clauses that hold only
structure -- `:select`, `:from`, `:order-by`, the DDL -- and checks
everything else, so a clause added to HoneySQL later is inspected rather
than ignored.

A join clause is a flat sequence alternating target and ON condition, and
both are keyword-headed vectors. The operator table tells them apart: a
condition starts with a known operator, anything else is a table.

Also drops `long*`, `longs` and `str*`. They were `clojure.core/long` and
`str` with docstrings -- an alias layer with no added behaviour, and nothing
used them once the marker existed. `[:auto/param]` is what the walk can
actually see; a coercion returns a plain scalar it cannot distinguish from
an uncoerced one. GHY-4475 wants amending to match.

* Load the check from app-db core, and move test scaffolding out of src

Removing the `keep-me` no-op exposed a real gap: nothing loaded this
namespace, so neither the marker lift nor the check ran in a running system.
It was masked while the adopting namespace still required it. `app-db.core`
now requires it the way it requires `honeysql-guard`, and `keep-me` is a
reference inside `comment` rather than a defn, matching the convention.

The test-only query that proved the check is scoped to its caller no longer
lives in `src`. The test namespace adds itself to the enforcing set instead,
which works now that the munged class prefixes are derived on each call
rather than captured once at load; two copies of that set could drift.

Adds a test that an unmarked string is compared as a literal. A string was
never the danger -- HoneySQL binds it -- and saying so keeps the marker's
purpose clear: it is for values that are not strings, which HoneySQL would
otherwise compile as structure.

Records that the marker belongs in a query map rather than a kv-arg. Toucan
builds kv-args into the where clause after the lift runs, so a marker passed
that way ends up inside a PARAM call instead of a comparison.

* Reduce to the value binding, and drop the walk

Review found the walk failing open in every position it did not explicitly
classify -- a map reaching `check-clause!` returned nil rather than throwing,
which left `{:where [:and [:= :a 1] {:raw "1=1 OR true"}]}` compiling to
`WHERE (a = ?) AND (1=1 OR true)`. The left-hand side of every comparison,
arguments past the declared indexes, `:join-by` conditions, and
`:limit`/`:offset`/`:fetch` were all unchecked for the same reason. A check
that advertises a guarantee it does not provide is worse than no check, so
the walk is gone rather than patched. It can come back built fail-closed.

What remains is the part that worked: a `[:auto/param v]` marker is lifted
at the compile step into HoneySQL's `[:param :kN]` plus its params map, so a
value written inline binds as `?`.

Four bugs in that half are fixed. `postwalk` descended into a marker's own
payload, so a nested marker bound a `[:param]` vector instead of the value;
it now uses `prewalk` and takes the payload whole. Toucan re-enters `compile`
with the `[sql & args]` vector it produced, so the method ran twice per
query; it now only walks a map. Param keys were sequential and guessable,
letting a `[:param :p1]` from request data bind another slot's value; keys
are now random. The options binding replaced `:params` wholesale, dropping an
enclosing compile's params; it merges.

`honeysql_guard.clj` is reverted. `:params` is never a key in a query map --
it reaches HoneySQL as a format option -- so that change was dead code in a
file backported to v58, v61 and v63, and the comment claiming `[:param]`
needed it was wrong.

* Reject a malformed marker, and do not treat a map entry as one

A marker-headed vector of the wrong arity now throws. HoneySQL does not
recognise the leftover form and compiles it into a call to a function named
PARAM, so `[:auto/param v extra]` previously failed at the database with an
error pointing nowhere near the typo.

A `MapEntry` is a two-element vector whose first element is its key, so the
map `{:auto/param v}` would otherwise look like a marker and the map holding
it would be rewritten away. That map means "a column named :auto/param", so
reaching it takes misuse rather than ordinary use -- but excluding the type
costs one check, so the walk does it rather than leaving the shape to chance.

Testing `vector?` instead of `sequential?` does not help here, since
`MapEntry` implements `IPersistentVector`.

This also drops support for a marker written as a seq. Nothing writes one,
and the looser test was widened for surface area rather than a caller.

Adds tests for the write path and for a query with no markers, neither of
which the unit tests reached: they exercise `auto-param` directly, so the
pipeline method's own branches were untested.

* Accept one marker shape, and never let one reach SQL

`[:auto/param value]` is the one supported form. Anything else headed by the
marker -- a seq, the wrong arity -- now throws instead of being left to
compile, because HoneySQL does not recognise the keyword and turns whatever
is left into a call to a function named PARAM that fails at the database
with nothing pointing back to the query.

Two assertions hold the line. Nothing marker-shaped may survive the rewrite,
and every value lifted out has to come back as a bound argument: a marker in
a slot HoneySQL formats as an identifier is rewritten but never consumed, so
`{:from [[:auto/param :t]]}` silently produced `FROM param AS p3lg...`,
dropping the value and putting the generated key in the SQL text. That now
throws.

A marked nil is left to HoneySQL rather than bound. It turns a literal nil in
a comparison into `IS NULL`, while a bound parameter gets `= ?`, which no row
satisfies -- so wrapping a nullable value silently returned nothing.

A `MapEntry` is still excluded from the marker test, so a column named
`:auto/param` is left alone rather than rewritten away.

* Look for the marked value among the arguments, not for a count

`assert-every-value-bound!` compared how many values were lifted to how many
arguments came back, which any arguments satisfy -- not specifically the ones
that were lifted. A query carrying unmarked literals has arguments to spare,
so a dropped marker hid behind them:

    {:from [[:auto/param :t]] :where [:and [:= :a 1] [:= :b 2]]}
    ;; => FROM param AS p16vl... WHERE (a = ?) AND (b = ?)   -- two args, one
    ;;    param, count satisfied, value silently gone

It now looks for each marked value among the arguments. A sequential value is
spread across one argument per element, so its elements are looked for rather
than the collection; anything else, including the map this exists to
neutralize, binds whole.

Tests cover the shapes where the number of arguments and the number of marked
values disagree: a collection, the same value marked twice, and marked values
sitting alongside unmarked ones.

* Drop the bound-value assertion, and say what is not checked

`assert-every-value-bound!` rejected ordinary queries. It inferred from a
value's Clojure type whether HoneySQL would spread it across several
arguments, but that is a decision HoneySQL makes per operator: `:in` spreads
a collection, `=` binds one whole. So a vector compared with `=`, and a set
in an `:in`, were each checked the wrong way and threw. A collection holding
`nil` or `false` threw as well, because the check used a set as a predicate
and a set returns nil for those members even when it contains them.

It was also unsound in the direction it existed for. Comparing against the
argument list loses which argument came from which value, so a dropped
marker passed whenever an unrelated literal elsewhere happened to equal it:
`{:from [[:auto/param "de"]] :where [:= :msgid "de"]}` was accepted, with the
value gone and the generated key left in the SQL as an identifier.

No post-hoc check can be sound here -- HoneySQL gives no signal for a param
it did not consume -- so the namespace docstring now states the limitation
instead of a check pretending to cover it. A marker still may not survive the
rewrite, and a malformed one still throws; both of those are about the marker
itself rather than about what HoneySQL did with it.

* Lint a converted namespace's values for a missing marker

Marking a value does nothing to keep it marked: a later edit adding an
unmarked value slot to a converted namespace passes silently. This adds the
author-time check that holds the line, extending the hook that already
enforces `db.clj` confinement rather than adding a second one -- the two are
halves of the same rule, and that hook already knows which namespaces are
`db.clj`.

It reports a symbol reaching a value slot without a marker, following the
clause shapes a query map uses. Only a symbol: a literal cannot carry a
request value, and a value built inside the function is out of reach of a
check that does not follow it across a call. Only a `db.clj`, which is what
makes it decidable -- the values there are named in the parameter vector a
few lines up.

Off by default, enabled per namespace. Run globally it reports 269 sites, of
which 196 are ids that Malli already constrains to an integer and that
therefore cannot become SQL. Warning about those would bury the 73 that
matter -- strings and enums like `perm-type`, `setting-key`, `namespace`. A
namespace is added to `:config-in-ns` once its own values are marked, so the
list grows with the migration rather than ahead of it.

`metabase.content-translation.db` is the first entry. Removing its marker now
fails the lint.

* Bind a marked kv-arg, which is how queries are actually written

`(t2/select :model/X :locale [:auto/param v])` did not work, and kv-args are
how nearly every query in the codebase is written: 1125 call sites across
`*/db.clj` against 3 that pass a query map. So the marker covered the style
almost nobody uses.

Toucan builds a kv-arg into the where clause before the compile step sees it,
taking the marker keyword for the operator and folding the column in, so
`:locale [:auto/param v]` arrives as `[:auto/param :locale v]`. That arity
was previously rejected as malformed. It is now recognised and rewritten to
`[:= :locale [:param k]]`, which is the comparison Toucan would have built.

A marked kv-arg holding nil still compares as `IS NULL` rather than binding,
matching an unmarked one.

The lint reads the same style now. With it enabled across `src` it reports
1006 sites rather than the 269 it saw when it only walked query maps, of
which 769 are id-shaped and 237 are not -- close to the 78% the project's
estimate predicted, and the 237 are the ones that need a marker rather than
a coercion.

* Pair kv-args from the model, and do not lint written values

Several query fns take an argument before the model -- (t2/select-one-fn
:value :model/Setting :key k) -- so pairing the kv-args from a fixed offset
misaligned them and the values went unreported. They are now paired from the
model keyword, which is where the pairs actually start.

Insert calls are no longer linted for unmarked values. A value written into a
row is not a where-clause value an attacker can turn into SQL structure, and
a define-before-insert hook may read it before the query compiles, so a
marker there breaks the write rather than protecting anything.

* Reject a marked operator form, and check every argument of one

A kv-arg value may itself be an operator -- `:locale [:in [...]]` -- and
Toucan folds that into `[:auto/param :locale [:in [...]]]`, which the kv
rewrite turned into `[:= :locale [:param k]]`. That binds the operator form
as a single value and changes the comparison, so `IN (?, ?)` became `= ?`
against a list. It now throws and points at the working form, which is the
marker inside the operator rather than around it: `[:in [:auto/param vs]]`.

The linter examined an operator's arguments only when there were exactly
two, so `[:between :col lo hi]` and any other arity went unchecked -- a false
negative, which is the expensive direction for this lint. It now takes every
argument after the column, and `between` joins the operator set.

`read-call?` no longer assumes the call head is Named. Kondo always passes a
resolved var so this was unreachable, but the failure would have been a
linter crash rather than one bad file.

Also drops a duplicated paragraph that described the key generation as
gensymed when it uses ThreadLocalRandom, and replaces the ns docstring's
description of a misplaced marker with the statement it actually produces.

* Lint the query shape as well as its values, under one finding

A query written as `:column value` pairs is now reported alongside a value
that reaches a slot unmarked, under a single `:metabase/unsafe-app-db-query`
finding. A namespace opts into both at once, and the pair can be flipped on
globally when every `db.clj` has been converted.

Toucan builds kv-args into the where clause itself, so the query's shape is
not on the page -- there is nothing for a reader, this linter, or the `.sql`
extraction that follows this project to read. Every call has a map form:
`{:where [...]}` for a select, and for `update!` a conditions map, since its
spec takes `conditions-map-or-query?` rather than only pairs.

Checking the shape rather than the pairs' values also removes the reason the
marker had to understand a kv-arg at all. The value check now walks query
maps only.
2026-09-16 03:27:26 +00:00
Alexander Polyankin
48d493c293 Fix the code review findings of the closed argument schemas PR (#82537) 2026-09-15 22:44:21 -04:00
Andrei Chernykh
01f2c4b624 Redesign the LDAP settings page and its group mapping card (#82389)
* Redesign the LDAP settings page and its group mapping card

* Keep the mapping editor fields aligned when a name is rejected

* Merge the LDAP group mapping e2e flows into one

* Address the LDAP page review feedback

* Drop comments that restate the code

* Lock the LDAP attributes card and group the locked cards
2026-09-15 19:37:24 -07:00
Romeo Van Snick
86a761845a Read a card's UI parameters off the question (#82466) 2026-09-16 12:03:10 +10:00
github-automation-metabase
c19c1bfdeb Auto-update documentation for master (#82532)
Update generated docs (master)

Co-authored-by: github-actions <github-actions@github.com>
2026-09-16 02:02:55 +00:00
bryan
47644f8e40 Raise describe timeout in handle-circular-refs-test (DEV-2333) (#82540)
`handle-circular-refs-test` has been failing intermittently since #82447.

The last assertion wraps `(umd/describe :metabase.lib.schema/query)` in a
500ms `with-timeout`. That timeout exists as a smoke test against runaway ref
expansion -- before `max-depth` was added to `metabase.util.malli.describe`,
describing this schema took literal minutes. It was never meant to be a
performance budget.

#82447 required closed map schemas in `mu/defn`, which enlarged the schemas
`describe` walks to its `max-depth` of 3. Since expansion cost is roughly
exponential in branching factor, this made the call ~4.5x slower. Measured on
one machine, isolated runs:

    before #82447: 89ms cold, 33-50ms warm
    after  #82447: 232ms cold, 158-193ms warm

Headroom against the 500ms budget dropped from ~14x to ~2.8x. The test is
`^:parallel` and `with-timeout` runs its body in a `future`, so it competes for
the thread pool with every other parallel test; under CI contention ~175ms
crosses 500ms. That is why it presents as flaky rather than consistently red,
and why it showed up on the Postgres 14 and Java 25 suites -- those runners are
slower/more contended, not semantically different.

The describe output itself is unchanged (identical 187-char string), so only
speed regressed, not correctness.

Raising the timeout to 5000ms keeps the runaway-recursion guard meaningful (that
failure mode is minutes, not milliseconds) while surviving pool contention.

Not treating the 4.5x as a perf bug to chase: `umd/describe`'s only production
caller is the error-explanation path in `metabase.api.macros`, reached only
after validation has already failed, and it describes the nested sub-schema
rather than the full query schema. Validation itself is unaffected by #82447
(sub-microsecond before and after).
2026-09-16 01:55:21 +00:00
Romeo Van Snick
980c6b0018 Measure each commit in German and in English (#82536)
Claude-Session: https://claude.ai/code/session_01171Lkmug15JzrcN2aMBLMk

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 11:18:22 +10:00
Fraser
e677b6b267 Use affected-test plans to narrow Loki visual tests (#82050) 2026-09-16 12:37:30 +12:00
Romeo Van Snick
548573f22d Move the public question and the expression editor off the Metadata object (#82313)
* Build the public question from the store

* Pass the expression editor its database, not the Metadata object

* Build the FieldValuesWidget test metadata with createMockMetadata

* Name the endpoints the public question's parameters come from
2026-09-16 08:56:28 +10:00
Aleksandr Lesnenko
00232ddc3a drop disable mcp tools setting (#82533) 2026-09-15 22:00:53 +00:00
bryan
f6caace66a Refuse to save a query handle that carries bound parameter values (#82434)
Refuse to save a query handle carrying bound parameters

A handle minted by execute_sql re-attaches the :parameters it ran with, but
:parameters is runtime-only and serialize-query strips it on the way into
dataset_query. Saving such a handle persisted the query without its filter and
without complaining, so the card returned rows the agent's own run excluded.

Fail closed in resolve-query-handle-for-save!, the chokepoint shared by
question_write, metric_write and transform_write, with a teaching error naming
the concrete alternative.

Closes GHY-4547
2026-09-15 14:51:30 -06:00
technomancy
ba327647bc Update oldest MySQL version to 8.4. (#82507)
According to https://endoflife.date/mysql version 8.0 went EOL in
April. Now 8.4 is the oldest supported release.
2026-09-15 23:34:51 +03:00
github-automation-metabase
dfbff65b08 Update translations 2026-09-15 (#82474)
* Update translation sources 2026-09-15

* Update translations

---------

Co-authored-by: github-actions <github-actions@github.com>
2026-09-15 20:26:58 +00:00
github-automation-metabase
f787cc5b89 Tighten ratchets (#82523)
Co-authored-by: github-actions <github-actions@github.com>
2026-09-15 20:02:24 +00:00
bryan
1b57292d09 Send the SAML SessionIndex in single logout requests (#82441)
* Send the SAML SessionIndex in single logout requests

Metabase's LogoutRequest identified the session to end by NameID alone, which
is the user's email. Auth0 issues a distinct user ID per connection, so one
email can own several sessions and Auth0 cannot tell which to end:

    client_error: No active session(s) found matching LogoutRequest

SAML Profiles 4.4.4.1 makes SessionIndex the element that scopes a
LogoutRequest to a session. Record the SessionIndex the IdP sends in its login
assertion on the session it creates, and send it back at logout.

SessionIndex is per session, not per user, so it lives on core_session rather
than auth_identity, which has a unique constraint on (user_id, provider) and
would let concurrent logins overwrite each other.

A hardcoded SessionIndex was tried once and broke Azure AD (#41600). Only an
index the IdP itself issued is ever sent; sessions without one (every session
predating this column, and every non-SAML session) produce exactly the request
they produce today. The existing saml-logout test, which asserts on the full
signed logout URL for a session with no SessionIndex, passes unchanged.

Requires saml20-clj 4.3.0, which parses SessionIndex out of login assertions
and can emit it in a LogoutRequest.

Fixes #77837

* Send the IdP's NameID, not the user's email, in logout requests

An IdP matches a LogoutRequest on NameID plus SessionIndex. We sent the user's
email as the NameID, but the IdP's NameID is not necessarily an email: Auth0
sends an opaque `auth0|<id>`. A LogoutRequest naming an identifier the IdP never
issued matches no session, so Auth0 answered:

    No active session(s) found matching LogoutRequest

even once we started sending the correct SessionIndex.

Record the NameID and its Format from the login assertion alongside the
SessionIndex, and send those at logout. Sessions predating these columns have a
NULL NameID and fall back to email, so logout keeps working across an upgrade.

Verified end to end against a live Auth0 tenant: the LogoutRequest now carries
NameID `auth0|<id>` with Format `unspecified` and the assertion's SessionIndex,
Auth0 returns StatusCode Success, and the session row is deleted.
2026-09-15 13:56:42 -06:00
Chris Vandoodewaard
6fdc4439da record E2E statistics (#82496)
* add script to upload e2e test timings

* update workflow, add tests

* fix team name
2026-09-15 14:27:09 -04:00
Chris Vandoodewaard
ad51ac8b78 only run some semantic search tests on PRs (#82440)
* only run some semantic search tests on PRs

* add label to force run semantic search tests

* fix whitespace

* trigger ci
2026-09-15 14:26:17 -04:00
Michael Eby
94aea9ae24 docs: 2FA v2 (#81962)
* docs: 2FA v2

* verify admin path and require options

* edit from jeff
2026-09-15 12:46:43 -04:00
Alexander Polyankin
e79f40e3d3 Require typed, closed argument schemas in mu/defn (#82447) 2026-09-15 16:45:35 +00:00
Nick Fitzpatrick
9ce408d114 Uxw 4979 manage unsynced content modal (#81226)
* Manage Unsynced Content modal

* add actions

* Show dependies when attempting to remote sync

* adjust spacing of dependency rows

* handling snippet and dashboard dependencies

* reshaping API response for FE consumption

* save from modal again

* cleanup

* PR Feedback, Design Feedback

* handle instance analytics case

* fixing function string

* Mike's feedback

* fix getIcon hook
2026-09-15 13:16:19 -03:00
Stas Gavrylov
6f423c147d GDGT-3199: Stabilize document comment tests (#82480)
* GDGT-3199: Stabilize document comment tests

* GDGT-3199: Assert editor focus before typing
2026-09-15 18:17:56 +03:00
Stas Gavrylov
7b271d25b6 GDGT-3108: Stabilize document revision history test (#82484)
* GDGT-3108: Stabilize document revision history test

* GDGT-3108: Apply review feedback

* GDGT-3108: Target final toast assertion by text
2026-09-15 18:17:46 +03:00
Will Fry
75398ad620 GDGT-3131: refactor ai usage stats page to use nested routes (#82420) 2026-09-15 15:48:28 +01:00
Stas Gavrylov
34c44f0f7f GDGT-3138: Make inline update rollback deterministic (#82418)
* GDGT-3138: Make inline update rollback deterministic

* GDGT-3138: Clarify reinitialization behavior

* GDGT-3138: Wait for incremental switch readiness

* GDGT-3138: Cleanup
2026-09-15 16:44:05 +03:00
Chris Truter
968b105665 Never send the model an empty user message from a Slack thread (#82443)
A file shared with no text, or a message that only mentions the bot, reached the thread history as an empty user
message, and Anthropic rejects the whole request for it. Name the attachments instead, or leave the message out.
2026-09-15 15:30:26 +02:00
Chris Truter
aecbc3c3cb Warm the Kondo cache so cache-reading hooks run in CI (#82388)
* Warm the Kondo cache so cache-reading hooks run in CI

`hooks/ns-analysis` reads only `.clj-kondo/.cache`, and Kondo writes that cache after every file in a run has
been analysed. `mage kondo` deleted the cache and linted in one pass, so a hook that asks about another
namespace's vars found nothing and skipped its check. `:metabase/prefer-with-dynamic-fn-redefs` therefore never
fired in CI while warning locally against a warm cache.

Lint once and discard the findings before the real pass. `kondo-redundant-ignores` and `kondo-insert-ignores`
get the same warm-up: on a cold cache the former reports needed suppressions as redundant.

Turning the linter on exposes 327 flagged `with-redefs` forms. 316 become `mt/with-dynamic-fn-redefs`. Nine keep
`with-redefs` where the redefined function runs on a thread a dynamic binding does not reach, or on a value the
proxy cannot wrap. Two `.cljc` sites are covered by a namespace entry in config.edn instead, since the macro is
JVM only.

Delegating replacements capture the original with `mt/original-fn`. A bare symbol or `@#'the-var` resolves to
the proxy once a var has been patched, which recurses on the second test in a JVM.

Inline ignore budget 28 to 37, config suppressions 3 to 4.

* Stop counting prefer-with-dynamic-fn-redefs ignores

Genuine cases are common: Quartz workers, pool callbacks, queue delivery workers and values the proxy cannot
wrap all need a root swap. A numeric ceiling adds churn to every PR that meets one. Each ignore still needs a
comment explaining why.

* Harden the Kondo warm-up and keep a primitive fn on with-redefs

`next-page-size` in the BigQuery driver takes and returns primitive longs, so compiled callers invoke it
through `IFn$LLLLL`. The dynamic-redefs proxy is a plain variadic fn, so the conversion threw
ClassCastException, and since patching is permanent every later BigQuery query in the JVM would too.

The warm pass now also lints `dev/src`, which defines vars that `dev/test` redefines. That exposes six sites
in the kondo ratchet tests, now converted. A warm pass that exits with anything but 0, 2 or 3 fails the
task rather than leaving a partial cache. `kondo-updated` warms when no cache exists, and the ratchet tools
warm the roots they lint.

* Only skip kondo-updated's warm pass after a finished one

A cache directory can be partial, left by an editor or an interrupted run, and `kondo-updated` then skipped
warming and missed cache-dependent findings. `warm-cache!` now writes a marker when it finishes, deleting it
first so an interrupted pass can't leave an old one, and `kondo-updated` checks for the marker instead.

* Only let a full warm pass mark the Kondo cache warm

A warm over a narrower set of roots leaves the cache incomplete, so it must not write the marker
`kondo-updated` trusts, nor clear one that a full pass wrote.

* Point the Athena TODO at the macro the test now uses

* Say why kondo-updated accepts a stale cache
2026-09-15 15:30:02 +02:00
github-automation-metabase
5c18aea2c7 Auto-update documentation for master (#82444)
Update generated docs (master)

Co-authored-by: github-actions <github-actions@github.com>
2026-09-15 09:14:04 -04:00
github-automation-metabase
18a6ad127c Widen snowplow-micro poll budget to de-flake event assertions (#77383)
Snowplow events are POSTed asynchronously and then ingested by
snowplow-micro before they can be read back via `micro/good`. The
1000ms retry budget in the Cypress helper was too tight for the event
to land on loaded CI runners or under network throttling, producing
spurious "Snowplow retry timeout" failures. Bump the budget to 5000ms;
this only affects how long a failing assertion waits, since matching
events still return early.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-09-15 14:30:40 +02:00
Fraser
afd98b94a5 Repoint embedding-sdk's ClickObject import to metabase-lib (#82455) 2026-09-15 23:22:26 +12:00
Phoomparin Mano
9a5524063b Infer semantic schema export scope from the data app prompt (#82221)
add scope guidance for agents
2026-09-15 17:01:39 +07:00
Romeo Van Snick
d46d7a47e4 Read the reference lists from their own responses (#82473)
* Drop the unused metadataFields prop from the reference detail pages

* List the reference databases from the fetched response

* Fetch the reference segment list where it is rendered
2026-09-15 19:54:14 +10:00
Romeo Van Snick
8d67bf7e3d Keep the segment retire action closed until its table loads (#82471) 2026-09-15 19:54:00 +10:00
Will Fry
4084c9254b GDGT-3219: fix broken navigation from unsaved sql (#82433) 2026-09-15 08:30:18 +01:00
Romeo Van Snick
5f8f065921 Read the embedding data picker's entities from the store (#82407) 2026-09-15 15:49:11 +10:00
Romeo Van Snick
7ff180c80a Read the table and database from their own requests, not the entities mirror (#82472) 2026-09-15 05:46:55 +00:00
Romeo Van Snick
6964b63d2b Record when the page draws on the second and later visits (#82467)
* Record when the page draws on the second and later visits

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01171Lkmug15JzrcN2aMBLMk

* Wrap the two long readings for the formatter

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01171Lkmug15JzrcN2aMBLMk

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-15 14:44:59 +10:00
Romeo Van Snick
98486ccd56 Restore the changes #82405 reverted (#82463) 2026-09-15 03:33:50 +00:00
Fraser
7e9e46602f Give types, settings-components and embedding-ee shared-tier levels (#82462) 2026-09-15 14:50:15 +12:00
Fraser
7ea204ed6e Enforce the shared tiers for api (#82460) 2026-09-15 14:50:04 +12:00
Romeo Van Snick
7e36117dbc Read the data selector's entities from the store (#82405)
* Add relation selectors to the metadata store

* Read the data selector's entities from the store

* Take the data selector's entities as plain objects in its consumers

* Format the data selector spec

* Inline the table schema lookup

* Name the data selector's lookups prop entityLookups
2026-09-15 11:59:21 +10:00
Nick Fitzpatrick
72ea707064 Add dataPointsVisible extension to eCharts (#81721)
* useRenderedDataVisibility

* static viz, other adjustments

* remove dead hook

* Split panels, simplification

* handling line series with a range very close to chart boundary, ignore goal line

* return true on traversal if intersection is found
2026-09-14 22:44:43 -03:00
Fraser
781e650301 Move security-center endpoints into enterprise security_center (#79351)
* Move security-center endpoints into enterprise security_center

Moves security-center.ts out of the shared metabase/api module into
enterprise/frontend/src/metabase-enterprise/security_center/api.ts. All
of its endpoints hit /api/ee/ URLs and every consumer lives in the EE
security_center module, so it was an OSS api file with zero OSS
consumers.

The file now uses EnterpriseApi.injectEndpoints, matching the existing
in-module EE precedents (monitor/ai-auditing/metabot-analytics/api.ts,
table-editing/api/index.ts). The security-advisory tag type moves from
the OSS TAG_TYPES to ENTERPRISE_TAG_TYPES alongside it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Cast test store state for EnterpriseApi selectors

The listSecurityAdvisories endpoint selector is now typed against the
enterprise tag union, while renderWithProviders builds a store typed
with the OSS State. The runtime state shape is unchanged, so the spec
casts through a helper with a justification comment.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Use relative imports for EnterpriseApi inside the enterprise module

security_center/api.ts was reaching its own module through the
metabase-enterprise alias; same-module imports are relative per the
import-style convention.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Import EnterpriseApi through the metabase-enterprise alias

../api/api reaches the sibling api module, not security_center's own
files, so the same-module-imports-are-relative convention does not
apply here. The alias form matches the other in-module EE endpoint
owners (metabot-analytics, table-editing) and is the specifier the
no-base-api-access rule tracks EnterpriseApi by.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Rewrap the cast comment in the promo card spec

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Verify security-center mutations invalidate the advisory cache

* Simplify security-center tests and flush query updates

* Match the security-center api spec titles to the folder's should idiom

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-09-15 11:37:42 +12:00
Fraser
53f48ed3be Use affected-test plans to narrow frontend unit tests (#82049)
* Use affected-test plans to narrow frontend unit tests

* Pass affected test plans by artifact and reject Jest runner failures

* Extract unit test selection preparation from workflow YAML

* Follow the shared Jest config files in the frontend path filters

* Drop the unused import/no-commonjs directive from the filter spec

* Leave heap retention specs out of the planner's unit list

* Check unit test results in a script and let suite hook failures reach quarantine

* Warn when a narrowed unit test run matches none of Jest's test paths

* Reword the workflow cast comment in the test plan spec
2026-09-15 11:04:56 +12:00
Romeo Van Snick
a0845f5373 Serve the locale catalogues as rspack chunks, remove locale api (#81933)
* Serve the locale catalogues as rspack chunks

The catalogues were fetched from `app/locales/<locale>.json`, a path outside
everything rspack does for an asset. They were unhashed, so `cacheable?` never
granted them far-future headers, and uncompressed, so the server gzipped each
one per request.

Importing them instead puts them through the normal pipeline: rspack hashes each
catalogue, emits it as its own chunk under `app/dist`, writes the `.br` and `.gz`
siblings, and resolves the URL at runtime. A hashed file there is already
cacheable, so no new rule is needed.

Production swaps `devtool` for SourceMapDevToolPlugin, which takes an exclude. A
source map for a catalogue maps JSON onto itself and would add about 29 MB.

* Format the specs the locale change touched

* Keep the catalogues out of the SDK bundle

A context module bundles its whole directory wherever it resolves, so importing
the catalogues added all 36 to the SDK's reachable graph: 66 files to 102, and
its total gzipped size up 275%.

The SDK should not carry them regardless of size. It renders in a customer's
page against whichever instance it is pointed at, so its catalogue has to come
from that instance rather than from whenever the SDK was built.

`locale-catalog` now names how a catalogue is loaded, the way `ee-plugins` and
`sdk-ee-plugins` name their implementations. The app imports from its own
bundle; the SDK bundle config aliases the fetch instead, which also keeps the
context module out of it.

* Let the SDK cache the catalogues it fetches

A catalogue served from `app/locales/<locale>.json` carries no hash, so
`cacheable?` never matched it and the SDK re-downloaded one on every load.

The SDK does not need the app's chunk hash to fix that, and it cannot have it:
the two bundles are separate compilations with their own chunk ids, filename
templates and runtime wrappers, so the same catalogue hashes differently in
each. What it needs is any token that changes when the catalogues do, and it
already has one. The instance serves the SDK bundle, so whatever build produced
it produced the catalogues too, and `EnvironmentPlugin` already compiles that
build's version and commit into it.

The SDK now names its build in the query, and a catalogue request that carries
one is cacheable. Without one it is not, so a caller that omits it cannot pin a
stale catalogue.

* Move the locale loading out of metabase/api

Nothing here calls an API endpoint. The catalogue is a static asset, loaded by
import in the app and by a plain fetch in the SDK, so it sits alongside
`utils/i18n`, which applies what these return.

* Swap the locale loader by path, not by alias

The loader was reached through a `locale-catalog` alias, which had to be
declared in the rspack aliases, tsconfig and jest config before the SDK bundle
could override it: four places to say one thing.

The importer now uses a plain relative import, which every resolver already
understands, and the SDK config keys its override on the resolved path. One
config knows a swap happens, and it is the one doing the swapping.

* Let the SDK load catalogues as chunks too

The SDK fetched catalogues from the instance over HTTP because a context
import pulled all 36 into its bundle. They were never in its initial
payload: an import() context always emits async chunks. What tripped CI
was the gate itself, which is `embedding-sdk-chunked/total`, the reachable
set. All 36 counted, 8.5 MB gzip against a 3.1 MB bundle, so 275%.

The SDK can use the same loader. `sdk-public-path.ts` points
`__webpack_public_path__` at the instance, and the package injects the
bundle as a script tag, so a host app's bundler never sees the import().
Chunks land under `app/embedding-sdk/chunks/`, which the backend already
serves with far-future immutable headers, so the build token and its
`cacheable?` clause are not needed either.

Measure the catalogues out of the reachable totals instead. One loads per
session and none for English, so counting all 36 overstates by 35x, and
their size tracks the translation process rather than any code change.

Drops the loader, its spec, the build-time alias and the backend clause.
The SDK's chunkFilename takes `[name]` so the gate can recognise them.

* Fold the loader into its only caller

Nothing overrides `loadLocaleCatalog` through the build any more, so it does
not need to be its own module.

Drop the SourceMapDevToolPlugin too. It existed to keep source maps off the
locale chunks, which only cost disk in a dev build, and it made `devtool`
conditional to work around the two not being able to run together.

Clear the comments that explained the change rather than the code: the specs
said what the catalogue used to do, and two references pointed at the file
this commit removes.

* Point the locale test hooks at the chunk

The catalogue arrives as a hashed rspack chunk, so the specs that stubbed or
waited on `GET /app/locales/<locale>.json` were watching a request that no
longer happens.

Five e2e specs used the intercept only as a synchronisation point, so they
now wait on the chunk instead. `content-translations` stubbed an empty
catalogue to keep the UI in English while it checks that content is
translated, and a chunk cannot carry a JSON body, so it blocks the chunk. Its
Arabic block asserts English chrome and `AR-` content only, nothing that
needs the catalogue applied.

That leans on LocaleProvider catching a failed load, so cover it: a rejected
catalogue still renders the children.

Also fix the import order the module move disturbed in three consumers.

* Spy on loadLocalization instead of wrapping the module

* Drop a tombstone comment

* Keep the locale set when its catalogue fails to load

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NUuaopJ7ZtMVvdwwxF6qyx

* Wait on the language landing, not on the catalogue request

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NUuaopJ7ZtMVvdwwxF6qyx

* Keep the locales directory when the translations step clears it

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NUuaopJ7ZtMVvdwwxF6qyx

* Assert a real Dutch string after the first language switch

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NUuaopJ7ZtMVvdwwxF6qyx

* Re-render the children once the catalogue is applied

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NUuaopJ7ZtMVvdwwxF6qyx

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-15 06:43:01 +10:00
Fraser
0fd3651033 Move the token refresh hooks into settings (#82402) 2026-09-15 08:22:28 +12:00
Chris Truter
a632ed1799 Interactive module graph viewers (#82375)
* Consolidate the module viewers into one explorer

`./bin/mage modules-tree --html` writes a single page with three views
of the module config: the tree, a coupling ranking, and a dependency
graph around focused modules. `dev.module-explorer/open!` builds the
same page in a dev REPL with namespace-level edges, so the graph can
show which API namespaces each consumer uses.

The graph replaces dev.module-viz and its Jetty server. Source paths,
stats and teams go through the shared resolver, so nested and renamed
modules report correctly. module-team moves into hooks.common.modules.

* Label module counts as used by and uses

* Refine module explorer semantics and state
2026-09-14 20:18:12 +00:00
Fraser
be3443b07d Stop querying from importing metabot (#82401)
* Move MetabotPromptButton into metabot behind a querying plugin slot

* Stop querying reading metabot permissions and sidebar state

* Test the data reference auto-open prop instead of metabot state

* Pass metabot's editor and error UI in as props instead of a plugin slot

* Fix editor specs for the metabot permissions hook and the barrel mock
2026-09-15 08:09:17 +12:00
Andrei Simionescu
02f5855a35 Don't hand raw driver errors to the model when a Slack upload fails (#81202)
Only the upload layer's own 4xx errors reach the model, and everything else gets a generic line. CSV validation errors had no status code, so they throw as 422s now and the upload endpoint answers 422 instead of 500 for a malformed file.
2026-09-14 15:52:09 -04:00
Ben Grabow
30346d6a64 Break requiring resolve cycles (#82011)
* Break requiring-resolve cycles

* Place moved code where its consumers can already reach it

Avoids widening module :api sets for three of the extracted namespaces.
dashboard->resolved-params moves to metabase.parameters.params.
template-tag-parameters and parameter-template-tag? move to metabase.query-processor.card.
Quartz's ConnectionProvider moves to metabase.app-db.quartz; task.bootstrap keeps the connection interceptor.
metabase.app-db.connection, metabase.dashboards.params and metabase.queries.template-tags leave :api.

* Restore comments lost when code moved

The c3p0 deadlock explanation goes back above the `locking` call in prometheus.clj.
The driver-implementation caveats go back on the requires that replaced their requiring-resolve calls.
The PulseChannel-hooks note and the placement rationales for the moved parameter functions come back too.

* Point mcp v2 projections at the moved dashboard->resolved-params

Master's mcp v2 merge landed a caller while the function was moving to metabase.parameters.params.

---------

Co-authored-by: Chris Truter <chris@metabase.com>
2026-09-14 15:46:55 -04:00
Andrei Simionescu
c74356b123 GHY-4070: Update source cards before dependents when a database changes (#82378)
A card's `database_id` is copied from its source card's row, so a dependent rewritten before its
source kept the old database id. The cascade now rewrites cards in dependency order, sources first.
2026-09-14 19:31:49 +00:00
Chris Vandoodewaard
9344c2bd56 Move PR env creation out of run-tests and remove unnecessary workflows (#81814)
* merge containerize-uberjar into uberjar workflow

* big refactor

* move things around, add inputs to fetch-artifact action

* update conditional logic

* clean up comments

cleanup whitespace

* simplify conditional logic

* guard conditional matrix

* review cleanup

* add job names

* cleanup whitespace

* update get-docker-info to resolve repo/tag inputs

* trim trailing whitespace

* add ecr auth if required

* add repo/tag inputs to containerize from uberjar

* use uberjar workflow to push container to ecr

* cleanup pr-env

* add proper label handling to pr-env

* clean up support for ecr

* simplify containerize

* move jar healthcheck into build step

* move containerize back out of uberjar

* review fixes

* a few more fixes

* code review fixes

* cleanup whitespace

* rename workflow to match scheme

* add verification on release, use DOCKERHUB_OWNER var

* fix conditional

* remove whitespace

* end with newline

* add extra guard for labels not pr env related
2026-09-14 14:36:49 -04:00
Aleksandr Lesnenko
a6ce02d1c6 nudge for mbql, fix pagination (#82391) 2026-09-14 13:51:51 -04:00
3349 changed files with 179761 additions and 113913 deletions

View File

@@ -74,12 +74,17 @@
:metabase/prefer-with-dynamic-fn-redefs {:level :warning}
:metabase/t2-query-namespace {:level :warning}
:metabase/table-or-field-query {:level :warning}
;; App-DB query hygiene: a value reaching a value slot unmarked, and a query written as
;; `:column value` pairs rather than a map. Off by default and enabled per namespace under
;; `:config-in-ns`; flip it on globally once every `db.clj` has been converted.
:metabase/unsafe-app-db-query {:level :off}
:metabase/toucan-model-ns {:level :warning}
:metabase/require-shape-checker {:level :warning}
:metabase/tests-must-live-in-known-modules {:level :warning}
:metabase/validate-defendpoint-route-uses-kebab-case {:level :warning}
:metabase/validate-defendpoint-query-params-use-kebab-case {:level :warning}
:metabase/validate-logging {:level :warning}
:metabase/validate-mb-app-db-migrations-test {:level :warning}
:metabase/validate-mb-driver-tests {:level :warning}
:metabase/validate-mb-once {:level :warning}
@@ -315,6 +320,12 @@
{metabase-enterprise.entity-retrieval.core/search-unfiltered {:message "Permission bypass risk: library-index hits are unfiltered — resolve each entity and drop what the current user can't read"}
metabase.entity-retrieval.core/search-unfiltered {:message "Permission bypass risk: library-index hits are unfiltered — resolve each entity and drop what the current user can't read"}
metabase.entity-retrieval.mirror/search-unfiltered {:message "Permission bypass risk: library-index hits are unfiltered — resolve each entity and drop what the current user can't read"}
metabase-enterprise.transform-testing.expectations.equals/->EqualsRows {:message "Build an expectation with metabase-enterprise.transform-testing.expectations.protocol/build, over a value the schema has passed — the generated constructors skip validation"}
metabase-enterprise.transform-testing.expectations.equals/map->EqualsRows {:message "Build an expectation with metabase-enterprise.transform-testing.expectations.protocol/build, over a value the schema has passed — the generated constructors skip validation"}
metabase-enterprise.transform-testing.expectations.equals/->EqualsSql {:message "Build an expectation with metabase-enterprise.transform-testing.expectations.protocol/build, over a value the schema has passed — the generated constructors skip validation"}
metabase-enterprise.transform-testing.expectations.equals/map->EqualsSql {:message "Build an expectation with metabase-enterprise.transform-testing.expectations.protocol/build, over a value the schema has passed — the generated constructors skip validation"}
metabase-enterprise.transform-testing.expectations.empty/->Empty {:message "Build an expectation with metabase-enterprise.transform-testing.expectations.protocol/build, over a value the schema has passed — the generated constructors skip validation"}
metabase-enterprise.transform-testing.expectations.empty/map->Empty {:message "Build an expectation with metabase-enterprise.transform-testing.expectations.protocol/build, over a value the schema has passed — the generated constructors skip validation"}
clojure.core/print {:message "Use metabase.util.log instead of clojure.core/print"}
clojure.core/println {:message "Use metabase.util.log instead of clojure.core/println"}
clojure.core/printf {:message "Use metabase.util.log instead of clojure.core/printf"}
@@ -591,6 +602,7 @@
clojure.core/defprotocol hooks.clojure.core.def/lint-def
clojure.core/defrecord hooks.clojure.core.def/lint-def
clojure.core/deftype hooks.clojure.core.def/lint-def
clojure.core/ex-info hooks.metabase.agent-message/lint-ex-info
clojure.core/ns hooks.clojure.core.ns/lint-ns
clojure.core/require hooks.clojure.core.require/lint-require
clojure.core/requiring-resolve hooks.clojure.core.require/lint-requiring-resolve
@@ -606,14 +618,13 @@
metabase-enterprise.cache.config-test/with-temp-persist-models hooks.common/with-seven-bindings
metabase-enterprise.dependencies.api-test/with-dependents-test! hooks.common/with-two-bindings
metabase-enterprise.metabot-v3.tools.dependencies-test/with-dependent-transforms! hooks.common/with-two-bindings
metabase-enterprise.metabot.tools.dependencies-test/with-dependent-transforms! hooks.common/with-two-bindings
metabase-enterprise.semantic-search.db.connection/with-migrate-tx hooks.common/let-one-with-optional-value
metabase-enterprise.serialization.api-test/with-serialization-test-data! hooks.common/with-three-bindings
metabase-enterprise.serialization.test-util/with-temp-dpc hooks.toucan2.tools.with-temp/with-temp
metabase-enterprise.transforms-python.drivers-test/with-test-table hooks.common/with-two-bindings
metabase.api.common.internal-test/with-jetty-server hooks.common/with-one-binding
metabase.api.macros/defendpoint hooks.metabase.api.macros/defendpoint
metabase.app-db.schema-migrations-test.impl/test-migrations hooks.metabase.app-db.schema-migrations-test.impl/test-migrations
metabase.app-db.schema-migrations-test.impl/test-migrations hooks.metabase.app-db.schema-migrations-test.impl/test-migrations!
metabase.channel.render.js.pool/with-static-viz-context hooks.common/with-one-top-level-binding
metabase.channel.template.handlebars-helper/defhelper hooks.metabase.channel.template.handlebars-helper/defhelper
metabase.channel.template.handlebars-test/with-temp-template! hooks.common/with-vec-first-binding
@@ -686,6 +697,15 @@
metabase.lib.test-util.macros/mbql-query hooks.metabase.test.data/mbql-query
metabase.lib.test-util.macros/mbql-5-query hooks.metabase.test.data/mbql-query
metabase.lib.test-util.macros/with-testing-against-standard-queries hooks.metabase.lib.test-util.macros/with-testing-against-standard-queries
metabase.mcp.transport/jsonrpc-error hooks.metabase.agent-message/lint-jsonrpc-error
metabase.mcp.v2.common/ellipsize hooks.metabase.agent-message/lint-ellipsize
metabase.mcp.v2.common/error-content hooks.metabase.agent-message/lint-teaching-exit
metabase.mcp.v2.common/list-content hooks.metabase.agent-message/lint-list-content
metabase.mcp.v2.common/success-content hooks.metabase.agent-message/lint-success-content
metabase.mcp.v2.common/throw-teaching-error hooks.metabase.agent-message/lint-teaching-exit
metabase.mcp.v2.dashboard-ops/op-error! hooks.metabase.agent-message/lint-op-error
metabase.mcp.v2.message/msg hooks.metabase.agent-message/lint-msg
metabase.mcp.v2.tools.browse/paged-list-content hooks.metabase.agent-message/lint-list-content
metabase.mcp.v2.tools.parameters-test/with-fixtures hooks.common/with-one-binding
metabase.metrics.api-dimension-test/with-seeded-metric hooks.common/with-one-binding
metabase.model-persistence.api-test/with-persistence-setup! hooks.common/with-one-top-level-binding
@@ -834,7 +854,13 @@
metabase.users.models.user-test/with-groups! macros.metabase.users.models.user-test/with-groups!
metabase.util.namespaces/import-fn macros.metabase.util.namespaces/import-fn
metabase.util.namespaces/import-macro macros.metabase.util.namespaces/import-macro
metabase.xrays.related-test/with-world macros.metabase.xrays.related-test/with-world}}
metabase.xrays.related-test/with-world macros.metabase.xrays.related-test/with-world
potemkin/import-def macros.potemkin.namespaces/import-def
potemkin/import-fn macros.potemkin.namespaces/import-fn
potemkin/import-macro macros.potemkin.namespaces/import-macro
potemkin.namespaces/import-def macros.potemkin.namespaces/import-def
potemkin.namespaces/import-fn macros.potemkin.namespaces/import-fn
potemkin.namespaces/import-macro macros.potemkin.namespaces/import-macro}}
:config-in-comment
{:linters {:unresolved-symbol {:level :off}}}
@@ -898,7 +924,19 @@
;; run Toucan 2 queries anywhere.
;;
{:pattern "^metabase\\.app-db\\..*"
:name app-db-namespaces}]
:name app-db-namespaces}
;;
;; MCP source namespaces, OSS and enterprise, whose prose reaches an LLM agent, excluding test namespaces.
;;
{:pattern "^(?!.*-test(?:\\..*)?$)(?!.*test-util)metabase(?:-enterprise)?\\.mcp\\..*"
:name agent-message-namespaces}
;;
;; MCP source namespaces whose caller-facing errors are MCP exits: `metabase.mcp.transport` and `metabase.mcp.v2.*`,
;; and their enterprise counterparts. Other MCP namespaces' 4xx `ex-info`s are admin API and middleware errors, not
;; MCP exits.
;;
{:pattern "^(?!.*-test(?:\\..*)?$)(?!.*test-util)metabase(?:-enterprise)?\\.mcp\\.(?:transport$|v2\\..*)"
:name agent-exit-namespaces}]
:config-in-ns
{metabase.app-db.custom-migrations
@@ -924,12 +962,25 @@
app-db-namespaces
{:linters {:metabase/t2-query-namespace {:level :off}}}
agent-message-namespaces
{:linters {:metabase/agent-message-lines {:level :warning}}}
agent-exit-namespaces
{:linters {:metabase/agent-message-exit {:level :warning}}}
;; Namespaces whose values are marked. Adding one here is what keeps it that way; see
;; `metabase.app-db.value-guard`.
metabase.content-translation.db
{:linters {:metabase/unsafe-app-db-query {:level :warning}}}
;; .cljc tests run in both Clojure and ClojureScript — `with-dynamic-fn-redefs` is JVM-only,
;; so leave these alone.
metabase.lib.aggregation-test
{:linters {:metabase/prefer-with-dynamic-fn-redefs {:level :off}}}
metabase.util.time-test
{:linters {:metabase/prefer-with-dynamic-fn-redefs {:level :off}}}
metabase.lib.metadata.calculation-test
{:linters {:metabase/prefer-with-dynamic-fn-redefs {:level :off}}}
;; classloader sits below `util` in the module graph, so it can't use `metabase.util.log` without creating
;; a cycle; it gets to use tools.logging directly.

View File

@@ -116,7 +116,10 @@
util
warehouse-schema-overlay
warehouses}
:model-exports #{:model/Action}
:model-exports #{:model/Action
:model/HTTPAction
:model/ImplicitAction
:model/QueryAction}
:model-imports #{:model/Card
:model/Collection
:model/Dashboard
@@ -134,6 +137,7 @@
:uses #{actions
analytics
api
api-scope
api.eid-translation
collections
lib
@@ -262,6 +266,7 @@
events
lib.schema
models
request
settings
util
util.config}
@@ -313,7 +318,9 @@
api-scope
{:team "Metabot"
:api #{metabase.api-scope.core
metabase.api-scope.init}}
metabase.api-scope.data-app
metabase.api-scope.init}
:uses #{util}}
api.eid-translation
{:team "UX West"
@@ -345,11 +352,12 @@
app-db
{:team "UX West"
:api #{metabase.app-db.cluster-lock ; TODO FIXME
:api #{metabase.app-db.cluster-lock ; TODO (Chris 2026-09-16) -- callers should go through app-db.core
metabase.app-db.core
metabase.app-db.init
metabase.app-db.setting
metabase.app-db.setup
metabase.app-db.sql-errors
metabase.app-db.transient-error}
:uses #{auth-provider
connection-pool
@@ -437,8 +445,15 @@
:api #{metabase.batch-processing.core
metabase.batch-processing.init}
:uses #{app-db
parameters
query-processor.cache-backend
settings
util}}
util}
:model-imports #{:model/Card
:model/Collection
:model/Dashboard
:model/Document
:model/Table}}
bookmarks
{:team "UX West"
@@ -505,6 +520,7 @@
metabase.channel.init
metabase.channel.models.channel
metabase.channel.render.core
metabase.channel.schema
metabase.channel.settings
metabase.channel.slack
metabase.channel.template.core
@@ -535,6 +551,7 @@
query-processor.pivot
request
settings
startup
system
task
tiles
@@ -627,6 +644,7 @@
:ns-prefix "metabase.collections-rest"
:api #{metabase.collections-rest.api}
:uses #{api
api-scope
api.eid-translation
collections
documents
@@ -672,11 +690,11 @@
content-verification
{:team "UX West"
:api #{metabase.content-verification.core
metabase.content-verification.init}
:api #{metabase.content-verification.core metabase.content-verification.init metabase.content-verification.schema}
:uses #{app-db
lib.schema
models
users
util}
:model-exports #{:model/ModerationReview}
:model-imports #{:model/Card :model/Dashboard :model/User}}
@@ -734,14 +752,17 @@
metabase.dashboards.schema
metabase.dashboards.settings
metabase.dashboards.write}
:uses #{analytics
:uses #{actions
analytics
api
app-db
audit-app
collections
content-verification
embedding
events
lib
lib.be
lib.schema
models
parameters
@@ -779,6 +800,7 @@
:uses #{actions
analytics
api
api-scope
api.eid-translation
app-db
channel
@@ -786,6 +808,7 @@
dashboards
embedding
events
lib
lib.be
lib.schema
models
@@ -887,7 +910,6 @@
metabase.driver.settings
metabase.driver.sql
metabase.driver.sql-jdbc
metabase.driver.sql-jdbc.connection
metabase.driver.sql-jdbc.execute
metabase.driver.sql-jdbc.execute.diagnostic
metabase.driver.sql-jdbc.sync
@@ -913,6 +935,7 @@
query-processor.driver-api
settings
sql-tools
startup
system
tracing
util
@@ -940,28 +963,6 @@
:model-imports #{:model/Card :model/Dashboard}
:model-exports #{:model/EmbeddingTheme}}
embedding-hub
{:team "Embedding"
:api #{metabase.embedding-hub.api}
:uses #{api
appearance
audit-app
embedding
metabot
permissions
premium-features
util.config
warehouse-schema-overlay}
:model-imports #{:model/Card
:model/Collection
:model/ConnectionImpersonation
:model/Dashboard
:model/Database
:model/DatabaseRouter
:model/EmbeddingTheme
:model/Sandbox
:model/Table
:model/Tenant}}
embedding.rest
{:team "Embedding"
@@ -1004,7 +1005,7 @@
explorations
{:team "UX West"
:api #{metabase.explorations.core metabase.explorations.init}
:api #{metabase.explorations.api metabase.explorations.core metabase.explorations.init metabase.explorations.schema}
:uses #{analytics
analytics.interface
api
@@ -1084,11 +1085,12 @@
glossary
{:team "Gadget"
:api #{metabase.glossary.api}
:api #{metabase.glossary.api metabase.glossary.core}
:uses #{api
events
lib.schema
models
remote-sync
util}
:model-exports #{:model/Glossary}
:model-imports #{:model/User}}
@@ -1108,6 +1110,7 @@
indexed-entities
{:team "UX West"
:api #{metabase.indexed-entities.api
metabase.indexed-entities.core
metabase.indexed-entities.init}
:uses #{analytics
api
@@ -1126,8 +1129,8 @@
interestingness
{:team "UX West"
:api #{metabase.interestingness.core}
:uses #{util}}
:api #{metabase.interestingness.chart.types metabase.interestingness.core}
:uses #{lib.be util}}
lib
{:team "Querying Platform"
@@ -1148,6 +1151,7 @@
metabase.lib.normalize
metabase.lib.options
metabase.lib.pivot
metabase.lib.ref
metabase.lib.types.isa
metabase.lib.util
metabase.lib.walk}
@@ -1312,6 +1316,7 @@
:uses #{activity-feed
ai-tracing
api
api-scope
api.eid-translation
app-db
bookmarks
@@ -1320,6 +1325,7 @@
comments
dashboards
documents
glossary
lib
lib.be
lib.metadata
@@ -1348,6 +1354,7 @@
util
util.config
warehouse-schema-overlay
warehouses
warehouses.schema}
:model-imports #{:model/Card
:model/Collection
@@ -1380,6 +1387,7 @@
mcp
models
request
server
settings
system
task
@@ -1394,14 +1402,17 @@
events
lib
lib.be
lib.metric
lib.schema
metrics
models
permissions
remote-sync
search
users
util
warehouse-schema-overlay}
warehouse-schema-overlay
warehouses.schema}
:model-exports #{:model/Measure}
:model-imports #{:model/Collection :model/Table :model/User}}
@@ -1469,17 +1480,16 @@
task
timeline
transforms
transforms.base
util
util.config
warehouse-schema-overlay
warehouses
warehouses.schema}
warehouses}
:model-exports #{:model/AiUsageLog
:model/Metabot
:model/MetabotConversation
:model/MetabotFeedback
:model/MetabotMessage}
:model/MetabotMessage
:model/MetabotPrompt}
:model-imports #{:model/Card
:model/Collection
:model/Dashboard
@@ -1560,6 +1570,7 @@
settings
setup
sql-tools
startup
sync
util
util.config
@@ -1594,10 +1605,7 @@
system
upload
util}
:model-imports #{:model/AuthIdentity
:model/Card
:model/Database
:model/MetabotMessage}}
:model-imports #{:model/AuthIdentity :model/Card :model/MetabotMessage}}
metrics
{:team "UX West"
@@ -1672,20 +1680,53 @@
lib.legacy-mbql
lib.metadata
lib.schema
parameters
util
util.classloader
warehouse-schema-overlay}
:model-imports #{:model/Card
:model-imports #{:model/Action
:model/Card
:model/Channel
:model/ChannelTemplate
:model/Collection
:model/CustomVizPlugin
:model/Dashboard
:model/DashboardCard
:model/DashboardCardSeries
:model/DashboardTab
:model/Database
:model/Dimension
:model/Document
:model/EmbeddingTheme
:model/Exploration
:model/Field
:model/FieldUserSettings
:model/FieldValues
:model/Glossary
:model/HTTPAction
:model/ImplicitAction
:model/Measure
:model/Metabot
:model/MetabotPrompt
:model/NativeQuerySnippet
:model/Notification
:model/NotificationCard
:model/NotificationHandler
:model/NotificationRecipient
:model/NotificationSubscription
:model/OsiAiContext
:model/PythonLibrary
:model/QueryAction
:model/Segment
:model/Table
:model/TableIndex
:model/Timeline
:model/TimelineEvent
:model/Transform
:model/TransformJob
:model/TransformJobTransformTag
:model/TransformTag
:model/TransformTransformTag
:model/User}}
mq
@@ -1701,9 +1742,9 @@
native-query-snippets
{:team "Graphy"
:api #{metabase.native-query-snippets.api
metabase.native-query-snippets.core}
:api #{metabase.native-query-snippets.api metabase.native-query-snippets.core metabase.native-query-snippets.schema}
:uses #{api
api-scope
collections
events
lib
@@ -1712,6 +1753,7 @@
permissions
premium-features
remote-sync
users
util}
:model-exports #{:model/NativeQuerySnippet}
:model-imports #{:model/Collection :model/User}}
@@ -1754,8 +1796,10 @@
util
util.config}
:model-exports #{:model/Notification
:model/NotificationCard
:model/NotificationHandler
:model/NotificationRecipient}
:model/NotificationRecipient
:model/NotificationSubscription}
:model-imports #{:model/Card
:model/Channel
:model/ChannelTemplate
@@ -1796,6 +1840,7 @@
metabase.parameters.shared}
:uses #{api
app-db
dashboards
lib
lib.be
lib.metadata
@@ -1817,6 +1862,7 @@
:api #{metabase.permissions.core
metabase.permissions.init
metabase.permissions.metric
metabase.permissions.path
metabase.permissions.schema}
:uses #{api
app-db
@@ -1834,6 +1880,8 @@
util
util.config
warehouse-schema-overlay
warehouses
warehouses.schema
enterprise/permissions}
:model-exports #{:model/ApplicationPermissionsRevision
:model/CollectionPermissionGraphRevision
@@ -1881,6 +1929,7 @@
{:team "UX West"
:api #{metabase.premium-features.api
metabase.premium-features.core
metabase.premium-features.defenterprise
metabase.premium-features.init}
:uses #{analytics.interface
analytics.internal-stats
@@ -2070,8 +2119,10 @@
:ns-prefix "metabase.queries-rest"
:api #{metabase.queries-rest.api}
:uses #{api
api-scope
api.eid-translation
collections
embedding
events
lib
lib.be
@@ -2154,6 +2205,7 @@
:uses #{analytics
analytics.interface
api
api-scope
app-db
appearance
audit-app
@@ -2264,6 +2316,7 @@
lib.schema
permissions
premium-features
server
session
settings
users
@@ -2275,18 +2328,23 @@
{:team "UX West"
:api #{metabase.revisions.api
metabase.revisions.core
metabase.revisions.init}
metabase.revisions.init
metabase.revisions.models.revision}
:uses #{api
collections
dashboards
events
explorations
lib
lib.be
lib.schema
measures
models
parameters
queries
query-permissions
segments
transforms
util
util.config}
:model-exports #{:model/Revision}
@@ -2301,7 +2359,8 @@
:model/Measure
:model/ParameterCard
:model/Segment
:model/Transform}}
:model/Transform
:model/TransformTest}}
run-tracking
{:team "Gadget"
@@ -2343,6 +2402,7 @@
:uses #{analytics
analytics.interface
api
api-scope
app-db
appearance
audit-app
@@ -2404,8 +2464,10 @@
permissions
remote-sync
search
users
util
warehouse-schema-overlay
warehouses.schema
xrays}
:model-exports #{:model/Segment}
:model-imports #{:model/Collection :model/Table :model/User}}
@@ -2422,6 +2484,7 @@
analytics.interface
api
api-keys
api-scope
app-db
appearance
core.initialization-status
@@ -2453,8 +2516,10 @@
:api #{metabase.session.api
metabase.session.core
metabase.session.init
metabase.session.schema
metabase.session.settings} ; TODO -- shouldn't be exposed as an API namespace
:uses #{api
api-scope
app-db
channel
events
@@ -2577,6 +2642,7 @@
server
settings
sso.auth-identity
startup
system
util
util.config}
@@ -2604,7 +2670,8 @@
staleness
{:team "UX West"
:api #{metabase.staleness.core}}
:api #{metabase.staleness.core}
:uses #{util}}
startup
{:team "DevEx"
@@ -2625,7 +2692,7 @@
metabase.sync.field-values
metabase.sync.init
metabase.sync.schedules
metabase.sync.task.sync-databases
metabase.sync.task.sync-databases-trigger
metabase.sync.util}
:uses #{analytics.interface
api
@@ -2751,6 +2818,7 @@
premium-features
query-processor
settings
startup
util}
:model-imports #{:model/Card :model/Dashboard :model/DashboardCard}}
@@ -2760,12 +2828,13 @@
metabase.timeline.core}
:uses #{analytics
api
api-scope
collections
events
lib.schema
models
util}
:model-exports #{:model/Timeline}
:model-exports #{:model/Timeline :model/TimelineEvent}
:model-imports #{:model/Collection :model/User}}
tracing
@@ -2824,7 +2893,8 @@
:model/TransformJobRun
:model/TransformJobTransformTag
:model/TransformRun
:model/TransformTag}
:model/TransformTag
:model/TransformTransformTag}
:model-imports #{:model/Collection
:model/Database
:model/Field
@@ -2866,11 +2936,12 @@
transforms.indexes
{:team "Gadget"
:ns-prefix "metabase.indexes"
:api #{metabase.indexes.models.table-index metabase.indexes.reconcile}
:api #{metabase.indexes.models.table-index metabase.indexes.reconcile metabase.indexes.schema}
:uses #{driver
lib.schema
models
util}
util
warehouses}
:model-exports #{:model/TableIndex}}
transforms.indexes.rest
@@ -2994,6 +3065,7 @@
:api #{metabase.user-key-value.api
metabase.user-key-value.init}
:uses #{api
api-scope
lib.schema
util
util.config}}
@@ -3037,6 +3109,7 @@
:ns-prefix "metabase.users-rest"
:api #{metabase.users-rest.api}
:uses #{api
api-scope
appearance
collections
events
@@ -3137,6 +3210,7 @@
:api #{metabase.warehouses-rest.api}
:uses #{analytics
api
api-scope
app-db
collections
database-routing
@@ -3156,13 +3230,11 @@
sync
upload
util
util.classloader
util.config
warehouse-schema-overlay
warehouses
warehouses.schema
warehouses.secrets
enterprise/permissions}
warehouses.secrets}
:model-imports #{:model/Card
:model/Collection
:model/Database
@@ -3205,11 +3277,8 @@
settings
sync.analyze
util
util.classloader
util.config
warehouse-schema-overlay
warehouses
enterprise/permissions}
warehouses}
:model-exports #{:model/Dimension
:model/Field
:model/FieldUserSettings
@@ -3230,6 +3299,7 @@
:api #{metabase.warehouse-schema-rest.api}
:uses #{analytics
api
api-scope
app-db
collections
database-routing
@@ -3279,6 +3349,7 @@
collections
dashboards
driver
indexed-entities
lib
lib.be
lib.legacy-mbql
@@ -3303,7 +3374,6 @@
:model/Database
:model/Field
:model/ModelIndex
:model/ModelIndexValue
:model/Query
:model/Revision
:model/Segment
@@ -3532,6 +3602,7 @@
{:team "UX West"
:api #{metabase-enterprise.content-translation.routes}
:uses #{api
api-scope
content-translation
embedding
premium-features
@@ -3561,6 +3632,7 @@
metabase-enterprise.custom-viz-plugin.init
metabase-enterprise.custom-viz-plugin.settings}
:uses #{api
api-scope
events
models
premium-features
@@ -3574,6 +3646,7 @@
{:team "Embedding"
:api #{metabase-enterprise.data-apps.api metabase-enterprise.data-apps.init metabase-enterprise.data-apps.sync}
:uses #{api
api-scope
collections
lib
lib.be
@@ -3635,10 +3708,7 @@
enterprise/dependencies
{:team "Graphy"
:module-exports #{enterprise/dependencies.replacement}
:api #{metabase-enterprise.dependencies.api
metabase-enterprise.dependencies.core
metabase-enterprise.dependencies.init
metabase-enterprise.dependencies.models.dependency}
:api #{metabase-enterprise.dependencies.api metabase-enterprise.dependencies.init metabase-enterprise.dependencies.models.dependency}
:uses #{api
app-db
collections
@@ -3649,14 +3719,18 @@
lib
lib.be
lib.metadata
lib.metric
lib.schema
measures
models
native-query-snippets
permissions
premium-features
queries
query-processor
request
revisions
segments
settings
sql-tools
task
@@ -3664,7 +3738,8 @@
transforms.base
util
util.graph
warehouse-schema-overlay}
warehouse-schema-overlay
enterprise/permissions.sandbox}
:model-imports #{:model/Card
:model/Collection
:model/Dashboard
@@ -3694,6 +3769,7 @@
measures
model-persistence
models
parameters
queries
segments
task
@@ -3726,8 +3802,13 @@
enterprise/embedding
{:team "Embedding"
:ns-prefix "metabase-enterprise.embedding-hub"
:api #{}
:uses #{lib.schema
:api #{metabase-enterprise.embedding-hub.api}
:uses #{api
appearance
audit-app
embedding
lib.schema
permissions
premium-features
util
warehouse-schema-overlay
@@ -3789,7 +3870,6 @@
metabase-enterprise.metabot.api.routes
metabase-enterprise.metabot.init}
:uses #{api
lib.be
lib.schema
metabot
metabot.llm
@@ -3799,14 +3879,8 @@
settings
task
util
enterprise/cloud.add-ons
enterprise/dependencies
enterprise/transforms.python}
:model-imports #{:model/AiUsageLog
:model/Card
:model/PermissionsGroup
:model/PermissionsGroupMembership
:model/Transform}}
enterprise/cloud.add-ons}
:model-imports #{:model/AiUsageLog :model/PermissionsGroup :model/PermissionsGroupMembership}}
enterprise/metabot.agent-api
{:team "Metabot"
@@ -3961,8 +4035,9 @@
enterprise/permissions.sandbox
{:team "UX West"
:ns-prefix "metabase-enterprise.sandbox"
:api #{metabase-enterprise.sandbox.api.routes metabase-enterprise.sandbox.api.util}
:api #{metabase-enterprise.sandbox.api.routes metabase-enterprise.sandbox.api.util metabase-enterprise.sandbox.schema}
:uses #{api
api-scope
app-db
audit-app
driver
@@ -4035,6 +4110,7 @@
:model/Document
:model/Field
:model/FieldUserSettings
:model/Glossary
:model/Measure
:model/NativeQuerySnippet
:model/PythonLibrary
@@ -4044,6 +4120,7 @@
:model/Timeline
:model/Transform
:model/TransformTag
:model/TransformTest
:model/User}}
enterprise/search
@@ -4278,6 +4355,31 @@
:model-exports #{:model/Tenant}
:model-imports #{:model/Collection :model/User}}
enterprise/transform-testing
{:team "Graphy"
:api #{metabase-enterprise.transform-testing.api metabase-enterprise.transform-testing.init}
:uses #{api
audit-app
driver
events
lib
lib.schema
models
premium-features
revisions
run-tracking
sql-tools
sql-tools.parsing
task
transforms.base
util
warehouses}
:model-exports #{:model/TransformTest}
:model-imports #{:model/Collection
:model/Database
:model/Transform
:model/User}}
enterprise/transforms
{:team "Graphy"
:module-exports #{enterprise/transforms.inspector

View File

@@ -0,0 +1,21 @@
(ns macros.potemkin.namespaces)
(defn- import-var-expansion
"Expansion shared by all of the `import-*` replacements below. Synthetic \"docstring\" is needed to silence
`:missing-docstring` linter."
([sym]
(import-var-expansion sym (-> sym name symbol)))
([sym new-name]
`(def ~new-name "docstring" ~sym)))
(defmacro import-def
([sym] (import-var-expansion sym))
([sym new-name] (import-var-expansion sym new-name)))
(defmacro import-fn
([sym] (import-var-expansion sym))
([sym new-name] (import-var-expansion sym new-name)))
(defmacro import-macro
([sym] (import-var-expansion sym))
([sym new-name] (import-var-expansion sym new-name)))

View File

@@ -14,10 +14,10 @@
:consistent-alias 2
:def-fn 1
:deprecated-namespace 95
:deprecated-var 96
:deprecated-var 85
:discouraged-java-method 4
:discouraged-namespace 71
:discouraged-var 130
:discouraged-namespace 68
:discouraged-var 132
:duplicate-require 1
:equals-true 1
:main-without-gen-class 1
@@ -26,13 +26,13 @@
:metabase/i-like-making-cams-eyes-bleed-with-horrifically-long-tests 13
:metabase/modules 24
:metabase/namespace-name 4
:metabase/prefer-with-dynamic-fn-redefs 28
:metabase/prefer-with-dynamic-fn-redefs :unlimited
:metabase/test-helpers-use-non-thread-safe-functions 18
:metabase/validate-defendpoint-has-response-schema 427
:metabase/validate-defendpoint-query-params-use-kebab-case 48
:metabase/validate-defendpoint-route-uses-kebab-case 44
:metabase/validate-deftest 9
:missing-docstring 12
:missing-docstring 4
:missing-protocol-method 3
:non-arg-vec-return-type-hint 1
:reduce-without-init 3
@@ -59,9 +59,10 @@
:main-without-gen-class 1
:metabase/defsetting-must-specify-export 1
:metabase/defsetting-namespace 1
:metabase/prefer-with-dynamic-fn-redefs 3
:metabase/prefer-with-dynamic-fn-redefs 4
:metabase/t2-query-namespace 2
:metabase/table-or-field-query 1
:metabase/unsafe-app-db-query 1
:missing-docstring 1
:private-call 1
:redundant-ignore 1

View File

@@ -89,6 +89,12 @@
[modules module ancestor]
(boolean (some #{ancestor} (take-while some? (iterate #(parent-module modules %) module)))))
(defn module-team
"Team owning `module`: its own `:team`, else its nearest ancestor's."
[modules module]
(some #(get-in modules [% :team])
(take-while some? (iterate #(parent-module modules %) module))))
(defn- rest-module? [module]
(re-find #"[.-]rest$" (str module)))
@@ -145,11 +151,11 @@
[config ns-symb]
(resolve-module (prefix->module config) ns-symb))
(defn- module-api-namespaces
(defn module-api-namespaces
"Set of API namespace symbols for a given module. `:any` means you can use anything, there are no API namespaces for
this module (yet). If unspecified, the default is just the `<module>.core` namespace."
[config module]
(let [module-config (get-in config [:metabase/modules module :api])]
this module (yet). If unspecified, the defaults are `<prefix>.api`, `<prefix>.core`, and `<prefix>.init`."
[modules module]
(let [module-config (get-in modules [module :api])]
(cond
(= module-config :any)
nil
@@ -158,7 +164,7 @@
module-config
:else
(let [ns-prefix (module-ns-prefix (:metabase/modules config) module)]
(let [ns-prefix (module-ns-prefix modules module)]
#{(symbol (str ns-prefix ".api"))
(symbol (str ns-prefix ".core"))
(symbol (str ns-prefix ".init"))}))))
@@ -181,7 +187,7 @@
(defn- allowed-module-namespace? [config current-module ns-symb]
(let [module (module config ns-symb)
module-api-namespaces (module-api-namespaces config module)
module-api-namespaces (module-api-namespaces (:metabase/modules config) module)
module-friends (module-friends config module)]
(or (nil? module-api-namespaces)
(contains? module-api-namespaces ns-symb)

View File

@@ -0,0 +1,375 @@
(ns hooks.metabase.agent-message
"Hooks linting agent-facing prose built with `metabase.mcp.v2.message/msg` (`:metabase/agent-message-lines`) and the
exits and helpers that carry it to the agent (`:metabase/agent-message-exit`). Each hook checks only when its
linter's level is configured and not `:off`, and returns its input unchanged."
(:require
[clj-kondo.hooks-api :as hooks]
[clojure.string :as str]))
(defn- level-on?
[config linter-key]
(not= :off (get-in config [:linters linter-key :level] :off)))
(defn- call-name
"The symbol at the head of list `node`, or nil."
[node]
(when (hooks/list-node? node)
(let [head (first (:children node))]
(when (hooks/token-node? head)
(let [v (hooks/sexpr head)]
(when (symbol? v) v))))))
(defn- calls?
"Whether `node` is a call to a var named `var-name` in one of `ns-names` (nil for an unqualified call)."
[node ns-names var-name]
(when-let [sym (call-name node)]
(and (= var-name (name sym))
(contains? ns-names (namespace sym)))))
(def ^:private core-ns #{nil "clojure.core"})
(def ^:private string-ns #{"str" "string" "clojure.string"})
(defn- literal-string
"The string value of `node` when it is a string literal or a `str` call over string literals, else nil."
[node]
(cond
(hooks/string-node? node)
(hooks/sexpr node)
(and (calls? node core-ns "str")
(every? hooks/string-node? (rest (:children node))))
(apply str (map hooks/sexpr (rest (:children node))))))
(defn- digits->long
"The value of a string of decimal digits."
[s]
;; Kondo's hook interpreter has no `parse-long`.
(reduce (fn [n c] (+ (* 10 n) (- (int c) (int \0)))) 0 s))
(def ^:private format-specifier
"A `java.util.Formatter` specifier; groups are explicit index, flags, width, precision, and conversion. The index is
capped at nine digits, so a longer one reads as malformed rather than a number too large to work with."
#"%(?:(\d{1,9})\$)?([-#+ 0,(<]*)(\d+)?(?:\.(\d+))?([tT]?[a-zA-Z%])")
(defn- consumed-args
"`[index conversion]` for each argument-consuming specifier in `fmt`, with zero-based argument indexes. An index of
-1 marks a reference no argument can satisfy: an explicit `0$`, or a `<` with no specifier before it to repeat."
[fmt]
(loop [[[_ explicit flags _width _precision conversion] & more] (re-seq format-specifier fmt)
next-index 0
last-index nil
acc []]
(if-not conversion
acc
(let [consumes? (not (contains? #{"%" "n"} conversion))
index (cond
(not consumes?) nil
explicit (dec (digits->long explicit))
(str/includes? flags "<") (or last-index -1)
:else next-index)
implicit? (and consumes? (not explicit) (not (str/includes? flags "<")))]
(recur more
(cond-> next-index implicit? inc)
(or index last-index)
(cond-> acc
index (conj [index conversion])))))))
(def ^:private msg-linter :metabase/agent-message-lines)
(defn- reg-msg-finding!
[node message]
(hooks/reg-finding! (assoc (meta node) :message message :type msg-linter)))
(defn- line-break?
"Whether format string `line` contains a line-breaking, control, or private-use character, or a `%n` specifier."
[line]
;; Private-use characters delimit the argument markers `render` formats with, so a line holding one could fake one.
(boolean (or (re-find #"[\p{Cc}\p{Cf}\p{Co}\p{Zl}\p{Zp}]" line)
(some #(= "n" (nth % 5)) (re-seq format-specifier line)))))
(defn- allowed-specifier?
"Whether specifier match `specifier` renders its value as `render` passes it: a bare `%%`, a `%s` with no flags but
`<`, or a `%d` with any flags and width but no precision. An explicit argument index is allowed on each."
[[_ explicit flags width precision conversion]]
(case conversion
"%" (not (or explicit (seq flags) width precision))
"s" (not (or width precision (seq (str/replace flags "<" ""))))
"d" (not precision)
false))
(defn- disallowed-specifier
"The first specifier in format string `line` that isn't [[allowed-specifier?]], as it is written, or nil."
[line]
(some #(when-not (allowed-specifier? %) (first %)) (re-seq format-specifier line)))
(def ^:private valid-conversion
"The conversions `java.util.Formatter` accepts, with each date/time conversion's required suffix."
#"[bBhHsScCdoxXeEfgGaA%n]|[tT][HIklMSLNpzZsQBbhAaCYyjmdeRTrDFc]")
(defn- malformed-specifier?
"Whether format string `line` has a `%` that doesn't begin a valid `java.util.Formatter` specifier."
[line]
(let [valid-removed (str/replace line format-specifier
(fn [[specifier & groups]]
(if (re-matches valid-conversion (last groups)) "" specifier)))]
(str/includes? valid-removed "%")))
(defn- line-problem
"The finding message for the first problem with a `msg` line, given its literal string or nil; nil when it has none."
[line]
(cond
(nil? line)
(str "Each line of a `msg` must be a string literal or a `str` of string literals; "
"pass values as arguments after the vector.")
(line-break? line)
(str "A `msg` line can't contain a line break, `%n`, or a control or private-use character; "
"put each line in its own string.")
(malformed-specifier? line)
"A `%` in a `msg` line must begin a format specifier; write a literal percent sign as `%%`."
:else
(when-let [specifier (disallowed-specifier line)]
(str "A `msg` line may only use `%s`, `%d`, or `%%`; `" specifier "` would alter or hide the value."))))
(defn- lint-lines!
"Flag each of `line-nodes` whose literal string, the matching element of `lines`, has a problem."
[line-nodes lines]
(doseq [[line-node line] (map vector line-nodes lines)]
(when-let [problem (line-problem line)]
(reg-msg-finding! line-node problem))))
(defn- plural
[n word]
(str n " " word (when-not (= 1 n) "s")))
(defn- lint-arg-count!
"Flag `msg` call `node` when a literal line refers to an argument that can't exist, or when its `args` don't number
the arguments its literal `lines` consume."
[node lines args]
(let [indexes (map first (consumed-args (str/join "\n" lines)))
expected (if (seq indexes) (inc (apply max indexes)) 0)
given (count args)]
(cond
;; `String/format` throws on these, so the message would render with every part quoted instead.
(some neg? indexes)
(reg-msg-finding! node (str "A `msg` line refers to an argument that can't exist; "
"argument indexes start at `1$`, and `%<` repeats the specifier before it."))
(not= expected given)
(reg-msg-finding! node (format "The `msg` lines take %s but %d %s given."
(plural expected "argument") given (if (= 1 given) "is" "are"))))))
(defn lint-msg
"Flag `msg` calls whose lines aren't a literal vector of single-line literal strings, that hold a malformed
specifier or a conversion other than `%s`, `%d`, or `%%`, or whose arguments don't match the lines' specifiers."
[{:keys [node config] :as input}]
(when (level-on? config msg-linter)
(let [[fn-node lines-node & args] (:children node)]
(if-not (and lines-node (hooks/vector-node? lines-node))
(reg-msg-finding! (or lines-node fn-node)
"`msg` takes a literal vector of line strings, one string per line of the message.")
(let [line-nodes (:children lines-node)
lines (map literal-string line-nodes)]
(lint-lines! line-nodes lines)
;; A malformed specifier can't be counted, and its line already has a finding.
(when (and (every? some? lines) (not-any? malformed-specifier? lines))
(lint-arg-count! node lines args))))))
input)
(def ^:private exit-linter :metabase/agent-message-exit)
(defn- exit-enabled?
[{:keys [config]}]
(level-on? config exit-linter))
(defn- reg-exit-finding!
[node message]
(hooks/reg-finding! (assoc (meta node) :message message :type exit-linter)))
(defn- calls-core?
"Whether `node` is a call to a `clojure.core` var or special form named in `var-names`."
[node var-names]
(when-let [sym (call-name node)]
(and (contains? var-names (name sym))
(contains? core-ns (namespace sym)))))
(defn- calls-i18n?
[node]
(when-let [sym (call-name node)]
(contains? #{"tru" "trs" "deferred-tru" "deferred-trs"} (name sym))))
(defn- calls-json-encode?
"Whether `node` calls `encode` in a namespace or alias whose name ends in `json`."
[node]
(when-let [sym (call-name node)]
(and (= "encode" (name sym))
(boolean (some-> (namespace sym) (str/ends-with? "json"))))))
(defn- builds-string?
"Whether `node` is a string literal or a call that builds a string."
[node]
(or (hooks/string-node? node)
(calls-core? node #{"str" "format" "pr-str" "ex-message"})
(calls-i18n? node)
(calls? node string-ns "join")
(calls-json-encode? node)))
(defn- clause-results
"The result nodes of `clauses`, alternating test and result nodes, with a trailing default when their count is odd."
[clauses]
(concat (take-nth 2 (rest clauses))
(when (odd? (count clauses)) [(last clauses)])))
(defn- result-fn-marker?
[node]
(and (hooks/keyword-node? node) (= :>> (hooks/sexpr node))))
(defn- condp-results
"The result nodes of `condp` `clauses`: each test's result, skipping `:>>` result functions, and any default."
[clauses]
(loop [[_test result & more :as clauses] clauses
acc []]
(cond
(empty? clauses) acc
(nil? (next clauses)) (conj acc (first clauses))
(result-fn-marker? result) (recur (next more) acc)
:else (recur more (conj acc result)))))
(defn- branch-results
"The nodes branching form `node` can evaluate to, or nil when it isn't one."
[node]
(let [args (rest (:children node))]
(cond
(calls-core? node #{"if" "if-not" "if-let"}) (rest args)
(calls-core? node #{"or"}) args
(calls-core? node #{"cond"}) (clause-results args)
(calls-core? node #{"case"}) (clause-results (rest args))
(calls-core? node #{"condp"}) (condp-results (drop 2 args)))))
(defn- step-call
"Threading step `node` as the call it stands for: a list as is, anything else as a call with no written arguments."
[node]
(if (hooks/list-node? node) node (hooks/list-node [node])))
(defn- threading-results
"The nodes whose text-ness decides what threading form `node` evaluates to, or nil when it isn't one."
[node]
(let [[init & steps] (rest (:children node))]
(cond
(calls-core? node #{"->" "->>" "some->" "some->>"}) [(if steps (step-call (last steps)) init)]
(calls-core? node #{"cond->" "cond->>"}) (cons init (map step-call (take-nth 2 (rest steps))))
(calls-core? node #{"as->"}) [(or (last (rest steps)) init)]
(calls-core? node #{"doto"}) [init])))
(defn- tail-result
"The node body form `node` evaluates to, or nil when it isn't one."
[node]
(let [args (rest (:children node))]
(cond
(calls-core? node #{"when" "when-not" "when-let" "let"}) (when (next args) (last args))
(calls-core? node #{"do"}) (last args))))
(defn- stringy?
"Whether `node` syntactically evaluates to text: a string literal, a call that builds a string, or a threading,
branching, or body form whose result is one."
[node]
(boolean
(or (builds-string? node)
(some stringy? (branch-results node))
(some stringy? (threading-results node))
(some-> (tail-result node) stringy?))))
(defn- lint-exit-text!
"Flag `arg`, the message passed to `target` (a description of where it goes), when it is text rather than a `msg`."
[arg target]
(when (and arg (stringy? arg))
(reg-exit-finding! arg (format (str "`%s` passes text to %s; "
"build agent-facing text with `msg` so interpolated values are cleaned.")
(pr-str (hooks/sexpr arg))
target))))
(defn- lint-exit-arg!
"Flag the argument at child `index` of call `node` when it is text rather than a `msg`."
[node index]
(lint-exit-text! (nth (:children node) index nil) (str "`" (name (call-name node)) "`")))
(defn lint-teaching-exit
"Flag a call throwing or returning a teaching error whose message, the first argument, is text rather than a `msg`."
[{:keys [node] :as input}]
(when (exit-enabled? input)
(lint-exit-arg! node 1))
input)
(defn lint-jsonrpc-error
"Flag a `jsonrpc-error` whose message, the third argument, is text rather than a `msg`."
[{:keys [node] :as input}]
(when (exit-enabled? input)
(lint-exit-arg! node 3))
input)
(defn lint-op-error
"Flag an `op-error!` whose detail message, the third argument, is text rather than a `msg`."
[{:keys [node] :as input}]
(when (exit-enabled? input)
(lint-exit-arg! node 3))
input)
(defn lint-ellipsize
"Flag an `ellipsize` whose message, the first argument, is text rather than a `msg`."
[{:keys [node] :as input}]
(when (exit-enabled? input)
(lint-exit-arg! node 1))
input)
(defn- map-value
"The value node of keyword `k` in map literal `node`, or nil."
[node k]
(when (hooks/map-node? node)
(some (fn [[key-node value-node]]
(when (and (hooks/keyword-node? key-node) (= k (hooks/sexpr key-node)))
value-node))
(partition 2 (:children node)))))
(defn lint-list-content
"Flag a `list-content`, or a wrapper taking its options map as the third argument too, whose literal options map
has an `:empty-hint` that is text rather than a `msg`."
[{:keys [node] :as input}]
(when (exit-enabled? input)
(lint-exit-text! (map-value (nth (:children node) 3 nil) :empty-hint)
(str "`" (name (call-name node)) "`'s `:empty-hint`")))
input)
(defn lint-success-content
"Flag a `success-content` whose first argument is text rather than a payload or a `msg`."
[{:keys [node] :as input}]
(when (exit-enabled? input)
(let [text (second (:children node))]
(when (and text (stringy? text))
(reg-exit-finding! text (str "`success-content` takes data to JSON-encode or a `msg`; "
"pass the payload itself, or build the text with `msg`.")))))
input)
(defn- caller-facing-data?
"Whether `node` is a map literal with a 4xx `:status-code` literal or an `error-code` key."
[node]
(and (hooks/map-node? node)
(some (fn [[k v]]
(when (hooks/keyword-node? k)
(let [kw (hooks/sexpr k)]
(or (= "error-code" (name kw))
(and (= :status-code kw)
(hooks/token-node? v)
(let [status (hooks/sexpr v)]
(and (number? status) (<= 400 status 499))))))))
(partition 2 (:children node)))))
(defn lint-ex-info
"Flag an `ex-info` whose data marks it as a caller-facing error."
[{:keys [node] :as input}]
(when (exit-enabled? input)
(when (caller-facing-data? (nth (:children node) 2 nil))
(reg-exit-finding! node (str "Caller-facing errors must be thrown with `throw-teaching-error` and a `msg`, "
"so their text is rendered and cleaned."))))
input)

View File

@@ -1,8 +1,22 @@
(ns hooks.metabase.app-db.schema-migrations-test.impl
(:require [clj-kondo.hooks-api :as hooks]))
(:require
[clj-kondo.hooks-api :as hooks]))
(defn test-migrations
[{{[_ migration-range {[binding] :children} & body] :children} :node}]
(defn- validate-mb-app-db-migrations-test!
"CI runs the namespaces tagged `:mb/app-db-migrations-test` on a shard of their own, because a count-based split
can't balance them (see `.github/workflows/app-db.yml`). An untagged one still runs, just on a shard sized for
everything else."
[{:keys [ns node]}]
(when-not (:mb/app-db-migrations-test (meta ns))
(hooks/reg-finding!
(assoc (meta node)
:message (str "Namespaces that run migration tests (test-migrations) should be marked"
" ^:mb/app-db-migrations-test [:metabase/validate-mb-app-db-migrations-test]")
:type :metabase/validate-mb-app-db-migrations-test))))
(defn test-migrations!
[{{[_ migration-range {[binding] :children} & body] :children} :node, :as x}]
(validate-mb-app-db-migrations-test! x)
(let [node* (hooks/list-node
(list*
(hooks/token-node `let)

View File

@@ -29,6 +29,286 @@
[filename]
(boolean (and filename (re-find #"(?:^|/)test/" filename))))
(def ^:private value-operators
"Operators whose second argument is a value rather than a column."
'#{= not= != <> is-distinct-from is-not-distinct-from
< > <= >= like not-like ilike not-ilike in not-in between not-between})
(defn- marked?
"Whether `node` is an `[:auto/param v]` marker."
[node]
(and (hooks/vector-node? node)
(= :auto/param (some-> (first (:children node)) hooks/sexpr))))
(defn- distinct-by
"`coll` with only the first element for each distinct `(f element)`."
[f coll]
(->> coll (reduce (fn [[seen acc] x]
(let [k (f x)]
(if (contains? seen k) [seen acc] [(conj seen k) (conj acc x)])))
[#{} []])
second))
(def ^:private identifier-clauses
"Clause keys whose entries name a column or a table rather than carrying a value.
A symbol here is a name, and `metabase.app-db.value-guard` refuses a marker in one, so reporting
it would ask for a fix that throws at compile. `:order-by` and `:group-by` are included even
though the guard allows a marker there: it compiles to `ORDER BY ?` and loses the ordering, so
asking for one is still wrong."
#{:select :select-distinct :select-top :from :join :left-join :right-join :inner-join :full-join
:cross-join :update :insert-into :delete-from :returning :with :with-columns :using
:order-by :group-by :partition-by :window})
(def ^:private expression-entry-clauses
"Identifier clauses where index 0 of an `[expr alias]` entry is an expression, not a name.
HoneySQL binds a parameter there -- `SELECT ? AS a` -- so the guard accepts a marker, and mirrors
this set as `expression-entry-clauses` in `metabase.app-db.value-guard`. Everything after index 0
is an alias, and a bare entry is a name; the guard refuses a marker in both."
#{:select :select-distinct :select-top :returning :partition-by})
(defn- comparison-node?
"Whether `node` is a keyword-headed comparison -- the one shape inside an identifier clause that
does hold values, as in the computed projection `[[:= :engine v] :is_match]`."
[node]
(and (hooks/vector-node? node)
(let [op (some-> (first (:children node)) hooks/sexpr)]
(and (keyword? op) (contains? value-operators (symbol (name op)))))))
(declare value-nodes)
(defn- identifier-clause-value-nodes
"The value slots reachable inside an identifier clause: a subquery's own clauses, and the
arguments of a comparison. The names themselves are not candidates."
[node]
(cond
(hooks/map-node? node) (value-nodes node)
(comparison-node? node) (value-nodes node)
(hooks/vector-node? node) (mapcat identifier-clause-value-nodes (:children node))
(hooks/list-node? node) (mapcat identifier-clause-value-nodes (:children node))
:else nil))
(defn- expression-entry-value-nodes
"The value slots of an [[expression-entry-clauses]] clause: index 0 of each `[expr alias]` entry."
[node]
(when (hooks/vector-node? node)
(mapcat (fn [entry]
(when (hooks/vector-node? entry)
(when-let [expr (first (:children entry))]
;; Keep the expression AND descend, so a bare symbol is reported and a computed
;; projection or a subquery sitting there has its own values checked.
(cons expr (value-nodes expr)))))
(:children node))))
(defn- clause-value-nodes
"The value slots of the value under clause key `clause`."
[clause node]
(cond
(contains? expression-entry-clauses clause) (expression-entry-value-nodes node)
(contains? identifier-clauses clause) (identifier-clause-value-nodes node)
:else (value-nodes node)))
(defn- value-nodes
"The nodes sitting in a value slot of `node`, following the clause shapes a query map uses."
[node]
(cond
;; A query map -- walk its clause values, not its keys. A clause that names columns or tables
;; contributes only what is nested inside it.
(hooks/map-node? node)
(mapcat (fn [[k v]]
(if (hooks/keyword-node? k)
(clause-value-nodes (hooks/sexpr k) v)
(value-nodes v)))
(partition 2 (:children node)))
(hooks/vector-node? node)
(let [[head & args] (:children node)
op (some-> head hooks/sexpr)]
(cond
;; A marker IS the value slot -- never descend into its payload, or the marked value
;; gets reported as though it were bare.
(marked? node)
nil
(contains? #{:and :or :not} op)
(mapcat value-nodes args)
;; `[:= col v]`, `[:in col vs]`, `[:between col lo hi]` -- the first argument is the
;; column and the rest are values, whatever the arity. Keep each value AND descend into it,
;; so a subquery in a value slot -- `[:in :id {:select ... :where [:= :z z]}]` -- has its
;; own values checked too. Descending alone would drop the plain-token case.
(and (keyword? op) (contains? value-operators (symbol (name op))))
(mapcat #(cons % (value-nodes %)) (rest args))
;; A function-call form -- `[:lower v]` -- or a literal collection -- `[a b]` in an `:in`.
;; Both hold values, so yield the children themselves as candidates as well as
;; descending, since a token child returns nothing on its own.
:else
(mapcat #(cons % (value-nodes %)) (:children node))))
;; A clause built conditionally -- `(when flag [:= :col v])`, `(if ... )`, `(cond-> ...)`. The
;; value slots are inside, so walk the children rather than stopping. Without this a value
;; wrapped in `when` is invisible to the check, which is a false negative in a security lint.
;;
;; An `assoc` call gets its clause context back: `(assoc q :order-by [[col dir]])` is the same
;; clause as `{:order-by [[col dir]]}`, but written outside a map literal, where the walk would
;; otherwise fall back to shape alone and report the column and the direction as values.
;; `cond->` is covered because the walk descends into the nested `assoc`.
(hooks/list-node? node)
(let [children (vec (:children node))
assoc? (= 'assoc (some-> (first children) hooks/sexpr))
;; index of a clause value -> its clause key
clauses (when assoc?
(into {} (keep-indexed (fn [i child]
(when (and (hooks/keyword-node? child)
(contains? identifier-clauses (hooks/sexpr child))
(< (inc i) (count children)))
[(inc i) (hooks/sexpr child)]))
children)))]
(mapcat (fn [i child]
(if-let [clause (get clauses i)]
(clause-value-nodes clause child)
(value-nodes child)))
(range)
children))
:else nil))
(def ^:private write-fns
"Calls whose values are written rather than filtered on. A value here is not a where-clause value
an attacker can turn into SQL structure, and a `define-before-insert` hook may read it before the
query compiles, so a marker would break it rather than protect anything."
'#{insert! insert-returning-instance! insert-returning-instances! insert-returning-pk!
insert-returning-pks! save!})
(defn- read-call?
"Whether `node` is a query call whose values are filtered on rather than written."
[node]
(let [f (some-> (first (:children node)) hooks/sexpr)]
(not (contains? write-fns (some-> f name symbol)))))
(def ^:private clause-keys
"Query-map clause keys. A map keyed by these is a query map rather than a map of conditions."
#{:select :select-distinct :from :where :join :left-join :right-join :inner-join :full-join
:cross-join :group-by :having :order-by :limit :offset :for :union :union-all :with
:with-columns :returning :values :set})
(defn- query-map-node?
"Whether `node` is a query map rather than a map of column/value conditions.
`(t2/delete! :model/X {:where [:= :k k]})` passes a query map, whose values [[value-nodes]]
already walks -- treating it as conditions too would report every value twice."
[node]
(boolean (some #(and (hooks/keyword-node? %) (contains? clause-keys (hooks/sexpr %)))
(take-nth 2 (:children node)))))
(def ^:private changes-map-fns
"Conditions-map fns whose arglist also ends in a changes map, so a lone trailing map is changes."
'#{update! update-or-insert!})
(def ^:private conditions-map-fns
"Calls whose first argument after the model is a map of CONDITIONS rather than a query map.
`(t2/update! :model/X {:key k} {:v 1})` filters on `k`, so it is a where-clause value -- but it
reaches the query as a plain map entry rather than a `:where` clause, so neither the query-map
walker nor the kv-arg walker sees it. It is a shape the sweep leaves in place, so it has to be
checked here."
'#{update! update-or-insert! delete!})
(defn- conditions-map-value-nodes
"The values of a conditions map passed to one of [[conditions-map-fns]]."
[f args]
(when (contains? conditions-map-fns (some-> f name symbol))
(let [after-model (->> args
(drop-while #(not (and (hooks/keyword-node? %)
(= "model" (namespace (hooks/sexpr %))))))
rest)]
;; Only the map immediately after the model is conditions. A later map is the changes map,
;; whose values are written rather than filtered on, and a written value is never marked: a
;; column's `:in` transform runs on the marker itself and stores it as data.
;; `update!`'s arglist ENDS in the changes map, so its first map is conditions only when
;; another argument follows: `(t2/update! model {:v written})` is the two-arity call whose
;; single map is CHANGES, so flagging it would ask for a marker that corrupts the write.
;; `delete!` has no changes map, so its map is always conditions.
(when-let [m (when (or (not (contains? changes-map-fns (some-> f name symbol)))
(next after-model))
(first after-model))]
(when (and (hooks/map-node? m) (not (query-map-node? m)))
(mapcat (fn [v]
;; The column comes from the map key, so an operator form here holds only
;; values -- `{:key [:in ks]}` -- exactly as a kv-arg pair does.
(if (and (hooks/vector-node? v) (not (marked? v)))
(let [[head & op-args] (:children v)
op (some-> head hooks/sexpr)]
(if (and (keyword? op) (contains? value-operators (symbol (name op))))
(mapcat #(cons % (value-nodes %)) op-args)
(cons v (value-nodes v))))
(cons v (value-nodes v))))
(take-nth 2 (rest (:children m)))))))))
(defn- kv-arg-value-nodes
"The value nodes of a call written as `:column value` pairs.
`(t2/select :model/X :locale locale)`. Several of these fns take an argument before the model --
`(t2/select-one-fn :value :model/X :key k)` -- so the pairs do not start at a fixed offset; they
start after the `:model/...` keyword. A call that does not name a literal model is not checked.
A trailing query map is not a pair, and its values are reached by [[value-nodes]] instead. A
value that is itself an operator form -- `:id [:in ids]` -- has its own values walked, so the
collection inside is checked rather than the form."
[args]
(let [after-model (->> args
(drop-while #(not (and (hooks/keyword-node? %)
(= "model" (namespace (hooks/sexpr %))))))
rest)]
(->> after-model
;; `partition-all` rather than `partition` so a trailing odd argument (a query map, which
;; is not a pair) is still seen and skipped by the keyword-node? test below.
(partition-all 2)
(mapcat (fn [[k v]]
(when (and v (hooks/keyword-node? k))
(if-not (hooks/vector-node? v)
[v]
(let [[head & op-args] (:children v)
op (some-> head hooks/sexpr)]
(cond
;; Already marked -- the marker is the value slot.
(marked? v) [v]
;; `:id [:in ids]`. Toucan folds the column in from the pair key, so
;; EVERY argument here is a value -- unlike a query-map clause, where
;; the first argument is the column.
(and (keyword? op)
(contains? value-operators (symbol (name op))))
(mapcat #(cons % (value-nodes %)) op-args)
:else (cons v (value-nodes v)))))))))))
(defn- lint-unmarked-values!
"Register a finding for each argument of the enclosing function that reaches a value slot unmarked.
Only a symbol is reported. A literal cannot carry a request value, and a value built inside the
function is out of reach of a check that does not follow it across a call."
[node]
(doseq [value (let [args (rest (:children node))
f (some-> (first (:children node)) hooks/sexpr)]
;; The three walkers overlap on some shapes -- a multi-arg operator in a kv-arg
;; reaches both `value-nodes` and `kv-arg-value-nodes` -- so dedupe by source
;; position rather than trying to keep them disjoint.
(->> (concat (mapcat value-nodes args)
(kv-arg-value-nodes args)
(conditions-map-value-nodes f args))
(distinct-by #(select-keys (meta %) [:row :col]))))
:when (and (hooks/token-node? value)
(symbol? (hooks/sexpr value))
(not (marked? value)))]
(hooks/reg-finding!
(assoc (meta value)
:message (format "`%s` reaches a SQL value slot unmarked. Write it as [:auto/param %s] so it is bound as a parameter."
(hooks/sexpr value) (hooks/sexpr value))
:type :metabase/unsafe-app-db-query))))
(defn lint-query-call
"Register a `:metabase/t2-query-namespace` finding when a Toucan 2 query call appears outside a `<module>.db`
namespace."
@@ -43,4 +323,9 @@
(hooks/sexpr fn-node))
:type :metabase/t2-query-namespace))))
((requiring-resolve 'hooks.metabase.warehouse-schema-overlay.table-or-field-query/lint-read) input)
(when (and ns
(db-namespace? (modules/config input) ns)
(not (test-file? filename))
(read-call? node))
(lint-unmarked-values! node))
input)

View File

@@ -0,0 +1,334 @@
(ns hooks.metabase.agent-message-test
(:require
[clj-kondo.hooks-api :as hooks]
[clj-kondo.impl.utils]
[clojure.edn :as edn]
[clojure.test :refer :all]
[hooks.metabase.agent-message :as agent-message]))
(defn- findings
"Run `hook-fn` over `form`, a form or its source string, with the linters at `level` (unconfigured when nil) in
namespace `ns` of file `filename`; return the findings' messages."
([hook-fn form]
(findings hook-fn form :warning))
([hook-fn form level]
(findings hook-fn form level 'metabase.mcp.v2.tools.browse "src/metabase/mcp/v2/tools/browse.clj"))
([hook-fn form level ns filename]
(let [config (if level
{:linters {:metabase/agent-message-lines {:level level}
:metabase/agent-message-exit {:level level}}}
{})]
(binding [clj-kondo.impl.utils/*ctx* {:config config
:ignores (atom nil)
:findings (atom [])
:namespaces (atom {})}]
(let [input {:node (hooks/parse-string (if (string? form) form (pr-str form)))
:ns ns
:filename filename
:config config}
output (hook-fn input)]
(is (identical? (:node input) (:node output))
"the hook must return the node unchanged so Kondo's normal analysis still runs")
(mapv :message @(:findings clj-kondo.impl.utils/*ctx*)))))))
(def ^:private hook-cases
"`[hook form]` pairs that each report one finding when their linter is enabled."
[[agent-message/lint-msg '(msg lines x)]
[agent-message/lint-teaching-exit '(throw-teaching-error "x")]
[agent-message/lint-jsonrpc-error '(jsonrpc-error id -32602 "x")]
[agent-message/lint-success-content '(success-content "x")]
[agent-message/lint-ex-info "(ex-info \"x\" {:status-code 404})"]
[agent-message/lint-op-error '(op-error! idx "add_link" "x")]
[agent-message/lint-ellipsize '(ellipsize "x" 300)]
[agent-message/lint-list-content '(list-content data total {:empty-hint "x"})]])
(defn- linter-enabled-in-ns?
"Whether `config` enables `linter` in namespace `ns-name` through an ns-group's `:config-in-ns` entry."
[config linter ns-name]
(boolean
(some (fn [{:keys [pattern name]}]
(and (string? pattern)
(re-find (re-pattern pattern) ns-name)
(not= :off (get-in config [:config-in-ns name :linters linter :level] :off))))
(:ns-groups config))))
(deftest ^:parallel config-scope-test
(let [config (edn/read-string (slurp ".clj-kondo/config.edn"))]
(testing "GHY-4544: `msg` lines are checked in every OSS and enterprise MCP source namespace, and no test one"
(doseq [[ns-name enabled?] {"metabase.mcp.ui-resource" true
"metabase.mcp.transport" true
"metabase.mcp.v2.tools.browse" true
"metabase-enterprise.mcp.usage" true
"metabase-enterprise.mcp.v2.tools.browse" true
"metabase.mcp.ui-resource-test" false
"metabase.mcp.v2.tools.browse-test" false
"metabase.mcp.test-util" false
"metabase-enterprise.mcp.usage-test" false
"metabase-enterprise.mcp.v2.tools.browse-test" false
"metabase.queries.models.card" false}]
(testing ns-name
(is (= enabled? (linter-enabled-in-ns? config :metabase/agent-message-lines ns-name))))))
(testing "GHY-4544: exits are checked in MCP transport and v2 source namespaces, whose 4xx errors reach the agent"
(doseq [[ns-name enabled?] {"metabase.mcp.transport" true
"metabase.mcp.v2.common" true
"metabase.mcp.v2.tools.browse" true
"metabase-enterprise.mcp.v2.common" true
"metabase.mcp.settings" false
"metabase.mcp.validation" false
"metabase-enterprise.mcp.usage" false
"metabase.mcp.transport-test" false
"metabase.mcp.v2.tools.browse-test" false
"metabase-enterprise.mcp.v2.tools.browse-test" false
"metabase.queries.models.card" false}]
(testing ns-name
(is (= enabled? (linter-enabled-in-ns? config :metabase/agent-message-exit ns-name))))))
(testing "GHY-4544: `:empty-hint` is checked on `list-content` and on wrappers passing their options map to it"
(doseq [fn-sym '[metabase.mcp.v2.common/list-content
metabase.mcp.v2.tools.browse/paged-list-content]]
(testing fn-sym
(is (= 'hooks.metabase.agent-message/lint-list-content
(get-in config [:hooks :analyze-call fn-sym]))))))
(testing (str "GHY-4544: `check-execute-sql-enabled!` isn't checked, since it interpolates its subject into "
"its own `msg` and a plain string is the intended way to name the tool")
(is (nil? (get-in config [:hooks
:analyze-call
(symbol "metabase.mcp.v2.queries" "check-execute-sql-enabled!")]))))))
(deftest ^:parallel scope-test
(testing "GHY-4544: the hooks leave scoping to config, reporting in any namespace where their linter is enabled"
(doseq [[hook form] hook-cases
[ns filename] [['metabase.mcp.v2.tools.browse "src/metabase/mcp/v2/tools/browse.clj"]
['metabase.queries.models.card "src/metabase/queries/models/card.clj"]
['metabase.mcp.v2.tools.browse-test "test/metabase/mcp/v2/tools/browse_test.clj"]]]
(testing (str ns " " (pr-str form))
(is (= 1 (count (findings hook form :warning ns filename)))))))
(testing "nothing is reported when the linter is off or unconfigured"
(doseq [[hook form] hook-cases
level [:off nil]]
(testing (str level " " (pr-str form))
(is (empty? (findings hook form level)))))))
(deftest ^:parallel msg-test
(testing "GHY-4544: a vector of single-line string literals with matching arguments is accepted"
(is (empty? (findings agent-message/lint-msg
'(msg ["Table %s: %d of %d fields." "Continue with `offset: %d`."] t n total next)))))
(testing "specifiers that consume no argument aren't counted, and an explicit index can repeat an argument"
(is (empty? (findings agent-message/lint-msg '(msg ["100%% done."]))))
(is (empty? (findings agent-message/lint-msg '(msg ["%1$s, then %1$s again, then %<s"] x)))))
(testing "the lines must be a literal vector"
(is (=? [#".*vector.*"]
(findings agent-message/lint-msg '(msg lines x))))
(is (=? [#".*vector.*"]
(findings agent-message/lint-msg '(msg "one line" x)))))
(testing "a line may be split across a `str` of string literals"
(is (empty? (findings agent-message/lint-msg '(msg [(str "Table %s has " "%d fields.")] t n))))
(is (=? [#".*1 argument.*2.*"]
(findings agent-message/lint-msg '(msg [(str "only " "%s")] a b)))))
(testing "each line must be a string literal or a `str` of string literals"
(is (=? [#".*string literal or a `str` of string literals.*"]
(findings agent-message/lint-msg '(msg ["ok" line]))))
(is (=? [#".*string literal or a `str` of string literals.*"]
(findings agent-message/lint-msg '(msg [(str "split " x)])))))
(testing "a line can't contain a line break, control, or private-use character; each line is its own string"
(is (=? [#".*own string.*"]
(findings agent-message/lint-msg '(msg ["first\nsecond"]))))
(is (=? [#".*own string.*"]
(findings agent-message/lint-msg '(msg [(str "first\n" "second")]))))
(is (=? [#".*own string.*"]
(findings agent-message/lint-msg (list 'msg [(str "first" (char 0x2028) "second")]))))
(is (=? [#".*own string.*"]
(findings agent-message/lint-msg '(msg ["tab\tseparated"]))))
(is (=? [#".*own string.*"]
(findings agent-message/lint-msg '(msg ["done.%n"]))))
(testing "GHY-4544: a private-use character, which the truncation markers use, is flagged too"
(is (=? [#".*own string.*"]
(findings agent-message/lint-msg (list 'msg [(str "first" (char 0xE000) "second")]))))
(is (=? [#".*own string.*"]
(findings agent-message/lint-msg (list 'msg [(str "marked" (char 0xF8FF))]))))))
(testing "the argument count must match the specifiers across all lines"
(is (=? [#".*2 arguments.*1.*"]
(findings agent-message/lint-msg '(msg ["%s and" "%s"] a))))
(is (=? [#".*1 argument.*2.*"]
(findings agent-message/lint-msg '(msg ["only %s"] a b)))))
(testing (str "GHY-4544: a reference `String/format` can't resolve is flagged, since the runtime would fall back "
"to rendering every part quoted")
(testing "`%<` with no specifier before it to repeat, which counts as no argument at all"
(is (=? [#".*can't exist.*"]
(findings agent-message/lint-msg '(msg ["Value: %<s"]))))
(is (=? [#".*can't exist.*"]
(findings agent-message/lint-msg '(msg ["100%% done," "then %<d"])))))
(testing "an explicit index of zero, which is below the first argument"
(is (=? [#".*can't exist.*"]
(findings agent-message/lint-msg '(msg ["Value: %0$s"]))))
(is (=? [#".*can't exist.*"]
(findings agent-message/lint-msg '(msg ["Value: %0$s"] x)))))
(testing "a `%<` that repeats the first argument is still accepted"
(is (empty? (findings agent-message/lint-msg '(msg ["%s then %<s"] x)))))))
(deftest ^:parallel msg-specifier-test
(testing "GHY-4544: only `%s` with no flags but `<`, `%d` with flags and width, and `%%` are allowed"
(doseq [line ["%s" "%d" "%1$s" "%,d" "%05d" "%-8d" "%(d" "%1$,10d"
"%s again %<s" "%d again %<d" "%s (100%%)"]]
(testing line
(is (empty? (findings agent-message/lint-msg (list 'msg [(str "Value: " line)] 'x)))))))
(testing "GHY-4544: any other conversion, or a `%s` or `%d` that reshapes its value, is flagged"
(doseq [line ["%b" "%h" "%c" "%x" "%X" "%f" "%e" "%o" "%g" "%a" "%tY" "%TY" "%B"
"%S" "%.20s" "%10s" "%-5s" "%#s" "%1$.3s" "%-10S" "%.1f" "%.2d" "%1$.2d"]]
(testing line
(is (=? [#"A `msg` line may only use `%s`, `%d`, or `%%`; .* would alter or hide the value\."]
(findings agent-message/lint-msg (list 'msg [(str "Value: " line)] 'x)))))))
(testing "GHY-4544: a `%` that doesn't begin a valid format specifier is flagged, since `String/format` rejects it"
(doseq [line ["Braces %}" "Dot %." "Trailing %" "Space % " "Unknown %q" "Bad date %tq" "%" "%1$"
;; GHY-4544: an argument index too large to be one is malformed, not an arithmetic overflow
"%999999999999999999999999$s"]]
(testing (pr-str line)
(is (=? [#"A `%` in a `msg` line must begin a format specifier.*`%%`.*"]
(findings agent-message/lint-msg (list 'msg [line]))))))
(testing "an index at the digit cap still parses, and is reported as an argument count"
(is (=? [#".*999999999 arguments.*"]
(findings agent-message/lint-msg '(msg ["%999999999$s"] x))))))
(testing "valid specifiers, including an escaped percent sign, aren't flagged as malformed"
(doseq [form ['(msg ["100%% done"])
'(msg ["%,d rows"] n)
'(msg ["%1$s, then %<s"] x)
'(msg [(str "50%" "% of %s")] x)]]
(testing (pr-str form)
(is (empty? (findings agent-message/lint-msg form)))))))
(def ^:private stringy-texts
"Message arguments built as text rather than with `msg`."
['"Query failed."
'(str "Query failed: " err)
'(format "Table %s not found" (pr-str t))
'(cond-> (json/encode body) message (str "\n" message))
'(if x "a" (str "b" y))
'(if x (msg ["a"]) (str "b" y))
'(when-let [e (:error r)] (str/join ", " e))
'(or (:message r) (tru "Unknown error"))
'(let [s (pr-str x)] (str "Bad value " s))
'(cond (:a x) (msg ["a"]) :else "b")
'(condp = x :a (str "a" y) (msg ["b"]))
'(condp = x :a (msg ["a"]) "b")
'(case x :a (msg ["a"]) "b")
'(case x :a "a" :b (msg ["b"]))
'(->> bad (map name) (str/join ", "))
'(-> x (str " suffix"))
'(-> x name str)
'(some-> e ex-message)
'(some->> xs (map name) (clojure.string/join ", "))
'(cond->> xs x (map name) y (str/join ", "))
'(cond-> "x" y (subs 1))
'(as-> x v (assoc v :a 1) (json/encode v))
'(doto (str "a" b) println)
'(let [s x] (-> s (format y)))])
(def ^:private message-texts
"Message arguments that aren't syntactically text."
['(msg ["Table %s not found."] t)
'message
'(if x (msg ["a"]) (msg ["b"]))
'(render-error e)
'(cond (str x) (msg ["a"]) :else (msg ["b"]))
'(condp = x "a" (msg ["a"]) (msg ["b"]))
'(condp get x "a" :>> render-error (msg ["b"]))
'(case x "a" (msg ["a"]) ("b" "c") (msg ["b"]))
'(-> "x" count)
'(->> "x" (map str) set)
'(some-> (str "x") render-error)
'(-> x)
'(-> e :message)
'(cond-> {:a 1} x (assoc :b "c"))
'(as-> "x" v (count v))
'(doto x (println "a"))])
(deftest ^:parallel teaching-exit-test
(doseq [fn-name ["throw-teaching-error" "error-content"]]
(testing (str "GHY-4544: text passed to `" fn-name "` is flagged")
(doseq [text stringy-texts]
(testing (pr-str text)
(is (=? [(re-pattern (str ".*passes text to `" fn-name "`.*`msg`.*"))]
(findings agent-message/lint-teaching-exit (list (symbol "common" fn-name) text)))))))
(testing (str "`" fn-name "` accepts a `msg`, a symbol, and other calls")
(doseq [text message-texts]
(testing (pr-str text)
(is (empty? (findings agent-message/lint-teaching-exit
(list (symbol fn-name) text {:status-code 404}))))))))
(testing "only the message argument is checked"
(is (empty? (findings agent-message/lint-teaching-exit '(error-content (msg ["x"]) (str "code"))))))
(testing "nothing is reported when the linter is off"
(is (empty? (findings agent-message/lint-teaching-exit '(throw-teaching-error "x") :off)))))
(def ^:private helper-cases
"`[hook fn-name form-fn]`: a message-taking helper's hook, the name its finding cites, and a function of a message
returning a call that passes it to the helper."
[[agent-message/lint-op-error "op-error!" #(list 'dashboard-ops/op-error! 'idx "add_link" %)]
[agent-message/lint-ellipsize "ellipsize" #(list 'common/ellipsize % 300)]
[agent-message/lint-list-content "list-content" #(list 'common/list-content 'rows 0 {:offset 0 :empty-hint %})]
[agent-message/lint-list-content "paged-list-content"
#(list 'paged-list-content 'args 'rows {:empty-hint %} 'identity)]])
(deftest ^:parallel message-helper-test
(doseq [[hook fn-name form-fn] helper-cases]
(testing (str "GHY-4544: text passed as `" fn-name "`'s message is flagged")
(doseq [text stringy-texts]
(testing (pr-str text)
(is (=? [(re-pattern (str ".*passes text to `" fn-name "`.*`msg`.*"))]
(findings hook (form-fn text)))))))
(testing (str "`" fn-name "` accepts a `msg`, a symbol, and other calls as its message")
(doseq [text message-texts]
(testing (pr-str text)
(is (empty? (findings hook (form-fn text))))))))
(testing "only the message argument is checked"
(is (empty? (findings agent-message/lint-op-error '(op-error! (str "0") (str "op") (msg ["x"])))))
(is (empty? (findings agent-message/lint-ellipsize '(ellipsize (msg ["x"]) (str "300")))))
(is (empty? (findings agent-message/lint-list-content '(list-content (str "data") 0 {:param (str "x")}))))
(is (empty? (findings agent-message/lint-list-content '(list-content data 0 opts))))
(is (empty? (findings agent-message/lint-list-content '(list-content data 0))))))
(deftest ^:parallel jsonrpc-error-test
(testing "GHY-4544: text passed as the JSON-RPC error message is flagged"
(doseq [text stringy-texts]
(testing (pr-str text)
(is (=? [#".*passes text to `jsonrpc-error`.*`msg`.*"]
(findings agent-message/lint-jsonrpc-error (list 'transport/jsonrpc-error 'id -32602 text)))))))
(testing "a `msg`, symbol, or other call as the message is accepted"
(doseq [text message-texts]
(testing (pr-str text)
(is (empty? (findings agent-message/lint-jsonrpc-error (list 'jsonrpc-error 'id -32602 text)))))))
(testing "only the third argument is the message"
(is (empty? (findings agent-message/lint-jsonrpc-error '(jsonrpc-error "id" (str "code") (msg ["x"])))))))
(deftest ^:parallel success-content-test
(testing "GHY-4544: text passed to `success-content` is flagged"
(doseq [text stringy-texts]
(testing (pr-str text)
(is (=? [#"`success-content` takes data to JSON-encode or a `msg`.*"]
(findings agent-message/lint-success-content (list 'common/success-content text)))))))
(testing "a payload, a `msg`, or a symbol is accepted"
(doseq [text (conj message-texts '{:data rows} '[a b])]
(testing (pr-str text)
(is (empty? (findings agent-message/lint-success-content (list 'success-content text))))))
(is (empty? (findings agent-message/lint-success-content
'(success-content {:data rows} (str "structured")))))))
(deftest ^:parallel ex-info-test
(testing "GHY-4544: an ex-info carrying a 4xx status code or an error code is flagged"
(doseq [form ["(ex-info \"Card not found\" {:status-code 404})"
"(ex-info (str \"Bad \" x) {:status-code 400 :field f})"
"(ex-info \"Nope\" {::common/error-code c})"
"(ex-info \"Nope\" {:error-code c})"
"(ex-info (msg [\"Nope\"]) {:error-code c})"]]
(testing form
(is (=? [#".*`throw-teaching-error` and a `msg`.*"]
(findings agent-message/lint-ex-info form))))))
(testing "other ex-info calls aren't flagged"
(doseq [form ["(ex-info \"boom\" {:status-code 500})"
"(ex-info \"boom\" {:status-code code})"
"(ex-info \"boom\" {:foo 1})"
"(ex-info \"boom\" data)"
"(ex-info \"boom\" (merge {:status-code 400} data))"
"(ex-info \"boom\")"]]
(testing form
(is (empty? (findings agent-message/lint-ex-info form))))))
(testing "nothing is reported when the linter is off"
(is (empty? (findings agent-message/lint-ex-info "(ex-info \"x\" {:status-code 404})" :off)))))

View File

@@ -0,0 +1,25 @@
(ns hooks.metabase.app-db.schema-migrations-test.impl-test
(:require
[clj-kondo.hooks-api :as hooks]
[clj-kondo.impl.utils]
[clojure.test :refer :all]
[hooks.metabase.app-db.schema-migrations-test.impl]))
(defn- finding-types!
[ns-symb]
(binding [clj-kondo.impl.utils/*ctx* {:config {:linters {:metabase/validate-mb-app-db-migrations-test {:level :warning}}}
:ignores (atom nil)
:findings (atom [])
:namespaces (atom {})}]
(hooks.metabase.app-db.schema-migrations-test.impl/test-migrations!
{:ns ns-symb
:node (hooks/parse-string (pr-str '(impl/test-migrations ["v64.2026-09-01T00:00:00"] [migrate!] (migrate!))))})
(mapv :type @(:findings clj-kondo.impl.utils/*ctx*))))
(deftest ^:synchronized require-app-db-migrations-test-tag-test
(testing "a namespace that runs migration tests must be tagged, so CI shards it with the other migration tests"
(is (= [:metabase/validate-mb-app-db-migrations-test]
(finding-types! 'metabase.some-migrations-test))))
(testing "a tagged namespace is fine"
(is (= []
(finding-types! (with-meta 'metabase.some-migrations-test {:mb/app-db-migrations-test true}))))))

View File

@@ -5,8 +5,15 @@
[clojure.test :refer :all]
[hooks.metabase.toucan.db-ns :as toucan.db-ns]))
(defn- confinement-findings
"Only the `t2-query-namespace` findings, so these tests are not affected by the query-hygiene
linter that shares this hook."
[findings]
(filter #(= :metabase/t2-query-namespace (:type %)) findings))
(defn- lint-query-call [form ns-sym & [filename modules]]
(binding [clj-kondo.impl.utils/*ctx* {:config {:linters {:metabase/t2-query-namespace {:level :warning}}}
(binding [clj-kondo.impl.utils/*ctx* {:config {:linters {:metabase/t2-query-namespace {:level :warning}
:metabase/unsafe-app-db-query {:level :warning}}}
:ignores (atom nil)
:findings (atom [])
:namespaces (atom {})}]
@@ -25,11 +32,11 @@
:message #".*`t2/select-one`.*"}]
(lint-query-call '(t2/select-one :model/Card :id 1) 'metabase.queries.models.card))))
(testing "metabase.<module>.db is allowed"
(is (empty? (lint-query-call '(t2/select-one :model/Card :id 1) 'metabase.queries.db))))
(is (empty? (confinement-findings (lint-query-call '(t2/select-one :model/Card :id 1) 'metabase.queries.db)))))
(testing "metabase-enterprise.<module>.db is allowed"
(is (empty? (lint-query-call '(t2/select-one :model/Card :id 1) 'metabase-enterprise.sandbox.db))))
(is (empty? (confinement-findings (lint-query-call '(t2/select-one :model/Card :id 1) 'metabase-enterprise.sandbox.db)))))
(testing "metabase.driver.<driver>.db is allowed for driver modules"
(is (empty? (lint-query-call '(t2/select-one :model/Database :id 1) 'metabase.driver.bigquery-cloud-sdk.db))))
(is (empty? (confinement-findings (lint-query-call '(t2/select-one :model/Database :id 1) 'metabase.driver.bigquery-cloud-sdk.db)))))
(testing "a nested db namespace is not a module db namespace"
(is (=? [{:type :metabase/t2-query-namespace}]
(lint-query-call '(t2/query {:select [:*]}) 'metabase.queries.models.db))))
@@ -71,3 +78,247 @@
(lint-query-call '(t2/query {:select [:*]}) 'metabase.metabot.llm.models.db nil modules))))
(testing "enterprise modules resolve through the metabase-enterprise root"
(is (empty? (lint-query-call '(t2/select :model/Card) 'metabase-enterprise.sandbox.db nil modules))))))
(deftest ^:parallel unmarked-value-in-a-db-namespace-test
(testing "a symbol reaching a value slot is flagged"
(is (=? [{:type :metabase/unsafe-app-db-query
:message #"`locale` reaches a SQL value slot unmarked.*"}]
(lint-query-call '(t2/select :model/X {:where [:= :locale locale]}) 'metabase.foo.db))))
(testing "a marked value is not flagged"
(is (empty? (lint-query-call '(t2/select :model/X {:where [:= :locale [:auto/param locale]]})
'metabase.foo.db))))
(testing "a literal cannot carry a request value and is not flagged"
(is (empty? (lint-query-call '(t2/select :model/X {:where [:= :locale "de"]}) 'metabase.foo.db))))
(testing "a column reference is not a value"
(is (empty? (lint-query-call '(t2/select :model/X {:where [:= :a.id :b.id]}) 'metabase.foo.db))))
(testing "values nested under a boolean connective are reached"
(is (=? [{:message #"`b`.*"}]
(lint-query-call '(t2/select :model/X {:where [:and [:= :x [:auto/param a]] [:= :y b]]})
'metabase.foo.db))))
(testing "only db namespaces are linted for unmarked values"
(is (empty? (filter #(= :metabase/unsafe-app-db-query (:type %))
(lint-query-call '(t2/select :model/X {:where [:= :locale locale]})
'metabase.foo.models.thing)))))
(testing "a test source tree is exempt"
(is (empty? (lint-query-call '(t2/select :model/X {:where [:= :locale locale]})
'metabase.foo.db "test/metabase/foo/db_test.clj")))))
(deftest ^:parallel kv-arg-style-is-not-flagged-test
(testing "call style is not linted: a query written as :column value pairs is left alone"
;; Extraction to a .sql file is semantic rather than a transcription of the HoneySQL shape, so a
;; query map is no closer to that target than kv-args are. Converting also skips Toucan's type
;; transforms, which only run in kv-arg position -- that bug shipped twice.
(are [form] (empty? (filter #(re-find #"Pass this query a map" (str (:message %)))
(lint-query-call form 'metabase.foo.db)))
'(t2/select :model/X :locale [:auto/param locale])
'(t2/select :model/X :archived false)
'(t2/select-one :model/X :id (long id))
'(t2/select-one-fn :value :model/Setting :key [:auto/param k])
'(t2/select :model/X {:where [:= :locale [:auto/param locale]]})))
(testing "an unmarked value in a kv-arg is still flagged -- only the style check went away"
(is (=? [{:type :metabase/unsafe-app-db-query, :message #".*reaches a SQL value slot unmarked.*"}]
(lint-query-call '(t2/select :model/X :locale locale) 'metabase.foo.db)))))
(deftest ^:parallel conditions-map-values-are-linted-test
(testing "a conditions map -- a shape the sweep leaves in place -- is checked"
;; `(t2/update! model {:col v} changes)` filters on v, but it is not a `:where` clause, so
;; neither the query-map walker nor the kv-arg walker used to see it. A namespace written this
;; way passed the lint vacuously.
(are [form] (=? [{:type :metabase/unsafe-app-db-query
:message #".*reaches a SQL value slot unmarked.*"}]
(lint-query-call form 'metabase.foo.db))
'(t2/update! :model/X {:key k} {:v 1})
'(t2/delete! :model/X {:key k})
'(t2/update! :model/X {:key [:in ks]} {:v 1})))
(testing "a marked or coerced conditions value is not flagged"
(are [form] (empty? (lint-query-call form 'metabase.foo.db))
'(t2/update! :model/X {:key [:auto/param k]} {:v 1})
'(t2/update! :model/X {:id (long id)} {:v 1})))
(testing "only the map right after the model is conditions -- the changes map is exempt"
;; rule 1: a written value must arrive as itself, so it is never flagged.
(is (empty? (lint-query-call '(t2/update! :model/X {:id (long id)} {:v written})
'metabase.foo.db)))))
(deftest ^:parallel subquery-values-are-linted-test
(testing "a value inside a subquery in a value slot is reached"
(is (=? [{:type :metabase/unsafe-app-db-query
:message #".*`z`.*reaches a SQL value slot unmarked.*"}]
(lint-query-call
'(t2/select :model/X {:where [:in :id {:select [:id] :from [:y] :where [:= :z z]}]})
'metabase.foo.db))))
(testing "the plain token case still works -- descending must not replace it"
(is (=? [{:message #"`v`.*reaches a SQL value slot unmarked.*"}]
(lint-query-call '(t2/select :model/X {:where [:= :k v]}) 'metabase.foo.db)))))
(deftest ^:parallel conditionally-built-clause-test
(testing "a value slot inside a conditional form is still reached"
;; `value-nodes` used to stop at a list node, so anything a `when`/`if`/`cond->` built was
;; invisible -- a false negative in a security lint. permissions/db.clj and collections/db.clj
;; both build clauses this way.
(are [form] (=? [{:type :metabase/unsafe-app-db-query
:message #".*reaches a SQL value slot unmarked.*"}]
(lint-query-call form 'metabase.foo.db))
'(t2/select :model/X {:where [:and (when k [:= :key k])]})))
(testing "both branches of an if are reached"
(is (= 2 (count (lint-query-call
'(t2/select :model/X {:where [:and (if flag [:= :key k] [:= :key other-k])]})
'metabase.foo.db)))))
(testing "a marked value inside a conditional form is not flagged"
(is (empty? (lint-query-call
'(t2/select :model/X {:where [:and (when k [:= :key [:auto/param k]])]})
'metabase.foo.db)))))
(deftest ^:parallel kv-arg-values-are-linted-test
(testing "an unmarked value in a kv-arg pair is flagged, so dropping the style check left no gap"
(are [form] (=? [{:type :metabase/unsafe-app-db-query
:message #".*reaches a SQL value slot unmarked.*"}]
(lint-query-call form 'metabase.foo.db))
'(t2/select :model/X :locale locale)
'(t2/select-one :model/X :id id)
;; the pairs do not start at a fixed offset -- :value comes before the model here
'(t2/select-one-fn :value :model/Setting :key k)
;; a value inside an operator form is reached, rather than the form being treated as a value
'(t2/select :model/X :id [:in ids])))
(testing "a marked or coerced kv-arg value is not flagged"
(are [form] (empty? (lint-query-call form 'metabase.foo.db))
'(t2/select :model/X :locale [:auto/param locale])
'(t2/select-one :model/X :id (long id))
'(t2/select :model/X :id [:in (mapv long ids)])
'(t2/select :model/X :id [:in [:auto/param ids]])))
(testing "a literal kv-arg value is not flagged"
(are [form] (empty? (lint-query-call form 'metabase.foo.db))
'(t2/select :model/X :archived false)
'(t2/select :model/X :status "pending")
'(t2/select :model/X :engine :appdb)))
(testing "one unmarked value per pair is reported"
(is (= 2 (count (lint-query-call '(t2/select :model/X :locale locale :msgid msgid)
'metabase.foo.db)))))
(testing "a call mixing kv-args with a trailing query map reports each value exactly once"
;; The two walkers run over the same arguments, so this pins that they do not double-report.
(is (= 2 (count (lint-query-call
'(t2/select :model/X :locale locale {:where [:= :other other]})
'metabase.foo.db))))))
(deftest ^:parallel write-calls-are-not-linted-for-values-test
(testing "an insert's values are written, not filtered on, so they are not flagged"
(are [form] (empty? (filter #(= :metabase/unsafe-app-db-query (:type %))
(lint-query-call form 'metabase.foo.db)))
'(t2/insert! :model/X :key k :value v)
'(t2/insert-returning-instances! :model/X :key k :value v)))
(testing "a select's values are still flagged"
(is (=? [{:type :metabase/unsafe-app-db-query}]
(lint-query-call '(t2/select :model/X :key k) 'metabase.foo.db)))))
(deftest ^:parallel operator-arity-test
(testing "a value operator with an unexpected arity still has its args examined"
(is (seq (lint-query-call '(t2/select :model/X {:where [:= :a b c]}) 'metabase.foo.db)))
(is (seq (lint-query-call '(t2/select :model/X {:where [:between :a lo hi]}) 'metabase.foo.db)))))
(deftest ^:parallel operator-coverage-test
(testing "comparison operators HoneySQL accepts as aliases for not= are checked"
;; The set used to list only `not=`, so `[:!= :col v]` -- live in collections/db.clj -- was
;; silently unchecked even though HoneySQL binds it as `<>`.
(are [form] (=? [{:message #"`v`.*reaches a SQL value slot unmarked.*"}]
(lint-query-call form 'metabase.foo.db))
'(t2/select :model/X {:where [:!= :k v]})
'(t2/select :model/X {:where [:<> :k v]})
'(t2/select :model/X {:where [:is-distinct-from :k v]})))
(testing "a two-value operator reports both values"
(is (= 2 (count (lint-query-call '(t2/select :model/X {:where [:not-between :k lo hi]})
'metabase.foo.db)))))
(testing "a value wrapped in a function-call form is reached"
;; `[:lower v]` compiles to `LOWER(?)`, so `v` is a real bind slot.
(are [form] (seq (lint-query-call form 'metabase.foo.db))
'(t2/select :model/X {:where [:= :k [:lower v]]})
'(t2/select :model/X {:where [:= :k [:cast v :text]]})
'(t2/select :model/X :k [:lower v])))
(testing "values in a literal collection are reached"
;; `[:in :k [a b]]` binds each element -- `IN (?, ?)`.
(is (= 2 (count (lint-query-call '(t2/select :model/X {:where [:in :k [a b]]})
'metabase.foo.db))))))
(deftest ^:parallel no-duplicate-findings-test
(testing "a multi-arg operator in a kv-arg reports each value exactly once"
;; value-nodes misreads `[:between lo hi]` as a clause (lo as column), while
;; kv-arg-value-nodes reads it correctly, so the walkers overlap on `hi`. Deduping by source
;; position makes that harmless.
(is (= 2 (count (lint-query-call '(t2/select :model/X :d [:between lo hi]) 'metabase.foo.db))))
(is (= 2 (count (lint-query-call '(t2/delete! :model/X {:d [:between lo hi]})
'metabase.foo.db)))))
(testing "a query map passed where conditions go reports each value once"
;; `{:where ...}` is a query map that value-nodes already walks; treating it as conditions too
;; reported everything twice. Live in mcp/db.clj, metabot/db.clj, testing_api/db.clj.
(is (= 1 (count (lint-query-call '(t2/delete! :model/X {:where [:= :k k]}) 'metabase.foo.db))))
(is (= 2 (count (lint-query-call '(t2/update! :model/X {:where [:and [:= :a a] [:= :b b]]} {:v 1})
'metabase.foo.db))))))
(deftest ^:parallel two-arity-update-changes-map-test
(testing "update!'s lone trailing map is CHANGES, not conditions, so its values are not flagged"
;; Toucan's arglist ends in the changes map. remote_sync/db.clj's mark-all-rsos-synced! is this
;; shape; flagging `written` would ask for a marker on a written value, which the column's
;; `:in` transform stores as data.
(is (empty? (lint-query-call '(t2/update! :model/X {:v written}) 'metabase.foo.db))))
(testing "delete! has no changes map, so its lone map is always conditions"
(is (=? [{:message #"`k`.*"}]
(lint-query-call '(t2/delete! :model/X {:k k}) 'metabase.foo.db))))
(testing "update!'s FIRST map is still conditions when a changes map follows"
(is (=? [{:message #"`k`.*"}]
(lint-query-call '(t2/update! :model/X {:k k} {:v 1}) 'metabase.foo.db)))))
(deftest ^:parallel identifier-clauses-are-not-value-slots-test
(testing "a symbol naming a column or a table is not reported"
;; `value-guard` refuses a marker in these clauses, so reporting one asks for a fix that throws
;; at compile. data_studio/db.clj, task_history/db.clj and permissions/db.clj are written this
;; way. `:select` is the exception: index 0 of its entries is an expression, so it has its own
;; test in expression-entry-clauses-test.
(are [form] (empty? (lint-query-call form 'metabase.foo.db))
'(t2/query {:select [:*] :from [[union-query :subquery]]})
'(t2/query {:select [:*] :from [:t] :order-by [[:sort_key sort-direction]]})
'(t2/query {:select [:*] :from [:t] :group-by [group-column]})))
(testing "a value nested inside an identifier clause is still reported"
(are [form] (=? [{:message #"`v`.*reaches a SQL value slot unmarked.*"}]
(lint-query-call form 'metabase.foo.db))
;; a computed projection is a real comparison
'(t2/query {:select [[[:= :engine v] :is_match]] :from [:t]})
;; so is a CASE sort key
'(t2/query {:select [:*] :from [:t] :order-by [[[:case [:= :a v] 1 :else 2] :asc]]})
;; a subquery carries its own where clause
'(t2/query {:select [:*] :from [[{:select [:*] :from [:u] :where [:= :k v]} :s]]}))))
(deftest ^:parallel expression-entry-clauses-test
(testing "index 0 of an entry is a value slot, because HoneySQL binds a parameter there"
;; `{:select [[[:auto/param "a"] :k]]}` compiles to `SELECT ? AS "K"`, so `value-guard` accepts
;; a marker there. Treating the whole clause as names would hide a request value.
(are [form] (=? [{:type :metabase/unsafe-app-db-query
:message #"`v`.*reaches a SQL value slot unmarked.*"}]
(lint-query-call form 'metabase.foo.db))
'(t2/query {:select [[v :label]] :from [:t]})
'(t2/query {:select-distinct [[v :label]] :from [:t]})
'(t2/query {:insert-into [:t] :returning [[v :label]]})))
(testing "the alias, a bare entry and a table entry are names"
;; `value-guard` refuses a marker in each of these, so asking for one would break the query.
(are [form] (empty? (lint-query-call form 'metabase.foo.db))
'(t2/query {:select [[:key alias-sym]] :from [:t]})
'(t2/query {:select [cols] :from [:t]})
'(t2/query {:select [:*] :from [[tbl :x]]})))
(testing "a computed projection and a subquery entry still reach their values"
(are [form] (=? [{:message #"`v`.*"}] (lint-query-call form 'metabase.foo.db))
'(t2/query {:select [[[:= :engine v] :is_match]] :from [:t]})
'(t2/query {:select [:*] :from [[{:select [:*] :from [:u] :where [:= :k v]} :s]]}))))
(deftest ^:parallel clause-context-survives-assoc-test
(testing "a clause key in an assoc call gives its value the clause context back"
;; task_history/db.clj:174 and five other namespaces build `:order-by` this way. Without this
;; the lint asks for a marker on a column name and on a sort direction, and the fix the message
;; names loses the ordering with no error.
(are [form] (empty? (lint-query-call form 'metabase.foo.db))
'(t2/query (assoc q :order-by [[sort-column sort-direction] [:id :desc]]))
'(t2/query (cond-> q flag (assoc :order-by [[sort-column sort-direction]])))
'(t2/query (assoc q :group-by [group-column]))
'(t2/query (assoc q :select [[:key alias-sym]]))))
(testing "a value clause built the same way is still reported"
(is (=? [{:message #"`v`.*"}]
(lint-query-call '(t2/query (assoc q :where [:= :k v])) 'metabase.foo.db))))
(testing "an expression entry built the same way is still reported"
(is (=? [{:message #"`v`.*"}]
(lint-query-call '(t2/query (assoc q :select [[v :label]])) 'metabase.foo.db)))))

View File

@@ -2,7 +2,7 @@
// imported, vs the *allowed* edges in module-boundaries.mjs). Consumed by
// .github/scripts/create-test-plan.ts.
//
// bunx depcruise frontend/src enterprise/frontend/src \
// bunx depcruise frontend/src enterprise/frontend/src .storybook/preview.tsx \
// --config .dependency-cruiser.cjs --output-type json > dependency-graph.json
/** @type {import('dependency-cruiser').IConfiguration} */
@@ -10,10 +10,13 @@ module.exports = {
options: {
// Keep only edges between our own source files (drops node_modules etc.),
// which keeps the JSON small.
includeOnly: "^(frontend/src|enterprise/frontend/src)/",
includeOnly: "^(frontend/src|enterprise/frontend/src|\\.storybook)/",
doNotFollow: { path: "node_modules" },
// No rules here, so the cycle and orphan analysis has no consumer and only costs time.
skipAnalysisNotInRules: true,
// Resolve the `"*": ["./frontend/src/*", ...]` aliases so bare imports like
// `metabase/lib/foo` map back to a real file.
tsConfig: { fileName: "tsconfig.json" },

View File

@@ -121,6 +121,9 @@ MB_EDITION="ee"
# Frontend dev server port (for hot reload). Must match the port used by `yarn build-hot`.
# MB_FRONTEND_DEV_PORT=8080
# Frontend dev server bind address. Defaults to 127.0.0.1; set 0.0.0.0 to reach it from other devices.
# MB_FRONTEND_DEV_HOST=127.0.0.1
# 3.2 E2E Local Runner Setup
# ------------------------------------------------------------------------------
@@ -161,3 +164,46 @@ TZ="US/Pacific"
# DO NOT USE Docker Desktop. Use Orbstack instead. Metabase will fail to connect
# to the dockerized Mongo QA database otherwise.
EXPERIMENTAL_DOCKER_DESKTOP_FORCE_QEMU=1
# ==============================================================================
# 4. BACKEND DRIVER TESTS
# ==============================================================================
#
# Connection details for running backend tests against a driver, e.g.
# ./bin/test-agent --drivers=snowflake :only '[metabase.foo-test]'
# 4.1 BigQuery
# ------------------------------------------------------------------------------
# Service account with BigQuery access in the project. The test user creates and drops the test-data datasets itself.
# https://cloud.google.com/iam/docs/keys-create-delete
# MB_BIGQUERY_CLOUD_SDK_TEST_PROJECT_ID="<project-id>"
# MB_BIGQUERY_CLOUD_SDK_TEST_SERVICE_ACCOUNT_JSON='{"type": "service_account", ...}'
# Only for database routing tests
# MB_BIGQUERY_CLOUD_SDK_TEST_SERVICE_ACCOUNT_JSON_ROUTING='{"type": "service_account", ...}'
# 4.2 Redshift
# ------------------------------------------------------------------------------
# Test data is loaded into a unique schema per test run.
# MB_REDSHIFT_TEST_HOST="<cluster>.<id>.<region>.redshift.amazonaws.com"
# Comma-separated hosts to pick one at random from, instead of MB_REDSHIFT_TEST_HOST
# MB_REDSHIFT_TEST_HOSTS=""
# MB_REDSHIFT_TEST_PORT=5439
# MB_REDSHIFT_TEST_DB="testdb"
# MB_REDSHIFT_TEST_USER="metabase_ci"
# MB_REDSHIFT_TEST_PASSWORD="<password>"
# Only for database routing tests
# MB_REDSHIFT_TEST_DB_ROUTING="dev"
# 4.3 Snowflake
# ------------------------------------------------------------------------------
# Key-pair authentication. The test user creates and drops the test-data databases itself.
# https://docs.snowflake.com/en/user-guide/key-pair-auth
# MB_SNOWFLAKE_TEST_ACCOUNT="<orgname>-<accountname>"
# MB_SNOWFLAKE_TEST_USER="<username>"
# MB_SNOWFLAKE_TEST_PRIVATE_KEY="-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----"
# MB_SNOWFLAKE_TEST_WAREHOUSE="<warehouse>"
# MB_SNOWFLAKE_TEST_ADDITIONAL_OPTIONS=""

View File

@@ -22,6 +22,9 @@ runs:
# `awk 'NR<=25'` rather than `head -25`: the runner ignores SIGPIPE, so `head` exiting early
# leaves `sort` with an EPIPE write error and `pipefail` fails the step. `|| true` covers `grep`
# finding no junit files.
#
# `-r --include` rather than a `target/junit/*.xml` glob: a multi-JVM run nests one directory
# per partition under `target/junit`, and a non-recursive glob matches none of it.
- name: Summarize test wall times
if: ${{ inputs.profiled == 'true' }}
continue-on-error: true
@@ -30,13 +33,13 @@ runs:
{
echo "### Slowest test namespaces (seconds)"
echo '```'
grep -h '<testsuite' target/junit/*.xml 2>/dev/null \
grep -rh --include='*.xml' '<testsuite' target/junit 2>/dev/null \
| sed -E 's|.*<testsuite name="([^"]*)" time="([^"]*)".*|\2 \1|' \
| sort -rn | awk 'NR<=25' || true
echo '```'
echo "### Slowest tests (seconds)"
echo '```'
grep -h '<testcase' target/junit/*.xml 2>/dev/null \
grep -rh --include='*.xml' '<testcase' target/junit 2>/dev/null \
| sed -E 's|.*classname="([^"]*)" name="([^"]*)" time="([^"]*)".*|\3 \1/\2|' \
| sort -rn | awk 'NR<=25' || true
echo '```'
@@ -53,9 +56,12 @@ runs:
uses: actions/upload-artifact@v7
with:
name: ${{ inputs.artifact-name }}
# Globs, because a multi-JVM run nests its junit XML one directory per partition and
# names each log `test-log-p<index>.json`. The flamegraphs stay flat, with that same
# `p<index>-` prefix.
path: |
target/clj-async-profiler/results/*flamegraph.html
target/junit/*.xml
logs/test-log.json
target/junit/**/*.xml
logs/test-log*.json
retention-days: 14
if-no-files-found: warn

View File

@@ -85,17 +85,27 @@ runs:
git checkout ${{ github.sha }}
- name: Approve and Merge the PR
# Approving takes a different token than the one that opened the PR: GitHub won't let a token
# approve its own pull request.
- name: Approve the backport PR
if: ${{ steps.create_backport_pull_request.outputs.has-conflicts == 'false' }}
run: |
gh pr review --approve "$PR_NUMBER"
gh pr merge --squash --auto "$PR_NUMBER"
run: gh pr review --approve "$PR_NUMBER"
shell: bash
env:
# uses a different token than the token that created the PR so we can approve and merge it
GH_TOKEN: ${{ inputs.github-token-2 }}
PR_NUMBER: ${{ steps.create_backport_pull_request.outputs.backport_pr_number }}
# Merging, though, has to use `github-token` rather than the `GITHUB_TOKEN` that approves: GitHub
# starts no workflow run for a push made with `GITHUB_TOKEN`, so an auto-merge enabled with it
# lands the backport on the release branch with no CI at all.
- name: Merge the backport PR
if: ${{ steps.create_backport_pull_request.outputs.has-conflicts == 'false' }}
run: gh pr merge --squash --auto "$PR_NUMBER"
shell: bash
env:
GH_TOKEN: ${{ inputs.github-token }}
PR_NUMBER: ${{ steps.create_backport_pull_request.outputs.backport_pr_number }}
- uses: ./.github/actions/notify-pull-request
if: ${{ steps.create_backport_pull_request.outputs.has-conflicts == 'true' }}
with:

View File

@@ -1,5 +1,5 @@
name: Create test plan
description: Cruise the frontend import graph and compute the affected-tests plan.
description: Compute the affected-tests plan for the changed files.
inputs:
changed-files:
@@ -28,21 +28,15 @@ outputs:
stats:
description: FE Affected Tests stats row (JSON).
value: ${{ steps.compute.outputs.stats }}
fe_unit_specs_to_run:
description: Unit specs to run (JSON array).
value: ${{ steps.compute.outputs.fe_unit_specs_to_run }}
loki_stories_to_run:
description: Loki stories to run (JSON array).
value: ${{ steps.compute.outputs.loki_stories_to_run }}
e2e_specs_to_run:
description: E2E specs to run (JSON array).
value: ${{ steps.compute.outputs.e2e_specs_to_run }}
artifact-id:
description: Artifact containing test-plan.json for this run.
value: ${{ steps.upload.outputs.artifact-id }}
runs:
using: composite
steps:
- name: Prepare Node + Bun
uses: ./.github/actions/prepare-node-bun
- name: Prepare front-end environment
uses: ./.github/actions/prepare-frontend
- name: Resolve closest-ancestor e2e coverage manifest
id: manifest
uses: actions/github-script@v9
@@ -61,16 +55,6 @@ runs:
headSha: context.payload.pull_request?.head?.sha ?? context.sha,
});
core.setOutput("path", result?.path ?? "");
- name: Build dependency graph
shell: bash
run: |
# Quiet on success; surface output only if install fails.
bun install --frozen-lockfile > /tmp/bun-install.log 2>&1 \
|| { echo "bun install issues (continuing):"; tail -20 /tmp/bun-install.log; }
bunx depcruise frontend/src enterprise/frontend/src \
--config .dependency-cruiser.cjs \
--output-type json > dependency-graph.json || true
- name: Create test plan
id: compute
shell: bash
@@ -82,15 +66,20 @@ runs:
LOKI_INFRA_TOUCHED: ${{ inputs.loki-infra-touched }}
E2E_INFRA_TOUCHED: ${{ inputs.e2e-infra-touched }}
SHARED_SOURCES_TOUCHED: ${{ inputs.shared-sources-touched }}
DEP_GRAPH_JSON: dependency-graph.json
E2E_SPEC_MANIFEST: ${{ steps.manifest.outputs.path }}
run: |
# stdout is the plan JSON; the planner's own info logs go to stderr.
# The planner writes this step's GITHUB_OUTPUT entries.
bun .github/scripts/create-test-plan.ts > test-plan.json
echo "stats=$(jq -c '.stats' test-plan.json)" >> "$GITHUB_OUTPUT"
echo "fe_unit_specs_to_run=$(jq -c '.fe_unit_specs_to_run' test-plan.json)" >> "$GITHUB_OUTPUT"
echo "loki_stories_to_run=$(jq -c '.loki_stories_to_run' test-plan.json)" >> "$GITHUB_OUTPUT"
echo "e2e_specs_to_run=$(jq -c '.e2e_specs_to_run' test-plan.json)" >> "$GITHUB_OUTPUT"
- name: Upload test plan
id: upload
uses: actions/upload-artifact@v7
with:
name: test-plan-${{ github.run_id }}-${{ github.run_attempt }}
path: test-plan.json
if-no-files-found: error
retention-days: 7
- name: Report test plan
shell: bash
@@ -110,6 +99,10 @@ runs:
echo
echo "Changed: $(jq '.stats.fe_files_changed' "$plan") FE / $(jq '.stats.be_files_changed' "$plan") BE file(s) · $(jq '.stats.fe_modules_affected_usage' "$plan") FE module(s) affected"
echo
if [ "$(jq '.stats.loki_preview_touched' "$plan")" = "true" ]; then
echo "**Loki: full run**: a file the Storybook preview imports changed."
echo
fi
e2e_run=$(jq '.stats.e2e_specs_to_run_usage' "$plan")
e2e_all=$(jq '.stats.e2e_specs_all' "$plan")
if [ "$e2e_run" -eq "$e2e_all" ]; then

View File

@@ -2,12 +2,16 @@ name: Get Docker Info
description: Determines the registry, repository, and tag a container image is pushed to.
inputs:
registry:
description: Registry the image is pushed to, either "docker" or "ecr".
required: false
default: docker
repo:
description: Registry and repo, e.g. "metabase/metabase". Overrides branch derived repo.
description: Repository name, e.g. "metabase-enterprise". Overrides a branch derived repo.
required: false
default: ''
tag:
description: Docker tag, e.g. v1.87.2. Overrides a branch derived tag.
description: Image tag, e.g. v1.87.2. Overrides a branch derived tag.
required: false
default: ''
branch:
@@ -18,14 +22,22 @@ inputs:
description: Edition (ee or oss)
required: false
default: ee
owner:
description: Repository owner (defaults to current repo owner)
aws-account-id:
description: AWS account id hosting the ECR registry. Required for the ecr registry.
required: false
default: ${{ github.repository_owner }}
default: ''
aws-region:
description: Region of the ECR registry.
required: false
default: us-east-1
dockerhub-owner:
description: DockerHub owner.
required: false
default: metabase
outputs:
repo:
description: Docker repository name
description: Docker repository, including the registry host (ecr) or the owner (docker)
value: ${{ steps.docker-info.outputs.repo }}
tag:
description: Docker tag
@@ -40,28 +52,49 @@ runs:
- name: Determine Docker info
id: docker-info
uses: actions/github-script@v9
env:
REGISTRY: ${{ inputs.registry }}
REPO: ${{ inputs.repo }}
TAG: ${{ inputs.tag }}
BRANCH: ${{ inputs.branch }}
EDITION: ${{ inputs.edition }}
AWS_ACCOUNT_ID: ${{ inputs.aws-account-id }}
AWS_REGION: ${{ inputs.aws-region }}
DOCKERHUB_OWNER: ${{ inputs.dockerhub-owner }}
with:
script: | # js
const branch = '${{ inputs.branch }}';
const edition = '${{ inputs.edition }}';
const owner = '${{ inputs.owner }}';
const { REGISTRY, OWNER, BRANCH, EDITION, AWS_ACCOUNT_ID, AWS_REGION, DOCKERHUB_OWNER } = process.env;
let tag = '${{ inputs.tag}}';
let tag = process.env.TAG;
if (!tag) {
tag = branch === 'master' ? 'latest' : `${branch.replaceAll('/', '-')}-${edition}`;
tag = BRANCH === 'master' ? 'latest' : `${BRANCH.replaceAll('/', '-')}-${EDITION}`;
}
let repo = '${{ inputs.repo }}';
if (!repo) {
if (branch === 'master') {
repo = edition === 'ee' ? `${owner}/metabase-enterprise-head` : `${owner}/metabase-head`;
let name = process.env.REPO;
if (!name) {
if (BRANCH === 'master') {
name = EDITION === 'ee' ? 'metabase-enterprise-head' : 'metabase-head';
} else {
repo = `${owner}/metabase-dev`;
name = 'metabase-dev';
}
}
let repo;
if (REGISTRY === 'ecr') {
if (!AWS_ACCOUNT_ID) {
core.setFailed('Pushing to AWS ECR requires aws-account-id to be set.');
return;
}
repo = `${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/${name}`;
} else if (REGISTRY === 'docker') {
repo = `${DOCKERHUB_OWNER}/${name}`;
} else {
core.setFailed(`Unknown registry "${REGISTRY}", expected "docker" or "ecr".`);
return;
}
core.setOutput('repo', repo);
core.setOutput('tag', tag);
core.setOutput('full_image', `${repo}:${tag}`);
console.log(`Resolved: repo = ${repo}, tag = ${tag}`);
console.log(`Resolved: registry = ${REGISTRY}, repo = ${repo}, tag = ${tag}`);

View File

@@ -0,0 +1,39 @@
name: Prepare test selection
description: Download this run's test plan and write the suite's selected test paths.
inputs:
suite:
description: The suite to select tests for (unit, loki or e2e).
required: true
test-plan-artifact-id:
description: ID of this run's test-plan.json artifact.
required: true
require-download:
description: Fail the step when the test plan download fails, instead of running the full suite.
required: false
default: "false"
outputs:
paths-file:
description: JSON file listing the selected test paths, or '' when the suite runs in full.
value: ${{ steps.selection.outputs.paths-file }}
selection:
description: full, narrowed or empty, or '' when the plan is unavailable and the suite runs in full.
value: ${{ steps.selection.outputs.selection }}
runs:
using: composite
steps:
- name: Download test plan
id: test-plan
continue-on-error: ${{ inputs.require-download != 'true' }}
uses: actions/download-artifact@v8
with:
artifact-ids: ${{ inputs.test-plan-artifact-id }}
path: ${{ runner.temp }}/test-plan
- name: Prepare test selection
id: selection
shell: bash
env:
PLAN_DOWNLOADED: ${{ steps.test-plan.outcome }}
run: bun .github/scripts/prepare-test-selection.ts "${{ inputs.suite }}"

View File

@@ -130,6 +130,7 @@ runs:
if [ "$PROFILE_TESTS" = "true" ]; then
echo "Profiling this run; a flamegraph will be attached to the job."
echo "profiled=true" >> "$GITHUB_OUTPUT"
ALIASES="$ALIASES:profile-tests"
BASE="$BASE:profile-tests"
fi
# An explicit selection wins over the rerun narrowing below: it was asked for by hand, on this
@@ -149,6 +150,15 @@ runs:
shell: bash
id: run-test
env:
# Both derived from test.profile-backend.yml, the sole dispatcher; its runs always profile. Inside
# a called workflow `github.event` is the caller's payload, so these resolve here rather than
# in each calling workflow. On a normal CI run there are no dispatch inputs, so these render
# `false` and empty and the job's own selection stands.
#
# ONLY_TESTS replaces the job's own test selection, so every matrix leg of the chosen job
# runs the whole list.
PROFILE_TESTS: ${{ github.event.inputs.suite != '' }}
ONLY_TESTS: ${{ github.event.inputs.tests }}
TEST_RUNNER_INDEX: ${{ inputs.runner-index }}
TEST_RUNNER_COUNT: ${{ inputs.runner-count }}
TEST_JVMS_PER_RUNNER: ${{ inputs.jvms-per-runner }}

View File

@@ -63,12 +63,10 @@ frontend_specs: &frontend_specs
- "frontend/**/tests/**"
- "frontend/**/*.unit.*"
- "enterprise/frontend/**/*.unit.*"
- ".github/**/*.unit.spec.*"
# file-paths.unit.spec.ts asserts what this file's own patterns match, so a change to it has to
# run the specs that guard it.
- ".github/file-paths.yaml"
- "jest.config.js"
- "jest.tz.unit.conf.json"
- "jest.base.conf.js"
- "jest.esm-packages.js"
- "jest.tz.unit.conf.js"
release_source: &release_sources
- "release/**"
@@ -80,6 +78,7 @@ frontend_all: &frontend_all
frontend_loki_ci: &frontend_loki_ci
- ".github/workflows/loki.yml"
- "loki.config.js"
- ".loki/**"
- ".storybook/**"
@@ -87,14 +86,52 @@ frontend_unit_infra: &frontend_unit_infra
- *frontend_configs
- *frontend_ci
- "jest.config.js"
- "jest.tz.unit.conf.json"
- "jest.base.conf.js"
- "jest.esm-packages.js"
- "jest.tz.unit.conf.js"
- "frontend/test/**"
- "frontend/**/__mocks__/**"
- "frontend/src/embedding-sdk-shared/jest/**"
frontend_loki_infra: &frontend_loki_infra
- *frontend_configs
- *shared_ci
- *frontend_loki_ci
- "frontend/build/shared/**"
- "frontend/test/__support__/**"
- "patches/**"
# dependency-cruiser doesn't follow CSS @import, so global styles must run every story.
- "frontend/src/metabase/css/**/*.css"
# Storybook serves resources/frontend_client as static files, and the stories bundle the app's SVGs.
- "resources/frontend_client/**"
- "frontend/**/*.svg"
- "enterprise/frontend/**/*.svg"
frontend_loki_all:
- *frontend_sources
- *frontend_loki_infra
# Everything the ci-scripts jest project runs or depends on.
# test-unit ignores that project, so this filter is the only thing that reruns it.
ci_scripts:
- *frontend_ci
- ".nvmrc"
# file-paths.unit.spec.ts asserts what this file's own patterns match, so a change to it has to
# run the specs that guard it.
- ".github/file-paths.yaml"
- ".github/scripts/**"
- ".github/actions/**"
- ".github/workflows/loki.yml"
- ".storybook/story-files.cjs"
- "frontend/lint/module-boundaries.mjs"
- "frontend/lint/shared-tiers.mjs"
- "frontend/test/jest-test-paths-filter.ts"
- ".dependency-cruiser.cjs"
- "jest.config.js"
- "jest.base.conf.js"
- "jest.esm-packages.js"
- "package.json"
- "bun.lock"
# e2e harness/support code (commands, support file, runner). Not in the
# instrumented bundle, so it maps to no covered module — a change here forces a

View File

@@ -105,6 +105,28 @@ export function getAffectedFiles(
return affected;
}
/** The file plus every file it transitively imports. */
export function getTransitiveDependencies(
fileDependencies: FileDependency[],
file: string,
): Set<string> {
const dependenciesOf = new Map(
fileDependencies.map(({ source, dependencies }) => [source, dependencies]),
);
const reached = new Set([file]);
const queue = [file];
while (queue.length > 0) {
const current = queue.shift()!;
for (const dependency of dependenciesOf.get(current) ?? []) {
if (!reached.has(dependency)) {
reached.add(dependency);
queue.push(dependency);
}
}
}
return reached;
}
export type ModuleGraph = {
nodes: ModuleNode[];
/**

View File

@@ -6,6 +6,7 @@ import {
getAffectedFiles,
getAffectedModules,
getChangedModules,
getTransitiveDependencies,
mapFileToModule,
parseCruiseModules,
} from "./affected-modules";
@@ -123,6 +124,31 @@ describe("affected modules", () => {
});
});
describe("getTransitiveDependencies", () => {
it("should return the file and everything it transitively imports", () => {
expect(
[...getTransitiveDependencies(FILE_DEPS, "src/app.js")].sort(),
).toEqual(["src/app.js", "src/foo/foo.tsx", "src/utils/colors.ts"]);
});
it("should return only the file when it imports nothing", () => {
expect([
...getTransitiveDependencies(FILE_DEPS, "src/utils/colors.ts"),
]).toEqual(["src/utils/colors.ts"]);
});
it("should stop at an import cycle", () => {
const cyclic = [
{ source: "src/a.ts", dependencies: ["src/b.ts"] },
{ source: "src/b.ts", dependencies: ["src/a.ts"] },
];
expect([...getTransitiveDependencies(cyclic, "src/a.ts")].sort()).toEqual(
["src/a.ts", "src/b.ts"],
);
});
});
describe("buildUsageModuleGraph + getAffectedModules", () => {
const usageGraph = buildUsageModuleGraph(ELEMENTS, FILE_DEPS);

View File

@@ -11,6 +11,7 @@ import {
buildUsageModuleGraph,
getAffectedModules,
getChangedModules,
getTransitiveDependencies,
mapFileToModule,
} from "./affected-modules";
@@ -21,6 +22,7 @@ export type TestPlanStats = {
be_files_total: number;
unit_infra_touched: boolean;
loki_infra_touched: boolean;
loki_preview_touched: boolean;
shared_sources_touched: boolean;
fe_modules_total: number;
fe_modules_changed: number;
@@ -48,9 +50,12 @@ export type CreateTestPlanInput = {
elements: ModuleDef[];
rules: Rule[];
changedFiles: string[];
// Parsed dependency-cruiser edges, or null to fall back to the rules graph.
fileDependencies: FileDependency[] | null;
// Loads the dependency-cruiser edges, called only when some suite can be narrowed.
// Null falls back to the rules graph.
loadFileDependencies: () => FileDependency[] | null;
testFilesBySuite: { unit: string[]; loki: string[]; e2e: string[] };
// Storybook's preview file, whose imports wrap every story.
lokiPreviewFile: string;
e2eSpecFiles: Record<string, string[]> | null;
unitInfraTouched: boolean;
lokiInfraTouched: boolean;
@@ -63,6 +68,7 @@ export type CreateTestPlanInput = {
};
// Tests whose owning module is affected.
// A spec outside the module tree always runs, because the graph cannot scope it.
export function filterAffectedTests(
nodes: ModuleNode[],
affected: Set<string>,
@@ -70,7 +76,7 @@ export function filterAffectedTests(
): string[] {
return testFiles.filter((file) => {
const module = mapFileToModule(nodes, file);
return module !== null && affected.has(module);
return module === null || affected.has(module);
});
}
@@ -130,8 +136,9 @@ export function createTestPlan({
elements,
rules,
changedFiles,
fileDependencies,
loadFileDependencies,
testFilesBySuite,
lokiPreviewFile,
e2eSpecFiles,
unitInfraTouched,
lokiInfraTouched,
@@ -143,16 +150,11 @@ export function createTestPlan({
beFilesTotal,
}: CreateTestPlanInput): TestPlan {
const rulesGraph = buildModuleGraph(elements, rules);
const usageGraph = fileDependencies
? buildUsageModuleGraph(elements, fileDependencies)
: rulesGraph;
const nodes = rulesGraph.nodes;
// Distinct module types (an element type can span several patterns).
const totalModules = new Set(elements.map((el) => el.type)).size;
const changedModules = getChangedModules(nodes, changedFiles);
const rulesAffected = getAffectedModules(rulesGraph, changedFiles);
const usageAffected = getAffectedModules(usageGraph, changedFiles);
// The coarse "feature" tier is the only set the e2e manifest is ever
// collapsed to (see filterAffectedE2eSpecs).
@@ -161,7 +163,7 @@ export function createTestPlan({
// cljc/cljs compile into the FE bundle, so they force a full run that module
// selection can't narrow — same effect as a suite's own infra changing.
const unitForceAll = unitInfraTouched || sharedSourcesTouched;
const lokiForceAll = lokiInfraTouched || sharedSourcesTouched;
const lokiInfraForceAll = lokiInfraTouched || sharedSourcesTouched;
// e2e is integration-level, so anything the FE-coverage manifest can't see
// forces a full run: cljc/cljs in the bundle (sharedSourcesTouched), a backend
// change that can break the UI (beFilesChanged), an e2e harness/support change
@@ -172,6 +174,28 @@ export function createTestPlan({
beFilesChanged > 0 ||
e2eSpecFiles === null;
// The cruise takes seconds, so it is skipped when every suite runs in full anyway.
const edges =
unitForceAll && lokiInfraForceAll && e2eForceAll
? null
: loadFileDependencies();
const usageGraph = edges
? buildUsageModuleGraph(elements, edges)
: rulesGraph;
const usageAffected = getAffectedModules(usageGraph, changedFiles);
// The preview wraps every story, so a change to a file it imports can change any screenshot.
const previewDependencies = edges
? getTransitiveDependencies(edges, lokiPreviewFile)
: new Set<string>();
const lokiPreviewTouched = changedFiles.some((file) =>
previewDependencies.has(file),
);
// Without the cruise, the rules graph cannot account for the preview's
// global dependencies, so it cannot safely narrow Loki.
const lokiForceAll =
lokiInfraForceAll || lokiPreviewTouched || edges === null;
const select = (forceAll: boolean, affected: Set<string>, files: string[]) =>
forceAll ? files : filterAffectedTests(nodes, affected, files);
@@ -208,6 +232,7 @@ export function createTestPlan({
be_files_total: beFilesTotal,
unit_infra_touched: unitInfraTouched,
loki_infra_touched: lokiInfraTouched,
loki_preview_touched: lokiPreviewTouched,
shared_sources_touched: sharedSourcesTouched,
fe_modules_total: totalModules,
fe_modules_changed: changedModules.size,

View File

@@ -26,8 +26,9 @@ const E2E_FILES = ["e2e/test/scenarios/a.cy.spec.ts"];
const baseInput = {
elements: ELEMENTS,
rules: RULES,
fileDependencies: null,
loadFileDependencies: () => null,
testFilesBySuite: { unit: UNIT_FILES, loki: LOKI_FILES, e2e: E2E_FILES },
lokiPreviewFile: ".storybook/preview.tsx",
e2eSpecFiles: null,
unitInfraTouched: false,
lokiInfraTouched: false,
@@ -41,7 +42,10 @@ const baseInput = {
describe("createTestPlan", () => {
it("runs only specs in affected modules (rules graph)", () => {
const plan = createTestPlan({ ...baseInput, changedFiles: ["src/foo/x.ts"] });
const plan = createTestPlan({
...baseInput,
changedFiles: ["src/foo/x.ts"],
});
expect(plan.stats.fe_modules_changed).toBe(1);
expect(plan.stats.fe_modules_affected_rules).toBe(1);
@@ -50,8 +54,8 @@ describe("createTestPlan", () => {
"src/foo/foo.unit.spec.ts",
]);
expect(plan.stats.unit_specs_to_run_rules).toBe(2);
expect(plan.stats.loki_stories_to_run_rules).toBe(1);
// With no file deps, usage falls back to rules.
expect(plan.loki_stories_to_run).toEqual(LOKI_FILES);
// With no file deps, unit usage falls back to rules, but Loki runs in full.
expect(plan.stats.unit_specs_to_run_usage).toBe(2);
});
@@ -60,7 +64,7 @@ describe("createTestPlan", () => {
...baseInput,
changedFiles: ["src/utils/colors.ts"],
// Only feature/foo actually imports lib/utils.
fileDependencies: [
loadFileDependencies: () => [
{ source: "src/foo/foo.tsx", dependencies: ["src/utils/colors.ts"] },
],
});
@@ -91,14 +95,69 @@ describe("createTestPlan", () => {
...baseInput,
changedFiles: ["docs/readme.md"],
unitInfraTouched: true,
loadFileDependencies: () => [],
});
expect(plan.stats.unit_specs_to_run_rules).toBe(UNIT_FILES.length);
expect(plan.stats.loki_stories_to_run_rules).toBe(0);
});
// The preview imports the theme, which imports the colors.
const PREVIEW_DEPS = [
{ source: ".storybook/preview.tsx", dependencies: ["src/utils/theme.ts"] },
{ source: "src/utils/theme.ts", dependencies: ["src/utils/colors.ts"] },
{ source: "src/foo/foo.tsx", dependencies: ["src/foo/x.ts"] },
];
it("forces a full Loki run when the Storybook preview imports a changed file", () => {
const plan = createTestPlan({
...baseInput,
changedFiles: ["src/utils/colors.ts"],
loadFileDependencies: () => PREVIEW_DEPS,
e2eSpecFiles: { "e2e/test/scenarios/a.cy.spec.ts": ["src/bar/b.ts"] },
});
expect(plan.stats.loki_preview_touched).toBe(true);
expect(plan.stats.loki_stories_to_run_rules).toBe(LOKI_FILES.length);
expect(plan.loki_stories_to_run).toEqual(LOKI_FILES);
// Unit and e2e still narrow to what the usage graph reaches from the change.
expect(plan.fe_unit_specs_to_run).toEqual(["src/utils/utils.unit.spec.ts"]);
expect(plan.e2e_specs_to_run).toEqual([]);
});
it("narrows Loki when the Storybook preview does not import the changed file", () => {
const plan = createTestPlan({
...baseInput,
changedFiles: ["src/foo/x.ts"],
loadFileDependencies: () => PREVIEW_DEPS,
});
expect(plan.stats.loki_preview_touched).toBe(false);
expect(plan.loki_stories_to_run).toEqual(["src/foo/Foo.stories.tsx"]);
});
it("runs all Loki stories when the cruise fails even if rules select fewer", () => {
const plan = createTestPlan({
...baseInput,
changedFiles: ["src/foo/x.ts"],
loadFileDependencies: () => null,
e2eSpecFiles: { "e2e/test/scenarios/a.cy.spec.ts": ["src/bar/b.ts"] },
});
expect(plan.stats.loki_preview_touched).toBe(false);
expect(plan.loki_stories_to_run).toEqual(LOKI_FILES);
expect(plan.stats.loki_stories_to_run_rules).toBe(LOKI_FILES.length);
expect(plan.stats.loki_stories_to_run_usage).toBe(LOKI_FILES.length);
// Unit and E2E keep their rules-graph fallback.
expect(plan.fe_unit_specs_to_run).toEqual(UNIT_FILES.slice(0, 2));
expect(plan.e2e_specs_to_run).toEqual([]);
});
it("runs the full e2e suite when no coverage manifest is available", () => {
const plan = createTestPlan({ ...baseInput, changedFiles: ["src/foo/x.ts"] });
const plan = createTestPlan({
...baseInput,
changedFiles: ["src/foo/x.ts"],
});
expect(plan.stats.e2e_specs_to_run_rules).toBe(E2E_FILES.length);
expect(plan.stats.e2e_specs_to_run_usage).toBe(E2E_FILES.length);
@@ -229,21 +288,53 @@ describe("createTestPlan", () => {
expect(plan.stats.fe_files_changed).toBe(5);
expect(plan.stats.be_files_changed).toBe(3);
});
it("does not build the usage graph when every suite runs in full", () => {
const loadFileDependencies = jest.fn(() => null);
createTestPlan({
...baseInput,
changedFiles: ["src/foo/x.ts"],
sharedSourcesTouched: true,
loadFileDependencies,
});
expect(loadFileDependencies).not.toHaveBeenCalled();
});
it("builds the usage graph when any suite can be narrowed", () => {
const loadFileDependencies = jest.fn(() => null);
createTestPlan({
...baseInput,
changedFiles: ["src/foo/x.ts"],
unitInfraTouched: true, // loki can still be narrowed
loadFileDependencies,
});
expect(loadFileDependencies).toHaveBeenCalledTimes(1);
});
});
describe("filterAffectedTests", () => {
it("keeps only tests inside affected modules", () => {
it("keeps tests inside affected modules, drops tests in unaffected ones", () => {
const nodes = buildNodes(ELEMENTS);
const affected = new Set(["feature/foo", "lib/utils"]);
const tests = [
"src/foo/foo.unit.spec.ts",
"src/utils/colors.unit.spec.ts",
"src/bar/bar.unit.spec.ts",
"docs/unrelated.unit.spec.ts",
];
expect(filterAffectedTests(nodes, affected, tests)).toEqual([
"src/foo/foo.unit.spec.ts",
"src/utils/colors.unit.spec.ts",
]);
});
it("keeps a test that maps to no module (unknown scope)", () => {
const nodes = buildNodes(ELEMENTS);
const affected = new Set(["feature/foo"]);
const tests = ["docs/unrelated.unit.spec.ts", "src/bar/bar.unit.spec.ts"];
expect(filterAffectedTests(nodes, affected, tests)).toEqual([
"docs/unrelated.unit.spec.ts",
]);
});
});

View File

@@ -0,0 +1,182 @@
import { spawnSync } from "node:child_process";
import { readFileSync, statSync } from "node:fs";
import { join } from "node:path";
const TEST_UNIT_COMMAND = ["bun", "run", "test-unit-keep-cljs"];
type SuiteResult = {
name: string;
status: string;
assertionResults: unknown[];
};
type JestResults = {
wasInterrupted: boolean;
testResults: SuiteResult[];
};
function isSuiteResult(value: unknown): value is SuiteResult {
return (
typeof value === "object" &&
value !== null &&
"name" in value &&
typeof value.name === "string" &&
"status" in value &&
typeof value.status === "string" &&
"assertionResults" in value &&
Array.isArray(value.assertionResults)
);
}
function readResults(file: string): JestResults | null {
const value = readJson(file);
if (
typeof value === "object" &&
value !== null &&
"wasInterrupted" in value &&
typeof value.wasInterrupted === "boolean" &&
"testResults" in value &&
Array.isArray(value.testResults) &&
value.testResults.every(isSuiteResult)
) {
return {
wasInterrupted: value.wasInterrupted,
testResults: value.testResults,
};
}
return null;
}
function parseJson(text: string): unknown {
try {
return JSON.parse(text);
} catch {
return null;
}
}
function readJson(file: string): unknown {
try {
return parseJson(readFileSync(file, "utf8"));
} catch {
return null;
}
}
function isStringArray(value: unknown): value is string[] {
return (
Array.isArray(value) && value.every((item) => typeof item === "string")
);
}
function hasContent(file: string): boolean {
try {
return statSync(file).size > 0;
} catch {
return false;
}
}
export function findRunnerFailure(env: NodeJS.ProcessEnv): string | null {
const {
UNIT_TEST_RESULTS_FILE,
JEST_JUNIT_OUTPUT_DIR,
JEST_JUNIT_OUTPUT_NAME,
} = env;
if (
!UNIT_TEST_RESULTS_FILE ||
!JEST_JUNIT_OUTPUT_DIR ||
!JEST_JUNIT_OUTPUT_NAME
) {
throw new Error(
"UNIT_TEST_RESULTS_FILE, JEST_JUNIT_OUTPUT_DIR and JEST_JUNIT_OUTPUT_NAME are required",
);
}
const results = readResults(UNIT_TEST_RESULTS_FILE);
if (results === null || results.wasInterrupted) {
return "Jest did not finish the run.";
}
if (!hasContent(join(JEST_JUNIT_OUTPUT_DIR, JEST_JUNIT_OUTPUT_NAME))) {
return "Jest did not write a JUnit report.";
}
const failedSuites = results.testResults.filter(
(suite) => suite.status === "failed",
);
// jest-junit leaves out suites with no test results, so the quarantine gate never sees them.
const suitesWithoutResults = failedSuites.filter(
(suite) => suite.assertionResults.length === 0,
);
if (suitesWithoutResults.length > 0) {
const names = suitesWithoutResults.map((suite) => suite.name).join(", ");
return `Test suites failed to run: ${names}`;
}
if (failedSuites.length === 0) {
return "Jest failed without a failing test suite.";
}
return null;
}
function listTests(
command: readonly string[],
env: NodeJS.ProcessEnv,
): string[] | null {
const [program, ...args] = command;
const result = spawnSync(program, [...args, "--listTests", "--json"], {
env,
encoding: "utf8",
stdio: ["ignore", "pipe", "inherit"],
});
if (result.status !== 0) {
return null;
}
const listed = parseJson(result.stdout);
return isStringArray(listed) ? listed : null;
}
// --passWithNoTests lets every shard pass when the selection matches nothing Jest discovers.
// A single empty shard is normal, so only an unsharded listing that is also empty warns.
export function findEmptySelection(
env: NodeJS.ProcessEnv,
command: readonly string[] = TEST_UNIT_COMMAND,
): string | null {
const { UNIT_TEST_RESULTS_FILE, JEST_TEST_PATHS_FILE } = env;
if (!UNIT_TEST_RESULTS_FILE || !JEST_TEST_PATHS_FILE) {
return null;
}
const results = readResults(UNIT_TEST_RESULTS_FILE);
const selection = readJson(JEST_TEST_PATHS_FILE);
if (
results === null ||
results.testResults.length > 0 ||
!isStringArray(selection) ||
selection.length === 0
) {
return null;
}
const listed = listTests(command, env);
if (listed === null) {
return "Could not list Jest's unit tests to check the selection.";
}
if (listed.length > 0) {
return null;
}
return `None of the ${selection.length} selected unit specs match a test Jest discovers, so no unit tests ran.`;
}
if (require.main === module) {
if (process.env.TEST_OUTCOME === "failure") {
const failure = findRunnerFailure(process.env);
if (failure !== null) {
console.error(`::error::${failure}`);
process.exitCode = 1;
}
} else {
const warning = findEmptySelection(process.env);
if (warning !== null) {
console.warn(`::warning::${warning}`);
}
}
}

View File

@@ -1,28 +1,31 @@
// I/O entrypoint: gather inputs (env vars, the cruise graph, the test-file
// lists) and hand them to createTestPlan, which does the computing.
// I/O entrypoint: gather inputs (env vars, the test-file lists, the cruise graph on request),
// hand them to createTestPlan, which does the computing, and publish its small stats output.
import { execFileSync } from "node:child_process";
import { existsSync, readFileSync } from "node:fs";
import { execFileSync, spawnSync } from "node:child_process";
import { appendFileSync, existsSync, readFileSync } from "node:fs";
import micromatch from "micromatch";
import { MAIN_APP_STORY_GLOBS } from "../../.storybook/story-files.cjs";
import { elements, rules } from "../../frontend/lint/module-boundaries.mjs";
import { type FileDependency, parseCruiseModules } from "./affected-modules";
import { createTestPlan } from "./affected-tests";
import { listSpecFiles } from "./e2e-spec-globs.mjs";
const UNIT_ROOTS = ["frontend/src", "enterprise/frontend/src"];
// The specs `bun run test-unit` runs.
const UNIT_ROOTS = ["."];
const UNIT_GLOBS = [
"frontend/src/**/*.unit.spec.{js,jsx,ts,tsx}",
"enterprise/frontend/src/**/*.unit.spec.{js,jsx,ts,tsx}",
"**/*.unit.spec.{js,jsx,ts,tsx}",
"!.github/**", // the ci-scripts project, which test-unit ignores
"!release/**", // has its own jest config
"!enterprise/frontend/src/custom-viz/**", // modulePathIgnorePatterns in jest.config.js
"!**/*.leak.unit.spec.{js,jsx,ts,tsx}", // run only by jest.memory.conf.js
];
const STORY_ROOTS = ["frontend", "enterprise/frontend"];
const STORY_GLOBS = [
"frontend/**/*.stories.{js,jsx,ts,tsx}",
"enterprise/frontend/**/*.stories.{js,jsx,ts,tsx}",
];
const FRONTEND_ROOTS = ["frontend", "enterprise/frontend"];
const LOKI_PREVIEW_FILE = ".storybook/preview.tsx";
// `git ls-files -- frontend enterprise/frontend` already prints just over a
// megabyte of paths, and node's default maxBuffer is exactly 1 MiB: past that
@@ -32,12 +35,11 @@ const LS_FILES_MAX_BUFFER = 64 * 1024 * 1024;
// Returns the tracked files under `roots` that match `globs`. The `dot: true`
// option means files inside dot-directories such as `.storybook` are included.
function listFiles(roots: string[], globs: string[]): string[] {
const tracked = execFileSync("git", ["ls-files", "--", ...roots], {
const tracked = execFileSync("git", ["ls-files", "-z", "--", ...roots], {
encoding: "utf8",
maxBuffer: LS_FILES_MAX_BUFFER,
})
.split("\n")
.map((line) => line.trim())
.split("\0")
.filter(Boolean);
return micromatch(tracked, globs, { dot: true });
}
@@ -49,26 +51,43 @@ const csvToList = (csv: string | undefined) =>
.map((s) => s.trim())
.filter(Boolean);
// Reads the dependency-cruiser graph (DEP_GRAPH_JSON). Null falls back to the
// rules graph, so a missing or unparseable file never breaks the plan.
function readFileDependencies(): FileDependency[] | null {
const path = process.env.DEP_GRAPH_JSON;
if (path && existsSync(path)) {
try {
const { modules } = JSON.parse(readFileSync(path, "utf8"));
process.stderr.write(`Using usage graph from ${path}.\n`);
return parseCruiseModules(modules);
} catch (error) {
process.stderr.write(
`Failed to read ${path}; falling back to rules graph: ${error}\n`,
);
}
} else {
process.stderr.write(
"No DEP_GRAPH_JSON found; falling back to rules graph.\n",
// Runs dependency-cruiser over the frontend sources and the Storybook preview and parses its edges.
// Null falls back to the rules graph, so a failed cruise never breaks the plan.
function loadFileDependencies(): FileDependency[] | null {
const output = "dependency-graph.json";
process.stderr.write("Building usage graph with dependency-cruiser.\n");
try {
// stdout is the plan JSON, so the cruise's own output goes to stderr.
const result = spawnSync(
"bunx",
[
"depcruise",
"frontend/src",
"enterprise/frontend/src",
LOKI_PREVIEW_FILE,
"--config",
".dependency-cruiser.cjs",
"--output-type",
"json",
"--output-to",
output,
],
{ stdio: ["ignore", 2, 2] },
);
if (result.error) {
throw result.error;
}
if (result.status !== 0) {
throw new Error(`depcruise exited with status ${result.status}`);
}
const { modules } = JSON.parse(readFileSync(output, "utf8"));
return parseCruiseModules(modules);
} catch (error) {
process.stderr.write(
`Failed to build usage graph; falling back to rules graph: ${error}\n`,
);
return null;
}
return null;
}
// Reads the nightly coverage manifest (E2E_SPEC_MANIFEST): { builtAt, specs:
@@ -98,12 +117,13 @@ const testPlan = createTestPlan({
elements,
rules,
changedFiles: csvToList(process.env.CHANGED_FILES),
fileDependencies: readFileDependencies(),
loadFileDependencies,
testFilesBySuite: {
unit: listFiles(UNIT_ROOTS, UNIT_GLOBS),
loki: listFiles(STORY_ROOTS, STORY_GLOBS),
loki: listFiles(FRONTEND_ROOTS, MAIN_APP_STORY_GLOBS),
e2e: listSpecFiles(),
},
lokiPreviewFile: LOKI_PREVIEW_FILE,
e2eSpecFiles: readE2eSpecFiles(),
unitInfraTouched: process.env.UNIT_INFRA_TOUCHED === "true",
lokiInfraTouched: process.env.LOKI_INFRA_TOUCHED === "true",
@@ -111,8 +131,15 @@ const testPlan = createTestPlan({
sharedSourcesTouched: process.env.SHARED_SOURCES_TOUCHED === "true",
feFilesChanged: csvToList(process.env.FE_CHANGED_FILES).length,
beFilesChanged: csvToList(process.env.BE_CHANGED_FILES).length,
feFilesTotal: listFiles(["frontend", "enterprise/frontend"], ["**"]).length,
feFilesTotal: listFiles(FRONTEND_ROOTS, ["**"]).length,
beFilesTotal: listFiles(["src", "enterprise/backend"], ["**"]).length,
});
process.stdout.write(JSON.stringify(testPlan) + "\n");
if (process.env.GITHUB_OUTPUT) {
appendFileSync(
process.env.GITHUB_OUTPUT,
`stats=${JSON.stringify(testPlan.stats)}\n`,
);
}

View File

@@ -11,12 +11,17 @@ type Filter = string | Filter[] | { [changeType: string]: Filter };
// A filter is a list of patterns, and an entry may name the change types it applies to as
// { "added|modified": pattern }. Either way the patterns are what matters here.
const patterns = (value: Filter): string[] => {
if (typeof value === "string") return [value];
if (Array.isArray(value)) return value.flatMap(patterns);
if (typeof value === "string") {
return [value];
}
if (Array.isArray(value)) {
return value.flatMap(patterns);
}
return Object.values(value).flatMap(patterns);
};
describe("file-paths.yaml", () => {
// js-yaml returns an untyped value, and this repository file defines named path filters.
const filters = load(readFileSync(FILTERS, "utf8")) as Record<string, Filter>;
// dorny/paths-filter matches with picomatch, which micromatch wraps, so this asks the same
@@ -44,14 +49,47 @@ describe("file-paths.yaml", () => {
"%s skips a ratchets-only change and still follows the rest of .clj-kondo",
(name) => {
expect(matches(name, ".clj-kondo/ratchets.edn")).toBe(false);
expect(
matches(name, ".clj-kondo/config/modules/ratchets.edn"),
).toBe(false);
expect(matches(name, ".clj-kondo/config/modules/ratchets.edn")).toBe(
false,
);
expect(matches(name, ".clj-kondo/config.edn")).toBe(true);
expect(matches(name, ".clj-kondo/config/modules/config.edn")).toBe(true);
},
);
it("runs CI script tests when the Node version changes", () => {
expect(matches("ci_scripts", ".nvmrc")).toBe(true);
});
it.each([
"jest.config.js",
"jest.base.conf.js",
"jest.esm-packages.js",
"jest.tz.unit.conf.js",
])("runs every frontend unit test when %s changes", (path) => {
expect(matches("frontend_specs", path)).toBe(true);
expect(matches("frontend_unit_infra", path)).toBe(true);
});
it("runs every frontend unit test when an SDK jest setup file changes", () => {
expect(
matches(
"frontend_unit_infra",
"frontend/src/embedding-sdk-shared/jest/setup-env.ts",
),
).toBe(true);
});
it.each([
"jest.config.js",
"jest.base.conf.js",
"jest.esm-packages.js",
".storybook/story-files.cjs",
".github/workflows/loki.yml",
])("runs CI script tests when %s changes", (path) => {
expect(matches("ci_scripts", path)).toBe(true);
});
it("runs the ratchet check on the ratchets file", () => {
expect(matches("project_ratchet_checks", ".clj-kondo/ratchets.edn")).toBe(
true,
@@ -63,4 +101,42 @@ describe("file-paths.yaml", () => {
),
).toBe(true);
});
it.each([
"loki.config.js",
".storybook/story-files.cjs",
".storybook/preview.tsx",
".loki/reference/chrome.laptop-Button.png",
".github/workflows/loki.yml",
".github/workflows/run-tests.yml",
".github/actions/prepare-frontend/action.yml",
"frontend/build/shared/rspack/css-config.js",
"frontend/test/__support__/custom-viz-fixtures/calendar-heatmap/index.js",
"patches/@loki+browser+0.35.0.patch",
"frontend/src/metabase/css/index.module.css",
"frontend/src/metabase/css/core/base.module.css",
"resources/frontend_client/app/fonts/Lato/lato-v16-latin-regular.woff2",
"resources/frontend_client/app/assets/img/no_results.svg",
"frontend/src/metabase/ui/components/icons/Icon/icons/warning.svg",
"enterprise/frontend/src/metabase-enterprise/google_drive/database-error.svg",
])("runs all Loki stories when %s changes", (file) => {
expect(matches("frontend_loki_all", file)).toBe(true);
expect(matches("frontend_loki_infra", file)).toBe(true);
});
it.each([
"frontend/src/metabase/ui/Button.stories.tsx",
"frontend/src/metabase/ui/components/feedback/Alert/Alert.module.css",
"frontend/build/embedding-sdk/rspack.config.js",
])("allows a change to %s to narrow the Loki run", (file) => {
expect(matches("frontend_loki_all", file)).toBe(true);
expect(matches("frontend_loki_infra", file)).toBe(false);
});
it.each([
".github/workflows/frontend.yml",
"frontend/test/metabase/scenarios/Button.unit.spec.tsx",
])("does not run Loki stories when %s changes", (file) => {
expect(matches("frontend_loki_all", file)).toBe(false);
});
});

View File

@@ -0,0 +1,237 @@
import { spawnSync } from "node:child_process";
import {
mkdirSync,
mkdtempSync,
readFileSync,
realpathSync,
rmSync,
writeFileSync,
} from "node:fs";
import { tmpdir } from "node:os";
import { join, resolve } from "node:path";
import { load } from "js-yaml";
import {
findEmptySelection,
findRunnerFailure,
} from "./check-unit-test-results";
import { prepareTestSelection } from "./prepare-test-selection";
type Step = { name: string; run?: string; "continue-on-error"?: boolean };
// js-yaml returns an untyped value, and this repository workflow defines the job's steps.
const workflow = load(
readFileSync(resolve(__dirname, "../workflows/frontend.yml"), "utf8"),
) as { jobs: { "fe-tests-unit": { steps: Step[] } } };
const steps = workflow.jobs["fe-tests-unit"].steps;
const checkScript = resolve(__dirname, "check-unit-test-results.ts");
const filter = resolve(
__dirname,
"../../frontend/test/jest-test-paths-filter.ts",
);
describe("frontend test plan handoff", () => {
let dir: string;
let env: NodeJS.ProcessEnv;
beforeEach(() => {
jest.spyOn(console, "warn").mockImplementation(() => {});
// Jest reports real paths, and macOS's temporary directory is behind a symlink.
dir = realpathSync(mkdtempSync(join(tmpdir(), "frontend-test-plan-")));
mkdirSync(join(dir, "test-plan"));
writeFileSync(join(dir, "output"), "");
env = {
...process.env,
NODE_OPTIONS: "",
JEST_TEST_PATHS_FILE: "",
RUNNER_TEMP: dir,
GITHUB_OUTPUT: join(dir, "output"),
PLAN_DOWNLOADED: "success",
JEST_JUNIT_OUTPUT_DIR: dir,
JEST_JUNIT_OUTPUT_NAME: "junit.xml",
UNIT_TEST_RESULTS_FILE: join(dir, "unit-test-results.json"),
};
});
afterEach(() => {
jest.restoreAllMocks();
rmSync(dir, { recursive: true, force: true });
});
function plan(files: string[], total = 2) {
return {
fe_unit_specs_to_run: files,
stats: { unit_specs_all: total, unit_specs_to_run_usage: files.length },
};
}
function writePlan(value: unknown) {
writeFileSync(join(dir, "test-plan/test-plan.json"), JSON.stringify(value));
}
function jestCommand(useFilter: boolean) {
return [
process.execPath,
require.resolve("jest/bin/jest"),
"--config",
JSON.stringify({
rootDir: dir,
testMatch: ["**/*.spec.cjs"],
reporters: ["default", require.resolve("jest-junit")],
...(useFilter && { filter }),
}),
];
}
function runJest(
files: Record<string, string>,
useFilter = false,
args: string[] = [],
) {
for (const [name, source] of Object.entries(files)) {
writeFileSync(join(dir, name), source);
}
const [program, ...commandArgs] = jestCommand(useFilter);
return spawnSync(
program,
[
...commandArgs,
"--runInBand",
"--watch=false",
"--json",
`--outputFile=${env.UNIT_TEST_RESULTS_FILE}`,
...(useFilter ? ["--passWithNoTests"] : []),
...args,
],
{ cwd: dir, env, encoding: "utf8" },
);
}
function readResults() {
return JSON.parse(
readFileSync(join(dir, "unit-test-results.json"), "utf8"),
);
}
function writeSelection(files: string[]) {
env.JEST_TEST_PATHS_FILE = join(dir, "unit-specs.json");
writeFileSync(env.JEST_TEST_PATHS_FILE, JSON.stringify(files));
}
it("lets ordinary assertion failures reach quarantine", () => {
expect(
runJest({ "fail.spec.cjs": "test('fails', () => expect(1).toBe(2));" })
.status,
).toBe(1);
expect(findRunnerFailure(env)).toBeNull();
});
it("lets suite hook failures reach quarantine", () => {
expect(
runJest({
"hook.spec.cjs":
"test('passes', () => {}); afterAll(() => { throw new Error('hook'); });",
}).status,
).toBe(1);
expect(readFileSync(join(dir, "junit.xml"), "utf8")).toContain(
"Test execution failure",
);
expect(findRunnerFailure(env)).toBeNull();
});
it("fails a filter crash even though the test step permits errors", () => {
env.JEST_TEST_PATHS_FILE = join(dir, "broken.json");
writeFileSync(env.JEST_TEST_PATHS_FILE, "[");
expect(
runJest({ "pass.spec.cjs": "test('passes', () => {});" }, true).status,
).toBe(1);
env.TEST_OUTCOME = "failure";
const result = spawnSync("bun", [checkScript], { env, encoding: "utf8" });
expect(result.status).toBe(1);
expect(result.stderr).toContain("::error::");
const step = steps.find((step) =>
step.run?.includes("check-unit-test-results.ts"),
);
expect(step?.["continue-on-error"]).toBeUndefined();
});
it("fails suite startup errors even alongside ordinary assertion failures", () => {
expect(
runJest({
"crash.spec.cjs": "throw new Error('startup crash');",
"fail.spec.cjs": "test('fails', () => expect(1).toBe(2));",
}).status,
).toBe(1);
expect(readFileSync(join(dir, "junit.xml"), "utf8")).not.toContain(
"crash.spec.cjs",
);
expect(findRunnerFailure(env)).toContain("crash.spec.cjs");
});
it("fails when JUnit results are missing", () => {
runJest({ "fail.spec.cjs": "test('fails', () => expect(1).toBe(2));" });
rmSync(join(dir, "junit.xml"));
expect(findRunnerFailure(env)).not.toBeNull();
});
it.each([true, false])(
"loads the TypeScript filter natively with an empty selection: %s",
(empty) => {
writePlan(plan(empty ? [] : [join(dir, "pass.spec.cjs")]));
prepareTestSelection("unit", env);
env.JEST_TEST_PATHS_FILE = join(dir, "unit-specs.json");
const result = runJest(
{
"pass.spec.cjs": "test('passes', () => {});",
"fail.spec.cjs":
"test('excluded', () => { throw new Error('must not run'); });",
},
true,
);
expect(result.status === 0 ? "" : result.stderr).toBe("");
expect(result.status).toBe(0);
expect(readResults()).toMatchObject({
numTotalTestSuites: empty ? 0 : 1,
});
},
);
it("warns when a narrowed run matches none of Jest's test paths", () => {
writeSelection([join(dir, "renamed.spec.cjs")]);
expect(
runJest({ "pass.spec.cjs": "test('passes', () => {});" }, true).status,
).toBe(0);
expect(findEmptySelection(env, jestCommand(true))).toContain(
"None of the 1 selected unit specs",
);
});
it("warns through Bun when the unit test step passed", () => {
writeSelection([join(dir, "renamed.spec.cjs")]);
runJest({ "pass.spec.cjs": "test('passes', () => {});" }, true);
env.TEST_OUTCOME = "success";
const result = spawnSync("bun", [checkScript], { env, encoding: "utf8" });
expect(result.status).toBe(0);
expect(result.stderr).toContain(
"::warning::None of the 1 selected unit specs",
);
});
it("does not warn for a shard the selection leaves empty", () => {
writeSelection([join(dir, "pass.spec.cjs")]);
const result = runJest(
{ "pass.spec.cjs": "test('passes', () => {});" },
true,
["--shard=2/2"],
);
expect(result.status).toBe(0);
expect(readResults()).toMatchObject({ numTotalTestSuites: 0 });
expect(findEmptySelection(env, jestCommand(true))).toBeNull();
});
it("does not list tests for an empty selection", () => {
writeSelection([]);
runJest({ "pass.spec.cjs": "test('passes', () => {});" }, true);
expect(findEmptySelection(env, ["false"])).toBeNull();
});
});

View File

@@ -0,0 +1,44 @@
import fs from "node:fs";
import os from "node:os";
import path from "node:path";
// eslint-disable-next-line @typescript-eslint/no-require-imports -- Match Jest's direct CommonJS loading contract.
const filter = require("../../frontend/test/jest-test-paths-filter");
describe("jest-test-paths-filter", () => {
const originalPathsFile = process.env.JEST_TEST_PATHS_FILE;
let tempDir: string;
let pathsFile: string;
beforeEach(() => {
tempDir = fs.mkdtempSync(path.join(os.tmpdir(), "jest-test-paths-"));
pathsFile = path.join(tempDir, "paths.json");
process.env.JEST_TEST_PATHS_FILE = pathsFile;
});
afterEach(() => {
fs.rmSync(tempDir, { recursive: true, force: true });
if (originalPathsFile === undefined) {
delete process.env.JEST_TEST_PATHS_FILE;
} else {
process.env.JEST_TEST_PATHS_FILE = originalPathsFile;
}
});
it("should keep the listed paths and drop the rest", () => {
const one = path.resolve("frontend/one.unit.spec.ts");
const two = path.resolve("frontend/two.unit.spec.ts");
const three = path.resolve("frontend/three.unit.spec.ts");
fs.writeFileSync(pathsFile, JSON.stringify([one, two]));
expect(filter([one, three, two])).toEqual({ filtered: [one, two] });
});
it("should resolve relative paths against the working directory", () => {
fs.writeFileSync(pathsFile, JSON.stringify(["frontend/one.unit.spec.ts"]));
const one = path.resolve("frontend/one.unit.spec.ts");
const two = path.resolve("frontend/two.unit.spec.ts");
expect(filter([one, two])).toEqual({ filtered: [one] });
});
});

View File

@@ -24,6 +24,15 @@ const SDK_ROOT = "resources/frontend_client/app/embedding-sdk";
const SDK_RUNTIME_RE = /embedding-sdk-chunk-runtime\./;
const notRuntime = name => !SDK_RUNTIME_RE.test(name);
// One locale catalogue per chunk, named by the `webpackChunkName: "locale-[request]"`
// comment in localization.ts. A session loads exactly one and an English session
// loads none, so counting all 36 in a reachable total overstates it by 35x: 8.5 MB
// gzip against the SDK's 3.1 MB. Their size also tracks the translation process
// rather than any code change, so counting them makes every catalogue update look
// like a bundle regression.
const LOCALE_CHUNK_RE = /(^|\/)locale-/;
const notLocale = name => !LOCALE_CHUNK_RE.test(name);
const assetFiles = (root, names) => names.map(name => path.join(root, name));
const unique = files => [...new Set(files)];
@@ -116,7 +125,8 @@ function selectAppAssets(stats) {
if (!initialAssets) {
throw new Error('app: stats-main.json / "app-main" entrypoint missing');
}
return { initialAssets, reachableAssets: entrypointJsAssets(stats, "app-main", "reachableAssets") };
const reachableAssets = entrypointJsAssets(stats, "app-main", "reachableAssets");
return { initialAssets, reachableAssets: reachableAssets?.filter(notLocale) ?? null };
}
/**
@@ -136,7 +146,9 @@ function selectLegacySdkAssets(stats, { sdkAsyncChunksLoadable }) {
const initial = unique(initialNames).filter(notRuntime);
return {
initialNames: initial,
reachableNames: sdkAsyncChunksLoadable ? unique(reachableNames).filter(notRuntime) : initial,
reachableNames: sdkAsyncChunksLoadable
? unique(reachableNames).filter(notRuntime).filter(notLocale)
: initial,
};
}
@@ -161,7 +173,9 @@ function selectChunkedSdkAssets(stats, { sdkAsyncChunksLoadable }) {
const bootstrapReachable = entrypointJsAssets(stats, "embedding-sdk-bootstrap", "reachableAssets") || bootstrapInitial;
return {
initialNames: unique([...bootstrapInitial, ...chunkedInitial]).filter(notRuntime),
reachableNames: unique([...bootstrapReachable, ...chunkedReachable]).filter(notRuntime),
reachableNames: unique([...bootstrapReachable, ...chunkedReachable])
.filter(notRuntime)
.filter(notLocale),
includesAsyncChunks: chunkedReachableAssets != null && sdkAsyncChunksLoadable,
};
}

View File

@@ -6,6 +6,7 @@ const {
} = require("./measure-bundle-sizes");
const RUNTIME = "embedding-sdk-chunk-runtime.deadbeef.js";
const LOCALE = "chunks/locale-fr-json.deadbeef.js";
const stats = entrypoints => ({ entrypoints });
@@ -30,6 +31,13 @@ describe("selectAppAssets", () => {
it("throws when the app-main entrypoint is missing", () => {
expect(() => selectAppAssets(stats({}))).toThrow("app-main");
});
it("leaves the locale catalogues out of the reachable total", () => {
const s = stats({
"app-main": { assets: ["main.js"], reachableAssets: ["main.js", "locale-fr-json.abc.js"] },
});
expect(selectAppAssets(s).reachableAssets).toEqual(["main.js"]);
});
});
describe("selectChunkedSdkAssets", () => {
@@ -50,6 +58,15 @@ describe("selectChunkedSdkAssets", () => {
expect(result.includesAsyncChunks).toBe(false);
});
it("leaves the locale catalogues out of the reachable total", () => {
const withLocale = {
...base,
"embedding-sdk-chunked": { assets: ["a.js", RUNTIME], reachableAssets: ["a.js", "b.js", LOCALE, RUNTIME] },
};
const result = selectChunkedSdkAssets(stats(withLocale), { sdkAsyncChunksLoadable: true });
expect(result.reachableNames).toEqual(["boot.js", "a.js", "b.js"]);
});
it("collapses total to initial when the stats carry no reachable graph", () => {
const noGraph = { "embedding-sdk-chunked": { assets: ["a.js"] } };
const result = selectChunkedSdkAssets(stats(noGraph), { sdkAsyncChunksLoadable: true });

View File

@@ -0,0 +1,122 @@
import { appendFileSync, readFileSync, writeFileSync } from "node:fs";
import { join } from "node:path";
import type { TestPlan, TestPlanStats } from "./affected-tests";
type NumberStat = {
[K in keyof TestPlanStats]: TestPlanStats[K] extends number ? K : never;
}[keyof TestPlanStats];
type Suite = {
name: string;
filesKey: Exclude<keyof TestPlan, "stats">;
totalKey: NumberStat;
selectedKey: NumberStat;
pathsFile: string;
};
export const SUITES = {
unit: {
name: "unit",
filesKey: "fe_unit_specs_to_run",
totalKey: "unit_specs_all",
selectedKey: "unit_specs_to_run_usage",
pathsFile: "unit-specs.json",
},
loki: {
name: "Loki",
filesKey: "loki_stories_to_run",
totalKey: "loki_stories_all",
selectedKey: "loki_stories_to_run_usage",
pathsFile: "loki-stories.json",
},
e2e: {
name: "E2E",
filesKey: "e2e_specs_to_run",
totalKey: "e2e_specs_all",
selectedKey: "e2e_specs_to_run_usage",
pathsFile: "e2e-specs.json",
},
} satisfies Record<string, Suite>;
function isSuiteName(name: string): name is keyof typeof SUITES {
return Object.hasOwn(SUITES, name);
}
function isRecord(value: unknown): value is Record<string, unknown> {
return typeof value === "object" && value !== null;
}
function isStringArray(value: unknown): value is string[] {
return (
Array.isArray(value) && value.every((item) => typeof item === "string")
);
}
function isCount(value: unknown): value is number {
return typeof value === "number" && Number.isInteger(value) && value >= 0;
}
// null means a full run, and [] is an explicit selection of no tests.
function readSelection(planFile: string, suite: Suite): string[] | null {
const plan: unknown = JSON.parse(readFileSync(planFile, "utf8"));
const files = isRecord(plan) ? plan[suite.filesKey] : undefined;
const stats = isRecord(plan) && isRecord(plan.stats) ? plan.stats : {};
const total = stats[suite.totalKey];
if (
!isStringArray(files) ||
files.some((file) => file.length === 0) ||
!isCount(total) ||
files.length !== stats[suite.selectedKey] ||
files.length > total
) {
throw new Error(`Invalid ${suite.name} test selection`);
}
return files.length === total ? null : files;
}
export function prepareTestSelection(
suiteName: keyof typeof SUITES,
env: NodeJS.ProcessEnv = process.env,
) {
const suite: Suite = SUITES[suiteName];
const { RUNNER_TEMP, GITHUB_OUTPUT, PLAN_DOWNLOADED } = env;
if (!RUNNER_TEMP || !GITHUB_OUTPUT) {
throw new Error("RUNNER_TEMP and GITHUB_OUTPUT are required");
}
let files: string[] | null;
try {
if (PLAN_DOWNLOADED !== "success") {
throw new Error("Test plan download failed");
}
files = readSelection(join(RUNNER_TEMP, "test-plan/test-plan.json"), suite);
} catch (error) {
console.warn(
`::warning::Test plan unavailable or invalid, running the full ${suite.name} suite: ${error}`,
);
return;
}
if (files === null) {
appendFileSync(GITHUB_OUTPUT, "selection=full\n");
return;
}
const pathsFile = join(RUNNER_TEMP, suite.pathsFile);
writeFileSync(pathsFile, JSON.stringify(files));
appendFileSync(
GITHUB_OUTPUT,
`paths-file=${pathsFile}\nselection=${files.length === 0 ? "empty" : "narrowed"}\n`,
);
}
if (require.main === module) {
const suiteName = process.argv[2] ?? "";
if (!isSuiteName(suiteName)) {
throw new Error(
`Unknown test suite "${suiteName}", expected one of: ${Object.keys(SUITES).join(", ")}`,
);
}
prepareTestSelection(suiteName);
}

View File

@@ -0,0 +1,268 @@
import { spawnSync } from "node:child_process";
import {
mkdirSync,
mkdtempSync,
readFileSync,
rmSync,
writeFileSync,
} from "node:fs";
import { tmpdir } from "node:os";
import { join, resolve } from "node:path";
import { load } from "js-yaml";
import { SUITES, prepareTestSelection } from "./prepare-test-selection";
type Suite = (typeof SUITES)[keyof typeof SUITES];
type Step = {
id?: string;
if?: string;
uses?: string;
run?: string;
env?: Record<string, string>;
with?: Record<string, string | boolean>;
};
const SCRIPT = resolve(__dirname, "prepare-test-selection.ts");
const SUITE_NAMES = ["unit", "loki", "e2e"] satisfies (keyof typeof SUITES)[];
function loadWorkflow(file: string) {
// js-yaml returns an untyped value, and these repository workflows define jobs with steps.
return load(
readFileSync(resolve(__dirname, "../workflows", file), "utf8"),
) as {
jobs: Record<
string,
{ if?: string; outputs?: Record<string, string>; steps?: Step[] }
>;
};
}
function loadAction() {
// js-yaml returns an untyped value, and this repository action defines outputs and steps.
return load(
readFileSync(
resolve(__dirname, "../actions/prepare-test-selection/action.yml"),
"utf8",
),
) as {
outputs: Record<string, { value: string }>;
runs: { steps: Step[] };
};
}
describe("prepareTestSelection", () => {
const { loki } = SUITES;
let dir: string;
let env: NodeJS.ProcessEnv;
beforeEach(() => {
jest.spyOn(console, "warn").mockImplementation(() => {});
dir = mkdtempSync(join(tmpdir(), "test-selection-"));
mkdirSync(join(dir, "test-plan"));
writeFileSync(join(dir, "output"), "");
env = {
...process.env,
RUNNER_TEMP: dir,
GITHUB_OUTPUT: join(dir, "output"),
PLAN_DOWNLOADED: "success",
};
});
afterEach(() => {
jest.restoreAllMocks();
rmSync(dir, { recursive: true, force: true });
});
function plan(files: unknown, total = 2, suite: Suite = loki) {
return {
[suite.filesKey]: files,
stats: {
[suite.totalKey]: total,
[suite.selectedKey]: Array.isArray(files) ? files.length : 0,
},
};
}
function writePlan(value: unknown) {
writeFileSync(join(dir, "test-plan/test-plan.json"), JSON.stringify(value));
}
const output = () => readFileSync(join(dir, "output"), "utf8");
it.each([{ files: ["one.spec.cjs"] }, { files: [] }])(
"preserves an explicit selection: $files",
({ files }) => {
writePlan(plan(files));
prepareTestSelection("loki", env);
expect(output()).toBe(
`paths-file=${join(dir, loki.pathsFile)}\nselection=${files.length > 0 ? "narrowed" : "empty"}\n`,
);
expect(
JSON.parse(readFileSync(join(dir, loki.pathsFile), "utf8")),
).toEqual(files);
},
);
it("writes the selection when invoked through Bun in CI", () => {
writePlan(plan(["one.spec.cjs"]));
const result = spawnSync("bun", [SCRIPT, "loki"], {
env,
encoding: "utf8",
});
expect(result.status === 0 ? "" : result.stderr).toBe("");
expect(result.status).toBe(0);
expect(output()).toBe(
`paths-file=${join(dir, loki.pathsFile)}\nselection=narrowed\n`,
);
});
it("writes no paths file for a full selection", () => {
writePlan(plan(["one.spec.cjs", "two.spec.cjs"]));
prepareTestSelection("loki", env);
expect(output()).toBe("selection=full\n");
});
it.each([
null,
{},
plan("[]"),
plan([null]),
{ ...plan([]), [loki.filesKey]: ["one.spec.cjs"] },
plan([""]),
plan(["one.spec.cjs"], 0),
])("runs in full for an invalid plan: %j", (value) => {
writePlan(value);
prepareTestSelection("loki", env);
expect(console.warn).toHaveBeenCalledWith(
expect.stringContaining("::warning::"),
);
expect(output()).toBe("");
});
it.each([
{ scenario: "the test plan is missing", setup: () => {} },
{
scenario: "the test plan is truncated",
setup: () => {
writeFileSync(
join(dir, "test-plan/test-plan.json"),
`{"${loki.filesKey}":[`,
);
},
},
{
scenario: "the download failed",
setup: () => {
writePlan(plan([])); // Even a partial download must not be trusted.
env.PLAN_DOWNLOADED = "failure";
},
},
])("runs in full when $scenario", ({ setup }) => {
setup();
prepareTestSelection("loki", env);
expect(console.warn).toHaveBeenCalledWith(
expect.stringContaining("::warning::"),
);
expect(output()).toBe("");
});
it.each(SUITE_NAMES)(
"reads the %s selection from its own plan keys",
(suiteName) => {
const suite = SUITES[suiteName];
writePlan(plan(["one.spec.cjs"], 2, suite));
prepareTestSelection(suiteName, env);
expect(output()).toBe(
`paths-file=${join(dir, suite.pathsFile)}\nselection=narrowed\n`,
);
expect(
JSON.parse(readFileSync(join(dir, suite.pathsFile), "utf8")),
).toEqual(["one.spec.cjs"]);
},
);
it("ignores the other suites' selections", () => {
writePlan({
...plan(["one.spec.cjs", "two.spec.cjs"]),
[SUITES.unit.filesKey]: [],
[SUITES.e2e.filesKey]: [],
});
prepareTestSelection("loki", env);
expect(console.warn).not.toHaveBeenCalled();
expect(output()).toBe("selection=full\n");
});
it("rejects an unknown suite", () => {
const result = spawnSync("bun", [SCRIPT, "toString"], {
env,
encoding: "utf8",
});
expect(result.status).not.toBe(0);
expect(result.stderr).toContain('Unknown test suite "toString"');
});
});
describe("test selection workflow steps", () => {
const ACTION = "./.github/actions/prepare-test-selection";
const CONSUMERS: Record<string, { suite: keyof typeof SUITES }> = {
"frontend.yml": { suite: "unit" },
"loki.yml": { suite: "loki" },
};
it("skips Loki visual tests only for an empty story selection", () => {
const { jobs } = loadWorkflow("loki.yml");
expect(jobs["story-selection"].outputs?.selection).toBe(
"${{ steps.loki-selection.outputs.selection }}",
);
expect(jobs["visual-test"].if).toContain(
"needs.story-selection.outputs.selection != 'empty'",
);
expect(jobs["visual-test"].if).not.toMatch(/outputs\.[\w-]+ *[!=]= *'\d+'/);
});
it.each(Object.entries(CONSUMERS))(
"selects tests through the composite action in %s",
(file, { suite, ...inputs }) => {
const steps = Object.values(loadWorkflow(file).jobs).flatMap(
({ steps = [] }) => steps.filter((step) => step.uses === ACTION),
);
expect(steps.length).toBeGreaterThan(0);
for (const step of steps) {
expect(step.with).toEqual({
suite,
"test-plan-artifact-id": expect.stringMatching(/^\$\{\{ .+ \}\}$/),
...inputs,
});
}
},
);
it("downloads the test plan before preparing the selection", () => {
const { outputs, runs } = loadAction();
const [download, prepare] = runs.steps;
expect(download).toMatchObject({
uses: expect.stringMatching(/^actions\/download-artifact@/),
"continue-on-error": "${{ inputs.require-download != 'true' }}",
with: {
"artifact-ids": "${{ inputs.test-plan-artifact-id }}",
path: "${{ runner.temp }}/test-plan",
},
});
expect(prepare).toMatchObject({
env: { PLAN_DOWNLOADED: `\${{ steps.${download.id}.outcome }}` },
run: expect.stringContaining(
'prepare-test-selection.ts "${{ inputs.suite }}"',
),
});
expect(outputs).toMatchObject({
"paths-file": {
value: `\${{ steps.${prepare.id}.outputs.paths-file }}`,
},
selection: { value: `\${{ steps.${prepare.id}.outputs.selection }}` },
});
});
});

View File

@@ -98,6 +98,15 @@ merge_outputs() {
if [ -f "$jvm_dir/logs/test-log.json" ]; then
cp -a "$jvm_dir/logs/test-log.json" "$WORKSPACE/logs/test-log-p$idx.json" || true
fi
# Flamegraphs, when the run was profiled. Flat with a `p$idx-` prefix, not nested: the
# collect-profile action globs `results/*flamegraph.html` without descending.
if [ -d "$jvm_dir/target/clj-async-profiler/results" ]; then
mkdir -p "$WORKSPACE/target/clj-async-profiler/results"
for f in "$jvm_dir"/target/clj-async-profiler/results/*flamegraph.html; do
[ -e "$f" ] || continue
cp -a "$f" "$WORKSPACE/target/clj-async-profiler/results/p$idx-$(basename "$f")" || true
done
fi
done
}
trap merge_outputs EXIT

226
.github/scripts/story-files.unit.spec.ts vendored Normal file
View File

@@ -0,0 +1,226 @@
import { execFileSync } from "node:child_process";
import { mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join, resolve } from "node:path";
import micromatch, { isMatch } from "micromatch";
import { normalizeStories } from "storybook/internal/common";
import {
MAIN_APP_STORY_GLOBS,
getStories,
hasExplicitTitle,
} from "../../.storybook/story-files.cjs";
const ROOT = resolve(__dirname, "../..");
const tracked = execFileSync(
"git",
["ls-files", "-z", "--", "frontend", "enterprise/frontend"],
{ cwd: ROOT, encoding: "utf8", maxBuffer: 64 * 1024 * 1024 },
)
.split("\0")
.filter(Boolean);
const planned = micromatch(tracked, MAIN_APP_STORY_GLOBS, { dot: true }).sort();
describe("getStories", () => {
let tempDir: string;
let pathsFile: string;
beforeEach(() => {
tempDir = mkdtempSync(join(tmpdir(), "story-files-"));
pathsFile = join(tempDir, "stories.json");
});
afterEach(() => {
rmSync(tempDir, { recursive: true, force: true });
});
it("loads the main app stories and docs when there is no plan", () => {
const patterns = getStories({ pathsFile: "", filter: "" });
expect(
isMatch("../frontend/src/metabase/ui/Button.stories.tsx", patterns),
).toBe(true);
expect(isMatch("../frontend/src/metabase/ui/Button.mdx", patterns)).toBe(
true,
);
expect(
isMatch(
"../frontend/src/embedding-sdk-shared/Button.stories.tsx",
patterns,
),
).toBe(false);
});
it("builds only the files in the JSON plan", () => {
const files = planned.slice(0, 2);
writeFileSync(pathsFile, JSON.stringify(files));
expect(getStories({ pathsFile })).toEqual(
files.map((file) => `../${file}`),
);
});
it("builds a planned story with an explicit title", () => {
const story = join(tempDir, "Titled.stories.tsx");
writeFileSync(
story,
'const meta = { title: "Components/Titled" };\nexport default meta;\n',
);
writeFileSync(pathsFile, JSON.stringify([story]));
expect(getStories({ pathsFile })).toEqual([`../${story}`]);
});
it("rejects a planned story without an explicit title", () => {
const story = join(tempDir, "Untitled.stories.tsx");
writeFileSync(story, "export default { component: Untitled };\n");
writeFileSync(pathsFile, JSON.stringify([story]));
expect(() => getStories({ pathsFile })).toThrow(story);
});
it("rejects an untitled Alert story even though its args have a title", () => {
const source = readFileSync(
join(
ROOT,
"frontend/src/metabase/ui/components/feedback/Alert/Alert.stories.tsx",
),
"utf8",
).replace(' title: "Components/Feedback/Alert",\n', "");
const story = join(tempDir, "Alert.stories.tsx");
writeFileSync(story, source);
writeFileSync(pathsFile, JSON.stringify([story]));
expect(() => getStories({ pathsFile })).toThrow(story);
});
it("rejects an empty plan even when the CSV filter is set", () => {
writeFileSync(pathsFile, "[]");
expect(() =>
getStories({ pathsFile, filter: "frontend/src/Unselected.stories.tsx" }),
).toThrow("No stories selected");
});
it("preserves the stress test's CSV filter", () => {
expect(
getStories({
filter: "frontend/One.stories.tsx,frontend/Two.stories.tsx",
}),
).toEqual(["../frontend/One.stories.tsx", "../frontend/Two.stories.tsx"]);
});
it.each(["{", "null", '""', "{}", "[null]"])(
"rejects an invalid plan: %s",
(contents) => {
writeFileSync(pathsFile, contents);
expect(() => getStories({ pathsFile })).toThrow();
},
);
});
describe("hasExplicitTitle", () => {
it.each([
{
form: "double quotes",
source: 'export default {\n title: "Components/Button",\n};\n',
},
{
form: "single quotes",
source: "export default {\n title: 'Components/Button',\n};\n",
},
{
form: "a title constant",
source: 'const title = "Components/Button"; export default { title };',
},
{
form: "a meta constant",
source:
'const meta: Meta<typeof Button> = {\n title: "Components/Button",\n};\n\nexport default meta;\n',
},
{
form: "a satisfies expression",
source:
'const meta = { title: "Components/Button" } satisfies Meta<typeof Button>; export default meta;',
},
])("accepts a title in $form", ({ source }) => {
expect(hasExplicitTitle(source)).toBe(true);
});
it.each([
{
form: "no title",
source: "export default {\n component: Button,\n};\n",
},
{
form: "only a title control",
source:
'const argTypes = {\n title: {\n control: { type: "text" },\n },\n};\n\nexport default { component: Button, argTypes };\n',
},
{
form: "only a title variable",
source:
"export default {\n component: Button,\n args: {\n title: defaultTitle,\n },\n};\n",
},
{
form: "only a string-valued args title",
source: 'export default { args: { title: "Button label" } };',
},
{
form: "only another object's title",
source:
'const other = { title: "Other" }; export default { component: Button };',
},
{
form: "only a commented title",
source:
'export default {\n // title: "Components/Button",\n component: Button };',
},
{
form: "an empty title",
source: 'export default { title: "" };',
},
{
form: "a template literal unsupported by Storybook",
source: "export default { title: `Components/Button` };",
},
])("rejects $form", ({ source }) => {
expect(hasExplicitTitle(source)).toBe(false);
});
});
describe("MAIN_APP_STORY_GLOBS", () => {
it("matches the same tracked files as Storybook's own story matcher", () => {
const specifiers = normalizeStories(getStories(), {
configDir: join(ROOT, ".storybook"),
workingDir: ROOT,
});
const built = tracked
.filter(
(file) =>
!file.endsWith(".mdx") &&
specifiers.some((specifier) =>
specifier.importPathMatcher.test(`./${file}`),
),
)
.sort();
expect(planned.length).toBeGreaterThan(0);
expect(planned).toEqual(built);
});
it("matches no docs entries", () => {
expect(planned.filter((file) => file.endsWith(".mdx"))).toEqual([]);
});
it("matches only stories with an explicit title", () => {
const untitled = planned.filter(
(file) => !hasExplicitTitle(readFileSync(join(ROOT, file), "utf8")),
);
expect(untitled).toEqual([]);
});
});

View File

@@ -23,9 +23,13 @@ interface Condition {
coldLargestPaintMs: number;
coldPageReadyMs: number;
warmPageReadyMs: number;
warmLargestPaintMs: number;
steadyLargestPaintMs: number;
steadyPageReadyMs: number;
locale: string;
scripts: number;
scriptKb: number;
cssKb: number;
totalKb: number;
runs: number;
}
@@ -69,9 +73,13 @@ function buildRows(
"Cold largest paint ms": condition.coldLargestPaintMs,
"Cold page ready ms": condition.coldPageReadyMs,
"Warm page ready ms": condition.warmPageReadyMs,
"Warm largest paint ms": condition.warmLargestPaintMs,
"Steady largest paint ms": condition.steadyLargestPaintMs,
"Steady page ready ms": condition.steadyPageReadyMs,
Locale: condition.locale,
Scripts: condition.scripts,
"Script kb": condition.scriptKb,
"Css kb": condition.cssKb,
"Total kb": condition.totalKb,
Runs: condition.runs,
}));

View File

@@ -0,0 +1,73 @@
/**
* Collect some stats about our e2e tests:
* - total number of spec files
* - total time to run all specs
* - average spec runtime
* - max spec runtime
*
*/
import { readFileSync } from "node:fs";
import { importStats, StatsRow } from "./stats-import";
const TABLE = "e2e_test_timings";
export interface Timings {
durations?: { spec?: string; duration?: unknown }[];
}
export const buildTestTimingsRow = (
timestamp: string,
timings: Timings,
): StatsRow => {
const durations = timings.durations ?? [];
const specCount = durations.length;
if (specCount === 0) {
throw new Error("No durations found.");
}
let totalRunTime = 0;
let maxSpecRuntime = 0;
for (const { duration } of durations) {
if (typeof duration !== "number") {
continue;
}
totalRunTime += duration;
maxSpecRuntime = duration > maxSpecRuntime ? duration : maxSpecRuntime;
}
const averageSpecRuntime = Math.round(totalRunTime / specCount);
return {
date: timestamp,
spec_count: specCount,
total_runtime: totalRunTime,
avg_spec_runtime: averageSpecRuntime,
max_spec_runtime: maxSpecRuntime,
};
};
export async function main() {
const path = process.env.TIMINGS_PATH || "e2e/support/timings.json";
try {
const timings = JSON.parse(readFileSync(path, "utf-8")) as Timings;
const row = buildTestTimingsRow(new Date().toISOString(), timings);
console.log(row);
await importStats({ table: TABLE, rows: [row] });
console.log(`Successfully uploaded e2e test timings to ${TABLE}.`);
} catch (error) {
const message = error instanceof Error ? error.message : String(error);
console.log(
`::warning::Uploading e2e test timings from ${path} failed; leaving run green: ${message}`,
);
}
}
if ((import.meta as ImportMeta & { main?: boolean }).main) {
main();
}

View File

@@ -0,0 +1,61 @@
import { buildTestTimingsRow } from "./upload-e2e-test-timings";
const timestamp = "2026-06-30T00:00:00.000Z";
describe("buildTestTimingsRow", () => {
it("summarizes spec durations", () => {
const row = buildTestTimingsRow(timestamp, {
durations: [
{ spec: "a.cy.spec.js", duration: 1000 },
{ spec: "b.cy.spec.js", duration: 3000 },
{ spec: "c.cy.spec.js", duration: 2000 },
],
});
expect(row).toEqual({
date: timestamp,
spec_count: 3,
total_runtime: 6000,
avg_spec_runtime: 2000,
max_spec_runtime: 3000,
});
});
it("rounds the average runtime", () => {
const row = buildTestTimingsRow(timestamp, {
durations: [
{ spec: "a.cy.spec.js", duration: 1000 },
{ spec: "b.cy.spec.js", duration: 1001 },
],
});
expect(row.avg_spec_runtime).toBe(1001);
});
it("counts specs with a non-numeric duration but ignores their value", () => {
const row = buildTestTimingsRow(timestamp, {
durations: [
{ spec: "a.cy.spec.js", duration: 1000 },
{ spec: "b.cy.spec.js", duration: null },
{ spec: "c.cy.spec.js" },
],
});
expect(row).toEqual({
date: timestamp,
spec_count: 3,
total_runtime: 1000,
avg_spec_runtime: 333,
max_spec_runtime: 1000,
});
});
it("throws when there are no durations", () => {
expect(() => buildTestTimingsRow(timestamp, { durations: [] })).toThrow(
"No durations found.",
);
expect(() => buildTestTimingsRow(timestamp, {})).toThrow(
"No durations found.",
);
});
});

View File

@@ -11,30 +11,6 @@ on:
type: boolean
default: false
# Run one app-DB job on demand, against the ref you dispatch on, optionally under a profiler.
# Everything else about the job -- services, matrix legs, test args, quarantine gating -- is
# whatever the normal CI run uses, because it is the same job.
workflow_dispatch:
inputs:
app-db:
description: 'Which app-DB job to run. Only this one runs; the others are skipped.'
type: choice
required: true
options:
- mariadb
- mysql
- postgres
tests:
description: 'Optional: run only these tests, comma- or space-separated, e.g. metabase.db.custom-migrations-test/some-test. Bare namespaces work too. Blank runs the job''s usual set.'
type: string
required: false
default: ''
profile:
description: 'Profile the test run and attach a flamegraph to the job. Adds sampling overhead.'
type: boolean
required: false
default: false
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }}-app-db
cancel-in-progress: true
@@ -45,14 +21,6 @@ concurrency:
env:
CI_CONDUCTOR_BASE_URL: ${{ vars.CI_CONDUCTOR_BASE_URL }}
CI_CONDUCTOR_WEBHOOK_SECRET: ${{ secrets.CI_CONDUCTOR_WEBHOOK_SECRET }}
# Both read by the test-driver action, which is what actually runs the tests. Set here rather than
# passed as inputs to each job so that turning either on is one line, not one line per job.
# Empty (and so off) on workflow_call, where neither input exists.
#
# ONLY_TESTS replaces the job's own test selection, so every matrix leg of the chosen job runs the
# whole list -- each app DB has three.
PROFILE_TESTS: ${{ inputs.profile }}
ONLY_TESTS: ${{ inputs.tests }}
jobs:
app-db-matrix:
@@ -64,8 +32,9 @@ jobs:
mariadb: ${{ steps.versions.outputs.mariadb }}
mysql: ${{ steps.versions.outputs.mysql }}
postgres: ${{ steps.versions.outputs.postgres }}
shards: ${{ steps.versions.outputs.shards }}
steps:
- name: Select versions to test
- name: Select versions and shards to test
id: versions
uses: actions/github-script@v9
env:
@@ -93,9 +62,9 @@ jobs:
],
mysql: [
{
name: 'MySQL 8.0',
image: 'mysql:8.0',
'junit-name': 'be-tests-mysql-8-0-ee',
name: 'MySQL 8',
image: 'mysql:8.4',
'junit-name': 'be-tests-mysql-8-ee',
env: { 'enable-ssl-tests': 'false', 'enable-aws-iam-tests': 'false' },
latest: false
},
@@ -133,35 +102,42 @@ jobs:
core.setOutput(db, JSON.stringify(selected));
}
const commonExcludes = ":mb/driver-tests :mb/transforms-python-test :mb/old-migrations-test";
const shards = [
{
name: 'Enterprise Tests',
'build-static-viz': false,
'test-args': `:only '"enterprise/backend/test"' :exclude-tags '[${commonExcludes}]'`,
},
{
name: 'EE App DB Tests (Part 1)', // migrations shard
'build-static-viz': true,
'test-args': `:only '"test"' :only-tags '[:mb/app-db-migrations-test]' :exclude-tags '[${commonExcludes}]'`,
},
{
name: 'EE App DB Tests (Part 2)',
'build-static-viz': true,
'test-args': `:only '"test"' :partition/total 2 :partition/index 0 :exclude-tags '[${commonExcludes} :mb/app-db-migrations-test]'`,
},
{
name: 'EE App DB Tests (Part 3)',
'build-static-viz': true,
'test-args': `:only '"test"' :partition/total 2 :partition/index 1 :exclude-tags '[${commonExcludes} :mb/app-db-migrations-test]'`,
},
];
core.setOutput('shards', JSON.stringify(shards));
be-tests-mariadb:
needs: app-db-matrix
if: ${{ !inputs.skip && (inputs.app-db == '' || inputs.app-db == 'mariadb') }}
if: ${{ !inputs.skip && (github.event.inputs.db == '' || github.event.inputs.db == 'mariadb') }}
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
timeout-minutes: 40
strategy:
fail-fast: false
matrix:
version: ${{ fromJSON(needs.app-db-matrix.outputs.mariadb) }}
job:
- name: Enterprise Tests
build-static-viz: false
test-args: >-
:only '"enterprise/backend/test"'
exclude-tag: ':mb/driver-tests :mb/transforms-python-test'
- name: EE App DB Tests (Part 1)
build-static-viz: true
test-args: >-
:only '"test"'
:partition/total 2
:partition/index 0
exclude-tag: ':mb/driver-tests :mb/transforms-python-test'
- name: EE App DB Tests (Part 2)
build-static-viz: true
test-args: >-
:only '"test"'
:partition/total 2
:partition/index 1
exclude-tag: ':mb/driver-tests :mb/transforms-python-test'
job: ${{ fromJSON(needs.app-db-matrix.outputs.shards) }}
services:
mysql:
@@ -175,10 +151,6 @@ jobs:
MARIADB_ALLOW_EMPTY_ROOT_PASSWORD: true
MARIADB_DATABASE: mb_test
env:
# __ADDITIONAL_EXCLUDED_TAG__: ${{ !inputs.force-run && ':mb/old-migrations-test' || '' }}
# GDGT-3126: re-enable and properly fix old migrations tests
__ADDITIONAL_EXCLUDED_TAG__: ':mb/old-migrations-test'
# actual serious env vars below
CI: 'true'
DRIVERS: mysql
MB_DB_TYPE: mysql
@@ -205,7 +177,8 @@ jobs:
# for historic reasons (I don't want to mess around with required jobs) the job name should be something like
# "be-tests-mariadb-10-6-ee (0)"
name: "${{ matrix.version.name }} ${{ matrix.job.name }}"
steps:
# Shared by every app DB job below via `*app-db-test-steps`.
steps: &app-db-test-steps
- uses: actions/checkout@v6
- name: Test ${{ matrix.version.name }}
id: test-driver
@@ -214,39 +187,18 @@ jobs:
build-static-viz: ${{ matrix.job.build-static-viz }}
junit-name: ${{ matrix.version.junit-name }}
artifact-suffix: ${{ matrix.job.name }}
test-args: >-
${{ matrix.job.test-args }}
:exclude-tags '[ ${{ matrix.job.exclude-tag }} ${{ env.__ADDITIONAL_EXCLUDED_TAG__ }}]'
test-args: ${{ matrix.job.test-args }}
be-tests-mysql:
needs: app-db-matrix
if: ${{ !inputs.skip && (inputs.app-db == '' || inputs.app-db == 'mysql') }}
if: ${{ !inputs.skip && (github.event.inputs.db == '' || github.event.inputs.db == 'mysql') }}
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
timeout-minutes: 40
strategy:
fail-fast: false
matrix:
version: ${{ fromJSON(needs.app-db-matrix.outputs.mysql) }}
job:
- name: Enterprise Tests
build-static-viz: false
test-args: >-
:only '"enterprise/backend/test"'
exclude-tag: ':mb/driver-tests :mb/transforms-python-test'
- name: EE App DB Tests (Part 1)
build-static-viz: true
test-args: >-
:only '"test"'
:partition/total 2
:partition/index 0
exclude-tag: ':mb/driver-tests :mb/transforms-python-test'
- name: EE App DB Tests (Part 2)
build-static-viz: true
test-args: >-
:only '"test"'
:partition/total 2
:partition/index 1
exclude-tag: ':mb/driver-tests :mb/transforms-python-test'
job: ${{ fromJSON(needs.app-db-matrix.outputs.shards) }}
services:
mysql:
@@ -260,10 +212,6 @@ jobs:
MYSQL_ALLOW_EMPTY_PASSWORD: true
MYSQL_DATABASE: mb_test
env:
# __ADDITIONAL_EXCLUDED_TAG__: ${{ !inputs.force-run && ':mb/old-migrations-test' || '' }}
# GDGT-3126: re-enable and properly fix old migrations tests
__ADDITIONAL_EXCLUDED_TAG__: ':mb/old-migrations-test'
# actual serious env vars below
CI: 'true'
DRIVERS: mysql
MB_DB_TYPE: mysql
@@ -298,54 +246,20 @@ jobs:
# for historic reasons (I don't want to mess around with required jobs) the job name should be something like
# "be-tests-mariadb-10-6-ee (0)"
name: "${{ matrix.version.name }} ${{ matrix.job.name }}"
steps:
- uses: actions/checkout@v6
- name: Test ${{ matrix.version.name }}
id: test-driver
uses: ./.github/actions/test-driver
with:
build-static-viz: ${{ matrix.job.build-static-viz }}
junit-name: ${{ matrix.version.junit-name }}
artifact-suffix: ${{ matrix.job.name }}
test-args: >-
${{ matrix.job.test-args }}
:exclude-tags '[ ${{ matrix.job.exclude-tag }} ${{ env.__ADDITIONAL_EXCLUDED_TAG__ }}]'
steps: *app-db-test-steps
be-tests-postgres:
needs: app-db-matrix
if: ${{ !inputs.skip && (inputs.app-db == '' || inputs.app-db == 'postgres') }}
if: ${{ !inputs.skip && (github.event.inputs.db == '' || github.event.inputs.db == 'postgres') }}
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
timeout-minutes: 40
strategy:
fail-fast: false
matrix:
version: ${{ fromJSON(needs.app-db-matrix.outputs.postgres) }}
job:
- name: Enterprise Tests
build-static-viz: false
test-args: >-
:only '"enterprise/backend/test"'
exclude-tag: ':mb/driver-tests :mb/transforms-python-test'
- name: EE App DB Tests (Part 1)
build-static-viz: true
test-args: >-
:only '"test"'
:partition/total 2
:partition/index 0
exclude-tag: ':mb/driver-tests :mb/transforms-python-test'
- name: EE App DB Tests (Part 2)
build-static-viz: true
test-args: >-
:only '"test"'
:partition/total 2
:partition/index 1
exclude-tag: ':mb/driver-tests :mb/transforms-python-test'
job: ${{ fromJSON(needs.app-db-matrix.outputs.shards) }}
name: "${{ matrix.version.name }} ${{ matrix.job.name }}"
env:
# __ADDITIONAL_EXCLUDED_TAG__: ${{ !inputs.force-run && ':mb/old-migrations-test' || '' }}
# GDGT-3126: re-enable and properly fix old migrations tests
__ADDITIONAL_EXCLUDED_TAG__: ':mb/old-migrations-test'
# actual serious env vars below
CI: 'true'
DRIVERS: postgres
MB_DB_TYPE: postgres
@@ -374,18 +288,7 @@ jobs:
POSTGRES_USER: mb_test
POSTGRES_DB: mb_test
POSTGRES_HOST_AUTH_METHOD: trust
steps:
- uses: actions/checkout@v6
- name: Test ${{ matrix.version.name }} (${{ matrix.job.name }})
id: test-driver
uses: ./.github/actions/test-driver
with:
build-static-viz: ${{ matrix.job.build-static-viz }}
junit-name: ${{ matrix.version.junit-name }}
artifact-suffix: ${{ matrix.job.name }}
test-args: >-
${{ matrix.job.test-args }}
:exclude-tags '[ ${{ matrix.job.exclude-tag }} ${{ env.__ADDITIONAL_EXCLUDED_TAG__ }}]'
steps: *app-db-test-steps
app-db-tests-result:
needs:

View File

@@ -23,22 +23,6 @@ on:
type: boolean
default: false
# Run the backend test suite on demand, against the ref you dispatch on, optionally under a
# profiler. Everything else about the jobs -- matrix legs, tag filters, quarantine gating -- is
# whatever the normal CI run uses, because they are the same jobs.
workflow_dispatch:
inputs:
tests:
description: 'Optional: run only these tests, comma- or space-separated, e.g. metabase.query-processor-test/some-test. Bare namespaces work too. Blank runs each leg''s usual set.'
type: string
required: false
default: ''
profile:
description: 'Profile the test run and attach a flamegraph to the job. Adds sampling overhead.'
type: boolean
required: false
default: false
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }}-backend
cancel-in-progress: true
@@ -49,13 +33,6 @@ concurrency:
env:
CI_CONDUCTOR_BASE_URL: ${{ vars.CI_CONDUCTOR_BASE_URL }}
CI_CONDUCTOR_WEBHOOK_SECRET: ${{ secrets.CI_CONDUCTOR_WEBHOOK_SECRET }}
# Both read by the be-tests run step below. Set here rather than passed per job so that turning
# either on is one line. Empty (and so off) on workflow_call, where neither input exists.
#
# ONLY_TESTS replaces the leg's own test selection, so every matrix leg runs the whole list: the
# partition args no longer apply, and legs that hold none of the named tests run nothing.
PROFILE_TESTS: ${{ inputs.profile }}
ONLY_TESTS: ${{ inputs.tests }}
jobs:
project-checks:
@@ -374,6 +351,16 @@ jobs:
else
$BASE :exclude-tags '[:mb/driver-tests ${{ env.__ADDITIONAL_EXCLUDED_TAG__ }}]' ${{ matrix.job.test-args }}
fi
env:
# Both derived from test.profile-backend.yml, the sole dispatcher; its runs always profile. On a
# normal CI run there are no dispatch inputs, so these render `false` and empty and the
# leg's own selection stands.
#
# ONLY_TESTS replaces the leg's own test selection, so every matrix leg runs the whole
# list: the partition args no longer apply, and legs that hold none of the named tests run
# nothing.
PROFILE_TESTS: ${{ github.event.inputs.suite != '' }}
ONLY_TESTS: ${{ github.event.inputs.tests }}
# Keep this step continue-on-error because the quarantine logic controls whether it ultimately fails.
continue-on-error: true

View File

@@ -8,11 +8,15 @@ on:
description: Metabase edition (ee or oss)
type: string
required: true
registry:
description: Registry the image is pushed to, either "docker" or "ecr".
type: string
default: docker
repo:
description: Docker owner and repo, e.g. "metabase/metabase".
description: Repository name the image is pushed to, e.g. "metabase-enterprise".
type: string
tag:
description: Docker tag, e.g. v1.87.2.
description: Image tag, e.g. v1.87.2.
type: string
commit:
type: string
@@ -21,6 +25,10 @@ on:
default: false
build-args:
type: string
platforms:
description: Platforms the image is built for.
type: string
default: linux/amd64,linux/arm64
workflow_dispatch:
inputs:
edition:
@@ -30,11 +38,18 @@ on:
- ee
- oss
required: true
registry:
description: Registry the image is pushed to
type: choice
options:
- docker
- ecr
default: docker
repo:
description: Docker owner and repo, e.g. "metabase/metabase".
description: Repository name the image is pushed to, e.g. "metabase-enterprise".
type: string
tag:
description: Docker tag, e.g. v1.87.2.
description: Image tag, e.g. v1.87.2.
type: string
commit:
description: the full commit hash
@@ -43,6 +58,10 @@ on:
description: Whether this is a release artifact (uses metabase-release- prefix)
type: boolean
default: false
platforms:
description: Platforms the image is built for.
type: string
default: linux/amd64,linux/arm64
jobs:
containerize:
@@ -50,95 +69,103 @@ jobs:
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
timeout-minutes: 15
outputs:
platforms: ${{ steps.platforms.outputs.result }}
repo: ${{ steps.image.outputs.repo }}
tag: ${{ steps.image.outputs.tag }}
full_image: ${{ steps.image.outputs.full_image }}
env:
AWS_REGION: us-east-1
steps:
- uses: actions/checkout@v6
with:
fetch-depth: 1
# we want the dockerfile from the commit we're building
- run: |
git fetch --depth=1 origin ${{ inputs.commit }}
git checkout ${{ inputs.commit }} -- bin/docker
- name: Retrieve uberjar artifact
uses: ./.github/actions/fetch-artifact
with:
commit: ${{ inputs.commit }}
edition: ${{ inputs.edition }}
release: ${{ inputs.release }}
- name: Prepare release scripts
uses: ./.github/actions/prepare-release-scripts
extract-path: bin/docker
- name: Resolve image repo and tag
id: image
uses: ./.github/actions/get-docker-info
with:
registry: ${{ inputs.registry }}
repo: ${{ inputs.repo }}
tag: ${{ inputs.tag }}
edition: ${{ inputs.edition }}
- name: Get platforms
id: platforms
uses: actions/github-script@v9
with:
result-encoding: string
script: | # js
const { getBuildRequirements, isValidVersionString} = require('${{ github.workspace }}/release/dist/index.cjs');
const version = "${{ steps.image.outputs.tag }}".split('-')[0];
aws-account-id: ${{ secrets.PR_ENV_AWS_ACCOUNT_ID }}
aws-region: ${{ env.AWS_REGION }}
dockerhub-owner: ${{ vars.DOCKERHUB_OWNER }}
- name: Configure AWS credentials
if: ${{ inputs.registry == 'ecr' }}
uses: aws-actions/configure-aws-credentials@d979d5b3a71173a29b74b5b88418bfda9437d885 # v6.1.1
with:
role-to-assume: ${{ secrets.PR_ENV_IAM_ROLE }}
role-session-name: GitHub_to_AWS_via_FederatedOIDC
aws-region: ${{ env.AWS_REGION }}
- name: Login to Amazon ECR
if: ${{ inputs.registry == 'ecr' }}
uses: aws-actions/amazon-ecr-login@183a1442edf41672e66566b7fc560e297a290896 # v2
with:
registries: "${{ secrets.PR_ENV_AWS_ACCOUNT_ID }}"
if (!isValidVersionString(version)) {
return "linux/amd64,linux/arm64";
}
// versions before 53 need to be handled differently
const buildInfo = getBuildRequirements(version);
return buildInfo.platforms;
- name: Move the Uberjar to the context dir
run: mv ./metabase.jar bin/docker/.
- name: Login to Docker Hub
if: ${{ inputs.registry != 'ecr' }}
uses: docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee # v4.2.0
with:
username: ${{ inputs.release && secrets.DOCKERHUB_RELEASE_USERNAME || secrets.DOCKERHUB_USERNAME }}
password: ${{ inputs.release && secrets.DOCKERHUB_RELEASE_TOKEN || secrets.DOCKERHUB_TOKEN }}
- name: Set up QEMU
if: contains(steps.platforms.outputs.result, 'arm')
uses: docker/setup-qemu-action@ce360397dd3f832beb865e1373c09c0e9f86d70a # v4.0.0
- name: Set up Docker Buildx
if: contains(steps.platforms.outputs.result, 'arm')
uses: docker/setup-buildx-action@d7f5e7f509e45cec5c76c4d5afdd7de93d0b3df5 # v4.1.0
- name: Build multi-arch container
if: contains(steps.platforms.outputs.result, 'arm')
- name: Build and Push Images
uses: docker/build-push-action@f9f3042f7e2789586610d6e8b85c8f03e5195baf # v7.2.0
with:
context: bin/docker/.
platforms: ${{ steps.platforms.outputs.result }}
platforms: ${{ inputs.platforms }}
network: host
tags: ${{ steps.image.outputs.full_image }}
no-cache: true
push: true
build-args: ${{ inputs.build-args }}
# we need to do this to get the nice fallback behavior of the legacy manifest file
- name: Build container (x86 only)
if: contains(steps.platforms.outputs.result, 'arm') == false
env:
BUILD_ARGS: ${{ inputs.build-args }}
FULL_IMAGE: ${{ steps.image.outputs.full_image }}
run: | # bash
args=()
while IFS= read -r line; do
[ -n "$line" ] && args+=(--build-arg "$line")
done <<< "$BUILD_ARGS"
- name: Post link to image on DockerHub
if: ${{ inputs.registry != 'ecr' && github.event_name == 'pull_request' }}
uses: marocchino/sticky-pull-request-comment@0ea0beb66eb9baf113663a64ec522f60e49231c0 # v3.0.4
with:
header: docker-build-status-${{ inputs.edition }}
message: |
✅ Docker build completed for commit ${{ inputs.commit || github.event.pull_request.head.sha || github.sha }}
docker build \
-t "$FULL_IMAGE" \
"${args[@]}" \
--network=host \
--push \
bin/docker/.
The uberjar has been built and containerized successfully.
verify-docker-pull:
**Branch:** `${{ github.head_ref || github.ref_name }}`
**Docker Image:** `${{ steps.image.outputs.full_image }}`
**Docker Hub:** [View on Docker Hub](https://hub.docker.com/r/${{ steps.image.outputs.repo }}/tags?name=${{ steps.image.outputs.tag }})
**Triggered by:** @${{ github.actor }}
Pull the image:
```bash
docker pull ${{ steps.image.outputs.full_image }}
```
verify-docker-pull-amd64:
needs: containerize
name: verify-docker-pull (${{ inputs.edition }})
if: >-
inputs.release &&
inputs.registry != 'ecr'
name: verify-docker-pull-amd64 (${{ inputs.edition }})
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
timeout-minutes: 15
steps:
@@ -158,7 +185,10 @@ jobs:
verify-docker-pull-arm:
needs: containerize
if: contains(needs.containerize.outputs.platforms, 'linux/arm64')
if: >-
inputs.release &&
inputs.registry != 'ecr' &&
contains(inputs.platforms, 'linux/arm64')
name: verify-docker-pull-arm (${{ inputs.edition }})
runs-on: arm-ubuntu-24-2core
timeout-minutes: 15
@@ -176,34 +206,3 @@ jobs:
- name: Wait for Metabase to start
run: while ! curl -s 'http://localhost:3000/api/health' | grep '{"status":"ok"}'; do sleep 1; done
timeout-minutes: 5
post-docker-info-comment:
needs: [containerize, verify-docker-pull, verify-docker-pull-arm]
# Only post a comment if all the steps pass and this is a PR.
if: |
!cancelled() &&
!failure() &&
!inputs.release &&
github.event_name == 'pull_request'
name: post-docker-info-comment (${{ inputs.edition }})
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
timeout-minutes: 5
steps:
- name: Post PR comment
uses: marocchino/sticky-pull-request-comment@0ea0beb66eb9baf113663a64ec522f60e49231c0 # v3.0.4
with:
header: docker-build-status-${{ inputs.edition }}
message: |
✅ Docker build completed for commit ${{ inputs.commit || github.event.pull_request.head.sha || github.sha }}
The uberjar has been built and containerized successfully.
**Branch:** `${{ github.head_ref || github.ref_name }}`
**Docker Image:** `${{ needs.containerize.outputs.full_image }}`
**Docker Hub:** [View on Docker Hub](https://hub.docker.com/r/${{ needs.containerize.outputs.repo }}/tags?name=${{ needs.containerize.outputs.tag }})
**Triggered by:** @${{ github.actor }}
Pull the image:
```bash
docker pull ${{ needs.containerize.outputs.full_image }}
```

View File

@@ -10,45 +10,6 @@ on:
description: Run the full suite regardless of the diff (master / release branches, ci:run-all label).
type: boolean
default: false
# Run one driver job on demand, against the ref you dispatch on, optionally under a profiler.
# Everything else about the job -- credentials, services, test args, quarantine gating -- is
# whatever the normal CI run uses, because it is the same job.
workflow_dispatch:
inputs:
driver:
description: 'Which driver job to run. Only this one runs; the usual decision rules are bypassed.'
type: choice
required: true
options:
- athena
- bigquery
- clickhouse
- databricks
- druid-jdbc
- h2
- mongo
- mongo-sharded-cluster
- mongo-ssl
- mysql-mariadb
- oracle
- postgres
- presto-jdbc
- redshift
- snowflake
- sparksql
- sqlite
- sqlserver
- vertica
tests:
description: 'Optional: run only these tests, comma- or space-separated, e.g. metabase.driver.bigquery-cloud-sdk-test/sync-test. Bare namespaces work too. Blank runs the job''s usual set.'
type: string
required: false
default: ''
profile:
description: 'Profile the test run and attach a flamegraph to the job. Adds sampling overhead.'
type: boolean
required: false
default: false
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }}-drivers
@@ -60,14 +21,6 @@ concurrency:
env:
CI_CONDUCTOR_BASE_URL: ${{ vars.CI_CONDUCTOR_BASE_URL }}
CI_CONDUCTOR_WEBHOOK_SECRET: ${{ secrets.CI_CONDUCTOR_WEBHOOK_SECRET }}
# Both read by the test-driver action, which is what actually runs the tests. Set here rather than
# passed as inputs to each job so that turning either on is one line, not one line per driver.
# Empty (and so off) on workflow_call, where neither input exists.
#
# ONLY_TESTS replaces the job's own test selection, so every matrix leg of the chosen driver runs
# the whole list -- BigQuery, for instance, has three legs.
PROFILE_TESTS: ${{ inputs.profile }}
ONLY_TESTS: ${{ inputs.tests }}
jobs:
# This job determines which driver tests should run based on PR context.
@@ -117,7 +70,7 @@ jobs:
--force-run="${FORCE_RUN}"
--pr-labels="${{ join(github.event.pull_request.labels.*.name, ',') }}"
--skip="${{ inputs.skip }}"
--only-driver="${ONLY_DRIVER}"
--only-driver="${PROFILE_DRIVER}"
)
echo "Running:"
echo "./bin/mage -driver-decisions ${MAGE_ARGS[*]}"
@@ -132,7 +85,9 @@ jobs:
env:
BASE_REF: ${{ github.event.pull_request.base.ref || github.ref_name }}
FORCE_RUN: ${{ inputs.force-run }}
ONLY_DRIVER: ${{ inputs.driver }}
# Set only by test.profile-backend.yml. Absent on a normal CI run, which leaves the decision rules
# alone; mage rejects a name that is not a driver.
PROFILE_DRIVER: ${{ github.event.inputs.db }}
shell: bash
be-tests-athena-ee:
@@ -450,9 +405,9 @@ jobs:
env:
enable-ssl-tests: 'false'
enable-aws-iam-tests: 'false'
- name: MySQL 8.0
junit-name: be-tests-mysql-8-0-ee
image: mysql:8.0
- name: MySQL 8
junit-name: be-tests-mysql-8-ee
image: mysql:8.4
env:
enable-ssl-tests: 'false'
enable-aws-iam-tests: 'false'

View File

@@ -1,42 +1,86 @@
name: CI for PR Review ENV
name: Deploy PR Environment
run-name: Building Dynamic PR Environment for ${{ github.ref_name }} by @${{ github.actor }}
on:
workflow_call:
inputs:
self_hosting:
description: "Deploy self-hosting stack"
required: false
type: boolean
default: false
analytics_dev_mode:
description: "Enable analytics development mode"
required: false
type: boolean
default: false
stats_dump:
description: "Load a recent app DB stats dump on first env creation"
required: false
type: boolean
default: false
pull_request:
types: [ opened, labeled, synchronize, reopened ]
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
permissions:
id-token: write
contents: read
actions: read
pull-requests: write
jobs:
build:
if: github.event.pull_request.head.repo.full_name == github.repository
name: Build Metabase Docker image
uses: ./.github/workflows/build-private-ee-docker-image.yml
secrets:
PR_ENV_IAM_ROLE: ${{ secrets.PR_ENV_IAM_ROLE }}
PR_ENV_AWS_ACCOUNT_ID: ${{ secrets.PR_ENV_AWS_ACCOUNT_ID }}
deploy_pr:
if: >-
${{
github.event.pull_request.head.repo.full_name == 'metabase/metabase' &&
(
github.event.action != 'labeled' ||
contains(fromJSON('["PR-Env", "PR-Env-self-hosting", "PR-Env-analytics-dev", "PR-Env-Stats-Dump"]'), github.event.label.name)
) &&
(
contains(github.event.pull_request.labels.*.name, 'PR-Env') ||
contains(github.event.pull_request.labels.*.name, 'PR-Env-self-hosting') ||
contains(github.event.pull_request.labels.*.name, 'PR-Env-analytics-dev') ||
contains(github.event.pull_request.labels.*.name, 'PR-Env-Stats-Dump')
)
}}
uses: ./.github/workflows/uberjar.yml
secrets: inherit
containerize:
needs: [build]
uses: ./.github/workflows/containerize-jar.yml
secrets: inherit
with:
edition: ee
registry: ecr
repo: metabase-enterprise
tag: ${{ github.event.pull_request.head.sha }}
commit: ${{ github.event.pull_request.head.sha }}
platforms: linux/amd64
deploy:
needs: [containerize]
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
name: PR Review ENV
permissions:
id-token: write
contents: read
timeout-minutes: 60
outputs:
self_hosting: ${{ steps.label.outputs.self_hosting }}
analytics_dev_mode: ${{ steps.label.outputs.analytics_dev_mode }}
stats_dump: ${{ steps.label.outputs.stats_dump }}
steps:
- name: Checkout source code
uses: actions/checkout@v6
- name: Resolve Labels
id: label
env:
LABELS: ${{ toJSON(github.event.pull_request.labels.*.name) }}
run: | # bash
self_hosting=false
analytics_dev_mode=false
stats_dump=false
while IFS= read -r label; do
case "$label" in
PR-Env-self-hosting) self_hosting=true ;;
PR-Env-analytics-dev) analytics_dev_mode=true ;;
PR-Env-Stats-Dump) stats_dump=true ;;
esac
done < <(jq -r '.[]' <<< "$LABELS")
echo "self_hosting=${self_hosting} analytics_dev_mode=${analytics_dev_mode} stats_dump=${stats_dump}"
{
echo "self_hosting=${self_hosting}"
echo "analytics_dev_mode=${analytics_dev_mode}"
echo "stats_dump=${stats_dump}"
} >> "$GITHUB_OUTPUT"
- name: Tailscale
uses: tailscale/github-action@306e68a486fd2350f2bfc3b19fcd143891a4a2d8 # v4.1.2
with:
@@ -75,7 +119,7 @@ jobs:
PR_ENV_DB_NAME: ${{ secrets.PR_ENV_DB_NAME }}
- name: Verify stats dump load status
id: verify_stats_dump
if: ${{ inputs.stats_dump }}
if: ${{ steps.label.outputs.stats_dump == 'true' }}
run: |
export PGPASSWORD="${PR_ENV_DB_PASSWORD}"
ADMIN="psql -h ${PR_ENV_DB_HOST} -U ${PR_ENV_DB_USER} -d ${PR_ENV_DB_NAME}"
@@ -98,7 +142,7 @@ jobs:
PR_ENV_DB_NAME: ${{ secrets.PR_ENV_DB_NAME }}
- name: Locate latest stats dump in S3
id: locate_stats_dump
if: ${{ inputs.stats_dump && steps.verify_stats_dump.outputs.should_load == 'true' }}
if: ${{ steps.label.outputs.stats_dump == 'true' && steps.verify_stats_dump.outputs.should_load == 'true' }}
env:
STATS_DUMP_BUCKET: metabase-cloud-stats-backups-coredev
run: |
@@ -113,7 +157,7 @@ jobs:
echo "Latest dump: $LATEST_KEY"
echo "key=$LATEST_KEY" >> "$GITHUB_OUTPUT"
- name: Download stats dump
if: ${{ inputs.stats_dump && steps.verify_stats_dump.outputs.should_load == 'true' }}
if: ${{ steps.label.outputs.stats_dump == 'true' && steps.verify_stats_dump.outputs.should_load == 'true' }}
env:
STATS_DUMP_BUCKET: metabase-cloud-stats-backups-coredev
run: |
@@ -123,7 +167,7 @@ jobs:
echo "Downloaded $(stat -c%s /tmp/stats-dump.dump) bytes"
file /tmp/stats-dump.dump
- name: Prepare app database schema
if: ${{ inputs.stats_dump && steps.verify_stats_dump.outputs.should_load == 'true' }}
if: ${{ steps.label.outputs.stats_dump == 'true' && steps.verify_stats_dump.outputs.should_load == 'true' }}
run: |
PGPASSWORD="${PR_ENV_DB_PASSWORD}" psql \
-h "${PR_ENV_DB_HOST}" \
@@ -136,7 +180,7 @@ jobs:
PR_ENV_DB_USER: ${{ secrets.PR_ENV_DB_USER }}
PR_ENV_DB_HOST: ${{ secrets.PR_ENV_DB_HOST }}
- name: Restore stats dump into app database
if: ${{ inputs.stats_dump && steps.verify_stats_dump.outputs.should_load == 'true' }}
if: ${{ steps.label.outputs.stats_dump == 'true' && steps.verify_stats_dump.outputs.should_load == 'true' }}
run: |
set +e
echo "::group::pg_restore / pg client versions"
@@ -173,7 +217,7 @@ jobs:
PR_ENV_DB_HOST: ${{ secrets.PR_ENV_DB_HOST }}
PR_ENV_DB_NAME: ${{ secrets.PR_ENV_DB_NAME }}
- name: Mark database as stats-dump loaded
if: ${{ inputs.stats_dump && steps.verify_stats_dump.outputs.should_load == 'true' }}
if: ${{ steps.label.outputs.stats_dump == 'true' && steps.verify_stats_dump.outputs.should_load == 'true' }}
run: |
PGPASSWORD="${PR_ENV_DB_PASSWORD}" psql \
-h "${PR_ENV_DB_HOST}" \
@@ -185,22 +229,22 @@ jobs:
PR_ENV_DB_USER: ${{ secrets.PR_ENV_DB_USER }}
PR_ENV_DB_HOST: ${{ secrets.PR_ENV_DB_HOST }}
- name: Download Deployment YAML template
if: ${{ !inputs.self_hosting }}
if: ${{ steps.label.outputs.self_hosting != 'true' }}
run: aws s3 cp s3://metabase-pr-env/metabase.yml.tmpl ./metabase.yml.tmpl
- name: Render Deployment YAML
if: ${{ !inputs.self_hosting }}
if: ${{ steps.label.outputs.self_hosting != 'true' }}
uses: nowactions/envsubst@c4b8e3977354d294659a7ec164279ce8fba5c2e1 # v1.0.1
with:
input: ./metabase.yml.tmpl
output: ./metabase.yml
env:
IMAGE_TAG: ${{ needs.build.outputs.tag }}
IMAGE_TAG: ${{ github.event.pull_request.head.sha }}
PR_NUMBER: ${{ github.event.number }}
MB_PREMIUM_EMBEDDING_TOKEN: ${{ secrets.PR_ENV_MB_PREMIUM_EMBEDDING_TOKEN }}
SHA: ${{ github.event.pull_request.head.sha || github.sha }}
MB_ANALYTICS_DEV_MODE: ${{ inputs.analytics_dev_mode && 'true' || 'false' }}
MB_ANALYTICS_DEV_MODE: ${{ steps.label.outputs.analytics_dev_mode }}
- name: Inject stats-dump env vars into Deployment YAML
if: ${{ !inputs.self_hosting && inputs.stats_dump }}
if: ${{ steps.label.outputs.self_hosting != 'true' && steps.label.outputs.stats_dump == 'true' }}
env:
MB_ENCRYPTION_SECRET_KEY: ${{ secrets.PR_ENV_STATS_DUMP_ENCRYPTION_KEY }}
run: |
@@ -242,20 +286,20 @@ jobs:
namespace: default
user: oidc-token
- name: Deploy PR Review ENV
if: ${{ !inputs.self_hosting }}
if: ${{ steps.label.outputs.self_hosting != 'true' }}
run: |
kubectl apply -f ./metabase.yml
- name: Download Helm values template (self-hosting)
if: ${{ inputs.self_hosting }}
if: ${{ steps.label.outputs.self_hosting == 'true' }}
run: aws s3 cp s3://metabase-pr-env/values.yaml.tmpl ./values.yaml.tmpl
- name: Render Helm values (self-hosting)
if: ${{ inputs.self_hosting }}
if: ${{ steps.label.outputs.self_hosting == 'true' }}
uses: nowactions/envsubst@c4b8e3977354d294659a7ec164279ce8fba5c2e1 # v1.0.1
with:
input: ./values.yaml.tmpl
output: ./values.yaml
env:
IMAGE_TAG: ${{ needs.build.outputs.tag }}
IMAGE_TAG: ${{ github.event.pull_request.head.sha }}
PR_NUMBER: ${{ github.event.number }}
MB_PREMIUM_EMBEDDING_TOKEN: ${{ secrets.PR_ENV_MB_PREMIUM_EMBEDDING_TOKEN }}
SHA: ${{ github.event.pull_request.head.sha || github.sha }}
@@ -266,15 +310,15 @@ jobs:
DB_USERNAME: ${{ secrets.PR_ENV_DB_USER }}
DB_PASSWORD: ${{ secrets.PR_ENV_DB_PASSWORD }}
CONFIG: ${{ secrets.PR_ENV_CONFIG }}
MB_ANALYTICS_DEV_MODE: ${{ inputs.analytics_dev_mode && 'true' || 'false' }}
MB_ANALYTICS_DEV_MODE: ${{ steps.label.outputs.analytics_dev_mode }}
- name: Setup Helm
if: ${{ inputs.self_hosting }}
if: ${{ steps.label.outputs.self_hosting == 'true' }}
uses: azure/setup-helm@dda3372f752e03dde6b3237bc9431cdc2f7a02a2 # v5.0.0
with:
version: "latest"
- name: Login to Quay.io
if: ${{ inputs.self_hosting }}
if: ${{ steps.label.outputs.self_hosting == 'true' }}
run: |
helm registry login quay.io \
--username "${QUAY_USERNAME}" \
@@ -283,7 +327,7 @@ jobs:
QUAY_USERNAME: ${{ secrets.QUAY_USERNAME }}
QUAY_PASSWORD: ${{ secrets.QUAY_PASSWORD }}
- name: Deploy PR Review ENV (self-hosting)
if: ${{ inputs.self_hosting }}
if: ${{ steps.label.outputs.self_hosting == 'true' }}
run: |
helm upgrade --install hosting-pr${{ github.event.number }} \
oci://quay.io/enterprisemetabase/chart-metabase-stack \
@@ -295,7 +339,7 @@ jobs:
preview_links:
runs-on: ${{ vars.SLIM_RUNNER_KEY }}
timeout-minutes: 5
needs: [deploy_pr]
needs: [deploy]
steps:
- uses: marocchino/sticky-pull-request-comment@0ea0beb66eb9baf113663a64ec522f60e49231c0 # v3.0.4
with:
@@ -305,4 +349,4 @@ jobs:
✅ Preview:
https://pr${{ github.event.number }}.coredev.metabase.com
🗒️ [login instructions](https://www.notion.so/metabase/Ephemeral-PR-Environments-c74a3710c87a460dbf4fc0007a001458)
📦 Deployment type: ${{ inputs.self_hosting && 'PR-Env-self-hosting' || (inputs.analytics_dev_mode && 'PR-Env-analytics-dev' || (inputs.stats_dump && 'PR-Env-Stats-Dump' || 'PR-Env')) }}
📦 Deployment type: ${{ needs.deploy.outputs.self_hosting == 'true' && 'PR-Env-self-hosting' || (needs.deploy.outputs.analytics_dev_mode == 'true' && 'PR-Env-analytics-dev' || (needs.deploy.outputs.stats_dump == 'true' && 'PR-Env-Stats-Dump' || 'PR-Env')) }}

View File

@@ -12,6 +12,13 @@ on:
skip-custom-viz:
type: boolean
default: false
# ID of this run's test-plan.json artifact, or '' to run the full suite.
test-plan-artifact-id:
type: string
default: ""
skip-ci-scripts:
type: boolean
default: false
workflow_dispatch:
inputs:
skip:
@@ -100,6 +107,7 @@ jobs:
SHARDS: 2 # should match matrix above
JEST_JUNIT_OUTPUT_DIR: ./target/junit
JEST_JUNIT_OUTPUT_NAME: test-report-frontend-unit.xml
UNIT_TEST_RESULTS_FILE: unit-test-results.json
steps:
- uses: actions/checkout@v6
- name: Prepare front-end environment
@@ -110,11 +118,29 @@ jobs:
# Resolve JOB_ID before the long test step so the ci-conductor reporter
# below can link failures to this exact job. Best-effort.
uses: ./.github/actions/resolve-job-id
- name: Prepare unit test selection
id: unit-selection
if: ${{ inputs.test-plan-artifact-id != '' }}
uses: ./.github/actions/prepare-test-selection
with:
suite: unit
test-plan-artifact-id: ${{ inputs.test-plan-artifact-id }}
- name: Run frontend unit tests
id: fe-tests
run: bun run test-unit --silent --shard=${{ matrix.shard }}/${{ env.SHARDS }}
# Keep this step continue-on-error because the quarantine logic controls whether it ultimately fails.
env:
JEST_TEST_PATHS_FILE: ${{ steps.unit-selection.outputs.paths-file }}
# Jest exits non-zero on a shard with no files unless --passWithNoTests is set.
run: |
rm -f "$UNIT_TEST_RESULTS_FILE" "$JEST_JUNIT_OUTPUT_DIR/$JEST_JUNIT_OUTPUT_NAME"
bun run test-unit --silent --shard=${{ matrix.shard }}/${{ env.SHARDS }} --json --outputFile="$UNIT_TEST_RESULTS_FILE" ${{ steps.unit-selection.outputs.paths-file && '--passWithNoTests' || '' }}
# Ordinary assertion failures can be quarantined, but runner failures cannot.
continue-on-error: true
- name: Check unit test results
if: ${{ !cancelled() && (steps.fe-tests.outcome == 'failure' || steps.unit-selection.outputs.paths-file) }}
env:
TEST_OUTCOME: ${{ steps.fe-tests.outcome }}
JEST_TEST_PATHS_FILE: ${{ steps.unit-selection.outputs.paths-file }}
run: bun .github/scripts/check-unit-test-results.ts
- name: Report failed tests to ci-conductor
if: ${{ !cancelled() && ( steps.fe-tests.outcome == 'failure' || github.run_attempt != '1' ) }}
# This step should NEVER fail the job!
@@ -144,6 +170,18 @@ jobs:
- name: Run frontend timezones tests
run: bun run test-timezones
# No CLJS compile: the CI-script specs don't import app code.
fe-tests-ci-scripts:
if: ${{ !inputs.skip-ci-scripts }}
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
timeout-minutes: 10
steps:
- uses: actions/checkout@v6
- name: Prepare front-end environment
uses: ./.github/actions/prepare-frontend
- name: Run CI script tests
run: bun run test-ci-scripts
# The custom-viz package is an isolated Bun workspace under
# enterprise/frontend/src/custom-viz
fe-custom-viz:
@@ -174,6 +212,7 @@ jobs:
- fe-type-check
- fe-tests-unit
- fe-tests-timezones
- fe-tests-ci-scripts
- fe-custom-viz
if: always() && !cancelled()
runs-on: ${{ vars.SLIM_RUNNER_KEY }}

View File

@@ -150,7 +150,7 @@ jobs:
run: NODE_ENV=development bun run build-pure:cljs
- name: Build Storybook
run: bun run build-storybook
run: bun run build-storybook:test
env:
STORYBOOK_STORIES_FILTER: ${{ needs.detect-changed-stories.outputs.story_files }}

View File

@@ -4,16 +4,52 @@ on:
workflow_call:
inputs:
skip:
description: "Skip the Loki visual tests."
type: boolean
default: false
test-plan-artifact-id:
description: "ID of this run's test-plan.json artifact, or '' to run the full suite."
type: string
default: ""
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }}-loki
cancel-in-progress: true
jobs:
story-selection:
if: ${{ !inputs.skip && inputs.test-plan-artifact-id != '' }}
runs-on: ${{ vars.SLIM_RUNNER_KEY }}
timeout-minutes: 5
outputs:
selection: ${{ steps.loki-selection.outputs.selection }}
test-plan-artifact-id: ${{ steps.loki-selection.outputs.paths-file && inputs.test-plan-artifact-id || '' }}
steps:
- name: Checkout Repository
uses: actions/checkout@v6
with:
sparse-checkout: |
.github
/.nvmrc
/package.json
sparse-checkout-cone-mode: false
- name: Prepare Node + Bun environment
uses: ./.github/actions/prepare-node-bun
- name: Prepare Loki story selection
id: loki-selection
uses: ./.github/actions/prepare-test-selection
with:
suite: loki
test-plan-artifact-id: ${{ inputs.test-plan-artifact-id }}
visual-test:
if: ${{ !inputs.skip }}
name: loki-run
needs: story-selection
# A missing job output compares equal to '0' in GitHub expressions,
# so an empty selection is marked with a non-numeric value.
if: ${{ !cancelled() && !inputs.skip && needs.story-selection.outputs.selection != 'empty' }}
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
timeout-minutes: 30
services:
@@ -41,8 +77,22 @@ jobs:
- name: Compile CLJS
run: NODE_ENV=development bun run build-pure:cljs
- name: Prepare Loki story selection
id: loki-selection
if: ${{ needs.story-selection.outputs.test-plan-artifact-id != '' }}
uses: ./.github/actions/prepare-test-selection
with:
suite: loki
test-plan-artifact-id: ${{ needs.story-selection.outputs.test-plan-artifact-id }}
- name: Build Storybook
env:
STORYBOOK_STORY_PATHS_FILE: ${{ steps.loki-selection.outputs.paths-file }}
run: bun run build-storybook:test
- name: Run Loki Visual Tests
run: bun run test-visual:loki:ci
# Loki drops stories with parameters.loki.skip, and exits non-zero when that leaves the selection empty.
run: bun run test-visual:loki --reactUri file:./storybook-static --verboseRenderer ${{ steps.loki-selection.outputs.paths-file && '--passWithNoStories --requireReference' || '' }}
- name: Generate Visual Report on Failure
if: failure()
@@ -57,3 +107,38 @@ jobs:
include-hidden-files: true
path: .loki/
if-no-files-found: ignore
loki-tests-result:
name: visual-test
needs: [story-selection, visual-test]
if: always() && !cancelled()
runs-on: ${{ vars.SLIM_RUNNER_KEY }}
timeout-minutes: 5
steps:
- name: Check Loki job status
uses: actions/github-script@v9
env:
needs: ${{ toJson(needs) }}
with:
script: | # js
const needs = JSON.parse(process.env.needs);
const jobs = Object.entries(needs).map(
([jobName, jobValues]) => ({
name: jobName,
result: jobValues.result
}));
// are all jobs skipped or successful?
if (jobs.every(job => (job.result === 'skipped' || job.result === 'success'))) {
console.log('All jobs are skipped or successful');
process.exit(0);
}
// otherwise, something failed
console.log('Some Loki jobs failed');
jobs.forEach((job) => {
if (job.result !== 'success') {
console.log(`${job.name} - ${job.result}`);
}
});
process.exit(1);

View File

@@ -1,52 +0,0 @@
name: PR-Env labeled
on:
pull_request:
types: [ labeled ]
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
uberjar:
if: ${{ github.event.label.name == 'PR-Env' || github.event.label.name == 'PR-Env-self-hosting' || github.event.label.name == 'PR-Env-analytics-dev' || github.event.label.name == 'PR-Env-Stats-Dump' }}
uses: ./.github/workflows/uberjar.yml
secrets: inherit
pr-env:
needs: [uberjar]
if: ${{ github.event.label.name == 'PR-Env' }}
uses: ./.github/workflows/pr-env.yml
secrets: inherit
with:
self_hosting: false
analytics_dev_mode: false
pr-env-self-hosting:
needs: [uberjar]
if: ${{ github.event.label.name == 'PR-Env-self-hosting' }}
uses: ./.github/workflows/pr-env.yml
secrets: inherit
with:
self_hosting: true
analytics_dev_mode: false
pr-env-analytics-dev:
needs: [uberjar]
if: ${{ github.event.label.name == 'PR-Env-analytics-dev' }}
uses: ./.github/workflows/pr-env.yml
secrets: inherit
with:
self_hosting: false
analytics_dev_mode: true
pr-env-stats-dump:
needs: [uberjar]
if: ${{ github.event.label.name == 'PR-Env-Stats-Dump' }}
uses: ./.github/workflows/pr-env.yml
secrets: inherit
with:
self_hosting: false
analytics_dev_mode: false
stats_dump: true

View File

@@ -252,7 +252,7 @@ jobs:
with:
edition: oss
commit: ${{ inputs.commit }}
repo: ${{ vars.DOCKERHUB_OWNER }}/${{ vars.DOCKERHUB_STAGING_REPO }}
repo: ${{ vars.DOCKERHUB_STAGING_REPO }}
tag: ${{ needs.release-artifact.outputs.oss_version }}-${{ inputs.commit }}
release: true
@@ -263,7 +263,7 @@ jobs:
with:
edition: ee
commit: ${{ inputs.commit }}
repo: ${{ vars.DOCKERHUB_OWNER }}/${{ vars.DOCKERHUB_STAGING_REPO }}
repo: ${{ vars.DOCKERHUB_STAGING_REPO }}
tag: ${{ needs.release-artifact.outputs.ee_version }}-${{ inputs.commit }}
release: true

View File

@@ -279,7 +279,7 @@ jobs:
with:
edition: oss
commit: ${{ inputs.commit }}
repo: ${{ vars.DOCKERHUB_OWNER }}/${{ vars.DOCKERHUB_REPO }}
repo: ${{ vars.DOCKERHUB_REPO }}
tag: ${{ needs.check-version.outputs.oss }}
release: true
@@ -290,7 +290,7 @@ jobs:
with:
edition: ee
commit: ${{ inputs.commit }}
repo: ${{ vars.DOCKERHUB_OWNER }}/${{ vars.DOCKERHUB_REPO }}-enterprise
repo: ${{ vars.DOCKERHUB_REPO }}-enterprise
tag: ${{ needs.check-version.outputs.ee }}
release: true
@@ -330,6 +330,7 @@ jobs:
get-extra-tags:
needs: [ check-version ]
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
timeout-minutes: 10
outputs:
extra_tags: ${{ steps.taglist.outputs.extra_tags }}
steps:

View File

@@ -56,6 +56,16 @@ jobs:
core.setOutput('artifact-count', result.artifactCount);
- uses: ./.github/actions/prepare-node-bun
- name: Upload e2e test timings to Stats
if: ${{ github.repository == 'metabase/metabase' }}
env:
TIMINGS_PATH: e2e/support/timings.json
ENG_STATS_URL: ${{ secrets.ENG_STATS_URL }}
API_KEY: ${{ secrets.ENG_STATS_API_KEY }}
run: bun .github/scripts/upload-e2e-test-timings.ts
- name: Configure git and commit changes
id: commit
run: |
@@ -110,7 +120,7 @@ jobs:
--body "$BODY" \
--label ".CI & Tests" \
--label "no-backport" \
--reviewer metabase/devexp)
--reviewer metabase/devex)
PR_NUMBER=${URL##*/}

View File

@@ -57,11 +57,10 @@ jobs:
build_scripts: ${{ steps.changes.outputs.build_scripts }}
embedder_plugin: ${{ steps.changes.outputs.embedder_plugin }}
frontend_all: ${{ steps.changes.outputs.frontend_all }}
frontend_ci: ${{ steps.changes.outputs.frontend_ci }}
frontend_sources: ${{ steps.changes.outputs.frontend_sources }}
frontend_unit_infra: ${{ steps.changes.outputs.frontend_unit_infra }}
frontend_loki_ci: ${{ steps.changes.outputs.frontend_loki_ci }}
frontend_loki_infra: ${{ steps.changes.outputs.frontend_loki_infra }}
frontend_loki_all: ${{ steps.changes.outputs.frontend_loki_all }}
shared_sources: ${{ steps.changes.outputs.shared_sources }}
documentation: ${{ steps.changes.outputs.documentation }}
embedding_documentation: ${{ steps.changes.outputs.embedding_documentation }}
@@ -69,6 +68,7 @@ jobs:
embedding_sdk_host_sample_apps: ${{ steps.changes.outputs.embedding_sdk_host_sample_apps }}
embedding_sdk_ci: ${{ steps.changes.outputs.embedding_sdk_ci }}
custom_viz_all: ${{ steps.changes.outputs.custom_viz_all }}
ci_scripts: ${{ steps.changes.outputs.ci_scripts }}
project_backend_checks: ${{ steps.changes.outputs.project_backend_checks }}
project_migration_checks: ${{ steps.changes.outputs.project_migration_checks }}
project_ratchet_checks: ${{ steps.changes.outputs.project_ratchet_checks }}
@@ -97,16 +97,14 @@ jobs:
filters: .github/file-paths.yaml
create-test-plan:
needs: files-changed
needs: [should-run, files-changed]
name: Create test plan
if: ${{ github.event_name == 'pull_request' }}
if: ${{ github.event_name == 'pull_request' && needs.should-run.outputs.verdict == 'defer' }}
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
timeout-minutes: 10
outputs:
stats: ${{ steps.compute.outputs.stats }}
fe_unit_specs_to_run: ${{ steps.compute.outputs.fe_unit_specs_to_run }}
loki_stories_to_run: ${{ steps.compute.outputs.loki_stories_to_run }}
e2e_specs_to_run: ${{ steps.compute.outputs.e2e_specs_to_run }}
artifact-id: ${{ steps.compute.outputs.artifact-id }}
steps:
- uses: actions/checkout@v6
- name: Create test plan
@@ -152,10 +150,30 @@ jobs:
(needs.should-run.outputs.verdict == 'force-run' ||
needs.files-changed.outputs.no_source_code_changes != 'true')
uses: ./.github/workflows/uberjar.yml
with:
containerize: ${{ (github.event_name == 'push' && github.ref_name == 'master') || contains(github.event.pull_request.labels.*.name, 'build-docker-uberjar') }}
secrets: inherit
containerize:
needs: [uberjar]
if: >-
!cancelled() &&
needs.uberjar.result == 'success' &&
(
(github.event_name == 'push' && github.ref_name == 'master') ||
(github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'build-docker-uberjar'))
)
name: containerize (${{ matrix.edition }})
strategy:
matrix:
edition: ['ee', 'oss']
uses: ./.github/workflows/containerize-jar.yml
secrets: inherit
with:
build-args: |
GIT_COMMIT_SHA=${{ github.event.pull_request.head.sha || github.sha }}
commit: ${{ github.event.pull_request.head.sha || github.sha }}
edition: ${{ matrix.edition }}
release: false
# Compile CLJS in parallel with the uberjar and pass it to each e2e group.
# The compile is doesn't depend on MB_EDITION
e2e-cljs:
@@ -242,6 +260,10 @@ jobs:
skip: >-
${{ needs.should-run.outputs.verdict != 'force-run' &&
needs.files-changed.outputs.backend_all != 'true' }}
smoke-only: >-
${{ needs.should-run.outputs.verdict != 'force-run' &&
!contains(github.event.pull_request.labels.*.name, 'ci:run-semantic-search-tests')
}}
app-db-tests:
if: |
@@ -272,7 +294,7 @@ jobs:
force-run: ${{ needs.should-run.outputs.verdict == 'force-run' }}
frontend-tests:
needs: [should-run, files-changed]
needs: [should-run, files-changed, create-test-plan]
if: |
!cancelled() &&
(needs.should-run.outputs.verdict == 'force-run' ||
@@ -283,6 +305,8 @@ jobs:
skip: >-
${{ needs.should-run.outputs.verdict != 'force-run' &&
needs.files-changed.outputs.frontend_all != 'true' }}
# A skipped/failed planner supplies no artifact, so the suite runs in full.
test-plan-artifact-id: ${{ needs.create-test-plan.result == 'success' && needs.create-test-plan.outputs.artifact-id || '' }}
skip-lint: >-
${{ needs.should-run.outputs.verdict != 'force-run' &&
needs.files-changed.outputs.frontend_all != 'true' &&
@@ -290,9 +314,12 @@ jobs:
skip-custom-viz: >-
${{ needs.should-run.outputs.verdict != 'force-run' &&
needs.files-changed.outputs.custom_viz_all != 'true' }}
skip-ci-scripts: >-
${{ needs.should-run.outputs.verdict != 'force-run' &&
needs.files-changed.outputs.ci_scripts != 'true' }}
loki-tests:
needs: [should-run, files-changed]
needs: [should-run, files-changed, create-test-plan]
if: |
!cancelled() &&
(needs.should-run.outputs.verdict == 'force-run' ||
@@ -301,9 +328,9 @@ jobs:
with:
skip: >-
${{ needs.should-run.outputs.verdict != 'force-run' &&
needs.files-changed.outputs.frontend_ci != 'true' &&
needs.files-changed.outputs.frontend_sources != 'true' &&
needs.files-changed.outputs.frontend_loki_ci != 'true' }}
needs.files-changed.outputs.frontend_loki_all != 'true' }}
# A skipped/failed planner supplies no artifact, so the suite runs in full.
test-plan-artifact-id: ${{ needs.create-test-plan.result == 'success' && needs.create-test-plan.outputs.artifact-id || '' }}
e2e-tests:
needs: [should-run, files-changed, uberjar]
@@ -403,50 +430,3 @@ jobs:
needs.files-changed.outputs.frontend_sources == 'true'
uses: ./.github/workflows/bundle-size.yml
secrets: inherit
pr-env:
needs: [uberjar]
if: |
!cancelled() &&
contains(github.event.pull_request.labels.*.name, 'PR-Env') &&
!contains(github.event.pull_request.labels.*.name, 'PR-Env-self-hosting') &&
!contains(github.event.pull_request.labels.*.name, 'PR-Env-analytics-dev')
uses: ./.github/workflows/pr-env.yml
secrets: inherit
with:
self_hosting: false
analytics_dev_mode: false
pr-env-self-hosting:
needs: [uberjar]
if: |
!cancelled() &&
contains(github.event.pull_request.labels.*.name, 'PR-Env-self-hosting')
uses: ./.github/workflows/pr-env.yml
secrets: inherit
with:
self_hosting: true
analytics_dev_mode: false
pr-env-analytics-dev:
needs: [uberjar]
if: |
!cancelled() &&
contains(github.event.pull_request.labels.*.name, 'PR-Env-analytics-dev')
uses: ./.github/workflows/pr-env.yml
secrets: inherit
with:
self_hosting: false
analytics_dev_mode: true
pr-env-stats-dump:
needs: [uberjar]
if: |
!cancelled() &&
contains(github.event.pull_request.labels.*.name, 'PR-Env-Stats-Dump')
uses: ./.github/workflows/pr-env.yml
secrets: inherit
with:
self_hosting: false
analytics_dev_mode: false
stats_dump: true

View File

@@ -14,14 +14,17 @@ jobs:
runs-on: ${{ vars.SLIM_RUNNER_KEY }}
timeout-minutes: 5
steps:
- name: Dispatch patch release for the current and previous major
- name: Dispatch patch release for the current and recent majors
uses: actions/github-script@v9
with:
script: | # js
// Nightly patches cover the current supported major and the one
// before it. release-auto.yml releases one major per dispatch.
const autoPatchVersionsCount = Number('${{ vars.AUTO_PATCH_VERSIONS_COUNT }}') || 0;
const currentRelease = Number('${{ vars.CURRENT_VERSION }}');
const majors = [currentRelease];
// [currentRelease, currentRelease - 1, ..., currentRelease - autoPatchVersionsCount]
const majors = Array.from(
{ length: autoPatchVersionsCount + 1 },
(_, i) => currentRelease - i,
);
await Promise.all(majors.map(version =>
github.rest.actions.createWorkflowDispatch({

View File

@@ -6,32 +6,10 @@ on:
skip:
type: boolean
default: false
# Run one semantic-search job on demand, against the ref you dispatch on, optionally under a
# profiler. Everything else about the job -- services, matrix legs, test args, quarantine gating --
# is whatever the normal CI run uses, because it is the same job.
workflow_dispatch:
inputs:
suite:
description: 'Which semantic-search job to run. Only this one runs; the others are skipped.'
type: choice
required: true
options:
- h2
- mariadb
- mysql
- postgres
- appdb-mode
tests:
description: 'Optional: run only these tests, comma- or space-separated, e.g. metabase-enterprise.semantic-search.index-test/some-test. Bare namespaces work too. Blank runs the job''s usual set.'
type: string
required: false
default: ''
profile:
description: 'Profile the test run and attach a flamegraph to the job. Adds sampling overhead.'
smoke-only:
type: boolean
required: false
default: false
description: Only run Postgres Latest to do a quick resource-efficient sanity check, intended to be set for PRs.
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }}-semantic-search
@@ -43,18 +21,10 @@ concurrency:
env:
CI_CONDUCTOR_BASE_URL: ${{ vars.CI_CONDUCTOR_BASE_URL }}
CI_CONDUCTOR_WEBHOOK_SECRET: ${{ secrets.CI_CONDUCTOR_WEBHOOK_SECRET }}
# Both read by the test-driver action, which is what actually runs the tests. Set here rather than
# passed as inputs to each job so that turning either on is one line, not one line per job.
# Empty (and so off) on workflow_call, where neither input exists.
#
# ONLY_TESTS replaces the job's own test selection, so every matrix leg of the chosen job runs the
# whole list.
PROFILE_TESTS: ${{ inputs.profile }}
ONLY_TESTS: ${{ inputs.tests }}
jobs:
semantic-search-tests-h2:
if: ${{ !inputs.skip && (inputs.suite == '' || inputs.suite == 'h2') }}
if: ${{ !inputs.skip && !inputs.smoke-only && (github.event.inputs.db == '' || github.event.inputs.db == 'h2') }}
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
timeout-minutes: 40
strategy:
@@ -98,7 +68,7 @@ jobs:
:exclude-tags '[${{ matrix.job.exclude-tag }}]'
semantic-search-tests-mariadb:
if: ${{ !inputs.skip && (inputs.suite == '' || inputs.suite == 'mariadb') }}
if: ${{ !inputs.skip && !inputs.smoke-only && (github.event.inputs.db == '' || github.event.inputs.db == 'mariadb') }}
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
timeout-minutes: 40
strategy:
@@ -163,7 +133,7 @@ jobs:
:exclude-tags '[${{ matrix.job.exclude-tag }}]'
semantic-search-tests-mysql:
if: ${{ !inputs.skip && (inputs.suite == '' || inputs.suite == 'mysql') }}
if: ${{ !inputs.skip && !inputs.smoke-only && (github.event.inputs.db == '' || github.event.inputs.db == 'mysql') }}
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
timeout-minutes: 40
strategy:
@@ -235,7 +205,7 @@ jobs:
:exclude-tags '[${{ matrix.job.exclude-tag }}]'
semantic-search-tests-postgres:
if: ${{ !inputs.skip && (inputs.suite == '' || inputs.suite == 'postgres') }}
if: ${{ !inputs.skip && (github.event.inputs.db == '' || github.event.inputs.db == 'postgres') }}
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
timeout-minutes: 40
strategy:
@@ -250,6 +220,10 @@ jobs:
junit-name: semantic-search-tests-postgres-latest
postgres-image: postgres:latest
pgvector-image: pgvector/pgvector:pg17
exclude: >-
${{ inputs.smoke-only
&& fromJSON('[{"version":{"name":"Postgres 14.x with pgvector pg17"}}]')
|| fromJSON('[]') }}
job:
- name: Semantic Search Tests
build-static-viz: false
@@ -302,7 +276,7 @@ jobs:
# MB_PGVECTOR_DB_URL is deliberately unset. The dedicated-harness tests gate themselves off; the
# appdb-pgvector-mode round trip (and the URL-independent unit tests) run for real.
semantic-search-tests-appdb-mode:
if: ${{ !inputs.skip && (inputs.suite == '' || inputs.suite == 'appdb-mode') }}
if: ${{ !inputs.skip && !inputs.smoke-only && (github.event.inputs.db == '' || github.event.inputs.db == 'appdb-mode') }}
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
timeout-minutes: 40
strategy:

View File

@@ -1,8 +1,8 @@
name: Bundle Load Stats
# Reuses the uberjar built on every master merge: boots it, signs in, loads the
# app in a throttled headless Chrome across four network and CPU conditions, and
# appends the times to eng-stats-importer.
# app in a throttled headless Chrome across four network and CPU conditions and
# two site locales, and appends the times to eng-stats-importer.
#
# The instance is blank on purpose. `index.html` is built from settings and the
# signed-in user alone, so seeding content would not change a single byte of the
@@ -59,7 +59,7 @@ jobs:
needs: [uberjar-built]
if: ${{ needs.uberjar-built.outputs.found == 'true' }}
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
timeout-minutes: 10
timeout-minutes: 20
env:
HEAD_SHA: ${{ inputs.commit || github.event.workflow_run.head_sha }}
MB_JETTY_PORT: "4000"
@@ -67,6 +67,9 @@ jobs:
# Loads per condition. The first is the cold reading and the rest give the
# median, so this trades run time for a steadier number.
RUNS: "8"
# Measured in this order, so the translated reading comes from the same
# state the instance is set up in.
LOCALES: "de en"
steps:
- name: Check out the code
uses: actions/checkout@v6
@@ -114,19 +117,36 @@ jobs:
env:
CHROME_PATH: ${{ steps.setup-chrome.outputs.chrome-path }}
run: |
bun frontend/test/bench/matrix.ts "$SITE/" "$RUNS" \
| tee artifacts/load-times.json
# The site locale decides which translation catalogue the document
# carries, and a user with no locale of their own falls back to it.
# Measure both, so every commit has a translated reading and an
# English one, and the difference is what translation costs.
for locale in $LOCALES; do
curl -sS -X PUT -o /dev/null \
-H "X-Metabase-Session: ${SESSION_COOKIE}" \
-H "content-type: application/json" \
-d "{\"value\":\"${locale}\"}" \
"$SITE/api/setting/site-locale"
# One request to absorb the settings cache miss the switch causes.
curl -sS -o /dev/null "$SITE/"
bun frontend/test/bench/matrix.ts "$SITE/" "$RUNS" \
| tee "artifacts/load-times-${locale}.json"
done
- name: Show the Metabase log if something went wrong
if: failure()
run: tail -100 artifacts/metabase.log || true
- name: Import load times into Stats
if: ${{ inputs.commit == '' }}
env:
ROWS: artifacts/load-times.json
# Passed via env (not interpolated into the shell) so the commit
# message can't inject commands; the script keeps only the subject.
COMMIT_MESSAGE: ${{ github.event.workflow_run.head_commit.message }}
COMMIT_TIMESTAMP: ${{ github.event.workflow_run.head_commit.timestamp }}
ENG_STATS_URL: ${{ secrets.ENG_STATS_URL }}
API_KEY: ${{ secrets.ENG_STATS_API_KEY }}
run: bun .github/scripts/upload-bundle-load-stats.ts
run: |
# Each file carries its own `Locale`, read from the served document.
for locale in $LOCALES; do
ROWS="artifacts/load-times-${locale}.json" \
bun .github/scripts/upload-bundle-load-stats.ts
done

View File

@@ -0,0 +1,65 @@
name: Profile Backend
# The only on-demand entry point for the backend test suites. Dispatching this workflow runs one
# job of one suite against the ref you dispatch on, under clj-async-profiler, and attaches a
# flamegraph to the job.
#
# The suite workflows below are called unchanged: services, matrix legs, test args and quarantine
# gating are whatever the normal CI run uses, because they are the same jobs. They declare nothing
# for this feature -- they read `suite`, `db` and `tests` off `github.event.inputs`, which inside a
# called workflow is this dispatch's payload.
#
# `suite` picks the workflow, `db` picks the leg within it. `db` is free text so that adding a
# driver or an app-DB leg needs no edit here; an unknown driver fails the drivers run through
# mage's `--only-driver`, and an unknown app-DB or semantic-search leg matches nothing and runs no
# jobs.
#
# Renaming an input here means updating every reader -- `grep -rn github.event.inputs .github/`.
# Nothing enforces that, deliberately: a stale name reads as empty, which runs every leg of the
# suite unprofiled, on the run you are watching. It cannot affect a normal CI run, where those
# references are empty either way.
on:
workflow_dispatch:
inputs:
suite:
description: 'Which suite to profile.'
type: choice
required: true
options:
- backend
- app-db
- drivers
- semantic-search
db:
description: 'Which leg of the suite: app-db takes mariadb, mysql or postgres; drivers takes a driver name, e.g. bigquery or mongo-ssl; semantic-search takes h2, mariadb, mysql, postgres or appdb-mode. Ignored by backend. Blank runs the whole suite -- every leg for app-db and semantic-search, mage''s usual decisions for drivers.'
type: string
required: false
default: ''
tests:
description: 'Optional: profile only these tests, comma- or space-separated, e.g. metabase.query-processor-test/some-test. Bare namespaces work too. Blank profiles the leg''s usual set.'
type: string
required: false
default: ''
# These jobs gate on `inputs.suite` -- this workflow's own input. Only the called workflows reach
# for `github.event.inputs.*`, because `inputs` there is their own `workflow_call` signature.
jobs:
backend:
if: ${{ inputs.suite == 'backend' }}
uses: ./.github/workflows/backend.yml
secrets: inherit
app-db:
if: ${{ inputs.suite == 'app-db' }}
uses: ./.github/workflows/app-db.yml
secrets: inherit
drivers:
if: ${{ inputs.suite == 'drivers' }}
uses: ./.github/workflows/drivers.yml
secrets: inherit
semantic-search:
if: ${{ inputs.suite == 'semantic-search' }}
uses: ./.github/workflows/semantic-search.yml
secrets: inherit

View File

@@ -144,12 +144,20 @@ jobs:
EOF
)"
- name: Approve and Merge the PR
run: |
gh pr review --approve "$PR_NUMBER"
gh pr merge --squash --auto "$PR_NUMBER"
# Approving takes a different token than the one that opened the PR (above, the automation
# user's): GitHub won't let a token approve its own pull request.
- name: Approve the PR
run: gh pr review --approve "$PR_NUMBER"
shell: bash
env:
# uses a different token than the token that created the PR so we can approve and merge it
GH_TOKEN: ${{ github.token }}
PR_NUMBER: ${{ steps.create_pr.outputs.pr_number }}
# Merging goes back through the automation user: GitHub starts no workflow run for a push made
# with `GITHUB_TOKEN`, so an auto-merge enabled with it lands on master with no CI.
- name: Merge the PR
run: gh pr merge --squash --auto "$PR_NUMBER"
shell: bash
env:
GH_TOKEN: ${{ secrets.METABASE_AUTOMATION_USER_TOKEN }}
PR_NUMBER: ${{ steps.create_pr.outputs.pr_number }}

View File

@@ -1,6 +1,6 @@
name: Build Uberjar
# This workflow builds and containerizes an uberjar and pushes it to dockerhub.
# This workflow builds the uberjar.
concurrency:
group: uberjar-${{ inputs.commit || github.event.pull_request.head.sha || github.sha }}
@@ -14,11 +14,6 @@ on:
type: string
default: ''
required: false
containerize:
description: "Set if the uberjar should be containerized and pushed to Dockerhub."
type: boolean
default: false
required: false
workflow_call:
inputs:
commit:
@@ -26,11 +21,6 @@ on:
type: string
default: ''
required: false
containerize:
description: "Set if the uberjar should be containerized and pushed to Dockerhub."
type: boolean
default: false
required: false
jobs:
setup:
@@ -101,14 +91,15 @@ jobs:
if: ${{ needs.setup.outputs.build_editions != '[]' }}
name: build (${{ matrix.edition }})
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
timeout-minutes: 15
timeout-minutes: 20
strategy:
matrix:
edition: ${{ fromJson(needs.setup.outputs.build_editions) }}
env:
MB_EDITION: ${{ matrix.edition }}
INTERACTIVE: false
SKIP_LICENSES: ${{ github.event_name == 'pull_request' && !inputs.containerize }} # faster builds for dev
# Faster builds for devs
SKIP_LICENSES: ${{ github.event_name == 'pull_request' }}
# Emit bundle stats on trunk/release builds (tracked over time) and on PRs
# (the bundle-size check reads them for both the PR head and its base ref).
EMIT_BUNDLE_STATS: ${{ (github.ref_name == 'master' || startsWith(github.ref_name, 'release-') || github.event_name == 'pull_request') && 'true' || 'false' }}
@@ -135,6 +126,19 @@ jobs:
name: metabase-bundle-stats-${{ matrix.edition }}-${{ inputs.commit || github.event.pull_request.head.sha || github.sha }}
path: target/bundle-stats
if-no-files-found: warn
- name: Verify license files are present in uberjar
if: ${{ env.SKIP_LICENSES != 'true' }}
run: |
echo "Checking for license files in uberjar..."
unzip -l ./target/uberjar/metabase.jar | grep -q "license-backend-third-party.txt" || (echo "ERROR: license-backend-third-party.txt not found in uberjar" && exit 1)
unzip -l ./target/uberjar/metabase.jar | grep -q "license-frontend-third-party.txt" || (echo "ERROR: license-frontend-third-party.txt not found in uberjar" && exit 1)
echo "Both license files found in uberjar"
- name: Launch uberjar
run: >-
java --add-opens java.base/java.nio=ALL-UNNAMED -jar ./target/uberjar/metabase.jar &
- name: Wait for Metabase to start
timeout-minutes: 5
run: while ! curl -s 'http://localhost:3000/api/health' | grep '{"status":"ok"}'; do sleep 1; done
- name: Mark with the commit hash
run: git rev-parse --short HEAD > COMMIT-ID
shell: bash
@@ -145,62 +149,9 @@ jobs:
uses: actions/upload-artifact@v7
with:
# Same key the artifact lookups use, so a build for an explicitly dispatched commit
# uploads where `setup` and `jar-healthcheck` go looking for it.
# uploads where `setup` and the containerize workflow go looking for it.
name: metabase-${{ matrix.edition }}-${{ inputs.commit || github.event.pull_request.head.sha || github.sha }}-uberjar
path: |
./target/uberjar/metabase.jar
./COMMIT-ID
./SHA256.sum
jar-healthcheck:
needs: [setup, build]
# Health-check whatever we are about to containerize, plus every build on master and release.
if: ${{ !failure() && !cancelled() && (inputs.containerize || github.ref_name == 'master' || startsWith(github.ref_name, 'release-')) }}
name: jar-healthcheck (${{ matrix.edition }})
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
timeout-minutes: 10
strategy:
matrix:
edition: [ee, oss]
steps:
- name: Check out the code
uses: actions/checkout@v6
- name: Prepare JRE (Java Run-time Environment)
uses: actions/setup-java@v5.2.0
with:
java-package: jre
java-version: ${{ needs.setup.outputs.java_version || 25 }}
distribution: "temurin"
- run: java -version
- name: Retrieve uberjar artifact
uses: ./.github/actions/fetch-artifact
with:
commit: ${{ inputs.commit || github.event.pull_request.head.sha || github.sha }}
edition: ${{ matrix.edition }}
- name: Verify license files are present in uberjar
run: |
echo "Checking for license files in uberjar..."
unzip -l ./metabase.jar | grep -q "license-backend-third-party.txt" || (echo "ERROR: license-backend-third-party.txt not found in uberjar" && exit 1)
unzip -l ./metabase.jar | grep -q "license-frontend-third-party.txt" || (echo "ERROR: license-frontend-third-party.txt not found in uberjar" && exit 1)
echo "Both license files found in uberjar"
- name: Launch uberjar
run: >-
java --add-opens java.base/java.nio=ALL-UNNAMED -jar ./metabase.jar &
- name: Wait for Metabase to start
run: while ! curl 'http://localhost:3000/api/health' | grep '{"status":"ok"}'; do sleep 1; done
containerize:
needs: [build, jar-healthcheck]
if: ${{ inputs.containerize && !failure() && !cancelled() }}
name: containerize (${{ matrix.edition }})
strategy:
matrix:
edition: [ee, oss]
uses: ./.github/workflows/containerize-jar.yml
secrets: inherit
with:
edition: ${{ matrix.edition }}
commit: ${{ inputs.commit || github.event.pull_request.head.sha || github.sha }}
release: false
build-args: |
GIT_COMMIT_SHA=${{ inputs.commit || github.event.pull_request.head.sha || github.sha }}

9
.gitignore vendored
View File

@@ -86,7 +86,11 @@ cypress.env.json
/resources/frontend_client/app/**/*.hot-update.js
/resources/frontend_client/app/**/*.hot-update.json
/resources/frontend_client/app/dist/
/resources/frontend_client/app/locales
/resources/frontend_client/app/locales/*
# Keep the directory itself: the frontend build resolves `locales/<locale>.json`
# through it, and rspack fails on a context whose directory does not exist. The
# `:translations` build step fills it in.
!/resources/frontend_client/app/locales/.gitkeep
/resources/frontend_client/app/embed.js
/resources/frontend_client/app/embedding-sdk.js
/resources/frontend_client/app/embedding-sdk.js.map
@@ -210,3 +214,6 @@ __pycache__/
# dependency-cruiser output
dependency-graph.json
.playwright-cli/
# mage modules-tree --html output
/modules.html

Binary file not shown.

Before

Width:  |  Height:  |  Size: 22 KiB

After

Width:  |  Height:  |  Size: 22 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 27 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 86 KiB

After

Width:  |  Height:  |  Size: 86 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 92 KiB

After

Width:  |  Height:  |  Size: 92 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 82 KiB

After

Width:  |  Height:  |  Size: 83 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 88 KiB

After

Width:  |  Height:  |  Size: 88 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 74 KiB

After

Width:  |  Height:  |  Size: 73 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 79 KiB

After

Width:  |  Height:  |  Size: 79 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 77 KiB

After

Width:  |  Height:  |  Size: 78 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 78 KiB

After

Width:  |  Height:  |  Size: 78 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 79 KiB

After

Width:  |  Height:  |  Size: 79 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 77 KiB

After

Width:  |  Height:  |  Size: 77 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 74 KiB

After

Width:  |  Height:  |  Size: 73 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 74 KiB

After

Width:  |  Height:  |  Size: 73 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 85 KiB

After

Width:  |  Height:  |  Size: 84 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 90 KiB

After

Width:  |  Height:  |  Size: 90 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 81 KiB

After

Width:  |  Height:  |  Size: 81 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 86 KiB

After

Width:  |  Height:  |  Size: 86 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 72 KiB

After

Width:  |  Height:  |  Size: 72 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 77 KiB

After

Width:  |  Height:  |  Size: 77 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 76 KiB

After

Width:  |  Height:  |  Size: 76 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 76 KiB

After

Width:  |  Height:  |  Size: 76 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 77 KiB

After

Width:  |  Height:  |  Size: 77 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 76 KiB

After

Width:  |  Height:  |  Size: 76 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 72 KiB

After

Width:  |  Height:  |  Size: 72 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 72 KiB

After

Width:  |  Height:  |  Size: 72 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 37 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 30 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 573 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 114 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 285 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 276 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 93 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 264 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 80 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 113 KiB

After

Width:  |  Height:  |  Size: 114 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 1.9 KiB

After

Width:  |  Height:  |  Size: 1.9 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 1.9 KiB

After

Width:  |  Height:  |  Size: 1.9 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 22 KiB

After

Width:  |  Height:  |  Size: 21 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 2.1 KiB

After

Width:  |  Height:  |  Size: 2.2 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 31 KiB

After

Width:  |  Height:  |  Size: 32 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 9.1 KiB

After

Width:  |  Height:  |  Size: 9.8 KiB

Some files were not shown because too many files have changed in this diff Show More