Compare commits

...

232 Commits

Author SHA1 Message Date
Bryan Maass
6b8829bd58 Spike: sqlc -> Clojure query codegen with Malli schemas
Generate Clojure query fns with Malli result schemas from .sql files, using
sqlc for SQL parsing and type inference and sqlc-gen-json to expose its
analysis as JSON.

Verified end to end: migrations against Postgres 16 (1427 changesets, 190
tables), pg_dump fed to sqlc unmodified, real queries ported from the
glossary/sso HugSQL spike branches, and the generated file compiles with
schemas that accept valid rows and reject bad ones.

See dev/src/dev/sqlc/README.md for open issues (no H2 engine, weak param
type inference, no dynamic SQL).
2026-09-15 23:06:24 -06:00
Chris Vandoodewaard
14fe567687 only auto-patch current version (#82368)
only auto-patch latest
2026-09-11 14:56:29 -04:00
technomancy
daceb36e65 Snowflake schema specified in :additional-options should take precedence (#82364)
Fixes https://github.com/metabase/metabase/issues/65493

If you have a `details.schema` property set in your connection (this was
removed in v42 in favor of schema filters), you can't change it in the
UI in the current Metabase version.  If you wanted to specify a schema
through the "Additional JDBC connection string options", the old
buried schema property would take precedence anyway.

This makes it so the deprecated field is removed when the schema is
set in additional-options.

I've verified this using a real connection and querying `SELECT current_schema()`
as suggested in the bug report.
2026-09-11 11:20:31 -07:00
github-automation-metabase
1c561d056c Tighten ratchets (#82365)
Co-authored-by: github-actions <github-actions@github.com>
2026-09-11 17:42:07 +00:00
Chris Truter
dd9d8d0579 Ratchet module boundary escape hatches (#82300)
Protect against bad things happening because agents and reviewers aren't aware that they're bad.
2026-09-11 17:36:42 +00:00
metamben
d0cbe49c68 Prime app-db connection-pool before use (#81601)
* Prime app-db connection-pool before use

Closes QUE2-833
Closes #81440

* Fix linter warnings
2026-09-11 17:08:38 +00:00
Nick Fitzpatrick
3e83533d06 BE PDFs - target 300dpi for rendered visualizations (#81078) 2026-09-11 13:55:16 -03:00
Chris Truter
eff3209a34 Add hierarchy to the backend module system (#82301)
This will allow us to resolve a lot a cycles in the module graph, and increase encapsulation considerably.
2026-09-11 18:48:15 +02:00
Alex Yarosh
93aabef5d3 docs: expand data modeling overview (#82309)
* data modeling overview

* copy paste strikes again

* Apply suggestions from code review

Co-authored-by: Michael Eby <38187276+michaeleby1@users.noreply.github.com>

---------

Co-authored-by: Michael Eby <38187276+michaeleby1@users.noreply.github.com>
2026-09-11 16:37:49 +00:00
Anderson
1a8883d604 Revert "Apply responsive spacing and typography to dashboard chart cards" (#82346)
* Revert "Apply responsive spacing and typography to dashboard chart cards (#81…"

This reverts commit 13e00d55b9.

* Update Loki Snapshots

---------

Co-authored-by: Metabase Automation <github-automation@metabase.com>
2026-09-11 13:10:36 -03:00
Timofey Kachalov
0cf4be2fd2 Fixed MCP search response schema validation (#82205) 2026-09-11 20:02:54 +04:00
Eric Normand
615cfdb48b Fix race in detached-fetch!-sweeps-stalled-fetches-test (#82347)
The test read `:future-ref` straight off the registry entry as soon as the
fetch signalled it had started. But `detached-fetch!` registers the entry
before the fetch begins and only stores the future just after submitting it,
so the thunk can deliver `started` before the `reset!` lands. When it does,
`stalled-future` is nil and `future-cancelled?` throws an NPE.

Under 300 spinning threads a probe caught the window 1-4 times per 20000
fetches; inserting a 5ms sleep between the `future` and the `reset!` makes
the old test fail every run.

Poll for the future instead. Only the test is affected: the sweep acts on
entries older than *fetch-max-age-ms* (an hour), so it can never observe one
whose `:future-ref` is still unset.


Claude-Session: https://claude.ai/code/session_01J1yFPpYiWH2ik8LB9rs7BT

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-11 10:56:49 -05:00
Alex Yarosh
19989ab207 docs: cloud migration (#82305)
* Revise migration steps 

Updated migration guide to clarify steps and include images.

* Delete docs/cloud/images/migrate-button.png
2026-09-11 15:36:37 +00:00
Alex Yarosh
efb95e2e18 docs: add glossary article (#82308)
* add glossary article

* Optimized 2 images, saved 219.4 KB (42.5% smaller)

* link

* Update docs/data-modeling/semantic-layer/glossary.md

Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>

---------

Co-authored-by: Metabase Automation <github-automation@metabase.com>
Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>
2026-09-11 15:29:28 +00:00
Riley Thompson
3ad97e619d Use multiple jvms per runner for bigquery tests (#81204)
* multiple jvms per runner for bigquery

* trigger CI

* add failing test

* lower timeout

* remove debug stuff

* clean up comments

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-09-11 11:27:34 -04:00
Anna Tomka
5b7c89d8ee GDGT-2478 Translate PasswordInput toggle label at call time (#82338)
* GDGT-2478 Translate PasswordInput toggle label at call time

* Update Loki Snapshots

---------

Co-authored-by: Metabase Automation <github-automation@metabase.com>
2026-09-11 15:26:10 +00:00
Riley Thompson
05d2f186a0 Use calendar days for bigquery datetimeDiff (#82194)
* BigQuery: use calendar days for datetimeDiff day unit

Fixes #82193

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Avoid :raw for date part in BigQuery day diff

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* clean up

* add timezone to test

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-11 10:53:37 -04:00
Phoomparin Mano
c8f77a0925 Data Apps Permissions (#79822)
* add saved question extraction prototype

* fix additional failure cases

* refactor: reconciler

* revert me: manual skill file edit for source swapping

* set data permissions to view for databases referenced in the query

* rename data app collection and group

* prevent agents from removing savedQuestionSourceId

* docs: clarify data app query location

* chore: cleanup query-sync canonical module

* refactor: split query tests into different files

* Swap query source for data app query hooks and apply dynamic filters on top of it (#79584)

* code improvements and bug-fixes for data apps permissions

* Add data apps model imports

* address reviews on data apps permissions

* run sync queries before vite build

* Add data app resource link (#79878)

add button to view data app resources

* Show permission error when users cannot access a data app (#79874)

show permission error page when user do not have permission to view data apps

* Add data app group link in overflow menu (#79873)

Add data app group link

* Store and reconcile data app entity ids in data_app.yaml (#79833)

* Store data app resource entity IDs

* make entity ids in data_app.yaml authoritative

* add top-level value after path key

* simplify data_app.yaml manifest parser

* only allow changing data app collection if empty

* consolidate backend test helpers and remove redundant tests

* chore: restore README changes

* ci: fix failing tests

* Add support of actions to Data App reconciler (#79842)

Add support of actions to Data App reconciler

* Fix bugs/inconsistency and add more e2e for data apps reconciler/syncing (#80002)

* Update semantic schema skill about preventing segments and measures in dynamic parts of a query in Data Apps (#80288)

* Grant view data access on a table-by-table basis in data app groups (#80189)

* grant view data on table instead of database level

* refactor table scan in reconciler

* update e2e tests for permission grants

* Grant data app view permissions per table

* Skip unchanged data app permissions

* add e2e data app test for sandboxing

* reset host app sources after runs

* Preserve sandboxing for data app users (#80406)

* add e2e data app test for sandboxing

* improve e2e tests for data apps

* Remove sandbox E2E from table permissions change

* preserve existing sandboxing from sandboxed groups in data apps

* assert that we grant non-sandboxed tables

* Explain data app sandbox exclusion

* Add data app sandbox E2E coverage

* Exercise sandbox preservation in data app E2E

* Disable saved question sources in data apps (#80139)

* Disable saved question sources in data apps

* Preserve question-only schema scope

* Remove the saved question schema parameter

* Copy table-sourced metrics into data app collection and do not codegen card-sourced metrics (#80086)

* copy table-sourced metrics into the data app collections

* copy table-sourced metrics into the data app collections

* Address metric sync review feedback

* Accept keyword metric display in query response

* Fix data app metric reconciliation ordering

* Remove old lockfile format

* exclude card-sourced metrics from schema generator

* do not use card terms in metrics to avoid ambiguity with card-sourced metrics

* fix model imports

* Fix data app metric reconciliation

* refactor: cleanup reconciler unit tests

* Fix duplicated metric client helper

* add safe metrics path

* Handle unsupported metric dependencies

* Fix data app CI test regressions

* Fix "Re-adding removed query fails to adopt deleted card" (#80341)

* Clean-up data apps permissions endpoints (#80825)

* Remove entity ids from data_app.yaml (#81317)

Remove data app entity ids from data_app.yaml

* Adjust data apps permissions code (#81093)

* Remove the data app table permission grants (#81584)

* Add permissions_group.is_data_app_group and hide data app permission groups from UI (#81705)

* Add data app user management and access warnings (#81839)

* Address data app user access review feedback

* additional e2e test cases for bugs in manage users page

* fix failing frontend type check on unit test

* fix migration id collision for data apps

* combine migration changes into one file

* fix data app metric sync after trashing copied metrics

* fix cli runtime dependency packaging for embedding sdk

* avoid loading data app bundle during resource setup

* prevent redundant data app collection permission updates

* allow admins to remove data app members after token expires

* update docs and add e2e test for missing resource collection case

* isolate data app sync using transactions

* do not leave user on an empty page when removing the last member on the page

* reserve sandbox-host slug in data app config

* ignore generated data app action fixtures

* move queries into db.clj

* preserve legacy permissions when adding data app members

* regenerate the kondo modules config

The data-apps module reaches for :model/Permissions and :model/DataPermissions,
neither of which was recorded in :model-imports, so metabase.core.modules-test
fails on the current tree.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011nNKkhQjjiUSM7qwLReyfP

* reject data app queries whose metrics read another card

Sync copies a referenced metric but rewrites nothing inside the copy, and
copies nothing that copy in turn reads, so those references still point at
the originals. The app publishes successfully and then fails for every
viewer without access to the originals' collections.

Checking while resolving the query, rather than at codegen, also catches a
metric edited into that shape after its schema was generated.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011nNKkhQjjiUSM7qwLReyfP

* exclude metrics that read a saved card anywhere from generated schemas

source-card-id only reads stage 0, so a table-sourced metric that joins a
saved question passed the codegen filter. The CLI checks the whole query and
rejects such a metric, aborting sync-resources for any app that uses it.

Checking the whole query here keeps codegen consistent with what sync
supports, and removes the database round trip that asked which of the
referenced cards were metrics -- the step that let a saved-question join
through in the first place.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011nNKkhQjjiUSM7qwLReyfP

* resolve copied resources' table dependencies through the server

Numeric source-table entries miss implicit joins: a copied model querying
Orders and reading Products.Category through a foreign key names Products
nowhere, so it was left out of the app's permission warnings.

Run the same metadata-based lookup the query endpoint uses, through a new
endpoint the CLI calls for copied models, actions and metrics, and drop the
JSON walk it replaces.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011nNKkhQjjiUSM7qwLReyfP

* cover the create-queries grant the database defaults leave on app groups

An app group that predates the first non-audit database is given
create-queries :query-builder-and-native by set-new-database-permissions!,
alongside the :blocked view-data it special-cases for app groups. A user with
data access through another group would gain native query authoring just by
joining the app.

The reconciler already resets it, but nothing failed when :perms/create-queries
was dropped from its list. The pairing cannot be produced through
set-database-permission!, which cascades view-data to :unrestricted for any
create-queries value other than :no, so the test writes the row the way the
defaults insert it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011nNKkhQjjiUSM7qwLReyfP

* prevent models based on saved questions from being copied

* return to a valid member page after overlapping removals

* do not resolve query objects when either enabled flag is false

* prevent shorthand properties and spreads from overriding generated ids

* prevent adding duplicate members and retry only members that are failed to add

* define schema for data app query endpoint

* reconcile only the current data app group during resource setup

* expand the schema fingerprint

* add data app group field in permission group schemas

* prevent new databases from granting query creation to data app groups

* throw validation errors when invalid query spec fields are passed

* do not fail validation on columns metadata

* add enabled metadata to query definition schema

* accept more data app specific query metadata and binning options

---------

Co-authored-by: Timofey Kachalov <timofei@metabase.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-09-11 15:41:01 +01:00
Timofey Kachalov
1626b31be7 Fix flaky child-frame realm gate in the data-app sandbox (#82324) 2026-09-11 18:07:04 +04:00
metamben
2659b87941 Throw on uncoerceable user attributes (#82118)
Closes QUE2-842
Closes #81821
2026-09-11 14:05:29 +00:00
metamben
b2c923e496 Remove explicit :fields on the last stage of pivot queries (#82111)
Closes QUE2-825
Closes #81203
2026-09-11 17:03:42 +03:00
Chris Vandoodewaard
cc1fa36312 add workflow combining quick and simple PR checks (#82126)
* combine workflows into single pr.checks.yml

* fixes and better checkout

* ignore parallel lint failure

* add filter to checkout to avoid slow checkouts

* remove check
2026-09-11 13:59:23 +00:00
github-automation-metabase
498dfc7435 Tighten ratchets (#82331)
Co-authored-by: github-actions <github-actions@github.com>
2026-09-11 13:16:25 +00:00
Ben Grabow
6780dbbf52 Allow multiple intermediate ns segments (#82125)
* Allow multiple intermediate ns segments

* Detangle llm settings

# Conflicts:
#	src/metabase/llm/provider.clj

* Move settings.providers to providers.settings

* Move providers.settings to provider.settings

* Loosen filter to include all `metabase.llm.*` settings

* Merge from master

* Delete obsolete linter ignore

* Clarify linter message
2026-09-11 08:11:09 -05:00
Jeff Bruemmer
86f64c3d9f docs - update docs job (#82327)
update docs job
2026-09-11 08:57:59 -04:00
Anna Tomka
ca17b89463 GDGT-2478 PasswordInput (#82117)
* test(ui): add PasswordInput Storybook story

* test(ui): add PasswordInput to Loki as baseline for restyle

* feat(ui): restyle PasswordInput from Figma and add input-background token

* test: cover PasswordInput visibility toggle in unit and e2e tests

* fix(PasswordInput): use highlight border color in focused error mode by overriding Input styles

* Update Loki Snapshots

* test(ui): add Loki reference for PasswordInput Overview story

* fix(PasswordInput): address review feedback

- keep border width constant and use an inset ring for focus/error to avoid layout shift
- tie right padding to the toggle button size so text clears it in lg
- restrict size prop to sm | md | lg
- order color keys alphabetically
- explicit visibility assertion in the e2e spec

* Update Loki Snapshots

---------

Co-authored-by: Metabase Automation <github-automation@metabase.com>
2026-09-11 14:18:41 +02:00
Anna Tomka
38b7b8c0b7 GDGT-3212 add missing inner border in dark theme (#82203)
* fix(Pill): add missing inner border in dark theme

* fix(Pill): drop explanatory comment per review
2026-09-11 14:18:11 +02:00
Kevin Kelleher
94e58da90d Fix auto-wiring dashboard filters to dashcards with multiple series (#81913)
* handle auto-wiring dashboard filters to dashcards with multiple series

* review feedback
2026-09-11 07:24:06 -04:00
Oleh K
6a035e234c Use JetBrains Mono as the code font (#82278)
* Use JetBrains Mono as the code font

Ship JetBrains Mono (regular + bold) and point the monospace font family
at it, replacing the hardcoded Monaco/Menlo stacks scattered across the
codebase.

- Add the woff2 files, @font-face declarations, and the type, fallback
  map, docs and backend test entries that keep the admin font picker in
  sync with the fonts directory.
- Route every monospace call site through
  --mb-default-monospace-font-family, and define that variable in the
  embedding SDK's variable set, where it was previously missing. With
  the variable always defined, the per-call-site `, monospace` fallback
  is redundant and is dropped.
- Inline the variable at the CodeMirror theme, which was the only
  consumer of monospaceFontFamily, and delete typography.ts.
- Set the Code component to 13px/22px per the design spec.

* Update Loki Snapshots

---------

Co-authored-by: Metabase Automation <github-automation@metabase.com>
2026-09-11 12:12:15 +02:00
Andrei Simionescu
b75caae5f3 AI settings docs: a provider connection takes credentials, not just an API key (#82026)
Bedrock takes an AWS access key pair and Google Gemini Enterprise a service
account key file, so the AI settings page can't keep calling every provider
connection an API key.
2026-09-11 09:18:39 +01:00
Fraser
a5af036433 Rehome the analytics export machinery to EE audit_app (#79808) 2026-09-11 19:04:08 +12:00
Romeo Van Snick
1c5ba041d6 Stop the metabot viewing-context tests from depending on absent entity ids (#82248)
The tests pass ids like 789 for entities they do not create. format-entity looks the
id up, and the test asserts on the fallback rendering that a 404 produces. When a
parallel test happens to hold that id, read-check gives a 403 instead, which renders
nothing, and the test fails.

Use Integer/MAX_VALUE, which no fixture allocates.


Claude-Session: https://claude.ai/code/session_01GCoSa7L7KrXh42rDRYR3Nk

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-11 15:11:30 +10:00
Romeo Van Snick
86c23eaf5e Read a card's UI parameters off its question (#82238)
* Read a card's UI parameters off its question

* Drop a tombstone comment

* Pin the agreement between getCardUiParameters and Question.parameters

* Type the unsaved card in the agreement test

* Format the agreement test
2026-09-11 15:11:21 +10:00
Fraser
1447d9cf97 Move the application permissions selector slot into current-user (#81846)
* Move the application permissions selector slot into current-user

PLUGIN_APPLICATION_PERMISSIONS.selectors was only read by current-user, so the slot now lives there as PLUGIN_APPLICATION_PERMISSIONS_SELECTORS with its OSS defaults, and the EE application permissions plugin registers into it from metabase/current-user. getRoutes and tabs stay in metabase/plugins for the admin permissions editor.
2026-09-11 16:42:11 +12:00
Romeo Van Snick
90688e49a7 Translate at call time, not at import time (#82147)
* Translate viz nouns and formatting operators on demand

`noun` was the one field on a visualization definition still translated at
import. `get noun()` matches what ObjectDetail and TreemapChart already do,
and the registry stores a definition by reference, so the getter survives.

The table-formatting operator names were four module-level objects. They are
functions now. `ALL_OPERATOR_NAMES` becomes `getAllOperatorNames()`, which
its three readers call.

25 of the 124 sites that translate before a catalogue can be applied.

* Translate settings titles on demand

`title` in a settings definition follows the same rule as the rest of the
object. 152 settings already use `get title()`; these 23 were the laggards.

The group-managers confirmation held two frozen strings, so it becomes a
function rather than an object with getters.

49 of 124.

* Lift translated containers into functions

Where a whole array or object is translations, the function goes on the
container so consumers keep a plain array or object: PieChart's ring titles,
the whitelabel illustration options and descriptions, PivotTable's partitions
(which held JSX built at import), the model-caching options, the cloud
migration messages, and the geo filter placeholders.

Where only a field or two of a mixed config is translated, and call sites read
it by property, the getter goes on the field instead. That covers the
collection configs, the permission option records and the loading messages,
none of which are spread anywhere.

Four files carried a whole-file disable rather than one per site, hiding 20
sites behind 4 directives. The real total is 140, not 124.

118 of 140.

* Clear the last module-scope translations

Validation schemas become functions, matching getOidcFormSchema and the other
lazy schemas already here. Module-local error strings become functions called
at their use sites. Three exported constants follow: NULL_DISPLAY_VALUE,
BEFORE_UNLOAD_UNSAVED_MESSAGE and NO_X_AXIS_VALUES_ERROR_MESSAGE, across 37
files.

The two Mantine `aria-label` defaults get a getter. Mantine merges
`defaultProps` by spread on each render, so the getter is read every time.

`ttag/no-module-declaration` now passes across all five trees it covers with
no opt-outs left, and `--report-unused-disable-directives` confirms none were
stranded.

* Stop spreads from flattening lazy translations

A spread or a rest destructure reads every getter it copies. Where an object
holds `get title()` so the string translates when it is read, spreading that
object at module scope translates it at import instead, which is the bug the
getters were meant to avoid.

Adds `mergeLazily` / `omitLazily`, which copy property descriptors instead,
and uses them where the flattening happened at import:

- `viz-core/lib/settings/utils.ts` flattened twice, once in the rest
  destructure and once in `...def`, so every setting built through
  `fieldSetting` / `metricSetting` / `dimensionSetting` froze at import.
- `palette/shortcuts/index.ts` spread five shortcut maps into one.
- `PaletteShortcutsModal` spread each shortcut again while grouping, so that
  now happens per render.

Measured with a test that imports every source module with `ttag` mocked:
40 modules and 370 translation calls at import, down to 35 and 256.

* Build parameter, admin and graph settings when read

* Build option lists when they are rendered

* Guard import-time translation with a test

* Make the instance analytics message a function so the plugin registry can replace it

* Resolve admin path names when the navbar renders them

* Give the enterprise caching tabs a lazy name too

* Skip generated cli-snippets from the side-effect scan

* Key the null slice by its untranslated label

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NUuaopJ7ZtMVvdwwxF6qyx

* Fail the import-time guard on unlisted import errors

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NUuaopJ7ZtMVvdwwxF6qyx

* Read lazy translations once per render, not per lookup

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NUuaopJ7ZtMVvdwwxF6qyx

* Let jest transform the MCP ext-apps package

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NUuaopJ7ZtMVvdwwxF6qyx

* Skip the SDK cli scripts in the import-time guard

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NUuaopJ7ZtMVvdwwxF6qyx

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-11 14:16:04 +10:00
Romeo Van Snick
60f2668137 Build dashboard and query-runner questions from the store (#82241)
* Build dashboard and query-runner questions from the store

* Cover the pivot endpoint selection
2026-09-11 14:13:19 +10:00
Romeo Van Snick
503c3bf068 Build the card embed and SDK questions from the store (#82253)
* Build the card embed questions from the store

* Read the SDK parameter values off the question
2026-09-11 10:27:16 +10:00
Fraser
a6de42f467 Re-tier styled-components to U4 and enforce its shared tiers (#82256) 2026-09-11 12:22:50 +12:00
github-automation-metabase
c998859487 Auto-update documentation for master (#82298)
Update generated docs (master)

Co-authored-by: github-actions <github-actions@github.com>
2026-09-10 23:37:23 +00:00
Alexander Polyankin
a491bd11be Namespace the table name BigQuery's compiled native form carries as :qp/table-name (#82275)
* Namespace the table name BigQuery's compiled native form carries as :qp/table-name

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Update the BigQuery compile tests for :qp/table-name

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Update the remaining BigQuery compile tests for :qp/table-name

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-10 19:23:06 -04:00
Anderson
0a30b63176 Trim redundant Data Studio TablePicker end-to-end tests (#81848)
Trim redundant Data Studio TablePicker E2Es
2026-09-10 19:18:50 -03:00
technomancy
f1ce332568 Refuse to create new tables in static datasets with Snowflake (#82222)
Refuse to create new tables in static datasets with Snowflake.

Table creation will always succeed on non-static datasets, but if it's
static then the table being created has to be a table that's in the
dataset definition.

Ensure all tests which alter the dataset by creating new tables
declare a non-static dataset. The transform-related tests switch over
to using transforms-dataset/transforms-test which most of them are
already using.

Upload tests have their dataset defined in a fixture so we can do it
all in one place instead of editing 60 different deftests.
2026-09-10 14:55:14 -07:00
Will Fry
1690028d0f GDGT-2653: uncomment E2E assertions (#82199) 2026-09-11 00:10:43 +03:00
Andrei Chernykh
0c76874681 Move SSO user provisioning a toggle card under server settings (#82232)
* Make SSO user provisioning a toggle card under server settings

* Debounce provisioning writes and drop the real timers from the spec
2026-09-10 13:52:45 -07:00
Chris Vandoodewaard
a6be073fb2 auto-merge e2e timing update prs (#82292)
* create PR using CLI, also auto-approve/merge

* cleanup string, check if things changed

* remove extra blank line

* namespace file
2026-09-10 15:05:29 -04:00
Ben Grabow
cb4541b965 Extract dispatch type hierarchy (#82226) 2026-09-10 13:37:56 -05:00
Ben Grabow
463fb9355b Extract entity types (#82265) 2026-09-10 13:23:50 -05:00
Andrei Chernykh
791b649cc0 bump qs to 6.16.0 (#82283) 2026-09-10 11:06:44 -07:00
Tyler Plude
085418edb8 [GIT-74095] Use git path separators in remote-sync export paths (#82284)
use git path separators in remote-sync export paths
2026-09-10 10:47:29 -07:00
Andrei Chernykh
1118c3e897 Bump js-yaml to 4.3.2 (#82282) 2026-09-10 10:36:08 -07:00
Alexander Polyankin
6e74e41eaf Strip undeclared keys from stored and legacy queries the way the request decoder does (#82273) 2026-09-10 13:32:50 -04:00
metabase-bot[bot]
38404fd9df [chore] Update E2E test timings with recent data (#81995)
Update E2E test timings with averaged data

Collected timing data from 7 days of test runs.
All specs with timing data were included (simple averages).

🤖 Generated automatically by update-e2e-timings workflow

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: GitHub Actions <actions@github.com>
2026-09-10 17:26:53 +00:00
Chris Vandoodewaard
8a7cd79f47 Bump backend deps to resolve security vulnerabilities (#82261)
* bump backend deps

* missed bump
2026-09-10 13:25:18 -04:00
Alex Yarosh
f3685652d3 docs: making semantic layer happen - org part (#82247)
* rename directories

* Revert "rename directories"

This reverts commit 13ebd4ae48.

* heard you like links

* what good is claude if it doesn't even find all the links

* Optimized 55 images, saved 698.8 KB (11.2% smaller)

* Fix docs links after merge

Co-authored-by: alexyarosh <16640109+alexyarosh@users.noreply.github.com>

* Fix merged docs regressions

Co-authored-by: alexyarosh <16640109+alexyarosh@users.noreply.github.com>

* Tighten merged docs links

Co-authored-by: alexyarosh <16640109+alexyarosh@users.noreply.github.com>

* Revert extra post-merge docs edits

Co-authored-by: alexyarosh <16640109+alexyarosh@users.noreply.github.com>

* link

---------

Co-authored-by: Metabase Automation <github-automation@metabase.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: alexyarosh <16640109+alexyarosh@users.noreply.github.com>
2026-09-10 12:49:43 -04:00
Andrei Simionescu
08497ded2f Bump csv-parse to 7.0.2 (#82266) 2026-09-10 15:48:24 +00:00
appleby
bfe0b19738 Update map tile server settings with suggested copy from DOC-301 (#81835)
* Update map tile server settings with suggested copy from DOC-301

And also include a link to the relevant page in the metabase docs.

* Add a note about not including private keys to the docstring for map-tile-server-url

so that the warning also shows up in the env var docs.
2026-09-10 08:44:01 -07:00
Riley Thompson
a155c1f4ee Document bigquery median behaviour (#82201)
* BigQuery: use PERCENTILE_CONT for median and percentile

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* update docs

* update doc

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-10 11:42:36 -04:00
Braden Shepherdson
9f48230b57 [PDFs] Respect the text wrapping settings on tables (#77090)
Fixes UXW-4714.

The static viz for tables uses a fixed width, but it was too big for the
PDFs, causing tables with long text in columns to lay out badly.

This PR disables the fixed width of `780px` on tables for PDFs, so that
the tables now lay out correctly.
2026-09-10 15:39:28 +00:00
Ben Grabow
503349c6ae Fix duplicates of the same namespace name (#81974)
* Fix collisions on metabase.notification.payload.core-test namespace name. Lint for future collisions.

* Include .clj-kondo/{src,test} in backend check triggers

* Expect the new project test namespace

* Symbolically reference project test suite namespaces

* Include resources, test_resources, test_config dirs in project checks

* Return [namespace relative-path] pairs.
Skip local/src

* Catch read-file errors and explain why some files may not have ns forms

* Sort paths that collide
2026-09-10 10:24:17 -05:00
Jeff Bruemmer
4d60c4bcf5 docs - nav link check (#82211) 2026-09-10 15:08:17 +00:00
Chris Truter
d9f905b7da Fix H2 migration failures after running transforms (#82135)
* Copy the transform tables in dump-to-h2 and load-from-h2

None of the nine transform_* models were in copy/entities, but metabase_table is,
and it carries a transform_id FK to transform. A table row reached the target
referencing a transform row that never arrived, and the deferred FK check
rejected it at commit.

transform_run_cancelation also goes in entities-without-autoinc-ids, since its
primary key is run_id rather than an autoincrement id.

Adds copied-tables-have-copied-fk-parents-test, which migrates a throwaway H2
database and fails if any copied table's FK parent is not copied.

* Simplify the copy foreign-key regression test

* Clarify the foreign-key coverage test names
2026-09-10 14:42:03 +00:00
Andrei Simionescu
36f273d523 BOT-1790: Redact Metabot chain-of-thought details in embedded sessions (#81872) 2026-09-10 14:01:55 +01:00
Alexander Polyankin
38cda45f69 Type every :any and [:map-of :keyword ...] property in the API (#82196)
* Make defendpoint reject request schemas that reach an open map, a keyword-keyed map-of or :any

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Close the MBQL 5 query and column schemas and declare the keys the query processor adds

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Type the shared schemas requests reach and drop ms/Map and ms/JSONString

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Close every map schema a request can reach

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Regenerate the modules config

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Keep action parameters string-keyed and type lens params and permission graph ids

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Declare the sandbox coercion keys a column's options can carry

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Declare the alias-info and Mongo keys a column can carry and strip internal keys from legacy column metadata

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Carry the Mongo column keys in the serialize test fixture

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Declare the keys the frontend and query processor put on columns, parameters and card embeds

- column metadata: :remapping, a typed :lib/options, the nest-breakouts option key
- query parameters accept the rest of a stored filter declaration
  (sectionId, isMultiSelect, values source config etc.)
- static-list values may be one-tuples; raw-value template tags may
  carry a nil widget-type
- card embed attrs declare chart_href, child_target_id and host_data;
  explore filter refs accept the legacy field_ref the chart column has
- notification card echoes a metric's query_description
- serialization deep-sort orders mixed keyword/string keys
- viz-settings column refs are normalized before matching; a Field
  hydrated with its table can be fingerprinted; broken segment
  definitions read back as nil
- test fixtures: real fingerprints, snippet tag type, http channel
  details, comment content, no min_duration_ms on duration strategies

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Wire the defendpoint check, keep every map single-keyed, and apply the review findings

- defendpoint now calls closed-schemas/check! on every request schema;
  the walker no longer lets :any leak through collections, treats
  map?/:some as :any, skips :=/:enum/:fn values and never skips the
  default entry
- no map mixes keyword and string keys: ms/string-keyed-object gives an
  opaque JSON object string-keyed declared entries; ProseMirror attrs
  (documents, comments, explorations embeds), CSV upload parts, search
  weights and testing query specs use it and their readers use string
  keys; the serdes deep-sort comparator is back to master
- the Metabot context is fully typed from the frontend types instead of
  an opaque default; a draft transform query without a database is
  accepted
- permission graphs, settings PUT and SCIM PATCH reject a body whose
  map-of entries the decoder dropped; ad-hoc query endpoints drop a
  client-sent cache strategy; action parameter values keep collections
  out without nil-ing them; SCIM meta.resourceType is optional; sandbox
  user attributes are LoginAttributes; a metric filter clause carries its
  offset options; explore filter refs are the legacy field_ref the
  chart column has; result_metadata written by older releases and
  hasVariableTemplateTagTarget on parameters are declared; a bare
  map-of on a request path is guarded against scalars

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Close the endpoints tests define and let a column's target and name field be an id reference

The check runs on every defendpoint, the ones test namespaces define included. A column's
:target and :name-field are hydrated Fields on the backend but id references in the
metadata the frontend hands over; the closed column schema nil'd the reference, which
lost the search field of a model's primary key.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Hand an action's query only the parameter values, and fix two fixtures the closed schemas caught

A query action's stored parameters carry the frontend's widget settings; the query gets
the value keys. The Metabot viewing-context tests use ids no card can have, and the
Mongo cache test spells the strategy key the schema does.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Declare the table name BigQuery's compiled native form carries on a native stage

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Keep a BigQuery RECORD's sub-field sync metadata off its result column, and fix a binning fixture

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-10 08:17:08 -04:00
Timofey Kachalov
fc18a66b6c Add remaining data apps sandboxing e2e (#82170) 2026-09-10 15:44:38 +04:00
Chris Vandoodewaard
433561ad5b Move Snyk scans into the Lint Security workflow (#82197)
* Move Snyk scans into the Lint Security workflow

Fold snyk.yml's three jobs into lint.security.yml as snyk-generate-poms,
snyk-frontend-projects, and snyk-maven-projects, and delete snyk.yml.

Snyk now runs on the existing daily 11:00 UTC cron instead of its own
05:00 cron and dependency-file push trigger. The notify job reports Snyk
failures to Slack alongside CodeQL and Trivy, and Snyk's upload-sarif is
bumped from v3 to v4 to match the rest of the workflow.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* add timeouts

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-09-10 07:16:23 -04:00
Will Fry
3c23f921e2 GDGT-3132 fix misaligned undo toast (#82045)
* GDGT-3132 fix misaligned undo toast

* migrate UndoListing to from styled components to CSS module

* absolute px -> rem, compact -> compact-md

* fix inconsistent toast height, use line-height instead of measuring the DOM to handle single vs multi line layouts

* strip comments
2026-09-10 11:06:19 +00:00
Oleh K
7f4883cc49 Fix inefficiencies in formatValue (#82103)
* GDGT-3195: Fix inefficiencies in formatValue

formatValue is fine per-cell but pathological in bulk; the "Copy results"
button made this visible, with a 100x2000 grid spending ~4.5s in this path.

- Drop a dead dayjs strict parse. The format literal 'T' is not valid dayjs
  escaping (dayjs wants [T]), so isValid() returned false for every value.
  It sat above the string/number/boolean branches, so every non-date cell
  paid for it.
- Only JSON.parse the value when a markdown_template will consume it,
  instead of on every call.
- Memoize the per-column type predicates (isURL/isEmail/isTime/isDate/
  isNumber/isCoordinate/isBoolean) in a WeakMap keyed on the column. Each
  fans out into recursive type-hierarchy lookups that cross into CLJS, and
  they only depend on the column. Entries snapshot the type fields they
  were computed from, since the custom-viz API hands third-party code
  mutable column objects.
- Skip new URL() construction for values with no ":" in getUrlProtocol.
- Cache Intl.NumberFormat by its constructor options, bounded at 100
  entries with oldest-first eviction.

Also make the value-formatting signatures honest: formatUrl, getLinkText,
getLinkUrl and getRemappedValue take unknown, which is what they actually
receive. getUrlProtocol keeps its string signature, with getLinkUrl
narrowing before the call. This removes both `value as string` casts.

Benchmarked with a synthetic 100x2000 copy path (medians of 5):
full table 4512ms -> 440ms, text 1231 -> 12, currency 637 -> 20,
integer 535 -> 15, boolean 343 -> 5, email 101 -> 4.

Claude-Session: https://claude.ai/code/session_01KUcocQj2bJPwVBi4dXKyEP

* Cleanup

* Revert the createPlainCellFormatter change

Hoisting the options spread out of the per-cell closure is not worth the
churn. Measured in isolation with a realistic 22-key columnSettings, the
spread costs 0.023us per cell against 0.002us to reuse a hoisted
reference, so it saves ~4ms per 200k cells — a rounding error next to the
dead dayjs parse at ~13.5us per cell, and nothing at all for columns with
click behaviour, where both shapes spread per cell anyway.

* Address review feedback

Guard the markdown_template JSON.parse with a typeof check instead of
carrying the placeholder cast comment.

Cover the two changes the review called out as untested:

- numberFormatterForOptions returns the same formatter for equivalent
  options, a separate one for different options, and evicts the oldest
  entry once the cache is full. The eviction test resets the module so
  the 100-entry boundary does not depend on what earlier tests cached.
- formatUrl passes strings and numbers through untouched and renders
  booleans and objects rather than returning them raw.

Both were checked by mutation: breaking the coercion and stubbing the
cache lookup to always miss fails exactly those assertions.
2026-09-10 12:49:27 +02:00
Fraser
667d038d46 Move the upload error modal slot into status (#82242) 2026-09-10 21:18:59 +12:00
Fraser
19df422bbd Move the storage setup provider slot beside its context (#82243) 2026-09-10 17:44:42 +12:00
Fraser
9fff12013e Move the DataReference library pane slot into querying (#81858)
* Move the DataReference library pane slot into querying

DataReference is the slot's only reader and querying owns its props type, so the slot lives next to it as PLUGIN_DATA_REFERENCE.LibraryPane.
The enterprise library plugin registers into it at boot, and plugins/oss/library.ts no longer imports querying's DataReference types.
2026-09-10 17:44:28 +12:00
Fraser
4bc6ccb00c Move the verified search filter slot into search (#81860)
* Move the verified search filter slot into search

SearchSidebar is the only reader of PLUGIN_CONTENT_VERIFICATION.VerifiedFilter, so the field becomes PLUGIN_SEARCH_FILTERS in metabase/search/plugin, next to the other search filter components. The EE content verification plugin registers the filter there from initializePlugin, and the rest of PLUGIN_CONTENT_VERIFICATION stays in plugins without the search import.

* Declare the search filters slot with definePluginSlot

* Import definePluginSlot from the plugins entry point

* Describe the verified filter default without naming its consumer

* Cover enterprise feature gates and plugin reset behaviour

* Remove redundant plugin registration tests
2026-09-10 17:44:12 +12:00
Fraser
81049d6069 Move the entity icon slot into hooks (#81857)
* Move the useGetIcon plugin slot into hooks/use-icon

The hook is the slot's only reader, so the slot and its OSS default live next to it as PLUGIN_ICON.
The enterprise collections plugin registers into it at boot, and plugins/oss/collections.ts no longer imports metabase/hooks/use-icon.

* Declare the entity icon slot with definePluginSlot

* Import definePluginSlot from the plugins entry point

* Cover enterprise feature gates and plugin reset behaviour

* Remove redundant plugin registration tests
2026-09-10 17:43:59 +12:00
Fraser
85ce065045 Remove the stale dashboard selectors side-effect entry (#82252) 2026-09-10 17:43:34 +12:00
Romeo Van Snick
dcb85ae924 Trigger the stats workflows from Run tests, gated on the uberjar artifact (#82244)
* Trigger the stats workflows from Run tests, gated on the uberjar artifact

Both workflows have recorded nothing since 2026-09-03. PR #81238 took the push
trigger off `uberjar.yml`, so master merges reach it only through
`workflow_call`. A reusable workflow reached that way raises no `workflow_run`
event, and both watchers went quiet. 143 master commits went unmeasured.

They now watch "Run tests", which is what builds the jar on a master merge.
Its conclusion is not a usable gate, because it is `failure` on master more
often than `success`, from jobs that have nothing to do with the jar. The
uberjar artifact is the right gate: a new job looks it up without downloading
it, and a docs-only merge that skipped the build finds nothing and stops.

Calling both from `run-tests.yml` would have been lower latency, but a stats
failure would then turn the master run red, and `rerun-workflows.yml` watches
that run and would re-run master CI over it. Keeping them in their own run
keeps a stats problem to itself.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01171Lkmug15JzrcN2aMBLMk

* Let a PR run the size benchmark against its own uberjar

A `workflow_run` workflow always runs the copy on the default branch, so a
change here cannot be tried out before it merges. This matches the trigger
Bundle Load Stats already carries. A called run measures and reports; the
upload step stays gated on the `workflow_run` path, so it writes no row.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01171Lkmug15JzrcN2aMBLMk

* Trim the trigger comments in the stats workflows

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-10 14:38:31 +10:00
Romeo Van Snick
3403f82c0a Build the dashcard question and metric series from the store (#82250) 2026-09-10 14:32:27 +10:00
Romeo Van Snick
95306c12fc Take metadata options at the store door (#82249)
* Take metadata options at the store door

* Build the conversation detail questions from the store

* Select the test fixtures through the store door
2026-09-10 14:32:17 +10:00
github-automation-metabase
5d1e1ad8f1 Auto-update documentation for master (#82227)
Update generated docs (master)

Co-authored-by: github-actions <github-actions@github.com>
2026-09-10 00:06:10 -04:00
Romeo Van Snick
6465693b78 Take a metadata provider where the database is already known (#82160)
* Take a metadata provider where the database is already known

* Fold the source-code provider into a hook
2026-09-10 13:37:15 +10:00
Romeo Van Snick
4e4cbac853 Give the visualizations a question builder instead of the metadata object (#82158)
* Give the visualizations a question builder instead of the metadata object

* Let the visualizations read the question builder themselves

* Read the store when a chart brush builds its question

* Drop the single-use question helper

* Give the map and object-detail views the question builder too

* Drop the last v1 metadata reads from the visualization tree
2026-09-10 13:37:10 +10:00
Romeo Van Snick
9762d0e164 Keep the CSP nonce out of the DOM, and off script-src (#82161)
* Give a script-src nonce only to responses that opt in

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DEcoNcbcoVpvPsGsX3ceua

* Hand the CSP nonce to page JS via the script element, not a DOM-readable JSON block

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DEcoNcbcoVpvPsGsX3ceua

* Pass the new script-nonce? arg at the common-test CSP call site

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GCoSa7L7KrXh42rDRYR3Nk

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-10 13:01:23 +10:00
Anderson
13e00d55b9 Apply responsive spacing and typography to dashboard chart cards (#81906)
* Apply responsive spacing and typography to dashboard chart cards (UXW-5181)

* Update Loki Snapshots

* Fix preview height regression and update e2e tests for dashcard size tiers

* Fix card heading typography and non-cartesian chart bottom padding

* Update Loki Snapshots

* Apply new styles only on cartesian charts

* Move changed styled components to css modules

* Update Loki Snapshots

---------

Co-authored-by: Metabase Automation <github-automation@metabase.com>
2026-09-09 23:51:25 -03:00
Romeo Van Snick
ad2d269251 Send the Field columns a parameter widget reads, and stop re-resolving them (#81936)
* Send the Field columns a parameter widget reads in param_fields

* Stop re-resolving a parameter widget's fields through the store

* Keep the empty default a parameter widget's fields prop had

* Read the driver-dependent Field columns in param_fields expectations
2026-09-10 12:04:48 +10:00
Romeo Van Snick
a22958cee3 Track page load times on every master merge (#80978)
* Add a harness for measuring bundle load time

* Report the second load and make the benchmark safe to run in CI

* Run the load benchmark across network and CPU conditions

* Sign the load benchmark in to a real instance

* Track page load times on every master merge

* Backfill load times for recent master commits

* Rewrite the load-time backfill in TypeScript with a tested core

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Launch the benchmark jar the way we ship it, and lint the .ts scripts

* Measure the whole load, not just DOMContentLoaded

* Do not let the performance marks gate a reading

* Drop the backfill workflow

The goal is a global trend, which the per-merge job builds on its own after a
few days. Backfilling only bought a readable chart sooner, at the cost of a
second workflow and the largest script in the PR.

The commit date stamp goes with it. It existed so a backfilled row could claim
its commit's date rather than today's, and the per-merge job never set it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012b67ubbXJVMuQrpuEksbeh

* Let a PR run the load benchmark against its own uberjar

A `workflow_run` workflow always runs the copy on the default branch, so a
change here cannot be tried out before it merges. The `workflow_call` trigger
lets CI hand the workflow a commit and reuse the uberjar it already built. A
called run measures and reports; it does not write rows to Stats.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01171Lkmug15JzrcN2aMBLMk

* Report each state from one load, not a median per reading

Every reading was medianed on its own, so a row described a load that never
happened. It could also put the readings out of order and print a page-ready
that precedes the TTFB beside it.

`measure.js` now picks the run whose DOMContentLoaded is the median of its
series and reports that run whole. The second visit happens once per browser
profile, so it stays the run itself, and its page-ready now comes from that same
load rather than from a median that included the series' first, uncached run.

`Cold spread %` still comes from all the cold runs. A spread is a property of the
series rather than of one load.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01171Lkmug15JzrcN2aMBLMk

* Read the largest paint with an observer, and cut the timeout

The first CI run reported a zero largest paint in all four conditions, and would
have done so forever. These entries never reach the performance timeline, so
`getEntriesByType` cannot see them. Only a `PerformanceObserver` can, and it has
to exist before the paint. Paint entries are buffered, which is why
first-contentful-paint needed none of this.

The browser keeps revising the value until the user interacts, and the reading is
taken once the page reports ready. It is a lower bound rather than the final
figure, which is what makes it comparable between commits.

That same run took 4.4 minutes, of which the measurement was 164 seconds. The
timeout drops from 25 minutes to 10.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01171Lkmug15JzrcN2aMBLMk

* Move the bench to frontend/test and write it in TypeScript

---------

Co-authored-by: Fraser <frasermcintosh1@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-09-10 11:42:14 +10:00
Fraser
e92dfab50e Refine TypeScript authoring and review skills (#82141) 2026-09-10 13:39:46 +12:00
Romeo Van Snick
7c4159eedd Read the query builder's cards and tables without the metadata object (#82159)
* Read the query builder's cards and tables without the metadata object

* Build the query builder's question from the store's builder
2026-09-10 11:19:39 +10:00
Romeo Van Snick
6dc53c3773 Keep the generated CLI snippets out of the side-effect registry (#82172) 2026-09-10 11:18:18 +10:00
Romeo Van Snick
86d1bf3ec7 Read table foreign keys from the store, not the metadata wrapper (#82163) 2026-09-10 11:17:59 +10:00
Chris Vandoodewaard
fe4e2beffa only re-run the failed specs for SDK component tests (#82220)
* granular re-run for sdk component tests

* update e2e-test to avoid clobering artifacts on multiple attempts
2026-09-10 12:27:57 +12:00
Romeo Van Snick
e096a361d8 Take a metadata provider in the detail-view query helpers (#82151)
* Take a metadata provider in the detail-view query helpers

* Select the detail-view queries instead of building them

* Take the provider for the table's own database

* Guard the table query selector's memoisation

* Type the table query guard against the API table
2026-09-10 09:13:56 +10:00
Ben Grabow
8b58b48375 Test registration fixes (#82130)
* Add missing requires

* Add missing requires

* Apply batched suggestions from code review

Co-authored-by: Ben Grabow <ben.grabow@metabase.com>

* Sort nses

* */keep-me
2026-09-09 17:28:31 -05:00
Fraser
ecde907aff Add definePluginSlot and a reset registry to metabase/plugins (#81926) 2026-09-10 10:01:23 +12:00
Andrei Chernykh
d1934fb897 Fix the JWT settings spec import after the store mocks move (#82228) 2026-09-09 14:45:58 -07:00
Jeff Bruemmer
03a39f47c8 docs - CLI updates (#81978) 2026-09-09 20:54:06 +00:00
thackervishal
ad22a8730c white_label - do not use hardcoded metabase in filenames for nonOSS (#81979)
* for EE/white_label - do not use hardcoded metabase in filenames
* test coverage includes test for special characters in application-name
* also use gibberish names in tests
2026-09-09 16:00:18 -04:00
Andrei Chernykh
3d145ccb73 Redesign the JWT group mapping section (#81928)
* Redesign the JWT group mapping section

* Tidy the JWT group mapping tests

* Type the group mapping hooks explicitly and tidy comments
2026-09-09 12:27:15 -07:00
metamben
1b93876984 Prevent inactive fields from breaking chain filtering (#82041)
Closes QUE2-820
Closes #80557

There are two changes:
1. handle `nil` nodes gracefully during traversal, and
2. don't add `nil` nodes to the graph in the first place.

Both of these changes alone fix the reported issue, but both make sense on their own.
2026-09-09 18:39:26 +00:00
Anderson
283b194a38 Stabilize Data Studio bulk-table e2e via shared TablePicker expand helpers (#81623)
* Stabilize Data Studio bulk-table e2e via shared TablePicker expand helpers

* Match unnamed-schema tables fetch and trim expand wait timeout to 25s
2026-09-09 15:33:00 -03:00
Alexander Polyankin
28b2ad9eb6 Remove Honey SQL arguments from db.clj (#82175) 2026-09-09 13:34:39 -04:00
Andrei Chernykh
5b705231d7 Collection filters UX followup (#79863)
* Stabilize collection type-filter options via an items-metadata endpoint, default the type checkboxes to unchecked, and hide the filters for collections of 10 or fewer items

* Show every type in the filter popover, disabling the ones without matching items

* Use kebab-case params on the metadata endpoint and adapt toolbar tests to the visibility threshold

* Scope the items metadata to the models the list shows, name its count total_items, and align root permissions with the items list

* Assert the restricted root metadata count against the items list instead of a fixture-dependent number

* Code review improvements

* Cleaned up kebob usage

* Adapt the branch to master's show-exploration-documents param and re-record kondo budgets

* Cover the items-metadata endpoint's exploration-document scoping

---------

Co-authored-by: Nathan Voxland <nathan@metabase.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-09 10:20:19 -07:00
Alex Yarosh
10f53bdcec Fix documentation URL for collection permissions (#82136) 2026-09-09 11:45:02 -04:00
Paolo Rascunà
2e1a0c1a2c BOT-1895: Stream reasoning tokens for 7 more LLM providers (#81956)
Every provider behind Metabot now streams reasoning to the chain-of-thought UI,
with one predicate per adapter deciding the settings gate, the request directive,
stream forwarding and replay together.

* BOT-1895: Shared groundwork. Drop the pre-registry saved-model fallback from
  Azure and Google; make tool_calls outrank reasoning in delta classification,
  which previously lost a tool call's only id/name chunk; consolidate
  reasoning-part coalescing into one helper.

* BOT-2027: Bedrock. Request thinking per model via the shared builders. The
  openai-family gate answers false — probes across all four GPT models confirm
  Bedrock never emits reasoning summaries.

* BOT-2028: Azure. Gate derived from the admin-typed deployment name, matched
  case-insensitively, which also fixes a latent temperature 400 on cased Claude
  deployments. Negative paths verified live; the positive path rests on docs.

* BOT-2025: Z.AI. Forward reasoning_content and send the thinking directive
  explicitly — GLM was already thinking invisibly on every chat. Thinking-only
  models get reasoning_effort and a 2048 floor on forced tool calls, flagged on
  the supported-models row so the sets cannot diverge.

* BOT-2024: OpenRouter. Forward flat delta.reasoning and classify all 26 models
  by live probe; the gpt-5.5/5.6 family streams under the server default, where
  an explicit enable verifiably suppresses it. Forced tool calls floored on
  mandatory-reasoning models.

* BOT-2026: Google Gemini. One catalog drives context windows, directive and
  gate. Thought summaries on chat, thinkingLevel LOW on structured (no off
  switch), and a 2048 floor — probed, a real prompt at 512 spent the whole
  budget thinking and returned no tool call.

* BOT-2030: Mistral. Net-new cost: the default sends no thinking and only
  "high"/"none" are accepted. Flattens vector content chunks into the shared
  xf's events, and adds the opt-in reasoning replay hook Mistral's docs require.

* BOT-2046: Moonshot. kimi-k3 always thinks — explicit effort, mandatory in-turn
  replay, 2048 floor; kimi-k2.6 is optional and disabled under a forced tool
  choice, which it rejects while thinking.

Co-authored-by: Paolo Rascuna <1936871+prascuna@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-09 17:42:56 +02:00
Chris Vandoodewaard
e49b98186b remove quarantine-engine paramater from test-driver action (#82107)
* remove quarantine-engine param from test-driver action

* remove uses of quarantine-engine param

* add timeout
2026-09-09 11:13:42 -04:00
Nicolò Pretto
f74d32c39a make sdk analytics use specific client names for data app and mcp apps (#82110) 2026-09-09 17:06:12 +02:00
Anderson
6431bc412b Fix chart color settings not applying in the query builder sidebar (#82140) 2026-09-09 11:45:11 -03:00
Tyler Plude
40a15bc485 [GIT-81733] Handle jar native filesystem paths (#81927)
* return a native filesystem path from get-jar-path

* extract code-location->path and add jimfs-backed regression test
2026-09-09 15:39:32 +01:00
Ben Grabow
b0c8103cdb Make batch timing stubs more reliable (#82073) 2026-09-09 09:30:16 -05:00
metamben
4b88b2ae79 Make sure we provide an order by clause when paging with SQL Server (#82039)
Closes QUE2-841
Closes #81988
2026-09-09 16:49:24 +03:00
Nathan Voxland
d3c7576ccb Validate local infile (#81967) 2026-09-09 08:41:57 -05:00
github-automation-metabase
7dbf078a40 Auto-update documentation for master (#82133)
Update generated docs (master)

Co-authored-by: github-actions <github-actions@github.com>
2026-09-09 09:32:21 -04:00
Jakub Chodorowicz
c0949b1d38 Gdgt 2465 popover (#81561) 2026-09-09 15:26:04 +02:00
Romeo Van Snick
ebdcd6c743 Take a metadata provider in the shared query helpers (#82150)
* Take a metadata provider in the shared query helpers

* Select the segment query instead of building it

* Take the provider for the segment definition's database

* Guard the segment query selector's memoisation

* Guard the segment query assertion against a vacuous pass
2026-09-09 22:05:52 +10:00
Andrei Simionescu
66038ba6d0 BOT-1427: link conversation user groups to filtered usage stats (#81869)
* BOT-1427: link conversation user groups to filtered usage stats

* Take the group id as an argument to the usage URL helper
2026-09-09 12:48:52 +01:00
Romeo Van Snick
97d871c78a Build the new metric page's queries from the store's builder (#82153)
* Build the new metric page's queries from the store's builder

* Select the new metric page's queries instead of building them
2026-09-09 20:57:50 +10:00
Romeo Van Snick
6bb98bce1d Build action-creator questions from the store's builders (#82154)
* Build action-creator questions from the store's builders

* Select the action-creator question instead of building it
2026-09-09 20:21:30 +10:00
Romeo Van Snick
c908146299 Build the browse-tables virtual question from the store's builder (#82155)
* Build the browse-tables virtual question from the store's builder

* Select the browse-tables url instead of building it
2026-09-09 20:20:56 +10:00
Romeo Van Snick
2d80c0e360 Build the model and source-card questions from the card the loader fetched (#82156) 2026-09-09 20:20:25 +10:00
Romeo Van Snick
7f589be657 Build draft questions from the store's builder (#82144)
* Build draft questions from the store's builder

* Read the store when normalizing a transform source
2026-09-09 20:18:36 +10:00
Kamil Mielnik
12e5f16918 GDGT-2796 Namespacing custom viz settings (#81181) 2026-09-09 16:38:51 +07:00
Egor Grushin
979b344073 Fix light theme border-highlight token (#82157)
Fix light theme border-highlight: brand[30] -> brand[60] (GDGT-2484)
2026-09-09 11:30:55 +02:00
Romeo Van Snick
4558ce654d Narrow the dashboard navigation cards, and build their questions from the store (#82148) 2026-09-09 19:11:28 +10:00
Romeo Van Snick
e584e95d48 Drop the question the click-behavior source filters never read (#82149) 2026-09-09 16:50:23 +10:00
Romeo Van Snick
6ed9837a03 Build parameter fields from param_fields, not the metadata store (#81935)
* Build parameter fields from param_fields, not the metadata store

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HNvsg9cPuNQ6BHiDwa1bif

* Answer a field's remapped value from the store, not a v1 method

The parameter pipeline now carries plain API fields, which have no
`hasRemappedValue` or `remappedValue`. Both callers read the store
through one selector instead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HNvsg9cPuNQ6BHiDwa1bif

* Carry an FK's target in param_fields for anonymous requests

The usual :target hydration reads the target Field through permissions,
which a public or embedded request does not have, so a public parameter
widget could not label an FK's values.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HNvsg9cPuNQ6BHiDwa1bif

* Give the number filter story a field with no values to list

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HNvsg9cPuNQ6BHiDwa1bif

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-09 15:17:40 +10:00
Romeo Van Snick
e4cc4a61ff Reduce the frontend to one memoize helper, and stop top-level caches leaking (#81866)
* Add a jest project for heap retention specs

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012b67ubbXJVMuQrpuEksbeh

* Scope the parsed x value cache to one getXValues pass

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012b67ubbXJVMuQrpuEksbeh

* Assert heap retention on the settled pass, not the first touch

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012b67ubbXJVMuQrpuEksbeh

* Key the log timestamp cache on the log object

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012b67ubbXJVMuQrpuEksbeh

* Key the schema viewer flow graph on the response object

The memo key serialized every table and field of the schema, so each distinct
schema held a megabyte-scale string forever. RTK Query returns a stable
reference per cache entry, so a WeakMap keeps the identity that useGraphSync
depends on without retaining the schema.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012b67ubbXJVMuQrpuEksbeh

* Key pie.rows on the dataset instead of a JSON copy of it

The memo key serialized every row and column, so each dataset the user viewed
held roughly its own size in cache key for the life of the tab. A WeakMap on
the dataset with a short settings subkey keeps the same hit rate.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012b67ubbXJVMuQrpuEksbeh

* Ban new uses of underscore's memoize

The 17 remaining call sites are allowlisted. The list may only shrink.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012b67ubbXJVMuQrpuEksbeh

* Narrow the memoize rule to module scope

A cache built inside a function, a component render, a class field or a
constructor is released with whatever owns it. Only a module scope cache lives
for the life of the tab, so the rule now flags just those.

This drops the allowlist from 17 files to 5, and each remaining entry carries
the reason its set of keys stays small.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012b67ubbXJVMuQrpuEksbeh

* Remove the memoize allowlist

Three of the five module scope caches are gone:

- getScrollBarSize takes no arguments, so it is a lazy value, not a cache.
- getScheduleExplanation returns a string that no caller compares by identity.
- cronToBuilderValue moves to the one caller that needs a stable identity.
  Schedule takes its value as a useMemo dependency, so StrategyForm now holds
  that memo itself. A user can write any number of cron strings, so a module
  cache for them was unbounded.

The remaining two carry an eslint-disable with the reason their set of keys
stays small.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012b67ubbXJVMuQrpuEksbeh

* Follow renderer_utils into the viz-core module

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012b67ubbXJVMuQrpuEksbeh

* Cover metabase/utils/memoize in the module scope rule

That memoize roots its WeakMap on the function it wraps, so it only releases
once that function does. At module scope the function never does, and the
nested Maps hold every argument ever passed. No call site does this today, but
the rule could not have caught one.

The rule now covers both sources, so it drops "underscore" from its name.
memoizeClass stays allowed: it keys on the instance.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Drop re-reselect from the dashboard selectors

createCachedSelector keyed getQuestionByCard and
getDashcardParameterMappingOptions on the card id, a primitive, in a
FlatObjectCache. That cache is a plain object with no eviction, so a session
kept one Question per card id it ever saw, and each Question pins the whole
Metadata snapshot it was built with.

reselect v5 defaults to weakMapMemoize, which keys on the card and the metadata
objects themselves. Entries are released when their card leaves the store or
the metadata is replaced, and identity still holds for every card in the
working set, which is what connect and useSelector compare on.

Measured over 300 cards, with the states dropped: 6.2 MB retained before,
0.0 MB after. Per entry, 21462 B before and 51 B after.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Drop re-reselect from the lockfile

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Back metabase/utils/memoize with reselect's weakMapMemoize

The hand-rolled implementation had two problems. It allocated two arrays on
every call, including cache hits: 225 ms against 20 ms for weakMapMemoize over
two million hits. And its WeakMap was keyed on the wrapped function rather than
the wrapper, so a cache built around a module-level helper outlived the wrapper
that owned it. A plain Map would have been safer.

weakMapMemoize matches on the same terms, identity for objects and value for
primitives, so no call site changes. It also weak-keys object arguments, which
the old one held strongly.

definition-cache passed weakMapMemoize as the memoize and argsMemoize options.
reselect v5 already defaults to both, so those are dropped.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Route all memoization through metabase/utils/memoize

underscore's memoize is gone from the app. Its 16 call sites now use the house
helper, which is reselect's weakMapMemoize. Three of them were in metabase-lib,
which cannot import metabase/utils, and all three took no arguments, so they
became plain lazy fields instead.

Two rules keep it that way:

- no-direct-memoize-import bans memoize from underscore and weakMapMemoize or
  lruMemoize from reselect, in any scope. Only metabase/utils/memoize may reach
  for an implementation.
- no-module-level-memoize keeps its job, now that the house helper is the only
  one left to police.

measureTextWidthSafely took a style object that was rebuilt on every render, so
identity keying would never have hit. Its third argument is now the font family
string, which is the only field any caller ever set.

useMemoizedCallback returns the callable shape rather than casting to T, since
the memoized function also carries clearCache and resultsCount.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Build memoizeClass on the house memoize

It kept a module-level WeakMap and walked a nested Map trie keyed on the
instance, the method, the argument count and then each argument, assembling
that path into an array on every call.

Keying weakMapMemoize on the instance and then the arguments gives the same
per-instance caching. One cache per decorated method replaces the shared trie,
so the method no longer needs to be part of the key, and neither does the
argument count. The file is down to 61 lines from 128.

A leak spec covers the property this rests on: an instance's results are
released with the instance.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Measure memoizeClass retention under load

2,000 instances per pass, 20 calls each, across a method with no arguments and
one taking a primitive and an object. Nothing is retained once the instances
are dropped.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Delete memoizeClass

It decorated a prototype method by rewriting the property descriptor, and put
the memoization 600 lines away from the method it applied to. Its one consumer
memoized a method whose body uses no `this`, so the same caching comes from
assigning a memoized arrow to a class field. The field is created per instance,
so the cache is released with the instance, and the memoization is now visible
at the definition.

metabase/utils/memoize is 14 lines, from 128.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Run the heap specs from their own jest config

They were a project inside jest.config.js, and a bare `jest` runs every
project, which is what CI does. So the sharded unit run picked them up, without
--expose-gc, and they failed as designed.

They now live in jest.memory.conf.js, the way the timezone specs live in their
own config. The shared base moves to jest.base.conf.js so neither file has to
duplicate it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Reduce the heap specs to one collection assertion each

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012b67ubbXJVMuQrpuEksbeh

* Follow the store mocks into test support

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012b67ubbXJVMuQrpuEksbeh

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-09 15:02:13 +10:00
Alexander Polyankin
c1805286bb Convert db.clj public functions to mu/defn with schemas (#82139) 2026-09-09 04:47:13 +00:00
Chris Truter
22b2577799 Require credentials when changing LLM provider base URLs (#81450)
A settings manager could change a connection's base URL without entering its
masked secret again. The update merged the stored secret back in, and
credential verification sent it to the new host before anything was persisted.
A network policy cannot tell a legitimate provider from a public host the
caller controls.

Changing a connection's effective base URL now requires freshly submitted
credentials for every sensitive field that would otherwise carry a secret to
the new destination. Environment-owned credentials must be repointed through
deployment configuration. Trusted provisioning such as config.yml is
unchanged.

Generic writes to the llm-providers setting run the same field validators, so
PUT /api/setting cannot bypass the connection API's checks.
2026-09-09 06:33:41 +02:00
Romeo Van Snick
2a35f6390b Let the measure and segment query hooks reach the store themselves (#82152)
Claude-Session: https://claude.ai/code/session_01HNvsg9cPuNQ6BHiDwa1bif

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-09 13:59:17 +10:00
Chris Truter
f508214657 Enforce a network policy on LLM provider base URLs (#81301)
* Enforce a network policy on LLM provider base URLs

A settings manager could point any provider's base URL at an internal
host; the model listing behind the admin Connect button then fetched
GET /v1/models from it with bare clj-http, and a 4xx body preview came
back in the error. Semi-blind SSRF.

New llm-allowed-networks setting, same shape as warehouse-allowed-networks:
external-only on Cloud, allow-all self-hosted (vLLM/Ollama on localhost
or a private network is the normal case there). Admin-only, so the
settings-manager class can't loosen it.

Every base-url registry field validates against it on write, so the
API rejects a forbidden URL before verify-credentials! makes the
request. core/request and llm.anthropic/chat-completion check again at
request time and resolve DNS through the policy resolver, which closes
the rebinding gap. The AI proxy URL is operator config and is exempt.

e2e pins MB_LLM_ALLOWED_NETWORKS=allow-all next to the warehouse pin,
since the mock LLM server is on localhost.

* Cover the embedding and Google token sinks too

Semantic search posts embeddings to llm-openai-api-base-url and to the
settings-manager-writable ee-embedding-service-base-url with bare
clj-http, so the same writer class reached the same kind of sink. Those
now run the llm-allowed-networks check and DNS resolver; the operator's
ai-service-base-url is exempt, and ee-embedding-service-base-url is
validated on write.

A Google service account key carries its own token_uri, and the
credential library posts to it to mint an access token before any
request we guard. Keys are now refused unless token_uri is one of
Google's OAuth hosts.

* Use var links in LLM policy docstrings

* Keep SSRF rejections out of the embedder breaker

* Normalize connection-time LLM policy errors

* Link LLM timeout settings in request docs

* Fix reflection warnings in LLM URL validation

* Use explicit policies for managed AI endpoints

* Address final Copilot review feedback

* Reject non-global special-purpose addresses

* Align network policy terminology

* Make external-only the LLM network policy everywhere

llm-allowed-networks is now environment-only (:internal, no setter)
and defaults to external-only on every instance. A settings manager is
who it defends against, and on Cloud a customer admin loosening it
would be reaching for our own infrastructure, so nobody sets it through
the API. Self-hosted operators with a vLLM server on a private network
opt in with MB_LLM_ALLOWED_NETWORKS; a value that is not one of the
three policies fails closed with one warning.

The rejection message says what to do: self-hosted, it names the
setting and which value fits; on Cloud, that private networks are out
of reach. The vLLM field's placeholder and help text no longer
advertise a private host as the default setup.

The managed endpoints' :allow-private is now a floor rather than a
replacement, so an explicit allow-all keeps an AI service on localhost
reachable. An embedding service URL supplied by the environment gets
the same floor, since the setter cannot vet it.

* Enforce the LLM network policy at connection time only

The request-time check resolved the host with a blocking lookup that
the connection's DNS resolver then repeated, and in the embedding path
it ran ahead of the breaker's fast-fail. Requests now run only the
I/O-free syntax check up front; the resolver decides on the addresses
the connection actually opens, and its refusal maps to the same 400
the set-time check gives. The set-time check still resolves, so a bad
URL is refused when it is entered.

One helper, llm-request-opts, replaces the three hand-copied
sequences; the refusal is thrown from one constructor. It carries
:status 400 beside :status-code because the semantic-search dead-letter
queue files errors by :status and was retrying a refused endpoint on
the fast schedule.

A base URL with a username or password is refused outright, and the
refusal names only the host, since the message reaches Metabot chat
users and the API's 400 body. A connection-time refusal is logged; it
is the only trace a host that rebinds leaves.

* Read llm-allowed-networks from the environment alone

get-value-of-type falls back to the application database, so a value
stored there before the setter went away kept working. The getter now
reads only the environment variable, and the setting carries a :doc
string so the generated environment-variable docs keep it now that it
has no setter. On Cloud the default applies; the operator's environment
is still honored, which the e2e runner's pin relies on.

* Disable redirects for LLM requests

* Normalize embedding service base URLs

* Normalize configured embedding URLs on read

* Clarify LLM network policy documentation

* Revert "Clarify LLM network policy documentation"

This reverts commit 8f2cff7c4963efba6756c117927ca56492cd49c5.

* Polish new LLM security comments

* Clarify embedding API key fallback

* Treat unallocated IPv6 space as non-global

* Grant private-network policy floors to environment-supplied URLs only

* Refuse IPv6 space IANA has not allocated

2000::/3 is the global-unicast space, but everything in it from 2d00:: up
is reserved: no ISP routes it, so a name resolving there is either a
mistake or an internal network squatting on it. `3000::1` passed as public
before.

* Pin Google's token endpoints by whole URI

The transport posts to whatever `token_uri` says, so checking only the
scheme and host let through another path, port, or query string on
accounts.google.com.

* Validate connections written as the llm-providers setting

The connection endpoints check a base URL against the network policy, but
the list they write is a setting in its own right: PUT /api/setting or a
config.yml entry saved a loopback URL straight past them. Run the field
validators in the setter too, on the connections a write changes.

* Hold the token-cache invalidation to the LLM network policy

/api/premium-features/token/refresh POSTs to ai-service-base-url with the
premium token in the URL path. That setting is admin-writable, so the
request needs the same syntax check, redirect ban and DNS policy as any
other LLM request; only an environment-supplied URL earns the private
floor.

* Check the target host when a JVM proxy is in the way

clj-http's route planner honours -Dhttps.proxyHost, so the connection is
opened to the proxy and the :dns-resolver is handed the proxy's hostname:
the target is resolved by the proxy, out of reach, and a private egress
proxy is itself refused under external-only. Treat an operator-configured
proxy as the trusted boundary it is and check the target host up front
instead.

* Judge a NAT64 address by the IPv4 it reaches

64:ff9b::/96 is globally reachable, but its low 32 bits are the IPv4 a
gateway translates to, so 64:ff9b::7f00:1 passed external-only and reached
127.0.0.1. RFC 6052 forbids the prefix carrying a non-global IPv4; nothing
on this side of the gateway enforces that.

* Validate only the connection fields a write changes

Comparing whole connections meant rotating an API key re-ran the base URL
check, so a private URL stored before the check existed made its own
connection unwritable.

* Name a network policy that would change the outcome

A refusal under a deployment endpoint's allow-private floor advised
setting MB_LLM_ALLOWED_NETWORKS=allow-private, which was already in force.
The resolver records the policy it refused under; use that for the message
and the log line.

* Refuse an unresolvable host behind a JVM proxy

Everywhere else a host that will not resolve is allowed, because the
connection then fails on its own. Behind a proxy there is no connection to
fail: the proxy resolves the target, so a split-horizon or cluster-internal
name went out with nothing checked at all.

* Keep proxy-only DNS working

* Document and test the LLM proxy enforcement boundary

* Match JVM proxy detection to Apache routing and fix tile fixture
2026-09-09 03:56:22 +00:00
Romeo Van Snick
a24ac528b3 Build reference questions from a provider and the field itself (#82142)
Claude-Session: https://claude.ai/code/session_01HNvsg9cPuNQ6BHiDwa1bif

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-09 12:51:21 +10:00
Alexander Polyankin
e0c40554da Accept a plaintext numeric example-dashboard-id in BackfillExampleDashboardIdValueWithAad (#82143) 2026-09-08 22:03:55 -04:00
Romeo Van Snick
c9c2552964 Build SDK and document questions from the store's builder (#81867)
* Build query builder questions from the store's builder

* Build document questions from the store's builder

* Build SDK and document questions from the store's builder
2026-09-09 11:46:12 +10:00
Romeo Van Snick
b57475ab38 Build query builder questions from the store's builder (#81861)
* Build query builder questions from the store's builder

* Build document questions from the store's builder
2026-09-09 11:42:14 +10:00
Romeo Van Snick
758aade121 Take metadata from the store's own doors, not the raw Metadata object (#81847)
* Build queries from a provider, not the raw Metadata object

* Build questions from cards through the metadata store

* Type the dashcard menu against the question dashcards its callers pass

* Name the guard the callers actually use

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HNvsg9cPuNQ6BHiDwa1bif

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-09 10:10:30 +10:00
Romeo Van Snick
5db23f4182 Move the test-side store mocks into test support (#81499)
* Move the test-side store mocks into test support

* Drop the moved store mocks from the side-effect registry

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HNvsg9cPuNQ6BHiDwa1bif

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-09 09:06:24 +10:00
Andrei Simionescu
0efc4c2161 BOT-1575: Show the illustrations toggle whenever illustrations are off (#81871)
The illustrations switch only appeared next to a custom icon, so turning illustrations off
and then removing the icon left no way to switch them back on. It now appears whenever
illustrations are off as well.
2026-09-08 23:40:41 +01:00
Andrei Simionescu
f4b04b6966 BOT-1573: Slackbot posts specific copy for known error codes (#81200)
Clackbot answered every failure with "Something went wrong", so a user denied by AI
permissions or an admin with a misconfigured provider had nothing to act on. Both the
streamed error part and the thrown-exception path now look the failure up by error
code and reply with copy that names the problem, and usage-limit errors pass their
server-authored message through. Anything unrecognised keeps the generic line, so raw
provider errors stay out of shared Slack channels.
2026-09-08 22:14:02 +00:00
Romeo Van Snick
83ed9b6c3d Serve the metabase-lib provider from the metadata store (#81664)
* Serve the metabase-lib provider from the metadata store

* Serve metric metadata providers from the store too

* Serve providers for metrics and for per-item database lookups

* Take a provider lookup in the transforms query helpers

* Resolve SDK query sources from the RTK cache, not the mirror

* Delete the unused isSourceEmpty helper and the dead tableId prop chain

* Canonicalise getMetadata's options so both call shapes share one Metadata

* Build v1 Questions through the metadata store (#81666)
2026-09-09 08:01:09 +10:00
Andrei Chernykh
0cd9134160 Keep closed group mapping dropdowns unmounted (#82124) 2026-09-08 14:56:21 -07:00
Sloan Sparger
d273dc56b6 [Metabot] Context window improvements (#81133) 2026-09-08 21:49:40 +00:00
Sloan Sparger
2d6db97c9d [Metabot] Frontend convo message shape refactor (#80967) 2026-09-08 16:10:26 -05:00
Ben Grabow
3fb5cbcb74 Use explicit hierarchy for action keywords (#79464)
* Use explicit hierarchy for action keywords

The :table.row/* -> :table.row/common and :data-grid.row/* -> :data-grid.row/common
relationships lived in Clojure's global hierarchy, sharing one mutable namespace with model
keywords, hook traits and every other subsystem. Move them into metabase.actions.hierarchy,
mirroring metabase.events.hierarchy: a defonce var holding a hierarchy map, mutated through
derive!.

Four multimethods consume those parents -- action-arg-map-schema, normalize-action-arg-map,
validate-inputs! and default-mapping -- and now dispatch on the new hierarchy. Driver-level
action methods are untouched: they dispatch on [driver action] through the driver hierarchy
and only ever name leaf action keywords.

The enterprise action_v2 module derives through metabase.actions.core, which re-exports
derive!.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* State the real reason driver action dispatch is unaffected

The hierarchy docstring said driver-level action methods are safe because they only ever name leaf action
keywords. That describes an incidental convention -- `[:postgres :model.row/create]` happens to be the only
such method -- and implies a fragile dependency that does not exist.

`driver/hierarchy` is its own `(make-hierarchy)`, and `isa?` against an explicit hierarchy never consults the
global one for keyword parents, so the derives this namespace moves had no effect on `prepare-query*` before
the move either. Say that instead, and note it at `prepare-query*` too, where someone adding a driver method
will actually be reading.

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Truter <chris@metabase.com>
2026-09-08 21:06:29 +00:00
Alex Yarosh
f52a94984a docs: typos (#82079)
* typos

* Update docs/developers-guide/api-changelog.md

Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>

* fine

* Apply suggestions from code review

Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>

---------

Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>
2026-09-08 20:38:33 +00:00
Jeff Bruemmer
8170a2ecdd docs - nav updates (#82116)
nav updates
2026-09-08 20:23:55 +00:00
Ben Grabow
ca2c341522 Use explicit hierarchy for auth identity providers (#79463)
* Use explicit hierarchy for auth providers

Provider relationships (:provider/saml -> ::provider/provider, etc.) lived in Clojure's
global hierarchy, sharing one mutable namespace with model keywords, hook traits and every
other subsystem. Move them into metabase.auth-identity.hierarchy, mirroring
metabase.events.hierarchy: a defonce var holding a hierarchy map, mutated through derive!
and underive!, read through isa? and descendants.

The three provider multimethods (validate, authenticate, login!) now dispatch on that
hierarchy via methodical's :hierarchy option.

Callers outside the module go through metabase.auth-identity.core, which re-exports the four
operations; namespaces inside the module use metabase.auth-identity.hierarchy directly, since
core requires them.

derive! is whitelisted as :parallel/safe because each test registers its own distinct
:provider/test-* keyword, so the additions never overlap. The plain `derive` calls it replaces
were equally global but escaped the linter by not ending in `!`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Register test providers at namespace level, not inside parallel tests

Each of these eight ^:parallel tests registered its provider by calling derive! and
methodical/defmethod in the test body. Both mutate process-wide state, so the tests were never
actually parallel-safe -- only derive! was visible to the linter, because defmethod does not
end in `!`.

Hoist both registrations above their deftest, alongside the test providers already registered
at the top of the namespace. The deftests are now pure assertions over providers that exist by
the time the namespace has loaded, which is what ^:parallel claims. No linter exemption for
metabase.auth-identity.core/derive! is needed, so .clj-kondo/ratchets.edn is untouched.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Truter <chris@metabase.com>
2026-09-08 22:04:07 +02:00
Ben Grabow
3390d69763 Use explicit hierarchy for domain entity types (#79459)
Domain entity types derived from :DomainEntity/* in Clojure's global hierarchy, sharing one
mutable namespace with model keywords, hook traits and every other subsystem. Specs are loaded
from YAML files, so adding a spec file was writing into that shared space.

Move them into metabase.xrays.domain-entities.hierarchy, mirroring metabase.events.hierarchy:
a defonce var holding a hierarchy map, mutated through derive!. A leaf namespace rather than
an in-file var because both specs (which registers types and validates them) and core (which
ranks candidate entities by ancestor count) read it.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-08 19:36:22 +00:00
Ben Grabow
adf3bd7254 Use explicit hierarchy for user-key-value namespaces (#79460)
Key-value namespaces registered by defnamespace derived from ::registered-namespace in
Clojure's global hierarchy, sharing one mutable namespace with model keywords, hook traits and
every other subsystem. Namespaces are loaded from EDN files on disk, so an operator adding a
file was writing into that shared space.

Move them into a hierarchy var private to this namespace, mutated through derive! -- the same
shape as metabase.events.hierarchy, kept in-file because defnamespace is the only writer and
known-namespaces the only reader.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-08 19:31:31 +00:00
Chris Vandoodewaard
2357c1b843 refactor pr env cleanup into single workflow (#81817)
* refactor pr env cleanup into single workflow

* add timeout

* rename to match naming scheme
2026-09-08 19:21:42 +00:00
Ben Grabow
e85fa9c446 Fix BigQuery tests (#82106)
* Avoid wildcard scan against all similar datasets on the warehouse

* Update bigquery page fetch tests to match new adaptive page sizing API
2026-09-08 14:08:28 -05:00
Nathan Voxland
250868257e Improve LLM token usage for add_reseach_group (#79556) 2026-09-08 14:08:26 -05:00
Jakub Chodorowicz
27752c76e0 Gdgt 3198 allow setting metabase instance url when starting embedding (#82063)
allow setting metabase instance url for embedding storybook
2026-09-08 20:38:18 +02:00
Alexander Polyankin
15106f17fd Add kondo rule keeping Toucan 2 query calls in module db namespaces (#82099) 2026-09-08 14:07:41 -04:00
Alexander Polyankin
4b2cfdc623 Add MB_DISABLE_LEGACY_STARTUP_ENCRYPTION to refuse startup instead of encrypting legacy plaintext (#82097) 2026-09-08 13:38:39 -04:00
Will Fry
3439faff23 GDGT-3094: remove attachment limit tooltip (#82055) 2026-09-08 17:49:35 +01:00
Chris Truter
18f216de52 Ensure mage -test runs every test namespace (#82051)
* Fail the mage suite when a test namespace is not required

`mage -test` runs `run-all-tests`, which only sees namespaces that are already
loaded, and loading happens solely through this namespace's `:require`. A test
file missing from it never runs, and nothing says so.

`mage.quick-test-runner-test` and `mage.start-db-test` were missing that way.
They are required by `mage/test/mage/runtests.clj`, but CI runs `mage -test`,
which does not read that file, so their four tests had not been running.

* Drop the dead mage test runner and a test that cannot fail

Nothing referenced mage/test/mage/runtests.clj. It required three namespaces
directly, which made mage.quick-test-runner-test and mage.start-db-test look
wired up while `mage -test` never loaded either.

mage.start-db-test asserted a string literal, so it passed unconditionally.
Wiring it in would have added a green test covering nothing. start-db still
deserves real tests; this makes their absence visible.
2026-09-08 18:37:44 +02:00
Anderson
e354134858 Show table publisher and publish date in the Library (#81076)
* Show table publishing info in Library

* address review comments

* Refine publishing info response

* Clarify publishing audit lookup

* Fix publishing info database queries
2026-09-08 12:56:46 -03:00
Ben Grabow
34da51c654 Fix OIDC test isolation (#81987)
Use the right localhost-site-url test helper, and make sure the web server is initialized
2026-09-08 10:47:40 -05:00
github-automation-metabase
55f3c0f268 Add docs/util/data/nav.yml for master (#82078)
Add docs/util/data/nav.yml (seeded from docs.metabase.github.io)

Co-authored-by: github-actions <github-actions@github.com>
2026-09-08 11:34:19 -04:00
Sébastien
4cb99413ed Deflake the standalone cardEmbed supporting text test (#81772) 2026-09-08 15:58:27 +02:00
Ben Grabow
beb22cd1b5 Add missing fixtures init (#82007) 2026-09-08 08:44:03 -05:00
Voytek Pituła
eb7bcaa06f Add krever to team.json and github-slack-map.json (#82058)
Adds @krever to the Metabot team in .github/team.json and maps the
GitHub username to Slack ID U0BUSKQRN80 in release/github-slack-map.json.
2026-09-08 15:34:58 +02:00
github-automation-metabase
7e11715582 Update translations 2026-09-08 (#82056)
* Update translation sources 2026-09-08

* Update translations

---------

Co-authored-by: github-actions <github-actions@github.com>
2026-09-08 13:20:14 +00:00
Stas Gavrylov
c7a5060ca8 GDGT-2462: Update Switch component (#81709)
* GDGT-2462: Update Switch component

* GDGT-2462: Add Switch Loki references

* Update Loki Snapshots

* Address PR findings

---------

Co-authored-by: Metabase Automation <github-automation@metabase.com>
2026-09-08 15:42:53 +03:00
Kevin Kelleher
b7500371ca Remove duplicate in Gauge color picker (#81920)
fix duplicate in gauge color picker
2026-09-08 12:41:00 +00:00
lbrdnk
bf0186730c Add goldens triggers (#81873)
* Enable golden scopes

* Add label triggers

* Temporarily remove label input

* drop the input completely

* Re-add label
2026-09-08 14:40:41 +02:00
Timofey Kachalov
6ca683c042 Re-enable Shoppy sample app e2e (#82061) 2026-09-08 16:27:53 +04:00
Sébastien
c16a8abfa6 Fix EditableDashboard collapsing when entering edit mode (#82019)
* Fix EditableDashboard collapsing when entering edit mode without a sized parent

* Also fix EditableDashboard collapsing while sharing

* Revert "Also fix EditableDashboard collapsing while sharing"

This reverts commit be2f86c738.
2026-09-08 11:28:56 +02:00
Oleksandr Yakushev
074e967be7 [liquibase] Remove obsolete proxy package hack (#81480) 2026-09-08 11:51:55 +03:00
Bartosz Rybacki
47bc44f775 GDGT-2028 Fix wrong column headers in alerts with a remapped foreign key (#81896)
GDGT-2028: Fix wrong column headers in alerts with a remapped foreign key

The header titles and the header row were built separately and zipped by
index. Only the header row dropped the remap target column, so every title
after it shifted by one. Both now derive from `visible-columns`.
2026-09-08 10:31:41 +02:00
Andrei Simionescu
06d46ec0c2 BOT-2113: Copy metabot_permissions on every edition
An H2 dump made by an OSS build kept the metabot_permissions rows that migration 061 seeds into the fresh target instead of copying the source's own rows, because the table only appears in cmd.copy/entities when the Enterprise backend is on the classpath. The seed references the four magic groups at ids 1 to 4, which on the source are often different groups or no groups at all, so restoring that dump into Cloud either failed on a foreign key or handed Metabot permissions to whoever happened to sit at ids 3 and 4.

The table is now copied on every edition. The load side also skips rows whose group is missing from the dump, so the dumps customers have already sent restore once Cloud picks this up, and magic groups left without rows get the 061 seed re-run by magic_group_type so a restored instance matches a fresh one. Grants that landed on a real custom group cannot be told apart from a deliberate grant and stay a data fix, tracked in CLO-5999.
2026-09-08 08:30:46 +00:00
Romeo Van Snick
28a72bb30c Answer field remapping from the field, not the metadata store (#81932)
* Answer field remapping from the field, not the metadata store

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HNvsg9cPuNQ6BHiDwa1bif

* Say what a remappable field has to carry, and simplify the format type

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HNvsg9cPuNQ6BHiDwa1bif

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-08 16:19:53 +10:00
Chris Truter
b57dc3f98e Prevent mage commands hanging on orphaned output pipes (#81699)
* Start subprocesses with ProcessBuilder

Runtime.exec's three-arg form has been deprecated since JDK 18, and it forced the command and environment
through String arrays with array type hints to match. ProcessBuilder takes both as collections, so the
call site is one line and the array juggling is gone. `:env` still replaces the parent environment rather
than adding to it, which is what the docstring promises: that is `.clear` then `.putAll`, not a bare
`.putAll`. read-lines now hints the BufferedReader the namespace already imports.

* Spend one timeout budget across a command's three waits

sh* waited for the exit code, then stdout, then stderr, giving each a fresh timeout-ms. A command that
dribbled output could take three times its budget, and the exception still claimed the single figure, so
"Timed out after 200 ms." could be thrown 600 ms in. The three waits now measure against one deadline
taken when the process starts.

* Measure the backend health-check wait with nanoTime

check-backend-health! polls for five minutes against a wall-clock start, so a clock adjustment mid-poll
could cut the wait short or stretch it. The interval literal is now written as a duration too.

* Kill the command however the wait ends, not only on timeout

Three fixes around sh*'s failure handling.

The catch was for ExceptionInfo carrying :timed-out?, so any other way out left the command running: a
reader future failing on the pipe, or the calling thread being interrupted. It now catches Throwable and
asks whether the process is still alive, which is the question that actually decides whether to kill.

:or does not fire on an explicit nil, so a caller computing :timeout-ms from an env var and handing us nil
got a NullPointerException building the deadline rather than the default.

The test's 200 ms budget left the shell to fork twice and write both pids with no margin. On a loaded
runner the file could come up a line short, failing the descendant assertion for the wrong reason. A
second is still far below the bound the test asserts.

* Stop waiting on a command that orphaned our output pipe

sh* could sit for as long as an orphan lived, whatever its timeout said: 62 s against a 500 ms budget in
the case that prompted this. with-open closed the BufferedReader, and that close takes the same lock a
blocked readLine holds, so it waited for EOF -- which a process the command left holding our stdout never
delivers. Closing the pipe underneath ends the read at once, so the streams are now closed directly.

Closing the pipe means a cut-off read reports EOF rather than an error, which would make truncation
invisible. Two things guard against that. The readers get a floor of their own once the exit code is in,
so a command that finished a moment inside its budget is not robbed of the output it had already written.
And if the pipe is still held when that floor runs out, sh* says so and throws, because half a command's
output returned as though it were all of it is worse than a failure.

The orphan itself is still running afterwards. Reaching it needs process-group handling, which is a
larger change and no mage command leaks one today.

* Rewrite the comments and docstrings in mage.shell

The docstrings now state the caller's contract and nothing else: what sh* returns, what it throws, what
the timeout actually covers now that one budget spans the whole command. Rationale about how the body
works moved down to the lines it explains -- why the descendants are listed before anything is signalled,
why the deadline is read off a monotonic clock, why the pipes are closed rather than the readers.

* Tighten the comments once more, and drop the duration glosses

A second pass over the comment wording, and a correction: a DST change does not move the epoch
milliseconds `currentTimeMillis` reports, so it was never one of the things nanoTime protects the deadline
from. NTP corrections and manual clock changes are.

The `; 5 seconds` style glosses are gone throughout. Writing the constant as `(* 5 1000)` already says
which unit it is in, and the name carries the rest.

* Serialize mage shell output and fix test namespace loading

`println` writes the text and its newline separately, so the stdout and
stderr readers could land halves of two lines on one line of the terminal.
Lock on the destination, which also keeps two commands streaming at once
from splitting each other.

`mage -test` runs `run-all-tests`, which only sees namespaces that are
already loaded, and nothing required mage.bot.pr-env-test or
mage.bot.repl-eval-test, so eight tests never ran. The `(comment ...)`
block meant to hold these requires open only referenced an alias that
existed for its own benefit; kondo does not flag alias-less requires, so
it was protecting nothing.
2026-09-08 01:03:13 +02:00
github-automation-metabase
ada3520e32 Auto-update documentation for master (#81981)
Update generated docs (master)

Co-authored-by: github-actions <github-actions@github.com>
2026-09-07 17:29:15 -04:00
Timofey Kachalov
33ebc8e671 Gate data apps remote sync logic under a token feature (#81946) 2026-09-07 20:04:42 +00:00
Andrei Simionescu
f1ea6b4ad5 BOT-1918: use the AWS default credentials chain for keyless Bedrock (#80512)
A Bedrock connection with no access key pair now signs with whatever the AWS
SDK default credentials chain resolves (environment, IRSA, EKS Pod Identity,
ECS task role, EC2 instance profile), and the SDK refreshes those on its own,
so a pod that already has a role needs no long-lived keys. Explicit keys still
override the chain, and half a pair or a lone session token is rejected.

On Metabase Cloud the chain would resolve our own identity rather than the
customer's, so a keyless connection is rejected there, and a token status the
license service could not confirm counts as hosted.
2026-09-07 18:24:18 +00:00
Will Fry
0de35e5dd3 Add Will Fry to team and slack map json files (#82028)
* Add Will Fry to team and slack map json files

* use exact casing for gh handle
2026-09-07 16:48:18 +00:00
Andrei Simionescu
0ed863ba11 BOT-1590: Test that the Conversations tab stays current on a conversation page (#81870) 2026-09-07 17:25:47 +01:00
Kamil Mielnik
da793ea976 GDGT-3189 Remove CustomVisualization['id'] and use CustomVisualization['getName'] (#82033) 2026-09-07 14:54:04 +00:00
Sébastien
7e6c62a105 Use consistent colors for the MCP errors-by-type pie chart (#81943)
* Avoid success-green slices in the MCP errors-by-type pie chart

* Use createMockStructuredDatasetQuery instead of a type cast
2026-09-07 16:51:27 +02:00
Paolo Rascunà
3ba6cb1479 BOT-2111: Stop the Gemini stream when the client cancels (#81959)
The streamGenerateContent xf walked each event's parts with a plain
`reduce`, which strips the `reduced` wrapper a cancelled consumer
returns. The stop never reached the outer reduction, so every event
that followed was still translated and pushed downstream — Google's
response was read to the end, and billed, after the client had gone.
The other adapters propagate the stop; only this one dropped it.

Use u/reduce-preserving-reduced so the early stop survives, and pin it
with a test that cancels on the first text delta: before the fix the
consumer kept receiving text from later events.

Co-authored-by: Paolo Rascuna <1936871+prascuna@users.noreply.github.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-07 14:23:37 +00:00
Stas Gavrylov
3c07580bcc GDGT-3188 Remove unintended elevation styles (#81958)
* GDGT-3188 Remove unintended elevation styles

* GDGT-3188 Fix remaining elevation regressions

* GDGT-3188 Update elevation guard expectations
2026-09-07 17:22:00 +03:00
Nicola Mometto
38fcd8e057 Scope tenant endpoints and pulse recipients by tenancy (#81765)
* Only serve tenant attributes and member_count to superusers

* Scope pulse recipients and the Slack directory by tenancy

* Update cross-tenant recipient test for non-superuser scoping
2026-09-07 15:15:20 +01:00
Nicola Mometto
c573e659f6 Read-check the transform in GET /api/transform/run/:run-id (#81761) 2026-09-07 14:45:17 +01:00
YB
f6ea5b6092 Fix ENOBUFS in the Create test plan CI step (#82025)
raise git ls-files maxbuffer so the test planner stops failing with enobufs
2026-09-07 21:33:49 +08:00
Sébastien
69256a104a Apply embedding theme colors to hand-picked chart series (#81014)
* fix: apply embedding theme colors to hand-picked chart series

* Clarify and dedupe chart color-picker naming

* Apply the instance's chart colors when rendering static visualizations
2026-09-07 15:00:00 +02:00
Kamil Mielnik
ff7b417ce6 GDGT-3184 Reject ':' in custom viz plugin identifiers (#81886) 2026-09-07 12:41:11 +00:00
Stas Gavrylov
0e7e969925 GDGT-2467: Curate Paper radius tokens (#81700)
* GDGT-2467: Restrict Paper padding/radius to curated token options

* Update Loki Snapshots

* GDGT-2467: Update entity picker width e2e expectation for Paper padding

* GDGT-2467: Keep Paper default padding at 0, add no-padding option to stories

* GDGT-2467: Keep all padding options in Paper stories, only curate radius

* Update Loki Snapshots

* GDGT-2467: Type PAPER_RADIUS_OPTIONS against RadiusScaleKey

---------

Co-authored-by: Metabase Automation <github-automation@metabase.com>
2026-09-07 15:07:41 +03:00
Nicola Mometto
1dfc2d1c90 Fix transform self cycle (#81676)
* Resolve the under-test transform's target live in cycle detection

* Split the change-target overwrite e2e test into the two cases it conflated

Retargeting onto the transform's own source table is a self-cycle (400), not a
target conflict, so exercise the conflict case with a table produced by another
transform and add a dedicated self-cycle case.

* Make the transform under test win target collisions in get-transform-cycle

* remove live-target-table-id
2026-09-07 09:49:13 +01:00
Nicola Mometto
78d3c10b00 perf: optimize fix-sql-params regex (#81893)
* Make fix-sql-params regex linear-time

The tag-body class excluded only `}`, so `\s*[^}]+\s*` had three overlapping
quantifiers and backtracked cubically on an unclosed `{ {` followed by
whitespace. Exclude `{` from the body and drop the redundant \s* so every
position matches in linear time.

* Use :standardsql dialect in test so no kondo ignore is needed
2026-09-07 09:48:49 +01:00
Kamil Mielnik
bbef4a31a2 GDGT-3185 Replace document viz settings instead of merging them (#81889) 2026-09-07 08:18:02 +00:00
Stas Gavrylov
ace7aa5dd0 Add Loki and Figma Desktop workflows to component skill (#81763) 2026-09-07 10:44:28 +03:00
Fraser
f531a39d14 Enforce the shared tiers for visualizations (#81990) 2026-09-07 16:59:45 +12:00
Romeo Van Snick
dbc843ce23 Derive each store root's state type from the slices it composes (#81481)
* Derive each store root's state type from the slices it composes

* Type the mirror's own selectors against the slice they read
2026-09-07 09:44:55 +10:00
metamben
e6216478c6 Introduce billing-project for BigQuery (#79489)
* Introduce billing-project for BigQuery

Closes QUE2-539
2026-09-05 22:29:39 +00:00
technomancy
42a8e9f77d Update create-transform-with-tags-test to be more specific about schema. (#81986)
Previously it was just picking a table at random and using that row,
but in redshift it turns out that table's row doesn't actually exist,
causing a bunch of failures. Let's look specifically at the people
test-data table instead of just using whatever.
2026-09-05 00:06:32 +00:00
Ben Grabow
6fdf223404 bgrabow/redshift-describe-database-probe (#81980)
* Probe for cause of describe database flakes

* Fix brittle test

* Delete obsolete probe test suite
2026-09-04 16:54:56 -05:00
Andrei Simionescu
70a3d8cb4a BOT-2110: sweep orphaned search index tables when a reindex starts (#81944)
Sweep orphaned search_index__* tables at the start of reindex! instead of at activation, so a rebuild
that dies partway is cleaned up by the next run instead of leaving its table behind. The sweep is best
effort and never aborts the reindex.
2026-09-04 21:52:11 +00:00
Fraser
8e3b90894e Move the drill layer from visualizations into querying (#79500) 2026-09-05 09:06:32 +12:00
Ben Grabow
a8db179087 Fix "The result set is closed" flaky reuse of connections (#80033) 2026-09-04 13:31:12 -05:00
Phoomparin Mano
ee9cd6a54e Fix getComputedStyle crash in data app sandbox (#81739)
* fix iframe parent distortion

Fix getComputedStyle calls in data app sandbox

fix frame parent distortion

* update distortion tests
2026-09-04 18:10:38 +00:00
Phoomparin Mano
96c905f643 Allow multiple useDataAppLocation consumers without crashing data apps (#81854)
Fix multiple data app location consumers
2026-09-05 01:06:40 +07:00
Riley Thompson
2a92db58c0 Don't merge db details if db engine was changed (#81968)
dont merge detials if new engine selected
2026-09-04 13:50:23 -04:00
Riley Thompson
cbfbeb3ce1 Fix implicit date conversion in postgres convertTimezone (#81912)
* add test

* Cast DATE inputs before timezone conversion on Oracle and SQL Server

`convertTimezone` on a `DATE` column failed on both drivers:

- Oracle: `FROM_TZ` only accepts a `TIMESTAMP` and refuses to implicitly
  promote a `DATE` (ORA-00932: inconsistent datatypes: expected TIMESTAMP got
  DATE). Cast date-typed inputs to `TIMESTAMP` first.
- SQL Server: `AT TIME ZONE` rejects a `date` argument (error 8116: Argument
  data type date is invalid for argument 1 of AT TIME ZONE function). Cast
  date-typed inputs to `datetime2` first.

Fixes `convert-timezone-date-column-test` for both drivers (#27186).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RYP35BadwdEL9LAC9zHWQZ

* Derive expected DATE column values from the results timezone in test

`convert-timezone-date-column-test` hardcoded the raw `d` column as
midnight `+07:00`, which is only right for drivers that support
`:set-timezone`. SQL Server can't set a session timezone, so its results
timezone falls back to the database timezone (UTC in CI) and the column
formats as `Z`. Compute the expected values the same way `format-rows`
does so the assertion holds on every driver with `:convert-timezone`.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RYP35BadwdEL9LAC9zHWQZ

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-09-04 17:37:51 +00:00
technomancy
234568bf20 Deprecate merged-column-info in driver-api. (#81965)
We want to drop as much usage of legacy-mbql as we can. This function
is never used internally, but it's exported in the driver-api. We
should mark it as deprecated since everything else in this namespace
is deprecated, so we can remove it when the time comes.

Need to get this in today so it's included in the 64 branch.
2026-09-04 17:32:16 +00:00
technomancy
4be1fd71c9 Replace Bigquery/Snowflake dataset tracking table; track by name instead (#81655)
* Replace Bigquery/Snowflake dataset tracking table; track by name instead.

The current system of using a tracking table was based on the
assumption that all datasets would need to be tracked by access
time. In fact, datasets fall into one of two buckets, either they are
static and immutable, or they are temporary one-offs. Static datasets
change *very* rarely and do not necessitate automated tracking. If we
neglect to clean them up, we could go for decades before the
accumulated datasets pose a real problem. But non-static datasets are
used once and accumulate *very* rapidly, needing to be cleaned up.

All datasets defined using `defdataset` are considered static.

Rather than using a database to track which ones must be cleaned up,
the logic can be much simpler; use the date embedded in the dataset's
name. It turns out it's even simpler because the logic to track
datasets based on their name has existed this entire time since 2023
in `metabase.driver.sql.test-util.unique-prefix`; we just stopped
using it in Snowflake/Bigquery in favor of the overly-complex tracking
table which A) leaked like mad and B) mistakenly marking static
datasets as old and causing them to get deleted while they were still
in use.

Redshift already uses the `unique-prefix` namespace generating its
schemas, so it only needs a few minor changes to adapt to the changes
in the gc job and the `unique-prefix` functions.

This also simplifies the gc nightly task; rather than having two
separate settings, we only need one setting for how many hours old a
non-static dataset has to be before cleaning it up. Also adds a
`dry-run?` option for manual testing.

* Address review feedback.

* Update workflow yaml to refer to single :hours parameter
* Remove 256-row limit from bigquery/snowflake
* Fix snowflake dataset :name in report data
* Get rid of with-write-stmt helper.
* Accept tracked? arg for manually GCing tracked datasets.
2026-09-04 09:24:56 -07:00
Nicolò Pretto
374e182461 add react_version to sdk analytics (#81954) 2026-09-04 17:03:33 +02:00
Anderson
35c9ea0a29 Fix flaky 'admin > allows to undo every action' test (#81020)
* Keep the data model in sync when a change is undone

* Extract shared TableSection update handlers into a hook

* Undo a custom field order with a single request

* Guard the optimistic table patch by table id

* Reproduce the dropped refetch scenario in the middleware test

* Link the upstream RTK issue from the dropped-refetch middleware and test
2026-09-04 13:52:40 +00:00
Nicolò Pretto
2176f18597 feat(sdk, eajs): add support to show "all" collections in the browser components (#80881)
* Accept "all" as a collection id in the browser public API

Adds the public api plumbing

* refactor: own the ItemsTable container query inside ItemsTable

The CSS container was defined in CollectionContent.module.css, which meant it only worked on ItemsTable when it was rendered inside CollectionItemsTable. The virtual "all collections" root renders ItemsTable on its own, so the container moves onto ItemsTable.

* refactor: split the collection breadcrumbs into a view and its data

We'll need to customize what breadcrumbs will be rendered for the virtual root.
This commit extracts the view that we'll use directly.

* Clear the browsed collection while the next one loads

RTK Query keeps serving the previous collection from `data` while the next one
loads, and keeps serving it once the argument becomes skipToken. A caller that
reads `can_write` during the switch therefore offers to create content in the
collection the user just left.

`currentData` drops to undefined on any argument change, so it covers both
cases and the manual clearing from 397a3e82e04 goes away.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Gdru2a8afSo6y24e99W6hy

* Don't let the popped stack entry override the clicked breadcrumb

Clicking a collection breadcrumb from inside a question landed on the
collection the browser was mounted on, not on the one the user clicked.

Opening a question, a metric or a model pushes a navigation stack entry, so the
back button keeps working. That entry carried an onPop that reset the view to
the mounted collection. The breadcrumb handler sets the view to the clicked
crumb and then pops the stack, so onPop ran last and overwrote the click.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Gdru2a8afSo6y24e99W6hy

* Render the virtual "all collections" root in the collection browser

collectionId="all" now lists everything the current user can reach, as flat
siblings, in the same order the entity picker uses: "Our analytics", the shared
tenant collections, "Our data" and the personal collection.

useAllCollectionsItems builds those rows:

- A 403 on GET /api/collection/root means the user cannot read "Our analytics"
  itself, so the hook falls back to /api/collection/root/items, which has no
  read check on the root. The models=collection argument is required: without
  it the endpoint returns no models at all.

- "Our analytics" needs a numeric id for the table, so it gets 0. Real
  collection ids start at 1, and the caller maps 0 back to "root".

- The shared tenant collections come from the collection tree, gated on the
  use-tenants setting and on the tenants plugin. shallow=true keeps the
  backend from building a nested tree the caller never reads.

- The backend excludes the tenant collection from /api/collection/root/items,
  so the hook adds it from the current user.

CollectionBrowser renders the rows with ItemsTable directly, without the
archive column: archiving resolves its permissions from a parent collection,
and the virtual root has none.

The breadcrumbs put a static "All collections" crumb in front of
CollectionBreadcrumbsView. The personal and the namespaced collections lose
their "Our analytics" crumb, because the API reports them as its children
while the virtual root shows them as siblings.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Gdru2a8afSo6y24e99W6hy

* Key the SDK breadcrumbs by type and id

Breadcrumb ids come from separate id spaces, so a collection and a dashboard
can both be 5. On an instance where collection 5 holds dashboard 5 the trail is
[all-collections "all", collection 5, dashboard 5], and two children of the
same list share the key 5. React warns about the duplicate key in the host
application's console.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Gdru2a8afSo6y24e99W6hy

* fix sdk package type

* e2e tests

* Unit test the virtual "all collections" root

* "all collections" -> "all-collections"

Co-authored-by: Mahatthana (Kelvin) Nomsawadi <me@bboykelvin.dev>

* pageSize support

* Assert the error message, not the error container

The virtual root error tests only checked that the sdk-error-container
testid rendered. That proves an error appeared, not which one, so any
other failure rendering into the same container would keep them green.
They now look for "Failed to load collections".

Verified by mutation: changing that string in CollectionBrowser turns
all four tests red, where the testid version stayed green.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01C9MrpBnd3VEiXVvYf15Qqd

* change root id from 0 to -1 and wrap the mapping

* import already existing defer

* not.toHaveLength(0) -> toBeGreaterThan(0)

* clarify that the root is 403 in the test

* `SdkBrowseCollectionId` -> `SdkBrowserCollectionId`

* docs(sdk): list the accepted collectionId values as bullets

The CollectionBrowser `collectionId` doc described five accepted values in
one running paragraph, and adding "all" pushed it past readable. Split it
into one bullet per value, and document the entity ID string, which the
paragraph never mentioned.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UTu6CuVJnRjzS1HrmrJQT3

* else if

* rename the virtual root identifiers to all mode root

* refactor root status into union

* clarify when the root could be empty

* fix eslint

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Mahatthana (Kelvin) Nomsawadi <me@bboykelvin.dev>
2026-09-04 14:59:46 +02:00
Kamil Mielnik
af22e8cf22 GDGT-3183 Sanitize ClickObject and HoverObject (#81885) 2026-09-04 19:57:23 +07:00
Kamil Mielnik
d3cb526e49 GDGT-3186 Custom viz SDK type-check (#81891) 2026-09-04 19:38:54 +07:00
Alexander Polyankin
6be1b2761e Warn when startup encrypts legacy plaintext, and make migrations leave none behind (#81898)
* Warn when startup or a migration encrypts data that was stored unencrypted

The settings backfill reports that settings saved by an older version are being
converted, the startup sweep reports each column whose plaintext values it
encrypted, and the migrations that encrypt existing rows report how many they
touched, so an operator can see when and why data changed at rest.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Find legacy unencrypted values first and hand them to a handler before encrypting them

`legacy-unencrypted-value?` decides what counts as legacy plaintext, the startup
sweep and the encrypting migrations filter their rows with it, and
`handle-legacy-unencrypted-values!` is called with the count before anything is
encrypted, so it can turn from a warning into something else later. The
migrations keep private mirrors of both, as they do for the encrypted JSON
helpers, and read their rows eagerly: reducing a reducible query closes the
transaction's connection.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Call the handler's location parameter title

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Make the encryption migrations no-ops and move the app-db module's queries to db.clj

The Encrypt* custom migrations did the same work `encrypt-plaintext-columns!` does
on every startup, so their forward bodies are gone; the reversible ones keep their
decrypting rollback for downgrades. The handler now takes only a `table.column`
location. The encryption and setting namespaces run their Toucan 2 queries through
the new `metabase.app-db.db`, like the other modules.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Drop the decrypting rollbacks of the no-op encryption migrations

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Make a fresh install come out fully encrypted from the migrations alone

The encryption-touching data migrations that only rewrote existing rows are no-ops
now that the startup sweep covers their columns, and MigrateMetricsToV2 goes with
them. Two v58 changesets left rows a fresh install would then see as legacy: the
SQL copy of password authentication into auth_identity is retired from the
changelog and redone in Clojure with the credentials encrypted, and sample content
is created by CreateSampleContentV3 once setting.value_with_aad exists so the
example-dashboard-id row is written whole. The settings backfill skips the
encryption-check sentinel, which the encryption tooling rewrites itself. The
fresh-install test asserts that setup logs no app-db warning and that the sweep
afterwards finds nothing.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Keep the sample-content migration tests in version order

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Stop inserting the plaintext encryption-check marker on new databases

The v53 changeset that inserted it is retired from the changelog: the encryption
tooling writes the sentinel itself, whole, and reads a missing row the same way, so
the settings backfill no longer needs to special-case it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Delete the plaintext encryption-check marker instead of retiring its changeset

The v53 changeset inserted the marker on every new database, encrypted ones
included, so it never said anything about the actual state. It stays in the
changelog as a checksum-agnostic no-op, since a rollback to a release that still
has it must not re-run the insert, and a new changeset deletes the marker rows: with
no row the state is unknown and judged from the content, which the encryption
tooling already does.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Leave the v53 encryption-check changeset untouched

Its checksum is recorded on every existing database and validCheckSum: ANY does
not lift the check, so it keeps inserting the plaintext marker on a new database
and the v58 changeset deletes the row right after.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Keep the data migrations, retire the Encrypt* changesets, and never change a row's encryption state

The migrations that rewrite encrypted JSON columns are back with their original
bodies, but `encrypted-json-in` now writes a row back the way it was stored:
encrypted only when the stored value decrypts with the current key, so a
migration never turns plaintext into ciphertext or the reverse. The Encrypt*
migrations are removed outright, with a changeset retiring their changelog
records, since encrypting legacy plaintext is the startup sweep's job.
CreateSampleContentV2 is untouched again and a small migration fills its
setting's value_with_aad from the strictly decrypted value; the LLM providers
migration writes both columns too.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Address the review: decide by decrypting, stream the sweep, and write NULL rather than an empty SET

The settings backfill now treats a value as encrypted only if it decrypts, so a
plaintext value that merely looks like ciphertext is filled in instead of being
skipped forever. update-setting-values! takes the columns to set as a map, so a
value that encrypts to nothing is written as NULL as before instead of compiling to
an empty SET. The startup sweep streams each column again, warning once before the
first row it encrypts. The unused bytes predicate is gone, the string one is
private, and the two changelog-retirement changesets are one, using the changelog
table property.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Keep the SQL copy of password authentication; the startup sweep encrypts the credentials

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Give the internal user no password

It is inactive and can never log in, and a password would be copied into
auth_identity bare by the v58 SQL changesets, leaving a fresh install with one
row for the startup sweep to encrypt.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-04 08:01:14 -04:00
Fraser
21b94190f6 Move FieldPanel into the table columns settings widget (#81929) 2026-09-04 21:24:50 +12:00
Nicola Mometto
72951848e5 Use tighter schema for POST /api/notification/send (#81877)
The endpoint sends an unsaved notification, so its body has no row of its
own. Decode it with the create schema instead of the full hydrated one.
2026-09-04 09:54:48 +01:00
Fraser
364ec28c56 Move template_tags and native parameters list into parameters (#79127) 2026-09-04 15:32:45 +12:00
Fraser
63e4a7ccb2 Create the segments module (#81864) 2026-09-04 14:59:02 +12:00
Anderson
3fafdbd58a Aggregate static viz grid map cells when the mapped dimensions are not unique (#80354)
* Aggregate static viz grid map cells that share a lat/long bin

* Address review
2026-09-04 01:13:19 +00:00
Riley Thompson
df8d8a2f41 Add test for presto struct fields (#81903)
add test
2026-09-03 21:33:06 +00:00
Tyler Plude
7d07cc9ac8 [GHY-4377] Data sensitivity table scanning (#81752)
* add deterministic data_sensitivity rules classifier

* fill data_sensitivity rule table to full enum breadth

* add data_sensitivity scan step and opt-in setting

* register classify-data-sensitivity as the last analyze step

* document PUBLIC as scanned-or-reviewed for data_sensitivity; add end-to-end tests for the data_sensitivity scan

* label non-Text base types and integer/band variants in the data sensitivity classifier

* add reset-data-sensitivity! to clear classifier-written labels
2026-09-03 14:33:01 -07:00
github-automation-metabase
cf03ebca8d Auto-update documentation for master (#81921)
Update generated docs (master)

Co-authored-by: github-actions <github-actions@github.com>
2026-09-03 17:29:02 -04:00
Fraser
a00b3d7fd1 Move CollectionBadge into common/collections (#81856) 2026-09-04 09:05:48 +12:00
Fraser
c82c593a00 Move the no-results alert prompt from querying into notifications (#81853) 2026-09-04 09:05:19 +12:00
Riley Thompson
b7a39e92b3 Add test for cumulative sum of a literal number (#81806)
* Add test for cumulative sum of a literal number (#27807)

Adds a QP test verifying that a custom aggregation expression of
CumulativeSum(1) on the Products table, broken out by Created At by
month and ordered ascending, runs successfully instead of failing
with 'Field 1: Field does not exist'.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011kKRbqUbfSKNViTEzS82ss

* clean up

* undo

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-09-03 23:43:36 +03:00
Aleksandr Lesnenko
8444d69eb9 Show errors from the body of http 500 responses to users (#74693)
surface http 500 error messages to users
2026-09-03 20:37:04 +00:00
Fraser
295d7eaabb Dispatch fetchTableMetadata directly instead of through loadMetadataForTable (#81852) 2026-09-04 08:35:08 +12:00
Cam Saul
cf6a95c3b8 Metadata for plain-:field expressions should propagate Field ID and Table ID (#81286)
* Metadata for plain-`:field` expressions should propagate Field ID and Table ID

* Fix 🔧

* Add tests 🔧

* Test fix 🔧

* Test fix 🔧

* Test fixes 🔧

* Test fix 🔧
2026-09-03 20:05:10 +00:00
Tyler Plude
e315710292 [GHY-4378] Add data_sensitivity col to field (#81744)
* add nullable data_sensitivity column to Field and its user-settings mirror

* Expose data_sensitivity on PUT /api/field/:id

* add tests for data_sensitivity survives sync, reactivation, and base-type changes

* add serdes round-trip tests for data_sensitivity on Field and FieldUserSettings

* Expose data_sensitivity in Lib column metadata

* tweak wording
2026-09-03 13:01:43 -07:00
Chris Vandoodewaard
2b35888297 containerize uberjar from uberjar workflow (#81238)
* merge containerize-uberjar into uberjar workflow

* big refactor

* move things around, add inputs to fetch-artifact action

* update conditional logic

* clean up comments

cleanup whitespace

* simplify conditional logic

* guard conditional matrix

* review cleanup

* add job names

* cleanup whitespace

* update get-docker-info to resolve repo/tag inputs

* trim trailing whitespace

* fix linting

* simplify fetch-artifact, use continue-on-error instead
2026-09-03 14:37:13 -04:00
Eric Normand
0dfca163f0 GHY-2937: On-demand field values can take forever to load (#81753)
* Keep on-demand FieldValues fetches alive past request cancellation

On-demand FieldValues are fetched on the request thread, so a canceled HTTP
request threw the warehouse scan away: every retry started the scan over and
nothing ever landed in the app DB, which is what makes these requests feel
like they take forever.

Run the fetch on a detached thread instead. It runs to completion and persists
what it fetched even when the caller stops waiting, and concurrent callers for
the same field (and sandboxing/constraint context) share the one in-flight run
rather than each kicking off their own scan.

Fixes GHY-2937 (#67843)

* Log detached FieldValues fetch failures

When every caller has walked away — the case detached-fetch! exists for —
nobody derefs the promise, so a thrown exception was dropped without a trace.

* Don't wipe cached FieldValues when the warehouse scan fails

`distinct-values` catches and logs warehouse errors, then returns nil — but
`create-or-update-full-field-values!` destructured that nil into empty values
and handed it to `persist-field-values!`, which reads empty as "this field has
no values" and calls `clear-field-values-for-field!`. So any failed or
timed-out scan deleted the values the field already had cached, and a field
slow enough to hit the QP timeout could never keep a cache at all.

Distinguish the two cases: nil means the scan failed, so leave whatever is
cached alone and return ::fv-fetch-failed. A scan that genuinely comes back
empty still clears, as before. The sync path already made this distinction
(see fetch-distinct-per-field's :failed-fields).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XwGt3aocjPgJeGXTuEjujp

* Guarantee detached FieldValues fetches always complete

A registry entry left behind with its promise undelivered is worse than a
leak: every later caller for that key parks on it forever, holding a request
thread each. Two paths could do that.

The `log/warnf` added for failed fetches sits inside the catch, so an appender
that throws skips both the registry cleanup and the delivery. And the registry
entry is claimed before the work is submitted, so a rejected submit — which is
what `future`'s pool does once the JVM starts shutting down — orphans it.

Route both through `complete-fetch!`, which still drops the entry before
delivering so a woken caller starts a fresh fetch rather than re-attaching to
a finished one.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XwGt3aocjPgJeGXTuEjujp

* Sweep and cancel stalled detached FieldValues fetches

Nothing bounded the in-flight registry except every code path remembering to
clean up after itself, which is a property that keeps not holding — the
logging call has now stranded entries twice.

Give each entry a timer and its future, and sweep on the way into
detached-fetch!, which is the only moment the registry can grow. The sweep
cancels the work and *completes* the entry rather than just forgetting it:
callers park on the promise, so dropping one undelivered leaves them blocked
forever, each holding a request thread.

Only a fetch wedged outside the query processor can reach the max age — the
warehouse query is already capped by *query-timeout-ms* (20 minutes in prod).

Removing an entry is now conditional on it still being the registered one, so
a swept fetch that later finishes can't evict its own replacement.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XwGt3aocjPgJeGXTuEjujp

* Filter stalled fetches in the doseq rather than up front

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XwGt3aocjPgJeGXTuEjujp

* Cap the in-flight FieldValues registry at 100 entries

Callers park on their fetch, so the number in flight is today limited by the
request thread pool — but that is an accident of how these endpoints are
served, not something this namespace can guarantee, and it stops holding the
moment anything frees a caller early.

Past the cap, detached-fetch! stops detaching and runs the work on the calling
thread. That degrades to what happened before detaching existed — no
single-flighting, no surviving cancellation — rather than failing the request,
so a ceiling that should never be reached can't take a filter dropdown down
with it. Capacity is checked inside the swap, so the ceiling holds exactly
under concurrent claims.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XwGt3aocjPgJeGXTuEjujp

* Refuse FieldValues fetches past the in-flight cap

Reaching the cap means something is wrong — callers park on their fetch, so
the number in flight is bounded by the request thread pool unless that stops
holding. Absorb it by running the work inline and the ceiling stops being a
signal; throw a 503 and it says what it means.

Raised to 200 now that hitting it fails the request rather than degrading it:
an instance has to have raised MB_JETTY_MAXTHREADS well past its default of 50
before this is reachable at all.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XwGt3aocjPgJeGXTuEjujp

* Pin down that joining an in-flight fetch bypasses the cap

The behavior was already right — claim-fetch!'s `contains?` short-circuits
before the count check — but it reads as incidental, and a caller that only
attaches to existing work adds no entry, so the ceiling has no reason to
refuse it. Say so, and test it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XwGt3aocjPgJeGXTuEjujp

* Bound how long a caller waits on a detached FieldValues fetch

sweep-stalled-fetches! only runs when a request arrives, which is exactly what
the case it guards against prevents: callers parked on a wedged fetch are
request threads, so enough of them and nothing reaches the sweep to cancel
anything. The watchdog was only reachable while the system was healthy.

Bound the waiter's deref by the same max age. The expensive thing here is the
parked request threads rather than the registry entry, and giving those back
lets requests flow again — at which point the next one sweeps the corpse. No
scheduler needed.

fetch-max-age-ms stays a backstop rather than a request-latency bound: a value
nearer a sensible HTTP timeout would return the thread far sooner, but
answering before the values are ready changes what the endpoint promises, and
that wants deciding on its own.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XwGt3aocjPgJeGXTuEjujp

* Report scan failures from POST /api/field/:id/rescan_values

The endpoint discarded the result and always answered {:status :success}. That
was already misleading, and the cache-wipe fix made it worse: a failed scan now
leaves the existing FieldValues untouched, so there is no observable trace of
it at all — the caller is told the scan succeeded and sees the values it had
before.

Return a 500 when the scan failed. The button already toasts on error, so this
is the difference between "Scan triggered!" and a visible failure.

Unlike the table and database endpoints, this one runs the scan synchronously,
so it is in a position to know.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XwGt3aocjPgJeGXTuEjujp

* Serve warm advanced FieldValues without a detached fetch

The advanced branch wrapped the whole select-or-insert in detached-fetch!, so a
cache hit — one SELECT — paid a thread handoff and a registry sweep for a plain
read. field-id->field-values-for-current-user walks advanced fields serially,
so a sandboxed user with N filters paid N of them.

Look the row up on the calling thread and detach only on a miss, which is the
only case that scans the warehouse. This is what the full-values path at
get-or-create-full-field-values! already did.

The miss path gains one extra SELECT, since select-or-insert! still does its
own read before inserting. That is noise next to a warehouse scan, and keeping
select-or-insert! there is what makes two callers racing a genuine miss safe —
detached-fetch! only single-flights within one JVM.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XwGt3aocjPgJeGXTuEjujp

* Stop the FieldValues tests mutating JVM-global state

Two tests could only be trusted because of a test-runner setting. The cap test
stuffed 200 fake entries into the shared in-flight registry, so anything else
asking for field values during that window would have been refused with a 503;
three tests swapped distinct-values with `with-redefs`, which rebinds the root
for the whole process. Under the current runner nothing else is running, but
that is a property of `:multithread? :vars`, not of the tests — flip it and the
failures land in other people's namespaces.

Make the two knobs dynamic so the cap test can bind the ceiling to 0 and drive
everything through the public API instead: one real fetch held in flight, a new
key refused, and a joiner admitted. That tests more than the fake entries did.

Use the thread-local `mt/with-dynamic-fn-redefs` where the work stays on
threads that inherit bindings. Where it delegates to the original, capture it
with `mt/original-fn`: `patch-vars!` installs its proxy as the root for the
life of the JVM, so a bare symbol captured after some earlier test patched the
same var would have recursed into the proxy.

The rescan test keeps `with-redefs` — `mt/user-http-request` is served on a
Jetty worker that inherits nothing, so only a root rebind is visible there.
Commented so nobody "fixes" it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XwGt3aocjPgJeGXTuEjujp

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-09-03 13:16:43 -05:00
Alexander Polyankin
30e5526fa7 Cover enable-encryption against the read-only-mode guard reading settings (#81890) 2026-09-03 14:00:47 -04:00
Jeff Bruemmer
196032c3fe docs - banner cleanup (#81895)
banner cleanup
2026-09-03 13:51:49 -04:00
Alexander Polyankin
a865fd1ad7 Address the remaining db.clj review findings: metabot, sync, search and friends (#81865) 2026-09-03 11:53:29 -04:00
Andrei Simionescu
9c04238b16 BOT-2052: stop reading six Metabot settings from the environment (#81217)
The three system prompts, the two usage limit settings and the quota message no
longer read their environment variables, so they are set through the admin page,
the API or a config file like every other setting. Name, icon and
show-illustrations keep theirs, which matches the MB_APPLICATION_* whitelabel
settings that self-hosted deployments set once at startup.

Demoting a setting used to drop it from the config file template as well, because
the template generator shared the environment variable docs filter. That filter
now sits with the docs alone, which puts use-tenants back in the template.
transforms-setup-complete is derived and never read back, so it is marked
read-only and the template stops advertising a key that does nothing.
2026-09-03 14:46:53 +00:00
Timofey Kachalov
25060e4397 Fixed outdated data app sync skill/comment (#81790) 2026-09-03 18:29:31 +04:00
Kamil Mielnik
a2979abd20 GDGT-3182 Make checkRenderable optional in custom viz (#81884) 2026-09-03 21:10:50 +07:00
3095 changed files with 122257 additions and 58116 deletions

View File

@@ -149,8 +149,7 @@ ms/NonBlankString ;; Non-empty string
ms/BooleanValue ;; String "true"/"false" or boolean
ms/MaybeBooleanValue ;; BooleanValue or nil
ms/TemporalString ;; ISO-8601 date/time string (for REQUEST params only!)
ms/Map ;; Any map
ms/JSONString ;; JSON-encoded string
ms/OpaqueJSONObject ;; JSON object whose keys are not ours to declare; string-keyed
ms/PositiveNum ;; Positive number
ms/IntGreaterThanOrEqualToZero ;; 0 or positive
```

View File

@@ -31,6 +31,11 @@ Phases 1, 2, 5 and 6 are shared. The middle two differ in **order**: update mode
- **The component's intended API** — variants, sizes, and states the design system supports for this component. This defines the axes of your showcase (Phase 3).
3. Decide the mode (see **Two modes** above): search for an existing component — in `metabase/ui` or as a legacy component elsewhere. Found → **update**; nothing equivalent anywhere → **create**. Confirm with the user when unsure.
4. **Verify a Figma desktop MCP is connected before doing the work** — this skill depends on it for exact tokens in Phase 4 (and styling can't proceed accurately without it). Do a quick read against the issue's node (e.g. `get_metadata`/`get_variable_defs`); if it errors with "nothing selected" or the server isn't connected, prompt the user to enable it (Switch to Dev Mode → MCP section in the right sidebar) and select the relevant layer before continuing.
5. **Figma Desktop MCP environments only:** open the issue's exact page with [`figma-open`](./figma-open) before MCP reads. Do not run this helper when Desktop MCP is unavailable.
```bash
.claude/skills/metabase-ui-component-from-figma/figma-open "<figma-url-with-node-id>"
```
A URL with `node-id` needs no token. Bare page-name and `--list` lookups require an already-configured `FIGMA_TOKEN`.
## Phase 2 — Understand the component
@@ -240,5 +245,6 @@ Only after sign-off:
- [ ] Storybook showcase matrix built from the component's full state space (theme excluded — global toggle drives light/dark), single panel, hover/pressed forced via `storybook-addon-pseudo-states`, controls scoped per story; lint + type-check pass. (Update mode: built before styling, **component styles untouched**. Create mode: built after the component is implemented in Phase 4.)
- [ ] Figma desktop MCP available; exact tokens extracted and mapped to semantic `--mb-color-*` (NO `color-mix`, NO primitives) and to existing scale vars for radius/spacing/elevation/type — any color *or* dimensional value with no matching token/scale step flagged to the user, not baked in as a literal.
- [ ] Component styled; stylelint + eslint(+CSS modules) + type-check pass.
- [ ] Loki coverage registered: `storiesFilter` matches exact story display names; targeted `loki update` generated references; reference PNGs and Playwright screenshots visually inspected; targeted `loki test` passes; `.loki/reference/*.png` included in the commit.
- [ ] Iterated with the user until satisfied.
- [ ] Committed (when asked); call sites migrated/verified (update) or export wired (create); findings doc handed back.

View File

@@ -0,0 +1,68 @@
#!/usr/bin/env bash
# figma-open — open a Figma file (or a specific page) in the Figma desktop app
#
# Usage:
# figma-open <url-or-file-key> open the file
# figma-open <url-or-file-key> <node-id> open at a node/page id (e.g. 12-3 or 12:3)
# figma-open <url-or-file-key> --list list pages with their ids
# figma-open <url-or-file-key> "Page name" open a page by name
#
# Page lookup by name/--list needs a Figma personal access token:
# export FIGMA_TOKEN=figd_... or put it in ~/.config/figma/token
set -euo pipefail
arg="${1:-}"
[[ -n "$arg" ]] || { awk 'NR==1 && /^#!/ {next} /^#/ {sub(/^# ?/,""); print; next} {exit}' "$0"; exit 1; }
token="${FIGMA_TOKEN:-}"
[[ -f "$HOME/.config/figma/token" ]] && token="$(<"$HOME/.config/figma/token")"
# Extract the file key from any Figma URL form, or use the arg as a bare key.
key="$arg"
case "$arg" in
*figma.com*|figma://*)
key="$(printf '%s' "$arg" | sed -E 's#https?://([a-z]+\.)?figma\.com##; s#^([a-z0-9-]+\.)?figma\.com##; s#^figma://##; s#^/?(file|design|proto|board)/##; s#^/##; s#/.*##; s#\?.*##')"
;;
esac
req="${2:-}"
# If no page arg was given but the URL carries a node-id, use it.
url_node="$(printf '%s' "$arg" | sed -nE 's/.*[?&]node-id=([0-9]+([-:][0-9]+)*).*/\1/p')"
open_fig() { # $1 = optional node id
if [[ -n "${1:-}" ]]; then
open "figma://file/${key}?node-id=${1//:/-}"
else
open "figma://file/${key}"
fi
}
# No page requested — open the file (or the node the URL points at).
[[ -z "$req" ]] && { open_fig "${url_node:-}"; exit 0; }
# Explicit node id (page ids work: pages are top-level nodes).
if [[ "$req" =~ ^[0-9]+[:-][0-9]+$ ]]; then
open_fig "$req"
exit 0
fi
# Everything else needs the REST API.
[[ -n "$token" ]] || { echo "error: page lookup needs FIGMA_TOKEN (env or ~/.config/figma/token)" >&2; exit 1; }
pages_json="$(curl -sf -H "X-Figma-Token: ${token}" "https://api.figma.com/v1/files/${key}?depth=1")" \
|| { echo "error: could not fetch file ${key} (bad key, no access, or bad token)" >&2; exit 1; }
if [[ "$req" == "--list" || "$req" == "-l" ]]; then
printf '%s\n' "$pages_json" | jq -r '.document.children[] | "\(.id // .guid)\t\(.name)"' | column -t -s $'\t'
exit 0
fi
# Page by name (case-insensitive, exact match first, then substring).
node_id="$(printf '%s' "$pages_json" | jq -r --arg n "$req" \
'first(.document.children[] | select((.name // "") | ascii_downcase == ($n | ascii_downcase)) | .id // .guid) //
first(.document.children[] | select((.name // "") | ascii_downcase | contains($n | ascii_downcase)) | .id // .guid) //
empty')"
[[ -n "$node_id" ]] || { echo "error: no page matching \"$req\" in file ${key}" >&2; exit 1; }
open_fig "$node_id"

View File

@@ -17,10 +17,10 @@ Adherence to `typescript-write` is the **highest-priority** review dimension: ra
Review in this priority order:
1. **Violations of [`typescript-write`](../typescript-write/SKILL.md) provisions** — no-`any`, type tightening, type modeling, null/undefined handling, naming, structure, comments. Highest priority; block on the no-`any` rule.
1. **Violations of [`typescript-write`](../typescript-write/SKILL.md) provisions** — no-`any`, type tightening, type modeling, function signatures, null/undefined handling, naming, structure, comments. Highest priority; block on the no-`any` rule. Apply conditional guidance in context: explain the unsupported type guarantee or concrete readability problem, rather than treating every preference as a blanket ban.
2. Compliance with `frontend/CLAUDE.md`.
3. Readability and maintainability.
4. Appropriate test coverage.
4. Appropriate test coverage. For internal typed callers, avoid requesting tests solely for inputs the type system excludes. External API data, deserialised values, storage and JavaScript callers can violate annotations: test runtime validation and nontrivial assumptions at those boundaries. Types do not replace behavioural, security or data-integrity tests.
## Blind spots — act as the missing reviewer
@@ -31,3 +31,4 @@ These rarely surface in team reviews, so this skill should raise them. They are
- **Security.** Evaluate potential security issues in new code.
- **Bundle size.** Flag new large dependencies, default imports from icon or util libs, and heavy modules imported at route-load time.
- **Analytics.** User-facing flows should emit tracking events. If a PR adds a new flow (button, modal, navigation) without a tracking event, ask whether one is expected.
- **Public API surface** (embedding SDK). Consumers should be able to use public signatures and name types they need to import. Export those types deliberately and document public behaviour, including `@deprecated` for deprecated APIs; a referenced structural type does not automatically need its own named export.

View File

@@ -20,20 +20,27 @@ description: Write TypeScript and JavaScript code following Metabase coding stan
- **Avoid type casts and loose `unknown`** — fix the signature instead.
- **If a function only needs one field of a wide object, accept that field** — not the wide object. The cast often disappears once the signature is right.
- **Reach for `Partial<T>`, `Pick<T, K>`, `Record<K, V>`, and generics** before reaching for a cast.
- **Match dictionary keys to the data.** Use a finite key union when the keys are known. Open dictionaries can use an index signature, `Record<string, T>`, or `Map`; index signatures still constrain values, and changing their spelling to `Record<string, T>` does not make missing-key access safe.
- **Prefer making props/components generic** (`<T>`) when a value flows through unchanged and the caller knows the type.
- **Prefer `unknown` over loose typing** and narrow before use — an `unknown` value forces a guard at the point of use.
- **`satisfies` for object literals** that must conform without widening (config objects, lookup maps, discriminated literals) — better than `: T` (widens) or `as T` (unsafe).
- **Avoid non-null assertions (`!`)**. Prefer a guard, early return, or `?.`. Use `!` only when non-nullness is provably true and localized, with a comment.
- **Guard indexed lookups that may miss.** Arrays and dictionaries can return `undefined` even when the inferred type omits it. Use an iteration form that preserves the key/value relationship, and only assert `keyof` when the runtime keys are known to belong to the declared type.
- **No redundant runtime coercion** — don't wrap already-typed values in `Number()` / `String()` / `Boolean()`.
- **Type guards belong in `frontend/src/metabase-types/guards/`**. Do not redefine them locally.
- **A cast you can't avoid needs a real justification comment.** The `metabase/no-unjustified-type-casts` rule accepts any preceding comment — state the actual reason the cast is safe. NEVER write the legacy `// Unjustified type cast. FIXME` placeholder; it exists only on casts that predated the rule, and copying it sneaks an unjustified cast past the linter. If you can't articulate why the cast is correct, the cast is wrong — fix the types.
- **Keep unavoidable assertions local.** Isolate a repeated or complex assertion behind a helper when that makes its invariant easier to enforce. Test nontrivial runtime assumptions that justify it; a trivial assertion does not automatically need a new helper or test. Do not weaken a public signature just to silence implementation errors.
## Type modeling
- **Reuse existing types; don't re-declare them.** Use canonical IDs and domain entity types from `metabase-types/api` (`FieldId`, `TableId`, `ConcreteTableId`, `SchemaName`, …) and key data structures by them (`new Map<ConcreteTableId, …>()`). Don't duplicate generated/API types — compose or derive (`Pick`, `Omit`, indexed access `SomeType["field"]`, `ReturnType`).
- **Use generics to allow TypeScript to infer correct types** when creating functions and components that need to be reusable and type-safe. Don't hesitate to introduce complex generics if they allow to derive types automatically instead of manual narrowing.
- **Generics must make promises the implementation can keep.** A caller-selected `get<T>(): T` must not disguise an unchecked assertion about external data. Return `unknown` and validate, or accept a validator that establishes `T`. A factory such as `function empty<T>(): T[] { return []; }` is valid; judge the implementation, not how often `T` appears in the signature.
- **Prefer an honest type over false precision.** Use generics when they express a real relationship. If a complex type cannot model the behaviour accurately, choose a simpler type or `unknown` with narrowing instead of asserting an unsupported guarantee.
- **Model the actual data contract; keep types narrow.** Optional `field?: T` for a key that may be absent, `field: T | undefined` only when the key is always present but the value may be undefined, `| null` for explicit API nulls. Prefer domain unions over broad `string` / `number` / loose `Record`.
- **Refer to API implementation** when defining or refining types to ensure they match the actual data structure. When considering a type cast, first consider if the type should be refined to match the actual data structure.
- **Optionality must reflect absence.** Keep required fields required and optional fields optional. Normalise input when the application has a meaningful default, not merely to remove a type error. Preserve the actual wire shape in API types.
- **Represent related absence together when modelling internal state.** If several fields exist or disappear together, consider a nullable containing object or a discriminated union. Do not reshape a raw API declaration unless the data actually has that shape.
- **Prefer explicit special states when designing a format.** Use nullability or named union variants when sentinel values such as `-1` hide meaning. Preserve established protocol sentinel values unless the behaviour is deliberately changed, or normalise them at an explicit boundary.
- **Discriminated unions for variant state, with exhaustive checks.** Model "one of N shapes" as a union with a literal discriminant rather than a bag of optional fields, and exhaust it with ts-pattern's `.exhaustive()` so adding a variant becomes a compile error:
```ts
import { match } from "ts-pattern";
@@ -45,20 +52,30 @@ description: Write TypeScript and JavaScript code following Metabase coding stan
.exhaustive(); // Compile-time guarantee all cases handled
```
- **Derive union types from constants** (`as const` + `typeof`/`keyof`) so the type and the values can't drift.
- **`readonly` / immutability where mutation isn't intended** — component props, shared constants, exported config. Prefer `readonly T[]` / `ReadonlyArray<T>` for inputs you don't mutate.
- **`readonly` / immutability where mutation isn't intended** — component props, shared constants, exported config, and unmutated parameters. Prefer `readonly T[]` / `ReadonlyArray<T>` for inputs you don't mutate. Functions that mutate caller-owned data should make that behaviour explicit.
- **Construct complete, well-typed objects where practical.** Prefer an object expression when it avoids partially initialised state or assertions. Incremental construction is fine when it is clearer and maintains the type's invariants.
- **Treat `metabase-lib` opaque types as black boxes.** Use `metabase-lib` functions to work with types such as `Lib.Query`, rather than casting into their internal representation.
- **Type async and error states explicitly** (a discriminated union or the data-layer's typed result) — never leave loading/error/empty implicit.
## Function signatures
- **Make public contracts explicit where it improves stability and clarity.** Annotate parameters and return types at shared boundaries when useful; let local values infer. Use `satisfies` or an annotation when a declaration needs an explicit shape check.
- **Accept the inputs the operation supports and return the most precise honest result.** Narrow avoidable uncertainty inside the function, but preserve meaningful nullability and union variants in its return type.
- **Use named options when positional arguments are easy to confuse.** Consecutive arguments with the same type are a useful warning sign, not an automatic requirement to rewrite a clear API.
- **Use `async`/`await` when it clarifies control flow or error handling.** Returning an existing promise directly is also valid; a Promise return type alone does not require adding `async`.
## Null and undefined
- **Narrow at the source**. If a value is optional only in a corner case, don't thread `undefined` through every layer — guard at the producer.
- **Sensible defaults for optional values**. Use `?.` and `??` at the consumer.
- **Lists should be filtered** before being used in a map or other iteration.
- **Narrow nullable list elements when the operation requires present values.** Filter with a type guard when missing entries should be discarded; preserve them when their absence or position carries meaning.
- **Avoid non-strict null comparisons** (`X != null`) when `X` can never be `null` — use a strict check or narrow the type. Use `checkNotNull` where necessary.
- **Check actual nullability against API implementation**. Find the API endpoint implementation and check if the field can actually be null.
## Naming
- **Names describe the entity, not the mechanism**. A name must reflect what the value holds.
- **Use domain vocabulary and include units where useful** (`timeoutMs`, `widthPx`, `temperatureC`). Prefer a more specific name when `Info`, `Data`, or `Entity` obscures the meaning, while keeping established terminology when it is clear.
- **Align sibling concepts**: keep verb conventions consistent across a related API.
- **No names that encode implementation history** rather than current meaning. Suffixes like `Base`, `New`, `Old`, `Initial` need a real semantic distinction, otherwise drop them.
- **Avoid cryptic identifiers** (`v`, `n`, `$n`) for domain values; short names are fine only in tiny conventional contexts (loop index `i`, coordinates `x`/`y`, generic params `T`/`K`/`V`).

View File

@@ -72,6 +72,7 @@
:metabase/namespace-name {:level :warning}
:metabase/no-jsqlparser-imports {:level :warning}
:metabase/prefer-with-dynamic-fn-redefs {:level :warning}
:metabase/t2-query-namespace {:level :warning}
:metabase/toucan-model-ns {:level :warning}
:metabase/require-shape-checker {:level :warning}
:metabase/tests-must-live-in-known-modules {:level :warning}
@@ -180,7 +181,7 @@
clojure.core.async/take!
clojure.core.async/to-chan!
clojure.core.async/to-chan!!
;; only writes .clj-kondo/ratchets.edn, from a :once fixture that finishes before the parallel tests
;; writes the ratchet files from a :once fixture that finishes before the parallel tests
dev.kondo-ratchet/fix!
metabase.ai-tracing.core/record!
metabase.api.macros.defendpoint.tools-manifest/assert-optional-fields-nullable!
@@ -469,6 +470,7 @@
metabase-enterprise.serialization.metadata-file-import.wire-format-test/with-tiny-fixture! clojure.core/fn
metabase-enterprise.serialization.test-util/with-dbs clojure.core/fn
metabase-enterprise.serialization.test-util/with-random-dump-dir clojure.core/let
metabase-enterprise.tenants.pulse-api-test/with-tenant-pulse-fixture! clojure.core/fn
metabase-enterprise.workspaces.test-util/with-resources! clojure.core/let
metabase-enterprise.workspaces.test-util/with-workspaces! clojure.core/let
metabase.actions.test-util/with-actions clojure.core/let
@@ -632,6 +634,45 @@
metabase.lib.schema.mbql-clause/define-tuple-mbql-clause hooks.metabase.lib.schema.mbql-clause/define-mbql-clause
methodical.core/defmethod hooks.metabase.toucan.table-name/lint-defmethod
methodical.macros/defmethod hooks.metabase.toucan.table-name/lint-defmethod
metabase.app-db.core/query hooks.metabase.toucan.db-ns/lint-query-call
metabase.app-db.core/streaming-reducible-query hooks.metabase.toucan.db-ns/lint-query-call
metabase.app-db.core/select-or-insert! hooks.metabase.toucan.db-ns/lint-query-call
metabase.app-db.core/update-or-insert! hooks.metabase.toucan.db-ns/lint-query-call
metabase.app-db.core/current-timestamp-string hooks.metabase.toucan.db-ns/lint-query-call
metabase.app-db.query/query hooks.metabase.toucan.db-ns/lint-query-call
metabase.app-db.query/streaming-reducible-query hooks.metabase.toucan.db-ns/lint-query-call
metabase.app-db.query/select-or-insert! hooks.metabase.toucan.db-ns/lint-query-call
metabase.app-db.query/update-or-insert! hooks.metabase.toucan.db-ns/lint-query-call
metabase.app-db.query/current-timestamp-string hooks.metabase.toucan.db-ns/lint-query-call
toucan2.core/count hooks.metabase.toucan.db-ns/lint-query-call
toucan2.core/delete! hooks.metabase.toucan.db-ns/lint-query-call
toucan2.core/exists? hooks.metabase.toucan.db-ns/lint-query-call
toucan2.core/insert! hooks.metabase.toucan.db-ns/lint-query-call
toucan2.core/insert-returning-instance! hooks.metabase.toucan.db-ns/lint-query-call
toucan2.core/insert-returning-instances! hooks.metabase.toucan.db-ns/lint-query-call
toucan2.core/insert-returning-pk! hooks.metabase.toucan.db-ns/lint-query-call
toucan2.core/insert-returning-pks! hooks.metabase.toucan.db-ns/lint-query-call
toucan2.core/query hooks.metabase.toucan.db-ns/lint-query-call
toucan2.core/query-one hooks.metabase.toucan.db-ns/lint-query-call
toucan2.core/reducible-query hooks.metabase.toucan.db-ns/lint-query-call
toucan2.core/reducible-select hooks.metabase.toucan.db-ns/lint-query-call
toucan2.core/reducible-update hooks.metabase.toucan.db-ns/lint-query-call
toucan2.core/reducible-update-returning-pks hooks.metabase.toucan.db-ns/lint-query-call
toucan2.core/save! hooks.metabase.toucan.db-ns/lint-query-call
toucan2.core/select hooks.metabase.toucan.db-ns/lint-query-call
toucan2.core/select-fn->fn hooks.metabase.toucan.db-ns/lint-query-call
toucan2.core/select-fn->pk hooks.metabase.toucan.db-ns/lint-query-call
toucan2.core/select-fn-reducible hooks.metabase.toucan.db-ns/lint-query-call
toucan2.core/select-fn-set hooks.metabase.toucan.db-ns/lint-query-call
toucan2.core/select-fn-vec hooks.metabase.toucan.db-ns/lint-query-call
toucan2.core/select-one hooks.metabase.toucan.db-ns/lint-query-call
toucan2.core/select-one-fn hooks.metabase.toucan.db-ns/lint-query-call
toucan2.core/select-one-pk hooks.metabase.toucan.db-ns/lint-query-call
toucan2.core/select-pk->fn hooks.metabase.toucan.db-ns/lint-query-call
toucan2.core/select-pks-set hooks.metabase.toucan.db-ns/lint-query-call
toucan2.core/select-pks-vec hooks.metabase.toucan.db-ns/lint-query-call
toucan2.core/update! hooks.metabase.toucan.db-ns/lint-query-call
toucan2.core/update-returning-pks! hooks.metabase.toucan.db-ns/lint-query-call
metabase.lib.test-util.macros/$ids hooks.metabase.test.data/$ids
metabase.lib.test-util.macros/mbql-query hooks.metabase.test.data/mbql-query
metabase.lib.test-util.macros/mbql-5-query hooks.metabase.test.data/mbql-query
@@ -803,7 +844,7 @@
:name test-namespaces}
{:pattern "^(?!.*-test(?:\\..*)?$)(?!.*test-util)(?!^metabase\\.test.*).*$"
:name source-namespaces}
{:pattern "(?:(?:^metabase\\.cmd.*)|(?:^build.*)|(?:^metabuild-common.*)|(?:^release.*)|(?:^i18n.*)|(?:^lint-migrations-file$))|(?:^mage.*)|(?:^load-namespaces.*)"
{:pattern "(?:(?:^metabase\\.cmd.*)|(?:^build.*)|(?:^metabuild-common.*)|(?:^release.*)|(?:^i18n.*)|(?:^lint-migrations-file$))|(?:^mage.*)|(?:^load-namespaces.*)|(?:^dev\\..*)"
:name printable-namespaces}
{:pattern "^metabase\\.(?:(?:lib)|(?:legacy-mbql))\\..*"
:name metabase-lib}
@@ -840,7 +881,13 @@
{:pattern "^metabase(?:((?:-enterprise\\.sandbox)?\\.query-processor)|(?:\\.driver\\.sql(?!server)(?!ite))).*-test$"
:name general-qp-and-driver-test-namespaces}
{:pattern "(?:^mage.*)"
:name mage-namespaces}]
:name mage-namespaces}
;;
;; the app-db module is the application database layer itself (connections, migrations, encryption), so it gets to
;; run Toucan 2 queries anywhere.
;;
{:pattern "^metabase\\.app-db\\..*"
:name app-db-namespaces}]
:config-in-ns
{test-namespaces
@@ -850,6 +897,7 @@
:private-call {:level :off}
:metabase/defsetting-must-specify-export {:level :off}
:metabase/defsetting-namespace {:level :off}
:metabase/t2-query-namespace {:level :off}
:metabase/test-helpers-use-non-thread-safe-functions {:level :warning} ; more config in `.clj-kondo/config/test-helpers/config.edn`
:discouraged-var
@@ -857,6 +905,9 @@
metabase.lib.convert/->legacy-MBQL {:level :off}
metabase.lib.core/->legacy-MBQL {:level :off}}}}
app-db-namespaces
{:linters {:metabase/t2-query-namespace {:level :off}}}
;; .cljc tests run in both Clojure and ClojureScript — `with-dynamic-fn-redefs` is JVM-only,
;; so leave these alone.
metabase.lib.aggregation-test

View File

@@ -109,6 +109,7 @@
query-processor
search
settings
users
util
warehouses}
:model-exports #{:model/Action}
@@ -146,6 +147,7 @@
collections
config
events
lib
models
permissions
util}
@@ -158,7 +160,10 @@
agent-api
{:team "Metabot"
:api #{metabase.agent-api.api metabase.agent-api.init metabase.agent-api.usage}
:api #{metabase.agent-api.api
metabase.agent-api.init
metabase.agent-api.schema
metabase.agent-api.usage}
:uses #{ai-tracing
api
auth-identity
@@ -222,6 +227,7 @@
driver
eid-translation
internal-stats
lib
lib-be
models
permissions
@@ -291,6 +297,7 @@
:uses #{api-scope
config
events
lib
models
settings
util}
@@ -325,6 +332,7 @@
:uses #{api
app-db
events
lib
models
permissions
users
@@ -363,17 +371,17 @@
{:team "UX West"
:api #{metabase.appearance.core
metabase.appearance.init}
:uses #{settings util}
:uses #{lib settings util}
:model-imports #{:model/Dashboard}}
audit-app
{:team "UX West"
:api #{metabase.audit-app.core
metabase.audit-app.init}
:api #{metabase.audit-app.core metabase.audit-app.init}
:friends #{enterprise/memoize-monitor}
:uses #{api
app-db
events
lib
models
premium-features
settings
@@ -408,14 +416,18 @@
auth-identity
{:team "UX West"
:api #{metabase.auth-identity.core
metabase.auth-identity.init}
metabase.auth-identity.db
metabase.auth-identity.init
metabase.auth-identity.schema}
:uses #{channel
events
lib
login-history
models
notification
premium-features
request
users
util}
:model-exports #{:model/AuthIdentity}
:model-imports #{:model/Session :model/User}}
@@ -439,6 +451,7 @@
:uses #{api
app-db
collections
lib
permissions
queries
util}
@@ -465,9 +478,7 @@
cache
{:team "UX West"
:api #{metabase.cache.api
metabase.cache.core
metabase.cache.init}
:api #{metabase.cache.api metabase.cache.core metabase.cache.init}
:uses #{api
app-db
config
@@ -602,6 +613,7 @@
audit-app
config
events
lib
models
permissions
premium-features
@@ -634,6 +646,7 @@
documents
eid-translation
events
lib
lib-be
models
notification
@@ -666,6 +679,7 @@
api
channel
events
lib
models
request
users
@@ -688,16 +702,16 @@
content-translation
{:team "UX West"
:api #{metabase.content-translation.models}
:api #{metabase.content-translation.models metabase.content-translation.schema}
:uses #{premium-features
util}
:model-exports #{:model/ContentTranslation}}
content-verification
{:team "UX West"
:api #{metabase.content-verification.core
metabase.content-verification.init}
:api #{metabase.content-verification.core metabase.content-verification.init}
:uses #{app-db
lib
models
util}
:model-exports #{:model/ModerationReview}
@@ -810,6 +824,7 @@
database-routing
driver
events
lib
request
sync
util
@@ -838,6 +853,7 @@
batch-processing
collections
events
lib
lib-be
models
parameters
@@ -962,6 +978,7 @@
{:team "Embedding"
:api #{metabase.embedding.init
metabase.embedding.jwt
metabase.embedding.schema
metabase.embedding.settings
metabase.embedding.util
metabase.embedding.validation}
@@ -1005,7 +1022,7 @@
{:team "Metabot"
:api #{metabase.entity-retrieval.core
metabase.entity-retrieval.mirror}
:uses #{premium-features}
:uses #{premium-features util}
:model-imports #{:model/OsiAiContext}}
events
@@ -1029,9 +1046,7 @@
explorations
{:team "UX West"
:api #{metabase.explorations.api
metabase.explorations.core
metabase.explorations.init}
:api #{metabase.explorations.api metabase.explorations.core metabase.explorations.init}
:uses #{analytics
analytics-interface
api
@@ -1104,6 +1119,7 @@
:api #{metabase.glossary.api}
:uses #{api
events
lib
models
util}
:model-exports #{:model/Glossary}
@@ -1128,8 +1144,7 @@
indexed-entities
{:team "UX West"
:api #{metabase.indexed-entities.api
metabase.indexed-entities.init}
:api #{metabase.indexed-entities.api metabase.indexed-entities.init}
:uses #{analytics
api
driver
@@ -1151,6 +1166,7 @@
metabase.indexes.reconcile
metabase.indexes.schema}
:uses #{driver
lib
models
util}
:model-exports #{:model/TableIndex}}
@@ -1160,6 +1176,7 @@
:api #{metabase.indexes-rest.api}
:uses #{api
indexes
lib
transforms-base
util}
:model-imports #{:model/Database
@@ -1179,8 +1196,7 @@
internal-stats
{:team "UX West"
:api #{metabase.internal-stats.core}
:uses #{app-db
models}
:uses #{app-db models util}
:model-imports #{:model/AiUsageLog
:model/Card
:model/Dashboard
@@ -1317,13 +1333,12 @@
login-history
{:team "UX West"
:api #{metabase.login-history.api
metabase.login-history.core
metabase.login-history.init}
:api #{metabase.login-history.api metabase.login-history.core metabase.login-history.init}
:uses #{analytics
api
app-db
channel
lib
request
settings
util}
@@ -1335,6 +1350,7 @@
metabase.mcp.callback-api
metabase.mcp.core
metabase.mcp.init
metabase.mcp.schema
metabase.mcp.usage}
:model-exports #{:model/McpSessionLog
:model/McpToolCallLog}
@@ -1343,6 +1359,7 @@
api
app-db
config
lib
llm
metabot
oauth-server
@@ -1356,7 +1373,7 @@
measures
{:team "Graphy"
:api #{metabase.measures.api}
:api #{metabase.measures.api metabase.measures.schema}
:uses #{api
events
lib
@@ -1377,11 +1394,11 @@
metabase.metabot.core
metabase.metabot.init
metabase.metabot.metadata-perms
metabase.metabot.schema
metabase.metabot.self
metabase.metabot.settings
metabase.metabot.tools.entity-details}
:friends #{agent-api
enterprise/metabot
enterprise/metabot-analytics
slackbot}
:uses #{activity-feed
@@ -1429,6 +1446,7 @@
transforms
transforms-base
util
warehouse-schema
warehouses}
:model-exports #{:model/AiUsageLog
:model/Metabot
@@ -1458,6 +1476,7 @@
metabase.metrics.core}
:uses #{api
app-db
collections
events
lib
lib-be
@@ -1465,7 +1484,6 @@
models
parameters
permissions
queries
query-processor
request
server
@@ -1474,9 +1492,7 @@
model-persistence
{:team "Graphy"
:api #{metabase.model-persistence.api
metabase.model-persistence.core
metabase.model-persistence.init}
:api #{metabase.model-persistence.api metabase.model-persistence.core metabase.model-persistence.init}
:uses #{api
app-db
driver
@@ -1493,13 +1509,15 @@
task
task-history
tracing
util}
util
warehouses}
:model-exports #{:model/PersistedInfo}
:model-imports #{:model/Card :model/Database}}
models
{:team "DevEx"
:api #{metabase.models.humanization
:api #{metabase.models.db
metabase.models.humanization
metabase.models.init
metabase.models.interface
metabase.models.resolution
@@ -1542,8 +1560,7 @@
native-query-snippets
{:team "Graphy"
:api #{metabase.native-query-snippets.api
metabase.native-query-snippets.core}
:api #{metabase.native-query-snippets.api metabase.native-query-snippets.core}
:uses #{api
collections
events
@@ -1580,6 +1597,7 @@
parameters
permissions
premium-features
queries
query-processor
request
session
@@ -1612,6 +1630,7 @@
api-scope
appearance
events
lib
mcp
models
request
@@ -1682,6 +1701,7 @@
remote-sync
settings
tracing
users
util
enterprise/advanced-permissions}
:model-exports #{:model/ApplicationPermissionsRevision
@@ -1776,9 +1796,7 @@
:api #{metabase.public-sharing.core
metabase.public-sharing.init
metabase.public-sharing.validation}
:uses #{api
settings
util}
:uses #{api settings util}
:model-imports #{:model/Action
:model/Card
:model/Dashboard
@@ -1811,9 +1829,7 @@
pulse
{:team "Gadget"
:api #{metabase.pulse.api
metabase.pulse.core
metabase.pulse.init}
:api #{metabase.pulse.api metabase.pulse.core metabase.pulse.init}
:uses #{api
app-db
channel
@@ -1873,6 +1889,7 @@
graph
lib
lib-be
lib-metric
metrics
models
parameters
@@ -2040,8 +2057,10 @@
metabase.remote-sync.init}
:uses #{api
events
lib
models
premium-features}
premium-features
util}
:model-imports #{:model/Collection}}
request
@@ -2055,7 +2074,8 @@
api
config
embedding
permissions ; FIXME
lib
permissions
premium-features
session
settings
@@ -2107,10 +2127,12 @@
:api #{metabase.sample-data.core
metabase.sample-data.init}
:uses #{driver
lib
plugins
settings
sync
util}
util
warehouses}
:model-imports #{:model/DataPermissions :model/Database :model/Table}}
search
@@ -2158,10 +2180,21 @@
:model/Table
:model/User}}
secrets
{:team "Graphy"
:api #{metabase.secrets.core}
:uses #{api driver models premium-features util}}
:uses #{api
driver
lib
models
premium-features
util}}
security-center
{:team "Gadget"
:api #{metabase.security-center.schema}
:uses #{util}}
segments
{:team "Graphy"
@@ -2199,6 +2232,7 @@
driver
embedding
initialization-status
lib
lib-be
mcp
oauth-server
@@ -2228,6 +2262,7 @@
channel
config
events
lib
premium-features
request
settings
@@ -2246,6 +2281,7 @@
app-db
config
events
lib
models
premium-features
util
@@ -2267,6 +2303,7 @@
metabase.setup.init}
:uses #{app-db
config
lib
settings
util}
:model-imports #{:model/User}}
@@ -2279,6 +2316,7 @@
appearance
auth-identity
events
lib
request
settings
setup
@@ -2349,10 +2387,10 @@
metabase.sso.core
metabase.sso.init
metabase.sso.settings}
:friends #{enterprise/sso}
:uses #{api
auth-identity
config
lib
permissions
premium-features
request
@@ -2433,11 +2471,10 @@
task-history
{:team "UX West"
:api #{metabase.task-history.api
metabase.task-history.core
metabase.task-history.init}
:api #{metabase.task-history.api metabase.task-history.core metabase.task-history.init}
:uses #{api
config
lib
models
permissions
premium-features
@@ -2467,10 +2504,13 @@
config
lib
lib-be
llm
mcp
metabot
permissions
premium-features
search
security-center
session
util}
:model-imports #{:model/AiUsageLog
@@ -2501,12 +2541,12 @@
timeline
{:team "UX West"
:api #{metabase.timeline.api
metabase.timeline.core}
:api #{metabase.timeline.api metabase.timeline.core}
:uses #{analytics
api
collections
events
lib
models
util}
:model-exports #{:model/Timeline}
@@ -2555,8 +2595,7 @@
tracing
transforms-base
util}
:friends #{models
enterprise/transforms-python}
:friends #{enterprise/transforms-python}
:model-exports #{:model/Transform
:model/TransformDagRun
:model/TransformJob
@@ -2609,6 +2648,7 @@
metabase.transforms-rest.api.transform-job
metabase.transforms-rest.api.transform-tag}
:uses #{api
lib
models
request
transforms
@@ -2670,6 +2710,7 @@
driver
events
lib
lib-be
model-persistence
models
permissions
@@ -2724,6 +2765,7 @@
channel
config
events
lib
models
notification
permissions
@@ -2750,6 +2792,7 @@
collections
config
events
lib
models
permissions
premium-features
@@ -2785,14 +2828,14 @@
view-log
{:team "UX West"
:api #{metabase.view-log.core
metabase.view-log.init}
:api #{metabase.view-log.core metabase.view-log.init}
:uses #{analytics
api
app-db
audit-app
batch-processing
events
lib
models
premium-features
query-permissions
@@ -2813,6 +2856,7 @@
metabase.warehouse-schema.models.field-user-settings
metabase.warehouse-schema.models.field-values
metabase.warehouse-schema.models.table
metabase.warehouse-schema.schema
metabase.warehouse-schema.table}
:uses #{analyze
api
@@ -2878,7 +2922,8 @@
{:team "UX West"
:api #{metabase.warehouses.core
metabase.warehouses.init
metabase.warehouses.models.database}
metabase.warehouses.models.database
metabase.warehouses.schema}
:uses #{analytics-interface
api
app-db
@@ -2954,6 +2999,7 @@
query-processor
segments
settings
types
util
warehouse-schema}
:model-imports #{:model/Card
@@ -3007,7 +3053,8 @@
setup
sync
users
util}
util
warehouses}
:model-imports #{:model/ApiKey :model/Database :model/User}}
enterprise/advanced-permissions
@@ -3050,6 +3097,7 @@
:api #{}
:uses #{driver
premium-features
util
enterprise/advanced-config
enterprise/mfa
enterprise/scim
@@ -3084,6 +3132,7 @@
permissions
plugins
premium-features
queries
query-permissions
query-processor
settings
@@ -3092,6 +3141,8 @@
sync
users
util
warehouse-schema
warehouses
enterprise/serialization}
:model-imports #{:model/AuditLog
:model/Card
@@ -3124,6 +3175,7 @@
{:team "UX West"
:api #{metabase-enterprise.billing.api.routes}
:uses #{api
lib
premium-features
store-api
util}
@@ -3133,6 +3185,7 @@
{:team "UX West"
:api #{metabase-enterprise.cache.init}
:uses #{cache
lib
premium-features
query-processor
task
@@ -3185,6 +3238,7 @@
:api #{metabase-enterprise.content-verification.api.routes}
:uses #{api
content-verification
lib
util
enterprise/api}
:model-imports #{:model/Card :model/Dashboard}}
@@ -3216,15 +3270,28 @@
:uses #{premium-features}}
enterprise/data-apps
{:team "Gadget"
{:team "Embedding"
:api #{metabase-enterprise.data-apps.api
metabase-enterprise.data-apps.init
metabase-enterprise.data-apps.sync}
:uses #{api
collections
lib
lib-be
models
permissions
premium-features
request
settings
util}}
util}
:model-imports #{:model/Card
:model/Collection
:model/DataPermissions
:model/Database
:model/Permissions
:model/PermissionsGroup
:model/Table
:model/User}}
enterprise/data-complexity-score
{:team "Metabot"
@@ -3239,6 +3306,7 @@
config
driver
embeddings
lib
metabot
models
premium-features
@@ -3262,14 +3330,17 @@
:uses #{api
collections
events
lib
models
permissions
premium-features
util}
:model-imports #{:model/Collection
:model-imports #{:model/AuditLog
:model/Collection
:model/Dimension
:model/Field
:model/Table}}
:model/Table
:model/User}}
enterprise/database-replication
{:team "Cloud"
@@ -3277,6 +3348,7 @@
metabase-enterprise.database-replication.init}
:uses #{api
driver
lib
premium-features
settings
util
@@ -3364,8 +3436,10 @@
appearance
audit-app
embedding
lib
permissions
premium-features
util
enterprise/sso}
:model-imports #{:model/Card
:model/Collection
@@ -3393,6 +3467,7 @@
collections
entity-retrieval
health-inspector
lib
premium-features
search
task
@@ -3408,6 +3483,7 @@
{:team "Gadget"
:api #{metabase-enterprise.erd.api}
:uses #{api
lib
models
permissions
util
@@ -3423,9 +3499,11 @@
:uses #{analytics
analytics-interface
api
lib
premium-features
settings
util
warehouses
enterprise/harbormaster}
:model-imports #{:model/Database :model/Setting}}
@@ -3463,13 +3541,16 @@
:api #{metabase-enterprise.library.api}
:uses #{api
collections
premium-features}
lib
premium-features
util}
:model-imports #{:model/Card :model/Collection :model/Table}}
enterprise/mcp
{:team "Metabot"
:api #{metabase-enterprise.mcp.init}
:uses #{analytics
lib
mcp
metabot
premium-features
@@ -3497,10 +3578,8 @@
:api #{metabase-enterprise.metabot.api
metabase-enterprise.metabot.api.routes
metabase-enterprise.metabot.init}
:friends #{slackbot
mcp
metabot}
:uses #{api
lib
lib-be
llm
metabot
@@ -3524,6 +3603,7 @@
:api #{metabase-enterprise.metabot-analytics.api}
:uses #{api
driver
lib
metabot
permissions
query-processor
@@ -3540,9 +3620,11 @@
metabase-enterprise.mfa.routes}
:uses #{api
appearance
auth-identity
channel
config
events
lib
premium-features
request
settings
@@ -3577,6 +3659,7 @@
app-db
collections
events
lib
models
premium-features
search
@@ -3611,12 +3694,15 @@
:uses #{analytics
api
app-db
dashboards
events
lib
lib-be
measures
model-persistence
models
queries
segments
source-swap
task
transforms
@@ -3636,8 +3722,7 @@
enterprise/sandbox
{:team "UX West"
:api #{metabase-enterprise.sandbox.api.routes
metabase-enterprise.sandbox.api.util}
:api #{metabase-enterprise.sandbox.api.routes metabase-enterprise.sandbox.api.util}
:uses #{api
app-db
audit-app
@@ -3649,6 +3734,7 @@
models
permissions
premium-features
queries
query-processor
request
tenants
@@ -3674,6 +3760,7 @@
:uses #{analytics-interface
api
api-keys
lib
models
permissions
server
@@ -3694,8 +3781,7 @@
enterprise/security-center
{:team "Gadget"
:api #{metabase-enterprise.security-center.api
metabase-enterprise.security-center.init}
:api #{metabase-enterprise.security-center.api metabase-enterprise.security-center.init}
:uses #{analytics
analytics-interface
api
@@ -3704,10 +3790,12 @@
config
driver
events
lib
models
notification
permissions
premium-features
security-center
settings
system
task
@@ -3735,12 +3823,14 @@
embeddings
events
health-inspector
lib
lib-be
llm
metabot
models
permissions
premium-features
request
search
settings
task
@@ -3778,7 +3868,8 @@
enterprise/snippet-collections
{:team "Graphy"
:api #{}
:uses #{models
:uses #{lib
models
native-query-snippets
permissions
premium-features
@@ -3814,6 +3905,7 @@
:uses #{analytics
api
collections
lib
premium-features
queries
request
@@ -3830,11 +3922,13 @@
app-db
auth-identity
events
lib
models
request
settings
system
task
users
util}
:model-imports #{:model/AuthIdentity :model/Session :model/User}}
@@ -3846,6 +3940,7 @@
auth-identity
collections
events
lib
models
permissions
premium-features
@@ -3858,7 +3953,8 @@
enterprise/transforms
{:team "Graphy"
:api #{metabase-enterprise.transforms.api}
:uses #{premium-features
:uses #{lib
premium-features
util
enterprise/transforms-inspector}
:model-imports #{:model/Transform}}
@@ -3915,6 +4011,7 @@
{:team "UX West"
:api #{metabase-enterprise.upload-management.api}
:uses #{api
lib
models
permissions
premium-features

View File

@@ -0,0 +1,6 @@
;; Budgets for escape hatches in config.edn. Counts may fall but not exceed these values.
;; Leave reductions to the post-merge shrink workflow; explain increases in the PR.
{:api-any 1
:friend-edges 22
:model-imports-bypass 2
:uses-any 4}

View File

@@ -5,7 +5,7 @@
;; Any ignore outside :comment-exempt needs an explanatory comment directly above or trailing on its line.
;; `./bin/mage kondo-ratchets-shrink` lowers budgets unless `--seed` explicitly adds or raises one.
;; The workflow runs it on master, so feature branches do not need to record reductions.
;; Raising or adding a budget (`--seed` for inline ignores, a manual edit for config) or widening the
;; Raising or adding a budget (`--seed` for inline ignores, a manual edit otherwise) or widening the
;; exemptions must be explained in the PR.
;; :all is the vector-less ignore form, which suppresses every linter on the next form.
{:ignore-counts {:aliased-namespace-symbol 3
@@ -21,7 +21,7 @@
:duplicate-require 1
:equals-true 1
:main-without-gen-class 1
:metabase/defsetting-namespace 2
:metabase/defsetting-namespace 1
:metabase/disallow-hardcoded-driver-names-in-tests 34
:metabase/discourage-millis-duration 1
:metabase/i-like-making-cams-eyes-bleed-with-horrifically-long-tests 13
@@ -29,8 +29,8 @@
:metabase/namespace-name 4
:metabase/prefer-with-dynamic-fn-redefs 28
:metabase/test-helpers-use-non-thread-safe-functions 18
:metabase/validate-defendpoint-has-response-schema 439
:metabase/validate-defendpoint-query-params-use-kebab-case 50
:metabase/validate-defendpoint-has-response-schema 427
:metabase/validate-defendpoint-query-params-use-kebab-case 48
:metabase/validate-defendpoint-route-uses-kebab-case 44
:metabase/validate-deftest 10
:missing-docstring 12
@@ -61,6 +61,7 @@
:metabase/defsetting-must-specify-export 1
:metabase/defsetting-namespace 1
:metabase/prefer-with-dynamic-fn-redefs 3
:metabase/t2-query-namespace 2
:missing-docstring 1
:private-call 1
:redundant-ignore 1

View File

@@ -92,7 +92,7 @@
(defn- lint-modules [ns-form-node config]
(let [ns-symb (ns-form-node->ns-symb ns-form-node)]
(when-not (modules/ignored-namespace? config ns-symb)
(when-let [current-module (modules/module ns-symb)]
(when-let [current-module (modules/module config ns-symb)]
(let [required-namespace-symb-nodes (-> ns-form-node
ns-form-node->require-node
require-node->namespace-symb-nodes)]

View File

@@ -11,10 +11,10 @@
(every? simple-symbol? required-namespaces)]}
(when-not (modules/ignored-namespace? config current-ns)
;; ignore namespaces outside of the module system
(when-let [current-module (modules/module current-ns)]
(when-let [current-module (modules/module config current-ns)]
(doseq [required-namespace required-namespaces]
;; ignore namespaces outside of the module system.
(when (modules/module required-namespace)
(when (modules/module config required-namespace)
(when-let [error (modules/usage-error config current-module required-namespace)]
(hooks/reg-finding! (assoc (meta node)
:message error
@@ -22,7 +22,7 @@
(defn- lint-dynamic-require* [node current-ns config]
(when (and (not (modules/ignored-namespace? config current-ns))
(modules/module current-ns))
(modules/module config current-ns))
(hooks/reg-finding! (assoc (meta node)
:message "Module dependency cannot be statically determined from a dynamic require"
:type :metabase/modules))))

View File

@@ -194,7 +194,7 @@
(not (contains? @warned-namespaces ns-symb)))
(swap! warned-namespaces conj ns-symb)
(let [known-modules (set (keys (:metabase/modules (modules/config input))))
current-module (modules/module ns-symb)]
current-module (modules/module (modules/config input) ns-symb)]
(when (or (not current-module)
(not (contains? known-modules current-module)))
(hooks/reg-finding! (assoc (meta node)

View File

@@ -1,4 +1,8 @@
(ns hooks.common.modules
"Resolves namespaces to modules and enforces module boundaries.
The Kondo linter, config generator, and CI tooling share this namespace so
they apply the same rules."
(:require
[clojure.string :as str]))
@@ -12,25 +16,130 @@
(merge (get-in config [:linters :metabase/modules])
(select-keys config [:metabase/modules])))
(defn module
;;;; Module tree. These functions take the `:metabase/modules` map, or the prefix map built from it.
;;;;
;;;; Dots express nesting: `lib.schema` is a child of `lib`. A declared
;;;; `enterprise/X` module is also a child of OSS module `X`. A namespace
;;;; belongs to the module with the most specific matching `:ns-prefix`.
(defn default-ns-prefix
"Namespace prefix a module owns unless it sets `:ns-prefix`: `metabase.lib.schema` for `lib.schema`,
`metabase-enterprise.foo` for `enterprise/foo`."
[module]
(str (if (= "enterprise" (namespace module)) "metabase-enterprise." "metabase.") (name module)))
(defn module-ns-prefix
"Namespace prefix `module` owns: its `:ns-prefix`, else [[default-ns-prefix]]."
[modules module]
(or (get-in modules [module :ns-prefix])
(default-ns-prefix module)))
(defn build-prefix->module
"Map each declared namespace prefix to its module, for [[declared-module]]."
[modules]
(into {}
(map (fn [module] [(module-ns-prefix modules module) module]))
(keys modules)))
(defn declared-module
"Resolve `ns-symb` to its declared module, or `nil`.
Chooses the most specific dotted prefix and ignores a trailing `-test`."
[prefix->module ns-symb]
(loop [candidate (str/replace (str ns-symb) #"-test$" "")]
(or (get prefix->module candidate)
(when-let [dot (str/last-index-of candidate ".")]
(recur (subs candidate 0 dot))))))
(defn resolve-module
"E.g.
(module 'metabase.qp.middleware.wow) => 'qp
(module 'metabase-enterprise.whatever.core) => enterprise/whatever"
[ns-symb]
(resolve-module prefix->module 'metabase.qp.middleware.wow) => 'qp
(resolve-module prefix->module 'metabase-enterprise.whatever.core) => enterprise/whatever
An unmatched Metabase namespace resolves to its first segment so the linter can report an undeclared module."
[prefix->module ns-symb]
{:pre [(simple-symbol? ns-symb)]}
;; treat something like `metabase.driver-test` (for a module that hasn't fully been updated to use `.core`
;; namespaces) as being in the `driver` module
(let [ns-symb (if (str/ends-with? (name ns-symb) "-test")
(symbol (str/replace (name ns-symb) #"-test$" ""))
ns-symb)]
(or (some->> (re-find #"^metabase-enterprise\.([^.]+)" (str ns-symb))
(or (declared-module prefix->module ns-symb)
(some->> (re-find #"^metabase-enterprise\.([^.]+)" (str ns-symb))
second
(symbol "enterprise"))
(some-> (re-find #"^metabase\.([^.]+)" (str ns-symb))
second
symbol))))
(defn parent-module
"The direct parent of `module`, or `nil` when it is top-level."
[modules module]
(let [module-name (name module)]
(if-let [dot (str/last-index-of module-name ".")]
(symbol (namespace module) (subs module-name 0 dot))
(when (= "enterprise" (namespace module))
(let [oss (symbol module-name)]
(when (contains? modules oss)
oss))))))
(defn descendant-of?
"Whether `module` is `ancestor` or one of its descendants."
[modules module ancestor]
(boolean (some #{ancestor} (take-while some? (iterate #(parent-module modules %) module)))))
(defn- exports-child?
[modules parent child]
(or (contains? (set (get-in modules [parent :module-exports])) child)
;; OSS module `X` implicitly exports its `enterprise/X` companion.
(and (not= (namespace parent) (namespace child))
(contains? modules child))))
(defn- blocking-export
"The first missing export from `module` toward the root.
Returns `{:ancestor parent, :child child}`, or `nil` when every link is
exported."
[modules module]
(loop [child module]
(when-let [ancestor (parent-module modules child)]
(if (exports-child? modules ancestor child)
(recur ancestor)
{:ancestor ancestor, :child child}))))
(defn namability-error
"Explain why `caller` may not refer to `target` in `:uses`, or return `nil` if it may.
A nested module is private to the nearest ancestor with a missing export.
Exporting every link makes it public."
[modules caller target]
(when-let [{:keys [ancestor child]} (blocking-export modules target)]
(when-not (descendant-of? modules caller ancestor)
(format (str "Module %s is nested and not exported by its ancestors; %s may not use it. Add %s to %s's "
":module-exports, or move the caller into the %s subtree. [:metabase/modules %s :module-exports]")
target caller child ancestor ancestor ancestor))))
;;;; Lint rules. These functions take the full linter config.
(defonce ^:private prefix->module-cache
(atom nil))
(defn- prefix->module
"[[build-prefix->module]] for `config`, cached because Kondo hands every hook call the same `:metabase/modules` map."
[config]
(let [modules (:metabase/modules config)
cached @prefix->module-cache]
(if (identical? modules (:modules cached))
(:prefix->module cached)
(:prefix->module (reset! prefix->module-cache {:modules modules
:prefix->module (build-prefix->module modules)})))))
(defn module
"Resolve `ns-symb` to its module under `config`; see [[resolve-module]]."
[config ns-symb]
(resolve-module (prefix->module config) ns-symb))
(defn- module-api-namespaces
"Set of API namespace symbols for a given module. `:any` means you can use anything, there are no API namespaces for
this module (yet). If unspecified, the default is just the `<module>.core` namespace."
@@ -44,9 +153,7 @@
module-config
:else
(let [ns-prefix (if (= (namespace module) "enterprise")
(str "metabase-enterprise." (name module))
(name module))]
(let [ns-prefix (module-ns-prefix (:metabase/modules config) module)]
#{(symbol (str ns-prefix ".api"))
(symbol (str ns-prefix ".core"))
(symbol (str ns-prefix ".init"))}))))
@@ -68,15 +175,17 @@
(contains? (set allowed-modules) required-module))))
(defn- allowed-module-namespace? [config current-module ns-symb]
(let [module (module ns-symb)
(let [module (module config ns-symb)
module-api-namespaces (module-api-namespaces config module)
module-friends (module-friends config module)]
(or (nil? module-api-namespaces)
(contains? module-api-namespaces ns-symb)
(contains? module-friends current-module))))
(contains? module-friends current-module)
;; a child may use its ancestors' internals; a parent still goes through its child's `:api`
(descendant-of? (:metabase/modules config) current-module module))))
(defn- rest-module? [module]
(str/ends-with? module "-rest"))
(re-find #"[.-]rest$" (str module)))
(defn- routes-module? [module]
(str/ends-with? module "-routes"))
@@ -89,7 +198,7 @@
error type if there is one, otherwise `nil` if there are no errors."
[config current-module required-namespace]
;; ignore stuff not in a module i.e. non-Metabase stuff.
(when-let [required-module (module required-namespace)]
(when-let [required-module (module config required-namespace)]
(when-not (= current-module required-module)
(cond
(not (allowed-module? config current-module required-module))
@@ -113,4 +222,8 @@
required-module
current-module
required-module
(symbol (str/replace required-module #"-rest$" "")))))))
(symbol (str/replace required-module #"[.-]rest$" "")))
;; Config tests check explicit `:uses`; wildcard callers are checked here.
(= :any (allowed-modules config current-module))
(namability-error (:metabase/modules config) current-module required-module)))))

View File

@@ -4,9 +4,9 @@
[hooks.common :as common]))
(defn lint-defsetting-namespace [node context]
(when-not (re-matches #"^metabase(?:-enterprise)?\.[^\.]+\.settings$" (name (:ns context)))
(when-not (re-matches #"^metabase(?:-enterprise)?\.([^\.]+\.)+settings$" (name (:ns context)))
(hooks/reg-finding! (assoc (meta node)
:message "All defsettings should live in metabase[-enterprise].<module>.settings namespaces"
:message "All defsettings should live in a namespace ending in .settings under metabase[-enterprise]"
:type :metabase/defsetting-namespace))))
;;; TODO -- move this into a Kondo config file in `.clj-kondo/config/`

View File

@@ -0,0 +1,45 @@
(ns hooks.metabase.toucan.db-ns
"Lint that application database query calls -- Toucan 2's `t2/select`, `t2/query`, `t2/insert!`, `t2/update!`,
`t2/delete!` and friends, plus the `metabase.app-db.core` wrappers around them such as `mdb/query` and
`mdb/update-or-insert!` -- live in a module's `db` namespace, i.e. `metabase[-enterprise].<module>.db` (or `metabase.driver.<driver>.db` for
driver modules). Every other namespace in a module goes through those functions instead of talking to the
application database directly.
Registered as an `:analyze-call` hook on each of those functions in `.clj-kondo/config.edn`. The hook
returns its input unchanged so Kondo's normal analysis of the call (arity, var usage) still runs."
(:require
[clj-kondo.hooks-api :as hooks]
[hooks.common.modules :as modules]))
(def ^:private driver-db-namespace
"Matches `metabase.driver.<driver>.db`; individual drivers are not modules."
#"^metabase\.driver\.[^.]+\.db$")
(defn- db-namespace? [config ns-sym]
(let [ns-str (name ns-sym)]
(boolean
;; Resolve ownership first: a nested `.db` name alone does not make a module.
(or (when-let [module (modules/module config ns-sym)]
(= ns-str (str (modules/module-ns-prefix (:metabase/modules config) module) ".db")))
(re-matches driver-db-namespace ns-str)))))
(defn- test-file?
"Whether `filename` is in a test source tree. Test namespaces are exempt through the `test-namespaces` group in
`config.edn`, but helper namespaces like `metabase.sso.test-helpers` don't match that group's pattern."
[filename]
(boolean (and filename (re-find #"(?:^|/)test/" filename))))
(defn lint-query-call
"Register a `:metabase/t2-query-namespace` finding when a Toucan 2 query call appears outside a `<module>.db`
namespace."
[{:keys [node ns filename] :as input}]
(when (and ns
(not (db-namespace? (modules/config input) ns))
(not (test-file? filename)))
(let [fn-node (first (:children node))]
(hooks/reg-finding!
(assoc (meta fn-node)
:message (format "Application database query calls like `%s` must live in metabase[-enterprise].<module>.db (or metabase.driver.<driver>.db) namespaces"
(hooks/sexpr fn-node))
:type :metabase/t2-query-namespace))))
input)

View File

@@ -5,6 +5,6 @@
(deftest ^:parallel module-test
(are [symb expected] (= expected
(hooks.common.modules/module symb))
(hooks.common.modules/module {} symb))
'metabase.qp.middleware.wow 'qp
'metabase-enterprise.whatever.core 'enterprise/whatever))

View File

@@ -0,0 +1,73 @@
(ns hooks.metabase.toucan.db-ns-test
(:require
[clj-kondo.hooks-api :as hooks]
[clj-kondo.impl.utils]
[clojure.test :refer :all]
[hooks.metabase.toucan.db-ns :as toucan.db-ns]))
(defn- lint-query-call [form ns-sym & [filename modules]]
(binding [clj-kondo.impl.utils/*ctx* {:config {:linters {:metabase/t2-query-namespace {:level :warning}}}
:ignores (atom nil)
:findings (atom [])
:namespaces (atom {})}]
(let [input {:node (hooks/parse-string (pr-str form))
:ns ns-sym
:filename (or filename "src/metabase/foo/bar.clj")
:config {:metabase/modules modules}}
output (toucan.db-ns/lint-query-call input)]
(is (identical? (:node input) (:node output))
"the hook must return the node unchanged so Kondo's normal analysis still runs")
@(:findings clj-kondo.impl.utils/*ctx*))))
(deftest ^:parallel t2-query-must-live-in-db-namespace-test
(testing "a query call outside a db namespace is flagged"
(is (=? [{:type :metabase/t2-query-namespace
:message #".*`t2/select-one`.*"}]
(lint-query-call '(t2/select-one :model/Card :id 1) 'metabase.queries.models.card))))
(testing "metabase.<module>.db is allowed"
(is (empty? (lint-query-call '(t2/select-one :model/Card :id 1) 'metabase.queries.db))))
(testing "metabase-enterprise.<module>.db is allowed"
(is (empty? (lint-query-call '(t2/select-one :model/Card :id 1) 'metabase-enterprise.sandbox.db))))
(testing "metabase.driver.<driver>.db is allowed for driver modules"
(is (empty? (lint-query-call '(t2/select-one :model/Database :id 1) 'metabase.driver.bigquery-cloud-sdk.db))))
(testing "a nested db namespace is not a module db namespace"
(is (=? [{:type :metabase/t2-query-namespace}]
(lint-query-call '(t2/query {:select [:*]}) 'metabase.queries.models.db))))
(testing "a namespace merely ending in db is not a module db namespace"
(is (=? [{:type :metabase/t2-query-namespace}]
(lint-query-call '(t2/delete! :model/Card :id 1) 'metabase.queries.dashboards-db))))
(testing "files in a test source tree are exempt even when the namespace name doesn't look like a test"
(is (empty? (lint-query-call '(t2/update! :model/User 1 {:sso_source nil}) 'metabase.sso.test-helpers
"test/metabase/sso/test_helpers.clj")))
(is (empty? (lint-query-call '(t2/select :model/Card) 'metabase-enterprise.remote-sync.test-helpers
"/repo/enterprise/backend/test/metabase_enterprise/remote_sync/test_helpers.clj"))))
(testing "app-db wrappers around Toucan 2 are reported the same way"
(is (=? [{:type :metabase/t2-query-namespace
:message #".*`mdb/update-or-insert!`.*"}]
(lint-query-call '(mdb/update-or-insert! :model/Foo {:id 1}) 'metabase.foo.models.foo))))
(testing "fully-qualified calls are reported by their written name"
(is (=? [{:type :metabase/t2-query-namespace
:message #".*`toucan2.core/insert!`.*"}]
(lint-query-call '(toucan2.core/insert! :model/Card {}) 'metabase.queries.models.card)))))
(deftest ^:parallel t2-query-namespace-follows-the-module-tree-test
(let [modules '{queries {}
metabot {}
metabot.llm {}
lib.be {:ns-prefix "metabase.lib-be"}
enterprise/sandbox {}}]
(testing "a nested module owns its own db namespace"
(is (empty? (lint-query-call '(t2/select :model/Card) 'metabase.metabot.llm.db nil modules))))
(testing "the parent's db namespace stays its own"
(is (empty? (lint-query-call '(t2/select :model/Card) 'metabase.metabot.db nil modules))))
(testing "a module with an explicit :ns-prefix is resolved through it, not its dotted name"
(is (empty? (lint-query-call '(t2/select :model/Card) 'metabase.lib-be.db nil modules)))
(is (=? [{:type :metabase/t2-query-namespace}]
(lint-query-call '(t2/select :model/Card) 'metabase.lib.be.db nil modules))))
(testing "a db namespace under a directory naming no module is still a finding"
(is (=? [{:type :metabase/t2-query-namespace}]
(lint-query-call '(t2/query {:select [:*]}) 'metabase.queries.models.db nil modules)))
(is (=? [{:type :metabase/t2-query-namespace}]
(lint-query-call '(t2/query {:select [:*]}) 'metabase.metabot.llm.models.db nil modules))))
(testing "enterprise modules resolve through the metabase-enterprise root"
(is (empty? (lint-query-call '(t2/select :model/Card) 'metabase-enterprise.sandbox.db nil modules))))))

View File

@@ -14,3 +14,6 @@ paths:
.github/workflows/lint.yml:
ignore:
- 'step must run script with "run" section or run action with "uses" section'
.github/workflows/pr.checks.yml:
ignore:
- 'step must run script with "run" section or run action with "uses" section'

View File

@@ -20,6 +20,15 @@ inputs:
description: Directory path where the jar should be extracted (defaults to current directory)
required: false
default: "."
download:
description: Default is true. Set to false to only look the artifact up and report `fetch-hit`.
required: false
default: "true"
outputs:
fetch-hit:
description: Boolean indicating if an artifact was successfully found and downloaded.
value: ${{ steps.find_release_artifact.outputs.fetch-hit }}
runs:
using: "composite"
@@ -28,11 +37,12 @@ runs:
id: find_release_artifact
uses: actions/github-script@v9
with:
result-encoding: string
script: | # js
const fs = require('fs');
const isRelease = "${{ inputs.release }}" === "true";
const failOnMiss = "${{ inputs.fail-on-fetch-miss }}" !== "false";
const shouldDownload = "${{ inputs.download }}" !== "false";
const prefix = isRelease ? "metabase-release" : "metabase";
const artifactName = `${prefix}-${{ inputs.edition }}-${{ inputs.commit }}-uberjar`;
console.log(`Looking for artifact: ${artifactName}`);
@@ -44,12 +54,18 @@ runs:
per_page: 1,
});
if (!artifacts.data?.artifacts?.[0]?.id) {
core.setFailed(`No artifacts found for ${artifactName}`);
return;
const artifact_id = artifacts.data?.artifacts?.[0]?.id;
if (!artifact_id) {
core.setOutput('fetch-hit', 'false');
} else {
core.setOutput('fetch-hit', 'true');
}
const artifact_id = artifacts.data.artifacts[0].id;
if (!shouldDownload) {
console.log(`skipping download (download: false)`);
return;
}
let download;
const maxAttempts = 3;
@@ -74,6 +90,7 @@ runs:
fs.writeFileSync(`${process.env.GITHUB_WORKSPACE}/mb.zip`, Buffer.from(download.data));
- name: Extract and rename JAR
if: ${{ steps.find_release_artifact.outputs.fetch-hit == 'true' && inputs.download != 'false' }}
shell: bash
run: | # bash
OUTPUT_NAME="${{ inputs.output-name }}"
@@ -119,12 +136,14 @@ runs:
fi
- name: Verify that this is a valid JAR file
if: ${{ steps.find_release_artifact.outputs.fetch-hit == 'true' && inputs.download != 'false' }}
shell: bash
run: |
echo "Verifying JAR at: $JAR_PATH"
file --mime-type "$JAR_PATH"
- name: Reveal its version.properties
if: ${{ steps.find_release_artifact.outputs.fetch-hit == 'true' && inputs.download != 'false' }}
shell: bash
run: |
echo "Extracting version.properties from: $JAR_PATH"

View File

@@ -1,7 +1,15 @@
name: Get Docker Info
description: Determines Docker repository and tag based on branch and edition
description: Determines the registry, repository, and tag a container image is pushed to.
inputs:
repo:
description: Registry and repo, e.g. "metabase/metabase". Overrides branch derived repo.
required: false
default: ''
tag:
description: Docker tag, e.g. v1.87.2. Overrides a branch derived tag.
required: false
default: ''
branch:
description: Branch name (defaults to current branch)
required: false
@@ -25,9 +33,6 @@ outputs:
full_image:
description: Full Docker image name (repo:tag)
value: ${{ steps.docker-info.outputs.full_image }}
editions:
description: JSON array of editions to build
value: ${{ steps.docker-info.outputs.editions }}
runs:
using: "composite"
@@ -41,34 +46,22 @@ runs:
const edition = '${{ inputs.edition }}';
const owner = '${{ inputs.owner }}';
console.log(`Determining Docker info for branch: ${branch}, edition: ${edition}`);
let repo, tag, editions;
if (branch === 'master') {
// Master builds both editions with specific repos
editions = ['ee', 'oss'];
tag = 'latest';
if (edition === 'ee') {
repo = `${owner}/metabase-enterprise-head`;
} else {
repo = `${owner}/metabase-head`;
}
} else {
// All other branches only build EE to metabase-dev
editions = ['ee'];
tag = branch.replaceAll('/', '-'); // slashes make docker sad
repo = `${owner}/metabase-dev`;
let tag = '${{ inputs.tag}}';
if (!tag) {
tag = branch === 'master' ? 'latest' : `${branch.replaceAll('/', '-')}-${edition}`;
}
const fullImage = `${repo}:${tag}`;
let repo = '${{ inputs.repo }}';
if (!repo) {
if (branch === 'master') {
repo = edition === 'ee' ? `${owner}/metabase-enterprise-head` : `${owner}/metabase-head`;
} else {
repo = `${owner}/metabase-dev`;
}
}
// Set outputs
core.setOutput('repo', repo);
core.setOutput('tag', tag);
core.setOutput('full_image', fullImage);
core.setOutput('editions', JSON.stringify(editions));
core.setOutput('full_image', `${repo}:${tag}`);
console.log(`Docker image: ${fullImage}`);
console.log(`Editions: ${JSON.stringify(editions)}`);
console.log(`Resolved: repo = ${repo}, tag = ${tag}`);

View File

@@ -14,21 +14,6 @@ inputs:
staging-mb-all-features-token:
description: "Staging MB all features token"
required: true
shoppy-iam-role:
description: "Shoppy IAM role for AWS credentials"
required: true
shoppy-metabase-app-db-s3-bucket:
description: "S3 bucket for Shoppy app DB dump"
required: true
shoppy-metabase-app-db-s3-key:
description: "S3 key for Shoppy app DB dump"
required: true
shoppy-metabase-app-db-user:
description: "Shoppy Metabase app DB user"
required: true
shoppy-metabase-app-db-database:
description: "Shoppy Metabase app DB database"
required: true
metastore-dev-server-url:
description: "Metastore dev server URL"
required: true
@@ -93,20 +78,6 @@ runs:
echo "sample_app_branch_exists=false" >> "$GITHUB_OUTPUT"
fi
- name: Configure AWS credentials
if: ${{ steps.check-sample-app-branch.outputs.sample_app_branch_exists == 'true' }}
uses: aws-actions/configure-aws-credentials@d979d5b3a71173a29b74b5b88418bfda9437d885
with:
role-to-assume: ${{ inputs.shoppy-iam-role }}
role-session-name: GitHub_to_AWS_via_FederatedOIDC
aws-region: us-east-1
- name: Download Shoppy's App DB PostgreSQL dump from S3
if: ${{ steps.check-sample-app-branch.outputs.sample_app_branch_exists == 'true' }}
shell: bash
run: |
aws s3 cp s3://${{ inputs.shoppy-metabase-app-db-s3-bucket }}/${{ inputs.shoppy-metabase-app-db-s3-key }} ./e2e/tmp/db_dumps/shoppy_metabase_app_db_dump.sql
- name: Prepare and launch Shoppy containers
if: ${{ steps.check-sample-app-branch.outputs.sample_app_branch_exists == 'true' }}
shell: bash
@@ -115,8 +86,6 @@ runs:
SAMPLE_APP_ENVIRONMENT: ${{ inputs.sample-app-environment }}
CYPRESS_MB_ALL_FEATURES_TOKEN: ${{ inputs.staging-mb-all-features-token }}
METASTORE_DEV_SERVER_URL: ${{ inputs.metastore-dev-server-url }}
METABASE_APP_DB_USER: ${{ inputs.shoppy-metabase-app-db-user }}
METABASE_APP_DB: ${{ inputs.shoppy-metabase-app-db-database }}
SHOPPY_DATADOG_APPLICATION_ID: ${{ inputs.shoppy-datadog-application-id }}
SHOPPY_DATADOG_CLIENT_TOKEN: ${{ inputs.shoppy-datadog-client-token }}
SHOPPY_DATADOG_SITE: "us5.datadoghq.com"

View File

@@ -20,8 +20,30 @@ inputs:
required: false
default: "ee"
description: Edition to test (oss or ee)
quarantine-engine:
runner-index:
required: false
description: >-
The index of this runner among those sharing a partitioned suite. See
`be-tests-bigquery-cloud-sdk-ee` in `.github/workflows/drivers.yml` for an example.
runner-count:
required: false
description: >-
How many runners share the suite. Recommended to set to `strategy.job-total` rather
than a literal so that it doesn't need updating if the `runner-index` matrix changes.
jvms-per-runner:
required: false
default: "1"
description: >-
Concurrent JVMs per runner. Recommended to raise above 1 for drivers whose tests
spend a lot of time waiting on a remote warehouse.
heap-per-jvm:
required: false
default: "4g"
description: >-
`-Xmx` per JVM, applied only when `jvms-per-runner` is more than 1 (otherwise the
the `:ci` alias's `-Xmx12g` is used). To size it: reserve ~1G for the OS and ~2G
of non-heap per JVM, then divide what's left. On a 16G runner that's ~4G each for
two JVMs.
granular-rerun:
required: false
default: "true"
@@ -100,6 +122,7 @@ runs:
# Otherwise run exactly as before. Quote the file so the whole `[ns/var ...]`
# vector reaches clojure -X as a single argument.
run: |
ALIASES="dev:ci:${{ inputs.edition }}:${{ inputs.edition }}-dev:drivers:drivers-dev:test"
BASE="clojure -X:dev:ci:${{ inputs.edition }}:${{ inputs.edition }}-dev:drivers:drivers-dev:test"
# `:profile-tests` overrides `:test`'s exec-fn, so it has to come last. See dev.profile-tests.
# The `profiled` output is what the collection steps below key off, so that the decision is
@@ -117,12 +140,19 @@ runs:
$BASE :only "$ONLY"
elif [ "${{ inputs.granular-rerun }}" = "true" ] && [ "${{ github.run_attempt }}" != "1" ] && [ -s target/failed-tests ]; then
echo "Granular rerun of previously-failed tests: $(cat target/failed-tests)"
$BASE :only "$(cat target/failed-tests)"
clojure -X:"$ALIASES" :only "$(cat target/failed-tests)"
elif [ -n "$TEST_RUNNER_INDEX" ]; then
CLOJURE_ALIASES="$ALIASES" ./.github/scripts/run-backend-test-jvms.sh ${{ inputs.test-args }}
else
$BASE ${{ inputs.test-args }}
clojure -X:"$ALIASES" ${{ inputs.test-args }}
fi
shell: bash
id: run-test
env:
TEST_RUNNER_INDEX: ${{ inputs.runner-index }}
TEST_RUNNER_COUNT: ${{ inputs.runner-count }}
TEST_JVMS_PER_RUNNER: ${{ inputs.jvms-per-runner }}
TEST_HEAP_PER_JVM: ${{ inputs.heap-per-jvm }}
# Keep this step continue-on-error because the quarantine gate below controls whether it
# ultimately fails.
continue-on-error: true
@@ -186,7 +216,9 @@ runs:
uses: actions/upload-artifact@v7
with:
name: ${{ format('test-logs-{0}{1}', github.job, inputs.artifact-suffix && format('-{0}', inputs.artifact-suffix) || '') }}
path: logs/test-log.json
# Glob: a single-JVM run writes `logs/test-log.json`; a multi-JVM run merges one
# `logs/test-log-p<index>.json` per JVM.
path: logs/test-log*.json
retention-days: 1
# Failing to upload the artifact should never affect the outcome of the whole test run.
continue-on-error: true
@@ -199,4 +231,4 @@ runs:
with:
adapter: backend
test-suite: ${{ inputs.junit-name }}
enforce-quarantine: ${{ inputs.quarantine-engine == 'ci-conductor' }}
enforce-quarantine: "true"

View File

@@ -115,10 +115,10 @@ backend_sources: &backend_sources
- "src/**"
- "modules/drivers/{*,*/{*,!(test)/**}}"
- "resources/**"
# The ratchets file drives project_ratchet_checks alone, so the automation PR that lowers the budgets
# runs the Project tests job by itself. Patterns are OR-ed, so a separate "!..." entry would match
# Ratchet files drive project_ratchet_checks alone, so automation PRs that lower budgets
# run the Project tests job by themselves. Patterns are OR-ed, so a separate "!..." entry would match
# every other path instead of excluding this one.
- ".clj-kondo/{!(ratchets.edn),*/**}"
- ".clj-kondo/**/!(ratchets.edn)"
backend_specs: &backend_specs
- *shared_specs
@@ -159,6 +159,8 @@ project_backend_checks:
- "modules/**/*.cljc"
- "modules/**/*.cljs"
- "**deps.edn"
- ".clj-kondo/{src,test}/**/*.{clj,cljc,cljs}"
- "{resources,test_resources,test_config}/**/*.{clj,cljc,cljs}"
- ".clj-kondo/config.edn"
- ".clj-kondo/config/modules/config.edn"
- ".github/team.json"
@@ -167,6 +169,7 @@ project_backend_checks:
# Only the ratchet check reads the budgets, so a diff that touches nothing else runs that suite alone.
project_ratchet_checks:
- ".clj-kondo/ratchets.edn"
- ".clj-kondo/config/modules/ratchets.edn"
project_migration_checks:
- *project_tests_infra

View File

@@ -24,11 +24,17 @@ const STORY_GLOBS = [
"enterprise/frontend/**/*.stories.{js,jsx,ts,tsx}",
];
// `git ls-files -- frontend enterprise/frontend` already prints just over a
// megabyte of paths, and node's default maxBuffer is exactly 1 MiB: past that
// the spawn dies with ENOBUFS. Give the listings room to grow.
const LS_FILES_MAX_BUFFER = 64 * 1024 * 1024;
// Returns the tracked files under `roots` that match `globs`. The `dot: true`
// option means files inside dot-directories such as `.storybook` are included.
function listFiles(roots: string[], globs: string[]): string[] {
const tracked = execFileSync("git", ["ls-files", "--", ...roots], {
encoding: "utf8",
maxBuffer: LS_FILES_MAX_BUFFER,
})
.split("\n")
.map((line) => line.trim())

View File

@@ -29,7 +29,7 @@ describe("file-paths.yaml", () => {
// dorny/paths-filter ORs the patterns in a filter, so a "!..." entry does not subtract from the
// other entries - it matches every path they don't, turning the filter permanently on. Excluding
// a path means writing the exclusion inside one pattern, the way backend_sources keeps
// ratchets.edn out with ".clj-kondo/{!(ratchets.edn),*/**}".
// ratchet files out with ".clj-kondo/**/!(ratchets.edn)".
it("never excludes a path with a standalone negation", () => {
const negated = Object.entries(filters).flatMap(([name, filter]) =>
patterns(filter)
@@ -44,6 +44,9 @@ describe("file-paths.yaml", () => {
"%s skips a ratchets-only change and still follows the rest of .clj-kondo",
(name) => {
expect(matches(name, ".clj-kondo/ratchets.edn")).toBe(false);
expect(
matches(name, ".clj-kondo/config/modules/ratchets.edn"),
).toBe(false);
expect(matches(name, ".clj-kondo/config.edn")).toBe(true);
expect(matches(name, ".clj-kondo/config/modules/config.edn")).toBe(true);
},
@@ -53,5 +56,11 @@ describe("file-paths.yaml", () => {
expect(matches("project_ratchet_checks", ".clj-kondo/ratchets.edn")).toBe(
true,
);
expect(
matches(
"project_ratchet_checks",
".clj-kondo/config/modules/ratchets.edn",
),
).toBe(true);
});
});

160
.github/scripts/run-backend-test-jvms.sh vendored Executable file
View File

@@ -0,0 +1,160 @@
#!/usr/bin/env bash
#
# Run this runner's share of a partitioned backend test suite as one or more concurrent
# JVMs. The caller says how many runners share the suite and how many JVMs each gets.
#
# WHY MULTIPLE JVMS PER RUNNER
# ----------------------------
# Driver tests spend most of their wall time waiting on the remote warehouse, and they
# can't use the idle CPU by using more threads because some tests mutate global state.
# A second JVM gets that isolation for free.
#
# Two paths are NOT safe to share:
# * `target/junit` — hawk deletes the whole directory at `:begin-test-run`, so a JVM
# starting late would wipe an earlier one's results.
# * `logs/test-log.json` — the log4j2 File appender truncates it on open.
# So each JVM runs in its own hardlinked copy of the checkout (inodes, not disk) and
# outputs are merged back into the workspace afterwards.
#
# INPUTS (environment)
# --------------------
# CLOJURE_ALIASES alias string for `clojure -X`, e.g. dev:ci:ee:ee-dev:drivers:drivers-dev:test
# TEST_RUNNER_INDEX index of this runner among those sharing the suite
# TEST_RUNNER_COUNT how many runners share the suite
# TEST_JVMS_PER_RUNNER concurrent JVMs per runner (default 1)
# TEST_HEAP_PER_JVM -Xmx per JVM, applied only when running more than one
#
# Exits non-zero if any JVM fails.
set -uo pipefail
: "${CLOJURE_ALIASES:?CLOJURE_ALIASES is required}"
: "${TEST_RUNNER_INDEX:?TEST_RUNNER_INDEX is required}"
: "${TEST_RUNNER_COUNT:?TEST_RUNNER_COUNT is required}"
TEST_ARGS=("$@")
TEST_JVMS_PER_RUNNER="${TEST_JVMS_PER_RUNNER:-1}"
TEST_HEAP_PER_JVM="${TEST_HEAP_PER_JVM:-4g}"
die() {
echo "::error::run-backend-test-jvms.sh: $1"
exit 1
}
[[ "$TEST_RUNNER_INDEX" =~ ^[0-9]+$ ]] || die "TEST_RUNNER_INDEX must be a non-negative integer, got '$TEST_RUNNER_INDEX'"
[[ "$TEST_RUNNER_COUNT" =~ ^[0-9]+$ ]] || die "TEST_RUNNER_COUNT must be a positive integer, got '$TEST_RUNNER_COUNT'"
[[ "$TEST_JVMS_PER_RUNNER" =~ ^[0-9]+$ ]] || die "TEST_JVMS_PER_RUNNER must be a positive integer, got '$TEST_JVMS_PER_RUNNER'"
[ "$TEST_RUNNER_COUNT" -ge 1 ] || die "TEST_RUNNER_COUNT must be at least 1"
[ "$TEST_JVMS_PER_RUNNER" -ge 1 ] || die "TEST_JVMS_PER_RUNNER must be at least 1"
[ "$TEST_RUNNER_INDEX" -lt "$TEST_RUNNER_COUNT" ] ||
die "runner index $TEST_RUNNER_INDEX is out of range for $TEST_RUNNER_COUNT runner(s) — a matrix entry was added without raising runner-count"
# Partition indexes are split across runners rather than handed out in contiguous
# blocks. E.g. with 2 runners and 2 JVMs the first runner gets partition 0 and 2, and
# the second runner gets partitions 1 and 3. Hawk partitions in namespace order, so
# neighbouring partitions have similar runtimes — contiguous blocks would group the slow
# ones together onto one runner.
PARTITION_TOTAL=$((TEST_RUNNER_COUNT * TEST_JVMS_PER_RUNNER))
INDEXES=()
for ((k = 0; k < TEST_JVMS_PER_RUNNER; k++)); do
INDEXES+=($((TEST_RUNNER_INDEX + k * TEST_RUNNER_COUNT)))
done
# One JVM: run directly in the workspace with no heap override, exactly as a
# partitioned run behaved before this script existed.
if [ "$TEST_JVMS_PER_RUNNER" -eq 1 ]; then
echo "Running partition ${INDEXES[0]} of $PARTITION_TOTAL"
exec clojure -X:"$CLOJURE_ALIASES" "${TEST_ARGS[@]}" \
:partition/total "$PARTITION_TOTAL" \
:partition/index "${INDEXES[0]}"
fi
WORKSPACE="$PWD"
# Sibling of the checkout: same filesystem (so `cp -al` can hardlink) but outside the
# repo, so one JVM's directory never gets copied into another's.
JVM_DIRS_ROOT="$(dirname "$WORKSPACE")/backend-test-jvms"
STATUS_DIR="$JVM_DIRS_ROOT/status"
# Merge every JVM's results into the workspace the rest of the job reads from.
#
# Each partition's junit XML goes into its own subdirectory: `mb_hawk_var_less_errors.xml`
# has a fixed name, so a flat copy would silently overwrite it. Nesting is safe because
# both consumers (dorny/test-reporter and ci-conductor's backend adapter) walk
# `target/junit` recursively and match on filename suffix.
#
# Runs from an EXIT trap so a cancelled or timed-out job still publishes whatever
# finished.
# shellcheck disable=SC2329 # invoked by the EXIT trap below, which shellcheck can't see.
merge_outputs() {
mkdir -p "$WORKSPACE/target/junit" "$WORKSPACE/logs"
local idx jvm_dir
for idx in "${INDEXES[@]}"; do
jvm_dir="$JVM_DIRS_ROOT/partition-$idx"
if [ -d "$jvm_dir/target/junit" ]; then
mkdir -p "$WORKSPACE/target/junit/p$idx"
cp -a "$jvm_dir/target/junit/." "$WORKSPACE/target/junit/p$idx/" || true
fi
# `logs/test-log*.json` is what the action uploads on failure.
if [ -f "$jvm_dir/logs/test-log.json" ]; then
cp -a "$jvm_dir/logs/test-log.json" "$WORKSPACE/logs/test-log-p$idx.json" || true
fi
done
}
trap merge_outputs EXIT
trap 'exit 143' TERM
trap 'exit 130' INT
# Hardlinked copy of the checkout. Skips the paths a run writes to (`target`, `logs`, `.cpcache`)
# and the two most expensive trees nothing on the test classpath reads (`.git`, `node_modules`).
prepare_jvm_dir() {
local jvm_dir="$1" entry
rm -rf "$jvm_dir"
mkdir -p "$jvm_dir"
for entry in "$WORKSPACE"/* "$WORKSPACE"/.[!.]*; do
[ -e "$entry" ] || continue
case "$(basename "$entry")" in
.git | node_modules | target | logs | .cpcache) continue ;;
esac
# Fall back to a real copy if the filesystem refuses the hardlink.
cp -al "$entry" "$jvm_dir/" 2>/dev/null || cp -a "$entry" "$jvm_dir/"
done
}
# Prefix output with the partition index so interleaved JVM logs stay readable.
run_partition() {
local idx="$1" jvm_dir="$2" rc line
(
cd "$jvm_dir" || exit 1
# `-J` options land after the aliases' `:jvm-opts`, overriding the `:ci` alias's
# `-Xms12g -Xmx12g`. Xms stays small so JVMs sharing a runner grow into memory on
# demand rather than each committing its ceiling up front.
exec clojure -J-Xms512m -J-Xmx"$TEST_HEAP_PER_JVM" -X:"$CLOJURE_ALIASES" \
"${TEST_ARGS[@]}" \
:partition/total "$PARTITION_TOTAL" \
:partition/index "$idx"
) 2>&1 | while IFS= read -r line || [ -n "$line" ]; do printf '[p%s] %s\n' "$idx" "$line"; done
rc="${PIPESTATUS[0]}"
echo "$rc" > "$STATUS_DIR/p$idx"
}
rm -rf "$JVM_DIRS_ROOT"
mkdir -p "$STATUS_DIR"
echo "Runner $TEST_RUNNER_INDEX of $TEST_RUNNER_COUNT: running $TEST_JVMS_PER_RUNNER concurrent JVMs for partition(s) ${INDEXES[*]} of $PARTITION_TOTAL (-Xmx$TEST_HEAP_PER_JVM each)"
for idx in "${INDEXES[@]}"; do
prepare_jvm_dir "$JVM_DIRS_ROOT/partition-$idx"
done
for idx in "${INDEXES[@]}"; do
run_partition "$idx" "$JVM_DIRS_ROOT/partition-$idx" &
done
wait
overall=0
for idx in "${INDEXES[@]}"; do
rc="$(cat "$STATUS_DIR/p$idx" 2>/dev/null || echo 1)"
echo "partition $idx exited $rc"
[ "$rc" -eq 0 ] || overall=1
done
exit "$overall"

View File

@@ -0,0 +1,97 @@
// Appends the measured load times to the "Bundle Load Times" table in
// eng-stats-importer. Reads ROWS (the JSON matrix.js prints) and API_KEY from
// env, and stamps each row with the commit it came from.
import { readFileSync } from "node:fs";
import { importStats } from "./stats-import";
/** One condition, as `frontend/test/bench/matrix.ts` reports it. */
interface Condition {
network: string;
networkMbps: number;
latencyMs: number;
cpu: string;
cpuThrottle: number;
coldMs: number;
warmMs: number;
steadyMs: number;
coldSpreadPercent: number;
coldTtfbMs: number;
coldFirstPaintMs: number;
coldAppMountedMs: number;
coldLargestPaintMs: number;
coldPageReadyMs: number;
warmPageReadyMs: number;
scripts: number;
scriptKb: number;
runs: number;
}
interface CommitStamp {
sha: string;
subject: string;
}
type LoadTimeRow = Record<string, string | number>;
function buildRows(
conditions: Condition[],
{ sha, subject }: CommitStamp,
): LoadTimeRow[] {
return conditions.map((condition) => ({
Date: new Date().toISOString().slice(0, 10),
// Truncated the same way the bundle-size table does, so the two join.
Commit: sha.slice(0, 12),
// The stats table carries a free-text Description column. Populate it with
// the commit subject so the chart's points are self-describing.
Description: subject.split("\n")[0],
Network: condition.network,
"Network mbps": condition.networkMbps,
"Latency ms": condition.latencyMs,
CPU: condition.cpu,
"CPU throttle": condition.cpuThrottle,
"Cold ms": condition.coldMs,
"Warm ms": condition.warmMs,
"Steady ms": condition.steadyMs,
"Cold spread %": condition.coldSpreadPercent,
// The cold load broken up. A zero means the browser or the route never
// reported that one, rather than that it happened at time zero.
"Cold ttfb ms": condition.coldTtfbMs,
"Cold first paint ms": condition.coldFirstPaintMs,
"Cold app mounted ms": condition.coldAppMountedMs,
"Cold largest paint ms": condition.coldLargestPaintMs,
"Cold page ready ms": condition.coldPageReadyMs,
"Warm page ready ms": condition.warmPageReadyMs,
Scripts: condition.scripts,
"Script kb": condition.scriptKb,
Runs: condition.runs,
}));
}
async function main() {
const path = process.env.ROWS || "artifacts/load-times.json";
// matrix.js wrote this file, and Condition is the shape it prints.
const conditions = JSON.parse(readFileSync(path, "utf8")) as Condition[];
const rows = buildRows(conditions, {
sha: process.env.HEAD_SHA || "",
subject: process.env.COMMIT_MESSAGE || "",
});
console.table(rows);
try {
await importStats({ table: "bundle_load_times", rows });
console.log("Load times uploaded successfully");
} catch (error) {
// Stats logging is best-effort. A point lost to an unreachable importer is
// worth less than a red build on master, and the next commit plots another.
const message = error instanceof Error ? error.message : String(error);
console.log(
`::warning::Load-time upload failed after retries; leaving run green: ${message}`,
);
}
}
main();

6
.github/team.json vendored
View File

@@ -63,7 +63,8 @@
"crisptrutski",
"undrfined",
"andreis",
"prascuna"
"prascuna",
"krever"
]
},
{
@@ -112,7 +113,8 @@
"metatyb",
"olehk-mb",
"annatomka",
"brybacki"
"brybacki",
"BillyBobFry"
]
},
{

View File

@@ -1,5 +1,5 @@
name: AI Benchmark
run-name: "Evals · ${{ github.event_name == 'pull_request' && 'label' || 'manual' }} · ${{ github.event.pull_request.number || inputs.pr || inputs.ref || github.sha }}"
run-name: "Evals · ${{ github.event_name == 'pull_request' && github.event.label.name || format('manual · {0}', inputs.scope) }} · ${{ github.event.pull_request.number || inputs.pr || inputs.ref || github.sha }}"
on:
pull_request:
@@ -18,7 +18,16 @@ on:
description: "Scope"
required: true
type: choice
options: [smoke, full, "all (EXPENSIVE)"]
options:
- smoke
- full
- embarrassing-plain
- embarrassing-human
- embarrassing-all
- goldens-plain
- goldens-human
- goldens-all
- "all (EXPENSIVE)"
default: smoke
sample:
description: "Sample (0 = full suite)"
@@ -29,15 +38,15 @@ on:
default: main
concurrency:
group: evals-dispatch-${{ (github.event.label.name == 'evals/smoke' && github.event.pull_request.number) || github.run_id }}
group: evals-dispatch-${{ github.event_name == 'pull_request' && format('{0}-{1}', github.event.pull_request.number, github.event.label.name) || github.run_id }}
cancel-in-progress: true
jobs:
prepare:
if: >-
github.event.pull_request.head.repo.full_name == github.repository
&& github.event_name == 'pull_request'
&& github.event.label.name == 'evals/smoke'
(github.event_name == 'pull_request'
&& github.event.pull_request.head.repo.full_name == github.repository
&& startsWith(github.event.label.name, 'evals:'))
|| github.event_name == 'workflow_dispatch'
runs-on: ${{ vars.SLIM_RUNNER_KEY }}
timeout-minutes: 5
@@ -46,6 +55,7 @@ jobs:
ref: ${{ steps.prepare.outputs.ref }}
scope: ${{ steps.prepare.outputs.scope }}
sample: ${{ steps.prepare.outputs.sample }}
label: ${{ steps.prepare.outputs.label }}
evals_branch: ${{ steps.prepare.outputs.evals_branch }}
steps:
- name: Prepare
@@ -53,6 +63,7 @@ jobs:
env:
EVENT_NAME: ${{ github.event_name }}
LABEL_PR: ${{ github.event.pull_request.number }}
LABEL: ${{ github.event.label.name }}
INPUT_PR: ${{ inputs.pr }}
INPUT_REF: ${{ inputs.ref }}
INPUT_SCOPE: ${{ inputs.scope }}
@@ -62,14 +73,24 @@ jobs:
set -euo pipefail
# The two paths are mutually exclusive, and `on:` admits no other trigger.
if [[ "$EVENT_NAME" == 'pull_request' ]]; then
PR="$LABEL_PR"; REF=''; SCOPE='smoke'
SAMPLE='5'; EVALS_BRANCH='main'
SCOPE="${LABEL#evals:}"
case "$SCOPE" in
embarrassing-plain|embarrassing-human|embarrassing-all|goldens-plain|goldens-human|goldens-all)
SAMPLE='0' ;;
smoke|full)
SAMPLE='5' ;;
*)
echo "::error::'${LABEL}' is not a known evals trigger label; add its scope here and to the workflow's \`scope\` options, or remove the label"
exit 1 ;;
esac
PR="$LABEL_PR"; REF=''; EVALS_BRANCH='main'
else
PR="$INPUT_PR"; REF="$INPUT_REF"; SCOPE="$INPUT_SCOPE"
SAMPLE="${INPUT_SAMPLE:-5}"; EVALS_BRANCH="$INPUT_EVALS_BRANCH"
LABEL='' # nothing to consume: a manual dispatch was not triggered by a label
fi
# None of these is ever multi-line; a newline would forge extra output entries.
for v in PR REF SCOPE SAMPLE EVALS_BRANCH; do
for v in PR REF SCOPE SAMPLE LABEL EVALS_BRANCH; do
if [[ "${!v}" == *$'\n'* ]]; then
echo "::error::${v} must be a single line"
exit 1
@@ -80,6 +101,7 @@ jobs:
echo "ref=$REF"
echo "scope=$SCOPE"
echo "sample=$SAMPLE"
echo "label=$LABEL"
echo "evals_branch=$EVALS_BRANCH"
} >> "$GITHUB_OUTPUT"
@@ -93,18 +115,26 @@ jobs:
steps:
- name: Comment (building)
id: post
env:
SCOPE: ${{ needs.prepare.outputs.scope }}
uses: actions/github-script@v9
with:
github-token: ${{ secrets.METABASE_AUTOMATION_USER_TOKEN }}
script: |
const runUrl = `${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`;
const body = `🤖 **Evals smoke** — ⏳ building the image… ([logs](${runUrl}))`;
const prefix = `🤖 **Evals (${process.env.SCOPE})**`;
const body = `${prefix} — ⏳ building the image… ([logs](${runUrl}))`;
const { owner, repo } = context.repo;
const issue_number = context.payload.pull_request.number;
// Drop earlier verdicts first: they describe shas that are no longer head, and a stale
// green one reads as this PR passing. The prefix survives eval-run's rewrites.
// Drop this scope's earlier verdicts first: they describe shas that are no longer head,
// and a stale green one reads as this PR passing. Only this scope's, though — a sibling
// scope's comment is a live report on a run still going.
const existing = await github.paginate(github.rest.issues.listComments, { owner, repo, issue_number });
for (const c of existing.filter(c => c.body.startsWith('🤖 **Evals'))) {
// The second form is what the single `evals/smoke` trigger wrote before the labels were
// split per scope; nothing writes it any more, so this clause is only here to tidy PRs
// that predate the split and can go once none are open.
const stale = c => c.body.startsWith(prefix) || c.body.startsWith('🤖 **Evals smoke**');
for (const c of existing.filter(stale)) {
await github.rest.issues.deleteComment({ owner, repo, comment_id: c.id }).catch(() => {});
}
const { data } = await github.rest.issues.createComment({ owner, repo, issue_number, body });
@@ -178,8 +208,11 @@ jobs:
SHA: ${{ needs.resolve.outputs.sha }}
SCOPE: ${{ needs.prepare.outputs.scope }}
SAMPLE: ${{ needs.prepare.outputs.sample }}
LABEL: ${{ needs.prepare.outputs.label }}
COMMENT_ID: ${{ needs.announce.outputs.comment_id }}
run: |
# `label` is the one eval-run consumes once it has taken the run over; empty on a manual
# dispatch, where there is nothing to consume.
gh api repos/metabase/evals/actions/workflows/eval-run.yml/dispatches \
-f ref="$EVALS_BRANCH" \
-f "inputs[pr]=$PR" \
@@ -187,16 +220,19 @@ jobs:
-f "inputs[scope]=$SCOPE" \
-f "inputs[surfaces]=all" \
-f "inputs[sample]=$SAMPLE" \
-f "inputs[label]=$LABEL" \
-f "inputs[evals_branch]=$EVALS_BRANCH" \
-f "inputs[comment_id]=$COMMENT_ID"
finalize:
needs: [announce, resolve, image, dispatch]
needs: [prepare, announce, resolve, image, dispatch]
if: always() && needs.announce.result == 'success' && needs.dispatch.result != 'success'
runs-on: ${{ vars.SLIM_RUNNER_KEY }}
timeout-minutes: 5
env:
COMMENT_ID: ${{ needs.announce.outputs.comment_id }}
SCOPE: ${{ needs.prepare.outputs.scope }}
LABEL: ${{ needs.prepare.outputs.label }}
NEEDS_JSON: ${{ toJSON(needs) }}
steps:
- name: Comment (build/dispatch failed)
@@ -207,9 +243,10 @@ jobs:
const runUrl = `${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`;
const cancelled = Object.values(JSON.parse(process.env.NEEDS_JSON))
.some(j => j.result === 'cancelled');
const prefix = `🤖 **Evals (${process.env.SCOPE})**`;
const body = cancelled
? `🤖 **Evals** — ⚪ cancelled before evals started ([logs](${runUrl}))`
: `🤖 **Evals** — ❌ build or dispatch failed before evals started ([logs](${runUrl}))`;
? `${prefix} — ⚪ cancelled before evals started ([logs](${runUrl}))`
: `${prefix} — ❌ build or dispatch failed before evals started ([logs](${runUrl}))`;
// The replacement run deletes this comment on its way in, so a concurrency cancel can
// land here with nothing to patch. Losing the stamp is fine; failing the job isn't.
await github.rest.issues.updateComment({
@@ -222,5 +259,5 @@ jobs:
await github.rest.issues.removeLabel({
owner: context.repo.owner, repo: context.repo.repo,
issue_number: context.payload.pull_request.number,
name: 'evals/smoke',
name: process.env.LABEL,
}).catch(() => {});

View File

@@ -217,7 +217,6 @@ jobs:
test-args: >-
${{ matrix.job.test-args }}
:exclude-tags '[ ${{ matrix.job.exclude-tag }} ${{ env.__ADDITIONAL_EXCLUDED_TAG__ }}]'
quarantine-engine: ${{ vars.QUARANTINE_ENGINE }}
be-tests-mysql:
needs: app-db-matrix
@@ -311,7 +310,6 @@ jobs:
test-args: >-
${{ matrix.job.test-args }}
:exclude-tags '[ ${{ matrix.job.exclude-tag }} ${{ env.__ADDITIONAL_EXCLUDED_TAG__ }}]'
quarantine-engine: ${{ vars.QUARANTINE_ENGINE }}
be-tests-postgres:
needs: app-db-matrix
@@ -388,7 +386,6 @@ jobs:
test-args: >-
${{ matrix.job.test-args }}
:exclude-tags '[ ${{ matrix.job.exclude-tag }} ${{ env.__ADDITIONAL_EXCLUDED_TAG__ }}]'
quarantine-engine: ${{ vars.QUARANTINE_ENGINE }}
app-db-tests-result:
needs:

View File

@@ -1,15 +0,0 @@
name: Block Master Branch PRs
on:
pull_request:
types: [opened, synchronize]
jobs:
check-source-branch:
if: github.head_ref == 'master'
runs-on: ${{ vars.SLIM_RUNNER_KEY }}
timeout-minutes: 5
steps:
- name: Block master branch PRs
run: |
echo "PRs from \`master\` branches are not allowed, please use a new branch name for your PR."
exit 1

View File

@@ -4,25 +4,54 @@ name: Bundle Size Stats
# emitted JS) plus the bundle-stats and embedding-sdk artifacts, measures
# initial/total bundle sizes, and appends them to eng-stats-importer.
# Triggered by "Run tests", which is the workflow that builds the uberjar on a
# master merge.
on:
workflow_run:
workflows: ["Build Uberjar"]
workflows: ["Run tests"]
types: [completed]
# Master only: releases form non-linear lines whose deltas are meaningless,
# so their sizes are backfilled separately rather than logged here.
branches: [master]
workflow_call:
inputs:
commit:
description: "Commit SHA to measure."
type: string
required: false
default: ""
concurrency:
group: bundle-size-stats-${{ github.event.workflow_run.head_sha }}
group: bundle-size-stats-${{ inputs.commit || github.event.workflow_run.head_sha }}
cancel-in-progress: true
jobs:
# "Run tests" concludes `failure` on master more often than `success`, from
# jobs that have nothing to do with the jar, so its conclusion is not a usable
# gate. The artifact is one: the uberjar job publishes it, and a docs-only
# merge that skipped that job publishes none.
uberjar-built:
if: ${{ github.repository == 'metabase/metabase' }}
runs-on: ${{ vars.SLIM_RUNNER_KEY }}
timeout-minutes: 5
outputs:
found: ${{ steps.probe.outputs.fetch-hit }}
steps:
- uses: actions/checkout@v6
- id: probe
uses: ./.github/actions/fetch-artifact
with:
commit: ${{ inputs.commit || github.event.workflow_run.head_sha }}
edition: ee
download: false
bundle-size-stats:
if: ${{ github.event.workflow_run.conclusion == 'success' && github.repository == 'metabase/metabase' }}
needs: [uberjar-built]
if: ${{ needs.uberjar-built.outputs.found == 'true' }}
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
timeout-minutes: 15
env:
HEAD_SHA: ${{ github.event.workflow_run.head_sha }}
HEAD_SHA: ${{ inputs.commit || github.event.workflow_run.head_sha }}
# Only record a data point when a bundle's as-served size (brotli where the
# build ships .br, else gzip) moved by at least this percent vs the last
# plotted point, keeping the chart to real changes.
@@ -31,12 +60,12 @@ jobs:
- name: Check out the code
uses: actions/checkout@v6
with:
ref: ${{ github.event.workflow_run.head_sha }}
ref: ${{ env.HEAD_SHA }}
- uses: ./.github/actions/prepare-node-bun
- name: Download EE uberjar
uses: ./.github/actions/fetch-artifact
with:
commit: ${{ github.event.workflow_run.head_sha }}
commit: ${{ env.HEAD_SHA }}
edition: ee
extract-path: temp/uberjar
- name: Extract front-end bundles from uberjar
@@ -108,7 +137,7 @@ jobs:
run: bun ./.github/scripts/bundle-size-stats-prepare.ts
- name: Import sizes (with deltas) into Stats
id: upload
if: steps.prepare.outputs.significant == 'true'
if: ${{ steps.prepare.outputs.significant == 'true' && inputs.commit == '' }}
env:
ROWS: artifacts/upload-rows.json
ENG_STATS_URL: ${{ secrets.ENG_STATS_URL }}

View File

@@ -1,21 +0,0 @@
name: Prevent backport.sh from being merged
on:
pull_request:
types: [opened, synchronize, reopened]
jobs:
check-backport-script:
runs-on: ${{ vars.SLIM_RUNNER_KEY }}
timeout-minutes: 5
steps:
- name: Checkout code
uses: actions/checkout@v6
- name: Check for backport.sh file
run: |
if [ -f "./backport.sh" ]; then
echo "backport.sh file exists. Please remove it before merging."
exit 1
else
echo "All good."
fi

View File

@@ -1,5 +1,5 @@
name: Containerize Uberjar
run-name: Containerize ${{ inputs.tag }} Uberjar
run-name: Containerize ${{ inputs.edition }} ${{ inputs.tag }} Uberjar
on:
workflow_call:
@@ -9,9 +9,10 @@ on:
type: string
required: true
repo:
description: owner and repo like "metabase/metabase"
description: Docker owner and repo, e.g. "metabase/metabase".
type: string
tag:
description: Docker tag, e.g. v1.87.2.
type: string
commit:
type: string
@@ -30,10 +31,10 @@ on:
- oss
required: true
repo:
description: docker owner and repo like "metabase/metabase"
description: Docker owner and repo, e.g. "metabase/metabase".
type: string
tag:
description: usually a version like v1.87.2
description: Docker tag, e.g. v1.87.2.
type: string
commit:
description: the full commit hash
@@ -45,10 +46,14 @@ on:
jobs:
containerize:
name: containerize (${{ inputs.edition }})
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
timeout-minutes: 15
outputs:
platforms: ${{ steps.platforms.outputs.result }}
repo: ${{ steps.image.outputs.repo }}
tag: ${{ steps.image.outputs.tag }}
full_image: ${{ steps.image.outputs.full_image }}
steps:
- uses: actions/checkout@v6
with:
@@ -65,6 +70,13 @@ jobs:
release: ${{ inputs.release }}
- name: Prepare release scripts
uses: ./.github/actions/prepare-release-scripts
- name: Resolve image repo and tag
id: image
uses: ./.github/actions/get-docker-info
with:
repo: ${{ inputs.repo }}
tag: ${{ inputs.tag }}
edition: ${{ inputs.edition }}
- name: Get platforms
id: platforms
uses: actions/github-script@v9
@@ -72,7 +84,7 @@ jobs:
result-encoding: string
script: | # js
const { getBuildRequirements, isValidVersionString} = require('${{ github.workspace }}/release/dist/index.cjs');
const version = "${{ inputs.tag }}".split('-')[0];
const version = "${{ steps.image.outputs.tag }}".split('-')[0];
if (!isValidVersionString(version)) {
return "linux/amd64,linux/arm64";
@@ -100,7 +112,7 @@ jobs:
context: bin/docker/.
platforms: ${{ steps.platforms.outputs.result }}
network: host
tags: ${{ inputs.repo }}:${{ inputs.tag }}
tags: ${{ steps.image.outputs.full_image }}
no-cache: true
push: true
build-args: ${{ inputs.build-args }}
@@ -108,16 +120,26 @@ jobs:
# we need to do this to get the nice fallback behavior of the legacy manifest file
- name: Build container (x86 only)
if: contains(steps.platforms.outputs.result, 'arm') == false
env:
BUILD_ARGS: ${{ inputs.build-args }}
FULL_IMAGE: ${{ steps.image.outputs.full_image }}
run: | # bash
args=()
while IFS= read -r line; do
[ -n "$line" ] && args+=(--build-arg "$line")
done <<< "$BUILD_ARGS"
docker build \
-t ${{ inputs.repo }}:${{ inputs.tag }} \
-t "$FULL_IMAGE" \
"${args[@]}" \
--network=host \
--push \
bin/docker/.
verify-docker-pull:
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
needs: containerize
name: verify-docker-pull (${{ inputs.edition }})
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
timeout-minutes: 15
steps:
- name: Login to Docker Hub
@@ -126,18 +148,19 @@ jobs:
username: ${{ inputs.release && secrets.DOCKERHUB_RELEASE_USERNAME || secrets.DOCKERHUB_USERNAME }}
password: ${{ inputs.release && secrets.DOCKERHUB_RELEASE_TOKEN || secrets.DOCKERHUB_TOKEN }}
- name: Pull the container image
run: docker pull ${{ inputs.repo }}:${{ inputs.tag }}
run: docker pull ${{ needs.containerize.outputs.full_image }}
- name: Launch container
run: docker run --rm -dp 3000:3000 ${{ inputs.repo }}:${{ inputs.tag }}
run: docker run --rm -dp 3000:3000 ${{ needs.containerize.outputs.full_image }}
timeout-minutes: 5
- name: Wait for Metabase to start
run: while ! curl -s 'http://localhost:3000/api/health' | grep '{"status":"ok"}'; do sleep 1; done
timeout-minutes: 5
verify-docker-pull-arm:
runs-on: arm-ubuntu-24-2core
needs: containerize
if: contains(needs.containerize.outputs.platforms, 'linux/arm64')
name: verify-docker-pull-arm (${{ inputs.edition }})
runs-on: arm-ubuntu-24-2core
timeout-minutes: 15
steps:
- name: Login to Docker Hub
@@ -146,10 +169,41 @@ jobs:
username: ${{ inputs.release && secrets.DOCKERHUB_RELEASE_USERNAME || secrets.DOCKERHUB_USERNAME }}
password: ${{ inputs.release && secrets.DOCKERHUB_RELEASE_TOKEN || secrets.DOCKERHUB_TOKEN }}
- name: Pull the container image
run: docker pull ${{ inputs.repo }}:${{ inputs.tag }}
run: docker pull ${{ needs.containerize.outputs.full_image }}
- name: Launch container
run: docker run --rm -dp 3000:3000 ${{ inputs.repo }}:${{ inputs.tag }}
run: docker run --rm -dp 3000:3000 ${{ needs.containerize.outputs.full_image }}
timeout-minutes: 5
- name: Wait for Metabase to start
run: while ! curl -s 'http://localhost:3000/api/health' | grep '{"status":"ok"}'; do sleep 1; done
timeout-minutes: 5
post-docker-info-comment:
needs: [containerize, verify-docker-pull, verify-docker-pull-arm]
# Only post a comment if all the steps pass and this is a PR.
if: |
!cancelled() &&
!failure() &&
!inputs.release &&
github.event_name == 'pull_request'
name: post-docker-info-comment (${{ inputs.edition }})
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
timeout-minutes: 5
steps:
- name: Post PR comment
uses: marocchino/sticky-pull-request-comment@0ea0beb66eb9baf113663a64ec522f60e49231c0 # v3.0.4
with:
header: docker-build-status-${{ inputs.edition }}
message: |
✅ Docker build completed for commit ${{ inputs.commit || github.event.pull_request.head.sha || github.sha }}
The uberjar has been built and containerized successfully.
**Branch:** `${{ github.head_ref || github.ref_name }}`
**Docker Image:** `${{ needs.containerize.outputs.full_image }}`
**Docker Hub:** [View on Docker Hub](https://hub.docker.com/r/${{ needs.containerize.outputs.repo }}/tags?name=${{ needs.containerize.outputs.tag }})
**Triggered by:** @${{ github.actor }}
Pull the image:
```bash
docker pull ${{ needs.containerize.outputs.full_image }}
```

View File

@@ -1,102 +0,0 @@
name: Containerize Uberjar
on:
workflow_call:
workflow_dispatch:
inputs:
commit:
description: the full commit hash to containerize (defaults to the triggering commit)
type: string
jobs:
get-container-info:
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
timeout-minutes: 5
name: Get container info
outputs:
editions: ${{ steps.ee-info.outputs.editions }}
ee: ${{ toJson(fromJson(steps.build-matrix.outputs.result).ee) }}
oss: ${{ toJson(fromJson(steps.build-matrix.outputs.result).oss) }}
steps:
- name: Check out the code
uses: actions/checkout@v6
- name: Get EE Docker info
id: ee-info
uses: ./.github/actions/get-docker-info
with:
edition: ee
- name: Get OSS Docker info
id: oss-info
uses: ./.github/actions/get-docker-info
with:
edition: oss
- name: Build matrix info
id: build-matrix
uses: actions/github-script@v9
with:
script: | # js
const editions = JSON.parse('${{ steps.ee-info.outputs.editions }}');
const result = {
editions: editions,
ee: {
repo: '${{ steps.ee-info.outputs.repo }}',
tag: '${{ steps.ee-info.outputs.tag }}'
},
oss: {
repo: '${{ steps.oss-info.outputs.repo }}',
tag: '${{ steps.oss-info.outputs.tag }}'
}
};
console.log('Build matrix:', result);
return result;
containerize:
needs: [get-container-info]
if: needs.get-container-info.result == 'success'
strategy:
matrix:
edition: ${{ fromJson(needs.get-container-info.outputs.editions) }}
uses: ./.github/workflows/containerize-jar.yml
secrets: inherit
with:
edition: ${{ matrix.edition }}
commit: ${{ github.event.inputs.commit || github.event.pull_request.head.sha || github.sha }}
repo: ${{ fromJson(needs.get-container-info.outputs[matrix.edition]).repo }}
tag: ${{ fromJson(needs.get-container-info.outputs[matrix.edition]).tag }}
release: false
build-args: |
GIT_COMMIT_SHA=${{ github.event.inputs.commit || github.event.pull_request.head.sha || github.sha }}
post-comment:
needs: [get-container-info, containerize]
if: github.event_name == 'pull_request'
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
name: Post PR comment
steps:
- name: Check out the code
uses: actions/checkout@v6
- name: Get Docker info for comment
id: docker-info
uses: ./.github/actions/get-docker-info
with:
branch: ${{ github.head_ref || github.ref_name }}
edition: ee
- name: Post PR comment
uses: marocchino/sticky-pull-request-comment@0ea0beb66eb9baf113663a64ec522f60e49231c0 # v3.0.4
with:
header: docker-build-status
message: |
✅ Docker build completed for commit ${{ github.event.pull_request.head.sha || github.sha }}
The uberjar has been built and containerized successfully.
**Branch:** `${{ github.head_ref || github.ref_name }}`
**Docker Image:** `${{ steps.docker-info.outputs.full_image }}`
**Docker Hub:** [View on Docker Hub](https://hub.docker.com/r/${{ steps.docker-info.outputs.repo }}/tags?name=${{ steps.docker-info.outputs.tag }})
**Triggered by:** @${{ github.actor }}
Pull the image:
```bash
docker pull ${{ steps.docker-info.outputs.full_image }}
```

View File

@@ -77,6 +77,9 @@ jobs:
- name: Generate AI provider documentation
id: generate_ai_provider_docs
# The generator only exists on branches that ship the multi-provider registry (master / 64+), but
# this reusable workflow always runs from master, so skip the step on release branches without it.
if: hashFiles('src/metabase/cmd/ai_provider_dox.clj') != ''
run: |
set -e # Exit on any error
echo "Generating AI provider documentation..."

View File

@@ -32,3 +32,9 @@ jobs:
fail: true
args: docs --config ./.lychee/config.toml --offline
format: markdown
# lychee can't read the YAML nav, so this script renders it as markdown first. lychee is on PATH
# from the step above.
- name: Check nav.yml links
if: ${{ !cancelled() }}
run: docs/util/check-links.sh nav

View File

@@ -74,6 +74,7 @@ jobs:
# All decision logic is consolidated in mage driver-decisions.
determine-driver-skips:
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
timeout-minutes: 10
outputs:
# All driver run decisions (from mage driver-decisions)
h2-should-run: ${{ steps.driver-decisions.outputs.h2-should-run }}
@@ -162,35 +163,16 @@ jobs:
test-args: >-
:only-tags [:mb/driver-tests]
:exclude-tags [:mb/transforms-python-test]
quarantine-engine: ${{ vars.QUARANTINE_ENGINE }}
be-tests-bigquery-cloud-sdk-ee:
needs: determine-driver-skips
if: ${{ needs.determine-driver-skips.outputs.bigquery-should-run == 'true' }}
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
timeout-minutes: 70
timeout-minutes: 40
strategy:
fail-fast: false
matrix:
job:
- name: Driver Tests Part 1
test-args: >-
:only-tags [:mb/driver-tests]
:exclude-tags [:mb/transforms-python-test]
:fail-fast? true
:partition/total 2
:partition/index 0
- name: Driver Tests Part 2
test-args: >-
:only-tags [:mb/driver-tests]
:exclude-tags [:mb/transforms-python-test]
:fail-fast? true
:partition/total 2
:partition/index 1
- name: Transforms Python Tests
test-args: >-
:only-tags [:mb/transforms-python-test]
needs-python-runner: true
runner-index: [0, 1]
env:
CI: 'true'
DRIVERS: bigquery-cloud-sdk
@@ -201,11 +183,43 @@ jobs:
MB_BIGQUERY_TEST_REFRESH_TOKEN: ${{ secrets.MB_BIGQUERY_TEST_REFRESH_TOKEN }}
MB_BIGQUERY_CLOUD_SDK_TEST_SERVICE_ACCOUNT_JSON: ${{ secrets.MB_BIGQUERY_CLOUD_SDK_TEST_SERVICE_ACCOUNT_JSON }}
MB_BIGQUERY_CLOUD_SDK_TEST_SERVICE_ACCOUNT_JSON_ROUTING: ${{ secrets.MB_BIGQUERY_CLOUD_SDK_TEST_SERVICE_ACCOUNT_JSON_ROUTING }}
name: BigQuery ${{ matrix.job.name }}
name: BigQuery Driver Tests ${{ matrix.runner-index }}
steps:
- uses: actions/checkout@v6
- name: Test BigQuery Cloud SDK driver
id: test-driver
uses: ./.github/actions/test-driver
with:
junit-name: 'be-tests-bigquery-cloud-sdk-ee'
test-args: >-
:only-tags [:mb/driver-tests]
:exclude-tags [:mb/transforms-python-test]
:fail-fast? true
runner-index: ${{ matrix.runner-index }}
runner-count: ${{ strategy.job-total }}
jvms-per-runner: '2'
heap-per-jvm: '4g'
artifact-suffix: runner-${{ matrix.runner-index }}
be-tests-bigquery-transforms-python-ee:
needs: determine-driver-skips
if: ${{ needs.determine-driver-skips.outputs.bigquery-should-run == 'true' }}
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
timeout-minutes: 30
env:
CI: 'true'
DRIVERS: bigquery-cloud-sdk
MB_BIGQUERY_TEST_PROJECT_ID: ${{ secrets.BIGQUERY_TEST_PROJECT_ID }}
MB_BIGQUERY_TEST_CLIENT_ID: ${{ secrets.MB_BIGQUERY_TEST_CLIENT_ID }}
MB_BIGQUERY_TEST_CLIENT_SECRET: ${{ secrets.MB_BIGQUERY_TEST_CLIENT_SECRET }}
MB_BIGQUERY_TEST_ACCESS_TOKEN: ${{ secrets.MB_BIGQUERY_TEST_ACCESS_TOKEN }}
MB_BIGQUERY_TEST_REFRESH_TOKEN: ${{ secrets.MB_BIGQUERY_TEST_REFRESH_TOKEN }}
MB_BIGQUERY_CLOUD_SDK_TEST_SERVICE_ACCOUNT_JSON: ${{ secrets.MB_BIGQUERY_CLOUD_SDK_TEST_SERVICE_ACCOUNT_JSON }}
MB_BIGQUERY_CLOUD_SDK_TEST_SERVICE_ACCOUNT_JSON_ROUTING: ${{ secrets.MB_BIGQUERY_CLOUD_SDK_TEST_SERVICE_ACCOUNT_JSON_ROUTING }}
name: BigQuery Transforms Python Tests
steps:
- uses: actions/checkout@v6
- name: Setup python-runner
if: ${{ matrix.job.needs-python-runner == true }}
uses: ./.github/actions/setup-python-runner
with:
metabase-automation-token: ${{ secrets.METABASE_AUTOMATION_USER_TOKEN }}
@@ -213,16 +227,17 @@ jobs:
id: test-driver
uses: ./.github/actions/test-driver
with:
# This is the same as the cloud-sdk job above to preserve quarantine history in ci-conductor.
junit-name: 'be-tests-bigquery-cloud-sdk-ee'
test-args: ${{ matrix.job.test-args }}
artifact-suffix: ${{ matrix.job.name }}
quarantine-engine: ${{ vars.QUARANTINE_ENGINE }}
test-args: >-
:only-tags [:mb/transforms-python-test]
artifact-suffix: transforms-python
- name: Cleanup python-runner
if: ${{ always() && matrix.job.needs-python-runner == true }}
if: ${{ always() }}
uses: ./.github/actions/cleanup-python-runner
with:
upload-logs: ${{ steps.test-driver.outcome == 'failure' }}
logs-artifact-suffix: ${{ matrix.job.name }}
logs-artifact-suffix: transforms-python
be-tests-clickhouse-ee:
needs: determine-driver-skips
@@ -279,7 +294,6 @@ jobs:
junit-name: 'be-tests-clickhouse-ee'
test-args: ${{ matrix.job.test-args }}
artifact-suffix: ${{ matrix.job.name }}
quarantine-engine: ${{ vars.QUARANTINE_ENGINE }}
- name: Cleanup python-runner
if: ${{ always() && matrix.job.needs-python-runner == true }}
uses: ./.github/actions/cleanup-python-runner
@@ -313,7 +327,6 @@ jobs:
test-args: >-
:only-tags [:mb/driver-tests]
:exclude-tags [:mb/transforms-python-test]
quarantine-engine: ${{ vars.QUARANTINE_ENGINE }}
be-tests-databricks-ee:
needs: determine-driver-skips
@@ -376,7 +389,6 @@ jobs:
junit-name: 'be-tests-databricks-ee'
artifact-suffix: ${{ matrix.job.name }}
test-args: ${{ matrix.job.test-args }}
quarantine-engine: ${{ vars.QUARANTINE_ENGINE }}
be-tests-h2:
needs: determine-driver-skips
@@ -396,7 +408,6 @@ jobs:
test-args: >-
:only-tags [:mb/driver-tests]
:exclude-tags [:mb/transforms-python-test]
quarantine-engine: ${{ vars.QUARANTINE_ENGINE }}
be-tests-h2-oss:
needs: determine-driver-skips
@@ -417,7 +428,6 @@ jobs:
test-args: >-
:only-tags [:mb/driver-tests]
:exclude-tags [:mb/transforms-python-test]
quarantine-engine: ${{ vars.QUARANTINE_ENGINE }}
be-tests-mysql-mariadb:
needs: determine-driver-skips
@@ -536,7 +546,6 @@ jobs:
test-args: >-
${{ matrix.job.test-args }}
:exclude-tags '[ ${{ matrix.job.exclude-tag }} ${{ env.__ADDITIONAL_EXCLUDED_TAG__ }}]'
quarantine-engine: ${{ vars.QUARANTINE_ENGINE }}
- name: Cleanup python-runner
if: always()
uses: ./.github/actions/cleanup-python-runner
@@ -596,7 +605,6 @@ jobs:
junit-name: ${{ matrix.version.junit-name }}
test-args: ${{ matrix.job.test-args }}
artifact-suffix: ${{ matrix.version.name }}-${{ matrix.job.name }}
quarantine-engine: ${{ vars.QUARANTINE_ENGINE }}
- name: Cleanup python-runner
if: ${{ always() && matrix.job.needs-python-runner == true }}
uses: ./.github/actions/cleanup-python-runner
@@ -658,7 +666,6 @@ jobs:
test-args: >-
:exclude-tags '[:mongo-sharded-cluster-tests :mb/transforms-python-test]'
:only-tags [:mb/driver-tests]
quarantine-engine: ${{ vars.QUARANTINE_ENGINE }}
be-tests-mongo-sharded-cluster-ee:
needs: determine-driver-skips
@@ -683,7 +690,6 @@ jobs:
junit-name: 'be-tests-mongo-sharded-cluster-ee'
test-args: >-
:only metabase.driver.mongo.sharded-cluster-test
quarantine-engine: ${{ vars.QUARANTINE_ENGINE }}
be-tests-oracle:
needs: determine-driver-skips
@@ -748,7 +754,6 @@ jobs:
test-args: >-
:only-tags [:mb/driver-tests]
:exclude-tags [:mb/transforms-python-test]
quarantine-engine: ${{ vars.QUARANTINE_ENGINE }}
be-tests-postgres:
needs: determine-driver-skips
@@ -841,7 +846,6 @@ jobs:
test-args: >-
${{ matrix.job.test-args }}
:exclude-tags '[ ${{ matrix.job.exclude-tag }} ${{ env.__ADDITIONAL_EXCLUDED_TAG__ }}]'
quarantine-engine: ${{ vars.QUARANTINE_ENGINE }}
- name: Cleanup python-runner
if: always()
uses: ./.github/actions/cleanup-python-runner
@@ -908,7 +912,6 @@ jobs:
test-args: >-
:only-tags [:mb/driver-tests]
:exclude-tags [:mb/transforms-python-test]
quarantine-engine: ${{ vars.QUARANTINE_ENGINE }}
be-tests-redshift-ee:
needs: determine-driver-skips
@@ -999,7 +1002,6 @@ jobs:
test-args: >-
${{ matrix.job.test-args }}
artifact-suffix: ${{ matrix.job.name }}
quarantine-engine: ${{ vars.QUARANTINE_ENGINE }}
- name: Cleanup python-runner
if: ${{ always() && matrix.job.needs-python-runner == true }}
uses: ./.github/actions/cleanup-python-runner
@@ -1062,7 +1064,6 @@ jobs:
junit-name: 'be-tests-snowflake-ee'
test-args: ${{ matrix.job.test-args }}
artifact-suffix: ${{ matrix.job.name }}
quarantine-engine: ${{ vars.QUARANTINE_ENGINE }}
- name: Cleanup python-runner
if: ${{ always() && matrix.job.needs-python-runner == true }}
uses: ./.github/actions/cleanup-python-runner
@@ -1094,7 +1095,6 @@ jobs:
test-args: >-
:only-tags [:mb/driver-tests]
:exclude-tags [:mb/transforms-python-test]
quarantine-engine: ${{ vars.QUARANTINE_ENGINE }}
be-tests-sqlite-ee:
needs: determine-driver-skips
@@ -1115,7 +1115,6 @@ jobs:
test-args: >-
:only-tags [:mb/driver-tests]
:exclude-tags [:mb/transforms-python-test]
quarantine-engine: ${{ vars.QUARANTINE_ENGINE }}
be-tests-sqlserver:
needs: determine-driver-skips
@@ -1171,7 +1170,6 @@ jobs:
junit-name: ${{ matrix.version.junit-name }}
test-args: ${{ matrix.job.test-args }}
artifact-suffix: ${{ matrix.version.name }}-${{ matrix.job.name }}
quarantine-engine: ${{ vars.QUARANTINE_ENGINE }}
- name: Cleanup python-runner
if: ${{ always() && matrix.job.needs-python-runner == true }}
uses: ./.github/actions/cleanup-python-runner
@@ -1268,7 +1266,6 @@ jobs:
test-args: >-
:only-tags [:mb/driver-tests]
:exclude-tags [:mb/transforms-python-test]
quarantine-engine: ${{ vars.QUARANTINE_ENGINE }}
drivers-tests-result:
needs:
@@ -1276,6 +1273,7 @@ jobs:
# if you add a driver job, you must add it to this list for it to be a required check
- be-tests-athena-ee
- be-tests-bigquery-cloud-sdk-ee
- be-tests-bigquery-transforms-python-ee
- be-tests-clickhouse-ee
- be-tests-databricks-ee
- be-tests-druid-jdbc-ee

View File

@@ -64,6 +64,7 @@ jobs:
# workspace where the gate step runs; an absolute path anchored to the
# workspace makes both agree regardless of cwd.
QUARANTINE_FAILURES_FILE: ${{ github.workspace }}/target/quarantine-failures.jsonl
RERUN_ARTIFACT_BASE: failed-tests-${{ inputs.name }}-${{ inputs.edition }}
# Cypress records video for every spec but the passing specs' videos are
# deleted in after:spec so recording is ~5-10% slowdown per shard and almost
# never yields an artifact. Failure screenshots are unaffected, and the
@@ -119,17 +120,28 @@ jobs:
id: cypress-prep
uses: ./.github/actions/prepare-cypress
- name: Download failed test artifacts
uses: actions/download-artifact@v8
if: github.run_attempt != '1'
- name: Resolve previous attempt
if: ${{ github.run_attempt != '1' }}
run: echo "RERUN_PREV_ATTEMPT=$((GITHUB_RUN_ATTEMPT - 1))" >> "$GITHUB_ENV"
- name: Restore previously-failed specs
if: ${{ github.run_attempt != '1' }}
continue-on-error: true
id: failed-specs
uses: actions/download-artifact@v8
with:
name: failed-tests-${{ inputs.name }}-${{ inputs.edition }}
pattern: ${{ env.RERUN_ARTIFACT_BASE }}-attempt-${{ env.RERUN_PREV_ATTEMPT }}
path: target/rerun
- name: Get failed specs
if: github.run_attempt != '1'
run: echo "specs=$(cat failed-specs)" >> "$GITHUB_ENV"
if: ${{ github.run_attempt != '1' }}
run: |
if [ -s target/rerun/failed-specs ]; then
echo "Re-running only the specs that failed on attempt $RERUN_PREV_ATTEMPT:"
cat target/rerun/failed-specs
echo "specs=$(cat target/rerun/failed-specs)" >> "$GITHUB_ENV"
else
echo "No failed-spec list from attempt $RERUN_PREV_ATTEMPT; running the full suite."
fi
# run-tests.yml compiles this in parallel with the uberjar build. Downloading it is ~10x faster than compiling it here.
# Callers without that job (stress tests, coverage shards, cross-run re-runs) fall through to compiling as before.
@@ -232,7 +244,7 @@ jobs:
if: |
!cancelled() && (steps.split-e2e.outcome == 'failure' || steps.tagged-e2e.outcome == 'failure')
with:
name: cypress-recording-${{ inputs.name }}-${{ inputs.edition }}
name: cypress-recording-${{ inputs.name }}-${{ inputs.edition }}-attempt-${{ github.run_attempt }}
path: |
./cypress
./logs/test.log
@@ -242,8 +254,9 @@ jobs:
uses: actions/upload-artifact@v7
if: |
!cancelled() && (steps.split-e2e.outcome == 'failure' || steps.tagged-e2e.outcome == 'failure')
continue-on-error: true
with:
name: failed-tests-${{ inputs.name }}-${{ inputs.edition }}
name: ${{ env.RERUN_ARTIFACT_BASE }}-attempt-${{ github.run_attempt }}
path: ./cypress/test-results
if-no-files-found: ignore

View File

@@ -26,6 +26,9 @@ jobs:
fail-fast: false
env:
CI_CONDUCTOR_TEST_SUITE: embedding-sdk-component-react-${{ matrix.react-version }}
# Base name of this leg's granular-rerun artifact. Each matrix leg needs its own, or a
# leg would restore the other leg's failed set. `-attempt-N` is appended below.
RERUN_ARTIFACT_BASE: failed-specs-embedding-sdk-component-react-${{ matrix.react-version }}
# Sensitive env vars (tokens, secrets) starting with `CYPRESS_` are accessed via async `cy.env()`
# Example: `cy.env(["FOO"]).then(({ FOO }) => ...)`
# For non-sensitive config, use `Cypress.expose()` in cypress config instead (synchronous)
@@ -73,6 +76,30 @@ jobs:
id: cypress-prep
uses: ./.github/actions/prepare-cypress
- name: Resolve previous attempt
if: ${{ github.run_attempt != '1' }}
run: echo "RERUN_PREV_ATTEMPT=$((GITHUB_RUN_ATTEMPT - 1))" >> "$GITHUB_ENV"
- name: Restore previously-failed specs
if: ${{ github.run_attempt != '1' }}
continue-on-error: true
uses: actions/download-artifact@v8
with:
# use pattern instead of name since this should still pass if the artifact doesn't exist
pattern: ${{ env.RERUN_ARTIFACT_BASE }}-attempt-${{ env.RERUN_PREV_ATTEMPT }}
path: target/rerun
- name: Get failed specs
if: ${{ github.run_attempt != '1' }}
run: |
if [ -s target/rerun/failed-specs ]; then
echo "Re-running failed specs from attempt $RERUN_PREV_ATTEMPT:"
cat target/rerun/failed-specs
echo "specs=$(cat target/rerun/failed-specs)" >> "$GITHUB_ENV"
else
echo "No failed-spec list from attempt $RERUN_PREV_ATTEMPT; running the full suite."
fi
# esbuild resolves every import in the spec bundle's graph, even the ones tree shaking then drops.
# The cljs imports resolve into target/cljs_dev, so the compiled CLJS has to exist before Cypress runs.
# run-tests.yml compiles it in parallel with the uberjar build, so downloading is ~10x faster than compiling here.
@@ -107,9 +134,18 @@ jobs:
- name: Run component tests for Embedding SDK
id: run-tests
shell: bash
env:
CYPRESS_BROWSER: ${{ steps.cypress-prep.outputs.chrome-path }}
run: node e2e/runner/run_cypress_ci.js component
SPECS: ${{ env.specs }}
# record video for re-runs of previously failed specs
CYPRESS_VIDEO: ${{ env.specs && 'true' || 'false' }}
run: |
SPEC_ARGS=()
if [ -n "$SPECS" ]; then
SPEC_ARGS=(--spec "$SPECS")
fi
node e2e/runner/run_cypress_ci.js component "${SPEC_ARGS[@]}"
# Keep this step continue-on-error because the quarantine logic controls whether it ultimately fails.
continue-on-error: true
@@ -140,12 +176,21 @@ jobs:
uses: actions/upload-artifact@v7
if: ${{ !cancelled() && steps.run-tests.outcome == 'failure' }}
with:
name: cypress-recording-embedding-sdk-react-${{ matrix.react-version }}
name: cypress-recording-embedding-sdk-react-${{ matrix.react-version }}-attempt-${{ github.run_attempt }}
path: |
./cypress
./logs/test.log
if-no-files-found: ignore
- name: "SDK component tests > Upload failed specs for rerun"
uses: actions/upload-artifact@v7
if: ${{ !cancelled() && steps.run-tests.outcome == 'failure' }}
continue-on-error: true
with:
name: ${{ env.RERUN_ARTIFACT_BASE }}-attempt-${{ github.run_attempt }}
path: ./cypress/test-results
if-no-files-found: ignore
- name: "SDK component tests > Publish Summary"
if: ${{ !cancelled() && steps.run-tests.outcome == 'failure' }}
uses: actions/github-script@v9
@@ -195,6 +240,7 @@ jobs:
MB_WAREHOUSE_ALLOWED_NETWORKS: "allow-all"
METASTORE_DEV_SERVER_URL: ${{ secrets.METASTORE_DEV_SERVER_URL }}
CI_CONDUCTOR_TEST_SUITE: embedding-sdk-component-backward-compatibility
RERUN_ARTIFACT_BASE: failed-specs-embedding-sdk-component-backward-compatibility
# Sensitive env vars (tokens, secrets) starting with `CYPRESS_` are accessed via async `cy.env()`
# Example: `cy.env(["FOO"]).then(({ FOO }) => ...)`
# For non-sensitive config, use `Cypress.expose()` in cypress config instead (synchronous)
@@ -261,6 +307,30 @@ jobs:
id: cypress-prep
uses: ./.github/actions/prepare-cypress
- name: Resolve previous attempt
if: ${{ github.run_attempt != '1' }}
run: echo "RERUN_PREV_ATTEMPT=$((GITHUB_RUN_ATTEMPT - 1))" >> "$GITHUB_ENV"
- name: Restore previously-failed specs
if: ${{ github.run_attempt != '1' }}
continue-on-error: true
uses: actions/download-artifact@v8
with:
# use pattern instead of name since this should still pass if the artifact doesn't exist
pattern: ${{ env.RERUN_ARTIFACT_BASE }}-attempt-${{ env.RERUN_PREV_ATTEMPT }}
path: target/rerun
- name: Get failed specs
if: ${{ github.run_attempt != '1' }}
run: |
if [ -s target/rerun/failed-specs ]; then
echo "Re-running only the specs that failed on attempt $RERUN_PREV_ATTEMPT:"
cat target/rerun/failed-specs
echo "specs=$(cat target/rerun/failed-specs)" >> "$GITHUB_ENV"
else
echo "No failed-spec list from attempt $RERUN_PREV_ATTEMPT; running the full suite."
fi
# esbuild resolves every import in the spec bundle's graph, even the ones tree shaking then drops.
# The cljs imports resolve into target/cljs_dev, so the compiled CLJS has to exist before Cypress runs.
# The checked-out tree is the previous commit, which the current run's CLJS artifact was not built from,
@@ -283,11 +353,20 @@ jobs:
- name: Run component tests for Embedding SDK
id: run-tests
shell: bash
env:
CYPRESS_BROWSER: ${{ steps.cypress-prep.outputs.chrome-path }}
SPECS: ${{ env.specs }}
# record video for re-runs of previously failed specs
CYPRESS_VIDEO: ${{ env.specs && 'true' || 'false' }}
run: |
SPEC_ARGS=()
if [ -n "$SPECS" ]; then
SPEC_ARGS=(--spec "$SPECS")
fi
node e2e/runner/run_cypress_ci.js component \
--expose grepTags="-@skip-backward-compatibility"
--expose grepTags="-@skip-backward-compatibility" \
"${SPEC_ARGS[@]}"
# Keep this step continue-on-error because the quarantine logic controls whether it ultimately fails.
continue-on-error: true
@@ -318,12 +397,21 @@ jobs:
uses: actions/upload-artifact@v7
if: ${{ !cancelled() && steps.run-tests.outcome == 'failure' }}
with:
name: cypress-recording-embedding-sdk-cross-version
name: cypress-recording-embedding-sdk-cross-version-attempt-${{ github.run_attempt }}
path: |
./cypress
./logs/test.log
if-no-files-found: ignore
- name: Upload failed specs for rerun
uses: actions/upload-artifact@v7
if: ${{ !cancelled() && steps.run-tests.outcome == 'failure' }}
continue-on-error: true
with:
name: ${{ env.RERUN_ARTIFACT_BASE }}-attempt-${{ github.run_attempt }}
path: ./cypress/test-results
if-no-files-found: ignore
- name: Publish Summary
if: ${{ !cancelled() && steps.run-tests.outcome == 'failure' }}
uses: actions/github-script@v9

View File

@@ -203,9 +203,7 @@ jobs:
fail-fast: false
matrix:
environment: ["production", "development"]
# TEMPORARILY DISABLED: re-enable by removing the `false &&` below.
if: |
false &&
!cancelled() && needs.get-sample-apps-data.result == 'success' &&
(github.ref == 'refs/heads/master' || github.base_ref == 'master')
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
@@ -235,11 +233,6 @@ jobs:
sample-app-branch-name: ${{ needs.get-sample-apps-data.outputs.branch_name }}
sample-app-environment: ${{ matrix.environment }}
staging-mb-all-features-token: ${{ secrets.STAGING_MB_ALL_FEATURES_TOKEN }}
shoppy-iam-role: ${{ secrets.SHOPPY_IAM_ROLE }}
shoppy-metabase-app-db-s3-bucket: ${{ secrets.SHOPPY_METABASE_APP_DB_S3_BUCKET }}
shoppy-metabase-app-db-s3-key: ${{ secrets.SHOPPY_METABASE_APP_DB_S3_KEY }}
shoppy-metabase-app-db-user: ${{ secrets.SHOPPY_METABASE_APP_DB_USER }}
shoppy-metabase-app-db-database: ${{ secrets.SHOPPY_METABASE_APP_DB_DATABASE }}
metastore-dev-server-url: ${{ secrets.METASTORE_DEV_SERVER_URL }}
shoppy-datadog-application-id: ${{ secrets.SHOPPY_METABASE_APP_DATADOG_APPLICATION_ID }}
shoppy-datadog-client-token: ${{ secrets.SHOPPY_METABASE_APP_DATADOG_CLIENT_TOKEN }}
@@ -272,9 +265,7 @@ jobs:
# They use the shoppy production environment and include performance measurement tests.
e2e-shoppy-synthetic-tests:
needs: [get-sample-apps-data]
# TEMPORARILY DISABLED: re-enable by removing the `false &&` below.
if: |
false &&
!cancelled() && needs.get-sample-apps-data.result == 'success' &&
github.event_name == 'push' &&
github.ref == 'refs/heads/master'
@@ -305,11 +296,6 @@ jobs:
sample-app-branch-name: ${{ needs.get-sample-apps-data.outputs.branch_name }}
sample-app-environment: production
staging-mb-all-features-token: ${{ secrets.STAGING_MB_ALL_FEATURES_TOKEN }}
shoppy-iam-role: ${{ secrets.SHOPPY_IAM_ROLE }}
shoppy-metabase-app-db-s3-bucket: ${{ secrets.SHOPPY_METABASE_APP_DB_S3_BUCKET }}
shoppy-metabase-app-db-s3-key: ${{ secrets.SHOPPY_METABASE_APP_DB_S3_KEY }}
shoppy-metabase-app-db-user: ${{ secrets.SHOPPY_METABASE_APP_DB_USER }}
shoppy-metabase-app-db-database: ${{ secrets.SHOPPY_METABASE_APP_DB_DATABASE }}
metastore-dev-server-url: ${{ secrets.METASTORE_DEV_SERVER_URL }}
shoppy-datadog-application-id: ${{ secrets.SHOPPY_METABASE_APP_DATADOG_APPLICATION_ID }}
shoppy-datadog-client-token: ${{ secrets.SHOPPY_METABASE_APP_DATADOG_CLIENT_TOKEN }}
@@ -438,15 +424,38 @@ jobs:
if: steps.prebuilt-cljs.outcome != 'success'
run: bun run build-pure:cljs
# The data-app suite synchronizes real actions, and Metabase only permits
# those against a writable warehouse, which comes from the `qa-db`
# snapshot below. That snapshot creator builds every QA database in one
# pass — Mongo and MySQL included — so all three have to be up even though
# only `postgres-writable` is restored. The other host apps read the sample
# database, so they skip the pull entirely.
- name: Prepare Docker containers
if: ${{ matrix.test_suite == 'vite-6-data-app-host-app-e2e' }}
uses: ./.github/actions/e2e-prepare-containers
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
postgres: true
mysql: true
mongo: true
- name: Run Metabase
run: node e2e/runner/run_cypress_ci.js start
- uses: ./.github/actions/resolve-job-id
# Without `--spec` the snapshot config picks up `qa-db.cy.snap.js` too,
# which is what creates the `postgres-writable` snapshot the data-app
# action tests restore.
- name: Generate database snapshots
run: |
node e2e/runner/run_cypress_ci.js snapshot \
--spec "e2e/snapshot-creators/default.cy.snap.js"
if [ "${{ matrix.test_suite }}" = "vite-6-data-app-host-app-e2e" ]; then
node e2e/runner/run_cypress_ci.js snapshot
else
node e2e/runner/run_cypress_ci.js snapshot \
--spec "e2e/snapshot-creators/default.cy.snap.js"
fi
- name: Prepare and launch Host App
run: npx tsx ./e2e/runner/embedding-sdk/host-apps/start-ci.ts ${{ matrix.test_suite }}

View File

@@ -24,6 +24,8 @@ jobs:
# unrelated to embedding (EMB-765)
echo "::remove-matcher owner=tsc::"
bun run build-embedding-sdk-package
- name: Smoke test the packed SDK CLI
run: node bin/embedding-sdk/test-sdk-package.mjs
- name: Prepare Embedding SDK artifact
uses: actions/upload-artifact@v7
with:

View File

@@ -1,29 +1,68 @@
name: CI for PR Review ENV Destroy
run-name: Destroying Dynamic PR Environment for ${{ github.ref_name }} by @${{ github.actor }}
name: Cleanup PR Environment
run-name: Destroying Dynamic PR Environment for PR ${{ inputs.pr_number || github.event.number }} by @${{ github.actor }}
on:
workflow_call:
pull_request:
types: [ closed, unlabeled ]
workflow_dispatch:
inputs:
pr_number:
description: 'PR number to destroy environment for'
required: true
type: string
self_hosting:
description: 'Self-hosting stack'
required: false
type: boolean
default: false
pr_number:
description: 'PR number (required for manual dispatch)'
required: false
type: string
env:
PR_NUMBER: ${{ inputs.pr_number || github.event.number }}
concurrency:
group: ${{ github.workflow }}-${{ inputs.pr_number || github.event.number }}
cancel-in-progress: false
jobs:
destroy_pr:
# Manual runs go through the pr-env-admin environment for approval.
# Skipped for pull_request events, which are already scoped to the PR's own labels.
approve:
if: ${{ github.event_name == 'workflow_dispatch' }}
runs-on: ${{ vars.SLIM_RUNNER_KEY }}
timeout-minutes: 5
environment: pr-env-admin
steps:
- name: Approved
env:
PR_NUMBER: ${{ inputs.pr_number }}
run: |
echo "Destroying PR environment for PR #${PR_NUMBER}"
cleanup:
needs: [ approve ]
# Runs when dispatched manually (after approval), when a PR carrying a PR-Env
# label is closed, or when one of those labels is removed.
if: >-
${{ !cancelled()
&& (needs.approve.result == 'success' || needs.approve.result == 'skipped')
&& (github.event_name == 'workflow_dispatch'
|| (github.event.action == 'closed'
&& (contains(github.event.pull_request.labels.*.name, 'PR-Env')
|| contains(github.event.pull_request.labels.*.name, 'PR-Env-self-hosting')
|| contains(github.event.pull_request.labels.*.name, 'PR-Env-analytics-dev')
|| contains(github.event.pull_request.labels.*.name, 'PR-Env-Stats-Dump')))
|| (github.event.action == 'unlabeled'
&& (github.event.label.name == 'PR-Env'
|| github.event.label.name == 'PR-Env-self-hosting'
|| github.event.label.name == 'PR-Env-analytics-dev'
|| github.event.label.name == 'PR-Env-Stats-Dump'))) }}
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
name: PR Review ENV Destroy
timeout-minutes: 10
permissions:
id-token: write
contents: read
env:
PR_NUMBER: ${{ inputs.pr_number || github.event.number }}
SELF_HOSTING: >-
${{ (github.event_name == 'workflow_dispatch' && inputs.self_hosting)
|| (github.event.action == 'unlabeled' && github.event.label.name == 'PR-Env-self-hosting')
|| (github.event.action == 'closed' && contains(github.event.pull_request.labels.*.name, 'PR-Env-self-hosting')) }}
steps:
- name: Checkout source code
uses: actions/checkout@v6
@@ -69,17 +108,17 @@ jobs:
namespace: default
user: oidc-token
- name: Destroy PR Review ENV
if: ${{ !inputs.self_hosting }}
if: ${{ env.SELF_HOSTING != 'true' }}
run: |
kubectl delete metabase -n hosting-pr${{ env.PR_NUMBER }} hosting-pr${{ env.PR_NUMBER }}
kubectl delete ns hosting-pr${{ env.PR_NUMBER }}
- name: Setup Helm
if: ${{ inputs.self_hosting }}
if: ${{ env.SELF_HOSTING == 'true' }}
uses: azure/setup-helm@dda3372f752e03dde6b3237bc9431cdc2f7a02a2 # v5.0.0
with:
version: 'latest'
- name: Destroy PR Review ENV (self-hosting)
if: ${{ inputs.self_hosting }}
if: ${{ env.SELF_HOSTING == 'true' }}
run: |
helm uninstall hosting-pr${{ env.PR_NUMBER }} --namespace hosting-pr${{ env.PR_NUMBER }} || true
kubectl delete ns hosting-pr${{ env.PR_NUMBER }} || true

View File

@@ -1,9 +1,9 @@
name: Shrink ratchets
# Lowers the budgets in .clj-kondo/ratchets.edn to match master and delivers the result through one
# Lowers the ratchet budgets to match master and delivers the result through one
# automation PR, because the rulesets require a pull request for every change to master. The PR is
# approved by github-actions[bot] and merges itself once the ratchet check passes. A merge of that PR
# touches only ratchets.edn, which matches `.clj-kondo/**` below, so it triggers one more run that finds
# touches only ratchet files, which match `.clj-kondo/**` below, so it triggers one more run that finds
# nothing to lower and stops: a single no-op, not a loop.
on:
@@ -32,7 +32,8 @@ concurrency:
env:
AUTOMATION_BRANCH: automation/shrink-ratchets
RATCHETS_FILE: .clj-kondo/ratchets.edn
KONDO_RATCHETS_FILE: .clj-kondo/ratchets.edn
MODULE_RATCHETS_FILE: .clj-kondo/config/modules/ratchets.edn
jobs:
shrink:
@@ -56,9 +57,10 @@ jobs:
with:
distribution: temurin
java-version: 25
- name: Shrink ignore limits
- name: Shrink ratchets
run: |
cp "$RATCHETS_FILE" "$RUNNER_TEMP/ratchets-before.edn"
cp "$KONDO_RATCHETS_FILE" "$RUNNER_TEMP/kondo-ratchets-before.edn"
cp "$MODULE_RATCHETS_FILE" "$RUNNER_TEMP/module-ratchets-before.edn"
./bin/mage kondo-ratchets-shrink
# Two independent outputs: `push` says the branch needs a new commit, `pr` names the open automation
# PR if there is one. Approval and auto-merge key on `pr` alone, so a run that failed after opening
@@ -77,23 +79,23 @@ jobs:
--jq '.[0].number // empty')
echo "pr=$open_pr" >> "$GITHUB_OUTPUT"
if git diff --quiet -- "$RATCHETS_FILE"; then
if git diff --quiet -- "$KONDO_RATCHETS_FILE" "$MODULE_RATCHETS_FILE"; then
echo "Ratchet budgets already match master $(git rev-parse --short HEAD)"
[ -z "$open_pr" ] || echo "The open automation PR #$open_pr is re-approved below"
echo "push=false" >> "$GITHUB_OUTPUT"
exit 0
fi
# The ratchet file is the whole payload, so an open PR that already carries this result is left
# The ratchet files are the whole payload, so an open PR that already carries this result is left
# alone. Rebasing it onto every master push would restart its CI and churn the diff without
# changing what it delivers. Compare the open PR with the working tree, which contains the
# shrunk ratchets file produced above, to determine whether the PR would deliver the same file.
# shrunk ratchet files produced above, to determine whether the PR would deliver the same result.
#
# A conflicted PR is the exception: only a fresh commit clears it. Mergeability is computed
# lazily, so a pending UNKNOWN reads as not conflicted and the next run settles it.
if [ -n "$open_pr" ] \
&& git fetch --quiet --depth=1 origin "refs/heads/$AUTOMATION_BRANCH" 2>/dev/null \
&& git diff --quiet FETCH_HEAD -- "$RATCHETS_FILE" \
&& git diff --quiet FETCH_HEAD -- "$KONDO_RATCHETS_FILE" "$MODULE_RATCHETS_FILE" \
&& [ "$(gh pr view "$open_pr" --repo "$GITHUB_REPOSITORY" \
--json mergeable --jq .mergeable)" != CONFLICTING ]; then
echo "The automation PR #$open_pr already carries this result"
@@ -102,7 +104,7 @@ jobs:
exit 0
fi
git diff --stat -- "$RATCHETS_FILE"
git diff --stat -- "$KONDO_RATCHETS_FILE" "$MODULE_RATCHETS_FILE"
echo "push=true" >> "$GITHUB_OUTPUT"
- name: Update the automation PR
id: deliver
@@ -116,14 +118,16 @@ jobs:
OPEN_PR: ${{ steps.compare.outputs.pr }}
run: |
./bin/mage -kondo-ratchets-pr-body \
"$RUNNER_TEMP/ratchets-before.edn" \
"$RATCHETS_FILE" \
"$RUNNER_TEMP/kondo-ratchets-before.edn" \
"$KONDO_RATCHETS_FILE" \
"$RUNNER_TEMP/module-ratchets-before.edn" \
"$MODULE_RATCHETS_FILE" \
"${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}" \
> "$RUNNER_TEMP/ratchets-pr-body.md"
git config user.name github-actions
git config user.email github-actions@github.com
git add -- "$RATCHETS_FILE"
git add -- "$KONDO_RATCHETS_FILE" "$MODULE_RATCHETS_FILE"
git commit --quiet -m "Tighten ratchets"
git push --force --quiet \
"https://x-access-token:${GH_TOKEN}@github.com/${GITHUB_REPOSITORY}.git" \

View File

@@ -1,25 +0,0 @@
name: Add Issue References to PR
on:
pull_request:
types: [opened, edited, synchronize]
jobs:
add-issue-references:
runs-on: ${{ vars.SLIM_RUNNER_KEY }}
timeout-minutes: 5
steps:
- name: Checkout code
uses: actions/checkout@v6
- name: Run the link script
uses: actions/github-script@v9
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
LINEAR_API_KEY: ${{ secrets.LINEAR_API_KEY }}
PR_NUMBER: ${{ github.event.pull_request.number }}
REPO_OWNER: ${{ github.repository_owner }}
REPO_NAME: ${{ github.event.repository.name }}
with:
script: |
const { link_issues } = require('${{ github.workspace }}/.github/scripts/update-pr-with-issues.js');
await link_issues(github)

View File

@@ -36,6 +36,7 @@ jobs:
uses: github/codeql-action/analyze@v4
trivy:
timeout-minutes: 15
strategy:
fail-fast: false
matrix:
@@ -68,8 +69,157 @@ jobs:
sarif_file: trivy-results-${{ matrix.edition }}.sarif
category: trivy-${{ matrix.edition }}
# Generate pom.xml files for all Maven projects
snyk-generate-poms:
name: Generate pom.xml files
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
timeout-minutes: 20
steps:
- uses: actions/checkout@v6
- name: Prepare back-end environment
uses: ./.github/actions/prepare-backend
- name: Generate all pom.xml
run: .github/scripts/write-poms.sh
- name: Upload pom files
uses: actions/upload-artifact@v7
with:
name: pom-files
path: |
pom.xml
modules/drivers/*/pom.xml
# Yarn/NPM projects that don't need pom files
snyk-frontend-projects:
name: ${{ matrix.project.name }}
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
timeout-minutes: 20
strategy:
matrix:
project:
- name: Main Frontend (bun w/ yarn.lock)
file: yarn.lock
package-manager: yarn
category: main-frontend-bun
- name: Release Helper
file: release/yarn.lock
package-manager: yarn
category: release-helper
steps:
- uses: actions/checkout@v6
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2.2.0
with:
bun-version-file: 'package.json'
token: ${{ secrets.GITHUB_TOKEN }}
# Snyk doesn't support bun.lock yet, so we generate yarn.lock and scan that.
# Bun has a bug where it normalizes "< 3.0.0" to "< 3" in alias entries
# but not in dependency specs, causing Snyk to fail matching dependencies.
# The sed commands fix this inconsistency. Bun also names aliased entries
# after the aliased package instead of the alias, which the script fixes.
# Both continue on error to let Snyk report parsing issues directly rather
# than failing on the workaround.
- name: Generate yarn.lock
if: matrix.project.file == 'yarn.lock'
run: |
bun install --yarn
sed -i -E 's/< ([0-9]+)\.0\.0"/< \1"/g; s/< ([0-9]+)\.0"/< \1"/g' yarn.lock || true
bun ${{ github.workspace }}/.github/scripts/fix-yarn-lock-aliases.ts || true
- name: Generate release/yarn.lock
if: matrix.project.file == 'release/yarn.lock'
run: |
bun install --yarn
sed -i -E 's/< ([0-9]+)\.0\.0"/< \1"/g; s/< ([0-9]+)\.0"/< \1"/g' yarn.lock || true
bun ${{ github.workspace }}/.github/scripts/fix-yarn-lock-aliases.ts || true
working-directory: release
- uses: snyk/actions/setup@9adf32b1121593767fc3c057af55b55db032dc04 # v1.0.0
- name: Run snyk test
env:
SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}
continue-on-error: true
run: snyk test --org="metabase-AnD9iS4Wfu4eBYMsG2DZ5N" --remote-repo-url="https://github.com/metabase/metabase" --file=${{ matrix.project.file }} --package-manager=${{ matrix.project.package-manager }} --sarif-file-output=snyk-${{ matrix.project.category }}.sarif
- name: Upload results
uses: github/codeql-action/upload-sarif@v4
with:
sarif_file: snyk-${{ matrix.project.category }}.sarif
category: ${{ matrix.project.category }}
# Maven projects that need pom files
snyk-maven-projects:
needs: snyk-generate-poms
name: ${{ matrix.project.name }}
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
timeout-minutes: 20
strategy:
matrix:
project:
- name: Main Backend (maven)
file: pom.xml
category: main-backend-maven
- name: Driver - Athena
file: modules/drivers/athena/pom.xml
category: driver-athena
- name: Driver - BigQuery Cloud SDK
file: modules/drivers/bigquery-cloud-sdk/pom.xml
category: driver-bigquery-cloud-sdk
- name: Driver - ClickHouse
file: modules/drivers/clickhouse/pom.xml
category: driver-clickhouse
- name: Driver - Databricks
file: modules/drivers/databricks/pom.xml
category: driver-databricks
- name: Driver - Druid JDBC
file: modules/drivers/druid-jdbc/pom.xml
category: driver-druid-jdbc
- name: Driver - Hive-like
file: modules/drivers/hive-like/pom.xml
category: driver-hive-like
- name: Driver - MongoDB
file: modules/drivers/mongo/pom.xml
category: driver-mongo
- name: Driver - Oracle
file: modules/drivers/oracle/pom.xml
category: driver-oracle
- name: Driver - Presto JDBC
file: modules/drivers/presto-jdbc/pom.xml
category: driver-presto-jdbc
- name: Driver - Redshift
file: modules/drivers/redshift/pom.xml
category: driver-redshift
- name: Driver - Snowflake
file: modules/drivers/snowflake/pom.xml
category: driver-snowflake
- name: Driver - SparkSQL
file: modules/drivers/sparksql/pom.xml
category: driver-sparksql
- name: Driver - SQL Server
file: modules/drivers/sqlserver/pom.xml
category: driver-sqlserver
- name: Driver - Starburst
file: modules/drivers/starburst/pom.xml
category: driver-starburst
- name: Driver - Vertica
file: modules/drivers/vertica/pom.xml
category: driver-vertica
steps:
- uses: actions/checkout@v6
- uses: snyk/actions/setup@9adf32b1121593767fc3c057af55b55db032dc04 # v1.0.0
- name: Download pom files
uses: actions/download-artifact@v8
with:
name: pom-files
- name: Run snyk test
env:
SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}
continue-on-error: true
run: snyk test --org="metabase-AnD9iS4Wfu4eBYMsG2DZ5N" --remote-repo-url="https://github.com/metabase/metabase" --file=${{ matrix.project.file }} --package-manager=maven --sarif-file-output=snyk-${{ matrix.project.category }}.sarif
- name: Upload results
uses: github/codeql-action/upload-sarif@v4
with:
sarif_file: snyk-${{ matrix.project.category }}.sarif
category: ${{ matrix.project.category }}
notify:
needs: [codeql, trivy]
needs: [codeql, trivy, snyk-frontend-projects, snyk-maven-projects]
if: ${{ failure() && github.repository == 'metabase/metabase' }}
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
timeout-minutes: 5
@@ -94,6 +244,8 @@ jobs:
const scans = [
['CodeQL', '${{ needs.codeql.result }}'],
['Trivy', '${{ needs.trivy.result }}'],
['Snyk (frontend)', '${{ needs.snyk-frontend-projects.result }}'],
['Snyk (maven)', '${{ needs.snyk-maven-projects.result }}'],
];
const failed = scans.filter(([, result]) => result === 'failure').map(([name]) => name);

View File

@@ -1,38 +0,0 @@
name: PR Closed or Unlabeled
on:
pull_request:
types: [ closed, unlabeled ]
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number }}
cancel-in-progress: false
jobs:
pr-env:
if: ${{ (github.event.action == 'closed' && contains(github.event.pull_request.labels.*.name, 'PR-Env') && !contains(github.event.pull_request.labels.*.name, 'PR-Env-self-hosting') && !contains(github.event.pull_request.labels.*.name, 'PR-Env-analytics-dev') && !contains(github.event.pull_request.labels.*.name, 'PR-Env-Stats-Dump')) || github.event.label.name == 'PR-Env' }}
uses: ./.github/workflows/pr-env-destroy.yml
secrets: inherit
with:
self_hosting: false
pr-env-self-hosting:
if: ${{ (github.event.action == 'closed' && contains(github.event.pull_request.labels.*.name, 'PR-Env-self-hosting')) || github.event.label.name == 'PR-Env-self-hosting' }}
uses: ./.github/workflows/pr-env-destroy.yml
secrets: inherit
with:
self_hosting: true
pr-env-analytics-dev:
if: ${{ (github.event.action == 'closed' && contains(github.event.pull_request.labels.*.name, 'PR-Env-analytics-dev')) || github.event.label.name == 'PR-Env-analytics-dev' }}
uses: ./.github/workflows/pr-env-destroy.yml
secrets: inherit
with:
self_hosting: false
pr-env-stats-dump:
if: ${{ (github.event.action == 'closed' && contains(github.event.pull_request.labels.*.name, 'PR-Env-Stats-Dump')) || github.event.label.name == 'PR-Env-Stats-Dump' }}
uses: ./.github/workflows/pr-env-destroy.yml
secrets: inherit
with:
self_hosting: false

View File

@@ -1,33 +0,0 @@
name: PR Env Destroy (Manual)
on:
workflow_dispatch:
inputs:
pr_number:
description: 'PR number to destroy environment for'
required: true
type: string
self_hosting:
description: 'Self-hosting stack'
type: boolean
default: false
jobs:
approve:
runs-on: ${{ vars.SLIM_RUNNER_KEY }}
timeout-minutes: 5
environment: pr-env-admin
steps:
- name: Approved
env:
PR_NUMBER: ${{ inputs.pr_number }}
run: |
echo "Destroying PR environment for PR #${PR_NUMBER}"
destroy:
needs: approve
uses: ./.github/workflows/pr-env-destroy.yml
secrets: inherit
with:
self_hosting: ${{ inputs.self_hosting }}
pr_number: ${{ inputs.pr_number }}

52
.github/workflows/pr.checks.yml vendored Normal file
View File

@@ -0,0 +1,52 @@
# Helpful (and quick) checks that run on PRs.
name: PR Checks
on: pull_request
jobs:
checks:
runs-on: ${{ vars.SLIM_RUNNER_KEY }}
timeout-minutes: 10
permissions:
pull-requests: write
issues: read
steps:
- uses: actions/checkout@v6
with:
sparse-checkout: |
/.github/scripts/update-pr-with-issues.js
/backport.sh
sparse-checkout-cone-mode: false
# download repo structure without file contents, fetched on demand since we only need a few files
# NOTE: this gets slow if we start accessing more files, remove this filter in that case.
filter: blob:none
- parallel:
- name: Block master branch PRs (from forks)
if: github.head_ref == 'master'
run: |
echo "PRs from \`master\` branches are not allowed, please use a new branch name for your PR."
exit 1
- name: Link PR and issues
uses: actions/github-script@v9
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
LINEAR_API_KEY: ${{ secrets.LINEAR_API_KEY }}
PR_NUMBER: ${{ github.event.pull_request.number }}
REPO_OWNER: ${{ github.repository_owner }}
REPO_NAME: ${{ github.event.repository.name }}
with:
script: |
const { link_issues } = require('${{ github.workspace }}/.github/scripts/update-pr-with-issues.js');
await link_issues(github)
- name: Prevent backport.sh from being merged
run: |
if [ -f "./backport.sh" ]; then
echo "backport.sh file exists. Please remove it before merging."
exit 1
else
echo "All good."
fi

View File

@@ -1,45 +0,0 @@
name: Pull Request Reminders
on:
pull_request:
types: [opened]
branches:
- "master"
jobs:
external-reminder:
name: External PR Helper
if: ${{ github.event.pull_request.head.repo.full_name != github.repository }}
runs-on: ${{ vars.SLIM_RUNNER_KEY }}
timeout-minutes: 5
permissions:
pull-requests: write
issues: read
steps:
- uses: actions/github-script@v9
with:
script: | # js
const owner = context.repo.owner;
const repo = context.repo.repo;
const pullNumber = '${{ github.event.pull_request.number }}';
const author = '${{ github.event.pull_request.user.login }}';
await github.rest.issues.addLabels({
owner,
repo,
issue_number: pullNumber,
labels: ["External"],
});
const guideLink = "https://www.notion.so/metabase/Third-Party-PRs-External-contributions-5954061854424c1ea6391d98746646e0";
const message = `@${author} Thanks for your contribution! ❤️ We'll try to review it as soon as we can.\n\n
Reviewers, make sure to check out our [External PR Review Guide](${guideLink}) when reviewing.
`;
await github.rest.issues.createComment({
owner,
repo,
issue_number: pullNumber,
body: message,
});

View File

@@ -382,6 +382,9 @@ jobs:
- name: Generate SDK package.json in the build directory
run: bun run embedding-sdk:generate-package
- name: Smoke test the packed SDK CLI
run: node bin/embedding-sdk/test-sdk-package.mjs
- name: Upload built SDK package as artifact
uses: actions/upload-artifact@v7
with:

View File

@@ -0,0 +1,128 @@
name: Update E2E Timings
permissions:
contents: write
pull-requests: write
on:
workflow_dispatch:
inputs:
days_back:
description: "Number of days to look back for timing artifacts"
type: number
default: 7
required: false
schedule:
# Run weekly on Sundays at 6 AM UTC
- cron: "0 6 * * 0"
jobs:
collect-and-average-timings:
if: ${{ github.event_name != 'schedule' || github.repository == 'metabase/metabase' }}
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
timeout-minutes: 30
env:
DAYS_BACK: ${{ inputs.days_back || 7 }}
steps:
- name: Generate GitHub App Token
uses: actions/create-github-app-token@v1
id: app-token
with:
app-id: ${{ secrets.METABASE_BOT_APP_ID }}
private-key: ${{ secrets.METABASE_BOT_APP_PRIVATE_KEY }}
- uses: actions/checkout@v6
with:
token: ${{ steps.app-token.outputs.token }}
- name: Setup Node.js
uses: actions/setup-node@v6.4.0
with:
node-version: 22
- name: Collect and average timing data
id: collect-timings
uses: actions/github-script@v9
with:
script: |
const { collectAndAverageTimings } = require('${{ github.workspace }}/.github/scripts/average-e2e-timings.js');
const result = await collectAndAverageTimings({
github,
context,
token: '${{ steps.app-token.outputs.token }}',
daysBack: ${{ env.DAYS_BACK }},
existingTimingsPath: 'e2e/support/timings.json'
});
core.setOutput('artifact-count', result.artifactCount);
- name: Configure git and commit changes
id: commit
run: |
git config user.name "github-actions[bot]"
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
git add e2e/support/timings.json
# Check if anything changed before commit/push
if git diff --cached --quiet; then
echo "changed=false" >> "$GITHUB_OUTPUT"
echo "Timing data is unchanged; skipping the pull request"
exit 0
fi
DATE=$(date +%Y%m%d-%H%M%S)
BRANCH_NAME="update-e2e-timings-$DATE"
echo "BRANCH_NAME=$BRANCH_NAME" >> "$GITHUB_ENV"
echo "Branch name will be: $BRANCH_NAME"
git checkout -b "$BRANCH_NAME"
git commit -m "update e2e timings $DATE" --no-verify
git push -u origin "$BRANCH_NAME"
echo "changed=true" >> "$GITHUB_OUTPUT"
- name: Create Pull Request
id: create_pr
if: steps.commit.outputs.changed == 'true'
env:
GH_TOKEN: ${{ secrets.METABASE_AUTOMATION_USER_TOKEN }}
run: |
BODY=$(cat <<'EOF'
## Summary
This PR updates the E2E test timing data with averaged values from recent test runs.
## Details
- Collected data from the last **${{ env.DAYS_BACK }} days** of test runs
- Averaged timing data from **${{ steps.collect-timings.outputs.artifact-count }}** test runs
## Generated automatically
🤖 This PR was created automatically by the `update-e2e-timings` workflow.
The timing data helps cypress-split optimize test distribution across parallel jobs.
Co-Authored-By: GitHub Actions <actions@github.com>
EOF
)
URL=$(gh pr create \
--base master \
--head "$BRANCH_NAME" \
--title "[chore] Update E2E test timings with recent data" \
--body "$BODY" \
--label ".CI & Tests" \
--label "no-backport" \
--reviewer metabase/devexp)
PR_NUMBER=${URL##*/}
echo "pr_number=$PR_NUMBER" >> "$GITHUB_OUTPUT"
- name: Approve and Merge the PR
if: steps.commit.outputs.changed == 'true'
env:
# uses a different token than the token that created the PR so we can approve and merge it
GH_TOKEN: ${{ github.token }}
PR_NUMBER: ${{ steps.create_pr.outputs.pr_number }}
shell: bash
run: |
gh pr review --approve "$PR_NUMBER"
gh pr merge --squash --auto "$PR_NUMBER"

View File

@@ -148,6 +148,8 @@ jobs:
(needs.should-run.outputs.verdict == 'force-run' ||
needs.files-changed.outputs.no_source_code_changes != 'true')
uses: ./.github/workflows/uberjar.yml
with:
containerize: ${{ (github.event_name == 'push' && github.ref_name == 'master') || contains(github.event.pull_request.labels.*.name, 'build-docker-uberjar') }}
secrets: inherit
# Compile CLJS in parallel with the uberjar and pass it to each e2e group.
@@ -414,15 +416,3 @@ jobs:
self_hosting: false
analytics_dev_mode: false
stats_dump: true
containerize:
needs: [uberjar]
if: |
!cancelled() &&
needs.uberjar.result == 'success' &&
(
(github.event_name == 'push' && contains(fromJson('["master", "metabot-v3-main", "internal-tools"]'), github.ref_name)) ||
(github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'build-docker-uberjar'))
)
uses: ./.github/workflows/containerize-uberjar.yml
secrets: inherit

View File

@@ -21,7 +21,7 @@ jobs:
// Nightly patches cover the current supported major and the one
// before it. release-auto.yml releases one major per dispatch.
const currentRelease = Number('${{ vars.CURRENT_VERSION }}');
const majors = [currentRelease - 1, currentRelease];
const majors = [currentRelease];
await Promise.all(majors.map(version =>
github.rest.actions.createWorkflowDispatch({

View File

@@ -96,7 +96,6 @@ jobs:
test-args: >-
${{ matrix.job.test-args }}
:exclude-tags '[${{ matrix.job.exclude-tag }}]'
quarantine-engine: ${{ vars.QUARANTINE_ENGINE }}
semantic-search-tests-mariadb:
if: ${{ !inputs.skip && (inputs.suite == '' || inputs.suite == 'mariadb') }}
@@ -162,7 +161,6 @@ jobs:
test-args: >-
${{ matrix.job.test-args }}
:exclude-tags '[${{ matrix.job.exclude-tag }}]'
quarantine-engine: ${{ vars.QUARANTINE_ENGINE }}
semantic-search-tests-mysql:
if: ${{ !inputs.skip && (inputs.suite == '' || inputs.suite == 'mysql') }}
@@ -235,7 +233,6 @@ jobs:
test-args: >-
${{ matrix.job.test-args }}
:exclude-tags '[${{ matrix.job.exclude-tag }}]'
quarantine-engine: ${{ vars.QUARANTINE_ENGINE }}
semantic-search-tests-postgres:
if: ${{ !inputs.skip && (inputs.suite == '' || inputs.suite == 'postgres') }}
@@ -300,7 +297,6 @@ jobs:
test-args: >-
${{ matrix.job.test-args }}
:exclude-tags '[${{ matrix.job.exclude-tag }}]'
quarantine-engine: ${{ vars.QUARANTINE_ENGINE }}
# pgvector-on-the-app-db mode: the app db itself is a pgvector-capable Postgres and
# MB_PGVECTOR_DB_URL is deliberately unset. The dedicated-harness tests gate themselves off; the
@@ -355,7 +351,6 @@ jobs:
test-args: >-
${{ matrix.job.test-args }}
:exclude-tags '[${{ matrix.job.exclude-tag }}]'
quarantine-engine: ${{ vars.QUARANTINE_ENGINE }}
semantic-search-tests-result:
needs:

View File

@@ -1,165 +0,0 @@
name: "Snyk"
on:
push:
branches:
- "master"
- "release-**"
paths:
- "**/deps.edn"
- "**/package.json"
- ".github/workflows/snyk.yml"
- ".github/scripts/write-poms.sh"
- ".github/scripts/fix-yarn-lock-aliases.ts"
schedule:
- cron: "0 5 * * *"
jobs:
# Generate pom.xml files for all Maven projects
generate-poms:
if: ${{ github.event_name != 'schedule' || github.repository == 'metabase/metabase' }}
name: Generate pom.xml files
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
steps:
- uses: actions/checkout@v6
- name: Prepare back-end environment
uses: ./.github/actions/prepare-backend
- name: Generate all pom.xml
run: .github/scripts/write-poms.sh
- name: Upload pom files
uses: actions/upload-artifact@v7
with:
name: pom-files
path: |
pom.xml
modules/drivers/*/pom.xml
# Yarn/NPM projects that don't need pom files
frontend-projects:
if: ${{ github.event_name != 'schedule' || github.repository == 'metabase/metabase' }}
name: ${{ matrix.project.name }}
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
strategy:
matrix:
project:
- name: Main Frontend (bun w/ yarn.lock)
file: yarn.lock
package-manager: yarn
category: main-frontend-bun
- name: Release Helper
file: release/yarn.lock
package-manager: yarn
category: release-helper
steps:
- uses: actions/checkout@v6
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2.2.0
with:
bun-version-file: 'package.json'
token: ${{ secrets.GITHUB_TOKEN }}
# Snyk doesn't support bun.lock yet, so we generate yarn.lock and scan that.
# Bun has a bug where it normalizes "< 3.0.0" to "< 3" in alias entries
# but not in dependency specs, causing Snyk to fail matching dependencies.
# The sed commands fix this inconsistency. Bun also names aliased entries
# after the aliased package instead of the alias, which the script fixes.
# Both continue on error to let Snyk report parsing issues directly rather
# than failing on the workaround.
- name: Generate yarn.lock
if: matrix.project.file == 'yarn.lock'
run: |
bun install --yarn
sed -i -E 's/< ([0-9]+)\.0\.0"/< \1"/g; s/< ([0-9]+)\.0"/< \1"/g' yarn.lock || true
bun ${{ github.workspace }}/.github/scripts/fix-yarn-lock-aliases.ts || true
- name: Generate release/yarn.lock
if: matrix.project.file == 'release/yarn.lock'
run: |
bun install --yarn
sed -i -E 's/< ([0-9]+)\.0\.0"/< \1"/g; s/< ([0-9]+)\.0"/< \1"/g' yarn.lock || true
bun ${{ github.workspace }}/.github/scripts/fix-yarn-lock-aliases.ts || true
working-directory: release
- uses: snyk/actions/setup@9adf32b1121593767fc3c057af55b55db032dc04 # v1.0.0
- name: Run snyk test
env:
SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}
continue-on-error: true
run: snyk test --org="metabase-AnD9iS4Wfu4eBYMsG2DZ5N" --remote-repo-url="https://github.com/metabase/metabase" --file=${{ matrix.project.file }} --package-manager=${{ matrix.project.package-manager }} --sarif-file-output=snyk-${{ matrix.project.category }}.sarif
- name: Upload results
uses: github/codeql-action/upload-sarif@v3
with:
sarif_file: snyk-${{ matrix.project.category }}.sarif
category: ${{ matrix.project.category }}
# Maven projects that need pom files
maven-projects:
needs: generate-poms
if: ${{ github.event_name != 'schedule' || github.repository == 'metabase/metabase' }}
name: ${{ matrix.project.name }}
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
strategy:
matrix:
project:
- name: Main Backend (maven)
file: pom.xml
category: main-backend-maven
- name: Driver - Athena
file: modules/drivers/athena/pom.xml
category: driver-athena
- name: Driver - BigQuery Cloud SDK
file: modules/drivers/bigquery-cloud-sdk/pom.xml
category: driver-bigquery-cloud-sdk
- name: Driver - ClickHouse
file: modules/drivers/clickhouse/pom.xml
category: driver-clickhouse
- name: Driver - Databricks
file: modules/drivers/databricks/pom.xml
category: driver-databricks
- name: Driver - Druid JDBC
file: modules/drivers/druid-jdbc/pom.xml
category: driver-druid-jdbc
- name: Driver - Hive-like
file: modules/drivers/hive-like/pom.xml
category: driver-hive-like
- name: Driver - MongoDB
file: modules/drivers/mongo/pom.xml
category: driver-mongo
- name: Driver - Oracle
file: modules/drivers/oracle/pom.xml
category: driver-oracle
- name: Driver - Presto JDBC
file: modules/drivers/presto-jdbc/pom.xml
category: driver-presto-jdbc
- name: Driver - Redshift
file: modules/drivers/redshift/pom.xml
category: driver-redshift
- name: Driver - Snowflake
file: modules/drivers/snowflake/pom.xml
category: driver-snowflake
- name: Driver - SparkSQL
file: modules/drivers/sparksql/pom.xml
category: driver-sparksql
- name: Driver - SQL Server
file: modules/drivers/sqlserver/pom.xml
category: driver-sqlserver
- name: Driver - Starburst
file: modules/drivers/starburst/pom.xml
category: driver-starburst
- name: Driver - Vertica
file: modules/drivers/vertica/pom.xml
category: driver-vertica
steps:
- uses: actions/checkout@v6
- uses: snyk/actions/setup@9adf32b1121593767fc3c057af55b55db032dc04 # v1.0.0
- name: Download pom files
uses: actions/download-artifact@v8
with:
name: pom-files
- name: Run snyk test
env:
SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}
continue-on-error: true
run: snyk test --org="metabase-AnD9iS4Wfu4eBYMsG2DZ5N" --remote-repo-url="https://github.com/metabase/metabase" --file=${{ matrix.project.file }} --package-manager=maven --sarif-file-output=snyk-${{ matrix.project.category }}.sarif
- name: Upload results
uses: github/codeql-action/upload-sarif@v3
with:
sarif_file: snyk-${{ matrix.project.category }}.sarif
category: ${{ matrix.project.category }}

View File

@@ -0,0 +1,125 @@
name: Bundle Load Stats
# Reuses the uberjar built on every master merge: boots it, signs in, loads the
# app in a throttled headless Chrome across four network and CPU conditions, and
# appends the times to eng-stats-importer.
#
# The instance is blank on purpose. `index.html` is built from settings and the
# signed-in user alone, so seeding content would not change a single byte of the
# document this measures.
# Triggered by "Run tests", which is the workflow that builds the uberjar on a
# master merge.
on:
workflow_run:
workflows: ["Run tests"]
types: [completed]
# Master only: a release forms a non-linear line whose times cannot be
# compared against the commit before it.
branches: [master]
workflow_call:
inputs:
commit:
description: "Commit SHA to measure."
type: string
required: false
default: ""
concurrency:
group: bundle-load-stats-${{ inputs.commit || github.event.workflow_run.head_sha }}
cancel-in-progress: true
jobs:
# "Run tests" concludes `failure` on master more often than `success`, from
# jobs that have nothing to do with the jar, so its conclusion is not a usable
# gate. The artifact is one: the uberjar job publishes it, and a docs-only
# merge that skipped that job publishes none.
uberjar-built:
if: ${{ github.repository == 'metabase/metabase' }}
runs-on: ${{ vars.SLIM_RUNNER_KEY }}
timeout-minutes: 5
outputs:
found: ${{ steps.probe.outputs.fetch-hit }}
steps:
- uses: actions/checkout@v6
- id: probe
uses: ./.github/actions/fetch-artifact
with:
commit: ${{ inputs.commit || github.event.workflow_run.head_sha }}
edition: ee
download: false
bundle-load-stats:
needs: [uberjar-built]
if: ${{ needs.uberjar-built.outputs.found == 'true' }}
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
timeout-minutes: 10
env:
HEAD_SHA: ${{ inputs.commit || github.event.workflow_run.head_sha }}
MB_JETTY_PORT: "4000"
SITE: http://localhost:4000
# Loads per condition. The first is the cold reading and the rest give the
# median, so this trades run time for a steadier number.
RUNS: "8"
steps:
- name: Check out the code
uses: actions/checkout@v6
with:
ref: ${{ env.HEAD_SHA }}
- uses: ./.github/actions/prepare-node-bun
- name: Prepare JDK 25
uses: actions/setup-java@v5.2.0
with:
java-version: "25"
distribution: "temurin"
- name: Install Chrome v144
uses: browser-actions/setup-chrome@2e1d749697dd1612b833dba4a722266286fbefcd # v2.1.2
with:
chrome-version: 144
id: setup-chrome
- name: Download EE uberjar
uses: ./.github/actions/fetch-artifact
with:
commit: ${{ env.HEAD_SHA }}
edition: ee
extract-path: temp/uberjar
- name: Start Metabase
run: |
mkdir -p artifacts
# --add-opens matches how the jar is launched everywhere else
# (uberjar.yml, pre-release.yml, bin/docker/run_metabase.sh), so the
# benchmark runs the app under the JVM configuration we ship.
MB_DB_FILE="${RUNNER_TEMP}/bench.db" \
nohup java --add-opens java.base/java.nio=ALL-UNNAMED \
-jar temp/uberjar/metabase.jar > artifacts/metabase.log 2>&1 &
- name: Sign in
run: |
# Waits for the instance to migrate and come up, then creates the
# first user. Masked because the token authenticates as that user.
session=$(bun frontend/test/bench/sign-in.ts "$SITE")
echo "::add-mask::$session"
echo "SESSION_COOKIE=$session" >> "$GITHUB_ENV"
- name: Warm the backend
run: |
# The first document a fresh JVM renders pays for its own JIT. Spend
# that here rather than inside the first condition measured.
for _ in 1 2 3 4 5; do curl -sS -o /dev/null "$SITE/"; done
- name: Measure load times
env:
CHROME_PATH: ${{ steps.setup-chrome.outputs.chrome-path }}
run: |
bun frontend/test/bench/matrix.ts "$SITE/" "$RUNS" \
| tee artifacts/load-times.json
- name: Show the Metabase log if something went wrong
if: failure()
run: tail -100 artifacts/metabase.log || true
- name: Import load times into Stats
if: ${{ inputs.commit == '' }}
env:
ROWS: artifacts/load-times.json
# Passed via env (not interpolated into the shell) so the commit
# message can't inject commands; the script keeps only the subject.
COMMIT_MESSAGE: ${{ github.event.workflow_run.head_commit.message }}
ENG_STATS_URL: ${{ secrets.ENG_STATS_URL }}
API_KEY: ${{ secrets.ENG_STATS_API_KEY }}
run: bun .github/scripts/upload-bundle-load-stats.ts

View File

@@ -10,21 +10,15 @@ on:
required: false
default: 'snowflake,bigquery-cloud-sdk,redshift'
type: string
temp-data-hours:
description: 'TTL in hours for per-run garbage (minimum 2)'
hours:
description: 'TTL in hours for per-run garbage (minimum 4)'
required: false
default: '2'
type: string
fixture-hours:
description: 'TTL in hours for datasets runs share (minimum 24)'
required: false
default: '96'
default: '12'
type: string
# `inputs.*` is empty on the scheduled run, so each falls back to the nightly value.
env:
GC_TEMP_HOURS: ${{ inputs.temp-data-hours || 2 }} # for temp data
GC_FIXTURE_HOURS: ${{ inputs.fixture-hours || 96 }} # for reusable data
GC_HOURS: ${{ inputs.hours || 12 }} # for temp data
GC_DRIVERS: ${{ inputs.drivers || 'snowflake,bigquery-cloud-sdk,redshift' }}
concurrency:
@@ -66,8 +60,7 @@ jobs:
run: |
clojure -X:dev:drivers:drivers-dev:test metabase.test.data.gc/gc-orphans! \
:drivers "\"${{ env.GC_DRIVERS }}\"" \
:temp-data-hours "${{ env.GC_TEMP_HOURS }}" \
:fixture-hours "${{ env.GC_FIXTURE_HOURS }}" \
:hours "${{ env.GC_HOURS }}" \
:report-dir '"target/gc-report"'
# The report is most wanted on the runs that fail, so upload it regardless of the sweep's outcome.

View File

@@ -1,21 +0,0 @@
name: Uberjar Containerize labeled
on:
pull_request:
types: [ labeled, opened, synchronize, reopened ]
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
uberjar:
if: ${{ contains(github.event.pull_request.labels.*.name, 'build-docker-uberjar') || github.event.label.name == 'build-docker-uberjar' }}
uses: ./.github/workflows/uberjar.yml
secrets: inherit
containerize:
needs: [uberjar]
if: ${{ contains(github.event.pull_request.labels.*.name, 'build-docker-uberjar') || github.event.label.name == 'build-docker-uberjar' }}
uses: ./.github/workflows/containerize-uberjar.yml
secrets: inherit

View File

@@ -1,108 +1,89 @@
name: Build Uberjar
# This workflow builds an uberjar and pushes it to dockerhub for any branch, either on-demand,
# or automatically for the listed branches
# This workflow builds and containerizes an uberjar and pushes it to dockerhub.
concurrency:
group: uberjar-${{ inputs.commit || github.event.pull_request.head.sha || github.sha }}
cancel-in-progress: false
on:
push:
branches:
- master
- internal-tools
- ai-service-port
- metabot-mbql-improvements
paths-ignore:
# config files
- ".**"
# documentation
- "docs/**"
- "**.md"
# this covers both BE and FE unit tests, as well as E2E tests
- "**test/**"
- "**_test.clj"
- "**/frontend/**.unit.*"
workflow_dispatch:
inputs:
commit:
description: "Optional full-length commit SHA-1 hash"
type: string
default: ''
required: false
containerize:
description: "Set if the uberjar should be containerized and pushed to Dockerhub."
type: boolean
default: false
required: false
workflow_call:
inputs:
commit:
description: "Optional full-length commit SHA-1 hash"
type: string
default: ''
required: false
containerize:
description: "Set if the uberjar should be containerized and pushed to Dockerhub."
type: boolean
default: false
required: false
jobs:
check-existing-artifacts:
setup:
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
timeout-minutes: 5
outputs:
ee_exists: ${{ steps.check-ee.outcome == 'success' }}
oss_exists: ${{ steps.check-oss.outcome == 'success' }}
should_build: ${{ steps.decide.outputs.should_build }}
build_editions: ${{ steps.matrix.outputs.build_editions }}
java_version: ${{ steps.dependencies.outputs.java_version }}
node_version: ${{ steps.dependencies.outputs.node_version }}
steps:
- name: Check out the code
uses: actions/checkout@v6
- name: Try to fetch existing EE artifact
- uses: actions/checkout@v6
- name: Check for existing EE uberjar
id: check-ee
continue-on-error: true
uses: ./.github/actions/fetch-artifact
continue-on-error: true
with:
commit: ${{ inputs.commit || github.event.pull_request.head.sha || github.sha }}
edition: ee
output-name: metabase-ee.jar
download: false
- name: Try to fetch existing OSS artifact
- name: Check for existing OSS uberjar
id: check-oss
continue-on-error: true
uses: ./.github/actions/fetch-artifact
continue-on-error: true
with:
commit: ${{ inputs.commit || github.event.pull_request.head.sha || github.sha }}
edition: oss
output-name: metabase-oss.jar
download: false
- name: Decide if build is needed
id: decide
run: |
EE_EXISTS="${{ steps.check-ee.outcome == 'success' }}"
OSS_EXISTS="${{ steps.check-oss.outcome == 'success' }}"
echo "EE exists: $EE_EXISTS"
echo "OSS exists: $OSS_EXISTS"
# Build only if artifacts don't exist for this commit hash
# Since we build by hash, if artifacts exist for this hash, we don't need to rebuild
if [[ "$EE_EXISTS" == "false" ]] || [[ "$OSS_EXISTS" == "false" ]]; then
SHOULD_BUILD="true"
echo "Will build: ee_exists=$EE_EXISTS, oss_exists=$OSS_EXISTS"
else
SHOULD_BUILD="false"
echo "Skipping build: artifacts exist for this commit hash"
fi
echo "should_build=$SHOULD_BUILD" >> "$GITHUB_OUTPUT"
# if this is on a release branch, we need to check the build requirements
get-build-requirements:
if: |
!cancelled() &&
contains(github.ref, 'release-x')
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
timeout-minutes: 10
outputs:
java_version: ${{ fromJson(steps.dependencies.outputs.result).java_version }}
node_version: ${{ fromJson(steps.dependencies.outputs.result).node_version }}
steps:
- uses: actions/checkout@v6
- name: Create build matrix
id: matrix
uses: actions/github-script@v9
with:
sparse-checkout: |
release
.github
script: | # js
const hit = {
ee: "${{ steps.check-ee.outputs.fetch-hit }}" === "true",
oss: "${{ steps.check-oss.outputs.fetch-hit }}" === "true",
};
// Don't rebuild an edition whose uberjar already exists for this commit.
const editions = ['ee', 'oss'].filter((edition) => !hit[edition]);
core.info(`build: ${editions.join(', ') || '(nothing, both artifacts already exist)'}`);
core.setOutput('build_editions', JSON.stringify(editions));
- name: Prepare release scripts
if: ${{ contains(github.ref, 'release-x') }}
uses: ./.github/actions/prepare-release-scripts
- name: Get build dependencies
if: ${{ contains(github.ref, 'release-x') }}
uses: actions/github-script@v9
id: dependencies
with:
@@ -112,26 +93,22 @@ jobs:
const version = getVersionFromReleaseBranch('${{ github.ref }}');
const requirements = getBuildRequirements(version);
return {
java_version: requirements.java,
node_version: requirements.node,
};
core.setOutput('java_version', requirements.java);
core.setOutput('node_version', requirements.node);
build:
needs:
[check-existing-artifacts, get-build-requirements]
if: always() && !cancelled() && needs.check-existing-artifacts.outputs.should_build == 'true' && (needs.get-build-requirements.result == 'success' || needs.get-build-requirements.result == 'skipped')
name: Build MB ${{ matrix.edition }}
needs: [setup]
if: ${{ needs.setup.outputs.build_editions != '[]' }}
name: build (${{ matrix.edition }})
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
timeout-minutes: 15
strategy:
matrix:
edition: ["ee", "oss"]
edition: ${{ fromJson(needs.setup.outputs.build_editions) }}
env:
MB_EDITION: ${{ matrix.edition }}
INTERACTIVE: false
SKIP_LICENSES: ${{ github.event_name == 'pull_request' }} # faster builds for dev
SKIP_LICENSES: ${{ github.event_name == 'pull_request' && !inputs.containerize }} # faster builds for dev
# Emit bundle stats on trunk/release builds (tracked over time) and on PRs
# (the bundle-size check reads them for both the PR head and its base ref).
EMIT_BUNDLE_STATS: ${{ (github.ref_name == 'master' || startsWith(github.ref_name, 'release-') || github.event_name == 'pull_request') && 'true' || 'false' }}
@@ -143,11 +120,11 @@ jobs:
- name: Prepare front-end environment
uses: ./.github/actions/prepare-frontend
with:
node-version: "${{ needs.get-build-requirements.outputs.node_version || '22' }}"
node-version: "${{ needs.setup.outputs.node_version || '22' }}"
- name: Prepare back-end environment
uses: ./.github/actions/prepare-backend
with:
java-version: "${{ needs.get-build-requirements.outputs.java_version || 25 }}"
java-version: "${{ needs.setup.outputs.java_version || 25 }}"
- name: Build
run: ./bin/build.sh
- name: Upload bundle stats
@@ -168,29 +145,23 @@ jobs:
uses: actions/upload-artifact@v7
with:
# Same key the artifact lookups use, so a build for an explicitly dispatched commit
# uploads where `check-existing-artifacts` and `check-jar-health` go looking for it.
# uploads where `setup` and `jar-healthcheck` go looking for it.
name: metabase-${{ matrix.edition }}-${{ inputs.commit || github.event.pull_request.head.sha || github.sha }}-uberjar
path: |
./target/uberjar/metabase.jar
./COMMIT-ID
./SHA256.sum
check-jar-health:
jar-healthcheck:
needs: [setup, build]
# Health-check whatever we are about to containerize, plus every build on master and release.
if: ${{ !failure() && !cancelled() && (inputs.containerize || github.ref_name == 'master' || startsWith(github.ref_name, 'release-')) }}
name: jar-healthcheck (${{ matrix.edition }})
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
name: Is ${{ matrix.edition }} (java ${{ matrix.java-version }}) healthy?
needs: [build, check-existing-artifacts]
# We only care about the explicit JAR health check on push to `master` and the `release-` branches.
# JARs created in random branches on PRs are 99% of the time treated as ephemeral, and used only in e2e tests.
# E2E tests already have a step where they wait for the instance health check.
if: |
always() && !cancelled() &&
(github.ref_name == 'master' || startsWith(github.ref_name, 'release-')) &&
(needs.build.result == 'success' || (needs.check-existing-artifacts.outputs.should_build == 'false' && needs.check-existing-artifacts.outputs.ee_exists == 'true' && needs.check-existing-artifacts.outputs.oss_exists == 'true'))
timeout-minutes: 10
strategy:
matrix:
edition: ["ee", "oss"]
java-version: [25]
edition: [ee, oss]
steps:
- name: Check out the code
uses: actions/checkout@v6
@@ -198,7 +169,7 @@ jobs:
uses: actions/setup-java@v5.2.0
with:
java-package: jre
java-version: ${{ matrix.java-version }}
java-version: ${{ needs.setup.outputs.java_version || 25 }}
distribution: "temurin"
- run: java -version
- name: Retrieve uberjar artifact
@@ -217,3 +188,19 @@ jobs:
java --add-opens java.base/java.nio=ALL-UNNAMED -jar ./metabase.jar &
- name: Wait for Metabase to start
run: while ! curl 'http://localhost:3000/api/health' | grep '{"status":"ok"}'; do sleep 1; done
containerize:
needs: [build, jar-healthcheck]
if: ${{ inputs.containerize && !failure() && !cancelled() }}
name: containerize (${{ matrix.edition }})
strategy:
matrix:
edition: [ee, oss]
uses: ./.github/workflows/containerize-jar.yml
secrets: inherit
with:
edition: ${{ matrix.edition }}
commit: ${{ inputs.commit || github.event.pull_request.head.sha || github.sha }}
release: false
build-args: |
GIT_COMMIT_SHA=${{ inputs.commit || github.event.pull_request.head.sha || github.sha }}

View File

@@ -1,103 +0,0 @@
name: Update E2E Timings
permissions:
contents: write
pull-requests: write
on:
workflow_dispatch:
inputs:
days_back:
description: "Number of days to look back for timing artifacts"
type: number
default: 7
required: false
schedule:
# Run weekly on Sundays at 6 AM UTC
- cron: "0 6 * * 0"
jobs:
collect-and-average-timings:
if: ${{ github.event_name != 'schedule' || github.repository == 'metabase/metabase' }}
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
timeout-minutes: 30
env:
DAYS_BACK: ${{ inputs.days_back || 7 }}
steps:
- name: Generate GitHub App Token
uses: actions/create-github-app-token@v1
id: app-token
with:
app-id: ${{ secrets.METABASE_BOT_APP_ID }}
private-key: ${{ secrets.METABASE_BOT_APP_PRIVATE_KEY }}
- uses: actions/checkout@v6
with:
token: ${{ steps.app-token.outputs.token }}
- name: Setup Node.js
uses: actions/setup-node@v6.4.0
with:
node-version: 22
- name: Collect and average timing data
id: collect-timings
uses: actions/github-script@v9
with:
script: |
const { collectAndAverageTimings } = require('${{ github.workspace }}/.github/scripts/average-e2e-timings.js');
const result = await collectAndAverageTimings({
github,
context,
token: '${{ steps.app-token.outputs.token }}',
daysBack: ${{ env.DAYS_BACK }},
existingTimingsPath: 'e2e/support/timings.json'
});
core.setOutput('artifact-count', result.artifactCount);
- name: Set up git configuration
run: |
git config user.name "github-actions[bot]"
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
BRANCH_NAME="update-e2e-timings-$(date +%Y%m%d-%H%M%S)"
echo "BRANCH_NAME=$BRANCH_NAME" >> "$GITHUB_ENV"
echo "Branch name will be: $BRANCH_NAME"
- name: Create Pull Request
uses: peter-evans/create-pull-request@5f6978faf089d4d20b00c7766989d076bb2fc7f1 # v8.1.1
with:
token: ${{ steps.app-token.outputs.token }}
branch: ${{ env.BRANCH_NAME }}
base: master
commit-message: |
Update E2E test timings with averaged data
Collected timing data from ${{ env.DAYS_BACK }} days of test runs.
All specs with timing data were included (simple averages).
🤖 Generated automatically by update-e2e-timings workflow
Co-Authored-By: GitHub Actions <actions@github.com>
committer: "github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>"
author: "github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>"
title: "[chore] Update E2E test timings with recent data"
body: |
## Summary
This PR updates the E2E test timing data with averaged values from recent test runs.
## Details
- Collected data from the last **${{ env.DAYS_BACK }} days** of test runs
- Averaged timing data from **${{ steps.collect-timings.outputs.artifact-count }}** test runs
## Generated automatically
🤖 This PR was created automatically by the `update-e2e-timings` workflow.
The timing data helps cypress-split optimize test distribution across parallel jobs.
Co-Authored-By: GitHub Actions <actions@github.com>
labels: |
.CI & Tests
no-backport
add-paths: e2e/support/timings.json
team-reviewers: |
devexp

Binary file not shown.

Before

Width:  |  Height:  |  Size: 8.0 KiB

After

Width:  |  Height:  |  Size: 7.2 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 84 KiB

After

Width:  |  Height:  |  Size: 86 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 78 KiB

After

Width:  |  Height:  |  Size: 79 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 73 KiB

After

Width:  |  Height:  |  Size: 75 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 76 KiB

After

Width:  |  Height:  |  Size: 77 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 90 KiB

After

Width:  |  Height:  |  Size: 92 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 79 KiB

After

Width:  |  Height:  |  Size: 82 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 87 KiB

After

Width:  |  Height:  |  Size: 88 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 72 KiB

After

Width:  |  Height:  |  Size: 74 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 78 KiB

After

Width:  |  Height:  |  Size: 79 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 75 KiB

After

Width:  |  Height:  |  Size: 77 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 76 KiB

After

Width:  |  Height:  |  Size: 78 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 78 KiB

After

Width:  |  Height:  |  Size: 79 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 75 KiB

After

Width:  |  Height:  |  Size: 77 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 72 KiB

After

Width:  |  Height:  |  Size: 74 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 72 KiB

After

Width:  |  Height:  |  Size: 74 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 75 KiB

After

Width:  |  Height:  |  Size: 77 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 83 KiB

After

Width:  |  Height:  |  Size: 85 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 76 KiB

After

Width:  |  Height:  |  Size: 77 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 73 KiB

After

Width:  |  Height:  |  Size: 74 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 74 KiB

After

Width:  |  Height:  |  Size: 75 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 89 KiB

After

Width:  |  Height:  |  Size: 90 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 79 KiB

After

Width:  |  Height:  |  Size: 81 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 85 KiB

After

Width:  |  Height:  |  Size: 86 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 71 KiB

After

Width:  |  Height:  |  Size: 72 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 76 KiB

After

Width:  |  Height:  |  Size: 77 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 74 KiB

After

Width:  |  Height:  |  Size: 76 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 74 KiB

After

Width:  |  Height:  |  Size: 76 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 76 KiB

After

Width:  |  Height:  |  Size: 77 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 74 KiB

After

Width:  |  Height:  |  Size: 76 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 71 KiB

After

Width:  |  Height:  |  Size: 72 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 71 KiB

After

Width:  |  Height:  |  Size: 72 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 74 KiB

After

Width:  |  Height:  |  Size: 75 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 40 KiB

After

Width:  |  Height:  |  Size: 37 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 117 KiB

After

Width:  |  Height:  |  Size: 109 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 212 KiB

After

Width:  |  Height:  |  Size: 210 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 382 KiB

After

Width:  |  Height:  |  Size: 367 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 87 KiB

After

Width:  |  Height:  |  Size: 84 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 71 KiB

After

Width:  |  Height:  |  Size: 71 KiB

Some files were not shown because too many files have changed in this diff Show More