Starting the poll on matchPending raced /stash: the branch is created before the
export task is, so the first poll read the previous task, saw a terminal status
and stopped -- leaving the two remaining remote-sync specs waiting on a second
poll that never came. Keying off matchFulfilled with a task_id present mirrors
what the settings-update listener already does.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The branch-and-push flows now make a single POST /stash instead of create-branch
followed by export, but the listener middleware only knew about exportChanges.
Nothing dispatched taskStarted, so the current-task poll stayed skipped, the
progress modal never opened, and three remote-sync e2e specs waited for a
request that was never made.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A revert must never move content between worktrees, so worktree_id joins the
excluded columns for card, dashboard, dashcard and dashboard-tab revisions.
The worktree transform test wrapped itself in with-temporary-setting-values for
transforms-enabled. That setting has a derived getter, so under premium features
without :hosting it reads false, and the restore writes that back as an explicit
false -- disabling transforms for every test that ran afterwards in the same
JVM. Granting :hosting lets the getter answer true on its own, with nothing to
restore. The namespace is also marked synchronous, matching its siblings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
TransformResponse is a closed map, so every transform endpoint started
answering 400 once the model gained worktree_id -- which took out all twelve
Transforms Python driver jobs and the Druid one.
document-serdes-spec-test asserts the skip list verbatim, and two git commit
messages the sync writes to the branch trip no-literal-metabase-strings; those
are branch history, not UI text, so they get the directive rather than
getApplicationName.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The previous commit narrowed worktree_id to Collection and derived everything
else from the collection a row lives in. That does not hold up: content needs
to answer which worktree it is in without a join, so the column goes back on
all thirteen tables, propagated from the parent at insert and immutable after,
with snippets, transforms and transform tag assignments back in scope.
What does not come back is the worktree_id_helper generated column and the
reworked entity_id uniques. entity_id stays globally unique, and
remote_sync_worktree_remapping keeps the per-worktree identity instead.
The reusable hooks live in OSS metabase.remote-sync.core, and the read gate now
mirrors the editable? family: collection-readable? / model-readable? are plain
OSS functions with an unconditional rule -- a row with a worktree_id is
admin-only, whatever the remote-sync settings say. On the write side the EE
editable? checks stop answering an unconditional true for worktree content and
answer is-superuser? instead, and the parent-collection perms set falls through
to the existing ___no-remote-sync-access string for both cases.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
worktree_id was on thirteen tables and propagated down the content tree,
which needed a hook on every model, an immutability guard on every model, and
a generated column plus a reworked entity_id unique on every table just so a
worktree could hold its own copy of an entity the main app already has.
Put it on Collection alone. A worktree materializes collections; everything it
checks out lives inside one, so the collection answers which worktree any row
belongs to. Collections take their worktree_id from their parent on create and
can never change it, and that is the whole propagation story.
entity_id stays globally unique. remote_sync_worktree_remapping pairs the id
the branch knows an entity by with the id of the row the worktree checked it
out into, and serdes reads the pair in both directions: an export writes the
source id, a load resolves it back to the worktree's own row -- and to nothing
at all when the worktree has not checked the entity out yet, which is what
makes the load insert a fresh copy rather than overwrite the main app's. A row
created inside a worktree has no branch id, so its first export mints one.
Extraction is scoped the same way: Collection by its own column, everything
with a collection_id by an anti-join on it.
Transforms, transform tags and snippets leave the feature's scope; a worktree
push no longer collects them as export roots.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
worktree_id was nullable, which meant a composite (parent_id, worktree_id)
foreign key could not enforce anything: with a null in the key the check is
skipped entirely, so main-app rows were free to sit inside worktree
collections. Model the main app as a worktree row instead -- the one flagged
is_default, with a null branch since it follows the remote-sync-branch setting
-- and make worktree_id NOT NULL everywhere.
That makes the containment rule a foreign key. Thirteen composite FKs point
each child at its parent's (id, worktree_id), so content cannot be attached to
a parent in another worktree, or moved across worktrees, at all. Where the
parent link is NOT NULL the check is unconditional; where it is nullable it
covers everything except root-level content.
It also retires the worktree_id_helper generated columns. They existed only to
give the entity_id uniques a non-null discriminator, which worktree_id now is,
so the composite uniques are plain (entity_id, worktree_id) -- and with them go
the four dialect-specific DDL variants per table, the after-select hooks that
stripped the column, its serdes skip entries, and its cmd/copy special case.
check-parent-same-worktree stays only on the six models a user can reparent
from the UI, so those moves still answer 400 rather than surfacing a constraint
violation; everything else relies on the foreign key.
Dropping the single-column entity_id uniques moves to its own migration file
running before this one, with one block per dialect covering all eleven tables
rather than a changeset each, and a rollback that recreates them with explicit
names so the next migration can drop them portably.
Verified on H2, Postgres 15, MySQL 8 and MariaDB 10.6.
- every worktree-scoped model declares worktree_id and worktree_id_helper in its
serialization spec, so the columns are accounted for per model rather than
exempted wholesale in the spec-completeness test
- a worktree_id in incoming data is never honoured: a load stamps the scope it is
pulling into and extraction is scoped to it, so the plain serdes API only ever
reads and writes main-app content
- transforms checked out into a worktree cannot be run, by hand or from a job or
DAG plan; their tags can still be edited
- transform dependencies resolve within the transform's own worktree
- worktree queries move from the model namespace into the API that uses them
GET /api/transform gains include-worktrees, false by default: a transform checked
out into a worktree only shows up when the caller asks for it, the same way
worktree collections do.
Browsing a collection's children now matches the parent collection's worktree
instead of taking an include_worktrees flag: a worktree collection lists its own
worktree's collections, a main-app one never lists worktree collections.
include_worktrees stays where it belongs, on the listing of all collections.
A worktree is a checkout of a git branch materialized alongside the main app: its
content lives in the same tables tagged with a nullable worktree_id (NULL = main
app), so several branches' content can coexist in one instance.
- migration: remote_sync_worktree table; worktree_id on the collection-based synced
content tables, remote_sync_object and remote_sync_task; a generated
worktree_id_helper column backing per-worktree (entity_id, worktree) uniqueness
- worktree_id is never ambient: a pull sets it on the root collections it
materializes, everything under them inherits it from its parent, and it can never
be changed afterwards. metabase.remote-sync.core holds the helpers each model calls
- worktree content is admin-only, to create as well as to read and write: the gate
sits in the collection perms layer, so every collection-based model inherits it
- listings leave worktree collections out unless asked for them (include_worktrees,
like include_library)
- serdes matches entity ids within the worktree being loaded (serdes/*worktree-id*)
- EE: RemoteSyncWorktree model + admin-only CRUD under /api/ee/remote-sync/worktree;
worktree_id/worktree-id params on the pull, push and status endpoints; the sync
ledger and task history are scoped per worktree
- branch endpoints are now GET /branch and POST /branch, and creating a branch no
longer switches the instance onto it - the FE creates-and-continues via stash
* Type the visualization registry as components or bare definitions
* Convert the dashboard viz register module to TypeScript and use typed named exports for Action and IFrameViz
* Strip large literal IN lists before GraalPy SQL parsing
Large literal IN lists (like large VALUES clauses) blow GraalPy's ~114KB/AST-node
memory cost and its 30s call timeout: a 300k-item list that sqlglot parses in
1.7s on CPython times out at 30s on GraalPy, poisoning a pooled context on every
background dependency-analysis pass over the offending query.
Extend the JVM-side pre-processing to also replace literal-only IN lists with
more than 100 items with `IN (NULL)`. Only lists consisting purely of literals
(numbers, quoted strings, NULL/TRUE/FALSE) are stripped; column references,
function calls, subqueries, and bind parameters disqualify a list, since
stripping those would change analysis results or parameter counts. Lists nested
inside unstripped IN subqueries are still found.
All callers get the same handling: the stripping happens in the one shared
strip-large-literal-lists (renamed from strip-large-values) used by every
analysis entry point, and is-single-stmt-of-type? keeps the #74284 restore
behavior — validate the stripped SQL, return the original text — so
impersonated queries still execute with their literal lists intact.
With the fix, field-references on a 300k-item IN list drops from a 30s timeout
to ~80ms.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Simplify literal-list stripping to a single shared pass
Review cleanup: collapse the two per-keyword scanners into one walk over the
SQL driven by a combined VALUES|IN pattern with small per-keyword handlers,
extract the quote/doubled-quote scanning into a shared skip-string-literal
(also fixing count-top-level-commas, which didn't handle doubled-quote
escapes), and replace the substring-allocating literal-only-list? check with a
single index-based early-exit pass, so IN (SELECT ...) is rejected without
scanning or copying the subquery region. No-strip calls now return the
original string identity.
field-references on a 300k-item IN list drops from ~80ms to ~27ms; all
existing stripping tests pass unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add exhaustive tests for literal-list stripping
Cover threshold boundaries (100 vs 101) for both VALUES and IN, every accepted
literal shape, a disqualifier grid (17 non-literal items x 3 positions),
byte-for-byte preservation of surrounding SQL, nesting (IN inside VALUES
tuples, stripped-VALUES swallowing nested lists, deep IN-subquery chains),
idempotency, original-instance identity on the no-strip path, malformed input
(unbalanced parens, unterminated strings, empty lists), keyword false
positives (margin, JOIN, INT, quoted identifiers), and an end-to-end pass of
every analysis entry point over stripped SQL.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Also strip large ARRAY literals and literal tuple IN lists
Two more static-data shapes reproduce the GraalPy 30s timeout / severe CPU
burn: id = ANY(ARRAY[...]) (the Postgres idiom ORMs emit instead of IN; 200k
items hit the 30s timeout) and composite-key tuple lists
(a, b) IN ((1, 1), (2, 2), ...) (50k tuples burned 21s).
Generalize the flat literal-list scanner to a parameterized closing delimiter
and reuse it for ARRAY[...] -> ARRAY[NULL], and add a tuple-list scanner that
requires every tuple to be literal-only, stripping to IN ((NULL, ...)) with
the first tuple's arity. Subscript access (arr[1], arr[1:5]), subquery tuples,
and any non-literal item still disqualify a list. The is-single-stmt-of-type?
restore behavior covers both new shapes.
After the change: tuple-IN 50k tuples 21.6s -> 525ms; ANY(ARRAY[200k])
30s timeout -> 1.1s. CASE WHEN chains and UNION ALL literal tables remain
unstripped (no lossless strip exists for them).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Treat double quotes as a disqualifier in literal-list scanning
In identifier-quoting dialects (Postgres, Snowflake, ...) "x" inside an IN
list or ARRAY literal is a column reference, not a string, so stripping a
large list of double-quoted items would drop real field references from
analysis. The literal-list scan is dialect-agnostic, so be conservative: only
single-quoted strings count as literals; a bare double quote leaves the list
untouched. skip-string-literal still skips both quote styles where it is used
for paren/comma balance scanning, where either interpretation is a
skip-region.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Use [[strip-threshold]] docstring links instead of backticks
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix Eastwood reflection warning: cast close-ch to primitive char
(= ch close-ch) compared a primitive char against the untyped close-ch
parameter, which Eastwood flags as an unresolvable Util.equiv call.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Ignore keyword matches in strings/comments; reject empty list elements
Review follow-ups:
The keyword matcher previously rewrote text inside string literals and
comments — SELECT 'IN (0, ..., 149)' became SELECT 'IN (NULL)', and an
unbalanced quote inside such a region could corrupt the statement. The walker
now gap-scans between matches with a string/comment-aware cursor
(code-region-end) and copies any match that sits inside '...', "...",
-- ..., or /* ... */ through untouched.
The flat-list scanner also accepted empty elements, so invalid SQL like
IN (,,,) with >100 commas was rewritten into valid IN (NULL), masking syntax
errors from validation. Both the flat and tuple scanners now require content
in every element and reject leading/trailing/consecutive commas.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Simplify IN/ARRAY stripping to one simple-list scanner
Stripping feeds fail-soft analysis only, and execution paths restore the
original SQL, so the defensive machinery accumulated from review rounds
(literal word allow-list, tuple/arity validation, empty-element checks,
string/comment-aware outer scanning) bought correctness for cases that don't
matter at the cost of real complexity. Replace it all with one rule: a list is
strippable when, from its opening ( or [, it contains only digits, signs,
single-quoted strings, commas, whitespace, and balanced parens (which covers
tuple lists with no special logic), and holds at least 100 commas. Flat IN,
tuple IN, and ARRAY all collapse into a single ~20-line scanner and one
decision fn; NULL/TRUE/FALSE items now conservatively block stripping. VALUES
keeps its pre-existing count-based path. The best-effort contract is stated in
the public docstring.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Seven sites across four drivers were escaping SQL string literals the same way. This consolidates, and there's a single place to warn not to use escape-sql to sanitize user input.
No behavior changes.
MBQL treats `java.math.BigInteger` and `clojure.lang.BigInt` interchangeably. Only the Clojure class was in the parameter table, so a BigInteger fell through in one of my tests to `:default` and bound as a string.
* Show the output schema and table for a transform run
Adds a TransformOutput component that renders the run's target as `schema › table`
breadcrumbs with a copy button. Both crumbs link out when they exist: the schema to
the data model, the table to the query builder. Wired into the run sidebar
("Output table" row) and the job run sidebar ("Output:" line).
Also fixes the job run sidebar not shrinking below its content width (missing
min-width: 0), which let long table names push it past the viewport.
* fix tests
* drop newTab prop, always open output links in new tab
* addressing PR's comments
* fix race condition
The three preceding PRs removed every shared-tier import from the module: the
route guards, the redux location mirror, and the styled-Link re-export. All that
is left is metabase/utils, which basic may import.
basic rather than lib: router is a React component library, the same shape as
metabase/ui, and nothing in basic/* imports it, so lib would only widen what
lib/utils is allowed to reach for no gain.
* fix(sdk): sdk crashes when resizing column
* do the same on ExecutionOutputTable too
* reword test to make it better
* make sure cols/rows are not new arrays every render when output is undefined
* Derive Tabs value type from the tab values
Mantine types the Tabs value and onChange argument as `string | null`. Make
TabsProps generic over the tab value so callers get their own union back in
onChange instead of a bare string.
* Drop Tabs onChange type workarounds at call sites
Every `<Tabs>` call site now annotates its own tab union instead of casting or
re-narrowing the change value: the six `as SomeTab` casts (CommentsSidesheet,
PythonEditorResults, GroupLimitsSettingsSection, McpAnalyticsPage,
CliAnalyticsPage, ConversationStatsPage), the four in the admin/permissions and
tenants listings, and the handlers that laundered `string | null` through a
lookup or a literal comparison (AddDataModal, NotificationsTabs, TableSection,
the date pickers, TabPanel). AddDataModal's `isValidTab` guard existed only for
this and is gone; DatasetFieldMetadataSidebar's TAB map is now `as const`.
* Cleanup
* Apply suggestions from code review
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
---------
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
* Mark metabase/router as side-effect free for rspack
* Route deep metabase/router imports through the barrel
* Let static-viz import the router barrel instead of a deep module
* Guard the side-effect-free contract and scope the include to the directory
metabase/router/Link.ts re-exported metabase/common/components/Link so every
routing import had one home. But that Link is not a routing primitive: it wraps
RouterLink with a Tooltip from metabase/ui, a CSS module, and disabled/variant
props, so the barrel dragged shared/common and basic/ui into the router module
and looped back into router/router-link.
Call sites import it from metabase/common/components/Link again. RouterLink and
RouterLinkProps stay in metabase/router, which is where the primitive belongs.
The boundaries linter never flagged this: element-types checks imports, not
'export ... from'. So the edge had to go rather than be waived, or a lower tier
would have been green while the dependency was still there.
* Fix N+1 query patterns in GET /api/database and GET /api/native-query-snippet
GET /api/database?saved=true&include=tables issued ~6-7 app-DB queries per
saved question (one metabase_database select-one plus a field select and
4-5 hydrations each) — ~3000 queries on instances with a few hundred
eligible cards:
- card->virtual-table: only fetch the database when include-database? is
set, instead of unconditionally per card
- extract cards->card-id->metadata-fields so the result_metadata fields of
all cards are fetched and hydrated in one batch (reused by
batch-fetch-card-query-metadatas), and pass it from cards-virtual-tables
- ids-of-dbs-that-support-source-queries: one select instead of
select-pks-set + per-id select-one
GET /api/native-query-snippet ran user-has-any-perms-of-type? — an
uncached EXISTS against data_permissions — once per snippet via
mi/can-read?, even though the answer is per-user, not per-snippet:
- user-has-any-perms-of-type? now primes the request-scoped permissions
cache for all databases (the same pattern query-creation-capabilities
already uses on GET /api/user/current) and answers by scanning the
cached entries; repeated calls in one request are pure cache hits
- prime-all-dbs-cache records completeness with an ::all-dbs sentinel in
the existing :db-ids set so the database id list is fetched only once
- the uncached EXISTS path is kept for cross-user checks and
disable-perms-cache
Note: the cache prime excludes permission rows for inactive tables, so a
grant existing only on a deactivated table no longer satisfies
user-has-any-perms-of-type?.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Track full-prime state under a dedicated :all-dbs-primed? cache key
Move the all-databases-primed marker out of the :db-ids set (where it sat
as a ::all-dbs sentinel among real database IDs) into its own
:all-dbs-primed? key on *permissions-for-user*. prime-db-cache now
preserves unrelated keys by assoc-ing instead of reset!-ing the atom.
Make prime-all-dbs-cache public with an optional pre-fetched-ids arity,
and switch query-creation-capabilities — the one caller that manually
primed all databases — to use it, so the full-prime work is shared with
user-has-any-perms-of-type? within a request.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Simplify prime-all-dbs-cache: single arity, :all-dbs? key
Drop the pre-fetched-ids arity — the function always fetches the database
ID list itself — and rename the completeness marker from :all-dbs-primed?
to :all-dbs?.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Align :all-dbs? docstring continuation line
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add call-count regression tests for the two fixed N+1 endpoints
Both tests measure warm app-DB call counts via t2/with-call-count at two
entity counts and assert the count doesn't scale:
- GET /api/database?saved=true&include=tables with 2 vs 18 eligible cards;
allows the 1-per-card toucan2 identity-query overhead from
do-after-select but fails on the pre-batching one-select-per-card
behavior (verified failing against the pre-fix code).
- GET /api/native-query-snippet with 2 vs 10 snippets as a non-admin;
fails on the per-snippet permission EXISTS check (verified failing
against the pre-fix code).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Use mt/with-model-cleanup in call-count tests
Replaces the try/finally + manual delete pattern, so fixture rows are
cleaned up even when insertion or the warm-up request throws (addresses
review feedback).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Rework any-perms-of-type caching to a dedicated request-scoped cache
Instead of priming the full permissions cache for every database and
scanning it, cache per [user-id perm-type] the set of database IDs
granting the most permissive value, loaded across all databases in a
single query. This answers every :exclude-db-ids variant from one small
fetch and keeps semantics identical to the uncached query.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* perms cache
* nil fix
* fix
* current state
* cleanup
* Test that inactive-table-only grants are ignored by aggregate perm checks
user-has-any-perms-of-type? previously counted grants existing only on
inactive tables (its raw EXISTS query had no active-table filter); now
that it is served by the distinct-perms cache it ignores them, matching
every other cached check. This is intentional — cover it with a test.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Apply code-review fixes to the permissions cache rework
- Restore batch priming in with-can-run-adhoc-query via a new public
prime-full-perms-cache (table-level checks otherwise load the cache
one database at a time, up to 2 queries per database on card lists)
- Scope the non-cached distinct-perms load to one database instead of
scanning all of the user's permission rows per call
- Extract the shared FROM/JOIN/WHERE of the two cache queries into
perm-rows-query-base so their row sets cannot drift apart
- Simplify the full-perms interner binding and drop a dead fnil and a
redundant double-check inside the distinct-perms swap!
- Select only :id and :engine in ids-of-dbs-that-support-source-queries
(full rows decrypt :details per database, twice per request)
- Own the batched virtual-table Field fetch in a new
cards->virtual-tables entry point; the helper is private again
- Use mt/with-dynamic-fn-redefs in permission-caches-load-per-type-test
- Document that the request caches are point-in-time snapshots
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Treat cached false as a hit in fast-interner
or-based lookups skip false, so interning false re-ran canon-fn and
putIfAbsent on every call (result was still correct). Use if-some so
presence, not truthiness, decides the hit.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Document why distinct perms load all databases and why the uncached interner exists
Capture the review-thread answers in the docstrings: distinct-perms
preloads every database because most consumers ask cross-database
questions, and full-perms builds a per-call interner even without a
request cache because group values are highly repetitive.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Batch-prime the full perms cache in the remaining per-table check loops
Endpoint tracing found three more loops that fill the full permissions
cache one database at a time now that priming is on demand: the
include=tables branch of GET /api/database (per-table can-read?/
can-query?/can-write?), batch-fetch-query-metadatas* (per-table
can-read? behind dashboard/card query_metadata), and the EE
filter-tables-by-data-model-perms loop. Each now batch-primes the perm
types it checks, one query per type instead of up to 3 per database.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Simplify perms caches to all-types loading; rename to table/schema caches
Per-type bookkeeping wasn't worth it — nearly every consumer needs
several permission types, and there are only a handful. Both caches now
load every permission type at once: the table-level cache (was
*full-perms-cache*) per database, the schema-level cache (was
*distinct-perms-cache*, table_id collapsed so schema is the finest
granularity it answers) for all databases in a single query on first
use. prime-table-perms-cache takes just database IDs and no-ops for
superusers, whose checks never read the cache.
Also batch-prime the remaining per-table check loops found by a
repo-wide audit: search result filtering, GET /api/table, recent views
(also reusing the already-fetched table instead of re-selecting it),
typed-schemas table listings, and the segment/measure list endpoints.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Drop the search prime and scope the recents prime to actual table recents
Benchmarks on a 10M-row data_permissions Postgres (200k rows for the
test user) showed both primes regressing their endpoints: search went
40ms to 840ms because priming all databases loads every permission row
of the user on each request while matching results only touch a few
databases, and recents primed the whole instance because table-recents
returns every table when given no IDs. Search now relies on lazy per-db
loads; table-recents returns nothing for empty IDs so the prime no-ops
naturally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Load the schema perms cache per database, with a priming function
Benchmarks showed the all-databases SELECT DISTINCT costing ~230ms per
request at 200k user rows, taxing endpoints that ask about one database
(dashcard query 228ms -> 445ms). The schema cache now mirrors the table
cache: single-database questions load one database at a time (indexed,
all permission types at once), loops over many databases call
prime-schema-perms-cache, and cross-database questions load everything
once, recorded under :all-dbs? so repeated checks stay one query.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Load perms caches via next.jdbc/plan instead of toucan2 rows
JFR allocation profiling showed toucan2's per-row machinery (row delays,
transient instances, realize maps) dominating the cache loads' share of
request allocation. plan reads columns straight off the ResultSet with
no per-row maps; the folds are unchanged. Runs on the current toucan2
connection so transactional writes stay visible. On the 10M-row bench
this cuts per-request allocation ~10-15% further (e.g. /api/user/current
570MB on master -> 376MB) and 50-90ms of latency on load-heavy
endpoints. Note plan queries are invisible to t2/with-call-count, so
one test now spies on the load fn instead of counting calls.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Store perms caches as mutable Java collections, skipping persistent copies
The folds now build nested HashMaps/HashSets that the caches store
as-is: no per-row persistent update-in churn and no whole-structure
persistent copy at the end. Clojure's lookup/seq functions read Java
collections directly, so consumers are unchanged. Only the two outer
levels (perm-type, db-id) are ConcurrentHashMaps that later loads extend
via putAll of missing databases; everything below is written once by its
load and then only read. Numeric keys are canonicalized to Long at both
fold and lookup time — JDBC hands back Integer for int columns and Java
maps compare boxed keys with equals, not Clojure's equiv. The entry
interner is gone with the persistent copy; instances with thousands of
identical per-database profiles lose that sharing (#76077), which is the
trade-off for the allocation win.
On the 10M-row Postgres bench (200k rows for the user) vs master:
/api/user/current 570 -> 268 MB allocated per request, GET /api/database
?saved=true&include=tables 1773 -> 1032 MB and 4.1s -> 2.5s,
GET /api/table 1152 -> 865 MB and 1.9s -> 0.9s.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Answer cross-database permission questions from a light per-function cache
Instances with thousands of databases and groups make loading tuples
for every database (schema cache with :all-dbs?) scale badly: the
distinct tuple set alone reaches hundreds of thousands of rows. The
questions that actually span databases only need one value per database
for one permission type, so most-permissive-database-permission-for-user
(signature unchanged) and user-has-any-perms-of-type? now share
*most-permissive-perms-cache*: {user-id {perm-type {db-id value}}},
loaded by one indexed query returning distinct (database, value) pairs
and coalesced per database. The :all-dbs? machinery is gone and
query-creation-capabilities needs no priming — its per-database loop
hits the cache after one query.
On the 10M-row bench this takes GET /api/user/current from 722ms/268MB
allocated to 556ms/182MB — 5MB above the response-validation floor
measured with all permission checks short-circuited (master: 833ms/570MB).
Also reorganize the caching section (atoms, then loaders, then the
functions using them; coalesce moves above since a loader needs it) and
relax the xrays call-count expectation: permission-cache loads go
through next.jdbc and are invisible to t2/with-call-count.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Coalesce permissions in SQL and give every check its own per-function cache
Each permission-checking function now owns one atom, one hand-written
aggregate query, and a cache structure fitting exactly its question,
organized per function. Values map to ranks (value-rank-case) and
GROUP BY collapses tables, duplicate values and — via two nested levels
of aggregation for the most-restrictive-within-group-then-most-
permissive-across-groups questions — group identity too, so result sets
scale with tables/schemas/databases rather than raw row counts
(dbs x groups x tables x values). A bucket's [min max] rank pair
reconstructs coalesce exactly, verified exhaustively per permission
type including :perms/view-data's special rule; full-schema keeps its
per-group db-level union by expanding db-level rows into every schema
inside its query. native-download delegates to full-db-permission
(identical semantics), user-has-any derives from the most-permissive
cache.
With the row-count multiplier gone the loads are back to plain t2/query
and persistent Clojure collections: the next.jdbc/plan machinery, the
mutable Java collections, the Long key canonicalization, and the
interner are all gone, and cache loads are visible to
t2/with-call-count again. No public API changes; the batch primer is
now prime-db-perms-cache and warms every per-database cache.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Drop a docstring reference to the removed full-perm-rows
The ~15x realize-before-key-access observation stands on its own; the
function it pointed at no longer exists now that the permission caches
load SQL-side aggregates.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Rewrite app-db NOT IN (SELECT ...) queries as NOT EXISTS anti-joins
NOT IN (subquery) degrades to a per-row subplan on Postgres once the
subquery result exceeds work_mem. Rewrites all runtime occurrences to
correlated NOT EXISTS (or a literal comparison where the subquery was
0/1 rows). MySQL keeps the double-wrapped subquery in
delete-database-fields! since it cannot reference the DELETE target
table in a subquery.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Keep is_audit predicate as NOT EXISTS in set-default-database-permissions!
Tests create audit DBs with random ids, so the fixed audit-db-id
constant is not equivalent to the is_audit = true subquery here.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Snowflake: don't over-escape schema names in the dynamic-table probe
The FK sync path pre-escaped schema/table names (`_` -> `\_`) before handing
them to `dynamic-table?`, which uses the schema as an identifier in
`IN SCHEMA <db>.<schema>`. `"RAW\_DATA"` is not a real schema, so Snowflake
threw for every table in any underscore-named schema and we logged a full
stack trace each time.
`dynamic-table?` now takes raw names and escapes only the `LIKE` argument,
where `_` is actually a wildcard. Escaping for the JDBC metadata call moved
to the call site. Also log the caught exception without the stack trace,
since it fires once per table.
Fixes#78541
Co-authored-by: da <paoliniluis@gmail.com>
* Generate a fresh title for a forked conversation from its first new message
* Show "Forked conversation" instead of "New conversation" in fork header
* Rename first-valid-user-message to first-non-forked-user-message
* Align map value columns in metabot conversation schemas (PR review nits)
* feat: fork a Metabot conversation at an assistant message
* feat: surface Metabot conversation fork origin in analytics
Record forked_from_conversation_id on the conversation and
forked_from_message_id on each copied message, expose them through the
metabot-analytics API and audit views, and link a fork back to its
original thread.
* Update git_branch icon
* Show source conversation title in fork boundary and header
* Revert fork boundary to the attempt where the fork happened
Showing the fork boundary divider on every regenerated attempt was
misleading: the fork only happened on the inherited attempt, so the
divider matches that single message again.
* Refine fork boundary rendering and tests
* Exclude forked (copied) messages from v_metabot_messages
* Address PR review: unify message-action guard, extract ForkBoundary, assert fork toasts via UI
* Reuse createMockMetabotConversationDetail for the fork test fixture
* Replace fork props on Messages with a renderAfterMessage slot; move ForkBoundary to EE
* Drop the redundant UndoListing now that the metabot test setup mounts it
* Index and FK-constrain forked_from_conversation_id so deleting an origin nulls the link
* Split forked messages out of usage metrics with new_message_count
v_metabot_conversations gains new_message_count, which leaves out the
messages a fork copied in from its origin. message_count keeps counting
the whole thread, so the conversations list, the detail page and the
shipped model all report what the reader can actually see.
The usage stats charts aggregate the new column instead, which makes
SUM(new_message_count) reconcile with COUNT(*) FROM v_metabot_messages.
metabot-stats leaves copies out of :metabot-queries and :metabot-users
for the same reason — copies inherit role/ai_proxied and are stamped
with a fresh created_at, so a fork was reporting LLM calls that never
happened.
* Don't offer to fork while a turn is streaming
Only the streaming message's actions were hidden, so earlier assistant
messages kept their fork button mid-turn. Forking then swapped the
conversation out from under an in-flight stream whose reducers are keyed
by agentId alone, so the old turn's remaining deltas rendered inside the
fork.
Withholding onFork while isDoingScience is set is preferable to
cancelling the request: the user asked to branch off an earlier message,
not to stop the response, and loadConversation already avoids cancelling
so an aborted turn isn't recorded.
* Realign the analytics schema maps and fix a migration author
The fork columns are longer than the alignment column the surrounding
maps used. ConversationDetail had already drifted into two alignment
columns before this branch; both maps are now uniform.
Extends the dynamic JSX formatter pattern from #77658 (which did links)
to markdown and email. formatValue and formatEmail now call a registered
renderer through the registry instead of importing ReactMarkdown/Mustache
and ExternalLink directly; the renderers live in ui.tsx and are wired up
at app boot by registerJsxFormatting (called from registerVisualizations).
This keeps the react-markdown/micromark/mustache stack and the mailto
ExternalLink out of the static-viz bundle, which imports the formatting
engine through shared echarts code but never registers the JSX renderers
(so jsx formatting degrades to plain text there, as it already did for
links). No behaviour change in the app.
Static-viz bundle: 3,450,947 -> 3,297,087 bytes (~150KB, 4.5%).
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
metabase.metrics.core re-exports sandbox-restricted-fields from metabase.metrics.permissions, and metabase.metabot.tools.entity-details calls it through that API. The passthrough to metabase.permissions.metric was missing, so importing the re-export threw at load and broke every namespace downstream of metabase.metrics.core.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Transform failure emails list root-cause failures by name and summarize dependents that never ran as a count, so the test asserts the tally instead of the dependent's name.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Add Metric Dimensions section
* [metrics] Field IDs are not unique! Multiple FKs to same table.
Fixes a few places where the field ID alone is assumed to be a unique
key for dimension mappings.
* [metrics] Add basic CRUD API for metric v2 dimensions
Fixes UXW-4631.
Fixes UXW-4633.
* [metrics] Add `/api/metric/$ID/dimension/set-default` endpoint
Fixes UXW-4632.
This sets the dimension of a Metric v2 with the specified
`dimension_id` (a body parameter) to be the `:default true`.
All other dimensions have `:default` removed, so there is at
most one default dimension for any metric.
* [metrics] Metric dimensions create dependencies on the tables
Fixes UXW-4636.
* [metrics] Serdes support for curated Metrics v2 dimensions
Fixes UXW-4634.
Previously `:dimensions` and `:dimension_mappings` were marked as
`:skip` on serdes since the dimensions were always synced on read.
Now that they're curated and only synced when blank, we need to
properly serialize dimensions and mappings.
* Adjust Metric dimensions UI + connect to api
* Add dimensions reorder with dnd
* Add metric dimensions usage in metric viewer
* Change Metric Viewer to group dimensions by type
* Implement shared dimensions logic, add e2e tests
* Use default dimension in Metric Viewer
* Align headings in panels / Make trash and add buttons the same height
* Move and simplify the Set as default button
* Make about page use the default curated dimension in the preview chart
* [dimensions] Bump `:card_schema` to 24; backfill `:dimensions` in old style
Fixes UXW-4808.
This gives pre-existing v2 Metrics the full `breakoutable-columns` as
their dimensions, rather than just the main group as is now the
default. That preserves the old behaviour for the dimensions of the
metrics.
* Respect default metric dimensions on dashboards / Add projectDimension lib function (#77910)
* Respect default metric dimensions on dashboards
* Consolidate metric dashboard E2E coverage
* Remove default time dimension from metric editor
* Implement a few design tweaks in Dimensions tab UI
* Use curated metric dimensions on dashboard metric card filters
* Add dimension picker to metric About page
* Auto-pick default dimensions for new metrics
* Additional cleanup to the About page dimension select element
* Expand all metrics in See all dimensions
* Remove table names from metric filters
* Don't show sub-headers in 'See all' dimension view in Metric Viewer
* Remove metric overview tab
* [UXW-4859] Handle empty curated dimension list and removed default dimension edge cases
* Fix default viz type when adding metric to dashboard
* Remove the 'See all' option on explorer page for single metric visualization
* [UXW-4863] Don't use custom display name as field reference in dimension detail column
* [backend] Fix persistence of upgraded metric dimensions
* [UXW-4880] Re-add Overview tab, show curated dimensions
* [UXW-4848] Make sure we show all shared dimensions in the Shared Dimensions view, if the underlying fields match
* [UXW-4771] Make pinned metric in collection respect default curated dimension
* Update e2e tests to account for PR updates [part 1]
* Seed explicitly joined metric dimensions
* Fix metric dimension serialization dependencies
* Fix/adjust backend unit tests
* Fix metric dimension lint issues
* Run metrics with saved card permissions
* Update e2e tests to account for PR updates [part 2]
* Update kondo ratchet for metric serialization test
* Update metric permissions e2e expectation [part 3]
* Fix BE sandbox unit test
* Prefix upgraded metric dimensions with connected table names (#78410)
[UXW-4896] Prefix upgraded metric dimension names
* Address FE issues from July 22 reviews
* use curated dimensions on series breakout list / fix dimension display name on breakout legend and dimension pill bar
* Optimize getMetricMappingOptions
* Adjust about page dimension select margin-bottom and dropdown positining
* Fix curated metric dimension selection across duplicate foreign key paths
* Improve logic to show correct dimension label in the CenterControls button
* Exclude orphaned dimensions from runtime metric metadata
* Address Ethan's comments / Clean up
* Use metadata field cache when checking user permission / Improve field validation when adding dimension
* Address latest reviews
* Update serialization baseline card_schema 23 -> 24
* Tighten docstrings and comments per review feedback
Remove caller references, ticket refs, and usage context from docstrings
and comments; describe contracts instead.
---------
Co-authored-by: Braden Shepherdson <braden@metabase.com>
Co-authored-by: Anderson <anderson@metabase.com>
Co-authored-by: Anderson <ands.alves.nunes@gmail.com>
* Stabilize SSO group-mapping delete flake by re-reading settled state
Selecting groups for a mapping optimistically patches the settings cache,
but the setting update also invalidates and refetches session properties.
A trailing stale refetch can revert the last-added group after the test's
per-selection checkbox assertion has already passed, leaving the mapping
with only one non-admin group. The delete then fires a single deleteGroup
request and cy.wait times out waiting for the second.
Re-load the auth settings page between adding groups and deleting the
mapping, and assert the row shows its full group summary before acting.
The fresh page load aborts in-flight refetches and reads a single settled
server state, so every mapped group is present when the delete fires.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Serialize SSO group-mapping edits against their setting round-trips
Adding a group to a mapping PUTs the group-mappings setting (built from the
widget's current mappings cache) and invalidates session properties, which
triggers a background refetch. The widget optimistically patches the cache,
but a trailing stale refetch can revert the just-added group; the next
selection then builds its PUT from that stale set and overwrites the previous
group server-side (last-write-wins). The mapping ends up with fewer groups
than the test expects, so fewer deleteGroup requests fire and cy.wait times
out waiting for the second one.
Wait for each mapping mutation's PUT and its trailing session-properties
refetch to settle before the next interaction, in createMapping,
addGroupsToMapping (per group), and deleteMappingWithGroups, so every edit is
built from a fully committed round-trip. deleteGroup/clearGroup invalidate
only permissions-group, so updateSetting is the sole source of session
refetches and the waits stay aligned one-to-one.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* update getExtraTagsForVersion to append -lts if isLts is set
* add ability to check if a version is an LTS version
* set islts
* update function to determine if we should tag with lts
* add condition for multiple LTS releases so we only tag the latest
* fix actionlint errors
* update lts tag to include major version
* await result of getExtraTagsForVersion
* fix type
* fix merge conflict, don't throw error
* GDGT-2851 Remove color-mix from theme color definitions
color-mix can only be resolved by a browser, so any theme color using it
breaks static viz, which renders on the server (see UXW-4690, where the
gauge failed on background_page-tertiary-inverse).
Per design's proposal, introduce a new orion-95 stop and use solid scale
values for the page backgrounds:
- dark: page primary/secondary/tertiary -> orion 95/100/110
- light: page primary/secondary/tertiary inverse -> orion 95/100/110
The remaining mixes are collapsed to their literal equivalents:
shadow-default to a plain rgba, and tooltip-background-focused to the
next stop down from tooltip-background.
The mixes that derive from whitelabel colors (core-brand, core-filter,
core-summarize) are left alone, since their values aren't known ahead of
time. A unit test now guards against reintroducing color-mix anywhere
else in the theme definitions.
* Update Loki Snapshots
* Remove var() references and non-hsla literals from theme definitions
Static viz feeds theme colors straight into Color(), which only parses
literals, so a var() throws there exactly like a color-mix does — even
when it points at a perfectly ordinary color. Collapse the remaining
var() references to the underlying base color, and normalize the few
hex and bare-keyword values to hsla so every theme color is a literal.
The guard test drops the transitive resolver it no longer needs and just
asserts that every non-whitelabel color is an hsla literal.
* Fix regression in pivot table
* Empty commit to trigger CI
* Manually update pivot table loki snapshot
* Update e2e colour assertions for the new background tokens
Dark background_page-primary moved from orion-100 to the new orion-95,
so anything asserting rgb(7, 23, 34) for it now expects rgb(12, 28, 39).
The dark mode body assertion was matching color(srgb 0.0204 0.06792
0.0996) — Chrome's computed form for the old color-mix on
background_page-secondary. That token is a plain hsla now, so Chrome
reports it as rgb(7, 23, 34).
* Fix e2e
* Fix e2e
* Revert changes in assertDarkMode
* Fix oss e2e spec
* Revert SDK theming assertions to the deprecated background token
The SDK theme presets resolve colours through the first entry of
SDK_TO_MAIN_APP_COLORS_MAPPING, and for `background` that is the
deprecated `background-primary`, which still points at orion-100.
`background_page-primary` sits at index 1 and never gets read, so
table-root is unaffected by the move to orion-95.
* Address review feedback on the colour theme changes
Move the gold and upsell-gem literals into base-colors, since each was
repeated across both theme files. Drop the intermediate staticColors
binding in the guard test in favour of a second filter, and drop the
`?? {}` on theme.colors, which is a required field.
---------
Co-authored-by: Metabase Automation <github-automation@metabase.com>
* Fix click-behavior multi-stage e2e flake: settle mapping list between selections
The "navigating to questions with filters applied in every stage" test sets 11
click-target mappings sequentially, using index-based selectors (.first(),
.last(), .eq(2)) against the unset-click-mappings list. That list re-renders as
each mapping moves from the unset to the set section, but there was no barrier
between steps, so an index-based lookup could resolve mid-re-render and land on
the wrong target — yielding a stable-but-wrong filter set (a duplicated
"Product → Title" filter replacing the dropped "Reviews - Product → Reviewer"
one, which is exactly the assertion that fails).
Extract source selection into selectClickMappingSource(), which asserts the
source picker popover has closed before returning, so every subsequent
index-based lookup runs against a settled list.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix click-mapping settle barrier to assert popover closed correctly
The previous settle barrier used `H.popover().should("not.exist")`, but the
`H.popover()` helper internally chains `.filter(":visible").should("be.visible")`,
so the assertion is self-contradictory: it requires a visible popover to exist
and then asserts it does not, failing deterministically the moment the source
selection closes the popover.
Assert the popover is gone with `cy.get(H.POPOVER_ELEMENT).should("not.exist")`
— the idiomatic pattern used across the e2e suite — so the mapping list still
settles between index-based selections without the contradictory visibility check.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Scope click-mapping settle barrier to the visible popover
The unset-click-mappings list re-renders as each mapping moves unset->set.
A bare `cy.get(POPOVER_ELEMENT).should("not.exist")` can never settle in the
click-behavior sidebar: it keeps ~60 mantine popover/combobox dropdowns
mounted-but-hidden, so they are continuously found in the DOM. Filter to
`:visible` so the barrier only waits for the open source picker to close.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Export queryToSearch and searchToQuery from metabase/router
* Serialize the query with queryToSearch at the navigation call sites
* Read search, not query, off a navigation descriptor
* Drop query from LocationDescriptorObject
* Thread QB popState off the routing slice
* Thread dashboard edit-mode off the routing slice
* Thread Metabot profile selection off the routing slice
* Thread document Metabot context off the routing slice
* Migrate component location readers to useLocation
* Retire the state.routing slice
* Convert getExpandedCollectionsById to TypeScript and fix the stale ExpandedCollection type
* Type fake collection ids as literals instead of casting to CollectionId, dropping the String() workaround
* Audit log statements: no stack traces, no queries/user input, no pretty-printing
- Replace throwable args to log macros with (ex-message e) so ex-data and
stack traces never reach the logs (~450 sites)
- Remove SQL/MBQL queries, prepared-statement params, HTTP request/response
bodies, PII, and user-authored names from log output (IDs kept)
- Replace u/pprint-to-str in log calls with pr-str, or drop the value
- Silence sqlglot parser warnings in the GraalPy bootstrap (they quote raw
chunks of customer SQL to stderr)
- Rewrite leftover functional colors (hsl/hsla/rgb/rgba) to hex in static-viz
SVG output so Batik never errors (and stack-dumps) on them
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Use logf variants for exception messages; restore serdes --full-stacktrace
- Convert print-style `(log/warn "msg:" (ex-message e))` calls to
`(log/warnf "msg: %s" (ex-message e))`
- Serdes import/export logs the full stack trace again when the
--full-stacktrace CLI option / full_stacktrace API param is passed
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Update swap-clauses tests for clause-free warning messages
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Update log-content test assertions for the new logging policy
Log calls no longer attach throwables (entries have :e nil) and append
ex-message text to the message instead; body/secret leak tests now assert
the payload never appears in log output.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Address review comments
- rethrow-api-error!: update docstrings — response body is never logged,
it travels on the thrown ex-data
- expand-macros: log the segment filter-clause count once instead of an
identical trace line per clause
- settings cache: log the whole SQLException message chain (the old
print-sql-exception-chain walked .getNextException; ex-message alone
dropped the chained causes)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Add Claude Opus 5 to the anthropic openrouter and bedrock whitelists
- anthropic claude-opus-5
- openrouter anthropic/claude-opus-5
- bedrock anthropic.claude-opus-5
* Scope the invite group picker to the groups with access to the shared item
* Imply All Users membership only for invites with a target, and mock the scoped groups endpoint in share menu specs
* Show all groups when inviting to view and mark which ones can access the item
* Make the invite picker's group section labels visible and tidy up UserForm's membership mapping
* Generalize the group picker into ordered labeled sections with group a11y roles, keeping invite-target semantics in UserForm
* Use the compact Alert size for the invite picker's no-access warning
Suppress api-key-missing exception when failing to generate sample question.
Not having an API key is the default situation and should not result
in an error being logged.
* Let library retrieval use pgvector on the app db
The tables were bare-named, so the drop-and-rebuild in ensure-tables! would have
resolved against the application database's default schema. That is what kept
the feature dedicated-only.
Table names now come from the pgvector mode, the way semantic search picks
between its two index-metadata configs: bare for a dedicated store, so no
existing index moves, and qualified by a library_retrieval schema when sharing
the app db. Its own schema, not semantic search's, whose migration drops every
table in it.
Raw SQL interpolating a qualified name into "%s" would read it as one
identifier, so those sites go through quote-table.
The sync task's scheduling gate stays boot-safe: a dedicated URL, or the license
plus a Postgres app db, decided without resolving the mode. available? checks
the license before the store for the same reason.
* Cover the app-db round trip end to end
Opt-in like the semantic-search one, and picked up by the same appdb-mode CI
job. Asserts the tables land in library_retrieval, the application schema is
untouched, retrieval returns the indexed entity, and a semantic-search wipe
leaves the library index alone -- the reason the schema is separate.
* Check our own schema before reporting the store ready
Semantic search's predicate answers for semantic_search. A role that can use
that schema but cannot create library_retrieval passed it, and then failed every
reconcile inside ensure-tables!. Probe for our schema too: present, or creatable
in a rolled-back transaction, cached because available? runs on every write
nudge and search.
* Gate the round trip on the capability the feature checks
Gating on semantic search's check let the harness pass where a role that can use
semantic_search but not create library_retrieval fails in production.
* Check schema privileges, not schema existence
A role with USAGE but not CREATE passed the existence check and then failed
creating tables. Ask Postgres for both privileges instead, and only fall back to
the rolled-back CREATE SCHEMA when the schema isn't there at all.
The whole probe is inside the try now: the existence query wasn't, so a
transient database failure could throw out of request-entity-sync!, which the
write hooks call fire-and-forget.
* Put the dedicated index in the schema too
A dedicated store kept the index in the default schema, which is exactly what
semantic search's migration wipes -- so enabling or migrating semantic search
dropped the library index and silently re-embedded the whole library. It fires
whenever the stored schema version trails the code, not just on first init.
Both modes now use the library_retrieval schema, so default-tables and
app-db-tables collapse into one. An index built before the schema existed is
moved into it with ALTER TABLE ... SET SCHEMA, keeping its rows; the move runs
before the IF NOT EXISTS creates, which would otherwise leave an empty table
beside it. Creating the schema now needs CREATE on the dedicated database, so a
failure says so.
* Drop only semantic search's own tables in a dedicated store
A dedicated store's default schema is shared -- library retrieval keeps its
index there until it moves -- so wiping every table in it could drop that index
before entity retrieval had a chance to adopt it. Both run as background jobs,
so nothing ordered them.
Scoped to the index_/dlq_/repair_ prefixes, and only in dedicated mode: the
module's own schema in app-db mode holds nothing else, so a reset there is still
free to clear leftovers. The app-db sentinel refusal still reads every table in
scope, since an app db is recognized by tables we would never drop.
* Match the names semantic search generates, not their prefixes
A cohabitant in a shared dedicated store is free to call something
index_history, which the prefix check would have dropped. Match the control
tables exactly, index tables through index-table-name? -- the same contract
orphan cleanup uses -- and DLQ and repair tables by their generated shapes.
Also covers the adoption path end to end: a legacy unqualified index with a row
in it, moved by ensure-tables! and still holding the row afterwards. Asserted
there rather than after the reconcile, which GCs documents the library no longer
implies.
* Stop the availability tests depending on the app db
available?-gates-test left the real schema probe live while stubbing only
the pgvector mode. The probe is Postgres-only SQL, so it threw and read as
unusable on the H2 and MySQL runs, and the case failed there.
The round trip's legacy table declares a vector(4) column before
ensure-tables! runs, and the support check verifies CREATE EXTENSION in a
transaction it rolls back, so nothing has installed the type by then.
* Answer review on the dedicated-store reset
Rename ours? to semantic-search-table?, and point the two docstring
references at namespaces the reader can resolve.
Cover the case where a legacy bare table and a qualified one both exist:
adoption has to leave them alone, since SET SCHEMA onto an occupied name
errors and would abort the whole ensure-tables! transaction.
* Latch the app-db schema check once it passes
The check catches everything and returns false, so on a plain TTL a single
transient app-db failure took retrieval offline for five minutes: available?
went false, search returned nothing and the write nudge no-oped.
A confirmed yes now sticks for the life of the process. Only a no keeps
re-checking, so a privilege granted while we run is still picked up.
* Stub the evals trigger
* use ref arg
* dummy to trigger build
* Clean up
* Use appropriate token
* Simplify pr message
* Use .evals_ref for default ref
* Comment cleanup
* Adjust ref
* Use evals branch
* Use general automation token
* readme for .evals_branch
* Use main of evals for label trigger
* Revert "dummy to trigger build"
This reverts commit e6502c6ff076a47d94eb829b516fc45b37cd7680.
* whitespace
* Address review remarks
* Use ai benchmark naming so we can trigger from pr
* Address review comments
* Remove now redundant branch pinning
* Review remarks
* feat: add lookback
* validate lookback
* feat: apply lookback
* tests
* fix
* fix test
* feat: only support lookback on temporal checkpoints
* update test
* FE
wip
* add :type/Time test
* add test with watermark row removed
* refactor to avoid double get-in
* only support date/datetime
* tests
* fix: support coerced cols
* add more tests
* refactor
* more tests
* test for coerced instant
* use utils in FE too
* chorec
* further validate
* use lib utils
* use strict comparison
* allow clearing lookback
* handle race
* chore: migrate registerVisualization to typescript
* chore: drop 'as unknown as Visualization' cast in custom viz plugin loading
applyDefaultVisualizationProps now takes the plain component and returns
the typed Visualization via Object.assign. The SDK's date_style format
option is widened to string | null to match the ColumnSettings value the
host actually passes to checkRenderable.
* Unit test the native editor run button's unmapped-tag tooltip
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Unit test the Save button's unmapped-tag tooltip and disabled state
Covers both dimension and temporal-unit template tags.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Cover temporal-unit tags in the canRun template tag validation test
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Drop the unmapped field filter e2e test in favour of unit coverage
Every assertion it made is now covered by NativeQueryEditorRunButton and
ViewTitleHeader unit tests.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Read useLocation and useParams straight from react-router v7
* Re-export react-router's Outlet instead of shimming it through context
* Disable the command palette by path instead of route props
* Drop the now-unused route props plumbing
* Add mdb/with-unshared-connection: appdb connections exempt from ambient reuse
A connection holding fragile long-lived state -- a streaming result set
(postgres portal/cursor), or FOR UPDATE row locks -- must not be picked
up by ambient connection reuse: a borrower that inherits it via
*current-connectable* and commits destroys that state. This is the
failure mode behind the first-boot hang in #78238, the revert (#77048)
of appdb streaming (#76645), and "cursor/portal does not exist" errors
when field updates run mid-sync-stream.
Built on toucan2's new unshared-connection! (camsaul/toucan2#212): the
connection is used when passed explicitly -- (t2/reducible-query conn
...), (t2/with-transaction [t conn] ...) -- but never bound as
*current-connectable*, so toucan calls that resolve their connection
ambiently run (and commit) on other pooled connections.
mdb/with-unshared-connection is checkout -> mark -> body -> close; in
every other respect the connection is ordinary JDBC, configured by the
body (e.g. .setAutoCommit false to stream). The pg test reproduces the
exact #77048 failure on an unmarked control connection (portal killed by
a nested commit mid-reduction) and shows the unshared connection
streaming to completion.
deps.edn temporarily pins toucan2 to the PR sha via :git/sha; a
follow-up commit will switch back to :mvn/version once a toucan2
release includes it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G7FfdpzW5pr7CX9aNwa98f
* Use released toucan2 1.0.572 for unshared-connection support
camsaul/toucan2#212 merged and released; drop the :git/sha pin.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G7FfdpzW5pr7CX9aNwa98f
* Hold the audit DB cluster lock on a detached connection (fixes boot stall)
Since #76551 the audit install/load/sync pipeline runs inside a cluster
lock whose SELECT FOR UPDATE ties lock lifetime to transaction lifetime:
the whole pipeline became one long transaction on the lock's connection.
The serdes load's ~3,600 per-entity nested transactions piled up as
savepoints and subtransaction XIDs, and every row lock was held until a
single commit at the end -- stalling first boot for 10-25+ minutes and
blocking concurrent work like config-file database provisioning (#78238).
with-cluster-lock now supports :detached? -- the lock rows are held on a
dedicated unshared connection (toucan2 unshared-connection!) while the
body runs on ordinary pooled connections, committing incrementally in
its own short transactions. Locks release at body completion (commit) or
throw (pool check-in rollback). Re-acquiring a lock this thread already
holds detached is a no-op in both detached and transactional form;
upgrading a detached :share hold to :exclusive throws instead of
self-deadlocking; :retry-transient? is rejected (a re-run would
double-apply incrementally-committed work).
The audit pipeline opts in: per-entity serdes isolation (#74412) is
restored, and a crash mid-pipeline is repaired by the next boot (the
checksum only advances on completion; reconcile runs every boot), which
is what makes incremental commit safe here.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G7FfdpzW5pr7CX9aNwa98f
* Address review: confine detached retry to acquisition; repair dialect-sync window
Review findings (galdre), all majors addressed:
- Major 1: retry/with-retry wrapped the whole detached body, so a
retryable-looking error thrown by the body (e.g. a nested cluster lock
timing out inside the serdes load) re-ran an incrementally-committed
body and then surfaced as "Failed to obtain cluster lock", which
ensure-audit-db-installed! swallows as benign contention. An acquired?
volatile now confines both the retry predicate and the acquisition-
failure wrapper to the pre-acquisition phase in detached mode; body
errors propagate raw with intact cause chains.
- Major 2: a failed or killed engine-changed dialect sync was never
repaired once the content checksum had advanced. New durable setting
audit-db-dialect-sync-pending: set before the sync attempt, cleared on
success, a third sync trigger on every boot.
- Major 3: the mysql/h2 engine-flip visibility window under detached
mode is documented as a KNOWN WINDOW at adjust-audit-db-to-source!
(restores pre-#76551 semantics; overlay-based resolution is follow-up).
- Majors 4-6: reviewer's contract tests adopted (throw-path lock
release, check-in rollback, behavioral pin on :detached?, crash-mid-
load self-heal, plus the two Major-1 reproductions); public docstrings
updated to describe all three lock modes and the upgrade throw.
Minors: unified cross-mode reentrancy registry with eager share->
exclusive upgrade validation (fail-fast in any detached interleaving;
legal same-transaction upgrade preserved); h2 read->write upgrade throws
instead of hanging; lock-row INSERT gets a query timeout (bounds the
N-node cold-start pile-up); install-database! and each reconcile group
are single transactions; unshared checkout goes through *application-db*
so the restore-time read-lock gate applies; registry conveyance and
explicit-tx-on-unshared-connection caveats documented; dual kondo
lint-as entries; test hygiene (finally cleanup, nil ambient contract).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G7FfdpzW5pr7CX9aNwa98f
* Right-size the engine-flip visibility comment
The committed, briefly-visible engine flip is this pipeline's
longstanding behavior; only #76551's single-transaction era hid it for
one release. A neutral note fits better than a warning block.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G7FfdpzW5pr7CX9aNwa98f
* Tighten comments to constraints; drop review-context narration
Comments now state what the code can't show (crash windows, why raw
JDBC, why the marker precedes the sync) and drop justification written
for reviewers: issue-number history, caller name-drops, repeated
tellings of the retry-confinement story, and cost reassurances.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G7FfdpzW5pr7CX9aNwa98f
* Make the detached lock a bespoke entry point instead of a with-cluster-lock mode
Reviewer pressure on the :detached? option had generalized
with-cluster-lock into a two-mode API whose cross-mode interleavings all
needed guarding: a held-locks registry, mode-upgrade validation, an h2
upgrade guard, and an acquired? volatile threading retry confinement
through the shared machinery — all protecting call patterns nothing in
the tree performs.
with-detached-cluster-lock is now its own function: exclusive-only,
single lock, documented non-reentrant. Structure replaces machinery —
the body sits outside the acquisition retry, so body errors are never
retried nor rewrapped as acquisition failure (no volatile), and
same-name re-entry throws via a two-line dynamic-set guard instead of a
registry (nested *different* locks, which the audit body does take,
work unchanged). with-cluster-lock returns to its previous shape; the
lock-row INSERT keeps its new query timeout, which bounds the
multi-node cold-start pile-up in both modes. Error shape is unchanged,
so audit-lock-contention? still classifies correctly.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G7FfdpzW5pr7CX9aNwa98f
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Compact in-memory format for the per-request data-permissions cache
The request-scoped cache in metabase.permissions.models.data-permissions
previously stored raw data_permissions rows as full Toucan instances, keyed
{user-id {perm-type {db-id [row ...]}}}. On instances with thousands of
databases and users in many groups a single prime-db-cache call could
materialize over a million instances (hundreds of MB retained per request,
plus the matching GC churn).
Store a compact entry per (perm-type, db-id) instead:
{:groups {group-id perm-value} ; db-level rows
:tables {table-id [schema-name {group-id perm-value}]}} ; table-level rows
and intern the group->value maps at load time: on big instances they are
typically identical across most databases, so the retained size becomes
proportional to the number of distinct permission profiles instead of
groups * databases.
Also fetch only the columns the checks actually consume (group_id, perm_type,
db_id, table_id, schema_name, perm_value) instead of p.*, and fix an O(n^2)
already-cached filter in prime-db-cache.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* small fix
* Use metabase.util.performance helpers in the cache hot path
perf/concat + perf/mapv in entry-group-value-pairs. Deliberately NOT
perf/update-vals for the interning pass: it skips replacing values that
are = to the old ones, which would undo the interning.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Drop perf helpers from entry-group-value-pairs
perf/concat over an already-built vector is just (into a b), and reduce-kv
iterates the map faster than mapv over entry seqs. Plain into + reduce-kv,
one less require.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Address code-review findings on the compact permissions cache
- Duplicate data_permissions rows (no unique constraint on
group/type/db/table) no longer silently last-win: cache slots promote to
a set when duplicate rows disagree, so every value reaches the consumers'
coalescing logic exactly as raw rows did. Regression test added.
- schema_name is stored per table-level row ({:schema .. :value ..})
instead of first-row-wins per table, since it is denormalized and rows
for one table can disagree.
- The interner now lives in the *permissions-for-user* atom, so value
sharing spans every prime in the request (db-at-a-time priming
previously got a fresh pool per call), and it reuses
metabase.util.memoize/fast-interner instead of a bespoke pool.
- rows->cache-entry returns nil for empty rows so cached and non-cached
get-permissions branches agree; the non-cached branch skips interning.
- schema-permission-for-user and most-permissive-database-permission-for-user
now go through entry-group-value-pairs instead of hand-rolled traversals.
- Dropped the impossible-nil guards in prime-db-cache.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix
* Address CodeRabbit nitpicks
- gid->slot -> group-id->slot (no abbreviations)
- native-download-permission-for-user reuses most-restrictive-per-group
instead of re-implementing the grouping inline
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Rename local intern -> interner (Eastwood local-shadows-var)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Explain the slot rationale in conj-slot/slot-seq docstrings
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* simplify slots
* Document interning with a worked example in the cache docstrings
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014V8QbHjoEkJ8eZLG13sYUe
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: dan sutton <dan@dpsutton.com>
* Restyle the sharing popover with Copy link and Embed buttons above the actions list
* Reserve the copy button width with hidden pseudo-elements instead of inline styles
* swap subscriptions action with auto-refresh action
* hide Subscriptions for non-admins if slack and email aren't set up
* small adjustments
* e2e adjustments
* more e2e adjustments
---------
Co-authored-by: Nick Fitzpatrick <nickfitz.582@gmail.com>
* Convert the legacy chart legend components to TypeScript function components
* Share the legend prop types and extract a typed getChartLayout helper with tests
Deleting a Database hard-deletes its Cards with raw SQL, but nothing
rewrites the `parameters` JSON on Cards/Dashboards that used one of
those Cards as a dropdown source. The parameter keeps a
`values_source_config.card_id` pointing at a row that is gone.
On export that made `*export-fk*` throw `:target-not-found`, which
`log-and-extract-one` swallows as a skip — silently dropping the whole
Card or Dashboard from the archive.
Export each parameter on its own now, and when one carries a card
values-source that can no longer be resolved, emit it without
`:values_source_type`/`:values_source_config`. That is the same shape
the app writes when a source Card is archived, so the parameter comes
back as a plain dropdown over its connected fields.
A source Card that merely isn't part of the export is unaffected —
`*export-fk*` only throws when the row is absent, so those still fail
dependency validation as before.
Fixes GHY-3259
The helper inserts straight into the table to bypass the model's transforms,
which bypasses the hook that fills in entity_id too. #78452 made that column NOT
NULL, so every App DB Tests job has failed on it since.
* Eagerly load tenant auth-provider; fail tenant SSO logins loudly (UXW-4898)
Root cause: metabase-enterprise.tenants.auth-provider — the login! method
that validates a login's tenant claim and stamps :tenant_id onto the user
data — was never eagerly loaded. The JWT/SAML providers only `derive` from
its dispatch keyword, which does not load the namespace, so on a freshly
started instance the tenant step was silently absent from the login!
method chain until unrelated traffic (e.g. /api/user/current hydrating
user attributes) lazily loaded it. A JWT provisioning login in that window
created the user with tenant_id = nil even though the token carried a
valid @tenant, the tenant-group assignment failed and was swallowed, and
the login returned 200 — permanently wedging the account (every later
login 403s with "Cannot add tenant claim to internal user").
Fix, plus defense in depth so no future tenant-drop can wedge an account:
- Add metabase-enterprise.tenants.init requiring the auth-provider, wired
into metabase-enterprise.core.init, so the tenant login step is always
registered at startup.
- create-user! throws instead of silently dropping :tenant_id when the
sso-user-fields field list would strip a tenant assignment stamped by
the upstream tenant flow.
- sync-group-memberships*! no longer swallows real tenant/group mismatch
errors (bad pairs whose :group-is-tenant? is non-nil); the login fails
loudly instead. Nonexistent groups keep log-and-continue. This also
covers EMB-2118 (JWT omitting @tenant with a mapped tenant group).
- The login! provisioning chain (tenant creation, user create/update,
group sync) runs in one transaction, so a failed login leaves no
half-provisioned user row behind.
* Fix cljfmt formatting in jwt_test.clj
* Stop the location-syncing call sites from replacing the url they just read
* Cover the documents /new to /new warning with a unit test
* Adopt v7 navigation semantics, drop the v3 parity guards
The v7 provider mirrored every location into state.routing by calling
useDispatch itself, which is what kept metabase/router bound to the store.
createLocationMirror takes a plain dispatch function instead, so the router
publishes locations without knowing redux exists. app.js and the test harness
supply it as RouterProvider's onLocationChange prop. V7ReduxBridge is renamed
V7RouterBridge now that nothing in it is redux-specific.
metabase/router now passes a basic-tier boundaries trial with zero violations.
Guards are authorization policy, not routing primitives. They were the largest
source of shared-tier imports inside metabase/router (redux, plugins, and four
selector modules), blocking the module from dropping to a lower tier.
The generic factory and the app-wide auth/setup guards move to a new shared
metabase/route-guards module. The monitor and data-studio guards move into their
own features, which lets getMonitorRoutes and getDataStudioRoutes stop taking
them as parameters.
Router violations under a basic-tier trial go from 13 to 3; the rest are the
redux location mirror in the v7 provider.
* Add GeoJSON error mocks and cover map load failures in unit tests
`setupGeoJSONEndpoint` could only produce a valid FeatureCollection, so the
error branches of the custom map flow had no unit coverage.
- `setupMalformedGeoJSONEndpoint` returns a 200 with a payload the client-side
validator in `metabase/api/geojson` rejects, since those checks run in
`transformResponse` and need the payload to reach the client.
- `setupGeoJSONEndpointWithError` returns a non-2xx with a bare string body,
matching what `api-exception-response` sends for an `ex-info` carrying only
a status code.
- `createMockProjectedGeoJSONFeatureCollection` supplies projected coordinates
so every bounds corner falls outside lat/lng range.
Covers the invalid-location, missing-features, and out-of-bounds errors in
CustomGeoJSONWidget.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Drop map GeoJSON error e2e tests in favour of unit coverage
All three asserted error strings are now covered more cheaply. The two
backend-owned messages are pinned by metabase.geojson.api-test against mocked
HTTP, and the modal rendering them is covered by CustomGeoJSONWidget. The two
client-side validation messages come from transformResponse in
metabase/api/geojson, which the e2e tests reached with the backend fully
stubbed or bypassed.
Two of them also drove real outbound requests from CI — one to www.metabase.com
and one to a file on the public repo's master branch.
projected.geojson had no other consumer; the equivalent payload is now built by
createMockProjectedGeoJSONFeatureCollection.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Drop admin settings e2e tests that duplicate existing unit coverage
Each of these asserts the same condition as a unit test that already exists,
not merely an adjacent one:
- the HTTPS redirect check failure message is asserted in HttpsOnlyWidget
("should display an error message if the site is not accessible via https")
- the billing info fetch failure is asserted in LicenseAndBillingSettings
("renders error for billing info for store managed billing and info request
fails")
- the absent license input for hosted instances is asserted in
LicenseAndBillingSettings ("does not render an input when token has hosting
feature enabled")
mockBillingTokenFeatures and its feature-key constants had no remaining
callers.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Persist disabling a collection from remote sync (GHY-4189)
Disabling a collection deleted its RemoteSyncObject rows outright, so the
removal left no dirty state to push and the collection's files stayed on the
remote. A later pull then re-imported the collection (its YAML still carried
is_remote_synced: true), re-enabling it.
Mark the collection's and its contents' RemoteSyncObject rows as "removed"
instead — a pending deletion the next export pushes to the remote — while
dropping rows still in "create" (never pushed, so nothing to delete remotely).
* Don't delete contents when a collection is re-synced before the push
Disabling a collection records a pending removal on its contents'
RemoteSyncObject rows. Re-enabling it re-tracked only the collection's own row,
so the contents stayed marked for deletion and the next export removed their
files from the remote.
Clear the recorded removal when a collection is re-synced before that removal
was pushed. Restore to "update" rather than "synced": edits made while the
collection was un-synced are not tracked, so the entity must be re-serialized
for the remote to be guaranteed to match local.
* Stop core-test leaking RemoteSyncObject rows between tests
Disabling a collection now marks its RemoteSyncObject rows rather than deleting
them, so rows seeded by a test outlive it and leak dirty state into every later
test. The old delete-on-disable behavior had been doing this cleanup by
accident.
Reset the table around each test with the clean-object fixture.
* Assert the pull succeeds in the disable round-trip test
The test discarded impl/import!'s result, so an errored pull would leave the
collection un-synced for the wrong reason and the following assertion would
pass vacuously.
* Re-check eligibility at write time so a pending removal isn't resurrected
An event handler computes eligibility from its event payload, then writes the
RemoteSyncObject row much later. A disable committing in between left the
handler writing a tracked status over the "removed" status the disable had
recorded, so the entity stayed on the remote after being un-synced.
Re-check eligibility against current DB state before letting a tracked status
overwrite a pending removal, and let the removal stand when it no longer holds.
* Row-lock the sync entry so the un-sync race has no remaining window
Re-checking eligibility at write time narrowed the race but did not close it: a
disable could still commit between the re-check and the update, and the losing
write would take effect.
Row-lock the entry for the transaction so a tracking write and a concurrent
un-sync settle in a fixed order instead of losing one of the two: whichever
locks first commits, and the other observes that result — the un-sync re-marking
the row, or the eligibility re-check seeing the entity leave the synced set.
The race test drives the disable on its own thread and unparks the handler
without waiting for it, since the parked handler holds the row lock the disable
needs. Its assertion holds for either order, so the choreography only biases
which one is exercised.
* Address review: honest race coverage, insert-path guard, tighter asserts
- Split the concurrency test into two, each parking the handler at the point
that exercises one ordering: disable-first (guarded by the write-time
eligibility re-check) and handler-first (guarded by the row lock, which the
disable now provably blocks on). The prior single test parked on the plain
existing-entry read, so it only ever exercised the re-check ordering while its
comment claimed lock contention.
- Re-check eligibility in the insert branch too: a stale tracked-status event
that finds no row inserts a fresh one, a phantom the row lock cannot cover.
This narrows, but cannot fully close, that window.
- Round-trip pull test asserts = :success rather than not= :error, so a
:conflict result can no longer slip through.
- restore-removed-rsos! docstring no longer claims it only clears removals from
record-removed-rsos!; it clears any 'removed' row in the subtree.
* Re-track never-pushed contents when a collection is re-synced
Disabling a collection drops its contents' never-pushed 'create' RemoteSyncObject
rows (the remote never had them, so nothing to delete there). Re-enabling only
restored the rows that were marked 'removed', so a card added-but-never-pushed
before the disable was left with no row — and the next incremental export, seeing
no dirty row for it, silently omitted it.
On re-enable, insert a 'create' row for every eligible content entity in the
re-enabled subtree that has none, so the next export pushes it. The
RemoteSyncObject table only records pending changes, so already-synced content is
legitimately untracked; re-marking it 'create' re-serializes it harmlessly (a
'create' onto its own path is a no-op).
* scaffold: two-step SDK release process (EMB-1518)
Replaces the hardcoded version/dist-tag computation in
release-embedding-sdk.yml with a new sdkRelease metadata block in
package.template.json (distTag, tagAsLatest), plus a new
sdk-version-bump-pr.yml workflow that bumps the version via `npm version`
and opens a PR labeled sdk-release-trigger for review. Merging that PR
triggers the release workflow directly.
Unverified: scaffolded per EMB-1518 ahead of Nicolo/Roman's review, since
this can only really be exercised by triggering a real SDK release. See
.claude/kelvin/2026-07-09-emb-1518-simplify-sdk-release-process-and-checklist-maintenance/
for the plan and open questions (particularly around distTag semantics for
stable/patch/minor releases).
* fix: correct distTag comment, <major>-stable is real (EMB-1518)
npm dist-tag ls @metabase/embedding-sdk-react confirms 51-stable
through 62-stable are already live, actively-used tags - not a new
concept as the prior comment assumed. Resolves the one open question
this scaffold couldn't settle from the proposal text alone.
* fix: address review cycle 1 findings (EMB-1518)
Fixes release-breaking and correctness issues found in review cycle 1
of the SDK release process scaffold:
- beta release_type now defaults to <major>-beta dist-tag instead of
the literal (non-existent) "beta" tag
- validate branch/release_type combinations that the old
getNextSdkVersion used to hard-error on (master requires alpha,
release branches reject alpha)
- pull_request trigger on release-embedding-sdk.yml now scoped to
package.template.json via a paths filter, and the concurrency group
no longer cancels an in-flight release on unrelated PR closures
- readme-PR follow-up steps now skip cleanly when the readme was
already up to date, instead of failing and firing a false alert
- tagAsLatest now always resets to false rather than carrying forward
a prior true, and the actual value is shown in the summary/PR body
- version-bump branch name includes the run id to avoid collisions on
retry
- renamed a step whose name didn't match what it does
See the "Review cycle 1 (fable)" section and the new "Fixes applied
for review cycle 1" subsection in the plan file for full detail on
each finding and its disposition.
* fix: address review cycle 2 findings + simplify version bump (EMB-1518)
Cycle-2 findings: add a preminor release_type so master's next-major
alpha bump no longer needs a hand-made PR (#9); make the Slack
failure notification independent of an artifact upload that may
never have run (#10); guard against two open version-bump PRs racing
each other (#11); stop interpolating the free-text branch input
directly into shell scripts (#12).
Also replaces the npm-version rename dance (rename to package.json,
bump, rename back, plus a .bak copy) with direct calls to the semver
package the bump logic already wraps, writing the result via jq
instead - no functional change, removes the dead .bak file (#13).
* fix: address review cycle 3 findings + rename resolve-release job (EMB-1518)
Fixes findings #14-#16 from the third fable review cycle:
- #14: the failure Slack alert's messageId interpolation was unquoted,
rendering a JS syntax error whenever send-slack-notification-start was
skipped (the exact scenario cycle-2's fix#10 was meant to cover). Now
quoted, and falls back to sendSlackMessage (new top-level message)
instead of sendSlackReply when there's no thread to reply to.
- #15: finished the cycle-2 injection-hardening pass by routing the two
remaining inline branch-value usages (workflow-summary's heredoc,
publish-npm's gh pr create --base) through env: instead of ${{ }}.
- #16: added a semver validity pre-check before the bump step, so an old
release branch's 0.N.x placeholder version fails with a clear ::error::
instead of a silent exit 1 from the semver CLI.
#17 (minor's version-collision caveat on old release branches) is left as
an open question for Nicolo/Roman rather than removed - documented with a
comment and Judgment call #6. #18 (merge-commit-sha race) is deferred -
it fails loudly rather than silently, lower priority than the others.
Also renames the resolve-release job to prepare-release throughout
release-embedding-sdk.yml, per explicit request (not a review finding).
See the plan file's "Fixes applied for review cycle 3" section for the
full verification detail.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U2Va4ryWuZbQvmUt2N85ec
* feat: auto-compute tagAsLatest from vars.LATEST_VERSION (EMB-1518)
Resolves Judgment call #2: the SDK version-bump workflow previously
always defaulted tagAsLatest to false because there was no reliable
way to know if a branch is the current "latest" major. The repo
already exposes vars.LATEST_VERSION (major version of the most recent
gold/stable release) for exactly this, so patch/minor bumps now set
tagAsLatest=true when MAJOR_VERSION matches it, false otherwise
(including when the variable is unset). alpha/beta/preminor stay
false. The PR review gate still allows overriding the computed value
before merging.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U2Va4ryWuZbQvmUt2N85ec
* refactor: use github.ref_name instead of inputs.branch (EMB-1518)
github.ref_name is already set by GitHub's "Use workflow from" ref
picker on every workflow_dispatch run, so the custom inputs.branch
free-text field was duplicating a selection that was already made -
and could silently diverge from it. Switch every functional read of
inputs.branch to github.ref_name in both release workflows; keep the
branch input field itself, but make it explicitly non-functional
(required: false, no default, description points at the ref picker).
type: choice was considered and rejected again here: it would close
the typo risk but can't accept an unregistered branch, breaking
one-off custom-branch releases without editing the workflow file per
release - exactly the maintenance burden EMB-1518 removes.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U2Va4ryWuZbQvmUt2N85ec
* fix: remove dead readme-update mechanism (EMB-1518)
update_readme's two sed patterns match zero occurrences in the current
README (restructured by 15f7d3d20e, no version-pinned content remains),
so publish-npm's readme-follow-up-PR step has been a silent no-op on
every release, not just re-runs. Deleted the entire dead block (7 steps,
the env vars and pull-requests:write permission it alone used) and the
now-unused update_readme function.
* fix: drop minor release_type option (EMB-1518)
`semver -i minor` on a release branch jumps to the next major's
version line, colliding with that major's own branch. `patch` already
covers graduation and `preminor` already covers the legitimate
next-major jump (scoped to master). Drop `minor` outright rather than
add compensating validation - resolves review-cycle-3 finding #17 /
Judgment call #6.
* fix: add explicit reviewer request for embedding team on bump PR (EMB-1518)
CODEOWNERS is deliberately stripped on release branches (a repo-wide
practice to cut auto-backport review noise, unrelated to this
workflow), so the bump PR's "code owner review is automatic" comment
was false there - it got zero reviewers requested on release
branches. EMB-1000 documented the identical gap in the old system's
follow-up PR. Fix targets the metabase/embedding team (not a named
individual - the old system's hardcoded reviewer has since left).
* fix: restrict release types by branch, add custom releases (EMB-1518)
A one-off branch could publish as the stable release. Validation only
guarded master and release-x.*, so `patch` on e.g. sdk-data-apps stripped
the prerelease id (0.62.5-data-apps.0 -> 0.62.5), computed the
<major>-stable dist-tag, and - when the major matched vars.LATEST_VERSION
- set tagAsLatest, shipping a feature branch to npm as the stable release.
Only check-git-tag stood in the way, and only if that version had already
shipped.
Each branch class now gets only the release types that make sense for it:
master takes alpha/preminor, release branches take beta/patch, any other
branch takes custom. tagAsLatest is additionally gated on a release-x.*
branch, so nothing published from master or a one-off branch can take the
npm `latest` tag.
The custom type keeps one-off branches (esbuild, data-apps) releasable
under that restriction: prerelease_id is set on the first cut and becomes
both the version's prerelease id and the <major>-<id> dist-tag, matching
the shape already live on npm (56-esbuild, 63-data-apps). Later bumps
leave it empty and semver reuses the id from the committed version.
Also drops the dead release_utils.bash (zero functions, zero callers),
replaces both free-text dispatch inputs with single-option choices that
cannot be typed into, points the release workflow's form at the bump PR,
and removes the scaffold banner comments.
* fix: read vars.LATEST_MAJOR_VERSION, not LATEST_VERSION (EMB-1518)
The variable is LATEST_MAJOR_VERSION - it is what release.yml already
reads to decide Docker's `latest` tag, and what the release checklist has
a github admin bump each cycle. `vars.LATEST_VERSION` does not exist, so
tagAsLatest would have computed false on every bump, including at gold,
without any error - the fallback for an unset variable is false.
* refactor: make the release concurrency group readable (EMB-1518)
Same three cases as before, but each arm now names what it produces
(sdk-release-<branch> for a dispatch or a merged bump PR, sdk-release-noop-<run id>
for a closed PR this workflow ignores), and the comment above maps them in
the order the expression tries them.
The noop arm is not decoration: concurrency is evaluated before any job's
`if:` runs, so a closed PR that prepare-release skips would still enter the
release group without it. cancel-in-progress stays false so a release is
never interrupted mid-publish.
Groups are prefixed rather than bare because concurrency groups are
repo-wide, not per-workflow.
* refactor: build the release scripts once, in their own job (EMB-1518)
The Slack jobs each got at release/dist/index.cjs a different way:
send-slack-notification-start checked out and built it (then uploaded it as
an artifact), send-slack-notification-failure built its own second copy, and
send-slack-notification-successful downloaded the artifact. The duplicate
build existed because the failure job has to fire even when the metadata read
fails - in which case the start job never ran, so its artifact never existed.
A build-release-scripts job now builds it once and uploads it, and all three
Slack jobs download it. The job hangs off prepare-release only, so the
artifact is there no matter how early the release fails - which removes the
reason the second build existed.
* fix: quote the Slack messageId in the success notification (EMB-1518)
A Slack ts is a string. Interpolated bare it becomes a JS number literal and
drops trailing zeros - 1783602196.214000 renders as 1783602196.214, and the
reply targets a thread that does not exist. Pre-existing on master; the
failure job already quoted it, so this makes the two consistent.
* refactor: resolve the release branch in one expression (EMB-1518)
The env: BRANCH indirection was injection hardening from when the value was
the free-text `inputs.branch`. It is github.ref_name now - a real branch
name, same as github.event.pull_request.base.ref, which the next line already
inlined - so it guarded nothing and made the step asymmetric.
base.ref is empty on a dispatch, so `base.ref || ref_name` picks the right one
without branching on github.event_name.
* refactor: drop the duplicate-bump-PR pre-flight (EMB-1518)
It guarded a case the design already handles safely. Two open bump PRs
against the same branch both compute from the branch's committed version, so
both propose the same next version: the first to merge publishes and tags it,
and the second dies at check-git-tag with "Tag ... already exists" (npm would
reject the republish too). Nothing is published twice, and the error says
exactly what happened.
Getting there needs someone to dispatch the bump twice and leave the first PR
open, with both PRs sitting labelled in the list. That is not worth an API
call and 16 lines on every bump.
* chore: trim the bump step's comments (EMB-1518)
* refactor: drop the semver validity pre-check (EMB-1518)
Every supported release branch (57 through 63) already carries a valid
semver, and the new process never writes a `0.N.x` placeholder - gold is a
patch bump (0.63.0-beta.3 -> 0.63.0). The placeholder only ever existed
transiently under the old checklist, which rewrote it back to a concrete
version after each release.
Also saves an npx invocation on every bump.
* chore: spell out PRERELEASE_ID in the bump workflow (EMB-1518)
* refactor: spell out the semver call in each release type (EMB-1518)
alpha and beta shared an arm that passed --preid="$RELEASE_TYPE", and patch
passed -i "$RELEASE_TYPE" - you had to know the variable's value to know what
command ran. Each arm now names its own call.
The release_type description now leads with the phase (before gold, at gold,
after gold), which is what the person dispatching actually knows. It also
corrects preminor: master moves to the next major *after gold*, not at branch
cut - the 62 cycle did it in the after-gold checklist (#75487, 0.62.0-alpha ->
0.63.0-alpha), and 61 before it (#74279).
* docs: stop calling the bump PR a review gate (EMB-1518)
Nothing blocks on approval: the bump PR is not auto-merged but it is not
gated on a reviewer either, and workflow_dispatch can publish with no PR at
all. The computed distTag/tagAsLatest are defaults made visible and editable
- whoever dispatched the bump has already decided.
* docs: note EMB-2095 (auto-merge the bump PR) in the workflow (EMB-1518)
* docs: tighten the tagAsLatest comment (EMB-1518)
* refactor: collapse the tagAsLatest case into one condition (EMB-1518)
* docs: say what the sdkRelease write does (EMB-1518)
* refactor: drop the label-creation step (EMB-1518)
The sdk-release-trigger label is repo-level, so it only ever needed creating
once - it now exists. Re-running `gh label create ... || true` on every bump
swallowed every error (including auth) to no purpose.
* fix: fail on a leftover bump branch instead of side-stepping it (EMB-1518)
The run-id suffix made every retry a fresh branch, so an abandoned bump left
a branch on origin forever and the name was unpredictable. The branch is now
named after the version, and an existing one is an error: an earlier bump to
the same version was never merged, which is worth knowing before opening a
second PR alongside it.
* refactor: drop the no-backport label step (EMB-1518)
Backporting is opt-in - auto-backport.yml only fires on a `backport` label -
and backport-reminder.yml only demands a decision label when a PR touches
docs/**. The bump PR touches one JSON file, so no-backport guarded nothing.
Keeps the reviewer rationale, moved next to the --reviewer flag it explains.
* feat: show the target branch and computed values in the bump PR body (EMB-1518)
The PR body did not say which branch it targeted, which is the first thing
you want to know when a release PR shows up. Branch, release type, version,
dist-tag and tagAsLatest are now a table at the top.
* fix: pin semver and validate prerelease_id (EMB-1518)
npx --yes semver resolved whatever the registry served at dispatch time, on
a job that installs no dependencies - so release-critical version math could
change under us while package.json already pins ^7.6.3. Pinned to semver@7.
prerelease_id was passed to semver unchecked. An id like `DataApps` bumped
fine and then died three steps later in the dist-tag read-back, whose regex
only matches [a-z][a-z0-9-]*, with an error blaming the computed version
instead of the input. It is now rejected up front, with a message naming the
rule.
* fix: take semver's version from package.json (EMB-1518)
Pinning the workflow to semver@7 fixed the drift but created a second
version to keep in step. Read the range the repo already pins instead
(devDependencies.semver, currently ^7.6.3) and hand it to npx, so bumping
semver in package.json is the only place it happens.
* refactor: rename bump workflow input to read_first (EMB-1518)
* feat: reject prerelease_id on non-custom release types (EMB-1518)
* refactor: drop the preminor bump-only warning from the PR body (EMB-1518)
* refactor: drop the duplicate rerun wording in tag-exists error (EMB-1518)
* fix: reject reserved alpha/beta as a custom prerelease_id (EMB-1518)
* refactor: tidy and clarify comments in the SDK bump workflow (EMB-1518)
* fix: satisfy actionlint in the SDK release workflow (EMB-1518)
* feat: extract embedding SDK release logic into a tested helper (EMB-1518)
* refactor: remove dead embedding SDK helpers from version-helpers (EMB-1518)
* fix: drop trailing blank line in the SDK bump workflow (EMB-1518)
* refactor: run the SDK version bump from a tested helper, not inline bash (EMB-1518)
* docs: note the env block guards github-script against injection (EMB-1518)
* refactor: drop the unused LATEST_MAJOR_VERSION fallback in the bump script (EMB-1518)
* feat: show the target branch in the bump PR title (EMB-1518)
* docs: drop the manual-edit invitation from the bump PR body (EMB-1518)
* refactor: tidy the SDK release helper and its tests (EMB-1518)
* refactor: read the SDK release metadata via the tested helper (EMB-1518)
* refactor: env-route the branch name in the bump PR title (EMB-1518)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0156DAaukfiU6xceXd3brbZN
* refactor: simplify SDK release concurrency group (EMB-1518)
Collapse the three-case concurrency expression to one per-branch group, and
switch every multi-item needs array to a block sequence.
The dropped noop group guarded a compound case: GitHub keeps one pending run
per group and cancels any earlier one, so a closed PR touching the template
could evict a release waiting to start. That needs two same-branch releases
in flight at once, so we take the simpler expression. The clean fix is
queue: max, which actionlint cannot parse yet.
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* Deflake transform revision history spec
The "view and revert transform revisions" test changed the transform
name twice, each firing a "Transform name updated" success toast that
lives for the default 5s undo timeout. Unlike every other update flow in
this spec, these toasts were never dismissed. On a fast runner the test
reached the failed-revert error toast within that window, so a lingering
success toast coexisted with it and H.undoToast() (a single-element
findByTestId on [data-testid="toast-undo"]) failed with "Found multiple
elements".
Dismiss each success toast right after it appears, matching the existing
convention used elsewhere in this file, so exactly one toast is present
when the error toast is asserted.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Nemanja <31325167+nemanjaglumac@users.noreply.github.com>
* Add semantic search health checks
* Split HNSW recovery from semantic health
* Use HoneySQL literals for repair queries
* Use dynamic redefs in semantic search test
* Auto-scroll to a dashcard whenever it becomes the scrollTo target, and keep the scroll until the grid settles
* Drop a stray brace from the auto-scroll spec's first visit URL
* Simplify the auto-scroll correction to a fixed frame window
* Delete the react-router v3 engine
* Sever the react-router package dependency
* Give the SDK data-app router its own browser history on v7
* Rename the react-router-v7 alias to react-router
* Rename getDataAppHistory to getRawBrowserHistory, drop dead getCurrentHistory
Scope existence preconditions to their table; lint against name-only checks
A foreignKeyConstraintExists precondition with only foreignKeyName (or
indexExists with only indexName) forces Liquibase to snapshot the entire
schema to answer the check -- ~2s per precondition on a large postgres
appdb. 27 such preconditions (24 FK, 3 index; 21 in one v59 workspace
block) added ~46s to every full migration: 50.2s of total changeset time,
of which 24 changesets cost 45.9s and the other 1,182 cost 3.7s.
Add the scoping key to all 27. Measured on a fresh postgres migration:
total changeset time drops 50.2s -> 5.1s, with zero changesets over 1s.
Editing released changesets' preconditions is safe: Liquibase checksums
cover the changes, not preConditions. Verified two ways -- ChangeSet
.generateCheckSum is byte-identical before/after for all 27 (v9
checksums), and a clone of an already-migrated appdb re-validates
cleanly against the edited files.
lint-migrations-file: generalize require-primary-key-exists-has-table-name
into require-table-scoped-existence-preconditions, covering
primaryKeyExists/indexExists/uniqueConstraintExists (tableName) and
foreignKeyConstraintExists (foreignKeyTableName), walking the full
precondition tree in both its flat-map and list forms.
Claude-Session: https://claude.ai/code/session_01G7FfdpzW5pr7CX9aNwa98f
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Fall back to .yaml if .yml not found
Update docs to mention `.yaml` support
* Fall back to .yaml if .yml not found
Update docs to mention `.yaml` support
* Update file.clj
* Fix tests
* Address feedback
* Drop field_usage table as it is no longer used.
* Stop saving preprocessed_query in metadata since it's not used any more.
This used to be used for saving off field usage, but that hasn't been
a thing for a very long time.
* Make model-illustration.svg work well in both light and dark mode
* whitelabel svg
---------
Co-authored-by: Nick Fitzpatrick <nickfitz.582@gmail.com>
* Migrate PinMap to a typed function component and extract a pure getPoints
* Type the pin metric as nullable, derive MapSettings from VisualizationSettings, and add createMockVisualizationProps
* Trim comments, replace the map ref cast with an instanceof guard, and cover instance clearing
* Don't block dashboard saves on pre-existing foreign internal cards
UXW-4870: the guard added in #77236 (reject assigning a dashcard's card
to a question internal to another dashboard) checked updated dashcards
unconditionally, so any dashboard with a pre-existing foreign internal
card association became unsavable — every edit, even a resize, returned
400 "Invalid Request.".
Scope the guard to newly-introduced card ids: card ids the dashboard
already references (as a dashcard's card or in a series) are
grandfathered, so existing associations never block saving — including
moving such a card to a different dashcard in the same save. Newly
assigning a question internal to another dashboard is still rejected.
* Clarify arg names in internal-dashcard guard
Rename the guard's params to existing-dashboard/new-dashcards to make
clear the grandfather set comes from stored state while the checked
dashcards are the incoming save.
* Guest embed filter dropdowns should use the router database (UXW-4881)
PR #68952 made guest-embed query execution bypass DB routing by wrapping it
in with-database-routing-off, but the parameter-values endpoints (dropdown
values, search, remapping) didn't get the same treatment: chain-filter runs
through the QP, where attach-destination-db-middleware throws "Anonymous
users cannot access a database with routing enabled."
- Wrap the query-running sections of the four param helpers in
metabase.embedding-rest.api.common in with-database-routing-off, fixing
all six /embed/{card,dashboard}/:token/params/* endpoints plus their
preview_embed equivalents.
- Wrap the guest-embed tiles endpoints, which have the same anonymous-user
bug.
- Wrap preview_embed's query-execution endpoints (query, dashcard, pivot,
tiles) so the admin's embed preview shows exactly what the published
embed will show (router DB data) instead of routing via the previewing
admin's user attribute.
- Replace the manual *current-user-permissions-set*/*is-superuser?*
bindings in the param helpers with the equivalent request/as-admin.
Public sharing has the same gap but whether routed DBs should work over
public links at all is an open product question, so it is intentionally
left out.
* Consolidate embed routing-off wraps into the shared execution helpers
Review feedback: rather than a macro that each endpoint must remember to
use, move with-database-routing-off inside the shared execution helpers in
metabase.embedding-rest.api.common (process-query-for-card-with-params,
process-query-for-dashcard, process-tiles-query-for-dashcard, and a new
process-tiles-query-for-card wrapper), which every /api/embed and
/api/preview_embed endpoint funnels through. Endpoints built on these
helpers now bypass routing automatically, so a future endpoint can't
reintroduce this class of bug.
Removes all per-endpoint wraps from embed.clj (including the two original
ones from #68952) and preview_embed.clj; the invariant now lives in one
file with one explanatory comment.
* Fix tooltip scrollbar flicker in TreeTable (GDGT-2822)
Hovering a cell tooltip in a virtualized TreeTable while scrolling made a
page-level vertical scrollbar flash on and off. The tooltip is portaled to
<body> and positioned with real top/left insets, and with the document root
sized to the exact viewport height, a tooltip positioned past the fold grew
the document and flashed a scrollbar.
Portal cell tooltips into the table's own scroll container instead of <body>,
so a tooltip positioned past the fold is clipped by the (already-scrolling)
table rather than stretching the page. This is wired through a small
TooltipPortalTargetProvider context so the metabase/ui Tooltip wrapper picks
up the target automatically — no per-tooltip wiring, and any TreeTable with
cell tooltips benefits (e.g. the notifications admin table).
* Address PR comment
* Fix Tooltip wrapper not forwarding ref
* Add monitor dir in metabase-enterprise
* GDGT-2676: Migrate Dependency Diagnostics under Monitor space
* GDGT-2677: Migrate Admin -> Tools under Monitor space
* GDGT-2676: Fix paths
* GDGT-2676: Create separate high-level directories for migrated features
* GDGT-2657, GDGT-2686: Monitor space + migrate Dependency Diagnostics (#76527)
* GDGT-2657: Introduce Monitor space layout and routing
* GDGT-2686: Migrate Dependency Diagnostics to Monitor
* Monitor: Add Monitor plugin. Cleanup + fixes
* Monitor: Update graphy ownership path for Dependency Diagnostics
* Monitor: Cleanup PR
* GDGT-2697: Update hotkeys hints for Toggle navbar
* GDGT-2684: Address Copilot review comments
* GDGT-2684: Address PR comments
* GDGT-2684: Rename SpaceLayout to AreaLayout. Split to separate files
* GDGT-2684: Refactor route guards + add tests
* GDGT-2684: Move Admin Tools pages to the Monitor space (#76606)
* GDGT-2684: Move Admin Tools to Monitor space
* Monitor: Extract diagnostics sidebar into shared Monitor sidebar outlet
Add a useMonitorSidebar context to MonitorContent so Monitor pages can
render sidebars in a dedicated right-hand region outside the padded,
scrollable main content area. Rewire Dependency Diagnostics to render its
detail sidebar through this outlet, and promote the resizable shell
(SidebarResizableBox) into shared monitor/components.
* Monitor: Add MonitorHeaderTitle component for consistent page headers
Introduce a shared MonitorHeaderTitle matching the small secondary text
style used by Data Studio breadcrumbs, and apply it across Monitor pages:
Dependency Diagnostics header, Jobs, Logs, Model cache log, Task list/runs,
Task and task-run details, Erroring questions, and Alerts management.
* GDGT-2684: Cleanup
* GDGT-2684: Update monitoring tools access permissions
* GDGT-2684: Add handling for not-found routes
* GDGT-2684: Fix not-found routes handling in Data Studio
* GDGT-2684: Extend route tests. Minor styling refactoring
* GDGT-2684: Refactor Alerts Management sidebar
* GDGT-2684: Move old SidebarResizableBox to ModelSibar
* GDGT-2684: Cleanup
* GDGT-2684: Refactor Sidebar solution to use Portal
* GDGT-2684: Fix Monitor Tools E2E specs (#76886)
* Format monitor router files after the master merge
* Import Route from metabase/router facade in AreaTab spec
Fixes the no-restricted-imports lint error banning direct react-router imports.
* GDGT-2713: Update Tasks page styling (#77088)
* GDGT-2713: Update Tasks page styling
Restyle the Monitor Tasks and Runs pages to match Dependency diagnostics:
- Replace the SettingsSection card + default tabs shell with the Monitor
frame (fixed-height page, pills tabs; only the table scrolls)
- Rebuild TasksTable and TaskRunsTable on the TreeTable data-grid
(sticky header, virtualized rows, skeleton loading, a11y labels)
- Move filters into a Group bar and pagination below the table with a
total results count
- Keep the 2-state sort contract (always sorted, desc/asc) on top of
TanStack's 3-state cycle; asc-first on all sortable columns
- Add MonitorHeaderTabs (Monitor-owned pills tabs, no data-studio
dependency) and extract the shared LinkTab router-link tab to
metabase/common/components
* GDGT-2713: Fix e2e test
* Migrate LinkTab to the metabase/router facade
* Remove unused Link import from PaneHeader
The Tasks styling switched the tabs to LinkTab, leaving Link unused.
* Address PR review comments and fix entity_name typing
Use strict undefined comparisons instead of loose null checks where
values can only be undefined (optional props, RTK Query error,
destructured SortingState), and strict null comparisons where the
value can only be null (Task.db_id).
Also fix TaskRun/RunEntity entity_name typing: the backend always
assocs this key (nil when the entity lookup misses), so it's
string | null on the wire, not optional.
* GDGT-2712: Update Jobs page styling (#77100)
* GDGT-2712: Update Jobs page styling
Restyle the Monitor Jobs page to match Dependency diagnostics:
- Rebuild the page on the Monitor frame (fixed-height page; only the
jobs table scrolls, with a sticky header)
- Render the scheduler info in a Code block instead of a bare pre
inside a settings card
- Rebuild the jobs table on the TreeTable data-grid (skeleton loading,
empty state, a11y label, keyboard row activation); row click still
opens the job triggers modal
- Move JobInfoApp into a folder and add unit coverage; add
TaskInfo/Job API mocks and a task-info server mock
* Make the Monitor sidebar outlet resizable
Move the resizing behavior from the Dependency diagnostics sidebar into
the shared Monitor Sidebar so every Monitor view gets it without
duplication:
- Relocate SidebarResizableBox from metabase-enterprise/monitor into
MonitorLayout and wrap the Sidebar portal content with it
- Sidebar API: optional resizable (default true) with containerWidth
required while resizable, optional defaultWidth (512 by default);
consumers measure their content area with useElementSize and pass it
- Protect the Monitor content area from text selection while dragging
inside Sidebar instead of per-view
- Simplify DiagnosticsSidebar/DependencyDiagnostics accordingly and
drop the now-empty metabase-enterprise/monitor/components tree
- Make the Alerts management detail sidebar resizable (fills the
resizable box, keeps its 560px default width)
* GDGT-2712: Jobs page styling fixes
* Replace the job triggers modal with a Monitor sidebar
Clicking a job on the Jobs page now opens the job's triggers in the
resizable Monitor sidebar outlet instead of a modal:
- New JobTriggersSidebar renders each trigger as a vertical key/value
block (all fields the modal table had), with a divider between
triggers and a close action that returns to /monitor/jobs
- The /monitor/jobs/:jobKey route stays, so deep links and back/forward
keep working; JobInfoApp reads the jobKey param and renders the
sidebar, measuring its content area for the resize constraint
- Delete JobTriggersModal; add a Trigger API mock
- Give the scheduler info Code block the old card padding and keep it
from shrinking
* Apply cross-review fixes for the jobs sidebar work
- Render trigger details as a dl/dt/dd list for proper label/value
semantics (no visual change)
- Cover the shared Sidebar resize side effect (user-select class on the
Monitor container across drag start/stop/unmount)
- Cover keyboard row activation, all trigger attributes with null
placeholders and the may-fire-again 'No' branch, and the no-triggers
empty state
* GDGT-2712: Cleanup
* GDGT-2712: Add distinct error message when job is not found
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Migrate JobInfoApp to the metabase/router facade
* Disable text selection on body while resizing the sidebar
Address review comment: dragging the resize handle otherwise causes
the browser to continuously select/deselect page text, since the
sidebar is portaled and doesn't share a DOM ancestor with the main
content to scope a CSS class to.
---------
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* GDGT-2711: Update Logs page styling (#77160)
* GDGT-2711: Extract LogsViewer to common Monitor components
* GDGT-2711: Update Logs page styling
* GDGT-2711: Cleanup
* GDGT-2711: Address PR comments
* GDGT-2711: Address PR review comments
Rename LogsViewer's process prop to processUUID, drop the unused
logs-content id, and remove a few comments flagged as redundant.
* GDGT-2661: Update Erroring questions page (styling + refactoring) (#77167)
* GDGT-2661: Restyle Erroring questions page
Rewrite the EE /monitor/errors page off the legacy audit_app viz onto the
Monitor recipe used by the other Monitor tools pages, keeping the same
backend audit query:
- TreeTable with the 11 audit columns, 2-state sortable headers, and native
row selection; "Rerun Selected" moved to the standard bottom BulkActionBar
- Dependency-diagnostics-style debounced filter inputs
- Page no longer scrolls; the table body is the sole scroll region
- Pagination outside the table with a total count (new bad-table-count audit
query), hidden until the data has loaded, and synced to a ?page URL param
- Requests abort the previous in-flight one on paging/sorting/filtering
Replaces the AuditParameters/AuditTable/ErrorMode stack; deletes the old
ErrorOverview and its click-mode.
* Remove dead legacy audit-table v1 code
The Erroring questions page was the last consumer of the Audit v1 table
stack. With it rebuilt on TreeTable, these are unreferenced and removable:
AuditTable, QuestionLoadAndDisplay, AuditTableVisualization (the last
"audit-table" display), AuditParameters, the bad_table card def, and
AuditMode. The rest of audit_app (unsubscribe UI, InsightsLink,
metabot-analytics) is untouched.
* Fix duplicate/incorrect cards in the erroring-questions audit query
The latest_qe CTE joined query_execution to the per-card MAX(started_at)
subquery on started_at alone, so executions across cards that shared a
timestamp fanned out — surfacing duplicate rows (and cards whose latest run
didn't actually error). Join on card_id as well. Regression test covers a
shared-timestamp collision.
* GDGT-2661: Address review feedback
- Surface count-query failures in the page error state so a failing count
can't silently hide pagination
- Filter bar reflows (wrap) instead of clipping at narrow widths / zoom
- Skeleton renders the checkbox column to avoid a layout shift on load
- Centralize the internal-dataset-query cast to a single boundary
- Add tests: count db/collection filter clauses (backend), 202-body
data.error path, real filter-reset assertion, rerun failure recovery
* GDGT-2661: Fold pagination total into the rows query via COUNT(*) OVER()
The page ran two /api/dataset requests per view — the bad-table rows and a
separate bad-table-count — each recomputing the expensive latest_qe scan over
all query executions. Add a COUNT(*) OVER () total_count column to bad-table so
the full filtered total rides on the rows response, and drop the count query.
Halves the round-trips and eliminates the second latest_qe scan (~2.1s -> ~1.4s
of DB work on the Stats dump); the window count adds no measurable overhead
since it runs over the already-materialized result set before LIMIT.
* Migrate ErrorOverview to the metabase/router facade
* GDGT-2661: Return erroring-questions total_count at the response root
The bad-table query computes COUNT(*) OVER () as a column, which repeated the
total on every row. Strip that column via an xform and surface the value once
as a top-level total_count on the response; the FE reads it from there.
* GDGT-2661: Non-blocking per-row rerun with loader in erroring questions
Rerun now runs in the background per card instead of blocking on the whole
batch: clicking Rerun Selected closes the bulk bar immediately and fires each
request independently, so more questions can be queued while slow ones run.
A rerunning row shows a loader in place of its checkbox, is excluded from
Select all, and its loader clears when the request settles; the list refetches
once the last rerun finishes.
TreeTable gains a generic isRowLoading prop and makes the whole checkbox cell
a click target so a near-miss no longer navigates to the row.
Also hoist total_count onto the Dataset type and drop the local helper.
* Extract shared generic sorting helpers for monitor tables
The monitor tables (erroring questions, tasks, dependency diagnostics) each
duplicated getSortingState/getSorting/isSortColumn. Extract them into a generic
metabase/utils/sorting module keyed by the column type. getSorting is overloaded
so the un-sort behaviour is type-driven: pass the current sort for always-sorted
tables (direction flips) or omit it to allow an unsorted result.
Refactor the three monitor tables onto the shared helpers.
* Update erroring questions E2E selectors for the new table
Point the tests at the TreeTable-based ErrorOverview: erroring-questions-table
container, erroring-question rows, and the always-enabled filters / bulk bar
that closes on rerun instead of a disabled button.
* Bind *driver* when reducing audit internal queries
The MySQL timestamp reader calls results-timezone-id, whose driver arg is
malli-validated as a keyword in tests. reduce-results* bound the timezone
override but not *driver*, so it was nil and audit queries returning timestamps
errored on MySQL/MariaDB. Bind *driver* to the app-db driver alongside the
override.
* GDGT-2661: Address PR comments
* GDGT-2661: Merge erroring questions input into one, extend to support name search
* GDGT-2661: Fix checkbox cell width in rows
* GDGT-2661: Fix e2e test
* GDGT-2661: Address PR comments
* GDGT-2710: Update Model cache log page styling (#77277)
* GDGT-2713: Update Model cache log page styling
Restyle to match Dependency diagnostics: the page no longer scrolls
vertically, only the table does. Replace the hand-rolled AdminS.ContentTable
with a metabase/ui TreeTable (skeleton while loading, empty state, refresh
action), and drop the emotion-styled ModelCacheRefreshJobs.styled.tsx in
favor of a CSS module.
Model and Collection columns are the two stretching columns (fill remaining
width) and support client-side sorting, since the underlying /api/persist
endpoint has no sort param.
* Unify monitor tables on 3-state sortable headers
Erroring questions and Tasks forced a 2-state flip-forever cycle by passing
the current sort back into getSorting(); drop that so a 3rd click clears the
header's sort arrow and falls back to the page's default sort, matching
Dependency diagnostics and Notifications.
The default-sorted column on each page needs sortDescFirst matching its
default direction (desc), otherwise the very first click on that column is a
no-op: TanStack's per-column cycle treats desc as the removable/last state
when sortDescFirst is false, so a column that starts at desc goes straight to
removed on click one.
* Make all applicable columns sortable in Model cache log and Jobs
Both are fully-loaded, unpaginated lists, so enabling client-side sorting on
the remaining columns is safe. Model cache log's action column stays
unsortable (no header text or data). Tasks and Task runs are left as-is:
their backends only support a fixed set of sort columns (or none), and
sorting the rest client-side would silently reorder only the current page.
* GDGT-2710: Address PR comments
* GDGT-2709: Update Alerts management page styling (#77364)
* GDGT-2709: Update Alerts management page styling
Restyle to match Dependency diagnostics: the page no longer scrolls
vertically, only the table does. Move pagination outside the table into a
plain footer and show the total result count via PaginationControls'
showTotal.
* Restrict Alerts management to admins only
The /api/notification/admin* endpoints are superuser-gated on the backend,
but the frontend route and nav link were reachable by anyone with the
broader can_access_monitoring permission (shared with the other Monitor
Tools pages), so a non-admin holding it could reach the page while every
API call inside it 403'd. Add a dedicated canAccessAlertsManagement
selector and route guard so the route/nav link match the backend's
admin-only restriction.
* Fix misleading test name in MonitorLayout spec
The test asserts the link is hidden for a non-admin analyst, not that it's
shown for an admin.
* GDGT-2794: Update Task/Run details page styling
Extract the "Back to Tasks"/"Back to Runs" breadcrumb into a shared
MonitorBackLink component, styled and positioned like other Monitor
page headers. Wrap both detail pages in the same non-scrolling shell
used elsewhere in Monitor so only the content area scrolls internally,
instead of the whole page scrolling behind the floating AppSwitcher.
* Extract MonitorPageContent to replace SettingsSection in Monitor
SettingsSection carries admin-page title/description logic that Monitor
pages don't use. MonitorPageContent keeps the same card styling with none
of that logic, dropping the last two /monitor call sites still importing
from metabase/admin.
* GDGT-2709: Fix layout in Tasks Details page
* GDGT-2709: Refactor NotificationsTable row selection logic. Fix selection with spacebar
* GDGT-2791: Abort superseded in-flight list requests in Monitor (#77694)
* GDGT-2791: Abort superseded in-flight list requests in Monitor
Changing page/sort/filter on a Monitor list page left the previous, often
expensive, list request running to completion on the server. Subscribed RTK
useXQuery hooks only abort when their last subscriber unmounts, so introduce
a shared useAbortableQuery hook built on lazy triggers (whose promise exposes
.abort()) that cancels the superseded request on arg change, unmount, and
skip transitions, and suppresses the abort error so cancellations never flash
the error view.
Caching mirrors useQuery: cache-first by default, refetchOnMountOrArgChange
to opt into always-fetch; manual refetch() always hits the network. Every
converted call site keeps its pre-migration caching semantics.
Converted: TaskListPage, TaskRunsPage, TaskRunEntityPicker,
ModelCacheRefreshJobs, NotificationsAdminPage (main list),
DependencyDiagnostics, BrokenDependentsSection, and ErrorOverview (its
one-line useAbortableAdhocQuery wrapper is inlined and hooks.ts deleted).
* GDGT-2791: Cleanup
* GDGT-2791: Fix type narrowing in TaskRunEntityPicker
* GDGT-2791: Clamp pagination range to total
* GDGT-2791: Test that tag invalidation does not refetch superseded args
* GDGT-2791: Scope query aborts per hook instance via injected cache key
* GDGT-2791: Migrate _refetchDeps to the shared __rtkCacheKey mechanism
* GDGT-2791: Refine baseQuery typings. Cleanup
* GDGT-2791: Replace interface in params declarations with type. Refine BaseQuery params type
* GDGT-2832: Group Monitor navbar items into sub-menus (#77706)
* GDGT-2832: Move AreaContent to AreaLayout and normalize Monitor page spacing
* GDGT-2832: Group Monitor navbar items into sub-menus
* GDGT-2832: Cleanup. Fix AreaTab outline overflow
* GDGT-2832: Match Alerts Management tab spacing to Dependency Diagnostics
* GDGT-2832: Replace collapsible navbar groups with static labelled sections
* GDGT-2832: Update Monitor views headings to match navbar. Fix e2e test
* Monitor section design tweaks (#78007)
* icon and spacing updates
* remove info icon from background tasks page heading
* improve empty states
* GDGT-2810: Enable sorting in all Monitor views (#77842)
* GDGT-2810: Enable sorting on all Background tasks and Task runs columns
* GDGT-2810: Remove bold styling from first Monitor table columns
* GDGT-2810: Fix reflection lint warnings in task history API test
* GDGT-2810: Qualify task_run/task_history filter columns
The entity_name and task_count sorts add joins (report_card and
report_dashboard both have their own entity_id column), so unqualified
filter columns become ambiguous: /api/task/runs?entity-id=N&sort_column=entity_name
returned a 500. Qualify the filter columns and add a regression test
combining filters with the derived-column sorts.
* GDGT-2810: Cleanup
* GDGT-2810: Use kebab-case sort params on the task runs endpoint
* GDGT-2810: Extract shared useSortingStateChange hook for TreeTable sorting
* GDGT-2810: Extract shared page/sort URL param parsers
* GDGT-2810: Fix e2e tests
---------
Co-authored-by: Ngoc Khuat <qn.khuat@gmail.com>
* GDGT-2843: Integrate Monitor upsell pages (#77948)
* GDGT-2843: Migrate Monitor upsell pages into monitor scope, link to /upgrade/monitor
* GDGT-2843: Tighten monitor module boundaries to standard shared tier
* GDGT-2843: Change Monitor app icon from gauge to pulse
* GDGT-2843: Update pulse icon and Monitor logo
* GDGT-2843: Address PR comments
* GDGT-2843: Fix e2e tests
* Fix Monitor code review findings (#78106)
* Fix Monitor code review findings
* Remove obsolete dependencies redirects
* Monitor review: Cleanup
* Address review comments
* Cleanup
* Remove Dependency diagnostics link from Data Studio nav
It already lives under Monitor (/monitor/dependency-diagnostics); the
Data Studio route just redirects there now.
---------
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
Co-authored-by: YB Tan <tybmeta@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Maz Ameli <maz@metabase.com>
Co-authored-by: Ngoc Khuat <qn.khuat@gmail.com>
* WIP: start flipping unit-test harness to v7
Default routerEngine to v7 and render bare (non-Route) children under a
catch-all route so v7's <Routes> renders them. role=link nav failures still
to diagnose. Suite not yet green.
* Give v7 tests a history handle and drop DOM-wrapped route config
The harness now owns the v7 memory history and hands specs a v3-shaped adapter
(getCurrentLocation/push/replace/goBack/listen), so the ~42 specs that drive and
assert navigation through the returned history keep working on v7.
Route config wrapped in a <div> is a v3 artifact that v7's <Routes> cannot read;
swap those two for fragments.
* Pin the v3-vs-v7 conformance suites to the v3 engine
They exist to prove the facade on v3 matches real v7 and are built from raw v3
route config, so they must not follow the harness default now that it is v7.
* Resolve a relative initial route in the v7 test history
* Buffer navigations dispatched before the v7 router registers
* Carry route props onto the v7 matched-route stubs
* Fix the v48 /browse/:dbId-:slug legacy redirect on v7
* Render facade routes in renderHookWithProviders on v7
* Migrate unit specs off v3-only routing behavior for the v7 flip
* Drop buffered v7 navigations when the router unmounts
* Default use-v7-router to true
* Re-home history@3 types onto the metabase/router facade
* Format the re-homed import blocks
* Merge duplicate router imports and drop unused lint directives
* Re-home the router-listen spec's history Location type
Wait for the DebouncedFrame to settle via ensureChartIsActive() before
clicking a group tile to drill in. While the frame transitions it carries
pointer-events: none, so the drill click was silently swallowed and the
breadcrumb button never rendered.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Migrate AuditTableVisualization to a typed function component
* Fix Erroring Questions column sorting by forwarding sorting props through Visualization
* Extract typed viz definition to drop casts per review
* GHY-4016: Replace Field YAMLs with FieldUserSettings in git sync (PR 1/3)
Git sync currently exports one Field YAML per database column. For a
typical customer this is ~6396 files — 84% of the entire repo. Most of
that data (active, database_type, position, dimensions, ...) is sync
metadata that git sync has no business tracking; the user-curated part
lives in FieldUserSettings.
This PR switches git sync to export FieldUserSettings instead of Fields.
Only fields with user-curated metadata get a YAML at all.
Measured against the stats-remote-sync repo (196 published tables,
~20 columns each):
Import: 43s → 14s (3x faster, 7659 → 1263 files)
Export: 6.3s → 2.0s (3x faster, 2155 → 239 entities)
Files: 7659 → 1263 (84% reduction)
For a large customer (563k files, 522k Fields), ingest time alone drops
from 116s to 7s.
Full Field serdes is unchanged — non-git-sync export/import is
unaffected. FieldUserSettings already had complete serdes support
(generate-path, storage-path, load-find-local, make-spec); this PR
wires it into the git sync paths.
Changes:
- `serdes/descendants "Table"`: branches on `:user-edits-only` opts.
When set, emits `["FieldUserSettings" field_id]` only for fields with
a FUS row; otherwise emits `["Field" field_id]` as before.
- `spec.clj`: named `git-sync-extract-opts` constant (with
`:user-edits-only true`); public `pk-col` helper for the non-standard
FieldUserSettings PK (`:field_id` not `:id`).
- `extract.clj` (OSS): local `pk-col` for `:where [:in pk-col ids]`
(module boundary prevents depending on EE spec).
- `impl.clj`: `extract-chunk` uses `spec/pk-col` for FieldUserSettings.
- `models.clj`: adds `"FieldUserSettings"` to `data-model-in-collection`.
- `events.clj`: removes `:model/Field` from `register-events-for-spec!`
so field-update events no longer create Field RSOs. A dedicated
`::field-update-event` handler creates FieldUserSettings RSOs only
when a FUS row exists. This completes the incremental path: full
export uses `descendants` + `:user-edits-only`; incremental export
uses FUS RSO rows.
Tests assert on the full RSO table so stray Field RSOs fail immediately.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Address PR review: defensive get for dynamic pk-col, fix content-hash test for FieldUserSettings
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* don't need maps, k/v pairs work
* Declare :model/FieldUserSettings module boundary export/import
remote-sync references :model/FieldUserSettings in events.clj, so it must
be exported by warehouse-schema and imported by enterprise/remote-sync.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Remove unused f1-id binding in extract test (kondo)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Test that Field and FieldUserSettings are serdes descendants leaf nodes
Cements the design discussed in review: Table's descendants method is the
only source of field-level entries, and the :user-edits-only branch is
exclusive — you get Field refs or FieldUserSettings refs, never both via
a transitive Field -> FieldUserSettings edge.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* WIP: dag run BE
* split into new table
* run utils
* remove dag changes on job runs
* new dag run model
* abstract over parent-run
* separate dag model/api
* factor common job/dag code in a coordinated-run primitive
* fixes
* dag test
* fix migration
* kondo
* missing resolution
* exclude model
* kondo config
* fix test
* fix: exclude for deser
* fixes
* fix: exclude for deser
* fix: missing fk
* random ids
* chore: rename
* feat: add unioned runs
* remove dead endpoints
* api tests
* extend filter semantics
* trim docstrings
* kondo
* gap: allow canceling job runs
* share code
* fix: job start should return proper job id
* fix: mig to 64
* simplify
* simplify further
* simplify
* dead code
* add unique constraint on dag runs
* tighten schemas
* tighten permissions
* fix test
* relax schema
* fix: set null on delete
* handle deleted jobs/transforms for dag runs
* appease the sql linter
* fix test
* simplify rollbacks
* simplify callsite
* fix tests from bad rebase
* fix: fallback when transform is deleted
* add transform_count
* fix tests
* log with context
* varchar 32 for status
* Add UI for running transform with upstream or downstream (#77367)
* GDGT-2625 wip: reprocess DAG UI for transforms
- Split "Run now" button (upstream/downstream) + confirmation modal
- runDag / dag-transforms / cancel-dag API + analytics event
- Tabbed Runs page: transform-graph runs (default) + individual transform runs
- Graph runs table (Run/Type cols), filter bar (Type/Transform/Started/Status),
sidebar with member transforms; shapes aligned to bronsa's /api/transform/runs
- List + members still served from a mock (transform-graph-runs.mock) via queryFn
* GDGT-2625 wip: align graph runs to /api/transform/runs, sidebar + cancel
- Reshape TransformGraphRun to backend RunSummaryResponse (run_type/entity_id/name)
- Filters mirror Transform runs: Type/Transform/Started/Ended/Status/Trigger
- Sidebar: title = Run name, Type as first metadata row, cancel section below
metadata (dag/job/transform), confirm modal; job-run cancel endpoint wired
- Mock covers job/dag/transform in all states (succeeded/started/failed/timeout)
- Reorder columns: Status/Trigger after Duration
* GDGT-2625 wip: sidebar header entity links
Add "View this job" / "View this transform" (+ dependency graph for
transforms) action links to the graph-run sidebar header, mirroring the
individual transform-runs sidebar.
* GDGT-2625 wip: migrate graph runs to real /api/transform/runs
- Swap list + member queryFn mocks for real queries (list endpoint, dag/job
member drill-down; standalone runs synthesized). Delete the mock fixture.
- Rewrite page spec against real endpoints via fetch-mock + new mock factories.
- Reword DAG run confirm modal (drop "in this order", bullet list) since members
can run in parallel.
* Adjust wording
* GDGT-2625 wip: scheduled state + cleanup
- Show a "scheduled" run state on the transform page after a DAG reprocess is
triggered, so the queued transform gives feedback instead of looking idle
- Drop dead DAG-run types/tag-providers/mock helpers left from earlier iterations
* Address code-review findings for reprocess DAG UI
Correctness:
- Migrate transforms.cy.spec.ts to /runs/individual (the /runs route now
renders the graph-runs page, not the individual run table)
- Gate the "Run all" DAG confirm button on the mutation, preventing a
double-click from triggering two DAG runs
- Toast when run-dag returns a null dag_run_id instead of silently no-op
- Stop the "Scheduled" banner from polling forever when the member-runs
request errors
- Keep the graph-run sidebar open when the selected run scrolls off the
current page during polling
- Invalidate transform state on cancelDagRun/cancelJobRun so member
transforms' last_run refreshes after a cancel
Cleanup:
- Translate whole phrases (with context) for DAG run names
- Replace a typecast with a type guard in TypeFilterWidget
- Memoize parsed params and filter options on the graph-runs page
- Drop the unused direction field from the scheduled-DAG-run state
- Convert the sidebar's IIFE switch to ts-pattern match
* Handle deleted entities in graph runs; fix run-list cache invalidation
- Graph runs table: grey a deleted job/transform's run name (italic +
"has been deleted" tooltip), keeping the row clickable
- Sidebar: keep showing member transforms for deleted DAG and standalone
runs (fetched by dag-run id / synthesized from the run); hide the section
only for deleted job runs, whose members can't be fetched without the
(now-null) job id
- api/transform.ts: invalidate the run lists on deleteTransform (a
transform's runs survive its deletion) and cancelCurrentTransformRun, so
the Runs pages reflect the change instead of relying on polling
- Trim narrating comments from the transform API types
* Cleanup
* Fix transforms cancel e2e: click renamed "Cancel run" confirm button
The transform-page cancel confirmation button was renamed from "Yes" to
"Cancel run", but two cancelation tests still clicked "Yes". The cancel was
never confirmed, so the slow (pg_sleep) transform kept running and held a lock
on the writable DB, hanging the next test's resetWritableDb and failing the
suite. Point those two clicks at the new label (the individual-runs-page cancel
still uses "Yes" and is left unchanged).
* Avoid type cast in RunSection spec to satisfy no-unjustified-type-casts
Replace `body as string` with a typeof guard when reading the run-dag request
body, which the new lint rule flags.
* Show dependency count in the graph-run label
Use bronsa's new transform_count to label DAG runs as
"{n} dependencies › {name}" (upstream) / "{name} › {n} dependents"
(downstream), n = transform_count - 1, joined by a small chevron icon.
Bold the job/transform name; fall back to the plain name when the count is
absent (job/standalone runs, or DAG runs predating the column).
* Replace runs tabs with Detailed view switch; polish graph-run name rendering
- Swap the two runs tabs for a Detailed view switch that toggles grouped
(/runs) and detailed (/runs/individual) views in place; add RunsPage shell
and DetailedViewSwitch, remove RunTabs
- Render the sidebar header run name like the table (dimmed deps count +
chevron), with wider gap and aligned font size/weight
- Keep the Detailed view switch label on one line
- Bump graph-runs page size to 25 to match the individual runs page
- Fix stale react-router-redux import
* Preserve shared filters across Detailed-view toggle; drop deprecated CSS util
Toggling the Detailed-view switch previously called the run-list URL helpers
with no arguments, discarding the active filters. Forward the params common to
both views (statuses, transform ids, start/end time, run methods) via a new
CommonRunListParams/pickCommonRunListParams helper; page and sort are dropped
since the two views paginate differently and use incompatible sort columns.
Also replace the deprecated CS.overflowHidden global utility class in the new
graph-run table with an inline overflow style prop.
* Track Detailed-view toggle in transform runs analytics
Emit a transform_runs_view_toggled simple event from the DetailedViewSwitch
toggle handler with the resulting view mode (detailed/grouped), so both the
grouped and individual run-list call sites are covered.
* Fix reprocess run status: don't re-show Scheduled after a member completes
- Derive the DAG scheduled state from this transform's own member run:
a member run row only exists once the transform starts, so its presence
means the transform already ran this cycle. Prevents isScheduled from
re-arming (and masking the completed status) after this transform's run
finishes while sibling DAG members are still running
- Keep the graph-run sidebar polling while the top-level run is canceling,
mirroring isPollingNeeded
* Cleanup
* Address PR review nits from Uladzimir
- Extract isActiveRunStatus helper and use it for the duplicated
started/canceling checks (RunButton, both isPollingNeeded, sidebar
shouldPoll, RunSection dagFinished)
- DetailedViewSwitch: replace styles prop with a CSS module class
- Convert renderRunName/renderRunEntityName to RunName/RunEntityName
components
- Drop redundant String() around an already-string cell value
- Cover RunButton onRun click and the run-options menu show/hide
* Adapt runs routes to the react-router migration on the rebased base
The base migrated transforms routes to element=/withRouteProps with
Outlet-based layouts. Make RunsPage an Outlet layout and render the runs
list pages via element in the specs.
* wip
---------
Co-authored-by: Oleh K <oleh@metabase.com>
* Treat a v7 Link with an empty `to` as a button, not a navigation
* Make a v7 replace to the current URL a no-op so URL-sync effects settle
* Provide router.listen on the v7 engine and keep the router shim stable
* Serialize a v3 location query into the search string on v7
* Fix lint and type issues in the v7 router shim
* Keep the v7 matched-route branch stable across renders
* Republish the v7 splat param under v3's `splat` name
* Enable the react-router v7 engine by default
* Commit the v7 location update synchronously instead of in a transition
* Mirror the v7 location into redux from the history subscription
* Format the v7 router provider
* Sort query keys when serializing a v3 location query
* Match v3 query encoding and resolve bare link paths against the root
* Keep v3 query encoding and leave absolute link urls alone
* Always push when a v7 link targets the current url
* conversation list endpoint
* add rtkquery endpoints
* generate conversation title
* better title generation logic, poll for title if not recieved before end of first message
* render conversation title in ui and show conversation list
* allow resuming historical conversations
* show exploration conversation history
* refresh current conversation ui for 409 case
* surface conversation titles in usage analytics
* add /metabot/conversation/:id route
* remove pending title data part, assume from lack of title data part we need to poll, fix loading a conversation that is mid-response
* title backfill job
* rename title event
* FE prevent polling multiple times for conversation title
* improve title generation prompt
* ci fixes
* automated review feedback
* e2e test fixes
* reduce diff size
* self-review
* Address PR review feedback
- api: pass profile-id through valid-usage-profile-id when ensuring a
conversation title, matching the backfill job
- title backfill: gate readiness on check-usage-limits! (instance-level)
so the job pauses and reschedules cleanly when a usage limit is hit
- title backfill: bound the title future .get with a 60s timeout so a
stuck generation can't block the backfill loop indefinitely
* Add Metabot option to the Transforms New menu (GDGT-1773)
Adds a "Metabot" entry as the first option in the Transforms "New" menu.
Clicking it opens the data-studio Metabot sidebar with a pre-seeded
"Create a transform that " prompt (not submitted), gated on Metabot access.
Also fixes the tiptap prompt input so a programmatically seeded prompt
places the caret at the end instead of the start (autofocus "end" plus a
caret-to-end after external setContent), which also improves the retry path.
* Cleanup
* Cleanup unnecessary code
* Test CreateTransformMenu Metabot item without mocking
Render the real CreateTransformMenu + Metabot via the shared metabot test
setup so the menu item, permission gating, and prompt seeding are exercised
end to end instead of through stubbed hooks.
Bump babashka postgresql pod to 0.1.6 so mage runs on Apple Silicon
The 0.1.0 pod manifest has no macos-aarch64 entry -- only macos-amd64.
Babashka downloads that x86_64 artifact and caches it under the aarch64 path
anyway, so on an Apple Silicon machine without Rosetta every mage command dies
before doing any work:
Cannot run program ".../mac_os_x/aarch64/pod-babashka-postgresql":
Exec failed, error: 86 (Bad CPU type in executable)
Clearing the pod cache does not help -- the re-download is the same x86_64
binary. 0.1.5 is the first release with a real macos-aarch64 artifact; this
bumps to the current 0.1.6, which resolves to a native arm64 executable.
0.1.6 keeps every platform 0.1.0 shipped (macos-amd64, linux-amd64,
windows-amd64) and adds macos-aarch64 and linux-aarch64, so no one loses
coverage. `pg/execute!` is the only pod fn used anywhere in the repo, and its
signature is unchanged.
Verified on arm64: the cached binary is now `Mach-O 64-bit executable arm64`,
`mage.bot.setup-worktree` loads, and `./bin/mage fix-kondo-ratchets` runs to
completion.
Claude-Session: https://claude.ai/code/session_01X5dVcVX59aLonqqsYsqU65
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Restore navigation blocking on the v7 engine via a blocking history
* Render the v7 Link on the v7 engine so links navigate in modals
* Scope v7 leave hooks to their route so within-route navigation is allowed
* Don't let a v7 Link with no destination navigate on click
* Type the app Link's `to` as optional to match the runtime contract
* Apply activeClassName on the v7 engine via NavLink
* Hide settings tab in TagEditorSidebar if there is no tags
* Don't crash query metadata fetch on un-normalizable queries
batch-fetch-query-metadata filtered empty queries before normalization,
but normalize-query returns {} when a query can't be normalized (e.g. a
native query with an incomplete field-filter/table template tag). That {}
then failed batch-fetch-query-metadata*'s strict ::query schema, 400ing the
whole request and tripping the query editor error boundary. Filter empties
after normalization too so an unnormalizable query just yields no metadata.
* Hide template-tag 'Try it' examples in transforms editor
The template-tag help sidebar's 'Try it' buttons load an example query
against the Sample Database. Transforms can't target the Sample Database,
so clicking 'Try it' switched the transform to an invalid database and left
the editor in a broken state. Add a canUseSampleDatabase QueryEditor option
(default on) that transforms set to false, which withholds the sample
database id so TagEditorHelp hides the 'Try it' buttons.
* Add regression tests for transform variables sidebar fixes
- batch-fetch-query-metadata skips queries that normalize to {} instead of
400ing the whole request (the P1 crash)
- TagEditorHelp hides the 'Try it' examples when no sample database is available
- TagEditorSidebar hides the Settings/Help tabs until the query has a variable
* Address PR review feedback
- Simplify batch-fetch-query-metadata with (keep (comp not-empty normalize-query))
- Replace the hidden-tabs settings behavior with an empty state in the Settings
tab (no more layout shift / auto-switch to Help); use the shared EmptyState
- Make sampleDatabaseId a required, DatabaseId-typed setup option
- Add a NativeQuerySidebar test asserting 'Try it' is hidden for transforms
(canUseSampleDatabase=false) and shown otherwise
- Re-add the getEditorOptions canUseSampleDatabase assertion
* split use-metabase-query.unit.spec.tsx into multiple files
* split test fixtures from setup.tsx
* move useMetabaseQuery tests into tests folder
* rename useMetabaseQuery hook test
* remove unused query test fixture fields
* rename mock dataset query helper
* rename bundle exports
* chore: tiny formatting changes
* test: rewrite stale query results test
* test: extract public api tests to type and runtime tests
* Convert the Table visualization from a class to a function component
* Stabilize the Table question so async metadata updates don't rebuild columns mid-hover
* Type useSyncedQuestion via VisualizationProps so the SDK d.ts doesn't leak internal types
* Author route paths in v7 syntax, shim to v3 in the facade
* Add use-v7-router flag and frontend engine reader
* Add flag-gated react-router v7 engine (provider, tree mapper, bridge)
* Re-point redux navigation and location to the v7 router
* Support both router engines in renderWithProviders
* Switch the app mount to the v7 router behind the flag
* Point setRouteLeaveHook no-op comment at DEV-2374
* Revoke schema-level grants when destroying Redshift workspace isolation
destroy-workspace-isolation! discovered schemas to revoke via
svv_relation_privileges, which only surfaces relation-level grants: a granted
input schema with no tables (or whose tables were dropped after provisioning)
kept its schema-level USAGE grant, and DROP USER then failed with 'user ...
cannot be dropped because the user has a privilege on some object'. Union in
svv_schema_privileges so schema-only grants are revoked too.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Rename workspace+det to workspace in the new destroy test
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Run Redshift workspace teardown statements individually, not as a JDBC batch
A failure now surfaces as the plain server error instead of the
BatchUpdateException wrapper ('Batch entry N ... Call getNextException to see
other errors in the batch.'); the transaction still aborts as a whole on the
first error.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Revert "Run Redshift workspace teardown statements individually, not as a JDBC batch"
This reverts commit 4bf2d241b5.
* Unwrap BatchUpdateException in Redshift workspace teardown
Keep the batch (there can be many granted schemas, two revokes each) but
rethrow .getNextException so failures read as the plain server error instead
of 'Batch entry N ... Call getNextException to see other errors in the batch.'
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* remove comment
* Unwrap BatchUpdateException across all workspace-isolation batch sites
New driver.u/batch-exception util returns the per-statement exception hiding
under .getNextException (or the throwable unchanged). Applied to every
workspace init/grant/destroy executeBatch site (h2, mysql, postgres,
clickhouse, redshift) — init sites unwrap before scrub-exceptions — so
failures read as the plain server error instead of 'Batch entry N ... Call
getNextException to see other errors in the batch.'
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Keep the batch exception on init-workspace-isolation! failures
The init statements carry secrets and pgjdbc echoes the aborted statement in
the BatchUpdateException message — scrub-exceptions redacts it, and the
GHY-3660 test (init-workspace-isolation-never-exposes-password-test) relies on
that redacted CREATE USER echo as proof the run reached the statement.
Unwrapping to the inner server error discarded the echo. grant/destroy keep
the unwrap: their statements carry no secrets and the plain server error is
the better message there.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Unwrap the batch envelope on init-workspace-isolation! too
All workspace batch sites now rethrow the underlying per-statement exception;
init scrubs the unwrapped exception so passwords stay out of the message. The
GHY-3660 test now proves the flow reached CREATE USER by asserting the
server's permission error instead of the redacted batch echo (which no longer
exists).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Test batch-envelope removal for grant! and password scrubbing after unwrap
- driver-level: a failed grant-workspace-read-access! must not surface a
BatchUpdateException (h2 runs locally; pg/mysql/clickhouse in driver CI;
non-batching impls pass trivially)
- unit: scrub-exceptions composed with batch-exception redacts the password
in the unwrapped per-statement message
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Batch init/grant/destroy on every SQL workspace driver
Redshift grant, and all three Snowflake / SQL Server workspace methods, now
run their statements as JDBC batches like the other drivers — one round trip,
per-statement exceptions unwrapped via driver.u/batch-exception (init unwraps
before scrub-exceptions). Redshift grants commit before grant! returns, so a
subsequent describe-database from another connection still sees the
privileges; the destroy pre-loop for foreign-grantor default-priv rows stays
per-statement on purpose (each must survive the others failing).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fold Redshift foreign-grantor revokes into the destroy batch
The per-statement autocommit pre-loop is gone: one transactional batch now
covers the FOR USER default-priv revokes, relation/schema revokes, and the
drops. A stale discovery row fails the whole destroy, which is fine — the
workspace layer records the failure and the deprovision retry re-discovers
fresh state.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Escape workspace passwords in DDL like every other interpolated value
CREATE/ALTER USER statements interpolated the generated password raw into the
SQL string on postgres, redshift, h2, clickhouse, and snowflake (mysql and
sqlserver already escaped). Escape with sql.u/escape-sql :ansi (quote
doubling, valid on all five engines) and scrub both the raw and escaped forms
from failure messages.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Unwrap batch exceptions from all JDBC connectors; fix sqlserver batch var scope
The grant-envelope test caught two real issues in driver CI:
- MariaDB's connector chains the per-statement exception via .getCause and
mssql-jdbc chains nothing at all, so batch-exception now falls back to the
cause and, failing that, rebuilds a plain SQLException with the same
message/SQLState/error code.
- mssql-jdbc executes a Statement batch in one T-SQL scope, so the two
sqlserver destroy statements both DECLAREing @sql collided; the variable
names are now unique.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Type-hint the rebuilt SQLException ctor to avoid reflection
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Return the batch exception as is when nothing is chained
mssql-jdbc's BatchUpdateException message is already the plain server error —
only pgjdbc adds the 'Batch entry N ... getNextException' wording. So the
unwrap falls back to the original exception instead of synthesizing a
SQLException, and the driver test asserts the envelope wording stays out of
the message rather than checking the exception class.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Interpolate the escaped password in Snowflake workspace user DDL
CREATE/ALTER USER spliced the raw password while every other workspace
driver uses the ANSI-escaped value; the escaped form was already
computed and scrubbed here.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
A STRUCT column's INFORMATION_SCHEMA data_type spells out its entire
nested schema, and the COLUMN_FIELD_PATHS LEFT JOIN repeats it on every
nested-leaf row -- O(n^2) bytes for a column with n leaves. Dynamic-key
STRUCTs (e.g. Cloud Functions log-sink tables whose JSON payloads use
user IDs as keys) reach thousands of leaves with ~270KB type strings;
one such table produced ~4,000 rows x 271KB = 1.09GB buffered in
sync-fields!'s partition-by, OOMing the instance mid-sync.
Only short type-string prefixes are ever consumed
(raw-type->database+base-type dispatches on prefixes up to 10 chars and
stores at most RANGE<TIMESTAMP> verbatim), so SUBSTR(..., 1, 200) in the
query -- applied server-side, before the wire -- is behaviorally
identical while collapsing response size, parse cost, and the buffered
partition (1.09GB -> ~2.8MB for the table above).
The test needs no dataset or network: it captures the compiled SQL via
*process-native* and asserts both data_type columns are truncated.
* remove the last test file from the legacy structure
* convert test table into format of newer tests
* remove test for undefined since this isn't a valid type for RowValue
Update E2E test timings with averaged data
Collected timing data from 7 days of test runs.
All specs with timing data were included (simple averages).
🤖 Generated automatically by update-e2e-timings workflow
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: GitHub Actions <actions@github.com>
* migrate TokenField
* fix RemappedOption type for Autocomplete ComboboxStringItem
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* address RecipientPicker review: memoize maps, share lookup/label helpers
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix single-value search: accept top match on Enter
Autocomplete only completes on Enter when an option is highlighted, but
search options load async so none is. Restore old TokenField behavior via
onKeyDown.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix letters swallowing
* addressing the PR's comments
* added missing tests
* introduce Autocomplete + refactoring
* apply filter by clicking on enter
* fix test
* refactor autocomplete
* refactor autocomplete
* fix edge case + added unit tests
* fix type
* fix test
* fix test
* get rid of lint suppression + allow autoselect on open for multiautocomplete
* revert autoselect destructuring, restore eslint suppression (tests failing)
* explain exhaustive-deps suppression in MultiAutocomplete effect
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Remove withRouter from production components
* Delete the public withRouter export
* Type the route-leave hook for PlainRoute and drop the route casts
* Add element/Outlet + useRoute support to the router facade
* Convert root route tree to element/Outlet
* Convert admin route tree to element/Outlet
* Convert data-studio, models and metrics route trees to element/Outlet
* Convert transforms, account, monitor and public route trees to element/Outlet
* Convert enterprise route trees to element/Outlet and drop dead library segment routes
* Convert tenants admin route tree to element/Outlet
* Convert redirect, createEntityIdRedirect and modalRoute to element
* Ban component= on routes with eslint
* Drop component= from the facade Route type and convert remaining routes
* Drop the component= eslint rule, the facade type now enforces it
* Remove the unused withOutlet helper
* Assert the required tenantId route param instead of coercing to NaN
* Bridge element={null} to a nothing-rendering component, matching v7
* Memoize router context value so element routes re-render once per navigation
* Convert data_apps and impersonation routes merged from master to element
* initial proof-of-concept for data apps
POC 2
POC 3
POC 4
Revert "POC 4"
This reverts commit 1f62e574dc.
POC 5
POC 6
* Expose all SDK components to child realm
* Setup app publishing and serving
* Refactor distortion helpers
* Adjust skill
* Add useQuestionQuery SDK hook (#74900)
Co-authored-by: Timofei <timofei@metabase.com>
* Mock METABASE_EMBEDDING_SDK_BUNDLE for data app sdk hooks/components
* Adjust skill for TS project
* Update skill to clearly mention TS
* Add blocked APIs list to skill
* Prototype schema exports and typed hooks for custom data apps (#74978)
* add prototype for schema generation
* add formatting for generated schemas
* add skill file
* useQuestionQuery should take only ID based on @sanex3339 feedback
* add segments, measures, tables and useMetabaseQuery hook
* Add useMetabaseQuery to endowments
* Fix queryDataset
---------
Co-authored-by: Timofei <timofei@metabase.com>
* Add ability to update app bundle
* Simplify hook
* Add spinner instruction
* Mode Data Apps to OSS
* Instruct to not not use blocked event handlers
* Improve API, add styles isolation, add router components (#75418)
* Prepare skills for data app template + adjust router components (#75490)
* Fixes after rebase
* Move migration to v63
* Move migration to v63 #2
* Fix config.edn
* Fix BE tests
* Fix team name
* Fix duplicated config
* Adjust version text in skill
* Fix data app bundle uploading after legacy client removal
* Remove `actions` from config edn
* Fix failing test
* Fix failing test #2
* Add custom actions for data apps (#75627)
* Fix imports
* Fix type annotations
* Adjust migration
* Adjust permissions
* Fixed iframe security
* Data Apps publishing via remote repository POC (#75702)
* Update gitignore with data-app.html
* [PoC] Type safety and modularity improvements for custom data apps (#75381)
* allow separate libraries in schemas
* add metric type checking
* add collections parameter
* check that segments and measures belongs to table in useMetabaseQuery
* require less type information for measures
* require less type information for segments
* use metric dataset endpoint when querying measures in metrics
* add validation for breakout buckets
* fix filter operation type scoping
* expose filter type helper for nicer autocompletions
* update skill file
* add breakout schema helper for better type errors
* update breakout schema helper to accept binning and bucket
* move fields that are not used in compile and runtime to comments to shorten type errors
* enforce data truthfulness in skill
* enforce semantic layer richness in skill
* handle dimension key collisions mapped from many tables
* update skill description
* prevent hallucinated measures and breakouts
* fix field serialization
* add guidance on using filter matcher
* add skill guidance on long results and units
* accept query object in StaticQuestion and InteractiveQuestion
* extract use-metabase-query to multiple files
* update skill file to use sdk question by default
* update skill file to avoid hallucinated derived calculations
* add new functions into sandbox
* kondo and cljfmt lint fixes
* update CSP rules to allow style tags for data apps
* address review feedback on scoping and metric filters
* address code review feedback
* Add aggregations support for breakouts in query hooks (#75723)
* add aggregations support
* add additional aggregation helpers
* Add metrics support in useMetabaseQueryObject hook (#75728)
* add metric queries
* indicate metric support in skill file for useMetabaseQueryObject
* add new version for release
* fix query dataset tests
* update release workflow temporarily
* update skill description
* update skill to use more generic examples
* hotfix: allow esm usage in vite dev mode
* skill file update for table caveat
* update sandbox
* fix type errors in useMetabaseQuery
* add v0.63.2 to release workflow
* update skill file to use the 63-data-apps version
* Move data apps to the EE directory and under `data-apps` token feature (#75871)
* remove skill file alias
* Add error handling for data apps
* Add spinner while iframe is loading
* Add OSS/no token upsell banner placeholder for data apps (#75893)
* ensure a fixed height is set if container has height
* Allow including everything in data and metric library in schema endpoint (#75944)
* add filter for including library and data collections
* address review on skill file
* remove id field from rendered comment
* remove useQuestionQuery and useMetricQuery hook
* Improve env files lookup in skills (#75955)
* Allow `unsafe-eval` for data app iframe (#76004)
* Fix lint
* Fixes after rebase
* Fix duplicated test
* Improve Providers and typesafety (#75934)
* chore: reformat single expression functions to arrow
* Use entity id in questionCollections when generating typed schemas (#76230)
* allow using entity ids for question collection filter
* guide the agent towards using entity ids
* add failing test for invalid entity id
* Make custom viz work in data apps (#76237)
* Allow blob: images in data-app CSP (EMB-1923)
SDK custom visualizations load their icon assets via blob: URLs (to avoid
the cross-origin auth issue), but the /embed/data-app/ route allowed
'unsafe-eval' without the blob: scheme in img-src, so icons were blocked
under a production CSP. Add blob: to img-src scoped to the data-app iframe
route only; the main app img-src stays strict.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* EMB-1880: support for `providerProps` in data apps
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Reduce schema bloat in dimensions of generated metrics when many tables are involved (#76003)
* Add useQuestionQuery SDK hook (#74900)
Co-authored-by: Timofei <timofei@metabase.com>
* Prototype schema exports and typed hooks for custom data apps (#74978)
* add prototype for schema generation
* add formatting for generated schemas
* add skill file
* useQuestionQuery should take only ID based on @sanex3339 feedback
* add segments, measures, tables and useMetabaseQuery hook
* Add useMetabaseQuery to endowments
* Fix queryDataset
---------
Co-authored-by: Timofei <timofei@metabase.com>
* [PoC] Type safety and modularity improvements for custom data apps (#75381)
* allow separate libraries in schemas
* add metric type checking
* add collections parameter
* check that segments and measures belongs to table in useMetabaseQuery
* require less type information for measures
* require less type information for segments
* use metric dataset endpoint when querying measures in metrics
* add validation for breakout buckets
* fix filter operation type scoping
* expose filter type helper for nicer autocompletions
* update skill file
* add breakout schema helper for better type errors
* update breakout schema helper to accept binning and bucket
* move fields that are not used in compile and runtime to comments to shorten type errors
* enforce data truthfulness in skill
* enforce semantic layer richness in skill
* handle dimension key collisions mapped from many tables
* update skill description
* prevent hallucinated measures and breakouts
* fix field serialization
* add guidance on using filter matcher
* add skill guidance on long results and units
* accept query object in StaticQuestion and InteractiveQuestion
* extract use-metabase-query to multiple files
* update skill file to use sdk question by default
* update skill file to avoid hallucinated derived calculations
* add new functions into sandbox
* kondo and cljfmt lint fixes
* update CSP rules to allow style tags for data apps
* address review feedback on scoping and metric filters
* address code review feedback
* Add aggregations support for breakouts in query hooks (#75723)
* add aggregations support
* add additional aggregation helpers
* Add metrics support in useMetabaseQueryObject hook (#75728)
* add metric queries
* indicate metric support in skill file for useMetabaseQueryObject
* add new version for release
* fix query dataset tests
* remove useQuestionQuery and useMetricQuery hook
* Data Apps publishing via remote repository POC (#75702)
* avoid duplicating dimension fields
namespace dimensions by table
add dimension id to mapping
pick from table fields for only allowed dimensions
fix incorrect type guards
allow querying metric using question as source
remove createMetabaseQuery from public interface
add sourceFieldId to metric
* doc generation type fixes
---------
Co-authored-by: Timofei <timofei@metabase.com>
* Add allowed hosts handling for data apps (#76094)
* Update version
* Properly include data app changes in `pull changes` modal (#76294)
* Move app template into main repo, handle skill versioning flow (#76288)
* feat(data-apps): allow passing down a card with viz settings (#75993)
support card prop in sdk question component
* Rename data app env variables and read them only in dev mode (#76604)
* Add custom data apps layout with sliding toolbar (#76600)
* Remove unused helper
* Dev env Sandboxing, Human Readable Errors and Copy Helper (#76597)
* Revert wrong rebase changes
* Improve creating filtering interfaces in custom data apps (#76401)
* update skill files to guide filter usages
* update skill file to prevent filtering mistakes
* move filtering ui patterns into a separate file
* do not treat mb_replace_me as valid creds
* use combobox for filters
* pass title false to sdk question
* combobox trigger before typing
* Add missing exported types
* Apply visualizations and visualization settings from SDK package (#76659)
* ask the agent to pull possible visualizations
* ask the agent to use card prop
* teach the agent when to use viz settings
* Support query-only SDK cards
* Remove cross references between data apps skills (#76661)
Remove data app skill cross references
* Move Data App Dev Vite App setup into SDK package (#76695)
* Don't navigate on ad-hoc question chart clicks (#76751)
* Fix sandboxing crashes and update skill file mistakes (#76858)
Fix data app sandboxing and skill guidance
* Block form submissions in data apps via CSP (#76882)
* Fix SDK theming hover text colors in generated data apps (#77083)
update guidance on colors
* Remove SDK question query prop for data apps use cases (#77080)
* Remove SDK question query prop
* update tests to use card prop instead
* Adopt Lib.createTestQuery for table sources when querying in data apps (#75957)
* Add base metabase-lib query implementation
* switch to Lib.createTestQuery DSL
* Load query metadata at runtime for metabase-lib instead of hard-coding (#76874)
* load query metadata dynamically
* accept id-only source references
* Wait for SDK login before resolving query metadata
* refactor: use useAsyncFn to handle states
* pass source-name to createTestQuery
* Add metric aggregations for useMetabaseQuery via Lib.createTestQuery (#76947)
* add metric sources for useMetabaseQuery via Lib.createTestQuery
* use MetabaseQueryObject instead of opaque type marker
* Fix metric schema and join source validation
* add metric library filtering to skill files
* make metric an aggregation expression instead of join sources
* mark tableId as never in joined fields
* export public MetabaseQueryObject type
* use camelCase for source name and field id
* Add initial data app e2e setup (#77101)
* Add sorting and fields to useMetabaseQuery via Lib.createTestQuery (#76966)
* add sorting, limiting and fields to useMetabaseQuery
* refactor: cleanup sorting and aggregations
* Fix metric order by validation
* Support saved questions for useMetabaseQuery via Lib.createTestQuery (#77073)
* add saved questions as useMetabaseQuery input
* remove extra operators for saved cards
* Add unit fe/be tests for data apps (#76979)
* Fix lint
* Use modern api in query-dataset.ts
* Fix BG color
* Fix execute-action tests
* Update SDK-data-apps version
* Move migrations to v64 folder
* Adjust router components to use metabase/router
* Allow opening a data app for any signed user
* Rename data app skills (#77266)
* Update DataAppLayout.tsx `go back` button logic
* Harden data apps skills and action validation errors (#77261)
* harden data apps skills and actions error validation
* do not display models as questions in data apps
* clarify action result types
* Agents should report errors when models, actions or questions cannot be schema generated (#77260)
* return an error if models or saved questions are invalid
* return an error if actions are invalid
* update skill file guidance on model generation
* fix model boundary
* Add data apps admin UI + routing/sandboxing/isolation e2e (#77271)
* Rename data-apps paths to /apps (#77298)
* Include all action-included models in the typed schema when a mutation is specified (#77291)
* add includeModels=true flag to include all models
* omit model columns and models without actions
* remove extra metadata
* Adjust admin UI for Data Apps (#77420)
* Add more e2e/unit tests for Data Apps (#77524)
* Update bun.lock
* Fix errors after rebase
* Parallelize dts rollup (#77663)
* revert EMB-1840 measures support in metric dataset endpoint
* chore: add data app module (#77705)
* test ci
* apply code review suggestion
* Setup separate entries for client and node dev data app entries (#77687)
* Fix type casts lint errors (#77696)
* Adjust data apps setup + add proper analytics setup (#77604)
* Extract common and scope helpers + tests for typed schema module (#77403)
* refactor: extract common and query param helpers
* refactor: use kebab-case for query params
* refactor: cleanup schema mapping
ci: fix module config
* refactor: rename schema scope helpers to scope.clj
* refactor: extract scope_test.clj
* Extract typed schema rendering helpers to render.clj (#77703)
* extract schema render function
* move runtime compact test out
* remove javascript and json endpoints from schema generator
* refactor: cleanup javascript.clj and render.clj
* Extract typed schema generators for tables and questions (#77729)
* refactor: extract schema generator modules
* remove question-details from model-schemas
* fix: batch queries in table and question schema generator
* remove N+1 in typed schema field table lookup
* Extract typed schema generators for models and actions (#77739)
* extract model schema generator
* batch model action querying to avoid n+1 queries
* Extract typed schema generator for metrics (#77806)
* Extract typed schema generators for models and actions (#77739)
* extract model schema generator
* batch model action querying to avoid n+1 queries
* extract metric schema generator
* test: simplify fields in metric_test fixtures
* refactor: clean up metric schema generation
* remove n+1 from metric field dimension read
* disable default temporal dimension breakout
* surface errors if metric schema cannot be built
* revert metrics/api.clj to master to remove outdated comments
* Fix routes
* Take a data app's slug from its directory name (#77688)
* Merge data-apps specific migrations into one
* ci: fix failing backend test and cljfmt lint
* Add a beta badge in data apps admin page heading (#77877)
marks the data app as beta
* Revert "Merge data-apps specific migrations into one"
This reverts commit 383a5fa28d.
* Revert "Revert "Merge data-apps specific migrations into one""
This reverts commit 948d904b62.
* ci: actionlint fix
* Fix data apps route
* Use getSubpathSafeUrl in attachIframeUrlMirror
* Fix comment
* Do not send apps with sync error for token check
* Add data app listing sidebar section for non-admins (#77977)
* show data apps in sidebar for non-admins
* add app icon to data apps
* use available=true to filter no sync error and enabled apps
* fix navbar keys in data apps
* test: use non-admin user to assert non-admin response
* fix toucan query in data apps listing endpoint
* re-export entity detail helpers and result column to metabot core
* open data apps in new tab from the sidebar
* Prevent race conditions in responses of useMetabaseQuery hook (#78023)
prevent stale responses in useMetabaseQuery
* ci: remove tools.util from module declaration
* Merge migrations into a single one
* reset release-embedding-sdk.yml to master
---------
Co-authored-by: Phoomparin Mano <poom@metabase.com>
Co-authored-by: Phoomparin Mano <phoomparin@gmail.com>
Co-authored-by: Nicolò Pretto <info@npretto.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Some cleanup of some false alarms I encountered in test environments while working on a different issue. Destination databases (those with `router_database_id` set) are reached only through their router: metadata, sync, permissions, etc. Only execution goes against the destination's connection. A `data_permissions` row pointing at a destination is never consulted -- it's already actively stripped.
There are test utilities that helpfully create in-test permissions for all databases created, so in a test environment it would look exactly like the invariant was being violated, but that was only because the test utilities deviated from production behavior. This brings the test behaviors into line with production (no permissions get written).
There should be no change in production behavior in this PR, but as a bonus, I figured that a safety guard filtering rows out on reads is not as good as filtering them out on reads *and* failing loudly on writes. I also added guards that a normal database can't be turned into a destination database, which was a behavior tests were relying on for easier setup (but again, the tests then deviated once more from production behavior).
Since 4.28 Liquibase ships an opt-out analytics feature that, on the first
migration run, fetches config from config-endpoint.liquibase.com and sends
anonymized usage events.
* Parse :absolute-datetime string literals in repr resolve
Metabot's representations repair pass wraps bare ISO `between` bounds as
`[:absolute-datetime {} "2024-01-01" :day]` to satisfy the lib `:between`
temporal schema, leaving the literal as a string. That form is schema-valid
(so it passes construct_notebook_query validation), but the JVM query
processor expects a real java.time value — optimize-temporal-filters throws
on a bare string, and the QP's own wrap-value-literals only parses temporal
strings that arrive as raw values, not ones already wrapped in
:absolute-datetime.
Parse those string literals into java.time values in resolve-query, where the
portable repr form becomes QP-ready pMBQL, alongside the existing
annotate-field-types pass.
* Address review: tighten docstrings, use update over assoc
- parseable-temporal-string?: contract-first two-line docstring
- parse-absolute-datetime-literals: move the "why" into inline comments at
the walk, keep the docstring to contract + idempotency
- use (update node 2 u.date/parse) instead of assoc
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Address review: year/year-month, fail-fast, export round-trip
- parse year/year-month :absolute-datetime literals too (u.date/parse maps
them to LocalDate), so they are QP-ready instead of crashing later
- fail fast on a structurally-ISO but invalid literal (e.g. "2024-13-45"):
re-throw as an :agent-error? ex-info so the LLM gets a clean message
- export: format java.time values back to ISO-8601 strings in
portable-repr-form so resolve -> export round-trips to the documented
string form
- tests: per-shape parsing (date/datetime/offset/year/year-month),
idempotency, invalid-literal, and export round-trip
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Move :absolute-datetime string parsing to the QP layer
Revert the resolve-layer parsing and instead teach wrap-value-literals to
parse string-valued :absolute-datetime clauses, not just raw values. The
repair pass wraps bare `between` bounds as `[:absolute-datetime {} "2024-01-01"
:day]`; wrap-value-literals previously only parsed temporal strings that
arrived as raw values, so a pre-wrapped string slipped through and
optimize-temporal-filters threw on the bare string.
The QP is the right layer: wrap-value-literals runs at execution time, where
both the report timezone (bound) and the comparison field's type are
available — so the parse is type- and timezone-aware for free, and the stored
query keeps its portable ISO-string form (no export round-trip to invert).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fail fast on malformed :absolute-datetime literals in resolve
Add a resolve-side validation pass that rejects temporal literals which pass
the schema's structural regex but are not real dates (e.g. "2024-13-45"), so
the agent gets an actionable error at construct time instead of a failure at
query execution.
Validation only — it does not coerce. The string -> java.time coercion stays
in the QP's wrap-value-literals, where the field type and report timezone are
available. Accepts every valid :absolute-datetime shape: date, datetime,
offset-datetime, year, and year-month.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Preserve unit and wrap mixed :between bounds for string :absolute-datetime
When re-parsing a string-valued :absolute-datetime in wrap-value-literals:
- Carry the clause's own temporal unit through instead of discarding it. The
comparison field's bucket still wins, but for an unbucketed field the literal's
unit is now the fallback (rather than :default), so a year/year-month literal
like [:absolute-datetime {} "2024" :year] keeps its bucket instead of silently
collapsing to a single instant.
- Wrap each :between bound independently so a mixed bound (string :absolute-datetime
on one side, :now / :relative-datetime / an already-parsed :absolute-datetime on
the other) still coerces the wrappable side instead of falling through unwrapped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Guard comparison literals and tidy :absolute-datetime validation
- wrap-value-literals: guard the field position in the :=/:!=/<…> arms so a
comparison with a wrappable literal on both sides can't bind a literal as the
field and leave it unparsed; it now falls through unchanged.
- resolve: validate :absolute-datetime literals via lib.walk/walk-clauses
(no tree rebuild, shares clause-of-type? detection with the QP) instead of
clojure.walk/postwalk.
- Move cross-subsystem rationale out of the string-valued-absolute-datetime
docstring into an inline comment.
- Add a test covering the two-literal comparison fall-through.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Parse fieldless :absolute-datetime comparisons and bound bad-literal errors
Wrap a string-valued :absolute-datetime on both sides of a comparison
(no field to wrap against) so each literal is parsed with its own unit
instead of leaving a bare string for later middleware to choke on.
Truncate the untrusted literal before it lands in the agent-facing
exception message and ex-data, so a pathological value can't flood logs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Treat :default field unit as unbucketed for :absolute-datetime literal fallback
A temporal field can carry :temporal-unit :default (add-default-temporal-unit
stamps it on :temporal/requires-default-unit drivers, and an LLM can author it
directly). The unit-fallback guard only skipped a nil unit, so a :default field
dropped the literal's own unit and collapsed e.g. [:absolute-datetime {} "2024"
:year] to a single :default-bucketed instant. Treat :default as unbucketed too.
* Validate :absolute-datetime literals with plain walk to keep resolve gate friendly
validate-absolute-datetime-literals walked the query with lib.walk/walk-clauses,
a mu/defn whose input schema validates the whole query against ::lib.schema/query.
Under test instrumentation that threw a raw "Invalid input" error on an otherwise-
malformed query (e.g. an :offset in :expressions), pre-empting the friendlier
query-not-runnable agent error the downstream canRun gate is meant to surface.
Use clojure.walk/postwalk and a bare [:absolute-datetime _ s _] shape check, matching
the sibling annotate-field-types pass, so only the literals are inspected.
* Use match/match-many for :absolute-datetime literal validation, add mixed-between raw-value test
Addresses PR review feedback: swap the manual walk/postwalk in
validate-absolute-datetime-literals for match/match-many, and cover a
:between with a raw (unwrapped) string bound plus a :relative-datetime
bound in wrap_value_literals_test.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Show honest "still working" state when a Git sync push stalls (GHY-4132)
A large push can stop reporting progress while the export is still running; the
backend then reports the task as :timed-out with ended_at nil. The push modal
treated that as neither success nor error and stayed frozen on "Exporting
content…" until the page was refreshed.
Render a distinct stalled state instead: drop the frozen progress bar and show
"Still working. No update for N minutes." (internationalized, singular/plural
aware). The counter derives from last_progress_report_at and stays fresh via the
existing 2s current-task poll — no separate ticker.
* Add throttled monotonic progress reporter for remote-sync tasks (GHY-4132)
* Ramp remote-sync export progress across plan + serialize phases (GHY-4132)
* Report remote-sync export commit checkpoint before push (GHY-4132)
* Report remote-sync push progress via JGit ProgressMonitor (GHY-4132)
* Fix throttle, dead 0.66 write, and push liveness in export progress (GHY-4132)
commit-staged! was passed a pre-forced report closure, so every push
ProgressMonitor tick bypassed the 10s throttle and hammered the DB. Force only
the one-shot commit checkpoint (0.8) explicitly in finish-commit!, and let
push ticks go through the throttled reporter.
The forced 0.66 serialize-boundary write ran after commit-staged! returned,
by which point the 0.8 checkpoint (and push ticks) had already advanced the
monotonic reporter past it, so it was silently dropped. Move it to the end of
each export's stage-fn, before the checkpoint/push run.
The push monitor only reported during the exactly-titled "Writing objects"
phase with a known positive total; a non-English JVM locale renames that
title and JGit may report an unknown (zero) total, so the push phase could go
silent for its whole duration and re-trip the stale-progress timeout. Report
on every update tick regardless of phase/total, falling back to the 0.8
push-start fraction — repeated equal values are monotonic no-ops but still
bump last_progress_report_at, throttled to at most once per 10s.
* Use dayjs diff for the stalled-progress minute counter (GHY-4132)
* Expose search's ranked set and page-hydration as separate steps
`search` already ran in two stages: rank the permission-filtered result set
(capped at max-filtered-results), then slice that set to the requested page
and hydrate only those rows. The comment on `search-results` spells out why
the slice comes first — hydration does per-row work, including the expensive
`can_write` check, so it should only touch rows that are actually returned.
Both stages were sealed inside `search`, so a caller holding several ranked
sets had no way to combine them and paginate the combination. Its only option
is to ask `search` for N hydrated pages and merge those, which hydrates N
pages to return one, and — because each page is sliced from its own ranking
before merging — cannot page the merged order correctly at all.
Split the two stages into `ranked-results` and `search-results` and export
both from the module API. `search` is now their composition and behaves
identically; no call site changes.
The motivating caller is metabot's multi-query search, which runs a search per
query and fuses the rankings with RRF. Fusing at the ranked-results seam lets
it hydrate only the page it returns, page the fused order coherently, and
report a total for the fused set.
* Remove comments from search core.clj
Removed comments explaining the purpose of `ranked-results` and `search-results`.
* kick ci
* Convert sandboxes and advanced_permissions plugin modules to TypeScript
* Address review comments: remove typecasting from sandboxes actions and unit spec
**Primary standard: the [`typescript-write`](../typescript-write/SKILL.md) skill.** Load it first — it defines the authoring rules this review enforces, alongside `frontend/CLAUDE.md` and `docs/developers-guide/frontend.md`.
Adherence to `typescript-write` is the **highest-priority** review dimension: rank any violation of its provisions above all other findings. Treat its **no-`any` hard rule** (no explicit *or* implicit `any` in new code) as **blocking**, and verify it with the LSP rather than by eye.
Adherence to `typescript-write` is the **highest-priority** review dimension: rank any violation of its provisions above all other findings. Treat its **no-`any` hard rule** (no explicit *or* implicit `any` in new code) as **blocking**. Use TypeScript LSP tools to inspect inferred types when available; otherwise rely on type-checking and linting.
@@ -13,7 +13,7 @@ description: Write TypeScript and JavaScript code following Metabase coding stan
- **New code must not introduce `any`, explicit or implicit.** No `any` annotations, no `as any` / `as unknown as`, no untyped parameters or returns that infer `any`, no implicitly-`any` destructures or array/object literals.
- **Untyped third-party / boundary values** must be typed at the boundary (a declared type, `unknown` + type guard, or a small typed wrapper) — never let `any` propagate inward.
- **Mandatory LSP verification.** After writing or editing any TS/TSX, inspect the changed symbols with the TypeScript Language Server (hover to read inferred types; `goToDefinition` to confirm sources) and run the project type-check.
- **Mandatory type verification.** Before finishing a TS/TSX change, run `bun run type-check-pure`. If TypeScript LSP tools are available, also inspect changed symbols with hover and go-to-definition; otherwise skip LSP check.
## Type tightening
@@ -26,6 +26,7 @@ description: Write TypeScript and JavaScript code following Metabase coding stan
- **Avoid non-null assertions (`!`)**. Prefer a guard, early return, or `?.`. Use `!` only when non-nullness is provably true and localized, with a comment.
- **Type guards belong in `frontend/src/metabase-types/guards/`**. Do not redefine them locally.
- **A cast you can't avoid needs a real justification comment.** The `metabase/no-unjustified-type-casts` rule accepts any preceding comment — state the actual reason the cast is safe. NEVER write the legacy `// Unjustified type cast. FIXME` placeholder; it exists only on casts that predated the rule, and copying it sneaks an unjustified cast past the linter. If you can't articulate why the cast is correct, the cast is wrong — fix the types.
{clojure.core/print{:message"Use metabase.util.log instead of clojure.core/print"}
{metabase-enterprise.entity-retrieval.core/search-unfiltered{:message"Permission bypass risk: library-index hits are unfiltered — resolve each entity and drop what the current user can't read"}
metabase.entity-retrieval.core/search-unfiltered{:message"Permission bypass risk: library-index hits are unfiltered — resolve each entity and drop what the current user can't read"}
metabase.entity-retrieval.mirror/search-unfiltered{:message"Permission bypass risk: library-index hits are unfiltered — resolve each entity and drop what the current user can't read"}
clojure.core/print{:message"Use metabase.util.log instead of clojure.core/print"}
clojure.core/println{:message"Use metabase.util.log instead of clojure.core/println"}
clojure.core/printf{:message"Use metabase.util.log instead of clojure.core/printf"}
clojure.core/prn{:message"Use metabase.util.log instead of clojure.core/prn"}
> Checkout the branch and run \`./backport.sh\` script. Force push your changes after cherry-picking. Check the box below when done.
> Checkout the branch and run \`./backport.sh\` script. Force push your changes after cherry-picking. Mark the PR as ready for review once conflicts have been addressed.
@${githubUsername} You have modified embedding code. Please make sure the PR title follows [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) style.
message:"⚠️ Main app initial bundle size (gzipped) increased by ${{ steps.compare.outputs.app_size_change_percent }}% compared to the ${{ github.base_ref }} branch (threshold: ${{ env.BUNDLE_SIZE_THRESHOLD_PERCENT }}%). This is the JavaScript every user downloads before the app renders, so check whether the new code can be loaded lazily instead."
include-log:false
- name:Comment on PR if app bundle size has decreased
- name:Check out the code using the provided branch
uses:actions/checkout@v6
with:
ref:${{ inputs.branch }}
ref:${{ needs.prepare-release.outputs.branch }}
fetch-depth:0
fetch-tags:true
- name:check tags
run:|
if [ $(git tag -l | grep $tag$) ]
if git tag -l | grep -q "${tag}$"
then
echo "::error::Tag '${{ env.tag }}' already exists. If you expect to run this workflow with the same tag, please remove the tag first and rerun this workflow again."
echo "::error::Tag '${{ env.tag }}' already exists. If you expect to run this workflow with the same tag, please remove the tag first and rerun this workflow."
exit 1
else
exit 0
fi
test:
needs:check-git-tag
needs:
- prepare-release
- check-git-tag
runs-on:${{ vars.DEFAULT_RUNNER_KEY }}
timeout-minutes:20
steps:
- name:Check out the code using the provided branch
uses:actions/checkout@v6
with:
ref:${{ inputs.branch }}
ref:${{ needs.prepare-release.outputs.branch }}
- name:Prepare front-end environment
uses:./.github/actions/prepare-frontend
@@ -262,14 +357,16 @@ jobs:
run:bun run embedding-sdk:test-unit
build-sdk:
needs:[test, determine-sdk-version]
needs:
- prepare-release
- test
runs-on:${{ vars.DEFAULT_RUNNER_KEY }}
timeout-minutes:20
steps:
- name:Check out the code using the provided branch
uses:actions/checkout@v6
with:
ref:${{ inputs.branch }}
ref:${{ needs.prepare-release.outputs.branch }}
fetch-depth:0
fetch-tags:true
@@ -279,11 +376,9 @@ jobs:
- name:Prepare back-end environment
uses:./.github/actions/prepare-backend
- name:Bump published npm package version
# NOTE: this should happen before "Build SDK bundle" as we inject SDK version into the code during build step
console.log(`Could not look up the author of ${BREAKING_COMMIT}: ${error.message}`);
}
}
const failedJobsList = failedJobs.map(job => ({
"type": "rich_text_section",
"elements": [
@@ -87,9 +115,9 @@ jobs:
]
}));
await slack.chat.postMessage({
channel:'engineering-ci',
text:'Failing tests',
await sendSlackMessage({
channelName:'engineering-ci',
message:'Failing tests',
blocks:[
{
"type": "header",
@@ -107,7 +135,7 @@ jobs:
"type": "section",
"text": {
"type": "mrkdwn",
"text": `Commit <https://github.com/metabase/metabase/commit/${BREAKING_COMMIT}|${BREAKING_COMMIT.slice(0,7)}> by ${AUTHOR} has failing <${FAILED_RUN_URL}|${FAILED_RUN_NAME}> tests on <https://github.com/${owner}/${repo}/commits/${BRANCH}|\`${BRANCH}\`>`
"text": `Commit <https://github.com/metabase/metabase/commit/${BREAKING_COMMIT}|${BREAKING_COMMIT.slice(0,7)}> by ${AUTHOR} (${mentionUserByGithubLogin(AUTHOR_LOGIN)}) has failing <${FAILED_RUN_URL}|${FAILED_RUN_NAME}> tests on <https://github.com/${owner}/${repo}/commits/${BRANCH}|\`${BRANCH}\`>`
• custom — one-off branch (esbuild, data-apps): publish under its own prerelease id
type:choice
required:true
options:
- alpha
- beta
- preminor
- patch
- custom
prerelease_id:
description:"release_type=custom only, and only on the FIRST cut of a one-off branch (e.g. data-apps). Becomes the prerelease id in the version (0.62.5-data-apps.0) and the npm dist-tag (62-data-apps). Leave empty on later bumps - the id is reused from the current version."
type:string
required:false
concurrency:
# one version-bump in flight per branch at a time, to avoid racing PRs
# A leftover branch means an earlier bump to this same version was
# abandoned. Say so, rather than pushing a near-duplicate alongside it.
if git ls-remote --exit-code --heads origin "$BUMP_BRANCH" >/dev/null 2>&1; then
echo "::error::'$BUMP_BRANCH' already exists on origin - an earlier bump to this version was never merged. Close its PR, delete the branch, then re-run."
exit 1
fi
git checkout -b "$BUMP_BRANCH"
git add "$package_template"
git commit -m "chore(sdk): bump SDK version to ${{ steps.metadata.outputs.new_version }} (${{ inputs.release_type }})"
| Tag as `latest` | `${{ steps.metadata.outputs.tag_as_latest }}` |
`distTag` and `tagAsLatest` are computed from the branch and release type. `tagAsLatest` is `true` only when patching the current gold major (`vars.LATEST_MAJOR_VERSION`).
Merging this PR (with the `sdk-release-trigger` label) triggers [release-embedding-sdk.yml](../actions/workflows/release-embedding-sdk.yml), which builds, publishes to npm, and tags the release.
_Opened by the [SDK Version Bump PR workflow](../actions/workflows/sdk-version-bump-pr.yml), triggered by @${{ github.event.sender.login }}._
if [[ "$dir_version" -lt "$DIR_LAYOUT_FLOOR" ]]; then
echo "ERROR: ${file} targets v${dir_version}, but per-feature directories start at v${DIR_LAYOUT_FLOOR} — a pre-${DIR_LAYOUT_FLOOR} changeset goes in resources/migrations/${dir}_update_migrations.yaml" >&2
(format "Migration '%s' has primaryKeyExists precondition without tableName" (:id changeSet))
(format "Migration '%s' has %s precondition without %s: without it Liquibase snapshots the entire schema to answer the check (~2s per precondition on a large postgres appdb)"
(testing"foreignKeyConstraintExists without foreignKeyTableName is rejected"
(is-thrown-with-error-info?
"Migration 'v49.2024-01-01T10:30:00' has foreignKeyConstraintExists precondition without foreignKeyTableName: without it Liquibase snapshots the entire schema to answer the check (~2s per precondition on a large postgres appdb)"
(testing"indexExists without tableName is rejected, including in flat-map not form"
(is-thrown-with-error-info?
"Migration 'v49.2024-01-01T10:30:00' has indexExists precondition without tableName: without it Liquibase snapshots the entire schema to answer the check (~2s per precondition on a large postgres appdb)"
(testing"primaryKeyExists without tableName is still rejected (previous rule subsumed)"
(is-thrown-with-error-info?
"Migration 'v49.2024-01-01T10:30:00' has primaryKeyExists precondition without tableName: without it Liquibase snapshots the entire schema to answer the check (~2s per precondition on a large postgres appdb)"
- [Metabase's officially supported databases](../databases/connecting.md#connecting-to-supported-databases) (with the exception of SQLite and H2).
Metabase Cloud doesn't support [community database drivers](../developers-guide/community-drivers.md), or file-based databases (SQLite and H2), because there's currently no file storage available.
Metabase Cloud doesn't support [community database drivers](../developers-guide/community-drivers.md), or file-based databases (SQLite), because there's currently no file storage available.
{% include plans-blockquote.html feature="Loading from a configuration file" self-hosted-only="true" %}
On self-hosted [Pro](https://www.metabase.com/product/pro) and [Enterprise](https://www.metabase.com/product/enterprise) plans, Metabase supports initialization on launch from a config file named `config.yml`. The config file should be located at:
On self-hosted [Pro](https://www.metabase.com/product/pro) and [Enterprise](https://www.metabase.com/product/enterprise) plans, Metabase supports initialization on launch from a config file. The config file should be located at:
-The current directory (the directory where the running Metabase JAR is located).
-A file named `config.yml` or `config.yaml` in the current directory (the directory where the running Metabase JAR is located).
- The path specified by the [`MB_CONFIG_FILE_PATH`](./environment-variables.md#mb_config_file_path) environment variable.
The settings in the config file work the same as if you'd set the settings in the Admin Settings in your Metabase. Settings defined in this configuration file will update any existing settings. If, for example, a database already exists (that is, you'd already added it via the initial set up or **Admin settings** > **Databases**, Metabase will update the database entry based on the data in the config file). Which means: if you define a setting in the config file, and then later change that setting in your Metabase application, keep in mind that the config file will overwrite that change whenever Metabase restarts. Let's reiterate that in a blockquote:
The AI provider and model for Metabot. Format: provider/model-name, e.g. `anthropic/claude-haiku-4-5`, `openai/gpt-5.4`, `openrouter/anthropic/claude-haiku-4.5`.
Refresh cached results this fraction of their cache duration before they expire, so requests keep being served from cache instead of waiting for a recomputation. Set to 0 to only refresh once results have expired.
### `MB_QUERY_CACHING_MAX_KB`
- Type: integer
@@ -1599,9 +1751,9 @@ The absolute maximum time to keep any cached query results, in seconds.
### `MB_QUEUE_BACKEND`
- Type: string
- Default: `quartz`
- Default: `null`
Which queue backend to use. Valid values: `quartz`, `memory`.
Which queue backend to use. Valid values: `quartz`, `memory`. When unset, defaults to `quartz`, or to `memory` when the task scheduler is disabled (`MB_DISABLE_SCHEDULER`).
### `MB_QUEUE_MAX_RETRIES`
@@ -1881,8 +2033,7 @@ on your IdP, this usually looks something like `http://www.example.com/141xkex60
@@ -82,7 +82,7 @@ You can use [Markdown](https://commonmark.org/help/) to format the text in your

To learn more, see [Fun with Markdown in your dashboards](https://www.metabase.com/learn/metabase-basics/querying-and-dashboards/dashboards/markdown).
To learn more, see [Format text with Markdown](./markdown.md).
### Including variables in text cards
@@ -321,12 +321,14 @@ See [Caching per dashboard](../configuring-metabase/caching.md#dashboard-caching
## Sharing dashboards with public links
If your Metabase administrator has enabled [public sharing](../embedding/public-links.md) on a saved question or dashboard, you can go to that question or dashboard and click on the **sharing** icon to find its public links.
If your Metabase administrator has enabled [public sharing](../embedding/public-links.md) on a saved question or dashboard, you can go to that question or dashboard and click on the **Share** icon to find its public links.

Public links can be viewed by anyone, even if they don't have access to Metabase. You can also use the public embedding code to embed your question or dashboard in a simple web page or blog post. Check out examples of simple apps with embedded dashboards in our [embedding-reference-apps repository](https://github.com/metabase/embedding-reference-apps). To learn more about [embedding](../embedding/introduction.md), check out our article on [How to use Metabase to deliver analytics to your customers](https://www.metabase.com/learn/metabase-basics/embedding/overview), as well as an article on how to combine branding, Single Sign-On, full app embedding, and row and column security to deliver [multi-tenant, self-service analytics](https://www.metabase.com/learn/metabase-basics/embedding/multi-tenant-self-service-analytics).
Admins can also [create an account](../people-and-groups/managing.md#creating-an-account) for someone and point them to a specific dashboard or question. On a dashboard or question, click the **Sharing** icon and select **Invite someone to view this**.
## Exporting results from a dashboard
You can export the entire dashboard as PDF, or export results of individual questions on the dashboard. See [Exporting results](../questions/exporting-results.md).
@@ -34,7 +34,7 @@ People in the Data Analyst group will have table metadata and data structure acc
_Data Studio > Tables_
You can see all tables in all databases connected to your Metabase in **Data Studio > Tables**, together with their owner, row count, and published state. For now, Metabase only displays row counts for PostgreSQL tables.
You can see all tables in all databases connected to your Metabase in **Data Studio > Tables**, together with their owner, visibility layer, row count, and published state. For now, Metabase only displays row counts for PostgreSQL tables.
You can search for table names, but the search will only match beginnings of words in table names. So for example, if you search for "base", results will include names like "Baseball stats" and "All your base are belong to us", but the results won't include tables like "Metabase secrets".
- **Metabase Cloud**: Basic transform functionality on Metabase Cloud - Starter, Pro, or Enterprise - comes with an additional small fee per successful transform run, see [Pricing](https://www.metabase.com/pricing).
Only people logged in with an email of a [Metabase Store admin](../../cloud/accounts-and-billing.md#add-managers) (not just Metabase _instance_ admins) can [enable basic transforms](./transforms-overview.md#enable-transforms).
Only people logged in with an email of a [Metabase Store admin](../../cloud/accounts-and-billing.md#add-managers) (not just Metabase _instance_ admins) can [enable basic transforms](./transforms-overview.md#enable-transforms).
### Cancel basic transforms
@@ -65,6 +65,8 @@ The Advanced transforms add-on comes with an additional charge per successful tr
To enable Advanced transforms functionality, you need to have [Basic transforms](#basic-transforms) already, see [Enable basic transforms](#enable-basic-transforms).
For the full self-hosted setup see [Set up transforms on a self-hosted Metabase](transforms-overview.md#set-up-transforms-on-a-self-hosted-metabase).
There are two ways to enable Advanced transforms:
- **From your Metabase instance**: you can navigate to a feature requiring advanced transforms (like Python transforms or transform inspector), and follow the prompts to upgrade.
@@ -38,7 +38,7 @@ Jobs have two components: schedule and tags.
- **Schedule** determines when the job will be executed: daily, hourly, etc. You can specify a custom cron schedule (e.g. "Every weekday at 9:05 AM"). The times are given in your Metabase's system timezone.
- **Tags** determine _which_ transforms a job runs, not when the job runs. For example, you can create a `Weekdays` tag, add that tag to a few transforms, then create a job that runs all the transforms with the `Weekdays` tag every weekday at 9:05AM.
Jobs will run all transforms tagged with any of the tags, plus any transforms that the tagged transforms depend on, see [Jobs will run all dependent transforms](#jobs-will-run-all-dependent-transforms).
Jobs will run all transforms tagged with any of the tags, plus any transforms that the tagged transforms depend on that aren't already up to date, see [Jobs include all dependent transforms](#jobs-include-all-dependent-transforms).
You can see which transforms a job will run and in which order on the job's page.
@@ -104,13 +104,22 @@ To delete a job:
2. Find the job you want to delete and click the **three dots** icon to the right of the job's name.
3. Select **Delete**.
## Jobs will run all dependent transforms
## Jobs include all dependent transforms
If one transform depends on another, Metabase will run the dependency first, even if that transform isn't tagged in the job. So if transform B depends on A, Metabase will first run A, even if A doesn't have a tag.
If one transform depends on another, Metabase adds the dependency to the job and runs it first, even if that transform isn't tagged in the job. So if transform B depends on A, Metabase makes sure A is up to date before running B.
This means that you can explicitly tag transform A to run daily, and transform B hourly, but because transform B depends on transform A, transform A will _also_ run hourly (in addition to daily), despite not having the tag.
## Jobs skip dependencies that are already up to date
You can see which transforms a job will run (and in which order) on the job's page in **Data Studio > Jobs**.
Metabase won't re-run a dependency that's still fresh. That way, a transform pulled into a frequent job doesn't get rebuilt more often than its own schedule calls for. So, for example, if you tag transform A `daily`, and transform B `hourly`, and hourly B depends on daily A, then A runs on its own daily schedule. The hourly job that runs B won't rerun A every hour.
Metabase skips a dependency when the dependency:
- Has its own tags and jobs, and none of those jobs' schedules have come due since the dependency last ran successfully.
- Has no tags at all, and it's already run successfully at least once.
## See which transforms a job will run
The job's page in **Data Studio > Jobs** lists every transform the job will run and in which order. The **Notes** column tells you when Metabase will skip a dependency, and flags dependencies that have no schedule of their own.
## Runs
@@ -120,4 +129,11 @@ You can see all past and current transform runs (both manual and scheduled) by g
You can click on any transform run to see more details about the run, like the error logs. To go to the transform definition from the transform run page, click on the icon next the transform name in the right sidebar.
The "Tags" column in the **Runs** table will only show the transform's specific tags. But the run might not have anything to do with those tags. Another job with different tags could have run the transform because [jobs will run _all_ dependent transforms](#jobs-will-run-all-dependent-transforms).
The "Tags" column in the **Runs** table will only show the transform's specific tags. But the run might not have anything to do with those tags. Another job with different tags could have run the transform because [jobs include all dependent transforms](#jobs-include-all-dependent-transforms).
## Emails about failed transforms
If your Metabase is [set up to send email](../../configuring-metabase/email.md), Metabase will let people know when transforms fail.
- **Individual transform failures**: when transforms fail during a scheduled job run, Metabase emails the last person to edit each failed transform (or the transform's creator, or admins if neither is active). Each email covers a single job run.
- **Daily digest of job failures**: each morning, Metabase emails all admins a summary of the scheduled job runs that failed or timed out the previous day. Manual runs aren't included in the digest, and Metabase skips the email entirely if no scheduled runs failed.
These settings can also be configured in the Metabase UI at **Admin** > **Settings** > **Python Runner**. Note that environment variables take precedence over UI settings.
| `MB_PYTHON_STORAGE_S_3_PATH_STYLE_ACCESS` | (Optional) Set to `true` for S3-compatible services like MinIO or LocalStack |
`MB_PYTHON_STORAGE_S_3_CONTAINER_ENDPOINT` is the host Metabase signs into the presigned URLs the runner uploads to and downloads from. Set it when the runner resolves storage by a different hostname than Metabase does.
## Using Docker Compose
Docker Compose simplifies managing multiple containers. Below are example configurations for different scenarios.
Below are example configurations for different scenarios.
### Simple setup with LocalStack
### Self-hosted storage with MinIO
For simple setup and local testing, you can use LocalStack to simulate S3:
For production, use an external S3-compatible storage service:
[MinIO](https://min.io/) is an S3-compatible server you can run yourself, which makes it a common pick for self-hosted Metabase. This Compose file runs Metabase, the Python runner, MinIO, and a short-lived container that creates the bucket Metabase will use — neither MinIO nor Metabase creates that bucket for you.
until mc alias set local http://minio:9000 "${MINIO_ROOT_USER}" "${MINIO_ROOT_PASSWORD}" >/dev/null 2>&1; do
echo "waiting for minio..."; sleep 2;
done
mc mb --ignore-existing local/metabase-python-runner
volumes:
minio-data:{}
```
Create a `.env` file with your configuration:
Both S3 endpoint variables point at `http://minio:9000` because Metabase and the runner share the Compose network and reach MinIO by the same hostname. `MB_PYTHON_STORAGE_S_3_CONTAINER_ENDPOINT` sets the host Metabase signs into the presigned URLs the runner uploads to and downloads from — if your runner resolves storage by a different hostname than Metabase does, set this to the runner's view of it.
Create a `.env` file:
```bash
AUTH_TOKEN=your-secure-token-here
S3_ENDPOINT=https://s3.amazonaws.com
S3_BUCKET=metabase-python-runner
S3_REGION=us-east-1
S3_ACCESS_KEY=your-access-key
S3_SECRET_KEY=your-secret-key
MINIO_ROOT_USER=minioadmin
MINIO_ROOT_PASSWORD=minioadmin
```
Then run:
Then bring everything up and try a transform:
```bash
docker compose up -d
```
1. Put a strong shared secret in `AUTH_TOKEN` (the runner and Metabase have to agree on it):
```bash
openssl rand -hex 32
```
2. Start the stack:
```bash
docker compose up -d
```
3. You'll need a Pro or Enterprise license with the [Advanced transforms add-on](addons.md). In Metabase, [enable transforms](transforms-overview.md#enable-transforms), then [create a Python transform](python-transforms.md#create-a-python-transform) and click **Run**.
4. Open **Data Studio > Jobs > Runs** and find your run. If it failed, the run's logs will usually tell you whether Metabase couldn't reach the runner, or the runner couldn't reach MinIO.
@@ -108,7 +108,7 @@ To use functions or classes from your Python library:
## Incremental Python transforms
By default, Metabase will process all the data in all input tables, drop the existing target table (if one exists), and create a new table with the processed data. You can tell Metabase to only write**new** data to your target table by marking your transform as incremental.
By default, Metabase will process all the data in all input tables, drop the existing target table (if one exists), and create a new table with the processed data. You can tell Metabase to only process**new** data by marking your transform as incremental.
### Prerequisites for incremental transforms
@@ -120,7 +120,8 @@ To make a Python transform incremental:
1. Go to the transform's page in **Data studio > Transforms**.
2. Switch to **Settings** tab.
3. In **Column to check for new values**, select the column in one of the source tables that Metabase should check to determine which values are new. Only some columns are eligible. See [prerequisites for incremental transforms](./transforms-overview.md#prerequisites-for-incremental-transforms).
3. In **Field to check for new values**, select the field in one of the source tables that Metabase should scan to determine which records are new or changed. Only some fields are eligible. See [prerequisites for incremental transforms](./transforms-overview.md#prerequisites-for-incremental-transforms).
4. (Optional) To update matching rows instead of appending new ones, [add a merge key](./transforms-overview.md#add-merge-keys-to-upsert-rows).
@@ -92,7 +92,7 @@ See [Run a transform](transforms-overview.md#run-a-transform). You'll see logs f
## Incremental query transforms
By default, on every transform run after the first one, Metabase will process all the data in all input tables, then drop the existing target table, and create a new table with the processed data. You can tell Metabase to only write**new** data to your target table by marking your transform as incremental.
By default, on every transform run after the first one, Metabase will process all the data in all input tables, then drop the existing target table, and create a new table with the processed data. You can tell Metabase to only process**new** data by marking your transform as incremental.
### Prerequisites for incremental transforms
@@ -102,6 +102,8 @@ Your data has to have certain structure for incremental transforms to work. See
For a transform to run incrementally, you'll need to pick a column ("checkpoint") that Metabase needs to check for new values. Then, behind the scenes, Metabase will add a filter around your transform query that will filter the results of the query for values greater than the last written checkpoint value.
If you set a [merge key](transforms-overview.md#add-merge-keys-to-upsert-rows), Metabase updates the target rows that match the key instead of adding new ones.
### Make a query transform incremental
#### Use table variables for incremental SQL transforms
@@ -152,7 +154,8 @@ To make a query transform incremental:
1. Go to the transform's page in **Data studio > Transforms**.
2. Switch to **Settings** tab.
3. In **Column to check for new values**, select the column in one of the source tables that Metabase should check to determine which values are new. Only some columns are eligible. See [prerequisites for incremental transforms](./transforms-overview.md#prerequisites-for-incremental-transforms).
3. In **Field to check for new values**, select the field in one of the source tables that Metabase should scan to determine which records are new or changed. Only some fields are eligible. See [prerequisites for incremental transforms](./transforms-overview.md#prerequisites-for-incremental-transforms).
4. (Optional) To update matching rows instead of appending new ones, [add a merge key](./transforms-overview.md#add-merge-keys-to-upsert-rows).
@@ -56,6 +56,16 @@ Permission configuration for transform depends on your plan.
- To **see** the list of transforms on your instance, people need to be able to access Data Studio, so they need to be either an Admin or a member of the special [Data Analyst group](../../people-and-groups/managing.md).
- To **execute** transforms on a database, people need to be either Admins on belong to the special [Data Analyst group](../../people-and-groups/managing.md). Additionally people need to have the [Transform permissions](../../permissions/data.md) for that database.
## Set up transforms on a self-hosted Metabase
If you run Metabase yourself, here's the path to working transforms:
1.**Check your plan.** [Basic transforms](addons.md#basic-transforms) (query-based) are included on self-hosted Metabase. [Advanced transforms](addons.md#advanced-transforms) — Python transforms, the [transform inspector](transform-inspector.md), and [writable connections](../../databases/writable-connection.md) — need a self-hosted Pro or Enterprise plan with the Advanced transforms add-on.
2.**Connect a database that can write.** Transforms create and replace tables in your database, so the database user needs create, drop, and write privileges. See [Database users, roles, and privileges](../../databases/users-roles-privileges.md). For cleaner isolation, point the transform at a [writable connection](../../databases/writable-connection.md). Only some databases [support transforms](#databases-that-support-transforms).
3.**For Python transforms, set up a runner.** Query-based transforms need nothing extra — they run inside your database. Python transforms run in a separate execution environment, so you'll point Metabase at a [self-hosted Python runner](python-runner.md) backed by S3-compatible storage (AWS S3, MinIO, and so on).
4.**[Enable transforms](#enable-transforms)** in Data Studio.
5.**Create and run a transform.** [Create a query-based transform](query-transforms.md#create-a-query-based-transform) or a [Python transform](python-transforms.md#create-a-python-transform), run it manually, and check the result under **Runs**. Once it works, [schedule it with jobs](jobs-and-runs.md).
## Enable transforms
Before you can start writing transforms, you'll need to enable transforms in your Metabase instance.
@@ -220,13 +230,17 @@ Metabase will track transform dependencies and execute transforms in a reasonabl
On Metabase Pro or Enterprise plans, you can see the transform dependencies graph by going to **Data Studio > Transforms** and go to the **Dependencies** tab.
If a job includes a transform that depends on a table created by another transform, then the job will run all the tagged transforms and their dependencies, even if they lack tags, see [Jobs and runs](jobs-and-runs.md) for more information.
If a job includes a transform that depends on a table created by another transform, then the job will run all the tagged transforms, plus any of their [dependencies that aren't already up to date](jobs-and-runs.md#jobs-include-all-dependent-transforms).
## Incremental transforms
_Data Studio > Transforms > Settings_
Incremental transforms only append new data since the previous transform run. For example, you might have new transaction data coming in every day, and run the transform nightly. With each run, the incremental transform would only write the rows added after the previous run the night before.
Incremental transforms only process the data that's new since the previous transform run. For example, you might have new transaction data coming in every day, and run the transform nightly. With each run, the incremental transform would only handle the rows added after the previous run the night before.
By default, Metabase appends those rows to the target table. If your source tables track changes to a record over time, you can use an incremental strategy by setting a [merge key](#add-merge-keys-to-upsert-rows) so that Metabase updates the existing rows instead of adding duplicate rows.
In this case, the checkpoint field decides which rows count as new, and the merge key decides whether Metabase appends those rows or updates the matching ones.
### Prerequisites for incremental transforms
@@ -234,6 +248,30 @@ Incremental transforms only append new data since the previous transform run. Fo
- The checkpoint column has to have increasing values, like a sequential ID or timestamp column. Metabase will determine what "new" data is by looking for values that are _greater than_ already-written checkpoint values.
- Your schema is stable, meaning that the structure of the tables is not going to change from run to run.
### Add merge keys to upsert rows
By default, Metabase will append rows when outputting a transform. You can add a merge key to upsert (update or append) instead.
Why you'd want to add a merge key: some source tables get a new row each time a record changes. So for example the same order shows up once as `created`, again as `paid`, and again as `shipped`. If you append those rows, you end up with three rows for the same order. That may be what you want. But if instead you want to update these records in place, you can select one or more columns as merge keys, like an order ID column. If a merge key is set, Metabase will first try to update existing records that match the key. If no match is found, Metabase will append a new record.
To add a merge key:
1. Go to the transform's page in **Data studio > Transforms**.
2. Switch to the **Settings** tab and turn on **Only process new data**.
3. In **Merge key**, choose the columns that identify a record. If the target table already exists, you'll pick from a list of its columns. If the transform hasn't run yet, there's no table to pick from, so type each column name and press comma or enter. In that case, type the column's name in the database, which may differ from the display name you'd see in the list.
The merge key refers to columns in the _target_ table (the columns your transform outputs), not columns in the _source_ tables. If identifying a record requires a combination of columns, like an `id` plus a `region`, pick more than one column as merge keys.
Merge keys need a checkpoint field that increases every time a row is written, like an `updated_at` timestamp, otherwise the transform never sees the change.
### Reprocess all data to rebuild the target table
Once an incremental transform has run, its **Settings** tab shows **Last processed [checkpoint field]**, followed by the highest checkpoint value Metabase has written so far. That value is how Metabase knows where to pick up, since each run only handles rows past it.
To clear that value and reprocess all rows, click **Reprocess all data**. The next run will then rebuild the target table from scratch, processing every row instead of only the new ones. The run doesn't start right away; you'll need to run the transform manually, or wait for the transform's next scheduled run.
Changing the checkpoint field also resets the stored value, so a transform will rebuild from scratch on its next run after you pick a different field to use for checkpointing.
### Make a transform incremental
Incremental transforms work differently for query-based transforms and Python transforms, so see [incremental query transforms](query-transforms.md#incremental-query-transforms) and [incremental Python transforms](./python-transforms.md#incremental-python-transforms) for more information.
@@ -48,7 +48,7 @@ The databases listed below have official drivers maintained by the Metabase team
If you don't see your database listed here, see [Community drivers](../developers-guide/community-drivers.md).
As of version 46.6.4, Metabase [no longer supports H2 connections](https://www.metabase.com/blog/vulnerability-post-mortem). But Metabase still ships with an H2 database to include an embedded application database, as well as to provide some sample data out of the box.
As of version 46.6.4, Metabase [no longer supports H2 connections](https://www.metabase.com/blog/vulnerability-post-mortem). But Metabase still ships with an H2 database to include an embedded application database.
## Connecting to databases hosted by a cloud provider
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.