memoized-supports?* and memoized-features* no longer exist; their
replacement is a plain atom the core.memoize-based monitor cannot measure.
Atom-backed cache telemetry is a planned follow-up.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WCo9W7tgK13nNSfWTtQoUc
Before, memoized-supports?* held one boolean per [driver feature app-db db-id
updated-at], and memoized-features* a parallel set per [driver app-db db-id
updated-at]:
[:postgres :describe-fields 1 3 #t "2026-07-08T17:45:11"] -> true
[:postgres :describe-indexes 1 3 #t "2026-07-08T17:45:11"] -> true
... 83 more rows for db 3 at this stamp ...
[:postgres :describe-fields 1 3 #t "2026-07-08T18:02:37"] -> true
... a full new block of 85, adjacent to the old one ...
After, one entry per database whose value is the whole answer:
[1 3] -> {:stamp #t "2026-07-08T18:02:37"
:features {:postgres #{:describe-fields :describe-indexes ...}}}
The set is the right unit because nobody asks these questions one at a time:
loading a Database row runs
(assoc :features (driver.u/features driver (t2.realize/realize database)))
in after-select, so features are fully enumerated on every row load, never
consulted atomically (heap-confirmed: entries only ever appear in complete
85-feature blocks). Features aren't meant to be atomic — cache them the way
we use them. The database-supports? multimethod stays per-feature as the
driver-side extension point; only the cache adopts the usage-side shape.
Going from the exploded rows to one enumerated set is safe because there is
no discovery to accommodate: driver/features is a closed, statically-defined
universe (dispatch throws on unqualified features outside it), so one sweep
at construction answers every question that can ever be asked. The exploded
cache filled in lazily per question and so could never say it was done; the
enumerated entry is complete the moment it is built and immutable until
replaced wholesale — nothing ever needs to be added to it as new questions
arrive. The only features outside the universe (test-only qualified
keywords) bypass the cache and hit the driver directly, which also preserves
dispatch validation of bogus features.
The old shape leaked because updated-at lives in the KEY: a row touch can't
replace anything, it mints 85 fresh entries adjacent to the previous
generation, which no lookup will hit again and nothing ever removes
(core.memoize memo is unbounded, and TTL caches only evict on later lookups
anyway). Freshness in the key = accumulation. The new shape moves the
invalidation datapoint INTO the value: same key, compare :stamp on read, swap
the entry in place. Freshness in the value = replacement; generations are
structurally impossible.
DB routing made this catastrophic. Every mirror is a Database row, every
mirror row load enumerates, so 5k mirrors x 85 features = 425k entries per
generation — and mirror rows are touched in bulk (secret rotation bumps
updated_at across all of them at once), so each rotation minted a complete
new generation. The 2026-07 heap dump: 521,900 entries (~130MB) = 6,140
(db-id x updated-at) generations x exactly 85 features, with Nov-2025 entries
still resident in Jul-2026. Now a mirror never gets an entry at all: its
reads key on its router_database_id (present on both Toucan rows and lib
metadata maps), a cold miss fetches the router's row once, and a mirror touch
changes nothing — a mirror's updated_at was never freshness information about
the router's features. Cache size is #routers, independent of mirror count
and touch frequency.
Mechanics that fall out:
- supports? is set membership on the cached sweep.
- the driver nests inside the value because before-update checks a
prospective new engine against the old row; eviction stays O(1) by db id.
- :event/database-update/-delete evict via the existing driver-notifications
handler; between events the stamp check keeps entries fresh.
- feature sets are interned, so equal sets share one instance instead of
~4KB of duplicated set structure per database.
- key reads are shape-agnostic (kebab first, snake fallback on Toucan rows;
lib metadata maps are SnakeHatingMaps that throw on snake_case access) and
deliberately independent of ensure-lib-database, pinned by tests covering
all four shape x role combinations (Toucan/lib x router/mirror).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WCo9W7tgK13nNSfWTtQoUc
* Bump `hawk` to `1.0.18`
This corresponds to
c17e2b9b4e
* fix bad tests
* select data key
---------
Co-authored-by: Riley Thompson <rileythompson99@gmail.com>
* Retry the quarantine list fetch before giving up
A dropped socket or a 5xx from ci-conductor made the gate report COULD NOT
CHECK on the first try. Transport errors, timeouts, 429s and 5xx now get three
attempts with exponential backoff and jitter; a 4xx returns straight away.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* improve retry api
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Ryan Laurie <iethree@gmail.com>
* Add allowedCustomVisualizations setting to EAJS embeds
This commit adds the plumbing necessary to pass down the
value of the allowlist for custom viz.
It warns when wrong names are passed (ie: not starting with `custom:`),
and forces [] for guest embed as the feature is not supported there.
* Move sandbox-host to its own namespace to remove +auth from it
The donor document is loaded as an iframe src, which cannot carry session
headers, so the endpoint must be reachable without authentication. Auth is
applied per-namespace by ns-handler in custom-viz-plugin.api and cannot be
lifted for a single endpoint, so the endpoint moves verbatim to its own
namespace, mounted without +auth next to the session-authed routes (both
still behind the :custom-viz premium gate). No behavior change other than
dropping the auth requirement.
* Serve the custom viz sandbox donor to EAJS embeds
Makes the donor iframe-able as long as either a valid session is passesed
or the correct sec-fetch-site headers are passed.
* Allow blob: images on the EAJS embed page for custom viz icons
Custom viz icons in the chart type picker are served by the SDK asset
manager as blob: URLs (a cross-origin img cannot carry the session
header, and EAJS has no cookies). The modular embedding page runs on the
Metabase origin under a strict img-src that omits blob:, so the browser
blocked the icons and left a broken image in the picker. Add blob: to
img-src only for the /embed/sdk/ entrypoint, gated on a URI predicate so
the main app and static or public embeds keep the stricter policy.
* chore: tidy the SDK release dispatch forms and bump output
The dispatch form renders an input's description as the control's label,
which is a single plain text node: newlines collapse to spaces and no
markdown is parsed. Both SDK release workflows had multi-line
description blocks written as if they rendered, so release_type's
occupied ten rendered lines and pushed the select below the fold.
- release_type: one-line description keyed by option, options reordered
into release-cycle order, and default: alpha pinned so the reorder
does not move the preselection. Option values are unchanged.
- Detail that no longer fits the form moved to the file header.
- Shortened the two read_first option strings that overflowed the
collapsed select at 44 and 81 characters.
- Gave the version-bump PR body's table a real header instead of an
empty one (EMB-2167).
- Capture the new PR's URL and append it to the job summary; the
existing summary step runs before the PR exists (EMB-2168).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ENnTtrTcig1vH3QADbsGgo
* chore: capitalise release type names in the dispatch label
The label renders as plain text, so no markdown emphasis is available.
Caps makes each option findable when scanning the wrapped paragraph.
Values stay lowercase - this is display emphasis only.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ENnTtrTcig1vH3QADbsGgo
* chore: say what each dispatch input is for, not what it isn't
prerelease_id read as an output description, so it never said what to
type. Phrase it as a mapping: enter data-apps, get 0.62.5-data-apps.0.
Same length, same six rendered lines.
The release workflow's label opened with a double negative - use it
'when neither the bump PR nor re-running the failed jobs will do it' -
which left the reader to infer the purpose. It is a manual override, so
say that first. Retry guidance stays in the file header, where there is
room for it. Eight rendered lines down to six.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ENnTtrTcig1vH3QADbsGgo
* chore: drop a redundant comment above the dispatch label
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ENnTtrTcig1vH3QADbsGgo
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Dependency upgrade: align both drivers that ship org.apache.thrift/libthrift on the latest release (0.24.0).
hive-like: bump the existing direct pin 0.16.0 → 0.24.0. Since 0.19.0, libthrift declares runtime deps (httpclient5, httpcore5, jakarta.servlet-api) that only serve its THttpClient/TServlet classes; hive-jdbc is built against httpclient 4.x (already pinned in this module) and cannot use them, so they are excluded to keep the driver jar lean. Verified by scanning every class in the built jar: the only classes referencing those packages are libthrift's own five HTTP/servlet classes - nothing in the hive/hadoop stack touches them.
databricks: pin 0.24.0 over the 0.23.0 bundled with databricks-jdbc-thin 3.4.1, following this module's existing pattern of pinning newer transitives (lz4, bouncycastle, jackson, commons-configuration2).
Previously the two driver jars bundled different libthrift versions (0.16.0 and 0.23.0) into the shared plugin classloader; this aligns them on a single version.
Also updates the Spark SQL connection form's "additional JDBC options" placeholder from ;transportMode=http to ;transportMode=binary - binary is the default and supported transport; hive-jdbc's http transport mode is not compatible with libthrift ≥ 0.19 (it passes an httpclient 4.x object to a constructor that now takes httpclient 5.x, see #46793), so the form should not suggest it.
Routine dependency upgrade: bumps the pinned at.yawk.lz4/lz4-java (maintained fork of the discontinued org.lz4:lz4-java) to the latest release, 1.11.1, in the ClickHouse and Databricks driver modules.
Upstream changes between the pinned versions and 1.11.1 are additive only (new fast-reset compressor classes, a linux-riscv64 native binary, bugfixes) — no classes removed, same net.jpountz packages, so it is a drop-in update for both JDBC drivers. Neither driver module calls the library directly.
* GDGT-2884: Move AI analytics to Monitor (#78222)
* GDGT-2874: Relocate AI analytics source under monitor/ai-auditing
* GDGT-2874: Add Monitor URL helpers for AI Auditing
* GDGT-2874: Repoint hardcoded legacy paths to Monitor URL helpers
* GDGT-2874: Swap MetabotAdminLayout for MonitorMain in AI Auditing pages
* GDGT-2874: Fix stale audit_app import in metabot-analytics test-support mock
Plan gap: this file wasn't in Task 1's moved-tree scope and wasn't listed as an
external consumer to fix in a later task. Caught during Task 4 verification
via type-check-pure.
* GDGT-2874: Add AI Auditing sections to monitor_section_clicked event
* GDGT-2874: Add canAccessAiAuditing selector and route guard
* GDGT-2874: Add combined AI Auditing routes getter and plugin index
* GDGT-2874: Move AI Auditing plugin wiring from PLUGIN_AUDIT to PLUGIN_MONITOR
* GDGT-2874: Mount /monitor/ai-auditing route block
* GDGT-2874: Add AI Auditing tab to Monitor nav
* Add regression tests for AI Auditing getActiveSection ordering
Lock in the order of the mcp/conversations/usage-stats route-matching
clauses in getActiveSection: the bare Urls.monitorAiAuditing() path
string-prefixes both sub-paths, so the more specific clauses must be
checked first (ts-pattern's P.string.startsWith is first-match-wins).
Without this, a future reorder would silently break active-tab
highlighting for the two sub-routes.
* GDGT-2874: Remove Auditing folder from Admin AI nav
* GDGT-2874: Add legacy redirects for /admin/metabot/usage-auditing
* GDGT-2874: Fix moved AI Auditing spec mocks and path assertions
* GDGT-2874: Repoint slash-command URL helper off the legacy admin path
The Metabot /inspect slash command still built its target URL via
metabase-enterprise/urls.ts's adminMetabotUsageAuditingConversation,
which hardcoded the old /admin/metabot/usage-auditing/conversations
path. It only worked because it bounced through the legacy redirect.
Delete that helper (its only caller) and use the canonical
Urls.monitorAiAuditingConversationDetail directly.
* GDGT-2874: Update E2E specs to canonical /monitor/ai-auditing URLs
usage-auditing.cy.spec.ts and mcp-analytics.cy.spec.ts predate the
Monitor migration and still visited/asserted on the legacy
/admin/metabot/usage-auditing* URLs, which only worked via the legacy
redirect. In-app navigation (row clicks, breadcrumbs, back button) now
lands on /monitor/ai-auditing* URLs, so any post-navigation URL
assertion in these specs was exercising stale paths. Point every
cy.visit()/pathname assertion at the new URLs, and add a small
regression test asserting the legacy URL still redirects, since the
main specs no longer exercise it directly.
* GDGT-2874: Apply AI Auditing cross-review fixes
* GDGT-2874: Add AI Auditing availability placeholders
* GDGT-2874: Gate AI Auditing pages with settings placeholders
* GDGT-2874: Keep unavailable AI Auditing links navigable
* GDGT-2874: Prioritize global AI settings for MCP availability
* GDGT-2874: Fix empty screen layout
* GDGT-2874: Use semantic AI Auditing navbar icons
* GDGT-2874: Restore Audit plugin ownership
* GDGT-2874: Test legacy redirects in app routes
* GDGT-2874: Add canonical Usage stats route
* GDGT-2874: Cleanup
* GDGT-2874: Address PR review feedback
- Rename AiAuditingAvailabilityLayout -> AvailabilityLayouts: the file
exports two layout components (Metabot and MCP), neither named after
the file/folder.
- Use !!useSetting(...) for the enabled/configured flags, matching the
convention used everywhere else these settings are read.
* GDGT-2883: Restyle AI Auditing to match Monitor (#78356)
* GDGT-2874: Fix moved AI Auditing spec mocks and path assertions
* GDGT-2874: Apply AI Auditing cross-review fixes
* GDGT-2883: Restyle AI Auditing to match Monitor
* GDGT-2883: Fix inconsistencies
* Fix issues surfaced by restacking onto gdgt-2874
- Remove index.unit.spec.ts: orphaned artifact from a superseded
gdgt-2874 draft, testing a ./index module that was never actually
relocated here (registration logic still lives in audit_app/index.js,
already covered by routes.unit.spec.tsx).
- Un-mock ConversationHeader in ConversationDetailPage.unit.spec.tsx:
the blanket mock predated the conversation-title test and was
hiding the real header it needs to assert against; the endpoint
mocks setup() already provides make the real component safe to
render.
- oxfmt fix in ConversationsTable.tsx.
* GDGT-2883: Fix Monitor layout consistency across AI Auditing pages
- Contain the Tool calls grid to the viewport (MonitorMain only clips
when its parent is a flex container, so wrap it in a Flex h="100%"
on the events tab; the charts tab keeps natural page scroll)
- Do the same for the Conversations grid page
- Hide the events table's pagination until it has loaded once, instead
of showing a nonsensical "1 - 0 of N" range
- Add showTotal to the Conversations page's pagination, which was
missing it entirely
- Match tabs/filters/content spacing (md) to Background Tasks on Usage
stats and MCP analytics, which had drifted to lg
- Vertically center the ellipsis menu button against the conversation
title in ConversationHeader
* GDGT-2883: Make all AI Auditing grid columns sortable, fix non-clickable cursor
- TreeTable: only show the pointer cursor on rows that actually respond
to a click (onRowClick or getRowHref set), instead of unconditionally
on every row. Fixes MCP analytics' Tool calls grid looking clickable
when it isn't, and the same latent bug on several other display-only
TreeTable consumers.
- MCP analytics Tool calls: wire up sorting for the remaining columns
(ID, Client version, Tenant, IP address, Error type, Error message).
The query builder already supported ordering by any valid view
column, so this is purely frontend wiring.
- Conversations: add Title as a sortable column, including the backend
allow-list/schema entry it needs. Queries and Searches stay
unsortable — they're computed in-memory per page from message
content, not real SQL columns.
- Fix a stale field name (summary -> title) in ConversationsPage's
test mock that was failing type-check.
* GDGT-2883: Show tool call details in a sidebar on the Conversation Detail page
Replace the tool-call-click modal with a Monitor sidebar, matching the
rest of the AI Auditing views. The live Metabot chat widget keeps its
modal — AgentToolCallMessage now takes an optional onSelect callback
that, when provided, defers to the host instead of opening the modal.
Also style the sidebar's Request/Response copy buttons with the same
Copy/Copied! tooltip used elsewhere in the conversation view, and fix
a pre-existing bug where copying re-stringified the already-formatted
JSON text, copying an escaped string instead of clean JSON.
* GDGT-2883: Toggle the tool call sidebar closed on a second click
Clicking the same tool call record that's already selected now closes
the sidebar instead of re-opening it with the same content.
* GDGT-2883: Constrain conversation content to an 800px column, fix page scroll
Wrap the whole page body (header, stats, feedback, conversation, and
generated queries) in a single horizontally-centered 800px column
instead of leaving it full width.
Fix the page-level scroll along the way: the Flex wrapper added for
the tool call sidebar's width measurement gave MonitorMain's parent
display:flex, which made MonitorMain's flex+overflow:hidden CSS
actually take effect and clip content instead of letting the page
scroll. This page has no internal scroll container of its own (unlike
table-driven Monitor views), so drop the bounded height and let it
scroll naturally.
Also add the missing Copy/Copied! tooltip to AgentToolCallMessage's
own copy button, matching the one already added to the sidebar's
Request/Response copy buttons.
* GDGT-2883: Make AI Auditing pagination abortable, sync page in URL instantly
Every paginated AI Auditing grid (MCP analytics Tool calls, Conversations)
kept a stale in-flight request running when the user paged again before it
resolved, and the URL's page param lagged behind clicks by the shared
useUrlState hook's 300ms debounce — unlike Dependency Diagnostics, where
page changes are both abortable and instant.
- Swap McpEventsTable's and ConversationsPage's queries onto
useAbortableQuery (the same hook DD already uses), so switching pages
cancels the stale request instead of letting it complete in the
background.
- Add an opt-in `immediate` option to useUrlState's patchUrlState that
bypasses the debounce for a single patch, and use it for pagination
clicks specifically. Filter/sort changes keep debouncing as before —
this only affects callers that pass the new option.
* GDGT-2883: Sync pagination to the URL instantly across other Monitor list pages
Same delayed-URL-on-pagination-click bug as AI Auditing, found on a sweep
of every useUrlState consumer with a paginated grid: TaskListPage,
TaskRunsPage, NotificationsAdminPage, and ErrorOverview. Pass the new
patchUrlState immediate option on their pagination handlers too.
TaskListPage's pagination test asserted the old debounced timing (URL
unchanged immediately after a click, then advance the fake timer); updated
it to assert the new instant update instead. The other pages' tests only
asserted the eventual URL state, so they needed no changes.
* GDGT-2883: Use the empty-dashboard illustration for the disabled MCP analytics page
Swap the generic metabot icon for the EmptyDashboardBot illustration and
make the CTA a secondary button, matching other Monitor empty states.
Scoped to the MCP-disabled case only — the AI-features-disabled and
not-configured states elsewhere in this file are unchanged.
EmptyState gains an optional actionVariant prop (defaults to "filled",
preserving every other caller's current look).
* GDGT-2883: Apply cross-review a11y/correctness fixes
- Empty-dashboard illustration + secondary CTA now apply to all four
AI Auditing unavailable states (AI disabled, not configured, MCP
disabled), not just MCP-disabled — that was the intended scope; a
narrower revert from a review pass was itself reverted here.
- Give the tool call copy buttons accessible names (were icon-only).
- Make TreeTable sort column headers keyboard-operable.
- Clear the selected tool call when navigating to a different
conversation, so the sidebar can't reopen with stale data.
- Move focus into the tool call details sidebar on open and back to
the trigger on close; label it as a region.
- Ignore a hidden/PII-gated sort column from a stale URL instead of
querying by it.
- Add a Title-sort UI test and a one-shot immediate-bypass regression
test.
* GDGT-2883: Fix code-review findings in TreeTable and AI Auditing
- Stop sortable header keydowns from bubbling to the treegrid, which
could also activate a stale keyboard-focused row
- Give every populated TreeTable header the columnheader role, not
just sortable ones
- Drop the redundant onRowClick push() on conversation rows, which
double-navigated alongside the row's own link
- Render the MCP analytics loading/empty states inside the active
Tabs.Panel so the selected tab's aria-controls stays valid
* GDGT-2883: Trim redundant comments in McpEventsTable and ConversationDetailPage
* GDGT-2883: Cleanup
* GDGT-2883: Fix e2e tests
* GDGT-2883: Address PR comments
* GDGT-2919: Migrate CLI usage analytics to Monitor (#78900)
* Project MCP events query to curated PII-aware columns
Restricts buildEventsQuery to exactly the columns the events table
displays, keyed off tenant/PII retention settings, instead of shipping
the view's full row (including ip_address/error_message) to the
browser regardless of the setting.
* GDGT-2919: Migrate CLI analytics to Monitor
* GDGT-2919: Drop ai-features-enabled? gate for CLI analytics
The gate was new, not preserved: the old admin route only checked
audit_app/admin access, with no ai-features-enabled? check, and CLI's
v_agent_api_calls data is independent of Metabot/LLM configuration.
* Fix oxfmt formatting
* GDGT-2919: Bring CLI analytics UI in line with the rest of Monitor
- Move the calls filter bar under the tabs and use MCP's spacing values,
keeping tabs/filters mounted (and visible) through loading and empty
states instead of swapping the whole Tabs block out.
- Replace CliEventsTable's AdminDataTable (admin-only, fixed
maxBodyHeight) with TreeTable, matching McpEventsTable's structure so
the Calls tab fills the available height like the rest of Monitor.
* Replace withRouteProps with useRouter across Monitor pages
withRouteProps is documented as a temporary v3-router migration shim
that reads location/params via useRouter() and re-injects them as
props. Have the Monitor pages call useRouter() directly instead and
mount them as plain route elements, dropping the wrapper.
* GDGT-2919: Make all Calls tab columns sortable
Matches MCP's events table, where every column (including the PK and
PII columns) is server-sortable. call_id/tenant_name/ip_address/
error_message are all plain columns on v_agent_api_calls, so ordering
by them needs no query changes beyond adding them to the sort list.
* GDGT-2919: Project CLI events query to curated PII-aware columns
buildEventsQuery shipped the view's full row (including ip_address and
error_message) to the browser regardless of the tenant/PII retention
settings; the frontend only hid those columns from the rendered table.
Add cliEventColumnKeys as the single source of truth for both the
query projection (Lib.withFields) and the display columns, mirroring
the same fix already applied to MCP analytics.
* Use absolute imports for cross-module paths in ai-auditing
Replace ../ relative imports (mcp-analytics/cli-analytics reaching
into metabot-analytics, and cross-directory imports within
metabot-analytics) with the metabase-enterprise/... absolute alias.
Same-directory ./ imports are left as-is.
* GDGT-2919: Show an error instead of spinning forever in CliEventsTable
useCliEventsQuery didn't surface the query's error, so a failed
per-page fetch left data undefined and the table stuck on the
skeleton indefinitely. Surface the error and render
LoadingAndErrorWrapper instead.
Also drop useRetainedValue: RTK Query's data (unlike currentData)
already stays sticky across page/sort changes until the next result
resolves, so the manual retention was redundant.
* GDGT-2919: Show an error instead of spinning forever in McpEventsTable
Mirrors the CliEventsTable fix: useMcpEventsQuery now surfaces the
query error, and McpEventsTable renders LoadingAndErrorWrapper instead
of leaving the skeleton up indefinitely on a failed fetch. Drops
useRetainedValue there too, for the same reason (RTK Query's data is
already sticky across page/sort changes).
Also trims redundant comments in both event tables.
* GDGT-2919: Address PR comments
* Revert gratuitous DependencyDiagnosticsPageOwnProps rename
Keep master's DependencyDiagnosticsPageProps naming so this Graphy-owned
file isn't touched by the master merge for no functional reason.
* Fix jest.mock("metabase/AppComponent") to match its named export
Master's router refactor changed AppComponent from a default-exported
connect() component to a plain named export (App). The mock still
returned a default export, so import { App } resolved to undefined and
broke every route render in routes.unit.spec.tsx.
* GDGT-2874: Fix usage audit e2e test
The treemap label model imported VisualizationGridSize from the
enterprise custom-viz package, a cross-boundary type import. A
structurally identical type already exists in
metabase/visualizations/types, so use that instead.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Resolve relative navigation against v7's own match
* Resolve redirect targets off the v7 match branch
* Drop the router argument from useDashboardUrlQuery
* Free Palette and McpAnalyticsPage from the router context
* Read location and params from v7 instead of useRouter
* Scope route-leave hooks from the match instead of an injected route
* Delete RouterBridge, the router contexts, and the v3 route types
HoneySQL munges dashes to underscore in a keyword's namespace, but not in its name. The code was relying on HoneySQL default behavior to turn namespaced keywords into table references, but the inconsistency bit me in some new development work.
Two paths currently pass schema-qualified keywords in cases where a table might leak:
- `drop-table!` targeted a nonexistent object and silently "succeeded" (not dropping the table)
- `compile-transform` (the SQL variant) could target a nonexistent database.
Both `drop-table-sql` and `create-table-sql` are updated (the latter via `quote-table`) to keep them in sync.
Collapsing the namespace into the keyword's name keeps everything verbatim as you'd expect. Callers already passing in `"schema.name"` weren't impacted.
* Fix QB perf
* Forbid loading every field via GET /api/database/:id/metadata
Make skip_fields a required `true` on the FE request type so no caller can ask
for field-level metadata, which grows with every column in the database.
The two callers that needed fields now scope their fetch to a single table:
- Data Reference field pages fetch the table's own query metadata (which also
pulls FK target tables and their fields)
- useAuditTable resolves the table from the plain database response, then loads
/api/dataset/query_metadata for that table
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Withhold the audit table until its fields load
ConversationStatsPage renders charts straight from useAuditTable's table
without checking isLoading, so handing back a table built from the field-less
database metadata made it paint once and re-render when the ad-hoc query
metadata arrived, detaching the chart nodes mid-assertion.
Also make the segment field list spec's database mock honour skip_fields, so
unit tests can catch a caller that stops asking for fields.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix be
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* BE Add Mistral metabot provider
* BE Add prompt-cache-key to LLMRequestOpts for Mistral prompt caching support
Mistral requires a `prompt_cache_key` to be set to enable prompt caching, so add it as an optional key to
`LLMRequestOpts` and thread it through `self/call-llm` and `self/call-llm-structured`.
* FE Add Mistral metabot provider
* Review feedback: fold all provider selectable tests into a single test
* Review feedback: use findPutRequests to reduce verbosity
* BE Add Z.AI metabot provider
* FE Add Z.AI metabot provider
* Move generic chat-completions code to metabase.metabot.self.openai.chat-completions
And use it in both the openrouter and zai providers.
hashSearchTerm and shouldReportSearchTerm are analytics reporting
policy (settings read + SHA-256 hashing), not search UI, so they move
from common/search/term.ts to the lib-tier analytics module and are
exported from its barrel. Consumers now import them from
metabase/analytics, and the common/search barrel no longer re-exports
them.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Relocate the isVisualizerDashboardCard type guard from
metabase/visualizer/utils to metabase-types/guards/dashboard, alongside
the other dashboard type guards, and update all importers to the new
path. Remove the re-export from the visualizer utils barrel and delete
the old file.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Add parseSearchQuery and a useNavigationType facade hook
* Move the location.action readers to useNavigationType
* Move the location.query readers to search params
* Drop query and action from the facade Location
* Replace connect() with hooks in the app shells and permissions pages
* Rewrite the reference containers as function components
* Delete withRouteProps
* Pass the props reference children read but do not select
* Cover every reference container page with a render test
* Fetch reference container data before the child's first render
* Add Snowplow event tracking to the metric Dimensions tab
* Update events' definitions to comply with new approach
* Update analytics-events claude skill to account for new approach defined in #71631
TableBrowser only needs the table list, but GET /api/database/:id/metadata
returns every field of every table. Pass skip_fields=true so the response
stays proportional to the number of tables.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
HoneySQL inlines true/false as TRUE/FLASE tokens even when the query is otherwise parameterized, and not every driver target has a boolean literal. SQL Server parses a bare TRUE as an identifier and `insert-into!` fails with `Invalid column name 'TRUE'`. Wrapping boolean row values in `[:lift v]` binds them as paremeters, which is what we do for other value types.
Impacts every `:sql-jdbc` driver's `insert-into!`. Drivers with boolean literals are not impacted.
Fix quadratic :is_personal hydration that made GET /api/collection take ~23s
On an instance with 28k collections — 6k of them personal — `GET /api/collection`
took 23.5s while making only 13 DB calls. Nearly all of it was the `:is_personal`
batched hydration: for every collection it scanned the full set of personal
collection IDs, building a prefix string with `format` on each step, so a request
did ~20k x 6k = 125M `String/format` calls.
Personal Collections only ever live in the Root Collection, so a collection is
inside one exactly when the *first* ID of its location path is a personal
collection. Test that single ID against the set instead of scanning.
Measured on a seeded app DB matching the reported shape (28.6k collections, 6,011
personal roots, 13.6k live non-personal, 7.8k archived, depth to 13):
GET /api/collection 11.2s -> 1.9s
:is_personal hydrate 9.2s -> 75ms
The remaining hot spot is `:can_write` (~1.1-1.5s), which linearly scans the
user's permission set per collection — left for a separate change since it
touches the shared permission check.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* Convert the store factory and whitelabel color bootstrap to TypeScript
* Name the deliberately loose reducer map type
* Condense the reducer map comment
* BigQuery: reject oversized queries client-side with a humanized error
BigQuery rejects jobs whose SQL exceeds 1 MB (1,048,576 characters,
including comments and whitespace) with a raw 400 INVALID_ARGUMENT,
which Metabase surfaces to the user as "Error executing query: <raw
message>" -- accurate but technical and not actionable.
Check the final compiled SQL (including the appended remark) before the
BigQuery client/job is created and throw a localized, actionable
:invalid-query error instead. The raw DB error path remains as a
fallback if this limit ever drifts from BigQuery's own.
Quota reference: https://cloud.google.com/bigquery/quotas#query_limitsCloses#79019
* Update error message for SQL query length validation
* Address review: express limit as (* 1024 1024), slim the tests
- max-sql-query-length-chars: write the value as (* 1024 1024) so the
1 MiB intent reads off the page.
- validate-query-length-test: collapse to the two boundary cases (at the
limit allowed; over the limit throws :invalid-query), error type only,
per review feedback to keep it simple.
* Address review: trim noise from max-sql-query-length-chars docstring
Drop the cross-reference to validate-query-length! and the "not ^:const"
justification — commentary, not contract. Leaves what the constant is,
how BigQuery counts, the failure mode, and the quota link.
* Add AI provider setup to onboarding
- Insert AI config step into setup flow
- Track connect and skip analytics events
- Update onboarding tests and schema version
* Preserve existing AI provider during setup
- Wait for the saved provider before mounting the form
- Show the connected provider in the setup step and cover it in tests
* Defer AI provider sync until after setup
- Keep the AI config step visible during onboarding
- Skip legacy Metabot provider migration on fresh instances
* Add AI provider setup to onboarding
- Hide the step when AI features are disabled
- Support Anthropic onboarding setup in e2e
- Mock Anthropic model catalog for credential validation
* Match license step layout in AI setup step
* Fix modules config and flaky setup toast assertion
* Add free_units to managed AI e2e add-on fixture
* Condense AI step ordering comment
* Make AI setup opt-in from the completed step
The AI provider step is no longer part of the wizard by default. The
"You're all set up!" screen offers a "Set up AI" button that appends
ai_config as the last step; connecting or skipping returns to it.
* Scope the settings-list mock to the test that needs it
settingOverrides was only used by one embedding test, so drop it from the
shared setup helper and let that test override the settings-list route.
* Type the visualization registry as components or bare definitions
* Convert the dashboard viz register module to TypeScript and use typed named exports for Action and IFrameViz
* Strip large literal IN lists before GraalPy SQL parsing
Large literal IN lists (like large VALUES clauses) blow GraalPy's ~114KB/AST-node
memory cost and its 30s call timeout: a 300k-item list that sqlglot parses in
1.7s on CPython times out at 30s on GraalPy, poisoning a pooled context on every
background dependency-analysis pass over the offending query.
Extend the JVM-side pre-processing to also replace literal-only IN lists with
more than 100 items with `IN (NULL)`. Only lists consisting purely of literals
(numbers, quoted strings, NULL/TRUE/FALSE) are stripped; column references,
function calls, subqueries, and bind parameters disqualify a list, since
stripping those would change analysis results or parameter counts. Lists nested
inside unstripped IN subqueries are still found.
All callers get the same handling: the stripping happens in the one shared
strip-large-literal-lists (renamed from strip-large-values) used by every
analysis entry point, and is-single-stmt-of-type? keeps the #74284 restore
behavior — validate the stripped SQL, return the original text — so
impersonated queries still execute with their literal lists intact.
With the fix, field-references on a 300k-item IN list drops from a 30s timeout
to ~80ms.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Simplify literal-list stripping to a single shared pass
Review cleanup: collapse the two per-keyword scanners into one walk over the
SQL driven by a combined VALUES|IN pattern with small per-keyword handlers,
extract the quote/doubled-quote scanning into a shared skip-string-literal
(also fixing count-top-level-commas, which didn't handle doubled-quote
escapes), and replace the substring-allocating literal-only-list? check with a
single index-based early-exit pass, so IN (SELECT ...) is rejected without
scanning or copying the subquery region. No-strip calls now return the
original string identity.
field-references on a 300k-item IN list drops from ~80ms to ~27ms; all
existing stripping tests pass unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add exhaustive tests for literal-list stripping
Cover threshold boundaries (100 vs 101) for both VALUES and IN, every accepted
literal shape, a disqualifier grid (17 non-literal items x 3 positions),
byte-for-byte preservation of surrounding SQL, nesting (IN inside VALUES
tuples, stripped-VALUES swallowing nested lists, deep IN-subquery chains),
idempotency, original-instance identity on the no-strip path, malformed input
(unbalanced parens, unterminated strings, empty lists), keyword false
positives (margin, JOIN, INT, quoted identifiers), and an end-to-end pass of
every analysis entry point over stripped SQL.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Also strip large ARRAY literals and literal tuple IN lists
Two more static-data shapes reproduce the GraalPy 30s timeout / severe CPU
burn: id = ANY(ARRAY[...]) (the Postgres idiom ORMs emit instead of IN; 200k
items hit the 30s timeout) and composite-key tuple lists
(a, b) IN ((1, 1), (2, 2), ...) (50k tuples burned 21s).
Generalize the flat literal-list scanner to a parameterized closing delimiter
and reuse it for ARRAY[...] -> ARRAY[NULL], and add a tuple-list scanner that
requires every tuple to be literal-only, stripping to IN ((NULL, ...)) with
the first tuple's arity. Subscript access (arr[1], arr[1:5]), subquery tuples,
and any non-literal item still disqualify a list. The is-single-stmt-of-type?
restore behavior covers both new shapes.
After the change: tuple-IN 50k tuples 21.6s -> 525ms; ANY(ARRAY[200k])
30s timeout -> 1.1s. CASE WHEN chains and UNION ALL literal tables remain
unstripped (no lossless strip exists for them).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Treat double quotes as a disqualifier in literal-list scanning
In identifier-quoting dialects (Postgres, Snowflake, ...) "x" inside an IN
list or ARRAY literal is a column reference, not a string, so stripping a
large list of double-quoted items would drop real field references from
analysis. The literal-list scan is dialect-agnostic, so be conservative: only
single-quoted strings count as literals; a bare double quote leaves the list
untouched. skip-string-literal still skips both quote styles where it is used
for paren/comma balance scanning, where either interpretation is a
skip-region.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Use [[strip-threshold]] docstring links instead of backticks
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix Eastwood reflection warning: cast close-ch to primitive char
(= ch close-ch) compared a primitive char against the untyped close-ch
parameter, which Eastwood flags as an unresolvable Util.equiv call.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Ignore keyword matches in strings/comments; reject empty list elements
Review follow-ups:
The keyword matcher previously rewrote text inside string literals and
comments — SELECT 'IN (0, ..., 149)' became SELECT 'IN (NULL)', and an
unbalanced quote inside such a region could corrupt the statement. The walker
now gap-scans between matches with a string/comment-aware cursor
(code-region-end) and copies any match that sits inside '...', "...",
-- ..., or /* ... */ through untouched.
The flat-list scanner also accepted empty elements, so invalid SQL like
IN (,,,) with >100 commas was rewritten into valid IN (NULL), masking syntax
errors from validation. Both the flat and tuple scanners now require content
in every element and reject leading/trailing/consecutive commas.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Simplify IN/ARRAY stripping to one simple-list scanner
Stripping feeds fail-soft analysis only, and execution paths restore the
original SQL, so the defensive machinery accumulated from review rounds
(literal word allow-list, tuple/arity validation, empty-element checks,
string/comment-aware outer scanning) bought correctness for cases that don't
matter at the cost of real complexity. Replace it all with one rule: a list is
strippable when, from its opening ( or [, it contains only digits, signs,
single-quoted strings, commas, whitespace, and balanced parens (which covers
tuple lists with no special logic), and holds at least 100 commas. Flat IN,
tuple IN, and ARRAY all collapse into a single ~20-line scanner and one
decision fn; NULL/TRUE/FALSE items now conservatively block stripping. VALUES
keeps its pre-existing count-based path. The best-effort contract is stated in
the public docstring.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Seven sites across four drivers were escaping SQL string literals the same way. This consolidates, and there's a single place to warn not to use escape-sql to sanitize user input.
No behavior changes.
MBQL treats `java.math.BigInteger` and `clojure.lang.BigInt` interchangeably. Only the Clojure class was in the parameter table, so a BigInteger fell through in one of my tests to `:default` and bound as a string.
* Show the output schema and table for a transform run
Adds a TransformOutput component that renders the run's target as `schema › table`
breadcrumbs with a copy button. Both crumbs link out when they exist: the schema to
the data model, the table to the query builder. Wired into the run sidebar
("Output table" row) and the job run sidebar ("Output:" line).
Also fixes the job run sidebar not shrinking below its content width (missing
min-width: 0), which let long table names push it past the viewport.
* fix tests
* drop newTab prop, always open output links in new tab
* addressing PR's comments
* fix race condition
The three preceding PRs removed every shared-tier import from the module: the
route guards, the redux location mirror, and the styled-Link re-export. All that
is left is metabase/utils, which basic may import.
basic rather than lib: router is a React component library, the same shape as
metabase/ui, and nothing in basic/* imports it, so lib would only widen what
lib/utils is allowed to reach for no gain.
* fix(sdk): sdk crashes when resizing column
* do the same on ExecutionOutputTable too
* reword test to make it better
* make sure cols/rows are not new arrays every render when output is undefined
* Derive Tabs value type from the tab values
Mantine types the Tabs value and onChange argument as `string | null`. Make
TabsProps generic over the tab value so callers get their own union back in
onChange instead of a bare string.
* Drop Tabs onChange type workarounds at call sites
Every `<Tabs>` call site now annotates its own tab union instead of casting or
re-narrowing the change value: the six `as SomeTab` casts (CommentsSidesheet,
PythonEditorResults, GroupLimitsSettingsSection, McpAnalyticsPage,
CliAnalyticsPage, ConversationStatsPage), the four in the admin/permissions and
tenants listings, and the handlers that laundered `string | null` through a
lookup or a literal comparison (AddDataModal, NotificationsTabs, TableSection,
the date pickers, TabPanel). AddDataModal's `isValidTab` guard existed only for
this and is gone; DatasetFieldMetadataSidebar's TAB map is now `as const`.
* Cleanup
* Apply suggestions from code review
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
---------
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
* Mark metabase/router as side-effect free for rspack
* Route deep metabase/router imports through the barrel
* Let static-viz import the router barrel instead of a deep module
* Guard the side-effect-free contract and scope the include to the directory
metabase/router/Link.ts re-exported metabase/common/components/Link so every
routing import had one home. But that Link is not a routing primitive: it wraps
RouterLink with a Tooltip from metabase/ui, a CSS module, and disabled/variant
props, so the barrel dragged shared/common and basic/ui into the router module
and looped back into router/router-link.
Call sites import it from metabase/common/components/Link again. RouterLink and
RouterLinkProps stay in metabase/router, which is where the primitive belongs.
The boundaries linter never flagged this: element-types checks imports, not
'export ... from'. So the edge had to go rather than be waived, or a lower tier
would have been green while the dependency was still there.
* Fix N+1 query patterns in GET /api/database and GET /api/native-query-snippet
GET /api/database?saved=true&include=tables issued ~6-7 app-DB queries per
saved question (one metabase_database select-one plus a field select and
4-5 hydrations each) — ~3000 queries on instances with a few hundred
eligible cards:
- card->virtual-table: only fetch the database when include-database? is
set, instead of unconditionally per card
- extract cards->card-id->metadata-fields so the result_metadata fields of
all cards are fetched and hydrated in one batch (reused by
batch-fetch-card-query-metadatas), and pass it from cards-virtual-tables
- ids-of-dbs-that-support-source-queries: one select instead of
select-pks-set + per-id select-one
GET /api/native-query-snippet ran user-has-any-perms-of-type? — an
uncached EXISTS against data_permissions — once per snippet via
mi/can-read?, even though the answer is per-user, not per-snippet:
- user-has-any-perms-of-type? now primes the request-scoped permissions
cache for all databases (the same pattern query-creation-capabilities
already uses on GET /api/user/current) and answers by scanning the
cached entries; repeated calls in one request are pure cache hits
- prime-all-dbs-cache records completeness with an ::all-dbs sentinel in
the existing :db-ids set so the database id list is fetched only once
- the uncached EXISTS path is kept for cross-user checks and
disable-perms-cache
Note: the cache prime excludes permission rows for inactive tables, so a
grant existing only on a deactivated table no longer satisfies
user-has-any-perms-of-type?.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Track full-prime state under a dedicated :all-dbs-primed? cache key
Move the all-databases-primed marker out of the :db-ids set (where it sat
as a ::all-dbs sentinel among real database IDs) into its own
:all-dbs-primed? key on *permissions-for-user*. prime-db-cache now
preserves unrelated keys by assoc-ing instead of reset!-ing the atom.
Make prime-all-dbs-cache public with an optional pre-fetched-ids arity,
and switch query-creation-capabilities — the one caller that manually
primed all databases — to use it, so the full-prime work is shared with
user-has-any-perms-of-type? within a request.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Simplify prime-all-dbs-cache: single arity, :all-dbs? key
Drop the pre-fetched-ids arity — the function always fetches the database
ID list itself — and rename the completeness marker from :all-dbs-primed?
to :all-dbs?.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Align :all-dbs? docstring continuation line
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add call-count regression tests for the two fixed N+1 endpoints
Both tests measure warm app-DB call counts via t2/with-call-count at two
entity counts and assert the count doesn't scale:
- GET /api/database?saved=true&include=tables with 2 vs 18 eligible cards;
allows the 1-per-card toucan2 identity-query overhead from
do-after-select but fails on the pre-batching one-select-per-card
behavior (verified failing against the pre-fix code).
- GET /api/native-query-snippet with 2 vs 10 snippets as a non-admin;
fails on the per-snippet permission EXISTS check (verified failing
against the pre-fix code).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Use mt/with-model-cleanup in call-count tests
Replaces the try/finally + manual delete pattern, so fixture rows are
cleaned up even when insertion or the warm-up request throws (addresses
review feedback).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Rework any-perms-of-type caching to a dedicated request-scoped cache
Instead of priming the full permissions cache for every database and
scanning it, cache per [user-id perm-type] the set of database IDs
granting the most permissive value, loaded across all databases in a
single query. This answers every :exclude-db-ids variant from one small
fetch and keeps semantics identical to the uncached query.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* perms cache
* nil fix
* fix
* current state
* cleanup
* Test that inactive-table-only grants are ignored by aggregate perm checks
user-has-any-perms-of-type? previously counted grants existing only on
inactive tables (its raw EXISTS query had no active-table filter); now
that it is served by the distinct-perms cache it ignores them, matching
every other cached check. This is intentional — cover it with a test.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Apply code-review fixes to the permissions cache rework
- Restore batch priming in with-can-run-adhoc-query via a new public
prime-full-perms-cache (table-level checks otherwise load the cache
one database at a time, up to 2 queries per database on card lists)
- Scope the non-cached distinct-perms load to one database instead of
scanning all of the user's permission rows per call
- Extract the shared FROM/JOIN/WHERE of the two cache queries into
perm-rows-query-base so their row sets cannot drift apart
- Simplify the full-perms interner binding and drop a dead fnil and a
redundant double-check inside the distinct-perms swap!
- Select only :id and :engine in ids-of-dbs-that-support-source-queries
(full rows decrypt :details per database, twice per request)
- Own the batched virtual-table Field fetch in a new
cards->virtual-tables entry point; the helper is private again
- Use mt/with-dynamic-fn-redefs in permission-caches-load-per-type-test
- Document that the request caches are point-in-time snapshots
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Treat cached false as a hit in fast-interner
or-based lookups skip false, so interning false re-ran canon-fn and
putIfAbsent on every call (result was still correct). Use if-some so
presence, not truthiness, decides the hit.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Document why distinct perms load all databases and why the uncached interner exists
Capture the review-thread answers in the docstrings: distinct-perms
preloads every database because most consumers ask cross-database
questions, and full-perms builds a per-call interner even without a
request cache because group values are highly repetitive.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Batch-prime the full perms cache in the remaining per-table check loops
Endpoint tracing found three more loops that fill the full permissions
cache one database at a time now that priming is on demand: the
include=tables branch of GET /api/database (per-table can-read?/
can-query?/can-write?), batch-fetch-query-metadatas* (per-table
can-read? behind dashboard/card query_metadata), and the EE
filter-tables-by-data-model-perms loop. Each now batch-primes the perm
types it checks, one query per type instead of up to 3 per database.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Simplify perms caches to all-types loading; rename to table/schema caches
Per-type bookkeeping wasn't worth it — nearly every consumer needs
several permission types, and there are only a handful. Both caches now
load every permission type at once: the table-level cache (was
*full-perms-cache*) per database, the schema-level cache (was
*distinct-perms-cache*, table_id collapsed so schema is the finest
granularity it answers) for all databases in a single query on first
use. prime-table-perms-cache takes just database IDs and no-ops for
superusers, whose checks never read the cache.
Also batch-prime the remaining per-table check loops found by a
repo-wide audit: search result filtering, GET /api/table, recent views
(also reusing the already-fetched table instead of re-selecting it),
typed-schemas table listings, and the segment/measure list endpoints.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Drop the search prime and scope the recents prime to actual table recents
Benchmarks on a 10M-row data_permissions Postgres (200k rows for the
test user) showed both primes regressing their endpoints: search went
40ms to 840ms because priming all databases loads every permission row
of the user on each request while matching results only touch a few
databases, and recents primed the whole instance because table-recents
returns every table when given no IDs. Search now relies on lazy per-db
loads; table-recents returns nothing for empty IDs so the prime no-ops
naturally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Load the schema perms cache per database, with a priming function
Benchmarks showed the all-databases SELECT DISTINCT costing ~230ms per
request at 200k user rows, taxing endpoints that ask about one database
(dashcard query 228ms -> 445ms). The schema cache now mirrors the table
cache: single-database questions load one database at a time (indexed,
all permission types at once), loops over many databases call
prime-schema-perms-cache, and cross-database questions load everything
once, recorded under :all-dbs? so repeated checks stay one query.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Load perms caches via next.jdbc/plan instead of toucan2 rows
JFR allocation profiling showed toucan2's per-row machinery (row delays,
transient instances, realize maps) dominating the cache loads' share of
request allocation. plan reads columns straight off the ResultSet with
no per-row maps; the folds are unchanged. Runs on the current toucan2
connection so transactional writes stay visible. On the 10M-row bench
this cuts per-request allocation ~10-15% further (e.g. /api/user/current
570MB on master -> 376MB) and 50-90ms of latency on load-heavy
endpoints. Note plan queries are invisible to t2/with-call-count, so
one test now spies on the load fn instead of counting calls.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Store perms caches as mutable Java collections, skipping persistent copies
The folds now build nested HashMaps/HashSets that the caches store
as-is: no per-row persistent update-in churn and no whole-structure
persistent copy at the end. Clojure's lookup/seq functions read Java
collections directly, so consumers are unchanged. Only the two outer
levels (perm-type, db-id) are ConcurrentHashMaps that later loads extend
via putAll of missing databases; everything below is written once by its
load and then only read. Numeric keys are canonicalized to Long at both
fold and lookup time — JDBC hands back Integer for int columns and Java
maps compare boxed keys with equals, not Clojure's equiv. The entry
interner is gone with the persistent copy; instances with thousands of
identical per-database profiles lose that sharing (#76077), which is the
trade-off for the allocation win.
On the 10M-row Postgres bench (200k rows for the user) vs master:
/api/user/current 570 -> 268 MB allocated per request, GET /api/database
?saved=true&include=tables 1773 -> 1032 MB and 4.1s -> 2.5s,
GET /api/table 1152 -> 865 MB and 1.9s -> 0.9s.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Answer cross-database permission questions from a light per-function cache
Instances with thousands of databases and groups make loading tuples
for every database (schema cache with :all-dbs?) scale badly: the
distinct tuple set alone reaches hundreds of thousands of rows. The
questions that actually span databases only need one value per database
for one permission type, so most-permissive-database-permission-for-user
(signature unchanged) and user-has-any-perms-of-type? now share
*most-permissive-perms-cache*: {user-id {perm-type {db-id value}}},
loaded by one indexed query returning distinct (database, value) pairs
and coalesced per database. The :all-dbs? machinery is gone and
query-creation-capabilities needs no priming — its per-database loop
hits the cache after one query.
On the 10M-row bench this takes GET /api/user/current from 722ms/268MB
allocated to 556ms/182MB — 5MB above the response-validation floor
measured with all permission checks short-circuited (master: 833ms/570MB).
Also reorganize the caching section (atoms, then loaders, then the
functions using them; coalesce moves above since a loader needs it) and
relax the xrays call-count expectation: permission-cache loads go
through next.jdbc and are invisible to t2/with-call-count.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Coalesce permissions in SQL and give every check its own per-function cache
Each permission-checking function now owns one atom, one hand-written
aggregate query, and a cache structure fitting exactly its question,
organized per function. Values map to ranks (value-rank-case) and
GROUP BY collapses tables, duplicate values and — via two nested levels
of aggregation for the most-restrictive-within-group-then-most-
permissive-across-groups questions — group identity too, so result sets
scale with tables/schemas/databases rather than raw row counts
(dbs x groups x tables x values). A bucket's [min max] rank pair
reconstructs coalesce exactly, verified exhaustively per permission
type including :perms/view-data's special rule; full-schema keeps its
per-group db-level union by expanding db-level rows into every schema
inside its query. native-download delegates to full-db-permission
(identical semantics), user-has-any derives from the most-permissive
cache.
With the row-count multiplier gone the loads are back to plain t2/query
and persistent Clojure collections: the next.jdbc/plan machinery, the
mutable Java collections, the Long key canonicalization, and the
interner are all gone, and cache loads are visible to
t2/with-call-count again. No public API changes; the batch primer is
now prime-db-perms-cache and warms every per-database cache.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Drop a docstring reference to the removed full-perm-rows
The ~15x realize-before-key-access observation stands on its own; the
function it pointed at no longer exists now that the permission caches
load SQL-side aggregates.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Rewrite app-db NOT IN (SELECT ...) queries as NOT EXISTS anti-joins
NOT IN (subquery) degrades to a per-row subplan on Postgres once the
subquery result exceeds work_mem. Rewrites all runtime occurrences to
correlated NOT EXISTS (or a literal comparison where the subquery was
0/1 rows). MySQL keeps the double-wrapped subquery in
delete-database-fields! since it cannot reference the DELETE target
table in a subquery.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Keep is_audit predicate as NOT EXISTS in set-default-database-permissions!
Tests create audit DBs with random ids, so the fixed audit-db-id
constant is not equivalent to the is_audit = true subquery here.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Snowflake: don't over-escape schema names in the dynamic-table probe
The FK sync path pre-escaped schema/table names (`_` -> `\_`) before handing
them to `dynamic-table?`, which uses the schema as an identifier in
`IN SCHEMA <db>.<schema>`. `"RAW\_DATA"` is not a real schema, so Snowflake
threw for every table in any underscore-named schema and we logged a full
stack trace each time.
`dynamic-table?` now takes raw names and escapes only the `LIKE` argument,
where `_` is actually a wildcard. Escaping for the JDBC metadata call moved
to the call site. Also log the caught exception without the stack trace,
since it fires once per table.
Fixes#78541
Co-authored-by: da <paoliniluis@gmail.com>
* Generate a fresh title for a forked conversation from its first new message
* Show "Forked conversation" instead of "New conversation" in fork header
* Rename first-valid-user-message to first-non-forked-user-message
* Align map value columns in metabot conversation schemas (PR review nits)
* feat: fork a Metabot conversation at an assistant message
* feat: surface Metabot conversation fork origin in analytics
Record forked_from_conversation_id on the conversation and
forked_from_message_id on each copied message, expose them through the
metabot-analytics API and audit views, and link a fork back to its
original thread.
* Update git_branch icon
* Show source conversation title in fork boundary and header
* Revert fork boundary to the attempt where the fork happened
Showing the fork boundary divider on every regenerated attempt was
misleading: the fork only happened on the inherited attempt, so the
divider matches that single message again.
* Refine fork boundary rendering and tests
* Exclude forked (copied) messages from v_metabot_messages
* Address PR review: unify message-action guard, extract ForkBoundary, assert fork toasts via UI
* Reuse createMockMetabotConversationDetail for the fork test fixture
* Replace fork props on Messages with a renderAfterMessage slot; move ForkBoundary to EE
* Drop the redundant UndoListing now that the metabot test setup mounts it
* Index and FK-constrain forked_from_conversation_id so deleting an origin nulls the link
* Split forked messages out of usage metrics with new_message_count
v_metabot_conversations gains new_message_count, which leaves out the
messages a fork copied in from its origin. message_count keeps counting
the whole thread, so the conversations list, the detail page and the
shipped model all report what the reader can actually see.
The usage stats charts aggregate the new column instead, which makes
SUM(new_message_count) reconcile with COUNT(*) FROM v_metabot_messages.
metabot-stats leaves copies out of :metabot-queries and :metabot-users
for the same reason — copies inherit role/ai_proxied and are stamped
with a fresh created_at, so a fork was reporting LLM calls that never
happened.
* Don't offer to fork while a turn is streaming
Only the streaming message's actions were hidden, so earlier assistant
messages kept their fork button mid-turn. Forking then swapped the
conversation out from under an in-flight stream whose reducers are keyed
by agentId alone, so the old turn's remaining deltas rendered inside the
fork.
Withholding onFork while isDoingScience is set is preferable to
cancelling the request: the user asked to branch off an earlier message,
not to stop the response, and loadConversation already avoids cancelling
so an aborted turn isn't recorded.
* Realign the analytics schema maps and fix a migration author
The fork columns are longer than the alignment column the surrounding
maps used. ConversationDetail had already drifted into two alignment
columns before this branch; both maps are now uniform.
Extends the dynamic JSX formatter pattern from #77658 (which did links)
to markdown and email. formatValue and formatEmail now call a registered
renderer through the registry instead of importing ReactMarkdown/Mustache
and ExternalLink directly; the renderers live in ui.tsx and are wired up
at app boot by registerJsxFormatting (called from registerVisualizations).
This keeps the react-markdown/micromark/mustache stack and the mailto
ExternalLink out of the static-viz bundle, which imports the formatting
engine through shared echarts code but never registers the JSX renderers
(so jsx formatting degrades to plain text there, as it already did for
links). No behaviour change in the app.
Static-viz bundle: 3,450,947 -> 3,297,087 bytes (~150KB, 4.5%).
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
metabase.metrics.core re-exports sandbox-restricted-fields from metabase.metrics.permissions, and metabase.metabot.tools.entity-details calls it through that API. The passthrough to metabase.permissions.metric was missing, so importing the re-export threw at load and broke every namespace downstream of metabase.metrics.core.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Transform failure emails list root-cause failures by name and summarize dependents that never ran as a count, so the test asserts the tally instead of the dependent's name.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Add Metric Dimensions section
* [metrics] Field IDs are not unique! Multiple FKs to same table.
Fixes a few places where the field ID alone is assumed to be a unique
key for dimension mappings.
* [metrics] Add basic CRUD API for metric v2 dimensions
Fixes UXW-4631.
Fixes UXW-4633.
* [metrics] Add `/api/metric/$ID/dimension/set-default` endpoint
Fixes UXW-4632.
This sets the dimension of a Metric v2 with the specified
`dimension_id` (a body parameter) to be the `:default true`.
All other dimensions have `:default` removed, so there is at
most one default dimension for any metric.
* [metrics] Metric dimensions create dependencies on the tables
Fixes UXW-4636.
* [metrics] Serdes support for curated Metrics v2 dimensions
Fixes UXW-4634.
Previously `:dimensions` and `:dimension_mappings` were marked as
`:skip` on serdes since the dimensions were always synced on read.
Now that they're curated and only synced when blank, we need to
properly serialize dimensions and mappings.
* Adjust Metric dimensions UI + connect to api
* Add dimensions reorder with dnd
* Add metric dimensions usage in metric viewer
* Change Metric Viewer to group dimensions by type
* Implement shared dimensions logic, add e2e tests
* Use default dimension in Metric Viewer
* Align headings in panels / Make trash and add buttons the same height
* Move and simplify the Set as default button
* Make about page use the default curated dimension in the preview chart
* [dimensions] Bump `:card_schema` to 24; backfill `:dimensions` in old style
Fixes UXW-4808.
This gives pre-existing v2 Metrics the full `breakoutable-columns` as
their dimensions, rather than just the main group as is now the
default. That preserves the old behaviour for the dimensions of the
metrics.
* Respect default metric dimensions on dashboards / Add projectDimension lib function (#77910)
* Respect default metric dimensions on dashboards
* Consolidate metric dashboard E2E coverage
* Remove default time dimension from metric editor
* Implement a few design tweaks in Dimensions tab UI
* Use curated metric dimensions on dashboard metric card filters
* Add dimension picker to metric About page
* Auto-pick default dimensions for new metrics
* Additional cleanup to the About page dimension select element
* Expand all metrics in See all dimensions
* Remove table names from metric filters
* Don't show sub-headers in 'See all' dimension view in Metric Viewer
* Remove metric overview tab
* [UXW-4859] Handle empty curated dimension list and removed default dimension edge cases
* Fix default viz type when adding metric to dashboard
* Remove the 'See all' option on explorer page for single metric visualization
* [UXW-4863] Don't use custom display name as field reference in dimension detail column
* [backend] Fix persistence of upgraded metric dimensions
* [UXW-4880] Re-add Overview tab, show curated dimensions
* [UXW-4848] Make sure we show all shared dimensions in the Shared Dimensions view, if the underlying fields match
* [UXW-4771] Make pinned metric in collection respect default curated dimension
* Update e2e tests to account for PR updates [part 1]
* Seed explicitly joined metric dimensions
* Fix metric dimension serialization dependencies
* Fix/adjust backend unit tests
* Fix metric dimension lint issues
* Run metrics with saved card permissions
* Update e2e tests to account for PR updates [part 2]
* Update kondo ratchet for metric serialization test
* Update metric permissions e2e expectation [part 3]
* Fix BE sandbox unit test
* Prefix upgraded metric dimensions with connected table names (#78410)
[UXW-4896] Prefix upgraded metric dimension names
* Address FE issues from July 22 reviews
* use curated dimensions on series breakout list / fix dimension display name on breakout legend and dimension pill bar
* Optimize getMetricMappingOptions
* Adjust about page dimension select margin-bottom and dropdown positining
* Fix curated metric dimension selection across duplicate foreign key paths
* Improve logic to show correct dimension label in the CenterControls button
* Exclude orphaned dimensions from runtime metric metadata
* Address Ethan's comments / Clean up
* Use metadata field cache when checking user permission / Improve field validation when adding dimension
* Address latest reviews
* Update serialization baseline card_schema 23 -> 24
* Tighten docstrings and comments per review feedback
Remove caller references, ticket refs, and usage context from docstrings
and comments; describe contracts instead.
---------
Co-authored-by: Braden Shepherdson <braden@metabase.com>
Co-authored-by: Anderson <anderson@metabase.com>
Co-authored-by: Anderson <ands.alves.nunes@gmail.com>
* Stabilize SSO group-mapping delete flake by re-reading settled state
Selecting groups for a mapping optimistically patches the settings cache,
but the setting update also invalidates and refetches session properties.
A trailing stale refetch can revert the last-added group after the test's
per-selection checkbox assertion has already passed, leaving the mapping
with only one non-admin group. The delete then fires a single deleteGroup
request and cy.wait times out waiting for the second.
Re-load the auth settings page between adding groups and deleting the
mapping, and assert the row shows its full group summary before acting.
The fresh page load aborts in-flight refetches and reads a single settled
server state, so every mapped group is present when the delete fires.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Serialize SSO group-mapping edits against their setting round-trips
Adding a group to a mapping PUTs the group-mappings setting (built from the
widget's current mappings cache) and invalidates session properties, which
triggers a background refetch. The widget optimistically patches the cache,
but a trailing stale refetch can revert the just-added group; the next
selection then builds its PUT from that stale set and overwrites the previous
group server-side (last-write-wins). The mapping ends up with fewer groups
than the test expects, so fewer deleteGroup requests fire and cy.wait times
out waiting for the second one.
Wait for each mapping mutation's PUT and its trailing session-properties
refetch to settle before the next interaction, in createMapping,
addGroupsToMapping (per group), and deleteMappingWithGroups, so every edit is
built from a fully committed round-trip. deleteGroup/clearGroup invalidate
only permissions-group, so updateSetting is the sole source of session
refetches and the waits stay aligned one-to-one.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* update getExtraTagsForVersion to append -lts if isLts is set
* add ability to check if a version is an LTS version
* set islts
* update function to determine if we should tag with lts
* add condition for multiple LTS releases so we only tag the latest
* fix actionlint errors
* update lts tag to include major version
* await result of getExtraTagsForVersion
* fix type
* fix merge conflict, don't throw error
* GDGT-2851 Remove color-mix from theme color definitions
color-mix can only be resolved by a browser, so any theme color using it
breaks static viz, which renders on the server (see UXW-4690, where the
gauge failed on background_page-tertiary-inverse).
Per design's proposal, introduce a new orion-95 stop and use solid scale
values for the page backgrounds:
- dark: page primary/secondary/tertiary -> orion 95/100/110
- light: page primary/secondary/tertiary inverse -> orion 95/100/110
The remaining mixes are collapsed to their literal equivalents:
shadow-default to a plain rgba, and tooltip-background-focused to the
next stop down from tooltip-background.
The mixes that derive from whitelabel colors (core-brand, core-filter,
core-summarize) are left alone, since their values aren't known ahead of
time. A unit test now guards against reintroducing color-mix anywhere
else in the theme definitions.
* Update Loki Snapshots
* Remove var() references and non-hsla literals from theme definitions
Static viz feeds theme colors straight into Color(), which only parses
literals, so a var() throws there exactly like a color-mix does — even
when it points at a perfectly ordinary color. Collapse the remaining
var() references to the underlying base color, and normalize the few
hex and bare-keyword values to hsla so every theme color is a literal.
The guard test drops the transitive resolver it no longer needs and just
asserts that every non-whitelabel color is an hsla literal.
* Fix regression in pivot table
* Empty commit to trigger CI
* Manually update pivot table loki snapshot
* Update e2e colour assertions for the new background tokens
Dark background_page-primary moved from orion-100 to the new orion-95,
so anything asserting rgb(7, 23, 34) for it now expects rgb(12, 28, 39).
The dark mode body assertion was matching color(srgb 0.0204 0.06792
0.0996) — Chrome's computed form for the old color-mix on
background_page-secondary. That token is a plain hsla now, so Chrome
reports it as rgb(7, 23, 34).
* Fix e2e
* Fix e2e
* Revert changes in assertDarkMode
* Fix oss e2e spec
* Revert SDK theming assertions to the deprecated background token
The SDK theme presets resolve colours through the first entry of
SDK_TO_MAIN_APP_COLORS_MAPPING, and for `background` that is the
deprecated `background-primary`, which still points at orion-100.
`background_page-primary` sits at index 1 and never gets read, so
table-root is unaffected by the move to orion-95.
* Address review feedback on the colour theme changes
Move the gold and upsell-gem literals into base-colors, since each was
repeated across both theme files. Drop the intermediate staticColors
binding in the guard test in favour of a second filter, and drop the
`?? {}` on theme.colors, which is a required field.
---------
Co-authored-by: Metabase Automation <github-automation@metabase.com>
* Fix click-behavior multi-stage e2e flake: settle mapping list between selections
The "navigating to questions with filters applied in every stage" test sets 11
click-target mappings sequentially, using index-based selectors (.first(),
.last(), .eq(2)) against the unset-click-mappings list. That list re-renders as
each mapping moves from the unset to the set section, but there was no barrier
between steps, so an index-based lookup could resolve mid-re-render and land on
the wrong target — yielding a stable-but-wrong filter set (a duplicated
"Product → Title" filter replacing the dropped "Reviews - Product → Reviewer"
one, which is exactly the assertion that fails).
Extract source selection into selectClickMappingSource(), which asserts the
source picker popover has closed before returning, so every subsequent
index-based lookup runs against a settled list.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix click-mapping settle barrier to assert popover closed correctly
The previous settle barrier used `H.popover().should("not.exist")`, but the
`H.popover()` helper internally chains `.filter(":visible").should("be.visible")`,
so the assertion is self-contradictory: it requires a visible popover to exist
and then asserts it does not, failing deterministically the moment the source
selection closes the popover.
Assert the popover is gone with `cy.get(H.POPOVER_ELEMENT).should("not.exist")`
— the idiomatic pattern used across the e2e suite — so the mapping list still
settles between index-based selections without the contradictory visibility check.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Scope click-mapping settle barrier to the visible popover
The unset-click-mappings list re-renders as each mapping moves unset->set.
A bare `cy.get(POPOVER_ELEMENT).should("not.exist")` can never settle in the
click-behavior sidebar: it keeps ~60 mantine popover/combobox dropdowns
mounted-but-hidden, so they are continuously found in the DOM. Filter to
`:visible` so the barrier only waits for the open source picker to close.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Export queryToSearch and searchToQuery from metabase/router
* Serialize the query with queryToSearch at the navigation call sites
* Read search, not query, off a navigation descriptor
* Drop query from LocationDescriptorObject
* Thread QB popState off the routing slice
* Thread dashboard edit-mode off the routing slice
* Thread Metabot profile selection off the routing slice
* Thread document Metabot context off the routing slice
* Migrate component location readers to useLocation
* Retire the state.routing slice
* Convert getExpandedCollectionsById to TypeScript and fix the stale ExpandedCollection type
* Type fake collection ids as literals instead of casting to CollectionId, dropping the String() workaround
* Audit log statements: no stack traces, no queries/user input, no pretty-printing
- Replace throwable args to log macros with (ex-message e) so ex-data and
stack traces never reach the logs (~450 sites)
- Remove SQL/MBQL queries, prepared-statement params, HTTP request/response
bodies, PII, and user-authored names from log output (IDs kept)
- Replace u/pprint-to-str in log calls with pr-str, or drop the value
- Silence sqlglot parser warnings in the GraalPy bootstrap (they quote raw
chunks of customer SQL to stderr)
- Rewrite leftover functional colors (hsl/hsla/rgb/rgba) to hex in static-viz
SVG output so Batik never errors (and stack-dumps) on them
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Use logf variants for exception messages; restore serdes --full-stacktrace
- Convert print-style `(log/warn "msg:" (ex-message e))` calls to
`(log/warnf "msg: %s" (ex-message e))`
- Serdes import/export logs the full stack trace again when the
--full-stacktrace CLI option / full_stacktrace API param is passed
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Update swap-clauses tests for clause-free warning messages
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Update log-content test assertions for the new logging policy
Log calls no longer attach throwables (entries have :e nil) and append
ex-message text to the message instead; body/secret leak tests now assert
the payload never appears in log output.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Address review comments
- rethrow-api-error!: update docstrings — response body is never logged,
it travels on the thrown ex-data
- expand-macros: log the segment filter-clause count once instead of an
identical trace line per clause
- settings cache: log the whole SQLException message chain (the old
print-sql-exception-chain walked .getNextException; ex-message alone
dropped the chained causes)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Add Claude Opus 5 to the anthropic openrouter and bedrock whitelists
- anthropic claude-opus-5
- openrouter anthropic/claude-opus-5
- bedrock anthropic.claude-opus-5
* Scope the invite group picker to the groups with access to the shared item
* Imply All Users membership only for invites with a target, and mock the scoped groups endpoint in share menu specs
* Show all groups when inviting to view and mark which ones can access the item
* Make the invite picker's group section labels visible and tidy up UserForm's membership mapping
* Generalize the group picker into ordered labeled sections with group a11y roles, keeping invite-target semantics in UserForm
* Use the compact Alert size for the invite picker's no-access warning
Suppress api-key-missing exception when failing to generate sample question.
Not having an API key is the default situation and should not result
in an error being logged.
* Let library retrieval use pgvector on the app db
The tables were bare-named, so the drop-and-rebuild in ensure-tables! would have
resolved against the application database's default schema. That is what kept
the feature dedicated-only.
Table names now come from the pgvector mode, the way semantic search picks
between its two index-metadata configs: bare for a dedicated store, so no
existing index moves, and qualified by a library_retrieval schema when sharing
the app db. Its own schema, not semantic search's, whose migration drops every
table in it.
Raw SQL interpolating a qualified name into "%s" would read it as one
identifier, so those sites go through quote-table.
The sync task's scheduling gate stays boot-safe: a dedicated URL, or the license
plus a Postgres app db, decided without resolving the mode. available? checks
the license before the store for the same reason.
* Cover the app-db round trip end to end
Opt-in like the semantic-search one, and picked up by the same appdb-mode CI
job. Asserts the tables land in library_retrieval, the application schema is
untouched, retrieval returns the indexed entity, and a semantic-search wipe
leaves the library index alone -- the reason the schema is separate.
* Check our own schema before reporting the store ready
Semantic search's predicate answers for semantic_search. A role that can use
that schema but cannot create library_retrieval passed it, and then failed every
reconcile inside ensure-tables!. Probe for our schema too: present, or creatable
in a rolled-back transaction, cached because available? runs on every write
nudge and search.
* Gate the round trip on the capability the feature checks
Gating on semantic search's check let the harness pass where a role that can use
semantic_search but not create library_retrieval fails in production.
* Check schema privileges, not schema existence
A role with USAGE but not CREATE passed the existence check and then failed
creating tables. Ask Postgres for both privileges instead, and only fall back to
the rolled-back CREATE SCHEMA when the schema isn't there at all.
The whole probe is inside the try now: the existence query wasn't, so a
transient database failure could throw out of request-entity-sync!, which the
write hooks call fire-and-forget.
* Put the dedicated index in the schema too
A dedicated store kept the index in the default schema, which is exactly what
semantic search's migration wipes -- so enabling or migrating semantic search
dropped the library index and silently re-embedded the whole library. It fires
whenever the stored schema version trails the code, not just on first init.
Both modes now use the library_retrieval schema, so default-tables and
app-db-tables collapse into one. An index built before the schema existed is
moved into it with ALTER TABLE ... SET SCHEMA, keeping its rows; the move runs
before the IF NOT EXISTS creates, which would otherwise leave an empty table
beside it. Creating the schema now needs CREATE on the dedicated database, so a
failure says so.
* Drop only semantic search's own tables in a dedicated store
A dedicated store's default schema is shared -- library retrieval keeps its
index there until it moves -- so wiping every table in it could drop that index
before entity retrieval had a chance to adopt it. Both run as background jobs,
so nothing ordered them.
Scoped to the index_/dlq_/repair_ prefixes, and only in dedicated mode: the
module's own schema in app-db mode holds nothing else, so a reset there is still
free to clear leftovers. The app-db sentinel refusal still reads every table in
scope, since an app db is recognized by tables we would never drop.
* Match the names semantic search generates, not their prefixes
A cohabitant in a shared dedicated store is free to call something
index_history, which the prefix check would have dropped. Match the control
tables exactly, index tables through index-table-name? -- the same contract
orphan cleanup uses -- and DLQ and repair tables by their generated shapes.
Also covers the adoption path end to end: a legacy unqualified index with a row
in it, moved by ensure-tables! and still holding the row afterwards. Asserted
there rather than after the reconcile, which GCs documents the library no longer
implies.
* Stop the availability tests depending on the app db
available?-gates-test left the real schema probe live while stubbing only
the pgvector mode. The probe is Postgres-only SQL, so it threw and read as
unusable on the H2 and MySQL runs, and the case failed there.
The round trip's legacy table declares a vector(4) column before
ensure-tables! runs, and the support check verifies CREATE EXTENSION in a
transaction it rolls back, so nothing has installed the type by then.
* Answer review on the dedicated-store reset
Rename ours? to semantic-search-table?, and point the two docstring
references at namespaces the reader can resolve.
Cover the case where a legacy bare table and a qualified one both exist:
adoption has to leave them alone, since SET SCHEMA onto an occupied name
errors and would abort the whole ensure-tables! transaction.
* Latch the app-db schema check once it passes
The check catches everything and returns false, so on a plain TTL a single
transient app-db failure took retrieval offline for five minutes: available?
went false, search returned nothing and the write nudge no-oped.
A confirmed yes now sticks for the life of the process. Only a no keeps
re-checking, so a privilege granted while we run is still picked up.
* Stub the evals trigger
* use ref arg
* dummy to trigger build
* Clean up
* Use appropriate token
* Simplify pr message
* Use .evals_ref for default ref
* Comment cleanup
* Adjust ref
* Use evals branch
* Use general automation token
* readme for .evals_branch
* Use main of evals for label trigger
* Revert "dummy to trigger build"
This reverts commit e6502c6ff076a47d94eb829b516fc45b37cd7680.
* whitespace
* Address review remarks
* Use ai benchmark naming so we can trigger from pr
* Address review comments
* Remove now redundant branch pinning
* Review remarks
* feat: add lookback
* validate lookback
* feat: apply lookback
* tests
* fix
* fix test
* feat: only support lookback on temporal checkpoints
* update test
* FE
wip
* add :type/Time test
* add test with watermark row removed
* refactor to avoid double get-in
* only support date/datetime
* tests
* fix: support coerced cols
* add more tests
* refactor
* more tests
* test for coerced instant
* use utils in FE too
* chorec
* further validate
* use lib utils
* use strict comparison
* allow clearing lookback
* handle race
* chore: migrate registerVisualization to typescript
* chore: drop 'as unknown as Visualization' cast in custom viz plugin loading
applyDefaultVisualizationProps now takes the plain component and returns
the typed Visualization via Object.assign. The SDK's date_style format
option is widened to string | null to match the ColumnSettings value the
host actually passes to checkRenderable.
* Unit test the native editor run button's unmapped-tag tooltip
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Unit test the Save button's unmapped-tag tooltip and disabled state
Covers both dimension and temporal-unit template tags.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Cover temporal-unit tags in the canRun template tag validation test
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Drop the unmapped field filter e2e test in favour of unit coverage
Every assertion it made is now covered by NativeQueryEditorRunButton and
ViewTitleHeader unit tests.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Read useLocation and useParams straight from react-router v7
* Re-export react-router's Outlet instead of shimming it through context
* Disable the command palette by path instead of route props
* Drop the now-unused route props plumbing
* Add mdb/with-unshared-connection: appdb connections exempt from ambient reuse
A connection holding fragile long-lived state -- a streaming result set
(postgres portal/cursor), or FOR UPDATE row locks -- must not be picked
up by ambient connection reuse: a borrower that inherits it via
*current-connectable* and commits destroys that state. This is the
failure mode behind the first-boot hang in #78238, the revert (#77048)
of appdb streaming (#76645), and "cursor/portal does not exist" errors
when field updates run mid-sync-stream.
Built on toucan2's new unshared-connection! (camsaul/toucan2#212): the
connection is used when passed explicitly -- (t2/reducible-query conn
...), (t2/with-transaction [t conn] ...) -- but never bound as
*current-connectable*, so toucan calls that resolve their connection
ambiently run (and commit) on other pooled connections.
mdb/with-unshared-connection is checkout -> mark -> body -> close; in
every other respect the connection is ordinary JDBC, configured by the
body (e.g. .setAutoCommit false to stream). The pg test reproduces the
exact #77048 failure on an unmarked control connection (portal killed by
a nested commit mid-reduction) and shows the unshared connection
streaming to completion.
deps.edn temporarily pins toucan2 to the PR sha via :git/sha; a
follow-up commit will switch back to :mvn/version once a toucan2
release includes it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G7FfdpzW5pr7CX9aNwa98f
* Use released toucan2 1.0.572 for unshared-connection support
camsaul/toucan2#212 merged and released; drop the :git/sha pin.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G7FfdpzW5pr7CX9aNwa98f
* Hold the audit DB cluster lock on a detached connection (fixes boot stall)
Since #76551 the audit install/load/sync pipeline runs inside a cluster
lock whose SELECT FOR UPDATE ties lock lifetime to transaction lifetime:
the whole pipeline became one long transaction on the lock's connection.
The serdes load's ~3,600 per-entity nested transactions piled up as
savepoints and subtransaction XIDs, and every row lock was held until a
single commit at the end -- stalling first boot for 10-25+ minutes and
blocking concurrent work like config-file database provisioning (#78238).
with-cluster-lock now supports :detached? -- the lock rows are held on a
dedicated unshared connection (toucan2 unshared-connection!) while the
body runs on ordinary pooled connections, committing incrementally in
its own short transactions. Locks release at body completion (commit) or
throw (pool check-in rollback). Re-acquiring a lock this thread already
holds detached is a no-op in both detached and transactional form;
upgrading a detached :share hold to :exclusive throws instead of
self-deadlocking; :retry-transient? is rejected (a re-run would
double-apply incrementally-committed work).
The audit pipeline opts in: per-entity serdes isolation (#74412) is
restored, and a crash mid-pipeline is repaired by the next boot (the
checksum only advances on completion; reconcile runs every boot), which
is what makes incremental commit safe here.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G7FfdpzW5pr7CX9aNwa98f
* Address review: confine detached retry to acquisition; repair dialect-sync window
Review findings (galdre), all majors addressed:
- Major 1: retry/with-retry wrapped the whole detached body, so a
retryable-looking error thrown by the body (e.g. a nested cluster lock
timing out inside the serdes load) re-ran an incrementally-committed
body and then surfaced as "Failed to obtain cluster lock", which
ensure-audit-db-installed! swallows as benign contention. An acquired?
volatile now confines both the retry predicate and the acquisition-
failure wrapper to the pre-acquisition phase in detached mode; body
errors propagate raw with intact cause chains.
- Major 2: a failed or killed engine-changed dialect sync was never
repaired once the content checksum had advanced. New durable setting
audit-db-dialect-sync-pending: set before the sync attempt, cleared on
success, a third sync trigger on every boot.
- Major 3: the mysql/h2 engine-flip visibility window under detached
mode is documented as a KNOWN WINDOW at adjust-audit-db-to-source!
(restores pre-#76551 semantics; overlay-based resolution is follow-up).
- Majors 4-6: reviewer's contract tests adopted (throw-path lock
release, check-in rollback, behavioral pin on :detached?, crash-mid-
load self-heal, plus the two Major-1 reproductions); public docstrings
updated to describe all three lock modes and the upgrade throw.
Minors: unified cross-mode reentrancy registry with eager share->
exclusive upgrade validation (fail-fast in any detached interleaving;
legal same-transaction upgrade preserved); h2 read->write upgrade throws
instead of hanging; lock-row INSERT gets a query timeout (bounds the
N-node cold-start pile-up); install-database! and each reconcile group
are single transactions; unshared checkout goes through *application-db*
so the restore-time read-lock gate applies; registry conveyance and
explicit-tx-on-unshared-connection caveats documented; dual kondo
lint-as entries; test hygiene (finally cleanup, nil ambient contract).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G7FfdpzW5pr7CX9aNwa98f
* Right-size the engine-flip visibility comment
The committed, briefly-visible engine flip is this pipeline's
longstanding behavior; only #76551's single-transaction era hid it for
one release. A neutral note fits better than a warning block.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G7FfdpzW5pr7CX9aNwa98f
* Tighten comments to constraints; drop review-context narration
Comments now state what the code can't show (crash windows, why raw
JDBC, why the marker precedes the sync) and drop justification written
for reviewers: issue-number history, caller name-drops, repeated
tellings of the retry-confinement story, and cost reassurances.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G7FfdpzW5pr7CX9aNwa98f
* Make the detached lock a bespoke entry point instead of a with-cluster-lock mode
Reviewer pressure on the :detached? option had generalized
with-cluster-lock into a two-mode API whose cross-mode interleavings all
needed guarding: a held-locks registry, mode-upgrade validation, an h2
upgrade guard, and an acquired? volatile threading retry confinement
through the shared machinery — all protecting call patterns nothing in
the tree performs.
with-detached-cluster-lock is now its own function: exclusive-only,
single lock, documented non-reentrant. Structure replaces machinery —
the body sits outside the acquisition retry, so body errors are never
retried nor rewrapped as acquisition failure (no volatile), and
same-name re-entry throws via a two-line dynamic-set guard instead of a
registry (nested *different* locks, which the audit body does take,
work unchanged). with-cluster-lock returns to its previous shape; the
lock-row INSERT keeps its new query timeout, which bounds the
multi-node cold-start pile-up in both modes. Error shape is unchanged,
so audit-lock-contention? still classifies correctly.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G7FfdpzW5pr7CX9aNwa98f
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Compact in-memory format for the per-request data-permissions cache
The request-scoped cache in metabase.permissions.models.data-permissions
previously stored raw data_permissions rows as full Toucan instances, keyed
{user-id {perm-type {db-id [row ...]}}}. On instances with thousands of
databases and users in many groups a single prime-db-cache call could
materialize over a million instances (hundreds of MB retained per request,
plus the matching GC churn).
Store a compact entry per (perm-type, db-id) instead:
{:groups {group-id perm-value} ; db-level rows
:tables {table-id [schema-name {group-id perm-value}]}} ; table-level rows
and intern the group->value maps at load time: on big instances they are
typically identical across most databases, so the retained size becomes
proportional to the number of distinct permission profiles instead of
groups * databases.
Also fetch only the columns the checks actually consume (group_id, perm_type,
db_id, table_id, schema_name, perm_value) instead of p.*, and fix an O(n^2)
already-cached filter in prime-db-cache.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* small fix
* Use metabase.util.performance helpers in the cache hot path
perf/concat + perf/mapv in entry-group-value-pairs. Deliberately NOT
perf/update-vals for the interning pass: it skips replacing values that
are = to the old ones, which would undo the interning.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Drop perf helpers from entry-group-value-pairs
perf/concat over an already-built vector is just (into a b), and reduce-kv
iterates the map faster than mapv over entry seqs. Plain into + reduce-kv,
one less require.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Address code-review findings on the compact permissions cache
- Duplicate data_permissions rows (no unique constraint on
group/type/db/table) no longer silently last-win: cache slots promote to
a set when duplicate rows disagree, so every value reaches the consumers'
coalescing logic exactly as raw rows did. Regression test added.
- schema_name is stored per table-level row ({:schema .. :value ..})
instead of first-row-wins per table, since it is denormalized and rows
for one table can disagree.
- The interner now lives in the *permissions-for-user* atom, so value
sharing spans every prime in the request (db-at-a-time priming
previously got a fresh pool per call), and it reuses
metabase.util.memoize/fast-interner instead of a bespoke pool.
- rows->cache-entry returns nil for empty rows so cached and non-cached
get-permissions branches agree; the non-cached branch skips interning.
- schema-permission-for-user and most-permissive-database-permission-for-user
now go through entry-group-value-pairs instead of hand-rolled traversals.
- Dropped the impossible-nil guards in prime-db-cache.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix
* Address CodeRabbit nitpicks
- gid->slot -> group-id->slot (no abbreviations)
- native-download-permission-for-user reuses most-restrictive-per-group
instead of re-implementing the grouping inline
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Rename local intern -> interner (Eastwood local-shadows-var)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Explain the slot rationale in conj-slot/slot-seq docstrings
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* simplify slots
* Document interning with a worked example in the cache docstrings
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014V8QbHjoEkJ8eZLG13sYUe
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: dan sutton <dan@dpsutton.com>
* Restyle the sharing popover with Copy link and Embed buttons above the actions list
* Reserve the copy button width with hidden pseudo-elements instead of inline styles
* swap subscriptions action with auto-refresh action
* hide Subscriptions for non-admins if slack and email aren't set up
* small adjustments
* e2e adjustments
* more e2e adjustments
---------
Co-authored-by: Nick Fitzpatrick <nickfitz.582@gmail.com>
* Convert the legacy chart legend components to TypeScript function components
* Share the legend prop types and extract a typed getChartLayout helper with tests
Deleting a Database hard-deletes its Cards with raw SQL, but nothing
rewrites the `parameters` JSON on Cards/Dashboards that used one of
those Cards as a dropdown source. The parameter keeps a
`values_source_config.card_id` pointing at a row that is gone.
On export that made `*export-fk*` throw `:target-not-found`, which
`log-and-extract-one` swallows as a skip — silently dropping the whole
Card or Dashboard from the archive.
Export each parameter on its own now, and when one carries a card
values-source that can no longer be resolved, emit it without
`:values_source_type`/`:values_source_config`. That is the same shape
the app writes when a source Card is archived, so the parameter comes
back as a plain dropdown over its connected fields.
A source Card that merely isn't part of the export is unaffected —
`*export-fk*` only throws when the row is absent, so those still fail
dependency validation as before.
Fixes GHY-3259
The helper inserts straight into the table to bypass the model's transforms,
which bypasses the hook that fills in entity_id too. #78452 made that column NOT
NULL, so every App DB Tests job has failed on it since.
* Eagerly load tenant auth-provider; fail tenant SSO logins loudly (UXW-4898)
Root cause: metabase-enterprise.tenants.auth-provider — the login! method
that validates a login's tenant claim and stamps :tenant_id onto the user
data — was never eagerly loaded. The JWT/SAML providers only `derive` from
its dispatch keyword, which does not load the namespace, so on a freshly
started instance the tenant step was silently absent from the login!
method chain until unrelated traffic (e.g. /api/user/current hydrating
user attributes) lazily loaded it. A JWT provisioning login in that window
created the user with tenant_id = nil even though the token carried a
valid @tenant, the tenant-group assignment failed and was swallowed, and
the login returned 200 — permanently wedging the account (every later
login 403s with "Cannot add tenant claim to internal user").
Fix, plus defense in depth so no future tenant-drop can wedge an account:
- Add metabase-enterprise.tenants.init requiring the auth-provider, wired
into metabase-enterprise.core.init, so the tenant login step is always
registered at startup.
- create-user! throws instead of silently dropping :tenant_id when the
sso-user-fields field list would strip a tenant assignment stamped by
the upstream tenant flow.
- sync-group-memberships*! no longer swallows real tenant/group mismatch
errors (bad pairs whose :group-is-tenant? is non-nil); the login fails
loudly instead. Nonexistent groups keep log-and-continue. This also
covers EMB-2118 (JWT omitting @tenant with a mapped tenant group).
- The login! provisioning chain (tenant creation, user create/update,
group sync) runs in one transaction, so a failed login leaves no
half-provisioned user row behind.
* Fix cljfmt formatting in jwt_test.clj
* Stop the location-syncing call sites from replacing the url they just read
* Cover the documents /new to /new warning with a unit test
* Adopt v7 navigation semantics, drop the v3 parity guards
The v7 provider mirrored every location into state.routing by calling
useDispatch itself, which is what kept metabase/router bound to the store.
createLocationMirror takes a plain dispatch function instead, so the router
publishes locations without knowing redux exists. app.js and the test harness
supply it as RouterProvider's onLocationChange prop. V7ReduxBridge is renamed
V7RouterBridge now that nothing in it is redux-specific.
metabase/router now passes a basic-tier boundaries trial with zero violations.
Guards are authorization policy, not routing primitives. They were the largest
source of shared-tier imports inside metabase/router (redux, plugins, and four
selector modules), blocking the module from dropping to a lower tier.
The generic factory and the app-wide auth/setup guards move to a new shared
metabase/route-guards module. The monitor and data-studio guards move into their
own features, which lets getMonitorRoutes and getDataStudioRoutes stop taking
them as parameters.
Router violations under a basic-tier trial go from 13 to 3; the rest are the
redux location mirror in the v7 provider.
* Add GeoJSON error mocks and cover map load failures in unit tests
`setupGeoJSONEndpoint` could only produce a valid FeatureCollection, so the
error branches of the custom map flow had no unit coverage.
- `setupMalformedGeoJSONEndpoint` returns a 200 with a payload the client-side
validator in `metabase/api/geojson` rejects, since those checks run in
`transformResponse` and need the payload to reach the client.
- `setupGeoJSONEndpointWithError` returns a non-2xx with a bare string body,
matching what `api-exception-response` sends for an `ex-info` carrying only
a status code.
- `createMockProjectedGeoJSONFeatureCollection` supplies projected coordinates
so every bounds corner falls outside lat/lng range.
Covers the invalid-location, missing-features, and out-of-bounds errors in
CustomGeoJSONWidget.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Drop map GeoJSON error e2e tests in favour of unit coverage
All three asserted error strings are now covered more cheaply. The two
backend-owned messages are pinned by metabase.geojson.api-test against mocked
HTTP, and the modal rendering them is covered by CustomGeoJSONWidget. The two
client-side validation messages come from transformResponse in
metabase/api/geojson, which the e2e tests reached with the backend fully
stubbed or bypassed.
Two of them also drove real outbound requests from CI — one to www.metabase.com
and one to a file on the public repo's master branch.
projected.geojson had no other consumer; the equivalent payload is now built by
createMockProjectedGeoJSONFeatureCollection.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Drop admin settings e2e tests that duplicate existing unit coverage
Each of these asserts the same condition as a unit test that already exists,
not merely an adjacent one:
- the HTTPS redirect check failure message is asserted in HttpsOnlyWidget
("should display an error message if the site is not accessible via https")
- the billing info fetch failure is asserted in LicenseAndBillingSettings
("renders error for billing info for store managed billing and info request
fails")
- the absent license input for hosted instances is asserted in
LicenseAndBillingSettings ("does not render an input when token has hosting
feature enabled")
mockBillingTokenFeatures and its feature-key constants had no remaining
callers.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Persist disabling a collection from remote sync (GHY-4189)
Disabling a collection deleted its RemoteSyncObject rows outright, so the
removal left no dirty state to push and the collection's files stayed on the
remote. A later pull then re-imported the collection (its YAML still carried
is_remote_synced: true), re-enabling it.
Mark the collection's and its contents' RemoteSyncObject rows as "removed"
instead — a pending deletion the next export pushes to the remote — while
dropping rows still in "create" (never pushed, so nothing to delete remotely).
* Don't delete contents when a collection is re-synced before the push
Disabling a collection records a pending removal on its contents'
RemoteSyncObject rows. Re-enabling it re-tracked only the collection's own row,
so the contents stayed marked for deletion and the next export removed their
files from the remote.
Clear the recorded removal when a collection is re-synced before that removal
was pushed. Restore to "update" rather than "synced": edits made while the
collection was un-synced are not tracked, so the entity must be re-serialized
for the remote to be guaranteed to match local.
* Stop core-test leaking RemoteSyncObject rows between tests
Disabling a collection now marks its RemoteSyncObject rows rather than deleting
them, so rows seeded by a test outlive it and leak dirty state into every later
test. The old delete-on-disable behavior had been doing this cleanup by
accident.
Reset the table around each test with the clean-object fixture.
* Assert the pull succeeds in the disable round-trip test
The test discarded impl/import!'s result, so an errored pull would leave the
collection un-synced for the wrong reason and the following assertion would
pass vacuously.
* Re-check eligibility at write time so a pending removal isn't resurrected
An event handler computes eligibility from its event payload, then writes the
RemoteSyncObject row much later. A disable committing in between left the
handler writing a tracked status over the "removed" status the disable had
recorded, so the entity stayed on the remote after being un-synced.
Re-check eligibility against current DB state before letting a tracked status
overwrite a pending removal, and let the removal stand when it no longer holds.
* Row-lock the sync entry so the un-sync race has no remaining window
Re-checking eligibility at write time narrowed the race but did not close it: a
disable could still commit between the re-check and the update, and the losing
write would take effect.
Row-lock the entry for the transaction so a tracking write and a concurrent
un-sync settle in a fixed order instead of losing one of the two: whichever
locks first commits, and the other observes that result — the un-sync re-marking
the row, or the eligibility re-check seeing the entity leave the synced set.
The race test drives the disable on its own thread and unparks the handler
without waiting for it, since the parked handler holds the row lock the disable
needs. Its assertion holds for either order, so the choreography only biases
which one is exercised.
* Address review: honest race coverage, insert-path guard, tighter asserts
- Split the concurrency test into two, each parking the handler at the point
that exercises one ordering: disable-first (guarded by the write-time
eligibility re-check) and handler-first (guarded by the row lock, which the
disable now provably blocks on). The prior single test parked on the plain
existing-entry read, so it only ever exercised the re-check ordering while its
comment claimed lock contention.
- Re-check eligibility in the insert branch too: a stale tracked-status event
that finds no row inserts a fresh one, a phantom the row lock cannot cover.
This narrows, but cannot fully close, that window.
- Round-trip pull test asserts = :success rather than not= :error, so a
:conflict result can no longer slip through.
- restore-removed-rsos! docstring no longer claims it only clears removals from
record-removed-rsos!; it clears any 'removed' row in the subtree.
* Re-track never-pushed contents when a collection is re-synced
Disabling a collection drops its contents' never-pushed 'create' RemoteSyncObject
rows (the remote never had them, so nothing to delete there). Re-enabling only
restored the rows that were marked 'removed', so a card added-but-never-pushed
before the disable was left with no row — and the next incremental export, seeing
no dirty row for it, silently omitted it.
On re-enable, insert a 'create' row for every eligible content entity in the
re-enabled subtree that has none, so the next export pushes it. The
RemoteSyncObject table only records pending changes, so already-synced content is
legitimately untracked; re-marking it 'create' re-serializes it harmlessly (a
'create' onto its own path is a no-op).
* scaffold: two-step SDK release process (EMB-1518)
Replaces the hardcoded version/dist-tag computation in
release-embedding-sdk.yml with a new sdkRelease metadata block in
package.template.json (distTag, tagAsLatest), plus a new
sdk-version-bump-pr.yml workflow that bumps the version via `npm version`
and opens a PR labeled sdk-release-trigger for review. Merging that PR
triggers the release workflow directly.
Unverified: scaffolded per EMB-1518 ahead of Nicolo/Roman's review, since
this can only really be exercised by triggering a real SDK release. See
.claude/kelvin/2026-07-09-emb-1518-simplify-sdk-release-process-and-checklist-maintenance/
for the plan and open questions (particularly around distTag semantics for
stable/patch/minor releases).
* fix: correct distTag comment, <major>-stable is real (EMB-1518)
npm dist-tag ls @metabase/embedding-sdk-react confirms 51-stable
through 62-stable are already live, actively-used tags - not a new
concept as the prior comment assumed. Resolves the one open question
this scaffold couldn't settle from the proposal text alone.
* fix: address review cycle 1 findings (EMB-1518)
Fixes release-breaking and correctness issues found in review cycle 1
of the SDK release process scaffold:
- beta release_type now defaults to <major>-beta dist-tag instead of
the literal (non-existent) "beta" tag
- validate branch/release_type combinations that the old
getNextSdkVersion used to hard-error on (master requires alpha,
release branches reject alpha)
- pull_request trigger on release-embedding-sdk.yml now scoped to
package.template.json via a paths filter, and the concurrency group
no longer cancels an in-flight release on unrelated PR closures
- readme-PR follow-up steps now skip cleanly when the readme was
already up to date, instead of failing and firing a false alert
- tagAsLatest now always resets to false rather than carrying forward
a prior true, and the actual value is shown in the summary/PR body
- version-bump branch name includes the run id to avoid collisions on
retry
- renamed a step whose name didn't match what it does
See the "Review cycle 1 (fable)" section and the new "Fixes applied
for review cycle 1" subsection in the plan file for full detail on
each finding and its disposition.
* fix: address review cycle 2 findings + simplify version bump (EMB-1518)
Cycle-2 findings: add a preminor release_type so master's next-major
alpha bump no longer needs a hand-made PR (#9); make the Slack
failure notification independent of an artifact upload that may
never have run (#10); guard against two open version-bump PRs racing
each other (#11); stop interpolating the free-text branch input
directly into shell scripts (#12).
Also replaces the npm-version rename dance (rename to package.json,
bump, rename back, plus a .bak copy) with direct calls to the semver
package the bump logic already wraps, writing the result via jq
instead - no functional change, removes the dead .bak file (#13).
* fix: address review cycle 3 findings + rename resolve-release job (EMB-1518)
Fixes findings #14-#16 from the third fable review cycle:
- #14: the failure Slack alert's messageId interpolation was unquoted,
rendering a JS syntax error whenever send-slack-notification-start was
skipped (the exact scenario cycle-2's fix#10 was meant to cover). Now
quoted, and falls back to sendSlackMessage (new top-level message)
instead of sendSlackReply when there's no thread to reply to.
- #15: finished the cycle-2 injection-hardening pass by routing the two
remaining inline branch-value usages (workflow-summary's heredoc,
publish-npm's gh pr create --base) through env: instead of ${{ }}.
- #16: added a semver validity pre-check before the bump step, so an old
release branch's 0.N.x placeholder version fails with a clear ::error::
instead of a silent exit 1 from the semver CLI.
#17 (minor's version-collision caveat on old release branches) is left as
an open question for Nicolo/Roman rather than removed - documented with a
comment and Judgment call #6. #18 (merge-commit-sha race) is deferred -
it fails loudly rather than silently, lower priority than the others.
Also renames the resolve-release job to prepare-release throughout
release-embedding-sdk.yml, per explicit request (not a review finding).
See the plan file's "Fixes applied for review cycle 3" section for the
full verification detail.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U2Va4ryWuZbQvmUt2N85ec
* feat: auto-compute tagAsLatest from vars.LATEST_VERSION (EMB-1518)
Resolves Judgment call #2: the SDK version-bump workflow previously
always defaulted tagAsLatest to false because there was no reliable
way to know if a branch is the current "latest" major. The repo
already exposes vars.LATEST_VERSION (major version of the most recent
gold/stable release) for exactly this, so patch/minor bumps now set
tagAsLatest=true when MAJOR_VERSION matches it, false otherwise
(including when the variable is unset). alpha/beta/preminor stay
false. The PR review gate still allows overriding the computed value
before merging.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U2Va4ryWuZbQvmUt2N85ec
* refactor: use github.ref_name instead of inputs.branch (EMB-1518)
github.ref_name is already set by GitHub's "Use workflow from" ref
picker on every workflow_dispatch run, so the custom inputs.branch
free-text field was duplicating a selection that was already made -
and could silently diverge from it. Switch every functional read of
inputs.branch to github.ref_name in both release workflows; keep the
branch input field itself, but make it explicitly non-functional
(required: false, no default, description points at the ref picker).
type: choice was considered and rejected again here: it would close
the typo risk but can't accept an unregistered branch, breaking
one-off custom-branch releases without editing the workflow file per
release - exactly the maintenance burden EMB-1518 removes.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U2Va4ryWuZbQvmUt2N85ec
* fix: remove dead readme-update mechanism (EMB-1518)
update_readme's two sed patterns match zero occurrences in the current
README (restructured by 15f7d3d20e, no version-pinned content remains),
so publish-npm's readme-follow-up-PR step has been a silent no-op on
every release, not just re-runs. Deleted the entire dead block (7 steps,
the env vars and pull-requests:write permission it alone used) and the
now-unused update_readme function.
* fix: drop minor release_type option (EMB-1518)
`semver -i minor` on a release branch jumps to the next major's
version line, colliding with that major's own branch. `patch` already
covers graduation and `preminor` already covers the legitimate
next-major jump (scoped to master). Drop `minor` outright rather than
add compensating validation - resolves review-cycle-3 finding #17 /
Judgment call #6.
* fix: add explicit reviewer request for embedding team on bump PR (EMB-1518)
CODEOWNERS is deliberately stripped on release branches (a repo-wide
practice to cut auto-backport review noise, unrelated to this
workflow), so the bump PR's "code owner review is automatic" comment
was false there - it got zero reviewers requested on release
branches. EMB-1000 documented the identical gap in the old system's
follow-up PR. Fix targets the metabase/embedding team (not a named
individual - the old system's hardcoded reviewer has since left).
* fix: restrict release types by branch, add custom releases (EMB-1518)
A one-off branch could publish as the stable release. Validation only
guarded master and release-x.*, so `patch` on e.g. sdk-data-apps stripped
the prerelease id (0.62.5-data-apps.0 -> 0.62.5), computed the
<major>-stable dist-tag, and - when the major matched vars.LATEST_VERSION
- set tagAsLatest, shipping a feature branch to npm as the stable release.
Only check-git-tag stood in the way, and only if that version had already
shipped.
Each branch class now gets only the release types that make sense for it:
master takes alpha/preminor, release branches take beta/patch, any other
branch takes custom. tagAsLatest is additionally gated on a release-x.*
branch, so nothing published from master or a one-off branch can take the
npm `latest` tag.
The custom type keeps one-off branches (esbuild, data-apps) releasable
under that restriction: prerelease_id is set on the first cut and becomes
both the version's prerelease id and the <major>-<id> dist-tag, matching
the shape already live on npm (56-esbuild, 63-data-apps). Later bumps
leave it empty and semver reuses the id from the committed version.
Also drops the dead release_utils.bash (zero functions, zero callers),
replaces both free-text dispatch inputs with single-option choices that
cannot be typed into, points the release workflow's form at the bump PR,
and removes the scaffold banner comments.
* fix: read vars.LATEST_MAJOR_VERSION, not LATEST_VERSION (EMB-1518)
The variable is LATEST_MAJOR_VERSION - it is what release.yml already
reads to decide Docker's `latest` tag, and what the release checklist has
a github admin bump each cycle. `vars.LATEST_VERSION` does not exist, so
tagAsLatest would have computed false on every bump, including at gold,
without any error - the fallback for an unset variable is false.
* refactor: make the release concurrency group readable (EMB-1518)
Same three cases as before, but each arm now names what it produces
(sdk-release-<branch> for a dispatch or a merged bump PR, sdk-release-noop-<run id>
for a closed PR this workflow ignores), and the comment above maps them in
the order the expression tries them.
The noop arm is not decoration: concurrency is evaluated before any job's
`if:` runs, so a closed PR that prepare-release skips would still enter the
release group without it. cancel-in-progress stays false so a release is
never interrupted mid-publish.
Groups are prefixed rather than bare because concurrency groups are
repo-wide, not per-workflow.
* refactor: build the release scripts once, in their own job (EMB-1518)
The Slack jobs each got at release/dist/index.cjs a different way:
send-slack-notification-start checked out and built it (then uploaded it as
an artifact), send-slack-notification-failure built its own second copy, and
send-slack-notification-successful downloaded the artifact. The duplicate
build existed because the failure job has to fire even when the metadata read
fails - in which case the start job never ran, so its artifact never existed.
A build-release-scripts job now builds it once and uploads it, and all three
Slack jobs download it. The job hangs off prepare-release only, so the
artifact is there no matter how early the release fails - which removes the
reason the second build existed.
* fix: quote the Slack messageId in the success notification (EMB-1518)
A Slack ts is a string. Interpolated bare it becomes a JS number literal and
drops trailing zeros - 1783602196.214000 renders as 1783602196.214, and the
reply targets a thread that does not exist. Pre-existing on master; the
failure job already quoted it, so this makes the two consistent.
* refactor: resolve the release branch in one expression (EMB-1518)
The env: BRANCH indirection was injection hardening from when the value was
the free-text `inputs.branch`. It is github.ref_name now - a real branch
name, same as github.event.pull_request.base.ref, which the next line already
inlined - so it guarded nothing and made the step asymmetric.
base.ref is empty on a dispatch, so `base.ref || ref_name` picks the right one
without branching on github.event_name.
* refactor: drop the duplicate-bump-PR pre-flight (EMB-1518)
It guarded a case the design already handles safely. Two open bump PRs
against the same branch both compute from the branch's committed version, so
both propose the same next version: the first to merge publishes and tags it,
and the second dies at check-git-tag with "Tag ... already exists" (npm would
reject the republish too). Nothing is published twice, and the error says
exactly what happened.
Getting there needs someone to dispatch the bump twice and leave the first PR
open, with both PRs sitting labelled in the list. That is not worth an API
call and 16 lines on every bump.
* chore: trim the bump step's comments (EMB-1518)
* refactor: drop the semver validity pre-check (EMB-1518)
Every supported release branch (57 through 63) already carries a valid
semver, and the new process never writes a `0.N.x` placeholder - gold is a
patch bump (0.63.0-beta.3 -> 0.63.0). The placeholder only ever existed
transiently under the old checklist, which rewrote it back to a concrete
version after each release.
Also saves an npx invocation on every bump.
* chore: spell out PRERELEASE_ID in the bump workflow (EMB-1518)
* refactor: spell out the semver call in each release type (EMB-1518)
alpha and beta shared an arm that passed --preid="$RELEASE_TYPE", and patch
passed -i "$RELEASE_TYPE" - you had to know the variable's value to know what
command ran. Each arm now names its own call.
The release_type description now leads with the phase (before gold, at gold,
after gold), which is what the person dispatching actually knows. It also
corrects preminor: master moves to the next major *after gold*, not at branch
cut - the 62 cycle did it in the after-gold checklist (#75487, 0.62.0-alpha ->
0.63.0-alpha), and 61 before it (#74279).
* docs: stop calling the bump PR a review gate (EMB-1518)
Nothing blocks on approval: the bump PR is not auto-merged but it is not
gated on a reviewer either, and workflow_dispatch can publish with no PR at
all. The computed distTag/tagAsLatest are defaults made visible and editable
- whoever dispatched the bump has already decided.
* docs: note EMB-2095 (auto-merge the bump PR) in the workflow (EMB-1518)
* docs: tighten the tagAsLatest comment (EMB-1518)
* refactor: collapse the tagAsLatest case into one condition (EMB-1518)
* docs: say what the sdkRelease write does (EMB-1518)
* refactor: drop the label-creation step (EMB-1518)
The sdk-release-trigger label is repo-level, so it only ever needed creating
once - it now exists. Re-running `gh label create ... || true` on every bump
swallowed every error (including auth) to no purpose.
* fix: fail on a leftover bump branch instead of side-stepping it (EMB-1518)
The run-id suffix made every retry a fresh branch, so an abandoned bump left
a branch on origin forever and the name was unpredictable. The branch is now
named after the version, and an existing one is an error: an earlier bump to
the same version was never merged, which is worth knowing before opening a
second PR alongside it.
* refactor: drop the no-backport label step (EMB-1518)
Backporting is opt-in - auto-backport.yml only fires on a `backport` label -
and backport-reminder.yml only demands a decision label when a PR touches
docs/**. The bump PR touches one JSON file, so no-backport guarded nothing.
Keeps the reviewer rationale, moved next to the --reviewer flag it explains.
* feat: show the target branch and computed values in the bump PR body (EMB-1518)
The PR body did not say which branch it targeted, which is the first thing
you want to know when a release PR shows up. Branch, release type, version,
dist-tag and tagAsLatest are now a table at the top.
* fix: pin semver and validate prerelease_id (EMB-1518)
npx --yes semver resolved whatever the registry served at dispatch time, on
a job that installs no dependencies - so release-critical version math could
change under us while package.json already pins ^7.6.3. Pinned to semver@7.
prerelease_id was passed to semver unchecked. An id like `DataApps` bumped
fine and then died three steps later in the dist-tag read-back, whose regex
only matches [a-z][a-z0-9-]*, with an error blaming the computed version
instead of the input. It is now rejected up front, with a message naming the
rule.
* fix: take semver's version from package.json (EMB-1518)
Pinning the workflow to semver@7 fixed the drift but created a second
version to keep in step. Read the range the repo already pins instead
(devDependencies.semver, currently ^7.6.3) and hand it to npx, so bumping
semver in package.json is the only place it happens.
* refactor: rename bump workflow input to read_first (EMB-1518)
* feat: reject prerelease_id on non-custom release types (EMB-1518)
* refactor: drop the preminor bump-only warning from the PR body (EMB-1518)
* refactor: drop the duplicate rerun wording in tag-exists error (EMB-1518)
* fix: reject reserved alpha/beta as a custom prerelease_id (EMB-1518)
* refactor: tidy and clarify comments in the SDK bump workflow (EMB-1518)
* fix: satisfy actionlint in the SDK release workflow (EMB-1518)
* feat: extract embedding SDK release logic into a tested helper (EMB-1518)
* refactor: remove dead embedding SDK helpers from version-helpers (EMB-1518)
* fix: drop trailing blank line in the SDK bump workflow (EMB-1518)
* refactor: run the SDK version bump from a tested helper, not inline bash (EMB-1518)
* docs: note the env block guards github-script against injection (EMB-1518)
* refactor: drop the unused LATEST_MAJOR_VERSION fallback in the bump script (EMB-1518)
* feat: show the target branch in the bump PR title (EMB-1518)
* docs: drop the manual-edit invitation from the bump PR body (EMB-1518)
* refactor: tidy the SDK release helper and its tests (EMB-1518)
* refactor: read the SDK release metadata via the tested helper (EMB-1518)
* refactor: env-route the branch name in the bump PR title (EMB-1518)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0156DAaukfiU6xceXd3brbZN
* refactor: simplify SDK release concurrency group (EMB-1518)
Collapse the three-case concurrency expression to one per-branch group, and
switch every multi-item needs array to a block sequence.
The dropped noop group guarded a compound case: GitHub keeps one pending run
per group and cancels any earlier one, so a closed PR touching the template
could evict a release waiting to start. That needs two same-branch releases
in flight at once, so we take the simpler expression. The clean fix is
queue: max, which actionlint cannot parse yet.
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* Deflake transform revision history spec
The "view and revert transform revisions" test changed the transform
name twice, each firing a "Transform name updated" success toast that
lives for the default 5s undo timeout. Unlike every other update flow in
this spec, these toasts were never dismissed. On a fast runner the test
reached the failed-revert error toast within that window, so a lingering
success toast coexisted with it and H.undoToast() (a single-element
findByTestId on [data-testid="toast-undo"]) failed with "Found multiple
elements".
Dismiss each success toast right after it appears, matching the existing
convention used elsewhere in this file, so exactly one toast is present
when the error toast is asserted.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Nemanja <31325167+nemanjaglumac@users.noreply.github.com>
* Add semantic search health checks
* Split HNSW recovery from semantic health
* Use HoneySQL literals for repair queries
* Use dynamic redefs in semantic search test
* Auto-scroll to a dashcard whenever it becomes the scrollTo target, and keep the scroll until the grid settles
* Drop a stray brace from the auto-scroll spec's first visit URL
* Simplify the auto-scroll correction to a fixed frame window
* Delete the react-router v3 engine
* Sever the react-router package dependency
* Give the SDK data-app router its own browser history on v7
* Rename the react-router-v7 alias to react-router
* Rename getDataAppHistory to getRawBrowserHistory, drop dead getCurrentHistory
Scope existence preconditions to their table; lint against name-only checks
A foreignKeyConstraintExists precondition with only foreignKeyName (or
indexExists with only indexName) forces Liquibase to snapshot the entire
schema to answer the check -- ~2s per precondition on a large postgres
appdb. 27 such preconditions (24 FK, 3 index; 21 in one v59 workspace
block) added ~46s to every full migration: 50.2s of total changeset time,
of which 24 changesets cost 45.9s and the other 1,182 cost 3.7s.
Add the scoping key to all 27. Measured on a fresh postgres migration:
total changeset time drops 50.2s -> 5.1s, with zero changesets over 1s.
Editing released changesets' preconditions is safe: Liquibase checksums
cover the changes, not preConditions. Verified two ways -- ChangeSet
.generateCheckSum is byte-identical before/after for all 27 (v9
checksums), and a clone of an already-migrated appdb re-validates
cleanly against the edited files.
lint-migrations-file: generalize require-primary-key-exists-has-table-name
into require-table-scoped-existence-preconditions, covering
primaryKeyExists/indexExists/uniqueConstraintExists (tableName) and
foreignKeyConstraintExists (foreignKeyTableName), walking the full
precondition tree in both its flat-map and list forms.
Claude-Session: https://claude.ai/code/session_01G7FfdpzW5pr7CX9aNwa98f
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Fall back to .yaml if .yml not found
Update docs to mention `.yaml` support
* Fall back to .yaml if .yml not found
Update docs to mention `.yaml` support
* Update file.clj
* Fix tests
* Address feedback
* Drop field_usage table as it is no longer used.
* Stop saving preprocessed_query in metadata since it's not used any more.
This used to be used for saving off field usage, but that hasn't been
a thing for a very long time.
* Make model-illustration.svg work well in both light and dark mode
* whitelabel svg
---------
Co-authored-by: Nick Fitzpatrick <nickfitz.582@gmail.com>
* Migrate PinMap to a typed function component and extract a pure getPoints
* Type the pin metric as nullable, derive MapSettings from VisualizationSettings, and add createMockVisualizationProps
* Trim comments, replace the map ref cast with an instanceof guard, and cover instance clearing
* Don't block dashboard saves on pre-existing foreign internal cards
UXW-4870: the guard added in #77236 (reject assigning a dashcard's card
to a question internal to another dashboard) checked updated dashcards
unconditionally, so any dashboard with a pre-existing foreign internal
card association became unsavable — every edit, even a resize, returned
400 "Invalid Request.".
Scope the guard to newly-introduced card ids: card ids the dashboard
already references (as a dashcard's card or in a series) are
grandfathered, so existing associations never block saving — including
moving such a card to a different dashcard in the same save. Newly
assigning a question internal to another dashboard is still rejected.
* Clarify arg names in internal-dashcard guard
Rename the guard's params to existing-dashboard/new-dashcards to make
clear the grandfather set comes from stored state while the checked
dashcards are the incoming save.
* Guest embed filter dropdowns should use the router database (UXW-4881)
PR #68952 made guest-embed query execution bypass DB routing by wrapping it
in with-database-routing-off, but the parameter-values endpoints (dropdown
values, search, remapping) didn't get the same treatment: chain-filter runs
through the QP, where attach-destination-db-middleware throws "Anonymous
users cannot access a database with routing enabled."
- Wrap the query-running sections of the four param helpers in
metabase.embedding-rest.api.common in with-database-routing-off, fixing
all six /embed/{card,dashboard}/:token/params/* endpoints plus their
preview_embed equivalents.
- Wrap the guest-embed tiles endpoints, which have the same anonymous-user
bug.
- Wrap preview_embed's query-execution endpoints (query, dashcard, pivot,
tiles) so the admin's embed preview shows exactly what the published
embed will show (router DB data) instead of routing via the previewing
admin's user attribute.
- Replace the manual *current-user-permissions-set*/*is-superuser?*
bindings in the param helpers with the equivalent request/as-admin.
Public sharing has the same gap but whether routed DBs should work over
public links at all is an open product question, so it is intentionally
left out.
* Consolidate embed routing-off wraps into the shared execution helpers
Review feedback: rather than a macro that each endpoint must remember to
use, move with-database-routing-off inside the shared execution helpers in
metabase.embedding-rest.api.common (process-query-for-card-with-params,
process-query-for-dashcard, process-tiles-query-for-dashcard, and a new
process-tiles-query-for-card wrapper), which every /api/embed and
/api/preview_embed endpoint funnels through. Endpoints built on these
helpers now bypass routing automatically, so a future endpoint can't
reintroduce this class of bug.
Removes all per-endpoint wraps from embed.clj (including the two original
ones from #68952) and preview_embed.clj; the invariant now lives in one
file with one explanatory comment.
* Fix tooltip scrollbar flicker in TreeTable (GDGT-2822)
Hovering a cell tooltip in a virtualized TreeTable while scrolling made a
page-level vertical scrollbar flash on and off. The tooltip is portaled to
<body> and positioned with real top/left insets, and with the document root
sized to the exact viewport height, a tooltip positioned past the fold grew
the document and flashed a scrollbar.
Portal cell tooltips into the table's own scroll container instead of <body>,
so a tooltip positioned past the fold is clipped by the (already-scrolling)
table rather than stretching the page. This is wired through a small
TooltipPortalTargetProvider context so the metabase/ui Tooltip wrapper picks
up the target automatically — no per-tooltip wiring, and any TreeTable with
cell tooltips benefits (e.g. the notifications admin table).
* Address PR comment
* Fix Tooltip wrapper not forwarding ref
* Add monitor dir in metabase-enterprise
* GDGT-2676: Migrate Dependency Diagnostics under Monitor space
* GDGT-2677: Migrate Admin -> Tools under Monitor space
* GDGT-2676: Fix paths
* GDGT-2676: Create separate high-level directories for migrated features
* GDGT-2657, GDGT-2686: Monitor space + migrate Dependency Diagnostics (#76527)
* GDGT-2657: Introduce Monitor space layout and routing
* GDGT-2686: Migrate Dependency Diagnostics to Monitor
* Monitor: Add Monitor plugin. Cleanup + fixes
* Monitor: Update graphy ownership path for Dependency Diagnostics
* Monitor: Cleanup PR
* GDGT-2697: Update hotkeys hints for Toggle navbar
* GDGT-2684: Address Copilot review comments
* GDGT-2684: Address PR comments
* GDGT-2684: Rename SpaceLayout to AreaLayout. Split to separate files
* GDGT-2684: Refactor route guards + add tests
* GDGT-2684: Move Admin Tools pages to the Monitor space (#76606)
* GDGT-2684: Move Admin Tools to Monitor space
* Monitor: Extract diagnostics sidebar into shared Monitor sidebar outlet
Add a useMonitorSidebar context to MonitorContent so Monitor pages can
render sidebars in a dedicated right-hand region outside the padded,
scrollable main content area. Rewire Dependency Diagnostics to render its
detail sidebar through this outlet, and promote the resizable shell
(SidebarResizableBox) into shared monitor/components.
* Monitor: Add MonitorHeaderTitle component for consistent page headers
Introduce a shared MonitorHeaderTitle matching the small secondary text
style used by Data Studio breadcrumbs, and apply it across Monitor pages:
Dependency Diagnostics header, Jobs, Logs, Model cache log, Task list/runs,
Task and task-run details, Erroring questions, and Alerts management.
* GDGT-2684: Cleanup
* GDGT-2684: Update monitoring tools access permissions
* GDGT-2684: Add handling for not-found routes
* GDGT-2684: Fix not-found routes handling in Data Studio
* GDGT-2684: Extend route tests. Minor styling refactoring
* GDGT-2684: Refactor Alerts Management sidebar
* GDGT-2684: Move old SidebarResizableBox to ModelSibar
* GDGT-2684: Cleanup
* GDGT-2684: Refactor Sidebar solution to use Portal
* GDGT-2684: Fix Monitor Tools E2E specs (#76886)
* Format monitor router files after the master merge
* Import Route from metabase/router facade in AreaTab spec
Fixes the no-restricted-imports lint error banning direct react-router imports.
* GDGT-2713: Update Tasks page styling (#77088)
* GDGT-2713: Update Tasks page styling
Restyle the Monitor Tasks and Runs pages to match Dependency diagnostics:
- Replace the SettingsSection card + default tabs shell with the Monitor
frame (fixed-height page, pills tabs; only the table scrolls)
- Rebuild TasksTable and TaskRunsTable on the TreeTable data-grid
(sticky header, virtualized rows, skeleton loading, a11y labels)
- Move filters into a Group bar and pagination below the table with a
total results count
- Keep the 2-state sort contract (always sorted, desc/asc) on top of
TanStack's 3-state cycle; asc-first on all sortable columns
- Add MonitorHeaderTabs (Monitor-owned pills tabs, no data-studio
dependency) and extract the shared LinkTab router-link tab to
metabase/common/components
* GDGT-2713: Fix e2e test
* Migrate LinkTab to the metabase/router facade
* Remove unused Link import from PaneHeader
The Tasks styling switched the tabs to LinkTab, leaving Link unused.
* Address PR review comments and fix entity_name typing
Use strict undefined comparisons instead of loose null checks where
values can only be undefined (optional props, RTK Query error,
destructured SortingState), and strict null comparisons where the
value can only be null (Task.db_id).
Also fix TaskRun/RunEntity entity_name typing: the backend always
assocs this key (nil when the entity lookup misses), so it's
string | null on the wire, not optional.
* GDGT-2712: Update Jobs page styling (#77100)
* GDGT-2712: Update Jobs page styling
Restyle the Monitor Jobs page to match Dependency diagnostics:
- Rebuild the page on the Monitor frame (fixed-height page; only the
jobs table scrolls, with a sticky header)
- Render the scheduler info in a Code block instead of a bare pre
inside a settings card
- Rebuild the jobs table on the TreeTable data-grid (skeleton loading,
empty state, a11y label, keyboard row activation); row click still
opens the job triggers modal
- Move JobInfoApp into a folder and add unit coverage; add
TaskInfo/Job API mocks and a task-info server mock
* Make the Monitor sidebar outlet resizable
Move the resizing behavior from the Dependency diagnostics sidebar into
the shared Monitor Sidebar so every Monitor view gets it without
duplication:
- Relocate SidebarResizableBox from metabase-enterprise/monitor into
MonitorLayout and wrap the Sidebar portal content with it
- Sidebar API: optional resizable (default true) with containerWidth
required while resizable, optional defaultWidth (512 by default);
consumers measure their content area with useElementSize and pass it
- Protect the Monitor content area from text selection while dragging
inside Sidebar instead of per-view
- Simplify DiagnosticsSidebar/DependencyDiagnostics accordingly and
drop the now-empty metabase-enterprise/monitor/components tree
- Make the Alerts management detail sidebar resizable (fills the
resizable box, keeps its 560px default width)
* GDGT-2712: Jobs page styling fixes
* Replace the job triggers modal with a Monitor sidebar
Clicking a job on the Jobs page now opens the job's triggers in the
resizable Monitor sidebar outlet instead of a modal:
- New JobTriggersSidebar renders each trigger as a vertical key/value
block (all fields the modal table had), with a divider between
triggers and a close action that returns to /monitor/jobs
- The /monitor/jobs/:jobKey route stays, so deep links and back/forward
keep working; JobInfoApp reads the jobKey param and renders the
sidebar, measuring its content area for the resize constraint
- Delete JobTriggersModal; add a Trigger API mock
- Give the scheduler info Code block the old card padding and keep it
from shrinking
* Apply cross-review fixes for the jobs sidebar work
- Render trigger details as a dl/dt/dd list for proper label/value
semantics (no visual change)
- Cover the shared Sidebar resize side effect (user-select class on the
Monitor container across drag start/stop/unmount)
- Cover keyboard row activation, all trigger attributes with null
placeholders and the may-fire-again 'No' branch, and the no-triggers
empty state
* GDGT-2712: Cleanup
* GDGT-2712: Add distinct error message when job is not found
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Migrate JobInfoApp to the metabase/router facade
* Disable text selection on body while resizing the sidebar
Address review comment: dragging the resize handle otherwise causes
the browser to continuously select/deselect page text, since the
sidebar is portaled and doesn't share a DOM ancestor with the main
content to scope a CSS class to.
---------
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* GDGT-2711: Update Logs page styling (#77160)
* GDGT-2711: Extract LogsViewer to common Monitor components
* GDGT-2711: Update Logs page styling
* GDGT-2711: Cleanup
* GDGT-2711: Address PR comments
* GDGT-2711: Address PR review comments
Rename LogsViewer's process prop to processUUID, drop the unused
logs-content id, and remove a few comments flagged as redundant.
* GDGT-2661: Update Erroring questions page (styling + refactoring) (#77167)
* GDGT-2661: Restyle Erroring questions page
Rewrite the EE /monitor/errors page off the legacy audit_app viz onto the
Monitor recipe used by the other Monitor tools pages, keeping the same
backend audit query:
- TreeTable with the 11 audit columns, 2-state sortable headers, and native
row selection; "Rerun Selected" moved to the standard bottom BulkActionBar
- Dependency-diagnostics-style debounced filter inputs
- Page no longer scrolls; the table body is the sole scroll region
- Pagination outside the table with a total count (new bad-table-count audit
query), hidden until the data has loaded, and synced to a ?page URL param
- Requests abort the previous in-flight one on paging/sorting/filtering
Replaces the AuditParameters/AuditTable/ErrorMode stack; deletes the old
ErrorOverview and its click-mode.
* Remove dead legacy audit-table v1 code
The Erroring questions page was the last consumer of the Audit v1 table
stack. With it rebuilt on TreeTable, these are unreferenced and removable:
AuditTable, QuestionLoadAndDisplay, AuditTableVisualization (the last
"audit-table" display), AuditParameters, the bad_table card def, and
AuditMode. The rest of audit_app (unsubscribe UI, InsightsLink,
metabot-analytics) is untouched.
* Fix duplicate/incorrect cards in the erroring-questions audit query
The latest_qe CTE joined query_execution to the per-card MAX(started_at)
subquery on started_at alone, so executions across cards that shared a
timestamp fanned out — surfacing duplicate rows (and cards whose latest run
didn't actually error). Join on card_id as well. Regression test covers a
shared-timestamp collision.
* GDGT-2661: Address review feedback
- Surface count-query failures in the page error state so a failing count
can't silently hide pagination
- Filter bar reflows (wrap) instead of clipping at narrow widths / zoom
- Skeleton renders the checkbox column to avoid a layout shift on load
- Centralize the internal-dataset-query cast to a single boundary
- Add tests: count db/collection filter clauses (backend), 202-body
data.error path, real filter-reset assertion, rerun failure recovery
* GDGT-2661: Fold pagination total into the rows query via COUNT(*) OVER()
The page ran two /api/dataset requests per view — the bad-table rows and a
separate bad-table-count — each recomputing the expensive latest_qe scan over
all query executions. Add a COUNT(*) OVER () total_count column to bad-table so
the full filtered total rides on the rows response, and drop the count query.
Halves the round-trips and eliminates the second latest_qe scan (~2.1s -> ~1.4s
of DB work on the Stats dump); the window count adds no measurable overhead
since it runs over the already-materialized result set before LIMIT.
* Migrate ErrorOverview to the metabase/router facade
* GDGT-2661: Return erroring-questions total_count at the response root
The bad-table query computes COUNT(*) OVER () as a column, which repeated the
total on every row. Strip that column via an xform and surface the value once
as a top-level total_count on the response; the FE reads it from there.
* GDGT-2661: Non-blocking per-row rerun with loader in erroring questions
Rerun now runs in the background per card instead of blocking on the whole
batch: clicking Rerun Selected closes the bulk bar immediately and fires each
request independently, so more questions can be queued while slow ones run.
A rerunning row shows a loader in place of its checkbox, is excluded from
Select all, and its loader clears when the request settles; the list refetches
once the last rerun finishes.
TreeTable gains a generic isRowLoading prop and makes the whole checkbox cell
a click target so a near-miss no longer navigates to the row.
Also hoist total_count onto the Dataset type and drop the local helper.
* Extract shared generic sorting helpers for monitor tables
The monitor tables (erroring questions, tasks, dependency diagnostics) each
duplicated getSortingState/getSorting/isSortColumn. Extract them into a generic
metabase/utils/sorting module keyed by the column type. getSorting is overloaded
so the un-sort behaviour is type-driven: pass the current sort for always-sorted
tables (direction flips) or omit it to allow an unsorted result.
Refactor the three monitor tables onto the shared helpers.
* Update erroring questions E2E selectors for the new table
Point the tests at the TreeTable-based ErrorOverview: erroring-questions-table
container, erroring-question rows, and the always-enabled filters / bulk bar
that closes on rerun instead of a disabled button.
* Bind *driver* when reducing audit internal queries
The MySQL timestamp reader calls results-timezone-id, whose driver arg is
malli-validated as a keyword in tests. reduce-results* bound the timezone
override but not *driver*, so it was nil and audit queries returning timestamps
errored on MySQL/MariaDB. Bind *driver* to the app-db driver alongside the
override.
* GDGT-2661: Address PR comments
* GDGT-2661: Merge erroring questions input into one, extend to support name search
* GDGT-2661: Fix checkbox cell width in rows
* GDGT-2661: Fix e2e test
* GDGT-2661: Address PR comments
* GDGT-2710: Update Model cache log page styling (#77277)
* GDGT-2713: Update Model cache log page styling
Restyle to match Dependency diagnostics: the page no longer scrolls
vertically, only the table does. Replace the hand-rolled AdminS.ContentTable
with a metabase/ui TreeTable (skeleton while loading, empty state, refresh
action), and drop the emotion-styled ModelCacheRefreshJobs.styled.tsx in
favor of a CSS module.
Model and Collection columns are the two stretching columns (fill remaining
width) and support client-side sorting, since the underlying /api/persist
endpoint has no sort param.
* Unify monitor tables on 3-state sortable headers
Erroring questions and Tasks forced a 2-state flip-forever cycle by passing
the current sort back into getSorting(); drop that so a 3rd click clears the
header's sort arrow and falls back to the page's default sort, matching
Dependency diagnostics and Notifications.
The default-sorted column on each page needs sortDescFirst matching its
default direction (desc), otherwise the very first click on that column is a
no-op: TanStack's per-column cycle treats desc as the removable/last state
when sortDescFirst is false, so a column that starts at desc goes straight to
removed on click one.
* Make all applicable columns sortable in Model cache log and Jobs
Both are fully-loaded, unpaginated lists, so enabling client-side sorting on
the remaining columns is safe. Model cache log's action column stays
unsortable (no header text or data). Tasks and Task runs are left as-is:
their backends only support a fixed set of sort columns (or none), and
sorting the rest client-side would silently reorder only the current page.
* GDGT-2710: Address PR comments
* GDGT-2709: Update Alerts management page styling (#77364)
* GDGT-2709: Update Alerts management page styling
Restyle to match Dependency diagnostics: the page no longer scrolls
vertically, only the table does. Move pagination outside the table into a
plain footer and show the total result count via PaginationControls'
showTotal.
* Restrict Alerts management to admins only
The /api/notification/admin* endpoints are superuser-gated on the backend,
but the frontend route and nav link were reachable by anyone with the
broader can_access_monitoring permission (shared with the other Monitor
Tools pages), so a non-admin holding it could reach the page while every
API call inside it 403'd. Add a dedicated canAccessAlertsManagement
selector and route guard so the route/nav link match the backend's
admin-only restriction.
* Fix misleading test name in MonitorLayout spec
The test asserts the link is hidden for a non-admin analyst, not that it's
shown for an admin.
* GDGT-2794: Update Task/Run details page styling
Extract the "Back to Tasks"/"Back to Runs" breadcrumb into a shared
MonitorBackLink component, styled and positioned like other Monitor
page headers. Wrap both detail pages in the same non-scrolling shell
used elsewhere in Monitor so only the content area scrolls internally,
instead of the whole page scrolling behind the floating AppSwitcher.
* Extract MonitorPageContent to replace SettingsSection in Monitor
SettingsSection carries admin-page title/description logic that Monitor
pages don't use. MonitorPageContent keeps the same card styling with none
of that logic, dropping the last two /monitor call sites still importing
from metabase/admin.
* GDGT-2709: Fix layout in Tasks Details page
* GDGT-2709: Refactor NotificationsTable row selection logic. Fix selection with spacebar
* GDGT-2791: Abort superseded in-flight list requests in Monitor (#77694)
* GDGT-2791: Abort superseded in-flight list requests in Monitor
Changing page/sort/filter on a Monitor list page left the previous, often
expensive, list request running to completion on the server. Subscribed RTK
useXQuery hooks only abort when their last subscriber unmounts, so introduce
a shared useAbortableQuery hook built on lazy triggers (whose promise exposes
.abort()) that cancels the superseded request on arg change, unmount, and
skip transitions, and suppresses the abort error so cancellations never flash
the error view.
Caching mirrors useQuery: cache-first by default, refetchOnMountOrArgChange
to opt into always-fetch; manual refetch() always hits the network. Every
converted call site keeps its pre-migration caching semantics.
Converted: TaskListPage, TaskRunsPage, TaskRunEntityPicker,
ModelCacheRefreshJobs, NotificationsAdminPage (main list),
DependencyDiagnostics, BrokenDependentsSection, and ErrorOverview (its
one-line useAbortableAdhocQuery wrapper is inlined and hooks.ts deleted).
* GDGT-2791: Cleanup
* GDGT-2791: Fix type narrowing in TaskRunEntityPicker
* GDGT-2791: Clamp pagination range to total
* GDGT-2791: Test that tag invalidation does not refetch superseded args
* GDGT-2791: Scope query aborts per hook instance via injected cache key
* GDGT-2791: Migrate _refetchDeps to the shared __rtkCacheKey mechanism
* GDGT-2791: Refine baseQuery typings. Cleanup
* GDGT-2791: Replace interface in params declarations with type. Refine BaseQuery params type
* GDGT-2832: Group Monitor navbar items into sub-menus (#77706)
* GDGT-2832: Move AreaContent to AreaLayout and normalize Monitor page spacing
* GDGT-2832: Group Monitor navbar items into sub-menus
* GDGT-2832: Cleanup. Fix AreaTab outline overflow
* GDGT-2832: Match Alerts Management tab spacing to Dependency Diagnostics
* GDGT-2832: Replace collapsible navbar groups with static labelled sections
* GDGT-2832: Update Monitor views headings to match navbar. Fix e2e test
* Monitor section design tweaks (#78007)
* icon and spacing updates
* remove info icon from background tasks page heading
* improve empty states
* GDGT-2810: Enable sorting in all Monitor views (#77842)
* GDGT-2810: Enable sorting on all Background tasks and Task runs columns
* GDGT-2810: Remove bold styling from first Monitor table columns
* GDGT-2810: Fix reflection lint warnings in task history API test
* GDGT-2810: Qualify task_run/task_history filter columns
The entity_name and task_count sorts add joins (report_card and
report_dashboard both have their own entity_id column), so unqualified
filter columns become ambiguous: /api/task/runs?entity-id=N&sort_column=entity_name
returned a 500. Qualify the filter columns and add a regression test
combining filters with the derived-column sorts.
* GDGT-2810: Cleanup
* GDGT-2810: Use kebab-case sort params on the task runs endpoint
* GDGT-2810: Extract shared useSortingStateChange hook for TreeTable sorting
* GDGT-2810: Extract shared page/sort URL param parsers
* GDGT-2810: Fix e2e tests
---------
Co-authored-by: Ngoc Khuat <qn.khuat@gmail.com>
* GDGT-2843: Integrate Monitor upsell pages (#77948)
* GDGT-2843: Migrate Monitor upsell pages into monitor scope, link to /upgrade/monitor
* GDGT-2843: Tighten monitor module boundaries to standard shared tier
* GDGT-2843: Change Monitor app icon from gauge to pulse
* GDGT-2843: Update pulse icon and Monitor logo
* GDGT-2843: Address PR comments
* GDGT-2843: Fix e2e tests
* Fix Monitor code review findings (#78106)
* Fix Monitor code review findings
* Remove obsolete dependencies redirects
* Monitor review: Cleanup
* Address review comments
* Cleanup
* Remove Dependency diagnostics link from Data Studio nav
It already lives under Monitor (/monitor/dependency-diagnostics); the
Data Studio route just redirects there now.
---------
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
Co-authored-by: YB Tan <tybmeta@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Maz Ameli <maz@metabase.com>
Co-authored-by: Ngoc Khuat <qn.khuat@gmail.com>
* WIP: start flipping unit-test harness to v7
Default routerEngine to v7 and render bare (non-Route) children under a
catch-all route so v7's <Routes> renders them. role=link nav failures still
to diagnose. Suite not yet green.
* Give v7 tests a history handle and drop DOM-wrapped route config
The harness now owns the v7 memory history and hands specs a v3-shaped adapter
(getCurrentLocation/push/replace/goBack/listen), so the ~42 specs that drive and
assert navigation through the returned history keep working on v7.
Route config wrapped in a <div> is a v3 artifact that v7's <Routes> cannot read;
swap those two for fragments.
* Pin the v3-vs-v7 conformance suites to the v3 engine
They exist to prove the facade on v3 matches real v7 and are built from raw v3
route config, so they must not follow the harness default now that it is v7.
* Resolve a relative initial route in the v7 test history
* Buffer navigations dispatched before the v7 router registers
* Carry route props onto the v7 matched-route stubs
* Fix the v48 /browse/:dbId-:slug legacy redirect on v7
* Render facade routes in renderHookWithProviders on v7
* Migrate unit specs off v3-only routing behavior for the v7 flip
* Drop buffered v7 navigations when the router unmounts
* Default use-v7-router to true
* Re-home history@3 types onto the metabase/router facade
* Format the re-homed import blocks
* Merge duplicate router imports and drop unused lint directives
* Re-home the router-listen spec's history Location type
Wait for the DebouncedFrame to settle via ensureChartIsActive() before
clicking a group tile to drill in. While the frame transitions it carries
pointer-events: none, so the drill click was silently swallowed and the
breadcrumb button never rendered.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Migrate AuditTableVisualization to a typed function component
* Fix Erroring Questions column sorting by forwarding sorting props through Visualization
* Extract typed viz definition to drop casts per review
* GHY-4016: Replace Field YAMLs with FieldUserSettings in git sync (PR 1/3)
Git sync currently exports one Field YAML per database column. For a
typical customer this is ~6396 files — 84% of the entire repo. Most of
that data (active, database_type, position, dimensions, ...) is sync
metadata that git sync has no business tracking; the user-curated part
lives in FieldUserSettings.
This PR switches git sync to export FieldUserSettings instead of Fields.
Only fields with user-curated metadata get a YAML at all.
Measured against the stats-remote-sync repo (196 published tables,
~20 columns each):
Import: 43s → 14s (3x faster, 7659 → 1263 files)
Export: 6.3s → 2.0s (3x faster, 2155 → 239 entities)
Files: 7659 → 1263 (84% reduction)
For a large customer (563k files, 522k Fields), ingest time alone drops
from 116s to 7s.
Full Field serdes is unchanged — non-git-sync export/import is
unaffected. FieldUserSettings already had complete serdes support
(generate-path, storage-path, load-find-local, make-spec); this PR
wires it into the git sync paths.
Changes:
- `serdes/descendants "Table"`: branches on `:user-edits-only` opts.
When set, emits `["FieldUserSettings" field_id]` only for fields with
a FUS row; otherwise emits `["Field" field_id]` as before.
- `spec.clj`: named `git-sync-extract-opts` constant (with
`:user-edits-only true`); public `pk-col` helper for the non-standard
FieldUserSettings PK (`:field_id` not `:id`).
- `extract.clj` (OSS): local `pk-col` for `:where [:in pk-col ids]`
(module boundary prevents depending on EE spec).
- `impl.clj`: `extract-chunk` uses `spec/pk-col` for FieldUserSettings.
- `models.clj`: adds `"FieldUserSettings"` to `data-model-in-collection`.
- `events.clj`: removes `:model/Field` from `register-events-for-spec!`
so field-update events no longer create Field RSOs. A dedicated
`::field-update-event` handler creates FieldUserSettings RSOs only
when a FUS row exists. This completes the incremental path: full
export uses `descendants` + `:user-edits-only`; incremental export
uses FUS RSO rows.
Tests assert on the full RSO table so stray Field RSOs fail immediately.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Address PR review: defensive get for dynamic pk-col, fix content-hash test for FieldUserSettings
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* don't need maps, k/v pairs work
* Declare :model/FieldUserSettings module boundary export/import
remote-sync references :model/FieldUserSettings in events.clj, so it must
be exported by warehouse-schema and imported by enterprise/remote-sync.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Remove unused f1-id binding in extract test (kondo)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Test that Field and FieldUserSettings are serdes descendants leaf nodes
Cements the design discussed in review: Table's descendants method is the
only source of field-level entries, and the :user-edits-only branch is
exclusive — you get Field refs or FieldUserSettings refs, never both via
a transitive Field -> FieldUserSettings edge.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* WIP: dag run BE
* split into new table
* run utils
* remove dag changes on job runs
* new dag run model
* abstract over parent-run
* separate dag model/api
* factor common job/dag code in a coordinated-run primitive
* fixes
* dag test
* fix migration
* kondo
* missing resolution
* exclude model
* kondo config
* fix test
* fix: exclude for deser
* fixes
* fix: exclude for deser
* fix: missing fk
* random ids
* chore: rename
* feat: add unioned runs
* remove dead endpoints
* api tests
* extend filter semantics
* trim docstrings
* kondo
* gap: allow canceling job runs
* share code
* fix: job start should return proper job id
* fix: mig to 64
* simplify
* simplify further
* simplify
* dead code
* add unique constraint on dag runs
* tighten schemas
* tighten permissions
* fix test
* relax schema
* fix: set null on delete
* handle deleted jobs/transforms for dag runs
* appease the sql linter
* fix test
* simplify rollbacks
* simplify callsite
* fix tests from bad rebase
* fix: fallback when transform is deleted
* add transform_count
* fix tests
* log with context
* varchar 32 for status
* Add UI for running transform with upstream or downstream (#77367)
* GDGT-2625 wip: reprocess DAG UI for transforms
- Split "Run now" button (upstream/downstream) + confirmation modal
- runDag / dag-transforms / cancel-dag API + analytics event
- Tabbed Runs page: transform-graph runs (default) + individual transform runs
- Graph runs table (Run/Type cols), filter bar (Type/Transform/Started/Status),
sidebar with member transforms; shapes aligned to bronsa's /api/transform/runs
- List + members still served from a mock (transform-graph-runs.mock) via queryFn
* GDGT-2625 wip: align graph runs to /api/transform/runs, sidebar + cancel
- Reshape TransformGraphRun to backend RunSummaryResponse (run_type/entity_id/name)
- Filters mirror Transform runs: Type/Transform/Started/Ended/Status/Trigger
- Sidebar: title = Run name, Type as first metadata row, cancel section below
metadata (dag/job/transform), confirm modal; job-run cancel endpoint wired
- Mock covers job/dag/transform in all states (succeeded/started/failed/timeout)
- Reorder columns: Status/Trigger after Duration
* GDGT-2625 wip: sidebar header entity links
Add "View this job" / "View this transform" (+ dependency graph for
transforms) action links to the graph-run sidebar header, mirroring the
individual transform-runs sidebar.
* GDGT-2625 wip: migrate graph runs to real /api/transform/runs
- Swap list + member queryFn mocks for real queries (list endpoint, dag/job
member drill-down; standalone runs synthesized). Delete the mock fixture.
- Rewrite page spec against real endpoints via fetch-mock + new mock factories.
- Reword DAG run confirm modal (drop "in this order", bullet list) since members
can run in parallel.
* Adjust wording
* GDGT-2625 wip: scheduled state + cleanup
- Show a "scheduled" run state on the transform page after a DAG reprocess is
triggered, so the queued transform gives feedback instead of looking idle
- Drop dead DAG-run types/tag-providers/mock helpers left from earlier iterations
* Address code-review findings for reprocess DAG UI
Correctness:
- Migrate transforms.cy.spec.ts to /runs/individual (the /runs route now
renders the graph-runs page, not the individual run table)
- Gate the "Run all" DAG confirm button on the mutation, preventing a
double-click from triggering two DAG runs
- Toast when run-dag returns a null dag_run_id instead of silently no-op
- Stop the "Scheduled" banner from polling forever when the member-runs
request errors
- Keep the graph-run sidebar open when the selected run scrolls off the
current page during polling
- Invalidate transform state on cancelDagRun/cancelJobRun so member
transforms' last_run refreshes after a cancel
Cleanup:
- Translate whole phrases (with context) for DAG run names
- Replace a typecast with a type guard in TypeFilterWidget
- Memoize parsed params and filter options on the graph-runs page
- Drop the unused direction field from the scheduled-DAG-run state
- Convert the sidebar's IIFE switch to ts-pattern match
* Handle deleted entities in graph runs; fix run-list cache invalidation
- Graph runs table: grey a deleted job/transform's run name (italic +
"has been deleted" tooltip), keeping the row clickable
- Sidebar: keep showing member transforms for deleted DAG and standalone
runs (fetched by dag-run id / synthesized from the run); hide the section
only for deleted job runs, whose members can't be fetched without the
(now-null) job id
- api/transform.ts: invalidate the run lists on deleteTransform (a
transform's runs survive its deletion) and cancelCurrentTransformRun, so
the Runs pages reflect the change instead of relying on polling
- Trim narrating comments from the transform API types
* Cleanup
* Fix transforms cancel e2e: click renamed "Cancel run" confirm button
The transform-page cancel confirmation button was renamed from "Yes" to
"Cancel run", but two cancelation tests still clicked "Yes". The cancel was
never confirmed, so the slow (pg_sleep) transform kept running and held a lock
on the writable DB, hanging the next test's resetWritableDb and failing the
suite. Point those two clicks at the new label (the individual-runs-page cancel
still uses "Yes" and is left unchanged).
* Avoid type cast in RunSection spec to satisfy no-unjustified-type-casts
Replace `body as string` with a typeof guard when reading the run-dag request
body, which the new lint rule flags.
* Show dependency count in the graph-run label
Use bronsa's new transform_count to label DAG runs as
"{n} dependencies › {name}" (upstream) / "{name} › {n} dependents"
(downstream), n = transform_count - 1, joined by a small chevron icon.
Bold the job/transform name; fall back to the plain name when the count is
absent (job/standalone runs, or DAG runs predating the column).
* Replace runs tabs with Detailed view switch; polish graph-run name rendering
- Swap the two runs tabs for a Detailed view switch that toggles grouped
(/runs) and detailed (/runs/individual) views in place; add RunsPage shell
and DetailedViewSwitch, remove RunTabs
- Render the sidebar header run name like the table (dimmed deps count +
chevron), with wider gap and aligned font size/weight
- Keep the Detailed view switch label on one line
- Bump graph-runs page size to 25 to match the individual runs page
- Fix stale react-router-redux import
* Preserve shared filters across Detailed-view toggle; drop deprecated CSS util
Toggling the Detailed-view switch previously called the run-list URL helpers
with no arguments, discarding the active filters. Forward the params common to
both views (statuses, transform ids, start/end time, run methods) via a new
CommonRunListParams/pickCommonRunListParams helper; page and sort are dropped
since the two views paginate differently and use incompatible sort columns.
Also replace the deprecated CS.overflowHidden global utility class in the new
graph-run table with an inline overflow style prop.
* Track Detailed-view toggle in transform runs analytics
Emit a transform_runs_view_toggled simple event from the DetailedViewSwitch
toggle handler with the resulting view mode (detailed/grouped), so both the
grouped and individual run-list call sites are covered.
* Fix reprocess run status: don't re-show Scheduled after a member completes
- Derive the DAG scheduled state from this transform's own member run:
a member run row only exists once the transform starts, so its presence
means the transform already ran this cycle. Prevents isScheduled from
re-arming (and masking the completed status) after this transform's run
finishes while sibling DAG members are still running
- Keep the graph-run sidebar polling while the top-level run is canceling,
mirroring isPollingNeeded
* Cleanup
* Address PR review nits from Uladzimir
- Extract isActiveRunStatus helper and use it for the duplicated
started/canceling checks (RunButton, both isPollingNeeded, sidebar
shouldPoll, RunSection dagFinished)
- DetailedViewSwitch: replace styles prop with a CSS module class
- Convert renderRunName/renderRunEntityName to RunName/RunEntityName
components
- Drop redundant String() around an already-string cell value
- Cover RunButton onRun click and the run-options menu show/hide
* Adapt runs routes to the react-router migration on the rebased base
The base migrated transforms routes to element=/withRouteProps with
Outlet-based layouts. Make RunsPage an Outlet layout and render the runs
list pages via element in the specs.
* wip
---------
Co-authored-by: Oleh K <oleh@metabase.com>
* Treat a v7 Link with an empty `to` as a button, not a navigation
* Make a v7 replace to the current URL a no-op so URL-sync effects settle
* Provide router.listen on the v7 engine and keep the router shim stable
* Serialize a v3 location query into the search string on v7
* Fix lint and type issues in the v7 router shim
* Keep the v7 matched-route branch stable across renders
* Republish the v7 splat param under v3's `splat` name
* Enable the react-router v7 engine by default
* Commit the v7 location update synchronously instead of in a transition
* Mirror the v7 location into redux from the history subscription
* Format the v7 router provider
* Sort query keys when serializing a v3 location query
* Match v3 query encoding and resolve bare link paths against the root
* Keep v3 query encoding and leave absolute link urls alone
* Always push when a v7 link targets the current url
* conversation list endpoint
* add rtkquery endpoints
* generate conversation title
* better title generation logic, poll for title if not recieved before end of first message
* render conversation title in ui and show conversation list
* allow resuming historical conversations
* show exploration conversation history
* refresh current conversation ui for 409 case
* surface conversation titles in usage analytics
* add /metabot/conversation/:id route
* remove pending title data part, assume from lack of title data part we need to poll, fix loading a conversation that is mid-response
* title backfill job
* rename title event
* FE prevent polling multiple times for conversation title
* improve title generation prompt
* ci fixes
* automated review feedback
* e2e test fixes
* reduce diff size
* self-review
* Address PR review feedback
- api: pass profile-id through valid-usage-profile-id when ensuring a
conversation title, matching the backfill job
- title backfill: gate readiness on check-usage-limits! (instance-level)
so the job pauses and reschedules cleanly when a usage limit is hit
- title backfill: bound the title future .get with a 60s timeout so a
stuck generation can't block the backfill loop indefinitely
* Add Metabot option to the Transforms New menu (GDGT-1773)
Adds a "Metabot" entry as the first option in the Transforms "New" menu.
Clicking it opens the data-studio Metabot sidebar with a pre-seeded
"Create a transform that " prompt (not submitted), gated on Metabot access.
Also fixes the tiptap prompt input so a programmatically seeded prompt
places the caret at the end instead of the start (autofocus "end" plus a
caret-to-end after external setContent), which also improves the retry path.
* Cleanup
* Cleanup unnecessary code
* Test CreateTransformMenu Metabot item without mocking
Render the real CreateTransformMenu + Metabot via the shared metabot test
setup so the menu item, permission gating, and prompt seeding are exercised
end to end instead of through stubbed hooks.
Bump babashka postgresql pod to 0.1.6 so mage runs on Apple Silicon
The 0.1.0 pod manifest has no macos-aarch64 entry -- only macos-amd64.
Babashka downloads that x86_64 artifact and caches it under the aarch64 path
anyway, so on an Apple Silicon machine without Rosetta every mage command dies
before doing any work:
Cannot run program ".../mac_os_x/aarch64/pod-babashka-postgresql":
Exec failed, error: 86 (Bad CPU type in executable)
Clearing the pod cache does not help -- the re-download is the same x86_64
binary. 0.1.5 is the first release with a real macos-aarch64 artifact; this
bumps to the current 0.1.6, which resolves to a native arm64 executable.
0.1.6 keeps every platform 0.1.0 shipped (macos-amd64, linux-amd64,
windows-amd64) and adds macos-aarch64 and linux-aarch64, so no one loses
coverage. `pg/execute!` is the only pod fn used anywhere in the repo, and its
signature is unchanged.
Verified on arm64: the cached binary is now `Mach-O 64-bit executable arm64`,
`mage.bot.setup-worktree` loads, and `./bin/mage fix-kondo-ratchets` runs to
completion.
Claude-Session: https://claude.ai/code/session_01X5dVcVX59aLonqqsYsqU65
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Restore navigation blocking on the v7 engine via a blocking history
* Render the v7 Link on the v7 engine so links navigate in modals
* Scope v7 leave hooks to their route so within-route navigation is allowed
* Don't let a v7 Link with no destination navigate on click
* Type the app Link's `to` as optional to match the runtime contract
* Apply activeClassName on the v7 engine via NavLink
* Hide settings tab in TagEditorSidebar if there is no tags
* Don't crash query metadata fetch on un-normalizable queries
batch-fetch-query-metadata filtered empty queries before normalization,
but normalize-query returns {} when a query can't be normalized (e.g. a
native query with an incomplete field-filter/table template tag). That {}
then failed batch-fetch-query-metadata*'s strict ::query schema, 400ing the
whole request and tripping the query editor error boundary. Filter empties
after normalization too so an unnormalizable query just yields no metadata.
* Hide template-tag 'Try it' examples in transforms editor
The template-tag help sidebar's 'Try it' buttons load an example query
against the Sample Database. Transforms can't target the Sample Database,
so clicking 'Try it' switched the transform to an invalid database and left
the editor in a broken state. Add a canUseSampleDatabase QueryEditor option
(default on) that transforms set to false, which withholds the sample
database id so TagEditorHelp hides the 'Try it' buttons.
* Add regression tests for transform variables sidebar fixes
- batch-fetch-query-metadata skips queries that normalize to {} instead of
400ing the whole request (the P1 crash)
- TagEditorHelp hides the 'Try it' examples when no sample database is available
- TagEditorSidebar hides the Settings/Help tabs until the query has a variable
* Address PR review feedback
- Simplify batch-fetch-query-metadata with (keep (comp not-empty normalize-query))
- Replace the hidden-tabs settings behavior with an empty state in the Settings
tab (no more layout shift / auto-switch to Help); use the shared EmptyState
- Make sampleDatabaseId a required, DatabaseId-typed setup option
- Add a NativeQuerySidebar test asserting 'Try it' is hidden for transforms
(canUseSampleDatabase=false) and shown otherwise
- Re-add the getEditorOptions canUseSampleDatabase assertion
* split use-metabase-query.unit.spec.tsx into multiple files
* split test fixtures from setup.tsx
* move useMetabaseQuery tests into tests folder
* rename useMetabaseQuery hook test
* remove unused query test fixture fields
* rename mock dataset query helper
* rename bundle exports
* chore: tiny formatting changes
* test: rewrite stale query results test
* test: extract public api tests to type and runtime tests
* Convert the Table visualization from a class to a function component
* Stabilize the Table question so async metadata updates don't rebuild columns mid-hover
* Type useSyncedQuestion via VisualizationProps so the SDK d.ts doesn't leak internal types
* Author route paths in v7 syntax, shim to v3 in the facade
* Add use-v7-router flag and frontend engine reader
* Add flag-gated react-router v7 engine (provider, tree mapper, bridge)
* Re-point redux navigation and location to the v7 router
* Support both router engines in renderWithProviders
* Switch the app mount to the v7 router behind the flag
* Point setRouteLeaveHook no-op comment at DEV-2374
* Revoke schema-level grants when destroying Redshift workspace isolation
destroy-workspace-isolation! discovered schemas to revoke via
svv_relation_privileges, which only surfaces relation-level grants: a granted
input schema with no tables (or whose tables were dropped after provisioning)
kept its schema-level USAGE grant, and DROP USER then failed with 'user ...
cannot be dropped because the user has a privilege on some object'. Union in
svv_schema_privileges so schema-only grants are revoked too.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Rename workspace+det to workspace in the new destroy test
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Run Redshift workspace teardown statements individually, not as a JDBC batch
A failure now surfaces as the plain server error instead of the
BatchUpdateException wrapper ('Batch entry N ... Call getNextException to see
other errors in the batch.'); the transaction still aborts as a whole on the
first error.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Revert "Run Redshift workspace teardown statements individually, not as a JDBC batch"
This reverts commit 4bf2d241b5.
* Unwrap BatchUpdateException in Redshift workspace teardown
Keep the batch (there can be many granted schemas, two revokes each) but
rethrow .getNextException so failures read as the plain server error instead
of 'Batch entry N ... Call getNextException to see other errors in the batch.'
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* remove comment
* Unwrap BatchUpdateException across all workspace-isolation batch sites
New driver.u/batch-exception util returns the per-statement exception hiding
under .getNextException (or the throwable unchanged). Applied to every
workspace init/grant/destroy executeBatch site (h2, mysql, postgres,
clickhouse, redshift) — init sites unwrap before scrub-exceptions — so
failures read as the plain server error instead of 'Batch entry N ... Call
getNextException to see other errors in the batch.'
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Keep the batch exception on init-workspace-isolation! failures
The init statements carry secrets and pgjdbc echoes the aborted statement in
the BatchUpdateException message — scrub-exceptions redacts it, and the
GHY-3660 test (init-workspace-isolation-never-exposes-password-test) relies on
that redacted CREATE USER echo as proof the run reached the statement.
Unwrapping to the inner server error discarded the echo. grant/destroy keep
the unwrap: their statements carry no secrets and the plain server error is
the better message there.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Unwrap the batch envelope on init-workspace-isolation! too
All workspace batch sites now rethrow the underlying per-statement exception;
init scrubs the unwrapped exception so passwords stay out of the message. The
GHY-3660 test now proves the flow reached CREATE USER by asserting the
server's permission error instead of the redacted batch echo (which no longer
exists).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Test batch-envelope removal for grant! and password scrubbing after unwrap
- driver-level: a failed grant-workspace-read-access! must not surface a
BatchUpdateException (h2 runs locally; pg/mysql/clickhouse in driver CI;
non-batching impls pass trivially)
- unit: scrub-exceptions composed with batch-exception redacts the password
in the unwrapped per-statement message
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Batch init/grant/destroy on every SQL workspace driver
Redshift grant, and all three Snowflake / SQL Server workspace methods, now
run their statements as JDBC batches like the other drivers — one round trip,
per-statement exceptions unwrapped via driver.u/batch-exception (init unwraps
before scrub-exceptions). Redshift grants commit before grant! returns, so a
subsequent describe-database from another connection still sees the
privileges; the destroy pre-loop for foreign-grantor default-priv rows stays
per-statement on purpose (each must survive the others failing).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fold Redshift foreign-grantor revokes into the destroy batch
The per-statement autocommit pre-loop is gone: one transactional batch now
covers the FOR USER default-priv revokes, relation/schema revokes, and the
drops. A stale discovery row fails the whole destroy, which is fine — the
workspace layer records the failure and the deprovision retry re-discovers
fresh state.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Escape workspace passwords in DDL like every other interpolated value
CREATE/ALTER USER statements interpolated the generated password raw into the
SQL string on postgres, redshift, h2, clickhouse, and snowflake (mysql and
sqlserver already escaped). Escape with sql.u/escape-sql :ansi (quote
doubling, valid on all five engines) and scrub both the raw and escaped forms
from failure messages.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Unwrap batch exceptions from all JDBC connectors; fix sqlserver batch var scope
The grant-envelope test caught two real issues in driver CI:
- MariaDB's connector chains the per-statement exception via .getCause and
mssql-jdbc chains nothing at all, so batch-exception now falls back to the
cause and, failing that, rebuilds a plain SQLException with the same
message/SQLState/error code.
- mssql-jdbc executes a Statement batch in one T-SQL scope, so the two
sqlserver destroy statements both DECLAREing @sql collided; the variable
names are now unique.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Type-hint the rebuilt SQLException ctor to avoid reflection
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Return the batch exception as is when nothing is chained
mssql-jdbc's BatchUpdateException message is already the plain server error —
only pgjdbc adds the 'Batch entry N ... getNextException' wording. So the
unwrap falls back to the original exception instead of synthesizing a
SQLException, and the driver test asserts the envelope wording stays out of
the message rather than checking the exception class.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Interpolate the escaped password in Snowflake workspace user DDL
CREATE/ALTER USER spliced the raw password while every other workspace
driver uses the ANSI-escaped value; the escaped form was already
computed and scrubbed here.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
A STRUCT column's INFORMATION_SCHEMA data_type spells out its entire
nested schema, and the COLUMN_FIELD_PATHS LEFT JOIN repeats it on every
nested-leaf row -- O(n^2) bytes for a column with n leaves. Dynamic-key
STRUCTs (e.g. Cloud Functions log-sink tables whose JSON payloads use
user IDs as keys) reach thousands of leaves with ~270KB type strings;
one such table produced ~4,000 rows x 271KB = 1.09GB buffered in
sync-fields!'s partition-by, OOMing the instance mid-sync.
Only short type-string prefixes are ever consumed
(raw-type->database+base-type dispatches on prefixes up to 10 chars and
stores at most RANGE<TIMESTAMP> verbatim), so SUBSTR(..., 1, 200) in the
query -- applied server-side, before the wire -- is behaviorally
identical while collapsing response size, parse cost, and the buffered
partition (1.09GB -> ~2.8MB for the table above).
The test needs no dataset or network: it captures the compiled SQL via
*process-native* and asserts both data_type columns are truncated.
* remove the last test file from the legacy structure
* convert test table into format of newer tests
* remove test for undefined since this isn't a valid type for RowValue
Update E2E test timings with averaged data
Collected timing data from 7 days of test runs.
All specs with timing data were included (simple averages).
🤖 Generated automatically by update-e2e-timings workflow
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: GitHub Actions <actions@github.com>
* migrate TokenField
* fix RemappedOption type for Autocomplete ComboboxStringItem
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* address RecipientPicker review: memoize maps, share lookup/label helpers
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix single-value search: accept top match on Enter
Autocomplete only completes on Enter when an option is highlighted, but
search options load async so none is. Restore old TokenField behavior via
onKeyDown.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix letters swallowing
* addressing the PR's comments
* added missing tests
* introduce Autocomplete + refactoring
* apply filter by clicking on enter
* fix test
* refactor autocomplete
* refactor autocomplete
* fix edge case + added unit tests
* fix type
* fix test
* fix test
* get rid of lint suppression + allow autoselect on open for multiautocomplete
* revert autoselect destructuring, restore eslint suppression (tests failing)
* explain exhaustive-deps suppression in MultiAutocomplete effect
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Remove withRouter from production components
* Delete the public withRouter export
* Type the route-leave hook for PlainRoute and drop the route casts
* Add element/Outlet + useRoute support to the router facade
* Convert root route tree to element/Outlet
* Convert admin route tree to element/Outlet
* Convert data-studio, models and metrics route trees to element/Outlet
* Convert transforms, account, monitor and public route trees to element/Outlet
* Convert enterprise route trees to element/Outlet and drop dead library segment routes
* Convert tenants admin route tree to element/Outlet
* Convert redirect, createEntityIdRedirect and modalRoute to element
* Ban component= on routes with eslint
* Drop component= from the facade Route type and convert remaining routes
* Drop the component= eslint rule, the facade type now enforces it
* Remove the unused withOutlet helper
* Assert the required tenantId route param instead of coercing to NaN
* Bridge element={null} to a nothing-rendering component, matching v7
* Memoize router context value so element routes re-render once per navigation
* Convert data_apps and impersonation routes merged from master to element
* initial proof-of-concept for data apps
POC 2
POC 3
POC 4
Revert "POC 4"
This reverts commit 1f62e574dc.
POC 5
POC 6
* Expose all SDK components to child realm
* Setup app publishing and serving
* Refactor distortion helpers
* Adjust skill
* Add useQuestionQuery SDK hook (#74900)
Co-authored-by: Timofei <timofei@metabase.com>
* Mock METABASE_EMBEDDING_SDK_BUNDLE for data app sdk hooks/components
* Adjust skill for TS project
* Update skill to clearly mention TS
* Add blocked APIs list to skill
* Prototype schema exports and typed hooks for custom data apps (#74978)
* add prototype for schema generation
* add formatting for generated schemas
* add skill file
* useQuestionQuery should take only ID based on @sanex3339 feedback
* add segments, measures, tables and useMetabaseQuery hook
* Add useMetabaseQuery to endowments
* Fix queryDataset
---------
Co-authored-by: Timofei <timofei@metabase.com>
* Add ability to update app bundle
* Simplify hook
* Add spinner instruction
* Mode Data Apps to OSS
* Instruct to not not use blocked event handlers
* Improve API, add styles isolation, add router components (#75418)
* Prepare skills for data app template + adjust router components (#75490)
* Fixes after rebase
* Move migration to v63
* Move migration to v63 #2
* Fix config.edn
* Fix BE tests
* Fix team name
* Fix duplicated config
* Adjust version text in skill
* Fix data app bundle uploading after legacy client removal
* Remove `actions` from config edn
* Fix failing test
* Fix failing test #2
* Add custom actions for data apps (#75627)
* Fix imports
* Fix type annotations
* Adjust migration
* Adjust permissions
* Fixed iframe security
* Data Apps publishing via remote repository POC (#75702)
* Update gitignore with data-app.html
* [PoC] Type safety and modularity improvements for custom data apps (#75381)
* allow separate libraries in schemas
* add metric type checking
* add collections parameter
* check that segments and measures belongs to table in useMetabaseQuery
* require less type information for measures
* require less type information for segments
* use metric dataset endpoint when querying measures in metrics
* add validation for breakout buckets
* fix filter operation type scoping
* expose filter type helper for nicer autocompletions
* update skill file
* add breakout schema helper for better type errors
* update breakout schema helper to accept binning and bucket
* move fields that are not used in compile and runtime to comments to shorten type errors
* enforce data truthfulness in skill
* enforce semantic layer richness in skill
* handle dimension key collisions mapped from many tables
* update skill description
* prevent hallucinated measures and breakouts
* fix field serialization
* add guidance on using filter matcher
* add skill guidance on long results and units
* accept query object in StaticQuestion and InteractiveQuestion
* extract use-metabase-query to multiple files
* update skill file to use sdk question by default
* update skill file to avoid hallucinated derived calculations
* add new functions into sandbox
* kondo and cljfmt lint fixes
* update CSP rules to allow style tags for data apps
* address review feedback on scoping and metric filters
* address code review feedback
* Add aggregations support for breakouts in query hooks (#75723)
* add aggregations support
* add additional aggregation helpers
* Add metrics support in useMetabaseQueryObject hook (#75728)
* add metric queries
* indicate metric support in skill file for useMetabaseQueryObject
* add new version for release
* fix query dataset tests
* update release workflow temporarily
* update skill description
* update skill to use more generic examples
* hotfix: allow esm usage in vite dev mode
* skill file update for table caveat
* update sandbox
* fix type errors in useMetabaseQuery
* add v0.63.2 to release workflow
* update skill file to use the 63-data-apps version
* Move data apps to the EE directory and under `data-apps` token feature (#75871)
* remove skill file alias
* Add error handling for data apps
* Add spinner while iframe is loading
* Add OSS/no token upsell banner placeholder for data apps (#75893)
* ensure a fixed height is set if container has height
* Allow including everything in data and metric library in schema endpoint (#75944)
* add filter for including library and data collections
* address review on skill file
* remove id field from rendered comment
* remove useQuestionQuery and useMetricQuery hook
* Improve env files lookup in skills (#75955)
* Allow `unsafe-eval` for data app iframe (#76004)
* Fix lint
* Fixes after rebase
* Fix duplicated test
* Improve Providers and typesafety (#75934)
* chore: reformat single expression functions to arrow
* Use entity id in questionCollections when generating typed schemas (#76230)
* allow using entity ids for question collection filter
* guide the agent towards using entity ids
* add failing test for invalid entity id
* Make custom viz work in data apps (#76237)
* Allow blob: images in data-app CSP (EMB-1923)
SDK custom visualizations load their icon assets via blob: URLs (to avoid
the cross-origin auth issue), but the /embed/data-app/ route allowed
'unsafe-eval' without the blob: scheme in img-src, so icons were blocked
under a production CSP. Add blob: to img-src scoped to the data-app iframe
route only; the main app img-src stays strict.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* EMB-1880: support for `providerProps` in data apps
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Reduce schema bloat in dimensions of generated metrics when many tables are involved (#76003)
* Add useQuestionQuery SDK hook (#74900)
Co-authored-by: Timofei <timofei@metabase.com>
* Prototype schema exports and typed hooks for custom data apps (#74978)
* add prototype for schema generation
* add formatting for generated schemas
* add skill file
* useQuestionQuery should take only ID based on @sanex3339 feedback
* add segments, measures, tables and useMetabaseQuery hook
* Add useMetabaseQuery to endowments
* Fix queryDataset
---------
Co-authored-by: Timofei <timofei@metabase.com>
* [PoC] Type safety and modularity improvements for custom data apps (#75381)
* allow separate libraries in schemas
* add metric type checking
* add collections parameter
* check that segments and measures belongs to table in useMetabaseQuery
* require less type information for measures
* require less type information for segments
* use metric dataset endpoint when querying measures in metrics
* add validation for breakout buckets
* fix filter operation type scoping
* expose filter type helper for nicer autocompletions
* update skill file
* add breakout schema helper for better type errors
* update breakout schema helper to accept binning and bucket
* move fields that are not used in compile and runtime to comments to shorten type errors
* enforce data truthfulness in skill
* enforce semantic layer richness in skill
* handle dimension key collisions mapped from many tables
* update skill description
* prevent hallucinated measures and breakouts
* fix field serialization
* add guidance on using filter matcher
* add skill guidance on long results and units
* accept query object in StaticQuestion and InteractiveQuestion
* extract use-metabase-query to multiple files
* update skill file to use sdk question by default
* update skill file to avoid hallucinated derived calculations
* add new functions into sandbox
* kondo and cljfmt lint fixes
* update CSP rules to allow style tags for data apps
* address review feedback on scoping and metric filters
* address code review feedback
* Add aggregations support for breakouts in query hooks (#75723)
* add aggregations support
* add additional aggregation helpers
* Add metrics support in useMetabaseQueryObject hook (#75728)
* add metric queries
* indicate metric support in skill file for useMetabaseQueryObject
* add new version for release
* fix query dataset tests
* remove useQuestionQuery and useMetricQuery hook
* Data Apps publishing via remote repository POC (#75702)
* avoid duplicating dimension fields
namespace dimensions by table
add dimension id to mapping
pick from table fields for only allowed dimensions
fix incorrect type guards
allow querying metric using question as source
remove createMetabaseQuery from public interface
add sourceFieldId to metric
* doc generation type fixes
---------
Co-authored-by: Timofei <timofei@metabase.com>
* Add allowed hosts handling for data apps (#76094)
* Update version
* Properly include data app changes in `pull changes` modal (#76294)
* Move app template into main repo, handle skill versioning flow (#76288)
* feat(data-apps): allow passing down a card with viz settings (#75993)
support card prop in sdk question component
* Rename data app env variables and read them only in dev mode (#76604)
* Add custom data apps layout with sliding toolbar (#76600)
* Remove unused helper
* Dev env Sandboxing, Human Readable Errors and Copy Helper (#76597)
* Revert wrong rebase changes
* Improve creating filtering interfaces in custom data apps (#76401)
* update skill files to guide filter usages
* update skill file to prevent filtering mistakes
* move filtering ui patterns into a separate file
* do not treat mb_replace_me as valid creds
* use combobox for filters
* pass title false to sdk question
* combobox trigger before typing
* Add missing exported types
* Apply visualizations and visualization settings from SDK package (#76659)
* ask the agent to pull possible visualizations
* ask the agent to use card prop
* teach the agent when to use viz settings
* Support query-only SDK cards
* Remove cross references between data apps skills (#76661)
Remove data app skill cross references
* Move Data App Dev Vite App setup into SDK package (#76695)
* Don't navigate on ad-hoc question chart clicks (#76751)
* Fix sandboxing crashes and update skill file mistakes (#76858)
Fix data app sandboxing and skill guidance
* Block form submissions in data apps via CSP (#76882)
* Fix SDK theming hover text colors in generated data apps (#77083)
update guidance on colors
* Remove SDK question query prop for data apps use cases (#77080)
* Remove SDK question query prop
* update tests to use card prop instead
* Adopt Lib.createTestQuery for table sources when querying in data apps (#75957)
* Add base metabase-lib query implementation
* switch to Lib.createTestQuery DSL
* Load query metadata at runtime for metabase-lib instead of hard-coding (#76874)
* load query metadata dynamically
* accept id-only source references
* Wait for SDK login before resolving query metadata
* refactor: use useAsyncFn to handle states
* pass source-name to createTestQuery
* Add metric aggregations for useMetabaseQuery via Lib.createTestQuery (#76947)
* add metric sources for useMetabaseQuery via Lib.createTestQuery
* use MetabaseQueryObject instead of opaque type marker
* Fix metric schema and join source validation
* add metric library filtering to skill files
* make metric an aggregation expression instead of join sources
* mark tableId as never in joined fields
* export public MetabaseQueryObject type
* use camelCase for source name and field id
* Add initial data app e2e setup (#77101)
* Add sorting and fields to useMetabaseQuery via Lib.createTestQuery (#76966)
* add sorting, limiting and fields to useMetabaseQuery
* refactor: cleanup sorting and aggregations
* Fix metric order by validation
* Support saved questions for useMetabaseQuery via Lib.createTestQuery (#77073)
* add saved questions as useMetabaseQuery input
* remove extra operators for saved cards
* Add unit fe/be tests for data apps (#76979)
* Fix lint
* Use modern api in query-dataset.ts
* Fix BG color
* Fix execute-action tests
* Update SDK-data-apps version
* Move migrations to v64 folder
* Adjust router components to use metabase/router
* Allow opening a data app for any signed user
* Rename data app skills (#77266)
* Update DataAppLayout.tsx `go back` button logic
* Harden data apps skills and action validation errors (#77261)
* harden data apps skills and actions error validation
* do not display models as questions in data apps
* clarify action result types
* Agents should report errors when models, actions or questions cannot be schema generated (#77260)
* return an error if models or saved questions are invalid
* return an error if actions are invalid
* update skill file guidance on model generation
* fix model boundary
* Add data apps admin UI + routing/sandboxing/isolation e2e (#77271)
* Rename data-apps paths to /apps (#77298)
* Include all action-included models in the typed schema when a mutation is specified (#77291)
* add includeModels=true flag to include all models
* omit model columns and models without actions
* remove extra metadata
* Adjust admin UI for Data Apps (#77420)
* Add more e2e/unit tests for Data Apps (#77524)
* Update bun.lock
* Fix errors after rebase
* Parallelize dts rollup (#77663)
* revert EMB-1840 measures support in metric dataset endpoint
* chore: add data app module (#77705)
* test ci
* apply code review suggestion
* Setup separate entries for client and node dev data app entries (#77687)
* Fix type casts lint errors (#77696)
* Adjust data apps setup + add proper analytics setup (#77604)
* Extract common and scope helpers + tests for typed schema module (#77403)
* refactor: extract common and query param helpers
* refactor: use kebab-case for query params
* refactor: cleanup schema mapping
ci: fix module config
* refactor: rename schema scope helpers to scope.clj
* refactor: extract scope_test.clj
* Extract typed schema rendering helpers to render.clj (#77703)
* extract schema render function
* move runtime compact test out
* remove javascript and json endpoints from schema generator
* refactor: cleanup javascript.clj and render.clj
* Extract typed schema generators for tables and questions (#77729)
* refactor: extract schema generator modules
* remove question-details from model-schemas
* fix: batch queries in table and question schema generator
* remove N+1 in typed schema field table lookup
* Extract typed schema generators for models and actions (#77739)
* extract model schema generator
* batch model action querying to avoid n+1 queries
* Extract typed schema generator for metrics (#77806)
* Extract typed schema generators for models and actions (#77739)
* extract model schema generator
* batch model action querying to avoid n+1 queries
* extract metric schema generator
* test: simplify fields in metric_test fixtures
* refactor: clean up metric schema generation
* remove n+1 from metric field dimension read
* disable default temporal dimension breakout
* surface errors if metric schema cannot be built
* revert metrics/api.clj to master to remove outdated comments
* Fix routes
* Take a data app's slug from its directory name (#77688)
* Merge data-apps specific migrations into one
* ci: fix failing backend test and cljfmt lint
* Add a beta badge in data apps admin page heading (#77877)
marks the data app as beta
* Revert "Merge data-apps specific migrations into one"
This reverts commit 383a5fa28d.
* Revert "Revert "Merge data-apps specific migrations into one""
This reverts commit 948d904b62.
* ci: actionlint fix
* Fix data apps route
* Use getSubpathSafeUrl in attachIframeUrlMirror
* Fix comment
* Do not send apps with sync error for token check
* Add data app listing sidebar section for non-admins (#77977)
* show data apps in sidebar for non-admins
* add app icon to data apps
* use available=true to filter no sync error and enabled apps
* fix navbar keys in data apps
* test: use non-admin user to assert non-admin response
* fix toucan query in data apps listing endpoint
* re-export entity detail helpers and result column to metabot core
* open data apps in new tab from the sidebar
* Prevent race conditions in responses of useMetabaseQuery hook (#78023)
prevent stale responses in useMetabaseQuery
* ci: remove tools.util from module declaration
* Merge migrations into a single one
* reset release-embedding-sdk.yml to master
---------
Co-authored-by: Phoomparin Mano <poom@metabase.com>
Co-authored-by: Phoomparin Mano <phoomparin@gmail.com>
Co-authored-by: Nicolò Pretto <info@npretto.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Some cleanup of some false alarms I encountered in test environments while working on a different issue. Destination databases (those with `router_database_id` set) are reached only through their router: metadata, sync, permissions, etc. Only execution goes against the destination's connection. A `data_permissions` row pointing at a destination is never consulted -- it's already actively stripped.
There are test utilities that helpfully create in-test permissions for all databases created, so in a test environment it would look exactly like the invariant was being violated, but that was only because the test utilities deviated from production behavior. This brings the test behaviors into line with production (no permissions get written).
There should be no change in production behavior in this PR, but as a bonus, I figured that a safety guard filtering rows out on reads is not as good as filtering them out on reads *and* failing loudly on writes. I also added guards that a normal database can't be turned into a destination database, which was a behavior tests were relying on for easier setup (but again, the tests then deviated once more from production behavior).
Since 4.28 Liquibase ships an opt-out analytics feature that, on the first
migration run, fetches config from config-endpoint.liquibase.com and sends
anonymized usage events.
* Parse :absolute-datetime string literals in repr resolve
Metabot's representations repair pass wraps bare ISO `between` bounds as
`[:absolute-datetime {} "2024-01-01" :day]` to satisfy the lib `:between`
temporal schema, leaving the literal as a string. That form is schema-valid
(so it passes construct_notebook_query validation), but the JVM query
processor expects a real java.time value — optimize-temporal-filters throws
on a bare string, and the QP's own wrap-value-literals only parses temporal
strings that arrive as raw values, not ones already wrapped in
:absolute-datetime.
Parse those string literals into java.time values in resolve-query, where the
portable repr form becomes QP-ready pMBQL, alongside the existing
annotate-field-types pass.
* Address review: tighten docstrings, use update over assoc
- parseable-temporal-string?: contract-first two-line docstring
- parse-absolute-datetime-literals: move the "why" into inline comments at
the walk, keep the docstring to contract + idempotency
- use (update node 2 u.date/parse) instead of assoc
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Address review: year/year-month, fail-fast, export round-trip
- parse year/year-month :absolute-datetime literals too (u.date/parse maps
them to LocalDate), so they are QP-ready instead of crashing later
- fail fast on a structurally-ISO but invalid literal (e.g. "2024-13-45"):
re-throw as an :agent-error? ex-info so the LLM gets a clean message
- export: format java.time values back to ISO-8601 strings in
portable-repr-form so resolve -> export round-trips to the documented
string form
- tests: per-shape parsing (date/datetime/offset/year/year-month),
idempotency, invalid-literal, and export round-trip
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Move :absolute-datetime string parsing to the QP layer
Revert the resolve-layer parsing and instead teach wrap-value-literals to
parse string-valued :absolute-datetime clauses, not just raw values. The
repair pass wraps bare `between` bounds as `[:absolute-datetime {} "2024-01-01"
:day]`; wrap-value-literals previously only parsed temporal strings that
arrived as raw values, so a pre-wrapped string slipped through and
optimize-temporal-filters threw on the bare string.
The QP is the right layer: wrap-value-literals runs at execution time, where
both the report timezone (bound) and the comparison field's type are
available — so the parse is type- and timezone-aware for free, and the stored
query keeps its portable ISO-string form (no export round-trip to invert).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fail fast on malformed :absolute-datetime literals in resolve
Add a resolve-side validation pass that rejects temporal literals which pass
the schema's structural regex but are not real dates (e.g. "2024-13-45"), so
the agent gets an actionable error at construct time instead of a failure at
query execution.
Validation only — it does not coerce. The string -> java.time coercion stays
in the QP's wrap-value-literals, where the field type and report timezone are
available. Accepts every valid :absolute-datetime shape: date, datetime,
offset-datetime, year, and year-month.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Preserve unit and wrap mixed :between bounds for string :absolute-datetime
When re-parsing a string-valued :absolute-datetime in wrap-value-literals:
- Carry the clause's own temporal unit through instead of discarding it. The
comparison field's bucket still wins, but for an unbucketed field the literal's
unit is now the fallback (rather than :default), so a year/year-month literal
like [:absolute-datetime {} "2024" :year] keeps its bucket instead of silently
collapsing to a single instant.
- Wrap each :between bound independently so a mixed bound (string :absolute-datetime
on one side, :now / :relative-datetime / an already-parsed :absolute-datetime on
the other) still coerces the wrappable side instead of falling through unwrapped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Guard comparison literals and tidy :absolute-datetime validation
- wrap-value-literals: guard the field position in the :=/:!=/<…> arms so a
comparison with a wrappable literal on both sides can't bind a literal as the
field and leave it unparsed; it now falls through unchanged.
- resolve: validate :absolute-datetime literals via lib.walk/walk-clauses
(no tree rebuild, shares clause-of-type? detection with the QP) instead of
clojure.walk/postwalk.
- Move cross-subsystem rationale out of the string-valued-absolute-datetime
docstring into an inline comment.
- Add a test covering the two-literal comparison fall-through.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Parse fieldless :absolute-datetime comparisons and bound bad-literal errors
Wrap a string-valued :absolute-datetime on both sides of a comparison
(no field to wrap against) so each literal is parsed with its own unit
instead of leaving a bare string for later middleware to choke on.
Truncate the untrusted literal before it lands in the agent-facing
exception message and ex-data, so a pathological value can't flood logs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Treat :default field unit as unbucketed for :absolute-datetime literal fallback
A temporal field can carry :temporal-unit :default (add-default-temporal-unit
stamps it on :temporal/requires-default-unit drivers, and an LLM can author it
directly). The unit-fallback guard only skipped a nil unit, so a :default field
dropped the literal's own unit and collapsed e.g. [:absolute-datetime {} "2024"
:year] to a single :default-bucketed instant. Treat :default as unbucketed too.
* Validate :absolute-datetime literals with plain walk to keep resolve gate friendly
validate-absolute-datetime-literals walked the query with lib.walk/walk-clauses,
a mu/defn whose input schema validates the whole query against ::lib.schema/query.
Under test instrumentation that threw a raw "Invalid input" error on an otherwise-
malformed query (e.g. an :offset in :expressions), pre-empting the friendlier
query-not-runnable agent error the downstream canRun gate is meant to surface.
Use clojure.walk/postwalk and a bare [:absolute-datetime _ s _] shape check, matching
the sibling annotate-field-types pass, so only the literals are inspected.
* Use match/match-many for :absolute-datetime literal validation, add mixed-between raw-value test
Addresses PR review feedback: swap the manual walk/postwalk in
validate-absolute-datetime-literals for match/match-many, and cover a
:between with a raw (unwrapped) string bound plus a :relative-datetime
bound in wrap_value_literals_test.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Show honest "still working" state when a Git sync push stalls (GHY-4132)
A large push can stop reporting progress while the export is still running; the
backend then reports the task as :timed-out with ended_at nil. The push modal
treated that as neither success nor error and stayed frozen on "Exporting
content…" until the page was refreshed.
Render a distinct stalled state instead: drop the frozen progress bar and show
"Still working. No update for N minutes." (internationalized, singular/plural
aware). The counter derives from last_progress_report_at and stays fresh via the
existing 2s current-task poll — no separate ticker.
* Add throttled monotonic progress reporter for remote-sync tasks (GHY-4132)
* Ramp remote-sync export progress across plan + serialize phases (GHY-4132)
* Report remote-sync export commit checkpoint before push (GHY-4132)
* Report remote-sync push progress via JGit ProgressMonitor (GHY-4132)
* Fix throttle, dead 0.66 write, and push liveness in export progress (GHY-4132)
commit-staged! was passed a pre-forced report closure, so every push
ProgressMonitor tick bypassed the 10s throttle and hammered the DB. Force only
the one-shot commit checkpoint (0.8) explicitly in finish-commit!, and let
push ticks go through the throttled reporter.
The forced 0.66 serialize-boundary write ran after commit-staged! returned,
by which point the 0.8 checkpoint (and push ticks) had already advanced the
monotonic reporter past it, so it was silently dropped. Move it to the end of
each export's stage-fn, before the checkpoint/push run.
The push monitor only reported during the exactly-titled "Writing objects"
phase with a known positive total; a non-English JVM locale renames that
title and JGit may report an unknown (zero) total, so the push phase could go
silent for its whole duration and re-trip the stale-progress timeout. Report
on every update tick regardless of phase/total, falling back to the 0.8
push-start fraction — repeated equal values are monotonic no-ops but still
bump last_progress_report_at, throttled to at most once per 10s.
* Use dayjs diff for the stalled-progress minute counter (GHY-4132)
* Expose search's ranked set and page-hydration as separate steps
`search` already ran in two stages: rank the permission-filtered result set
(capped at max-filtered-results), then slice that set to the requested page
and hydrate only those rows. The comment on `search-results` spells out why
the slice comes first — hydration does per-row work, including the expensive
`can_write` check, so it should only touch rows that are actually returned.
Both stages were sealed inside `search`, so a caller holding several ranked
sets had no way to combine them and paginate the combination. Its only option
is to ask `search` for N hydrated pages and merge those, which hydrates N
pages to return one, and — because each page is sliced from its own ranking
before merging — cannot page the merged order correctly at all.
Split the two stages into `ranked-results` and `search-results` and export
both from the module API. `search` is now their composition and behaves
identically; no call site changes.
The motivating caller is metabot's multi-query search, which runs a search per
query and fuses the rankings with RRF. Fusing at the ranked-results seam lets
it hydrate only the page it returns, page the fused order coherently, and
report a total for the fused set.
* Remove comments from search core.clj
Removed comments explaining the purpose of `ranked-results` and `search-results`.
* kick ci
* Convert sandboxes and advanced_permissions plugin modules to TypeScript
* Address review comments: remove typecasting from sandboxes actions and unit spec
@@ -10,23 +10,23 @@ This skill helps you add product analytics (Snowplow) events to track user inter
## Quick Reference
Analytics events in Metabase use Snowplow with typed event schemas. All events must be defined in TypeScript types before use.
Analytics events in Metabase use Snowplow with typed event schemas. Simple events are declared **where they are used** — `trackSimpleEvent` is generic and validates the payload at the call site.
**Key Files:**
-`frontend/src/metabase-types/analytics/event.ts` - Event type definitions
**Primary standard: the [`typescript-write`](../typescript-write/SKILL.md) skill.** Load it first — it defines the authoring rules this review enforces, alongside `frontend/CLAUDE.md` and `docs/developers-guide/frontend.md`.
Adherence to `typescript-write` is the **highest-priority** review dimension: rank any violation of its provisions above all other findings. Treat its **no-`any` hard rule** (no explicit *or* implicit `any` in new code) as **blocking**, and verify it with the LSP rather than by eye.
Adherence to `typescript-write` is the **highest-priority** review dimension: rank any violation of its provisions above all other findings. Treat its **no-`any` hard rule** (no explicit *or* implicit `any` in new code) as **blocking**. Use TypeScript LSP tools to inspect inferred types when available; otherwise rely on type-checking and linting.
@@ -13,7 +13,7 @@ description: Write TypeScript and JavaScript code following Metabase coding stan
- **New code must not introduce `any`, explicit or implicit.** No `any` annotations, no `as any` / `as unknown as`, no untyped parameters or returns that infer `any`, no implicitly-`any` destructures or array/object literals.
- **Untyped third-party / boundary values** must be typed at the boundary (a declared type, `unknown` + type guard, or a small typed wrapper) — never let `any` propagate inward.
- **Mandatory LSP verification.** After writing or editing any TS/TSX, inspect the changed symbols with the TypeScript Language Server (hover to read inferred types; `goToDefinition` to confirm sources) and run the project type-check.
- **Mandatory type verification.** Before finishing a TS/TSX change, run `bun run type-check-pure`. If TypeScript LSP tools are available, also inspect changed symbols with hover and go-to-definition; otherwise skip LSP check.
## Type tightening
@@ -26,6 +26,7 @@ description: Write TypeScript and JavaScript code following Metabase coding stan
- **Avoid non-null assertions (`!`)**. Prefer a guard, early return, or `?.`. Use `!` only when non-nullness is provably true and localized, with a comment.
- **Type guards belong in `frontend/src/metabase-types/guards/`**. Do not redefine them locally.
- **A cast you can't avoid needs a real justification comment.** The `metabase/no-unjustified-type-casts` rule accepts any preceding comment — state the actual reason the cast is safe. NEVER write the legacy `// Unjustified type cast. FIXME` placeholder; it exists only on casts that predated the rule, and copying it sneaks an unjustified cast past the linter. If you can't articulate why the cast is correct, the cast is wrong — fix the types.
{clojure.core/print{:message"Use metabase.util.log instead of clojure.core/print"}
{metabase-enterprise.entity-retrieval.core/search-unfiltered{:message"Permission bypass risk: library-index hits are unfiltered — resolve each entity and drop what the current user can't read"}
metabase.entity-retrieval.core/search-unfiltered{:message"Permission bypass risk: library-index hits are unfiltered — resolve each entity and drop what the current user can't read"}
metabase.entity-retrieval.mirror/search-unfiltered{:message"Permission bypass risk: library-index hits are unfiltered — resolve each entity and drop what the current user can't read"}
clojure.core/print{:message"Use metabase.util.log instead of clojure.core/print"}
clojure.core/println{:message"Use metabase.util.log instead of clojure.core/println"}
clojure.core/printf{:message"Use metabase.util.log instead of clojure.core/printf"}
clojure.core/prn{:message"Use metabase.util.log instead of clojure.core/prn"}
> Checkout the branch and run \`./backport.sh\` script. Force push your changes after cherry-picking. Check the box below when done.
> Checkout the branch and run \`./backport.sh\` script. Force push your changes after cherry-picking. Mark the PR as ready for review once conflicts have been addressed.
@${githubUsername} You have modified embedding code. Please make sure the PR title follows [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) style.
message:"⚠️ Main app initial bundle size (gzipped) increased by ${{ steps.compare.outputs.app_size_change_percent }}% compared to the ${{ github.base_ref }} branch (threshold: ${{ env.BUNDLE_SIZE_THRESHOLD_PERCENT }}%). This is the JavaScript every user downloads before the app renders, so check whether the new code can be loaded lazily instead."
include-log:false
- name:Comment on PR if app bundle size has decreased
description:"Branch we want to release the SDK from (to release from other branches use the workflow from those branches)"
read_first:
description:"⚠️ Manual override. Publishes whatever version is already committed in package.template.json; it never bumps. Normally you want the SDK Version Bump PR workflow, where merging the PR publishes."
type:choice
required:true
default:"master"
required:false
options:
- master
- Publishes the branch picked above
# Step 2 of the new release process: merging a version-bump PR (opened by
# sdk-version-bump-pr.yml, labeled `sdk-release-trigger`) triggers a release
# of whatever branch that PR targeted. See `prepare-release` below.
#
# The `paths` filter keeps this from creating a run for every PR closed
# anywhere in the repo - version-bump PRs always touch package.template.json,
# so this is a reliable pre-filter ahead of the label/merge check below.
- name:Check out the code using the provided branch
uses:actions/checkout@v6
with:
ref:${{ inputs.branch }}
ref:${{ needs.prepare-release.outputs.branch }}
fetch-depth:0
fetch-tags:true
- name:check tags
run:|
if [ $(git tag -l | grep $tag$) ]
if git tag -l | grep -q "${tag}$"
then
echo "::error::Tag '${{ env.tag }}' already exists. If you expect to run this workflow with the same tag, please remove the tag first and rerun this workflow again."
echo "::error::Tag '${{ env.tag }}' already exists. If you expect to run this workflow with the same tag, please remove the tag first and rerun this workflow."
exit 1
else
exit 0
fi
test:
needs:check-git-tag
needs:
- prepare-release
- check-git-tag
runs-on:${{ vars.DEFAULT_RUNNER_KEY }}
timeout-minutes:20
steps:
- name:Check out the code using the provided branch
uses:actions/checkout@v6
with:
ref:${{ inputs.branch }}
ref:${{ needs.prepare-release.outputs.branch }}
- name:Prepare front-end environment
uses:./.github/actions/prepare-frontend
@@ -262,14 +354,16 @@ jobs:
run:bun run embedding-sdk:test-unit
build-sdk:
needs:[test, determine-sdk-version]
needs:
- prepare-release
- test
runs-on:${{ vars.DEFAULT_RUNNER_KEY }}
timeout-minutes:20
steps:
- name:Check out the code using the provided branch
uses:actions/checkout@v6
with:
ref:${{ inputs.branch }}
ref:${{ needs.prepare-release.outputs.branch }}
fetch-depth:0
fetch-tags:true
@@ -279,11 +373,9 @@ jobs:
- name:Prepare back-end environment
uses:./.github/actions/prepare-backend
- name:Bump published npm package version
# NOTE: this should happen before "Build SDK bundle" as we inject SDK version into the code during build step
console.log(`Could not look up the author of ${BREAKING_COMMIT}: ${error.message}`);
}
}
const failedJobsList = failedJobs.map(job => ({
"type": "rich_text_section",
"elements": [
@@ -87,9 +115,9 @@ jobs:
]
}));
await slack.chat.postMessage({
channel:'engineering-ci',
text:'Failing tests',
await sendSlackMessage({
channelName:'engineering-ci',
message:'Failing tests',
blocks:[
{
"type": "header",
@@ -107,7 +135,7 @@ jobs:
"type": "section",
"text": {
"type": "mrkdwn",
"text": `Commit <https://github.com/metabase/metabase/commit/${BREAKING_COMMIT}|${BREAKING_COMMIT.slice(0,7)}> by ${AUTHOR} has failing <${FAILED_RUN_URL}|${FAILED_RUN_NAME}> tests on <https://github.com/${owner}/${repo}/commits/${BRANCH}|\`${BRANCH}\`>`
"text": `Commit <https://github.com/metabase/metabase/commit/${BREAKING_COMMIT}|${BREAKING_COMMIT.slice(0,7)}> by ${AUTHOR} (${mentionUserByGithubLogin(AUTHOR_LOGIN)}) has failing <${FAILED_RUN_URL}|${FAILED_RUN_NAME}> tests on <https://github.com/${owner}/${repo}/commits/${BRANCH}|\`${BRANCH}\`>`
# Step 1 of the SDK release process: bump the version (and the sdkRelease
# metadata that controls the npm dist-tag / `latest` promotion) in
# package.template.json, then open a PR labeled `sdk-release-trigger`.
# Merging that PR is what actually triggers a release, via the
# `pull_request` trigger on release-embedding-sdk.yml.
#
# Release types (the dispatch form lists them in release-cycle order):
# beta - release branch, before gold: publish a beta.
# patch - release branch: graduate the beta to 0.N.0 at gold,
# then ship 0.N.1, 0.N.2 and so on after gold.
# preminor - master, at gold: move master to the next major.
# alpha - master, any time: bump the development version.
# custom - one-off branch (esbuild, data-apps): publish under its
# own prerelease id, which is reused on later bumps.
#
# The dispatch form's descriptions render as the input's label, which is a
# single plain text node - newlines collapse to spaces and no markdown is
# parsed. Keep them short and single-line; put detail here instead.
name:SDK Version Bump PR
run-name:Bump Embedding SDK version (${{ inputs.release_type }}) on ${{ github.ref_name }}
on:
workflow_dispatch:
inputs:
read_first:
description:"Bumps the version on the branch chosen in 'Use workflow from' above."
type:choice
required:false
options:
- Bumps the branch picked above
release_type:
# Option names are capitalised for emphasis only - the values stay
# lowercase. The label renders as plain text, so caps is the only
# emphasis available (same reason prerelease_id below shouts FIRST).
description:"BETA - before gold, on a release branch. PATCH - at gold and after, on a release branch. PREMINOR - at gold, on master. ALPHA - any time, on master. CUSTOM - any time, on a one-off branch."
type:choice
required:true
# Pinned so the release-cycle ordering below does not decide the
# preselection: without a default, GitHub preselects the first option.
default:alpha
options:
- beta
- patch # one entry, two phases:the version decides 0.N.0 vs 0.N.1
- preminor
- alpha
- custom
prerelease_id:
# Left empty on later bumps, the id is reused from the current version.
description:"For release_type=custom only, and only on the FIRST cut of a one-off branch. Enter a name like data-apps: the version becomes 0.62.5-data-apps.0 and the dist-tag 62-data-apps. Leave empty on later bumps."
type:string
required:false
concurrency:
# one version-bump in flight per branch at a time, to avoid racing PRs
# A leftover branch means an earlier bump to this same version was
# abandoned. Say so, rather than pushing a near-duplicate alongside it.
if git ls-remote --exit-code --heads origin "$BUMP_BRANCH" >/dev/null 2>&1; then
echo "::error::'$BUMP_BRANCH' already exists on origin - an earlier bump to this version was never merged. Close its PR, delete the branch, then re-run."
exit 1
fi
git checkout -b "$BUMP_BRANCH"
git add "$package_template"
git commit -m "chore(sdk): bump SDK version to ${{ steps.metadata.outputs.new_version }} (${{ inputs.release_type }})"
| Tag as `latest` | `${{ steps.metadata.outputs.tag_as_latest }}` |
`distTag` and `tagAsLatest` are computed from the branch and release type. `tagAsLatest` is `true` only when patching the current gold major (`vars.LATEST_MAJOR_VERSION`).
Merging this PR (with the `sdk-release-trigger` label) triggers [release-embedding-sdk.yml](../actions/workflows/release-embedding-sdk.yml), which builds, publishes to npm, and tags the release.
_Opened by the [SDK Version Bump PR workflow](../actions/workflows/sdk-version-bump-pr.yml), triggered by @${{ github.event.sender.login }}._
BODY
)")
echo "pr_url=$pr_url" >> "$GITHUB_OUTPUT"
# Separate from "Append workflow summary" above: that step runs before the
# PR exists, so the link has to be appended after this one opens it.
if [[ "$dir_version" -lt "$DIR_LAYOUT_FLOOR" ]]; then
echo "ERROR: ${file} targets v${dir_version}, but per-feature directories start at v${DIR_LAYOUT_FLOOR} — a pre-${DIR_LAYOUT_FLOOR} changeset goes in resources/migrations/${dir}_update_migrations.yaml" >&2
(format "Migration '%s' has primaryKeyExists precondition without tableName" (:id changeSet))
(format "Migration '%s' has %s precondition without %s: without it Liquibase snapshots the entire schema to answer the check (~2s per precondition on a large postgres appdb)"
(testing"foreignKeyConstraintExists without foreignKeyTableName is rejected"
(is-thrown-with-error-info?
"Migration 'v49.2024-01-01T10:30:00' has foreignKeyConstraintExists precondition without foreignKeyTableName: without it Liquibase snapshots the entire schema to answer the check (~2s per precondition on a large postgres appdb)"
(testing"indexExists without tableName is rejected, including in flat-map not form"
(is-thrown-with-error-info?
"Migration 'v49.2024-01-01T10:30:00' has indexExists precondition without tableName: without it Liquibase snapshots the entire schema to answer the check (~2s per precondition on a large postgres appdb)"
(testing"primaryKeyExists without tableName is still rejected (previous rule subsumed)"
(is-thrown-with-error-info?
"Migration 'v49.2024-01-01T10:30:00' has primaryKeyExists precondition without tableName: without it Liquibase snapshots the entire schema to answer the check (~2s per precondition on a large postgres appdb)"
io.github.camsaul/humane-are{:mvn/version"1.0.2"}; cleaner test output
io.github.metabase/hawk{:mvn/version"1.0.13"}; test runner
io.github.metabase/hawk{:mvn/version"1.0.18"}; test runner
io.opentelemetry/opentelemetry-sdk-testing{:mvn/version"1.54.1"}; InMemorySpanExporter for tracing tests; matches sdk impl (clj-otel-sdk 0.2.10 → 1.54.1); api is separately pinned to 1.62.0
io.zonky.test/embedded-postgres{:mvn/version"2.2.2"}; In-process postgres for dev and testing
;; Native postgres binaries for every OS/arch we're likely to run on in
- [Metabase's officially supported databases](../databases/connecting.md#connecting-to-supported-databases) (with the exception of SQLite and H2).
Metabase Cloud doesn't support [community database drivers](../developers-guide/community-drivers.md), or file-based databases (SQLite and H2), because there's currently no file storage available.
Metabase Cloud doesn't support [community database drivers](../developers-guide/community-drivers.md), or file-based databases (SQLite), because there's currently no file storage available.
{% include plans-blockquote.html feature="Loading from a configuration file" self-hosted-only="true" %}
On self-hosted [Pro](https://www.metabase.com/product/pro) and [Enterprise](https://www.metabase.com/product/enterprise) plans, Metabase supports initialization on launch from a config file named `config.yml`. The config file should be located at:
On self-hosted [Pro](https://www.metabase.com/product/pro) and [Enterprise](https://www.metabase.com/product/enterprise) plans, Metabase supports initialization on launch from a config file. The config file should be located at:
-The current directory (the directory where the running Metabase JAR is located).
-A file named `config.yml` or `config.yaml` in the current directory (the directory where the running Metabase JAR is located).
- The path specified by the [`MB_CONFIG_FILE_PATH`](./environment-variables.md#mb_config_file_path) environment variable.
The settings in the config file work the same as if you'd set the settings in the Admin Settings in your Metabase. Settings defined in this configuration file will update any existing settings. If, for example, a database already exists (that is, you'd already added it via the initial set up or **Admin settings** > **Databases**, Metabase will update the database entry based on the data in the config file). Which means: if you define a setting in the config file, and then later change that setting in your Metabase application, keep in mind that the config file will overwrite that change whenever Metabase restarts. Let's reiterate that in a blockquote:
- Environment variable only: you can't set this in the Admin settings or in a [configuration file](./config-file.md).
Number of days to retain rows in the ai_usage_log, metabot_conversation, and metabot_message tables. Minimum value is 30; set to 0 to retain data indefinitely.
@@ -294,6 +294,7 @@ Range: 1-100. To limit the total number of rows included in the file attachment
- Type: string
- Default: `null`
- Environment variable only: you can't set this in the Admin settings or in a [configuration file](./config-file.md).
Number of days to retain data in audit-related tables. Minimum value is 30; set to 0 to retain data indefinitely.
@@ -903,6 +904,7 @@ Number of threads to use for batched index updates, including embedding requests
- Type: boolean
- Default: `true`
- Environment variable only: you can't set this in the Admin settings or in a [configuration file](./config-file.md).
Whether or not we should install the Metabase analytics database on startup. Defaults to true, but can be disabled
via environmment variable.
@@ -1287,6 +1289,14 @@ When set to `true`, users who log in via LDAP will automatically get a Metabase
The array of last two ISO8601 dates when an admin dismissed the license token missing banner.
The AI provider and model for Metabot. Format: provider/model-name, e.g. `anthropic/claude-haiku-4-5`, `openai/gpt-5.4`, `openrouter/anthropic/claude-haiku-4.5`.
Refresh cached results this fraction of their cache duration before they expire, so requests keep being served from cache instead of waiting for a recomputation. Set to 0 to only refresh once results have expired.
### `MB_QUERY_CACHING_MAX_KB`
- Type: integer
@@ -1599,9 +1755,9 @@ The absolute maximum time to keep any cached query results, in seconds.
### `MB_QUEUE_BACKEND`
- Type: string
- Default: `quartz`
- Default: `null`
Which queue backend to use. Valid values: `quartz`, `memory`.
Which queue backend to use. Valid values: `quartz`, `memory`. When unset, defaults to `quartz`, or to `memory` when the task scheduler is disabled (`MB_DISABLE_SCHEDULER`).
### `MB_QUEUE_MAX_RETRIES`
@@ -1881,8 +2037,7 @@ on your IdP, this usually looks something like `http://www.example.com/141xkex60
@@ -82,7 +82,7 @@ You can use [Markdown](https://commonmark.org/help/) to format the text in your

To learn more, see [Fun with Markdown in your dashboards](https://www.metabase.com/learn/metabase-basics/querying-and-dashboards/dashboards/markdown).
To learn more, see [Format text with Markdown](./markdown.md).
### Including variables in text cards
@@ -321,12 +321,14 @@ See [Caching per dashboard](../configuring-metabase/caching.md#dashboard-caching
## Sharing dashboards with public links
If your Metabase administrator has enabled [public sharing](../embedding/public-links.md) on a saved question or dashboard, you can go to that question or dashboard and click on the **sharing** icon to find its public links.
If your Metabase administrator has enabled [public sharing](../embedding/public-links.md) on a saved question or dashboard, you can go to that question or dashboard and click on the **Share** icon to find its public links.

Public links can be viewed by anyone, even if they don't have access to Metabase. You can also use the public embedding code to embed your question or dashboard in a simple web page or blog post. Check out examples of simple apps with embedded dashboards in our [embedding-reference-apps repository](https://github.com/metabase/embedding-reference-apps). To learn more about [embedding](../embedding/introduction.md), check out our article on [How to use Metabase to deliver analytics to your customers](https://www.metabase.com/learn/metabase-basics/embedding/overview), as well as an article on how to combine branding, Single Sign-On, full app embedding, and row and column security to deliver [multi-tenant, self-service analytics](https://www.metabase.com/learn/metabase-basics/embedding/multi-tenant-self-service-analytics).
Admins can also [create an account](../people-and-groups/managing.md#creating-an-account) for someone and point them to a specific dashboard or question. On a dashboard or question, click the **Sharing** icon and select **Invite someone to view this**.
## Exporting results from a dashboard
You can export the entire dashboard as PDF, or export results of individual questions on the dashboard. See [Exporting results](../questions/exporting-results.md).
@@ -34,7 +34,7 @@ People in the Data Analyst group will have table metadata and data structure acc
_Data Studio > Tables_
You can see all tables in all databases connected to your Metabase in **Data Studio > Tables**, together with their owner, row count, and published state. For now, Metabase only displays row counts for PostgreSQL tables.
You can see all tables in all databases connected to your Metabase in **Data Studio > Tables**, together with their owner, visibility layer, row count, and published state. For now, Metabase only displays row counts for PostgreSQL tables.
You can search for table names, but the search will only match beginnings of words in table names. So for example, if you search for "base", results will include names like "Baseball stats" and "All your base are belong to us", but the results won't include tables like "Metabase secrets".
- **Metabase Cloud**: Basic transform functionality on Metabase Cloud - Starter, Pro, or Enterprise - comes with an additional small fee per successful transform run, see [Pricing](https://www.metabase.com/pricing).
Only people logged in with an email of a [Metabase Store admin](../../cloud/accounts-and-billing.md#add-managers) (not just Metabase _instance_ admins) can [enable basic transforms](./transforms-overview.md#enable-transforms).
Only people logged in with an email of a [Metabase Store admin](../../cloud/accounts-and-billing.md#add-managers) (not just Metabase _instance_ admins) can [enable basic transforms](./transforms-overview.md#enable-transforms).
### Cancel basic transforms
@@ -65,6 +65,8 @@ The Advanced transforms add-on comes with an additional charge per successful tr
To enable Advanced transforms functionality, you need to have [Basic transforms](#basic-transforms) already, see [Enable basic transforms](#enable-basic-transforms).
For the full self-hosted setup see [Set up transforms on a self-hosted Metabase](transforms-overview.md#set-up-transforms-on-a-self-hosted-metabase).
There are two ways to enable Advanced transforms:
- **From your Metabase instance**: you can navigate to a feature requiring advanced transforms (like Python transforms or transform inspector), and follow the prompts to upgrade.
@@ -38,7 +38,7 @@ Jobs have two components: schedule and tags.
- **Schedule** determines when the job will be executed: daily, hourly, etc. You can specify a custom cron schedule (e.g. "Every weekday at 9:05 AM"). The times are given in your Metabase's system timezone.
- **Tags** determine _which_ transforms a job runs, not when the job runs. For example, you can create a `Weekdays` tag, add that tag to a few transforms, then create a job that runs all the transforms with the `Weekdays` tag every weekday at 9:05AM.
Jobs will run all transforms tagged with any of the tags, plus any transforms that the tagged transforms depend on, see [Jobs will run all dependent transforms](#jobs-will-run-all-dependent-transforms).
Jobs will run all transforms tagged with any of the tags, plus any transforms that the tagged transforms depend on that aren't already up to date, see [Jobs include all dependent transforms](#jobs-include-all-dependent-transforms).
You can see which transforms a job will run and in which order on the job's page.
@@ -104,13 +104,22 @@ To delete a job:
2. Find the job you want to delete and click the **three dots** icon to the right of the job's name.
3. Select **Delete**.
## Jobs will run all dependent transforms
## Jobs include all dependent transforms
If one transform depends on another, Metabase will run the dependency first, even if that transform isn't tagged in the job. So if transform B depends on A, Metabase will first run A, even if A doesn't have a tag.
If one transform depends on another, Metabase adds the dependency to the job and runs it first, even if that transform isn't tagged in the job. So if transform B depends on A, Metabase makes sure A is up to date before running B.
This means that you can explicitly tag transform A to run daily, and transform B hourly, but because transform B depends on transform A, transform A will _also_ run hourly (in addition to daily), despite not having the tag.
## Jobs skip dependencies that are already up to date
You can see which transforms a job will run (and in which order) on the job's page in **Data Studio > Jobs**.
Metabase won't re-run a dependency that's still fresh. That way, a transform pulled into a frequent job doesn't get rebuilt more often than its own schedule calls for. So, for example, if you tag transform A `daily`, and transform B `hourly`, and hourly B depends on daily A, then A runs on its own daily schedule. The hourly job that runs B won't rerun A every hour.
Metabase skips a dependency when the dependency:
- Has its own tags and jobs, and none of those jobs' schedules have come due since the dependency last ran successfully.
- Has no tags at all, and it's already run successfully at least once.
## See which transforms a job will run
The job's page in **Data Studio > Jobs** lists every transform the job will run and in which order. The **Notes** column tells you when Metabase will skip a dependency, and flags dependencies that have no schedule of their own.
## Runs
@@ -120,4 +129,11 @@ You can see all past and current transform runs (both manual and scheduled) by g
You can click on any transform run to see more details about the run, like the error logs. To go to the transform definition from the transform run page, click on the icon next the transform name in the right sidebar.
The "Tags" column in the **Runs** table will only show the transform's specific tags. But the run might not have anything to do with those tags. Another job with different tags could have run the transform because [jobs will run _all_ dependent transforms](#jobs-will-run-all-dependent-transforms).
The "Tags" column in the **Runs** table will only show the transform's specific tags. But the run might not have anything to do with those tags. Another job with different tags could have run the transform because [jobs include all dependent transforms](#jobs-include-all-dependent-transforms).
## Emails about failed transforms
If your Metabase is [set up to send email](../../configuring-metabase/email.md), Metabase will let people know when transforms fail.
- **Individual transform failures**: when transforms fail during a scheduled job run, Metabase emails the last person to edit each failed transform (or the transform's creator, or admins if neither is active). Each email covers a single job run.
- **Daily digest of job failures**: each morning, Metabase emails all admins a summary of the scheduled job runs that failed or timed out the previous day. Manual runs aren't included in the digest, and Metabase skips the email entirely if no scheduled runs failed.
These settings can also be configured in the Metabase UI at **Admin** > **Settings** > **Python Runner**. Note that environment variables take precedence over UI settings.
| `MB_PYTHON_STORAGE_S_3_PATH_STYLE_ACCESS` | (Optional) Set to `true` for S3-compatible services like MinIO or LocalStack |
`MB_PYTHON_STORAGE_S_3_CONTAINER_ENDPOINT` is the host Metabase signs into the presigned URLs the runner uploads to and downloads from. Set it when the runner resolves storage by a different hostname than Metabase does.
## Using Docker Compose
Docker Compose simplifies managing multiple containers. Below are example configurations for different scenarios.
Below are example configurations for different scenarios.
### Simple setup with LocalStack
### Self-hosted storage with MinIO
For simple setup and local testing, you can use LocalStack to simulate S3:
For production, use an external S3-compatible storage service:
[MinIO](https://min.io/) is an S3-compatible server you can run yourself, which makes it a common pick for self-hosted Metabase. This Compose file runs Metabase, the Python runner, MinIO, and a short-lived container that creates the bucket Metabase will use — neither MinIO nor Metabase creates that bucket for you.
until mc alias set local http://minio:9000 "${MINIO_ROOT_USER}" "${MINIO_ROOT_PASSWORD}" >/dev/null 2>&1; do
echo "waiting for minio..."; sleep 2;
done
mc mb --ignore-existing local/metabase-python-runner
volumes:
minio-data:{}
```
Create a `.env` file with your configuration:
Both S3 endpoint variables point at `http://minio:9000` because Metabase and the runner share the Compose network and reach MinIO by the same hostname. `MB_PYTHON_STORAGE_S_3_CONTAINER_ENDPOINT` sets the host Metabase signs into the presigned URLs the runner uploads to and downloads from — if your runner resolves storage by a different hostname than Metabase does, set this to the runner's view of it.
Create a `.env` file:
```bash
AUTH_TOKEN=your-secure-token-here
S3_ENDPOINT=https://s3.amazonaws.com
S3_BUCKET=metabase-python-runner
S3_REGION=us-east-1
S3_ACCESS_KEY=your-access-key
S3_SECRET_KEY=your-secret-key
MINIO_ROOT_USER=minioadmin
MINIO_ROOT_PASSWORD=minioadmin
```
Then run:
Then bring everything up and try a transform:
```bash
docker compose up -d
```
1. Put a strong shared secret in `AUTH_TOKEN` (the runner and Metabase have to agree on it):
```bash
openssl rand -hex 32
```
2. Start the stack:
```bash
docker compose up -d
```
3. You'll need a Pro or Enterprise license with the [Advanced transforms add-on](addons.md). In Metabase, [enable transforms](transforms-overview.md#enable-transforms), then [create a Python transform](python-transforms.md#create-a-python-transform) and click **Run**.
4. Open **Data Studio > Jobs > Runs** and find your run. If it failed, the run's logs will usually tell you whether Metabase couldn't reach the runner, or the runner couldn't reach MinIO.
@@ -108,7 +108,7 @@ To use functions or classes from your Python library:
## Incremental Python transforms
By default, Metabase will process all the data in all input tables, drop the existing target table (if one exists), and create a new table with the processed data. You can tell Metabase to only write**new** data to your target table by marking your transform as incremental.
By default, Metabase will process all the data in all input tables, drop the existing target table (if one exists), and create a new table with the processed data. You can tell Metabase to only process**new** data by marking your transform as incremental.
### Prerequisites for incremental transforms
@@ -120,7 +120,8 @@ To make a Python transform incremental:
1. Go to the transform's page in **Data studio > Transforms**.
2. Switch to **Settings** tab.
3. In **Column to check for new values**, select the column in one of the source tables that Metabase should check to determine which values are new. Only some columns are eligible. See [prerequisites for incremental transforms](./transforms-overview.md#prerequisites-for-incremental-transforms).
3. In **Field to check for new values**, select the field in one of the source tables that Metabase should scan to determine which records are new or changed. Only some fields are eligible. See [prerequisites for incremental transforms](./transforms-overview.md#prerequisites-for-incremental-transforms).
4. (Optional) To update matching rows instead of appending new ones, [add a merge key](./transforms-overview.md#add-merge-keys-to-upsert-rows).
## Current limitations of Python transforms
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.