master
638 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Alexander Polyankin
|
5ce0b19bc0 | Actions optional model (#83435) | ||
|
YB
|
53e53addbc |
Bump jackson-core and jackson-databind to 2.21.7 (#82997)
bump jackson-core and jackson-databind to 2.21.7 |
||
|
Aleksandr Lesnenko
|
3d924036db | bind dev Docker ports, socket REPL, and rspack dev server to loopback only (#82707) | ||
|
bryan
|
1b57292d09 |
Send the SAML SessionIndex in single logout requests (#82441)
* Send the SAML SessionIndex in single logout requests
Metabase's LogoutRequest identified the session to end by NameID alone, which
is the user's email. Auth0 issues a distinct user ID per connection, so one
email can own several sessions and Auth0 cannot tell which to end:
client_error: No active session(s) found matching LogoutRequest
SAML Profiles 4.4.4.1 makes SessionIndex the element that scopes a
LogoutRequest to a session. Record the SessionIndex the IdP sends in its login
assertion on the session it creates, and send it back at logout.
SessionIndex is per session, not per user, so it lives on core_session rather
than auth_identity, which has a unique constraint on (user_id, provider) and
would let concurrent logins overwrite each other.
A hardcoded SessionIndex was tried once and broke Azure AD (#41600). Only an
index the IdP itself issued is ever sent; sessions without one (every session
predating this column, and every non-SAML session) produce exactly the request
they produce today. The existing saml-logout test, which asserts on the full
signed logout URL for a session with no SessionIndex, passes unchanged.
Requires saml20-clj 4.3.0, which parses SessionIndex out of login assertions
and can emit it in a LogoutRequest.
Fixes #77837
* Send the IdP's NameID, not the user's email, in logout requests
An IdP matches a LogoutRequest on NameID plus SessionIndex. We sent the user's
email as the NameID, but the IdP's NameID is not necessarily an email: Auth0
sends an opaque `auth0|<id>`. A LogoutRequest naming an identifier the IdP never
issued matches no session, so Auth0 answered:
No active session(s) found matching LogoutRequest
even once we started sending the correct SessionIndex.
Record the NameID and its Format from the login assertion alongside the
SessionIndex, and send those at logout. Sessions predating these columns have a
NULL NameID and fall back to email, so logout keeps working across an upgrade.
Verified end to end against a live Auth0 tenant: the LogoutRequest now carries
NameID `auth0|<id>` with Format `unspecified` and the assertion's SessionIndex,
Auth0 returns StatusCode Success, and the session row is deleted.
|
||
|
Chris Vandoodewaard
|
8a7cd79f47 |
Bump backend deps to resolve security vulnerabilities (#82261)
* bump backend deps * missed bump |
||
|
Tyler Plude
|
40a15bc485 |
[GIT-81733] Handle jar native filesystem paths (#81927)
* return a native filesystem path from get-jar-path * extract code-location->path and add jimfs-backed regression test |
||
|
Nathan Voxland
|
52ba78451d | Added map-tile-server-allowed-networks setting (#81243) | ||
|
Chris Truter
|
3c5d1111d7 | Run project checks in a separate job (#81318) | ||
|
YB
|
72edd1917c |
Bump mariadb-java-client to 2.7.15 (#81456)
bump mariadb-java-client to 2.7.15 |
||
|
YB
|
bb4b24c65f |
Bump httpclient5 to 5.6.4 (#81556)
bump httpclient5 to 5.6.4 |
||
|
Oleksandr Yakushev
|
105262e022 | Bump Toucan2 to 1.0.575 (#81449) | ||
|
YB
|
3866917fba |
Bump jsoup to 1.23.2 (#81457)
bump jsoup to 1.23.2 and pin it in the snowflake driver |
||
|
YB
|
1866aaccd9 |
Bump sshd-core to 2.19.0 (#81461)
bump sshd-core to 2.19.0 |
||
|
YB
|
52fa5626a2 |
Bump httpclient5 to 5.6.3 (#81459)
bump httpclient5 to 5.6.3 and pin it in the clickhouse and databricks drivers |
||
|
YB
|
3ef8df323a |
Bump Bouncy Castle to 1.85.2 (#81287)
bump bouncy castle to 1.85.2 |
||
|
Ben Grabow
|
06f7a3ca8b |
Bgrabow/profile tests workflow (#80258)
* Add workflow dispatch/profiling options to app-db, backend, semantic-search workflows |
||
|
Chris Truter
|
3989cb5eb3 | Standardize nonparallel tests on ^:synchronized (#80064) | ||
|
Ben Grabow
|
ff8e7993d3 |
Internal load-namespaces tool to check for compilation errors (#80009)
* Internal load-namespaces tool to check for compilation errors * Ignore load-namespaces in module checks * lint load-namespaces * Reject positional args * Test the new utility * Guard against nonexistent root. Make the code more testable. |
||
|
Nathan Voxland
|
dbe9a0acc1 | Improve geojson and http requests (#80111) | ||
|
Chris Truter
|
2550ef763e |
Add embedding provider SPI and Arctic embedder plugin (#78310)
* Add a versioned embedding provider SPI Summary - Add a runtime registry for embedding providers with readiness, model resolution, preparation, and batch embedding. - Define immutable embedding-space descriptors and legacy deterministic identities. - Validate provider results, dimensions, counts, and finite numeric vector components. - Make SPI version 1 a compile-time constant; provider artifacts must embed the literal version they were built against. Scope This defines the neutral provider contract only. It does not migrate semantic-search consumers or ship a provider plugin. Testing - Registration and version compatibility. - Identity stability and revision round trips. - Registry snapshot consistency and malformed provider output. * Rename shadowing local in provider/prepare! The local bound from :prepare! shadowed the prepare! var (eastwood local-shadows-var). * Add the Arctic embedder plugin Summary - Build a separate runtime plugin containing the pinned Snowflake Arctic embedding model and DJL/ONNX dependencies. - Bundle arm64 and AVX2 exports, lazy-load native inference, and cap local batches. - Register literal embedding SPI version 1 and derive immutable space identity from the exact model/runtime bundle. - Fail readiness closed for unsupported platforms, musl, and glibc older than 2.34. - Disable DJL network telemetry and revision-address bundle URLs so model upgrades cannot reuse stale cached content. Scope This PR ships and registers the provider but does not yet select it from semantic-search settings. Consumer integration follows upstack. Testing - Model bundle and plugin uberjar build tests. - Catalog, classloader, platform-readiness, identity, cache-key, and batching coverage. - Artifact-only real inference with zero outbound network attempts. * Follow prune-provided-libs! rename in the embedder plugin build * Merge nested let in embedder-plugin artifact test Kondo flagged the inner let as redundant; fold both binding vectors into one. * Deref the now-delayed core provided-libs in the embedder build * Bundle MiniLM for in-process complexity scoring * Activate the embedder plugin in its artifact smoke test * Activate lazy embedding providers on demand * Use plugin initializer in artifact smoke test * Make lazy provider activation best effort * Log lazy embedding provider activation failures * Declare the embeddings plugin dependency * Keep embeddings provider internal * Activate the in-process embedder during bootstrap * Fix embedder validation and optional plugin startup * Declare the embedder plugin as a module The deps graph only scanned src, enterprise/backend/src, and modules/drivers, so the plugin's namespaces belonged to no module and anyone linting them got six unresolved-dependency warnings. Scanning modules/embedder/src too lets the generator record what the plugin actually takes from the host. * Move embedder model caching into the weekly cache generator Review feedback on #78310: - Drop the redundant persist-credentials: false checkout options. - Stop writing the model cache from the CI job. The weekly cache-generator workflow is the single cache writer; the build job now restores only. The downloader is content-addressed (SHA-256 verified, refetches deltas), so a stale weekly cache still protects against HuggingFace rate limits without affecting correctness. * Bundle snowflake-arctic-embed-xs instead of Arctic-L The bundled models are a test tier: they exercise the packaging, DJL runtime, tokenizer, and pooling machinery without shipping the 1.1 GB Arctic-L artifacts. xs keeps the same family, query-prefix scheme, and cls pooling as Arctic-L at 23 MB per architecture. A release-tier build with the full model comes later, when the plugin is published for customers or offline use. At ~90 MB of pinned downloads there is nothing worth caching, so the cache-generator wiring is reverted and the CI job downloads directly. * Fail loudly when a catalog entry lacks this architecture's export resolved-model computed the embedding-space-id from get-in spec [:architectures arch :sha256] without checking it was present. A model missing this architecture's export would silently bake a nil into the identity hash instead of failing, producing a syntactically valid but bogus embedding-space-id. * Pin the embedder plugin manifest name to one constant "Metabase In-Process Embedder" was duplicated in core.clj (the registered?/load-plugin! lookup key) and the artifact test. A rename in modules/embedder/resources/metabase/embedder/metabase-plugin.yaml without updating both call sites would silently disable the plugin, since registered? just returns false for a name it doesn't recognize. * Downgrade the hashed-table-name log line to debug Hashing fires deterministically for every identifier derived from a long provider/model pairing (e.g. "in-process" + the bundled xs model), so it was logging a warning on every lookup rather than once at creation. * Cover the missing-architecture catalog guard The error path added alongside the guard had no test, so the earlier silent nil-hashing behaviour could come back unnoticed. * Assert the reason on the missing-architecture guard The message alone leaves `:reason :architecture-not-bundled` untested, which is the part callers actually branch on. * Drop the hashed-table-name log line It fires from a pure name-derivation helper on every lookup. The table creation it hints at is already logged at info by the caller, and the parameters behind the hash are in index_metadata. * Extract the missing-architecture check out of the smoke test The guard's assertions pushed the smoke test to 155 lines, past the 150-line deftest limit, so Clj-Kondo failed. Pull it into a helper the way the platform-support checks already are. |
||
|
Nicola Mometto
|
af35f7f0af |
Backend hardening (#80003)
Co-authored-by: Alexander Polyankin <alexander.polyankin@metabase.com> Co-authored-by: Alexander Lesnenko <alxnddr@users.noreply.github.com> Co-authored-by: Ben Grabow <ben.grabow@gmail.com> Co-authored-by: Bryan Maass <bryan.maass@gmail.com> Co-authored-by: Dan Sutton <dan@dpsutton.com> Co-authored-by: Eric Normand <ericwnormand@gmail.com> Co-authored-by: Nemanja Glumac <31325167+nemanjaglumac@users.noreply.github.com> Co-authored-by: Timothy Dean <timothy@timothydean.dev> Co-authored-by: Tyler Plude <tsplude@gmail.com> |
||
|
appleby
|
c6e544cb1c |
Gemini Agent Platform: BE support Gemini models (#78193)
* Add llm-google-* settings for the Gemini Enterprise Agent Platform The location, project ID and model become path segments of every request, so each is validated as URL-safe. * Wire the google provider into Metabot provider settings * Add Google Gemini provider via streamGenerateContent API Add new provider supporting Gemini models via the Gemini Enterprise Agent Platform - New ns `metabase.metabot.self.google`. Handles generic provider concerns like - contructing appropriate endpoint urls for the given model and location - validating caller-provided credentials via a countToken call - credentials handling - validating config - remapping upstream errors with improved error messages - toplevel request entrypoint - New ns `metabase.metabot.self.google.stream-generate-content` - handles wire format and aisdk chunk tranlation for Google's streamGenerateContent API * Wire Google credentials into the admin API Optional credential settings that are set by an env var no longer block connect or disconnect. * Skip the provider setting write when shadowed by an env var * Drop the unused llm-bedrock-configured? setting Nothing read it: the frontend never referenced it, and its last backend caller went away when configured-provider-credentials switched to provider-credentials-complete? for the bedrock branch. * Extract the shared tool->function conversion into schema/tool-function The mu/defn doc-stripping + filter-schema-by-features + JSON Schema transform boilerplate was copy-pasted across the claude, openai, chat-completions, and google adapters. * Review feedback: allow settings to be cleared even if env shadowed There was a mismatch where check-not-env-shadowed! allowed nil values but set-setting-unless-env-shadowed! did not. Update set-setting-unless-env-shadowed! to preform nil writes so that a disconnect fully clears the appdb settings even if some are env shadowed. |
||
|
Andrei Simionescu
|
0e70b3eb71 |
BOT-1956: Read the Python library source from :source (#79573)
Change `get_transform_python_library_details` to read from `:source` instead of `:content` to match what the API sends. The code is wrapped in a Markdown fence longer than any backtick run inside it, prefixed with a line telling the model to treat it as data, and cut off at 100k characters with a truncation marker when the library is bigger than that. |
||
|
Nathan Voxland
|
5dbc8db1d2 | Only default warehouse-allowed-networks to external-only on prod (#79672) | ||
|
Nathan Voxland
|
7b58c182a3 | Better management of database metadata (#79181) | ||
|
Kevin Kelleher
|
789d334847 |
Explorations (#73244)
* Fixed failing CI * Add query timestamp, refactor error marker * no-mbql-introspection pass * Order temporal-axis exploration variants by date, not aggregation When the row cap fires on a time series, aggregation-desc ordering keeps the highest-count buckets scattered across the full range: gaps in the x-axis, old spikes beating recent data. Temporal axes now order by the temporal breakout desc (aggregation desc as tiebreak) before the limit, so the most recent contiguous window survives. - time-facet: order by the temporal breakout (second breakout) desc - default: branch on dim type — temporal dims by recency, categorical dims keep aggregation desc - per-value-time-series: gains the same date-desc cap (had none, so a single fine-grained series could blow query-caching-max-kb) - all three also filter the NULL date bucket: it can't render on a time axis, and its position under ORDER BY DESC is warehouse-dependent (Postgres NULLs first, MySQL last), making cap survival non-deterministic otherwise - temporal-pattern-day/hour unchanged (bounded, no cap interaction) * Added debug info to interestingness calculations * even less mbql introspection * oops, delete .scratch * Sidebar UI fixes - error marker, loading shimmer * Interestingness >= 0.7 * Adjust prompts list animation * Fix e2e * Note that the LLM query planner is a POC, not used in production * Fix searching for explorations * Use llm-metabot-provider for everything * Log interestingless calculation as info for now * Log interestingless calculation back as debug * Fix FE lint * fix ai summary showing dirty while generating * Use llm-metabot-provider * collection breadcrumbs * add back null values for temporal variants * order by for temporal pattern variants * fix top-n-other * doh, fix temporal pattern queries * fix name in viz header, don't show be error in tooltip * sort numeric bins in heat map header, fix tests * UXW-4483 Add matching colors for category + breakout charts (#75912) * Fix CI * Fix units, fix loki * UXW-4522 Update last query run timestamp in background * Fix loki * Fix e2e * Improve Explorations viz types (#75958) * UXW-4483 Add matching colors for category + breakout charts * improve viz types first pass * add error boundary * add row chart support for dimension_value_colors * Fix typecheck * fix tests * better handling of segments with row fallback, fix null color bug --------- Co-authored-by: Denis Berezin <denis.berezin@metabase.com> * UXW-4402: Make explorations respect sandboxing/impersonation/db routing (#75572) * Make explorations respect sandboxing/impersonation/db routing * some self-review fixes * more cleanup * test fixes * doh * Fixing CI failures * capitalize 'By' consistently * Fixed CI failure * add loki snapshots * Review fix * Fixes after merge * Rework Explorations UX (#76300) * comment on exploration viz first pass * wip * wip * wip * UX changes first pass finished * code cleanup, fix, add tests * address review comments * Code cleanup --------- Co-authored-by: Nathan Voxland <nathan@metabase.com> * Refine explorations with metabot (#75326) * teach metabot about new exploration ux * refine explorations * address self-cr * pass over FE * CR response * fix test * Add :model/Exploration to api + eid-translation model-imports These modules reference :model/Exploration after merging the latest explorations base; declare the imports to satisfy model boundaries. --------- Co-authored-by: Kevin Kelleher <kevin.kelleher@metabase.com> * Harden explorations runner: - planner crash recovery - Bound timeline scoring - add metrics * UX tweaks * Additional logging of errors on ai summary, plus increased thinking budget * Improved timeline interestingness calculation * Handle sonnet 5 etc better * Make explorations groups a real thing (#76643) * Rename ExplorationThreadGroup model to ExplorationBlock Explorations organize their queries in a hierarchy: an Exploration has Threads (one per run), a Thread has Blocks, and a Block is one metrics-by-dimensions grouping that renders as a single sidebar heading. That persisted grouping entity was named ExplorationThreadGroup, and the word "group" was overloaded across the backend and frontend. Rename the entity to Block — the name the frontend already uses — so the larger data-model changes that follow start from one clear name. No behavior change. Renames :model/ExplorationThreadGroup -> :model/ExplorationBlock and table exploration_thread_group -> exploration_block. - Append a renameTable changeset rather than editing the createTable in place: the original is already applied to running dev DBs, so an in-place edit would fail checksum validation and force a DB rebuild. - Rename the model namespace/file; update all call sites, the require alias (:as block), and insert-blocks!. - Update model registrations (resolution, serdes models, copy, entity-id test) and re-sort the entries. - Refresh the model's docstrings to describe a "block". The planner's :thread-groups ctx key and the read-side group-tree are left untouched here; they belong to later read-side work. * Persist exploration pages with stable ids reconciled across reruns Within a Block, an Exploration's queries are organized into Pages: a Page is the bundle of queries for one (metric, dimension, query_type). Until now a Page existed only as a derived, content-addressed key computed at read time, so anything that wanted to point at a Page had to reference a string that changed whenever the grouping definition changed. This makes a Page a real persisted row whose surrogate id is its identity, so comments (and stars, coming next) can anchor to a Page and survive both a rerun — where queries are deleted and regenerated — and a future re-grouping. Introduce ExplorationPage: the persisted (block, card, dimension, query_type) page within a block. - New exploration_page table + model, FK to exploration_block (cascade) and report_card. Registered in resolution, serdes (excluded), copy, and the entity-id test. - Add exploration_query.page_id (NOT NULL, FK, cascade): the reconciler stamps it on every query, so it's required from the start. group_id stays for now as the block pointer the read-side and deep-link URLs use; dropping it waits for the read-side rework. - Reconcile in insert-plan-rows!, the single insert point for both initial Start and rerun: find-or-create a page per distinct key (stamping page_id), then GC pages left with no queries. reset-thread- for-rerun! is unchanged — pages survive a query delete (cascade is page->query), so find-or-create recovers the same page ids on rerun. - No DB unique index on the key: dimension_id is text (not portably indexable on MySQL) and planning is single-threaded, so uniqueness is enforced by find-or-create and matching is done in Clojure. - Migration changesets appended (the table/group_id changesets are already applied to the running dev db). Tests cover page-id stamping, rerun id-stability, orphan GC, and perms. * Return nested blocks→pages from the read API; drop query.group_id An Exploration Thread's structure is Blocks, each containing its Pages, each Page bundling the ids of its queries. Until now the backend instead returned a hand-rolled flat list of typed nodes (sidebar/singleton/page) linked by parent_group_id and keyed by auto:<...> strings, which the frontend then re-nested. Now that Blocks and Pages are real rows, return the nested shape the tables already encode: GET /:id returns :blocks, each with its :pages, built from ExplorationBlock + ExplorationPage rows and queries grouped by page_id. - New metabase.explorations.blocks (renamed from .groups): blocks-tree builds the nested shape; block headings + page names + score-sorting ported over; block-display-name made public for the LLM planner ctx. - Deep links go to a page by id: page-url → /question/research/<exp>/ page/<page-id>; the AI-summary and document-append builders pass the primary query's page_id. - Drop exploration_query.group_id (and its FK/index); a query reaches its block through its page now — build-row-context (used by the runner to finalize each row's MBQL) resolves the block via the page, which is what the group_id read there used to do. - API schemas: ExplorationQueryGroup → ExplorationBlockNode + ExplorationPageNode. - Migration changesets appended (the table is live on the dev db). - Rewrite the read-model + endpoint tests for the nested shape; every ExplorationQuery test fixture now hangs off a page (page_id NOT NULL). The create payload key stays :groups for now (separate input contract, touches the data-picker flow). FE consumption of :blocks lands next. * Consume nested blocks→pages on the frontend The frontend renders an Exploration Thread as sidebar headings — one per Block — with a navigable item per Page (a Page being the bundle of queries for one metric/dimension/variant). It used to receive a flat typed-node `groups` list and re-nest it itself; this switches it to the nested `blocks` array (block → pages) the read API now returns, and renames the navigable unit from "group" to "page" throughout. Users see the same sidebar and charts — this lines the frontend up with the backend's shape. - exploration.ts: add ExplorationBlockNode / ExplorationPageNode; thread `.groups` → `.blocks`; remove ExplorationQueryGroup / its display-type; the navigable id is now a page node id (kept as an opaque string). - Sidebar utils build the tree directly from blocks→pages (one heading per block, one child per page); score-sorting + empty-block filtering preserved. - The navigable unit is renamed group→page end to end: the route segment becomes /page/<id>, matching the backend's AI-summary deep links. The selection/url/comment-draft id stays a string via String(page.id), so the existing string-keyed plumbing is unchanged. - Visualization toolbar/header/comment-draft props renamed group→page; the mark-as-interesting handler is left a stub for the stars feature. - Specs, stories, and test-utils rewritten to the nested shape. The create-payload request key stays `groups` (separate input contract). * Retain comment-anchored pages across rerun GC; point comment URLs at /page/ A Page bundles the queries for one (metric, dimension, query_type), and a comment can be anchored to a Page — comment.child_target_id holds the page id. On a rerun an Exploration's queries are deleted and regenerated, and any Page left with no queries is garbage-collected. A Page a comment points at must survive that cleanup even when it ends up empty, or the comment would resolve to nothing. This makes the rerun GC comment-aware and moves comment deep-links onto the page route. - gc-orphan-pages! now retains an orphan page when some "exploration"- targeted comment's child_target_id equals the page id (compared as strings). Empty + no comment is still GC'd as before. Page ids are globally-unique PKs, so matching on child_target_id alone is unambiguous. - exploration-comment-url: /group/<child> → /page/<child>, matching the page route and blocks/page-url; stale "group ID" docstring fixed. - explorations imports :model/Comment for the retention lookup (comments doesn't depend on explorations, so no dependency cycle). - No backfill: the dev Postgres has zero exploration comments and zero auto:* child_target_id rows, so the optional migration is skipped. - Test: comment on a page → drop that page's selection → rerun → the page is retained and the comment still resolves; a no-comment empty page in the same rerun is still GC'd. * Generate short and long page names from metric, dimension, and variant A Page — the bundle of queries for one (metric, dimension, query_type) — is shown in the sidebar under its Block's heading, but is also referenced on its own elsewhere (comment deep-links, AI-summary chart titles, search). Page names are derived, not stored, so a Page needs both a short label that omits whatever its Block heading already shows (e.g. under a "Number of Orders" heading: "By Category over time") and a full, self-describing label for context-free places ("Number of Orders by Category over time"). This generates both from the Page's parts — the metric name, the dimension name, and a short per-variant qualifier — rather than by string-slicing a finished name. - variants/variant-qualifier: query_type → a short, param-free localized qualifier ("over time", "(Day of week)", ...) or nil for the default breakdown. Lives next to query-name so all variant naming is in one place. Unit-tested. - blocks: page-long-name and page-short-name generate from parts (metric + dimension + qualifier) using tru templates and the block's anchor; page-node emits :name and :long_name. - API ::ExplorationPageNode and the FE ExplorationPageNode type gain :long_name (FE display wiring left for separate work). - Same-dimension pages that differ only by variant (default vs top-n-other) are now distinctly named instead of colliding; a blocks-tree test that asserted the collision now asserts they differ. * Add ability to star pages, sidebar tabs UX (#76755) * add ability to star, sidebar tabs * - renamed "user interestingness" to "starred" - Let starred just be true|false, no null - Removed DELETE starred API, just PUT now --------- Co-authored-by: Nathan Voxland <nathan@metabase.com> * Fix CI: test compile error, migration boolean type, e2e formatting - documents/search_test.clj: qualify text->prose-mirror-ast with the documents.test-util/ alias in the exploration-thread exclusion test. The unresolved symbol broke backend test compilation, which cascaded to every app-db/driver test job and clj-kondo ("0 tests" everywhere). - 20260428_explorations.yaml: use ${boolean.type} instead of bare boolean for exploration_page.starred (required post-v49 so MySQL uses BIT(1)). - oxfmt formatting for two explorations frontend files (fe-lint). * Update e2e specs for reworked comment URL + explorations sidebar - comment helper: match the node comment link by path only. The `?new=true` marker was removed from comment URLs (base commit "respect query params in all urls"); the CommentsMenu unit test was updated but this e2e helper was missed. Verified the comment flow still works (opening a node's thread shows the new-comment editor). - explorations spec: align with the reworked sidebar — * `by <dim>` leaf label is now `By <dim>` * Scratchpad doc test: only Scratchpad is created synchronously; the AI Summary doc is created asynchronously, so stop asserting it (and the removed "Findings" grouping) at creation time. * AI-summary completion test: the AI Summary doc row now hangs under the thread heading ("Initial investigation"), not a "Findings" group. * remove the two "Add to document" tests — that control was removed. * Fix stylelint rule-empty-line-before in explorations CSS modules Add blank line before nested &:hover rules in ActionToolbar.module.css and ExplorationGroupVisualization.module.css. These surfaced in fe-lint's lint-css step once the oxfmt formatting failure was resolved. * - cleaned up migration file - Improve group renaming in code - preserve starts between re-runs like comments * Fixing CI failures --------- Co-authored-by: Kevin Kelleher <kevin.kelleher@metabase.com> Co-authored-by: Nathan Voxland <nathan@metabase.com> * Don't send empty messages to the LLM * address review comments from 76755 * UXW-4765 Hide charts (#77488) * UXW-4765 Hide charts * Review fixes * Fix the failing explorations unit tests and Loki stories (#77564) * Fix the failing explorations unit tests and Loki stories at the branch tip * Restore the original all-hidden copy and drop thread headings from it * Delete the outdated ExplorationGroupVisualization stories and their Loki references * Rephrase the hide toast to put the chart name first * UXW-4764 Remove interestingness badges on results (#77544) * UXW-4764 Remove interestingness badges on results * Drop e2e assertion for the removed sidebar interestingness markers * Colocate the interesting and error markers with their only consumers * Remove the interestingness marker from the timeline picker too * fix types, lints, formatting * Uxw 4779 update tab menu to match designs (#77543) * UXW-4779 Update sidebar tab menu to match designs * Show a pointer cursor across the whole SegmentedControl * Give the icon tabs img semantics and hide the decorative icon names * Let getExplorationSidebarTabsInfo own the tab icon config * Revert the SegmentedControl root pointer cursor, Mantine already puts it on the labels * Regenerate modules config after master merge The merge left 'explorations' and master's new 'entity-retrieval' out of sorted order, which metabase.core.modules-test asserts on. * Float the action toolbar pinned to the bottom of the chart canvas (#77667) * fix types * revert timelines in documents * Review fixes: stop requiring the QP's internal cache.impl from other modules Explorations code that saves and loads query-result snapshots was requiring metabase.query-processor.middleware.cache.impl directly — an internal namespace of the query-processor module that other modules aren't supposed to reach into. Since the module's :api set is generated from actual usage, those requires had put cache.impl on the module's public surface — something master never exposed. Modules talk to each other through a small public facade (metabase.query-processor.core) instead. The two helpers those callers need — do-with-serialization (write a QP result as a gzipped blob) and with-reducible-deserialized-results (read one back) — are now re-exported from the qp.core facade, and the explorations callers go through it. One consumer can't: queries/cached_result.clj is itself loaded while qp.core is loading, so requiring the facade from there would be a circular-load error at boot. It gets a proper home instead — a new small, documented public namespace, metabase.query-processor.result-serialization, re-exporting the same two helpers. With nothing outside the module requiring cache.impl anymore, the regenerated :api drops it, matching master. * Review fixes: explorations planner and AI-summary polish - Nested cardEmbeds: the AI-summary postprocessor wrapped only top-level cardEmbed nodes in resizeNode containers, while the card materializer walks the whole document — so a cardEmbed the LLM emitted inside e.g. a blockquote got a real card but rendered collapsed to zero height. The wrapper now walks the tree too (skipping already-wrapped nodes). - Awareness-tier sort validation: phase 2 validates that embedded categorical charts carry an explicit sort instruction, but the validation set was built only from top-tier breakouts — an awareness-tier categorical chart escaped validation entirely, and its prompt block (slim-block) never told the model a sort was required. Both tiers are validated now and slim-block emits the same per-chart sort directive as full-block. - Contextual-scoring gate order: safe-score+describe resolved its scoring context (a Card select and a SQL compile) before checking whether the row is scorable at all (chart config present, creator known, non-blank thread prompt) — wasted DB reads and a query compile for every unscorable row. The cheap gates run first now. Its docstring also claimed the LLM gate throws and is swallowed; it actually returns nil from a pre-flight cond. - Discovery-cache TTL: the top-K category discovery cache was a process-lifetime LRU, so a stale top-K result could be served forever on a long-lived instance; entries now also expire after 15 minutes. - Chart-name i18n: filtered-subset chart names built their multi-value summary as (str (count values) " values") — a hardcoded, unpluralized English fragment inside an otherwise-translated name. It uses trun now. - Docstring/comment corrections: effective-cardinality documented (and branched on) a :num-bins bucket that no caller can produce — branch and docs removed; run-one-plan-iteration! said it runs "the LLM planner" when the default is the mechanical one; a variants-test section comment claimed temporal variants add a not-null filter that no variant adds. Also repoints this module's result-blob serialization calls at the qp.core facade (the consumer half of the cache.impl cleanup in this stack's first commit). * Review fixes: explorations API permissions, documents, and search - Creating an exploration accepted arbitrary card ids (in blocks) and timeline ids without checking the caller could read them, so you could attach someone else's private metric or timeline and see its name/events later. Both are read-checked at create time now, and timeline hydration filters out timelines the viewer can't read. A duplicated timeline id used to hit a unique constraint and 500 the request; ids are deduped. - The restart endpoint re-selected the Exploration row after resetting the thread, but the reset never touches that row — the redundant select is dropped. - GET /api/document listed documents that belong to exploration threads, which search, recents, and collection items all deliberately hide. It applies the same filter now — which as a side effect removes an N+1 pattern where each listed doc could trigger several permission queries. - The document validator rejected {:type "doc" :content []} — the exact empty document the backend itself creates for new threads. An empty root doc is valid now. - Search: is_published was deliberately hidden from search responses on master (#66841) and this branch accidentally re-exposed it — hidden again in both search engines. Exploration descriptions are searchable in the legacy engine too, matching the new engine. - Data picker: metrics and dimension groups now use the same interestingness filter (a metric could previously reference dimension ids that appeared in no group), search matches the displayed "<group> - <dimension>" text, and a collection-ids query that ran twice per request runs once. - Consistency: the create event publishes after the transaction commits (observers could previously fire before the data was visible). * Review fixes: metabot / LLM infrastructure - Asking Claude for extended thinking while also forcing a tool call (tool_choice "required") produced a request Anthropic rejects with a 400. The guard that already handled this for structured output now covers the general tools path too. - The e2e guard that stops tests from hitting a real LLM provider threw a raw MalformedURLException on unparseable URLs; it now fails closed with its normal friendly message — and it was moved out of a catch-all that would have rerouted it, so it fails loudly. - The timeline tools handed the LLM broken XML (an attribute quote and a tag never closed) and interpolated user-written names/descriptions unescaped, letting content break the markup. Output is well-formed and escaped. - Hardening/docs: the retryable-error check bounds its exception cause-chain walk at 10 (a cyclic getCause could loop forever); the contextual-interestingness response schema now really forbids extra properties (its docstring claimed it already did); docstrings about thinking model buckets, return shapes, and inputs corrected. - Also repairs three test breaks that already existed at HEAD: an arity mismatch from the OpenRouter merge, a test asserting thinking blocks are dropped (they now stream as reasoning parts), and tests that didn't stub the LLM gate's newer usage/permission checks. * Review fixes: lib-metric, metrics transforms, startup sync, interestingness - group-by-source was re-exported for JVM Clojure but missing from the ClojureScript branch of lib-metric.core; added. - Metric/measure display-info inherited a default :long-display-name even where the method overrode :display-name, so the two could disagree; they're set together now. - A failure in the startup metric-dimension sync aborted server boot entirely; it's wrapped to log an error and continue. - The interestingness "leftovers" backfill loaded every unscored field for the database into memory in one select; it now streams one field at a time, and the namespace docstring mentions this second recompute trigger. - Added a unit test covering the dimensions :out transform (kebab-key renames + type-value keywordization). * Review fixes: enterprise perms/sync/deps, cached card query, test quality - Remote sync: an event payload that simply omitted a key used to satisfy a nil condition (e.g. "exploration_thread_id must be nil") — conditions now require the key to be present with the right value, checked with (find object k), which distinguishes a present-but-nil entry from an absent key. Vector values in removal conditions rendered as broken equality clauses; they render as IN. - Dependency graph: smartLink nodes with nil/placeholder ids added nil to the dependency set (the sibling cardEmbed branch already guarded this); guarded, with regression tests for both branches. - Test quality: the revisions exclusion test asserted the impl against its own private set, so it could never fail; it checks a hardcoded expected list now. Exploration bookmarks get real API coverage (create/list/delete/ordering/archived), and a mislabeled test string was fixed. * Review fixes: v4 instance-analytics views with explorations usage sources The AI-usage analytics views translate machine source names ("exploration", "contextual_interestingness", …) into human-readable labels with a SQL CASE. The explorations stack added its new sources to the v1 SQL files — but the views actually installed on instances are v3, and nothing reinstalled them, so users saw the raw machine names fall through the ELSE branch. Adds v4 of ai_usage_log and metabot_conversations (v3 plus the new CASE branches, with one consistent "Explorations" label — v1 said "Exploration" for the source but "Explorations" for the profile), a migration that reinstalls the views at v4, an updated pointer comment in usage.clj, and a parity test that fails whenever a known source/profile is missing a CASE branch in the latest views — exactly the drift that happened here. * Review fixes: frontend (minimal) and e2e Only the low-risk frontend fixes from the review pass, plus the e2e test fixes; the other frontend findings are deliberately left out to keep this branch's FE diff small. - getDimensionIcon existed byte-for-byte in two files; the duplicate module is deleted and its one importer now uses the copy in common/utils/columns.ts. - e2e: an assertion that searched for text inside another text node (and so always passed) is meaningful now; readSelectedText no longer breaks Cypress's retry behavior by asserting after a .then; and the seemingly pointless GET /api/metric/:id in seedMetrics got a comment explaining the side effect it exists for (it persists metric dimensions). * Remove the LLM query planner Explorations plans queries with the mechanical planner only now. Removed: the LLM planner namespace, its prompt templates (plan.selmer / repair.selmer) and prompt loader, the prompt-markdown renderers in the planner-context namespace, the explorations-query-planner setting (mechanical/llm/auto), and the :skip-no-llm exit for "planner requires an LLM but none is configured". pick-planner! now always returns the mechanical planner — kept as a seam so tests can stub it and a future planner has an obvious dispatch point. The planner-contract protocol and the shared context builder stay (they are planner-agnostic), and the query_plan_transcript column stays (every planner writes a transcript); its remark no longer describes an LLM exchange. * remove graph._dimension_value_colors (per bar colors) * Add row count to explorations API (#77740) Already available, we just need to persist it and thread it through the API. * Format fix * Exploration UX... um explorations. (#76901) * click thru * better comment links * chart avatars * avatars * allow subsubexplorations * disable explore further for Other group in top n charts and dimension anchored blocks * fix explore further for binned numeric columns * comment highlighting, remove avatars * comment label formatting * map and table highlighting * use built in viz click actions popover instead of custom * fix comment badge in dark mode * self review cleanup * fix and simplify new thread toast * fix be lints, add tests * fix tests * add fe, e2e tests * - POST /:id/explore-further now verifies the clicked page belongs to this exploration before copying its block, closing a hole where any page in the instance could be copied into any writable exploration. - key on the stable :explore-filters spec instead of a hash of the per-row-reconstructed :dataset_query (which minted fresh :lib/uuids and so never hit). build-row-context exposes :explore-filters on the row context for the cache to key on. Also reuse metabase.util/capitalize-first-char in the block title and thread name helpers instead of hand-rolled str/upper-case. * Code cleanup * Gate exploration derived metadata by viewer's permissions * fix lints * Code review improvements * Den/uxw 4741 explorations new sidebar (#77674) * Update tabs view, add filter button menu * Fix tests * Add sidebar items vertical lines + icons * Other sidebar changes - viewed state, sticky hierarchy * Show sub-explorations near it's parent item * Fix issues * Self review * Fixes after merge * Self review * Review fixes * Fix empty states * Review fixes --------- Co-authored-by: John Swanson <john.swanson@metabase.com> Co-authored-by: Kevin Kelleher <kevin.kelleher@metabase.com> Co-authored-by: Nathan Voxland <nathan@metabase.com> Co-authored-by: Denis Berezin <denis.berezin@metabase.com> * Explorations self-review cleanup (#77717) * Consolidate the explorations migration in place The 063 explorations migration had grown by accretion: tables were created and then altered by later changesets in the same unreleased file. Since the branch convention is to edit unmerged changesets in place, the final shapes are folded into the createTables: - exploration_query gains params, nullable dataset_query/entity_id, and page_id inline (the page_id FK and index stay separate — exploration_page is created later in the file, so they can't run earlier). - stored_result gains data_access_token inline; its backfill changeset was deleted outright — created and backfilled in the same release, it could only ever run against an empty table. - exploration_query_result gains chart_stats / metric_description / chart_description inline, which also retires the randomly-named changeset id v63.gr324asd. - idx_eqti_query_id deleted: the unique constraint on the same leading column already provides that index. The completed_at and analysis_started_at backfills are deleted outright: the explorations feature merges to master in one piece, so at upgrade time no instance can have pre-existing exploration threads to backfill. (They were also dangerous — they stamped columns the cancel endpoint and the completion handler gate on being NULL, permanently locking out any thread in flight during the migration.) Cosmetics: backfilled default docs are named "Scratchpad" to match runtime (the backfill said "Findings"); stale remarks corrected (chart_stats is EDN not JSON, the real query_type variant list, what collection_id actually defaults to). Existing dev databases will need a migration reset. * Set exploration thread started_at in the INSERT The thread was inserted without started_at and then immediately updated to set it, on the theory that the background worker must not see the thread before its dependent document/block/timeline rows exist. The whole create shares one transaction, so intra-transaction ordering is invisible to the worker (it reads on another connection; the commit is atomic) — the flag can simply be set in the INSERT, saving a write. * Guard exploration restart against in-flight work The restart endpoint could fire while the planner was still working on the thread, deleting its rows mid-plan and wedging the thread in a state no worker would ever pick up again. reset-thread-for-rerun! is now a CAS: it applies only to a terminal thread (completed_at set) with no query still mid-execution, and the endpoint returns 409 ("cancel it before restarting") otherwise. A successful reset leaves the thread exactly in the shape the planning worker's claim predicate picks up. The endpoint also published :event/exploration-update, which provably did nothing: restart changes no Exploration column, and the revision system skips unchanged objects. The publish is removed; the AI-summary doc reset still records a Document revision, which is what /mine's last-touched ordering composes. * Route mid-stream LLM errors through the provider-friendly rethrow LLM responses stream lazily; a connection timeout between SSE chunks escaped as a raw SocketTimeoutException because the provider error handling only wrapped request setup — the reduction happens later, in the agent loop. Every provider adapter's SSE reducible (claude, openai, openrouter, azure, bedrock) is now wrapped so consumption-time failures get the same translation as request-time errors. * Grant the read-only timelines scope under metabot-nlq The explorations metabot profile instructs the model to use the timeline tools, but those tools were granted only under :permission/metabot-other-tools while the profile is gated on NLQ — so for NLQ-only users they were silently filtered out of scope while the prompt kept referring to them. agent:timelines:* (whose only member is the read-only list/details scope) is now granted under both permissions. * Don't retry deterministically-failing fields in the interestingness leftovers pass The leftovers pass selects on dimension_interestingness IS NULL, so a field whose scoring deterministically throws was re-attempted on every sync forever. A per-process attempted-marker now skips fields that already failed in this process; a restart re-enables retry, so transient failures still converge. Process-local by design — no schema change, and no sentinel score leaking into product surfaces that read the column as a number. * Fail closed when a sandboxed field's user attributes are missing sandbox-token-for-table returned nil when the field was sandboxed but the user's sandbox-attribute lookup came up empty — and the downstream compatibility gate reads nil as "unrestricted". That's fail-open. It now returns a user-scoped "indeterminate" sentinel that matches nobody else's token (two indeterminate users don't match each other either): fail-closed. Also adds direct and end-to-end tests for the impersonation and routing token producers, which previously had none. * Require the (card, stored_result) pairing on the cached card-query path POST /api/card/:id/query with a stored_result_id read-checked the card but then served whatever snapshot id the client sent — nothing verified the snapshot actually belonged to that card. The pairing must now exist in stored_result_use (404 when it doesn't, so an unpaired probe can't confirm a snapshot exists). Both materialization paths already insert the pairing rows, and the table ships in the same unreleased release, so no legacy data lacks them. The viewer-lens check also denied a nil dataset_query only by accident (a schema exception swallowed by the catch — and in prod, where mu/defn logs instead of throwing, it would have failed open); it denies explicitly now. Plus: consistent t2/select-one forms in the endpoint, a stored_result_use docstring update, and the generated module config picking up the new StoredResultUse consumption. * Apply the personal-collection default when it resolves after first render The "default new explorations to my personal collection" logic captured personalCollectionId once in a useState initializer — if the selector resolved after the first render, the default never applied. It now reconciles when the id arrives, without overriding an explicit user choice. * Move the explorations migrations to v64 The v63 release branch has been cut, so the explorations feature will ship in v64 and its migrations belong in the 064 directory. Both files move (20260428_explorations.yaml and 20260519_exploration_bookmark.yaml) and every changeset ID is reprefixed v63. -> v64. No changeset content changes. Since we're already editing unreleased changesets in place on this branch, existing dev databases needed a migration reset anyway. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * v_metabot_conversations v4: restore include-deleted semantics lost in merge Master's #77542 (conversation branches) and this branch both created metabot_conversations/v4 in parallel: master's dropped the deleted_at IS NULL filters so soft-deleted (rewound/regenerated) messages are counted; ours added CASE branches for the new explorations usage sources and profile. The 'Merged from master' resolution kept only our copy, silently losing the deleted-message semantics — which is why deleted-messages-included-test and last-message-at-test fail: they assert master's behavior against our view. This drops the three filter lines from each dialect's v4 file so v4 is the union of both changes. Both installing changesets (our v63 and master's v64) are runOnChange: true and checksum the referenced file, so every instance reinstalls the corrected view on upgrade — no new version or migration needed. * Drop the redundant v_metabot_conversations v4 installer Master's #77542 (064/20260713_metabot_analytics_views_include_deleted.yaml) already installs metabot_conversations v4 via a runOnChange changeset against the same SQL files, which now carry both sides' changes: the explorations CASE branches and the include-deleted semantics. A second installer here was redundant — and would have collided with master's changeset IDs (v64.2026-07-13T00:00:00/01) had this file been mechanically renumbered to v64 like the other explorations migrations. The ai_usage_log v4 changeset stays: nothing on master installs that view. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Move the ai_usage_log v4 migration to 064 Same reasoning as the explorations tables: v63 is cut, this ships in v64. The changeset ID takes the next free slot for its date (v64.2026-07-13T00:00:02 — master's include-deleted file owns 00 and 01) rather than a mechanical v63->v64 rewrite that would have collided. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> * Show a crossed-eye indicator with a tooltip on hidden pages in the exploration sidebar (#77907) * Explorations no AI Summary (#77878) * Extract AI Summary + document-in-exploration out of explorations Removes the LLM-generated "AI Summary" feature and the ability to display a document within an exploration. The feature-restoring (inverse) commit is kept on branch `explorations-ai-summary`; cherry-pick it to bring the functionality back. Backend: - Delete the explorations.ai-summary.* pipeline, composite, document-perms, prompt templates, and their tests - Drop exploration->Document creation, the /thread/:id/documents endpoints, the :documents hydrate, ai_summary_document_id, and the AI-summary reset - Revert the documents-module coupling: content-visibility gate, the document.exploration_thread_id FK/column, the view-log skip, and the GET /document list filter; delete documents/validate.clj - Drop the collection-items show_exploration_documents option and the exploration-doc exclusions in recents / search / serdes / remote-sync; drop the search exploration-thread-id attribute - Remove the ai_summary_document_id and document.exploration_thread_id migration changesets (edited in place) Frontend: - Delete the ExplorationDocument component and its tests - Remove the AI-summary sidebar node, completion toast, and the document API / urls / types; keep the shared comments sidesheet and page routing * Extract static card-embed in documents out of explorations Removes the static (cached-snapshot) card-embed capability from documents, so it can land on its own branch (`static-embed`) independently of the AI-Summary feature. The feature-restoring (inverse) commit lives on `static-embed`. Keeps the base `stored_result` model, the exploration query-result cache, and the cached-result permission helpers (`assert-can-view-cached-result!`) that the exploration `/query/:id` streaming endpoint uses. Also keeps the general `report_card.document_id` "card in a document" feature (it predates this work). Removed: - The cached-result serve path (`cached-dataset`, `apply-sort`, `deserialize-cached-result`, `allowed-chart-sorts`, `viewer-can-view-cached-result?`) and the `stored_result_id`/`sort` branch of `POST /api/card/:id/query` - `stored_result_use.card_id` (column + model arm; keeps `exploration_id`) - The documents serdes static-node filter (reverted to master's live-only rewrite) and the prose-mirror static docstrings - Frontend: the tiptap `cardEmbed` static rendering (the `stored_result_id` + `sort` + `chart_href` trio) across the CardEmbed node/menu, document hooks, redux, and `card.ts`; deleted the CardEmbedNode static unit spec * Fix pre-existing cljfmt/oxfmt formatting to unblock CI These 4 files (a stray blank line each) are unchanged from the base explorations branch — the format checks were failing on inherited, not extraction, changes. * more documents cleanup * Update Loki Snapshots * change references from entity (which was page or document) to just page --------- Co-authored-by: Kevin Kelleher <kevin.kelleher@metabase.com> Co-authored-by: Metabase Automation <github-automation@metabase.com> * Restore fix * Fix Metabot exploration tools streaming empty results to the client (#77990) Since |
||
|
Oleksandr Yakushev
|
37fccf450c | Bump Kondo version (#78647) | ||
|
Oleksandr Yakushev
|
5918cc9a6b | [test] Optimize modules-test (#77065) | ||
|
Nemanja Glumac
|
4f4264a011 |
Bump hawk to 1.0.18 (#79070)
* Bump `hawk` to `1.0.18`
This corresponds to
|
||
|
dpsutton
|
0ed95e05f6 |
Unshared appdb connections + detached audit cluster lock (fix first-boot stall) (#78276)
* Add mdb/with-unshared-connection: appdb connections exempt from ambient reuse A connection holding fragile long-lived state -- a streaming result set (postgres portal/cursor), or FOR UPDATE row locks -- must not be picked up by ambient connection reuse: a borrower that inherits it via *current-connectable* and commits destroys that state. This is the failure mode behind the first-boot hang in #78238, the revert (#77048) of appdb streaming (#76645), and "cursor/portal does not exist" errors when field updates run mid-sync-stream. Built on toucan2's new unshared-connection! (camsaul/toucan2#212): the connection is used when passed explicitly -- (t2/reducible-query conn ...), (t2/with-transaction [t conn] ...) -- but never bound as *current-connectable*, so toucan calls that resolve their connection ambiently run (and commit) on other pooled connections. mdb/with-unshared-connection is checkout -> mark -> body -> close; in every other respect the connection is ordinary JDBC, configured by the body (e.g. .setAutoCommit false to stream). The pg test reproduces the exact #77048 failure on an unmarked control connection (portal killed by a nested commit mid-reduction) and shows the unshared connection streaming to completion. deps.edn temporarily pins toucan2 to the PR sha via :git/sha; a follow-up commit will switch back to :mvn/version once a toucan2 release includes it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G7FfdpzW5pr7CX9aNwa98f * Use released toucan2 1.0.572 for unshared-connection support camsaul/toucan2#212 merged and released; drop the :git/sha pin. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G7FfdpzW5pr7CX9aNwa98f * Hold the audit DB cluster lock on a detached connection (fixes boot stall) Since #76551 the audit install/load/sync pipeline runs inside a cluster lock whose SELECT FOR UPDATE ties lock lifetime to transaction lifetime: the whole pipeline became one long transaction on the lock's connection. The serdes load's ~3,600 per-entity nested transactions piled up as savepoints and subtransaction XIDs, and every row lock was held until a single commit at the end -- stalling first boot for 10-25+ minutes and blocking concurrent work like config-file database provisioning (#78238). with-cluster-lock now supports :detached? -- the lock rows are held on a dedicated unshared connection (toucan2 unshared-connection!) while the body runs on ordinary pooled connections, committing incrementally in its own short transactions. Locks release at body completion (commit) or throw (pool check-in rollback). Re-acquiring a lock this thread already holds detached is a no-op in both detached and transactional form; upgrading a detached :share hold to :exclusive throws instead of self-deadlocking; :retry-transient? is rejected (a re-run would double-apply incrementally-committed work). The audit pipeline opts in: per-entity serdes isolation (#74412) is restored, and a crash mid-pipeline is repaired by the next boot (the checksum only advances on completion; reconcile runs every boot), which is what makes incremental commit safe here. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G7FfdpzW5pr7CX9aNwa98f * Address review: confine detached retry to acquisition; repair dialect-sync window Review findings (galdre), all majors addressed: - Major 1: retry/with-retry wrapped the whole detached body, so a retryable-looking error thrown by the body (e.g. a nested cluster lock timing out inside the serdes load) re-ran an incrementally-committed body and then surfaced as "Failed to obtain cluster lock", which ensure-audit-db-installed! swallows as benign contention. An acquired? volatile now confines both the retry predicate and the acquisition- failure wrapper to the pre-acquisition phase in detached mode; body errors propagate raw with intact cause chains. - Major 2: a failed or killed engine-changed dialect sync was never repaired once the content checksum had advanced. New durable setting audit-db-dialect-sync-pending: set before the sync attempt, cleared on success, a third sync trigger on every boot. - Major 3: the mysql/h2 engine-flip visibility window under detached mode is documented as a KNOWN WINDOW at adjust-audit-db-to-source! (restores pre-#76551 semantics; overlay-based resolution is follow-up). - Majors 4-6: reviewer's contract tests adopted (throw-path lock release, check-in rollback, behavioral pin on :detached?, crash-mid- load self-heal, plus the two Major-1 reproductions); public docstrings updated to describe all three lock modes and the upgrade throw. Minors: unified cross-mode reentrancy registry with eager share-> exclusive upgrade validation (fail-fast in any detached interleaving; legal same-transaction upgrade preserved); h2 read->write upgrade throws instead of hanging; lock-row INSERT gets a query timeout (bounds the N-node cold-start pile-up); install-database! and each reconcile group are single transactions; unshared checkout goes through *application-db* so the restore-time read-lock gate applies; registry conveyance and explicit-tx-on-unshared-connection caveats documented; dual kondo lint-as entries; test hygiene (finally cleanup, nil ambient contract). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G7FfdpzW5pr7CX9aNwa98f * Right-size the engine-flip visibility comment The committed, briefly-visible engine flip is this pipeline's longstanding behavior; only #76551's single-transaction era hid it for one release. A neutral note fits better than a warning block. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G7FfdpzW5pr7CX9aNwa98f * Tighten comments to constraints; drop review-context narration Comments now state what the code can't show (crash windows, why raw JDBC, why the marker precedes the sync) and drop justification written for reviewers: issue-number history, caller name-drops, repeated tellings of the retry-confinement story, and cost reassurances. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G7FfdpzW5pr7CX9aNwa98f * Make the detached lock a bespoke entry point instead of a with-cluster-lock mode Reviewer pressure on the :detached? option had generalized with-cluster-lock into a two-mode API whose cross-mode interleavings all needed guarding: a held-locks registry, mode-upgrade validation, an h2 upgrade guard, and an acquired? volatile threading retry confinement through the shared machinery — all protecting call patterns nothing in the tree performs. with-detached-cluster-lock is now its own function: exclusive-only, single lock, documented non-reentrant. Structure replaces machinery — the body sits outside the acquisition retry, so body errors are never retried nor rewrapped as acquisition failure (no volatile), and same-name re-entry throws via a two-line dynamic-set guard instead of a registry (nested *different* locks, which the audit body does take, work unchanged). with-cluster-lock returns to its previous shape; the lock-row INSERT keeps its new query timeout, which bounds the multi-node cold-start pile-up in both modes. Error shape is unchanged, so audit-lock-contention? still classifies correctly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G7FfdpzW5pr7CX9aNwa98f --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
da
|
5cfbe5423c | Update pdfbox version from 3.0.5 to 3.0.8 (#78230) | ||
|
da
|
43d61f4514 |
Update Jetty dependencies to version 12.1.11 (#77860)
https://github.com/jetty/jetty.project/releases check 12.1.7...12.1.11 |
||
|
Timofey Kachalov
|
7cc5a515f1 | Update log4j-* versions (#77808) | ||
|
Nathan Voxland
|
b3cfb8d510 | Persistent Queue (#76352) | ||
|
Timofey Kachalov
|
31a0379d5d | Update jackson.* dependencies to 2.21.5 (#77427) | ||
|
Timothy S. Dean
|
f57fdc8258 | Bump postgresql 42.7.11 -> 42.7.12 (#77181) | ||
|
Timofey Kachalov
|
01fa952fa7 | Update httpcore5, httpcore5-h2 and httpclient5 versions (#77109) | ||
|
Timothy S. Dean
|
d9ea875545 |
Bump handlebars 4.3.1 -> 4.5.3 (#77134)
* Bump handlebars 4.3.1 -> 4.5.3 * Add license for jnack/handlebars |
||
|
Timofey Kachalov
|
eac68e3fe9 | Update c3p0 version (#77000) | ||
|
Oleksandr Yakushev
|
f8cb4ca1bf | Bump kixi.stats to 0.5.9 (#76789) | ||
|
Mahatthana (Kelvin) Nomsawadi
|
6df842cd19 |
fix: pin opentelemetry-api to 1.62.0 (#76531)
* fix: pin opentelemetry-api to 1.62.0 * fix: move io.opentelemetry pin to correct alphabetical position in deps.edn * fix: correct OTel comment attribution, fix stale sdk-testing comment, bump bigquery netty to 4.2.15 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EBYTJTjQQY4qo5EBCt5YwH --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
Mahatthana (Kelvin) Nomsawadi
|
31034d8487 |
fix: bump jackson-databind to 2.21.4 (#76529)
* fix: bump jackson-databind to 2.21.4 and handlebars to 4.5.2 * fix: bypass handlebars 4.4+ root-classpath loader check * fix: revert handlebars bump; keep only jackson-databind 2.21.4 * fix: bump jackson-core to 2.21.4 to match jackson-databind Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EBYTJTjQQY4qo5EBCt5YwH --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
Ben Grabow
|
ebb8e4838b | Replace H2 with SQLite for sample data | ||
|
Braden Shepherdson
|
57991c5b67 |
[subs] Generate PDFs for dashboard subscriptions on the BE (#75205)
The PDFs are laid out on either A4 or US Letter paper, portrait orientation, following the size and positioning of cards on the dashboard itself.
This is a nearly complete rendering of the dashboard, akin to what you see in the brower:
- Directly building a PDF in memory with Apache PDFBox, a low-level Java library.
- All chart types supported by static viz
- Recently expanded to include maps and pivot tables!
- Markdown rendering, including code blocks, clickable links etc.
- Unicode support (Europe, Cyrillic, Greek, Hebrew, Arabic, Hindi, CJK and others) including RTL
- iframe cards have their target (e.g. a YouTube video) rendered as a link
- Dashboard is run as the requesting user, and can have parameter values supplied
- Parameters on dashboard and individual cards are shown with their values.
- Solid API endpoint that allows for parameter values and directly responds with the PDF.
- Reusing query results when sending charts via email or Slack along with a PDF.
---------
Co-authored-by: Nick Fitzpatrick <nick@metabase.com>
|
||
|
bryan
|
fdf5e72f54 | dev.module-viz: interactive modules graph visualizer (#75256) | ||
|
dpsutton
|
718c730aa9 |
downgrade apache sshd-core (#76104)
Why? Because 3.0.0-M2 leaks connections.
Revert mina-sshd 3.0.0-M2 → 2.16 to stop SSH channel leak OOMs
3.0.0-M2 leaks one KexListener per SSH port-forward channel, never
removed on channel close. Each Metabase DB query over an SSH tunnel
opens and closes a `direct-tcpip` channel, so the listener list grows
unbounded for the life of the (long-lived) SSH connection.
Heap dump from a crashed prod instance (highway):
- 1 SSH connection, but 67,221 closed TcpipClientChannel /
Nio2Session / socket-channel objects retained.
- KexFilter.listeners (a session-scoped CopyOnWriteArrayList) retained
572 MB ≈ 38% of heap — full of AbstractChannel$$Lambda instances,
each capturing one closed channel.
Root cause in mina-sshd 3.x (dev_3.0 branch):
- AbstractChannel.init() registers an anonymous KexListener on the
session for every channel and never removes it on close.
- AbstractSession.removeKexListener() is also broken: it calls
addKexListener() instead of remove (copy-paste bug), so even a
correct removal attempt adds a duplicate.
The 2.x lineage has no KexFilter / addKexListener / per-channel KEX
listener — the filter-chain mechanism is entirely new in 3.x — so the
leak cannot occur on 2.16.
|
||
|
Cam Saul
|
4eb1408f48 |
Remove legacy :druid driver / migrate to :druid-jdbc (#75143)
* Remove legacy `:druid` driver / migrate to `:druid-jdbc` * Add no-op migration rollback * Update e2e tests * Remove additional places mentioning removed Druid driver * Kondo * Update modules config |
||
|
Cam Saul
|
cc84d93477 |
Enable cljfmt fixing comment alignment (#74613)
* Enable cljfmt fixing comment alignment * Align bb cljfmt version * Revert cljfmt version bump for now * Update with latest master |
||
|
YB
|
74a33038ba |
GDGT-2076 add observability to incremental transforms (#73364)
Incremental transforms lack visibility into their checkpoint-based behavior. Adding these would help monitor data freshness and efficiency: * Add transform/incremental boolean attribute to existing tracing spans * I'm also adding transform/first-incremental-run to differentiate what happens to the target (first runs are drop-and-recreate rather than append) * Add transform/checkpoint-lo and transform/checkpoint-hi span attributes when incremental Also extracted `first-incremental-run?` which was a predicate we have been using in-line in a couple of other callsites but had not formalized as a function. **New public API: `metabase.tracing.core/add-span-attrs!`**: Enriches the *active* `with-span` body's span with attrs that are only known mid-execution (e.g. `checkpoint-lo`/`hi` comes from `get-source-range-params`, which runs inside the span body). To prevent silent trace pollution, **re-writing a key already on the span is a duplicate** is considered an error. |
||
|
Riley Thompson
|
50e36cc6d3 |
Pin com.google.code.gson/gson dependency at root deps.edn (#74220)
pin gson dep at root |