Compare commits

...

15 Commits

Author SHA1 Message Date
Nicola Mometto
3ae1585c7e bump 2026-07-17 09:40:23 +01:00
Nicola Mometto
b0b11dca91 test seccomp 2026-07-17 09:39:53 +01:00
Oleh K
8984233763 Align transform job header to fix tab layout shift (#77803)
* Fix layout shift when switching tabs on the transform job page

The job Overview and Run history pages each rendered their own header:
Overview used an editable name input plus a `...` menu, while Run history
used a static title with no menu. The differing heights shifted the tab
row, so the clicked tab jumped when switching between the two.

Extract a shared JobHeader component (and a useJobHeaderState hook for the
readOnly/name-change logic) rendered identically by both pages, mirroring
the MetricHeader pattern. Run history now also shows the `...` menu and the
editable name, and Metabot is consistent across both tabs.

* Unify JobPage header logic with useJobHeaderState

JobPage duplicated the readOnly computation and name-change handler that
now live in useJobHeaderState. Source them from the hook instead; keep the
local useTransformPermissions call only for the page-level loading gate.

* Use strict undefined comparisons in job header code

Address review: job/jobId are never null, so compare with undefined
strictly and derive jobId from job.id in the JobHeader render.
2026-07-17 08:51:59 +02:00
Egor Grushin
8ab2269f04 GDGT-2767 Replace deprecated MaybeLink (#77908)
* Replace deprecated MaybeLink

* refactor
2026-07-17 03:12:48 +01:00
Kevin Kelleher
4385d33640 Fix failing test introduced by #77981 (#78000)
fix test
2026-07-17 01:03:20 +00:00
bryan
64588d02fb test(mfa): auth-mechanism matrix + lockout-corner coverage (#77624) 2026-07-16 22:11:37 +00:00
github-automation-metabase
20b1d51e6b Auto-update documentation for master (#77997)
Update generated docs (master)

Co-authored-by: github-actions <github-actions@github.com>
2026-07-16 17:41:49 -04:00
Michael Eby
838d5a7a50 docs: add data isolation content (#77876)
* docs: add data isolation content

* Optimized 4 images, saved 70.0 KB (15.8% smaller)

* Fix links

* Edits from Alex

---------

Co-authored-by: Metabase Automation <github-automation@metabase.com>
2026-07-16 16:47:42 -04:00
dpsutton
5fb83b02e7 Fix MCP OAuth consent flow when Metabase is hosted under a subpath (#77765)
Fixes GIT-10551: connecting an MCP client to a Metabase instance served
under a subpath (site-url like https://example.com/metabase behind a
prefix-stripping reverse proxy) failed at the consent step.

The OAuth flow already derives every advertised URL from site-url —
discovery metadata, authorization/token/registration endpoints, and the
WWW-Authenticate resource_metadata header are all subpath-correct. But
three browser-facing pieces assumed Metabase lives at the domain root:

1. The consent form posted to root-relative /oauth/authorize/decision,
   so the browser sent the POST outside the subpath and the proxy
   returned 404 (the reported bug). The form action is now the absolute
   site-url-based URL, via the same absolute-url helper the page already
   uses for the logo. The bundled @font-face URLs on the page get the
   same treatment.

2. The CSRF cookie was scoped to Path=/oauth/authorize. Under a subpath
   the consent page lives at /metabase/oauth/authorize, so the browser
   would never send the cookie back and the decision POST would fail
   CSRF validation even with the form action fixed. The cookie path now
   carries the path component of site-url.

3. The logged-out flow: the backend bounces to the login page with a
   basename-relative redirect target (the server sits behind the
   prefix-stripping proxy and never sees the subpath). The frontend
   guard resolved that target against the bare origin, losing the
   subpath on the full-page redirect back to /oauth/authorize. The
   guard now normalizes targets through the router basename in both
   directions: paths without an explicit origin get the basename joined
   (browser-real href), and pathnames get it stripped (router path, so
   the backend-only-path prefix check still matches).

Why this is safe:

- Root deployments are bit-for-bit unchanged. site-url has no path
  component there, so the cookie path is still /oauth/authorize, the
  form action still targets /oauth/authorize/decision, and with an
  empty basename the guard resolves every input shape (relative path,
  bare path, absolute URL) to exactly the URLs it produced before.
- Under a subpath, the only outputs that change are ones that were
  previously unroutable (404 through the proxy or a path the SPA
  router would double-prefix).
- The server never sees the subpath, so no routes, parameters, or
  validation logic change; every fix is to a browser-visible artifact
  derived from site-url.

Tests pin both halves: existing root-deployment assertions are
untouched and still pass, and new tests cover the subpath form action,
cookie path, login redirect target, and the full consent -> decision
round-trip with a subpath site-url. Also verified live against a local
prefix-stripping Caddy proxy with a real MCP client flow (DCR, consent,
PKCE token exchange, tool calls).

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-16 15:06:24 -05:00
Rodrigo López Dato
31b35fb452 Automatically focus new Key field when adding user attributes (#77715)
* Automatically focus new Key field when adding user attributes

* Only focus new entry if no error
2026-07-16 16:55:13 -03:00
Anderson
f82bdbdd52 Restore table color range saturation for single conditional formatting rules (#77981)
Restore color range saturation for single rules
2026-07-16 16:41:02 -03:00
Alexander Polyankin
2819934fdc Unify image-buffer pool (#77836)
* Rewrite image-buffer pool on dirigiste (same pool as the GraalVM contexts)

Replace the hand-rolled ConcurrentLinkedDeque pool, stats counters, sweeper
daemon, and logging with the dirigiste Pool already used for static-viz
contexts, with the same semantics: 0-3 pooled int[] arrays of the fixed
oversized size, held exclusively per render, idle arrays dropped after ~10
minutes. acquire blocks until an array is free when the requested size fits;
larger requests get a fresh unpooled BufferedImage.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Extract the dirigiste pool factory into metabase.util.pool

create-pool was duplicated verbatim between the static-viz graal renderer
(channel.render.js.common) and the sqlglot graal runner (sql-parsing.common),
and image-buffer had just picked up a third use. Move it to metabase.util.pool
and use it from all three; the pool behavior tests move from
sql-parsing.common-test to util.pool-test with it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Add prometheus counters for image-buffer pool usage

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Rename image-buffer metrics to metabase-notification namespace

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-16 15:40:38 -04:00
Timofey Kachalov
ce264b2c85 Update SDK version to 0.64.0-alpha (#77978) 2026-07-16 18:33:44 +00:00
lbrdnk
07f62f33c3 Log suggested prompts generation exception on debug only (#77962)
Log exception on debug only
2026-07-16 18:15:54 +00:00
Andrei Chernykh
adf574b57b Convert Scalar visualization from a class to a function component (#77774) 2026-07-16 11:00:04 -07:00
61 changed files with 1337 additions and 1030 deletions

View File

@@ -19,7 +19,8 @@ while [[ $# -gt 0 ]]; do
done
echo "Cloning python-runner repository..."
git clone --depth 1 https://${METABASE_AUTOMATION_USER_TOKEN}@github.com/metabase/python-runner-container.git python-runner
git clone --depth 1 https://${METABASE_AUTOMATION_USER_TOKEN}@github.com/metabase/python-runner-container.git --branch seccomp python-runner
if [ "$SKIP_START" = false ]; then
echo "Starting python-runner..."

82
docs/api.json generated
View File

@@ -5033,7 +5033,13 @@
}
},
"required" : [ "channel_type", "details" ]
}, { } ]
}, {
"type" : "object",
"properties" : {
"channel_type" : { }
},
"required" : [ "channel_type" ]
} ]
},
"metabase.channel.models.channel.ChannelTemplateEmailDetails" : {
"oneOf" : [ {
@@ -5102,7 +5108,13 @@
}
},
"required" : [ "channel_type", "details" ]
}, { } ]
}, {
"type" : "object",
"properties" : {
"channel_type" : { }
},
"required" : [ "channel_type" ]
} ]
},
"metabase.collections-rest.api.DashboardQuestionCandidate" : {
"type" : "object",
@@ -37214,7 +37226,7 @@
"post" : {
"operationId" : "post-api-measure",
"summary" : "POST /api/measure",
"description" : "Create a new `Measure`.",
"description" : "Create a new `Measure`. The Measure's table is derived from its `definition`.",
"parameters" : [ ],
"responses" : {
"2XX" : {
@@ -37255,14 +37267,9 @@
"name" : {
"type" : "string",
"minLength" : 1
},
"table_id" : {
"description" : "value must be an integer greater than zero.",
"type" : "integer",
"minimum" : 1
}
},
"required" : [ "name", "table_id", "definition" ]
"required" : [ "name", "definition" ]
}
}
}
@@ -44356,7 +44363,7 @@
"get" : {
"operationId" : "get-api-search",
"summary" : "GET /api/search",
"description" : "Search for items in Metabase.\n For the list of supported models, check [[metabase.search.config/all-models]].\n\n `context` identifies the surface issuing the search; it selects the ranking weights and filter defaults.\n It defaults to `api`, the value for programmatic callers.\n\n Filters:\n - `archived`: set to true to search archived items only, default is false\n - `table_db_id`: search for tables, cards, and models of a certain DB\n - `models`: only search for items of specific models. If not provided, search for all models\n - `filter_items_in_personal_collection`: only search for items in personal collections\n - `created_at`: search for items created at a specific timestamp\n - `created_by`: search for items created by a specific user\n - `last_edited_at`: search for items last edited at a specific timestamp\n - `last_edited_by`: search for items last edited by a specific user\n - `search_native_query`: set to true to search the content of native queries\n - `vector_search_strategy`: for the semantic engine: `hnsw` (approximate index search, default),\n `brute-force` (exact filter-first search), or `hnsw-iterative-relaxed` / `hnsw-iterative-strict`\n (index-backed iterative scans with inline filters); ignored by other engines\n - `vector_search_ef_search`: override pgvector's `hnsw.ef_search` (1-1000) for the iterative strategies; (admin only)\n - `vector_search_max_scan_tuples`: override pgvector's `hnsw.max_scan_tuples` for the iterative strategies; (admin only)\n - `vector_search_explain`: set to true to record vector-scan instrumentation for this search (expensive); (admin only)\n - `verified`: set to true to search for verified items only (requires Content Management or Official Collections premium feature)\n - `ids`: search for items with those ids, works iff single value passed to `models`\n - `display_type`: search for cards/models with specific display types\n - `non_temporal_dim_ids`: search for cards/metrics/datasets with this exact set of non temporal dimension field IDs (requires appdb engine)\n - `has_temporal_dim`: set to true for cards/metrics/datasets with 1 or more temporal dimensions (requires appdb engine)\n\n Note that not all item types support all filters, and the results will include only models that support the provided\n filters. For example:\n\n - The `created-by` filter supports dashboards, models, actions, and cards.\n - The `verified` filter supports models and cards.\n\n A search query that has both filters applied will only return models and cards.",
"description" : "Search for items in Metabase.\n For the list of supported models, check [[metabase.search.config/all-models]].\n\n `context` identifies the surface issuing the search; it selects the ranking weights and filter defaults.\n It defaults to `api`, the value for programmatic callers.\n\n Filters:\n - `archived`: set to true to search archived items only, default is false\n - `table_db_id`: search for tables, cards, and models of a certain DB\n - `models`: only search for items of specific models. If not provided, search for all models\n - `filter_items_in_personal_collection`: only search for items in personal collections\n - `created_at`: search for items created at a specific timestamp\n - `created_by`: search for items created by a specific user\n - `last_edited_at`: search for items last edited at a specific timestamp\n - `last_edited_by`: search for items last edited by a specific user\n - `search_native_query`: set to true to search the content of native queries\n - `vector_search_strategy`: for the semantic engine: `hnsw` (approximate index search, default),\n `brute-force` (exact filter-first search), or `hnsw-iterative-relaxed` / `hnsw-iterative-strict`\n (index-backed iterative scans with inline filters); ignored by other engines\n - `vector_search_ef_search`: override pgvector's `hnsw.ef_search` (1-1000) for the iterative strategies; (admin only)\n - `vector_search_max_scan_tuples`: override pgvector's `hnsw.max_scan_tuples` for the iterative strategies; (admin only)\n - `vector_search_explain`: set to true to record vector-scan instrumentation for this search (expensive); (admin only)\n - `verified`: set to true to search for verified items only (requires Content Management or Official Collections premium feature)\n - `ids`: search for items with those ids, works iff single value passed to `models`\n - `display_type`: search for cards/models with specific display types\n\n Note that not all item types support all filters, and the results will include only models that support the provided\n filters. For example:\n\n - The `created-by` filter supports dashboards, models, actions, and cards.\n - The `verified` filter supports models and cards.\n\n A search query that has both filters applied will only return models and cards.",
"parameters" : [ {
"in" : "query",
"name" : "q",
@@ -44658,29 +44665,6 @@
"type" : "null"
} ]
}
}, {
"in" : "query",
"name" : "non_temporal_dim_ids",
"required" : false,
"schema" : {
"oneOf" : [ {
"type" : "string",
"minLength" : 1
}, {
"type" : "null"
} ]
}
}, {
"in" : "query",
"name" : "has_temporal_dim",
"required" : false,
"schema" : {
"oneOf" : [ {
"type" : "boolean"
}, {
"type" : "null"
} ]
}
} ],
"responses" : {
"2XX" : {
@@ -45002,29 +44986,6 @@
"type" : "null"
} ]
}
}, {
"in" : "query",
"name" : "non_temporal_dim_ids",
"required" : false,
"schema" : {
"oneOf" : [ {
"type" : "string",
"minLength" : 1
}, {
"type" : "null"
} ]
}
}, {
"in" : "query",
"name" : "has_temporal_dim",
"required" : false,
"schema" : {
"oneOf" : [ {
"type" : "boolean"
}, {
"type" : "null"
} ]
}
}, {
"in" : "query",
"name" : "expected_result_type",
@@ -45173,7 +45134,7 @@
"post" : {
"operationId" : "post-api-segment",
"summary" : "POST /api/segment",
"description" : "Create a new `Segment`.",
"description" : "Create a new `Segment`. The Segment's table is derived from its `definition`.",
"parameters" : [ ],
"responses" : {
"2XX" : {
@@ -45207,14 +45168,9 @@
"name" : {
"type" : "string",
"minLength" : 1
},
"table_id" : {
"description" : "value must be an integer greater than zero.",
"type" : "integer",
"minimum" : 1
}
},
"required" : [ "name", "table_id", "definition" ]
"required" : [ "name", "definition" ]
}
}
}

View File

@@ -197,6 +197,8 @@ See [Modular embedding parameters](./parameters.md#pass-parameter-values-to-embe
Locked parameters let you filter data without exposing the filter to the end-user. Locking parameters is useful for restricting data based on who's viewing the embed (for example, showing each customer only their own data).
![Locked parameters](./images/locked-parameters.png)
To use locked parameters, you need to:
1. Set the parameter to **Locked** in the embed settings.

Binary file not shown.

After

Width:  |  Height:  |  Size: 73 KiB

View File

@@ -308,6 +308,8 @@ For an overview of how data permissions work in Metabase, see [Data permissions]
If every tenant has their data in a separate database, then instead of using permissions for data access control, you can use [**database routing**](../permissions/database-routing.md) to route queries to appropriate databases directly.
For a comparison of data isolation methods, see [Choosing a data isolation method](../permissions/data-isolation-methods.md).
- **Create queries** controls whether tenant users can create queries on the data they see. If you want to give your tenant users the ability to drill through (e.g., through `drills` parameter in [modular embedding](../embedding/modular-embedding.md)), you need to give them "Create queries" permissions, because a drill through is a new query.
- **Download results** controls, unsurprisingly, whether people can download results of queries. You need to set download permissions if you want to give your users the option to download their data as a spreadsheet (for example, through `with-downloads` parameter in [modular embedding](../embedding/modular-embedding.md)).
@@ -437,3 +439,4 @@ If you disable multi-tenant strategy, _all your tenant users will be deactivated
- [Embedding overview](./start.md)
- [JWT authentication](../people-and-groups/authenticating-with-jwt.md)
- [Permissions overview](../permissions/start.md)
- [Data isolation methods](../permissions/data-isolation-methods.md)

View File

@@ -0,0 +1,90 @@
---
title: Data isolation methods
summary: Compare the ways Metabase can restrict data access for internal analytics or multi-tenant setups, including row and column security, impersonation, database routing, and locked parameters.
---
# Data isolation methods
> Row and column security, impersonation, and database routing are only available on Metabase [Pro](https://www.metabase.com/product/pro) and [Enterprise](https://www.metabase.com/product/enterprise) plans.
Metabase offers several ways to ensure people only see the data they're supposed to see, whether you're restricting data by team for internal analytics or isolating customer data in a multi-tenant setup. The best method depends on how you store your data and where you want to enforce access rules:
- If all data lives in the same tables, use [row and column security](#row-and-column-security) to filter data in Metabase, or [impersonation](#impersonation) to enforce access rules at the database level.
- If each customer or team has their own database, use [database routing](#database-routing).
- If you're embedding dashboards for people without Metabase accounts, use [locked parameters](#locked-parameters).
You can combine these methods. For example, you can use locked parameters for a guest embed while using row and column security for signed-in users.
## Row and column security
[Row and column security](./row-and-column-security.md) lets you control which rows and columns each group can see. You can define rules that filter data based on [user attributes](../people-and-groups/managing.md#adding-a-user-attribute). Metabase applies your rules to questions you build in the query builder.
![Row and column security](./images/row-and-column-level-security.png)
Use row and column security when:
- You need granular control over which rows and columns each group or customer can see.
- You want to manage access rules in the Metabase UI instead of in your database.
Don't use row and column security when:
- People need to write native SQL queries. Row and column security doesn't apply to the SQL editor.
- You need column-level security on MongoDB or Druid, which only support row-level security.
## Impersonation
[Impersonation](./impersonation.md) lets your database control what each person can see. You define roles and row-level security policies in your database. Metabase connects using the role specified by each person's [user attribute](../people-and-groups/managing.md#adding-a-user-attribute), and your database enforces what that role can see.
![Impersonation](./images/connection-impersonation.png)
Use impersonation when:
- You want to centralize access logic in your database instead of in Metabase.
- People need to write native SQL queries. Since your database enforces the rules, the rules apply in the SQL editor too.
Don't use impersonation when:
- Your database [doesn't support it](./impersonation.md#databases-that-support-impersonation).
- You don't want to set up and maintain roles and policies in your database.
- You rely heavily on caching. You can't share cached results between roles.
## Database routing
[Database routing](./database-routing.md) sends each person's queries to their own database. You build questions and dashboards once against a primary database, and Metabase routes queries to the right destination database based on each person's [user attribute](../people-and-groups/managing.md#adding-a-user-attribute). Database routing requires that each group's data lives in a separate database, and that all destination databases have identical schemas.
![Database routing](./images/database-routing-diagram.png)
Use database routing when:
- You want the strongest isolation. Each group's data lives in a separate database.
- You want to avoid one group's queries affecting the performance of another group's queries.
Don't use database routing when:
- Groups share tables. Use [row and column security](#row-and-column-security) or [impersonation](#impersonation) instead.
- You want to manage access rules in the Metabase UI instead of maintaining a separate database for each group.
## Locked parameters
[Locked parameters](../embedding/guest-embedding.md#locked-parameters) filter an embedded dashboard or question before Metabase displays it. Your app passes the filter value in a signed JWT. The filter is invisible to the person viewing the embed, and they can't change it.
![Locked parameters](../embedding/images/locked-parameters.png)
Use locked parameters when:
- You want to restrict data without setting up Metabase accounts, user attributes, or permissions.
- You want to filter data without exposing the filter or its values to the people viewing the embed.
Don't use locked parameters when:
- You want account-level or role-based access control across your Metabase.
- You're working with sensitive data. Use a method that enforces access at the permissions or database level instead, like [row and column security](#row-and-column-security), [impersonation](#impersonation), or [database routing](#database-routing).
## Further reading
- [Permissions overview](./start.md)
- [Tenants](../embedding/tenants.md)
- [Row and column security](./row-and-column-security.md)
- [Impersonation](./impersonation.md)
- [Database routing](./database-routing.md)
- [Locked parameters for guest embeds](../embedding/guest-embedding.md#locked-parameters)

View File

@@ -9,6 +9,8 @@ summary: Route queries to different databases based on who's viewing them. Great
With database routing, an admin can build a question once using one database, and the question will run its query against a different database with the same schema depending on who is viewing the question.
![Database routing diagram](./images/database-routing-diagram.png)
Database routing is useful for:
- Managing modular and full app embedding setups where each customer has their own database with identical schemas.

Binary file not shown.

After

Width:  |  Height:  |  Size: 86 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 70 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 144 KiB

View File

@@ -10,6 +10,8 @@ This page covers the [View data](./data.md#view-data-permissions) permission lev
**Impersonation access** allows admins to "outsource" View data permissions to roles in your database. Admins can associate user attributes with database-defined roles and their privileges. If someone is in a group with their View data permission set to Impersonation, the person will be able to view and query data based on the privileges granted to the role specified by their user attribute.
![Connection impersonation](./images/connection-impersonation.png)
## Databases that support impersonation
For now, impersonation access is only available for the following databases:

View File

@@ -14,6 +14,8 @@ Row and column security lets you give granular permissions for different groups
You can use row and column security to set up [self-service analytics](https://www.metabase.com/learn/metabase-basics/embedding/multi-tenant-self-service-analytics), so that each of your customers can only view the rows that match their customer ID. For example, if you have an Accounts table with information about your customers, you can add permissions to the table so that each customer only sees the data relevant to them.
![Row and column security](./images/row-and-column-level-security.png)
> Row and column security was formerly called data sandboxing. It's the same feature, it just now has a more descriptive name.
## Row and column security examples

View File

@@ -28,6 +28,10 @@ Granting group access to different Metabase features.
Setting up row and column security to restrict access to rows and columns in tables.
## [Choosing a data isolation method](./data-isolation-methods.md)
Compare row and column security, impersonation, database routing, and locked parameters for multi-tenant setups.
## [Snippet folder permissions](./snippets.md)
Organize snippets into folders that require permissions to view.

View File

@@ -25,6 +25,7 @@
[metabase.sso.core :as sso]
[metabase.util.encryption :as encryption]
[metabase.util.i18n :refer [tru]]
[metabase.util.log :as log]
[metabase.util.malli.schema :as ms]
[metabase.util.password :as u.password]
[throttle.core :as throttle]
@@ -67,9 +68,15 @@
(t2/select-one-fn :credentials :model/AuthIdentity :user_id user-id :provider "password")]
(and password_hash (u.password/verify-password password password_salt password_hash)))
(when (sso/ldap-enabled)
(when-let [user-email (t2/select-one-fn :email :model/User user-id)]
(when-let [user-info (sso/find-user user-email)]
(sso/verify-password user-info password))))))))
;; an unreachable directory fails closed: re-auth is denied (the user retries when the
;; directory is back; admin/remove needs no re-auth), never an unhandled 500
(try
(when-let [user-email (t2/select-one-fn :email :model/User user-id)]
(when-let [user-info (sso/find-user user-email)]
(sso/verify-password user-info password)))
(catch Exception e
(log/warn e "LDAP re-auth failed because the directory is unreachable")
false)))))))
;; Notification emails here are fire-and-log by construction: the messages/send-mfa-*-email!
;; senders route through email/send-message!, which catches and logs delivery failures — so an
@@ -164,14 +171,23 @@
;;; -------------------------------------------------- Admin --------------------------------------------------
(api.macros/defendpoint :post "/admin/remove" :- nil
"Admin: remove a user's two-factor enrollment entirely — the lockout escape hatch for a lost
authenticator with no recovery codes. Never feature-gated (a lapsed license must not make
lockouts permanent). The affected user is notified by email. They re-enroll from scratch —
there is nothing to \"reset\", the secret lives on their device."
"Admin: remove *another* user's two-factor enrollment entirely — the lockout escape hatch for a
lost authenticator with no recovery codes. Never feature-gated (a lapsed license must not make
lockouts permanent). The affected user is notified by email. They re-enroll from scratch — there
is nothing to \"reset\", the secret lives on their device.
Removing your *own* enrollment is deliberately refused here: this is a user-management endpoint
that takes no second factor (the target can't produce one — that's why an admin is removing it),
and the admin UI never collects a code. Self-removal goes through the normal Security page, which
re-auths with a fresh factor via `/disable`. Without this guard a hijacked admin session could
strip its own 2FA with only a cookie, turning transient access into a permanent password bypass."
[_route-params
_query-params
{user-id :user_id} :- [:map [:user_id ms/PositiveInt]]]
(api/check-superuser)
(when (= user-id api/*current-user-id*)
(throw (ex-info (tru "You cannot administratively remove your own two-factor authentication. Please use the normal removal method in your account settings.")
{:status-code 400})))
(when (enrollment/disable! user-id)
(let [user (t2/select-one :model/User :id user-id)]
(messages/send-mfa-removed-by-admin-email! (:email user))

View File

@@ -5,7 +5,8 @@
[metabase-enterprise.mfa.enrollment :as enrollment]
[metabase-enterprise.mfa.totp :as totp]
[metabase.test :as mt]
[metabase.test.fixtures :as fixtures]))
[metabase.test.fixtures :as fixtures]
[toucan2.core :as t2]))
(use-fixtures :once (fixtures/initialize :db :web-server :test-users))
@@ -22,6 +23,23 @@
(mt/user-http-request :crowberto :post 204 "ee/mfa/admin/remove" {:user_id user-id})
(is (nil? (enrollment/enrolled-method user-id)))))))
(deftest admin-cannot-self-remove-test
;; admin/remove is "manage OTHER users" — self-removal is refused so a hijacked admin cookie
;; can't strip its own 2FA (this endpoint takes no factor; a stolen session has none to give).
;; Removing your own enrollment goes through /disable, which re-auths with a fresh code.
(mt/with-temp [:model/AuthIdentity _ {:user_id (mt/user->id :crowberto)
:provider "totp"
:confirmed_at (t/instant)
:credentials {:secret (totp/generate-secret)}}]
(try
(testing "removing your own id is refused, enrollment intact"
(is (re-find #"cannot administratively remove your own"
(mt/user-http-request :crowberto :post 400 "ee/mfa/admin/remove"
{:user_id (mt/user->id :crowberto)})))
(is (= :totp (enrollment/enrolled-method (mt/user->id :crowberto)))))
(finally
(t2/delete! :model/AuthIdentity :user_id (mt/user->id :crowberto) :provider "totp")))))
(deftest admin-remove-is-audited-test
;; audit rows require the :audit-app feature at event time (premium-features/log-enabled?)
(mt/with-premium-features #{:audit-app}

View File

@@ -68,6 +68,81 @@
(is (true? (:mfa/pending? gated)))
(is (nil? (:mfa/first-factor gated)))))))))))
(deftest gate-sso-provider-matrix-test
(testing "every SSO provider is exempt for an enrolled user — the IdP owns MFA there"
(mt/with-premium-features #{:multi-factor-auth}
(mt/with-temporary-setting-values [mfa-enforcement :optional]
(with-enrolled-user!
(fn [user-id]
(let [result {:success? true :user {:id user-id}}]
(doseq [provider [:provider/google
:provider/saml
:provider/jwt
:provider/oidc
:provider/custom-oidc
:provider/slack-connect]]
(testing provider
(is (= result (auth-identity.provider/apply-mfa-gate provider result))))))))))))
(deftest gate-provider-registry-exhaustiveness-test
;; The gate's fall-through branch is exempt, so a provider nobody classified silently bypasses
;; MFA. This test forces the decision: every provider in the auth-identity hierarchy must appear
;; below. If you added a provider and landed here, decide its MFA posture — challenged providers
;; also go in `metabase-enterprise.mfa.gate/challenged-providers`, session-suppressed ones in
;; `session-suppressed-providers`; exemption is the default and needs only this map.
(let [classification {:provider/password :challenged
:provider/ldap :challenged
:provider/emailed-secret-password-reset :session-suppressed
;; IdP owns MFA for SSO logins
:provider/google :exempt
:provider/saml :exempt
:provider/jwt :exempt
:provider/oidc :exempt
:provider/custom-oidc :exempt
:provider/slack-connect :exempt
;; admin-granted, time-boxed, audited; staff can't hold the second factor
:provider/support-access-grant :exempt
;; abstract parent (reset/verification/magic links) — a concrete login
;; child must be classified here on its own when it ships
:provider/emailed-secret :exempt
;; the second factor itself, never a first factor
:provider/totp :exempt}]
(testing "every registered provider has a deliberate MFA classification"
(is (= (set (keys classification))
(set (descendants :metabase.auth-identity.provider/provider)))))
(testing "the gate's behavior matches each classification for an enrolled user"
(mt/with-premium-features #{:multi-factor-auth}
(mt/with-temporary-setting-values [mfa-enforcement :optional]
(with-enrolled-user!
(fn [user-id]
(let [result {:success? true :user {:id user-id}}]
(doseq [[provider posture] classification]
(testing provider
(let [gated (auth-identity.provider/apply-mfa-gate provider result)]
(case posture
:challenged (is (= :mfa-required (:success? gated)))
:session-suppressed (do (is (true? (:success? gated)))
(is (true? (:mfa/pending? gated))))
:exempt (is (= result gated))))))))))))))
(deftest gate-env-var-escape-hatch-test
;; The last-admin lockout recovery: the only superuser loses authenticator + recovery codes, so
;; nobody can log in to flip the setting. Restarting with MB_MFA_ENFORCEMENT=off must win over
;; the :optional stored in the DB and let them in single-step.
(mt/with-premium-features #{:multi-factor-auth}
(mt/with-temporary-setting-values [mfa-enforcement :optional]
(with-enrolled-user!
(fn [user-id]
(let [result {:success? true :user {:id user-id}}]
(is (= :mfa-required
(:success? (auth-identity.provider/apply-mfa-gate :provider/password result)))
"sanity: challenged before the override")
(mt/with-temp-env-var-value! [mb-mfa-enforcement "off"]
(is (false? (mfa.settings/mfa-enabled?))
"the env var beats the DB value")
(is (= result (auth-identity.provider/apply-mfa-gate :provider/password result))
"the gate no-ops, so the locked-out admin can log in"))))))))
(deftest mfa-enforcement-write-path-test
(mt/with-temporary-setting-values [mfa-enforcement :off]
(testing "enabling (:optional) requires the :multi-factor-auth feature (setup is gated; enforcement is not)"

View File

@@ -5,9 +5,11 @@
(:require
[clojure.test :refer :all]
[java-time.api :as t]
[metabase-enterprise.mfa.enrollment :as enrollment]
[metabase-enterprise.mfa.management :as mfa.management]
[metabase-enterprise.mfa.totp :as totp]
[metabase.session.api :as api.session]
[metabase.sso.core :as sso]
[metabase.sso.ldap-test-util :as ldap.test]
[metabase.test :as mt]
[metabase.test.fixtures :as fixtures]
@@ -73,6 +75,35 @@
(testing "the enroll endpoint's schema also refuses a blank password"
(mt/client session-key :post 400 "ee/mfa/enroll" {:password ""})))))))
(deftest ldap-directory-unreachable-fails-closed-test
;; Lockout-corner: the directory goes down while a user is enrolled. Re-auth (disable) must fail
;; closed as a denied verification — not a 500 — and must not touch the enrollment. The recovery
;; path that needs no directory, admin/remove, keeps working.
(mt/with-premium-features #{:multi-factor-auth}
(mt/with-temporary-setting-values [mfa-enforcement :optional]
(ldap.test/with-ldap-server!
(let [{session-key :id} (mt/client :post 200 "session" {:username sally-email
:password sally-directory-password})
user-id (t2/select-one-fn :id :model/User :email sally-email)
secret (totp/generate-secret)]
(try
(t2/insert! :model/AuthIdentity {:user_id user-id
:provider "totp"
:confirmed_at (t/instant)
:credentials {:secret secret}})
(with-redefs [sso/find-user (fn [& _] (throw (ex-info "connection refused" {})))]
(testing "enroll re-auth with the directory down is a 400, not a 500"
(is (=? {:errors {:password some?}}
(mt/client session-key :post 400 "ee/mfa/enroll"
{:password sally-directory-password}))))
(testing "the enrollment is untouched"
(is (= :totp (enrollment/enrolled-method user-id))))
(testing "admin/remove needs no directory and still recovers the account"
(mt/user-http-request :crowberto :post 204 "ee/mfa/admin/remove" {:user_id user-id})
(is (nil? (enrollment/enrolled-method user-id)))))
(finally
(t2/delete! :model/AuthIdentity :user_id user-id :provider "totp"))))))))
(deftest ldap-only-user-enrolls-with-directory-password-test
(mt/with-premium-features #{:multi-factor-auth}
(mt/with-temporary-setting-values [mfa-enforcement :optional]

View File

@@ -2,6 +2,7 @@
(:require
[clojure.test :refer :all]
[java-time.api :as t]
[metabase-enterprise.mfa.enrollment :as enrollment]
[metabase-enterprise.mfa.recovery-codes :as recovery-codes]
[metabase-enterprise.mfa.totp :as totp]
[metabase-enterprise.mfa.verification :as verification]
@@ -194,3 +195,25 @@
(t2/select-one-fn :credentials :model/AuthIdentity
:user_id user-id :provider "totp"))
"sanity: the old key can no longer read the row"))))))
(deftest lost-encryption-key-fails-closed-test
(testing "an instance started with the wrong key: verify errors (no bypass, no session), and
admin removal — which never decrypts — still recovers the account"
(let [secret (totp/generate-secret)
k1 "0123456789abcdef-key-A"
k2 "totally-different-key-B!"]
(mt/with-temp [:model/User {user-id :id} {}]
(encryption-test/with-secret-key k1
(t2/insert! :model/AuthIdentity {:user_id user-id
:provider "totp"
:confirmed_at (t/instant)
:credentials {:secret secret}}))
(encryption-test/with-secret-key k2
(testing "verification fails closed — an error, never a false-positive login"
(is (thrown? Exception
(verification/verify-attempt! user-id (totp/generate-code secret) (fresh-jti)))))
(testing "the enrollment row survives the failed attempt"
(is (= 1 (t2/count :auth_identity :user_id user-id :provider "totp"))))
(testing "disable! deletes without decrypting, so the admin escape hatch works"
(is (true? (enrollment/disable! user-id)))
(is (zero? (t2/count :auth_identity :user_id user-id :provider "totp")))))))))

View File

@@ -1,6 +1,6 @@
{
"name": "@metabase/embedding-sdk-react",
"version": "0.63.0-alpha",
"version": "0.64.0-alpha",
"description": "Metabase Embedding SDK for React",
"bin": "./dist/cli.js",
"repository": {

View File

@@ -41,17 +41,7 @@
}
}
.ItemLink {
font-size: inherit !important;
&:hover {
color: var(--mb-color-core-brand);
}
}
.ItemLink,
.CollectionLink {
:hover {
color: var(--mb-color-core-brand);
}
.ItemLink:hover,
.CollectionLink :hover {
color: var(--mb-color-core-brand);
}

View File

@@ -3,7 +3,6 @@ import { useMemo } from "react";
import { getShortStrategyLabel } from "metabase/admin/performance/utils";
import { EllipsifiedCollectionPath } from "metabase/common/components/EllipsifiedPath/EllipsifiedCollectionPath";
import { Link } from "metabase/common/components/Link";
import { MaybeLink } from "metabase/common/components/MaybeLink/MaybeLink.styled";
import { useGetIcon } from "metabase/hooks/use-icon";
import { Box, Button, Ellipsified, FixedSizeIcon, Flex } from "metabase/ui";
import * as Urls from "metabase/urls";
@@ -48,6 +47,20 @@ export const TableRowForCacheableItem = ({
}
};
const cellContent = (
<Flex
className={StrategyEditorForQuestionsAndDashboardsS.ItemLink}
align="center"
wrap="nowrap"
gap="sm"
fw={700}
style={{ overflow: "hidden" }}
>
{iconName ? <FixedSizeIcon name={iconName} /> : <Box h="sm" w="md" />}
<Ellipsified>{name}</Ellipsified>
</Flex>
);
return (
<tr
className={
@@ -56,26 +69,7 @@ export const TableRowForCacheableItem = ({
: undefined
}
>
<td>
<MaybeLink
className={StrategyEditorForQuestionsAndDashboardsS.ItemLink}
to={url}
>
<Flex
align="center"
wrap="nowrap"
gap="sm"
style={{ overflow: "hidden" }}
>
{iconName ? (
<FixedSizeIcon name={iconName} />
) : (
<Box h="sm" w="md" />
)}
<Ellipsified style={{ fontWeight: "bold" }}>{name}</Ellipsified>
</Flex>
</MaybeLink>
</td>
<td>{url ? <Link to={url}>{cellContent}</Link> : cellContent}</td>
<td>
{collection && (
<Link

View File

@@ -103,6 +103,7 @@ export const LoginAttributeMappingEditor = ({
const [entries, setEntries] = useState<MappingEditorEntry[]>(
structuredAttributes ? [] : buildEntries(simpleAttributes),
);
const [autoFocusIndex, setAutoFocusIndex] = useState<number | null>(null);
useEffect(() => {
// structuredAttributes can change if a different tenant is selected
@@ -121,6 +122,7 @@ export const LoginAttributeMappingEditor = ({
onError(hasError(newEntries));
} else {
onChange(buildMapping(newEntries));
setAutoFocusIndex(newEntries.length - 1);
}
};
@@ -143,6 +145,7 @@ export const LoginAttributeMappingEditor = ({
}}
keyOpts={keyOpts}
error={entryError(entries, key)}
autoFocus={index === autoFocusIndex}
/>
</Flex>
<Flex gap="sm" w="50%">
@@ -175,7 +178,9 @@ export const LoginAttributeMappingEditor = ({
variant="light"
size="xs"
leftSection={<Icon name="add" />}
onClick={() => handleChange(addEntry(entries))}
onClick={() => {
handleChange(addEntry(entries));
}}
mt="md"
disabled={_.some(
entries,
@@ -193,11 +198,13 @@ const KeyInput = ({
value,
onChange,
error,
autoFocus,
}: {
value: string;
onChange: (e: React.ChangeEvent<HTMLInputElement>) => void;
keyOpts: MappingEditorEntry["keyOpts"];
error?: string | boolean;
autoFocus?: boolean;
}) => {
if (keyOpts?.disabled) {
return (
@@ -215,6 +222,7 @@ const KeyInput = ({
w="100%"
onChange={onChange}
error={error}
autoFocus={autoFocus}
/>
);
};

View File

@@ -162,6 +162,20 @@ describe("LoginAttributeMappingEditor", () => {
});
});
it("focuses the key input of a newly added attribute", async () => {
setup({ simpleAttributes });
const keyInputs = await screen.findAllByPlaceholderText("Key");
keyInputs.forEach((input) => expect(input).not.toHaveFocus());
await userEvent.click(
await screen.findByRole("button", { name: /Add an attribute/ }),
);
const newKeyInput = (await screen.findAllByPlaceholderText("Key")).at(-1);
expect(newKeyInput).toHaveFocus();
});
it("can remove an attribute", async () => {
const { onChange } = setup({ simpleAttributes });

View File

@@ -14,7 +14,6 @@ import { SortableColumnHeader } from "metabase/common/components/ItemsTable/Base
import {
ColumnHeader,
ItemNameCell,
MaybeItemLink,
TBody,
Table,
TableColumn,
@@ -229,36 +228,28 @@ function preventDefault(event: MouseEvent) {
function NameCell({ metric }: { metric?: MetricResult }) {
const headingId = `metric-${metric?.id ?? "dummy"}-heading`;
return (
<ItemNameCell
data-testid="metric-name"
aria-labelledby={headingId}
{...nameProps}
>
<MaybeItemLink
to={
metric
? Urls.metric({ id: metric.id, name: metric.name, type: "metric" })
: undefined
}
style={{
// To align the icons with "Name" in the <th>
paddingInlineStart: "1.4rem",
paddingInlineEnd: ".5rem",
}}
onClick={preventDefault}
>
<Flex align="center" gap="0.5rem" ps="1.4rem" pe="0.5rem">
{metric ? (
<EntityItem.Name
name={metric?.name || ""}
variant="list"
id={headingId}
/>
<Link
to={Urls.metric({
id: metric.id,
name: metric.name,
type: "metric",
})}
onClick={preventDefault}
>
<EntityItem.Name name={metric.name} variant="list" id={headingId} />
</Link>
) : (
<SkeletonText />
)}
</MaybeItemLink>
</Flex>
</ItemNameCell>
);
}

View File

@@ -9,7 +9,6 @@ import { EntityItem } from "metabase/common/components/EntityItem";
import { SortableColumnHeader } from "metabase/common/components/ItemsTable/BaseItemsTable";
import {
ItemNameCell,
MaybeItemLink,
TBody,
Table,
TableColumn,
@@ -210,35 +209,28 @@ function NameCell({ model }: { model?: ModelResult }) {
const getIcon = useGetIcon();
const headingId = `model-${model?.id || "dummy"}-heading`;
const icon = getIcon(model ?? { model: "dataset" }) ?? { name: "folder" };
const name = <EntityItem.Name name={model?.name || ""} variant="list" />;
return (
<ItemNameCell data-testid="model-name" aria-labelledby={headingId}>
<MaybeItemLink
to={
model
? Urls.model({ id: model.id, name: model.name, type: "model" })
: undefined
}
style={{
// To align the icons with "Name" in the <th>
paddingInlineStart: "1.4rem",
paddingInlineEnd: ".5rem",
}}
onClick={preventDefault}
>
<Flex id={headingId} align="center" gap="0.5rem" ps="1.4rem" pe="0.5rem">
<EntityIcon
size="1rem"
{...icon}
color="icon-brand"
style={{ flexShrink: 0 }}
/>
{
<EntityItem.Name
name={model?.name || ""}
variant="list"
id={headingId}
/>
}
</MaybeItemLink>
{model ? (
<Link
to={Urls.model({ id: model.id, name: model.name, type: "model" })}
onClick={preventDefault}
style={{ overflow: "hidden" }}
>
{name}
</Link>
) : (
name
)}
</Flex>
</ItemNameCell>
);
}

View File

@@ -1,25 +1,9 @@
// eslint-disable-next-line no-restricted-imports
import { css } from "@emotion/react";
// eslint-disable-next-line no-restricted-imports
import styled from "@emotion/styled";
import { MarkdownPreview } from "metabase/common/components/MarkdownPreview";
import { RawMaybeLink } from "metabase/common/components/MaybeLink/MaybeLink.styled";
import { Box, type BoxProps } from "metabase/ui";
export const ItemLink = styled(RawMaybeLink)<{ to?: string }>`
display: block;
height: min-content;
${(props) =>
props.to
? ""
: css`
${Body} {
cursor: default;
}
`}
`;
// Unjustified type cast. FIXME
export const ActionsContainer = styled(Box)<BoxProps>`
display: flex;

View File

@@ -9,9 +9,10 @@ import type {
} from "metabase/common/collections/types";
import { EntityIcon } from "metabase/common/components/EntityIcon";
import { EventSandbox } from "metabase/common/components/EventSandbox";
import { Link } from "metabase/common/components/Link";
import { useGetIcon } from "metabase/hooks/use-icon";
import { PLUGIN_MODERATION } from "metabase/plugins";
import { Card, Flex, Skeleton, Tooltip } from "metabase/ui";
import { Box, Card, Flex, Skeleton, Tooltip } from "metabase/ui";
import { modelToUrl } from "metabase/urls";
import type Database from "metabase-lib/v1/metadata/Database";
import type {
@@ -27,7 +28,6 @@ import {
Body,
Description,
Header,
ItemLink,
Title,
} from "./PinnedItemCard.styled";
@@ -119,71 +119,74 @@ function PinnedItemCard({
isCollectionItem(item) &&
(onCopy || onMove || createBookmark || deleteBookmark || collection);
const card = (
<Card p={0} radius="md" shadow="none" withBorder>
<Body style={item ? undefined : { cursor: "default" }}>
<Header>
<EntityIcon {...iconData} size="1.5rem" color="core-brand" />
<ActionsContainer h={item ? undefined : "2rem"}>
{hasActions && (
// This component is used within a `<Link>` component,
// so we must prevent events from triggering the activation of the link
<EventSandbox preventDefault sandboxedEvents={["onClick"]}>
<ActionMenu
databases={databases}
bookmarks={bookmarks}
createBookmark={createBookmark}
deleteBookmark={deleteBookmark}
item={item}
collection={collection}
onCopy={onCopy}
onMove={onMove}
/>
</EventSandbox>
)}
</ActionsContainer>
</Header>
{item ? (
<>
<Tooltip
label={item.name}
position="bottom"
disabled={!showTitleTooltip}
maw={TOOLTIP_MAX_WIDTH}
>
<Title
onMouseEnter={(e) => maybeEnableTooltip(e, setShowTitleTooltip)}
>
<Flex align="center" gap="0.5rem">
{item.name}
<PLUGIN_MODERATION.ModerationStatusIcon
status={item.moderated_status}
filled
size={14}
/>
</Flex>
</Title>
</Tooltip>
<Description tooltipMaxWidth={TOOLTIP_MAX_WIDTH}>
{item.description || DEFAULT_DESCRIPTION[item.model] || ""}
</Description>
</>
) : (
<>
<Skeleton natural h="1.5rem" />
<Skeleton natural mt="xs" mb="4px" h="1rem" />
</>
)}
</Body>
</Card>
);
const to = item ? (modelToUrl(item) ?? "/") : undefined;
return (
<ItemLink
<Box
display="block"
h="min-content"
className={className}
to={item ? (modelToUrl(item) ?? "/") : undefined}
onClick={onClick}
>
<Card p={0} radius="md" shadow="none" withBorder>
<Body>
<Header>
<EntityIcon {...iconData} size="1.5rem" color="core-brand" />
<ActionsContainer h={item ? undefined : "2rem"}>
{hasActions && (
// This component is used within a `<Link>` component,
// so we must prevent events from triggering the activation of the link
<EventSandbox preventDefault sandboxedEvents={["onClick"]}>
<ActionMenu
databases={databases}
bookmarks={bookmarks}
createBookmark={createBookmark}
deleteBookmark={deleteBookmark}
item={item}
collection={collection}
onCopy={onCopy}
onMove={onMove}
/>
</EventSandbox>
)}
</ActionsContainer>
</Header>
{item ? (
<>
<Tooltip
label={item.name}
position="bottom"
disabled={!showTitleTooltip}
maw={TOOLTIP_MAX_WIDTH}
>
<Title
onMouseEnter={(e) =>
maybeEnableTooltip(e, setShowTitleTooltip)
}
>
<Flex align="center" gap="0.5rem">
{item.name}
<PLUGIN_MODERATION.ModerationStatusIcon
status={item.moderated_status}
filled
size={14}
/>
</Flex>
</Title>
</Tooltip>
<Description tooltipMaxWidth={TOOLTIP_MAX_WIDTH}>
{item.description || DEFAULT_DESCRIPTION[item.model] || ""}
</Description>
</>
) : (
<>
<Skeleton natural h="1.5rem" />
<Skeleton natural mt="xs" mb="4px" h="1rem" />
</>
)}
</Body>
</Card>
</ItemLink>
{to ? <Link to={to}>{card}</Link> : card}
</Box>
);
}

View File

@@ -12,7 +12,6 @@ import {
import { EntityItem } from "metabase/common/components/EntityItem";
import { IconButtonWrapper } from "metabase/common/components/IconButtonWrapper";
import { Link } from "metabase/common/components/Link";
import { RawMaybeLink } from "metabase/common/components/MaybeLink/MaybeLink.styled";
import AdminS from "metabase/css/admin.module.css";
import type { IconProps, TextProps } from "metabase/ui";
import { FixedSizeIcon, Text } from "metabase/ui";
@@ -112,17 +111,15 @@ export const ItemButton = styled(Text)<
export const ItemLink = styled(Link)(itemLinkStyle);
export const MaybeItemLink = styled(RawMaybeLink)(itemLinkStyle);
export const ItemNameCell = styled.td`
padding: 0 !important;
${ItemLink}, ${MaybeItemLink}, ${ItemButton} {
& > * {
padding: 1em;
}
&:hover {
${ItemLink}, ${MaybeItemLink}, ${ItemButton} {
& > * {
color: var(--mb-color-core-brand);
}

View File

@@ -1,48 +0,0 @@
// eslint-disable-next-line no-restricted-imports
import { css } from "@emotion/react";
// eslint-disable-next-line no-restricted-imports
import styled from "@emotion/styled";
import type { HTMLAttributes } from "react";
import type { LinkProps } from "metabase/common/components/Link";
import { Link } from "metabase/common/components/Link";
import { maybeColor } from "metabase/ui/utils/colors";
interface RawMaybeLinkProps {
to?: string;
activeColor?: string;
inactiveColor?: string;
isSingleLine?: boolean;
}
export function RawMaybeLink({
to,
activeColor,
inactiveColor,
isSingleLine,
...props
}: RawMaybeLinkProps & (LinkProps | HTMLAttributes<HTMLSpanElement>)) {
return to ? <Link to={to} {...props} /> : <span {...props} />;
}
const hoverStyle = (props: RawMaybeLinkProps) => css`
cursor: pointer;
${props.activeColor ? `color: ${maybeColor(props.activeColor)};` : ""}
`;
/**
* @deprecated styled components are deprecated
*/
export const MaybeLink = styled(RawMaybeLink)`
display: flex;
align-items: center;
font-size: 0.875em;
font-weight: bold;
${(props) =>
props.inactiveColor ? `color: ${maybeColor(props.inactiveColor)};` : ""}
min-width: ${(props) => (props.isSingleLine ? 0 : "")};
:hover {
${(props) => (props.to || props.onClick) && hoverStyle(props)}
}
`;

View File

@@ -8,6 +8,7 @@ import { getAdminPaths } from "metabase/selectors/admin";
import { getIsEmbeddingIframe } from "metabase/selectors/embed";
import { getCanAccessOnboardingPage } from "metabase/selectors/onboarding";
import { getSetting } from "metabase/selectors/settings";
import { getBasename } from "metabase/utils/basename";
import { isSameOrSiteUrlOrigin, replaceLocation } from "metabase/utils/dom";
import { Navigate } from "./Navigate";
@@ -32,6 +33,37 @@ const getRedirectUrl = () => {
: "/";
};
/**
* A redirect target arrives in one of two shapes: an absolute URL
* (`https://host/metabase/x`, `//host/x`) whose pathname is already what the
* browser sees, or a basename-relative path (`/oauth/x`, sloppily `oauth/x`) —
* the convention for every SPA path, including the backend's login redirect
* (the server sits behind the prefix-stripping proxy and never sees the
* subpath). Normalize to a browser-real URL.
*/
const toBrowserUrl = (url: string) => {
const hasExplicitOrigin = /^[a-z][a-z0-9+.-]*:|^\/\//i.test(url);
// String-join for paths, not URL resolution: a leading "/" is *root*-relative
// and would discard the basename from a URL base.
return hasExplicitOrigin
? new URL(url, window.location.origin)
: new URL(
`${getBasename()}/${url.replace(/^\//, "")}`,
window.location.origin,
);
};
/** The inverse: strip the basename back off for the SPA router, which prepends
* it itself (and for prefix checks like `isBackendOnlyPath`). */
const toRouterPath = ({ pathname, search, hash }: URL) => {
const basename = getBasename();
const relativePathname =
basename && pathname.startsWith(`${basename}/`)
? pathname.slice(basename.length)
: pathname;
return `${relativePathname}${search}${hash}`;
};
/**
* `getRedirectUrl` accepts site-url-origin targets as well as same-origin ones,
* so it can hand back an absolute URL. The SPA router only understands in-app
@@ -39,13 +71,13 @@ const getRedirectUrl = () => {
* can serve that origin at all.
*/
const resolveRedirectTarget = (url: string) => {
const target = new URL(url, window.location.origin);
const target = toBrowserUrl(url);
return {
// Absolute, so a full-page redirect is not re-resolved against the current
// path (a bare `auth/sso/x` would otherwise land under `/auth/login`).
href: target.href,
path: `${target.pathname}${target.search}${target.hash}`,
path: toRouterPath(target),
isInAppOrigin: target.origin === window.location.origin,
};
};

View File

@@ -4,6 +4,7 @@ import {
createMockSettingsState,
createMockState,
} from "metabase/redux/store/mocks";
import { setBasename } from "metabase/utils/basename";
import { replaceLocation } from "metabase/utils/dom";
import { createMockUser } from "metabase-types/api/mocks";
@@ -154,6 +155,7 @@ describe("route-guards", () => {
window.history.replaceState({}, "", "/");
// reset the global MetabaseSettings singleton mutated by mockSettings
mockSettings();
setBasename("");
});
it("does a full-page redirect for a relative backend-only path", async () => {
@@ -211,6 +213,66 @@ describe("route-guards", () => {
});
expect(history?.getCurrentLocation().pathname).toBe("/auth/login");
});
describe("when Metabase is hosted under a subpath (GIT-10551)", () => {
beforeEach(() => {
setBasename("/metabase");
});
it("prefixes the subpath on a full-page redirect to a backend-only path", async () => {
const { history } = setup("/oauth/authorize?client_id=abc");
await waitFor(() => {
expect(replaceLocationMock).toHaveBeenCalledWith(
`${ORIGIN}/metabase/oauth/authorize?client_id=abc`,
);
});
expect(history?.getCurrentLocation().pathname).toBe("/auth/login");
});
it("does not double the subpath when navigating in-app", async () => {
const { history } = setup("/dashboard/1");
await waitFor(() => {
expect(history?.getCurrentLocation().pathname).toBe("/dashboard/1");
});
expect(replaceLocationMock).not.toHaveBeenCalled();
});
it("recognizes a backend-only path behind the subpath in an absolute URL", async () => {
const { history } = setup(`${ORIGIN}/metabase/oauth/authorize`);
await waitFor(() => {
expect(replaceLocationMock).toHaveBeenCalledWith(
`${ORIGIN}/metabase/oauth/authorize`,
);
});
expect(history?.getCurrentLocation().pathname).toBe("/auth/login");
});
it("does not strip a lookalike path prefix that is not the basename", async () => {
const { history } = setup(`${ORIGIN}/metabase-docs/foo`);
await waitFor(() => {
expect(history?.getCurrentLocation().pathname).toBe(
"/metabase-docs/foo",
);
});
expect(replaceLocationMock).not.toHaveBeenCalled();
});
it("handles a nested subpath basename", async () => {
setBasename("/bi/metabase");
const { history } = setup("/oauth/authorize?client_id=abc");
await waitFor(() => {
expect(replaceLocationMock).toHaveBeenCalledWith(
`${ORIGIN}/bi/metabase/oauth/authorize?client_id=abc`,
);
});
expect(history?.getCurrentLocation().pathname).toBe("/auth/login");
});
});
});
describe("isBackendOnlyPath", () => {

View File

@@ -1,21 +1,10 @@
import type { ReactNode } from "react";
import { t } from "ttag";
import { DataStudioBreadcrumbs } from "metabase/common/data-studio/components/DataStudioBreadcrumbs";
import { PageContainer } from "metabase/common/data-studio/components/PageContainer";
import {
PaneHeader,
PaneHeaderInput,
} from "metabase/common/data-studio/components/PaneHeader";
import { useSetting } from "metabase/common/hooks";
import { Link } from "metabase/router";
import { Group, Stack } from "metabase/ui";
import * as Urls from "metabase/urls";
import { Stack } from "metabase/ui";
import type { ScheduleDisplayType, TransformTagId } from "metabase-types/api";
import { NAME_MAX_LENGTH } from "../../constants";
import { LockedTransformsHoverCard } from "../LockedTransformsHoverCard/LockedTransformsHoverCard";
import { TransformBadge } from "../TransformBadge/TransformBadge";
import { JobHeader } from "../JobHeader";
import { ScheduleSection } from "./ScheduleSection";
import { TagSection } from "./TagSection";
@@ -29,6 +18,7 @@ type JobEditorProps = {
tabs?: ReactNode;
readOnly?: boolean;
isCheckingPermissions?: boolean;
showMetabotButton?: boolean;
onNameChange: (name: string) => void;
onScheduleChange: (
schedule: string,
@@ -44,48 +34,21 @@ export function JobEditor({
tabs,
readOnly,
isCheckingPermissions,
showMetabotButton,
onNameChange,
onScheduleChange,
onTagListChange,
}: JobEditorProps) {
const isMeterLocked = useSetting("transforms-meter-locked");
return (
<PageContainer data-testid="transforms-job-editor" gap="2.5rem">
<PaneHeader
title={
<Group align="center" gap="sm" wrap="nowrap">
<PaneHeaderInput
initialValue={job.name}
maxLength={NAME_MAX_LENGTH}
onChange={onNameChange}
readOnly={readOnly}
/>
{isMeterLocked && (
<LockedTransformsHoverCard>
<TransformBadge bg="background_surface-warning-strong">{t`Disabled`}</TransformBadge>
</LockedTransformsHoverCard>
)}
{!isMeterLocked && !job.active && (
<TransformBadge bg="background_surface-warning-strong">
{t`Disabled`}
</TransformBadge>
)}
</Group>
}
py={0}
breadcrumbs={
<DataStudioBreadcrumbs>
<Link key="transform-job-list" to={Urls.transformJobList()}>
{t`Jobs`}
</Link>
{job.name}
</DataStudioBreadcrumbs>
}
<JobHeader
job={job}
menu={menu}
actions={actions}
tabs={tabs}
data-testid="jobs-header"
readOnly={readOnly}
showMetabotButton={showMetabotButton}
onNameChange={onNameChange}
/>
<Stack gap="3.5rem">
<ScheduleSection

View File

@@ -0,0 +1,78 @@
import type { ReactNode } from "react";
import { t } from "ttag";
import { DataStudioBreadcrumbs } from "metabase/common/data-studio/components/DataStudioBreadcrumbs";
import {
PaneHeader,
PaneHeaderInput,
} from "metabase/common/data-studio/components/PaneHeader";
import { useSetting } from "metabase/common/hooks";
import { Link } from "metabase/router";
import { Group } from "metabase/ui";
import * as Urls from "metabase/urls";
import { NAME_MAX_LENGTH } from "../../constants";
import type { TransformJobInfo } from "../JobEditor/types";
import { LockedTransformsHoverCard } from "../LockedTransformsHoverCard/LockedTransformsHoverCard";
import { TransformBadge } from "../TransformBadge/TransformBadge";
type JobHeaderProps = {
job: TransformJobInfo;
menu?: ReactNode;
actions?: ReactNode;
tabs?: ReactNode;
readOnly?: boolean;
showMetabotButton?: boolean;
onNameChange: (name: string) => void;
};
export function JobHeader({
job,
menu,
actions,
tabs,
readOnly,
showMetabotButton,
onNameChange,
}: JobHeaderProps) {
const isMeterLocked = useSetting("transforms-meter-locked");
return (
<PaneHeader
title={
<Group align="center" gap="sm" wrap="nowrap">
<PaneHeaderInput
initialValue={job.name}
maxLength={NAME_MAX_LENGTH}
onChange={onNameChange}
readOnly={readOnly}
/>
{isMeterLocked && (
<LockedTransformsHoverCard>
<TransformBadge bg="background_surface-warning-strong">{t`Disabled`}</TransformBadge>
</LockedTransformsHoverCard>
)}
{!isMeterLocked && !job.active && (
<TransformBadge bg="background_surface-warning-strong">
{t`Disabled`}
</TransformBadge>
)}
</Group>
}
py={0}
breadcrumbs={
<DataStudioBreadcrumbs>
<Link key="transform-job-list" to={Urls.transformJobList()}>
{t`Jobs`}
</Link>
{job.name}
</DataStudioBreadcrumbs>
}
menu={menu}
actions={actions}
tabs={tabs}
showMetabotButton={showMetabotButton}
data-testid="jobs-header"
/>
);
}

View File

@@ -0,0 +1 @@
export { JobHeader } from "./JobHeader";

View File

@@ -0,0 +1,46 @@
import { useMemo } from "react";
import { t } from "ttag";
import {
skipToken,
useListTransformJobTransformsQuery,
useUpdateTransformJobMutation,
} from "metabase/api";
import { useMetadataToasts } from "metabase/metadata/hooks";
import type { TransformJobId } from "metabase-types/api";
import {
canEditTransform,
useTransformPermissions,
} from "./use-transform-permissions";
export function useJobHeaderState(jobId: TransformJobId | undefined) {
const [updateJob] = useUpdateTransformJobMutation();
const { sendErrorToast, sendSuccessToast } = useMetadataToasts();
const { transformsDatabases } = useTransformPermissions();
const { data: transforms, isLoading: isCheckingPermissions } =
useListTransformJobTransformsQuery(jobId ?? skipToken);
const readOnly = useMemo(() => {
if (!transformsDatabases || !transforms) {
return true;
}
return !transforms.every((transform) =>
canEditTransform(transform, transformsDatabases),
);
}, [transforms, transformsDatabases]);
const handleNameChange = async (name: string) => {
if (jobId === undefined) {
return;
}
const { error } = await updateJob({ id: jobId, name });
if (error) {
sendErrorToast(t`Failed to update job name`);
} else {
sendSuccessToast(t`Job name updated`);
}
};
return { readOnly, isCheckingPermissions, onNameChange: handleNameChange };
}

View File

@@ -1,18 +1,15 @@
import { useMemo, useState } from "react";
import { useState } from "react";
import { t } from "ttag";
import {
skipToken,
useGetTransformJobQuery,
useListTransformJobTransformsQuery,
useUpdateTransformJobMutation,
} from "metabase/api";
import { LoadingAndErrorWrapper } from "metabase/common/components/LoadingAndErrorWrapper";
import { useMetadataToasts } from "metabase/metadata/hooks";
import {
canEditTransform,
useTransformPermissions,
} from "metabase/transforms/hooks/use-transform-permissions";
import { useJobHeaderState } from "metabase/transforms/hooks/use-job-header-state";
import { useTransformPermissions } from "metabase/transforms/hooks/use-transform-permissions";
import { Center } from "metabase/ui";
import * as Urls from "metabase/urls";
import type {
@@ -50,16 +47,9 @@ export function JobPage({ params }: JobPageProps) {
pollingInterval: isPolling ? POLLING_INTERVAL : undefined,
});
const { transformsDatabases, isLoadingDatabases, databasesError } =
useTransformPermissions();
const { data: transforms, isLoading: isLoadingTransforms } =
useListTransformJobTransformsQuery(jobId || skipToken);
const readOnly = useMemo(() => {
if (!transformsDatabases || !transforms) {
return true;
}
return !transforms.every((t) => canEditTransform(t, transformsDatabases));
}, [transforms, transformsDatabases]);
const { isLoadingDatabases, databasesError } = useTransformPermissions();
const { readOnly, isCheckingPermissions, onNameChange } =
useJobHeaderState(jobId);
if (isPolling !== isPollingNeeded(job)) {
setIsPolling(isPollingNeeded(job));
@@ -79,7 +69,8 @@ export function JobPage({ params }: JobPageProps) {
<JobPageBody
job={job}
readOnly={readOnly}
isCheckingPermissions={isLoadingTransforms}
isCheckingPermissions={isCheckingPermissions}
onNameChange={onNameChange}
/>
);
}
@@ -88,30 +79,19 @@ type JobPageBodyProps = {
job: TransformJob;
readOnly?: boolean;
isCheckingPermissions?: boolean;
onNameChange: (name: string) => void;
};
function JobPageBody({
job,
readOnly,
isCheckingPermissions,
onNameChange,
}: JobPageBodyProps) {
const [updateJob] = useUpdateTransformJobMutation();
const { sendErrorToast, sendSuccessToast, sendUndoToast } =
useMetadataToasts();
const handleNameChange = async (name: string) => {
const { error } = await updateJob({
id: job.id,
name,
});
if (error) {
sendErrorToast(t`Failed to update job name`);
} else {
sendSuccessToast(t`Job name updated`);
}
};
const handleScheduleChange = async (
schedule: string,
uiDisplayType: ScheduleDisplayType,
@@ -161,7 +141,8 @@ function JobPageBody({
tabs={<JobTabs jobId={job.id} />}
readOnly={readOnly}
isCheckingPermissions={isCheckingPermissions}
onNameChange={handleNameChange}
showMetabotButton
onNameChange={onNameChange}
onScheduleChange={handleScheduleChange}
onTagListChange={handleTagListChange}
/>

View File

@@ -1,8 +1,10 @@
import fetchMock from "fetch-mock";
import { setupDatabaseListEndpoint } from "__support__/server-mocks";
import {
setupDatabaseListEndpoint,
setupUserMetabotPermissionsEndpoint,
} from "__support__/server-mocks";
import {
setupGetTransformJobEndpoint,
setupListTransformJobTransformsEndpoint,
setupListTransformTagsEndpoint,
} from "__support__/server-mocks/transform";
import { act, renderWithProviders, screen, within } from "__support__/ui";
@@ -30,7 +32,8 @@ const setup = ({
setupGetTransformJobEndpoint(job);
setupDatabaseListEndpoint([createMockDatabase()]);
setupListTransformTagsEndpoint([]);
fetchMock.get(`path:/api/transform-job/${job.id}/transforms`, [], {
setupUserMetabotPermissionsEndpoint();
setupListTransformJobTransformsEndpoint(job.id, [], {
delay: transformsDelay,
});

View File

@@ -11,15 +11,11 @@ import {
} from "metabase/api";
import { LoadingAndErrorWrapper } from "metabase/common/components/LoadingAndErrorWrapper";
import { PaginationControls } from "metabase/common/components/PaginationControls";
import { DataStudioBreadcrumbs } from "metabase/common/data-studio/components/DataStudioBreadcrumbs";
import {
PaneHeader,
PanelHeaderTitle,
} from "metabase/common/data-studio/components/PaneHeader";
import { usePageTitle } from "metabase/hooks/use-page-title";
import { useDispatch } from "metabase/redux";
import { Link, replace } from "metabase/router";
import { replace } from "metabase/router";
import { POLLING_INTERVAL } from "metabase/transforms/constants";
import { useJobHeaderState } from "metabase/transforms/hooks/use-job-header-state";
import { formatRunMethod, formatStatus } from "metabase/transforms/utils";
import { Center, Flex, Group, Select, Stack } from "metabase/ui";
import * as Urls from "metabase/urls";
@@ -32,6 +28,8 @@ import {
type TransformRunMethod,
} from "metabase-types/api";
import { JobHeader } from "../../components/JobHeader";
import { JobMoreMenu } from "../../components/JobMoreMenu";
import { JobTabs } from "../../components/JobTabs";
import S from "./JobRunListPage.module.css";
@@ -63,6 +61,7 @@ export function JobRunListPage({ params, location }: JobRunListPageProps) {
const dispatch = useDispatch();
const { data: job } = useGetTransformJobQuery(jobId ?? skipToken);
const { readOnly, onNameChange } = useJobHeaderState(jobId);
const { data, isLoading, error } = useListTransformJobRunsQuery(
jobId != null
@@ -164,18 +163,16 @@ export function JobRunListPage({ params, location }: JobRunListPageProps) {
data-testid="job-run-list"
>
<Stack className={S.main} flex={1} px="3.5rem" pb="md" gap={0}>
<PaneHeader
title={job != null && <PanelHeaderTitle>{job.name}</PanelHeaderTitle>}
tabs={jobId != null && <JobTabs jobId={jobId} />}
breadcrumbs={
<DataStudioBreadcrumbs>
<Link to={Urls.transformJobList()}>{t`Jobs`}</Link>
{job?.name ?? t`Run history`}
</DataStudioBreadcrumbs>
}
py={0}
showMetabotButton
/>
{job !== undefined && (
<JobHeader
job={job}
menu={!readOnly && <JobMoreMenu job={job} />}
tabs={<JobTabs jobId={job.id} />}
readOnly={readOnly}
showMetabotButton
onNameChange={onNameChange}
/>
)}
{isLoading || error != null ? (
<Center h="100%">
<LoadingAndErrorWrapper loading={isLoading} error={error} />

View File

@@ -2,9 +2,11 @@ import userEvent from "@testing-library/user-event";
import fetchMock from "fetch-mock";
import {
setupDatabaseListEndpoint,
setupGetTransformJobEndpoint,
setupListJobRunTransformRunsEndpoint,
setupListTransformJobRunsEndpoint,
setupListTransformJobTransformsEndpoint,
setupUserMetabotPermissionsEndpoint,
} from "__support__/server-mocks";
import {
@@ -22,6 +24,7 @@ import type {
TransformRunForJobRun,
} from "metabase-types/api";
import {
createMockDatabase,
createMockListTransformJobRunsResponse,
createMockTransformJob,
createMockTransformJobRun,
@@ -48,6 +51,8 @@ function setup({
setupUserMetabotPermissionsEndpoint();
setupGetTransformJobEndpoint(job);
setupDatabaseListEndpoint([createMockDatabase()]);
setupListTransformJobTransformsEndpoint(JOB_ID, []);
setupListTransformJobRunsEndpoint(JOB_ID, () =>
createMockListTransformJobRunsResponse({
data: currentRuns,

View File

@@ -144,6 +144,7 @@ export function compileFormatter(
columnName: string | null = null,
columnExtents: ColumnExtents | null = null,
isRowFormatter: boolean = false,
matchOnlyWithinRange: boolean = false,
): Formatter {
if (format.type === "single") {
let { operator, value, color } = format;
@@ -200,7 +201,10 @@ export function compileFormatter(
}),
).clamp(true);
return (value) => {
if (!isNumber(value) || value < lowerBound || value > upperBound) {
if (!isNumber(value)) {
return null;
}
if (matchOnlyWithinRange && (value < lowerBound || value > upperBound)) {
return null;
}
const colorValue = scale(value);
@@ -273,11 +277,27 @@ function compileFormatters(
columnExtents: ColumnExtents,
): Formatters {
const formatters: Formatters = {};
const rangeFormatterCounts: Record<string, number> = {};
formats.forEach((format) => {
if (format.type === "range") {
format.columns.forEach((columnName) => {
rangeFormatterCounts[columnName] =
(rangeFormatterCounts[columnName] ?? 0) + 1;
});
}
});
formats.forEach((format) => {
format.columns.forEach((columnName) => {
formatters[columnName] = formatters[columnName] || [];
formatters[columnName].push(
compileFormatter(format, columnName, columnExtents, false),
compileFormatter(
format,
columnName,
columnExtents,
false,
rangeFormatterCounts[columnName] > 1,
),
);
});
});

View File

@@ -245,6 +245,60 @@ describe("compileFormatter with extreme ranges", () => {
});
describe("makeCellBackgroundGetter", () => {
it("saturates values outside custom bounds when there is one range rule (#77957)", () => {
const getter = makeCellBackgroundGetter(
[[-0.1], [0], [0.6], [0.8]],
[createMockNumericColumn({ name: "score" })],
[
createMockColumnRangeFormattingSetting({
columns: ["score"],
colors: ["#ff0000", "#00ff00"],
min_type: "custom",
min_value: 0,
max_type: "custom",
max_value: 0.6,
}),
],
false,
);
expect(getter(-0.1, 0, "score")).toBe(getter(0, 1, "score"));
expect(getter(0.8, 3, "score")).toBe(getter(0.6, 2, "score"));
});
it("saturates values when range rules target different columns (#77957)", () => {
const getter = makeCellBackgroundGetter(
[
[0.6, 0.6],
[0.8, 0.8],
],
[
createMockNumericColumn({ name: "score" }),
createMockNumericColumn({ name: "total" }),
],
[
createMockColumnRangeFormattingSetting({
columns: ["score"],
min_type: "custom",
min_value: 0,
max_type: "custom",
max_value: 0.6,
}),
createMockColumnRangeFormattingSetting({
columns: ["total"],
min_type: "custom",
min_value: 0,
max_type: "custom",
max_value: 0.6,
}),
],
false,
);
expect(getter(0.8, 1, "score")).toBe(getter(0.6, 0, "score"));
expect(getter(0.8, 1, "total")).toBe(getter(0.6, 0, "total"));
});
it("applies each range rule to values in its own range when ranges do not overlap (#75933)", () => {
const getter = makeCellBackgroundGetter(
[[-1000], [1000]],

View File

@@ -1,4 +1,4 @@
import { Component } from "react";
import { useRef } from "react";
import _ from "underscore";
import DashboardS from "metabase/css/dashboard.module.css";
@@ -20,7 +20,6 @@ import type {
VisualizationProps,
} from "metabase/visualizations/types";
import { BarChart } from "metabase/visualizations/visualizations/BarChart";
import type { DatasetColumn } from "metabase-types/api/dataset";
import { ScalarValueContainer } from "./ScalarValueContainer";
import { SCALAR_CHART_DEFINITION } from "./definition";
@@ -40,157 +39,133 @@ function legacyScalarSettingsToFormatOptions(
.value();
}
// Scalar visualization shows a single number
// Multiseries Scalar is transformed to a Funnel
export class Scalar extends Component<
VisualizationProps & VisualizationPassThroughProps
> {
_scalar: HTMLElement | null = null;
// Scalar visualization shows a single number; multiseries is transformed to a bar chart
function ScalarComponent(
props: VisualizationProps & VisualizationPassThroughProps,
) {
const scalarRef = useRef<HTMLDivElement>(null);
_getColumnIndex(
cols: DatasetColumn[],
settings: ComputedVisualizationSettings,
) {
const columnIndex = _.findIndex(
cols,
(col) => col.name === settings["scalar.field"],
const {
series: [
{
data: { cols, rows },
},
],
settings,
visualizationIsClickable,
onVisualizationClick,
height,
width,
gridSize,
totalNumGridCols,
fontFamily,
rawSeries,
} = props;
if (rawSeries.length > 1) {
return (
<TransformedVisualization
transformSeries={scalarToBarTransform}
originalProps={props}
VisualizationComponent={BarChart}
/>
);
return columnIndex < 0 ? 0 : columnIndex;
}
render() {
const {
series: [
{
data: { cols, rows },
},
],
settings,
visualizationIsClickable,
onVisualizationClick,
height,
width,
gridSize,
totalNumGridCols,
fontFamily,
rawSeries,
} = this.props;
// clamp the -1 of a missing "scalar.field" to the first column
const columnIndex = Math.max(
0,
cols.findIndex((col) => col.name === settings["scalar.field"]),
);
const value = rows[0] && rows[0][columnIndex];
const column = cols[columnIndex];
if (rawSeries.length > 1) {
return (
<TransformedVisualization
transformSeries={scalarToBarTransform}
originalProps={this.props}
VisualizationComponent={BarChart}
/>
);
const formatOptions = {
...legacyScalarSettingsToFormatOptions(settings),
...settings.column?.(column),
jsx: true,
};
const segments = settings["scalar.segments"]?.filter((segment) =>
segmentIsValid(segment, { allowOpenEnded: true }),
);
const color = getColor(value, segments);
const tooltipContent = getTooltipContent(segments);
const { displayValue, fullScalarValue } = compactifyValue(
value,
width,
formatOptions,
);
const label = settings["scalar.label"];
const sublabel = settings["scalar.sublabel"];
const isMetricsViewer = label !== undefined;
const isClickable = onVisualizationClick != null && !isMetricsViewer;
const handleClick = () => {
const element = scalarRef.current;
if (element == null) {
return;
}
const columnIndex = this._getColumnIndex(cols, settings);
const value = rows[0] && rows[0][columnIndex];
const column = cols[columnIndex];
const formatOptions = {
...legacyScalarSettingsToFormatOptions(settings),
...settings.column?.(column),
jsx: true,
};
const segments = settings["scalar.segments"]?.filter((segment) =>
segmentIsValid(segment, { allowOpenEnded: true }),
);
const color = getColor(value, segments);
const tooltipContent = getTooltipContent(segments);
const { displayValue, fullScalarValue } = compactifyValue(
const clickData = {
value,
width,
formatOptions,
);
const label = settings["scalar.label"];
const sublabel = settings["scalar.sublabel"];
const isMetricsViewer = label !== undefined;
const isClickable = onVisualizationClick != null && !isMetricsViewer;
const handleClick = () => {
if (this._scalar == null) {
return;
}
const clickData = {
value,
column,
data: rows[0]?.map((value, index) => ({ value, col: cols[index] })),
settings,
element: this._scalar,
};
if (
this._scalar &&
onVisualizationClick &&
visualizationIsClickable(clickData)
) {
onVisualizationClick(clickData);
}
column,
data: rows[0]?.map((value, index) => ({ value, col: cols[index] })),
settings,
element,
};
return (
<ScalarWrapper>
<ScalarValueContainer
className={DashboardS.fullscreenNormalText}
tooltip={fullScalarValue}
alwaysShowTooltip={fullScalarValue !== displayValue}
isClickable={isClickable}
if (onVisualizationClick && visualizationIsClickable(clickData)) {
onVisualizationClick(clickData);
}
};
return (
<ScalarWrapper>
<ScalarValueContainer
className={DashboardS.fullscreenNormalText}
tooltip={fullScalarValue}
alwaysShowTooltip={fullScalarValue !== displayValue}
isClickable={isClickable}
>
<Tooltip
label={tooltipContent}
position="bottom"
px="0.375rem"
py="xs"
disabled={!tooltipContent}
>
<Tooltip
label={tooltipContent}
position="bottom"
px="0.375rem"
py="xs"
disabled={!tooltipContent}
>
<Stack
onClick={handleClick}
ref={(scalar) => (this._scalar = scalar)}
align="center"
gap={0}
>
<ScalarValue
color={color}
disableHover={isMetricsViewer}
fontFamily={fontFamily}
gridSize={gridSize}
height={Math.max(height - PADDING * 2, 0)}
totalNumGridCols={totalNumGridCols}
// Unjustified type cast. FIXME
value={displayValue as string}
width={Math.max(width - PADDING, 0)}
/>
{label && (
<Text fz="14px" lh="16px" c="text-primary" mt="md" ta="center">
{label}
</Text>
)}
{sublabel && (
<Text
fz="12px"
lh="16px"
c="text-secondary"
mt="xs"
ta="center"
>
{sublabel}
</Text>
)}
</Stack>
</Tooltip>
</ScalarValueContainer>
</ScalarWrapper>
);
}
<Stack onClick={handleClick} ref={scalarRef} align="center" gap={0}>
<ScalarValue
color={color}
disableHover={isMetricsViewer}
fontFamily={fontFamily}
gridSize={gridSize}
height={Math.max(height - PADDING * 2, 0)}
totalNumGridCols={totalNumGridCols}
// Unjustified type cast. FIXME
value={displayValue as string}
width={Math.max(width - PADDING, 0)}
/>
{label && (
<Text fz="14px" lh="16px" c="text-primary" mt="md" ta="center">
{label}
</Text>
)}
{sublabel && (
<Text fz="12px" lh="16px" c="text-secondary" mt="xs" ta="center">
{sublabel}
</Text>
)}
</Stack>
</Tooltip>
</ScalarValueContainer>
</ScalarWrapper>
);
}
Object.assign(Scalar, SCALAR_CHART_DEFINITION);
export const Scalar = Object.assign(ScalarComponent, SCALAR_CHART_DEFINITION);

View File

@@ -110,6 +110,46 @@ describe("Scalar", () => {
overflowY: "visible",
});
});
it("should call onVisualizationClick with the clicked element when clickable", async () => {
const onVisualizationClick = jest.fn();
render(
<Scalar
{...mockedProps}
series={series(12345)}
rawSeries={series(12345)}
settings={settings}
visualizationIsClickable={() => true}
onVisualizationClick={onVisualizationClick}
width={230}
/>,
);
await userEvent.click(screen.getByText("12,345"));
expect(onVisualizationClick).toHaveBeenCalledWith(
expect.objectContaining({
value: 12345,
column: expect.objectContaining({ name: "count" }),
element: expect.any(HTMLElement),
}),
);
});
it("should fall back to the first column when scalar.field matches no column", () => {
render(
<Scalar
{...mockedProps}
series={series(12345)}
rawSeries={series(12345)}
settings={{ ...settings, "scalar.field": "not-a-column" }}
visualizationIsClickable={() => false}
width={230}
/>,
);
expect(screen.getByText("12,345")).toBeInTheDocument();
});
});
describe("scalar viz settings", () => {

View File

@@ -43,8 +43,13 @@ export function setupListTransformJobsEndpoint(jobs: TransformJob[]) {
export function setupListTransformJobTransformsEndpoint(
jobId: TransformJobId,
transforms: Transform[],
options?: { delay?: number },
) {
fetchMock.get(`path:/api/transform-job/${jobId}/transforms`, transforms);
fetchMock.get(
`path:/api/transform-job/${jobId}/transforms`,
transforms,
options,
);
}
export function setupListTransformJobTransformsEndpointWithError(

View File

@@ -531,6 +531,10 @@
(prometheus/counter :metabase-notification/template-update
{:description "Number of notification templates updated."
:labels [:channel-type]})
(prometheus/counter :metabase-notification/image-buffer-pooled
{:description "Number of chart rasterization images backed by a pooled buffer."})
(prometheus/counter :metabase-notification/image-buffer-unpooled
{:description "Number of chart rasterization images too large for the pool, allocated fresh."})
(prometheus/counter :metabase-gsheets/connection-creation-began
{:description "How many times the instance has initiated a Google Sheets connection creation."})
(prometheus/counter :metabase-gsheets/connection-creation-error

View File

@@ -1,195 +1,66 @@
(ns metabase.channel.render.image-buffer
"A small pool of reusable, fixed-size `int[]` pixel arrays that back the ARGB [[java.awt.image.BufferedImage]]s chart
rasterization renders into, shared across all rendering threads.
"Pools the oversized `int[]` pixel arrays that back the ARGB [[java.awt.image.BufferedImage]]s chart rasterization
renders into. The arrays are humongous allocations under G1 (several MB each) and subscriptions render in bursts,
so reusing a few fixed-size arrays keeps them from piling up faster than GC can reclaim them.
Why: rasterizing a chart (SVG via Batik) paints into a `width * height * 4` byte buffer -- 3-6 MB at dashboard
sizes. Under the default G1 GC any object over half a region (~1 MB at a 2 MB region) is \"humongous\": allocated
into dedicated regions a young GC can't reclaim. Subscriptions render in bursts, so without reuse these pile up
faster than mixed/full GCs reclaim the regions and the JVM OOMs on region exhaustion while most of the heap is free.
What we reuse and why this shape: the only expensive, humongous part of a `BufferedImage` is its backing `int[]`.
Constructing the wrapper objects (`DataBufferInt` + raster + the *cached* `ColorModel/getRGBdefault`) around an
existing array is ~1 us (verified) -- and critically, one oversized `int[]` can back an image of ANY `w` x `h` whose
`w*h` fits it, because `createPackedRaster` re-interprets the flat array at the new scanline stride. So we don't
need a per-size cache: we pool a few `int[]`s of one fixed size big enough to cover the vast majority of renders,
and on each [[acquire]] wrap a throwaway correctly-sized image around one. A render that needs *more* pixels than
the fixed size just gets a fresh standard `BufferedImage` (the original behavior) -- a rare one-off, not pooled. We
pool the arrays, not the images, to keep the minimum in memory.
Reuse is safe because Batik PNG-encodes the buffer eagerly and returns only a `byte[]`; the image never escapes. The
one load-bearing rule is clearing the used region on acquire (Batik composites with `SrcOver` and skips its
background fill when the background is nil, so stale pixels would bleed a previous render through).
Concurrency: the idle arrays live in a `ConcurrentLinkedDeque`, so every operation is lock-free and thread-safe on
its own; there is no cross-operation invariant to protect. [[acquire]] never blocks and never refuses. Retention is
bounded to `:max-arrays` idle arrays (a best-effort check on [[release]] -- occasionally keeping one extra under a
race is harmless, it is only a retention cap). A daemon thread periodically clears idle arrays so memory drains to
zero when rendering stops.
Pools are plain maps built by [[->pool]]; the core fns take one. The no-arg arities target the process-wide
[[default-pool]], which is built lazily on first use (so AOT compilation, which loads every namespace, doesn't
allocate it or spawn its sweeper thread). Only the default pool has a sweeper; tests build isolated, sweeper-free
pools with [[->pool]]."
One oversized `int[]` can back an image of any `w` x `h` whose `w*h` fits it, because `createPackedRaster`
re-interprets the flat array at the new scanline stride, and wrapping it in a [[BufferedImage]] is ~1 us. The pool
is the same dirigiste `Pool` the GraalVM static-viz contexts use, with the same semantics: 0-3 arrays, each held
exclusively from [[acquire]] to [[release]], idle arrays dropped after ~10 minutes."
(:require
[metabase.util.log :as log])
[metabase.analytics-interface.core :as analytics]
[metabase.util.pool :as u.pool])
(:import
(io.aleph.dirigiste Pool)
(java.awt Point)
(java.awt.image BufferedImage ColorModel DataBufferInt Raster WritableRaster)
(java.util.concurrent ConcurrentLinkedDeque Executors ScheduledExecutorService ThreadFactory TimeUnit)
(java.util.concurrent.atomic LongAdder)))
(java.awt.image BufferedImage ColorModel DataBufferInt Raster WritableRaster)))
(set! *warn-on-reflection* true)
(def ^:private default-array-pixels
"Fixed size (in pixels = ints) of every pooled array. ~2x the size that covers the vast majority of dashboard renders
(1200 wide x up to ~1350 tall ~= 1.6M px), so it also fits 2x-supersampled variants. A render needing more pixels
falls back to a fresh standard image. At 4 bytes/pixel a pooled array is ~13 MB."
(def ^:private array-pixels
"Size in pixels (= ints) of every pooled array: ~2x the largest common dashboard render (1200 wide x up to ~1350
tall), so it also fits 2x-supersampled variants. ~13 MB at 4 bytes/pixel."
(* 1200 2700))
(def ^:private default-max-arrays
"Max idle pooled arrays retained. Bounded by render concurrency (a few Jetty/notification threads), not request rate."
6)
(def ^:private default-idle-ttl-ms
"How often the daemon drops idle arrays; if nothing was rendered in the interval, the pool drains to zero."
60000)
;; Standard non-premultiplied sRGB ARGB color model -- a cached singleton. Using this (rather than constructing a
;; DirectColorModel) keeps wrapping cheap: a premultiplied/mismatched color model makes the BufferedImage ctor run a
;; full-raster per-pixel `coerceData` conversion (~7 ms), whereas getRGBdefault matches the raster (no-op).
;; Standard non-premultiplied sRGB ARGB color model (a cached singleton); anything else makes the BufferedImage ctor
;; run a per-pixel coerceData pass over the whole raster.
(def ^:private ^ColorModel rgb-default (ColorModel/getRGBdefault))
(def ^:private ^"[I" argb-masks (int-array [0x00ff0000 0x0000ff00 0x000000ff (unchecked-int 0xff000000)]))
(declare default-pool)
(def ^:private pool-key ::arrays)
(defn ->pool
"Construct a fresh, independent pool: a plain map of its idle-array store, counters, and config. With no args uses
production sizing. Tests pass their own to isolate from the [[default-pool]] (and get no sweeper).
`:idle` is a `ConcurrentLinkedDeque` of idle `int[]`, each of length `:array-pixels`. `:hits`/`:misses`/`:one-offs`
are counters (a \"one-off\" is a render too large for the fixed array, handed a fresh standard image)."
([] (->pool {}))
([{:keys [array-pixels max-arrays idle-ttl-ms]
:or {array-pixels default-array-pixels max-arrays default-max-arrays idle-ttl-ms default-idle-ttl-ms}}]
{:idle (ConcurrentLinkedDeque.)
:hits (LongAdder.)
:misses (LongAdder.)
:one-offs (LongAdder.)
:last-logged (atom {:hits 0 :misses 0})
:array-pixels array-pixels
:max-arrays max-arrays
:idle-ttl-ms idle-ttl-ms}))
(def ^:private ^Pool array-pool
(u.pool/create-pool #(int-array array-pixels) (fn [_array]) {:max-size 3, :idle-minutes 10}))
(defn- wrap-array
"Wrap an existing `int[]` (length >= w*h) as a `w` x `h` ARGB [[BufferedImage]]. Throwaway wrapper (~1 us); only
`backing` carries the humongous cost."
"A `w` x `h` ARGB [[BufferedImage]] over the first `w*h` ints of `backing`."
^BufferedImage [^"[I" backing ^long w ^long h]
(let [db (DataBufferInt. backing (int (* w h)))
^WritableRaster raster (Raster/createPackedRaster db (int w) (int h) (int w) argb-masks (Point. 0 0))]
(let [buffer (DataBufferInt. backing (int (* w h)))
^WritableRaster raster (Raster/createPackedRaster buffer (int w) (int h) (int w) argb-masks (Point. 0 0))]
(BufferedImage. rgb-default raster false nil)))
(defn- cleared-array
"An `int[]` of `array-pixels` length with its first `need` ints zeroed (the region this render will use). Reuses one
from `idle` if available (a hit), else allocates (a miss). Clearing is load-bearing -- see ns docstring."
^"[I" [pool ^long need]
(let [^"[I" backing (if-let [^"[I" reused (.pollFirst ^ConcurrentLinkedDeque (:idle pool))]
(do (.increment ^LongAdder (:hits pool)) reused)
(do (.increment ^LongAdder (:misses pool)) (int-array (long (:array-pixels pool)))))]
(java.util.Arrays/fill backing 0 (int need) (int 0))
backing))
(defn acquire
"Return a cleared `w` x `h` ARGB [[BufferedImage]]. If `w*h` fits the pool's fixed array size, reuses a pooled `int[]`
(or allocates a fixed-size one if none is idle); otherwise returns a fresh standard `BufferedImage` (a one-off, not
pooled). NEVER blocks or refuses. With no pool arg, uses the [[default-pool]].
Pass the result to [[release]] when done. The image is shared mutable state -- it must not escape the rasterization
(Batik PNG-encodes it before returning, which is what makes reuse safe)."
(^BufferedImage [w h] (acquire (default-pool) w h))
(^BufferedImage [pool ^long w ^long h]
(if (> (* w h) (long (:array-pixels pool)))
(do (.increment ^LongAdder (:one-offs pool)) ; too large: fresh standard image, not pooled
(BufferedImage. (int w) (int h) BufferedImage/TYPE_INT_ARGB))
(wrap-array (cleared-array pool (* w h)) w h))))
(defn- backing-of
"The backing `int[]` of an image produced by [[acquire]]."
^"[I" [^BufferedImage img]
(.getData ^DataBufferInt (.getDataBuffer (.getRaster img))))
"Return a cleared `w` x `h` ARGB [[BufferedImage]]. When `w*h` fits the pooled array size, the image wraps a pooled
`int[]` -- blocking until one is available -- and must be handed back with [[release]]. Larger images are freshly
allocated and not pooled. The image is shared mutable state and must not escape the rasterization; clearing the
used region on acquire is load-bearing (Batik composites with `SrcOver`, so stale pixels would bleed a previous
render through)."
^BufferedImage [^long w ^long h]
(if (> (* w h) (long array-pixels))
(do
(analytics/inc! :metabase-notification/image-buffer-unpooled)
(BufferedImage. (int w) (int h) BufferedImage/TYPE_INT_ARGB))
(let [^"[I" backing (.acquire array-pool pool-key)]
(analytics/inc! :metabase-notification/image-buffer-pooled)
(java.util.Arrays/fill backing 0 (int (* w h)) (int 0))
(wrap-array backing w h))))
(defn release
"Return `img`'s backing array to `pool` for reuse, unless the pool already holds `:max-arrays` idle arrays, or the
array is a one-off (a different size than the pooled arrays) -- in either case it is simply dropped (GC reclaims it).
Safe with `nil`; never throws. With no pool arg, uses the [[default-pool]]."
([img] (release (default-pool) img))
([pool img]
(when img
(let [^"[I" backing (backing-of img)
^ConcurrentLinkedDeque idle (:idle pool)]
(when (and (= (alength backing) (long (:array-pixels pool)))
(< (.size idle) (long (:max-arrays pool))))
(.addFirst idle backing))))))
(defn stats
"Snapshot of a pool's counters `{:hits n :misses n :one-offs n :idle n}`. With no arg, the [[default-pool]]."
([] (stats (default-pool)))
([pool]
{:hits (.sum ^LongAdder (:hits pool))
:misses (.sum ^LongAdder (:misses pool))
:one-offs (.sum ^LongAdder (:one-offs pool))
:idle (.size ^ConcurrentLinkedDeque (:idle pool))}))
(defn sweep!
"Drop the pool's idle arrays so memory is reclaimed when rendering stops. The daemon calls this periodically: if
rendering is active, acquire/release repopulate within microseconds; if idle, the pool stays empty. With no arg, the
[[default-pool]]."
([] (sweep! (default-pool)))
([pool] (.clear ^ConcurrentLinkedDeque (:idle pool))))
;; --- stats logging (cumulative + interval; periodic, NOT per-send -- a per-send rate is dominated by cold-start) ---
(defn- rate-str [hits misses one-offs]
(let [total (+ hits misses)]
(format "%d/%d hits (%.1f%%), %d allocs, %d one-offs"
hits total (if (pos? total) (* 100.0 (/ hits (double total))) 0.0) misses one-offs)))
(defn log-stats!
"Log `pool` reuse: the hit rate since this pool's previous call and the cumulative rate, plus idle/one-off counts.
No-op for the interval line when nothing was rendered since the last call. With no arg, the [[default-pool]]."
([] (log-stats! (default-pool)))
([pool]
(let [now (stats pool)
prev @(:last-logged pool)
dh (- (:hits now) (:hits prev))
dm (- (:misses now) (:misses prev))]
(reset! (:last-logged pool) {:hits (:hits now) :misses (:misses now)})
(when (pos? (+ dh dm))
(log/infof "Image buffer reuse (interval): %s; (lifetime): %s; idle arrays: %d"
(rate-str dh dm 0)
(rate-str (:hits now) (:misses now) (:one-offs now))
(:idle now))))))
;; --- the default pool, built lazily on first use, with its one sweeper daemon ------------------------------------
(defn- start-sweeper!
"Start the daemon that every `:idle-ttl-ms` sweeps `pool` and logs reuse, so the pool drains to zero when rendering
stops. No Quartz needed -- a process-local timer on a daemon thread (so it never blocks shutdown)."
[pool]
(let [exec (Executors/newScheduledThreadPool
1 (reify ThreadFactory
(newThread [_ r]
(doto (Thread. ^Runnable r "image-buffer-maintenance") (.setDaemon true)))))]
(.scheduleWithFixedDelay ^ScheduledExecutorService exec
(fn [] (try (sweep! pool) (log-stats! pool)
(catch Throwable e (log/warn e "Error in image-buffer maintenance"))))
(long (:idle-ttl-ms pool)) (long (:idle-ttl-ms pool)) TimeUnit/MILLISECONDS)))
;; A delay so nothing runs at namespace load (AOT-safe). Realized exactly once on first use, which is also when -- and
;; the only place where -- the single sweeper daemon is started.
(defonce ^:private the-pool
(delay (doto (->pool) start-sweeper!)))
(defn- default-pool
"THE process-wide pool, built (and its sweeper started) on first call."
[]
@the-pool)
"Return the pooled array backing `img` (an image from [[acquire]]) to the pool; unpooled oversized images are just
dropped for GC to reclaim. Safe with `nil`."
[^BufferedImage img]
(when img
(let [^"[I" backing (.getData ^DataBufferInt (.getDataBuffer (.getRaster img)))]
(when (= (alength backing) (long array-pixels))
(.release array-pool pool-key backing)))))

View File

@@ -1,11 +1,8 @@
(ns metabase.channel.render.js.common
"Shared helpers for pooled static-viz renderers (see [[metabase.channel.render.js.graal]]): the graal
bundle's classpath path, the test-init guard, and the dirigiste worker-pool factory."
bundle's classpath path and the test-init guard."
(:require
[metabase.config.core :as config])
(:import
(io.aleph.dirigiste IPool$Generator Pool Pools)
(java.util.concurrent TimeUnit)))
[metabase.config.core :as config]))
(set! *warn-on-reflection* true)
@@ -20,24 +17,3 @@
[]
(when config/tests-available?
((requiring-resolve 'mb.hawk.init/assert-tests-are-not-initializing) "(mt/id ...) or (data/id ...)")))
(defn create-pool
"Build a dirigiste `Pool` of static-viz workers, each held exclusively per render (so at most `:max-size`
renders run at once). `:max-size` is the maximum number of concurrent workers. The utilization
controller targets 100% utilization with a min of 0, so when nothing is rendering the pool shrinks to 0
and `destroy` is called on each idle worker; it rechecks every `:idle-minutes`, so a worker
lingers up to that long before being reaped (keeping it warm through gaps between renders). `(generate)`
mints a worker; `(destroy worker)` tears one down. The other constructor args (queue size, sampling
interval) don't matter much."
^Pool [generate destroy {:keys [max-size idle-minutes]}]
(let [max-queued-acquires 65000
sample-period-ms (.toMillis TimeUnit/MILLISECONDS 25)
control-period-ms (.toMillis TimeUnit/MINUTES idle-minutes)]
(Pool. (reify IPool$Generator
(generate [_ _] (generate))
(destroy [_ _ worker] (destroy worker)))
(Pools/utilizationController 1.0 max-size max-size)
max-queued-acquires
sample-period-ms
control-period-ms
TimeUnit/MILLISECONDS)))

View File

@@ -20,7 +20,8 @@
[metabase.config.core :as config]
[metabase.util.i18n :refer [trs]]
[metabase.util.json :as json]
[metabase.util.malli :as mu])
[metabase.util.malli :as mu]
[metabase.util.pool :as u.pool])
(:import
(io.aleph.dirigiste Pool)
(org.graalvm.polyglot Context Engine HostAccess Source Value)))
@@ -154,8 +155,8 @@
three renders run at once — one per context, on the shared engine. When idle for up to 10 minutes the
pool shrinks to 0 and the generator's `destroy` closes the context (and, on the last one, the shared
engine); the first render after an idle gap rebuilds them. See
[[metabase.channel.render.js.common/create-pool]]."
(common/create-pool generate-context! destroy-context! {:max-size 3, :idle-minutes 10}))
[[metabase.util.pool/create-pool]]."
(u.pool/create-pool generate-context! destroy-context! {:max-size 3, :idle-minutes 10}))
(defn- do-with-static-viz-context
"Borrow a pooled static-viz context and call `f` with it, held exclusively for the call (never let it —

View File

@@ -201,6 +201,8 @@
(defmethod initialize! :default [_]) ; no-op
;; bump
(defmulti display-name
"A nice name for the driver that we'll display to in the admin panel, e.g. \"PostgreSQL\" for `:postgres`. Default
implementation capitializes the name of the driver, e.g. `:oracle` becomes \"Oracle\".

View File

@@ -165,7 +165,8 @@
(try
{:ok (generate-fn item)}
(catch Throwable e
(log/warn e "Example question generation failed for one item")
(log/warnf "Example question generation failed for one item: %s" (ex-message e))
(log/debug e "Example question geenration failure detail")
{:error e}))))
batch)]
(mapv deref futures)))

View File

@@ -23,7 +23,7 @@
[throttle.core :as throttle])
(:import
(clojure.lang ExceptionInfo)
(java.net URLEncoder)))
(java.net URI URLEncoder)))
(set! *warn-on-reflection* true)
@@ -34,12 +34,22 @@
[]
(codecs/bytes->hex (nonce/random-bytes 16)))
(defn- site-path-prefix
"Path component of site-url with no trailing slash (e.g. `/metabase` when Metabase is hosted
under a subpath), or an empty string when site-url has no path or is unset."
[]
(or (some-> (system/site-url) (URI.) (.getPath) (str/replace #"/$" "") not-empty)
""))
(defn- csrf-cookie-opts
"Cookie options for the CSRF cookie. Sets `:secure` when site-url is HTTPS."
"Cookie options for the CSRF cookie. Sets `:secure` when site-url is HTTPS.
The cookie `:path` is the path the *browser* sees, so it must include the subpath prefix
when Metabase is hosted under one — otherwise the cookie is never sent back with the
consent form POST and CSRF validation fails."
[max-age]
(cond-> {:http-only true
:same-site :strict
:path "/oauth/authorize"
:path (str (site-path-prefix) "/oauth/authorize")
:max-age max-age}
(some-> (system/site-url) (str/starts-with? "https"))
(assoc :secure true)))

View File

@@ -32,16 +32,17 @@
[font-name]
(let [dir-name (str/replace font-name " " "_")
file-stem (str/replace font-name " " "")
css-font-name (css-escape-font-name font-name)]
css-font-name (css-escape-font-name font-name)
fonts-url (absolute-url "/app/fonts")]
(if (= font-name "Lato")
(str "@font-face { font-family: 'Lato'; font-weight: 400; font-style: normal; font-display: swap;"
" src: url('/app/fonts/Lato/lato-v16-latin-regular.woff2') format('woff2'); }\n"
" src: url('" fonts-url "/Lato/lato-v16-latin-regular.woff2') format('woff2'); }\n"
"@font-face { font-family: 'Lato'; font-weight: 700; font-style: normal; font-display: swap;"
" src: url('/app/fonts/Lato/lato-v16-latin-700.woff2') format('woff2'); }\n")
" src: url('" fonts-url "/Lato/lato-v16-latin-700.woff2') format('woff2'); }\n")
(str "@font-face { font-family: '" css-font-name "'; font-weight: 400; font-style: normal; font-display: swap;"
" src: url('/app/fonts/" dir-name "/" file-stem "-Regular.woff2') format('woff2'); }\n"
" src: url('" fonts-url "/" dir-name "/" file-stem "-Regular.woff2') format('woff2'); }\n"
"@font-face { font-family: '" css-font-name "'; font-weight: 700; font-style: normal; font-display: swap;"
" src: url('/app/fonts/" dir-name "/" file-stem "-Bold.woff2') format('woff2'); }\n"))))
" src: url('" fonts-url "/" dir-name "/" file-stem "-Bold.woff2') format('woff2'); }\n"))))
(def ^:private default-logo-url "app/assets/img/logo.svg")
@@ -168,7 +169,9 @@
(render-scope-list scopes)
(when-let [redirect-host (some-> (:redirect_uri oauth-params) not-empty (java.net.URI.) (.getHost))]
[:p.destination "Redirects to " [:strong redirect-host]])
[:form {:method "POST" :action "/oauth/authorize/decision"}
;; Absolute action: a root-relative path would drop the subpath when Metabase is hosted
;; under one (site-url like https://example.com/metabase).
[:form {:method "POST" :action (absolute-url "/oauth/authorize/decision")}
[:input {:type "hidden" :name "csrf_token" :value csrf-token}]
[:input {:type "hidden" :name "params_sig" :value params-sig}]
(for [[k v] oauth-params

View File

@@ -1,7 +1,6 @@
(ns metabase.sql-parsing.common
"Shared helpers for pooled sqlglot parsers (see [[metabase.sql-parsing.graal]]): the python-sources
locations, the lazy dev sqlglot install, the protocol reify over a transport function, and the
dirigiste worker-pool factory (copied from [[metabase.channel.render.js.common]])."
locations, the lazy dev sqlglot install, and the protocol reify over a transport function."
(:require
[clojure.java.io :as io]
[clojure.java.shell :as shell]
@@ -9,9 +8,7 @@
[metabase.util.json :as json]
[metabase.util.log :as log])
(:import
(io.aleph.dirigiste IPool$Generator Pool Pools)
(java.io File)
(java.util.concurrent TimeUnit)))
(java.io File)))
(set! *warn-on-reflection* true)
@@ -154,26 +151,3 @@
(json/decode+kw (call "is_single_stmt_of_type" sql stmt-type dialect)))
(transpile-sql [_ sql from-dialect to-dialect]
(json/decode+kw (call "transpile_sql" sql from-dialect to-dialect)))))
;;; -------------------------------------------------- pool -----------------------------------------------
(defn create-pool
"Build a dirigiste `Pool` of workers, each held exclusively per call (so at most `:max-size`
calls run at once). `:max-size` is the maximum number of concurrent workers. The utilization
controller targets 100% utilization with a min of 0, so when nothing is running the pool shrinks to 0
and `destroy` is called on each idle worker; it rechecks every `:idle-minutes`, so a worker
lingers up to that long before being reaped (keeping it warm through gaps between calls). `(generate)`
mints a worker; `(destroy worker)` tears one down. The other constructor args (queue size, sampling
interval) don't matter much."
^Pool [generate destroy {:keys [max-size idle-minutes]}]
(let [max-queued-acquires 65000
sample-period-ms (.toMillis TimeUnit/MILLISECONDS 25)
control-period-ms (.toMillis TimeUnit/MINUTES idle-minutes)]
(Pool. (reify IPool$Generator
(generate [_ _] (generate))
(destroy [_ _ worker] (destroy worker)))
(Pools/utilizationController 1.0 max-size max-size)
max-queued-acquires
sample-period-ms
control-period-ms
TimeUnit/MILLISECONDS)))

View File

@@ -16,7 +16,8 @@
[metabase.analytics-interface.core :as analytics]
[metabase.sql-parsing.common :as common]
[metabase.util.files :as u.files]
[metabase.util.log :as log])
[metabase.util.log :as log]
[metabase.util.pool :as u.pool])
(:import
(io.aleph.dirigiste Pool)
(java.net URI)
@@ -189,8 +190,8 @@
three sqlglot calls run at once — one per context, on the shared engine. When idle for up to 10
minutes the pool shrinks to 0 and the generator's `destroy` closes the context (and, on the last one,
the shared engine); the first call after an idle gap rebuilds them. See
[[metabase.sql-parsing.common/create-pool]]."
(common/create-pool generate-context! destroy-context! {:max-size 3, :idle-minutes 10}))
[[metabase.util.pool/create-pool]]."
(u.pool/create-pool generate-context! destroy-context! {:max-size 3, :idle-minutes 10}))
;;; ---------------------------------------------- timeout handling ---------------------------------------

View File

@@ -0,0 +1,27 @@
(ns metabase.util.pool
"Dirigiste worker-pool factory, shared by the GraalVM static-viz contexts, the sqlglot contexts, and the
image-render pixel buffers."
(:import
(io.aleph.dirigiste IPool$Generator Pool Pools)
(java.util.concurrent TimeUnit)))
(set! *warn-on-reflection* true)
(defn create-pool
"Build a dirigiste `Pool` of workers, each held exclusively from acquire to release (so at most `:max-size` are in
use at once; further acquires block). The utilization controller targets 100% utilization with a min of 0, so an
idle pool shrinks to 0 and `destroy` is called on each idle worker; it rechecks every `:idle-minutes`, so a worker
lingers up to that long between uses (keeping it warm through short gaps). `(generate)` mints a worker;
`(destroy worker)` tears one down. The other constructor args (queue size, sampling interval) don't matter much."
^Pool [generate destroy {:keys [max-size idle-minutes]}]
(let [max-queued-acquires 65000
sample-period-ms (.toMillis TimeUnit/MILLISECONDS 25)
control-period-ms (.toMillis TimeUnit/MINUTES idle-minutes)]
(Pool. (reify IPool$Generator
(generate [_ _] (generate))
(destroy [_ _ worker] (destroy worker)))
(Pools/utilizationController 1.0 max-size max-size)
max-queued-acquires
sample-period-ms
control-period-ms
TimeUnit/MILLISECONDS)))

View File

@@ -7,129 +7,56 @@
(set! *warn-on-reflection* true)
;; Every test runs against its OWN pool built with [[image-buffer/->pool]], not the process-wide default pool. That
;; gives deterministic counters (no cross-test residue) and no background sweeper racing the assertions, so we can
;; assert exact hit/miss/idle counts rather than fuzzy "some reuse" thresholds.
;; All tests share the process-wide pool, which holds at most three arrays at a time and blocks further acquires, so
;; every test must release everything it acquires (and these tests must not run in parallel with each other).
(deftest acquire-returns-argb-of-requested-size-test
(testing "acquire yields an ARGB BufferedImage of exactly the requested size, for any size that fits the pool"
(let [p (image-buffer/->pool)
sizes [[320 240] [1200 800] [1 1] [1200 1322] [64 480]]]
(doseq [[w h] sizes]
(let [^BufferedImage img (image-buffer/acquire p w h)]
(doseq [[w h] [[320 240] [1200 800] [1 1] [1200 1322] [64 480]]]
(let [^BufferedImage img (image-buffer/acquire w h)]
(try
(is (instance? BufferedImage img) (format "%dx%d is a BufferedImage" w h))
(is (= w (.getWidth img)) (format "%dx%d width" w h))
(is (= h (.getHeight img)) (format "%dx%d height" w h)))))))
(deftest sequential-cycle-reuses-after-one-allocation-test
(testing "cycling one size allocates once, then reuses every time"
(let [p (image-buffer/->pool)
n 100]
(dotimes [_ n]
(image-buffer/release p (image-buffer/acquire p 1207 503)))
(let [{:keys [hits misses idle]} (image-buffer/stats p)]
(is (= 1 misses) "only the very first acquire allocates; the rest must reuse")
(is (= (dec n) hits) "every acquire after the first reused a pooled array")
(is (= 1 idle) "one array cycles through the whole run")))))
(deftest different-sizes-reuse-the-same-array-test
(testing "the headline property: one fixed-size array backs ANY size that fits, so different sizes reuse it (NOT a
per-size cache). After the first allocation every differently-sized acquire is a hit."
(let [p (image-buffer/->pool)]
(image-buffer/release p (image-buffer/acquire p 1200 800)) ;; allocate once
(doseq [[w h] [[1000 600] [200 100] [1199 1000] [50 50]]] ;; all different sizes, all fit
(image-buffer/release p (image-buffer/acquire p w h)))
(let [{:keys [hits misses idle]} (image-buffer/stats p)]
(is (= 1 misses) "only the first acquire allocated; differently-sized acquires reused the same array")
(is (= 4 hits) "every differently-sized acquire after the first was a reuse")
(is (= 1 idle) "still just one array")))))
(is (= h (.getHeight img)) (format "%dx%d height" w h))
(finally
(image-buffer/release img)))))))
(deftest reused-array-is-cleared-test
(testing "a reused array is wiped in the region the next render uses (no data leak between renders)"
(let [p (image-buffer/->pool)]
(let [^BufferedImage img (image-buffer/acquire p 64 64)]
(.setRGB img 10 10 (unchecked-int 0xFF00FF00))
(is (not (zero? (.getRGB img 10 10))) "sanity: pixel was dirtied")
(image-buffer/release p img))
(let [^BufferedImage img (image-buffer/acquire p 64 64)]
(is (zero? (.getRGB img 10 10)) "the reused array no longer shows the previous render's pixel")))))
(testing "a reused array is wiped in the region the next render uses (no pixel leak between renders)"
(let [^BufferedImage img (image-buffer/acquire 64 64)]
(.setRGB img 10 10 (unchecked-int 0xFF00FF00))
(is (not (zero? (.getRGB img 10 10))) "sanity: pixel was dirtied")
(image-buffer/release img))
(let [^BufferedImage img (image-buffer/acquire 64 64)]
(try
(is (zero? (.getRGB img 10 10)) "the reused array no longer shows the previous render's pixel")
(finally
(image-buffer/release img))))))
(deftest oversized-request-is-a-one-off-test
(testing "a request larger than the fixed array size falls back to a fresh standard image and is NOT pooled"
(let [p (image-buffer/->pool {:array-pixels (* 100 100)}) ;; tiny fixed size
^BufferedImage big (image-buffer/acquire p 500 500)] ;; 250k px > 10k cap
(is (= 500 (.getWidth big)))
(is (= 500 (.getHeight big)))
(is (= BufferedImage/TYPE_INT_ARGB (.getType big)) "one-off uses a standard TYPE_INT_ARGB image")
(image-buffer/release p big)
(let [{:keys [one-offs idle]} (image-buffer/stats p)]
(is (= 1 one-offs) "counted as a one-off")
(is (= 0 idle) "the oversized array was not pooled on release")))))
(deftest oversized-request-is-not-pooled-test
(testing "a request larger than the pooled array size gets a fresh standard image; releasing it is a no-op"
(let [side 2200 ; 2200^2 px > the 1200x2700 pooled array size
^BufferedImage big (image-buffer/acquire side side)]
(is (= side (.getWidth big)))
(is (= side (.getHeight big)))
(is (= BufferedImage/TYPE_INT_ARGB (.getType big)) "oversized images use a standard TYPE_INT_ARGB image")
(is (nil? (image-buffer/release big))))))
(deftest oversized-and-normal-coexist-test
(testing "one-offs don't pollute the pool: a normal-size array still pools while oversized requests pass through"
(let [p (image-buffer/->pool {:array-pixels (* 100 100)})]
(image-buffer/release p (image-buffer/acquire p 80 80)) ;; fits -> pooled
(image-buffer/release p (image-buffer/acquire p 500 500)) ;; oversized -> one-off, dropped
(let [{:keys [idle one-offs misses]} (image-buffer/stats p)]
(is (= 1 idle) "only the fitting array is pooled")
(is (= 1 one-offs))
(is (= 1 misses) "the fitting acquire allocated once; the one-off is counted separately")))))
(deftest max-arrays-bounds-retention-test
(testing "the pool retains at most :max-arrays idle arrays; surplus releases are dropped"
(let [cap 3
p (image-buffer/->pool {:max-arrays cap})
;; acquire more arrays than the cap (force misses by holding them all before releasing any)
held (vec (repeatedly (* 2 cap) #(image-buffer/acquire p 700 350)))]
(run! #(image-buffer/release p %) held)
(is (= cap (:idle (image-buffer/stats p))) "retained exactly :max-arrays; surplus discarded"))))
(deftest concurrent-allocates-at-most-thread-count-test
(testing "concurrent threads cycling renders allocate at most thread-count arrays"
(let [thread-count 8
cycles-per-thread 50
timeout-ms 5000
pool (image-buffer/->pool {:max-arrays thread-count})
;; A barrier so all threads start at the same moment (genuine overlap, so the "<= thread-count allocations"
;; assertion exercises real concurrency). `await` is bounded and the worker reaches it directly, so a thread
;; that throws can't strand the others on the barrier.
started (java.util.concurrent.CyclicBarrier. thread-count)
worker (fn []
(.await started (long timeout-ms) java.util.concurrent.TimeUnit/MILLISECONDS)
(dotimes [_ cycles-per-thread]
(image-buffer/release pool (image-buffer/acquire pool 909 405)))
::done)
futures (doall (repeatedly thread-count #(future (worker))))
results (mapv #(deref % timeout-ms ::timeout) futures)]
(is (every? #(= ::done %) results)
"every worker finished within the timeout (no deadlock / no thrown worker)")
(let [{:keys [hits misses]} (image-buffer/stats pool)]
(is (= (* thread-count cycles-per-thread) (+ hits misses)) "every acquire counted as a hit or a miss")
(is (<= misses thread-count) "never allocated more than the number of concurrent threads")
(is (pos? hits) "reuse happened")))))
(deftest sweep-drains-to-zero-test
(testing "sweep! drops idle arrays so memory is reclaimed when rendering stops"
(let [p (image-buffer/->pool)]
(image-buffer/release p (image-buffer/acquire p 480 480))
(image-buffer/release p (image-buffer/acquire p 481 481))
(is (pos? (:idle (image-buffer/stats p))) "arrays are pooled before the sweep")
(image-buffer/sweep! p)
(is (= 0 (:idle (image-buffer/stats p))) "sweep drains the pool to zero")
;; and rendering after a sweep just repopulates
(image-buffer/release p (image-buffer/acquire p 480 480))
(is (= 1 (:idle (image-buffer/stats p))) "post-sweep acquire/release repopulates"))))
(deftest stats-idle-reflects-reality-test
(testing "reported idle count matches reality and never goes negative"
(let [p (image-buffer/->pool)]
(dotimes [i 20]
(let [img (image-buffer/acquire p (+ 600 (mod i 3)) 300)]
(when (even? i) (image-buffer/release p img))))
(let [idle (:idle (image-buffer/stats p))]
(is (>= idle 0) "idle count is never negative")
(is (<= idle (:max-arrays (image-buffer/->pool))) "idle never exceeds the retention cap")))))
(deftest acquire-blocks-at-capacity-test
(testing "a fourth concurrent acquire blocks until one of the three held images is released"
(let [held (vec (repeatedly 3 #(image-buffer/acquire 100 100)))
fourth (future
(let [img (image-buffer/acquire 100 100)]
(image-buffer/release img)
::acquired))]
(try
(is (= ::still-blocked (deref fourth 300 ::still-blocked))
"no pooled array is free while three are held")
(image-buffer/release (first held))
(is (= ::acquired (deref fourth 5000 ::timed-out))
"unblocks as soon as an array is released")
(finally
(run! image-buffer/release (rest held)))))))
(deftest release-nil-is-safe-test
(is (nil? (image-buffer/release (image-buffer/->pool) nil))))
(is (nil? (image-buffer/release nil))))

View File

@@ -83,8 +83,8 @@
"7" "rgba(255, 0, 0, 0.65)"
"8" "rgba(255, 0, 0, 0.2)"
"9" "rgba(0, 0, 255, 0.75)"
"1.001,5" "rgba(255, 0, 0, 0.2)"
"1,001.5" "rgba(255, 0, 0, 0.2)"}
"1.001,5" "rgba(0, 0, 255, 0.75)"
"1,001.5" "rgba(0, 0, 255, 0.75)"}
(-> (select-keys query-results [:cols :rows])
(#'table/render-table {:col-names ["a" "b" "c"]
:cols-for-color-lookup ["a" "b" "c"]}

View File

@@ -2,6 +2,7 @@
(:require
[clojure.string :as str]
[clojure.test :refer [deftest is testing use-fixtures]]
[metabase.oauth-server.api.oauth :as api.oauth]
[metabase.oauth-server.core :as oauth-server]
[metabase.test :as mt]
[metabase.test.http-client :as client]
@@ -408,6 +409,84 @@
(is (str/includes? location "error=access_denied"))
(is (str/includes? location "state=test-state")))))))
;;; -------------------------------------- Subpath hosting (GIT-10551) ---------------------------------------------
;; Metabase doesn't officially support being hosted under a subpath, but the OAuth consent flow is entirely
;; driven by site-url, so these tests pin the parts of the flow the *browser* sees — the consent form action,
;; the CSRF cookie path, and the login redirect — to subpath-aware values. The server never sees the subpath
;; (the reverse proxy strips it), so the routes themselves are unchanged.
(defn- extract-csrf-cookie-path
"Extract the Path attribute of the CSRF cookie from the response.
Checks both the :cookies map and the Set-Cookie header (which may be a string or vector of strings)."
[response]
(or (get-in response [:cookies "metabase.OAUTH_CSRF" :path])
(let [set-cookie (get-in response [:headers "Set-Cookie"])
cookies (cond
(string? set-cookie) [set-cookie]
(sequential? set-cookie) (vec set-cookie)
:else [])]
(some #(when (and (string? %) (str/includes? % "metabase.OAUTH_CSRF="))
(second (re-find #"Path=([^;]+)" %)))
cookies))))
(deftest csrf-cookie-path-follows-site-url-test
(doseq [[site-url expected-path] {"http://localhost:3000" "/oauth/authorize"
"http://localhost:3000/metabase" "/metabase/oauth/authorize"
"http://localhost:3000/metabase/" "/metabase/oauth/authorize"
"http://localhost:3000/bi/metabase" "/bi/metabase/oauth/authorize"}]
(testing (str "site-url " site-url)
(mt/with-temporary-setting-values [site-url site-url]
(is (= expected-path (:path (#'api.oauth/csrf-cookie-opts 600))))))))
(deftest subpath-consent-flow-test
(mt/with-temporary-setting-values [site-url "http://localhost:3000/metabase"]
(t2/with-transaction [_conn nil {:rollback-only true}]
(let [client (create-test-client!)
client-id (:client_id client)
consent-resp (get-consent-page! :crowberto client-id)
consent-body (:body consent-resp)]
(testing "the consent form posts to the absolute, subpath-prefixed decision URL"
(is (str/includes? consent-body
"action=\"http://localhost:3000/metabase/oauth/authorize/decision\"")))
(testing "the CSRF cookie path includes the subpath so the browser sends the cookie with the form POST"
(is (= "/metabase/oauth/authorize" (extract-csrf-cookie-path consent-resp))))
(testing "the consent -> decision flow still round-trips"
(let [csrf-token (extract-csrf-token-from-consent consent-body)
csrf-cookie (extract-csrf-cookie consent-resp)
params-sig (extract-params-sig-from-consent consent-body)
response (form-post-decision!
:crowberto
{:approved "true"
:csrf_token csrf-token
:params_sig params-sig
:client_id client-id
:redirect_uri "https://example.com/callback"
:response_type "code"
:scope "profile"
:state "test-state"}
302
:csrf-cookie csrf-cookie)
location (get-in response [:headers "Location"])]
(is (str/starts-with? location "https://example.com/callback?"))
(is (str/includes? location "code="))
(testing "the decision response clears the CSRF cookie at the same subpath-prefixed path
(a clear at a different path would leave the old cookie shadowing the next flow)"
(is (= "/metabase/oauth/authorize" (extract-csrf-cookie-path response))))))))))
(deftest subpath-login-redirect-test
(testing "unauthenticated GET /oauth/authorize under a subpath"
(mt/with-temporary-setting-values [site-url "http://localhost:3000/metabase"]
(let [response (client/client-full-response :get 302 "oauth/authorize"
:client_id "some-client"
:redirect_uri "https://example.com/callback"
:response_type "code")
location (get-in response [:headers "Location"])]
(testing "the login page URL carries the subpath"
(is (str/starts-with? location "http://localhost:3000/metabase/auth/login?redirect=")))
(testing "the redirect param stays basename-relative — the SPA router re-adds the subpath"
(is (str/includes? location (str "redirect=" (URLEncoder/encode "/oauth/authorize" "UTF-8")))))))))
(defn- approve-or-deny!
"Drive the consent + decision flow for `client-id` and return the decision response.
`approved?` chooses approve vs deny."

View File

@@ -77,6 +77,21 @@
(is (re-find #"http://localhost:3000/app/assets/img/custom\.png" html)
"relative path should be resolved to absolute URL")))))
(deftest consent-page-subpath-form-action-test
(testing "form action is absolute so it survives hosting under a subpath (GIT-10551)"
(let [html (mt/with-temporary-setting-values [site-url "https://example.com/metabase"]
(consent-page/render-consent-page
{:client-name "Test App"
:oauth-params {:response_type "code" :client_id "abc123"}
:nonce "test-nonce"
:csrf-token "test-csrf"}))]
(is (re-find #"action=\"https://example\.com/metabase/oauth/authorize/decision\"" html))
(testing "bundled fonts are also loaded from under the subpath"
(is (re-find #"url\('https://example\.com/metabase/app/fonts/" html)))))
(testing "at the domain root the action still targets /oauth/authorize/decision"
(let [html (render!)]
(is (re-find #"action=\"http://localhost:3000/oauth/authorize/decision\"" html)))))
(defn- render-with-scopes! [scopes]
(mt/with-temporary-setting-values [site-url "http://localhost:3000"]
(consent-page/render-consent-page

View File

@@ -1,151 +1,12 @@
(ns metabase.sql-parsing.common-test
(:require
[clojure.test :refer :all]
[metabase.sql-parsing.common :as common])
(:import
(io.aleph.dirigiste Pool)
(java.util.concurrent CountDownLatch)))
[metabase.sql-parsing.common :as common]))
(set! *warn-on-reflection* true)
(defn- mock-worker
"Create a mock 'worker' object. The pool doesn't care what type the objects are - it just stores and
returns them."
[id]
{:worker-id id
:created-at (System/currentTimeMillis)})
(defn- counting-generator
"Returns a generator function that counts how many workers it has created."
[counter-atom]
(fn []
(mock-worker (swap! counter-atom inc))))
(defn- failing-generator
"Returns a generator that fails after creating n workers."
[counter-atom fail-after]
(fn []
(let [id (swap! counter-atom inc)]
(when (> id fail-after)
(throw (ex-info "Generator limit reached" {:count id :limit fail-after})))
(mock-worker id))))
;;; ------------------------------------------------- Test Helpers ---------------------------------------------------
(defn- test-pool
"Create a pool for testing with sensible defaults.
Returns [pool created-count-atom destroyed-atom]."
([]
(test-pool {}))
([config]
(let [created (atom 0)
destroyed (atom [])
pool (common/create-pool (counting-generator created)
(fn [worker] (swap! destroyed conj worker))
(merge {:max-size 2, :idle-minutes 1} config))]
[pool created destroyed])))
(defn- with-worker
"Execute f with a worker from pool, releasing it afterwards."
[^Pool pool f]
(let [worker (.acquire pool :test)]
(try
(f worker)
(finally
(.release pool :test worker)))))
(defn- run-concurrent
"Run n threads concurrently, each executing f with a worker from pool.
Returns vector of results."
[pool n f]
(let [latch (CountDownLatch. n)
results (atom [])
futures (doall
(for [i (range n)]
(future
(.countDown latch)
(.await latch)
(let [result (with-worker pool (fn [worker] (f i worker)))]
(swap! results conj result)))))]
(doseq [fut futures] @fut)
@results))
;;; ------------------------------------------------ Pool Size Tests -------------------------------------------------
(deftest pool-respects-maximum-size-test
(testing "Pool never creates more than max workers even under concurrent load"
(let [[pool created] (test-pool {:max-size 2})
num-threads 20
results (run-concurrent pool num-threads
(fn [_i worker]
(Thread/sleep (long (+ 10 (rand-int 50))))
worker))]
(is (<= @created 2)
(str "Pool created " @created " workers but max is 2"))
(is (= num-threads (count results))
"All threads should have received a worker"))))
(deftest pool-reuses-idle-workers-test
(testing "Sequential uses reuse the same worker instead of creating new ones"
(let [[pool created] (test-pool)]
(with-worker pool (fn [_worker] (is (= 1 @created) "First use creates one worker")))
(with-worker pool (fn [worker]
(is (= 1 @created) "Second use reuses the same worker")
(is (= 1 (:worker-id worker))))))))
;;; ------------------------------------------------ Disposal Tests --------------------------------------------------
(deftest pool-dispose-destroys-and-replaces-workers-test
(testing "Disposing a worker calls destroy and the next acquire generates a fresh one"
(let [[pool created destroyed] (test-pool)
worker (.acquire ^Pool pool :test)]
(.dispose ^Pool pool :test worker)
(is (= [worker] @destroyed) "Disposed worker should be destroyed")
(with-worker pool (fn [_worker] (is (= 2 @created) "Disposed worker should be replaced"))))))
;;; --------------------------------------------- Generator Failure Tests --------------------------------------------
(deftest pool-handles-generator-failures-test
(testing "Pool gracefully handles when generator fails"
(let [counter (atom 0)
pool (common/create-pool (failing-generator counter 1) (fn [_worker]) {:max-size 2, :idle-minutes 1})]
(is (some? (with-worker pool identity)) "First worker creation succeeds")
;; Dispose to force new creation
(let [worker (.acquire ^Pool pool :test)]
(.dispose ^Pool pool :test worker))
(is (thrown? Exception (with-worker pool identity)) "Generator failure should propagate"))))
;;; ------------------------------------------- Concurrent Access Tests ----------------------------------------------
(deftest pool-handles-concurrent-access-safely-test
(testing "Pool handles concurrent access without race conditions"
(let [[pool created] (test-pool {:max-size 2})
num-threads 50
results (run-concurrent pool num-threads
(fn [i _worker]
(Thread/sleep (long (rand-int 20)))
i))]
(is (= num-threads (count results)) "All threads should complete successfully")
(is (<= @created 2) (str "Created " @created " workers, expected <= 2")))))
;;; ----------------------------------------- Version Parsing --------------------------------------------------------
(deftest expected-sqlglot-version-test
(testing "Sqlglot version is properly parsed from pyproject.toml"
(let [version (common/expected-sqlglot-version)]
(is (string? version))
(is (re-matches #"\d+\.\d+\.\d+" version)))))
;;; -------------------------------------------- Pool Behavior Tests -------------------------------------------------
(deftest pool-stores-and-returns-objects-test
(testing "Pool correctly stores and returns arbitrary objects"
(let [[pool created] (test-pool)
result1 (with-worker pool identity)
result2 (with-worker pool identity)]
(is (map? result1))
(is (= 1 (:worker-id result1)))
(is (= 1 @created))
(is (map? result2))
(is (= 1 (:worker-id result2))) ;; Same worker ID
(is (= 1 @created)))))

View File

@@ -0,0 +1,143 @@
(ns metabase.util.pool-test
(:require
[clojure.test :refer :all]
[metabase.util.pool :as u.pool])
(:import
(io.aleph.dirigiste Pool)
(java.util.concurrent CountDownLatch)))
(set! *warn-on-reflection* true)
(defn- mock-worker
"Create a mock 'worker' object. The pool doesn't care what type the objects are - it just stores and
returns them."
[id]
{:worker-id id
:created-at (System/currentTimeMillis)})
(defn- counting-generator
"Returns a generator function that counts how many workers it has created."
[counter-atom]
(fn []
(mock-worker (swap! counter-atom inc))))
(defn- failing-generator
"Returns a generator that fails after creating n workers."
[counter-atom fail-after]
(fn []
(let [id (swap! counter-atom inc)]
(when (> id fail-after)
(throw (ex-info "Generator limit reached" {:count id :limit fail-after})))
(mock-worker id))))
;;; ------------------------------------------------- Test Helpers ---------------------------------------------------
(defn- test-pool
"Create a pool for testing with sensible defaults.
Returns [pool created-count-atom destroyed-atom]."
([]
(test-pool {}))
([config]
(let [created (atom 0)
destroyed (atom [])
pool (u.pool/create-pool (counting-generator created)
(fn [worker] (swap! destroyed conj worker))
(merge {:max-size 2, :idle-minutes 1} config))]
[pool created destroyed])))
(defn- with-worker
"Execute f with a worker from pool, releasing it afterwards."
[^Pool pool f]
(let [worker (.acquire pool :test)]
(try
(f worker)
(finally
(.release pool :test worker)))))
(defn- run-concurrent
"Run n threads concurrently, each executing f with a worker from pool.
Returns vector of results."
[pool n f]
(let [latch (CountDownLatch. n)
results (atom [])
futures (doall
(for [i (range n)]
(future
(.countDown latch)
(.await latch)
(let [result (with-worker pool (fn [worker] (f i worker)))]
(swap! results conj result)))))]
(doseq [fut futures] @fut)
@results))
;;; ------------------------------------------------ Pool Size Tests -------------------------------------------------
(deftest pool-respects-maximum-size-test
(testing "Pool never creates more than max workers even under concurrent load"
(let [[pool created] (test-pool {:max-size 2})
num-threads 20
results (run-concurrent pool num-threads
(fn [_i worker]
(Thread/sleep (long (+ 10 (rand-int 50))))
worker))]
(is (<= @created 2)
(str "Pool created " @created " workers but max is 2"))
(is (= num-threads (count results))
"All threads should have received a worker"))))
(deftest pool-reuses-idle-workers-test
(testing "Sequential uses reuse the same worker instead of creating new ones"
(let [[pool created] (test-pool)]
(with-worker pool (fn [_worker] (is (= 1 @created) "First use creates one worker")))
(with-worker pool (fn [worker]
(is (= 1 @created) "Second use reuses the same worker")
(is (= 1 (:worker-id worker))))))))
;;; ------------------------------------------------ Disposal Tests --------------------------------------------------
(deftest pool-dispose-destroys-and-replaces-workers-test
(testing "Disposing a worker calls destroy and the next acquire generates a fresh one"
(let [[pool created destroyed] (test-pool)
worker (.acquire ^Pool pool :test)]
(.dispose ^Pool pool :test worker)
(is (= [worker] @destroyed) "Disposed worker should be destroyed")
(with-worker pool (fn [_worker] (is (= 2 @created) "Disposed worker should be replaced"))))))
;;; --------------------------------------------- Generator Failure Tests --------------------------------------------
(deftest pool-handles-generator-failures-test
(testing "Pool gracefully handles when generator fails"
(let [counter (atom 0)
pool (u.pool/create-pool (failing-generator counter 1) (fn [_worker]) {:max-size 2, :idle-minutes 1})]
(is (some? (with-worker pool identity)) "First worker creation succeeds")
;; Dispose to force new creation
(let [worker (.acquire ^Pool pool :test)]
(.dispose ^Pool pool :test worker))
(is (thrown? Exception (with-worker pool identity)) "Generator failure should propagate"))))
;;; ------------------------------------------- Concurrent Access Tests ----------------------------------------------
(deftest pool-handles-concurrent-access-safely-test
(testing "Pool handles concurrent access without race conditions"
(let [[pool created] (test-pool {:max-size 2})
num-threads 50
results (run-concurrent pool num-threads
(fn [i _worker]
(Thread/sleep (long (rand-int 20)))
i))]
(is (= num-threads (count results)) "All threads should complete successfully")
(is (<= @created 2) (str "Created " @created " workers, expected <= 2")))))
;;; -------------------------------------------- Pool Behavior Tests -------------------------------------------------
(deftest pool-stores-and-returns-objects-test
(testing "Pool correctly stores and returns arbitrary objects"
(let [[pool created] (test-pool)
result1 (with-worker pool identity)
result2 (with-worker pool identity)]
(is (map? result1))
(is (= 1 (:worker-id result1)))
(is (= 1 @created))
(is (map? result2))
(is (= 1 (:worker-id result2))) ;; Same worker ID
(is (= 1 @created)))))