Remove old llms.txt superseded by Jekyll plugin on docs site (#69132)
remove llms.txt and llms-embedding-full.txt superseded by Jekyll build plugin
Co-authored-by: Phoomparin Mano <poom@metabase.com>
Don't use the deprecated enable-query-caching value (#69256)
Don't use the deprecated enable-query-caching value which may have been stored as "false" in the database.
Co-authored-by: Nathan Voxland <nathan@metabase.com>
[Athena] Correctly recognize `:base_type` when db type is uppercase (#68369)
Athena's JDBC driver returns type names with the same capitalization as
the user types into the admin interface, but the
`sql-jdbc.sync/database-type->base-type` implementation for Athena only
recognized lower-case names (which are the defaults, I gather).
I think most databases don't have this problem, since they ignore the
case you typed in e.g. `CREATE TABLE` statements and have the JDBC
driver always return the canonical spelling of the type names.
Co-authored-by: Braden Shepherdson <braden@metabase.com>
Fix reindex connection management (#68973)
Move reindex connection management to use a connection only around the writes, not try to use a connection across all writes, which may commit and therefore close server-side cursors still being used to read the next batch.
Co-authored-by: Nathan Voxland <nathan@metabase.com>
Co-authored-by: dpsutton <dan@dpsutton.com>
* Bigquery and snowflake failing test QUE-3013 (#68916)
BigQuery can't do case statements with different types in each branch. Other dbs can. This fixes a test that ran a case statement with different types to skip BigQuery, but it adds a new assertion to check correct behavior without it.
This also corrects a fault in the SnowFlake driver that incorrectly assumed that convertTimezone() returned a timestamp_ntz but it returns a timestamp_tz. This wraps it in a cast back to timestamp_ntz.
* Whitespace
---------
Co-authored-by: Eric Normand <ericwnormand@gmail.com>
* Don't double-convert expressions with convertTimezone() (#68740)
In the case statement, the default case did not TIMEZONE() to alter the timezone. The cases with TIMEZONE() returned a timestamp. The default case without TIMEZONE() returned timestamptz. Postgres reconciled the different types by converting all of them to timestamptz. So the timestamp was altered (by TIMEZONE()), but a new UTC timestamp was added (by the cast).
When the post-processor formats the rows, it checks that the metadata for the column has :converted_timezone. If it has that, it is formatted in the converted timezone, and since it's UTC, it will convert it again, a second time, which is incorrect.
The fix here is to check if the returned column is a timestamptz and strip it before formatting. We assume the database has already done the conversion. This avoids the second conversion.
* Fix import problem
---------
Co-authored-by: Eric Normand <ericwnormand@gmail.com>
Change log level from info to debug for ignored params on qp (#68748)
reducing a bit more the verbosity of our logs, this time on cards with old metadata
Co-authored-by: da <paoliniluis@gmail.com>
Co-authored-by: dpsutton <dan@dpsutton.com>
* implicit joins get implicit fields added (#68394)
Say you are doing an implicit join from orders to products in order to summarize by products.category. Previously, with implicit joins, we join with the actual products table instead of a select blah from products wrapper, and so the various fields returned by the join were returned under their original name. However, the code would abbreviate excessively long column names from the joined table and break the query.
That difference was due to add-implicit-fields running before add-implicit-joins. Explicit joins got implicit fields added, while implicit joins didn't get those fields. This pr runs add-implicit-fields a second time after add-implicit-joins, which makes implicit and explicit joins compile to the same (correct) sql.
* Fix bigquery implicit join tests
* rename dataset to fix caching
* bump tests
* update comments to account for data loading hacks
* make athena tests use current version of test-data
* revert comment changes because of other work
* bump tests
---------
Co-authored-by: William <william@metabase.com>
prometheus metrics for jvm startup (#68728)
logs:
```
2026-01-27 15:02:23,442 INFO util.queue :: Starting listener field-value-invalidate with 1 threads 🎧
2026-01-27 15:02:23,448 INFO util.queue :: Starting listener search-index-update with 1 threads 🎧
2026-01-27 15:02:23,448 INFO core.core :: Metabase Initialization COMPLETE
2026-01-27 15:02:23,453 INFO core.core :: Metabase Initialization COMPLETE in 4.8 s (JVM uptime: 57.8 s)
```
metrics:
```
❯ http get :9191/metrics | grep startup
metabase_startup_jvm_to_complete_millis 57764.0
metabase_startup_init_duration_millis 4816.0
```
Note that there two different notions of startup time: from jvm
instantiation, and from before init! was called. This maintains the two
separately so we can diagnose if class load time gets longer
independently of amount of work we do during startup itself
Co-authored-by: dpsutton <dan@dpsutton.com>
I have a suspicion that the token check query is taking a long
time. Timestamps in some logs are
2026-01-27 00:06:11.561 {"m":"Checking with the MetaStore to see whether token...
2026-01-27 00:06:28.951 {"lvl":"INFO","lgr":"metabase.premium-features.token-check","cluster":"metabase-1","m":"Reporting Metabase stats:
2026-01-27 00:06:28.951
{"lvl":"INFO","lgr":"metabase.premium-features.token-check","cluster":"metabase-1","m":"Reporting
Metabase stats: Cannot throw exception because the return value of \"java.lang.Throwable.getCause()\" is null"
So:
I'm memoizing the token request stats for 2 minutes. This is not long
enough to affect reported numbers, just to make sure that we
successively checking we don't need to hit the db again
and don't rethrow an empty (.getCause e). I don't understand why it
would be empty at all but I'm defensive here and logging
Better logging around token checks (#68625)
* Better logging around token checks
We used to just log we were checking but not give error responses. So it
would look like a ton of token checks for no real reason.
Now includes error responses if any, and also include logs when engaging
the token check circuit breaker, and when it is "half open", when it
lets one real token check happen every 30 seconds to see if the service
is back.
```
2026-01-23 12:26:47,175 INFO premium-features.token-check :: Checking with the MetaStore to see whether token 'f325...d01a' is valid...
2026-01-23 12:26:47,186 INFO premium-features.token-check :: Reporting Metabase stats: {:embedding-dashboard-count 0, :enabled-embedding-sdk false, :internal-users 1, :embedding-question-count 0, :query_executions_static_embed 0, :query_executions_public_link 0, :domains 1, :metabot-usage-date 2026-01-22, :enabled-embedding-static false, :enabled-embedding-simple false, :metabot-queries 0, :external-users 0, :metabot-tokens 0, :query_executions_simple_embed 0, :enabled-embedding-interactive false, :query_executions_internal 0, :metabot-users 0, :query_executions_sdk_embed 0, :query_executions_interactive_embed 0, :users 1}
2026-01-23 12:26:49,190 INFO premium-features.token-check :: Error checking token: Connect timed out †
2026-01-23 12:26:49,194 INFO premium-features.token-check :: Checking with the MetaStore to see whether token 'f325...d01a' is valid...
2026-01-23 12:26:49,201 INFO premium-features.token-check :: Reporting Metabase stats: {:embedding-dashboard-count 0, :enabled-embedding-sdk false, :internal-users 1, :embedding-question-count 0, :query_executions_static_embed 0, :query_executions_public_link 0, :domains 1, :metabot-usage-date 2026-01-22, :enabled-embedding-static false, :enabled-embedding-simple false, :metabot-queries 0, :external-users 0, :metabot-tokens 0, :query_executions_simple_embed 0, :enabled-embedding-interactive false, :query_executions_internal 0, :metabot-users 0, :query_executions_sdk_embed 0, :query_executions_interactive_embed 0, :users 1}
2026-01-23 12:26:51,205 INFO premium-features.token-check :: Engaging circuit breaker in token check ††
2026-01-23 12:26:51,206 INFO premium-features.token-check :: Error checking token: Connect timed out
2026-01-23 12:27:44,548 INFO premium-features.token-check :: In token check circuit breaker but attempting check
2026-01-23 12:27:44,551 INFO premium-features.token-check :: Checking with the MetaStore to see whether token 'f325...d01a' is valid...
2026-01-23 12:27:44,565 INFO premium-features.token-check :: Reporting Metabase stats: {:embedding-dashboard-count 0, :enabled-embedding-sdk false, :internal-users 1, :embedding-question-count 0, :query_executions_static_embed 0, :query_executions_public_link 0, :domains 1, :metabot-usage-date 2026-01-22, :enabled-embedding-static false, :enabled-embedding-simple false, :metabot-queries 0, :external-users 0, :metabot-tokens 0, :query_executions_simple_embed 0, :enabled-embedding-interactive false, :query_executions_internal 0, :metabot-users 0, :query_executions_sdk_embed 0, :query_executions_interactive_embed 0, :users 1}
2026-01-23 12:27:46,574 INFO premium-features.token-check :: Engaging circuit breaker in token check
2026-01-23 12:27:46,575 INFO premium-features.token-check :: Error checking token: Connect timed out
...
2026-01-23 12:32:50,432 INFO premium-features.token-check :: In token check circuit breaker but attempting check †††
2026-01-23 12:32:50,433 INFO premium-features.token-check :: Checking with the MetaStore to see whether token 'f325...d01a' is valid...
2026-01-23 12:32:50,444 INFO premium-features.token-check :: Reporting Metabase stats: {:embedding-dashboard-count 0, :enabled-embedding-sdk false, :internal-users 1, :embedding-question-count 0, :query_executions_static_embed 0, :query_executions_public_link 0, :domains 1, :metabot-usage-date 2026-01-22, :enabled-embedding-static false, :enabled-embedding-simple false, :metabot-queries 0, :external-users 0, :metabot-tokens 0, :query_executions_simple_embed 0, :enabled-embedding-interactive false, :query_executions_internal 0, :metabot-users 0, :query_executions_sdk_embed 0, :query_executions_interactive_embed 0, :users 1}
2026-01-23 12:32:52,450 INFO premium-features.token-check :: Engaging circuit breaker in token check
2026-01-23 12:32:52,450 INFO premium-features.token-check :: Error checking token: Connect timed out
```
The highlights are now we include:
- † `Error checkign token: Connect timed out` the error response when
appropriate
- †† `Engaging circuit breaker in token check` when we engage the
circuit breaker to no longer do real token checks
- ††† `In token check circuit breaker but attempting check` indicating
we are doing a real one from the circuit breaker to determine if the
service is restored
* Whew. Lots of stuff
add a namespace to the circuit breaker keyword:
`:token-check/circuit-breaker`. Change the keyword holding
`:token-check/app-db-not-ready` to cause. I had used reason but nothing
checked that.
this is because want to log when a token check fails, but not when it is
- failure due to circuit breaker
- failure due to appdb not ready yet.
token checks can happen anywhere and with an enormous frequency. they
fail before the appdb is setup (we can't check settigns yet) and of
course they fail during the circuit breaker opening. that's the point of
the circuit breaker. So we don't have to log these.
We _do_ want to log other failures.
We also want to do some prometheus metrics. Here's a journey:
You previously couldn't import the prometheus analytics namespace
because analytics.settings required appearance.settings. It does so
because a docstring in analytics settings used to include Metabase in
the string. I removed that so now it says "in order to help us
improve". And then things would be smooth sailing.
Except we had an issue.
```
026-01-23 20:50:03,514 INFO analytics.prometheus :: Starting prometheus metrics collector
2026-01-23 20:50:03,539 INFO analytics.prometheus :: Starting prometheus metrics web-server on port 9191
thread229238 we made the system!
thread229238 and now we are done with the lock!
2026-01-23 20:50:03,607 INFO analytics.prometheus :: Starting prometheus metrics web-server on port 9191
2026-01-23 20:50:03,611 ERROR premium-features.token-check :: Error validating token: Failed to initialize Prometheus on port 9,191
2026-01-23 20:50:03,611 INFO analytics.prometheus :: Starting prometheus metrics web-server on port 9191
2026-01-23 20:50:03,612 INFO analytics.prometheus :: Starting prometheus metrics web-server on port 9191
2026-01-23 20:50:03,614 INFO analytics.prometheus :: Starting prometheus metrics web-server on port 9191
2026-01-23 20:50:03,616 INFO analytics.prometheus :: Starting prometheus metrics web-server on port 9191
2026-01-23 20:50:03,617 INFO analytics.prometheus :: Starting prometheus metrics web-server on port 9191
2026-01-23 20:50:03,618 INFO analytics.prometheus :: Starting prometheus metrics web-server on port 9191
2026-01-23 20:50:03,619 INFO analytics.prometheus :: Starting prometheus metrics web-server on port 9191
2026-01-23 20:50:03,619 INFO analytics.prometheus :: Starting prometheus metrics web-server on port 9191
2026-01-23 20:50:03,653 INFO core.core :: Setting up prometheus metrics
```
We have locking around the system but it is starting tons of times! Lots
of debugging, swearing that this can't happen. but then i finally threw
in my favorite trick: `(clojure.repl/pst (Exception. "how did we get
here"))`. And it turns out we are doing this:
1. token check fails, so we call `prometheus/inc!`
2. inc is helpful, sees system isn't yet setup, and sets it up for us
3. Setup metric system
4. setup metrics in system
5. we hit the following code which gets initial values for labels
```clojure
(defn- initial-labelled-metric-values []
(for [metric (keys (methods known-labels))
labels (known-labels metric)]
{:metric metric
:labels labels
:value (initial-value metric labels)}))
```
6. We hit the following search code about which search we have
```clojure
(defmethod analytics/initial-value :metabase-search/engine-active
[_ {:keys [engine]}]
(if (search.engine/supported-engine? (keyword "search.engine" engine))
1 0))
```
7. This takes us to semantic search:
```clojure
(defmethod search.engine/supported-engine? :search.engine/semantic [_]
(try
(supported?)
(catch Exception e
(log/warn e "Semantic search engine not supported")
false)))
```
8. `supported?` here checks to see if you support a particular search
impelementation, which of course depends on _YOUR FEATURES_.
9. Checking features fails, so we should increment our prometheus
counter
10. the system is not yet initialized yet. Let's initialize it. (this is
also step 2)
So I have added a simple `inc-if-initialized!` function. It's not
important if the system is down. It's ok if these events are lost. The
system will be initialized in core anyways.
```
❯ http get :9191/metrics | grep metabase_token
HELP metabase_token_check_attempt_total Total number of token checks. Includes a status label.
TYPE metabase_token_check_attempt_total counter
metabase_token_check_attempt_total{status="failure",} 8.0
```
logs:
```
2026-01-23 22:58:26,668 INFO premium-features.token-check :: Checking with the MetaStore to see whether token '82bf...c3fe' is valid...
2026-01-23 22:58:28,706 INFO premium-features.token-check :: Error checking token: Connect timed out
2026-01-23 22:58:28,750 INFO core.core :: Setting up prometheus metrics
2026-01-23 22:58:28,751 INFO analytics.prometheus :: Starting prometheus metrics collector
2026-01-23 22:58:28,789 INFO premium-features.token-check :: Checking with the MetaStore to see whether token '82bf...c3fe' is valid...
2026-01-23 22:58:30,793 INFO premium-features.token-check :: Error checking token: Connect timed out
2026-01-23 22:58:30,798 INFO analytics.prometheus :: Starting prometheus metrics web-server on port 9191
...
2026-01-23 22:58:53,441 INFO premium-features.token-check :: Checking with the MetaStore to see whether token '82bf...c3fe' is valid... {mb-quartz-job-type=SyncAndAnalyzeDatabase}
...
2026-01-23 22:58:55,445 INFO premium-features.token-check :: Engaging circuit breaker in token check {mb-quartz-job-type=SyncAndAnalyzeDatabase}
2026-01-23 22:58:55,445 INFO premium-features.token-check :: Error checking token: Connect timed out {mb-quartz-job-type=SyncAndAnalyzeDatabase}
```
```
Exception how here??
metabase.analytics.prometheus/setup! (prometheus.clj:584)
metabase.analytics.prometheus/setup! (prometheus.clj:576)
metabase.analytics.prometheus/inc! (prometheus.clj:668)
metabase.analytics.prometheus/inc! (prometheus.clj:658)
metabase.analytics.prometheus/inc! (prometheus.clj:665)
metabase.analytics.prometheus/inc! (prometheus.clj:658)
metabase.premium-features.token-check/error-catching-token-checker/reify--98262 (token_check.clj:431)
metabase.premium-features.token-check/check-token (token_check.clj:488)
metabase.premium-features.token-check/check-token (token_check.clj:484)
metabase.premium-features.token-check/check-token (token_check.clj:486)
metabase.premium-features.token-check/check-token (token_check.clj:484)
metabase.premium-features.token-check/eval98325/*token-features*98348--98349 (token_check.clj:534)
metabase.premium-features.token-check/eval98325/fn--98353 (token_check.clj:529)
metabase.premium-features.token-check/has-feature? (token_check.clj:575)
metabase.premium-features.token-check/has-feature? (token_check.clj:569)
clojure.lang.Var.invoke (Var.java:386)
metabase.premium-features.defenterprise/check-feature (defenterprise.clj:41)
metabase.premium-features.defenterprise/check-feature (defenterprise.clj:35)
metabase.premium-features.defenterprise/dynamic-ee-oss-fn/fn--71956 (defenterprise.clj:53)
metabase.search.semantic.core/eval180600/fn--180601 (core.clj:58)
clojure.lang.MultiFn.invoke (MultiFn.java:229)
metabase.search.core/eval139302/fn--139304 (core.clj:77)
clojure.lang.MultiFn.invoke (MultiFn.java:234)
metabase.analytics.prometheus/initial-labelled-metric-values/iter--97802--97808/fn--97809/iter--97804--97810/fn--97811 (prometheus.clj:521)
clojure.lang.LazySeq.force (LazySeq.java:50)
clojure.lang.LazySeq.realize (LazySeq.java:89)
clojure.lang.LazySeq.seq (LazySeq.java:106)
clojure.lang.RT.seq (RT.java:555)
clojure.core/seq--5488 (core.clj:139)
clojure.core/concat/fn--5579 (core.clj:727)
clojure.lang.LazySeq.force (LazySeq.java:50)
clojure.lang.LazySeq.realize (LazySeq.java:89)
clojure.lang.LazySeq.seq (LazySeq.java:106)
clojure.lang.Cons.next (Cons.java:41)
clojure.lang.RT.next (RT.java:733)
clojure.core/next--5472 (core.clj:64)
clojure.core/next--5472 (core.clj:64)
metabase.analytics.prometheus/setup-metrics! (prometheus.clj:546)
metabase.analytics.prometheus/setup-metrics! (prometheus.clj:533)
metabase.analytics.prometheus/make-prometheus-system (prometheus.clj:60)
metabase.analytics.prometheus/make-prometheus-system (prometheus.clj:55)
metabase.analytics.prometheus/setup!/fn--97866 (prometheus.clj:591)
metabase.analytics.prometheus/setup! (prometheus.clj:590)
metabase.analytics.prometheus/setup! (prometheus.clj:576)
metabase.analytics.prometheus/inc! (prometheus.clj:668)
metabase.analytics.prometheus/inc! (prometheus.clj:658)
metabase.analytics.prometheus/inc! (prometheus.clj:665)
metabase.analytics.prometheus/inc! (prometheus.clj:658)
metabase.premium-features.token-check/error-catching-token-checker/reify--98262 (token_check.clj:431)
metabase.premium-features.token-check/check-token (token_check.clj:488)
```
* module config fixup
Co-authored-by: dpsutton <dan@dpsutton.com>
Delete from cache in batches of 1000 (#68622)
There's apparently an instance on cloud that is trying to remove 70k queries from the cache at once, and postgres
doesn't allow 70k params to a query. Batching the deletes in groups of 1000 should fix that.
Co-authored-by: William <william@metabase.com>
* Delete invalid `source-card-id`s on native SQL questions (#68100)
When a user creates a question based on a model, `source_card_id` was set to the model's ID. If that question was later converted to native SQL, `source_card_id` wouldn't get cleared. This means that deleting the model would cause the question to be deleted even though they no longer have any relationship to one another.
The code fix won't help anyone whose data is already incorrect, so this adds a migration to correct the inconsistent data. I added an assertion to the t2 hook to catch this at dev time in the future. And ultimately, the delete cascade seems overkill, so there's a migration to replace that with SET NULL instead.
* Move 057 migrations to 056.
handle empty error message (#68581)
Fixes https://github.com/metabase/metabase/issues/68578
This issue is not about the stack overflow per-se but blowing up on a
string includes:
```javascript
{
"app": "metabase",
"time": "2026-01-22T12:08:53.660601485Z",
"stream": "stderr",
"app_version": "v1.57.10-X01",
"message": "Exception in thread \"Thread-41\" java.lang.NullPointerException: Cannot invoke \"Object.toString()\" because \"s\" is null
at clojure.string$includes_QMARK_.invokeStatic(string.clj:377)
at clojure.string$includes_QMARK_.invoke(string.clj:373)
at metabase.app_db.cluster_lock$retryable_QMARK_.invokeStatic(cluster_lock.clj:27)
at metabase.app_db.cluster_lock$retryable_QMARK_.invoke(cluster_lock.clj:19)
at metabase.app_db.cluster_lock$do_with_cluster_lock.invokeStatic(cluster_lock.clj:87)
at metabase.app_db.cluster_lock$do_with_cluster_lock.invoke(cluster_lock.clj:65)
at metabase.app_db.cluster_lock$do_with_cluster_lock.invokeStatic(cluster_lock.clj:79)
at metabase.app_db.cluster_lock$do_with_cluster_lock.invoke(cluster_lock.clj:65)
at metabase.search.task.search_index$init_BANG_.invokeStatic(search_index.clj:36)
at metabase.search.task.search_index$init_BANG_.invoke(search_index.clj:32)
at clojure.lang.AFn.run(AFn.java:22)
at java.base/java.lang.Thread.run(Unknown Source)",
"namespace": "hosting-eb9b3875e12d465a",
"container": "metabase",
"cluster": "metabase-1",
"_p": "F"
}
```
The error message itself comes from a stackoverflow which perhaps does
not have a message? Either way, it's possible for this to be missing so
be a bit defensive
Co-authored-by: dpsutton <dan@dpsutton.com>
Tweak data loading behavior for a test after MySQL 9.6.0 release (#68634)
Closes DEV-1289
### Description
Within the last day or two, `metabase.query-processor-test.remapping-test/self-referencing-test` began failing on MySQL Latest. I hypothesize this has to do with the [latest release](https://dev.mysql.com/doc/relnotes/mysql/9.6/en/news-9-6-0.html) fixing an unknown bug around batch inserts in InnoDB and the deprecation of a recusive [sic] fk-related flag.
The error is thrown not in the test conditions, but in the test setup -- MySQL apparently now checks foreign key constraints row-per-row even on a batch insert.
This adds a flag that the test can set which changes the test data-loading behavior to disable the FK check before insertion and re-enable it after.
Co-authored-by: Timothy S. Dean <timothy.dean@metabase.com>
Bump toucan2 version to fix dashboards sometimes failing to load (#68494)
If the first table mentioned by a dashboard has no fields (presumably because the table and its field are all
inactive), /api/dashboard/<id>/query_metadata would fail due to a toucan2 bug. This bumps toucan2 to fix the bug and
adds a test to verify that this case now works.
Co-authored-by: William <william@metabase.com>
Deduplicate field refs by ignoring `:source-field-name` unless it matters (#68166)
* Tentative fix for #67808 -- deduplicate by ignoring :source-field-name if possible
* Make sure `:fields` remains a vector
* Add note
* cljfmt
Co-authored-by: Cam Saul <1455846+camsaul@users.noreply.github.com>
Transforms job runs keep trying other transforms if one fails. (#68141)
Previously, when running a transform job (e.g., daily), it would fail if a single transform in the run failed. This PR makes it so that the job continues to run transforms that don't depend on any failed transform. As previously, if any transform in the run fails, the job is marked as failed.
Co-authored-by: Eric Normand <ericwnormand@gmail.com>
Fix Horror Show fingerprint OOMs (#68391)
* Fix Horror Show fingerprint OOMs
This may likely fix many OOMs that others are seeing. Anyone with large
schema will likely benefit from this.
```clojure
(let [database (toucan2.core/select-one :model/Database :id 2)]
(metabase.sync.analyze.fingerprint/fingerprint-fields-for-db!
database (fn [s t] (println "synced " s "for " (:name t)))))
```
This is how we fingerprint fields. DB 2 here is "Horror Show", a
database with 200 tables, each of which has 900 fields in it. This
database has zero rows in it, it it just a large schema.
Syncing this recently has caused huge cpu spikes and then OOM errors
until the instance is killed.
Profiling locally with
```
MB_JETTY_PORT=3091 java "$(socket-repl 6091)" -Xmx2g -jar metabase.jar
```
and visual vm hooked up, I run
```clojure
(let [database (toucan2.core/select-one :model/Database :id 2)]
(metabase.sync.analyze.fingerprint/fingerprint-fields-for-db!
database (fn [s t] (println "synced " s "for " (:name t)))))
```
And observe this behavior:
The ram usage approaches the 2gb limit and then bounces there. The CPU
begins to spike erratically and spend 10% of its time in GC.
Taking a heap dump we can see that we have 2.8 million ResultSetMetaData
objects occuping 183 Mb.
The ResultSetMetaData are closed over and kept in the metadata of the
snake hating maps stored in the cached metadata provider.
This brings into stark relief one simple way to fix this: don't use a
caching metadata provider. This PR just raising the body of the
fingerprint-tables-for-db function to not include
`(qp.store/with-metadata-provider (u/the-id database)` . There is no
benefit to caching here as this is an enumeration of the database. It is
by design not meant to reuse anything. If we didn't OOM we would end up
with 200 tables and 200*900 = 180,000 fields in the cache, each of which
was accessed only once.
That is what this PR fixes.
I also followed one other thread: why are we holding so many
ResultSetMetaData objects in memory. This seems to come about from a
change in https://github.com/metabase/metabase/pull/62330 specifically
this line:
https://github.com/metabase/metabase/blob/master/src/metabase/lib_be/metadata/jvm.clj#L63
```clojure
(mu/defn instance->metadata
"Convert a (presumably) Toucan 2 instance of an application database model with `snake_case` keys to a MLv2 style
metadata instance with `:lib/type` and `kebab-case` keys."
[instance :- :map
metadata-type :- :keyword]
(let [normalize (if-let [schema (get metadata-type->schema metadata-type)]
(fn [instance]
(lib.normalize/normalize schema instance))
identity)]
(-> instance
(perf/update-keys memoized-kebab-key)
(assoc :lib/type metadata-type)
normalize
u.snake-hating-map/snake-hating-map
;; the line below here captures the result coming back from
;; toucan and has a closure over
;; toucan2.jdbc.read/make-column-thunk which has a closure over
;; (.getMetaData rset)
(vary-meta assoc :metabase/toucan-instance instance))))
```
As it stands without any changes, the instance gets quite unstable and
falls over. If you comment out the `var-meta` form above, the graph
returns to a very normal sawtooth graph of memory usage and
reclaimation. But central to this is that it does not need to retain any
of this memory at all. But without this metadata containing the instance
and db classes, we only cache the table and field data and the
fingerprinting can finish. When we cache this stuff and also
inadvertently keep these db classes, we quite quickly run out of
headroom and the instance becomes unstable and falls over.
* fix kondo lint (removes a legacy usage as well :) 💰
Co-authored-by: dpsutton <dan@dpsutton.com>
Custom ref schema that reuses the cache (#68379)
Add ref schema caching
Introduces a custom :ref IntoSchema that permits
reuse of Schema objects for references when safe to do so.
Early tests indicate a dramatic reduction in heap retention, after
loading a single dashboard, I see a 10x resident heap reduction from ~300MB to ~30MB.
It should also act as knot tie / runtime combinatorics limiter, so we should not see 'leaky' growth at runtime.
More work could be done as there are still _some_ gaps here.
- Bare pointers (keywords) can still lead to exponential allocation
- We do not cache when :documentation is present on the ref
(as it would specialise the Schema object delivered for that specific reference)
Co-authored-by: Dan Stone <wotbrew@gmail.com>
Log error instead of warning if a subscription is not sent through the channel rather than a warning (#68214)
* Why is it a warning when it should be an error?
* Fix error logging format in notification worker
* Improve error logging in notification send process
Fix silly mistakes that I do when I'm sleepy
Co-authored-by: da <paoliniluis@gmail.com>
Uxw 2596 color range on table crashes it if values span a large scale (#68022)
# Conflicts:
# e2e/test/scenarios/models/models-list-view.cy.spec.js
# jest.config.js
# jest.tz.unit.conf.js
# yarn.lock
Co-authored-by: Jakub Chodorowicz <jakub.chodorowicz@metabase.com>
Increase verbosity in the caching middleware (#68121)
* Increase verbosity in the caching middleware
* Fix linter
* Make it simpler
* Broke so many tests
* Make the linter happy
* Fix tests (going back to old implementation)
Co-authored-by: da <paoliniluis@gmail.com>
UXW-2586: Acquire locks before changing data permissions (#67201)
* Acquire locks before changing data permissions
Data permission have a read-modify-write cycle:
- we read the existing permissions to see what's currently set
- then we decide what to do (e.g. delete the existing DB level
permissions to write a granular table-level permission)
- then we do the write
This has a race condition if multiple Metabase instances are modifying
permissions at the same time.
For example, say we have:
Initial state: `{:db_id 1 :table_id nil :perm_value :unrestricted}`
Instance A: set table 1 to :blocked
Instance B: set database to :blocked
T1: A reads state, sees db-level permission, plans to expand to table-level
T2: B reads state, sees db-level permission, plans to replace it
T3: A deletes db-level, inserts table-level rows for all tables
T4: B deletes db-level (no-op), inserts new db-level :blocked
Result (CORRUPT):
```
{:db_id 1 :table_id 1 :perm_value :blocked} ;; from A
{:db_id 1 :table_id 2 :perm_value :unrestricted} ;; from A
{:db_id 1 :table_id nil :perm_value :blocked} ;; from B
```
Both db-level AND table-level permissions now exist.
For the permissions API, we take a cluster lock, but sync operations
bypassed this cluster lock. This PR creates a lower-level cluster lock
per `[database, group, perm-type]` so only one instance/thread can do
the read/decide/write cycle for a particular db/group/perm at a time.
* Add a test for race conditions setting data perms
This took a while before I was able to track down the fact that using
`(mt/with-temp )` caused both threads to share the same connection, so
they could obtain the lock simultaneously 🤦
* Use a slightly coarser lock
I realized that if an instance has a huge number of groups, we'll obtain
a lock and ask Postgres to maintain an active savepoint for each group,
which might be untenable.
I think the best solution here is to use slightly coarser locks. This
might increase contention slightly, since we won't be able to change
permissions for `db 1, group 1, perm type view-data` at the same time
that we change permissions for `db 1, group 2, perm type view-data`. But
I think the tradeoff here is necessary.
Co-authored-by: John Swanson <john.swanson@metabase.com>
fix: limit can_run_adhoc_query hydration (#68028)
We can't hydrate unbounded collections of cards with the
can_run_adhoc_query hydration since it linearly adds db queries to the
request. Users with > 10000k can have severe CPU/memory issues
processing this.
Co-authored-by: Edward Paget <edpaget@users.noreply.github.com>
Previously if you had:
```
A (curate)
B (curate)
C (no permissions)
D (curate)
```
Then if C was archived, you would be unable to move or archive A or
B.
Co-authored-by: John Swanson <john.swanson@metabase.com>
* Fix `text-secondary` not working on dark theme on charts
* Support all colors so it's easier to reason about
* Add tests
* Fix type error
* Revert part of the code that's not used anymore
* Add test snapshots
* Make overriding font more consistency across environments
e.g. From storybook to SDK to the main app.
* Fix formatting
Co-authored-by: Mahatthana (Kelvin) Nomsawadi <me@bboykelvin.dev>
Co-authored-by: Aleksandr Lesnenko <alxnddr@users.noreply.github.com>
* fix MB in an iframe running on a subpath
* fix
* extracted out the util fn and added unit tests for it
* addressed comments
* fixed unit tests
Co-authored-by: Sébastien <sebastien@metabase.com>
* Fix parsing of :ERROR chunks in metabase-enterprise.metabot-v3.util/aisdk->messages
There was a typo in the case expr where :ERROR chunks were being handled like :DATA chunks, but in fact should be
handled like :TEXT chunks. This was resulting in an error like
1. Unhandled java.lang.ClassCastException
class java.lang.String cannot be cast to class
clojure.lang.Associative (java.lang.String is in module java.base
of loader 'bootstrap'; clojure.lang.Associative is in unnamed
module of loader 'app')
RT.java: 847 clojure.lang.RT/assoc
core.clj: 193 clojure.core/assoc
core.clj: 192 clojure.core/assoc
util.clj: 56 metabase-enterprise.metabot-v3.util/aisdk->messages/fn/fn
...
whenever an :ERROR chunk was encountered as we tried to ASSOC a string where we were expecting a map.
https://v4.ai-sdk.dev/docs/ai-sdk-ui/stream-protocol#text-parthttps://v4.ai-sdk.dev/docs/ai-sdk-ui/stream-protocol#error-part059d904d6e/ai_service/ai_sdk.py (L50)059d904d6e/ai_service/ai_sdk.py (L203)059d904d6e/ai_service/ai_sdk.py (L324)
* Add some error chunk parsing tests to aisdk-line-parse-test
* Mark aisdk-line-parse-test as ^:parallel
This does io to read some files, but should be threadsafe as no other tests read or modify those files.
* Add make-mock-error-stream-response test util
And rename make-mock-stream-response to make-mock-text-stream-response
* Add client-error-test to test streaming-request with error chunks in response
Co-authored-by: appleby <86076+appleby@users.noreply.github.com>
We have a grace periods and circuit breakers for token checks. But one
issue is that if the token check doesn't throw an error we can't catch
it.
An easy way to see this is to use a local network address that won't
respond like 192.168.random.random (i used 192.168.40.40) then see the
behavior. Without the timeouts, the threads all just get stuck here and
the page never loads. With timeouts, you'll see a few stutters until the
circuit breaker steps in, at which point you have a non-enterprise
version.
https://www.loom.com/share/6ae056dc126b48c9abe8a7ad9366fd05
Co-authored-by: dpsutton <dan@dpsutton.com>
* Run suggested prompts generator as admin & fix test
* Add a prompt suggestions GET perms test
* small cleanups
* use mbql5
Co-authored-by: Noah Moss <32746338+noahmoss@users.noreply.github.com>
Updated curl command formatting and clarified instructions for exporting and importing Metabase data.
Co-authored-by: Alex Yarosh <alexandra@metabase.com>
* kick ci
* Disable remaps when calculating columns for a query.
* Add tests for result_metadata remappings
* Add comments
* Whitespace
Co-authored-by: Eric Normand <ericwnormand@gmail.com>
Fixes#65838.
This issue was about dashboard filters showing the wrong value when
the filter was bound to an FK field from a table with multiple FKs to
the same foreign table. The dashboard parameters logic on the BE was
choosing which FK to use for the filter values arbitrarily, meaning that
binding the filter to one of them would work fine, but the other would
give incoherent results.
The fix is to be deliberate about selecting the parameter's column. That
doesn't solve all ambiguous cases, but we really can't help if there's
a multi-hop join that has parallel edges in the middle!
Co-authored-by: Braden Shepherdson <braden@metabase.com>
* Prevent unnecessary height calculation
* Prioritize local layout from `react-grid-layout` for height calculation
* Refix the issue for real
* Add tests
Co-authored-by: Mahatthana (Kelvin) Nomsawadi <me@bboykelvin.dev>
* fix: hide untested and unsupported db engines in table editing section
* move allowed engines to constants file
* test: add hidden editing toggle test case
Co-authored-by: Jakub Chodorowicz <jakub.chodorowicz@metabase.com>
* Default to base-type when it differs from effective type
* Make 'er parallel
* Does this clause make it better?
* Remove parallel.
* New test and fix for two literals
* Use y-type for effective type
* Update test to try number
Co-authored-by: Eric Normand <ericwnormand@gmail.com>
* Add failing test for #63822, DEV-1190
* Add new feature flag
* Update FE config
* Fix tests 🔧
* Disable for 7 more drivers
* Fix MySQL + MariaDB
* Fix TypeScript
Co-authored-by: Cam Saul <1455846+camsaul@users.noreply.github.com>
* fix: custom home page settings are not being exported
* fix: update seralization baseline test
Co-authored-by: Jakub Chodorowicz <jakub.chodorowicz@metabase.com>
* Don't expose `question` in SDK's `entityTypes`
Because it's only applicable to staged picker
* Fix type
* Clarify the use of `question` in SDK's `entityTypes`
* Fix SDK not building
Co-authored-by: Mahatthana (Kelvin) Nomsawadi <me@bboykelvin.dev>
* fix: respect time format in exlude date picker filter QUE-2611
* rework: move constant out of function body
* test: fix failing tests due to lack of redux provider
* fix: add missing redux provider in story
* test: improve testing logic for ExcludeDatePicker
* internal: use specialized useSetting hook
* test: adjust looping with async action
Co-authored-by: Jakub Chodorowicz <jakub.chodorowicz@metabase.com>
* Check that the number was parsed correctly
* Don't parse int columns that are remapped
* oops, remove old change
* Better fix and specific test
* kick ci
* kick ci
* Make sure it still hits the bigint branch
* Move check into parseNumber()
* Move check back to value.tsx
* Single describe in tests
* Add blank lines (fe-lint)
Co-authored-by: Eric Normand <ericwnormand@gmail.com>
* prototype
* open chat sidebar on navigate
* update copy for new route
* use nlq profile and make sticky, redirect to empty query builder if no navigate_to is recieved
* responsiveness
* move code to enterprise folder
* imple error states, better loading state
* handle user leaving the page while a response is being loaded
* lint fix, change inline sql shortcut key, disable prompt suggestions and fade out once a prompt has been sent
* fix e2e test
* add test coverage, update copy
* fix failing e2e tests
* attempt to fix e2e flake
Co-authored-by: Sloan Sparger <sloansparger@users.noreply.github.com>
Log each entity (card) as its dependencies get backfilled by the job. This is to be able to see if there is a particular entity that's causing the OOM.
Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>
* Support arbitrary expressions inside `:is-null` and `:not-null`
* Remove unnecessary blank line in schema_test.cljc
Co-authored-by: Cam Saul <1455846+camsaul@users.noreply.github.com>
* WIP
* Working test! (it reproduces)
* Fhew! I found it!
* Remove printlns
* Also remove sensitive fks in calculation
* Remove stray marker comment
* Move tests to fields-test
* Exclude :retired too
* Move tests and add one for retired
* Add another retired
* Better tests
* Appease the kondo
* Fix broken test
Co-authored-by: Eric Normand <ericwnormand@gmail.com>
* Fix normalization of MBQL 4 `:aggregation-options` with a `nil` options map
* Update tests
* Fix normalization of Template Tag :options
* Allow raw values as expression definitions
* Automatically remove :ident from joins
* Properly handle aggregations expressions that contain `:if` (alias for `:case`)
* Allow `:coalesce`, `:if`, and `:case` as datetime exprs in MBQL 4
* Test & Kondo fixes 🔧
Co-authored-by: Cam Saul <1455846+camsaul@users.noreply.github.com>
### Description
We want to communicate per-billing-period quantities with customer, instead of always communicate monthly quantities. Invoiced totals will stay the same.
### How to verify
Describe the steps to verify that the changes are working as expected.
1. Ensure you have a token with the `offer-metabase-ai-tiered` feature, but without the `metabot-v3` feature.
2. Navigate to `/admin/metabot` and see the quantities as per billing period, and not per month.
### Checklist
- [x] Tests have been added/updated to cover changes in this PR
Fixes: 2b76c38e1f
Closes: https://metaboat.slack.com/archives/C07RMUQ0GE5/p1765907134400039
Co-authored-by: Dennis Schridde <63082+devurandom@users.noreply.github.com>
* UXW-2451 Fix opposite light/dark mode colors leaking when table is present (#67017)
* WIP UXW-2451 Fixes issue where changing light/dark mode messes up color variables by removing nested providers in useBodyCellMeasure which probably breaks other things
* Update Loki Snapshots
---------
Co-authored-by: Metabase Automation <github-automation@metabase.com>
* Update Loki Snapshots
---------
Co-authored-by: Brad Anderson <brad@metabase.com>
This generally indicates an MBQL 4 -> 5 conversion issue, and we need
the signal in the 57 rollout.
Also improves the detail of the logging for when `:dataset_query {}`.
Co-authored-by: Braden Shepherdson <braden@metabase.com>
* Add Australia (Sydney) to available regions
Added Australia (Sydney) to the list of regions.
* Add Asia Pacific (Sydney) IP addresses to whitelist
Added IP addresses for the Asia Pacific (Sydney) region.
* Reorder South America IP addresses section
* Update cloud region list in change-region.md
Removed duplicate entry for Australia (Sydney) and updated Asia Pacific region.
Co-authored-by: Alex Yarosh <alexandra@metabase.com>
This reverts commit 9191713854.
used to do this in-house but now we use trunk for this. Old way has
ultimate flexibility. But trunk has some really nice workflows built
in (reporting, creating issues in linear, etc).
Co-authored-by: dpsutton <dan@dpsutton.com>
The summary: we are offering drill-throughs on custom columns defined after aggregation, but those custom columns don't exist in the base query. The error presented in such a way that `lib.underlying/top-level-column` for a dimension was returning `nil`, and we were happily putting that `nil` into the query we constructed.
I created a utility to filter dimensions by whether or not the top-level-column can be resolved, and used this for the underlying-records drill-through (from the original issue) as well as the pivot drill-through (which behaved similarly). If no dimensions have a top-level column, the drill-through is not available. If some do, we offer the partial drill-through.
Co-authored-by: Timothy S. Dean <timothy.dean@metabase.com>
`source` is a legacy key on the frontend that gets translated to `lib/source`. Some maps are using both `source` and `lib/source`, and the translation layer translates keys and values in completely separate steps. When the time comes to translate the legacy value of `source` into a valid value for `lib/source`, the code can't tell what the key was originally named. So already-valid values were going through the translation process, producing invalid values (e.g., `:source/source/aggregations`).
This adds a step to object parsing to detect that both keys are present; when they are, it sets an option to simply skip `source` during the parsing; secondly, it adds a step in the value parsing to check whether the value is already a keyword namespaced by `source/`.
Co-authored-by: Timothy S. Dean <timothy.dean@metabase.com>
`util.perf/mapv` checks the `count` of all its collection args, so it's
not safe to call with an infinite `seq`.
`lib.metadata.result-metadata/merge-cols` was doing exactly that in the
case where there was a mismatch between the column counts from lib and
driver. It was passing `(concat short-list (repeat nil))` to `mapv` and
causing an infinite loop!
Co-authored-by: Braden Shepherdson <braden@metabase.com>
OOMs in cloud are linked to this weak map retaining too much. The
exported heap dump reports are suggestive but not conclusive, and I
can't reproduce locally.
Adds `:metabase-query-processor/computed-weak-map-queries` to measure
the size of the weak map. That should sawtooth with GCs as it
accumulates queries and then discards them, but we'll see if it rises
forever on the problem instances.
This also adjusts the caching for a `MetadataProvider`-level entity
(`:metadata/card`) from the weak map to the `MetadataProvider`'s
internal cache. That's speculative, since there's no evidence to link
it to the issue, but it should be at least as effective a cache and
there's less risk of accidentally retaining too much.
Co-authored-by: Braden Shepherdson <braden@metabase.com>
docs scoop up all settings and has ways to avoid. In
`metabase.cmd.env-var-dox/avoid?` This had a docstring and then an extra
docstring. We can just copy the nice docstring to the docstring slot and
put `:doc false`.
```clojure
(defn- avoid?
"Used to filter out environment variables with high foot-gun indices."
[env-var]
(or (false? (:doc env-var))
(false? (:can-read-from-env? env-var))
;; Ideally, we'd move off of this list completely,
;; but not all environment variables are defsettings.
(contains? env-vars-not-to-mess-with (format-prefix env-var))))
```
Co-authored-by: dpsutton <dan@dpsutton.com>
* fix broken text wrapping in tables in documents
* render measure root in a portal
Co-authored-by: Aleksandr Lesnenko <alxnddr@users.noreply.github.com>
* Prevent looping past current-schema-version when already past it.
* Add test for upgrade-card-schema-to-latest
* Make test parallel
Co-authored-by: Eric Normand <ericwnormand@gmail.com>
* Treat postgres bit strings as text (#66618)
* treat bit strings as text
* Add tests
* cast bit string values instead of fields
* only use string value if bit type
* fix kondo
* Add comment
* kondo
---------
Co-authored-by: Riley Thompson <riley@metabase.com>
Co-authored-by: Riley Thompson <rileythompson99@gmail.com>
motivating example:
we ran the redshift tests, and had three failures: redshift 1, redshift
2, and redshift transforms.
redshifts 1 and 2 timedout at 60 minutes. But the log upload failed
because the python transform tests failed in 24 minutes and then
uploaded:
from the transform failure run:
```
Run actions/upload-artifact@v4
with:
name: test-logs-be-tests-redshift-ee
path: logs/test-log.json
retention-days: 1
if-no-files-found: warn
compression-level: 6
overwrite: false
include-hidden-files: false
env:
CI: true
...
With the provided path, there will be 1 file uploaded
Artifact name is valid!
Root directory input is valid!
Beginning upload of artifact content to blob storage
Uploaded bytes 1072768
Finished uploading artifact content to blob storage!
SHA256 digest of uploaded artifact zip is 267d4b085d49ca4d682fedca97832fecb603bbb5a52c0e1d99bd4daa49984ba3
Finalizing artifact upload
Artifact test-logs-be-tests-redshift-ee.zip successfully finalized. Artifact ID 4816135520
Artifact test-logs-be-tests-redshift-ee has been successfully uploaded! Final size is 1072768 bytes. Artifact ID is 4816135520
```
And then from the redshift tests:
```
Run actions/upload-artifact@v4
with:
name: test-logs-be-tests-redshift-ee
path: logs/test-log.json
retention-days: 1
if-no-files-found: warn
compression-level: 6
overwrite: false
include-hidden-files: false
env:
CI: true
...
With the provided path, there will be 1 file uploaded
Artifact name is valid!
Root directory input is valid!
Error: Failed to CreateArtifact: Received non-retryable error: Failed request: (409) Conflict: an artifact with this name already exists on the workflow run
```
The same happens on redshift part 1.
```
Error: Failed to CreateArtifact: Received non-retryable error: Failed
request: (409) Conflict: an artifact with this name already exists on
the workflow run
```
This just introduces on the driver action to look for a
logs-artifact-suffix and use it if present. And on driver jobs that use
a matrix include the matrix.job.name (or matrix.version.name depending
on the exact structure).
Co-authored-by: dpsutton <dan@dpsutton.com>
* Make card schema upgrade 23 a no-op (#66719)
* Make card schema upgrade 23 a no-op
* Fix kondo error
* Yay, fix kondo error in other file
* Appease the module check
* Use clojure.core/get-in
* kick ci
* kick ci
---------
Co-authored-by: Eric Normand <ericwnormand@gmail.com>
* #66585 Updates static embeds to omit color-scheme settings to preserve legacy transparent iframe behavior
* support public embed too (#66693)
* support public embed too
* Imports isPublicEmbedding to prevent app crash
---------
---------
Co-authored-by: Brad Anderson <brad@metabase.com>
Co-authored-by: Nicolò Pretto <info@npretto.com>
If you have a query like "WITH query_a AS (...), query_b AS (select ... from query_a) SELECT ... from query_b",
native query parsing understood that query_b in the final query referred to query_b in the WITH, but it didn't realize
that query_a in query_b referred to query_a in the WITH. Now, it should understand these sorts of depedent ctes.
Co-authored-by: William <william@metabase.com>
this should get all audit log entries for a given model and
model-id. but previously it was using t2/select-one. now it uses
t2/select. whoops
Co-authored-by: dpsutton <dan@dpsutton.com>
* If dataset_query is empty and legacy_query is not, copy legacy query over
* Fix kondo errors and bump schema to 23
* Remove parens
* Baseline update
Co-authored-by: Eric Normand <ericwnormand@gmail.com>
* add events when making cards/dash pubic or retracting
https://github.com/metabase/metabase/issues/64417
When we make a card public, we put a uuid on it and include
`made_public_by_id`. But when you retract the public link, we remove
both bits of information. This is a nice invariant but it removes
information about who publicized a card.
Imagine a card is discovered by have been public. Of course you want to
immediately remove it. But this erases the audit trail (so to speak) of
who made it public. It is not a great idea to delay remediation during
the investigation. Now you can just consult the audit log
* make events
* formatting
* clean up tests a bit
tests were a bit wack. We should have utility functions to get audit
events. Also noticed we are going off of the object id and ignoring the
object. Could lead to gross issues if there's dashboard with id 4 in
there and also card with id 4. so require model on new one
* fmt
Co-authored-by: dpsutton <dan@dpsutton.com>
Fixes#66464.
This was a corner case in `lib.field.resolution` and `qp.util.add-alias-info`
that regressed in 57. It made the QP less smart about repairing a certain
class of broken queries, which regrettably exist in the wild.
The breakage in question was a `:field` ref in later stage, which had a
numeric field ID and a `:join-alias`. If that field ID was not actually
returned as part of the join anymore, fallback metadata would be
generated by looking up the field by ID. That "works" for non-joined
fields but for an explicitly joined field it would use
`source."SOME_FIELD"` rather than `"The Join Alias"."SOME_FIELD"`, which
causes SQL compilation to fail.
Co-authored-by: Braden Shepherdson <braden@metabase.com>
* Closes EMB-1042 removed flaky e2e test and adds a unit test instead
* yeah no timeout
* fixed poor rebase
* better use of fetchmock in unit test
Co-authored-by: Sébastien <sebastien@metabase.com>
* When setting settings, include 5 minute cookie of updated timestamp
if other instances have outdated settings cache, this cookie will let
them know what the requestor has seen. The instance can then know it is
out of date. This won't help for instances that aren't hit by the person
setting the setting.
Motivating example is someone who is doing an embedding onboarding
checklist. And then hits a new instance that hasn't yet loaded the "show
embedding homepage" setting so it flips back to the regular
homepage. The hope here is the person doing this checklist will let the
other instance know it is out of date, reload the settings cache and
then see the embedding setting.
* Interesting strategy here for cookies.
I hated that the route would would set a cookie at a particular key, do
timestamp stuff, etc. I know ring is a map and abstraction, but i added
a new paradigm:
```clojure
(defn- add-settings-last-updated-cookie
"Add a cookie with the current settings-last-updated timestamp to the response."
[response]
(assoc-in response [:mb/cookies :cookie/settings-cache-timestamp] true))
```
This means that just the middleware is responsible for checking and
setting this key. This was largely driven by two different namespaces
having a private def of:
```clojure
(def ^:private settings-last-updated-cookie-name "metabase.SETTINGS_LAST_UPDATED")
```
And it's nice if the ring map can indicate to this middleware to handle
all the key and timestamp stuff and it just has to indicate to set this
with `cookie/settings-cache-timestamp`.
So the request looks like this:
```clojure
{:status 204
:headers {"X-Frame-Options" "DENY"
"X-XSS-Protection" "1; mode=block"
"Last-Modified" "Mon, 1 Dec 2025 17:09:59 GMT"
"Strict-Transport-Security" "max-age=31536000"
"X-Permitted-Cross-Domain-Policies" "none"
"Cache-Control" "max-age=0, no-cache, must-revalidate, proxy-revalidate"
"X-Content-Type-Options" "nosniff"
"Content-Security-Policy" "...<lots of junk>..."
"Expires" "Tue, 03 Jul 2001 06:00:00 GMT"}
:body nil
:mb/cookies {:cookie/settings-cache-timestamp true} ;; <---- our new addition
:cookies {"metabase.DEVICE" {:value "60ac9454-9dd1-439a-afa1-921ff029686d"
:http-only true
:path "/"
:expires "Fri, 1 Dec 2045 17:09:59 GMT"
:same-site :lax}}}
```
And then in practice, it looks like this:
Set the setting and get a cookie with the timestamp in response:
```
❯ http put :3000/api/setting/site-name value=foo x-api-key:$KEY
HTTP/1.1 204 No Content
...
Set-Cookie: metabase.SETTINGS_LAST_UPDATED=2025-12-01+17%3A28%3A15.450873%2B00; Path=/; Max-Age=300; SameSite=Lax
...
```
Simulate sending to another instance with a timestamp that exceeds the
currently know timestamp (i'll increment the hour based sent back in
this response)
```
❯ http get :3000/api/user/current x-api-key:$KEY Cookie:metabase.SETTINGS_LAST_UPDATED="2025-12-01+18%3A23%3A51.059155%2B00" | jq '.date_joined'
"2025-12-01T16:51:08.137822Z"
```
(note the 18 vs the 17)
And the logs are this:
```
INFO middleware.settings-cache :: Settings cookie indicates cache is out of date. Refreshing...
DEBUG middleware.log :: GET /api/user/current 200 19ms (11 DB calls) App DB connections: 0/4 Jetty threads: 5/50 (5 idle, 0 queued) (245 total active threads) Queries in flight: 0 (0 queued) {:metabase-user-id 346}
```
* fixes:
- logged known timestamp not weird cookie value
- timestamp is optional, so handle that
- typo in log konwn -> known
* Add tests, add functionality
tests:
ensure it (a) sets the cookie, (b) honors the cookie, and when it
restores the cache, to set the cookie so it doesn't keep thrashing. But
presumably it won't since it's up to date. My "in the future" version is
a bit synthetic.
Functionality wise, updating the cache now returns and this will saet
the header with the most recent cache setting
Co-authored-by: dpsutton <dan@dpsutton.com>
* Closes EMB-1055 static embedding without a theme defaults to dark mode or system settings
* types
* .
Co-authored-by: Sébastien <sebastien@metabase.com>
* fix: avoid breaking spaces on upsell CTA link
* fix: exchange `h` for `mih` on stepper card components and allow for flexible height
Co-authored-by: Gabriel Moreno <gabriel@metabase.com>
* Add Metabase Sample Database to editable tables support
Updated the list of databases that support editable tables to include Metabase Sample Database.
* Update docs/data-modeling/editable-tables.md
* Update docs/data-modeling/editable-tables.md
Co-authored-by: Alex Yarosh <alexandra@metabase.com>
Fixes#66199.
`metabase.legacy-mbql.schema`, now used for normalizing MBQL 4 in
place of the old bespoke normalization in earlier versions, regressed in
this weird but legitimate use case: Writing a custom aggregation like
`MAX("String Literal")`.
That's an odd thing to do, but it can be useful to provide a label in an
aggregated query. This is the MBQL equivalent of
`(SELECT ..., 'A' as label FROM ...) UNION ALL (SELECT ..., 'B' as label ...)`
Co-authored-by: Braden Shepherdson <braden@metabase.com>
* updating watermark svg color and opacity
* static viz development mark has same color and opacity as dynamic viz
Co-authored-by: Nick Fitzpatrick <nick@metabase.com>
This pull request refactors and streamlines the configuration for CI workflows and file path groupings in .github/file-paths.yaml and updates related workflow logic in .github/workflows/run-tests.yml. The changes focus on consolidating shared configuration, improving maintainability, and removing redundant or unused path groups.
Co-authored-by: Nemanja Glumac <31325167+nemanjaglumac@users.noreply.github.com>
* Add :database_type to metabot tools that return column metadata
Add :database_type to metabot tools that return column metadata in order to give metabot information about the real
database type when generating SQL.
Examples of places the :database_type now appears:
- :used_tables in :user_is_viewing context
- {table,card,metric}-details
- get-query-details
- enhanced-database-tables
* Fix test after rebase onto #66271
Fix get-table-details-database-type-test after rebase (1) use new field-id formatting (2) get-table-details no longer returns implicitly-joinable fields
in the top-level `:fields`.
Co-authored-by: appleby <86076+appleby@users.noreply.github.com>
* More flexible approach to resolving columns from metabot field_ids
* fix one field-stats-test
* fix field-stats-test
* support nano-ids in field ids (?)
* remove some dead code
* Exclude implicitly joinable fields when fetching cols for table
* try switching field format to use dash
* fix field IDs in a number of tests
* Kondo
* add some unit tests for new field parsing & column lookup helper fns
* Fix more tests
* Fix more tests
* fix regex
* New approach -- use wide field IDs in related_tables and add a related_by field for each table
* Fix some tests
* remove unneeded table
* cleanup, new test
* Revert to passing in & matching on expected prefix
* Address tyler's comments
* accept a regex for the prefix as well
* add back small malli group-by fix from the last PR - removed unintentionally during revision
Co-authored-by: Noah Moss <32746338+noahmoss@users.noreply.github.com>
* fix: add missing channels to copy & rendering logic
Mentioned webhooks and Slack where necessary to stay consistent with the
admin UI
* fix: standardize lowercase 'admin' & prefer 'email' over 'SMTP'
* fix: remove extra comma on wording
* test: adjust e2e test assertions
* test: add `DashboardSubscriptionMenuItem.unit.spec.tsx`
* test: extract single case from `describe` block
* test: fix subscriptions e2e case & text lookup on alert spec
Co-authored-by: Gabriel Moreno <gabriel@metabase.com>
Since `position` was set to `top`, the popover had little space to
expand upwards, and the box was cut-off before reaching the anchor
point.
Co-authored-by: Gabriel Moreno <gantoreno@gmail.com>
Co-authored-by: Gabriel Moreno <gabriel@metabase.com>
* GDGT-1252: Make list view consider mini-bar-chart setting
* GDGT-1252: Fix incorrect model settings update when changing columns settings
* GDGT-1252: e2e test
* GDGT-1252: Fix unit test for ListViewItem
* GDGT-1252: Address PR comments. Minor refactoring
* GDGT-1252: Refactor DatasetEditor to get rid of modelSettings hack
* GDGT-1252: Update e2e test to cover settings persistence
* Update Loki Snapshots
* Revert Loki update
---------
Co-authored-by: Stas Gavrylov <stas@metabase.com>
These tools already included :database_id in the results; we now include the :database_engine as well, to give metabot
a reminder about which SQL dialect it should generate.
Co-authored-by: appleby <86076+appleby@users.noreply.github.com>
Fixes#65714.
These are sent as `:number/<=` or `:number/>=` and the validation was
too strict from 57 onward. Added a test for none, min, max, and both
ends of a Between filter.
Co-authored-by: Braden Shepherdson <braden@metabase.com>
Co-authored-by: dpsutton <dan@dpsutton.com>
### Description
Missing billing information is treated as an error for purposes of
displaying the purchase page with prices. Non-store users cannot
retrieve billing information. Thus, previously, they would always see
an error page: "Error fetching information about available add-ons."
With this change, they see the "Please ask a Metabase Store Admin" page
as intended.
We achieve this by splitting the page content for store users from that
of non-store users. Only the former needs to load data (billing and
add-on information), so only that page needs `LoadingAndErrorWrapper`.
For store users we show no dynamic content that could fail to load
(apart from a store user's email address, which we gracefully omit if
it is missing).
### How to verify
1. Ensure you have a token with the `offer-metabase-ai-tiered`
feature, but without the `metabot-v3` and `offer-metabase-ai`
features.
2. As a Metabase user with an email address that *does not* match any
of the token's organization's store user email addresses, navigate
to `/admin/metabot` and see "Please ask a Metabase Store Admin".
3. As a Metabase user with an email address that *does* match any
of the token's organization's store user email addresses, navigate
to `/admin/metabot` and see the purchase page with tier selector.
### Checklist
- [x] Tests have been added/updated to cover changes in this PR
Fixes: 2b76c38e1f
Closes: https://metaboat.slack.com/archives/C07RMUQ0GE5/p1763644357893079
Co-authored-by: Dennis Schridde <63082+devurandom@users.noreply.github.com>
* Move audit installation after search init; change some logs to debug
* change another warn to debug
Co-authored-by: Noah Moss <32746338+noahmoss@users.noreply.github.com>
* minimal tests to be run when the label is `minimal-tests`
* include enterprise tests, fix syntax
* bump CI
* don't fail if label is absent
Co-authored-by: dpsutton <dan@dpsutton.com>
* chore(sdk,ci): fix stress tests for sdk tests
* fix baseUrl, cypress explodes if we pass it in ct test config
Co-authored-by: Nicolò Pretto <info@npretto.com>
* fix cut not removing text from input
* ensure we fetch all data for all possible models for smart links
* remove unused comment
* fixes smart links in metabot input sometimes degrading into their text representation and fixes some cases where wrong model name ("model" instead of "dataset") was used for metabot links
* fix unit tests
* ui for cancelling a request, handles state updates for canceled requests
* press esc to cancel metabot request, fix state being incorrect on requests after an aborted request
* add test coverage for ui, update test utils to enable aborting requests in unit tests, small bits of clean up
* add tests to processStream function
* update snapshot
Co-authored-by: Sloan Sparger <sloansparger@users.noreply.github.com>
* fix: allow analytics collection perms to be set
In #65098, we did not fully replicate the behavior of the old graph
endpoint where collection namespacing rules aren't applied to the
"analytics" namespace. Collections in this namespace can be modified as
though they are part of the root `nil` namespace. The bug was in
removing collections not part of the targetted namespace of the
`update-graph!` funciton from the graph to update.
This restores the old behavior by not excluding collection permissions
updates for collections in the "analytics" namespace in the
`remove-collections-from-other-namespaces` function.
it also updates our tests for the collection graph to make sure mock
analytics collections have the right namespace set rather than a `nil` namespace.
* fixup! reset revisions
* fixup! tests to EE module fix tests in EE module
Co-authored-by: Edward Paget <edpaget@users.noreply.github.com>
Running Cypress tests doesn't have to be hard.
This PR is moving us in a direction where (1) there's some logic that sets up the environment for e2e testing, and then (2) there's a simple command to invoke cypress (aka "decoupling" or "separation of concerns")
Cypress only needs to know which config to look at. It contains all the information it needs to run.
There are two ways to invoke Cypress:
1. Cypress cli
2. Cypress Module API (programmatic entry point)
They should be compatible and interchangeable. Anything you pass to the node module as options can be passed as individual flags to the cli.
```
npx cypress run --config-file path/to/some/config --spec path/to/spec.cy.spec.ts
```
is equivalent to
```
cypress.run({ configFile: "path/to/some/config", spec: "path/to/spec.cy.spec.ts" });
```
This PR removes all abstractions from the actual node module runner.
Co-authored-by: Nemanja Glumac <31325167+nemanjaglumac@users.noreply.github.com>
* Metabot tool endpoitns should support MBQL5
* Update filter tool to work with MBQL 5
* Fix find-outliers-wrong-query-test
Was attempting to assoc-in/dissoc-in MBQL4 query keys on an MBQL5 query.
---------
Co-authored-by: Thomas Schmidt <thomas@metabase.com>
Co-authored-by: Mike Appleby <86076+appleby@users.noreply.github.com>
* Turn off db routing for clickhouse and it's trick.
DB routing needs a "degree of freedom" to run the exact same query
against a different schema/db/instance whatever. Clickhouse only has
database.table. so to db route on the same clickhouse instance you have
to omit the database and let it resolve during query time.
`select * from db.table` can't work. Have to do `select * from table`
and let the connection dictate db1 or db2.
In 56, tables look like this:
```clojure
{:name "upload_records_10k_20251118164620", :schema "db_eafc4346850f435e"}
```
In 57, now they look like this for single db clickhouse things:
```clojure
{:name "upload_records_10k_20251118164620", :schema nil}
```
There are two critical problems here:
1. When uploading, it's not aware to omit the schema. So a table is
created with that schema. During the next sync that table is archived
and a new one with a nil'd schema is created. So the model created for
the uploaded CSV would break as it points to an inactive table. This has
a further compounding issue because now there "appear" to be two schema:
the faked nil one and also a new one like `db_eafc...`.
2. But _all_ tables will be archived in the same manner which were
initially found in 56 or below. These tables will be marked as inactive
and new ones created. Meaning all mbql queries based on clickhouse will
fail.
I wanted to revert this change but there's actually a bit more in the PR
and some conflicts. It also does more than just enabling db-routing so I
think we can perhaps undo the damage in this way more easily.
* remove outdated test
Co-authored-by: dpsutton <dan@dpsutton.com>
* move sdk bundle code to the oss folder (#65753)
* move the actual sdk folders
* update rspack config and eslint config
* fix relative import in CommonSdkStoryWrapper
* allow default exports in preview.tsx
* update eslint config to allow imports from frontend/src/embedding-sdk-shared
* remove legacy eslint disable directive
* temporarily 'disable' eslint-disable-next-line no-external-references-for-sdk-package-code that has not been ported yet
* update rspack config to use new paths
* update jest config to use new paths
* add `embedding-sdk-package` to restricted imports
* copy eslint files to apply to the new sdk folders
* add eslint-disable-next-line no-restricted-imports to some places where we need emotion
* fix jest setup
* add and use setupEmbeddingSdkEnterprisePlugins
* use `useSetting` instead of `getApplicationColors` to remove enterprise import
* move defineMetabaseAuthConfig to embedding-sdk-shared
* move MetabotProvider to embedding-sdk-bundle
* fix docs generation issue about types
* fix prettier ignore file
* move back tests to ee folder
* remove ../../../ from imports
* fix eslint issues
* fix, again, jest config :sad:
* chore: empty commit to make ci restart
* chore: empty commit to make ci restart
---------
Co-authored-by: Nicolò Pretto <info@npretto.com>
Fixes#64135.
Previously this code would throw if multiple dashboard and/or card
params were bound to the same SQL variable. Now multiple links are
allowed, provided that only one is actually given a value.
Actually, this logic de-duplicates the values, so if multiple filters
all set the same value, that also works.
Co-authored-by: Braden Shepherdson <braden@metabase.com>
* implement plugins for auth
* move some stuff to the oss folder as those file create issues later on
* make embedding-sdk:docs:generate pass
* move metabot to use plugins
* remove imports from package from other folders
Co-authored-by: Nicolò Pretto <info@npretto.com>
Removed multiple tests related to info popovers for field picker options and saved questions.
This behavior is already covered with a unit test at:
`frontend/src/metabase/common/components/MetadataInfo/ColumnInfoIcon/ColumnInfoIcon.unit.spec.tsx`
Co-authored-by: Nemanja Glumac <31325167+nemanjaglumac@users.noreply.github.com>
* Fix the update keys for postgres search idx upsert
The upsert keys were calculated by looking at the first entry we were
inserting and removing `:id`, `:model`, and `:model_id`.
But the entries are not necessarily homogeneous. In this case, the first
entry is a dashboard (has no `:dashboard_id`) and the second entry is a
card (yes `:dashboard_id`).
This meant we never actually updated the dashboard_id when updating a card.
I attempted to add a test for this:
```
(deftest updates-work-properly
(mt/with-test-user :rasta
(mt/with-temp [:model/Dashboard {dash-id :id} {}]
(let [card-name (str (random-uuid))
card-id-1 (:id (mt/user-http-request :rasta :post 200 "card/" {:name card-name
:collection_id nil
:database_id 1
:display "table"
:creator_id (mt/user->id :rasta)
:visualization_settings {}
:dataset_query {:type "native" :native {:query "select 123"}
:database 1}}))
card-id-2 (:id (mt/user-http-request :rasta :post 200 "card/" {:name card-name
:collection_id nil
:database_id 1
:display "table"
:creator_id (mt/user->id :rasta)
:visualization_settings {}
:dataset_query {:type "native" :native {:query "select 123"}
:database 1}}))]
(mt/user-http-request :rasta :put 200 (str "card/" card-id-1) {:dashboard_id dash-id})
(is (= [card-name]
(map :name (:data (mt/user-http-request :rasta :get 200 "search" :q card-name)))))))))
```
The good news is that this test passes after my change. The bad news is
that it also passes without my change. I think what's going on is that
the behavior is different with `*force-sync*` on/off: in async mode, we
get the full batch that contains the updates for `[dashboard, card]`.
In synchronous mode, we get separate batches for `[dashboard]` and then
`[card]`.
I think we might want to rework tests somehow - it would be quite nice
to be able to still run things asynchronously but drain the queue before
making assertions, so the behavior wouldn't be different between
tests/prod.
* add a test
Co-authored-by: John Swanson <john.swanson@metabase.com>
* replace password with key pair auth
* re enable snowflake tests
* add private key env var
* update private key details for temp dbs
* small changes
* fix describe database test
* fix multi method params
* update test
* update test
* try fix
* check CI
i noticed that this tests pass if we don't put a connection-uri in
there. I'm not sure what the original intent was, but the following
works just fine:
```
{:url "jdbc❄️//<account>.snowflakecomputing.com/?schema=INFORMATION_SCHEMA",
:properties {"use-password" "false",
"connection-uri" "jdbc❄️//<account>.snowflakecomputing.com/?user=<user>CI&private_key_file=%2Fprivate%2Fvar%2Ffolders%2F1d%2F3ns5s1gs7xjgb09bh1yb6wpc0000gn%2FT%2Fmetabase-secret_8808579641027884666.tmp&schema=INFORMATION_SCHEMA",
"client_session_keep_alive" "true",
"timezone" "UTC",
"warehouse" "<warehouse>",
"user" "<user>",
"ssl" "true",
"private_key_file" "/var/folders/1d/3ns5s1gs7xjgb09bh1yb6wpc0000gn/T/metabase-secret_8808579641027884666.tmp",
"week_start" "7",
"application" "Metabase_Metabase",
"account" "<account>",
"db" "\"<db>\"",
"time_output_format" "HH24:MI:SS.FF3",
"client_metadata_request_use_connection_ctx" "true"}}
```
When it goes through clojure.java.jdbc. Somehow we are getting a
connection string that doesn't play nice. I _think_ it may just be an
issue with snowflake but i'm not positive.
* remove parallel test markers
* remove parallel marker
when running tests locally with these removed:
```clojure
(run/find-and-run-tests-repl {:only 'metabase.driver.snowflake-test})
...
Ran 49 tests in 116.525 seconds
502 assertions, 1 failure, 0 errors.
{:test 49, :pass 501, :fail 1, :error 0, :type :summary, :duration 116525.006833, :single-threaded 32, :parallel 17}
```
yields only one expected error. Previously using regular stuff also
worked
```clojure
snowflake-test=> (run-tests)
Testing metabase.driver.snowflake-test
Ran 49 tests containing 502 assertions.
1 failures, 0 errors.
{:test 49, :pass 501, :fail 1, :error 0, :type :summary}
```
(again the single test that failed). but using the CLI tool i would get
errors when running them in parallel.
* fixup stupid snowflake
1. i don't know why we are running into multithreading issues now. But
i've removed those tags and the issues are gone.
2.
```clojure
snowflake-test=> (let [spec {:connection-uri "jdbc❄️//account.snowflakecomputing.com/?schema=INFORMATION_SCHEMA&user=<user>&password=<password>&db=%22v5_airports%22"}]
[(jdbc/query spec ["select CURRENT_DATABASE() db"])
(jdbc/query spec ["select CURRENT_SCHEMA() db"])])
[({:db "v5_airports"}) ({:db "INFORMATION_SCHEMA"})]
snowflake-test=> (let [spec {:connection-uri "jdbc❄️//account.snowflakecomputing.com/?schema=INFORMATION_SCHEMA&user=<user>&password=<password>&db=v5_airports"}]
[(jdbc/query spec ["select CURRENT_DATABASE() db"])
(jdbc/query spec ["select CURRENT_SCHEMA() db"])])
[({:db nil}) ({:db nil})]
```
In the above, note that putting the db in the connection string _and
url-encoding it_ (it's already quoted) is necessary.
* conditionally add db
in tests there's not a database so just be smart about it:
```clojure
{:warehouse "<warehouse>",
:additional-options nil,
:timezone "UTC",
:db nil,
:account "<account>",
:use-password false,
:time_output_format "HH24:MI:SS.FF3",
:user "<user>",
:subname "//<account>.snowflakecomputing.com/"}
```
* address pr comments
* remove outdated comment
pretty freaky comment eh? I added this when i commented out the
`handle-conn-uri` call underneath it and it caused tests to pass. That
was helpful in identifying the mechanism, but not a good solution. Put a
comment here so that that was not a final state, forgot to remove
comment when we got to a proper final state and `handle-conn-uri` was
fixed.
---------
Co-authored-by: Riley Thompson <riley@metabase.com>
Co-authored-by: dan sutton <dan@dpsutton.com>
Fixes#65726.
If you implicitly joined a field that happened to be the target of a
remapping, the QP was labeling it as `:remapped_from` the FK even
though (a) it wasn't really remapped and (b) there was no
corresponding FK column marked as `:remapped_to` the target column.
(Eg. `Orders.PRODUCT_ID` is remapped to `Products.TITLE`. Then I
implicitly join `Products.TITLE` directly and the QP marks it as
remapped from `Orders.PRODUCT_ID` even though it wasn't.)
That caused the FE to do odd things like drop and relabel columns.
This change adjusts the QP so it will ignore plausible remappings that
don't actually get returned from the last stage.
Co-authored-by: Braden Shepherdson <braden@metabase.com>
We get a bunch of this useless stuff in CI:
```
Placeholder until other changesets
UPDATE SUMMARY
Run: 641
Previously run: 0
Filtered out: 69
-------------------------------
Total change sets: 710
FILTERED CHANGE SETS SUMMARY
DBMS mismatch: 69
```
Looking for this in liquibase this is not logged but printed to the
console :(.
It goes through this:
```java
private static void writeMessage(String message, UpdateSummaryOutputEnum showSummaryOutput, OutputStream outputStream) throws LiquibaseException {
switch (showSummaryOutput) {
case CONSOLE:
writeToOutput(outputStream, message);
break;
case LOG:
writeToLog(message);
break;
default:
writeToOutput(outputStream, message);
writeToLog(message);
}
}
```
So we want that enum to be LOG.
```clojure
liquibase=> (clojure.reflect/reflect liquibase.Liquibase)
{:bases #{java.lang.Object java.lang.AutoCloseable},
:flags #{:public},
:members #{ ,,,
#clojure.reflect.Method{:name setShowSummaryOutput,
:return-type void,
:declaring-class liquibase.Liquibase,
:parameter-types [liquibase.UpdateSummaryOutputEnum],
:exception-types [],
:flags #{:public}}
,,,}}
```
So easy fix. Set this enum and hopefully it shuts up.
Co-authored-by: dpsutton <dan@dpsutton.com>
* claude fix
* simplify
* handle nested fields for athena
* nfc have position 0
* set non leafs as details only
* clean up parser and add tests
* whitespace
* disable nested fields
* kondo
* kondo
* disable nested schema test
Co-authored-by: Riley Thompson <riley@metabase.com>
* Revert "Adjust cards-with-filters to work with action cards (#65805)"
This reverts commit d6b857a2c6.
* Revert "Fix mapping of params to later query stages (#65651)"
This reverts commit c98e640426.
* Revert "Compute param values from card results (#64318)"
This reverts commit 2693173e7f.
Co-authored-by: lbrdnk <lbrdnk@users.noreply.github.com>
* Fix SDK save dashboard question to not show collection/dashboard picker
* Fix EditableDashboard sometimes doesn't show the correct tab on first render
* Add an assertion
* left some breadcrumbs
* Update unit tests to be async
Co-authored-by: Mahatthana (Kelvin) Nomsawadi <me@bboykelvin.dev>
* Fix code that populates :used_tables in metabot context
This was causing :used_tables to be absent in the :user_is_viewing
context, limiting metabot's ability to get sql fixing right without
information about the available tables and their fields.
Call lib-be/normalize-query and lib/native-only-query? in
metabase-enterprise.metabot-v3.context/query-for-sql-parsing and
likewise use lib funcs to get template-tags in
metabase-enterprise.metabot-v3.query-analyzer.parameter-substitution/replace-tags
* Update ai-sql-fixer and tests to work with MBQL5 queries
* Fix metabot-v3.table-utils-test to use MBQL5 queries
Conflicts:
enterprise/backend/test/metabase_enterprise/metabot_v3/context_test.clj
Co-authored-by: appleby <86076+appleby@users.noreply.github.com>
In #62513 we added a page to offer a Metabase AI (Metabot) trial. Now
we want to offer to purchase the paid Metabot add-on.
### How to verify
1. Ensure you have a token with the `offer-metabase-ai-tiered`
feature, but without the `metabot-v3` and `offer-metabase-ai`
features.
2. Ensure you have a Metabase API key linked to that same token.
3. Navigate to `/admin/metabot` and see a page offering you to purchase
different quantities of Metabot queries.
4. Select a quantity (or go with the default of 500), check "I agree",
then click "confirm purchase".
5. Confirm that your token now has the `metabot-v3` feature and the
associated subscription has an appropriate Metabase AI product.
### Checklist
- [x] Tests have been added/updated to cover changes in this PR
### Notes
* Rename MetabotPurchasePage to MetabotTrialPage
- and add the page for the purchase of the paid add-on as
`MetabotPurchasePage`, to make the name more clearly represent the
purpose (paid vs. trial).
* Move frontend client for /api/ee/cloud-add-ons into metabase-enterprise/api
* Use ts-pattern instead of hand-crafted type guards in Metabot trial page
* Don't swallow errors when submitting Metabot purchase form
* Create Redux selector to derive isStoreUser
* Test failures to purchase Metabot trial
* Adjust implementation of POST /api/ee/cloud-add-ons/:product-type to dashes instead of underscores
* Require accepting terms of service for Metabot trial
* Move RTK and Formik error handling for Metabot into utils.ts
* Avoid use of any type in Metabot Formik error handling
* Improve control flow with side effects
* Fix typo in log message
This also backports "Proxy apis for store to get plan and addons (#65423)",
but without the python-execution bits.
Closes: https://linear.app/metabase/issue/CLO-4423/allow-to-buy-metabase-ai-from-the-admin-under-a-token-feature
References: https://github.com/metabase/metabase/pull/62513
Co-authored-by: Benoit Vinay <ben@metabase.com>
Co-authored-by: Ngoc Khuat <ngoc@metabase.com>
* Add click behavior setup information to introduction
* Update docs/embedding/sdk/introduction.md
---------
Co-authored-by: Alex Yarosh <alexandra@metabase.com>
Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>
* Update the delete button copy for better UX (#65688)
* Update the delete button copy for better UX
* Apply suggestions from code review
Co-authored-by: Phoomparin Mano <poom@metabase.com>
---------
Co-authored-by: Phoomparin Mano <poom@metabase.com>
* Update screen reader description to match the new menu copy (#65747)
* Update screen reader description to match the new menu copy
* Update frontend/src/metabase/embedding/components/ArchiveButton.tsx
Co-authored-by: metabase-bot[bot] <109303359+metabase-bot[bot]@users.noreply.github.com>
---------
Co-authored-by: metabase-bot[bot] <109303359+metabase-bot[bot]@users.noreply.github.com>
---------
Co-authored-by: Mahatthana (Kelvin) Nomsawadi <me@bboykelvin.dev>
Co-authored-by: Phoomparin Mano <poom@metabase.com>
Co-authored-by: metabase-bot[bot] <109303359+metabase-bot[bot]@users.noreply.github.com>
* metabot: handle (and propagate) request cancellation (#65461)
* metabot request cancelation (#65538)
this time it is actually working and tests are synchronized with real-world behavior
* request cancelation: this time for real (#65660)
Transfer-Encoding: chunked will cause Apache HttpClient to wrap body in ContentLengthInputStream,
which, in turn, quote: "NEVER closes the underlying stream".
So 'with-open' closing the stream was never propagated. And the new test with transfer-encoding
header captures that.
Cloud never used this to pre-fill the user on MB setup.
We have the same functionality with way less code available today, either way.
Instead of:
```
/setup/user_defaults#123456
```
do this:
```
/setup?first_name=John&last_name=Doe
```
Co-authored-by: Nemanja Glumac <31325167+nemanjaglumac@users.noreply.github.com>
When we introduced a new e2e run mode, we didn't realize this piece of code needs to be updated as well.
For people who were running the backend from source (but using e2e mode) some drivers might not have worked. Example:
```
"message": "No method in multimethod 'connection-properties' for dispatch value: :sqlite"
```
This commit fixes that.
Co-authored-by: Nemanja Glumac <31325167+nemanjaglumac@users.noreply.github.com>
* Fix static embedding or public embed outside iframe not working with #theme
* Add tests
* Make embedding prefix/suffix naming consistent
Co-authored-by: Mahatthana (Kelvin) Nomsawadi <me@bboykelvin.dev>
This is done by putting them in the full-text index using a 'simple' stemmer, just with a lower priority than a normal
stemmed words are.
Fixes#60649
Co-authored-by: Oleksandr Solovyov <alexander@solovyov.net>
* UXW-2235 Prevents 403s from SmartLinks from preventing the whole document from showing (SmartLink renders with "No access" version instead)
* UXW-2235 Adds e2e tests
Co-authored-by: Brad Anderson <brad@metabase.com>
* Add /livez and /readyz endpoints
* nit comment
* Update tests with new endpoints
* Update src/metabase/server/routes.clj
* bad bot
* fix tests
* fix license test failure
https://github.com/metabase/metabase/pull/65034 bumped some jetty deps
from 12.0.24 to 12.1.3. Along for the ride came a new org:
```
org.eclipse.jetty.ee9.websocket/jetty-ee9-websocket-jetty-server 12.1.3
X org.eclipse.jetty.ee9/jetty-ee9-servlet 12.1.3 :use-top
. org.eclipse.jetty.ee9/jetty-ee9-webapp 12.1.3
. org.eclipse.jetty/jetty-xml 12.1.3
. org.eclipse.jetty/jetty-util 12.1.3
X org.slf4j/slf4j-api 2.0.17 :use-top
. org.eclipse.jetty.ee/jetty-ee-webapp 12.1.3 <--- new
```
This has a new org of `org.eclipse.jetty.ee` and no discernable license
in the pom or the jar.
Refresher we look in the pom for things like
```
<licenses>
<license>
<name>The Apache Software License, Version 2.0</name>
<url>http://www.apache.org/licenses/LICENSE-2.0</url>
<distribution>repo</distribution>
</license>
</licenses>
```
or in the jar for some common names of licenses, like in this mysql
connector jar:
```
-rw-r--r-- 134 18-Sep-2023 19:11:56 INFO_SRC
-rw-r--r-- 71049 18-Sep-2023 19:11:56 LICENSE
-rw-r--r-- 1624 18-Sep-2023 19:11:56 README
-rw-r--r-- 846 18-Sep-2023 19:12:04 META-INF/INDEX.LIST
```
I didn't see a license at either of these spots, but consulted
https://mvnrepository.com/artifact/org.eclipse.jetty.ee/jetty-ee-webapp
to see it listed as Apache 2.0 and EPL 2.0
---------
Co-authored-by: Rodrigo López Dato <rodrigo@metabase.com>
Co-authored-by: metabase-bot[bot] <109303359+metabase-bot[bot]@users.noreply.github.com>
Co-authored-by: dan sutton <dan@dpsutton.com>
Part of #64774.
Replaces quadratic (refs * columns) search in `add-alias-info` with map lookups
by ID and `:lib/source-column-alias`. They don't always succeed, but I ran the tests with a fallback case
that checked the old quadratic way too - in no case does the new match fail but the old way succeed.
- Speeds up the basic query of wide tables (see `qp.perf-test`) about 7x, from 8825ms to 1435ms.
- Reduces the big user query from #64774 from 205ms to 145ms on my machine.
Co-authored-by: Braden Shepherdson <braden@metabase.com>
Part of #64774.
Now that most of the query processor machinery has been ported to use MBQL 5 and the `metabase.lib.core`
functionality, there were several needless round trips between MBQL 4 and 5.
The `driver.sql.qp/preprocess` function was converting queries from MBQL 5 to 4 up front,
and then each of three inner operations (now implemented in MBQL 5) were wrapped to convert 4 to 5 and back.
Removing this wasted effort reduced the time on my machine for the big user query from #64774 from 205ms to 145ms.
Co-authored-by: Braden Shepherdson <braden@metabase.com>
* fix cut not removing text from input
* ensure we fetch all data for all possible models for smart links
* remove unused comment
* fixes smart links in metabot input sometimes degrading into their text representation and fixes some cases where wrong model name ("model" instead of "dataset") was used for metabot links
* fix unit tests
* fix phantom selection when cutting value from input (context: https://github.com/metabase/metabase/pull/65255\#pullrequestreview-3436250849)
Co-authored-by: Sloan Sparger <sloansparger@users.noreply.github.com>
- catalog metadata is not transactional in redshift
- calls like pg_catalog.has_schema_privilege when using a string argument will resolve the oid of the schema/table etc
- if the oid cannot be resolved (e.g. table/schema does not exist) an error is thrown
We occasionally see sync failed due to relation does not exist, one cause might be concurrent test runs / cleanup deleting tables/schemas while running.
I have fixed the first half of the union to use the oid form of these calls which will result in NULL instead of an error while racing with DDL in other runs.
Need to research on external tables before changing the external table sub-query: probably want to join to svv_external_schemas and use the oid there, but not 100% yet.
Co-authored-by: Dan Stone <wotbrew@gmail.com>
* fix: update harbormaster API use
The Harbormaster API was updated to use snake_case in all params
and returns in https://github.com/metabase/harbormaster/pull/6663.
There's an backwards compatible interim period where it will support both kebab-case and snake_case in params, and keep existing returns.
The changes in this PR will make Metabase forwards and backwards compatible with the Harbormaster API changes.
After the interim period, the Harbormaster API will only take and return snake_case data.
* feat: support missing api key on hm.client/call
The OpenAPI spec is now public and the api key is not mandatory.
Co-authored-by: Filipe Silva <filipematossilva@gmail.com>
This has been a very valuable benchmark for processing complex queries,
especially large `:case` statements and nested aggregations.
Adding it with some REPL-able benchmarks in `query-processor.perf-test`.
Co-authored-by: Braden Shepherdson <braden@metabase.com>
If you have an empty first stage followed by "real" later stages, the frontend produces duplicate buttons in the query
builder. Cleaning those stages up removes those buttons.
Co-authored-by: William <william@metabase.com>
* add debug message type for tool calls
* rename toolCalls -> activeToolCalls
* Basic debug mode slash command to hide/show tool calls
* ToolCallDetailsModal + better formatting
* Fix type
* Fix types, some simplification
* Address minor bot comments
* small changes
* move AgentToolCallMessage into its own file and add copy buttons + tool call id to modal
* fix copied value
* add unit tests
* move failure case further downstream to mitigate any possible customer issue
* ci fixes
---------
Co-authored-by: Noah Moss <32746338+noahmoss@users.noreply.github.com>
Co-authored-by: Sloan Sparger <sloansparger@gmail.com>
* ditch our manual batches of rows for bufferedwriter
Q: Why did we use a manual batch at first?
A: i noticed it was super slow and worked around this. Then realized i
was gzipping it so that was the actual slowdown. The batching was
already done and it didn't hurt anything.
Q: Why get rid of it then?
A: want to move to "cell" based counting. Rows are fine until they get
really wide and then 300 columns * 5,000 rows = 1,500,000 cells in
memory at once. Put some strings in there and we blow up. So lets switch
to cell counting rather than row counting. This is way easier if we only
care about number of rows or cells when holding them in memory. Don't
have to count the batch ourself when streaming. So if that goes row by
row, we just watch for the number of cells as the trigger to switch to
streaming and we are done.
```clojure
temp-storage-test=> (time
(mt/test-driver :postgres
(select-keys (mt/process-query (mt/native-query {:query "SELECT
row_num AS id,
-- Random Integer between 1 and 100000
floor(random() * 100000 + 1)::INTEGER AS random_int,
-- Random Float
(random() * 1000)::NUMERIC(10, 4) AS random_float,
-- Random Date within the last 5 years
(current_date - (random() * 5 * 365)::INTEGER)::DATE AS random_date,
-- Random Timestamp within the last 30 days
(NOW() - (random() * 30 * '1 day'::interval)) AS random_timestamp,
-- Random String: Generates a random alphanumeric string about 1000 characters long
(
SELECT string_agg(
chr(
(CASE WHEN random() < 0.5 THEN 65 ELSE 97 END) + (random() * 25)::INTEGER
),
''
)
FROM generate_series(1, 1000)
) AS random_string
FROM
generate_series(1, 500000) AS row_num;"})
(temp-storage/notification-rff 2000 :testing))
[:row_count :status :data.rows-file-size :notification/truncated?])))
"Elapsed time: 1457.921083 msecs"
{:row_count 15184,
:status :completed,
:data.rows-file-size 16369108,
:notification/truncated? true}
```
* Switch to cell based counts, not rows
We found in practice some people might have queries with 300
columns. When we keep 5,000 of those in memory, we end up having
1,500,000 cells in memory at once. We then are open again to OOM
issues. So this swaps over to a cell basec count rather than rows to be
a bit more granular.
* Change from 2,000 rows to 20,000 cells
the thinking here is 2,000 rows with 10 values. It's pretty arbitrary though
* rename var for cells to disk threshold
* i have to delete this test because i used `time` in a comment block
* formatting fix
Co-authored-by: dpsutton <dan@dpsutton.com>
When a join's conditions came from multiple tables, we used the first table as the table name, even though some of the cols came from a different table. We now display "Previous results" to denote that the lhs columns are from different tables.
Co-authored-by: William <william@metabase.com>
* updated segments ui
* sober up the needle
* setting up unit tests
* make gauge needle opaque
* remove key, update unit test
Co-authored-by: Nick Fitzpatrick <nick@metabase.com>
* new click and link behavior
* Apply suggestions from code review
* handle handlelink
* fix typo
---------
Co-authored-by: Alex Yarosh <alexandra@metabase.com>
Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>
* reverts the color-scheme meta tag
* adds the color scheme meta tag if entry point is not embed-sdk
* add backend tests for color-scheme meta tags
* add a cypress test for color-scheme
Co-authored-by: Phoomparin Mano <poom@metabase.com>
* Fix docs
it throws an error in the js console about webassembly not having the
right permissions
Two ways to fix: give permissions, or rollback to a version that works.
I'm pinning to a version that works since we are close to a release.
39.2 breaks, 39.1 works.
I'm also including the recommended stylesheets
In the meantime, i'm going to investigate
resources/openapi/index_external.html.mustache as i'm not sure what uses
this and if it needs to be updated or deleted. Perhaps used by `clojure
-M:ee:doc api-documentation`?
* switch template to pinned 39.1 version
Co-authored-by: dpsutton <dan@dpsutton.com>
* Fix typo in 'Checking for breaking changes' section
* Fix typo in 'Checking for breaking changes' section
Co-authored-by: Alex Yarosh <alexandra@metabase.com>
* Add breaking changes detection information
Added section on checking for breaking changes when saving questions, models, or metrics.
* Add breaking changes check to model editing documentation
Added section on checking for breaking changes when saving a model.
* Apply suggestions from code review
---------
Co-authored-by: Alex Yarosh <alexandra@metabase.com>
Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>
* show new library icon
* UXW-2115 Address PR feedback - makes it so only the Library uses the synced collection icon, not the nested subdirectories
* UXW-2115 Fix getCollectionType condition
* UXW-2115 Removes unnecessary type changes
* UXW-2115 Reverts changes for synced collection icon
---------
Co-authored-by: Aleksandr Lesnenko <alxnddr@users.noreply.github.com>
Co-authored-by: Brad Anderson <brad@metabase.com>
* Updates 35 SVG illustrations
The same asset now works well for both light and dark mode.
* branding OnboardingIllustration, adding cloug and bug metabot to metabot logo
* PR Feedback
---------
Co-authored-by: AlessioLaiso <7041012+AlessioLaiso@users.noreply.github.com>
Co-authored-by: Nick Fitzpatrick <nickfitz.582@gmail.com>
* fix large command palette subtext elements, add loki testing
* how did I miss this?
* updating Overlay demo
Co-authored-by: Nick Fitzpatrick <nick@metabase.com>
* spark failure fix
I cannot run spark locally so from CI:
```
FAIL in metabase.driver.sparksql-test/friendly-inline-strings-in-convert-to-sql-test (sparksql_test.clj:67)
:sparksql
The native query returned in query results should use user-friendly splicing
expected: "SELECT COUNT(*) AS `count` FROM `test_data`.`venues` AS `t1` WHERE `t1`.`name` = 'wow'"
actual: "SELECT COUNT(*) AS `count` FROM `test_data`.`venues` AS `t1` WHERE `t1`.`name` = 'wow'"
diff:
expected: "SELECT COUNT(*) AS `count` FROM `test_data`.`venues` AS `t1` WHERE `t1`.`name` = 'wow'"
actual: "SELECT COUNT(*) AS `count` FROM `test_data`.`venues` AS `t1` WHERE `t1`.`name` = decode(unhex('776f77'), 'utf-8')"
```
Locally hit this on an m1 mac:
```
impl.get-or-create :: create-database! failed; destroying :sparksql database “test-data” {mb-test=metabase.driver.sparksql-test/friendly-inline-strings-in-convert-to-sql-test}
java.util.concurrent.TimeoutException: Timed out after 30.0 mins
```
* fix license test failure
https://github.com/metabase/metabase/pull/65034 bumped some jetty deps
from 12.0.24 to 12.1.3. Along for the ride came a new org:
```
org.eclipse.jetty.ee9.websocket/jetty-ee9-websocket-jetty-server 12.1.3
X org.eclipse.jetty.ee9/jetty-ee9-servlet 12.1.3 :use-top
. org.eclipse.jetty.ee9/jetty-ee9-webapp 12.1.3
. org.eclipse.jetty/jetty-xml 12.1.3
. org.eclipse.jetty/jetty-util 12.1.3
X org.slf4j/slf4j-api 2.0.17 :use-top
. org.eclipse.jetty.ee/jetty-ee-webapp 12.1.3 <--- new
```
This has a new org of `org.eclipse.jetty.ee` and no discernable license
in the pom or the jar.
Refresher we look in the pom for things like
```
<licenses>
<license>
<name>The Apache Software License, Version 2.0</name>
<url>http://www.apache.org/licenses/LICENSE-2.0</url>
<distribution>repo</distribution>
</license>
</licenses>
```
or in the jar for some common names of licenses, like in this mysql
connector jar:
```
-rw-r--r-- 134 18-Sep-2023 19:11:56 INFO_SRC
-rw-r--r-- 71049 18-Sep-2023 19:11:56 LICENSE
-rw-r--r-- 1624 18-Sep-2023 19:11:56 README
-rw-r--r-- 846 18-Sep-2023 19:12:04 META-INF/INDEX.LIST
```
I didn't see a license at either of these spots, but consulted
https://mvnrepository.com/artifact/org.eclipse.jetty.ee/jetty-ee-webapp
to see it listed as Apache 2.0 and EPL 2.0
Co-authored-by: dpsutton <dan@dpsutton.com>
* Make Trunk analytics always run
This prevents false positives because Trunk analytics actio will now
always analyze test results, and compare them with their own test
quarantine list. It uses this information to know if it should fail the
job or let it pass.
* Introduce dry-run
When a PR is in a draft mode, we don't want to upload results but still
have to quarantine tests.
Co-authored-by: Nemanja Glumac <31325167+nemanjaglumac@users.noreply.github.com>
Part of #64774.
Hopefully this is sufficient to unblock users seeing the high CPU times
like in #64774. I discovered other, less significant performance issues
during this investigation that I'll be addressing in follow-up PRs.
The problem query and relevant table metadata from a helpful customer
was taking 950ms to preprocess and compile on my 2022 M1 Max, running
Metabase in dev mode but with Malli schema enforcement disabled.
The query is pretty big! It uses 3 tables with 200-300 columns each,
has several stages including some from a joined card, and ends up with
several stages doing two levels of aggregation. It also includes
several *huge* `:case` expressions.
The optimizations below have reduce the time required from 950ms to 310ms
on my machine. A 3x improvement, but I think there's more to be gained in
the follow-up PRs I mentioned.
Main optimizations:
- `lib.util/find-stage-index-and-clause-by-uuid` pre-computed and cached
on each stage, saved 120ms on the above query.
- `lib.metadata.cached-provider/fields` attempts to pre-load the cache
of individual `lib.metadata/field` calls, having just fetched them
all, but it used the wrong key and they were all misses instead!
- `expression-metadata` and `aggregation-metadata` are recomputed over
and over again by their stage and later stages; cache them by value
- `returned-columns` was cached at the top level and frequently missed
because of dumb details like explicit stage 1 vs. -1 for last stage.
- Switched it to caching on individual `returned-columns-method`s
with more precise cache keys.
Both of the latter two are using a new caching approach: a weak map
using queries as keys. Since CLJ(S) maps are immutable, any change to
the query means a new pointer.
That's still far more eviction than optimal! Most edits to a query
don't actually invalidate most of the cached values! But it's a lot
better than what we had before, and because the map's keys are weak
references, the queries and their cached data are not retained any
longer than the query itself.
I also added a `qp.perf-test` namespace with some metadata providers
with 10-, 100-, 1000- and 10,000-column tables for testing.
So far the tests it performs are quite basic, though. I'll be adapting
the user-provided query into a test case in the future, since it's an
excellent real-world example of a complex but not pathological query.
Co-authored-by: Braden Shepherdson <braden@metabase.com>
* Update wording for subscriptions and alerts section
* Update docs/permissions/application.md
---------
Co-authored-by: Alex Yarosh <alexandra@metabase.com>
Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>
* upgrade to echarts v6
* Update Loki Snapshots
* Fix BE test
* update gridlines helper function
* adjust test
---------
Co-authored-by: Aleksandr Lesnenko <alxnddr@users.noreply.github.com>
Co-authored-by: Nick Fitzpatrick <nickfitz.582@gmail.com>
* Make sure /dirty is called when moving an entity out of the library
* Untouch embedding code / Clean up / Add some tests
* Check cache from both entities and RTK / Undo unnecessary changes
* Make sure getCollectionFromCollectionsTree gets collection from latest query
Co-authored-by: Anderson <ands.alves.nunes@gmail.com>
* Recreate model index quartz triggers on job initialization when needed
* whitespace
* rename var + add comment
Co-authored-by: Noah Moss <32746338+noahmoss@users.noreply.github.com>
* fix: allow stacktraces to be suppressed in API
Allows administrators to configure their metabase instance using
MB_HIDE_STACKTRACES to prevent stacktraces from appearing in API
response. For standard responses it replaces unhandled exceptiosn with a
generic message and 500 response. For streaming responses it leaves the
error message and strips out the trace/via keys to prevent exception
info from leaking.
* fixup! dissoc traces from map errors
Co-authored-by: Edward Paget <edpaget@users.noreply.github.com>
* Compute param values from card results
seq for bool
Tests, multi-stage queries
Compute values also for params on nested stages
whitespace
Unused ns
Last stage check first
Docstring
Drop base query's last stage clauses
Address review remarks
Apply suggestions from code review
Remove redundant ops
Adjust for test
Search for col also in aggregated result columns
Look for breakoutable column only if source-uuid is present
* symb
* Rebase fix
* fix rebase, remove unused module
* post rebase, fix getting database id
* kondo
* Revamp
* limit
* Adjust the solution for impl joins
* Fix e2e
* Comment
* Cleanup
* Cleanup
* comment
* comment
* Update src/metabase/parameters/dashboard.clj
* indent
---------
Co-authored-by: lbrdnk <lbrdnk@users.noreply.github.com>
Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>
* UXW-1991 Fixes various issues both new and old with the Branding page like the colors not updating until refresh and swatches not updating when setting colors back to their defaults
* UXW-1991 Uses hex format for brand colors placeholders
* UXW-1991 Fix persistence issues with setting/unsetting accent colors
Co-authored-by: Brad Anderson <brad@metabase.com>
* Remove mention of "night mode" and image
Removed mention of 'Night mode' from fullscreen dashboard instructions.
* Delete dark mode/night mode image
Co-authored-by: Robert Kosara <rkosara@me.com>
we want to not hammer the network when things are down. But during
startup we don't need to circuit break these errors that we throw before
the app-db is read
* Enhance glossary section with context for definitions
This is a smaller change than I expected, since glossary terms don't show up in searches yet. Might just cut this if it doesn't add anything useful. I also wasn't sure if we should mention Metabot here.
* Add dark mode options
Adds the available options and explains that you don't have to reload the page
* Add a little clarification to the computed goal example
This confused me, so I'm adding this little clarification
* Jeff's suggested change
* Remove the bit about LLMs again
---------
Co-authored-by: Robert Kosara <rkosara@me.com>
Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>
* Fix logs upload for drivers
When we introduced `continue-on-error` to the "test" step, we had to
update the conditionals for the test report steps as well. I've
introduced a bug by checking for a failure, instead of saying - upload
test reports whenever the job is not successful.
Hint: test.outcome means "step status BEFORE continue-on-error overrides
the exit code"
* Prevent trashed documents from being edited
* Add test
* Soften the test name tone
* Disallow even updating document when archived via API
* Add BE test
* Fix Cypress test issue reference
* Update tests after merging master
* Only check for archived when not unarchiving
* Don't check archived documents when archiving or unarchiving them
* Add missing condition from merging
Co-authored-by: Mahatthana (Kelvin) Nomsawadi <me@bboykelvin.dev>
* Add a grace period to token checks
There's more work on this namespace to be done i think, but this is a
first cut at it. There's a lot of control flow by exceptions which is
how we jump around in here. Fundamentally this should be far
simpler. Will take a crack at that shortly. I also don't love that this
just hammers the `fetch-token-status` function and then memoizes inside
of that. Would prefer simpler layout of `token-features` function and
that can be aware of population issues.
Adds a simple protocol: GracePeriod, which has two functions: save! and
retrieve. On successful token check, we `save!` the response. On errors,
we `retrieve` previous responses. This obviously can be nil in the case
we have never had a successful response, and will become nil once the
grace period's internal notion of time expires.
This implementation is based on guava's cache which has a nifty
expireafterwrite functionality.
```
(*token-features*) ;; call this function to simulate the check of features
Checking with the MetaStore to see whether token 'f325...d01a' is valid...
Reporting Metabase stats: {... user counts etc}
-> #{"features come back"}
;; meanwhile clear cache to simulate elapsing of 12 hours
(*token-features*) ;; call function again to simulate check of features
2025-10-22 21:30:26,620 INFO premium-features.token-check :: Checking with the MetaStore to see whether token 'f325...d01a' is valid...
2025-10-22 21:30:26,622 ERROR premium-features.token-check :: Error fetching token status from https://token-check.metabase.com:
clojure.lang.ExceptionInfo: network issues ;; hit network issue
2025-10-22 21:30:26,632 INFO premium-features.token-check :: Using token from grace period
->> #{graceperiod network features}
...
;; here after 5 seconds (in real life will be 36 hours)
2025-10-22 21:31:00,064 WARN premium-features.token-check :: Removing token: f325...d01a from grace period cache {mb-quartz-job-type=Cache}
```
* tests that updating value populates grace period cache
* couple changes
protect against storing nil value. guava cache doesn't like it so gone
it is.
added some tests:
```clojure
(testing "Setting a new value populates the grace period cache"
(let [new-token (tu/random-token)]
(binding [http/request (fn [& _] {:status 200
:body "{\"valid\":true,\"status\":\"fake\",\"features\":[\"fake\",\"features\"]}"})]
(mt/discard-setting-changes [premium-embedding-token]
(premium-features.settings/premium-embedding-token! new-token)
(is (= #{"fake" "features"} (token-check/*token-features*)))
(is (= {:valid true :status "fake" :features ["fake" "features"]}
(token-check/retrieve token-check/grace-period new-token)))))))
```
But this ended up leaving a setting table value for this value. i don't
feel like looking into discard setting changes so i'm gonna just remove it
* attempt to simplify the api a bit
the thee important functions are now `token-information`, `harness` and
`decode-token`.
Everyone should use token-information. Harness hooks up the grace period
and retry strategies, and `decode-token` only cares about decoding a
token. It should return the token-status map {:valid true ...} or throw.
* fixup test
* clean up tests and rename a bit
* remove comment block
* remove commented out grace period
* bit of cleanup
* fix with-random-premium-token! macro
* function composes them instead of caller
rather than a caller needing to know all of the layers, i thread them
for you and let you provide all arugments in one function call.
dynamic var for needing to customize layers in tests. largely for
removing the circuit-breaker which is stateful and hard to restart if
you want to simulate errors in tests
* rename test
* Update error message for remote sync dep checking
Try to make these error messages less jargony because they are presented
by the user interface when dependency checking content fails.
Closes UXW-2142 and UXW-1879.
* Update some missed tests
Co-authored-by: Edward Paget <edpaget@users.noreply.github.com>
* fix: improve collections graph performance
Fixes performance generating and updating a collection graph by avoiding
the cross join of all negative permissions (blocks) between groups and
collections. Instead it builds sparse graph where missing entries for
group/collection paris can be assumed to be :none permissions.
In practice this drops a collection permission update on a graph of 5K
groups and 5K collections from a greater than 10min operation on my
local machine against MySQL to a .5s operation.
* fixup! explicilty disj :root in other places
* fixup! improve some docs
* fixup! improve more docs
* fixup! seq checks
* fixup! blank lines
* fixup! convert raw sql concat into honeysql
* fixup! e2e test
Co-authored-by: Edward Paget <edpaget@users.noreply.github.com>
* remove border from dashboard header where header=false and no tabs
* move css code up to keep no border in fullscreen
Co-authored-by: Nicolò Pretto <info@npretto.com>
2025-10-30 15:43:17 +00:00
2374 changed files with 698066 additions and 569929 deletions
### Don't: Use `ms/TemporalString` for Java Time objects in response schemas
```clojure
;; WRONG - Java Time objects aren't strings yet
[:date_joinedms/TemporalString]
;; RIGHT - schemas validate BEFORE JSON serialization
[:date_joined:any];; Java Time object, serialized to string by middleware
[:last_login[:maybe:any]];; Java Time object or nil
```
**Why:** Response schemas validate the internal Clojure data structures BEFORE they are serialized to JSON. Java Time objects like `OffsetDateTime` get converted to ISO-8601 strings by the JSON middleware, so the schema needs to accept the raw Java objects.
### Don't: Use `[:sequential X]` when the data is actually a set
```clojure
;; WRONG - group_ids is actually a set
[:group_ids{:optionaltrue}[:sequentialpos-int?]]
;; RIGHT - matches the actual data structure
[:group_ids{:optionaltrue}[:maybe[:setpos-int?]]]
```
**Why:** Toucan hydration methods often return sets. The JSON middleware will serialize sets to arrays, but the schema validates before serialization.
### Don't: Create anonymous schemas for reused structures
Use `mr/def` for schemas used in multiple places:
```clojure
(mr/def::User
[:map
[:idpos-int?]
[:emailstring?]
[:namestring?]])
```
## Finding Return Types
1.**Look at the function being called**
```clojure
(api.macros/defendpoint:get"/:id"
[{:keys[id]}]
(t2/select-one:model/Field:idid));; Returns a Field instance
```
2.**Check Toucan models for structure**
Look in `src/metabase/*/models/*.clj` for model definitions.
org.bouncycastle/bcpkix-jdk18on{:mvn/version"1.81"}; Bouncy Castle crypto library -- explicit version of BC specified to resolve illegal reflective access errors
@@ -37,7 +37,7 @@ The core team runs a pretty well defined product process. It is actively being t
### A) Identify product needs from the community
We actively look for new feature ideas from our community, user base and our own use of Metabase internally. We concentrate on the underlying _problem_ or _need_ as opposed to requests for specific features. While sometimes suggested features are built as requested, often we find that they involve changes to existing features, and perhaps an entirely different solution to the underlying problem. These will typically be collected in a number of issues, and tagged [Proposal](https://github.com/metabase/metabase/labels/.Proposal)
We actively look for new feature ideas from our community, user base and our own use of Metabase internally. We concentrate on the underlying _problem_ or _need_ as opposed to requests for specific features. While sometimes suggested features are built as requested, often we find that they involve changes to existing features, and potentially an entirely different solution to the underlying problem. These will typically be collected in a number of issues, and tagged [Proposal](https://github.com/metabase/metabase/labels/.Proposal)
### B) Synthesize these needs into a concrete feature
@@ -107,4 +107,4 @@ Generally speaking, any issue in [Help Wanted](https://github.com/metabase/metab
### #YOLO JUST SUBMIT A PR
If you come up with something really cool, and want to share it with us, just submit a PR. If it hasn't gone through the above process, we probably won't merge it as is, but if it's compelling, we're more than willing to help you via code review, design review and generally OCD nitpicking so that it fits into the rest of our codebase.
If you come up with something cool, and want to share it with us, just submit a PR. If it hasn't gone through the above process, we probably won't merge it as is, but if it's compelling, we're more than willing to help you via code review, design review and generally OCD nitpicking so that it fits into the rest of our codebase.
@@ -55,7 +55,7 @@ There are multiple ways to run actions:
Just something to clarify here: actions, even though they are added to models, make their changes to the underlying table that a model queries. Which means that anyone who has access to the underlying table, or to questions or other models based on that table, will be able to see the effects of an action. Tools other than Metabase that are connected to that database will also pick up these changes.
In this sense, models are containers for actions; models are a way to organize actions. In fact, you could (in theory) add a [custom action](./custom.md) to a model that performs some update unrelated to its model's data. For example, you could write a custom action that updates the `Accounts` table, and add that action to a model that only queries an unrelated table (e.g., the `Orders` table). But, you know, maybe don't do that (unless you have a really good reason). [Basic actions](./basic.md), however, are only be available for models that wrap a single raw table.
In this sense, models are containers for actions; models are a way to organize actions. In fact, you could (in theory) add a [custom action](./custom.md) to a model that performs some update unrelated to its model's data. For example, you could write a custom action that updates the `Accounts` table, and add that action to a model that only queries an unrelated table (e.g., the `Orders` table). But, you know, maybe don't do that (unless you have a good reason). [Basic actions](./basic.md), however, are only be available for models that wrap a single raw table.
Before using actions in production, consider playing around with actions on some sample data (like the Sample Database included with Metabase) to get a feel for how they work.
@@ -5,14 +5,14 @@ summary: Metabot is Metabase's AI assistant that helps you analyze data, create
# Metabot - Metabase's AI assistant
{% include beta-blockquote.html
message="For now, <a href='https://www.metabase.com/features/metabot-ai'>Metabot</a> is only available as an add-on for Pro and Enterprise plans on Metabase Cloud."
%}
> Metabot is only available as an add-on on Metabase Cloud.

Metabot helps you analyze your data by creating charts from natural language, generating SQL queries, fixing query errors, and analyzing existing visualizations.
## Set up Metabot
To set up Metabot, see [Metabot settings](./settings.md).
## What Metabot can do
@@ -21,6 +21,7 @@ Metabot can help you to:
- [Create a chart using the query builder](#how-metabot-uses-the-query-builder) from a natural language query.
- [Generate SQL in the native editor](../questions/native-editor/writing-sql.md) from natural language. (Currently, only SQL is supported.)
- [Edit SQL directly in the native editor](#inline-sql-editing).
- [Analyze a chart](#analyze-charts-with-metabot).
- [Fix errors in SQL code](#have-metabot-fix-sql-queries).
- Answer questions from our documentation (as in, the literature you're reading right now).
@@ -46,6 +47,7 @@ Some tips:
- Whenever you want Metabot to do something completely different, you should reset the conversation, as Metabot might find that irrelevant historical context to be confusing.
- Once Metabot creates a question for you, you can follow up with more questions or take over yourself. You can drill through the chart or step into the editor to tweak the query (both in the query builder and the SQL editor).
- Metabot works best with English prompts. While it might understand other languages, you'll get the most reliable results by asking your questions in English.
- Define domain-specific terms in the [glossary](../exploration-and-organization/data-model-reference.md#glossary) to help Metabot understand your organization's terminology.
### Metabot response menu
@@ -92,6 +94,17 @@ If you don't specify a specific table in a natural language question, Metabot wi
When you get an error in a SQL query, you can click the **Have Metabot fix it** button, and Metabot will try to correct the query. You can also ask Metabot to fix your SQL in the chat.
You can ask Metabot to generate or edit code in the SQL editor, without using the [chat sidebar](#the-metabot-chat-sidebar):
1. Open the [SQL editor](../questions/native-editor/writing-sql.md).
2. Press Cmd+; on Mac or Ctrl+; on Windows.
3. Describe the SQL you want to generate or modify.
4. Review the generated code and accept or reject it.
## Navigating after Metabot creates a chart
If Metabot creates a query or takes you to a new item but you want to return to the previous screen, you can navigate using your browser's back button.
@@ -5,9 +5,7 @@ summary: Configure Metabot settings, including which collections Metabot can acc
# Metabot AI settings
{% include beta-blockquote.html
message="For now, <a href='https://www.metabase.com/features/metabot-ai'>Metabot</a> is only available as an add-on for Pro and Enterprise plans on Metabase Cloud."
%}
> Metabot is only available as an add-on on Metabase Cloud.
_Settings > Admin settings > AI_
@@ -15,17 +13,33 @@ This page covers admin settings for Metabase's AI assistant, [Metabot](./metabot

Before using Metabot in your Metabase, you'll need to add the Metabot add-on to your instance in Metabase Store.
## Add Metabot from the Metabase Store
For now, Metabot is only available for Metabase Cloud. Before you can set up Metabot in your Metabase, you'll need to add Metabot to your subscription from the Metabase Store.
1. Go to [store.metabase.com](https://store.metabase.com).
2. Log in with your **Metabase Store account** (distinct from the account you use to log into your Metabase).
3. In the **Instances** tab, find the instance you'd like to add Metabot to, and click "Add Metabot AI".
4. Pick the plan based on the number of requests you expect you'll need.
A "request" is any message anyone in your Metabase sends to Metabot. Several messages sent within the same chat session are counted as separate requests. Requests are added across the entire instance.
5. Read through the [terms of service](https://www.metabase.com/license/hosting) and click **Add Metabot AI**.
Once you've added Metabot AI in the Metabase store, you can log into your Metabase and configure it in _Admin settings > AI_.
## Verified content
Admins on Pro and Enterprise plans can tell Metabot to only work with [models](../data-modeling/models.md) and [metrics](../data-modeling/metrics.md) that have been [verified](../exploration-and-organization/content-verification.md).
Restricting Metabot to verified models and metrics (and only models and metrics) can help Metabot produce more reliable answers, since you know someone has at least vetted the data Metabot can use.
Restricting Metabot to verified models and metrics (and only models and metrics) helps Metabot produce more reliable answers, since you know someone has at least vetted the data Metabot can use.
## When embedding Metabot, you can pick a collection for Metabot to have access to
When embedding Metabot in your app, you can select a collection for Metabot:
1. Go to **Settings** > **Admin settings** > **AI**.
2. Click **Embedded Metabot**.
3. In the **Collection Embedded Metabot can use** section, click **Pick a collection**.
4. Select the collection that contains the models and metrics you want Metabot to use.
@@ -38,17 +52,11 @@ Alternatively (or additionally), you can restrict Metabot to [verified content](
The best thing you can do to improve Metabot's performance is to prep your data like you would for onboarding a new (human) hire to your data. In practice, this means you should:
- [Add models and metrics to your Metabot collection](#add-models-and-metrics-to-your-metabot-collection)
- [Add descriptions for your data and content](#add-descriptions-for-your-data-and-content)
- [Make sure the semantic types for each field are correct](#make-sure-the-semantic-types-for-each-field-are-correct)
- [Define domain-specific terms in the glossary](#define-domain-specific-terms-in-the-glossary)
### Add models and metrics to your Metabot collection
Create models that make it easy for Metabot to find answers to the kinds of questions you expect people to ask about your data. Create metrics that capture key business calculations that people frequently need to reference. Add these models and metrics to the collection you've designated for Metabot to learn from.
For example, if people often ask questions about customer lifetime value (LTV), create a model that joins customer data with order history and calculates LTV. Or if people frequently need to know monthly active users (MAU), create a metric that defines exactly how MAU should be calculated.
### Add descriptions for your data and content
Add descriptions to your [models](../data-modeling/models.md#add-metadata-to-columns-in-a-model), [metrics](../data-modeling/metrics.md), [dashboards](../dashboards/introduction.md), and [questions](../questions/introduction.md). Write descriptions to provide context, define terms, and explain business logic.
@@ -69,6 +77,12 @@ Make sure the semantic types for each field accurately describe the field's "mea
Metabase will try to set semantic types automatically, but you should confirm that each field has the relevant semantic type. See [Data types and semantic types](../data-modeling/semantic-types.md). You can also set semantic types for [models](../data-modeling/models.md#add-metadata-to-columns-in-a-model).
### Define domain-specific terms in the glossary
Add your organization's terminology, acronyms, and business-specific terms to the [glossary](../exploration-and-organization/data-model-reference.md#glossary). When Metabot receives a prompt, it can look up terms in the glossary to better understand your request.
For example, if you define "MRR" as "Monthly Recurring Revenue" in your glossary, Metabot will know what you mean when you ask "What's our MRR for Q4?" This is especially helpful for industry-specific jargon, internal product names, or abbreviations unique to your organization.
### Curate prompt suggestions
When you select a collection for Metabot to "learn", Metabot will suggest a series of prompts based on the content it finds in that collection. These prompts just give people a feel for the kinds of things people can ask Metabot to do.
@@ -81,9 +95,17 @@ Metabot inherits the permissions of the current user, so you don't need to set p
In other words, to restrict what data Metabot can see for each person, simply apply [data](../permissions/data.md) and [collection](../permissions/collections.md) permissions to their groups as you would normally, and those permissions will apply to their use of Metabot as well.
## Metabot is only available instance-wide, not per person
## Viewing Metabot usage
Currently, Metabot will be available to everyone who uses your Metabase.
You can see how many Metabot requests people have made this month by going to **Admin settings > Settings > License**.
If you aren't logged into the [Metabase Store](../cloud/accounts-and-billing.md).(, you'll need to log in to the store before you can view the usage. Once logged in to the store, go back to your Metabase and view the license page.
The **Metabot AI requests used, this month (updated daily)** field shows how many requests your Metabase has used this month. Each message sent to Metabot counts as a request.
## Metabot can't be enabled per person
Currently, Metabot is available to everyone who uses your Metabase.
## Metabot uses a variety of generative AI models to answer your questions
@@ -99,7 +121,7 @@ We've intentionally limited what Metabot can do. Metabot lacks access to API key
Metabot has access to your Metabase metadata and some data values to help answer your questions:
- **Table, Question, Model, Dashboard, and Metric metadata**: Metabot can see the structure and configuration of your content
- **Table, Question, Model, Dashboard, and Metric metadata**: Metabot can see the structure and configuration of your content.
- **Sample field values**: When you ask questions like "Filter everyone from Wisconsin," Metabot might check the values in the state field to understand how the data is stored (like "WI" vs "Wisconsin"). See [syncs](../databases/sync-scan.md).
- **Timeseries data**: For chart analysis, Metabot might see the timeseries data used to draw certain visualizations, depending on the chart type.
5. Give the service account **Viewer** permissions. Metabase will only have access to this folder or sheet; it won't have access to any other files in your Google Drive. If you select folder, Metabase will also sync any sheets it finds in any subfolders.
6. Click **Send** to share the folder with the Metabase service account.
7. Copy the sharing link for the folder or sheet.
8. Return to Metabase and paste the sharing link into the field that asks for "the sharing link for this folder/file".
8. Return to Metabase and paste the sharing link into "the sharing link for this folder/file" field.
**If you synced a Google Drive folder**, Metabase will sync with the Google Drive folder (and its subfolders) and import all Google Sheets, creating a new table in your Metabase Cloud Storage database for each sheet. For sheets with multiple tabs, Metabase will create a table for each tab. Metabase will only sync Google Sheets; it'll ignore other file types in the folder. After the initial sync, Metabase will sync every 15 minutes.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.