add split openapi to the repo

* Convert parameters to TypeScript

* Fix types

* Fix types in normalizeValue

* Fix types in dashboard tests

* Fix more types

* Fix test

* Remove duplicate check

* Naming

.claude/skills/mutation-testing/SKILL.md

@@ -5,227 +5,110 @@ description: Run mutation testing on a Clojure namespace, generate tests to kill
 
 # Mutation Testing Skill
 
-This skill runs mutation testing on a target namespace, writes tests to kill surviving mutations, and creates draft PRs with Linear issues for each function.
+This skill runs mutation testing end-to-end: generates a coverage report, groups functions, shells out to `claude -p` to write tests, verifies mutations are killed, and creates draft PRs with Linear issues.
 
 ## Prerequisites
 
 - A running nREPL connected to the Metabase dev environment
 - `LINEAR_API_KEY` environment variable set with a valid Linear personal API key
 - `gh` CLI authenticated with GitHub
+- `claude` CLI available on PATH
 
 ## Reference Files
 
-- `dev/src/dev/coverage.clj` — mutation testing tool (generates reports, runs mutations)
-- `dev/src/dev/mutation_testing.clj` — Linear API client and PR template helpers
-- `mutation-testing-report.lib.card.md` — example report from the pilot
+- `dev/src/dev/coverage.clj` — mutation testing engine (generates reports, runs mutations)
+- `dev/src/dev/mutation_testing.clj` — orchestration, Linear API, PR helpers
 
 ## Invocation
 
-The argument is a Clojure namespace, e.g.:
+The argument is a Clojure namespace, optionally followed by a base branch:
 
 ```
 /mutation-testing metabase.lib.order-by
+/mutation-testing metabase.lib.order-by --base-branch release-x.52.x
+/mutation-testing metabase.lib.order-by --project-id abc-123
+/mutation-testing metabase.lib.order-by --base-branch release-x.52.x --project-id abc-123
 ```
 
-## Workflow
+- `--base-branch`: defaults to the value in config (set via `set-config!`), or `"master"` if not configured.
+- `--project-id`: if provided, issues are added to this existing Linear project instead of creating a new one.
 
-### Step 1: Parse the namespace
+## Steps
 
-From the argument (e.g., `metabase.lib.order-by`), derive:
-- **Test namespace**: append `-test` (e.g., `metabase.lib.order-by-test`)
-- **Short name**: the last segment (e.g., `order-by`)
-- **Source path**: `src/metabase/lib/order_by.cljc` (replace `-` with `_` in path segments)
-- **Test path**: `test/metabase/lib/order_by_test.cljc`
+### 1. Parse arguments
 
-### Step 2: Set up Linear and load tools
+Parse `$ARGUMENTS` to extract:
+- **namespace** (required) — the first positional argument
+- **`--base-branch`** (optional) — if present, the next argument is the base branch name
+- **`--project-id`** (optional) — if present, the next argument is an existing Linear project ID
+
+### 2. Load and configure
 
 ```clojure
-(require '[dev.coverage :as cov] :reload)
 (require '[dev.mutation-testing :as mut-test] :reload)
+(require '[dev.coverage :as cov] :reload)
 ```
 
 If this is the first invocation (or the REPL was restarted), set up the Linear team:
 
 ```clojure
-;; Find the team
 (mut-test/list-teams!)
-;; Set team ID from the output
-(mut-test/set-config! {:team-id "<team-id>"})
+(mut-test/set-config! {:team-id "<id>"})
 ```
 
-If config is already set from a previous invocation, skip this.
-
-### Step 3: Generate baseline report and create Linear project
+### 3. Run
 
 ```clojure
-(cov/generate-report
-  '<target-ns>
-  ['<test-ns>]
-  "mutation-testing-report.lib.<short-name>.before.md")
+(mut-test/run! '<target-ns>)
+;; Or with options:
+(mut-test/run! '<target-ns> {:base-branch "release-x.52.x"})
+(mut-test/run! '<target-ns> {:project-id "abc-123"})
+(mut-test/run! '<target-ns> {:base-branch "release-x.52.x" :project-id "abc-123"})
 ```
 
-Read the generated report file. It has three sections:
-- **Uncovered Functions** — never called by any test
-- **Partially Covered Functions** — called but with surviving mutations
-- **Fully Covered Functions** — all mutations killed
+Pass the opts map with `:base-branch` and/or `:project-id` if those flags were provided in the arguments.
 
-Then create a Linear project for this namespace (uses report stats in the description):
+This will:
+1. Generate a baseline mutation testing report
+2. Create a Linear project for the namespace
+3. Group functions by coverage relationships
+4. For each group: create branch → invoke Claude to write tests → verify → retry if needed → commit & push → create Linear issue → create draft PR
+5. Print a summary with PR links
+
+### 4. Handle failures
+
+If a group fails mid-processing, `run!` catches the error and continues to the next group. To retry a single group manually:
 
 ```clojure
-(mut-test/create-project-for-namespace!
-  "<target-ns>"
-  "mutation-testing-report.lib.<short-name>.before.md")
-;; => {:project-id "...", :name "Mutation Testing: metabase.lib.order-by"}
-;; Automatically sets :project-id in config for subsequent create-issue! calls
+;; Get the parsed namespace info
+(def parsed (mut-test/parse-namespace '<target-ns>))
+
+;; Run coverage to get the data
+(def coverage-results (cov/test-namespace (:target-ns parsed) [(:test-ns parsed)]))
+
+;; Group and find the one you want
+(def groups (mut-test/group-functions coverage-results))
+
+;; Process just that group
+(mut-test/process-group! parsed (nth groups <index>))
 ```
 
-### Step 4: Plan the work
+### 5. Verify the prompt (dry run)
 
-Before creating branches, analyze the report to plan how to group the work:
-
-1. **Identify killable functions.** For each function with surviving mutations, assess whether any mutations are killable. Run `cov/test-mutations` to see the full mutation list. If ALL mutations for a function are unkillable (schema-only, semantically equivalent, etc.), skip that function — no branch or PR needed.
-
-2. **Group private functions by public entry point.** Private functions cannot be tested directly. Group them with the public function that exercises them. Create one branch/PR per group, not per private function.
-
-3. **Track mutations per group.** The `:mutations-before` and `:not-killed` fields in the PR should only reflect mutations for functions covered by that specific PR — never mutations from other functions.
-
-### Step 5: Process each group
-
-For each group of functions that has killable mutations:
-
-#### 5a. Create a branch
+To inspect what Claude will see without invoking it:
 
 ```clojure
-(mut-test/create-branch! "<target-ns>" "<primary-fn-name>")
-;; => "mutation-testing-lib-order-by-orderable-columns"
+(def parsed (mut-test/parse-namespace '<target-ns>))
+(def coverage-results (cov/test-namespace (:target-ns parsed) [(:test-ns parsed)]))
+(def groups (mut-test/group-functions coverage-results))
+(println (mut-test/build-test-prompt
+           (merge (select-keys parsed [:target-ns :test-ns :source-path :test-path])
+                  (select-keys (first groups) [:fn-names :mutations]))))
 ```
 
-Use the primary public function name for the branch. If the group covers private functions too, name the branch after the public entry point.
-
-#### 5b. Read context
-
-- Read the source function(s) from the source file
-- Read the full test file to understand existing test patterns, helpers, and metadata providers used
-- If the function is private, identify which public function(s) call it
-
-#### 5c. Write tests
-
-Generate the **simplest tests** that kill the surviving mutations while still making semantic sense. Guidelines:
-
-- **Never call private functions directly.** Always test through public API functions. Use the coverage data to identify which public functions exercise the private function.
-- **Keep tests simple.** Each test should verify one meaningful behavior. Don't over-engineer tests just to chase mutation kills.
-- **Follow existing patterns.** Match the style of the existing tests: same metadata providers, same helper functions, same assertion patterns.
-- **Insert tests near related existing tests** for the same function, not at the end of the file. This minimizes merge conflicts between branches. If there are no existing tests for the function, find the most logical location based on the order of functions in the source file.
-
-**IMPORTANT: Use `file_edit` (not `clojure_edit`) when editing test files.** The `clojure_edit` tool reformats the entire file, introducing unnecessary whitespace changes that pollute the diff. Use `file_edit` with exact string matching to make surgical insertions that only touch the lines you intend to change.
-
-#### 5d. Verify mutations are killed
-
-Use the REPL to check that the new tests kill the targeted mutations:
-
-```clojure
-;; Reload the test namespace to pick up new tests
-(require '<test-ns> :reload)
-
-;; Test mutations for the specific function
-(cov/test-mutations
-  '<target-ns>/<fn-name>
-  ;; Set of test names that cover this function
-  #{'<test-ns>/<test-name-1> '<test-ns>/<test-name-2>})
-```
-
-Check the `:survived` key in the result. If mutations still survive:
-- Try writing additional tests
-- If a mutation is truly unkillable (semantically equivalent), note it for the PR description
-- If a mutation reveals dead/unreachable code, consider suggesting removal as a code improvement
-
-#### 5e. Handle unkillable mutations and improvements
-
-**Do NOT edit the source namespace just for documentation.** Instead:
-
-- **Unkillable mutations**: Note them in the PR description with rationale
-- **Code improvements** (dead code removal, expression simplification) that directly relate to surviving mutants:
-  1. Make the change in the source file
-  2. Verify all tests still pass with the change
-  3. Note the file path, line range, and the improved code
-  4. **Reset the change**: `git checkout -- <source-file>` — do NOT commit it
-  5. After creating the PR, use `add-suggested-change!` to post the improvement as a GitHub suggested change (Step 5h)
-  6. The reviewer decides whether to accept it
-
-#### 5f. Commit and push
-
-```clojure
-;; Only commit the test file — never commit source changes for improvements
-(mut-test/commit-and-push! "<target-ns>" "<primary-fn-name>"
-  ["test/metabase/lib/<short_name>_test.cljc"])
-```
-
-#### 5g. Create Linear issue and draft PR
-
-```clojure
-;; Create a Linear issue (use the primary function name)
-(def issue (mut-test/create-issue-for-function! "<target-ns>" "<primary-fn-name>"))
-;; => {:identifier "QUE-1234", :url "https://linear.app/...", ...}
-
-;; Create draft PR linked to the Linear issue
-;; :fn-names lists ALL functions covered by this PR
-;; :mutations-before counts only mutations for functions in this PR
-;; :not-killed lists only unkillable mutations for functions in this PR
-(def pr-url
-  (mut-test/create-draft-pr!
-    {:target-ns         "<target-ns>"
-     :fn-name           "<primary-fn-name>"
-     :linear-identifier (:identifier issue)
-     :mutations-before  9       ;; surviving mutations for these functions only
-     :tests-added       2       ;; number of new test functions
-     :killed            ["Replace :segment-id with :segment-id__"
-                         "Replace cycle-path with nil"]
-     :not-killed        [{:description "Replace acc with nil"
-                          :rationale   "reduce always has single iteration"}]
-     :suggested-changes []}))
-;; => "https://github.com/metabase/metabase/pull/12345"
-```
-
-#### 5h. Add code improvement suggestions (if any)
-
-If you identified code improvements in step 5e (and reset them), post each as a GitHub suggested change:
-
-```clojure
-(mut-test/add-suggested-change!
-  {:pr-url     pr-url
-   :path       "src/metabase/lib/<short_name>.cljc"
-   :start-line 42    ;; first line of the code to replace
-   :end-line   45    ;; last line of the code to replace
-   :suggestion "(improved-code-here)"  ;; the replacement code
-   :comment    "**Suggested improvement:** <description of the change and why it's related to the surviving mutant>"})
-```
-
-### Step 6: Return to master and repeat
-
-```clojure
-(mut-test/return-to-master!)
-```
-
-Repeat Step 5 for the next group.
-
-### Step 7: Summary
-
-Print a summary of what was done:
-- Link to the Linear project for this namespace
-- Number of functions processed
-- Number of functions skipped (all mutations unkillable)
-- Number of draft PRs created
-- Number of mutations killed vs. unkillable
-- List of test PRs with links
-- List of mutation rule PRs with links (if any were created)
-
-## Suggesting New Mutation Rules
-
-If you notice patterns in surviving mutations that suggest the mutation testing tool should have additional rules (e.g., a common Clojure form that isn't being mutated), open a separate PR proposing additions to `dev.coverage/mutation-rules` with an explanation.
-
 ## Configuration
 
-The `dev.mutation-testing` namespace reads `LINEAR_API_KEY` from the environment and stores team/project IDs in an atom.
-
-- **Team ID**: Set once per REPL session via `(mut-test/list-teams!)` and `(mut-test/set-config! {:team-id "..."})` (Step 2)
-- **Project ID**: Set automatically when `create-project-for-namespace!` is called (Step 3) — one project per namespace
+- **Team ID**: Set once per REPL session via `(mut-test/list-teams!)` and `(mut-test/set-config! {:team-id "..."})`
+- **Project ID**: Set automatically when `run!` calls `create-project-for-namespace!`
+- **Base Branch**: Defaults to `"master"`. Override via `(mut-test/set-config! {:base-branch "release-x.52.x"})` or pass `{:base-branch "..."}` as the second arg to `run!`
+- **Project ID**: If set (via `set-config!` or `run!` opts), `run!` reuses the existing Linear project instead of creating a new one

.github/workflows/e2e-test.yml

@@ -45,8 +45,9 @@ jobs:
     name: e2e-tests-${{ inputs.name }}-${{ inputs.edition }}
     env:
       MB_EDITION: ${{ inputs.edition }}
-      # Any env starting with `CYPRESS_` will be available to all Cypress tests via `Cypress.env()`
-      # Example: you can get `CYPRESS_FOO` with `Cypress.env("FOO")`
+      # Sensitive env vars (tokens, secrets) starting with `CYPRESS_` are accessed via async `cy.env()`
+      # Example: `cy.env(["FOO"]).then(({ FOO }) => ...)`
+      # For non-sensitive config, use `Cypress.expose()` in cypress config instead (synchronous)
       CYPRESS_MB_ALL_FEATURES_TOKEN: ${{ secrets.STAGING_MB_ALL_FEATURES_TOKEN }}
       CYPRESS_MB_STARTER_CLOUD_TOKEN: ${{ secrets.STAGING_MB_STARTER_CLOUD_TOKEN }}
       CYPRESS_MB_PRO_CLOUD_TOKEN: ${{ secrets.STAGING_MB_PRO_CLOUD_TOKEN }}

.github/workflows/embedding-sdk-component-tests.yml

@@ -19,8 +19,9 @@ jobs:
         react-version: [18, 19]
       fail-fast: false
     env:
-      # Any env starting with `CYPRESS_` will be available to all Cypress tests via `Cypress.env()`
-      # Example: you can get `CYPRESS_FOO` with `Cypress.env("FOO")`
+      # Sensitive env vars (tokens, secrets) starting with `CYPRESS_` are accessed via async `cy.env()`
+      # Example: `cy.env(["FOO"]).then(({ FOO }) => ...)`
+      # For non-sensitive config, use `Cypress.expose()` in cypress config instead (synchronous)
       CYPRESS_MB_ALL_FEATURES_TOKEN: ${{ secrets.STAGING_MB_ALL_FEATURES_TOKEN }}
       CYPRESS_MB_STARTER_CLOUD_TOKEN: ${{ secrets.STAGING_MB_STARTER_CLOUD_TOKEN }}
       CYPRESS_MB_PRO_CLOUD_TOKEN: ${{ secrets.STAGING_MB_PRO_CLOUD_TOKEN }}
@@ -155,8 +156,9 @@ jobs:
     env:
       MB_RUN_MODE: "e2e"
       METASTORE_DEV_SERVER_URL: ${{ secrets.METASTORE_DEV_SERVER_URL }}
-      # Any env starting with `CYPRESS_` will be available to all Cypress tests via `Cypress.env()`
-      # Example: you can get `CYPRESS_FOO` with `Cypress.env("FOO")`
+      # Sensitive env vars (tokens, secrets) starting with `CYPRESS_` are accessed via async `cy.env()`
+      # Example: `cy.env(["FOO"]).then(({ FOO }) => ...)`
+      # For non-sensitive config, use `Cypress.expose()` in cypress config instead (synchronous)
       CYPRESS_MB_ALL_FEATURES_TOKEN: ${{ secrets.STAGING_MB_ALL_FEATURES_TOKEN }}
       CYPRESS_MB_STARTER_CLOUD_TOKEN: ${{ secrets.STAGING_MB_STARTER_CLOUD_TOKEN }}
       CYPRESS_MB_PRO_CLOUD_TOKEN: ${{ secrets.STAGING_MB_PRO_CLOUD_TOKEN }}

.github/workflows/openapi.yml

@@ -3,8 +3,8 @@ name: Validate OpenAPI
 on:
   push:
     branches:
-      - 'master'
-      - 'release-**'
+      - "master"
+      - "release-**"
   pull_request:
 
 concurrency:
@@ -39,6 +39,8 @@ jobs:
       - name: Prepare back-end environment
         uses: ./.github/actions/prepare-backend
       - name: Generate OpenAPI spec
-        run: bun run generate-openapi
+        run: bun run openapi:generate
       - name: Validate OpenAPI spec
-        run: bun run lint-openapi
+        run: bun run openapi:lint
+      - name: Split OpenAPI spec
+        run: bun run openapi:split

.gitignore

@@ -170,5 +170,5 @@ CLAUDE.local.md
 # Python dependencies (pip-installed, not committed)
 /resources/python-sources/sqlglot/
 /resources/python-sources/sqlglot-*.dist-info/
+/resources/python-sources/.lock
 __pycache__/
-

bin/openapi-check.sh

@@ -14,7 +14,7 @@ fi
 # Generate the current OpenAPI spec from Malli schemas using write-openapi-spec-to-file!
 # This creates $OPENAPI_SPEC directly
 # Using the generate-openapi-spec command from metabase.cmd.core
-bun run generate-openapi
+bun run openapi:generate
 ret=$?
 
 if [[ $ret -ne 0 ]]; then
@@ -82,7 +82,7 @@ if ! git diff --exit-code "$OPENAPI_SPEC" >/dev/null 2>&1; then
         echo ""
         echo "To fix this:"
         echo "  1. Pull the latest changes from master: git pull origin master"
-        echo "  2. Run 'bun run generate-openapi' to update $OPENAPI_SPEC"
+        echo "  2. Run 'bun run openapi:generate' to update $OPENAPI_SPEC"
         echo "  3. Commit the updated openapi.json file"
         echo ""
         echo "Differences:"

dev/src/dev/debug_qp/analyze_user_query.clj

@@ -0,0 +1,157 @@
+(ns dev.debug-qp.analyze-user-query
+  "Given access to e.g. a Cloud instance with a querying issue, this script can use a login token borrowed from your
+  browser to make API calls to the instance, and assemble a local mock `MetadataProvider` which contains the tables,
+  fields and cards necessary to run QP preprocessing and compilation locally against the same metadata the user
+  instance is seeing.
+
+  Of course the queries can't be executed without a connection to the user's DWH, which is usually impossible."
+  (:require
+   [clj-http.client :as http]
+   [medley.core :as m]
+   [metabase.lib.js.metadata :as lib.js.metadata]
+   [metabase.lib.metadata :as lib.metadata]
+   [metabase.query-processor.compile :as qp.compile]
+   [metabase.query-processor.preprocess :as qp.preprocess]
+   [metabase.util.json :as json]
+   [metabase.util.malli :as mu]))
+
+;; XXX: INSTRUCTIONS:
+;; - Log into the instance, open DevTools, and navigate to the problem question.
+;; - Open the Network tab, find a `/api/card/1234` or similar request, right click and "Copy as cURL".
+;; - Paste that somewhere, probably into your shell to test that the `curl` call is working.
+;; - Grab its `cookie` and edit it into `my-ctx` below. (It's the `-b` arg to curl.)
+;; - Edit the `instance-url` of `my-ctx` below likewise.
+;; - Edit the `database-id` of `my-ctx` below to match the problem database on the user instance.
+;;   - That can be seen in eg. `/api/card` requests and similar.
+;; - Now you can run the analysis functions below, like compile-card.
+;;   - See the examples at the bottom of this file.
+(def ^:private my-ctx
+  "Contains instance details, for passing to the various functions below."
+  {:instance-url        "https://acustomer.metabaseapp.com"  ; No trailing slash.
+   :cookie-from-browser "COPY ME FROM YOUR curl COMMAND"
+   :database-id         123})
+
+(defn- mk-headers [{:keys [cookie-from-browser instance-url] :as _ctx}]
+  {"accept" "application/json"
+   "accept-language" "en-US,en;q=0.9"
+   "cache-control" "no-cache"
+   "content-type" "application/json"
+   "cookie" cookie-from-browser
+   "pragma" "no-cache"
+   "priority" "u=1, i"
+   "referer" instance-url
+   "sec-ch-ua" "\"Google Chrome\";v=\"141\", \"Not?A_Brand\";v=\"8\", \"Chromium\";v=\"141\""
+   "sec-ch-ua-mobile" "?0"
+   "sec-ch-ua-platform" "\"macOS\""
+   "sec-fetch-dest" "empty"
+   "sec-fetch-mode" "cors"
+   "sec-fetch-site" "same-origin"
+   "user-agent" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36"})
+
+(defn- instance-fetch [{:keys [instance-url] :as ctx} path]
+  (let [headers  (mk-headers ctx)
+        request  {:method           :get
+                  :url              (str instance-url path)
+                  :accept           :json
+                  :content-type     :json
+                  :throw-exceptions false
+                  :headers          headers}
+        response (-> (http/request request)
+                     (select-keys [:body :headers :status])
+                     (update :body json/decode))]
+    (if (<= 200 (:status response) 299)
+      (:body response)
+      (throw (ex-info "Non-200 response" response)))))
+
+(defn- qm-index [objs]
+  (m/index-by (comp str #(get % "id")) objs))
+
+(defn- qm->metadata [raw]
+  ;; /query_metadata requests return a map in a different shape from what lib.js.metadata expects.
+  ;; We get {"databases" [{...}], "tables" [{..., "fields": [{...}]}]} but we need
+  ;; {"databases" {"12" {...}}
+  ;;  "tables" {"333" {...}}
+  ;;  "fields" {"4343" {...}}
+  ;;  "questions" {"12" {...}}}
+  (let [fields (for [table (get raw "tables")
+                     :when (number? (get table "id"))
+                     field (get table "fields")
+                     :when (number? (get field "id"))]
+                 field)]
+    {"databases" (qm-index (get raw "databases"))
+     "tables"    (qm-index (get raw "tables"))
+     "fields"    (qm-index fields)}))
+
+(defn- recursive-card-metadata
+  "Helper for [[metadata-for]]. Prefer calling that function instead."
+  [ctx main-card-id]
+  (loop [metadata          {}
+         cards-seen        #{main-card-id}
+         [card-id & cards] [main-card-id]]
+    (if-not card-id
+      metadata
+      (let [card         (instance-fetch ctx (str "/api/card/" card-id))
+            raw          (instance-fetch ctx (str "/api/card/" card-id "/query_metadata"))
+            nested-cards (->> (get raw "tables")
+                              (map #(get % "id"))
+                              (filter string?)
+                              (map (comp Long/parseLong #(subs % 6)))
+                              (remove cards-seen))]
+        (recur (m/deep-merge metadata
+                             {"questions" {(str card-id) card}}
+                             (qm->metadata raw))
+               (into cards-seen nested-cards)
+               (concat cards nested-cards))))))
+
+(defn- table-metadata
+  "Helper for [[metadata-for]]. Prefer calling that function instead."
+  [ctx table-id]
+  (qm->metadata {"tables" [(-> (instance-fetch ctx (str "/api/table/" table-id "/query_metadata"))
+                               (dissoc "db"))]}))
+
+;; =============================================== External API ===================================================
+(defn metadata-for
+  "Given a map like `{:card [123 456], :table [789]}` returns a `MetadataProvider` with metadata for all the tables,
+  cards and fields transitively required by the specified roots.
+
+  Generally you only need a single `{:card [123]}` for the problem card you're investigating.
+
+  Occasionally a broken card might reference fields by ID which are not really part of their query - then they're
+  missing from the `/query_metadata` on that card and will be missed by this function unless you explicitly add
+  `{:table [789]}` to the map of roots.
+
+  Returns a complete `MetadataProvider` ready to use in a QP call."
+  [{:keys [database-id] :as ctx} kind->ids]
+  (->> (for [[kind ids] kind->ids
+             id         ids]
+         (case kind
+           :card  (recursive-card-metadata ctx id)
+           :table (table-metadata ctx id)))
+       (apply m/deep-merge)
+       (lib.js.metadata/metadata-provider database-id)))
+
+(defn preprocess-card
+  "Given a `MetadataProvider` built by [[metadata-for]] and a card ID, tries to preprocess that card."
+  [metadata-provider card-id]
+  (mu/disable-enforcement
+    (-> (lib.metadata/card metadata-provider card-id)
+        :dataset-query
+        qp.preprocess/preprocess)))
+
+(defn compile-card
+  "Given a `MetadataProvider` built by [[metadata-for]] and a card ID, tries to compile that card to e.g. SQL."
+  [metadata-provider card-id]
+  (mu/disable-enforcement
+    (-> (lib.metadata/card metadata-provider card-id)
+        :dataset-query
+        qp.compile/compile)))
+
+(comment
+  ;; Example calls
+  (let [card-id 456
+        mp      (metadata-for my-ctx {:card [card-id]})]
+    (compile-card mp card-id))
+
+  (let [card-id 456
+        mp      (metadata-for my-ctx {:card [card-id]})]
+    (preprocess-card mp card-id)))

dev/src/dev/mutation_testing.clj

@@ -7,11 +7,15 @@
    Configuration:
    - LINEAR_API_KEY env var must be set
    - Call (set-config! {:team-id \"...\" :project-id \"...\"}) before creating issues"
+  (:refer-clojure :exclude [run!])
   (:require
    [cheshire.core :as json]
    [clj-http.client :as http]
+   [clojure.java.io :as io]
    [clojure.java.shell :as shell]
-   [clojure.string :as str]))
+   [clojure.set :as set]
+   [clojure.string :as str]
+   [dev.coverage :as coverage]))
 
 (set! *warn-on-reflection* true)
 
@@ -20,14 +24,21 @@
 (defonce ^:private config (atom {}))
 
 (defn set-config!
-  "Set configuration for Linear API calls.
+  "Set configuration for mutation testing.
 
    Required keys:
    - :team-id   — Linear team ID
-   - :project-id — Linear project ID (can be set after creating a project)"
+   - :project-id — Linear project ID (can be set after creating a project)
+
+   Optional keys:
+   - :base-branch — git branch to create feature branches from (default: \"master\")
+   - :project-id  — Linear project ID to add issues to (skips project creation in run!)"
   [m]
   (swap! config merge m))
 
+(defn- base-branch []
+  (get @config :base-branch "master"))
+
 (defn- api-key []
   (or (System/getenv "LINEAR_API_KEY")
       (throw (ex-info "LINEAR_API_KEY environment variable is not set" {}))))
@@ -215,14 +226,14 @@
   "Generate a branch name for mutation testing a function.
    Example: mutation-testing-lib-order-by-orderable-columns"
   [target-ns fn-name]
-  (let [short-ns (last (clojure.string/split (str target-ns) #"\."))]
+  (let [short-ns (last (str/split (str target-ns) #"\."))]
     (str "mutation-testing-lib-" short-ns "-" fn-name)))
 
 (defn create-branch!
-  "Create and checkout a new branch from master for mutation testing a function."
+  "Create and checkout a new branch from the base branch for mutation testing a function."
   [target-ns fn-name]
   (let [branch (branch-name target-ns fn-name)]
-    (sh "git" "checkout" "master")
+    (sh "git" "checkout" (base-branch))
     (sh "git" "pull")
     (sh "git" "checkout" "-b" branch)
     branch))
@@ -255,10 +266,11 @@
   (let [title (pr-title target-ns fn-name)
         body (pr-description opts)
         result (sh "gh" "pr" "create" "--draft"
+                   "--base" (base-branch)
                    "--title" title
                    "--body" body
                    "--label" "no-backport")]
-    (clojure.string/trim (:out result))))
+    (str/trim (:out result))))
 
 (defn add-pr-comment!
   "Add a general comment to a PR. `pr-url-or-number` can be a PR URL or number."
@@ -308,12 +320,17 @@
                       "-f" "start_side=RIGHT"]))]
     (apply sh args)))
 
-(defn return-to-master!
-  "Checkout master branch."
+(defn return-to-base!
+  "Checkout the configured base branch."
   []
-  (sh "git" "checkout" "master")
+  (sh "git" "checkout" (base-branch))
   nil)
 
+(defn return-to-master!
+  "Checkout the configured base branch. Deprecated: use return-to-base! instead."
+  []
+  (return-to-base!))
+
 ;;; --- Report helpers ---
 
 (defn report-stats
@@ -409,6 +426,358 @@
   (create-issue! {:title (linear-issue-title target-ns fn-name)
                   :description (linear-issue-description target-ns fn-name)}))
 
+;;; --- Orchestration ---
+
+(defn parse-namespace
+  "Parse a namespace symbol into all derived paths and names.
+   Auto-detects .cljc vs .clj by checking which file exists."
+  [target-ns-sym]
+  (let [target-ns (str target-ns-sym)
+        test-ns (str target-ns "-test")
+        short-name (last (str/split target-ns #"\."))
+        ns-path (-> target-ns
+                    (str/replace "." "/")
+                    (str/replace "-" "_"))
+        test-path-base (-> test-ns
+                           (str/replace "." "/")
+                           (str/replace "-" "_"))
+        source-ext (cond
+                     (.exists (io/file (str "src/" ns-path ".cljc"))) ".cljc"
+                     (.exists (io/file (str "src/" ns-path ".clj"))) ".clj"
+                     :else ".cljc")
+        test-ext (cond
+                   (.exists (io/file (str "test/" test-path-base ".cljc"))) ".cljc"
+                   (.exists (io/file (str "test/" test-path-base ".clj"))) ".clj"
+                   :else ".cljc")]
+    {:target-ns (symbol target-ns)
+     :test-ns (symbol test-ns)
+     :short-name short-name
+     :source-path (str "src/" ns-path source-ext)
+     :test-path (str "test/" test-path-base test-ext)
+     :report-path (str "mutation-testing-report." target-ns ".before.md")}))
+
+(defn group-functions
+  "Group functions from coverage results for batch processing.
+   Takes the output of coverage/test-namespace.
+
+   Logic:
+   - Public functions with surviving mutations each become a group
+   - Private functions get assigned to the public function with the most test overlap
+   - Functions with zero surviving mutations are skipped
+   - Uncovered or unassignable private functions get their own group"
+  [coverage-results]
+  (let [with-survivors (into {} (filter (fn [[_ v]] (seq (:survived v))) coverage-results))
+        public? (fn [fn-sym]
+                  (let [v (find-var fn-sym)]
+                    (and v (not (:private (meta v))))))
+        public-fns (into {} (filter (fn [[k _]] (public? k)) with-survivors))
+        private-fns (into {} (remove (fn [[k _]] (public? k)) with-survivors))
+        assignments (into {}
+                          (for [[priv-fn priv-data] private-fns
+                                :let [priv-tests (set (:tests priv-data))
+                                      best (when (seq priv-tests)
+                                             (->> public-fns
+                                                  (map (fn [[pub-fn pub-data]]
+                                                         [pub-fn (count (set/intersection
+                                                                         priv-tests
+                                                                         (set (:tests pub-data))))]))
+                                                  (filter (fn [[_ n]] (pos? n)))
+                                                  (sort-by second >)
+                                                  first))]
+                                :when best]
+                            [priv-fn (first best)]))
+        unassigned-private (into {} (remove (fn [[k _]] (contains? assignments k)) private-fns))
+        public-groups (for [[pub-fn pub-data] public-fns
+                            :let [attached (keep (fn [[priv-fn assigned-to]]
+                                                   (when (= assigned-to pub-fn) priv-fn))
+                                                 assignments)
+                                  all-fn-names (vec (cons pub-fn attached))
+                                  all-mutations (vec (concat (:survived pub-data)
+                                                             (mapcat #(:survived (get coverage-results %))
+                                                                     attached)))
+                                  all-tests (apply set/union
+                                                   (keep #(:tests (get coverage-results %))
+                                                         all-fn-names))]]
+                        {:primary-fn pub-fn
+                         :fn-names all-fn-names
+                         :mutations all-mutations
+                         :tests (or all-tests #{})})
+        private-groups (for [[priv-fn priv-data] unassigned-private]
+                         {:primary-fn priv-fn
+                          :fn-names [priv-fn]
+                          :mutations (vec (:survived priv-data))
+                          :tests (or (:tests priv-data) #{})})]
+    (vec (concat public-groups private-groups))))
+
+(defn build-test-prompt
+  "Build a prompt for Claude to write tests that kill surviving mutations."
+  [{:keys [target-ns test-ns source-path test-path fn-names mutations]}]
+  (let [fn-sources (str/join "\n\n"
+                             (for [fn-name fn-names
+                                   :let [fn-info (coverage/find-function-source fn-name)
+                                         source (when fn-info (coverage/read-function-from-file fn-info))]
+                                   :when source]
+                               (str ";;; " fn-name "\n" source)))
+        mutation-list (str/join "\n\n"
+                                (map-indexed
+                                 (fn [i {:keys [description mutation]}]
+                                   (str "### Mutation " (inc i) ": " description "\n```clojure\n" mutation "\n```"))
+                                 mutations))
+        test-content (slurp test-path)]
+    (str "You are writing mutation tests for functions in `" target-ns "`.\n"
+         "\n"
+         "## Source Code of Functions Under Test\n"
+         "\n"
+         "```clojure\n"
+         fn-sources
+         "\n```\n"
+         "\n"
+         "## Existing Test File (" test-path ")\n"
+         "\n"
+         "```clojure\n"
+         test-content
+         "\n```\n"
+         "\n"
+         "## Surviving Mutations to Kill\n"
+         "\n"
+         "Each mutation below represents a change to the source code that no existing test catches.\n"
+         "Write the simplest tests that would fail if these mutations were applied.\n"
+         "\n"
+         mutation-list
+         "\n"
+         "\n"
+         "## Instructions\n"
+         "\n"
+         "- Write the simplest tests that kill these surviving mutations\n"
+         "- Follow the patterns in the existing test file (same helpers, same assertion style)\n"
+         "- Never call private functions directly — test through public API\n"
+         "- Insert tests near related existing tests, not at the end of the file\n"
+         "- Use the Edit tool to modify the test file at: " test-path "\n"
+         "- After writing tests, verify they compile by running: clj-nrepl-eval -p $(clj-nrepl-eval --discover-ports | head -1) \"(require '" test-ns " :reload)\"\n"
+         "- Each test should verify one meaningful behavior — don't over-engineer\n")))
+
+(defn invoke-claude!
+  "Shell out to claude CLI to write tests. Returns the output string."
+  [prompt]
+  (let [result (shell/sh "claude" "-p"
+                         "--allowedTools" "Edit,Read,Bash(clj-nrepl-eval*)"
+                         :in prompt)]
+    (when-not (zero? (:exit result))
+      (throw (ex-info "Claude invocation failed" {:exit (:exit result)
+                                                  :err (:err result)})))
+    (:out result)))
+
+(defn- all-test-names
+  "Get all test var names from a test namespace."
+  [test-ns-sym]
+  (require test-ns-sym :reload)
+  (let [ns-obj (find-ns test-ns-sym)]
+    (->> (ns-interns ns-obj)
+         vals
+         (filter #(:test (meta %)))
+         (map #(symbol (str test-ns-sym) (name (.sym ^clojure.lang.Var %))))
+         set)))
+
+(defn- count-tests
+  "Count test vars in a namespace."
+  [test-ns-sym]
+  (count (filter #(:test (meta %)) (vals (ns-interns (find-ns test-ns-sym))))))
+
+(defn verify-and-retry!
+  "Verify mutations are killed. Retry with Claude if needed.
+   Returns {:status :success|:partial, :killed [...], :survived [...]}."
+  [{:keys [fn-names test-ns test-path max-retries]
+    :or {max-retries 2}}]
+  (loop [retries-left max-retries]
+    (let [test-names (all-test-names test-ns)
+          fn-results (into {}
+                           (for [fn-name fn-names
+                                 :let [result (coverage/test-mutations fn-name test-names)]
+                                 :when result]
+                             [fn-name result]))
+          all-killed (vec (mapcat :killed (vals fn-results)))
+          all-survived (vec (mapcat :survived (vals fn-results)))]
+      (if (empty? all-survived)
+        {:status :success :killed all-killed :survived []}
+        (if (pos? retries-left)
+          (do
+            (println "  " (count all-survived) "mutations still survive, retrying..."
+                     "(" retries-left "retries left)")
+            (let [retry-prompt (str "Some mutations still survive after your previous tests. "
+                                    "Please write additional tests to kill them.\n\n"
+                                    "## Still-Surviving Mutations\n\n"
+                                    (str/join "\n\n"
+                                              (for [{:keys [description mutation]} all-survived]
+                                                (str "### " description "\n```clojure\n" mutation "\n```")))
+                                    "\n\n## Instructions\n"
+                                    "- The test file is at: " test-path "\n"
+                                    "- Use the Edit tool to add more tests\n"
+                                    "- Focus specifically on the mutations listed above\n"
+                                    "- Do NOT duplicate existing tests\n")]
+              (invoke-claude! retry-prompt))
+            (recur (dec retries-left)))
+          {:status :partial :killed all-killed :survived all-survived})))))
+
+(defn process-group!
+  "Process one group of functions: branch → tests → verify → commit → PR.
+   Returns {:pr-url ... :issue ... :killed ... :not-killed ...}."
+  [session group]
+  (let [{:keys [target-ns test-ns test-path]} session
+        {:keys [primary-fn fn-names mutations]} group
+        primary-fn-name (name primary-fn)]
+    ;; 1. Create branch
+    (println "  Creating branch...")
+    (create-branch! (str target-ns) primary-fn-name)
+
+    ;; 2. Count tests before
+    (require test-ns :reload)
+    (let [tests-before (count-tests test-ns)]
+
+      ;; 3. Build prompt and invoke Claude
+      (println "  Invoking Claude to write tests...")
+      (let [prompt (build-test-prompt {:target-ns target-ns
+                                       :test-ns test-ns
+                                       :source-path (:source-path session)
+                                       :test-path test-path
+                                       :fn-names fn-names
+                                       :mutations mutations})]
+        (invoke-claude! prompt))
+
+      ;; 4. Verify and retry
+      (println "  Verifying mutations killed...")
+      (let [verify-result (verify-and-retry! {:fn-names fn-names
+                                              :test-ns test-ns
+                                              :test-path test-path
+                                              :max-retries 2})
+            killed (:killed verify-result)
+            survived (:survived verify-result)
+            tests-after (count-tests test-ns)
+            tests-added (- tests-after tests-before)]
+
+        ;; 5. Commit and push
+        (println "  Committing and pushing...")
+        (commit-and-push! (str target-ns) primary-fn-name [test-path])
+
+        ;; 6. Create Linear issue
+        (println "  Creating Linear issue...")
+        (let [issue (create-issue-for-function! (str target-ns) primary-fn-name)
+
+              ;; 7. Create draft PR
+              _ (println "  Creating draft PR...")
+              pr-url (create-draft-pr!
+                      {:target-ns (str target-ns)
+                       :fn-name primary-fn-name
+                       :fn-names (mapv name fn-names)
+                       :linear-identifier (:identifier issue)
+                       :mutations-before (count mutations)
+                       :tests-added tests-added
+                       :killed (mapv :description killed)
+                       :not-killed (mapv (fn [m] {:description (:description m)
+                                                  :rationale "Could not be killed automatically"})
+                                         survived)
+                       :suggested-changes []})]
+
+          ;; 8. Return to base branch
+          (return-to-base!)
+
+          (println "  Done! PR:" pr-url)
+          {:pr-url pr-url
+           :issue issue
+           :killed killed
+           :not-killed survived
+           :tests-added tests-added})))))
+
+(defn print-summary!
+  "Print a summary of the mutation testing run."
+  [{:keys [project groups results]}]
+  (println "\n========================================")
+  (println "  Mutation Testing Summary")
+  (println "========================================")
+  (println "Project:" (:name project))
+  (println "Groups:" (count groups))
+  (let [successful (filter :pr-url results)
+        failed (filter :error results)
+        total-killed (count (mapcat :killed successful))
+        total-unkilled (count (mapcat :not-killed successful))
+        total-tests (reduce + 0 (keep :tests-added successful))]
+    (println "PRs created:" (count successful))
+    (when (seq failed)
+      (println "Errors:" (count failed)))
+    (println "Tests added:" total-tests)
+    (println "Mutations killed:" total-killed)
+    (println "Mutations not killed:" total-unkilled)
+    (println)
+    (when (seq successful)
+      (println "PRs:")
+      (doseq [r successful]
+        (println "  " (:pr-url r))))
+    (when (seq failed)
+      (println "\nFailed:")
+      (doseq [r failed]
+        (println "  " (:primary-fn r) "-" (:error r))))
+    (println "========================================")))
+
+(defn run!
+  "Main entry point. Runs the full mutation testing workflow for a namespace.
+
+   Prerequisites:
+   - nREPL running
+   - LINEAR_API_KEY env var set
+   - gh CLI authenticated
+   - team-id configured via (set-config! {:team-id \"...\"})
+
+   Options (optional second arg):
+   - :base-branch — git branch to create feature branches from (default: from config, or \"master\")
+   - :project-id  — existing Linear project ID to use (skips creating a new project)"
+  ([target-ns-sym] (run! target-ns-sym {}))
+  ([target-ns-sym opts]
+   (when-let [bb (:base-branch opts)]
+     (set-config! {:base-branch bb}))
+   (when-let [pid (:project-id opts)]
+     (set-config! {:project-id pid}))
+   (let [parsed (parse-namespace target-ns-sym)
+         {:keys [target-ns test-ns report-path]} parsed]
+     ;; 1. Generate report
+     (println "Generating mutation testing report for" (str target-ns) "...")
+     (coverage/generate-report target-ns [test-ns] report-path)
+     (println "Report written to" report-path)
+
+     ;; 2. Create or reuse Linear project
+     (let [project (if (:project-id @config)
+                     (do (println "Using existing Linear project:" (:project-id @config))
+                         {:project-id (:project-id @config)
+                          :name (str "Existing project " (:project-id @config))})
+                     (do (println "Creating Linear project...")
+                         (let [p (create-project-for-namespace! (str target-ns) report-path)]
+                           (println "Created project:" (:name p))
+                           p)))]
+
+       ;; 3. Run coverage and group functions
+       (println "Running coverage analysis and grouping...")
+       (let [coverage-results (coverage/test-namespace target-ns [test-ns])
+             groups (group-functions coverage-results)]
+         (println "Found" (count groups) "function groups to process")
+
+         ;; 4. Process each group
+         (let [results (vec
+                        (for [group groups]
+                          (do
+                            (println "\n--- Processing:" (:primary-fn group)
+                                     "(" (count (:mutations group)) "surviving mutations) ---")
+                            (try
+                              (process-group! parsed group)
+                              (catch Exception e
+                                (println "ERROR:" (.getMessage e))
+                                (try (return-to-base!) (catch Exception _))
+                                {:error (.getMessage e)
+                                 :primary-fn (:primary-fn group)})))))]
+
+           ;; 5. Print summary
+           (print-summary! {:project project
+                            :groups groups
+                            :results results})
+           results))))))
+
 (comment
   ;; Setup:
   ;; 1. Set LINEAR_API_KEY env var before starting the REPL

docs/api.json

@@ -9408,10 +9408,6 @@
         "description" : "schema for a valid MBQL 4 :value clause",
         "allOf" : [ ]
       },
-      "metabase.lib.metadata.column.column-unique-key" : {
-        "type" : "string",
-        "pattern" : "^column-unique-key-v\\d+\\$.+$"
-      },
       "metabase.lib.metadata.protocols.metadata-provider" : {
         "description" : "Schema for something that satisfies the [[metabase.lib.metadata.protocols/MetadataProvider]] protocol."
       },
@@ -9635,6 +9631,10 @@
           "$ref" : "#/components/schemas/metabase.lib.schema.util.distinct-mbql-clauses"
         } ]
       },
+      "metabase.lib.schema.column-unique-key" : {
+        "type" : "string",
+        "pattern" : "^column-unique-key-v\\d+\\$.+$"
+      },
       "metabase.lib.schema.common.base-type" : {
         "type" : "string"
       },
@@ -14531,7 +14531,7 @@
             "type" : "string"
           },
           "checkpoint-filter-unique-key" : {
-            "$ref" : "#/components/schemas/metabase.lib.metadata.column.column-unique-key"
+            "$ref" : "#/components/schemas/metabase.lib.schema.column-unique-key"
           },
           "type" : {
             "const" : "checkpoint"
@@ -39537,135 +39537,6 @@
         "tags" : [ "/api/pulse" ]
       }
     },
-    "/api/pulse/preview_card/{id}" : {
-      "get" : {
-        "operationId" : "get-api-pulse-preview_card-id",
-        "summary" : "GET /api/pulse/preview_card/{id}",
-        "description" : "Get HTML rendering of a Card with `id`.",
-        "parameters" : [ {
-          "in" : "path",
-          "name" : "id",
-          "required" : true,
-          "schema" : {
-            "type" : "integer",
-            "minimum" : 1
-          },
-          "description" : "value must be an integer greater than zero."
-        } ],
-        "responses" : {
-          "2XX" : {
-            "description" : "Successful response"
-          },
-          "4XX" : {
-            "description" : "Client error response"
-          },
-          "5XX" : {
-            "description" : "Server error response"
-          }
-        },
-        "tags" : [ "/api/pulse" ]
-      }
-    },
-    "/api/pulse/preview_card_info/{id}" : {
-      "get" : {
-        "operationId" : "get-api-pulse-preview_card_info-id",
-        "summary" : "GET /api/pulse/preview_card_info/{id}",
-        "description" : "Get JSON object containing HTML rendering of a Card with `id` and other information.",
-        "parameters" : [ {
-          "in" : "path",
-          "name" : "id",
-          "required" : true,
-          "schema" : {
-            "type" : "integer",
-            "minimum" : 1
-          },
-          "description" : "value must be an integer greater than zero."
-        } ],
-        "responses" : {
-          "2XX" : {
-            "description" : "Successful response"
-          },
-          "4XX" : {
-            "description" : "Client error response"
-          },
-          "5XX" : {
-            "description" : "Server error response"
-          }
-        },
-        "tags" : [ "/api/pulse" ]
-      }
-    },
-    "/api/pulse/preview_card_png/{id}" : {
-      "get" : {
-        "operationId" : "get-api-pulse-preview_card_png-id",
-        "summary" : "GET /api/pulse/preview_card_png/{id}",
-        "description" : "Get PNG rendering of a Card with `id`. Optionally specify `width` as a query parameter.",
-        "parameters" : [ {
-          "in" : "path",
-          "name" : "id",
-          "required" : true,
-          "schema" : {
-            "type" : "integer",
-            "minimum" : 1
-          },
-          "description" : "value must be an integer greater than zero."
-        }, {
-          "in" : "query",
-          "name" : "width",
-          "required" : false,
-          "schema" : {
-            "oneOf" : [ {
-              "description" : "value must be an integer greater than zero.",
-              "type" : "integer",
-              "minimum" : 1
-            }, {
-              "type" : "null"
-            } ]
-          }
-        } ],
-        "responses" : {
-          "2XX" : {
-            "description" : "Successful response"
-          },
-          "4XX" : {
-            "description" : "Client error response"
-          },
-          "5XX" : {
-            "description" : "Server error response"
-          }
-        },
-        "tags" : [ "/api/pulse" ]
-      }
-    },
-    "/api/pulse/preview_dashboard/{id}" : {
-      "get" : {
-        "operationId" : "get-api-pulse-preview_dashboard-id",
-        "summary" : "GET /api/pulse/preview_dashboard/{id}",
-        "description" : "Get HTML rendering of a Dashboard with `id`.\n\n  This endpoint relies on a custom middleware defined in `metabase.channel.render.core/style-tag-nonce-middleware` to\n  allow the style tag to render properly, given our Content Security Policy setup. This middleware is attached to these\n  routes at the bottom of this namespace using `metabase.api.common/define-routes`.",
-        "parameters" : [ {
-          "in" : "path",
-          "name" : "id",
-          "required" : true,
-          "schema" : {
-            "type" : "integer",
-            "minimum" : 1
-          },
-          "description" : "value must be an integer greater than zero."
-        } ],
-        "responses" : {
-          "2XX" : {
-            "description" : "Successful response"
-          },
-          "4XX" : {
-            "description" : "Client error response"
-          },
-          "5XX" : {
-            "description" : "Server error response"
-          }
-        },
-        "tags" : [ "/api/pulse" ]
-      }
-    },
     "/api/pulse/test" : {
       "post" : {
         "operationId" : "post-api-pulse-test",

docs/databases/connections/athena.md

@@ -209,6 +209,12 @@ If Metabase also needs to create tables, you'll need additional AWS Glue permiss
 }
 ```
 
+## Writeable connection
+
+{% include plans-blockquote.html feature="Writeable connection" is_plural=true%}
+
+Set up an additional connection used for write operations. See [Writeable connections](../writeable-connection.md).
+
 ## Model features
 
 There aren't (yet) any model features available for Athena.

docs/databases/connections/bigquery.md

@@ -161,6 +161,12 @@ FROM [my_dataset.my_table]
 
 If you're having trouble with your BigQuery connection, you can check out this [troubleshooting guide](../../troubleshooting-guide/bigquery-drive.md) that covers BigQuery issues, [this one](../../troubleshooting-guide/db-connection.md) on data warehouse connections, or visit [Metabase's discussion forum](https://discourse.metabase.com/search?q=bigquery) to see if someone has encountered and resolved a similar issue.
 
+## Writeable connection
+
+{% include plans-blockquote.html feature="Writeable connection" is_plural=true%}
+
+Set up an additional connection used for write operations. See [Writeable connections](../writeable-connection.md).
+
 ## Model features
 
 There aren't (yet) any model features available for BigQuery.

docs/databases/connections/clickhouse.md

@@ -101,6 +101,12 @@ Turn this option **ON** to scan a sample of values every time Metabase runs a [s
 
 A fingerprinting query examines the first 10,000 rows from each column and uses that data to guesstimate how many unique values each column has, what the minimum and maximum values are for numeric and timestamp columns, and so on. If you leave this option **OFF**, Metabase will only fingerprint your columns once during setup.
 
+## Writeable connection
+
+{% include plans-blockquote.html feature="Writeable connection" is_plural=true%}
+
+Set up an additional connection used for write operations. See [Writeable connections](../writeable-connection.md).
+
 ## Model features
 
 There aren't (yet) any model features for ClickHouse.

docs/databases/connections/databricks.md

@@ -101,6 +101,12 @@ Turn this option **ON** to scan a sample of values every time Metabase runs a [s
 
 A fingerprinting query examines the first 10,000 rows from each column and uses that data to guesstimate how many unique values each column has, what the minimum and maximum values are for numeric and timestamp columns, and so on. If you leave this option **OFF**, Metabase will only fingerprint your columns once during setup.
 
+## Writeable connection
+
+{% include plans-blockquote.html feature="Writeable connection" is_plural=true%}
+
+Set up an additional connection used for write operations. See [Writeable connections](../writeable-connection.md).
+
 ## Model features
 
 There aren't (yet) any model features available for Databricks.

docs/databases/connections/druid.md

@@ -64,6 +64,12 @@ Turn this option **ON** to scan a sample of values every time Metabase runs a [s
 
 A fingerprinting query examines the first 10,000 rows from each column and uses that data to guesstimate how many unique values each column has, what the minimum and maximum values are for numeric and timestamp columns, and so on. If you leave this option **OFF**, Metabase will only fingerprint your columns once during setup.
 
+## Writeable connection
+
+{% include plans-blockquote.html feature="Writeable connection" is_plural=true%}
+
+Set up an additional connection used for write operations. See [Writeable connections](../writeable-connection.md).
+
 ## Model features
 
 There aren't (yet) any model features available for Druid.

docs/databases/connections/mariadb.md

@@ -101,6 +101,12 @@ JSON schema inference doesn't work with MariaDB, due to implementation differenc
 
 If your password contains characters that aren't UTF-8, then you might need to add an additional variable to the connection string `passwordCharacterEncoding=<your_encoding_here>`. This ensures that MariaDB understands the special characters in the password during authentication.
 
+## Writeable connection
+
+{% include plans-blockquote.html feature="Writeable connection" is_plural=true%}
+
+Set up an additional connection used for write operations. See [Writeable connections](../writeable-connection.md).
+
 ## Database routing
 
 With database routing, an admin can build a question once using one database, and the question will run its query against a different database with the same schema depending on who is viewing the question.

docs/databases/connections/mongodb.md

@@ -130,6 +130,12 @@ Instead, Metabase gets a sample of the fields in a collection by scanning a samp
 
 If you're not seeing all of the fields show up for a collection in Metabase, one workaround is to include all possible keys in the first document of the collection, and give those keys null values. That way, Metabase will be able to recognize the correct schema for the entire collection.
 
+## Writeable connection
+
+{% include plans-blockquote.html feature="Writeable connection" is_plural=true%}
+
+Set up an additional connection used for write operations. See [Writeable connections](../writeable-connection.md).
+
 ## Database routing
 
 With database routing, an admin can build a question once using one database, and the question will run its query against a different database with the same schema depending on who is viewing the question.

docs/databases/connections/mysql.md

@@ -187,6 +187,12 @@ mysql:
 
 If your password contains characters that aren't UTF-8, then you might need to add an additional variable to the connection string `passwordCharacterEncoding=<your_encoding_here>`. This ensures that MySQL understands the special characters in the password during authentication.
 
+## Writeable connection
+
+{% include plans-blockquote.html feature="Writeable connection" is_plural=true%}
+
+Set up an additional connection used for write operations. See [Writeable connections](../writeable-connection.md).
+
 ## Model features
 
 Choose whether to enable features related to [Metabase models](../../data-modeling/models.md). These features will often require that the database user account, the one you use to connect to your database, has both read and write privileges.

docs/databases/connections/oracle.md

@@ -131,6 +131,12 @@ For example, if you're running Metabase from a directory called `/app/`, you sho
 
 The process for adding plugins when running via Docker is similar, but you'll need to mount the `plugins` directory. Refer to instructions [here](../../installation-and-operation/running-metabase-on-docker.md#adding-external-dependencies-or-plugins) for more details.
 
+## Writeable connection
+
+{% include plans-blockquote.html feature="Writeable connection" is_plural=true%}
+
+Set up an additional connection used for write operations. See [Writeable connections](../writeable-connection.md).
+
 ## Danger zone
 
 See [Danger zone](../danger-zone.md).

docs/databases/connections/postgresql.md

@@ -186,6 +186,12 @@ Turn this option **ON** to scan a sample of values every time Metabase runs a [s
 
 A fingerprinting query examines the first 10,000 rows from each column and uses that data to guesstimate how many unique values each column has, what the minimum and maximum values are for numeric and timestamp columns, and so on. If you leave this option **OFF**, Metabase will only fingerprint your columns once during setup.
 
+## Writeable connection
+
+{% include plans-blockquote.html feature="Writeable connection" is_plural=true%}
+
+Set up an additional connection used for write operations. See [Writeable connections](../writeable-connection.md).
+
 ## Model features
 
 Choose whether to enable features related to [Metabase models](../../data-modeling/models.md). These features will often require that the database user account, the one you use to connect to your database, has both read and write privileges.

docs/databases/connections/presto.md

@@ -94,6 +94,12 @@ Turn this option **ON** to scan a sample of values every time Metabase runs a [s
 
 A fingerprinting query examines the first 10,000 rows from each column and uses that data to guesstimate how many unique values each column has, what the minimum and maximum values are for numeric and timestamp columns, and so on. If you leave this option **OFF**, Metabase will only fingerprint your columns once during setup.
 
+## Writeable connection
+
+{% include plans-blockquote.html feature="Writeable connection" is_plural=true%}
+
+Set up an additional connection used for write operations. See [Writeable connections](../writeable-connection.md).
+
 ## Database routing
 
 With database routing, an admin can build a question once using one database (data catalog), and the question will run its query against a different data catalog with the same schema depending on who is viewing the question.

docs/databases/connections/redshift.md

@@ -98,6 +98,12 @@ Turn this option **ON** to scan a sample of values every time Metabase runs a [s
 
 A fingerprinting query examines the first 10,000 rows from each column and uses that data to guesstimate how many unique values each column has, what the minimum and maximum values are for numeric and timestamp columns, and so on. If you leave this option **OFF**, Metabase will only fingerprint your columns once during setup.
 
+## Writeable connection
+
+{% include plans-blockquote.html feature="Writeable connection" is_plural=true%}
+
+Set up an additional connection used for write operations. See [Writeable connections](../writeable-connection.md).
+
 ## Model features
 
 Choose whether to enable features related to Metabase models. These will often require a write connection.

docs/databases/connections/snowflake.md

@@ -14,17 +14,17 @@ You can edit these settings at any time. Just remember to save your changes.
 
 Paste a connection string here to pre-fill the remaining fields below.
 
-## Display name
+### Display name
 
 The display name for the database in the Metabase interface.
 
-## Account or hostname
+### Account or hostname
 
 You have the option of using an account name or hostname. Your hostname is your database's IP address or its domain name (e.g., database.example.com).
 
 If your URL includes your database's region, you can connect using an account name, otherwise connect using a hostname.
 
-### AWS
+#### AWS
 
 Connect with an account name.
 
@@ -37,7 +37,7 @@ You'd enter `az12345.ca-central-1.aws` as the account name in Metabase.
 
 Not all regions require the cloud platform identifier. If you're in `us-west-2`, you would enter `az12345` as the account name. For the requirements per region, see [the official Snowflake's documentation](https://docs.snowflake.com/en/user-guide/admin-account-identifier.html#non-vps-account-locator-formats-by-cloud-platform-and-region).
 
-### app.snowflake.com
+#### app.snowflake.com
 
 Connect with a host name.
 
@@ -47,31 +47,31 @@ You can get your host name by going to Admin > Accounts. The Account name will b
 
 Learn more about [Account Identifiers in Snowflake](https://docs.snowflake.com/en/user-guide/admin-account-identifier).
 
-## Username
+### Username
 
 The database username for the account that you want to use to connect to your database. You can set up multiple connections to the same database using different user accounts to connect to the same database, each with different sets of [privileges](../users-roles-privileges.md).
 
 On your app.snowflake.com account page, you can find Users and roles under Admin > Users & Roles.
 
-## Password
+### Password
 
 The password for the username that you use to connect to the database.
 
-## RSA private key (PEM)
+### RSA private key (PEM)
 
 Not required. You have the option of using a **Local file path** or an **Uploaded file path**. If you're on Metabase Cloud, you'll need to select **Uploaded file path** and upload your certificate.
 
-## Warehouse
+### Warehouse
 
 Snowflake warehouse. If the user lacks a default warehouse, you'll need to enter the warehouse to connect to.
 
 On app.snowflake.com, you can find warehouses under Admin > Warehouses.
 
-## Database name (case sensitive)
+### Database name (case sensitive)
 
 The name of the database you want to connect to in Snowflake. On app.snowflake.com you can find databases under Data > Databases.
 
-## Schemas (optional)
+### Schemas (optional)
 
 You can specify which schemas you want to sync and scan. If no schema is passed, then all schema available to that user and role will be listed as folders in Metabase.
 
@@ -96,7 +96,7 @@ Let's say you have three schemas: FOO, BAR, and BAZ.
 
 Note that only the `*` wildcard is supported; you can't use other special characters or regexes.
 
-## Role (optional)
+### Role (optional)
 
 > **WARNING:** Metabase gets _all the permissions_ combined from _every_ role you grant the user in Snowflake. Setting the 'Role' here just adds the role to the connection string to tell Snowflake which role to default to – it **won't restrict** access on its own. So **be careful which roles you assign to the connection user**: if down the line you grant that user broader roles in Snowflake, Metabase will automatically gets those new permissions, potentially showing data you didn't mean for it to see. If instead you want Metabase to issue `SET ROLE` commands before querying anything, check out [connection impersonation](../../permissions/impersonation.md).
 
@@ -107,25 +107,25 @@ Specify a role to override the database user's default role. For example, if the
 
 You can enter `ANALYTICS` in the Role field to ensure that the `METABASE` user connects to Snowflake using the `ANALYTICS` role by default.
 
-## Use an SSH tunnel
+### Use an SSH tunnel
 
 See our [guide to SSH tunneling](../ssh-tunnel.md).
 
-## Additional JDBC connection string options
+### Additional JDBC connection string options
 
 Some databases allow you to append options to the connection string that Metabase uses to connect to your database.
 
-## Re-run queries for simple explorations
+### Re-run queries for simple explorations
 
 Turn this option **OFF** if people want to click **Run** (the play button) before applying any [Summarize](../../questions/query-builder/summarizing-and-grouping.md) or filter selections.
 
 By default, Metabase will execute a query as soon as you choose an grouping option from the **Summarize** menu or a filter condition from the [drill-through menu](https://www.metabase.com/learn/metabase-basics/querying-and-dashboards/questions/drill-through). If your database is slow, you may want to disable re-running to avoid loading data on each click.
 
-## Choose when syncs and scans happen
+### Choose when syncs and scans happen
 
 See [syncs and scans](../sync-scan.md#choose-when-syncs-and-scans-happen).
 
-### Periodically refingerprint tables
+#### Periodically refingerprint tables
 
 > Periodic refingerprinting will increase the load on your database.
 
@@ -133,6 +133,12 @@ Turn this option **ON** to scan a sample of values every time Metabase runs a [s
 
 A fingerprinting query examines the first 10,000 rows from each column and uses that data to guesstimate how many unique values each column has, what the minimum and maximum values are for numeric and timestamp columns, and so on. If you leave this option **OFF**, Metabase will only fingerprint your columns once during setup.
 
+## Writeable connection
+
+{% include plans-blockquote.html feature="Writeable connection" is_plural=true%}
+
+Set up an additional connection used for write operations. See [Writeable connections](../writeable-connection.md).
+
 ## Model features
 
 There aren't (yet) any model features available for Snowflake.

docs/databases/connections/sparksql.md

@@ -77,6 +77,12 @@ Turn this option **ON** to scan a sample of values every time Metabase runs a [s
 
 A fingerprinting query examines the first 10,000 rows from each column and uses that data to guesstimate how many unique values each column has, what the minimum and maximum values are for numeric and timestamp columns, and so on. If you leave this option **OFF**, Metabase will only fingerprint your columns once during setup.
 
+## Writeable connection
+
+{% include plans-blockquote.html feature="Writeable connection" is_plural=true%}
+
+Set up an additional connection used for write operations. See [Writeable connections](../writeable-connection.md).
+
 ## Danger zone
 
 See [Danger zone](../danger-zone.md).

docs/databases/connections/sql-server.md

@@ -97,6 +97,12 @@ A fingerprinting query examines the first 10,000 rows from each column and uses
 
 To connect to Azure SQL, you'll need to set the port to 1433.
 
+## Writeable connection
+
+{% include plans-blockquote.html feature="Writeable connection" is_plural=true%}
+
+Set up an additional connection used for write operations. See [Writeable connections](../writeable-connection.md).
+
 ## Database routing
 
 With database routing, an admin can build a question once using one database, and the question will run its query against a different database with the same schema depending on who is viewing the question.

docs/databases/connections/sqlite.md

@@ -61,6 +61,12 @@ Turn this option **ON** to scan a sample of values every time Metabase runs a [s
 
 A fingerprinting query examines the first 10,000 rows from each column and uses that data to guesstimate how many unique values each column has, what the minimum and maximum values are for numeric and timestamp columns, and so on. If you leave this option **OFF**, Metabase will only fingerprint your columns once during setup.
 
+## Writeable connection
+
+{% include plans-blockquote.html feature="Writeable connection" is_plural=true%}
+
+Set up an additional connection used for write operations. See [Writeable connections](../writeable-connection.md).
+
 ## Database routing
 
 With database routing, an admin can build a question once using one database, and the question will run its query against a different database with the same schema depending on who is viewing the question.

docs/databases/connections/starburst.md

@@ -91,6 +91,12 @@ Turn this option **ON** to scan a sample of values every time Metabase runs a [s
 
 A fingerprinting query examines the first 10,000 rows from each column and uses that data to guesstimate how many unique values each column has, what the minimum and maximum values are for numeric and timestamp columns, and so on. If you leave this option **OFF**, Metabase will only fingerprint your columns once during setup.
 
+## Writeable connection
+
+{% include plans-blockquote.html feature="Writeable connection" is_plural=true%}
+
+Set up an additional connection used for write operations. See [Writeable connections](../writeable-connection.md).
+
 ## Model features
 
 There aren't (yet) any model features for Starburst.

docs/databases/connections/vertica.md

@@ -41,6 +41,12 @@ For example, if you're running Metabase from a directory called `/app/`, you sho
 
 The process for adding plugins when running via Docker is similar, but you'll need to mount the `plugins` directory. Refer to instructions [here](../../installation-and-operation/running-metabase-on-docker.md#adding-external-dependencies-or-plugins) for more details.
 
+## Writeable connection
+
+{% include plans-blockquote.html feature="Writeable connection" is_plural=true%}
+
+Set up an additional connection used for write operations. See [Writeable connections](../writeable-connection.md).
+
 ## Model features
 
 There aren't (yet) any model features for Vertica.

docs/databases/users-roles-privileges.md

@@ -1,21 +1,28 @@
 ---
 title: Database users, roles, and privileges
+summary: Set up database users and roles for Metabase: read-only access for queries, optional write permissions for transforms, uploads, actions, and model persistence.
 ---
 
 # Database users, roles, and privileges
 
-We recommend creating a `metabase` database user with the following database roles:
+## Recommended setup
+
+We recommend creating a dedicated `metabase` database user with the **read-only** access to your database with [minimum database privileges for read access](#minimum-database-privileges) to any schemas or tables used for analysis.
+
+Some Metabase functionality requires write access to the database. Depending on other Metabase features you're planning to use, you might want to create additional roles:
 
-- [`analytics` for read access](#minimum-database-privileges) to any schemas or tables used for analysis.
-- Optional [`metabase_actions` for write access](#privileges-to-enable-actions-and-editable-table-data) to tables used for Metabase actions.
-- Optional [`metabase_model_persistence` for write access](#privileges-to-enable-model-persistence) to the schema used for Metabase model persistence.
 - Optional [`metabase_transforms` for write access](#privileges-to-enable-transforms) to the schema used for Metabase transforms.
+- Optional [`metabase_uploads` for write access](#privileges-to-enable-uploads) to schema used for CSV uploads.
+- Optional [`metabase_actions` for write access](#privileges-to-enable-actions-and-editable-table-data) to tables used for editable tables or Metabase actions .
+- Optional [`metabase_model_persistence` for write access](#privileges-to-enable-model-persistence) to the schema used for Metabase model persistence.
 
 Bundling your privileges into roles based on use cases makes it easier to manage privileges in the future (especially in [multi-tenant situations](#multi-tenant-permissions)). For example, you could:
 
 - Use the same `analytics` role for other BI tools in your [data stack](https://www.metabase.com/learn/grow-your-data-skills/data-fundamentals/data-landscape) that need read-only access to the analytics tables in your database.
 - Revoke the write access for `metabase_model_persistence` without affecting the write access for `metabase_actions`.
 
+On Metabase Pro/Enterprise plans, we recommend using separate [Writeable connections](./writeable-connection.md) for operations that require write access.
+
 ## Minimum database privileges
 
 In order to view and query your tables in Metabase, you'll have to give Metabase's database user:
@@ -85,17 +92,17 @@ CREATE USER metabase WITH PASSWORD "your_password";
 GRANT ALL PRIVILEGES ON "database" TO metabase;
 ```
 
-This is a good option if you're connecting to a local database for development or testing.
+This is a good option if you're connecting to a local database for development or testing. Avoid using this setup in production.
 
 ## Privileges to enable actions and editable table data
 
 Both [actions](../actions/introduction.md) and the [editable table data](../data-modeling/editable-tables.md) let Metabase write back to specific tables in your database.
 
-In addition to the [minimum database privileges](#minimum-database-privileges), you'll need to grant write access to any tables you want to be able to write to.
+if you're using actions or editable tables, then in addition to the [minimum database privileges](#minimum-database-privileges), you'll need to grant write access to any tables you want to be able to write to.
 
 - Create a new role called `metabase_writer`.
 - Give the role `INSERT`, `UPDATE`, and `DELETE` privileges to the relevant tables.
-- Give the `metabase_writer` role to the `metabase` user.
+- Give the `metabase_writer` role to the user used for your database connection. We recommend using a [writeable connection](./writeable-connection.md) and only granting this role to the user used for the writeable connection.
 
 ```sql
 -- Create a role to bundle database privileges for Metabase writing to your database.
@@ -117,7 +124,7 @@ In addition to the [minimum database privileges](#minimum-database-privileges):
 - Create a new role called `metabase_model_persistence`.
 - Give the role `CREATE` access to the database.
 - Give the role `INSERT`, `UPDATE`, and `DELETE` privileges to the schema used for model persistence.
-- Give the `metabase_model_persistence` role to the `metabase` user.
+- Give the `metabase_model_persistence` role to the user used for your database connection. We recommend using a [writeable connection](./writeable-connection.md) and only granting this role to the user used for the writeable connection.
 
 ```sql
 -- Create a role to bundle database privileges for Metabase model persistence.
@@ -137,20 +144,20 @@ GRANT metabase_model_persistence TO metabase;
 
 ## Privileges to enable transforms
 
-[Transforms](../data-studio/transforms/transforms-overview.md) let Metabase write query results back to your database. We suggest that you create a dedicated schema for your transforms. Metabase's database user will need to be able to create and drop transform tables.
+[Transforms](../data-studio/transforms/transforms-overview.md) let Metabase write query results back to your database. We suggest that you create a dedicated schema for your transforms. Metabase's database user will need to be able to create and drop transform tables. We recommend using a [writeable connection](./writeable-connection.md) and only granting write access to the user used for the writeable connection.
 
 In addition to the [minimum database privileges](#minimum-database-privileges):
 
 - Create a new role called `metabase_transforms`.
 - Create a dedicated schema for transforms.
 - Give the `metabase_transforms` role `CREATE TABLE` / `ALTER` / `DROP` privileges for the specified schema.
-- Give the `metabase_transforms` role to the `metabase` user.
+- Give the `metabase_transforms` role to the user used for your database connection. We recommend using a [writeable connection](./writeable-connection.md) and only granting this role to the user used for the writeable connection.
 
 In certain cases, Metabase will give you an option to create new schemas for transforms, in which case you'll also need `CREATE SCHEMA` privileges for your metabase users (or `CREATE DATABASE` for ClickHouse).
 
 ## Privileges to enable uploads
 
-You can [upload CSVs](../databases/uploads.md) to supported databases. Metabase's database user should have write access (`INSERT`, `UPDATE`, `DELETE`) to the schema where you want to store the uploads.
+You can [upload CSVs](../databases/uploads.md) to supported databases. Metabase's database user should have write access (`INSERT`, `UPDATE`, `DELETE`) to the schema where you want to store the uploads. We recommend using a [writeable connection](./writeable-connection.md) and only granting write access to the user used for the writeable connection.
 
 You'll first need to create a schema to store uploads (or use an existing schema) and tell Metabase that you want to [use that schema to store uploads](./uploads.md#select-the-database-and-schema-that-you-want-to-store-the-data-in).
 
@@ -158,7 +165,7 @@ In addition to the [minimum database privileges](#minimum-database-privileges):
 
 - Create a new role called `metabase_uploads`.
 - Give the role `INSERT`, `UPDATE`, and `DELETE` privileges to the schema where you want to store uploads.
-- Give the `metabase_uploads` role to the `metabase` user.
+- Give the `metabase_uploads` role to the user used for your database connection.
 
 ```sql
 -- Create a role to bundle database privileges for uploads.
@@ -221,6 +228,7 @@ We recommend bundling privileges into roles based on use cases per customer. Tha
 
 ## Further reading
 
+- [Writeable connections](./writeable-connection.md)
 - [Permissions strategies](https://www.metabase.com/learn/metabase-basics/administration/permissions/strategy)
 - [Permissions introduction](../permissions/introduction.md)
 - [People overview](../people-and-groups/start.md)

docs/databases/writeable-connection.md

@@ -0,0 +1,60 @@
+---
+title: Writeable connections
+summary: Add an optional separate database connection for write operations like transforms or editable tables to keep the main Metabase connection read-only for analytics.
+---
+
+## Writeable connections
+
+{% include plans-blockquote.html feature="Writeable connections" is_plural=true%}
+
+_Admin > Databases > Writeable connection_
+
+Writeable connection is an optional connection that Metabase will use for functionality that involves writing into the database (for example, transforms or CSV uploads). This allows you to keep your main Metabase connection read-only, and only to switch to connection with write permissions for the operations that require write access.
+
+![Writeable connection in database settings](./images/writeable-connection.png)
+
+## How writeable connection works
+
+Writeable connection is an additional connection option added to an existing database connection.
+
+When a writeable connection is configured for a database, Metabase automatically switches between connections behind the scenes based on what people are doing with the database. The main connection is used for querying, and the writeable connection is used for [features that require writes](#features-that-use-writeable-connection).
+
+Writeable connection can use a different database user than the main connection, or even connect to a different database. For example, your main connection (used for querying) can go to a read replica, but the writeable connection can be to a primary database.
+
+## Add a writeable connection
+
+To add a writeable connection to a database:
+
+1. Go to **Admin > Databases**.
+2. Select the database to add a writeable connection to.
+3. Under **Writeable connection**, click **Add writeable connection**.
+4. Metabase will pre-fill some fields with information from your main connection. Input the credentials for the writeable connection.
+5. Click **Save**. Metabase will try to connect using the new connection.
+
+## Remove a writeable connection
+
+To remove a writeable connection from a database:
+
+1. Go to **Admin > Databases**.
+2. Select the database you want to remove the writeable connection from.
+3. Under **Writeable connection**, click **Remove writeable connection**.
+
+Once writeable connection is removed, Metabase will revert to using the main connection for [features that need write access](#features-that-use-writeable-connection), so your main connection will need to have appropriate database permissions, see [Users, roles, and privileges](./users-roles-privileges.md)
+
+## Features that use writeable connection
+
+When configured, writeable connection will be used for:
+
+- [Transforms](../data-studio/transforms/transforms-overview.md)
+- [CSV uploads](../databases/uploads.md)
+- [Editable tables](../data-modeling/editable-tables.md)
+- [Actions](../actions/introduction.md)
+- [Model persistence](../data-modeling/model-persistence.md)
+
+Your writeable connection will need to have the privileges required for these operations. See [Users, roles, and privileges](./users-roles-privileges.md).
+
+## Limitations
+
+- Writeable connection can't be used with [database routing](../permissions/database-routing.md).
+
+- Writeable connection must use the same database engine as the primary connection - so for example, if the primary connection is to a PostgreSQL database, the writeable connection must be to a PostgreSQL database as well.

docs/embedding/tenants.md

@@ -192,16 +192,18 @@ You can create tenant-level [user attributes](#tenant-attributes) which all user
 
 ![Edit tenant](./images/edit-tenant.png)
 
-To add a tenant attribute:
+There are three ways to add a tenant attribute, using the Metabase UI, the API, or JWT SSO.
+
+To create a tenant attribute manually using the Metabase UI:
 
 1. Go to **Admin settings > People**
 2. Select **Tenants** on the left sidebar.
 3. Click on **three dots** next to the tenant.
 4. Input the attribute key and value.
 
-Once you add a tenant attribute, all users of that tenant will inherit the attribute, but the value can be overridden for any particular user, see [Edit user attributes](../people-and-groups/managing.md#adding-a-user-attribute).
+For details on how to set tenant attributes using JWT tenant claims, see [Setting tenant attributes using tenant claims](#setting-tenant-attributes-using-tenant-claims) below.
 
-Currently, you can't assign custom tenant attributes with SSO. The only way to assign attributes is through the Metabase UI (but you can provision attributes for _individual users_ through SSO, see [JWT user attributes](../people-and-groups/authenticating-with-jwt.md)). However, if you're using user attribute to set permissions, then you can use the [special slug attribute](#special-tenant-slug-attribute) which Metabase creates automatically.
+Once you add a tenant attribute, all users of that tenant will inherit the attribute, but the value can be overridden for any particular user, see [Edit user attributes](../people-and-groups/managing.md#adding-a-user-attribute).
 
 ### Special tenant slug attribute
 
@@ -264,6 +266,24 @@ When user provisioning with JWT is enabled:
 2. If the tenant doesn't exist, Metabase automatically creates it. Metabase will use the value of the `@tenant` key (or your chosen assignment attribute) as the tenant slug.
 3. New users are automatically assigned to the tenant from their JWT.
 
+### Setting tenant attributes using tenant claims
+
+To create tenant attributes from JWT SSO, include a claim `@tenant.attributes`:
+
+```json
+{
+  "@tenant": "meowdern_solutions",
+  "@tenant.attributes": {
+    "industry": "cat food"
+  },
+  "email": "mittens@example.com",
+  "first_name": "Mister",
+  "last_name": "Mittens"
+}
+```
+
+If a tenant attribute with this name doesn't exist, Metabase will create the attribute and assign the value from the JWT claim. However, if the tenant attribute already exists, Metabase will **not** update the value.
+
 ### Troubleshooting JWT authentication with tenants
 
 Some common auth error messages and what they mean:
@@ -329,8 +349,7 @@ Permissions are granted to groups. Which permissions are available to each group
 ### Internal user collection permissions
 
 - Metabase Admins will have **Curate** access to all shared collections and all tenant collections.
-- Other internal groups can be granted **View** or **Curate** access to **shared collections**, see [Configuring shared collection permissions](#configuring-shared-collections-permissions).
-- Non-admin internal users will have **No** access to tenant-specific collections. Currently, this can't be configured.
+- Other internal groups and non-admin users have **No** access by default, but can be granted **View** or **Curate** access to **shared collections**. See [Configuring shared collection permissions](#configuring-shared-collections-permissions) for details.
 
 For configuring permissions to _internal_ collections for internal users, see [general docs on collection permissions](../permissions/collections.md).
 

docs/permissions/database-routing.md

@@ -21,13 +21,32 @@ Database routing is useful for:
 
 ## Database routing limitations
 
-> Database routing is **not supported** on ClickHouse, Oracle, Spark SQL, and Vertica.
+### Driver limitations
+
+Database routing is **not supported** on:
+
+- ClickHouse
+- Oracle
+- Spark SQL
+- Vertica.
 
 Different database have different setups, so _what_ you can route between (database, schema, data catalog, etc.) will differ slightly depending on which data warehouse you're using.
 
-- [Athena](../databases/connections/athena.md): Only routing between different connections is supported (e.g., different buckets, roles, or catalogs).
-- [BigQuery](../databases/connections/bigquery.md): Only routing between databases in different projects is supported.
-- [Databricks](../databases/connections/databricks.md): When multi-catalog is not enabled, you can route between catalogs on the same host. If multi-catalog is enabled, then you can only route between databases on separate hosts.
+- Athena: Only routing between different connections is supported (e.g., different buckets, roles, or catalogs).
+- BigQuery: Only routing between databases in different projects is supported.
+- Databricks: When multi-catalog is not enabled, you can route between catalogs on the same host. If multi-catalog is enabled, then you can only route between databases on separate hosts.
+
+### Functionality limitations
+
+Database routing **can't be used** on databases with:
+
+- [Writeable connections](../databases/writeable-connection.md)
+- [Editable tables](../data-modeling/editable-tables.md)
+- [Actions](../actions/introduction.md)
+- [CSV uploads](../databases/uploads.md)
+- [Model persistence](../data-modeling/model-persistence.md)
+
+With **guest embedding**, database queries will always be routed to the router database. This is because guest embed users don't have Metabase accounts, so there are no user attributes available to determine which destination database to route to.
 
 ## How database routing works
 

e2e/snapshot-creators/default.cy.snap.js

@@ -38,7 +38,7 @@ const {
 } = USER_GROUPS;
 const { admin } = USERS;
 
-const { IS_ENTERPRISE } = Cypress.env();
+const IS_ENTERPRISE = Cypress.expose("IS_ENTERPRISE");
 
 describe("snapshots", () => {
   describe("default", () => {

e2e/support/config.js

@@ -14,7 +14,6 @@ import {
   removeDirectory,
   verifyDownloadTasks,
 } from "./commands/downloads/downloadUtils";
-import webpackConfig from "./component-webpack.config";
 import * as dbTasks from "./db_tasks";
 import { signJwt } from "./helpers/e2e-jwt-tasks";
 
@@ -47,6 +46,25 @@ const assetsResolverPlugin = {
 };
 
 const defaultConfig = {
+  // Expose non-sensitive environment variables synchronously via Cypress.expose()
+  // These are safe to expose in the browser and are used for configuration
+  expose: {
+    CI: isCI,
+    IS_ENTERPRISE: isEnterprise,
+    MB_EDITION: process.env["MB_EDITION"],
+    ENABLE_NETWORK_THROTTLING: !!process.env["ENABLE_NETWORK_THROTTLING"],
+    SNOWPLOW_MICRO_URL: snowplowMicroUrl,
+    CLIENT_PORT: process.env["CLIENT_PORT"],
+    feHealthcheck: process.env["FE_HEALTHCHECK_URL"]
+      ? { enabled: true, url: process.env["FE_HEALTHCHECK_URL"] }
+      : undefined,
+  },
+
+  // Note: We can't set `allowCypressEnv: false` yet because @cypress/grep
+  // plugin still uses Cypress.env() internally
+  // FIXME: enable once we upgrade (DEV-1620)
+  // allowCypressEnv: false,
+
   // This is the functionality of the old cypress-plugins.js file
   setupNodeEvents(cypressOn, config) {
     // `on` is used to hook into various events Cypress emits
@@ -130,9 +148,6 @@ const defaultConfig = {
     config.env.grepFilterSpecs = true;
     config.env.grepOmitFiltered = true;
 
-    config.env.IS_ENTERPRISE = isEnterprise;
-    config.env.SNOWPLOW_MICRO_URL = snowplowMicroUrl;
-
     require("@cypress/grep/src/plugin")(config);
 
     if (isCI) {
@@ -156,9 +171,6 @@ const defaultConfig = {
   },
   baseUrl: `http://${BACKEND_HOST}:${BACKEND_PORT}`,
   defaultBrowser: process.env.CYPRESS_BROWSER ?? "chrome",
-  env: {
-    CI: isCI,
-  },
   supportFile: "e2e/support/cypress.js",
   chromeWebSecurity: false,
   modifyObstructiveCode: false,
@@ -201,34 +213,12 @@ const mainConfig = {
     },
   },
   retries: {
-    runMode: 1,
+    runMode: 2,
     openMode: 0,
   },
 };
 
-const embeddingSdkComponentTestConfig = {
-  ...defaultConfig,
-  baseUrl: undefined, // baseUrl should not be set for component tests,
-  defaultCommandTimeout: 10000,
-  requestTimeout: 10000,
-  video: false,
-  specPattern: "e2e/test-component/scenarios/embedding-sdk/**/*.cy.spec.tsx",
-  indexHtmlFile: "e2e/support/component-index.html",
-  supportFile: "e2e/support/component-cypress.js",
-
-  reporter: mainConfig.reporter,
-  reporterOptions: mainConfig.reporterOptions,
-  retries: mainConfig.retries,
-
-  devServer: {
-    framework: "react",
-    bundler: "webpack",
-    webpackConfig: webpackConfig,
-  },
-};
-
 module.exports = {
   defaultConfig,
   mainConfig,
-  embeddingSdkComponentTestConfig,
 };

e2e/support/cypress-embedding-sdk-component-test.config.js

@@ -1,4 +1,26 @@
-const { embeddingSdkComponentTestConfig } = require("./config");
+import webpackConfig from "./component-webpack.config";
+import { defaultConfig, mainConfig } from "./config";
+
+const embeddingSdkComponentTestConfig = {
+  ...defaultConfig,
+  baseUrl: undefined, // baseUrl should not be set for component tests,
+  defaultCommandTimeout: 10000,
+  requestTimeout: 10000,
+  video: false,
+  specPattern: "e2e/test-component/scenarios/embedding-sdk/**/*.cy.spec.tsx",
+  indexHtmlFile: "e2e/support/component-index.html",
+  supportFile: "e2e/support/component-cypress.js",
+
+  reporter: mainConfig.reporter,
+  reporterOptions: mainConfig.reporterOptions,
+  retries: mainConfig.retries,
+
+  devServer: {
+    framework: "react",
+    bundler: "webpack",
+    webpackConfig: webpackConfig,
+  },
+};
 
 module.exports = {
   component: {

e2e/support/cypress.js

@@ -8,8 +8,8 @@ import "cypress-real-events/support";
 import addContext from "mochawesome/addContext";
 import "./commands";
 
-const isCI = Cypress.env("CI");
-const isNetworkThrottlingEnabled = Cypress.env("ENABLE_NETWORK_THROTTLING");
+const isCI = Cypress.expose("CI");
+const isNetworkThrottlingEnabled = Cypress.expose("ENABLE_NETWORK_THROTTLING");
 
 // remove default html output on test failure
 configure({
@@ -31,7 +31,7 @@ Cypress.on("uncaught:exception", (err, runnable) => false);
 
 Cypress.on("test:before:run", () => {
   // Check wether FE is running in dev mode
-  const feHealthcheck = Cypress.env().feHealthcheck;
+  const feHealthcheck = Cypress.expose("feHealthcheck");
   if (feHealthcheck?.enabled) {
     fetch(feHealthcheck.url).catch(() =>
       alert(
@@ -160,7 +160,7 @@ if (isCI) {
 beforeEach(function () {
   const isCurrentTesOss =
     this.currentTest._testConfig.unverifiedTestConfig.tags === "@OSS";
-  const isBuildOss = Cypress.env("MB_EDITION") === "oss";
+  const isBuildOss = Cypress.expose("MB_EDITION") === "oss";
   const testName = this.currentTest.title;
   if (Cypress.config("isInteractive") && isCurrentTesOss && !isBuildOss) {
     console.log(

e2e/support/helpers/e2e-embedding-iframe-sdk-helpers.ts

@@ -15,7 +15,7 @@ import {
   mockAuthProviderAndJwtSignIn,
 } from "./embedding-sdk-testing";
 
-const { IS_ENTERPRISE } = Cypress.env();
+const IS_ENTERPRISE = Cypress.expose("IS_ENTERPRISE");
 
 const EMBED_JS_PATH = "http://localhost:4000/app/embed.js";
 
@@ -373,7 +373,7 @@ export const visitCustomHtmlPage = (
  * to point it to the rspack dev server.
  */
 export const mockEmbedJsToDevServer = () => {
-  if (Cypress.env("CI")) {
+  if (Cypress.expose("CI")) {
     // we don't need this logic in CI, let's skip the check to avoid slowing down the tests
     return;
   }

e2e/support/helpers/e2e-token-helpers.ts

@@ -6,7 +6,7 @@ import { match } from "ts-pattern";
  *
  * `IS_ENTERPRISE` means enterprise instance without a token and `isOSS` means open-source instance.
  */
-const { IS_ENTERPRISE } = Cypress.env();
+const IS_ENTERPRISE = Cypress.expose("IS_ENTERPRISE");
 
 const throwIfNotEnterprise = () => {
   if (!IS_ENTERPRISE) {
@@ -28,29 +28,25 @@ export const activateToken = (
     .with("pro-self-hosted", () => "MB_PRO_SELF_HOSTED_TOKEN")
     .exhaustive();
 
-  const fallbackToken =
-    tokenReference === "starter" ? "NO_FEATURES_TOKEN" : "ALL_FEATURES_TOKEN";
-  const token = Cypress.env(tokenReference) || Cypress.env(fallbackToken);
+  // Use cy.env() for sensitive token values (async API)
+  return cy.env([tokenReference]).then((envVars) => {
+    const token = envVars[tokenReference];
 
-  if (!token) {
-    const errorMessage = `
-    Please make sure you have correctly set the following tokens in your environment variables:
-      - CYPRESS_MB_ALL_FEATURES_TOKEN
-      - CYPRESS_MB_STARTER_CLOUD_TOKEN
-      - CYPRESS_MB_PRO_CLOUD_TOKEN
-      - CYPRESS_MB_PRO_SELF_HOSTED_TOKEN
-    `;
-    throw new Error(errorMessage);
-  }
+    if (!token) {
+      throw new Error(
+        `Missing CYPRESS_${tokenReference} environment variable for "${tokenName}" token`,
+      );
+    }
 
-  cy.log(`Set the "${tokenReference}" token`);
-  return cy.request({
-    method: "PUT",
-    url: "/api/setting/premium-embedding-token",
-    failOnStatusCode: false,
-    body: {
-      value: token,
-    },
+    cy.log(`Set the "${tokenName}" token`);
+    return cy.request({
+      method: "PUT",
+      url: "/api/setting/premium-embedding-token",
+      failOnStatusCode: false,
+      body: {
+        value: token,
+      },
+    });
   });
 };
 

e2e/test-host-app/shared/compatibility.cy.spec.js

@@ -5,7 +5,7 @@ import {
 
 const TIMEOUT_MS = 40000;
 
-const CLIENT_PORT = Cypress.env("CLIENT_PORT");
+const CLIENT_PORT = Cypress.expose("CLIENT_PORT");
 const CLIENT_HOST = `http://localhost:${CLIENT_PORT}`;
 
 describe("Embedding SDK: shared Host Apps compatibility tests", () => {

e2e/test/scenarios/admin/databases.cy.spec.js

@@ -12,7 +12,7 @@ import { ORDERS_QUESTION_ID } from "e2e/support/cypress_sample_instance_data";
 import { visitDatabase, waitForDbSync } from "./helpers/e2e-database-helpers";
 
 const { H } = cy;
-const { IS_ENTERPRISE } = Cypress.env();
+const IS_ENTERPRISE = Cypress.expose("IS_ENTERPRISE");
 const { ORDERS_ID, ORDERS } = SAMPLE_DATABASE;
 
 describe(

e2e/test/scenarios/data-studio/data-studio-library.cy.spec.ts

@@ -19,7 +19,23 @@ describe("scenarios > data studio > library", () => {
     cy.intercept("GET", "/api/collection/tree*").as("getCollectionTree");
 
     cy.log("Navigate to Data Studio Library");
-    H.DataModel.visitDataStudio();
+    /**
+     * Let's use the profile menu to navigate to Data Studio in this test, just
+     * to make sure it works, and to test the analytics event
+     */
+    cy.visit("/");
+    H.getProfileLink().click();
+    H.popover()
+      .findByText(/Data studio/)
+      .click();
+
+    cy.log(
+      "Verify tracking event when opening Data Studio from the profile menu",
+    );
+    H.expectUnstructuredSnowplowEvent({
+      event: "data_studio_opened",
+      triggered_from: "nav_menu",
+    });
     H.DataStudio.nav().findByLabelText("Library").click();
 
     cy.log("Create library via inline empty state");

e2e/test/scenarios/data-studio/table-collection-permissions.cy.spec.ts

@@ -481,7 +481,7 @@ describe("scenarios > data studio > table collection permissions", () => {
 
       cy.signIn("nodata");
       H.visitQuestionAdhoc(productsQuestionDetails);
-      assertPermissionError();
+      assertQueryPermissionError();
     });
 
     it("should not be able to create questions when all published tables are unpublished", () => {
@@ -509,7 +509,7 @@ describe("scenarios > data studio > table collection permissions", () => {
 
       cy.signIn("nodata");
       H.visitQuestionAdhoc(productsQuestionDetails);
-      assertPermissionError();
+      assertQueryPermissionError();
     });
 
     it("should not be able to create questions even if there are published tables", () => {
@@ -544,12 +544,6 @@ function librarySidebarSection() {
   return H.navigationSidebar().findByRole("section", { name: "Library" });
 }
 
-function assertPermissionError() {
-  H.main()
-    .findByText("Sorry, you don’t have permission to see that.")
-    .should("be.visible");
-}
-
 function assertQueryPermissionError() {
   H.main()
     .findByText("Sorry, you don't have permission to run this query.")

e2e/test/scenarios/embedding/sdk-iframe-embedding/guest-embed.cy.spec.ts

@@ -4,7 +4,7 @@ import { createQuestion, getSignedJwtForResource } from "e2e/support/helpers";
 const { H } = cy;
 const { ORDERS, ORDERS_ID } = SAMPLE_DATABASE;
 
-const { IS_ENTERPRISE } = Cypress.env();
+const IS_ENTERPRISE = Cypress.expose("IS_ENTERPRISE");
 const IS_OSS = !IS_ENTERPRISE;
 const MB_EDITION = IS_ENTERPRISE ? "ee" : "oss";
 

e2e/test/scenarios/onboarding/setup/setup.cy.spec.ts

@@ -1,5 +1,5 @@
 const { H } = cy;
-const { IS_ENTERPRISE } = Cypress.env();
+const IS_ENTERPRISE = Cypress.expose("IS_ENTERPRISE");
 import { USERS } from "e2e/support/cypress_data";
 import { SUBSCRIBE_URL } from "metabase/setup/constants";
 
@@ -573,7 +573,10 @@ describe("scenarios > setup (EE)", () => {
 
       cy.findByText("Activate your commercial license").should("exist");
 
-      typeToken(Cypress.env("MB_STARTER_CLOUD_TOKEN"));
+      // Use cy.env() for sensitive token values (async API)
+      cy.env(["MB_STARTER_CLOUD_TOKEN"]).then(({ MB_STARTER_CLOUD_TOKEN }) => {
+        typeToken(MB_STARTER_CLOUD_TOKEN);
+      });
 
       cy.button("Activate").click();
 

e2e/test/scenarios/permissions/sandboxing/helpers/e2e-sandboxing-helpers.ts

@@ -551,7 +551,11 @@ export const getCardResponses = (questions: SimpleCollectionItem[]) => {
   expect(questions.length).to.be.greaterThan(0);
   return H.cypressWaitAll(
     questions.map((question) =>
-      cy.request<DatasetResponse>("POST", `/api/card/${question.id}/query`),
+      cy.request<DatasetResponse>({
+        method: "POST",
+        url: `/api/card/${question.id}/query`,
+        failOnStatusCode: false,
+      }),
     ),
   ).then((responses) => {
     return { responses, questions };

e2e/test/scenarios/visualizations-charts/visualizations-charts-reproductions.cy.spec.ts

@@ -850,6 +850,68 @@ describe("UXW-2696", () => {
   });
 });
 
+describe("issue 69882", () => {
+  const DOOHICKEY_BAR_COLOR = "#227FD2";
+  const GADGET_LINE_COLOR = "#689636";
+
+  const questionDetails: StructuredQuestionDetails = {
+    name: "69882",
+    display: "combo" as const,
+    query: {
+      "source-table": ORDERS_ID,
+      aggregation: [["count"]],
+      breakout: [
+        ["field", PRODUCTS.CATEGORY, { "source-field": ORDERS.PRODUCT_ID }],
+        ["field", ORDERS.CREATED_AT, { "temporal-unit": "year" }],
+      ],
+    },
+    visualization_settings: {
+      series_settings: {
+        Doohickey: {
+          color: DOOHICKEY_BAR_COLOR,
+          title: "_Doohickey",
+          display: "bar",
+        },
+        Gadget: {
+          color: GADGET_LINE_COLOR,
+          title: "_Gadget",
+          display: "line",
+        },
+      },
+      "graph.x_axis.scale": "timeseries",
+      "graph.dimensions": ["CREATED_AT", "CATEGORY"],
+      "graph.metrics": ["count"],
+    },
+  };
+
+  beforeEach(() => {
+    H.restore();
+    cy.signInAsAdmin();
+  });
+
+  it("should apply per-series display type on a question and dashcard (metabase#69882)", () => {
+    H.createQuestion(questionDetails, { visitQuestion: true, wrapId: true });
+
+    H.echartsContainer().should("be.visible");
+    H.chartPathWithFillColor(DOOHICKEY_BAR_COLOR).should("exist");
+    H.cartesianChartCircleWithColor(GADGET_LINE_COLOR);
+
+    H.createDashboard({ name: "Test Dashboard" }, { wrapId: true });
+    cy.get<number>("@questionId").then((cardId) => {
+      cy.get<number>("@dashboardId").then((dashboardId) => {
+        H.addQuestionToDashboard({ dashboardId, cardId });
+        H.visitDashboard(dashboardId);
+      });
+    });
+
+    cy.findByTestId("dashcard").within(() => {
+      H.echartsContainer().should("be.visible");
+      H.chartPathWithFillColor(DOOHICKEY_BAR_COLOR).should("exist");
+      H.cartesianChartCircleWithColor(GADGET_LINE_COLOR);
+    });
+  });
+});
+
 describe("issue #68819", () => {
   beforeEach(() => {
     H.restore();

e2e/test/scenarios/visualizations-tabular/drillthroughs/column_extract_drill.cy.spec.js

@@ -92,7 +92,7 @@ const DATE_QUESTION = {
   },
 };
 
-describe("extract action", () => {
+describe("extract action", { viewportWidth: 1600 }, () => {
   beforeEach(() => {
     H.restore();
     cy.signInAsAdmin();
@@ -256,7 +256,7 @@ describe("extract action", () => {
     });
 
     EMAIL_CASES.forEach(({ option, value, example }) => {
-      it(option, { viewportWidth: 1600 }, () => {
+      it(option, () => {
         H.openPeopleTable({ limit: 1 });
         extractColumnAndCheck({
           column: "Email",
@@ -395,7 +395,7 @@ function extractColumnAndCheck({
   }
 }
 
-describe("extract action", () => {
+describe("extract action", { viewportWidth: 1600 }, () => {
   beforeEach(() => {
     H.restore();
     H.resetSnowplow();

enterprise/backend/src/metabase_enterprise/dependencies/api.clj

@@ -67,7 +67,7 @@
                                           (update :dashboard #(some-> % (select-keys [:id :name])))
                                           (update :document #(some-> % (select-keys [:id :name])))))))
                       broken-cards)
-     :bad_transforms (into [] broken-transforms)}))
+     :bad_transforms (into [] (filter mi/can-read?) broken-transforms)}))
 
 (api.macros/defendpoint :post "/check-card" :- ::broken-cards-response
   "Check a proposed edit to a card, and return the card IDs for those cards this edit will break."

enterprise/backend/src/metabase_enterprise/transforms_python/execute.clj

@@ -120,28 +120,6 @@
     (reify Closeable
       (close [_] (cleanup fut)))))
 
-(defn- maybe-retry-with-backoff
-  [driver thunk]
-  ;; bigquery can have eventual consistency hiccups where the table is not yet available for insertion
-  ;; immediately after creation, so we retry a few times with backoff
-  (if (= driver :bigquery-cloud-sdk)
-    (loop [attempt 0]
-      (let [ret (try
-                  (thunk)
-                  (catch Exception e
-                    (if (and (< attempt 5)
-                             (or (str/includes? (str e) "not found")
-                                 (str/includes? (str e) "Not found")))
-                      (let [delay-ms (* 150 (Math/pow 2 attempt))]
-                        (log/debugf "BigQuery operation failed (attempt %d), retrying in %dms" (inc attempt) delay-ms)
-                        (Thread/sleep (long delay-ms))
-                        ::recur)
-                      (throw e))))]
-        (if (= ::recur ret)
-          (recur (inc attempt))
-          ret)))
-    (thunk)))
-
 (defn- table-schema [table-name metadata]
   {:name (if (keyword? table-name) table-name (keyword table-name))
    :columns (mapv (fn [{:keys [name base_type #_database_type]}]
@@ -155,8 +133,7 @@
   "Insert data from source into an existing table."
   [driver db-id table-schema data-source]
   (let [data-source (assoc data-source :table-schema table-schema)]
-    (->> #(driver/insert-from-source! driver db-id table-schema data-source)
-         (maybe-retry-with-backoff driver))))
+    (driver/insert-from-source! driver db-id table-schema data-source)))
 
 (defn- create-table-and-insert-data!
   "Create a table from metadata and insert data from source."

enterprise/backend/test/metabase_enterprise/advanced_permissions/query_processor/middleware/permissions_test.clj

@@ -282,15 +282,16 @@
       (with-download-perms-for-db! (mt/id) :none
         (streaming-test/do-test!
          "A user with no download perms for a DB receives an error response"
-         {:query      {:database (mt/id)
-                       :type     :query
-                       :query    {:source-table (mt/id :venues)
-                                  :limit        10}}
-          :endpoints  [:card :dataset]
-          :assertions {:csv (fn [results]
-                              (is (partial=
-                                   {:error "You do not have permissions to download the results of this query."}
-                                   results)))}})
+         {:query           {:database (mt/id)
+                            :type     :query
+                            :query    {:source-table (mt/id :venues)
+                                       :limit        10}}
+          :expected-status 403
+          :endpoints       [:card :dataset]
+          :assertions      {:csv (fn [results]
+                                   (is (partial=
+                                        {:error "You do not have permissions to download the results of this query."}
+                                        results)))}})
 
         (streaming-test/do-test!
          "An admin can always run download queries, even if the All Users group has no download permissions "
@@ -305,15 +306,16 @@
       (with-download-perms! (mt/id) {:schemas {"PUBLIC" :none}}
         (streaming-test/do-test!
          "A user with no download perms for a schema receives an error response for download queries on that schema"
-         {:query      {:database (mt/id)
-                       :type     :query
-                       :query    {:source-table (mt/id :venues)
-                                  :limit        10}}
-          :endpoints  [:card :dataset]
-          :assertions {:csv (fn [results]
-                              (is (partial=
-                                   {:error "You do not have permissions to download the results of this query."}
-                                   results)))}}))
+         {:query           {:database (mt/id)
+                            :type     :query
+                            :query    {:source-table (mt/id :venues)
+                                       :limit        10}}
+          :expected-status 403
+          :endpoints       [:card :dataset]
+          :assertions      {:csv (fn [results]
+                                   (is (partial=
+                                        {:error "You do not have permissions to download the results of this query."}
+                                        results)))}}))
 
       (with-download-perms! (mt/id) {:schemas {"PUBLIC" {(mt/id :venues)     :none
                                                          (mt/id :checkins)   :full
@@ -321,15 +323,16 @@
                                                          (mt/id :categories) :full}}}
         (streaming-test/do-test!
          "A user with no download perms for a table receives an error response for download queries on that table"
-         {:query      {:database (mt/id)
-                       :type     :query
-                       :query    {:source-table (mt/id :venues)
-                                  :limit        10}}
-          :endpoints  [:card :dataset]
-          :assertions {:csv (fn [results]
-                              (is (partial=
-                                   {:error "You do not have permissions to download the results of this query."}
-                                   results)))}})
+         {:query           {:database (mt/id)
+                            :type     :query
+                            :query    {:source-table (mt/id :venues)
+                                       :limit        10}}
+          :expected-status 403
+          :endpoints       [:card :dataset]
+          :assertions      {:csv (fn [results]
+                                   (is (partial=
+                                        {:error "You do not have permissions to download the results of this query."}
+                                        results)))}})
 
         (streaming-test/do-test!
          "A user with no download perms for a table still has full download perms for MBQL queries on other tables"
@@ -342,12 +345,13 @@
 
         (streaming-test/do-test!
          "A user with no download perms for a table has no download perms for native queries on all tables"
-         {:query      (mt/native-query {:query "SELECT * FROM checkins LIMIT 10;"})
-          :endpoints  [:card :dataset]
-          :assertions {:csv (fn [results]
-                              (is (partial=
-                                   {:error "You do not have permissions to download the results of this query."}
-                                   results)))}})))))
+         {:query           (mt/native-query {:query "SELECT * FROM checkins LIMIT 10;"})
+          :expected-status 403
+          :endpoints       [:card :dataset]
+          :assertions      {:csv (fn [results]
+                                   (is (partial=
+                                        {:error "You do not have permissions to download the results of this query."}
+                                        results)))}})))))
 
 (deftest joins-test
   (mt/with-full-data-perms-for-all-users!
@@ -362,11 +366,12 @@
                    {:joins [{:source-table $$categories
                              :condition    [:= $category_id 1]}]
                     :limit 10})
-          :endpoints  [:card :dataset]
-          :assertions {:csv (fn [results]
-                              (is (partial=
-                                   {:error "You do not have permissions to download the results of this query."}
-                                   results)))}})
+          :expected-status 403
+          :endpoints       [:card :dataset]
+          :assertions      {:csv (fn [results]
+                                   (is (partial=
+                                        {:error "You do not have permissions to download the results of this query."}
+                                        results)))}})
 
         (streaming-test/do-test!
          "A user has limited downloads for a query with a join if they have limited permissions for one of the tables"
@@ -425,12 +430,13 @@
      (fn [query]
        (streaming-test/do-test!
         "A table joined to a card, with no download perms for the card, results in blocked download"
-        {:query      query
-         :endpoints  [:card :dataset]
-         :assertions {:csv (fn [results]
-                             (is (partial=
-                                  {:error "You do not have permissions to download the results of this query."}
-                                  results)))}})))))
+        {:query           query
+         :expected-status 403
+         :endpoints       [:card :dataset]
+         :assertions      {:csv (fn [results]
+                                  (is (partial=
+                                       {:error "You do not have permissions to download the results of this query."}
+                                       results)))}})))))
 
 (deftest sandbox-card-test
   (testing "Do we correctly check download perms for queries that involve a sandbox? (#57861)"
@@ -463,15 +469,16 @@
                                                              (mt/id :checkins)   :none}}}
             (streaming-test/do-test!
              "A table with sandbox and not download perms"
-             {:query      {:database (mt/id)
-                           :type     :query
-                           :query    {:source-table (mt/id :checkins)
-                                      :limit        10}}
-              :endpoints  [:card :dataset]
-              :assertions {:csv (fn [results]
-                                  (is (partial=
-                                       {:error "You do not have permissions to download the results of this query."}
-                                       results)))}})))))))
+             {:query           {:database (mt/id)
+                                :type     :query
+                                :query    {:source-table (mt/id :checkins)
+                                           :limit        10}}
+              :expected-status 403
+              :endpoints       [:card :dataset]
+              :assertions      {:csv (fn [results]
+                                       (is (partial=
+                                            {:error "You do not have permissions to download the results of this query."}
+                                            results)))}})))))))
 
 (defn- do-joined-cards-with-native-query-test! [f]
   (testing "Do we correctly apply the least permissive download perms when joining cards where one has a native query?"
@@ -520,21 +527,22 @@
                                                           (mt/id :users)    :limited}}}
          (streaming-test/do-test!
           "Join between MBQL card (venues:full) and native card - should block due to checkins:none"
-          {:query      {:database (mt/id)
-                        :type     :query
-                        :query    {:source-table (format "card__%d" mbql-card-id)
-                                   :joins        [{:fields       [[:field "ID" {:base-type :type/Integer}]]
-                                                   :source-table (format "card__%d" native-card-id)
-                                                   :alias        "native_card"
-                                                   :condition    [:= [:field "ID" {:base-type :type/Integer}]
-                                                                  [:field "VENUE_ID" {:base-type :type/Integer, :join-alias "native_card"}]]
-                                                   :strategy     :left-join}]
-                                   :limit        10}}
-           :endpoints  [:card :dataset]
-           :assertions {:csv (fn [results]
-                               (is (partial=
-                                    {:error "You do not have permissions to download the results of this query."}
-                                    results)))}}))))))
+          {:query           {:database (mt/id)
+                             :type     :query
+                             :query    {:source-table (format "card__%d" mbql-card-id)
+                                        :joins        [{:fields       [[:field "ID" {:base-type :type/Integer}]]
+                                                        :source-table (format "card__%d" native-card-id)
+                                                        :alias        "native_card"
+                                                        :condition    [:= [:field "ID" {:base-type :type/Integer}]
+                                                                       [:field "VENUE_ID" {:base-type :type/Integer, :join-alias "native_card"}]]
+                                                        :strategy     :left-join}]
+                                        :limit        10}}
+           :expected-status 403
+           :endpoints       [:card :dataset]
+           :assertions      {:csv (fn [results]
+                                    (is (partial=
+                                         {:error "You do not have permissions to download the results of this query."}
+                                         results)))}}))))))
 
 (deftest joined-cards-with-native-query-test-3
   (do-joined-cards-with-native-query-test!
@@ -598,15 +606,16 @@
                                                             (mt/id :users)    :full}}}
            (streaming-test/do-test!
             "Card with native source card - should block due to checkins:none in native source"
-            {:query      {:database (mt/id)
-                          :type     :query
-                          :query    {:source-table (format "card__%d" mbql-card-id)
-                                     :limit        10}}
-             :endpoints  [:card :dataset]
-             :assertions {:csv (fn [results]
-                                 (is (partial=
-                                      {:error "You do not have permissions to download the results of this query."}
-                                      results)))}})))))))
+            {:query           {:database (mt/id)
+                               :type     :query
+                               :query    {:source-table (format "card__%d" mbql-card-id)
+                                          :limit        10}}
+             :expected-status 403
+             :endpoints       [:card :dataset]
+             :assertions      {:csv (fn [results]
+                                      (is (partial=
+                                           {:error "You do not have permissions to download the results of this query."}
+                                           results)))}})))))))
 
 (deftest card-with-native-source-card-test-3
   (testing "Do we correctly apply download perms when downloading from a card that uses a source card with a native query?"

enterprise/backend/test/metabase_enterprise/agent_api/api_test.clj

@@ -325,7 +325,7 @@
         (let [decoded (decode-query response)]
           (is (= :mbql/query (lib/normalized-query-type decoded)))
           (is (= (mt/id) (lib/database-id decoded)))
-          (is (= (mt/id :orders) (lib/source-table-id decoded))))))
+          (is (= (mt/id :orders) (lib/primary-source-table-id decoded))))))
 
     (testing "Applies default limit of 100 when no limit is specified"
       (let [table-id (mt/id :orders)

enterprise/backend/test/metabase_enterprise/data_studio/permissions/query_test.clj

@@ -132,7 +132,8 @@
               (perms/set-table-permission! all-users (mt/id :venues) :perms/create-queries :no)
               ;; Without collection permission, query should fail at database check (403)
               (testing "Query should fail because user has no collection permission on published table"
-                (is (= "You don't have permissions to do that."
-                       (mt/with-current-user user-id
-                         (mt/user-http-request user-id :post 403 "dataset"
-                                               (mt/mbql-query venues {:limit 1})))))))))))))
+                (is (=? {:status "failed"
+                         :error  "You do not have permissions to run this query."}
+                        (mt/with-current-user user-id
+                          (mt/user-http-request user-id :post 403 "dataset"
+                                                (mt/mbql-query venues {:limit 1})))))))))))))

enterprise/backend/test/metabase_enterprise/database_routing/e2e_test.clj

@@ -2,6 +2,7 @@
   (:require
    [clojure.java.jdbc :as jdbc]
    [clojure.set :as set]
+   [clojure.string :as str]
    [clojure.test :refer [deftest is testing]]
    [clojurewerkz.quartzite.conversion :as qc]
    [metabase-enterprise.test :as met]
@@ -216,7 +217,7 @@
 
 (defmethod router-dataset-name :default [_driver] "db-router-data")
 
-(doseq [driver [:redshift :databricks :presto-jdbc]]
+(doseq [driver [:redshift :databricks :presto-jdbc :bigquery-cloud-sdk]]
   (defmethod router-dataset-name driver [_driver] "db-routing-data"))
 
 (defmulti routed-dataset-name
@@ -227,7 +228,7 @@
 
 (defmethod routed-dataset-name :default [_driver] "db-routed-data")
 
-(doseq [driver [:redshift :databricks :presto-jdbc]]
+(doseq [driver [:redshift :databricks :presto-jdbc :bigquery-cloud-sdk]]
   (defmethod routed-dataset-name driver [_driver] "db-routing-data"))
 
 (defmulti router-dataset-details
@@ -242,8 +243,8 @@
   {:dbname "db_router_data"
    :enable-multiple-db false})
 
-(defmethod router-dataset-details :bigquery-cloud-sdk [_driver]
-  {:dataset-filters-patterns "metabase_routing_dataset"})
+(defmethod router-dataset-details :bigquery-cloud-sdk [driver]
+  {:dataset-filters-patterns (str "*" (str/replace (router-dataset-name driver) "-" "_"))})
 
 (defmethod router-dataset-details :databricks [driver]
   {:multi-level-schema false
@@ -262,8 +263,8 @@
    :enable-multiple-db false})
 
 (defmethod routed-dataset-details :bigquery-cloud-sdk [driver]
-  {:service-account-json (tx/db-test-env-var-or-throw driver :service-account-json-routing)
-   :dataset-filters-patterns "metabase_routing_dataset"})
+  {:service-account-json     (tx/db-test-env-var-or-throw driver :service-account-json-routing)
+   :dataset-filters-patterns (str "*" (str/replace (routed-dataset-name driver) "-" "_"))})
 
 (defmethod routed-dataset-details :redshift [driver]
   {:db (tx/db-test-env-var-or-throw driver :db-routing)})
@@ -273,22 +274,36 @@
    :multi-level-schema false
    :schema-filters-patterns (routed-dataset-name driver)})
 
+;; the test requires both datasets share a hash (for the name qualifier)
+;; this is because the qp when routing a db does not map to the routed table (it reuses metadata of original)
+;; the end result is the compiled query would retain the routed sha_* qualifier rather than the hash of the original.
+;; using :hash-key now forces the hash key used to sha to be shared between the two datasets.
+
+(defn- router-dataset [driver]
+  (let [dataset (mt/dataset-definition (router-dataset-name driver)
+                                       [["t"
+                                         [{:field-name "f", :base-type :type/Text}]
+                                         [["original-foo"]
+                                          ["original-bar"]]]])]
+    (assoc dataset :hash-key dataset)))
+
+(defn- routed-dataset [driver]
+  (let [router (router-dataset driver)
+        dataset (mt/dataset-definition (routed-dataset-name driver)
+                                       [["t"
+                                         [{:field-name "f", :base-type :type/Text}]
+                                         [["routed-foo"]
+                                          ["routed-bar"]]]])]
+    (assoc dataset :hash-key (:hash-key router))))
+
 (deftest db-routing-e2e-test
   (mt/test-drivers (mt/normal-driver-select {:+features [:database-routing]})
     (mt/with-premium-features #{:database-routing}
       (binding [tx/*use-routing-details* true]
-        (mt/dataset (mt/dataset-definition (routed-dataset-name driver/*driver*)
-                                           [["t"
-                                             [{:field-name "f", :base-type :type/Text}]
-                                             [["routed-foo"]
-                                              ["routed-bar"]]]])
+        (mt/dataset (routed-dataset driver/*driver*)
           (let [routed (mt/db)]
             (binding [tx/*use-routing-details* false]
-              (mt/dataset (mt/dataset-definition (router-dataset-name driver/*driver*)
-                                                 [["t"
-                                                   [{:field-name "f", :base-type :type/Text}]
-                                                   [["original-foo"]
-                                                    ["original-bar"]]]])
+              (mt/dataset (router-dataset driver/*driver*)
                 (let [router (mt/db)]
                   (t2/update! :model/Database (u/the-id routed)
                               {:details (merge (:details routed)

enterprise/backend/test/metabase_enterprise/dependencies/api_test.clj

@@ -3,6 +3,7 @@
    [clojure.test :refer :all]
    [medley.core :as m]
    [metabase-enterprise.dependencies.api :as deps.api]
+   [metabase-enterprise.dependencies.core :as dependencies]
    [metabase-enterprise.dependencies.events]
    [metabase-enterprise.dependencies.findings :as dependencies.findings]
    [metabase-enterprise.dependencies.task.backfill :as dependencies.backfill]
@@ -680,6 +681,31 @@
                                          :source (:source transform)
                                          :target (:target transform)}))))))))
 
+(deftest check-card-bad-transforms-filtered-by-can-read-test
+  (testing "POST /api/ee/dependencies/check-card filters bad_transforms by mi/can-read?"
+    (mt/dataset test-data
+      (mt/with-premium-features #{:dependencies :transforms}
+        (mt/with-temp [:model/User user {:email "test@test.com"}
+                       :model/Transform transform {}]
+          (mt/with-model-cleanup [:model/Card :model/Dependency]
+            (let [base-card (card/create-card! (basic-card) user)
+                  proposed-card {:id (:id base-card)
+                                 :type :question
+                                 :dataset_query (:dataset_query base-card)}]
+              ;; Mock errors-from-proposed-edits to report the transform as broken
+              (with-redefs [dependencies/errors-from-proposed-edits
+                            (constantly {:transform {(:id transform) #{:some-error}}})]
+                (testing "Admin sees broken transforms in response"
+                  (let [response (mt/user-http-request :crowberto :post 200 "ee/dependencies/check-card"
+                                                       proposed-card)]
+                    (is (false? (:success response)))
+                    (is (=? [{:id (:id transform)}]
+                            (:bad_transforms response)))))
+                (testing "Non-admin user cannot see broken transforms they lack read permission for"
+                  (let [response (mt/user-http-request :rasta :post 200 "ee/dependencies/check-card"
+                                                       proposed-card)]
+                    (is (= [] (:bad_transforms response)))))))))))))
+
 (deftest graph-permissions-test
   (testing "GET /api/ee/dependencies/graph requires read permissions on the starting entity"
     (mt/with-premium-features #{:dependencies}

enterprise/backend/test/metabase_enterprise/sandbox/api/card_test.clj

@@ -131,10 +131,11 @@
             (data-perms/set-table-permission! group (mt/id :venues) :perms/view-data view-perm)
             (data-perms/set-table-permission! group (mt/id :venues) :perms/create-queries create-perm)
             (testing (str "view-data: " view-perm ", create-queries: " create-perm)
-              (let [response (mt/user-http-request user-id :post 202 (str "card/" (u/the-id card) "/query"))]
-                (if should-succeed?
-                  (is (= 1 (count (mt/rows response))))
-                  (is (thrown? clojure.lang.ExceptionInfo (mt/rows response))))))))))))
+              (if should-succeed?
+                (let [response (mt/user-http-request user-id :post 202 (str "card/" (u/the-id card) "/query"))]
+                  (is (= 1 (count (mt/rows response)))))
+                (let [response (mt/user-http-request user-id :post 403 (str "card/" (u/the-id card) "/query"))]
+                  (is (partial= {:error_type "missing-required-permissions"} response)))))))))))
 
 (deftest sandbox-join-permissions-test
   (testing "Sandboxed query can't be saved when sandboxed table is joined to a table that the current user doesn't have access to"

enterprise/backend/test/metabase_enterprise/serialization/v2/e2e_test.clj

@@ -430,6 +430,7 @@
                   (is (= [{:id                   "abc",
                            :name                 "CATEGORY",
                            :type                 :category,
+                           :position             0,
                            :values_source_config {:card_id     (:entity_id card1s),
                                                   :value_field [:field
                                                                 ["my-db" nil "CUSTOMERS" (:name field1s)]
@@ -442,6 +443,7 @@
                            [{:id                   "abc",
                              :name                 "CATEGORY",
                              :type                 :category,
+                             :position             0,
                              :values_source_config {:card_id     (:entity_id card1s),
                                                     :value_field [:field
                                                                   ["my-db" nil "CUSTOMERS" (:name field1s)]

enterprise/backend/test/metabase_enterprise/serialization/v2/extract_test.clj

@@ -1296,6 +1296,7 @@
           (is (= [{:id                   "abc",
                    :type                 :category,
                    :name                 "CATEGORY",
+                   :position             0,
                    :values_source_type   :card
                    :values_source_config {:card_id card-eid-1,
                                           :value_field [:field ["My Database" nil "Schemaless Table" "A Field"] nil]}}]
@@ -2555,8 +2556,12 @@
                 {:id "alpha" :name "A param" :type :category}
                 {:id "middle" :name "M param" :type :category}]
         result (serdes/export-parameters params)]
-    (is (= ["alpha" "middle" "zebra"]
-           (map :id result)))))
+    (testing "export sorts parameters by :id for stable diffs"
+      (is (= ["alpha" "middle" "zebra"]
+             (map :id result))))
+    (testing "export stamps :position from original array index"
+      (is (= [1 2 0]
+             (map :position result))))))
 
 (deftest ^:parallel export-parameters-empty-input-test
   (is (= [] (serdes/export-parameters []))))
@@ -2567,14 +2572,16 @@
 (deftest ^:parallel export-parameters-single-element-test
   (let [params [{:id "only" :name "Only param"}]
         result (serdes/export-parameters params)]
-    (is (= ["only"] (map :id result)))))
+    (is (= ["only"] (map :id result)))
+    (is (= [0] (map :position result)))))
 
 (deftest ^:parallel export-parameters-nil-id-sorts-first-test
   (let [params [{:id "beta" :name "B"}
                 {:id nil :name "Nil"}
                 {:id "alpha" :name "A"}]
         result (serdes/export-parameters params)]
-    (is (= [nil "alpha" "beta"] (map :id result)))))
+    (is (= [nil "alpha" "beta"] (map :id result)))
+    (is (= [1 2 0] (map :position result)))))
 
 (deftest ^:parallel export-parameter-mappings-sorts-by-parameter-id-test
   (let [mappings [{:parameter_id "z-param" :target [:dimension [:field 1 nil]]}
@@ -2614,7 +2621,32 @@
                                                                      :card_id card-id
                                                                      :target [:dimension [:field 2 nil]]}]}]
       (let [ser (ts/extract-one "Dashboard" dash-id)]
-        (is (= ["alpha" "beta" "zebra"]
-               (map :id (:parameters ser))))
+        (testing "parameters are sorted by :id"
+          (is (= ["alpha" "beta" "zebra"]
+                 (map :id (:parameters ser)))))
+        (testing "positions preserve original display order"
+          (is (= [1 2 0]
+                 (map :position (:parameters ser)))))
         (is (= ["alpha" "zebra"]
                (map :parameter_id (-> ser :dashcards first :parameter_mappings))))))))
+
+(deftest ^:parallel import-parameters-round-trip-test
+  (testing "export stamps position, import restores original display order"
+    (let [params   [{:id "zebra" :name "Z param" :type :category}
+                    {:id "alpha" :name "A param" :type :category}
+                    {:id "middle" :name "M param" :type :category}]
+          exported (serdes/export-parameters params)
+          imported (serdes/import-parameters exported)]
+      (is (= ["alpha" "middle" "zebra"] (map :id exported))
+          "export sorts by id")
+      (is (= [1 2 0] (map :position exported))
+          "export stamps position from original array index")
+      (is (= ["zebra" "alpha" "middle"] (map :id imported))
+          "import restores original display order via position"))))
+
+(deftest ^:parallel import-parameters-without-position-preserves-order-test
+  (testing "importing legacy params without :position preserves existing order"
+    (let [params   [{:id "zebra" :name "Z param" :type :category}
+                    {:id "alpha" :name "A param" :type :category}]
+          imported (serdes/import-parameters params)]
+      (is (= ["zebra" "alpha"] (map :id imported))))))

enterprise/backend/test/metabase_enterprise/transforms_python/drivers_test.clj

@@ -87,10 +87,9 @@
                     [2 nil]]}
 
    :bigquery-cloud-sdk {:columns [{:name "id" :type :type/Integer :nullable? false}
-                                  {:name "json_field" :type :type/JSON :nullable? true}
-                                  {:name "dict_field" :type :type/Dictionary :nullable? true :database-type "STRUCT<key STRING, value INT64>"}]
-                        :data [[1 "{\"key\": \"value\"}" {"key" "test", "value" 42}]
-                               [2 nil nil]]}
+                                  {:name "json_field" :type :type/JSON :nullable? true}]
+                        :data [[1 "{\"key\": \"value\"}"]
+                               [2 nil]]}
    :sqlserver {:columns [{:name "id" :type :type/Integer :nullable? false}
                          {:name "uuid_field" :type :type/UUID :nullable? true}
                          {:name "datetimeoffset_field" :type :type/DateTimeWithTZ :nullable? true :database-type "datetimeoffset"}]
@@ -118,8 +117,9 @@
         qualified-table-name (if schema-name
                                (keyword schema-name table-name)
                                (keyword table-name))
+        columns (:columns schema)
         table-schema {:name qualified-table-name
-                      :columns (:columns schema)}]
+                      :columns columns}]
     (mt/as-admin
       (transforms.util/create-table-from-schema! driver db-id table-schema))
 
@@ -127,15 +127,21 @@
       (driver/insert-from-source! driver db-id table-schema
                                   {:type :rows
                                    :data (map (fn [row]
-                                                (map #(cond
-                                                        (and (= :sqlserver driver) (boolean? %))
-                                                        (if % 1 0)
+                                                (map-indexed
+                                                 (fn [i v]
+                                                   (cond
+                                                     (and (= :bigquery-cloud-sdk driver) (string? v) (= :type/JSON (:type (nth columns i))))
+                                                     ;; it might be better to have a generic tagged value abstraction for drivers to work with
+                                                     ((requiring-resolve 'metabase.driver.bigquery-cloud-sdk.params/param) "JSON" v)
 
-                                                        (and (= :mongo driver) (string? %) (try (u.date/parse %) (catch Exception _)))
-                                                        (u.date/parse %)
+                                                     (and (= :sqlserver driver) (boolean? v))
+                                                     (if v 1 0)
 
-                                                        :else
-                                                        %) row))
+                                                     (and (= :mongo driver) (string? v) (try (u.date/parse v) (catch Exception _)))
+                                                     (u.date/parse v)
+
+                                                     :else
+                                                     v)) row))
                                               data)}))
 
     (sync/sync-database! (mt/db) {:scan :schema})
@@ -147,7 +153,7 @@
   [table-id]
   (try
     (when-let [table (t2/select-one :model/Table :id table-id)]
-      (let [table-name (:name table)
+      (let [table-name     (:name table)
             qualified-name (case driver/*driver*
                              :bigquery-cloud-sdk
                              (let [schema (sql.tx/session-schema driver/*driver*)]
@@ -254,9 +260,7 @@
                            (is (isa? (type-map "json_field") :type/Text #_:type/JSON))
                            (is (isa? (type-map "timestamp") :type/DateTimeWithLocalTZ))))
 
-                :bigquery-cloud-sdk (do
-                                      (is (isa? (type-map "json_field") :type/Text #_:type/JSON))
-                                      (is (isa? (type-map "dict_field") :type/Text #_:type/JSON)))
+                :bigquery-cloud-sdk (is (isa? (type-map "json_field") :type/Text #_:type/JSON))
 
                 :mongo (do
                          (is (isa? (type-map "array_field") :type/Text #_:type/Array))
@@ -359,7 +363,7 @@
                     (is (isa? (type-map "bool_inverted") :type/Boolean)))
 
                   (testing "Edge case data handling"
-                    (let [[row1 row2 row3] rows]
+                    (let [[row1 row2 row3] (sort-by #(get % "id") rows)]
                       (is (= 1 (get row1 "id")))
                       (is (= 0 (get row1 "text_length"))) ; empty string length
                       (is (= 0 (get row1 "int_doubled"))) ; 0 * 2
@@ -458,7 +462,10 @@
                                       (assoc (str source-table-name "_exotic") exotic-table-id))
 
                       result (execute-e2e-transform! table-name transform-code source-tables)
-                      {:keys [columns] result-rows :rows} result]
+                      {:keys [columns] result-rows :rows} result
+                      ;; deal with non-deterministic order (e.g. bigquery)
+                      id-idx      (u/index-of #{"id"} columns)
+                      result-rows (sort-by #(nth % id-idx) result-rows)]
 
                   (testing "E2E transform execution succeeded"
                     (is (seq result-rows) "Transform should produce results"))

enterprise/backend/test/metabase_enterprise/transforms_python/python_runner_test.clj

@@ -480,7 +480,7 @@
                  (set (map :name (:fields metadata))))))
 
         (testing "types are preserved correctly"
-          (is (= {"id"           (if (= :snowflake driver) :type/Number :type/Integer)
+          (is (= {"id"           (if (= :snowflake driver) :type/BigInteger :type/Integer)
                   "price"        :type/Float
                   "active"       :type/Boolean
                   "created_tz"   (case driver

enterprise/backend/test/metabase_enterprise/transforms_python/transforms_api_test.clj

@@ -158,8 +158,7 @@
                 (transforms.tu/test-run transform-id)
                 (transforms.tu/wait-for-table table-name 10000)
                 (is (true? (driver/table-exists? driver/*driver* (mt/db) target)))
-                (is (= [["Alice" 25] ["Bob" 30]]
-                       (transforms.tu/table-rows table-name)))))))))))
+                (is (= [["Alice" 25] ["Bob" 30]] (sort-by first (transforms.tu/table-rows table-name))))))))))))
 
 (defn- subsequence?
   "Returns true if sequence ys is a subsequence of xs:

enterprise/frontend/src/metabase-enterprise/content_translation/use-translate-content.ts

@@ -24,12 +24,14 @@ export const useTranslateContent = (): ContentTranslationFunction => {
 
   const tc = useCallback<ContentTranslationFunction>(
     <T = string | null | undefined>(msgid: T) =>
-      dictionary?.length
-        ? translateContentString<T>(dictionary || [], locale, msgid)
-        : leaveUntranslated(msgid),
+      translateContentString<T>(dictionary || [], locale, msgid),
     [locale, dictionary],
   );
 
+  if (!dictionary?.length) {
+    return leaveUntranslated;
+  }
+
   return tc;
 };
 

enterprise/frontend/src/metabase-enterprise/tools/ErrorOverview.jsx

@@ -7,6 +7,7 @@ import {
   SettingsSection,
 } from "metabase/admin/components/SettingsSection";
 import CS from "metabase/css/core/index.css";
+import { fetchDataOrError } from "metabase/dashboard/utils";
 import { CardApi } from "metabase/services";
 
 import AuditParameters from "../audit_app/components/AuditParameters";
@@ -59,7 +60,8 @@ export default function ErrorOverview(props) {
 
     await Promise.all(
       checkedCardIds.map(
-        async (member) => await CardApi.query({ cardId: member }),
+        async (member) =>
+          await fetchDataOrError(CardApi.query({ cardId: member })),
       ),
     );
     setRowChecked({});

frontend/src/embedding-sdk-bundle/components/private/SdkAdHocQuestion/SdkAdHocQuestion.tsx

@@ -65,7 +65,9 @@ export const SdkAdHocQuestion = ({
 
   const { options, deserializedCard } = useMemo(() => {
     const { options, serializedCard } = parseHash(location.hash);
-    const deserializedCard = serializedCard && deserializeCard(serializedCard);
+    const deserializedCard = serializedCard
+      ? deserializeCard(serializedCard)
+      : undefined;
 
     return { options, deserializedCard };
   }, [location.hash]);

frontend/src/metabase-lib/v1/Question.ts

@@ -4,7 +4,12 @@ import { assoc, assocIn, chain, dissoc, getIn } from "icepick";
 import slugg from "slugg";
 import _ from "underscore";
 
-import { utf8_to_b64url } from "metabase/lib/encoding";
+// eslint-disable-next-line no-restricted-imports
+import {
+  type SerializeCardOptions,
+  serializeCardForUrl,
+} from "metabase/lib/card";
+import { equals } from "metabase/lib/utils";
 import { applyParameter } from "metabase/querying/parameters/utils/query";
 import * as Lib from "metabase-lib";
 import {
@@ -24,10 +29,9 @@ import NativeQuery, {
 } from "metabase-lib/v1/queries/NativeQuery";
 import { STRUCTURED_QUERY_TEMPLATE } from "metabase-lib/v1/queries/StructuredQuery";
 import { isTransientId } from "metabase-lib/v1/queries/utils/card";
-import { sortObject } from "metabase-lib/v1/utils";
 import type {
+  Card,
   CardDisplayType,
-  Card as CardObject,
   CardType,
   CollectionId,
   DashCardId,
@@ -79,7 +83,7 @@ class Question {
    * The plain object presentation of this question, equal to the format that Metabase REST API understands.
    * It is called `card` for both historical reasons and to make a clear distinction to this class.
    */
-  _card: CardObject;
+  _card: Card;
 
   /**
    * The Question wrapper requires a metadata object because the queries it contains (like {@link StructuredQuery})
@@ -135,7 +139,7 @@ class Question {
     return this._card;
   }
 
-  setCard(card: CardObject): Question {
+  setCard(card: Card): Question {
     const q = this.clone();
     q._card = card;
     return q;
@@ -677,19 +681,10 @@ class Question {
     } else {
       // If it's saved, then it's dirty when the current card doesn't match the last saved version.
       // Omit `entity_id` and `dataset_query` as they have randomized idents
-      const origCardSerialized =
-        originalQuestion &&
-        originalQuestion._serializeForUrl({
-          includeEntityId: false,
-          includeDatasetQuery: false,
-          includeOriginalCardId: false,
-        });
-      const currentCardSerialized = this._serializeForUrl({
-        includeEntityId: false,
-        includeDatasetQuery: false,
-        includeOriginalCardId: false,
-      });
-      if (currentCardSerialized !== origCardSerialized) {
+      const originalCard = originalQuestion?._getValueForComparison();
+      const currentCard = this._getValueForComparison();
+
+      if (!equals(originalCard, currentCard)) {
         return true;
       }
 
@@ -724,60 +719,44 @@ class Question {
     );
   }
 
-  // Internal methods
-  _serializeForUrl({
-    includeEntityId = true,
-    includeDatasetQuery = true,
-    includeOriginalCardId = true,
-    includeDisplayIsLocked = false,
-    creationType,
-  }: {
-    includeEntityId?: boolean;
-    includeDatasetQuery?: boolean;
-    includeOriginalCardId?: boolean;
-    includeDisplayIsLocked?: boolean;
-    creationType?: string;
-  } = {}) {
-    const card = this._card;
-    const cardCopy = {
-      name: card.name,
-      description: card.description,
-      collection_id: card.collection_id,
-      dashboard_id: card.dashboard_id,
-      ...(includeEntityId ? { entity_id: card.entity_id } : {}),
-      ...(includeDatasetQuery
-        ? { dataset_query: Lib.toJsQuery(this.query()) }
-        : {}),
-      display: card.display,
-      ...(_.isEmpty(card.parameters)
-        ? undefined
-        : {
-            parameters: card.parameters,
-          }),
-      type: card.type,
-      ...(_.isEmpty(this._parameterValues)
-        ? undefined
-        : {
-            parameterValues: this._parameterValues,
-          }),
-      // this is kinda wrong. these values aren't really part of the card, but this is a convenient place to put them
-      visualization_settings: card.visualization_settings,
-      ...(includeOriginalCardId
-        ? {
-            original_card_id: card.original_card_id,
-          }
-        : {}),
-      ...(includeDisplayIsLocked
-        ? {
-            displayIsLocked: card.displayIsLocked,
-          }
-        : {}),
-
-      ...(creationType ? { creationType } : {}),
-      dashboardId: card.dashboardId,
-      dashcardId: card.dashcardId,
+  serializeForUrl(opts: SerializeCardOptions = {}) {
+    const card = {
+      ...this.card(),
+      dataset_query: Lib.toJsQuery(this.query()),
     };
-    return utf8_to_b64url(JSON.stringify(sortObject(cardCopy)));
+    return serializeCardForUrl(card, {
+      ...opts,
+      parameterValues: this._parameterValues,
+    });
+  }
+
+  // Internal methods
+  _getValueForComparison() {
+    const value = {
+      ...this._card,
+      ...this._parameterValues,
+    };
+
+    const keys = [
+      "collection_id",
+      "dashboard_id",
+      "dashboardId",
+      "dashcardId",
+      "description",
+      "display",
+      "name",
+      "parameters",
+      "parameterValues",
+      "type",
+      "visualization_settings",
+    ];
+
+    const res = {};
+    for (const key of keys) {
+      res[key] = value[key] ?? undefined;
+    }
+
+    return res;
   }
 
   _convertParametersToMbql({ isComposed }: { isComposed: boolean }): Question {
@@ -911,7 +890,7 @@ class Question {
       ? NATIVE_QUERY_TEMPLATE
       : STRUCTURED_QUERY_TEMPLATE,
   }: QuestionCreatorOpts = {}) {
-    let card: CardObject = {
+    let card: Card = {
       name,
       collection_id: collectionId,
       dashboard_id: dashboardId,

frontend/src/metabase-lib/v1/urls.ts

@@ -19,12 +19,7 @@ type UrlBuilderOpts = {
 
 export function getUrl(
   question: Question,
-  {
-    originalQuestion,
-    query,
-    includeDisplayIsLocked,
-    creationType,
-  }: UrlBuilderOpts = {},
+  { originalQuestion, query, creationType }: UrlBuilderOpts = {},
 ) {
   question = question.omitTransientCardIds();
 
@@ -33,8 +28,8 @@ export function getUrl(
     (originalQuestion && question.isDirtyComparedTo(originalQuestion))
   ) {
     return Urls.question(null, {
-      hash: question._serializeForUrl({
-        includeDisplayIsLocked,
+      hash: question.serializeForUrl({
+        includeDisplayIsLocked: true,
         creationType,
       }),
       query,

frontend/src/metabase-types/analytics/event.ts

@@ -586,6 +586,11 @@ export type BookmarkEvent =
   | BookmarkCollectionEvent
   | BookmarkDocumentEvent;
 
+export type DataStudioOpenedEvent = ValidateEvent<{
+  event: "data_studio_opened";
+  triggered_from: "nav_menu";
+}>;
+
 export type DataStudioLibraryCreatedEvent = ValidateEvent<{
   event: "data_studio_library_created";
   target_id: number | null;
@@ -654,6 +659,7 @@ export type DataStudioTableFieldValuesDiscardedEvent = ValidateEvent<{
 }>;
 
 export type DataStudioEvent =
+  | DataStudioOpenedEvent
   | DataStudioLibraryCreatedEvent
   | DataStudioTablePublishedEvent
   | DataStudioGlossaryCreatedEvent

frontend/src/metabase-types/api/card.ts

@@ -89,6 +89,7 @@ export interface Card<Q extends DatasetQuery = DatasetQuery>
   view_count?: number;
 
   download_perms?: DownloadPermission;
+  displayIsLocked?: boolean;
 }
 
 export interface PublicCard {

frontend/src/metabase-types/api/dataset.ts

@@ -23,6 +23,12 @@ import type { TableId } from "./table";
 export type RowValue = string | number | null | boolean | object;
 export type RowValues = RowValue[];
 
+export function getRowsForStableKeys(
+  data: Pick<DatasetData, "rows" | "untranslatedRows">,
+): RowValues[] {
+  return data.untranslatedRows ?? data.rows;
+}
+
 export type BinningMetadata = {
   binning_strategy?: "default" | "bin-width" | "num-bins";
   bin_width?: number;
@@ -102,6 +108,7 @@ export interface DatasetData {
     "show-row-totals"?: boolean;
     "show-column-totals"?: boolean;
   };
+  untranslatedRows?: RowValues[];
 }
 
 export type JsonQuery = DatasetQuery & {

frontend/src/metabase-types/api/mocks/dashboard.ts

@@ -1,6 +1,7 @@
 import type {
   ActionDashboardCard,
   Dashboard,
+  DashboardParameterMapping,
   DashboardQueryMetadata,
   DashboardTab,
   QuestionDashboardCard,
@@ -81,6 +82,15 @@ export const createMockDashboardCard = (
   ...opts,
 });
 
+export const createMockParameterMapping = (
+  opts?: Partial<DashboardParameterMapping>,
+): DashboardParameterMapping => ({
+  card_id: 1,
+  target: ["variable", ["template-tag", "foo"]],
+  parameter_id: "foo",
+  ...opts,
+});
+
 export const createMockVirtualCard = (
   opts?: Partial<VirtualCard>,
 ): VirtualCard => ({

frontend/src/metabase/app-main.js

@@ -18,7 +18,6 @@ const NOT_AUTHORIZED_TRIGGERS = [
   /\/api\/collection\/\d+(?:\/items)?$/,
   /\/api\/card\/\d+$/,
   /\/api\/pulse\/\d+$/,
-  /\/api\/dataset$/,
 ];
 
 init(mainReducers, getRoutes, (store) => {

frontend/src/metabase/dashboard/actions/data-fetching.ts

@@ -330,7 +330,7 @@ export const fetchCardDataAction = createAsyncThunk<
         ),
       );
     } else if (dashboardType === "public") {
-      result = await fetchDataOrError(
+      result = (await fetchDataOrError(
         maybeUsePivotEndpoint(
           PublicApi.dashboardCardQuery,
           card,
@@ -347,9 +347,9 @@ export const fetchCardDataAction = createAsyncThunk<
           },
           queryOptions,
         ),
-      );
+      )) as Dataset | { error: unknown };
     } else if (dashboardType === "embed") {
-      result = await fetchDataOrError(
+      result = (await fetchDataOrError(
         maybeUsePivotEndpoint(
           EmbedApi.dashboardCardQuery,
           card,
@@ -366,15 +366,15 @@ export const fetchCardDataAction = createAsyncThunk<
           },
           queryOptions,
         ),
-      );
+      )) as Dataset | { error: unknown };
     } else if (dashboardType === "transient" || dashboardType === "inline") {
-      result = await fetchDataOrError(
+      result = (await fetchDataOrError(
         maybeUsePivotEndpoint(
           MetabaseApi.dataset,
           card,
           metadata,
         )({ ...datasetQuery, ignore_cache: ignoreCache }, queryOptions),
-      );
+      )) as Dataset | { error: unknown };
     } else {
       const dashcardBeforeEditing = getDashCardBeforeEditing(
         getState(),
@@ -407,13 +407,13 @@ export const fetchCardDataAction = createAsyncThunk<
             dashboard_id: dashcard.dashboard_id,
             dashboard_load_id: dashboardLoadId,
           };
-      result = await fetchDataOrError(
+      result = (await fetchDataOrError(
         maybeUsePivotEndpoint(
           endpoint,
           card,
           metadata,
         )(requestBody, queryOptions),
-      );
+      )) as Dataset | { error: unknown };
     }
 
     // If the request was not previously cancelled, then clear the defer for the card

frontend/src/metabase/dashboard/actions/tabs.ts

@@ -219,7 +219,11 @@ export function getPrevDashAndTabs({
   filterRemovedTabs?: boolean;
 }) {
   const dashId = state.dashboardId;
-  const prevDash = dashId ? state.dashboards[dashId] : null;
+  // eslint-disable-next-line @typescript-eslint/ban-ts-comment
+  // @ts-ignore - possibly infinite type error
+  const prevDash: StoreDashboard | null = dashId
+    ? state.dashboards[dashId]
+    : null;
   const prevTabs =
     prevDash?.tabs?.filter((t) => !filterRemovedTabs || !t.isRemoved) ?? [];
 

frontend/src/metabase/dashboard/components/DashboardCopyModal.tsx

@@ -88,7 +88,7 @@ const DashboardCopyModal = ({
       }}
       title={title}
       overwriteOnInitialValuesChange
-      copy={async (object: Record<string, unknown>) =>
+      copy={async (object) =>
         await copyDashboard({ id: dashboardIdFromSlug }, dissoc(object, "id"))
       }
       onClose={onClose}

frontend/src/metabase/dashboard/components/DashboardLeaveConfirmationModal/utils.ts

@@ -17,7 +17,7 @@ export const isNavigatingToCreateADashboardQuestion = (
     const isDashboardQuestion =
       deserializedCard && deserializedCard.dashboard_id !== null;
 
-    return isDashboardQuestion;
+    return Boolean(isDashboardQuestion);
   }
 
   return false;

frontend/src/metabase/dashboard/utils.ts

@@ -266,6 +266,23 @@ export async function fetchDataOrError<T>(dataPromise: Promise<T>) {
   try {
     return await dataPromise;
   } catch (error) {
+    // For 4xx errors from streaming query endpoints, the error response body
+    // contains the actual error data that should be displayed (just like the old
+    // 202-with-error-in-body behavior). Treat these as successful responses.
+    if (
+      error &&
+      typeof error === "object" &&
+      "status" in error &&
+      typeof error.status === "number" &&
+      error.status >= 400 &&
+      error.status < 500 &&
+      "data" in error &&
+      typeof error.data === "object"
+    ) {
+      // Return the error data as if it were a successful response
+      return error.data;
+    }
+    // For 5xx errors or other errors, maintain the original behavior
     return { error };
   }
 }

frontend/src/metabase/dashboard/utils.unit.spec.ts

@@ -75,7 +75,8 @@ describe("Dashboard utils", () => {
       const failedFetch = Promise.reject(error);
 
       const result = await fetchDataOrError(failedFetch);
-      expect(result.error).toEqual(error);
+      expect(result).toBeTruthy();
+      expect((result as { error: unknown }).error).toEqual(error);
     });
 
     it("should return true if a database has model actions enabled", () => {

frontend/src/metabase/data-studio/analytics.ts

@@ -139,3 +139,10 @@ export const trackDependencyEntitySelected = ({
     target_id: entityId,
   });
 };
+
+export const trackDataStudioOpened = () => {
+  trackSimpleEvent({
+    event: "data_studio_opened",
+    triggered_from: "nav_menu",
+  });
+};

frontend/src/metabase/documents/components/DocumentCopyForm/DocumentCopyForm.tsx

@@ -11,8 +11,9 @@ import {
 } from "metabase/forms";
 import { Button, Stack } from "metabase/ui";
 import type { CollectionId } from "metabase-types/api";
+import type { Document } from "metabase-types/api/document";
 
-type CopyDocumentProperties = {
+export type CopyDocumentProperties = {
   collection_id: CollectionId | null;
   name: string;
 };

frontend/src/metabase/documents/components/DocumentPage.tsx

@@ -549,7 +549,7 @@ export const DocumentPage = ({
           />
         )}
 
-        {duplicateModalMode === "duplicate" && (
+        {duplicateModalMode === "duplicate" && documentData && (
           <EntityCopyModal
             entityType="documents"
             onClose={() => setDuplicateModalMode(null)}
@@ -569,18 +569,20 @@ export const DocumentPage = ({
                 );
               }
 
-              return await copyDocument({
+              const response = await copyDocument({
                 ...object,
                 id: documentData.id,
-              }).then((response) => {
-                if (response.data) {
-                  const _document = response.data;
-                  trackDocumentDuplicated(_document);
-                  return _document;
-                } else if (response.error) {
-                  throw response.error;
-                }
               });
+
+              if (!response.data) {
+                throw (
+                  response.error ?? new Error("Failed to duplicate document")
+                );
+              }
+
+              const _document = response.data;
+              trackDocumentDuplicated(_document);
+              return _document;
             }}
           />
         )}

frontend/src/metabase/documents/documents.slice.ts

@@ -84,7 +84,8 @@ const documentsSlice = createSlice({
       state.cardEmbeds = action.payload;
     },
     setCurrentDocument: (state, action: PayloadAction<Document | null>) => {
-      // @ts-expect-error - possibly infinite type error
+      // eslint-disable-next-line @typescript-eslint/ban-ts-comment
+      // @ts-ignore - possibly infinite type error
       state.currentDocument = action.payload;
     },
     resetDocuments: () => {

frontend/src/metabase/entities/containers/EntityCopyModal.tsx

@@ -3,35 +3,32 @@ import { t } from "ttag";
 
 import { useGetDefaultCollectionId } from "metabase/collections/hooks";
 import { useEscapeToCloseModal } from "metabase/common/hooks/use-escape-to-close-modal";
-import {
-  CopyDashboardFormConnected,
-  type CopyDashboardFormProperties,
-} from "metabase/dashboard/containers/CopyDashboardForm";
+import { CopyDashboardFormConnected } from "metabase/dashboard/containers/CopyDashboardForm";
 import { DocumentCopyForm } from "metabase/documents/components/DocumentCopyForm/DocumentCopyForm";
 import { CopyCardForm } from "metabase/questions/components/CopyCardForm/CopyCardForm";
 import { Modal } from "metabase/ui";
 
-interface EntityCopyModalProps {
-  entityType: string;
-  entityObject: any;
-  copy: (data: any) => Promise<any>;
-  title?: string;
-  onClose: () => void;
-  onSaved: (newEntityObject?: any) => void;
-  overwriteOnInitialValuesChange?: boolean;
-  onValuesChange?: (values: CopyDashboardFormProperties) => void;
-  form?: any;
-}
+import type {
+  CopyableEntityType,
+  EntityCopyModalProps,
+  GenericEntityCopyModalProps,
+} from "./EntityCopyModal.types";
+
+function EntityCopyModal<T extends CopyableEntityType>(
+  props: EntityCopyModalProps<T>,
+): React.JSX.Element;
+function EntityCopyModal(props: GenericEntityCopyModalProps): React.JSX.Element;
+function EntityCopyModal(props: GenericEntityCopyModalProps) {
+  const {
+    entityType,
+    entityObject,
+    copy,
+    title,
+    onClose,
+    onSaved,
+    onValuesChange,
+  } = props;
 
-const EntityCopyModal = ({
-  entityType,
-  entityObject,
-  copy,
-  title,
-  onClose,
-  onSaved,
-  ...props
-}: EntityCopyModalProps) => {
   const resolvedObject =
     typeof entityObject?.getPlainObject === "function"
       ? entityObject.getPlainObject()
@@ -66,7 +63,7 @@ const EntityCopyModal = ({
           onClose={onClose}
           onSaved={onSaved}
           initialValues={initialValues}
-          {...props}
+          onValuesChange={onValuesChange}
           originalDashboardId={resolvedObjectWithDefaultCollection.id}
         />
       )}
@@ -77,7 +74,6 @@ const EntityCopyModal = ({
           onSaved={onSaved}
           initialValues={initialValues}
           model={entityObject?.type}
-          {...props}
         />
       )}
       {entityType === "documents" && (
@@ -86,12 +82,11 @@ const EntityCopyModal = ({
           onCancel={onClose}
           onSaved={onSaved}
           initialValues={initialValues}
-          {...props}
         />
       )}
     </Modal>
   );
-};
+}
 
 // eslint-disable-next-line import/no-default-export -- deprecated usage
 export default EntityCopyModal;

frontend/src/metabase/entities/containers/EntityCopyModal.types.ts

@@ -0,0 +1,59 @@
+import type { CopyDashboardFormProperties } from "metabase/dashboard/containers/CopyDashboardForm";
+import type { CopyDocumentProperties } from "metabase/documents/components/DocumentCopyForm/DocumentCopyForm";
+import type { CopyCardProperties } from "metabase/questions/components/CopyCardForm/CopyCardForm";
+import type { Card, Dashboard } from "metabase-types/api";
+import type { Document } from "metabase-types/api/document";
+
+/**
+ * Registry mapping entityType strings to their form properties and entity types.
+ * To add a new entity type, extend this interface.
+ */
+export interface EntityCopyTypeRegistry {
+  dashboards: {
+    formProperties: CopyDashboardFormProperties;
+    entity: Dashboard;
+  };
+  cards: {
+    formProperties: CopyCardProperties;
+    entity: Card;
+  };
+  documents: {
+    formProperties: CopyDocumentProperties;
+    entity: Document;
+  };
+}
+
+export type CopyableEntityType = keyof EntityCopyTypeRegistry;
+
+export interface EntityCopyModalProps<T extends CopyableEntityType> {
+  entityType: T;
+  entityObject: EntityCopyTypeRegistry[T]["entity"];
+  copy: (
+    data: EntityCopyTypeRegistry[T]["formProperties"],
+  ) => Promise<EntityCopyTypeRegistry[T]["entity"]>;
+  title?: string;
+  onClose: () => void;
+  onSaved: (newEntity: EntityCopyTypeRegistry[T]["entity"]) => void;
+  overwriteOnInitialValuesChange?: boolean;
+  onValuesChange?: (
+    values: EntityCopyTypeRegistry[T]["formProperties"],
+  ) => void;
+}
+
+/**
+ * Generic fallback props used by the implementation signature.
+ *
+ * Call sites should prefer the typed overloads using `EntityCopyModalProps<T>`.
+ * The implementation stays broadly typed to avoid overly complex unions while
+ * still being safe in practice (forms validate their own values).
+ */
+export interface GenericEntityCopyModalProps {
+  entityType: string;
+  entityObject: any;
+  copy: (data: any) => Promise<any>;
+  title?: string;
+  onClose: () => void;
+  onSaved: (newEntity?: any) => void;
+  overwriteOnInitialValuesChange?: boolean;
+  onValuesChange?: (values: any) => void;
+}

frontend/src/metabase/lib/card.js

@@ -1,45 +0,0 @@
-import { b64hash_to_utf8, utf8_to_b64url } from "metabase/lib/encoding";
-import { equals } from "metabase/lib/utils";
-
-export function createCard(name = null) {
-  return {
-    name: name,
-    display: "table",
-    visualization_settings: {},
-    dataset_query: {},
-  };
-}
-
-function getCleanCard(card) {
-  return {
-    name: card.name,
-    collectionId: card.collectionId,
-    description: card.description,
-    dataset_query: card.dataset_query,
-    display: card.display,
-    displayIsLocked: card.displayIsLocked,
-    parameters: card.parameters,
-    dashboardId: card.dashboardId,
-    dashcardId: card.dashcardId,
-    visualization_settings: card.visualization_settings,
-    original_card_id: card.original_card_id,
-    type: card.type,
-  };
-}
-
-export function isEqualCard(card1, card2) {
-  if (card1 && card2) {
-    return equals(getCleanCard(card1), getCleanCard(card2));
-  } else {
-    return false;
-  }
-}
-
-// TODO Atte Keinänen 5/31/17 Deprecated, we should move tests to Questions.spec.js
-export function serializeCardForUrl(card) {
-  return utf8_to_b64url(JSON.stringify(getCleanCard(card)));
-}
-
-export function deserializeCardFromUrl(serialized) {
-  return JSON.parse(b64hash_to_utf8(serialized));
-}

frontend/src/metabase/lib/card.ts

@@ -0,0 +1,83 @@
+import { b64hash_to_utf8, utf8_to_b64url } from "metabase/lib/encoding";
+import { stableStringify } from "metabase/lib/objects";
+import { equals } from "metabase/lib/utils";
+import type { Card, ParameterValuesMap, UnsavedCard } from "metabase-types/api";
+
+export type SerializeCardOptions = {
+  includeDatasetQuery?: boolean;
+  includeOriginalCardId?: boolean;
+  includeDisplayIsLocked?: boolean;
+  creationType?: string;
+  parameterValues?: ParameterValuesMap;
+};
+
+function getCleanCard(
+  card: Card | UnsavedCard,
+  {
+    includeDatasetQuery = true,
+    includeOriginalCardId = true,
+    includeDisplayIsLocked = false,
+    creationType,
+    parameterValues,
+  }: SerializeCardOptions = {},
+) {
+  const value = { ...card, creationType, parameterValues };
+  const keysToInclude = [
+    "collection_id",
+    "dashboard_id",
+    "dashboardId",
+    "dashcardId",
+    "description",
+    "display",
+    "name",
+    "parameters",
+    "type",
+    "visualization_settings",
+  ];
+
+  if (includeDatasetQuery) {
+    keysToInclude.push("dataset_query");
+  }
+  if (includeOriginalCardId) {
+    keysToInclude.push("original_card_id");
+  }
+  if (includeDisplayIsLocked) {
+    keysToInclude.push("displayIsLocked");
+  }
+  if (creationType) {
+    keysToInclude.push("creationType");
+  }
+  if (parameterValues) {
+    keysToInclude.push("parameterValues");
+  }
+
+  type Key = keyof typeof value;
+
+  const res: { [key in Key]?: unknown } = {};
+  for (const key of keysToInclude) {
+    // coerce to undefined to omit
+    res[key as Key] = value[key as Key] ?? undefined;
+  }
+
+  return res;
+}
+
+export function isEqualCard(card1?: Card | null, card2?: Card | null) {
+  if (card1 && card2) {
+    return equals(getCleanCard(card1), getCleanCard(card2));
+  } else {
+    return false;
+  }
+}
+
+// TODO Atte Keinänen 5/31/17 Deprecated, we should move tests to Questions.spec.js
+export function serializeCardForUrl(
+  card: Card | UnsavedCard,
+  options: SerializeCardOptions = {},
+) {
+  return utf8_to_b64url(stableStringify(getCleanCard(card, options)));
+}
+
+export function deserializeCardFromUrl(serialized: string): Card {
+  return JSON.parse(b64hash_to_utf8(serialized));
+}

frontend/src/metabase/lib/types.ts

@@ -4,6 +4,13 @@ export const isNotNull = <T>(value: T | null | undefined): value is T => {
   return value != null;
 };
 
+export const asNotNull = <T>(value: T | null | undefined): T => {
+  if (isNotNull(value)) {
+    return value;
+  }
+  throw new TypeError("value is null or undefined");
+};
+
 export const isNotFalsy = <T>(
   value: T | null | undefined | false | "",
 ): value is T => {

frontend/src/metabase/lib/urls/questions.ts

@@ -5,7 +5,11 @@ import MetabaseSettings from "metabase/lib/settings";
 import type { QuestionCreatorOpts } from "metabase-lib/v1/Question";
 import Question from "metabase-lib/v1/Question";
 import * as ML_Urls from "metabase-lib/v1/urls";
-import type { CardId, Card as SavedCard } from "metabase-types/api";
+import type {
+  CardId,
+  Card as SavedCard,
+  UnsavedCard,
+} from "metabase-types/api";
 import type { EntityToken } from "metabase-types/api/entity";
 
 import { appendSlug, getEncodedUrlSearchParams } from "./utils";
@@ -17,7 +21,7 @@ type Card = Partial<SavedCard> & {
 
 export type QuestionUrlBuilderParams = {
   mode?: "view" | "notebook" | "query";
-  hash?: Card | string;
+  hash?: SavedCard | UnsavedCard | string;
   query?: Record<string, unknown> | string;
   objectId?: number | string;
 };
@@ -42,21 +46,11 @@ export function question(
     objectId,
   }: QuestionUrlBuilderParams = {},
 ) {
-  if (hash && typeof hash === "object") {
-    hash = serializeCardForUrl(hash);
-  }
+  hash = encodeIfNeeded(hash, serializeCardForUrl);
+  hash = prefixIfNeeded(hash, "#");
 
-  if (query && typeof query === "object") {
-    query = String(getEncodedUrlSearchParams(query));
-  }
-
-  if (hash && hash.charAt(0) !== "#") {
-    hash = "#" + hash;
-  }
-
-  if (query && query.charAt(0) !== "?") {
-    query = "?" + query;
-  }
+  query = encodeIfNeeded(query, getEncodedUrlSearchParams);
+  query = prefixIfNeeded(query, "?");
 
   const isModel = card?.type === "model" || card?.model === "dataset";
   const fallbackPath = isModel ? "model" : "question";
@@ -101,7 +95,24 @@ export function question(
   return `${path}${query}${hash}`;
 }
 
-export function serializedQuestion(card: Card, opts = {}) {
+function encodeIfNeeded<T extends object>(
+  value: T | string,
+  fn: (value: T) => string,
+) {
+  if (typeof value === "string") {
+    return value;
+  }
+  return fn(value);
+}
+
+function prefixIfNeeded(value: string, prefix: string) {
+  if (value === "" || value.startsWith(prefix)) {
+    return value;
+  }
+  return `${prefix}${value}`;
+}
+
+export function serializedQuestion(card: SavedCard | UnsavedCard, opts = {}) {
   return question(null, { ...opts, hash: card });
 }
 

frontend/src/metabase/nav/components/AppSwitcher/AppSwitcher.tsx

@@ -8,6 +8,7 @@ import { trackErrorDiagnosticModalOpened } from "metabase/common/components/Erro
 import { ExternalLink } from "metabase/common/components/ExternalLink";
 import { ForwardRefLink } from "metabase/common/components/Link";
 import { userInitials } from "metabase/common/utils/user";
+import { trackDataStudioOpened } from "metabase/data-studio/analytics";
 import { canAccessDataStudio as canAccessDataStudioSelector } from "metabase/data-studio/selectors";
 import {
   getCanAccessOnboardingPage,
@@ -99,6 +100,8 @@ export const AppSwitcher = ({ className }: { className?: string }) => {
           key="data-studio-app-link"
           component={ForwardRefLink}
           to={Urls.dataStudio()}
+          onAuxClick={trackDataStudioOpened}
+          onClickCapture={trackDataStudioOpened}
           leftSection={
             <Icon
               name="table"

frontend/src/metabase/notifications/AddEditSidebar/DefaultParametersSection.tsx

@@ -1,15 +1,25 @@
 import cx from "classnames";
-import PropTypes from "prop-types";
 import { t } from "ttag";
 
 import { Badge } from "metabase/common/components/Badge";
 import CS from "metabase/css/core/index.css";
 import { conjunct } from "metabase/lib/formatting";
+import { isNotNull } from "metabase/lib/types";
 import { formatDateValue } from "metabase/parameters/utils/date-formatting";
 import { Icon, Title } from "metabase/ui";
+import type { Parameter, ParameterValueOrArray } from "metabase-types/api";
+
+interface FormattedParam {
+  name: string;
+  value: string;
+}
+
+function toStringArray(value: ParameterValueOrArray): string[] {
+  return (Array.isArray(value) ? value : [value]).map(String);
+}
 
 // TODO: will need improved formatting for operator parameter filters
-function formatDefaultParamValues(parameters) {
+function formatDefaultParamValues(parameters: Parameter[]): FormattedParam[] {
   return parameters
     .map((parameter) => {
       const { name, type, default: defaultValue } = parameter;
@@ -20,16 +30,15 @@ function formatDefaultParamValues(parameters) {
 
       let formattedValue;
       if (type.startsWith("date/")) {
-        const values = [].concat(defaultValue);
-        const formattedValues = values
+        const formattedValues = toStringArray(defaultValue)
           .map((val) => formatDateValue(parameter, val))
-          .filter(Boolean);
+          .filter(isNotNull);
 
         if (formattedValues.length > 0) {
           formattedValue = conjunct(formattedValues, t`and`);
         }
       } else {
-        formattedValue = conjunct([].concat(defaultValue), t`and`);
+        formattedValue = conjunct(toStringArray(defaultValue), t`and`);
       }
 
       if (formattedValue) {
@@ -37,10 +46,18 @@ function formatDefaultParamValues(parameters) {
       }
       return null;
     })
-    .filter(Boolean);
+    .filter(isNotNull);
 }
 
-function DefaultParametersSection({ className, parameters }) {
+interface DefaultParametersSectionProps {
+  className?: string;
+  parameters: Parameter[];
+}
+
+function DefaultParametersSection({
+  className,
+  parameters,
+}: DefaultParametersSectionProps) {
   const formattedParameterValues = formatDefaultParamValues(parameters);
 
   return (
@@ -73,10 +90,5 @@ function DefaultParametersSection({ className, parameters }) {
   );
 }
 
-DefaultParametersSection.propTypes = {
-  className: PropTypes.string,
-  parameters: PropTypes.array.isRequired,
-};
-
 // eslint-disable-next-line import/no-default-export -- deprecated usage
 export default DefaultParametersSection;

frontend/src/metabase/parameters/components/widgets/ParameterFieldWidget/FieldValuesWidget/SingleSelectListField/SingleSelectListField.unit.spec.tsx

@@ -2,23 +2,30 @@ import { waitForElementToBeRemoved } from "@testing-library/react";
 import userEvent from "@testing-library/user-event";
 
 import { render, screen } from "__support__/ui";
+import type { OptionsType } from "metabase/lib/formatting";
+import type Field from "metabase-lib/v1/metadata/Field";
+import type { FieldValue, RowValue } from "metabase-types/api";
 
 import { Value as ValueComponent } from "../../Value";
 
 import SingleSelectListField from "./index";
 
-const value = [];
+const value: RowValue[] = [];
 const firstOption = "AK";
 const secondOption = "AL";
-const options = [[firstOption], [secondOption]];
-const fields = [];
+const options: FieldValue[] = [[firstOption], [secondOption]];
+const fields: Field[] = [];
 const formatOptions = {};
 
-function showRemapping(fields) {
+function showRemapping(fields: Field[]) {
   return fields.length === 1;
 }
 
-function renderValue(fields, formatOptions, value, options) {
+function renderValue(
+  fields: Field[],
+  formatOptions: Partial<OptionsType>,
+  value: unknown,
+) {
   return (
     <ValueComponent
       value={value}

frontend/src/metabase/parameters/components/widgets/ParameterFieldWidget/FieldValuesWidget/SingleSelectListField/types.ts

@@ -8,7 +8,7 @@ export interface SingleSelectListFieldProps {
   options: Option[];
   optionRenderer: (option: Option) => JSX.Element;
   onSearchChange?: (value: string) => void;
-  placeholder: string;
+  placeholder?: string;
   isDashboardFilter?: boolean;
   isLoading?: boolean;
   checkedColor?: string;

frontend/src/metabase/parameters/components/widgets/ParameterFieldWidget/FieldValuesWidget/utils.ts

@@ -46,7 +46,7 @@ export function shouldList({
 }: {
   parameter?: Parameter;
   fields: Field[];
-  disableSearch: boolean;
+  disableSearch?: boolean;
 }) {
   if (disableSearch) {
     return false;
@@ -90,10 +90,10 @@ function getNonSearchableTokenFieldPlaceholder(
 }
 
 export function searchField(
-  field: Field,
-  disablePKRemappingForSearch: boolean,
+  field: Field | null,
+  disablePKRemappingForSearch?: boolean,
 ) {
-  return field.searchField(disablePKRemappingForSearch);
+  return field?.searchField(disablePKRemappingForSearch) ?? null;
 }
 
 function getSearchableTokenFieldPlaceholder(
@@ -172,8 +172,8 @@ export function isSearchable({
   valuesMode,
 }: {
   parameter?: Parameter;
-  fields: Field[];
-  disableSearch: boolean;
+  fields?: Field[];
+  disableSearch?: boolean;
   disablePKRemappingForSearch?: boolean;
   valuesMode?: ValuesMode;
 }) {
@@ -183,8 +183,10 @@ export function isSearchable({
     return true;
   } else if (parameter) {
     return canSearchParameterValues(parameter, disablePKRemappingForSearch);
-  } else {
+  } else if (fields) {
     return canSearchFieldValues(fields, disablePKRemappingForSearch);
+  } else {
+    return false;
   }
 }
 
@@ -248,7 +250,7 @@ export function getValuesMode({
 }: {
   parameter?: Parameter;
   fields: Field[];
-  disableSearch: boolean;
+  disableSearch?: boolean;
   disablePKRemappingForSearch?: boolean;
 }): ValuesMode {
   if (

frontend/src/metabase/parameters/components/widgets/ParameterFieldWidget/FieldValuesWidget/utils.unit.spec.tsx

@@ -1,3 +1,5 @@
+import { asNotNull } from "metabase/lib/types";
+import type { FieldId } from "metabase-types/api";
 import { ORDERS, PEOPLE, PRODUCTS } from "metabase-types/api/mocks/presets";
 
 import {
@@ -7,15 +9,14 @@ import {
 } from "./testMocks.spec";
 import { getValuesMode, isSearchable, searchField } from "./utils";
 
+const getField = (id: FieldId) => asNotNull(metadata.field(id));
+
 describe("Components > FieldValuesWidget > utils", () => {
   describe("isSearchable", () => {
-    const listField = metadata.field(PRODUCTS.CATEGORY);
-    const searchField = metadata.field(PEOPLE.EMAIL);
-    const nonExhaustiveListField = metadata.field(
-      LISTABLE_FIELD_WITH_MANY_VALUES_ID,
-    );
-
-    const idField = metadata.field(PRODUCTS.ID);
+    const listField = getField(PRODUCTS.CATEGORY);
+    const searchField = getField(PEOPLE.EMAIL);
+    const nonExhaustiveListField = getField(LISTABLE_FIELD_WITH_MANY_VALUES_ID);
+    const idField = getField(PRODUCTS.ID);
 
     describe("when the `valuesMode` is already set to 'search'", () => {
       it("should return return true unless fully disabled", () => {
@@ -58,21 +59,21 @@ describe("Components > FieldValuesWidget > utils", () => {
 
     describe("when passed fields that are searchable", () => {
       it("should return 'search'", () => {
-        const fields = [metadata.field(PEOPLE.EMAIL)];
+        const fields = [getField(PEOPLE.EMAIL)];
         expect(getValuesMode({ fields })).toBe("search");
       });
     });
 
     describe("when passed fields that are not searchable but listable", () => {
       it("should return 'list'", () => {
-        const fields = [metadata.field(PRODUCTS.CATEGORY)];
+        const fields = [getField(PRODUCTS.CATEGORY)];
         expect(getValuesMode({ fields })).toBe("list");
       });
     });
 
     describe("when passed fields that are not searchable and not listable", () => {
       it("should return 'none'", () => {
-        const fields = [metadata.field(ORDERS.SUBTOTAL)];
+        const fields = [getField(ORDERS.SUBTOTAL)];
         expect(getValuesMode({ fields })).toBe("none");
       });
     });
@@ -82,8 +83,8 @@ describe("Components > FieldValuesWidget > utils", () => {
     describe("`disablePKRemappingForSearch` is true and field is a PK", () => {
       const disablePKRemappingForSearch = true;
 
-      const stringPKField = metadata.field(STRING_PK_FIELD_ID);
-      const numberPKField = metadata.field(PRODUCTS.ID);
+      const stringPKField = getField(STRING_PK_FIELD_ID);
+      const numberPKField = getField(PRODUCTS.ID);
 
       it("should return same field when the field is searchable (the field is a string AND a PK)", () => {
         expect(searchField(stringPKField, disablePKRemappingForSearch)).toBe(
@@ -99,8 +100,8 @@ describe("Components > FieldValuesWidget > utils", () => {
     });
 
     describe("when the field is remapped to a searchable field", () => {
-      const stringField = metadata.field(PRODUCTS.TITLE);
-      const remappedField = metadata.field(PRODUCTS.CATEGORY).clone();
+      const stringField = getField(PRODUCTS.TITLE);
+      const remappedField = getField(PRODUCTS.CATEGORY).clone();
       remappedField.remappedExternalField = () => stringField;
 
       it("should return the remapped field", () => {
@@ -110,12 +111,12 @@ describe("Components > FieldValuesWidget > utils", () => {
 
     describe("when the field is remapped to a non-searchable field", () => {
       it("should ignore it and return the original field, assuming it is searchable", () => {
-        const numberField = metadata.field(ORDERS.TOTAL);
+        const numberField = getField(ORDERS.TOTAL);
 
-        const remappedField = metadata.field(PRODUCTS.CATEGORY).clone();
+        const remappedField = getField(PRODUCTS.CATEGORY).clone();
         remappedField.remappedExternalField = () => numberField;
 
-        const nonSearchableRemappedField = metadata.field(PRODUCTS.ID);
+        const nonSearchableRemappedField = getField(PRODUCTS.ID);
         nonSearchableRemappedField.remappedExternalField = () => numberField;
 
         expect(searchField(remappedField)).toBe(remappedField);
@@ -124,12 +125,12 @@ describe("Components > FieldValuesWidget > utils", () => {
     });
 
     it("should return the field if it is searchable", () => {
-      const searchableField = metadata.field(PRODUCTS.TITLE);
+      const searchableField = getField(PRODUCTS.TITLE);
       expect(searchField(searchableField)).toBe(searchableField);
     });
 
     it("should return null if the field is not searchable", () => {
-      const nonSearchableField = metadata.field(PRODUCTS.ID);
+      const nonSearchableField = getField(PRODUCTS.ID);
       expect(searchField(nonSearchableField)).toBeNull();
     });
   });

frontend/src/metabase/parameters/components/widgets/ParameterFieldWidget/ParameterFieldWidgetValue/ParameterFieldWidgetValue.tsx

@@ -8,7 +8,7 @@ import { normalizeValue } from "../normalizeValue";
 type ParameterFieldWidgetValueProps = {
   value: unknown;
   fields: Field[];
-  parameter: Parameter;
+  parameter?: Parameter;
   cardId?: CardId;
   dashboardId?: DashboardId;
   displayValue?: string;

frontend/src/metabase/parameters/components/widgets/ParameterFieldWidget/Value/RemappedValue.tsx

@@ -1,13 +1,39 @@
-/* eslint-disable react/prop-types */
+import type { ReactNode } from "react";
+
 import CS from "metabase/css/core/index.css";
 import AutoLoadRemapped from "metabase/hoc/Remapped";
 import { formatValue } from "metabase/lib/formatting";
+import type Field from "metabase-lib/v1/metadata/Field";
 
-const defaultRenderNormal = ({ value }) => <span>{value}</span>;
+type RenderNormal = (opts: { value?: unknown; column?: Field }) => ReactNode;
+type RenderRemapped = (opts: {
+  value: unknown;
+  column?: Field;
+  displayValue?: unknown;
+  displayColumn?: Field;
+}) => ReactNode;
 
-const defaultRenderRemapped = ({ value, displayValue, column }) => (
+export type RemappedValueProps = {
+  value: unknown;
+  column?: Field;
+  displayValue: unknown;
+  displayColumn?: Field;
+  renderNormal?: RenderNormal;
+  renderRemapped?: RenderRemapped;
+  autoLoad?: boolean;
+};
+
+const defaultRenderNormal: RenderNormal = ({ value }) => (
+  <span>{value as ReactNode}</span>
+);
+
+const defaultRenderRemapped: RenderRemapped = ({
+  value,
+  displayValue,
+  column,
+}) => (
   <span>
-    <span className={CS.textBold}>{displayValue}</span>
+    <span className={CS.textBold}>{displayValue as ReactNode}</span>
     {/* Show the underlying ID for PK/FK */}
     {column?.isID() && <span style={{ opacity: 0.5 }}>{" - " + value}</span>}
   </span>
@@ -21,11 +47,11 @@ const RemappedValueContent = ({
   renderNormal = defaultRenderNormal,
   renderRemapped = defaultRenderRemapped,
   ...props
-}) => {
+}: Omit<RemappedValueProps, "autoLoad">) => {
   if (column != null) {
     value = formatValue(value, {
       ...props,
-      column: column,
+      column,
       jsx: true,
       remap: false,
     });
@@ -47,14 +73,16 @@ const RemappedValueContent = ({
 
 export const AutoLoadRemappedValue = AutoLoadRemapped(RemappedValueContent);
 
-export const FieldRemappedValue = (props) => (
+export const FieldRemappedValue = (props: RemappedValueProps) => (
   <RemappedValueContent
     {...props}
-    displayValue={props.displayValue ?? props.column.remappedValue(props.value)}
+    displayValue={
+      props.displayValue ?? props.column?.remappedValue(props.value)
+    }
   />
 );
 
-const RemappedValue = ({ autoLoad = true, ...props }) =>
+const RemappedValue = ({ autoLoad = true, ...props }: RemappedValueProps) =>
   autoLoad && !props.displayValue ? (
     <AutoLoadRemappedValue {...props} />
   ) : (

frontend/src/metabase/parameters/components/widgets/ParameterFieldWidget/normalizeValue.ts

@@ -1,7 +1,15 @@
-export function normalizeValue(value: unknown) {
+type NonArray<T> = Exclude<T, any[]>;
+
+export function normalizeValue<T>(
+  value: NonArray<T> | NonArray<T>[] | undefined,
+): T[] {
   if (Array.isArray(value)) {
-    return value;
+    return value as T[];
   }
 
-  return value || value === 0 ? [value] : [];
+  if (value || value === 0) {
+    return [value];
+  }
+
+  return [];
 }

frontend/src/metabase/parameters/components/widgets/ParameterFieldWidget/normalizeValue.unit.spec.tsx

@@ -3,11 +3,10 @@ import { normalizeValue } from "./normalizeValue";
 describe("normalizeValue", () => {
   it("returns empty array if value is null", () => {
     const value = null;
-    const expected = [];
 
     const normalized = normalizeValue(value);
 
-    expect(normalized).toEqual(expected);
+    expect(normalized).toEqual([]);
   });
 
   it("returns value if value is an array", () => {

frontend/src/metabase/parameters/utils/dashboard-options.unit.spec.tsx

@@ -1,5 +1,7 @@
 import _ from "underscore";
 
+import type { ParameterSectionId } from "metabase-lib/v1/parameters/utils/operators";
+
 import { getDashboardParameterSections } from "./dashboard-options";
 
 describe("parameters/utils/dashboard-options", () => {
@@ -15,7 +17,7 @@ describe("parameters/utils/dashboard-options", () => {
         id: "location",
       });
       expect(
-        locationSection.options.every((option) =>
+        locationSection?.options.every((option) =>
           option.type.startsWith("string"),
         ),
       ).toBe(true);
@@ -23,7 +25,9 @@ describe("parameters/utils/dashboard-options", () => {
 
     it("should have a string section", () => {
       expect(
-        _.findWhere(getDashboardParameterSections(), { id: "category" }),
+        _.findWhere(getDashboardParameterSections(), {
+          id: "bad-section-id" as ParameterSectionId,
+        }),
       ).not.toBeDefined();
 
       expect(

frontend/src/metabase/parameters/utils/dashboards.unit.spec.tsx

@@ -7,8 +7,20 @@ import {
   hasMatchingParameters,
   setParameterName,
 } from "metabase/parameters/utils/dashboards";
+import * as Lib from "metabase-lib";
+import { SAMPLE_PROVIDER } from "metabase-lib/test-helpers";
 import Question from "metabase-lib/v1/Question";
 import Field from "metabase-lib/v1/metadata/Field";
+import { createMockUiParameter } from "metabase-lib/v1/parameters/mock";
+import type { UiParameter } from "metabase-lib/v1/parameters/types";
+import type { Parameter } from "metabase-types/api";
+import {
+  createMockCard,
+  createMockDashboard,
+  createMockDashboardCard,
+  createMockParameter,
+  createMockParameterMapping,
+} from "metabase-types/api/mocks";
 import {
   PRODUCTS,
   createSampleDatabase,
@@ -84,7 +96,12 @@ describe("metabase/parameters/utils/dashboards", () => {
 
   describe("setParameterName", () => {
     it("should set a name and a slug on parameter", () => {
-      expect(setParameterName({ abc: 123 }, "foo")).toEqual({
+      expect(
+        // @ts-expect-error: extra options like abc are preserved
+        setParameterName(createMockParameter({ abc: 123 }), "foo"),
+      ).toEqual({
+        id: expect.any(String),
+        type: expect.any(String),
         abc: 123,
         name: "foo",
         slug: "foo",
@@ -92,7 +109,9 @@ describe("metabase/parameters/utils/dashboards", () => {
     });
 
     it("should not default", () => {
-      expect(setParameterName({}, "")).toEqual({
+      expect(setParameterName(createMockParameter(), "")).toEqual({
+        id: expect.any(String),
+        type: expect.any(String),
         name: "",
         slug: "",
       });
@@ -100,85 +119,85 @@ describe("metabase/parameters/utils/dashboards", () => {
   });
 
   describe("hasMapping", () => {
-    const parameter = { id: "foo" };
+    const parameter = createMockParameter({ id: "foo" });
 
     it("should return false when there are no cards on the dashboard", () => {
-      const dashboard = {
+      const dashboard = createMockDashboard({
         dashcards: [],
-      };
+      });
       expect(hasMapping(parameter, dashboard)).toBe(false);
     });
 
     it("should return false when there are no cards with parameter mappings", () => {
-      const dashboard = {
+      const dashboard = createMockDashboard({
         dashcards: [
-          {
+          createMockDashboardCard({
             parameter_mappings: [],
-          },
-          {
+          }),
+          createMockDashboardCard({
             parameter_mappings: [],
-          },
+          }),
         ],
-      };
+      });
 
       expect(hasMapping(parameter, dashboard)).toBe(false);
     });
 
     it("should return false when missing parameter mappings", () => {
-      const dashboard = {
-        dashcards: [{ parameter_mappings: undefined }],
-      };
+      const dashboard = createMockDashboard({
+        dashcards: [createMockDashboardCard({ parameter_mappings: undefined })],
+      });
       expect(hasMapping(parameter, dashboard)).toBe(false);
     });
 
     it("should return false when there are no matching parameter mapping parameter_ids", () => {
-      const dashboard = {
+      const dashboard = createMockDashboard({
         dashcards: [
-          {
+          createMockDashboardCard({
             parameter_mappings: [
-              {
+              createMockParameterMapping({
                 parameter_id: "bar",
-              },
+              }),
             ],
-          },
-          {
+          }),
+          createMockDashboardCard({
             parameter_mappings: [
-              {
+              createMockParameterMapping({
                 parameter_id: "baz",
-              },
-              {
+              }),
+              createMockParameterMapping({
                 parameter_id: "abc",
-              },
+              }),
             ],
-          },
+          }),
         ],
-      };
+      });
 
       expect(hasMapping(parameter, dashboard)).toBe(false);
     });
 
     it("should return true when the given parameter's id is found in a parameter_mappings object", () => {
-      const dashboard = {
+      const dashboard = createMockDashboard({
         dashcards: [
-          {
+          createMockDashboardCard({
             parameter_mappings: [
-              {
+              createMockParameterMapping({
                 parameter_id: "bar",
-              },
+              }),
             ],
-          },
-          {
+          }),
+          createMockDashboardCard({
             parameter_mappings: [
-              {
+              createMockParameterMapping({
                 parameter_id: "baz",
-              },
-              {
+              }),
+              createMockParameterMapping({
                 parameter_id: "foo",
-              },
+              }),
             ],
-          },
+          }),
         ],
-      };
+      });
 
       expect(hasMapping(parameter, dashboard)).toBe(true);
     });
@@ -186,21 +205,21 @@ describe("metabase/parameters/utils/dashboards", () => {
 
   describe("hasMatchingParameters", () => {
     it("should return false when the given card is not found on the dashboard", () => {
-      const dashboard = {
+      const dashboard = createMockDashboard({
         dashcards: [
-          {
+          createMockDashboardCard({
             id: 1,
             card_id: 123,
-            card: { id: 123 },
+            card: createMockCard({ id: 123 }),
             parameter_mappings: [
-              {
+              createMockParameterMapping({
                 card_id: 123,
                 parameter_id: "foo",
-              },
+              }),
             ],
-          },
+          }),
         ],
-      };
+      });
 
       expect(
         hasMatchingParameters({
@@ -208,7 +227,6 @@ describe("metabase/parameters/utils/dashboards", () => {
           dashcardId: 1,
           cardId: 456,
           parameters: [],
-          metadata,
         }),
       ).toBe(false);
 
@@ -218,38 +236,37 @@ describe("metabase/parameters/utils/dashboards", () => {
           dashcardId: 2,
           cardId: 123,
           parameters: [],
-          metadata,
         }),
       ).toBe(false);
     });
 
     it("should return false when a given parameter is not found in the dashcard mappings", () => {
-      const dashboard = {
+      const dashboard = createMockDashboard({
         dashcards: [
-          {
+          createMockDashboardCard({
             id: 1,
             card_id: 123,
-            card: { id: 123 },
+            card: createMockCard({ id: 123 }),
             parameter_mappings: [
-              {
+              createMockParameterMapping({
                 card_id: 123,
                 parameter_id: "foo",
-              },
+              }),
             ],
-          },
-          {
+          }),
+          createMockDashboardCard({
             id: 2,
             card_id: 456,
-            card: { id: 456 },
+            card: createMockCard({ id: 456 }),
             parameter_mappings: [
-              {
+              createMockParameterMapping({
                 card_id: 456,
                 parameter_id: "bar",
-              },
+              }),
             ],
-          },
+          }),
         ],
-      };
+      });
 
       expect(
         hasMatchingParameters({
@@ -257,38 +274,37 @@ describe("metabase/parameters/utils/dashboards", () => {
           dashcardId: 1,
           cardId: 123,
           parameters: [
-            {
+            createMockParameter({
               id: "foo",
-            },
-            {
+            }),
+            createMockParameter({
               id: "bar",
-            },
+            }),
           ],
-          metadata,
         }),
       ).toBe(false);
     });
 
     it("should return true when all given parameters are found mapped to the dashcard", () => {
-      const dashboard = {
+      const dashboard = createMockDashboard({
         dashcards: [
-          {
+          createMockDashboardCard({
             id: 1,
             card_id: 123,
-            card: { id: 123 },
+            card: createMockCard({ id: 123 }),
             parameter_mappings: [
-              {
+              createMockParameterMapping({
                 card_id: 123,
                 parameter_id: "foo",
-              },
-              {
+              }),
+              createMockParameterMapping({
                 card_id: 123,
                 parameter_id: "bar",
-              },
+              }),
             ],
-          },
+          }),
         ],
-      };
+      });
 
       expect(
         hasMatchingParameters({
@@ -296,48 +312,47 @@ describe("metabase/parameters/utils/dashboards", () => {
           dashcardId: 1,
           cardId: 123,
           parameters: [
-            {
+            createMockParameter({
               id: "foo",
-            },
-            {
+            }),
+            createMockParameter({
               id: "bar",
-            },
+            }),
           ],
-          metadata,
         }),
       ).toBe(true);
     });
   });
 
   describe("getFilteringParameterValuesMap", () => {
-    const undefinedFilteringParameters = {};
-    const emptyFilteringParameters = {
+    const undefinedFilteringParameters = {} as UiParameter;
+    const emptyFilteringParameters = createMockUiParameter({
       filteringParameters: [],
-    };
+    });
 
-    const parameter = {
+    const parameter = createMockUiParameter({
       filteringParameters: ["a", "b", "c", "d"],
-    };
-    const parameters = [
-      {
+    });
+    const parameters: Parameter[] = [
+      createMockParameter({
         id: "a",
         value: "aaa",
-      },
-      {
+      }),
+      createMockParameter({
         id: "b",
         value: "bbb",
-      },
-      {
+      }),
+      createMockParameter({
         id: "c",
-      },
-      {
+      }),
+      createMockParameter({
         id: "d",
         value: null,
-      },
-      {
+      }),
+      createMockParameter({
         id: "e",
         value: "eee",
-      },
+      }),
     ];
 
     it("should create a map of any defined parameterValues found in a specific parameter's filteringParameters property", () => {
@@ -370,19 +385,19 @@ describe("metabase/parameters/utils/dashboards", () => {
   });
 
   describe("getDashboardUiParameters", () => {
-    const dashboard = {
+    const dashboard = createMockDashboard({
       id: 1,
       dashcards: [
-        {
+        createMockDashboardCard({
           id: 1,
           card_id: 123,
-          card: { id: 123, dataset_query: { type: "query" } },
-          series: [{ id: 789, dataset_query: { type: "query" } }],
+          card: createMockCard({ id: 123 }),
+          series: [createMockCard({ id: 789 })],
           parameter_mappings: [
             {
               card_id: 123,
               parameter_id: "b",
-              target: ["breakout", 0],
+              target: ["dimension", ["field", PRODUCTS.RATING, null]],
             },
             {
               card_id: 789,
@@ -400,51 +415,49 @@ describe("metabase/parameters/utils/dashboards", () => {
               target: ["dimension", ["field", PRODUCTS.TITLE, null]],
             },
           ],
-        },
-        {
+        }),
+        createMockDashboardCard({
           id: 2,
           card_id: 456,
-          card: {
+          card: createMockCard({
             id: 456,
-            dataset_query: {
-              type: "native",
-              native: {
+            dataset_query: Lib.toJsQuery(
+              Lib.createTestNativeQuery(SAMPLE_PROVIDER, {
                 query: "{{foo}}",
-                "template-tags": {
+                templateTags: {
                   foo: {
                     type: "text",
                   },
                   bar: {
                     type: "dimension",
                     "widget-type": "string/contains",
-                    dimension: ["field", PRODUCTS.TITLE, null],
+                    dimension: PRODUCTS.TITLE,
                   },
                 },
-              },
-            },
-          },
+              }),
+            ),
+          }),
           parameter_mappings: [
-            {
+            createMockParameterMapping({
               card_id: 456,
               parameter_id: "e",
-              target: ["variable", "foo"],
-            },
-            {
+            }),
+            createMockParameterMapping({
               card_id: 456,
               parameter_id: "f",
               target: ["dimension", ["template-tag", "bar"]],
-            },
-            {
+            }),
+            createMockParameterMapping({
               card_id: 456,
               parameter_id: "h",
-              target: ["variable", "foo"],
-            },
+              target: ["variable", ["template-tag", "foo"]],
+            }),
           ],
-        },
-        {
+        }),
+        createMockDashboardCard({
           id: 3,
           card_id: 999,
-          card: { id: 999, dataset_query: { type: "query" } },
+          card: createMockCard({ id: 999 }),
           parameter_mappings: [
             {
               card_id: 999,
@@ -457,67 +470,75 @@ describe("metabase/parameters/utils/dashboards", () => {
               target: ["dimension", ["field", PRODUCTS.CATEGORY, null]],
             },
           ],
-        },
-        {
+        }),
+        createMockDashboardCard({
           id: 4,
           card_id: 888,
-          card: { id: 888, dataset_query: { type: "query" } },
+          card: createMockCard({ id: 888 }),
           parameter_mappings: [],
-        },
+        }),
       ],
       parameters: [
         // unmapped, not field filter
-        {
+        createMockParameter({
           id: "a",
+          name: "a",
           slug: "slug-a",
           type: "foo",
-        },
+        }),
         // mapped, not field filter
-        {
+        createMockParameter({
           id: "b",
+          name: "b",
           slug: "slug-b",
           type: "granularity",
           default: ["day"],
-        },
+        }),
         // unmapped, field filter
-        {
+        createMockParameter({
           id: "c",
+          name: "c",
           slug: "slug-c",
           type: "string/=",
-        },
+        }),
         // mapped, field filter
-        {
+        createMockParameter({
           id: "d",
+          name: "d",
           slug: "slug-d",
           type: "number/=",
           default: [1, 2, 3],
-        },
+        }),
         // mapped to variable, field filter
-        {
+        createMockParameter({
           id: "e",
+          name: "e",
           slug: "slug-e",
           type: "category",
-        },
+        }),
         // field filter, mapped to two cards, same field
-        {
+        createMockParameter({
           id: "f",
+          name: "f",
           slug: "slug-f",
           type: "string/contains",
-        },
+        }),
         // field filter, mapped to two, different fields
-        {
+        createMockParameter({
           id: "g",
+          name: "g",
           slug: "slug-g",
           type: "string/starts-with",
-        },
+        }),
         // field filter, mapped to field and variable
-        {
+        createMockParameter({
           id: "h",
+          name: "h",
           slug: "slug-h",
           type: "string/=",
-        },
+        }),
       ],
-    };
+    });
 
     it("should return a list of UiParameter objects from the given dashboard", () => {
       const questions = Object.fromEntries(
@@ -536,17 +557,20 @@ describe("metabase/parameters/utils/dashboards", () => {
       ).toEqual([
         {
           id: "a",
+          name: "a",
           slug: "slug-a",
           type: "foo",
         },
         {
           id: "b",
+          name: "b",
           slug: "slug-b",
           type: "granularity",
           default: ["day"],
         },
         {
           id: "c",
+          name: "c",
           slug: "slug-c",
           type: "string/=",
           fields: [],
@@ -554,6 +578,7 @@ describe("metabase/parameters/utils/dashboards", () => {
         },
         {
           id: "d",
+          name: "d",
           slug: "slug-d",
           type: "number/=",
           default: [1, 2, 3],
@@ -562,6 +587,7 @@ describe("metabase/parameters/utils/dashboards", () => {
         },
         {
           id: "e",
+          name: "e",
           slug: "slug-e",
           type: "category",
           fields: [],
@@ -569,6 +595,7 @@ describe("metabase/parameters/utils/dashboards", () => {
         },
         {
           id: "f",
+          name: "f",
           slug: "slug-f",
           type: "string/contains",
           fields: [expect.any(Field)],
@@ -576,6 +603,7 @@ describe("metabase/parameters/utils/dashboards", () => {
         },
         {
           id: "g",
+          name: "g",
           slug: "slug-g",
           type: "string/starts-with",
           fields: [expect.any(Field), expect.any(Field)],
@@ -583,6 +611,7 @@ describe("metabase/parameters/utils/dashboards", () => {
         },
         {
           id: "h",
+          name: "h",
           slug: "slug-h",
           type: "string/=",
           fields: [expect.any(Field)],