add split openapi to the repo

* Convert parameters to TypeScript

* Fix types

* Fix types in normalizeValue

* Fix types in dashboard tests

* Fix more types

* Fix test

* Remove duplicate check

* Naming

.claude/skills/mutation-testing/SKILL.md

@@ -5,227 +5,110 @@ description: Run mutation testing on a Clojure namespace, generate tests to kill
 
 # Mutation Testing Skill
 
-This skill runs mutation testing on a target namespace, writes tests to kill surviving mutations, and creates draft PRs with Linear issues for each function.
+This skill runs mutation testing end-to-end: generates a coverage report, groups functions, shells out to `claude -p` to write tests, verifies mutations are killed, and creates draft PRs with Linear issues.
 
 ## Prerequisites
 
 - A running nREPL connected to the Metabase dev environment
 - `LINEAR_API_KEY` environment variable set with a valid Linear personal API key
 - `gh` CLI authenticated with GitHub
+- `claude` CLI available on PATH
 
 ## Reference Files
 
-- `dev/src/dev/coverage.clj` — mutation testing tool (generates reports, runs mutations)
-- `dev/src/dev/mutation_testing.clj` — Linear API client and PR template helpers
-- `mutation-testing-report.lib.card.md` — example report from the pilot
+- `dev/src/dev/coverage.clj` — mutation testing engine (generates reports, runs mutations)
+- `dev/src/dev/mutation_testing.clj` — orchestration, Linear API, PR helpers
 
 ## Invocation
 
-The argument is a Clojure namespace, e.g.:
+The argument is a Clojure namespace, optionally followed by a base branch:
 
 ```
 /mutation-testing metabase.lib.order-by
+/mutation-testing metabase.lib.order-by --base-branch release-x.52.x
+/mutation-testing metabase.lib.order-by --project-id abc-123
+/mutation-testing metabase.lib.order-by --base-branch release-x.52.x --project-id abc-123
 ```
 
-## Workflow
+- `--base-branch`: defaults to the value in config (set via `set-config!`), or `"master"` if not configured.
+- `--project-id`: if provided, issues are added to this existing Linear project instead of creating a new one.
 
-### Step 1: Parse the namespace
+## Steps
 
-From the argument (e.g., `metabase.lib.order-by`), derive:
-- **Test namespace**: append `-test` (e.g., `metabase.lib.order-by-test`)
-- **Short name**: the last segment (e.g., `order-by`)
-- **Source path**: `src/metabase/lib/order_by.cljc` (replace `-` with `_` in path segments)
-- **Test path**: `test/metabase/lib/order_by_test.cljc`
+### 1. Parse arguments
 
-### Step 2: Set up Linear and load tools
+Parse `$ARGUMENTS` to extract:
+- **namespace** (required) — the first positional argument
+- **`--base-branch`** (optional) — if present, the next argument is the base branch name
+- **`--project-id`** (optional) — if present, the next argument is an existing Linear project ID
+
+### 2. Load and configure
 
 ```clojure
-(require '[dev.coverage :as cov] :reload)
 (require '[dev.mutation-testing :as mut-test] :reload)
+(require '[dev.coverage :as cov] :reload)
 ```
 
 If this is the first invocation (or the REPL was restarted), set up the Linear team:
 
 ```clojure
-;; Find the team
 (mut-test/list-teams!)
-;; Set team ID from the output
-(mut-test/set-config! {:team-id "<team-id>"})
+(mut-test/set-config! {:team-id "<id>"})
 ```
 
-If config is already set from a previous invocation, skip this.
-
-### Step 3: Generate baseline report and create Linear project
+### 3. Run
 
 ```clojure
-(cov/generate-report
-  '<target-ns>
-  ['<test-ns>]
-  "mutation-testing-report.lib.<short-name>.before.md")
+(mut-test/run! '<target-ns>)
+;; Or with options:
+(mut-test/run! '<target-ns> {:base-branch "release-x.52.x"})
+(mut-test/run! '<target-ns> {:project-id "abc-123"})
+(mut-test/run! '<target-ns> {:base-branch "release-x.52.x" :project-id "abc-123"})
 ```
 
-Read the generated report file. It has three sections:
-- **Uncovered Functions** — never called by any test
-- **Partially Covered Functions** — called but with surviving mutations
-- **Fully Covered Functions** — all mutations killed
+Pass the opts map with `:base-branch` and/or `:project-id` if those flags were provided in the arguments.
 
-Then create a Linear project for this namespace (uses report stats in the description):
+This will:
+1. Generate a baseline mutation testing report
+2. Create a Linear project for the namespace
+3. Group functions by coverage relationships
+4. For each group: create branch → invoke Claude to write tests → verify → retry if needed → commit & push → create Linear issue → create draft PR
+5. Print a summary with PR links
+
+### 4. Handle failures
+
+If a group fails mid-processing, `run!` catches the error and continues to the next group. To retry a single group manually:
 
 ```clojure
-(mut-test/create-project-for-namespace!
-  "<target-ns>"
-  "mutation-testing-report.lib.<short-name>.before.md")
-;; => {:project-id "...", :name "Mutation Testing: metabase.lib.order-by"}
-;; Automatically sets :project-id in config for subsequent create-issue! calls
+;; Get the parsed namespace info
+(def parsed (mut-test/parse-namespace '<target-ns>))
+
+;; Run coverage to get the data
+(def coverage-results (cov/test-namespace (:target-ns parsed) [(:test-ns parsed)]))
+
+;; Group and find the one you want
+(def groups (mut-test/group-functions coverage-results))
+
+;; Process just that group
+(mut-test/process-group! parsed (nth groups <index>))
 ```
 
-### Step 4: Plan the work
+### 5. Verify the prompt (dry run)
 
-Before creating branches, analyze the report to plan how to group the work:
-
-1. **Identify killable functions.** For each function with surviving mutations, assess whether any mutations are killable. Run `cov/test-mutations` to see the full mutation list. If ALL mutations for a function are unkillable (schema-only, semantically equivalent, etc.), skip that function — no branch or PR needed.
-
-2. **Group private functions by public entry point.** Private functions cannot be tested directly. Group them with the public function that exercises them. Create one branch/PR per group, not per private function.
-
-3. **Track mutations per group.** The `:mutations-before` and `:not-killed` fields in the PR should only reflect mutations for functions covered by that specific PR — never mutations from other functions.
-
-### Step 5: Process each group
-
-For each group of functions that has killable mutations:
-
-#### 5a. Create a branch
+To inspect what Claude will see without invoking it:
 
 ```clojure
-(mut-test/create-branch! "<target-ns>" "<primary-fn-name>")
-;; => "mutation-testing-lib-order-by-orderable-columns"
+(def parsed (mut-test/parse-namespace '<target-ns>))
+(def coverage-results (cov/test-namespace (:target-ns parsed) [(:test-ns parsed)]))
+(def groups (mut-test/group-functions coverage-results))
+(println (mut-test/build-test-prompt
+           (merge (select-keys parsed [:target-ns :test-ns :source-path :test-path])
+                  (select-keys (first groups) [:fn-names :mutations]))))
 ```
 
-Use the primary public function name for the branch. If the group covers private functions too, name the branch after the public entry point.
-
-#### 5b. Read context
-
-- Read the source function(s) from the source file
-- Read the full test file to understand existing test patterns, helpers, and metadata providers used
-- If the function is private, identify which public function(s) call it
-
-#### 5c. Write tests
-
-Generate the **simplest tests** that kill the surviving mutations while still making semantic sense. Guidelines:
-
-- **Never call private functions directly.** Always test through public API functions. Use the coverage data to identify which public functions exercise the private function.
-- **Keep tests simple.** Each test should verify one meaningful behavior. Don't over-engineer tests just to chase mutation kills.
-- **Follow existing patterns.** Match the style of the existing tests: same metadata providers, same helper functions, same assertion patterns.
-- **Insert tests near related existing tests** for the same function, not at the end of the file. This minimizes merge conflicts between branches. If there are no existing tests for the function, find the most logical location based on the order of functions in the source file.
-
-**IMPORTANT: Use `file_edit` (not `clojure_edit`) when editing test files.** The `clojure_edit` tool reformats the entire file, introducing unnecessary whitespace changes that pollute the diff. Use `file_edit` with exact string matching to make surgical insertions that only touch the lines you intend to change.
-
-#### 5d. Verify mutations are killed
-
-Use the REPL to check that the new tests kill the targeted mutations:
-
-```clojure
-;; Reload the test namespace to pick up new tests
-(require '<test-ns> :reload)
-
-;; Test mutations for the specific function
-(cov/test-mutations
-  '<target-ns>/<fn-name>
-  ;; Set of test names that cover this function
-  #{'<test-ns>/<test-name-1> '<test-ns>/<test-name-2>})
-```
-
-Check the `:survived` key in the result. If mutations still survive:
-- Try writing additional tests
-- If a mutation is truly unkillable (semantically equivalent), note it for the PR description
-- If a mutation reveals dead/unreachable code, consider suggesting removal as a code improvement
-
-#### 5e. Handle unkillable mutations and improvements
-
-**Do NOT edit the source namespace just for documentation.** Instead:
-
-- **Unkillable mutations**: Note them in the PR description with rationale
-- **Code improvements** (dead code removal, expression simplification) that directly relate to surviving mutants:
-  1. Make the change in the source file
-  2. Verify all tests still pass with the change
-  3. Note the file path, line range, and the improved code
-  4. **Reset the change**: `git checkout -- <source-file>` — do NOT commit it
-  5. After creating the PR, use `add-suggested-change!` to post the improvement as a GitHub suggested change (Step 5h)
-  6. The reviewer decides whether to accept it
-
-#### 5f. Commit and push
-
-```clojure
-;; Only commit the test file — never commit source changes for improvements
-(mut-test/commit-and-push! "<target-ns>" "<primary-fn-name>"
-  ["test/metabase/lib/<short_name>_test.cljc"])
-```
-
-#### 5g. Create Linear issue and draft PR
-
-```clojure
-;; Create a Linear issue (use the primary function name)
-(def issue (mut-test/create-issue-for-function! "<target-ns>" "<primary-fn-name>"))
-;; => {:identifier "QUE-1234", :url "https://linear.app/...", ...}
-
-;; Create draft PR linked to the Linear issue
-;; :fn-names lists ALL functions covered by this PR
-;; :mutations-before counts only mutations for functions in this PR
-;; :not-killed lists only unkillable mutations for functions in this PR
-(def pr-url
-  (mut-test/create-draft-pr!
-    {:target-ns         "<target-ns>"
-     :fn-name           "<primary-fn-name>"
-     :linear-identifier (:identifier issue)
-     :mutations-before  9       ;; surviving mutations for these functions only
-     :tests-added       2       ;; number of new test functions
-     :killed            ["Replace :segment-id with :segment-id__"
-                         "Replace cycle-path with nil"]
-     :not-killed        [{:description "Replace acc with nil"
-                          :rationale   "reduce always has single iteration"}]
-     :suggested-changes []}))
-;; => "https://github.com/metabase/metabase/pull/12345"
-```
-
-#### 5h. Add code improvement suggestions (if any)
-
-If you identified code improvements in step 5e (and reset them), post each as a GitHub suggested change:
-
-```clojure
-(mut-test/add-suggested-change!
-  {:pr-url     pr-url
-   :path       "src/metabase/lib/<short_name>.cljc"
-   :start-line 42    ;; first line of the code to replace
-   :end-line   45    ;; last line of the code to replace
-   :suggestion "(improved-code-here)"  ;; the replacement code
-   :comment    "**Suggested improvement:** <description of the change and why it's related to the surviving mutant>"})
-```
-
-### Step 6: Return to master and repeat
-
-```clojure
-(mut-test/return-to-master!)
-```
-
-Repeat Step 5 for the next group.
-
-### Step 7: Summary
-
-Print a summary of what was done:
-- Link to the Linear project for this namespace
-- Number of functions processed
-- Number of functions skipped (all mutations unkillable)
-- Number of draft PRs created
-- Number of mutations killed vs. unkillable
-- List of test PRs with links
-- List of mutation rule PRs with links (if any were created)
-
-## Suggesting New Mutation Rules
-
-If you notice patterns in surviving mutations that suggest the mutation testing tool should have additional rules (e.g., a common Clojure form that isn't being mutated), open a separate PR proposing additions to `dev.coverage/mutation-rules` with an explanation.
-
 ## Configuration
 
-The `dev.mutation-testing` namespace reads `LINEAR_API_KEY` from the environment and stores team/project IDs in an atom.
-
-- **Team ID**: Set once per REPL session via `(mut-test/list-teams!)` and `(mut-test/set-config! {:team-id "..."})` (Step 2)
-- **Project ID**: Set automatically when `create-project-for-namespace!` is called (Step 3) — one project per namespace
+- **Team ID**: Set once per REPL session via `(mut-test/list-teams!)` and `(mut-test/set-config! {:team-id "..."})`
+- **Project ID**: Set automatically when `run!` calls `create-project-for-namespace!`
+- **Base Branch**: Defaults to `"master"`. Override via `(mut-test/set-config! {:base-branch "release-x.52.x"})` or pass `{:base-branch "..."}` as the second arg to `run!`
+- **Project ID**: If set (via `set-config!` or `run!` opts), `run!` reuses the existing Linear project instead of creating a new one

.clj-kondo/config/modules/config.edn

@@ -1597,7 +1597,6 @@
            util}
    :friends #{models
               enterprise/transforms-python
-              enterprise/transforms-runner
               enterprise/workspaces}}
 
   transforms-inspector
@@ -2258,7 +2257,7 @@
               util
               warehouses
               enterprise/dependencies
-           enterprise/transforms-runner}}
+              enterprise/transforms-python}}
 
   enterprise/permission-debug
   {:team "UX West"
@@ -2293,7 +2292,7 @@
            task-history
            util
            enterprise/serialization
-           enterprise/transforms-runner}}
+           enterprise/transforms-python}}
 
   enterprise/sandbox
   {:team "UX West"
@@ -2458,27 +2457,12 @@
    :api  #{}
    :uses #{premium-features}}
 
-  enterprise/transforms-clojure
-  {:team "Querying"
-   :api #{metabase-enterprise.transforms-clojure.init}
-   :uses #{transforms}}
-
-  enterprise/transforms-javascript
-  {:team "Querying"
-   :api #{metabase-enterprise.transforms-javascript.init}
-   :uses #{transforms}}
-
   enterprise/transforms-python
   {:team "Querying"
-   :api #{metabase-enterprise.transforms-python.init}
-   :uses #{transforms}}
-
-  enterprise/transforms-runner
-  {:team "Querying"
-   :api #{metabase-enterprise.transforms-runner.api
-          metabase-enterprise.transforms-runner.init
-          metabase-enterprise.transforms-runner.models.transform-library
-          metabase-enterprise.transforms-runner.runner}
+   :api  #{metabase-enterprise.transforms-python.api
+           metabase-enterprise.transforms-python.core
+           metabase-enterprise.transforms-python.init
+           metabase-enterprise.transforms-python.python-runner}
    :uses #{analytics
            api
            app-db
@@ -2526,7 +2510,7 @@
            system
            transforms
            util
-           enterprise/transforms-runner}}}
+           enterprise/transforms-python}}}
 
  ;; namespaces matching these patterns (with `re-find`) are excluded from the module linter. Since regex literals
  ;; aren't allowed in EDN just used the `(str <regex>)` version i.e. two slashes instead of one.

.github/workflows/e2e-test.yml

@@ -45,8 +45,9 @@ jobs:
     name: e2e-tests-${{ inputs.name }}-${{ inputs.edition }}
     env:
       MB_EDITION: ${{ inputs.edition }}
-      # Any env starting with `CYPRESS_` will be available to all Cypress tests via `Cypress.env()`
-      # Example: you can get `CYPRESS_FOO` with `Cypress.env("FOO")`
+      # Sensitive env vars (tokens, secrets) starting with `CYPRESS_` are accessed via async `cy.env()`
+      # Example: `cy.env(["FOO"]).then(({ FOO }) => ...)`
+      # For non-sensitive config, use `Cypress.expose()` in cypress config instead (synchronous)
       CYPRESS_MB_ALL_FEATURES_TOKEN: ${{ secrets.STAGING_MB_ALL_FEATURES_TOKEN }}
       CYPRESS_MB_STARTER_CLOUD_TOKEN: ${{ secrets.STAGING_MB_STARTER_CLOUD_TOKEN }}
       CYPRESS_MB_PRO_CLOUD_TOKEN: ${{ secrets.STAGING_MB_PRO_CLOUD_TOKEN }}

.github/workflows/embedding-sdk-component-tests.yml

@@ -19,8 +19,9 @@ jobs:
         react-version: [18, 19]
       fail-fast: false
     env:
-      # Any env starting with `CYPRESS_` will be available to all Cypress tests via `Cypress.env()`
-      # Example: you can get `CYPRESS_FOO` with `Cypress.env("FOO")`
+      # Sensitive env vars (tokens, secrets) starting with `CYPRESS_` are accessed via async `cy.env()`
+      # Example: `cy.env(["FOO"]).then(({ FOO }) => ...)`
+      # For non-sensitive config, use `Cypress.expose()` in cypress config instead (synchronous)
       CYPRESS_MB_ALL_FEATURES_TOKEN: ${{ secrets.STAGING_MB_ALL_FEATURES_TOKEN }}
       CYPRESS_MB_STARTER_CLOUD_TOKEN: ${{ secrets.STAGING_MB_STARTER_CLOUD_TOKEN }}
       CYPRESS_MB_PRO_CLOUD_TOKEN: ${{ secrets.STAGING_MB_PRO_CLOUD_TOKEN }}
@@ -155,8 +156,9 @@ jobs:
     env:
       MB_RUN_MODE: "e2e"
       METASTORE_DEV_SERVER_URL: ${{ secrets.METASTORE_DEV_SERVER_URL }}
-      # Any env starting with `CYPRESS_` will be available to all Cypress tests via `Cypress.env()`
-      # Example: you can get `CYPRESS_FOO` with `Cypress.env("FOO")`
+      # Sensitive env vars (tokens, secrets) starting with `CYPRESS_` are accessed via async `cy.env()`
+      # Example: `cy.env(["FOO"]).then(({ FOO }) => ...)`
+      # For non-sensitive config, use `Cypress.expose()` in cypress config instead (synchronous)
       CYPRESS_MB_ALL_FEATURES_TOKEN: ${{ secrets.STAGING_MB_ALL_FEATURES_TOKEN }}
       CYPRESS_MB_STARTER_CLOUD_TOKEN: ${{ secrets.STAGING_MB_STARTER_CLOUD_TOKEN }}
       CYPRESS_MB_PRO_CLOUD_TOKEN: ${{ secrets.STAGING_MB_PRO_CLOUD_TOKEN }}

.github/workflows/openapi.yml

@@ -3,8 +3,8 @@ name: Validate OpenAPI
 on:
   push:
     branches:
-      - 'master'
-      - 'release-**'
+      - "master"
+      - "release-**"
   pull_request:
 
 concurrency:
@@ -39,6 +39,8 @@ jobs:
       - name: Prepare back-end environment
         uses: ./.github/actions/prepare-backend
       - name: Generate OpenAPI spec
-        run: bun run generate-openapi
+        run: bun run openapi:generate
       - name: Validate OpenAPI spec
-        run: bun run lint-openapi
+        run: bun run openapi:lint
+      - name: Split OpenAPI spec
+        run: bun run openapi:split

.gitignore

@@ -170,5 +170,5 @@ CLAUDE.local.md
 # Python dependencies (pip-installed, not committed)
 /resources/python-sources/sqlglot/
 /resources/python-sources/sqlglot-*.dist-info/
+/resources/python-sources/.lock
 __pycache__/
-

bin/openapi-check.sh

@@ -14,7 +14,7 @@ fi
 # Generate the current OpenAPI spec from Malli schemas using write-openapi-spec-to-file!
 # This creates $OPENAPI_SPEC directly
 # Using the generate-openapi-spec command from metabase.cmd.core
-bun run generate-openapi
+bun run openapi:generate
 ret=$?
 
 if [[ $ret -ne 0 ]]; then
@@ -82,7 +82,7 @@ if ! git diff --exit-code "$OPENAPI_SPEC" >/dev/null 2>&1; then
         echo ""
         echo "To fix this:"
         echo "  1. Pull the latest changes from master: git pull origin master"
-        echo "  2. Run 'bun run generate-openapi' to update $OPENAPI_SPEC"
+        echo "  2. Run 'bun run openapi:generate' to update $OPENAPI_SPEC"
         echo "  3. Commit the updated openapi.json file"
         echo ""
         echo "Differences:"

dev/src/dev/debug_qp/analyze_user_query.clj

@@ -0,0 +1,157 @@
+(ns dev.debug-qp.analyze-user-query
+  "Given access to e.g. a Cloud instance with a querying issue, this script can use a login token borrowed from your
+  browser to make API calls to the instance, and assemble a local mock `MetadataProvider` which contains the tables,
+  fields and cards necessary to run QP preprocessing and compilation locally against the same metadata the user
+  instance is seeing.
+
+  Of course the queries can't be executed without a connection to the user's DWH, which is usually impossible."
+  (:require
+   [clj-http.client :as http]
+   [medley.core :as m]
+   [metabase.lib.js.metadata :as lib.js.metadata]
+   [metabase.lib.metadata :as lib.metadata]
+   [metabase.query-processor.compile :as qp.compile]
+   [metabase.query-processor.preprocess :as qp.preprocess]
+   [metabase.util.json :as json]
+   [metabase.util.malli :as mu]))
+
+;; XXX: INSTRUCTIONS:
+;; - Log into the instance, open DevTools, and navigate to the problem question.
+;; - Open the Network tab, find a `/api/card/1234` or similar request, right click and "Copy as cURL".
+;; - Paste that somewhere, probably into your shell to test that the `curl` call is working.
+;; - Grab its `cookie` and edit it into `my-ctx` below. (It's the `-b` arg to curl.)
+;; - Edit the `instance-url` of `my-ctx` below likewise.
+;; - Edit the `database-id` of `my-ctx` below to match the problem database on the user instance.
+;;   - That can be seen in eg. `/api/card` requests and similar.
+;; - Now you can run the analysis functions below, like compile-card.
+;;   - See the examples at the bottom of this file.
+(def ^:private my-ctx
+  "Contains instance details, for passing to the various functions below."
+  {:instance-url        "https://acustomer.metabaseapp.com"  ; No trailing slash.
+   :cookie-from-browser "COPY ME FROM YOUR curl COMMAND"
+   :database-id         123})
+
+(defn- mk-headers [{:keys [cookie-from-browser instance-url] :as _ctx}]
+  {"accept" "application/json"
+   "accept-language" "en-US,en;q=0.9"
+   "cache-control" "no-cache"
+   "content-type" "application/json"
+   "cookie" cookie-from-browser
+   "pragma" "no-cache"
+   "priority" "u=1, i"
+   "referer" instance-url
+   "sec-ch-ua" "\"Google Chrome\";v=\"141\", \"Not?A_Brand\";v=\"8\", \"Chromium\";v=\"141\""
+   "sec-ch-ua-mobile" "?0"
+   "sec-ch-ua-platform" "\"macOS\""
+   "sec-fetch-dest" "empty"
+   "sec-fetch-mode" "cors"
+   "sec-fetch-site" "same-origin"
+   "user-agent" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36"})
+
+(defn- instance-fetch [{:keys [instance-url] :as ctx} path]
+  (let [headers  (mk-headers ctx)
+        request  {:method           :get
+                  :url              (str instance-url path)
+                  :accept           :json
+                  :content-type     :json
+                  :throw-exceptions false
+                  :headers          headers}
+        response (-> (http/request request)
+                     (select-keys [:body :headers :status])
+                     (update :body json/decode))]
+    (if (<= 200 (:status response) 299)
+      (:body response)
+      (throw (ex-info "Non-200 response" response)))))
+
+(defn- qm-index [objs]
+  (m/index-by (comp str #(get % "id")) objs))
+
+(defn- qm->metadata [raw]
+  ;; /query_metadata requests return a map in a different shape from what lib.js.metadata expects.
+  ;; We get {"databases" [{...}], "tables" [{..., "fields": [{...}]}]} but we need
+  ;; {"databases" {"12" {...}}
+  ;;  "tables" {"333" {...}}
+  ;;  "fields" {"4343" {...}}
+  ;;  "questions" {"12" {...}}}
+  (let [fields (for [table (get raw "tables")
+                     :when (number? (get table "id"))
+                     field (get table "fields")
+                     :when (number? (get field "id"))]
+                 field)]
+    {"databases" (qm-index (get raw "databases"))
+     "tables"    (qm-index (get raw "tables"))
+     "fields"    (qm-index fields)}))
+
+(defn- recursive-card-metadata
+  "Helper for [[metadata-for]]. Prefer calling that function instead."
+  [ctx main-card-id]
+  (loop [metadata          {}
+         cards-seen        #{main-card-id}
+         [card-id & cards] [main-card-id]]
+    (if-not card-id
+      metadata
+      (let [card         (instance-fetch ctx (str "/api/card/" card-id))
+            raw          (instance-fetch ctx (str "/api/card/" card-id "/query_metadata"))
+            nested-cards (->> (get raw "tables")
+                              (map #(get % "id"))
+                              (filter string?)
+                              (map (comp Long/parseLong #(subs % 6)))
+                              (remove cards-seen))]
+        (recur (m/deep-merge metadata
+                             {"questions" {(str card-id) card}}
+                             (qm->metadata raw))
+               (into cards-seen nested-cards)
+               (concat cards nested-cards))))))
+
+(defn- table-metadata
+  "Helper for [[metadata-for]]. Prefer calling that function instead."
+  [ctx table-id]
+  (qm->metadata {"tables" [(-> (instance-fetch ctx (str "/api/table/" table-id "/query_metadata"))
+                               (dissoc "db"))]}))
+
+;; =============================================== External API ===================================================
+(defn metadata-for
+  "Given a map like `{:card [123 456], :table [789]}` returns a `MetadataProvider` with metadata for all the tables,
+  cards and fields transitively required by the specified roots.
+
+  Generally you only need a single `{:card [123]}` for the problem card you're investigating.
+
+  Occasionally a broken card might reference fields by ID which are not really part of their query - then they're
+  missing from the `/query_metadata` on that card and will be missed by this function unless you explicitly add
+  `{:table [789]}` to the map of roots.
+
+  Returns a complete `MetadataProvider` ready to use in a QP call."
+  [{:keys [database-id] :as ctx} kind->ids]
+  (->> (for [[kind ids] kind->ids
+             id         ids]
+         (case kind
+           :card  (recursive-card-metadata ctx id)
+           :table (table-metadata ctx id)))
+       (apply m/deep-merge)
+       (lib.js.metadata/metadata-provider database-id)))
+
+(defn preprocess-card
+  "Given a `MetadataProvider` built by [[metadata-for]] and a card ID, tries to preprocess that card."
+  [metadata-provider card-id]
+  (mu/disable-enforcement
+    (-> (lib.metadata/card metadata-provider card-id)
+        :dataset-query
+        qp.preprocess/preprocess)))
+
+(defn compile-card
+  "Given a `MetadataProvider` built by [[metadata-for]] and a card ID, tries to compile that card to e.g. SQL."
+  [metadata-provider card-id]
+  (mu/disable-enforcement
+    (-> (lib.metadata/card metadata-provider card-id)
+        :dataset-query
+        qp.compile/compile)))
+
+(comment
+  ;; Example calls
+  (let [card-id 456
+        mp      (metadata-for my-ctx {:card [card-id]})]
+    (compile-card mp card-id))
+
+  (let [card-id 456
+        mp      (metadata-for my-ctx {:card [card-id]})]
+    (preprocess-card mp card-id)))

dev/src/dev/mutation_testing.clj

@@ -7,11 +7,15 @@
    Configuration:
    - LINEAR_API_KEY env var must be set
    - Call (set-config! {:team-id \"...\" :project-id \"...\"}) before creating issues"
+  (:refer-clojure :exclude [run!])
   (:require
    [cheshire.core :as json]
    [clj-http.client :as http]
+   [clojure.java.io :as io]
    [clojure.java.shell :as shell]
-   [clojure.string :as str]))
+   [clojure.set :as set]
+   [clojure.string :as str]
+   [dev.coverage :as coverage]))
 
 (set! *warn-on-reflection* true)
 
@@ -20,14 +24,21 @@
 (defonce ^:private config (atom {}))
 
 (defn set-config!
-  "Set configuration for Linear API calls.
+  "Set configuration for mutation testing.
 
    Required keys:
    - :team-id   — Linear team ID
-   - :project-id — Linear project ID (can be set after creating a project)"
+   - :project-id — Linear project ID (can be set after creating a project)
+
+   Optional keys:
+   - :base-branch — git branch to create feature branches from (default: \"master\")
+   - :project-id  — Linear project ID to add issues to (skips project creation in run!)"
   [m]
   (swap! config merge m))
 
+(defn- base-branch []
+  (get @config :base-branch "master"))
+
 (defn- api-key []
   (or (System/getenv "LINEAR_API_KEY")
       (throw (ex-info "LINEAR_API_KEY environment variable is not set" {}))))
@@ -215,14 +226,14 @@
   "Generate a branch name for mutation testing a function.
    Example: mutation-testing-lib-order-by-orderable-columns"
   [target-ns fn-name]
-  (let [short-ns (last (clojure.string/split (str target-ns) #"\."))]
+  (let [short-ns (last (str/split (str target-ns) #"\."))]
     (str "mutation-testing-lib-" short-ns "-" fn-name)))
 
 (defn create-branch!
-  "Create and checkout a new branch from master for mutation testing a function."
+  "Create and checkout a new branch from the base branch for mutation testing a function."
   [target-ns fn-name]
   (let [branch (branch-name target-ns fn-name)]
-    (sh "git" "checkout" "master")
+    (sh "git" "checkout" (base-branch))
     (sh "git" "pull")
     (sh "git" "checkout" "-b" branch)
     branch))
@@ -255,10 +266,11 @@
   (let [title (pr-title target-ns fn-name)
         body (pr-description opts)
         result (sh "gh" "pr" "create" "--draft"
+                   "--base" (base-branch)
                    "--title" title
                    "--body" body
                    "--label" "no-backport")]
-    (clojure.string/trim (:out result))))
+    (str/trim (:out result))))
 
 (defn add-pr-comment!
   "Add a general comment to a PR. `pr-url-or-number` can be a PR URL or number."
@@ -308,12 +320,17 @@
                       "-f" "start_side=RIGHT"]))]
     (apply sh args)))
 
-(defn return-to-master!
-  "Checkout master branch."
+(defn return-to-base!
+  "Checkout the configured base branch."
   []
-  (sh "git" "checkout" "master")
+  (sh "git" "checkout" (base-branch))
   nil)
 
+(defn return-to-master!
+  "Checkout the configured base branch. Deprecated: use return-to-base! instead."
+  []
+  (return-to-base!))
+
 ;;; --- Report helpers ---
 
 (defn report-stats
@@ -409,6 +426,358 @@
   (create-issue! {:title (linear-issue-title target-ns fn-name)
                   :description (linear-issue-description target-ns fn-name)}))
 
+;;; --- Orchestration ---
+
+(defn parse-namespace
+  "Parse a namespace symbol into all derived paths and names.
+   Auto-detects .cljc vs .clj by checking which file exists."
+  [target-ns-sym]
+  (let [target-ns (str target-ns-sym)
+        test-ns (str target-ns "-test")
+        short-name (last (str/split target-ns #"\."))
+        ns-path (-> target-ns
+                    (str/replace "." "/")
+                    (str/replace "-" "_"))
+        test-path-base (-> test-ns
+                           (str/replace "." "/")
+                           (str/replace "-" "_"))
+        source-ext (cond
+                     (.exists (io/file (str "src/" ns-path ".cljc"))) ".cljc"
+                     (.exists (io/file (str "src/" ns-path ".clj"))) ".clj"
+                     :else ".cljc")
+        test-ext (cond
+                   (.exists (io/file (str "test/" test-path-base ".cljc"))) ".cljc"
+                   (.exists (io/file (str "test/" test-path-base ".clj"))) ".clj"
+                   :else ".cljc")]
+    {:target-ns (symbol target-ns)
+     :test-ns (symbol test-ns)
+     :short-name short-name
+     :source-path (str "src/" ns-path source-ext)
+     :test-path (str "test/" test-path-base test-ext)
+     :report-path (str "mutation-testing-report." target-ns ".before.md")}))
+
+(defn group-functions
+  "Group functions from coverage results for batch processing.
+   Takes the output of coverage/test-namespace.
+
+   Logic:
+   - Public functions with surviving mutations each become a group
+   - Private functions get assigned to the public function with the most test overlap
+   - Functions with zero surviving mutations are skipped
+   - Uncovered or unassignable private functions get their own group"
+  [coverage-results]
+  (let [with-survivors (into {} (filter (fn [[_ v]] (seq (:survived v))) coverage-results))
+        public? (fn [fn-sym]
+                  (let [v (find-var fn-sym)]
+                    (and v (not (:private (meta v))))))
+        public-fns (into {} (filter (fn [[k _]] (public? k)) with-survivors))
+        private-fns (into {} (remove (fn [[k _]] (public? k)) with-survivors))
+        assignments (into {}
+                          (for [[priv-fn priv-data] private-fns
+                                :let [priv-tests (set (:tests priv-data))
+                                      best (when (seq priv-tests)
+                                             (->> public-fns
+                                                  (map (fn [[pub-fn pub-data]]
+                                                         [pub-fn (count (set/intersection
+                                                                         priv-tests
+                                                                         (set (:tests pub-data))))]))
+                                                  (filter (fn [[_ n]] (pos? n)))
+                                                  (sort-by second >)
+                                                  first))]
+                                :when best]
+                            [priv-fn (first best)]))
+        unassigned-private (into {} (remove (fn [[k _]] (contains? assignments k)) private-fns))
+        public-groups (for [[pub-fn pub-data] public-fns
+                            :let [attached (keep (fn [[priv-fn assigned-to]]
+                                                   (when (= assigned-to pub-fn) priv-fn))
+                                                 assignments)
+                                  all-fn-names (vec (cons pub-fn attached))
+                                  all-mutations (vec (concat (:survived pub-data)
+                                                             (mapcat #(:survived (get coverage-results %))
+                                                                     attached)))
+                                  all-tests (apply set/union
+                                                   (keep #(:tests (get coverage-results %))
+                                                         all-fn-names))]]
+                        {:primary-fn pub-fn
+                         :fn-names all-fn-names
+                         :mutations all-mutations
+                         :tests (or all-tests #{})})
+        private-groups (for [[priv-fn priv-data] unassigned-private]
+                         {:primary-fn priv-fn
+                          :fn-names [priv-fn]
+                          :mutations (vec (:survived priv-data))
+                          :tests (or (:tests priv-data) #{})})]
+    (vec (concat public-groups private-groups))))
+
+(defn build-test-prompt
+  "Build a prompt for Claude to write tests that kill surviving mutations."
+  [{:keys [target-ns test-ns source-path test-path fn-names mutations]}]
+  (let [fn-sources (str/join "\n\n"
+                             (for [fn-name fn-names
+                                   :let [fn-info (coverage/find-function-source fn-name)
+                                         source (when fn-info (coverage/read-function-from-file fn-info))]
+                                   :when source]
+                               (str ";;; " fn-name "\n" source)))
+        mutation-list (str/join "\n\n"
+                                (map-indexed
+                                 (fn [i {:keys [description mutation]}]
+                                   (str "### Mutation " (inc i) ": " description "\n```clojure\n" mutation "\n```"))
+                                 mutations))
+        test-content (slurp test-path)]
+    (str "You are writing mutation tests for functions in `" target-ns "`.\n"
+         "\n"
+         "## Source Code of Functions Under Test\n"
+         "\n"
+         "```clojure\n"
+         fn-sources
+         "\n```\n"
+         "\n"
+         "## Existing Test File (" test-path ")\n"
+         "\n"
+         "```clojure\n"
+         test-content
+         "\n```\n"
+         "\n"
+         "## Surviving Mutations to Kill\n"
+         "\n"
+         "Each mutation below represents a change to the source code that no existing test catches.\n"
+         "Write the simplest tests that would fail if these mutations were applied.\n"
+         "\n"
+         mutation-list
+         "\n"
+         "\n"
+         "## Instructions\n"
+         "\n"
+         "- Write the simplest tests that kill these surviving mutations\n"
+         "- Follow the patterns in the existing test file (same helpers, same assertion style)\n"
+         "- Never call private functions directly — test through public API\n"
+         "- Insert tests near related existing tests, not at the end of the file\n"
+         "- Use the Edit tool to modify the test file at: " test-path "\n"
+         "- After writing tests, verify they compile by running: clj-nrepl-eval -p $(clj-nrepl-eval --discover-ports | head -1) \"(require '" test-ns " :reload)\"\n"
+         "- Each test should verify one meaningful behavior — don't over-engineer\n")))
+
+(defn invoke-claude!
+  "Shell out to claude CLI to write tests. Returns the output string."
+  [prompt]
+  (let [result (shell/sh "claude" "-p"
+                         "--allowedTools" "Edit,Read,Bash(clj-nrepl-eval*)"
+                         :in prompt)]
+    (when-not (zero? (:exit result))
+      (throw (ex-info "Claude invocation failed" {:exit (:exit result)
+                                                  :err (:err result)})))
+    (:out result)))
+
+(defn- all-test-names
+  "Get all test var names from a test namespace."
+  [test-ns-sym]
+  (require test-ns-sym :reload)
+  (let [ns-obj (find-ns test-ns-sym)]
+    (->> (ns-interns ns-obj)
+         vals
+         (filter #(:test (meta %)))
+         (map #(symbol (str test-ns-sym) (name (.sym ^clojure.lang.Var %))))
+         set)))
+
+(defn- count-tests
+  "Count test vars in a namespace."
+  [test-ns-sym]
+  (count (filter #(:test (meta %)) (vals (ns-interns (find-ns test-ns-sym))))))
+
+(defn verify-and-retry!
+  "Verify mutations are killed. Retry with Claude if needed.
+   Returns {:status :success|:partial, :killed [...], :survived [...]}."
+  [{:keys [fn-names test-ns test-path max-retries]
+    :or {max-retries 2}}]
+  (loop [retries-left max-retries]
+    (let [test-names (all-test-names test-ns)
+          fn-results (into {}
+                           (for [fn-name fn-names
+                                 :let [result (coverage/test-mutations fn-name test-names)]
+                                 :when result]
+                             [fn-name result]))
+          all-killed (vec (mapcat :killed (vals fn-results)))
+          all-survived (vec (mapcat :survived (vals fn-results)))]
+      (if (empty? all-survived)
+        {:status :success :killed all-killed :survived []}
+        (if (pos? retries-left)
+          (do
+            (println "  " (count all-survived) "mutations still survive, retrying..."
+                     "(" retries-left "retries left)")
+            (let [retry-prompt (str "Some mutations still survive after your previous tests. "
+                                    "Please write additional tests to kill them.\n\n"
+                                    "## Still-Surviving Mutations\n\n"
+                                    (str/join "\n\n"
+                                              (for [{:keys [description mutation]} all-survived]
+                                                (str "### " description "\n```clojure\n" mutation "\n```")))
+                                    "\n\n## Instructions\n"
+                                    "- The test file is at: " test-path "\n"
+                                    "- Use the Edit tool to add more tests\n"
+                                    "- Focus specifically on the mutations listed above\n"
+                                    "- Do NOT duplicate existing tests\n")]
+              (invoke-claude! retry-prompt))
+            (recur (dec retries-left)))
+          {:status :partial :killed all-killed :survived all-survived})))))
+
+(defn process-group!
+  "Process one group of functions: branch → tests → verify → commit → PR.
+   Returns {:pr-url ... :issue ... :killed ... :not-killed ...}."
+  [session group]
+  (let [{:keys [target-ns test-ns test-path]} session
+        {:keys [primary-fn fn-names mutations]} group
+        primary-fn-name (name primary-fn)]
+    ;; 1. Create branch
+    (println "  Creating branch...")
+    (create-branch! (str target-ns) primary-fn-name)
+
+    ;; 2. Count tests before
+    (require test-ns :reload)
+    (let [tests-before (count-tests test-ns)]
+
+      ;; 3. Build prompt and invoke Claude
+      (println "  Invoking Claude to write tests...")
+      (let [prompt (build-test-prompt {:target-ns target-ns
+                                       :test-ns test-ns
+                                       :source-path (:source-path session)
+                                       :test-path test-path
+                                       :fn-names fn-names
+                                       :mutations mutations})]
+        (invoke-claude! prompt))
+
+      ;; 4. Verify and retry
+      (println "  Verifying mutations killed...")
+      (let [verify-result (verify-and-retry! {:fn-names fn-names
+                                              :test-ns test-ns
+                                              :test-path test-path
+                                              :max-retries 2})
+            killed (:killed verify-result)
+            survived (:survived verify-result)
+            tests-after (count-tests test-ns)
+            tests-added (- tests-after tests-before)]
+
+        ;; 5. Commit and push
+        (println "  Committing and pushing...")
+        (commit-and-push! (str target-ns) primary-fn-name [test-path])
+
+        ;; 6. Create Linear issue
+        (println "  Creating Linear issue...")
+        (let [issue (create-issue-for-function! (str target-ns) primary-fn-name)
+
+              ;; 7. Create draft PR
+              _ (println "  Creating draft PR...")
+              pr-url (create-draft-pr!
+                      {:target-ns (str target-ns)
+                       :fn-name primary-fn-name
+                       :fn-names (mapv name fn-names)
+                       :linear-identifier (:identifier issue)
+                       :mutations-before (count mutations)
+                       :tests-added tests-added
+                       :killed (mapv :description killed)
+                       :not-killed (mapv (fn [m] {:description (:description m)
+                                                  :rationale "Could not be killed automatically"})
+                                         survived)
+                       :suggested-changes []})]
+
+          ;; 8. Return to base branch
+          (return-to-base!)
+
+          (println "  Done! PR:" pr-url)
+          {:pr-url pr-url
+           :issue issue
+           :killed killed
+           :not-killed survived
+           :tests-added tests-added})))))
+
+(defn print-summary!
+  "Print a summary of the mutation testing run."
+  [{:keys [project groups results]}]
+  (println "\n========================================")
+  (println "  Mutation Testing Summary")
+  (println "========================================")
+  (println "Project:" (:name project))
+  (println "Groups:" (count groups))
+  (let [successful (filter :pr-url results)
+        failed (filter :error results)
+        total-killed (count (mapcat :killed successful))
+        total-unkilled (count (mapcat :not-killed successful))
+        total-tests (reduce + 0 (keep :tests-added successful))]
+    (println "PRs created:" (count successful))
+    (when (seq failed)
+      (println "Errors:" (count failed)))
+    (println "Tests added:" total-tests)
+    (println "Mutations killed:" total-killed)
+    (println "Mutations not killed:" total-unkilled)
+    (println)
+    (when (seq successful)
+      (println "PRs:")
+      (doseq [r successful]
+        (println "  " (:pr-url r))))
+    (when (seq failed)
+      (println "\nFailed:")
+      (doseq [r failed]
+        (println "  " (:primary-fn r) "-" (:error r))))
+    (println "========================================")))
+
+(defn run!
+  "Main entry point. Runs the full mutation testing workflow for a namespace.
+
+   Prerequisites:
+   - nREPL running
+   - LINEAR_API_KEY env var set
+   - gh CLI authenticated
+   - team-id configured via (set-config! {:team-id \"...\"})
+
+   Options (optional second arg):
+   - :base-branch — git branch to create feature branches from (default: from config, or \"master\")
+   - :project-id  — existing Linear project ID to use (skips creating a new project)"
+  ([target-ns-sym] (run! target-ns-sym {}))
+  ([target-ns-sym opts]
+   (when-let [bb (:base-branch opts)]
+     (set-config! {:base-branch bb}))
+   (when-let [pid (:project-id opts)]
+     (set-config! {:project-id pid}))
+   (let [parsed (parse-namespace target-ns-sym)
+         {:keys [target-ns test-ns report-path]} parsed]
+     ;; 1. Generate report
+     (println "Generating mutation testing report for" (str target-ns) "...")
+     (coverage/generate-report target-ns [test-ns] report-path)
+     (println "Report written to" report-path)
+
+     ;; 2. Create or reuse Linear project
+     (let [project (if (:project-id @config)
+                     (do (println "Using existing Linear project:" (:project-id @config))
+                         {:project-id (:project-id @config)
+                          :name (str "Existing project " (:project-id @config))})
+                     (do (println "Creating Linear project...")
+                         (let [p (create-project-for-namespace! (str target-ns) report-path)]
+                           (println "Created project:" (:name p))
+                           p)))]
+
+       ;; 3. Run coverage and group functions
+       (println "Running coverage analysis and grouping...")
+       (let [coverage-results (coverage/test-namespace target-ns [test-ns])
+             groups (group-functions coverage-results)]
+         (println "Found" (count groups) "function groups to process")
+
+         ;; 4. Process each group
+         (let [results (vec
+                        (for [group groups]
+                          (do
+                            (println "\n--- Processing:" (:primary-fn group)
+                                     "(" (count (:mutations group)) "surviving mutations) ---")
+                            (try
+                              (process-group! parsed group)
+                              (catch Exception e
+                                (println "ERROR:" (.getMessage e))
+                                (try (return-to-base!) (catch Exception _))
+                                {:error (.getMessage e)
+                                 :primary-fn (:primary-fn group)})))))]
+
+           ;; 5. Print summary
+           (print-summary! {:project project
+                            :groups groups
+                            :results results})
+           results))))))
+
 (comment
   ;; Setup:
   ;; 1. Set LINEAR_API_KEY env var before starting the REPL

docs/api.json

@@ -9408,10 +9408,6 @@
         "description" : "schema for a valid MBQL 4 :value clause",
         "allOf" : [ ]
       },
-      "metabase.lib.metadata.column.column-unique-key" : {
-        "type" : "string",
-        "pattern" : "^column-unique-key-v\\d+\\$.+$"
-      },
       "metabase.lib.metadata.protocols.metadata-provider" : {
         "description" : "Schema for something that satisfies the [[metabase.lib.metadata.protocols/MetadataProvider]] protocol."
       },
@@ -9635,6 +9631,10 @@
           "$ref" : "#/components/schemas/metabase.lib.schema.util.distinct-mbql-clauses"
         } ]
       },
+      "metabase.lib.schema.column-unique-key" : {
+        "type" : "string",
+        "pattern" : "^column-unique-key-v\\d+\\$.+$"
+      },
       "metabase.lib.schema.common.base-type" : {
         "type" : "string"
       },
@@ -14531,7 +14531,7 @@
             "type" : "string"
           },
           "checkpoint-filter-unique-key" : {
-            "$ref" : "#/components/schemas/metabase.lib.metadata.column.column-unique-key"
+            "$ref" : "#/components/schemas/metabase.lib.schema.column-unique-key"
           },
           "type" : {
             "const" : "checkpoint"
@@ -39537,135 +39537,6 @@
         "tags" : [ "/api/pulse" ]
       }
     },
-    "/api/pulse/preview_card/{id}" : {
-      "get" : {
-        "operationId" : "get-api-pulse-preview_card-id",
-        "summary" : "GET /api/pulse/preview_card/{id}",
-        "description" : "Get HTML rendering of a Card with `id`.",
-        "parameters" : [ {
-          "in" : "path",
-          "name" : "id",
-          "required" : true,
-          "schema" : {
-            "type" : "integer",
-            "minimum" : 1
-          },
-          "description" : "value must be an integer greater than zero."
-        } ],
-        "responses" : {
-          "2XX" : {
-            "description" : "Successful response"
-          },
-          "4XX" : {
-            "description" : "Client error response"
-          },
-          "5XX" : {
-            "description" : "Server error response"
-          }
-        },
-        "tags" : [ "/api/pulse" ]
-      }
-    },
-    "/api/pulse/preview_card_info/{id}" : {
-      "get" : {
-        "operationId" : "get-api-pulse-preview_card_info-id",
-        "summary" : "GET /api/pulse/preview_card_info/{id}",
-        "description" : "Get JSON object containing HTML rendering of a Card with `id` and other information.",
-        "parameters" : [ {
-          "in" : "path",
-          "name" : "id",
-          "required" : true,
-          "schema" : {
-            "type" : "integer",
-            "minimum" : 1
-          },
-          "description" : "value must be an integer greater than zero."
-        } ],
-        "responses" : {
-          "2XX" : {
-            "description" : "Successful response"
-          },
-          "4XX" : {
-            "description" : "Client error response"
-          },
-          "5XX" : {
-            "description" : "Server error response"
-          }
-        },
-        "tags" : [ "/api/pulse" ]
-      }
-    },
-    "/api/pulse/preview_card_png/{id}" : {
-      "get" : {
-        "operationId" : "get-api-pulse-preview_card_png-id",
-        "summary" : "GET /api/pulse/preview_card_png/{id}",
-        "description" : "Get PNG rendering of a Card with `id`. Optionally specify `width` as a query parameter.",
-        "parameters" : [ {
-          "in" : "path",
-          "name" : "id",
-          "required" : true,
-          "schema" : {
-            "type" : "integer",
-            "minimum" : 1
-          },
-          "description" : "value must be an integer greater than zero."
-        }, {
-          "in" : "query",
-          "name" : "width",
-          "required" : false,
-          "schema" : {
-            "oneOf" : [ {
-              "description" : "value must be an integer greater than zero.",
-              "type" : "integer",
-              "minimum" : 1
-            }, {
-              "type" : "null"
-            } ]
-          }
-        } ],
-        "responses" : {
-          "2XX" : {
-            "description" : "Successful response"
-          },
-          "4XX" : {
-            "description" : "Client error response"
-          },
-          "5XX" : {
-            "description" : "Server error response"
-          }
-        },
-        "tags" : [ "/api/pulse" ]
-      }
-    },
-    "/api/pulse/preview_dashboard/{id}" : {
-      "get" : {
-        "operationId" : "get-api-pulse-preview_dashboard-id",
-        "summary" : "GET /api/pulse/preview_dashboard/{id}",
-        "description" : "Get HTML rendering of a Dashboard with `id`.\n\n  This endpoint relies on a custom middleware defined in `metabase.channel.render.core/style-tag-nonce-middleware` to\n  allow the style tag to render properly, given our Content Security Policy setup. This middleware is attached to these\n  routes at the bottom of this namespace using `metabase.api.common/define-routes`.",
-        "parameters" : [ {
-          "in" : "path",
-          "name" : "id",
-          "required" : true,
-          "schema" : {
-            "type" : "integer",
-            "minimum" : 1
-          },
-          "description" : "value must be an integer greater than zero."
-        } ],
-        "responses" : {
-          "2XX" : {
-            "description" : "Successful response"
-          },
-          "4XX" : {
-            "description" : "Client error response"
-          },
-          "5XX" : {
-            "description" : "Server error response"
-          }
-        },
-        "tags" : [ "/api/pulse" ]
-      }
-    },
     "/api/pulse/test" : {
       "post" : {
         "operationId" : "post-api-pulse-test",

docs/databases/connections/athena.md

@@ -209,6 +209,12 @@ If Metabase also needs to create tables, you'll need additional AWS Glue permiss
 }
 ```
 
+## Writeable connection
+
+{% include plans-blockquote.html feature="Writeable connection" is_plural=true%}
+
+Set up an additional connection used for write operations. See [Writeable connections](../writeable-connection.md).
+
 ## Model features
 
 There aren't (yet) any model features available for Athena.

docs/databases/connections/bigquery.md

@@ -161,6 +161,12 @@ FROM [my_dataset.my_table]
 
 If you're having trouble with your BigQuery connection, you can check out this [troubleshooting guide](../../troubleshooting-guide/bigquery-drive.md) that covers BigQuery issues, [this one](../../troubleshooting-guide/db-connection.md) on data warehouse connections, or visit [Metabase's discussion forum](https://discourse.metabase.com/search?q=bigquery) to see if someone has encountered and resolved a similar issue.
 
+## Writeable connection
+
+{% include plans-blockquote.html feature="Writeable connection" is_plural=true%}
+
+Set up an additional connection used for write operations. See [Writeable connections](../writeable-connection.md).
+
 ## Model features
 
 There aren't (yet) any model features available for BigQuery.

docs/databases/connections/clickhouse.md

@@ -101,6 +101,12 @@ Turn this option **ON** to scan a sample of values every time Metabase runs a [s
 
 A fingerprinting query examines the first 10,000 rows from each column and uses that data to guesstimate how many unique values each column has, what the minimum and maximum values are for numeric and timestamp columns, and so on. If you leave this option **OFF**, Metabase will only fingerprint your columns once during setup.
 
+## Writeable connection
+
+{% include plans-blockquote.html feature="Writeable connection" is_plural=true%}
+
+Set up an additional connection used for write operations. See [Writeable connections](../writeable-connection.md).
+
 ## Model features
 
 There aren't (yet) any model features for ClickHouse.

docs/databases/connections/databricks.md

@@ -101,6 +101,12 @@ Turn this option **ON** to scan a sample of values every time Metabase runs a [s
 
 A fingerprinting query examines the first 10,000 rows from each column and uses that data to guesstimate how many unique values each column has, what the minimum and maximum values are for numeric and timestamp columns, and so on. If you leave this option **OFF**, Metabase will only fingerprint your columns once during setup.
 
+## Writeable connection
+
+{% include plans-blockquote.html feature="Writeable connection" is_plural=true%}
+
+Set up an additional connection used for write operations. See [Writeable connections](../writeable-connection.md).
+
 ## Model features
 
 There aren't (yet) any model features available for Databricks.

docs/databases/connections/druid.md

@@ -64,6 +64,12 @@ Turn this option **ON** to scan a sample of values every time Metabase runs a [s
 
 A fingerprinting query examines the first 10,000 rows from each column and uses that data to guesstimate how many unique values each column has, what the minimum and maximum values are for numeric and timestamp columns, and so on. If you leave this option **OFF**, Metabase will only fingerprint your columns once during setup.
 
+## Writeable connection
+
+{% include plans-blockquote.html feature="Writeable connection" is_plural=true%}
+
+Set up an additional connection used for write operations. See [Writeable connections](../writeable-connection.md).
+
 ## Model features
 
 There aren't (yet) any model features available for Druid.

docs/databases/connections/mariadb.md

@@ -101,6 +101,12 @@ JSON schema inference doesn't work with MariaDB, due to implementation differenc
 
 If your password contains characters that aren't UTF-8, then you might need to add an additional variable to the connection string `passwordCharacterEncoding=<your_encoding_here>`. This ensures that MariaDB understands the special characters in the password during authentication.
 
+## Writeable connection
+
+{% include plans-blockquote.html feature="Writeable connection" is_plural=true%}
+
+Set up an additional connection used for write operations. See [Writeable connections](../writeable-connection.md).
+
 ## Database routing
 
 With database routing, an admin can build a question once using one database, and the question will run its query against a different database with the same schema depending on who is viewing the question.

docs/databases/connections/mongodb.md

@@ -130,6 +130,12 @@ Instead, Metabase gets a sample of the fields in a collection by scanning a samp
 
 If you're not seeing all of the fields show up for a collection in Metabase, one workaround is to include all possible keys in the first document of the collection, and give those keys null values. That way, Metabase will be able to recognize the correct schema for the entire collection.
 
+## Writeable connection
+
+{% include plans-blockquote.html feature="Writeable connection" is_plural=true%}
+
+Set up an additional connection used for write operations. See [Writeable connections](../writeable-connection.md).
+
 ## Database routing
 
 With database routing, an admin can build a question once using one database, and the question will run its query against a different database with the same schema depending on who is viewing the question.

docs/databases/connections/mysql.md

@@ -187,6 +187,12 @@ mysql:
 
 If your password contains characters that aren't UTF-8, then you might need to add an additional variable to the connection string `passwordCharacterEncoding=<your_encoding_here>`. This ensures that MySQL understands the special characters in the password during authentication.
 
+## Writeable connection
+
+{% include plans-blockquote.html feature="Writeable connection" is_plural=true%}
+
+Set up an additional connection used for write operations. See [Writeable connections](../writeable-connection.md).
+
 ## Model features
 
 Choose whether to enable features related to [Metabase models](../../data-modeling/models.md). These features will often require that the database user account, the one you use to connect to your database, has both read and write privileges.

docs/databases/connections/oracle.md

@@ -131,6 +131,12 @@ For example, if you're running Metabase from a directory called `/app/`, you sho
 
 The process for adding plugins when running via Docker is similar, but you'll need to mount the `plugins` directory. Refer to instructions [here](../../installation-and-operation/running-metabase-on-docker.md#adding-external-dependencies-or-plugins) for more details.
 
+## Writeable connection
+
+{% include plans-blockquote.html feature="Writeable connection" is_plural=true%}
+
+Set up an additional connection used for write operations. See [Writeable connections](../writeable-connection.md).
+
 ## Danger zone
 
 See [Danger zone](../danger-zone.md).

docs/databases/connections/postgresql.md

@@ -186,6 +186,12 @@ Turn this option **ON** to scan a sample of values every time Metabase runs a [s
 
 A fingerprinting query examines the first 10,000 rows from each column and uses that data to guesstimate how many unique values each column has, what the minimum and maximum values are for numeric and timestamp columns, and so on. If you leave this option **OFF**, Metabase will only fingerprint your columns once during setup.
 
+## Writeable connection
+
+{% include plans-blockquote.html feature="Writeable connection" is_plural=true%}
+
+Set up an additional connection used for write operations. See [Writeable connections](../writeable-connection.md).
+
 ## Model features
 
 Choose whether to enable features related to [Metabase models](../../data-modeling/models.md). These features will often require that the database user account, the one you use to connect to your database, has both read and write privileges.

docs/databases/connections/presto.md

@@ -94,6 +94,12 @@ Turn this option **ON** to scan a sample of values every time Metabase runs a [s
 
 A fingerprinting query examines the first 10,000 rows from each column and uses that data to guesstimate how many unique values each column has, what the minimum and maximum values are for numeric and timestamp columns, and so on. If you leave this option **OFF**, Metabase will only fingerprint your columns once during setup.
 
+## Writeable connection
+
+{% include plans-blockquote.html feature="Writeable connection" is_plural=true%}
+
+Set up an additional connection used for write operations. See [Writeable connections](../writeable-connection.md).
+
 ## Database routing
 
 With database routing, an admin can build a question once using one database (data catalog), and the question will run its query against a different data catalog with the same schema depending on who is viewing the question.

docs/databases/connections/redshift.md

@@ -98,6 +98,12 @@ Turn this option **ON** to scan a sample of values every time Metabase runs a [s
 
 A fingerprinting query examines the first 10,000 rows from each column and uses that data to guesstimate how many unique values each column has, what the minimum and maximum values are for numeric and timestamp columns, and so on. If you leave this option **OFF**, Metabase will only fingerprint your columns once during setup.
 
+## Writeable connection
+
+{% include plans-blockquote.html feature="Writeable connection" is_plural=true%}
+
+Set up an additional connection used for write operations. See [Writeable connections](../writeable-connection.md).
+
 ## Model features
 
 Choose whether to enable features related to Metabase models. These will often require a write connection.

docs/databases/connections/snowflake.md

@@ -14,17 +14,17 @@ You can edit these settings at any time. Just remember to save your changes.
 
 Paste a connection string here to pre-fill the remaining fields below.
 
-## Display name
+### Display name
 
 The display name for the database in the Metabase interface.
 
-## Account or hostname
+### Account or hostname
 
 You have the option of using an account name or hostname. Your hostname is your database's IP address or its domain name (e.g., database.example.com).
 
 If your URL includes your database's region, you can connect using an account name, otherwise connect using a hostname.
 
-### AWS
+#### AWS
 
 Connect with an account name.
 
@@ -37,7 +37,7 @@ You'd enter `az12345.ca-central-1.aws` as the account name in Metabase.
 
 Not all regions require the cloud platform identifier. If you're in `us-west-2`, you would enter `az12345` as the account name. For the requirements per region, see [the official Snowflake's documentation](https://docs.snowflake.com/en/user-guide/admin-account-identifier.html#non-vps-account-locator-formats-by-cloud-platform-and-region).
 
-### app.snowflake.com
+#### app.snowflake.com
 
 Connect with a host name.
 
@@ -47,31 +47,31 @@ You can get your host name by going to Admin > Accounts. The Account name will b
 
 Learn more about [Account Identifiers in Snowflake](https://docs.snowflake.com/en/user-guide/admin-account-identifier).
 
-## Username
+### Username
 
 The database username for the account that you want to use to connect to your database. You can set up multiple connections to the same database using different user accounts to connect to the same database, each with different sets of [privileges](../users-roles-privileges.md).
 
 On your app.snowflake.com account page, you can find Users and roles under Admin > Users & Roles.
 
-## Password
+### Password
 
 The password for the username that you use to connect to the database.
 
-## RSA private key (PEM)
+### RSA private key (PEM)
 
 Not required. You have the option of using a **Local file path** or an **Uploaded file path**. If you're on Metabase Cloud, you'll need to select **Uploaded file path** and upload your certificate.
 
-## Warehouse
+### Warehouse
 
 Snowflake warehouse. If the user lacks a default warehouse, you'll need to enter the warehouse to connect to.
 
 On app.snowflake.com, you can find warehouses under Admin > Warehouses.
 
-## Database name (case sensitive)
+### Database name (case sensitive)
 
 The name of the database you want to connect to in Snowflake. On app.snowflake.com you can find databases under Data > Databases.
 
-## Schemas (optional)
+### Schemas (optional)
 
 You can specify which schemas you want to sync and scan. If no schema is passed, then all schema available to that user and role will be listed as folders in Metabase.
 
@@ -96,7 +96,7 @@ Let's say you have three schemas: FOO, BAR, and BAZ.
 
 Note that only the `*` wildcard is supported; you can't use other special characters or regexes.
 
-## Role (optional)
+### Role (optional)
 
 > **WARNING:** Metabase gets _all the permissions_ combined from _every_ role you grant the user in Snowflake. Setting the 'Role' here just adds the role to the connection string to tell Snowflake which role to default to – it **won't restrict** access on its own. So **be careful which roles you assign to the connection user**: if down the line you grant that user broader roles in Snowflake, Metabase will automatically gets those new permissions, potentially showing data you didn't mean for it to see. If instead you want Metabase to issue `SET ROLE` commands before querying anything, check out [connection impersonation](../../permissions/impersonation.md).
 
@@ -107,25 +107,25 @@ Specify a role to override the database user's default role. For example, if the
 
 You can enter `ANALYTICS` in the Role field to ensure that the `METABASE` user connects to Snowflake using the `ANALYTICS` role by default.
 
-## Use an SSH tunnel
+### Use an SSH tunnel
 
 See our [guide to SSH tunneling](../ssh-tunnel.md).
 
-## Additional JDBC connection string options
+### Additional JDBC connection string options
 
 Some databases allow you to append options to the connection string that Metabase uses to connect to your database.
 
-## Re-run queries for simple explorations
+### Re-run queries for simple explorations
 
 Turn this option **OFF** if people want to click **Run** (the play button) before applying any [Summarize](../../questions/query-builder/summarizing-and-grouping.md) or filter selections.
 
 By default, Metabase will execute a query as soon as you choose an grouping option from the **Summarize** menu or a filter condition from the [drill-through menu](https://www.metabase.com/learn/metabase-basics/querying-and-dashboards/questions/drill-through). If your database is slow, you may want to disable re-running to avoid loading data on each click.
 
-## Choose when syncs and scans happen
+### Choose when syncs and scans happen
 
 See [syncs and scans](../sync-scan.md#choose-when-syncs-and-scans-happen).
 
-### Periodically refingerprint tables
+#### Periodically refingerprint tables
 
 > Periodic refingerprinting will increase the load on your database.
 
@@ -133,6 +133,12 @@ Turn this option **ON** to scan a sample of values every time Metabase runs a [s
 
 A fingerprinting query examines the first 10,000 rows from each column and uses that data to guesstimate how many unique values each column has, what the minimum and maximum values are for numeric and timestamp columns, and so on. If you leave this option **OFF**, Metabase will only fingerprint your columns once during setup.
 
+## Writeable connection
+
+{% include plans-blockquote.html feature="Writeable connection" is_plural=true%}
+
+Set up an additional connection used for write operations. See [Writeable connections](../writeable-connection.md).
+
 ## Model features
 
 There aren't (yet) any model features available for Snowflake.

docs/databases/connections/sparksql.md

@@ -77,6 +77,12 @@ Turn this option **ON** to scan a sample of values every time Metabase runs a [s
 
 A fingerprinting query examines the first 10,000 rows from each column and uses that data to guesstimate how many unique values each column has, what the minimum and maximum values are for numeric and timestamp columns, and so on. If you leave this option **OFF**, Metabase will only fingerprint your columns once during setup.
 
+## Writeable connection
+
+{% include plans-blockquote.html feature="Writeable connection" is_plural=true%}
+
+Set up an additional connection used for write operations. See [Writeable connections](../writeable-connection.md).
+
 ## Danger zone
 
 See [Danger zone](../danger-zone.md).

docs/databases/connections/sql-server.md

@@ -97,6 +97,12 @@ A fingerprinting query examines the first 10,000 rows from each column and uses
 
 To connect to Azure SQL, you'll need to set the port to 1433.
 
+## Writeable connection
+
+{% include plans-blockquote.html feature="Writeable connection" is_plural=true%}
+
+Set up an additional connection used for write operations. See [Writeable connections](../writeable-connection.md).
+
 ## Database routing
 
 With database routing, an admin can build a question once using one database, and the question will run its query against a different database with the same schema depending on who is viewing the question.

docs/databases/connections/sqlite.md

@@ -61,6 +61,12 @@ Turn this option **ON** to scan a sample of values every time Metabase runs a [s
 
 A fingerprinting query examines the first 10,000 rows from each column and uses that data to guesstimate how many unique values each column has, what the minimum and maximum values are for numeric and timestamp columns, and so on. If you leave this option **OFF**, Metabase will only fingerprint your columns once during setup.
 
+## Writeable connection
+
+{% include plans-blockquote.html feature="Writeable connection" is_plural=true%}
+
+Set up an additional connection used for write operations. See [Writeable connections](../writeable-connection.md).
+
 ## Database routing
 
 With database routing, an admin can build a question once using one database, and the question will run its query against a different database with the same schema depending on who is viewing the question.

docs/databases/connections/starburst.md

@@ -91,6 +91,12 @@ Turn this option **ON** to scan a sample of values every time Metabase runs a [s
 
 A fingerprinting query examines the first 10,000 rows from each column and uses that data to guesstimate how many unique values each column has, what the minimum and maximum values are for numeric and timestamp columns, and so on. If you leave this option **OFF**, Metabase will only fingerprint your columns once during setup.
 
+## Writeable connection
+
+{% include plans-blockquote.html feature="Writeable connection" is_plural=true%}
+
+Set up an additional connection used for write operations. See [Writeable connections](../writeable-connection.md).
+
 ## Model features
 
 There aren't (yet) any model features for Starburst.

docs/databases/connections/vertica.md

@@ -41,6 +41,12 @@ For example, if you're running Metabase from a directory called `/app/`, you sho
 
 The process for adding plugins when running via Docker is similar, but you'll need to mount the `plugins` directory. Refer to instructions [here](../../installation-and-operation/running-metabase-on-docker.md#adding-external-dependencies-or-plugins) for more details.
 
+## Writeable connection
+
+{% include plans-blockquote.html feature="Writeable connection" is_plural=true%}
+
+Set up an additional connection used for write operations. See [Writeable connections](../writeable-connection.md).
+
 ## Model features
 
 There aren't (yet) any model features for Vertica.

docs/databases/users-roles-privileges.md

@@ -1,21 +1,28 @@
 ---
 title: Database users, roles, and privileges
+summary: Set up database users and roles for Metabase: read-only access for queries, optional write permissions for transforms, uploads, actions, and model persistence.
 ---
 
 # Database users, roles, and privileges
 
-We recommend creating a `metabase` database user with the following database roles:
+## Recommended setup
+
+We recommend creating a dedicated `metabase` database user with the **read-only** access to your database with [minimum database privileges for read access](#minimum-database-privileges) to any schemas or tables used for analysis.
+
+Some Metabase functionality requires write access to the database. Depending on other Metabase features you're planning to use, you might want to create additional roles:
 
-- [`analytics` for read access](#minimum-database-privileges) to any schemas or tables used for analysis.
-- Optional [`metabase_actions` for write access](#privileges-to-enable-actions-and-editable-table-data) to tables used for Metabase actions.
-- Optional [`metabase_model_persistence` for write access](#privileges-to-enable-model-persistence) to the schema used for Metabase model persistence.
 - Optional [`metabase_transforms` for write access](#privileges-to-enable-transforms) to the schema used for Metabase transforms.
+- Optional [`metabase_uploads` for write access](#privileges-to-enable-uploads) to schema used for CSV uploads.
+- Optional [`metabase_actions` for write access](#privileges-to-enable-actions-and-editable-table-data) to tables used for editable tables or Metabase actions .
+- Optional [`metabase_model_persistence` for write access](#privileges-to-enable-model-persistence) to the schema used for Metabase model persistence.
 
 Bundling your privileges into roles based on use cases makes it easier to manage privileges in the future (especially in [multi-tenant situations](#multi-tenant-permissions)). For example, you could:
 
 - Use the same `analytics` role for other BI tools in your [data stack](https://www.metabase.com/learn/grow-your-data-skills/data-fundamentals/data-landscape) that need read-only access to the analytics tables in your database.
 - Revoke the write access for `metabase_model_persistence` without affecting the write access for `metabase_actions`.
 
+On Metabase Pro/Enterprise plans, we recommend using separate [Writeable connections](./writeable-connection.md) for operations that require write access.
+
 ## Minimum database privileges
 
 In order to view and query your tables in Metabase, you'll have to give Metabase's database user:
@@ -85,17 +92,17 @@ CREATE USER metabase WITH PASSWORD "your_password";
 GRANT ALL PRIVILEGES ON "database" TO metabase;
 ```
 
-This is a good option if you're connecting to a local database for development or testing.
+This is a good option if you're connecting to a local database for development or testing. Avoid using this setup in production.
 
 ## Privileges to enable actions and editable table data
 
 Both [actions](../actions/introduction.md) and the [editable table data](../data-modeling/editable-tables.md) let Metabase write back to specific tables in your database.
 
-In addition to the [minimum database privileges](#minimum-database-privileges), you'll need to grant write access to any tables you want to be able to write to.
+if you're using actions or editable tables, then in addition to the [minimum database privileges](#minimum-database-privileges), you'll need to grant write access to any tables you want to be able to write to.
 
 - Create a new role called `metabase_writer`.
 - Give the role `INSERT`, `UPDATE`, and `DELETE` privileges to the relevant tables.
-- Give the `metabase_writer` role to the `metabase` user.
+- Give the `metabase_writer` role to the user used for your database connection. We recommend using a [writeable connection](./writeable-connection.md) and only granting this role to the user used for the writeable connection.
 
 ```sql
 -- Create a role to bundle database privileges for Metabase writing to your database.
@@ -117,7 +124,7 @@ In addition to the [minimum database privileges](#minimum-database-privileges):
 - Create a new role called `metabase_model_persistence`.
 - Give the role `CREATE` access to the database.
 - Give the role `INSERT`, `UPDATE`, and `DELETE` privileges to the schema used for model persistence.
-- Give the `metabase_model_persistence` role to the `metabase` user.
+- Give the `metabase_model_persistence` role to the user used for your database connection. We recommend using a [writeable connection](./writeable-connection.md) and only granting this role to the user used for the writeable connection.
 
 ```sql
 -- Create a role to bundle database privileges for Metabase model persistence.
@@ -137,20 +144,20 @@ GRANT metabase_model_persistence TO metabase;
 
 ## Privileges to enable transforms
 
-[Transforms](../data-studio/transforms/transforms-overview.md) let Metabase write query results back to your database. We suggest that you create a dedicated schema for your transforms. Metabase's database user will need to be able to create and drop transform tables.
+[Transforms](../data-studio/transforms/transforms-overview.md) let Metabase write query results back to your database. We suggest that you create a dedicated schema for your transforms. Metabase's database user will need to be able to create and drop transform tables. We recommend using a [writeable connection](./writeable-connection.md) and only granting write access to the user used for the writeable connection.
 
 In addition to the [minimum database privileges](#minimum-database-privileges):
 
 - Create a new role called `metabase_transforms`.
 - Create a dedicated schema for transforms.
 - Give the `metabase_transforms` role `CREATE TABLE` / `ALTER` / `DROP` privileges for the specified schema.
-- Give the `metabase_transforms` role to the `metabase` user.
+- Give the `metabase_transforms` role to the user used for your database connection. We recommend using a [writeable connection](./writeable-connection.md) and only granting this role to the user used for the writeable connection.
 
 In certain cases, Metabase will give you an option to create new schemas for transforms, in which case you'll also need `CREATE SCHEMA` privileges for your metabase users (or `CREATE DATABASE` for ClickHouse).
 
 ## Privileges to enable uploads
 
-You can [upload CSVs](../databases/uploads.md) to supported databases. Metabase's database user should have write access (`INSERT`, `UPDATE`, `DELETE`) to the schema where you want to store the uploads.
+You can [upload CSVs](../databases/uploads.md) to supported databases. Metabase's database user should have write access (`INSERT`, `UPDATE`, `DELETE`) to the schema where you want to store the uploads. We recommend using a [writeable connection](./writeable-connection.md) and only granting write access to the user used for the writeable connection.
 
 You'll first need to create a schema to store uploads (or use an existing schema) and tell Metabase that you want to [use that schema to store uploads](./uploads.md#select-the-database-and-schema-that-you-want-to-store-the-data-in).
 
@@ -158,7 +165,7 @@ In addition to the [minimum database privileges](#minimum-database-privileges):
 
 - Create a new role called `metabase_uploads`.
 - Give the role `INSERT`, `UPDATE`, and `DELETE` privileges to the schema where you want to store uploads.
-- Give the `metabase_uploads` role to the `metabase` user.
+- Give the `metabase_uploads` role to the user used for your database connection.
 
 ```sql
 -- Create a role to bundle database privileges for uploads.
@@ -221,6 +228,7 @@ We recommend bundling privileges into roles based on use cases per customer. Tha
 
 ## Further reading
 
+- [Writeable connections](./writeable-connection.md)
 - [Permissions strategies](https://www.metabase.com/learn/metabase-basics/administration/permissions/strategy)
 - [Permissions introduction](../permissions/introduction.md)
 - [People overview](../people-and-groups/start.md)

docs/databases/writeable-connection.md

@@ -0,0 +1,60 @@
+---
+title: Writeable connections
+summary: Add an optional separate database connection for write operations like transforms or editable tables to keep the main Metabase connection read-only for analytics.
+---
+
+## Writeable connections
+
+{% include plans-blockquote.html feature="Writeable connections" is_plural=true%}
+
+_Admin > Databases > Writeable connection_
+
+Writeable connection is an optional connection that Metabase will use for functionality that involves writing into the database (for example, transforms or CSV uploads). This allows you to keep your main Metabase connection read-only, and only to switch to connection with write permissions for the operations that require write access.
+
+![Writeable connection in database settings](./images/writeable-connection.png)
+
+## How writeable connection works
+
+Writeable connection is an additional connection option added to an existing database connection.
+
+When a writeable connection is configured for a database, Metabase automatically switches between connections behind the scenes based on what people are doing with the database. The main connection is used for querying, and the writeable connection is used for [features that require writes](#features-that-use-writeable-connection).
+
+Writeable connection can use a different database user than the main connection, or even connect to a different database. For example, your main connection (used for querying) can go to a read replica, but the writeable connection can be to a primary database.
+
+## Add a writeable connection
+
+To add a writeable connection to a database:
+
+1. Go to **Admin > Databases**.
+2. Select the database to add a writeable connection to.
+3. Under **Writeable connection**, click **Add writeable connection**.
+4. Metabase will pre-fill some fields with information from your main connection. Input the credentials for the writeable connection.
+5. Click **Save**. Metabase will try to connect using the new connection.
+
+## Remove a writeable connection
+
+To remove a writeable connection from a database:
+
+1. Go to **Admin > Databases**.
+2. Select the database you want to remove the writeable connection from.
+3. Under **Writeable connection**, click **Remove writeable connection**.
+
+Once writeable connection is removed, Metabase will revert to using the main connection for [features that need write access](#features-that-use-writeable-connection), so your main connection will need to have appropriate database permissions, see [Users, roles, and privileges](./users-roles-privileges.md)
+
+## Features that use writeable connection
+
+When configured, writeable connection will be used for:
+
+- [Transforms](../data-studio/transforms/transforms-overview.md)
+- [CSV uploads](../databases/uploads.md)
+- [Editable tables](../data-modeling/editable-tables.md)
+- [Actions](../actions/introduction.md)
+- [Model persistence](../data-modeling/model-persistence.md)
+
+Your writeable connection will need to have the privileges required for these operations. See [Users, roles, and privileges](./users-roles-privileges.md).
+
+## Limitations
+
+- Writeable connection can't be used with [database routing](../permissions/database-routing.md).
+
+- Writeable connection must use the same database engine as the primary connection - so for example, if the primary connection is to a PostgreSQL database, the writeable connection must be to a PostgreSQL database as well.

docs/embedding/tenants.md

@@ -192,16 +192,18 @@ You can create tenant-level [user attributes](#tenant-attributes) which all user
 
 ![Edit tenant](./images/edit-tenant.png)
 
-To add a tenant attribute:
+There are three ways to add a tenant attribute, using the Metabase UI, the API, or JWT SSO.
+
+To create a tenant attribute manually using the Metabase UI:
 
 1. Go to **Admin settings > People**
 2. Select **Tenants** on the left sidebar.
 3. Click on **three dots** next to the tenant.
 4. Input the attribute key and value.
 
-Once you add a tenant attribute, all users of that tenant will inherit the attribute, but the value can be overridden for any particular user, see [Edit user attributes](../people-and-groups/managing.md#adding-a-user-attribute).
+For details on how to set tenant attributes using JWT tenant claims, see [Setting tenant attributes using tenant claims](#setting-tenant-attributes-using-tenant-claims) below.
 
-Currently, you can't assign custom tenant attributes with SSO. The only way to assign attributes is through the Metabase UI (but you can provision attributes for _individual users_ through SSO, see [JWT user attributes](../people-and-groups/authenticating-with-jwt.md)). However, if you're using user attribute to set permissions, then you can use the [special slug attribute](#special-tenant-slug-attribute) which Metabase creates automatically.
+Once you add a tenant attribute, all users of that tenant will inherit the attribute, but the value can be overridden for any particular user, see [Edit user attributes](../people-and-groups/managing.md#adding-a-user-attribute).
 
 ### Special tenant slug attribute
 
@@ -264,6 +266,24 @@ When user provisioning with JWT is enabled:
 2. If the tenant doesn't exist, Metabase automatically creates it. Metabase will use the value of the `@tenant` key (or your chosen assignment attribute) as the tenant slug.
 3. New users are automatically assigned to the tenant from their JWT.
 
+### Setting tenant attributes using tenant claims
+
+To create tenant attributes from JWT SSO, include a claim `@tenant.attributes`:
+
+```json
+{
+  "@tenant": "meowdern_solutions",
+  "@tenant.attributes": {
+    "industry": "cat food"
+  },
+  "email": "mittens@example.com",
+  "first_name": "Mister",
+  "last_name": "Mittens"
+}
+```
+
+If a tenant attribute with this name doesn't exist, Metabase will create the attribute and assign the value from the JWT claim. However, if the tenant attribute already exists, Metabase will **not** update the value.
+
 ### Troubleshooting JWT authentication with tenants
 
 Some common auth error messages and what they mean:
@@ -329,8 +349,7 @@ Permissions are granted to groups. Which permissions are available to each group
 ### Internal user collection permissions
 
 - Metabase Admins will have **Curate** access to all shared collections and all tenant collections.
-- Other internal groups can be granted **View** or **Curate** access to **shared collections**, see [Configuring shared collection permissions](#configuring-shared-collections-permissions).
-- Non-admin internal users will have **No** access to tenant-specific collections. Currently, this can't be configured.
+- Other internal groups and non-admin users have **No** access by default, but can be granted **View** or **Curate** access to **shared collections**. See [Configuring shared collection permissions](#configuring-shared-collections-permissions) for details.
 
 For configuring permissions to _internal_ collections for internal users, see [general docs on collection permissions](../permissions/collections.md).
 

docs/permissions/database-routing.md

@@ -21,13 +21,32 @@ Database routing is useful for:
 
 ## Database routing limitations
 
-> Database routing is **not supported** on ClickHouse, Oracle, Spark SQL, and Vertica.
+### Driver limitations
+
+Database routing is **not supported** on:
+
+- ClickHouse
+- Oracle
+- Spark SQL
+- Vertica.
 
 Different database have different setups, so _what_ you can route between (database, schema, data catalog, etc.) will differ slightly depending on which data warehouse you're using.
 
-- [Athena](../databases/connections/athena.md): Only routing between different connections is supported (e.g., different buckets, roles, or catalogs).
-- [BigQuery](../databases/connections/bigquery.md): Only routing between databases in different projects is supported.
-- [Databricks](../databases/connections/databricks.md): When multi-catalog is not enabled, you can route between catalogs on the same host. If multi-catalog is enabled, then you can only route between databases on separate hosts.
+- Athena: Only routing between different connections is supported (e.g., different buckets, roles, or catalogs).
+- BigQuery: Only routing between databases in different projects is supported.
+- Databricks: When multi-catalog is not enabled, you can route between catalogs on the same host. If multi-catalog is enabled, then you can only route between databases on separate hosts.
+
+### Functionality limitations
+
+Database routing **can't be used** on databases with:
+
+- [Writeable connections](../databases/writeable-connection.md)
+- [Editable tables](../data-modeling/editable-tables.md)
+- [Actions](../actions/introduction.md)
+- [CSV uploads](../databases/uploads.md)
+- [Model persistence](../data-modeling/model-persistence.md)
+
+With **guest embedding**, database queries will always be routed to the router database. This is because guest embed users don't have Metabase accounts, so there are no user attributes available to determine which destination database to route to.
 
 ## How database routing works
 

e2e/snapshot-creators/default.cy.snap.js

@@ -38,7 +38,7 @@ const {
 } = USER_GROUPS;
 const { admin } = USERS;
 
-const { IS_ENTERPRISE } = Cypress.env();
+const IS_ENTERPRISE = Cypress.expose("IS_ENTERPRISE");
 
 describe("snapshots", () => {
   describe("default", () => {

e2e/support/config.js

@@ -14,7 +14,6 @@ import {
   removeDirectory,
   verifyDownloadTasks,
 } from "./commands/downloads/downloadUtils";
-import webpackConfig from "./component-webpack.config";
 import * as dbTasks from "./db_tasks";
 import { signJwt } from "./helpers/e2e-jwt-tasks";
 
@@ -47,6 +46,25 @@ const assetsResolverPlugin = {
 };
 
 const defaultConfig = {
+  // Expose non-sensitive environment variables synchronously via Cypress.expose()
+  // These are safe to expose in the browser and are used for configuration
+  expose: {
+    CI: isCI,
+    IS_ENTERPRISE: isEnterprise,
+    MB_EDITION: process.env["MB_EDITION"],
+    ENABLE_NETWORK_THROTTLING: !!process.env["ENABLE_NETWORK_THROTTLING"],
+    SNOWPLOW_MICRO_URL: snowplowMicroUrl,
+    CLIENT_PORT: process.env["CLIENT_PORT"],
+    feHealthcheck: process.env["FE_HEALTHCHECK_URL"]
+      ? { enabled: true, url: process.env["FE_HEALTHCHECK_URL"] }
+      : undefined,
+  },
+
+  // Note: We can't set `allowCypressEnv: false` yet because @cypress/grep
+  // plugin still uses Cypress.env() internally
+  // FIXME: enable once we upgrade (DEV-1620)
+  // allowCypressEnv: false,
+
   // This is the functionality of the old cypress-plugins.js file
   setupNodeEvents(cypressOn, config) {
     // `on` is used to hook into various events Cypress emits
@@ -130,9 +148,6 @@ const defaultConfig = {
     config.env.grepFilterSpecs = true;
     config.env.grepOmitFiltered = true;
 
-    config.env.IS_ENTERPRISE = isEnterprise;
-    config.env.SNOWPLOW_MICRO_URL = snowplowMicroUrl;
-
     require("@cypress/grep/src/plugin")(config);
 
     if (isCI) {
@@ -156,9 +171,6 @@ const defaultConfig = {
   },
   baseUrl: `http://${BACKEND_HOST}:${BACKEND_PORT}`,
   defaultBrowser: process.env.CYPRESS_BROWSER ?? "chrome",
-  env: {
-    CI: isCI,
-  },
   supportFile: "e2e/support/cypress.js",
   chromeWebSecurity: false,
   modifyObstructiveCode: false,
@@ -201,34 +213,12 @@ const mainConfig = {
     },
   },
   retries: {
-    runMode: 1,
+    runMode: 2,
     openMode: 0,
   },
 };
 
-const embeddingSdkComponentTestConfig = {
-  ...defaultConfig,
-  baseUrl: undefined, // baseUrl should not be set for component tests,
-  defaultCommandTimeout: 10000,
-  requestTimeout: 10000,
-  video: false,
-  specPattern: "e2e/test-component/scenarios/embedding-sdk/**/*.cy.spec.tsx",
-  indexHtmlFile: "e2e/support/component-index.html",
-  supportFile: "e2e/support/component-cypress.js",
-
-  reporter: mainConfig.reporter,
-  reporterOptions: mainConfig.reporterOptions,
-  retries: mainConfig.retries,
-
-  devServer: {
-    framework: "react",
-    bundler: "webpack",
-    webpackConfig: webpackConfig,
-  },
-};
-
 module.exports = {
   defaultConfig,
   mainConfig,
-  embeddingSdkComponentTestConfig,
 };

e2e/support/cypress-embedding-sdk-component-test.config.js

@@ -1,4 +1,26 @@
-const { embeddingSdkComponentTestConfig } = require("./config");
+import webpackConfig from "./component-webpack.config";
+import { defaultConfig, mainConfig } from "./config";
+
+const embeddingSdkComponentTestConfig = {
+  ...defaultConfig,
+  baseUrl: undefined, // baseUrl should not be set for component tests,
+  defaultCommandTimeout: 10000,
+  requestTimeout: 10000,
+  video: false,
+  specPattern: "e2e/test-component/scenarios/embedding-sdk/**/*.cy.spec.tsx",
+  indexHtmlFile: "e2e/support/component-index.html",
+  supportFile: "e2e/support/component-cypress.js",
+
+  reporter: mainConfig.reporter,
+  reporterOptions: mainConfig.reporterOptions,
+  retries: mainConfig.retries,
+
+  devServer: {
+    framework: "react",
+    bundler: "webpack",
+    webpackConfig: webpackConfig,
+  },
+};
 
 module.exports = {
   component: {

e2e/support/cypress.js

@@ -8,8 +8,8 @@ import "cypress-real-events/support";
 import addContext from "mochawesome/addContext";
 import "./commands";
 
-const isCI = Cypress.env("CI");
-const isNetworkThrottlingEnabled = Cypress.env("ENABLE_NETWORK_THROTTLING");
+const isCI = Cypress.expose("CI");
+const isNetworkThrottlingEnabled = Cypress.expose("ENABLE_NETWORK_THROTTLING");
 
 // remove default html output on test failure
 configure({
@@ -31,7 +31,7 @@ Cypress.on("uncaught:exception", (err, runnable) => false);
 
 Cypress.on("test:before:run", () => {
   // Check wether FE is running in dev mode
-  const feHealthcheck = Cypress.env().feHealthcheck;
+  const feHealthcheck = Cypress.expose("feHealthcheck");
   if (feHealthcheck?.enabled) {
     fetch(feHealthcheck.url).catch(() =>
       alert(
@@ -160,7 +160,7 @@ if (isCI) {
 beforeEach(function () {
   const isCurrentTesOss =
     this.currentTest._testConfig.unverifiedTestConfig.tags === "@OSS";
-  const isBuildOss = Cypress.env("MB_EDITION") === "oss";
+  const isBuildOss = Cypress.expose("MB_EDITION") === "oss";
   const testName = this.currentTest.title;
   if (Cypress.config("isInteractive") && isCurrentTesOss && !isBuildOss) {
     console.log(

e2e/support/helpers/e2e-embedding-iframe-sdk-helpers.ts

@@ -15,7 +15,7 @@ import {
   mockAuthProviderAndJwtSignIn,
 } from "./embedding-sdk-testing";
 
-const { IS_ENTERPRISE } = Cypress.env();
+const IS_ENTERPRISE = Cypress.expose("IS_ENTERPRISE");
 
 const EMBED_JS_PATH = "http://localhost:4000/app/embed.js";
 
@@ -373,7 +373,7 @@ export const visitCustomHtmlPage = (
  * to point it to the rspack dev server.
  */
 export const mockEmbedJsToDevServer = () => {
-  if (Cypress.env("CI")) {
+  if (Cypress.expose("CI")) {
     // we don't need this logic in CI, let's skip the check to avoid slowing down the tests
     return;
   }

e2e/support/helpers/e2e-token-helpers.ts

@@ -6,7 +6,7 @@ import { match } from "ts-pattern";
  *
  * `IS_ENTERPRISE` means enterprise instance without a token and `isOSS` means open-source instance.
  */
-const { IS_ENTERPRISE } = Cypress.env();
+const IS_ENTERPRISE = Cypress.expose("IS_ENTERPRISE");
 
 const throwIfNotEnterprise = () => {
   if (!IS_ENTERPRISE) {
@@ -28,29 +28,25 @@ export const activateToken = (
     .with("pro-self-hosted", () => "MB_PRO_SELF_HOSTED_TOKEN")
     .exhaustive();
 
-  const fallbackToken =
-    tokenReference === "starter" ? "NO_FEATURES_TOKEN" : "ALL_FEATURES_TOKEN";
-  const token = Cypress.env(tokenReference) || Cypress.env(fallbackToken);
+  // Use cy.env() for sensitive token values (async API)
+  return cy.env([tokenReference]).then((envVars) => {
+    const token = envVars[tokenReference];
 
-  if (!token) {
-    const errorMessage = `
-    Please make sure you have correctly set the following tokens in your environment variables:
-      - CYPRESS_MB_ALL_FEATURES_TOKEN
-      - CYPRESS_MB_STARTER_CLOUD_TOKEN
-      - CYPRESS_MB_PRO_CLOUD_TOKEN
-      - CYPRESS_MB_PRO_SELF_HOSTED_TOKEN
-    `;
-    throw new Error(errorMessage);
-  }
+    if (!token) {
+      throw new Error(
+        `Missing CYPRESS_${tokenReference} environment variable for "${tokenName}" token`,
+      );
+    }
 
-  cy.log(`Set the "${tokenReference}" token`);
-  return cy.request({
-    method: "PUT",
-    url: "/api/setting/premium-embedding-token",
-    failOnStatusCode: false,
-    body: {
-      value: token,
-    },
+    cy.log(`Set the "${tokenName}" token`);
+    return cy.request({
+      method: "PUT",
+      url: "/api/setting/premium-embedding-token",
+      failOnStatusCode: false,
+      body: {
+        value: token,
+      },
+    });
   });
 };
 

e2e/support/helpers/e2e-transform-helpers.ts

@@ -1,6 +1,5 @@
 import { WRITABLE_DB_ID } from "e2e/support/cypress_data";
 import type {
-  AdvancedTransformType,
   Collection,
   CollectionId,
   PythonTransformTableAliases,
@@ -167,9 +166,7 @@ export function createSqlTransform({
   );
 }
 
-export function createAdvancedTransform({
-  name = "Python transform",
-  type = "python",
+export function createPythonTransform({
   body,
   sourceTables,
   targetTable,
@@ -177,8 +174,6 @@ export function createAdvancedTransform({
   tagIds,
   visitTransform,
 }: {
-  name?: string;
-  type?: AdvancedTransformType;
   body: string;
   sourceTables: PythonTransformTableAliases;
   targetTable: string;
@@ -188,9 +183,9 @@ export function createAdvancedTransform({
 }) {
   return createTransform(
     {
-      name,
+      name: "Python transform",
       source: {
-        type,
+        type: "python",
         "source-database": WRITABLE_DB_ID,
         "source-tables": sourceTables,
         body,

e2e/test-host-app/shared/compatibility.cy.spec.js

@@ -5,7 +5,7 @@ import {
 
 const TIMEOUT_MS = 40000;
 
-const CLIENT_PORT = Cypress.env("CLIENT_PORT");
+const CLIENT_PORT = Cypress.expose("CLIENT_PORT");
 const CLIENT_HOST = `http://localhost:${CLIENT_PORT}`;
 
 describe("Embedding SDK: shared Host Apps compatibility tests", () => {

e2e/test/scenarios/admin/databases.cy.spec.js

@@ -12,7 +12,7 @@ import { ORDERS_QUESTION_ID } from "e2e/support/cypress_sample_instance_data";
 import { visitDatabase, waitForDbSync } from "./helpers/e2e-database-helpers";
 
 const { H } = cy;
-const { IS_ENTERPRISE } = Cypress.env();
+const IS_ENTERPRISE = Cypress.expose("IS_ENTERPRISE");
 const { ORDERS_ID, ORDERS } = SAMPLE_DATABASE;
 
 describe(

e2e/test/scenarios/collections/instance-analytics.cy.spec.js

@@ -215,7 +215,7 @@ describe("scenarios > Metabase Analytics Collection (AuditV2) ", () => {
         H.visitModel(id);
       });
 
-      H.newButton(/SQL/).click();
+      H.newButton("SQL query").click();
 
       // sample DB should be the only one
       cy.findByTestId("gui-builder-data")

e2e/test/scenarios/data-studio/data-studio-library.cy.spec.ts

@@ -19,7 +19,23 @@ describe("scenarios > data studio > library", () => {
     cy.intercept("GET", "/api/collection/tree*").as("getCollectionTree");
 
     cy.log("Navigate to Data Studio Library");
-    H.DataModel.visitDataStudio();
+    /**
+     * Let's use the profile menu to navigate to Data Studio in this test, just
+     * to make sure it works, and to test the analytics event
+     */
+    cy.visit("/");
+    H.getProfileLink().click();
+    H.popover()
+      .findByText(/Data studio/)
+      .click();
+
+    cy.log(
+      "Verify tracking event when opening Data Studio from the profile menu",
+    );
+    H.expectUnstructuredSnowplowEvent({
+      event: "data_studio_opened",
+      triggered_from: "nav_menu",
+    });
     H.DataStudio.nav().findByLabelText("Library").click();
 
     cy.log("Create library via inline empty state");

e2e/test/scenarios/data-studio/table-collection-permissions.cy.spec.ts

@@ -481,7 +481,7 @@ describe("scenarios > data studio > table collection permissions", () => {
 
       cy.signIn("nodata");
       H.visitQuestionAdhoc(productsQuestionDetails);
-      assertPermissionError();
+      assertQueryPermissionError();
     });
 
     it("should not be able to create questions when all published tables are unpublished", () => {
@@ -509,7 +509,7 @@ describe("scenarios > data studio > table collection permissions", () => {
 
       cy.signIn("nodata");
       H.visitQuestionAdhoc(productsQuestionDetails);
-      assertPermissionError();
+      assertQueryPermissionError();
     });
 
     it("should not be able to create questions even if there are published tables", () => {
@@ -544,12 +544,6 @@ function librarySidebarSection() {
   return H.navigationSidebar().findByRole("section", { name: "Library" });
 }
 
-function assertPermissionError() {
-  H.main()
-    .findByText("Sorry, you don’t have permission to see that.")
-    .should("be.visible");
-}
-
 function assertQueryPermissionError() {
   H.main()
     .findByText("Sorry, you don't have permission to run this query.")

e2e/test/scenarios/data-studio/transforms/python-library.cy.spec.ts

@@ -21,7 +21,6 @@ describe("scenarios > data studio > transforms > python library", () => {
 
   it("should allow editing the python library", () => {
     H.DataStudio.Transforms.visit();
-    H.DataStudio.Transforms.list().findByText("Code library").click();
     cy.findByRole("link", { name: /Python library/ }).click();
 
     H.DataStudio.PythonLibrary.editor().should("be.visible");
@@ -31,7 +30,7 @@ describe("scenarios > data studio > transforms > python library", () => {
       .findByText(/# This is your Python library/)
       .should("be.visible");
     H.DataStudio.PythonLibrary.editor()
-      .findByText(/# You can add code here that can be reused/)
+      .findByText(/# You can add functions and classes here/)
       .should("be.visible");
 
     cy.log("modify and save the python library");
@@ -43,42 +42,7 @@ describe("scenarios > data studio > transforms > python library", () => {
     cy.findByRole("button", { name: "Save" }).should("be.visible").click();
 
     H.undoToast()
-      .findByText(/library saved/i)
-      .should("be.visible");
-
-    cy.log("refresh the page and check the content is persisted");
-    cy.url().reload();
-
-    H.DataStudio.PythonLibrary.editor()
-      .findByText(/hello world/)
-      .should("be.visible");
-  });
-
-  it("should allow editing the JavaScript library", () => {
-    H.DataStudio.Transforms.visit();
-    H.DataStudio.Transforms.list().findByText("Code library").click();
-    cy.findByRole("link", { name: /JavaScript library/ }).click();
-
-    H.DataStudio.PythonLibrary.editor().should("be.visible");
-
-    cy.log("make sure placeholder with help comment is displayed");
-    H.DataStudio.PythonLibrary.editor()
-      .findByText(/\/\/ This is your JavaScript library/)
-      .should("be.visible");
-    H.DataStudio.PythonLibrary.editor()
-      .findByText(/\/\/ You can add code here that can be reused/)
-      .should("be.visible");
-
-    cy.log("modify and save the JavaScript library");
-    H.DataStudio.PythonLibrary.editor()
-      .findByRole("textbox")
-      .click()
-      .realType("console.log('hello world');");
-
-    cy.findByRole("button", { name: "Save" }).should("be.visible").click();
-
-    H.undoToast()
-      .findByText(/library saved/i)
+      .findByText(/Python library saved/)
       .should("be.visible");
 
     cy.log("refresh the page and check the content is persisted");

e2e/test/scenarios/data-studio/transforms/reproductions.cy.spec.ts

@@ -18,7 +18,7 @@ describe("issue #68378", () => {
   it("should show empty schemas when picking a target schema (metabase#68378)", () => {
     visitTransformListPage();
     cy.button("Create a transform").click();
-    H.popover().findByText(/SQL/).click();
+    H.popover().findByText("SQL query").click();
     H.popover().findByText("Writable Postgres12").click();
     H.NativeEditor.type("SELECT 42", { allowFastSet: true }).blur();
 

e2e/test/scenarios/data-studio/transforms/template-tags.cy.spec.ts

@@ -37,7 +37,7 @@ describe("scenarios > admin > transforms", () => {
 
     visitTransformListPage();
     cy.button("Create a transform").click();
-    H.popover().findByText(/SQL/).click();
+    H.popover().findByText("SQL query").click();
     H.popover().findByText(DB_NAME).click();
 
     function testDataReference() {
@@ -270,7 +270,7 @@ describe("scenarios > admin > transforms", () => {
     cy.log("create a new transform");
     visitTransformListPage();
     cy.button("Create a transform").click();
-    H.popover().findByText(/SQL/).click();
+    H.popover().findByText("SQL query").click();
 
     cy.log("Add a query with multiple template tags");
     H.NativeEditor.clear()

e2e/test/scenarios/data-studio/transforms/transforms.cy.spec.ts

@@ -13,7 +13,6 @@ import {
 import { createLibraryWithItems } from "e2e/support/test-library-data";
 import { DataPermissionValue } from "metabase/admin/permissions/types";
 import type {
-  AdvancedTransformType,
   CardType,
   CollectionId,
   PythonTransformTableAliases,
@@ -123,6 +122,7 @@ describe("scenarios > admin > transforms", () => {
       });
 
       H.goToMainApp();
+      H.modal().button("Discard changes").click();
       H.newButton("Question").click();
 
       H.miniPicker().within(() => {
@@ -137,7 +137,7 @@ describe("scenarios > admin > transforms", () => {
       cy.log("create a new transform");
       visitTransformListPage();
       cy.button("Create a transform").click();
-      H.popover().findByText(/SQL/).click();
+      H.popover().findByText("SQL query").click();
       H.expectUnstructuredSnowplowEvent({
         event: "transform_create",
         event_detail: "native",
@@ -177,12 +177,10 @@ describe("scenarios > admin > transforms", () => {
         cy.log("create a new transform");
         visitTransformListPage();
         cy.button("Create a transform").click();
-        H.popover()
-          .findByText(/Python/)
-          .click();
+        H.popover().findByText("Python script").click();
         H.expectUnstructuredSnowplowEvent({
           event: "transform_create",
-          event_detail: "advanced",
+          event_detail: "python",
         });
 
         cy.findByTestId("python-transform-top-bar")
@@ -562,7 +560,7 @@ LIMIT
       visitTransformListPage();
       cy.button("Create a transform").click();
 
-      H.popover().findByText(/SQL/).click();
+      H.popover().findByText("SQL query").click();
 
       cy.findByTestId("gui-builder-data")
         .findByText("Writable Postgres12")
@@ -1618,7 +1616,7 @@ LIMIT
       cy.log("create a new transform");
       H.getTableId({ name: "Animals", databaseId: WRITABLE_DB_ID }).then(
         (id) => {
-          createAdvancedTransform({
+          createPythonTransform({
             body: dedent`
           import pandas as pd
 
@@ -1656,7 +1654,7 @@ LIMIT
         cy.log("create a new Python transform");
         H.getTableId({ name: "Animals", databaseId: WRITABLE_DB_ID }).then(
           (id) => {
-            createAdvancedTransform({
+            createPythonTransform({
               body: dedent`
               import pandas as pd
 
@@ -1696,7 +1694,7 @@ LIMIT
         cy.log("create a new Python transform");
         H.getTableId({ name: "Animals", databaseId: WRITABLE_DB_ID }).then(
           (id) => {
-            createAdvancedTransform({
+            createPythonTransform({
               body: dedent`
               import pandas as pd
 
@@ -1732,7 +1730,7 @@ LIMIT
         cy.log("create a new Python transform");
         H.getTableId({ name: "Animals", databaseId: WRITABLE_DB_ID }).then(
           (id) => {
-            createAdvancedTransform({
+            createPythonTransform({
               body: dedent`
               import pandas as pd
 
@@ -1827,7 +1825,7 @@ LIMIT
         cy.log("create a new SQL transform with a complex query");
         visitTransformListPage();
         cy.button("Create a transform").click();
-        H.popover().findByText(/SQL/).click();
+        H.popover().findByText("SQL query").click();
         H.popover().findByText(DB_NAME).click();
 
         H.NativeEditor.type(
@@ -2171,9 +2169,7 @@ LIMIT
 
         visitTransformListPage();
         cy.button("Create a transform").click();
-        H.popover()
-          .findByText(/Python/)
-          .click();
+        H.popover().findByText("Python script").click();
 
         cy.log("import common should be included by default");
         H.PythonEditor.value().should("contain", "import common");
@@ -2246,10 +2242,9 @@ LIMIT
       () => {
         visitTransformListPage();
         cy.button("Create a transform").click();
-        H.popover()
-          .findByText(/Python/)
-          .click();
+        H.popover().findByText("Python script").click();
         cy.get(".cm-clickable-token").should("be.visible").click();
+        H.modal().button("Discard changes").click();
         cy.url().should("include", "/data-studio/transforms/library/common.py");
         cy.findByTestId("python-library-header").should("be.visible");
       },
@@ -2264,9 +2259,7 @@ LIMIT
           cy.stub(win, "open").as("windowOpen");
         });
         cy.button("Create a transform").click();
-        H.popover()
-          .findByText(/Python/)
-          .click();
+        H.popover().findByText("Python script").click();
         cy.get(".cm-clickable-token").should("be.visible").click(H.holdMetaKey);
 
         cy.get("@windowOpen").should(
@@ -2284,9 +2277,7 @@ LIMIT
 
         visitTransformListPage();
         cy.button("Create a transform").click();
-        H.popover()
-          .findByText(/Python/)
-          .click();
+        H.popover().findByText("Python script").click();
 
         cy.log("import common should be included by default");
         H.PythonEditor.value().should("contain", "import common");
@@ -2587,8 +2578,8 @@ LIMIT
       getTransformsList().findByText("Name").click();
 
       getRowNames().should("deep.equal", [
-        "Code library",
         "Middle Transform",
+        "Python library",
         "Reports",
         "Alpha Transform",
         "Zebra Transform",
@@ -2600,8 +2591,8 @@ LIMIT
         "Zebra Transform",
         "Reports",
         "Alpha Transform",
+        "Python library",
         "Middle Transform",
-        "Code library",
       ]);
 
       cy.log("verify sorting by output table column ascending");
@@ -2609,7 +2600,7 @@ LIMIT
       getRowNames().should("deep.equal", [
         "Reports",
         "Alpha Transform",
-        "Code library",
+        "Python library",
         "Middle Transform",
         "Zebra Transform",
       ]);
@@ -2621,7 +2612,7 @@ LIMIT
         "Middle Transform",
         "Reports",
         "Alpha Transform",
-        "Code library",
+        "Python library",
       ]);
     });
 
@@ -2718,7 +2709,6 @@ LIMIT
 
       cy.log("Python library should be visible in the list");
       getTransformsList().within(() => {
-        cy.findByText("Code library").click();
         cy.findByText("Python library").should("be.visible");
       });
 
@@ -2814,13 +2804,12 @@ LIMIT
       cy.button("Create a transform").should("not.exist");
 
       cy.log("clicking Python library navigates to the library editor");
-      getTransformsList().findByText("Code library").click();
       getTransformsList().findByText("Python library").click();
 
       cy.log("python library editor is read-only");
       cy.url().should("include", "/data-studio/transforms/library/common.py");
       cy.findByRole("alert")
-        .contains(/The library is not editable/)
+        .contains(/The Python library is not editable/)
         .should("be.visible");
 
       H.DataStudio.PythonLibrary.editor().within(() => {
@@ -3679,7 +3668,7 @@ describe(
       H.expectNoBadSnowplowEvents();
     });
 
-    it("should be possible to test run a Python transform", () => {
+    it("should be possible to test run a Python script", () => {
       H.getTableId({ name: "Animals", databaseId: WRITABLE_DB_ID }).then(
         (id) => {
           createPythonLibrary(
@@ -3690,7 +3679,7 @@ describe(
             `,
           );
 
-          createAdvancedTransform({
+          createPythonTransform({
             body: dedent`
           import pandas as pd
           import common
@@ -3735,7 +3724,7 @@ describe(
     it("should display preview notice message", () => {
       H.getTableId({ name: "Animals", databaseId: WRITABLE_DB_ID }).then(
         (id) => {
-          createAdvancedTransform({
+          createPythonTransform({
             body: dedent`
               import pandas as pd
 
@@ -3786,7 +3775,7 @@ describe("scenarios > admin > transforms", () => {
     cy.log("create a new transform");
     visitTransformListPage();
     cy.button("Create a transform").click();
-    H.popover().findByText(/SQL/).click();
+    H.popover().findByText("SQL query").click();
 
     cy.findByTestId("gui-builder-data")
       .findByText("Select a database")
@@ -3797,9 +3786,7 @@ describe("scenarios > admin > transforms", () => {
     cy.log("create a new transform");
     visitTransformListPage();
     cy.button("Create a transform").click();
-    H.popover()
-      .findByText(/Python/)
-      .click();
+    H.popover().findByText("Python script").click();
 
     cy.findByTestId("python-transform-top-bar")
       .findByText("Select a database")
@@ -4015,9 +4002,7 @@ function createSqlTransform(opts: {
   });
 }
 
-function createAdvancedTransform(opts: {
-  name?: string;
-  type?: AdvancedTransformType;
+function createPythonTransform(opts: {
   body: string;
   sourceTables: PythonTransformTableAliases;
   targetTable?: string;
@@ -4025,7 +4010,7 @@ function createAdvancedTransform(opts: {
   tagIds?: TransformTagId[];
   visitTransform?: boolean;
 }) {
-  return H.createAdvancedTransform({
+  return H.createPythonTransform({
     targetTable: TARGET_TABLE,
     targetSchema: TARGET_SCHEMA,
     ...opts,
@@ -4070,7 +4055,7 @@ function getPythonDataPicker() {
 }
 
 function createPythonLibrary(path: string, source: string) {
-  cy.request("PUT", `/api/ee/advanced-transforms/library/${path}`, {
+  cy.request("PUT", `/api/ee/transforms-python/library/${path}`, {
     source,
   });
 }

e2e/test/scenarios/data-studio/workspaces.cy.spec.ts

@@ -1685,7 +1685,7 @@ describe("scenarios > data studio > workspaces", () => {
       cy.log("Create transform via UI with model reference");
       Workspaces.visitTransformListPage();
       cy.button("Create a transform").click();
-      H.popover().findByText(/SQL/).click();
+      H.popover().findByText("SQL query").click();
       H.popover().findByText("Writable Postgres12").click();
 
       cy.get("@modelId").then((modelId) => {
@@ -1763,7 +1763,7 @@ describe("scenarios > data studio > workspaces", () => {
           databaseId: WRITABLE_DB_ID,
           schema: "Schema B",
         }).then((id2) => {
-          createAdvancedTransform({
+          createPythonTransform({
             body: TEST_PYTHON_TRANSFORM,
             sourceTables: { foo: id1, bar: id2 },
             visitTransform: true,
@@ -1906,7 +1906,7 @@ describe("scenarios > data studio > workspaces", () => {
             databaseId: WRITABLE_DB_ID,
             schema: "Schema B",
           }).then((id2) => {
-            createAdvancedTransform({
+            createPythonTransform({
               body: TEST_PYTHON_TRANSFORM_MULTI_TABLE,
               sourceTables: { animals_a: id1, animals_b: id2 },
               visitTransform: true,
@@ -2269,7 +2269,7 @@ describe("scenarios > data studio > workspaces", () => {
     it("should not show error when editing a new transform in a workspace (GDGT-1445)", () => {
       Workspaces.visitTransformListPage();
       cy.findByLabelText("Create a transform").click();
-      H.popover().findByText(/SQL/).click();
+      H.popover().findByText("SQL query").click();
       // to avoid flakiness on editor's focus
       cy.findByTestId("native-query-top-bar").should(
         "contain.text",
@@ -2515,7 +2515,7 @@ function createTransforms({ visit }: { visit?: boolean } = { visit: false }) {
   });
 
   H.getTableId({ name: "Animals", databaseId: WRITABLE_DB_ID }).then((id) => {
-    createAdvancedTransform({
+    createPythonTransform({
       body: TEST_PYTHON_TRANSFORM,
       sourceTables: { foo: id },
     });
@@ -2561,7 +2561,7 @@ function createSqlTransform(opts: {
   });
 }
 
-function createAdvancedTransform(opts: {
+function createPythonTransform(opts: {
   body: string;
   sourceTables: PythonTransformTableAliases;
   targetTable?: string;
@@ -2569,7 +2569,7 @@ function createAdvancedTransform(opts: {
   tagIds?: TransformTagId[];
   visitTransform?: boolean;
 }) {
-  return H.createAdvancedTransform({
+  return H.createPythonTransform({
     targetTable: TARGET_TABLE_PYTHON,
     targetSchema: TARGET_SCHEMA,
     ...opts,

e2e/test/scenarios/embedding/sdk-iframe-embedding/guest-embed.cy.spec.ts

@@ -4,7 +4,7 @@ import { createQuestion, getSignedJwtForResource } from "e2e/support/helpers";
 const { H } = cy;
 const { ORDERS, ORDERS_ID } = SAMPLE_DATABASE;
 
-const { IS_ENTERPRISE } = Cypress.env();
+const IS_ENTERPRISE = Cypress.expose("IS_ENTERPRISE");
 const IS_OSS = !IS_ENTERPRISE;
 const MB_EDITION = IS_ENTERPRISE ? "ee" : "oss";
 

e2e/test/scenarios/metabot/transforms-codegen.cy.spec.ts

@@ -182,10 +182,7 @@ describe(
 
           cy.log("Should be able to visit and the see new transform");
           viewLastSuggestion();
-          cy.url().should(
-            "include",
-            "/data-studio/transforms/new/advanced/python",
-          );
+          cy.url().should("include", "/data-studio/transforms/new/python");
           assertEditorDiffState({ exists: false }); // nothing to diff, so we shouldn't show the UI for it
           assertEditorContent("python", "pd.DataFrame({'value': [1]})");
 
@@ -378,7 +375,7 @@ describe(
         it("should update existing Python transform via metabot", () => {
           H.getTableId({ name: SOURCE_TABLE, databaseId: WRITABLE_DB_ID }).then(
             (tableId) => {
-              H.createAdvancedTransform({
+              H.createPythonTransform({
                 targetTable: "transform_table",
                 targetSchema: "Schema A",
                 body: dedent`

e2e/test/scenarios/native/native-reproductions.cy.spec.ts

@@ -398,7 +398,7 @@ describe("issue 52806", () => {
 
   it("should remove parameter values from the URL when leaving the query builder and discarding changes (metabase#52806)", () => {
     cy.visit("/");
-    H.newButton(/SQL/).click();
+    H.newButton("SQL query").click();
     H.NativeEditor.focus().type("select {{x}}");
     cy.location().should((location) => expect(location.search).to.eq("?x="));
     cy.findByTestId("main-logo-link").click();
@@ -441,7 +441,7 @@ describe("issue 55951", () => {
       },
     );
 
-    H.newButton(/SQL/).click();
+    H.newButton("SQL query").click();
     H.popover()
       .should("be.visible")
       .within(() => {

e2e/test/scenarios/native/native.cy.spec.js

@@ -801,7 +801,7 @@ describe("no native access", { tags: ["@external", "@skip"] }, () => {
 
     cy.log("#32387");
     cy.findByRole("button", { name: /New/ }).click();
-    H.popover().findByText(/SQL/).click();
+    H.popover().findByText("SQL query").click();
 
     cy.wait("@database");
     cy.go("back");

e2e/test/scenarios/onboarding/setup/setup.cy.spec.ts

@@ -1,5 +1,5 @@
 const { H } = cy;
-const { IS_ENTERPRISE } = Cypress.env();
+const IS_ENTERPRISE = Cypress.expose("IS_ENTERPRISE");
 import { USERS } from "e2e/support/cypress_data";
 import { SUBSCRIBE_URL } from "metabase/setup/constants";
 
@@ -573,7 +573,10 @@ describe("scenarios > setup (EE)", () => {
 
       cy.findByText("Activate your commercial license").should("exist");
 
-      typeToken(Cypress.env("MB_STARTER_CLOUD_TOKEN"));
+      // Use cy.env() for sensitive token values (async API)
+      cy.env(["MB_STARTER_CLOUD_TOKEN"]).then(({ MB_STARTER_CLOUD_TOKEN }) => {
+        typeToken(MB_STARTER_CLOUD_TOKEN);
+      });
 
       cy.button("Activate").click();
 

e2e/test/scenarios/permissions/sandboxing/helpers/e2e-sandboxing-helpers.ts

@@ -551,7 +551,11 @@ export const getCardResponses = (questions: SimpleCollectionItem[]) => {
   expect(questions.length).to.be.greaterThan(0);
   return H.cypressWaitAll(
     questions.map((question) =>
-      cy.request<DatasetResponse>("POST", `/api/card/${question.id}/query`),
+      cy.request<DatasetResponse>({
+        method: "POST",
+        url: `/api/card/${question.id}/query`,
+        failOnStatusCode: false,
+      }),
     ),
   ).then((responses) => {
     return { responses, questions };

e2e/test/scenarios/visualizations-charts/visualizations-charts-reproductions.cy.spec.ts

@@ -850,6 +850,68 @@ describe("UXW-2696", () => {
   });
 });
 
+describe("issue 69882", () => {
+  const DOOHICKEY_BAR_COLOR = "#227FD2";
+  const GADGET_LINE_COLOR = "#689636";
+
+  const questionDetails: StructuredQuestionDetails = {
+    name: "69882",
+    display: "combo" as const,
+    query: {
+      "source-table": ORDERS_ID,
+      aggregation: [["count"]],
+      breakout: [
+        ["field", PRODUCTS.CATEGORY, { "source-field": ORDERS.PRODUCT_ID }],
+        ["field", ORDERS.CREATED_AT, { "temporal-unit": "year" }],
+      ],
+    },
+    visualization_settings: {
+      series_settings: {
+        Doohickey: {
+          color: DOOHICKEY_BAR_COLOR,
+          title: "_Doohickey",
+          display: "bar",
+        },
+        Gadget: {
+          color: GADGET_LINE_COLOR,
+          title: "_Gadget",
+          display: "line",
+        },
+      },
+      "graph.x_axis.scale": "timeseries",
+      "graph.dimensions": ["CREATED_AT", "CATEGORY"],
+      "graph.metrics": ["count"],
+    },
+  };
+
+  beforeEach(() => {
+    H.restore();
+    cy.signInAsAdmin();
+  });
+
+  it("should apply per-series display type on a question and dashcard (metabase#69882)", () => {
+    H.createQuestion(questionDetails, { visitQuestion: true, wrapId: true });
+
+    H.echartsContainer().should("be.visible");
+    H.chartPathWithFillColor(DOOHICKEY_BAR_COLOR).should("exist");
+    H.cartesianChartCircleWithColor(GADGET_LINE_COLOR);
+
+    H.createDashboard({ name: "Test Dashboard" }, { wrapId: true });
+    cy.get<number>("@questionId").then((cardId) => {
+      cy.get<number>("@dashboardId").then((dashboardId) => {
+        H.addQuestionToDashboard({ dashboardId, cardId });
+        H.visitDashboard(dashboardId);
+      });
+    });
+
+    cy.findByTestId("dashcard").within(() => {
+      H.echartsContainer().should("be.visible");
+      H.chartPathWithFillColor(DOOHICKEY_BAR_COLOR).should("exist");
+      H.cartesianChartCircleWithColor(GADGET_LINE_COLOR);
+    });
+  });
+});
+
 describe("issue #68819", () => {
   beforeEach(() => {
     H.restore();

e2e/test/scenarios/visualizations-tabular/drillthroughs/column_extract_drill.cy.spec.js

@@ -92,7 +92,7 @@ const DATE_QUESTION = {
   },
 };
 
-describe("extract action", () => {
+describe("extract action", { viewportWidth: 1600 }, () => {
   beforeEach(() => {
     H.restore();
     cy.signInAsAdmin();
@@ -256,7 +256,7 @@ describe("extract action", () => {
     });
 
     EMAIL_CASES.forEach(({ option, value, example }) => {
-      it(option, { viewportWidth: 1600 }, () => {
+      it(option, () => {
         H.openPeopleTable({ limit: 1 });
         extractColumnAndCheck({
           column: "Email",
@@ -395,7 +395,7 @@ function extractColumnAndCheck({
   }
 }
 
-describe("extract action", () => {
+describe("extract action", { viewportWidth: 1600 }, () => {
   beforeEach(() => {
     H.restore();
     H.resetSnowplow();

enterprise/backend/src/metabase_enterprise/api_routes/routes.clj

@@ -39,7 +39,7 @@
    [metabase-enterprise.stale.api]
    [metabase-enterprise.support-access-grants.api]
    [metabase-enterprise.tenants.api]
-   [metabase-enterprise.transforms-runner.api]
+   [metabase-enterprise.transforms-python.api]
    [metabase-enterprise.upload-management.api]
    [metabase-enterprise.workspaces.api]
    [metabase.api.macros :as api.macros]
@@ -130,7 +130,7 @@
    "/metabot-tools"                metabase-enterprise.metabot-v3.tools.api/routes
    "/metabot-v3"                   (premium-handler metabase-enterprise.metabot-v3.api/routes :metabot-v3)
    "/permission_debug"             (premium-handler metabase-enterprise.permission-debug.api/routes :advanced-permissions)
-   "/advanced-transforms" (premium-handler metabase-enterprise.transforms-runner.api/routes :transforms-python)
+   "/transforms-python"            (premium-handler metabase-enterprise.transforms-python.api/routes :transforms-python)
    "/scim"                         (premium-handler metabase-enterprise.scim.routes/routes :scim)
    "/semantic-search"              (premium-handler metabase-enterprise.semantic-search.api/routes :semantic-search)
    "/serialization"                (premium-handler metabase-enterprise.serialization.api/routes :serialization)

enterprise/backend/src/metabase_enterprise/core/init.clj

@@ -21,8 +21,5 @@
    [metabase-enterprise.sso.init]
    [metabase-enterprise.stale.init]
    [metabase-enterprise.support-access-grants.init]
-   [metabase-enterprise.transforms-clojure.init]
-   [metabase-enterprise.transforms-javascript.init]
    [metabase-enterprise.transforms-python.init]
-   [metabase-enterprise.transforms-runner.init]
    [metabase-enterprise.workspaces.init]))

enterprise/backend/src/metabase_enterprise/dependencies/api.clj

@@ -67,7 +67,7 @@
                                           (update :dashboard #(some-> % (select-keys [:id :name])))
                                           (update :document #(some-> % (select-keys [:id :name])))))))
                       broken-cards)
-     :bad_transforms (into [] broken-transforms)}))
+     :bad_transforms (into [] (filter mi/can-read?) broken-transforms)}))
 
 (api.macros/defendpoint :post "/check-card" :- ::broken-cards-response
   "Check a proposed edit to a card, and return the card IDs for those cards this edit will break."

enterprise/backend/src/metabase_enterprise/metabot_v3/tools/transforms.clj

@@ -1,7 +1,7 @@
 (ns metabase-enterprise.metabot-v3.tools.transforms
   (:require
    [metabase-enterprise.metabot-v3.tools.util :as metabot-v3.tools.u]
-   [metabase-enterprise.transforms-runner.api :as transforms-runner.api]
+   [metabase-enterprise.transforms-python.api :as transforms-python.api]
    [metabase.transforms.core :as transforms]
    [metabase.transforms.schema :as transforms.schema]
    [metabase.util.malli.registry :as mr]))
@@ -15,7 +15,7 @@
   (try
     {:structured_output
      (->> (transforms/get-transforms)
-          (into [] (comp (filter #(or (transforms/runner-transform? %)
+          (into [] (comp (filter #(or (transforms/python-transform? %)
                                       (transforms/native-query-transform? %)))
                          (filter :source_readable)
                          (map #(select-keys % [:id :entity_id :name :type :description :source])))))}
@@ -36,6 +36,6 @@
   [{:keys [path]}]
   (try
     {:structured_output
-     (transforms-runner.api/get-library-by-path path nil)}
+     (transforms-python.api/get-python-library-by-path path)}
     (catch Exception e
       (metabot-v3.tools.u/handle-agent-error e))))

enterprise/backend/src/metabase_enterprise/remote_sync/impl.clj

@@ -24,7 +24,7 @@
   (:import (metabase_enterprise.remote_sync.source.protocol SourceSnapshot)))
 
 (defn- snapshot-has-transforms?
-  "Checks if the snapshot contains any Transform, TransformLibrary entities,
+  "Checks if the snapshot contains any Transform, PythonLibrary entities,
    non-built-in TransformTags, or transforms-namespace collections.
    Used to auto-enable/disable remote-sync-transforms setting during import.
 
@@ -179,7 +179,7 @@
    Conflict types detected:
    - :entity-id-conflict - Items with existing entity IDs that are NOT already synced
    - :library-conflict - First import only, local Library exists, import has Library
-   - :transforms-not-enabled - Import has Transform/TransformTag/TransformLibrary, setting disabled
+   - :transforms-not-enabled - Import has Transform/TransformTag/PythonLibrary, setting disabled
    - :snippets-without-library - Import has NativeQuerySnippet, Library not remote-synced"
   [ingestable first-import?]
   (let [ingest-list (serialization/ingest-list ingestable)
@@ -242,7 +242,7 @@
               last-imported-version (remote-sync.task/last-version)
               first-import? (nil? last-imported-version)
               path-filters [#"collections/.*" #"databases/.*" #"actions/.*"
-                            #"transforms/.*" #"transform-libraries/.*" #"snippets/.*"]
+                            #"transforms/.*" #"python-libraries/.*" #"snippets/.*"]
               base-ingestable (source.p/->ingestable snapshot {:path-filters path-filters})
               has-transforms? (snapshot-has-transforms? base-ingestable)
               {:keys [conflicts summary]} (get-conflicts base-ingestable first-import?)

enterprise/backend/src/metabase_enterprise/remote_sync/spec.clj

@@ -15,7 +15,7 @@
    [clojure.string :as str]
    [clojure.walk :as walk]
    [metabase-enterprise.remote-sync.settings :as rs-settings]
-   [metabase-enterprise.transforms-runner.models.transform-library :as transform-library]
+   [metabase-enterprise.transforms-python.core :as transforms-python]
    [metabase.collections.core :as collections]
    [metabase.collections.models.collection :as collection]
    [metabase.models.serialization :as serdes]
@@ -71,7 +71,7 @@
    - :export-scope   - Export scope for query-export-roots:
                        :root-collections - Query root-level remote-synced + namespace collections (Collection)
                        :root-only        - Query root instances with collection_id = nil (Transform)
-                       :all              - Query all instances (TransformTag, TransformLibrary, NativeQuerySnippet)
+                       :all              - Query all instances (TransformTag, PythonLibrary, NativeQuerySnippet)
                        nil/:derived      - No root query; derived from other models via serdes/descendants
    - :enabled?       - true, or setting keyword (e.g., :remote-sync-transforms, :library-synced).
                        When :library-synced, uses the library-is-remote-synced? setting."
@@ -302,18 +302,18 @@
     :export-scope   :all  ; query for all instances
     :enabled?       :remote-sync-transforms}
 
-   :model/TransformLibrary
-   {:model-type "TransformLibrary"
-    :model-key :model/TransformLibrary
+   :model/PythonLibrary
+   {:model-type     "PythonLibrary"
+    :model-key      :model/PythonLibrary
     :identity       :entity-id
-    :events {:prefix :event/transform-library
-             :types  [:create :update]}  ; no delete - upsert only
+    :events         {:prefix :event/python-library
+                     :types  [:create :update]}  ; no delete - upsert only
     :eligibility    {:type    :setting
                      :setting :remote-sync-transforms}
     :archived-key   nil  ; no archived field
     :tracking       {:select-fields  [:path]
                      :field-mappings {:model_name :path}}
-    :conditions {:entity_id [:not-in (transform-library/all-builtin-entity-ids)]} ; exclude built-in libraries from sync
+    :conditions     {:entity_id [:not= transforms-python/builtin-entity-id]}  ; exclude built-in common.py from sync
     :removal        {:statuses               #{"removed" "delete"}  ; no scope-key = global deletion
                      :all-on-setting-disable :remote-sync-transforms}
     :export-scope   :all  ; query for all instances
@@ -506,7 +506,7 @@
 
 (defn- has-unsynced-entities-for-feature?
   "Returns true if any model in the feature group has local entities not tracked in RemoteSyncObject.
-   Excludes built-in TransformTags and built-in TransformLibrary entries from the count since they are
+   Excludes built-in TransformTags and the built-in PythonLibrary from the count since they are
    system-created and not user data. Namespace collections are not checked here because they are
    organizational containers, not user data that would be lost on import."
   [specs-for-feature]
@@ -516,7 +516,7 @@
                 ;; Exclude built-in entities from count (they are system-created, not user data)
                 local-count (case model-key
                               :model/TransformTag   (t2/count model-key :built_in_type nil)
-                              :model/TransformLibrary (t2/count model-key :entity_id [:not-in (transform-library/all-builtin-entity-ids)])
+                              :model/PythonLibrary  (t2/count model-key :entity_id [:not= transforms-python/builtin-entity-id])
                               (t2/count model-key))
                 synced-count (t2/count :model/RemoteSyncObject :model_type model-type)]
             (and (pos? local-count)

enterprise/backend/src/metabase_enterprise/serialization/v2/extract.clj

@@ -35,7 +35,7 @@
     (conj "Setting")
 
     (not (:no-transforms opts))
-    (conj "Transform" "TransformTag" "TransformJob" "TransformLibrary")))
+    (conj "Transform" "TransformTag" "TransformJob" "PythonLibrary")))
 
 (defn make-targets-of-type
   "Returns a targets seq with model type and given ids"

enterprise/backend/src/metabase_enterprise/serialization/v2/ingest.clj

@@ -74,7 +74,7 @@
       read-timestamps))
 
 (def legal-top-level-paths "Known top-level paths for directory with serialization output"
-  #{"actions" "collections" "databases" "glossary" "transform-libraries" "snippets" "transforms"})
+  #{"actions" "collections" "databases" "glossary" "python-libraries" "snippets" "transforms"})
 
 (defn- ingest-all [^File root-dir]
   ;; This returns a map {unlabeled-hierarchy [original-hierarchy File]}.

enterprise/backend/src/metabase_enterprise/serialization/v2/models.clj

@@ -35,7 +35,7 @@
           content
           ["FieldValues"
            "Metabot"
-           "TransformLibrary"
+           "PythonLibrary"
            "Setting"
            "Transform"
            "TransformJob"

enterprise/backend/src/metabase_enterprise/transforms_clojure/impl.clj

@@ -1,13 +0,0 @@
-(ns metabase-enterprise.transforms-clojure.impl
-  "Registers :clojure as a runner language. Loaded by transforms-clojure/init.clj for side effects.
-  See transforms.interface/register-runner! and transforms.interface/lang-config."
-  (:require
-   [metabase.transforms.interface :as transforms.i]))
-
-(transforms.i/register-runner! :clojure)
-
-(defmethod transforms.i/lang-config :clojure [_]
-  {:runtime "clojure"
-   :label "Clojure"
-   :timing-key :clojure-execution
-   :extension ".clj"})

enterprise/backend/src/metabase_enterprise/transforms_clojure/init.clj

@@ -1,3 +0,0 @@
-(ns metabase-enterprise.transforms-clojure.init
-  (:require
-   [metabase-enterprise.transforms-clojure.impl]))

enterprise/backend/src/metabase_enterprise/transforms_javascript/impl.clj

@@ -1,13 +0,0 @@
-(ns metabase-enterprise.transforms-javascript.impl
-  "Registers :javascript as a runner language. Loaded by transforms-javascript/init.clj for side effects.
-  See transforms.interface/register-runner! and transforms.interface/lang-config."
-  (:require
-   [metabase.transforms.interface :as transforms.i]))
-
-(transforms.i/register-runner! :javascript)
-
-(defmethod transforms.i/lang-config :javascript [_]
-  {:runtime "javascript"
-   :label "JavaScript"
-   :timing-key :javascript-execution
-   :extension ".js"})

enterprise/backend/src/metabase_enterprise/transforms_javascript/init.clj

@@ -1,3 +0,0 @@
-(ns metabase-enterprise.transforms-javascript.init
-  (:require
-   [metabase-enterprise.transforms-javascript.impl]))

enterprise/backend/src/metabase_enterprise/transforms_python/api.clj

@@ -1,9 +1,9 @@
-(ns metabase-enterprise.transforms-runner.api
+(ns metabase-enterprise.transforms-python.api
   (:require
    [clojure.string :as str]
-   [metabase-enterprise.transforms-runner.models.transform-library :as transform-library]
-   [metabase-enterprise.transforms-runner.runner :as runner]
-   [metabase-enterprise.transforms-runner.settings :as runner.settings]
+   [metabase-enterprise.transforms-python.models.python-library :as python-library]
+   [metabase-enterprise.transforms-python.python-runner :as python-runner]
+   [metabase-enterprise.transforms-python.settings :as transforms-python.settings]
    [metabase.api.common :as api]
    [metabase.api.macros :as api.macros]
    [metabase.api.routes.common :refer [+auth]]
@@ -13,10 +13,10 @@
    [metabase.util.malli.schema :as ms]
    [toucan2.core :as t2]))
 
-(defn get-library-by-path
-  "Get library details by path. Type is \"python\" (default), \"javascript\", or \"clojure\"."
-  [path type]
-  (-> (transform-library/get-library-by-path (or type "python") path)
+(defn get-python-library-by-path
+  "Get Python library details by path for use by other APIs."
+  [path]
+  (-> (python-library/get-python-library-by-path path)
       api/read-check
       (select-keys [:source :path :created_at :updated_at])))
 
@@ -25,26 +25,24 @@
 ;;
 #_{:clj-kondo/ignore [:metabase/validate-defendpoint-has-response-schema]}
 (api.macros/defendpoint :get "/library/:path"
-  "Get the library for user modules. Pass `?type=javascript` for JavaScript, defaults to Python."
+  "Get the Python library for user modules."
   [{:keys [path]} :- [:map [:path ms/NonBlankString]]
-   {:keys [type]} :- [:map [:type {:optional true} [:maybe :string]]]]
-  (get-library-by-path path type))
+   _query-params]
+  (get-python-library-by-path path))
 
 ;; TODO (Cam 2025-11-25) please add a response schema to this API endpoint, it makes it easier for our customers to
 ;; use our API + we will need it when we make auto-TypeScript-signature generation happen
 ;;
 #_{:clj-kondo/ignore [:metabase/validate-defendpoint-has-response-schema]}
 (api.macros/defendpoint :put "/library/:path"
-  "Update the library source code for user modules. Pass `type` in the body for JavaScript."
+  "Update the Python library source code for user modules."
   [{:keys [path]} :- [:map [:path ms/NonBlankString]]
    _query-params
-   body :- [:map
-            [:source :string]
-            [:type {:optional true} [:maybe :string]]]]
+   body :- [:map {:closed true}
+            [:source :string]]]
   ;; Check permission directly since this is an upsert endpoint - the library may not exist yet.
   (api/check-403 (perms/has-any-transforms-permission? api/*current-user-id*))
-  (let [{:keys [source type]} body]
-    (transform-library/update-library-source! (or type "python") path source)))
+  (python-library/update-python-library-source! path (:source body)))
 
 (api.macros/defendpoint :post "/test-run"
   :- [:map
@@ -53,13 +51,11 @@
       [:output {:optional true} [:map
                                  [:cols [:sequential [:map [:name :string]]]]
                                  [:rows [:sequential :any]]]]]
-  "Evaluate an ad-hoc transform on a sample of input data.
-  Pass `type` as \"javascript\" to run JavaScript; defaults to \"python\".
+  "Evaluate an ad-hoc python transform on a sample of input data.
   Intended for short runs for early feedback. Input/output/timeout limits apply."
   [_
    _
    {:keys [code
-           type
            source_tables
            output_row_limit
            per_input_row_limit]
@@ -67,21 +63,19 @@
            per_input_row_limit 100}}
    :- [:map
        [:code                                 :string]
-       [:type {:optional true} [:maybe :string]]
        [:source_tables                        [:map-of {:min 1} :string :int]]
        [:output_row_limit    {:optional true} [:and :int [:> 1] [:<= 100]]]
        [:per_input_row_limit {:optional true} [:and :int [:> 1] [:<= 100]]]]]
   (let [db-ids (t2/select-fn-set :db_id [:model/Table :db_id] :id [:in (vals source_tables)])]
-    (doseq [db-id db-ids]
-      (api/check-403 (perms/has-db-transforms-permission? api/*current-user-id* db-id))))
+    (api/check-400 (= (count db-ids) 1) (i18n/deferred-tru "All source tables must belong to the same database."))
+    (api/check-403 (perms/has-db-transforms-permission? api/*current-user-id* (first db-ids))))
   ;; NOTE: we do not test database support, as there is no write target.
-  (let [result (runner/execute-and-read-output!
+  (let [result (python-runner/execute-and-read-output!
                 {:code            code
                  :source-tables   source_tables
                  :per-input-limit per_input_row_limit
                  :row-limit       output_row_limit
-                 :timeout-secs (runner.settings/python-runner-test-run-timeout-seconds)
-                 :runtime (or type "python")})
+                 :timeout-secs    (transforms-python.settings/python-runner-test-run-timeout-seconds)})
         logs   (str/join "\n" (map :message (:logs result)))]
     (if (= :succeeded (:status result))
       {:logs   logs
@@ -91,6 +85,6 @@
        :error {:message (:message result)}})))
 
 (def ^{:arglists '([request respond raise])} routes
-  "`/api/ee/advanced-transforms` routes."
+  "`/api/ee/transforms-python` routes."
   (handlers/routes
    (api.macros/ns-handler *ns* +auth)))

enterprise/backend/src/metabase_enterprise/transforms_python/core.clj

@@ -0,0 +1,10 @@
+(ns metabase-enterprise.transforms-python.core
+  "Core namespace for Python transforms functionality.
+   Re-exports commonly used vars from internal namespaces."
+  (:require
+   [metabase-enterprise.transforms-python.models.python-library :as python-library]
+   [potemkin :as p]))
+
+(p/import-vars
+ [python-library
+  builtin-entity-id])

enterprise/backend/src/metabase_enterprise/transforms_python/execute.clj

@@ -0,0 +1,369 @@
+(ns metabase-enterprise.transforms-python.execute
+  (:require
+   [clojure.core.async :as a]
+   [clojure.java.io :as io]
+   [clojure.string :as str]
+   [metabase-enterprise.transforms-python.python-runner :as python-runner]
+   [metabase-enterprise.transforms-python.s3 :as s3]
+   [metabase-enterprise.transforms-python.settings :as transforms-python.settings]
+   [metabase.app-db.core :as app-db]
+   [metabase.driver :as driver]
+   [metabase.driver.connection :as driver.conn]
+   [metabase.driver.util :as driver.u]
+   [metabase.transforms.core :as transforms]
+   [metabase.transforms.instrumentation :as transforms.instrumentation]
+   [metabase.transforms.interface :as transforms.i]
+   [metabase.transforms.util :as transforms.util]
+   [metabase.util :as u]
+   [metabase.util.format :as u.format]
+   [metabase.util.i18n :as i18n]
+   [metabase.util.jvm :as u.jvm]
+   [metabase.util.log :as log]
+   [toucan2.core :as t2])
+  (:import
+   (java.io Closeable)
+   (java.net SocketException)
+   (java.nio.file Files)
+   (java.nio.file.attribute FileAttribute)
+   (java.time Duration)))
+
+(set! *warn-on-reflection* true)
+
+(defn- empty-message-log
+  "Returns a new message log.
+
+  Python transforms produce output while they run. The message log is a stateful structure that buffers log outputs during the run.
+  The goal is to assist in debugging in the case of python runtime or syntax errors, and to provide some immediate feedback for longer running transforms.
+
+  Important Note: in memory buffering is considered acceptable as we expect /logs output to be truncated or small.
+
+  The intended use of this structure is to produce a string suitable for `transform_run.message`, saving it every few seconds."
+  []
+  (atom {:pre-python  []                                    ; log! outputs previous to the python execution, i.e table read progress
+         :python      nil                                   ; events json structured logs from the /logs endpoint
+         :post-python []}))                                 ; log! outputs after the python execution, i.e. output reads, writes to target
+
+(defn- log!
+  "Appends a string to the message log, the string is user facing and should be suitable for presentation as part of the `transform_run.message` field."
+  [message-log s]
+  (swap! message-log (fn [m] (update m (if (:python m) :post-python :pre-python) conj s)))
+  nil)
+
+(defn- replace-python-logs!
+  "Called when events output is received from a /logs call to copy those events into the message-log."
+  [message-log events]
+  (swap! message-log assoc :python events)
+  nil)
+
+(defn- message-log->transform-run-message
+  "Presents the content of the log as a string suitable for storage in transform_run.message."
+  [message-log]
+  (let [{:keys [pre-python python post-python]} @message-log]
+    (str/join "\n" (concat pre-python
+                           (map :message python)
+                           post-python))))
+
+(defn- save-log-to-transform-run-message!
+  "Saves the content of the log into the transform_run's message field. Lossy, string contains user facing information only."
+  [run-id message-log]
+  (t2/update! :model/TransformRun
+              :id run-id
+              {:message (message-log->transform-run-message message-log)}))
+
+(def ^:private ^Duration python-message-loop-sleep-duration
+  (Duration/ofMillis 1000))
+
+(defn- python-message-update-loop!
+  "Block while relevant log data is replicated from the runner into the message log.
+  When new logs are received, the log data will be flushed to the transform_run.message field as a string.
+
+  The loop will exit on error or when interrupted."
+  [run-id message-log]
+  (try
+    (loop []
+      (if (.isInterrupted (Thread/currentThread))
+        (log/debug "Message update loop interrupted")
+        (do (let [sleep-ms (.toMillis python-message-loop-sleep-duration)]
+              (when (pos? sleep-ms) (Thread/sleep sleep-ms)))
+            (let [{:keys [status body]} (python-runner/get-logs run-id)]
+              (cond
+                (<= 200 status 299)
+                (let [{:keys [execution_id events]} body]
+                  (if-not (= run-id execution_id)
+                    (do (log/debugf "Run id did not match expected: %s actual: %s" run-id execution_id)
+                        (recur))
+                    (do
+                      (replace-python-logs! message-log events)
+                      (save-log-to-transform-run-message! run-id message-log)
+                      (recur))))
+                (= 404 status)
+                (do
+                  (log/debugf "No logs yet (or run finished), run-id: %s" run-id)
+                  (recur))
+                :else
+                (do
+                  (log/warnf "Unexpected status polling for logs %s %s, run-id: %s" status body run-id)
+                  (log/debug "Exiting due to poll error")))))))
+    (catch SocketException se (when-not (= "Closed by interrupt" (ex-message se)) (throw se)))
+    (catch InterruptedException _)
+    (catch Throwable e
+      (log/errorf e "An exception was caught during msg update loop, run-id: %s" run-id))))
+
+(defn- open-python-message-update-future! ^Closeable [run-id message-log]
+  (let [cleanup (fn [fut]
+                  (future-cancel fut)
+                  (if (= ::timeout (try (deref fut 10000 ::timeout) (catch Throwable _)))
+                    (log/fatalf "Log polling task did not respond to interrupt, run-id: %s" run-id)
+                    (log/debugf "Log polling task done, run-id: %s" run-id)))
+        fut     (u.jvm/in-virtual-thread*
+                 (python-message-update-loop! run-id message-log))]
+    (reify Closeable
+      (close [_] (cleanup fut)))))
+
+(defn- table-schema [table-name metadata]
+  {:name (if (keyword? table-name) table-name (keyword table-name))
+   :columns (mapv (fn [{:keys [name base_type #_database_type]}]
+                    {:name name
+                     :type (python-runner/restricted-insert-type base_type)
+                     ;; :database-type database_type
+                     :nullable? true})
+                  (:fields metadata))})
+
+(defn- insert-data!
+  "Insert data from source into an existing table."
+  [driver db-id table-schema data-source]
+  (let [data-source (assoc data-source :table-schema table-schema)]
+    (driver/insert-from-source! driver db-id table-schema data-source)))
+
+(defn- create-table-and-insert-data!
+  "Create a table from metadata and insert data from source."
+  [driver db-id table-schema data-source]
+  (transforms.util/create-table-from-schema! driver db-id table-schema)
+  (insert-data! driver db-id table-schema data-source))
+
+(defn- transfer-with-rename-tables-strategy!
+  "Transfer data using the rename-tables*! multimethod with atomicity guarantees.
+   Creates new table, then atomically renames target->old and new->target, then drops old."
+  [driver db-id table-name metadata data-source]
+  (let [source-table-name (driver.u/temp-table-name driver table-name)
+        temp-table-name   (driver.u/temp-table-name driver table-name)]
+    (log/info "Using rename-tables strategy with atomicity guarantees")
+    (try
+
+      (create-table-and-insert-data! driver db-id (table-schema source-table-name metadata) data-source)
+      (transforms.util/rename-tables! driver db-id {table-name temp-table-name
+                                                    source-table-name table-name})
+      (transforms.util/drop-table! driver db-id temp-table-name)
+
+      (catch Exception e
+        (log/error e "Failed to transfer data using rename-tables strategy")
+        (try
+          (transforms.util/drop-table! driver db-id source-table-name)
+          (catch Exception _))
+        (throw e)))))
+
+(defn- transfer-with-create-drop-rename-strategy!
+  "Transfer data using create + drop + rename to minimize time without data.
+   Creates new table, drops old table, then renames new->target."
+  [driver db-id table-name metadata data-source]
+  (let [source-table-name (driver.u/temp-table-name driver table-name)]
+    (log/info "Using create-drop-rename strategy to minimize downtime")
+    (try
+
+      (create-table-and-insert-data! driver db-id (table-schema source-table-name metadata) data-source)
+      (transforms.util/drop-table! driver db-id table-name)
+      (driver/rename-table! driver db-id source-table-name table-name)
+
+      (catch Exception e
+        (log/error e "Failed to transfer data using create-drop-rename strategy")
+        (try
+          (transforms.util/drop-table! driver db-id source-table-name)
+          (catch Exception _))
+        (throw e)))))
+
+(defn- transfer-with-drop-create-fallback-strategy!
+  "Transfer data using drop + create fallback strategy.
+   Drops old table, then creates new table directly with target name."
+  [driver db-id table-name metadata data-source]
+  (log/info "Using drop-create fallback strategy")
+  (try
+
+    (transforms.util/drop-table! driver db-id table-name)
+    (create-table-and-insert-data! driver db-id (table-schema table-name metadata) data-source)
+
+    (catch Exception e
+      (log/error e "Failed to transfer data using drop-create fallback strategy")
+      (throw e))))
+
+(defmulti ^:private transfer-file-to-db
+  {:arglists '([driver db transform metadata temp-file])}
+  (fn [_ _ transform _ _] (-> transform :target :type keyword)))
+
+(defmethod transfer-file-to-db :table-incremental
+  [driver {db-id :id}
+   {:keys [target] :as transform}
+   metadata temp-file]
+  (let [table-name (transforms.util/qualified-table-name driver target)
+        table-exists? (transforms.util/target-table-exists? transform)
+        data-source {:type :jsonl-file
+                     :file temp-file}]
+
+    ;; once we have more than just append, dispatch on :target-incremental-strategy
+
+    (if (not table-exists?)
+      (do
+        (log/info "New table")
+        (create-table-and-insert-data! driver db-id (table-schema table-name metadata) data-source))
+      (insert-data! driver db-id (table-schema table-name metadata) data-source))))
+
+(defmethod transfer-file-to-db :table
+  [driver {db-id :id :as db}
+   {:keys [target] :as transform}
+   metadata temp-file]
+  (let [table-name (transforms.util/qualified-table-name driver target)
+        table-exists? (transforms.util/target-table-exists? transform)
+        data-source {:type :jsonl-file
+                     :file temp-file}]
+    (cond
+      (not table-exists?)
+      (do
+        (log/info "New table")
+        (create-table-and-insert-data! driver db-id (table-schema table-name metadata) data-source))
+
+      (driver.u/supports? driver :atomic-renames db)
+      (transfer-with-rename-tables-strategy! driver db-id table-name metadata data-source)
+
+      (driver.u/supports? driver :rename db)
+      (transfer-with-create-drop-rename-strategy! driver db-id table-name metadata data-source)
+
+      :else
+      (transfer-with-drop-create-fallback-strategy! driver db-id table-name metadata data-source))))
+
+(defn- start-cancellation-process!
+  "Starts a core.async process that optimistically sends a cancellation request to the python executor if cancel-chan receives a value.
+  Returns a channel that will receive either the async http call j.u.c.FutureTask in the case of cancellation, or nil when the cancel-chan is closed."
+  [server-url run-id cancel-chan]
+  (a/go (when (a/<! cancel-chan)
+          (python-runner/cancel-python-code-http-call! server-url run-id))))
+
+(defn- run-python-transform! [{:keys [source] :as transform} db run-id cancel-chan message-log]
+  ;; Resolve name-based source table refs to table IDs (throws if any not found)
+  (let [resolved-source-tables (transforms.util/resolve-source-tables (:source-tables source))]
+    ;; TODO restructure things such that s3 can we swapped out for other transfer mechanisms
+    (with-open [^Closeable log-future-ref
+                (if (app-db/in-transaction?)
+                  ;; if in a transaction (such as under mt/with-temp), it is not safe to poll for logs (close race / contention)
+                  ;; tests that want to test async log behaviour should opt out of thread-local test helpers
+                  (reify Closeable (close [_]))
+                  (open-python-message-update-future! run-id message-log))
+                shared-storage-ref (s3/open-shared-storage! resolved-source-tables)]
+      (let [driver          (:engine db)
+            server-url      (transforms-python.settings/python-runner-url)
+            _               (python-runner/copy-tables-to-s3! {:run-id         run-id
+                                                               :shared-storage @shared-storage-ref
+                                                               :source         (assoc source :source-tables resolved-source-tables)
+                                                               :cancel-chan    cancel-chan
+                                                               :limit          (:limit source)
+                                                               :transform-id   (:id transform)})
+            _               (start-cancellation-process! server-url run-id cancel-chan) ; inherits lifetime of cancel-chan
+            {:keys [status body] :as response}
+            (python-runner/execute-python-code-http-call!
+             {:server-url     server-url
+              :code           (:body source)
+              :run-id         run-id
+              :table-name->id resolved-source-tables
+              :shared-storage @shared-storage-ref})
+
+            output-manifest (python-runner/read-output-manifest @shared-storage-ref)
+            events          (python-runner/read-events @shared-storage-ref)]
+        (.close ^Closeable log-future-ref)                 ; early close to force any writes to flush
+        (replace-python-logs! message-log events)
+        (if (not= 200 status)
+          (do
+            (when (:timeout body)
+              (transforms/timeout-run! run-id))
+            (throw (ex-info "Python runner call failed"
+                            {:transform-message (i18n/tru "Python execution failure (exit code {0})" (:exit_code body "?"))
+                             :status-code       400
+                             :api-status-code   status
+                             :body              body
+                             :events            events})))
+          (try
+            (let [temp-path (Files/createTempFile "transform-output-" ".jsonl" (u/varargs FileAttribute))
+                  temp-file (.toFile temp-path)]
+              (when-not (seq (:fields output-manifest))
+                (throw (ex-info "No fields in metadata"
+                                {:metadata               output-manifest
+                                 :raw-body               body
+                                 :events                 events})))
+              (try
+                (with-open [in (python-runner/open-output @shared-storage-ref)]
+                  (io/copy in temp-file))
+                (let [file-size (.length temp-file)]
+                  (transforms.instrumentation/with-stage-timing [run-id [:import :file-to-dwh]]
+                    (transfer-file-to-db driver db transform output-manifest temp-file))
+                  (transforms.instrumentation/record-data-transfer! run-id :file-to-dwh file-size nil))
+                (finally
+                  (.delete temp-file))))
+            response
+            (catch Exception e
+              (log/error e "Failed to to create resulting table")
+              (throw (ex-info "Failed to create the resulting table"
+                              {:transform-message (or (:transform-message (ex-data e))
+                                                      ;; TODO keeping messaging the same at this level
+                                                      ;;  should be more specific in underlying calls
+                                                      (i18n/tru "Failed to create the resulting table"))}
+                              e)))))))))
+
+(defn- exceptional-run-message [message-log ex]
+  (str/join "\n" (remove str/blank? [(message-log->transform-run-message message-log)
+                                     (or (:transform-message (ex-data ex))
+                                         (if (instance? InterruptedException ex)
+                                           (i18n/tru "Transform interrupted")
+                                           (i18n/tru "Something went wrong")))])))
+
+(defn execute-python-transform!
+  "Execute a Python transform by calling the python runner.
+
+  Blocks until the transform returns."
+  [transform {:keys [run-method start-promise user-id]}]
+  (assert (transforms.util/python-transform? transform) "Transform must be a python transform")
+  (try
+    (let [message-log                                                (empty-message-log)
+          {:keys [target owner_user_id creator_id] transform-id :id} transform
+          {driver :engine :as db}                                    (t2/select-one :model/Database (transforms.i/target-db-id transform))
+          ;; For manual runs, use the triggering user; for cron, use owner/creator
+          run-user-id                                                (if (and (= run-method :manual) user-id)
+                                                                       user-id
+                                                                       (or owner_user_id creator_id))
+          {run-id :id}                                               (transforms.util/try-start-unless-already-running transform-id run-method run-user-id)]
+      (some-> start-promise (deliver [:started run-id]))
+      (log! message-log (i18n/tru "Executing Python transform"))
+      (driver.conn/with-write-connection
+        (log/info "Executing Python transform" transform-id "with target" (pr-str target)
+                  (when (driver.conn/write-connection-requested?)
+                    " using write connection"))
+        (let [start-ms          (u/start-timer)
+              conn-spec         (driver/connection-spec driver db)
+              transform-details {:db-id          (:id db)
+                                 :transform-id   transform-id
+                                 :transform-type (keyword (:type target))
+                                 :conn-spec      conn-spec
+                                 :output-schema  (:schema target)
+                                 :output-table   (transforms.util/qualified-table-name driver target)}
+              run-fn            (fn [cancel-chan]
+                                  (run-python-transform! transform db run-id cancel-chan message-log)
+                                  (log! message-log (i18n/tru "Python execution finished successfully in {0}" (u.format/format-milliseconds (u/since-ms start-ms))))
+                                  (save-log-to-transform-run-message! run-id message-log))
+              ex-message-fn     #(exceptional-run-message message-log %)
+              result            (transforms.instrumentation/with-stage-timing [run-id [:computation :python-execution]]
+                                  (transforms.util/run-cancelable-transform! run-id driver transform-details run-fn :ex-message-fn ex-message-fn))]
+          (transforms.util/handle-transform-complete!
+           :run-id run-id
+           :transform transform
+           :db db)
+          {:run_id run-id
+           :result result})))
+    (catch Throwable t
+      (log/error t "Error executing Python transform")
+      (throw t))))

enterprise/backend/src/metabase_enterprise/transforms_python/impl.clj

@@ -1,13 +1,39 @@
 (ns metabase-enterprise.transforms-python.impl
-  "Registers :python as a runner language. Loaded by transforms-python/init.clj for side effects.
-  See transforms.interface/register-runner! and transforms.interface/lang-config."
   (:require
+   [metabase-enterprise.transforms-python.execute :as transforms-python.execute]
    [metabase.transforms.interface :as transforms.i]))
 
-(transforms.i/register-runner! :python)
+(defmethod transforms.i/target-db-id :python
+  [transform]
+  (-> transform :target :database))
 
-(defmethod transforms.i/lang-config :python [_]
-  {:runtime "python"
-   :label "Python"
-   :timing-key :python-execution
-   :extension ".py"})
+(defmethod transforms.i/source-db-id :python
+  [transform]
+  (-> transform :source :source-database))
+
+#_{:clj-kondo/ignore [:discouraged-var]}
+(defmethod transforms.i/execute! :python
+  [transform options]
+  (transforms-python.execute/execute-python-transform! transform options))
+
+(defn- source-table-value->dependency
+  "Convert a source table value (int or ref map) to a dependency map."
+  [v]
+  (cond
+    ;; Integer table ID - direct table dependency
+    (int? v)
+    {:table v}
+
+    ;; Map with resolved table_id - direct table dependency
+    (:table_id v)
+    {:table (:table_id v)}
+
+    ;; Map without table_id - return ref for ordering system to resolve
+    :else
+    {:table-ref (select-keys v [:database_id :schema :table])}))
+
+(defmethod transforms.i/table-dependencies :python
+  [transform]
+  (into #{}
+        (map source-table-value->dependency)
+        (vals (get-in transform [:source :source-tables]))))

enterprise/backend/src/metabase_enterprise/transforms_python/init.clj

@@ -1,3 +1,4 @@
 (ns metabase-enterprise.transforms-python.init
   (:require
-   [metabase-enterprise.transforms-python.impl]))
+   [metabase-enterprise.transforms-python.impl]
+   [metabase-enterprise.transforms-python.settings]))

enterprise/backend/src/metabase_enterprise/transforms_python/models/python_library.clj

@@ -0,0 +1,115 @@
+(ns metabase-enterprise.transforms-python.models.python-library
+  (:require
+   [metabase.api.common :as api]
+   [metabase.app-db.core :as app-db]
+   [metabase.events.core :as events]
+   [metabase.models.interface :as mi]
+   [metabase.models.serialization :as serdes]
+   [metabase.permissions.core :as perms]
+   [metabase.util.i18n :refer [tru]]
+   [methodical.core :as methodical]
+   [toucan2.core :as t2]))
+
+(set! *warn-on-reflection* true)
+
+(methodical/defmethod t2/table-name :model/PythonLibrary [_model] :python_library)
+
+(defn- normalize-path
+  "Ensure the path ends with .py extension."
+  [path]
+  (if (.endsWith ^String path ".py")
+    path
+    (str path ".py")))
+
+(t2/define-before-insert :model/PythonLibrary
+  [library]
+  (update library :path normalize-path))
+
+(t2/define-before-update :model/PythonLibrary
+  [library]
+  (if (:path library)
+    (update library :path normalize-path)
+    library))
+
+(doseq [trait [:metabase/model :hook/timestamped? :hook/entity-id]]
+  (derive :model/PythonLibrary trait))
+
+(defmethod mi/can-read? :model/PythonLibrary
+  ([_instance]
+   (perms/has-any-transforms-permission? api/*current-user-id*))
+  ([_model _pk]
+   (perms/has-any-transforms-permission? api/*current-user-id*)))
+
+(defmethod mi/can-write? :model/PythonLibrary
+  ([_instance]
+   (perms/has-any-transforms-permission? api/*current-user-id*))
+  ([_model _pk]
+   (perms/has-any-transforms-permission? api/*current-user-id*)))
+
+(def ^:private allowed-paths
+  "Set of allowed library paths. Currently only 'common' is supported."
+  #{"common.py"})
+
+(def builtin-entity-id
+  "The entity_id of the built-in common.py PythonLibrary created by migration.
+   Used to protect it from deletion during remote-sync import."
+  "cWWH9qJPvHNB3rP2vLZrK")
+
+(defn- validate-path!
+  "Validates that the given path is allowed. Throws an exception if not."
+  [path]
+  (let [normalized-path (normalize-path path)]
+    (when-not (contains? allowed-paths normalized-path)
+      (throw (ex-info (tru "Invalid library path. Only ''common'' is currently supported.")
+                      {:status-code 400
+                       :path normalized-path
+                       :allowed-paths allowed-paths})))))
+
+(defn get-python-library-by-path
+  "Get the Python library by path."
+  [path]
+  (let [normalized-path (normalize-path path)]
+    (validate-path! normalized-path)
+    (t2/select-one :model/PythonLibrary :path normalized-path)))
+
+(defn update-python-library-source!
+  "Update the Python library source code. Creates a new record if none exists. Returns the updated library."
+  [path source]
+  (let [normalized-path (normalize-path path)]
+    (validate-path! normalized-path)
+    (let [id (app-db/update-or-insert! :model/PythonLibrary
+                                       {:path normalized-path}
+                                       (constantly {:path normalized-path :source source}))]
+      (t2/select-one :model/PythonLibrary id))))
+
+;;; ------------------------------------------------- Serialization --------------------------------------------------
+
+(defmethod serdes/make-spec "PythonLibrary"
+  [_model-name _opts]
+  {:copy      [:path :source :entity_id]
+   :transform {:created_at (serdes/date)}})
+
+(defmethod serdes/hash-fields :model/PythonLibrary
+  [_model]
+  [:path])
+
+(defmethod serdes/storage-path "PythonLibrary" [entity _ctx]
+  (let [{:keys [id label]} (-> entity serdes/path last)]
+    ["python-libraries" (serdes/storage-leaf-file-name id label)]))
+
+;;; ------------------------------------------------ Event Hooks -----------------------------------------------------
+
+;; Event type hierarchy for remote-sync tracking
+(derive ::event :metabase/event)
+(doseq [e [:event/python-library-create :event/python-library-update :event/python-library-delete]]
+  (derive e ::event))
+
+(t2/define-after-insert :model/PythonLibrary
+  [library]
+  (events/publish-event! :event/python-library-create {:object library})
+  library)
+
+(t2/define-after-update :model/PythonLibrary
+  [library]
+  (events/publish-event! :event/python-library-update {:object library})
+  library)

enterprise/backend/src/metabase_enterprise/transforms_python/python_runner.clj

@@ -1,12 +1,12 @@
-(ns metabase-enterprise.transforms-runner.runner
+(ns metabase-enterprise.transforms-python.python-runner
   (:require
    [clj-http.client :as http]
    [clojure.core.async :as a]
    [clojure.java.io :as io]
    [clojure.string :as str]
    [medley.core :as m]
-   [metabase-enterprise.transforms-runner.s3 :as s3]
-   [metabase-enterprise.transforms-runner.settings :as runner.settings]
+   [metabase-enterprise.transforms-python.s3 :as s3]
+   [metabase-enterprise.transforms-python.settings :as transforms-python.settings]
    [metabase.analytics.prometheus :as prometheus]
    [metabase.config.core :as config]
    [metabase.lib-be.core :as lib-be]
@@ -33,7 +33,7 @@
   "Returns HTTP headers with Authorization bearer token if configured.
   Throws configuration error in production if token is not set."
   []
-  (let [api-token (runner.settings/python-runner-api-token)]
+  (let [api-token (transforms-python.settings/python-runner-api-token)]
     (if api-token
       {"Authorization" (str "Bearer " api-token)}
       (if config/is-prod?
@@ -175,7 +175,7 @@
 (defn get-logs
   "Return the logs of the current running python process"
   [run-id]
-  (let [server-url (runner.settings/python-runner-url)]
+  (let [server-url (transforms-python.settings/python-runner-url)]
     (python-runner-request server-url :get "/logs" {:query-params {:request_id run-id}})))
 
 (mu/defn record-python-api-call!
@@ -200,25 +200,23 @@
 (defn execute-python-code-http-call!
   "Calls the /execute endpoint of the python runner. Blocks until the run either succeeds or fails and returns
   the response from the server."
-  [{:keys [server-url code request-id run-id table-name->id shared-storage timeout-secs runtime]}]
-  (let [{:keys [objects]}                       shared-storage
+  [{:keys [server-url code request-id run-id table-name->id shared-storage timeout-secs]}]
+  (let [{:keys [objects]} shared-storage
         {:keys [output output-manifest events]} objects
-        runtime                                 (or runtime "python")
-        url-for-path                            (fn [path] (:url (get objects path)))
-        table-name->url                         (update-vals table-name->id #(url-for-path [:table % :data]))
-        table-name->manifest-url                (update-vals table-name->id #(url-for-path [:table % :manifest]))
-        payload                                 {:code                code
-                                                 :library             (t2/select-fn->fn :path :source :model/TransformLibrary :language runtime)
-                                                 :timeout             (or timeout-secs (runner.settings/python-runner-timeout-seconds))
-                                                 :request_id          (or request-id run-id)
-                                                 :output_url          (:url output)
-                                                 :output_manifest_url (:url output-manifest)
-                                                 :events_url          (:url events)
-                                                 :table_mapping       table-name->url
-                                                 :manifest_mapping    table-name->manifest-url
-                                                 :runtime             runtime}
-        response                                (with-python-api-timing [run-id]
-                                                  (python-runner-request server-url :post "/execute" {:body (json/encode payload)}))]
+        url-for-path             (fn [path] (:url (get objects path)))
+        table-name->url          (update-vals table-name->id #(url-for-path [:table % :data]))
+        table-name->manifest-url (update-vals table-name->id #(url-for-path [:table % :manifest]))
+        payload                  {:code                code
+                                  :library             (t2/select-fn->fn :path :source :model/PythonLibrary)
+                                  :timeout             (or timeout-secs (transforms-python.settings/python-runner-timeout-seconds))
+                                  :request_id          (or request-id run-id)
+                                  :output_url          (:url output)
+                                  :output_manifest_url (:url output-manifest)
+                                  :events_url          (:url events)
+                                  :table_mapping       table-name->url
+                                  :manifest_mapping    table-name->manifest-url}
+        response                 (with-python-api-timing [run-id]
+                                   (python-runner-request server-url :post "/execute" {:body (json/encode payload)}))]
     ;; when a 500 is returned we observe a string in the body (despite the python returning json)
     ;; always try to parse the returned string as json before yielding (could tighten this up at some point)
     (update response :body (fn [string-if-error]
@@ -344,7 +342,6 @@
      :source-tables - Map of table-name -> table-id (already resolved)
      :row-limit     - Max rows to return (also limits input rows)
      :timeout-secs  - Optional timeout override
-     :runtime       - Optional runtime override (default: \"python\")
 
    Returns:
      {:status  :succeeded/:failed
@@ -353,9 +350,9 @@
       :logs    [{:message ...} ...]   ; events from Python execution
       :message \"error message\"}     ; on failure
 "
-  [{:keys [code source-tables per-input-limit row-limit timeout-secs runtime]}]
+  [{:keys [code source-tables per-input-limit row-limit timeout-secs]}]
   (with-open [shared-storage-ref (s3/open-shared-storage! source-tables)]
-    (let [server-url (runner.settings/python-runner-url)
+    (let [server-url (transforms-python.settings/python-runner-url)
           _          (copy-tables-to-s3! {:shared-storage @shared-storage-ref
                                           :source         {:source-tables source-tables}
                                           :limit          (or per-input-limit row-limit)})
@@ -366,8 +363,7 @@
             :request-id     (u/generate-nano-id)
             :table-name->id source-tables
             :timeout-secs   timeout-secs
-            :shared-storage @shared-storage-ref
-            :runtime runtime})
+            :shared-storage @shared-storage-ref})
           events (read-events @shared-storage-ref)]
       (cond
         (:timeout body)

enterprise/backend/src/metabase_enterprise/transforms_python/s3.clj

@@ -1,6 +1,6 @@
-(ns metabase-enterprise.transforms-runner.s3
+(ns metabase-enterprise.transforms-python.s3
   (:require
-   [metabase-enterprise.transforms-runner.settings :as runner.settings]
+   [metabase-enterprise.transforms-python.settings :as transforms-python.settings]
    [metabase.util.log :as log])
   (:import
    (clojure.lang IDeref)
@@ -29,12 +29,12 @@
 (defn- working-dir-for-run
   "The path within the target bucket within which to store all the files for this run."
   []
-  (let [shared-prefix (some-> (runner.settings/python-storage-s-3-prefix) (str "/"))]
+  (let [shared-prefix (some-> (transforms-python.settings/python-storage-s-3-prefix) (str "/"))]
     (str shared-prefix "run-" (System/nanoTime) "-" (rand-int 10000))))
 
 (defmacro ^:private maybe-with-endpoint* [builder endpoint]
   `(let [builder# ~builder]
-     (when-let [region# (runner.settings/python-storage-s-3-region)]
+     (when-let [region# (transforms-python.settings/python-storage-s-3-region)]
        (.region builder# (Region/of region#)))
      (when ~endpoint (.endpointOverride builder# (URI/create ~endpoint)))
      builder#))
@@ -46,7 +46,7 @@
   (maybe-with-endpoint* builder endpoint))
 
 (defn- s3-configuration ^S3Configuration []
-  (let [path-style-access (runner.settings/python-storage-s-3-path-style-access)]
+  (let [path-style-access (transforms-python.settings/python-storage-s-3-path-style-access)]
     (-> (S3Configuration/builder)
         (.pathStyleAccessEnabled path-style-access)
         ;; Disable chunked encoding when using path-style access, as most S3-compatible
@@ -65,8 +65,8 @@
 
 (defn- maybe-with-credentials*
   [credentials-provider]
-  (let [access-key (runner.settings/python-storage-s-3-access-key)
-        secret-key (runner.settings/python-storage-s-3-secret-key)]
+  (let [access-key (transforms-python.settings/python-storage-s-3-access-key)
+        secret-key (transforms-python.settings/python-storage-s-3-secret-key)]
     (if (or access-key secret-key)
       (if-not (and access-key secret-key)
         (do (log/warnf "Ignoring %s because %s is not defined"
@@ -90,15 +90,15 @@
   ^S3Client []
   (.build
    (doto (S3Client/builder)
-     (maybe-with-endpoint-s3-client (runner.settings/python-storage-s-3-endpoint))
+     (maybe-with-endpoint-s3-client (transforms-python.settings/python-storage-s-3-endpoint))
      maybe-with-credentials-s3-client
      (.serviceConfiguration (s3-configuration)))))
 
 (defn- create-s3-presigner-for-container
   "Create S3 presigner for container operations (presigned URLs). Uses distinct container-endpoint if relevant."
   ^S3Presigner []
-  (let [container-endpoint (runner.settings/python-storage-s-3-container-endpoint)
-        endpoint           (or container-endpoint (runner.settings/python-storage-s-3-endpoint))]
+  (let [container-endpoint (transforms-python.settings/python-storage-s-3-container-endpoint)
+        endpoint           (or container-endpoint (transforms-python.settings/python-storage-s-3-endpoint))]
     (.build
      (doto (S3Presigner/builder)
        (maybe-with-endpoint-s3-presigner endpoint)
@@ -126,7 +126,7 @@
 (defn- s3-shared-storage [table-name->id]
   (let [work-dir-name       (working-dir-for-run)
         container-presigner (create-s3-presigner-for-container)
-        bucket-name         (runner.settings/python-storage-s-3-bucket)
+        bucket-name         (transforms-python.settings/python-storage-s-3-bucket)
         ref                 (fn [method relative-path]
                               (let [path (str work-dir-name "/" relative-path)]
                                 {:path   path

enterprise/backend/src/metabase_enterprise/transforms_python/settings.clj

@@ -1,5 +1,4 @@
-;; TODO(Timothy, 02-25-2026): these settings point to a common runner, not python
-(ns metabase-enterprise.transforms-runner.settings
+(ns metabase-enterprise.transforms-python.settings
   (:require
    [metabase.config.core :as config]
    [metabase.settings.core :as setting]

enterprise/backend/src/metabase_enterprise/transforms_runner/execute.clj

@@ -1,428 +0,0 @@
-(ns metabase-enterprise.transforms-runner.execute
-  "Shared execution logic for runner-based transforms (Python, JavaScript, etc.).
-
-  All runner-based transforms follow the same pattern:
-  1. Upload source tables to S3
-  2. Call the runner service with a `language` parameter
-  3. Read output from S3
-  4. Transfer results to the target database
-
-  Each language provides a small config map; this namespace does the rest."
-  (:require
-   [clojure.core.async :as a]
-   [clojure.java.io :as io]
-   [clojure.string :as str]
-   [metabase-enterprise.transforms-runner.runner :as runner]
-   [metabase-enterprise.transforms-runner.s3 :as s3]
-   [metabase-enterprise.transforms-runner.settings :as runner.settings]
-   [metabase.app-db.core :as app-db]
-   [metabase.driver :as driver]
-   [metabase.driver.connection :as driver.conn]
-   [metabase.driver.util :as driver.u]
-   [metabase.transforms.core :as transforms]
-   [metabase.transforms.instrumentation :as transforms.instrumentation]
-   [metabase.transforms.interface :as transforms.i]
-   [metabase.transforms.util :as transforms.util]
-   [metabase.util :as u]
-   [metabase.util.format :as u.format]
-   [metabase.util.i18n :as i18n]
-   [metabase.util.jvm :as u.jvm]
-   [metabase.util.log :as log]
-   [toucan2.core :as t2])
-  (:import
-   (java.io Closeable)
-   (java.net SocketException)
-   (java.nio.file Files)
-   (java.nio.file.attribute FileAttribute)
-   (java.time Duration)))
-
-(set! *warn-on-reflection* true)
-
-;;; ------------------------------------------------ Message Log -----------------------------------------------------
-
-(defn- empty-message-log
-  "Returns a new message log for transform execution.
-  Buffers log outputs during the run to produce a string suitable for `transform_run.message`."
-  []
-  (atom {:pre []
-         :runner nil
-         :post []}))
-
-(defn- log!
-  "Appends a user-facing string to the message log."
-  [message-log s]
-  (swap! message-log (fn [m] (update m (if (:runner m) :post :pre) conj s)))
-  nil)
-
-(defn- replace-runner-logs!
-  "Called when events output is received from a /logs call to copy those events into the message-log."
-  [message-log events]
-  (swap! message-log assoc :runner events)
-  nil)
-
-(defn- message-log->transform-run-message
-  "Presents the content of the log as a string suitable for storage in transform_run.message."
-  [message-log]
-  (let [{:keys [pre runner post]} @message-log]
-    (str/join "\n" (concat pre
-                           (map :message runner)
-                           post))))
-
-(defn- save-log-to-transform-run-message!
-  [run-id message-log]
-  (t2/update! :model/TransformRun
-              :id run-id
-              {:message (message-log->transform-run-message message-log)}))
-
-;;; ------------------------------------------------ Log Polling Loop ------------------------------------------------
-
-(def ^:private ^Duration message-loop-sleep-duration
-  (Duration/ofMillis 1000))
-
-(defn- message-update-loop!
-  "Block while relevant log data is replicated from the runner into the message log."
-  [run-id message-log]
-  (try
-    (loop []
-      (if (.isInterrupted (Thread/currentThread))
-        (log/debug "Message update loop interrupted")
-        (do (let [sleep-ms (.toMillis message-loop-sleep-duration)]
-              (when (pos? sleep-ms) (Thread/sleep sleep-ms)))
-            (let [{:keys [status body]} (runner/get-logs run-id)]
-              (cond
-                (<= 200 status 299)
-                (let [{:keys [execution_id events]} body]
-                  (if-not (= run-id execution_id)
-                    (do (log/debugf "Run id did not match expected: %s actual: %s" run-id execution_id)
-                        (recur))
-                    (do
-                      (replace-runner-logs! message-log events)
-                      (save-log-to-transform-run-message! run-id message-log)
-                      (recur))))
-                (= 404 status)
-                (do
-                  (log/debugf "No logs yet (or run finished), run-id: %s" run-id)
-                  (recur))
-                :else
-                (do
-                  (log/warnf "Unexpected status polling for logs %s %s, run-id: %s" status body run-id)
-                  (log/debug "Exiting due to poll error")))))))
-    (catch SocketException se (when-not (= "Closed by interrupt" (ex-message se)) (throw se)))
-    (catch InterruptedException _)
-    (catch Throwable e
-      (log/errorf e "An exception was caught during msg update loop, run-id: %s" run-id))))
-
-(defn- open-message-update-future! ^Closeable [run-id message-log]
-  (let [cleanup (fn [fut]
-                  (future-cancel fut)
-                  (if (= ::timeout (try (deref fut 10000 ::timeout) (catch Throwable _)))
-                    (log/fatalf "Log polling task did not respond to interrupt, run-id: %s" run-id)
-                    (log/debugf "Log polling task done, run-id: %s" run-id)))
-        fut (u.jvm/in-virtual-thread*
-             (message-update-loop! run-id message-log))]
-    (reify Closeable
-      (close [_] (cleanup fut)))))
-
-;;; ------------------------------------------------ Table Transfer --------------------------------------------------
-
-(defn- maybe-retry-with-backoff
-  [driver thunk]
-  (if (= driver :bigquery-cloud-sdk)
-    (loop [attempt 0]
-      (let [ret (try
-                  (thunk)
-                  (catch Exception e
-                    (if (and (< attempt 5)
-                             (or (str/includes? (str e) "not found")
-                                 (str/includes? (str e) "Not found")))
-                      (let [delay-ms (* 150 (Math/pow 2 attempt))]
-                        (log/debugf "BigQuery operation failed (attempt %d), retrying in %dms" (inc attempt) delay-ms)
-                        (Thread/sleep (long delay-ms))
-                        ::recur)
-                      (throw e))))]
-        (if (= ::recur ret)
-          (recur (inc attempt))
-          ret)))
-    (thunk)))
-
-(defn- table-schema [table-name metadata]
-  {:name (if (keyword? table-name) table-name (keyword table-name))
-   :columns (mapv (fn [{:keys [name base_type]}]
-                    {:name name
-                     :type (runner/restricted-insert-type base_type)
-                     :nullable? true})
-                  (:fields metadata))})
-
-(defn- insert-data!
-  [driver db-id table-schema data-source]
-  (let [data-source (assoc data-source :table-schema table-schema)]
-    (->> #(driver/insert-from-source! driver db-id table-schema data-source)
-         (maybe-retry-with-backoff driver))))
-
-(defn- create-table-and-insert-data!
-  [driver db-id table-schema data-source]
-  (transforms.util/create-table-from-schema! driver db-id table-schema)
-  (insert-data! driver db-id table-schema data-source))
-
-(defn- transfer-with-rename-tables-strategy!
-  [driver db-id table-name metadata data-source]
-  (let [source-table-name (driver.u/temp-table-name driver table-name)
-        temp-table-name (driver.u/temp-table-name driver table-name)]
-    (log/info "Using rename-tables strategy with atomicity guarantees")
-    (try
-      (create-table-and-insert-data! driver db-id (table-schema source-table-name metadata) data-source)
-      (transforms.util/rename-tables! driver db-id {table-name temp-table-name
-                                                    source-table-name table-name})
-      (transforms.util/drop-table! driver db-id temp-table-name)
-      (catch Exception e
-        (log/error e "Failed to transfer data using rename-tables strategy")
-        (try
-          (transforms.util/drop-table! driver db-id source-table-name)
-          (catch Exception _))
-        (throw e)))))
-
-(defn- transfer-with-create-drop-rename-strategy!
-  [driver db-id table-name metadata data-source]
-  (let [source-table-name (driver.u/temp-table-name driver table-name)]
-    (log/info "Using create-drop-rename strategy to minimize downtime")
-    (try
-      (create-table-and-insert-data! driver db-id (table-schema source-table-name metadata) data-source)
-      (transforms.util/drop-table! driver db-id table-name)
-      (driver/rename-table! driver db-id source-table-name table-name)
-      (catch Exception e
-        (log/error e "Failed to transfer data using create-drop-rename strategy")
-        (try
-          (transforms.util/drop-table! driver db-id source-table-name)
-          (catch Exception _))
-        (throw e)))))
-
-(defn- transfer-with-drop-create-fallback-strategy!
-  [driver db-id table-name metadata data-source]
-  (log/info "Using drop-create fallback strategy")
-  (try
-    (transforms.util/drop-table! driver db-id table-name)
-    (create-table-and-insert-data! driver db-id (table-schema table-name metadata) data-source)
-    (catch Exception e
-      (log/error e "Failed to transfer data using drop-create fallback strategy")
-      (throw e))))
-
-(defmulti ^:private transfer-file-to-db
-  {:arglists '([driver db transform metadata temp-file])}
-  (fn [_ _ transform _ _] (-> transform :target :type keyword)))
-
-(defmethod transfer-file-to-db :table-incremental
-  [driver {db-id :id}
-   {:keys [target] :as transform}
-   metadata temp-file]
-  (let [table-name (transforms.util/qualified-table-name driver target)
-        table-exists? (transforms.util/target-table-exists? transform)
-        data-source {:type :jsonl-file
-                     :file temp-file}]
-    (if (not table-exists?)
-      (do
-        (log/info "New table")
-        (create-table-and-insert-data! driver db-id (table-schema table-name metadata) data-source))
-      (insert-data! driver db-id (table-schema table-name metadata) data-source))))
-
-(defmethod transfer-file-to-db :table
-  [driver {db-id :id :as db}
-   {:keys [target] :as transform}
-   metadata temp-file]
-  (let [table-name (transforms.util/qualified-table-name driver target)
-        table-exists? (transforms.util/target-table-exists? transform)
-        data-source {:type :jsonl-file
-                     :file temp-file}]
-    (cond
-      (not table-exists?)
-      (do
-        (log/info "New table")
-        (create-table-and-insert-data! driver db-id (table-schema table-name metadata) data-source))
-
-      (driver.u/supports? driver :atomic-renames db)
-      (transfer-with-rename-tables-strategy! driver db-id table-name metadata data-source)
-
-      (driver.u/supports? driver :rename db)
-      (transfer-with-create-drop-rename-strategy! driver db-id table-name metadata data-source)
-
-      :else
-      (transfer-with-drop-create-fallback-strategy! driver db-id table-name metadata data-source))))
-
-;;; ------------------------------------------------ Runner Execution ------------------------------------------------
-
-(defn- start-cancellation-process!
-  [server-url run-id cancel-chan]
-  (a/go (when (a/<! cancel-chan)
-          (runner/cancel-python-code-http-call! server-url run-id))))
-
-(defn- run-runner-transform!
-  "Core execution: upload tables to S3, call runner, read output, transfer to DB.
-  `runtime` is the language string passed to the runner (e.g. \"python\", \"javascript\")."
-  [{:keys [source] :as transform} db run-id cancel-chan message-log runtime]
-  (let [resolved-source-tables (transforms.util/resolve-source-tables (:source-tables source))]
-    (with-open [^Closeable log-future-ref
-                (if (app-db/in-transaction?)
-                  (reify Closeable (close [_]))
-                  (open-message-update-future! run-id message-log))
-                shared-storage-ref (s3/open-shared-storage! resolved-source-tables)]
-      (let [driver (:engine db)
-            server-url (runner.settings/python-runner-url)
-            _ (runner/copy-tables-to-s3! {:run-id run-id
-                                          :shared-storage @shared-storage-ref
-                                          :source (assoc source :source-tables resolved-source-tables)
-                                          :cancel-chan cancel-chan
-                                          :limit (:limit source)
-                                          :transform-id (:id transform)})
-            _ (start-cancellation-process! server-url run-id cancel-chan)
-            {:keys [status body] :as response}
-            (runner/execute-python-code-http-call!
-             {:server-url server-url
-              :code (:body source)
-              :run-id run-id
-              :table-name->id resolved-source-tables
-              :shared-storage @shared-storage-ref
-              :runtime runtime})
-
-            output-manifest (runner/read-output-manifest @shared-storage-ref)
-            events (runner/read-events @shared-storage-ref)]
-        (.close ^Closeable log-future-ref)
-        (replace-runner-logs! message-log events)
-        (if (not= 200 status)
-          (do
-            (when (:timeout body)
-              (transforms/timeout-run! run-id))
-            (throw (ex-info (str runtime " runner call failed")
-                            {:transform-message (i18n/tru "{0} execution failure (exit code {1})"
-                                                          (str/capitalize runtime)
-                                                          (:exit_code body "?"))
-                             :status-code 400
-                             :api-status-code status
-                             :body body
-                             :events events})))
-          (try
-            (let [temp-path (Files/createTempFile "transform-output-" ".jsonl" (u/varargs FileAttribute))
-                  temp-file (.toFile temp-path)]
-              (when-not (seq (:fields output-manifest))
-                (throw (ex-info "No fields in metadata"
-                                {:metadata output-manifest
-                                 :raw-body body
-                                 :events events})))
-              (try
-                (with-open [in (runner/open-output @shared-storage-ref)]
-                  (io/copy in temp-file))
-                (let [file-size (.length temp-file)]
-                  (transforms.instrumentation/with-stage-timing [run-id [:import :file-to-dwh]]
-                    (transfer-file-to-db driver db transform output-manifest temp-file))
-                  (transforms.instrumentation/record-data-transfer! run-id :file-to-dwh file-size nil))
-                (finally
-                  (.delete temp-file))))
-            response
-            (catch Exception e
-              (log/error e "Failed to create resulting table")
-              (throw (ex-info "Failed to create the resulting table"
-                              {:transform-message (or (:transform-message (ex-data e))
-                                                      (i18n/tru "Failed to create the resulting table"))}
-                              e)))))))))
-
-;;; ------------------------------------------------ Public API ------------------------------------------------------
-
-(defn- exceptional-run-message [message-log ex]
-  (str/join "\n" (remove str/blank? [(message-log->transform-run-message message-log)
-                                     (or (:transform-message (ex-data ex))
-                                         (if (instance? InterruptedException ex)
-                                           (i18n/tru "Transform interrupted")
-                                           (i18n/tru "Something went wrong")))])))
-
-(defn execute-runner-transform!
-  "Execute a runner-based transform (Python, JavaScript, Clojure, R, Julia, etc.).
-
-  `lang-config` is a map with:
-    :runtime           - string passed to runner as language param (e.g. \"python\", \"javascript\")
-    :label             - human-readable label for log messages (e.g. \"Python\", \"JavaScript\")
-    :timing-key        - keyword for instrumentation (e.g. :python-execution, :javascript-execution)
-
-  Blocks until the transform returns."
-  [transform {:keys [run-method start-promise user-id]} lang-config]
-  (let [{:keys [runtime label timing-key]} lang-config]
-    (assert (transforms.util/runner-transform? transform)
-            (str "Transform must be a runner-based transform, got: "
-                 (transforms.util/transform-type transform)))
-    (try
-      (let [message-log (empty-message-log)
-            {:keys [target owner_user_id creator_id] transform-id :id} transform
-            {driver :engine :as db} (t2/select-one :model/Database (transforms.i/target-db-id transform))
-            run-user-id (if (and (= run-method :manual) user-id)
-                          user-id
-                          (or owner_user_id creator_id))
-            {run-id :id} (transforms.util/try-start-unless-already-running transform-id run-method run-user-id)]
-        (some-> start-promise (deliver [:started run-id]))
-        (log! message-log (i18n/tru "Executing {0} transform" label))
-        (driver.conn/with-write-connection
-          (log/info "Executing" label "transform" transform-id "with target" (pr-str target)
-                    (when (driver.conn/write-connection-requested?)
-                      " using write connection"))
-          (let [start-ms (u/start-timer)
-                conn-spec (driver/connection-spec driver db)
-                transform-details {:db-id (:id db)
-                                   :transform-id transform-id
-                                   :transform-type (keyword (:type target))
-                                   :conn-spec conn-spec
-                                   :output-schema (:schema target)
-                                   :output-table (transforms.util/qualified-table-name driver target)}
-                run-fn (fn [cancel-chan]
-                         (run-runner-transform! transform db run-id cancel-chan message-log runtime)
-                         (log! message-log (i18n/tru "{0} execution finished successfully in {1}"
-                                                     label (u.format/format-milliseconds (u/since-ms start-ms))))
-                         (save-log-to-transform-run-message! run-id message-log))
-                ex-message-fn #(exceptional-run-message message-log %)
-                result (transforms.instrumentation/with-stage-timing [run-id [:computation timing-key]]
-                         (transforms.util/run-cancelable-transform! run-id driver transform-details run-fn :ex-message-fn ex-message-fn))]
-            (transforms.util/handle-transform-complete!
-             :run-id run-id
-             :transform transform
-             :db db)
-            {:run_id run-id
-             :result result})))
-      (catch Throwable t
-        (log/error t (str "Error executing " label " transform"))
-        (throw t)))))
-
-;;; ------------------------------------------------ Impl Helpers ----------------------------------------------------
-
-(defn source-table-value->dependency
-  "Convert a source table value (int or ref map) to a dependency map.
-  Shared by all runner-based transform implementations."
-  [v]
-  (cond
-    (int? v)
-    {:table v}
-
-    (:table_id v)
-    {:table (:table_id v)}
-
-    :else
-    {:table-ref (select-keys v [:database_id :schema :table])}))
-
-;;; ---------------------------------------- Shared ::runner defmethods ----------------------------------------
-;; All runner-based languages share these implementations. Per-language dispatch
-;; is handled via the hierarchy in transforms.i — each language only needs to
-;; define (defmethod transforms.i/lang-config :lang-kw ...).
-
-(defmethod transforms.i/target-db-id ::transforms.i/runner
-  [transform]
-  (-> transform :target :database))
-
-(defmethod transforms.i/source-db-id ::transforms.i/runner
-  [transform]
-  (-> transform :source :source-database))
-
-(defmethod transforms.i/table-dependencies ::transforms.i/runner
-  [transform]
-  (into #{}
-        (map source-table-value->dependency)
-        (vals (get-in transform [:source :source-tables]))))
-
-#_{:clj-kondo/ignore [:discouraged-var]}
-(defmethod transforms.i/execute! ::transforms.i/runner
-  [transform options]
-  (let [lang-kw (-> transform :source :type keyword)]
-    (execute-runner-transform! transform options (transforms.i/lang-config lang-kw))))

enterprise/backend/src/metabase_enterprise/transforms_runner/init.clj

@@ -1,8 +0,0 @@
-(ns metabase-enterprise.transforms-runner.init
-  (:require
-   [metabase-enterprise.transforms-runner.api]
-   [metabase-enterprise.transforms-runner.execute]
-   [metabase-enterprise.transforms-runner.models.transform-library]
-   [metabase-enterprise.transforms-runner.runner]
-   [metabase-enterprise.transforms-runner.s3]
-   [metabase-enterprise.transforms-runner.settings]))

enterprise/backend/src/metabase_enterprise/transforms_runner/models/transform_library.clj

@@ -1,151 +0,0 @@
-(ns metabase-enterprise.transforms-runner.models.transform-library
-  (:require
-   [metabase.api.common :as api]
-   [metabase.app-db.core :as app-db]
-   [metabase.events.core :as events]
-   [metabase.models.interface :as mi]
-   [metabase.models.serialization :as serdes]
-   [metabase.permissions.core :as perms]
-   [metabase.transforms.interface :as transforms.i]
-   [metabase.util.i18n :refer [tru]]
-   [methodical.core :as methodical]
-   [toucan2.core :as t2]))
-
-(set! *warn-on-reflection* true)
-
-(methodical/defmethod t2/table-name :model/TransformLibrary [_model] :transform_library)
-
-(doseq [trait [:metabase/model :hook/timestamped? :hook/entity-id]]
-  (derive :model/TransformLibrary trait))
-
-(defn- language-extension
-  "Return the file extension (e.g. \".py\") for a language string, looked up from lang-config."
-  [language]
-  (:extension (transforms.i/lang-config (keyword language))))
-
-(defn- normalize-path
-  [language path]
-  (let [ext (language-extension language)]
-    (if (and ext (not (.endsWith ^String path ext)))
-      (str path ext)
-      path)))
-
-(t2/define-before-insert :model/TransformLibrary
-  [library]
-  (update library :path (partial normalize-path (:language library))))
-
-(t2/define-before-update :model/TransformLibrary
-  [library]
-  (if (:path library)
-    (update library :path (partial normalize-path (or (:language library)
-                                                      (:language (t2/original library)))))
-    library))
-
-(defmethod mi/can-read? :model/TransformLibrary
-  ([_instance]
-   (perms/has-any-transforms-permission? api/*current-user-id*))
-  ([_model _pk]
-   (perms/has-any-transforms-permission? api/*current-user-id*)))
-
-(defmethod mi/can-write? :model/TransformLibrary
-  ([_instance]
-   (perms/has-any-transforms-permission? api/*current-user-id*))
-  ([_model _pk]
-   (perms/has-any-transforms-permission? api/*current-user-id*)))
-
-(def ^:private legacy-entity-ids
-  "Stable entity IDs for built-in libraries created by Liquibase migrations.
-  New languages get deterministic IDs via [[builtin-entity-id]]."
-  {"python" "cWWH9qJPvHNB3rP2vLZrK"
-   "javascript" "aHWo_0yPLwKqpQPlFNzCg"})
-
-(defn builtin-entity-id
-  "Return the stable entity ID for a language's built-in common library.
-  Legacy languages (python, javascript) use migration-baked IDs; new languages
-  get a deterministic 21-char NanoID-style hash derived from the language name."
-  [language]
-  (or (get legacy-entity-ids language)
-      (let [bytes (.digest (java.security.MessageDigest/getInstance "SHA-256")
-                           (.getBytes (str "transform-library:" language) "UTF-8"))
-            encoded (-> (java.util.Base64/getUrlEncoder)
-                        (.withoutPadding)
-                        (.encodeToString bytes))]
-        (subs encoded 0 21))))
-
-(defn all-builtin-entity-ids
-  "Return the set of entity IDs for all registered runner languages' built-in libraries."
-  []
-  (into #{}
-        (map (comp builtin-entity-id name))
-        (transforms.i/runner-languages)))
-
-(defn- validate-path!
-  [language path]
-  (let [ext (language-extension language)
-        normalized (normalize-path language path)
-        allowed-path (str "common" ext)]
-    (when-not (= normalized allowed-path)
-      (throw (ex-info (tru "Invalid library path. Only ''common'' is currently supported.")
-                      {:status-code 400
-                       :path normalized
-                       :language language
-                       :allowed-paths [allowed-path]})))))
-
-(defn get-library-by-path
-  "Look up a TransformLibrary record by `language` and `path`, validating the path first."
-  [language path]
-  (let [normalized (normalize-path language path)]
-    (validate-path! language normalized)
-    (t2/select-one :model/TransformLibrary :language language :path normalized)))
-
-(defn update-library-source!
-  "Create or update the TransformLibrary record for `language` and `path` with the given `source`."
-  [language path source]
-  (let [normalized (normalize-path language path)]
-    (validate-path! language normalized)
-    (let [id (app-db/update-or-insert! :model/TransformLibrary
-                                       {:language language :path normalized}
-                                       (constantly {:language language :path normalized :source source}))]
-      (t2/select-one :model/TransformLibrary id))))
-
-(defn ensure-builtin-library!
-  "Ensure the built-in common library row exists for `language`.
-  Inserts idempotently — no-op if the row already exists.
-  New runner languages can call this instead of adding a Liquibase migration.
-
-  NOTE: Must be called after the app DB is initialized and migrations have run.
-  Do NOT call at namespace load time (e.g. from init.clj) — the table may not exist yet."
-  [language]
-  (let [ext (language-extension language)
-        path (str "common" ext)
-        eid (builtin-entity-id language)]
-    (when-not (t2/exists? :model/TransformLibrary :language language :path path)
-      (t2/insert! :model/TransformLibrary
-                  {:path path :source "" :language language :entity_id eid}))))
-
-(defmethod serdes/make-spec "TransformLibrary"
-  [_model-name _opts]
-  {:copy [:language :path :source :entity_id]
-   :transform {:created_at (serdes/date)}})
-
-(defmethod serdes/hash-fields :model/TransformLibrary
-  [_model]
-  [:language :path])
-
-(defmethod serdes/storage-path "TransformLibrary" [entity _ctx]
-  (let [{:keys [id label]} (-> entity serdes/path last)]
-    ["transform-libraries" (serdes/storage-leaf-file-name id label)]))
-
-(derive ::event :metabase/event)
-(doseq [e [:event/transform-library-create :event/transform-library-update :event/transform-library-delete]]
-  (derive e ::event))
-
-(t2/define-after-insert :model/TransformLibrary
-  [library]
-  (events/publish-event! :event/transform-library-create {:object library})
-  library)
-
-(t2/define-after-update :model/TransformLibrary
-  [library]
-  (events/publish-event! :event/transform-library-update {:object library})
-  library)

enterprise/backend/src/metabase_enterprise/workspaces/execute.clj

@@ -9,7 +9,7 @@
   decoupled from AppDb."
   (:require
    [clojure.string :as str]
-   [metabase-enterprise.transforms-runner.runner :as runner]
+   [metabase-enterprise.transforms-python.python-runner :as python-runner]
    [metabase.api.common :as api]
    [metabase.query-processor :as qp]
    [metabase.sql-tools.core :as sql-tools]
@@ -115,7 +115,7 @@
         remapped-source        (remap-python-source table-mapping source)
         resolved-source-tables (transforms.util/resolve-source-tables (:source-tables remapped-source))
         {:as   result
-         :keys [cols rows]}    (runner/execute-and-read-output!
+         :keys [cols rows]}    (python-runner/execute-and-read-output!
                                 {:code          (:body remapped-source)
                                  :source-tables resolved-source-tables
                                  :row-limit     preview-row-limit})

enterprise/backend/test/metabase_enterprise/advanced_permissions/query_processor/middleware/permissions_test.clj

@@ -282,15 +282,16 @@
       (with-download-perms-for-db! (mt/id) :none
         (streaming-test/do-test!
          "A user with no download perms for a DB receives an error response"
-         {:query      {:database (mt/id)
-                       :type     :query
-                       :query    {:source-table (mt/id :venues)
-                                  :limit        10}}
-          :endpoints  [:card :dataset]
-          :assertions {:csv (fn [results]
-                              (is (partial=
-                                   {:error "You do not have permissions to download the results of this query."}
-                                   results)))}})
+         {:query           {:database (mt/id)
+                            :type     :query
+                            :query    {:source-table (mt/id :venues)
+                                       :limit        10}}
+          :expected-status 403
+          :endpoints       [:card :dataset]
+          :assertions      {:csv (fn [results]
+                                   (is (partial=
+                                        {:error "You do not have permissions to download the results of this query."}
+                                        results)))}})
 
         (streaming-test/do-test!
          "An admin can always run download queries, even if the All Users group has no download permissions "
@@ -305,15 +306,16 @@
       (with-download-perms! (mt/id) {:schemas {"PUBLIC" :none}}
         (streaming-test/do-test!
          "A user with no download perms for a schema receives an error response for download queries on that schema"
-         {:query      {:database (mt/id)
-                       :type     :query
-                       :query    {:source-table (mt/id :venues)
-                                  :limit        10}}
-          :endpoints  [:card :dataset]
-          :assertions {:csv (fn [results]
-                              (is (partial=
-                                   {:error "You do not have permissions to download the results of this query."}
-                                   results)))}}))
+         {:query           {:database (mt/id)
+                            :type     :query
+                            :query    {:source-table (mt/id :venues)
+                                       :limit        10}}
+          :expected-status 403
+          :endpoints       [:card :dataset]
+          :assertions      {:csv (fn [results]
+                                   (is (partial=
+                                        {:error "You do not have permissions to download the results of this query."}
+                                        results)))}}))
 
       (with-download-perms! (mt/id) {:schemas {"PUBLIC" {(mt/id :venues)     :none
                                                          (mt/id :checkins)   :full
@@ -321,15 +323,16 @@
                                                          (mt/id :categories) :full}}}
         (streaming-test/do-test!
          "A user with no download perms for a table receives an error response for download queries on that table"
-         {:query      {:database (mt/id)
-                       :type     :query
-                       :query    {:source-table (mt/id :venues)
-                                  :limit        10}}
-          :endpoints  [:card :dataset]
-          :assertions {:csv (fn [results]
-                              (is (partial=
-                                   {:error "You do not have permissions to download the results of this query."}
-                                   results)))}})
+         {:query           {:database (mt/id)
+                            :type     :query
+                            :query    {:source-table (mt/id :venues)
+                                       :limit        10}}
+          :expected-status 403
+          :endpoints       [:card :dataset]
+          :assertions      {:csv (fn [results]
+                                   (is (partial=
+                                        {:error "You do not have permissions to download the results of this query."}
+                                        results)))}})
 
         (streaming-test/do-test!
          "A user with no download perms for a table still has full download perms for MBQL queries on other tables"
@@ -342,12 +345,13 @@
 
         (streaming-test/do-test!
          "A user with no download perms for a table has no download perms for native queries on all tables"
-         {:query      (mt/native-query {:query "SELECT * FROM checkins LIMIT 10;"})
-          :endpoints  [:card :dataset]
-          :assertions {:csv (fn [results]
-                              (is (partial=
-                                   {:error "You do not have permissions to download the results of this query."}
-                                   results)))}})))))
+         {:query           (mt/native-query {:query "SELECT * FROM checkins LIMIT 10;"})
+          :expected-status 403
+          :endpoints       [:card :dataset]
+          :assertions      {:csv (fn [results]
+                                   (is (partial=
+                                        {:error "You do not have permissions to download the results of this query."}
+                                        results)))}})))))
 
 (deftest joins-test
   (mt/with-full-data-perms-for-all-users!
@@ -362,11 +366,12 @@
                    {:joins [{:source-table $$categories
                              :condition    [:= $category_id 1]}]
                     :limit 10})
-          :endpoints  [:card :dataset]
-          :assertions {:csv (fn [results]
-                              (is (partial=
-                                   {:error "You do not have permissions to download the results of this query."}
-                                   results)))}})
+          :expected-status 403
+          :endpoints       [:card :dataset]
+          :assertions      {:csv (fn [results]
+                                   (is (partial=
+                                        {:error "You do not have permissions to download the results of this query."}
+                                        results)))}})
 
         (streaming-test/do-test!
          "A user has limited downloads for a query with a join if they have limited permissions for one of the tables"
@@ -425,12 +430,13 @@
      (fn [query]
        (streaming-test/do-test!
         "A table joined to a card, with no download perms for the card, results in blocked download"
-        {:query      query
-         :endpoints  [:card :dataset]
-         :assertions {:csv (fn [results]
-                             (is (partial=
-                                  {:error "You do not have permissions to download the results of this query."}
-                                  results)))}})))))
+        {:query           query
+         :expected-status 403
+         :endpoints       [:card :dataset]
+         :assertions      {:csv (fn [results]
+                                  (is (partial=
+                                       {:error "You do not have permissions to download the results of this query."}
+                                       results)))}})))))
 
 (deftest sandbox-card-test
   (testing "Do we correctly check download perms for queries that involve a sandbox? (#57861)"
@@ -463,15 +469,16 @@
                                                              (mt/id :checkins)   :none}}}
             (streaming-test/do-test!
              "A table with sandbox and not download perms"
-             {:query      {:database (mt/id)
-                           :type     :query
-                           :query    {:source-table (mt/id :checkins)
-                                      :limit        10}}
-              :endpoints  [:card :dataset]
-              :assertions {:csv (fn [results]
-                                  (is (partial=
-                                       {:error "You do not have permissions to download the results of this query."}
-                                       results)))}})))))))
+             {:query           {:database (mt/id)
+                                :type     :query
+                                :query    {:source-table (mt/id :checkins)
+                                           :limit        10}}
+              :expected-status 403
+              :endpoints       [:card :dataset]
+              :assertions      {:csv (fn [results]
+                                       (is (partial=
+                                            {:error "You do not have permissions to download the results of this query."}
+                                            results)))}})))))))
 
 (defn- do-joined-cards-with-native-query-test! [f]
   (testing "Do we correctly apply the least permissive download perms when joining cards where one has a native query?"
@@ -520,21 +527,22 @@
                                                           (mt/id :users)    :limited}}}
          (streaming-test/do-test!
           "Join between MBQL card (venues:full) and native card - should block due to checkins:none"
-          {:query      {:database (mt/id)
-                        :type     :query
-                        :query    {:source-table (format "card__%d" mbql-card-id)
-                                   :joins        [{:fields       [[:field "ID" {:base-type :type/Integer}]]
-                                                   :source-table (format "card__%d" native-card-id)
-                                                   :alias        "native_card"
-                                                   :condition    [:= [:field "ID" {:base-type :type/Integer}]
-                                                                  [:field "VENUE_ID" {:base-type :type/Integer, :join-alias "native_card"}]]
-                                                   :strategy     :left-join}]
-                                   :limit        10}}
-           :endpoints  [:card :dataset]
-           :assertions {:csv (fn [results]
-                               (is (partial=
-                                    {:error "You do not have permissions to download the results of this query."}
-                                    results)))}}))))))
+          {:query           {:database (mt/id)
+                             :type     :query
+                             :query    {:source-table (format "card__%d" mbql-card-id)
+                                        :joins        [{:fields       [[:field "ID" {:base-type :type/Integer}]]
+                                                        :source-table (format "card__%d" native-card-id)
+                                                        :alias        "native_card"
+                                                        :condition    [:= [:field "ID" {:base-type :type/Integer}]
+                                                                       [:field "VENUE_ID" {:base-type :type/Integer, :join-alias "native_card"}]]
+                                                        :strategy     :left-join}]
+                                        :limit        10}}
+           :expected-status 403
+           :endpoints       [:card :dataset]
+           :assertions      {:csv (fn [results]
+                                    (is (partial=
+                                         {:error "You do not have permissions to download the results of this query."}
+                                         results)))}}))))))
 
 (deftest joined-cards-with-native-query-test-3
   (do-joined-cards-with-native-query-test!
@@ -598,15 +606,16 @@
                                                             (mt/id :users)    :full}}}
            (streaming-test/do-test!
             "Card with native source card - should block due to checkins:none in native source"
-            {:query      {:database (mt/id)
-                          :type     :query
-                          :query    {:source-table (format "card__%d" mbql-card-id)
-                                     :limit        10}}
-             :endpoints  [:card :dataset]
-             :assertions {:csv (fn [results]
-                                 (is (partial=
-                                      {:error "You do not have permissions to download the results of this query."}
-                                      results)))}})))))))
+            {:query           {:database (mt/id)
+                               :type     :query
+                               :query    {:source-table (format "card__%d" mbql-card-id)
+                                          :limit        10}}
+             :expected-status 403
+             :endpoints       [:card :dataset]
+             :assertions      {:csv (fn [results]
+                                      (is (partial=
+                                           {:error "You do not have permissions to download the results of this query."}
+                                           results)))}})))))))
 
 (deftest card-with-native-source-card-test-3
   (testing "Do we correctly apply download perms when downloading from a card that uses a source card with a native query?"

enterprise/backend/test/metabase_enterprise/agent_api/api_test.clj

@@ -325,7 +325,7 @@
         (let [decoded (decode-query response)]
           (is (= :mbql/query (lib/normalized-query-type decoded)))
           (is (= (mt/id) (lib/database-id decoded)))
-          (is (= (mt/id :orders) (lib/source-table-id decoded))))))
+          (is (= (mt/id :orders) (lib/primary-source-table-id decoded))))))
 
     (testing "Applies default limit of 100 when no limit is specified"
       (let [table-id (mt/id :orders)

enterprise/backend/test/metabase_enterprise/data_studio/permissions/query_test.clj

@@ -132,7 +132,8 @@
               (perms/set-table-permission! all-users (mt/id :venues) :perms/create-queries :no)
               ;; Without collection permission, query should fail at database check (403)
               (testing "Query should fail because user has no collection permission on published table"
-                (is (= "You don't have permissions to do that."
-                       (mt/with-current-user user-id
-                         (mt/user-http-request user-id :post 403 "dataset"
-                                               (mt/mbql-query venues {:limit 1})))))))))))))
+                (is (=? {:status "failed"
+                         :error  "You do not have permissions to run this query."}
+                        (mt/with-current-user user-id
+                          (mt/user-http-request user-id :post 403 "dataset"
+                                                (mt/mbql-query venues {:limit 1})))))))))))))

enterprise/backend/test/metabase_enterprise/database_routing/e2e_test.clj

@@ -2,6 +2,7 @@
   (:require
    [clojure.java.jdbc :as jdbc]
    [clojure.set :as set]
+   [clojure.string :as str]
    [clojure.test :refer [deftest is testing]]
    [clojurewerkz.quartzite.conversion :as qc]
    [metabase-enterprise.test :as met]
@@ -216,7 +217,7 @@
 
 (defmethod router-dataset-name :default [_driver] "db-router-data")
 
-(doseq [driver [:redshift :databricks :presto-jdbc]]
+(doseq [driver [:redshift :databricks :presto-jdbc :bigquery-cloud-sdk]]
   (defmethod router-dataset-name driver [_driver] "db-routing-data"))
 
 (defmulti routed-dataset-name
@@ -227,7 +228,7 @@
 
 (defmethod routed-dataset-name :default [_driver] "db-routed-data")
 
-(doseq [driver [:redshift :databricks :presto-jdbc]]
+(doseq [driver [:redshift :databricks :presto-jdbc :bigquery-cloud-sdk]]
   (defmethod routed-dataset-name driver [_driver] "db-routing-data"))
 
 (defmulti router-dataset-details
@@ -242,8 +243,8 @@
   {:dbname "db_router_data"
    :enable-multiple-db false})
 
-(defmethod router-dataset-details :bigquery-cloud-sdk [_driver]
-  {:dataset-filters-patterns "metabase_routing_dataset"})
+(defmethod router-dataset-details :bigquery-cloud-sdk [driver]
+  {:dataset-filters-patterns (str "*" (str/replace (router-dataset-name driver) "-" "_"))})
 
 (defmethod router-dataset-details :databricks [driver]
   {:multi-level-schema false
@@ -262,8 +263,8 @@
    :enable-multiple-db false})
 
 (defmethod routed-dataset-details :bigquery-cloud-sdk [driver]
-  {:service-account-json (tx/db-test-env-var-or-throw driver :service-account-json-routing)
-   :dataset-filters-patterns "metabase_routing_dataset"})
+  {:service-account-json     (tx/db-test-env-var-or-throw driver :service-account-json-routing)
+   :dataset-filters-patterns (str "*" (str/replace (routed-dataset-name driver) "-" "_"))})
 
 (defmethod routed-dataset-details :redshift [driver]
   {:db (tx/db-test-env-var-or-throw driver :db-routing)})
@@ -273,22 +274,36 @@
    :multi-level-schema false
    :schema-filters-patterns (routed-dataset-name driver)})
 
+;; the test requires both datasets share a hash (for the name qualifier)
+;; this is because the qp when routing a db does not map to the routed table (it reuses metadata of original)
+;; the end result is the compiled query would retain the routed sha_* qualifier rather than the hash of the original.
+;; using :hash-key now forces the hash key used to sha to be shared between the two datasets.
+
+(defn- router-dataset [driver]
+  (let [dataset (mt/dataset-definition (router-dataset-name driver)
+                                       [["t"
+                                         [{:field-name "f", :base-type :type/Text}]
+                                         [["original-foo"]
+                                          ["original-bar"]]]])]
+    (assoc dataset :hash-key dataset)))
+
+(defn- routed-dataset [driver]
+  (let [router (router-dataset driver)
+        dataset (mt/dataset-definition (routed-dataset-name driver)
+                                       [["t"
+                                         [{:field-name "f", :base-type :type/Text}]
+                                         [["routed-foo"]
+                                          ["routed-bar"]]]])]
+    (assoc dataset :hash-key (:hash-key router))))
+
 (deftest db-routing-e2e-test
   (mt/test-drivers (mt/normal-driver-select {:+features [:database-routing]})
     (mt/with-premium-features #{:database-routing}
       (binding [tx/*use-routing-details* true]
-        (mt/dataset (mt/dataset-definition (routed-dataset-name driver/*driver*)
-                                           [["t"
-                                             [{:field-name "f", :base-type :type/Text}]
-                                             [["routed-foo"]
-                                              ["routed-bar"]]]])
+        (mt/dataset (routed-dataset driver/*driver*)
           (let [routed (mt/db)]
             (binding [tx/*use-routing-details* false]
-              (mt/dataset (mt/dataset-definition (router-dataset-name driver/*driver*)
-                                                 [["t"
-                                                   [{:field-name "f", :base-type :type/Text}]
-                                                   [["original-foo"]
-                                                    ["original-bar"]]]])
+              (mt/dataset (router-dataset driver/*driver*)
                 (let [router (mt/db)]
                   (t2/update! :model/Database (u/the-id routed)
                               {:details (merge (:details routed)

enterprise/backend/test/metabase_enterprise/dependencies/api_test.clj

@@ -3,6 +3,7 @@
    [clojure.test :refer :all]
    [medley.core :as m]
    [metabase-enterprise.dependencies.api :as deps.api]
+   [metabase-enterprise.dependencies.core :as dependencies]
    [metabase-enterprise.dependencies.events]
    [metabase-enterprise.dependencies.findings :as dependencies.findings]
    [metabase-enterprise.dependencies.task.backfill :as dependencies.backfill]
@@ -680,6 +681,31 @@
                                          :source (:source transform)
                                          :target (:target transform)}))))))))
 
+(deftest check-card-bad-transforms-filtered-by-can-read-test
+  (testing "POST /api/ee/dependencies/check-card filters bad_transforms by mi/can-read?"
+    (mt/dataset test-data
+      (mt/with-premium-features #{:dependencies :transforms}
+        (mt/with-temp [:model/User user {:email "test@test.com"}
+                       :model/Transform transform {}]
+          (mt/with-model-cleanup [:model/Card :model/Dependency]
+            (let [base-card (card/create-card! (basic-card) user)
+                  proposed-card {:id (:id base-card)
+                                 :type :question
+                                 :dataset_query (:dataset_query base-card)}]
+              ;; Mock errors-from-proposed-edits to report the transform as broken
+              (with-redefs [dependencies/errors-from-proposed-edits
+                            (constantly {:transform {(:id transform) #{:some-error}}})]
+                (testing "Admin sees broken transforms in response"
+                  (let [response (mt/user-http-request :crowberto :post 200 "ee/dependencies/check-card"
+                                                       proposed-card)]
+                    (is (false? (:success response)))
+                    (is (=? [{:id (:id transform)}]
+                            (:bad_transforms response)))))
+                (testing "Non-admin user cannot see broken transforms they lack read permission for"
+                  (let [response (mt/user-http-request :rasta :post 200 "ee/dependencies/check-card"
+                                                       proposed-card)]
+                    (is (= [] (:bad_transforms response)))))))))))))
+
 (deftest graph-permissions-test
   (testing "GET /api/ee/dependencies/graph requires read permissions on the starting entity"
     (mt/with-premium-features #{:dependencies}

enterprise/backend/test/metabase_enterprise/metabot_v3/tools/api_test.clj

@@ -1360,9 +1360,9 @@
     (let [conversation-id (str (random-uuid))
           rasta-ai-token (ai-session-token)
           crowberto-ai-token (ai-session-token :crowberto (str (random-uuid)))
-          saved-python-library (t2/select-one :model/TransformLibrary :language "python" :path "common.py")]
+          saved-python-library (t2/select-one :model/PythonLibrary :path "common.py")]
       (when (seq saved-python-library)
-        (t2/delete! :model/TransformLibrary :language "python"))
+        (t2/delete! :model/PythonLibrary))
       (try
         (testing "With no Python library present"
           (is (= "Not found."
@@ -1370,9 +1370,8 @@
                                        {:request-options {:headers {"x-metabase-session" crowberto-ai-token}}}
                                        {:arguments {:path "common.py"}
                                         :conversation_id conversation-id}))))
-        (mt/with-temp [:model/TransformLibrary lib1 {:language "python"
-                                                     :path "common.py"
-                                                     :source "def hello():\n    return 'world'"}]
+        (mt/with-temp [:model/PythonLibrary lib1 {:path "common.py"
+                                                  :source "def hello():\n    return 'world'"}]
           (testing "With insufficient permissions"
             (is (= "You don't have permissions to do that."
                    (mt/user-http-request :rasta :post 403 "ee/metabot-tools/get-transform-python-library-details"
@@ -1396,7 +1395,7 @@
                                            :conversation_id conversation-id})))))
         (finally
           (when (seq saved-python-library)
-            (t2/insert! :model/TransformLibrary saved-python-library)))))))
+            (t2/insert! :model/PythonLibrary saved-python-library)))))))
 
 (deftest get-snippets-test
   (mt/with-premium-features #{:metabot-v3}

enterprise/backend/test/metabase_enterprise/remote_sync/impl_test.clj

@@ -1070,17 +1070,16 @@
               (is (true? (remote-sync.settings/remote-sync-transforms))
                   "remote-sync-transforms should be auto-enabled after successful import with transforms"))))))))
 
-(deftest import!-auto-enables-transforms-setting-when-transform-libraries-detected-test
-  (testing "import! auto-enables remote-sync-transforms setting only after successful import with transform-libraries"
+(deftest import!-auto-enables-transforms-setting-when-python-libraries-detected-test
+  (testing "import! auto-enables remote-sync-transforms setting only after successful import with python-libraries"
     (mt/with-premium-features #{:transforms}
-      (mt/with-model-cleanup [:model/RemoteSyncTask :model/TransformLibrary :model/RemoteSyncObject]
+      (mt/with-model-cleanup [:model/RemoteSyncTask :model/PythonLibrary :model/RemoteSyncObject]
         (mt/with-temporary-setting-values [remote-sync-transforms false
                                            remote-sync-enabled true]
           (let [task-id (t2/insert-returning-pk! :model/RemoteSyncTask {:sync_task_type "import" :initiated_by (mt/user->id :rasta)})
                 lib-entity-id "auto-enable-lib-xxxxx"
-                test-files {"main" {(str "transform-libraries/" lib-entity-id ".yaml")
-                                    (format "language: python
-path: uncommon.py
+                test-files {"main" {(str "python-libraries/" lib-entity-id ".yaml")
+                                    (format "path: uncommon.py
 source: |
   # shared code
   def shared_func():
@@ -1089,7 +1088,7 @@ entity_id: %s
 created_at: '2024-08-28T09:46:18.671622Z'
 serdes/meta:
 - id: %s
-  model: TransformLibrary
+  model: PythonLibrary
 " lib-entity-id lib-entity-id)}}
                 mock-source (test-helpers/create-mock-source :initial-files test-files)]
             (is (false? (remote-sync.settings/remote-sync-transforms))
@@ -1098,7 +1097,7 @@ serdes/meta:
               (is (= :success (:status result))
                   "Import should succeed")
               (is (true? (remote-sync.settings/remote-sync-transforms))
-                  "remote-sync-transforms should be auto-enabled after successful import with transform-libraries"))))))))
+                  "remote-sync-transforms should be auto-enabled after successful import with python-libraries"))))))))
 
 (deftest import!-keeps-transforms-setting-disabled-when-no-transforms-present-test
   (testing "import! keeps remote-sync-transforms setting disabled when no transforms are present"
@@ -1133,7 +1132,7 @@ serdes/meta:
                     "remote-sync-transforms should be disabled when no transforms in remote")))))))))
 
 (deftest import!-includes-all-optional-paths-regardless-of-settings-test
-  (testing "import! always includes all optional paths (transforms, transform-libraries, snippets)"
+  (testing "import! always includes all optional paths (transforms, python-libraries, snippets)"
     (mt/with-model-cleanup [:model/RemoteSyncTask]
       (let [task-id (t2/insert-returning-pk! :model/RemoteSyncTask {:sync_task_type "import" :initiated_by (mt/user->id :rasta)})]
         (mt/with-temporary-setting-values [remote-sync-transforms false]
@@ -1152,8 +1151,8 @@ serdes/meta:
                     (let [filter-strs (map str @paths-passed)]
                       (is (some #(str/includes? % "transforms") filter-strs)
                           "transforms path should be included in filters")
-                      (is (some #(str/includes? % "transform-libraries") filter-strs)
-                          "transform-libraries path should be included in filters")
+                      (is (some #(str/includes? % "python-libraries") filter-strs)
+                          "python-libraries path should be included in filters")
                       (is (some #(str/includes? % "snippets") filter-strs)
                           "snippets path should be included in filters"))))))))))))
 

enterprise/backend/test/metabase_enterprise/remote_sync/test_helpers.clj

@@ -4,7 +4,7 @@
    [clojure.string :as str]
    [clojure.test :as t]
    [metabase-enterprise.remote-sync.source.protocol :as source.p]
-   [metabase-enterprise.transforms-runner.models.transform-library :as transform-library]
+   [metabase-enterprise.transforms-python.core :as transforms-python]
    [metabase.util :as u]
    [toucan2.core :as t2]))
 
@@ -287,45 +287,44 @@ width: fixed
           (t2/insert! :model/RemoteSyncTask old-models))))))
 
 (def ^:private builtin-python-library
-  "The built-in TransformLibrary (python) created by migration. Recreated by fixture after cleanup."
-  {:path "common.py" :source "" :language "python"
-   :entity_id (transform-library/builtin-entity-id "python")})
+  "The built-in PythonLibrary created by migration. Recreated by fixture after cleanup."
+  {:path "common.py" :source "" :entity_id transforms-python/builtin-entity-id})
 
 (defn- ensure-builtin-python-library!
-  "Ensures the built-in common.py TransformLibrary exists, creating it if missing."
+  "Ensures the built-in common.py PythonLibrary exists, creating it if missing."
   []
-  (when-not (t2/exists? :model/TransformLibrary :path (:path builtin-python-library) :language "python")
-    (t2/insert! :model/TransformLibrary builtin-python-library)))
+  (when-not (t2/exists? :model/PythonLibrary :path (:path builtin-python-library))
+    (t2/insert! :model/PythonLibrary builtin-python-library)))
 
 (defn clean-optional-feature-models
-  "Test fixture that cleans Transform, TransformTag, TransformLibrary, and namespace collection
+  "Test fixture that cleans Transform, TransformTag, PythonLibrary, and namespace collection
   tables to prevent conflict detection during first-import tests. Preserves built-in TransformTags
-  and recreates the built-in common.py TransformLibrary after cleanup."
+  and recreates the built-in common.py PythonLibrary after cleanup."
   [f]
   (let [old-transforms (t2/select :model/Transform)
         old-tags (t2/select :model/TransformTag :built_in_type nil)
-        old-libs (t2/select :model/TransformLibrary)
+        old-libs (t2/select :model/PythonLibrary)
         old-ns-colls (t2/select :model/Collection :namespace [:in ["transforms" "snippets"]])]
     (try
       (t2/delete! :model/TransformTag :built_in_type nil)
       (t2/delete! :model/Transform)
-      (t2/delete! :model/TransformLibrary)
+      (t2/delete! :model/PythonLibrary)
       (t2/delete! :model/Collection :namespace [:in ["transforms" "snippets"]])
       (f)
       (finally
         (t2/delete! :model/TransformTag :built_in_type nil)
         (t2/delete! :model/Transform)
-        (t2/delete! :model/TransformLibrary)
+        (t2/delete! :model/PythonLibrary)
         (t2/delete! :model/Collection :namespace [:in ["transforms" "snippets"]])
         (when (seq old-transforms) (t2/insert! :model/Transform old-transforms))
         (when (seq old-tags) (t2/insert! :model/TransformTag old-tags))
-        (when (seq old-libs) (t2/insert! :model/TransformLibrary old-libs))
+        (when (seq old-libs) (t2/insert! :model/PythonLibrary old-libs))
         (when (seq old-ns-colls) (t2/insert! :model/Collection old-ns-colls))
         (ensure-builtin-python-library!)))))
 
 (def clean-remote-sync-state
   "Composed test fixture that ensures RemoteSyncObject, RemoteSyncTask, and optional feature
-  model tables (Transform, TransformTag, TransformLibrary) are clean."
+  model tables (Transform, TransformTag, PythonLibrary) are clean."
   (t/compose-fixtures clean-object (t/compose-fixtures clean-task-table clean-optional-feature-models)))
 
 (defn generate-table-yaml

enterprise/backend/test/metabase_enterprise/remote_sync/transforms_test.clj

@@ -10,7 +10,7 @@
    [metabase-enterprise.remote-sync.source.protocol :as source.p]
    [metabase-enterprise.remote-sync.spec :as spec]
    [metabase-enterprise.remote-sync.test-helpers :as test-helpers]
-   [metabase-enterprise.transforms-runner.models.transform-library :as transform-library]
+   [metabase-enterprise.transforms-python.core :as transforms-python]
    [metabase.collections.models.collection :as collection]
    [metabase.events.core :as events]
    [metabase.search.core :as search]
@@ -499,164 +499,163 @@ is_sample: false
                 (is (t2/exists? :model/Collection :entity_id remote-coll-entity-id :namespace "transforms")
                     "Remote transforms collection should be imported")))))))))
 
-;;; ------------------------------------------- TransformLibrary Tests -------------------------------------------
+;;; ------------------------------------------- PythonLibrary Tests -------------------------------------------
 
-(defn- generate-transform-library-yaml
-  "Generates YAML content for a TransformLibrary."
+(defn- generate-python-library-yaml
+  "Generates YAML content for a PythonLibrary."
   [entity-id path source]
-  (format "language: python
-path: %s
+  (format "path: %s
 source: |
 %s
 entity_id: %s
 created_at: '2024-08-28T09:46:18.671622Z'
 serdes/meta:
 - id: %s
-  model: TransformLibrary
+  model: PythonLibrary
 "
           path
           (str/join "\n" (map #(str "  " %) (str/split-lines source)))
           entity-id
           entity-id))
 
-(deftest transform-library-event-creates-sync-object-when-setting-enabled-test
-  (testing "Creating a TransformLibrary creates a RemoteSyncObject entry when remote-sync-transforms is enabled"
+(deftest python-library-event-creates-sync-object-when-setting-enabled-test
+  (testing "Creating a PythonLibrary creates a RemoteSyncObject entry when remote-sync-transforms is enabled"
     (mt/with-premium-features #{:transforms}
       (mt/with-temporary-setting-values [remote-sync-transforms true
                                          remote-sync-enabled true]
-        (let [library (t2/insert-returning-instance! :model/TransformLibrary {:path "common.py" :source "# test" :language "python"})]
+        (let [library (t2/insert-returning-instance! :model/PythonLibrary {:path "common.py" :source "# test"})]
           (is (t2/exists? :model/RemoteSyncObject
-                          :model_type "TransformLibrary"
+                          :model_type "PythonLibrary"
                           :model_id (:id library))
-              "TransformLibrary should be tracked when remote-sync-transforms is enabled"))))))
+              "PythonLibrary should be tracked when remote-sync-transforms is enabled"))))))
 
-(deftest transform-library-event-ignored-when-setting-disabled-test
-  (testing "TransformLibrary events are ignored when remote-sync-transforms is disabled"
+(deftest python-library-event-ignored-when-setting-disabled-test
+  (testing "PythonLibrary events are ignored when remote-sync-transforms is disabled"
     (mt/with-premium-features #{:transforms}
       (mt/with-temporary-setting-values [remote-sync-transforms false
                                          remote-sync-enabled true]
-        (let [library (t2/insert-returning-instance! :model/TransformLibrary {:path "common.py" :source "# test" :language "python"})]
+        (let [library (t2/insert-returning-instance! :model/PythonLibrary {:path "common.py" :source "# test"})]
           (is (not (t2/exists? :model/RemoteSyncObject
-                               :model_type "TransformLibrary"
+                               :model_type "PythonLibrary"
                                :model_id (:id library)))
-              "TransformLibrary should NOT be tracked when remote-sync-transforms is disabled"))))))
+              "PythonLibrary should NOT be tracked when remote-sync-transforms is disabled"))))))
 
-(deftest transform-library-event-updates-sync-object-test
-  (testing "Updating a TransformLibrary updates the RemoteSyncObject entry when setting is enabled"
+(deftest python-library-event-updates-sync-object-test
+  (testing "Updating a PythonLibrary updates the RemoteSyncObject entry when setting is enabled"
     (mt/with-premium-features #{:transforms}
       (mt/with-temporary-setting-values [remote-sync-transforms true
                                          remote-sync-enabled true]
-        (let [library (t2/insert-returning-instance! :model/TransformLibrary {:path "common.py" :source "# test" :language "python"})]
-          (t2/update! :model/RemoteSyncObject {:model_type "TransformLibrary" :model_id (:id library)}
+        (let [library (t2/insert-returning-instance! :model/PythonLibrary {:path "common.py" :source "# test"})]
+          (t2/update! :model/RemoteSyncObject {:model_type "PythonLibrary" :model_id (:id library)}
                       {:status "synced"})
-          (t2/update! :model/TransformLibrary (:id library) {:source "# updated"})
+          (t2/update! :model/PythonLibrary (:id library) {:source "# updated"})
           (let [entry (t2/select-one :model/RemoteSyncObject
-                                     :model_type "TransformLibrary"
+                                     :model_type "PythonLibrary"
                                      :model_id (:id library))]
             (is (= "update" (:status entry))
-                "TransformLibrary should have 'update' status after modification")))))))
+                "PythonLibrary should have 'update' status after modification")))))))
 
-(deftest export-includes-transform-library-when-setting-enabled-test
-  (testing "Export includes TransformLibrary when remote-sync-transforms is enabled"
+(deftest export-includes-python-library-when-setting-enabled-test
+  (testing "Export includes PythonLibrary when remote-sync-transforms is enabled"
     (mt/with-premium-features #{:transforms}
       (mt/with-temporary-setting-values [remote-sync-type :read-write
                                          remote-sync-transforms true
                                          remote-sync-enabled true]
         (mt/with-model-cleanup [:model/RemoteSyncTask]
           (let [task-id (t2/insert-returning-pk! :model/RemoteSyncTask {:sync_task_type "export" :initiated_by (mt/user->id :rasta)})
-                library (t2/insert-returning-instance! :model/TransformLibrary {:path "common.py" :source "# shared code" :language "python"})]
-            (t2/update! :model/RemoteSyncObject {:model_type "TransformLibrary" :model_id (:id library)}
+                library (t2/insert-returning-instance! :model/PythonLibrary {:path "common.py" :source "# shared code"})]
+            (t2/update! :model/RemoteSyncObject {:model_type "PythonLibrary" :model_id (:id library)}
                         {:status "create"})
             (let [mock-source (test-helpers/create-mock-source)
                   result (impl/export! (source.p/snapshot mock-source) task-id "Test export")]
               (is (= :success (:status result))
                   (str "Export should succeed. Result: " result))
               (let [files-after-export (get @(:files-atom mock-source) "main")]
-                (is (some #(str/includes? % "transform-libraries/") (keys files-after-export))
-                    "Export should include the TransformLibrary file")))))))))
+                (is (some #(str/includes? % "python-libraries/") (keys files-after-export))
+                    "Export should include the PythonLibrary file")))))))))
 
-(deftest import-transform-library-from-yaml-test
-  (testing "Import brings in TransformLibrary from YAML files"
+(deftest import-python-library-from-yaml-test
+  (testing "Import brings in PythonLibrary from YAML files"
     (mt/with-premium-features #{:transforms}
       (mt/with-temporary-setting-values [remote-sync-transforms true
                                          remote-sync-enabled true]
         (mt/with-model-cleanup [:model/RemoteSyncTask]
           (let [task-id (t2/insert-returning-pk! :model/RemoteSyncTask {:sync_task_type "import" :initiated_by (mt/user->id :rasta)})
                 lib-entity-id (u/generate-nano-id)
-                test-files {"main" {(str "transform-libraries/" lib-entity-id ".yaml")
-                                    (generate-transform-library-yaml lib-entity-id "common.py" "def shared_func():\n    return 42")}}
+                test-files {"main" {(str "python-libraries/" lib-entity-id ".yaml")
+                                    (generate-python-library-yaml lib-entity-id "common.py" "def shared_func():\n    return 42")}}
                 mock-source (test-helpers/create-mock-source :initial-files test-files)
                 snapshot (source.p/snapshot mock-source)
                 result (impl/import! snapshot task-id)]
             (is (= :success (:status result))
                 (str "Import should succeed. Result: " result))
-            (is (t2/exists? :model/TransformLibrary :path "common.py")
-                "TransformLibrary should be imported")
-            (when-let [library (t2/select-one :model/TransformLibrary :path "common.py")]
+            (is (t2/exists? :model/PythonLibrary :path "common.py")
+                "PythonLibrary should be imported")
+            (when-let [library (t2/select-one :model/PythonLibrary :path "common.py")]
               (is (str/includes? (:source library) "def shared_func()")
-                  "TransformLibrary source should be imported correctly"))))))))
+                  "PythonLibrary source should be imported correctly"))))))))
 
-(deftest import-removes-transform-library-not-on-remote-test
-  (testing "Import removes local TransformLibrary that doesn't exist on the remote"
+(deftest import-removes-python-library-not-on-remote-test
+  (testing "Import removes local PythonLibrary that doesn't exist on the remote"
     (mt/with-premium-features #{:transforms}
       (mt/with-temporary-setting-values [remote-sync-transforms true
                                          remote-sync-enabled true]
         (mt/with-model-cleanup [:model/RemoteSyncTask]
           (let [task-id (t2/insert-returning-pk! :model/RemoteSyncTask {:sync_task_type "import" :initiated_by (mt/user->id :rasta)})
-                local-library (t2/insert-returning-instance! :model/TransformLibrary {:path "common.py" :source "# local only" :language "python"})]
-            (is (t2/exists? :model/TransformLibrary :id (:id local-library))
-                "Local TransformLibrary should exist before import")
+                local-library (t2/insert-returning-instance! :model/PythonLibrary {:path "common.py" :source "# local only"})]
+            (is (t2/exists? :model/PythonLibrary :id (:id local-library))
+                "Local PythonLibrary should exist before import")
             (let [test-files {"main" {}}
                   mock-source (test-helpers/create-mock-source :initial-files test-files)
                   result (impl/import! (source.p/snapshot mock-source) task-id)]
               (is (= :success (:status result))
                   (str "Import should succeed. Result: " result))
-              (is (not (t2/exists? :model/TransformLibrary :id (:id local-library)))
-                  "Local TransformLibrary should be deleted after import since it wasn't on remote"))))))))
+              (is (not (t2/exists? :model/PythonLibrary :id (:id local-library)))
+                  "Local PythonLibrary should be deleted after import since it wasn't on remote"))))))))
 
-(deftest import-preserves-builtin-transform-library-test
-  (testing "Import does NOT delete the built-in TransformLibrary even when it's not on remote"
+(deftest import-preserves-builtin-python-library-test
+  (testing "Import does NOT delete the built-in PythonLibrary even when it's not on remote"
     (mt/with-premium-features #{:transforms}
       (mt/with-temporary-setting-values [remote-sync-transforms true
                                          remote-sync-enabled true]
         (mt/with-model-cleanup [:model/RemoteSyncTask]
           (let [task-id (t2/insert-returning-pk! :model/RemoteSyncTask {:sync_task_type "import" :initiated_by (mt/user->id :rasta)})]
-            (t2/insert-returning-instance! :model/TransformLibrary
+            ;; Create the builtin library explicitly (fixture cleans PythonLibrary before test)
+            (t2/insert-returning-instance! :model/PythonLibrary
                                            {:path "common.py"
                                             :source "# builtin"
-                                            :language "python"
-                                            :entity_id (transform-library/builtin-entity-id "python")})
-            (is (t2/exists? :model/TransformLibrary :entity_id (transform-library/builtin-entity-id "python"))
-                "Built-in TransformLibrary should exist before import")
-            (let [test-files {"main" {}}
+                                            :entity_id transforms-python/builtin-entity-id})
+            (is (t2/exists? :model/PythonLibrary :entity_id transforms-python/builtin-entity-id)
+                "Built-in PythonLibrary should exist before import")
+            (let [test-files {"main" {}}  ;; Empty remote - no python libraries
                   mock-source (test-helpers/create-mock-source :initial-files test-files)
                   result (impl/import! (source.p/snapshot mock-source) task-id)]
               (is (= :success (:status result))
                   (str "Import should succeed. Result: " result))
-              (is (t2/exists? :model/TransformLibrary :entity_id (transform-library/builtin-entity-id "python"))
-                  "Built-in TransformLibrary should NOT be deleted after import"))))))))
+              (is (t2/exists? :model/PythonLibrary :entity_id transforms-python/builtin-entity-id)
+                  "Built-in PythonLibrary should NOT be deleted after import"))))))))
 
-(deftest import-replaces-transform-library-with-remote-version-test
-  (testing "Import updates local TransformLibrary when remote has same entity_id"
+(deftest import-replaces-python-library-with-remote-version-test
+  (testing "Import updates local PythonLibrary when remote has same entity_id"
     (mt/with-premium-features #{:transforms}
       (mt/with-temporary-setting-values [remote-sync-transforms true
                                          remote-sync-enabled true]
         (mt/with-model-cleanup [:model/RemoteSyncTask]
           (let [task-id (t2/insert-returning-pk! :model/RemoteSyncTask {:sync_task_type "import" :initiated_by (mt/user->id :rasta)})
-                local-library (t2/insert-returning-instance! :model/TransformLibrary {:path "common.py" :source "# local version" :language "python"})
+                local-library (t2/insert-returning-instance! :model/PythonLibrary {:path "common.py" :source "# local version"})
                 local-entity-id (:entity_id local-library)]
-            (is (t2/exists? :model/TransformLibrary :path "common.py")
-                "Local TransformLibrary should exist before import")
-            (let [test-files {"main" {(str "transform-libraries/" local-entity-id ".yaml")
-                                      (generate-transform-library-yaml local-entity-id "common.py" "# remote version\ndef new_func():\n    pass")}}
+            (is (t2/exists? :model/PythonLibrary :path "common.py")
+                "Local PythonLibrary should exist before import")
+            (let [test-files {"main" {(str "python-libraries/" local-entity-id ".yaml")
+                                      (generate-python-library-yaml local-entity-id "common.py" "# remote version\ndef new_func():\n    pass")}}
                   mock-source (test-helpers/create-mock-source :initial-files test-files)
                   result (impl/import! (source.p/snapshot mock-source) task-id)]
               (is (= :success (:status result))
                   (str "Import should succeed. Result: " result))
-              (let [library (t2/select-one :model/TransformLibrary :path "common.py")]
-                (is (some? library) "TransformLibrary should still exist")
+              (let [library (t2/select-one :model/PythonLibrary :path "common.py")]
+                (is (some? library) "PythonLibrary should still exist")
                 (is (str/includes? (:source library) "# remote version")
-                    "TransformLibrary source should be updated with remote version")))))))))
+                    "PythonLibrary source should be updated with remote version")))))))))
 
 ;;; ------------------------------------------- Transform Sync Behavior Tests -------------------------------------------
 
@@ -830,11 +829,11 @@ serdes/meta:
                                                                                      :collection_id coll-id}
                        :model/TransformTag {tag-id :id tag-eid :entity_id} {:name "Test Tag"
                                                                             :built_in_type nil}]
-          (let [library (t2/insert-returning-instance! :model/TransformLibrary {:path "common.py" :source "# test" :language "python"})
+          (let [library (t2/insert-returning-instance! :model/PythonLibrary {:path "common.py" :source "# test"})
                 lib-eid (:entity_id library)]
             (is (t2/exists? :model/Transform :id transform-id))
             (is (t2/exists? :model/TransformTag :id tag-id))
-            (is (t2/exists? :model/TransformLibrary :id (:id library)))
+            (is (t2/exists? :model/PythonLibrary :id (:id library)))
             (let [paths-before (spec/build-all-removal-paths)]
               (is (not (some #(str/includes? % transform-eid) paths-before))
                   "Transform should not be in removal paths before setting is disabled"))
@@ -851,7 +850,7 @@ serdes/meta:
               (is (some #(str/includes? % tag-eid) paths-after)
                   "TransformTag should be in removal paths after setting is disabled")
               (is (some #(str/includes? % lib-eid) paths-after)
-                  "TransformLibrary should be in removal paths after setting is disabled"))))))))
+                  "PythonLibrary should be in removal paths after setting is disabled"))))))))
 
 (deftest build-all-removal-paths-excludes-builtin-transform-tags-test
   (testing "build-all-removal-paths respects :conditions and excludes built-in tags"

enterprise/backend/test/metabase_enterprise/sandbox/api/card_test.clj

@@ -131,10 +131,11 @@
             (data-perms/set-table-permission! group (mt/id :venues) :perms/view-data view-perm)
             (data-perms/set-table-permission! group (mt/id :venues) :perms/create-queries create-perm)
             (testing (str "view-data: " view-perm ", create-queries: " create-perm)
-              (let [response (mt/user-http-request user-id :post 202 (str "card/" (u/the-id card) "/query"))]
-                (if should-succeed?
-                  (is (= 1 (count (mt/rows response))))
-                  (is (thrown? clojure.lang.ExceptionInfo (mt/rows response))))))))))))
+              (if should-succeed?
+                (let [response (mt/user-http-request user-id :post 202 (str "card/" (u/the-id card) "/query"))]
+                  (is (= 1 (count (mt/rows response)))))
+                (let [response (mt/user-http-request user-id :post 403 (str "card/" (u/the-id card) "/query"))]
+                  (is (partial= {:error_type "missing-required-permissions"} response)))))))))))
 
 (deftest sandbox-join-permissions-test
   (testing "Sandboxed query can't be saved when sandboxed table is joined to a table that the current user doesn't have access to"

enterprise/backend/test/metabase_enterprise/serialization/api_test.clj

@@ -39,7 +39,7 @@
    #"/databases/.*/schemas/(.*)"               :schema
    #"/databases/(.*)\.yaml"                    :database
    #"/transforms/(.*)\.yaml"                   :transform
-   #"/transform-libraries/(.*)\.yaml" :transform-library])
+   #"/python-libraries/(.*)\.yaml"             :python-library])
 
 (defn- file-type
   "Find out entity type by file path"
@@ -122,13 +122,13 @@
               (testing "API respects parameters"
                 (let [f (mt/user-http-request :crowberto :post 200 "ee/serialization/export" {}
                                               :all_collections false :data_model false :settings true)]
-                  (is (= #{:log :dir :settings :transform :transform-library}
+                  (is (= #{:log :dir :settings :transform :python-library}
                          (tar-file-types f)))))
 
               (testing "We can export just a single collection"
                 (let [f (mt/user-http-request :crowberto :post 200 "ee/serialization/export" {}
                                               :collection (:id coll) :data_model false :settings false)]
-                  (is (= #{:log :dir :dashboard :card :collection :transform :transform-library}
+                  (is (= #{:log :dir :dashboard :card :collection :transform :python-library}
                          (tar-file-types f)))))
 
               (testing "We can export two collections"
@@ -144,18 +144,18 @@
                 (let [f (mt/user-http-request :crowberto :post 200 "ee/serialization/export" {}
                                               ;; eid:... syntax is kept for backward compat
                                               :collection (str "eid:" (:entity_id coll)) :data_model false :settings false)]
-                  (is (= #{:log :dir :dashboard :card :collection :transform :transform-library}
+                  (is (= #{:log :dir :dashboard :card :collection :transform :python-library}
                          (tar-file-types f)))))
 
               (testing "We can export that collection using entity id"
                 (let [f (mt/user-http-request :crowberto :post 200 "ee/serialization/export" {}
                                               :collection (:entity_id coll) :data_model false :settings false)]
-                  (is (= #{:log :dir :dashboard :card :collection :transform :transform-library}
+                  (is (= #{:log :dir :dashboard :card :collection :transform :python-library}
                          (tar-file-types f)))))
 
               (testing "Default export: all-collections, data-model, settings"
                 (let [f (mt/user-http-request :crowberto :post 200 "ee/serialization/export" {})]
-                  (is (= #{:transform :log :dir :dashboard :card :collection :settings :schema :database :transform-library}
+                  (is (= #{:transform :log :dir :dashboard :card :collection :settings :schema :database :python-library}
                          (tar-file-types f)))))
 
               (testing "On exception API returns log"
@@ -221,7 +221,7 @@
                     io/input-stream)
             ba  (#'api.serialization/ba-copy res)]
         (testing "Archive contains correct number of files with proper log entries"
-          (is (= 14
+          (is (= 13
                  (with-open [tar (open-tar ba)]
                    (count
                     (for [^TarArchiveEntry e (u.compress/entries tar)
@@ -229,7 +229,7 @@
                       (do
                         (condp re-find (.getName e)
                           #"/export.log$" (testing "Log contains extract and store entries"
-                                            (is (= (+ #_extract 13 #_store 13)
+                                            (is (= (+ #_extract 12 #_store 12)
                                                    (count (line-seq (io/reader tar))))))
                           nil)
                         (.getName e))))))))
@@ -243,7 +243,7 @@
                    "settings"        false
                    "field_values"    false
                    "duration_ms"     (every-pred number? pos?)
-                   "count"           13
+                   "count"           12
                    "error_count"     0
                    "source"          "api"
                    "secrets"         false
@@ -273,7 +273,7 @@
                                                   {:request-options {:headers {"content-type" "multipart/form-data"}}}
                                                   {:file ba}))]
           (testing "Log contains imported entity types"
-            (is (= #{"Collection" "Dashboard" "Card" "TransformLibrary" "TransformJob" "TransformTag"}
+            (is (= #{"Collection" "Dashboard" "Card" "PythonLibrary" "TransformJob" "TransformTag"}
                    (log-types (line-seq (io/reader (io/input-stream res)))))))
 
           (testing "Entities are restored in the database"
@@ -285,8 +285,8 @@
                      "direction"     "import"
                      "duration_ms"   pos?
                      "source"        "api"
-                     "models"        "Card,Collection,Dashboard,TransformJob,TransformLibrary,TransformTag"
-                     "count"         13
+                     "models"        "Card,Collection,Dashboard,PythonLibrary,TransformJob,TransformTag"
+                     "count"         12
                      "error_count"   0
                      "success"       true
                      "error_message" nil}
@@ -349,7 +349,7 @@
                                           :continue_on_error true)
                 log (slurp (io/input-stream res))]
             (testing "Log shows loaded entities and error"
-              (is (= #{"Dashboard" "Card" "Collection" "TransformJob" "TransformTag" "TransformLibrary"}
+              (is (= #{"Dashboard" "Card" "Collection" "TransformJob" "TransformTag" "PythonLibrary"}
                      (log-types (str/split-lines log))))
               (is (re-find #"Collection 'DoesNotExist' was not found" log)))
 
@@ -359,9 +359,9 @@
                        "direction"   "import"
                        "source"      "api"
                        "duration_ms" int?
-                       "count"       12
+                       "count"       11
                        "error_count" 1
-                       "models"      "Collection,Dashboard,TransformJob,TransformLibrary,TransformTag"}
+                       "models"      "Collection,Dashboard,PythonLibrary,TransformJob,TransformTag"}
                       (-> (snowplow-test/pop-event-data-and-user-id!) last :data))))))))))
 
 (deftest import-invalid-archive-test
@@ -429,7 +429,7 @@
             (doseq [^TarArchiveEntry e (u.compress/entries tar)]
               (condp re-find (.getName e)
                 #"/export.log$" (testing "Log shows extract entries, error, and store entries"
-                                  (is (= (+ #_extract 13 #_error 1 #_store 12)
+                                  (is (= (+ #_extract 12 #_error 1 #_store 11)
                                          (count (line-seq (io/reader tar))))))
                 nil))))
 
@@ -442,7 +442,7 @@
                    "settings"        false
                    "field_values"    false
                    "duration_ms"     pos?
-                   "count"           13
+                   "count"           11
                    "error_count"     1
                    "source"          "api"
                    "secrets"         false

enterprise/backend/test/metabase_enterprise/serialization/cmd_test.clj

@@ -45,7 +45,7 @@
                          "settings"        true
                          "field_values"    false
                          "duration_ms"     pos?
-                         "count"           13
+                         "count"           12
                          "source"          "cli"
                          "secrets"         false
                          "success"         true
@@ -60,8 +60,8 @@
                          "direction"     "import"
                          "duration_ms"   pos?
                          "source"        "cli"
-                         "models"        "Card,Collection,Setting,TransformJob,TransformLibrary,TransformTag"
-                         "count"         13
+                         "models"        "Card,Collection,PythonLibrary,Setting,TransformJob,TransformTag"
+                         "count"         12
                          "success"       true
                          "error_message" nil}
                         (-> (snowplow-test/pop-event-data-and-user-id!) first :data))))

enterprise/backend/test/metabase_enterprise/serialization/v2/e2e_test.clj

@@ -430,6 +430,7 @@
                   (is (= [{:id                   "abc",
                            :name                 "CATEGORY",
                            :type                 :category,
+                           :position             0,
                            :values_source_config {:card_id     (:entity_id card1s),
                                                   :value_field [:field
                                                                 ["my-db" nil "CUSTOMERS" (:name field1s)]
@@ -442,6 +443,7 @@
                            [{:id                   "abc",
                              :name                 "CATEGORY",
                              :type                 :category,
+                             :position             0,
                              :values_source_config {:card_id     (:entity_id card1s),
                                                     :value_field [:field
                                                                   ["my-db" nil "CUSTOMERS" (:name field1s)]

enterprise/backend/test/metabase_enterprise/serialization/v2/extract_test.clj

@@ -1296,6 +1296,7 @@
           (is (= [{:id                   "abc",
                    :type                 :category,
                    :name                 "CATEGORY",
+                   :position             0,
                    :values_source_type   :card
                    :values_source_config {:card_id card-eid-1,
                                           :value_field [:field ["My Database" nil "Schemaless Table" "A Field"] nil]}}]
@@ -1516,7 +1517,7 @@
                                                                                                     :dimension dimension}}}}})}}}]
 
       (testing "selecting a collection includes settings metabot and data model by default"
-        (is (= #{"Card" "Collection" "Dashboard" "Database" "TransformLibrary" "Setting" "TransformTag" "TransformJob"}
+        (is (= #{"Card" "Collection" "Dashboard" "Database" "PythonLibrary" "Setting" "TransformTag" "TransformJob"}
                (->> (extract/extract {:targets [["Collection" coll1-id]]})
                     (map (comp :model first serdes/path))
                     set))))
@@ -2530,23 +2531,23 @@
           (testing "fields from non-published table are NOT exported"
             (is (= 2 (count field-ids)))))))))
 
-(deftest transform-library-test
+(deftest python-library-test
   (mt/with-empty-h2-app-db!
-    (t2/delete! :model/TransformLibrary)
-    (ts/with-temp-dpc [:model/TransformLibrary {lib-id :id lib-entity-id :entity_id} {:language "python"
-                                                                                      :path "common"
-                                                                                      :source "def helper():\n    return 42"}]
-      (testing "transform library extraction"
-        (let [ser (serdes/extract-one "TransformLibrary" {} (t2/select-one :model/TransformLibrary :id lib-id))]
-          (is (=? {:serdes/meta [{:model "TransformLibrary"
+    ;; Delete any pre-existing python_library entries from migrations
+    (t2/delete! :model/PythonLibrary)
+    (ts/with-temp-dpc [:model/PythonLibrary {lib-id :id lib-entity-id :entity_id} {:path   "common"
+                                                                                   :source "def helper():\n    return 42"}]
+      (testing "python library extraction"
+        (let [ser (serdes/extract-one "PythonLibrary" {} (t2/select-one :model/PythonLibrary :id lib-id))]
+          (is (=? {:serdes/meta [{:model "PythonLibrary"
                                   :id    lib-entity-id}]
-                   :language "python"
                    :path        "common.py"
                    :source      "def helper():\n    return 42"
                    :entity_id   lib-entity-id
                    :created_at  string?}
                   ser))
           (is (not (contains? ser :id)))
+
           (testing "has no dependencies"
             (is (empty? (serdes/dependencies ser)))))))))
 
@@ -2555,8 +2556,12 @@
                 {:id "alpha" :name "A param" :type :category}
                 {:id "middle" :name "M param" :type :category}]
         result (serdes/export-parameters params)]
-    (is (= ["alpha" "middle" "zebra"]
-           (map :id result)))))
+    (testing "export sorts parameters by :id for stable diffs"
+      (is (= ["alpha" "middle" "zebra"]
+             (map :id result))))
+    (testing "export stamps :position from original array index"
+      (is (= [1 2 0]
+             (map :position result))))))
 
 (deftest ^:parallel export-parameters-empty-input-test
   (is (= [] (serdes/export-parameters []))))
@@ -2567,14 +2572,16 @@
 (deftest ^:parallel export-parameters-single-element-test
   (let [params [{:id "only" :name "Only param"}]
         result (serdes/export-parameters params)]
-    (is (= ["only"] (map :id result)))))
+    (is (= ["only"] (map :id result)))
+    (is (= [0] (map :position result)))))
 
 (deftest ^:parallel export-parameters-nil-id-sorts-first-test
   (let [params [{:id "beta" :name "B"}
                 {:id nil :name "Nil"}
                 {:id "alpha" :name "A"}]
         result (serdes/export-parameters params)]
-    (is (= [nil "alpha" "beta"] (map :id result)))))
+    (is (= [nil "alpha" "beta"] (map :id result)))
+    (is (= [1 2 0] (map :position result)))))
 
 (deftest ^:parallel export-parameter-mappings-sorts-by-parameter-id-test
   (let [mappings [{:parameter_id "z-param" :target [:dimension [:field 1 nil]]}
@@ -2614,7 +2621,32 @@
                                                                      :card_id card-id
                                                                      :target [:dimension [:field 2 nil]]}]}]
       (let [ser (ts/extract-one "Dashboard" dash-id)]
-        (is (= ["alpha" "beta" "zebra"]
-               (map :id (:parameters ser))))
+        (testing "parameters are sorted by :id"
+          (is (= ["alpha" "beta" "zebra"]
+                 (map :id (:parameters ser)))))
+        (testing "positions preserve original display order"
+          (is (= [1 2 0]
+                 (map :position (:parameters ser)))))
         (is (= ["alpha" "zebra"]
                (map :parameter_id (-> ser :dashcards first :parameter_mappings))))))))
+
+(deftest ^:parallel import-parameters-round-trip-test
+  (testing "export stamps position, import restores original display order"
+    (let [params   [{:id "zebra" :name "Z param" :type :category}
+                    {:id "alpha" :name "A param" :type :category}
+                    {:id "middle" :name "M param" :type :category}]
+          exported (serdes/export-parameters params)
+          imported (serdes/import-parameters exported)]
+      (is (= ["alpha" "middle" "zebra"] (map :id exported))
+          "export sorts by id")
+      (is (= [1 2 0] (map :position exported))
+          "export stamps position from original array index")
+      (is (= ["zebra" "alpha" "middle"] (map :id imported))
+          "import restores original display order via position"))))
+
+(deftest ^:parallel import-parameters-without-position-preserves-order-test
+  (testing "importing legacy params without :position preserves existing order"
+    (let [params   [{:id "zebra" :name "Z param" :type :category}
+                    {:id "alpha" :name "A param" :type :category}]
+          imported (serdes/import-parameters params)]
+      (is (= ["zebra" "alpha"] (map :id imported))))))

enterprise/backend/test/metabase_enterprise/serialization/v2/storage_test.clj

@@ -238,16 +238,16 @@
                    ["parent.child.yaml"]}
                  (file-set (io/file dump-dir "databases" "mydb" "tables" "table" "fields")))))))))
 
-(deftest transform-library-storage-test
+(deftest python-library-storage-test
   (ts/with-random-dump-dir [dump-dir "serdesv2-"]
     (mt/with-empty-h2-app-db!
-      (t2/delete! :model/TransformLibrary)
-      (ts/with-temp-dpc [:model/TransformLibrary lib {:language "python" :path "common" :source "def test(): pass"}]
+      (t2/delete! :model/PythonLibrary)
+      (ts/with-temp-dpc [:model/PythonLibrary lib {:path "common" :source "def test(): pass"}]
         (let [export (into [] (extract/extract {:no-settings true :no-data-model true}))]
           (storage/store! export dump-dir)
-          (testing "transform library stored at top-level transform-libraries/"
+          (testing "python library stored at top-level python-libraries/"
             (is (= #{[(str (:entity_id lib) ".yaml")]}
-                   (file-set (io/file dump-dir "transform-libraries"))))))))))
+                   (file-set (io/file dump-dir "python-libraries"))))))))))
 
 (deftest name-too-long-test
   (ts/with-random-dump-dir [dump-dir "serdesv2-"]