wtf

.claude/skills/clojure-eval/SKILL.md

@@ -0,0 +1,173 @@
+---
+name: clojure-eval
+description: Evaluate Clojure code via nREPL using clj-nrepl-eval. Use this when you need to test code, check if edited files compile, verify function behavior, or interact with a running REPL session.
+---
+
+# Clojure REPL Evaluation
+
+## When to Use This Skill
+
+Use this skill when you need to:
+- **Verify that edited Clojure files compile and load correctly**
+- Test function behavior interactively
+- Check the current state of the REPL
+- Debug code by evaluating expressions
+- Require or load namespaces for testing
+- Validate that code changes work before committing
+
+## How It Works
+
+The `clj-nrepl-eval` command evaluates Clojure code against an nREPL server. **Session state persists between evaluations**, so you can require a namespace in one evaluation and use it in subsequent calls. Each host:port combination maintains its own session file.
+
+## Instructions
+
+### 0. Discover and select nREPL server
+
+First, discover what nREPL servers are running in the current directory:
+
+```bash
+clj-nrepl-eval --discover-ports
+```
+
+This will show all nREPL servers (Clojure, Babashka, shadow-cljs, etc.) running in the current project directory.
+
+**Then use the AskUserQuestion tool:**
+
+- **If ports are discovered:** Prompt user to select which nREPL port to use:
+  - **question:** "Which nREPL port would you like to use?"
+  - **header:** "nREPL Port"
+  - **options:** Present each discovered port as an option with:
+    - **label:** The port number 
+    - **description:** The server type and status (e.g., "Clojure nREPL server in current directory")
+  - Include up to 4 discovered ports as options
+  - The user can select "Other" to enter a custom port number
+
+- **If no ports are discovered:** Prompt user how to start an nREPL server:
+  - **question:** "No nREPL servers found. How would you like to start one?"
+  - **header:** "Start nREPL"
+  - **options:**
+    - **label:** "deps.edn alias", **description:** "Find and use an nREPL alias in deps.edn"
+    - **label:** "Leiningen", **description:** "Start nREPL using 'lein repl'"
+  - The user can select "Other" for alternative methods or if they already have a server running on a specific port
+
+IMPORTANT: IF you start a REPL do not supply a port let the nREPL start and return the port that it was started on.
+
+### 1. Evaluate Clojure Code
+
+> Evaluation automatically connects to the given port
+
+Use the `-p` flag to specify the port and pass your Clojure code.
+
+**Recommended: Pass code as a command-line argument:**
+```bash
+clj-nrepl-eval -p <PORT> "(+ 1 2 3)"
+```
+
+**For multiple expressions (single line):**
+```bash
+clj-nrepl-eval -p <PORT> "(def x 10) (+ x 20)"
+```
+
+**Alternative: Using heredoc (may require permission approval for multiline commands):**
+```bash
+clj-nrepl-eval -p <PORT> <<'EOF'
+(def x 10)
+(+ x 20)
+EOF
+```
+
+**Alternative: Via stdin pipe:**
+```bash
+echo "(+ 1 2 3)" | clj-nrepl-eval -p <PORT>
+```
+
+### 2. Display nREPL Sessions
+
+**Discover all nREPL servers in current directory:**
+```bash
+clj-nrepl-eval --discover-ports
+```
+Shows all running nREPL servers in the current project directory, including their type (clj/bb/basilisp) and whether they match the current working directory.
+
+**Check previously connected sessions:**
+```bash
+clj-nrepl-eval --connected-ports
+```
+Shows only connections you have made before (appears after first evaluation on a port).
+
+### 3. Common Patterns
+
+**Require a namespace (always use :reload to pick up changes):**
+```bash
+clj-nrepl-eval -p <PORT> "(require '[my.namespace :as ns] :reload)"
+```
+
+**Test a function after requiring:**
+```bash
+clj-nrepl-eval -p <PORT> "(ns/my-function arg1 arg2)"
+```
+
+**Check if a file compiles:**
+```bash
+clj-nrepl-eval -p <PORT> "(require 'my.namespace :reload)"
+```
+
+**Multiple expressions:**
+```bash
+clj-nrepl-eval -p <PORT> "(def x 10) (* x 2) (+ x 5)"
+```
+
+**Complex multiline code (using heredoc):**
+```bash
+clj-nrepl-eval -p <PORT> <<'EOF'
+(def x 10)
+(* x 2)
+(+ x 5)
+EOF
+```
+*Note: Heredoc syntax may require permission approval.*
+
+**With custom timeout (in milliseconds):**
+```bash
+clj-nrepl-eval -p <PORT> --timeout 5000 "(long-running-fn)"
+```
+
+**Reset the session (clears all state):**
+```bash
+clj-nrepl-eval -p <PORT> --reset-session
+clj-nrepl-eval -p <PORT> --reset-session "(def x 1)"
+```
+
+## Available Options
+
+- `-p, --port PORT` - nREPL port (required)
+- `-H, --host HOST` - nREPL host (default: 127.0.0.1)
+- `-t, --timeout MILLISECONDS` - Timeout (default: 120000 = 2 minutes)
+- `-r, --reset-session` - Reset the persistent nREPL session
+- `-c, --connected-ports` - List previously connected nREPL sessions
+- `-d, --discover-ports` - Discover nREPL servers in current directory
+- `-h, --help` - Show help message
+
+## Important Notes
+
+- **Prefer command-line arguments:** Pass code as quoted strings: `clj-nrepl-eval -p <PORT> "(+ 1 2 3)"` - works with existing permissions
+- **Heredoc for complex code:** Use heredoc (`<<'EOF' ... EOF`) for truly multiline code, but note it may require permission approval
+- **Sessions persist:** State (vars, namespaces, loaded libraries) persists across invocations until the nREPL server restarts or `--reset-session` is used
+- **Automatic delimiter repair:** The tool automatically repairs missing or mismatched parentheses
+- **Always use :reload:** When requiring namespaces, use `:reload` to pick up recent changes
+- **Default timeout:** 2 minutes (120000ms) - increase for long-running operations
+- **Input precedence:** Command-line arguments take precedence over stdin
+
+## Typical Workflow
+
+1. Discover nREPL servers: `clj-nrepl-eval --discover-ports`
+2. Use **AskUserQuestion** tool to prompt user to select a port
+3. Require namespace:
+   ```bash
+   clj-nrepl-eval -p <PORT> "(require '[my.ns :as ns] :reload)"
+   ```
+4. Test function:
+   ```bash
+   clj-nrepl-eval -p <PORT> "(ns/my-fn ...)"
+   ```
+5. Iterate: Make changes, re-require with `:reload`, test again

.clj-kondo/config/modules/config.edn

@@ -2117,6 +2117,7 @@
            warehouses-rest
            enterprise/dependencies
            enterprise/transforms
+           enterprise/transforms-base
            enterprise/transforms-python}}
 
   enterprise/permission-debug
@@ -2338,9 +2339,24 @@
            revisions
            search
            settings
-           sync
            task
            task-history
+           util
+           enterprise/transforms-base}}
+
+  enterprise/transforms-base
+  {:team "Querying"
+   :api  #{metabase-enterprise.transforms-base.core
+           metabase-enterprise.transforms-base.interface
+           metabase-enterprise.transforms-base.ordering
+           metabase-enterprise.transforms-base.util}
+   :uses #{driver
+           events
+           lib
+           lib-be
+           premium-features
+           query-processor
+           sync
            util}}
 
   enterprise/transforms-python
@@ -2357,7 +2373,8 @@
            query-processor
            settings
            util
-           enterprise/transforms}}
+           enterprise/transforms
+           enterprise/transforms-base}}
 
   enterprise/upload-management
   {:team "UX West"

enterprise/backend/src/metabase_enterprise/metabot_v3/context.clj

@@ -3,7 +3,7 @@
    [clojure.java.io :as io]
    [medley.core :as m]
    [metabase-enterprise.metabot-v3.table-utils :as table-utils]
-   [metabase-enterprise.transforms.util :as transforms.util]
+   [metabase-enterprise.transforms-base.util :as transforms-base.util]
    [metabase.activity-feed.core :as activity-feed]
    [metabase.api.common :as api]
    [metabase.config.core :as config]
@@ -146,9 +146,9 @@
                   (try
                     (if (and (= (:type item) "transform")
                              (not (:source_type item)))
-                      (let [transform (transforms.util/normalize-transform item)]
+                      (let [transform (transforms-base.util/normalize-transform item)]
                         (assoc transform
-                               :source_type (transforms.util/transform-source-type (:source transform))))
+                               :source_type (transforms-base.util/transform-source-type (:source transform))))
                       item)
                     (catch Exception e
                       (log/error e "Error annotating transform source type for metabot context")

enterprise/backend/src/metabase_enterprise/transforms/api.clj

@@ -1,17 +1,17 @@
 (ns metabase-enterprise.transforms.api
   (:require
    [macaw.core :as macaw]
+   [metabase-enterprise.transforms-base.interface :as transforms-base.i]
+   [metabase-enterprise.transforms-base.ordering :as transforms-base.ordering]
+   [metabase-enterprise.transforms-base.util :as transforms-base.util]
    [metabase-enterprise.transforms.api.transform-job]
    [metabase-enterprise.transforms.api.transform-tag]
    [metabase-enterprise.transforms.canceling :as transforms.canceling]
    [metabase-enterprise.transforms.execute :as transforms.execute]
-   [metabase-enterprise.transforms.interface :as transforms.i]
    [metabase-enterprise.transforms.models.transform :as transform.model]
    [metabase-enterprise.transforms.models.transform-run :as transform-run]
    [metabase-enterprise.transforms.models.transform-run-cancelation :as transform-run-cancelation]
-   [metabase-enterprise.transforms.ordering :as transforms.ordering]
    [metabase-enterprise.transforms.schema :as transforms.schema]
-   [metabase-enterprise.transforms.util :as transforms.util]
    [metabase.api.common :as api]
    [metabase.api.macros :as api.macros]
    [metabase.api.routes.common :refer [+auth]]
@@ -54,7 +54,7 @@
   We now supported table-ref as source but since FE is still expecting table-id we need to temporarily do this.
   Should update FE to fully use table-ref"
   [transform]
-  (if (transforms.util/python-transform? transform)
+  (if (transforms-base.util/python-transform? transform)
     (update-in transform [:source :source-tables]
                (fn [source-tables]
                  (update-vals source-tables #(if (int? %) % (:table_id %)))))
@@ -75,21 +75,21 @@
 
 (defn- check-database-feature
   [transform]
-  (let [database (api/check-400 (t2/select-one :model/Database (transforms.i/target-db-id transform))
+  (let [database (api/check-400 (t2/select-one :model/Database (transforms-base.i/target-db-id transform))
                                 (deferred-tru "The target database cannot be found."))
-        features (transforms.util/required-database-features transform)]
+        features (transforms-base.util/required-database-features transform)]
     (api/check-400 (not (:is_sample database))
                    (deferred-tru "Cannot run transforms on the sample database."))
     (api/check-400 (not (:is_audit database))
                    (deferred-tru "Cannot run transforms on audit databases."))
     (api/check-400 (every? (fn [feature] (driver.u/supports? (:engine database) feature database)) features)
                    (deferred-tru "The database does not support the requested transform features."))
-    (api/check-400 (not (transforms.util/db-routing-enabled? database))
+    (api/check-400 (not (transforms-base.util/db-routing-enabled? database))
                    (deferred-tru "Transforms are not supported on databases with DB routing enabled."))))
 
 (defn- check-feature-enabled!
   [transform]
-  (api/check (transforms.util/check-feature-enabled transform)
+  (api/check (transforms-base.util/check-feature-enabled transform)
              [402 (deferred-tru "Premium features required for this transform type are not enabled.")]))
 
 (defn get-transforms
@@ -98,10 +98,10 @@
   (api/check-superuser)
   (let [transforms (t2/select :model/Transform {:order-by [[:id :asc]]})]
     (into []
-          (comp (transforms.util/->date-field-filter-xf [:last_run :start_time] last_run_start_time)
-                (transforms.util/->status-filter-xf [:last_run :status] last_run_statuses)
-                (transforms.util/->tag-filter-xf [:tag_ids] tag_ids)
-                (map #(update % :last_run transforms.util/localize-run-timestamps))
+          (comp (transforms-base.util/->date-field-filter-xf [:last_run :start_time] last_run_start_time)
+                (transforms-base.util/->status-filter-xf [:last_run :status] last_run_statuses)
+                (transforms-base.util/->tag-filter-xf [:tag_ids] tag_ids)
+                (map #(update % :last_run transforms-base.util/localize-run-timestamps))
                 (map python-source-table-ref->table-id))
           (t2/hydrate transforms :last_run :transform_tag_ids :creator :owner))))
 
@@ -223,7 +223,7 @@
   (check-feature-enabled! body)
   (validate-incremental-column-type! body)
 
-  (api/check (not (transforms.util/target-table-exists? body))
+  (api/check (not (transforms-base.util/target-table-exists? body))
              403
              (deferred-tru "A table with that name already exists."))
   (let [transform (t2/with-transaction [_]
@@ -248,10 +248,10 @@
   "Get a specific transform."
   [id]
   (let [{:keys [target] :as transform} (api/read-check :model/Transform id)
-        target-table (transforms.util/target-table (transforms.i/target-db-id transform) target :active true)]
+        target-table (transforms-base.util/target-table (transforms-base.i/target-db-id transform) target :active true)]
     (-> transform
         (t2/hydrate :last_run :transform_tag_ids :creator :owner)
-        (u/update-some :last_run transforms.util/localize-run-timestamps)
+        (u/update-some :last_run transforms-base.util/localize-run-timestamps)
         (assoc :table target-table)
         python-source-table-ref->table-id)))
 
@@ -275,7 +275,8 @@
                     [:id ms/PositiveInt]]]
   (api/read-check :model/Transform id)
   (let [id->transform (t2/select-pk->fn identity :model/Transform)
-        global-ordering (transforms.ordering/transform-ordering (vals id->transform))
+        _ (api/check-404 (get id->transform id))
+        global-ordering (transforms-base.ordering/transform-ordering (vals id->transform))
         dep-ids (get global-ordering id)
         dependencies (map id->transform dep-ids)]
     (mapv python-source-table-ref->table-id (t2/hydrate dependencies :creator :owner))))
@@ -305,7 +306,7 @@
   (-> (transform-run/paged-runs (assoc query-params
                                        :offset (request/offset)
                                        :limit  (request/limit)))
-      (update :data #(map transforms.util/localize-run-timestamps %))))
+      (update :data #(map transforms-base.util/localize-run-timestamps %))))
 
 ;; TODO (Cam 2025-11-25) please add a response schema to this API endpoint, it makes it easier for our customers to
 ;; use our API + we will need it when we make auto-TypeScript-signature generation happen
@@ -336,12 +337,12 @@
                       (check-feature-enabled! new)
                       (check-database-feature new)
                       (validate-incremental-column-type! new)
-                      (when (transforms.util/query-transform? old)
-                        (when-let [{:keys [cycle-str]} (transforms.ordering/get-transform-cycle new)]
+                      (when (transforms-base.util/query-transform? old)
+                        (when-let [{:keys [cycle-str]} (transforms-base.ordering/get-transform-cycle new)]
                           (throw (ex-info (str "Cyclic transform definitions detected: " cycle-str)
                                           {:status-code 400}))))
                       (api/check (not (and (not= (target-fields old) (target-fields new))
-                                           (transforms.util/target-table-exists? new)))
+                                           (transforms-base.util/target-table-exists? new)))
                                  403
                                  (deferred-tru "A table with that name already exists.")))
                     (t2/update! :model/Transform id (dissoc body :tag_ids))
@@ -376,7 +377,7 @@
   [{:keys [id]} :- [:map
                     [:id ms/PositiveInt]]]
   (api/write-check :model/Transform id)
-  (transforms.util/delete-target-table-by-id! id)
+  (transforms-base.util/delete-target-table-by-id! id)
   nil)
 
 ;; TODO (Cam 2025-11-25) please add a response schema to this API endpoint, it makes it easier for our customers to
@@ -390,7 +391,7 @@
   (let [transform (api/write-check :model/Transform id)
         run (api/check-404 (transform-run/running-run-for-transform-id id))]
     (transform-run-cancelation/mark-cancel-started-run! (:id run))
-    (when (transforms.util/python-transform? transform)
+    (when (transforms-base.util/python-transform? transform)
       (transforms.canceling/cancel-run! (:id run))))
   nil)
 

enterprise/backend/src/metabase_enterprise/transforms/api/transform_job.clj

@@ -1,10 +1,10 @@
 (ns metabase-enterprise.transforms.api.transform-job
   (:require
    [medley.core :as m]
+   [metabase-enterprise.transforms-base.util :as transforms-base.util]
    [metabase-enterprise.transforms.jobs :as transforms.jobs]
    [metabase-enterprise.transforms.models.transform-job :as transform-job]
    [metabase-enterprise.transforms.schedule :as transforms.schedule]
-   [metabase-enterprise.transforms.util :as transforms.util]
    [metabase.api.common :as api]
    [metabase.api.macros :as api.macros]
    [metabase.api.routes.common :refer [+auth]]
@@ -159,7 +159,7 @@
 (defn- add-next-run
   [{id :id :as job}]
   (if-let [start-time (-> id transforms.schedule/existing-trigger :next-fire-time)]
-    (assoc job :next_run {:start_time (str (transforms.util/->instant start-time))})
+    (assoc job :next_run {:start_time (str (transforms-base.util/->instant start-time))})
     job))
 
 ;; TODO (Cam 2025-11-25) please add a response schema to this API endpoint, it makes it easier for our customers to
@@ -198,12 +198,12 @@
   (let [jobs (t2/select :model/TransformJob {:order-by [[:created_at :desc]]})]
     (into []
           (comp (map add-next-run)
-                (transforms.util/->date-field-filter-xf [:last_run :start_time] last_run_start_time)
-                (transforms.util/->date-field-filter-xf [:next_run :start_time] next_run_start_time)
-                (transforms.util/->status-filter-xf [:last_run :status] last_run_statuses)
-                (transforms.util/->tag-filter-xf [:tag_ids] tag_ids)
-                (map #(update % :last_run transforms.util/localize-run-timestamps))
-                (map #(update % :next_run transforms.util/localize-run-timestamps)))
+                (transforms-base.util/->date-field-filter-xf [:last_run :start_time] last_run_start_time)
+                (transforms-base.util/->date-field-filter-xf [:next_run :start_time] next_run_start_time)
+                (transforms-base.util/->status-filter-xf [:last_run :status] last_run_statuses)
+                (transforms-base.util/->tag-filter-xf [:tag_ids] tag_ids)
+                (map #(update % :last_run transforms-base.util/localize-run-timestamps))
+                (map #(update % :next_run transforms-base.util/localize-run-timestamps)))
           (t2/hydrate jobs :tag_ids :last_run))))
 
 (def ^{:arglists '([request respond raise])} routes

enterprise/backend/src/metabase_enterprise/transforms/core.clj

@@ -2,9 +2,9 @@
   "API namespace for the `metabase-enterprise.transform` module."
   (:require
    [java-time.api :as t]
+   [metabase-enterprise.transforms-base.util]
    [metabase-enterprise.transforms.models.transform-run]
    [metabase-enterprise.transforms.settings]
-   [metabase-enterprise.transforms.util]
    [metabase.premium-features.core :refer [defenterprise]]
    [potemkin :as p]
    [toucan2.core :as t2]))
@@ -12,7 +12,7 @@
 (p/import-vars
  [metabase-enterprise.transforms.settings
   transform-timeout]
- [metabase-enterprise.transforms.util
+ [metabase-enterprise.transforms-base.util
   native-query-transform?
   python-transform?
   query-transform?]

enterprise/backend/src/metabase_enterprise/transforms/interface.clj

@@ -1,32 +1,21 @@
-(ns metabase-enterprise.transforms.interface)
+(ns metabase-enterprise.transforms.interface
+  "Interface for scheduled transform execution.
 
-(defn- transform->transform-type
-  [transform]
-  (-> transform :source :type keyword))
+   For core interface methods (source-db-id, target-db-id, table-dependencies),
+   use metabase-enterprise.transforms-base.interface directly.
 
-(defmulti source-db-id
-  "Return the ID of the source database for a given `transform`. The source database is where the data originates from
-  before being transformed and written to the target destination.
-
-  NOTE: Currently returns a single database ID, but we may want to change this to return multiple database IDs in the
-  future since transforms can have multiple input sources."
-  {:added "0.57.0" :arglists '([transform])}
-  transform->transform-type)
-
-(defmulti target-db-id
-  "Return the ID of the target database for a given `transform`. The target database is the destination where the
-  transformed data will be written. Often this is the same database as the source database."
-  {:added "0.57.0" :arglists '([transform])}
-  transform->transform-type)
+   This namespace defines the scheduled-execution-specific `execute!` multimethod
+   which handles transform_run tracking."
+  (:require
+   [metabase-enterprise.transforms-base.interface :as transforms-base.i]))
 
+;; Scheduled execution multimethod - dispatches on transform type
+;; This is separate from transforms-base.i/execute-base! which does NOT track transform_run
 (defmulti execute!
-  "Execute a transform operation. Runs the actual transformation process, which might involve running SQL queries,
-  Python scripts, or other transformation logic depending on the transform type.
+  "Execute a transform operation with transform_run tracking.
 
-  This method blocks and may take significant time depending on the data volume. Implementations
-  should handle errors gracefully and provide appropriate logging.
-
-  Returns nil (or a result that can be discarded).
+  This method creates transform_run rows, tracks status, and handles cancellation.
+  For base execution without database tracking, use transforms-base.i/execute-base! instead.
 
   Options:
   - `:start-promise`
@@ -45,24 +34,4 @@
   Do not use this directly. Use [[metabase-enterprise.transforms.execute/execute!]] instead."
   {:added "0.57.0" :arglists '([transform options])}
   (fn [transform _options]
-    (transform->transform-type transform)))
-
-(defmulti table-dependencies
-  "Return a set of logical table dependencies of the transform, including indirect dependencies via cards.
-  The transform execution system uses these dependencies to determine the correct order of execution
-  and to detect circular dependencies.
-
-  Each dependency is represented as one of:
-  - `{:table <table-id>}`
-     A dependency on a table that exists and has been synced.
-  - `{:transform <transform-id>}`
-     A dependency on a table that does not yet exist, but is known to be the target of another transform.
-     Represents a 'placeholder' table (as we no table id / metadata) for the same purposes.
-  - `{:table-ref {:database_id <db-id> :schema <schema> :table <name>}}`
-     A dependency on a table by name, for cases where the table_id may not exist yet (e.g., when
-     bulk importing transforms that depend on each other's outputs). The ordering system will
-     resolve these to transforms by matching against target table definitions.
-
-  An empty set indicates no dependencies."
-  {:added "0.57.0" :arglists '([transform])}
-  transform->transform-type)
+    (transforms-base.i/transform->transform-type transform)))

enterprise/backend/src/metabase_enterprise/transforms/jobs.clj

@@ -5,13 +5,13 @@
    [clojurewerkz.quartzite.schedule.calendar-interval :as calendar-interval]
    [clojurewerkz.quartzite.triggers :as triggers]
    [flatland.ordered.set :as ordered-set]
+   [metabase-enterprise.transforms-base.ordering :as transforms-base.ordering]
+   [metabase-enterprise.transforms-base.util :as transforms-base.util]
    [metabase-enterprise.transforms.execute :as transforms.execute]
    [metabase-enterprise.transforms.instrumentation :as transforms.instrumentation]
    [metabase-enterprise.transforms.models.job-run :as transforms.job-run]
    [metabase-enterprise.transforms.models.transform-run :as transform-run]
-   [metabase-enterprise.transforms.ordering :as transforms.ordering]
    [metabase-enterprise.transforms.settings :as transforms.settings]
-   [metabase-enterprise.transforms.util :as transforms.util]
    [metabase.channel.urls :as urls]
    [metabase.events.core :as events]
    [metabase.revisions.core :as revisions]
@@ -34,7 +34,7 @@
       found)))
 
 (defn- next-transform [ordering transforms-by-id complete]
-  (-> (transforms.ordering/available-transforms ordering #{} complete)
+  (-> (transforms-base.ordering/available-transforms ordering #{} complete)
       first
       transforms-by-id))
 
@@ -52,7 +52,7 @@
 
 (defn- get-plan [transform-ids]
   (let [all-transforms   (t2/select :model/Transform)
-        global-ordering  (transforms.ordering/transform-ordering all-transforms)
+        global-ordering  (transforms-base.ordering/transform-ordering all-transforms)
         relevant-ids     (get-deps global-ordering transform-ids)
         transforms-by-id (into {}
                                (keep (fn [{:keys [id] :as transform}]
@@ -60,7 +60,7 @@
                                          [id transform])))
                                all-transforms)
         ordering         (sorted-ordering (select-keys global-ordering relevant-ids) transforms-by-id)]
-    (when-let [cycle (transforms.ordering/find-cycle ordering)]
+    (when-let [cycle (transforms-base.ordering/find-cycle ordering)]
       (let [id->name (into {} (map (juxt :id :name)) all-transforms)]
         (throw (ex-info (str "Cyclic transform definitions detected: "
                              (str/join " → " (map id->name cycle)))
@@ -72,7 +72,7 @@
          :deps global-ordering}))))
 
 (defn- run-transform! [run-id run-method user-id {transform-id :id :as transform}]
-  (if-not (transforms.util/check-feature-enabled transform)
+  (if-not (transforms-base.util/check-feature-enabled transform)
     (log/warnf "Skip running transform %d due to lacking premium features" transform-id)
     (do
       (when (transform-run/running-run-for-transform-id transform-id)

enterprise/backend/src/metabase_enterprise/transforms/models/transform.clj

@@ -2,9 +2,9 @@
   (:require
    [clojure.set :as set]
    [medley.core :as m]
-   [metabase-enterprise.transforms.interface :as transforms.i]
+   [metabase-enterprise.transforms-base.interface :as transforms-base.i]
+   [metabase-enterprise.transforms-base.util :as transforms-base.util]
    [metabase-enterprise.transforms.models.transform-run :as transform-run]
-   [metabase-enterprise.transforms.util :as transforms.util]
    [metabase.collections.models.collection :as collection]
    [metabase.events.core :as events]
    [metabase.lib-be.core :as lib-be]
@@ -61,7 +61,7 @@
 
 (defn- transform-source-in [m]
   (-> m
-      (m/update-existing :source-tables transforms.util/normalize-source-tables)
+      (m/update-existing :source-tables transforms-base.util/normalize-source-tables)
       (m/update-existing :query (comp lib/prepare-for-serialization lib-be/normalize-query))
       mi/json-in))
 
@@ -80,7 +80,7 @@
   (collection/check-collection-namespace :model/Transform collection_id)
   (when collection_id
     (collection/check-allowed-content :model/Transform collection_id))
-  (assoc transform :source_type (transforms.util/transform-source-type source)))
+  (assoc transform :source_type (transforms-base.util/transform-source-type source)))
 
 (t2/define-before-update :model/Transform
   [{:keys [source] :as transform}]
@@ -88,13 +88,13 @@
     (collection/check-collection-namespace :model/Transform new-collection)
     (collection/check-allowed-content :model/Transform new-collection))
   (if source
-    (assoc transform :source_type (transforms.util/transform-source-type source))
+    (assoc transform :source_type (transforms-base.util/transform-source-type source))
     transform))
 
 (t2/define-after-select :model/Transform
   [{:keys [source] :as transform}]
   (if source
-    (assoc transform :source_type (transforms.util/transform-source-type source))
+    (assoc transform :source_type (transforms-base.util/transform-source-type source))
     transform))
 
 (methodical/defmethod t2/batched-hydrate [:model/TransformRun :transform]
@@ -246,7 +246,7 @@
   "Fetch tables with their fields. The tables show up under the `:table` property."
   [transforms]
   (let [table-key-fn (fn [{:keys [target] :as transform}]
-                       [(transforms.i/target-db-id transform) (:schema target) (:name target)])
+                       [(transforms-base.i/target-db-id transform) (:schema target) (:name target)])
         table-keys (into #{} (map table-key-fn) transforms)
         table-keys-with-schema (filter second table-keys)
         table-keys-without-schema (keep (fn [[db-id schema table-name]]

enterprise/backend/src/metabase_enterprise/transforms/query_impl.clj

@@ -1,96 +1,44 @@
 (ns metabase-enterprise.transforms.query-impl
+  "Query transform implementation for scheduled execution.
+
+   This namespace provides the scheduled wrapper that creates transform_run rows
+   and tracks status. The actual execution logic is in transforms-base.query."
   (:require
-   [metabase-enterprise.transforms.instrumentation :as transforms.instrumentation]
    [metabase-enterprise.transforms.interface :as transforms.i]
    [metabase-enterprise.transforms.util :as transforms.util]
-   [metabase.driver :as driver]
-   [metabase.driver.util :as driver.u]
-   [metabase.events.core :as events]
-   [metabase.lib.schema.common :as schema.common]
-   [metabase.query-processor.compile :as qp.compile]
-   [metabase.util.log :as log]
-   [metabase.util.malli.registry :as mr]
-   [toucan2.core :as t2]))
+   [metabase.util.log :as log]))
 
 (set! *warn-on-reflection* true)
 
-(defmethod transforms.i/source-db-id :query
-  [transform]
-  (-> transform :source :query :database))
-
-(defmethod transforms.i/target-db-id :query
-  [transform]
-  (-> transform :source :query :database))
-
-(mr/def ::transform-details
-  [:map
-   [:transform-type [:enum {:decode/normalize schema.common/normalize-keyword} :table :table-incremental]]
-   [:conn-spec :any]
-   [:query ::qp.compile/compiled]
-   [:output-table [:keyword {:decode/normalize schema.common/normalize-keyword}]]])
-
-(mr/def ::transform-opts
-  [:map
-   [:overwrite? :boolean]])
-
-(defn- transform-opts [{:keys [transform-type]}]
-  (case transform-type
-    :table {:overwrite? true}
-
-    ;; once we have more than just append, dispatch on :target-incremental-strategy
-    :table-incremental {}))
+;;; ------------------------------------------------- Scheduled Execution Wrapper -------------------------------------------------
 
 (defn- run-mbql-transform!
+  "Execute a query transform with transform_run tracking.
+
+   This wrapper:
+   1. Creates a transform_run row via try-start-unless-already-running
+   2. Calls the base execution via run-cancelable-transform!
+   3. Updates transform_run status on completion/failure"
   ([transform] (run-mbql-transform! transform nil))
-  ([{:keys [id source target owner_user_id creator_id] :as transform} {:keys [run-method start-promise user-id]}]
+  ([{:keys [id owner_user_id creator_id] :as transform} {:keys [run-method start-promise user-id]}]
    (try
-     (let [db (get-in source [:query :database])
-           {driver :engine :as database} (t2/select-one :model/Database db)
-           transform-details {:db-id db
-                              :database database
-                              :transform-id   id
-                              :transform-type (keyword (:type target))
-                              :conn-spec (driver/connection-spec driver database)
-                              :query (transforms.util/compile-source transform)
-                              :output-schema (:schema target)
-                              :output-table (transforms.util/qualified-table-name driver target)}
-           opts (transform-opts transform-details)
-           features (transforms.util/required-database-features transform)
-           ;; For manual runs, use the triggering user; for cron, use owner/creator
+     (let [;; For manual runs, use the triggering user; for cron, use owner/creator
            run-user-id (if (and (= run-method :manual) user-id)
                          user-id
-                         (or owner_user_id creator_id))]
-
-       (when (transforms.util/db-routing-enabled? database)
-         (throw (ex-info "Transforms are not supported on databases with DB routing enabled."
-                         {:driver driver, :database database})))
-       (when-not (every? (fn [feature] (driver.u/supports? (:engine database) feature database)) features)
-         (throw (ex-info "The database does not support the requested transform target type."
-                         {:driver driver, :database database, :features features})))
-       ;; mark the execution as started and notify any observers
-       (let [{run-id :id} (transforms.util/try-start-unless-already-running id run-method run-user-id)]
-         (when start-promise
-           (deliver start-promise [:started run-id]))
-         (log/info "Executing transform" id "with target" (pr-str target))
-         (transforms.instrumentation/with-stage-timing [run-id [:computation :mbql-query]]
-           (transforms.util/run-cancelable-transform!
-            run-id driver transform-details
-            (fn [_cancel-chan] (driver/run-transform! driver transform-details opts))))
-         (transforms.instrumentation/with-stage-timing [run-id [:import :table-sync]]
-           (transforms.util/sync-target! target database)
-           ;; This event must be published only after the sync is complete - the new table needs to be in AppDB.
-           (events/publish-event! :event/transform-run-complete {:object transform-details}))
-         ;; Creating an index after sync means the filter column is known in the appdb.
-         ;; The index would be synced the next time sync runs, but at time of writing, index sync is disabled.
-         (transforms.util/execute-secondary-index-ddl-if-required! transform run-id database target)))
+                         (or owner_user_id creator_id))
+           {run-id :id} (transforms.util/try-start-unless-already-running id run-method run-user-id)]
+       (when start-promise
+         (deliver start-promise [:started run-id]))
+       (transforms.util/run-cancelable-transform! run-id transform {}))
      (catch Throwable t
        (log/error t "Error executing transform")
        (when start-promise
-         ;; if the start-promise has been delivered, this is a no-op,
-         ;; but we assume nobody would catch the exception anyway
+         ;; if the start-promise has been delivered, this is a no-op
          (deliver start-promise t))
        (throw t)))))
 
+;;; ------------------------------------------------- Interface Implementation -------------------------------------------------
+
 #_{:clj-kondo/ignore [:discouraged-var]}
 (defmethod transforms.i/execute! :query [transform opts]
   (run-mbql-transform! transform opts))

enterprise/backend/src/metabase_enterprise/transforms/util.clj

@@ -1,102 +1,20 @@
 (ns metabase-enterprise.transforms.util
+  "Scheduled execution utilities for transforms.
+
+   These functions require transform_run database access and are NOT in the base module.
+   For base utilities, use metabase-enterprise.transforms-base.util directly."
   (:require
-   [buddy.core.codecs :as codecs]
-   [buddy.core.hash :as buddy-hash]
    [clojure.core.async :as a]
-   [clojure.string :as str]
-   [java-time.api :as t]
+   [metabase-enterprise.transforms-base.core :as transforms-base]
+   [metabase-enterprise.transforms-base.util :as transforms-base.util]
    [metabase-enterprise.transforms.canceling :as canceling]
    [metabase-enterprise.transforms.instrumentation :as transforms.instrumentation]
-   [metabase-enterprise.transforms.interface :as transforms.i]
    [metabase-enterprise.transforms.models.transform-run :as transform-run]
    [metabase-enterprise.transforms.settings :as transforms.settings]
-   [metabase.driver :as driver]
-   [metabase.driver.sql.query-processor :as sql.qp]
-   [metabase.driver.util :as driver.u]
-   [metabase.lib-be.core :as lib-be]
-   [metabase.lib.core :as lib]
-   [metabase.lib.metadata :as lib.metadata]
-   [metabase.lib.query :as lib.query]
-   [metabase.lib.schema.common :as lib.schema.common]
-   [metabase.premium-features.core :as premium-features :refer [defenterprise]]
-   [metabase.query-processor :as qp]
-   [metabase.query-processor.compile :as qp.compile]
-   [metabase.query-processor.parameters.dates :as params.dates]
-   [metabase.query-processor.pipeline :as qp.pipeline]
-   [metabase.sync.core :as sync]
-   [metabase.util :as u]
-   [metabase.util.date-2 :as u.date]
-   [metabase.util.honey-sql-2 :as h2x]
-   [metabase.util.log :as log]
-   [metabase.util.malli :as mu]
-   [metabase.util.malli.registry :as mr]
-   [toucan2.core :as t2])
-  (:import
-   (java.time Instant LocalDate LocalDateTime LocalTime OffsetDateTime OffsetTime ZonedDateTime)
-   (java.util Date)))
+   [metabase.query-processor.pipeline :as qp.pipeline]))
 
 (set! *warn-on-reflection* true)
 
-(def ^:const transform-temp-table-prefix
-  "Prefix used for temporary tables created during transform execution."
-  "mb_transform_temp_table")
-
-(defn qualified-table-name
-  "Return the name of the target table of a transform as a possibly qualified symbol."
-  [_driver {:keys [schema name]}]
-  (if schema
-    (keyword schema name)
-    (keyword name)))
-
-(defn query-transform?
-  "Check if this is a query transform: native query / mbql query."
-  [transform]
-  (= :query (-> transform :source :type keyword)))
-
-(defn native-query-transform?
-  "Check if this is a native query transform.
-  Note: The transform should be normalized (via `normalize-transform`) before calling this function."
-  [transform]
-  (when (query-transform? transform)
-    (let [query (-> transform :source :query)]
-      (lib/native-only-query? query))))
-
-(defn python-transform?
-  "Check if this is a Python transform."
-  [transform]
-  (= :python (-> transform :source :type keyword)))
-
-(defn normalize-transform
-  "Normalize a transform's source query, similar to how transforms are normalized when read from the database.
-  This should be called on transforms before processing them to ensure queries are in the expected format."
-  [transform]
-  (if (and (map? transform)
-           (= (:type transform) "transform")
-           (get-in transform [:source :query]))
-    (update-in transform [:source :query] lib-be/normalize-query)
-    transform))
-
-(defn transform-source-type
-  "Returns the type of a transform's source: :python, :native, or :mbql.
-  Throws if the source type cannot be detected.
-  Note: The transform should be normalized (via `normalize-transform`) before calling this function."
-  [source]
-  (case (keyword (:type source))
-    :python :python
-    :query  (if (lib/native-only-query? (:query source))
-              :native
-              :mbql)
-    (throw (ex-info (str "Unknown transform source type: " (:type source))
-                    {:source source}))))
-
-(defn check-feature-enabled
-  "Checking whether we have proper feature flags for using a given transform."
-  [transform]
-  (if (python-transform? transform)
-    (and (premium-features/has-feature? :transforms)
-         (premium-features/has-feature? :transforms-python))
-    (premium-features/has-feature? :transforms)))
-
 (defn try-start-unless-already-running
   "Start a transform run, throwing an informative error if already running.
    If `user-id` is provided, it will be stored with the run for attribution purposes."
@@ -113,553 +31,65 @@
         (throw e)))))
 
 (defn run-cancelable-transform!
-  "Execute a transform with cancellation support and proper error handling.
+  "Execute a transform with cancellation support, status tracking, and proper error handling.
+
+  This is the wrapper for scheduled execution that:
+  1. Sets up cancellation channel and timeout
+  2. Calls the base execution via `cancelled?` callback
+  3. Updates transform_run status on completion/failure
 
   Options:
-  - `:ex-message-fn` change how caught exceptions are presented to the user in run logs, by default the same as clojure.core/ex-message"
-  [run-id driver {:keys [db-id conn-spec output-schema]} run-transform! & {:keys [ex-message-fn] :or {ex-message-fn ex-message}}]
-  ;; local run is responsible for status, using canceling lifecycle
+  - `:ex-message-fn` - customize how caught exceptions are presented in run logs"
+  [run-id transform {:keys [ex-message-fn] :or {ex-message-fn ex-message}}]
   (try
-    (when-not (driver/schema-exists? driver db-id output-schema)
-      (driver/create-schema-if-needed! driver conn-spec output-schema))
     (canceling/chan-start-timeout-vthread! run-id (transforms.settings/transform-timeout))
     (let [cancel-chan (a/promise-chan)
-          ret (binding [qp.pipeline/*canceled-chan* cancel-chan]
-                (canceling/chan-start-run! run-id cancel-chan)
-                (run-transform! cancel-chan))]
-      (transform-run/succeed-started-run! run-id)
-      ret)
+          cancelled? (fn [] (boolean (a/poll! cancel-chan)))
+          ;; Create timing wrapper that uses instrumentation
+          with-stage-timing-fn (fn [rid stage thunk]
+                                 (transforms.instrumentation/with-stage-timing [rid stage]
+                                   (thunk)))
+          result (binding [qp.pipeline/*canceled-chan* cancel-chan]
+                   (canceling/chan-start-run! run-id cancel-chan)
+                   (transforms-base/execute! transform
+                                             {:cancelled? cancelled?
+                                              :run-id run-id
+                                              :with-stage-timing-fn with-stage-timing-fn}))]
+      (case (:status result)
+        :succeeded
+        (do
+          (transform-run/succeed-started-run! run-id)
+          result)
+
+        :cancelled
+        (do
+          (transform-run/cancel-run! run-id {:message (or (:logs result) "Canceled by user")})
+          (throw (ex-info "Transform cancelled" (assoc result :status :cancelled))))
+
+        :timeout
+        (do
+          (transform-run/timeout-run! run-id)
+          (throw (ex-info "Transform timed out" (assoc result :status :timeout))))
+
+        :failed
+        ;; if we have it, throw an error using original message so it's not as generic
+        (throw (ex-info (or (some-> (:error result) ex-message)
+                            "Transform failed")
+                        result
+                        (:error result)))))
     (catch Throwable t
-      (transform-run/fail-started-run! run-id {:message (ex-message-fn t)})
+      (transform-run/fail-started-run! run-id {:message (or (:logs (ex-data t))
+                                                            (ex-message-fn t))})
       (throw t))
     (finally
       (canceling/chan-end-run! run-id))))
 
-(declare activate-table-and-mark-computed! target-table)
-
-(defn sync-target!
-  "Sync target of a transform"
-  [target database]
-  ;; sync the new table (note that even a failed sync status means that the execution succeeded)
-  (log/info "Syncing target" (pr-str target) "for transform")
-  (activate-table-and-mark-computed! database target))
-
-(defn target-table-exists?
-  "Test if the target table of a transform already exists."
-  [{:keys [target] :as transform}]
-  (let [db-id (transforms.i/target-db-id transform)
-        {driver :engine :as database} (t2/select-one :model/Database db-id)]
-    (driver/table-exists? driver database target)))
-
-(defn target-table
-  "Load the `target` table of a transform from the database specified by `database-id`."
-  [database-id target & kv-args]
-  (some-> (apply t2/select-one :model/Table
-                 :db_id database-id
-                 :schema (:schema target)
-                 :name (:name target)
-                 kv-args)
-          (t2/hydrate :db)))
-
-(defn- sync-table!
-  ([database target] (sync-table! database target nil))
-  ([database target {:keys [create?]}]
-   (when-let [table (or (target-table (:id database) target)
-                        (when create?
-                          (sync/create-table! database (select-keys target [:schema :name :data_source :data_authority]))))]
-     (sync/sync-table! table)
-     table)))
-
-(defn activate-table-and-mark-computed!
-  "Activate table for `target` in `database` in the app db."
-  [database target]
-  (when-let [table (sync-table! database (assoc target
-                                                :data_authority :computed
-                                                :data_source :metabase-transform)
-                                {:create? true})]
-    (when-not (:active table)
-      (t2/update! :model/Table (:id table) {:active true}))
-    table))
-
-(defn deactivate-table!
-  "Deactivate table for `target` in `database` in the app db."
-  [database target]
-  (when-let [table (sync-table! database target)]
-    ;; TODO this should probably be a function in the sync module
-    (t2/update! :model/Table (:id table) {:active false})))
-
-(defn delete-target-table!
-  "Delete the target table of a transform and sync it from the app db."
-  [{:keys [id target], :as transform}]
-  (when target
-    (let [target (update target :type keyword)
-          database-id (transforms.i/target-db-id transform)
-          {driver :engine :as database} (t2/select-one :model/Database database-id)]
-      (driver/drop-transform-target! driver database target)
-      (log/info "Deactivating  target " (pr-str target) "for transform" id)
-      (deactivate-table! database target))))
-
-(defn delete-target-table-by-id!
-  "Delete the target table of the transform specified by `transform-id`."
-  [transform-id]
-  (delete-target-table! (t2/select-one :model/Transform transform-id)))
-
-(defn massage-sql-query
-  "Adjusts mbql query for use in a transform."
-  [query]
-  (assoc-in query [:middleware :disable-remaps?] true))
-
-(defn- checkpoint-incremental?
-  "Returns true if `source` uses checkpoint-based incremental strategy."
-  [source]
-  (= :checkpoint (some-> source :source-incremental-strategy :type keyword)))
-
-(defn supported-incremental-filter-type?
-  "Returns true if the given base-type is supported for incremental filtering.
-
-  We only support temporal (timestamp/tz) and numeric (int/float) types."
-  [base-type]
-  (or (isa? base-type :type/Temporal)
-      (isa? base-type :type/Number)))
-
-(defn- source->checkpoint-filter-unique-key
-  "Extract the checkpoint filter column from `query` using the unique key specified in `source-incremental-strategy`."
-  [query source-incremental-strategy]
-  (some->> source-incremental-strategy :checkpoint-filter-unique-key (lib/column-with-unique-key query)))
-
-(defn- source->checkpoint-filter-column
-  "Resolve the checkpoint filter column for an incremental transform.
-
-  Tries to resolve the column using the unique key first.
-  Falls back to looking up the column by name from the target table if a `:checkpoint-filter` is specified.
-
-  Validates that the resolved column has a supported type for checkpoint filtering (numeric or temporal).
-  Throws an exception if the column type is not supported."
-  [query source-incremental-strategy table metadata-provider]
-  (let [{:keys [checkpoint-filter checkpoint-filter-unique-key]} source-incremental-strategy]
-    (when-some [{column-name :name
-                 :keys [base-type]
-                 :as column}
-                (cond
-                  checkpoint-filter-unique-key
-                  (source->checkpoint-filter-unique-key query source-incremental-strategy)
-                  checkpoint-filter
-                  (when-some [field-id (t2/select-one-pk :model/Field
-                                                         :table_id (:id table)
-                                                         :name checkpoint-filter)]
-                    (lib.metadata/field metadata-provider field-id)))]
-      (when-not (supported-incremental-filter-type? base-type)
-        (throw (ex-info (str "Checkpoint column '" column-name "' has unsupported type " (pr-str base-type) ". "
-                             "Only numeric and temporal columns are supported for incremental filtering.")
-                        {:column-name column-name
-                         :base-type   base-type})))
-      column)))
-
-(defn next-checkpoint
-  "Build a query to compute the MAX of the checkpoint column from the target table.
-
-  Returns a map with `:query` (MBQL query selecting the max) and `:filter-column` (column metadata),
-  or `nil` if the transform doesn't use checkpoint-based incremental strategy or the target table doesn't exist."
-  [{:keys [source target] :as transform}]
-  (let [db-id (transforms.i/target-db-id transform)]
-    (when (checkpoint-incremental? source)
-      (when-let [table (target-table db-id target)]
-        (let [metadata-provider (lib-be/application-database-metadata-provider db-id)
-              table-metadata (lib.metadata/table metadata-provider (:id table))
-              query (lib/query metadata-provider table-metadata)]
-          (when-let [filter-column (source->checkpoint-filter-column query
-                                                                     (:source-incremental-strategy source)
-                                                                     table metadata-provider)]
-            {:query (-> query (lib/aggregate (lib/max filter-column)))
-             :filter-column filter-column}))))))
-
-(defn- next-checkpoint-value
-  "Execute the checkpoint query and normalize the result for database insertion.
-  Returns `nil` if the target table is empty."
-  [{:keys [query filter-column]}]
-  (let [{:keys [base-type]} filter-column
-        v (some-> query qp/process-query :data :rows first first)]
-    ;; QP return values are lossy, we do a bit of parsing to ensure they're of the right
-    ;; shape for reinsertion
-    (cond
-      (nil? v)
-      nil
-
-      (isa? base-type :type/Integer)
-      (bigint v)
-
-      ;; any other number that's not an integer, should be a decimal/float
-      (number? v)
-      (bigdec v)
-
-      :else v)))
-
-(defn preprocess-incremental-query
-  "Add checkpoint checkpoint filtering to a query for incremental execution.
-
-  For native queries with a `checkpoint` template tag, adds the checkpoint as a parameter.
-  For MBQL queries, adds a filter clause `WHERE checkpoint_column > checkpoint`.
-  Returns the query unchanged on first run (no checkpoint) or for native queries without the checkpoint tag."
-  [query source-incremental-strategy checkpoint]
-  (if-let [checkpoint-value (next-checkpoint-value checkpoint)]
-    (if (lib.query/native? query)
-      ;; native query with explicit checkpoint filter
-      (if (get-in query [:stages 0 :template-tags "checkpoint"])
-        (update query :parameters conj
-                {:type (if (number? checkpoint-value) :number :text)
-                 :target [:variable [:template-tag "checkpoint"]]
-                 :value checkpoint-value})
-        query)
-      ;; mbql query
-      (lib/filter query (lib/> (source->checkpoint-filter-unique-key query source-incremental-strategy) checkpoint-value)))
-    query))
-
-(defn- post-process-incremental-query
-  "Wrap a compiled native query with checkpoint filtering for native queries without explicit checkpoint tags.
-
-  Generates SQL that wraps the original query as a subquery and filters by `checkpoint_filter > (checkpoint_query)`. "
-  [outer-query driver {:keys [source-incremental-strategy] :as source} {checkpoint-query :query :as checkpoint}]
-  (let [{:keys [checkpoint-filter]} source-incremental-strategy]
-    (if (and (lib.query/native? (:query source))
-             (not (get-in (:query source) [:stages 0 :template-tags "checkpoint"]))
-             (next-checkpoint-value checkpoint))
-      (let [wrap-query (fn [query]
-                         (let [honeysql-query {:select [:*]
-                                               :from [[[:raw (str "(" query ")")] :subquery]]
-                                               :where [:> (h2x/identifier :field checkpoint-filter)
-                                                       [:raw (str "(" (:query (qp.compile/compile checkpoint-query)) ")")]]}]
-                           (first (sql.qp/format-honeysql driver honeysql-query))))]
-        (update outer-query :query wrap-query))
-      outer-query)))
-
-(defn compile-source
-  "Compile the source query of a transform to SQL, applying incremental filtering if required."
-  [{:keys [source] :as transform}]
-  (let [{:keys [source-incremental-strategy] query-type :type} source]
-    (case (keyword query-type)
-      :query
-      (let [checkpoint (next-checkpoint transform)
-            query (:query source)
-            driver (some->> query :database (t2/select-one :model/Database) :engine keyword)]
-        (binding [driver/*compile-with-inline-parameters*
-                  (or (= :clickhouse driver)
-                      driver/*compile-with-inline-parameters*)]
-          (-> query
-              (preprocess-incremental-query source-incremental-strategy checkpoint)
-              massage-sql-query
-              qp.compile/compile
-              (post-process-incremental-query driver source checkpoint)))))))
-
-(defn required-database-features
-  "Returns the database features necessary to execute `transform`."
-  [transform]
-  (if (python-transform? transform)
-    [:transforms/python]
-    [:transforms/table]))
-
-(defn ->instant
-  "Convert a temporal value `t` to an Instant in the system timezone."
-  ^Instant [t]
-  (when t
-    (condp instance? t
-      Instant        t
-      Date           (.toInstant ^Date t)
-      OffsetDateTime (.toInstant ^OffsetDateTime t)
-      ZonedDateTime  (.toInstant ^ZonedDateTime t)
-      LocalDateTime  (recur (.atZone ^LocalDateTime t (t/zone-id)))
-      String         (recur (u.date/parse t))
-      LocalTime      (recur (.atDate ^LocalTime t (t/local-date)))
-      OffsetTime     (recur (.atDate ^OffsetTime t (t/local-date)))
-      LocalDate      (recur (.atStartOfDay ^LocalDate t))
-      (throw (ex-info (str "Cannot convert temporal " t " of type " (type t) " to an Instant")
-                      {:temporal t})))))
-
-(defn utc-timestamp-string
-  "Convert the timestamp t to a string encoding the it in the system timezone."
-  [t]
-  (-> t ->instant str))
-
-(defn localize-run-timestamps
-  "Convert the timestamps of a `run` to ISO strings in UTC."
-  [run]
-  (-> run
-      (u/update-some :start_time utc-timestamp-string)
-      (u/update-some :end_time   utc-timestamp-string)))
-
-(mr/def ::column-definition
-  [:map
-   [:name :string]
-   [:type ::lib.schema.common/base-type]
-   [:nullable? {:optional true} :boolean]])
-
-(mr/def ::table-definition
-  [:map
-   [:name :keyword]
-   [:columns [:sequential ::column-definition]]
-   [:primary-key {:optional true} [:sequential :string]]])
-
-(mu/defn create-table-from-schema!
-  "Create a table from a table-schema"
-  [driver :- :keyword
-   database-id :- pos-int?
-   table-schema :- ::table-definition]
-  (let [{:keys [columns] table-name :name} table-schema
-        column-definitions (mapv (fn [{:keys [name type database-type]}]
-                                   (let [db-type (if database-type
-                                                   [[:raw database-type]]
-                                                   (try
-                                                     (driver/type->database-type driver type)
-                                                     (catch IllegalArgumentException _
-                                                       (log/warnf "Couldn't determine database type for type %s, fallback to Text" type)
-                                                       (driver/type->database-type driver :type/Text))))]
-                                     [name db-type]))
-                                 columns)
-        primary-key-opts (select-keys table-schema [:primary-key])]
-    (log/infof "Creating table %s with %d columns" table-name (count columns))
-    (driver/create-table! driver database-id table-name column-definitions primary-key-opts)))
-
-(defn drop-table!
-  "Drop a table in the database."
-  [driver database-id table-name]
-  (log/infof "Dropping table %s" table-name)
-  (driver/drop-table! driver database-id table-name))
-
-(defn temp-table-name
-  "Generate a temporary table name with current timestamp in milliseconds.
-  If table name would exceed max table name length for the driver, fallback to using a shorter timestamp"
-  [driver schema]
-  (let [max-len   (max 1 (or (driver/table-name-length-limit driver) Integer/MAX_VALUE))
-        timestamp (str (System/currentTimeMillis))
-        prefix    (str transform-temp-table-prefix "_")
-        available (- max-len (count prefix))
-        ;; If we don't have enough space, take the later digits of the timestamp
-        suffix    (if (>= available (count timestamp))
-                    timestamp
-                    (subs timestamp (- (count timestamp) available)))
-        table-name (str prefix suffix)]
-    (keyword schema table-name)))
-
-(defn rename-tables!
-  "Rename multiple tables atomically within a transaction using the new driver/rename-tables method.
-   This is a simpler, composable operation that only handles renaming."
-  [driver database-id rename-map]
-  (log/infof "Renaming tables: %s" (pr-str rename-map))
-  (driver/rename-tables! driver database-id rename-map))
-
-(defenterprise is-temp-transform-table?
-  "Return true when `table` matches the transform temporary table naming pattern and transforms are enabled."
-  :feature :transforms
-  [table]
-  (when-let [table-name (:name table)]
-    (str/starts-with? (u/lower-case-en table-name) transform-temp-table-prefix)))
-
-(defn db-routing-enabled?
-  "Returns whether or not the given database is either a router or destination database"
-  [db-or-id]
-  (or (t2/exists? :model/DatabaseRouter :database_id (u/the-id db-or-id))
-      (some->> (:router-database-id db-or-id)
-               (t2/exists? :model/DatabaseRouter :database_id))))
-
-;;; ------------------------------------------------- Source Table Resolution -----------------------------------------
-
-(def ^:private ^:const batch-lookup-chunk-size
-  "Maximum number of table refs to query at once to avoid SQL query size limits."
-  100)
-
-(defn batch-lookup-table-ids
-  "Batch lookup table IDs from ref maps. Returns {[db_id schema name] -> table_id}.
-  Queries the exact conjunction of each [database_id, schema, table] triple rather than
-  a Cartesian product of all values. Uses chunking to avoid query size limits."
-  [refs]
-  (when (seq refs)
-    (let [unique-refs (distinct (map (juxt :database_id :schema :table) refs))
-          ref->clause (fn [[db-id schema table-name]]
-                        [:and
-                         [:= :db_id db-id]
-                         (if (some? schema)
-                           [:= :schema schema]
-                           [:is :schema nil])
-                         [:= :name table-name]])
-          fetch-batch (fn [batch]
-                        (t2/select-fn->fn (juxt :db_id :schema :name) :id
-                                          [:model/Table :id :db_id :schema :name]
-                                          {:where (into [:or] (map ref->clause batch))}))]
-      (into {} (mapcat fetch-batch) (partition-all batch-lookup-chunk-size unique-refs)))))
-
-(defn- source-table-ref->key
-  "Convert a source table ref map to a lookup key [db_id schema name]."
-  [{:keys [database_id schema table]}]
-  [database_id schema table])
-
-(defn- missing-table-id?
-  "Returns true if `v` is a source table ref map that needs table_id lookup."
-  [v]
-  (and (map? v) (nil? (:table_id v))))
-
-(defn normalize-source-tables
-  "Normalize source-tables to consistent map format {:database_id :schema :table :table_id}.
-
-  The old format stored just integer table IDs. New transforms store maps on write.
-  Old data is converted on read via transform-source-out for backwards compatibility.
-
-  Throws if an integer table ID references a non-existent table.
-  Map refs with non-existent tables get nil table_id (resolved later at execute time)."
-  [source-tables]
-  (let [int-table-ids    (into #{} (filter int?) (vals source-tables))
-        int-id->metadata (when (seq int-table-ids)
-                           (t2/select-pk->fn (fn [{:keys [db_id schema name]}]
-                                               {:database_id db_id :schema schema :table name})
-                                             [:model/Table :id :db_id :schema :name]
-                                             :id [:in int-table-ids]))
-        missing-ids      (when (seq int-table-ids)
-                           (remove int-id->metadata int-table-ids))
-        refs-needing-id  (filter missing-table-id? (vals source-tables))
-        ref-lookup       (or (batch-lookup-table-ids refs-needing-id) {})]
-    (when (seq missing-ids)
-      (throw (ex-info (str "Tables not found for ids: " (str/join ", " (sort missing-ids)))
-                      {:table_ids (vec missing-ids)})))
-    (update-vals source-tables
-                 (fn [v]
-                   (cond
-                     (int? v)     (assoc (int-id->metadata v) :table_id v)
-                     (:table_id v) v
-                     :else        (assoc v :table_id (ref-lookup (source-table-ref->key v))))))))
-
-(defn resolve-source-tables
-  "Resolve source-tables to {alias -> table_id}. Throws if any table not found.
-  For execute time - all entries must resolve to valid table IDs.
-  Handles both integer IDs (old format) and map refs (new format)."
-  [source-tables]
-  (let [needs-lookup (filter missing-table-id? (vals source-tables))
-        lookup       (or (batch-lookup-table-ids needs-lookup) {})
-        resolved     (u/for-map [[alias v] source-tables]
-                       [alias (if (int? v)
-                                v
-                                (or (:table_id v) (lookup (source-table-ref->key v))))])
-        unresolved   (for [[alias table-id] resolved
-                           :when (nil? table-id)
-                           :let [v (get source-tables alias)]]
-                       {:alias alias
-                        :table (if-let [schema (:schema v)]
-                                 (str schema "." (:table v))
-                                 (:table v))
-                        :ref   v})]
-    (when (seq unresolved)
-      (throw (ex-info (str "Tables not found: " (str/join ", " (map :table unresolved)))
-                      {:unresolved unresolved})))
-    resolved))
-
-(defn- matching-timestamp?
-  [job field-path {:keys [start end]}]
-  (when-let [field-instant (->instant (get-in job field-path))]
-    (let [start-instant (some-> start u.date/parse ->instant)
-          end-instant (some-> end u.date/parse ->instant)]
-      (and (or (nil? start)
-               (not (.isBefore field-instant start-instant)))
-           (or (nil? end)
-               (.isAfter end-instant field-instant))))))
-
-(defn ->date-field-filter-xf
-  "Returns an xform for a date filter."
-  [field-path filter-value]
-  (let [range (some-> filter-value (params.dates/date-string->range {:inclusive-end? false}))]
-    (if range
-      (filter #(matching-timestamp? % field-path range))
-      identity)))
-
-(defn ->status-filter-xf
-  "Returns an xform for a transform run status filter."
-  [field-path statuses]
-  (let [statuses (->> statuses (map keyword) set not-empty)]
-    (if statuses
-      (filter #(statuses (get-in % field-path)))
-      identity)))
-
-(defn ->tag-filter-xf
-  "Returns an xform for a transform tag filter."
-  [field-path tag-ids]
-  (let [tag-ids (-> tag-ids set not-empty)]
-    (if tag-ids
-      (filter #(some tag-ids (get-in % field-path)))
-      identity)))
-
-(def ^:private metabase-index-prefix "mb_transform_idx_")
-
-(defn- incremental-filter-index-name [schema table-name filter-column-name]
-  (let [prefix metabase-index-prefix
-        suffix (codecs/bytes->hex (buddy-hash/md5 (str/join "|" [schema table-name filter-column-name])))]
-    (str prefix suffix)))
-
-(defn- should-index-incremental-filter-column? [driver database]
-  (and (driver.u/supports? driver :describe-indexes database)
-       (driver.u/supports? driver :transforms/index-ddl database)))
-
-(defn- metabase-owned-index? [index]
-  (str/starts-with? (:index-name index) metabase-index-prefix))
-
-(defn- metabase-incremental-filter-index [filter-column target]
-  (let [table-name  (:name target)
-        schema      (:schema target)
-        filter-name (:name filter-column)]
-    ;; match the schema used by describe-table-indexes (:value denotes the column name, assumed leading column if a composite index)
-    {:index-name (incremental-filter-index-name schema table-name filter-name)
-     :value      filter-name}))
-
-(defn- decide-secondary-index-ddl
-  "Decides which indexes should be dropped/created on the target table.
-
-  e.g. Indexing the incremental :filter-column to accelerate MAX(target.checkpoint) queries on OLTP databases.
-
-  Indexes are represented as maps with at least the keys :index-name, :value.
-  This matches the form used by [[metabase.driver/describe-table-indexes]].
-
-  Returns a map:
-  :drop   - a vector of indexes that are redundant and should be dropped.
-  :create - a vector of desirable indexes that do not exist and that should be created.
-
-  Notes:
-  - If user covering indexes exist they should be reused.
-  - Will never drop user indexes.
-  - Indexes we previously created and are no longer required are dropped."
-  [{:keys [filter-column indexes target database]}]
-  (let [[mb-indexes user-indexes] ((juxt filter remove) metabase-owned-index? indexes)
-        default-mb-index    (when filter-column (metabase-incremental-filter-index filter-column target))
-        column-name         (:name filter-column)
-        existing-user-index (first (filter #(= (:value %) column-name) user-indexes))
-        existing-mb-index   (first (filter #(= (:index-name %) (:index-name default-mb-index)) mb-indexes))
-        driver              (:engine database)
-        intended-index      (when (should-index-incremental-filter-column? driver database)
-                              ;; Prefer reuse to not create redundant indexes
-                              (or existing-user-index default-mb-index))
-        drop                (if intended-index
-                              (remove #(= (:index-name intended-index) (:index-name %)) mb-indexes)
-                              mb-indexes)
-        create              (when (and intended-index
-                                       (not= (:index-name intended-index) (:index-name existing-user-index))
-                                       (not= (:index-name intended-index) (:index-name existing-mb-index)))
-                              [intended-index])]
-    {:drop   (vec drop)
-     :create (vec create)}))
-
 (defn execute-secondary-index-ddl-if-required!
-  "If target table index modifications are required, executes those CREATE/DROP commands.
-  See [[metabase.transforms-util/decide-secondary-index-ddl]] for details."
+  "Execute secondary index DDL with instrumentation timing.
+  Wraps base function with transforms.instrumentation timing."
   [transform run-id database target]
-  (when (driver.u/supports? (:engine database) :describe-indexes database)
-    (let [driver     (:engine database)
-          indexes    (driver/describe-table-indexes driver database target)
-          checkpoint (next-checkpoint transform)
-          {:keys [drop create]}
-          (decide-secondary-index-ddl
-           {:filter-column (:filter-column checkpoint)
-            :database      database
-            :target        target
-            :indexes       indexes})]
-      (doseq [{:keys [index-name value]} drop]
-        (transforms.instrumentation/with-stage-timing [run-id [:import :drop-incremental-filter-index]]
-          (log/infof "Dropping secondary index %s(%s) for target %s" index-name value (pr-str target))
-          (driver/drop-index! driver (:id database) (:schema target) (:name target) index-name)))
-      (doseq [{:keys [index-name value]} create]
-        (transforms.instrumentation/with-stage-timing [run-id [:import :create-incremental-filter-index]]
-          (log/infof "Creating secondary index %s(%s) for target %s" index-name value (pr-str target))
-          (driver/create-index! driver (:id database) (:schema target) (:name target) index-name [value]))))))
+  (let [with-timing (fn [rid stage thunk]
+                      (transforms.instrumentation/with-stage-timing [rid stage]
+                        (thunk)))]
+    (transforms-base.util/execute-secondary-index-ddl-if-required!
+     transform run-id database target with-timing)))

enterprise/backend/src/metabase_enterprise/transforms_base/core.clj

@@ -0,0 +1,38 @@
+(ns metabase-enterprise.transforms-base.core
+  "Entry point for base transform execution.
+
+   This namespace provides the main `execute!` function for running transforms
+   without writing to transform_run rows. Results are returned in memory.
+
+   For query transforms, this module handles execution directly.
+   For python transforms, implementations are loaded separately via
+   metabase-enterprise.transforms-python.base to avoid circular dependencies."
+  (:require
+   ;; Interface for dispatch
+   [metabase-enterprise.transforms-base.interface :as transforms-base.i]
+   ;; Load query implementation - registers multimethods
+   [metabase-enterprise.transforms-base.query]))
+
+(defn execute!
+  "Execute transform and return result map without writing transform_run rows.
+
+   This is the main entry point for base transform execution. Use this when you
+   want to run a transform without database tracking (e.g., from workspaces).
+
+   Options:
+   - `:cancelled?` - (fn [] boolean), polled during execution to check for cancellation
+   - `:run-id` - optional, for instrumentation/metrics (nil skips metrics recording)
+   - `:with-stage-timing-fn` - optional, (fn [run-id stage thunk] result)
+
+   Returns a map:
+   {:status :succeeded | :failed | :cancelled | :timeout
+    :result <implementation-specific result>
+    :error <exception if failed>
+    :logs <string, for python transforms>}
+
+   Target table sync (metabase_table writes) still occurs - this is necessary
+   for the table to be usable after execution."
+  ([transform]
+   (execute! transform nil))
+  ([transform opts]
+   (transforms-base.i/execute-base! transform opts)))

enterprise/backend/src/metabase_enterprise/transforms_base/interface.clj

@@ -0,0 +1,67 @@
+(ns metabase-enterprise.transforms-base.interface
+  "Interface definitions for transform execution.
+
+   This namespace defines the multimethods that different transform types
+   (query, python) must implement. The base module uses these for dispatch
+   without any database writes for transform_run tracking.")
+
+(defn transform->transform-type
+  "Extract the transform type from a transform's source."
+  [transform]
+  (-> transform :source :type keyword))
+
+(defmulti source-db-id
+  "Return the ID of the source database for a given `transform`. The source database is where the data originates from
+  before being transformed and written to the target destination.
+
+  NOTE: Currently returns a single database ID, but we may want to change this to return multiple database IDs in the
+  future since transforms can have multiple input sources."
+  {:added "0.57.0" :arglists '([transform])}
+  transform->transform-type)
+
+(defmulti target-db-id
+  "Return the ID of the target database for a given `transform`. The target database is the destination where the
+  transformed data will be written. Often this is the same database as the source database."
+  {:added "0.57.0" :arglists '([transform])}
+  transform->transform-type)
+
+(defmulti table-dependencies
+  "Return a set of logical table dependencies of the transform, including indirect dependencies via cards.
+  The transform execution system uses these dependencies to determine the correct order of execution
+  and to detect circular dependencies.
+
+  Each dependency is represented as one of:
+  - `{:table <table-id>}`
+     A dependency on a table that exists and has been synced.
+  - `{:transform <transform-id>}`
+     A dependency on a table that does not yet exist, but is known to be the target of another transform.
+     Represents a 'placeholder' table (as we no table id / metadata) for the same purposes.
+  - `{:table-ref {:database_id <db-id> :schema <schema> :table <name>}}`
+     A dependency on a table by name, for cases where the table_id may not exist yet (e.g., when
+     bulk importing transforms that depend on each other's outputs). The ordering system will
+     resolve these to transforms by matching against target table definitions.
+
+  An empty set indicates no dependencies."
+  {:added "0.57.0" :arglists '([transform])}
+  transform->transform-type)
+
+(defmulti execute-base!
+  "Execute a transform and return results in memory. Does NOT write transform_run rows.
+
+  Options:
+  - `:cancelled?` - (fn [] boolean), polled during execution to check for cancellation
+  - `:run-id` - optional, for instrumentation/metrics (nil skips metrics recording)
+
+  Returns a map:
+  {:status :succeeded | :failed | :cancelled
+   :result <implementation-specific result>
+   :error <exception if failed>
+   :logs <string, for python transforms>}
+
+  Implementations should:
+  - Check `(cancelled?)` at safe points and return {:status :cancelled} if true
+  - Still sync target tables (metabase_table writes are allowed)
+  - NOT create/update transform_run rows"
+  {:added "0.57.0" :arglists '([transform options])}
+  (fn [transform _options]
+    (transform->transform-type transform)))

enterprise/backend/src/metabase_enterprise/transforms_base/ordering.clj

@@ -1,10 +1,13 @@
-(ns metabase-enterprise.transforms.ordering
+(ns metabase-enterprise.transforms-base.ordering
+  "Transform dependency ordering and cycle detection.
+
+   Pure functions for computing execution order based on table dependencies."
   (:require
    [clojure.set :as set]
    [clojure.string :as str]
    [flatland.ordered.set :refer [ordered-set]]
-   [metabase-enterprise.transforms.interface :as transforms.i]
-   [metabase-enterprise.transforms.util :as transforms.util]
+   [metabase-enterprise.transforms-base.interface :as transforms-base.i]
+   [metabase-enterprise.transforms-base.util :as transforms-base.util]
    [metabase.driver :as driver]
    [metabase.lib-be.core :as lib-be]
    [metabase.lib.core :as lib]
@@ -17,10 +20,10 @@
 
 (set! *warn-on-reflection* true)
 
-(defmethod transforms.i/table-dependencies :query
+(defmethod transforms-base.i/table-dependencies :query
   [{:keys [source]}]
   (let [query (-> (:query source)
-                  transforms.util/massage-sql-query
+                  transforms-base.util/massage-sql-query
                   qp.preprocess/preprocess)
         driver (-> query
                    lib.metadata/database
@@ -34,7 +37,7 @@
 
 (defn- dependency-map [transforms]
   (into {}
-        (map (juxt :id transforms.i/table-dependencies))
+        (map (juxt :id transforms-base.i/table-dependencies))
         transforms))
 
 (mu/defn- output-table-map
@@ -78,7 +81,7 @@
   (let [;; Group all transforms by their database
         transforms-by-db (->> transforms
                               (map (fn [transform]
-                                     (let [db-id (transforms.i/target-db-id transform)]
+                                     (let [db-id (transforms-base.i/target-db-id transform)]
                                        {db-id [transform]})))
                               (apply merge-with into))
         transform-ids    (into #{} (map :id) transforms)
@@ -144,7 +147,7 @@
         output-tables    (output-table-map mp db-transforms)
         transform-ids    (into #{} (map :id) db-transforms)
         target-refs      (target-ref-map transforms)
-        node->children   #(->> % transforms-by-id transforms.i/table-dependencies
+        node->children   #(->> % transforms-by-id transforms-base.i/table-dependencies
                                (keep (fn [dep] (resolve-dependency dep output-tables transform-ids target-refs))))
         id->name         (comp :name transforms-by-id)
         cycle            (find-cycle node->children [transform-id])]

enterprise/backend/src/metabase_enterprise/transforms_base/query.clj

@@ -0,0 +1,147 @@
+(ns metabase-enterprise.transforms-base.query
+  "Base query transform execution - core logic without transform_run tracking.
+
+   This namespace handles MBQL/native query transform execution and returns
+   results in memory rather than writing to transform_run rows."
+  (:require
+   [metabase-enterprise.transforms-base.interface :as transforms-base.i]
+   [metabase-enterprise.transforms-base.util :as transforms-base.util]
+   [metabase.driver :as driver]
+   [metabase.driver.util :as driver.u]
+   [metabase.events.core :as events]
+   [metabase.lib.schema.common :as schema.common]
+   [metabase.query-processor.compile :as qp.compile]
+   [metabase.util.log :as log]
+   [metabase.util.malli.registry :as mr]
+   [toucan2.core :as t2]))
+
+(set! *warn-on-reflection* true)
+
+;;; ------------------------------------------------- Interface Implementations -------------------------------------------------
+
+(defmethod transforms-base.i/source-db-id :query
+  [transform]
+  (-> transform :source :query :database))
+
+(defmethod transforms-base.i/target-db-id :query
+  [transform]
+  (-> transform :source :query :database))
+
+;;; ------------------------------------------------- Schemas -------------------------------------------------
+
+(mr/def ::transform-details
+  [:map
+   [:transform-type [:enum {:decode/normalize schema.common/normalize-keyword} :table :table-incremental]]
+   [:conn-spec :any]
+   [:query ::qp.compile/compiled]
+   [:output-table [:keyword {:decode/normalize schema.common/normalize-keyword}]]])
+
+(mr/def ::transform-opts
+  [:map
+   [:overwrite? :boolean]])
+
+;;; ------------------------------------------------- Helpers -------------------------------------------------
+
+(defn- transform-opts [{:keys [transform-type]}]
+  (case transform-type
+    :table {:overwrite? true}
+    ;; once we have more than just append, dispatch on :target-incremental-strategy
+    :table-incremental {}))
+
+;;; ------------------------------------------------- Base Execution -------------------------------------------------
+
+(defn run-query-transform!
+  "Execute query transform (MBQL/native). Returns result map.
+
+   Does:
+   - Compile query
+   - Create schema if needed
+   - Call driver/run-transform!
+   - Sync target table
+   - Manage secondary indexes
+
+   Does NOT:
+   - Create transform_run row
+   - Update transform_run status
+
+   Options:
+   - `:cancelled?` - (fn [] boolean), polled to check for cancellation
+   - `:run-id` - optional, for instrumentation (nil skips metrics)
+   - `:with-stage-timing-fn` - optional, (fn [run-id stage thunk] result)
+
+   Returns:
+   {:status :succeeded | :failed | :cancelled
+    :result <driver result>
+    :error <exception if failed>}"
+  [{:keys [id source target] :as transform}
+   {:keys [cancelled? run-id with-stage-timing-fn]}]
+  (try
+    ;; Check cancellation before starting
+    (when (and cancelled? (cancelled?))
+      (throw (ex-info "Transform cancelled before start" {:status :cancelled})))
+
+    (let [db (get-in source [:query :database])
+          {driver :engine :as database} (t2/select-one :model/Database db)
+          transform-details {:db-id db
+                             :database database
+                             :transform-id   id
+                             :transform-type (keyword (:type target))
+                             :conn-spec (driver/connection-spec driver database)
+                             :query (transforms-base.util/compile-source transform)
+                             :output-schema (:schema target)
+                             :output-table (transforms-base.util/qualified-table-name driver target)}
+          opts (transform-opts transform-details)
+          features (transforms-base.util/required-database-features transform)]
+
+      (when (transforms-base.util/db-routing-enabled? database)
+        (throw (ex-info "Transforms are not supported on databases with DB routing enabled."
+                        {:driver driver, :database database})))
+      (when-not (every? (fn [feature] (driver.u/supports? (:engine database) feature database)) features)
+        (throw (ex-info "The database does not support the requested transform target type."
+                        {:driver driver, :database database, :features features})))
+
+      (log/info "Executing transform" id "with target" (pr-str target))
+
+      ;; Create schema if needed
+      (when-not (driver/schema-exists? driver db (:schema target))
+        (driver/create-schema-if-needed! driver (:conn-spec transform-details) (:schema target)))
+
+      ;; Check cancellation before running query
+      (when (and cancelled? (cancelled?))
+        (throw (ex-info "Transform cancelled before query execution" {:status :cancelled})))
+
+      ;; Run the actual transform
+      (let [result (driver/run-transform! driver transform-details opts)]
+
+        ;; Check cancellation after query but before sync
+        (when (and cancelled? (cancelled?))
+          (throw (ex-info "Transform cancelled after query execution" {:status :cancelled})))
+
+        ;; Sync target table
+        (transforms-base.util/sync-target! target database)
+
+        ;; Publish event (after sync, so table exists in AppDB)
+        (events/publish-event! :event/transform-run-complete {:object transform-details})
+
+        ;; Create secondary indexes if needed
+        (transforms-base.util/execute-secondary-index-ddl-if-required!
+         transform run-id database target with-stage-timing-fn)
+
+        {:status :succeeded
+         :result result}))
+
+    (catch Exception e
+      (let [data (ex-data e)]
+        (if (= :cancelled (:status data))
+          {:status :cancelled
+           :error e}
+          (do
+            (log/error e "Error executing transform")
+            {:status :failed
+             :error e}))))))
+
+;;; ------------------------------------------------- Interface Implementation -------------------------------------------------
+
+(defmethod transforms-base.i/execute-base! :query
+  [transform opts]
+  (run-query-transform! transform opts))

enterprise/backend/src/metabase_enterprise/transforms_base/util.clj

@@ -0,0 +1,662 @@
+(ns metabase-enterprise.transforms-base.util
+  "Shared utilities for transform execution.
+
+   This namespace contains functions that are used by both query and python transforms
+   and do NOT require transform_run database access."
+  (:require
+   [buddy.core.codecs :as codecs]
+   [buddy.core.hash :as buddy-hash]
+   [clojure.string :as str]
+   [java-time.api :as t]
+   [metabase-enterprise.transforms-base.interface :as transforms-base.i]
+   [metabase.driver :as driver]
+   [metabase.driver.sql.query-processor :as sql.qp]
+   [metabase.driver.util :as driver.u]
+   [metabase.lib-be.core :as lib-be]
+   [metabase.lib.core :as lib]
+   [metabase.lib.metadata :as lib.metadata]
+   [metabase.lib.query :as lib.query]
+   [metabase.lib.schema.common :as lib.schema.common]
+   [metabase.premium-features.core :as premium-features :refer [defenterprise]]
+   [metabase.query-processor :as qp]
+   [metabase.query-processor.compile :as qp.compile]
+   [metabase.query-processor.parameters.dates :as params.dates]
+   [metabase.sync.core :as sync]
+   [metabase.util :as u]
+   [metabase.util.date-2 :as u.date]
+   [metabase.util.honey-sql-2 :as h2x]
+   [metabase.util.log :as log]
+   [metabase.util.malli :as mu]
+   [metabase.util.malli.registry :as mr]
+   [toucan2.core :as t2])
+  (:import
+   (java.time Instant LocalDate LocalDateTime LocalTime OffsetDateTime OffsetTime ZonedDateTime)
+   (java.util Date)))
+
+(set! *warn-on-reflection* true)
+
+;;; ------------------------------------------------- Constants -------------------------------------------------
+
+(def ^:const transform-temp-table-prefix
+  "Prefix used for temporary tables created during transform execution."
+  "mb_transform_temp_table")
+
+;;; ------------------------------------------------- Transform Type Predicates -------------------------------------------------
+
+(defn query-transform?
+  "Check if this is a query transform: native query / mbql query."
+  [transform]
+  (= :query (-> transform :source :type keyword)))
+
+(defn native-query-transform?
+  "Check if this is a native query transform.
+  Note: The transform should be normalized (via `normalize-transform`) before calling this function."
+  [transform]
+  (when (query-transform? transform)
+    (let [query (-> transform :source :query)]
+      (lib/native-only-query? query))))
+
+(defn python-transform?
+  "Check if this is a Python transform."
+  [transform]
+  (= :python (-> transform :source :type keyword)))
+
+;;; ------------------------------------------------- Transform Normalization -------------------------------------------------
+
+(defn normalize-transform
+  "Normalize a transform's source query, similar to how transforms are normalized when read from the database.
+  This should be called on transforms before processing them to ensure queries are in the expected format."
+  [transform]
+  (if (and (map? transform)
+           (= (:type transform) "transform")
+           (get-in transform [:source :query]))
+    (update-in transform [:source :query] lib-be/normalize-query)
+    transform))
+
+(defn transform-source-type
+  "Returns the type of a transform's source: :python, :native, or :mbql.
+  Throws if the source type cannot be detected.
+  Note: The transform should be normalized (via `normalize-transform`) before calling this function."
+  [source]
+  (case (keyword (:type source))
+    :python :python
+    :query  (if (lib/native-only-query? (:query source))
+              :native
+              :mbql)
+    (throw (ex-info (str "Unknown transform source type: " (:type source))
+                    {:source source}))))
+
+;;; ------------------------------------------------- Feature Checks -------------------------------------------------
+
+(defn check-feature-enabled
+  "Checking whether we have proper feature flags for using a given transform."
+  [transform]
+  (if (python-transform? transform)
+    (and (premium-features/has-feature? :transforms)
+         (premium-features/has-feature? :transforms-python))
+    (premium-features/has-feature? :transforms)))
+
+(defn required-database-features
+  "Returns the database features necessary to execute `transform`."
+  [transform]
+  (if (python-transform? transform)
+    [:transforms/python]
+    [:transforms/table]))
+
+;;; ------------------------------------------------- Table Names -------------------------------------------------
+
+(defn qualified-table-name
+  "Return the name of the target table of a transform as a possibly qualified symbol."
+  [_driver {:keys [schema name]}]
+  (if schema
+    (keyword schema name)
+    (keyword name)))
+
+(defn temp-table-name
+  "Generate a temporary table name with current timestamp in milliseconds.
+  If table name would exceed max table name length for the driver, fallback to using a shorter timestamp"
+  [driver schema]
+  (let [max-len   (max 1 (or (driver/table-name-length-limit driver) Integer/MAX_VALUE))
+        timestamp (str (System/currentTimeMillis))
+        prefix    (str transform-temp-table-prefix "_")
+        available (- max-len (count prefix))
+        ;; If we don't have enough space, take the later digits of the timestamp
+        suffix    (if (>= available (count timestamp))
+                    timestamp
+                    (subs timestamp (- (count timestamp) available)))
+        table-name (str prefix suffix)]
+    (keyword schema table-name)))
+
+;;; ------------------------------------------------- Query Helpers -------------------------------------------------
+
+(defn massage-sql-query
+  "Adjusts mbql query for use in a transform."
+  [query]
+  (assoc-in query [:middleware :disable-remaps?] true))
+
+;;; ------------------------------------------------- Incremental/Checkpoint Helpers -------------------------------------------------
+
+(defn- checkpoint-incremental?
+  "Returns true if `source` uses checkpoint-based incremental strategy."
+  [source]
+  (= :checkpoint (some-> source :source-incremental-strategy :type keyword)))
+
+(defn supported-incremental-filter-type?
+  "Returns true if the given base-type is supported for incremental filtering.
+
+  We only support temporal (timestamp/tz) and numeric (int/float) types."
+  [base-type]
+  (or (isa? base-type :type/Temporal)
+      (isa? base-type :type/Number)))
+
+(defn- source->checkpoint-filter-unique-key
+  "Extract the checkpoint filter column from `query` using the unique key specified in `source-incremental-strategy`."
+  [query source-incremental-strategy]
+  (some->> source-incremental-strategy :checkpoint-filter-unique-key (lib/column-with-unique-key query)))
+
+(defn- source->checkpoint-filter-column
+  "Resolve the checkpoint filter column for an incremental transform.
+
+  Tries to resolve the column using the unique key first.
+  Falls back to looking up the column by name from the target table if a `:checkpoint-filter` is specified.
+
+  Validates that the resolved column has a supported type for checkpoint filtering (numeric or temporal).
+  Throws an exception if the column type is not supported."
+  [query source-incremental-strategy table metadata-provider]
+  (let [{:keys [checkpoint-filter checkpoint-filter-unique-key]} source-incremental-strategy]
+  (when-some [{column-name :name
+               :keys [base-type]
+               :as column}
+              (cond
+                checkpoint-filter-unique-key
+                (source->checkpoint-filter-unique-key query source-incremental-strategy)
+                checkpoint-filter
+                (when-some [field-id (t2/select-one-pk :model/Field
+                                                       :table_id (:id table)
+                                                       :name checkpoint-filter)]
+                  (lib.metadata/field metadata-provider field-id)))]
+    (when-not (supported-incremental-filter-type? base-type)
+      (throw (ex-info (str "Checkpoint column '" column-name "' has unsupported type " (pr-str base-type) ". "
+                           "Only numeric and temporal columns are supported for incremental filtering.")
+                      {:column-name column-name
+                       :base-type   base-type})))
+    column)))
+
+(declare target-table)
+
+(defn next-checkpoint
+  "Build a query to compute the MAX of the checkpoint column from the target table.
+
+  Returns a map with `:query` (MBQL query selecting the max) and `:filter-column` (column metadata),
+  or `nil` if the transform doesn't use checkpoint-based incremental strategy or the target table doesn't exist."
+  [{:keys [source target] :as transform}]
+  (let [db-id (transforms-base.i/target-db-id transform)]
+    (when (checkpoint-incremental? source)
+      (when-let [table (target-table db-id target)]
+        (let [metadata-provider (lib-be/application-database-metadata-provider db-id)
+              table-metadata (lib.metadata/table metadata-provider (:id table))
+              query (lib/query metadata-provider table-metadata)]
+          (when-let [filter-column (source->checkpoint-filter-column query
+                                                                     (:source-incremental-strategy source)
+                                                                     table metadata-provider)]
+            {:query (-> query (lib/aggregate (lib/max filter-column)))
+             :filter-column filter-column}))))))
+
+(defn- next-checkpoint-value
+  "Execute the checkpoint query and normalize the result for database insertion.
+  Returns `nil` if the target table is empty."
+  [{:keys [query filter-column]}]
+  (let [{:keys [base-type]} filter-column
+        v (some-> query qp/process-query :data :rows first first)]
+    ;; QP return values are lossy, we do a bit of parsing to ensure they're of the right
+    ;; shape for reinsertion
+    (cond
+      (nil? v)
+      nil
+
+      (isa? base-type :type/Integer)
+      (bigint v)
+
+      ;; any other number that's not an integer, should be a decimal/float
+      (number? v)
+      (bigdec v)
+
+      :else v)))
+
+(defn preprocess-incremental-query
+  "Add checkpoint checkpoint filtering to a query for incremental execution.
+
+  For native queries with a `checkpoint` template tag, adds the checkpoint as a parameter.
+  For MBQL queries, adds a filter clause `WHERE checkpoint_column > checkpoint`.
+  Returns the query unchanged on first run (no checkpoint) or for native queries without the checkpoint tag."
+  [query source-incremental-strategy checkpoint]
+  (if-let [checkpoint-value (next-checkpoint-value checkpoint)]
+    (if (lib.query/native? query)
+      ;; native query with explicit checkpoint filter
+      (if (get-in query [:stages 0 :template-tags "checkpoint"])
+        (update query :parameters conj
+                {:type (if (number? checkpoint-value) :number :text)
+                 :target [:variable [:template-tag "checkpoint"]]
+                 :value checkpoint-value})
+        query)
+      ;; mbql query
+      (lib/filter query (lib/> (source->checkpoint-filter-unique-key query source-incremental-strategy) checkpoint-value)))
+    query))
+
+(defn- post-process-incremental-query
+  "Wrap a compiled native query with checkpoint filtering for native queries without explicit checkpoint tags.
+
+  Generates SQL that wraps the original query as a subquery and filters by `checkpoint_filter > (checkpoint_query)`. "
+  [outer-query driver {:keys [source-incremental-strategy] :as source} {checkpoint-query :query :as checkpoint}]
+  (let [{:keys [checkpoint-filter]} source-incremental-strategy]
+    (if (and (lib.query/native? (:query source))
+             (not (get-in (:query source) [:stages 0 :template-tags "checkpoint"]))
+             (next-checkpoint-value checkpoint))
+      (let [wrap-query (fn [query]
+                         (let [honeysql-query {:select [:*]
+                                               :from [[[:raw (str "(" query ")")] :subquery]]
+                                               :where [:> (h2x/identifier :field checkpoint-filter)
+                                                       [:raw (str "(" (:query (qp.compile/compile checkpoint-query)) ")")]]}]
+                           (first (sql.qp/format-honeysql driver honeysql-query))))]
+        (update outer-query :query wrap-query))
+      outer-query)))
+
+(defn compile-source
+  "Compile the source query of a transform to SQL, applying incremental filtering if required."
+  [{:keys [source] :as transform}]
+  (let [{:keys [source-incremental-strategy] query-type :type} source]
+    (case (keyword query-type)
+      :query
+      (let [checkpoint (next-checkpoint transform)
+            query (:query source)
+            driver (some->> query :database (t2/select-one :model/Database) :engine keyword)]
+        (binding [driver/*compile-with-inline-parameters*
+                  (or (= :clickhouse driver)
+                      driver/*compile-with-inline-parameters*)]
+          (-> query
+              (preprocess-incremental-query source-incremental-strategy checkpoint)
+              massage-sql-query
+              qp.compile/compile
+              (post-process-incremental-query driver source checkpoint)))))))
+
+;;; ------------------------------------------------- Target Table Management -------------------------------------------------
+
+(defn target-table
+  "Load the `target` table of a transform from the database specified by `database-id`."
+  [database-id target & kv-args]
+  (some-> (apply t2/select-one :model/Table
+                 :db_id database-id
+                 :schema (:schema target)
+                 :name (:name target)
+                 kv-args)
+          (t2/hydrate :db)))
+
+(defn target-table-exists?
+  "Test if the target table of a transform already exists."
+  [{:keys [target] :as transform}]
+  (let [db-id (transforms-base.i/target-db-id transform)
+        {driver :engine :as database} (t2/select-one :model/Database db-id)]
+    (driver/table-exists? driver database target)))
+
+(defn- sync-table!
+  ([database target] (sync-table! database target nil))
+  ([database target {:keys [create?]}]
+   (when-let [table (or (target-table (:id database) target)
+                        (when create?
+                          (sync/create-table! database (select-keys target [:schema :name :data_source :data_authority]))))]
+     (sync/sync-table! table)
+     table)))
+
+(defn activate-table-and-mark-computed!
+  "Activate table for `target` in `database` in the app db."
+  [database target]
+  (when-let [table (sync-table! database (assoc target
+                                                :data_authority :computed
+                                                :data_source :metabase-transform)
+                                {:create? true})]
+    (when-not (:active table)
+      (t2/update! :model/Table (:id table) {:active true}))
+    table))
+
+(defn sync-target!
+  "Sync target of a transform"
+  [target database]
+  ;; sync the new table (note that even a failed sync status means that the execution succeeded)
+  (log/info "Syncing target" (pr-str target) "for transform")
+  (activate-table-and-mark-computed! database target))
+
+(defn deactivate-table!
+  "Deactivate table for `target` in `database` in the app db."
+  [database target]
+  (when-let [table (sync-table! database target)]
+    ;; TODO this should probably be a function in the sync module
+    (t2/update! :model/Table (:id table) {:active false})))
+
+(defn delete-target-table!
+  "Delete the target table of a transform and sync it from the app db."
+  [{:keys [id target], :as transform}]
+  (when target
+    (let [target (update target :type keyword)
+          database-id (transforms-base.i/target-db-id transform)
+          {driver :engine :as database} (t2/select-one :model/Database database-id)]
+      (driver/drop-transform-target! driver database target)
+      (log/info "Deactivating  target " (pr-str target) "for transform" id)
+      (deactivate-table! database target))))
+
+(defn delete-target-table-by-id!
+  "Delete the target table of the transform specified by `transform-id`."
+  [transform-id]
+  (delete-target-table! (t2/select-one :model/Transform transform-id)))
+
+;;; ------------------------------------------------- Table DDL -------------------------------------------------
+
+(mr/def ::column-definition
+  [:map
+   [:name :string]
+   [:type ::lib.schema.common/base-type]
+   [:nullable? {:optional true} :boolean]])
+
+(mr/def ::table-definition
+  [:map
+   [:name :keyword]
+   [:columns [:sequential ::column-definition]]
+   [:primary-key {:optional true} [:sequential :string]]])
+
+(mu/defn create-table-from-schema!
+  "Create a table from a table-schema"
+  [driver :- :keyword
+   database-id :- pos-int?
+   table-schema :- ::table-definition]
+  (let [{:keys [columns] table-name :name} table-schema
+        column-definitions (mapv (fn [{:keys [name type database-type]}]
+                                   (let [db-type (if database-type
+                                                   [[:raw database-type]]
+                                                   (try
+                                                     (driver/type->database-type driver type)
+                                                     (catch IllegalArgumentException _
+                                                       (log/warnf "Couldn't determine database type for type %s, fallback to Text" type)
+                                                       (driver/type->database-type driver :type/Text))))]
+                                     [name db-type]))
+                                 columns)
+        primary-key-opts (select-keys table-schema [:primary-key])]
+    (log/infof "Creating table %s with %d columns" table-name (count columns))
+    (driver/create-table! driver database-id table-name column-definitions primary-key-opts)))
+
+(defn drop-table!
+  "Drop a table in the database."
+  [driver database-id table-name]
+  (log/infof "Dropping table %s" table-name)
+  (driver/drop-table! driver database-id table-name))
+
+(defn rename-tables!
+  "Rename multiple tables atomically within a transaction using the new driver/rename-tables method.
+   This is a simpler, composable operation that only handles renaming."
+  [driver database-id rename-map]
+  (log/infof "Renaming tables: %s" (pr-str rename-map))
+  (driver/rename-tables! driver database-id rename-map))
+
+;;; ------------------------------------------------- Secondary Index Management -------------------------------------------------
+
+(def ^:private metabase-index-prefix "mb_transform_idx_")
+
+(defn- incremental-filter-index-name [schema table-name filter-column-name]
+  (let [prefix metabase-index-prefix
+        suffix (codecs/bytes->hex (buddy-hash/md5 (str/join "|" [schema table-name filter-column-name])))]
+    (str prefix suffix)))
+
+(defn- should-index-incremental-filter-column? [driver database]
+  (and (driver.u/supports? driver :describe-indexes database)
+       (driver.u/supports? driver :transforms/index-ddl database)))
+
+(defn- metabase-owned-index? [index]
+  (str/starts-with? (:index-name index) metabase-index-prefix))
+
+(defn- metabase-incremental-filter-index [filter-column target]
+  (let [table-name  (:name target)
+        schema      (:schema target)
+        filter-name (:name filter-column)]
+    ;; match the schema used by describe-table-indexes (:value denotes the column name, assumed leading column if a composite index)
+    {:index-name (incremental-filter-index-name schema table-name filter-name)
+     :value      filter-name}))
+
+(defn- decide-secondary-index-ddl
+  "Decides which indexes should be dropped/created on the target table.
+
+  e.g. Indexing the incremental :filter-column to accelerate MAX(target.checkpoint) queries on OLTP databases.
+
+  Indexes are represented as maps with at least the keys :index-name, :value.
+  This matches the form used by [[metabase.driver/describe-table-indexes]].
+
+  Returns a map:
+  :drop   - a vector of indexes that are redundant and should be dropped.
+  :create - a vector of desirable indexes that do not exist and that should be created.
+
+  Notes:
+  - If user covering indexes exist they should be reused.
+  - Will never drop user indexes.
+  - Indexes we previously created and are no longer required are dropped."
+  [{:keys [filter-column indexes target database]}]
+  (let [[mb-indexes user-indexes] ((juxt filter remove) metabase-owned-index? indexes)
+        default-mb-index    (when filter-column (metabase-incremental-filter-index filter-column target))
+        column-name         (:name filter-column)
+        existing-user-index (first (filter #(= (:value %) column-name) user-indexes))
+        existing-mb-index   (first (filter #(= (:index-name %) (:index-name default-mb-index)) mb-indexes))
+        driver              (:engine database)
+        intended-index      (when (should-index-incremental-filter-column? driver database)
+                              ;; Prefer reuse to not create redundant indexes
+                              (or existing-user-index default-mb-index))
+        drop                (if intended-index
+                              (remove #(= (:index-name intended-index) (:index-name %)) mb-indexes)
+                              mb-indexes)
+        create              (when (and intended-index
+                                       (not= (:index-name intended-index) (:index-name existing-user-index))
+                                       (not= (:index-name intended-index) (:index-name existing-mb-index)))
+                              [intended-index])]
+    {:drop   (vec drop)
+     :create (vec create)}))
+
+(defn execute-secondary-index-ddl-if-required!
+  "If target table index modifications are required, executes those CREATE/DROP commands.
+  See [[decide-secondary-index-ddl]] for details.
+
+  `run-id` is optional - if nil, instrumentation/timing is skipped.
+  `with-stage-timing-fn` is optional - if provided, will be called as (with-stage-timing-fn run-id stage thunk)."
+  ([transform database target]
+   (execute-secondary-index-ddl-if-required! transform nil database target nil))
+  ([transform run-id database target with-stage-timing-fn]
+   (when (driver.u/supports? (:engine database) :describe-indexes database)
+     (let [driver     (:engine database)
+           indexes    (driver/describe-table-indexes driver database target)
+           checkpoint (next-checkpoint transform)
+           {:keys [drop create]}
+           (decide-secondary-index-ddl
+            {:filter-column (:filter-column checkpoint)
+             :database      database
+             :target        target
+             :indexes       indexes})
+           maybe-timed (fn [stage thunk]
+                         (if (and run-id with-stage-timing-fn)
+                           (with-stage-timing-fn run-id stage thunk)
+                           (thunk)))]
+       (doseq [{:keys [index-name value]} drop]
+         (maybe-timed [:import :drop-incremental-filter-index]
+                      (fn []
+                        (log/infof "Dropping secondary index %s(%s) for target %s" index-name value (pr-str target))
+                        (driver/drop-index! driver (:id database) (:schema target) (:name target) index-name))))
+       (doseq [{:keys [index-name value]} create]
+         (maybe-timed [:import :create-incremental-filter-index]
+                      (fn []
+                        (log/infof "Creating secondary index %s(%s) for target %s" index-name value (pr-str target))
+                        (driver/create-index! driver (:id database) (:schema target) (:name target) index-name [value]))))))))
+
+;;; ------------------------------------------------- Source Table Resolution -------------------------------------------------
+
+(def ^:private ^:const batch-lookup-chunk-size
+  "Maximum number of table refs to query at once to avoid SQL query size limits."
+  100)
+
+(defn batch-lookup-table-ids
+  "Batch lookup table IDs from ref maps. Returns {[db_id schema name] -> table_id}.
+  Queries the exact conjunction of each [database_id, schema, table] triple rather than
+  a Cartesian product of all values. Uses chunking to avoid query size limits."
+  [refs]
+  (when (seq refs)
+    (let [unique-refs (distinct (map (juxt :database_id :schema :table) refs))
+          ref->clause (fn [[db-id schema table-name]]
+                        [:and
+                         [:= :db_id db-id]
+                         (if (some? schema)
+                           [:= :schema schema]
+                           [:is :schema nil])
+                         [:= :name table-name]])
+          fetch-batch (fn [batch]
+                        (t2/select-fn->fn (juxt :db_id :schema :name) :id
+                                          [:model/Table :id :db_id :schema :name]
+                                          {:where (into [:or] (map ref->clause batch))}))]
+      (into {} (mapcat fetch-batch) (partition-all batch-lookup-chunk-size unique-refs)))))
+
+(defn- source-table-ref->key
+  "Convert a source table ref map to a lookup key [db_id schema name]."
+  [{:keys [database_id schema table]}]
+  [database_id schema table])
+
+(defn- missing-table-id?
+  "Returns true if `v` is a source table ref map that needs table_id lookup."
+  [v]
+  (and (map? v) (nil? (:table_id v))))
+
+(defn normalize-source-tables
+  "Normalize source-tables to consistent map format {:database_id :schema :table :table_id}.
+
+  The old format stored just integer table IDs. New transforms store maps on write.
+  Old data is converted on read via transform-source-out for backwards compatibility.
+
+  Throws if an integer table ID references a non-existent table.
+  Map refs with non-existent tables get nil table_id (resolved later at execute time)."
+  [source-tables]
+  (let [int-table-ids    (into #{} (filter int?) (vals source-tables))
+        int-id->metadata (when (seq int-table-ids)
+                           (t2/select-pk->fn (fn [{:keys [db_id schema name]}]
+                                               {:database_id db_id :schema schema :table name})
+                                             [:model/Table :id :db_id :schema :name]
+                                             :id [:in int-table-ids]))
+        missing-ids      (when (seq int-table-ids)
+                           (remove int-id->metadata int-table-ids))
+        refs-needing-id  (filter missing-table-id? (vals source-tables))
+        ref-lookup       (or (batch-lookup-table-ids refs-needing-id) {})]
+    (when (seq missing-ids)
+      (throw (ex-info (str "Tables not found for ids: " (str/join ", " (sort missing-ids)))
+                      {:table_ids (vec missing-ids)})))
+    (update-vals source-tables
+                 (fn [v]
+                   (cond
+                     (int? v)     (assoc (int-id->metadata v) :table_id v)
+                     (:table_id v) v
+                     :else        (assoc v :table_id (ref-lookup (source-table-ref->key v))))))))
+
+(defn resolve-source-tables
+  "Resolve source-tables to {alias -> table_id}. Throws if any table not found.
+  For execute time - all entries must resolve to valid table IDs.
+  Handles both integer IDs (old format) and map refs (new format)."
+  [source-tables]
+  (let [needs-lookup (filter missing-table-id? (vals source-tables))
+        lookup       (or (batch-lookup-table-ids needs-lookup) {})
+        resolved     (u/for-map [[alias v] source-tables]
+                       [alias (if (int? v)
+                                v
+                                (or (:table_id v) (lookup (source-table-ref->key v))))])
+        unresolved   (for [[alias table-id] resolved
+                           :when (nil? table-id)
+                           :let [v (get source-tables alias)]]
+                       {:alias alias
+                        :table (if-let [schema (:schema v)]
+                                 (str schema "." (:table v))
+                                 (:table v))
+                        :ref   v})]
+    (when (seq unresolved)
+      (throw (ex-info (str "Tables not found: " (str/join ", " (map :table unresolved)))
+                      {:unresolved unresolved})))
+    resolved))
+
+;;; ------------------------------------------------- Timestamp Helpers -------------------------------------------------
+
+(defn ->instant
+  "Convert a temporal value `t` to an Instant in the system timezone."
+  ^Instant [t]
+  (when t
+    (condp instance? t
+      Instant        t
+      Date           (.toInstant ^Date t)
+      OffsetDateTime (.toInstant ^OffsetDateTime t)
+      ZonedDateTime  (.toInstant ^ZonedDateTime t)
+      LocalDateTime  (recur (.atZone ^LocalDateTime t (t/zone-id)))
+      String         (recur (u.date/parse t))
+      LocalTime      (recur (.atDate ^LocalTime t (t/local-date)))
+      OffsetTime     (recur (.atDate ^OffsetTime t (t/local-date)))
+      LocalDate      (recur (.atStartOfDay ^LocalDate t))
+      (throw (ex-info (str "Cannot convert temporal " t " of type " (type t) " to an Instant")
+                      {:temporal t})))))
+
+(defn utc-timestamp-string
+  "Convert the timestamp t to a string encoding the it in the system timezone."
+  [t]
+  (-> t ->instant str))
+
+(defn localize-run-timestamps
+  "Convert the timestamps of a `run` to ISO strings in UTC."
+  [run]
+  (-> run
+      (u/update-some :start_time utc-timestamp-string)
+      (u/update-some :end_time   utc-timestamp-string)))
+
+;;; ------------------------------------------------- Filter Transforms -------------------------------------------------
+
+(defn- matching-timestamp?
+  [job field-path {:keys [start end]}]
+  (when-let [field-instant (->instant (get-in job field-path))]
+    (let [start-instant (some-> start u.date/parse ->instant)
+          end-instant (some-> end u.date/parse ->instant)]
+      (and (or (nil? start)
+               (not (.isBefore field-instant start-instant)))
+           (or (nil? end)
+               (.isAfter end-instant field-instant))))))
+
+(defn ->date-field-filter-xf
+  "Returns an xform for a date filter."
+  [field-path filter-value]
+  (let [range (some-> filter-value (params.dates/date-string->range {:inclusive-end? false}))]
+    (if range
+      (filter #(matching-timestamp? % field-path range))
+      identity)))
+
+(defn ->status-filter-xf
+  "Returns an xform for a transform run status filter."
+  [field-path statuses]
+  (let [statuses (->> statuses (map keyword) set not-empty)]
+    (if statuses
+      (filter #(statuses (get-in % field-path)))
+      identity)))
+
+(defn ->tag-filter-xf
+  "Returns an xform for a transform tag filter."
+  [field-path tag-ids]
+  (let [tag-ids (-> tag-ids set not-empty)]
+    (if tag-ids
+      (filter #(some tag-ids (get-in % field-path)))
+      identity)))
+
+;;; ------------------------------------------------- Misc -------------------------------------------------
+
+(defenterprise is-temp-transform-table?
+  "Return true when `table` matches the transform temporary table naming pattern."
+  :feature :transforms
+  [table]
+  (when-let [table-name (:name table)]
+    (str/starts-with? (u/lower-case-en table-name) transform-temp-table-prefix)))
+
+(defn db-routing-enabled?
+  "Returns whether or not the given database is either a router or destination database"
+  [db-or-id]
+  (or (t2/exists? :model/DatabaseRouter :database_id (u/the-id db-or-id))
+      (some->> (:router-database-id db-or-id)
+               (t2/exists? :model/DatabaseRouter :database_id))))

enterprise/backend/src/metabase_enterprise/transforms_python/base.clj

@@ -0,0 +1,402 @@
+(ns metabase-enterprise.transforms-python.base
+  "Base Python transform execution - core logic without transform_run tracking.
+
+   This namespace handles Python transform execution and returns results in memory
+   rather than writing to transform_run rows. Logs are returned in the result map
+   instead of being saved to transform_run.message."
+  (:require
+   [clojure.core.async :as a]
+   [clojure.java.io :as io]
+   [clojure.string :as str]
+   [metabase-enterprise.transforms-base.interface :as transforms-base.i]
+   [metabase-enterprise.transforms-base.util :as transforms-base.util]
+   [metabase-enterprise.transforms-python.python-runner :as python-runner]
+   [metabase-enterprise.transforms-python.s3 :as s3]
+   [metabase-enterprise.transforms-python.settings :as transforms-python.settings]
+   [metabase-enterprise.transforms.instrumentation :as transforms.instrumentation]
+   [metabase.driver :as driver]
+   [metabase.driver.util :as driver.u]
+   [metabase.util :as u]
+   [metabase.util.format :as u.format]
+   [metabase.util.i18n :as i18n]
+   [metabase.util.log :as log]
+   [toucan2.core :as t2])
+  (:import
+   (java.io File)
+   (java.nio.file Files)
+   (java.nio.file.attribute FileAttribute)))
+
+(set! *warn-on-reflection* true)
+
+;;; ------------------------------------------------- Interface Implementations -------------------------------------------------
+
+(defmethod transforms-base.i/target-db-id :python
+  [transform]
+  (-> transform :target :database))
+
+(defmethod transforms-base.i/source-db-id :python
+  [transform]
+  (-> transform :source :database))
+
+(defn- source-table-value->dependency
+  "Convert a source table value (int or ref map) to a dependency map."
+  [v]
+  (cond
+    ;; Integer table ID - direct table dependency
+    (int? v)
+    {:table v}
+
+    ;; Map with resolved table_id - direct table dependency
+    (:table_id v)
+    {:table (:table_id v)}
+
+    ;; Map without table_id - return ref for ordering system to resolve
+    :else
+    {:table-ref (select-keys v [:database_id :schema :table])}))
+
+(defmethod transforms-base.i/table-dependencies :python
+  [transform]
+  (into #{}
+        (map source-table-value->dependency)
+        (vals (get-in transform [:source :source-tables]))))
+
+;;; ------------------------------------------------- Message Log (in-memory only) -------------------------------------------------
+
+(defn- empty-message-log
+  "Returns a new message log for accumulating execution logs in memory."
+  []
+  (atom {:pre-python  []
+         :python      nil
+         :post-python []}))
+
+(defn- log!
+  "Appends a string to the message log."
+  [message-log s]
+  (swap! message-log (fn [m] (update m (if (:python m) :post-python :pre-python) conj s)))
+  nil)
+
+(defn- replace-python-logs!
+  "Called when events output is received to copy those events into the message-log."
+  [message-log events]
+  (swap! message-log assoc :python events)
+  nil)
+
+(defn- message-log->string
+  "Presents the content of the log as a string."
+  [message-log]
+  (let [{:keys [pre-python python post-python]} @message-log]
+    (str/join "\n" (concat pre-python
+                           (map :message python)
+                           post-python))))
+
+;;; ------------------------------------------------- Transfer Strategies -------------------------------------------------
+
+(defn- table-schema [table-name metadata]
+  {:name (if (keyword? table-name) table-name (keyword table-name))
+   :columns (mapv (fn [{:keys [name base_type]}]
+                    {:name name
+                     :type (python-runner/restricted-insert-type base_type)
+                     :nullable? true})
+                  (:fields metadata))})
+
+(defn- maybe-retry-with-backoff
+  [driver thunk]
+  ;; bigquery can have eventual consistency hiccups where the table is not yet available for insertion
+  ;; immediately after creation, so we retry a few times with backoff
+  (if (= driver :bigquery-cloud-sdk)
+    (loop [attempt 0]
+      (let [ret (try
+                  (thunk)
+                  (catch Exception e
+                    (if (and (< attempt 5)
+                             (or (str/includes? (str e) "not found")
+                                 (str/includes? (str e) "Not found")))
+                      (let [delay-ms (* 150 (Math/pow 2 attempt))]
+                        (log/debugf "BigQuery operation failed (attempt %d), retrying in %dms" (inc attempt) delay-ms)
+                        (Thread/sleep (long delay-ms))
+                        ::recur)
+                      (throw e))))]
+        (if (= ::recur ret)
+          (recur (inc attempt))
+          ret)))
+    (thunk)))
+
+(defn- insert-data!
+  "Insert data from source into an existing table."
+  [driver db-id table-schema data-source]
+  (let [data-source (assoc data-source :table-schema table-schema)]
+    (->> #(driver/insert-from-source! driver db-id table-schema data-source)
+         (maybe-retry-with-backoff driver))))
+
+(defn- create-table-and-insert-data!
+  "Create a table from metadata and insert data from source."
+  [driver db-id table-schema data-source]
+  (transforms-base.util/create-table-from-schema! driver db-id table-schema)
+  (insert-data! driver db-id table-schema data-source))
+
+(defn- transfer-with-rename-tables-strategy!
+  "Transfer data using the rename-tables*! multimethod with atomicity guarantees."
+  [driver db-id table-name metadata data-source]
+  (let [source-table-name (transforms-base.util/temp-table-name driver (namespace table-name))
+        temp-table-name (u/poll {:thunk #(transforms-base.util/temp-table-name driver (namespace table-name))
+                                 :done? #(not= source-table-name %)
+                                 :interval-ms 1})]
+    (log/info "Using rename-tables strategy with atomicity guarantees")
+    (try
+      (create-table-and-insert-data! driver db-id (table-schema source-table-name metadata) data-source)
+      (transforms-base.util/rename-tables! driver db-id {table-name temp-table-name
+                                                         source-table-name table-name})
+      (transforms-base.util/drop-table! driver db-id temp-table-name)
+      (catch Exception e
+        (log/error e "Failed to transfer data using rename-tables strategy")
+        (try
+          (transforms-base.util/drop-table! driver db-id source-table-name)
+          (catch Exception _))
+        (throw e)))))
+
+(defn- transfer-with-create-drop-rename-strategy!
+  "Transfer data using create + drop + rename to minimize time without data."
+  [driver db-id table-name metadata data-source]
+  (let [source-table-name (transforms-base.util/temp-table-name driver (namespace table-name))]
+    (log/info "Using create-drop-rename strategy to minimize downtime")
+    (try
+      (create-table-and-insert-data! driver db-id (table-schema source-table-name metadata) data-source)
+      (transforms-base.util/drop-table! driver db-id table-name)
+      (driver/rename-table! driver db-id source-table-name table-name)
+      (catch Exception e
+        (log/error e "Failed to transfer data using create-drop-rename strategy")
+        (try
+          (transforms-base.util/drop-table! driver db-id source-table-name)
+          (catch Exception _))
+        (throw e)))))
+
+(defn- transfer-with-drop-create-fallback-strategy!
+  "Transfer data using drop + create fallback strategy."
+  [driver db-id table-name metadata data-source]
+  (log/info "Using drop-create fallback strategy")
+  (try
+    (transforms-base.util/drop-table! driver db-id table-name)
+    (create-table-and-insert-data! driver db-id (table-schema table-name metadata) data-source)
+    (catch Exception e
+      (log/error e "Failed to transfer data using drop-create fallback strategy")
+      (throw e))))
+
+(defmulti ^:private transfer-file-to-db
+  {:arglists '([driver db transform metadata temp-file])}
+  (fn [_ _ transform _ _] (-> transform :target :type keyword)))
+
+(defmethod transfer-file-to-db :table-incremental
+  [driver {db-id :id}
+   {:keys [target] :as transform}
+   metadata temp-file]
+  (let [table-name (transforms-base.util/qualified-table-name driver target)
+        table-exists? (transforms-base.util/target-table-exists? transform)
+        data-source {:type :jsonl-file
+                     :file temp-file}]
+    (if (not table-exists?)
+      (do
+        (log/info "New table")
+        (create-table-and-insert-data! driver db-id (table-schema table-name metadata) data-source))
+      (insert-data! driver db-id (table-schema table-name metadata) data-source))))
+
+(defmethod transfer-file-to-db :table
+  [driver {db-id :id :as db}
+   {:keys [target] :as transform}
+   metadata temp-file]
+  (let [table-name (transforms-base.util/qualified-table-name driver target)
+        table-exists? (transforms-base.util/target-table-exists? transform)
+        data-source {:type :jsonl-file
+                     :file temp-file}]
+    (cond
+      (not table-exists?)
+      (do
+        (log/info "New table")
+        (create-table-and-insert-data! driver db-id (table-schema table-name metadata) data-source))
+
+      (driver.u/supports? driver :atomic-renames db)
+      (transfer-with-rename-tables-strategy! driver db-id table-name metadata data-source)
+
+      (driver.u/supports? driver :rename db)
+      (transfer-with-create-drop-rename-strategy! driver db-id table-name metadata data-source)
+
+      :else
+      (transfer-with-drop-create-fallback-strategy! driver db-id table-name metadata data-source))))
+
+;;; ------------------------------------------------- Cancellation -------------------------------------------------
+
+(defn- start-cancellation-process!
+  "Starts a core.async process that sends a cancellation request to the python executor if cancel-chan receives a value."
+  [server-url run-id cancel-chan]
+  (a/go (when (a/<! cancel-chan)
+          (python-runner/cancel-python-code-http-call! server-url run-id))))
+
+;;; ------------------------------------------------- Core Execution -------------------------------------------------
+
+(defn- run-python-transform-impl!
+  "Core Python transform execution. Returns {:status :result :logs :events}.
+
+   Options:
+   - `with-stage-timing-fn` - optional, (fn [run-id stage thunk] result) for instrumentation"
+  [{:keys [source] :as transform} db run-id cancel-chan message-log {:keys [with-stage-timing-fn]}]
+  ;; Resolve name-based source table refs to table IDs (throws if any not found)
+  (let [resolved-source-tables (transforms-base.util/resolve-source-tables (:source-tables source))]
+    (with-open [shared-storage-ref (s3/open-shared-storage! resolved-source-tables)]
+      (let [driver          (:engine db)
+            server-url      (transforms-python.settings/python-runner-url)
+            _               (python-runner/copy-tables-to-s3! {:run-id         run-id
+                                                               :shared-storage @shared-storage-ref
+                                                               :source         (assoc source :source-tables resolved-source-tables)
+                                                               :cancel-chan    cancel-chan
+                                                               :limit          (:limit source)
+                                                               :transform-id   (:id transform)})
+            _               (start-cancellation-process! server-url run-id cancel-chan)
+            {:keys [status body] :as response}
+            (python-runner/execute-python-code-http-call!
+             {:server-url     server-url
+              :code           (:body source)
+              :run-id         run-id
+              :table-name->id resolved-source-tables
+              :shared-storage @shared-storage-ref})
+
+            output-manifest (python-runner/read-output-manifest @shared-storage-ref)
+            events          (python-runner/read-events @shared-storage-ref)]
+        (replace-python-logs! message-log events)
+        (if (not= 200 status)
+          (throw (ex-info "Python runner call failed"
+                          {:transform-message (i18n/tru "Python execution failure (exit code {0})" (:exit_code body "?"))
+                           :status-code       400
+                           :api-status-code   status
+                           :body              body
+                           :events            events
+                           :timeout           (:timeout body)}))
+          (try
+            (let [temp-path (Files/createTempFile "transform-output-" ".jsonl" (u/varargs FileAttribute))
+                  temp-file ^File (.toFile temp-path)]
+              (when-not (seq (:fields output-manifest))
+                (throw (ex-info "No fields in metadata"
+                                {:metadata               output-manifest
+                                 :raw-body               body
+                                 :events                 events})))
+              (try
+                (with-open [in (python-runner/open-output @shared-storage-ref)]
+                  (io/copy in temp-file))
+                ;; Transfer file to database with instrumentation
+                (let [file-size (.length temp-file)
+                      do-transfer (fn [] (transfer-file-to-db driver db transform output-manifest temp-file))]
+                  (if with-stage-timing-fn
+                    (with-stage-timing-fn run-id [:import :file-to-dwh] do-transfer)
+                    (do-transfer))
+                  (transforms.instrumentation/record-data-transfer! run-id :file-to-dwh file-size nil))
+                (finally
+                  (.delete temp-file))))
+            response
+            (catch Exception e
+              (log/error e "Failed to create resulting table")
+              (throw (ex-info "Failed to create the resulting table"
+                              {:transform-message (or (:transform-message (ex-data e))
+                                                      (i18n/tru "Failed to create the resulting table"))}
+                              e)))))))))
+
+;;; ------------------------------------------------- Base Execution Entry Point -------------------------------------------------
+
+(defn run-python-transform!
+  "Execute Python transform. Returns result map.
+
+   Does:
+   - Resolve source tables
+   - Copy to S3
+   - Call python runner
+   - Transfer result to DB
+   - Sync target
+
+   Does NOT:
+   - Create transform_run row
+   - Poll/save logs to transform_run.message
+
+   Options:
+   - `:cancelled?` - (fn [] boolean), polled to check for cancellation
+   - `:run-id` - optional, for cancellation signaling and instrumentation
+   - `:with-stage-timing-fn` - optional, (fn [run-id stage thunk] result)
+
+   Returns:
+   {:status :succeeded | :failed | :cancelled
+    :result <http response>
+    :logs <string>
+    :error <exception if failed>}"
+  [transform {:keys [cancelled? run-id with-stage-timing-fn]}]
+  (assert (transforms-base.util/python-transform? transform) "Transform must be a python transform")
+  (let [message-log (empty-message-log)]
+    (try
+      ;; Check cancellation before starting
+      (when (and cancelled? (cancelled?))
+        (throw (ex-info "Transform cancelled before start" {:status :cancelled})))
+
+      (let [{:keys [target] transform-id :id} transform
+            db (t2/select-one :model/Database (:database target))
+            ;; Use run-id if provided, otherwise generate a temp one for python runner
+            effective-run-id (or run-id (rand-int Integer/MAX_VALUE))
+            cancel-chan (a/promise-chan)
+            ;; Bridge cancelled? to cancel-chan
+            _ (when cancelled?
+                (a/go
+                  (loop []
+                    (when-not (a/poll! cancel-chan)
+                      (if (cancelled?)
+                        (a/>! cancel-chan :cancel!)
+                        (do
+                          (a/<! (a/timeout 100))
+                          (recur)))))))
+            start-ms (u/start-timer)]
+
+        (log! message-log (i18n/tru "Executing Python transform"))
+        (log/info "Executing Python transform" transform-id "with target" (pr-str target))
+
+        (let [result (run-python-transform-impl! transform db effective-run-id cancel-chan message-log
+                                                 {:with-stage-timing-fn with-stage-timing-fn})]
+          (log! message-log (i18n/tru "Python execution finished successfully in {0}"
+                                      (u.format/format-milliseconds (u/since-ms start-ms))))
+
+          ;; Check cancellation after python but before sync
+          (when (and cancelled? (cancelled?))
+            (throw (ex-info "Transform cancelled after python execution" {:status :cancelled})))
+
+          ;; Sync target table
+          (transforms-base.util/sync-target! target db)
+
+          ;; Create secondary indexes if needed
+          (transforms-base.util/execute-secondary-index-ddl-if-required!
+           transform run-id db target with-stage-timing-fn)
+
+          {:status :succeeded
+           :result result
+           :logs (message-log->string message-log)}))
+
+      (catch Exception e
+        (let [data (ex-data e)
+              logs (message-log->string message-log)
+              error-message (or (:transform-message data) (ex-message e))]
+          (cond
+            (= :cancelled (:status data))
+            {:status :cancelled
+             :error e
+             :logs logs}
+
+            (:timeout data)
+            {:status :timeout
+             :error e
+             :logs (or (when-let [events (:events data)]
+                         (str/join "\n" (map :message events)))
+                       logs)}
+
+            :else
+            (do
+              (log/error e "Error executing Python transform")
+              {:status :failed
+               :error e
+               :logs (str logs "\n" error-message)})))))))
+
+;;; ------------------------------------------------- Interface Implementation -------------------------------------------------
+
+(defmethod transforms-base.i/execute-base! :python
+  [transform opts]
+  (run-python-transform! transform opts))

enterprise/backend/src/metabase_enterprise/transforms_python/execute.clj

@@ -1,387 +1,134 @@
 (ns metabase-enterprise.transforms-python.execute
+  "Python transform implementation for scheduled execution.
+
+   This namespace provides the scheduled wrapper that creates transform_run rows,
+   tracks status, and saves logs. The actual execution logic is in transforms-base.python."
   (:require
-   [clojure.core.async :as a]
-   [clojure.java.io :as io]
    [clojure.string :as str]
+   [metabase-enterprise.transforms-base.util :as transforms-base.util]
    [metabase-enterprise.transforms-python.python-runner :as python-runner]
-   [metabase-enterprise.transforms-python.s3 :as s3]
-   [metabase-enterprise.transforms-python.settings :as transforms-python.settings]
-   [metabase-enterprise.transforms.core :as transforms]
-   [metabase-enterprise.transforms.instrumentation :as transforms.instrumentation]
    [metabase-enterprise.transforms.util :as transforms.util]
-   [metabase.app-db.core :as app-db]
-   [metabase.driver :as driver]
-   [metabase.driver.util :as driver.u]
-   [metabase.util :as u]
-   [metabase.util.format :as u.format]
-   [metabase.util.i18n :as i18n]
    [metabase.util.jvm :as u.jvm]
    [metabase.util.log :as log]
    [toucan2.core :as t2])
   (:import
    (java.io Closeable)
    (java.net SocketException)
-   (java.nio.file Files)
-   (java.nio.file.attribute FileAttribute)
    (java.time Duration)))
 
 (set! *warn-on-reflection* true)
 
-(defn- empty-message-log
-  "Returns a new message log.
-
-  Python transforms produce output while they run. The message log is a stateful structure that buffers log outputs during the run.
-  The goal is to assist in debugging in the case of python runtime or syntax errors, and to provide some immediate feedback for longer running transforms.
-
-  Important Note: in memory buffering is considered acceptable as we expect /logs output to be truncated or small.
-
-  The intended use of this structure is to produce a string suitable for `transform_run.message`, saving it every few seconds."
-  []
-  (atom {:pre-python  []                                    ; log! outputs previous to the python execution, i.e table read progress
-         :python      nil                                   ; events json structured logs from the /logs endpoint
-         :post-python []}))                                 ; log! outputs after the python execution, i.e. output reads, writes to target
-
-(defn- log!
-  "Appends a string to the message log, the string is user facing and should be suitable for presentation as part of the `transform_run.message` field."
-  [message-log s]
-  (swap! message-log (fn [m] (update m (if (:python m) :post-python :pre-python) conj s)))
-  nil)
-
-(defn- replace-python-logs!
-  "Called when events output is received from a /logs call to copy those events into the message-log."
-  [message-log events]
-  (swap! message-log assoc :python events)
-  nil)
-
-(defn- message-log->transform-run-message
-  "Presents the content of the log as a string suitable for storage in transform_run.message."
-  [message-log]
-  (let [{:keys [pre-python python post-python]} @message-log]
-    (str/join "\n" (concat pre-python
-                           (map :message python)
-                           post-python))))
+;;; ------------------------------------------------- Log Saving -------------------------------------------------
 
 (defn- save-log-to-transform-run-message!
-  "Saves the content of the log into the transform_run's message field. Lossy, string contains user facing information only."
-  [run-id message-log]
-  (t2/update! :model/TransformRun
-              :id run-id
-              {:message (message-log->transform-run-message message-log)}))
+  "Saves logs to the transform_run's message field."
+  [run-id logs]
+  (when (and run-id logs)
+    (t2/update! :model/TransformRun run-id {:message logs})))
+
+;;; ------------------------------------------------- Log Polling Loop -------------------------------------------------
 
 (def ^:private ^Duration python-message-loop-sleep-duration
-  (Duration/ofMillis 1000))
+  "How often to poll for log updates during Python execution.
+  Should be short enough to provide meaningful feedback during execution."
+  (Duration/ofMillis 50))
 
 (defn- python-message-update-loop!
-  "Block while relevant log data is replicated from the runner into the message log.
-  When new logs are received, the log data will be flushed to the transform_run.message field as a string.
+  "Block while relevant log data is replicated from the runner into the transform_run.message field.
+  The loop will poll the python runner /logs endpoint and save the logs to the database.
 
   The loop will exit on error or when interrupted."
-  [run-id message-log]
+  [run-id]
   (try
     (loop []
       (if (.isInterrupted (Thread/currentThread))
         (log/debug "Message update loop interrupted")
-        (do (let [sleep-ms (.toMillis python-message-loop-sleep-duration)]
-              (when (pos? sleep-ms) (Thread/sleep sleep-ms)))
-            (let [{:keys [status body]} (python-runner/get-logs run-id)]
-              (cond
-                (<= 200 status 299)
-                (let [{:keys [execution_id events]} body]
-                  (if-not (= run-id execution_id)
-                    (do (log/debugf "Run id did not match expected: %s actual: %s" run-id execution_id)
-                        (recur))
-                    (do
-                      (replace-python-logs! message-log events)
-                      (save-log-to-transform-run-message! run-id message-log)
-                      (recur))))
-                (= 404 status)
-                (do
-                  (log/debugf "No logs yet (or run finished), run-id: %s" run-id)
-                  (recur))
-                :else
-                (do
-                  (log/warnf "Unexpected status polling for logs %s %s, run-id: %s" status body run-id)
-                  (log/debug "Exiting due to poll error")))))))
+        (let [{:keys [status body]} (python-runner/get-logs run-id)]
+          (cond
+            (<= 200 status 299)
+            (let [{:keys [execution_id events]} body]
+              (if-not (= run-id execution_id)
+                (do (log/debugf "Run id did not match expected: %s actual: %s" run-id execution_id)
+                    (Thread/sleep (.toMillis python-message-loop-sleep-duration))
+                    (recur))
+                (let [message (str "Executing Python transform\n"
+                                   (->> events (map :message) (str/join "\n")))]
+                  (save-log-to-transform-run-message! run-id message)
+                  (Thread/sleep (.toMillis python-message-loop-sleep-duration))
+                  (recur))))
+            (= 404 status)
+            (do
+              (log/debugf "No logs yet (or run finished), run-id: %s" run-id)
+              (Thread/sleep (.toMillis python-message-loop-sleep-duration))
+              (recur))
+            :else
+            (do
+              (log/warnf "Unexpected status polling for logs %s %s, run-id: %s" status body run-id)
+              (log/debug "Exiting due to poll error"))))))
     (catch SocketException se (when-not (= "Closed by interrupt" (ex-message se)) (throw se)))
     (catch InterruptedException _)
     (catch Throwable e
       (log/errorf e "An exception was caught during msg update loop, run-id: %s" run-id))))
 
-(defn- open-python-message-update-future! ^Closeable [run-id message-log]
+(defn- open-python-message-update-future!
+  "Start a background thread that polls for logs and updates transform_run.message.
+  Returns a Closeable that should be closed when execution completes."
+  ^Closeable [run-id]
   (let [cleanup (fn [fut]
                   (future-cancel fut)
                   (if (= ::timeout (try (deref fut 10000 ::timeout) (catch Throwable _)))
                     (log/fatalf "Log polling task did not respond to interrupt, run-id: %s" run-id)
                     (log/debugf "Log polling task done, run-id: %s" run-id)))
         fut     (u.jvm/in-virtual-thread*
-                 (python-message-update-loop! run-id message-log))]
+                 (python-message-update-loop! run-id))]
     (reify Closeable
       (close [_] (cleanup fut)))))
 
-(defn- maybe-retry-with-backoff
-  [driver thunk]
-  ;; bigquery can have eventual consistency hiccups where the table is not yet available for insertion
-  ;; immediately after creation, so we retry a few times with backoff
-  (if (= driver :bigquery-cloud-sdk)
-    (loop [attempt 0]
-      (let [ret (try
-                  (thunk)
-                  (catch Exception e
-                    (if (and (< attempt 5)
-                             (or (str/includes? (str e) "not found")
-                                 (str/includes? (str e) "Not found")))
-                      (let [delay-ms (* 150 (Math/pow 2 attempt))]
-                        (log/debugf "BigQuery operation failed (attempt %d), retrying in %dms" (inc attempt) delay-ms)
-                        (Thread/sleep (long delay-ms))
-                        ::recur)
-                      (throw e))))]
-        (if (= ::recur ret)
-          (recur (inc attempt))
-          ret)))
-    (thunk)))
-
-(defn- table-schema [table-name metadata]
-  {:name (if (keyword? table-name) table-name (keyword table-name))
-   :columns (mapv (fn [{:keys [name base_type #_database_type]}]
-                    {:name name
-                     :type (python-runner/restricted-insert-type base_type)
-                     ;; :database-type database_type
-                     :nullable? true})
-                  (:fields metadata))})
-
-(defn- insert-data!
-  "Insert data from source into an existing table."
-  [driver db-id table-schema data-source]
-  (let [data-source (assoc data-source :table-schema table-schema)]
-    (->> #(driver/insert-from-source! driver db-id table-schema data-source)
-         (maybe-retry-with-backoff driver))))
-
-(defn- create-table-and-insert-data!
-  "Create a table from metadata and insert data from source."
-  [driver db-id table-schema data-source]
-  (transforms.util/create-table-from-schema! driver db-id table-schema)
-  (insert-data! driver db-id table-schema data-source))
-
-(defn- transfer-with-rename-tables-strategy!
-  "Transfer data using the rename-tables*! multimethod with atomicity guarantees.
-   Creates new table, then atomically renames target->old and new->target, then drops old."
-  [driver db-id table-name metadata data-source]
-  (let [source-table-name (transforms.util/temp-table-name driver (namespace table-name))
-        temp-table-name (u/poll {:thunk #(transforms.util/temp-table-name driver (namespace table-name))
-                                 :done? #(not= source-table-name %)
-                                 ;; Poll every 1ms to quickly generate a different timestamp-based table name
-                                 :interval-ms 1})]
-    (log/info "Using rename-tables strategy with atomicity guarantees")
-    (try
-
-      (create-table-and-insert-data! driver db-id (table-schema source-table-name metadata) data-source)
-      (transforms.util/rename-tables! driver db-id {table-name temp-table-name
-                                                    source-table-name table-name})
-      (transforms.util/drop-table! driver db-id temp-table-name)
-
-      (catch Exception e
-        (log/error e "Failed to transfer data using rename-tables strategy")
-        (try
-          (transforms.util/drop-table! driver db-id source-table-name)
-          (catch Exception _))
-        (throw e)))))
-
-(defn- transfer-with-create-drop-rename-strategy!
-  "Transfer data using create + drop + rename to minimize time without data.
-   Creates new table, drops old table, then renames new->target."
-  [driver db-id table-name metadata data-source]
-  (let [source-table-name (transforms.util/temp-table-name driver (namespace table-name))]
-    (log/info "Using create-drop-rename strategy to minimize downtime")
-    (try
-
-      (create-table-and-insert-data! driver db-id (table-schema source-table-name metadata) data-source)
-      (transforms.util/drop-table! driver db-id table-name)
-      (driver/rename-table! driver db-id source-table-name table-name)
-
-      (catch Exception e
-        (log/error e "Failed to transfer data using create-drop-rename strategy")
-        (try
-          (transforms.util/drop-table! driver db-id source-table-name)
-          (catch Exception _))
-        (throw e)))))
-
-(defn- transfer-with-drop-create-fallback-strategy!
-  "Transfer data using drop + create fallback strategy.
-   Drops old table, then creates new table directly with target name."
-  [driver db-id table-name metadata data-source]
-  (log/info "Using drop-create fallback strategy")
-  (try
-
-    (transforms.util/drop-table! driver db-id table-name)
-    (create-table-and-insert-data! driver db-id (table-schema table-name metadata) data-source)
-
-    (catch Exception e
-      (log/error e "Failed to transfer data using drop-create fallback strategy")
-      (throw e))))
-
-(defmulti ^:private transfer-file-to-db
-  {:arglists '([driver db transform metadata temp-file])}
-  (fn [_ _ transform _ _] (-> transform :target :type keyword)))
-
-(defmethod transfer-file-to-db :table-incremental
-  [driver {db-id :id}
-   {:keys [target] :as transform}
-   metadata temp-file]
-  (let [table-name (transforms.util/qualified-table-name driver target)
-        table-exists? (transforms.util/target-table-exists? transform)
-        data-source {:type :jsonl-file
-                     :file temp-file}]
-
-    ;; once we have more than just append, dispatch on :target-incremental-strategy
-
-    (if (not table-exists?)
-      (do
-        (log/info "New table")
-        (create-table-and-insert-data! driver db-id (table-schema table-name metadata) data-source))
-      (insert-data! driver db-id (table-schema table-name metadata) data-source))))
-
-(defmethod transfer-file-to-db :table
-  [driver {db-id :id :as db}
-   {:keys [target] :as transform}
-   metadata temp-file]
-  (let [table-name (transforms.util/qualified-table-name driver target)
-        table-exists? (transforms.util/target-table-exists? transform)
-        data-source {:type :jsonl-file
-                     :file temp-file}]
-    (cond
-      (not table-exists?)
-      (do
-        (log/info "New table")
-        (create-table-and-insert-data! driver db-id (table-schema table-name metadata) data-source))
-
-      (driver.u/supports? driver :atomic-renames db)
-      (transfer-with-rename-tables-strategy! driver db-id table-name metadata data-source)
-
-      (driver.u/supports? driver :rename db)
-      (transfer-with-create-drop-rename-strategy! driver db-id table-name metadata data-source)
-
-      :else
-      (transfer-with-drop-create-fallback-strategy! driver db-id table-name metadata data-source))))
-
-(defn- start-cancellation-process!
-  "Starts a core.async process that optimistically sends a cancellation request to the python executor if cancel-chan receives a value.
-  Returns a channel that will receive either the async http call j.u.c.FutureTask in the case of cancellation, or nil when the cancel-chan is closed."
-  [server-url run-id cancel-chan]
-  (a/go (when (a/<! cancel-chan)
-          (python-runner/cancel-python-code-http-call! server-url run-id))))
-
-(defn- run-python-transform! [{:keys [source] :as transform} db run-id cancel-chan message-log]
-  ;; Resolve name-based source table refs to table IDs (throws if any not found)
-  (let [resolved-source-tables (transforms.util/resolve-source-tables (:source-tables source))]
-    ;; TODO restructure things such that s3 can we swapped out for other transfer mechanisms
-    (with-open [^Closeable log-future-ref
-                (if (app-db/in-transaction?)
-                  ;; if in a transaction (such as under mt/with-temp), it is not safe to poll for logs (close race / contention)
-                  ;; tests that want to test async log behaviour should opt out of thread-local test helpers
-                  (reify Closeable (close [_]))
-                  (open-python-message-update-future! run-id message-log))
-                shared-storage-ref (s3/open-shared-storage! resolved-source-tables)]
-      (let [driver          (:engine db)
-            server-url      (transforms-python.settings/python-runner-url)
-            _               (python-runner/copy-tables-to-s3! {:run-id         run-id
-                                                               :shared-storage @shared-storage-ref
-                                                               :source         (assoc source :source-tables resolved-source-tables)
-                                                               :cancel-chan    cancel-chan
-                                                               :limit          (:limit source)
-                                                               :transform-id   (:id transform)})
-            _               (start-cancellation-process! server-url run-id cancel-chan) ; inherits lifetime of cancel-chan
-            {:keys [status body] :as response}
-            (python-runner/execute-python-code-http-call!
-             {:server-url     server-url
-              :code           (:body source)
-              :run-id         run-id
-              :table-name->id resolved-source-tables
-              :shared-storage @shared-storage-ref})
-
-            output-manifest (python-runner/read-output-manifest @shared-storage-ref)
-            events          (python-runner/read-events @shared-storage-ref)]
-        (.close ^Closeable log-future-ref)                 ; early close to force any writes to flush
-        (replace-python-logs! message-log events)
-        (if (not= 200 status)
-          (do
-            (when (:timeout body)
-              (transforms/timeout-run! run-id))
-            (throw (ex-info "Python runner call failed"
-                            {:transform-message (i18n/tru "Python execution failure (exit code {0})" (:exit_code body "?"))
-                             :status-code       400
-                             :api-status-code   status
-                             :body              body
-                             :events            events})))
-          (try
-            (let [temp-path (Files/createTempFile "transform-output-" ".jsonl" (u/varargs FileAttribute))
-                  temp-file (.toFile temp-path)]
-              (when-not (seq (:fields output-manifest))
-                (throw (ex-info "No fields in metadata"
-                                {:metadata               output-manifest
-                                 :raw-body               body
-                                 :events                 events})))
-              (try
-                (with-open [in (python-runner/open-output @shared-storage-ref)]
-                  (io/copy in temp-file))
-                (let [file-size (.length temp-file)]
-                  (transforms.instrumentation/with-stage-timing [run-id [:import :file-to-dwh]]
-                    (transfer-file-to-db driver db transform output-manifest temp-file))
-                  (transforms.instrumentation/record-data-transfer! run-id :file-to-dwh file-size nil))
-                (finally
-                  (.delete temp-file))))
-            response
-            (catch Exception e
-              (log/error e "Failed to to create resulting table")
-              (throw (ex-info "Failed to create the resulting table"
-                              {:transform-message (or (:transform-message (ex-data e))
-                                                    ;; TODO keeping messaging the same at this level
-                                                    ;;  should be more specific in underlying calls
-                                                      (i18n/tru "Failed to create the resulting table"))}
-                              e)))))))))
-
-(defn- exceptional-run-message [message-log ex]
-  (str/join "\n" (remove str/blank? [(message-log->transform-run-message message-log)
-                                     (or (:transform-message (ex-data ex))
-                                         (if (instance? InterruptedException ex)
-                                           (i18n/tru "Transform interrupted")
-                                           (i18n/tru "Something went wrong")))])))
+;;; ------------------------------------------------- Scheduled Execution Wrapper --------------------------------
 
 (defn execute-python-transform!
-  "Execute a Python transform by calling the python runner.
+  "Execute a Python transform with transform_run tracking.
 
-  Blocks until the transform returns."
+   This wrapper:
+   1. Creates a transform_run row via try-start-unless-already-running
+   2. Starts a background log polling loop for immediate feedback
+   3. Calls the base execution
+   4. Saves logs to transform_run.message
+   5. Updates transform_run status on completion/failure
+
+   Blocks until the transform returns."
   [transform {:keys [run-method start-promise user-id]}]
-  (assert (transforms.util/python-transform? transform) "Transform must be a python transform")
-  (try
-    (let [message-log (empty-message-log)
-          {:keys [target owner_user_id creator_id] transform-id :id} transform
-          {driver :engine :as db} (t2/select-one :model/Database (:database target))
-          ;; For manual runs, use the triggering user; for cron, use owner/creator
-          run-user-id (if (and (= run-method :manual) user-id)
-                        user-id
-                        (or owner_user_id creator_id))
-          {run-id :id} (transforms.util/try-start-unless-already-running transform-id run-method run-user-id)]
-      (some-> start-promise (deliver [:started run-id]))
-      (log! message-log (i18n/tru "Executing Python transform"))
-      (log/info "Executing Python transform" transform-id "with target" (pr-str target))
-      (let [start-ms          (u/start-timer)
-            transform-details {:db-id          (:id db)
-                               :transform-type (keyword (:type target))
-                               :conn-spec      (driver/connection-spec driver db)
-                               :output-schema  (:schema target)
-                               :output-table   (transforms.util/qualified-table-name driver target)}
-            run-fn            (fn [cancel-chan]
-                                (run-python-transform! transform db run-id cancel-chan message-log)
-                                (log! message-log (i18n/tru "Python execution finished successfully in {0}" (u.format/format-milliseconds (u/since-ms start-ms))))
-                                (save-log-to-transform-run-message! run-id message-log))
-            ex-message-fn     #(exceptional-run-message message-log %)
-            result            (transforms.instrumentation/with-stage-timing [run-id [:computation :python-execution]]
-                                (transforms.util/run-cancelable-transform! run-id driver transform-details run-fn :ex-message-fn ex-message-fn))]
-        (transforms.instrumentation/with-stage-timing [run-id [:import :table-sync]]
-          (transforms.util/sync-target! target db))
-        (transforms.util/execute-secondary-index-ddl-if-required! transform run-id db target)
-        {:run_id run-id
-         :result result}))
-    (catch Throwable t
-      (log/error t "Error executing Python transform")
-      (throw t))))
+  (assert (transforms-base.util/python-transform? transform) "Transform must be a python transform")
+  (let [*run-id (atom nil)]
+    (try
+      (let [{:keys [owner_user_id creator_id] transform-id :id} transform
+            ;; For manual runs, use the triggering user; for cron, use owner/creator
+            run-user-id  (if (and (= run-method :manual) user-id)
+                           user-id
+                           (or owner_user_id creator_id))
+            {run-id :id} (transforms.util/try-start-unless-already-running transform-id run-method run-user-id)]
+        (reset! *run-id run-id)
+
+        (some-> start-promise (deliver [:started run-id]))
+        (log/info "Executing Python transform" transform-id)
+
+        ;; Start log polling loop for immediate feedback during execution
+        (with-open [_log-poller (open-python-message-update-future! run-id)]
+          ;; Call the base execution - it handles all the work and returns results
+          (let [result (transforms.util/run-cancelable-transform!
+                        run-id transform
+                        {:ex-message-fn (fn [ex]
+                                          (or (:transform-message (ex-data ex))
+                                              (ex-message ex)))})]
+            ;; Save final logs (this overwrites the polled logs with the complete log)
+            (when-let [logs (:logs result)]
+              (save-log-to-transform-run-message! run-id logs))
+            {:run_id run-id
+             :result result})))
+      (catch Throwable t
+        ;; Save logs from exception data if available
+        (when @*run-id
+          (when-let [logs (:logs (ex-data t))]
+            (save-log-to-transform-run-message! @*run-id logs)))
+        (log/error t "Error executing Python transform")
+        (throw t)))))

enterprise/backend/src/metabase_enterprise/transforms_python/impl.clj

@@ -1,39 +1,16 @@
 (ns metabase-enterprise.transforms-python.impl
+  "Python transform interface implementation for scheduled execution.
+
+   The core interface methods (source-db-id, target-db-id, table-dependencies)
+   are implemented in transforms-python.base. This namespace registers the
+   scheduled execution method."
   (:require
+   ;; Load base implementations - registers source-db-id, target-db-id, table-dependencies, execute-base!
+   [metabase-enterprise.transforms-python.base]
    [metabase-enterprise.transforms-python.execute :as transforms-python.execute]
    [metabase-enterprise.transforms.interface :as transforms.i]))
 
-(defmethod transforms.i/target-db-id :python
-  [transform]
-  (-> transform :target :database))
-
-(defmethod transforms.i/source-db-id :python
-  [transform]
-  (-> transform :source :database))
-
 #_{:clj-kondo/ignore [:discouraged-var]}
 (defmethod transforms.i/execute! :python
   [transform options]
   (transforms-python.execute/execute-python-transform! transform options))
-
-(defn- source-table-value->dependency
-  "Convert a source table value (int or ref map) to a dependency map."
-  [v]
-  (cond
-    ;; Integer table ID - direct table dependency
-    (int? v)
-    {:table v}
-
-    ;; Map with resolved table_id - direct table dependency
-    (:table_id v)
-    {:table (:table_id v)}
-
-    ;; Map without table_id - return ref for ordering system to resolve
-    :else
-    {:table-ref (select-keys v [:database_id :schema :table])}))
-
-(defmethod transforms.i/table-dependencies :python
-  [transform]
-  (into #{}
-        (map source-table-value->dependency)
-        (vals (get-in transform [:source :source-tables]))))

enterprise/backend/src/metabase_enterprise/transforms_python/python_runner.clj

@@ -4,10 +4,10 @@
    [clojure.core.async :as a]
    [clojure.java.io :as io]
    [medley.core :as m]
+   [metabase-enterprise.transforms-base.util :as transforms-base.u]
    [metabase-enterprise.transforms-python.s3 :as s3]
    [metabase-enterprise.transforms-python.settings :as transforms-python.settings]
    [metabase-enterprise.transforms.instrumentation :as transforms.instrumentation]
-   [metabase-enterprise.transforms.util :as transforms.u]
    [metabase.analytics.prometheus :as prometheus]
    [metabase.config.core :as config]
    [metabase.lib-be.core :as lib-be]
@@ -274,7 +274,7 @@
         table-metadata    (lib.metadata/table metadata-provider table-id)
         transform         (t2/select-one :model/Transform transform-id)]
     (cond-> (lib/query metadata-provider table-metadata)
-      source-incremental-strategy (transforms.u/preprocess-incremental-query source-incremental-strategy (transforms.u/next-checkpoint transform))
+      source-incremental-strategy (transforms-base.u/preprocess-incremental-query source-incremental-strategy (transforms-base.u/next-checkpoint transform))
       limit                       (lib/limit limit))))
 
 ;; TODO break this up such that s3 can be swapped out for other transfer mechanisms.

enterprise/backend/test/metabase_enterprise/transforms/api_test.clj

@@ -3,6 +3,7 @@
   (:require
    [clojure.test :refer :all]
    [medley.core :as m]
+   [metabase-enterprise.transforms-base.util :as transforms-base.util]
    [metabase-enterprise.transforms.api]
    [metabase-enterprise.transforms.query-test-util :as query-test-util]
    [metabase-enterprise.transforms.test-dataset :as transforms-dataset]
@@ -10,7 +11,6 @@
                                                      parse-instant
                                                      utc-timestamp
                                                      with-transform-cleanup!]]
-   [metabase-enterprise.transforms.util :as transforms.util]
    [metabase.driver :as driver]
    [metabase.lib.core :as lib]
    [metabase.lib.metadata :as lib.metadata]
@@ -471,7 +471,7 @@
                         (m/dissoc-in [:source :query :lib/metadata]))
                     (-> (mt/user-http-request :crowberto :put 200 (format "ee/transform/%s" (:id resp)) updated)
                         (update-in [:source :query] lib/normalize))))
-            (is (false? (transforms.util/target-table-exists? original)))))))))
+            (is (false? (transforms-base.util/target-table-exists? original)))))))))
 
 (deftest delete-transforms-test
   (mt/test-drivers (mt/normal-drivers-with-feature :transforms/table)
@@ -592,7 +592,7 @@
                                                              original)
                     _                  (do (test-run transform-id)
                                            (wait-for-table table1-name 5000))
-                    _                  (is (true? (transforms.util/target-table-exists? original)))
+                    _                  (is (true? (transforms-base.util/target-table-exists? original)))
                     _                  (check-query-results table1-name [5 11 16] "Gadget")
                     updated            {:name        "Doohickey Products"
                                         :description "Desc"
@@ -606,8 +606,8 @@
                          (update-in [:source :query] lib/normalize))))
                 (test-run transform-id)
                 (wait-for-table table2-name 5000)
-                (is (true? (transforms.util/target-table-exists? original)))
-                (is (true? (transforms.util/target-table-exists? updated)))
+                (is (true? (transforms-base.util/target-table-exists? original)))
+                (is (true? (transforms-base.util/target-table-exists? updated)))
                 (check-query-results table2-name [2 3 4 13] "Doohickey")))))))))
 
 (deftest get-runs-filter-by-single-transform-id-test

enterprise/backend/test/metabase_enterprise/transforms/execute_test.clj

@@ -1,11 +1,11 @@
 (ns ^:mb/driver-tests metabase-enterprise.transforms.execute-test
   (:require
    [clojure.test :refer :all]
+   [metabase-enterprise.transforms-base.util :as transforms-base.util]
    [metabase-enterprise.transforms.execute :as transforms.execute]
    [metabase-enterprise.transforms.query-test-util :as query-test-util]
    [metabase-enterprise.transforms.test-dataset :as transforms-dataset]
    [metabase-enterprise.transforms.test-util :as transforms.tu :refer [with-transform-cleanup! delete-schema!]]
-   [metabase-enterprise.transforms.util :as transforms.util]
    [metabase.driver :as driver]
    [metabase.driver.sql-jdbc.connection :as sql-jdbc.conn]
    [metabase.lib.core :as lib]
@@ -138,7 +138,7 @@
                                   {:name "name" :type :type/Text :nullable? true}]}]
       (mt/as-admin
         (testing "create-table-from-schema! should create the table successfully"
-          (transforms.util/create-table-from-schema! driver db-id table-schema)
+          (transforms-base.util/create-table-from-schema! driver db-id table-schema)
           (let [table-exists? (driver/table-exists? driver db-id {:schema schema-name :name table-name})]
             (is (some? table-exists?) "Table should exist in the database schema")
             (driver/drop-table! driver db-id (:name table-schema))))))))

enterprise/backend/test/metabase_enterprise/transforms/incremental_test.clj

@@ -5,11 +5,11 @@
    [clojure.string :as str]
    [clojure.test :refer :all]
    [honey.sql :as sql]
+   [metabase-enterprise.transforms-base.util :as transforms-base.u]
    [metabase-enterprise.transforms-python.python-runner :as python-runner]
    [metabase-enterprise.transforms.execute :as transforms.execute]
    [metabase-enterprise.transforms.test-dataset :as transforms-dataset]
    [metabase-enterprise.transforms.test-util :as transforms.tu :refer [with-transform-cleanup!]]
-   [metabase-enterprise.transforms.util :as transforms.u]
    [metabase.driver :as driver]
    [metabase.driver.sql-jdbc.connection :as sql-jdbc.conn]
    [metabase.driver.sql.query-processor :as sql.qp]
@@ -184,8 +184,8 @@
     (some-> result :data :rows first first bigint)))
 
 (defn get-checkpoint-value [transform]
-  (#'transforms.u/next-checkpoint-value
-   (transforms.u/next-checkpoint transform)))
+  (#'transforms-base.u/next-checkpoint-value
+   (transforms-base.u/next-checkpoint transform)))
 
 (defn- compare-checkpoint-values
   "Compare two checkpoint values with type-appropriate logic. "
@@ -605,7 +605,7 @@
             (testing "sync has picked up table"
               (is (=? {:name target-table, :fields [{:name "id"}]} (-> (t2/select-one :model/Table :name target-table) (t2/hydrate :fields)))))
             (testing "checkpoint is recognized"
-              (is (some? (transforms.u/next-checkpoint transform))))))))))
+              (is (some? (transforms-base.u/next-checkpoint transform))))))))))
 
 (deftest checkpoint-field-does-not-exist-test
   (mt/test-drivers #{:postgres}                             ; no db specifics
@@ -632,7 +632,7 @@
             (testing "target table has expected data"
               (is (= [{:id 42}] (pg-table-rows db-spec target-table))))
             (testing "checkpoint is not recognized, so transform acts as if no checkpoint"
-              (is (nil? (transforms.u/next-checkpoint transform))))
+              (is (nil? (transforms-base.u/next-checkpoint transform))))
             ;; Maybe this is unrealistic - you cannot select a column that does not exist
             ;; But the source tables schema can change, e.g. rename: you change event_time to event_ts or something
             ;; so one would have to be careful to disable or delete transforms ahead of a schema change like this

enterprise/backend/test/metabase_enterprise/transforms/test_util.clj

@@ -83,55 +83,73 @@
   ^String [timestamp-string]
   (-> timestamp-string parse-instant str))
 
+(def ^:private max-poll-iterations
+  "Hard limit on polling iterations to prevent runaway loops in tests."
+  200)
+
 (defn wait-for-table
   "Wait for a table to appear in metadata, with timeout."
   [^String table-name timeout-ms]
   (let [timer (u/start-timer)]
-    (loop []
+    (loop [iterations 0]
       (let [table (t2/select-one :model/Table :name table-name)
             fields (t2/select :model/Field :table_id (:id table))]
         (cond
           (and table (seq fields)) table
+          (>= iterations max-poll-iterations)
+          (throw (ex-info (format "Table %s exceeded max poll iterations: %d" table-name max-poll-iterations)
+                          {:table-name table-name}))
           (> (u/since-ms timer) timeout-ms)
           (throw (ex-info (format "Table %s did not appear after %dms" table-name timeout-ms)
                           {:table-name table-name :timeout-ms timeout-ms}))
           :else (do (Thread/sleep 100)
-                    (recur)))))))
+                    (recur (inc iterations))))))))
 
-(defn test-run
+(defn test-run!
   [transform-id]
   (let [resp      (mt/user-http-request :crowberto :post 202 (format "ee/transform/%s/run" transform-id))
         timeout-s 10 ; 10 seconds is our timeout to finish execution and sync
         limit     (+ (System/currentTimeMillis) (* timeout-s 1000))]
     (is (=? {:message "Transform run started"}
             resp))
-    (loop [last-resp nil]
-      (when (> (System/currentTimeMillis) limit)
-        (throw (ex-info (str "Transform run timed out after " timeout-s " seconds") {:resp last-resp})))
-      (let [resp   (mt/user-http-request :crowberto :get 200 (format "ee/transform/%s" transform-id))
-            status (some-> resp :last_run :status keyword)]
-        (when-not (contains? #{:started :succeeded} status)
-          (throw (ex-info (str "Transform run failed with status " status) {:resp resp :status status})))
-        (when-not (some? (:table resp))
-          (Thread/sleep 100)
-          (recur resp))))))
+    ;; Suppress DEBUG logs during polling to avoid log spam
+    (mt/with-log-level [metabase.server.middleware.log :warn]
+      (loop [last-resp nil
+             iterations 0]
+        (when (>= iterations max-poll-iterations)
+          (throw (ex-info (str "Transform run exceeded max poll iterations: " max-poll-iterations) {:resp last-resp})))
+        (when (> (System/currentTimeMillis) limit)
+          (throw (ex-info (str "Transform run timed out after " timeout-s " seconds") {:resp last-resp})))
+        (let [resp   (mt/user-http-request :crowberto :get 200 (format "ee/transform/%s" transform-id))
+              status (some-> resp :last_run :status keyword)]
+          (when-not (contains? #{:started :succeeded} status)
+            (throw (ex-info (str "Transform run failed with status " status) {:resp resp :status status})))
+          ;; Wait for both table existence AND succeeded status
+          (when-not (and (some? (:table resp)) (= :succeeded status))
+            (Thread/sleep 100)
+            (recur resp (inc iterations))))))))
 
-(defn wait-for-transform-completion
+(defn wait-for-transform-completion!
   "Wait for a transform run to complete without triggering a new run.
    Polls the transform status until it succeeds or times out."
   [transform-id timeout-ms]
-  (let [start-time (u/start-timer)]
-    (loop []
-      (when (> (u/since-ms start-time) timeout-ms)
-        (throw (ex-info (format "Transform %d did not complete after %dms" transform-id timeout-ms)
-                        {:transform-id transform-id :timeout-ms timeout-ms})))
-      (let [resp (mt/user-http-request :crowberto :get 200 (format "ee/transform/%d" transform-id))
-            status (some-> resp :last_run :status keyword)]
-        (case status
-          :succeeded resp
-          (:started :running) (do (Thread/sleep 100) (recur))
-          (throw (ex-info (format "Transform run failed with status %s" status)
-                          {:resp resp :status status})))))))
+  ;; Suppress DEBUG logs during polling to avoid log spam
+  (mt/with-log-level [metabase.server.middleware.log :warn]
+    (let [start-time (u/start-timer)]
+      (loop [iterations 0]
+        (when (>= iterations max-poll-iterations)
+          (throw (ex-info (format "Transform %d exceeded max poll iterations: %d" transform-id max-poll-iterations)
+                          {:transform-id transform-id})))
+        (when (> (u/since-ms start-time) timeout-ms)
+          (throw (ex-info (format "Transform %d did not complete after %dms" transform-id timeout-ms)
+                          {:transform-id transform-id :timeout-ms timeout-ms})))
+        (let [resp (mt/user-http-request :crowberto :get 200 (format "ee/transform/%d" transform-id))
+              status (some-> resp :last_run :status keyword)]
+          (case status
+            :succeeded resp
+            (:started :running) (do (Thread/sleep 100) (recur (inc iterations)))
+            (throw (ex-info (format "Transform run failed with status %s" status)
+                            {:resp resp :status status}))))))))
 
 (defn get-test-schema
   "Get the schema from the products table in the test dataset.

enterprise/backend/test/metabase_enterprise/transforms_base/core_test.clj

@@ -0,0 +1,99 @@
+(ns ^:mb/driver-tests metabase-enterprise.transforms-base.core-test
+  "Tests for base transform execution without transform_run tracking.
+
+   These tests verify the core contract of transforms-base.core/execute!:
+   - Returns result in memory (not writing transform_run)
+   - Respects cancelled? callback
+   - Returns error info on failure"
+  (:require
+   [clojure.test :refer :all]
+   [metabase-enterprise.transforms-base.core :as transforms-base]
+   [metabase-enterprise.transforms.test-dataset :as transforms-dataset]
+   [metabase-enterprise.transforms.test-util :refer [with-transform-cleanup!]]
+   [metabase.lib.core :as lib]
+   [metabase.lib.metadata :as lib.metadata]
+   [metabase.test :as mt]
+   [toucan2.core :as t2]))
+
+(set! *warn-on-reflection* true)
+
+(deftest execute-returns-result-map-test
+  (testing "Base execute! returns result map and does NOT create transform_run"
+    (mt/test-drivers (mt/normal-drivers-with-feature :transforms/table)
+      (mt/with-premium-features #{:transforms}
+        (mt/dataset transforms-dataset/transforms-test
+          (let [schema (t2/select-one-fn :schema :model/Table (mt/id :transforms_products))]
+            (with-transform-cleanup! [target-table {:type   :table
+                                                    :schema schema
+                                                    :name   "base_exec_test"}]
+              (let [mp (mt/metadata-provider)
+                    transforms-products (lib.metadata/table mp (mt/id :transforms_products))
+                    products-category (lib.metadata/field mp (mt/id :transforms_products :category))
+                    query (-> (lib/query mp transforms-products)
+                              (lib/filter (lib/= products-category "Widget"))
+                              (lib/limit 5))]
+                (mt/with-temp [:model/Transform transform {:name   "base-test-transform"
+                                                           :source {:type  :query
+                                                                    :query query}
+                                                           :target target-table}]
+                  (let [transform-run-count-before (t2/count :model/TransformRun :transform_id (:id transform))
+                        result (transforms-base/execute! transform)
+                        transform-run-count-after (t2/count :model/TransformRun :transform_id (:id transform))]
+
+                    (testing "Returns result map with :status"
+                      (is (map? result))
+                      (is (= :succeeded (:status result))))
+
+                    (testing "Result contains :result key"
+                      (is (contains? result :result)))
+
+                    (testing "Does NOT create transform_run row"
+                      (is (= transform-run-count-before transform-run-count-after)
+                          "Base execute! should not write to transform_run table"))))))))))))
+
+(deftest execute-respects-cancelled-callback-test
+  (testing "Base execute! respects cancelled? callback"
+    (mt/test-drivers (mt/normal-drivers-with-feature :transforms/table)
+      (mt/with-premium-features #{:transforms}
+        (mt/dataset transforms-dataset/transforms-test
+          (let [schema (t2/select-one-fn :schema :model/Table (mt/id :transforms_products))]
+            (with-transform-cleanup! [target-table {:type   :table
+                                                    :schema schema
+                                                    :name   "cancel_test"}]
+              (let [mp (mt/metadata-provider)
+                    transforms-products (lib.metadata/table mp (mt/id :transforms_products))
+                    query (lib/query mp transforms-products)]
+                (mt/with-temp [:model/Transform transform {:name   "cancel-test-transform"
+                                                           :source {:type  :query
+                                                                    :query query}
+                                                           :target target-table}]
+
+                  (testing "When cancelled? returns true before execution, returns :cancelled"
+                    (let [result (transforms-base/execute! transform {:cancelled? (constantly true)})]
+                      (is (= :cancelled (:status result)))
+                      (is (some? (:error result))))))))))))))
+
+(deftest execute-returns-error-on-failure-test
+  (testing "Base execute! returns error info on failure"
+    (mt/test-drivers (mt/normal-drivers-with-feature :transforms/table)
+      (mt/with-premium-features #{:transforms}
+        (mt/dataset transforms-dataset/transforms-test
+          (let [schema (t2/select-one-fn :schema :model/Table (mt/id :transforms_products))]
+            (with-transform-cleanup! [target-table {:type   :table
+                                                    :schema schema
+                                                    :name   "error_test"}]
+              ;; Create a transform with invalid query (nonexistent table)
+              (mt/with-temp [:model/Transform transform {:name   "error-test-transform"
+                                                         :source {:type  :query
+                                                                  :query {:database (mt/id)
+                                                                          :type     :native
+                                                                          :native   {:query "SELECT * FROM nonexistent_table_xyz"}}}
+                                                         :target target-table}]
+                (let [result (transforms-base/execute! transform)]
+
+                  (testing "Returns :failed status"
+                    (is (= :failed (:status result))))
+
+                  (testing "Contains :error with exception"
+                    (is (some? (:error result)))
+                    (is (instance? Exception (:error result)))))))))))))

enterprise/backend/test/metabase_enterprise/transforms_base/ordering_test.clj

@@ -1,9 +1,9 @@
-(ns ^:mb/driver-tests metabase-enterprise.transforms.ordering-test
+(ns ^:mb/driver-tests metabase-enterprise.transforms-base.ordering-test
   (:require
    [clojure.test :refer :all]
+   [metabase-enterprise.transforms-base.interface :as transforms-base.i]
+   [metabase-enterprise.transforms-base.ordering :as ordering]
    [metabase-enterprise.transforms.execute :as transforms.execute]
-   [metabase-enterprise.transforms.interface :as transforms.i]
-   [metabase-enterprise.transforms.ordering :as ordering]
    [metabase.driver :as driver]
    [metabase.driver.sql :as driver.sql]
    [metabase.query-processor.compile :as qp.compile]
@@ -54,7 +54,7 @@
 
 (defn- transform-deps-for-db [transform]
   (mt/with-metadata-provider (mt/id)
-    (#'transforms.i/table-dependencies transform)))
+    (#'transforms-base.i/table-dependencies transform)))
 
 (deftest not-run-transform-dependency-ordering-test
   (mt/test-driver :postgres
@@ -386,7 +386,7 @@
                                                           :table "intermediate_output"}}
                                                "final_output")]
       (testing "table-dependencies returns table-ref for unresolved name reference"
-        (let [deps (transforms.i/table-dependencies (t2/select-one :model/Transform :id t-b))]
+        (let [deps (transforms-base.i/table-dependencies (t2/select-one :model/Transform :id t-b))]
           (is (contains? deps {:table-ref {:database_id (mt/id)
                                            :schema "public"
                                            :table "intermediate_output"}}))))
@@ -410,7 +410,7 @@
                                                                   :table "output_a"}}
                                                "output_b")]
       (testing "table-dependencies includes both types"
-        (let [deps (transforms.i/table-dependencies (t2/select-one :model/Transform :id t-b))]
+        (let [deps (transforms-base.i/table-dependencies (t2/select-one :model/Transform :id t-b))]
           (is (contains? deps {:table (mt/id :products)}))
           (is (contains? deps {:table-ref {:database_id (mt/id)
                                            :schema "public"

enterprise/backend/test/metabase_enterprise/transforms_base/util_test.clj

@@ -1,9 +1,9 @@
-(ns ^:mb/driver-tests metabase-enterprise.transforms.util-test
-  "Tests for transform utility functions."
+(ns ^:mb/driver-tests metabase-enterprise.transforms-base.util-test
+  "Tests for transform base utility functions."
   (:require
    [clojure.string :as str]
    [clojure.test :refer :all]
-   [metabase-enterprise.transforms.util :as transforms.util]
+   [metabase-enterprise.transforms-base.util :as transforms-base.util]
    [metabase.driver :as driver]
    [metabase.test :as mt]
    [metabase.test.data.sql :as sql.tx]))
@@ -16,7 +16,7 @@
       (let [driver driver/*driver*]
 
         (testing "Basic table name generation"
-          (let [result (transforms.util/temp-table-name driver nil)
+          (let [result (transforms-base.util/temp-table-name driver nil)
                 table-name (name result)]
             (is (keyword? result))
             (is (nil? (namespace result)))
@@ -24,7 +24,7 @@
             (is (re-matches #"mb_transform_temp_table_\d+" table-name))))
 
         (testing "Table name preserves namespace when present"
-          (let [result (transforms.util/temp-table-name driver "schema")]
+          (let [result (transforms-base.util/temp-table-name driver "schema")]
             (is (= "schema" (namespace result)))
             (is (str/starts-with? (name result) "mb_transform_temp_table_"))))))))
 
@@ -34,7 +34,7 @@
       (let [driver driver/*driver*
             db-id (mt/id)
 
-            table-name (transforms.util/temp-table-name driver nil)
+            table-name (transforms-base.util/temp-table-name driver nil)
             schema-name (when (get-method sql.tx/session-schema driver)
                           (sql.tx/session-schema driver))
             qualified-table-name (if schema-name
@@ -56,19 +56,19 @@
 
 (deftest is-temp-transform-tables-test
   (testing "tables with shcema"
-    (let [table-with-schema    {:name (name (transforms.util/temp-table-name :postgres "schema"))}
-          table-without-schema {:name (name (transforms.util/temp-table-name :postgres "schema"))}]
+    (let [table-with-schema    {:name (name (transforms-base.util/temp-table-name :postgres "schema"))}
+          table-without-schema {:name (name (transforms-base.util/temp-table-name :postgres "schema"))}]
       (mt/with-premium-features #{}
-        (is (false? (transforms.util/is-temp-transform-table? table-with-schema)))
-        (is (false? (transforms.util/is-temp-transform-table? table-without-schema))))
+        (is (false? (transforms-base.util/is-temp-transform-table? table-with-schema)))
+        (is (false? (transforms-base.util/is-temp-transform-table? table-without-schema))))
       (mt/with-premium-features #{:transforms}
-        (is (transforms.util/is-temp-transform-table? table-without-schema))
-        (is (transforms.util/is-temp-transform-table? table-with-schema)))))
+        (is (transforms-base.util/is-temp-transform-table? table-without-schema))
+        (is (transforms-base.util/is-temp-transform-table? table-with-schema)))))
 
   (testing "Ignores non-transform tables"
     (mt/with-premium-features #{:transforms}
-      (is (false? (transforms.util/is-temp-transform-table? {:name :orders})))
-      (is (false? (transforms.util/is-temp-transform-table? {:name :public/orders}))))))
+      (is (false? (transforms-base.util/is-temp-transform-table? {:name :orders})))
+      (is (false? (transforms-base.util/is-temp-transform-table? {:name :public/orders}))))))
 
 (deftest create-table-from-schema!-test
   (testing "create-table-from-schema! preserves column order from schema definition"
@@ -88,7 +88,7 @@
         (mt/as-admin
           (try
             (testing "Creating table with ordered columns"
-              (transforms.util/create-table-from-schema! driver db-id table-schema)
+              (transforms-base.util/create-table-from-schema! driver db-id table-schema)
               (is (driver/table-exists? driver (mt/db) {:schema schema-name :name (name table-name)})))
 
             (when (get-method driver/describe-table driver)
@@ -117,30 +117,30 @@
                    :model/Table    t1 {:db_id (:id db) :name "table_one" :schema nil}
                    :model/Table    t2 {:db_id (:id db) :name "table_two" :schema "my_schema"}]
       (testing "returns nil for empty input"
-        (is (nil? (transforms.util/batch-lookup-table-ids [])))
-        (is (nil? (transforms.util/batch-lookup-table-ids nil))))
+        (is (nil? (transforms-base.util/batch-lookup-table-ids [])))
+        (is (nil? (transforms-base.util/batch-lookup-table-ids nil))))
 
       (testing "looks up table without schema"
         (let [refs [{:database_id (:id db) :schema nil :table "table_one"}]
-              result (transforms.util/batch-lookup-table-ids refs)]
+              result (transforms-base.util/batch-lookup-table-ids refs)]
           (is (= {[(:id db) nil "table_one"] (:id t1)} result))))
 
       (testing "looks up table with schema"
         (let [refs [{:database_id (:id db) :schema "my_schema" :table "table_two"}]
-              result (transforms.util/batch-lookup-table-ids refs)]
+              result (transforms-base.util/batch-lookup-table-ids refs)]
           (is (= {[(:id db) "my_schema" "table_two"] (:id t2)} result))))
 
       (testing "handles mixed refs with and without schema"
         (let [refs [{:database_id (:id db) :schema nil :table "table_one"}
                     {:database_id (:id db) :schema "my_schema" :table "table_two"}]
-              result (transforms.util/batch-lookup-table-ids refs)]
+              result (transforms-base.util/batch-lookup-table-ids refs)]
           (is (= {[(:id db) nil "table_one"] (:id t1)
                   [(:id db) "my_schema" "table_two"] (:id t2)}
                  result))))
 
       (testing "returns empty for non-existent table"
         (let [refs [{:database_id (:id db) :schema nil :table "nonexistent"}]
-              result (transforms.util/batch-lookup-table-ids refs)]
+              result (transforms-base.util/batch-lookup-table-ids refs)]
           (is (= {} result)))))))
 
 (deftest normalize-source-tables-test
@@ -148,7 +148,7 @@
     (mt/with-temp [:model/Database db {}
                    :model/Table    t1 {:db_id (:id db) :name "existing_table" :schema nil}]
       (testing "converts integer table ID to map format"
-        (let [result (transforms.util/normalize-source-tables {"t" (:id t1)})]
+        (let [result (transforms-base.util/normalize-source-tables {"t" (:id t1)})]
           (is (map? (get result "t")))
           (is (= (:id db) (get-in result ["t" :database_id])))
           (is (= "existing_table" (get-in result ["t" :table])))
@@ -156,27 +156,27 @@
 
       (testing "throws for non-existent integer table ID"
         (is (thrown-with-msg? clojure.lang.ExceptionInfo #"Tables not found for ids: 999999"
-                              (transforms.util/normalize-source-tables {"t" 999999}))))
+                              (transforms-base.util/normalize-source-tables {"t" 999999}))))
 
       (testing "populates table_id for existing table"
         (let [source-tables {"t" {:database_id (:id db) :schema nil :table "existing_table"}}
-              result (transforms.util/normalize-source-tables source-tables)]
+              result (transforms-base.util/normalize-source-tables source-tables)]
           (is (= (:id t1) (get-in result ["t" :table_id])))))
 
       (testing "preserves existing table_id"
         (let [source-tables {"t" {:database_id (:id db) :schema nil :table "existing_table" :table_id 999}}
-              result (transforms.util/normalize-source-tables source-tables)]
+              result (transforms-base.util/normalize-source-tables source-tables)]
           (is (= 999 (get-in result ["t" :table_id])))))
 
       (testing "leaves table_id nil for non-existent table ref"
         (let [source-tables {"t" {:database_id (:id db) :schema nil :table "nonexistent"}}
-              result (transforms.util/normalize-source-tables source-tables)]
+              result (transforms-base.util/normalize-source-tables source-tables)]
           (is (nil? (get-in result ["t" :table_id])))))
 
       (testing "handles mixed int and map entries"
         (let [source-tables {"t1" (:id t1)
                              "t2" {:database_id (:id db) :schema nil :table "existing_table"}}
-              result (transforms.util/normalize-source-tables source-tables)]
+              result (transforms-base.util/normalize-source-tables source-tables)]
           (is (map? (get result "t1")))
           (is (= (:id t1) (get-in result ["t1" :table_id])))
           (is (= (:id t1) (get-in result ["t2" :table_id]))))))))
@@ -187,28 +187,28 @@
                    :model/Table    t1 {:db_id (:id db) :name "table_one" :schema nil}
                    :model/Table    t2 {:db_id (:id db) :name "table_two" :schema nil}]
       (testing "passes through integer entries (old format)"
-        (is (= {"t" 123} (transforms.util/resolve-source-tables {"t" 123}))))
+        (is (= {"t" 123} (transforms-base.util/resolve-source-tables {"t" 123}))))
 
       (testing "resolves map with table_id"
         (let [source-tables {"t" {:database_id (:id db) :schema nil :table "table_one" :table_id (:id t1)}}]
-          (is (= {"t" (:id t1)} (transforms.util/resolve-source-tables source-tables)))))
+          (is (= {"t" (:id t1)} (transforms-base.util/resolve-source-tables source-tables)))))
 
       (testing "looks up table_id for map without it"
         (let [source-tables {"t" {:database_id (:id db) :schema nil :table "table_one"}}]
-          (is (= {"t" (:id t1)} (transforms.util/resolve-source-tables source-tables)))))
+          (is (= {"t" (:id t1)} (transforms-base.util/resolve-source-tables source-tables)))))
 
       (testing "throws for non-existent table"
         (let [source-tables {"t" {:database_id (:id db) :schema nil :table "nonexistent"}}]
           (is (thrown-with-msg? clojure.lang.ExceptionInfo #"Tables not found: nonexistent"
-                                (transforms.util/resolve-source-tables source-tables)))))
+                                (transforms-base.util/resolve-source-tables source-tables)))))
 
       (testing "throws with schema in error message"
         (let [source-tables {"t" {:database_id (:id db) :schema "my_schema" :table "nonexistent"}}]
           (is (thrown-with-msg? clojure.lang.ExceptionInfo #"Tables not found: my_schema\.nonexistent"
-                                (transforms.util/resolve-source-tables source-tables)))))
+                                (transforms-base.util/resolve-source-tables source-tables)))))
 
       (testing "handles mixed entries (old and new format)"
         (let [source-tables {"t1" (:id t1)
                              "t2" {:database_id (:id db) :schema nil :table "table_two"}}]
           (is (= {"t1" (:id t1) "t2" (:id t2)}
-                 (transforms.util/resolve-source-tables source-tables))))))))
+                 (transforms-base.util/resolve-source-tables source-tables))))))))

enterprise/backend/test/metabase_enterprise/transforms_python/drivers_test.clj

@@ -3,11 +3,11 @@
   (:require
    [clojure.string :as str]
    [clojure.test :refer :all]
+   [metabase-enterprise.transforms-base.util :as transforms-base.util]
    [metabase-enterprise.transforms-python.execute :as transforms-python.execute]
    [metabase-enterprise.transforms-python.python-runner :as python-runner]
    [metabase-enterprise.transforms-python.python-runner-test :as python-runner-test]
    [metabase-enterprise.transforms.test-util :as transforms.tu :refer [with-transform-cleanup!]]
-   [metabase-enterprise.transforms.util :as transforms.util]
    [metabase.driver :as driver]
    [metabase.driver.mysql :as mysql]
    [metabase.sync.core :as sync]
@@ -120,7 +120,7 @@
         table-schema {:name qualified-table-name
                       :columns (:columns schema)}]
     (mt/as-admin
-      (transforms.util/create-table-from-schema! driver db-id table-schema))
+      (transforms-base.util/create-table-from-schema! driver db-id table-schema))
 
     (when (seq data)
       (driver/insert-from-source! driver db-id table-schema

enterprise/backend/test/metabase_enterprise/transforms_python/e2e_test.clj

@@ -49,7 +49,7 @@
                                          :target (assoc target :database (mt/id))}
                       {transform-id :id} (mt/user-http-request :crowberto :post 200 "ee/transform"
                                                                transform-payload)]
-                  (transforms.tu/test-run transform-id)
+                  (transforms.tu/test-run! transform-id)
                   (transforms.tu/wait-for-table table-name 5000)
                   (is (true? (driver/table-exists? driver/*driver* (mt/db) target)))
 
@@ -79,7 +79,7 @@
                                         :target  (assoc target :database (mt/id))}
                     {transform-id :id} (mt/user-http-request :crowberto :post 200 "ee/transform"
                                                              original)]
-                (transforms.tu/test-run transform-id)
+                (transforms.tu/test-run! transform-id)
                 (transforms.tu/wait-for-table table-name 5000)
                 (is (true? (driver/table-exists? driver/*driver* (mt/db) target)))
                 (is (= [["Alice" 25] ["Bob" 30]]

enterprise/backend/test/metabase_enterprise/transforms_python/execute_test.clj

@@ -1,10 +1,10 @@
 (ns ^:mb/driver-tests ^:mb/transforms-python-test metabase-enterprise.transforms-python.execute-test
   (:require
    [clojure.test :refer :all]
+   [metabase-enterprise.transforms-base.util :as transforms-base.util]
    [metabase-enterprise.transforms-python.execute :as transforms-python.execute]
    [metabase-enterprise.transforms.test-dataset :as transforms-dataset]
    [metabase-enterprise.transforms.test-util :as transforms.tu :refer [with-transform-cleanup!]]
-   [metabase-enterprise.transforms.util :as transforms.util]
    [metabase.test :as mt]
    [metabase.test.util :as test.util]
    [toucan2.core :as t2])
@@ -46,10 +46,10 @@
                                                      "    return pd.DataFrame({'name': ['Charlie', 'Diana', 'Eve'], 'age': [35, 40, 45]})")}}))
 
                   (let [swap-latch (CountDownLatch. 1)
-                        original-rename-tables-atomic! transforms.util/rename-tables!]
-                    (with-redefs [transforms.util/rename-tables! (fn [driver db-id rename-pairs]
-                                                                   (.await swap-latch)
-                                                                   (original-rename-tables-atomic! driver db-id rename-pairs))]
+                        original-rename-tables-atomic! transforms-base.util/rename-tables!]
+                    (with-redefs [transforms-base.util/rename-tables! (fn [driver db-id rename-pairs]
+                                                                        (.await swap-latch)
+                                                                        (original-rename-tables-atomic! driver db-id rename-pairs))]
                       (let [transform-future (future
                                                (transforms-python.execute/execute-python-transform!
                                                 (t2/select-one :model/Transform (:id transform))
@@ -88,7 +88,7 @@
 
                   (let [db-id (mt/id)
                         tables (t2/select :model/Table :db_id db-id :active true)]
-                    (is (not-any? transforms.util/is-temp-transform-table? tables)
+                    (is (not-any? transforms-base.util/is-temp-transform-table? tables)
                         "No temp tables should remain after successful Python transform")
 
                     (is (= [[1 "a"] [2 "b"] [3 "c"]] (transforms.tu/table-rows table-name))

enterprise/backend/test/metabase_enterprise/transforms_python/ordering_test.clj

@@ -1,8 +1,8 @@
 (ns ^:mb/driver-tests ^:mb/transforms-python-test metabase-enterprise.transforms-python.ordering-test
   (:require
    [clojure.test :refer :all]
-   [metabase-enterprise.transforms.interface :as transforms.i]
-   [metabase-enterprise.transforms.ordering :as ordering]
+   [metabase-enterprise.transforms-base.interface :as transforms-base.i]
+   [metabase-enterprise.transforms-base.ordering :as ordering]
    [metabase.driver :as driver]
    [metabase.driver.sql :as driver.sql]
    [metabase.test :as mt]
@@ -37,7 +37,7 @@
 
 (defn- transform-deps-for-db [transform]
   (mt/with-metadata-provider (mt/id)
-    (#'transforms.i/table-dependencies transform)))
+    (#'transforms-base.i/table-dependencies transform)))
 
 (deftest python-transform-basic-dependencies-test
   (testing "Python transforms with source-tables dependencies are extracted correctly"

enterprise/backend/test/metabase_enterprise/transforms_python/python_runner_test.clj

@@ -5,10 +5,10 @@
    [clojure.string :as str]
    [clojure.test :refer :all]
    [java-time.api :as t]
+   [metabase-enterprise.transforms-base.util :as transforms-base.util]
    [metabase-enterprise.transforms-python.python-runner :as python-runner]
    [metabase-enterprise.transforms-python.s3 :as s3]
    [metabase-enterprise.transforms-python.settings :as transforms-python.settings]
-   [metabase-enterprise.transforms.util :as transforms.util]
    [metabase.driver :as driver]
    [metabase.driver.sql-jdbc.connection :as sql-jdbc.conn]
    [metabase.sync.core :as sync]
@@ -452,7 +452,7 @@
                                     {:name "created_date"  :type :type/Date     :nullable? true}
                                     {:name "description"   :type :type/Text     :nullable? true}]}
             _ (mt/as-admin
-                (transforms.util/create-table-from-schema! driver db-id table-schema))
+                (transforms-base.util/create-table-from-schema! driver db-id table-schema))
 
             row-values   [[1
                            19.99

enterprise/backend/test/metabase_enterprise/transforms_python/transforms_api_test.clj

@@ -2,6 +2,7 @@
   (:require
    [clojure.string :as str]
    [clojure.test :refer :all]
+   [metabase-enterprise.transforms-python.base :as transforms-python.base]
    [metabase-enterprise.transforms-python.execute :as transforms-python.execute]
    [metabase-enterprise.transforms-python.init]
    [metabase-enterprise.transforms-python.python-runner :as transforms-python.python-runner]
@@ -22,63 +23,65 @@
 
 (deftest create-python-transform-test
   (mt/test-drivers (mt/normal-drivers-with-feature :transforms/table)
-    (mt/dataset transforms-dataset/transforms-test
-      (letfn [(create-transform! []
-                (let [schema            (get-test-schema)
-                      transform-payload {:name   "My beautiful python runner"
-                                         :source {:type          "python"
-                                                  :body          "print('hello world')"
-                                                  :source-tables {}}
-                                         :target {:type     "table"
-                                                  :schema   schema
-                                                  :name     "gadget_products"
-                                                  :database (mt/id)}}]
-                  (mt/user-http-request :crowberto :post "ee/transform"
-                                        transform-payload)))]
+    (mt/with-log-level [metabase.driver.sql-jdbc.sync.describe-table :error]
+      (mt/with-log-level [metabase.server.middleware.log :info]
+        (mt/dataset transforms-dataset/transforms-test
+          (letfn [(create-transform! []
+                    (let [schema            (get-test-schema)
+                          transform-payload {:name   "My beautiful python runner"
+                                             :source {:type          "python"
+                                                      :body          "print('hello world')"
+                                                      :source-tables {}}
+                                             :target {:type     "table"
+                                                      :schema   schema
+                                                      :name     "gadget_products"
+                                                      :database (mt/id)}}]
+                      (mt/user-http-request :crowberto :post "ee/transform"
+                                            transform-payload)))]
 
-        (testing "without any feature flags"
-          (mt/with-premium-features #{}
-            (testing "creating python transform without any features fails"
-              (is (= "error-premium-feature-not-available"
-                     (:status (mt/user-http-request :crowberto :post 402 "ee/transform"
-                                                    {:name   "My beautiful python runner"
-                                                     :source {:type          "python"
-                                                              :body          "print('hello world')"
-                                                              :source-tables {}}
-                                                     :target {:type     "table"
-                                                              :schema   (get-test-schema)
-                                                              :name     "gadget_products"
-                                                              :database (mt/id)}})))))))
+            (testing "without any feature flags"
+              (mt/with-premium-features #{}
+                (testing "creating python transform without any features fails"
+                  (is (= "error-premium-feature-not-available"
+                         (:status (mt/user-http-request :crowberto :post 402 "ee/transform"
+                                                        {:name   "My beautiful python runner"
+                                                         :source {:type          "python"
+                                                                  :body          "print('hello world')"
+                                                                  :source-tables {}}
+                                                         :target {:type     "table"
+                                                                  :schema   (get-test-schema)
+                                                                  :name     "gadget_products"
+                                                                  :database (mt/id)}})))))))
 
-        (testing "with only transforms feature flag (no transforms-python)"
-          (mt/with-premium-features #{:transforms}
-            (testing "creating python transform without transforms-python feature fails"
-              (is (= "Premium features required for this transform type are not enabled."
-                     (mt/user-http-request :crowberto :post 402 "ee/transform"
-                                           {:name   "My beautiful python runner"
-                                            :source {:type          "python"
-                                                     :body          "print('hello world')"
-                                                     :source-tables {}}
-                                            :target {:type     "table"
-                                                     :schema   (get-test-schema)
-                                                     :name     "gadget_products"
-                                                     :database (mt/id)}}))))))
+            (testing "with only transforms feature flag (no transforms-python)"
+              (mt/with-premium-features #{:transforms}
+                (testing "creating python transform without transforms-python feature fails"
+                  (is (= "Premium features required for this transform type are not enabled."
+                         (mt/user-http-request :crowberto :post 402 "ee/transform"
+                                               {:name   "My beautiful python runner"
+                                                :source {:type          "python"
+                                                         :body          "print('hello world')"
+                                                         :source-tables {}}
+                                                :target {:type     "table"
+                                                         :schema   (get-test-schema)
+                                                         :name     "gadget_products"
+                                                         :database (mt/id)}}))))))
 
-        (testing "with transforms-python feature flag"
-          (mt/with-premium-features #{:transforms :transforms-python}
-            (with-transform-cleanup! [table-name "gadget_products"]
-              (let [transform         (create-transform!)]
-                (is (= "print('hello chris')"
-                       (-> (mt/user-http-request :crowberto :put 200 (format "ee/transform/%s" (:id transform))
-                                                 {:name   "My beautiful python runner"
-                                                  :source {:type          "python"
-                                                           :body          "print('hello chris')"
-                                                           :source-tables {}}
-                                                  :target {:type     "table"
-                                                           :schema   (get-test-schema)
-                                                           :name     table-name
-                                                           :database (mt/id)}})
-                           :source :body)))))))))))
+            (testing "with transforms-python feature flag"
+              (mt/with-premium-features #{:transforms :transforms-python}
+                (with-transform-cleanup! [table-name "gadget_products"]
+                  (let [transform (create-transform!)]
+                    (is (= "print('hello chris')"
+                           (-> (mt/user-http-request :crowberto :put 200 (format "ee/transform/%s" (:id transform))
+                                                     {:name   "My beautiful python runner"
+                                                      :source {:type          "python"
+                                                               :body          "print('hello chris')"
+                                                               :source-tables {}}
+                                                      :target {:type     "table"
+                                                               :schema   (get-test-schema)
+                                                               :name     table-name
+                                                               :database (mt/id)}})
+                               :source :body)))))))))))))
 
 (deftest update-python-transform-feature-flag-test
   (mt/test-drivers (mt/normal-drivers-with-feature :transforms/table)
@@ -144,7 +147,7 @@
                                                              "    return pd.DataFrame({'name': ['Alice', 'Bob'], 'age': [25, 30]})")}
                                         :target  (assoc target :database (mt/id))}
                     {transform-id :id} (mt/user-http-request :crowberto :post 200 "ee/transform" original)]
-                (transforms.tu/test-run transform-id)
+                (transforms.tu/test-run! transform-id)
                 (transforms.tu/wait-for-table table-name 10000)
                 (is (true? (driver/table-exists? driver/*driver* (mt/db) target)))
                 (is (= [["Alice" 25] ["Bob" 30]]
@@ -215,7 +218,7 @@
 
           (block-on-run [{:keys [expect-status]} target transform-id]
             (try
-              (transforms.tu/test-run transform-id)
+              (transforms.tu/test-run! transform-id)
               (catch Throwable e
                 ;; test-run throws for non success, but we want to test failures too
                 (when-not (= expect-status (:status (ex-data e)))
@@ -225,9 +228,9 @@
 
           (run-scenario [scenario schema]
             (with-redefs [transforms-python.execute/python-message-loop-sleep-duration (Duration/ofMillis fast-log-polling-ms)
-                          transforms-python.execute/transfer-file-to-db                (if-some [e (:writeback-ex scenario)]
+                          transforms-python.base/transfer-file-to-db                   (if-some [e (:writeback-ex scenario)]
                                                                                          (fn [& _] (throw e))
-                                                                                         @#'transforms-python.execute/transfer-file-to-db)]
+                                                                                         @#'transforms-python.base/transfer-file-to-db)]
               (with-transform-cleanup! [target {:type   "table"
                                                 :schema schema
                                                 :name   "result"}]
@@ -371,7 +374,7 @@
                  (fn [os fields-meta col-meta]
                    (rf-proxy ready-signal wait-signal (f os fields-meta col-meta)))})
               :write
-              (let [f-ref #'transforms-python.execute/transfer-file-to-db
+              (let [f-ref #'transforms-python.base/transfer-file-to-db
                     f     @f-ref]
                 {f-ref
                  (fn [& args]
@@ -478,27 +481,27 @@
 
                           (when (some? expect-write)
                             (testing "table existence"
-                              (is (= expect-write (driver/table-exists? driver/*driver* (mt/db) target)))))))))
+                              (is (= expect-write (driver/table-exists? driver/*driver* (mt/db) target))))))))
 
                   ; todo We have not yet covered the case where we rerun the same transform while there might be some hangover.
                   ;      Cancellation addresses the transform and not the run, there is shared mutable state and races on it are possible
-                  (testing "the runner is not blocked for a new run"
-                    (with-transform-cleanup! [target {:type   "table"
-                                                      :schema schema
-                                                      :name   "result"}]
-                      (let [{transform-id :id} (create-transform {:desc "normal run", :program non-blocking-script} target)
-                            _          (transforms.tu/test-run transform-id)
-                            {:keys [status start_time end_time]} (get-last-run transform-id)
-                            start-inst (some-> start_time Instant/parse)
-                            end-inst   (some-> end_time Instant/parse)
-                            duration   (when (and start-inst end-inst) (Duration/between start-inst end-inst))]
-                        (is (= "succeeded" status))
-                        (when duration
+                    (testing "the runner is not blocked for a new run"
+                      (with-transform-cleanup! [target {:type   "table"
+                                                        :schema schema
+                                                        :name   "result"}]
+                        (let [{transform-id :id} (create-transform {:desc "normal run", :program non-blocking-script} target)
+                              _          (transforms.tu/test-run! transform-id)
+                              {:keys [status start_time end_time]} (get-last-run transform-id)
+                              start-inst (some-> start_time Instant/parse)
+                              end-inst   (some-> end_time Instant/parse)
+                              duration   (when (and start-inst end-inst) (Duration/between start-inst end-inst))]
+                          (is (= "succeeded" status))
+                          (when duration
                           ;; 10 seconds is an estimate of the maximum time we should expect the normal run to take.
                           ;; if the runner was blocked for 30 seconds, and only then did the new run get scheduled - we would exceed this time.
-                          (is (< (.toSeconds duration) 10)))))))))))))))
+                            (is (< (.toSeconds duration) 10))))))))))))))))
 
-(deftest python-transform-schema-change-integration-test
+(deftest python-transform-schema-change-integration-test!
   (testing "Python transform handles schema changes using appropriate rename strategy"
     (mt/test-drivers (disj (mt/normal-drivers-with-feature :transforms/python)
                            ;; takes too long on CI
@@ -522,7 +525,7 @@
                     {transform-id :id} (mt/user-http-request :crowberto :post 200 "ee/transform" initial-transform)]
 
                 ;; Run initial transform and validate
-                (transforms.tu/test-run transform-id)
+                (transforms.tu/test-run! transform-id)
                 (transforms.tu/wait-for-table table-name 10000)
                 (let [initial-rows (transforms.tu/table-rows table-name)]
                   (is (= [["Alice" 25] ["Bob" 30]] initial-rows) "Initial data should be Alice and Bob with ages"))
@@ -540,8 +543,8 @@
                   (is (some? update-response) "Transform update should succeed"))
 
                 ;; Run updated transform and validate schema change
-                (transforms.tu/test-run transform-id)
-                (transforms.tu/wait-for-transform-completion transform-id 10000)
+                (transforms.tu/test-run! transform-id)
+                (transforms.tu/wait-for-transform-completion! transform-id 10000)
 
                 ;; hmmm, looks like QP needs a bit more time to update metadata
                 (Thread/sleep 2000)

src/metabase/sync/sync_metadata/tables.clj

@@ -84,7 +84,7 @@
 
 (defenterprise is-temp-transform-table?
   "Return true if `table` references a temporary transform table created during transforms execution."
-  metabase-enterprise.transforms.util
+  metabase-enterprise.transforms-base.util
   [_table]
   false)