Files
metabase/resources/migrations/060_update_migrations.yaml
William 8ceeb1fce4 source replacement (#69257)
* fixes

* test parser changes

* schema test

* fix linter

* implicit join error

* fe tests

* fix linter

* test fixes

* more tests

* tests

* upgrade field refs for dashboardcard_series and dashboards in transitive deps

- Add dashboard-upgrade-field-refs! for Phase 1b: upgrades parameter_mappings
  on dashboards found in transitive deps (cards are upgraded first in Phase 1)
- Fix dashboard-card-update-field-refs! to find dashcards via
  DashboardCardSeries, not just by primary card_id
- Split dashboard-ids out of transitive in runner, pass separately to run-swap*
- Add tests for series card parameter mapping swap and dedup

* fixup some tests, add promise to progress instance

useful for test so we don't have to loop on the db row

```clojure
     (reify replacement.protocols/IRunnerProgress
       (set-total! [_ total] ...)
       (advance! [this] ...)
       (advance! [this n]
         (let [c    (swap! completed* + n)
               t    @total*
               prev (- c n)
               crossed-boundary? (or (= c t)
                                     (not= (quot prev progress-batch-size)
                                           (quot c progress-batch-size)))]
           (when crossed-boundary?
             (let [progress (if (pos? t) (double (/ c t)) 0.0)]
               (update-progress! run-id progress)
               (when (replacement.protocols/canceled? this)
                 (log/infof "Replacement run %d was canceled" run-id)
                 (when on-complete
                   (deliver on-complete :run/cancelled))  ;; <-----------
                 (throw (ex-info "Run canceled" {:run-id run-id})))))))
       (canceled? [_] ...)
       (start-run! [_] ...)
       (succeed-run! [_]
         (log/infof "Replacement run %d succeeded." run-id)
         (succeed-run! run-id)
         (when on-complete
           (deliver on-complete :run/success)))  ;; <--------
       (fail-run! [_ throwable]
         (log/errorf throwable "Replacement run %d failed" run-id)
         (fail-run! run-id (ex-message throwable))
         (when on-complete
           (deliver on-complete :run/fail))))     :<-----
```

and in tests we can do like so:

```clojure
(deftest advance!-progress-write-boundaries-test
  (testing "advance! writes progress at batch boundaries and on final item"
    (mt/with-premium-features #{:dependencies}
      (mt/with-model-cleanup [:model/ReplacementRun]
        (let [writes (atom [])
              done?  (promise)]
          (with-redefs [replacement-run/update-progress!
                        (fn [_run-id progress]
                          (swap! writes conj progress))]
            (let [record (replacement-run/create-run! :card 1 :card 2 (mt/user->id :rasta))
                  progress (replacement-run/run-row->progress record done?)]
              (execute/execute-async!
               (fn [progress]
                 (replacement.protocols/set-total! progress 120)
                 ;; advance one-at-a-time for 120 items
                 (dotimes [_ 120]
                   (replacement.protocols/advance! progress)))
               progress)
              ;; wait for virtual thread
              (is (= :run/success (u/deref-with-timeout done? 500)))  ;; <------------
              ;; Writes at items 50, 100, 120 (final) → progress 50/120, 100/120, 120/120
              (is (= [(double (/ 50 120)) (double (/ 100 120)) 1.0]
                     @writes))
              (is (=? {:is_active nil :progress 1.0 :status :succeeded}
                      (t2/select-one :model/ReplacementRun :id (:id record)))))))))))
```

* fixes

* add new flag to resolution

* fix

* fix sandboxing

* fix

* wip be

* wip  be

* wip be

* wip

* wip be

* wip be

* missing join-alias

* wip be

* more field ref cases

* wip be

* wip be

* wip be

* remove healing

* implicit join second level

* swap refactoring

* both loops process all

[GHY-3182: swap all dependencies](https://linear.app/metabase/issue/GHY-3182/swap-all-dependencies)

* field refs

* fix types

* fix ref types

* short curcuit when swapping

* base-type test

* fix order by

* test

* Viz settings in cards

* handle field->expression conversion

* base-type fix

* unique fix

* deduplicate test

* only upgrade/swap when well-formed query

`(-> card :dataset_query :stages seq)`

degenerate cases are `{}` and `{:lib/type :lib/query}`

* remove tests with broken schema

* test that :field -> :expression works in parameters, too

* viz_settings!

* legacy->pMBQL

* preserve native queries metadata

* fix typo in a comment

* Tests for field_refs.clj

* swap function order

* remove transitive card ids, add usages tests

we don't need to keep a set of all card ids. it's huge, and we're gonna
upgrade cards either way. So no need to not upgrade card references

* Fix upgrade-field-ref-to-name crash on :aggregation refs

Viz-settings locations (pivot_table.column_split etc.) can contain
[:aggregation 0] refs, not just [:field ...]. Wrap ->pMBQL in
with-aggregation-list so it can resolve legacy index-based aggregation
refs to UUID-based pMBQL. Add try/catch as safety net for other
unexpected ref types.

Fixes 13 cards failing with "no aggregation at index 0" error.

* fix expressions

* add a test

* fix the test

* add a proper test

* Skip non-field refs in upgrade-field-ref-to-name

Per Alexander's feedback: since we only resolve :field refs downstream,
skip :aggregation/:expression refs at the gate instead of setting up
aggregation context for ->pMBQL.

* remove log call in source-swap

* add schemas

* add schemas

* repro test

* guard against missing tables

* more fixes

* wip tests

* add tests for resolution with missing metadata

* Mutation testing improvements

* Some more tests

* revertme: adds comilation tracker for statsdb swap+upgrade analysis

* add checks for wrong queries

* fixes

* test fixes

* fix

* remove old code

* temp test fixes

* fix viz

* fixes

* One more tests.

* Singular mutation testing run

* tests

* fix tests

* remove not needed column checks

* fix tests

* run-swap throws with old-source new-source and check-result

* fix dependencies check test

```clojure
source-swap-test=> (clojure.test/run-test swap-native-card-source!-updates-dependencies-test)

Testing metabase-enterprise.replacement.source-swap-test

Ran 1 tests containing 1 assertions.
0 failures, 0 errors.
{:test 1, :pass 1, :fail 0, :error 0, :type :summary}
```

So an interesting bit:
we must update dependencies regardless. It's important that the dep
graph is updated. Imagine if Card A is swapped for Card B. All cards
that use Card A will be listed as deps of Card A even if they don't
reference it and don't require changes. We are still swapping the source
so we need to update the graph.

We don't have a real api for dependencies. So to ensure the cards are in
the graph we call `(events/publish-event! :event/card-create {:object
card :user-id (mt/user->id :rasta)})` and when swapping we prefer to use
`(models.dependency/swap-dependency! :measure entity-id old-source
new-source)`. But for dashboards we'll just do the full recompute
accomplished with

```clojure
(events/publish-event! :event/dashboard-update
                       {:object (t2/select-one :model/Dashboard :id dashboard-id)})
```

* working field-refs tests

* Fix logic

* Strong logic (but is it possible?)

* fix ::entity-id schema to include all entity types

* refactor

* fixes

* fixes

* fixes

* fixes

* fixes

* wip

* fix

* fixes

* fixes

* fixes

* fixes

* fixes

* fixes

* wip be

* wip be

* wip be

* remove unused

* wip be

* remove invalid queries

* wip fixes

* wip walk function

* Measure and segment tests

* fixes

* aliases

* fixes

* fix tests

* make source_swap accept the preloaded entity

* renaming

* unify sources

* database id fixes

* fix table id check

* deps fixes

* reflection warning

* fixes

* fixes

* fixes

* fix team

* fix CI

* check for admin

* disable plugin

* Disable api endpoints in prod

* wip tests

* fire events; namespace doc

```clojure
clj -X:dev:test:ee:ee-dev :only metabase-enterprise.replacement.source-swap-test
Ran 29 tests in 29.144 seconds
219 assertions, 0 failures, 0 errors.
{:test 29, :pass 219, :fail 0, :error 0, :type :summary, :duration 29144.393083, :single-threaded 29}
Ran 0 tests in parallel, 29 single-threaded.
Finding and running tests took 40.1 s.
```

All tests passing

Fire event only when things change, swap sources before. That's faster
and must happen regardless, change event only on change.

Namespace docstring indicating why

* fix constraint error if dependency already exists:

```
custom-swap=> (source-swap/swap-source! [:card 27452] [:card 6759] [:table 52728])
Execution error (PSQLException) at org.postgresql.core.v3.QueryExecutorImpl/receiveErrorResponse (QueryExecutorImpl.java:2736).
ERROR: duplicate key value violates unique constraint "idx_unique_dependency"
  Detail: Key (from_entity_type, from_entity_id, to_entity_type, to_entity_id)=(card, 27452, table, 52728) already exists.
```

it seems to have already have a dependency. So now we just remove
dependency if the one we are adding already exists, otherwise we swap.

swap card A: dep on B -> dep on C
if it already depends on C, just delete the dep on B
otherwise swap where card A dep B -> dep C

* simplify check

* upgrade parameter value_source_config

* field refs

* tests

* viz tests

* parameters

* tests

* fixes

* tests

* check sources fixes

* fix tests

* tests

* wip tests

* fixes

* wip tests

* wip tests

* wip tests

* wip cleanup

* cleanup

* move test

* lint

* fix

* fix

* fixes

* fixes

* fixes

* fix

* fixes

* fix

* fix

* fixes

* fix

* FE cache fixes

* fix checks

* fix

* polish: remove tracing code, update docstring, add :pending to ::run-status enum

* Add test for table->card in check-column-mappings

* Remove unneeded argument.

* Remove unused function.

* Test that all api endpoints are gated behind dependencies feature flag

* use api/*current-user-id* instead of (:id @api/*current-user*) in source swap

Avoids unnecessarily forcing the *current-user* delay (which loads the
full User record from the DB) when only the integer ID is needed.

* stop swapping card_id in parameters

* throw on partial swap failures so the run is marked as failed

Previously run-swap* returned {:failures [...]} which execute-async!
ignored, marking the run as succeeded. Now it throws with a descriptive
message (count + first 10 failures) that gets persisted in the run's
message column.

* Check that all routes are protected behind superuser

* Add test for runner.run-swap

* FE checks

* Remove source-error schema (unused)

* Reuse lib-be.schema.source-swap/column-error schema instead of remaking it

* Kondo error

* move code to lib-be

* Better mutation testing

* fixes

* More tests for field-refs

* Better test

* More source swap tests.

* suppress dependency event handlers during source replacement

Source replacement manages dependencies itself via swap-dependency!, so
the event-driven dependency recalculation, stale-marking, and metadata
propagation triggered by entity update events is redundant work.

Adds *suppress-dependency-events?* dynamic var, checked by all dependency
event handlers. Bound to true in run-swap* phase 2.

* Fewer deps! Yay!

* Revert "suppress dependency event handlers during source replacement"

This reverts commit 03625e9c26.

* Clearer code

* update 2 docstrings

* publish dashboard-update event only when a dashcard changed

* simplify implementation

* simplify

* bulk mbql replacement

* Revert "bulk mbql replacement"

This reverts commit cf12636c42.

* simplify

* fix reseting selection when the graph changes

* fix

* add source_swap module

* fix lint

* fix comment

* fix

* fixes

* fix module order

* fix module order

* fix order

---------

Co-authored-by: Alexander Polyankin <alexander.polyankin@metabase.com>
Co-authored-by: Bryan Maass <bryan.maass@gmail.com>
Co-authored-by: dan sutton <dan@dpsutton.com>
Co-authored-by: Eric Normand <ericwnormand@gmail.com>
2026-03-10 14:25:01 -04:00

248 lines
9.4 KiB
YAML

databaseChangeLog:
- objectQuotingStrategy: QUOTE_ALL_OBJECTS
- changeSet:
id: v60.2026-02-13T21:43:00
author: epaget
comment: Add dimensions column to measure table
changes:
- addColumn:
tableName: measure
columns:
- column:
name: dimensions
type: ${text.type}
remarks: JSON array of persisted dimension definitions (id, display-name, types)
constraints:
nullable: true
rollback:
- dropColumn:
tableName: measure
columnName: dimensions
- changeSet:
id: v60.2026-02-13T21:43:01
author: epaget
comment: Add dimension_mappings column to measure table
changes:
- addColumn:
tableName: measure
columns:
- column:
name: dimension_mappings
type: ${text.type}
remarks: JSON array of dimension-to-field mappings (dimension-id, target, table-id)
constraints:
nullable: true
rollback:
- dropColumn:
tableName: measure
columnName: dimension_mappings
- changeSet:
id: v60.2026-02-13T21:43:02
author: epaget
comment: Add dimensions column to report_card table for metrics
changes:
- addColumn:
tableName: report_card
columns:
- column:
name: dimensions
type: ${text.type}
remarks: JSON array of persisted dimension definitions (id, display-name, types)
constraints:
nullable: true
rollback:
- dropColumn:
tableName: report_card
columnName: dimensions
- changeSet:
id: v60.2026-02-13T21:43:03
author: epaget
comment: Add dimension_mappings column to report_card table for metrics
changes:
- addColumn:
tableName: report_card
columns:
- column:
name: dimension_mappings
type: ${text.type}
remarks: JSON array of dimension-to-field mappings (dimension-id, target, table-id)
constraints:
nullable: true
rollback:
- dropColumn:
tableName: report_card
columnName: dimension_mappings
- changeSet:
id: v60.2026-03-06T12:00:00
author: bcm
comment: Create source_replacement_run table to track async source replacement executions
changes:
- createTable:
tableName: source_replacement_run
remarks: Table to track source replacement executions and their status
columns:
- column:
name: id
type: int
remarks: Unique ID
autoIncrement: true
constraints:
primaryKey: true
nullable: false
- column:
name: source_entity_type
type: varchar(50)
remarks: "Type of the source entity (card or table)"
constraints:
nullable: false
- column:
name: source_entity_id
type: int
remarks: "ID of the source entity"
constraints:
nullable: false
- column:
name: target_entity_type
type: varchar(50)
remarks: "Type of the target entity (card or table)"
constraints:
nullable: false
- column:
name: target_entity_id
type: int
remarks: "ID of the target entity"
constraints:
nullable: false
- column:
name: status
type: varchar(50)
remarks: "Current status (started, succeeded, failed, canceled, timeout)"
constraints:
nullable: false
- column:
name: is_active
type: ${boolean.type}
remarks: "True only for currently running replacements, null for completed"
constraints:
nullable: true
- column:
name: progress
type: float
remarks: "Progress from 0.0 to 1.0"
constraints:
nullable: true
- column:
name: message
type: ${text.type}
remarks: "Human-readable message; may contain error info"
constraints:
nullable: true
- column:
name: user_id
type: int
remarks: "User who initiated the replacement"
constraints:
nullable: true
- column:
name: start_time
type: ${timestamp_type}
remarks: "When the replacement started"
defaultValueComputed: current_timestamp
constraints:
nullable: false
- column:
name: end_time
type: ${timestamp_type}
remarks: "When the replacement completed (null if still running)"
constraints:
nullable: true
- changeSet:
id: v60.2026-03-06T12:00:01
author: bcm
comment: Enforce at most one active source replacement run cluster-wide
changes:
- addUniqueConstraint:
tableName: source_replacement_run
constraintName: idx_unique_active_source_replacement_run
columnNames: is_active
- changeSet:
id: v60.2026-03-06T12:00:02
author: bcm
comment: Add index on source_replacement_run.user_id
changes:
- createIndex:
tableName: source_replacement_run
indexName: idx_source_replacement_run_user_id
columns:
- column:
name: user_id
- changeSet:
id: v60.2026-03-06T12:00:03
author: bcm
comment: Add foreign key on source_replacement_run(user_id)
changes:
- addForeignKeyConstraint:
baseTableName: source_replacement_run
baseColumnNames: user_id
referencedTableName: core_user
referencedColumnNames: id
constraintName: fk_source_replacement_run_user_id
# >>>>>>>>>> DO NOT ADD NEW MIGRATIONS BELOW THIS LINE! ADD THEM ABOVE <<<<<<<<<<
########################################################################################################################
#
# ADVICE:
#
# 1) Think through some of these points when writing a migration, learn from our past mistakes:
# - Do you really need a migration? Could the feature work without it? Prefer not to if possible.
# Data in the wild can be vastly different from what you expect, and it's hard to get right.
# - Make sure your migration is backward compatible: it might not be possible to add a constraint back
# if you drop it in a migration.
# - Postgres, MySQL and H2 have their differences. Make sure your migration works for all.
# - Database encryption is a major issue:
# - Fields like `metabase_database.details` or `setting.value` can be encrypted, so you need to decrypt them in
# your migration. See #42617, #44048.
# - Database could be partially encrypted, see https://www.notion.so/72575933ef2a446bafd16909e05a7387
# - Custom migrations:
# - Prefer SQL migrations when possible.
# - Never use application code: it can change and *will* break your migration.
# - Do not use Toucan models - refer table names directly.
# - If it's a big change, consider using Quartz, see #42279
# - More in `metabase.app_db.custom_migrations` namespace doc.
# - Never delete a migration: users won't be able to downgrade. If you really need to, see #44908
# - Migration id (`vXX.<date>`) must match its earliest released version:
# - Do not backport `v51....` to version 50, Metabase will try to downgrade it.
# - Instead, write a migration with an oldest target you plan to backport to in mind.
#
# 2) Migrations should go in the ###_update_migrations.yaml file for the target version.
#
# 3) Run mage lint-migrations to run core.spec checks against any changes you make here. Liquibase is pretty
# forgiving and won't complain if you accidentally mix up things like deleteCascade and onDelete: CASCADE. CI runs
# this check but it's nicer to know now instead of waiting for CI.
#
# 3) Migration IDs should follow the format
#
# vMM.TIMESTAMP
#
# Where
#
# M = major version
# TIMESTAMP = the current timestamp with format `yyyy-MM-dd'T'HH:mm:ss`
# To get this timestamp, evaluate this in your REPL: `(dev/migration-timestamp)`
#
# E.g: You're adding a new migration for version 49, And it's 10:30:00AM on April 1, 2023 (UTC),
# your migration id should be: `v49.2023-04-01T10:30:00`.
#
# PLEASE KEEP THIS MESSAGE AT THE BOTTOM OF THIS FILE!!!!! Add new migrations above the message.
#
########################################################################################################################