* fixes
* test parser changes
* schema test
* fix linter
* implicit join error
* fe tests
* fix linter
* test fixes
* more tests
* tests
* upgrade field refs for dashboardcard_series and dashboards in transitive deps
- Add dashboard-upgrade-field-refs! for Phase 1b: upgrades parameter_mappings
on dashboards found in transitive deps (cards are upgraded first in Phase 1)
- Fix dashboard-card-update-field-refs! to find dashcards via
DashboardCardSeries, not just by primary card_id
- Split dashboard-ids out of transitive in runner, pass separately to run-swap*
- Add tests for series card parameter mapping swap and dedup
* fixup some tests, add promise to progress instance
useful for test so we don't have to loop on the db row
```clojure
(reify replacement.protocols/IRunnerProgress
(set-total! [_ total] ...)
(advance! [this] ...)
(advance! [this n]
(let [c (swap! completed* + n)
t @total*
prev (- c n)
crossed-boundary? (or (= c t)
(not= (quot prev progress-batch-size)
(quot c progress-batch-size)))]
(when crossed-boundary?
(let [progress (if (pos? t) (double (/ c t)) 0.0)]
(update-progress! run-id progress)
(when (replacement.protocols/canceled? this)
(log/infof "Replacement run %d was canceled" run-id)
(when on-complete
(deliver on-complete :run/cancelled)) ;; <-----------
(throw (ex-info "Run canceled" {:run-id run-id})))))))
(canceled? [_] ...)
(start-run! [_] ...)
(succeed-run! [_]
(log/infof "Replacement run %d succeeded." run-id)
(succeed-run! run-id)
(when on-complete
(deliver on-complete :run/success))) ;; <--------
(fail-run! [_ throwable]
(log/errorf throwable "Replacement run %d failed" run-id)
(fail-run! run-id (ex-message throwable))
(when on-complete
(deliver on-complete :run/fail)))) :<-----
```
and in tests we can do like so:
```clojure
(deftest advance!-progress-write-boundaries-test
(testing "advance! writes progress at batch boundaries and on final item"
(mt/with-premium-features #{:dependencies}
(mt/with-model-cleanup [:model/ReplacementRun]
(let [writes (atom [])
done? (promise)]
(with-redefs [replacement-run/update-progress!
(fn [_run-id progress]
(swap! writes conj progress))]
(let [record (replacement-run/create-run! :card 1 :card 2 (mt/user->id :rasta))
progress (replacement-run/run-row->progress record done?)]
(execute/execute-async!
(fn [progress]
(replacement.protocols/set-total! progress 120)
;; advance one-at-a-time for 120 items
(dotimes [_ 120]
(replacement.protocols/advance! progress)))
progress)
;; wait for virtual thread
(is (= :run/success (u/deref-with-timeout done? 500))) ;; <------------
;; Writes at items 50, 100, 120 (final) → progress 50/120, 100/120, 120/120
(is (= [(double (/ 50 120)) (double (/ 100 120)) 1.0]
@writes))
(is (=? {:is_active nil :progress 1.0 :status :succeeded}
(t2/select-one :model/ReplacementRun :id (:id record)))))))))))
```
* fixes
* add new flag to resolution
* fix
* fix sandboxing
* fix
* wip be
* wip be
* wip be
* wip
* wip be
* wip be
* missing join-alias
* wip be
* more field ref cases
* wip be
* wip be
* wip be
* remove healing
* implicit join second level
* swap refactoring
* both loops process all
[GHY-3182: swap all dependencies](https://linear.app/metabase/issue/GHY-3182/swap-all-dependencies)
* field refs
* fix types
* fix ref types
* short curcuit when swapping
* base-type test
* fix order by
* test
* Viz settings in cards
* handle field->expression conversion
* base-type fix
* unique fix
* deduplicate test
* only upgrade/swap when well-formed query
`(-> card :dataset_query :stages seq)`
degenerate cases are `{}` and `{:lib/type :lib/query}`
* remove tests with broken schema
* test that :field -> :expression works in parameters, too
* viz_settings!
* legacy->pMBQL
* preserve native queries metadata
* fix typo in a comment
* Tests for field_refs.clj
* swap function order
* remove transitive card ids, add usages tests
we don't need to keep a set of all card ids. it's huge, and we're gonna
upgrade cards either way. So no need to not upgrade card references
* Fix upgrade-field-ref-to-name crash on :aggregation refs
Viz-settings locations (pivot_table.column_split etc.) can contain
[:aggregation 0] refs, not just [:field ...]. Wrap ->pMBQL in
with-aggregation-list so it can resolve legacy index-based aggregation
refs to UUID-based pMBQL. Add try/catch as safety net for other
unexpected ref types.
Fixes 13 cards failing with "no aggregation at index 0" error.
* fix expressions
* add a test
* fix the test
* add a proper test
* Skip non-field refs in upgrade-field-ref-to-name
Per Alexander's feedback: since we only resolve :field refs downstream,
skip :aggregation/:expression refs at the gate instead of setting up
aggregation context for ->pMBQL.
* remove log call in source-swap
* add schemas
* add schemas
* repro test
* guard against missing tables
* more fixes
* wip tests
* add tests for resolution with missing metadata
* Mutation testing improvements
* Some more tests
* revertme: adds comilation tracker for statsdb swap+upgrade analysis
* add checks for wrong queries
* fixes
* test fixes
* fix
* remove old code
* temp test fixes
* fix viz
* fixes
* One more tests.
* Singular mutation testing run
* tests
* fix tests
* remove not needed column checks
* fix tests
* run-swap throws with old-source new-source and check-result
* fix dependencies check test
```clojure
source-swap-test=> (clojure.test/run-test swap-native-card-source!-updates-dependencies-test)
Testing metabase-enterprise.replacement.source-swap-test
Ran 1 tests containing 1 assertions.
0 failures, 0 errors.
{:test 1, :pass 1, :fail 0, :error 0, :type :summary}
```
So an interesting bit:
we must update dependencies regardless. It's important that the dep
graph is updated. Imagine if Card A is swapped for Card B. All cards
that use Card A will be listed as deps of Card A even if they don't
reference it and don't require changes. We are still swapping the source
so we need to update the graph.
We don't have a real api for dependencies. So to ensure the cards are in
the graph we call `(events/publish-event! :event/card-create {:object
card :user-id (mt/user->id :rasta)})` and when swapping we prefer to use
`(models.dependency/swap-dependency! :measure entity-id old-source
new-source)`. But for dashboards we'll just do the full recompute
accomplished with
```clojure
(events/publish-event! :event/dashboard-update
{:object (t2/select-one :model/Dashboard :id dashboard-id)})
```
* working field-refs tests
* Fix logic
* Strong logic (but is it possible?)
* fix ::entity-id schema to include all entity types
* refactor
* fixes
* fixes
* fixes
* fixes
* fixes
* wip
* fix
* fixes
* fixes
* fixes
* fixes
* fixes
* fixes
* wip be
* wip be
* wip be
* remove unused
* wip be
* remove invalid queries
* wip fixes
* wip walk function
* Measure and segment tests
* fixes
* aliases
* fixes
* fix tests
* make source_swap accept the preloaded entity
* renaming
* unify sources
* database id fixes
* fix table id check
* deps fixes
* reflection warning
* fixes
* fixes
* fixes
* fix team
* fix CI
* check for admin
* disable plugin
* Disable api endpoints in prod
* wip tests
* fire events; namespace doc
```clojure
clj -X:dev:test:ee:ee-dev :only metabase-enterprise.replacement.source-swap-test
Ran 29 tests in 29.144 seconds
219 assertions, 0 failures, 0 errors.
{:test 29, :pass 219, :fail 0, :error 0, :type :summary, :duration 29144.393083, :single-threaded 29}
Ran 0 tests in parallel, 29 single-threaded.
Finding and running tests took 40.1 s.
```
All tests passing
Fire event only when things change, swap sources before. That's faster
and must happen regardless, change event only on change.
Namespace docstring indicating why
* fix constraint error if dependency already exists:
```
custom-swap=> (source-swap/swap-source! [:card 27452] [:card 6759] [:table 52728])
Execution error (PSQLException) at org.postgresql.core.v3.QueryExecutorImpl/receiveErrorResponse (QueryExecutorImpl.java:2736).
ERROR: duplicate key value violates unique constraint "idx_unique_dependency"
Detail: Key (from_entity_type, from_entity_id, to_entity_type, to_entity_id)=(card, 27452, table, 52728) already exists.
```
it seems to have already have a dependency. So now we just remove
dependency if the one we are adding already exists, otherwise we swap.
swap card A: dep on B -> dep on C
if it already depends on C, just delete the dep on B
otherwise swap where card A dep B -> dep C
* simplify check
* upgrade parameter value_source_config
* field refs
* tests
* viz tests
* parameters
* tests
* fixes
* tests
* check sources fixes
* fix tests
* tests
* wip tests
* fixes
* wip tests
* wip tests
* wip tests
* wip cleanup
* cleanup
* move test
* lint
* fix
* fix
* fixes
* fixes
* fixes
* fix
* fixes
* fix
* fix
* fixes
* fix
* FE cache fixes
* fix checks
* fix
* polish: remove tracing code, update docstring, add :pending to ::run-status enum
* Add test for table->card in check-column-mappings
* Remove unneeded argument.
* Remove unused function.
* Test that all api endpoints are gated behind dependencies feature flag
* use api/*current-user-id* instead of (:id @api/*current-user*) in source swap
Avoids unnecessarily forcing the *current-user* delay (which loads the
full User record from the DB) when only the integer ID is needed.
* stop swapping card_id in parameters
* throw on partial swap failures so the run is marked as failed
Previously run-swap* returned {:failures [...]} which execute-async!
ignored, marking the run as succeeded. Now it throws with a descriptive
message (count + first 10 failures) that gets persisted in the run's
message column.
* Check that all routes are protected behind superuser
* Add test for runner.run-swap
* FE checks
* Remove source-error schema (unused)
* Reuse lib-be.schema.source-swap/column-error schema instead of remaking it
* Kondo error
* move code to lib-be
* Better mutation testing
* fixes
* More tests for field-refs
* Better test
* More source swap tests.
* suppress dependency event handlers during source replacement
Source replacement manages dependencies itself via swap-dependency!, so
the event-driven dependency recalculation, stale-marking, and metadata
propagation triggered by entity update events is redundant work.
Adds *suppress-dependency-events?* dynamic var, checked by all dependency
event handlers. Bound to true in run-swap* phase 2.
* Fewer deps! Yay!
* Revert "suppress dependency event handlers during source replacement"
This reverts commit 03625e9c26.
* Clearer code
* update 2 docstrings
* publish dashboard-update event only when a dashcard changed
* simplify implementation
* simplify
* bulk mbql replacement
* Revert "bulk mbql replacement"
This reverts commit cf12636c42.
* simplify
* fix reseting selection when the graph changes
* fix
* add source_swap module
* fix lint
* fix comment
* fix
* fixes
* fix module order
* fix module order
* fix order
---------
Co-authored-by: Alexander Polyankin <alexander.polyankin@metabase.com>
Co-authored-by: Bryan Maass <bryan.maass@gmail.com>
Co-authored-by: dan sutton <dan@dpsutton.com>
Co-authored-by: Eric Normand <ericwnormand@gmail.com>
248 lines
9.4 KiB
YAML
248 lines
9.4 KiB
YAML
databaseChangeLog:
|
|
- objectQuotingStrategy: QUOTE_ALL_OBJECTS
|
|
|
|
- changeSet:
|
|
id: v60.2026-02-13T21:43:00
|
|
author: epaget
|
|
comment: Add dimensions column to measure table
|
|
changes:
|
|
- addColumn:
|
|
tableName: measure
|
|
columns:
|
|
- column:
|
|
name: dimensions
|
|
type: ${text.type}
|
|
remarks: JSON array of persisted dimension definitions (id, display-name, types)
|
|
constraints:
|
|
nullable: true
|
|
rollback:
|
|
- dropColumn:
|
|
tableName: measure
|
|
columnName: dimensions
|
|
|
|
- changeSet:
|
|
id: v60.2026-02-13T21:43:01
|
|
author: epaget
|
|
comment: Add dimension_mappings column to measure table
|
|
changes:
|
|
- addColumn:
|
|
tableName: measure
|
|
columns:
|
|
- column:
|
|
name: dimension_mappings
|
|
type: ${text.type}
|
|
remarks: JSON array of dimension-to-field mappings (dimension-id, target, table-id)
|
|
constraints:
|
|
nullable: true
|
|
rollback:
|
|
- dropColumn:
|
|
tableName: measure
|
|
columnName: dimension_mappings
|
|
|
|
- changeSet:
|
|
id: v60.2026-02-13T21:43:02
|
|
author: epaget
|
|
comment: Add dimensions column to report_card table for metrics
|
|
changes:
|
|
- addColumn:
|
|
tableName: report_card
|
|
columns:
|
|
- column:
|
|
name: dimensions
|
|
type: ${text.type}
|
|
remarks: JSON array of persisted dimension definitions (id, display-name, types)
|
|
constraints:
|
|
nullable: true
|
|
rollback:
|
|
- dropColumn:
|
|
tableName: report_card
|
|
columnName: dimensions
|
|
|
|
- changeSet:
|
|
id: v60.2026-02-13T21:43:03
|
|
author: epaget
|
|
comment: Add dimension_mappings column to report_card table for metrics
|
|
changes:
|
|
- addColumn:
|
|
tableName: report_card
|
|
columns:
|
|
- column:
|
|
name: dimension_mappings
|
|
type: ${text.type}
|
|
remarks: JSON array of dimension-to-field mappings (dimension-id, target, table-id)
|
|
constraints:
|
|
nullable: true
|
|
rollback:
|
|
- dropColumn:
|
|
tableName: report_card
|
|
columnName: dimension_mappings
|
|
|
|
- changeSet:
|
|
id: v60.2026-03-06T12:00:00
|
|
author: bcm
|
|
comment: Create source_replacement_run table to track async source replacement executions
|
|
changes:
|
|
- createTable:
|
|
tableName: source_replacement_run
|
|
remarks: Table to track source replacement executions and their status
|
|
columns:
|
|
- column:
|
|
name: id
|
|
type: int
|
|
remarks: Unique ID
|
|
autoIncrement: true
|
|
constraints:
|
|
primaryKey: true
|
|
nullable: false
|
|
- column:
|
|
name: source_entity_type
|
|
type: varchar(50)
|
|
remarks: "Type of the source entity (card or table)"
|
|
constraints:
|
|
nullable: false
|
|
- column:
|
|
name: source_entity_id
|
|
type: int
|
|
remarks: "ID of the source entity"
|
|
constraints:
|
|
nullable: false
|
|
- column:
|
|
name: target_entity_type
|
|
type: varchar(50)
|
|
remarks: "Type of the target entity (card or table)"
|
|
constraints:
|
|
nullable: false
|
|
- column:
|
|
name: target_entity_id
|
|
type: int
|
|
remarks: "ID of the target entity"
|
|
constraints:
|
|
nullable: false
|
|
- column:
|
|
name: status
|
|
type: varchar(50)
|
|
remarks: "Current status (started, succeeded, failed, canceled, timeout)"
|
|
constraints:
|
|
nullable: false
|
|
- column:
|
|
name: is_active
|
|
type: ${boolean.type}
|
|
remarks: "True only for currently running replacements, null for completed"
|
|
constraints:
|
|
nullable: true
|
|
- column:
|
|
name: progress
|
|
type: float
|
|
remarks: "Progress from 0.0 to 1.0"
|
|
constraints:
|
|
nullable: true
|
|
- column:
|
|
name: message
|
|
type: ${text.type}
|
|
remarks: "Human-readable message; may contain error info"
|
|
constraints:
|
|
nullable: true
|
|
- column:
|
|
name: user_id
|
|
type: int
|
|
remarks: "User who initiated the replacement"
|
|
constraints:
|
|
nullable: true
|
|
- column:
|
|
name: start_time
|
|
type: ${timestamp_type}
|
|
remarks: "When the replacement started"
|
|
defaultValueComputed: current_timestamp
|
|
constraints:
|
|
nullable: false
|
|
- column:
|
|
name: end_time
|
|
type: ${timestamp_type}
|
|
remarks: "When the replacement completed (null if still running)"
|
|
constraints:
|
|
nullable: true
|
|
|
|
- changeSet:
|
|
id: v60.2026-03-06T12:00:01
|
|
author: bcm
|
|
comment: Enforce at most one active source replacement run cluster-wide
|
|
changes:
|
|
- addUniqueConstraint:
|
|
tableName: source_replacement_run
|
|
constraintName: idx_unique_active_source_replacement_run
|
|
columnNames: is_active
|
|
|
|
- changeSet:
|
|
id: v60.2026-03-06T12:00:02
|
|
author: bcm
|
|
comment: Add index on source_replacement_run.user_id
|
|
changes:
|
|
- createIndex:
|
|
tableName: source_replacement_run
|
|
indexName: idx_source_replacement_run_user_id
|
|
columns:
|
|
- column:
|
|
name: user_id
|
|
|
|
- changeSet:
|
|
id: v60.2026-03-06T12:00:03
|
|
author: bcm
|
|
comment: Add foreign key on source_replacement_run(user_id)
|
|
changes:
|
|
- addForeignKeyConstraint:
|
|
baseTableName: source_replacement_run
|
|
baseColumnNames: user_id
|
|
referencedTableName: core_user
|
|
referencedColumnNames: id
|
|
constraintName: fk_source_replacement_run_user_id
|
|
|
|
# >>>>>>>>>> DO NOT ADD NEW MIGRATIONS BELOW THIS LINE! ADD THEM ABOVE <<<<<<<<<<
|
|
|
|
########################################################################################################################
|
|
#
|
|
# ADVICE:
|
|
#
|
|
# 1) Think through some of these points when writing a migration, learn from our past mistakes:
|
|
# - Do you really need a migration? Could the feature work without it? Prefer not to if possible.
|
|
# Data in the wild can be vastly different from what you expect, and it's hard to get right.
|
|
# - Make sure your migration is backward compatible: it might not be possible to add a constraint back
|
|
# if you drop it in a migration.
|
|
# - Postgres, MySQL and H2 have their differences. Make sure your migration works for all.
|
|
# - Database encryption is a major issue:
|
|
# - Fields like `metabase_database.details` or `setting.value` can be encrypted, so you need to decrypt them in
|
|
# your migration. See #42617, #44048.
|
|
# - Database could be partially encrypted, see https://www.notion.so/72575933ef2a446bafd16909e05a7387
|
|
# - Custom migrations:
|
|
# - Prefer SQL migrations when possible.
|
|
# - Never use application code: it can change and *will* break your migration.
|
|
# - Do not use Toucan models - refer table names directly.
|
|
# - If it's a big change, consider using Quartz, see #42279
|
|
# - More in `metabase.app_db.custom_migrations` namespace doc.
|
|
# - Never delete a migration: users won't be able to downgrade. If you really need to, see #44908
|
|
# - Migration id (`vXX.<date>`) must match its earliest released version:
|
|
# - Do not backport `v51....` to version 50, Metabase will try to downgrade it.
|
|
# - Instead, write a migration with an oldest target you plan to backport to in mind.
|
|
#
|
|
# 2) Migrations should go in the ###_update_migrations.yaml file for the target version.
|
|
#
|
|
# 3) Run mage lint-migrations to run core.spec checks against any changes you make here. Liquibase is pretty
|
|
# forgiving and won't complain if you accidentally mix up things like deleteCascade and onDelete: CASCADE. CI runs
|
|
# this check but it's nicer to know now instead of waiting for CI.
|
|
#
|
|
# 3) Migration IDs should follow the format
|
|
#
|
|
# vMM.TIMESTAMP
|
|
#
|
|
# Where
|
|
#
|
|
# M = major version
|
|
# TIMESTAMP = the current timestamp with format `yyyy-MM-dd'T'HH:mm:ss`
|
|
# To get this timestamp, evaluate this in your REPL: `(dev/migration-timestamp)`
|
|
#
|
|
# E.g: You're adding a new migration for version 49, And it's 10:30:00AM on April 1, 2023 (UTC),
|
|
# your migration id should be: `v49.2023-04-01T10:30:00`.
|
|
#
|
|
# PLEASE KEEP THIS MESSAGE AT THE BOTTOM OF THIS FILE!!!!! Add new migrations above the message.
|
|
#
|
|
########################################################################################################################
|