* Add Metabase Sample Database to editable tables support
Updated the list of databases that support editable tables to include Metabase Sample Database.
* Update docs/data-modeling/editable-tables.md
* Update docs/data-modeling/editable-tables.md
- If you decide to build the backend, you should run it
- There are no more derived options - everything is explicit
- Running Docker containers is a non-negotiable
- Tests need these supporting services to run
- If you really insist on not running them, either comment out those lines temporarily or do not use this script to set up the e2e environment
- Because of this, there are no more custom envs `QA_DB_ENABLED` and `CYPRESS_QA_DB_MONGO`
- Alternative location (directory) for the e2e copy of the sample-application.db is now set to be `e2e/tmp` since it's already gitignored
* UXW-2209 Updates datamodel tracking triggered_from to be either "data_studio" or "admin" depending on where the metadata is being edited
* Updates e2e tests
* UXW-2352 Fix issue where CopyCardForm resets when the destination collection is changed
* Unskips now-passing e2e tests
* Fix issue where Duplicate modal doesn't close on success
The previous version of the build script (`build-for-test`) had a weird quirk that could be both confusing and dangerous. It was building a backend-only JAR in the same location and under same name as the full-blown, all-in-one Metabase JAR. This means that, by running Cypress locally, one could easily override the previously built healthy JAR. It's not catastrophic but it was confusing.
This PR makes it perfectly clear what we're doing, and there should be no conflicts between the JARs any longer.
* library collection search
* Fix e2e tests from group-16
* Fix a couple tests from group 42
* Fix e2e tests from group-17
* Fix e2e tests from group-18
* tests: add bulk publishing scenario
fix wrong collection link in case of root collection
* UXW-2315, UXW-2314
* UXW-2294
* more UXW-2315, UXW-2314
* more UXW-2315, UXW-2314
* correct collection types
* Remove Browsing Memory Feature
* Revert "Revert "Don't return library collections by default from APIs""
This reverts commit 814c49473b.
* Revert "Revert "Fixed null type filtering""
This reverts commit 25e0d78408.
* fetch full tree to have library collections visible
* show library in permissions
* Fixing tests
* fix infinite re-render in tree
* rewrite unit tests and fix max depth problem
* fix: publish property not validated in the tree view after table publishing
* Move bulk editing apis to enterprise (#65968)
* Make sidebar sticky
* test: bulk attribute actions for tables, schemas and dbs
* fix more unit tests
* Copy preview section tests
* Copy error handling tests
* Copy undos tests
* Update test
* test: update new ee endpoints paths
* Make new bulk table APIs ee-only (fe)
- Closes GDGT-1251
* Fix global H.DataModel.visit overrides
* Remove redundant cy.reload
* Remove redundant cy.reload
* add select/deselect tests
* use helper
* data picker using proper library endpoint. Fixing types I broke. Endless Shame
* Fix Tree skeleton resizing multiple times during loading
- Closes GDGT-1253
* Improve icon
* Fill content
* don't call getCollectionItems on database collection
* add search tests and support shift click in search
* default table publish collection
* fox: reset bulk form values when selection changes
* Fix e2e tests from group-19
* fix: fix missing dependencies in useEffect gdgt-1245
* test: refactor test setup action order
* Fix e2e tests from group-20
* Fix e2e tests from group-22
* Fix e2e tests from group-23
* test: restore default db as the first step
* Kondo the kondo config
* default collections for metrics and models outside of data studio
* Move edit test to ee
* Fix list filtering now default is bronze
* Move bulk-edit-test to ee
* Fix table metadata for test
* Revert "Added `collection` argument to search"
This reverts commits:
- 21999e01
- b0f71c03ea
- 4fa0c3b422
* Fixed OSS loading of tables
* Fixed OSS loading of tables
* add tests for filtering
* Fix e2e tests from group 41 / joins-repros
* more tests
* Fix e2e tests from group-20
* Fix e2e tests from group-23
* Fix e2e tests from group-24
* Fix e2e tests from group-26
* Fix e2e tests from group-27
* Fix e2e tests from group-28
* Fix e2e tests from group-30
* constuct initial paths from library collections
* segments editor, hasEnabledNestedQueries
* Fix a couple more (something between groups 40 and 43)
* Fixed module config
* keyboard navigation
* Added library/tree API
* Fix some e2e tests from groups 40 and 42
* Fixing tests
* disable scoped search on the 'database collection', models.cy.spec should be passing now
* Fix e2e tests from group-33
* Minor cleanup
* Fix e2e tests from group-34
* Fixing tests
* Fix e2e tests from group 37 (except one depending on Recents tab)
* Starts fixing e2e tests from oss group
* cleanup
* wait to focus
* hide our analytics in the data studio
* Added library/tree API
* merge library and data studio
* change backend library flag validation to data-studio
* tidy things up
* Make bulk edit inputs non-clearable
- This prevents sending "empty" requests when clearing values
- And clearing does not really make sense here
* fix: test: table rescan and discard mocks are wrong
* fix: test: wrong mock in oss sync table schema
* Remove ^:parallel from bulk edit tests
They can contend and cause deadlocks
* fix: test: failure due to repeated mocking of same route
* test: refactor: cypress table tests
* Update e2e test
* Update e2e test
* Update e2e test
* fix browse nav section test
* Add missing token
* add check icon for published tables
* Removed unused import
* Updates some e2e tests from oss group, removes some now-invalid tests
* test: use default external db, skip transform test
* Starts fixing e2e tests from mongo group
* remove background from not active table in search
* specs
* Fix e2e tests from group-15
* hack fix ci while i'm on meetings
* scoped search is back
* Fix some e2e tests from group-32
* table picker fixes
* Fixed merge
* Fix e2e test from group-42
* merge fixes
* re-enable the recents tab on the data picker modal
* Avoid race condition on list-table-filtering-test
* fix embedding types
* mini picker unit tests
* Fixing tests
* ci, fixes
* fix library collection bugs
* cleanup data studio plugin
* fix lots of unit tests
* fix import
* fix
* Fixed unused import
* Fixed accidental #p
* ci
* revert util
* specs
* specs
* specs
* fix bug / bad type in SchemaFormSelect (#65995)
* Use actual anti join in GET /table query
* Remove old comments
* Activate token after restore
* test: unskip table metadata transform test
* test: fix search tables e2e test
* fix: test: selection in table picker
* Fix Source input disabled when filtering by Metabase transform
- Closes GDGT-1259
* test: fix: filter unused table
* Fix e2e test
* fix: test: datamodel toasts and errors
* Fix e2e test
* Fix e2e tests from group-9
* Fix flaky tests
* GDGT-1244: Fix DB tree virtualization (#66125)
* GDGT-1244: Fix tree virtualization. Replace top with translateY to improve performance
* GDGT-1244: Fix scroll bug. Refactor border styling
* bring back modeling empty state
* test: adjust publishing test
* Fix e2e test from group-22
* Fix e2e test from group-25
* fix allowing cross DB joins
* properly disable invalid items in MiniPicker
* Fix e2e test from group-26
* Fix straggler from oss group
* Starts fixing e2e tests from group-32
* unit test adjustments
* Fix remaining issue with e2e group 37
* fix item filtering
* Update Loki Snapshots
* add recent views to mini picker
* test recents and backparsing
* additional filtering test
* fix minipicker shows items it should not show
* Fixed :here in library API
* Fixed :here in library API
* UXW-2358 publishing outside of the data studio
* Fixed :here in library API
* snapshot
* Fix most e2e tests from group-32
* Lint fixes
* Fixed failing test
* revert fix
* Reapply "Added `collection` argument to search"
This reverts commit f28d9d5af9.
* Quick fix in CollectionPicker/utils.ts
* Fix flaky Remote Sync test (#65862)
* Clean up backend-test matrix (#66158)
* Fix timeseries value detection (#66161)
* Prevent driver tests (#66156)
Co-authored-by: Bryan Maass <bryan.maass@gmail.com>
* Fetch uberjar using a proper action (#66165)
* fix embedding column picking
* Trying CI with search_index.verified
* Fixed failing test
* Translate label
* Fixed failing test
* fix: published info not updated in the tree picker after deleting model
* fix: refactor: publish collection link tests
* Fixed failing test
* Fix remaining e2e tests from group-32
* FileUploadStatus unit test, quesiton reproductions
* create data studio plugin file
* fix hidden items some more
* Fix issue where confirmation modal closes then reopens again then closes again when you archive a dashboard
* fix dark mode
* UXW-2367
* UXW-2378
* Fixed merge issues
* this element is not visible because you can't even open incompatible sample database db
* Fix e2e test from group-14
* fix embedding data picker clicks
* filtering fixes for transforms
* UXW-2372
* update mini picker specs after a fix
* fix flaky join test
* Fixing in-place support for collections
* fix: cannot scroll search results
* ui: tweak search results checkbox spacing to match tree results
* fix: scroll position jump when entering field sort mode gdgt-1269
* Revert "fix: scroll position jump when entering field sort mode gdgt-1269"
This reverts commit bfecdec661.
* UXW-2279 Changes Data Studio's Glossary page background color to background-secondary
* Fix linters
* Fix e2e tests
* fixes for sqlserver
* fix driver changelog
* fix
* more fix
* fix: support params in bigq
* fix: inline params for sqlserver
* fix test
* fix: we do want the unique key
* fix: missing nil in test
* fix clickhouse
---------
Co-authored-by: Ryan Laurie <iethree@gmail.com>
Co-authored-by: Brad Anderson <brad@metabase.com>
Co-authored-by: Anderson <anderson@metabase.com>
Co-authored-by: Jakub Chodorowicz <jakub@chodorowicz.com>
Co-authored-by: Aleksandr Lesnenko <alxnddr@gmail.com>
Co-authored-by: Nathan Voxland <nathan@metabase.com>
Co-authored-by: Uladzimir Havenchyk <uladzimir@metabase.com>
Co-authored-by: Uladzimir Havenchyk <125459446+uladzimirdev@users.noreply.github.com>
Co-authored-by: Ngoc Khuat <qn.khuat@gmail.com>
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
Co-authored-by: Nick Fitzpatrick <nickfitz.582@gmail.com>
Co-authored-by: Dan Stone <wotbrew@gmail.com>
Co-authored-by: Sloan Sparger <sloansparger@users.noreply.github.com>
Co-authored-by: Aleksandr Lesnenko <alxnddr@users.noreply.github.com>
Co-authored-by: Metabase Automation <github-automation@metabase.com>
Co-authored-by: John Swanson <john.swanson@metabase.com>
Co-authored-by: Nemanja Glumac <31325167+nemanjaglumac@users.noreply.github.com>
Co-authored-by: Denis Berezin <denis.berezin@metabase.com>
Co-authored-by: dpsutton <dan@dpsutton.com>
Co-authored-by: Bryan Maass <bryan.maass@gmail.com>
Co-authored-by: Nicola Mometto <nicola.mometto@metabase.com>
This pull request refactors and streamlines the configuration for CI workflows and file path groupings in .github/file-paths.yaml and updates related workflow logic in .github/workflows/run-tests.yml. The changes focus on consolidating shared configuration, improving maintainability, and removing redundant or unused path groups.
* Add :database_type to metabot tools that return column metadata
Add :database_type to metabot tools that return column metadata in order to give metabot information about the real
database type when generating SQL.
Examples of places the :database_type now appears:
- :used_tables in :user_is_viewing context
- {table,card,metric}-details
- get-query-details
- enhanced-database-tables
* Fix test after rebase onto #66271
Fix get-table-details-database-type-test after rebase (1) use new field-id formatting (2) get-table-details no longer returns implicitly-joinable fields
in the top-level `:fields`.
* More flexible approach to resolving columns from metabot field_ids
* fix one field-stats-test
* fix field-stats-test
* support nano-ids in field ids (?)
* remove some dead code
* Exclude implicitly joinable fields when fetching cols for table
* try switching field format to use dash
* fix field IDs in a number of tests
* Kondo
* add some unit tests for new field parsing & column lookup helper fns
* Fix more tests
* Fix more tests
* fix regex
* New approach -- use wide field IDs in related_tables and add a related_by field for each table
* Fix some tests
* remove unneeded table
* cleanup, new test
* Revert to passing in & matching on expected prefix
* Address tyler's comments
* accept a regex for the prefix as well
* add back small malli group-by fix from the last PR - removed unintentionally during revision
* Make uberjar (artifact) file name configurable via env var
Add the ability to customize the output filename for the uberjar build
via the MB_JAR_FILENAME environment variable. This allows backend-only
test builds to use a separate filename (e.g., metabase-backend.jar)
to avoid overwriting the regular all-in-one metabase.jar.
* fix: add missing channels to copy & rendering logic
Mentioned webhooks and Slack where necessary to stay consistent with the
admin UI
* fix: standardize lowercase 'admin' & prefer 'email' over 'SMTP'
* fix: remove extra comma on wording
* test: adjust e2e test assertions
* test: add `DashboardSubscriptionMenuItem.unit.spec.tsx`
* test: extract single case from `describe` block
* test: fix subscriptions e2e case & text lookup on alert spec
Fixes#66199.
`metabase.legacy-mbql.schema`, now used for normalizing MBQL 4 in
place of the old bespoke normalization in earlier versions, regressed in
this weird but legitimate use case: Writing a custom aggregation like
`MAX("String Literal")`.
That's an odd thing to do, but it can be useful to provide a label in an
aggregated query. This is the MBQL equivalent of
`(SELECT ..., 'A' as label FROM ...) UNION ALL (SELECT ..., 'B' as label ...)`
This pull request refactors the Cypress runner utility to simplify error handling and improve consistency in how test failures are reported. The main change is that runCypress now throws errors on test failures instead of relying on an external exit or cleanup function, which streamlines its usage across different scripts.
This ensures consistent error propagation and simplifies the control flow for callers.
These tools already included :database_id in the results; we now include the :database_engine as well, to give metabot
a reminder about which SQL dialect it should generate.
* Bump org.apache.arrow/arrow-memory-netty from 17.0.0 to 18.3.0
Inspect dependency changes here:
* Upgrade arrow-memory-core to version 18.3.0
Updated the version of arrow-memory-core from 17.0.0 to 18.3.0.
* Consolidate changes
* Fix formatting
---------
Co-authored-by: bryan <bryan.maass@gmail.com>
Co-authored-by: Luis Paolini <paoliniluis@gmail.com>
Co-authored-by: Cam Saul <github@camsaul.com>
Co-authored-by: Cam Saul <1455846+camsaul@users.noreply.github.com>
* Fix some e2e tests from group-9
* test: add bulk sync action e2e test
* fix column selection
* Fixed module config
* Fix a few e2e tests from oss group
* Fix e2e tests from groups 46 and 47
* Fix e2e tests from groups 44 and 45
* Fix group-3 straggler, some of group-8
* Potentially disastrous change (increases item-picker-level of SchemaList and TableList) to fix test in group-8
* Fix group-6 straggler, some more of group-8
* Updates some more e2e tests from group-9 (some actual failures left)
* Fix e2e tests from group-14
* do enterprise code correctly
* library collection search
* Fix e2e tests from group-16
* Fix a couple tests from group 42
* Fix e2e tests from group-17
* Fix e2e tests from group-18
* tests: add bulk publishing scenario
fix wrong collection link in case of root collection
* UXW-2315, UXW-2314
* UXW-2294
* more UXW-2315, UXW-2314
* more UXW-2315, UXW-2314
* correct collection types
* Remove Browsing Memory Feature
* Revert "Revert "Don't return library collections by default from APIs""
This reverts commit 814c49473b.
* Revert "Revert "Fixed null type filtering""
This reverts commit 25e0d78408.
* fetch full tree to have library collections visible
* show library in permissions
* Fixing tests
* fix infinite re-render in tree
* rewrite unit tests and fix max depth problem
* fix: publish property not validated in the tree view after table publishing
* Move bulk editing apis to enterprise (#65968)
* Make sidebar sticky
* test: bulk attribute actions for tables, schemas and dbs
* fix more unit tests
* Copy preview section tests
* Copy error handling tests
* Copy undos tests
* Update test
* test: update new ee endpoints paths
* Make new bulk table APIs ee-only (fe)
- Closes GDGT-1251
* Fix global H.DataModel.visit overrides
* Remove redundant cy.reload
* Remove redundant cy.reload
* add select/deselect tests
* use helper
* data picker using proper library endpoint. Fixing types I broke. Endless Shame
* Fix Tree skeleton resizing multiple times during loading
- Closes GDGT-1253
* Improve icon
* Fill content
* don't call getCollectionItems on database collection
* add search tests and support shift click in search
* default table publish collection
* fox: reset bulk form values when selection changes
* Fix e2e tests from group-19
* fix: fix missing dependencies in useEffect gdgt-1245
* test: refactor test setup action order
* Fix e2e tests from group-20
* Fix e2e tests from group-22
* Fix e2e tests from group-23
* test: restore default db as the first step
* Kondo the kondo config
* default collections for metrics and models outside of data studio
* Move edit test to ee
* Fix list filtering now default is bronze
* Move bulk-edit-test to ee
* Fix table metadata for test
* Revert "Added `collection` argument to search"
This reverts commits:
- 21999e01
- b0f71c03ea
- 4fa0c3b422
* Fixed OSS loading of tables
* Fixed OSS loading of tables
* add tests for filtering
* Fix e2e tests from group 41 / joins-repros
* more tests
* Fix e2e tests from group-20
* Fix e2e tests from group-23
* Fix e2e tests from group-24
* Fix e2e tests from group-26
* Fix e2e tests from group-27
* Fix e2e tests from group-28
* Fix e2e tests from group-30
* constuct initial paths from library collections
* segments editor, hasEnabledNestedQueries
* Fix a couple more (something between groups 40 and 43)
* Fixed module config
* keyboard navigation
* Added library/tree API
* Fix some e2e tests from groups 40 and 42
* Fixing tests
* disable scoped search on the 'database collection', models.cy.spec should be passing now
* Fix e2e tests from group-33
* Minor cleanup
* Fix e2e tests from group-34
* Fixing tests
* Fix e2e tests from group 37 (except one depending on Recents tab)
* Starts fixing e2e tests from oss group
* cleanup
* wait to focus
* hide our analytics in the data studio
* Added library/tree API
* merge library and data studio
* change backend library flag validation to data-studio
* tidy things up
* Make bulk edit inputs non-clearable
- This prevents sending "empty" requests when clearing values
- And clearing does not really make sense here
* fix: test: table rescan and discard mocks are wrong
* fix: test: wrong mock in oss sync table schema
* Remove ^:parallel from bulk edit tests
They can contend and cause deadlocks
* fix: test: failure due to repeated mocking of same route
* test: refactor: cypress table tests
* Update e2e test
* Update e2e test
* Update e2e test
* fix browse nav section test
* Add missing token
* add check icon for published tables
* Removed unused import
* Updates some e2e tests from oss group, removes some now-invalid tests
* test: use default external db, skip transform test
* Starts fixing e2e tests from mongo group
* remove background from not active table in search
* specs
* Fix e2e tests from group-15
* hack fix ci while i'm on meetings
* scoped search is back
* Fix some e2e tests from group-32
* table picker fixes
* Fixed merge
* Fix e2e test from group-42
* merge fixes
* re-enable the recents tab on the data picker modal
* Avoid race condition on list-table-filtering-test
* fix embedding types
* mini picker unit tests
* Fixing tests
* ci, fixes
* fix library collection bugs
* cleanup data studio plugin
* fix lots of unit tests
* fix import
* fix
* Fixed unused import
* Fixed accidental #p
* ci
* revert util
* specs
* specs
* specs
* fix bug / bad type in SchemaFormSelect (#65995)
* Use actual anti join in GET /table query
* Remove old comments
* Activate token after restore
* test: unskip table metadata transform test
* test: fix search tables e2e test
* fix: test: selection in table picker
* Fix Source input disabled when filtering by Metabase transform
- Closes GDGT-1259
* test: fix: filter unused table
* Fix e2e test
* fix: test: datamodel toasts and errors
* Fix e2e test
* Fix e2e tests from group-9
* Fix flaky tests
* GDGT-1244: Fix DB tree virtualization (#66125)
* GDGT-1244: Fix tree virtualization. Replace top with translateY to improve performance
* GDGT-1244: Fix scroll bug. Refactor border styling
* bring back modeling empty state
* test: adjust publishing test
* Fix e2e test from group-22
* Fix e2e test from group-25
* fix allowing cross DB joins
* properly disable invalid items in MiniPicker
* Fix e2e test from group-26
* Fix straggler from oss group
* Starts fixing e2e tests from group-32
* unit test adjustments
* Fix remaining issue with e2e group 37
* fix item filtering
* Update Loki Snapshots
* add recent views to mini picker
* test recents and backparsing
* additional filtering test
* fix minipicker shows items it should not show
* Fixed :here in library API
* Fixed :here in library API
* UXW-2358 publishing outside of the data studio
* Fixed :here in library API
* snapshot
* Fix most e2e tests from group-32
* Lint fixes
* Fixed failing test
* revert fix
* Reapply "Added `collection` argument to search"
This reverts commit f28d9d5af9.
* Quick fix in CollectionPicker/utils.ts
* Fix flaky Remote Sync test (#65862)
* Clean up backend-test matrix (#66158)
* Fix timeseries value detection (#66161)
* Prevent driver tests (#66156)
Co-authored-by: Bryan Maass <bryan.maass@gmail.com>
* Fetch uberjar using a proper action (#66165)
* fix embedding column picking
* Trying CI with search_index.verified
* Fixed failing test
* Translate label
* Fixed failing test
* fix: published info not updated in the tree picker after deleting model
* fix: refactor: publish collection link tests
* Fixed failing test
* Fix remaining e2e tests from group-32
* FileUploadStatus unit test, quesiton reproductions
* create data studio plugin file
* fix hidden items some more
* Fix issue where confirmation modal closes then reopens again then closes again when you archive a dashboard
* fix dark mode
* UXW-2367
* UXW-2378
* Fixed merge issues
* this element is not visible because you can't even open incompatible sample database db
* Fix e2e test from group-14
* fix embedding data picker clicks
* filtering fixes for transforms
* UXW-2372
* update mini picker specs after a fix
* fix flaky join test
* Fixing in-place support for collections
* fix: cannot scroll search results
* ui: tweak search results checkbox spacing to match tree results
* fix: scroll position jump when entering field sort mode gdgt-1269
* Revert "fix: scroll position jump when entering field sort mode gdgt-1269"
This reverts commit bfecdec661.
* UXW-2279 Changes Data Studio's Glossary page background color to background-secondary
---------
Co-authored-by: Nathan Voxland <nathan@metabase.com>
Co-authored-by: Brad Anderson <brad@metabase.com>
Co-authored-by: Jakub Chodorowicz <jakub@chodorowicz.com>
Co-authored-by: Ryan Laurie <iethree@gmail.com>
Co-authored-by: Anderson <anderson@metabase.com>
Co-authored-by: Aleksandr Lesnenko <alxnddr@gmail.com>
Co-authored-by: Uladzimir Havenchyk <uladzimir@metabase.com>
Co-authored-by: Uladzimir Havenchyk <125459446+uladzimirdev@users.noreply.github.com>
Co-authored-by: Ngoc Khuat <qn.khuat@gmail.com>
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
Co-authored-by: Nick Fitzpatrick <nickfitz.582@gmail.com>
Co-authored-by: Dan Stone <wotbrew@gmail.com>
Co-authored-by: Sloan Sparger <sloansparger@users.noreply.github.com>
Co-authored-by: Stas Gavrylov <stas@metabase.com>
Co-authored-by: Aleksandr Lesnenko <alxnddr@users.noreply.github.com>
Co-authored-by: Metabase Automation <github-automation@metabase.com>
Co-authored-by: John Swanson <john.swanson@metabase.com>
Co-authored-by: Nemanja Glumac <31325167+nemanjaglumac@users.noreply.github.com>
Co-authored-by: Denis Berezin <denis.berezin@metabase.com>
Co-authored-by: dpsutton <dan@dpsutton.com>
Co-authored-by: Bryan Maass <bryan.maass@gmail.com>
* GDGT-1252: Make list view consider mini-bar-chart setting
* GDGT-1252: Fix incorrect model settings update when changing columns settings
* GDGT-1252: e2e test
* GDGT-1252: Fix unit test for ListViewItem
* GDGT-1252: Address PR comments. Minor refactoring
* GDGT-1252: Refactor DatasetEditor to get rid of modelSettings hack
* GDGT-1252: Update e2e test to cover settings persistence
* Update Loki Snapshots
* Revert Loki update
---------
Co-authored-by: Metabase Automation <github-automation@metabase.com>
### Description
Missing billing information is treated as an error for purposes of
displaying the purchase page with prices. Non-store users cannot
retrieve billing information. Thus, previously, they would always see
an error page: "Error fetching information about available add-ons."
With this change, they see the "Please ask a Metabase Store Admin" page
as intended.
We achieve this by splitting the page content for store users from that
of non-store users. Only the former needs to load data (billing and
add-on information), so only that page needs `LoadingAndErrorWrapper`.
For store users we show no dynamic content that could fail to load
(apart from a store user's email address, which we gracefully omit if
it is missing).
### How to verify
1. Ensure you have a token with the `offer-metabase-ai-tiered`
feature, but without the `metabot-v3` and `offer-metabase-ai`
features.
2. As a Metabase user with an email address that *does not* match any
of the token's organization's store user email addresses, navigate
to `/admin/metabot` and see "Please ask a Metabase Store Admin".
3. As a Metabase user with an email address that *does* match any
of the token's organization's store user email addresses, navigate
to `/admin/metabot` and see the purchase page with tier selector.
### Checklist
- [x] Tests have been added/updated to cover changes in this PR
Fixes: 2b76c38e1f
Closes: https://metaboat.slack.com/archives/C07RMUQ0GE5/p1763644357893079
This PR adds a non-destructive dev script whose only job is to detect and report on your local development environment setup.
## What it does
**`./bin/dev-requirements`** detects:
- Version managers:
- nvm, fnm, n (Node.js version managers)
- volta, asdf, mise (multi-language support)
- jenv, sdkman (Java version managers)
- Installed development tools and their versions
- Whether your setup meets Metabase's requirements
It cross-references detected versions against project configuration files:
- Node.js version from `.nvmrc`
- Clojure version from `deps.edn`
- Java 21 Temurin distribution requirement
## What it doesn't do
- No installations
- No modifications
- No assumptions about your preferences
The script is purely informational - it shows you what you have, what's required, and recommends mise for version management. Perfect for new contributors or when troubleshooting setup issues.
* feat: use boolean flag for remote-sync status
It turns out type is too overloaded to have a 'remote-synced' collection
type. This is behavior that we might want to share around other typed
collections, so this migrates us to use a boolean column for this value instead.
* fixup! e2e test find synced collection
* fixup! e2e collection fixture has correct values
* fixup! make sure flag is set on insert/update
* bump databricks
This moves over to a thin jar (or regular jar) which provides jdbc
classes and lists its dependencies. We can now bump or pin as we
like. The previous versions were uberjars that had their dependencies
packaged inside.
```clojure
clj -Sdeps '{:deps {com.databricks/databricks-jdbc-thin {:mvn/version "3.0.5"}}}'
Clojure 1.12.1
user=> (Class/forName "com.databricks.client.jdbc.Driver")
com.databricks.client.jdbc.Driver
```
```clojure
clj -Xdeps tree :extra '{:deps {com.databricks/databricks-jdbc-thin {:mvn/version "3.0.5"}}}'
org.clojure/clojure 1.12.1
. org.clojure/spec.alpha 0.5.238
. org.clojure/core.specs.alpha 0.4.74
com.databricks/databricks-jdbc-thin 3.0.5
. com.databricks/databricks-sdk-java 0.69.0
X org.slf4j/slf4j-api 2.0.9 :older-version
. org.apache.commons/commons-configuration2 2.11.0 :newer-version
X org.apache.commons/commons-lang3 3.14.0 :older-version
. org.apache.commons/commons-text 1.12.0
X org.apache.commons/commons-lang3 3.14.0 :older-version
. commons-logging/commons-logging 1.3.2 :newer-version
X com.fasterxml.jackson.core/jackson-databind 2.15.2 :older-version
...
```
contrasted with
```clojure
❯ clj -Xdeps tree :extra '{:deps {com.databricks/databricks-jdbc {:mvn/version "3.0.1"}}}'
org.clojure/clojure 1.12.1
. org.clojure/spec.alpha 0.5.238
. org.clojure/core.specs.alpha 0.4.74
com.databricks/databricks-jdbc 3.0.1
```
* bump to run drivers
* include test namespace change to trigger driver tests
* reorg
* added reinit fns to oss plugins
* ಠ_ಠ
* Added doc and eslint to prevent import of individual reinitialize functions
* added short doc to reinitialize functions to indicate they're internal
* Try to fix model references in transforms generated by metabot
* Add an e2e test
* move test later in file
* simplify implementation
* address bot comments
* add comment
...migrations.
Handles cases where emails were set to longer than the valid length of
254 bytes. Those emails would fail on this migration previously. We're
not currently using auth-identity to lookup email addresses so this
won't cause any incorrect behavior, but also if any emails like this
exist they weren't valid anyway.
Also sets the validCheckSum to ANY like some other SQL migrations to
handle cases where this migration has already been run successfully, as
on stats.
* feat: add api for manage support access grants
* Setup frontend endpoint logic and start work on UI
* Implement create access grant; revoke; UI clean up
* Move section to tools > help / Adjust main UI
* feat: migration to new auth-identity module
* Fix schema migration tests
* Adjust backend fields structure on the backend / Update types and table in the UI
* Implement invitation mechanism and auth provider
* Create a route for the access grant modal
* Fix fe types / Fix clj spaces
* Add some test placeholders
* Remove stray #p
* Add missing index
* refix broken migration
* Change the way we temporarily make the new feature always enabled
* Add email sending on grant creation
* Add login and tests
* Make auths expirable
* Remove extraneous test
* Fix bad syntax in test
* Split two column migration into two migrations
* Fix test syntax issue
* Better expires at test
* Create e2e tests / Split troubleshooting.cy.spec and move to ts
* Add fe unit tests
* Add fe unit tests
* Fix kondo
* Fix config file and settings require
* Revert added disallowed exported namespaces
* Correctly test expiration
* Add tests for revoked at behavior
* Better better expires_at test
* Fix help.cy.spec.ts / Self review updates
* Fixup last issues with module config
* Make password reset is disabled for support users
Improve some email copy
* Real token check
* Move test relying on EE-code out of OSS code
* Send invitation notification after sso user creations
* Fix module config
* fixup! BE feedback
* fixup! altered not found msg
* Use user email as fallack when name is not set on support grants table
---------
Co-authored-by: Edward Paget <ed.paget@gmail.com>
Co-authored-by: Edward Paget <edpaget@users.noreply.github.com>
* fix cut not removing text from input
* ensure we fetch all data for all possible models for smart links
* remove unused comment
* fixes smart links in metabot input sometimes degrading into their text representation and fixes some cases where wrong model name ("model" instead of "dataset") was used for metabot links
* fix unit tests
* ui for cancelling a request, handles state updates for canceled requests
* press esc to cancel metabot request, fix state being incorrect on requests after an aborted request
* add test coverage for ui, update test utils to enable aborting requests in unit tests, small bits of clean up
* add tests to processStream function
* update snapshot
Fixes#65714.
These are sent as `:number/<=` or `:number/>=` and the validation was
too strict from 57 onward. Added a test for none, min, max, and both
ends of a Between filter.
* Metabot tool endpoitns should support MBQL5
* Update filter tool to work with MBQL 5
* Fix find-outliers-wrong-query-test
Was attempting to assoc-in/dissoc-in MBQL4 query keys on an MBQL5 query.
---------
Co-authored-by: Mike Appleby <86076+appleby@users.noreply.github.com>
Running Cypress tests doesn't have to be hard.
This PR is moving us in a direction where (1) there's some logic that sets up the environment for e2e testing, and then (2) there's a simple command to invoke cypress (aka "decoupling" or "separation of concerns")
Cypress only needs to know which config to look at. It contains all the information it needs to run.
There are two ways to invoke Cypress:
1. Cypress cli
2. Cypress Module API (programmatic entry point)
They should be compatible and interchangeable. Anything you pass to the node module as options can be passed as individual flags to the cli.
```
npx cypress run --config-file path/to/some/config --spec path/to/spec.cy.spec.ts
```
is equivalent to
```
cypress.run({ configFile: "path/to/some/config", spec: "path/to/spec.cy.spec.ts" });
```
This PR removes all abstractions from the actual node module runner.
* Turn off db routing for clickhouse and it's trick.
DB routing needs a "degree of freedom" to run the exact same query
against a different schema/db/instance whatever. Clickhouse only has
database.table. so to db route on the same clickhouse instance you have
to omit the database and let it resolve during query time.
`select * from db.table` can't work. Have to do `select * from table`
and let the connection dictate db1 or db2.
In 56, tables look like this:
```clojure
{:name "upload_records_10k_20251118164620", :schema "db_eafc4346850f435e"}
```
In 57, now they look like this for single db clickhouse things:
```clojure
{:name "upload_records_10k_20251118164620", :schema nil}
```
There are two critical problems here:
1. When uploading, it's not aware to omit the schema. So a table is
created with that schema. During the next sync that table is archived
and a new one with a nil'd schema is created. So the model created for
the uploaded CSV would break as it points to an inactive table. This has
a further compounding issue because now there "appear" to be two schema:
the faked nil one and also a new one like `db_eafc...`.
2. But _all_ tables will be archived in the same manner which were
initially found in 56 or below. These tables will be marked as inactive
and new ones created. Meaning all mbql queries based on clickhouse will
fail.
I wanted to revert this change but there's actually a bit more in the PR
and some conflicts. It also does more than just enabling db-routing so I
think we can perhaps undo the damage in this way more easily.
* remove outdated test
* fix: allow analytics collection perms to be set
In #65098, we did not fully replicate the behavior of the old graph
endpoint where collection namespacing rules aren't applied to the
"analytics" namespace. Collections in this namespace can be modified as
though they are part of the root `nil` namespace. The bug was in
removing collections not part of the targetted namespace of the
`update-graph!` funciton from the graph to update.
This restores the old behavior by not excluding collection permissions
updates for collections in the "analytics" namespace in the
`remove-collections-from-other-namespaces` function.
it also updates our tests for the collection graph to make sure mock
analytics collections have the right namespace set rather than a `nil` namespace.
* fixup! reset revisions
* fixup! tests to EE module fix tests in EE module
* Update Table Editing e2e tests to use Postgres instead of SampleDB
* Update table editing e2e tests to include different column types
* Address PR comments
* move the actual sdk folders
* update rspack config and eslint config
* fix relative import in CommonSdkStoryWrapper
* allow default exports in preview.tsx
* update eslint config to allow imports from frontend/src/embedding-sdk-shared
* remove legacy eslint disable directive
* temporarily 'disable' eslint-disable-next-line no-external-references-for-sdk-package-code that has not been ported yet
* update rspack config to use new paths
* update jest config to use new paths
* add `embedding-sdk-package` to restricted imports
* copy eslint files to apply to the new sdk folders
* add eslint-disable-next-line no-restricted-imports to some places where we need emotion
* fix jest setup
* add and use setupEmbeddingSdkEnterprisePlugins
* use `useSetting` instead of `getApplicationColors` to remove enterprise import
* move defineMetabaseAuthConfig to embedding-sdk-shared
* move MetabotProvider to embedding-sdk-bundle
* fix docs generation issue about types
* fix prettier ignore file
* move back tests to ee folder
* remove ../../../ from imports
* fix eslint issues
* fix, again, jest config :sad:
* Split warehouses and warehouses-rest
* Split warehouse-schema and warehouse-schema-rest
* Split actions and actions-rest
* Split collections and collections-rest
* Split `dashboards` and `dashboards-rest`
* Split public-sharing and public-sharing-rest
* Split embedding and embedding-rest; eliminated driver deps: xrays segments sample-data tiles
* Split users and users-rest
* Split setup and setup-rest
* Some more dev functions
* Update Kondo modules config
* Fix test namespace name
* Fix test failure 🔧
* MAGE modules
* Trivial change
* Trivial change 2
* Add note
* Let's try this
* Don't rely on team label
* Update GH Action
* Fix shell script syntax
* Use same runner type for all jobs in drivers workflow
* We can skip Postgres tests too
* Fix Mage command??
* debug github output
* Test ignoring tests in Lib
* Can we skip fetching the base ref??
* Fix test-ignoring logic
* Ok so maybe we do need to fetch the base ref
* Try git-fetch --depth=1
* Fix drivers not waiting for job
* Remove TODO
* Cache Babashka dependencies
* Update step name
* Make sure Babashka deps are actually saved
* Flip the status codes around
* PR feedback and cleanup
Fixes#64135.
Previously this code would throw if multiple dashboard and/or card
params were bound to the same SQL variable. Now multiple links are
allowed, provided that only one is actually given a value.
Actually, this logic de-duplicates the values, so if multiple filters
all set the same value, that also works.
* feat: migration to new auth-identity module
* Fix schema migration tests
* Make auths expirable
* Remove extraneous test
* Fix bad syntax in test
* Split two column migration into two migrations
* Fix test syntax issue
* Better expires at test
* Better better expires_at test
* Send invitation notification after sso user creations
* Fix module config
* UXW-1866: Make MultiAutocomplete dropdown preserve open state when selecting options
* UXW-1866: Fix e2e test
* Add note on running e2e tests for translation functionality
* Fix e2e test
* Fix e2e test
Removed multiple tests related to info popovers for field picker options and saved questions.
This behavior is already covered with a unit test at:
`frontend/src/metabase/common/components/MetadataInfo/ColumnInfoIcon/ColumnInfoIcon.unit.spec.tsx`
* Fix the update keys for postgres search idx upsert
The upsert keys were calculated by looking at the first entry we were
inserting and removing `:id`, `:model`, and `:model_id`.
But the entries are not necessarily homogeneous. In this case, the first
entry is a dashboard (has no `:dashboard_id`) and the second entry is a
card (yes `:dashboard_id`).
This meant we never actually updated the dashboard_id when updating a card.
I attempted to add a test for this:
```
(deftest updates-work-properly
(mt/with-test-user :rasta
(mt/with-temp [:model/Dashboard {dash-id :id} {}]
(let [card-name (str (random-uuid))
card-id-1 (:id (mt/user-http-request :rasta :post 200 "card/" {:name card-name
:collection_id nil
:database_id 1
:display "table"
:creator_id (mt/user->id :rasta)
:visualization_settings {}
:dataset_query {:type "native" :native {:query "select 123"}
:database 1}}))
card-id-2 (:id (mt/user-http-request :rasta :post 200 "card/" {:name card-name
:collection_id nil
:database_id 1
:display "table"
:creator_id (mt/user->id :rasta)
:visualization_settings {}
:dataset_query {:type "native" :native {:query "select 123"}
:database 1}}))]
(mt/user-http-request :rasta :put 200 (str "card/" card-id-1) {:dashboard_id dash-id})
(is (= [card-name]
(map :name (:data (mt/user-http-request :rasta :get 200 "search" :q card-name)))))))))
```
The good news is that this test passes after my change. The bad news is
that it also passes without my change. I think what's going on is that
the behavior is different with `*force-sync*` on/off: in async mode, we
get the full batch that contains the updates for `[dashboard, card]`.
In synchronous mode, we get separate batches for `[dashboard]` and then
`[card]`.
I think we might want to rework tests somehow - it would be quite nice
to be able to still run things asynchronously but drain the queue before
making assertions, so the behavior wouldn't be different between
tests/prod.
* add a test
* replace password with key pair auth
* re enable snowflake tests
* add private key env var
* update private key details for temp dbs
* small changes
* fix describe database test
* fix multi method params
* update test
* update test
* try fix
* check CI
i noticed that this tests pass if we don't put a connection-uri in
there. I'm not sure what the original intent was, but the following
works just fine:
```
{:url "jdbc❄️//<account>.snowflakecomputing.com/?schema=INFORMATION_SCHEMA",
:properties {"use-password" "false",
"connection-uri" "jdbc❄️//<account>.snowflakecomputing.com/?user=<user>CI&private_key_file=%2Fprivate%2Fvar%2Ffolders%2F1d%2F3ns5s1gs7xjgb09bh1yb6wpc0000gn%2FT%2Fmetabase-secret_8808579641027884666.tmp&schema=INFORMATION_SCHEMA",
"client_session_keep_alive" "true",
"timezone" "UTC",
"warehouse" "<warehouse>",
"user" "<user>",
"ssl" "true",
"private_key_file" "/var/folders/1d/3ns5s1gs7xjgb09bh1yb6wpc0000gn/T/metabase-secret_8808579641027884666.tmp",
"week_start" "7",
"application" "Metabase_Metabase",
"account" "<account>",
"db" "\"<db>\"",
"time_output_format" "HH24:MI:SS.FF3",
"client_metadata_request_use_connection_ctx" "true"}}
```
When it goes through clojure.java.jdbc. Somehow we are getting a
connection string that doesn't play nice. I _think_ it may just be an
issue with snowflake but i'm not positive.
* remove parallel test markers
* remove parallel marker
when running tests locally with these removed:
```clojure
(run/find-and-run-tests-repl {:only 'metabase.driver.snowflake-test})
...
Ran 49 tests in 116.525 seconds
502 assertions, 1 failure, 0 errors.
{:test 49, :pass 501, :fail 1, :error 0, :type :summary, :duration 116525.006833, :single-threaded 32, :parallel 17}
```
yields only one expected error. Previously using regular stuff also
worked
```clojure
snowflake-test=> (run-tests)
Testing metabase.driver.snowflake-test
Ran 49 tests containing 502 assertions.
1 failures, 0 errors.
{:test 49, :pass 501, :fail 1, :error 0, :type :summary}
```
(again the single test that failed). but using the CLI tool i would get
errors when running them in parallel.
* fixup stupid snowflake
1. i don't know why we are running into multithreading issues now. But
i've removed those tags and the issues are gone.
2.
```clojure
snowflake-test=> (let [spec {:connection-uri "jdbc❄️//account.snowflakecomputing.com/?schema=INFORMATION_SCHEMA&user=<user>&password=<password>&db=%22v5_airports%22"}]
[(jdbc/query spec ["select CURRENT_DATABASE() db"])
(jdbc/query spec ["select CURRENT_SCHEMA() db"])])
[({:db "v5_airports"}) ({:db "INFORMATION_SCHEMA"})]
snowflake-test=> (let [spec {:connection-uri "jdbc❄️//account.snowflakecomputing.com/?schema=INFORMATION_SCHEMA&user=<user>&password=<password>&db=v5_airports"}]
[(jdbc/query spec ["select CURRENT_DATABASE() db"])
(jdbc/query spec ["select CURRENT_SCHEMA() db"])])
[({:db nil}) ({:db nil})]
```
In the above, note that putting the db in the connection string _and
url-encoding it_ (it's already quoted) is necessary.
* conditionally add db
in tests there's not a database so just be smart about it:
```clojure
{:warehouse "<warehouse>",
:additional-options nil,
:timezone "UTC",
:db nil,
:account "<account>",
:use-password false,
:time_output_format "HH24:MI:SS.FF3",
:user "<user>",
:subname "//<account>.snowflakecomputing.com/"}
```
* address pr comments
* remove outdated comment
pretty freaky comment eh? I added this when i commented out the
`handle-conn-uri` call underneath it and it caused tests to pass. That
was helpful in identifying the mechanism, but not a good solution. Put a
comment here so that that was not a final state, forgot to remove
comment when we got to a proper final state and `handle-conn-uri` was
fixed.
---------
Co-authored-by: dan sutton <dan@dpsutton.com>
Fixes#65726.
If you implicitly joined a field that happened to be the target of a
remapping, the QP was labeling it as `:remapped_from` the FK even
though (a) it wasn't really remapped and (b) there was no
corresponding FK column marked as `:remapped_to` the target column.
(Eg. `Orders.PRODUCT_ID` is remapped to `Products.TITLE`. Then I
implicitly join `Products.TITLE` directly and the QP marks it as
remapped from `Orders.PRODUCT_ID` even though it wasn't.)
That caused the FE to do odd things like drop and relabel columns.
This change adjusts the QP so it will ignore plausible remappings that
don't actually get returned from the last stage.
We get a bunch of this useless stuff in CI:
```
Placeholder until other changesets
UPDATE SUMMARY
Run: 641
Previously run: 0
Filtered out: 69
-------------------------------
Total change sets: 710
FILTERED CHANGE SETS SUMMARY
DBMS mismatch: 69
```
Looking for this in liquibase this is not logged but printed to the
console :(.
It goes through this:
```java
private static void writeMessage(String message, UpdateSummaryOutputEnum showSummaryOutput, OutputStream outputStream) throws LiquibaseException {
switch (showSummaryOutput) {
case CONSOLE:
writeToOutput(outputStream, message);
break;
case LOG:
writeToLog(message);
break;
default:
writeToOutput(outputStream, message);
writeToLog(message);
}
}
```
So we want that enum to be LOG.
```clojure
liquibase=> (clojure.reflect/reflect liquibase.Liquibase)
{:bases #{java.lang.Object java.lang.AutoCloseable},
:flags #{:public},
:members #{ ,,,
#clojure.reflect.Method{:name setShowSummaryOutput,
:return-type void,
:declaring-class liquibase.Liquibase,
:parameter-types [liquibase.UpdateSummaryOutputEnum],
:exception-types [],
:flags #{:public}}
,,,}}
```
So easy fix. Set this enum and hopefully it shuts up.
* claude fix
* simplify
* handle nested fields for athena
* nfc have position 0
* set non leafs as details only
* clean up parser and add tests
* whitespace
* disable nested fields
* kondo
* kondo
* disable nested schema test
* implement plugins for auth
* move some stuff to the oss folder as those file create issues later on
* make embedding-sdk:docs:generate pass
* move metabot to use plugins
* remove imports from package from other folders
* Revert "Adjust cards-with-filters to work with action cards (#65805)"
This reverts commit d6b857a2c6.
* Revert "Fix mapping of params to later query stages (#65651)"
This reverts commit c98e640426.
* Revert "Compute param values from card results (#64318)"
This reverts commit 2693173e7f.
* Fix SDK save dashboard question to not show collection/dashboard picker
* Fix EditableDashboard sometimes doesn't show the correct tab on first render
* Add an assertion
* left some breadcrumbs
* Update unit tests to be async
* Fix code that populates :used_tables in metabot context
This was causing :used_tables to be absent in the :user_is_viewing
context, limiting metabot's ability to get sql fixing right without
information about the available tables and their fields.
Call lib-be/normalize-query and lib/native-only-query? in
metabase-enterprise.metabot-v3.context/query-for-sql-parsing and
likewise use lib funcs to get template-tags in
metabase-enterprise.metabot-v3.query-analyzer.parameter-substitution/replace-tags
* Update ai-sql-fixer and tests to work with MBQL5 queries
* Fix metabot-v3.table-utils-test to use MBQL5 queries
* Add click behavior setup information to introduction
* Update docs/embedding/sdk/introduction.md
Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>
---------
Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>
* Update screen reader description to match the new menu copy
* Update frontend/src/metabase/embedding/components/ArchiveButton.tsx
Co-authored-by: metabase-bot[bot] <109303359+metabase-bot[bot]@users.noreply.github.com>
---------
Co-authored-by: metabase-bot[bot] <109303359+metabase-bot[bot]@users.noreply.github.com>
When we introduced a new e2e run mode, we didn't realize this piece of code needs to be updated as well.
For people who were running the backend from source (but using e2e mode) some drivers might not have worked. Example:
```
"message": "No method in multimethod 'connection-properties' for dispatch value: :sqlite"
```
This commit fixes that.
Cloud never used this to pre-fill the user on MB setup.
We have the same functionality with way less code available today, either way.
Instead of:
```
/setup/user_defaults#123456
```
do this:
```
/setup?first_name=John&last_name=Doe
```
### Description
In #62513 we added a page to offer a Metabase AI (Metabot) trial. Now
we want to offer to purchase the paid Metabot add-on.
### How to verify
1. Ensure you have a token with the `offer-metabase-ai-tiered`
feature, but without the `metabot-v3` and `offer-metabase-ai`
features.
2. Ensure you have a Metabase API key linked to that same token.
3. Navigate to `/admin/metabot` and see a page offering you to purchase
different quantities of Metabot queries.
4. Select a quantity (or go with the default of 500), check "I agree",
then click "confirm purchase".
5. Confirm that your token now has the `metabot-v3` feature and the
associated subscription has an appropriate Metabase AI product.
### Checklist
- [x] Tests have been added/updated to cover changes in this PR
### Notes
* Rename MetabotPurchasePage to MetabotTrialPage
- and add the page for the purchase of the paid add-on as
`MetabotPurchasePage`, to make the name more clearly represent the
purpose (paid vs. trial).
* Move frontend client for /api/ee/cloud-add-ons into metabase-enterprise/api
* Use ts-pattern instead of hand-crafted type guards in Metabot trial page
* Don't swallow errors when submitting Metabot purchase form
* Create Redux selector to derive isStoreUser
* Test failures to purchase Metabot trial
* Adjust implementation of POST /api/ee/cloud-add-ons/:product-type to dashes instead of underscores
* Require accepting terms of service for Metabot trial
* Move RTK and Formik error handling for Metabot into utils.ts
* Avoid use of any type in Metabot Formik error handling
* Improve control flow with side effects
* Fix typo in log message
Closes: https://linear.app/metabase/issue/CLO-4423/allow-to-buy-metabase-ai-from-the-admin-under-a-token-feature
References: https://github.com/metabase/metabase/pull/62513
Co-authored-by: Benoit Vinay <ben@metabase.com>
* Update the delete button copy for better UX
* Apply suggestions from code review
Co-authored-by: Phoomparin Mano <poom@metabase.com>
---------
Co-authored-by: Phoomparin Mano <poom@metabase.com>
* Convert plugin system to deferred initialization with undo capability
* fixed wrong rebase
* better initialization
* .
* fixed unit tests
* .
* duh
* added back whitelabel overrides for SDK
* fixup
* fixed unit test
* cleanup
* UXW-2235 Prevents 403s from SmartLinks from preventing the whole document from showing (SmartLink renders with "No access" version instead)
* UXW-2235 Adds e2e tests
* Add /livez and /readyz endpoints
* nit comment
* Update tests with new endpoints
* Update src/metabase/server/routes.clj
Co-authored-by: metabase-bot[bot] <109303359+metabase-bot[bot]@users.noreply.github.com>
* bad bot
* fix tests
* fix license test failure
https://github.com/metabase/metabase/pull/65034 bumped some jetty deps
from 12.0.24 to 12.1.3. Along for the ride came a new org:
```
org.eclipse.jetty.ee9.websocket/jetty-ee9-websocket-jetty-server 12.1.3
X org.eclipse.jetty.ee9/jetty-ee9-servlet 12.1.3 :use-top
. org.eclipse.jetty.ee9/jetty-ee9-webapp 12.1.3
. org.eclipse.jetty/jetty-xml 12.1.3
. org.eclipse.jetty/jetty-util 12.1.3
X org.slf4j/slf4j-api 2.0.17 :use-top
. org.eclipse.jetty.ee/jetty-ee-webapp 12.1.3 <--- new
```
This has a new org of `org.eclipse.jetty.ee` and no discernable license
in the pom or the jar.
Refresher we look in the pom for things like
```
<licenses>
<license>
<name>The Apache Software License, Version 2.0</name>
<url>http://www.apache.org/licenses/LICENSE-2.0</url>
<distribution>repo</distribution>
</license>
</licenses>
```
or in the jar for some common names of licenses, like in this mysql
connector jar:
```
-rw-r--r-- 134 18-Sep-2023 19:11:56 INFO_SRC
-rw-r--r-- 71049 18-Sep-2023 19:11:56 LICENSE
-rw-r--r-- 1624 18-Sep-2023 19:11:56 README
-rw-r--r-- 846 18-Sep-2023 19:12:04 META-INF/INDEX.LIST
```
I didn't see a license at either of these spots, but consulted
https://mvnrepository.com/artifact/org.eclipse.jetty.ee/jetty-ee-webapp
to see it listed as Apache 2.0 and EPL 2.0
---------
Co-authored-by: metabase-bot[bot] <109303359+metabase-bot[bot]@users.noreply.github.com>
Co-authored-by: dan sutton <dan@dpsutton.com>
Part of #64774.
Replaces quadratic (refs * columns) search in `add-alias-info` with map lookups
by ID and `:lib/source-column-alias`. They don't always succeed, but I ran the tests with a fallback case
that checked the old quadratic way too - in no case does the new match fail but the old way succeed.
- Speeds up the basic query of wide tables (see `qp.perf-test`) about 7x, from 8825ms to 1435ms.
- Reduces the big user query from #64774 from 205ms to 145ms on my machine.
Transfer-Encoding: chunked will cause Apache HttpClient to wrap body in ContentLengthInputStream,
which, in turn, quote: "NEVER closes the underlying stream".
So 'with-open' closing the stream was never propagated. And the new test with transfer-encoding
header captures that.
Part of #64774.
Now that most of the query processor machinery has been ported to use MBQL 5 and the `metabase.lib.core`
functionality, there were several needless round trips between MBQL 4 and 5.
The `driver.sql.qp/preprocess` function was converting queries from MBQL 5 to 4 up front,
and then each of three inner operations (now implemented in MBQL 5) were wrapped to convert 4 to 5 and back.
Removing this wasted effort reduced the time on my machine for the big user query from #64774 from 205ms to 145ms.
* fix cut not removing text from input
* ensure we fetch all data for all possible models for smart links
* remove unused comment
* fixes smart links in metabot input sometimes degrading into their text representation and fixes some cases where wrong model name ("model" instead of "dataset") was used for metabot links
* fix unit tests
* fix phantom selection when cutting value from input (context: https://github.com/metabase/metabase/pull/65255\#pullrequestreview-3436250849)
* ditch our manual batches of rows for bufferedwriter
Q: Why did we use a manual batch at first?
A: i noticed it was super slow and worked around this. Then realized i
was gzipping it so that was the actual slowdown. The batching was
already done and it didn't hurt anything.
Q: Why get rid of it then?
A: want to move to "cell" based counting. Rows are fine until they get
really wide and then 300 columns * 5,000 rows = 1,500,000 cells in
memory at once. Put some strings in there and we blow up. So lets switch
to cell counting rather than row counting. This is way easier if we only
care about number of rows or cells when holding them in memory. Don't
have to count the batch ourself when streaming. So if that goes row by
row, we just watch for the number of cells as the trigger to switch to
streaming and we are done.
```clojure
temp-storage-test=> (time
(mt/test-driver :postgres
(select-keys (mt/process-query (mt/native-query {:query "SELECT
row_num AS id,
-- Random Integer between 1 and 100000
floor(random() * 100000 + 1)::INTEGER AS random_int,
-- Random Float
(random() * 1000)::NUMERIC(10, 4) AS random_float,
-- Random Date within the last 5 years
(current_date - (random() * 5 * 365)::INTEGER)::DATE AS random_date,
-- Random Timestamp within the last 30 days
(NOW() - (random() * 30 * '1 day'::interval)) AS random_timestamp,
-- Random String: Generates a random alphanumeric string about 1000 characters long
(
SELECT string_agg(
chr(
(CASE WHEN random() < 0.5 THEN 65 ELSE 97 END) + (random() * 25)::INTEGER
),
''
)
FROM generate_series(1, 1000)
) AS random_string
FROM
generate_series(1, 500000) AS row_num;"})
(temp-storage/notification-rff 2000 :testing))
[:row_count :status :data.rows-file-size :notification/truncated?])))
"Elapsed time: 1457.921083 msecs"
{:row_count 15184,
:status :completed,
:data.rows-file-size 16369108,
:notification/truncated? true}
```
* Switch to cell based counts, not rows
We found in practice some people might have queries with 300
columns. When we keep 5,000 of those in memory, we end up having
1,500,000 cells in memory at once. We then are open again to OOM
issues. So this swaps over to a cell basec count rather than rows to be
a bit more granular.
* Change from 2,000 rows to 20,000 cells
the thinking here is 2,000 rows with 10 values. It's pretty arbitrary though
* rename var for cells to disk threshold
* i have to delete this test because i used `time` in a comment block
* formatting fix
If you have an empty first stage followed by "real" later stages, the frontend produces duplicate buttons in the query
builder. Cleaning those stages up removes those buttons.
* BE modularization: decouple `server` from `core`
* Split up enterprise/api and enterprise/api-routes
* Split up `settings` and `settings-rest`
* Dox for new util
* Add another util function
* Split `permissions` and `permissions-rest`
* Code fixes and additional utils
* Improved deps graph dox
When a join's conditions came from multiple tables, we used the first table as the table name, even though some of the cols came from a different table. We now display "Previous results" to denote that the lhs columns are from different tables.
* fix: update harbormaster API use
The Harbormaster API was updated to use snake_case in all params
and returns in https://github.com/metabase/harbormaster/pull/6663.
There's an backwards compatible interim period where it will support both kebab-case and snake_case in params, and keep existing returns.
The changes in this PR will make Metabase forwards and backwards compatible with the Harbormaster API changes.
After the interim period, the Harbormaster API will only take and return snake_case data.
* feat: support missing api key on hm.client/call
The OpenAPI spec is now public and the api key is not mandatory.
* new click and link behavior
* Apply suggestions from code review
Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>
* handle handlelink
* fix typo
---------
Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>
This has been a very valuable benchmark for processing complex queries,
especially large `:case` statements and nested aggregations.
Adding it with some REPL-able benchmarks in `query-processor.perf-test`.
* reverts the color-scheme meta tag
* adds the color scheme meta tag if entry point is not embed-sdk
* add backend tests for color-scheme meta tags
* add a cypress test for color-scheme
- catalog metadata is not transactional in redshift
- calls like pg_catalog.has_schema_privilege when using a string argument will resolve the oid of the schema/table etc
- if the oid cannot be resolved (e.g. table/schema does not exist) an error is thrown
We occasionally see sync failed due to relation does not exist, one cause might be concurrent test runs / cleanup deleting tables/schemas while running.
I have fixed the first half of the union to use the oid form of these calls which will result in NULL instead of an error while racing with DDL in other runs.
Need to research on external tables before changing the external table sub-query: probably want to join to svv_external_schemas and use the oid there, but not 100% yet.
* Fix docs
it throws an error in the js console about webassembly not having the
right permissions
Two ways to fix: give permissions, or rollback to a version that works.
I'm pinning to a version that works since we are close to a release.
39.2 breaks, 39.1 works.
I'm also including the recommended stylesheets
In the meantime, i'm going to investigate
resources/openapi/index_external.html.mustache as i'm not sure what uses
this and if it needs to be updated or deleted. Perhaps used by `clojure
-M:ee:doc api-documentation`?
* switch template to pinned 39.1 version
* Add breaking changes detection information
Added section on checking for breaking changes when saving questions, models, or metrics.
* Add breaking changes check to model editing documentation
Added section on checking for breaking changes when saving a model.
* Apply suggestions from code review
Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>
---------
Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>
* add aws jdbc wrapper
* add aws-iam option to auth-provider-options
* WIP: hook ssl iam in postgres
* wip
* working
* style
* postgres appdb aws iam support
* auth-provider-options-only-aws
* aws iam for mysql
* ssl
* warnings
* as error
* support appdb uri
* add unit tests
* warning
* don't automatically include useSSL
* support ssl-cert configs for mysql appdb
* more unit tests
* refactor
* integration test
* fix: always set useSSL for aws-iam in mysql appdb
* fix mysql uri
* chore
* style
* fix
* connection tests
* setup for CI
* temporarily disable until secrets are setup
* chore: set
* add warning
* Revert "temporarily disable until secrets are setup"
This reverts commit 4ca6eb4607.
* show new library icon
* UXW-2115 Address PR feedback - makes it so only the Library uses the synced collection icon, not the nested subdirectories
* UXW-2115 Fix getCollectionType condition
* UXW-2115 Removes unnecessary type changes
* UXW-2115 Reverts changes for synced collection icon
---------
Co-authored-by: Brad Anderson <brad@metabase.com>
* refactor HomeGreeting and convert to use css module over styled components
* update colors for metabot illustration
* bad ai
* rename
* pr feedback
* small adjustments
* spark failure fix
I cannot run spark locally so from CI:
```
FAIL in metabase.driver.sparksql-test/friendly-inline-strings-in-convert-to-sql-test (sparksql_test.clj:67)
:sparksql
The native query returned in query results should use user-friendly splicing
expected: "SELECT COUNT(*) AS `count` FROM `test_data`.`venues` AS `t1` WHERE `t1`.`name` = 'wow'"
actual: "SELECT COUNT(*) AS `count` FROM `test_data`.`venues` AS `t1` WHERE `t1`.`name` = 'wow'"
diff:
expected: "SELECT COUNT(*) AS `count` FROM `test_data`.`venues` AS `t1` WHERE `t1`.`name` = 'wow'"
actual: "SELECT COUNT(*) AS `count` FROM `test_data`.`venues` AS `t1` WHERE `t1`.`name` = decode(unhex('776f77'), 'utf-8')"
```
Locally hit this on an m1 mac:
```
impl.get-or-create :: create-database! failed; destroying :sparksql database “test-data” {mb-test=metabase.driver.sparksql-test/friendly-inline-strings-in-convert-to-sql-test}
java.util.concurrent.TimeoutException: Timed out after 30.0 mins
```
* fix license test failure
https://github.com/metabase/metabase/pull/65034 bumped some jetty deps
from 12.0.24 to 12.1.3. Along for the ride came a new org:
```
org.eclipse.jetty.ee9.websocket/jetty-ee9-websocket-jetty-server 12.1.3
X org.eclipse.jetty.ee9/jetty-ee9-servlet 12.1.3 :use-top
. org.eclipse.jetty.ee9/jetty-ee9-webapp 12.1.3
. org.eclipse.jetty/jetty-xml 12.1.3
. org.eclipse.jetty/jetty-util 12.1.3
X org.slf4j/slf4j-api 2.0.17 :use-top
. org.eclipse.jetty.ee/jetty-ee-webapp 12.1.3 <--- new
```
This has a new org of `org.eclipse.jetty.ee` and no discernable license
in the pom or the jar.
Refresher we look in the pom for things like
```
<licenses>
<license>
<name>The Apache Software License, Version 2.0</name>
<url>http://www.apache.org/licenses/LICENSE-2.0</url>
<distribution>repo</distribution>
</license>
</licenses>
```
or in the jar for some common names of licenses, like in this mysql
connector jar:
```
-rw-r--r-- 134 18-Sep-2023 19:11:56 INFO_SRC
-rw-r--r-- 71049 18-Sep-2023 19:11:56 LICENSE
-rw-r--r-- 1624 18-Sep-2023 19:11:56 README
-rw-r--r-- 846 18-Sep-2023 19:12:04 META-INF/INDEX.LIST
```
I didn't see a license at either of these spots, but consulted
https://mvnrepository.com/artifact/org.eclipse.jetty.ee/jetty-ee-webapp
to see it listed as Apache 2.0 and EPL 2.0
* Make Trunk analytics always run
This prevents false positives because Trunk analytics actio will now
always analyze test results, and compare them with their own test
quarantine list. It uses this information to know if it should fail the
job or let it pass.
* Introduce dry-run
When a PR is in a draft mode, we don't want to upload results but still
have to quarantine tests.
* Update wording for subscriptions and alerts section
* Update docs/permissions/application.md
Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>
---------
Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>
* toggle dark mode from command palette
* I'm sorry I built this in a dumb way Ryan
---------
Co-authored-by: Nick Fitzpatrick <nickfitz.582@gmail.com>
* upgrade to echarts v6
* Update Loki Snapshots
* Fix BE test
* update gridlines helper function
* adjust test
---------
Co-authored-by: Metabase Automation <github-automation@metabase.com>
Co-authored-by: Nick Fitzpatrick <nickfitz.582@gmail.com>
* Make sure /dirty is called when moving an entity out of the library
* Untouch embedding code / Clean up / Add some tests
* Check cache from both entities and RTK / Undo unnecessary changes
* Make sure getCollectionFromCollectionsTree gets collection from latest query
* ci: allow running stress tests on sdk tests
* type choice
* yeah this should work 🤞
* ok we should also build the sdk
* maybe this time
* do we need to setup the BE?
* Revert "do we need to setup the BE?"
This reverts commit 7e9092e2c3.
* trying the auto infer thing out
* refactor and move the logic up to dedup it
* rename output
* move up env?
* Revert "move up env?"
This reverts commit f78f72e396.
* increase timeout to make stress test with throttling pass
* increase the timeout even more :shrugs:
Part of #64774.
Hopefully this is sufficient to unblock users seeing the high CPU times
like in #64774. I discovered other, less significant performance issues
during this investigation that I'll be addressing in follow-up PRs.
### Investigation Notes
The problem query and relevant table metadata from a helpful customer
was taking 950ms to preprocess and compile on my 2022 M1 Max, running
Metabase in dev mode but with Malli schema enforcement disabled.
The query is pretty big! It uses 3 tables with 200-300 columns each,
has several stages including some from a joined card, and ends up with
several stages doing two levels of aggregation. It also includes
several *huge* `:case` expressions.
The optimizations below have reduce the time required from 950ms to 310ms
on my machine. A 3x improvement, but I think there's more to be gained in
the follow-up PRs I mentioned.
### Optimizations
Main optimizations:
- `lib.util/find-stage-index-and-clause-by-uuid` pre-computed and cached
on each stage, saved 120ms on the above query.
- `lib.metadata.cached-provider/fields` attempts to pre-load the cache
of individual `lib.metadata/field` calls, having just fetched them
all, but it used the wrong key and they were all misses instead!
- `expression-metadata` and `aggregation-metadata` are recomputed over
and over again by their stage and later stages; cache them by value
- `returned-columns` was cached at the top level and frequently missed
because of dumb details like explicit stage 1 vs. -1 for last stage.
- Switched it to caching on individual `returned-columns-method`s
with more precise cache keys.
#### New Caching Approach
Both of the latter two are using a new caching approach: a weak map
using queries as keys. Since CLJ(S) maps are immutable, any change to
the query means a new pointer.
That's still far more eviction than optimal! Most edits to a query
don't actually invalidate most of the cached values! But it's a lot
better than what we had before, and because the map's keys are weak
references, the queries and their cached data are not retained any
longer than the query itself.
### New Benchmarking Tools
I also added a `qp.perf-test` namespace with some metadata providers
with 10-, 100-, 1000- and 10,000-column tables for testing.
So far the tests it performs are quite basic, though. I'll be adapting
the user-provided query into a test case in the future, since it's an
excellent real-world example of a complex but not pathological query.
```
Updating table metadata for Table 5465 ''properati_ar.properties_rent_201607'' {mb-quartz-job-type=SyncAndAnalyzeDatabase}
:is_writable of Table 5465 ''properati_ar.properties_rent_201607'' changed from null to null {mb-quartz-job-type=SyncAndAnalyzeDatabase}
Updating table metadata for Table 5485 ''properati_ar.properties_sell_201706'' {mb-quartz-job-type=SyncAndAnalyzeDatabase}
:is_writable of Table 5485 ''properati_ar.properties_sell_201706'' changed from null to null {mb-quartz-job-type=SyncAndAnalyzeDatabase}
Updating table metadata for Table 5468 ''properati_ar.properties_rent_201602'' {mb-quartz-job-type=SyncAndAnalyzeDatabase}
:is_writable of Table 5468 ''properati_ar.properties_rent_201602'' changed from null to null {mb-quartz-job-type=SyncAndAnalyzeDatabase}
Updating table metadata for Table 5511 ''properati_ar.properties_rent_201612'' {mb-quartz-job-type=SyncAndAnalyzeDatabase}
:is_writable of Table 5511 ''properati_ar.properties_rent_201612'' changed from null to null {mb-quartz-job-type=SyncAndAnalyzeDatabase}
Updating table metadata for Table 5431 ''properati_ar.properties_sell_201802'' {mb-quartz-job-type=SyncAndAnalyzeDatabase}
:is_writable of Table 5431 ''properati_ar.properties_sell_201802'' changed from null to null {mb-quartz-job-type=SyncAndAnalyzeDatabase}
Updating table metadata for Table 5441 ''properati_ar.properties_sell_201707'' {mb-quartz-job-type=SyncAndAnalyzeDatabase}
:is_writable of Table 5441 ''properati_ar.properties_sell_201707'' changed from null to null {mb-quartz-job-type=SyncAndAnalyzeDatabase}
Updating table metadata for Table 5467 ''personal_bq_repro.events_20230519'' {mb-quartz-job-type=SyncAndAnalyzeDatabase}
:is_writable of Table 5467 ''personal_bq_repro.events_20230519'' changed from null to null {mb-quartz-job-type=SyncAndAnalyzeDatabase}
Updating table metadata for Table 5484 ''properati_ar.properties_sell_201708'' {mb-quartz-job-type=SyncAndAnalyzeDatabase}
:is_writable of Table 5484 ''properati_ar.properties_sell_201708'' changed from null to null {mb-quartz-job-type=SyncAndAnalyzeDatabase}
Updating table metadata for Table 5434 ''properati_ar.resultados_apto_credito'' {mb-quartz-job-type=SyncAndAnalyzeDatabase}
:is_writable of Table 5434 ''properati_ar.resultados_apto_credito'' changed from null to null {mb-quartz-job-type=SyncAndAnalyzeDatabase}
Updating table metadata for Table 5501 ''properati_ar.properties_sell_201704'' {mb-quartz-job-type=SyncAndAnalyzeDatabase}
:is_writable of Table 5501 ''properati_ar.properties_sell_201704'' changed from null to null {mb-quartz-job-type=SyncAndAnalyzeDatabase}
Updating table metadata for Table 5433 ''properati_ar.properties_rent_201604'' {mb-quartz-job-type=SyncAndAnalyzeDatabase}
:is_writable of Table 5433 ''properati_ar.properties_rent_201604'' changed from null to null {mb-quartz-job-type=SyncAndAnalyzeDatabase}
Updating table metadata for Table 5478 ''properati_ar.properties_sell_201702'' {mb-quartz-job-type=SyncAndAnalyzeDatabase}
:is_writable of Table 5478 ''properati_ar.properties_sell_201702'' changed from null to null {mb-quartz-job-type=SyncAndAnalyzeDatabase}
Updating table metadata for Table 5498 ''properati_ar.properties_sell_201512'' {mb-quartz-job-type=SyncAndAnalyzeDatabase}
:is_writable of Table 5498 ''properati_ar.properties_sell_201512'' changed from null to null {mb-quartz-job-type=SyncAndAnalyzeDatabase}
Updating table metadata for Table 5447 ''properati_ar.properties_rent_201709'' {mb-quartz-job-type=SyncAndAnalyzeDatabase}
:is_writable of Table 5447 ''properati_ar.properties_rent_201709'' changed from null to null {mb-quartz-job-type=SyncAndAnalyzeDatabase}
Updating table metadata for Table 5463 ''properati_ar.properties_rent_201609'' {mb-quartz-job-type=SyncAndAnalyzeDatabase}
:is_writable of Table 5463 ''properati_ar.properties_rent_201609'' changed from null to null {mb-quartz-job-type=SyncAndAnalyzeDatabase}
Updating table metadata for Table 5448 ''properati_ar.properties_rent_201509'' {mb-quartz-job-type=SyncAndAnalyzeDatabase}
:is_writable of Table 5448 ''properati_ar.properties_rent_201509'' changed from null to null {mb-quartz-job-type=SyncAndAnalyzeDatabase}
```
We unconditionally say we are updating table metadata, but then when we
actually change anything we of course include the table name.
Note here there is some kind of bug that we are changing the
table (which property??) from null to null.
* Updates 35 SVG illustrations
The same asset now works well for both light and dark mode.
* branding OnboardingIllustration, adding cloug and bug metabot to metabot logo
* PR Feedback
---------
Co-authored-by: Nick Fitzpatrick <nickfitz.582@gmail.com>
* Annotate draft transforms in user_is_viewing context with source type when possible
* add unit tests
* Move transform-source-type helper to transform utils; some refactor
* switch to use lib/native-only-query; change default transform query in tests to use mbql5
* switch batch to using lib.query/native?
* switch back to native-only-query?
* clean ns
* Compute param values from card results
seq for bool
Tests, multi-stage queries
Compute values also for params on nested stages
whitespace
Unused ns
Last stage check first
Docstring
Drop base query's last stage clauses
Address review remarks
Apply suggestions from code review
Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>
Remove redundant ops
Adjust for test
Search for col also in aggregated result columns
Look for breakoutable column only if source-uuid is present
* symb
* Rebase fix
* fix rebase, remove unused module
* post rebase, fix getting database id
* kondo
* Revamp
* limit
* Adjust the solution for impl joins
* Fix e2e
* Comment
* Cleanup
* Cleanup
* comment
* comment
* Update src/metabase/parameters/dashboard.clj
Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>
* indent
---------
Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>
* UXW-1991 Fixes various issues both new and old with the Branding page like the colors not updating until refresh and swatches not updating when setting colors back to their defaults
* UXW-1991 Uses hex format for brand colors placeholders
* UXW-1991 Fix persistence issues with setting/unsetting accent colors
we want to not hammer the network when things are down. But during
startup we don't need to circuit break these errors that we throw before
the app-db is read
* Enhance glossary section with context for definitions
This is a smaller change than I expected, since glossary terms don't show up in searches yet. Might just cut this if it doesn't add anything useful. I also wasn't sure if we should mention Metabot here.
* Add dark mode options
Adds the available options and explains that you don't have to reload the page
* Add a little clarification to the computed goal example
This confused me, so I'm adding this little clarification
* Jeff's suggested change
Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>
* Remove the bit about LLMs again
---------
Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>
* fix: allow stacktraces to be suppressed in API
Allows administrators to configure their metabase instance using
MB_HIDE_STACKTRACES to prevent stacktraces from appearing in API
response. For standard responses it replaces unhandled exceptiosn with a
generic message and 500 response. For streaming responses it leaves the
error message and strips out the trace/via keys to prevent exception
info from leaking.
* fixup! dissoc traces from map errors
* Prevent trashed documents from being edited
* Add test
* Soften the test name tone
* Disallow even updating document when archived via API
* Add BE test
* Fix Cypress test issue reference
* Update tests after merging master
* Only check for archived when not unarchiving
* Don't check archived documents when archiving or unarchiving them
* Add missing condition from merging
* Fix logs upload for drivers
When we introduced `continue-on-error` to the "test" step, we had to
update the conditionals for the test report steps as well. I've
introduced a bug by checking for a failure, instead of saying - upload
test reports whenever the job is not successful.
Hint: test.outcome means "step status BEFORE continue-on-error overrides
the exit code"
Collected timing data from 7 days of test runs.
All specs with timing data were included (simple averages).
🤖 Generated automatically by update-e2e-timings workflow
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: GitHub Actions <actions@github.com>
* add debug message type for tool calls
* rename toolCalls -> activeToolCalls
* Basic debug mode slash command to hide/show tool calls
* ToolCallDetailsModal + better formatting
* Fix type
* Fix types, some simplification
* Address minor bot comments
* small changes
* move AgentToolCallMessage into its own file and add copy buttons + tool call id to modal
* fix copied value
* add unit tests
* move failure case further downstream to mitigate any possible customer issue
* ci fixes
---------
Co-authored-by: Sloan Sparger <sloansparger@gmail.com>
* fix: improve collections graph performance
Fixes performance generating and updating a collection graph by avoiding
the cross join of all negative permissions (blocks) between groups and
collections. Instead it builds sparse graph where missing entries for
group/collection paris can be assumed to be :none permissions.
In practice this drops a collection permission update on a graph of 5K
groups and 5K collections from a greater than 10min operation on my
local machine against MySQL to a .5s operation.
* fixup! explicilty disj :root in other places
* fixup! improve some docs
* fixup! improve more docs
* fixup! seq checks
* fixup! blank lines
* fixup! convert raw sql concat into honeysql
* fixup! e2e test
* Update error message for remote sync dep checking
Try to make these error messages less jargony because they are presented
by the user interface when dependency checking content fails.
Closes UXW-2142 and UXW-1879.
* Update some missed tests
* Add a grace period to token checks
There's more work on this namespace to be done i think, but this is a
first cut at it. There's a lot of control flow by exceptions which is
how we jump around in here. Fundamentally this should be far
simpler. Will take a crack at that shortly. I also don't love that this
just hammers the `fetch-token-status` function and then memoizes inside
of that. Would prefer simpler layout of `token-features` function and
that can be aware of population issues.
Adds a simple protocol: GracePeriod, which has two functions: save! and
retrieve. On successful token check, we `save!` the response. On errors,
we `retrieve` previous responses. This obviously can be nil in the case
we have never had a successful response, and will become nil once the
grace period's internal notion of time expires.
This implementation is based on guava's cache which has a nifty
expireafterwrite functionality.
```
(*token-features*) ;; call this function to simulate the check of features
Checking with the MetaStore to see whether token 'f325...d01a' is valid...
Reporting Metabase stats: {... user counts etc}
-> #{"features come back"}
;; meanwhile clear cache to simulate elapsing of 12 hours
(*token-features*) ;; call function again to simulate check of features
2025-10-22 21:30:26,620 INFO premium-features.token-check :: Checking with the MetaStore to see whether token 'f325...d01a' is valid...
2025-10-22 21:30:26,622 ERROR premium-features.token-check :: Error fetching token status from https://token-check.metabase.com:
clojure.lang.ExceptionInfo: network issues ;; hit network issue
2025-10-22 21:30:26,632 INFO premium-features.token-check :: Using token from grace period
->> #{graceperiod network features}
...
;; here after 5 seconds (in real life will be 36 hours)
2025-10-22 21:31:00,064 WARN premium-features.token-check :: Removing token: f325...d01a from grace period cache {mb-quartz-job-type=Cache}
```
* tests that updating value populates grace period cache
* couple changes
protect against storing nil value. guava cache doesn't like it so gone
it is.
added some tests:
```clojure
(testing "Setting a new value populates the grace period cache"
(let [new-token (tu/random-token)]
(binding [http/request (fn [& _] {:status 200
:body "{\"valid\":true,\"status\":\"fake\",\"features\":[\"fake\",\"features\"]}"})]
(mt/discard-setting-changes [premium-embedding-token]
(premium-features.settings/premium-embedding-token! new-token)
(is (= #{"fake" "features"} (token-check/*token-features*)))
(is (= {:valid true :status "fake" :features ["fake" "features"]}
(token-check/retrieve token-check/grace-period new-token)))))))
```
But this ended up leaving a setting table value for this value. i don't
feel like looking into discard setting changes so i'm gonna just remove it
* attempt to simplify the api a bit
the thee important functions are now `token-information`, `harness` and
`decode-token`.
Everyone should use token-information. Harness hooks up the grace period
and retry strategies, and `decode-token` only cares about decoding a
token. It should return the token-status map {:valid true ...} or throw.
* fixup test
* clean up tests and rename a bit
* remove comment block
* remove commented out grace period
* bit of cleanup
* fix with-random-premium-token! macro
* function composes them instead of caller
rather than a caller needing to know all of the layers, i thread them
for you and let you provide all arugments in one function call.
dynamic var for needing to customize layers in tests. largely for
removing the circuit-breaker which is stateful and hard to restart if
you want to simulate errors in tests
* rename test
* Add documents to content view
* add document to audit_log
* added comment events to the audit log, publish events
* wip: transform events
* glossary
* reversion the content migrations
* include transforms in content
* remote sync audit events
* wip: action v2 event
* kondo
* fix action tests
* transform run event
* add tests for new events
* bump
* tests
* task data
* let's not change the old migration in case we might break the hashsum
* typo
* more typos
* fix bugs in h2/mysql
* add :previous-object
* fix tests
* move branch into details
* fix test
* fix: run tests only on ee
---------
Co-authored-by: Nicola Mometto <nicola.mometto@metabase.com>
* Normalize parameter type category/= to category
Stats had exactly one dashboard with one such parameter and that dashboard was immutable because the
before-update hook failed before any update could happen.
* - Improve remote-sync error messages
- Do a fetch when checking git settings to ensure the current version can be read and is being checked
* Validate git url
* Validate branch name in production mode
* Validate branch name in production mode
* Removed unneeded assoc
* Fixed formatting
* Code improvements
* Code improvements
* Allow direct path URLs
* Allow direct path URLs
* E2e Fixes
* E2e Fixes
* Allow empty branch to be passed
* Legacy MBQL normalization overhaul
* Kondo + test fixes 🔧
* Cleanup and test fixes 🔧
* Code cleanup
* Test fixes 🔧
* Extra test 🔧
* Some test fixes 🔧
* Test fixes 🔧
* Test fixes 🔧
* Test fixes 🔧
* Fix SerDes test failure
* cljfmt
* Code cleanup
* Fix normalizing :datetime with bytes modes
* Fix legacy MBQL normalization of :datetime
* Handle broken case clauses
* Set Card metadata to nil when invalid
* Fixx
* Fix Kondo
* Fix/unskip some tests / try to make flaky FE test to pass in CI
* adding branching tests, unskipping admin tests
* PR Feedback
* PR Feedback
---------
Co-authored-by: Anderson <anderson@metabase.com>
* Add type field to transforms with temporary backfill job
* add a default value for tests
* add some new tests, fix serdes
* lint
* remove debugging defs
* revert serdes spec change
* Revert "remove transforms from serdes baseline (?)"
This reverts commit e8fc8fe80bfe15e00eb031b4050877f271b8081f.
* try again to fix serialization tests
* address bot comments
* Revert "try again to fix serialization tests"
This reverts commit 13d78664b79b7f13406d89e1f8f910610ea2cce9.
* Fix search ingestion functions to use :type field + some cleanup
* Update baseline to fix serialization test
* Revert "Update baseline to fix serialization test"
This reverts commit 60cb7909e5a73d04c42d36c915a32bb28d684e37.
* fix up comment
* reset serialization_baseline to version on master
* let's try this again: manually update baseline to add type field
* fix search tests by adding correct types to transfomrs
* update metabot tool apis
* fix revision test
* simplify test
* address feedback - move type setting to the model layer
* simplify tests
* Fix up transform serdes tests: remove top-level type fields and updated_at from being serialized, and fix shape of transforms in extraction tests
* set :type after select just in case; fix serdes tests again
* guard against empty source on select
* remove redundant search spec for transforms (introduced via a previous merge)
* switch from type to source_type to avoid conflict with type field on the frontend
* add better error handling
Basically, the same case found two unrelated bugs. For one, we don't support table functions like `select * from
my_function()`. For another, `select orders from orders` is also unsupported but apparently legal (at least in
postgres). This updates sql parsing/validation to support both things.
This pull request updates the configuration and environment handling for E2E and compatibility tests, mainly to support running tests against a staging license/token server and to improve test reliability. The changes affect GitHub Actions workflows, environment variables, and some test and backend logic.
*Backend Logic for E2E Mode*
Added `is-e2e?` flag in core.clj to distinguish E2E test runs from dev/prod/test modes, allowing for custom behavior in E2E scenarios.
Modified token check logic in token_check.clj to allow E2E tests to use a staging server for license validation.
Changed store API settings in settings.clj to use staging endpoints when in dev or E2E mode, ensuring tests interact with the correct backend.
* poc of handleLink
* safer implementation
* make it return a boolean to prevent default
* fix types?
* fix static viz build
* adds tests
* make tests more consistents with fns
* test return value undefined
* Update enterprise/frontend/src/embedding-sdk-shared/lib/sdk-global-plugins.ts
Co-authored-by: metabase-bot[bot] <109303359+metabase-bot[bot]@users.noreply.github.com>
* refactor plugin call
* fix types
* removed reduntant test
* change the api to use `{handled: boolean}`
* support links rendered as anchor tags, refactor the logic and tests
* add story
---------
Co-authored-by: metabase-bot[bot] <109303359+metabase-bot[bot]@users.noreply.github.com>
* init branch
* FE - add codemirror diffing, add diff support to native query editor, fake a proposed query
* Add /ee/metabot-tools/get-transforms endpoint
Returns a sequence of all python or native query transforms.
BOT-391 [Backend] Create transform index tool API
* Add test that get-transforms does not include non-native MBQL transforms
* Add /ee/metabot-tools/get-transform-details endpoint
* allow opening metabot on transforms admin pages
* use transforms_codegen profile for metabot on transforms page
* send transform in context
* show metabot on edit + create transforms page
* accept tranform suggestions from metabot
* Cleanup malli schemas for get-transforms and get-transform-details
* add transform context on edit query page
* use transforms instead of data queries
* Parse out tables from transforms in metabot user context
* wip on creation flow
* hack in a message to metabot on save
* quick fix preview error not displaying correctly
* misc. fe improvements
* fix transform context on subsequent edits for the same transform
* half hacky way to use current editor state for context in transform editing
* Switch /ee/metabot-v3/get-transforms from :get to :post
This endpoint doesn't expect args of it's own in the body, but does accept the standard metabot conversation_id that
it echoes back.
* Show 'Apply Changes' button when a transform change is generated by the agent
* handle new transforms better
* Ensure we have a fallback for context items that don't have SQL to parse
* persist diff of transform to the conversation history
* temp - make metabot always visible on transforms admin pages
* Change button label to "Proposed changes to <transform>"
* copy change for create transform buttons in chat
* better hidden message ui
* hack - better emtpy convo text
* hide suggested prompts
* Add basic snippets tools
* ensure proposed queries go through nativeQuery.setQueryText so that template tags are parsed
* better todos
* even better todo list ui
* crude sidebar ui for query transforms diffs
* better suggestion chat ui
* good'nuff @mentions
* debug logging for search table bug
* further logging
* Ensure we parse template tags for transform edits in Metabot sidebar (#64001)
* cleaner, more copy/pastable, better flowing @mentions in metabot
* Fix metabot get-transforms api test to :post not :get
* Update metabot tools test with basic python transform tests
* Ensure transform :source and :target keys get snake cased when returned from metabot tool apis
* Add get-transform-python-library-details metabot tool api endpoint
* Enhance context with :used_tables for python transforms
* surface transforms in @-mention search
* some type fixes
* python FE progress
* python fe improvements + edit experience now works
* clear proposed on cancel
* fix accept in sidebar not saving
* better diff colors
* smaller todos
* cleaner code for todos
* improve ranking of transforms in search by hardcoding a last_viewed_at time in user-recency-expr
* first pass at nested @mentions
* Conditionally add transforms to model->db-model
* Add transforms-python to module deps for metabot
* Move native-query-transform? into transforms.util and re-use existing utils
metabot-v3.tools.transforms defined a couple of helper functions that were nearly identical to ones that have since
been added to transforms.util ns. Move some things around and make use of the existing transforms.util predicates,
which are now exported from the module's core namespace.
* lots of fe clean up
* prevent rewinding conversation for admin pages
* @mention solution that plays nicely with documents
* fix search bug
* fix type issues and few small feedback things
* Add transforms to search for Metabot (#64134)
* index transforms content
* Add support for transforms & query search in metabot search tool API
* fixup whitespace
* update to use new :fields syntax
* Fix up serach-native-query malli types
* Fix the 15-day interval for transforms to be db-agnostic
* move engine loading to metabase.search.init to break circular dep
* fix kondo
* Revert "Fix the 15-day interval for transforms to be db-agnostic"
This reverts commit 11b34e4cd4a8a5a0cf8041a1d82d487ed8c81025.
* add transform to set of appdb scorer models; unrevert db type fix
* fix kondo
* implement requisite multimethods for transforms not to error in in-place search engine
* ensure we enforce metabot/unknown contexts for transforms in in-place search backend as well
* fix filter test
* transform->postprocess to reduce naming ambiguity; add a basic transform ingestion test
* add a metabot result postprocessing test
* add transforms scoring test
* expand ingestion tests
* fix tests + kondo
* address feedback
* make suggestion models configurable for a Metabot instance
* fix @mention recents in documents, update suggestion sidebar component to not accept / reject items + calculate isViewing better
* quick fixes
* for rye guy
* Handful of fixes for transforms search (#64287)
* Add both search and search_v2 endpoints, and fix bug with search-native-query option
* add a docstring
* add a basic unit test for search-native-query option
* address comments
* better state management for suggested transofrms, move css around, clean up suggestion message code
* Update the list of source tables in Python editor when metabot changes it (#64258)
If metabot proposes changes to the list of input tables for a Python transform, update the PythonDataPicker component
to keep it in sync with the text of the transform source.
* python tables reference fix
* bug fixes and better state organization
* better draft support
* fix types
* add on leave modal, clear transforms on route leave, calc dirty state, fix onChange but for native query editor, better logic for showing cta text for transform suggestions
* Ensure transforms get deleted synchronously from the search index (#64382)
* Delete transforms from search index in t2/define-before-delete
* kondo
* fix up tests
* Stop pretending to snake_case transform source and target keys
The malli schemas were wrong, so these weren't actually getting snake_cased anyway.
See also:
https://github.com/metabase/ai-service/pull/387#discussion_r2414569482
* Fix failing test & kondo warning
* Add transforms dependency check API for metabot (#64542)
* WIP - add dependency check API for metabot
* omit bad_cards for now
* update return format
* add tests
* Fix fn call
* revert unneeded refactor
* Fix search transform-ingestion-test to use mt/native-query
* switch test to mt/native-query
* Fix module usage test
* fix up tests to use lib/native-query instead
* try to fix last test
* switch native-query-transform? to use lib.query/native?
* more mbql 5/metabase lib updates
* revert the 1-pixel fix
* Subtract a pixel when computing metabot chat filler height to avoid scrollbar flicker (#64690)
* Fix MetabotAgentSuggestionMessage only show "create" for new transforms (#64598)
https://metaboat.slack.com/archives/C09CR3MT1KR/p1760034396345789
* Surface broken cards from Metabot dependencies check API (#64781)
* fix missing sidebar diff after mbql5 change
* surface broken cards
* add test
* fix kondo
* remove unused code
* Fix up dep check question terminology & schema
* fix up test
* fix up another test
* address feedback
* Update the transform source schema to use ::queries.schema/query
* remove unused require
* address FE feedback
* fix modules config test
* @mentions tests and improvements
* couple of e2e tests fixes
* prevent leave confirmation when saving
* fix most of unit tests
* add back searchable models in metabot @mentions
* More tests for transform search + transform dependencies tool, and some new future-proofing read checks (#64896)
* add some future-proofing read checks to transform deps checks + related tests
* add some more transform search tests
* remove 404 check; move read check later in impl
* convert from toucan transform instance to metadata
* clean ns
* Move honeysql helper to simplify module dependencies
* fix more function calls in tests
* fix more calls
* fix (hopefully) last two calls
* kondo fix
* fix module config
* Fix detection of requires in threading macros
* Add metabase.driver.sql.query-processor back as a driver API namespace
* Add tests for metabot tools snippets and check-transform-dependencies endpoints (#64894)
* Add tests for metabot tools snippets API endpoints
* Add test for metabot tools check-transform-dependencies endpoint
* Remove trailing blank line to appease linter
* copy over appdb implementations of add-interval-honeysql-form to h2x; add back the sql.qp version for other drivers
* switch other driver methods back to sql.qp
* update relevant callsites
* move some test logic
* fix up error
* restore :added meta
* fix tests, move over one more pg timestamp helper
* oops, update callsites
* make function public
* add MetabotChatEditor.unit.spec.tsx
* add links.unit.spec.ts
* add AIMarkdown.unit.spec.tsx
* add MetabotAgentTodoMessage.unit.spec.tsx
* add MetabotAgentSuggestionMessage.unit.spec.tsx
* add metabot/state/reducer.unit.spec.ts w/ tests for suggested transforms state
* add e2e test coverage for create + update flow for both native sql + python transforms
* FE pr feedback and lint fix
* unit test fixes
* fix search test
* add temp metabot toggle button in admin nav
* fix a couple of failing FE tests
* attempt fix ci flake issue with metabot.unit.spec.tsx
* remove metabot-only restriction on transform search
* remove redundant logs in transform api endpoints to address ngoc's comment
* fix a number of transform search tests
* fix another test
* fix hopefully last search test
* bot feedback
* bot feedback
* fix function name
---------
Co-authored-by: Sloan Sparger <sloansparger@gmail.com>
Co-authored-by: Mike Appleby <86076+appleby@users.noreply.github.com>
Co-authored-by: Mihael Konjevic <konjevic@gmail.com>
Co-authored-by: Tyler Plude <tsplude@gmail.com>
Make native validation allow using current-level aliases in other clauses
`select a as b from table where b > 1` wasn't passing validation because the b alias wasn't in scope in the where.
This should now be fixed.
Also, native validation keeps more context when generating references, which should allow for better error messages.
Instrumentation adds a lot of overhead. Unfortunately, even 30 seconds timeout wasn't enough for some of our unit tests. We're doubling that value in order to minimize the chances of this workflow halting/failing again.
* Add workflow_dispatch trigger
* Rename workflow to 'Frontend Test Coverage'
* Fix remote sync settings test
* Fix sync type setting validation test
* Fix check sync task type is production
* Update plain type to collection_type
* Cascade changes to collection type
* Update deps checks when collection type changes
* Fix task type check in export
* move setting check to api for better status report
* add tests for sync-type
* Don't mark newly created entities as updated or removed UXW-1806
* UXW-1821 enabled badge
* UXW-1818: hide branch picker and push button for non-admins
* cleanup and UXW-1814
* remove border from the branch picker button
* cleanup
* UXW-1829 update diff modal ui
* UXW-1807: token should appear masked
* Allow setting of nil token
* UXW-1833: show backend error when saving invalid git settings
* Add new collection update/create events
* Remove per collection imp/exp; change dirty status table
* Redo event handling add removed status
* Sync object table on syncs
* Fix status names
* Added /current-task/cancel API and track version in remote_sync_task
* Add is_remote_synced to api responses
* Fix adding is_remote_sync hydration
* Fixed compilation error
* remove hashp
* Fix loading is_remote_sync hydration
* Resolve ambiguous reference
* Fix n+1 detector
if it gets a heterogeneous set of things we should hydrate each type
separately.
* Add can-create? handlers
* remote sync: a few bugfixes and improvements from my bugbash (#64372)
* Set `collection.type` on archive/unarchive
When unarchiving or archiving collections that have the `remote-synced`
type, we need to set their new type appropriately.
When archiving, we should give them a normal (nil) type.
When unarchiving any collection *to* a remote-synced collection, we
should set the type to `remote-synced`.
* Allow serdes without archived items
and use it in remote sync export.
During dependency checking, we do *not* want to exclude archived items:
if card A uses card B, and card B is archived, we need to include Card B
in the list of dependencies! Otherwise we'll say that it's fine to
remote sync card A when it's not.
* Add tests
* Documents are descendants of collections
* don't include cards as document descendants
* fix tests, dashboard descendants args
* remove #p and unnecessary query
* remove another `#p`
* Update auto sync token to enable import job
* More is_remote_sync hydration fixes
* Fix lints and document oss serdes
* Fix cards to be valid for update events
* Don't actually import if nothing has changed in the remote repo
* Remove hashp
* UXW-1817: change branches should update content
* UXW-1813 ensure sync dirty state is properly invalidated
* UXW-1864 fix how collections changes appear in the diff
* Fix ee-available? check
* Improve is_remote_synced hydration
* Fix some more oss/ee issues
finish remote ChangeLog model
* fix invalidation on removal
* fix missing require
* Don't need to handle transforms directories specially
* fix tests to make sure state is fully reset
* remove duplicate require
* Fix deadlock on import
* Fixing tests
* Fix document tests
* Fix kondos
* handle removed sync_status
* UXW-1851: show moving out of synced collection error
* feature flag, misc fixes
* dirty state tracking for snippets
* Fix ns typo
* Add remote-sync token and config
* enforce remote_sync token feature on the FE
* allow cancelling the remote-sync
* Track timeline
* Force re-import when configuring for production mode
* Fixed compilation error
* Force re-import when configuring for production mode only when auto-sync is enabled
* Throwa 400 error if there are dirty objects when doing an import
* force import button
* Better archiving checks with remote synced collections
* Better tests for documents and remote-sync deps
* Remove hashp
* remote hashp
* Remove other hashp
* More consistent collection write checks
* Fix broken collection tests
* Fix broken documents tests
* improve imports/exports frontend
* Implement can-create? for timeline-event
* tweak copy
* Always update branch setting
* Initial Commit
* Fix more tests
* Remove bad parallel annotations
* Show error toast when adding a collection that has references outside of the repo
* reasonable structure, error handling
* timelines editing is less frequent, we can invalidate dirty state on any edit
* Fix dirty state api tests for mysql
* bring error details back
* Fix collection move tests
* Fix remove parallel from can_restore tests
Splitting this into smaller tests seems to introduce some kind of state
that causes tests to fail in parallel
* Fix make logs use unique names for be tests
* Bump toucan2 versions
* Fix var name
* Fix check that there are exportable collections
* Add test for exportable check
* Started on e2e test
* Fix issue with merge
* Fix arguments to card descendants
* Handle new ingestable interface
* Put back changes in cypress-runner
* Fix remaining failing enterprise tests
* Fix one set of app tests
* Clean remote sync state in obj tests
* single collection
* Fix unit test - enterprise getIcon function only applied when new feature is enabled (ResultItem.unit.spec.tsx)
* Fix ts issues / Clean up
* FE fixes
* Remove parallel annotations
* Fix missing import from merging master
* Fix the fix
* always create collection to sync
* no unsyncing
* Remove associating location to trash from search
* Prevent users from switching branches with unsynced changes + show warning
* Create remote-sync collection when it doesn't exist
* Prevent changing to production with unsynced changes
* Make sure dirty state refetches on admin settings load
* single synced collection ui update
* Cannot change to production mode when there are unsynced changes
* Fix failing query
* unquote branches
* fix eastwood
* fix timeline event publishing
* fix import
* Move dashboard remote sync api tests into ee
...where they can reference the setting
* Created /stash API and renamed create branch api to /create-branch
* Fix settings issue
* Fix remaining permissions message tests
* Make sure drop target always works for trash
* Add checks for if we're in trash elsewhere
* cleanup
* git-sync to remote-sync
* more git-sync to remote-sync
* Don't allow export if it overwrites an external commit
* - Don't assume a "main" branch
- Attempt an import when no remote-sync branch exists when enabling dev mode
* Fix creating remote sync library
* refactor to middleware
* hack: add a delay for refetching instance settings or dirty state after remote sync task finishes
* UXW-1953: confirmation flow when switching branches in prod
* Update modal when switching branches with dirty state / Component refactorings
* Create a branch off the last-used commit, not the branch itself
* Add options modal when trying to push while remote is ahead
* Support force option on export
* Update branch setting and complete task in txn
* Let export! set branch in stash and fix tests
* Fix PR feedback nits
* Update enterprise/backend/src/metabase_enterprise/remote_sync/api.clj
* Consistent documentation style in remote-sync
* Add option to pull from remote / Adjust push/pull icons / Adjust out-of-sync options modal
* modals UI polish
* constants clean up
* Add api schemas and make error messages consistent
* fix check-403 to write-check
* Apply suggestions from code review
Co-authored-by: metabase-bot[bot] <109303359+metabase-bot[bot]@users.noreply.github.com>
* Clean up a couple components / Move things around
* Address renaming suggestions
* Fix var names and bad merge
* double->float in schema
* Share error handling
* Update enterprise/backend/src/metabase_enterprise/remote_sync/settings.clj
* Update enterprise/backend/src/metabase_enterprise/remote_sync/settings.clj
* Update enterprise/backend/src/metabase_enterprise/remote_sync/settings.clj
* Addressing small PR comments
* Refetch properties when branch is created (current branch changed)
* Make sure /current-task is only called when import and export are not in progress
* Replace unset color variables
* Improve element padding on side panel
* Remove a couple trailing spaces reported by clj whitespace linter
* Show error when settings request fails / Make sure current-task also waits for the PUT settings mutation
* Fix document perm tests
* Allow empty remote-url, moved import and export calls to impl
* Fix minor prettier issue
* Fix another lil lint issue
* temp
* e2e push / pull
* Unify more of the import implementation
* fix CI failures
* fix mysql ci failures
* Address metabot critiques
* Address additional metabot feedback
* last round of docstring feedback
* library admin settings e2e
* Add attribution to TODO
* fix ws
* fix missed commiting file
* fix another missing file
* production switch branches
* More clojure like docstrings
* More docstring fixes
* adding branch switching test
* Address Nick's PR review comments (most of them)
* Fix more nits
* Resolving Nick's comments part 2
* Improving remote-sync settings handling
* attempt to repair synced_folder icon
* Fail if attempting to connect to an uninitialized repo
* correct the name of synced_collection.svg
* add new repository icon
* Add default-branch to source protocol
* Update Git Sync to Remote Sync in Admin settings (#64993)
* Make api docstrings and errors more generic
* Use check-400 to simplify api
* fix ws
* Rollback BaseSMTPConnectionForm.tsx (fixes a e2e test)
* fix e2e tests?
* Fix small merge issue
* Always use `last-version` instead of caring about import vs. export version
* starting warning modal tests
* Fetch source at the beginning of import/export operations
* Fix broken test
* Delete branches that were deleted on remote
* Fix missing test suffix
* Fix eastwood
* Adjust branching e2e test, skip failing tests
---------
Co-authored-by: Edward Paget <ed.paget@gmail.com>
Co-authored-by: Nathan Voxland <nathan@metabase.com>
Co-authored-by: Aleksandr Lesnenko <alxnddr@gmail.com>
Co-authored-by: Edward Paget <edpaget@users.noreply.github.com>
Co-authored-by: Aleksandr Lesnenko <alxnddr@users.noreply.github.com>
Co-authored-by: Anderson <anderson@metabase.com>
Co-authored-by: eliotmetabase <eliot@metabase.com>
Co-authored-by: metabase-bot[bot] <109303359+metabase-bot[bot]@users.noreply.github.com>
Co-authored-by: Nick Fitzpatrick <nickfitz.582@gmail.com>
Co-authored-by: Maz Ameli <maz@metabase.com>
* init branch
* FE - add codemirror diffing, add diff support to native query editor, fake a proposed query
* Add /ee/metabot-tools/get-transforms endpoint
Returns a sequence of all python or native query transforms.
BOT-391 [Backend] Create transform index tool API
* Add test that get-transforms does not include non-native MBQL transforms
* Add /ee/metabot-tools/get-transform-details endpoint
* allow opening metabot on transforms admin pages
* use transforms_codegen profile for metabot on transforms page
* send transform in context
* show metabot on edit + create transforms page
* accept tranform suggestions from metabot
* Cleanup malli schemas for get-transforms and get-transform-details
* add transform context on edit query page
* use transforms instead of data queries
* Parse out tables from transforms in metabot user context
* wip on creation flow
* hack in a message to metabot on save
* quick fix preview error not displaying correctly
* misc. fe improvements
* fix transform context on subsequent edits for the same transform
* half hacky way to use current editor state for context in transform editing
* Switch /ee/metabot-v3/get-transforms from :get to :post
This endpoint doesn't expect args of it's own in the body, but does accept the standard metabot conversation_id that
it echoes back.
* Show 'Apply Changes' button when a transform change is generated by the agent
* handle new transforms better
* Ensure we have a fallback for context items that don't have SQL to parse
* persist diff of transform to the conversation history
* temp - make metabot always visible on transforms admin pages
* Change button label to "Proposed changes to <transform>"
* copy change for create transform buttons in chat
* better hidden message ui
* hack - better emtpy convo text
* hide suggested prompts
* Add basic snippets tools
* ensure proposed queries go through nativeQuery.setQueryText so that template tags are parsed
* better todos
* even better todo list ui
* crude sidebar ui for query transforms diffs
* better suggestion chat ui
* good'nuff @mentions
* debug logging for search table bug
* further logging
* Ensure we parse template tags for transform edits in Metabot sidebar (#64001)
* cleaner, more copy/pastable, better flowing @mentions in metabot
* Fix metabot get-transforms api test to :post not :get
* Update metabot tools test with basic python transform tests
* Ensure transform :source and :target keys get snake cased when returned from metabot tool apis
* Add get-transform-python-library-details metabot tool api endpoint
* Enhance context with :used_tables for python transforms
* surface transforms in @-mention search
* some type fixes
* python FE progress
* python fe improvements + edit experience now works
* clear proposed on cancel
* fix accept in sidebar not saving
* better diff colors
* smaller todos
* cleaner code for todos
* improve ranking of transforms in search by hardcoding a last_viewed_at time in user-recency-expr
* first pass at nested @mentions
* Conditionally add transforms to model->db-model
* Add transforms-python to module deps for metabot
* Move native-query-transform? into transforms.util and re-use existing utils
metabot-v3.tools.transforms defined a couple of helper functions that were nearly identical to ones that have since
been added to transforms.util ns. Move some things around and make use of the existing transforms.util predicates,
which are now exported from the module's core namespace.
* lots of fe clean up
* prevent rewinding conversation for admin pages
* @mention solution that plays nicely with documents
* fix search bug
* fix type issues and few small feedback things
* Add transforms to search for Metabot (#64134)
* index transforms content
* Add support for transforms & query search in metabot search tool API
* fixup whitespace
* update to use new :fields syntax
* Fix up serach-native-query malli types
* Fix the 15-day interval for transforms to be db-agnostic
* move engine loading to metabase.search.init to break circular dep
* fix kondo
* Revert "Fix the 15-day interval for transforms to be db-agnostic"
This reverts commit 11b34e4cd4a8a5a0cf8041a1d82d487ed8c81025.
* add transform to set of appdb scorer models; unrevert db type fix
* fix kondo
* implement requisite multimethods for transforms not to error in in-place search engine
* ensure we enforce metabot/unknown contexts for transforms in in-place search backend as well
* fix filter test
* transform->postprocess to reduce naming ambiguity; add a basic transform ingestion test
* add a metabot result postprocessing test
* add transforms scoring test
* expand ingestion tests
* fix tests + kondo
* address feedback
* make suggestion models configurable for a Metabot instance
* fix @mention recents in documents, update suggestion sidebar component to not accept / reject items + calculate isViewing better
* quick fixes
* for rye guy
* Handful of fixes for transforms search (#64287)
* Add both search and search_v2 endpoints, and fix bug with search-native-query option
* add a docstring
* add a basic unit test for search-native-query option
* address comments
* better state management for suggested transofrms, move css around, clean up suggestion message code
* Update the list of source tables in Python editor when metabot changes it (#64258)
If metabot proposes changes to the list of input tables for a Python transform, update the PythonDataPicker component
to keep it in sync with the text of the transform source.
* python tables reference fix
* bug fixes and better state organization
* better draft support
* fix types
* add on leave modal, clear transforms on route leave, calc dirty state, fix onChange but for native query editor, better logic for showing cta text for transform suggestions
* Ensure transforms get deleted synchronously from the search index (#64382)
* Delete transforms from search index in t2/define-before-delete
* kondo
* fix up tests
* Stop pretending to snake_case transform source and target keys
The malli schemas were wrong, so these weren't actually getting snake_cased anyway.
See also:
https://github.com/metabase/ai-service/pull/387#discussion_r2414569482
* Fix failing test & kondo warning
* Add transforms dependency check API for metabot (#64542)
* WIP - add dependency check API for metabot
* omit bad_cards for now
* update return format
* add tests
* Fix fn call
* revert unneeded refactor
* Fix search transform-ingestion-test to use mt/native-query
* switch test to mt/native-query
* Fix module usage test
* fix up tests to use lib/native-query instead
* try to fix last test
* switch native-query-transform? to use lib.query/native?
* more mbql 5/metabase lib updates
* revert the 1-pixel fix
* Subtract a pixel when computing metabot chat filler height to avoid scrollbar flicker (#64690)
* Fix MetabotAgentSuggestionMessage only show "create" for new transforms (#64598)
https://metaboat.slack.com/archives/C09CR3MT1KR/p1760034396345789
* Surface broken cards from Metabot dependencies check API (#64781)
* fix missing sidebar diff after mbql5 change
* surface broken cards
* add test
* fix kondo
* remove unused code
* Fix up dep check question terminology & schema
* fix up test
* fix up another test
* address feedback
* Update the transform source schema to use ::queries.schema/query
* remove unused require
* address FE feedback
* fix modules config test
* @mentions tests and improvements
* couple of e2e tests fixes
* prevent leave confirmation when saving
* fix most of unit tests
* add back searchable models in metabot @mentions
* More tests for transform search + transform dependencies tool, and some new future-proofing read checks (#64896)
* add some future-proofing read checks to transform deps checks + related tests
* add some more transform search tests
* remove 404 check; move read check later in impl
* convert from toucan transform instance to metadata
* clean ns
* Add tests for metabot tools snippets and check-transform-dependencies endpoints (#64894)
* Add tests for metabot tools snippets API endpoints
* Add test for metabot tools check-transform-dependencies endpoint
* Remove trailing blank line to appease linter
* add MetabotChatEditor.unit.spec.tsx
* add links.unit.spec.ts
* add AIMarkdown.unit.spec.tsx
* add MetabotAgentTodoMessage.unit.spec.tsx
* add MetabotAgentSuggestionMessage.unit.spec.tsx
* add metabot/state/reducer.unit.spec.ts w/ tests for suggested transforms state
* add e2e test coverage for create + update flow for both native sql + python transforms
* FE pr feedback and lint fix
* unit test fixes
* fix search test
* add temp metabot toggle button in admin nav
* fix a couple of failing FE tests
* attempt fix ci flake issue with metabot.unit.spec.tsx
* remove metabot-only restriction on transform search
* remove redundant logs in transform api endpoints to address ngoc's comment
* fix a number of transform search tests
* fix another test
* fix hopefully last search test
---------
Co-authored-by: Sloan Sparger <sloansparger@gmail.com>
Co-authored-by: Mike Appleby <86076+appleby@users.noreply.github.com>
Co-authored-by: Noah Moss <noahbmoss@gmail.com>
Co-authored-by: Mihael Konjevic <konjevic@gmail.com>
Co-authored-by: Noah Moss <32746338+noahmoss@users.noreply.github.com>
Due to `useEngineDocMarkdownContent` changes the dynamic imports are needed for `import(`docs/databases/connections/${docFileName}.md`)`
Co-authored-by: Jonathan Sheely <jsheely@thirtytech.net>
* Collections are delete-able
The `can_delete` flag for collections was the only missing bit to have
collections be deletable, since I added an API endpoint to delete
collections a while ago! I thought FE work was required, doh.
Fixes UXW-1090
* Fix e2e tests
* Rework X-Rays to use Lib
* Test change
* Fix the last few failing tests
* Remove dead code
* PR feedback
* Claude updates
* Fix typos
* Oops
* e2e test fix 🔧
* Assert premium features for some middleware
* Fix an existing bug
These tests started failing here, which revealed that they formerly
never actually were testing what they thought they were.
A few reasons:
- they turned on `advanced-permissions` feature by itself, so sandboxing
was always off here
- they set up sandboxing in a nonstandard way
- one had a typo so we weren't even wrapping the body with
`with-download-perms!` (it was a separate block entirely)
This hid the bug that some of the tests were intended to reproduce - if
a card used for sandboxing had a native query, then we required that you
have download permissions on the entire database, which is wrong.
* fix test
* make assertions with/without tokens for sandboxing
* Add comments about why `:feature :none`
* Update enterprise/backend/test/metabase_enterprise/sandbox/query_processor/middleware/sandboxing_test.clj
Co-authored-by: Cam Saul <1455846+camsaul@users.noreply.github.com>
* add a nicer docstring and change fn name
replace `any-native-unsandboxed-stage?` with
`any-native-stage-not-introduced-by-sandbox?`
* Block without feature enabled
* CR response
---------
Co-authored-by: Cam Saul <1455846+camsaul@users.noreply.github.com>
* Allow any scheme for URLs, not just http/https
* Explicitly allow app:// and capacitor:// schemes
* fix flakiness in e2e test
* Only accept HTTP/HTTPS when no protocol is specified
---------
Co-authored-by: Uladzimir Havenchyk <uladzimir@metabase.com>
* ugly solution #1
* revert the ugly part
* refactor and tests
* fix(modular): make links/click behavior that link to the same origin as the host app work and open them on a new tab
* update tests to expect a click behavior
* clean up tests
* tests for the same domain issue
* add also basic support for links, just in case
* Update e2e/test-component/scenarios/embedding-sdk/dashboard-click-behavior.cy.spec.tsx
* refactor tests as per feedback to use mockIsEmbeddingSdk
* don't remove column settings
* Remove unnecessary default value
* Remove unnecessary export
* Show move to trash icon in CollectionBrowser
* Make the trash button easier to work with
* Add a dropdown
* Fix icon
* Archive resources in the collection browser
* Add tests
* Update UX
* Prevent external type change
* Remove custom style
* subtler add data button in side nav
* fix tests
* update test
* fix unrelated broken test
* fix another unrelated broken test
* try fixing test again
* undo changes to unrelated tests
* Cursor's attempt at fixing related test
* Extend db connection timeout in tests
We allow for 10 seconds in the app, and 3 seconds in test. I extend that
3 seconds to 5 seconds. This should really only affect bigquery.
In bigquery, we test if you "can connect" by connecting to the project
and then attmepting to find any datasets that you are able to see. This
is important because each test run connects to the same project but is
scoped down to fewer datsets. It's possible we could change this
behavior and return if you can list datasets at all, but it is nice if
you can see at least one dataset based on your inclusion and exclusion
filters.
For reference, the filters for my current CI run are:
```clojure
bigquery-cloud-sdk=> (update details :service-account-json (constantly "<<redacted>>"))
{:project-id nil,
:service-account-json "<<redacted>>",
:dataset-filters-type "inclusion",
:dataset-filters-patterns "sha_4c3ca6238018e2b60025297f3e5c39671588adf2_QLGKZKIAOSQXOGGDURQY",
:include-user-id-and-hash true,
:project-id-from-credentials "metabase-bigquery-driver"}
```
In my repl, i'm seeing the following behavior:
```clojure
bigquery-cloud-sdk=> (time (let [client (database-details->client details)
project-id (get-project-id details)
datasets (.listDatasets client project-id (u/varargs BigQuery$DatasetListOption))
]
(vec (for [^Dataset dataset (.iterateAll datasets)
:let [dataset-id (.. dataset getDatasetId getDataset)]
]
dataset-id))))
"Elapsed time: 2692.292292 msecs"
["BQ_tupac_sightings"
"Shakespeare"
...
"v6_sample_dataset"]
bigquery-cloud-sdk=> (count *1)
273
```
So just enumerating all datasets takes ~2.7 seconds so we are flirting
with the fail limit of 3 seconds.
I extend this to 5 seconds and this should only give a bit of breathing
room to bigquery and everything else remains the same.
* fix flakiness in e2e test
---------
Co-authored-by: Uladzimir Havenchyk <uladzimir@metabase.com>
* Add migrations
* Add new columns to serde spec
* Update serde baseline
Will change again in follow up once sql-jdbc behaviour is ported
* Extend sql-jdbc describe-table
* Sync new field columns
* Extend field our-metadata to the new sync columns
* Fix tests
* Add postgres support new field columns
* Add generic driver tests for nullability and defaults
* Extend clickhouse test driver to :not-null? :default field defs
* Add new columns to mysql describe-fields-sql
* Add new features for better test specification
- :describe-is-nullable
- :describe-default-expr
In both cases if the feature is unsupported, the driver must return `nil` for the respective field metadata keyword.
* Add :database-is-generated test and driver feature
* Fix load-data inserts when rows empty
* Do not return generated-column-sql for redshift
* Fix snowflake-test/describe-table-test
* Fix bigquery load-tabledef when table has no rows
* Use generation_expression rather than :extra for mysql/mariadb
* Fixup driver features
* Amend pk-sync-test to verify the new :database_is_pk behaviour more explicitly as the semantic type changes
* Add sync tests for default/generated/nullable
* Formatting
* Add SQLite support for generated/nullable/default
* Add SQL-Server support for generated/nullable/default
* Only test dynamic dataset databases for now
* For druid JDBC all columns (apart from __time) are nullable
* Vertica does not support generated columns
* Starburst does not support generated columns
* Presto does not support generated columns
* Revert accidental database-required behaviour change
* Assert table synced
* Use def'd datasets rather than inline
* Oracle features
* Vertica features
* Opt starburst, presto out of defaults
* Add :test/create-table-without-data to deal with mongo
* Add some new lines to mig yml
* Add note about sync to driver test
* Do not spam logs for pk changes on upgrade
* Further logging guard refinement
* Fix migration conflicts
* [Search] Separate out embeddable_text from searchable_text and use labeled format for embedding
* add embeddable_text to test data
* update mock-embeddings in tests to use new format for embeddable strings
* Fix more tests
* Fix more tests
* Fix last tests
* Fix last test
* improve comments & add a test that transform fns are not applied
Avoids FE ordering depending on hash iteration order on initial sync - in particular avoids
breakage in tests relying on id positionally if hashes change.
* Remove double beforeEach setup probably from a mistake
* Fix the flaky test. I hope...
* Attempt to fix the test that failed the interception
* Revert change
* Fix the flake on Cypress CI
Closes QUE-2613.
`convertTimezone` now checks the type of the field ref, since we might not know to make the compiled
column a timezone-backed column when the previous stage is a native query.
* Remove double border
* Remove overflowing corners on functionbrowser
* Fix HelpText header background in dark mode
* Remove dot in customexpression editor
* Fix dark mode completions in native editor
* Move color fixes to CodeMirror component
* Move CodeMirror cursor fix in darkmode to shared CodeMirror module
* Move completion styles to shared CodeMirror component
* Enable highlighting of partials in Python completions
* Remove more shared CM code
* Fix e2e test
* Fix timout in e2e test
This addresses most of QUE-2686 #64774 but a second PR is required to fix a
56-specific issue that doesn't apply to `master`.
The FE was not using the new
`lib.metadata.cached-provider/cached-metadata-provider` except where
it gets added by `lib/query`. Previously, the FE constructed the
*uncached* MP once (ish) and then passed it to `lib/query` many times.
Each of those queries got its MP wrapped with `cached-metadata-provider`
but the next time the query got converted via `lib/query` its cached
values (eg. `visible-columns`, `returned-columns`) were lost.
This PR wraps the `lib.js.metadata/metadata-provider*` with
`cached-metadata-provider` from the start, so the cached values have
the same lifespan as the `MetadataProvider`, rather than each individual
query.
I also noticed that `/query_metadata` requests don't use
`lib-be.metadata.jvm/with-metadata-provider-cache`, so they end up
recreating `MetadataProvider`s for the same underlying database over and
over. That increases AppDB traffic and CPU usage to recompute
`returned-columns` etc. repeatedly. Both of those scale fast with the
complexity of the query.
* disable excessive spam in the prs
* to-revert: change source code to trigger the build
* test2
* :shrugs:
* Revert "to-revert: change source code to trigger the build"
This reverts commit 37f6abb085.
* clean up and comment
This fixup broke because QP started returning the db-shaped
viz-settings (e.g. `:table.columns`) instead of the normalized
viz-settings (e.g.
`:metabase.models.visualization-settings/table-columns `) under
`[:json_query :viz-settings]`.
The existing test for this fix did not catch the regression because the
shape of viz settings was faked out for testing, and did not get
transformed like real viz settings. `{:ver 2}` stayed as `{:ver 2}`,
while `{:table.columns []}` is transformed into
`{::visualization-settings/table-columns []}`
* Fix date range picker text inputs not in sync with calendar inputs (#64602)
* Unskip and update test
* Add repro for #64602
* Add unit tests
* Add issue tags
* Rename date to displayedDate
* Update the last instance of deprecated `FormRadioWidget`
This commit replaces a deprecated `FormRadioWidget` with the
`FormRadioGroup` component (which is a light wrapper around
the plain Radio.Group from Mantine).
* Remove the deprecated `FormRadio` component
* Revise alert creation instructions in documentation
Updated instructions for creating alerts, including clearer steps and options for different alert types. Adjusted formatting for consistency and added details on editing and managing alerts.
* Enhance clarity in delete and restore documentation
Updated the documentation for deleting and restoring items to clarify the impact on related content and improve phrasing.
* Apply suggestions from code review
Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>
---------
Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>
We call native-query-deps on query save on mongo queries. This ensure that doesn't error, while still flagging if we don't implement native-query-deps for a driver that is supposed to support it.
* use disk at query time of notification payload creation
We have some nice work to dump query results to disk if they are larger
than a row-count threshold. But this presupposes that results can fit
into memory. It's completely possible, and relatively easy, to do some
kind of "select *" operation in a dashboard where the only limit that
prevents billions of rows coming back is the excel limit we add in.
This tries to optimistically use the disk. If creates a new rff that has
two modes: in-memory and a gzipped dataoutput stream. The first is in
operation for the first 2,000 rows. If we never exceed this limit it
remains a regular reducing function. Once this threshold is exceeded, we
open a new file and stuff the rows in there and don't keep them in
memory any longer. This prevents us from having to hold the entire
result set in memory at once.
On the consumption side, we have a threshold such that if a file is
_larger_ than this, we won't attempt to load it and render it for static
viz. This is what lets us defensively decline to do an operation that
would reasonably lead to an OOM.
Optimizations for the future:
- give up on file creation once it exceeds the threshold size. We can
easily monitor the size of the file as we go and stop the transduction
or results.
- clear out files more frequently. This will probably litter lots of
temp files, some possibly quite large, on teh disk. We need to more
frequently clear them up. The directory is marked as deleteOnExit, but
we need to ensure we don't put lots of files in there during process
lifetime.
- better error rendering of "too large" error. Right now it hits the
default error rendering (previously it would oom). We can make this
better.
- explore how parallel we can get now. Presumably we are now quite
memory effecient. Can we go back to 3, or 10, worker threads?
* fix typo and reintroduce constant for row threshold
* more references to typo constant
* missed one somehow?
* include pulses (alerts) for autocommit off for downloads
we want to stream results. and now we do into a file so this can respect
the page size which only happens in pg when using autocommit false
* special error msg for too large results/dont attach large results
feedback that query results were too large
alerts also by default include results. let's check for the
render/too-large and not include them in that case. Was actually getting
a seq error on the deftype that holds the file. maybe want to give that
a seqable method?
* fix tests
we had an unrealized temp storage file going through
* unification all to streaming before moving to temp-storage
* move streaming to the temp-storage namespace
* don't deref results if no one wants attachments
* don't log the error here. it's pretty normal
* doc strings
* better logging and truncate results above threshold
once we are above the size that we won't read the file, we don't have to
keep piping results to it.
Concretely, what does this get us? If someone absentmindedly did
`select * from transactions` it could be 100s of GBs or more. Without
this, we would fill up the disk for the instance even though we don't
try to render it in the email. But once we know that there's no way we
will read the contents of the file, we really don't have to write to it
any longer.
* Make settings for size limit and whether to enforce
maybe someone has a beefy machine and they don't mind chugging through
843mb of query results to get a graph. they can easily set
`MB_ENFORCE_NOTIFICATION_TEMP_FILE_SIZE_LIMIT` to false and the old
behavior will be restored. If someone doesn't like the 10mb limit and
prefers a 50mb limit they can do
`MB_NOTIFICATION_TEMP_FILE_SIZE_MAX_BYTES=52428800` and then they will
have the higher limit
* Huge performance improvements
The real gist here is removing the gzip on the output. This was crushing
our performance. 18 seconds instead of ~4 seconds. Real bad. And it's
better to not gzip anyways. We really want to be tied pretty closely to
the size of the results. GZiping uses a ton of cpu and gets it very
compact. But we want a number that is pretty representative of the
results. Repeated values get gzipped down to almost nothing and then
they could still OOM the instance. But 50mb of results might get
squashed down to ~8mb of text. We want to be tethered close the size in
a csv or in memory.
Also changed the serialization to write chunks of rows and not a single
row. Its totally possible that the gzip fix did all of the work and
chunks or individual rows all get put together into the
bufferedoutputstream so it doesn't matter anyways. But the good thing is
that the format of this file is entirely the work of this namespace and
no one else gets to see it.
* Check for fidelity of regular results and disk based
timing wise: running the same card 356 (a join of people, products,
orders) and verify timing stays consistent (we are only 100ms longer
with disk)
```clojure
temp-storage=> (time
(-> (toucan2.core/select-one-fn :dataset_query :model/Card :id 356)
metabase.query-processor/process-query
:data :rows count))
"Elapsed time: 740.709583 msecs"
18760
temp-storage=>
(time (->
(metabase.notification.payload.execute/execute-card 1 356)
:result
(select-keys [:notification/truncated?
:status :running_time
:data.rows-file-size
:data])
(update :data (fn [d] (select-keys d [:rows])))))
"Elapsed time: 840.608 msecs"
{:status :completed,
:running_time 724,
:data.rows-file-size 8056845,
:data {:rows #object[metabase.notification.payload.temp_storage.StreamingTempFileStorage
"0x3336077d"
{:status :pending, :val nil}]}}
```
Compare in memory results:
```clojure
temp-storage=> (let [card-id 356
regular-rows (-> (toucan2.core/select-one-fn :dataset_query :model/Card :id card-id)
(metabase.query-processor/process-query)
:data :rows)
disk-rows (-> (metabase.notification.payload.execute/execute-card 1 356)
:result :data :rows)
dereffed @disk-rows]
(doseq [[f results] [["/tmp/results.from-disk" dereffed]
["/tmp/results.from-qp" regular-rows]]]
(with-open [w (clojure.java.io/writer f)]
(doseq [row results]
(io/copy (str (pr-str row) \newline) w))))
[(= regular-rows dereffed) (= (count regular-rows) (count dereffed))])
[true true]
```
and then compare the files written above:
```shell
/tmp
❯ diff results.from-disk results.from-qp
/tmp
❯ echo $?
0
```
* weird postgres types
* postgres 12 didn't like gen_random_uuid so just hardcode one
everyone else promised they wouldn't use
7e3cd49d-bfe1-4620-83dd-0c163719175c for anything important
* formatting, lints
* alignment
* reformat, remove unused aliases
* remove errant comment
* cleanup logic and add some comments
unconditionally add to rowcount and in-memory collection of rows. For
two of three branches, this is "done": (1) collecting in memory, and (2)
streaming to disk but our "batch" isn't full. Only when the batch is a
multiple of 5,000 rows (number completely picked at random, balancing
this) we can then write the row block to disk.
Totally possible this can switch to just writing to disk on each row and
rely on the buffered writer to handle batching. But the use of the
in-memory collection being similar across almost all of the branches
makes this solution feel a bit cleaner for the moment.
also does things like inverting `if-not` so the short branch is at the
top and not hanging at the end where it's esay to miss.
* unify two file size settings
used to have some notion of `enforce` and `file-size-limits` Rather than
juggle both, you can just set the filesize to 0 to make it unbounded
* reuse human-readable-size in more places, fix some typos
* typos, fixes
* prometheus metrics
* followup to metrics
I’ve added a new metric
```clojure
(prometheus/counter :metabase-notification/temp-storage
{:description "Number and type of temporary storage uses"
;; memory, disk, above-threshold, truncated, not-limited
:labels [:storage]})
```
to record the different scenarios of query results:
- stayed in `memory`
- wrote them to `disk` but remained under the threshold
- wrote to disk and they went `above-threshold`
- wrote to disk and they went so far above threshold that we abandoned the query so the results were `truncated`
- wrote to disk but the limit was set to 0 so the results were `not-limited`
and these look like: (replaced # with ♯ so git message will include)
```
❯ http get :9191/metrics | grep temp
♯ HELP metabase_notification_temp_storage_total Number and type of temporary storage uses
♯ TYPE metabase_notification_temp_storage_total counter
metabase_notification_temp_storage_total{storage="memory",} 2.0
metabase_notification_temp_storage_total{storage="disk",} 2.0
metabase_notification_temp_storage_total{storage="truncated",} 2.0
```
* make predicates a bit more clojurey
* missed a refactor somehow
* better styling
thread the human readable size through
* styling changes
* bit of formatting
---------
Co-authored-by: Alexander Polyankin <alexander.polyankin@metabase.com>
* Add currency label style options to formatting.md
Added options for displaying currency labels in the formatting section.
* Clarify currency label style options
Updated the currency label style section to clarify options for displaying currency labels, including examples for Canadian dollars.
* Apply suggestions from code review
Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>
---------
Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>
* subtler background lines on cartesian charts
* increase font size on cartesian charts and legends to 13px
* improve size and spacing of legends for pies and choropleths
* change legend color to text-primary
* change case and color of pie chart center subtext
* change small funnel text to 13px
* tweak the number of y-axis tick marks when set to auto for small charts
* Update Loki Snapshots
* Update Loki Snapshots
* update specs, polish
* specs
* specs
* cleanup
* Update Loki Snapshots
* subtler background lines on cartesian charts
* increase font size on cartesian charts and legends to 13px
* improve size and spacing of legends for pies and choropleths
* change legend color to text-primary
* change case and color of pie chart center subtext
* change small funnel text to 13px
* tweak the number of y-axis tick marks when set to auto for small charts
* Update Loki Snapshots
merge in updated loki tests
* update specs, polish
* specs
* specs
* cleanup
* Update Loki Snapshots
merge master's loki snapshots
* Update Loki Snapshots
* update be specs
* move border-subtle definition from colors.module to colors.ts
* table border color fixes
* fix overly thick dashboard heading border-bottom
* Cursor's attempt at fixing related tests
* Cursor again attempts to fix tests
* specs
* cleanup
* another attempt to fix specs
* Update Loki Snapshots
* fix cartesian chart and pivot table lines in dark mode
* Update Loki Snapshots
* attempt to fix chart test
* spec
* unrelated test fix but I need to merge this PR
---------
Co-authored-by: Metabase Automation <github-automation@metabase.com>
Co-authored-by: Aleksandr Lesnenko <alxnddr@gmail.com>
Co-authored-by: Aleksandr Lesnenko <alxnddr@users.noreply.github.com>
* command palette update
Updated keyboard shortcuts for opening the chat sidebar in Metabot documentation.
* Simplify custom action creation instructions
Removed instructions for creating a custom action from the command palette.
* Update appearance settings reference to 'Branding'
* Remove Metabot reference from exploration.md
Removed mention of asking Metabot in the exploration section.
* Add 'boolean' to filters page
Added boolean expressions support to custom filter documentation.
* Add boolean keywords to expressions doc
Updated section on conditional operators to include comparison and boolean logic operators.
* Fix anchor so old links work
* Fix missing period
Co-authored-by: Alex Yarosh <alexandra@metabase.com>
---------
Co-authored-by: Alex Yarosh <alexandra@metabase.com>
When using the `underlying-records` drill ("See these Orders") on a bucketed temporal value, such as a data point in
a time series, use a `:between` filter with the endpoints of the bucket, rather than `:=`.
The queries work with `:=` (because the QP expands them to `:between`) but some parts of the filter UI
render it badly. With `:between`, the filter is rendered properly as a date range.
* Make Metabot logo gray when brand color customization is defined
* Add MetabotLogo stories and test them with Loki
Signed-off-by: Jakub Chodorowicz <jakub@chodorowicz.com>
---------
Signed-off-by: Jakub Chodorowicz <jakub@chodorowicz.com>
Fixes QUE-2561
For example, in our Sample Database, `Orders.PRODUCT_ID` is remapped to
`Products.TITLE` by default. If you go to the **Table Metadata** in admin
and set `Products.TITLE` to "Do not include"
(which is `:visibility-type :sensitive` internally) it should not be
used for remapping.
* Add useEffectOnceIf hook
* Fix entity picker list not scrolling to chosen element
* Remove explicit dependencies
* Move useEffectOnceIf to VirtualizedList folder
We had tried this previously, but the athena driver has these libraries
uberjar'd into it. So we need this to be in lockstep. See
https://github.com/metabase/metabase/pull/63894 for more info.
but i'll include some info here:
there are two copies on the classpath:
```clojure
user=> (enumeration-seq (.. (Thread/currentThread) getContextClassLoader (getResources "io/netty/buffer/ByteBufAllocator.class")))
(#object[java.net.URL 0x3bb34fe "jar:file:/private/tmp/fixed-athena/plugins/bigquery-cloud-sdk.metabase-driver.jar!/io/netty/buffer/ByteBufAllocator.class"]
#object[java.net.URL 0x4ac1867d "jar:file:/private/tmp/fixed-athena/plugins/athena.metabase-driver.jar!/io/netty/buffer/ByteBufAllocator.class"])
```
one in the bigquery driver jar and one in the athena driver jar. The
bigquery driver jar has an explicit dep and we can control it
independently. The athena jar has it uberjar'd in and we can't control
it independently. So we want to make sure that we are relying on the
version that is (unfortunately) baked into athena.
For this we can see the athena 3.6.0 jar has the following
pom.properties file:
`META-INF/maven/io.netty/netty-buffer/pom.properties`
```properties
artifactId=netty-buffer
groupId=io.netty
version=4.1.124.Final
```
So we will pin that version in bigquery as well so we only have a
single version (but still two copies) on the classpath.
This way whichever classfile is resolved (and this can vary depending on
whether an athena db or bigquery db is synced first, or health checked)
it should behave the same
* fix long snippet names overflow
* add percent width so it uses most space both in SQL models and SQL editor
* fix sidebar width in transforms page
* for SnippetRow, use flex instead of width percentages, fix collection row
* fix snippetrow background so it expands snippet instead of inserting it into the editor
When investigating a separate issue I've noticed that we're logging "Loading ..." even for entities that are not actually being loaded. That's because we were printing this line when something is a dependency - even if it exists in the database and is not provided in the export.
This changes the approach somewhat, we will still have some indication we need some entity, but only at DEBUG level, and instead the "Loading ..." will appear only if we're genuinely about to read something from an export and put it in the database.
Not only it's easier to understand what's going on, it's going to be way less noisy.
* WRK-903 Update command palette modal (#64039)
* Remove metabot from command palette
* Remove footer
* Style search input
* Icon color
* Remove actions section when there is no search query
* Rename 'recent items' to 'recents'
* Show 10 recents by default
* Style item
* Update copy (Search results -> Results)
* Use project react import convention
* Move link to search results to group header
* Close palette on results link click
* Fix types
* Use disabled prop in useCommandPalette
* Update item bg
* Align search icon
* Update color whem is active
* Update table icon
* Fix wrong item being selected
* Use stack
* Fix icons shrinking
* Hide subtext on overflow
* Update unit tests
* Fix types
* Clean up
* Hide actions in a different way
* Update helper
* Remove redundant changes
* Update unit tests
* Update unit tests
* Update helper
* Update test
* Update test
* Update tests
* Update test
* Update test
* Update tests
* Add a border to search input
* Update test
* Simplify text overflow handling
* Avoid parent().parent()
* ESLint
* Improve assertion
* Improve assertion
* Remove borders between result groups in command palette
* update styles of the app bar (#64105)
* update styles of the app bar
* rework profilelink to ts
* fix some tests
* fix another e2e test
* update search style
* fix shortcuts conflict
* unify search buttons between oss and enterprise
* don't translate metakey
* fix colors
* let's have two keydown handlers
* fix test
* fix typecheck
* address review comments
* fix showplow event in test
* show metabot icon on small screens
* fix test
* do not show metabot button within iframe
* attempt to fix flaky test
* lint
* address review comments
* address review comments
* add a unit test
* update colors
* address comments
* update skeleton
---------
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
* support returning one action with "onClick" to call it immediately
* feat(sdk): add question to the clicked object, part of EMB-796
* feat(sdk): clean up the data we pass as `clicked`, part of EMB-796
makes export types checker happy
remove source
* add stories
* only require `onClick` for the "immediate" actions
* address feedback: fix type of onClick and add warning in case of wrong
shape
* allow cc review to be skipped
* Update .github/workflows/claude-code-review.yaml
Co-authored-by: metabase-bot[bot] <109303359+metabase-bot[bot]@users.noreply.github.com>
* trying to retrigger when labels change
---------
Co-authored-by: metabase-bot[bot] <109303359+metabase-bot[bot]@users.noreply.github.com>
* Claude code review (experimental)
* Run my job PLEASE!
* Fix syntax
* More guidelines
* Minor change to see if we can get some useful feedback
* Seriously run my damn workflow
* Add GH token
* Remove paths
* Appease Claude
* Tell Claude how to get the Clojure Style Guide
* Attempt Clojure style guide again
* Give Claude ReadFile perms for the style guide
* Move info about community style guide to CLAUDE.md
* Try again
* Another pass
* Seriously please follow Community style guidelines
* More advice.
* LEARN ABOUT TEST NAMING CLAUDE!!!!
* More advice
* Appease Kondo
* I have a lot to say I guess
* Iterate on CLAUDE.md
* Clarify module linter cheating rule
* CLAUDE u iz dumb
* Chaos
* Dumb as a rocc
* EVEN. MORE. GUIDELINES.
* Clarify semicolon rules more
* Threaten our AI overlords
* Are you dense?
* Annoyed
* BE SUPER DUPER EXPLICIT
* Fun time
* fun
* More dumb comment examples
* Clarify defmethod rules
* SERIOUSLY!!!
* I don't think Claude can count
* More examples of what to avoid
* U not listening
* Clarify
* Use emojys
* Parens
* Be. Even. More. Explicit. About. Semicolons.
* Be SUPER EXPLICIT
* Clarify
* Discourage `qp.store/with-metadata-provider`
* Move SmartScalar subcomponents to separate files
* Update getValueHeight to support multiple comparisons
* Fix dashcard header height
* Fix overflow with cheap tactics
* Hide all but 1 comparisons if they don't fit
* Fix line height
* Move PreviousValueComparison to its own directory
* Extract DetailCandidate
* Extract VariationPercent
* Extract DetailCandidate
* Remove inTooltip prop in VariationPercent
* Add hidden comparisons to the tooltip
* Update constant
* Add ellipsis badge
* Do not use " • " as text separator
* Make tooltip text white
* Wrap text in the tooltip
* Fix text alignment
* Revert static viz change
* Add e2e test
* Update tests
* Update Loki Snapshots
* Remove separator in static viz
* Update Loki Snapshots
---------
Co-authored-by: Metabase Automation <github-automation@metabase.com>
* Allow passing error/friendly in malli schema's to declare them to be be safe for users to see
* Add a user-friendly error message when aggregations don't contain aggregations
* Allow :error/message to be a thunk
* Add repro for metabase#61893
* Use deferred-tru for error message
* Remove deferred-tru
* Fix regression with :async? requests
- callbacks were not passed and clj-http would throw
* Fix data correctness issue if cancelled during table read
Fixes an issue where cancellation during the read stage
could cause the python run to receive (and output) truncated/empty incorrect data and write back / succeed anyway.
* Add cancellation test
* Give script a bit more time for CI
* Disable cancellation test for problematic drivers
* Also disable redshift
* Also disable mongo
* Fix regression due to merge conflict
* disable comments in an iframe
* Remove comments from UI for embedded scenario
* Make isWithinIframe check external for CommentsMenu
* Add e2e test to check that comments are not available in embedded app
* Update Loki Snapshots
---------
Co-authored-by: stasgavrylov <stasgavrylov@gmail.com>
Co-authored-by: Metabase Automation <github-automation@metabase.com>
* Add missing class to ClickActionIcon
* Remove superflous transition
* Move ClickActionsTextIcon to css
* Move ClickActionControl to css modules and metabase/ui
* Tweak css to match prevouis button height
* Split fields returned for cards/models into core fields and related_tables
* Update tests, and fix implementation to always include :related_tables
* typo
* fix kondo
* add separate with_related_tables flag to get-table-details API
* add test for new flag
* fix malli recursive schema
* add back query-needed?
* support related-tables when fetching table details as well, not just models
* add get-table-details-related-tables-test
* don't overwrite tyler's 404 fix
* Extract lazy-evaluation of currencyMapCache to an iife
* Add getCurrencySymbolNative
* Add narrow symbol support
* Update unit tests
* Add test
* Add e2e test
* Fix issue with current value not being shown
* Fix#54317: Object detail navigation state doesn't reset
Added a verification that caps the selected object index.
* Fix#54317: Changed useState and reseted index.
Now using useEffect and reseting to the first index.
* Fix#54317: Added an e2e test
* Adjust test to new filters
---------
Co-authored-by: Sébastien <sebastien@metabase.com>
Co-authored-by: Jakub Chodorowicz <jakub.chodorowicz@metabase.com>
Co-authored-by: Jakub Chodorowicz <jakub@chodorowicz.com>
* js source of truth for colors
* use more betterer colors
* keyboard shortcut for toggling dark mode
* Adds colorConfig for defining a light and dark color for each color variable
* Copies over dark colors from `dark-mode` branch (excluding variables that don't exist in colorConfig`
* Ports other style changes from `dark-mode` branch
* Revert 3301ed93693
* Ports mantine theme logic from `dark-mode` but uses SSOT colorConfig instead of SEMANTIC_COLOR_SCHEMES
* Fix TS errors
* Fix text color and random white bg on permissions table
* Removing a bunch of usages of JS color... probably breaking things
* Updates mantine color scheme when resolvedColorScheme changes
* a bunch more
* fix progress bars. shame on Nick
* Stubs out forcing light mantine theme when isEmbedding (since it has its own dark mode)
* Fix hard-to-read dark mode straggler caused by one-off bg color
* add orionAlphaInverse colors
* make lighthouse illustration work with CSS vars
* Fixes hover states of query builder header buttons in dark mode
* Updates PivotTables in dark mode to behave similarly to night mode
* Fix upsell CTA hover color in dark mode
* Tweaks admin nav dark mode color
* some small fixes
* Fixes some incorrect brand color values when white-labeling
* Tweaks admin nav bar colors and uses same colors on dark mode as light mode (if it ain't broke...)
* Fix gauge visualization text/input colors in dark mode
* Makes things legible and more consistent in query builder in dark mode
* Fixes a few filter chip stragglers in dark mode
* metabot visits the dark side
* Use ColorSchemeToggle for dashboard night mode button
* Prevents dashboard from using night mode when theme=night is set in the hash
* Update color.ts
* Revert "Prevents dashboard from using night mode when theme=night is set in the hash"
This reverts commit 06a504c088e31c60aca7c7550aa24a0f32edba6c.
* set color scheme for scrollbars
* hard to see toggle in database info
* Embedding and other fixed
* fix blue brand
* Uses canonical dark mode for dashboard night mode and static embedding's dark mode
* Fixes issue where Static Embedding preview pane doesn't show checkerboard background when "Dashboard background" is turned off
* Updates find-never-defined-css-variables.ts so it looks for defined variables in colors.ts. Removes references to now-removed base colors.
* Fixes some TS errors
* happy-ish static viz
* only resolve to color var if name is known
* type corrections, removing auto color scheme
* Passes storybook theme arg to ThemeProvider
* add konami code shortcut for Vamsi
* Fix dashboard context TS errors
* Fix runtime warning
* Prevents whitelabel colors from being used in colors.ts when token lacks whitelabel feature. Updates `filter` and `summarize` colors to use whitelabel value if present.
* re-enable updateColors
* happy types
* Updates storybook renderer to work with stories that set the theme via theme: "dark"
* Fix issue where DatesRangeSdk story doesn't pick up brand color
* green unit tests?
* Reverts unnecessary change carried over from original `dark-mode` POC which changed some loki/storybook backgrounds
* Fix issue where theme switching themes in Question embed previews doesn't appear to do anything
* handfull of e2e suites
* Revives some `theme` infrastructure needed to support `theme=transparent`
* Restores SdkError colors on dark mode (#64306)
* happy unit and e2e tests?
* Fix lint errors
* feat(sdk-bundle): use existing tooltip configuration for dark mode (#64325)
* fix card shadow crash
* backdrop color fix
* Refines ColorSchemeToggle. Color scheme is "auto" by default. When "auto", color scheme is automatically updated when browser/device settings is changed. Switching to the theme that's defined as the browser/device preference will set the color scheme to "auto".
* Adds clarifying "(system preference)" to color scheme toggle tooltip
* embedding e2e, unit tests
* Fix TS error
* update colorConfig for sdkThemeProvider
* force e2e to run in light mode
* Fix embedding-sdk:docs:generate error
* Updates ThemeProvider to ignore from hash if we're outside of static embeds. This prevents old night mode dashboard urls from messing with the new dark mode.
* Fix issue where dark mode shortcut isn't pulling in the latest values (not toggling right)
* Replace hardcoded value with already-defined constant
* fix quotes in css
Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>
* use firefox compatible scale
Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>
* fix css quotes
* fix color scheme toggle
* fix test
* fix conflict
* update test
* PR Feedback
* fix embedding homepage flicker
* e2e adjustment
* adding text-hover, fixing checkbox border
* setup.cy green plz
* e2e test for toggling color scheme
* e2e plz
* type check
* e2e adjustment
* Update Loki Snapshots
* test colors in RGB
* OS check for shortcut
---------
Co-authored-by: Ryan Laurie <iethree@gmail.com>
Co-authored-by: Nick Fitzpatrick <nickfitz.582@gmail.com>
Co-authored-by: Brad Anderson <brad@metabase.com>
Co-authored-by: AlessioLaiso <7041012+AlessioLaiso@users.noreply.github.com>
Co-authored-by: Phoomparin Mano <poom@metabase.com>
Co-authored-by: Ryan Laurie <30528226+iethree@users.noreply.github.com>
Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>
Co-authored-by: Metabase Automation <github-automation@metabase.com>
Instead of hard-coding a "metabase-ai" as the only add-on,
we're now parametrizing this endpoint to accept different
product types, which translates into buying different add-ons.
* Alternative fix for internal queries
reenable the admin screen for http://localhost:3000/admin/tools/errors
I know we aren't supposed to look at any piece of a query, including
type here, but there are a few reasons i'm doing so.
1. i don't know of any predicates in mbql lib for this
metabase.lib.core doesn't have a `native?` predicate, nor an `mbql?`
predicate. It does seem have predicates about stages but that doesn't
seem applicable here
2. even if it _did_, the internal query is not recognized by lib, so we
then have to teach it internal queries which means the other PR (#64375)
has to be implemented either way, defeating the purpose of this very
simple PR.
```clojure
user=> (metabase.lib.core/any-native-stage? {:limit 1,
:type :internal,
:fn "metabase-enterprise.audit-app.pages.queries/bad-table",
:args [nil nil nil "last_run_at" "desc"],
:offset 0,
:parameters []})
Execution error (ExceptionInfo) at metabase.lib.walk.util/fn$fn (util.cljc:72).
Invalid input: {:lib/type ["missing required key, got: nil"], :stages ["missing required key, got: nil"]}
```
Why aren't there tests?
🤠
Want to contrast the two approaches first. Happy to backfill tests once
the approach is agreeable. This approach here is _very_ simple and lets
the working qp approach continue working without disturbing
metabase.lib. This seems pretty nice because ultimately these internal
queries should/will be replaced. But in the meantime, do we want to
teach lib about them, or just skirt it altogether?
* test that api works for these queries
* enable feature in test
The python code is executed by the runner but its output is not saved back. The endpoint blocks
until the output is received.
Any objects in S3 should be deleted before the request returns (or will otherwise be collected via retention)
Limits:
- Run timeout is set to 60 seconds
- Samples up to 100 rows from each input table.
- Outputs up to 100 rows from the python output.
Fixes#63245.
Add `:reversed? true` option to `lib.walk/walk-stages` that reverses the order.
Then use that option for `qp.middleware.add-implicit-joins/second-pass`, since that `second-pass`
is about making sure if stage N needs an implicitly joined column from the previous stage, that the
previous stage actually returns it. #63245 happens when the join clause and usage are 2+ stages apart.
Flipping the iteration order will push the need for the implicitly joined column through as many stages
as necessary.
* use local datetime instead of zoned datetime when creating exclusion filters from parameters
* use local datetime instead of zoned datetime when creating exclusion filters from parameters
* Configure ESLint to prevent test files going into SDK build
* Fix SDK build to not include test files
* Fix the config to allow __support__/metadata
* Revert a file I thought was a test file
* Make rules consistent
* Clean up tsconfig pattern
* Make the error message less repeatable
* Fix incorrect separator on FE and add guard to setting
* Reproduction
* when-some
* Add FE test
* naming
* Update frontend/src/metabase/admin/settings/components/widgets/FormattingWidget.unit.spec.tsx
Co-authored-by: Ryan Laurie <30528226+iethree@users.noreply.github.com>
---------
Co-authored-by: Ryan Laurie <30528226+iethree@users.noreply.github.com>
* WRK-905: Allow commenting on an unsaved document
* Fix e2e tests
* Remove 'disabled' prop from CommentsMenu. Remove force click in comments test
* Allow editing document when comments sidebar is opened
* Fix linting
* Fix e2e tests
* Address PR comments. Allow charts editing when comments sidebar is open
* store transform revisions
* Add tests
* remove unneeded arity
* fix whitespace
* include impl.transform in revisions.init
* move transform test to enterprise folder
* use =? and check unneeded keys were removed
* revert =? use
* dissoc source from transform comparison
* Allow token without exp, but raise a warning
* Fix typing
* Fix wrong doc link anchor
* Fix doc link accessible label
* Add a test
* Fix sso endpoint calling repeating when jwt doesn't contain `exp`
* Fix unit tests
* Fix test for real!
* Fix test name typo
Co-authored-by: Nicolò Pretto <info@npretto.com>
* Use correct terminology "session"
Co-authored-by: Nicolò Pretto <info@npretto.com>
* Fix showing no EXP claim warning when the token is being fetched
Co-authored-by: Nicolò Pretto <info@npretto.com>
---------
Co-authored-by: Nicolò Pretto <info@npretto.com>
Fixes https://github.com/metabase/metabase/issues/62179
Before (and bad) (wordwrap introduced for clarity):
```
2025-08-14 16:17:36,662 WARN app-db.liquibase :: The following
changesets were not rolled back. Likely because they are not in the
changelog file: v56.2025-06-05T16:48:48, v56.2025-06-06T20:11:53,
v56.2025-06-06T20:11:54, v56.2025-06-06T20:11:55,
v56.2025-06-06T20:11:56, v56.2025-06-06T20:11:58,
v56.2025-06-13T15:00:00, v56.2025-07-07T08:02:43,
v56.2025-07-17T20:11:55, v56.2025-07-17T20:11:56 2025-08-14 16:17:36,662
INFO liquibase.logging :: Successfully released change log lock ```
```
And now:
```
2025-10-01 16:00:19,326 INFO app-db.liquibase :: Rolling back app database schema to version 55
2025-10-01 16:00:19,326 INFO app-db.liquibase :: Going to roll back changeset v56.2025-07-17T20:11:56
Rolling Back Changeset: migrations/056_update_migrations.yaml::v56.2025-07-17T20:11:56::edpaget
2025-10-01 16:00:19,328 ERROR app-db.liquibase :: Error rolling back migration v56.2025-07-17T20:11:56
liquibase.exception.RollbackFailedException: liquibase.exception.DatabaseException: ERROR: column "jwt_attributes" does not exist
Position: 41 [Failed SQL: (0) UPDATE core_user SET login_attributes = jwt_attributes WHERE sso_source = 'jwt']
at liquibase.changelog.ChangeSet.rollback(ChangeSet.java:1041)
...
Caused by: liquibase.exception.DatabaseException: ERROR: column "jwt_attributes" does not exist
Position: 41 [Failed SQL: (0) UPDATE core_user SET login_attributes = jwt_attributes WHERE sso_source = 'jwt']
at liquibase.executor.jvm.JdbcExecutor$ExecuteStatementCallback.doInStatement(JdbcExecutor.java:520)
... omitted
... 65 more
Caused by: org.postgresql.util.PSQLException: ERROR: column "jwt_attributes" does not exist
Position: 41
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2734)
... 70 more
2025-10-01 16:00:19,330 WARN app-db.liquibase :: The following
changesets were not rolled back. Likely because there were errors in
rollback (v56.2025-07-17T20:11:56): v56.2025-06-06T20:11:53,
v56.2025-06-06T20:11:54, v56.2025-06-06T20:11:55,
v56.2025-06-06T20:11:56, v56.2025-06-06T20:11:58,
v56.2025-06-13T15:00:00, v56.2025-07-07T08:02:43,
v56.2025-07-17T20:11:55, v56.2025-07-17T20:11:56 2025-10-01 16:00:19,330
INFO liquibase.logging :: Successfully released change log lock
```
How i recreated:
connected to a jar:
```
MB_DB_CONNECTION_URI="postgres://dan:password@localhost:5432/downgrade-checks" MB_JETTY_PORT=2999 java "$(socket-repl 6000)" -jar $JARS/1.56.6.2.jar
```
```sql
downgrade-checks=# alter table core_user drop column jwt_attributes;
ALTER TABLE
```
this will make a downgrade fail:
```yaml
- changeSet:
id: v56.2025-07-17T20:11:56
author: edpaget
comment: |
Add sync login_attributes with jwt_attributes, we need to replace jwt_attributes
with login attributes for users with sso_source jwt so that they will get the new
behavior where login_attributes can override jwt_attributes
changes:
- sql: >-
UPDATE core_user
SET jwt_attributes = login_attributes, login_attributes = '{}'
WHERE sso_source = 'jwt';
rollback:
- sql: >-
UPDATE core_user
SET login_attributes = jwt_attributes
WHERE sso_source = 'jwt';
```
This will fail as the `jwt_attributes` column has been removed.
```
Error rolling back migration v56.2025-07-17T20:11:56
liquibase.exception.RollbackFailedException: liquibase.exception.DatabaseException: ERROR: column "jwt_attributes" does not exist
Position: 41 [Failed SQL: (0) UPDATE core_user SET login_attributes = jwt_attributes WHERE sso_source = 'jwt']
```
And then connect to the jar again, redefine the changes here, and run
```clojure
(let [data-source (metabase.app-db.core/data-source)]
(with-open [conn (.getConnection ^javax.sql.DataSource data-source)]
(with-liquibase [liquibase conn]
(rollback-major-version! conn liquibase false))))
```
* Fix `u/slugify` function's max-length option
`u/slugify` url-encoded non-ASCII characters, then took `max-length`
chars off the result.
Both make sense in isolation - but if we're slugifying, e.g., the string
勇士, with max-length 10, this results in:
- first, url encode non-ASCII characters => "%E5%8B%87%E5%A3%AB"
- then, take 10 characters => "%E5%8B%87%"
But a bare `%` character, as in the trailing character above, is not
valid URI syntax.
Discovered in [UXW-666: Users can't download results for questions with
long cyrillic names, error "URL
malformed"](https://linear.app/metabase/issue/UXW-666/users-cant-download-results-for-questions-with-long-cyrillic-names)
but there are probably more cases where this has caused mischief.
* fix test
These log lines will now be debug level warnings rather warning level warnings.
```
<lib.normalize> Error normalizing MBQL 5
<lib.field.resolution> We tried every trick we could think
<lib.join> Failed to find matching column in join
```
Intrepid readers will also notice that moved "Resolved field %s to an
expression. ..." to debug as well.
* Fix downloading tables grouped by datetime->int
The Linear task has a slightly more involved setup. I found that I could
replicate the issue with a slightly simpler setup:
- set up a simple query:
- select COUNT from ORDERS
- group by the day of the month
- log in as the sandboxed user
- try to download the card results (as CSV, XLSX, etc)
- see the error
Why does this happen? I found that, for a sandboxed user, the query
results show the "day of the month" column as having
```
:base_type :type/DateTime,
:database_type "TIMESTAMP",
:effective_type :type/Integer,
```
while an admin user sees the same column as having
```
:base_type :type/Integer,
:database_type "INTEGER",
:effective_type :type/Integer,
```
This seems bizarre and worth looking more into, but honestly shouldn't
really matter - if the effective type is an integer, we certainly
shouldn't be formatting it as a datetime.
`field-is-type?` is only used for checking whether a field is temporal
for formatting purposes, so I think this is a relatively low risk
change, and note that `metabase.lib.types.isa/temporal?` (the
replacement suggested by the deprecation notice for
`metabase.types.core/temporal-field?`) returns `false` for both
cases.
[UXW-660: Pivot tables with integer breakout columns fail to download as formatted XLSX for sandboxed tables](https://linear.app/metabase/issue/UXW-660/pivot-tables-with-integer-breakout-columns-fail-to-download-as)
* Write a test
* add python runner timeout as setting
* set timeout status
* add timeout status test
* appease kondo
* proper script
* Fix if: expr in upload runner logs
* Fix python runner log name
* Remove setting qualifier
* Add python-runner-timeout-seconds to FE settings type
* Ensure premium feature enabled so setting can be changed
---------
Co-authored-by: Dan Stone <wotbrew@gmail.com>
* Add context to Save verb
* Add key
* Extract button from nested t
* Find other instances of nested c(...).jt`...${<x>...</x>}...` calls and unnest them
* Remove extra interval tick when there is only one interval
* Add repro for metabase#63671
* Update Loki Snapshots
---------
Co-authored-by: Metabase Automation <github-automation@metabase.com>
* Fix can't drill SQL questions in embedding SDK
* Add tests
* Fix the audit DB should not be able to see SQL
* Fix every CI check failing from updating from master
* Address code feedback, make it more clean
* Fix transducer order
* pin shape
* Update docs/questions/visualizations/map.md
Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>
* wording
Clarified limitations on data points for maps and added details about pin types.
* image
---------
Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>
Very silly bug, we were just populating the ID with the wrong
thing (`(:database-id table)` instead of `(:db_id table)`), so the FE
couldn't build links correctly.
There was a mismatch between `recent_views` and `popular_items`
endpoints. They have different queries, but are using the same
postprocessing function. So I needed to unify them: both now use
`:db_id`.
* Fix ssh tunneling for clickhouse
* Remove questionmark
* Use kebab case keys
* Fix kondo;
This is also fixed in https://github.com/metabase/metabase/pull/63645,
so there may be a conflict.
* Don't squash exceptions.
* remove some tests, fix some tests
I removed some tests. Let's be very clear why this is happening. Push
back if you disagree:
```clojure
(deftest clickhouse-connection-fails-test
(mt/test-driver :clickhouse
(mt/with-temp [:model/Database db {:details (assoc (:details (mt/db)) :password "wrongpassword") :engine :clickhouse}]
(testing "`driver/database-supports? :uploads` does not throw even if the connection fails."
(is (false? (driver/database-supports? :clickhouse :uploads db)))
(is (false? (driver/database-supports? :clickhouse :uploads nil))))
(testing "`driver/database-supports? :connection-impersonation` does not throw even if the connection fails."
(is (false? (driver/database-supports? :clickhouse :connection-impersonation db)))
(is (false? (driver/database-supports? :clickhouse :connection-impersonation nil))))
(testing (str "`sql-jdbc.conn/connection-details->spec` does not throw even if the connection fails, "
"and doesn't include the `select_sequential_consistency` parameter.")
(is (nil? (:select_sequential_consistency (sql-jdbc.conn/connection-details->spec :clickhouse db))))
(is (nil? (:select_sequential_consistency (sql-jdbc.conn/connection-details->spec :clickhouse nil))))))))
```
This test is _GONE_ because it's insane that we are testing that
`conneciton-details->spec` and `database-supports?` is resilient to
opening a connection to a server. That's nuts. We don't do that any
more. Should be the same as if checking `(+ 1 1)` doesn't open a network
connection. We get version information in
```clojure
(def ^:private ^{:arglists '([database])} get-clickhouse-version
(memoize/ttl
^{:clojure.core.memoize/args-fn (fn [[database]] (:details database))}
(fn [database]
(sql-jdbc.execute/do-with-connection-with-options
:clickhouse
(sql-jdbc.conn/db->pooled-connection-spec database)
nil
(fn [^java.sql.Connection conn]
(with-open [ver-stmt (.createStatement conn)
ver-rset (.executeQuery ver-stmt clickhouse-version-query)
cloud-stmt (.createStatement conn)
cloud-rset (.executeQuery cloud-stmt "SELECT value='1' FROM system.settings WHERE name='cloud_mode'")]
(cond-> nil
(.next ver-rset)
(assoc :version (.getString ver-rset 1)
:semantic-version {:major (.getInt ver-rset 2)
:minor (.getInt ver-rset 3)})
(.next cloud-rset)
(assoc :cloud (.getBoolean cloud-rset 1)))))))
:ttl/threshold default-cache-ttl))
(defmethod driver/dbms-version :clickhouse
[_driver db]
(get-clickhouse-version db))
```
We don't need to run this query every time you get a connection spec or
ask if the database supports something. We have the version and if it's
cloud and that's sufficient. The only reason it was opening a connection
was to run the cloud check (`select value=1 from system.settings where
name=cloud_mode`) above.
Also removed:
```clojure
(deftest impersonation-support-check-over-ssh-tunnel-test
(testing "driver-supports? :connection-impersonation should work through an SSH tunnel (#62377)"
;; Use an in-memory SSH server that forwards to the correct port through localhost.
;; Since the real database is still reachable locally, it's possible for the driver to ignore the SSH tunnel and
;; still connect to the DB. So this test runs twice, once with all correct and once with an incorrect SSH
;; password. The latter check should fail (returning false for `driver-supports?`) if the SSH tunnel is being
;; respected.
(mt/test-driver :clickhouse
(let [username "username", password "password"]
(with-open [ssh-server (driver.tu/basic-auth-ssh-server username password)]
(doseq [[correct-password? ssh-password] [[true password] [false "wrong-password"]]]
(memoize/memo-clear! @#'clickhouse.version/get-clickhouse-version)
(let [ssh-port (.getPort ssh-server)]
(with-ssh-tunnel! {:tunnel-enabled true
:tunnel-host "localhost"
:tunnel-auth-option "password"
:tunnel-port ssh-port
:tunnel-user username
:tunnel-pass ssh-password}
(testing (str "using " ssh-password)
(is (= correct-password?
(driver/database-supports? :clickhouse :connection-impersonation (mt/id)))))))))))))
```
Again, this is verifying connection resiliency in a call
to `(driver/database-supports? :clickhouse
:connection-impersonation (mt/id))` and ensuring that it uses the ssh
tunnel. This entire concern is gone, and a red flag that it ever
existed.
Changes:
a few tests used the `db` from `with-temp` to do stuff, but now that we
require them to have dbms_version i populated that
in the version check i thought i saw one in test going through with
`dbms-version` and not `dbms_version` so i'm checking both just to be
sure. it's a bit sloppy
* small changes and add back test
* handle dbms-version vs. dbms_version
* fix kondo
---------
Co-authored-by: Riley Thompson <riley@metabase.com>
Co-authored-by: dan sutton <dan@dpsutton.com>
Co-authored-by: Riley Thompson <rileythompson99@gmail.com>
* Factor out setup/log actions for python runner, use in driver tests
* Remove redundant cd
Co-authored-by: Nicolò Maria Mezzopera <nicolo.mezzopera@metabase.com>
---------
Co-authored-by: Nicolò Maria Mezzopera <nicolo.mezzopera@metabase.com>
* Pass className to native editor
* Add missing border on top of native editor
* Set minimal editor height so the results panel is always visible
* Move initial height calculation to hook
* Add missing import
* Use late-binding for handlers so they can change without having to recreate markers
* Allow isEditing props to re-render Map component
* Allow disabling zoom control on maps
* Allow clicking the Map action buttons
* Disable zoom and hover when in editing mode
* Add repro for metabase#64138
* Fix e2e tests
Eric and I duplicated some work. This merges our efforts. This also builds in ref patching on import and export and provides an `import-dir!` method that #worksonmymachine.
Co-authored-by: Eric Normand <ericwnormand@gmail.com>
* UXW-1766 Fixes issue where getDashboardHeaderParameters looks at all cached dashcards rather than just the ones from the currently viewed dashboard
* UXW-1766 Adds/updates unit tests
We're introducing a new endpoint in metabase/metabase. It sends the feedback to the backend which then (1) fetches the premium token and (2) constructs the correct store api URL with the same feedback as the payload. Our backend acts as a proxy because the api url contains the token and we can't send it from the FE.
* Extract native query content for search indexing rather than the entire dataset_query structure
* remove inline def
* Fix doc string
* Update search spec field-fields logic to support the :fn syntax to fix test
* Remvoe query_type from expected card spec fields as we're no longer using it to detect native query changes
* Address feedback on :fn semantics, and add a more detailed docstring to define-spec macro
* feat: use new detail view in metadata preview
* drop relationships
* fix e2e helper
* use correct '
* make table responsive
* update helper
* adjust values in tests
* use useSampleData
* rework approach
* update helper
* oops, lost changes
* restore test
* let's add table just in case
* another attempt
* responsive
* restore indexes
* rename index -> rowNumber
* show details group in more cases
* remove force: true
* fix index
* When deleting DB, hide cards from search
We have a current issue with search: deleting items does not actually
remove them from search.
This is a particularly nasty version of this, because while usually a
delete follows an update (that marks the card as archived,
which *does* remove it from search), in this case we're just deleting
the items directly as part of an `ON DELETE CASCADE`.
This is a bit hacky, but for now we can just mark them as `archived`
before we delete them to hide them from search.
[UXW-602: Search API returns inexisting cards](https://linear.app/metabase/issue/UXW-602/search-api-returns-inexisting-cards)
* Add i18n / formatting to existing log calls
* Cleanup log conveyance to failure :message
* Improve error feedback capability
- Ensure exception messages do not leak to user logs by default 'Something went wrong'
- Exceptions can carry a `transform-message` key to provide a custom failure message (with i18n)
- Defined messaging for:
- Error during table copy
- Error during execution
- Error during writeback
I would like to follow up with additional context in the form of message log entries for further progress / context of any failure.
Logs can be correlated to metabase logs for stack traces or further debugging. Python output remains presented as-is (traceback etc).
* [match] Prevent match-lite from extracting common result that depends on uncommon bindings
* [match] Fail if match-lite's :guard is provided with incorrect predicate syntax
Also fix its treatment of named functions passed as predicates.
* Inline AdminPaneLayout
* Do not show invite button on deactivated tab
* Simplify
* Use modern button
* Add tabs in People page
* Move mb
* Fix typo
* Make people search input full width
* Move tabs to the top
* UXW-333 Prevents `updateDashboardAndCards` from being called when discarding changes (still called when clicking "Save changes")
* UXW-333 Adds e2e test
* for fetching tiles encode field refs as safe base64
* clojure implementation
* update public api
* pass lat and lon fields in query params
* specs
* fix specs
* update preview endpoints and tests
* clean up previous url safe base64 implementation
* fix specs
* UXW-709 When drilling into a dashboard card that had a filter applied, the resulting query can now be saved outside of the original dashboard. "New Question"/"New SQL query" via dashboard edit screen behavior is unaffected (still forces you to save to that dashboard).
* UXW-709 Updates unit tests
* Allow redirect url from password reset page
* Add await when inviting user so that errors will be displayed
* Allow for different email content when inviting user from setup / Set returnUrl param
* Code refinements
* Add backend tests
* Add frontend tests
* Run formatter
* handle email content when invitor's first name is empty/unset
* Allow running dev on custom MB_FRONTEND_DEV_PORT
Signed-off-by: Jakub Chodorowicz <jakub@chodorowicz.com>
* Unify port env var naming between JS and Clojure
* Add dev server port info
* Update devenv docs
Co-authored-by: Sébastien <sebastien@metabase.com>
* update docs
---------
Signed-off-by: Jakub Chodorowicz <jakub@chodorowicz.com>
Co-authored-by: Sébastien <sebastien@metabase.com>
* Add working table with 2 columns
* Table layout and dnd polishing
* ResizeNode
* Refactor to use grid container
* Add Card node title ellipsification
* cleanup, add min height
* Fix FlexContainer to work with ResizeNode
* Fix empty ResizeNode after card dnd
* Document layout no dnd kit (#63363)
* using tiptap drag and drop
* Fix some issues with dnd
* Add support for resize node, fix styles
* Remove comments
---------
Co-authored-by: Nick Fitzpatrick <nickfitz.582@gmail.com>
* Fixes after merge comments
* Fix issues with dnd logic
* Fix issue where rearranging columns within a flexContainer didn't respect which side of a cardEmbed we dropped; it always inserted dragged column to the left.
* Fix issue where moving a cardEmbed out of a flexContainer and placing it right after doesn't wrap the card in a resizer
* Fix issues with dropzones and card id
* Fix issue with column resize handlers
* Fix issues with cards previews on dnd
* Feat: UXW-717 Reconcile comments with document layout (#63593)
* UXW-717 Add comment option to cardEmbed menu
* UXW-717 Wires up comment option in cardEmbed menu
* UXW-717 Stubs out CommentsMenu reorganization so logic can be reused inside cardEmbeds (button and menu item)
* UXW-717 Design tweaks per design feedback (and fixes ts error)
* UXW-717 Updates hover state of cardEmbed ellipsis menu trigger per design feedback
* Fix issues with funnel chart and resizer
* Fix issues with column rearrange
* Fix adding third column position
* UXW-739 Adds ResizeNode wrapper to charts added by metabot
* UXW-749 [1/2] Fix: Cutting/copying a chart from a group then pasting it (cmd+v) results in a chart without a resizeNode
* UXW-749 [2/2] Fix: Copying and pasting from one chart to another doesn't preserve the structure correctly
* Refactor dnd logic - consolidate all use cases in HandleEditorDrop
* remove redundant message and add_message icons
* actually remove redundant message icons
* replace references to old icons with new icons
* Fix issue with dropping a card into existing flexContainer
* Fix issue with moving a card from one flexContainer into another
* UXW-748 - Fix broken document when typing with a card selected
* Handle dropping cards onto paragraphs (#63754)
* Handle dropping cards onto paragraphs
* PR Feedback
* cleanup flex containers
---------
Co-authored-by: Denis Berezin <denis.berezin@metabase.com>
* small feedbacks
* breaking up document drop handler
* UXW-689 - add basic e2e tests
* type fix
* loki diff
* Add more e2e tests
* Fix typecheck
* happy types
* passing document.cy.spec
* Fix e2e tests
* e2e test for card resizing
* flex container column resize tests
* Add node deletion e2e tests
* Fix review issues
* Try to fix typecheck
* Refactor useCommentsButtonProps to a react component
* final PR feedback
* fix removing a card via menu
---------
Co-authored-by: Nick Fitzpatrick <nickfitz.582@gmail.com>
Co-authored-by: Brad Anderson <brad@metabase.com>
Co-authored-by: Maz Ameli <maz@metabase.com>
Co-authored-by: Nick Fitzpatrick <nick@metabase.com>
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
Previously `extract-query "NativeQuerySnippet"` would use standard method for exporting entities, which checks for
collection_id in entity and prevents its export if it's outside requested collections.
While that makes sense for Cards (and we do escape analysis for those), snippets live in their own separate collection
namespace and this logic is not relevant to them. So now we export either those in requested collections _or_
dependencies of cards that are being exported.
* cherry pick driver code (#63628)
* cherry pick driver code
* update changelog
* add type->database-type unit tests
* kondo
* create/insert table
* insert-from-jsonl test
* missing rename-tables for snowflake
* extend rename-tables-test to more drivers
* line breaks
* add driver python transform feature
* handle review comments
* kondo
* feat: replace string->val with a more generic insert-col->val
* remove non transactional impls
* chore: docsstrings
* TODO
* extra test
* fix extra test
* test all base-types
* fix sqls
* setup ci properly (#63693)
* S3 transport layer for python-runner (#63697)
* Add s3 dependencies (#63706)
* port prometheus/instrumentation (#63631)
* port prometheus metrics
* port instrumentation
* instrument existing transform paths
* wider buckets
* remove nil handling
* warn on nil bytes/rows
* add stage-type mapping
* typo
* split CI into separate jobs (#63746)
* Add python_runner.clj (#63747)
* Add python runner
* Kondo
* Traceback fix
* Remove dead code
* start the runner by default
* Cancellation not used yet
* Remove unused keys from fields-metadata
* cherry pick driver fixes
* start for postgres and mysql as well
* clickhouse fixes
* relax for snowflake
* typo
* exclude transform-python-test from appdb
* exclude transforms-python-test as well
* try exclude-tag here too
---------
Co-authored-by: Ngoc Khuat <qn.khuat@gmail.com>
Co-authored-by: Nicola Mometto <nicola.mometto@metabase.com>
* execute.clj (#63755)
* execute.clj
* with drivers test and execute test
* with wait tables
* missing utils
* /s/open/start
* fix failing tests due to getting the wrong source and target
* kondo
* move cancel-python-code-http-call to python-runner
* Dan's feedback
* we have 2 implementstation of cancel
* and 2 db-routing-enabled?
---------
Co-authored-by: Nicola Mometto <nicola.mometto@metabase.com>
* Transforms python APIs (#63762)
* transforms python api
* new tests pass locally
* clean up the db id utils
* move to util
* fix: missing refer
* chore: missing tests/kondo
* kondo
* more kondo
* fix: make function resilient
* eh, validate on old
* on new
* Add python transform support for jobs (#63764)
* Add python transforms job support
* Fix whitespace in when
* Avoid churn due to new local
* Add feature test
* Add ignore for logs (there might be away to avoid this)
* Use new target-database-id util
* Remove duplicated fn
---------
Co-authored-by: Nicola Mometto <nicola.mometto@metabase.com>
Co-authored-by: Dan Stone <wotbrew@gmail.com>
* Move transform ordering tests to the python module (#63827)
* Add transform ordering tests to the python module
- Consistent with api tests for python transforms being in the python
module
- In practice transforms module 'knows' about python, so this is
organizational only
- Copied test helpers for now, not sure if functions strong enough to be
shared
* Check :transforms/python
* Proj python/table name len (#63842)
* feat: ensure temp-table-name respects max length
* add unit test
* add limit for missing drivers
* depend on it
* fix: use db instead of db-id
* Proj ptyhon/table swap (#63878)
* add rename feature, rename-table! multimethod
* add table swap with atomic or rename or fallback
* changelog
* drop requirement
* ensure we're testing replace flow
* wait for completion, table already exists
* wip
* add rename-table-test
* fix for snowflake
* fix: support bigints (#63898)
* fix: support bigints
* merge meta
* Frontend for Python Transforms (#63699)
* cherry pick driver code (#63628)
* cherry pick driver code
* update changelog
* add type->database-type unit tests
* kondo
* create/insert table
* insert-from-jsonl test
* missing rename-tables for snowflake
* extend rename-tables-test to more drivers
* line breaks
* add driver python transform feature
* handle review comments
* kondo
* feat: replace string->val with a more generic insert-col->val
* remove non transactional impls
* chore: docsstrings
* TODO
* extra test
* fix extra test
* test all base-types
* fix sqls
* setup ci properly (#63693)
* feat: add Python transform frontend pages
* Integerate with FE
* Fix wrong source field name used for showing script in transform view
* Revert "Fix wrong source field name used for showing script in transform view"
This reverts commit c7dfb541f3.
* Fix edit button not doing anything for python scripts
Closes QUE-2300
* call api proxy instead of internal call, also have settings page for transform
* python transforms now properly being scheduled
* FE select table when creating python transform
* multiple tables
* ensure tables are unique in tables selector
* source alignments
* run test
* dead code
* Better default schema, better docstring
* fixex test run
* add cancel button
* better endpoint name
* add cancel button to test python endpoint
* working button
* Auto-update Python transform function signature when table aliases change
- Automatically updates def transform(...) signature when aliases are modified
- Updates docstring Args section with aligned descriptions and table names
- Adds reset button to restore default slugified aliases
- Validates alias input to only allow valid Python identifiers
- Handles multi-line signatures for many parameters (>80 chars)
- Shows visual distinction between custom and auto-generated aliases
- Fixes code editor layout to prevent overlap with Run button
* feat: settings page with s3 configs
* feat: split settings
* remove placeholders
* eh, rename to appease the munging
* Logs are viewable inline during execution of python transforms
* Some minor fixes around docstring rewriting
* solve more fe lints
* fix type errors
* more type check fixes - including some smelly optional override stuff
* fix delete transform + table
* fix: don't remove indentation before """
* chore: allow empty schemas
* Critical fix: docstring indentation
* Separate file / s3 lifetimes / writes / reads from http calls (#63294)
- python-runner no longer simulates a single http api that does
everything (incl s3 interaction)
- Separates file lifetimes (shared storage) from execution
- Easier to see how to avoid reading into strings (e.g. for piping into
tmp files, follow up)
Earlier on we had a service that managed much of the I/O, metabase now
coordinates it so it makes little sense pretending there is still
an overall http api.
# Conflicts:
# enterprise/backend/src/metabase_enterprise/transforms/api.clj
# enterprise/backend/src/metabase_enterprise/transforms/execute.clj
# enterprise/backend/src/metabase_enterprise/transforms/python_runner.clj
# enterprise/backend/test/metabase_enterprise/transforms/python_runner_test.clj
* Add first-class PythonTransformSource type
# Conflicts:
# enterprise/frontend/src/metabase-enterprise/transforms/components/QueryEditor/EditorBody/EditorBody.tsx
* Lay python editor out in two columns
* Make python editor resizable
* Move parseCSV to utils
* Split off results into separate component
* Add empty and loading state components
* Split up ExecutionResult component
* Move python execution to separate hook
* Remove color in editor gutter
* Increase indentUnit for python
* Allow extension to be passed to CodeEditor
* Add pandas completions
* Use DataGrid for displaying results
* Python library APIs (#63378)
Co-authored-by: Romeo Van Snick <romeo@romeovansnick.be>
# Conflicts:
# enterprise/backend/src/metabase_enterprise/api/routes.clj
# enterprise/backend/src/metabase_enterprise/transforms/python_runner.clj
# enterprise/backend/test/metabase_enterprise/transforms/python_runner_test.clj
# enterprise/frontend/src/metabase-enterprise/api/tags.ts
# resources/migrations/057_update_migrations.yaml
* transforms python module (#63503)
# Conflicts:
# enterprise/backend/src/metabase_enterprise/transforms/api.clj
# enterprise/backend/src/metabase_enterprise/transforms/execute.clj
# enterprise/backend/src/metabase_enterprise/transforms/jobs.clj
# enterprise/backend/src/metabase_enterprise/transforms/util.clj
# enterprise/backend/src/metabase_enterprise/transforms_python/python_runner.clj
# enterprise/backend/test/metabase_enterprise/transforms_python/api_test.clj
# enterprise/backend/test/metabase_enterprise/transforms_python/e2e_test.clj
# enterprise/backend/test/metabase_enterprise/transforms_python/python_runner_test.clj
* Use normal title case
* Handle json-line output
* Remove unused props
* Rename PythonEditor to PythonQueryEditor
* Move script rewriter to utils
* Flatten Python transform page
* Rename shared to common
* Remove dead code
* Reduce specialization in Transform editor
* Remove unused props
* Move PythonQueryEditor to top-level components
* Nest Python editor components in PythonTransformEditor
* Restructure Python transform editor
* Split up component more
* Add helpers for python editor
* Disable default schema
* Add test ids
* Fix import path
* Reuse transform save modal
* Fix parameter
* Add e2e tests for python common lib
* Move python transform api to general transforms folder
* Move type to metabase-api/types
* Disable debugger for python transforms for now, allow to open it with query param
* Disable debugger hotkey for python transforms
* Add syntax highlighting to Python script on Transform page
* Remove cancelation code for now
# Conflicts:
# enterprise/frontend/src/metabase-enterprise/transforms/components/RunButton/RunButton.tsx
# enterprise/frontend/src/metabase-enterprise/transforms/pages/TransformPage/RunSection/RunSection.tsx
* Add more generic RunInfo
* Remove unused LogOutput
* Set up python runner in CI (#63566)
* Check out with submodules in CI
* Include python-runner in docker compose for tests
* Set up python runner in CI
* Fix import
* Update python-runner
# Conflicts:
# python-runner
* Fix font sizes
* Fix table layout
* Revert "Remove unused LogOutput"
This reverts commit 90c0704344.
* Add LogOutput back to runsection
* Remove duplicate schema field
* Use css instead of styled components
* Fix LogOutput styling
* Add transforms/python feature
* Do not allow picking unsupported databases
* Use slugify from metabase/lib/formatting
* Add type for transform alias map
* Fix some type imports
* Refactor PythonDataPicker
# Conflicts:
# enterprise/frontend/src/metabase-enterprise/transforms/components/PythonTransformEditor/PythonDataPicker/PythonDataPicker.tsx
* Further refactors
* Use data selector for database picker
* Use data picker to pick tables
* Remove unused var
* Improve tests for python transform
* Fix duplicate imports
* Use DatabaseId in python transform
* Remove source/draft ambiguity in types
* Move library types to metabase-types/api
* Remove api transforms
* Remove vibe-comment
* Remove unused component
* Validate source more
* Move TableInput to separate file
* Remove TableOption type
* Simplify props
* Move AliasInput to separate file
* Remove unused props
* Add border to empty results
* Fix import
* Remove extension
* Move draft back to new python transform page
* Fix new python transform page nesting
* Simpify new transform page structure
* Add analytics to history revision and data reference (#63587)
* Transform run cancellation (#62387)
* Add cancellation to transform api
* Use effect to set polling state
* Add cancellation button to run button
* Set up cancellation on transform page
* Use confirmation modal to cancel transform
* Add run cancel button to run list
* Make api endpoint take transform id instead of run id.
* Fix cancelation api
* Add canceled state to TransformRunStatus
* Add canceled state to transform page
* Add polling to RunListPage
* Remove comment in code
* Set status to canceling when you click cancel
* Add canceling status
* Poll when the transform is canceling
* Add copy for runs that are being canceled
* Manually track fetching status for running and canceling
* Handle canceling status in utils
* Avoid wrapping cancel button in run list
* Add test for canceling run on transform page
* Add e2e for canceling test from runs page
* Fix prop type
* Change copy to Canceling
* Add custom color to canceled state
* Remove mock code
* Increase cancelation tiemout
* Fix copy in text
* Fix snowplow event
* Remove void specifier and close modal in handler directyle
* Add optimistic update for run cancelation
* Avoid manually tracking cancelation status
* Avoid useEffect when changing isPolling
* Add in missing helper
* Remove another useEffect
* Add placeholder run when starting
---------
Co-authored-by: Eric Normand <ericwnormand@gmail.com>
Co-authored-by: Alexander Polyankin <alexander.polyankin@metabase.com>
* Fix command z not working in connection string input (#63689)
- Stop preventing default event handling onPaste.
- Improve Cypress paste emulation to be compatible with React way of handling paste events.
* Transforms prepare for python (#63698)
* pull out transform run utils to prepare for python transforms
* fix: check running-run on transform-id, not run-id
* chore: put on a new line
* Fix#63629: Filter on cards causes an unnecessary horizontal scroll bar for public/static embedding (#63650)
* Closes EMB-50 tabs dimmed out on iOS (#63690)
* S3 transport layer for python-runner (#63697)
* Add s3 dependencies (#63706)
* port prometheus/instrumentation (#63631)
* port prometheus metrics
* port instrumentation
* instrument existing transform paths
* wider buckets
* remove nil handling
* warn on nil bytes/rows
* add stage-type mapping
* typo
* Implement data picker according to designs
* Further adjust to design file
* Fix some e2e tests
* Clone python runner when running CI
* Avoid starting the container in e2e tests twice
* Hide python runner admin route when user does not have the feature for it
* Remove unused variables
* Move Python runner amdin page to transforms plugin
* Do not show Python library link when the user does not have python transforms support
* Hide Python transforms features when the user does not have transforms-python token
* Use useHasTokenFeature hook where possible
* Add transforms-python feature
* Fix e2e tests
* Do not include python-runner in test docker-compose yet
* Tag tests for python transforms with @transforms-python
* Add helper to update Python library
* Add check for python output
* Fix Revert on python library when it does not exist
* Fix e2e test helper
* Add python runner settings in CI
* WIP
* Translate pandas definitions
* Add AnsiLogs component
* Use AnsiLogs in Logs components
* Use AnsiLogs in LogOutput component
* Remove token feature gate in enterprise folder
* Return alias instead of throwing
* Check type in isPythonTransform assertion
* Use isResourceNotFoundError
* Make transforms-python it's own plugin
* /s/localhost/localstack
* Add test for editing Python transform
* Verify that running the transform with the library works
* Add missing transforms-python feature to mocks
* Simplify source layout
* Change setting type of python runner token
* Fix tests
* Use ConcreteTableIds in python transforms only
* Deduplicate codemirror helpers
* Ignore errors in non-parseable JSON lines
* Use useLocation hook to enable debugger
* Use useErrorMessage helper
* Move SourceSection to transforms-python plugin
* Fix codemirror helpers
* Remove undefined color
* Fix types
* Fix codemirror helpers more
* Move transform ordering tests to the python module (#63827)
* Add transform ordering tests to the python module
- Consistent with api tests for python transforms being in the python
module
- In practice transforms module 'knows' about python, so this is
organizational only
- Copied test helpers for now, not sure if functions strong enough to be
shared
* Check :transforms/python
* Proj python/table name len (#63842)
* feat: ensure temp-table-name respects max length
* add unit test
* add limit for missing drivers
* depend on it
* fix: use db instead of db-id
* Proj ptyhon/table swap (#63878)
* add rename feature, rename-table! multimethod
* add table swap with atomic or rename or fallback
* changelog
* drop requirement
* ensure we're testing replace flow
* wait for completion, table already exists
* wip
* add rename-table-test
* fix for snowflake
* Extract sourceDatabaseId helper
* Fix e2e helpers
* Fix error in table picker
* Add missing test-id
* Fix transform tests
* Validate python source
* Fix issues after merge
* Fix tag test
* Fix other tag test
---------
Co-authored-by: Nicola Mometto <nicola.mometto@metabase.com>
Co-authored-by: Ngoc Khuat <qn.khuat@gmail.com>
Co-authored-by: Nicolo-Maria <nicolo.mezzopera@metabase.com>
Co-authored-by: Dan Stone <wotbrew@gmail.com>
Co-authored-by: Chris Truter <chris@metabase.com>
Co-authored-by: Nemanja Glumac <31325167+nemanjaglumac@users.noreply.github.com>
Co-authored-by: Eric Normand <ericwnormand@gmail.com>
Co-authored-by: Alexander Polyankin <alexander.polyankin@metabase.com>
Co-authored-by: Jakub Chodorowicz <jakub.chodorowicz@metabase.com>
Co-authored-by: Brad Anderson <brad@metabase.com>
Co-authored-by: Sébastien <sebastien@metabase.com>
Co-authored-by: Chris Truter <crisptrutski@users.noreply.github.com>
* Proj python fix tests (#63970)
* add premium feature
* fix tests that didn't run previously
* exclude from vertica
* remove internal id from mongo table-rows
* fix: require transforms/python db feature for python transforms
* missing in stats
* fix for mongo
* whitespaces to appease the linter
* more more whitespace for the linter gods
* disambiguate for the e2e tests
* Some small type fixes for Python transforms (#64021)
* Remove duplicate helper
* Add updateEnterpriseSetting helper for type-correctness
* Add missing python runner settings
* Use correct setting name
---------
Co-authored-by: Ngoc Khuat <qn.khuat@gmail.com>
Co-authored-by: Chris Truter <crisptrutski@users.noreply.github.com>
Co-authored-by: Dan Stone <wotbrew@gmail.com>
Co-authored-by: Romeo Van Snick <romeo@romeovansnick.be>
Co-authored-by: Nicolo-Maria <nicolo.mezzopera@metabase.com>
Co-authored-by: Chris Truter <chris@metabase.com>
Co-authored-by: Nemanja Glumac <31325167+nemanjaglumac@users.noreply.github.com>
Co-authored-by: Eric Normand <ericwnormand@gmail.com>
Co-authored-by: Alexander Polyankin <alexander.polyankin@metabase.com>
Co-authored-by: Jakub Chodorowicz <jakub.chodorowicz@metabase.com>
Co-authored-by: Brad Anderson <brad@metabase.com>
Co-authored-by: Sébastien <sebastien@metabase.com>
* Allow passing language as a plugin to CodeEditor
* Pull out NativeQueryPreview
* Render NativeQueryPreview in the native transform editor
* Relax prop types
* Add e2e test for query preview in transforms
* Fix e2e tests
* Pass query instead of question
* Use CopyButton instead of bespoke component
* Fix unit test
Attempting to use a snippet template-tag with a trailing space in the snippet name (e.g., `{{snippet: foobar }}`) caused an exception to be thrown when running the native query (missing required parameter).
The regex used for identifying snippet tags will now strip trailing spaces appropriately.
* Misc Lib improvements
* Restore deleted function... for now
* Test fix 🔧
* Mark more stuff in Legacy MBQL utils as deprecated
* Use Lib for query->source-card
* Update modules config
* Update tests 🔧
* Fix the X-Rays
* Druid test fix 🔧
* Appease Kondo
* docs: db routing with static embedding
Clarified limitations of database routing in Metabase.
* Update docs/permissions/database-routing.md
Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>
---------
Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>
* [perf] Suggest drop-in replacements for clojure.core functions from metabase.util.performance
* Fix walk for nil
* [perf] Make metabase.util.performance/reduce compatible with non-iterables (like strings)
Changes to any user attribute for sso users were failing preventing them
from being to set their user locale. This was due to us not checking if
`email` is present in the body of the update request before failing a
`check-403` call if the user has an sso source.
rather than 4.2.4.Final, they will now be "io.netty/netty-buffer
4.1.118.Final" which puts it the same as that in 3.5.0.
The issue is that both the bigquery-cloud-sdk driver and athena driver
get put onto the classpath and it depends on which order the drivers are
added for resolving which version of this netty buffer dep you get.
```clojure
user=> (metabase.classloader.core/the-classloader)
user=> (enumeration-seq (.. (Thread/currentThread) getContextClassLoader (getResources "io/netty/buffer/ByteBufAllocator.class")))
(#object[java.net.URL 0x3bb34fe "jar:file:/private/tmp/fixed-athena/plugins/bigquery-cloud-sdk.metabase-driver.jar!/io/netty/buffer/ByteBufAllocator.class"]
#object[java.net.URL 0x4ac1867d "jar:file:/private/tmp/fixed-athena/plugins/athena.metabase-driver.jar!/io/netty/buffer/ByteBufAllocator.class"])
```
One annoying bit to keep track of this is that bigquery uses a regular
jar that specifies its dependencies, so you can you regular tooling:
```
clj -X:deps tree :aliases '[:drivers]' > /tmp/deps.with.netty.downgrade
```
And then see those deps:
```
. metabase/bigquery-cloud-sdk /Users/dan/projects/work/metabase/modules/drivers/bigquery-cloud-sdk
. com.google.cloud/google-cloud-bigquery 2.54.2
. com.google.cloud/google-cloud-core 2.60.0
. ... [lots omitted]
. org.apache.arrow/arrow-memory-netty 17.0.0
X org.apache.arrow/arrow-memory-core 17.0.0 :older-version
. org.apache.arrow/arrow-memory-netty-buffer-patch 17.0.0
X org.apache.arrow/arrow-memory-core 17.0.0 :older-version
. io.netty/netty-buffer 4.1.110.Final
```
This version was created without any pinning of netty. Checking the
athena version is a bit more tricky. We use a fat jar. But you can
download the source from amazon as a jar with deps:
```
athena-jdbc-3.5.0-lean-jar-and-separate-dependencies-jars
```
https://docs.aws.amazon.com/athena/latest/ug/jdbc-v3-driver-previous-versions.html
This zip includes `runtime-dependencies` which includes
`netty-buffer-4.1.jar`
The pom inside of this jar specifies
```xml
<parent>
<groupId>io.netty</groupId>
<artifactId>netty-parent</artifactId>
<version>4.1.118.Final</version>
</parent>
```
And `META-INF/maven/io.netty/netty-buffer/pom.properties` specifies
```
artifactId=netty-buffer
groupId=io.netty
version=4.1.118.Final
```
So i am confident now that with the pinning in bigquery and this version
here that regardless of class resolution from
"jar:file:/private/tmp/fixed-athena/plugins/bigquery-cloud-sdk.metabase-driver.jar!/io/netty/buffer/ByteBufAllocator.class"
or
"jar:file:/private/tmp/fixed-athena/plugins/athena.metabase-driver.jar!/io/netty/buffer/ByteBufAllocator.class"
you will get the same class and athena shall work again.
* ensure type/* for base-type if otherwise not known
Using the follow query that returns pg arrays:
```sql
SELECT *
FROM (
VALUES
(1, ARRAY['https://example.com', 'https://example.org']),
(2, ARRAY['https://test.com']),
(3, ARRAY['https://foo.com', 'https://bar.com', 'https://baz.com'])
) AS adhoc_table (id, urls);
```
In 55 this returned a nil base-type
```clojure
user=> (metabase.query-processor.metadata/legacy-result-metadata
{:database 2
:type :native,
:native {:template-tags {},
:query "SELECT *\nFROM (\n VALUES\n (1, ARRAY['https://example.com', 'https://example.org']),\n (2, ARRAY['https://test.com']),\n (3, ARRAY['https://foo.com', 'https://bar.com', 'https://baz.com'])\n) AS adhoc_table (id, urls);"}}
1)
[{:display_name "id",
:source :native,
:field_ref [:field "id" {:base-type :type/Integer}],
:ident "native[]__id",
:name "id",
:database_type "int4",
:base_type :type/Integer,
:semantic_type :type/PK}
{:display_name "urls",
:source :native,
:field_ref [:field "urls" {:base-type nil}],
:ident "native[]__urls",
:name "urls",
:database_type "_text",
:base_type nil}]
```
In master, this just fails. If you remove the type checking in places,
it will eventually return similar.
This base-type is computed in sql-jdbc/metadata
```clojure
(mu/defn query-result-metadata :- [:sequential driver-api/schema.metadata.column]
"Default implementation of [[metabase.driver/query-result-metadata]] for JDBC-based drivers. Gets metadata without
actually running a query."
([driver :- :keyword
query :- :map]
(driver-api/with-qp-setup [query query]
(let [database (driver-api/database (driver-api/metadata-provider))
{:keys [query params]} (driver-api/compile query)]
(query-result-metadata driver database query params))))
([driver :- :keyword
database :- driver-api/schema.metadata.database
^String sql :- :string
params :- [:maybe [:sequential :any]]]
(sql-jdbc.execute/do-with-connection-with-options
driver
database
nil
(fn [^java.sql.Connection conn]
(with-open [stmt (sql-jdbc.execute/prepared-statement driver conn sql params)]
(let [rsmeta (.getMetaData stmt)]
(mapv (fn [i]
(let [database-type (.getColumnTypeName rsmeta i)]
{:lib/type :metadata/column
:name (.getColumnLabel rsmeta i)
:database-type database-type
:base-type (or (sql-jdbc.sync.interface/database-type->base-type driver (keyword database-type))
:type/*)}))
(range 1 (inc (.getColumnCount rsmeta))))))))))
```
Before this change, there was no `or` and
`sql-jdbc.sync.interface/database-type->base-type` would just return
nil.
```clojure
user=> (metabase.driver.sql-jdbc.sync/database-type->base-type :postgres (keyword "_text"))
nil
```
Unfortunately this is a multimethod so we can't easily update it in one
spot to fall back to type/*. Each implementation must take care to do
this. A few do (clickhouse does) but a few dont (snowflake does not). So
we update this call site to ensure we have something from the
multimethod or add our own type/*
Why did this start happening now?
> I think it's because I added :metadata/model-metadata to :metabase.lib.schema.info/info at some point
Query info stashes the model metadata which is
```clojure
[:metadata/model-metadata {:optional true} [:maybe [:sequential ::lib.schema.metadata/lib-or-legacy-column]]]
```
We had been throwing errors asynchronously when trying to save metadata,
but now we have on-thread type mismatch for model metadata, which is
what is in the ticket.
* add test to ensure we can handle existing base_type nil's
this got annoying because legacy metadata from the app-db comes from a
different location. I was adding a card schema migration but this is
better here.
The previous commit makes sure we don't put out more bad data, but this
commit here ensures that we can update the card.
* Move cron info up into cron section
* Move schedule explanation logic to utils
* Allow ReactNode as explain message
* Render explain message in cron component
* Add status to transform job page
* Add e2e tests for job error message
* Add assertion to lookup
* Add extract shared logic for run status
* Add missing test-ids
* Select correct section
* Don't send new user login for SDK or Embedded Analytics JS
* Update tests
* Stop using hard-coded string to detect modular embedding headers
* Conform to spacing rule
* add isClosed impls to proxys
* use mt/id in test
* test all drivers
* check if statements are supported, refactor statement-supported? multimethod to drive feature
* add namespace to statements feature
* Only report analytics when not in the preview
This would avoid confusing preview usage with the actual usage
* Ensure analytics would be report even if iframe isn't ready
* Add E2E test
* Make the test pass
* Stop relying on redux store to fetch `site-url`
* Clarify test with comment
* Add tests
* Fix linter
I was naming a file wrong, so it doesn't match unit test spec. This rule
apparently doesn't take tests into account, which is great.
* Add search endpoint for Metabot and adjusted metadata
* More tests and better split out of logic into separate files
* dedupe search terms and description if they happen to be the same
* fix bad merge conflict
* Answer blending sql querying via search tables (#62415)
* Only put tables used in SQL into context
* Change format of injected database content by reusing existing models
* Create separate function for new schema enchancement
Co-authored-by: Sloan Sparger <sloansparger@gmail.com>
* Add table reference for implicitly joinable fields in context injection (#62753)
* Update Metabot profile IDs (#62864)
* [Answer blending] Updates to the metabot admin configs (#62551)
* remove ability to configure the selected collection for default metabot
* add verified toggle
* add migrations for metabot collection configuration changes
* add metabot PUT endpoint, rip out metabot entities endpoints and usages, convert FE over to new FE, fix tests
* remove metabot_entity table, model, usage and refactor code to use updated metabot and metabot_prompts schemas
* enforce verification and regenerate prompts on verification + collection id change
* fix bad references
* fix bad regexp for fetch-mock collections by id endpoints breaking unit tests
* fix kondo
* fix + increase tests for enterprise/backend/test/metabase_enterprise/metabot_v3/tools/api_test.clj
* fix migration boolean default value syntax
* maybe fix be tests
* fix some tests
* enforce verified filter on the search endpoint
* factor in verified check into find-metric
* hide verification setting behind token feature check, fix migration lint, do not migrate collection id for default metabot, fix kondo errors
* fix core tests
* fix kondo
* fix most of the failing serialization tests
* fix migration lints
* fix one test file
* maybe fix migration failures
* move test resource file
* prevent default metabot from getting a collection id configured
* yolo
* add cascade delete to entity drop table migration
* increase unit tests for verification config UI
* remove extra add column
* fix migration... i think for real this time
* for real for real fix migrations
* no really
* tests fixes and add preventing verification toggle if the instance does not have the content_verification feature
* fix failing feature tests in ci
* fix one more
* fix cljfmt issue
---------
Co-authored-by: Tyler Plude <tsplude@gmail.com>
* [Metabot] Quick slash command + dynamic profile ui (#62909)
* quick slash command + dyanmic profile ui
* add back code i did not mean to remove
* fix failing test
* Revert "fix failing test"
This reverts commit b1c7e9efb9.
* [BOT-284] BE for notebook query tool table support (#62710)
* init
* moved query-table endpoint to query-datasource; add support for both model and table ids
* unit test fixes
* use existing card-query and table-query utils
* adding unit tests
* [Answer blending] Respect collection id in search (#63150)
* remove find metric tool
* adjustments to search endpoint
* add tests
* fix failing test in ci
* add way to override the metabot id
* add way to override metabot request id
* fix whitespace lint
* missing paren...
* fix failing test
* New BE `/search` endpoint for ai-service `SearchTool` (#63154)
* new metabot search tool BE endpoint
* remove debug prints
* parallel searches; incomplete effort to return more fields for results
* Simplify result transforming & response schemas
* unit tests
* kondo fixes; pr feedback
* fallback search model list
* remove references to MetabotEntity model which got re-added incorrectly in rebase
* Make internal metabot use verified content by default (#63529)
Update linter for usage of task in metabot-v3
Move suggested-prompts to separate namespace
Add collection condition only when current-user-id
Add startup task
migration
Update tests
Verified cards only
Test
Docstrings
Limit number of entities
Review feedback
* [METABOT] Default use_verified_content to false; tweak prompts (#63843)
* default use_verified_content false; limit amount of prompts
* reworked unit test to not depend on ai-service or appdb contents
* PR feedback
* remove mock of get metrics and models in favor of mt/with-empty-h2-app-db
* Small refactors & simplifications to metabot PUT endpoint and suggested
prompts job
* small simplification, use m/distinct-by for deduping tables
* rm deprecated search_data_sources
* Cleanup and prioritize verified content (#63877)
* Remove now redundant content verification guard
* Prioritize verified content
* Redundant merge
* Test
* ensure field IDs in user_is_viewing context are snake_cased
* Use proper source column alias as name for the model / table fields (#63826)
* Use proper source column alias as name for the model / table fields
* Add name as fallback
* test fix
* switch to using :lib/desired-column-alias, revert test fix
---------
Co-authored-by: Tyler Plude <tsplude@gmail.com>
Co-authored-by: Noah Moss <noahbmoss@gmail.com>
* Fix some tests: consistent use of snake_case, and a :lib/source-column-alias fallback for model field :name
* fix more field_id snake_case issues
* fix another snake case issue - revert result_columns back to result-columns
---------
Co-authored-by: Thomas Schmidt <thomas.heinz.schmidt@gmail.com>
Co-authored-by: Sloan Sparger <sloansparger@gmail.com>
Co-authored-by: Thomas Schmidt <thomas@metabase.com>
Co-authored-by: Sloan Sparger <sloansparger@users.noreply.github.com>
Co-authored-by: Noah Moss <noahbmoss@gmail.com>
Co-authored-by: lbrdnk <lbrdnk@users.noreply.github.com>
* fix: jetty stats handler tests flake
These tests could hang indefinitely on the `(.join server)` statement
because the core.async/go block handling async requests was incorrectly
doing a blocking take `<!!`. This also updates our :dev alias to enable
core.async to print warnings when using blocking operations in `go` blocks.
* fixup! oncomplete dispatch in go block
* fixup! using alts!! in go block
* WRK-893: Update API to store comment html content
* WRK-891: Add HTML content sending in comments. Add proper SmartLink serialization to HTML.
* WRK-891: Update comment email template to display comment content
* WRK-891: Fixes for comment email template. Add space insertion after smartLink
* trigger comment notification by id
* put old api behavior back and cleanup comment notification context
* whitespace
* fix tests
* Fix changeset author
* WRK-891: Fix smart links serialization. Add parent comment display in email
* Add style payload for comments email template
* fix types
* 🙄
* test email changes
* test email again
* make email formatting looking perfect
* cleanup
* require html content to be provided when posting a comment
* fix e2e test
* fix another e2e test
* pass 0 as tiptap placeholder when no label available
---------
Co-authored-by: Oleksandr Solovyov <o@solovyov.net>
Co-authored-by: Uladzimir Havenchyk <uladzimir@metabase.com>
Co-authored-by: Uladzimir Havenchyk <125459446+uladzimirdev@users.noreply.github.com>
* Adjust DB form page width and form fields spacing
* Extract layout field settings to one file
* Use lg spacing between form fields
* Conditionally add labelProps property
* Fix typo
* Refactor deprecated default export
* Simplify responsive max width
* Simplify DB page responsive styles
* Add card background to add db form
having issues with classpath. I _think_ that athena is finding the
bigquery provided version of a netty class and that is failing. but it's
a bit difficult to follow.
downgrading now until we can take time to understand and sort it.
* Remove the +New option for creating new models
* Bring back the from-scratch approach
Based on Alex's requested change, but also realized that you can use the command palette
* Move note about models getting pinned back into the new model section
* Format UI terms using bold instead of italics
* docs: send only attachements
* Apply suggestions from code review
Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>
---------
Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>
* update test and implementation
* update models query to make more sense
* flake fix: try adding timeouts to Done
* remove cy.wait from tests
* wait for model creation
* modify create model tests
* Don't force reindex on serialization load to be async
* Allow reindex to be controlled when calling load
* Keep load reindex sync for now
* Make reindex async again
* Add a controlled version of NotebookNativePreview
* Use ReactNode instead of JSX.Element
* Add a button to toggle native sql
* Add sql preview to transform query editor
* Pull out resizehandle component
* Add e2e test for converting an MBQL transform to a native transform
* Move native query toggle out of header
* Move Native editor sidebar to separate component
* Fix overflow in native sidebar
* Fix copy on button
* Move connected NotebookNativePreview to query_builder
* Remove canConvertToNative prop
* Rename prop
* Move styles to css
given
```
❯ tree /tmp/swaperoo
/tmp/swaperoo
├── c-1-card196164.card.yml
├── c-2-card196324.card.yml
└── sub-collection
└── c-3-card12345.card.yml
2 directories, 3 files
```
the running
```
clj -M:dev:ee:ee-dev:drivers:drivers-dev:socket -m metabase.core.bootstrap static --directory /tmp/swaperoo
```
Will serve the assets from /tmp/swaperoo.
It does this by loading them in MEMORY and assigning them sequential
negative ids. These are not loaded into the app-db.
```
❯ tree /tmp/swaperoo
/tmp/swaperoo (collection -1: swaperoo)
├── c-1-card196164.card.yml (card -1)
├── c-2-card196324.card.yml (card -2)
└── sub-collection (collection -2: sub-collection)
└── c-3-card12345.card.yml (card -3)
```
* Closes EMB-816 misleading text for number of rows
* updated e2e tests
* marked a test as flaky until we figure out what is up with it
* addressed legit comment
* Update Loki Snapshots
* loki
---------
Co-authored-by: Metabase Automation <github-automation@metabase.com>
* Add Embedded Analytics JS usage snowplow schema
* Add analytics
* Clarify code with comments
* Fix analytics logic + fix typing
* Add unit tests
* Make E2E embedded analytics JS util supports multiple elements
* Fix code not working in Cypress for some reason
* Add E2E test suite
* Fix import order
* Remove extra line
* Fix E2E from util API change
* Update snowplow/iglu-client-embedded/schemas/com.metabase/embedded_analytics_js/jsonschema/1-0-0
Co-authored-by: Phoomparin Mano <poom@metabase.com>
* Update e2e/test/scenarios/embedding/sdk-iframe-embedding/sdk-iframe-embedding.cy.spec.ts
Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>
* Update e2e/support/helpers/e2e-embedding-iframe-sdk-helpers.ts
Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>
* Fix E2E by updating a function API
* Pass snowplow schema lint
---------
Co-authored-by: Phoomparin Mano <poom@metabase.com>
Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>
Note: I copied the export for questions, including the logic and schema. I don't know if there's something else in
models (metadata?) that I am not capturing.
* Add cancellation to transform api
* Use effect to set polling state
* Add cancellation button to run button
* Set up cancellation on transform page
* Use confirmation modal to cancel transform
* Add run cancel button to run list
* Make api endpoint take transform id instead of run id.
* Fix cancelation api
* Add canceled state to TransformRunStatus
* Add canceled state to transform page
* Add polling to RunListPage
* Remove comment in code
* Set status to canceling when you click cancel
* Add canceling status
* Poll when the transform is canceling
* Add copy for runs that are being canceled
* Manually track fetching status for running and canceling
* Handle canceling status in utils
* Avoid wrapping cancel button in run list
* Add test for canceling run on transform page
* Add e2e for canceling test from runs page
* Fix prop type
* Change copy to Canceling
* Add custom color to canceled state
* Remove mock code
* Increase cancelation tiemout
* Fix copy in text
* Fix snowplow event
* Remove void specifier and close modal in handler directyle
* Add optimistic update for run cancelation
* Avoid manually tracking cancelation status
* Avoid useEffect when changing isPolling
* Add in missing helper
* Remove another useEffect
* Add placeholder run when starting
---------
Co-authored-by: Eric Normand <ericwnormand@gmail.com>
Co-authored-by: Alexander Polyankin <alexander.polyankin@metabase.com>
* use iframe for changelog
* add query params and tabs
* always show update info
it's useful even if you're on the latest version
* remove metabase as configurable host
* always allow changelog/releases in iframes
* fix tests
* fix tests
* update tests
* use root metabase domains
* fix test
---------
Co-authored-by: John Swanson <john.d.swanson@gmail.com>
* Send parse connection string parsing events only on blur
* Use synchronous last parsing result for monitoring
* Improve test assertions for connection string events
* Make sure we don' track the same parsing event again
* Refactor ts-match to simple if conditions
* Make lastStatus have the same success value depending on non undefined values
* Improve the snowplow Cypress test setup for connection string
* [Admin settings] Show help documentation to connect DB in a side panel
* Add unit tests
* Add more tests / tweaks
* Add e2e tests
* Make sure help content is not rendered if not available
* Self-review tweaks
* Use useDocsUrl in DatabaseHelpSidePanel.tsx
* Use aside element for the side panel
* Use await findBy when possible
* Use JSDoc / Remove unnecessary comment
* Add some style to blockquote
* Add test file for useHelpLink
* Fix failing test
* Add analytics event
* Document password handling for special characters
* Add section on special character passwords in MariaDB
Added information about handling passwords with special characters in MariaDB connection strings.
* Apply suggestions from code review
Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>
* Update docs/databases/connections/mysql.md
---------
Co-authored-by: Alex Yarosh <alexandra@metabase.com>
Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>
* Fix vertical legend position for cartesian charts
* Set dashcard title font size to 14px
* Decrease legend dot size
* Decrease value labels density for cartesian charts
* Use normal font weight for axis names, labels, etc
* Make y axis split lines solid vs dashed
* Show scalar card title in dashcard header
* Always show pie chart total
* Hide axis names on small dashcards
* Update screenshots
* Fix tests
* Fix number cards font size
* Fix hiding cartesian chart axis
* Fix font styles in other legend components
* Fix e2e tests
* Hide cartesian chart axes independently
* Use different legend style only in dashboards
* Apply min spacing to labels per series
* Fix legend dot size in edit mode
* Update label sizes for funnels and sankey charts
* Use bold font for data labels
* Update screenshots
* prettier
* Update Loki Snapshots
---------
Co-authored-by: Metabase Automation <github-automation@metabase.com>
* Trivial preprocessing middleware => Lib
* Handle already-native MBQL 5 queries
* substitute-persisted-query => Lib
* A bunch of post-processing middleware is actually MBQL-5 ready
* Fixx 🔧
* Test fixes 🔧
* More fixes 🔧
* Fix more stuff
* Wow even more test fixes =(
* (Hopefully) final round of test fixes
* Kondo
* More test fixes 🔧
* Fixes 🔧
* Swap cancel and save button
* Render sidebar in native query editor for transforms
* use toggleSnippetSidebar from props
* Fix snippets sidebar
* Export SelectionRange
* Add aria label for data reference button
* Open database on mount
* Add aria-label to snippets button
* Add e2e tests for sidebar in SQL transforms
* Rename prop for clarity
* Return instead of throwing
* Use Lib.rawNativeQuery
* Pass selected text to native editor
* Fix types
* Move longer handlers to hooks in utils
* Handle empty text and reshuffle code
* add-implicit-clauses middleware => Lib
* Test fixes 🔧
* Some lib.convert fixes from #61398
* Test fixes 🔧
* More test fixes
* Test fixes 🔧
* Handful of test fixes 🔧
* Discourage use of legacy MBQL utils in QP
* Integrate the platinum metadata provider
* Only one failing test left
* Test fix 🔧
* Kondo
* Test fixes 🔧
* Test fix 🔧
* Test fixes 🔧
* Handle legacy pivot viz settings where `base-type` might not be present
* Handle legacy pivot viz settings where `base-type` might not be present
* Handle legacy pivot viz settings where `base-type` might not be present
* Handle legacy pivot viz settings where `base-type` might not be present
* Handle legacy pivot viz settings where `base-type` might not be present
* Handle legacy pivot viz settings where `base-type` might not be present
* Handle legacy pivot viz settings where `base-type` might not be present
* Handle legacy pivot viz settings where `base-type` might not be present
* Handle legacy pivot viz settings where `base-type` might not be present
* Test fix 🔧
* Test fix 🔧
* Kondo
* Test fix
* Test fix 🔧
* Skip failing transforms tests
* Mark failing tests flaky
* Setup sidebar and onboarding page for embedding hub (#62517)
* Add the add data and new dashboard modals to embedding hub (#62528)
* mark sandboxed step as done (#62758)
* Auto-mark 'secure your embeds' step as done when either JWT or SAML is configured (#62736)
Co-authored-by: Nicolò Pretto <info@npretto.com>
* Add UTM tags to documentation links in the embedding hub (#62747)
* Add a wizard to select a table to x-ray in embedding hub (#62715)
* Mark the create embed steps in embedding hub as done from embed flow (#62759)
* Add a dropdown to switch embedding solutions in embedding hub (#62870)
* fix(embed-hub): adds key prop to component passed to jt (#63060)
* feat(embed-hub): checklist endpoint (#62875)
* feat(embed-hub): EMB-770 and EMB-742, add checklist endpoint with
'create-dashboard'
* feat(embed-hub): EMB-740 auto mark "Add data" as done
* Revert "mark sandboxed step as done (#62758)"
This reverts commit be775aceed.
* feat(embed-hub): EMB-768 move sandbox check to the BE
* EMB-772: move all "done" checks on the be, move endpoints to ee
* cleanup
fmt
configure enterprise module to make linting pass
test: activate license to make e2e tests pass again
remove fe unit test now that all the logic is on the be
add doc to the endpoint to maybe the tests will pass?
whitespace?
fixes
idk anymore how to make all the linters be happy
last try
refactor has-configured-sso to a function
revert visibility to see if tests pass
sort config.edn
* tests: remove cy.wait from embed hub tests (#63066)
* Bit of cleanup
- remove warn-on-reflection; there is no interop
- clean up a bit of duplication in sql clause generation
- remove unncessary try/catch around t2 access
- remove a few boolean calls on boolean functions
---------
Co-authored-by: dan sutton <dan@dpsutton.com>
* fix(embed-js): disable the next button when embedded js is disabled (#63134)
* Replace the accordion in embedding hub with steppers (#63271)
* Move embedding settings into its own top-level menu (#63339)
* Add related setting cards to modular embedding and setup guide (#63340)
* Remove the Embed menu from the New menu (#63385)
* Completing later steps on embedding hub should not tick off the former steps (#63393)
* Modular embedding toggle label should depend on toggle states (#63391)
* Add the interactive embedding upsell to the settings page (#63399)
* Lock embed in production step in embedding hub checklist until sso configured (#63449)
* Use ee plugins to check if embedding is enabled in nav (#63446)
* Check for CSV uploads in embedding-hub checklist endpoint (#63501)
* update embedded-analytics-js.md to have consistent quickstart pages
---------
Co-authored-by: Nicolò Pretto <info@npretto.com>
Co-authored-by: dan sutton <dan@dpsutton.com>
Co-authored-by: Alexander Polyankin <alexander.polyankin@metabase.com>
Fixes issue where undo history is deleted on save. Also fixes issue where CommentsSidesheet stays open (big unclosable grey box) when navigating to a different document when it was open.
* Move sqlserver description to extra info tooltip
* Fix mongo additional-options not appearing in advanced-options section
# Conflicts:
# e2e/test/scenarios/admin/databases.cy.spec.js
* Add new DB engine icons, add icon to regular add DB flow
* Refactor old export style
* update h2 icon
* Fix layout of browse databases cards
* Adjust padding of browse databases cards
* Remove empty comment
* Render default db icon in case of empty engine key
* Fall back to DESCRIBE if Athena JDBC reports duplicate column names
* Make test fail another way
* Fix test
* Verify that the hijacked function gets called
* Metadata Provider Platinum Edition
* Revert some formatting changes
* Revert changes to QP tests
* Revert NOCOMMIT changes
* Fix missing ns alias
* Pull in missing changes from other branches
* Misc fixes and changes
* Fixes
* Fix bugs in JVM metadatas
* Fix composed provider not propagating inactive
* JS fixes 🔧
* Test fix 🔧
* Test fixes 🔧
* Test fixes 🔧
* Another JS test fix 🔧
* More FE fixes 🔧
* Don't be too strict
* Set database
* Fix merge conflict
* Fix kondo
---------
Co-authored-by: Alexander Polyankin <alexander.polyankin@metabase.com>
* Sandboxing middleware => Lib
* Test fix 🔧
* Remove some unused stuff
* rename `row-level-restricitions` middleware => `sandboxing`
* Remove TODO
* :model/GroupTableAccessPolicy => :model/Sandbox
* Test fixes 🔧
* Test fix and mark more stuff deprecated
* Test and Kondo fixes 🔧
* "Fix" sandboxes with extra columns because I don't want to update e2e tests
* cljfmt
* docs - add modular embedding
Also adjust headers a bit because ToC is difficult to scan
* Update docs/embedding/securing-embeds.md
* Update docs/embedding/securing-embeds.md
* Implement new DatabaseFormError component & break DatabaseForm component down
* Implement scrolling actions / Avoid 'div in p' console error
* Fix typing check issues / Add 'Read the docs' link url
* Define 'invite teammate' and 'contact support' urls
* add more tests / further refactoring
* css tweaks
* Simplify tests a bit
* Remove comment / Fix translation cases
* Refactor tests
* Adjust 'Check Host and Port settings' button / Fix css lint
* Fix test
* Add target=_blank to doc links / Use existing color for the error container border
* Use cloud-gateway-ips setting when showing IPs to whitelist / Move some helpers to a local utils.tsx
* Use 'be.visible' over 'exist' in cypress test
* Use useDisclosure
* Make Alert title have dark text color by default / Add storybook case to show warning color variant
Trino's Hive plugin shows the tables in question in its metadata, but if you try to get a handle on these tables, it will always throw an exception with error code `UNSUPPORTED_TABLE_TYPE`. The Iceberg plugin has a similar exception for such circumstances.
For us, there doesn't seem to be much difference between not being able to query a table because it's an unsupported type and not being able to query because we don't have permissions, so`have-select-privilege?` is modified to return false in these cases. We can examine the vendor-specific error code on the SQLException to detect them.
* Update Start new thread UI
* Refactor
* Lint migrations
* Lint migrations
* Lint
* Fix comments count
* Thread submit input improvements
* Align 'Start new thread' button
* mention metabase entities
* Stupid link copying
* Deep linking to comments
* fix: sidebar style improvements
* Fix line breaking bug on submit
* Maz Comments styling tweaks (#62964)
* add tabs hiding/showing logic
* only show persistent button for open comments
* move comment buttons to the right
* attempt to fix tests
* always display the add-a-comment input in the sidebar
* Remove unused attr
* Placeholders without flags
* Refactor comments' tabs markup
* Refactor CommentsMenu
---------
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
Co-authored-by: Stas Gavrylov <stasgavrylov@gmail.com>
* Fix incorrect counter for unresolved comments
* Fix warning
* Fix key warning in Discussions
* Make document read-only when comments sidebar is open
* Refactor URL generation
* Automatically open correct tab for the comment targeted by URL hash
* Clean up
* Use createPortal instead of Portal from metabase/ui to fix CommentsMenu appearing above CommentsSidesheet when visiting a specific comment via URL
* Remove more/less feature
* Set max-height when editing comments, fix scrolling area
* Extra padding
* Make it possible to mention yourself
* Add ability to resolve deleted threads
* Remove ability to comment on resolved threads
* Remove ability to add new comments on Resolved tab
* allow using @ character without showing mention list
* Fix type errors
* Fix types
* Resolve/unresolve deleted comments
* Remove DocumentContext
* Fix inserting mentions on Enter press
* Minor styling fixes for mentions
* Remove unused endpoint
* Clean up
* Remove unused component
* Remove stories
* Fix storybook
* Remove TODO
* Remove unused file
* Remove outdated comment
* Avoid cast
* Remove redundant code
* Deal with any
* Don't trim empty nodes
* Simplify Discussions component
* Align avatar, submit button and code editor
* Do not show last divider if not necessary
* Get rid of currentIndex
* Fix typo
* Remove hack
* Check only unresolved comments when deciding whether to autofocus comment editor when sidebar opens
* Enable inline_code styling in comments
* Rename css file
* Fix cast
* Use top instead of transform
* Use a flag
* Remove commented-out code
* Move getCommentsUrl to utils
* Use Menu
* Add Avatar to metabase/ui
* Bring back accidentally removed code
* Resolve CSS duplication
* Use void
* Add comment
* Add all comments sidebar
* Use createSelector
* Fix no results styling
* Autofocus editor when editing comments
* Disable menu when editing is disabled
* Add a toast when copying the link
* Prevent deleting and editing someone else's comments
* Fix types
* Update copy
* Cleanup all comments sidebar
* Extract isMetabotBlock
* Use CustomStarterKit
* Copy
* feat: add Emoji picker (#62991)
* feat: add Emoji picker
* fix: styles
* fix: class for loading label
* Add ability to inline emojis in CommentEditor
* fix: keyboard events for emoji picker
* fix: focus visible for emoji picker viewport
* Implement proper keyboard support for inline emojis
* Temporarily hide emoji reaction button
* Improve UX around selecting first emoji
---------
Co-authored-by: Stas Gavrylov <stasgavrylov@gmail.com>
* Pass editable
* Use mentions plugin
* Fix z-index
* Add reactions BE
* Improve list paddings
* Add users support to SmartLink and MentionExtension
* Remove old mentions
* use semantic colors for Avatar
similar logic to what do for users in admin
* show add comment input when we switch from resolved tab
* Limit number of user results
* feat: add comment reactions (#63072)
* feat: add comment reactions
* fix: typescript
* fix: click to react
* fix: styling
* tweak spacing between replies (#63100)
* fix: reply styles
* Update submit shortcut to Cmd-Enter. Fix emoji picking behavior
* fix: do not close discussion sidesheet on some escape events
* fix: cancel editing on escape press
* Make DocumentPage 400px narrower when comments modal is opened
* Extract shared tiptap-related code
* Move EmojiPicker to common/components
* Fix DiscussionComment error when creator is null
* Fix missing formatting menu in CommentEditor
* Add tests boilerplate
* Allow many reactions on a comment
* Hydrate creator info on deleted comments
* Add first test and helpers
* Add tests for more types of nodes
* Add comment
* Improve test name
* Test highlighting
* Add test for splitting paragraphs
* fix: blur comment editor on submit
* Add schema upgrade test
* Add test for creating comments and threads
* Add tests for deleting comments
* Editing a comment
* Test for Esc
* fix: limit editable functionality and add comment styles (#63161)
* Fix test
* Update test
* Fix flaky test
* Fix tooltip
* fix: do not react on deleted comments
* fix: show resolved threads
* fix duplicate replies display
* Add mentions tests
* Make suggestions taller to accommodate users search results without a vertical scrollbar
* don't count deleted comments for the comment badge
* don't show threads if all comments deleted
* Add emoji tests
* fix: can react true by default
* Fix floating menu positioning in CommentEditor
* fix document / menu selected item styling
* Sort comment reactions by created_at ASC
* Make Esc close emoji popup
* first pass at comments api
* Fix reactions ascending order
* WIP
* fix linting
* model resolution
* update test setup to include _id, so equality check works correctly
* replace is_deleted with deleted_at to detect deleted comments
* fix types
* remove ?new=true from url after adding a new comment
* Fix tests
* Improve arrow navigation when emoji popup is opened. Allow selecting with Tab
* Add test
* Refactor
* Update test
* Fix emoji popup Esc handler removal
* Add formatting tests
* Rename test
* Add e2e API helpers to create and delete comments
* Update test name
* Refactor
* Fix types
* update is not deleted eh
* Add test for the comment button
* Add test
* Add a test
* Add test
* test: basic resolve/unresolve tests (#63243)
* test: add resolve other person thread test
* Add test for disabling document editing when comments are open
* Add test for unsaved document changes
* Add test for show all comments button and fix the issue
* Add tests for deleting/editing other people's comments
* Add e2e tests for comment links
* add e2e tests for all comments
* way better/less verbose permission checks
* test: cannot reply to resolved threads
* ESLint
* Update assertion
* Fix unresolved comments count
* e2e tests for comments reactions
* first iteration of notifications
* fix migration lint errors
* fix module tests
* Add label attribute for SmartLinks to simplify server-side notifications rendering
* add e2e test for removing ?new=true after creating a comment
* Fix
* fix clj syntax
* Remove commented-out code
* Use H.readClipboard()
* Refactor
* Update reaction icon
* Move util
* Lint
* Fix
* Lift conditional rendering, move Reaction component to a separate file
* Refactor
* Do not render invalid links
* Refactor
* Update new comment notification template
* add e2e test for top blocks
* fix: comment date to respect browser locale
* Update reactions icon
* Add 1st test for email subscriptions
* Update email template to match figma, remove the comment content
* Add document icon to email template
* Update test
* Remove protocol from the link
* Extract verifyEmail helper
* fix migrations because mysql is majorly unhappy about those rollbacks
* correct urls for comments, notifications on mentions
* Update assertion
* Insert both comment URL and document URL into the email template
* Fix url
* fix copy tests
* Add a setup for next test
* Implement assertions
* 2nd email variant
* Add another test
* Add an assertion
* add more tests for email template
* Ignore whitespace
* Error handling for mutations - toasts
* add more tests
* feedback: drop blue border from paragraph
* feedback: placeholder
* fix e2e test to pass
* Update setup
* Comments design polish (#63352)
* use new Send icon instead of up_arrow
* remove extra space below a Discussion
* restyle Discussions
* fix CommentEditor spacing
* restyle current user reaction
* only show No Comments message on the All Comments sheet
* add on-hover highlighting for All Comments section
* fix for fe-type-check
* fix test
* Cleanup
---------
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
* put user permissions check back and new api for mentions
* pagination
* Use new GET /api/ee/comment/mentions in frontend
* Use new GET /api/ee/comment/mentions in smart link node
* paid feature in tests
* seed notifications
* oh
* clj-kondo 🤦
* port numbers
* Try to fix flaky test
* attempt to fix flaky e2e
* disable returning users for mentions
* disable mention related tests
---------
Co-authored-by: Stas Gavrylov <stasgavrylov@gmail.com>
Co-authored-by: Anton Kulyk <kuliks.anton@gmail.com>
Co-authored-by: Konstantin Darutkin <spalt08@mail.ru>
Co-authored-by: Maz Ameli <maz@metabase.com>
Co-authored-by: Uladzimir Havenchyk <uladzimir@metabase.com>
Co-authored-by: Oleksandr Solovyov <o@solovyov.net>
Fixes#62492.
A literal UUID was reaching the driver as
`[:value "uuid-string" {:base_type :type/UUID}]`
but this was not recognized as a UUID-valued field because `:value`
clauses have `:base_type` and not `:base-type` like `:field` clauses.
Therefore the `:sql` core driver was rewriting this clause to cast the
UUID field to a string and compare that against the literal string.
With that logic now correctly recognizing such a `:value` clause, the
UUID fields are compared directly. This is much better for indexing,
but also works with *nullable* UUID columns, where the
`CAST(my_uuid AS text)` would cause SQL compilation errors.
* Make with-index! use new, clean, separate database per test
* Add with-test-db! setup modes
* Comment
* Adjust with-test-db! and related
* Drop with-index! in favor of modal with-test-db! in first test
* semantic.tu/db removal
* Drop use of semantic.tu/db in favor of get-pgvector-datasource!
* Drop semantic.tu/db from tu
* Remove once-fixture
* Remove ensure-no-migration-table-fixture
* Comment
* Comment
* Cleanup
* Make the rest of the query-test ns use with-test-db!
* Remove redundant test
* Remove remaining usage of with-index!
* Drop remnants of with-index! and make linter happy
* Cleanup
* Comment
* Remove with-mock-index-metadata! in favor of with-temp-db!
* Adjust with-index-contents!
* Fixture to avoid running app-db tests
* Fixture
* Require
* whitespace
Problem: Perf tests were failing because they couldn't find the uberjar artifact.
The wait_for_uberjar mechanism was trying to wait for a job named 'build (ee)'
but the actual job name is 'Build MB ee'. Additionally, this wait mechanism was
unnecessary complexity.
Solution: Align with pr-env-labeled.yml pattern:
1. Build uberjar first when 'Run-perf' label is added
2. Run perf test after uberjar is built
3. Remove unnecessary wait_for_uberjar mechanism
Changes:
- perf-test-labeled.yml: Add uberjar build job before running perf test
- perf-test.yml: Remove wait_for_uberjar parameter and 56-line wait mechanism
This simplifies the workflow and ensures the uberjar artifact exists before
the perf test tries to fetch it, following the same proven pattern used by
pr-env-labeled.yml.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-authored-by: Claude <noreply@anthropic.com>
* Make dataset-query->dimensions more robust
* *disable-updates*
* Add macro impls for easier debugging
* Make dataset-query->dimensions more resilient
* Whitespace
* fix: don't ignore db sync/rescan promise
When the call returns non-200, the button still says it's fine instead of showing a failure.
You can test this by trying to sync "Metabase Cloud Storage" DWH.
See https://metaboat.slack.com/archives/CKZEMT1MJ/p1749830993738189
* add unit tests
* fix mocks
---------
Co-authored-by: Ryan Laurie <iethree@gmail.com>
* Ensure results are realized when falling back to in-place results in semantic search
Unlike the semantic and appdb backends, the in-place backend returns a reducible result set from
search.engine/results. Make sure we realize the results before attempting to pass them to `m/indexed-by` to dedup
them. This avoid a "Don't know how to convert reducible to ISeq" exception when falling back to in-place results.
* Add comment
want to check that the new table exists before adding it into the
app-db. The old way worked for all databases but is pretty dumb: it
enumerates all tables in the db (as it would for regular sync) and then
checks if the new table is in there.
We can be better with the `driver/table-exists?` function.
The shape of data coming back from describe-data is the following:
```clojure
notify=> (let [database (toucan2.core/select-one :model/Database :id 1)]
(driver/describe-database :h2 database))
{:tables #{{:name "ORDERS",
:schema "PUBLIC",
:description nil,
:is_writable true}
{:name "PEOPLE",
:schema "PUBLIC",
:description nil,
:is_writable true}
{:name "FEEDBACK",
:schema "PUBLIC",
:description nil,
:is_writable true}
...
```
So we can create the same shape (i omit description and is_writable to
seemingly no deleterious effect).
reports that a customer has around 200k tables spread across 300
schemas. the fields query by default would just get a single cursor into
these fields, passing all schema into the query.
This can take quite a long time. Our own horror show schema would take
~45 minutes to sync. But holding a cursor open this long just seems to
invite bad times.
I've taken the logic and just called it multiple times, once per schema.
* WRK-874: Populate preview for List View customization with sample values
* WRK-874: Set default and min size for list view on dashboard
* WRK-874: Fix List view card size on the dashboard
* WRK-874: Add e2e test for empty column placeholder
* Fix lint errors
* Add horizontal padding for List view
* Refactor generatePreviewSample
* Stretch join data sections when needed
* Tweak padding to avoid jump
* Center 'on' in join condition
* Make gap consistent
* Tighten margins to avoid jump
* Tweak padding to make all wells the same height
* BOT-310 lower the weight on the :rrf score for semantic search results
This seems to give a better blend between the :rrf score coming from the semantic search hybrid query and the other
post-search metadata scorers.
* Add with-weights and with-only-semantic-weights test util macros
* Use semantic.tu/with-weights in scoring tests
* Update query tests to use semantic.tu/with-only-semantic-weights
This prevents other metadata scorers from overriding the expected rankings for these tests. The query tests are
largely focused on making sure the basic machinery of semantic queries are working. Tests exercising individual
scorers are in scoring_test.clj
* Add tests for :official_collection and :verified "premium" scorers
Fixes#35561
We currently don't allow filtering on null foreign key values, but that is a reasonable thing to want to do. This extends the foreign key drill to support null values.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-authored-by: Claude Code[bot] <noreply@anthropic.com>
Co-authored-by: William Zimrin <william@metabase.com>
* Unpolished WIP hackathon code for additional driver support
* fix kondo
* add transforms/table featuer for supported drivers
* fix transforms for dbs with no schema
* rename transform datasets to prevent foreign key overlap
* Add clickhouse support
* fix clickhouse tests
* fix redshift tests
* fix bigquery failures
* kondo and filter tests by feature
* fix transform dataset for bigquery
* fix sqlserver transforms with joins
* fix clickhouse order by when no non nullable field is available
* add more complex test with join, filter, aggregation and ordering
* use different metadata provider
* filter for left join feature in test
* improve sqlserver compile-transform
* done use legacy mbql in test, refactor normalize-name
* clean up
* add sqlserver error test
* Squashed commit of the following:
commit a11b602e19
Author: Romeo Van Snick <romeo@romeovansnick.be>
Date: Thu Aug 21 17:44:13 2025 +1000
Add message when a new transform will be created
commit 2cfdbfaba3
Author: Romeo Van Snick <romeo@romeovansnick.be>
Date: Thu Aug 21 17:37:36 2025 +1000
Allow creating values in schema form select
commit 17ff4dcb3f
Author: Romeo Van Snick <romeo@romeovansnick.be>
Date: Thu Aug 21 16:51:13 2025 +1000
Allow selecting schemas always, not only when there are schemas
* create schemas
* fix redshift
* add snowflake
* add sqlserver
* add clickhouse
* add bigquery
* Add test
* add default impl
* fix redshift tests
* Move SchemaFormSelect to components folder
* Allow creating schemas in the target update modal
* Allow initial value in SchemaFormSelect
* Only render schema field when the database supports :schemas
* Do not render new schema item if value is empty
* Add unit tests for SchemaFormSelect
* Add e2e tests for schema creation
* check schema exists first
* fix sql jdbc schema exists and add create schema permissions test
* update changelog
* parse query and add into for sqlserver
* add type hints for linter
* Address comments
* realize bigquery execute-raw-queries
* Disable target schema link if it does not exists (yet)
* Add tooltip when schema is new
* Use hydrated table from transform
* Add e2e tests
* Allow existing schema link to render as enabled even when table does not exist yet
* Do not add default to databaseId
* Add tests for existing schema with new table
* Fix e2e test
* address comments
* rename connection-details to conn-spec
* Compare against null not empty string
* use find first
* Await request to avoid flakiness in hover
---------
Co-authored-by: Timothy Dean <timothy@timothydean.dev>
Co-authored-by: Romeo Van Snick <romeo@romeovansnick.be>
* Add last_run, next_run and tag id filters
Closes QUE-2156
* Placate kondo
* Test with scheduled jobs and filtering by tags
* Fix next_run_start_time test
* Return run timestamps as timezone localized strings
* Handle Zulu
* Prepare job list table for next_run and transforms
* Add JobListParams type and use it in getJobListUrl
* Add new parameters to listTransformJobs handler
* Parse params and pass them down on job list page
* Add filters to job list page
* Add tooltipPosition prop to Icon
* Add TimezoneIndicator component
* Allow rendering React nodes in table headers
* Add timezone indicators to job list and transform list
* Add timezoneToUTCOffset helper
* Add timezone to cron explanation
* Format timestamps as ISO-8601 strings with timezone
* Add next run timestamp
* Hide transform count in list for now
* Add next_run to TransformJob
* Add timezone indicators to run list
* Avoid wrapping timestamps in run list
* Move shared widgets to transforms/components
* Allow passing available shortcuts to filter picker
* Disable shortcuts that are in the past for Next run, which is always in the future
* Add e2e test for filters
* Add UTC offset in tests
* Report run timestamps in UTC and expose system timezone
The system timezone is exposed in /api/session/properties as "system-timezone".
* Expose system-timezone to admins only
* Fix setting docstring
* Fix serdes baseline-comparison-test
* Make transform_job API return run timestamps in UTC
* Placate kondo
* Timezone
* Use system-timezone instead of report-timezone
* Add tests
* Fix test string
---------
Co-authored-by: Romeo Van Snick <romeo@romeovansnick.be>
Co-authored-by: Alexander Polyankin <alexander.polyankin@metabase.com>
The Store API's OpenAPI description (OAD) can change over time, but
Metabase's Store client would not notice. For example, the client
would fail a HTTP request, not because the server rejected it, but
because the client "believes" it needs to send a field that the server
no longer requires.
Instead of caching the client, and thus OAD, for the whole lifetime of
the Metabase instance, cache it only for 5min.
Fixes: 1dd56e5f14
References: https://metaboat.slack.com/archives/C010ZSXQY87/p1757002462314229
2025-09-05 15:37:19 +02:00
6227 changed files with 666751 additions and 364753 deletions
This guide covers Clojure and ClojureScript coding conventions for Metabase. See also: `CLOJURE_STYLE_GUIDE.adoc` for the Community Clojure Style Guide.
Write like you're talking to a colleague. Be conversational, not formal. Get people what they need quickly. Know your audience and match the complexity.
## Tone and Voice
**Do:**
- Use contractions ("can't" not "cannot")
- Say "people" or "companies" instead of "users"
- Be friendly but not peppy
- Acknowledge limitations honestly ("that's on us, not them")
- Jokes and Easter eggs are okay (permit them, don't suggest them)
**Don't:**
- Use exclamation points excessively
- Rely on tired tropes about nerdiness
- Use corporate jargon ("utilize", "offerings", "actionable insights")
- Tell people something is cool (show them instead)
## Structure and Clarity
**Lead with the important stuff:**
- Most important information first
- Lead with the ask, then provide context
- Cut text that adds little value (when in doubt, cut it)
- Each paragraph should have one clear purpose
**Make headings do the work:**
- Convey your actual point, not just the topic
- "Use headings to highlight key points" not "How to write a good heading"
- Use sentence case, no punctuation except question marks
- No links in headings unless entire heading is a link
## Instructions and Examples
**Tell people what to do:**
- Give the action before explaining why
- Put commands in execution order
- Provide context, steps, and reasoning
- Don't describe tasks as "easy" or "simple"
**Answer unasked questions:**
- Include answers to "dumb" questions you had when learning
- Don't literally include the question, just give the answer
## Formatting
**Code vs. UI elements:**
- Backticks for code, variable names, parameters only
- **Bold** for UI elements and labels (e.g., "Click the **Save** button")
- Don't use backticks or quotes for UI elements
## Writing Mechanics
- Limit pronouns when introducing new terms (repeat the term to reinforce it)
- Ampersands only in proper nouns, never as substitute for "and"
## Terminology Preferences
Use familiar terms from tools people already know:
- "Summarize" (like Excel) instead of "aggregate"
- "Take a look at" instead of "reference"
- "Filter" instead of technical database terms when appropriate
description: Review Clojure and ClojureScript code changes for compliance with Metabase coding standards, style violations, and code quality issues. Use when reviewing pull requests or diffs containing Clojure/ClojureScript code.
allowed-tools: Read, Grep, Bash, Glob
---
# Clojure Code Review Skill
@./../_shared/clojure-style-guide.md
@./../_shared/clojure-commands.md
## Review guidelines
**What to flag:**
- Check compliance with the Metabase Clojure style guide (included above)
- If `CLOJURE_STYLE_GUIDE.adoc` exists in the working directory, also check compliance with the community Clojure style guide
- Flag all style guide violations
**What NOT to post:**
- Do not post comments congratulating someone for trivial changes or for following style guidelines
- Do not post comments confirming things "look good" or telling them they did something correctly
- Only post comments about style violations or potential issues
Example bad code review comments to avoid:
> This TODO comment is properly formatted with author and date - nice work!
> Good addition of limit 1 to the query - this makes the test more efficient without changing its behavior.
> The kondo ignore comment is appropriately placed here
> Test name properly ends with -test as required by the style guide.
**Special cases:**
- Do not post comments about missing parentheses (these will be caught by the linter)
## Quick review checklist
Use this to scan through changes efficiently:
### Naming
- [ ] Descriptive names (no `tbl`, `zs'`)
- [ ] Pure functions named as nouns describing their return value
- [ ]`kebab-case` for all variables and functions
- [ ] Side-effect functions end with `!`
- [ ] No namespace-alias repetition in function names
### Documentation
- [ ] Public vars in `src` or `enterprise/backend/src` have useful docstrings
- [ ] Docstrings use Markdown conventions
- [ ] References use `[[other-var]]` not backticks
- [ ]`TODO` comments include author and date: `;; TODO (Name 1/1/25) -- description`
### Code Organization
- [ ] Everything `^:private` unless used elsewhere
- [ ] No `declare` when avoidable (public functions near end)
- [ ] Functions under 20 lines when possible
- [ ] No blank lines within definition forms (except pairwise constructs in `let`/`cond`)
- [ ] Lines ≤ 120 characters
### Tests
- [ ] Separate `deftest` forms for distinct test cases
- [ ] Pure tests marked `^:parallel`
- [ ] Test names end in `-test` or `-test-<number>`
description: Guide Clojure and ClojureScript development using REPL-driven workflow, coding conventions, and best practices. Use when writing, developing, or refactoring Clojure/ClojureScript code.
---
# Clojure Development Skill
@./../_shared/development-workflow.md
@./../_shared/clojure-style-guide.md
@./../_shared/clojure-commands.md
## REPL-Driven Development Workflow
- Start with small, fundamental functions:
- Identify the core features or functionalities required for your task.
- Break each feature down into the smallest, most basic functions that can be developed and tested independently.
- Write and test in the REPL:
- Write the code for each small function directly in the REPL (Read-Eval-Print Loop).
- Test it thoroughly with a variety of inputs, including typical use cases and relevant edge cases, to ensure it
behaves as expected.
- Integrate into source code:
- Once a function works correctly in the REPL, move it from the REPL environment into your source code files (e.g.,
within appropriate namespaces).
- Gradually increase complexity:
- Build upon tested, basic functions to create more complex functions or components.
- Compose smaller functions together, testing each new composition in the REPL to verify correctness step by step.
- Ensure dependency testing:
- Make sure every function is fully tested in the REPL before it is depended upon by other functions.
- This ensures that each layer of your application is reliable before you build on it.
- Use the REPL fully:
- Use the REPL as your primary tool to experiment with different approaches, iterate quickly, and get immediate
feedback on your code.
- Follow functional programming principles:
- Keep functions small, focused, and composable.
- Use Clojure's functional programming features—like immutability, higher-order functions, and the standard
library—to write concise, effective code.
## How to Evaluate Code
### Bottom-up Dev Loop
1. Write code into a file.
2. Evaluate the file's namespace and make sure it loads correctly with:
description: Review documentation changes for compliance with the Metabase writing style guide. Use when reviewing pull requests, files, or diffs containing documentation markdown files.
allowed-tools: Read, Grep, Bash, Glob
---
# Documentation Review Skill
@./../_shared/metabase-style-guide.md
## Review mode detection
**IMPORTANT: Before starting the review, determine which mode to use:**
1.**PR review mode**: If the `mcp__github__create_pending_pull_request_review` tool is available, you are reviewing a GitHub PR
- Use the pending review workflow to post all issues as one cohesive review
- Follow the workflow steps in "PR review mode format" below
2.**Local review mode**: If the MCP tool is NOT available, output issues in the conversation
- Format all issues in a numbered markdown list (as described in "Feedback format" below)
## Review process
1.**Detect review mode** - Check if `mcp__github__create_pending_pull_request_review` is available
2. Read the changes through once to understand intent
3. Check all issues that violate style guide or significantly impact readability
4. Only flag issues worth mentioning - if it won't make a material difference to the reader, skip it
5.**REQUIRED: Number ALL feedback sequentially** - Start from Issue 1 and increment for each issue found
## Review checklist
Run through the diff looking for these issues:
**Tone and voice:**
- [ ] Formal/corporate language ("utilize" not "use", "offerings", etc.)
- [ ] "Users" instead of "people" or "companies"
- [ ] Excessive exclamation points or overly peppy tone
- [ ] Telling readers something is cool instead of showing them
**Structure and clarity:**
- [ ] Important information buried instead of leading
- [ ] Verbose text that adds little value
- [ ] Paragraphs without clear purpose
- [ ] Vague headings that don't convey the point
- [ ] Instructions explain "why" before telling "what to do"
- [ ] Tasks described as "easy" or "simple"
**Links and references:**
- [ ] Linking the word "here" instead of descriptive text
- [ ] Links in headings (unless entire heading is a link)
**Formatting:**
- [ ] Backticks on UI elements (should use **bold**)
- [ ] Backticks used as quotation marks
- [ ] Ampersands as "and" substitute (except proper nouns)
- [ ] Inconsistent list formatting
**Code and examples:**
- [ ] Code examples that don't work or would error
- [ ] Commands not in execution order
- [ ] Full-width screenshots instead of scoped UI elements
- [ ] Excessive or unnecessary images
**Sentence construction:**
- [ ] Overuse of pronouns when introducing new terms
| `Button name` or `UI element` | Should use **bold** not backticks |
| we can do X, our feature | Should be "Metabase" or "it" |
| click here, read more here | Need descriptive link text |
| easy, simple, just | Remove condescending qualifiers |
| users | Should be "people" or "companies" if possible |
## Feedback format
**MANDATORY REQUIREMENT: Every single issue MUST be numbered sequentially starting from Issue 1.**
This numbered format is NON-NEGOTIABLE. It allows users to efficiently reference specific issues (e.g., "fix issues 1, 3, and 5") and track which feedback has been addressed.
### Local review mode format
When outputting issues in the conversation (local mode), use this format:
```markdown
## Issues
**Issue 1: [Brief title]**
Line X: Succinct description of the issue
[code or example]
Suggested fix or succinct explanation
**Issue 2: [Brief title]**
Line Y: Description of the issue
Suggested fix or explanation
**Issue 3: [Brief title]**
...
```
**Examples:**
> **Issue 1: Backticks on UI elements**
> Line 42: This uses backticks for the UI element. Use **bold** instead: **Filter** not `Filter`.
> **Issue 2: Formal tone**
> Line 15: This could be more conversational. Consider: "You can't..." instead of "You cannot..."
> **Issue 3: Vague heading**
> Line 8: The heading could be more specific. Try stating the point directly: "Run migrations before upgrading" vs "Upgrade process"
### PR review mode format
When posting to GitHub (PR mode), use the **pending review workflow**:
**Workflow steps:**
1.**Start a review**: Use `mcp__github__create_pending_pull_request_review` to begin a pending review
- This creates a draft review that won't be visible until submitted
2.**Get diff information**: Use `mcp__github__get_pull_request_diff` to understand the code changes and line numbers
- This helps you determine the correct file paths and line numbers for comments
3.**Identify ALL issues**: Read through all changes and identify every issue worth mentioning
- Collect all issues before posting any comments
- Number them sequentially (Issue 1, Issue 2, Issue 3, etc.)
4.**Add review comments**: Use `mcp__github__add_pull_request_review_comment_to_pending_review` for each issue
- **CRITICAL**: Post ALL comments in a SINGLE response using multiple tool calls in parallel
- Each comment should reference a specific file path and line number from the diff
- Start each comment body with `**Issue N: [Brief title]**`
- Include the description and suggested fix
5.**Submit the review**: Use `mcp__github__submit_pending_pull_request_review` to publish all comments at once
- Use event type `"COMMENT"` (NOT "REQUEST_CHANGES") to make it non-blocking
- **Do NOT include a body message** - Leave the body empty or omit it entirely
- All comments will appear together as one cohesive review
**Comment format example:**
```
**Issue 1: Backticks on UI elements**
This uses backticks for the UI element. Use **bold** instead: **Filter** not `Filter`.
```
**IMPORTANT**:
- Each issue gets its own review comment attached to the pending review
- Number ALL comments sequentially (Issue 1, Issue 2, Issue 3, etc.)
- Always start the comment body with `**Issue N: [Brief title]**`
- **MUST add all comments in parallel in a single response** - Do NOT add them one after another in separate responses
- Do NOT output a summary message to the conversation - only post GitHub review comments
- When submitting the review, do NOT include a body parameter (or leave it empty) to avoid cluttering the PR with summary text
- The review will appear as a single review with multiple comments when submitted
## Final check
1. Remove any issues from your assessment that won't make a material difference to the reader if addressed. Only flag issues worth the author's time to fix.
2.**Verify all issues are numbered sequentially** starting from Issue 1 with no gaps in numbering.
3. Confirm the format exactly matches: `**Issue N: [Brief title]**` where N is the issue number.
4.**In PR mode**: Verify each issue was posted as a separate GitHub comment (not output to conversation).
description: Write documentation following Metabase's conversational, clear, and user-focused style. Use when creating or editing documentation files (markdown, MDX, etc.).
allowed-tools: Read, Write, Grep, Bash, Glob
---
# Documentation Writing Skill
@./../_shared/metabase-style-guide.md
## When writing documentation
### Start here
1.**Who is this for?** Match complexity to audience. Don't oversimplify hard things or overcomplicate simple ones.
2.**What do they need?** Get them to the answer fast. Nobody wants to be in docs longer than necessary.
3.**What did you struggle with?** Those common questions you had when learning? Answer them (without literally including the question).
### Writing process
**Draft:**
- Write out the steps/explanation as you'd tell a colleague
- Lead with what to do, then explain why
- Use headings that state your point: "Set SAML before adding users" not "SAML configuration timing"
**Edit:**
- Read aloud. Does it sound like you talking? If it's too formal, simplify.
- Cut anything that doesn't directly help the reader
- Check each paragraph has one clear purpose
- Verify examples actually work (don't give examples that error)
**Polish:**
- Make links descriptive (never "here")
- Backticks only for code/variables, **bold** for UI elements
- American spelling, serial commas
- Keep images minimal and scoped tight
**Format:**
- Run prettier on the file after making edits: `yarn prettier --write <file-path>`
- This ensures consistent formatting across all documentation
### Common patterns
**Instructions:**
```markdown
Run:
\`\`\`
command-to-run
\`\`\`
Then:
\`\`\`
next-command
\`\`\`
This ensures you're getting the latest changes.
```
Not: "(remember to run X before Y...)" buried in a paragraph.
**Headings:**
- "Use environment variables for configuration" ✅
- "Environment variables" ❌ (too vague)
- "How to use environment variables for configuration" ❌ (too wordy)
**Links:**
- "Check out the [SAML documentation](link)" ✅
- "Read the docs [here](link)" ❌
### Watch out for
- Describing tasks as "easy" (you don't know the reader's context)
- Using "we" when talking about Metabase features (use "Metabase" or "it")
description: Review TypeScript and JavaScript code changes for compliance with Metabase coding standards, style violations, and code quality issues. Use when reviewing pull requests or diffs containing TypeScript/JavaScript code.
allowed-tools: Read, Grep, Bash, Glob
---
# TypeScript/JavaScript Code Review Skill
@./../_shared/typescript-commands.md
## Code Review Guidelines
Review pull requests with a focus on:
- Compliance with project coding standards and conventions
description: Write TypeScript and JavaScript code following Metabase coding standards and best practices. Use when developing or refactoring TypeScript/JavaScript code.
clojure.java.jdbc/with-db-connection{:message"Use sql-jdbc.execute/do-with-connection-with-options (for drivers) or t2/with-connection (for the app DB) instead of jdbc/with-db-connection"}
clojure.java.jdbc/with-db-metadata{:message"Use sql-jdbc.execute/do-with-connection-with-options + .getMetaData instead of jdbc/with-db-metadata"}
clojure.pprint/pprint{:message"Use metabase.util.log instead of clojure.pprint/pprint"}
clojure.pprint/print-table{:message"Use metabase.util.log instead of printing directly to the console"}
clojure.string/lower-case{:message"Use metabase.util/lower-case-en instead of clojure.string/lower-case"}
clojure.string/upper-case{:message"Use metabase.util/upper-case-en instead of clojure.string/upper-case"}
compojure.core/context{:message"Use metabase.api.util.handlers/route-map-handler instead of compojure.core/context, it is much faster and supports OpenAPI spec generation"}
clojure.core/eval{:message"Don't use eval outside of tests"}}
:discouraged-namespace
{metabase.query-processor.store{:message"Please do not use the QP Store outside of legacy QP or driver code. Use MBQL 5/Lib or lib.metadata.jvm/applicaton-database-metadata-provider instead."}}}}
{metabase.legacy-mbql.jvm-util{:message"Avoid legacy MBQL utils, use Lib instead"}
metabase.legacy-mbql.normalize{:message"Avoid legacy MBQL normalization, use MBQL 5 + lib/normalize instead"}
metabase.legacy-mbql.predicates{:message"Avoid legacy MBQL utils, use Lib instead"}
metabase.legacy-mbql.schema{:message"Avoid the legacy MBQL schema, use MBQL 5 + lib.schema instead"}
metabase.legacy-mbql.schema.helpers{:message"Avoid the legacy MBQL schema, use MBQL 5 + lib.schema instead"}
metabase.legacy-mbql.schema.macros{:message"Avoid the legacy MBQL schema, use MBQL 5 + lib.schema instead"}
metabase.legacy-mbql.util{:message"Avoid legacy MBQL utils, use Lib instead"}
metabase.query-processor.store{:message"Please do not use the QP Store outside of legacy QP or driver code. Use MBQL 5/Lib or lib.metadata.jvm/applicaton-database-metadata-provider instead."}}}}
printable-namespaces
{:linters
@@ -779,7 +807,7 @@
clojure.string/lower-case{:level:off}
clojure.string/upper-case{:level:off}}}}
;; enable all the extra linters we haven't enabled in the core app yet in MLv2. It's all new code so we can adhere
;; enable all the extra linters we haven't enabled in the core app yet in Lib. It's all new code so we can adhere
;; to the new stricter rules from day 1
metabase-lib
{:linters
@@ -799,48 +827,120 @@
metabase.test/with-temp{:message"Don't use application database inside Lib code"}}
:discouraged-namespace
{metabase.driver{:message"MLv2 code should be independent of driver code/methods"}
metabase.test{:message"MLv2 tests should only use MLv2 test helpers"}
metabase.test.data{:message"MLv2 tests should only use MLv2 test helpers"}
metabase.util.malli.schema{:message"Prefer metabase.lib.schema schemas in MLv2 code"}
toucan.db{:message"Don't use application database inside MLv2 code"}
toucan.models{:message"Don't use application database inside MLv2 code"}
toucan.util.test{:message"Don't use application database inside MLv2 code"}
toucan2.core{:message"Don't use application database inside MLv2 code"}}}}
{metabase.driver{:message"Lib code should be independent of driver code/methods"}
metabase.test{:message"Lib tests should only use Lib test helpers"}
metabase.test.data{:message"Lib tests should only use Lib test helpers"}
metabase.util.malli.schema{:message"Prefer metabase.lib.schema schemas in Lib code"}
toucan.db{:message"Don't use application database inside Lib code"}
toucan.models{:message"Don't use application database inside Lib code"}
toucan.util.test{:message"Don't use application database inside Lib code"}
toucan2.core{:message"Don't use application database inside Lib code"}}}}
metabase-lib-excluding-legacy-mbql
{}
legacy-mbql-namespaces
{:linters
{:discouraged-var
{metabase.legacy-mbql.util/unique-name-generator{:message"Use metabase.lib.util/unique-name-generator or non-truncating-unique-name-generator"}}}}
{:discouraged-namespace
{metabase.legacy-mbql.jvm-util{:level:off}
metabase.legacy-mbql.normalize{:level:off}
metabase.legacy-mbql.predicates{:level:off}
metabase.legacy-mbql.schema{:level:off}
metabase.legacy-mbql.schema.helpers{:level:off}
metabase.legacy-mbql.schema.macros{:level:off}
metabase.legacy-mbql.util{:level:off}
metabase.query-processor.store{:level:off}}
;; the `metabase.legacy-mbql` stuff is all deprecated, so we don't need to warn inside the module about other stuff
;; in the module being marked deprecated or discouraged.
metabase.app-db.core/query{:message"Don't use application database directly in QP code; use QP Store/metadata provider"}
metabase.app-db.core/reducible-query{:message"Don't use application database directly in QP code; use QP Store/metadata provider"}
metabase.app-db.core/query{:message"Don't use application database directly in QP code; use the MetadataProvider"}
metabase.app-db.core/reducible-query{:message"Don't use application database directly in QP code; use the MetadataProvider"}
metabase.lib.convert/->legacy-MBQL{:level:off}
metabase.lib.core/->legacy-MBQL{:level:off}}}}
qp-source-namespaces
{}
general-qp-and-driver-test-namespaces
{:linters
{:discouraged-var
@@ -850,4 +950,11 @@
toucan2.tools.with-temp/with-temp{:message"Use lib.tu/mock-metadata-provider in QP tests. Exceptions: tests for code that doesn't use MPs or models not supported by MPs"}}
:message"Don't use System/exit directly. Instead throw an ex-info with {:babashka/exit <exit-code>} (and optionally :mage/error) let the mage runner handle exiting the process"}}}}}}}
> If that is your first contribution to Metabase, please sign the [Contributor License Agreement](https://docs.google.com/a/metabase.com/forms/d/1oV38o7b9ONFSwuzwmERRMi9SYrhYeOrkbmNaq9pOJ_E/viewform) (unless it's a tiny documentation change). Also, if you're attempting to fix a translation issue, please submit your changes to our [Crowdin project](https://crowdin.com/project/metabase-i18n) instead of opening a PR.
> If that is your first contribution to Metabase, please sign the [Contributor License Agreement](https://docs.google.com/a/metabase.com/forms/d/1oV38o7b9ONFSwuzwmERRMi9SYrhYeOrkbmNaq9pOJ_E/viewform). Also, if you're attempting to fix a translation issue, please submit your changes to our [Crowdin project](https://crowdin.com/project/metabase-i18n) instead of opening a PR.
@${githubUsername} You have modified embedding SDK code. Please make sure the PR title follows [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) style.
Here are all the supported types that will be shown in the changelog:
constcomment=`${CONVENTIONAL_COMMENT_IDENTIFIER}
@${githubUsername} You have modified embedding code. Please make sure the PR title follows [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) style.
Here are all the supported types:
- \`feat\`
- \`fix\`
- \`perf\`
- \`docs\`
- \`style\`
- \`refactor\`
- \`test\`
- \`build\`
- \`ci\`
These types \`docs\`, \`style\`, \`refactor\`, \`test\`, \`build\`, and \`ci\` are also encouraged for non-changelog related tasks.
Please also make sure to include \`sdk\` scope, otherwise, the changelog will not include this task.
message:"⚠️ Changes were made to the <b>Embedding SDK Package</b>. Review this PR and apply one of the following labels.<br><br>- <code>release-notes</code> — select if this PR must be included in the main Metabase changelog release notes.<br>- <code>no-release-notes</code> — select if this PR should not be included in the main Metabase changelog release notes."
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.