`entity_id` columns on Databases, Tables and Fields are new in #53076.
This PR uses `entity_id` on Fields as their `:ident` in the metadata,
rather than the concatenated names previously used.
The names were a bit too fragile, since occasionally people rename
Fields or Tables in the appdb to retain the identity of something that
got renamed in place in their data warehouse. (That isn't exactly
supported, but it's also not blocked.)
With random (or hashed and saved) `entity_id`s for Fields used as their
`:ident`s instead, references to fields will not need to be updated if
fields or tables are renamed but their `entity_id`s preserved.
**On memoization**
Since the `entity_id` is `NULL` for existing fields and they're only
being populated in the background, we need to populate them on demand
when reading metadata. Since fields depend on their tables and tables on
their databases, this meant a lot of extra appdb reads at first.
This change includes memoization added to `serdes/hydrated-hash`, such
that any table or database only gets read once, and memoized forever.
There's a big comment about the memory impact of this; in summary it
should be safe since this is on demand, and holds only DBs/tables
and not fields.
A later PR will add a background job as mentioned, that will populate
all the `entity_id` columns with hashed values. After that, the cache
will always be empty.
Note that this memoization is not going to work if a table or field is
renamed before the background job catches up! I consider this an
acceptable loss - renaming fields is an admin action that requires
editing the appdb. The guide for doing that can include making sure all
`entity_id` fields are populated first.
* fix: use standard user order sort for group list
* fixup! remove alba from user fixtures
* fixup! better user model creation
* fixup! remove test-user? filter
* fixup! remove added test and use exisiting sort test
* fix: allow password reset if google sso disabled
* fixup! correct arg type to keyword in message test
* fixup! remove empty line
* fixup! add public docstring
* fixup! add temp google client id
* Use NativeEditor.type in test
* Use NativeEditor.type in all tests that aren't
* Stop using .as(editor) for native editor
* Do not return anything from startNewNativeQuestion to avoid confusion
* Fix more tests
* Fix movetostart/movetoend
- Ctrl + A is "select all" on non-apple machines, but on apple machines it's same as "home"
---------
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
* Error explicitly when sandboxing/impersonation configs conflict
* Fix tests
* Fix kondo warning, use consistent ns aliases
* Fix another test
* Cleanup, consistent aliases
* Add an extra check of *current-user-id*
* Wrap east/west filterBounds to [-180,180] in LeafletMap.jsx
* Split filters that cross the antimerdian into two parts in update-lat-lon-filter
If the bounds cross the 180th meridian, split the filter into two parts ORed togehter. One part covering the
[west, 180] range and the other covering [-180, east].
* Add tests for bounds crossing the antimerdian
* Rename longDelta to lngDelta to conform to standard leaflet abbreviation
* Review feedback: use shorthand notation for east and west filterBounds
* Add e2e tests
* Fix existing e2e test
realMouseDown expects to have x and y coords passed in an options map.
* Use common pinMapSelectRegion test helper in new e2e tests
* wip logic to handle 'personal' for collectionId when creating dashboards
* unit test for useCreateDashboardApi supporting 'personal'
* add e2e test for createdashboardmodal
* collection browser support for 'personal'
* some actual decent logic
* change jest tests to use root collcetion
* fix type error in stories
* fix create dashboard modal support for root
* yeah i shouldn't have commited a .only
* self review, better docs + removed un-used type
* typo
* another step of self review: changed comments and removed useless
beforeEach
* updated docs
* Apply suggestions from code review
Co-authored-by: Phoomparin Mano <poom@metabase.com>
* re-format file after suggestions were applied from gh
* refactor(e2e): use cy.wait().then instead of let =
---------
Co-authored-by: Phoomparin Mano <poom@metabase.com>
* Enable pivot tables with only pivot columns
* Re-add one condition so that tests pass
* Tweak logic
* Fix logic again, and add cypress test
* Try again to fix conditional
* Enable results metadata during qp/process-query as it seems to also control insights which we need for slack alert trendlines
* Removing outdated alerts w/ models test
* fix: add logging during connection health checks
Trying to get ahead of #52458, add logs around startup health checks
* Add log for when health check is queued
* Use with-generator
* Replace for tu.rng
* Add ref to workflow, rename env
* Rename vars
* Remove former random ns
* Typo
* Adjust log level
* random-query-from
* Use random-query-from
* Adjust random exception handling
* Add with-gentest kondo hook
* Initial version of with-gentest
* Logging in test assertions
* Workflow, env var names, logging
* Remove redundant code
* Newline
* Seed for non exceptional queries
* Naming
* Cleanup
* Atom for seed
* Cleanup
* Whitespace
* Kondo
* Temporarily fix schema and suppress full-join, to be rebased out later
* Update .github/workflows/generative-query-test.yml
Co-authored-by: appleby <86076+appleby@users.noreply.github.com>
* Update test/metabase/test/gentest.clj
Co-authored-by: appleby <86076+appleby@users.noreply.github.com>
* Remove stale comment
* Use tu.rng
---------
Co-authored-by: appleby <86076+appleby@users.noreply.github.com>
* Improve ID test
* Improve ID test
* Improve ID test
* Improve ID test
* Add a test
* Between test
* Between test
* Between test
* Between test
* tests
* tests
* tests
* tests
* tests
* tests
* tests
* tests
* tests
* tests
* tests
* tests
* tests
* tests
* tests
* Make the UpsellBigCard more flexible
The product requirements for the Cloud upsell dictate that
the main CTA doesn't directly lead to Metabase website like in
all other cases. It opens a Migration modal instead.
UpsellBigCard had to be adjusted to enable that.
* Add Upsell Cloud component
* Resolve CLO-3576
* Use the cloud upsell instead of the previous CTA
* Adjust the UI
* Update stories
* Update related test
* Fix e2e test
* Add context for translators
* Adjust the vertical spacing
* Don't :return-drills-for-dimensions? for header-context-only drills
Dimensions can never have a "missing" value, i.e. (nil? value), and so none of these drills will ever apply when
called with a context built from dimension-contexts.
* Add comment with updated table for Pivot Cell clicks
#53076 added `:hook/entity-id` for databases, tables and fields, but this is incorrect behaviour.
The sync process periodically detects new tables and fields and adds them to the appdb. If multiple instances of Metabase are connected via serdes, and linked to the same data warehouses, then eg. a new table would get duplicated before the serdes process would be run.
Instead, the `entity_id`s should be based only on the hashes, the same as serdes for databases, tables and fields.
* Cast date binning back to date in postgres
* Remove unnecessary conversion to timestamp
* Fix existing test.
* Test that temporal columns don't change type when binned.
* Fix bucketing for postgres days
* Improve comment about casting interval addition
* Select single field in test for robustness
* Tag :to_timestamp to avoid other areas adding a type
* Whitespace
* Remove blank line at end of file
* Don't cast if type is nil.
* Add alias to native query in test
* Fix redshift
* Pass temporal unit when listing quick-filter options.
* Use with-temporal-bucket like a gentleman
* Use lib.temporal-bucket/temporal-bucket
* Whitespace
* check if the project is a nextjs project
* add nextjs imports and exports for nextjs projects
* add comments
* add notice on analytics.css and link to guide
* fix typo
* update snippet
* normalize path
* quote import path
* update generated directory
* use fileName for sampleComponents
* remove getImportSnippet
* use nextjs imports from the compatibility layer
* remove use client and export default from snippets
* fix checkIsInNextJsProject import
* add use client directive to pages with useReducer
* add suggestions on module resolution
* make import paths reasonable for next.js
* add example _app.tsx snippet
* trim snippets for Next.js
* show custom app snippets separately for next
* auto-generate custom app
* fix file extension
* do not show the react import step in nextjs
* add default root layout or app
* refine root layout generation logic
* add comments on root layout generation
* add next.js route to app and pages router
* add notes on API routes
* only generate AnalyticsPage when not using nextjs
* use writeFileSync
* create directory if it does not exist
* add messages on route created
* remove AppProps type from snippets
* inform users that a custom root layout has been added
* wait for root layout snippets to be ready
* use the correct layout file name in message
* cleanup snippet message
* highlight the layout file
* temporarily remove the moduleResolution warning
* cleanup messages helper
* cleanup exports
* cleanup analytics page snippet
* restore analytics.css
* add more info on the default provider
* fix whitespace in imports
* add a use-client directive to EmbeddingProvider
* highlight the analytics demo route
* add a sans-serif font by default
* do not show the Express server message if Express server is not there
* use glob to simplify path matching
* make analytics page snippet function pure
* make custom app snippet function pure
* add support for src directory
* add source prefix in router resolution
* trim layout snippet header and add eta
* strip the source prefix from import paths
* make src dir a constant
* add lightweight margin reset to sample analytics.css
* consistent source prefix
* parameterize the route name
* cleanup snippet whitespace
Co-authored-by: Mahatthana (Kelvin) Nomsawadi <me@bboykelvin.dev>
* rename to getNextJsPagesWrapperOrAppWrapperSnippet
* consolidate snippets file
* inline hasNextJsCustomAppOrRootLayout
Co-authored-by: Mahatthana (Kelvin) Nomsawadi <me@bboykelvin.dev>
---------
Co-authored-by: Mahatthana (Kelvin) Nomsawadi <me@bboykelvin.dev>
* Fix#50731
* Revert first fix as design team said there's a better way
* Remove HackyInvisibleTargetFiller
* Use max-width instead of width to prevent overflow if there's less than 300px available
* Add a repro for #50731
When wrapping literal values in :value clauses, fields with string-type ids (for example, fields after an aggregation)
were getting their type info from source-metadata. When the field in question was a postgres enum, the source
metadata was missing important data. Fetching info from both source-metadata and the type info of field-ref fixes
that.
These three entities have previously used their names as their serdes
keys. The names are still used for serdes but we want opaque, stable
`entity_id`s as well so that they can be used as `:ident`s in the field
refs overhaul. Then the name of a database, schema, table or field can
be changed without its `:ident` changing, and therefore without breaking
any questions that reference it.
* Updating the v2 serdes baseline export - it has `entity_id`s now
* Updating the exported audit database; it has entity_ids now
* Fix MySQL appdbs
Use lower case for all table/field names; hard-code the schema
"PUBLIC". (The actual schema for appdb is ignored.)
* run all smoke tests on OSS tests regularly
* skip certain smoke tests for oss v EE
* or not and
* union
* update prerelease tests
* run smoke tests on OSS except EE
* Add upsell for whitelabel/appearance
The most important change this PR introduces is a whitelabel route that's now present in the OSS and starter plans. How this works now:
- When the whitelabel token feature is not present, the route `/admin/settings/whitelabel` shows the upsell page
- When the whitelabel token feature gets activated, the route `/admin/settings/whitelabel` gets redirected to `/admin/settings/whitelabel/branding` and the upsell page is not visible anymore.
* Mantine v7-1 squash
update deps
update UI configs
easy fixes
happy little typescript
updates after rebase
test fixes
fix rebase issues
admin-2, collections, custom-columns, dashbaord and dashboard-cards suites
fix clause popovers
fixing rebase issue
more rebase sadness:
more e2e
autocomplete adjustment
fixing nested popovers
adjusting a couple more pickers
more e2e fixes
fix mantine provider, searchbox errors
unit test adjustments
theme colors, filter labels
Relative Date Picker
more unit test fixes
* e2e adjustments
* [MANTINE] ci: find css variables that are used but never defined (#52947)
* ci: find css variables that are used but never defined
* only search in frontend and enterprise/frontend
* whitelist mantine css variables
* temporarily allow some css variables
* read mantine variables directly from mantine css file
* UI Regressions
* lint and type fixes. green unit test
* Cherry pick updates to the never defined variables script (#53049)
* Update bin/find-never-defined-css-variables/find-never-defined-css-variables.ts
Co-authored-by: Mahatthana (Kelvin) Nomsawadi <me@bboykelvin.dev>
* test: checks that it doesn't confuse definitions with usages
---------
Co-authored-by: Mahatthana (Kelvin) Nomsawadi <me@bboykelvin.dev>
* more adjustments for e2e tests
* adjustments
* embedding sdk green, maybe unit tests?
* loki regressions
* test(sdk): migrate theme provider test to e2e to work with mantine v7 (#53290)
* test: migrate theme provider test to e2e to work with mantine v7
* re-put there things i removed by mistake
* preview and type fixes
* I think I'm making this better?
* fixing colors, unit tests
* scoping css vars and static classes, other things
* class prefix, scoping input css
* small adjustments
* shamelessly poach Nicolo's commit
* small fixes, should make Kamil happy :)
* finally green!
* type check
* chartsettingsinput fixes
* silly number types
* Adjustments for loki tests
* type fix
* undo Question View Download changes
* adjusting smart scalar
* updating loki reference
---------
Co-authored-by: Nick Fitzpatrick <nickfitz.582@gmail.com>
Co-authored-by: Nicolò Pretto <info@npretto.com>
Co-authored-by: Mahatthana (Kelvin) Nomsawadi <me@bboykelvin.dev>
* smooth out some rough edges for local e2e tests
* note for the normies
* remove premium embedding token
* add note about h2 app db
* update config file stuff
* Allow passing native extras to withDifferentDatabase
* Pass empty nativeExtras when switching databases
* Add reproduction for 53299
* Do not pass nativeExtras to withDifferentDatabase
* Do not update native-extras when chaning databases
* Remove with-extras overload on with-different-database
* Remove native-extras for with-different-database call sites
* Remove required throw from test
* Remove collection requirement from test
* convert pgarray to java array
* convert array to vector early
* add test and todo comment
* fix whitespace
* add tests for caching and emailing
* whitespace
* use feature and test other drivers
* update test names
* deduplicate native array methods
* remove vertica
* remove http from array agg test
* remove let
* fixes offset images when dnd-ing columns in dashcards
* we should be grateful for e2e tests
* -> mantine style prop
* fix this in the sortable list while I'm at it
* add clarification message to the cli
* update cli startup message
Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>
* line-wrap the cli startup message
* use metaba.se/sdk for docs link
---------
Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>
* Remove silent flag and re-enable multi-arch build
Check what's going on here
* test code
* fix test code
* Update Dockerfile_ubuntu
Fixes the multi-arch build
* Update Dockerfile_ubuntu
Have a feeling that the ca-certificates-java is making a mess here
* Update uberjar.yml
Add the logging to the process
* Update Dockerfile_ubuntu
* Update uberjar.yml
* Update run_metabase.sh
ensure the path lives with the user
* Update uberjar.yml
Alpine 21 is multi-arch, we should have that as well in the build script along ubuntu
* Update uberjar.yml
push explicitly all platforms
* Update uberjar.yml
Update actions to see if platform flag works
* Update uberjar.yml
This should push the multi-arch images without building again 🤞🏻
* Update uberjar.yml
return to old versions, just to keep stuff more consistent
* Update uberjar.yml
fix typo
* remove test code
---------
Co-authored-by: Ryan Laurie <iethree@gmail.com>
* feat: use liquibase session lock instead of transaction lock
Prevent the migration lock class of errors in non-h2 dbs by using a liquibase session lock instead of a transaction lock.
This type of lock would clear automatically when the connection closes for any reason.
Using this approach there is no need for the metabase release lock command except with h2.
The force migration command still is applicable though, as it does more things than just release the lock.
See https://github.com/blagerweij/liquibase-sessionlockFix#50212
* fix: mark migration 109 as none db
From Nathan Voxland:
From a liquibase-standpoint, deleting the changesets are just fine. As long as you don't accidentally reuse the id+author+path combo in the future. Databases that already ran it will not care that it's missing and new databases will just never see it to know. So it's more of an our-standards questions.
Alternately, you can set dbms="none" to keep it in the file but not have it actually apply anywhere. That doesn't change the checksum
* fix: error when force-releasing non-h2 locks
* test: add session lock test
* WIP - lots of streamlining
* revert dotenv addition
* ci fix
* some fixes
* make component testing work
* set some default ports
* fix component test ci
* fix all the e2e tests
* fix the other sdk tests
* update e2e docs
* fix embedding more
* fix tagged tests
* documentation
* split user and derived options
* better exit behavior
* more error handling
* revert saml change
* ❤️ typescript and booleans
* fix some stuff
* fixes
* stupid lockfile
* maybe keep the lockfile
* remove unnecessary diff
When fetching a param's possible values, we were originally looking for concrete columns and checking result metadata.
However, if the query is against a model and has an aggregation, the thing you are filtering on isn't a concrete field
and probably isn't in result metadata. Checking filterable-columns lets us find the appropriate field in that case. The practical result is that we actually find remapped values for a param when it is linked to an aggregated query against a model.
* Check cycles on PUT to /card/:id
* Remove extra protection from saving cycles.
* cljfmt
* kondo
* Better threading
* Cleaner tests
* Fix test
* Better naming in metabase.api.card/dataset-query->query
* Rename `dataset-query` arg to `dq-fragment`.
* Normalize query on backend.
* Throw exceptions with user messages.
Also rework `dataset-query->query`.
* Unshadow card-type in doseq
* Make consistent use of query/dataset-query
* Remove normalization from lib.query
* Don't shadow `query`.
* Rename `dq-fragment` to `query-fragment`
* `timelines` module
* Move some render-related timeline stuff into the timeline module
* Remove unused /api/card/:id/timelines endpoint
* PR feedback and test fixes 🔧
* Create dashboard questions for x-rays
* Don't make subcollections for autogened dashboards
Before, for an admin we'd create a structure like:
```
- Our analytics
- Automatically Generated Dashboards
- A look at Accounts
- Dashboard
- Card 1
- Card 2
- ...
```
Or, for regular users, a structure like:
```
- Your personal collection
- A look at Accounts
- Dashboard (A look at Accounts)
- Card 1
- ...
```
Now, we'll create the following structure:
```
- Our Analytics (OR Your personal collection, for non-admins)
- Automatically Generated Dashboards
- Dashboard (A look at Accounts)
```
Other autogenerated dashboards will be placed alongside.
* Fix question not found error showing after navigating from dashboards
* Add interactive dashboard tests
* Add editable dashboard tests
* Fix failed tests
* Add an assertion to ensure we display ad-hoc question
* update enable-embedding-sdk based on embedding-app-origins-sdk
|enable-embedding-sdk key exists? | then we:
|---------------------------------|---------
|false | no-op
|true | set enable-embedding-sdk to true when `embedding-app-origins-sdk` exists
* actually need to set it to false when embedding-app-origins-sdk dne
* reformat liquibase migration
* make the query run on h2, too
* per-db sql + update comment
* use preconditions to simplify update sql
* Allow private slack channels to be listed for alerts/subscriptions.
Requires the groups:read scope to see private channels
* manifest.yaml changed so new installations get it by default
* things are kept compatible if the token does not have the scope
* you can manually add the scope to the app in slack.
private channels will then up when the channels are refreshed
* Update the slack manifest.yaml inlined in the docs
* Add re-installation guidance to slack docs for `groups:read`
* Ensure slack token is set in conversations-list test
* Refactor conversations-list to take :private-channels opt
* Add refresh-channels-and-usernames! test to check groups:read behaviour
* prevent duplicated SET_EDITING_DASHBOARD dispatches from overriding edit dashboard state if the editing dashboard is the same dashboard
* small improvements
* add test coverage
* impl pr feedback
* unmark as flaky: should not reload the model for record in the same model
* unmark as flaky: scenarios > search should allow users to paginate results
* unmark as flaky: scenarios > model indexes should be able to see details of a record outside the first 2000
* unmark as flaky: should be able to search model index values and visit detail records
* Remove the custom permissions upsell component
* Implement upsell for advanced permissions
This applies to both:
- `/admin/permissions/data/group` and
- `/admin/permissions/data/database/` routes.
* Remove the old feature from the enterprise feature level permissions
* make dashboard dirty if tabs are edited and add test coverage for leaving dashboard after editing a dashboard or its tabs
* handle rename tab, add rename tab test coverage, simplify test coverage
* makes ci ts happy
* clean up test a bit
* rename helper
* Return column corresponding to aggregation value in SmartScalar getCurrentMetricData
Closes#46168
The returned clicked object had the `value` from the aggregation cell, but the `column` from the date dimension, which
breaks the quick filter drill (and probably others).
Instead, return the `column` that corresponds to the `value`.
* Add e2e test for smartscalar quick filter drill
* remove unsafe lifecycles from visualization component
* add e2e tests
* remove extra comma in test helpers
* only run tests under strict mode when a flag is passed
* extract console assertion to a test helper
* update tests for known console errors
* swap new and previous props
* combine old state for transform props to state
* combine initial state
* use getDerivedStateFromProps to fix failures
* only store a subset of lastProps to compare
* remove timeline events check from getDerivedStateFromProps
* reset state when timeline events changed
* get rid of mutation in strict mode
* remove existing state spread from state updates
Co-authored-by: Mahatthana (Kelvin) Nomsawadi <me@bboykelvin.dev>
* remove deriveStateFromProps from constructor
* use sdkProviderProps to pass provider props
* replace UNSAFE_componentWillReceiveProps with getDerivedStateFromProps
* add unsafe lifecycle check for DashboardGrid
* lastProps is already part of DashboardGrid state
* optional chain state.initialCardSizes
* remove state comparison logic
* fix misleading hasVisibleDashcardsChanged condition
* fix editor issue
* remove unused dashcardData from lastProps
---------
Co-authored-by: Mahatthana (Kelvin) Nomsawadi <me@bboykelvin.dev>
* Add generative test
* Use per test generator with adjustable seed
* Add workflow file
* Temporarily add trigger
* Make rand function work with default generator
* Update log4j2 test config
* Info log level
* Whitespace
* Add iteration count and env to make run on-demand
* cljfmt
* cljfmt
* remove unsafe lifecycles from visualization component
* add e2e tests
* remove extra comma in test helpers
* only run tests under strict mode when a flag is passed
* extract console assertion to a test helper
* update tests for known console errors
* swap new and previous props
* combine old state for transform props to state
* combine initial state
* use getDerivedStateFromProps to fix failures
* only store a subset of lastProps to compare
* remove timeline events check from getDerivedStateFromProps
* reset state when timeline events changed
* get rid of mutation in strict mode
* remove existing state spread from state updates
Co-authored-by: Mahatthana (Kelvin) Nomsawadi <me@bboykelvin.dev>
* remove deriveStateFromProps from constructor
* use sdkProviderProps to pass provider props
---------
Co-authored-by: Mahatthana (Kelvin) Nomsawadi <me@bboykelvin.dev>
When replacing a join, we were taking a valid join (with a correctly set up join alias), dropping the alias, creating
a new alias, and then inferring which fields belonged to the new alias. This caused issues when joining models using
a field that could plausibly belong to either model, because we inferred that the lhs join condition belonged to the
rhs model. This in turn confused the join naming code and caused us to refer to the lhs as "previous results" instead
of the correct table name.
With the new code, if we want to rename the new join, we explicitly replace the old alias with the new alias instead
of just removing the new alias and inferring what goes with what. This avoids potentially incorrect inferences and
fixes this issue (and possibly others?).
Last year we had a P1 where a high-visibility page in Brazilian Portuguese was broken. Let's add some basic e2e tests that check high-visibility admin/webapp pages (accessible in one click from the homepage) in our most popular locales.
* update ui to show trash icon, tooltip, and undo message if the dashcard being removed is the last dashcard for a dashboard question
* add e2e test, fix types
* Add pivot-drill-details
Similar to filter-drill-details and combine-columm-drill-details.
pivot-drill-details returns the info needed by FE to render the pivot drill popover UI.
* Merge pivot-types into pivot-drill-details
* Add e2e test
* Revert unintentional whitespace change
* Re-enable Breakout by e2e test
* Re-enable multi-stage pivot drill tests
* Add pivot drill test for query with explict join
* Add hostname support to BigQuery
* Fix formatting
---------
Co-authored-by: Case Nelson <snoe@snoe.ca>
Co-authored-by: Case Nelson <case@metabase.com>
Fixes#52843.
* Redshift: Adding an explicit dep on `aws-java-sdk-core`
This is listed as an optional dep, but it seems that at least one class
from it (`com.amazonaws.util.StringUtils`) is now dynamically required
by our driver - tests were throwing `NoClassDefFoundError`s without it.
Redshift tests are passing locally with this dependency added.
* fix(entity-id): show 404 for entity id that starts with digits
* simplify logic by removing support for sequential ids with slugs for the
tab
* clean up the tests and rename things
Year of era aggregations weren't working because our legacy malli schema didnt recognize it as a valid temporal unit.
Adding it to the schema makes it work.
* partition indexes and use separate queries
* test no exception thrown
* improve helper func name
* improve test
* remove test debugging
* address pr comments
This should ensure tasks will find classes in our plugin system. The
issue arose when `submit-task!` was called to sync an unhidden bigquery
table before any query was run on the main thread before hand.
This ensures that if either of these tasks runs before classloader has
been set, that their threads will inherit it.
`task.cache/pool` was found as an audit of other threads that may
exhibit this behavior.
Testing this from source is not possible right away since the
classloaders behave so differently between jars and sources.
* ci: find css variables that are used but never defined
* only search in frontend and enterprise/frontend
* Update bin/find-never-defined-css-variables/find-never-defined-css-variables.ts
Co-authored-by: Mahatthana (Kelvin) Nomsawadi <me@bboykelvin.dev>
* test: checks that it doesn't confuse definitions with usages
---------
Co-authored-by: Mahatthana (Kelvin) Nomsawadi <me@bboykelvin.dev>
These date from the beginnings of the `metabase.lib` codebase,
and are unused outside of a couple of tests.
I deleted the tests for these functions specifically, and updated a few
other tests that used these functions to retrieve columns on a query.
Prework for #49182.
* Start refactoring Field to TS
* Revert "Start refactoring Field to TS"
This reverts commit 88af47fdac.
* Add description input
* Use translation
* Make placeholder match data model page UI
* Add repro for #37907
* Fixing xlsx pivot table exports to respect table column names
* Adding fallback to column title to use display_name if custom title is empty; fixing kondo error
* Fix cljfmt spacing
* Adding unit test for xlsx pivot exports respecting custom column name
* Improving logic to be more idiomatic
* OpenAPI dox generation part 2
* Fix docstring
* metabase.billing.billing => metabase.billing.api
* Fix dox generation code fr fr
* Revert changes to dox for now
* Remove define-routes
* Fix missing routes
* Fix even more missing routes
* Third time's a charm
* Remove extra newlines at end of file
Databricks has static test data, by creating views we run the risk of
concurrent test runs seeing the temporary views or those views not
getting cleaned up if a job gets killed.
This is part of ENG-14545
* Add upsell component for better support
* Implement upsell for better support on a troubleshooting page
* Snow better support upsell only for OSS
* Update import path
* Remove whitespace
* Account for smaller screens
* fix(entity-id): don't make call to the card endpoint if the entity id
starts with digits
* prevent `isXXXPath` to return true if the path is the one of entity ids
* Remove flaky tags in e2e/test/scenarios/native/native.cy.spec.js
* Remove flaky tags in e2e/test/scenarios/native/native_subquery.cy.spec.js
* Remove flaky tags in e2e/test/scenarios/native/native-reproductions.cy.spec.js
* Remove flaky tag from admin test that uses native editor
* Remove flaky tag from visualization test that uses native editor
* Remove flaky tag from dashboard test that uses native editor
* Add helper to clear browser cache
* Reduce flakiness in autocomplete test
* Remove flaky tag in e2e/test/scenarios/native-filters/native-filters-reproductions.cy.spec.js
* Fix SDK cross-version testing failure after test util refactor (#52900)
* Revert the code before the fix
* Revert "Revert the code before the fix"
This reverts commit be11146bf1.
* Another attempt by checking out the whole e2e dir
* Can't checkout twice it will override everything 😭
* Reset everything back to the start
* Add missing files
* Keep only the latest webpack config
* Clarify the temporary bandaid with a comment
* Ensure when we cut 54, we won't ran into SDK cross-version testing
Future work on the refs overhaul #49182 changes how queries are
represented. To enable testing that new behaviour at a per-test level,
we can't let global queries like `lib.tu/venues-query` be defined once
and reused forever: in that case, tests of the new refs would see global
queries defined under old refs, and vice versa.
This change transforms the global queries, cards and other helpers in
`lib.test-util` from static `def`s to functions, so they build the
queries etc. in the context of the current test.
(If this introduces a performance hit for the lib tests, we can update
the functions to cache the two versions up front and choose between them
at runtime. So far it does not seem to have made a significant impact.)
* Use CodeMirror 6 (#50484)
* Add @uiw/react-codemirror
* Add language plugins for codemirror
* Add @codemirror/autocomplete
* Add basic codemirror editor
* Add selection handler to CodeMirror editor
* Set height
* Add basic styling to codemirror editor
* Add custom selection background
* Add simple theme with font settings
* Move extensions to a separate file
* Implement right click on selection
* Add language support
* Add @codemirror/language
* Add @lezer/highlight
* Add custom syntax colors
* Hightlight tags
* Add highlight for matching bracket
* Add completions
* Add snippet completions
* Make the editor full height by default
* Remove duplicate dependencies
* Match autocomplete colors with ace editor
* Render completion items in subgrid
* Tweak completion margin
* Use nesting
* Add hover state on completions
* Make getTagAtCursor more sueful
* Move matchTagAtCursor to utils
* Avoid work when there's no need
* Implement onCursorMoveOnCardTag
* Move autocomplete into language helper
* Disable language completions when inside a tag
* Allow open ended tags to return without suffix
* Add testid to codemirror editor
* Update test helpers
* Fix native subquery tests
* Fix native reproductions
* Fix snippet tests
* Remove ensureAceEditorClosed helper
* Add nativeEditorType helper
* Fix action tests
* Do not use native editor helper for custom expression test
* Add more native editor helpers
* Fix model metadata tests
* Use better type helpers
* Add more TODO comments
* Fix sharing test
* Simplify e2e helper
* Use native editor type helper
* Fix clearNativeEditor helper
* Fix missing namespace
* Update Loki Snapshots
* Remove unused lang extension
* Be explicit about h2 engine
* Nest language plugin return
* Export useLazyGetCardQuery
* Add completions for referenced card columns
* Split up autocompleters
* Simplify completion info
* Use cached cards
* Move useMemoized to util
* Add TODO comment about engine type
* Fix subquery test
* Use helpers in native reproduction
* Fix e2e test meta helper
* Do not assume we've seen all suggestions when narrowing down
* Fix selectAll
* Unskip unbroken test
* Fix broken reproduction
* Increase the delay for autocompletions to kick in
* Fix nativeEditorSelectAll on non-Mac systems
* Rename completer
* Move completers to separate file and make them handle their own dependencies
* Add autocomplete endpoints to metabase/api
* Use autocompletion hooks from metabase/api
* Reuse AutocompleteMatchStyle type
* Simplify helper
* Whitespace formatting
* Test sub-limit completion
* Fix reproduction
* Add tests for matchTagAtCursor
* Remove commented out code
* Typos
* Clarify why we disable rtk-query's cache
* Simplify cx call
* Use Cypress.platform over navigator.platform
* Tweak fold caret position
* Pass font to cm-content too
* Add synteax highlighting for the name token
* Rename AutocompleteSuggestionsRequest to AutocompleteRequest
* Fix broken overflow on notebook preview
* Add more fonts to monospaceFontFamily
* Use the same line height as the ACE editor
* Autocomplete much faster
* Handle overflow in completion meta
* Use shared object
* Add padding to editor scroll pane
* Add more fonts to monospaceFontFamily
* Set line-height to normal
* Fold on parens, braces and brackets
---------
Co-authored-by: Metabase Automation <github-automation@metabase.com>
* Namespace native editor e2e helpers (#51093)
* Add namespace for helpers
* Replace H.nativeEditor with namespaced version
* Replace H.focusNativeEditor with namespaced H.NativeEditor.focus()
* Replace H.nativeEditor completions helpers with namespaced versions
* Replace H.nativeEditorSelectAll with namespaced version
* Replace H.nativeEditorClear with namespaced version
* Replace H.nativeEditorType with namespaced version
* Use NativeEditor.type where possible
* Fix imports
* Replace H.focusNativeEditor with namespaced H.NativeEditor.focus()
* Use NativeEditor.type where possible
* Use NativeEditor.clear where possible
* Do not call NativeEditor
* Add missing backspace in helpers
* Fix case of selectall
* Remove use of focus().as(...)
* Fix NativeEditor.get call
* Fix e2e test
* Remove focus where it is wrong
* Fix text replacement test
* Add small delay
* Remove ACE version of the Native editor (#51437)
* Remove sql_behaviour
* Remove getEngineNativeAceMode
* Remove CARD_TAG_REGEX
* Do not re-export ACE_EDITOR_ID
* Remove AceEditor
* Remove ACE_EDITOR_ID from NativeEdiror mock
* Remove AceEditor reference in test
* Remove intermediate editor
* Update Loki Snapshots
* Fix formatting
* Fix broken test
* Various fixes for the native editor revamp (#52321)
* Deduplicate schema completions
* Align description to the right
* Add keyword description to comletions
* Render completion description in normal font
* Set indentation to use tabs
* Remove boost from completions
* Stop cmd+enter from entering a newline
* Use same indentation as previous line when pressing Enter
* Remove empty resize handler in CodeMirrorEditor
* Add defaults to NativeEditor.type
* Make config parameter optional
* Add custom keymaps for next and previous completions
* Add custom interpolations for previous and next completion
* Add helpers up and down arrows
* Add reproduction for #48712
* Add tests for suggestion issues
* Allow completing from inside a word
* Add unit test for useSchemaCompletions
* Add unit test for useSnippetCompletions
* Add unit test for useCardTagCompletion
* Add unit test for useReferencedCardCompletion
* Accept completion with tab
* Do not indent with tab, but just insert a tab
* Use Lib.Query in CodeMirrorEditor
* Simplif CodeMirror editor prop and ref names
* Add simple unit test for CodeMirrorEditor
* Use isNotNull helper
* Use Lib.Query in NotebookNativePreview
* Handle nil in engine
* Handle empty engine
* Use correct meta kay in test
* Reduce flakiness in dashboard test
* Reduce flakiness in native test
* Add completion for query locals
* Make database optional
* Add test for locals
* Add test for command enter
* Add tests for indenting with tab
* Handle non-darwin platform in test
* Allow typing {tab}
* Add test for indentation
* Handle quoted locals
* Add typing to source
* Improve syntax highlighting for mongo
* WIP: add custom keywords completion for mongo
* Make collection_id optional
* Remove test return
* Handle SQLDialct shape
* Handle keyword completion all in the completer
* Lift completers out of language plugin
* Move language helpers to separate file
* Simplify keywords extraction
* Add tests for keyword completion
* Add more mongo keywords
* Fix quoted local test
* Add suggestion tests for mongo
* Fix e2e test with NativeEditor
* Update Loki Snapshots
* Fix e2e test with NativeEditor
* Revert "Update Loki Snapshots"
This reverts commit 7d1ea82579.
* Revert "Revert "Update Loki Snapshots""
This reverts commit fd7b552770.
* update screenshot
* Fix autocomplete API calls in the new native query editor (#52688)
* Enable autoFocus
---------
Co-authored-by: Metabase Automation <github-automation@metabase.com>
Co-authored-by: Uladzimir Havenchyk <uladzimir.dev@gmail.com>
Co-authored-by: Alexander Polyankin <alexander.polyankin@metabase.com>
* Sync state in useStringFilter
* Sync state in useBooleanOperatorFilter
* Sync state in useBooleanOptionFilter
* Add FilterModalResult type
* Fix setTab type
* Make comments work as JSDocs
* Move FilterModalResult
* Add filter modal context
* Drop FilterModalHeader and use FilterSearchInput directly
* Start to use context in BooleanFilterEditor
* Start to use context in TimeFilterEditor
* Start to use context in DateFilterEditor
* Start to use context in CoordinateFilterEditor
* Start to use context in NumberFilterEditor
* Start to use context in StringFilterEditor
* Start to use context in DefaultFilterEditor
* Stop drilling isSearching
* Fix tests
* Stop drilling version prop
* Remove FilterModalBody and FilterModalFooter exports
* Make naming consistent
* Remove props drilling
* Remove props drilling
* Remove props drilling
* Inline FilterTabContent into FilterModalBody
* Revert "Inline FilterTabContent into FilterModalBody"
This reverts commit cfd5738e52.
* Remove props drilling
* Sort attributes
* Consolidate imports
* Fix a mistake
* Improve JSDoc
* Revert "Sync state in useBooleanOptionFilter"
This reverts commit 33a979dfdf.
* Update approach after merging refactoring branch
* Add a repro for string filters
* Go back and forth on search input
* Assert operator
* Move effect to concrete hook
* Add test for boolean filters
* Fix time filters
* Revert "Move effect to concrete hook"
This reverts commit 82b94f4345.
* Move effect outside use-time-filter
* Improve naming
* Remove TODO
* Extract setup48319
* Rename setup48319 to setupCustomColumns
* Rename handle* to on*
* Fix coordinate filters
* Fix default filters
* Fix number filters
* Revert 1st approach
* Use key prop
* Remove detailed tests and leave only one
* Use current value
* Add a test for "version" hack
* Use only version for the key
* Rename version to remountKey and add explanatory comments
* Rename forceDescendantsRemount to forceRemountDescendants
* Rename
* fix ultra wide tooltips (closes#52199)
* all tooltips wrap now, and better test
* sturdier test
* use setup() in test case
* removed unncessary code path
* use features instead of hard coding drivers in tests
* fix formatting
* fix details-name-is-db feature
* bad conn details helper instead of features
* address pr comments
* exclude vertica from materialized view tests
* fix whitespace
Update YAMLs for 53 release.
Move all questions into usage analytics dashboards. In some cases, questions had to be duplicated in order to be moved to dashboards.
* update copy and avoid showing two messages about moving when going between collections and dashboards in different collections
* pr feedback
* impl pr feedback
* Add dashboard empty states
* Separate empty states from the actual dashboard content
* Fix public dashboards view
* Add translation context
* Localize the `alt` tag
* Make sure the content doesn't get cut off
In sql server, language settings change what day weeks start on. The sql server driver assumes that weeks always
start on sunday (ie the US default), which means that if you change the db to french, our math is off and we start
calculating weeks and days of the week wrong. This adds back in an offset based on DATEFIRST to fix that.
* POC of entity id aware route
* wip e2e tests
* clean up logic
* changing approach as everything was broken
* revert unwanted changes
* Revert "revert unwanted changes"
This reverts commit f6ce10ac7b.
* reset routes to master
* /by-entity-id/ routes
* update tests to check for subpaths and search param preservation
* cleaning up un-needed code
* simplified even more the logic by merging the two config maps to one
array
* pass down router instead of single props
* split the tests to their files
* fix typo in test
* adds a test for a unlikely to be needed scenario
* fix import after merging master
* fix e2e copy
* /by-entity-id/ -> /entity/ as per product decision
* tag the test as flaky as it fails depending on the order of test files
being run
* Auto-rewrite defendpoint forms in dataset.clj
* Auto-rewrite defendpoint forms in dataset.clj
* Auto-rewrite defendpoint forms in dataset.clj
* Test fixes 🔧
* More test fixes 🔧
* A few more test fixes.
* Test fixes 🔧
* Test fixes 🔧
* Finally handle form-encoded bodies correctly
- Cleaned up session management
- Don't send "new device" email from embedded logins
---------
Co-authored-by: John Swanson <john.swanson@metabase.com>
* Avoid cartesian products in pivot csv exports
* Get row and column sorting working properly
* Add a bunch of unit tests
* Test fixes, various tweaks to better match the FE output
* Fix another test
* address feedback
* Add FilterModalResult type
* Fix setTab type
* Make comments work as JSDocs
* Move FilterModalResult
* Add filter modal context
* Drop FilterModalHeader and use FilterSearchInput directly
* Start to use context in BooleanFilterEditor
* Start to use context in TimeFilterEditor
* Start to use context in DateFilterEditor
* Start to use context in CoordinateFilterEditor
* Start to use context in NumberFilterEditor
* Start to use context in StringFilterEditor
* Start to use context in DefaultFilterEditor
* Stop drilling isSearching
* Fix tests
* Stop drilling version prop
* Remove FilterModalBody and FilterModalFooter exports
* Make naming consistent
* Remove props drilling
* Remove props drilling
* Remove props drilling
* Inline FilterTabContent into FilterModalBody
* Revert "Inline FilterTabContent into FilterModalBody"
This reverts commit cfd5738e52.
* Remove props drilling
* Sort attributes
* Consolidate imports
* Fix a mistake
* Improve JSDoc
* Rename handle* to on*
* Auto-rewrite defendpoint forms in session.clj
* Auto-rewrite defendpoint forms in session.clj
* Remove &request anaphor and make it a normal arg
* Add test
* Defendpoint 2.1
* base-handler => handler
* Fix Kondo errors
* Make sure async exception handling is done correctly
* Remove defendpoint-async
* Rewrite all the rest of the remaining defendpoints except for those in public
* Remove card endpoints from this PR
* Update src/metabase/setup/api.clj
Co-authored-by: bryan <bryan.maass@gmail.com>
* cljfmt
* Test fixes 🔧
---------
Co-authored-by: bryan <bryan.maass@gmail.com>
* Drop Java 11/17 Support
* Update Dockerfile
Change the Java version of the builder as well
* mention that we are dropping support for earlier Java in docs
---------
Co-authored-by: Luis Paolini <paoliniluis@gmail.com>
Co-authored-by: Alexander Solovyov <alexander@solovyov.net>
* throw down ui
* pixel freaking perfect
* wire in real endpoint for fetching dashboard question candidates
* wire in move endpoint
* loading and error handling for mutation
* persist info step acknowledgement
* optionally show item in menu, fix flickering of info modal is acked previously, add a toast on successful bulk move
* fix some of the failing existing tests
* add unit tests for collection menu
* add unit tests for main migration modal
* add e2e test
* fix failing unit test
* fix another failing unit test
* fix candidates result being stale if dashcards added/removed from dashboards creates/removes candidates
* adjust url of feature
* copy feedback
* fix type-check
* scope feature to only admins
* use pagination to fetch count but not items when detecting if to show the migration tool menu item
* test + style improvements, impl pr feedback
* fix unit tests, add unit coverage for is admin
* Add lib.native/has-template-tag-variables?
* Disable drills for native queries with template tag variables
Closes#49051
* Re-export lib.native/has-template-tag-variables? from lib.core
* Add tests for disabled drills for native queries with variables
* Review feedback
Closes: https://github.com/metabase/metabase/issues/52116
Note that of icu4j was already a dependency, via graalvm.
We have just made it explicit now, ~~and updated to the latest version.~~ ([upgrading version breaks rendering](https://github.com/metabase/metabase/actions/runs/12926546646/job/36049844305?pr=52581))
```
org.graalvm.js/js 22.3.5
. org.graalvm.regex/regex 22.3.5
. org.graalvm.truffle/truffle-api 22.3.5
. com.ibm.icu/icu4j 71.1
. org.graalvm.truffle/truffle-api 22.3.5
. org.graalvm.sdk/graal-sdk 22.3.5
. org.graalvm.sdk/graal-sdk 22.3.5
. com.ibm.icu/icu4j 71.1
```
* Auto-rewrite defendpoint forms in src/metabase/api/api_key.clj
* Auto-rewrite defendpoint forms in src/metabase/api/api_key.clj
* Fix formatting
* Auto-rewrite defendpoint forms in api_key.clj
* Auto-rewrite defendpoint forms in api_key.clj
* updated some flaky screenshots
* simplified ci job
* updated types
* registered visualizations and added a way to register multiple visualizations in storybook
* replaced @storybook/testing-library with @storybook/test
* Paginate the DQ candidates endpoints
* Paginate in Clojure
I think this should be fine - the number of cards we're working with
should be relatively small, since this is working one collection at a time.
* CR response
* Auto-rewrite defendpoint forms in database.clj
* Some cleanup
* Update src/metabase/api/database.clj
Co-authored-by: bryan <bryan.maass@gmail.com>
* cljfmt
* Auto-rewrite defendpoint forms in database.clj
* Test fix?
---------
Co-authored-by: bryan <bryan.maass@gmail.com>
* Exclude DQs from DQ candidate endpoint
... oops
I also moved the `(= (:collection_id card) (:collection_id dashboard))`
check out of `card/sole-dashboard-id`, since it's not really part of
being a sole dashboard.
* Add `dashboard_id IS NULL` to the query
... instead of filtering them out in clojure.
This was already legal-ish, as [:* 1 [:sum ...]] was a valid argument to coalesce. However, you couldn't pass in the
sum directly. Under the hood, NumericExpressionArg allowed aggregations, but ExpressionArg didn't. Presumably, the
one is supposed to be a superset of the other, and this change made that actually be true.
* filter out collections and dashboards where can_write is false from recents tab in collection picker
* move modal filter out locations you can not save to from recents tab
* add e2e test
* pr feedback - make cleaner and add comment
* Auto-rewrite defendpoint forms in src/metabase/api/automagic_dashboards.clj
* Manual cleanup
* Auto-rewrite defendpoint forms in bookmark.clj
* Manual cleanup
* Auto-rewrite defendpoint forms in automagic_dashboards.clj
* Fix endpoints with no route params but no regexes
* Auto-rewrite defendpoint forms in automagic_dashboards.clj
* Auto-rewrite defendpoint forms in automagic_dashboards.clj
* Auto-rewrite defendpoint forms in automagic_dashboards.clj
* Don't include changes to dev namespace
* Revert changes to bookmark
* Some manual cleanup
* Show should be optional
* Auto-rewrite defendpoint forms in user_key_value.clj
* Auto-rewrite defendpoint forms in user_key_value.clj
* Auto-rewrite defendpoint forms in user_key_value.clj
* Cleanup + fixes 🔧
* Auto-rewrite defendpoint forms in cache.clj
* Auto-rewrite defendpoint forms in cache.clj
* Auto-rewrite defendpoint forms in cache.clj
* Auto-rewrite defendpoint forms in cache.clj
* Fix schema for PUT /api/cache
* Fix cache schema (part 2)
* Do not render "Make public" toggle in Create Action Modal
* Disable the "make public" toggle and add a tooltip explaining why it's disabled
- Use Mantine's Switch
- Use Mantine's Tooltip
* Add a repro for #51282
* Update test name
* Fix tests
* Update unit tests
* Add unit test repro for #51282
* Remove e2e repro for #51282
* Format code
* Format code
* fix unknown attribute error
* fix another error message about unknow HTML attribute
* fix invisible ellipsis when title is empty string
* added e2e test
* Update labels
* Revert "Update labels"
This reverts commit be632aec2e.
* Simplify the fix
* Update unit tests
* Update test
* Revert "Simplify the fix"
This reverts commit a9c267c4c0.
* Update labels
* Update test
* Make it work in native editor
* Revert "Update unit tests"
This reverts commit 3614a8ff2f.
* Update test
* Update test
* Update test
* Update test
* Using 1 as max value in minibar chart rendering logic if the column is pure percent
* Adding MiniBar unit spec
* Prettier --write
* fixing eslint errors
* polish specs
---------
Co-authored-by: Aleksandr Lesnenko <alxnddr@gmail.com>
* Defendpoint 2 misc fixes
* More fixes 🔧
* Remove some requires for faster loading; always make all query params optional
* Fix comment
* Make :boolean default false as well
* Auto-rewrite defendpoint forms in activity.clj
* Auto-rewrite defendpoint forms in activity.clj
* Formatting
* Auto-rewrite defendpoint forms in activity.clj
* Auto-rewrite defendpoint forms in activity.clj
* Auto rewrite defendpoint ones
* My bird is typing
* Make sure response format matches old defendpoint format
* Cleanup
* Clean up alert tests
* Test fixes 🔧
* One last test fix 🔧
* Make actions execute body optional
As a follow-up to #52284, this PR adds lint rules banning the pattern that caused a P1 (#48144). These rules require us to use the browser default with `String.localeCompare` and related functions in the `Intl` library. Since we cannot pass these functions our own locale codes, since some of these are in the wrong format.
This is designed to be a good-enough, "80 percent of the value with 20 percent of the effort" solution, even though better solutions could be pursued such as aligning FE and BE around an international standard (BCP 47).
- [x] Tests have been added/updated to cover changes in this PR
This is a strange one. But in practice, this was still doing a seq scan
on postgres.
```sql
clean=# explain DELETE FROM query_execution WHERE id IN (SELECT id FROM query_execution WHERE started_at <= '2025-01-02 01:10:15.651839+00' LIMIT 50000);
QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------
Delete on query_execution (cost=526084.17..1066675.96 rows=0 width=0)
-> Nested Loop (cost=526084.17..1066675.96 rows=50000 width=34)
-> Unique (cost=526076.73..526326.73 rows=50000 width=32)
-> Sort (cost=526076.73..526201.73 rows=50000 width=32)
Sort Key: "ANY_subquery".id
-> Subquery Scan on "ANY_subquery" (cost=0.00..522174.32 rows=50000 width=32)
-> Limit (cost=0.00..521674.32 rows=50000 width=4)
table scan >>>>> -> Seq Scan on query_execution query_execution_1 (cost=0.00..1704153.49 rows=163335 width=4)
Filter: (started_at <= '2025-01-01 19:10:15.651839-06'::timestamp with time zone)
-> Bitmap Heap Scan on query_execution (cost=7.44..11.46 rows=1 width=10)
Recheck Cond: (id = "ANY_subquery".id)
-> Bitmap Index Scan on query_execution_pkey (cost=0.00..7.44 rows=1 width=0)
Index Cond: (id = "ANY_subquery".id)
(13 rows)
```
The odd thing is that it is sensitive to what timestamp you use. Here's
the same query but with a timestamp in 2024:
```sql
clean=# explain DELETE FROM query_execution WHERE id IN (SELECT id FROM query_execution WHERE started_at <= '2024-12-19 01:10:15.651839+00' LIMIT 50000);
QUERY PLAN
-------------------------------------------------------------------------------------------------------------------------------------
Delete on query_execution (cost=225736.06..641819.90 rows=0 width=0)
-> Nested Loop (cost=225736.06..641819.90 rows=28490 width=34)
-> HashAggregate (cost=225724.95..226009.85 rows=28490 width=32)
Group Key: "ANY_subquery".id
-> Subquery Scan on "ANY_subquery" (cost=122917.34..225653.72 rows=28490 width=32)
-> Limit (cost=122917.34..225368.82 rows=28490 width=4)
-> Bitmap Heap Scan on query_execution query_execution_1 (cost=122917.34..225368.82 rows=28490 width=4)
Recheck Cond: (started_at <= '2024-12-18 19:10:15.651839-06'::timestamp with time zone)
-> Bitmap Index Scan on idx_query_execution_started_at (cost=0.00..122910.22 rows=28490 width=0)
Index Cond: (started_at <= '2024-12-18 19:10:15.651839-06'::timestamp with time zone)
-> Bitmap Heap Scan on query_execution (cost=11.11..15.12 rows=1 width=10)
Recheck Cond: (id = "ANY_subquery".id)
-> Bitmap Index Scan on query_execution_pkey (cost=0.00..11.11 rows=1 width=0)
Index Cond: (id = "ANY_subquery".id)
(14 rows)
```
And this is because the db engine knows that most of my data is around
2025 and will just hit the whole table rather than hit an index and then
go return most of the table.
The logic we want here is basically, "work up the index on created_at
until we get to a notion of recent".
If we add an order by, we're close to that and things go much faster
again
```sql
clean=# explain DELETE FROM query_execution WHERE id IN (SELECT id FROM query_execution WHERE started_at <= '2025-01-02 01:10:15.651839+00' order by id asc LIMIT 50000);
QUERY PLAN
-------------------------------------------------------------------------------------------------------------------------------------------------------------
Delete on query_execution (cost=851615.84..1392207.62 rows=0 width=0)
-> Nested Loop (cost=851615.84..1392207.62 rows=50000 width=34)
-> Unique (cost=851608.39..851858.39 rows=50000 width=32)
-> Sort (cost=851608.39..851733.39 rows=50000 width=32)
Sort Key: "ANY_subquery".id
-> Subquery Scan on "ANY_subquery" (cost=0.55..847705.98 rows=50000 width=32)
-> Limit (cost=0.55..847205.98 rows=50000 width=4)
-> Index Scan using query_execution_pkey on query_execution query_execution_1 (cost=0.55..2767566.54 rows=163335 width=4)
Filter: (started_at <= '2025-01-01 19:10:15.651839-06'::timestamp with time zone)
-> Bitmap Heap Scan on query_execution (cost=7.44..11.46 rows=1 width=10)
Recheck Cond: (id = "ANY_subquery".id)
-> Bitmap Index Scan on query_execution_pkey (cost=0.00..7.44 rows=1 width=0)
Index Cond: (id = "ANY_subquery".id)
(13 rows)
```
It caused problems as lint-staged stashes files and if developer is not familiar with the mechanism, he thinks that his work is lost. To avoid confusion we remove this git hook
* skips model popup
* fix types
* port to ts
* reorganize acknowledgement code
* improve ack api, add unit test, fix missing key prop console warning for question action menu
* move schema file to correct location, improve e2e
* Make sure we can load User KV schemas when in a JAR (#52246)
* Make sure we can load user KV schemas in a JAR
* A few minor tweaks
- removed some unused code
- added some more comments to clarify when each function is used
- move `update-user-key-value-schema!` outside the `doseq`
* Fix tests
* A bit more clarity
* actually fix tests
* move prod implementation below dev implementation; couple other tweaks
* fix fn name
---------
Co-authored-by: John Swanson <john.swanson@metabase.com>
* remove only usage in test
* lint and unit fixes
* fix unit test for real this time
* but actually... unit tests are fixed this time
---------
Co-authored-by: Noah Moss <32746338+noahmoss@users.noreply.github.com>
Co-authored-by: John Swanson <john.swanson@metabase.com>
When we replace a breakout, we also update any related order-by. However, we weren't updating all fields. This
meant that the order-by and breakout weren't identical. This was fine-ish on postgres, because they compiled to the
same thing. However, on google bigquery, it expects that group bys and order bys will refer to the AS alias instead
of just duplicating the code for the aliased thing. The differences between order-by and breakout meant that we
didn't realize that the one was supposed to match the other, and so we duplicated the code and bigquery barfed.
Syncing more fields fixes that.
**Joined columns**
- `join__${IDENT_OF_JOIN_CLAUSE}__${IDENT_OF_COLUMN}` for explicit
joins
- `implicit_via__${IDENT_OF_FK_COLUMN}__->__${IDENT_OF_TARGET_COLUMN}`
for implicit joins
These fulfill the contract for column `:ident`s:
- Unique identity within one stage of a query
- Globally unique
- Note that the same implicitly joined column (eg.
`Products via Orders.PRODUCT_ID`) might be referenced in
many queries, but it's the **same column** - same `display_name`,
types, etc.
**New test metadata**
In order to test these properly, I needed test metadata with more
permutations of FKs.
I've added three new tables to `metabase.lib.test-metadata`, which are a
simplified version of Github issues, users and comments. The tables are
`:gh/issues`, `:gh/users` and `:gh/comments`.
Some notable things:
- `:gh/users` has a `:type/Text` PK, ie. Github usernames
- `:gh/issues` and `:gh/comments` have `:type/UUID` PKs
- `:gh/issues` has **two** FKs to `:gh/users`: `:reporter-id` and
`:assignee-id`.
- `:gh/comments` has a **self FK**: `:reply-to`
- `:gh/issues :closed-at` is a `NULL`able `:type/DateTime` column, which
we lacked previously
* phase #1: remove all uses of openNativeEditor without a parameter
* phase #2: remove all uses with parameter
* remove useless test
* fix failing test cases
* fix failing tests
* add db id and native filter interaction
* fix flaky tests
* Support multiple operators in expectOp
* Refactor
* Show errors from parser
* Revert recursive-parser changes
* Pass human-friendly operator name to expectOp
* Fix typo
* Require comma if next token is not closing parenthesis
* Add a repro for #15892
* Update existing test
This drill thru was recently enabled for multi-stage queries in #51950, but there is a bug which causes the
QueryColumnPicker to fail if a join is present. Disable the drill thru for now while we work on a fix.
Closes#52221
Related to #52236
* Upgrade postgres app db version
The more the marrier
* Update docs/installation-and-operation/configuring-application-database.md
Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>
---------
Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>
* Add repro for #50925
* Add a test
* Fix the case with whitespace following an incomplete token
* Add test case with bracket identifier following an incomplete token
* Add comments
* Thanks vscode
* Fix the case with bracket identifier following an incomplete token
* Simplify unit tests
* Fix the logic
* Update comment
* Fix it
* Refactor
* Throw different error when using a known name that is not a metric
* Drop checking for segments as they cannot be used in aggregation expressions anyway
* Add tests
* Remove redundant withName: true
* Wrap with describe
* Make naming consistent
* Improve copy
If a card with a default value doesn't actually require a value, letting people clear the default value can be useful.
However, if the card does require a value, letting people clear the default just throws an error. This is a
completely useless result, so we should use the default value in that case. See [here](https://metaboat.slack.com/archives/C0645JP1W81/p1719233020129649) for more info.
* Add a warning/disclaimer for H2 uploads
* Expand the component test
* Use the built in margin bottom
* Show the additional warning on hosted instances only
* Remove `mb/once` tag since we don't use it anymore
* Add `^:mb/old-migrations-tests` tag.
* Update GH Actions config
* Cleanup
* Keep the migrations test files in order
* Quote excluded tags
* Update running.md
Some little details about running Java apps on prod
* Apply suggestions from code review
Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>
---------
Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>
Co-authored-by: Alex Yarosh <alexandra@metabase.com>
* Update mysql.md
Add a note about vitess and planetscale
* Update docs/databases/connections/mysql.md
Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>
---------
Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>
* [Backend modularization] Remove `metabase.models`
* I guess we don't really need an ordered map
* Remove unneeded require of models
* Small fixes 🔧
* Test fix 🔧
* Fix metabase.api.embed.common
* Fix some insane code that was using symbols for raw table names
* We need to do model resolution for table names
* Don't test EE namespaces in OSS
* Secret handling in details
Secret handling in details was very confusing. This PR consolidates most
of the logic into models.secret around toucan2 lifecycles of
model/Database.
Some of the problems fixed here:
- path and value confusion. It was possible to swap between uploaded
files and file-paths and end up with the uploaded file being treated
as a file-path and leaking to users in logs, APIs, and UIs.
- Unclear management of secret properties in details. The
responsibility for taking in secrets in details, saving to a Secret,
and then ensuring those incoming secrets do not get saved in details
was spread out all over the place. This made it possible to save a
secret-value in details.
- Unclear hydration of secret values for connection testing. How a
driver was supposed to get a secret and which properties might be in
details was unclear and inconsistent. Often mangling strings and
files returned by Secret api was necessary.
- Encoding problems. Uploaded files were being mangled by taking bytes
into a string and back to bytes to save. Ensure that the bytes given
are the bytes saved.
* Fix merge
* Fix tests
* Fix db creation
* Fix db creation in other tests
* Manually add updated_at to database
* Fix tests
* Fix tests
* Fix after-select input
* Wrap in data-uri
* Fallback for -value if base64 encoding failed
* If not data-uri, then just turn string to utf8 bytes
* Poking at oracle failures
* Oracle wrap bytes in data-uri
* Don't clean secret properties from oracle
* Fix base64 typo and add test util for data-uri wrap
* Fix snowflake test
* Another snowflake test
* Update src/metabase/models/secret.clj
Co-authored-by: Noah Moss <32746338+noahmoss@users.noreply.github.com>
* Update src/metabase/models/secret.clj
Co-authored-by: Noah Moss <32746338+noahmoss@users.noreply.github.com>
* Update src/metabase/models/secret.clj
Co-authored-by: Noah Moss <32746338+noahmoss@users.noreply.github.com>
* Update test/metabase/test/util.clj
Co-authored-by: Noah Moss <32746338+noahmoss@users.noreply.github.com>
* Update src/metabase/util/jvm.clj
Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>
* Update src/metabase/util/jvm.clj
Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>
---------
Co-authored-by: Noah Moss <32746338+noahmoss@users.noreply.github.com>
Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>
* Update comment in automatic-insights drill to point to new github issue
* Move helpers from lib.drill-thru.common to lib.underlying
These funcs deal with "underlying" columns, so move them into lib.underlying.
aggregation-sourced?
breakout-sourced?
strictly-underlying-aggregation?
* Consistent naming for drill-thru multi-stage query tests
* Update multi-stage query drills tests to use new test helpers
* Homogenize apis for drill thru test helpers
Rename prepend-filter-to-stage to prepend-filter-to-test-expectation-stage and make the calling interface match
that of append-filter-stage-to-test-expectation.
* cljfmt
* Revise new table permissions logic to handle sandboxes
* tweak new perm logic
* make sure cond has a default
* fix comments
* tweak defenterprise conditions
* [Backend modularization] `query-analysis` cleanup
* Update Kondo config
* Add CODEOWNERS
* metabase.task.sweep-query-analysis => metabase.query-analysis.task.sweep-query-analysis
* Move test util
* Module linter should ignore my test-setup namespace
* Make sure query analysis init is loaded in tests that need it
* Deparallelize some of the tests
This change happened in peer
https://github.com/metabase/metabase/pull/51023 and is manifesting now
it seems.
The removal of the call to `(classloader/the-classloader)` is what is
hurting us.
Only clickhouse cloud instances can be set as a suitable upload spot so
i'm doing the following changes in the local jar repro to fix it up:
```clojure
(in-ns 'metabase.driver.clickhouse)
(defmethod driver/database-supports? [:clickhouse :uploads] [_driver _feature db]
true)
(in-ns 'metabase.public-settings)
(let [db_id 2, schema_name "default", table_prefix "upload_"]
(t2/update! :model/Database db_id {:uploads_enabled true
:uploads_schema_name schema_name
:uploads_table_prefix table_prefix}))
```
* [debug] add storybook helper
* Make error on InteractiveQuestion consistent
* Make error on StaticQuestion consistent
* [debug] static question stories
* [debug] InteractiveDashboard Stories
* Make error on InteractiveDashboard consistent
* [debug] StaticDashboard stories
* Make error on StaticDashboard consistent
* [debug] EditableDashboard stories
* Make error on EditableDashboard consistent
* Fix CreateQuestion not working after fixing entity ID states
* Add tests for InteractiveQuestion
* Add StaticQuestion tests
* Add InteractiveDashboard tests
* Add StaticDashboard tests
* Add EditableDashboard tests
* Make variable names consistent
* Extract resource not found error to a component
* Fix missing return
* Make storybook usable and reduce duplications
* Don't roll my own util
* Update error colors following the new design
* Update loki snapshots
* Fix unit test expectation to hex
* Make colors easier to understand with comments
* Update SDK error style
* Update error components copy
* Use Code from metabase/ui
* Remove change accidentally commited
* Fix InteractiveQuestion renders question not found briefly
metabase#51969
* WIP First attempt at passing start of week option to static viz bundle
* Cleaning debug lines
* set moment and dayjs start of week day setting in static viz
* reverting index.ts
* Adding backend test for checking first day of week setting is respected in static viz
---------
Co-authored-by: Aleksandr Lesnenko <alxnddr@gmail.com>
Co-authored-by: Aleksandr Lesnenko <alxnddr@users.noreply.github.com>
* Update mysql.md
Add an additional flag as a note, since I've hit this one today with a customer
* Update docs/databases/connections/mysql.md
Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>
---------
Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>
* Update activating-the-enterprise-edition.md
another little nitpick, since people tend to get creative with these things and then we end up debugging endless hours
* edit
---------
Co-authored-by: jeff bruemmer <jeff.bruemmer@gmail.com>
* initial impl
* better error handling, smooth scroll for adding existing cards, clean up of code in general
* add e2e test
* fix type error
* pr feedback
* reworks implementation to consume value from url then clear it, follows existing code patterns better
* clean up and test improvements
* fix e2e tests
* fix type issues
* pr feedback
* mostly working solution w/ tests
* add basic type impl, change interface to return obj from hook, update tests to account for both
* add types to setup mock endpoint fns
* hacks solution together to test if everything passes CI
* real implementation, removes hacked approach
* patch types
* add test coverage for patched package
* add one more unit test
* Update error colors following the new design
* Update loki snapshots
* Fix unit test expectation to hex
* Make colors easier to understand with comments
* Fix E2E tests
* Make setupModelPersistenceEndpoints work with multiple models
* Make test wait for data
* Improve tags invalidation
* Improve types
* Remove entity.ListLoader usage in UploadSettingsForm (#51909)
* Remove entity.ListLoader usage from ModelCacheRefreshJobs (#51910)
* Remove entity.ListLoader usage from SavedEntityList (#51914)
* Remove entity.ListLoader usage from SearchApp (#51915)
* Remove entity.ListLoader usage from QuestionList (#51912)
* Remove entity.ListLoader usage from SearchResults (#51913)
* Remove `entity.ListLoader` from `CollectionContentView` (#51916)
* Remove entity.ListLoader usage from CollectionContentView
* CI Failure notice improvements (#51885)
* don't notify slack on flaky job failing
also add more info to failure notice
* this is nicer
* add entity insertion library (#47895)
* add entity insertion library
- this lets us use `add-load/from-script`
* update add load examples
* Update year in the README (#51923)
* reset cache when starting cypress locally (#51921)
* Don't build FE for driver tests that don't need it; shave 2 minutes off of driver tests (#51884)
* Don't build FE for driver tests
* Try this again
* Consolidate MySQL job configs
* Oops fix conditional flags for test-driver
* More MySQL consolidation
* Oops
* Add a new job that waits for all other jobs (so we can require one single job going forward)
* Restore deleted code
* Try adding a Driver Tests (Summary) workflow
* How about a nice job name
* Don't love YAML inline array syntax
* Try fixing the build-static-viz configuration again
* Try fixing the build-static-viz configuration again
* Try this again
* Fix MySQL SSL config
* Try fixing the drivers summary workflow
* re-remove docs update block (#51928)
* upload title (#51931)
* Fix formatting of month-of-year when there are 53 weeks in the year (#51892)
* New deactivated_at field on users (#51765)
* New deactivated_at field on users
* fix migration syntax
* Setting alignment to top for icons in subscription and alert emails (#51886)
* Correct docstring for series (#51840)
Minor nitpicks on some endpoint docstrings
* Dont escape chars of header dashcards (#51880)
* fix rerun job
* feat(sdk): Interactive Question Chart Settings Dropdown #50976
---------
Co-authored-by: Ryan Laurie <30528226+iethree@users.noreply.github.com>
Co-authored-by: bryan <bryan.maass@gmail.com>
Co-authored-by: Nemanja Glumac <31325167+nemanjaglumac@users.noreply.github.com>
Co-authored-by: Cam Saul <1455846+camsaul@users.noreply.github.com>
Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>
Co-authored-by: Alexander Polyankin <alexander.polyankin@metabase.com>
Co-authored-by: Noah Moss <32746338+noahmoss@users.noreply.github.com>
Co-authored-by: Tyler Plude <tsplude@gmail.com>
Co-authored-by: Luis Paolini <paoliniluis@gmail.com>
Co-authored-by: Oisin Coveney <oisin@metabase.com>
* Remove `entity.ListLoader` from `CollectionItemsTable` - approach v3 (#51958)
* Break CollectionItemsTable down to avoid using Search.ListLoader
* ESLint
* Oops
* Fix auto-upload effect in ModelUploadModal
* Fix types
* Fix the effect running too often
* Remove redundant code
* Reduce diff
* Remove old EntityListLoader and rename new EntityListLoaderRtkQuery to EntityListLoader
* Remove PaginationState
* Remove EntityType
* Fix effect
* Fix effect
---------
Co-authored-by: Ryan Laurie <30528226+iethree@users.noreply.github.com>
Co-authored-by: bryan <bryan.maass@gmail.com>
Co-authored-by: Nemanja Glumac <31325167+nemanjaglumac@users.noreply.github.com>
Co-authored-by: Cam Saul <1455846+camsaul@users.noreply.github.com>
Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>
Co-authored-by: Alexander Polyankin <alexander.polyankin@metabase.com>
Co-authored-by: Noah Moss <32746338+noahmoss@users.noreply.github.com>
Co-authored-by: Tyler Plude <tsplude@gmail.com>
Co-authored-by: Luis Paolini <paoliniluis@gmail.com>
Co-authored-by: Oisin Coveney <oisin@metabase.com>
* Make setupModelPersistenceEndpoints work with multiple models
* Make test wait for data
* Improve tags invalidation
* Improve types
* Remove entity.ListLoader usage in UploadSettingsForm (#51909)
* Remove entity.ListLoader usage from ModelCacheRefreshJobs (#51910)
* Remove entity.ListLoader usage from SavedEntityList (#51914)
* Remove entity.ListLoader usage from SearchApp (#51915)
* Remove entity.ListLoader usage from QuestionList (#51912)
* Remove entity.ListLoader usage from SearchResults (#51913)
* Remove `entity.ListLoader` from `CollectionContentView` (#51916)
* Remove entity.ListLoader usage from CollectionContentView
* CI Failure notice improvements (#51885)
* don't notify slack on flaky job failing
also add more info to failure notice
* this is nicer
* add entity insertion library (#47895)
* add entity insertion library
- this lets us use `add-load/from-script`
* update add load examples
* Update year in the README (#51923)
* reset cache when starting cypress locally (#51921)
* Don't build FE for driver tests that don't need it; shave 2 minutes off of driver tests (#51884)
* Don't build FE for driver tests
* Try this again
* Consolidate MySQL job configs
* Oops fix conditional flags for test-driver
* More MySQL consolidation
* Oops
* Add a new job that waits for all other jobs (so we can require one single job going forward)
* Restore deleted code
* Try adding a Driver Tests (Summary) workflow
* How about a nice job name
* Don't love YAML inline array syntax
* Try fixing the build-static-viz configuration again
* Try fixing the build-static-viz configuration again
* Try this again
* Fix MySQL SSL config
* Try fixing the drivers summary workflow
* re-remove docs update block (#51928)
* upload title (#51931)
* Fix formatting of month-of-year when there are 53 weeks in the year (#51892)
* New deactivated_at field on users (#51765)
* New deactivated_at field on users
* fix migration syntax
* Setting alignment to top for icons in subscription and alert emails (#51886)
* Correct docstring for series (#51840)
Minor nitpicks on some endpoint docstrings
* Dont escape chars of header dashcards (#51880)
* fix rerun job
* feat(sdk): Interactive Question Chart Settings Dropdown #50976
---------
Co-authored-by: Ryan Laurie <30528226+iethree@users.noreply.github.com>
Co-authored-by: bryan <bryan.maass@gmail.com>
Co-authored-by: Nemanja Glumac <31325167+nemanjaglumac@users.noreply.github.com>
Co-authored-by: Cam Saul <1455846+camsaul@users.noreply.github.com>
Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>
Co-authored-by: Alexander Polyankin <alexander.polyankin@metabase.com>
Co-authored-by: Noah Moss <32746338+noahmoss@users.noreply.github.com>
Co-authored-by: Tyler Plude <tsplude@gmail.com>
Co-authored-by: Luis Paolini <paoliniluis@gmail.com>
Co-authored-by: Oisin Coveney <oisin@metabase.com>
* Remove `entity.ListLoader` from `CollectionItemsTable` - approach v3 (#51958)
* Break CollectionItemsTable down to avoid using Search.ListLoader
* ESLint
* Oops
* Fix auto-upload effect in ModelUploadModal
* Fix types
* Fix the effect running too often
* Remove redundant code
* Reduce diff
* Fix effect
---------
Co-authored-by: Ryan Laurie <30528226+iethree@users.noreply.github.com>
Co-authored-by: bryan <bryan.maass@gmail.com>
Co-authored-by: Nemanja Glumac <31325167+nemanjaglumac@users.noreply.github.com>
Co-authored-by: Cam Saul <1455846+camsaul@users.noreply.github.com>
Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>
Co-authored-by: Alexander Polyankin <alexander.polyankin@metabase.com>
Co-authored-by: Noah Moss <32746338+noahmoss@users.noreply.github.com>
Co-authored-by: Tyler Plude <tsplude@gmail.com>
Co-authored-by: Luis Paolini <paoliniluis@gmail.com>
Co-authored-by: Oisin Coveney <oisin@metabase.com>
* Don't select entire collections for permission checks
* Pull collection perm checks out as a CTE (wip)
* Cljfmt
* Refactor, fix tests
* Move comment, formatting fixes
* do a single query to fetch both the limited results and the total row count
* select more fields from collections
* inline values in the CTE to get it to work on H2
* fix e2e tests
* fix handling of limit = 0 edge case
* inline last ID
* Split Kondo lint fixes off into #51747
* Merge changes from #51660
* Reworked API documentation generation code
* Oops Markdown not YAML
* BIG documentation generation improvements
* Don't try to style tables ourselves
* Fix lint errors
* Fix tests 🔧
* Fix Kondo warnings
* Update OpenAPI dox generator to handle Defendpoint v2
* Update OpenAPI dox generator to handle Defendpoint v2
* Test fixes 🔧
* Fix uberjar compilation
* Static OpenAPI documentation generation
* change output to docs/api.html
* Update test
* PR feedback from @escherize
* add api.html
* Deprecate the old `defendpoint`
* Nuke the old version of the API documentation code (new version will be merged in #51872)
* Remove empty file
* Disable output validation in prod
* Fix the way we define defendpoint 2 handlers
* Fix deps.edn
---------
Co-authored-by: jeff bruemmer <jeff.bruemmer@gmail.com>
* Update docstring in zoom-in.timeseries
The comment is stale. Construction of dimensions from row data is now done in available-drill-thrus.
* Fix zoom-in.geographic drill thru for mutli-stage queries
* Fix return type schema for lib.underlying/top-level-column
* Add append-filter-stage-to-test-expectation to drill-thru test-utils
* Add tests for multi-stage queries to zoom-in.geographic tests
* add New Dashboard button and rework Create a new collection button to New Collection
* fix failing e2e tests
* fix unit tests
* allows users to create a new dashboard to save their dashboard question into
* remove usage of ref
* adds e2e test coverage for creating a dashboard on the go for a new question
* add unit tests
* fix type error
* clean up
* make test less flaky
* Fix quick-filter drill thru for multi-staged queries
* Add column-by-name and append-filter-stage to drill-thru test utils
* Add quick-filter tests for multi-stage queries
* Allow drill-thru tests to pass an update fn for :custom-{query,native,row}
The update fn will get passed the default value for the test case based on query-table and query-type. This allows
callers to easily modify or extend the default, rather than having to look it up or recreate it from scratch.
;; same as the standard test query, but with an extra filter stage
:custom-query #(lib.drill-thru.tu/append-filter-stage % "count")
* Use mu/defmethod and mu/defn for column-by-name and append-filter-stage
* Add lib.drill-thru.test-util/prepend-filter-to-stage
* Add lib.drill-thru.tu/test-drill-variants-with-merged-args
A helper function to make it easy to call test-returns-drill or test-drill-application with minor variations of some
base case (for example, drill behavior should remain unchanged when called with a query with filter stage after
aggregations).
* Use test-drill-variants-with-merged-args for quick-filter multi-stage tests
* Add multi-stage quick-filter tests for a couple of unaggregated queries
* Use lib.drill-thru.tu/column-by-name in metabase.lib.drill-thru-test
* Use lib.drill-thru.tu/column-by-name in lib.drill-thru.test-util.canned
* Support skipping tests in test-drill-variants-with-merged-args
* Allow variant-case to be a fn in test-drill-variants-with-merged-args
* Clean up test formatting
* Reduce diff
* feat(QB): combine chart type selector with viz settings sidebar
* fix typecheck
* fix unit tests
* chore(e2e): create helpers for viz settings and viz type buttons
* fix some tests
* fix most of the e2e tests
* fixup
* more fixes
* another test
* add a warning
* revert cross test changes
* chore: cleanup unused code
* render component directly
* Don't build FE for driver tests
* Try this again
* Consolidate MySQL job configs
* Oops fix conditional flags for test-driver
* More MySQL consolidation
* Oops
* Add a new job that waits for all other jobs (so we can require one single job going forward)
* Restore deleted code
* Try adding a Driver Tests (Summary) workflow
* How about a nice job name
* Don't love YAML inline array syntax
* Try fixing the build-static-viz configuration again
* Try fixing the build-static-viz configuration again
* Try this again
* Fix MySQL SSL config
* Try fixing the drivers summary workflow
* feat(QB): combine chart type selector with viz settings sidebar
* fix typecheck
* fix unit tests
* chore(e2e): create helpers for viz settings and viz type buttons
* fix some tests
* fix most of the e2e tests
* fixup
* more fixes
* another test
* add a warning
* revert cross test changes
* chore(QB): show small Join and CC icons for aggregation without breakout
* fix linting
* tests
* use compact prop and add another test
* revert provider
* fix memo
* Fix typo
* Fix broken memoization
* Use undefined instead of 0
* Introduce entityOptions
* Fix memoization
* Enable EntityListLoaderRtkQuery for alerts (#51501)
* Port bookmarks to use new `EntityListLoaderRtkQuery` (#51502)
* Fix types
* Enable EntityListLoaderRtkQuery for bookmarks
* Revert "Fix types"
This reverts commit f855c3a460.
* Enable EntityListLoaderRtkQuery for dashboards (#51503)
* Enable EntityListLoaderRtkQuery for persisted models (#51505)
* Enable EntityListLoaderRtkQuery for pulses (#51507)
* Enable EntityListLoaderRtkQuery for revisions (#51509)
* Enable EntityListLoaderRtkQuery for persisted segments (#51512)
* Enable EntityListLoaderRtkQuery for snippets (#51513)
* Enable EntityListLoaderRtkQuery for persisted questions (#51508)
* Enable EntityListLoaderRtkQuery for actions (#51515)
* Port tables to use new `EntityListLoaderRtkQuery` (#51518)
* Enable EntityListLoaderRtkQuery for tables
* Fix polling
* Port timelines to use new `EntityListLoaderRtkQuery` (#51516)
* Enable EntityListLoaderRtkQuery for timelines
* Add missing options
* Fix flaky test
* Port groups to use new `EntityListLoaderRtkQuery` (#51504)
* Enable EntityListLoaderRtkQuery for groups
* Bind action creators
* Fix flaky test
* Port users to use new `EntityListLoaderRtkQuery` (#51514)
* Enable EntityListLoaderRtkQuery for users
* Add missing options
* Fix crash
* Reimplement pagination
* Enable EntityListLoaderRtkQuery for search (#51568)
* Port schemas to use new `EntityListLoaderRtkQuery` (#51511)
* Enable EntityListLoaderRtkQuery for persisted schemas
* Add missing options
* Temporary commit
- just to save work at EOY
* Fix file upload
* Fix unit tests
* Fix data being set for different entityQuery
* Port collections to use new `EntityListLoaderRtkQuery` (#51519)
* Enable EntityListLoaderRtkQuery for collections
* Add missing options
* Improve typing
* Fix data being set for different entityQuery
* Fix destructuring crash
* Fix typing
* Pass options to RTK query where missing
* Use undefined instead of 0 for reloadInterval
* Port snippet collections to use new `EntityListLoaderRtkQuery` (#51847)
* Enable EntityListLoaderRtkQuery for snippet collections
* Introduce custom useListQuery
* Reduce diff
* Reduce diff
* Split Kondo lint fixes off into #51747
* Merge changes from #51660
* Reworked API documentation generation code
* Oops Markdown not YAML
* BIG documentation generation improvements
* Don't try to style tables ourselves
* Fix lint errors
* Fix tests 🔧
* Fix Kondo warnings
* Update OpenAPI dox generator to handle Defendpoint v2
* Update OpenAPI dox generator to handle Defendpoint v2
* Test fixes 🔧
* Fix uberjar compilation
* PR feedback from @escherize
* Deprecate the old `defendpoint`
* Nuke the old version of the API documentation code (new version will be merged in #51872)
* Remove empty file
* Disable output validation in prod
* Fix the way we define defendpoint 2 handlers
* start moving code around
* WIP fixes
* Test reorg, some more fixes
* Test fixes
* cljfmt, fix a test
* more cljfmt and linting
* cleanup, formatting fixes
* remove debugging code
* format again
* add some ns docstrings
* fix: Change pool acquireRetryAttemps to 1
Fixes: #51176
* Address PR review and only test dbs with username auth
* Fix model ref
* Exclude hive-like connections
* Ensure user and password
Updates JVM, JS and unit test `MetadataProvider` implementations to include `:ident` fields for all columns.
These idents need to be globally unique and serdes portable; therefore they are not randomized.
Instead they are built from the database, schema, table and field names, eg.
```clojure
{:ident "field__Sample Database__PUBLIC__Orders__CREATED_AT"
#_...}
```
**Note:** These `:ident` fields are not included on `/query_metadata` and other such API calls (currently).
They could be, but they can just as easily be synthesized from the DB, table and field in the JS metadata
implementation.
When you order by an aggregation, mbql v2 references the aggregation by uuid. However, that makes the query hash to a different value each time, because you get a different uuid each time you hash it. Converting the query to reference by index during the hashing process fixes this.
* Ensure database is fully encrypted if key is set, fail f database is encrypted and key is not set
* Added "remove-encryption" command as a more obvious alternative to rotate-encryption-key with an empty key
* fix broken link on BE
* Frontend
* update switch styles
* Fix type errors
* Fix BE tests
* fix malli schema
* logic tweak - don't show the switch if the target model is database
* cljfmt
* Add cypress test
* localize strings
* simplify form updates
* remove unneeded description style
* remove unused enabled? prop
* revert unintended change to cypress helper
* Make renaming options optional in lib.underlying/top-level-column
* Construct dimensions from row data in available-drill-thrus
Construct dimensions from row data once in available-drill-thrus, rather than doing it in each drill that needs it
separately. The dimensions must be constructed in available-drill-thrus since some drills are expected to
return-drills-for-dimensions?, but this also has the side benefit of reducing code duplication by lifting the
rows->dimensions conversion up into a single place.
* Fix the zoom-in.binning drill thru for multi-stage queries
* Add tests for zoom-in.binning with multi-stage queries
* cljfmt
* Explicitly disable the automatic-insights drill for multi-stage queries
Previously, this drill will was implicitly disabled for multi-stage queries because the FE would not pass dimensions
for queries with underlying breakouts / aggregations. Now that dimensions are reconstructed in available-drill-thrus,
automatic-insights drill is available again, but if you try to actually run the drill you get a big scary "Something
went wrong" error, so just explicitly disable the drill until we fix it.
* Update test/metabase/lib/drill_thru/zoom_in_bins_test.cljc
Co-authored-by: Braden Shepherdson <braden@metabase.com>
* Review feedback: not-empty -> seq
* Reivew feeback: shadow rather than using a single-quote suffix
* Review feedback: underyling -> underlying
---------
Co-authored-by: Braden Shepherdson <braden@metabase.com>
* Fix date widgets dark theme
* Update more outdated loki snapshots
* Add a story to theme date filters missing dark theme
* Theme relative current widget and co.
* Theme date exclude widget
* Theme relative date interval widget and co.
* Update more snapshots from border color change
* Fix comment on the wrong line
* Fix light theme story switching + mock relative date stories
* Review: Remove unused function arg
* Update snapshots that change once a year 🤦
That is: aggregations, breakouts, expressions and joins.
Nothing is consuming these yet, but the BE populates them when reading cards.
MBQL changes
=============
These new `:ident`s have been present for new clauses for a few weeks, but now they'll be generated for existing queries on read.
The new idents are stable, including across serialization. They aren't written back until the card is saved again.
Backfilling
===========
Here's the constraints for generating idents for pre-existing clauses:
- If the same query exists across serdes instances, produce the same ident
- Globally unique, even if:
- two cards have identical queries
- two stages of the same card have identical `:count` aggregations.
- a single stage has multiple aggregations of the same type
- a single stage has duplicate breakouts with different time granularity
- Never change, even if the query does
- mbql-lib assumes these are (in theory) randomly generated, and never examines or updates them.
- If new clauses are added, they get random idents.
- If clauses are removed, that's no problem.
- If clauses are edited (eg. changing an expression name) then the BE *would* generate a new ident!
- But when the card gets saved, the original ident with the original expression name is enshrined forever as the ident - it just needs to be unique!
The backfilled idents have this form:
```clojure
"${kind}_${entity_id of the card}@${stage-number}__${name or index}"
;; eg.
"aggregation_ThisIsAValidNanoIdxxx@0__1"
"breakout_ThisIsAValidNanoIdxxx@1__1"
"expression_ThisIsAValidNanoIdxxx@0__exprname"
"join_ThisIsAValidNanoIdxxx@1__alias"
```
* Add M2M Oauth to Databricks Driver
* Add toggle for PAT/M2M Oauth
Introduces a 'Use Personal Access Token' toggle button which shows different UI components when setting up the connection. The implementation was inspired by @snoe in #51249
* Address feedback from @ibrdnk
Addresses the feedback related to #pullrequestreview-2515818008
* Default M2M, Add conditional Merge, Reduce whitespace
* Update docs/databases/connections/databricks.md
Co-authored-by: Jeff Bruemmer <jeffbruemmer@gmail.com>
* Reset FieldValuesWidget to v50 state
* Revert some changes in SingleSelectListField to v50
* Make onSearchChange optional
* Use RowValue instead of string
* Add test-ids to FieldValuesWidget inputs
* Add testing matrix for all parameter combinations
* Add filter matrix test to github workflow
* Fix unit test
* Render custom option value when it exists
* Fix e2e test
* Fix remove value helper
* Fix e2e test
* Render custom label on TokenField
* Update Loki Snapshots
* Use css colors for TokenFieldItem
* Make TokenField dark-mode aware
* Make TokenFieldItem dark-mode aware
* Update Loki Snapshots
* Fix e2e test after merge
* Add listbox role to tokenfield values
* Fix more e2e tests
* Fix e2e test for listbox
* Remove commented out code
* Fix e2e test
---------
Co-authored-by: Metabase Automation <github-automation@metabase.com>
* fixes showing admin links to not admins in the command palette
* updates calculation to take into account EE advanced permissions allowing certain users permissions to change settings who are not admins
* admin routes for non-admins
* fix unit tests
* actually fix unit tests
* adding checks for other permissions
---------
Co-authored-by: Nick Fitzpatrick <nickfitz.582@gmail.com>
clojure.core/pr{:message"Use metabase.util.log instead of clojure.core/pr"}
clojure.core/tap>{:message"Don't use tap> is source code"}
clojure.core/time{:message"Don't use time in source code"}
clojure.test.check/quick-check{:message"You probably meant to use clojure.test.check.clojure-test/defspec instead -- quick check will run the tests when it is evaluated rather than when we run the test suite."}
clojure.java.jdbc/get-connection{:message"Use sql-jdbc.execute/do-with-connection-with-options (for drivers) or t2/with-connection (for the app DB) instead of jdbc/get-connection"}
clojure.java.jdbc/with-db-connection{:message"Use sql-jdbc.execute/do-with-connection-with-options (for drivers) or t2/with-connection (for the app DB) instead of jdbc/with-db-connection"}
clojure.java.jdbc/with-db-metadata{:message"Use sql-jdbc.execute/do-with-connection-with-options + .getMetaData instead of jdbc/with-db-metadata"}
clojure.pprint/pprint{:message"Use metabase.util.log instead of clojure.pprint/pprint"}
clojure.string/lower-case{:message"Use metabase.util/lower-case-en instead of clojure.string/lower-case"}
clojure.string/upper-case{:message"Use metabase.util/upper-case-en instead of clojure.string/upper-case"}
compojure.core/context{:message"Use metabase.api.util.handlers/route-map-handler instead of compojure.core/context, it is much faster and supports OpenAPI spec generation"}
compojure.core/defroutes{:message"Use metabase.api.util.handlers/defroutes instead of compojure.core/routes, it supports OpenAPI spec generation."}
compojure.core/routes{:message"Use metabase.api.util.handlers/routes instead of compojure.core/routes, it supports OpenAPI spec generation."}
fipp.edn/pprint{:message"Use metabase.util.log instead of fipp.edn/pprint"}
honeysql.core/call{:message"Use hx/call instead because it is Honey SQL 2 friendly"}
honeysql.core/raw{:message"Use hx/raw instead because it is Honey SQL 2 friendly"}
java-time/with-clock{:message"Use mt/with-clock"}
java.util.UUID/randomUUID{:message"Use clojure.core/random-uuid instead of java.util.UUID/randomUUID"}
malli.core/explainer{:message"Use metabase.util.malli.registry/explainer instead of malli.core/explainer"}
malli.core/explain{:message"Use metabase.util.malli.registry/explain instead of malli.core/explain because it has built-in caching"}
malli.core/explainer{:message"Use metabase.util.malli.registry/explainer instead of malli.core/explainer because it has built-in caching"}
malli.core/validate{:message"Use metabase.util.malli.registry/validate instead of malli.core/validate because it has built-in caching"}
malli.core/validator{:message"Use metabase.util.malli.registry/validator instead of malli.core/validator because it has built-in caching"}
me.flowthing.pp/pprint{:message"Use metabase.util.log instead of me.flowthing.pp/pprint"}
medley.core/random-uuid{:message"Use clojure.core/random-uuid instead of medley.core/random-uuid"}
metabase.driver/database-supports?{:message"Use metabase.driver.util/supports? instead of metabase.driver/database-supports?"}
metabase.lib.equality/find-column-indexes-for-refs{:message"Use lib.equality/closest-matches-in-metadata or lib.equality/find-closest-matches-for-refs instead of lib.equality/find-column-indexes-for-refs"}
metabase.test/with-temp*{:message"Use mt/with-temp instead of mt/with-temp*"}
toucan2.tools.with-temp{:message"Use mt/with-temp instead of t2.with-temp/with-temp"}}
metabase.lib.equality/find-column-indexes-for-refs{:message"Use lib.equality/closest-matches-in-metadata or lib.equality/find-closest-matches-for-refs instead of lib.equality/find-column-indexes-for-refs"}}
:discouraged-namespace
{camel-snake-kebab.core{:message"CSK is not Turkish-safe, use the versions in metabase.util instead."}
@@ -286,213 +296,12 @@
honeysql.format{:message"Use Honey SQL 2. Honey SQL 1 is removed as a dependency in Metabase 49+."}
honeysql.helpers{:message"Use Honey SQL 2. Honey SQL 1 is removed as a dependency in Metabase 49+."}
honeysql.types{:message"Use Honey SQL 2. Honey SQL 1 is removed as a dependency in Metabase 49+."}
honeysql.util{:message"Use Honey SQL 2. Honey SQL 1 is removed as a dependency in Metabase 49+."}
metabase.util.jvm{:message"All of metabase.util.jvm is re-exported from metabase.util; prefer that"}}
honeysql.util{:message"Use Honey SQL 2. Honey SQL 1 is removed as a dependency in Metabase 49+."}}
:unresolved-var
{:exclude
[colorize.core]}
;; A "module" is any `metabase.x` namespace in `src`.
:metabase/ns-module-checker
{:level:warning
;; Map of module name => the "core" external public-facing API namespace(s). You have three options here:
;;
;; 1. Special sentinel value `:any` means means this module does not (yet) have external public-facing API
;; namespace(s). This is mostly a temporary placeholder until we go in and create module namespaces, which means
;; you should go create one.
;;
;; 2. A set of namespace symbols. All namespaces in other modules will only be allowed to use namespaces from
;; this set. Ideally this set should only have one namespace, but restricting it to a set of several is still
;; better than `:any`.
;;
;; 3. `nil` or not listed here -- we default to assuming there is one API namespace that matches the name of the
;; module itself, e.g. the module namespace for `metabase.setup` would be `metabase.setup`.
;;
;; `nil` or an empty set Otherwise this should be a set of namespace symbols.
;;
;; PRO TIP: Check out the `dev.deps-graph` namespace for helpful tools to see where a module is used externally.
:api-namespaces
{metabase.actions#{metabase.actions.core}
metabase.analytics#{metabase.analytics.prometheus
metabase.analytics.snowplow
metabase.analytics.stats
metabase.analytics.sdk}; TODO -- consolidate these into a real API namespace.
metabase.analyze#{metabase.analyze}
metabase.api#{metabase.api.common
metabase.api.dataset
metabase.api.permission-graph
metabase.api.routes}; TODO -- consolidate these into a real API namespace. I think the `*current-user*` type stuff might need to be moved into a separate module.
metabase.async#{metabase.async.streaming-response
metabase.async.streaming-response.thread-pool
metabase.async.util}; TODO -- consolidate these into a real API namespace.
metabase.audit#{metabase.audit}
metabase.bootstrap#{metabase.bootstrap}
metabase.cmd#{}; there are no namespaces here since you shouldn't be using this in any other module.
metabase.channel#{metabase.channel.core
metabase.channel.render.core
metabase.channel.template.core}
metabase.config#{metabase.config}
metabase.core#{metabase.core.initialization-status}; TODO -- only namespace used outside of EE, this probably belongs in `metabase.server` anyway since that's the only place it's used.
metabase.db#{metabase.db
metabase.db.metadata-queries; TODO this should probably be separate from metabase.db
metabase.db.query; TODO this is mostly util stuff like `metabase.db.query/query` that we don't even need anymore.
metabase.db.setup}; TODO these are only calling `metabase.db.setup/setup-db!` and there's a slightly different version in `metabase.db`
metabase.driver:any; TODO -- 19 namespaces!!!! CRY
metabase.email#{metabase.email
metabase.email.messages
metabase.email.result-attachment}; TODO -- consolidate these into a real API namespace.
metabase.legacy-mbql.util}; TODO -- consolidate these into a real API namespace.
metabase.lib:any; TODO -- :cry: 34 externally referenced namespaces, but maybe half of them are schema namespaces which technically don't need to be required.
metabase.logger#{metabase.logger}
metabase.metabot#{metabase.metabot}
metabase.models:any; TODO -- scream, 62 namespaces used elsewhere, but to be fair a lot of these don't *need* to be required.
metabase.moderation#{metabase.moderation}
metabase.notification#{metabase.notification.core
metabase.notification.payload.core}
metabase.permissions#{metabase.permissions.util}; TODO -- this is currently the only namespace in this module. Give it a real API namespace?
metabase.plugins#{metabase.plugins
metabase.plugins.classloader}; TODO -- not 100% sure the classloader belongs here
:message(format"The name of this %s should end with `!` because it contains calls to non thread safe form `%s`. [:metabase/test-helpers-use-non-thread-safe-functions]"
:message(format"The name of this %s should end with `!` because it contains calls to non thread safe form `%s`. [:metabase/test-helpers-use-non-thread-safe-functions]"
body: "The backport command can only be used after the pull request has been merged. Please wait until the PR is merged before attempting to backport."
// don't bother slack if only the flaky job failed
if (failedJobs.every(job => job.name.toLowerCase().includes("flaky"))) {
return;
}
const failedJobsList = failedJobs.map(job => ({
"type": "rich_text_section",
"elements": [
{
"type": "link",
"text": job.name,
"url": job.html_url,
}
]
}));
await slack.chat.postMessage({
channel:'engineering-ci',
text:'Failing tests',
@@ -70,9 +99,20 @@ jobs:
"type": "section",
"text": {
"type": "mrkdwn",
"text": `Commit <https://github.com/metabase/metabase/commit/${BREAKING_COMMIT}|${BREAKING_COMMIT.slice(0,7)}> by ${AUTHOR} has failing <${FAILED_RUN_URL}|${FAILED_RUN_NAME}> tests on the <https://github.com/metabase/metabase/commits/${BRANCH}|\`${BRANCH}\`> branch. :sad-panda:`
}
"text": `Commit <https://github.com/metabase/metabase/commit/${BREAKING_COMMIT}|${BREAKING_COMMIT.slice(0,7)}> by ${AUTHOR} has failing <${FAILED_RUN_URL}|${FAILED_RUN_NAME}> tests on <https://github.com/${owner}/${repo}/commits/${BRANCH}|\`${BRANCH}\`>`
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.