Fixes#48046
Manual backport of d3a515858a to fix customer escalation in
Not renamed fields in a same table join inherit the renamed name in exports #48046.
Conflicts:
e2e/test/scenarios/dashboard-filters/temporal-unit-parameters.cy.spec.js
frontend/src/metabase/static-viz/lib/settings.ts
frontend/src/metabase/visualizations/lib/settings/column.js
frontend/src/metabase/visualizations/lib/utils.js
frontend/src/metabase/visualizations/shared/settings/column.ts
frontend/src/metabase/visualizations/visualizations/Table.unit.spec.js
src/metabase/query_processor/streaming/xlsx.clj
This was old logic to support certain drivers (eg. pre-JDBC Druid) and
isn't required for most. It's perfectly sound to filter or even break
out on a datetime column without bucketing.
Adds a new `:temporal/requires-default-unit` driver feature, and enables it only for the legacy Druid driver.
Fixes#47341
Co-authored-by: Braden Shepherdson <braden@metabase.com>
* Pivot Options are Properly Calculated for Questions based on Models with Joins (#47830)
Fixes 46575
Creating a Pivot Table Question that is based off of a model that has at least one column derived from a join failed
to display row totals.
This is because the pivot-options map was being mis-calculated; not all column indices were correctly found/passed in
to the :pivot-rows or :pivot-cols keys, causing the pivot query not to compute all necessary data.
Here, I just modify the :lib/source key of the columns whose source is a card (as determined by the existence of
:lib/card-id). The columns being checked will all have :source/breakout, which caused, in the issue's repro example,
the "NAME" column to be missed. If it instead has :lib/source :source/card, the logic inside
`lib/find-matching-column` works.
* include changes from #46594 that this backported changed relies on
---------
Co-authored-by: adam-james <21064735+adam-james-v@users.noreply.github.com>
Co-authored-by: Braden Shepherdson <braden@metabase.com>
* Require `:encryption` for string settings (#48067)
* Require `:encryption` for string settings
For settings that are not typed as JSON, CSV, or strings, encryption now
defaults to `:no` (*except* if you have
explicitly marked your setting as `:sensitive?` - these will default to
`:when-encryption-key-set`).
I went through all our settings and provided what I think are reasonable
values here. I tried to be conservative - when I wasn't sure whether a
stored setting was sensitive, I kept it as encrypted. For example, the
`ldap-port` setting is probably non-sensitive but theoretically someone
could be using a weird port for security-by-obscurity, so I kept that
encrypted.
* Change possible values for `:encryption`
`:maybe` was confusing: let's be more explicit that the value will be
encrypted `:when-encryption-key-set` to make it obvious what actually
turns encryption on and off.
* Don't encrypt `site-locale` (#47731)
* Don't encrypt `site-locale`
* Fix test for `site-locale`
The previous test required encryption. Theoretically the referenced
error just depends on the fact that something is logged during the
process of getting the value, so I reworked it to depend on that
directly rather than triggering it through an encryption key rotation.
* Reset the dashboard parameter when setting the default value
* Fix test for filter default
* Add test for setting parameters when editing the defaults
* Use plain dispatch
* Use cy.log instead of comments
* Use icon selector
---------
Co-authored-by: Romeo Van Snick <romeo@romeovansnick.be>
Co-authored-by: Uladzimir Havenchyk <125459446+uladzimirdev@users.noreply.github.com>
* Pre-fill cloud user info from search params
* Autofocus password field for hosted instances
* Add tests
* Parse setup user info and store it immediatelly
* Autofocus password input field for hosted instances
* Add e2e reproduction
* Fix the user info
* Fix unit test
Co-authored-by: Nemanja Glumac <31325167+nemanjaglumac@users.noreply.github.com>
* Reset filters in dashboard click action (#48075)
* Use default values for parameters in dashboard filters when used from click action
* Add test for setting non-specified parameters to default in dashboard accessed via click action
* Fix multiAutocomplete in tests
* Fix e2e tests
---------
Co-authored-by: Romeo Van Snick <romeo@romeovansnick.be>
* removes the default-ee snapshot in favor for a blockUserGroupPermissions util
* commit default-ee removal i forgot, replace default strings with undefined
* fix oopsie
* fixes failures for not removing enough permissions for tests in sandboxes.cy.spec.js
* rename var
Co-authored-by: Sloan Sparger <sloansparger@users.noreply.github.com>
* Pivot Exports/Downloads Should Not include pivot-grouping or 'Extra' Rows (#47832)
* Pivot Exports/Downloads Should Not include pivot-grouping or 'Extra' Rows
Fixes 46561
When downloading a pivot table's data, a column 'pivot-grouping' is included as well as 'extra' rows that correspond
to various totals in the table.
This PR removes the pivot-grouping column and any of these 'extra' rows from the downloads/exports.
* fix xlsx
* Fix csv
* fix json
* add test and fix some failing tests
* fmt
* Update src/metabase/query_processor/streaming/csv.clj
Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>
* static viz table renders filter out pivot-grouping
* address review feedback
* add some comments to try explain the changes
---------
Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>
* fix xlsx impl that I didn't completely backport properly at first
* pivot-grouping-key available even if not exporting as pivoted
---------
Co-authored-by: adam-james <21064735+adam-james-v@users.noreply.github.com>
Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>
Co-authored-by: Adam James <adam.vermeer2@gmail.com>
* fixes some tests where data permissions were overwriting sandboxing permissions, makes assertions in tests where sandboxing user is used but the test also passes if you were to sign in as the normal user, fixes one test that sandboxes the wrong table for the test
* do not push - wip changes
* add assertions to that network requests are provided sandboxed data, fix faulty test in dashboard-filters-source.cy.spec.js
* use helpers to clean up parts of pagination assertions
* rename assertDatasetIsSandboxed to assertDatasetReqIsSandboxed
* adds assertions on api response data
* clean up
* fix typo
* refactor helper to be easier to follow
* golfing
* pr feedback
* fix
* fix
Co-authored-by: Sloan Sparger <sloansparger@users.noreply.github.com>
* add snowplow mention to e2e docs
* tweak new snowplow doc page to be more in line with old section
* Update docs/developers-guide/e2e-tests.md
---------
Co-authored-by: Filip Hric <filip@filiphric.sk>
Co-authored-by: Uladzimir Havenchyk <125459446+uladzimirdev@users.noreply.github.com>
Fixes#47426
Date operations return values based on the timestamp offset rather than
the session/db timezone. Converting to `timestamp_ltz` first ensures
that we get predictable results.
In the test, it is important that database timezone matches
report_timezone so that post-processing gives proper results.
Co-authored-by: Case Nelson <case@metabase.com>
* fix: bigquery add null checks to result processing
Fixes#47339
On the related issue there are different stacktraces indicating
likely sources of null pointer exceptions.
1. `.getNextPage` is likely returning a nil value. I was unable to reproduce this but one thing I did notice is that `hasNextPage` is recommended over checking `.getNextPageToken`. Added nil handling around `page` possibly being nil.
2. `cancel-chan` may be triggered before processing begins and such `execute-bigquery` would pass nil as a TableResult to the initial reducer. Testing if cancel-chan happens at just the right moment would be too flaky for CI testing but I was able to reproduce this locally and is fixed by the nil handling added.
3. `cancel-chan` may be triggered during query processing. This is covered by a test now.
* Check hasNextPage in test:
* Add test for null getNextPage
* Fix cljfmt
Co-authored-by: Case Nelson <case@metabase.com>
* Ensure :effective-type for columns from an aggregation in a card
Fixes#47184
* Synchronize effective-type with the base-type on override
When :base-type in the column metadata is overridden with the :base-type in the field ref, set it as :effective-type
too. If :effective-type is also set in the field ref, it wins.
Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>
* Fix typo in MetadataProvider docstring
* Fix typo in caching-test: s/Second/Third/
* Store ::nil markers in CachedMetadataProvider for failed lookups
Store ::nil markers in CachedMetadataProvider for any ids for which the wrapped/uncached upstream provider fails to return
metadatas. This prevents repeatedly querying the uncached-provider for ids that don't exist.
The downside is that if the uncached-provider suddenly starts returning metadata for an id that previously did not
exist, we won't pick up on it, but the assumption here is that this is no different / worse than cache invalidation
for existing ids that happen to change after we cache them.
* PR suggestion: remove unnecessary test assertion
Co-authored-by: appleby <86076+appleby@users.noreply.github.com>
* Rollback to legacy unique-name-generator in annotate-native-cols
* Add native query long column alias test to nested queries tests
* Add native query long column alias test to dataset api tests
* Add native query long column alias test to models e2e tests
* Move limit into inner native query
* PR suggestion: simplify lambda argslist
* PR suggestion: remove unnecessary mt/with-temp-copy-of-db
* Tweak test name
---------
Closes#47584
Co-authored-by: appleby <86076+appleby@users.noreply.github.com>
Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>
* Initial commit of controlled upgrades
- new setting upgrade-threshold (`MB_UPGRADE_THRESHOLD`)
number 0-100
- conditionally removing latest from upgrade checks
AS OF RIGHT NOW IT ALWAYS REMOVES the latest.
Should help FE make this optional
* dumb mistake
* handle no latest version info
* implement and basic test
* More tests, refactor name
* more tests
* cljfmt does not like commas in source code.
* Just move site-uuid to the declaration and don't declare it
nothing is sacred in the settings order. no reason. i think i was just
trying to minimize the diff, which a dumb and ignoble goal.
* add context strings
* update unit tests
* use display version
* Have upgrade threshold include current major version
hopefully helps rotate people from early to late, and late to early
across major versions. Idea from sanya and quite nice!
---------
Co-authored-by: Ryan Laurie <iethree@gmail.com>
# Conflicts:
# frontend/src/metabase/admin/settings/components/SettingsUpdatesForm/VersionUpdateNotice/VersionUpdateNotice.tsx
Co-authored-by: Ryan Laurie <iethree@gmail.com>
* progress bar
* rephrase
* better word
* streamline the article
* rename image
* Apply suggestions from code review
---------
Co-authored-by: Alex Yarosh <alexandra@metabase.com>
Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>
so we need to account for that in serialization paths, storage
paths and when exporting/importing foreign keys
Co-authored-by: Alexander Solovyov <alexander@solovyov.net>
Fixes#45938.
Remapping is handled at the top level of the query, and should be
ignored for inner queries. This PR hides it from `results_metadata` on
source queries.
It also fixes the interaction with `large-int-id` logic to make big
integers JS-safe.
Co-authored-by: Braden Shepherdson <braden@metabase.com>
* fixup: Static embed dashboard and question endpoints (#47270)
* entity id translation + tests
* add api level test
* simplify definition of eid-table->model + add test
* update tests to take keywords
* improve comment
* generate the eid-table->model map
* delete now-obsolete test
* make it work in oss
* put the resulting response into a key, so we can add more information later
* formatting
* use model names without the model/ prefix as keys
* Creates list of `api/model->db-model`
- update keys for util/entity_id request
- update shape of util/entity_id response
- add test for not-found eids
* formatting
* Respond to code review feedback
* entity-ids work on /embed/{card,dashboard}* routes
* make entity id translation work for all tokens
* Merge branch 'static-embed-dashboard-and-question-endpoints' of github.com:metabase/metabase into static-embed-dashboard-and-question-endpoints
* massage schema. add tons of tests
* decode more eids
* test alignment
* don't require ns inside of itself :think:
* make the test functions to create tokens better
* bring back `dashboard-url`
---------
Co-authored-by: Oisin Coveney <oisin@metabase.com>
* fixup more tests
* fix a test in api.embed-test
* fix typo
* fix dox typo
* api.embed-test is passing
* Add backport resolution script
* perf: Miscellaneous card rendering improvements (#47679) (#47691)
* perf: Optimize reading rows into vectors in JDBC driver
* perf: Use direct interop when computing with-time-zone-same-instant
Co-authored-by: Oleksandr Yakushev <alex@bytopia.org>
* Catch Errors in Download process and only write error message (#47482) (#47696)
* walk the error to remove any visualization settings
Thanks Dan for the patch, this approach is nice because it doesn't require changing all 3 streaming-response-writer
implementations (json csv and xlsx). It keeps behaviour basically the same, but just cleans up the output to not
include potentially sensitive column/dataset details.
* add a test that confirms json/csv errors are cleaned
* walk the error to remove any visualization settings
Thanks Dan for the patch, this approach is nice because it doesn't require changing all 3 streaming-response-writer
implementations (json csv and xlsx). It keeps behaviour basically the same, but just cleans up the output to not
include potentially sensitive column/dataset details.
* add a test that confirms json/csv errors are cleaned
* fix dataset error test
* cljfmt
* try to be a little more surgical with what we take out of the error
* formatting again, oops
* pass export format, only modify the obj for file exports
In the case of errors being saved to files, the obj is a map with a :status :failed key assoc'd by the qp somewhere
along the way. The format-exception cond branch is not used in this case, so we can't rely on modifying
`format-exception` to just dissoc ex-data.
Instead, I've kept the obj modification, but it excludes the query and the preprocessed keys, which are the only 2
keys where we see :viz-settings. This doesn't eliminate the problem of needing to exclude other keys in the future,
but it does improve the surface area - the query and viz settings are the most likely candidates for holding
potentially sensitive info.
I'd prefer to stick with this solution, as we can get a fix for the security concern in quicker, and perhaps a
non-security bug can be opened regarding the most correct way to indicate errors to users whose downloads have failed.
* keep format-exception the same
* cleaner passing of export-format to write-error!
Co-authored-by: adam-james <21064735+adam-james-v@users.noreply.github.com>
* Revert "Hide upload config when data warehouse is attached" (#47611) (#47666)
The way this was implemented, it also hid the section for management
of uploaded CSV files. Further, admins might want to intentionally
switch from uploading to the data warehouse provided by Metabase to
one of their own.
This reverts commit fbaf58ad03.
References: https://github.com/metabase/harbormaster/issues/5121
References: https://metaboat.slack.com/archives/C032LFJFANL/p1725296255637989
Co-authored-by: Dennis Schridde <63082+devurandom@users.noreply.github.com>
* Hide attached DWH database details (#47419)
== Goal ==
Hide attached DWH database details from anyone incl. admins:
* Do not show them in the UI
* Do not permit to change them
* Do not serialize them
The aim is that customers cannot gain access to (parts of) credentials,
and they cannot break a feature they are paying for by changing
connection details.
Database details are already generally excluded from H2 dump snapshots
(see `metabase.cmd.copy/*copy-h2-database-details*`), thus nothing
changes there.
== Implementation ==
The Metabase backend already contains provisions in the implementation
of `metabase.models.interface/to-json` for `:model/Database` to hide the
`details` of the database in HTTP responses, if the user lacks write
permission on the database. We utilize this by adding an
`is_attached_dwh` column to the `database` table and rejecting
`metabase.models.interface/can-write?` when this flag is enabled. In
the "admin" UI, we show a replacement text instead of the edit form when
the flag is set. (It might be correct to show this whenever `details`
is absent. See below for possible follow-up work.)
However, several sections of the frontend code expected the `details`
field to always be present. In order to make `details` optional, as the
backend seems to handle it, we fix the respective code to treat this
case in the way that appears appropriate in the context.
== How to test ==
=== New behaviour ===
Setting the `is_attached_dwh` field hides the database details:
1. Configure a database as described in https://www.metabase.com/docs/latest/configuring-metabase/config-file#databases.
- In addition to the fields you would normally set, also set
`is_attached_dwh: true`.
- This also works when adding this flag to a database that previously
did not have this flag set.
2. Start your Metabase instance.
3. Verify the database shows up in the "admin" section
(`/admin/databases`).
4. Verify that clicking the database to see its details only reveals
"This database cannot be modified."
5. Verify that responses from the backend do not include a `details`
field for this database.
=== Original behaviour ===
Behaviour without setting the `is_attached_dwh` field is unchanged:
1. Configure a database as described in https://www.metabase.com/docs/latest/configuring-metabase/config-file#databases.
- Only set the fields you would normally set. Do not set
`is_attached_dwh` (or set it to `false`).
2. Start your Metabase instance.
3. Verify the database shows up in the "admin" section
(`/admin/databases`).
4. Verify that clicking the database to see its details only reveal the
regular edit form, showing connection fields like `host`, `user`,
etc. with the values you configured.
== How this will be rolled out ==
1. Upgrade existing Metabase Cloud instances with data warehouse to a
Metabase version that supports `is_attached_dwh`.
2. Set `is_attached_dwh` in the database section of the config file for
Metabase Cloud instances with a data warehouse.
== Possible follow-up work ==
In https://github.com/metabase/metabase/issues/25715, absent
`database.details` was identified as a bug. Since then, `details` was
made `NOT NULL` in the application database, so this bug can no longer
occur. However, today backend responses can be missing the `details`
field, if the current user lacks write permission to the database
setting (see above). Fully re-evaluating #25715 in this context is
outside the scope of this PR.
Closes: https://github.com/metabase/harbormaster/issues/5051
Backports: https://github.com/metabase/metabase/pull/47247
Backports: https://github.com/metabase/metabase/pull/47517
Backports: https://github.com/metabase/metabase/pull/47469
Backports: https://github.com/metabase/metabase/pull/47419
Co-authored-by: Dennis Schridde <63082+devurandom@users.noreply.github.com>
* Fix Transparent disabled state on mantine switch label (#47677) (#47718)
* make custom disabled state more specific than transparent style
* make it less nesty
Co-authored-by: Ryan Laurie <30528226+iethree@users.noreply.github.com>
* update docs (#47725)
* Support field remapping in pivot queries (#47592) (#47693)
* Support field remapping in pivot queries
* Generate breakouts without inversions
Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>
* perf: Optimize some query processor functions (#47658) (#47716)
* perf: [metabase.util.performance] Faster mapv for small collections
* perf: Optimize some query processor functions
Co-authored-by: Oleksandr Yakushev <alex@bytopia.org>
* 🤖 backported "recents updates to better support metrics" (#47141)
* recents updates to better support metrics (#47032)
* recents updates to better support metrics
allows updating metric views through the API
improves parent_collection calculation
* add 2 tests for metrics
* 🤖 backported "Entity id translation service"
---------
Co-authored-by: Bryan Maass <bryan.maass@gmail.com>
* fix: sqlserver handle uniqueidentifier uuids (#47544) (#47638)
* fix: sqlserver handle uniqueidentifier uuids
Fixes#46148
Include sqlserver in `uuid-type` handling as its `uniqueidentifier` type
stores uuids.
* Don't be so precise with varchar size
* Add seam for drivers to cast to text type
* Fix arg order
Co-authored-by: Case Nelson <case@metabase.com>
* Always allow localhost:* on CORS (#47663) (#47710)
* Always allow localhost:* on CORS
* Use `embedding-app-origin-sdk` method for the headers
* Fix BE code formatting
* Fix BE formatting
Co-authored-by: Mahatthana (Kelvin) Nomsawadi <kelvin@metabase.com>
* 🤖 backported "Revert multiple inheritance with SnowflakeVariant" (#47695)
* Revert multiple inheritance with SnowflakeVariant (#47680)
* Revert multiple inheritance with SnowflakeVariant (#47680)
* CI
---------
Co-authored-by: Alexander Polyankin <alexander.polyankin@metabase.com>
* Entity id translation analytics (#47432)
* tracking entity id translation
* add total and more tests
* indentation
* realign the last test
* Backport fixups
* 🤖 backported "upgrade webpack to address security issue" (#47604)
Co-authored-by: Uladzimir Havenchyk <125459446+uladzimirdev@users.noreply.github.com>
* Hide attached DWH database details (#47419)
== Goal ==
Hide attached DWH database details from anyone incl. admins:
* Do not show them in the UI
* Do not permit to change them
* Do not serialize them
The aim is that customers cannot gain access to (parts of) credentials,
and they cannot break a feature they are paying for by changing
connection details.
Database details are already generally excluded from H2 dump snapshots
(see `metabase.cmd.copy/*copy-h2-database-details*`), thus nothing
changes there.
== Implementation ==
The Metabase backend already contains provisions in the implementation
of `metabase.models.interface/to-json` for `:model/Database` to hide the
`details` of the database in HTTP responses, if the user lacks write
permission on the database. We utilize this by adding an
`is_attached_dwh` column to the `database` table and rejecting
`metabase.models.interface/can-write?` when this flag is enabled. In
the "admin" UI, we show a replacement text instead of the edit form when
the flag is set. (It might be correct to show this whenever `details`
is absent. See below for possible follow-up work.)
However, several sections of the frontend code expected the `details`
field to always be present. In order to make `details` optional, as the
backend seems to handle it, we fix the respective code to treat this
case in the way that appears appropriate in the context.
== How to test ==
=== New behaviour ===
Setting the `is_attached_dwh` field hides the database details:
1. Configure a database as described in https://www.metabase.com/docs/latest/configuring-metabase/config-file#databases.
- In addition to the fields you would normally set, also set
`is_attached_dwh: true`.
- This also works when adding this flag to a database that previously
did not have this flag set.
2. Start your Metabase instance.
3. Verify the database shows up in the "admin" section
(`/admin/databases`).
4. Verify that clicking the database to see its details only reveals
"This database cannot be modified."
5. Verify that responses from the backend do not include a `details`
field for this database.
=== Original behaviour ===
Behaviour without setting the `is_attached_dwh` field is unchanged:
1. Configure a database as described in https://www.metabase.com/docs/latest/configuring-metabase/config-file#databases.
- Only set the fields you would normally set. Do not set
`is_attached_dwh` (or set it to `false`).
2. Start your Metabase instance.
3. Verify the database shows up in the "admin" section
(`/admin/databases`).
4. Verify that clicking the database to see its details only reveal the
regular edit form, showing connection fields like `host`, `user`,
etc. with the values you configured.
== How this will be rolled out ==
1. Upgrade existing Metabase Cloud instances with data warehouse to a
Metabase version that supports `is_attached_dwh`.
2. Set `is_attached_dwh` in the database section of the config file for
Metabase Cloud instances with a data warehouse.
== Possible follow-up work ==
In https://github.com/metabase/metabase/issues/25715, absent
`database.details` was identified as a bug. Since then, `details` was
made `NOT NULL` in the application database, so this bug can no longer
occur. However, today backend responses can be missing the `details`
field, if the current user lacks write permission to the database
setting (see above). Fully re-evaluating #25715 in this context is
outside the scope of this PR.
Closes: https://github.com/metabase/harbormaster/issues/5051
Backports: https://github.com/metabase/metabase/pull/47247
Backports: https://github.com/metabase/metabase/pull/47517
Backports: https://github.com/metabase/metabase/pull/47469
Backports: https://github.com/metabase/metabase/pull/47419
Co-authored-by: Dennis Schridde <63082+devurandom@users.noreply.github.com>
* update docs (#47725)
* Support field remapping in pivot queries (#47592) (#47693)
* Support field remapping in pivot queries
* Generate breakouts without inversions
Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>
* perf: Optimize some query processor functions (#47658) (#47716)
* perf: [metabase.util.performance] Faster mapv for small collections
* perf: Optimize some query processor functions
Co-authored-by: Oleksandr Yakushev <alex@bytopia.org>
* fix: sqlserver handle uniqueidentifier uuids (#47544) (#47638)
* fix: sqlserver handle uniqueidentifier uuids
Fixes#46148
Include sqlserver in `uuid-type` handling as its `uniqueidentifier` type
stores uuids.
* Don't be so precise with varchar size
* Add seam for drivers to cast to text type
* Fix arg order
Co-authored-by: Case Nelson <case@metabase.com>
* Always allow localhost:* on CORS (#47663) (#47710)
* Always allow localhost:* on CORS
* Use `embedding-app-origin-sdk` method for the headers
* Fix BE code formatting
* Fix BE formatting
Co-authored-by: Mahatthana (Kelvin) Nomsawadi <kelvin@metabase.com>
* 🤖 backported "Revert multiple inheritance with SnowflakeVariant" (#47695)
* Revert multiple inheritance with SnowflakeVariant (#47680)
* Revert multiple inheritance with SnowflakeVariant (#47680)
* CI
---------
Co-authored-by: Alexander Polyankin <alexander.polyankin@metabase.com>
* 🤖 backported "upgrade webpack to address security issue" (#47604)
Co-authored-by: Uladzimir Havenchyk <125459446+uladzimirdev@users.noreply.github.com>
---------
Co-authored-by: Bryan Maass <bryan.maass@gmail.com>
Co-authored-by: Oisin Coveney <oisin@metabase.com>
Co-authored-by: Metabase bot <metabase-bot@metabase.com>
Co-authored-by: Oleksandr Yakushev <alex@bytopia.org>
Co-authored-by: adam-james <21064735+adam-james-v@users.noreply.github.com>
Co-authored-by: Dennis Schridde <63082+devurandom@users.noreply.github.com>
Co-authored-by: Ryan Laurie <30528226+iethree@users.noreply.github.com>
Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>
Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>
Co-authored-by: Case Nelson <case@metabase.com>
Co-authored-by: Bryan <bryanl@metabase.com>
Co-authored-by: Mahatthana (Kelvin) Nomsawadi <kelvin@metabase.com>
Co-authored-by: Alexander Polyankin <alexander.polyankin@metabase.com>
Co-authored-by: Uladzimir Havenchyk <125459446+uladzimirdev@users.noreply.github.com>
* fixup: Static embed dashboard and question endpoints (#47270)
* entity id translation + tests
* add api level test
* simplify definition of eid-table->model + add test
* update tests to take keywords
* improve comment
* generate the eid-table->model map
* delete now-obsolete test
* make it work in oss
* put the resulting response into a key, so we can add more information later
* formatting
* use model names without the model/ prefix as keys
* Creates list of `api/model->db-model`
- update keys for util/entity_id request
- update shape of util/entity_id response
- add test for not-found eids
* formatting
* Respond to code review feedback
* entity-ids work on /embed/{card,dashboard}* routes
* make entity id translation work for all tokens
* Merge branch 'static-embed-dashboard-and-question-endpoints' of github.com:metabase/metabase into static-embed-dashboard-and-question-endpoints
* massage schema. add tons of tests
* decode more eids
* test alignment
* don't require ns inside of itself :think:
* make the test functions to create tokens better
* bring back `dashboard-url`
---------
Co-authored-by: Oisin Coveney <oisin@metabase.com>
* fixup more tests
* fix a test in api.embed-test
* fix typo
* fix dox typo
* api.embed-test is passing
---------
Co-authored-by: Bryan Maass <bryan.maass@gmail.com>
Co-authored-by: Oisin Coveney <oisin@metabase.com>
* Always allow localhost:* on CORS
* Use `embedding-app-origin-sdk` method for the headers
* Fix BE code formatting
* Fix BE formatting
Co-authored-by: Mahatthana (Kelvin) Nomsawadi <me@bboykelvin.dev>
* fix: sqlserver handle uniqueidentifier uuids
Fixes#46148
Include sqlserver in `uuid-type` handling as its `uniqueidentifier` type
stores uuids.
* Don't be so precise with varchar size
* Add seam for drivers to cast to text type
* Fix arg order
Co-authored-by: Case Nelson <case@metabase.com>
* recents updates to better support metrics (#47032)
* recents updates to better support metrics
allows updating metric views through the API
improves parent_collection calculation
* add 2 tests for metrics
* 🤖 backported "Entity id translation service"
---------
Co-authored-by: Bryan Maass <bryan.maass@gmail.com>
== Goal ==
Hide attached DWH database details from anyone incl. admins:
* Do not show them in the UI
* Do not permit to change them
* Do not serialize them
The aim is that customers cannot gain access to (parts of) credentials,
and they cannot break a feature they are paying for by changing
connection details.
Database details are already generally excluded from H2 dump snapshots
(see `metabase.cmd.copy/*copy-h2-database-details*`), thus nothing
changes there.
== Implementation ==
The Metabase backend already contains provisions in the implementation
of `metabase.models.interface/to-json` for `:model/Database` to hide the
`details` of the database in HTTP responses, if the user lacks write
permission on the database. We utilize this by adding an
`is_attached_dwh` column to the `database` table and rejecting
`metabase.models.interface/can-write?` when this flag is enabled. In
the "admin" UI, we show a replacement text instead of the edit form when
the flag is set. (It might be correct to show this whenever `details`
is absent. See below for possible follow-up work.)
However, several sections of the frontend code expected the `details`
field to always be present. In order to make `details` optional, as the
backend seems to handle it, we fix the respective code to treat this
case in the way that appears appropriate in the context.
== How to test ==
=== New behaviour ===
Setting the `is_attached_dwh` field hides the database details:
1. Configure a database as described in https://www.metabase.com/docs/latest/configuring-metabase/config-file#databases.
- In addition to the fields you would normally set, also set
`is_attached_dwh: true`.
- This also works when adding this flag to a database that previously
did not have this flag set.
2. Start your Metabase instance.
3. Verify the database shows up in the "admin" section
(`/admin/databases`).
4. Verify that clicking the database to see its details only reveals
"This database cannot be modified."
5. Verify that responses from the backend do not include a `details`
field for this database.
=== Original behaviour ===
Behaviour without setting the `is_attached_dwh` field is unchanged:
1. Configure a database as described in https://www.metabase.com/docs/latest/configuring-metabase/config-file#databases.
- Only set the fields you would normally set. Do not set
`is_attached_dwh` (or set it to `false`).
2. Start your Metabase instance.
3. Verify the database shows up in the "admin" section
(`/admin/databases`).
4. Verify that clicking the database to see its details only reveal the
regular edit form, showing connection fields like `host`, `user`,
etc. with the values you configured.
== How this will be rolled out ==
1. Upgrade existing Metabase Cloud instances with data warehouse to a
Metabase version that supports `is_attached_dwh`.
2. Set `is_attached_dwh` in the database section of the config file for
Metabase Cloud instances with a data warehouse.
== Possible follow-up work ==
In https://github.com/metabase/metabase/issues/25715, absent
`database.details` was identified as a bug. Since then, `details` was
made `NOT NULL` in the application database, so this bug can no longer
occur. However, today backend responses can be missing the `details`
field, if the current user lacks write permission to the database
setting (see above). Fully re-evaluating #25715 in this context is
outside the scope of this PR.
Closes: https://github.com/metabase/harbormaster/issues/5051
Backports: https://github.com/metabase/metabase/pull/47247
Backports: https://github.com/metabase/metabase/pull/47517
Backports: https://github.com/metabase/metabase/pull/47469
Backports: https://github.com/metabase/metabase/pull/47419
Co-authored-by: Dennis Schridde <63082+devurandom@users.noreply.github.com>
* walk the error to remove any visualization settings
Thanks Dan for the patch, this approach is nice because it doesn't require changing all 3 streaming-response-writer
implementations (json csv and xlsx). It keeps behaviour basically the same, but just cleans up the output to not
include potentially sensitive column/dataset details.
* add a test that confirms json/csv errors are cleaned
* walk the error to remove any visualization settings
Thanks Dan for the patch, this approach is nice because it doesn't require changing all 3 streaming-response-writer
implementations (json csv and xlsx). It keeps behaviour basically the same, but just cleans up the output to not
include potentially sensitive column/dataset details.
* add a test that confirms json/csv errors are cleaned
* fix dataset error test
* cljfmt
* try to be a little more surgical with what we take out of the error
* formatting again, oops
* pass export format, only modify the obj for file exports
In the case of errors being saved to files, the obj is a map with a :status :failed key assoc'd by the qp somewhere
along the way. The format-exception cond branch is not used in this case, so we can't rely on modifying
`format-exception` to just dissoc ex-data.
Instead, I've kept the obj modification, but it excludes the query and the preprocessed keys, which are the only 2
keys where we see :viz-settings. This doesn't eliminate the problem of needing to exclude other keys in the future,
but it does improve the surface area - the query and viz settings are the most likely candidates for holding
potentially sensitive info.
I'd prefer to stick with this solution, as we can get a fix for the security concern in quicker, and perhaps a
non-security bug can be opened regarding the most correct way to indicate errors to users whose downloads have failed.
* keep format-exception the same
* cleaner passing of export-format to write-error!
Co-authored-by: adam-james <21064735+adam-james-v@users.noreply.github.com>
* perf: Optimize reading rows into vectors in JDBC driver
* perf: Use direct interop when computing with-time-zone-same-instant
Co-authored-by: Oleksandr Yakushev <alex@bytopia.org>
* newsletter-toggle-clicked event
* use ValidateEvent,
* rename event -> simple_event and trackEvent -> trackSimpleEvent
* use the renamed trackSimpleEvent function
Co-authored-by: Nicolò Pretto <info@npretto.com>
* fixup backport of: Entity id translation service
* entity id translation + tests
* add api level test
* simplify definition of eid-table->model + add test
* update tests to take keywords
* improve comment
* generate the eid-table->model map
* delete now-obsolete test
* make it work in oss
* put the resulting response into a key, so we can add more information later
* formatting
* use model names without the model/ prefix as keys
* Creates list of `api/model->db-model`
- update keys for util/entity_id request
- update shape of util/entity_id response
- add test for not-found eids
* formatting
* Respond to code review feedback
---------
Co-authored-by: Oisin Coveney <oisin@metabase.com>
* remove unrelated test
---------
Co-authored-by: Bryan Maass <bryan.maass@gmail.com>
Co-authored-by: Oisin Coveney <oisin@metabase.com>
* fix time series with gaps stacked bar charts crash when enabling all labels (#47599)
* Update Loki Snapshots
---------
Co-authored-by: Aleksandr Lesnenko <alxnddr@users.noreply.github.com>
* rename event -> simple_event and trackEvent -> trackSimpleEvent
* add empty line at the end of the file
---------
Co-authored-by: Nicolò Pretto <info@npretto.com>
Co-authored-by: Thomas Schmidt <thomas@metabase.com>
Fixes#47341.
The QP now supports temporal bucketing of custom expressions, but that has
not been surfaced in the UI until now.
Describe the steps to verify that the changes are working as expected.
1. Use a database other than H2, since it doesn't support time zones.
1. New question -> Anything with a DateTime column
1. Create a custom expression like `convertTimezone([Date Column], "America/New_York", "UTC")`
1. Break out by that expression
The temporal bucketing should work in the UI and in the query results, save properly, etc.
Co-authored-by: Braden Shepherdson <braden@metabase.com>
* Simplify finding metrics in Summarize (#46887)
* Use global search for aggregation picker
* Add names to aggregation picker items so they can be searched
* Render info icon metric descriptions
* Add unit tests for global aggregation picker search
* Add test for searching metrics
* Add a test for metrics descriptions
* Sort metrics
* Fix e2e test
* Use Flex from metabase/ui
* Handle markdown in metric description
* Add test for sorted metrics
* Update Markdown styling in metric description
* Fix clojure formatting
* Test for markdown in the Metric description
* Fix lint
* Fix test for markdown
* Use simpler equality test
* Fix overflow in test
* Add e2e test for legacy metrics searching
* Maintain results as a vec
---------
Co-authored-by: Romeo Van Snick <romeo@romeovansnick.be>
I still don't want users' eyes to bleed looking at (unnecessary?) stacktraces, but we need to be able to see exception
causes to determine what is going wrong
references #47520
Co-authored-by: Alexander Solovyov <alexander@solovyov.net>
Resolved minor conflicts in the following files due to unbackported PRs
src/metabase/lib/schema/util.cljc
- perf: Optimize validation with Malli #46485
src/metabase/query_processor/util.clj
- Use the new private defn #46013
-------------------------------------------------------------------------------
Disallow duplicate order-by clauses in ::lib.schema.order-by/order-bys (#47489)
* Move query-processor.util/remove-lib-uuids to lib.schema.util
This function will soon be needed in lib.schema.util in order to implement distinct order-by clause schema checking.
This function feels like it should live in lib.util instead of lib.schema.util, but that would create the following
circular import dependency since lib.util indirectly requires stuff from lib.schema.util:
lib.schema.util -> lib.util -> lib.schema.util
* Don't create queries with duplicate order-bys in remove_replace_test.cljc
Otherwise, theses tests would fail soon when we add update the order-by schema to reject duplicates.
* Declare lib.order-by/order-bys to return ::lib.schema.order-by/order-bys
Previously it inlined the schema instead.
* Disallow duplicate order-by clauses in ::lib.schema.order-by/order-bys
Fixes: 39384
Co-authored-by: Mike Appleby <86076+appleby@users.noreply.github.com>
* Snowflake Variant cols should also work in attachments
A previous PR fixed the snowflake variant bug by using a try/catch aroung the global-type-settings function.
I realized there was a second codepath using the same function and it gets used in xlsx attachment code, so I've made
the change there as well to prevent the same error in attachments.
* add a test to confirm the viz-settings-for-col function doesn't throw
Co-authored-by: adam-james <21064735+adam-james-v@users.noreply.github.com>
* Incrementally add rows to Native Pivot xlsx exports
Adds a boolean env var `MB_NATIVE_PIVOT_EXPORTS` which enables the 'native pivot' feature in Downloads of xlsx files.
* Simplify the init
* remove unnecessary step
* properly use streaming workbook for pivot export too
This change initializes a non-streaming workbook so that a Pivot table can be added. Then, it turns that workbook into
a streaming workbook, so that we never have to keep the entire output in memory; just like non-pivot xlsx exports, we
can ensure more efficient output.
* Env var name more indicative of its temporary nature
I mean, all things eventually succumb to the inexorable march of time that sands away all lustre, it's just that this
environment variable will disappear much sooner, and intentionally.
* change env var name, add default row count in case no counts exist
* try to improve the area-ref code
* just kidding, figured out how to do A:D style refs
Still need to avoid the zip bomb problem, but at least now we don't need to know or guess the row count beforehand.
And, the fileszie ends up being smaller, though the time to download goes up a little bit.
* fix xlsx test
* should keep col name format behaviour the same
Fixes#30602
We were overriding localdate in bigquery driver to produce
offsetdatetimes for no apparent reason. This causes a number of problems
when dealing with date type columns.
Co-authored-by: Case Nelson <case@metabase.com>
* Snowflake Variant Type should not prevent downloads or attachments
Fixes#46981
The :type/SnowflakeVariant key matches 2 methods in `metabase.query-processor.streaming.common/global-type-settings`.
In this case, it seems that the variant type can be any type, so we shouldn't try to guess anything here, except if
the user has provided a :semantic_type, which we can use.
Otherwise, we'll keep going without formatting details about the variant column, which is likely fine already, as it
should result in strings in the export/attachments.
* add a test
Co-authored-by: adam-james <21064735+adam-james-v@users.noreply.github.com>
* locale query string support on public links and static embeds (#47186)
* keep locale in url query params
* e2e tests
* ugly fix for the missing baseUrl error
* applies suggestion from Kelvin to make code more demure
* Update e2e/test/scenarios/sharing/public-question.cy.spec.js
Co-authored-by: Anton Kulyk <kuliks.anton@gmail.com>
* remove it.skip added by mistake
* sort imports
* handle native question/ SyncedParametersList too
* shorter and more accurate comment 😅
* Update e2e/support/helpers/e2e-embedding-helpers.js
Co-authored-by: Mahatthana (Kelvin) Nomsawadi <me@bboykelvin.dev>
---------
Co-authored-by: Anton Kulyk <kuliks.anton@gmail.com>
Co-authored-by: Mahatthana (Kelvin) Nomsawadi <me@bboykelvin.dev>
* fix tests, the button i was looking for is not on v50
---------
Co-authored-by: Nicolò Pretto <info@npretto.com>
Co-authored-by: Anton Kulyk <kuliks.anton@gmail.com>
Co-authored-by: Mahatthana (Kelvin) Nomsawadi <me@bboykelvin.dev>
These migrations will be backported to v49.
Three changes:
- update IDs/locations of the migrations to v49 vs v51
- add one preCondition (when adding the foreign key constraint)
- add `onFail: MARK_RAN` to the index preconditions. I forgot this
before, so it blows up when the precondition doesn't hold.
Co-authored-by: John Swanson <john.swanson@metabase.com>
* Increase log level to INFO from error
Closes#47396
logs before:
```
2024-08-29 08:37:03,836 INFO db.liquibase :: Checking if Database has unrun migrations...
2024-08-29 08:37:04,045 INFO db.liquibase :: Database has unrun migrations. Checking if migration lock is taken...
2024-08-29 08:37:04,047 INFO db.liquibase :: No migration lock found.
2024-08-29 08:37:04,047 INFO db.liquibase :: Migration lock acquired.
2024-08-29 08:37:04,167 INFO db.liquibase :: Running 83 migrations ...
```
logs after:
```
2024-08-29 08:39:45,019 INFO db.liquibase :: Running 83 migrations ...
...
2024-08-29 08:39:45,166 INFO liquibase.changelog :: ChangeSet migrations/001_update_migrations.yaml::v49.2024-06-27T00:00:00::calherries ran successfully in 0ms
2024-08-29 08:39:45,182 INFO liquibase.changelog :: Columns is_defective_duplicate(boolean) added to metabase_field
2024-08-29 08:39:45,182 INFO liquibase.changelog :: ChangeSet migrations/001_update_migrations.yaml::v49.2024-06-27T00:00:01::calherries ran successfully in 14ms
2024-08-29 08:39:45,199 INFO liquibase.changelog :: Custom SQL executed
2024-08-29 08:39:45,200 INFO liquibase.changelog :: ChangeSet migrations/001_update_migrations.yaml::v49.2024-06-27T00:00:02::calherries ran successfully in 16ms
2024-08-29 08:39:45,202 INFO liquibase.changelog :: Foreign key fk_field_parent_ref_field_id dropped
2024-08-29 08:39:45,202 INFO liquibase.changelog :: ChangeSet migrations/001_update_migrations.yaml::v49.2024-06-27T00:00:03::calherries ran successfully in 1ms
2024-08-29 08:39:45,204 INFO liquibase.changelog :: Foreign key constraint added to metabase_field (parent_id)
2024-08-29 08:39:45,205 INFO liquibase.changelog :: ChangeSet migrations/001_update_migrations.yaml::v49.2024-06-27T00:00:04::calherries ran successfully in 2ms
...
```
* empty commit to bump CI
Co-authored-by: dpsutton <dan@dpsutton.com>
* handle cache config overflowed
* fix root too
* do not comment out test
* make cljfmt happy
---------
Co-authored-by: Alexander Solovyov <alexander@solovyov.net>
* Make metrics inherit the name of the aggregation
Fixes#40355.
The query processor expects the columns resulting from metric aggregations to be named based on the aggregation in the
metric definition, not by the name of the metric. This change brings metabase lib in sync with the query processor.
* Make metrics inherit the name of the aggregation
Fixes#40355.
The query processor expects the columns resulting from metric aggregations to be named based on the aggregation in the
metric definition, not by the name of the metric. This change brings metabase lib in sync with the query processor.
* Fix test
* Resolve metric once only
* Much better collection permission performance (#46942)
PREFACE: THIS IS A BACKPORT COMMIT. And it was a painful one, because some
previous very relevant changes hadn't been backported. So there are some
changes from the original, non-backported, commit:
- removed all support for getting the trash collection, getting things
that were `archived_directly`, or getting collections by
`archive_operation_id`. None of these exist in this branch.
- in this branch, functions like
`permissions-set->visible-collection-ids` did not check for archived
status or anything other than permissions. On `master`, they do. So, the
new function that replaced them,
`collection/visible-collection-filter-clause`, also checks for archived
status as well as permissions. To resolve this, I chose to just default
`collection/visible-collection-filter-clause` to ignore archived status
and return both archived and unarchived collections.
On to the original commit message:
There are a few separate changes here:
- Migrations: add and populate indexed columns perm_value, perm_type, and collection_id to permissions
These fields allows us to efficiently run queries based on collection permissions in the DB without string manipulation. Keeping the table as permissions allows us to do this migration in-place. Note that in some cases collections may have been deleted from the database without deleting the associated permissions row (since there was no foreign key before). We need to be defensive here: if we have a permissions row without a corresponding collection, delete the row before running the rest of the migration.
- Write perm_value, perm_type, and collection_id for new collection permissions
A very simple before-insert method sets these fields before a collection permission is written to the DB.
- Replace collection/permissions-set->visible-collection-ids and collection/visible-collection-ids->honeysql-filter-clause with collection/honeysql-filter-clause
Previously, just about everywhere we used permissions-set->visible-collection-ids, what we were essentially doing was an in-app join: select all the collection IDs you have permission on, then convert it to a SQL clause like WHERE collection_id IN ( all of those collection IDs).
Replace both of these with honeysql-filter-clause, which uses the new fields we added to permissions above to construct a honeysql filter representing "all the collections I have permissions on", without needing to round-trip them to the application and back to the DB.
Of course, we can then write a function visible-collection-ids, which uses honeysql-filter-clause, for those cases where we do actually need the whole bunch in the application (we use this, for example, when constructing the effective-location for a collection).
I also added one more toggle to the VisibilityConfig that's passed into the honeysql-filter-clause (and used to be passed to permissions-set->visible-collection-ids), allowing you to select only the effective children of some collection.
* Backport: Speed up calculation of effective_ancestors
https://github.com/metabase/metabase/pull/47324
Previously, `visible-collection-ids` was effectively "free" in that we'd
cached `collection-id->collection` for *all* collections, within a
single request, and then locally filtered it for permissions without
needing to hit the database again.
To be honest, there is probably a better fix for this - we're repeatedly
calling `visible-collection-ids` when we probably could just save a
single copy of it and use it when calculating the effective location of
every collection.
However, this is a very quick and low-risk fix, and I want to prioritize
getting this done, and then we can improve it later.
Locally, I copied down the database from stats and timed the
`/api/search?model_ancestors=true` endpoint.
Before my "speedup" PR (https://github.com/metabase/metabase/pull/46942)
it took ~7 seconds to return results.
After my "speedup" PR, it took ~15s to return results. :grimace:
With this change, it takes 818ms to return results. 🎉
---------
Co-authored-by: John Swanson <john.swanson@metabase.com>
Previously, truncating a `:type/DateTime` column by `:month` or `:day`
would return a `:type/Date`, which subtly broke the query.
In particular, if you try to order-by the breakout column `Created At (month)`
then it would not get de-duplicated, causing a SQL error about conflicting
ORDER BY clauses.
Fixes#46992.
Co-authored-by: Braden Shepherdson <braden@metabase.com>
Resolved conflicts in legacy-ref-test related to a metrics test that depends on changes that weren't backported.
Relevant unbackported changes in 460ac41f6f
Conflicts:
test/metabase/lib/js_test.cljs
Ensure that CI checks fail if cljs tests fail (#47192)
* Fix typo in expression-clause-normalization-test
s/time-interval/relative-time-interval/
* Add missing table-id to card def to fix failing legacy-ref-test
* Ensure that if yarn test-cljs fails it fails the CI checks
When configured to autorun, the command
shadow-cljs compile test
will exit success even if test failures occur in the node test. See
https://github.com/thheller/shadow-cljs/issues/425
In order to ensure our CI checks fail when cljs tests fail, we need to invoke
node directly. To prevent running the tests twice, remove `:autorun true` from
the `:test` target in shadow-cljs.edn.
Co-authored-by: appleby <86076+appleby@users.noreply.github.com>
Resolve conflicts in zoom_in_timeseries.cljc by replacing unbackported mu/defn- with "mu/defn ^:private".
Conflicts:
src/metabase/lib/drill_thru/zoom_in_timeseries.cljc
-----------------------
zoom-in-timeseries-drill: do not include :hour and :minute for DATE columns (#46997)
* zoom-in-timeseries-drill: do not include :hour and :minute for DATE columns
When the user attempts to drill through a DATE column, we should not return :hour or :minute drill throughs from
available-drill-thrus. Previously we did, which resulted in an error being displayed when the user attempted to select
them.
Modify next-breakout-unit in zoom_in_timeseries.cljc to not return :hour or :minute units for :type/Date columns,
which prevents these from being returned by available-drill-thrus, which prevents them appearing in UI context menus.
* Use lib.temporal-bucket/available-temporal-buckets
Per review feedback, use lib.temporal-bucket/available-temporal-buckets to get the valid-current-units in
metabase.lib.drill-thru.zoom-in-timeseries.
* Do not construct zipmap just to lookup the next unit
Rather than constructing a zipmap from `valid-current-units', which requires iterating over the seq twice, just
traverse the seq once and return the next item in the seq (if any).
Constructing a zipmap made sense in the old code because it was constructed once from static data and used many times
for lookup. In the new version, the vec of `valid-current-units` is dynamic and dependent on the type of the breakout.
* PR suggestion: use threading macro rather than nested sexps
Co-authored-by: Braden Shepherdson <braden@metabase.com>
* PR suggestion: use threading macro rather than transducer / eduction
* PR suggestion: reduce boilerplate in zoom-in-timeseries tests
Reduce code dup by looping over aggregation and bucketing options rather than having a separate deftest for each.
---------
Co-authored-by: Braden Shepherdson <braden@metabase.com>
Fixes#39366
Co-authored-by: appleby <86076+appleby@users.noreply.github.com>
* Don't encrypt boolean settings (by default)
We have tooling to disable encryption on settings even when the
`MB_ENCRYPTION_SECRET_KEY` is set. Turn it on by default for all boolean
settings, which don't need to be encrypted.
I also optimized the code that decrypts settings on startup because I
didn't want to delay startup if someone had set a bunch of boolean
settings. With 20 set, the old version added about 200+ ms to startup,
about 10ms per boolean setting, whether or not it was encrypted or in
the DB already. The optimized version selects all the never-encrypt
values from the database at once (a bit silly, but we also just exclude
raw `true` and `false` values so we don't bother checking them) and
updates them if they're encrypted - this adds ~40ms to startup with 20
encrypted boolean settings (about 2ms per boolean setting) and ~5ms to
startup on subsequent runs, when no encrypted values are in the DB.
Co-authored-by: John Swanson <john.swanson@metabase.com>
This testing is too strict to support some legitimate use cases that
have no workaround at present. We can bring back this type-checking
eventually, once it's possible to correctly express things like
automatic coercion of strings to numbers in `SUM` aggregations.
Fixes#44431.
Co-authored-by: Braden Shepherdson <braden@metabase.com>
* Update appearance.md
Adding information about how colors are selected by Metabase because there is the confusion that the order in which you select colors is the order you get on the chart
* Update appearance.md
Removed code reference and tried to simplify the context
* Update docs/configuring-metabase/appearance.md
---------
Co-authored-by: Tony-metabase <110378427+Tony-metabase@users.noreply.github.com>
Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>
* Add database type adjustments to server side generated temporal values
* Speculative base type change
* Infer timezone only from _actual_ user, not global user level
* Set timezone for actual user in test extensions
* Add test case
* Update type info generation
* Adjust server side relative datetime
* Update src/metabase/driver/sql/query_processor.clj
* Fix base type logic
* Fix typo
* Fix formatting
* Update user name handling
* cljfmt
* Address review comments
---------
Co-authored-by: lbrdnk <lbrdnk@users.noreply.github.com>
Co-authored-by: Braden Shepherdson <braden@metabase.com>
* [serdes] move most of the models over to using spec (#46966)
* [serdes] keep nulls in export when there are nulls in the db
this retains previous behavior
* [serialization] extract subtree should hydrate entities (#47171)
* [serdes] a few improvements, instance_analytics cleanup script
* source_card_id is in master, not 50
* [serdes] move Action to using spec (#46973)
* action fixes
---------
Co-authored-by: Alexander Solovyov <alexander@solovyov.net>
Co-authored-by: Ngoc Khuat <qn.khuat@gmail.com>
* reorganizes data-permissions.ts to be a folder split by a file per concern, moves some logic out into other utils files where it makes sense
* pr feedback and fix lints
Co-authored-by: Sloan Sparger <sloansparger@users.noreply.github.com>
* extending table component features
* untangle pagination controls
* adding pagination control roll
---------
Co-authored-by: Nick Fitzpatrick <nickfitz.582@gmail.com>
* broken state - fixes fe/e2e merge conflicts, commit be conflicts
* fix migration conflict
* fix merge conflicts in recent_views model ns
* Recent views events conflict fix
* commiting backend and frontend fixes
* migration ordering fixes
- also removed: Migrate v1 metrics to v2 metrics as that is not in 50
* fix test namespace broken merges
* remove notion of metric from recent views
* fix tests in api/activity_test
* e2e and CI adjustments
* linting woes
* migration archeology
---------
Co-authored-by: Sloan Sparger <sloansparger@gmail.com>
Co-authored-by: Adam James <adam.vermeer2@gmail.com>
Co-authored-by: Bryan Maass <bryan.maass@gmail.com>
Co-authored-by: Nick Fitzpatrick <nickfitz.582@gmail.com>
* make the newsletter signup a checkbox instead of a button (#46783)
* make the newsletter signup a checkbox instead of a button
* update copy on unit test
* fix describe nesting
* update the ui to be like the design
* make tests use userEvent.click to avoid warning about missing act
* use the subscribeToNewsletter function
* add basic e2e test for newsletter subscription
* move wait to the end of the test to see if it passes in ci
* remove async to see if it fixes the issue
* trying to mock the response
* make it clear why we're mocking the response
* use getUserEmail as suggested by Sloan
* inline var a suggested on code review
* fix padding on setup completed step when on cloud (#47021)
* skip analytics opt out step for people on cloud (#46965)
* poc of removing data usage step for the tech plan
* pretend it's not cloud if hasAddedPaidPlanInPreviousStep is true
* split the jest tests files to simulate pro cloud, pro selfhosted and starter cloud scenarios
* bump onboarding version to 1.3.0 (#47055)
* fix linting issue
* fix: Handle tokens in strings when parsing params
Fixes#38051Fixes#36262Fixes#11755
* Fix handling snippets with single quotes in name
Co-authored-by: Case Nelson <case@metabase.com>
* Split utils from `QuestionDataSource`
* Do the computation inside the component
* Compute query only once
* Fix imports order
Co-authored-by: Nemanja Glumac <31325167+nemanjaglumac@users.noreply.github.com>
* perf: Use more efficient iteration and formatting in CSV export
* perf: Replace clojure.data.csv functions with more efficient implementations
* fix: Fix formatting for integer numbers in scientific notation
Co-authored-by: Oleksandr Yakushev <alex@bytopia.org>
* Refactor e2e JavaScript API commands to TypeScript function helpers (#46934)
* Remove unused file
* Refactor getCurrentUser command to a TS function util
* Refactor createAlert command to a TS function util
* Refactor createModerationReview command to a TS function util
* Reuse createModerationReview
* Refactor createPulse command to a TS function util
* Remove references to deleted files
* Improve typing
* ESLint
---------
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
* refactor: memoize getSteps selector
* screen.getXXX.click() => userEvent.click(...) to avoid missing act() warning
* use mockSettings so that MetabaseSettings.setAll is called and we avoid the unknown features warning
Co-authored-by: Nicolò Pretto <info@npretto.com>
* Add Timeseries insights even when Custom Columns exist (#46253)
* Add Timeseries insights even when Custom Columns exist
WIP
Fixes 46244
I don't know why it was (maybe still is) required that the :other columns be empty before providing insights.
* might be a solution to changing trendlines when data doesn't change
The insights code that powers the frontend trendline requires taking a sample of the dataset.
This sampling has some randomness associated by design, and should maintain its randomness. But it makes some sense to
use the exact same sample when the input has not changed at all, hence trying this memoize approach out.
Might not be the final solution, but it's a start
* add test to show that insights are computed
* use java.util.random with a seed to keep insights stable
* fix failing e2e test
I don't understand why other runs on the release branch have been passing this? But in any case, I've copied this
change from master to fix it.
---------
Co-authored-by: adam-james <21064735+adam-james-v@users.noreply.github.com>
Co-authored-by: Adam James <adam.vermeer2@gmail.com>
* Allow disabling encryption for settings
For two settings, we want to disable encryption entirely.
On the way out of the data source, we don't need to worry - the
decryption is already backwards-compatible with non-encrypted data.
On the way in, things are a bit less straightforward. The core of the
change is replacing the `t2/deftransform` that *always* runs
`encryption/maybe-encrypt` on the value with:
- a define-after-select that always tries to decrypt the value (so we
can handle existing encrypted values in the db)
- a define-before-insert that encrypts the value if the `defsetting`
told us to, and
- a define-before-update that does the same thing.
For now, set two settings to be never-encrypted: `analytics-uuid` and
`anon-tracking-enabled`.
* Show loading state while fetching data in a new model page (#46694)
* Show new model loading state while fetching
* Show new model loading state while fetching
* Add unit test
* Use `DelayedLoadingSpinner` instead
* Update the test
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
* Prettier
---------
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
* Fix the test
---------
Co-authored-by: Nemanja Glumac <31325167+nemanjaglumac@users.noreply.github.com>
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
* Enable type info computation for fields coming from model source query
* Add test
* Add handling for fields coming from joins
Co-authored-by: lbrdnk <lbrdnk@users.noreply.github.com>
* Export Formatting Correctness
WIP
Several bugs exist around the correctness of export formatting, this aims to fix them.
Fixes#27374
When a model has viz settings/metadata changed after save, the download won't incorporate said changes. At least it's
clear that scale doesn't work.
* number formatter merges settings from columns too
* column formatting on aggregated cols works in xlsx now too
* fixes 43040 by grabbing global-type-settings in csv formatter
Co-authored-by: adam-james <21064735+adam-james-v@users.noreply.github.com>
* new version of data_access migration + run ANALYZE beforehand
* relax constraints on migration timestamps
* more migration rewrites
* fix bug in migration
* fix another bug and remove extraneous logging
* reorder migrations to fix ID validation
* start adapting the changes to mysql as well
* wip
* add more ANALYZE calls
* revert logging changes
* finish duplicating work for mysql + a bit of clean up
* create H2 versions of migrations without the ANALYZE calls
* remove ANALYZE call from view_data.sql (probably not necessary)
* fix mysql syntax
* fix another mysql bug
* another mysql fix
* Don't exclude inactive tables for data_access
* Don't write rows for Administrators group
* Don't write rows for Admins (H2/MySQL)
* disable triggers before perms migration
---------
Co-authored-by: Noah Moss <32746338+noahmoss@users.noreply.github.com>
Co-authored-by: John Swanson <john.swanson@metabase.com>
Fixes: #46558
In #45575 we introduced casting uuid fields for drilldowns.
However we failed to include a check that the rhs was itself another
field of uuid type and so no casting should occur. This broke joins on
models when the database_type of the field may not be known.
Co-authored-by: Case Nelson <case@metabase.com>
* Nicer error messages for linting migrations
In general, just throw exceptions instead of using clojure spec here.
You'll only get one error per run, but that seems fine.
I did keep some specs around, but run them slightly differently. Rather
than validating the whole collection of changeSets at once with `s/+`,
just `doseq` through the changeSets and validate each one separately.
That way, the value that's presented as erroneous is much smaller (a
single changeSet) and it's easier to see what went wrong.
* Update bin/lint-migrations-file/test/lint_migrations_file_test.clj
* Update bin/lint-migrations-file/src/lint_migrations_file.clj
* Require preConditions for certain liquibase change types to encourage idempotence (#44578)
* remove `!`s from non-side-effecty fns
* add preConditions to a few v51 migrations
---------
Co-authored-by: Noah Moss <32746338+noahmoss@users.noreply.github.com>
Co-authored-by: Ngoc Khuat <qn.khuat@gmail.com>
* Fix relative date options popover
* Add repro for #46342
* Revert "Fix relative date options popover"
This reverts commit ceaf62d90b.
* Use the Mantine Popover z-index
* conditional dismissal step
* check for download status in the container
* try fixing flakes again
* try again
* chartPathWithFillColor already verifies chart has been rendered
Co-authored-by: Aleksandr Lesnenko <alxnddr@users.noreply.github.com>
* Dowload Row Limit Env Var
Adds `MB_DOWNLOAD_ROW_LIMIT` to enable changing the row limit on downloads and alert/subscription attachments (not the
rendered tables, but the .csv, etc.).
Based on: #44982 (Thanks, @r-kot)
Partially implements: #28144
The difference in this PR compared to #44982 is that the download limit applies to all downloads, the only exception
being when the limit is above `qp.i/absolute-max-results` (1048575, based on Excel's limitation); in such a case, the
user supplied limit is only used if the download is csv or json, and `qp.i/absolute-max-results` is used for xlsx.
This PR also fixes alert/subscription attachment limits; prior to this, they were set to the in-app limit of 2000
rows, but now they will follow the user supplied download-row-limit.
This PR also adds a test to the downloads-and-exports test namespace, confirming that they follow the supplied limit,
or the max limit if none is supplied.
* add test confirming the default limit works
* fix test to use the download-row-limit
* address review feedback
* Update src/metabase/public_settings.clj
* add test covering case where download-row-limit is unset
---------
Co-authored-by: adam-james <21064735+adam-james-v@users.noreply.github.com>
Co-authored-by: Cal Herries <39073188+calherries@users.noreply.github.com>
* Revert "Remove unused humanizeCoercionStrategy and its test suite (#36136)"
This reverts commit 5375347e33.
* humanize coercion strategies, add e2e test
* unit test adjustment
* pr feedback
Co-authored-by: Nick Fitzpatrick <nick@metabase.com>
* Last Used Param Value doesn't fill in with default value if cleared
Fixes#46368
The Last used parameter value is cleared when the frontend sends `nil` as the parameter's value. This works except in
cases where a user clears a parameter with a default value. In this case, we want to indicate that the parameter is
cleared, so we store `nil` in the appdb in such a case.
* Update src/metabase/query_processor/dashboard.clj
* add assertion showing that 'nil' with default is still stored
---------
Co-authored-by: adam-james <21064735+adam-james-v@users.noreply.github.com>
Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>
== Goal ==
Prevent admins of Metabase Cloud instances with Metabase Cloud Storage
from accidentally breaking their upload configuration, since in this
scenario that is automatically managed by Metabase Cloud to use the
attached data warehouse.
== How to test ==
On a subscription with Metabase Cloud Storage:
* The "Uploads" section in the left sidebar of `/admin/settings` does
not show
* A "not found" page is shown at `/admin/settings/uploads` where one
would otherwise expect the "Uploads Settings" page
* `PUT /api/setting` with an `uploads-settings` field in the body fails
with `403 Forbidden` and the setting is not changed
* `PUT /api/setting/uploads-settings` fails with `403 Forbidden` and
the setting is not changed
* `GET /api/setting` contains an element with `key: "uploads-settings"`
and a value
* `GET /api/setting/uploads-settings` responds with the value
* `GET /api/session/properties` contains the `uploads-settings` field
Closes: https://github.com/metabase/harbormaster/issues/5121
Co-authored-by: Dennis Schridde <63082+devurandom@users.noreply.github.com>
Co-authored-by: Ryan Laurie <iethree@gmail.com>
* Add unit test for #46574
* Handle non-numeric values in NumberFingerprint
* Be more realistic about NumberFingerprintDisplayInfo type
* Add comment on why the types are unknown
Co-authored-by: Romeo Van Snick <romeo@romeovansnick.be>
Fixes#46440
couple things come together here.
Bad escaping:
The `{` characters are used for format strings and placeholders like
```clojure
impl=> (translate "fr" "After {0}" ["Bob"])
"Après Bob"
```
And if you want _literal_ {'s in your translation string, you must
escape them like "'{" or "'}" using an apostrophe. But this isn't the
end of our worries.
in `resources/zh-CN.edn`, we actually have _two_ translations for the
original string:
```clojure
"User lookup filter. The placeholder '{login'} will be replaced by the user supplied login."
"用户找过滤器,占位符 '{login'} 将替换为用户提供的登录名。"
,,,
"User lookup filter. The placeholder '{login}' will be replaced by the user supplied login."
"用户查找筛选器。占位符“{login}”将替换为用户提供的登录名。"
```
And this later has unescaped curlies.
The fix for that is at
https://poeditor.com/projects/po_edit?per_page=20&id=200535&id_language=31&filter=&order=&search=User+lookup+filter
but will need to wait for the new source translation string in this
instance with the corrected escaping.
```clojure
;; simulate what happens at runtime
impl=> (translated-format-string "zh_CN"
"User lookup filter. The placeholder '{login}' will be replaced by the user supplied login."
{})
"用户查找筛选器。占位符“{login}”将替换为用户提供的登录名。"
impl=> (MessageFormat. *1 (locale "zh_CN"))
Execution error (NumberFormatException) at java.lang.NumberFormatException/forInputString (NumberFormatException.java:67).
For input string: "login"
;; the problematic zh-CN string
impl=> (MessageFormat. "用户查找筛选器。占位符“{login}”将替换为用户提供的登录名。")
Execution error (NumberFormatException) at java.lang.NumberFormatException/forInputString (NumberFormatException.java:67).
For input string: "login"
;; minimal reproduction
impl=> (MessageFormat. "“{login}”")
Execution error (NumberFormatException) at java.lang.NumberFormatException/forInputString (NumberFormatException.java:67).
For input string: "login"
```
Co-authored-by: dpsutton <dan@dpsutton.com>
== Goal ==
Currently the config file (`MB_CONFIG_FILE_PATH`) only allows adding or
changing databases, but not removing them. Since connected databases
can be managed as a service by us, it becomes necessary to delete any
information Metabase has about them, including any stored credentials.
Since this is a destructive operation with no way to revert (without
restoring from a backup), we raise the bar for accidental deletion and
require not just a (boolean) flag, but a magic string to be set.
== How to test ==
1. Configure a database as described in https://www.metabase.com/docs/latest/configuring-metabase/config-file#databases.
2. Start your Metabase instance.
3. Verify the database shows up in the "admin" section and it works.
Create a dashboard and questions using the database.
4. Add a `delete: "DELETE_WITH_DEPENDENTS:${name}"` line to your
database configuration section.
5. Restart your Metabase instance.
6. Verify the database no longer shows up in the "admin" section.
Notice that questions using it are gone, too.
References: https://github.com/metabase/harbormaster/issues/5173
Co-authored-by: Dennis Schridde <63082+devurandom@users.noreply.github.com>
* fix: mongo ObjectId should be orderable
Fixes#46259
Mongo ObjectId derive :type/TextLike but that can't be an orderable
type, so we mark the specific Mongo type as orderable for malli schemas.
* Fix test
Co-authored-by: Case Nelson <case@metabase.com>
mt/with-temp opens transaction which is used by toucan to make requests, so `mt/with-db` had no impact on database used inside of mt/with-temp
Co-authored-by: Alexander Solovyov <alexander@solovyov.net>
* Update "Clear" and "Reset" buttons (#46050)
* Refactor dashboard helpers to TypeScript
* Reuse existing helper
* Refactor cy.createQuestionAndDashboard to a function helper
- there was no good place to put it, so I also created new helpers/api directory
- and I moved all helpers using cy.request (with 1 exception, where a CSV download helper also does a bunch of assertions) in there
* Merge TS command definitions into a single file
* Export types
* Make dashboardDetails optional
* Export StructuredQuestionDetails
* Remove duplicated function
* Extract createNativeQuestion to separate file
- 1 exported helper per file, no exceptions!
* Reuse const
* Add test for unit of time parameters
* Improve assertions
* Extract createDashboardWithParameters
* Add tests for time parameters
* Fix hardcoded parameter_mappings ids
* Format code
* Extract checkButtonVisible
* Improve test names
* Add tests for location parameters
* Reuse const
* Update test name
* Add WidgetStatusButton
* Add unit tests for WidgetStatusButton
* Rename handler
* Remove redundant assertions
* Add iconSize prop
* Fix styling
* Replace WidgetStatusIcon with WidgetStatusButton
- Remove WidgetStatusIcon
* Fix tooltip issue
* Fix tooltip for good
* Use default compact button size
* Prevent overflow
* Fix alignment issues in parameter sidebar
- Remove iconSize prop
- Simplify StatusConfig
* Update padding in Unit of Time input to make it consistent with other inputs in dashboard parameter sidebar
* Update unit tests
* Rename WidgetStatusButton to WidgetStatus
* Explicitly specify what is a button
* Update helpers in tests
* Update when buttons are displayed according to the new spec, update tests
* Simplify label
* Remove utils
* Adjust padding
* Update reset icon
* Make the button gray when color is undesired
* Remove old definitions
* Add missing export
* Remove unused type
* Add test for location (single value)
* Add test for id (single value)
* Add more assertions
* Add test for id (multiple values)
* Improve name
* Add tests for number (single and multi value)
* Add tests for text (single and multi value)
* Rename file
* Update comment
* Update test name
* Use unique ids
* Use new color name
* Add a test for chevron icons alignment
* Lift className up
* Make types more specific
* Sort attributes
* Revert "Make types more specific"
This reverts commit 2e6994b5ed.
* Revert "Use new color name"
This reverts commit 2cc68df6ba.
* Remove TODO
* Introduce editFilter util, refactor chevron assertions to work with any number of chevrons
* Connect label to the default value input in ParameterSettings
* Add a whitespace in the default value label when required "(required)" is shown
* Add parameter sidebar tests for unit of time
* Add parameter sidebar tests for date filters
* Extract checkParameterSidebarDefaultValue
* Add parameter sidebar tests for location - single values
* Add parameter sidebar tests for location - multiple values
* Use constants for test case names
* Add parameter sidebar tests for id parameters - single values
* Add parameter sidebar tests for id parameters - multi values
* Add parameter sidebar tests for number parameters - single values
* Add parameter sidebar tests for number parameters - multi values
* Add parameter sidebar tests for text parameters - single values
* Add parameter sidebar tests for text parameters - multi values
* Extract checkDashboardFilters
* Use checkDashboardFilters in time parameters
* Use checkParameterSidebarDefaultValue in checkDashboardFilters
* Use checkDashboardFilters everywhere
* Sort helpers
* Rename checkDashboardFilters to checkDashboardParameters
* Rename checkOnlyOneButtonVisible to checkStatusIcon
* Add test suite for sql filters, coverage for text filter - single value
* Add a whitespace in the default value label when required "(required)" is shown - SQL filters
- Similar to 930dd3a50a
* Add test suite for sql filters, coverage for number filters
* Add test for sql filters - date
* Remove unused things, rename placeholder to labelOrPlaceholder
* Apply label only to 1 input
* Add test for sql filters - field parameters
* Fix date tests
* Make all sql filters tests work
* Add assertions for required without default in text filters
* Add assertions for required without default in date filters
* Add assertions for required without default in field filters
* Allow clearing required SQL filters without default value
* Add a test to ensure comma-separated numeric values work
* Update unit test
* Update helper
* Update e2e tests
* Use != null and == null checks with defaultValue instead of just checking if its truthy/falsy
* Run `yarn test-visual:loki` and then `yarn loki approve`
- see https://metaboat.slack.com/archives/C505ZNNH4/p1721289287037069
* Use parameterHasNoDisplayValue to account for empty strings
* Run `yarn test-visual:loki` and then `yarn loki approve` after merging master
- see https://metaboat.slack.com/archives/C505ZNNH4/p1721289287037069
* Run `yarn test-visual:loki` and then `yarn loki approve`
- see https://metaboat.slack.com/archives/C505ZNNH4/p1721289287037069
* Run `yarn test-visual:loki` and then `yarn loki approve`
- see https://metaboat.slack.com/archives/C505ZNNH4/p1721289287037069
* Remove unit of time parameters tests
* Downgrade tests to v50 behavior
* Downgrade tests to v50 behavior
* Remove unit of time test
---------
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
* fix viz settings not persisted during question edit (#46308)
* fix lost settings
* add e2e test
* trigger ci
* fix e2e test
* fix lint
---------
Co-authored-by: Emmad Usmani <emmadusmani@berkeley.edu>
* fix db connection form icons (#46290)
* fix db connection form icons
* fix db connection form icons
* appease typescript
* update tooltip helper for v50
---------
Co-authored-by: Ryan Laurie <30528226+iethree@users.noreply.github.com>
Co-authored-by: Ryan Laurie <iethree@gmail.com>
* updates the ordering of create queries permissions options so that granular permissions is the last option
* fix unit test
* move granular option above no
Co-authored-by: Sloan Sparger <sloansparger@users.noreply.github.com>
Previously we unpacked everything and picked up first entry as the directory
containing the data. This works well when archives are passed as is. However,
if a user unpacks and then packs an archive, there is a good chance that some
hidden directories will appear - and they are first in order.
So we skip those right now, and additionally check known markers of the
serialization data directory.
Co-authored-by: Alexander Solovyov <alexander@solovyov.net>
Since I set auto-merge to #45691, I missed some bits of review feedback.
Luckily, nothing was blocking, so these are small readability/docs changes
Co-authored-by: adam-james <21064735+adam-james-v@users.noreply.github.com>
* Use function helpers
* Only assert the resize down
* Refactor repro 31701
* Remove flaky tag
* Better organize describe blocks
* Move custom `requestTimeout` config to the common block for resizing tests
Co-authored-by: Nemanja Glumac <31325167+nemanjaglumac@users.noreply.github.com>
The `dashboard_load_id` is attached to the body for queries, not as a
query parameter. This correctly receives the `dashboard_load_id` and
uses it for `/query` calls.
Co-authored-by: Braden Shepherdson <braden@metabase.com>
Expands and refactors `metabase.util.memoize` to support better cross-platform memoization.
**Better basic memoization**
Use `clojure.core.memoize/memo` (and its clone in CLJS) rather than `clojure.core/memoize` for
the "everything forever" strategy.
**Bounded memoization**
There are a few places (eg. `u/kebab-case-en`) where we want zero-overhead *hits* and expect the input space
to be fixed. To guard against runaway memory usage, this **bounded** strategy dumps the entire cache when it
overflows. It logs an INFO level note when that happens - the intent here is that the threshold should never
actually get hit!
**Fast JVM versions**
In the special case where the function to memoize takes exactly 1 argument which is a valid map key, we can
use `ConcurrentHashMap.computeIfAbsent` as the basis of the cache and have much less overhead.
`fast-memo` and `fast-bounded` implement this in CLJ; in CLJS they just call `memo` and `bounded`.
Co-authored-by: Braden Shepherdson <braden@metabase.com>
* Refactor dashboard helpers to TypeScript
* Reuse existing helper
* Refactor cy.createQuestionAndDashboard to a function helper
- there was no good place to put it, so I also created new helpers/api directory
- and I moved all helpers using cy.request (with 1 exception, where a CSV download helper also does a bunch of assertions) in there
* Merge TS command definitions into a single file
* Export types
* Make dashboardDetails optional
* Export StructuredQuestionDetails
* Remove duplicated function
* Extract createNativeQuestion to separate file
- 1 exported helper per file, no exceptions!
* Reuse const
* Remove old definitions
* Add missing export
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
* filter invalid recent and search items in move modal
* e2e test disabled moving search and recent items
* update test
Co-authored-by: Ryan Laurie <30528226+iethree@users.noreply.github.com>
* fix conditional formatting color lookup with correct col names
The column names for color lookup are expected to be the same as the :name key of each column, and applying column
formatting broke this lookup. This fixes it.
* move overflow-x into pulse body div to exlude card title from scroll
* add a test for conditional formatting w/ a hidden row.
This still fails at the moment, working on a fix yet.
* apply conditional formatting in static viz even when col is hidden
Conditional formatting that depends on a hidden column was not being rendered in the static viz, but now we pass all
columns into the color selector fn, thus when rendering the table with the ordered/hidden columns, all other columns
will still use their appropriate conditional formatting
* I missed a linter warning
* fix tab rendering
* fix test
* fix up tests with some better structural searching
Co-authored-by: adam-james <21064735+adam-james-v@users.noreply.github.com>
* add skip_graph to coll perm PUT
- also no longer ignores revision and groups
so we don't print this warning:
2024-07-11 16:17:32,293 WARN api.common :: Unexpected parameters at [:put "/api/collection/graph"]: [:revision :groups]
* namespace should be a keyword
* decode the keyword
* add a typehint to avoid reflection
* Adds test for skip_graph
* handle revision 0 🤦
Co-authored-by: bryan <bryan.maass@gmail.com>
* Bigquery Pagination Use next page to get proper results
Fixes 45953
WIP
* use new-page in values-iterator
* disable just the part of the test that is still flaky
Want to keep the other part of the test that does prove we're not getting duplicates due to incorrect pagination.
Co-authored-by: adam-james <21064735+adam-james-v@users.noreply.github.com>
* Rewrite repro for 27643
* Add TODO for the remaining tasks
* Make a test name consistent with others
* Finish the E2E test for #27643
* Fix the test to use PostgresDB
Since H2 originally doesn't fail the test
* Attempt to fix a flake on CI by waiting for table to finish syncing
* Create QA snapshot with fully synced field metadata
* Fix failed tests from field metadata change
* Simplify question scenarios
* Simplify dashboard scenarios
* Fix permissions
* Rework titles
* Add comment about PG
* Add limit to speed things up
* Fix and improve dashboard assertions
* Fix and improve question assertions
---------
Co-authored-by: Nemanja Glumac <31325167+nemanjaglumac@users.noreply.github.com>
Co-authored-by: Mahatthana (Kelvin) Nomsawadi <me@bboykelvin.dev>
* Number formatter Reports correct Decimal places for negative Numbers
Fixes#41640
Prior to this, the number formatter would get the incorrect number of decimal places for negative numbers.
The way the decimal places count is derived works as follows:
- convert number to a string
- split the string on the separator (decimal, comma, whatever the instance has set)
- destructure that split into [int decimal]
- count decimal to get the number of decimals.
That all works for positive numbers, but for negative numbers, the regex was just splitting on 'not digits', so we got
an extra group due to the leading negative sign. This would incorrectly destructure the 'int' part of the number into
the count, thus we would always get 1 instead of the correct value.
This fixes it by just taking last instead of destructuring.
* correct comparison to always use abs value in conditonal check
* tiny incorrectness in the test
Co-authored-by: adam-james <21064735+adam-james-v@users.noreply.github.com>
* ci: Fix request timeout in E2E dashboard filters flake (#46091)
* Make filter corrupted again™
* Speed up the test by limiting the number of rows
* Make the repro more correct, and more resilient
* Remove intercepts
* Improve assertions
* Fix the test
---------
Co-authored-by: Nemanja Glumac <31325167+nemanjaglumac@users.noreply.github.com>
* Make drivers build in parallel rather than sequentially
* Make the whitespace linter happy
* Add duration to steps
Co-authored-by: Luis Paolini <paoliniluis@gmail.com>
It's based on a (global, synchronized) `WeakHashMap` using the query map
itself as the key. That gives the cache the same lifespan as the query
map, which is generally brief but often includes many calls to
expensive, cached functions like `visible-columns`.
Co-authored-by: Braden Shepherdson <braden@metabase.com>
* Embedding Parameter Values set in Tokens work properly when `nil` or empty list (#45771)
* Embedding Parameter Values set in Tokens work properly when `nil` or empty list
Fixes: #40292
If a locked parameter's value is set to `nil` or `[]` in the embedding JWT, we want to treat this as 'no value has
been set' on that filter, but we should still run the query as normal. Likewise, any linked filters should still
properly derive their values from the unfiltered values of the filter to which they are linked.
* when a parameter's value is nil or empty seq, don't include it
This is a partial solution as I still need to find the place that's still erroring on the empty list case in terms of
the linked parameter's values.
* Treat empty list like `nil` for the token param
When you're using a filter in the App, it makes intuitive sense that when you've not selected anything from a filter's
drop down list (or entered any text, etc.), then the filter is effectively not applied.
For token params, we should behave similarly; `nil` and `[]` can both be accepted, and signal that the user intends
for that filter to basically NOT apply.
This also means we should not pass the constraint to the linked filters, since it errors, but even if it worked, it
would return 0 results, making it impossible to properly apply the linked filter at all.
So, this change causes empty lists to be treated as nil, AND will remove any parameters with `nil` as a value before
being passed to the query processor, which results in the proper behaviour of not having constraints on the linked
filter, and having the 'parent' filter not filtering anything.
* Embed preview should always use params from the token
In making a screenshot, I discovered a slight bug in previews, where if the embed had been previously published, it
could pull embedding params from the db that don't match the settings on screen. So, always pull the settings from the
token, which updates as the user is making changes in the Preview Embed UI.
* for previews, merge the embedding params sources instead
* alter 'nil' behaviour to work just as if the key doesn't exist
* fix one spot where I was passing maps into set functions
* Adjust tests to check for consistent behaviour in dash/qstn embeds
* fix an e2e test
* fix up last 2 failing e2e tests
* add the test case of a locked param having no value set in token
* some further e2e changes
* this test wasnt supposed to be in this PR. oops
---------
Co-authored-by: adam-james <21064735+adam-james-v@users.noreply.github.com>
Co-authored-by: Adam James <adam.vermeer2@gmail.com>
Also modify the tests to not actually make HTTP requests. We have the
clj-http-fake client which works well for this sort of thing.
Co-authored-by: John Swanson <john.swanson@metabase.com>
* fix(admin): Don't use regex lookbehinds in Schedule utility functions since these fail on Safari 16.3
* Remove blank describe block
Co-authored-by: Raphael Krut-Landau <raphael.kl@gmail.com>
* Wait for the native editor to be ready
* Add comment
* Make ace editor slightly more resilient to re-renderings
* Improve the test readability
Co-authored-by: Nemanja Glumac <31325167+nemanjaglumac@users.noreply.github.com>
* Remove flakiness from e2e repro
* add to flaky group anyway
* add to flaky group anyway
* fixup! add to flaky group anyway
Co-authored-by: Uladzimir Havenchyk <125459446+uladzimirdev@users.noreply.github.com>
* Uninstall ReplayIO
* Remove Replay config
* Install and use custom Chrome v111
* Simplify comment
* Set a specific Chrome version commit
* Skip repro for #22517
Co-authored-by: Nemanja Glumac <31325167+nemanjaglumac@users.noreply.github.com>
* tests for drivers with spaces in their identifiers
* Add driver feature
* Fix tests
* Remove ambiguous alias for sqlite
* Try to fix bigquery and sqlite tests not seeing left-join support
* Add explicit support for left-join because of foreign-keys during test hacks
* Handle left-join support weirdness
* Fix test
Co-authored-by: Case Nelson <case@metabase.com>
* [WIP] Testing driver filters
* Only test returned values in CI
* Add :uuid-type for dbs that support that type (test oracle's support)
* Try again for oracle test, redshift does not support uuid-type
* Add tests, support where available
* Add empty/null tests
* Adjust test query for athena
* Reset logging
* Fix comments
* Add Athena uuid conversion
* Address PR review
* Add comments
* Address PR comments
* Use varchar for portability. Add inline uuid casting for athena
---------
Co-authored-by: Case Nelson <case@metabase.com>
Co-authored-by: Ngoc Khuat <qn.khuat@gmail.com>
* Support out of order expression references (#45872)
* Support out of order expression references
* Return just the cycle for cyclic expression references
* Prevent "yarn test-cljs" from testing twice
* Fix expected error
---------
Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>
Co-authored-by: Tamás Benkő <tamas@metabase.com>
* fix: replacing joins and conditions updates alias
Fixes#37074
* Only change alias if conditions or table changed
* Fix test
* Fix fe test
* Address PR feedback
Co-authored-by: Case Nelson <case@metabase.com>
* Don't show Invalid questions for Joins in Query Builder (#45768)
* adds database_id to recents, and question collection items
* don't show unjoinable things in recents either
* unskip repro
* fix types
* fix repro
* fix tests in model/recent_views
* add unit test for invalid join in recents
* fix tests in test/metabase/api/collection_test.clj
* update type
* improve unit tests
---------
Co-authored-by: Ryan Laurie <iethree@gmail.com>
* fix recents tests
:collection-id vs :collection_id
---------
Co-authored-by: Bryan Maass <bryan.maass@gmail.com>
Co-authored-by: Ryan Laurie <iethree@gmail.com>
* Add a repro for "Cannot create another filter value if it starts with the same value as an existing value" (#45837)
* Fix lint
---------
Co-authored-by: Alexander Polyankin <alexander.polyankin@metabase.com>
* fix tab actions not working in firefox
* add e2e test
* update e2e tests that need awaitRequest: false
Co-authored-by: Emmad Usmani <emmadusmani@berkeley.edu>
* Fix parameter mapping in auto-wiring a new card
* Fix the test for v50
---------
Co-authored-by: Alexander Polyankin <alexander.polyankin@metabase.com>
* E2E repro for Field Filter's Default Value Not Working on Dashboard with UNIX timestamp (#45811)
* Fix helper
---------
Co-authored-by: Alexander Polyankin <alexander.polyankin@metabase.com>
* docs: extract requires a db with regex
* Update docs/questions/sharing/visualizations/table.md
---------
Co-authored-by: Alex Yarosh <alexandra@metabase.com>
Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>
* Fix - App crashes when navigating back during model creation (#45709)
* Refactor EditoTabs
- get rid of options prop
- use named export
* Remove unused data-testid
* Remove redundant id and aria-labelledby
* Remove redundant props spread
* Bring back data-testid in use
- Thanks VSCode for randomly showing me search results or not...
* Fix navigation issue
* Add a repro for first issue
* Adjust EntityPickerModal root element so that it works with "be.visible" assertion in Cypress
* Add a repro for second issue
* Refactor tests
- Fix not enough history entries issue
* Change order of tests to start with simpler one
* Fix invalid request issue
* Add more assertions
* Format code
* Try to fix a flaking test
* Add waits to fix a flaky test
* Add one more wait
* Fix order of commands
* Improve flaky test
---------
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
* Do not render if there is no time to format
* Add a unit test
* Render time if 0
* Simplify the logic with explicit check for `undefined`
Co-authored-by: Nemanja Glumac <31325167+nemanjaglumac@users.noreply.github.com>
* Field Filter Value Normalization in Embedding Endpoints (#45447)
* Field Filter Value Normalization in Embedding Endpoints
Fixes 27643
Since we use query-params to pass paramter values, we have to normalize the parameter map values.
This PR is a WIP, as I'm trying to find a way to do this hollistically/properly.
First commit here is mostly as a marker for where to start working on this properly, as I think the json parsing won't
catch all cases, nor should the normalization happen without some attempt to use a schema to properly validate, maybe
even coerce things.
* read strings if boolean or numbers, replace blank str with nil
Also update the common dashboard param values fn to use embedding params from the URL blob if the app db doesn't have
embedding params. This happens upon first preview where nothing will have been written to the Appdb yet.
* Add query-param normalization test, add related embedding test
* schema didn't work in this case after all.
Worth a shot, but don't want this to block the PR, so reverting to the previous schema
* Add comment pointing out the downsides/reasons for this solution
It's nto a perfect solution, but it's probably the best we can get at the moment, without adding a new filter
type (boolean) and/or improving the design of hte embedding endpoints and how filters/parameters work
* empty commit to unstick CI
---------
Co-authored-by: adam-james <21064735+adam-james-v@users.noreply.github.com>
Co-authored-by: Adam James <adam.vermeer2@gmail.com>
When we run queries, we preprocess them, during which the sandboxing
middleware adds information that we use when checking permissions
later on in query processing.
When we checked data permissions before saving a card, we were checking
permissions on the raw query, without preprocessing. Without the data
provided by the sandboxing middleware, the permission check sometimes
failed (depending on the permissions for other tables in the DB).
Note the big hack here: preprocessing can fail, for lots of reasons.
This broke a *lot* of tests that were passing in invalid queries. If an
exception occurs during preprocessing, we catch and just use the
original query like we were before.
Co-authored-by: John Swanson <john.swanson@metabase.com>
* Add fallback filter widgets for uncommon (or unknown by Metabase) data types
* Fix type
---------
Co-authored-by: Alexander Polyankin <alexander.polyankin@metabase.com>
* Refactor EditoTabs
- get rid of options prop
- use named export
* Remove unused data-testid
* Remove redundant id and aria-labelledby
* Remove redundant props spread
* Bring back data-testid in use
- Thanks VSCode for randomly showing me search results or not...
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
* Alert on asset download error
* moving function to metabase global
* removing alert, adjusting e2e test
* update CSP
Co-authored-by: Nick Fitzpatrick <nick@metabase.com>
* fix: add timezone to bigquery param substitution
Fixes#43597
* Fix indentation and remove report-timezone lookup from loop
Co-authored-by: Case Nelson <case@metabase.com>
* updates the fe to skip refetching the graph when updating the collection graph, adds tests
* Moves the skip_graph flag to be part of the body.
- the FE tooling makes it awkward for PUTs to include query params
* fixes failing unit tests
* pr feedback
---------
Co-authored-by: Bryan Maass <bryan.maass@gmail.com>
This makes the names inconsistent with what's in `data.cols` on the same
API response(!) but consistent with what MBQL lib will return.
Co-authored-by: Braden Shepherdson <braden@metabase.com>
* Get rid of tolerance
* Use range rect and element rect instead of clientWidth/clientHeight/scrollWidth/scrollHeight
* Fix types
* Mock Range.prototype.getBoundingClientRect
* Mock getBoundingClientRect in unit tests
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
* Add repro for #32252
* Have a test for archiving a question and a collection
* Refactor cy.createCollection and cy.archiveCollection to TS
* Refactor collection and user helpers to TS
* Keep collection helpers together
* Sort code
* Format code
* Use new createCollection helper
* Add return type to createCollection
* Fix typing
* Fix#32252
* Fix typing
* Deprecate openNewCollectionItemFlowFor
* Revert "Fix typing"
This reverts commit ecf3179868.
* Fix TS error
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
* Refactor cy.createCollection and cy.archiveCollection to TS
* Refactor collection and user helpers to TS
* Keep collection helpers together
* Sort code
* Format code
* Add return type to createCollection
* Fix typing
* Deprecate openNewCollectionItemFlowFor
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
* Fix - More than 10 Series (using Add Series Dashboard Feature) cause Filter UI to be inaccessible (#45550)
* Add a repro for #43219
* Fix#43219
* Create questions in parallel
* Make 2 elements scrollable so that the 'Make sure to make a selection ...' element is not weirdly cut off when scrolling horizontally to see inside of parameter-mapper-container
* Fix assertion
* Be more specific which element is to be scrolled
* Avoid realMouseWheel which does not work reliably in CI
* Fix conflict
- Partially port changes from https://github.com/metabase/metabase/pull/43323
- Partially port changes from https://github.com/metabase/metabase/pull/43816
---------
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
* fix cache config migration fails with float setting on mysql (#45539)
* update check sum of v50.2024-06-12T12:33:08 migration that was changed in #45539 (#45563)
---------
Co-authored-by: Ngoc Khuat <qn.khuat@gmail.com>
* fix: limit xrays on agg models to source fields
Fixes#43793
1. `table-like?` was checking aggregations but not breakouts. This meant
that legacy metric definitions would try to apply to the second stage
which is generally illegal and would cause things like implicit joins
to come in where they make no sense.
2. Trying to bin a `count` metric-like model would give a bin span of
0.0 which would blow up those questions.
3. Trying to self reference breakout fields, in the precense of a
question's aggregation meant that most fields would not be exposed to
the next stage and should be ignored. (similar to 1.)
* Fix tests
Co-authored-by: Case Nelson <case@metabase.com>
* do not allow to render empty SaveQuestionModal content (#45476)
* do not allow to render empty SaveQuestionModal content
when question is already saved and QB rerenders still open SaveQuestionModal there is a moment when content is empty before modal is closed
* fixup! do not allow to render empty SaveQuestionModal content
* add missing importg
---------
Co-authored-by: Uladzimir Havenchyk <125459446+uladzimirdev@users.noreply.github.com>
Co-authored-by: Uladzimir Havenchyk <uladzimir.dev@gmail.com>
* Do not reset `table.columns` when some column is deleted
* Do not reset `table.columns` when some column is deleted
* Fix
* Fix
* Make API calls more obvious
Co-authored-by: Alexander Polyankin <alexander.polyankin@metabase.com>
* Fix type/PK -> type/Name and type/FK field remapping in the QB
* Fix type/PK -> type/Name and type/FK field remapping in the QB
* Fix types
* Make API calls more obvious
---------
Co-authored-by: Alexander Polyankin <alexander.polyankin@metabase.com>
* parallelize coll perm graph group lookup w/ core.async
* respond to code review
* add + uses ExecutorCompletionService map function
* rename test
* more tests
* no concurrency for cljs
* fix timeout + add more tests
* use claypoole
* cleanup
* add kondo hooks for claypoole
* cleanup
* Can't use libraries that don't have a license
Revert "add kondo hooks for claypoole"
This reverts commit 9d93b55b28.
* ignore unused value
-- TODO: check back in on claypoole to see if the kondo config gets merged
* fix linter
Co-authored-by: bryan <bryan.maass@gmail.com>
This was a huge footgun.
We were accidentally limiting *ALL* queries run in our transducer to
1000 results. This included the query to fetch permissions, so when we
checked permissions we were checking a small percentage of your *actual*
permissions (e.g. 1000 / ~6300 for me on stats).
Co-authored-by: John Swanson <john.swanson@metabase.com>
This comes in two parts.
**Rebuilt query logic**
I rewrote the query exec logic to use one `future` and one `a/go` block,
reporting results to a `promise`. This is easier to follow and doesn't leak
threads like the original implementation with two `future`s.
The single future is guaranteed to exit. It blocks on the BigQuery call.
That will either complete or throw an exception. Either the exception or
the BigQuery result is sent to the promise, and then the future is
complete. The only reference to that future is overwritten after the `promise`
resolves, so the `future`, its closure, and its thread don't persist longer than needed.
**Return a reducible result**
Previously the results were returned as a lazy sequence, but that brings
a lot of `seq` overhead, and is a risk of retaining the entire result
set in memory until the query is all done. Above all, it doesn't support
`reduced?` to explicitly drop any further results.
The `thunk` function style of `qp.reducible/reducible-rows` doesn't fit
neatly here, so I implemented the reducible logic directly on top of the
`Iterable` BigQuery results.
**Verifying**
This is a bit tricky. I've been testing it like this:
- Add a BigQuery database to my instance
- Create a saved question and note its ID (say, 123)
- Create an API key in (EE) Metabase and save it in an env var
- Install the `siege` load testing tool
- Install VisualVM and monitor my Clojure REPL process's thread count
- `siege --concurrent=15 --time=30s --header="x-api-key: $API_KEY" "http://localhost:3000/api/card/123/query POST"`
That will create a bunch of new threads while it's getting called, but
within a minute of the last queries they'll gradually drop out of the
thread pools and it will return to the baseline number of threads.
Co-authored-by: Braden Shepherdson <braden@metabase.com>
* Implement transfer queues for performing async backfills (#45408)
* Refer to old position of helper
* It's private
---------
Co-authored-by: Chris Truter <crisptrutski@users.noreply.github.com>
Co-authored-by: Chris Truter <chris@metabase.com>
* Fix - Ellipsis tooltip cut off/not visible in GUI editor (#45030)
* Fix#39771
* Add repro for #39771
* Add explanatory comment
* Fix z-index
* Use Mantine Tooltip in Ellipsified to fix z-index issues
* Revert "Fix z-index"
This reverts commit 951e3dca86.
* Revert "Fix #39771"
This reverts commit 4165da5022.
* Update repro to new tooltip
* Update assertions
* Try to fix flake
* Try to fix flake
* Disable Replay to see if it affects tests
* Revert "Disable Replay to see if it affects tests"
This reverts commit e44d9a0a6f.
* Revert "Try to fix flake"
This reverts commit e2f23eb597.
* Revert "Try to fix flake"
This reverts commit 0df6554b38.
* Update assertion and add an explanatory comment
---------
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
* funnel chart handle duplicated steps (#45333)
* handle duplicated funnel chart steps
* fix import
* Update Loki Snapshots
* drop dynamic viz stories for funnel since the setup code is not in the release branch
---------
Co-authored-by: Aleksandr Lesnenko <alxnddr@users.noreply.github.com>
Co-authored-by: Aleksandr Lesnenko <alxnddr@gmail.com>
* Dashboard Paramter Value Now Works in Preview Embed
Fixes: #44962
Prior, a common util fn used both in the embed and preview_embed endpoints was causing the preview_embed to error out
when fetching dashboard parameter values because it was checking if embedding was enabled. This check is needed for
the normal embedding endpoints, but we want the preview embed endpoints to work without needing to publish the
dashboard first.
* pass proper args
* put args back in proper order.
oops
* change form back to previous, but keep preview check
* autocorrect mistake? Or perhaps a silly typo?
* change one test to show that the preview flag is working
Didn't create a whole new test for this flag as it is provable just by adjusting this test to not bother enabling
embedding on the dashboard.
* improve the docstring
Co-authored-by: adam-james <21064735+adam-james-v@users.noreply.github.com>
- getAllCollectionItemNames was breaking the chain in Cypress, so assertions did not know how to wait for DOM to be updated
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
* Log changes when they make sense + async coll perm-graph revision
* - Make revision-number consistent
Decouples the creation of a revision number from the (potentially
expensive) writing of before and after for the coll perms-graph
revision.
* get id from the perms revision entity
* Move the revision id creation into the transaction
The revision id creation should only happen iff the perm updates went
through successfully.
* write a test that checks the incomplete revision
* remove sleep from tests
Co-authored-by: bryan <bryan.maass@gmail.com>
* Static Viz and Exports/Downloads Respect Dashcard Viz Settings (#45164)
* start adding some util fns to download-exports-test ns
* wip improving test util fns
* add subscription attachment util fn
* add tests to catch download/export differences
* viz-settings from card and dashcard merged properly
* revert these changes which probably aren't needed
* split test: downloads can be parallel, email attachments cannot
* normalize viz-settings no matter their origin
* Update test/metabase/api/downloads_exports_test.clj
Co-authored-by: Alexander Solovyov <alexander@solovyov.net>
* Incorporate some review feedback
* rendered table cards also respect column names
* Clean up table render tests and remove unhelpful render test utils
The big removal of code in the render test utils namespace is a clean up task that could've happened sooner, but this
table test adjustment is the perfect opportunity to clean up.
I had made the test utils before I truly knew how best to use `mt/with-temp`, so it had a lot of poorly implemented
'generators' that were doing a poor job of creating mock data for the rendering functions anyway.
So, reworking a few table tests here to just use a temp card is way more inline with the rest of the codebase, and
actually improves confidence that the tests are testing on well-formed data to begin with.
---------
Co-authored-by: Alexander Solovyov <alexander@solovyov.net>
* Use pre-51 method for getting attachments
---------
Co-authored-by: adam-james <21064735+adam-james-v@users.noreply.github.com>
Co-authored-by: Alexander Solovyov <alexander@solovyov.net>
Co-authored-by: Adam James <adam.vermeer2@gmail.com>
This introduces a new metadata on commands, `:requires-init`. If set,
we'll run `metabase.core/init!` before running the command. `init!` will
initialize the database and run our normal startup, though it won't
actually start the Metabase server.
This allows you to use
```
env MB_CONFIG_FILE_PATH=... java -jar metabase.jar import /my/metabase/export
```
We'll run the normal init process (creating the internal user, loading
from the config file, etc), then run the command.
On my machine this adds about 2-3 seconds to the time it takes to run an
`import`.
Co-authored-by: John Swanson <john.swanson@metabase.com>
* Add a test verifying that fonts are loaded
* Verify font is mentioned in HTML
* Bring back working assertions
* Add wait
* Disable speedy
* Disable CSP and emotion cache
* Revert "Disable CSP and emotion cache"
This reverts commit 573e99bfa4.
* Revert "Disable speedy"
This reverts commit 6f9962b869.
* Don't use local fonts
* Revert "Don't use local fonts"
This reverts commit 17c58aa73d.
* Force Lato
* Fix storybook file name
* ESLint
* Add more !important
* Revert !important
* Add font-family assertion
* Move test
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
* perf: pass uuid to BE to enable metadata caching during fetching dashboard
* fixup! perf: pass uuid to BE to enable metadata caching during fetching dashboard
* Add wildcard on interceptions
* Add tests
Co-authored-by: Uladzimir Havenchyk <125459446+uladzimirdev@users.noreply.github.com>
A bunch of small improvements, notably a timeout and support for MSSQL square bracket quotes.
Co-authored-by: Chris Truter <crisptrutski@users.noreply.github.com>
Before
====
```
2024-07-04 15:54:39,084 INFO metabase.server :: Shutting Down Embedded Jetty Webserver
Exception in thread "Thread-9" java.util.IllegalFormatConversionException: f != java.lang.Long
2024-07-04 15:54:39,122 WARN db.liquibase :: (#object[liquibase.Liquibase 0x2b7e6614 liquibase.Liquibase@2b7e6614])
at java.base/java.util.Formatter$FormatSpecifier.failConversion(Unknown Source)
at java.base/java.util.Formatter$FormatSpecifier.printFloat(Unknown Source)
at java.base/java.util.Formatter$FormatSpecifier.print(Unknown Source)
at java.base/java.util.Formatter.format(Unknown Source)
at java.base/java.util.Formatter.format(Unknown Source)
at java.base/java.lang.String.format(Unknown Source)
at clojure.core$format.invokeStatic(core.clj:5770)
at clojure.core$format.doInvoke(core.clj:5764)
at clojure.lang.RestFn.invoke(RestFn.java:423)
at metabase.db.liquibase$wait_for_all_locks.invokeStatic(liquibase.clj:258)
at metabase.db.liquibase$wait_for_all_locks.invoke(liquibase.clj:251)
at metabase.db.setup$release_migration_locks_BANG_.invokeStatic(setup.clj:175)
at metabase.db.setup$release_migration_locks_BANG_.invoke(setup.clj:170)
at metabase.db$release_migration_locks_BANG_.invokeStatic(db.clj:91)
at metabase.db$release_migration_locks_BANG_.invoke(db.clj:86)
at metabase.core$destroy_BANG_.invokeStatic(core.clj:90)
at metabase.core$destroy_BANG_.invoke(core.clj:81)
at clojure.lang.AFn.run(AFn.java:22)
at java.base/java.lang.Thread.run(Unknown Source)
```
And now it will correctly log
```
2024-07-09 16:04:28,345 INFO db.liquibase :: Waiting for migration lock(s) to be released (max 0.4 secs)
```
Why no tests?
====
There _is_ a test that would have caught this. And in fact running at
the repl will catch it:
```clojure
liquibase=> (do (require 'clojure.test 'metabase.db.liquibase-test)
(clojure.test/run-test metabase.db.liquibase-test/wait-for-all-locks-test))
Testing metabase.db.liquibase-test
ERROR in (wait-for-all-locks-test) (Formatter.java:4515)
:h2
Will timeout if a lock is not released
expected: (= :timed-out (liquibase/wait-for-all-locks sleep-ms timeout-ms))
actual: java.util.IllegalFormatConversionException: f != clojure.lang.Ratio
```
But this _passes_ in CI:
```
❯ clj -X:dev:test :only metabase.db.liquibase-test/wait-for-all-locks-test
Finding tests took 23.5 s.
Running 1 tests
Running before-run hooks...
Ran 1 tests in 1.461 seconds
3 assertions, 0 failures, 0 errors.
{:test 1, :pass 3, :fail 0, :error 0, :type :summary, :duration 1461.175042, :single-threaded 1}
Ran 0 tests in parallel, 1 single-threaded.
Finding and running tests took 25.0 s.
All tests passed.
Running after-run hooks...
```
And that's because the log level in testing is FATAL and this is an INFO
level log.
Discussed ways to improve this like adding a null logger during CI but
we aren't ready to increase CI times for that just yet.
Co-authored-by: dpsutton <dan@dpsutton.com>
* Use Mantine Modal for QB Newb disclaimer
* Add repro for #44754
* Use theme spacing units
* Tweak the design slightly
* Fix E2E test
Co-authored-by: Nemanja Glumac <31325167+nemanjaglumac@users.noreply.github.com>
* fix combined cards crash when some cards have invalid viz settings
* fix mock column was not metric
* ensure combined scalar column has proper type to handle the case when the card returns empty results
Co-authored-by: Aleksandr Lesnenko <alxnddr@users.noreply.github.com>
This PR also specifies the 'filled' variant for the question duplication
modal, since the previous default 'filled' variant for submit button was
added to make that specific button filled.
Co-authored-by: Raphael Krut-Landau <raphael.kl@gmail.com>
* Split unrelated changes into separate PR
* Fix test since user perms set reset fix split into other PR
* Fix syntax error
* Test fixes 🔧
* Test fix??? 🔧
* Test fix 🔧
Co-authored-by: Cam Saul <1455846+camsaul@users.noreply.github.com>
* fix: param-fields when native model is aggregated
Fixes#42829
The param-field lookup required result_metadata to match
filterable-columns in order to find parameter targets. This adds a
fallback where if the param field is missing from result_metadata,
then we look at filterable-columns.
* Unskip test
Co-authored-by: Case Nelson <case@metabase.com>
* Repro #37380 - Users with no self-service access may update pivot tables and leave them in a non-working state (#43437)
* Fix crash and remove redundant casting
* Add repro for metabase#37380
* Be extra-safe with an empty array
* [repro] Add e2e repro for Zoom In on pivot tables row headers (#45066)
This works on master, but this reproduction will prevent #38265 from
reoccurring.
* Repro #39504: Pivot table disappears (#45143)
---------
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
Co-authored-by: Braden Shepherdson <braden@metabase.com>
* E2E repro for "Editing Column description of Model is confusing if it contains aggregations" (#45113)
* Fix checks and tests
* Fix checks and tests
* Fix conflict
---------
Co-authored-by: Alexander Polyankin <alexander.polyankin@metabase.com>
* Refine the reproduction for 44794
* Move skip to top-level again
* Remove superflouous click
* Verify wether or not the recents tab exists and is open
* Make the test description more descriptive
* Move question details to a variable
---------
Co-authored-by: Romeo Van Snick <romeo@romeovansnick.be>
Co-authored-by: Nemanja Glumac <31325167+nemanjaglumac@users.noreply.github.com>
* User Parameter Values Are Unique Per Dashboard (#45064)
* User Parameter Values Are Unique Per Dashboard
Fixes: #43001
Might be related: #44858
User Parameter Values previously stored just the `parameter-id` and `user-id` as data to look up the last used
parameter value. This isn't sufficient as the parameter id is not guaranteed to be unique, in particular this is true
when a dashboard is duplicated: the parameters are copied to the new dashboard without changing the parameter id at
all.
This means that we need to also store the dashboard-id in the user_parameter_value table and use that to update/get
the last used value.
The migration removes existing entries to the user_parameter_value table because I want a non-nullable constraint on
the dashboard_id column, and existing entries will not have a value. The only way to backfill those values would be to
look through every dashboard containing parameters, and for every parameter check for a matching ID. Even if you can
do this, there's no way to disambiguate if the same paramter exists on 2 dashboards anyway, so one of them will be
wrong. I think it's not worth the trouble considering that removing the entries in this table doesn't break anything;
it just becomes a mild inconvenience that a user has to select a filter value again (since the dashboard will use the
default value).
* alter test to check for uniqueness across dashboard
This test makes 2 dashboards with parameters of the same ID and asserts that changing the value on dashboard-a does
not change the value on dashboard-b.
* adjust migration to pass linter rules
* remove the extra rollback on migration
* Update src/metabase/models/user_parameter_value.clj
Co-authored-by: bryan <bryan.maass@gmail.com>
* Update src/metabase/models/user_parameter_value.clj
Co-authored-by: bryan <bryan.maass@gmail.com>
* adjust parameters in test so we don't get logged warnings
* put update!/insert! in a transaction to avoid any race conditions
---------
Co-authored-by: bryan <bryan.maass@gmail.com>
* change to proper migration IDs
---------
Co-authored-by: Adam James <adam.vermeer2@gmail.com>
Co-authored-by: bryan <bryan.maass@gmail.com>
* include dark background to PNG exports when night theme is enabled
* fix css
* override border for png downloads
Co-authored-by: Aleksandr Lesnenko <alxnddr@users.noreply.github.com>
`Lib.diagnoseExpression` is stricter about this than the QP or FE
expression parser, making it reject expressions that tried to compare
datetimes to strings like `2024-07-02 14:34:17Z`.
Those used to work in 49.x, but a new `Lib.diagnoseExpression` call
while editing expressions caused it to regress.
Fixes#44916.
Co-authored-by: Braden Shepherdson <braden@metabase.com>
**MetadataProvider cache**
The reusable MetadataProviders is a generally useful thing - within some
dynamic scope like an API request, a `clojure.core.cache` instance can
be provided as `lib.metadata.jvm/*metadata-provider-cache*`, and that
will be used to cache the provider for each `database-id`.
Reusing those providers can significantly reduce appdb traffic,
re-fetching the same metadata in different `MetadataProvider`
instances.
**Dashboard loads**
The FE can supply `?dashboard_load_id=uuid_string` on requests for a
dashboard load:
- `/api/dashboard/:id`
- `/api/dashboard/:id/query_metadata`
- `/api/dashboard/:id/dashcard/:dashcard_id/card/:card_id/query`
That provides a good cache key across that burst of N+2 API requests
that are part of a single load. Currently it is used for two things:
- Caching `get-dashboard` (used by the first two requests)
- Selecting the `clojure.core.cache` for `*metadata-provider-cache*`
Both use a TTL of 10 seconds, which is generally enough for the first
screenful of a dashboard load, if not the whole thing. (Browsers will
only issue a handful of requests at once, so a few long-running queries
might delay later parts of the dashboard from loading until after the
TTL expires.)
Co-authored-by: Braden Shepherdson <braden@metabase.com>
* Make sure the query is native
* Convert the comment to JSDoc
* Add E2E reproduction
* Fix mapping
* Add test scenarios for the public sharing and embedding
* Simplify API payload
* Extract and significantly expand the reproduction scope
---------
Co-authored-by: Nemanja Glumac <31325167+nemanjaglumac@users.noreply.github.com>
Co-authored-by: Alexander Polyankin <alexander.polyankin@metabase.com>
* only send permissions updates for groups that have been modified in the graph
* Support partial permission graph updates on the BE (#44797)
* handles keeping track of advanced permissions changes to send relevant group data when updating the graph, merges response
* tests and optimizations
* respect group-ids set when fetching connection impersonations, sandboxes, and admin group perms
* fix typo
* add e2e test
* fix BE test
* impls pr feedback, reduces code needed since advanced permissions state is always reset on save
* last fixes
---------
Co-authored-by: Sloan Sparger <sloansparger@users.noreply.github.com>
Co-authored-by: Noah Moss <32746338+noahmoss@users.noreply.github.com>
Co-authored-by: Noah Moss <noahbmoss@gmail.com>
* Reset JoinDraft state on database change (fixes metabase#42385)
* Add repro for #42385
* Use admin user
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
* support sorting on funnel bar charts (#44943)
* support sorting on funnel bar charts
* fix funnel bar row display setting
* spec
* Update Loki Snapshots
---------
Co-authored-by: Aleksandr Lesnenko <alxnddr@users.noreply.github.com>
* Add reproduction for #38989
* Use saveQuestion instead of clicking trough the UI
* Use openOrdersTable instead of manually clicking through the ui
* Create question programmatically in test
---------
Co-authored-by: Romeo Van Snick <romeo@romeovansnick.be>
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
* Combine and reorganize admin e2e specs (#43853)
* combine and reorganize admin specs
* fix import and reorganize more
* Revert React 18 changes
* Revert Metrics v2 changes
* Bring back #44497
* Add a reproduction for "Cannot find the new column in pick columns after alter schema and synced" (#44882)
* Add a reproduction for metabase#41765
* Navigate manually to the admin page
* Regroup (metric v1) repro for 36422
* fix: Remove help email from error diagnostic modal (#44165)
---------
Co-authored-by: Ryan Laurie <30528226+iethree@users.noreply.github.com>
Co-authored-by: Romeo Van Snick <romeo@romeovansnick.be>
Co-authored-by: Raphael Krut-Landau <raphael.kl@gmail.com>
* Limit the height of the breakout popover
* Allow the click action popover to untether to prevent overflowing the screen
* Add reproduction for #43075
* Rename breaktrhough to breakout
* Use findAllByTestId
* Remove copy pasted question name
* Remove unused api intercept
---------
Co-authored-by: Romeo Van Snick <romeo@romeovansnick.be>
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
* Wrap non-latin characters in a span specifying working font
Fixes: #38753
CSSBox seems to have a bug where font fallback doesn't work properly. This is noticeable when a font does not contain
glyphs that are present in the string being rendered. For example, Lato does not have many international characters,
so the rendered version of tables (that show up in Slack messages) will not render properly, making the card
unreadable.
Since this appears to be a downstream bug, I've opted to work around this limitation by wrapping any non-latin
characters in a <span> that specifies the font family to be sans-serif, which should contain the glyphs to properly
render.
This leaves Lato in place for other characters.
For now, I figured it's worth trying this solution over using Noto for 2 reasons:
- we can keep Lato, which has been the decided font style for the app for some time (this keeps things consistent
where possible)
- the Noto font containing all glyphs is quite a large font (>20mb, I think) and it would be nice to avoid bundling
that if we can.
* stub installed fonts test
* typo
* Do wrapping, but now per-string, and in Clojure data not html string
I've decided that a reasonable solution is to still wrap strings containing non-lato characters. But it's not done
with str/replace to the html string but rather in a postwalk over the Hiccup data prior to rendering.
Seems like a decent compromise of issues without patching CSSBox or fixing upstream (may be good to do, but will take
longer to get a fix in).
* add test checking that glyphs render correctly
* Add a test that directly checks the wrapping fn
* Change the string to keep the linter quiet
* Change how we check if string can be rendered to faster method, new Lato Font
With Sanya's help, the way I check if a given string is renderable with Lato is now faster.
Also use the full Lato font, not just the 'latin' lato so we can cover more chars
* change lato so that it loads the fn even in a fresh test run
Co-authored-by: adam-james <21064735+adam-james-v@users.noreply.github.com>
Co-authored-by: Nemanja Glumac <31325167+nemanjaglumac@users.noreply.github.com>
* prevent pin maps from filtering out rows with null values
- since pin maps are just displaying where the data is by latitude and longitude, we do not need to filter the pin just because one of the rows is null.
- the metric column selection does not make sense for pins as they just represent that data through coordinates (no aggregations or values displayed), so we should not be filtering out any data just because it is null
* add unit test
Co-authored-by: Jesse Devaney <22608765+JesseSDevaney@users.noreply.github.com>
* Rather support old files, than duplicate columns
* Cope with driver specific failure modes
* Restore bytes comment
Co-authored-by: Chris Truter <crisptrutski@users.noreply.github.com>
* Single Log Out saml-slo-enabled defsetting (#44794)
- backport:
- use `client/client-real-response`, instead of `client-full-response`
* stop using strings for enum values (prefer kw)
* only do SLO when saml-slo-enabled is true
* make defsetting docstring formatting consistent
* log-direction is a string
* add test for enabling / disabling saml slo
- also return a 403 when the client tries, but it's disabled
* use client-full-response, which won't add '/api'.
* fix typo in deftest name
---------
Co-authored-by: bryan <bryan.maass@gmail.com>
I was thinking about the potential options here.
On one hand, we could initialize the app db ourselves. This is a little
problematic, because:
- our `init!` function is in `metabase.core` and calling it yields a
painful cyclical dependency. It's also pretty tied to outputting things
to the user that they might not expect during an import, e.g. "Looks
like this is a new installation ... preparing setup wizard", and starts
up jobs, etc.
- if we choose to *only* initialize the database ourselves (only the
bits that we actually *need* initialized), I'm a little nervous about
maybe missing something. Right now we need to create the internal user
and create users from the config file before running the import. If we
added a third thing, would we remember to update this?
- we could refactor to put all of the above in one spot, but this is a
P1 and it feels important to close this out as quickly as possible. We
also don't claim to support initializing the app DB from an import in
the first place.
So: just throw an exception if the app db hasn't been set up yet. Our
heuristic here is whether a single non-internal user has been created.
So if you want to use your config, you can run:
```
MB_CONFIG_FILE_PATH=dev/config.yml java -jar metabase.jar
java -jar metabase.jar import /my/exported/dump
```
If you try to run `java -jar metabase.jar import ...` on a fresh
DB, it'll now fail with the error message:
```
You cannot `import` into an empty database. Please set up Metabase
normally, then retry.
```
Co-authored-by: John Swanson <john.swanson@metabase.com>
* Fix order by field refs
* Update test to handle all drivers
* Untangle fix-order-by-field-refs
Co-authored-by: lbrdnk <lbrdnk@users.noreply.github.com>
* Repro #41785 - Using Column picker to remove columns on model breaks question with ERROR: column source.name_2 does not exist Position (#44808)
* Add repro for #41785
* Update comment
* Fix failing test
---------
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
* fix: dont always optimize between expressions
Fixes#42291
The frontend produces expressions like
```
[:between
[+ [:field ... {:temporal-bucket :day] [:interval 2 :day]]
[:relative-datetime -1 :week]
[:relative-datetime 0 :week]]
```
This should not be optimized because of the mixed `:day` and `:week`
units. However, it was being optimized since the compatible units
weren't being properly picked up by the match.
* Disable suspcicios args eastwood lint, kondo does it better
* Stop autobucket to day when adding a time interval
Co-authored-by: Case Nelson <case@metabase.com>
Co-authored-by: Nemanja Glumac <31325167+nemanjaglumac@users.noreply.github.com>
* Port the original `reproductions-3.cy.spec.js`
* Port changes from #44053
* E2E repro for "Fix issue with root collection in QuestionPicker" (#44318)
* Port 44415 and 44532
* Port 41381
* Add reproduction for "Result inconsistencies with Preview in GUI editor #40399" (#44630)
* Add e2e test to verify that step previews only care about the step being previewed
* Use getNotebookStep and find preview button using cy.icon
* Remove empty line
Co-authored-by: Nemanja Glumac <31325167+nemanjaglumac@users.noreply.github.com>
---------
Co-authored-by: Nemanja Glumac <31325167+nemanjaglumac@users.noreply.github.com>
* Fix expressions with the same name as their base column (#44641)
* Pass expressionPosition
* Add a test
* Update e2e/test/scenarios/question-reproductions/reproductions-3.cy.spec.js
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
* Rename expressionPosition to expressionIndex
---------
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
* Add E2E repro for convertTimezone (#44705)
* E2E test for custom date expression error (#44694)
* Add reproduction for "Closing Drop-down in Join column selector causes non-drop-down columns to disappear" (#44675)
* Add reproduction for #19894
* Use startNewQuestion
* Fix 43294
---------
Co-authored-by: Alexander Polyankin <alexander.polyankin@metabase.com>
Co-authored-by: Romeo Van Snick <romeo@romeovansnick.be>
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
* Add new e2e test group
* Consolidate `reproductions-1.cy.spec.js`
* Revert changes specific to `master`
* Port the original `reproductions-2.cy.spec.js`
* Port a change from #43481
* Port changes from #43660
* Add a repro for 'Cannot change question data source to raw data after selecting saved question' (#44757)
* Add a repro for 'Cannot change question data source to raw data after selecting saved question'
* Update e2e/test/scenarios/question-reproductions/reproductions-2.cy.spec.js
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
* Update e2e/test/scenarios/question-reproductions/reproductions-2.cy.spec.js
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
* Update e2e/test/scenarios/question-reproductions/reproductions-2.cy.spec.js
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
---------
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
* Add a repro for 'Visible Columns errors when source query is a table joined on itself' (#44737)
* Add a repro for 'Visible Columns errors when source query is a table joined on itself'
* verify warning icon is not rendered
* Fix repro 9027
Revert "move to trash" changes
---------
Co-authored-by: Uladzimir Havenchyk <125459446+uladzimirdev@users.noreply.github.com>
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
* remove invalid else conditional
* fix RowChart firing invalid API request on legend item click
* fix clicking on non-breakout legend on dashboards
* add E2E test to validate and enforce the desired behavior
Co-authored-by: Jesse Devaney <22608765+JesseSDevaney@users.noreply.github.com>
* Add a repro for 'Cannot change question data source to raw data after selecting saved question'
---------
Co-authored-by: Uladzimir Havenchyk <uladzimir.dev@gmail.com>
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
* fix map default display type with un-binned coordinates
- default to `"map.type"` = "pin" when coordinates are not binned
- default to `"map.type"` = "grid" when coordinates are binned
* add tests
* refactor breakouts usage
Co-authored-by: Jesse Devaney <22608765+JesseSDevaney@users.noreply.github.com>
* Allow clicking "Remove all" for the first group in `FieldPanel` and omit removing the first field in the group (#44645)
* Backport some test helpers
---------
Co-authored-by: Alexander Polyankin <alexander.polyankin@metabase.com>
* Fix broken field normalization in metadata
* Handle Fields.actionTypes.UPDATE action in tables reducer to keep table's original_fields in sync
* Add an e2e test for fields
* Format code
* Use a different table, since accounts table is not available in e2e tests
---------
Co-authored-by: Alexander Polyankin <alexander.polyankin@metabase.com>
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
* Consolidate filters reproductions
* Consolidate dashboard filters reproductions
* Add new group to the CI
* Fix recently modified tests that have different behavior in the release branch
* Unskip the repro for #18770 (Post-aggregation filters) (#44748)
* Bump dashboard description max length limit to 1.5K (#44470)
* Bump dashboard description max len limit to 1.5K
* Extract `DASHBOARD_DESCRIPTION_MAX_LENGTH` const
* Handle description max length in dashboard sidebar
* Add red border to dashboard description box
* Add test
* Use `color` fn instead of CSS variables
* Add e2e test to verify that step previews only care about the step being previewed
* Use getNotebookStep and find preview button using cy.icon
* Remove empty line
---------
Co-authored-by: Romeo Van Snick <romeo@romeovansnick.be>
Co-authored-by: Nemanja Glumac <31325167+nemanjaglumac@users.noreply.github.com>
* convert embed.ts to ts and to use createSlice from rtk (#44496)
* convert embed.ts to use createSlice from rtk
* only store in the state what we care about for interactive embedding
* use DEFAULT_EMBED_OPTIONS keys to filter searchOptions
* removed unused hash parameter
* adds basic unit test
* add back in the font to the redux state, as in v50 that's where we store it
* remove validation to not change logic in a minor in the backport
---------
Co-authored-by: Nicolò Pretto <info@npretto.com>
This commit adds `$size` operator wrapping to `$addToSet`. That enables use of `:distinct`
clause in expressions on Mongo.
Co-authored-by: lbrdnk <lbrdnk@users.noreply.github.com>
* Effective location on collection recent items (#43018)
* add effective_location to recent-collection-items
* cleanup
* fix one failing test
---------
Co-authored-by: bryan <bryan.maass@gmail.com>
Co-authored-by: Adam James <adam.vermeer2@gmail.com>
* Add Temporal Units List to Dashboard Parameters Schema
Closes: #44361
This adds an entry to the Dashboard Parameter schema so that we can validate
the list of temporal_units that might be passed from the frontend when adding
or updating the `Unit of Time` type parameters.
* Sanitize the SVG before parsing it with Batik
WIP
Fixes: #43677
There are characters that are invalid according to the Batik XML Parser. This sanitizes the svg string to strip out
some of those characters so that the render can continue.
* Use the xml 1.0 spec allowed unicode chars list for the regex
* Remove accidentally merged code
* still miseed a line
Co-authored-by: adam-james <21064735+adam-james-v@users.noreply.github.com>
* Fix the operator list for `type/TextLike` PKs
* Fix the bug with using wrong operators
* Fix the bug with using wrong operators
* Update test/metabase/lib/filter_test.cljc
* Test FKs too
* Update test/metabase/lib/filter_test.cljc
---------
Co-authored-by: Alexander Polyankin <alexander.polyankin@metabase.com>
Co-authored-by: lbrdnk <lbrdnk@users.noreply.github.com>
This PR removes `:role` removal from spec, even if it was added in `:connection-uri`.
It seems that `:connection-uri` is ignored on `do-with-connection-options` code path.
Hence use of non-default roles is again available on Snowflake.
Co-authored-by: lbrdnk <lbrdnk@users.noreply.github.com>
This not only does not make sense, but also fails for models with aggregations (and thus not having those fields for filtering available).
Co-authored-by: Alexander Solovyov <alexander@solovyov.net>
This PR disables the 'make exports look pivoted' post processing step for csv and xlsx exports.
This is in response to a reported issue that Large Pivot Table downloads are failing entirely: #44556
A more hollistic fix and improvement to the post processing feature will be worked on, so the post processing code is
left untouched in this PR.
The dynamic vars in the csv and xlsx namespaces each default `false` and disable the export from running any post
processing. The var should remain `false` until we've got a plan for these types of exports, and there is no way for
users (either through ENV or UI) to change this, so the feature is effectively disabled.
Co-authored-by: adam-james <21064735+adam-james-v@users.noreply.github.com>
* Set `useLocalSessionState=true` for MySQL DW connections
* Fix GH issue number
* Test fixes 🔧
* Lint fix 🔧
Co-authored-by: Cam Saul <1455846+camsaul@users.noreply.github.com>
* Make the App DB transaction isolation level READ_COMMITTED for MySQL [MEGA PERFORMANCE BOOST] (#44505)
* Make the App DB transaction isolation level READ_COMMITTED for MySQL/MariaDB [MEGA PERFORMANCE BOOST]
* Add test
* empty commit to tickle CI
---------
Co-authored-by: Cam Saul <1455846+camsaul@users.noreply.github.com>
Co-authored-by: Braden Shepherdson <braden@metabase.com>
* Don't run sandboxing rollback test on mysql
* Update test/metabase/db/schema_migrations_test.clj
Co-authored-by: Noah Moss <32746338+noahmoss@users.noreply.github.com>
* fix(cache): Fix two bugs in Admin / Performance (#44255)
Ensures that minimum query durations are correctly displayed in the form post-save (#44249) and that all times can be used in the Schedule component (#44257)
---------
Co-authored-by: Raphael Krut-Landau <raphael.kl@gmail.com>
* UserParameterValue transforms wrapped to properly escape string vals
The `json-in` and `json-out` functions used for toucan model transforms do not perform any string escaping logic. This
makes sense as we don't want to make assumptions about the shape of data flowing in/out of the db. But, this did mean
that for the User paramter values table, string values were causing an error to be logged. This doesn't break
anything, as the function will still correctly return the string, but it can clutter up logs.
So, this PR wraps all incoming values in a map and unwraps it for outgoing values.
* Add migration to wrap existing user param values with ::wrapper key
Since the in/out transform for UserParameterValues is updated, we need to migrate any existing values to have this wrapping.
* Simplify the solution to the problem.
Since we're already getting what we need from the json-in/json-out *except* that it's logging a parse error, we can
create a json-out that works the same way (tries to parse and returns the string as-is if it fails) without logging an
error at all.
* take away the arg and keywordize inside the json-out fn
* remove irrelevant comment
* Add a little more test coverage for different value types
* faster test thanks to Dan!
* Add a comment about the data being tested
---------
Signed-off-by: Adam James <adam.vermeer2@gmail.com>
Co-authored-by: adam-james <21064735+adam-james-v@users.noreply.github.com>
* XLSX Pivot Table Downloads have 'Native' Pivot Table (#43791)
* XLSX Pivot Table Downloads have 'Native' Pivot Table
Addresses: #2473
The xlsx downloads work fine for regular tables. However, on Pivot Table downloads, the resulting file isn't pivoted
and is formatted somewhat strangely (it contains additional rows related to totals, has an extra 'pivot-grouping'
column).
Now, the downloaded xlsx file contains 2 sheets:
- the data sheet containing the question's rows unpivoted. If you were constructing a pivot table yourself, you would
probably start with this shape of data.
- the pivot sheet containing a 'native' Pivot table whose reference points to the data on the data sheet, and whose
columns, rows, and values align with the cols, rows, and measures set up in Metabase
* Silly typo!
* Exported Pivot uses correct aggregation (sum, count, stddev, avg, min, max)
* data for pivot exports now also uses the formatting
* Add a test asserting that a pivot table exists in the xlsx file
* add-row! is a method dispatching on sheet class
* remove unnecessary hint
* address review feedback:
- consolidate the aggregation function key fns into one function in the .xlsx namespace
this was moved out of postprocessing so that the xlsx specific fns don't litter the post processing namespace at
this time
- clean up an unnecessary destructiring inside the native-pivot function
- don't shadow 'name' inside the body of a fn
* Add tests that confirm zero-col and zero-row situations work
* Consolidate the atoms used to store pivot rows/options
* unstick be checks?
---------
Co-authored-by: adam-james <21064735+adam-james-v@users.noreply.github.com>
Co-authored-by: Adam James <adam.vermeer2@gmail.com>
Fixes#43993 for real.
The earlier fix#44182 was needlessly restricting the
"nominal refs" check to fields which were using numeric IDs,
when really any match on nominal refs is valid.
Co-authored-by: Braden Shepherdson <braden@metabase.com>
* Pivot Tables with no Pivot Columns should still download
Fixes#44159
Pivot Tables can still be valid without columns, and therefore the download should respect such a query and be
successfully exported.
In this PR, the postprocessing of pivot results into 'visual pivot' exports for csv takes into account the zero-column
configuration and the download no longer fails.
* add a test to confirm that zero cols and >1 measures works as well
* Handle the case where there are zero pivot-rows.
This is a similar problem to the zero pivot-cols case, just with rows.
The app actually doesn't render these tables correctly, but that's a separate frontend bug
Co-authored-by: adam-james <21064735+adam-james-v@users.noreply.github.com>
* use effective location when building initial path in entity picker
* unit test adjustment
* more unit test adjustments
Co-authored-by: Nick Fitzpatrick <nick@metabase.com>
* Change run to make-run in process-query-for-card
Fixes#44160
There were two levers where a card could change how it was run.
`qp` this is basically `qp/process-query` with some optional userland
flags and constraints marked on the passed in query.
In the case of pivot tables in the
`/pivot/:card-id/query` endpoints it was `qp.pivot/process-query`
'run' this defaults to a streaming response that takes the `qp` param
and `export-format` and passes a query to qp. Most often this is
overriden to not stream the response.
Unfortunately, if `run` is overwritten, then `qp` is thrown away, and
the caller who overrides `run` must determine the correct `qp` to use.
However, when running `process-query-for-dashcard` it is difficult to properly set
`qp` for all of the dashcards without doing card lookups. So [this commit](https://github.com/metabase/metabase/blob/release-x.50.x/src/metabase/query_processor/card.clj#L230-L232) did a check to swap the runner in the presence of a pivot query that calls the **default** runner with a `qp.pivot/run-pivot-query`. The problem is that pulse needs to use a non-default runner, but there's no way for `process-query-for-card` to do both.
This commit changes `run` to `make-run` which raises the abstraction to
a function that produces a runer given a `qp`. This allows
`process-query-for-card` to change the `qp` as needed while keeping the
passed in runner.
There's really three things going on that could be teased apart
further/better.
1. `qp` - this conflates the processor (`qp/process-query` `qp.pivot/run-pivot-query`)
and modifying query (`qp/userland-query`,
`qp/userland-query-with-default-constraints` `(update query :info merge info)`)
2. `make-run` - How the above are processed: (streaming/realized)
* Fix tests
Co-authored-by: Case Nelson <case@metabase.com>
* fix charts crash in safari when dataset timezone is not in iana format (#44193)
* fix charts crash in safari when dataset timezone is not in iana format
* Update Loki Snapshots
* better fix: apply offset instead of finding a timezone with such offset
* remove unused specs
* Update Loki Snapshots
* review
---------
Co-authored-by: Metabase Automation <github-automation@metabase.com>
* snapshots
---------
Co-authored-by: Aleksandr Lesnenko <alxnddr@users.noreply.github.com>
Co-authored-by: Aleksandr Lesnenko <alxnddr@gmail.com>
* adding snippets back into getCollection
* e2e test
---------
Co-authored-by: Nick Fitzpatrick <nick@metabase.com>
Co-authored-by: Sloan Sparger <sloansparger@gmail.com>
* Handle errors when inferring separator of csv
This file
```
"c1","c2"
"a,b,c","d"
```
was failing to upload for a silly reason: when we were checking for
which separator it used, we had a parse error when using a semi-colon
and that blew up the whole pipeline.
* Use cal's helpful test suggestions
Co-authored-by: dpsutton <dan@dpsutton.com>
* certain settings should be decrypted for cache config migration to run (#44048)
* ~ fix test-migrations-for-driver! when rolling back and migrating again
---------
Co-authored-by: Alexander Solovyov <alexander@solovyov.net>
Co-authored-by: Cal Herries <calherries@gmail.com>
* Fix the typo
* Replace `realHover` with the `mouseMove` trigger
* Be explicit when targetting elements by their index
* fix combo chart tooltip e2e specs
* cy.wait as a last resort
---------
Co-authored-by: Nemanja Glumac <31325167+nemanjaglumac@users.noreply.github.com>
Co-authored-by: Aleksandr Lesnenko <alxnddr@gmail.com>
* Remind embedding SDK PR author to use conventional commits
* Only remind PR author when PR title isn't conventional commits
* Add branch for testing
* Fix gh cli command
* Fix getting wrong user login
* Ensure we don't create more than 1 comment
* Improve comment format
* Add more info for non-changelog related type
* Check non-changelog related types
* Update the comment for consistency
* Revert "Add branch for testing"
This reverts commit 4a1be68828.
Co-authored-by: Mahatthana (Kelvin) Nomsawadi <me@bboykelvin.dev>
* Fix stacked redirects to homepage dashboard crashing the app (#43836)
* Add repro test
* Move home selectors to `metabase/selectors/app`
* Prevent unnecessary redirects to home dashboard
* Rename
* Handle navbar home link
* Fix unit tests
* Fix navigating away from an archived dashboard
* Handle sub-root Metabase URLs
* Fix dashboard state sync
---------
Co-authored-by: Anton Kulyk <kuliks.anton@gmail.com>
I'm not sure it's any less hacky in the CLJS but it keeps the TS
wrappers thin.
Context:
- #43799 is the original issue
- #43848 added the hack being removed here
Co-authored-by: Braden Shepherdson <braden@metabase.com>
Co-authored-by: Alexander Polyankin <alexander.polyankin@metabase.com>
This PR simplifies a bit of text used in the cache cron schedule component: we can say "spaces" (which is normal English) not "white spaces" (which is developerese)
Co-authored-by: Raphael Krut-Landau <raphael.kl@gmail.com>
* Initial commit
* unit test, always white on active
* Run Prettier 🙄
* removing yellow color from verified model icon
* linter and type fix
Co-authored-by: Nick Fitzpatrick <nick@metabase.com>
* fix data labels under trend/goal lines (#42928)
* fix data labels under trend/goal lines
* fix fontFamily missing
* update specs
* fix type errors
* snapshots
---------
Co-authored-by: Aleksandr Lesnenko <alxnddr@gmail.com>
* Set min bar height for non-zero bars (#43485)
* set min bar height for non-zero values
* fix y-axis split logic uses transformed dataset instead of actual data
* fix timeline events covers series (#43780)
* make compact formatting consistent across all series (#43472)
* make compact formatting consistent across all series
* fix rebase
* resolve yarn.lock conflicts
* snapshots
---------
Co-authored-by: Emmad Usmani <emmadusmani@berkeley.edu>
* Run smaller smoke test suite for pre-release check (#43633)
* run smaller smoke test suite for pre-release check
* fix run command
* kick ci
---------
Co-authored-by: Ryan Laurie <30528226+iethree@users.noreply.github.com>
Co-authored-by: Ryan Laurie <iethree@gmail.com>
* Automate SDK doc PR
* Prevent auto merge
* Dry run npm publish for testing
* Speed up testing by not building jars
* Fix auto approve step
* disable concurrency so I could test multiple runs in parallel
* Test auto-merge
* Fix auto merge
* Revert "disable concurrency so I could test multiple runs in parallel"
This reverts commit 60e90076eb.
* Revert "Speed up testing by not building jars"
This reverts commit 4e53a54414.
* Revert "Dry run npm publish for testing"
This reverts commit 9b5d761c8d.
* Enable all commented out steps
* Remove unnecessary steps
Co-authored-by: Mahatthana (Kelvin) Nomsawadi <me@bboykelvin.dev>
* Mock that the changelog is different from the one in master
* Apply changelog patch incase the changelog is created from another branch
* Dry run npm publish so I could test
* Fix failed patch
* Speed up testing by not building jar
* Fix adding new changelog to master
* feat(sdk): A placeholder for an actual feature in the changelog
* Mock that the changelog diverges from master
* Revert "Speed up testing by not building jar"
This reverts commit acb80d7465.
* Revert "Dry run npm publish so I could test"
This reverts commit f4e679efd9.
* Revert "Mock that the changelog diverges from master"
This reverts commit 0e77a68e1b.
Co-authored-by: Mahatthana (Kelvin) Nomsawadi <me@bboykelvin.dev>
* Multi Series Dashcards Render with Filters Applied To All Series
Fixes: #39083
Multi-series dashcards now properly render when a subscription is set up with filters that impact the values of each
card in the multi-series card.
* Small refactor for executing dashcards in a pulse context.
- move execute-dashboard-subscription to pulse.util
- this fn now will also execute all card series cards
and put those results into a list on the key :series-results in the dashcard
which is then used in the render multimethod to send the correct data to the javascript context
A small todo was taken care of as well, we can now use :dashboard-subscription in the :context key
* Fix up render fn
The shape of hte data going into the javascript was a bit unclear before. This keeps things a little more
understandable.
Next step is probably to create some Malli schemas inside the js_svg namespace to document and enforce the needed data shapes
* pull :data up a level from :result in the series cards
* Update src/metabase/pulse/render/body.clj
* Remove unneeded let
* set/rename-keys won't work here because we raise the :data key
I've realised a suggested change to just use set/rename-keys won't do what we need. We need instead to raise :data
from within the result key up to the same level (and dissoc result).
I've renamed the helper function to make its purpose a bit clearer
---------
Co-authored-by: adam-james <21064735+adam-james-v@users.noreply.github.com>
Co-authored-by: Ngoc Khuat <qn.khuat@gmail.com>
Liquibase changed their boolean type in MySQL from `bit(1)` to
`tinyint(4)` in version 4.25.1. Our JDBC driver does not recognize these
as booleans, so we needed to migrate them to `bit(1)`s.
As discussed [here](#36964), we
changed all existing `boolean` types that were in the
`001_update_migrations.yml` but not the SQL initialization file.
For new installations, this works: things in the SQL initialization file
get created with the `bit(1)` type.
However, for existing installations, there's a potential issue. Say I'm
on v42 and am upgrading to v49. In v43, a new `boolean` was added.
In this case, I'll get the `boolean` from the liquibase migration rather
than from the SQL initialization file, and it need to be changed to a
`bit(1)`.
I installed Metabase v41 with MySQL, migrated the database, and then
installed Metabase v49 and migrated again. I made a list of all the
columns that had the type `tinyint`:
```
mysql> SELECT TABLE_SCHEMA, TABLE_NAME, COLUMN_NAME, COLUMN_TYPE, COLUMN_DEFAULT, IS_NULLABLE FROM INFORMATION_SCHEMA.COLUMNS WHERE COLUMN_TYPE = 'tinyint' AND TABLE_SCHEMA='metabase_test';
+---------------+------------------------------+-------------------+-------------+----------------+-------------+
| TABLE_SCHEMA | TABLE_NAME | COLUMN_NAME | COLUMN_TYPE | COLUMN_DEFAULT | IS_NULLABLE |
+---------------+------------------------------+-------------------+-------------+----------------+-------------+
| metabase_test | core_user | is_datasetnewb | tinyint | 1 | NO |
| metabase_test | metabase_field | database_required | tinyint | 0 | NO |
| metabase_test | metabase_fieldvalues | has_more_values | tinyint | 0 | YES |
| metabase_test | permissions_group_membership | is_group_manager | tinyint | 0 | NO |
| metabase_test | persisted_info | active | tinyint | 0 | NO |
| metabase_test | report_card | dataset | tinyint | 0 | NO |
| metabase_test | timeline | archived | tinyint | 0 | NO |
| metabase_test | timeline | default | tinyint | 0 | NO |
| metabase_test | timeline_event | archived | tinyint | 0 | NO |
| metabase_test | timeline_event | time_matters | tinyint | NULL | NO |
+---------------+------------------------------+-------------------+-------------+----------------+-------------+
10 rows in set (0.01 sec)
```
Then wrote migrations. For each column, we:
- turn it into a `bit(1)`,
- re-set the previously existing default value, and
- re-add the NOT NULL constraint, if applicable.
---------
Co-authored-by: SakuragiYoshimasa <ysrhsp@outlook.com>
On the Browse models page we have a filter icon which opens a popover with a switch that toggles "Show only verified models". With this change, the icon is not rendered if there are no verified models at all.
Co-authored-by: Raphael Krut-Landau <raphael.kl@gmail.com>
* Fix column scaling ("Multiply by a number") bugs (#43668)
* return empty string so non-empty labels do not get hidden by empty labels
* fix function name
* apply scaling in transformed dataset
* add missing option to OptionsType
* fix double scaling
* add column scaling to waterfall transformed dataset
* add column value scaling to dataset transform
* fix stacked bar segment labels not responding to column scaling
* fix tooltip values displayed
* since scaling as added to transformedDataset, use transformedDataset for formatters
* revert changes to display percentage instead of absolute value
* fix stacked data labels
* add loki variant for custom y-axis range with column scaling
* fix stack segmented label formatters compact calculation
* scale original dataset and use that instead of modifying transformedDataset
* update loki snapshots
* fix scatter plot dataset scaling
* fix tooltip values being double scaled
* add E2E test to validate behavior
* add helper for not duplicating the column scaling
* port multiple y-axis metrics with column scaling fixes to RowChart
* add waterfall and scatterplot loki variants
* fix type errors and failing unit tests
* fix failing unit tests
* update loki snapshots
* update loki snapshots
---------
Co-authored-by: Jesse Devaney <22608765+JesseSDevaney@users.noreply.github.com>
* Fix OSS tests in CI (#43660)
* fix OSS tests in CI
* update tests that went on vacation for a while
* do safer cypress things
* better helper name
# Conflicts:
# e2e/support/helpers/e2e-ui-elements-helpers.js
# e2e/test/scenarios/collections/instance-analytics.cy.spec.js
# e2e/test/scenarios/models/reproductions.cy.spec.js
# e2e/test/scenarios/permissions/permissions-reproductions.cy.spec.js
# e2e/test/scenarios/question-reproductions/reproductions-2.cy.spec.js
# e2e/test/scenarios/question/new.cy.spec.js
* manual backport to changed spec files
* gross manual backporting
* fix conflict
---------
Co-authored-by: Ryan Laurie <30528226+iethree@users.noreply.github.com>
Co-authored-by: Ryan Laurie <iethree@gmail.com>
* update translations for v50
* add Danish
* update translations for v50 release
* alphabetical order is nice
Co-authored-by: Ryan Laurie <30528226+iethree@users.noreply.github.com>
* Check git tag for an exact match
* Set npm publish to dry run, so I can test
* Uses new github automation account for creating a PR
* Remove unnecessary step
* Automatically set `no-backport` on the SDK doc version bump PR
* Fix step to edit PR with correct permission
* Remove npm dry run after finished testing
Co-authored-by: Mahatthana (Kelvin) Nomsawadi <me@bboykelvin.dev>
* Fix flaky test
* Fix the flake for real
* Fix stress test failing because of mismatched height
* Review: Move viewport setting to shared config
Co-authored-by: Mahatthana (Kelvin) Nomsawadi <me@bboykelvin.dev>
* Revert "Revert "minimal support for multiple values for Access-Control-Allow-Origin (#42888)" (#43125)"
This reverts commit a4312230a8.
* refactor: extracted parse-approved-origins + supported "*" without special case
* fix: parse-approved-origins won't break on multiple spaces in a row
* handles invalid urls in the origins + some formatting
* log instead of throwing in parse-url + simplify the logic in parse-approved-origins
* simplify parse-approved-origins logic by splitting on "at least one space"
* remove extra boolean call
* approved-origins -> approved-origins-raw to better reflect that it's a raw string
* swap the order of the arguments to `is` in the tests
* indentation 🙃
Co-authored-by: Nicolò Pretto <info@npretto.com>
(When scrolling through CI logs I always do a double-take when I see a
huge stack trace with "Just an error". I think this is a little
clearer?)
Co-authored-by: Tim Macdonald <tim@metabase.com>
On the Browse models page we have a filter icon which opens a popover with a switch that toggles "Show only verified models". With this change, the icon is not rendered if there are no verified models at all.
Co-authored-by: Raphael Krut-Landau <raphael.kl@gmail.com>
* move parameters and dashboard reducers to the common ones as they're used everywhere
* convert reducers-{common,main,public}.js to typescript, done with --no-verify
* convert reducers to named exports
Co-authored-by: Nicolò Pretto <info@npretto.com>
* Do not use asReturned to determine columns for extractions and combine columns
* Add reproduction for #43226 for combine columns
* Add reproduction for #43226 for extract columns
* Remove commented out code
* Only render extraction shortcut when there are extractions
* Only render combination shortcut when there are combinations
* Use hasCombinations from CombineColumns
* Add hasExtractions from ExtractColumn
* Use appendStageIfAggregated over Lib.asReturned
* Only show combine column shortcut when there are two or more columns to be combined
* Lift appendStageIfAggregated to top-level drill
* Add test for extraction on table with just breakout
* Add test for combinations on table with just breakouts
* Disable the 2-column requirement for combinations
* Remove Lib.asReturned in appendStageIfAggregated
* Remove + 1 to stageIndex in appendStageIfAggregated
* Check for empty breakouts too in as-returned
* Switch back to Lib.asReturned
* Fix test for combine column on breakouts
* Reference the correct query and stageIndex
* add CLJS unit tests for as-returned with only aggs, breakouts
---------
Co-authored-by: Romeo Van Snick <romeo@romeovansnick.be>
Co-authored-by: Braden Shepherdson <braden@metabase.com>
* remove unused variable
* conditionally render labels sparsely if large amount of data points
* update loki snapshots
* refactor data label formatter conditions
* setup intelligent label plotting
* handle the case when there are no labels
* add sparse data label rendering to bar charts
* do not compute if no labels need to be shown or hidden
* add sparse data labeling to waterfall charts
* adjustment for large series waterfall labels
* revert change
* refactor name
* add sparse labels to stacked bar and area
* improve scaleFactor
* fix type errors
* remove unused labelFormatter
* update snapshots
* patch ECharts
- labels with no text, i.e. "", were still being considered as plot-able labels which was causing labels with real text "asdfec.." to be hidden because of overlap.
- This patch makes it so that labels with no text, i.e. "", are not considered to be candidates for overlap checking.
* update loki snapshots
* only loop over dataset once for getWaterfallChartDataDensity
* improve performance of chart data density calculations
* update loki snapshots
* update E2E spec
* update E2E spec
* increase cartesian label density allotment
* update loki snapshots
* update loki snapshots
* comment out node_modules cache since it cannot be skipped by commit and is breaking CI
* fix type errors
* Reset node modules if patches are changed
* refactor type naming
---------
Co-authored-by: Jesse Devaney <22608765+JesseSDevaney@users.noreply.github.com>
Co-authored-by: Uladzimir Havenchyk <uladzimir.dev@gmail.com>
Co-authored-by: Uladzimir Havenchyk <125459446+uladzimirdev@users.noreply.github.com>
* add hook to check token features
* add cache upsell to performance page
* upsell cards are white
* fix rebase
# Conflicts:
# frontend/src/metabase/admin/performance/components/StrategyEditorForDatabases.tsx
Co-authored-by: Ryan Laurie <30528226+iethree@users.noreply.github.com>
* Auto tag embedding team for PR review + dry run npm publish
* Install changelog generation package
* Add the initial changelog
* feat(sdk): This is an empty commit. For testing purposes.
* Update SDk build script to include changelog
* Check if a tag exists
* Ensure changelog is in the released package
* Fix workflow
* Remove unnecessary step
* Ensure we can have access to release utils script when checking out master
* Fix the step to create a PR updating readme and changelog
* Fix changelog generation
* Fix couldn't download changelog
* Clean up steps
Co-authored-by: Mahatthana (Kelvin) Nomsawadi <me@bboykelvin.dev>
Models are treated as opaque here, but for a saved question we expose
filters on the query before its final aggregation step.
This drops the aggregation clauses and gets the possible values for the
column of interest.
Fixes#43154.
Co-authored-by: Braden Shepherdson <braden@metabase.com>
* refactor scatter plot to separate code path (#43032)
* refactor scatter plot model to separate function
* (2/3) refactor scatter plot option to separate function (#43033)
* use getScatterPlotOption
* use buildEChartsScatterSeries directly
* remove hoveredSeriesDataKey from scatter
* (3/3) improve chart model types (#43034)
* create WaterfallChartModel type
* create ScatterPlotModel type
* rename BaseCartesianChartModel to CartesianChartModel
* simplify chartModel creation in hook
* support stacked series values labels (#42803)
* drop comment
---------
Co-authored-by: Emmad Usmani <emmadusmani@berkeley.edu>
Co-authored-by: Aleksandr Lesnenko <alxnddr@users.noreply.github.com>
Co-authored-by: Aleksandr Lesnenko <alxnddr@gmail.com>
* Multiple Measures Requires Additional Header Row in Pivot Exports
Mulitple measures can be set in our pivots. We indicate these properly in the fronted with a header row that shows
which measure is reprsented in that particular column.
The header builder fn in the pivot export post processing didn't handle this case, but after this PR, it correctly
adds the final header when necessary.
* Fix up header tests
* Remove comment form; the tests are derived from that work already
Co-authored-by: adam-james <21064735+adam-james-v@users.noreply.github.com>
These databases previously had impoverished support for regular
expressions. The `:regex` driver feature requires advanced,
"Perl-compatible" regular expressions that can do things like lookahead
assertions.
- MySQL supports this from 8.0.4; we recommend 8.0.33+.
- MariaDB supports this from 10.0.5; we recommend 10.4+.
Fixes#43180.
Co-authored-by: Braden Shepherdson <braden@metabase.com>
* [Manual backport] Backfill `:effective-type` for coerced fields (#43144) (#43357)
- Update how base type and effective type is added in query-method :mbql/query.
- Base type and effective type is added also during the conversion of query-method :query.
- Adds warning suppression for expressions to places where normalize is
called something that does not conform schme (eg. ->pMBQL :dispatch-type/map called
in query-method :query).
---------
Co-authored-by: lbrdnk <lbrdnk@users.noreply.github.com>
* Single metadata endpoint for a dashboard (#43283)
* Fix backport
* Backport dependent-metadata changes
* Fix lint
---------
Co-authored-by: Case Nelson <case@metabase.com>
Co-authored-by: Alexander Polyankin <alexander.polyankin@metabase.com>
* pulls out test changes from react 18 / rtl 15 refactor that are comptable with react 17 / rtl 12
* impls pr feedback
Co-authored-by: Sloan Sparger <sloansparger@users.noreply.github.com>
* Keep a single null dimension value
* Add Loki test
* Remove Loki test
* Throw an error if dataset is empty after filtering
* Extract error message into a constant
Co-authored-by: Anton Kulyk <kuliks.anton@gmail.com>
* Update expression suggestion design (#43232)
* Use sentence case for suggestion headers
* Remove suggestion group dividers
* Highlight suggestions when mousing over them
* Remove unused hoverparent
* Remove custom highlighting for icons
* Allow passing color to metadata info icon
* Make suggestions icons have a consisten color
* Update documentation suggestion style
* Add margin to suggestion sections
* Reorder suggestion sections
* Add test for mouse over highlight on suggestions
* Fix test for suggestion footers
* Remove debugging true
* Rename 'Most used functions/aggregations' to 'Common functions/aggregations'
* Highlight suggestion on mouse move
* Only call onHighlightSuggestion if it is not already highlighted
* Use arrow functions
* Fix test for suggestion highlight
* Add delay to fix broken test
* Use findAllByRole instead of wait
---------
Co-authored-by: Romeo Van Snick <romeo@romeovansnick.be>
* Use internal Link for relative and same-origin URLs
* Update frontend/test/metabase/lib/formatting.unit.spec.js
* Update frontend/test/metabase/lib/formatting.unit.spec.js
---------
Co-authored-by: Anton Kulyk <kuliks.anton@gmail.com>
Co-authored-by: Uladzimir Havenchyk <125459446+uladzimirdev@users.noreply.github.com>
* entityPicker in technicolor
* more precise import
* add unit test
# Conflicts:
# frontend/src/metabase/lib/icon.ts
Co-authored-by: Ryan Laurie <iethree@gmail.com>
* Do not render 'Add column' shortcut when viewing a raw table
* Do not show 'Add column' shortcut when a query has aggregations
* List shortcut guards out of component
* Make 'Add column' icon sticky in the header
* Add test for hiding the column shortcut
* Remove Number wrapper
* Move padding to a constant
* Use actions directly to determine if there are any shortcuts
* Make sure there are expressionable columns for the combine columns shortcut
* Make sure there are extractable columns for the extract column shortcut
* Remove padding when not overflowing
* Do not shadow query in popover components
* Allow combine and extract for raw tables
* Add test for combining summarization columns
* Add test for combining summarization columns
* Use a more precise way of calculating the offset for the 'Add column' button
* Handle null question
* Change to the outline button
* Use actual column widths to calculate total width
* Handle empty mode
* Remove extra measure call
* Move shortcutColumn to separate method and avoid looping over all available actions when possible
Co-authored-by: Romeo Van Snick <romeo@romeovansnick.be>
* improve models table performance
* add loading state for large sets of models
* less icon padding
Co-authored-by: Ryan Laurie <30528226+iethree@users.noreply.github.com>
* fix trend charts erroring on latest value being null
* refactor previousRowIndex calculation
* handle errors gracefully
* refactor error messages
* update computeTrend unit tests
* add error handling integration test
* add E2E test to check that errors are gracefully handled
* add integration test to enforce the fix
* fix static viz
* display more meaningful error on static-viz trend charts
* move error handling out of trend chart and fix error handling in Visualization.tsx
* update compute.unit.spec.js
* update SmartScalar.unit.spec.js
* use separate error handler for ErrorBoundary errors in Visualization.jsx
* update E2E test
* update error handling
* update compute.unit.spec.js
* fix react state change error in render function
* show generic error message for ErrorBoundary caught errors
* revert ErrorBoundary changes
* remove generic error border for visualizations
Co-authored-by: Jesse Devaney <22608765+JesseSDevaney@users.noreply.github.com>
* Fix missing React key prop
* Fix missing React key prop
* Fix order of conditions and make the code easier to follow
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
* Generate different shouldShowItem for different pickers
* Add repro for #42957
* Update test to use models instead of metrics
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
* highlight hovered timeline event
* add e2e test
* use TIMELINE_EVENT_DATA_NAME constant
* add test case for clicking icon
Co-authored-by: Emmad Usmani <emmadusmani@berkeley.edu>
* Restore default styles for subtle button (#42854)
* fix(sdk): Use theme font in charts and tooltips (#42855)
* Allow charts font override in SDK
* Fix default not configured font
(cherry picked from commit 0278b8d221)
* fix(sdk): Various fixes for InteractiveQuestion theming (#42932)
* Fixes for InteractiveQuestion theming
* Update readme
* Fix for unset filter theme color
* Fix unit tests
* Fix unit tests
(cherry picked from commit a3c3193474)
# Conflicts:
# enterprise/frontend/src/embedding-sdk/lib/theme/embedding-color-palette.ts
# enterprise/frontend/src/embedding-sdk/lib/theme/get-embedding-theme.ts
# frontend/src/metabase/ui/utils/colors.ts
These were added in #42602 but I neglected to update the changelog.
Their defaults should serve for most drivers.
Co-authored-by: Braden Shepherdson <braden@metabase.com>
* Extract EmailAdmin and add tests for it
* Rename EmailAdmin to AdminEmail
* Remove redundant async
* Add a test case for no email
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
* hide symbols for charts with too many of them
* update loki
* prevent series dots showing up after manually disabling another series
* update loki snapshots
* refactor conditional
* update loki snapshots
Co-authored-by: Jesse Devaney <22608765+JesseSDevaney@users.noreply.github.com>
This is a bit painful. I merged this change, but realized we need to
backport the fix to v49. However:
- we don't want to have two versions of the migration (one with a v49
id, one with a v50 id) because then if someone upgrades to 50, then
downgrades to 49, the `rollback` will run and change the type back,
leading to a bug.
- we don't want to push a v51 changeSet ID to v49 or v50, because we
give the user a helpful notice when their database needs a downgrade.
We do this by checking for the latest *executed* migration in the
database and comparing it to the latest migration that Liquibase knows
about, and making sure the executed isn't bigger than the known (e.g.
you can't have executed a v51 migration if it isn't in the local
migration yaml). That would all work fine, except that then we want to
tell you how to downgrade your database, and we use the latest-executed
version for that. So if, for example, someone upgraded from 48 to 49 and
got a v51 changeset, then downgraded back to 48, they would get an error
telling them to run the *v51* jar to roll back their DB.
In this case though, I think it's fine to just move the migration around
to v49, then we can backport it to 49 and 50.
* [MBQL lib] Reject broken uses of `offset` in expressions, filters (#42662)
- Using `offset` in custom expressions is supported only when there is
an order-by defined on that stage.
- Using `offset` in custom filters is not supported at all, currently.
* Support `Offset()` in custom columns [frontend] (#42326)
* Only nest expressions referenced in breakouts or aggregations
* Support Offset() as expression with no breakouts
* Test fixes 🔧
* Oracle test update
* Improved Oracle test
* Test update 🔧
* Fix busted test
* Add naive support for Offset() in expressions
* Introduce MBQLClauseFunctionReturnType
* Add "window" to MBQLClauseFunctionReturnType and use it for the offset function
* Handle offset type inference
* Remove unused export
* Use "expression" instead of "window" return type
- Rename identifiers in isCompatible
- Make isCompatible accept a clause object instead of just the type
- Handle offset as a special case in isCompatible
* Use any type
* Rename expectedArgumentType to expectedType
* Format code
* Sort types
* Do not suggest offset function in filters
* Fix offset not working in case
* Revert "Do not suggest offset function in filters"
This reverts commit e63790b780.
* Fix order of adjustments
---------
* Disable offsets in filters expressions
* Group existing aggregations-specific tests
* Remove repro for a closed issue
* Use findByText instead of contains
* Add a test for filter expressions
* Add a test for aggregation expressions suggestions
* Disable offsets in filters expressions (#42755)
* Add a test for custom column suggestions
* Add "typing" to enterCustomColumnDetails
* Use enterCustomColumnDetails, improve assertions
* Add more assertions
* Optimize queries
* Add typing for offset expressions
* Add a repro for metabase#42764
* Add a TODO
* Add a TODO
* Add a TODO
* Use NumericLiteral
* Post-merge fixes
* Update test
* Add tests for other custom expressions
* Test drills
* Format code
* Update test name
* Add an assertion
* Add assertions for the preview
* Unskip fixed issue
---------
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
Co-authored-by: Braden Shepherdson <braden@metabase.com>
Co-authored-by: Cam Saul <github@camsaul.com>
Co-authored-by: Cam Saul <1455846+camsaul@users.noreply.github.com>
* Set scrollToLastColumn ui control from column extract action
* Scroll to last column when the uiControl is set
* Scroll to last column when combining a column
* Scroll to last column when combining a column via the + shortcut
* Scroll to last column when extracting a column via the + shortcut
* Remove all references to settingsSyncOptions
* Check that the column is added at the end
* Disable scrollToLastColumn once used
* Add test that verifies scrollToLastColumn ui control has been reset
Co-authored-by: Romeo Van Snick <romeo@romeovansnick.be>
2024-05-17 15:07:00 +00:00
4810 changed files with 507293 additions and 222318 deletions
malli.core/explainer{:message"Use metabase.util.malli.registry/explainer instead of malli.core/explainer"}
me.flowthing.pp/pprint{:message"Use metabase.util.log instead of me.flowthing.pp/pprint"}
medley.core/random-uuid{:message"Use clojure.core/random-uuid instead of medley.core/random-uuid"}
metabase.driver/database-supports?{:message"Use metabase.driver.util/supports? instead of metabase.driver/database-supports?"}
metabase.lib.equality/find-column-indexes-for-refs{:message"Use lib.equality/closest-matches-in-metadata or lib.equality/find-closest-matches-for-refs instead of lib.equality/find-column-indexes-for-refs"}
metabase.test/with-temp*{:message"Use mt/with-temp instead of mt/with-temp*"}
toucan2.tools.with-temp{:message"Use mt/with-temp instead of t2.with-temp/with-temp"}}
@@ -114,6 +115,7 @@
:discouraged-namespace
{camel-snake-kebab.core{:message"CSK is not Turkish-safe, use the versions in metabase.util instead."}
clojure.tools.logging{:message"Use metabase.util.log instead of clojure.tools.logging directly"}
grouper.core{:message"Use metabase.util.grouper instead of grouper.core"}
honeysql.core{:message"Use Honey SQL 2. Honey SQL 1 is removed as a dependency in Metabase 49+."}
honeysql.format{:message"Use Honey SQL 2. Honey SQL 1 is removed as a dependency in Metabase 49+."}
honeysql.helpers{:message"Use Honey SQL 2. Honey SQL 1 is removed as a dependency in Metabase 49+."}
@@ -135,7 +137,7 @@
;; namespace(s). This is mostly a temporary placeholder until we go in and create module namespaces, which means
;; you should go create one.
;;
;; 2. A set of namespace symbols. All namespaces in other modules will only be allowed ;; to use namespaces from
;; 2. A set of namespace symbols. All namespaces in other modules will only be allowed to use namespaces from
;; this set. Ideally this set should only have one namespace, but restricting it to a set of several is still
;; better than `:any`.
;;
@@ -162,6 +164,7 @@
metabase.async#{metabase.async.streaming-response
metabase.async.streaming-response.thread-pool
metabase.async.util}; TODO -- consolidate these into a real API namespace.
metabase.audit#{metabase.audit}
metabase.bootstrap#{metabase.bootstrap}
metabase.cmd#{}; there are no namespaces here since you shouldn't be using this in any other module.
comment='@${{ github.event.pull_request.user.login }} You have modified embedding SDK code. Please make sure the PR title follows [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) style.
Here are all the supported types that will be shown in the changelog:
- `feat`
- `fix`
- `perf`
These types `docs`, `style`, `refactor`, `test`, `build`, and `ci` are also encouraged for non-changelog related tasks.
Please also make sure to include `sdk` scope, otherwise, the changelog will not include this task.
run:echo "Tag '${{ env.tag }}' already exists. If you expect to run this workflow with the same tag, please remove the tag first and rerun this workflow again."
if [ $(git tag -l | grep $tag$) ]
then
echo "::error::Tag '${{ env.tag }}' already exists. If you expect to run this workflow with the same tag, please remove the tag first and rerun this workflow again."
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.