This was a huge footgun.
We were accidentally limiting *ALL* queries run in our transducer to
1000 results. This included the query to fetch permissions, so when we
checked permissions we were checking a small percentage of your *actual*
permissions (e.g. 1000 / ~6300 for me on stats).
Co-authored-by: John Swanson <john.swanson@metabase.com>
This comes in two parts.
**Rebuilt query logic**
I rewrote the query exec logic to use one `future` and one `a/go` block,
reporting results to a `promise`. This is easier to follow and doesn't leak
threads like the original implementation with two `future`s.
The single future is guaranteed to exit. It blocks on the BigQuery call.
That will either complete or throw an exception. Either the exception or
the BigQuery result is sent to the promise, and then the future is
complete. The only reference to that future is overwritten after the `promise`
resolves, so the `future`, its closure, and its thread don't persist longer than needed.
**Return a reducible result**
Previously the results were returned as a lazy sequence, but that brings
a lot of `seq` overhead, and is a risk of retaining the entire result
set in memory until the query is all done. Above all, it doesn't support
`reduced?` to explicitly drop any further results.
The `thunk` function style of `qp.reducible/reducible-rows` doesn't fit
neatly here, so I implemented the reducible logic directly on top of the
`Iterable` BigQuery results.
**Verifying**
This is a bit tricky. I've been testing it like this:
- Add a BigQuery database to my instance
- Create a saved question and note its ID (say, 123)
- Create an API key in (EE) Metabase and save it in an env var
- Install the `siege` load testing tool
- Install VisualVM and monitor my Clojure REPL process's thread count
- `siege --concurrent=15 --time=30s --header="x-api-key: $API_KEY" "http://localhost:3000/api/card/123/query POST"`
That will create a bunch of new threads while it's getting called, but
within a minute of the last queries they'll gradually drop out of the
thread pools and it will return to the baseline number of threads.
Co-authored-by: Braden Shepherdson <braden@metabase.com>
* Implement transfer queues for performing async backfills (#45408)
* Refer to old position of helper
* It's private
---------
Co-authored-by: Chris Truter <crisptrutski@users.noreply.github.com>
Co-authored-by: Chris Truter <chris@metabase.com>
* Fix - Ellipsis tooltip cut off/not visible in GUI editor (#45030)
* Fix#39771
* Add repro for #39771
* Add explanatory comment
* Fix z-index
* Use Mantine Tooltip in Ellipsified to fix z-index issues
* Revert "Fix z-index"
This reverts commit 951e3dca86.
* Revert "Fix #39771"
This reverts commit 4165da5022.
* Update repro to new tooltip
* Update assertions
* Try to fix flake
* Try to fix flake
* Disable Replay to see if it affects tests
* Revert "Disable Replay to see if it affects tests"
This reverts commit e44d9a0a6f.
* Revert "Try to fix flake"
This reverts commit e2f23eb597.
* Revert "Try to fix flake"
This reverts commit 0df6554b38.
* Update assertion and add an explanatory comment
---------
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
* funnel chart handle duplicated steps (#45333)
* handle duplicated funnel chart steps
* fix import
* Update Loki Snapshots
* drop dynamic viz stories for funnel since the setup code is not in the release branch
---------
Co-authored-by: Aleksandr Lesnenko <alxnddr@users.noreply.github.com>
Co-authored-by: Aleksandr Lesnenko <alxnddr@gmail.com>
* Dashboard Paramter Value Now Works in Preview Embed
Fixes: #44962
Prior, a common util fn used both in the embed and preview_embed endpoints was causing the preview_embed to error out
when fetching dashboard parameter values because it was checking if embedding was enabled. This check is needed for
the normal embedding endpoints, but we want the preview embed endpoints to work without needing to publish the
dashboard first.
* pass proper args
* put args back in proper order.
oops
* change form back to previous, but keep preview check
* autocorrect mistake? Or perhaps a silly typo?
* change one test to show that the preview flag is working
Didn't create a whole new test for this flag as it is provable just by adjusting this test to not bother enabling
embedding on the dashboard.
* improve the docstring
Co-authored-by: adam-james <21064735+adam-james-v@users.noreply.github.com>
- getAllCollectionItemNames was breaking the chain in Cypress, so assertions did not know how to wait for DOM to be updated
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
* Log changes when they make sense + async coll perm-graph revision
* - Make revision-number consistent
Decouples the creation of a revision number from the (potentially
expensive) writing of before and after for the coll perms-graph
revision.
* get id from the perms revision entity
* Move the revision id creation into the transaction
The revision id creation should only happen iff the perm updates went
through successfully.
* write a test that checks the incomplete revision
* remove sleep from tests
Co-authored-by: bryan <bryan.maass@gmail.com>
* Static Viz and Exports/Downloads Respect Dashcard Viz Settings (#45164)
* start adding some util fns to download-exports-test ns
* wip improving test util fns
* add subscription attachment util fn
* add tests to catch download/export differences
* viz-settings from card and dashcard merged properly
* revert these changes which probably aren't needed
* split test: downloads can be parallel, email attachments cannot
* normalize viz-settings no matter their origin
* Update test/metabase/api/downloads_exports_test.clj
Co-authored-by: Alexander Solovyov <alexander@solovyov.net>
* Incorporate some review feedback
* rendered table cards also respect column names
* Clean up table render tests and remove unhelpful render test utils
The big removal of code in the render test utils namespace is a clean up task that could've happened sooner, but this
table test adjustment is the perfect opportunity to clean up.
I had made the test utils before I truly knew how best to use `mt/with-temp`, so it had a lot of poorly implemented
'generators' that were doing a poor job of creating mock data for the rendering functions anyway.
So, reworking a few table tests here to just use a temp card is way more inline with the rest of the codebase, and
actually improves confidence that the tests are testing on well-formed data to begin with.
---------
Co-authored-by: Alexander Solovyov <alexander@solovyov.net>
* Use pre-51 method for getting attachments
---------
Co-authored-by: adam-james <21064735+adam-james-v@users.noreply.github.com>
Co-authored-by: Alexander Solovyov <alexander@solovyov.net>
Co-authored-by: Adam James <adam.vermeer2@gmail.com>
This introduces a new metadata on commands, `:requires-init`. If set,
we'll run `metabase.core/init!` before running the command. `init!` will
initialize the database and run our normal startup, though it won't
actually start the Metabase server.
This allows you to use
```
env MB_CONFIG_FILE_PATH=... java -jar metabase.jar import /my/metabase/export
```
We'll run the normal init process (creating the internal user, loading
from the config file, etc), then run the command.
On my machine this adds about 2-3 seconds to the time it takes to run an
`import`.
Co-authored-by: John Swanson <john.swanson@metabase.com>
* Add a test verifying that fonts are loaded
* Verify font is mentioned in HTML
* Bring back working assertions
* Add wait
* Disable speedy
* Disable CSP and emotion cache
* Revert "Disable CSP and emotion cache"
This reverts commit 573e99bfa4.
* Revert "Disable speedy"
This reverts commit 6f9962b869.
* Don't use local fonts
* Revert "Don't use local fonts"
This reverts commit 17c58aa73d.
* Force Lato
* Fix storybook file name
* ESLint
* Add more !important
* Revert !important
* Add font-family assertion
* Move test
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
* perf: pass uuid to BE to enable metadata caching during fetching dashboard
* fixup! perf: pass uuid to BE to enable metadata caching during fetching dashboard
* Add wildcard on interceptions
* Add tests
Co-authored-by: Uladzimir Havenchyk <125459446+uladzimirdev@users.noreply.github.com>
A bunch of small improvements, notably a timeout and support for MSSQL square bracket quotes.
Co-authored-by: Chris Truter <crisptrutski@users.noreply.github.com>
Before
====
```
2024-07-04 15:54:39,084 INFO metabase.server :: Shutting Down Embedded Jetty Webserver
Exception in thread "Thread-9" java.util.IllegalFormatConversionException: f != java.lang.Long
2024-07-04 15:54:39,122 WARN db.liquibase :: (#object[liquibase.Liquibase 0x2b7e6614 liquibase.Liquibase@2b7e6614])
at java.base/java.util.Formatter$FormatSpecifier.failConversion(Unknown Source)
at java.base/java.util.Formatter$FormatSpecifier.printFloat(Unknown Source)
at java.base/java.util.Formatter$FormatSpecifier.print(Unknown Source)
at java.base/java.util.Formatter.format(Unknown Source)
at java.base/java.util.Formatter.format(Unknown Source)
at java.base/java.lang.String.format(Unknown Source)
at clojure.core$format.invokeStatic(core.clj:5770)
at clojure.core$format.doInvoke(core.clj:5764)
at clojure.lang.RestFn.invoke(RestFn.java:423)
at metabase.db.liquibase$wait_for_all_locks.invokeStatic(liquibase.clj:258)
at metabase.db.liquibase$wait_for_all_locks.invoke(liquibase.clj:251)
at metabase.db.setup$release_migration_locks_BANG_.invokeStatic(setup.clj:175)
at metabase.db.setup$release_migration_locks_BANG_.invoke(setup.clj:170)
at metabase.db$release_migration_locks_BANG_.invokeStatic(db.clj:91)
at metabase.db$release_migration_locks_BANG_.invoke(db.clj:86)
at metabase.core$destroy_BANG_.invokeStatic(core.clj:90)
at metabase.core$destroy_BANG_.invoke(core.clj:81)
at clojure.lang.AFn.run(AFn.java:22)
at java.base/java.lang.Thread.run(Unknown Source)
```
And now it will correctly log
```
2024-07-09 16:04:28,345 INFO db.liquibase :: Waiting for migration lock(s) to be released (max 0.4 secs)
```
Why no tests?
====
There _is_ a test that would have caught this. And in fact running at
the repl will catch it:
```clojure
liquibase=> (do (require 'clojure.test 'metabase.db.liquibase-test)
(clojure.test/run-test metabase.db.liquibase-test/wait-for-all-locks-test))
Testing metabase.db.liquibase-test
ERROR in (wait-for-all-locks-test) (Formatter.java:4515)
:h2
Will timeout if a lock is not released
expected: (= :timed-out (liquibase/wait-for-all-locks sleep-ms timeout-ms))
actual: java.util.IllegalFormatConversionException: f != clojure.lang.Ratio
```
But this _passes_ in CI:
```
❯ clj -X:dev:test :only metabase.db.liquibase-test/wait-for-all-locks-test
Finding tests took 23.5 s.
Running 1 tests
Running before-run hooks...
Ran 1 tests in 1.461 seconds
3 assertions, 0 failures, 0 errors.
{:test 1, :pass 3, :fail 0, :error 0, :type :summary, :duration 1461.175042, :single-threaded 1}
Ran 0 tests in parallel, 1 single-threaded.
Finding and running tests took 25.0 s.
All tests passed.
Running after-run hooks...
```
And that's because the log level in testing is FATAL and this is an INFO
level log.
Discussed ways to improve this like adding a null logger during CI but
we aren't ready to increase CI times for that just yet.
Co-authored-by: dpsutton <dan@dpsutton.com>
* Use Mantine Modal for QB Newb disclaimer
* Add repro for #44754
* Use theme spacing units
* Tweak the design slightly
* Fix E2E test
Co-authored-by: Nemanja Glumac <31325167+nemanjaglumac@users.noreply.github.com>
* fix combined cards crash when some cards have invalid viz settings
* fix mock column was not metric
* ensure combined scalar column has proper type to handle the case when the card returns empty results
Co-authored-by: Aleksandr Lesnenko <alxnddr@users.noreply.github.com>
This PR also specifies the 'filled' variant for the question duplication
modal, since the previous default 'filled' variant for submit button was
added to make that specific button filled.
Co-authored-by: Raphael Krut-Landau <raphael.kl@gmail.com>
* Split unrelated changes into separate PR
* Fix test since user perms set reset fix split into other PR
* Fix syntax error
* Test fixes 🔧
* Test fix??? 🔧
* Test fix 🔧
Co-authored-by: Cam Saul <1455846+camsaul@users.noreply.github.com>
* fix: param-fields when native model is aggregated
Fixes#42829
The param-field lookup required result_metadata to match
filterable-columns in order to find parameter targets. This adds a
fallback where if the param field is missing from result_metadata,
then we look at filterable-columns.
* Unskip test
Co-authored-by: Case Nelson <case@metabase.com>
* Repro #37380 - Users with no self-service access may update pivot tables and leave them in a non-working state (#43437)
* Fix crash and remove redundant casting
* Add repro for metabase#37380
* Be extra-safe with an empty array
* [repro] Add e2e repro for Zoom In on pivot tables row headers (#45066)
This works on master, but this reproduction will prevent #38265 from
reoccurring.
* Repro #39504: Pivot table disappears (#45143)
---------
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
Co-authored-by: Braden Shepherdson <braden@metabase.com>
* E2E repro for "Editing Column description of Model is confusing if it contains aggregations" (#45113)
* Fix checks and tests
* Fix checks and tests
* Fix conflict
---------
Co-authored-by: Alexander Polyankin <alexander.polyankin@metabase.com>
* Refine the reproduction for 44794
* Move skip to top-level again
* Remove superflouous click
* Verify wether or not the recents tab exists and is open
* Make the test description more descriptive
* Move question details to a variable
---------
Co-authored-by: Romeo Van Snick <romeo@romeovansnick.be>
Co-authored-by: Nemanja Glumac <31325167+nemanjaglumac@users.noreply.github.com>
* User Parameter Values Are Unique Per Dashboard (#45064)
* User Parameter Values Are Unique Per Dashboard
Fixes: #43001
Might be related: #44858
User Parameter Values previously stored just the `parameter-id` and `user-id` as data to look up the last used
parameter value. This isn't sufficient as the parameter id is not guaranteed to be unique, in particular this is true
when a dashboard is duplicated: the parameters are copied to the new dashboard without changing the parameter id at
all.
This means that we need to also store the dashboard-id in the user_parameter_value table and use that to update/get
the last used value.
The migration removes existing entries to the user_parameter_value table because I want a non-nullable constraint on
the dashboard_id column, and existing entries will not have a value. The only way to backfill those values would be to
look through every dashboard containing parameters, and for every parameter check for a matching ID. Even if you can
do this, there's no way to disambiguate if the same paramter exists on 2 dashboards anyway, so one of them will be
wrong. I think it's not worth the trouble considering that removing the entries in this table doesn't break anything;
it just becomes a mild inconvenience that a user has to select a filter value again (since the dashboard will use the
default value).
* alter test to check for uniqueness across dashboard
This test makes 2 dashboards with parameters of the same ID and asserts that changing the value on dashboard-a does
not change the value on dashboard-b.
* adjust migration to pass linter rules
* remove the extra rollback on migration
* Update src/metabase/models/user_parameter_value.clj
Co-authored-by: bryan <bryan.maass@gmail.com>
* Update src/metabase/models/user_parameter_value.clj
Co-authored-by: bryan <bryan.maass@gmail.com>
* adjust parameters in test so we don't get logged warnings
* put update!/insert! in a transaction to avoid any race conditions
---------
Co-authored-by: bryan <bryan.maass@gmail.com>
* change to proper migration IDs
---------
Co-authored-by: Adam James <adam.vermeer2@gmail.com>
Co-authored-by: bryan <bryan.maass@gmail.com>
* include dark background to PNG exports when night theme is enabled
* fix css
* override border for png downloads
Co-authored-by: Aleksandr Lesnenko <alxnddr@users.noreply.github.com>
`Lib.diagnoseExpression` is stricter about this than the QP or FE
expression parser, making it reject expressions that tried to compare
datetimes to strings like `2024-07-02 14:34:17Z`.
Those used to work in 49.x, but a new `Lib.diagnoseExpression` call
while editing expressions caused it to regress.
Fixes#44916.
Co-authored-by: Braden Shepherdson <braden@metabase.com>
**MetadataProvider cache**
The reusable MetadataProviders is a generally useful thing - within some
dynamic scope like an API request, a `clojure.core.cache` instance can
be provided as `lib.metadata.jvm/*metadata-provider-cache*`, and that
will be used to cache the provider for each `database-id`.
Reusing those providers can significantly reduce appdb traffic,
re-fetching the same metadata in different `MetadataProvider`
instances.
**Dashboard loads**
The FE can supply `?dashboard_load_id=uuid_string` on requests for a
dashboard load:
- `/api/dashboard/:id`
- `/api/dashboard/:id/query_metadata`
- `/api/dashboard/:id/dashcard/:dashcard_id/card/:card_id/query`
That provides a good cache key across that burst of N+2 API requests
that are part of a single load. Currently it is used for two things:
- Caching `get-dashboard` (used by the first two requests)
- Selecting the `clojure.core.cache` for `*metadata-provider-cache*`
Both use a TTL of 10 seconds, which is generally enough for the first
screenful of a dashboard load, if not the whole thing. (Browsers will
only issue a handful of requests at once, so a few long-running queries
might delay later parts of the dashboard from loading until after the
TTL expires.)
Co-authored-by: Braden Shepherdson <braden@metabase.com>
* Make sure the query is native
* Convert the comment to JSDoc
* Add E2E reproduction
* Fix mapping
* Add test scenarios for the public sharing and embedding
* Simplify API payload
* Extract and significantly expand the reproduction scope
---------
Co-authored-by: Nemanja Glumac <31325167+nemanjaglumac@users.noreply.github.com>
Co-authored-by: Alexander Polyankin <alexander.polyankin@metabase.com>
* only send permissions updates for groups that have been modified in the graph
* Support partial permission graph updates on the BE (#44797)
* handles keeping track of advanced permissions changes to send relevant group data when updating the graph, merges response
* tests and optimizations
* respect group-ids set when fetching connection impersonations, sandboxes, and admin group perms
* fix typo
* add e2e test
* fix BE test
* impls pr feedback, reduces code needed since advanced permissions state is always reset on save
* last fixes
---------
Co-authored-by: Sloan Sparger <sloansparger@users.noreply.github.com>
Co-authored-by: Noah Moss <32746338+noahmoss@users.noreply.github.com>
Co-authored-by: Noah Moss <noahbmoss@gmail.com>
* Reset JoinDraft state on database change (fixes metabase#42385)
* Add repro for #42385
* Use admin user
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
* support sorting on funnel bar charts (#44943)
* support sorting on funnel bar charts
* fix funnel bar row display setting
* spec
* Update Loki Snapshots
---------
Co-authored-by: Aleksandr Lesnenko <alxnddr@users.noreply.github.com>
* Add reproduction for #38989
* Use saveQuestion instead of clicking trough the UI
* Use openOrdersTable instead of manually clicking through the ui
* Create question programmatically in test
---------
Co-authored-by: Romeo Van Snick <romeo@romeovansnick.be>
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
* Combine and reorganize admin e2e specs (#43853)
* combine and reorganize admin specs
* fix import and reorganize more
* Revert React 18 changes
* Revert Metrics v2 changes
* Bring back #44497
* Add a reproduction for "Cannot find the new column in pick columns after alter schema and synced" (#44882)
* Add a reproduction for metabase#41765
* Navigate manually to the admin page
* Regroup (metric v1) repro for 36422
* fix: Remove help email from error diagnostic modal (#44165)
---------
Co-authored-by: Ryan Laurie <30528226+iethree@users.noreply.github.com>
Co-authored-by: Romeo Van Snick <romeo@romeovansnick.be>
Co-authored-by: Raphael Krut-Landau <raphael.kl@gmail.com>
* Limit the height of the breakout popover
* Allow the click action popover to untether to prevent overflowing the screen
* Add reproduction for #43075
* Rename breaktrhough to breakout
* Use findAllByTestId
* Remove copy pasted question name
* Remove unused api intercept
---------
Co-authored-by: Romeo Van Snick <romeo@romeovansnick.be>
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
* Wrap non-latin characters in a span specifying working font
Fixes: #38753
CSSBox seems to have a bug where font fallback doesn't work properly. This is noticeable when a font does not contain
glyphs that are present in the string being rendered. For example, Lato does not have many international characters,
so the rendered version of tables (that show up in Slack messages) will not render properly, making the card
unreadable.
Since this appears to be a downstream bug, I've opted to work around this limitation by wrapping any non-latin
characters in a <span> that specifies the font family to be sans-serif, which should contain the glyphs to properly
render.
This leaves Lato in place for other characters.
For now, I figured it's worth trying this solution over using Noto for 2 reasons:
- we can keep Lato, which has been the decided font style for the app for some time (this keeps things consistent
where possible)
- the Noto font containing all glyphs is quite a large font (>20mb, I think) and it would be nice to avoid bundling
that if we can.
* stub installed fonts test
* typo
* Do wrapping, but now per-string, and in Clojure data not html string
I've decided that a reasonable solution is to still wrap strings containing non-lato characters. But it's not done
with str/replace to the html string but rather in a postwalk over the Hiccup data prior to rendering.
Seems like a decent compromise of issues without patching CSSBox or fixing upstream (may be good to do, but will take
longer to get a fix in).
* add test checking that glyphs render correctly
* Add a test that directly checks the wrapping fn
* Change the string to keep the linter quiet
* Change how we check if string can be rendered to faster method, new Lato Font
With Sanya's help, the way I check if a given string is renderable with Lato is now faster.
Also use the full Lato font, not just the 'latin' lato so we can cover more chars
* change lato so that it loads the fn even in a fresh test run
Co-authored-by: adam-james <21064735+adam-james-v@users.noreply.github.com>
Co-authored-by: Nemanja Glumac <31325167+nemanjaglumac@users.noreply.github.com>
* prevent pin maps from filtering out rows with null values
- since pin maps are just displaying where the data is by latitude and longitude, we do not need to filter the pin just because one of the rows is null.
- the metric column selection does not make sense for pins as they just represent that data through coordinates (no aggregations or values displayed), so we should not be filtering out any data just because it is null
* add unit test
Co-authored-by: Jesse Devaney <22608765+JesseSDevaney@users.noreply.github.com>
* Rather support old files, than duplicate columns
* Cope with driver specific failure modes
* Restore bytes comment
Co-authored-by: Chris Truter <crisptrutski@users.noreply.github.com>
* Single Log Out saml-slo-enabled defsetting (#44794)
- backport:
- use `client/client-real-response`, instead of `client-full-response`
* stop using strings for enum values (prefer kw)
* only do SLO when saml-slo-enabled is true
* make defsetting docstring formatting consistent
* log-direction is a string
* add test for enabling / disabling saml slo
- also return a 403 when the client tries, but it's disabled
* use client-full-response, which won't add '/api'.
* fix typo in deftest name
---------
Co-authored-by: bryan <bryan.maass@gmail.com>
I was thinking about the potential options here.
On one hand, we could initialize the app db ourselves. This is a little
problematic, because:
- our `init!` function is in `metabase.core` and calling it yields a
painful cyclical dependency. It's also pretty tied to outputting things
to the user that they might not expect during an import, e.g. "Looks
like this is a new installation ... preparing setup wizard", and starts
up jobs, etc.
- if we choose to *only* initialize the database ourselves (only the
bits that we actually *need* initialized), I'm a little nervous about
maybe missing something. Right now we need to create the internal user
and create users from the config file before running the import. If we
added a third thing, would we remember to update this?
- we could refactor to put all of the above in one spot, but this is a
P1 and it feels important to close this out as quickly as possible. We
also don't claim to support initializing the app DB from an import in
the first place.
So: just throw an exception if the app db hasn't been set up yet. Our
heuristic here is whether a single non-internal user has been created.
So if you want to use your config, you can run:
```
MB_CONFIG_FILE_PATH=dev/config.yml java -jar metabase.jar
java -jar metabase.jar import /my/exported/dump
```
If you try to run `java -jar metabase.jar import ...` on a fresh
DB, it'll now fail with the error message:
```
You cannot `import` into an empty database. Please set up Metabase
normally, then retry.
```
Co-authored-by: John Swanson <john.swanson@metabase.com>
* Fix order by field refs
* Update test to handle all drivers
* Untangle fix-order-by-field-refs
Co-authored-by: lbrdnk <lbrdnk@users.noreply.github.com>
* Repro #41785 - Using Column picker to remove columns on model breaks question with ERROR: column source.name_2 does not exist Position (#44808)
* Add repro for #41785
* Update comment
* Fix failing test
---------
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
* fix: dont always optimize between expressions
Fixes#42291
The frontend produces expressions like
```
[:between
[+ [:field ... {:temporal-bucket :day] [:interval 2 :day]]
[:relative-datetime -1 :week]
[:relative-datetime 0 :week]]
```
This should not be optimized because of the mixed `:day` and `:week`
units. However, it was being optimized since the compatible units
weren't being properly picked up by the match.
* Disable suspcicios args eastwood lint, kondo does it better
* Stop autobucket to day when adding a time interval
Co-authored-by: Case Nelson <case@metabase.com>
Co-authored-by: Nemanja Glumac <31325167+nemanjaglumac@users.noreply.github.com>
* Port the original `reproductions-3.cy.spec.js`
* Port changes from #44053
* E2E repro for "Fix issue with root collection in QuestionPicker" (#44318)
* Port 44415 and 44532
* Port 41381
* Add reproduction for "Result inconsistencies with Preview in GUI editor #40399" (#44630)
* Add e2e test to verify that step previews only care about the step being previewed
* Use getNotebookStep and find preview button using cy.icon
* Remove empty line
Co-authored-by: Nemanja Glumac <31325167+nemanjaglumac@users.noreply.github.com>
---------
Co-authored-by: Nemanja Glumac <31325167+nemanjaglumac@users.noreply.github.com>
* Fix expressions with the same name as their base column (#44641)
* Pass expressionPosition
* Add a test
* Update e2e/test/scenarios/question-reproductions/reproductions-3.cy.spec.js
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
* Rename expressionPosition to expressionIndex
---------
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
* Add E2E repro for convertTimezone (#44705)
* E2E test for custom date expression error (#44694)
* Add reproduction for "Closing Drop-down in Join column selector causes non-drop-down columns to disappear" (#44675)
* Add reproduction for #19894
* Use startNewQuestion
* Fix 43294
---------
Co-authored-by: Alexander Polyankin <alexander.polyankin@metabase.com>
Co-authored-by: Romeo Van Snick <romeo@romeovansnick.be>
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
* Add new e2e test group
* Consolidate `reproductions-1.cy.spec.js`
* Revert changes specific to `master`
* Port the original `reproductions-2.cy.spec.js`
* Port a change from #43481
* Port changes from #43660
* Add a repro for 'Cannot change question data source to raw data after selecting saved question' (#44757)
* Add a repro for 'Cannot change question data source to raw data after selecting saved question'
* Update e2e/test/scenarios/question-reproductions/reproductions-2.cy.spec.js
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
* Update e2e/test/scenarios/question-reproductions/reproductions-2.cy.spec.js
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
* Update e2e/test/scenarios/question-reproductions/reproductions-2.cy.spec.js
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
---------
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
* Add a repro for 'Visible Columns errors when source query is a table joined on itself' (#44737)
* Add a repro for 'Visible Columns errors when source query is a table joined on itself'
* verify warning icon is not rendered
* Fix repro 9027
Revert "move to trash" changes
---------
Co-authored-by: Uladzimir Havenchyk <125459446+uladzimirdev@users.noreply.github.com>
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
* remove invalid else conditional
* fix RowChart firing invalid API request on legend item click
* fix clicking on non-breakout legend on dashboards
* add E2E test to validate and enforce the desired behavior
Co-authored-by: Jesse Devaney <22608765+JesseSDevaney@users.noreply.github.com>
* Add a repro for 'Cannot change question data source to raw data after selecting saved question'
---------
Co-authored-by: Uladzimir Havenchyk <uladzimir.dev@gmail.com>
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
* fix map default display type with un-binned coordinates
- default to `"map.type"` = "pin" when coordinates are not binned
- default to `"map.type"` = "grid" when coordinates are binned
* add tests
* refactor breakouts usage
Co-authored-by: Jesse Devaney <22608765+JesseSDevaney@users.noreply.github.com>
* Allow clicking "Remove all" for the first group in `FieldPanel` and omit removing the first field in the group (#44645)
* Backport some test helpers
---------
Co-authored-by: Alexander Polyankin <alexander.polyankin@metabase.com>
* Fix broken field normalization in metadata
* Handle Fields.actionTypes.UPDATE action in tables reducer to keep table's original_fields in sync
* Add an e2e test for fields
* Format code
* Use a different table, since accounts table is not available in e2e tests
---------
Co-authored-by: Alexander Polyankin <alexander.polyankin@metabase.com>
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
* Consolidate filters reproductions
* Consolidate dashboard filters reproductions
* Add new group to the CI
* Fix recently modified tests that have different behavior in the release branch
* Unskip the repro for #18770 (Post-aggregation filters) (#44748)
* Bump dashboard description max length limit to 1.5K (#44470)
* Bump dashboard description max len limit to 1.5K
* Extract `DASHBOARD_DESCRIPTION_MAX_LENGTH` const
* Handle description max length in dashboard sidebar
* Add red border to dashboard description box
* Add test
* Use `color` fn instead of CSS variables
* Add e2e test to verify that step previews only care about the step being previewed
* Use getNotebookStep and find preview button using cy.icon
* Remove empty line
---------
Co-authored-by: Romeo Van Snick <romeo@romeovansnick.be>
Co-authored-by: Nemanja Glumac <31325167+nemanjaglumac@users.noreply.github.com>
* convert embed.ts to ts and to use createSlice from rtk (#44496)
* convert embed.ts to use createSlice from rtk
* only store in the state what we care about for interactive embedding
* use DEFAULT_EMBED_OPTIONS keys to filter searchOptions
* removed unused hash parameter
* adds basic unit test
* add back in the font to the redux state, as in v50 that's where we store it
* remove validation to not change logic in a minor in the backport
---------
Co-authored-by: Nicolò Pretto <info@npretto.com>
This commit adds `$size` operator wrapping to `$addToSet`. That enables use of `:distinct`
clause in expressions on Mongo.
Co-authored-by: lbrdnk <lbrdnk@users.noreply.github.com>
* Effective location on collection recent items (#43018)
* add effective_location to recent-collection-items
* cleanup
* fix one failing test
---------
Co-authored-by: bryan <bryan.maass@gmail.com>
Co-authored-by: Adam James <adam.vermeer2@gmail.com>
* Add Temporal Units List to Dashboard Parameters Schema
Closes: #44361
This adds an entry to the Dashboard Parameter schema so that we can validate
the list of temporal_units that might be passed from the frontend when adding
or updating the `Unit of Time` type parameters.
* Sanitize the SVG before parsing it with Batik
WIP
Fixes: #43677
There are characters that are invalid according to the Batik XML Parser. This sanitizes the svg string to strip out
some of those characters so that the render can continue.
* Use the xml 1.0 spec allowed unicode chars list for the regex
* Remove accidentally merged code
* still miseed a line
Co-authored-by: adam-james <21064735+adam-james-v@users.noreply.github.com>
* Fix the operator list for `type/TextLike` PKs
* Fix the bug with using wrong operators
* Fix the bug with using wrong operators
* Update test/metabase/lib/filter_test.cljc
* Test FKs too
* Update test/metabase/lib/filter_test.cljc
---------
Co-authored-by: Alexander Polyankin <alexander.polyankin@metabase.com>
Co-authored-by: lbrdnk <lbrdnk@users.noreply.github.com>
This PR removes `:role` removal from spec, even if it was added in `:connection-uri`.
It seems that `:connection-uri` is ignored on `do-with-connection-options` code path.
Hence use of non-default roles is again available on Snowflake.
Co-authored-by: lbrdnk <lbrdnk@users.noreply.github.com>
This not only does not make sense, but also fails for models with aggregations (and thus not having those fields for filtering available).
Co-authored-by: Alexander Solovyov <alexander@solovyov.net>
This PR disables the 'make exports look pivoted' post processing step for csv and xlsx exports.
This is in response to a reported issue that Large Pivot Table downloads are failing entirely: #44556
A more hollistic fix and improvement to the post processing feature will be worked on, so the post processing code is
left untouched in this PR.
The dynamic vars in the csv and xlsx namespaces each default `false` and disable the export from running any post
processing. The var should remain `false` until we've got a plan for these types of exports, and there is no way for
users (either through ENV or UI) to change this, so the feature is effectively disabled.
Co-authored-by: adam-james <21064735+adam-james-v@users.noreply.github.com>
* Set `useLocalSessionState=true` for MySQL DW connections
* Fix GH issue number
* Test fixes 🔧
* Lint fix 🔧
Co-authored-by: Cam Saul <1455846+camsaul@users.noreply.github.com>
* Make the App DB transaction isolation level READ_COMMITTED for MySQL [MEGA PERFORMANCE BOOST] (#44505)
* Make the App DB transaction isolation level READ_COMMITTED for MySQL/MariaDB [MEGA PERFORMANCE BOOST]
* Add test
* empty commit to tickle CI
---------
Co-authored-by: Cam Saul <1455846+camsaul@users.noreply.github.com>
Co-authored-by: Braden Shepherdson <braden@metabase.com>
* Don't run sandboxing rollback test on mysql
* Update test/metabase/db/schema_migrations_test.clj
Co-authored-by: Noah Moss <32746338+noahmoss@users.noreply.github.com>
* fix(cache): Fix two bugs in Admin / Performance (#44255)
Ensures that minimum query durations are correctly displayed in the form post-save (#44249) and that all times can be used in the Schedule component (#44257)
---------
Co-authored-by: Raphael Krut-Landau <raphael.kl@gmail.com>
* UserParameterValue transforms wrapped to properly escape string vals
The `json-in` and `json-out` functions used for toucan model transforms do not perform any string escaping logic. This
makes sense as we don't want to make assumptions about the shape of data flowing in/out of the db. But, this did mean
that for the User paramter values table, string values were causing an error to be logged. This doesn't break
anything, as the function will still correctly return the string, but it can clutter up logs.
So, this PR wraps all incoming values in a map and unwraps it for outgoing values.
* Add migration to wrap existing user param values with ::wrapper key
Since the in/out transform for UserParameterValues is updated, we need to migrate any existing values to have this wrapping.
* Simplify the solution to the problem.
Since we're already getting what we need from the json-in/json-out *except* that it's logging a parse error, we can
create a json-out that works the same way (tries to parse and returns the string as-is if it fails) without logging an
error at all.
* take away the arg and keywordize inside the json-out fn
* remove irrelevant comment
* Add a little more test coverage for different value types
* faster test thanks to Dan!
* Add a comment about the data being tested
---------
Signed-off-by: Adam James <adam.vermeer2@gmail.com>
Co-authored-by: adam-james <21064735+adam-james-v@users.noreply.github.com>
* XLSX Pivot Table Downloads have 'Native' Pivot Table (#43791)
* XLSX Pivot Table Downloads have 'Native' Pivot Table
Addresses: #2473
The xlsx downloads work fine for regular tables. However, on Pivot Table downloads, the resulting file isn't pivoted
and is formatted somewhat strangely (it contains additional rows related to totals, has an extra 'pivot-grouping'
column).
Now, the downloaded xlsx file contains 2 sheets:
- the data sheet containing the question's rows unpivoted. If you were constructing a pivot table yourself, you would
probably start with this shape of data.
- the pivot sheet containing a 'native' Pivot table whose reference points to the data on the data sheet, and whose
columns, rows, and values align with the cols, rows, and measures set up in Metabase
* Silly typo!
* Exported Pivot uses correct aggregation (sum, count, stddev, avg, min, max)
* data for pivot exports now also uses the formatting
* Add a test asserting that a pivot table exists in the xlsx file
* add-row! is a method dispatching on sheet class
* remove unnecessary hint
* address review feedback:
- consolidate the aggregation function key fns into one function in the .xlsx namespace
this was moved out of postprocessing so that the xlsx specific fns don't litter the post processing namespace at
this time
- clean up an unnecessary destructiring inside the native-pivot function
- don't shadow 'name' inside the body of a fn
* Add tests that confirm zero-col and zero-row situations work
* Consolidate the atoms used to store pivot rows/options
* unstick be checks?
---------
Co-authored-by: adam-james <21064735+adam-james-v@users.noreply.github.com>
Co-authored-by: Adam James <adam.vermeer2@gmail.com>
Fixes#43993 for real.
The earlier fix#44182 was needlessly restricting the
"nominal refs" check to fields which were using numeric IDs,
when really any match on nominal refs is valid.
Co-authored-by: Braden Shepherdson <braden@metabase.com>
* Pivot Tables with no Pivot Columns should still download
Fixes#44159
Pivot Tables can still be valid without columns, and therefore the download should respect such a query and be
successfully exported.
In this PR, the postprocessing of pivot results into 'visual pivot' exports for csv takes into account the zero-column
configuration and the download no longer fails.
* add a test to confirm that zero cols and >1 measures works as well
* Handle the case where there are zero pivot-rows.
This is a similar problem to the zero pivot-cols case, just with rows.
The app actually doesn't render these tables correctly, but that's a separate frontend bug
Co-authored-by: adam-james <21064735+adam-james-v@users.noreply.github.com>
* use effective location when building initial path in entity picker
* unit test adjustment
* more unit test adjustments
Co-authored-by: Nick Fitzpatrick <nick@metabase.com>
* Change run to make-run in process-query-for-card
Fixes#44160
There were two levers where a card could change how it was run.
`qp` this is basically `qp/process-query` with some optional userland
flags and constraints marked on the passed in query.
In the case of pivot tables in the
`/pivot/:card-id/query` endpoints it was `qp.pivot/process-query`
'run' this defaults to a streaming response that takes the `qp` param
and `export-format` and passes a query to qp. Most often this is
overriden to not stream the response.
Unfortunately, if `run` is overwritten, then `qp` is thrown away, and
the caller who overrides `run` must determine the correct `qp` to use.
However, when running `process-query-for-dashcard` it is difficult to properly set
`qp` for all of the dashcards without doing card lookups. So [this commit](https://github.com/metabase/metabase/blob/release-x.50.x/src/metabase/query_processor/card.clj#L230-L232) did a check to swap the runner in the presence of a pivot query that calls the **default** runner with a `qp.pivot/run-pivot-query`. The problem is that pulse needs to use a non-default runner, but there's no way for `process-query-for-card` to do both.
This commit changes `run` to `make-run` which raises the abstraction to
a function that produces a runer given a `qp`. This allows
`process-query-for-card` to change the `qp` as needed while keeping the
passed in runner.
There's really three things going on that could be teased apart
further/better.
1. `qp` - this conflates the processor (`qp/process-query` `qp.pivot/run-pivot-query`)
and modifying query (`qp/userland-query`,
`qp/userland-query-with-default-constraints` `(update query :info merge info)`)
2. `make-run` - How the above are processed: (streaming/realized)
* Fix tests
Co-authored-by: Case Nelson <case@metabase.com>
* fix charts crash in safari when dataset timezone is not in iana format (#44193)
* fix charts crash in safari when dataset timezone is not in iana format
* Update Loki Snapshots
* better fix: apply offset instead of finding a timezone with such offset
* remove unused specs
* Update Loki Snapshots
* review
---------
Co-authored-by: Metabase Automation <github-automation@metabase.com>
* snapshots
---------
Co-authored-by: Aleksandr Lesnenko <alxnddr@users.noreply.github.com>
Co-authored-by: Aleksandr Lesnenko <alxnddr@gmail.com>
* adding snippets back into getCollection
* e2e test
---------
Co-authored-by: Nick Fitzpatrick <nick@metabase.com>
Co-authored-by: Sloan Sparger <sloansparger@gmail.com>
* Handle errors when inferring separator of csv
This file
```
"c1","c2"
"a,b,c","d"
```
was failing to upload for a silly reason: when we were checking for
which separator it used, we had a parse error when using a semi-colon
and that blew up the whole pipeline.
* Use cal's helpful test suggestions
Co-authored-by: dpsutton <dan@dpsutton.com>
* certain settings should be decrypted for cache config migration to run (#44048)
* ~ fix test-migrations-for-driver! when rolling back and migrating again
---------
Co-authored-by: Alexander Solovyov <alexander@solovyov.net>
Co-authored-by: Cal Herries <calherries@gmail.com>
* Fix the typo
* Replace `realHover` with the `mouseMove` trigger
* Be explicit when targetting elements by their index
* fix combo chart tooltip e2e specs
* cy.wait as a last resort
---------
Co-authored-by: Nemanja Glumac <31325167+nemanjaglumac@users.noreply.github.com>
Co-authored-by: Aleksandr Lesnenko <alxnddr@gmail.com>
* Remind embedding SDK PR author to use conventional commits
* Only remind PR author when PR title isn't conventional commits
* Add branch for testing
* Fix gh cli command
* Fix getting wrong user login
* Ensure we don't create more than 1 comment
* Improve comment format
* Add more info for non-changelog related type
* Check non-changelog related types
* Update the comment for consistency
* Revert "Add branch for testing"
This reverts commit 4a1be68828.
Co-authored-by: Mahatthana (Kelvin) Nomsawadi <me@bboykelvin.dev>
* Fix stacked redirects to homepage dashboard crashing the app (#43836)
* Add repro test
* Move home selectors to `metabase/selectors/app`
* Prevent unnecessary redirects to home dashboard
* Rename
* Handle navbar home link
* Fix unit tests
* Fix navigating away from an archived dashboard
* Handle sub-root Metabase URLs
* Fix dashboard state sync
---------
Co-authored-by: Anton Kulyk <kuliks.anton@gmail.com>
I'm not sure it's any less hacky in the CLJS but it keeps the TS
wrappers thin.
Context:
- #43799 is the original issue
- #43848 added the hack being removed here
Co-authored-by: Braden Shepherdson <braden@metabase.com>
Co-authored-by: Alexander Polyankin <alexander.polyankin@metabase.com>
This PR simplifies a bit of text used in the cache cron schedule component: we can say "spaces" (which is normal English) not "white spaces" (which is developerese)
Co-authored-by: Raphael Krut-Landau <raphael.kl@gmail.com>
* Initial commit
* unit test, always white on active
* Run Prettier 🙄
* removing yellow color from verified model icon
* linter and type fix
Co-authored-by: Nick Fitzpatrick <nick@metabase.com>
* fix data labels under trend/goal lines (#42928)
* fix data labels under trend/goal lines
* fix fontFamily missing
* update specs
* fix type errors
* snapshots
---------
Co-authored-by: Aleksandr Lesnenko <alxnddr@gmail.com>
* Set min bar height for non-zero bars (#43485)
* set min bar height for non-zero values
* fix y-axis split logic uses transformed dataset instead of actual data
* fix timeline events covers series (#43780)
* make compact formatting consistent across all series (#43472)
* make compact formatting consistent across all series
* fix rebase
* resolve yarn.lock conflicts
* snapshots
---------
Co-authored-by: Emmad Usmani <emmadusmani@berkeley.edu>
* Run smaller smoke test suite for pre-release check (#43633)
* run smaller smoke test suite for pre-release check
* fix run command
* kick ci
---------
Co-authored-by: Ryan Laurie <30528226+iethree@users.noreply.github.com>
Co-authored-by: Ryan Laurie <iethree@gmail.com>
* Automate SDK doc PR
* Prevent auto merge
* Dry run npm publish for testing
* Speed up testing by not building jars
* Fix auto approve step
* disable concurrency so I could test multiple runs in parallel
* Test auto-merge
* Fix auto merge
* Revert "disable concurrency so I could test multiple runs in parallel"
This reverts commit 60e90076eb.
* Revert "Speed up testing by not building jars"
This reverts commit 4e53a54414.
* Revert "Dry run npm publish for testing"
This reverts commit 9b5d761c8d.
* Enable all commented out steps
* Remove unnecessary steps
Co-authored-by: Mahatthana (Kelvin) Nomsawadi <me@bboykelvin.dev>
* Mock that the changelog is different from the one in master
* Apply changelog patch incase the changelog is created from another branch
* Dry run npm publish so I could test
* Fix failed patch
* Speed up testing by not building jar
* Fix adding new changelog to master
* feat(sdk): A placeholder for an actual feature in the changelog
* Mock that the changelog diverges from master
* Revert "Speed up testing by not building jar"
This reverts commit acb80d7465.
* Revert "Dry run npm publish so I could test"
This reverts commit f4e679efd9.
* Revert "Mock that the changelog diverges from master"
This reverts commit 0e77a68e1b.
Co-authored-by: Mahatthana (Kelvin) Nomsawadi <me@bboykelvin.dev>
* Multi Series Dashcards Render with Filters Applied To All Series
Fixes: #39083
Multi-series dashcards now properly render when a subscription is set up with filters that impact the values of each
card in the multi-series card.
* Small refactor for executing dashcards in a pulse context.
- move execute-dashboard-subscription to pulse.util
- this fn now will also execute all card series cards
and put those results into a list on the key :series-results in the dashcard
which is then used in the render multimethod to send the correct data to the javascript context
A small todo was taken care of as well, we can now use :dashboard-subscription in the :context key
* Fix up render fn
The shape of hte data going into the javascript was a bit unclear before. This keeps things a little more
understandable.
Next step is probably to create some Malli schemas inside the js_svg namespace to document and enforce the needed data shapes
* pull :data up a level from :result in the series cards
* Update src/metabase/pulse/render/body.clj
* Remove unneeded let
* set/rename-keys won't work here because we raise the :data key
I've realised a suggested change to just use set/rename-keys won't do what we need. We need instead to raise :data
from within the result key up to the same level (and dissoc result).
I've renamed the helper function to make its purpose a bit clearer
---------
Co-authored-by: adam-james <21064735+adam-james-v@users.noreply.github.com>
Co-authored-by: Ngoc Khuat <qn.khuat@gmail.com>
Liquibase changed their boolean type in MySQL from `bit(1)` to
`tinyint(4)` in version 4.25.1. Our JDBC driver does not recognize these
as booleans, so we needed to migrate them to `bit(1)`s.
As discussed [here](#36964), we
changed all existing `boolean` types that were in the
`001_update_migrations.yml` but not the SQL initialization file.
For new installations, this works: things in the SQL initialization file
get created with the `bit(1)` type.
However, for existing installations, there's a potential issue. Say I'm
on v42 and am upgrading to v49. In v43, a new `boolean` was added.
In this case, I'll get the `boolean` from the liquibase migration rather
than from the SQL initialization file, and it need to be changed to a
`bit(1)`.
I installed Metabase v41 with MySQL, migrated the database, and then
installed Metabase v49 and migrated again. I made a list of all the
columns that had the type `tinyint`:
```
mysql> SELECT TABLE_SCHEMA, TABLE_NAME, COLUMN_NAME, COLUMN_TYPE, COLUMN_DEFAULT, IS_NULLABLE FROM INFORMATION_SCHEMA.COLUMNS WHERE COLUMN_TYPE = 'tinyint' AND TABLE_SCHEMA='metabase_test';
+---------------+------------------------------+-------------------+-------------+----------------+-------------+
| TABLE_SCHEMA | TABLE_NAME | COLUMN_NAME | COLUMN_TYPE | COLUMN_DEFAULT | IS_NULLABLE |
+---------------+------------------------------+-------------------+-------------+----------------+-------------+
| metabase_test | core_user | is_datasetnewb | tinyint | 1 | NO |
| metabase_test | metabase_field | database_required | tinyint | 0 | NO |
| metabase_test | metabase_fieldvalues | has_more_values | tinyint | 0 | YES |
| metabase_test | permissions_group_membership | is_group_manager | tinyint | 0 | NO |
| metabase_test | persisted_info | active | tinyint | 0 | NO |
| metabase_test | report_card | dataset | tinyint | 0 | NO |
| metabase_test | timeline | archived | tinyint | 0 | NO |
| metabase_test | timeline | default | tinyint | 0 | NO |
| metabase_test | timeline_event | archived | tinyint | 0 | NO |
| metabase_test | timeline_event | time_matters | tinyint | NULL | NO |
+---------------+------------------------------+-------------------+-------------+----------------+-------------+
10 rows in set (0.01 sec)
```
Then wrote migrations. For each column, we:
- turn it into a `bit(1)`,
- re-set the previously existing default value, and
- re-add the NOT NULL constraint, if applicable.
---------
Co-authored-by: SakuragiYoshimasa <ysrhsp@outlook.com>
On the Browse models page we have a filter icon which opens a popover with a switch that toggles "Show only verified models". With this change, the icon is not rendered if there are no verified models at all.
Co-authored-by: Raphael Krut-Landau <raphael.kl@gmail.com>
* Fix column scaling ("Multiply by a number") bugs (#43668)
* return empty string so non-empty labels do not get hidden by empty labels
* fix function name
* apply scaling in transformed dataset
* add missing option to OptionsType
* fix double scaling
* add column scaling to waterfall transformed dataset
* add column value scaling to dataset transform
* fix stacked bar segment labels not responding to column scaling
* fix tooltip values displayed
* since scaling as added to transformedDataset, use transformedDataset for formatters
* revert changes to display percentage instead of absolute value
* fix stacked data labels
* add loki variant for custom y-axis range with column scaling
* fix stack segmented label formatters compact calculation
* scale original dataset and use that instead of modifying transformedDataset
* update loki snapshots
* fix scatter plot dataset scaling
* fix tooltip values being double scaled
* add E2E test to validate behavior
* add helper for not duplicating the column scaling
* port multiple y-axis metrics with column scaling fixes to RowChart
* add waterfall and scatterplot loki variants
* fix type errors and failing unit tests
* fix failing unit tests
* update loki snapshots
* update loki snapshots
---------
Co-authored-by: Jesse Devaney <22608765+JesseSDevaney@users.noreply.github.com>
* Fix OSS tests in CI (#43660)
* fix OSS tests in CI
* update tests that went on vacation for a while
* do safer cypress things
* better helper name
# Conflicts:
# e2e/support/helpers/e2e-ui-elements-helpers.js
# e2e/test/scenarios/collections/instance-analytics.cy.spec.js
# e2e/test/scenarios/models/reproductions.cy.spec.js
# e2e/test/scenarios/permissions/permissions-reproductions.cy.spec.js
# e2e/test/scenarios/question-reproductions/reproductions-2.cy.spec.js
# e2e/test/scenarios/question/new.cy.spec.js
* manual backport to changed spec files
* gross manual backporting
* fix conflict
---------
Co-authored-by: Ryan Laurie <30528226+iethree@users.noreply.github.com>
Co-authored-by: Ryan Laurie <iethree@gmail.com>
* update translations for v50
* add Danish
* update translations for v50 release
* alphabetical order is nice
Co-authored-by: Ryan Laurie <30528226+iethree@users.noreply.github.com>
* Check git tag for an exact match
* Set npm publish to dry run, so I can test
* Uses new github automation account for creating a PR
* Remove unnecessary step
* Automatically set `no-backport` on the SDK doc version bump PR
* Fix step to edit PR with correct permission
* Remove npm dry run after finished testing
Co-authored-by: Mahatthana (Kelvin) Nomsawadi <me@bboykelvin.dev>
* Fix flaky test
* Fix the flake for real
* Fix stress test failing because of mismatched height
* Review: Move viewport setting to shared config
Co-authored-by: Mahatthana (Kelvin) Nomsawadi <me@bboykelvin.dev>
* Revert "Revert "minimal support for multiple values for Access-Control-Allow-Origin (#42888)" (#43125)"
This reverts commit a4312230a8.
* refactor: extracted parse-approved-origins + supported "*" without special case
* fix: parse-approved-origins won't break on multiple spaces in a row
* handles invalid urls in the origins + some formatting
* log instead of throwing in parse-url + simplify the logic in parse-approved-origins
* simplify parse-approved-origins logic by splitting on "at least one space"
* remove extra boolean call
* approved-origins -> approved-origins-raw to better reflect that it's a raw string
* swap the order of the arguments to `is` in the tests
* indentation 🙃
Co-authored-by: Nicolò Pretto <info@npretto.com>
(When scrolling through CI logs I always do a double-take when I see a
huge stack trace with "Just an error". I think this is a little
clearer?)
Co-authored-by: Tim Macdonald <tim@metabase.com>
On the Browse models page we have a filter icon which opens a popover with a switch that toggles "Show only verified models". With this change, the icon is not rendered if there are no verified models at all.
Co-authored-by: Raphael Krut-Landau <raphael.kl@gmail.com>
* move parameters and dashboard reducers to the common ones as they're used everywhere
* convert reducers-{common,main,public}.js to typescript, done with --no-verify
* convert reducers to named exports
Co-authored-by: Nicolò Pretto <info@npretto.com>
* Do not use asReturned to determine columns for extractions and combine columns
* Add reproduction for #43226 for combine columns
* Add reproduction for #43226 for extract columns
* Remove commented out code
* Only render extraction shortcut when there are extractions
* Only render combination shortcut when there are combinations
* Use hasCombinations from CombineColumns
* Add hasExtractions from ExtractColumn
* Use appendStageIfAggregated over Lib.asReturned
* Only show combine column shortcut when there are two or more columns to be combined
* Lift appendStageIfAggregated to top-level drill
* Add test for extraction on table with just breakout
* Add test for combinations on table with just breakouts
* Disable the 2-column requirement for combinations
* Remove Lib.asReturned in appendStageIfAggregated
* Remove + 1 to stageIndex in appendStageIfAggregated
* Check for empty breakouts too in as-returned
* Switch back to Lib.asReturned
* Fix test for combine column on breakouts
* Reference the correct query and stageIndex
* add CLJS unit tests for as-returned with only aggs, breakouts
---------
Co-authored-by: Romeo Van Snick <romeo@romeovansnick.be>
Co-authored-by: Braden Shepherdson <braden@metabase.com>
* remove unused variable
* conditionally render labels sparsely if large amount of data points
* update loki snapshots
* refactor data label formatter conditions
* setup intelligent label plotting
* handle the case when there are no labels
* add sparse data label rendering to bar charts
* do not compute if no labels need to be shown or hidden
* add sparse data labeling to waterfall charts
* adjustment for large series waterfall labels
* revert change
* refactor name
* add sparse labels to stacked bar and area
* improve scaleFactor
* fix type errors
* remove unused labelFormatter
* update snapshots
* patch ECharts
- labels with no text, i.e. "", were still being considered as plot-able labels which was causing labels with real text "asdfec.." to be hidden because of overlap.
- This patch makes it so that labels with no text, i.e. "", are not considered to be candidates for overlap checking.
* update loki snapshots
* only loop over dataset once for getWaterfallChartDataDensity
* improve performance of chart data density calculations
* update loki snapshots
* update E2E spec
* update E2E spec
* increase cartesian label density allotment
* update loki snapshots
* update loki snapshots
* comment out node_modules cache since it cannot be skipped by commit and is breaking CI
* fix type errors
* Reset node modules if patches are changed
* refactor type naming
---------
Co-authored-by: Jesse Devaney <22608765+JesseSDevaney@users.noreply.github.com>
Co-authored-by: Uladzimir Havenchyk <uladzimir.dev@gmail.com>
Co-authored-by: Uladzimir Havenchyk <125459446+uladzimirdev@users.noreply.github.com>
* add hook to check token features
* add cache upsell to performance page
* upsell cards are white
* fix rebase
# Conflicts:
# frontend/src/metabase/admin/performance/components/StrategyEditorForDatabases.tsx
Co-authored-by: Ryan Laurie <30528226+iethree@users.noreply.github.com>
* Auto tag embedding team for PR review + dry run npm publish
* Install changelog generation package
* Add the initial changelog
* feat(sdk): This is an empty commit. For testing purposes.
* Update SDk build script to include changelog
* Check if a tag exists
* Ensure changelog is in the released package
* Fix workflow
* Remove unnecessary step
* Ensure we can have access to release utils script when checking out master
* Fix the step to create a PR updating readme and changelog
* Fix changelog generation
* Fix couldn't download changelog
* Clean up steps
Co-authored-by: Mahatthana (Kelvin) Nomsawadi <me@bboykelvin.dev>
Models are treated as opaque here, but for a saved question we expose
filters on the query before its final aggregation step.
This drops the aggregation clauses and gets the possible values for the
column of interest.
Fixes#43154.
Co-authored-by: Braden Shepherdson <braden@metabase.com>
* refactor scatter plot to separate code path (#43032)
* refactor scatter plot model to separate function
* (2/3) refactor scatter plot option to separate function (#43033)
* use getScatterPlotOption
* use buildEChartsScatterSeries directly
* remove hoveredSeriesDataKey from scatter
* (3/3) improve chart model types (#43034)
* create WaterfallChartModel type
* create ScatterPlotModel type
* rename BaseCartesianChartModel to CartesianChartModel
* simplify chartModel creation in hook
* support stacked series values labels (#42803)
* drop comment
---------
Co-authored-by: Emmad Usmani <emmadusmani@berkeley.edu>
Co-authored-by: Aleksandr Lesnenko <alxnddr@users.noreply.github.com>
Co-authored-by: Aleksandr Lesnenko <alxnddr@gmail.com>
* Multiple Measures Requires Additional Header Row in Pivot Exports
Mulitple measures can be set in our pivots. We indicate these properly in the fronted with a header row that shows
which measure is reprsented in that particular column.
The header builder fn in the pivot export post processing didn't handle this case, but after this PR, it correctly
adds the final header when necessary.
* Fix up header tests
* Remove comment form; the tests are derived from that work already
Co-authored-by: adam-james <21064735+adam-james-v@users.noreply.github.com>
These databases previously had impoverished support for regular
expressions. The `:regex` driver feature requires advanced,
"Perl-compatible" regular expressions that can do things like lookahead
assertions.
- MySQL supports this from 8.0.4; we recommend 8.0.33+.
- MariaDB supports this from 10.0.5; we recommend 10.4+.
Fixes#43180.
Co-authored-by: Braden Shepherdson <braden@metabase.com>
* [Manual backport] Backfill `:effective-type` for coerced fields (#43144) (#43357)
- Update how base type and effective type is added in query-method :mbql/query.
- Base type and effective type is added also during the conversion of query-method :query.
- Adds warning suppression for expressions to places where normalize is
called something that does not conform schme (eg. ->pMBQL :dispatch-type/map called
in query-method :query).
---------
Co-authored-by: lbrdnk <lbrdnk@users.noreply.github.com>
* Single metadata endpoint for a dashboard (#43283)
* Fix backport
* Backport dependent-metadata changes
* Fix lint
---------
Co-authored-by: Case Nelson <case@metabase.com>
Co-authored-by: Alexander Polyankin <alexander.polyankin@metabase.com>
* pulls out test changes from react 18 / rtl 15 refactor that are comptable with react 17 / rtl 12
* impls pr feedback
Co-authored-by: Sloan Sparger <sloansparger@users.noreply.github.com>
* Keep a single null dimension value
* Add Loki test
* Remove Loki test
* Throw an error if dataset is empty after filtering
* Extract error message into a constant
Co-authored-by: Anton Kulyk <kuliks.anton@gmail.com>
* Update expression suggestion design (#43232)
* Use sentence case for suggestion headers
* Remove suggestion group dividers
* Highlight suggestions when mousing over them
* Remove unused hoverparent
* Remove custom highlighting for icons
* Allow passing color to metadata info icon
* Make suggestions icons have a consisten color
* Update documentation suggestion style
* Add margin to suggestion sections
* Reorder suggestion sections
* Add test for mouse over highlight on suggestions
* Fix test for suggestion footers
* Remove debugging true
* Rename 'Most used functions/aggregations' to 'Common functions/aggregations'
* Highlight suggestion on mouse move
* Only call onHighlightSuggestion if it is not already highlighted
* Use arrow functions
* Fix test for suggestion highlight
* Add delay to fix broken test
* Use findAllByRole instead of wait
---------
Co-authored-by: Romeo Van Snick <romeo@romeovansnick.be>
* Use internal Link for relative and same-origin URLs
* Update frontend/test/metabase/lib/formatting.unit.spec.js
* Update frontend/test/metabase/lib/formatting.unit.spec.js
---------
Co-authored-by: Anton Kulyk <kuliks.anton@gmail.com>
Co-authored-by: Uladzimir Havenchyk <125459446+uladzimirdev@users.noreply.github.com>
* entityPicker in technicolor
* more precise import
* add unit test
# Conflicts:
# frontend/src/metabase/lib/icon.ts
Co-authored-by: Ryan Laurie <iethree@gmail.com>
* Do not render 'Add column' shortcut when viewing a raw table
* Do not show 'Add column' shortcut when a query has aggregations
* List shortcut guards out of component
* Make 'Add column' icon sticky in the header
* Add test for hiding the column shortcut
* Remove Number wrapper
* Move padding to a constant
* Use actions directly to determine if there are any shortcuts
* Make sure there are expressionable columns for the combine columns shortcut
* Make sure there are extractable columns for the extract column shortcut
* Remove padding when not overflowing
* Do not shadow query in popover components
* Allow combine and extract for raw tables
* Add test for combining summarization columns
* Add test for combining summarization columns
* Use a more precise way of calculating the offset for the 'Add column' button
* Handle null question
* Change to the outline button
* Use actual column widths to calculate total width
* Handle empty mode
* Remove extra measure call
* Move shortcutColumn to separate method and avoid looping over all available actions when possible
Co-authored-by: Romeo Van Snick <romeo@romeovansnick.be>
* improve models table performance
* add loading state for large sets of models
* less icon padding
Co-authored-by: Ryan Laurie <30528226+iethree@users.noreply.github.com>
* fix trend charts erroring on latest value being null
* refactor previousRowIndex calculation
* handle errors gracefully
* refactor error messages
* update computeTrend unit tests
* add error handling integration test
* add E2E test to check that errors are gracefully handled
* add integration test to enforce the fix
* fix static viz
* display more meaningful error on static-viz trend charts
* move error handling out of trend chart and fix error handling in Visualization.tsx
* update compute.unit.spec.js
* update SmartScalar.unit.spec.js
* use separate error handler for ErrorBoundary errors in Visualization.jsx
* update E2E test
* update error handling
* update compute.unit.spec.js
* fix react state change error in render function
* show generic error message for ErrorBoundary caught errors
* revert ErrorBoundary changes
* remove generic error border for visualizations
Co-authored-by: Jesse Devaney <22608765+JesseSDevaney@users.noreply.github.com>
* Fix missing React key prop
* Fix missing React key prop
* Fix order of conditions and make the code easier to follow
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
* Generate different shouldShowItem for different pickers
* Add repro for #42957
* Update test to use models instead of metrics
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
* highlight hovered timeline event
* add e2e test
* use TIMELINE_EVENT_DATA_NAME constant
* add test case for clicking icon
Co-authored-by: Emmad Usmani <emmadusmani@berkeley.edu>
* Restore default styles for subtle button (#42854)
* fix(sdk): Use theme font in charts and tooltips (#42855)
* Allow charts font override in SDK
* Fix default not configured font
(cherry picked from commit 0278b8d221)
* fix(sdk): Various fixes for InteractiveQuestion theming (#42932)
* Fixes for InteractiveQuestion theming
* Update readme
* Fix for unset filter theme color
* Fix unit tests
* Fix unit tests
(cherry picked from commit a3c3193474)
# Conflicts:
# enterprise/frontend/src/embedding-sdk/lib/theme/embedding-color-palette.ts
# enterprise/frontend/src/embedding-sdk/lib/theme/get-embedding-theme.ts
# frontend/src/metabase/ui/utils/colors.ts
These were added in #42602 but I neglected to update the changelog.
Their defaults should serve for most drivers.
Co-authored-by: Braden Shepherdson <braden@metabase.com>
* Extract EmailAdmin and add tests for it
* Rename EmailAdmin to AdminEmail
* Remove redundant async
* Add a test case for no email
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
* hide symbols for charts with too many of them
* update loki
* prevent series dots showing up after manually disabling another series
* update loki snapshots
* refactor conditional
* update loki snapshots
Co-authored-by: Jesse Devaney <22608765+JesseSDevaney@users.noreply.github.com>
This is a bit painful. I merged this change, but realized we need to
backport the fix to v49. However:
- we don't want to have two versions of the migration (one with a v49
id, one with a v50 id) because then if someone upgrades to 50, then
downgrades to 49, the `rollback` will run and change the type back,
leading to a bug.
- we don't want to push a v51 changeSet ID to v49 or v50, because we
give the user a helpful notice when their database needs a downgrade.
We do this by checking for the latest *executed* migration in the
database and comparing it to the latest migration that Liquibase knows
about, and making sure the executed isn't bigger than the known (e.g.
you can't have executed a v51 migration if it isn't in the local
migration yaml). That would all work fine, except that then we want to
tell you how to downgrade your database, and we use the latest-executed
version for that. So if, for example, someone upgraded from 48 to 49 and
got a v51 changeset, then downgraded back to 48, they would get an error
telling them to run the *v51* jar to roll back their DB.
In this case though, I think it's fine to just move the migration around
to v49, then we can backport it to 49 and 50.
* [MBQL lib] Reject broken uses of `offset` in expressions, filters (#42662)
- Using `offset` in custom expressions is supported only when there is
an order-by defined on that stage.
- Using `offset` in custom filters is not supported at all, currently.
* Support `Offset()` in custom columns [frontend] (#42326)
* Only nest expressions referenced in breakouts or aggregations
* Support Offset() as expression with no breakouts
* Test fixes 🔧
* Oracle test update
* Improved Oracle test
* Test update 🔧
* Fix busted test
* Add naive support for Offset() in expressions
* Introduce MBQLClauseFunctionReturnType
* Add "window" to MBQLClauseFunctionReturnType and use it for the offset function
* Handle offset type inference
* Remove unused export
* Use "expression" instead of "window" return type
- Rename identifiers in isCompatible
- Make isCompatible accept a clause object instead of just the type
- Handle offset as a special case in isCompatible
* Use any type
* Rename expectedArgumentType to expectedType
* Format code
* Sort types
* Do not suggest offset function in filters
* Fix offset not working in case
* Revert "Do not suggest offset function in filters"
This reverts commit e63790b780.
* Fix order of adjustments
---------
* Disable offsets in filters expressions
* Group existing aggregations-specific tests
* Remove repro for a closed issue
* Use findByText instead of contains
* Add a test for filter expressions
* Add a test for aggregation expressions suggestions
* Disable offsets in filters expressions (#42755)
* Add a test for custom column suggestions
* Add "typing" to enterCustomColumnDetails
* Use enterCustomColumnDetails, improve assertions
* Add more assertions
* Optimize queries
* Add typing for offset expressions
* Add a repro for metabase#42764
* Add a TODO
* Add a TODO
* Add a TODO
* Use NumericLiteral
* Post-merge fixes
* Update test
* Add tests for other custom expressions
* Test drills
* Format code
* Update test name
* Add an assertion
* Add assertions for the preview
* Unskip fixed issue
---------
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
Co-authored-by: Braden Shepherdson <braden@metabase.com>
Co-authored-by: Cam Saul <github@camsaul.com>
Co-authored-by: Cam Saul <1455846+camsaul@users.noreply.github.com>
* Set scrollToLastColumn ui control from column extract action
* Scroll to last column when the uiControl is set
* Scroll to last column when combining a column
* Scroll to last column when combining a column via the + shortcut
* Scroll to last column when extracting a column via the + shortcut
* Remove all references to settingsSyncOptions
* Check that the column is added at the end
* Disable scrollToLastColumn once used
* Add test that verifies scrollToLastColumn ui control has been reset
Co-authored-by: Romeo Van Snick <romeo@romeovansnick.be>
2024-05-17 15:07:00 +00:00
2324 changed files with 326050 additions and 160522 deletions
malli.core/explainer{:message"Use metabase.util.malli.registry/explainer instead of malli.core/explainer"}
me.flowthing.pp/pprint{:message"Use metabase.util.log instead of me.flowthing.pp/pprint"}
medley.core/random-uuid{:message"Use clojure.core/random-uuid instead of medley.core/random-uuid"}
metabase.driver/database-supports?{:message"Use metabase.driver.util/supports? instead of metabase.driver/database-supports?"}
metabase.lib.equality/find-column-indexes-for-refs{:message"Use lib.equality/closest-matches-in-metadata or lib.equality/find-closest-matches-for-refs instead of lib.equality/find-column-indexes-for-refs"}
metabase.test/with-temp*{:message"Use mt/with-temp instead of mt/with-temp*"}
toucan2.tools.with-temp{:message"Use mt/with-temp instead of t2.with-temp/with-temp"}}
@@ -135,7 +136,7 @@
;; namespace(s). This is mostly a temporary placeholder until we go in and create module namespaces, which means
;; you should go create one.
;;
;; 2. A set of namespace symbols. All namespaces in other modules will only be allowed ;; to use namespaces from
;; 2. A set of namespace symbols. All namespaces in other modules will only be allowed to use namespaces from
;; this set. Ideally this set should only have one namespace, but restricting it to a set of several is still
;; better than `:any`.
;;
@@ -162,6 +163,7 @@
metabase.async#{metabase.async.streaming-response
metabase.async.streaming-response.thread-pool
metabase.async.util}; TODO -- consolidate these into a real API namespace.
metabase.audit#{metabase.audit}
metabase.bootstrap#{metabase.bootstrap}
metabase.cmd#{}; there are no namespaces here since you shouldn't be using this in any other module.
comment='@${{ github.event.pull_request.user.login }} You have modified embedding SDK code. Please make sure the PR title follows [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) style.
Here are all the supported types that will be shown in the changelog:
- `feat`
- `fix`
- `perf`
These types `docs`, `style`, `refactor`, `test`, `build`, and `ci` are also encouraged for non-changelog related tasks.
Please also make sure to include `sdk` scope, otherwise, the changelog will not include this task.
run:echo "Tag '${{ env.tag }}' already exists. If you expect to run this workflow with the same tag, please remove the tag first and rerun this workflow again."
if [ $(git tag -l | grep $tag$) ]
then
echo "::error::Tag '${{ env.tag }}' already exists. If you expect to run this workflow with the same tag, please remove the tag first and rerun this workflow again."
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.