Compare commits

..

354 Commits

Author SHA1 Message Date
Thomas Schmidt
349f48ce44 Basic sql generation draft 2025-03-26 17:03:56 +01:00
Thomas Schmidt
b08b714b0e Do not convert tool names (#55505) 2025-03-24 07:39:43 +01:00
Tamás Benkő
523cd3812b Always set :queryable-foreign-key-tables for models 2025-03-20 22:45:46 +03:00
Tamás Benkő
d67c024717 Remove unused schema definition 2025-03-20 22:45:09 +03:00
Tamás Benkő
e52be87e3a Convert history sent by the FE 2025-03-20 22:44:39 +03:00
metamben
4f83b5eff3 Merge branch 'master' into metabot-v3-main 2025-03-20 21:42:49 +03:00
metamben
02ef883215 Expose table references for metrics (#55456)
Fixes BOT-66
2025-03-20 21:39:37 +03:00
metamben
a654c4025b Expose field values for metrics, models, and tables (#55413)
Fixes BOT-69
2025-03-20 21:32:33 +03:00
metamben
8c236418de Normalize metadata IDs (#55383)
Fixes BOT-68
2025-03-20 21:26:20 +03:00
metamben
ffb7af68c8 Keep history including the initial user message (#55473) 2025-03-20 21:25:25 +03:00
metamben
5333eeda6b Merge branch 'master' into metabot-v3-main 2025-03-19 20:53:04 +03:00
metamben
9981e3db93 Merge branch 'master' into metabot-v3-main 2025-03-18 22:39:39 +03:00
Tamás Benkő
93ab435a9d Merge remote-tracking branch 'origin/master' into metabot-v3-main
# Conflicts:
#	.github/workflows/uberjar.yml
#	.loki/reference/chrome_laptop_Design_System_Icons_List.png
#	frontend/src/metabase/query_builder/components/view/View/ViewNativeQueryEditor/ViewNativeQueryEditor.jsx
2025-03-18 22:38:59 +03:00
metamben
805f066b8c Add answer source tool (#55172) 2025-03-18 12:36:57 +03:00
Tamás Benkő
8af934fecd fix: Allow strings as tool call arguments 2025-03-17 15:07:51 +03:00
metamben
79a24d5800 Merge branch 'master' into metabot-v3-main 2025-03-17 14:40:56 +03:00
Tamás Benkő
ccc58d6578 Merge remote-tracking branch 'origin/master' into metabot-v3-main 2025-03-16 18:00:15 +03:00
Tamás Benkő
c06e62576b Merge remote-tracking branch 'origin/master' into metabot-v3-main
# Conflicts:
#	.loki/reference/chrome_laptop_Design_System_Icons_List.png
2025-03-14 10:59:57 +03:00
metamben
2aec76218d Add query model tool (#55089)
Fixes BOT-61
2025-03-14 10:46:52 +03:00
metamben
c14d51b22e Add support for temporal extraction filters (#55037)
* Add support for temporal extraction filters

Fixes BOT-60.

Actually, both extraction and bucketing is supported. The following values can be provided in the filter's `bucket`
property:

```
millisecond
second
minute
hour
day
week
month
quarter
year
second-of-minute
minute-of-hour
hour-of-day
day-of-week
day-of-month
day-of-year
week-of-year
month-of-year
quarter-of-year
year-of-era
```

The values of the form x-of-y result in extraction, the other result in truncation. The extractions `week-of-year` and
`day-of-week` have ISO semantics.

* Placate kondo
2025-03-13 20:41:53 +03:00
metamben
ef69ec3129 Merge branch 'master' into metabot-v3-main 2025-03-12 14:52:36 +03:00
Tamás Benkő
4fee5cd3c4 Update expected native query result text 2025-03-11 23:48:06 +03:00
Tamás Benkő
644f935654 Merge remote-tracking branch 'origin/master' into metabot-v3-main 2025-03-11 23:11:48 +03:00
Tamás Benkő
2e6734d105 Merge remote-tracking branch 'origin/master' into metabot-v3-main 2025-03-11 22:55:57 +03:00
metamben
1db581dd1b Merge branch 'master' into metabot-v3-main 2025-03-10 15:18:55 +03:00
metamben
71d913d8d1 Merge branch 'master' into metabot-v3-main 2025-03-09 14:11:37 +03:00
metamben
51ac49727e Merge branch 'master' into metabot-v3-main 2025-03-08 01:56:24 +03:00
metamben
806d623462 Remove dummy tools and Metabot loop related stuff (#54812)
* Remove dummy tools and clean up
2025-03-08 01:56:13 +03:00
metamben
8f62da32cc Merge branch 'master' into metabot-v3-main 2025-03-07 13:49:24 +03:00
metamben
edd7bc6a9c Distinguish temporal types and add semantic type (#54741)
Fixes BOT-56.
2025-03-07 13:49:10 +03:00
metamben
8f62fb33a5 Merge branch 'master' into metabot-v3-main 2025-03-06 17:55:00 +03:00
metamben
15f333d81f Add type agnostic and disjunctive filters (#54731)
Fixes BOT-54.
2025-03-06 17:54:51 +03:00
metamben
2543a6af72 Merge branch 'master' into metabot-v3-main 2025-03-06 15:01:23 +03:00
Tamás Benkő
961a73ff4c Merge remote-tracking branch 'origin/master' into metabot-v3-main 2025-03-05 21:36:59 +03:00
Tamás Benkő
383591f75b Update icon list in Loki 2025-03-05 21:36:19 +03:00
Tamás Benkő
2e1c2d7205 Merge remote-tracking branch 'origin/master' into metabot-v3-main 2025-03-05 17:12:37 +03:00
Tamás Benkő
57ca9f0aac Merge remote-tracking branch 'origin/master' into metabot-v3-main 2025-03-05 17:10:36 +03:00
metamben
4c1fe3d767 Merge branch 'master' into metabot-v3-main 2025-03-04 14:27:27 +03:00
Tamás Benkő
8c758e3674 Merge remote-tracking branch 'origin/master' into metabot-v3-main 2025-03-03 23:56:37 +03:00
Tamás Benkő
2bf18a43ec Send "user_is_viewing" to the agent in the context 2025-03-03 23:55:44 +03:00
metamben
4846e8a94b Merge branch 'master' into metabot-v3-main 2025-03-03 19:44:48 +03:00
metamben
a62e4e10ef Merge branch 'master' into metabot-v3-main 2025-03-03 18:31:54 +03:00
metamben
9778f8c12b Convert dummy tools to normal tools (#54519)
Fixes BOT-30
Fixes BOT-31
Fixes BOT-32
Fixes BOT-33
Fixes BOT-34
Fixes BOT-35
2025-03-03 18:31:43 +03:00
metamben
486d792c9a Remove V1 metabot agent endpoint (#54380)
* Remove V1 metabot agent endpoint

Fixes BOT-36
2025-03-03 17:18:23 +03:00
Tamás Benkő
4975f725a9 Merge remote-tracking branch 'origin/metabot-bot-47-tool-api-endpoint-tests' into metabot-v3-main 2025-03-03 17:09:45 +03:00
Tamás Benkő
b5e38e84b5 Merge remote-tracking branch 'origin/master' into metabot-v3-main 2025-03-03 17:06:48 +03:00
Tamás Benkő
ce62f30ffc Merge remote-tracking branch 'origin/master' into metabot-v3-main 2025-03-03 17:06:16 +03:00
Tamás Benkő
8374ed8450 Fix namespace 2025-02-28 23:37:38 +03:00
metamben
e2660bf3ef Merge branch 'master' into metabot-v3-main 2025-02-28 23:28:36 +03:00
Alexander Polyankin
08ebf5a3e3 Merge remote-tracking branch 'origin/metabot-v3-main' into metabot-v3-main 2025-02-28 15:20:15 -05:00
Alexander Polyankin
485df761d0 Use CSS colors 2025-02-28 15:20:06 -05:00
Alexander Polyankin
f2f4f57866 Use CSS colors 2025-02-28 15:19:45 -05:00
Alexander Polyankin
3abeeb9ab2 Use CSS colors 2025-02-28 15:15:56 -05:00
metamben
cb22a6bcf4 Merge branch 'master' into metabot-v3-main 2025-02-28 23:11:50 +03:00
Tamás Benkő
ad34b73d94 Placate clj-kondo 2025-02-28 22:49:37 +03:00
metamben
351daa8851 Merge branch 'master' into metabot-v3-main 2025-02-28 16:46:47 +03:00
Tamás Benkő
61b153a918 Run tests with the metabot-v3 feature enabled 2025-02-27 18:32:27 +03:00
Tamás Benkő
d1b0bf48fd Add V2 tool API endpoint tests
Fixes BOT-47.
2025-02-27 18:05:43 +03:00
metamben
1671d222cd Merge branch 'master' into metabot-v3-main 2025-02-27 17:09:46 +03:00
metamben
c0ef616c6d Merge branch 'master' into metabot-v3-main 2025-02-26 16:27:15 +03:00
Ngoc Khuat
78743ba645 fix whitespace 🤦 2025-02-26 15:32:03 +07:00
Ngoc Khuat
01e98249a2 fix invalid reducers 2025-02-26 15:13:46 +07:00
Ngoc Khuat
1dabc6ccbb Merge branch 'master' into metabot-v3-main 2025-02-26 14:55:59 +07:00
metamben
752287a56d Merge branch 'master' into metabot-v3-main 2025-02-25 20:23:55 +03:00
metamben
865d8a0d81 Generate site-wide session tokens for the AI service (#54269)
* Generate site-wide session tokens for the AI service

Fixes BOT-38.

This PR also allows the V2 tool API endpoints to be used by other existing authentication methods like API keys and
Metabase sessions.
2025-02-25 20:23:42 +03:00
metamben
26dc17b31d Merge branch 'master' into metabot-v3-main 2025-02-25 16:53:05 +03:00
metamben
2f6a39b40e Merge branch 'master' into metabot-v3-main 2025-02-24 22:10:14 +03:00
Tamás Benkő
dfab957510 Fix tests 2025-02-24 21:15:30 +03:00
metamben
ea3b49aa0a Merge branch 'master' into metabot-v3-main 2025-02-24 18:48:30 +03:00
metamben
5d81a60a64 Introduce V2 tool endpoints (#54192) 2025-02-24 18:48:17 +03:00
metamben
10bd22a3b9 Merge branch 'master' into metabot-v3-main 2025-02-21 16:45:24 +03:00
Ngoc Khuat
1e75309a10 Merge branch 'master' into metabot-v3-main 2025-02-21 11:27:27 +07:00
metamben
68d24b95b4 Merge branch 'master' into metabot-v3-main 2025-02-20 18:46:51 +03:00
metamben
0574137a29 Merge branch 'master' into metabot-v3-main 2025-02-19 17:48:32 +03:00
metamben
455713606e Merge branch 'master' into metabot-v3-main 2025-02-18 17:44:02 +03:00
metamben
fcf86ea731 Skip NOT IN condition if there are no used-ids (#53845) 2025-02-18 17:43:46 +03:00
Mihael Konjević
168c055c8f Keep Metabot palette item stable between query changes (#53827) 2025-02-18 12:41:24 +01:00
Tamás Benkő
1e715f418b Merge remote-tracking branch 'origin/master' into metabot-v3-main 2025-02-17 18:46:06 +03:00
Tamás Benkő
5aca4ec11d Fix test 2025-02-17 18:45:41 +03:00
Tamás Benkő
27045cf9a7 Remove .clj-kondo/src/hooks/metabase/models/setting.clj changes 2025-02-17 17:06:36 +03:00
Tamás Benkő
d358286383 Merge remote-tracking branch 'origin/master' into metabot-v3-main
# Conflicts:
#	.clj-kondo/src/hooks/metabase/models/setting.clj
2025-02-17 17:03:06 +03:00
metamben
c19b8310dd Fix module dependencies (#53763)
* Fix module dependencies
2025-02-17 16:03:54 +03:00
metamben
3b535a542c Add find-outliers tool endpoint (#53624)
* Add find-outliers tool endpoint
* Also fix the decoding of find-metric tool API results.
* Add tests for explicit query handling.
* Add context when running the query for finding outliers
2025-02-14 17:00:24 +03:00
metamben
291d9e12e1 Merge branch 'master' into metabot-v3-main 2025-02-14 16:56:41 +03:00
Thomas Schmidt
bd1da7cdb9 Include query in request to ai-service (#53745) 2025-02-14 14:48:37 +01:00
Tamás Benkő
96cd8ba5b7 Appease the module checker even more vigorously 2025-02-14 00:44:10 +03:00
Tamás Benkő
b5e2e66b27 Skip SQL fixer e2e test and mute the module linter for now 2025-02-14 00:17:44 +03:00
Tamás Benkő
5b8a479fe4 Merge remote-tracking branch 'origin/master' into metabot-v3-main 2025-02-14 00:13:41 +03:00
Tamás Benkő
bf5df2cd38 Merge remote-tracking branch 'origin/master' into metabot-v3-main 2025-02-13 20:10:58 +03:00
Tamás Benkő
98332749de Address linter issues 2025-02-13 20:09:29 +03:00
metamben
827c0c2eab Merge branch 'master' into metabot-v3-main 2025-02-13 18:21:44 +03:00
Tamás Benkő
e74c254612 Merge remote-tracking branch 'origin/master' into metabot-v3-main 2025-02-13 18:20:33 +03:00
Tamás Benkő
04b7020aef Merge remote-tracking branch 'origin/master' into metabot-v3-main 2025-02-13 18:19:51 +03:00
Romeo Van Snick
edb6b83feb Fix loop in codemirror (#53599)
* Correctly compare selections

* Add reproduction for QUE-614

* Use github issue number in regression

* Unnest issue from other issue reproduction

* Log in before test
2025-02-13 15:16:42 +01:00
Thomas Schmidt
10550112f5 Only send path for xray dashboard not full link (#53594) 2025-02-13 12:34:55 +01:00
Mihael Konjevic
5c99127943 Merge branch 'master' into metabot-v3-main 2025-02-12 20:56:08 +01:00
Mihael Konjević
5b5ab1fa16 Metabot v3 command palette item (#53568)
* Initial implementation of Metabot palette action

* Use default styles, correct ordering of palette items, tests

* Setup tests with metabot_v3 feature enabled

* Recompute "mainReducers" on demand in the renderWithProviders function

This solves a race condition where reducers were computed before the
enterprise plugins were required, which made it impossible to test their
behavior (in this case in PaletteResult tests). makeMainReducers function is
added which computes the reducers object, and the previous behavior is
preserved (mainReducers const points to the output of the makeMainReducers fn)
2025-02-12 19:40:26 +01:00
metamben
76d904ba3a Merge branch 'master' into metabot-v3-main 2025-02-12 19:21:20 +03:00
metamben
ca3698711b SQL fixer: generate schema_ddl (#53381)
* Generate ddl_schema
2025-02-12 19:21:05 +03:00
metamben
271fab9a30 Add find-metric tool endpoint (#53492)
* Add find-metric tool endpoint

* Add missing result conversion functions
2025-02-12 19:20:01 +03:00
metamben
ff0ed9be47 Add filter-records tool endpoint (#53561)
* Add filter-records tool endpoint

* Adapt tests
2025-02-12 18:40:00 +03:00
Tamás Benkő
a69afcb929 Merge remote-tracking branch 'origin/master' into metabot-v3-main 2025-02-12 17:59:55 +03:00
Tamás Benkő
c4b2f29b0f Remove unused imports 2025-02-11 20:02:45 +03:00
Tamás Benkő
44a15f3a77 Merge remote-tracking branch 'origin/master' into metabot-v3-main 2025-02-11 19:52:14 +03:00
Tamás Benkő
6630e72403 Stop re-using metabase.api.card/get-card
This function is private now, and it's also doing more than what we need right now.
2025-02-11 19:51:35 +03:00
Tamás Benkő
318e3e6b6f Merge remote-tracking branch 'origin/master' into metabot-v3-main 2025-02-11 19:10:06 +03:00
Thomas Schmidt
91948e24ab Redirect user when navigate_to property is present in agent response (#53518) 2025-02-11 16:36:04 +01:00
Tamás Benkő
6c20dc9c17 Merge remote-tracking branch 'origin/master' into metabot-v3-main 2025-02-11 17:54:38 +03:00
metamben
a6f8cc0919 Fix input encoding for the create-dashboard-subscription tool (#53486) 2025-02-10 23:45:17 +03:00
Alexander Polyankin
3ebda589f5 Merge remote-tracking branch 'refs/remotes/origin/master' into metabot-v3-main
# Conflicts:
#	frontend/src/metabase/query_builder/components/NativeQueryEditor/CodeMirrorEditor/extensions.tsx
2025-02-10 11:05:15 -05:00
Tamás Benkő
e51023ca43 Remove describeEE and onlyOnEE 2025-02-08 02:33:40 +03:00
metamben
44da1f4d67 Merge branch 'master' into metabot-v3-main 2025-02-07 23:24:46 +03:00
metamben
4a1c0c5029 Check table_id parameter in generate-insights tool (#53310) 2025-02-07 20:33:35 +03:00
metamben
f2792cdbfd Merge branch 'master' into metabot-v3-main 2025-02-07 18:47:20 +03:00
metamben
10c042dbdc Add create-dashboard-subscription and generate-insights endpoints (#53253)
* Add create-dashboard-subscription and generate-insights endpoints

* Rename session_id to conversation_id for the tool endpoints
2025-02-07 18:46:40 +03:00
metamben
7badf9005d Merge branch 'master' into metabot-v3-main 2025-02-07 16:39:14 +03:00
metamben
c3d157b9cd Merge branch 'master' into metabot-v3-main 2025-02-07 01:27:05 +03:00
Tamás Benkő
aeaa811281 Fix ns require formatting 2025-02-07 01:26:33 +03:00
metamben
d44226c7fa Merge branch 'master' into metabot-v3-main 2025-02-07 01:24:12 +03:00
Alexander Polyankin
c4578261a8 Fix SQL button tests (#53198) 2025-02-06 11:38:10 -05:00
metamben
ca0352d6ad Merge branch 'master' into metabot-v3-main 2025-02-06 17:03:41 +03:00
metamben
32dc706a39 Merge branch 'master' into metabot-v3-main 2025-02-05 17:40:15 +03:00
metamben
1b44cb8bb9 Implement query-metric tool endpoint (#53195) 2025-02-05 17:40:01 +03:00
Alexander Polyankin
9fa2f59d64 Autofix SQL button (#53153) 2025-02-04 11:32:49 -05:00
metamben
e62bfc2004 Merge branch 'master' into metabot-v3-main 2025-02-04 18:08:04 +03:00
metamben
95726e9d1b Add /api/ee/ai-fix-sql/fix endpoint (#53143)
* Add POST /api/ee/ai-sql-fixer/fix endpoint

* Never prettify the generated SQL
2025-02-04 18:07:06 +03:00
metamben
ec7595557a Merge branch 'master' into metabot-v3-main 2025-02-04 16:20:13 +03:00
metamben
da64908493 Merge branch 'master' into metabot-v3-main 2025-02-04 01:26:31 +03:00
metamben
9bd61f3557 Merge branch 'master' into metabot-v3-main 2025-02-03 21:26:55 +03:00
metamben
49a83e9f41 Merge branch 'master' into metabot-v3-main 2025-02-03 16:38:28 +03:00
metamben
bddc369c0d Merge branch 'master' into metabot-v3-main 2025-02-02 14:31:29 +03:00
metamben
baa6baf893 Merge branch 'master' into metabot-v3-main 2025-02-01 02:43:33 +03:00
metamben
c9fe612a50 Merge branch 'master' into metabot-v3-main 2025-01-31 18:13:39 +03:00
metamben
6480f7b841 Add standalone tests for the invite-user tool (#52984)
* Add standalone tests for the invite-user tool
2025-01-31 18:13:26 +03:00
Tamás Benkő
64fa2f2bc1 Merge remote-tracking branch 'origin/master' into metabot-v3-main 2025-01-31 18:07:36 +03:00
Tamás Benkő
4c690a44dc Merge remote-tracking branch 'origin/master' into metabot-v3-main 2025-01-31 18:07:13 +03:00
metamben
d74631e48b Merge branch 'master' into metabot-v3-main 2025-01-31 00:37:25 +03:00
metamben
c9f5a0e3b6 Merge branch 'master' into metabot-v3-main 2025-01-30 17:57:42 +03:00
metamben
123ec0e340 Add standalone filter-records tests (#52933)
* Add standalone filter-records tests
2025-01-30 17:57:33 +03:00
metamben
3929342ac0 Merge branch 'master' into metabot-v3-main 2025-01-30 00:39:04 +03:00
metamben
ed537dcbc7 Add standalone find-outliers tests (#52906)
* Add standalone find-outliers tests
2025-01-30 00:38:46 +03:00
Tamás Benkő
60b85b61ac Switch to api.macros/defendpoint 2025-01-29 18:23:57 +03:00
metamben
e3347d10a5 Merge branch 'master' into metabot-v3-main 2025-01-29 17:24:25 +03:00
Tamás Benkő
ca0cd2b5a5 Add back React import 2025-01-28 21:53:33 +03:00
metamben
10107fcf5c Merge branch 'master' into metabot-v3-main 2025-01-28 21:39:38 +03:00
Tamás Benkő
42aa717241 Add missing type keyword 2025-01-28 21:37:13 +03:00
Tamás Benkő
cb30ee8a4b Fix wrong merge 2025-01-28 21:30:14 +03:00
metamben
dee6da84cc Merge branch 'master' into metabot-v3-main 2025-01-28 21:16:32 +03:00
metamben
9d6db122ca Add tests for the query-metric tool (#52765)
* Ignore bucketing for non-temporal columns

This was implemented in #52271 but has been reverted by a bad merge.

* Use long names when generating columns

The standard names can easily result in duplicates which will certainly confuse the agent.

* Break long lines

* Add tests for the query-metric tool

* Handle the strange metric-id error cases explicitly
2025-01-28 21:16:14 +03:00
metamben
bdccec2d1f Add tests for the generate-insights tool (#52767) 2025-01-28 21:15:17 +03:00
metamben
653797622b Merge branch 'master' into metabot-v3-main 2025-01-28 16:55:33 +03:00
Tamás Benkő
77fb7c62fa Merge remote-tracking branch 'origin/master' into metabot-v3-main 2025-01-27 18:33:01 +03:00
Tamás Benkő
c1eb2559ec Merge remote-tracking branch 'origin/master' into metabot-v3-main 2025-01-27 18:32:33 +03:00
Aleksandr Lesnenko
8643fc3aef update import 2025-01-24 16:08:10 -05:00
metamben
67abded9e8 Add default_time_dimension_field_id and adhoc context handling (#52683)
* Add default_time_dimension_field_id and adhoc context handling

* Add show-results-to-user dummy call for adhoc queries in context
2025-01-24 23:33:49 +03:00
Tamás Benkő
dab935167b Fix formatting 2025-01-24 22:54:54 +03:00
metamben
087ad1a2b1 Merge branch 'master' into metabot-v3-main 2025-01-24 22:15:47 +03:00
metamben
c3cbeffc70 Merge branch 'master' into metabot-v3-main 2025-01-24 21:45:31 +03:00
Tamás Benkő
75b33df5bc Merge remote-tracking branch 'origin/master' into metabot-v3-main 2025-01-24 20:36:28 +03:00
metamben
b87ce054cb Switch the uberjar build branch to metabot-v3-main (#52678)
* Switch the build branch to metabot-v3-main

* Switch containerize_test_and_push_container too
2025-01-24 20:32:36 +03:00
metamben
89490c9a16 Merge branch 'master' into metabot-v3-main 2025-01-23 22:37:23 +03:00
metamben
9b78a0014f Return a friendly message when the number of round trips is exceeded (#52621)
Fixes #52591.
2025-01-23 22:36:57 +03:00
metamben
74c29161f3 Re-enable the filter-records tool (#52615)
It turns out that a fair number of our test cases rely on this tool, and they worked before, so it might be stable
enough.
2025-01-23 21:01:50 +03:00
metamben
59dcb7f115 Stop checking that parameter names are kebab-case (#52613) 2025-01-23 20:59:15 +03:00
metamben
2f817742e6 Consistently generate and look for query_id and not query-id (#52599) 2025-01-23 18:04:18 +03:00
metamben
be840bb7bd Prevent converting :structured-content to JSON twice (#52556) 2025-01-23 16:54:21 +03:00
metamben
086783dd68 Merge branch 'master' into metabot-v3-main 2025-01-23 16:42:51 +03:00
metamben
3cd107d6d3 Ignore result-columns for saved questions (#52535)
The result-columns of saved pivot questions are a mess, the names and the effective types don't match the values.
Instead of using these columns, we generate the columns from the dataset-query field.

The assumption is that the result-columns for saved questions (but not for models) don't play an important role.
2025-01-23 00:16:29 +03:00
metamben
2d47254583 Disable Metabot's filter-records tool (#52546)
Fixes #52499
2025-01-23 00:16:03 +03:00
Tamás Benkő
ec6033870c Merge remote-tracking branch 'origin/master' into metabot-v3-main 2025-01-23 00:05:13 +03:00
metamben
db9e5b1a49 Merge branch 'master' into metabot-v3-main 2025-01-22 23:14:38 +03:00
metamben
96c9346eda Use integers for report_id, strings for table_id (#52528) 2025-01-22 20:28:11 +03:00
metamben
57af6af23a Add standalone test for the create-dashboard-subscription tool (#52519) 2025-01-22 20:27:49 +03:00
metamben
8598fc59d5 Merge branch 'master' into metabot-v3-main 2025-01-22 16:54:50 +03:00
John Swanson
6bcbe2f35a Don't send queries to llm (#51599)
* Don't send queries to the LLM

This introduces `envelope/llm-message`, a private function that formats
messages for the LLM. Currently it does two things: removes `query`s and
json-encodes the structured content.

There's also `envelope/llm-history`, which replaces
`envelope/full-history` by providing the full history appropriate for
the LLM (instead of the full "internal" history for our own use).

* Rework confusing fn

Split this out into a few functions:

- `stringified-content` takes a message and returns the stringified
content (from `structured-content` or `content` as appropriate)

- `stringify-content` takes a message and returns a message with a
stringified `:content` form (and no `:structured-content`)

- `maybe-remove-query` takes a message and removes the `:query` if
necessary

Now `llm-message` just threads the message through `maybe-remove-query`
and then `stringify-content` rather than a confusing `cond->` form.
2025-01-21 21:11:04 +03:00
metamben
57af242b8f Disable invite-user tool (#52450) 2025-01-21 20:35:18 +03:00
metamben
22a3a5fe44 Merge branch 'master' into metabot-v3-main 2025-01-21 18:35:53 +03:00
metamben
8ddeee90f3 Support find-outliers for reports and queries in simple cases (#52414)
When the result has a temporal column, the first in the list is taken as the basis of the analysis.
2025-01-21 17:27:47 +03:00
metamben
74574ed2aa Merge branch 'master' into metabot-v3-second-try-rebased 2025-01-21 00:22:01 +03:00
metamben
d3804ab2bc Merge branch 'master' into metabot-v3-second-try-rebased 2025-01-20 21:11:53 +03:00
metamben
c76e64407b Merge branch 'master' into metabot-v3-second-try-rebased 2025-01-20 19:40:33 +03:00
metamben
7330ca489e Merge branch 'master' into metabot-v3-second-try-rebased 2025-01-20 17:05:33 +03:00
Mihael Konjević
7a0db83be5 Implement redirect reaction for Metabot (#52289)
* Implement redirect reaction for Metabot

Redirect reaction redirects the user to the results page generated by a tool.
Currently `query-metric` and `generate-insights` tools use this reaction.

This also introduces a concept of "terminating" reactions. When a tool returns a
terminating reaction, this tool call result is not sent back to the LLM, instead
it returns the envelope and the frontend handles any sideffects.
2025-01-20 13:26:31 +01:00
metamben
abdfa33583 Ignore archived metrics in the find-metric tool (#52322)
* Ignore archived metrics in the `find-metric` tool

* Use yarn.lock generated by yarn
2025-01-17 22:07:15 +03:00
Tamás Benkő
dee067ffff Merge remote-tracking branch 'origin/master' into metabot-v3-second-try-rebased 2025-01-17 19:33:57 +03:00
Tamás Benkő
8cbea88330 Fix formatting 2025-01-17 18:58:38 +03:00
metamben
d4944b478b Normalize tool error responses (#52230) 2025-01-17 18:02:28 +03:00
metamben
5603565c7c Look for field_granularity instead of field-granularity (#52271)
* Look for field_granularity instead of field-granularity

* Don't report an error when bucketing non-temporal columns

Just ignore the bucket instead.
2025-01-17 17:50:53 +03:00
metamben
fd44da515e Handle dashcards without card when creating subscriptions (#52248) 2025-01-17 17:50:41 +03:00
metamben
56419f6be9 Merge branch 'master' into metabot-v3-second-try-rebased 2025-01-17 16:44:16 +03:00
metamben
43dab70197 Merge branch 'master' into metabot-v3-second-try-rebased 2025-01-16 23:30:08 +03:00
metamben
26bdc4ca18 Distinguish structured and textual tool results (#51698)
* Distinguish structured and textual tool results
2025-01-16 15:58:33 +03:00
metamben
c17bbb8ca3 Merge branch 'master' into metabot-v3-second-try-rebased 2025-01-16 15:57:27 +03:00
metamben
d30c95d30b Merge branch 'master' into metabot-v3-second-try-rebased 2025-01-16 00:41:22 +03:00
metamben
ce8558949a Return metric IDs as plain integers instead of card__<id> (#52059)
See https://metaboat.slack.com/archives/C07SJT1P0ET/p1736237110786299?thread_ts=1735243354.390349&cid=C07SJT1P0ET for context.
2025-01-15 23:52:37 +03:00
Thomas Schmidt
d027f538f1 Update tool spec with examples (#52195)
* Update tool spec with examples

* Support string table_id values in filter-records

---------

Co-authored-by: Tamás Benkő <tamas@metabase.com>
2025-01-15 19:57:43 +03:00
metamben
13a73a6b9a Implement show-results-to-user tool (#52121) 2025-01-14 23:12:42 +03:00
metamben
80c322f8b1 Merge branch 'master' into metabot-v3-second-try-rebased 2025-01-14 16:56:04 +03:00
metamben
acc331c5fb Merge branch 'master' into metabot-v3-second-try-rebased 2025-01-13 21:43:31 +03:00
metamben
4edd922f10 Merge branch 'master' into metabot-v3-second-try-rebased 2025-01-13 21:06:06 +03:00
Tamás Benkő
6108f5d699 Merge remote-tracking branch 'origin/master' into metabot-v3-second-try-rebased 2025-01-13 19:20:40 +03:00
Tamás Benkő
83af40a49d Fix linter errors from wrong merge 2025-01-13 19:19:55 +03:00
Tamás Benkő
71d828be07 Fix require from wrong merge 2025-01-13 18:49:51 +03:00
metamben
129f1ce7f4 Merge branch 'master' into metabot-v3-second-try-rebased 2025-01-13 18:24:23 +03:00
Tamás Benkő
0e0aac3f46 Merge remote-tracking branch 'origin/master' into metabot-v3-second-try-rebased 2025-01-13 18:11:23 +03:00
metamben
474ecbeec5 Properly report unknown users from create-dashboard-subscription (#51662) 2025-01-13 17:41:26 +03:00
metamben
1b3ed70dec Fix typo (#51702) 2024-12-31 15:03:26 +03:00
metamben
91abc51ec5 Merge branch 'master' into metabot-v3-second-try-rebased 2024-12-31 12:37:47 +03:00
John Swanson
1c02aebe07 Find outliers tool (#51534)
* QOL improvements for the REPL namespace

* Find outliers tool

There are also some unrelated changes I threw in here:

- remove a bit of unnecessary serialization of output (we do this
automatically now)

- delete a stale tool that is no longer part of toolbox v1 (I noticed
the LLM trying to call it)

* CR resp

* Userland query

* Userland query *with default constraints*

* kondo
2024-12-27 12:17:41 -08:00
John Swanson
0af736a234 Adjust response format for invite-user (#51536) 2024-12-27 12:17:23 -08:00
metamben
0c3b93131c Generate field_ids by index (#51606)
* Identify fields by their index

* Generate short field IDs

* Truncate columns to day when filtering without time component

* Decode tool history contents when searching for generated queries
2024-12-27 01:04:37 +03:00
metamben
ada0e86826 Merge branch 'master' into metabot-v3-second-try-rebased 2024-12-23 14:33:12 +03:00
Alexander Polyankin
160be30f2e Fix tests 2024-12-22 15:11:14 -05:00
Alexander Polyankin
f6d87b174d Fix tests 2024-12-22 15:10:57 -05:00
metamben
1d8350e6b7 Merge branch 'master' into metabot-v3-second-try-rebased 2024-12-21 02:29:19 +03:00
metamben
88605995e1 Merge branch 'master' into metabot-v3-second-try-rebased 2024-12-20 17:24:51 +03:00
Tamás Benkő
1516add0f2 Handwave FE merge 2024-12-20 00:03:19 +03:00
Tamás Benkő
751c628779 Use metabase.util.json instead of cheshire.core 2024-12-20 00:03:19 +03:00
Tamás Benkő
1c7d841199 Fix bad var ref 2024-12-20 00:03:19 +03:00
Alexander Polyankin
73c8f0e87c Add user_is_viewing to FE->BE context (#51482)
* Add user_is_viewing to FE->BE context

* Use the context provided by the FE

* Fix bad merge

---------

Co-authored-by: Tamás Benkő <tamas@metabase.com>
2024-12-20 00:03:19 +03:00
metamben
5306cc9e37 Send current_user_time in LLM context (#51477) 2024-12-20 00:03:19 +03:00
John Swanson
1545b358dd green CI? (#51459)
* green CI?

* don't swap in test context for dummy tools

* convert non-string tool responses to json

* CR feedback and fix circular dep

* clean up reaction handling in envelope

* fix another test
2024-12-20 00:03:19 +03:00
metamben
2d68c5569e Register generate-insights and invite-user (#51470) 2024-12-20 00:03:19 +03:00
metamben
e0f0fac373 Implement filter-records (#51452)
* Implement filter-data

* Switch dummy tools to new tool parameters

* Rename filter-data to filter-records

* Register find-metric tool

* Select only the fields from table we really need
2024-12-20 00:03:19 +03:00
John Swanson
c98477e607 fix dummy tools (#51457) 2024-12-20 00:03:19 +03:00
John Swanson
974424174c Rework flow to allow for a query store (#51425)
* Rework flow to allow for a query store

Dummy tools need to be able to store queries for later use. Right now
dummy tools work differently than regular tools: they can
only (effectively) return an `output`. They can't change context, they
can't return reactions, etc.

I was imagining that when we wanted to add queries to the query store,
we'd do so in the `envelope`, and probably do this in a functional way
by returning `:queries` or a `:query-id->query` map from the tool call
itself. Then, we'd write a function in `envelope` that would basically
just store this state for later retrieval.

The issue we had is that (before this commit) we don't actually invoke
dummy tools in a way that changes the envelope - we just invoke them as
needed before sending the history to the LLM.

So, this PR reworks things a bit:

- we invoke the dummy tool calls once, immediately after creating the
envelope

- right now, this only means that we store the results of those tool
calls in the env, for later retrieval to send to the LLM when we are
actually making a call. (we don't just store the tool calls in normal
history, because we don't want to return them to the FE when we
respond).

- in the immediate future though, we'll want to *also*, in addition to
adding tool `output` to the response, handle other tool results. One
thing could be changing the context in response to a dummy tool call -
which seems maybe a bit crazy, but on the other hand maybe dummy tool
calls should be able to do anything a regular tool call can do?

- more immediately, we'll probably want to handle a new key from tool
invocations, `:query-id->query-map`, that will store queries for later
retrieval.

* Add a function to find a query

* refactor `invoke-tool` to take entire env

I also changed the response handling so that if you don't return
`:context` it's ignored, rather than associng `nil` onto the existing
env.

* actually use `find-query` in generate-insights
2024-12-20 00:03:19 +03:00
Thomas Schmidt
2df3d84fed Sync tool schemas with ai-service (#51446) 2024-12-20 00:03:19 +03:00
metamben
f1d23fa5d9 Implement run-query (#51420)
* Implement run-query as dummy tool

* Return kebab-cased keys from query-metric tool

The keys will be automatically converted to snake_case in the message.

* Skip context processing of the previous attempt
2024-12-20 00:03:19 +03:00
metamben
de6b854244 Implement query-metrics and get-metric-details tools (#51315)
* Implement ISO day of week extraction (#50475)

* Implement ISO day of week extraction

Fixes #50424

* Support display-name for ISO get-day-of-week exclusion expressions (#50606)

* Support display-name for ISO get-day-of-week exclusion expressions

Fixes #50574

* Support extraction expressions in filter-args-display-name (#50694)

Fixes #50670

* Add get-metric-details and query-metric tools

* Pass history to tool invocations

* Tool implementations (#51244)

* Skeleton for `generate-insights` tool

For now I just implemented the easy things: tables, metrics, and
reports (which I think equal cards on our end?), and totally skipped
`query_id`.

* Pull in all the tools from ai-service

* `get-report-details` and `get-metric-details`

* find metrics using the select metrics endpoint

* reimplement invite-user tool

skipping all confirmation for now

* Update enterprise/backend/src/metabase_enterprise/metabot_v3/tools/generate_insights.clj

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

---------

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

* Align table and report details with metric details

* Fix tool JSON schemata

---------

Co-authored-by: John Swanson <john.swanson@metabase.com>
2024-12-20 00:03:19 +03:00
metamben
b6af5b1a2e Implement get-table-details dummy tool (#51143)
* Implement get-table-details dummy tool

* Only return readable tables
2024-12-20 00:03:19 +03:00
metamben
15adc8994c Send auth token via request header instead of the body (#51174)
* Send auth token via request header instead of the body

* Remove instance-info from request body
2024-12-20 00:03:19 +03:00
metamben
da69aaf7d0 Implement dashboard subscription using dummy tools (#51074) 2024-12-20 00:03:19 +03:00
Alexander Polyankin
e1c39f6b5e Add dashboard_id to FE->BE context (#51027) 2024-12-20 00:03:19 +03:00
Sloan Sparger
c76b952c5b [Metabot] Fix e2e failures (#50112)
* force ci run

* test fix

* fix people tests

* revert router changes

* removes force enabling metabot that does not do anything

* disable metabot tests since the token feature was disabled

* change

* change

* fix documentation
2024-12-20 00:03:19 +03:00
Alexander Polyankin
0ad961096b Throw error 2024-12-20 00:03:19 +03:00
Alexander Polyankin
8fac6159ec Hacky multi-tool chain (#50179)
* Hacky multi-tool chain

* Hacky multi-tool chain
2024-12-20 00:03:19 +03:00
Alexander Polyankin
409d7a2251 Add granularity 2024-12-20 00:03:19 +03:00
Alexander Polyankin
325c6a17a1 filter-data and aggregate-data tools (#50168) 2024-12-20 00:03:19 +03:00
Sloan Sparger
6a6312c8e2 [Metabot] UI Access improvements (#49859)
* auto close the metabot input on route changes

* prevents auto-closing metabot if the base path segment is the same

* pr feedback

* makes metabot only visible while the user is signed in

* add test coverage

* fix infinite type instantiation errors
2024-12-20 00:03:19 +03:00
Alexander Polyankin
3a98e2d44f Bring back change-chart-appearance tool (#50104) 2024-12-20 00:03:19 +03:00
Alexander Polyankin
a604ab351b change-query cleanup (#49921) 2024-12-20 00:03:19 +03:00
Sloan Sparger
560f2e1455 [Metabot] Auto-close on route change (#49810)
* auto close the metabot input on route changes

* prevents auto-closing metabot if the base path segment is the same

* pr feedback
2024-12-20 00:03:19 +03:00
Sloan Sparger
a283463d05 [Metabot] Sessions (#49785)
* base e2e tests

* unit tests

* fe implementation for sessions

* BE: Send back a `session_id` if missing

* change to uuid

* send the session id along to ai-service

* pr feedback

* require `session_id` to be passed, and be a UUID

* backend: send a dummy session id to unbreak stats (#49894)

* backend: send a dummy session id to unbreak stats

* preserve session-id across calls

one shortcoming: it does not preserve the session id across FE
invocations. Need to send it back and the FE parrot it back, and the
envelope create function should reuse that when present

```
{
  "messages" : [ {
    "role" : "user",
    "content" : "filter to where the state is CA"
  } ],
  "context" : {...}
  "tools" : [,,,],
  "instance_info" : {,,,},
  "user_id" : 1,
  "session_id" : "6e67dafe-30e6-4b0a-8387-5dc5968e7f28" // <----
}

"LLM -----------------> BE"
{
  "message" : {
    "content" : "",
    "role" : "assistant",
    "tool_calls" : [ {
      "id" : "call_NpA5WEA2gzrRDWt7pC8i0GZR",
      "name" : "change_query",
      "arguments" : "{\"relative_date_filters\":null,\"specific_date_filters\":null,\"boolean_filters\":null,\"breakouts\":null,\"string_filters\":[{\"column\":\"State\",\"operator\":\"=\",\"value\":\"CA\"}],\"order_bys\":null,\"limits\":null,\"number_filters\":null,\"aggregations\":null}"
    } ]
  },
  "metadata" : {
    "usage" : {
      "model" : "gpt-4o-mini-2024-07-18",
      "total" : 7404,
      "prompt" : 7335,
      "completion" : 69,
      "estimated_costs" : 0.0011519999999999998
    }
  }
}

"BE -----------------> LLM"
{
  "messages" : [ {
    "role" : "user",
    "content" : "filter to where the state is CA"
  }, {
    "content" : "",
    "role" : "assistant",
    "tool_calls" : [ {
      "id" : "call_NpA5WEA2gzrRDWt7pC8i0GZR",
      "name" : "change_query",
      "arguments" : "{\"order-bys\":null,\"specific-date-filters\":null,\"breakouts\":null,\"boolean-filters\":null,\"relative-date-filters\":null,\"string-filters\":[{\"column\":\"State\",\"operator\":\"=\",\"value\":\"CA\"}],\"number-filters\":null,\"limits\":null,\"aggregations\":null}"
    } ]
  }, {
    "role" : "tool",
    "tool_call_id" : "call_NpA5WEA2gzrRDWt7pC8i0GZR",
    "content" : "success"
  } ],
  "context" : {},
  "tools" : []..,
  "instance_info" : {,,,}
  },
  "user_id" : 1,
  "session_id" : "6e67dafe-30e6-4b0a-8387-5dc5968e7f28" // <---
}
```

This session_id above is missing in the BE -> FE and then on subsequent FE -> BE payloads

---------

Co-authored-by: dan sutton <dan@dpsutton.com>

* right some wrongs from merge conflict resolutions

* fix

* fix

---------

Co-authored-by: John Swanson <john.swanson@metabase.com>
Co-authored-by: dan sutton <dan@dpsutton.com>
2024-12-20 00:03:19 +03:00
Sloan Sparger
ecc3e7ba0b [Metabot] Initial e2e + unit tests (#49732)
* base e2e tests

* unit tests

* pr feedback

* run metabot tests

* force enable metabot for testing

* test something in ci

* test something in ci 2

* test something in ci 3

* test something in ci 4

* test something in ci 5

* fix for now
2024-12-20 00:03:19 +03:00
John Swanson
57f58d6009 backend: send a dummy session id to unbreak stats (#49894)
* backend: send a dummy session id to unbreak stats

* preserve session-id across calls

one shortcoming: it does not preserve the session id across FE
invocations. Need to send it back and the FE parrot it back, and the
envelope create function should reuse that when present

```
{
  "messages" : [ {
    "role" : "user",
    "content" : "filter to where the state is CA"
  } ],
  "context" : {...}
  "tools" : [,,,],
  "instance_info" : {,,,},
  "user_id" : 1,
  "session_id" : "6e67dafe-30e6-4b0a-8387-5dc5968e7f28" // <----
}

"LLM -----------------> BE"
{
  "message" : {
    "content" : "",
    "role" : "assistant",
    "tool_calls" : [ {
      "id" : "call_NpA5WEA2gzrRDWt7pC8i0GZR",
      "name" : "change_query",
      "arguments" : "{\"relative_date_filters\":null,\"specific_date_filters\":null,\"boolean_filters\":null,\"breakouts\":null,\"string_filters\":[{\"column\":\"State\",\"operator\":\"=\",\"value\":\"CA\"}],\"order_bys\":null,\"limits\":null,\"number_filters\":null,\"aggregations\":null}"
    } ]
  },
  "metadata" : {
    "usage" : {
      "model" : "gpt-4o-mini-2024-07-18",
      "total" : 7404,
      "prompt" : 7335,
      "completion" : 69,
      "estimated_costs" : 0.0011519999999999998
    }
  }
}

"BE -----------------> LLM"
{
  "messages" : [ {
    "role" : "user",
    "content" : "filter to where the state is CA"
  }, {
    "content" : "",
    "role" : "assistant",
    "tool_calls" : [ {
      "id" : "call_NpA5WEA2gzrRDWt7pC8i0GZR",
      "name" : "change_query",
      "arguments" : "{\"order-bys\":null,\"specific-date-filters\":null,\"breakouts\":null,\"boolean-filters\":null,\"relative-date-filters\":null,\"string-filters\":[{\"column\":\"State\",\"operator\":\"=\",\"value\":\"CA\"}],\"number-filters\":null,\"limits\":null,\"aggregations\":null}"
    } ]
  }, {
    "role" : "tool",
    "tool_call_id" : "call_NpA5WEA2gzrRDWt7pC8i0GZR",
    "content" : "success"
  } ],
  "context" : {},
  "tools" : []..,
  "instance_info" : {,,,}
  },
  "user_id" : 1,
  "session_id" : "6e67dafe-30e6-4b0a-8387-5dc5968e7f28" // <---
}
```

This session_id above is missing in the BE -> FE and then on subsequent FE -> BE payloads

---------

Co-authored-by: dan sutton <dan@dpsutton.com>
2024-12-20 00:03:19 +03:00
Alexander Polyankin
782d167f75 Query tool - no context, fetching metadata on demand (#49812) 2024-12-20 00:03:19 +03:00
John Swanson
a90addd18c Move user_id out of instance_info (#49787) 2024-12-20 00:03:19 +03:00
Aleksandr Lesnenko
9c6a2571d4 merge more viz tools 2024-12-20 00:03:19 +03:00
Aleksandr Lesnenko
2dc9749a64 group into other 2024-12-20 00:03:19 +03:00
Aleksandr Lesnenko
aa4d5180da merge a few viz settings tools into one 2024-12-20 00:03:19 +03:00
Aleksandr Lesnenko
27a1329c00 metabot v3: change series settings tool (#49644)
* change series settings tool

* metabot v3: change column settings tool (#49646)

* change column settings tool

* metabot v3: stacked setting tool (#49649)

* stacked setting tool

* metabot v3: add change goal line tool (#49650)

* add change goal line tool

* y axis range tool (#49651)
2024-12-20 00:03:19 +03:00
Aleksandr Lesnenko
6ed742c176 change axes names tool (#49640) 2024-12-20 00:03:19 +03:00
Aleksandr Lesnenko
888333c654 provide complete visualization settings to llm (#49639) 2024-12-20 00:03:19 +03:00
Alexander Polyankin
b6a9a3d851 Single query tool (#49722)
* MetaBot v3 PoC

* API endpoint

* Fix formatting

* Change tools shape to JSON schema and update API shapes

* initial FE prototype

* FE - update styles, add toggle button for chat, put behind feature flag, PR feedback

* FE - remove previous message history snapshot from being saved on each history entry

* FE - hide behind feature flag

* FE - use history not messages key on agent requests

* Enforce tool naming rules

* Add encoding for reactions

* don't use ::metabot-v3.client/history schema until it exists

put :any in there, we should use the schema when we know what it looks
like

* `defreaction` can be a plain function.

* Remove empty file

* Code cleanup

* Make sure history includes the previous user message

* `:context` should get snake-case encoding for AI Proxy

* Fix encoding of requests => AI Proxy

* Make sure we use :additionalProperties, not :additional_properties

* Fix error with API endpoint and add (mocked) e2e tests for the endpoint

* Some improvements to error handling in client and REPL.

* Add placeholder tool call result messages

* FE - account for schema changes

* Decode api request into internal format

```clojure
api=> (mc/decode [:maybe ::metabot-v3.client.schema/messages]
                 [{:role "user"
                   :content "Invite sloan to metabase"}
                  {:content "Sorry, I cant help you with that. Could you please provide the email address of the user you want to invite to Metabase?"
                   :role "assistant"}]
                 (mtx/transformer
                   {:name :api-request}))
[{:role :user, :content "Invite sloan to metabase"}
 {:content "Sorry, I cant help you with that. Could you please provide the email address of the user you want to invite to Metabase?",
  :role :assistant}]
```

This just ensures that the role gets keywordized.

`:role "user"` -> `:role :user`.

* ensure we decode LLM response

LLM sends back the tool we are using as "invite-user". But our dispatch
stuff needs it to end up as `:metabot.tool/invite-user`. Just a simple
namespace for the keyword.

```shell
❯ echo '{"message":"sloan@metabase.com","context":{},"history":[{"role":"user","content":"Invite sloan to metabase"},{"content":"Sorry, I cant help you with that. Could you please provide the email address of the user you want to invite to Metabase?","role":"assistant"}]}' | http post "localhost:3000/api/ee/metabot-v3/agent" x-api-key:$KEY -pb
{
    "history": [
        {
            "content": "Invite sloan to metabase",
            "role": "user"
        },
        {
            "content": "Sorry, I cant help you with that. Could you please provide the email address of the user you want to invite to Metabase?",
            "role": "assistant"
        },
        {
            "content": "sloan@metabase.com",
            "role": "user"
        },
        {
            "content": "",
            "role": "assistant",
            "tool-calls": [
                {
                    "arguments": {
                        "email": "sloan@metabase.com"
                    },
                    "id": "call_yt9LvJS6bmB2oAekhU94VwqR",
                    "name": "metabot.tool/invite-user"
                }
            ]
        }
    ],
    "reactions": [
        {
            "email": "sloan@metabase.com",
            "type": "metabot.reaction/user-invite-sent"
        }
    ]
}

```

* decode tools

```clojure
api=> (mc/decode ::metabot-v3.client.schema/messages
                 [{:content "",
                   :role "assistant",
                   :tool-calls [{:id "call_Ma3wQEylZIT27t8z4lx8ch2y",
                                 :name "metabot.tool/say-hello",
                                 :arguments {"name" "User", "greeting" "hi"}}]}]
                 (mtx/transformer {:name :api-request}))
[{:content "",
  :role :assistant,
  :tool-calls [{:id "call_Ma3wQEylZIT27t8z4lx8ch2y",
                :name :metabot.tool/say-hello,
                :arguments {"name" "User", "greeting" "hi"}}]}]
```

from the api:

```shell
❯ echo '{"message":"sloan@metabase.com","context":{},"history":[{"role":"user","content":"Invite sloan to metabase"},{"content":"Sorry, I cant help you with that. Could you please provide the email address of the user you want to invite to Metabase?","role":"assistant"}]}' | http post "localhost:3000/api/ee/metabot-v3/agent" x-api-key:$KEY -pb | jq
 '.reactions'
[
  {
    "type": "metabot.reaction/user-invite-sent",
    "email": "sloan@metabase.com"
  }
]
```

and

```
❯ echo '{"message":"hi","context":{},"history":[{"role":"user","content":"hi"},{"content":"","role":"assistant","tool-calls":[{"id":"call_Ma3wQEylZIT27t8z4lx8ch2y","name":"metabot.tool/say-hello","arguments":{"name":"User","greeting":"hi"}}]}]}' | http post "localhost:3000/api/ee/metabot-v3/agent" x-api-key:$KEY -pb | jq '.reactions'
[
  {
    "type": "metabot.reaction/message",
    "message": "Sorry I don't understand that. Could you please provide more context or let me know how I can assist you?",
    "repl/message_color": "green",
    "repl/message_emoji": "🤖"
  }
]
```

* Add tool-call-ids to less recent history items

* Playing around with other tools

* FE - placeholder for demo steps which are ready for their be reactions

* FE - update how metabot can be accessed

* Add `defreactions` for the new reactions

* include comparison operators

* include other tools

* small fixes

* oops - value is a string or number, not an enum

* An actually working schema for filter value

* go to question tool/reaction

* tell the LLM not to change display with filter

* small fixes for malli schemas

* change the question to sales over time

* hardcode the default hosted proxy URL

* correct URL for ai-proxy

* FE - connect fe reaction implementation with real BE reactions

* FE - shorter animation duration

* change Ask Metabot to Metabot request

* hardcode URL in the getter instead of default

oop, `(premium-features/is-hosted?)` isn't available at compile time so
we need to just hardcode this in the getter. This means we can't
actually set the setting now, but that should be fine for now.

* Respond with simple messages after invite/favorite

* Hide any error details for the demo

* simpler error message

* FE - impl loading state

* Improve invite user schema descriptions

* Round trips to the LLM

* Hopefully fix clj-kondo hook test

This is my first time actually touching a hook, so hopefully I didn't
mess it up.

* Fix a few more tests

* Log traffic to `llm-payloads` file

```json
"FE -----------------> BE"
{
  "message" : "invite thomas to metabase",
  "context" : { },
  "history" : [ ]
}

"BE -----------------> LLM"
{
  "messages" : [ {
    "role" : "user",
    "content" : "invite thomas to metabase"
  } ],
  "context" : { },
  "tools" : [ {
    "name" : "apply_visualizations",
    ...
    }]}

"LLM -----------------> BE"
{
  "message" : {
    "content" : "Could you please provide me with Thomas's email address so I can send the invitation to Metabase?",
    "role" : "assistant"
  },
  "metadata" : {
    "usage" : {
      "model" : "gpt-4o-mini-2024-07-18",
      "total" : 1163,
      "prompt" : 1142,
      "completion" : 21,
      "estimated_costs" : 1.8704999999999997E-4
    }
  }
}
...
"BE -----------------> FE"
{
  "reactions" : [ {
    "type" : "metabot.reaction/message",
    "repl/message_color" : "green",
    "repl/message_emoji" : "🤖",
    "message" : "Thomas has been successfully invited to Metabase! If you need anything else, just let me know."
  } ],
  "history" : [ {
    "role" : "user",
    "content" : "invite thomas to metabase"
    ...
```

* Fix the REPL namespace

* FE - multi-line editor, css based animations, error handling, dismissable messages, updated loading state

* max 6 round trips

* Remove unused tools + real invite users

* Oops, fix tool requirements

* fix token feature test

* another small test fix

* oops - eastwood

* delete empty config

* FE - implments context solution so any component or route can set context for metabot, adds handling for change display type tool

* Observability, fix endpoint URL, and fix table viz

* Rework, but turn off, the invite tool

* Don't log tool args/results

* only log paylods in dev

* FE - impl hide/show columns reaction

* check metabot-v3 feature

* add tests for tools on the backend

* FE - make Dan happy

* FE - make Dan happier

* lint

* FE - huge code improvement pass, keep isProcessing state for metabot and await completion of each reaction before processing next, allow reactions to stop further reaction processing in the case of an error

* gate metabot on a different feature for now

* Some documentation

also moved `handle-response` to `handle-envelope`, since that's what
it's doing now.

* add llm-payloads to gitignore (#49505)

* Add back backend support for adding a user

After talking with Sloan, we reworked the reactions here a bit.

The `confirmation` reaction has a `description` and `options` (the
options here means "the options the user can choose between", not
something like "arguments").

The keys of the options are just names for the choices, in this case
"yes" and "no". When the frontend makes the choice, it carries out the
reactions located under that key in the `confirmation` reaction.

* FE - documentation and make context deregistering O(1)

* FE - fix enter key bug

* Include description in column context

* Always send the LLM's messages as reactions

Previously, we would only send the LLM's message as a reaction if there
were no other reactions. I think it makes more sense to send a list
containing:

- all reactions explicitly set by tool call invocations, PLUS

- all messages from the LLM since your last message (formatted as
reaction messages)

* Fix double message on the who's your favorite tool

* Try something with invite user, and fix up setting

* WIP tools testing

* add ? to confirmation description

* fix confirmation flow

* don't send tool call id on tool invocation

* set default ai host to localhost

* exclude metabot-v3 tests from test run

* FE - invite user improvements, implement confirmation + writeback + api call reactions, update users page to use RTKQuery to allow for reactive updates

* oops, fix failing tool tests

* use the real metabot feature

* Add sort-query tool

* Add direction

* Fix formatting

* Fix formatting

* Fix lint

* Add `aggregate_query` tool (#49622)

* aggregate_query tool

* Fix lint

* Add `limit_query` tool (#49629)

* limit_query tool

* Add breakout_query tool (#49631)

* Fix conflicts

* Single tool for queries

* Add grouping

* Add aggregations

* Add aggregations

* Add aggregations

* Add aggregations

* Add filters

* Cleanup

* Cleanup

* Cleanup

* Booleans

* Dates

* Dates

---------

Co-authored-by: Cam Saul <github@camsaul.com>
Co-authored-by: Sloan Sparger <sloansparger@gmail.com>
Co-authored-by: dan sutton <dan@dpsutton.com>
Co-authored-by: John Swanson <john.swanson@metabase.com>
Co-authored-by: Maz Ameli <maz@metabase.com>
Co-authored-by: Thomas Schmidt <thomas.heinz.schmidt@gmail.com>
Co-authored-by: Aleksandr Lesnenko <alxnddr@users.noreply.github.com>
2024-12-20 00:03:19 +03:00
Alexander Polyankin
af08a7488c Add sort_query tool (#49619)
* MetaBot v3 PoC

* API endpoint

* Fix formatting

* Change tools shape to JSON schema and update API shapes

* initial FE prototype

* FE - update styles, add toggle button for chat, put behind feature flag, PR feedback

* FE - remove previous message history snapshot from being saved on each history entry

* FE - hide behind feature flag

* FE - use history not messages key on agent requests

* Enforce tool naming rules

* Add encoding for reactions

* don't use ::metabot-v3.client/history schema until it exists

put :any in there, we should use the schema when we know what it looks
like

* `defreaction` can be a plain function.

* Remove empty file

* Code cleanup

* Make sure history includes the previous user message

* `:context` should get snake-case encoding for AI Proxy

* Fix encoding of requests => AI Proxy

* Make sure we use :additionalProperties, not :additional_properties

* Fix error with API endpoint and add (mocked) e2e tests for the endpoint

* Some improvements to error handling in client and REPL.

* Add placeholder tool call result messages

* FE - account for schema changes

* Decode api request into internal format

```clojure
api=> (mc/decode [:maybe ::metabot-v3.client.schema/messages]
                 [{:role "user"
                   :content "Invite sloan to metabase"}
                  {:content "Sorry, I cant help you with that. Could you please provide the email address of the user you want to invite to Metabase?"
                   :role "assistant"}]
                 (mtx/transformer
                   {:name :api-request}))
[{:role :user, :content "Invite sloan to metabase"}
 {:content "Sorry, I cant help you with that. Could you please provide the email address of the user you want to invite to Metabase?",
  :role :assistant}]
```

This just ensures that the role gets keywordized.

`:role "user"` -> `:role :user`.

* ensure we decode LLM response

LLM sends back the tool we are using as "invite-user". But our dispatch
stuff needs it to end up as `:metabot.tool/invite-user`. Just a simple
namespace for the keyword.

```shell
❯ echo '{"message":"sloan@metabase.com","context":{},"history":[{"role":"user","content":"Invite sloan to metabase"},{"content":"Sorry, I cant help you with that. Could you please provide the email address of the user you want to invite to Metabase?","role":"assistant"}]}' | http post "localhost:3000/api/ee/metabot-v3/agent" x-api-key:$KEY -pb
{
    "history": [
        {
            "content": "Invite sloan to metabase",
            "role": "user"
        },
        {
            "content": "Sorry, I cant help you with that. Could you please provide the email address of the user you want to invite to Metabase?",
            "role": "assistant"
        },
        {
            "content": "sloan@metabase.com",
            "role": "user"
        },
        {
            "content": "",
            "role": "assistant",
            "tool-calls": [
                {
                    "arguments": {
                        "email": "sloan@metabase.com"
                    },
                    "id": "call_yt9LvJS6bmB2oAekhU94VwqR",
                    "name": "metabot.tool/invite-user"
                }
            ]
        }
    ],
    "reactions": [
        {
            "email": "sloan@metabase.com",
            "type": "metabot.reaction/user-invite-sent"
        }
    ]
}

```

* decode tools

```clojure
api=> (mc/decode ::metabot-v3.client.schema/messages
                 [{:content "",
                   :role "assistant",
                   :tool-calls [{:id "call_Ma3wQEylZIT27t8z4lx8ch2y",
                                 :name "metabot.tool/say-hello",
                                 :arguments {"name" "User", "greeting" "hi"}}]}]
                 (mtx/transformer {:name :api-request}))
[{:content "",
  :role :assistant,
  :tool-calls [{:id "call_Ma3wQEylZIT27t8z4lx8ch2y",
                :name :metabot.tool/say-hello,
                :arguments {"name" "User", "greeting" "hi"}}]}]
```

from the api:

```shell
❯ echo '{"message":"sloan@metabase.com","context":{},"history":[{"role":"user","content":"Invite sloan to metabase"},{"content":"Sorry, I cant help you with that. Could you please provide the email address of the user you want to invite to Metabase?","role":"assistant"}]}' | http post "localhost:3000/api/ee/metabot-v3/agent" x-api-key:$KEY -pb | jq
 '.reactions'
[
  {
    "type": "metabot.reaction/user-invite-sent",
    "email": "sloan@metabase.com"
  }
]
```

and

```
❯ echo '{"message":"hi","context":{},"history":[{"role":"user","content":"hi"},{"content":"","role":"assistant","tool-calls":[{"id":"call_Ma3wQEylZIT27t8z4lx8ch2y","name":"metabot.tool/say-hello","arguments":{"name":"User","greeting":"hi"}}]}]}' | http post "localhost:3000/api/ee/metabot-v3/agent" x-api-key:$KEY -pb | jq '.reactions'
[
  {
    "type": "metabot.reaction/message",
    "message": "Sorry I don't understand that. Could you please provide more context or let me know how I can assist you?",
    "repl/message_color": "green",
    "repl/message_emoji": "🤖"
  }
]
```

* Add tool-call-ids to less recent history items

* Playing around with other tools

* FE - placeholder for demo steps which are ready for their be reactions

* FE - update how metabot can be accessed

* Add `defreactions` for the new reactions

* include comparison operators

* include other tools

* small fixes

* oops - value is a string or number, not an enum

* An actually working schema for filter value

* go to question tool/reaction

* tell the LLM not to change display with filter

* small fixes for malli schemas

* change the question to sales over time

* hardcode the default hosted proxy URL

* correct URL for ai-proxy

* FE - connect fe reaction implementation with real BE reactions

* FE - shorter animation duration

* change Ask Metabot to Metabot request

* hardcode URL in the getter instead of default

oop, `(premium-features/is-hosted?)` isn't available at compile time so
we need to just hardcode this in the getter. This means we can't
actually set the setting now, but that should be fine for now.

* Respond with simple messages after invite/favorite

* Hide any error details for the demo

* simpler error message

* FE - impl loading state

* Improve invite user schema descriptions

* Round trips to the LLM

* Hopefully fix clj-kondo hook test

This is my first time actually touching a hook, so hopefully I didn't
mess it up.

* Fix a few more tests

* Log traffic to `llm-payloads` file

```json
"FE -----------------> BE"
{
  "message" : "invite thomas to metabase",
  "context" : { },
  "history" : [ ]
}

"BE -----------------> LLM"
{
  "messages" : [ {
    "role" : "user",
    "content" : "invite thomas to metabase"
  } ],
  "context" : { },
  "tools" : [ {
    "name" : "apply_visualizations",
    ...
    }]}

"LLM -----------------> BE"
{
  "message" : {
    "content" : "Could you please provide me with Thomas's email address so I can send the invitation to Metabase?",
    "role" : "assistant"
  },
  "metadata" : {
    "usage" : {
      "model" : "gpt-4o-mini-2024-07-18",
      "total" : 1163,
      "prompt" : 1142,
      "completion" : 21,
      "estimated_costs" : 1.8704999999999997E-4
    }
  }
}
...
"BE -----------------> FE"
{
  "reactions" : [ {
    "type" : "metabot.reaction/message",
    "repl/message_color" : "green",
    "repl/message_emoji" : "🤖",
    "message" : "Thomas has been successfully invited to Metabase! If you need anything else, just let me know."
  } ],
  "history" : [ {
    "role" : "user",
    "content" : "invite thomas to metabase"
    ...
```

* Fix the REPL namespace

* FE - multi-line editor, css based animations, error handling, dismissable messages, updated loading state

* max 6 round trips

* Remove unused tools + real invite users

* Oops, fix tool requirements

* fix token feature test

* another small test fix

* oops - eastwood

* delete empty config

* FE - implments context solution so any component or route can set context for metabot, adds handling for change display type tool

* Observability, fix endpoint URL, and fix table viz

* Rework, but turn off, the invite tool

* Don't log tool args/results

* only log paylods in dev

* FE - impl hide/show columns reaction

* check metabot-v3 feature

* add tests for tools on the backend

* FE - make Dan happy

* FE - make Dan happier

* lint

* FE - huge code improvement pass, keep isProcessing state for metabot and await completion of each reaction before processing next, allow reactions to stop further reaction processing in the case of an error

* gate metabot on a different feature for now

* Some documentation

also moved `handle-response` to `handle-envelope`, since that's what
it's doing now.

* add llm-payloads to gitignore (#49505)

* Add back backend support for adding a user

After talking with Sloan, we reworked the reactions here a bit.

The `confirmation` reaction has a `description` and `options` (the
options here means "the options the user can choose between", not
something like "arguments").

The keys of the options are just names for the choices, in this case
"yes" and "no". When the frontend makes the choice, it carries out the
reactions located under that key in the `confirmation` reaction.

* FE - documentation and make context deregistering O(1)

* FE - fix enter key bug

* Include description in column context

* Always send the LLM's messages as reactions

Previously, we would only send the LLM's message as a reaction if there
were no other reactions. I think it makes more sense to send a list
containing:

- all reactions explicitly set by tool call invocations, PLUS

- all messages from the LLM since your last message (formatted as
reaction messages)

* Fix double message on the who's your favorite tool

* Try something with invite user, and fix up setting

* WIP tools testing

* add ? to confirmation description

* fix confirmation flow

* don't send tool call id on tool invocation

* set default ai host to localhost

* exclude metabot-v3 tests from test run

* FE - invite user improvements, implement confirmation + writeback + api call reactions, update users page to use RTKQuery to allow for reactive updates

* oops, fix failing tool tests

* use the real metabot feature

* Add sort-query tool

* Add direction

* Fix formatting

* Fix formatting

* Fix lint

* Add `aggregate_query` tool (#49622)

* aggregate_query tool

* Fix lint

* Add `limit_query` tool (#49629)

* limit_query tool

* Add breakout_query tool (#49631)

* Fix conflicts

---------

Co-authored-by: Cam Saul <github@camsaul.com>
Co-authored-by: Sloan Sparger <sloansparger@gmail.com>
Co-authored-by: dan sutton <dan@dpsutton.com>
Co-authored-by: John Swanson <john.swanson@metabase.com>
Co-authored-by: Maz Ameli <maz@metabase.com>
Co-authored-by: Thomas Schmidt <thomas.heinz.schmidt@gmail.com>
Co-authored-by: Aleksandr Lesnenko <alxnddr@users.noreply.github.com>
2024-12-20 00:03:19 +03:00
John Swanson
8933965e77 Instance info
For now, put `user-id` into the instance info. I'll move this over once
we've added that field to the schema on the `ai-service` side.
2024-12-20 00:03:19 +03:00
Sloan Sparger
d4c0b24558 FE - change error messages 2024-12-20 00:03:19 +03:00
Sloan Sparger
7b34e3c04f FE - copy changes 2024-12-20 00:03:19 +03:00
Sloan Sparger
b498a5ad59 FE - removes the feature from being hardcoded as enabled 2024-12-20 00:03:19 +03:00
John Swanson
4530bb4472 use the real metabot feature 2024-12-20 00:03:19 +03:00
John Swanson
f7bc32c37f oops, fix failing tool tests 2024-12-20 00:03:19 +03:00
Sloan Sparger
b11ace9933 FE - invite user improvements, implement confirmation + writeback + api call reactions, update users page to use RTKQuery to allow for reactive updates 2024-12-20 00:03:19 +03:00
John Swanson
4ac8936dda exclude metabot-v3 tests from test run 2024-12-20 00:03:18 +03:00
John Swanson
da80d576fc set default ai host to localhost 2024-12-20 00:03:18 +03:00
John Swanson
75af27d1ef don't send tool call id on tool invocation 2024-12-20 00:03:18 +03:00
John Swanson
b71f81de3a fix confirmation flow 2024-12-20 00:03:18 +03:00
John Swanson
8ae959af40 add ? to confirmation description 2024-12-20 00:03:18 +03:00
John Swanson
54bea92e33 WIP tools testing 2024-12-20 00:03:18 +03:00
John Swanson
c1cec51039 Try something with invite user, and fix up setting 2024-12-20 00:03:18 +03:00
John Swanson
619faab76a Fix double message on the who's your favorite tool 2024-12-20 00:03:18 +03:00
John Swanson
fa679be3f5 Always send the LLM's messages as reactions
Previously, we would only send the LLM's message as a reaction if there
were no other reactions. I think it makes more sense to send a list
containing:

- all reactions explicitly set by tool call invocations, PLUS

- all messages from the LLM since your last message (formatted as
reaction messages)
2024-12-20 00:03:18 +03:00
Thomas Schmidt
3e151e39ce Include description in column context 2024-12-20 00:03:18 +03:00
Sloan Sparger
2086a5bcee FE - fix enter key bug 2024-12-20 00:03:18 +03:00
Sloan Sparger
94658fdbab FE - documentation and make context deregistering O(1) 2024-12-20 00:03:18 +03:00
John Swanson
c4d52fb574 Add back backend support for adding a user
After talking with Sloan, we reworked the reactions here a bit.

The `confirmation` reaction has a `description` and `options` (the
options here means "the options the user can choose between", not
something like "arguments").

The keys of the options are just names for the choices, in this case
"yes" and "no". When the frontend makes the choice, it carries out the
reactions located under that key in the `confirmation` reaction.
2024-12-20 00:03:18 +03:00
Aleksandr Lesnenko
1444a637e3 add llm-payloads to gitignore (#49505) 2024-12-20 00:03:18 +03:00
John Swanson
fdbbe3e246 Some documentation
also moved `handle-response` to `handle-envelope`, since that's what
it's doing now.
2024-12-20 00:03:18 +03:00
John Swanson
4d722317bf gate metabot on a different feature for now 2024-12-20 00:03:18 +03:00
Sloan Sparger
07fb93c1dd FE - huge code improvement pass, keep isProcessing state for metabot and await completion of each reaction before processing next, allow reactions to stop further reaction processing in the case of an error 2024-12-20 00:03:18 +03:00
John Swanson
3082c28ce6 lint 2024-12-20 00:03:18 +03:00
Sloan Sparger
c43ffdf1f6 FE - make Dan happier 2024-12-20 00:03:18 +03:00
Sloan Sparger
30e1b63230 FE - make Dan happy 2024-12-20 00:03:18 +03:00
John Swanson
35f2719fc7 add tests for tools on the backend 2024-12-20 00:03:18 +03:00
John Swanson
5d2b8b4192 check metabot-v3 feature 2024-12-20 00:03:18 +03:00
Sloan Sparger
dd305e34c2 FE - impl hide/show columns reaction 2024-12-20 00:03:18 +03:00
dan sutton
7c47d4908f only log paylods in dev 2024-12-20 00:03:18 +03:00
John Swanson
718602271b Don't log tool args/results 2024-12-20 00:03:18 +03:00
John Swanson
cbbc868d07 Rework, but turn off, the invite tool 2024-12-20 00:03:18 +03:00
John Swanson
ca5aadc113 Observability, fix endpoint URL, and fix table viz 2024-12-20 00:03:18 +03:00
Sloan Sparger
313f03307a FE - implments context solution so any component or route can set context for metabot, adds handling for change display type tool 2024-12-20 00:03:18 +03:00
John Swanson
e92eba2a34 delete empty config 2024-12-20 00:03:18 +03:00
John Swanson
eefe925be2 oops - eastwood 2024-12-20 00:03:18 +03:00
John Swanson
6ed9935bf0 another small test fix 2024-12-20 00:03:18 +03:00
John Swanson
7032631e79 fix token feature test 2024-12-20 00:03:18 +03:00
John Swanson
b0e5c8f306 Oops, fix tool requirements 2024-12-20 00:03:18 +03:00
John Swanson
06b61e9d11 Remove unused tools + real invite users 2024-12-20 00:03:18 +03:00
John Swanson
74c7533d9f max 6 round trips 2024-12-20 00:03:18 +03:00
Sloan Sparger
eca6d725db FE - multi-line editor, css based animations, error handling, dismissable messages, updated loading state 2024-12-20 00:03:18 +03:00
John Swanson
1ee39a695f Fix the REPL namespace 2024-12-20 00:03:18 +03:00
dan sutton
7d97203f80 Log traffic to llm-payloads file
```json
"FE -----------------> BE"
{
  "message" : "invite thomas to metabase",
  "context" : { },
  "history" : [ ]
}

"BE -----------------> LLM"
{
  "messages" : [ {
    "role" : "user",
    "content" : "invite thomas to metabase"
  } ],
  "context" : { },
  "tools" : [ {
    "name" : "apply_visualizations",
    ...
    }]}

"LLM -----------------> BE"
{
  "message" : {
    "content" : "Could you please provide me with Thomas's email address so I can send the invitation to Metabase?",
    "role" : "assistant"
  },
  "metadata" : {
    "usage" : {
      "model" : "gpt-4o-mini-2024-07-18",
      "total" : 1163,
      "prompt" : 1142,
      "completion" : 21,
      "estimated_costs" : 1.8704999999999997E-4
    }
  }
}
...
"BE -----------------> FE"
{
  "reactions" : [ {
    "type" : "metabot.reaction/message",
    "repl/message_color" : "green",
    "repl/message_emoji" : "🤖",
    "message" : "Thomas has been successfully invited to Metabase! If you need anything else, just let me know."
  } ],
  "history" : [ {
    "role" : "user",
    "content" : "invite thomas to metabase"
    ...
```
2024-12-20 00:03:18 +03:00
John Swanson
53df1cc48f Fix a few more tests 2024-12-20 00:03:18 +03:00
John Swanson
445689bd61 Hopefully fix clj-kondo hook test
This is my first time actually touching a hook, so hopefully I didn't
mess it up.
2024-12-20 00:03:18 +03:00
John Swanson
1afd8db757 Round trips to the LLM 2024-12-20 00:03:18 +03:00
Thomas Schmidt
f36b61d095 Improve invite user schema descriptions 2024-12-20 00:03:18 +03:00
Sloan Sparger
1320442e81 FE - impl loading state 2024-12-20 00:03:18 +03:00
John Swanson
9a5c9f6b43 simpler error message 2024-12-20 00:03:18 +03:00
John Swanson
a77f8d3186 Hide any error details for the demo 2024-12-20 00:03:18 +03:00
John Swanson
d18a41c615 Respond with simple messages after invite/favorite 2024-12-20 00:03:18 +03:00
John Swanson
23e07e333b hardcode URL in the getter instead of default
oop, `(premium-features/is-hosted?)` isn't available at compile time so
we need to just hardcode this in the getter. This means we can't
actually set the setting now, but that should be fine for now.
2024-12-20 00:03:18 +03:00
Maz Ameli
c032343abb change Ask Metabot to Metabot request 2024-12-20 00:03:18 +03:00
Sloan Sparger
85454134b8 FE - shorter animation duration 2024-12-20 00:03:18 +03:00
Sloan Sparger
75a61b3ac1 FE - connect fe reaction implementation with real BE reactions 2024-12-20 00:03:18 +03:00
John Swanson
3855c55a9a correct URL for ai-proxy 2024-12-20 00:03:18 +03:00
John Swanson
7023192f86 hardcode the default hosted proxy URL 2024-12-20 00:03:18 +03:00
John Swanson
e52e7b9ca4 change the question to sales over time 2024-12-20 00:03:18 +03:00
John Swanson
1917280df3 small fixes for malli schemas 2024-12-20 00:03:18 +03:00
John Swanson
6a5ac5a8c5 tell the LLM not to change display with filter 2024-12-20 00:03:18 +03:00
John Swanson
54d966fb98 go to question tool/reaction 2024-12-20 00:03:18 +03:00
John Swanson
89e065a77a An actually working schema for filter value 2024-12-20 00:03:18 +03:00
John Swanson
f46fb69020 oops - value is a string or number, not an enum 2024-12-20 00:03:18 +03:00
John Swanson
e5d0c154e6 small fixes 2024-12-20 00:03:18 +03:00
dan sutton
c937c135db include other tools 2024-12-20 00:03:18 +03:00
dan sutton
0052fdfdec include comparison operators 2024-12-20 00:03:18 +03:00
John Swanson
3847d4067c Add defreactions for the new reactions 2024-12-20 00:03:18 +03:00
Sloan Sparger
ba6d81f561 FE - update how metabot can be accessed 2024-12-20 00:03:18 +03:00
Sloan Sparger
50f3c789b8 FE - placeholder for demo steps which are ready for their be reactions 2024-12-20 00:03:18 +03:00
John Swanson
9c71ce697a Playing around with other tools 2024-12-20 00:03:18 +03:00
John Swanson
ba2c9e9eca Add tool-call-ids to less recent history items 2024-12-20 00:03:18 +03:00
dan sutton
4a61e5a42c decode tools
```clojure
api=> (mc/decode ::metabot-v3.client.schema/messages
                 [{:content "",
                   :role "assistant",
                   :tool-calls [{:id "call_Ma3wQEylZIT27t8z4lx8ch2y",
                                 :name "metabot.tool/say-hello",
                                 :arguments {"name" "User", "greeting" "hi"}}]}]
                 (mtx/transformer {:name :api-request}))
[{:content "",
  :role :assistant,
  :tool-calls [{:id "call_Ma3wQEylZIT27t8z4lx8ch2y",
                :name :metabot.tool/say-hello,
                :arguments {"name" "User", "greeting" "hi"}}]}]
```

from the api:

```shell
❯ echo '{"message":"sloan@metabase.com","context":{},"history":[{"role":"user","content":"Invite sloan to metabase"},{"content":"Sorry, I cant help you with that. Could you please provide the email address of the user you want to invite to Metabase?","role":"assistant"}]}' | http post "localhost:3000/api/ee/metabot-v3/agent" x-api-key:$KEY -pb | jq
 '.reactions'
[
  {
    "type": "metabot.reaction/user-invite-sent",
    "email": "sloan@metabase.com"
  }
]
```

and

```
❯ echo '{"message":"hi","context":{},"history":[{"role":"user","content":"hi"},{"content":"","role":"assistant","tool-calls":[{"id":"call_Ma3wQEylZIT27t8z4lx8ch2y","name":"metabot.tool/say-hello","arguments":{"name":"User","greeting":"hi"}}]}]}' | http post "localhost:3000/api/ee/metabot-v3/agent" x-api-key:$KEY -pb | jq '.reactions'
[
  {
    "type": "metabot.reaction/message",
    "message": "Sorry I don't understand that. Could you please provide more context or let me know how I can assist you?",
    "repl/message_color": "green",
    "repl/message_emoji": "🤖"
  }
]
```
2024-12-20 00:03:18 +03:00
dan sutton
7007095b35 ensure we decode LLM response
LLM sends back the tool we are using as "invite-user". But our dispatch
stuff needs it to end up as `:metabot.tool/invite-user`. Just a simple
namespace for the keyword.

```shell
❯ echo '{"message":"sloan@metabase.com","context":{},"history":[{"role":"user","content":"Invite sloan to metabase"},{"content":"Sorry, I cant help you with that. Could you please provide the email address of the user you want to invite to Metabase?","role":"assistant"}]}' | http post "localhost:3000/api/ee/metabot-v3/agent" x-api-key:$KEY -pb
{
    "history": [
        {
            "content": "Invite sloan to metabase",
            "role": "user"
        },
        {
            "content": "Sorry, I cant help you with that. Could you please provide the email address of the user you want to invite to Metabase?",
            "role": "assistant"
        },
        {
            "content": "sloan@metabase.com",
            "role": "user"
        },
        {
            "content": "",
            "role": "assistant",
            "tool-calls": [
                {
                    "arguments": {
                        "email": "sloan@metabase.com"
                    },
                    "id": "call_yt9LvJS6bmB2oAekhU94VwqR",
                    "name": "metabot.tool/invite-user"
                }
            ]
        }
    ],
    "reactions": [
        {
            "email": "sloan@metabase.com",
            "type": "metabot.reaction/user-invite-sent"
        }
    ]
}

```
2024-12-20 00:03:18 +03:00
dan sutton
d32f8c5940 Decode api request into internal format
```clojure
api=> (mc/decode [:maybe ::metabot-v3.client.schema/messages]
                 [{:role "user"
                   :content "Invite sloan to metabase"}
                  {:content "Sorry, I cant help you with that. Could you please provide the email address of the user you want to invite to Metabase?"
                   :role "assistant"}]
                 (mtx/transformer
                   {:name :api-request}))
[{:role :user, :content "Invite sloan to metabase"}
 {:content "Sorry, I cant help you with that. Could you please provide the email address of the user you want to invite to Metabase?",
  :role :assistant}]
```

This just ensures that the role gets keywordized.

`:role "user"` -> `:role :user`.
2024-12-20 00:03:18 +03:00
Sloan Sparger
a1174e82b1 FE - account for schema changes 2024-12-20 00:03:18 +03:00
Cam Saul
6b500e15f9 Add placeholder tool call result messages 2024-12-20 00:03:18 +03:00
Cam Saul
b02e3c9aa1 Some improvements to error handling in client and REPL. 2024-12-20 00:03:18 +03:00
Cam Saul
6abab13f62 Fix error with API endpoint and add (mocked) e2e tests for the endpoint 2024-12-20 00:03:18 +03:00
Cam Saul
f9919999ff Make sure we use :additionalProperties, not :additional_properties 2024-12-20 00:03:18 +03:00
Cam Saul
22ae538b0c Fix encoding of requests => AI Proxy 2024-12-20 00:03:18 +03:00
Cam Saul
ae093274ff :context should get snake-case encoding for AI Proxy 2024-12-20 00:03:18 +03:00
Cam Saul
ac3a171308 Make sure history includes the previous user message 2024-12-20 00:03:18 +03:00
Cam Saul
de00816b7b Code cleanup 2024-12-20 00:03:18 +03:00
Cam Saul
2c647c0637 Remove empty file 2024-12-20 00:03:18 +03:00
Cam Saul
c4866381a3 defreaction can be a plain function. 2024-12-20 00:03:17 +03:00
dan sutton
749a5e6a56 don't use ::metabot-v3.client/history schema until it exists
put :any in there, we should use the schema when we know what it looks
like
2024-12-20 00:03:17 +03:00
Cam Saul
556acb9b10 Add encoding for reactions 2024-12-20 00:03:17 +03:00
Cam Saul
d60072bcc9 Enforce tool naming rules 2024-12-20 00:03:17 +03:00
Sloan Sparger
d478a79856 FE - use history not messages key on agent requests 2024-12-20 00:03:17 +03:00
Sloan Sparger
6067de5ba7 FE - hide behind feature flag 2024-12-20 00:03:17 +03:00
Sloan Sparger
b98ed219e8 FE - remove previous message history snapshot from being saved on each history entry 2024-12-20 00:03:17 +03:00
Sloan Sparger
d6f4458fba FE - update styles, add toggle button for chat, put behind feature flag, PR feedback 2024-12-20 00:03:17 +03:00
Sloan Sparger
4b5457848f initial FE prototype 2024-12-20 00:03:17 +03:00
Cam Saul
acd4ed01c2 Change tools shape to JSON schema and update API shapes 2024-12-20 00:03:17 +03:00
Cam Saul
06212efee6 Fix formatting 2024-12-20 00:03:17 +03:00
Cam Saul
2c4c399e29 API endpoint 2024-12-20 00:03:17 +03:00
Cam Saul
71ac4f02f1 MetaBot v3 PoC 2024-12-20 00:03:17 +03:00
13594 changed files with 1615731 additions and 1850367 deletions

View File

@@ -1 +0,0 @@
Fix flakey test $ARGUMENTS

View File

@@ -1 +0,0 @@
Fix issue $ARGUMENTS

View File

@@ -1 +0,0 @@
Fetch unresolved comments for this branch's PR, then fix them

View File

@@ -1 +0,0 @@
Repro issue $ARGUMENTS in a failing test

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

View File

@@ -13,10 +13,7 @@
toucan2.core/insert-returning-pk!
toucan2.core/insert-returning-pks!
clojure.core.async/<!!
clojure.core.async/<!
clojure.core.async/>!!
clojure.core.async/>!
clojure.core.async/alts!
clojure.core.async/alts!!
clojure.core.async/close!
clojure.core.async/poll!
@@ -52,43 +49,36 @@
metabase.actions.models/insert!
metabase.analytics.snowplow-test/fake-track-event-impl!
metabase.analytics.snowplow/track-event-impl!
metabase.app-db.core/finish-db-setup!
metabase.app-db.core/migrate!
metabase.app-db.core/setup-db!
metabase.app-db.schema-migrations-test.impl/run-migrations-in-range!
metabase.app-db.setup/migrate!
metabase.app-db.setup/setup-db!
metabase.channel.email-test/reset-inbox!
metabase.channel.email/send-email!
metabase.cmd.dump-to-h2/dump-to-h2!
metabase.cmd.load-from-h2/load-from-h2!
metabase.content-verification.models.moderation-review/create-review!
metabase.core.core/ensure-audit-db-installed!
metabase.db.schema-migrations-test.impl/run-migrations-in-range!
metabase.db.setup/migrate!
metabase.db.setup/setup-db!
metabase.db/migrate!
metabase.db/setup-db!
metabase.driver.mongo-test/create-database-from-row-maps!
metabase.driver.postgres-test/create-enums-db!
metabase.driver.postgres-test/drop-if-exists-and-create-db!
metabase.driver.sql-jdbc.execute/execute-statement!
metabase.indexed-entities.models.model-index/add-values!
metabase.driver.sql-jdbc.connection.ssh-tunnel-test/start-mock-servers!
metabase.driver.sql-jdbc.connection.ssh-tunnel-test/stop-mock-servers!
metabase.indexed-entities.task.index-values/job-init!
metabase.model-persistence.models.persisted-info/ready-database!
metabase.model-persistence.task.persist-refresh/job-init!
metabase.model-persistence.task.persist-refresh/refresh-tables!
metabase.model-persistence.task.persist-refresh/schedule-persistence-for-database!
metabase.models.collection.graph-test/clear-graph-revisions!
metabase.models.collection.graph-test/do-with-n-temp-users-with-personal-collections!
metabase.models.field-values/create-or-update-full-field-values!
metabase.models.moderation-review/create-review!
metabase.models.on-demand-test/add-dashcard-with-parameter-mapping!
metabase.permissions.core/grant-application-permissions!
metabase.permissions.core/grant-collection-read-permissions!
metabase.permissions.core/grant-collection-readwrite-permissions!
metabase.permissions.core/grant-full-data-permissions!
metabase.permissions.core/grant-native-readwrite-permissions!
metabase.permissions.core/grant-permissions!
metabase.permissions.core/revoke-application-permissions!
metabase.permissions.core/revoke-data-perms!
metabase.permissions.core/update-data-perms-graph!
metabase.permissions.core/update-group-permissions!
metabase.permissions.models.collection.graph-test/clear-graph-revisions!
metabase.permissions.models.collection.graph-test/do-with-n-temp-users-with-personal-collections!
metabase.models.setting-test/test-user-local-allowed-setting!
metabase.models.setting-test/test-user-local-only-setting!
metabase.models.setting.cache/restore-cache!
metabase.models.setting/set!
metabase.models.setting/validate-settings-formatting!
metabase.permissions.models.permissions/grant-application-permissions!
metabase.permissions.models.permissions/grant-collection-read-permissions!
metabase.permissions.models.permissions/grant-collection-readwrite-permissions!
@@ -100,18 +90,13 @@
metabase.permissions.models.permissions/update-data-perms-graph!
metabase.permissions.models.permissions/update-group-permissions!
metabase.permissions.test-util/with-restored-perms!
metabase.public-sharing-rest.api-test/add-card-to-dashboard!
metabase.public-sharing.api-test/add-card-to-dashboard!
metabase.pulse.send/send-notifications!
metabase.pulse.send/send-pulse!
metabase.query-processor.streaming.interface/begin!
metabase.query-processor.streaming.interface/finish!
metabase.query-processor.streaming.interface/write-row!
metabase.sample-data.impl/try-to-extract-sample-database!
metabase.settings.models.setting-test/test-user-local-allowed-setting!
metabase.settings.models.setting-test/test-user-local-only-setting!
metabase.settings.models.setting.cache/restore-cache!
metabase.settings.models.setting/set!
metabase.settings.models.setting/validate-settings-formatting!
metabase.sample-data/try-to-extract-sample-database!
metabase.setup.core/create-token!
metabase.sso.ldap-test-util/start-ldap-server!
metabase.sync.core/sync-database!
@@ -119,8 +104,7 @@
metabase.sync.sync-metadata/sync-db-metadata!
metabase.sync.util-test/sync-database!
metabase.sync.util/store-sync-summary!
metabase.task.core/delete-task!
metabase.task.impl/delete-task!
metabase.task/delete-task!
metabase.test.data.bigquery-cloud-sdk/execute!
metabase.test.data.impl.get-or-create/set-test-db-permissions!
metabase.test.data.impl/copy-db-tables-and-fields!
@@ -142,6 +126,7 @@
metabase.test/initialize-if-needed!
metabase.test/test-helpers-set-global-values!
metabase.test/with-temp-env-var-value!
metabase.upload.impl-test/set-local-infile!
metabase.upload-test/set-local-infile!
metabase.util.files/create-dir-if-not-exists!
metabase.warehouse-schema.models.field-values/create-or-update-full-field-values!}}}}
metabase.util.ssh-test/start-mock-servers!
metabase.util.ssh-test/stop-mock-servers!}}}}

View File

@@ -0,0 +1,13 @@
(ns macros.metabase.api.card-test)
(defmacro with-ordered-items
[collection model-and-name-syms & body]
`(let ~(into []
(comp (partition-all 2)
(mapcat (fn [[model binding]]
[binding model])))
model-and-name-syms)
~collection
~@(->> (partition-all 2 model-and-name-syms)
(map second))
~@body))

View File

@@ -0,0 +1,24 @@
(ns macros.metabase.api.collection-test)
(defmacro with-collection-hierarchy! [bindings & body]
`(let [~'a nil
~'b nil
~'c nil
~'d nil
~'e nil
~'f nil
~'g nil]
~'a
~'b
~'c
~'d
~'e
~'f
~'g
~bindings
~@body))
(defmacro with-some-children-of-collection! [collection-or-id-or-nil & body]
`(let [~'&ids ~collection-or-id-or-nil]
~'&ids
~@body))

View File

@@ -0,0 +1,13 @@
(ns macros.metabase.api.embed-test)
(defmacro with-embedding-enabled-and-temp-card-referencing!
[table-kw field-kw [card-binding] & body]
`(let [~card-binding [~table-kw ~field-kw]]
~@body))
(defmacro with-embedding-enabled-and-temp-dashcard-referencing!
[table-kw field-kw [dash-binding card-binding dashcard-binding] & body]
`(let [~dash-binding [~table-kw ~field-kw]
~(or card-binding '_) nil
~(or dashcard-binding '_) nil]
~@body))

View File

@@ -1,24 +0,0 @@
(ns macros.metabase.collections-rest.api-test)
(defmacro with-collection-hierarchy! [bindings & body]
`(let [~'a nil
~'b nil
~'c nil
~'d nil
~'e nil
~'f nil
~'g nil]
~'a
~'b
~'c
~'d
~'e
~'f
~'g
~bindings
~@body))
(defmacro with-some-children-of-collection! [collection-or-id-or-nil & body]
`(let [~'&ids ~collection-or-id-or-nil]
~'&ids
~@body))

View File

@@ -1,13 +0,0 @@
(ns macros.metabase.embedding-rest.api.embed-test)
(defmacro with-embedding-enabled-and-temp-card-referencing!
[table-kw field-kw [card-binding] & body]
`(let [~card-binding [~table-kw ~field-kw]]
~@body))
(defmacro with-embedding-enabled-and-temp-dashcard-referencing!
[table-kw field-kw [dash-binding card-binding dashcard-binding] & body]
`(let [~dash-binding [~table-kw ~field-kw]
~(or card-binding '_) nil
~(or dashcard-binding '_) nil]
~@body))

View File

@@ -0,0 +1,4 @@
(ns macros.metabase.models.params.chain-filter-test)
(defmacro chain-filter [& _]
`(metabase.models.params.chain-filter/chain-filter nil nil))

View File

@@ -0,0 +1,12 @@
(ns macros.metabase.models.query-analysis-test)
(defmacro with-test-setup [& body]
`(let [~'card-id 1
~'table-id 2
~'tax-id 3
~'total-id 4]
~'card-id
~'table-id
~'tax-id
~'total-id
~@body))

View File

@@ -0,0 +1,9 @@
(ns macros.metabase.models.user-test)
(defmacro with-groups [bindings & body]
`(let ~(into []
(comp (partition-all 3)
(mapcat (fn [[group-binding properties members]]
[group-binding [properties members]])))
bindings)
~@body))

View File

@@ -1,4 +0,0 @@
(ns macros.metabase.parameters.chain-filter-test)
(defmacro chain-filter [& _]
`(metabase.parameters.chain-filter/chain-filter nil nil))

View File

@@ -0,0 +1,13 @@
(ns macros.metabase.public-sharing.api-test)
(defmacro with-sharing-enabled-and-temp-card-referencing!
[table-kw field-kw [card-binding] & body]
`(let [~card-binding [~table-kw ~field-kw]]
~@body))
(defmacro with-sharing-enabled-and-temp-dashcard-referencing!
[table-kw field-kw [dash-binding card-binding dashcard-binding] & body]
`(let [~dash-binding [~table-kw ~field-kw]
~(or card-binding '_) nil
~(or dashcard-binding '_) nil]
~@body))

View File

@@ -1,13 +0,0 @@
(ns macros.metabase.public-sharing-rest.api-test)
(defmacro with-sharing-enabled-and-temp-card-referencing!
[table-kw field-kw [card-binding] & body]
`(let [~card-binding [~table-kw ~field-kw]]
~@body))
(defmacro with-sharing-enabled-and-temp-dashcard-referencing!
[table-kw field-kw [dash-binding card-binding dashcard-binding] & body]
`(let [~dash-binding [~table-kw ~field-kw]
~(or card-binding '_) nil
~(or dashcard-binding '_) nil]
~@body))

View File

@@ -1,13 +0,0 @@
(ns macros.metabase.queries.api.card-test)
(defmacro with-ordered-items
[collection model-and-name-syms & body]
`(let ~(into []
(comp (partition-all 2)
(mapcat (fn [[model binding]]
[binding model])))
model-and-name-syms)
~collection
~@(->> (partition-all 2 model-and-name-syms)
(map second))
~@body))

View File

@@ -1,12 +0,0 @@
(ns macros.metabase.query-analysis.models.query-analysis-test)
(defmacro with-test-setup [& body]
`(let [~'card-id 1
~'table-id 2
~'tax-id 3
~'total-id 4]
~'card-id
~'table-id
~'tax-id
~'total-id
~@body))

View File

@@ -1,9 +0,0 @@
(ns macros.metabase.server.statistics-handler-test)
(defmacro with-server
[_ _ [connector chan-start-handle chan-finish-handle chan-done-request] & body]
`(let [~connector nil
~chan-start-handle nil
~chan-finish-handle nil
~chan-done-request nil]
~@body))

View File

@@ -1,9 +0,0 @@
(ns macros.metabase.users.models.user-test)
(defmacro with-groups! [bindings & body]
`(let ~(into []
(comp (partition-all 3)
(mapcat (fn [[group-binding properties members]]
[group-binding [properties members]])))
bindings)
~@body))

View File

@@ -0,0 +1,16 @@
(ns macros.metabase.xrays.domain-entities.malli)
(defmacro define-getters-and-setters [schema & defs]
`(do
(comment ~schema) ;; Reference the schema
~@(for [[sym _path] (partition 2 defs)]
`(do
(defn ~(vary-meta sym
assoc :export true)
"docs" [_x#] nil)
(defn ~(vary-meta (symbol (str "with-" (name sym)))
assoc :export true)
"docs" [x# _new#] x#)
(defn ~(vary-meta (symbol (str (name sym) "-js"))
assoc :export true)
"docs" [x#] x#)))))

View File

@@ -1,5 +0,0 @@
(ns macros.metabase-enterprise.database-routing.e2e-test)
(defmacro with-temp-dbs! [bindings & body]
`(let [~@(interleave bindings (repeat nil))]
~@body))

View File

@@ -1,7 +1,6 @@
(ns hooks.clojure.core.ns
(:require
[clj-kondo.hooks-api :as hooks]
[clojure.string :as str]
[hooks.common]
[hooks.common.modules :as modules]))
@@ -60,15 +59,12 @@
(printf "Don't know how to figure out what namespace is being required in %s\n" (pr-str node)))))
args)))
(defn- ns-form-node->ns-symb-node [ns-form-node]
(some (fn [node]
(when (and (hooks/token-node? node)
(not= (hooks/sexpr node) 'ns))
node))
(:children ns-form-node)))
(defn- ns-form-node->ns-symb [ns-form-node]
(some-> (ns-form-node->ns-symb-node ns-form-node)
(some-> (some (fn [node]
(when (and (hooks/token-node? node)
(not= (hooks/sexpr node) 'ns))
node))
(:children ns-form-node))
hooks/sexpr))
(defn- lint-modules [ns-form-node config]
@@ -87,33 +83,9 @@
:message error
:type :metabase/modules))))))))
(defn- lint-namespace-name [ns-form-node]
(when-let [ns-symb-node (ns-form-node->ns-symb-node ns-form-node)]
(when-not (contains? (hooks.common/ignored-linters ns-symb-node) :metabase/namespace-name)
(let [parts (str/split (str (hooks/sexpr ns-symb-node)) #"\.")]
(when (and (<= (count parts) 2)
(#{"metabase" "metabase-enterprise"} (first parts))
;; exclude test namespaces for now
(not (str/ends-with? (last parts) "test")))
(hooks/reg-finding! (assoc (meta ns-symb-node)
:message "Metabase namespaces should have the form metabase[-enterprise].<module>.* [:metabase/namespace-name]"
:type :metabase/namespace-name)))))))
(defn lint-ns [x]
(doto (:node x)
lint-require-shapes
lint-requires-on-new-lines
(lint-modules (modules/config x))
lint-namespace-name)
(lint-modules (modules/config x)))
x)
(comment
(lint-ns
{:node (hooks/parse-string
(pr-str
'(ns hooks.clojure.core.ns
(:require
[clj-kondo.hooks-api :as hooks]
[clojure.string :as str]
[hooks.common]
[hooks.common.modules :as modules]))))}))

View File

@@ -52,13 +52,7 @@
(defn- lint-require* [node current-ns config]
(let [[_require & args] (:children node)
required-namespaces (->> (keep unwrap-require args)
;; `require` runs through `cc/load-libs` which allows
;; #{:as :reload :reload-all :require :use :verbose :refer :as-alias}
;; as args. If you use a REPL tool that applies clj-kondo linting (e.g., Clojure MCP)
;; you may wish to `:reload` or `:reload-all` a namespace. So we shouldn't assume all
;; args are keywords:
(remove keyword?))]
required-namespaces (keep unwrap-require args)]
(lint* node current-ns config required-namespaces)))
(defn- lint-requiring-resolve* [node current-ns config]

View File

@@ -1,67 +1,7 @@
(ns hooks.metabase.api.macros
(:require
[clj-kondo.hooks-api :as api]
[clojure.string :as str]))
(defn- validate-route!
"Make sure the route uses kebab-case rather than snake_case."
[route]
(letfn [(validate-route-string! [route-string]
(when (and (api/string-node? route-string)
(str/includes? (api/sexpr route-string) "_"))
(api/reg-finding! (assoc (meta route-string)
:message "REST API routes and query params should use kebab-case, not snake_case"
:type :metabase/validate-defendpoint-route-uses-kebab-case))))]
(cond
(api/string-node? route)
(validate-route-string! route)
;; otherwise route is a vector like ["/abc/:d/" :d <regex>]
(api/vector-node? route)
(validate-route-string! (first (:children route))))))
(defn- validate-query-params!
"Make sure query params use kebab-case rather than snake_case.
This only works on inline schemas for now. If you define the schema elsewhere then we'd have to implement this
validation in Clojure-land since we can't resolve the schemas in Kondo. It should however still work on complicated
schemas like
[:merge <map-1> <map-2>]"
[query-param-schema-node]
(letfn [(map-schema? [node]
(and (api/vector-node? node)
(when-let [first-child (first (:children node))]
(and (api/keyword-node? first-child)
(= (api/sexpr first-child) :map)))))
(map-schema-keys [{:keys [children], :as _node}]
(let [child-schemas (if (api/map-node? (second children))
(drop 2 children)
(drop 1 children))]
(keep (fn [child-schema]
(when (api/vector-node? child-schema)
(when-let [k (first (:children child-schema))]
(when (api/keyword-node? k)
k))))
child-schemas)))
(validate-map-schema! [form]
;; only warn once per endpoint -- just once on the first bad node
(when-let [bad-node (some (fn [key-node]
(when (str/includes? (str (api/sexpr key-node)) "_")
key-node))
(map-schema-keys form))]
(api/reg-finding! (assoc (meta bad-node)
:message "REST API routes and query params should use kebab-case, not snake_case"
:type :metabase/validate-defendpoint-query-params-use-kebab-case))))
(validate! [node]
(cond
(and (api/vector-node? node)
(map-schema? node))
(validate-map-schema! node)
(api/vector-node? node)
(doseq [child (:children node)]
(validate! child))))]
(validate! query-param-schema-node)))
[hooks.common]))
(defn defendpoint
[arg]
@@ -78,21 +18,16 @@
(cons nil more))
[params & body] more
[bindings schemas] (when (api/vector-node? params)
(loop [bindings [], schemas [], [x y z & more] (:children params)]
(loop [bindings [], schemas [], [x y & more] (:children params)]
(cond
(not x)
[bindings schemas]
(and (api/keyword-node? y)
(= (api/sexpr y) :-))
(recur (conj bindings x) (conj schemas z) more)
(= (api/sexpr x) :-)
(recur bindings (conj schemas y) more)
:else
(recur (conj bindings x) (conj schemas nil) (list* y z more)))))]
(validate-route! route)
(let [[_route-param-schema query-param-schema _body-param-schema] schemas]
(when query-param-schema
(validate-query-params! query-param-schema)))
(recur (conj bindings x) schemas (cons y more)))))]
(-> (api/list-node
(list
(api/token-node 'do)
@@ -119,27 +54,3 @@
body))))
(with-meta (meta node)))))]
(update arg :node update-defendpoint)))
(comment
(defn -defendpoint [form]
(-> {:node (-> form pr-str api/parse-string)}
defendpoint
:node
api/sexpr))
(defn x []
(-defendpoint '(api.macros/defendpoint :get "/:id/syncable_schemas" ; snake_case route should trigger a warning
"Returns a list of all syncable schemas found for the database `id`."
[{:keys [id]} :- [:map
[:id ms/PositiveInt]]
;; snake_case query parameter, should trigger a warning
{:keys [filter_by]} :- [:map
[:filter_by {:optional true} :string]]]
(let [db (get-database id)]
(api/check-403 (or (:is_attached_dwh db)
(and (mi/can-write? db)
(mi/can-read? db))))
(->> db
(driver/syncable-schemas (:engine db))
(vec)
(sort)))))))

View File

@@ -1,14 +0,0 @@
(ns hooks.metabase.app-db.schema-migrations-test.impl
(:require [clj-kondo.hooks-api :as hooks]))
(defn test-migrations
[{{[_ migration-range {[binding] :children} & body] :children} :node}]
(let [node* (hooks/list-node
(list*
(hooks/token-node `let)
(hooks/vector-node
[binding (hooks/list-node (list (hooks/token-node `fn)
(hooks/vector-node [(hooks/token-node '&) (hooks/token-node '_args)])))])
migration-range
body))]
{:node node*}))

View File

@@ -0,0 +1,14 @@
(ns hooks.metabase.db.schema-migrations-test.impl
(:require [clj-kondo.hooks-api :as hooks]))
(defn test-migrations
[{{[_ migration-range {[binding] :children} & body] :children} :node}]
(let [node* (hooks/list-node
(list*
(hooks/token-node `let)
(hooks/vector-node
[binding (hooks/list-node (list (hooks/token-node `fn)
(hooks/vector-node [(hooks/token-node '&) (hooks/token-node '_args)])))])
migration-range
body))]
{:node node*}))

View File

@@ -1,18 +0,0 @@
(ns hooks.metabase.driver.mongo-test
(:require
[clj-kondo.hooks-api :as hooks]
[hooks.common]))
(defn with-describe-table-for-sample
"Forward bindings for results of functions used in mongo's describe-table implementation."
[{{[_ _ & body] :children} :node}]
(let [node* (hooks/list-node
(list*
(hooks/token-node 'let)
(hooks/vector-node
[(hooks/token-node 'dbfields) (hooks/token-node '(delay nil))
(hooks/token-node 'ftree) (hooks/token-node '(delay nil))
(hooks/token-node 'nested-fields) (hooks/token-node '(delay nil))])
body))]
;; Avoid warnings when not all from the dbfields, ftree and nested-fields are used.
{:node (hooks.common/update-ignored-linters node* conj :unused-binding)}))

View File

@@ -33,36 +33,18 @@
(defclause [ag:var var] field-or-expression FieldOrExpressionDef)
=>
(def ag:var [FieldOrExpressionDef])"
[{:keys [node], :as x}]
(let [[_defclause clause-name & arg-specs] (:children node)]
(assoc x :node (-> (hooks/list-node
(list
(hooks/token-node 'do)
(-> (hooks/keyword-node (keyword "metabase.legacy-mbql.schema" (name (hooks/sexpr clause-name))))
(with-meta (meta clause-name))
(hooks/reg-keyword! 'metabase.legacy-mbql.schemas.macros/defclause))
(collect-defclause-body-schemas arg-specs)))
(with-meta (meta node))))))
(defn defclause*
"e.g.
(defclause [ag:var var] field-or-expression FieldOrExpressionDef)
=>
(def ag:var [FieldOrExpressionDef])"
[{:keys [node], :as x}]
(let [[_defclause clause-name schema] (:children node)]
(assoc x :node (-> (hooks/list-node
(list
(hooks/token-node 'do)
(-> (hooks/keyword-node (keyword "metabase.legacy-mbql.schema" (name (hooks/sexpr clause-name))))
(with-meta (meta clause-name))
(hooks/reg-keyword! 'metabase.legacy-mbql.schemas.macros/defclause*))
schema))
(with-meta (meta node))))))
[{:keys [node]}]
(let [[_defclause clause-name-form & arg-specs] (:children node)]
{:node (-> (hooks/list-node
(list
(hooks/token-node 'def)
(unwrap-defclause-clause-name clause-name-form)
(hooks/string-node "Docstring.")
(collect-defclause-body-schemas arg-specs)))
(with-meta (meta node)))}))
(comment
(defn- expand-defclause [form]
(defn- defclause* [form]
(hooks/sexpr
(:node
(defclause
@@ -73,35 +55,15 @@
form)))}))))
(defn- x []
(expand-defclause '(defclause var
field-or-expression FieldOrExpressionDef)))
(defclause* '(defclause [ag:var var]
field-or-expression FieldOrExpressionDef)))
;; =>
(do :metabase.legacy-mbql.schema/var [FieldOrExpressionDef])
(def ag:var "Docstring." [FieldOrExpressionDef])
(defn- y []
(expand-defclause '(defclause var
x (some-list-schema)
y (optional FieldOrExpressionDef)
z (rest (some-other-schema)))))
(defclause* '(defclause var
x (some-list-schema)
y (optional FieldOrExpressionDef)
z (rest (some-other-schema)))))
;; =>
(do :metabase.legacy-mbql.schema/var [(some-list-schema) FieldOrExpressionDef (some-other-schema)]))
(defn one-of [{:keys [node], :as x}]
(let [[_one-of & schema-nodes] (:children node)
node' (-> (hooks/vector-node
(mapv (fn [schema-node]
(if (and (hooks/token-node? schema-node)
(simple-symbol? (hooks/sexpr schema-node)))
(-> (hooks/keyword-node (keyword "mteabase.legacy-mbql.schema" (name (hooks/sexpr schema-node))))
(with-meta (meta schema-node)))
schema-node))
schema-nodes))
(with-meta (meta node)))]
(assoc x :node node')))
(comment
(defn- expand-one-of [form]
(hooks/sexpr (:node (one-of {:node (hooks/parse-string (with-out-str (clojure.pprint/pprint form)))}))))
(defn- z []
(expand-one-of '(one-of absolute-datetime relative-datetime time))))
(def var "Docstring." [(some-list-schema) FieldOrExpressionDef (some-other-schema)]))

View File

@@ -0,0 +1,294 @@
(ns hooks.metabase.models.setting
(:require
[clj-kondo.hooks-api :as hooks]
[hooks.common :as common]))
;;; TODO -- move this into
(def ^:private ignored-implicit-export?
'#{active-users-count
admin-email
analytics-uuid
anon-tracking-enabled
api-key
attachment-table-row-limit
audit-max-retention-days
bcc-enabled?
breakout-bin-width
check-for-updates
cloud-gateway-ips
config-from-file-sync-databases
custom-homepage
custom-homepage-dashboard
database-enable-actions
deprecation-notice-version
dismissed-custom-dashboard-toast
dismissed-onboarding-sidebar-link
email-configured?
email-from-address
email-from-name
email-reply-to
email-smtp-host
email-smtp-password
email-smtp-port
email-smtp-security
email-smtp-username
embedding-app-origin
embedding-secret-key
enable-password-login
enable-public-sharing
enable-query-caching
engines
enum-cardinality-threshold
follow-up-email-sent
google-auth-auto-create-accounts-domain
google-auth-client-id
google-auth-configured
google-auth-enabled
has-sample-database?
has-user-setup
help-link
help-link-custom-destination
instance-creation
is-hosted?
is-metabot-enabled
jdbc-data-warehouse-max-connection-pool-size
jdbc-data-warehouse-unreturned-connection-timeout-seconds
jwt-attribute-email
jwt-attribute-firstname
jwt-attribute-groups
jwt-attribute-lastname
jwt-configured
jwt-enabled
jwt-group-mappings
jwt-group-sync
jwt-identity-provider-uri
jwt-user-provisioning-enabled?
jwt-shared-secret
last-acknowledged-version
last-used-native-database-id
ldap-attribute-email
ldap-attribute-firstname
ldap-attribute-lastname
ldap-bind-dn
ldap-configured?
ldap-enabled
ldap-group-base
ldap-group-mappings
ldap-group-membership-filter
ldap-group-sync
ldap-host
ldap-password
ldap-port
ldap-user-provisioning-enabled?
ldap-security
ldap-sync-user-attributes
ldap-sync-user-attributes-blacklist
ldap-user-base
ldap-user-filter
load-analytics-content
map-tile-server-url
metabot-default-embedding-model
metabot-feedback-url
metabot-get-prompt-templates-url
metabot-prompt-generator-token-limit
multi-setting-read-only
notebook-native-preview-shown
notebook-native-preview-sidebar-width
notification-link-base-url
num-metabot-choices
openai-api-key
openai-available-models
openai-model
openai-organization
other-sso-enabled?
password-complexity
persist-models-enabled
persisted-model-refresh-cron-schedule
premium-embedding-token
prometheus-server-port
query-caching-max-kb
query-caching-max-ttl
query-caching-min-ttl
query-caching-ttl-ratio
redirect-all-requests-to-https
reset-token-ttl-hours
retired-setting
retry-initial-interval
retry-max-attempts
retry-max-interval-millis
retry-multiplier
retry-randomization-factor
saml-application-name
saml-attribute-email
saml-attribute-firstname
saml-attribute-group
saml-attribute-lastname
saml-configured
saml-enabled
saml-group-mappings
saml-group-sync
saml-identity-provider-certificate
saml-identity-provider-issuer
saml-identity-provider-uri
saml-keystore-alias
saml-keystore-password
saml-keystore-path
saml-user-provisioning-enabled?
send-email-on-first-login-from-new-device
send-new-sso-user-admin-email?
session-cookie-samesite
session-cookies
session-timeout
setup-token
show-database-syncing-modal
show-metabase-links
site-url
site-uuid
site-uuid-for-premium-features-token-checks
site-uuid-for-unsubscribing-url
site-uuid-for-version-info-fetching
slack-app-token
slack-cached-channels-and-usernames
slack-channels-and-usernames-last-updated
slack-files-channel
slack-token
slack-token-valid?
snowplow-available
snowplow-enabled
snowplow-url
sql-jdbc-fetch-size
ssh-heartbeat-interval-sec
ssl-certificate-public-key
startup-time-millis
token-features
token-status
toucan-name
trial-banner-dismissal-timestamp
uncached-setting
user-visibility
version
version-info
version-info-last-checked})
(defn- defsetting-lint [node setting-name docstring options-list]
(let [anon-binding (common/with-macro-meta (hooks/token-node '_) node)
;; (defn my-setting [] ...)
getter-node (-> (list
(hooks/token-node 'defn)
setting-name
(hooks/string-node "Docstring.")
(hooks/vector-node []))
hooks/list-node
(with-meta (meta node)))
;; (defn my-setting! [_x] ...)
setter-node (-> (list
(hooks/token-node 'defn)
(with-meta
(hooks/token-node (symbol (str (hooks/sexpr setting-name) \!)))
(meta setting-name))
(hooks/string-node "Docstring.")
(hooks/vector-node [(hooks/token-node '_value-or-nil)]))
hooks/list-node
(with-meta (meta node))
common/add-lsp-ignore-unused-public-var-metadata)]
(when (nil? (second (drop-while (comp not #{[:k :export?]} first) options-list)))
(when-not (contains? ignored-implicit-export? (:value setting-name))
(hooks/reg-finding! (assoc (meta node)
:message "Setting definition must provide an explicit value for :export? indicating whether the setting should be exported or not with serialization."
:type :metabase/defsetting-must-specify-export))))
{:node (-> (hooks/list-node
(list
(hooks/token-node 'let)
;; include description and the options map so they can get validated as well.
(hooks/vector-node
[anon-binding docstring
anon-binding (hooks/map-node options-list)])
(hooks/reg-keyword! (-> (hooks/keyword-node (keyword (hooks/sexpr setting-name)))
(with-meta (meta setting-name)))
'metabase.models.setting/defsetting)
getter-node
setter-node))
(with-meta (meta node)))}))
(defn defsetting
"Rewrite a [[metabase.models.defsetting]] form like
(defsetting my-setting \"Description\" :type :boolean)
as
(let [_ \"Description\"
_ {:type :boolean}]
(defn my-setting \"Docstring.\" [])
(defn my-setting! \"Docstring.\" [_value-or-nil]))
for linting purposes."
[{:keys [node], :as context}]
(let [[setting-name docstring & options] (rest (:children node))]
(merge
context
(defsetting-lint node setting-name docstring options))))
(defn define-multi-setting
"Rewrite a [[metabase.models.define-multi-setting]] form like
(defsetting my-setting \"Description\" :key :type :boolean)
as
(let [_ \"Description\"
_ {:type :boolean, :multi-thunk :key}]
(defn my-setting \"Docstring.\" [])
(defn my-setting! \"Docstring.\" [_value-or-nil]))
for linting purposes."
[{:keys [node], :as context}]
(let [[setting-name docstring thunk & options] (rest (:children node))]
(merge
context
(defsetting-lint node setting-name docstring (concat options [(hooks/token-node :multi-thunk) thunk])))))
(defn define-multi-setting-impl
[context]
(letfn [(update-node [node]
(let [[_define-multi-setting-impl setting-name & more] (:children node)]
(-> (hooks/list-node
(list*
(hooks/token-node 'do)
;; For a qualified setting name like `oss-namespace/setting`, then leave the symbol as is, so the
;; usage of that namespace/var gets recorded. Otherwise for an unqualified name convert it to a
;; keyword so we don't get unresolved var errors.
(if (and (hooks/token-node? setting-name)
(qualified-symbol? (hooks/sexpr setting-name)))
setting-name
(-> (hooks/keyword-node (keyword (hooks/sexpr setting-name)))
(with-meta (meta setting-name))))
more))
(with-meta (meta node)))))]
(update context :node update-node)))
(comment
(defn- defsetting* [form]
(hooks/sexpr
(:node
(defsetting
{:node
(hooks/parse-string
(with-out-str
#_{:clj-kondo/ignore [:unresolved-namespace]}
(clojure.pprint/pprint
form)))}))))
(defn x []
(defsetting*
'(defsetting active-users-count
(deferred-tru "Cached number of active users. Refresh every 5 minutes.")
:visibility :admin
:type :integer
:default 0
:getter (fn []
(if-not ((requiring-resolve 'metabase.db/db-is-set-up?))
0
(cached-active-users-count)))))))

View File

@@ -1,310 +0,0 @@
(ns hooks.metabase.settings.models.setting
(:require
[clj-kondo.hooks-api :as hooks]
[hooks.common :as common]))
(defn lint-defsetting-namespace [node context]
(when-not (re-matches #"^metabase(?:-enterprise)?\.[^\.]+\.settings$" (name (:ns context)))
(hooks/reg-finding! (assoc (meta node)
:message "All defsettings should live in metabase[-enterprise].<module>.settings namespaces"
:type :metabase/defsetting-namespace))))
;;; TODO -- move this into a Kondo config file in `.clj-kondo/config/`
(def ^:private ignored-implicit-export?
'#{active-users-count
admin-email
analytics-uuid
anon-tracking-enabled
api-key
attachment-table-row-limit
audit-max-retention-days
bcc-enabled?
breakout-bin-width
check-for-updates
cloud-gateway-ips
config-from-file-sync-databases
custom-homepage
custom-homepage-dashboard
database-enable-actions
deprecation-notice-version
dismissed-custom-dashboard-toast
email-configured?
email-from-address
email-from-name
email-reply-to
email-smtp-host
email-smtp-password
email-smtp-port
email-smtp-security
email-smtp-username
embedding-app-origin
embedding-secret-key
enable-password-login
enable-public-sharing
enable-query-caching
engines
enum-cardinality-threshold
follow-up-email-sent
google-auth-auto-create-accounts-domain
google-auth-client-id
google-auth-configured
google-auth-enabled
has-sample-database?
has-user-setup
help-link
help-link-custom-destination
instance-creation
is-hosted?
is-metabot-enabled
jdbc-data-warehouse-max-connection-pool-size
jdbc-data-warehouse-unreturned-connection-timeout-seconds
jwt-attribute-email
jwt-attribute-firstname
jwt-attribute-groups
jwt-attribute-lastname
jwt-configured
jwt-enabled
jwt-group-mappings
jwt-group-sync
jwt-identity-provider-uri
jwt-user-provisioning-enabled?
jwt-shared-secret
last-acknowledged-version
last-used-native-database-id
ldap-attribute-email
ldap-attribute-firstname
ldap-attribute-lastname
ldap-bind-dn
ldap-configured?
ldap-enabled
ldap-group-base
ldap-group-mappings
ldap-group-membership-filter
ldap-group-sync
ldap-host
ldap-password
ldap-port
ldap-user-provisioning-enabled?
ldap-security
ldap-sync-user-attributes
ldap-sync-user-attributes-blacklist
ldap-user-base
ldap-user-filter
load-analytics-content
map-tile-server-url
metabot-default-embedding-model
metabot-feedback-url
metabot-get-prompt-templates-url
metabot-prompt-generator-token-limit
multi-setting-read-only
notebook-native-preview-shown
notebook-native-preview-sidebar-width
notification-link-base-url
num-metabot-choices
openai-api-key
openai-available-models
openai-model
openai-organization
other-sso-enabled?
password-complexity
persist-models-enabled
persisted-model-refresh-cron-schedule
premium-embedding-token
prometheus-server-port
query-caching-max-kb
query-caching-max-ttl
query-caching-min-ttl
query-caching-ttl-ratio
redirect-all-requests-to-https
reset-token-ttl-hours
retired-setting
retry-initial-interval
retry-max-attempts
retry-max-interval-millis
retry-multiplier
retry-randomization-factor
saml-application-name
saml-attribute-email
saml-attribute-firstname
saml-attribute-group
saml-attribute-lastname
saml-configured
saml-enabled
saml-group-mappings
saml-group-sync
saml-identity-provider-certificate
saml-identity-provider-issuer
saml-identity-provider-uri
saml-keystore-alias
saml-keystore-password
saml-keystore-path
saml-user-provisioning-enabled?
sdk-iframe-embed-setup-settings
send-email-on-first-login-from-new-device
send-new-sso-user-admin-email?
session-cookie-samesite
session-cookies
session-timeout
setup-token
show-database-syncing-modal
show-metabase-links
site-url
site-uuid
site-uuid-for-premium-features-token-checks
site-uuid-for-unsubscribing-url
site-uuid-for-version-info-fetching
slack-app-token
slack-cached-channels-and-usernames
slack-channels-and-usernames-last-updated
slack-files-channel
slack-token
slack-token-valid?
snowplow-available
snowplow-enabled
snowplow-url
sql-jdbc-fetch-size
ssh-heartbeat-interval-sec
ssl-certificate-public-key
startup-time-millis
token-features
token-status
toucan-name
trial-banner-dismissal-timestamp
uncached-setting
user-visibility
version
version-info
version-info-last-checked})
(defn- lint-defsetting-export [node]
(let [[_defsetting setting-name _docstring & options] (:children node)]
(when (nil? (second (drop-while (comp not #{[:k :export?]} first) options)))
(when-not (contains? ignored-implicit-export? (:value setting-name))
(hooks/reg-finding! (assoc (meta node)
:message "Setting definition must provide an explicit value for :export? indicating whether the setting should be exported or not with serialization."
:type :metabase/defsetting-must-specify-export))))))
(defn- defsetting-replacement-node [node]
(let [[_defsetting setting-name docstring & options-list] (:children node)
anon-binding (common/with-macro-meta (hooks/token-node '_) node)
;; (defn my-setting [] ...)
getter-node (-> (list
(hooks/token-node 'defn)
setting-name
(hooks/string-node "Docstring.")
(hooks/vector-node []))
hooks/list-node
(with-meta (meta node)))
;; (defn my-setting! [_x] ...)
setter-node (-> (list
(hooks/token-node 'defn)
(with-meta
(hooks/token-node (symbol (str (hooks/sexpr setting-name) \!)))
(meta setting-name))
(hooks/string-node "Docstring.")
(hooks/vector-node [(hooks/token-node '_value-or-nil)]))
hooks/list-node
(with-meta (meta node))
common/add-lsp-ignore-unused-public-var-metadata)]
(-> (hooks/list-node
(list
(hooks/token-node 'let)
;; include description and the options map so they can get validated as well.
(hooks/vector-node
[anon-binding docstring
anon-binding (hooks/map-node options-list)])
(hooks/reg-keyword! (-> (hooks/keyword-node (keyword (hooks/sexpr setting-name)))
(with-meta (meta setting-name)))
'metabase.settings.models.setting/defsetting)
getter-node
setter-node))
(with-meta (meta node)))))
(defn defsetting
"Rewrite a [[metabase.models.defsetting]] form like
(defsetting my-setting \"Description\" :type :boolean)
as
(let [_ \"Description\"
_ {:type :boolean}]
(defn my-setting \"Docstring.\" [])
(defn my-setting! \"Docstring.\" [_value-or-nil]))
for linting purposes."
[{:keys [node], :as context}]
(lint-defsetting-namespace node context)
(lint-defsetting-export node)
(update context :node defsetting-replacement-node))
(defn define-multi-setting-replacement-node [node]
(let [[defsetting setting-name docstring thunk & options] (:children node)]
(-> (hooks/list-node
(list*
defsetting
setting-name
docstring
(hooks/token-node :multi-thunk) thunk
options))
(with-meta (meta node))
defsetting-replacement-node)))
(defn define-multi-setting
"Rewrite a [[metabase.models.define-multi-setting]] form like
(defsetting my-setting \"Description\" :key :type :boolean)
as
(let [_ \"Description\"
_ {:type :boolean, :multi-thunk :key}]
(defn my-setting \"Docstring.\" [])
(defn my-setting! \"Docstring.\" [_value-or-nil]))
for linting purposes."
[{:keys [node], :as context}]
(update context :node define-multi-setting-replacement-node))
(defn define-multi-setting-impl
[context]
(letfn [(update-node [node]
(let [[_define-multi-setting-impl setting-name & more] (:children node)]
(-> (hooks/list-node
(list*
(hooks/token-node 'do)
;; For a qualified setting name like `oss-namespace/setting`, then leave the symbol as is, so the
;; usage of that namespace/var gets recorded. Otherwise for an unqualified name convert it to a
;; keyword so we don't get unresolved var errors.
(if (and (hooks/token-node? setting-name)
(qualified-symbol? (hooks/sexpr setting-name)))
setting-name
(-> (hooks/keyword-node (keyword (hooks/sexpr setting-name)))
(with-meta (meta setting-name))))
more))
(with-meta (meta node)))))]
(update context :node update-node)))
(comment
(defn- defsetting* [form]
(hooks/sexpr
(:node
(defsetting
{:node
(hooks/parse-string
(with-out-str
#_{:clj-kondo/ignore [:unresolved-namespace]}
(clojure.pprint/pprint
form)))}))))
(defn x []
(defsetting*
'(defsetting active-users-count
(deferred-tru "Cached number of active users. Refresh every 5 minutes.")
:visibility :admin
:type :integer
:default 0
:getter (fn []
(if-not ((requiring-resolve 'metabase.app-db.core/db-is-set-up?))
0
(cached-active-users-count)))))))

View File

@@ -2,8 +2,8 @@
(:require
[clj-kondo.hooks-api :as hooks]))
(defn iterate*
"(iterate {:as limit-spec} [s1 val1 & more] body-form body-form...)"
(defn with-gentest
"(with-gentest {:as limit-spec} [s1 val1 & more] body-form body-form...)"
[{{:keys [children] :as node} :node :as orig}]
(let [[token limit-spec bindings & body-nodes] children
node* (-> (hooks/list-node

View File

@@ -1,6 +1,6 @@
(ns hooks.metabase.test.util
(:require
[clj-kondo.hooks-api :as hooks]))
(:require [clj-kondo.hooks-api :as hooks]
[hooks.common :as common]))
(defn- namespaced-symbol-node? [node]
(when (hooks/token-node? node)

View File

@@ -1,15 +0,0 @@
(ns hooks.metabase.test.util.generators.jvm
(:require
[clj-kondo.hooks-api :as hooks]))
(defn with-random-cards
"(with-random-cards <metadata-provider> <integer-count-value> & body-forms)"
[{{:keys [children] :as node} :node :as orig}]
(let [[token num-cards & body-nodes] children
node* (-> (hooks/list-node
(list token num-cards
(hooks/list-node (list* (hooks/token-node 'let)
(hooks/vector-node [(hooks/token-node '&cards) (hooks/list-node ())])
body-nodes))))
(with-meta (meta node)))]
(assoc orig :node node*)))

View File

@@ -1,16 +0,0 @@
(ns hooks.metabase.util.timing
"Custom CLJ Kondo hooks for timing-related linting rules."
(:require [clj-kondo.hooks-api :as api]))
(defn discourage-millis-duration
"Discourage slow and unreliable duration calculations using System/currentTimeMillis "
[{:keys [node]}]
(let [sexpr (api/sexpr node)]
(when (and (list? sexpr) (= (first sexpr) '-) (>= (count sexpr) 2))
(let [first-arg (second sexpr)]
(when (= first-arg (list 'System/currentTimeMillis))
(api/reg-finding!
(assoc (meta node)
:message "Avoid System/currentTimeMillis for calculating durations. Use u/start-timer and u/since-ms for more reliability."
:type :metabase/discourage-millis-duration
:level :warning)))))))

View File

@@ -20,17 +20,15 @@
(lint-ns form (assoc-in config [:linters :metabase/modules :level] :warning)))
(deftest ^:parallel module-checker-allowed-modules-test
(are [task-namespace] (=? [{:message "Module task should not be used in the api module. [:metabase/modules api :uses]"
:type :metabase/modules}]
(lint-modules
'(ns metabase.api.search
(:require
[metabase.request.core :as request]
[task-namespace :as task]))
'{:metabase/modules {api {:uses #{request}}
request {:api #{metabase.request.core}}}}))
metabase.task
metabase.task.core))
(is (=? [{:message "Module task should not be used in the api module. [:metabase/modules api :uses]"
:type :metabase/modules}]
(lint-modules
'(ns metabase.api.search
(:require
[metabase.request.core :as request]
[metabase.task :as task]))
'{:metabase/modules {api {:uses #{request}}
request {:api #{metabase.request.core}}}}))))
(deftest ^:parallel module-checker-api-namespaces-test
(is (=? [{:message "Namespace metabase.search.config is not an allowed external API namespace for the search module. [:metabase/modules search :api]"

View File

@@ -1,7 +1,6 @@
(ns hooks.metabase.api.macros-test
(:require
[clj-kondo.hooks-api :as api]
[clj-kondo.impl.utils]
[clojure.test :refer :all]
[hooks.metabase.api.macros]))
@@ -70,41 +69,3 @@
(respond :x)
(catch Throwable e
(raise e))))))))
(defn- defendpoint [form]
(hooks.metabase.api.macros/defendpoint
{:node (-> form pr-str api/parse-string)}))
(defn- defendpoint-warnings
[form]
(binding [clj-kondo.impl.utils/*ctx* {:config {:linters {:metabase/validate-defendpoint-route-uses-kebab-case {:level :warning}
:metabase/validate-defendpoint-query-params-use-kebab-case {:level :warning}}}
:ignores (atom nil)
:findings (atom [])
:namespaces (atom {})}]
(defendpoint form)
@(:findings clj-kondo.impl.utils/*ctx*)))
(deftest ^:parallel validate-route-kebab-case-test
(is (=? [{:message "REST API routes and query params should use kebab-case, not snake_case"
:type :metabase/validate-defendpoint-route-uses-kebab-case
:row pos-int?
:col pos-int?}]
(defendpoint-warnings
'(api.macros/defendpoint :get "/:id/syncable_schemas" ; snake_case route should trigger a warning
"Returns a list of all syncable schemas found for the database `id`."
[{:keys [id]}]
id)))))
(deftest ^:parallel validate-query-params-kebab-case-test
(is (=? [{:message "REST API routes and query params should use kebab-case, not snake_case",
:type :metabase/validate-defendpoint-query-params-use-kebab-case
:row pos-int?
:col pos-int?}]
(defendpoint-warnings
'(api.macros/defendpoint :get "/:id/syncable-schemas"
"Returns a list of all syncable schemas found for the database `id`."
[{:keys [id]}
{:keys [filter_by]} :- [:map
[:filter_by {:optional true} :string]]]
id)))))

View File

@@ -0,0 +1,34 @@
(ns hooks.metabase.models.setting-test
(:require
[clj-kondo.hooks-api :as hooks]
[clojure.test :refer :all]
[hooks.metabase.models.setting]))
#_{:clj-kondo/ignore [:metabase/validate-deftest]}
(deftest ^:parallel defsetting-test
(let [node (-> '(defsetting active-users-count
(deferred-tru "Cached number of active users. Refreshed every 5 minutes.")
:visibility :admin
:type :integer
:default 0
:getter (fn []
(if-not ((requiring-resolve 'metabase.db/db-is-set-up?))
0
(cached-active-users-count))))
pr-str
hooks/parse-string)]
(is (= '(let [_ (deferred-tru "Cached number of active users. Refreshed every 5 minutes.")
_ {:default 0
:type :integer
:getter (fn []
(if-not ((requiring-resolve 'metabase.db/db-is-set-up?))
0
(cached-active-users-count)))
:visibility :admin}]
:active-users-count
(defn active-users-count "Docstring." [])
(defn active-users-count! "Docstring." [_value-or-nil]))
(-> {:node node}
hooks.metabase.models.setting/defsetting
:node
hooks/sexpr)))))

View File

@@ -1,34 +0,0 @@
(ns hooks.metabase.settings.models.setting-test
(:require
[clj-kondo.hooks-api :as hooks]
[clojure.test :refer :all]
[hooks.metabase.settings.models.setting]))
#_{:clj-kondo/ignore [:metabase/validate-deftest]}
(deftest ^:parallel defsetting-test
(let [node (-> '(defsetting active-users-count
(deferred-tru "Cached number of active users. Refreshed every 5 minutes.")
:visibility :admin
:type :integer
:default 0
:getter (fn []
(if-not ((requiring-resolve 'metabase.app-db.core/db-is-set-up?))
0
(cached-active-users-count))))
pr-str
hooks/parse-string)]
(is (= '(let [_ (deferred-tru "Cached number of active users. Refreshed every 5 minutes.")
_ {:default 0
:type :integer
:getter (fn []
(if-not ((requiring-resolve 'metabase.app-db.core/db-is-set-up?))
0
(cached-active-users-count)))
:visibility :admin}]
:active-users-count
(defn active-users-count "Docstring." [])
(defn active-users-count! "Docstring." [_value-or-nil]))
(-> {:node node, :ns 'hooks.metabase.settings.models.setting-test}
hooks.metabase.settings.models.setting/defsetting
:node
hooks/sexpr)))))

View File

@@ -38,321 +38,311 @@
;; zero get indented 2 spaces if not on the first line regardless of number of elements on the first line.
:extra-indents
{;; clojure.core stuff
fn* [[:inner 0]]
let* [[:block 1]]
with-meta [[:default]]
fn* [[:inner 0]]
let* [[:block 1]]
with-meta [[:default]]
;; library stuff. Libraries that specify their own `:style/indent` specs are included in the section after this
;; since it is generated automatically.
clojure.core.logic/fresh [[:block 1]]
clojure.core.logic/matcha [[:block 1]]
clojure.core.logic/run [[:block 2]]
clojure.core.match/match [[:block 1]]
clojure.test.check.properties/for-all [[:block 1]]
clojure.test.check.properties/for-all* [[:block 1]]
clojure.test.check/quick-check [[:block 1]]
clojure.tools.macro/macrolet [[:block 1] [:inner 2 0]]
clojurewerkz.quartzite.jobs/build [[:default]]
clojurewerkz.quartzite.jobs/with-description [[:default]]
clojurewerkz.quartzite.jobs/with-identity [[:default]]
clojurewerkz.quartzite.schedule.cron/schedule [[:default]]
clojurewerkz.quartzite.schedule.cron/with-misfire-handling-instruction-do-nothing [[:default]]
clojurewerkz.quartzite.triggers/build [[:default]]
clojurewerkz.quartzite.triggers/with-description [[:default]]
clojurewerkz.quartzite.triggers/with-identity [[:default]]
clojurewerkz.quartzite.triggers/with-priority [[:default]]
clojurewerkz.quartzite.triggers/with-schedule [[:default]]
def-map-type [[:block 2] [:inner 1]]
defprotocol+ [[:block 1] [:inner 1]]
defrecord+ [[:block 2] [:inner 1]]
deftype+ [[:block 2] [:inner 1]]
net.cgrand.macrovich/case [[:block 0]]
potemkin.namespaces/import-vars [[:default]]
potemkin.types/def-map-type [[:block 2] [:inner 1]]
potemkin.types/defprotocol+ [[:block 1] [:inner 1]]
potemkin.types/defrecord+ [[:block 2] [:inner 1]]
potemkin.types/deftype+ [[:block 2] [:inner 1]]
potemkin/def-map-type [[:block 2] [:inner 1]]
potemkin/defprotocol+ [[:block 1] [:inner 1]]
potemkin/defrecord+ [[:block 2] [:inner 1]]
potemkin/deftype+ [[:block 2] [:inner 1]]
potemkin/import-vars [[:default]]
taoensso.nippy/extend-freeze [[:inner 0]]
taoensso.nippy/extend-thaw [[:inner 0]]
throttle.core/with-throttling [[:block 1]]
clojure.core.logic/fresh [[:block 1]]
clojure.core.logic/matcha [[:block 1]]
clojure.core.logic/run [[:block 2]]
clojure.core.match/match [[:block 1]]
clojure.test.check.properties/for-all [[:block 1]]
clojure.test.check.properties/for-all* [[:block 1]]
clojure.test.check/quick-check [[:block 1]]
clojure.tools.macro/macrolet [[:block 1] [:inner 2 0]]
clojurewerkz.quartzite.jobs/build [[:default]]
clojurewerkz.quartzite.jobs/with-description [[:default]]
clojurewerkz.quartzite.jobs/with-identity [[:default]]
clojurewerkz.quartzite.schedule.cron/schedule [[:default]]
clojurewerkz.quartzite.schedule.cron/with-misfire-handling-instruction-do-nothing [[:default]]
clojurewerkz.quartzite.triggers/build [[:default]]
clojurewerkz.quartzite.triggers/with-description [[:default]]
clojurewerkz.quartzite.triggers/with-identity [[:default]]
clojurewerkz.quartzite.triggers/with-priority [[:default]]
clojurewerkz.quartzite.triggers/with-schedule [[:default]]
def-map-type [[:block 2] [:inner 1]]
defprotocol+ [[:block 1] [:inner 1]]
defrecord+ [[:block 2] [:inner 1]]
deftype+ [[:block 2] [:inner 1]]
net.cgrand.macrovich/case [[:block 0]]
potemkin.namespaces/import-vars [[:default]]
potemkin.types/def-map-type [[:block 2] [:inner 1]]
potemkin.types/defprotocol+ [[:block 1] [:inner 1]]
potemkin.types/defrecord+ [[:block 2] [:inner 1]]
potemkin.types/deftype+ [[:block 2] [:inner 1]]
potemkin/def-map-type [[:block 2] [:inner 1]]
potemkin/defprotocol+ [[:block 1] [:inner 1]]
potemkin/defrecord+ [[:block 2] [:inner 1]]
potemkin/deftype+ [[:block 2] [:inner 1]]
potemkin/import-vars [[:default]]
taoensso.nippy/extend-freeze [[:inner 0]]
taoensso.nippy/extend-thaw [[:inner 0]]
throttle.core/with-throttling [[:block 1]]
;; the rest of the stuff in this list is generated automatically based on `:style/indent` specs
;; by running [[dev.generate-cljfmt-config/specs]]... use that code to update this list
metabase-enterprise.cache.config-test/with-temp-persist-models [[:block 1]]
metabase-enterprise.impersonation.util-test/with-impersonations! [[:inner 0]]
metabase-enterprise.sandbox.test-util/with-gtaps! [[:inner 0]]
metabase-enterprise.sandbox.test-util/with-gtaps-for-user! [[:inner 0]]
metabase-enterprise.sandbox.test-util/with-user-attributes [[:block 2]]
metabase-enterprise.serialization.test-util/with-dbs [[:inner 0]]
metabase-enterprise.serialization.test-util/with-random-dump-dir [[:block 1]]
metabase-enterprise.serialization.test-util/with-world [[:block 0]]
metabase-enterprise.sso.integrations.jwt-test/with-users-with-email-deleted [[:block 1]]
metabase-enterprise.test/with-gtaps! [[:inner 0]]
metabase-enterprise.test/with-gtaps-for-user! [[:inner 0]]
metabase-enterprise.test/with-user-attributes [[:block 2]]
metabase.actions.actions-test/with-actions-test-data-and-actions-permissively-enabled [[:block 0]]
metabase.actions.test-util/with-actions [[:block 1]]
metabase.actions.test-util/with-actions-disabled [[:block 0]]
metabase.actions.test-util/with-actions-enabled [[:block 0]]
metabase.actions.test-util/with-actions-test-data [[:inner 0]]
metabase.actions.test-util/with-actions-test-data-and-actions-enabled [[:block 0]]
metabase.actions.test-util/with-actions-test-data-tables [[:block 1]]
metabase.actions.test-util/with-empty-db [[:inner 0]]
metabase.actions.test-util/with-temp-test-data [[:inner 0]]
metabase.api.common/check [[:default]]
metabase.api.common/define-routes [[:block 0]]
metabase.api.common/let-404 [[:block 1]]
metabase.app-db.core/with-application-db [[:inner 0]]
metabase.app-db.liquibase/with-liquibase [[:block 1]]
metabase.app-db.liquibase/with-scope-locked [[:block 1]]
metabase.app-db.schema-migrations-test.impl/test-migrations [[:block 2]]
metabase.cmd.copy/step [[:block 1]]
metabase.cmd.copy/with-connection-rollback-only [[:block 1]]
metabase.cmd.copy/with-disabled-db-constraints [[:block 2]]
metabase.collections-rest.api-test/with-collection-hierarchy [[:block 1]]
metabase.collections-rest.api-test/with-french-user-and-personal-collection [[:block 2]]
metabase.collections-rest.api-test/with-some-children-of-collection [[:block 1]]
metabase.collections.models.collection-test/with-collection-hierarchy [[:block 1]]
metabase.collections.models.collection-test/with-current-user-perms-for-collections [[:block 1]]
metabase.collections.models.collection-test/with-personal-and-impersonal-collections [[:block 1]]
metabase.config/build-type-case [[:block 0]]
metabase.dashboards.models.dashboard-test/with-dash-in-collection [[:inner 0]]
metabase.driver-api.core/match [[:inner 0]]
metabase.driver-api.core/match-one [[:inner 0]]
metabase.driver-api.core/replace [[:inner 0]]
metabase.driver-api.core/replace-in [[:inner 0]]
metabase.driver.bigquery-cloud-sdk.query-processor/with-temporal-type [[:default]]
metabase.driver.druid.client/DELETE [[:default]]
metabase.driver.druid.client/GET [[:default]]
metabase.driver.druid.client/POST [[:default]]
metabase.driver.druid.query-processor-test/druid-query [[:block 0]]
metabase.driver.druid.query-processor-test/druid-query-returning-rows [[:block 0]]
metabase.driver.mongo.query-processor/mongo-let [[:block 1]]
metabase.driver.sql-jdbc.actions/with-jdbc-transaction [[:block 1]]
metabase.driver.sql-jdbc.connection/with-connection-spec-for-testing-connection [[:block 1]]
metabase.driver.sql-jdbc.execute.diagnostic/capturing-diagnostic-info [[:block 1]]
metabase.driver.sql.query-processor/apply-top-level-clause [[:default]]
metabase.driver/with-driver [[:block 1]]
metabase.embedding-rest.api.embed-test/do-response-formats [[:block 1]]
metabase.embedding-rest.api.embed-test/with-embedding-enabled-and-new-secret-key [[:block 0]]
metabase.embedding-rest.api.embed-test/with-embedding-enabled-and-temp-card-referencing [[:block 3]]
metabase.embedding-rest.api.embed-test/with-embedding-enabled-and-temp-dashcard-referencing [[:block 3]]
metabase.embedding-rest.api.embed-test/with-new-secret-key [[:block 0]]
metabase.embedding-rest.api.embed-test/with-temp-card [[:block 1]]
metabase.embedding-rest.api.embed-test/with-temp-dashcard [[:block 1]]
metabase.embedding-rest.api.preview-embed-test/with-embedding-enabled-and-new-secret-key [[:block 0]]
metabase.embedding-rest.api.preview-embed-test/with-new-secret-key [[:block 0]]
metabase.lib.binning/with-binning [[:default]]
metabase.lib.core/with-binning [[:default]]
metabase.lib.core/with-join-alias [[:default]]
metabase.lib.core/with-join-conditions [[:default]]
metabase.lib.join/with-join-alias [[:default]]
metabase.lib.join/with-join-conditions [[:default]]
metabase.lib.test-util.macros/$ids [[:inner 0]]
metabase.lib.test-util.macros/mbql-5-query [[:inner 0]]
metabase.lib.test-util.macros/mbql-query [[:inner 0]]
metabase.lib.util.match/match [[:inner 0]]
metabase.lib.util.match/match-one [[:inner 0]]
metabase.lib.util.match/match-lite [[:inner 0]]
metabase.lib.util.match/match-lite-recursive [[:inner 0]]
metabase.lib.util.match/replace [[:inner 0]]
metabase.lib.util.match/replace-in [[:inner 0]]
metabase.models.interface/define-batched-hydration-method [[:inner 0]]
metabase.models.interface/define-simple-hydration-method [[:inner 0]]
metabase.parameters.chain-filter-test/with-clean-field-values-for-field [[:block 1]]
metabase.parameters.chain-filter-test/with-fk-field-to-field-remapping! [[:block 0]]
metabase.parameters.chain-filter-test/with-human-readable-values-remapping! [[:block 0]]
metabase.permissions.core/allow-changing-all-users-group-members [[:block 0]]
metabase.permissions.core/disable-perms-cache [[:block 0]]
metabase.public-sharing-rest.api-test/with-sharing-enabled-and-temp-card-referencing [[:block 3]]
metabase.public-sharing-rest.api-test/with-sharing-enabled-and-temp-dashcard-referencing [[:block 3]]
metabase.public-sharing-rest.api-test/with-temp-public-card [[:block 1]]
metabase.public-sharing-rest.api-test/with-temp-public-dashboard [[:block 1]]
metabase.public-sharing-rest.api-test/with-temp-public-dashboard-and-card [[:block 1]]
metabase.pulse.api.alert-test/with-alert-in-collection [[:block 1]]
metabase.pulse.dashboard-subscription-test/with-link-card-fixture-for-dashboard [[:block 2]]
metabase.pulse.models.pulse-test/with-dashboard-subscription-in-collection [[:block 1]]
metabase.pulse.models.pulse-test/with-pulse-in-collection [[:inner 0]]
metabase.queries.api.card-test/with-cards-in-readable-collection [[:block 1]]
metabase.queries.api.card-test/with-cards-in-writeable-collection [[:block 1]]
metabase.queries.api.card-test/with-ordered-items [[:inner 0]]
metabase.queries.api.card-test/with-temp-native-card [[:block 1]]
metabase.queries.api.card-test/with-temp-native-card-with-params [[:block 1]]
metabase.query-processor-test.pipeline-queries-test/pmbql-query [[:block 1]]
metabase.query-processor-test.pipeline-queries-test/run-pmbql-query [[:block 1]]
metabase.query-processor.debug/debug> [[:block 0]]
metabase.query-processor.error-type/deferror [[:block 1]]
metabase.query-processor.middleware.cache-backend.interface/with-cached-results [[:block 4]]
metabase.query-processor.middleware.process-userland-query-test/with-query-execution [[:block 1]]
metabase.query-processor.store/cached [[:block 1]]
metabase.query-processor.store/with-metadata-provider [[:inner 0]]
metabase.query-processor.streaming/streaming-response [[:block 1]]
metabase.query-processor.test-util/format-rows-by [[:default]]
metabase.request.core/as-admin [[:block 0]]
metabase.request.core/with-current-request [[:inner 0]]
metabase.request.core/with-current-user [[:inner 0]]
metabase.request.core/with-limit-and-offset [[:inner 0]]
metabase.request.session/as-admin [[:block 0]]
metabase.request.session/with-current-user [[:inner 0]]
metabase.sample-data.impl-test/with-temp-sample-database-db [[:block 1]]
metabase.search.api-test/do-test-users [[:block 1]]
metabase.server.middleware.session/as-admin [[:block 0]]
metabase.server.middleware.session/with-current-user [[:inner 0]]
metabase.server.streaming-response-test/with-streaming-response-thread-pool [[:block 0]]
metabase.server.streaming-response-test/with-test-driver-db! [[:block 0]]
metabase.server.streaming-response/streaming-response [[:block 2]]
metabase.session.api/http-401-on-error [[:block 0]]
metabase.settings.core/define-multi-setting [[:inner 0]]
metabase.settings.core/defsetting [[:block 1]]
metabase.settings.models.setting.multi-setting/define-multi-setting [[:inner 0]]
metabase.settings.models.setting/defsetting [[:block 1]]
metabase.sync.fetch-metadata/log-if-error [[:block 1]]
metabase.sync.sync-metadata.fields-test/with-test-db [[:block 0]]
metabase.sync.util/do-with-start-and-finish-debug-logging [[:default]]
metabase.sync.util/sum-for [[:block 1]]
metabase.sync.util/sync-operation [[:block 3]]
metabase.sync.util/with-db-logging-disabled [[:default]]
metabase.sync.util/with-duplicate-ops-prevented [[:default]]
metabase.sync.util/with-emoji-progress-bar [[:block 1]]
metabase.sync.util/with-error-handling [[:block 1]]
metabase.sync.util/with-start-and-finish-logging [[:default]]
metabase.sync.util/with-start-and-finish-logging* [[:default]]
metabase.sync.util/with-sync-events [[:default]]
metabase.task-history.models.task-history/with-task-history [[:block 1]]
metabase.task.core/rerun-on-error [[:block 1]]
metabase.task.impl/rerun-on-error [[:block 1]]
metabase.test.data.datasets/test-driver [[:inner 0]]
metabase.test.data.datasets/test-drivers [[:inner 0]]
metabase.test.data.one-off-dbs/with-blank-db [[:block 0]]
metabase.test.data.one-off-dbs/with-blueberries-db [[:block 0]]
metabase.test.data.users/with-test-user [[:inner 0]]
metabase.test.data/$ids [[:inner 0]]
metabase.test.data/dataset [[:inner 0]]
metabase.test.data/mbql-query [[:inner 0]]
metabase.test.data/query [[:block 1]]
metabase.test.data/run-mbql-query [[:inner 0]]
metabase.test.data/with-empty-h2-app-db! [[:block 0]]
metabase.test.data/with-temp-copy-of-db [[:block 0]]
metabase.test.util.public-settings/with-additional-premium-features [[:block 1]]
metabase.test.util.public-settings/with-premium-features [[:block 1]]
metabase.test.util.thread-local/test-helpers-set-global-values! [[:block 0]]
metabase.test.util/discard-setting-changes [[:block 1]]
metabase.test.util/let-url [[:block 1]]
metabase.test.util/throw-if-called! [[:block 1]]
metabase.test.util/with-temp-scheduler [[:block 0]]
metabase.test.util/with-temp-vals-in-db [[:block 3]]
metabase.test.util/with-user-in-groups [[:inner 0]]
metabase.test/$ids [[:inner 0]]
metabase.test/as-admin [[:inner 0]]
metabase.test/dataset [[:inner 0]]
metabase.test/discard-setting-changes [[:block 1]]
metabase.test/format-rows-by [[:default]]
metabase.test/let-url [[:block 1]]
metabase.test/mbql-query [[:inner 0]]
metabase.test/query [[:block 1]]
metabase.test/run-mbql-query [[:inner 0]]
metabase.test/test-driver [[:inner 0]]
metabase.test/test-drivers [[:inner 0]]
metabase.test/test-helpers-set-global-values! [[:block 0]]
metabase.test/throw-if-called! [[:block 1]]
metabase.test/with-actions [[:block 1]]
metabase.test/with-actions-disabled [[:block 0]]
metabase.test/with-actions-enabled [[:block 0]]
metabase.test/with-actions-test-data [[:inner 0]]
metabase.test/with-actions-test-data-and-actions-enabled [[:block 0]]
metabase.test/with-actions-test-data-tables [[:block 1]]
metabase.test/with-additional-premium-features [[:block 1]]
metabase.test/with-current-user [[:inner 0]]
metabase.test/with-driver [[:block 1]]
metabase.test/with-empty-db [[:inner 0]]
metabase.test/with-empty-h2-app-db! [[:block 0]]
metabase.test/with-metadata-provider [[:inner 0]]
metabase.test/with-premium-features [[:block 1]]
metabase.test/with-temp [[:inner 0]]
metabase.test/with-temp-copy-of-db [[:block 0]]
metabase.test/with-temp-scheduler [[:block 0]]
metabase.test/with-temp-test-data [[:inner 0]]
metabase.test/with-temp-vals-in-db [[:block 3]]
metabase.test/with-test-user [[:inner 0]]
metabase.test/with-user-in-groups [[:inner 0]]
metabase.timeseries-query-processor-test.util/test-timeseries-drivers [[:block 0]]
metabase.timeseries-query-processor-test/run-mbql-query [[:inner 0]]
metabase.upload.impl-test/with-upload-table! [[:inner 0]]
metabase.upload.types/with-currency [[:default]]
metabase.upload.types/with-parens [[:default]]
metabase.util/for-map [[:block 1]]
metabase.util.async/cancelable-thread [[:block 0]]
metabase.util.date-2/with-time-zone-same-instant [[:default]]
metabase.util.encryption-test/with-secret-key [[:block 1]]
metabase.util.honey-sql-2/with-database-type-info [[:default]]
metabase.util.i18n/deferred-trs [[:default]]
metabase.util.i18n/deferred-tru [[:default]]
metabase.util.i18n/trs [[:default]]
metabase.util.i18n/trsn [[:default]]
metabase.util.i18n/tru [[:default]]
metabase.util.i18n/trun [[:default]]
metabase.util.jvm/auto-retry [[:block 1]]
metabase.util.jvm/varargs [[:block 1]]
metabase.util.jvm/with-us-locale [[:block 0]]
metabase.util.malli/disable-enforcement [[:block 0]]
metabase.util.malli/with [[:default]]
metabase.util.malli/with-api-error-message [[:default]]
metabase.util.namespaces/import-fns [[:default]]
metabase.util.queue-test/await-test-while [[:block 1]]
metabase.util/auto-retry [[:block 1]]
metabase.util/case-enum [[:block 1]]
metabase.util/ignore-exceptions [[:block 0]]
metabase.util/or-with [[:block 1]]
metabase.util/profile [[:block 1]]
metabase.util/prog1 [[:inner 0]]
metabase.util/varargs [[:inner 0]]
metabase.util/with-us-locale [[:block 0]]
metabuild-common.core/exit-when-finished-nonzero-on-exception [[:block 0]]
metabuild-common.core/step [[:block 1]]
metabuild-common.entrypoint/exit-when-finished-nonzero-on-exception [[:block 0]]
metabuild-common.steps/step [[:block 1]]
methodical.core/defmethod [[:inner 0]]
methodical.core/defmulti [[:inner 0]]
methodical.core/with-dispatcher [[:default]]
methodical.core/with-method-table [[:default]]
methodical.core/with-prefers [[:default]]
methodical.interface/with-dispatcher [[:default]]
methodical.interface/with-method-table [[:default]]
methodical.interface/with-prefers [[:default]]
methodical.macros/defmethod [[:inner 0]]
methodical.macros/defmulti [[:inner 0]]
saml20-clj.sp.response/validate-confirmation-datas [[:block 1]]
toucan2.connection/with-transaction [[:block 1]]
toucan2.core/build [[:block 0]]
toucan2.core/compile [[:block 0]]
toucan2.core/debug [[:block 0]]
toucan2.core/define-after-insert [[:inner 0]]
toucan2.core/define-after-select [[:inner 0]]
toucan2.core/define-after-update [[:inner 0]]
toucan2.core/define-before-delete [[:inner 0]]
toucan2.core/define-before-insert [[:inner 0]]
toucan2.core/define-before-select [[:inner 0]]
toucan2.core/define-default-fields [[:inner 0]]
toucan2.core/define-named-query [[:block 1]]
toucan2.core/deftransforms [[:block 1]]
toucan2.core/with-transaction [[:block 1]]
toucan2.tools.after-insert/define-after-insert [[:inner 0]]
toucan2.tools.after-select/define-after-select [[:inner 0]]
toucan2.tools.after-update/define-after-update [[:inner 0]]
toucan2.tools.after/define-after-impl [[:default]]
toucan2.tools.before-delete/define-before-delete [[:inner 0]]
toucan2.tools.before-insert/define-before-insert [[:inner 0]]
toucan2.tools.before-select/define-before-select [[:inner 0]]
toucan2.tools.compile/build [[:block 0]]
toucan2.tools.compile/compile [[:block 0]]
toucan2.tools.compile/resolved [[:block 0]]
toucan2.tools.debug/debug [[:block 0]]
toucan2.tools.default-fields/define-default-fields [[:inner 0]]
toucan2.tools.named-query/define-named-query [[:block 1]]
toucan2.tools.simple-out-transform/define-out-transform [[:inner 0]]
toucan2.tools.transformed/deftransforms [[:block 1]]
toucan2.tools.with-temp/with-temp [[:inner 0]]
toucan2.util/try-with-error-context [[:inner 0]]}}
metabase-enterprise.impersonation.util-test/with-impersonations! [[:inner 0]]
metabase-enterprise.cache.config-test/with-temp-persist-models [[:block 1]]
metabase-enterprise.sandbox.test-util/with-gtaps! [[:inner 0]]
metabase-enterprise.sandbox.test-util/with-gtaps-for-user! [[:inner 0]]
metabase-enterprise.sandbox.test-util/with-user-attributes [[:block 2]]
metabase-enterprise.serialization.test-util/with-dbs [[:inner 0]]
metabase-enterprise.serialization.test-util/with-random-dump-dir [[:block 1]]
metabase-enterprise.serialization.test-util/with-world [[:block 0]]
metabase-enterprise.sso.integrations.jwt-test/with-users-with-email-deleted [[:block 1]]
metabase-enterprise.test/with-gtaps! [[:inner 0]]
metabase-enterprise.test/with-gtaps-for-user! [[:inner 0]]
metabase-enterprise.test/with-user-attributes [[:block 2]]
metabase.actions.actions-test/with-actions-test-data-and-actions-permissively-enabled [[:block 0]]
metabase.actions.test-util/with-actions [[:block 1]]
metabase.actions.test-util/with-actions-disabled [[:block 0]]
metabase.actions.test-util/with-actions-enabled [[:block 0]]
metabase.actions.test-util/with-actions-test-data [[:inner 0]]
metabase.actions.test-util/with-actions-test-data-and-actions-enabled [[:block 0]]
metabase.actions.test-util/with-actions-test-data-tables [[:block 1]]
metabase.actions.test-util/with-empty-db [[:inner 0]]
metabase.actions.test-util/with-temp-test-data [[:inner 0]]
metabase.api.card-test/with-cards-in-readable-collection [[:block 1]]
metabase.api.card-test/with-cards-in-writeable-collection [[:block 1]]
metabase.api.card-test/with-ordered-items [[:inner 0]]
metabase.api.card-test/with-temp-native-card! [[:block 1]]
metabase.api.card-test/with-temp-native-card-with-params! [[:block 1]]
metabase.api.collection-test/with-collection-hierarchy [[:block 1]]
metabase.api.collection-test/with-french-user-and-personal-collection [[:block 2]]
metabase.api.collection-test/with-some-children-of-collection [[:block 1]]
metabase.api.common/check [[:default]]
metabase.api.common/define-routes [[:block 0]]
metabase.api.common/let-404 [[:block 1]]
metabase.api.embed-test/do-response-formats [[:block 1]]
metabase.api.embed-test/with-embedding-enabled-and-new-secret-key [[:block 0]]
metabase.api.embed-test/with-embedding-enabled-and-temp-card-referencing [[:block 3]]
metabase.api.embed-test/with-embedding-enabled-and-temp-dashcard-referencing [[:block 3]]
metabase.api.embed-test/with-new-secret-key [[:block 0]]
metabase.api.embed-test/with-temp-card [[:block 1]]
metabase.api.embed-test/with-temp-dashcard [[:block 1]]
metabase.api.preview-embed-test/with-embedding-enabled-and-new-secret-key [[:block 0]]
metabase.api.preview-embed-test/with-new-secret-key [[:block 0]]
metabase.cmd.copy/step [[:block 1]]
metabase.cmd.copy/with-connection-rollback-only [[:block 1]]
metabase.cmd.copy/with-disabled-db-constraints [[:block 2]]
metabase.config/build-type-case [[:block 0]]
metabase.dashboard-subscription-test/with-link-card-fixture-for-dashboard [[:block 2]]
metabase.db.liquibase/with-liquibase [[:block 1]]
metabase.db.liquibase/with-scope-locked [[:block 1]]
metabase.db.schema-migrations-test.impl/test-migrations [[:block 2]]
metabase.db/with-application-db [[:inner 0]]
metabase.driver.bigquery-cloud-sdk.query-processor/with-temporal-type [[:default]]
metabase.driver.druid.client/DELETE [[:default]]
metabase.driver.druid.client/GET [[:default]]
metabase.driver.druid.client/POST [[:default]]
metabase.driver.druid.query-processor-test/druid-query [[:block 0]]
metabase.driver.druid.query-processor-test/druid-query-returning-rows [[:block 0]]
metabase.driver.mongo.query-processor/mongo-let [[:block 1]]
metabase.driver.sql-jdbc.actions/with-jdbc-transaction [[:block 1]]
metabase.driver.sql-jdbc.connection/with-connection-spec-for-testing-connection [[:block 1]]
metabase.driver.sql-jdbc.execute.diagnostic/capturing-diagnostic-info [[:block 1]]
metabase.driver.sql.query-processor/apply-top-level-clause [[:default]]
metabase.driver/with-driver [[:block 1]]
metabase.lib.binning/with-binning [[:default]]
metabase.lib.core/with-binning [[:default]]
metabase.lib.core/with-join-alias [[:default]]
metabase.lib.core/with-join-conditions [[:default]]
metabase.lib.join/with-join-alias [[:default]]
metabase.lib.join/with-join-conditions [[:default]]
metabase.lib.test-util.macros/$ids [[:inner 0]]
metabase.lib.test-util.macros/mbql-query [[:inner 0]]
metabase.lib.util.match/match [[:inner 0]]
metabase.lib.util.match/match-one [[:inner 0]]
metabase.lib.util.match/replace [[:inner 0]]
metabase.lib.util.match/replace-in [[:inner 0]]
metabase.models.collection-test/with-collection-hierarchy [[:block 1]]
metabase.models.collection-test/with-current-user-perms-for-collections [[:block 1]]
metabase.models.collection-test/with-personal-and-impersonal-collections [[:block 1]]
metabase.models.dashboard-test/with-dash-in-collection [[:inner 0]]
metabase.models.interface/define-batched-hydration-method [[:inner 0]]
metabase.models.interface/define-simple-hydration-method [[:inner 0]]
metabase.models.params.chain-filter-test/with-clean-field-values-for-field [[:block 1]]
metabase.models.params.chain-filter-test/with-fk-field-to-field-remapping [[:block 0]]
metabase.models.params.chain-filter-test/with-human-readable-values-remapping [[:block 0]]
metabase.models.setting.multi-setting/define-multi-setting [[:inner 0]]
metabase.models.setting/defsetting [[:block 1]]
metabase.models.task-history/with-task-history [[:block 1]]
metabase.permissions.core/allow-changing-all-users-group-members [[:block 0]]
metabase.permissions.core/disable-perms-cache [[:block 0]]
metabase.public-sharing.api-test/with-sharing-enabled-and-temp-card-referencing [[:block 3]]
metabase.public-sharing.api-test/with-sharing-enabled-and-temp-dashcard-referencing [[:block 3]]
metabase.public-sharing.api-test/with-temp-public-card [[:block 1]]
metabase.public-sharing.api-test/with-temp-public-dashboard [[:block 1]]
metabase.public-sharing.api-test/with-temp-public-dashboard-and-card [[:block 1]]
metabase.pulse.api.alert-test/with-alert-in-collection [[:block 1]]
metabase.pulse.models.pulse-test/with-dashboard-subscription-in-collection [[:block 1]]
metabase.pulse.models.pulse-test/with-pulse-in-collection [[:inner 0]]
metabase.query-processor-test.pipeline-queries-test/pmbql-query [[:block 1]]
metabase.query-processor-test.pipeline-queries-test/run-pmbql-query [[:block 1]]
metabase.query-processor.debug/debug> [[:block 0]]
metabase.query-processor.error-type/deferror [[:block 1]]
metabase.query-processor.middleware.cache-backend.interface/with-cached-results [[:block 4]]
metabase.query-processor.middleware.process-userland-query-test/with-query-execution [[:block 1]]
metabase.query-processor.store/cached [[:block 1]]
metabase.query-processor.store/with-metadata-provider [[:inner 0]]
metabase.query-processor.streaming/streaming-response [[:block 1]]
metabase.query-processor.test-util/format-rows-by [[:default]]
metabase.request.core/as-admin [[:block 0]]
metabase.request.core/with-current-request [[:inner 0]]
metabase.request.core/with-current-user [[:inner 0]]
metabase.request.core/with-limit-and-offset [[:inner 0]]
metabase.request.session/as-admin [[:block 0]]
metabase.request.session/with-current-user [[:inner 0]]
metabase.sample-data-test/with-temp-sample-database-db [[:block 1]]
metabase.search.api-test/do-test-users [[:block 1]]
metabase.server.middleware.session/as-admin [[:block 0]]
metabase.server.middleware.session/with-current-user [[:inner 0]]
metabase.server.streaming-response-test/with-streaming-response-thread-pool [[:block 0]]
metabase.server.streaming-response-test/with-test-driver-db! [[:block 0]]
metabase.server.streaming-response/streaming-response [[:block 2]]
metabase.session.api/http-401-on-error [[:block 0]]
metabase.sync.fetch-metadata/log-if-error [[:block 1]]
metabase.sync.sync-metadata.fields-test/with-test-db [[:block 0]]
metabase.sync.util/do-with-start-and-finish-debug-logging [[:default]]
metabase.sync.util/sum-for [[:block 1]]
metabase.sync.util/sync-operation [[:block 3]]
metabase.sync.util/with-db-logging-disabled [[:default]]
metabase.sync.util/with-duplicate-ops-prevented [[:default]]
metabase.sync.util/with-emoji-progress-bar [[:block 1]]
metabase.sync.util/with-error-handling [[:block 1]]
metabase.sync.util/with-start-and-finish-logging [[:default]]
metabase.sync.util/with-start-and-finish-logging* [[:default]]
metabase.sync.util/with-sync-events [[:default]]
metabase.task/rerun-on-error [[:block 1]]
metabase.test/$ids [[:inner 0]]
metabase.test/as-admin [[:inner 0]]
metabase.test/dataset [[:inner 0]]
metabase.test/discard-setting-changes [[:block 1]]
metabase.test/format-rows-by [[:default]]
metabase.test/let-url [[:block 1]]
metabase.test/mbql-query [[:inner 0]]
metabase.test/native-query [[:block 0]]
metabase.test/query [[:block 1]]
metabase.test/run-mbql-query [[:inner 0]]
metabase.test/test-driver [[:inner 0]]
metabase.test/test-drivers [[:inner 0]]
metabase.test/test-helpers-set-global-values! [[:block 0]]
metabase.test/throw-if-called! [[:block 1]]
metabase.test/with-actions [[:block 1]]
metabase.test/with-actions-disabled [[:block 0]]
metabase.test/with-actions-enabled [[:block 0]]
metabase.test/with-actions-test-data [[:inner 0]]
metabase.test/with-actions-test-data-and-actions-enabled [[:block 0]]
metabase.test/with-actions-test-data-tables [[:block 1]]
metabase.test/with-additional-premium-features [[:block 1]]
metabase.test/with-current-user [[:inner 0]]
metabase.test/with-driver [[:block 1]]
metabase.test/with-empty-db [[:inner 0]]
metabase.test/with-empty-h2-app-db [[:block 0]]
metabase.test/with-metadata-provider [[:inner 0]]
metabase.test/with-premium-features [[:block 1]]
metabase.test/with-temp [[:inner 0]]
metabase.test/with-temp-copy-of-db [[:block 0]]
metabase.test/with-temp-scheduler [[:block 0]]
metabase.test/with-temp-test-data [[:inner 0]]
metabase.test/with-temp-vals-in-db [[:block 3]]
metabase.test/with-test-user [[:inner 0]]
metabase.test/with-user-in-groups [[:inner 0]]
metabase.test.data/$ids [[:inner 0]]
metabase.test.data/dataset [[:inner 0]]
metabase.test.data/mbql-query [[:inner 0]]
metabase.test.data/native-query [[:block 0]]
metabase.test.data/query [[:block 1]]
metabase.test.data/run-mbql-query [[:inner 0]]
metabase.test.data/with-empty-h2-app-db [[:block 0]]
metabase.test.data/with-temp-copy-of-db [[:block 0]]
metabase.test.data.datasets/test-driver [[:inner 0]]
metabase.test.data.datasets/test-drivers [[:inner 0]]
metabase.test.data.one-off-dbs/with-blank-db [[:block 0]]
metabase.test.data.one-off-dbs/with-blueberries-db [[:block 0]]
metabase.test.data.users/with-test-user [[:inner 0]]
metabase.test.util/discard-setting-changes [[:block 1]]
metabase.test.util/let-url [[:block 1]]
metabase.test.util/throw-if-called! [[:block 1]]
metabase.test.util/with-temp-scheduler [[:block 0]]
metabase.test.util/with-temp-vals-in-db [[:block 3]]
metabase.test.util/with-user-in-groups [[:inner 0]]
metabase.test.util.public-settings/with-additional-premium-features [[:block 1]]
metabase.test.util.public-settings/with-premium-features [[:block 1]]
metabase.test.util.thread-local/test-helpers-set-global-values! [[:block 0]]
metabase.timeseries-query-processor-test.util/test-timeseries-drivers [[:block 0]]
metabase.timeseries-query-processor-test/run-mbql-query [[:inner 0]]
metabase.upload-test/with-upload-table! [[:inner 0]]
metabase.upload.types/with-currency [[:default]]
metabase.upload.types/with-parens [[:default]]
metabase.util/auto-retry [[:block 1]]
metabase.util/case-enum [[:block 1]]
metabase.util/ignore-exceptions [[:block 0]]
metabase.util/or-with [[:block 1]]
metabase.util/profile [[:block 1]]
metabase.util/prog1 [[:inner 0]]
metabase.util/varargs [[:inner 0]]
metabase.util/with-us-locale [[:block 0]]
metabase.util.async/cancelable-thread [[:block 0]]
metabase.util.date-2/with-time-zone-same-instant [[:default]]
metabase.util.encryption-test/with-secret-key [[:block 1]]
metabase.util.honey-sql-2/with-database-type-info [[:default]]
metabase.util.i18n/deferred-trs [[:default]]
metabase.util.i18n/deferred-tru [[:default]]
metabase.util.i18n/trs [[:default]]
metabase.util.i18n/trsn [[:default]]
metabase.util.i18n/tru [[:default]]
metabase.util.i18n/trun [[:default]]
metabase.util.jvm/auto-retry [[:block 1]]
metabase.util.jvm/varargs [[:block 1]]
metabase.util.jvm/with-us-locale [[:block 0]]
metabase.util.malli/disable-enforcement [[:block 0]]
metabase.util.malli/with [[:default]]
metabase.util.malli/with-api-error-message [[:default]]
metabase.util.namespaces/import-fns [[:default]]
metabuild-common.core/exit-when-finished-nonzero-on-exception [[:block 0]]
metabuild-common.core/step [[:block 1]]
metabuild-common.entrypoint/exit-when-finished-nonzero-on-exception [[:block 0]]
metabuild-common.steps/step [[:block 1]]
methodical.core/defmethod [[:inner 0]]
methodical.core/defmulti [[:inner 0]]
methodical.core/with-dispatcher [[:default]]
methodical.core/with-method-table [[:default]]
methodical.core/with-prefers [[:default]]
methodical.interface/with-dispatcher [[:default]]
methodical.interface/with-method-table [[:default]]
methodical.interface/with-prefers [[:default]]
methodical.macros/defmethod [[:inner 0]]
methodical.macros/defmulti [[:inner 0]]
saml20-clj.sp.response/validate-confirmation-datas [[:block 1]]
toucan2.connection/with-transaction [[:block 1]]
toucan2.core/build [[:block 0]]
toucan2.core/compile [[:block 0]]
toucan2.core/debug [[:block 0]]
toucan2.core/define-after-insert [[:inner 0]]
toucan2.core/define-after-select [[:inner 0]]
toucan2.core/define-after-update [[:inner 0]]
toucan2.core/define-before-delete [[:inner 0]]
toucan2.core/define-before-insert [[:inner 0]]
toucan2.core/define-before-select [[:inner 0]]
toucan2.core/define-default-fields [[:inner 0]]
toucan2.core/define-named-query [[:block 1]]
toucan2.core/deftransforms [[:block 1]]
toucan2.core/with-transaction [[:block 1]]
toucan2.tools.after/define-after-impl [[:default]]
toucan2.tools.after-insert/define-after-insert [[:inner 0]]
toucan2.tools.after-select/define-after-select [[:inner 0]]
toucan2.tools.after-update/define-after-update [[:inner 0]]
toucan2.tools.before-delete/define-before-delete [[:inner 0]]
toucan2.tools.before-insert/define-before-insert [[:inner 0]]
toucan2.tools.before-select/define-before-select [[:inner 0]]
toucan2.tools.compile/build [[:block 0]]
toucan2.tools.compile/compile [[:block 0]]
toucan2.tools.compile/resolved [[:block 0]]
toucan2.tools.debug/debug [[:block 0]]
toucan2.tools.default-fields/define-default-fields [[:inner 0]]
toucan2.tools.named-query/define-named-query [[:block 1]]
toucan2.tools.simple-out-transform/define-out-transform [[:inner 0]]
toucan2.tools.transformed/deftransforms [[:block 1]]
toucan2.tools.with-temp/with-temp [[:inner 0]]
toucan2.util/try-with-error-context [[:inner 0]]}}

View File

@@ -1,23 +0,0 @@
Brite
afterAll
ags
altho
cancelation
cancelations
datas
edn
eduction
hasTables
horizontalX
horizontalY
juxt
methodd
notIn
pass-thru
ser
serieses
sortings
trs
trsn
tru
trun

View File

@@ -1,3 +0,0 @@
[codespell]
skip = *.csv,*.edn,*.geojson,*.json,*.tsv,*.xml,docs/developers-guide/drivers/driver-tests.md,docs/developers-guide/e2e-tests.md,docs/questions/query-builder/expressions-list.md,docs/questions/visualizations/country-codes.md,e2e/test/scenarios/admin/i18n/content-translation/constants.ts,frontend/src/cljs,frontend/src/metabase/querying/expressions/tokenizer/lezer.js,frontend/src/metabase/static-viz/constants/char-sizes.ts,frontend/src/metabase/visualizations/lib/mapping_codes.ts,modules/drivers/mongo/src/metabase/driver/mongo/operators.clj
ignore-words = .codespellignore

View File

@@ -1,10 +1,11 @@
FROM mcr.microsoft.com/vscode/devcontainers/java:21
FROM mcr.microsoft.com/vscode/devcontainers/java:11
# Set up nodesource, install node, yarn, fontconfig for static viz, rlwrap for dev ergonomics
RUN ( curl -fsSL https://deb.nodesource.com/setup_22.x | bash ) \
RUN ( curl -fsSL https://deb.nodesource.com/setup_18.x | bash ) \
&& export DEBIAN_FRONTEND=noninteractive \
&& apt-key adv --refresh-keys --keyserver keyserver.ubuntu.com \
&& apt-get update && apt-get -y install --no-install-recommends nodejs yarn rlwrap fontconfig
# install Clojure
RUN curl https://download.clojure.org/install/linux-install-1.12.0.1488.sh | bash
RUN curl https://download.clojure.org/install/linux-install-1.11.1.1262.sh | bash

View File

@@ -1,4 +1,5 @@
.babel_cache
docs/*
target/*
.circleci
.cpcache

View File

@@ -2,4 +2,3 @@ frontend/src/cljs
frontend/src/cljs_release
e2e/support/cypress_sample_database.js
e2e/support/cypress_sample_instance_data.js
e2e/embedding-sdk-host-apps

View File

@@ -1,20 +1,17 @@
/* eslint-disable import/no-commonjs */
/* eslint-disable no-undef */
const path = require("path");
// `postcss-modules` lints css modules class names, but it currently crashes
// eslint on vscode. If you use webstorm or want to run the lint for the cli, you
// can use this flag to enable it. This is set to true in CI
const shouldLintCssModules =
process.env.LINT_CSS_MODULES === "true" || process.env.CI;
const plugins = ["react", "no-only-tests", "ttag", "i18next"];
const plugins = ["react", "no-only-tests"];
if (shouldLintCssModules) {
plugins.push("postcss-modules");
}
module.exports = {
ignorePatterns: ["!.storybook"],
rules: {
strict: [2, "never"],
"no-undef": 2,
@@ -49,7 +46,6 @@ module.exports = {
],
curly: [1, "all"],
eqeqeq: [1, "smart"],
"import/no-duplicates": ["warn", { considerQueryString: true }],
"import/no-default-export": 2,
"import/no-named-as-default": 0,
"import/no-commonjs": 1,
@@ -94,7 +90,10 @@ module.exports = {
"react/jsx-no-target-blank": 2,
"react/jsx-key": 2,
"react/forbid-component-props": [2, { forbid: ["sx"] }],
"react-hooks/exhaustive-deps": ["warn"],
"react-hooks/exhaustive-deps": [
"warn",
{ additionalHooks: "(useSafeAsyncFunction)" },
],
"prefer-const": [1, { destructuring: "all" }],
"no-restricted-globals": ["error", "close"],
"no-useless-escape": 0,
@@ -115,10 +114,11 @@ module.exports = {
"And",
"When",
"Then",
"describeWithSnowplow",
],
},
],
complexity: ["error", { max: 55 }],
complexity: ["error", { max: 54 }],
...(shouldLintCssModules
? {
"postcss-modules/no-undef-class": "error",
@@ -132,7 +132,7 @@ module.exports = {
},
env: {
browser: true,
es2020: true,
es6: true,
commonjs: true,
jest: true,
"jest/globals": true,
@@ -149,13 +149,11 @@ module.exports = {
"plugin:import/typescript",
"plugin:depend/recommended",
"plugin:storybook/recommended",
"plugin:i18next/recommended",
],
settings: {
"import/internal-regex": "^metabase/|^metabase-lib/",
"import/resolver": {
webpack: {
config: path.resolve(__dirname, "./rspack.main.config.js"),
typescript: true,
},
},
@@ -176,7 +174,6 @@ module.exports = {
{
files: ["*.js", "*.jsx", "*.ts", "*.tsx"],
rules: {
"jtag-missing-key": "error",
"no-unconditional-metabase-links-render": "error",
"no-color-literals": "error",
"no-literal-metabase-strings": "error",
@@ -198,19 +195,12 @@ module.exports = {
"*.unit.spec.*",
"frontend/src/metabase/admin/**/*",
"frontend/src/metabase/setup/**/*",
"enterprise/frontend/src/metabase-enterprise/whitelabel/**/*",
"enterprise/frontend/src/metabase-enterprise/embedding/**/*",
"frontend/lint/**/*",
"*.stories.*",
"**/.storybook/*",
"stories-data.*",
"e2e/**/*",
"**/tests/*",
"release/**/*",
"rspack.config.js",
"rspack.main.config.js",
"rspack.embedding-sdk-package.config.js",
"rspack.embedding-sdk-bundle.config.js",
],
rules: {
"no-color-literals": "off",
@@ -218,21 +208,6 @@ module.exports = {
"no-literal-metabase-strings": "off",
},
},
{
files: [
"*.unit.spec.*",
"frontend/lint/**/*",
"*.stories.*",
"stories-data.*",
"e2e/**/*",
"**/tests/*",
"release/**/*",
"rspack.main.config.js",
],
rules: {
"i18next/no-literal-string": "off",
},
},
{
extends: ["plugin:@typescript-eslint/recommended"],
files: ["*.ts", "*.tsx"],
@@ -280,13 +255,6 @@ module.exports = {
],
rules: {
"jest/valid-title": ["error", { ignoreTypeOfDescribeName: true }],
"jest/expect-expect": [
"error",
{
assertFunctionNames: ["expect*", "assert*"],
additionalTestBlockFunctions: [],
},
],
},
},
{
@@ -316,33 +284,9 @@ module.exports = {
selector: "Literal[value=/mb-base-color-/]",
message:
"You may not use base colors in the application, use semantic colors instead. (see colors.module.css)",
},
}
],
},
},
{
files: ["frontend/src/metabase/query_builder/**/*"],
rules: {
"import/no-cycle": "error",
},
},
{
files: ["docs/**/snippets/**/*.{ts,tsx,js,jsx}"],
rules: {
"@typescript-eslint/no-unused-vars": "off",
"@typescript-eslint/no-var-requires": "off",
"import/no-commonjs": "off",
"import/no-default-export": "off",
"import/order": "off",
"import/no-unresolved": "off",
"no-color-literals": "off",
},
},
{
files: ["frontend/build/**/*.js"],
rules: {
"import/no-commonjs": "off",
},
},
],
};

51
.github/CODEOWNERS vendored
View File

@@ -1,29 +1,24 @@
# Before changing this file, make sure you have read how we use it:
# https://www.notion.so/metabase/Pull-Request-Code-Review-Guide-374f05df889748b69d67af753f9bc9b8?pvs=4#36550c04ceab40b5a8b8bee6264b090c
enterprise/frontend/src/embedding-sdk-package @metabase/embedding
enterprise/frontend/src/embedding-sdk-bundle @metabase/embedding
enterprise/frontend/src/embedding-sdk-shared @metabase/embedding
enterprise/frontend/src/embedding-sdk @metabase/embedding
frontend/src/metabase/embedding-sdk @metabase/embedding
frontend/src/metabase/public/containers/PublicOrEmbeddedQuestion @metabase/embedding
frontend/src/metabase/public/containers/PublicOrEmbeddedDashboard @metabase/embedding
frontend/src/metabase/ui @metabase/core-frontend-ux-west
frontend/src/metabase/ui @metabase/core-frontend-admin-webapp
# snowplow/* @metabase/data
src/metabase/analytics/prometheus.clj @metabase/cloud-ops
.github @metabase/devexp
.clj-kondo @metabase/devexp
deps.edn @metabase/backend-developers
docs @metabase/tech-writers
docs/developers-guide # exclude from tech-writer ownership
#
# Backend modules <=> teams that own them
#
# modules/drivers @metabase/drivers
src/metabase/driver/sql_jdbc/actions.clj @metabase/core-ux-east
test/metabase/driver/sql_jdbc/actions_test.clj @metabase/core-ux-east
# enterprise/backend/src/metabase_enterprise/advanced_config @metabase/core-backend-admin-webapp
# enterprise/backend/test/metabase_enterprise/advanced_config @metabase/core-backend-admin-webapp
@@ -55,8 +50,8 @@ test/metabase/driver/sql_jdbc/actions_test.clj @metabase/core-ux-east
# enterprise/backend/src/metabase_enterprise/core @metabase/devexp
# enterprise/backend/test/metabase_enterprise/core @metabase/devexp
# enterprise/backend/src/metabase_enterprise/dashboard_subscription_filters @metabase/core-ux-east
# enterprise/backend/test/metabase_enterprise/dashboard_subscription_filters @metabase/core-ux-east
# enterprise/backend/src/metabase_enterprise/dashboard_subscription_filters @metabase/core-workflows
# enterprise/backend/test/metabase_enterprise/dashboard_subscription_filters @metabase/core-workflows
# enterprise/backend/src/metabase_enterprise/enhancements @metabase/core-backend-admin-webapp
# enterprise/backend/test/metabase_enterprise/enhancements @metabase/core-backend-admin-webapp
@@ -73,11 +68,11 @@ test/metabase/driver/sql_jdbc/actions_test.clj @metabase/core-ux-east
# enterprise/backend/src/metabase_enterprise/scim @metabase/core-backend-admin-webapp
# enterprise/backend/test/metabase_enterprise/scim @metabase/core-backend-admin-webapp
# enterprise/backend/src/metabase_enterprise/search ...
# enterprise/backend/test/metabase_enterprise/search ...
# enterprise/backend/src/metabase_enterprise/search @metabase/core-workflows
# enterprise/backend/test/metabase_enterprise/search @metabase/core-workflows
# enterprise/backend/src/metabase_enterprise/serialization @metabase/core-ux-east
# enterprise/backend/test/metabase_enterprise/serialization @metabase/core-ux-east
# enterprise/backend/src/metabase_enterprise/serialization @metabase/core-workflows
# enterprise/backend/test/metabase_enterprise/serialization @metabase/core-workflows
# enterprise/backend/src/metabase_enterprise/snippet_collections @metabase/core-backend-admin-webapp
# enterprise/backend/test/metabase_enterprise/snippet_collections @metabase/core-backend-admin-webapp
@@ -97,8 +92,8 @@ test/metabase/driver/sql_jdbc/actions_test.clj @metabase/core-ux-east
# enterprise/backend/src/metabase_enterprise/upload_management @metabase/core-backend-admin-webapp
# enterprise/backend/test/metabase_enterprise/upload_management @metabase/core-backend-admin-webapp
src/metabase/actions @metabase/core-ux-east
test/metabase/actions @metabase/core-ux-east
# src/metabase/actions @metabase/core-backend-admin-webapp
# test/metabase/actions @metabase/core-backend-admin-webapp
# src/metabase/activity_feed @metabase/core-backend-admin-webapp
# test/metabase/activity_feed @metabase/core-backend-admin-webapp
@@ -124,11 +119,11 @@ test/metabase/actions @metabase/core-ux-east
# src/metabase/cmd @metabase/devexp
# test/metabase/cmd @metabase/devexp
# src/metabase/channel @metabase/core-ux-east
# test/metabase/channel @metabase/core-ux-east
# src/metabase/channel @metabase/core-workflows
# test/metabase/channel @metabase/core-workflows
# src/metabase/cloud_migration ...
# test/metabase/cloud_migration ...
# src/metabase/cloud_migration @metabase/core-workflows
# test/metabase/cloud_migration @metabase/core-workflows
# src/metabase/cmd @metabase/devexp
# test/metabase/cmd @metabase/devexp
@@ -160,8 +155,8 @@ test/metabase/actions @metabase/core-ux-east
# src/metabase/indexed_entities @metabase/core-backend-querying
# test/metabase/indexed_entities @metabase/core-backend-querying
# src/metabase/integrations ...
# test/metabase/integrations ...
# src/metabase/integrations @metabase/core-workflows
# test/metabase/integrations @metabase/core-workflows
# src/metabase/legacy_mbql @metabase/core-backend-querying
# test/metabase/legacy_mbql @metabase/core-backend-querying
@@ -184,8 +179,8 @@ test/metabase/actions @metabase/core-ux-east
# src/metabase/moderation @metabase/core-backend-admin-webapp
# test/metabase/moderation @metabase/core-backend-admin-webapp
# src/metabase/notification @metabase/core-ux-east
# test/metabase/notification @metabase/core-ux-east
# src/metabase/notification @metabase/core-workflows
# test/metabase/notification @metabase/core-workflows
# src/metabase/permissions @metabase/core-backend-admin-webapp
# test/metabase/permissions @metabase/core-backend-admin-webapp
@@ -202,8 +197,8 @@ test/metabase/actions @metabase/core-ux-east
# src/metabase/public_sharing @metabase/core-backend-admin-webapp
# test/metabase/public_sharing @metabase/core-backend-admin-webapp
# src/metabase/pulse @metabase/core-ux-east
# test/metabase/pulse @metabase/core-ux-east
# src/metabase/pulse @metabase/core-workflows
# test/metabase/pulse @metabase/core-workflows
# src/metabase/query_analysis @metabase/core-backend-querying
# test/metabase/query_analysis @metabase/core-backend-querying
@@ -220,8 +215,8 @@ test/metabase/actions @metabase/core-ux-east
# src/metabase/sample_data @metabase/core-backend-admin-webapp
# test/metabase/sample_data @metabase/core-backend-admin-webapp
# src/metabase/search ...
# test/metabase/search ...
# src/metabase/search @metabase/core-workflows
# test/metabase/search @metabase/core-workflows
# src/metabase/segments @metabase/core-backend-querying
# test/metabase/segments @metabase/core-backend-querying

View File

@@ -63,8 +63,7 @@ body:
attributes:
label: Information about your Metabase installation
description: |
For versions >= 56, copy and paste the Diagnostic info from `Admin -> Tools -> Help`.
For versions < 56, copy and paste the Diagnostic info from `Admin -> Troubleshooting`.
Copy and paste the JSON from `Admin -> Troubleshooting`.
_This will be automatically formatted into code, so no need for backticks._
placeholder: |
{"browser-info": {

View File

@@ -3,7 +3,7 @@
> **Warning**
>
> If that is your first contribution to Metabase, please sign the [Contributor License Agreement](https://docs.google.com/a/metabase.com/forms/d/1oV38o7b9ONFSwuzwmERRMi9SYrhYeOrkbmNaq9pOJ_E/viewform). Also, if you're attempting to fix a translation issue, please submit your changes to our [Crowdin project](https://crowdin.com/project/metabase-i18n) instead of opening a PR.
> If that is your first contribution to Metabase, please sign the [Contributor License Agreement](https://docs.google.com/a/metabase.com/forms/d/1oV38o7b9ONFSwuzwmERRMi9SYrhYeOrkbmNaq9pOJ_E/viewform) (unless it's a tiny documentation change). Also, if you're attempting to fix a translation issue, please submit your changes to our [Crowdin project](https://crowdin.com/project/metabase-i18n) instead of opening a PR.
Closes https://github.com/metabase/metabase/issues/[issue_number]

View File

@@ -1,5 +1,4 @@
name: await-port
description: Wait for localhost port to be ready
name: Wait for localhost port to be ready
inputs:
port:

View File

@@ -0,0 +1,116 @@
name: Build ee-extra
description: Build Metabase Enterprise Edition with extra features
inputs:
java-version:
required: true
default: 21
iam-role:
description: "The IAM role to assume"
required: true
runs:
using: composite
steps:
- name: Prepare Java
uses: actions/setup-java@v3
with:
java-version: ${{ inputs.java-version }}
distribution: 'temurin'
- name: Reveal its original version.properties
run: jar xf ./bin/docker/metabase.jar version.properties && cat version.properties
shell: bash
- name: get major version
id: major_version
run: echo "major_version=$(cat version.properties | grep 'tag=' | cut -d'.' -f2)" >> $GITHUB_OUTPUT
shell: bash
- name: get drivers registry
run: |
curl -OL https://raw.githubusercontent.com/metabase/metabase-registry/main/registry.yaml
shell: bash
- uses: actions/setup-node@v4
with:
node-version: lts/Jod # 22.x.x
- run: rm package.json && npm install yaml
shell: bash
- name: parse drivers registry
uses: actions/github-script@v7
id: drivers_registry
with:
script: | # js
const { parse } = require('yaml');
const fs = require('fs');
const registry = parse(fs.readFileSync('registry.yaml', 'utf8'));
const drivers = registry.drivers;
const majorVersion = '${{ steps.major_version.outputs.major_version }}';
const getDriverUrl = (driverName) => {
const driver = drivers.find(driver => driver.name === driverName);
if (!driver) {
throw new Error(`Driver ${driverName} not found in the registry`);
}
const driverUrl = driver.versions[majorVersion] ?? driver.versions.default;
if (!driverUrl) {
throw new Error(`Driver ${driverName} version ${majorVersion} not found in the registry`);
}
return driverUrl;
};
const extraDrivers = {
firebolt: getDriverUrl('firebolt'),
starburst: getDriverUrl('starburst'),
clickhouse: getDriverUrl('clickhouse'),
materialize: getDriverUrl('materialize'),
};
console.log(extraDrivers);
return extraDrivers;
- name: Download Firebolt driver
run: |
curl -OL ${{ fromJson(steps.drivers_registry.outputs.result).firebolt }}
working-directory: modules
shell: bash
- name: Download Starburst driver
run: |
curl -OL ${{ fromJson(steps.drivers_registry.outputs.result).starburst }}
shell: bash
working-directory: modules
- name: Download Clickhouse driver
run: |
curl -OL ${{ fromJson(steps.drivers_registry.outputs.result).clickhouse }}
shell: bash
working-directory: modules
- name: Download Materialize driver
run: |
curl -OL ${{ fromJson(steps.drivers_registry.outputs.result).materialize }}
shell: bash
working-directory: modules
- name: Bundle additional drivers into the Uberjar
run: jar uf bin/docker/metabase.jar modules/*.jar
shell: bash
- name: Launch uberjar
run: >-
java --add-opens java.base/java.nio=ALL-UNNAMED -jar ./bin/docker/metabase.jar &
shell: bash
- name: Wait for Metabase to start
run: while ! curl 'http://localhost:3000/api/health' | grep '{"status":"ok"}'; do sleep 1; done
shell: bash
- name: Kill the java process
run: pkill -f metabase.jar
shell: bash
- name: configure aws credentials
if: ${{ github.ref_name == 'master' }}
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: ${{ inputs.iam-role }}
role-session-name: GitHub_to_AWS_via_FederatedOIDC
aws-region: us-east-1
- name: Login to Amazon ECR
if: ${{ github.ref_name == 'master' }}
uses: aws-actions/amazon-ecr-login@v2

View File

@@ -1,36 +0,0 @@
name: Cleanup Python Runner
description: Cleanup python-runner services and upload logs as artifacts
inputs:
upload-logs:
description: 'Whether to upload logs (typically on failure)'
required: false
default: 'false'
logs-artifact-suffix:
description: 'The unique name of the .zip artifact containing the logs'
required: false
default: ''
runs:
using: composite
steps:
- name: Cleanup python-runner services
if: always()
run: |
echo "Stopping python-runner services..."
if [ -d "python-runner" ]; then
cd python-runner
make stop || echo "WARNING: Failed to stop python runner"
fi
shell: bash
- name: Upload python-runner logs
if: ${{inputs.upload-logs == 'true' && inputs.logs-artifact-suffix != ''}}
uses: actions/upload-artifact@v4
with:
name: python-runner-logs-${{inputs.logs-artifact-suffix}}
path: |
python-runner.out.log
python-runner.err.log
retention-days: 1
if-no-files-found: warn

View File

@@ -23,7 +23,7 @@ runs:
GITHUB_TOKEN: ${{ inputs.github-token }}
COMMIT: ${{ inputs.backport-commit }}
shell: bash
run: | # bash
run: |
git config --global user.email "metabase-bot@metabase.com"
git config --global user.name "Metabase bot"
@@ -35,8 +35,6 @@ runs:
git checkout -b ${BACKPORT_BRANCH}
git cherry-pick ${COMMIT} || true
CONFLICT_LIST=$(git ls-files -u)
CONFLICTS=$(git ls-files -u | wc -l)
if [ "$CONFLICTS" -gt 0 ]; then
git cherry-pick --abort
@@ -48,7 +46,7 @@ runs:
echo "git reset HEAD~1" > ./backport.sh
echo "rm ./backport.sh" >> ./backport.sh
echo "git cherry-pick ${COMMIT}" >> ./backport.sh
echo "echo 'Resolve conflicts and force push this branch.\n\nTo backport translations run: bin/i18n/merge-translations <release-branch>'" >> ./backport.sh
echo "echo 'Resolve conflicts and force push this branch'" >> ./backport.sh
chmod +x ./backport.sh
git add ./backport.sh
git commit -m "Add backport resolution script"
@@ -57,14 +55,7 @@ runs:
#${ORIGINAL_PULL_REQUEST_NUMBER}
> [!IMPORTANT]
> Manual conflict resolution is required.
> Checkout the branch and run \`./backport.sh\` script. Force push your changes after cherry-picking. Check the box below when done.
Conflicts:
\`\`\`shell
${CONFLICT_LIST}
\`\`\`
- [ ] Conflicts resolved
Checkout the branch and run \`./backport.sh\` script. Force push your changes after cherry-picking.
END
)
else
@@ -74,7 +65,7 @@ runs:
git push -u origin ${BACKPORT_BRANCH}
BACKPORT_PR_URL=$(gh pr create --base "${TARGET_BRANCH}" --head "${BACKPORT_BRANCH}" --label "was-backported" --assignee "${{ github.event.pull_request.user.login }}" --title "🤖 backported \"${ORIGINAL_TITLE}\"" --body "${PR_BODY}")
BACKPORT_PR_URL=$(gh pr create --base "${TARGET_BRANCH}" --head "${BACKPORT_BRANCH}" --label "was-backported" --assignee "${GITHUB_ACTOR}" --title "🤖 backported \"${ORIGINAL_TITLE}\"" --body "${PR_BODY}")
BACKPORT_PR_NUMBER=${BACKPORT_PR_URL##*/}
echo "backport_pr_number=$BACKPORT_PR_NUMBER" >> $GITHUB_OUTPUT

View File

@@ -1,15 +1,12 @@
name: Cross-branch migration test
inputs:
base-ref: ## Base branch PR came from
required: true
head-ref: ## Source branch of the pull request
base-ref:
required: true
runs:
using: "composite"
steps:
## First we deploy the base branch
- name: Check out base branch
uses: actions/checkout@v4
with:
@@ -17,6 +14,8 @@ runs:
- name: Prepare backend environment
uses: ./.github/actions/prepare-backend
with:
m2-cache-key: cross-branch-migrations
- name: Run backend on base branch
run: |
@@ -40,11 +39,10 @@ runs:
run: kill $(cat metabase_pid)
shell: bash
## Then we deploy the PR branch to ensure it migrates up successfully
- name: Check out pull request branch
uses: actions/checkout@v4
with:
ref: ${{ inputs.head-ref }}
ref: ${{ github.head_ref }}
- name: Run backend on pull request branch
run: |
@@ -67,38 +65,3 @@ runs:
- name: Stop Metabase on pull request branch
run: kill $(cat metabase_pid)
shell: bash
## Testing that originally deploying this PR allows you to roll back to a previous release
## Reset the checksums so when we go back to the base branch, it's as if the PR branch was what was originally deployed
- name: Reset the checksums as if the PR branch was what was originally deployed
run: |
clojure -M:dev:ee:migrate reset-checksums
shell: bash
- name: Rollback the last deployment
run: |
clojure -M:dev:ee:migrate rollback last-deployment
shell: bash
- name: Re-check out base branch
uses: actions/checkout@v4
with:
ref: ${{ inputs.base-ref }}
- name: Re-run backend on base branch
run: |
clojure -M:run:ee > metabase.log 2>&1 & echo $! > metabase_pid
tail -F metabase.log &
shell: bash
- name: Wait for Metabase to start
run: |
while ! curl -s 'http://localhost:3000/api/health' | grep '{"status":"ok"}' > /dev/null; do
if ! kill -0 $(cat metabase_pid) 2>/dev/null; then
cat metabase.log
echo "Metabase process has exited."
exit 1
fi
sleep 1
done
shell: bash

View File

@@ -8,7 +8,7 @@ inputs:
description: DockerHub password/token
required: true
maildev:
description: Maildev containers
description: Maildev container
required: true
default: false
openldap:
@@ -31,10 +31,6 @@ inputs:
description: Webhook tester container
required: false
default: false
snowplow:
description: Snowplow micro container
required: false
default: false
runs:
using: "composite"
@@ -46,68 +42,56 @@ runs:
password: ${{ inputs.password }}
- name: Start Containers
shell: bash
run: | # bash
run: |
Y='\033[33m'
G='\033[1;32m'
cd e2e/test/scenarios/
docker compose up \
$([ ${{ inputs.postgres }} = true ] && echo "postgres-sample") \
$([ ${{ inputs.mysql }} = true ] && echo "mysql-sample") \
$([ ${{ inputs.mongo }} = true ] && echo "mongo-sample") \
$([ ${{ inputs.maildev }} = true ] && echo "maildev") \
$([ ${{ inputs.maildev }} = true ] && echo "maildev-ssl") \
$([ ${{ inputs.openldap }} = true ] && echo "ldap") \
$([ ${{ inputs.webhook }} = true ] && echo "webhook-tester") \
$([ ${{ inputs.snowplow }} = true ] && echo "snowplow-micro") \
-d
if ${{ inputs.maildev }}; then
echo -e "${Y}checking maildev container..." &&
echo -e "${Y}Starting maildev container..." &&
docker run -d -p 1080:1080 -p 1025:1025 maildev/maildev:2.0.5 &&
while ! nc -z localhost 1080; do sleep 1; done &&
while ! nc -z localhost 1025; do sleep 1; done &&
echo -e "${G}Maildev is up and running!"
echo -e "${Y}checking maildev-ssl container..." &&
while ! nc -z localhost 1081; do sleep 1; done &&
while ! nc -z localhost 465; do sleep 1; done &&
echo -e "${G}Maildev-ssl is up and running!"
fi
if ${{ inputs.openldap }}; then
echo -e "${Y}checking openldap container..." &&
echo -e "${Y}Starting openldap container..." &&
docker run -d -p 389:389 \
--env LDAP_ADMIN_PASSWORD=adminpass \
--env LDAP_USERS=user01@example.org,user02@example.org \
--env LDAP_PASSWORDS=123456,123465 \
--env LDAP_ROOT=dc=example,dc=org \
--env LDAP_PORT_NUMBER=389 \
bitnami/openldap:2.6.4 &&
while ! nc -z localhost 389; do sleep 1; done &&
echo -e "${G}Openldap is up and running!"
fi
if ${{ inputs.postgres }}; then
echo -e "${Y}checking postgres container..." &&
echo -e "${Y}Starting postgres container..." &&
docker run -d -p 5404:5432 -e TZ='US/Pacific' metabase/qa-databases:postgres-sample-12 &&
while ! nc -z localhost 5404; do sleep 1; done &&
echo -e "${G}Postgres is up and running!"
fi
if ${{ inputs.mysql }}; then
echo -e "${Y}checking mysql container..." &&
echo -e "${Y}Starting mysql container..." &&
docker run -d -p 3304:3306 -e TZ='US/Pacific' metabase/qa-databases:mysql-sample-8 &&
while ! nc -z localhost 3304; do sleep 1; done &&
echo -e "${G}MySQL is up and running!"
fi
if ${{ inputs.mongo }}; then
echo -e "${Y}checking mongo container..." &&
echo -e "${Y}Starting mongo container..." &&
docker run -d -p 27004:27017 metabase/qa-databases:mongo-sample-4.4 &&
while ! nc -z localhost 27004; do sleep 1; done &&
echo -e "${G}Mongo is up and running!"
fi
if ${{ inputs.webhook }}; then
echo -e "${Y}checking webhook test container..." &&
echo -e "${Y}Starting webhook test container..." &&
docker run -d -p 9080:8080/tcp tarampampam/webhook-tester:1.1.0 serve --create-session 00000000-0000-0000-0000-000000000000 &&
while ! nc -z localhost 9080; do sleep 1; done &&
echo -e "${G}Webhook tester is up and running!"
fi
if ${{ inputs.snowplow }}; then
echo -e "${Y}checking snowplow micro container..." &&
while ! nc -z localhost 9090; do sleep 1; done &&
echo -e "${G}Snowplow micro is up and running!"
fi
shell: bash

View File

@@ -5,14 +5,6 @@ inputs:
name:
description: The name of the artifact to fetch
required: true
output-name:
description: Optional custom name for the output jar file (defaults to metabase.jar)
required: false
default: "metabase.jar"
extract-path:
description: Directory path where the jar should be extracted (defaults to current directory)
required: false
default: "."
runs:
using: "composite"
@@ -33,7 +25,7 @@ runs:
});
if (!artifacts.data?.artifacts?.[0]?.id) {
throw new Error(`No artifacts found for ${{ inputs.name }}`);
throw new Error(`No artifacts found for ${{ inputs.commit }}`);
}
const artifact_id = artifacts.data.artifacts[0].id;
@@ -47,59 +39,25 @@ runs:
fs.writeFileSync(`${process.env.GITHUB_WORKSPACE}/mb.zip`, Buffer.from(download.data));
- name: Extract and rename JAR
- name: unzip uberjar artifact
shell: bash
run: unzip mb.zip
- name: Move the Uberjar
shell: bash
run: | # bash
OUTPUT_NAME="${{ inputs.output-name }}"
EXTRACT_PATH="${{ inputs.extract-path }}"
# Create a temporary directory for extraction to avoid conflicts
TEMP_DIR=$(mktemp -d)
# Extract to temporary directory
unzip mb.zip -d "$TEMP_DIR"
# Create target directory
mkdir -p "$EXTRACT_PATH"
# Find the jar and move all files from its directory
if [ -e "$TEMP_DIR/metabase.jar" ]; then
# Move all files to extract path
mv "$TEMP_DIR"/* "$EXTRACT_PATH/" 2>/dev/null || true
# Rename the jar to custom name if different
if [ "$OUTPUT_NAME" != "metabase.jar" ]; then
mv "$EXTRACT_PATH/metabase.jar" "$EXTRACT_PATH/$OUTPUT_NAME"
fi
elif [ -e "$TEMP_DIR/target/uberjar/metabase.jar" ]; then
# Move all files from uberjar directory to extract path
mv "$TEMP_DIR/target/uberjar"/* "$EXTRACT_PATH/" 2>/dev/null || true
# Rename the jar to custom name if different
if [ "$OUTPUT_NAME" != "metabase.jar" ]; then
mv "$EXTRACT_PATH/metabase.jar" "$EXTRACT_PATH/$OUTPUT_NAME"
fi
if [ -e "metabase.jar" ]; then
echo "metabase.jar is already in the right place"
elif [ -e "target/uberjar/metabase.jar" ]; then
echo "moving metabase.jar to current directory"
mv target/uberjar/metabase.jar metabase.jar
else
echo "Could not find metabase.jar in extracted files"
echo "Could not find the metabase.jar file"
exit 1
fi
# Clean up temporary directory
rm -rf "$TEMP_DIR"
# Set the JAR_PATH environment variable
if [ "$EXTRACT_PATH" == "." ]; then
echo "JAR_PATH=$OUTPUT_NAME" >> $GITHUB_ENV
else
echo "JAR_PATH=$EXTRACT_PATH/$OUTPUT_NAME" >> $GITHUB_ENV
fi
- name: Verify that this is a valid JAR file
shell: bash
run: |
echo "Verifying JAR at: $JAR_PATH"
file --mime-type "$JAR_PATH"
run: file --mime-type ./metabase.jar
- name: Reveal its version.properties
shell: bash
run: |
echo "Extracting version.properties from: $JAR_PATH"
jar xf "$JAR_PATH" version.properties && cat version.properties
run: jar xf metabase.jar version.properties && cat version.properties

View File

@@ -1,32 +0,0 @@
name: Get cache keys
description: |
Get deterministic cache keys for Maven and Node.js caches based on the centrally controlled cache prefix,
and the suffix based on the runner OS and the current week of the year.
outputs:
m2-cache-key:
description: "The cache key for the Maven cache."
value: ${{ steps.keys-factory.outputs.m2-cache-key }}
node-cache-key:
description: "The cache key for the Node.js cache."
value: ${{ steps.keys-factory.outputs.node-cache-key }}
eslint-cache-key:
description: "The cache key for Eslint cache"
value: ${{ steps.keys-factory.outputs.eslint-cache-key }}
runs:
using: "composite"
steps:
- name: Generate and output cache keys
id: keys-factory
run: |
M2_CACHE_PREFIX='M2'
NODE_CACHE_PREFIX='node-modules'
ESLINT_CACHE_PREFIX='eslint'
WEEK_OF_YEAR=$(/bin/date "+%G-%V")
CACHE_SUFFIX="${{ runner.os }}-$WEEK_OF_YEAR"
echo "m2-cache-key=$M2_CACHE_PREFIX-$CACHE_SUFFIX" >> $GITHUB_OUTPUT
echo "node-cache-key=$NODE_CACHE_PREFIX-$CACHE_SUFFIX" >> $GITHUB_OUTPUT
echo "eslint-cache-key=$ESLINT_CACHE_PREFIX-$CACHE_SUFFIX" >> $GITHUB_OUTPUT
shell: bash

View File

@@ -1,74 +0,0 @@
name: Get Docker Info
description: Determines Docker repository and tag based on branch and edition
inputs:
branch:
description: Branch name (defaults to current branch)
required: false
default: ${{ github.head_ref || github.ref_name }}
edition:
description: Edition (ee or oss)
required: false
default: ee
owner:
description: Repository owner (defaults to current repo owner)
required: false
default: ${{ github.repository_owner }}
outputs:
repo:
description: Docker repository name
value: ${{ steps.docker-info.outputs.repo }}
tag:
description: Docker tag
value: ${{ steps.docker-info.outputs.tag }}
full_image:
description: Full Docker image name (repo:tag)
value: ${{ steps.docker-info.outputs.full_image }}
editions:
description: JSON array of editions to build
value: ${{ steps.docker-info.outputs.editions }}
runs:
using: "composite"
steps:
- name: Determine Docker info
id: docker-info
uses: actions/github-script@v7
with:
script: | # js
const branch = '${{ inputs.branch }}';
const edition = '${{ inputs.edition }}';
const owner = '${{ inputs.owner }}';
console.log(`Determining Docker info for branch: ${branch}, edition: ${edition}`);
let repo, tag, editions;
if (branch === 'master') {
// Master builds both editions with specific repos
editions = ['ee', 'oss'];
tag = 'latest';
if (edition === 'ee') {
repo = `${owner}/metabase-enterprise-head`;
} else {
repo = `${owner}/metabase-head`;
}
} else {
// All other branches only build EE to metabase-dev
editions = ['ee'];
tag = branch.replaceAll('/', '-'); // slashes make docker sad
repo = `${owner}/metabase-dev`;
}
const fullImage = `${repo}:${tag}`;
// Set outputs
core.setOutput('repo', repo);
core.setOutput('tag', tag);
core.setOutput('full_image', fullImage);
core.setOutput('editions', JSON.stringify(editions));
console.log(`Docker image: ${fullImage}`);
console.log(`Editions: ${JSON.stringify(editions)}`);

View File

@@ -1,13 +1,5 @@
name: Notify Pull Request
inputs:
remove-previous-comment:
required: false
type: boolean
default: false
add-new-comment:
required: false
type: boolean
default: true
message:
required: true
include-log:
@@ -17,44 +9,7 @@ inputs:
runs:
using: "composite"
steps:
- name: Remove previous pull-request comment
uses: actions/github-script@v7
if: ${{ inputs.remove-previous-comment == 'true' && always() && github.event_name == 'pull_request' }}
with:
script: |
const pr = context.payload.pull_request ?? context.payload.issue;
if (!pr) {
console.error("This action can only be run on pull requests");
return;
}
const { owner, repo } = context.repo;
const { data: comments } = await github.rest.issues.listComments({
owner,
repo,
issue_number: pr.number,
}).catch(console.error);
if (!comments) {
return;
}
const message = "${{ inputs.message }}";
const comment = comments.find((comment) => comment.body.includes(message));
if (comment) {
await github.rest.issues.deleteComment({
owner,
repo,
comment_id: comment.id,
});
}
- name: Notify pull-request
uses: actions/github-script@v7
if: ${{ inputs.add-new-comment == 'true' && always() && github.event_name == 'pull_request' }}
- uses: actions/github-script@v7
id: notify-pull-request
with:
result-encoding: string
@@ -66,7 +21,7 @@ runs:
return;
}
const message = "${{ inputs.message }}";
const message = "${{ inputs.message }}"
const shouldIncludeLog = ${{ inputs.include-log }}.toString() === "true";
const author = context.payload.sender.login;

View File

@@ -1,7 +1,4 @@
name: Prepare back-end environment
description: |
Prepare the backend. Installs jdk, Clojure, and fetches the cache for m2.
The CI config fetches `ci-test-config.json` from master and lets us ignore tests at the var level if they are flaking.
inputs:
java-version:
required: true
@@ -9,6 +6,10 @@ inputs:
clojure-version:
required: true
default: '1.12.0.1488'
m2-cache-key:
description: 'Key to cache M2 packages from Maven Central'
required: true
default: 'm2'
runs:
using: "composite"
@@ -20,24 +21,20 @@ runs:
distribution: 'temurin'
- name: Install Clojure CLI
shell: bash
continue-on-error: true
run: |
curl -O https://download.clojure.org/install/linux-install-${{ inputs.clojure-version }}.sh &&
sudo bash ./linux-install-${{ inputs.clojure-version }}.sh
- name: Fetch CI config
- name: Check to see if dependencies should be cached
if: ${{ contains(github.event.head_commit.message, '[ci nocache]') }}
run: echo "Commit message includes [ci nocache]; dependencies will NOT be cached"
shell: bash
run: |
curl -o ci-test-config.json https://raw.githubusercontent.com/metabase/metabase/refs/heads/master/ci-test-config.json
- name: Get M2 cache key
uses: ./.github/actions/get-cache-keys
id: get-cache-keys
- name: Restore M2 cache
uses: actions/cache/restore@v4
- name: Get M2 cache
uses: actions/cache@v4
if: ${{ !contains(github.event.head_commit.message, '[ci nocache]') }}
with:
path: |
~/.m2
~/.gitlibs
key: ${{ steps.get-cache-keys.outputs.m2-cache-key }}
key: ${{ runner.os }}-${{ inputs.m2-cache-key }}-${{ hashFiles('**/deps.edn') }}
restore-keys: |
${{ runner.os }}-${{ inputs.m2-cache-key }}-

View File

@@ -1,12 +1,6 @@
name: Prepare cypress environment
description: Cypress preparation steps
inputs:
working-directory:
description: "Optional working directory for yarn commands"
required: false
default: "."
outputs:
chrome-path:
description: Path to the custom Chrome executable
@@ -22,16 +16,16 @@ runs:
chrome-version: 1097615
id: setup-chrome
- name: Get current week of year
id: get-date
run: |
echo "week-of-year=$(/bin/date "+%G-%V")" >> $GITHUB_OUTPUT
- name: Check to see if dependencies should be cached
if: ${{ contains(github.event.head_commit.message, '[ci nocache]') }}
run: echo "Commit message includes [ci nocache]; dependencies will NOT be cached"
shell: bash
- name: Get Cypress cache
uses: actions/cache@v4
if: ${{ !contains(github.event.head_commit.message, '[ci nocache]') }}
with:
path: ~/.cache/Cypress
key: ${{ runner.os }}-Cypress-${{ steps.get-date.outputs.week-of-year }}
key: ${{ runner.os }}-Cypress-${{ hashFiles('**/yarn.lock') }}
- name: Ensure that Cypress executable is ready
run: |
yarn cypress install
@@ -39,4 +33,3 @@ runs:
yarn cypress cache list
yarn cypress verify
shell: bash
working-directory: ${{ inputs.working-directory }}

View File

@@ -3,10 +3,6 @@ inputs:
node-version:
description: "The version of Node.js to use, overriding .nvmrc"
required: false
apply-patches:
description: "Whether to patch frontend dependencies with patch-package."
required: true
default: 'true'
runs:
using: "composite"
steps:
@@ -16,32 +12,26 @@ runs:
node-version: ${{ inputs.node-version }} # this overrides, if present
node-version-file: ".nvmrc"
- name: Get cache keys
uses: ./.github/actions/get-cache-keys
id: get-cache-keys
- name: Restore M2 cache
uses: actions/cache/restore@v4
- name: Check to see if dependencies should be cached
if: ${{ contains(github.event.head_commit.message, '[ci nocache]') }}
run: |
echo "Commit message includes [ci nocache]; dependencies will NOT be cached"
yarn cache clean
shell: bash
- name: Get M2 cache
uses: actions/cache@v4
if: ${{ !contains(github.event.head_commit.message, '[ci nocache]') }}
with:
path: |
~/.m2
~/.gitlibs
key: ${{ steps.get-cache-keys.outputs.m2-cache-key }}
path: ~/.m2
key: ${{ runner.os }}-cljs-${{ hashFiles('**/shadow-cljs.edn') }}
- name: Restore node_modules cache
uses: actions/cache/restore@v4
- name: Get node_modules cache
uses: actions/cache@v4
id: node-modules-cache
if: ${{ !contains(github.event.head_commit.message, '[ci nocache]') }}
with:
path: node_modules
key: ${{ steps.get-cache-keys.outputs.node-cache-key }}
key: ${{ runner.os }}-node-modules-${{ hashFiles('**/yarn.lock', '**/patches/*.patch') }}
# postinstall lifecycle hook can be dangerous in CI so we only run it locally (see package.json)
# If a job wants to patch packages, it does so explicitly in the CI. This setting is opt-out.
# For example, we never want to cache node_modules with patches already applied so cache-generator.yml skips that step.
- name: Clean install
run: yarn install --frozen-lockfile --prefer-offline
shell: bash
- name: Patch frontend dependencies
if: inputs.apply-patches == 'true'
run: yarn patch-package
- run: yarn install --frozen-lockfile --prefer-offline
shell: bash

View File

@@ -0,0 +1,33 @@
name: Run Embedding SDK Storybook
runs:
using: "composite"
steps:
- name: Prepare frontend environment
uses: ./.github/actions/prepare-frontend
- name: Prepare back-end environment
uses: ./.github/actions/prepare-backend
with:
m2-cache-key: "cljs"
- name: Compile CLJS
run: yarn build-pure:cljs
shell: bash
- name: Build Storybook
run: yarn build-storybook-embedding-sdk
shell: bash
- name: Serve Storybook
run: yarn http-server storybook-static -p 6006 &
shell: bash
- name: Wait for Storybook to become available
run: |
echo "Waiting for Storybook to be served..."
while ! curl --silent --fail http://localhost:6006; do
sleep 5
echo "Retrying..."
done
echo "Storybook is being served!"
shell: bash

View File

@@ -0,0 +1,13 @@
name: Run Snowplow micro
runs:
using: "composite"
steps:
- name: Download Snowplow micro JAR
run: curl -L -o snowplow-micro.jar https://github.com/snowplow-incubator/snowplow-micro/releases/download/micro-1.2.1/snowplow-micro-1.2.1.jar
shell: bash
- name: Launch Snowplow micro
run: java -cp snowplow-micro.jar:snowplow com.snowplowanalytics.snowplow.micro.Main --collector-config snowplow/micro.conf --iglu snowplow/iglu.json &
shell: bash
- name: Wait until Snowplow micro is ready
run: while ! curl -s 'http://localhost:9090' | grep '404'; do sleep 1; done
shell: bash

View File

@@ -1,14 +0,0 @@
name: Setup Python Runner
description: Setup python-runner for tests
inputs:
metabase-automation-token:
description: Token for accessing the python-runner repository
required: true
runs:
using: composite
steps:
- name: Setup python-runner
run: ./.github/scripts/setup-python-runner.sh
shell: bash
env:
METABASE_AUTOMATION_USER_TOKEN: ${{ inputs.metabase-automation-token }}

View File

@@ -1,5 +1,4 @@
name: Test database driver
description: Test database driver
inputs:
junit-name:
required: true
@@ -15,6 +14,8 @@ runs:
steps:
- name: Prepare back-end environment
uses: ./.github/actions/prepare-backend
with:
m2-cache-key: driver
- name: Prepare front-end environment
uses: ./.github/actions/prepare-frontend
if: ${{ inputs.build-static-viz == 'true' }}
@@ -22,22 +23,12 @@ runs:
run: yarn build-static-viz
shell: bash
if: ${{ inputs.build-static-viz == 'true' }}
env:
MB_EDITION: ee
- name: Test database driver
run: clojure -X:dev:ci:ee:ee-dev:drivers:drivers-dev:test ${{ inputs.test-args }}
shell: bash
id: run-test
# Trunk Quarantine controls the final state of the run in the CI!
# Even if this step fails we let it pass, and then `trunk-io/analytics-uploader` determines
# whether to fail or to pass the whole job, based on the quarantine list.
continue-on-error: true
- name: Publish Test Report (JUnit)
uses: dorny/test-reporter@v1
# As per the documentation: https://docs.github.com/en/actions/reference/workflows-and-actions/contexts#steps-context
# test.outcome gives us access to the result before `continue-on-error` overrides it and sets it to 'success'.
# We want test reports even if the test step gets cancelled (e.g. times out), hence we match anything that is not a 'success'.
if: steps.run-test.outcome != 'success'
if: failure()
with:
path: "target/junit/**/*_test.xml"
name: JUnit Test Report ${{ inputs.junit-name }}
@@ -45,12 +36,3 @@ runs:
list-suites: failed
list-tests: failed
fail-on-error: false
- name: Upload Logs on Failure
if: always() && steps.run-test.outcome != 'success'
uses: actions/upload-artifact@v4
with:
name: test-logs-${{ github.job }}
path: logs/test-log.json
retention-days: 1
# Failing to upload the artifact should never affect the outcome of the whole test run.
continue-on-error: true

View File

@@ -1,41 +0,0 @@
name: Update cache
description: |
Update a GitHub Actions cache based on a provided path and key.
If a cache entry already exists for the given key, it is first removed.
This ensures immutability of cache entries, and prevents runtime errors.
inputs:
path:
description: "The path(s) to cache, separated by newlines."
required: true
key:
description: "The cache key to use."
required: true
runs:
using: "composite"
steps:
- name: Check if the cache entry exists
id: cache-lookup
uses: actions/cache/restore@v4
with:
path: ${{ inputs.path }}
key: ${{ inputs.key }}
lookup-only: true # Important bit: the action will not fetch anything, just check for the cache existence.
- name: Delete the cache entry if it exists
if: ${{ steps.cache-lookup.outputs.cache-hit == 'true' }}
run: |
echo "Cache entry found for key '${{ inputs.key }}'. Deleting it to ensure immutability."
gh api \
--method DELETE \
-H "Accept: application/vnd.github+json" \
-H "X-GitHub-Api-Version: 2022-11-28" \
/repos/${{ github.repository }}/actions/caches?key=${{ inputs.key }}
env:
GH_TOKEN: ${{ github.token }}
shell: bash
- name: Save the new cache entry
uses: actions/cache/save@v4
with:
path: ${{ inputs.path }}
key: ${{ inputs.key }}

View File

@@ -11,7 +11,4 @@ runs:
env:
GITHUB_TOKEN: ${{ inputs.github-token }}
shell: bash
run: |
git config --global user.email "github-automation@metabase.com"
git config --global user.name "Metabase Automation"
bash ${{ github.action_path }}/upgrade-deps.sh
run: bash ${{ github.action_path }}/upgrade-deps.sh

View File

@@ -59,11 +59,6 @@ for upgrade in $UPGRADES; do
# Commit the dependency update, and link to the diff
git add deps.edn modules
# Use an email and name that will pass the CLA check:
git config --global user.email "github-automation@metabase.com"
git config --global user.name "Metabase Automation"
git commit -m ${TITLE} -m ${BODY} --no-verify
git push -u origin $BRANCH_NAME

View File

@@ -18,9 +18,6 @@ inputs:
required: true
trunk-api-token:
required: true
variant:
required: false
default: ''
runs:
using: "composite"
@@ -48,11 +45,10 @@ runs:
aws s3 cp ${OUTPUT_FILE}.zip s3://$BUCKET/$DATE/$GITHUB_RUN_ID/$GITHUB_RUN_ATTEMPT/
- name: Upload results to Trunk
if: always() # Important! This step must always run because it controls job's exit code.
if: ${{ always() }}
uses: trunk-io/analytics-uploader@main
with:
junit-paths: ${{ inputs.input-path }}
variant: ${{ inputs.variant || inputs.output-name }}
org-slug: metabase
dry-run: ${{ github.event_name == 'pull_request' && github.event.pull_request.draft == true }} # quarantine, but do not upload results
token: ${{ inputs.trunk-api-token }}
continue-on-error: true

17
.github/auto_request_review.yml vendored Normal file
View File

@@ -0,0 +1,17 @@
# The reviewers configuration for the auto_request_review workflow
# See https://github.com/necojackarc/auto-request-review/tree/master?tab=readme-ov-file#reviewers-configuration
# for the spec
files:
# migrations related files
'resources/migrations/**':
- team:core-workflows
'src/metabase/db/custom_migrations.clj':
- team:core-workflows
options:
ignore_draft: true
ignored_keywords:
- DO NOT REVIEW
- WIP
enable_group_assignment: false

View File

@@ -1,5 +1,3 @@
version: 1
name: "Metabase CodeQL config"
paths-ignore:

View File

@@ -1,52 +0,0 @@
# Metabase Developer Assistant Instructions
## Frontend Development
### Repository Structure
```
frontend/src/
├── metabase/ # Main application components and pages
├── metabase-lib/ # Query building and data modeling utilities
├── metabase-types/ # TypeScript type definitions
└── metabase/ui/ # Design system components
e2e/ # Cypress end-to-end tests
frontend/test/ # Jest unit tests
```
### Technology Stack
- **Framework**: React 18 with TypeScript
- **State**: Redux Toolkit
- **Styling**: CSS Modules (preferred) > Emotion styled-components
- **UI**: `metabase/ui` components built with Mantine v8
- **Build**: Rspack (primary), Webpack (legacy)
- **Testing**: Jest + React Testing Library, Cypress
### Coding Standards
#### Component Preferences
- Prefer `metabase/ui`components over `metabase/common/components`
- Use `.tsx` for components, `.ts` for utilities
#### Styling
ALWAYS prefer Mantine style props,then CSS modules. DO NOT suggest styled components, they are deprecated.
#### TypeScript Migration
When heavily editing `.js`/`.jsx` files, create a separate PR to convert to TypeScript first, then implement changes.
#### Enterprise Features
Enterprise functionality MUST use the plugin system. It is very important to not expose enterprise code in the OSS version.
#### Testing Requirements
All PRs should include tests. Prefer Unit tests over E2E tests.
#### Localization
All user-facing strings MUST be localized using the ttag library. Localized strings should be complete phrases, do not concatenate a few separately localized strings. You should add context to strings where the meaning of the string might not be obvious in isolation: e.g. "Home" might have different words in some languages depending on whether you're talking about a dwelling or the landing page for a website.

View File

@@ -16,49 +16,29 @@ frontend_ci: &frontend_ci
- ".github/actions/prepare-backend/**"
- ".github/workflows/frontend.yml"
frontend_embedding_sdk_ci: &frontend_embedding_sdk_ci
- ".github/actions/prepare-frontend/**"
- ".github/actions/prepare-backend/**"
- ".github/workflows/*embedding-sdk*.yml"
frontend_configs: &frontend_configs
frontend_sources: &frontend_sources
- *shared_sources
- "enterprise/frontend/!(test)**/**/{!(*.spec),}.{js,jsx,ts,tsx,css}"
- "frontend/!(test)**/**/{!(*.spec),}.{js,jsx,ts,tsx,css}"
- "yarn.lock"
- "**/tsconfig*.json"
- "package.json"
- "babel.config.json"
- ".eslintrc.js"
- "postcss.config.js"
- "rspack.config.js"
- "rspack.main.config.js"
- "rspack.embedding-sdk-package.config.js"
- "rspack.embedding-sdk-bundle.config.js"
- "rspack.static-viz.config.js"
- "webpack.config.js"
- "webpack.embedding-sdk.config.js"
- "webpack.static-viz.config.js"
- ".nvmrc"
frontend_sources: &frontend_sources
- *shared_sources
- *frontend_configs
- "enterprise/frontend/!(test)**/**/{!(*.spec),}.{js,jsx,ts,tsx,css}"
- "frontend/!(test)**/**/{!(*.spec),}.{js,jsx,ts,tsx,css}"
frontend_embedding_sources:
- "enterprise/frontend/src/{embedding,embedding-*}/**/{!(*.spec),}.{js,jsx,ts,tsx,css}"
- "enterprise/frontend/src/metabase-enterprise/{embedding,embedding-*}**/{!(*.spec),}.{js,jsx,ts,tsx,css}"
frontend_embedding_sdk_sources:
- *frontend_configs
- "enterprise/frontend/!(test)**/**/{!(*.spec),}.{js,jsx,ts,tsx,css}"
frontend_embedding_sdk_package_sources:
- "enterprise/frontend/src/embedding-sdk-package/**/!(*.spec).{js,jsx,ts,tsx,css}"
- "enterprise/frontend/src/embedding-sdk/**"
frontend_specs: &frontend_specs
- *shared_specs
- "frontend/test/**"
- "frontend/**/tests/**"
- "frontend/**/*.unit.*"
- "enterprise/frontend/**/*.unit.*"
- ".github/**/*.unit.spec.*"
- "jest.config.js"
- "jest.tz.unit.conf.json"
@@ -122,51 +102,39 @@ e2e_ci: &e2e_ci
- ".github/actions/prepare-uberjar-artifact/**"
- ".github/actions/e2e-prepare-containers/**"
- ".github/actions/prepare-cypress/**"
- ".github/actions/run-snowplow-micro/**"
- ".github/workflows/e2e-*"
e2e_specs: &e2e_specs
- e2e/!(test-host-app)**/*.cy.*.{js,ts}
- "**/*.cy.*.js"
- "**/*.cy.*.ts"
- "e2e/runner/**"
- "e2e/support/**"
- "e2e/snapshot*/**"
e2e_embedding_sdk:
- *frontend_embedding_sdk_ci
- *sources
- "e2e/component/**/*.cy.*.{js,ts}"
e2e_embedding_sdk_compatibility:
- *default
- *frontend_embedding_sdk_ci
- *frontend_sources
- *backend_sources
- "e2e/embedding-sdk-host-apps/**"
- "e2e/runner/embedding-sdk/**"
- "e2e/test-host-app/**/*.cy.*.{js,ts}"
snowplow: &snowplow
- ".github/workflows/snowplow.yml"
- "snowplow/**"
e2e_cross_version: &e2e_cross_version
- ".github/workflows/e2e-cross-version.yml"
- "e2e/test/scenarios/cross-version/**"
e2e_all:
- *default
- *e2e_ci
- *sources
- *e2e_specs
- "e2e/**"
- *snowplow
snowplow:
- ".github/workflows/snowplow.yml"
- "snowplow/**"
documentation:
- "docs/**"
- "**/*.md"
embedding_documentation:
- ".typedoc/**"
- "docs/embedding/**"
yaml:
- "**/*.yml"
- "**/*.yaml"
codeql:
- ".github/workflows/codeql.yml"
- "frontend/src/**"
- "enterprise/frontend/src/**"

View File

@@ -1,171 +0,0 @@
const fs = require("fs");
const path = require("path");
const { execSync } = require("child_process");
const collectAndAverageTimings = async ({
github,
context,
daysBack = 7,
existingTimingsPath = "e2e/support/timings.json",
}) => {
try {
const artifacts = await findMergedTimingArtifacts({
github,
context,
daysBack,
});
if (artifacts.length === 0) {
throw new Error("No timing artifacts found in the specified time range");
}
const timingData = await downloadAndExtractArtifacts(
artifacts,
github,
context,
);
if (timingData.length === 0) {
throw new Error("No valid timing data extracted from artifacts");
}
const averagedTimings = averageTimings(timingData);
averagedTimings.durations.sort((a, b) => a.spec.localeCompare(b.spec));
fs.writeFileSync(
existingTimingsPath,
JSON.stringify(averagedTimings, null, 2),
);
console.log(`✅ Updated ${existingTimingsPath} with new timing data`);
return { artifactCount: timingData.length };
} catch (error) {
console.error("❌ Error in collectAndAverageTimings:", error.message);
throw error;
}
};
const findMergedTimingArtifacts = async ({ github, context, daysBack }) => {
const cutoffDate = new Date();
cutoffDate.setDate(cutoffDate.getDate() - daysBack);
console.log(
`Searching for merged timing artifacts created after ${cutoffDate.toISOString()}`,
);
const artifacts = [];
let page = 1;
while (true) {
try {
const { data } = await github.rest.actions.listArtifactsForRepo({
owner: context.repo.owner,
repo: context.repo.repo,
name: "merged-e2e-timings",
per_page: 100,
page,
});
const validArtifacts = data.artifacts
.filter((a) => new Date(a.created_at) > cutoffDate && !a.expired)
.map(({ name, created_at, id }) => ({ name, created_at, id }));
artifacts.push(...validArtifacts);
if (
data.artifacts.some((a) => new Date(a.created_at) <= cutoffDate) ||
data.artifacts.length < 100
) {
break;
}
page++;
} catch (error) {
console.log(
`Error fetching artifacts page ${page - 1}: ${error.message}`,
);
break;
}
}
console.log(`Found ${artifacts.length} merged timing artifacts total`);
return artifacts;
};
const downloadAndExtractArtifacts = async (artifacts, github, context) => {
const tempDir = "timing-artifacts-temp";
fs.mkdirSync(tempDir, { recursive: true });
const timingData = [];
for (const artifact of artifacts) {
try {
console.log(`Downloading artifact: ${artifact.name}`);
const zipPath = path.join(tempDir, `${artifact.name}.zip`);
const extractPath = path.join(tempDir, artifact.name);
const { data } = await github.rest.actions.downloadArtifact({
owner: context.repo.owner,
repo: context.repo.repo,
artifact_id: artifact.id,
archive_format: "zip",
});
fs.writeFileSync(zipPath, Buffer.from(data));
fs.mkdirSync(extractPath, { recursive: true });
execSync(`unzip -q "${zipPath}" -d "${extractPath}"`);
const timingFilePath = path.join(extractPath, "runTimings.json");
if (fs.existsSync(timingFilePath)) {
const content = JSON.parse(fs.readFileSync(timingFilePath, "utf8"));
if (content.durations?.length) {
timingData.push(content);
console.log(
`✅ Extracted timing data from ${artifact.name} (${content.durations.length} specs)`,
);
} else {
console.log(`⚠ Invalid timing data structure in ${artifact.name}`);
}
} else {
console.log(`⚠ No timing data found in ${artifact.name}`);
}
fs.rmSync(extractPath, { recursive: true });
} catch (error) {
console.log(`❌ Error processing ${artifact.name}: ${error.message}`);
}
}
fs.rmSync(tempDir, { recursive: true });
console.log(
`Successfully extracted timing data from ${timingData.length}/${artifacts.length} artifacts`,
);
return timingData;
};
const averageTimings = (timingDataArray) => {
if (!timingDataArray.length) {
throw new Error("No timing data to process");
}
const specTimings = timingDataArray
.flatMap((data) => data.durations)
.filter((item) => item.spec && typeof item.duration === "number")
.reduce((acc, { spec, duration }) => {
(acc[spec] = acc[spec] || []).push(duration);
return acc;
}, {});
const durations = Object.entries(specTimings).map(([spec, durations]) => ({
spec,
duration: Math.round(
durations.reduce((sum, d) => sum + d, 0) / durations.length,
),
}));
return { durations };
};
module.exports = { collectAndAverageTimings, averageTimings };

View File

@@ -1,132 +0,0 @@
const { averageTimings } = require("./average-e2e-timings");
describe("averageTimings", () => {
it("should throw error when given empty array", () => {
expect(() => averageTimings([])).toThrow("No timing data to process");
});
it("should work for n=1", () => {
const input = [
{
durations: [
{ spec: "test1.cy.js", duration: 1000 },
{ spec: "test2.cy.js", duration: 2000 },
],
},
];
const result = averageTimings(input);
expect(result.durations).toHaveLength(2);
expect(result.durations).toContainEqual({
spec: "test1.cy.js",
duration: 1000,
});
expect(result.durations).toContainEqual({
spec: "test2.cy.js",
duration: 2000,
});
});
it("should calculate average for multiple specs", () => {
const input = [
{
durations: [
{ spec: "test1.cy.js", duration: 1000 },
{ spec: "test2.cy.js", duration: 2000 },
],
},
{
durations: [
{ spec: "test1.cy.js", duration: 2000 },
{ spec: "test2.cy.js", duration: 4000 },
],
},
];
const result = averageTimings(input);
expect(result.durations).toHaveLength(2);
expect(result.durations).toContainEqual({
spec: "test1.cy.js",
duration: 1500,
});
expect(result.durations).toContainEqual({
spec: "test2.cy.js",
duration: 3000,
});
});
it("should round averages to nearest integer", () => {
const input = [
{
durations: [{ spec: "test1.cy.js", duration: 1001 }],
},
{
durations: [{ spec: "test1.cy.js", duration: 1002 }],
},
];
const result = averageTimings(input);
expect(result.durations[0].duration).toBe(1002);
});
it("should ignore invalid entries", () => {
const input = [
{
durations: [
{ spec: "test1.cy.js", duration: 1000 },
{ spec: "test2.cy.js", duration: "invalid" },
{ duration: 2000 }, // missing spec
{ spec: "test3.cy.js", duration: 3000 },
],
},
];
const result = averageTimings(input);
expect(result.durations).toHaveLength(2);
expect(result.durations).toContainEqual({
spec: "test1.cy.js",
duration: 1000,
});
expect(result.durations).toContainEqual({
spec: "test3.cy.js",
duration: 3000,
});
});
it("should handle merging different specs", () => {
const input = [
{
durations: [
{ spec: "test1.cy.js", duration: 1000 },
{ spec: "test2.cy.js", duration: 2000 },
],
},
{
durations: [
{ spec: "test1.cy.js", duration: 3000 },
{ spec: "test3.cy.js", duration: 1000 },
],
},
];
const result = averageTimings(input);
expect(result.durations).toHaveLength(3);
expect(result.durations).toContainEqual({
spec: "test1.cy.js",
duration: 2000,
});
expect(result.durations).toContainEqual({
spec: "test2.cy.js",
duration: 2000,
});
expect(result.durations).toContainEqual({
spec: "test3.cy.js",
duration: 1000,
});
});
});

View File

@@ -1,79 +0,0 @@
// This script is used in .github/workflows/e2e-matrix-builder.yml
// its aim is to split the e2e test matrix into multiple jobs
// grouping some specific tests together, other tests are split into chunks
const DEFAULT_SPEC_PATTERN = "./e2e/test/scenarios/**/*.cy.spec.*";
const specialTestConfigs = [
{
name: "oss-subset",
edition: "oss",
tags: "@OSS @smoke+-@EE",
specs: DEFAULT_SPEC_PATTERN,
},
{ name: "mongo", tags: "@mongo", specs: DEFAULT_SPEC_PATTERN },
{ name: "python", tags: "@python", specs: DEFAULT_SPEC_PATTERN },
];
/**
*
* @param {*} inputSpecs - specs, which were changed in the PR, separated by comma
* .e.g e2e/test/scenarios/onboarding/command-palette.cy.spec.js,e2e/test/scenarios/question/document-title.cy.spec.js
* or a pattern like DEFAULT_SPEC_PATTERN
*/
function buildMatrix(options, inputSpecs, inputChunks) {
const { java, defaultRunner } = options;
// number of specs per chunk when running specific specs
const SPECS_PER_CHUNK = 5;
const defaultOptions = {
"java-version": java,
runner: defaultRunner,
edition: "ee",
};
const isDefaultSpecPattern =
inputSpecs === "" || inputSpecs === DEFAULT_SPEC_PATTERN;
console.log("Processed specs value:", inputSpecs);
console.log("Is default pattern:", isDefaultSpecPattern);
let regularChunks;
if (isDefaultSpecPattern) {
regularChunks = inputChunks - specialTestConfigs.length;
} else {
// when pattern is not default, it means we passed some custom list of the changed specs
// so we need to calculate how many chunks we need to run
const matchingSpecsCount = inputSpecs.split(",").length;
regularChunks = Math.max(
1,
Math.ceil(matchingSpecsCount / SPECS_PER_CHUNK),
);
}
const regularTests = new Array(regularChunks).fill(1).map((files, index) => ({
name: `e2e-group-${index + 1}`,
// works when specs less than 5, otherwise seems all chunks will contain
// same specs
...(!isDefaultSpecPattern && {
specs: inputSpecs
.split(",")
.slice(SPECS_PER_CHUNK * index, SPECS_PER_CHUNK * (index + 1))
.join(","),
}),
}));
const testSets = isDefaultSpecPattern
? regularTests.concat(specialTestConfigs)
: regularTests;
const config = testSets.map((options) => ({
...defaultOptions,
...options,
}));
return { config, regularChunks, isDefaultSpecPattern };
}
module.exports = { buildMatrix };

View File

@@ -1,97 +0,0 @@
const { buildMatrix } = require("./build-e2e-matrix");
describe("buildMatrix", () => {
const options = {
java: "11",
defaultRunner: "ubuntu-22.04",
};
const testBuildMatrix = (entry, chunks) =>
buildMatrix(options, entry, chunks);
it("should calculate correct regular chunks with default parameters", () => {
const result = testBuildMatrix("./e2e/test/scenarios/**/*.cy.spec.*", 30);
expect(result.regularChunks).toBe(27);
});
it("should create correct number of configuration objects", () => {
const result = testBuildMatrix("./e2e/test/scenarios/**/*.cy.spec.*", 10);
expect(result.config.length).toBe(10);
});
it("should apply default specs path when empty entry is provided", () => {
const result = testBuildMatrix("", 6);
expect(result.regularChunks).toBe(3);
expect(result.config.length).toBe(6);
});
it("should name regular test groups sequentially", () => {
const result = testBuildMatrix("./e2e/test/scenarios/**/*.cy.spec.*", 10);
const regularTests = result.config.filter((cfg) =>
cfg.name.startsWith("e2e-group-"),
);
regularTests.forEach((test, index) => {
expect(test.name).toBe(`e2e-group-${index + 1}`);
});
});
it("should handle case when input chunks equals special tests count", () => {
const result = testBuildMatrix("./e2e/test/scenarios/**/*.cy.spec.*", 3);
expect(result.regularChunks).toBe(0);
expect(result.config.length).toBe(3);
expect(result.config[0].name).toBe("oss-subset");
expect(result.config[1].name).toBe("mongo");
expect(result.config[2].name).toBe("python");
});
it("should correctly calculate regularChunks for non-default spec pattern", () => {
const entry = "file.cy.spec.js,".repeat(12).slice(0, -1);
const result = testBuildMatrix(entry, 10);
// For 12 matching specs with SPECS_PER_CHUNK=5, we expect ceil(12/5) = 3 chunks
expect(result.regularChunks).toBe(3);
const firstGroupSpecs = result.config[0].specs.split(",").length;
const secondGroupSpecs = result.config[1].specs.split(",").length;
const thirdGroupSpecs = result.config[2].specs.split(",").length;
expect(firstGroupSpecs).toBe(5);
expect(secondGroupSpecs).toBe(5);
expect(thirdGroupSpecs).toBe(2);
});
it("should return isDefaultSpecPattern as true when empty or default value is provided", () => {
let result = testBuildMatrix("", 5);
expect(result.isDefaultSpecPattern).toBe(true);
result = testBuildMatrix("./e2e/test/scenarios/**/*.cy.spec.*", 5);
expect(result.isDefaultSpecPattern).toBe(true);
});
it("should return isDefaultSpecPattern as false when non-default spec pattern is provided", () => {
const result = testBuildMatrix("test1.cy.spec.js,test2.cy.spec.js", 1);
expect(result.isDefaultSpecPattern).toBe(false);
});
it("should return special test configs when default spec pattern is provided", () => {
const result = testBuildMatrix("./e2e/test/scenarios/**/*.cy.spec.*", 50);
expect(result.config[47].name).toBe("oss-subset");
expect(result.config[48].name).toBe("mongo");
expect(result.config[49].name).toBe("python");
});
it("should not return special test configs when non-default spec pattern is provided", () => {
const result = testBuildMatrix("test1.cy.spec.js,test2.cy.spec.js", 50);
expect(result.config.length).toBe(1);
expect(result.config[0].name).toBe("e2e-group-1");
});
});

View File

@@ -1,66 +1,24 @@
const TYPES = new Set([
"feat",
"fix",
"perf",
"docs",
"style",
"refactor",
"test",
"build",
"ci",
]);
/**
* @param title {string}
*/
function isConventionalTitle(title) {
const match = /^(?<type>\w+)\((?<rawScopes>[^)]+)\):/.exec(title);
if (!match) {
return false;
}
const [, type, rawScopes] = match;
const scopes = rawScopes
.split(",")
.map((scope) => scope.trim())
.filter(Boolean);
return TYPES.has(type) && scopes.length > 0;
}
exports.updateComment = updateComment;
/** @param {import('@types/github-script').AsyncFunctionArguments} AsyncFunctionArguments */
async function updateCommentForEmbedding({ github, context }) {
async function updateComment({ github, context }) {
const githubUsername = context.payload.sender.login;
const comment = `${CONVENTIONAL_COMMENT_IDENTIFIER}
@${githubUsername} You have modified embedding code. Please make sure the PR title follows [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) style.
Here are all the supported types:
const comment = `${CONVENTIONAL_COMMIT_REMINDER_COMMENT_IDENTIFIER}
@${githubUsername} You have modified embedding SDK code. Please make sure the PR title follows [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) style.
Here are all the supported types that will be shown in the changelog:
- \`feat\`
- \`fix\`
- \`perf\`
- \`docs\`
- \`style\`
- \`refactor\`
- \`test\`
- \`build\`
- \`ci\`
Please also make sure to include a scope.
These types \`docs\`, \`style\`, \`refactor\`, \`test\`, \`build\`, and \`ci\` are also encouraged for non-changelog related tasks.
Please also make sure to include \`sdk\` scope, otherwise, the changelog will not include this task.
For example, these are valid PR titles:
- \`feat(sdk-bundle): Add interactive dashboard component\`
- \`fix(sdk-package): Fixed MetabaseProviderPropsStore logic\`
- \`fix(sdk-package, embed-js): Fix API\``;
- \`feat(sdk): Add interactive dashboard component\`
- \`feat(sdk): Support theming pie chart\`
- \`fix(sdk): Fix static dashboard crash\``;
return updateComment(comment, { github, context });
}
/**
* @param comment {string}
* @param {import('@types/github-script').AsyncFunctionArguments} AsyncFunctionArguments
*/
async function updateComment(comment, { github, context }) {
const owner = context.repo.owner;
const repo = context.repo.repo;
const pullRequestNumber = context.issue.number;
@@ -91,19 +49,20 @@ async function updateComment(comment, { github, context }) {
}
}
const CONVENTIONAL_COMMENT_IDENTIFIER =
"<!---conventional comment identifier-->";
const CONVENTIONAL_COMMIT_REMINDER_COMMENT_IDENTIFIER =
"<!---conventional commit reminder comment ID-->";
/**
* @param comments {{body: string}[]}
* @typedef Comment
* @property {string} body
*
* @param {Comment[]} comments
*/
function getExistingConventionalCommitReminderComment(comments) {
for (const comment of comments) {
if (comment.body.startsWith(CONVENTIONAL_COMMENT_IDENTIFIER)) {
if (
comment.body.startsWith(CONVENTIONAL_COMMIT_REMINDER_COMMENT_IDENTIFIER)
) {
return comment;
}
}
}
exports.isConventionalTitle = isConventionalTitle;
exports.updateCommentForEmbedding = updateCommentForEmbedding;

View File

@@ -1,45 +0,0 @@
const { isConventionalTitle } = require("./utils");
describe("isConventionalTitle", () => {
it("returns true for a simple conventional commit", () => {
expect(isConventionalTitle("feat: add new feature")).toBe(false);
});
it("returns true for a commit with a scope", () => {
expect(isConventionalTitle("fix(api): correct API bug")).toBe(true);
});
it("returns true for a commit with multiple scopes, comma separated", () => {
expect(isConventionalTitle("feat(build,deps): update dependencies")).toBe(
true,
);
});
it("returns true for a commit with multiple scopes, comma separated with space", () => {
expect(isConventionalTitle("feat(build, deps): update dependencies")).toBe(
true,
);
});
it("returns false for a commit with round brackets in a message", () => {
expect(isConventionalTitle("feat: (something) update dependencies")).toBe(
false,
);
});
it("returns false for missing colon", () => {
expect(isConventionalTitle("feat add feature")).toBe(false);
});
it("returns false for invalid type", () => {
expect(isConventionalTitle("invalidType: some message")).toBe(false);
});
it("returns false for empty message", () => {
expect(isConventionalTitle("feat:")).toBe(false);
});
it("returns false for empty input", () => {
expect(isConventionalTitle("")).toBe(false);
});
});

View File

@@ -1,39 +0,0 @@
const extractChangedTimings = ({ oldTimings, newTimings }) => {
const oldTimingMap =
oldTimings.durations?.reduce((map, item) => {
if (item.spec) {
map[item.spec] = item.duration;
}
return map;
}, {}) || {};
const changedDurations = newTimings.durations
.filter((item) => {
if (!item || !item.spec || typeof item.duration !== "number") {
return false;
}
const newSpec = convertPathFormat(item.spec);
const oldDuration = oldTimingMap[newSpec] || oldTimingMap[item.spec];
// Include if it's a new spec or the duration has changed
return oldDuration === undefined || oldDuration !== item.duration;
})
.map((item) => ({
spec: convertPathFormat(item.spec),
duration: item.duration,
}));
return {
changedTimings: { durations: changedDurations },
hasChanges: changedDurations.length > 0,
};
};
// Convert cypress-split path format to timings.json path format
// from "e2e/test/scenarios/..." to "../test/scenarios/..."
function convertPathFormat(specPath) {
return specPath.startsWith("e2e/test/")
? specPath.replace("e2e/test/", "../test/")
: specPath;
}
module.exports = { extractChangedTimings };

View File

@@ -1,195 +0,0 @@
const { extractChangedTimings } = require("./extract-changed-timings");
describe("extractChangedTimings", () => {
it("should return empty durations when no changes", () => {
const oldTimings = {
durations: [
{ spec: "test1.cy.js", duration: 1000 },
{ spec: "test2.cy.js", duration: 2000 },
],
};
const newTimings = {
durations: [
{ spec: "test1.cy.js", duration: 1000 },
{ spec: "test2.cy.js", duration: 2000 },
],
};
const result = extractChangedTimings({ oldTimings, newTimings });
expect(result.changedTimings.durations).toHaveLength(0);
expect(result.hasChanges).toBe(false);
});
it("should detect new specs", () => {
const oldTimings = {
durations: [{ spec: "test1.cy.js", duration: 1000 }],
};
const newTimings = {
durations: [
{ spec: "test1.cy.js", duration: 1000 },
{ spec: "test2.cy.js", duration: 2000 },
],
};
const result = extractChangedTimings({ oldTimings, newTimings });
expect(result.changedTimings.durations).toHaveLength(1);
expect(result.changedTimings.durations[0]).toEqual({
spec: "test2.cy.js",
duration: 2000,
});
expect(result.hasChanges).toBe(true);
});
it("should detect duration changes", () => {
const oldTimings = {
durations: [
{ spec: "test1.cy.js", duration: 1000 },
{ spec: "test2.cy.js", duration: 2000 },
],
};
const newTimings = {
durations: [
{ spec: "test1.cy.js", duration: 1000 },
{ spec: "test2.cy.js", duration: 2200 },
],
};
const result = extractChangedTimings({ oldTimings, newTimings });
expect(result.changedTimings.durations).toHaveLength(1);
expect(result.changedTimings.durations[0]).toEqual({
spec: "test2.cy.js",
duration: 2200,
});
expect(result.hasChanges).toBe(true);
});
it("should handle missing old timings", () => {
const oldTimings = { durations: [] };
const newTimings = {
durations: [
{ spec: "test1.cy.js", duration: 1000 },
{ spec: "test2.cy.js", duration: 2000 },
],
};
const result = extractChangedTimings({ oldTimings, newTimings });
// All timings should be considered "changed" since old timings is empty
expect(result.changedTimings.durations).toHaveLength(2);
expect(result.changedTimings.durations).toContainEqual({
spec: "test1.cy.js",
duration: 1000,
});
expect(result.changedTimings.durations).toContainEqual({
spec: "test2.cy.js",
duration: 2000,
});
expect(result.hasChanges).toBe(true);
});
it("should handle invalid timing entries", () => {
const oldTimings = {
durations: [{ spec: "test1.cy.js", duration: 1000 }],
};
const newTimings = {
durations: [
{ spec: "test1.cy.js", duration: 1000 },
{ spec: "test2.cy.js", duration: "invalid" },
{ duration: 2000 }, // missing spec
{ spec: "test3.cy.js", duration: 3000 },
],
};
const result = extractChangedTimings({ oldTimings, newTimings });
// Only test3.cy.js should be included (new spec with valid data)
expect(result.changedTimings.durations).toHaveLength(1);
expect(result.changedTimings.durations[0]).toEqual({
spec: "test3.cy.js",
duration: 3000,
});
expect(result.hasChanges).toBe(true);
});
it("should handle timing data with different specs", () => {
const oldTimings = {
durations: [
{ spec: "test1.cy.js", duration: 1000 },
{ spec: "test2.cy.js", duration: 2000 },
],
};
const newTimings = {
durations: [
{ spec: "test1.cy.js", duration: 3000 },
{ spec: "test3.cy.js", duration: 1000 },
],
};
const result = extractChangedTimings({ oldTimings, newTimings });
expect(result.changedTimings.durations).toHaveLength(2);
expect(result.changedTimings.durations).toContainEqual({
spec: "test1.cy.js",
duration: 3000,
});
expect(result.changedTimings.durations).toContainEqual({
spec: "test3.cy.js",
duration: 1000,
});
expect(result.hasChanges).toBe(true);
});
it("should convert cypress-split paths to relative paths", () => {
const oldTimings = {
durations: [
{
spec: "../test/scenarios/dashboard/dashboard.cy.spec.js",
duration: 1000,
},
{
spec: "../test/scenarios/admin/databases.cy.spec.js",
duration: 2000,
},
],
};
const newTimings = {
durations: [
{
spec: "e2e/test/scenarios/dashboard/dashboard.cy.spec.js",
duration: 1200,
},
{
spec: "e2e/test/scenarios/admin/databases.cy.spec.js",
duration: 2000,
},
{
spec: "e2e/test/scenarios/collections/collections.cy.spec.js",
duration: 1500,
},
],
};
const result = extractChangedTimings({ oldTimings, newTimings });
expect(result.changedTimings.durations).toHaveLength(2);
expect(result.changedTimings.durations).toContainEqual({
spec: "../test/scenarios/dashboard/dashboard.cy.spec.js",
duration: 1200,
});
expect(result.changedTimings.durations).toContainEqual({
spec: "../test/scenarios/collections/collections.cy.spec.js",
duration: 1500,
});
expect(result.hasChanges).toBe(true);
});
});

View File

@@ -1,84 +0,0 @@
const githubSlackMap = require("../../release/github-slack-map.json");
async function getIssueData({ context, github, manualIssueNumber }) {
let issueNumber;
let issueData;
let prReviews = [];
const triggerType = context.eventName;
if (context.eventName === "issues") {
issueNumber = context.payload.issue.number;
issueData = context.payload.issue;
} else if (context.eventName === "workflow_dispatch") {
issueNumber = parseInt(manualIssueNumber);
const { data } = await github.rest.issues.get({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: issueNumber,
});
issueData = data;
} else if (context.eventName === "pull_request_review") {
const prNumber = context.payload.pull_request.number;
const { data: prData } = await github.rest.pulls.get({
owner: context.repo.owner,
repo: context.repo.repo,
pull_number: prNumber,
});
// Extract issue number from branch name (e.g., "claude-fix/issue-12345-some-title")
const branchMatch = prData.head.ref.match(/claude-fix\/issue-(\d+)/);
if (!branchMatch) {
throw new Error(
`Could not extract issue number from branch name: ${prData.head.ref}`,
);
}
issueNumber = parseInt(branchMatch[1]);
const { data } = await github.rest.issues.get({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: issueNumber,
});
issueData = data;
const { data: reviews } = await github.rest.pulls.listReviews({
owner: context.repo.owner,
repo: context.repo.repo,
pull_number: prNumber,
});
prReviews = reviews.filter(
(review) =>
review.state === "CHANGES_REQUESTED" &&
isMetabaseMember(review.user.login),
);
}
const { data: comments } = await github.rest.issues.listComments({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: issueNumber,
});
const employeeComments = comments.filter((comment) =>
isMetabaseMember(comment.user.login),
);
return {
issueNumber,
issueData,
employeeComments,
prReviews,
triggerType,
isIssueAuthorEmployee: isMetabaseMember(issueData.user.login),
};
}
function isMetabaseMember(username) {
return username in githubSlackMap;
}
module.exports = {
getIssueData,
};

View File

@@ -1,30 +0,0 @@
#!/bin/bash
set -e
# Parse command line arguments
SKIP_START=false
while [[ $# -gt 0 ]]; do
case $1 in
--skip-start)
SKIP_START=true
shift
;;
*)
echo "Unknown option: $1"
echo "Usage: $0 [--skip-start]"
exit 1
;;
esac
done
echo "Cloning python-runner repository..."
git clone --depth 1 https://${METABASE_AUTOMATION_USER_TOKEN}@github.com/metabase/python-runner-container.git python-runner
if [ "$SKIP_START" = false ]; then
echo "Starting python-runner..."
cd python-runner
make run-ci 1> ../python-runner.out.log 2> ../python-runner.err.log
make logs 1>> ../python-runner.out.log 2>> ../python-runner.err.log &
cd ..
fi

View File

@@ -49,55 +49,6 @@ function httpsRequest(options, data = null) {
});
}
// Makes an HTTPS request with exponential backoff retry logic
async function httpsRequestWithBackoff(options, data = null, backoffOptions = {}) {
// Set default backoff options:
const {
maxRetries = 8,
totalDurationSeconds = 60
} = backoffOptions;
let retries = 0;
let delay = 500; // Start with a .5 second delay
// Calculate a reasonable initial delay based on total duration and max retries
// This helps distribute the retry attempts across the total allowed duration
const totalDurationMs = totalDurationSeconds * 1000;
// Track the start time
const startTime = Date.now();
// Ensure data is stringified if it's an object
const requestData = data ? (typeof data === 'string' ? data : JSON.stringify(data)) : null;
while (true) {
try {
const response = await httpsRequest(options, requestData);
return response; // Success: Return the data
} catch (error) {
retries++;
const elapsedMs = Date.now() - startTime;
const timeRemaining = totalDurationMs - elapsedMs;
// Stop retrying if we've hit max retries or exceeded the total duration
if (retries >= maxRetries || timeRemaining <= 0) {
console.error(`Failed after ${retries} retries (${Math.round(elapsedMs/1000)}s elapsed):`);
throw error;
}
// Calculate the next delay with exponential backoff and jitter
delay = Math.min(delay * 2 * (0.8 + Math.random() * 0.4), timeRemaining);
console.log(`Request failed (attempt ${retries}/${maxRetries}). Retrying in ${Math.round(delay)}ms... (${Math.round(timeRemaining/1000)}s remaining)`);
// Wait for the calculated delay before trying again
await new Promise(resolve => setTimeout(resolve, delay));
}
}
}
async function link_issues(github) {
try {
@@ -124,14 +75,6 @@ async function link_issues(github) {
console.log('PR Info:\n' + JSON.stringify(prInfo, null, 2) + "\n");
if (prInfo &&
prInfo.data &&
prInfo.data.labels &&
prInfo.data.labels.some(label => label.name === "no-auto-issue-links")) {
console.log('PR has "no-auto-issue-links" label, skipping...');
return;
}
const prUrl = prInfo.data.html_url;
console.log(`Processing PR: ${prUrl}`);
@@ -147,11 +90,7 @@ async function link_issues(github) {
attachments {
nodes {url}}}}}}`});
console.log('--- Linear API query ---');
console.log(linearQuery);
console.log('----- End of query -----');
const linearData = await httpsRequestWithBackoff({
const linearData = await httpsRequest({
hostname: 'api.linear.app',
path: '/graphql',
method: 'POST',
@@ -159,9 +98,7 @@ async function link_issues(github) {
'Content-Type': 'application/json',
'Authorization': linearApiKey
}
},
linearQuery,
{ maxRetries: 8, totalDurationSeconds: 60 });
}, linearQuery);
console.log('Linear API response:');
console.log(JSON.stringify(linearData) + "\n");
@@ -216,9 +153,7 @@ async function link_issues(github) {
// Generate closing references
const closingRefs = issueNumbersToAdd.map(num => `closes #${num}`).join(' ');
const messagePreamble = "<!-- Added by 'Add Issue References to PR' GitHub Action. To disable linking, add 'no-auto-issue-links' label to your PR. -->"
newBody = `${messagePreamble} ${closingRefs}\n` + newBody;
newBody = `<!-- Added by 'Add Issue References to PR' GitHub Action -->${closingRefs}\n` + newBody;
// Update PR body with new closing references
await github.rest.issues.update({

114
.github/team.json vendored
View File

@@ -8,9 +8,18 @@
"romeovs",
"bshepherdson",
"metamben",
"appleby",
"uladzimirdev",
"wzimrin",
"ericnormand",
"galdre"
"ericnormand"
]
},
{
"name": "Semantic Layer",
"label": ".Team/SemanticLayer",
"members": [
"kamilmielnik",
"lbrdnk"
]
},
{
@@ -26,17 +35,48 @@
"label": ".Team/Embedding",
"members": [
"WiNloSt",
"oisincoveney",
"npretto",
"heypoom",
"sanex3339",
"lorem--ipsum"
"sanex3339"
]
},
{
"name": "Workflows",
"label": ".Team/Workflows",
"members": [
"qnkhuat",
"piranha",
"crisptrutski",
"deniskaber",
"wotbrew",
"spalt08",
"stasgavrylov"
]
},
{
"name": "Admin Webapp",
"label": ".Team/AdminWebapp",
"members": [
"eliotmetabase"
"npfitz",
"johnswanson",
"rafpaf",
"sloansparger",
"iethree",
"nvoxland",
"fraserdrops",
"edpaget"
]
},
{
"name": "DashViz",
"label": ".Team/DashViz",
"members": [
"alxnddr",
"kulyk",
"tsplude",
"lorem--ipsum",
"noahmoss"
]
},
{
@@ -44,69 +84,7 @@
"label": ".Team/DevEx",
"members": [
"camsaul",
"dpsutton",
"escherize",
"fraserdrops",
"nemanjaglumac"
]
},
{
"name": "Metabot",
"label": ".Team/Metabot",
"members": [
"appleby",
"noahmoss",
"retro",
"sloansparger",
"somtom",
"tsplude",
"lbrdnk",
"piranha",
"crisptrutski"
]
},
{
"name": "Cloud",
"label": "cloud",
"members": [
"filipesilva"
]
},
{
"name": "DashViz",
"label": ".Team/DashViz",
"members": [
"eliotmetabase"
]
},
{
"name": "UX West",
"label": ".Team/UXWest",
"members": [
"eliotmetabase",
"npfitz",
"johnswanson",
"iethree",
"nvoxland",
"edpaget",
"deniskaber",
"alxnddr",
"bpander",
"andsalves"
]
},
{
"name": "Gadget",
"label": ".Team/Gadget",
"members": [
"nmezzopera",
"bronsa",
"wotbrew",
"qnkhuat",
"chodorowicz",
"kamilmielnik",
"stasgavrylov",
"uladzimirdev"
"escherize"
]
}
]

View File

@@ -1,420 +0,0 @@
name: App DB Tests
on:
workflow_call:
inputs:
skip:
type: boolean
default: false
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }}-app-db
cancel-in-progress: true
jobs:
be-tests-mariadb:
if: ${{ !inputs.skip }}
runs-on: ubuntu-22.04
timeout-minutes: 60
strategy:
fail-fast: false
matrix:
version:
- name: MariaDB 10.2
junit-name: be-tests-mariadb-10-2-ee
image: circleci/mariadb:10.2.23
env:
enable-ssl-tests: 'false'
- name: MariaDB Latest
junit-name: be-tests-mariadb-latest-ee
image: circleci/mariadb:latest
env:
enable-ssl-tests: 'false'
job:
- name: Enterprise Tests
build-static-viz: false
test-args: >-
:only '"enterprise/backend/test"'
exclude-tag: ':mb/driver-tests :mb/transforms-python-test'
- name: EE App DB Tests (Part 1)
build-static-viz: true
test-args: >-
:only '"test"'
:partition/total 2
:partition/index 0
exclude-tag: ':mb/driver-tests :mb/transforms-python-test'
- name: EE App DB Tests (Part 2)
build-static-viz: true
test-args: >-
:only '"test"'
:partition/total 2
:partition/index 1
exclude-tag: ':mb/driver-tests :mb/transforms-python-test'
services:
mysql:
image: ${{ matrix.version.image }}
ports:
- "3306:3306"
env:
MYSQL_ALLOW_EMPTY_PASSWORD: true
MYSQL_DATABASE: circle_test
env:
# Only run old migrations tests on pushes to master or release branches. All other branches should skip tests
# with the tag `mb/old-migrations-test`. `__ADDITIONAL_EXCLUDED_TAG__` is not used anywhere outside of splicing
# it in to the command below.
__ADDITIONAL_EXCLUDED_TAG__: >-
${{
(
github.event_name == 'push' &&
(
github.ref == 'master' ||
startsWith(github.ref, 'release-')
) &&
''
) ||
':mb/old-migrations-test'
}}
# actual serious env vars below
CI: 'true'
DRIVERS: mysql
MB_DB_TYPE: mysql
MB_DB_HOST: localhost
MB_DB_PORT: 3306
MB_DB_DBNAME: circle_test
MB_DB_USER: root
MB_MYSQL_TEST_USER: root
#
# SSL config: only for mysql latest tests.
#
# set up env vars for something named "MYSQL_SSL" to run MySQL SSL tests verifying connectivity with PEM cert
# they are deliberately given a different name to prevent them from affecting the regular test run against
# the configured MySQL instance, but there is one particular test (mysql-connect-with-ssl-and-pem-cert-test)
# that overrides the MB_MYSQL_TEST_* values with them
# the MYSQL_RDS_SSL_INSTANCE vars are defined as secrets and can be altered
MB_MYSQL_SSL_TEST_SSL: ${{ matrix.version.env.enable-ssl-tests }}
MB_MYSQL_SSL_TEST_HOST: ${{ secrets.MYSQL_RDS_SSL_INSTANCE_HOST }}
MB_MYSQL_SSL_TEST_ADDITIONAL_OPTIONS: 'verifyServerCertificate=true'
# the contents of the ./resources/certificates/rds-combined-ca-bundle.pem file
MB_MYSQL_SSL_TEST_SSL_CERT: ${{ secrets.MB_MYSQL_SSL_TEST_SSL_CERT }}
MB_MYSQL_SSL_TEST_USER: metabase
MB_MYSQL_SSL_TEST_PASSWORD: ${{ secrets.MYSQL_RDS_SSL_INSTANCE_PASSWORD }}
# for historic reasons (I don't want to mess around with required jobs) the job name should be something like
# "be-tests-mariadb-10-2-ee (0)"
name: "${{ matrix.version.name }} ${{ matrix.job.name }}"
steps:
- uses: actions/checkout@v4
- name: Test ${{ matrix.version.name }}
uses: ./.github/actions/test-driver
with:
build-static-viz: ${{ matrix.job.build-static-viz }}
junit-name: ${{ matrix.version.junit-name }}
test-args: >-
${{ matrix.job.test-args }}
:exclude-tags '[ ${{ matrix.job.exclude-tag }} ${{ env.__ADDITIONAL_EXCLUDED_TAG__ }}]'
- name: Upload Test Results
uses: ./.github/actions/upload-test-results
if: always()
with:
input-path: ./target/junit/
output-name: ${{ github.job }}
bucket: ${{ vars.AWS_S3_TEST_RESULTS_BUCKET }}
aws-access-key-id: ${{ secrets.AWS_TEST_RESULTS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_TEST_RESULTS_SECRET_ACCESS_KEY }}
aws-region: ${{ vars.AWS_REGION }}
trunk-api-token: ${{ secrets.TRUNK_API_TOKEN }}
be-tests-mysql:
if: ${{ !inputs.skip }}
runs-on: ubuntu-22.04
timeout-minutes: 60
strategy:
fail-fast: false
matrix:
version:
- name: MySQL 8.0
junit-name: be-tests-mysql-8-0-ee
image: cimg/mysql:8.0
env:
enable-ssl-tests: 'false'
enable-aws-iam-tests: 'false'
- name: MySQL Latest
junit-name: be-tests-mysql-latest-ee
image: mysql:latest
env:
enable-ssl-tests: 'true'
enable-aws-iam-tests: 'true'
job:
- name: Enterprise Tests
build-static-viz: false
test-args: >-
:only '"enterprise/backend/test"'
exclude-tag: ':mb/driver-tests :mb/transforms-python-test'
- name: EE App DB Tests (Part 1)
build-static-viz: true
test-args: >-
:only '"test"'
:partition/total 2
:partition/index 0
exclude-tag: ':mb/driver-tests :mb/transforms-python-test'
- name: EE App DB Tests (Part 2)
build-static-viz: true
test-args: >-
:only '"test"'
:partition/total 2
:partition/index 1
exclude-tag: ':mb/driver-tests :mb/transforms-python-test'
services:
mysql:
image: ${{ matrix.version.image }}
ports:
- "3306:3306"
env:
MYSQL_ALLOW_EMPTY_PASSWORD: true
MYSQL_DATABASE: circle_test
env:
# Only run old migrations tests on pushes to master or release branches. All other branches should skip tests
# with the tag `mb/old-migrations-test`. `__ADDITIONAL_EXCLUDED_TAG__` is not used anywhere outside of splicing
# it in to the command below.
__ADDITIONAL_EXCLUDED_TAG__: >-
${{
(
github.event_name == 'push' &&
(
github.ref == 'master' ||
startsWith(github.ref, 'release-')
) &&
''
) ||
':mb/old-migrations-test'
}}
# actual serious env vars below
CI: 'true'
DRIVERS: mysql
MB_DB_TYPE: mysql
MB_DB_HOST: localhost
MB_DB_PORT: 3306
MB_DB_DBNAME: circle_test
MB_DB_USER: root
MB_MYSQL_TEST_USER: root
#
# SSL config: only for mysql latest tests.
#
# set up env vars for something named "MYSQL_SSL" to run MySQL SSL tests verifying connectivity with PEM cert
# they are deliberately given a different name to prevent them from affecting the regular test run against
# the configured MySQL instance, but there is one particular test (mysql-connect-with-ssl-and-pem-cert-test)
# that overrides the MB_MYSQL_TEST_* values with them
# the MYSQL_RDS_SSL_INSTANCE vars are defined as secrets and can be altered
MB_MYSQL_SSL_TEST_SSL: ${{ matrix.version.env.enable-ssl-tests }}
MB_MYSQL_SSL_TEST_HOST: ${{ secrets.MYSQL_RDS_SSL_INSTANCE_HOST }}
MB_MYSQL_SSL_TEST_ADDITIONAL_OPTIONS: 'verifyServerCertificate=true'
# the contents of the ./resources/certificates/rds-combined-ca-bundle.pem file
MB_MYSQL_SSL_TEST_SSL_CERT: ${{ secrets.MB_MYSQL_SSL_TEST_SSL_CERT }}
MB_MYSQL_SSL_TEST_USER: metabase
MB_MYSQL_SSL_TEST_PASSWORD: ${{ secrets.MYSQL_RDS_SSL_INSTANCE_PASSWORD }}
MB_MYSQL_AWS_IAM_TEST: ${{ matrix.version.env.enable-aws-iam-tests }}
MB_MYSQL_AWS_IAM_TEST_HOST: ${{ secrets.MYSQL_AURORA_IAM_INSTANCE_HOST }}
MB_MYSQL_AWS_IAM_TEST_PORT: 3306
MB_MYSQL_AWS_IAM_TEST_USER: my_iam_user
MB_MYSQL_AWS_IAM_TEST_DBNAME: test_db
MB_MYSQL_AWS_IAM_TEST_SSL_CERT: trust
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_AURORA_IAM_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_AURORA_IAM_SECRET_ACCESS_KEY }}
# for historic reasons (I don't want to mess around with required jobs) the job name should be something like
# "be-tests-mariadb-10-2-ee (0)"
name: "${{ matrix.version.name }} ${{ matrix.job.name }}"
steps:
- uses: actions/checkout@v4
- name: Test ${{ matrix.version.name }}
uses: ./.github/actions/test-driver
with:
build-static-viz: ${{ matrix.job.build-static-viz }}
junit-name: ${{ matrix.version.junit-name }}
test-args: >-
${{ matrix.job.test-args }}
:exclude-tags '[ ${{ matrix.job.exclude-tag }} ${{ env.__ADDITIONAL_EXCLUDED_TAG__ }}]'
- name: Upload Test Results
uses: ./.github/actions/upload-test-results
if: always()
with:
input-path: ./target/junit/
output-name: ${{ github.job }}
bucket: ${{ vars.AWS_S3_TEST_RESULTS_BUCKET }}
aws-access-key-id: ${{ secrets.AWS_TEST_RESULTS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_TEST_RESULTS_SECRET_ACCESS_KEY }}
aws-region: ${{ vars.AWS_REGION }}
trunk-api-token: ${{ secrets.TRUNK_API_TOKEN }}
be-tests-postgres:
if: ${{ !inputs.skip }}
runs-on: ubuntu-22.04
timeout-minutes: 60
strategy:
fail-fast: false
matrix:
version:
- name: Postgres 12.x
junit-name: postgres-ee
docker-image: postgres:12-alpine
env:
enable-ssl-tests: 'false'
enable-aws-iam-tests: 'false'
- name: Postgres Latest
junit-name: postgres-latest-ee
docker-image: postgres:latest
env:
enable-ssl-tests: 'true'
enable-aws-iam-tests: 'true'
job:
- name: Enterprise Tests
build-static-viz: false
test-args: >-
:only '"enterprise/backend/test"'
exclude-tag: ':mb/driver-tests :mb/transforms-python-test'
- name: EE App DB Tests (Part 1)
build-static-viz: true
test-args: >-
:only '"test"'
:exclude-tags [:mb/driver-tests]
:partition/total 2
:partition/index 0
exclude-tag: ':mb/driver-tests :mb/transforms-python-test'
- name: EE App DB Tests (Part 2)
build-static-viz: true
test-args: >-
:only '"test"'
:partition/total 2
:partition/index 1
exclude-tag: ':mb/driver-tests :mb/transforms-python-test'
name: "${{ matrix.version.name }} ${{ matrix.job.name }}"
env:
# Only run old migrations tests on pushes to master or release branches. All other branches should skip tests
# with the tag `mb/old-migrations-test`. `__ADDITIONAL_EXCLUDED_TAG__` is not used anywhere outside of splicing
# it in to the command below.
__ADDITIONAL_EXCLUDED_TAG__: >-
${{
(
github.event_name == 'push' &&
(
github.ref == 'master' ||
startsWith(github.ref, 'release-')
) &&
''
) ||
':mb/old-migrations-test'
}}
# actual serious env vars below
CI: 'true'
DRIVERS: postgres
MB_DB_TYPE: postgres
MB_DB_PORT: 5432
MB_DB_HOST: localhost
MB_DB_DBNAME: mb_test
MB_DB_USER: mb_test
MB_POSTGRESQL_TEST_USER: mb_test
# SSL tests are only enabled for the postgres-latest job.
MB_POSTGRES_SSL_TEST_SSL: ${{ matrix.version.env.enable-ssl-tests }}
MB_POSTGRES_SSL_TEST_SSL_MODE: verify-full
MB_POSTGRES_SSL_TEST_SSL_ROOT_CERT_PATH: 'test-resources/certificates/us-east-2-bundle.pem'
MB_POSTGRES_AWS_IAM_TEST: ${{ matrix.version.env.enable-aws-iam-tests }}
MB_POSTGRES_AWS_IAM_TEST_HOST: ${{ secrets.POSTGRES_AURORA_IAM_INSTANCE_HOST }}
MB_POSTGRES_AWS_IAM_TEST_PORT: 5432
MB_POSTGRES_AWS_IAM_TEST_USER: my_iam_user
MB_POSTGRES_AWS_IAM_TEST_DBNAME: test_db
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_AURORA_IAM_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_AURORA_IAM_SECRET_ACCESS_KEY }}
services:
postgres:
image: ${{ matrix.version.docker-image }}
ports:
- "5432:5432"
env:
POSTGRES_USER: mb_test
POSTGRES_DB: mb_test
POSTGRES_HOST_AUTH_METHOD: trust
steps:
- uses: actions/checkout@v4
- name: Test ${{ matrix.version.name }} (${{ matrix.job.name }})
uses: ./.github/actions/test-driver
with:
build-static-viz: ${{ matrix.job.build-static-viz }}
junit-name: 'be-tests-${{ matrix.version.junit-name }}'
test-args: >-
${{ matrix.job.test-args }}
:exclude-tags '[ ${{ matrix.job.exclude-tag }} ${{ env.__ADDITIONAL_EXCLUDED_TAG__ }}]'
- name: Upload Test Results
uses: ./.github/actions/upload-test-results
if: always()
with:
input-path: ./target/junit/
output-name: ${{ github.job }}
bucket: ${{ vars.AWS_S3_TEST_RESULTS_BUCKET }}
aws-access-key-id: ${{ secrets.AWS_TEST_RESULTS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_TEST_RESULTS_SECRET_ACCESS_KEY }}
aws-region: ${{ vars.AWS_REGION }}
trunk-api-token: ${{ secrets.TRUNK_API_TOKEN }}
app-db-tests-result:
needs:
- be-tests-mariadb
- be-tests-mysql
- be-tests-postgres
runs-on: ubuntu-latest
timeout-minutes: 5
name: app-db-tests-result
if: always() && !cancelled()
steps:
- name: Check job status
uses: actions/github-script@v7
env:
needs: ${{ toJson(needs) }}
with:
script: | # js
const needs = JSON.parse(process.env.needs);
const jobs = Object.entries(needs).map(
([jobName, jobValues]) => ({
name: jobName,
result: jobValues.result
}));
// are all jobs skipped or successful?
if (jobs.every(job => (job.result === 'skipped' || job.result === 'success'))) {
console.log("");
console.log(" _------. ");
console.log(" / , \_ ");
console.log(" / / /{}\ |o\_ ");
console.log(" / \ `--' /-' \ ");
console.log(" | \ \ | ");
console.log(" | |`-, | ");
console.log(" / /__/)/ ");
console.log(" | | ");
console.log("");
console.log("All app db tests have passed (or have been skipped). Cam is very proud of you.");
process.exit(0);
}
// otherwise, something failed
console.log("");
console.log(" .::::::::::. .::::::::::. ");
console.log(" .::::''''''::::. .::::''''''::::. ");
console.log(" .:::' `::::.... ....::::' `:::. ");
console.log(" .::' `:::::::| |:::::::' `::. ");
console.log(" .::| |::::::|_ ___ __|::::::| |::. ");
console.log(" `--' |::::::|_()__()_|::::::| `--' ");
console.log(" ::: |::-o::| |::o-::| ::: ");
console.log(" `::. .|::::::| |::::::|. .::' ");
console.log(" `:::. .::\-----' `-----/::. .:::' ");
console.log(" `::::......::::' `::::......::::' ");
console.log(" `::::::::::' `::::::::::' ");
console.log("");
console.log("App db tests have failed. You have been sentenced to MetaJail.");
jobs.forEach((job) => {
if (job.result !== 'success') {
console.log(`${job.name} - ${job.result}`);
}
});
process.exit(1);

View File

@@ -1,38 +0,0 @@
name: Publish Athena JDBC Driver
on:
workflow_dispatch:
inputs:
version:
description: 'Athena JDBC driver version to publish'
required: true
type: string
jobs:
deploy:
name: Publish Athena JDBC Driver
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Check out the code
uses: actions/checkout@v4
- name: Set up Babashka
uses: turtlequeue/setup-babashka@v1.7.0
with:
babashka-version: 1.12.197
- name: Set up AWS CLI
uses: aws-actions/configure-aws-credentials@v4
with:
aws-access-key-id: ${{ secrets.AWS_S3_MAVEN_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_S3_MAVEN_SECRET_ACCESS_KEY }}
aws-region: ${{ vars.AWS_REGION }}
- name: Deploy Athena JDBC driver
run: ./bin/athena.bb ${{ github.event.inputs.version }} --deploy
- name: Verify deployment
run: |
aws s3 ls s3://metabase-maven-downloads/com/metabase/athena-jdbc/${{ github.event.inputs.version }}/
echo "Athena JDBC driver version ${{ github.event.inputs.version }} has been deployed"

View File

@@ -23,7 +23,7 @@ jobs:
uses: actions/github-script@v7
with:
github-token: ${{ steps.generate-token.outputs.token }}
script: | #js
script: |
const fs = require('fs');
const teamConfig = JSON.parse(fs.readFileSync('.github/team.json', 'utf-8'));
const prAuthor = context.payload.pull_request.user.login;
@@ -65,3 +65,10 @@ jobs:
issue_number: prNumber,
labels: [team.label]
});
return team.projectUrl || null;
- uses: actions/add-to-project@v0.5.0
if: ${{ steps.auto-assign.outputs.result != 'null' }}
with:
project-url: ${{ steps.auto-assign.outputs.result }}
github-token: ${{ steps.generate-token.outputs.token }}

View File

@@ -1,24 +1,18 @@
name: Auto-backport
run-name: Auto Backport ${{ github.event.pull_request.title }}
on:
pull_request:
types: [closed, labeled]
jobs:
create-backport:
if:
| # the PR must have a backport label when it's merged, or be labeled with a backport label after merge
github.event.pull_request.merged == true && (
github.event.action == 'closed' && (
contains(github.event.pull_request.labels.*.name, 'backport') ||
contains(github.event.pull_request.labels.*.name, 'double-backport') ||
contains(github.event.pull_request.labels.*.name, 'triple-backport')
) ||
github.event.action == 'labeled' && (
github.event.label.name == 'backport' ||
github.event.label.name == 'double-backport' ||
github.event.label.name == 'triple-backport'
)
if: | # the PR must be closed and have a backport label
(github.event.pull_request.merged == true || github.event.action == 'closed') && (
contains(github.event.pull_request.labels.*.name, 'backport') ||
contains(github.event.pull_request.labels.*.name, 'double-backport') ||
contains(github.event.pull_request.labels.*.name, 'triple-backport') ||
github.event.label.name == 'backport' ||
github.event.label.name == 'double-backport' ||
github.event.label.name == 'triple-backport'
)
runs-on: ubuntu-latest
steps:
@@ -82,7 +76,7 @@ jobs:
github-token-2: ${{ secrets.GITHUB_TOKEN }}
- name: Double Backport
if: ${{ always() && fromJson(steps.how-many-backports.outputs.result) >= 2 }}
if: ${{ fromJson(steps.how-many-backports.outputs.result) >= 2 }}
uses: ./.github/actions/create-backport
with:
target-version: ${{ fromJson(steps.get-latest-release-version.outputs.result).secondLatest }}
@@ -91,7 +85,7 @@ jobs:
github-token-2: ${{ secrets.GITHUB_TOKEN }}
- name: Triple Backport
if: ${{ always() && fromJson(steps.how-many-backports.outputs.result) >= 3 }}
if: ${{ fromJson(steps.how-many-backports.outputs.result) >= 3 }}
uses: ./.github/actions/create-backport
with:
target-version: ${{ fromJson(steps.get-latest-release-version.outputs.result).thirdLatest }}

View File

@@ -0,0 +1,21 @@
# Config is in .github/auto_request_review.yml
name: Auto Request Review
on:
pull_request:
types: [opened, ready_for_review, reopened, synchronize]
jobs:
auto-request-review:
# Don't run on forked repos
if: ${{ !github.event.pull_request.head.repo.fork }}
permissions:
contents: read
pull-requests: write
name: Auto Request Review
runs-on: ubuntu-latest
steps:
- name: Request review based on files changes and/or groups the author belongs to
uses: necojackarc/auto-request-review@v0.13.0
with:
token: '${{ secrets.METABASE_AUTOMATION_USER_TOKEN }}'

View File

@@ -10,202 +10,19 @@ jobs:
# don't run this workflow on a cron for forks
if: ${{ github.event_name != 'schedule' || github.repository == 'metabase/metabase' }}
runs-on: ubuntu-22.04
timeout-minutes: 30
strategy:
fail-fast: false
matrix:
modules:
- actions
- activity-feed
- analytics
- analyze
- api
- api-keys
- api-routes
- app-db
- appearance
- audit-app
- auth-provider
- batch-processing
- bookmarks
- bug-reporting
- cache
- channel
- classloader
- cloud-migration
- cmd
- collections
- config
- connection-pool
- content-translation
- content-verification
- core
- dashboards
- database-routing
- driver
- driver-api
- eid-translation
- embedding
- events
- formatter
- geojson
- glossary
- graph
- indexed-entities
- internal-stats
- legacy-mbql
- lib
- lib-be
- logger
- login-history
- model-persistence
- models
- native-query-snippets
- notification
- parameters
- permissions
- pivot
- plugins
- premium-features
- product-feedback
- public-sharing
- pulse
- queries
- query-permissions
- query-processor
- remote-sync
- request
- revisions
- sample-data
- search
- secrets
- segments
- server
- session
- settings
- setup
- sso
- startup
- store-api
- sync
- system
- task
- task-history
- testing-api
- tiles
- timeline
- types
- upload
- user-key-value
- users
- util
- version
- view-log
- warehouse-schema
- warehouses
- xrays
timeout-minutes: 90
steps:
- uses: actions/checkout@v4
- name: Prepare front-end environment
uses: ./.github/actions/prepare-frontend
- name: Prepare back-end environment
uses: ./.github/actions/prepare-backend
with:
m2-cache-key: "cloverage"
- name: Build static viz frontend
run: yarn build-static-viz
env:
MB_EDITION: ee
- name: Collect the test coverage
run: |
clojure -X:dev:ci:ee:ee-dev:test:cloverage \
:ns-regex '["^metabase\\.${{ matrix.modules }}\\..*"]' \
:test-ns-path '["test/metabase/${{ matrix.modules }}"]'
continue-on-error: true
run: clojure -X:dev:ci:ee:ee-dev:test:cloverage
- name: Upload coverage to codecov.io
if: always()
uses: codecov/codecov-action@v5
with:
token: ${{ secrets.CODECOV_TOKEN }}
files: ./target/coverage/codecov.json
flags: back-end
be-linter-cloverage-enterprise:
# don't run this workflow on a cron for forks
if: ${{ github.event_name != 'schedule' || github.repository == 'metabase/metabase' }}
runs-on: ubuntu-22.04
timeout-minutes: 30
strategy:
fail-fast: false
matrix:
modules:
- action-v2
- advanced-config
- advanced-permissions
- ai-entity-analysis
- ai-sql-fixer
- ai-sql-generation
- analytics
- api
- audit-app
- auth-provider
- billing
- bookmarks
- cache
- cloud-add-ons
- comments
- content-translation
- content-verification
- core
- dashboard-subscription-filters
- database-replication
- database-routing
- dependencies
- documents
- email
- embedding-hub
- gsheets
- harbormaster
- impersonation
- internal-stats
- llm
- metabot-v3
- permission-debug
- premium-features
- public-sharing
- remote-sync
- sandbox
- scim
- search
- semantic-search
- serialization
- snippet-collections
- sso
- stale
- transforms
- transforms-python
- upload-management
steps:
- uses: actions/checkout@v4
- name: Prepare front-end environment
uses: ./.github/actions/prepare-frontend
- name: Prepare back-end environment
uses: ./.github/actions/prepare-backend
- name: Build static viz frontend
run: yarn build-static-viz
env:
MB_EDITION: ee
- name: Collect the test coverage
run: |
clojure -X:dev:ci:ee:ee-dev:test:cloverage \
:ns-regex '["^metabase-enterprise\\.${{ matrix.modules }}\\..*"]' \
:test-ns-path '["enterprise/backend/test/metabase/${{ matrix.modules }}"]'
continue-on-error: true
- name: Upload coverage to codecov.io
if: always()
uses: codecov/codecov-action@v5
uses: codecov/codecov-action@v4
with:
token: ${{ secrets.CODECOV_TOKEN }}
files: ./target/coverage/codecov.json

View File

@@ -21,8 +21,10 @@ jobs:
- uses: actions/checkout@v4
- name: Prepare back-end environment
uses: ./.github/actions/prepare-backend
with:
m2-cache-key: 'kondo'
- name: Run clj-kondo
run: ./bin/mage kondo
run: ./bin/kondo.sh
be-linter-eastwood:
if: ${{ !inputs.skip }}
@@ -33,9 +35,32 @@ jobs:
- uses: actions/checkout@v4
- name: Prepare back-end environment
uses: ./.github/actions/prepare-backend
with:
m2-cache-key: "eastwood"
- run: clojure -X:dev:ee:ee-dev:drivers:drivers-dev:test:eastwood
name: Run Eastwood linter
mage-tests:
if: ${{ !inputs.skip }}
runs-on: ubuntu-22.04
name: Mage
steps:
- uses: actions/checkout@v4
- name: Prepare back-end environment
uses: ./.github/actions/prepare-backend
with:
m2-cache-key: 'kondo'
- name: Setup Babashka
uses: turtlequeue/setup-babashka@v1.7.0
with:
babashka-version: 1.12.196
- name: Check bb runs
run: bb --version
- name: bb task list
run: bb tasks
- name: Run mage tests
run: bb ./test/mage/core_test.clj
be-tests:
if: ${{ !inputs.skip }}
runs-on: ubuntu-22.04
@@ -47,13 +72,11 @@ jobs:
job:
- name: Enterprise Tests
edition: ee
test-group-name: ee
build-static-viz: 'false'
test-args: >-
:only '"enterprise/backend/test"'
- name: EE App DB Tests (Part 1)
edition: ee
test-group-name: ee-app-db-1
build-static-viz: 'true'
test-args: >-
:only '["test" ".clj-kondo/test"]'
@@ -61,7 +84,6 @@ jobs:
:partition/index 0
- name: EE App DB Tests (Part 2)
edition: ee
test-group-name: ee-app-db-2
build-static-viz: 'true'
test-args: >-
:only '["test" ".clj-kondo/test"]'
@@ -69,7 +91,6 @@ jobs:
:partition/index 1
- name: OSS App DB Tests (Part 1)
edition: oss
test-group-name: oss-app-db-1
build-static-viz: 'true'
test-args: >-
:only '["test" ".clj-kondo/test"]'
@@ -77,7 +98,6 @@ jobs:
:partition/index 0
- name: OSS App DB Tests (Part 2)
edition: oss
test-group-name: oss-app-db-2
build-static-viz: 'true'
test-args: >-
:only '["test" ".clj-kondo/test"]'
@@ -108,14 +128,13 @@ jobs:
- name: Prepare back-end environment
uses: ./.github/actions/prepare-backend
with:
m2-cache-key: be-tests
java-version: "${{ matrix.java-version }}"
# Depending on which files changed, we might either have to build static-viz
# from scratch or download the previously built artifact
- name: Build static-viz frontend
if: ${{ matrix.job.build-static-viz == 'true' && !inputs.skip }}
run: yarn build-static-viz
env:
MB_EDITION: ${{ matrix.job.edition }}
- name: Download Static Viz Bundle Artifact
if: ${{ matrix.job.build-static-viz == 'true' && !inputs.skip }}
uses: actions/download-artifact@v4
@@ -129,10 +148,6 @@ jobs:
clojure -X:dev:ci:test:${{ matrix.job.edition }}:${{ matrix.job.edition }}-dev
:exclude-tags '[:mb/driver-tests ${{ env.__ADDITIONAL_EXCLUDED_TAG__ }}]'
${{ matrix.job.test-args }}
# Trunk Quarantine controls the final state of the run in the CI!
# Even if this step fails we let it pass, and then `trunk-io/analytics-uploader` determines
# whether to fail or to pass the whole job, based on the quarantine list.
continue-on-error: true
- name: Upload Test Results
uses: ./.github/actions/upload-test-results
@@ -157,14 +172,6 @@ jobs:
list-tests: failed
fail-on-error: false
- name: Upload Logs on Failure
if: always() && !success()
uses: actions/upload-artifact@v4
with:
name: test-logs-${{ matrix.java-version }}-${{ matrix.job.edition }}-${{ github.job }}-${{ matrix.job.test-group-name }}
path: logs/test-log.json
retention-days: 1
# checks that all the namespaces we actually ship can be compiled, without any dependencies that we don't ship (such
# as `:dev` dependencies). See #27009 for more context.
be-check:
@@ -179,6 +186,8 @@ jobs:
- uses: actions/checkout@v4
- name: Prepare backend
uses: ./.github/actions/prepare-backend
with:
m2-cache-key: be-check
# TODO -- we should probably also check WITHOUT ee -- do a check with only OSS namespaces on the classpath
- name: Check namespaces
run: clojure -M:ee:drivers:check
@@ -192,8 +201,10 @@ jobs:
- uses: actions/checkout@v4
- name: Prepare back-end environment
uses: ./.github/actions/prepare-backend
with:
m2-cache-key: 'cljfmt'
- name: Run cljfmt
run: ./bin/mage cljfmt-all --force-check
run: clojure -T:cljfmt check
be-tests-result:
needs:

View File

@@ -5,7 +5,6 @@ on:
types: [auto_merge_enabled, review_requested, labeled, unlabeled, synchronize]
branches:
- "master"
merge_group:
jobs:
backport-reminder:

View File

@@ -1,14 +0,0 @@
name: Block Master Branch PRs
on:
pull_request:
types: [opened, synchronize]
jobs:
check-source-branch:
runs-on: ubuntu-latest
steps:
- name: Block master branch PRs
if: github.head_ref == 'master'
run: |
echo "PRs from `master` branches are not allowed, please use a new branch name for your PR."
exit 1

View File

@@ -0,0 +1,52 @@
name: Bugs Slack Notification
on:
issues:
types:
- labeled
jobs:
on-issue-labeled:
runs-on: ubuntu-22.04
timeout-minutes: 5
if: |
github.event.label.name == '.Needs Triage' && contains(github.event.issue.labels.*.name, 'Type:Bug')
steps:
- name: Setting title
uses: actions/github-script@v7
id: vars
with:
script: |
core.setOutput('issue_title', ${{ toJson(github.event.issue.title) }}.replaceAll(/"/g, '\\"'));
- name: Check for labels containing team name
id: check_labels
run: |
labels_json=$(sed 's/&/\$/g' <<< '${{ toJson(github.event.issue.labels) }}')
echo "has_team_label=false" >> "$GITHUB_OUTPUT"
for label in "$labels_json"
do
if [[ $label == *".Team"* ]]; then
echo "Label contains the team name."
echo "has_team_label=true" >> "$GITHUB_OUTPUT"
break
fi
done
- name: Send the issue ${{ github.event.issue.number }} to Slack
id: slack
if: steps.check_labels.outputs.has_team_label == 'false'
uses: slackapi/slack-github-action@v1.26.0
with:
payload: |
{
"blocks": [
{
"type": "section",
"text": {
"type": "mrkdwn",
"text": "${{ steps.vars.outputs.issue_title }}\n${{ github.event.issue.html_url }}"
}
}
]
}
env:
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_BUG_TRIAGE_WEBHOOK_URL }}
SLACK_WEBHOOK_TYPE: INCOMING_WEBHOOK

View File

@@ -16,6 +16,8 @@ jobs:
- name: Prepare back-end environment
timeout-minutes: 5
uses: ./.github/actions/prepare-backend
with:
m2-cache-key: 'build-scripts'
- name: Run build and release script tests
run: clojure -X:dev:drivers:build:build-dev:build-test:ci

View File

@@ -1,107 +0,0 @@
# This workflow is the single source of truth for generating and updating various caches used by our CI pipeline.
# It sets the fresh cache each Monday but provides a manual trigger to update it on-demand.
#
# All other jobs only restore from this cache; *they never update it*!
# They should always use `actions/cache/restore@v4` because `actions/cache@v4` would attempt to save the cache.
#
# Our agreed-upon strategy is to use a deterministic cache key based on the week of the year.
# Jobs can utilize the prior week's cache if the current week's cache is not found.
#
#
# PRINCIPLES:
# - The cache is never wrong!
# - Caches are immutable once created.
# - Caches are for speeding up builds, not for correctness.
# - It is faster to download a large cache than to re-resolve missing dependencies.
#
#
# DESIGN:
#
# 1. Prepare
# - A job first prepares the local environment (e.g., installs JDK, Clojure, Node.js, etc.)
# - This fetches the dependencies from the cache (if it exists) and warms up the local cache for this workflow.
# 2. Resolve
# - The next step pre-resolves all dependencies, ensuring that the local cache is fully populated and up-to-date.
# 3. Update
# - The final step saves the local cache back to GitHub Actions cache with a deterministic key.
# - If a cache entry already exists for that key, it is deleted first to ensure immutability.
#
name: Cache Generator
on:
schedule:
- cron: '0 3 * * 1' # Mondays at 03:00 UTC
workflow_dispatch: # manually update the cache
concurrency:
group: cache-generator
cancel-in-progress: true
jobs:
get-cache-keys:
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v4
- name: Get cache keys
uses: ./.github/actions/get-cache-keys
id: get-cache-keys
outputs:
m2-cache-key: ${{ steps.get-cache-keys.outputs.m2-cache-key }}
node-cache-key: ${{ steps.get-cache-keys.outputs.node-cache-key }}
eslint-cache-key: ${{ steps.get-cache-keys.outputs.eslint-cache-key }}
generate-m2-cache:
runs-on: ubuntu-22.04
timeout-minutes: 15
needs: get-cache-keys
env:
M2_CACHE_KEY: ${{ needs.get-cache-keys.outputs.m2-cache-key }}
steps:
- uses: actions/checkout@v4
- name: Prepare backend
uses: ./.github/actions/prepare-backend
# Add or remove aliases to affect the cache size and its contents.
# Adjust to taste, then redeploy this workflow.
- name: Pre-resolve dependencies
run: clojure -A:build:dev:ee:ee-dev:drivers:drivers-dev:cljs -P
- name: Update M2 cache
uses: ./.github/actions/update-cache
with:
path: |
~/.m2
~/.gitlibs
key: ${{ env.M2_CACHE_KEY }}
generate-frontend-caches:
runs-on: ubuntu-22.04
timeout-minutes: 15
needs: get-cache-keys
env:
NODE_CACHE_KEY: ${{ needs.get-cache-keys.outputs.node-cache-key }}
ESLINT_CACHE_KEY: ${{ needs.get-cache-keys.outputs.eslint-cache-key }}
steps:
- uses: actions/checkout@v4
# `prepare-frontend` action (1) prepares Node.js and (2) fetches the node_modules cache if it exists.
# It then (3) runs `yarn install --frozen-lockfile --prefer-offline` which ensures the local cache is fully populated.
# IMPORTANT: Caching node_modules after patches are already applied will break any PR that attempts to either:
# - upgrade a patched library
# - change a patch
# Hence why we explicitly exclude patches from the cache (controlled by the `apply-patches` action input).
- name: Install front-end dependencies
uses: ./.github/actions/prepare-frontend
with:
apply-patches: 'false'
- name: Update node_modules cache
uses: ./.github/actions/update-cache
with:
path: node_modules
key: ${{ env.NODE_CACHE_KEY }}
- name: Update eslint cache
uses: ./.github/actions/update-cache
with:
path: .eslintcache
key: ${{ env.ESLINT_CACHE_KEY }}

Some files were not shown because too many files have changed in this diff Show More