Compare commits

..

600 Commits

Author SHA1 Message Date
Nicola Mometto
3b49e17e43 support bigints 2025-09-23 15:32:06 +01:00
Romeo Van Snick
4ca3759f4a Make transforms-python it's own plugin 2025-09-23 16:28:15 +10:00
Romeo Van Snick
65af07681e Use isResourceNotFoundError 2025-09-23 15:34:58 +10:00
Romeo Van Snick
7218316f22 Check type in isPythonTransform assertion 2025-09-23 15:32:33 +10:00
Romeo Van Snick
bf647fe6d9 Return alias instead of throwing 2025-09-23 15:30:54 +10:00
Romeo Van Snick
c07520cc93 Remove token feature gate in enterprise folder 2025-09-23 15:29:59 +10:00
Romeo Van Snick
a45ac1119b Use AnsiLogs in LogOutput component 2025-09-23 15:28:07 +10:00
Romeo Van Snick
58d8a20c5b Use AnsiLogs in Logs components 2025-09-23 15:27:48 +10:00
Romeo Van Snick
9c23b6f68b Add AnsiLogs component 2025-09-23 15:27:36 +10:00
Romeo Van Snick
5e42728ca2 Translate pandas definitions 2025-09-23 13:25:45 +10:00
Romeo Van Snick
585b73d250 Add python runner settings in CI 2025-09-22 22:55:23 +10:00
Romeo Van Snick
3d3680e8e9 Fix e2e test helper 2025-09-22 22:34:43 +10:00
Romeo Van Snick
92a2e083ec Fix Revert on python library when it does not exist 2025-09-22 21:37:31 +10:00
Romeo Van Snick
c1ebb1882c Add check for python output 2025-09-22 21:32:53 +10:00
Romeo Van Snick
b8208dc345 Add helper to update Python library 2025-09-22 21:25:22 +10:00
Romeo Van Snick
9fc3a644c0 Tag tests for python transforms with @transforms-python 2025-09-22 21:25:22 +10:00
Dan Stone
e03713fd9d Fix private 2025-09-22 12:05:20 +01:00
Dan Stone
cf9d57e584 Fix missing require 2025-09-22 12:05:06 +01:00
Dan Stone
05464a05f7 Remove dead code 2025-09-22 12:03:04 +01:00
Dan Stone
37c2c41b14 Merge branch 'proj-python/feature-branch' into proj-python/s01e04
# Conflicts:
#	enterprise/backend/src/metabase_enterprise/transforms/api.clj
#	enterprise/backend/src/metabase_enterprise/transforms/canceling.clj
#	enterprise/backend/src/metabase_enterprise/transforms/jobs.clj
#	enterprise/backend/src/metabase_enterprise/transforms/ordering.clj
#	enterprise/backend/src/metabase_enterprise/transforms/util.clj
#	enterprise/backend/test/metabase_enterprise/transforms/api_test.clj
#	enterprise/backend/test/metabase_enterprise/transforms/test_util.clj
2025-09-22 11:42:25 +01:00
Ngoc Khuat
473e07809b Transforms python APIs (#63762)
* transforms python api

* new tests pass locally

* clean up the db id utils

* move to util

* fix: missing refer

* chore: missing tests/kondo

* kondo

* more kondo

* fix: make function resilient

* eh, validate on old

* on new

* Add python transform support for jobs (#63764)

* Add python transforms job support

* Fix whitespace in when

* Avoid churn due to new local

* Add feature test

* Add ignore for logs (there might be away to avoid this)

* Use new target-database-id util

* Remove duplicated fn

---------

Co-authored-by: Nicola Mometto <nicola.mometto@metabase.com>
Co-authored-by: Dan Stone <wotbrew@gmail.com>
2025-09-22 12:12:48 +02:00
Romeo Van Snick
160d867260 Do not include python-runner in test docker-compose yet 2025-09-22 19:24:10 +10:00
Romeo Van Snick
5a5fde8a54 Fix e2e tests 2025-09-22 18:56:29 +10:00
Romeo Van Snick
3d767af895 Add transforms-python feature 2025-09-22 18:30:25 +10:00
Romeo Van Snick
83ed774cbc Use useHasTokenFeature hook where possible 2025-09-22 18:24:10 +10:00
Romeo Van Snick
8936750aa5 Hide Python transforms features when the user does not have transforms-python token 2025-09-22 16:55:46 +10:00
Romeo Van Snick
8bd2d83ae8 Do not show Python library link when the user does not have python transforms support 2025-09-22 16:03:06 +10:00
Romeo Van Snick
c7b47f276f Move Python runner amdin page to transforms plugin 2025-09-22 16:01:14 +10:00
Romeo Van Snick
96ecb29881 Remove unused variables 2025-09-22 15:47:17 +10:00
Romeo Van Snick
715e082a94 Hide python runner admin route when user does not have the feature for it 2025-09-22 14:58:12 +10:00
Nicola Mometto
92e8270aa1 Merge remote-tracking branch 'origin/proj-python/feature-branch' into proj-python/s01e04 2025-09-19 16:42:54 +01:00
Ngoc Khuat
57618b5245 execute.clj (#63755)
* execute.clj

* with drivers test and execute test

* with wait tables

* missing utils

* /s/open/start

* fix failing tests due to getting the wrong source and target

* kondo

* move cancel-python-code-http-call to python-runner

* Dan's feedback

* we have 2 implementstation of cancel

* and 2 db-routing-enabled?

---------

Co-authored-by: Nicola Mometto <nicola.mometto@metabase.com>
2025-09-19 17:41:43 +02:00
Nicola Mometto
2a3f25b6c0 Merge branch 'python-transform-execute' into proj-python/s01e04 2025-09-19 16:15:14 +01:00
Ngoc Khuat
ca2d010f96 we have 2 implementstation of cancel 2025-09-19 22:09:22 +07:00
Ngoc Khuat
f43429605d Dan's feedback 2025-09-19 22:08:30 +07:00
Nicola Mometto
3bdb8b3745 move cancel-python-code-http-call to python-runner 2025-09-19 16:07:48 +01:00
Nicola Mometto
0bd78f3d99 kondo 2025-09-19 16:04:39 +01:00
Ngoc Khuat
e9c82648a9 fix failing tests due to getting the wrong source and target 2025-09-19 21:52:00 +07:00
Ngoc Khuat
d3c66a05a7 /s/open/start 2025-09-19 21:41:29 +07:00
Ngoc Khuat
98fdefb3af missing utils 2025-09-19 21:40:31 +07:00
Ngoc Khuat
5829f65cc8 with wait tables 2025-09-19 21:35:15 +07:00
Ngoc Khuat
e2a8599507 with drivers test and execute test 2025-09-19 21:30:50 +07:00
Ngoc Khuat
45b432db7f execute.clj 2025-09-19 21:18:55 +07:00
Nicola Mometto
2d2b6826af Merge remote-tracking branch 'origin/proj-python/feature-branch' into proj-python/s01e04 2025-09-19 15:09:20 +01:00
Nicola Mometto
6e39a2451e Merge branch 'proj-python/runner' into proj-python/s01e04 2025-09-19 15:05:35 +01:00
Dan Stone
13b011b48b Add python_runner.clj (#63747)
* Add python runner

* Kondo

* Traceback fix

* Remove dead code

* start the runner by default

* Cancellation not used yet

* Remove unused keys from fields-metadata

* cherry pick driver fixes

* start for postgres and mysql as well

* clickhouse fixes

* relax for snowflake

* typo

* exclude transform-python-test from appdb

* exclude transforms-python-test as well

* try exclude-tag here too

---------

Co-authored-by: Ngoc Khuat <qn.khuat@gmail.com>
Co-authored-by: Nicola Mometto <nicola.mometto@metabase.com>
2025-09-19 15:59:41 +02:00
Nicola Mometto
aa914db7c7 disable flake 2025-09-19 14:48:12 +01:00
Nicola Mometto
81657554bc try exclude-tag here too 2025-09-19 14:38:49 +01:00
Ngoc Khuat
007e322139 exclude transforms-python-test as well 2025-09-19 20:27:09 +07:00
Nicola Mometto
894b446311 further relax 2025-09-19 14:14:09 +01:00
Nicola Mometto
e2ffa4481a exclude transform-python-test from appdb 2025-09-19 13:58:47 +01:00
Nicola Mometto
fee9768887 typo 2025-09-19 13:57:45 +01:00
Nicola Mometto
fbf7b96655 typo 2025-09-19 13:56:59 +01:00
Nicola Mometto
a38b97c894 fix 2025-09-19 13:48:11 +01:00
Nicola Mometto
1b45e41bf4 relax for snowflake 2025-09-19 13:34:54 +01:00
Nicola Mometto
f909d81aaf relax for snowflake 2025-09-19 13:34:38 +01:00
Nicola Mometto
3808c9e403 clickhouse fixes 2025-09-19 13:21:26 +01:00
Nicola Mometto
37ae7feec5 clickhouse fixes 2025-09-19 13:18:11 +01:00
Ngoc Khuat
3c35fff1b3 start for postgres and mysql as well 2025-09-19 19:14:27 +07:00
Nicola Mometto
5b9f005c94 order 2025-09-19 13:01:13 +01:00
Nicola Mometto
4027e703bb cherry pick driver fixes 2025-09-19 13:00:50 +01:00
Nicola Mometto
ae341c30e8 Merge remote-tracking branch 'origin/proj-python/feature-branch' into proj-python/s01e04 2025-09-19 12:56:17 +01:00
Nicola Mometto
f21066e627 fix for redshift 2025-09-19 12:55:35 +01:00
Dan Stone
b78c9d0322 Remove unused keys from fields-metadata 2025-09-19 12:49:28 +01:00
Dan Stone
efd2076773 Cancellation not used yet 2025-09-19 12:46:56 +01:00
Nicola Mometto
2e9b7dfc04 bump 2025-09-19 12:44:12 +01:00
Nicola Mometto
743d6f2ce5 fix 2025-09-19 12:43:08 +01:00
Ngoc Khuat
bf4382bccf start the runner by default 2025-09-19 18:31:41 +07:00
Dan Stone
460e29f0be Remove dead code 2025-09-19 12:10:41 +01:00
Dan Stone
8a3eb3e46c Traceback fix 2025-09-19 12:09:20 +01:00
Nicola Mometto
8b81aaef89 fix test 2025-09-19 12:00:46 +01:00
Dan Stone
357d18f987 Kondo 2025-09-19 11:57:46 +01:00
Dan Stone
f3282ed482 Add python runner 2025-09-19 11:57:46 +01:00
Ngoc Khuat
a350606c67 split CI into separate jobs (#63746) 2025-09-19 11:53:38 +01:00
Nicola Mometto
620d15cfa8 fixes for clickhouse 2025-09-19 11:51:56 +01:00
Ngoc Khuat
98df5a72fd Transforms python tests with its own CI (#63744) 2025-09-19 15:57:42 +07:00
Romeo Van Snick
7bcbc93848 Avoid starting the container in e2e tests twice 2025-09-19 18:33:29 +10:00
Romeo Van Snick
ad553839b7 Clone python runner when running CI 2025-09-19 18:22:23 +10:00
Dan Stone
68e7486947 Reorder the final message-log flush to occur before the succeed call
Should fix issues in tests that block on until a 'terminal' state, log output could be late and cause the test to fail.
2025-09-19 09:19:21 +01:00
Romeo Van Snick
a15d6eeec8 Fix some e2e tests 2025-09-19 16:42:16 +10:00
Romeo Van Snick
eab7a7a23e Further adjust to design file 2025-09-19 16:28:52 +10:00
Romeo Van Snick
1171f638a9 Implement data picker according to designs 2025-09-19 16:28:47 +10:00
Romeo Van Snick
409391e434 Simpify new transform page structure 2025-09-19 16:28:15 +10:00
Romeo Van Snick
eeb7c748a0 Fix new python transform page nesting 2025-09-19 16:28:13 +10:00
Romeo Van Snick
ee2b0e47fb Move draft back to new python transform page 2025-09-19 16:28:10 +10:00
Romeo Van Snick
fd9ba52e7c Remove extension 2025-09-19 16:28:08 +10:00
Romeo Van Snick
772abb574f Fix import 2025-09-19 16:28:04 +10:00
Romeo Van Snick
8716c6763a Add border to empty results 2025-09-19 16:28:01 +10:00
Romeo Van Snick
f4d559099f Remove unused props 2025-09-19 16:27:59 +10:00
Romeo Van Snick
e51b465624 Move AliasInput to separate file 2025-09-19 16:27:57 +10:00
Romeo Van Snick
572674ee4d Simplify props 2025-09-19 16:27:51 +10:00
Romeo Van Snick
4273556b67 Remove TableOption type 2025-09-19 16:27:49 +10:00
Romeo Van Snick
8b311b71ad Move TableInput to separate file 2025-09-19 16:27:47 +10:00
Romeo Van Snick
b9a69ace7e Validate source more 2025-09-19 16:27:46 +10:00
Romeo Van Snick
b33910f807 Remove unused component 2025-09-19 16:27:43 +10:00
Romeo Van Snick
4130322ad7 Remove vibe-comment 2025-09-19 16:27:40 +10:00
Romeo Van Snick
8a262ba858 Remove api transforms 2025-09-19 16:27:34 +10:00
Romeo Van Snick
d9a052ff23 Move library types to metabase-types/api 2025-09-19 16:27:33 +10:00
Romeo Van Snick
5ae6151a15 Remove source/draft ambiguity in types 2025-09-19 16:27:30 +10:00
Romeo Van Snick
2f91103e4d Use DatabaseId in python transform 2025-09-19 16:27:27 +10:00
Nicola Mometto
c2645a07a3 Merge remote-tracking branch 'origin/proj-python/feature-branch' into proj-python/s01e04 2025-09-18 23:46:49 +01:00
Nicola Mometto
cfb45c4ddf port prometheus/instrumentation (#63631)
* port prometheus metrics

* port instrumentation

* instrument existing transform paths

* wider buckets

* remove nil handling

* warn on nil bytes/rows

* add stage-type mapping

* typo
2025-09-18 22:05:45 +00:00
Nicola Mometto
4b1c3e50a2 eh, that was worse, we're probably nondeterministic somewhere, TODO 2025-09-18 21:53:35 +01:00
Nicola Mometto
559727b0fa fix: order 2025-09-18 21:13:54 +01:00
Nicola Mometto
4ca3c253be Merge remote-tracking branch 'origin/proj-python/feature-branch' into proj-python/s01e04 2025-09-18 20:40:49 +01:00
Nicola Mometto
413c50b888 shorter 2025-09-18 20:19:15 +01:00
Chris Truter
25c66f6028 Add s3 dependencies (#63706) 2025-09-18 19:14:43 +00:00
Nicola Mometto
61ab46858c shorter name 2025-09-18 20:00:05 +01:00
Nicola Mometto
ac0945a0f2 less fancy for maria 2025-09-18 19:51:29 +01:00
Nicola Mometto
82704fd83e switch to wait-for-table 2025-09-18 19:27:22 +01:00
Nicola Mometto
d6d52b002a wait for fields 2025-09-18 19:25:38 +01:00
Nicola Mometto
2742939e11 remove exotic maria type not in CI 2025-09-18 19:20:22 +01:00
Nicola Mometto
14ac84fd36 fix yet another test that never worked 2025-09-18 18:55:51 +01:00
Nicola Mometto
39c99824aa fix broken test 2025-09-18 18:50:36 +01:00
Nicola Mometto
d33886138a fix: proper sleep 2025-09-18 18:38:08 +01:00
Nicola Mometto
fa388420fa fix: missing feature 2025-09-18 18:34:40 +01:00
Nicola Mometto
bc829a898d fix 2025-09-18 18:16:35 +01:00
Nicola Mometto
6713ee3e63 kondo 2025-09-18 18:05:17 +01:00
Nicola Mometto
e216611038 disable h2 here too 2025-09-18 18:01:32 +01:00
Nicola Mometto
c1033e494d log/warnf is a macro, we can't with-redefs. how did this test ever work? 2025-09-18 17:59:57 +01:00
Chris Truter
be695dff40 Merge branch 'merge-s3-fixes' into proj-python/s01e04 2025-09-18 18:52:07 +02:00
Chris Truter
a79cbf4662 Remove stray stub 2025-09-18 18:51:25 +02:00
Nicola Mometto
420631e095 h2 ain't enabled for now 2025-09-18 17:50:30 +01:00
Chris Truter
6615d5ba1a Merge branch 'add-s3-deps' into proj-python/s01e04 2025-09-18 18:49:25 +02:00
Chris Truter
e4d7bb13e0 Disable s3 tests when run in the wrong group 2025-09-18 18:46:07 +02:00
Nicola Mometto
d8ae17ac38 fix case 2025-09-18 17:41:04 +01:00
Chris Truter
98bf85bae6 Simplify control flow a bit 2025-09-18 18:35:29 +02:00
Chris Truter
847be6f69c Remove rich comments 2025-09-18 18:34:41 +02:00
Nicola Mometto
c503083864 missed 2025-09-18 17:32:15 +01:00
Nicola Mometto
d12c7d40b5 fix test 2025-09-18 17:24:16 +01:00
Nicola Mometto
c4cb1b88ae comment out postgres driver tests during appddb workflow 2025-09-18 17:20:42 +01:00
Nicola Mometto
72463b8b6d Merge remote-tracking branch 'origin/add-s3-deps' into proj-python/s01e04 2025-09-18 17:20:28 +01:00
Nicola Mometto
831eb23131 typo 2025-09-18 17:16:53 +01:00
Nicola Mometto
3dae67b634 linter fixes 2025-09-18 16:57:41 +01:00
Nicola Mometto
a7af06ac42 tests are passing 2025-09-18 16:55:18 +01:00
Nicola Mometto
fccba413b0 fix driver matrix 2025-09-18 16:37:16 +01:00
Nicola Mometto
4ff9edeb28 pull out type restriction helper 2025-09-18 16:30:28 +01:00
Nicola Mometto
970535439c add stage-type mapping 2025-09-18 16:30:27 +01:00
Nicola Mometto
f9ca2ad6a9 warn on nil bytes/rows 2025-09-18 16:30:06 +01:00
Nicola Mometto
2f46305503 remove nil handling 2025-09-18 16:30:06 +01:00
Nicola Mometto
0886e7eaca wider buckets 2025-09-18 16:30:06 +01:00
Nicola Mometto
60504bdcc9 instrument existing transform paths 2025-09-18 16:30:04 +01:00
Dan Stone
4c39b11d08 Fix SocketException handling on interrupt during logging (#63705) 2025-09-18 16:28:22 +01:00
Chris Truter
1a40bac681 Make s3 a driver test to get localstack 2025-09-18 16:57:48 +02:00
Chris Truter
be815dedd1 Add dependencies 2025-09-18 16:45:29 +02:00
Nicola Mometto
5aa5c9206f chore: update drivers matrix to type restriction 2025-09-18 15:24:00 +01:00
Chris Truter
1b95cd0ddd Merge branch 'proj-python/feature-branch' into proj-python/s01e04 2025-09-18 16:22:52 +02:00
Chris Truter
57e68d2a62 S3 transport layer for python-runner (#63697) 2025-09-18 15:02:57 +01:00
Ngoc Khuat
a19b3a46da setup ci properly (#63693) 2025-09-18 15:02:57 +01:00
Nicola Mometto
90eb7e53cd cherry pick driver code (#63628)
* cherry pick driver code

* update changelog

* add type->database-type unit tests

* kondo

* create/insert table

* insert-from-jsonl test

* missing rename-tables for snowflake

* extend rename-tables-test to more drivers

* line breaks

* add driver python transform feature

* handle review comments

* kondo

* feat: replace string->val with a more generic insert-col->val

* remove non transactional impls

* chore: docsstrings

* TODO

* extra test

* fix extra test

* test all base-types

* fix sqls
2025-09-18 15:02:57 +01:00
Nicola Mometto
eecc0b9a10 Merge remote-tracking branch 'origin/master' into proj-python/s01e04 2025-09-18 14:51:49 +01:00
Nicola Mometto
1b5e427878 cherry pick changes from feature branch 2025-09-18 14:03:15 +01:00
Ngoc Khuat
3d1682e966 spin python runner for postgres app db tests as well 2025-09-18 19:53:37 +07:00
Ngoc Khuat
4fea38de35 fix drivers ci 2025-09-18 19:50:00 +07:00
Chris Truter
554890a168 Merge branch 'merge-s3-changes' into proj-python/s01e04 2025-09-18 14:49:20 +02:00
Chris Truter
403623894f Merge branch 's3-helpers' into proj-python/s01e04 2025-09-18 14:48:01 +02:00
Ngoc Khuat
94d9921324 check feature flag for proper transform when create and run (#63641) 2025-09-18 19:42:15 +07:00
Chris Truter
38dd9b1a82 Fix whitespace 2025-09-18 14:38:06 +02:00
Chris Truter
e00d3a0351 Fix-up rename 2025-09-18 14:35:24 +02:00
Chris Truter
32702eadbb Better names 2025-09-18 14:34:34 +02:00
Chris Truter
9c08b6777f Better docstring for the most complex method 2025-09-18 14:23:53 +02:00
Nicola Mometto
ee3ab9acde Merge branch 'proj-python-master' into proj-python/s01e04 2025-09-18 13:10:04 +01:00
Chris Truter
9b5fed78aa Test it 2025-09-18 14:09:12 +02:00
Chris Truter
fbc328ba2b Yoinked s3 code 2025-09-18 13:03:26 +02:00
Chris Truter
723a19e001 Remove staging settings from defaults 2025-09-18 13:02:58 +02:00
Chris Truter
f3e96cff26 Fix lints and minor bug 2025-09-18 12:57:13 +02:00
Chris Truter
1515d90cc7 Super embarrassing 2025-09-18 12:52:06 +02:00
Chris Truter
78d1cee978 Yoink s3 code out 2025-09-18 12:50:11 +02:00
Chris Truter
463bb865e8 Merge branch 'merge-from-master-sep-18' into proj-python/s01e04 2025-09-18 12:33:31 +02:00
Chris Truter
8a9bb6df63 Merge branch 'master' into proj-python/s01e04 2025-09-18 12:31:22 +02:00
Nicola Mometto
c3096b0bd1 maintain ordering 2025-09-18 11:23:44 +01:00
Romeo Van Snick
8518d7124f Improve tests for python transform 2025-09-18 20:21:11 +10:00
Romeo Van Snick
a7a740a0bf Use data picker to pick tables 2025-09-18 20:21:11 +10:00
Romeo Van Snick
de0a212a8e Use data selector for database picker 2025-09-18 20:21:11 +10:00
Romeo Van Snick
fb9704ce22 Further refactors 2025-09-18 20:21:11 +10:00
Nicola Mometto
82b64197e4 chore 2025-09-18 11:11:05 +01:00
Ngoc Khuat
1585d8cf00 setup ci properly (#63693) 2025-09-18 10:59:58 +01:00
Chris Truter
95a2ab4e8c Missed a line somehow 2025-09-18 11:11:19 +02:00
Nicola Mometto
c40ab083ef cherry pick driver code (#63628)
* cherry pick driver code

* update changelog

* add type->database-type unit tests

* kondo

* create/insert table

* insert-from-jsonl test

* missing rename-tables for snowflake

* extend rename-tables-test to more drivers

* line breaks

* add driver python transform feature

* handle review comments

* kondo

* feat: replace string->val with a more generic insert-col->val

* remove non transactional impls

* chore: docsstrings

* TODO

* extra test

* fix extra test

* test all base-types

* fix sqls
2025-09-18 11:04:01 +02:00
Chris Truter
7c0580d524 Remove staging creds from code 2025-09-18 10:50:27 +02:00
Chris Truter
68167fd68a Rename setting for consistency 2025-09-18 10:46:29 +02:00
Chris Truter
8ce510608b Track versioning (#63534) 2025-09-18 10:31:36 +02:00
Romeo Van Snick
1c3d229286 Refactor PythonDataPicker 2025-09-18 18:01:51 +10:00
Romeo Van Snick
ff5160741f Fix some type imports 2025-09-18 17:06:03 +10:00
Romeo Van Snick
b060c51715 Add type for transform alias map 2025-09-18 16:07:33 +10:00
Romeo Van Snick
3ac0532a1e Use slugify from metabase/lib/formatting 2025-09-18 16:02:24 +10:00
Romeo Van Snick
b325cbb292 Do not allow picking unsupported databases 2025-09-18 16:01:37 +10:00
Romeo Van Snick
d152afb7cb Add transforms/python feature 2025-09-18 16:01:20 +10:00
Ngoc Khuat
f889842faa Proper CI setup: clone instead of sub modules (#63654) 2025-09-17 23:40:34 +07:00
Nicola Mometto
14933f5d94 typo 2025-09-17 15:36:47 +01:00
Nicola Mometto
7eb8454293 s/keep/filter 2025-09-17 15:31:30 +01:00
Nicola Mometto
eba026fe3f fix: order 2025-09-17 15:29:52 +01:00
Nicola Mometto
47cc2574ad type restriction. 2025-09-17 15:26:16 +01:00
Chris Truter
5b6e257871 Fix for empty input tables with LocalStack (#63639) 2025-09-17 16:24:57 +02:00
Nicola Mometto
a36ce63e50 fix sqls 2025-09-17 15:06:53 +01:00
Romeo Van Snick
b0574988f8 Fix LogOutput styling 2025-09-17 22:58:12 +10:00
Romeo Van Snick
3bceb9f3db Use css instead of styled components 2025-09-17 22:37:59 +10:00
Romeo Van Snick
37c6a82298 Remove duplicate schema field 2025-09-17 22:37:58 +10:00
Romeo Van Snick
9f79d2b5f7 Add LogOutput back to runsection 2025-09-17 22:37:58 +10:00
Romeo Van Snick
1c56af8cb1 Revert "Remove unused LogOutput"
This reverts commit 90c0704344.
2025-09-17 22:37:58 +10:00
Nicola Mometto
d3be8d4347 TODO 2025-09-17 13:21:08 +01:00
Nicola Mometto
947a52c1ea chore: docsstrings 2025-09-17 13:21:08 +01:00
Nicola Mometto
4eafb237c7 remove non transactional impls 2025-09-17 13:21:08 +01:00
Nicola Mometto
fa72ebd4ee feat: replace string->val with a more generic insert-col->val 2025-09-17 13:21:08 +01:00
Nicola Mometto
c10c14324b kondo 2025-09-17 13:21:08 +01:00
Nicola Mometto
025544d742 handle review comments 2025-09-17 13:21:08 +01:00
Nicola Mometto
892cfce52d add driver python transform feature 2025-09-17 13:21:08 +01:00
Nicola Mometto
dd04f39158 line breaks 2025-09-17 13:21:07 +01:00
Nicola Mometto
39810bee7d extend rename-tables-test to more drivers 2025-09-17 13:21:07 +01:00
Nicola Mometto
7f662b9321 missing rename-tables for snowflake 2025-09-17 13:21:07 +01:00
Romeo Van Snick
69ca0b2c6b Fix table layout 2025-09-17 22:07:40 +10:00
Romeo Van Snick
a6d3f4e488 Fix font sizes 2025-09-17 21:50:38 +10:00
Romeo Van Snick
cff794b6c2 Set up python runner in CI (#63566)
* Check out with submodules in CI

* Include python-runner in docker compose for tests

* Set up python runner in CI

* Fix import

* Update python-runner
2025-09-17 21:41:06 +10:00
Romeo Van Snick
90c0704344 Remove unused LogOutput 2025-09-17 21:35:44 +10:00
Romeo Van Snick
3ee4162738 Add more generic RunInfo 2025-09-17 21:35:24 +10:00
Romeo Van Snick
3877cdb6eb Remove cancelation code for now 2025-09-17 21:22:55 +10:00
Romeo Van Snick
b2ff9c847e Add syntax highlighting to Python script on Transform page 2025-09-17 21:15:50 +10:00
Romeo Van Snick
8c0ac25468 Disable debugger hotkey for python transforms 2025-09-17 21:13:22 +10:00
Romeo Van Snick
e07e10062c Disable debugger for python transforms for now, allow to open it with query param 2025-09-17 21:13:19 +10:00
Romeo Van Snick
b3dfb7548f Move type to metabase-api/types 2025-09-17 21:13:17 +10:00
Romeo Van Snick
4a34306c4b Move python transform api to general transforms folder 2025-09-17 21:13:15 +10:00
Romeo Van Snick
020c546b53 Add e2e tests for python common lib 2025-09-17 21:13:12 +10:00
Romeo Van Snick
f5f6155b1c Fix parameter 2025-09-17 21:13:10 +10:00
Romeo Van Snick
d8f02a8545 Fix types 2025-09-17 21:13:06 +10:00
Romeo Van Snick
71fe53f41f Add e2e test for Python transform cretion 2025-09-17 21:12:59 +10:00
Romeo Van Snick
bbef002e2d Reuse transform save modal 2025-09-17 21:12:31 +10:00
Romeo Van Snick
fbb1774d67 Fix import path 2025-09-17 21:12:29 +10:00
Romeo Van Snick
9cf2e10377 Add test ids 2025-09-17 21:12:26 +10:00
Romeo Van Snick
0da7a46952 Disable default schema 2025-09-17 21:12:20 +10:00
Romeo Van Snick
e2689d6350 Add helpers for python editor 2025-09-17 21:12:10 +10:00
Romeo Van Snick
03ef177df5 Split up component more 2025-09-17 21:11:33 +10:00
Romeo Van Snick
8981f73a67 Restructure Python transform editor 2025-09-17 21:11:31 +10:00
Romeo Van Snick
9cd205e099 Nest Python editor components in PythonTransformEditor 2025-09-17 21:11:28 +10:00
Romeo Van Snick
59188cdae2 Move PythonQueryEditor to top-level components 2025-09-17 21:11:25 +10:00
Romeo Van Snick
36edbab3f3 Remove unused props 2025-09-17 21:10:58 +10:00
Romeo Van Snick
7b2ffce97a Reduce specialization in Transform editor 2025-09-17 21:10:52 +10:00
Romeo Van Snick
68dd3fe87f Remove dead code 2025-09-17 21:10:20 +10:00
Romeo Van Snick
08aca57e82 Rename shared to common 2025-09-17 21:09:43 +10:00
Romeo Van Snick
a5f3577d71 Flatten Python transform page 2025-09-17 21:09:43 +10:00
Romeo Van Snick
c0b1ce6763 Move script rewriter to utils 2025-09-17 21:09:42 +10:00
Romeo Van Snick
9c2a8da244 Rename PythonEditor to PythonQueryEditor 2025-09-17 21:09:42 +10:00
Romeo Van Snick
d5c2d719b2 Remove unused props 2025-09-17 21:09:42 +10:00
Romeo Van Snick
25e16b10eb Handle json-line output 2025-09-17 21:09:42 +10:00
Romeo Van Snick
af38cff267 Use normal title case 2025-09-17 21:09:31 +10:00
Nicola Mometto
9d76366f07 Changes from cherry-pick branch 2025-09-17 10:03:49 +01:00
Nicola Mometto
8bb58b8ed9 insert-from-jsonl test 2025-09-16 18:05:25 +01:00
Nicola Mometto
08496e13ec create/insert table 2025-09-16 17:14:02 +01:00
Nicola Mometto
f3ccf8fb12 kondo 2025-09-16 17:14:02 +01:00
Nicola Mometto
3e45ec2456 add type->database-type unit tests 2025-09-16 17:14:02 +01:00
Nicola Mometto
dc0f77f702 update changelog 2025-09-16 16:49:22 +01:00
Nicola Mometto
2a4ef82f04 cherry pick driver code 2025-09-16 16:42:00 +01:00
Nicola Mometto
c3fcb6c4f1 snowflake test 2025-09-16 15:54:46 +01:00
Nicola Mometto
8efca18157 text fallback 2025-09-16 15:54:34 +01:00
Dan Stone
5fa08ec51a Fix regression where we lost ex-message at end of logs
Test case extended, subsequence was convenient for now
could instead respecify test cases using regex to deal with intermediate
messages
2025-09-16 15:44:50 +01:00
Nicola Mometto
c5f626609a fix clickhouse rename-tables 2025-09-16 13:56:22 +01:00
Nicola Mometto
b36868cd08 safeguard 2025-09-16 13:37:49 +01:00
Nicola Mometto
6be338e592 test fixes 2025-09-16 13:34:02 +01:00
Nicola Mometto
7f6fb642db more test refactoring 2025-09-16 13:22:48 +01:00
Nicola Mometto
3bd98820b8 refactoring 2025-09-16 13:17:31 +01:00
Nicola Mometto
6186daa722 remove dead test 2025-09-16 13:17:31 +01:00
Nicola Mometto
4f810f2195 chore: mysql reduce precision for non-representable types 2025-09-16 13:17:31 +01:00
Nicola Mometto
2138b76093 default to type/Text if no type present 2025-09-16 13:17:31 +01:00
Nicola Mometto
096b6ae885 create resulting table 2025-09-16 13:17:31 +01:00
Nicola Mometto
c7c2471b20 fixes for postgres arrays 2025-09-16 13:17:31 +01:00
Nicola Mometto
c4423114c7 more test 2025-09-16 13:17:31 +01:00
Ngoc Khuat
51016ac69a lots of test fixes 2025-09-16 18:52:15 +07:00
Ngoc Khuat
b54256b448 our own settings 2025-09-16 18:24:44 +07:00
Dan Stone
6c461c2e2a Factor out python specific api tests (#63552)
* Factor out python specific api tests

* Kondo
2025-09-16 11:21:53 +01:00
Chris Truter
c2e93f408e The least encapsulated module of all time 2025-09-16 12:01:49 +02:00
Chris Truter
a4ea8c5e8e Fixing linters 2025-09-16 11:58:29 +02:00
Chris Truter
36df56509a Remove hacky script tester API and some shadowed code 2025-09-16 11:48:21 +02:00
Ngoc Khuat
f48349a46a Merge branch 'master' into proj-python/s01e04 2025-09-16 16:33:16 +07:00
Ngoc Khuat
9d0c57d594 use new feature flag for python transform tests 2025-09-16 15:28:07 +07:00
Ngoc Khuat
0d777beb36 fix mixing imports from old modules 2025-09-16 15:28:07 +07:00
Dan Stone
e566259f01 Fix logging regression / test breakage 2025-09-16 09:27:04 +01:00
Dan Stone
7c905eea0a Shorter name for sleep variable 2025-09-16 09:24:49 +01:00
Dan Stone
43428da019 Indentation 2025-09-16 09:24:20 +01:00
Dan Stone
fdb337e991 Remove dev time comment 2025-09-16 09:24:03 +01:00
Dan Stone
c8374c760c Remove dead code 2025-09-16 09:19:30 +01:00
Chris Truter
74bb4fe4f6 Send token auth header (#63532) 2025-09-15 22:56:11 +02:00
Nicola Mometto
b36b3d8560 fix: no json for maria 2025-09-15 17:36:29 +01:00
Nicola Mometto
e35f7b54b3 mongo 2025-09-15 17:36:29 +01:00
Chris Truter
fdb1eb9e8b Fix lints in python_runner_test 2025-09-15 18:20:47 +02:00
Chris Truter
9a14667493 Fix another test 2025-09-15 18:18:31 +02:00
Chris Truter
c8919412a3 Fix a test - marking a regression? 2025-09-15 18:13:12 +02:00
Chris Truter
cf2ef23e8f Unique run ids 2025-09-15 18:12:57 +02:00
Ngoc Khuat
663374b067 fix import on test 2025-09-15 22:56:19 +07:00
Chris Truter
ba92a1150a Bump submodule 2025-09-15 17:55:38 +02:00
Nicola Mometto
cdf95855c3 bump 2025-09-15 16:52:13 +01:00
Nicola Mometto
e69d8882ed fix: bad rebase 2025-09-15 16:51:37 +01:00
Chris Truter
9050c12b6c Fix resolution path after module move 2025-09-15 17:47:33 +02:00
Ngoc Khuat
a278a27ef1 feature check for python db too 2025-09-15 22:45:35 +07:00
Nicola Mometto
a889243fcb bump python runner 2025-09-15 16:42:42 +01:00
Chris Truter
3bd6ef4e5b Fix submodule reference 2025-09-15 17:40:33 +02:00
Chris Truter
b03ffdf1e2 Don't swallow /cancel responses. Small fixes. 2025-09-15 17:38:24 +02:00
Chris Truter
92ab4507e8 Note two routes in the wrong place 2025-09-15 17:37:46 +02:00
Nicola Mometto
8067a89464 python transforms matrix test (#63495)
* tests

* cleanup unused tests for now

* fix for missing dtypes
2025-09-15 16:37:30 +01:00
Ngoc Khuat
2b88496689 fix case where transform have target db is not the same with source db 2025-09-15 22:29:14 +07:00
Nicola Mometto
3394ee6a66 add string->val 2025-09-15 16:25:26 +01:00
Nicola Mometto
8f65216db5 remove debug 2025-09-15 16:25:23 +01:00
Chris Truter
9ac0b94414 Less verbose name 2025-09-15 17:11:33 +02:00
Chris Truter
d2998fad92 Differentiate read and write durations 2025-09-15 17:11:06 +02:00
Chris Truter
f92d9914e4 Trim some fat 2025-09-15 17:07:05 +02:00
Ngoc Khuat
7bbd95fe3e transforms python module (#63503) 2025-09-15 21:37:54 +07:00
Dan Stone
28a9104b27 Docstring tweak 2025-09-15 15:28:06 +01:00
Dan Stone
65231151b3 Logs pass 1 (#63384)
* Fix bad variable name

* Ensure logs are not lost for certain kinds of error

Step change:
- reduced number of unique error handling / redundant paths
- prefer a custom error message if provided in ex-info during error
handling
- improved documentation, names

* Fix bad debug format string

* Add tests for log behaviour

* Kondo
2025-09-15 15:08:13 +01:00
Chris Truter
212595564e Cleaning up 2025-09-15 15:35:37 +02:00
Nicola Mometto
314633b3cf stop parsing dtype, driver fixes 2025-09-15 13:56:01 +01:00
Nicola Mometto
7d595c03d1 feat: update rename-table! impls to rename-tables!* 2025-09-15 10:39:25 +01:00
Nicola Mometto
b6e9f3f0df dead code 2025-09-15 10:33:09 +01:00
Nicola Mometto
36413f0c35 Merge remote-tracking branch 'origin/master' into proj-python/s01e04 2025-09-15 10:28:05 +01:00
Nicola Mometto
3214a98103 Python jsonl (#63437)
* feat: jsonl

* bump
2025-09-15 10:15:34 +01:00
Ngoc Khuat
dae2fe7378 Atomically swapping tables for python transform (#63292) 2025-09-12 20:29:39 +07:00
Ngoc Khuat
a2dd096507 Python library APIs (#63378)
Co-authored-by: Romeo Van Snick <romeo@romeovansnick.be>
2025-09-12 18:44:49 +07:00
Nicola Mometto
2f5655beec fix 2025-09-12 12:28:51 +01:00
Nicola Mometto
3a23ad9566 docstrings 2025-09-12 11:15:41 +01:00
Nicola Mometto
577c760848 feat: move driver-specific hacks during preprocessing 2025-09-12 10:55:02 +01:00
Chris Truter
2a19738f66 Upgrade to latest python-runner 2025-09-12 11:02:40 +02:00
Nicola Mometto
c6f0ec1dcf reintroduce fix for floats on bigquery
Closes QUE-2444
2025-09-11 23:09:07 +01:00
Nicola Mometto
dd36cb7eea feat: better insert-from-source abstraction 2025-09-11 15:06:14 +01:00
Nicola Mometto
e94c2b2c0c typo 2025-09-11 14:45:05 +01:00
Nicola Mometto
5317293228 bump 2025-09-11 14:45:05 +01:00
Nicola Mometto
cbf30cdd64 typo 2025-09-11 13:53:55 +01:00
Nicola Mometto
0d5072d0c7 Merge remote-tracking branch 'origin/master' into proj-python/s01e03 2025-09-11 13:51:11 +01:00
Dan Stone
fe35332a62 Remove unused import 2025-09-11 12:27:28 +01:00
Dan Stone
cc45389256 Interop with future task rather than doing old-school thread stuff on log process shutdown 2025-09-11 12:25:59 +01:00
Nicola Mometto
4eac2ceba3 fix(ci): make run 2025-09-11 11:34:02 +01:00
Nicola Mometto
a8f5ff5193 sqlserver rename-table 2025-09-11 11:22:48 +01:00
Nicola Mometto
b8fef1a67f chore 2025-09-11 11:11:17 +01:00
Nicola Mometto
ce6420ebc7 missing require 2025-09-11 11:10:00 +01:00
Nicola Mometto
6f74fd4593 type conversion for mongo 2025-09-11 11:09:47 +01:00
Nicola Mometto
33593abd60 format datetimes for bigq 2025-09-11 10:52:48 +01:00
Nicola Mometto
702bd9a3f2 typo 2025-09-11 10:40:34 +01:00
Nicola Mometto
8284646d15 more parsing for bigq 2025-09-11 10:35:58 +01:00
Romeo Van Snick
1e0246d2f1 Use DataGrid for displaying results 2025-09-11 19:33:17 +10:00
Romeo Van Snick
d82c40a1db Swap buttons 2025-09-11 19:33:16 +10:00
Romeo Van Snick
e39983b3db Add pandas completions 2025-09-11 19:33:16 +10:00
Romeo Van Snick
01f4bcfb43 Allow extension to be passed to CodeEditor 2025-09-11 19:33:16 +10:00
Romeo Van Snick
cae08116f0 Increase indentUnit for python 2025-09-11 19:33:16 +10:00
Romeo Van Snick
a314bb587c Remove color in editor gutter 2025-09-11 19:33:16 +10:00
Romeo Van Snick
09731e48fa Move python execution to separate hook 2025-09-11 19:33:15 +10:00
Romeo Van Snick
cdba119297 Split up ExecutionResult component 2025-09-11 19:32:37 +10:00
Romeo Van Snick
5957b1f5df Add empty and loading state components 2025-09-11 19:32:37 +10:00
Romeo Van Snick
2357bea601 Split off results into separate component 2025-09-11 19:32:37 +10:00
Romeo Van Snick
32281e2096 Move parseCSV to utils 2025-09-11 19:32:36 +10:00
Romeo Van Snick
f17e329af5 Make python editor resizable 2025-09-11 19:32:36 +10:00
Romeo Van Snick
c481f1ceb6 Lay python editor out in two columns 2025-09-11 19:32:36 +10:00
Romeo Van Snick
f63e05fda1 Add first-class PythonTransformSource type 2025-09-11 19:32:36 +10:00
Nicola Mometto
928f6f830b bump 2025-09-11 10:15:33 +01:00
Nicola Mometto
29e936e2f8 chore: Dictionary as root type 2025-09-11 10:15:21 +01:00
Nicola Mometto
802fe0fcfe fix: schema-exists? check 2025-09-11 10:15:21 +01:00
Nicola Mometto
0ae5027542 feat: decode json strings 2025-09-11 10:15:21 +01:00
Nicola Mometto
e64e7ec93b assoc table-schema to data-source 2025-09-11 10:15:21 +01:00
Chris Truter
fcb4ca1f80 Bump to latest submodule 2025-09-11 10:58:11 +02:00
Dan Stone
54695561ad Add a few comments regarding optimistic nature of s3 deletes 2025-09-11 09:41:07 +01:00
Nicola Mometto
e965057f1b fix for clickhouse driver 2025-09-11 08:53:07 +01:00
Nicola Mometto
7ed61da2df feat: json 2025-09-10 17:47:32 +01:00
Nicola Mometto
0fba1b34a9 fix support for clickhouse 2025-09-10 16:37:44 +01:00
Chris Truter
6ca68fc067 Pin to a fixed python-runner, with slightly changed logs 2025-09-10 17:36:27 +02:00
Dan Stone
93bcf2f1b8 Remove some commented out / dead code 2025-09-10 15:39:17 +01:00
Ngoc Khuat
c830ff5164 Start python runner with driver tests (#63335) 2025-09-10 20:51:56 +07:00
Nicola Mometto
97296942ab fixes for bigq rename-table 2025-09-10 12:35:18 +01:00
Nicola Mometto
a9b58f9dd7 fixes for bigq insertion 2025-09-10 12:26:20 +01:00
Nicola Mometto
54d2a79a1a eh, array/dictionary for bigq 2025-09-10 11:43:37 +01:00
Nicola Mometto
4147277e28 kondo 2025-09-10 10:51:33 +01:00
Nicola Mometto
1815fb8afa feat: allow overriding base_type/db_type 2025-09-10 10:51:02 +01:00
Nicola Mometto
058e4b3bc5 wip: filter out nested fields 2025-09-10 10:51:02 +01:00
Dan Stone
8c5cd34ec9 Separate file / s3 lifetimes / writes / reads from http calls (#63294)
- python-runner no longer simulates a single http api that does
everything (incl s3 interaction)
- Separates file lifetimes (shared storage) from execution
- Easier to see how to avoid reading into strings (e.g. for piping into
tmp files, follow up)

Earlier on we had a service that managed much of the I/O, metabase now
coordinates it so it makes little sense pretending there is still
an overall http api.
2025-09-10 10:35:50 +01:00
Nicola Mometto
eea9b658da fix: retrieve meta from appdb 2025-09-09 15:27:51 +01:00
Nicola Mometto
17fb99163f chore: bind cancel-chan 2025-09-09 14:58:37 +01:00
Nicola Mometto
7257a4d1d2 only deal with schemas if schema exist 2025-09-09 11:59:23 +01:00
Nicola Mometto
0287e639b1 fix(mongo): guard against lack of cancel-chan 2025-09-09 11:59:23 +01:00
Nicola Mometto
0999cfab54 wip: DDL/DML for mongo 2025-09-09 11:59:23 +01:00
Nicola Mometto
9e18d5b9f7 stub rename-table for bigquery 2025-09-09 10:49:01 +01:00
Nicola Mometto
9910f6d0ca kondo 2025-09-09 10:45:25 +01:00
Chris Truter
6939615ce2 Merge branch 'master' into proj-python/s01e03 2025-09-08 20:24:26 +02:00
Chris Truter
b7bb799e14 Test for ordering across databasees, if we enable python support 2025-09-08 20:23:57 +02:00
Ngoc Khuat
20d91af3a5 proper transform ordering 2025-09-08 22:16:23 +07:00
Ngoc Khuat
2d43e09f84 add tests and polishing 2025-09-08 21:40:26 +07:00
Ngoc Khuat
8cb6965fa7 Table swapping instead of delete then create 2025-09-08 21:40:26 +07:00
Nicola Mometto
1279c1f226 chore: kondo 2025-09-08 12:29:52 +01:00
Nicola Mometto
9efe8835fa typo 2025-09-08 12:22:53 +01:00
Nicola Mometto
23b4d7ece0 types for clickhouse 2025-09-08 12:18:43 +01:00
Nicola Mometto
40e71f4baa fix for redshift 2025-09-08 12:11:36 +01:00
Nicola Mometto
2dd76b303e add session-schema for snowflake 2025-09-08 11:45:37 +01:00
Nicola Mometto
f7eb4ca800 fix ddl for bigquery 2025-09-08 11:34:37 +01:00
Nicola Mometto
f3860aee72 fix bigquery types 2025-09-08 11:34:05 +01:00
Nicola Mometto
5d93b6fe71 fix roundtrip 2025-09-08 11:00:07 +01:00
Nicola Mometto
0783824721 Merge remote-tracking branch 'origin/master' into proj-python/s01e03 2025-09-08 10:46:17 +01:00
Chris Truter
e6d90444c2 delete old test file 2025-09-08 10:30:48 +02:00
Chris Truter
bda65e63db bump python-runner 2025-09-05 16:19:23 +02:00
Nicola Mometto
74d6c3f356 more fixes 2025-09-05 15:03:36 +01:00
Nicola Mometto
815d7a3186 fixes 2025-09-05 14:41:40 +01:00
Nicola Mometto
92171ab506 fix: cleanup 2025-09-05 14:33:05 +01:00
Nicola Mometto
4fc51375be add test 2025-09-05 14:32:48 +01:00
Nicola Mometto
de8a48277f more generic types 2025-09-05 14:28:48 +01:00
Nicola Mometto
dad98c137d fix test 2025-09-05 14:08:43 +01:00
Nicola Mometto
68d9786697 Merge remote-tracking branch 'origin/master' into proj-python/s01e02 2025-09-05 13:59:54 +01:00
Nicola Mometto
db02774c9c for snoflake and sqlserver too 2025-09-05 13:52:17 +01:00
Nicola Mometto
3b955d9067 feat: per driver impl 2025-09-05 13:40:03 +01:00
Nicola Mometto
6b190fdcb1 unused 2025-09-05 11:43:31 +01:00
Nicola Mometto
d3d865dda9 fix tests 2025-09-05 11:42:58 +01:00
Nicola Mometto
558c69ee84 simpler 2025-09-05 11:41:31 +01:00
Nicola Mometto
86bc79d826 typo 2025-09-05 11:41:31 +01:00
Nicola Mometto
6739de6bfc bump 2025-09-05 11:41:31 +01:00
Nicola Mometto
f21fda68f1 feat: use new methods 2025-09-05 11:41:31 +01:00
Nicola Mometto
350561daf9 add new type->database-type 2025-09-05 11:41:31 +01:00
Nicola Mometto
cc99a79835 root-type instead of upload-type 2025-09-05 11:41:31 +01:00
Ngoc Khuat
8a5c220e5c add scheduled job e2e tests 2025-09-05 16:59:07 +07:00
Nicola Mometto
daeb777e4d fix: sort data-rows 2025-09-05 10:57:16 +01:00
Nicola Mometto
708f4696ca move utils into transforms.util 2025-09-05 10:50:03 +01:00
Nicola Mometto
fc453f6bbc chore: move insert-from-source into! driver 2025-09-05 10:31:02 +01:00
Dan Stone
f50d99d709 Pass run-id to /logs endpoint 2025-09-05 09:29:03 +01:00
Chris Truter
c19e0fa75f Date times can roundtrip losslessly and custom metadata is supported 2025-09-05 09:53:44 +02:00
Dan Stone
fe9b74f261 Remove stdout/err from clj side (single :events output is used now) 2025-09-04 17:38:30 +01:00
Dan Stone
3f720c9624 Read new events jsonl file and use that for the message 2025-09-04 17:34:57 +01:00
Ngoc Khuat
0f2d049daf CI for Python Runner (#63113) 2025-09-04 21:54:04 +07:00
Nicola Mometto
3d295d8025 unused 2025-09-04 14:40:34 +01:00
Nicola Mometto
456dc422fc don't go through qp for create-table/drop-table 2025-09-04 14:34:21 +01:00
Nicola Mometto
008c87b572 automatically parse datetime fields 2025-09-04 12:18:02 +01:00
Nicola Mometto
b21d3365b7 wip: pass upload_type to manifest 2025-09-04 12:07:55 +01:00
Nicola Mometto
42b06a759f fix: timezone aware datetime 2025-09-04 12:00:14 +01:00
Nicola Mometto
b3088ecfe4 note 2025-09-04 11:38:13 +01:00
Nicola Mometto
a1d8b0650a an in/out type test 2025-09-04 11:37:26 +01:00
Chris Truter
9740c5fe74 Replace python-runner folder with git submodule 2025-09-04 12:12:32 +02:00
Chris Truter
bfdd0754ed Critical fix: docstring indentation 2025-09-04 12:02:50 +02:00
Nicola Mometto
df6404e2f5 hack 2025-09-04 10:46:17 +01:00
Nicola Mometto
22224b8304 simpler api 2025-09-04 10:26:32 +01:00
Nicola Mometto
11c83cab30 switch to delete-target-table 2025-09-04 10:04:15 +01:00
Nicola Mometto
ee3c3ff237 fix hints 2025-09-04 09:22:13 +01:00
Nicola Mometto
467734b04e type hints 2025-09-04 09:12:31 +01:00
Dan Stone
e1df9b5c14 Use timestamps to interleave python stdin/err output in msg 2025-09-03 17:54:28 +01:00
Dan Stone
e1083a25f7 Wrap stdin/stderr with timestamp prefixes for each line 2025-09-03 17:54:28 +01:00
Dan Stone
fe3ea7401d Increase max processes
For me 50 was exceeded today and I could not start transforms. I am
guessing these are starter numbers and we'll need to figure out values
anyway
2025-09-03 17:54:28 +01:00
Nicola Mometto
1815a69384 chore: add error 2025-09-03 17:34:02 +01:00
Nicola Mometto
af7a73b5f7 chore: allow empty schemas 2025-09-03 17:26:05 +01:00
Nicola Mometto
9035fda953 fix: don't remove indentation before """ 2025-09-03 17:07:59 +01:00
Nicola Mometto
b4df47d8c8 drop table instead of truncating 2025-09-03 17:00:35 +01:00
Ngoc Khuat
4d270c04d1 remove hashp 2025-09-03 22:30:35 +07:00
Chris Truter
ea56586c04 Add default creds provider to pick up STS 2025-09-03 17:07:09 +02:00
Arkadi Yaroslavtsev
ee1aa7620c adding sts dependency 2025-09-03 18:03:02 +03:00
Ngoc Khuat
0c871cbd68 fix delete transform + table 2025-09-03 20:58:08 +07:00
Nicola Mometto
8f038a2476 note 2025-09-03 13:11:58 +01:00
Nicola Mometto
52dad58417 move to common ns 2025-09-03 13:10:36 +01:00
Nicola Mometto
97bbf28724 todo 2025-09-03 13:09:08 +01:00
Nicola Mometto
b06558f7fc with double dispatch 2025-09-03 13:03:14 +01:00
Nicola Mometto
9e4bb24349 creation tests 2025-09-03 12:53:09 +01:00
Nicola Mometto
2c269334ab chore: fix tests 2025-09-03 12:53:09 +01:00
Nicola Mometto
b594e954d8 relax schema 2025-09-03 12:53:09 +01:00
Nicola Mometto
de545a639e wip: move away from csv upload 2025-09-03 12:53:08 +01:00
Ngoc Khuat
cc9f0cc6da fix build error s3 v2 2025-09-03 18:49:33 +07:00
Ngoc Khuat
35588c7f9e switch s3 sdk to v2 2025-09-03 17:22:21 +07:00
Nicola Mometto
c5920c39e8 remove whitespaces 2025-09-03 10:48:03 +01:00
Nicola Mometto
1ffcaee719 add logging 2025-09-03 10:45:44 +01:00
Nicola Mometto
9f619f71cc fix args 2025-09-03 10:42:17 +01:00
Nicola Mometto
1e63d81675 fix import typo 2025-09-03 10:37:19 +01:00
Nicola Mometto
a13415c31f cleanup, remove csv metrics (as it's going to change), generic with-timing 2025-09-03 10:36:41 +01:00
Chris Truter
22554bed89 rough metadata transfer (just return raw dtype for now) 2025-09-03 11:30:09 +02:00
Ngoc Khuat
db1155d958 fix scheduled python run 2025-09-03 15:31:23 +07:00
Chris Truter
e0b446f108 more type check fixes - including some smelly optional override stuff 2025-09-03 09:48:36 +02:00
Chris Truter
ba51fad9d2 fix type errors 2025-09-03 09:42:49 +02:00
Chris Truter
d4b7faa3ff solve more fe lints 2025-09-03 07:43:42 +02:00
Chris Truter
8c8fba1874 trouble with kms - support only region without custom endpoint 2025-09-03 07:34:38 +02:00
Chris Truter
ca970b80e7 fix unit test 2025-09-03 00:43:13 +02:00
Chris Truter
2b9ead20f9 fix kondo 2025-09-03 00:15:53 +02:00
Chris Truter
f0f9b98f0f Fix build - oops 2025-09-03 00:13:48 +02:00
Chris Truter
1cccfd3c66 Don't need this negative comment 2025-09-03 00:05:09 +02:00
Chris Truter
79f2bc3a06 Various fixes for staging and general life
- Measure time properly
- Make endpoint config optional, so it's skipped in staging
- Make auth key config option, so it's skipped in staging
- Skip dev-only settings defaults in staging
- Add prefix, so staging permissions work
- Use short names for Java classes (Cursive lint)
- Remove trivial comments, answer some others
- Fix test-id that broke malli
2025-09-03 00:00:16 +02:00
Chris Truter
6171621ea3 lock down script permissions a bit more 2025-09-02 22:28:48 +02:00
Chris Truter
f6d0e32a02 Remove sharp hidden failures edges around bucket creation 2025-09-02 22:10:24 +02:00
Chris Truter
1697548096 Use system credentials for s3 2025-09-02 22:10:24 +02:00
Nicola Mometto
32b49c788c snuck back in 2025-09-02 16:38:55 +01:00
Nicola Mometto
f447bdaa8f just use run-id 2025-09-02 16:36:16 +01:00
Nicola Mometto
fbb2824944 simplify 2025-09-02 16:12:24 +01:00
Nicola Mometto
34331c5a79 instrument python execution 2025-09-02 16:04:07 +01:00
Nicola Mometto
935e458161 instrument the job 2025-09-02 16:04:07 +01:00
Nicola Mometto
31f5c10ffe chore: allow analytics for transforms 2025-09-02 16:04:07 +01:00
Nicola Mometto
c77efb4ba1 feat: add instrumentation util namespace 2025-09-02 16:04:07 +01:00
Nicola Mometto
e7510b606e wip: add transform metrics 2025-09-02 16:04:07 +01:00
Chris Truter
ff2dc465f7 more debugging stuff 2025-09-02 16:58:01 +02:00
Chris Truter
43de9c4f6a Some minor fixes around docstring rewriting 2025-09-02 16:30:06 +02:00
Dan Stone
126098289b Logs are viewable inline during execution of python transforms 2025-09-02 15:23:09 +01:00
Dan Stone
fb808245e3 Logs endpoint vibes
Needs to be destroyed at some future point
2025-09-02 11:46:57 +01:00
Dan Stone
159462606f Tests include boto3 now, so add to venv 2025-09-02 11:46:57 +01:00
Nicola Mometto
7d1ba026db eh, rename to appease the munging 2025-09-02 11:21:06 +01:00
Nicola Mometto
18199cf553 remove placeholders 2025-09-02 11:21:05 +01:00
Nicola Mometto
1fd87edf97 use new settings 2025-09-02 11:21:05 +01:00
Nicola Mometto
76097a2c65 feat: split settings 2025-09-02 11:21:05 +01:00
Nicola Mometto
cc5eb4332c feat: settings page with s3 configs 2025-09-02 11:21:05 +01:00
Nicola Mometto
54ad1db5c8 remove old setting 2025-09-02 11:21:05 +01:00
Chris Truter
a3af25f5b0 Auto-update Python transform function signature when table aliases change
- Automatically updates def transform(...) signature when aliases are modified
- Updates docstring Args section with aligned descriptions and table names
- Adds reset button to restore default slugified aliases
- Validates alias input to only allow valid Python identifiers
- Handles multi-line signatures for many parameters (>80 chars)
- Shows visual distinction between custom and auto-generated aliases
- Fixes code editor layout to prevent overlap with Run button
2025-09-02 12:17:43 +02:00
Chris Truter
6ec9182ad0 silly move, from demo #p 2025-09-02 11:22:49 +02:00
Nicola Mometto
efb033096e chore: docstrings in right position 2025-09-02 10:06:51 +01:00
Chris Truter
91bd482969 Fix for 404s 2025-09-01 18:48:40 +02:00
Chris Truter
010ab29959 Fix for find-replace breaking signed url 2025-09-01 18:48:40 +02:00
Chris Truter
3f897aa4f8 get rid of vestigial working_dir 2025-09-01 18:48:40 +02:00
Nicola Mometto
d17158db77 working button 2025-09-01 17:43:07 +01:00
Nicola Mometto
29c80be876 add cancel python test endpoing 2025-09-01 17:37:59 +01:00
Nicola Mometto
76dc608ed8 add cancel button to test python endpoint 2025-09-01 17:37:50 +01:00
Nicola Mometto
ded4524241 better endpoint name 2025-09-01 16:31:02 +01:00
Nicola Mometto
65b3027992 fix /test error reporting 2025-09-01 16:14:47 +01:00
Chris Truter
d72ca3c6cc Minor code and object cleanup 2025-09-01 15:13:39 +02:00
Chris Truter
0622b4004a writing to a tempfile first is a feature - we can close the dwh query faster 2025-09-01 15:12:08 +02:00
Nicola Mometto
8348a94edb Merge remote-tracking branch 'origin/proj-python/s01e02' into proj-python/s01e02 2025-09-01 13:59:30 +01:00
Nicola Mometto
c96bd66166 Merge remote-tracking branch 'origin/proj-python/s01e02' into proj-python/s01e02 2025-09-01 13:55:49 +01:00
Chris Truter
21a54d7591 s3 config setting 2025-09-01 14:48:51 +02:00
Chris Truter
b8fb092432 Linter stuff 2025-09-01 14:38:30 +02:00
Nicola Mometto
5f6d3ee14e leave async cancel for sql 2025-09-01 13:35:45 +01:00
Chris Truter
f6b143d489 Makefile creates bucket 2025-09-01 14:31:45 +02:00
Chris Truter
8e5d085924 Vile vibe verbiage - but it works (locally) 2025-09-01 14:30:29 +02:00
Nicola Mometto
e1b4627f33 cancel working e2e 2025-09-01 13:22:00 +01:00
Chris Truter
fd50372b5d Hacky as all hell - input files use s3 2025-09-01 14:14:23 +02:00
Nicola Mometto
322edf01ba immediately cancel run 2025-09-01 13:01:26 +01:00
Nicola Mometto
50059a9790 fix transform-run-cancelation 2025-09-01 12:57:19 +01:00
Dan Stone
1115e30d3b Add docker-compose.yml with localstack
Makefile/tests now uses docker-compose
2025-09-01 12:53:50 +01:00
Nicola Mometto
518877e2f6 fix running-run for transform-id 2025-09-01 12:52:34 +01:00
Nicola Mometto
b0749f1b2b add cancel button 2025-09-01 12:45:08 +01:00
Nicola Mometto
c7cba11e85 fix cancel endpoint 2025-09-01 12:45:02 +01:00
Nicola Mometto
a1bcd98682 pass run-id 2025-09-01 12:44:59 +01:00
Nicola Mometto
95b077f039 fix tests 2025-09-01 12:31:10 +01:00
Nicola Mometto
840a2b71e8 feat: cancel in python server 2025-09-01 12:30:23 +01:00
Nicola Mometto
8325bb3375 feat: interrupt request on cancel 2025-09-01 12:20:51 +01:00
Nicola Mometto
00e078cfa4 chore: pipe cancel-chan down to python-runner 2025-09-01 10:54:43 +01:00
Nicola Mometto
105039ee2f Merge remote-tracking branch 'origin/master' into proj-python/s01e02 2025-09-01 10:41:02 +01:00
Ngoc Khuat
35e712e21b fix permissions when running transform as background job 2025-08-29 21:58:22 +07:00
Nicola Mometto
5003a1e5ca explicitely pass cancel-chan 2025-08-29 14:10:20 +01:00
Nicola Mometto
949096259a style 2025-08-29 14:00:02 +01:00
Nicola Mometto
4feb3d4230 refactor: use common function for running python-transform 2025-08-29 13:58:01 +01:00
Nicola Mometto
4be8421fae chore: lift utility function 2025-08-29 13:06:02 +01:00
Nicola Mometto
652c2a1799 fix 2025-08-29 12:13:06 +01:00
Nicola Mometto
668289e248 run python transform within virtual thread, conform to async api
Closes QUE-2302
2025-08-29 12:08:31 +01:00
Nicola Mometto
37417f1e1f chore: make sure permission issues don't impact file deletion 2025-08-29 10:53:56 +01:00
Nicola Mometto
6901a209e2 dead code 2025-08-29 10:26:28 +01:00
Chris Truter
67e37c58b4 Remove unused python client 2025-08-29 10:52:28 +02:00
Ngoc Khuat
65e54c1d17 fixex test run 2025-08-29 12:34:29 +07:00
Chris Truter
57ef35fd81 Better default schema, better docstring 2025-08-28 19:11:21 +02:00
Chris Truter
576c774001 Demo guide for Nico 2025-08-28 17:21:48 +02:00
Chris Truter
07c8dfab7c Slightly less wonky readme 2025-08-28 17:15:15 +02:00
Nicola Mometto
fe26b3ba57 chore: simpler code 2025-08-28 16:12:05 +01:00
Nicola Mometto
a7ab68a82a kondo 2025-08-28 14:21:26 +01:00
Nicola Mometto
8a5b76955d dead code 2025-08-28 14:18:03 +01:00
Ngoc Khuat
dd3e9ffb8d run test 2025-08-28 20:11:02 +07:00
Chris Truter
c1adff4357 Get rid of anemic proxy layer 2025-08-28 15:01:39 +02:00
Nicola Mometto
b90e090fec wip: disable pyexec user for now 2025-08-28 13:08:34 +01:00
Nicola Mometto
b8d7c3fe1b re-enable auth 2025-08-28 11:40:29 +01:00
Nicola Mometto
74efddd5f4 feat: communicate via files in python runner/server 2025-08-28 11:39:40 +01:00
Nicola Mometto
945263e788 feat: use jsonl files as transport layer 2025-08-28 11:23:22 +01:00
Nicola Mometto
11cf0a20b7 fix: actually delete temp files 2025-08-28 11:11:30 +01:00
Ngoc Khuat
04027d1310 source alignments 2025-08-28 15:57:17 +07:00
Ngoc Khuat
fdc0cdeb8b ensure tables are unique in tables selector 2025-08-28 15:44:11 +07:00
Ngoc Khuat
10d46bb28e Ensure we have the right permission when execute python 2025-08-28 15:14:41 +07:00
Chris Truter
02afe10001 Persistent container (#62839) 2025-08-27 23:22:05 +02:00
Chris Truter
4d8a8e93fb Fix up last few multi-table things 2025-08-27 18:47:09 +02:00
Chris Truter
7528f2f3b0 Makefile 2025-08-27 18:35:54 +02:00
Ngoc Khuat
7755c56386 multiple tables 2025-08-27 23:33:09 +07:00
Chris Truter
65978ae49a Better convention 2025-08-27 18:03:16 +02:00
Chris Truter
7875745bfe Fixups 2025-08-27 18:03:16 +02:00
Chris Truter
2f778b9dd9 Support multiple tables on the runner side 2025-08-27 18:03:15 +02:00
Nicola Mometto
3b3ec3e396 cleanup 2025-08-27 15:44:37 +01:00
Nicola Mometto
fa1946c7d9 wip: empty token rather than nil, else shell/sh args parsing gets confused 2025-08-27 15:37:54 +01:00
Nicola Mometto
decf483773 feat: don't go through QP, reducible-query 2025-08-27 15:37:54 +01:00
Ngoc Khuat
0a82070f27 FE select table when creating python transform 2025-08-27 20:43:14 +07:00
Chris Truter
21486dd01b Assume exactly 1 source table, preselected 2025-08-27 15:41:18 +02:00
Chris Truter
4ce5db9db3 WIP get table passed in 2025-08-27 15:30:35 +02:00
Ngoc Khuat
781045b019 TODO for python transform dependencies 2025-08-27 18:41:05 +07:00
Ngoc Khuat
fa2cbffbd5 python transforms now properly being scheduled 2025-08-27 18:39:55 +07:00
Ngoc Khuat
0695681d7c remove the hack of using local files as well 2025-08-27 00:49:55 +07:00
Ngoc Khuat
b5a67bf540 call api proxy instead of internal call, also have settings page for transform 2025-08-27 00:29:55 +07:00
Ngoc Khuat
83744d69a7 Transform life cycle macro and fix unable to run failed transform
repeatedly
2025-08-27 00:29:19 +07:00
Dan Stone
31650d2f87 Save message including stdout/err to the database on failure
Resolves QUE-2296
2025-08-26 18:05:36 +01:00
Dan Stone
c19a5dab9e Fix edit button not doing anything for python scripts
Closes QUE-2300
2025-08-26 17:44:52 +01:00
Dan Stone
b32310dad9 Revert "Fix wrong source field name used for showing script in transform view"
This reverts commit c7dfb541f3.
2025-08-26 16:31:47 +01:00
Nicola Mometto
e1e553e39c fix tests 2025-08-26 15:56:56 +01:00
Nicola Mometto
b6c062b365 fix test 2025-08-26 15:47:44 +01:00
Nicola Mometto
22a294c135 Merge branch 'QUE-2298' into proj-python/s01e01 2025-08-26 15:38:33 +01:00
Nicola Mometto
012c0e5661 disable auth for now.. 2025-08-26 15:33:17 +01:00
Dan Stone
c7dfb541f3 Fix wrong source field name used for showing script in transform view 2025-08-26 15:30:00 +01:00
Nicola Mometto
1ea8c670f7 fix for docker/win 2025-08-26 15:29:52 +01:00
Nicola Mometto
7ed9553ee3 fix 2025-08-26 15:12:46 +01:00
Nicola Mometto
ba67bef0d2 remove connection string, pass metabase-url instead 2025-08-26 14:59:33 +01:00
Dan Stone
6902bb5020 Use :dbname if :db is not present for conn
For me this was the case
2025-08-26 14:54:05 +01:00
Nicola Mometto
e6cf33c740 query tables via api 2025-08-26 14:23:54 +01:00
Nicola Mometto
39daa3258a fix: docker compose 2025-08-26 14:13:16 +01:00
Nicola Mometto
ef60fa5ee5 wip: table/:id/data endpoint 2025-08-26 14:04:28 +01:00
Nicola Mometto
386f8ce2ec fix: source-database-id for python transforms 2025-08-26 11:53:52 +01:00
Nicola Mometto
34c3a5f9e6 quick fix for error detection 2025-08-26 09:57:14 +01:00
Chris Truter
ce0372fd73 fixes 2025-08-25 17:18:55 +02:00
Chris Truter
6df12e022a Extract util that we'll need for demo 2025-08-25 16:55:47 +02:00
Chris Truter
5e5ff95ef7 test with temp db 2025-08-25 16:54:59 +02:00
Chris Truter
c8c39cfeb1 kondo 2025-08-25 16:54:59 +02:00
Ngoc Khuat
89056a9d5d placeholder connection string 2025-08-25 21:31:28 +07:00
Ngoc Khuat
db4f81cb38 wire up with upload 2025-08-25 21:27:34 +07:00
Chris Truter
2daba8f0b4 read frames 2025-08-25 16:14:04 +02:00
Chris Truter
4e1bb2a25a Inversion of control to hide output file 2025-08-25 16:14:03 +02:00
Ngoc Khuat
7e9d0e6a2b wiring up with python executor 2025-08-25 19:44:15 +07:00
Ngoc Khuat
0e9f773d55 Integerate with FE 2025-08-25 19:24:42 +07:00
Ngoc Khuat
8ac019cefb CRUD python transform 2025-08-25 19:24:42 +07:00
Chris Truter
ac78a328eb Stub python execution service api 2025-08-25 14:01:12 +02:00
Nicolo-Maria
0e915e0437 feat: add Python transform frontend pages 2025-08-25 12:42:08 +02:00
Chris Truter
d23bea5a3a Stub python container to execute user code 2025-08-25 12:30:30 +02:00
1183 changed files with 499243 additions and 588453 deletions

View File

@@ -1,7 +1,7 @@
;; -*- comment-column: 100; -*-
{:config-paths ["macros" "src" "config"]
:linters
{ ;;
{;;
;; enabled optional linters (these are disabled by default)
;;
@@ -187,7 +187,7 @@
:metabase/disallow-hardcoded-driver-names-in-tests
;; don't include `:h2` since that will probably lead to a mountain of tests where we are specifically testing
;; middleware or compilation behavior with just H2 since it's the default.
{:drivers #{ ;; core drivers
{:drivers #{;; core drivers
#_:h2
:postgres
:mysql
@@ -627,7 +627,7 @@
metabase.test/with-temp hooks.toucan2.tools.with-temp/with-temp
metabase.test/with-temp-file hooks.metabase.test.util/with-temp-file
metabase.test/with-temporary-setting-values hooks.metabase.test.util/with-temporary-setting-values
metabase.users.api-test/with-temp-user-email! hooks.common/with-one-binding
metabase.users.api-test/with-temp-user-email! hooks.common/with-one-binding
metabase.util.i18n/deferred-trs hooks.metabase.util.i18n/deferred-trs
metabase.util.i18n/deferred-tru hooks.metabase.util.i18n/deferred-tru
metabase.util.i18n/trs hooks.metabase.util.i18n/trs
@@ -742,7 +742,9 @@
;; `metabase-enterprise.sandbox.query-processor*-test`
;;
{:pattern "^metabase(?:((?:-enterprise\\.sandbox)?\\.query-processor)|(?:\\.driver\\.sql(?!server)(?!ite))).*-test$"
:name general-qp-and-driver-test-namespaces}]
:name general-qp-and-driver-test-namespaces}
{:pattern "^metabase\\.python-runner\\..*"
:name python-runner-namespaces}]
:config-in-ns
{test-namespaces
@@ -767,8 +769,8 @@
source-namespaces
{:linters
{:metabase/validate-escaped-single-quotes-in-i18n {:level :warning}
:metabase/validate-string-or-str-args-to-i18n {:level :warning}
:metabase/discourage-millis-duration {:level :warning}
:metabase/validate-string-or-str-args-to-i18n {:level :warning}
:metabase/discourage-millis-duration {:level :warning}
:discouraged-var
{clojure.core/with-redefs {:message "Don't use with-redefs outside of tests"}
@@ -835,16 +837,7 @@
performance-sensitive
{:linters
{:discouraged-var
{clojure.core/every? {:message "Use metabase.util.performance/every?"}
clojure.core/mapv {:message "Use metabase.util.performance/mapv"}
clojure.core/run! {:message "Use metabase.util.performance/run!"}
clojure.core/select-keys {:message "Use metabase.util.performance/select-keys"}
clojure.core/some {:message "Use metabase.util.performance/some"}
clojure.core/update-keys {:message "Use metabase.util.performance/update-keys"}
clojure.walk/keywordize-keys {:message "Use metabase.util.performance/keywordize-keys"}
clojure.walk/postwalk {:message "Use metabase.util.performance/postwalk"}
clojure.walk/prewalk {:message "Use metabase.util.performance/prewalk"}
clojure.walk/walk {:message "Use metabase.util.performance/walk"}}}}
{clojure.core/update-keys {:message "Use metabase.util.performance/update-keys"}}}}
qp-and-driver-source-namespaces
{:linters

View File

@@ -502,7 +502,6 @@
metabase.embedding.init
metabase.embedding.jwt
metabase.embedding.settings
metabase.embedding.util
metabase.embedding.validation}
:uses #{analytics
api
@@ -517,7 +516,6 @@
public-sharing
queries
query-processor
request
server
settings
tiles
@@ -549,13 +547,6 @@
settings
util}}
glossary
{:team "Workflows"
:api #{metabase.glossary.api}
:uses #{api
models
util}}
indexed-entities
{:team "UX West"
:api #{metabase.indexed-entities.api
@@ -952,7 +943,6 @@
:uses #{api
legacy-mbql
lib
lib-be
models
permissions
query-processor
@@ -1035,7 +1025,6 @@
metabase.request.schema}
:uses #{api
config
embedding
permissions ; FIXME
session
settings
@@ -1241,7 +1230,6 @@
events
lib
models
premium-features
query-processor
settings
task
@@ -1487,7 +1475,6 @@
driver
legacy-mbql
lib
lib-be
models
queries
query-permissions
@@ -1643,7 +1630,6 @@
:api #{metabase-enterprise.comments.api
metabase-enterprise.comments.init}
:uses #{api
channel
events
models
request
@@ -1831,12 +1817,10 @@
pulse
query-processor
request
search
server
settings
sync
system
task
users
util
warehouse-schema

View File

@@ -13,10 +13,7 @@
toucan2.core/insert-returning-pk!
toucan2.core/insert-returning-pks!
clojure.core.async/<!!
clojure.core.async/<!
clojure.core.async/>!!
clojure.core.async/>!
clojure.core.async/alts!
clojure.core.async/alts!!
clojure.core.async/close!
clojure.core.async/poll!

View File

@@ -3,7 +3,7 @@
> **Warning**
>
> If that is your first contribution to Metabase, please sign the [Contributor License Agreement](https://docs.google.com/a/metabase.com/forms/d/1oV38o7b9ONFSwuzwmERRMi9SYrhYeOrkbmNaq9pOJ_E/viewform). Also, if you're attempting to fix a translation issue, please submit your changes to our [Crowdin project](https://crowdin.com/project/metabase-i18n) instead of opening a PR.
> If that is your first contribution to Metabase, please sign the [Contributor License Agreement](https://docs.google.com/a/metabase.com/forms/d/1oV38o7b9ONFSwuzwmERRMi9SYrhYeOrkbmNaq9pOJ_E/viewform) (unless it's a tiny documentation change). Also, if you're attempting to fix a translation issue, please submit your changes to our [Crowdin project](https://crowdin.com/project/metabase-i18n) instead of opening a PR.
Closes https://github.com/metabase/metabase/issues/[issue_number]

View File

@@ -11,6 +11,18 @@ runs:
- name: Wait for port ${{ inputs.port }} to be ready
shell: bash
run: |
echo "Waiting for port ${{ inputs.port }} to be ready..."
timeout=300 # 5 minutes timeout
elapsed=0
while ! nc -z localhost ${{ inputs.port }}; do
sleep 0.1;
if [ $elapsed -ge $timeout ]; then
echo "❌ Timeout waiting for port ${{ inputs.port }} after ${timeout} seconds"
exit 1
fi
sleep 1
elapsed=$((elapsed + 1))
if [ $((elapsed % 10)) -eq 0 ]; then
echo " Still waiting for port ${{ inputs.port }}... ($elapsed seconds)"
fi
done
echo "✅ Port ${{ inputs.port }} is ready!"

View File

@@ -74,7 +74,7 @@ runs:
git push -u origin ${BACKPORT_BRANCH}
BACKPORT_PR_URL=$(gh pr create --base "${TARGET_BRANCH}" --head "${BACKPORT_BRANCH}" --label "was-backported" --assignee "${{ github.event.pull_request.user.login }}" --title "🤖 backported \"${ORIGINAL_TITLE}\"" --body "${PR_BODY}")
BACKPORT_PR_URL=$(gh pr create --base "${TARGET_BRANCH}" --head "${BACKPORT_BRANCH}" --label "was-backported" --assignee "${GITHUB_ACTOR}" --title "🤖 backported \"${ORIGINAL_TITLE}\"" --body "${PR_BODY}")
BACKPORT_PR_NUMBER=${BACKPORT_PR_URL##*/}
echo "backport_pr_number=$BACKPORT_PR_NUMBER" >> $GITHUB_OUTPUT

View File

@@ -1,172 +0,0 @@
const fs = require("fs");
const path = require("path");
const { execSync } = require("child_process");
const collectAndAverageTimings = async ({
github,
context,
daysBack = 7,
existingTimingsPath = "e2e/support/timings.json",
}) => {
try {
const artifacts = await findMergedTimingArtifacts({
github,
context,
daysBack,
});
if (artifacts.length === 0) {
throw new Error("No timing artifacts found in the specified time range");
}
const timingData = await downloadAndExtractArtifacts(
artifacts,
token,
github,
context,
);
if (timingData.length === 0) {
throw new Error("No valid timing data extracted from artifacts");
}
const averagedTimings = averageTimings(timingData);
averagedTimings.durations.sort((a, b) => a.spec.localeCompare(b.spec));
fs.writeFileSync(
existingTimingsPath,
JSON.stringify(averagedTimings, null, 2),
);
console.log(`✅ Updated ${existingTimingsPath} with new timing data`);
return { artifactCount: timingData.length };
} catch (error) {
console.error("❌ Error in collectAndAverageTimings:", error.message);
throw error;
}
};
const findMergedTimingArtifacts = async ({ github, context, daysBack }) => {
const cutoffDate = new Date();
cutoffDate.setDate(cutoffDate.getDate() - daysBack);
console.log(
`Searching for merged timing artifacts created after ${cutoffDate.toISOString()}`,
);
const artifacts = [];
let page = 1;
while (true) {
try {
const { data } = await github.rest.actions.listArtifactsForRepo({
owner: context.repo.owner,
repo: context.repo.repo,
name: "merged-e2e-timings",
per_page: 100,
page,
});
const validArtifacts = data.artifacts
.filter((a) => new Date(a.created_at) > cutoffDate && !a.expired)
.map(({ name, created_at, id }) => ({ name, created_at, id }));
artifacts.push(...validArtifacts);
if (
data.artifacts.some((a) => new Date(a.created_at) <= cutoffDate) ||
data.artifacts.length < 100
) {
break;
}
page++;
} catch (error) {
console.log(
`Error fetching artifacts page ${page - 1}: ${error.message}`,
);
break;
}
}
console.log(`Found ${artifacts.length} merged timing artifacts total`);
return artifacts;
};
const downloadAndExtractArtifacts = async (artifacts, github, context) => {
const tempDir = "timing-artifacts-temp";
fs.mkdirSync(tempDir, { recursive: true });
const timingData = [];
for (const artifact of artifacts) {
try {
console.log(`Downloading artifact: ${artifact.name}`);
const zipPath = path.join(tempDir, `${artifact.name}.zip`);
const extractPath = path.join(tempDir, artifact.name);
const { data } = await github.rest.actions.downloadArtifact({
owner: context.repo.owner,
repo: context.repo.repo,
artifact_id: artifact.id,
archive_format: "zip",
});
fs.writeFileSync(zipPath, Buffer.from(data));
fs.mkdirSync(extractPath, { recursive: true });
execSync(`unzip -q "${zipPath}" -d "${extractPath}"`);
const timingFilePath = path.join(extractPath, "runTimings.json");
if (fs.existsSync(timingFilePath)) {
const content = JSON.parse(fs.readFileSync(timingFilePath, "utf8"));
if (content.durations?.length) {
timingData.push(content);
console.log(
`✅ Extracted timing data from ${artifact.name} (${content.durations.length} specs)`,
);
} else {
console.log(`⚠ Invalid timing data structure in ${artifact.name}`);
}
} else {
console.log(`⚠ No timing data found in ${artifact.name}`);
}
fs.rmSync(extractPath, { recursive: true });
} catch (error) {
console.log(`❌ Error processing ${artifact.name}: ${error.message}`);
}
}
fs.rmSync(tempDir, { recursive: true });
console.log(
`Successfully extracted timing data from ${timingData.length}/${artifacts.length} artifacts`,
);
return timingData;
};
const averageTimings = (timingDataArray) => {
if (!timingDataArray.length) {
throw new Error("No timing data to process");
}
const specTimings = timingDataArray
.flatMap((data) => data.durations)
.filter((item) => item.spec && typeof item.duration === "number")
.reduce((acc, { spec, duration }) => {
(acc[spec] = acc[spec] || []).push(duration);
return acc;
}, {});
const durations = Object.entries(specTimings).map(([spec, durations]) => ({
spec,
duration: Math.round(
durations.reduce((sum, d) => sum + d, 0) / durations.length,
),
}));
return { durations };
};
module.exports = { collectAndAverageTimings, averageTimings };

View File

@@ -1,132 +0,0 @@
const { averageTimings } = require("./average-e2e-timings");
describe("averageTimings", () => {
it("should throw error when given empty array", () => {
expect(() => averageTimings([])).toThrow("No timing data to process");
});
it("should work for n=1", () => {
const input = [
{
durations: [
{ spec: "test1.cy.js", duration: 1000 },
{ spec: "test2.cy.js", duration: 2000 },
],
},
];
const result = averageTimings(input);
expect(result.durations).toHaveLength(2);
expect(result.durations).toContainEqual({
spec: "test1.cy.js",
duration: 1000,
});
expect(result.durations).toContainEqual({
spec: "test2.cy.js",
duration: 2000,
});
});
it("should calculate average for multiple specs", () => {
const input = [
{
durations: [
{ spec: "test1.cy.js", duration: 1000 },
{ spec: "test2.cy.js", duration: 2000 },
],
},
{
durations: [
{ spec: "test1.cy.js", duration: 2000 },
{ spec: "test2.cy.js", duration: 4000 },
],
},
];
const result = averageTimings(input);
expect(result.durations).toHaveLength(2);
expect(result.durations).toContainEqual({
spec: "test1.cy.js",
duration: 1500,
});
expect(result.durations).toContainEqual({
spec: "test2.cy.js",
duration: 3000,
});
});
it("should round averages to nearest integer", () => {
const input = [
{
durations: [{ spec: "test1.cy.js", duration: 1001 }],
},
{
durations: [{ spec: "test1.cy.js", duration: 1002 }],
},
];
const result = averageTimings(input);
expect(result.durations[0].duration).toBe(1002);
});
it("should ignore invalid entries", () => {
const input = [
{
durations: [
{ spec: "test1.cy.js", duration: 1000 },
{ spec: "test2.cy.js", duration: "invalid" },
{ duration: 2000 }, // missing spec
{ spec: "test3.cy.js", duration: 3000 },
],
},
];
const result = averageTimings(input);
expect(result.durations).toHaveLength(2);
expect(result.durations).toContainEqual({
spec: "test1.cy.js",
duration: 1000,
});
expect(result.durations).toContainEqual({
spec: "test3.cy.js",
duration: 3000,
});
});
it("should handle merging different specs", () => {
const input = [
{
durations: [
{ spec: "test1.cy.js", duration: 1000 },
{ spec: "test2.cy.js", duration: 2000 },
],
},
{
durations: [
{ spec: "test1.cy.js", duration: 3000 },
{ spec: "test3.cy.js", duration: 1000 },
],
},
];
const result = averageTimings(input);
expect(result.durations).toHaveLength(3);
expect(result.durations).toContainEqual({
spec: "test1.cy.js",
duration: 2000,
});
expect(result.durations).toContainEqual({
spec: "test2.cy.js",
duration: 2000,
});
expect(result.durations).toContainEqual({
spec: "test3.cy.js",
duration: 1000,
});
});
});

View File

@@ -1,39 +0,0 @@
const extractChangedTimings = ({ oldTimings, newTimings }) => {
const oldTimingMap =
oldTimings.durations?.reduce((map, item) => {
if (item.spec) {
map[item.spec] = item.duration;
}
return map;
}, {}) || {};
const changedDurations = newTimings.durations
.filter((item) => {
if (!item || !item.spec || typeof item.duration !== "number") {
return false;
}
const newSpec = convertPathFormat(item.spec);
const oldDuration = oldTimingMap[newSpec] || oldTimingMap[item.spec];
// Include if it's a new spec or the duration has changed
return oldDuration === undefined || oldDuration !== item.duration;
})
.map((item) => ({
spec: convertPathFormat(item.spec),
duration: item.duration,
}));
return {
changedTimings: { durations: changedDurations },
hasChanges: changedDurations.length > 0,
};
};
// Convert cypress-split path format to timings.json path format
// from "e2e/test/scenarios/..." to "../test/scenarios/..."
function convertPathFormat(specPath) {
return specPath.startsWith("e2e/test/")
? specPath.replace("e2e/test/", "../test/")
: specPath;
}
module.exports = { extractChangedTimings };

View File

@@ -1,195 +0,0 @@
const { extractChangedTimings } = require("./extract-changed-timings");
describe("extractChangedTimings", () => {
it("should return empty durations when no changes", () => {
const oldTimings = {
durations: [
{ spec: "test1.cy.js", duration: 1000 },
{ spec: "test2.cy.js", duration: 2000 },
],
};
const newTimings = {
durations: [
{ spec: "test1.cy.js", duration: 1000 },
{ spec: "test2.cy.js", duration: 2000 },
],
};
const result = extractChangedTimings({ oldTimings, newTimings });
expect(result.changedTimings.durations).toHaveLength(0);
expect(result.hasChanges).toBe(false);
});
it("should detect new specs", () => {
const oldTimings = {
durations: [{ spec: "test1.cy.js", duration: 1000 }],
};
const newTimings = {
durations: [
{ spec: "test1.cy.js", duration: 1000 },
{ spec: "test2.cy.js", duration: 2000 },
],
};
const result = extractChangedTimings({ oldTimings, newTimings });
expect(result.changedTimings.durations).toHaveLength(1);
expect(result.changedTimings.durations[0]).toEqual({
spec: "test2.cy.js",
duration: 2000,
});
expect(result.hasChanges).toBe(true);
});
it("should detect duration changes", () => {
const oldTimings = {
durations: [
{ spec: "test1.cy.js", duration: 1000 },
{ spec: "test2.cy.js", duration: 2000 },
],
};
const newTimings = {
durations: [
{ spec: "test1.cy.js", duration: 1000 },
{ spec: "test2.cy.js", duration: 2200 },
],
};
const result = extractChangedTimings({ oldTimings, newTimings });
expect(result.changedTimings.durations).toHaveLength(1);
expect(result.changedTimings.durations[0]).toEqual({
spec: "test2.cy.js",
duration: 2200,
});
expect(result.hasChanges).toBe(true);
});
it("should handle missing old timings", () => {
const oldTimings = { durations: [] };
const newTimings = {
durations: [
{ spec: "test1.cy.js", duration: 1000 },
{ spec: "test2.cy.js", duration: 2000 },
],
};
const result = extractChangedTimings({ oldTimings, newTimings });
// All timings should be considered "changed" since old timings is empty
expect(result.changedTimings.durations).toHaveLength(2);
expect(result.changedTimings.durations).toContainEqual({
spec: "test1.cy.js",
duration: 1000,
});
expect(result.changedTimings.durations).toContainEqual({
spec: "test2.cy.js",
duration: 2000,
});
expect(result.hasChanges).toBe(true);
});
it("should handle invalid timing entries", () => {
const oldTimings = {
durations: [{ spec: "test1.cy.js", duration: 1000 }],
};
const newTimings = {
durations: [
{ spec: "test1.cy.js", duration: 1000 },
{ spec: "test2.cy.js", duration: "invalid" },
{ duration: 2000 }, // missing spec
{ spec: "test3.cy.js", duration: 3000 },
],
};
const result = extractChangedTimings({ oldTimings, newTimings });
// Only test3.cy.js should be included (new spec with valid data)
expect(result.changedTimings.durations).toHaveLength(1);
expect(result.changedTimings.durations[0]).toEqual({
spec: "test3.cy.js",
duration: 3000,
});
expect(result.hasChanges).toBe(true);
});
it("should handle timing data with different specs", () => {
const oldTimings = {
durations: [
{ spec: "test1.cy.js", duration: 1000 },
{ spec: "test2.cy.js", duration: 2000 },
],
};
const newTimings = {
durations: [
{ spec: "test1.cy.js", duration: 3000 },
{ spec: "test3.cy.js", duration: 1000 },
],
};
const result = extractChangedTimings({ oldTimings, newTimings });
expect(result.changedTimings.durations).toHaveLength(2);
expect(result.changedTimings.durations).toContainEqual({
spec: "test1.cy.js",
duration: 3000,
});
expect(result.changedTimings.durations).toContainEqual({
spec: "test3.cy.js",
duration: 1000,
});
expect(result.hasChanges).toBe(true);
});
it("should convert cypress-split paths to relative paths", () => {
const oldTimings = {
durations: [
{
spec: "../test/scenarios/dashboard/dashboard.cy.spec.js",
duration: 1000,
},
{
spec: "../test/scenarios/admin/databases.cy.spec.js",
duration: 2000,
},
],
};
const newTimings = {
durations: [
{
spec: "e2e/test/scenarios/dashboard/dashboard.cy.spec.js",
duration: 1200,
},
{
spec: "e2e/test/scenarios/admin/databases.cy.spec.js",
duration: 2000,
},
{
spec: "e2e/test/scenarios/collections/collections.cy.spec.js",
duration: 1500,
},
],
};
const result = extractChangedTimings({ oldTimings, newTimings });
expect(result.changedTimings.durations).toHaveLength(2);
expect(result.changedTimings.durations).toContainEqual({
spec: "../test/scenarios/dashboard/dashboard.cy.spec.js",
duration: 1200,
});
expect(result.changedTimings.durations).toContainEqual({
spec: "../test/scenarios/collections/collections.cy.spec.js",
duration: 1500,
});
expect(result.hasChanges).toBe(true);
});
});

View File

@@ -19,7 +19,6 @@ while [[ $# -gt 0 ]]; do
done
echo "Cloning python-runner repository..."
rm -r python-runner
git clone --depth 1 https://${METABASE_AUTOMATION_USER_TOKEN}@github.com/metabase/python-runner-container.git python-runner
if [ "$SKIP_START" = false ]; then

View File

@@ -330,6 +330,10 @@ jobs:
test-args: >-
${{ matrix.job.test-args }}
:exclude-tags '[ ${{ matrix.job.exclude-tag }} ${{ env.__ADDITIONAL_EXCLUDED_TAG__ }}]'
- name: Setup python-runner
env:
METABASE_AUTOMATION_USER_TOKEN: ${{ secrets.METABASE_AUTOMATION_USER_TOKEN }}
run: ./.github/scripts/setup-python-runner.sh
- name: Upload Test Results
uses: ./.github/actions/upload-test-results
if: always()

View File

@@ -6,19 +6,11 @@ on:
jobs:
create-backport:
if:
| # the PR must have a backport label when it's merged, or be labeled with a backport label after merge
if: | # the PR must be merged and have a backport label
github.event.pull_request.merged == true && (
github.event.action == 'closed' && (
contains(github.event.pull_request.labels.*.name, 'backport') ||
contains(github.event.pull_request.labels.*.name, 'double-backport') ||
contains(github.event.pull_request.labels.*.name, 'triple-backport')
) ||
github.event.action == 'labeled' && (
github.event.label.name == 'backport' ||
github.event.label.name == 'double-backport' ||
github.event.label.name == 'triple-backport'
)
contains(github.event.pull_request.labels.*.name, 'backport') ||
contains(github.event.pull_request.labels.*.name, 'double-backport') ||
contains(github.event.pull_request.labels.*.name, 'triple-backport')
)
runs-on: ubuntu-latest
steps:

View File

@@ -195,6 +195,7 @@ jobs:
- name: Run cljfmt
run: ./bin/mage cljfmt-all --force-check
be-tests-result:
needs:
- be-linter-eastwood

View File

@@ -689,57 +689,25 @@ jobs:
junit-name: 'be-tests-starburst-ee'
test-args: ":only ${{ inputs.test }} :times ${{ inputs.times }}"
login-to-ecr:
if: inputs.driver == 'Vertica'
runs-on: ubuntu-22.04
timeout-minutes: 15
permissions:
id-token: write
contents: read
steps:
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: ${{ secrets.CI_IAM_ROLE }}
role-session-name: GitHub_to_AWS_via_FederatedOIDC
aws-region: us-east-1
- name: Get ECR password
id: login-ecr
run: |
PASSWORD=$(aws ecr get-login-password --region us-east-1)
REGISTRY="$(aws sts get-caller-identity --query Account --output text).dkr.ecr.us-east-1.amazonaws.com"
echo "docker_username=AWS" >> $GITHUB_OUTPUT
echo "docker_password=$PASSWORD" >> $GITHUB_OUTPUT
echo "registry=$REGISTRY" >> $GITHUB_OUTPUT
outputs:
registry: ${{ steps.login-ecr.outputs.registry }}
docker_username: ${{ steps.login-ecr.outputs.docker_username }}
docker_password: ${{ steps.login-ecr.outputs.docker_password }}
be-tests-vertica-ee:
if: inputs.driver == 'Vertica'
runs-on: ubuntu-22.04
needs: login-to-ecr
timeout-minutes: 90
env:
CI: 'true'
DRIVERS: vertica
services:
vertica:
image: ${{ needs.login-to-ecr.outputs.registry }}/vertica-ce:25.3.0-0
ports:
- "5433:5433"
credentials:
username: ${{ needs.login-to-ecr.outputs.docker_username }}
password: ${{ needs.login-to-ecr.outputs.docker_password }}
steps:
- uses: actions/checkout@v4
- name: Make plugins directory
run: mkdir plugins
- name: Test Vertica driver
uses: ./.github/actions/test-driver
with:
junit-name: 'be-tests-vertica-ee'
test-args: ":only ${{ inputs.test }} :times ${{ inputs.times }}"
# TODO(rileythomp, 2025-08-20): Re-enable vertica tests once the docker image is updated
# be-tests-vertica-ee:
# if: ${{ false && inputs.driver == 'Vertica' }}
# runs-on: ubuntu-22.04
# timeout-minutes: 90
# env:
# CI: 'true'
# DRIVERS: vertica
# services:
# vertica:
# image: vertica/vertica-ce:12.0.2-0
# ports:
# - "5433:5433"
# steps:
# - uses: actions/checkout@v4
# - name: Make plugins directory
# run: mkdir plugins
# - name: Test Vertica driver
# uses: ./.github/actions/test-driver
# with:
# junit-name: 'be-tests-vertica-ee'
# test-args: ":only ${{ inputs.test }} :times ${{ inputs.times }}"

View File

@@ -115,11 +115,10 @@ jobs:
test-args: >-
:only-tags [:mb/driver-tests]
:exclude-tags [:mb/transforms-python-test]
# DS 2025-09-26 temporarily disabled while we diagnose flakes
# - name: Transforms Python Tests
# test-args: >-
# :only-tags [:mb/transforms-python-test]
# needs-python-runner: true
- name: Transforms Python Tests
test-args: >-
:only-tags [:mb/transforms-python-test]
needs-python-runner: true
env:
CI: 'true'
DRIVERS: bigquery-cloud-sdk
@@ -983,11 +982,10 @@ jobs:
test-args: >-
:only-tags [:mb/driver-tests]
:exclude-tags [:mb/transforms-python-test]
# DS 2025-09-26 temporarily disabled while we diagnose flakes
# - name: Transforms Python Tests
# test-args: >-
# :only-tags [:mb/transforms-python-test]
# needs-python-runner: true
- name: Transforms Python Tests
test-args: >-
:only-tags [:mb/transforms-python-test]
needs-python-runner: true
env:
CI: 'true'
DRIVERS: snowflake
@@ -1199,75 +1197,42 @@ jobs:
# aws-region: ${{ vars.AWS_REGION }}
# trunk-api-token: ${{ secrets.TRUNK_API_TOKEN }}
login-to-ecr:
needs: determine-driver-skips
if: ${{ !inputs.skip && !(needs.determine-driver-skips.outputs.skip == 'true') }}
runs-on: ubuntu-22.04
timeout-minutes: 15
permissions:
id-token: write
contents: read
steps:
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: ${{ secrets.CI_IAM_ROLE }}
role-session-name: GitHub_to_AWS_via_FederatedOIDC
aws-region: us-east-1
- name: Get ECR password
id: login-ecr
run: |
PASSWORD=$(aws ecr get-login-password --region us-east-1)
REGISTRY="$(aws sts get-caller-identity --query Account --output text).dkr.ecr.us-east-1.amazonaws.com"
echo "docker_username=AWS" >> $GITHUB_OUTPUT
echo "docker_password=$PASSWORD" >> $GITHUB_OUTPUT
echo "registry=$REGISTRY" >> $GITHUB_OUTPUT
outputs:
registry: ${{ steps.login-ecr.outputs.registry }}
docker_username: ${{ steps.login-ecr.outputs.docker_username }}
docker_password: ${{ steps.login-ecr.outputs.docker_password }}
be-tests-vertica-ee:
needs: [determine-driver-skips, login-to-ecr]
if: ${{ !inputs.skip && !(needs.determine-driver-skips.outputs.skip == 'true') }}
runs-on: ubuntu-22.04
timeout-minutes: 60
env:
CI: 'true'
DRIVERS: vertica
services:
vertica:
image: ${{ needs.login-to-ecr.outputs.registry }}/vertica-ce:25.3.0-0
ports:
- "5433:5433"
credentials:
username: ${{ needs.login-to-ecr.outputs.docker_username }}
password: ${{ needs.login-to-ecr.outputs.docker_password }}
name: Vertica
steps:
- uses: actions/checkout@v4
- name: Make plugins directory
run: mkdir plugins
- name: Test Vertica driver
uses: ./.github/actions/test-driver
with:
junit-name: 'be-tests-vertica-ee'
test-args: >-
:only-tags [:mb/driver-tests]
:exclude-tags [:mb/transforms-python-test]
- name: Upload Test Results
uses: ./.github/actions/upload-test-results
if: always()
with:
input-path: ./target/junit/
output-name: ${{ github.job }}
bucket: ${{ vars.AWS_S3_TEST_RESULTS_BUCKET }}
aws-access-key-id: ${{ secrets.AWS_TEST_RESULTS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_TEST_RESULTS_SECRET_ACCESS_KEY }}
aws-region: ${{ vars.AWS_REGION }}
trunk-api-token: ${{ secrets.TRUNK_API_TOKEN }}
# TODO(rileythomp, 2025-08-20): Re-enable vertica tests once the docker image is updated
# be-tests-vertica-ee:
# needs: determine-driver-skips
# if: ${{ false && !inputs.skip && !(needs.determine-driver-skips.outputs.skip == 'true') }}
# runs-on: ubuntu-22.04
# timeout-minutes: 60
# env:
# CI: 'true'
# DRIVERS: vertica
# services:
# vertica:
# image: vertica/vertica-ce:12.0.2-0
# ports:
# - "5433:5433"
# name: Vertica
# steps:
# - uses: actions/checkout@v4
# - name: Make plugins directory
# run: mkdir plugins
# - name: Test Vertica driver
# uses: ./.github/actions/test-driver
# with:
# junit-name: 'be-tests-vertica-ee'
# test-args: >-
# :only-tags [:mb/driver-tests]
# - name: Upload Test Results
# uses: ./.github/actions/upload-test-results
# if: always()
# with:
# input-path: ./target/junit/
# output-name: ${{ github.job }}
# bucket: ${{ vars.AWS_S3_TEST_RESULTS_BUCKET }}
# aws-access-key-id: ${{ secrets.AWS_TEST_RESULTS_ACCESS_KEY_ID }}
# aws-secret-access-key: ${{ secrets.AWS_TEST_RESULTS_SECRET_ACCESS_KEY }}
# aws-region: ${{ vars.AWS_REGION }}
# trunk-api-token: ${{ secrets.TRUNK_API_TOKEN }}
drivers-tests-result:
needs:

View File

@@ -131,7 +131,7 @@ jobs:
SPLIT_INDEX: ${{ inputs.split_index }}
SPLIT_FILE: "./e2e/support/timings.json"
SPLIT_OUTPUT_FILE: "newTimes.json"
SPLIT_TIME_THRESHOLD: 0.01 # 1% threshold for changes
SPLIT_TIME_THRESHOLD: 5.0
# do not write cypress-split summary to github
SPLIT_SUMMARY: false
SPEC_PATH: ${{ env.specs || inputs.specs }}
@@ -185,51 +185,6 @@ jobs:
name: failed-tests-${{ inputs.name }}-${{ inputs.edition }}
path: ./cypress/test-results
- name: Extract chunk-specific changed timings
id: extract-timings
if: inputs.split_index >= 0 && github.ref == 'refs/heads/master'
continue-on-error: true
uses: actions/github-script@v7
with:
script: |
const fs = require('fs');
const oldTimingsPath = "./e2e/support/timings.json";
const newTimingsPath = "./e2e/support/newTimes.json";
if (!fs.existsSync(newTimingsPath)) {
throw new Error(`newTimes.json not found - cypress-split may not have generated it`);
}
if (!fs.existsSync(oldTimingsPath)) {
throw new Error(`Original timings file ${oldTimingsPath} not found`);
}
const oldTimings = fs.existsSync(oldTimingsPath)
? JSON.parse(fs.readFileSync(oldTimingsPath, "utf8"))
: { durations: [] };
const newTimings = fs.existsSync(newTimingsPath)
? JSON.parse(fs.readFileSync(newTimingsPath, "utf8"))
: { durations: [] };
const { extractChangedTimings } = require('${{ github.workspace }}/.github/scripts/extract-changed-timings.js');
const result = extractChangedTimings({ oldTimings, newTimings });
if (result.hasChanges) {
core.setOutput('has-changes', 'true');
fs.writeFileSync('chunkTimings.json', JSON.stringify(result.changedTimings, null, 2));
console.log(`Extracted ${Object.keys(result.changedTimings).length} changed timings for chunk index ${{ inputs.split_index }}`);
}
- name: Upload Timing Data
uses: actions/upload-artifact@v4
if: steps.extract-timings.outcome == 'success' && steps.extract-timings.outputs.has-changes == 'true'
with:
name: timing-data-${{ inputs.name }}-${{ inputs.split_index }}
path: chunkTimings.json
retention-days: 1 # gets merged immediately by parent workflow
- name: Publish Summary
if: failure()
uses: actions/github-script@v7

View File

@@ -87,67 +87,6 @@ jobs:
name: Flaky
tags: "@flaky"
collect-timings:
name: Collect E2E Timings
runs-on: ubuntu-22.04
timeout-minutes: 10
needs: e2e-tests
if: always() && !cancelled() && github.ref == 'refs/heads/master'
steps:
- uses: actions/checkout@v4
with:
sparse-checkout: |
.github
e2e/support
sparse-checkout-cone-mode: false
- name: Download all timing artifacts
uses: actions/download-artifact@v4
with:
pattern: timing-data-*
path: timing-files
merge-multiple: false
continue-on-error: true
- name: Merge split timings
id: merge
continue-on-error: true
uses: actions/github-script@v7
with:
script: |
const fs = require('fs');
const path = require('path');
const chunkFiles = glob.sync('timing-files/*/chunkTimings.json');
const specs = new Map();
chunkFiles.forEach(file => {
const { durations = [] } = JSON.parse(fs.readFileSync(file, 'utf8'));
durations.forEach(({ spec, duration }) => specs.set(spec, duration));
});
if (specs.size === 0) {
throw new Error('No timing data found - no specs had timing changes');
}
const outputPath = 'e2e/support/runTimings.json';
fs.mkdirSync(path.dirname(outputPath), { recursive: true });
const runTimings = {
durations: [...specs].map(([spec, duration]) => ({ spec, duration }))
};
fs.writeFileSync(outputPath, JSON.stringify(runTimings, null, 2));
console.log(`Merged timing data for ${specs.size} specs`);
- name: Upload merged timing files
if: steps.merge.outcome == 'success'
uses: actions/upload-artifact@v4
with:
name: merged-e2e-timings
path: e2e/support/runTimings.json
retention-days: 7
e2e-tests-result:
needs:
- e2e-tests

View File

@@ -26,8 +26,26 @@ jobs:
- name: Compile CLJS
run: yarn build-pure:cljs
- name: Build Storybook
# We won't be able to use Storybook play's `userEvent` with production build
# see https://github.com/storybookjs/storybook/issues/19758
run: NODE_ENV=development yarn build-storybook
- name: Serve Storybook
run: yarn http-server storybook-static -p 6006 &
shell: bash
- name: Wait for Storybook to become available
run: |
echo "Waiting for Storybook to be served..."
while ! curl --silent --fail http://localhost:6006; do
sleep 5
echo "Retrying..."
done
echo "Storybook is being served!"
- name: Update Loki Snapshots
run: yarn test-visual:loki:ci || yarn test-visual:loki-approve-diff && yarn test-visual:loki-prune
run: yarn test-visual:loki || yarn test-visual:loki-approve-diff && yarn test-visual:loki-prune
- name: Check for changes
id: check-changes

View File

@@ -53,8 +53,14 @@ jobs:
- name: Compile CLJS
run: NODE_ENV=development yarn build-pure:cljs
- name: Build Storybook
# We won't be able to use Storybook play's `userEvent` with production build
# see https://github.com/storybookjs/storybook/issues/19758
run: NODE_ENV=development yarn build-storybook
- name: Run Loki Visual Tests
run: yarn test-visual:loki:ci
run: |
yarn loki --requireReference --reactUri file:./storybook-static --chromeFlags='--headless --disable-gpu' --verboseRenderer
- name: Generate Visual Report on Failure
if: failure()

View File

@@ -8,7 +8,6 @@ on:
pull_request:
types: [opened, synchronize, reopened]
merge_group:
workflow_dispatch:
jobs:
files-changed:
@@ -75,7 +74,7 @@ jobs:
secrets: inherit
backend-tests:
if: ${{ !cancelled() && needs.files-changed.result == 'success' }}
if: ${{ !cancelled() }}
needs: [files-changed, static-viz-files-changed]
uses: ./.github/workflows/backend.yml
secrets: inherit
@@ -113,7 +112,7 @@ jobs:
e2e-tests:
needs: [files-changed, uberjar]
if: always() && !cancelled() && needs.files-changed.result == 'success'
if: always() && !cancelled()
uses: ./.github/workflows/e2e-tests.yml
secrets: inherit
with:

View File

@@ -1,92 +0,0 @@
name: Update E2E Timings
permissions:
contents: write
pull-requests: write
on:
workflow_dispatch:
inputs:
days_back:
description: "Number of days to look back for timing artifacts"
type: number
default: 7
required: false
schedule:
# Run weekly on Sundays at 6 AM UTC
- cron: "0 6 * * 0"
jobs:
collect-and-average-timings:
runs-on: ubuntu-22.04
timeout-minutes: 30
env:
DAYS_BACK: ${{ inputs.days_back || 7 }}
steps:
- uses: actions/checkout@v4
with:
token: ${{ secrets.GITHUB_TOKEN }}
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: 22
- name: Collect and average timing data
id: collect-timings
uses: actions/github-script@v7
with:
script: |
const { collectAndAverageTimings } = require('${{ github.workspace }}/.github/scripts/average-e2e-timings.js');
const result = await collectAndAverageTimings({
github,
context,
token: '${{ secrets.GITHUB_TOKEN }}',
daysBack: ${{ env.DAYS_BACK }},
existingTimingsPath: 'e2e/support/timings.json'
});
core.setOutput('artifact-count', result.artifactCount);
- name: Set up git configuration
run: |
git config user.name "github-actions[bot]"
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
BRANCH_NAME="update-e2e-timings-$(date +%Y%m%d-%H%M%S)"
echo "BRANCH_NAME=$BRANCH_NAME" >> $GITHUB_ENV
echo "Branch name will be: $BRANCH_NAME"
- name: Create Pull Request
uses: peter-evans/create-pull-request@v6
with:
token: ${{ secrets.GITHUB_TOKEN }}
branch: ${{ env.BRANCH_NAME }}
base: master
commit-message: |
Update E2E test timings with averaged data
Collected timing data from ${{ env.DAYS_BACK }} days of test runs.
All specs with timing data were included (simple averages).
🤖 Generated automatically by update-e2e-timings workflow
Co-Authored-By: GitHub Actions <actions@github.com>
committer: "github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>"
author: "github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>"
title: "[chore] Update E2E test timings with recent data"
body: |
## Summary
This PR updates the E2E test timing data with averaged values from recent test runs.
## Details
- Collected data from the last **${{ env.DAYS_BACK }} days** of test runs
- Averaged timing data from **${{ steps.collect-timings.outputs.artifact-count }}** test runs
## Generated automatically
🤖 This PR was created automatically by the `update-e2e-timings` workflow.
The timing data helps cypress-split optimize test distribution across parallel jobs.
Co-Authored-By: GitHub Actions <actions@github.com>
labels: |
.CI & Tests
add-paths: e2e/support/timings.json

1
.gitignore vendored
View File

@@ -163,3 +163,4 @@ CLAUDE.local.md
/logs
/.clojure-mcp
__pycache__/

4
.gitmodules vendored Normal file
View File

@@ -0,0 +1,4 @@
[submodule "python-runner"]
path = python-runner
url = https://github.com/metabase/python-runner-container.git
branch = prototype

Binary file not shown.

Before

Width:  |  Height:  |  Size: 66 KiB

After

Width:  |  Height:  |  Size: 66 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 66 KiB

After

Width:  |  Height:  |  Size: 66 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 59 KiB

After

Width:  |  Height:  |  Size: 58 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 59 KiB

After

Width:  |  Height:  |  Size: 58 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 65 KiB

After

Width:  |  Height:  |  Size: 64 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 66 KiB

After

Width:  |  Height:  |  Size: 65 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 59 KiB

After

Width:  |  Height:  |  Size: 58 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 59 KiB

After

Width:  |  Height:  |  Size: 58 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 64 KiB

After

Width:  |  Height:  |  Size: 63 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 66 KiB

After

Width:  |  Height:  |  Size: 64 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 59 KiB

After

Width:  |  Height:  |  Size: 58 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 58 KiB

After

Width:  |  Height:  |  Size: 56 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 23 KiB

After

Width:  |  Height:  |  Size: 23 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 24 KiB

After

Width:  |  Height:  |  Size: 24 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 31 KiB

After

Width:  |  Height:  |  Size: 31 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 26 KiB

After

Width:  |  Height:  |  Size: 25 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 84 KiB

After

Width:  |  Height:  |  Size: 84 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 76 KiB

After

Width:  |  Height:  |  Size: 76 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 72 KiB

After

Width:  |  Height:  |  Size: 71 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 74 KiB

After

Width:  |  Height:  |  Size: 74 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 89 KiB

After

Width:  |  Height:  |  Size: 89 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 80 KiB

After

Width:  |  Height:  |  Size: 79 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 85 KiB

After

Width:  |  Height:  |  Size: 85 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 70 KiB

After

Width:  |  Height:  |  Size: 70 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 76 KiB

After

Width:  |  Height:  |  Size: 76 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 74 KiB

After

Width:  |  Height:  |  Size: 74 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 74 KiB

After

Width:  |  Height:  |  Size: 74 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 76 KiB

After

Width:  |  Height:  |  Size: 75 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 74 KiB

After

Width:  |  Height:  |  Size: 74 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 70 KiB

After

Width:  |  Height:  |  Size: 69 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 70 KiB

After

Width:  |  Height:  |  Size: 69 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 74 KiB

After

Width:  |  Height:  |  Size: 73 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 83 KiB

After

Width:  |  Height:  |  Size: 82 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 75 KiB

After

Width:  |  Height:  |  Size: 74 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 71 KiB

After

Width:  |  Height:  |  Size: 70 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 74 KiB

After

Width:  |  Height:  |  Size: 73 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 88 KiB

After

Width:  |  Height:  |  Size: 88 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 78 KiB

After

Width:  |  Height:  |  Size: 77 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 84 KiB

After

Width:  |  Height:  |  Size: 83 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 70 KiB

After

Width:  |  Height:  |  Size: 69 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 76 KiB

After

Width:  |  Height:  |  Size: 75 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 74 KiB

After

Width:  |  Height:  |  Size: 73 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 74 KiB

After

Width:  |  Height:  |  Size: 73 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 76 KiB

After

Width:  |  Height:  |  Size: 75 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 74 KiB

After

Width:  |  Height:  |  Size: 73 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 70 KiB

After

Width:  |  Height:  |  Size: 69 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 70 KiB

After

Width:  |  Height:  |  Size: 69 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 73 KiB

After

Width:  |  Height:  |  Size: 72 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 109 KiB

After

Width:  |  Height:  |  Size: 108 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 58 KiB

After

Width:  |  Height:  |  Size: 64 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 10 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 114 KiB

After

Width:  |  Height:  |  Size: 113 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 116 KiB

After

Width:  |  Height:  |  Size: 115 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 114 KiB

After

Width:  |  Height:  |  Size: 113 KiB

12
bb.edn
View File

@@ -160,18 +160,6 @@
:requires [[mage.stats-repl :as stats-repl]]
:task (stats-repl/connect (cli/parse! (current-task)))}
notify
{:doc "Send notification for a comment (that already exists"
:requires [[mage.be-dev :refer [nrepl-eval]]]
:options [["-c" "--comment-id ID" "Notify about this comment id"]
["-p" "--port PORT" "Port to use for the task, defaults to value in .nrepl-port"]]
:task (let [opts (:options (cli/parse! (current-task)))
;; note: no quoting -> expect it to be int
code (format "(notify-comment-id! %d)" (parse-long (:comment-id opts)))]
(if (:port opts)
(nrepl-eval "metabase-enterprise.comments.core" code (:port opts))
(nrepl-eval "metabase-enterprise.comments.core" code)))}
ls {:doc "List all mage public and private tasks"
:examples [["./bin/mage ls" "List all tasks"]]
:requires [[mage.util :as u]]

View File

@@ -36,7 +36,7 @@
com.clearspring.analytics/stream {:mvn/version "2.9.8" ; Various sketching algorithms
:exclusions [it.unimi.dsi/fastutil
org.slf4j/slf4j-api]}
com.clojure-goes-fast/jvm-alloc-rate-meter{:mvn/version "0.1.4"} ; Monitor heap allocation rate.
com.clojure-goes-fast/jvm-alloc-rate-meter {:mvn/version "0.1.4"} ; Monitor heap allocation rate.
com.clojure-goes-fast/jvm-hiccup-meter {:mvn/version "0.1.1"} ; Monitor GC pauses and other system-induced pauses.
com.draines/postal {:mvn/version "2.0.5" ; SMTP library
:exclusions [com.sun.mail/jakarta.mail]}
@@ -45,7 +45,7 @@
com.github.albfernandez/juniversalchardet {:mvn/version "2.5.0"}
com.github.blagerweij/liquibase-sessionlock {:mvn/version "1.6.9"} ; session lock for liquibase migrations
com.github.jknack/handlebars ^:antq/exclude ; 4.4.0 requires java 17+
{:mvn/version "4.3.1"} ; templating engine
{:mvn/version "4.3.1"} ; templating engine
com.github.oliyh/martian-clj-http {:mvn/version "0.1.31"} ; openapi client
com.github.pangloss/transducers {:git/sha "c91d39046c8c64dc31b84ebb27154a8084fcda4c" ; like medley for transducers
:git/url "https://github.com/pangloss/transducers"}
@@ -55,14 +55,14 @@
com.github.vertical-blank/sql-formatter {:mvn/version "2.0.5"} ; Java SQL formatting library https://github.com/vertical-blank/sql-formatter
com.google.guava/guava {:mvn/version "33.4.8-jre"} ; dep for BigQuery, Spark, and GA. Require here rather than letting different dep versions stomp on each other — see comments on #9697
com.h2database/h2 ^:antq/exclude ; https://metaboat.slack.com/archives/CKZEMT1MJ/p1727191010259979
{:mvn/version "2.1.214"} ; embedded SQL database
{:mvn/version "2.1.214"} ; embedded SQL database
com.ibm.icu/icu4j {:mvn/version "77.1"} ; Used to transliterate Unicode characters into Latin characters
com.knuddels/jtokkit {:mvn/version "1.1.0"} ; Java tokenizer library for OpenAI models, used for token counting in semantic search
com.mchange/c3p0 {:mvn/version "0.11.2"} ; DB connection pool
com.mchange/c3p0 {:mvn/version "0.10.1"} ; DB connection pool
com.snowplowanalytics/snowplow-java-tracker ^:antq/exclude ; 2+ depend on apache httpclient 5.x, but saml and clj-http depend on 4.x
{:mvn/version "1.0.1"} ; Snowplow analytics
{:mvn/version "1.0.1"} ; Snowplow analytics
com.sun.mail/jakarta.mail ^:antq/exclude ; 2+ changes imports from javax.mail to jakarta.mail
{:mvn/version "1.6.8"} ; SMTP lib, required by postal
{:mvn/version "1.6.8"} ; SMTP lib, required by postal
com.taoensso/nippy {:mvn/version "3.6.0"} ; Fast serialization (i.e., GZIP) library for Clojure
com.vladsch.flexmark/flexmark {:mvn/version "0.64.8"} ; Markdown parsing
com.vladsch.flexmark/flexmark-ext-autolink
@@ -85,14 +85,14 @@
hiccup/hiccup {:mvn/version "2.0.0"} ; HTML templating
inflections/inflections {:mvn/version "0.15.0"} ; Clojure/Script library used for prularizing words
instaparse/instaparse {:mvn/version "1.5.0"} ; Make your own parser
io.github.camsaul/toucan2 {:mvn/version "1.0.567"}
io.github.camsaul/toucan2 {:mvn/version "1.0.566"}
io.github.eerohele/pp {#_#_:git/tag "2024-11-13.77" ; super fast pretty-printing library
:git/sha "4998a1fdd9d3713d7034d04509e9212204773bf2"
:git/url "https://github.com/eerohele/pp"}
;; TODO: replace back to normal artifact
io.github.metabase/macaw {:mvn/version "0.2.27"} ; Parse native SQL queries
io.github.resilience4j/resilience4j-retry ^:antq/exclude ; 2.x do not support Java 8
{:mvn/version "1.7.1"} ; Support for retrying operations
{:mvn/version "1.7.1"} ; Support for retrying operations
io.prometheus/simpleclient_hotspot {:mvn/version "0.16.0"} ; prometheus jvm collector
junegunn/grouper {:mvn/version "0.1.1"} ; Batch processing helper
kixi/stats {:mvn/version "0.5.7" ; Various statistic measures implemented as transducers
@@ -198,8 +198,8 @@
org.eclipse.jetty.ee9/jetty-ee9-servlet
org.eclipse.jetty.ee9.websocket/jetty-ee9-websocket-jetty-server]}
slingshot/slingshot {:mvn/version "0.12.2"} ; enhanced throw/catch, used by other deps
software.amazon.awssdk/s3 {:mvn/version "2.33.0"} ; AWS SDK v2 for S3
software.amazon.awssdk/sts {:mvn/version "2.33.0"} ; Required for S3 client to use service account
software.amazon.awssdk/s3 {:mvn/version "2.33.0"}
software.amazon.awssdk/sts {:mvn/version "2.33.0"} ; AWS SDK v2 for S3
stencil/stencil {:mvn/version "0.5.0"} ; Mustache templates for Clojure
user-agent/user-agent {:mvn/version "0.1.1"} ; User-Agent string parser, for Login History page & elsewhere
weavejester/dependency {:mvn/version "1.0.0"} ; Dependency graphs and topological sorting
@@ -286,8 +286,6 @@
"-Duser.country=US"
;; Allow clojure goes fast tooling to work
"-Djdk.attach.allowAttachSelf"
;; have core.async report usages of blocking operations in go blocks
"-Dclojure.core.async.go-checking=true"
;; This will suppress the warning about dynamically loaded agents (like clj-memory-meter)
"-XX:+EnableDynamicAgentLoading"
;; set the logging properties set in `metabase.core.bootstrap`. calling (dev) will load it but
@@ -840,8 +838,14 @@
:test/lib
{:exec-args {:only ["test/metabase/lib"
"test/metabase/lib_be"]}}
;; test the python transforms
;; clj -X:dev:ee:ee-dev:driver:test:test/python-transform
:test/python-transform
{:exec-args {:only [metabase-enterprise.transforms.api-test
"enterprise/backend/test/metabase_enterprise/transforms"
"enterprise/backend/test/metabase_enterprise/transforms_python"]}}
;; test the Query Processor
;; test the Query Processor
;;
;; clj -X:dev:ee:ee-dev:test:test/qp
:test/qp

View File

@@ -61,7 +61,7 @@ Once one or more people have started to work on a feature, it should be marked [
### E) Verification and merging
All PRs that involve more than an insignificant change should be reviewed. See our [Code Review Process](https://www.metabase.com/docs/master/developers-guide/code-reviews).
All PRs that involve more than an insignificant change should be reviewed. See our [Code Review Process](./developers-guide/code-reviews.md).
If all goes well, the feature gets coded up, verified and then the pull request gets merged! High-fives all around.
@@ -95,7 +95,7 @@ By our definition, "Bugs" are situations where the program doesn't do what it wa
There are a lot of docs, which means keeping them up to date is hard. If you notice inconsistencies, errors, or outdated information, please help us keep them current!
Note that **we cannot accept translations for documentation at this time**. We support [in-app translations](https://www.metabase.com/docs/master/configuring-metabase/localization), and only support languages that have 100% coverage. But 1) the in-app text is orders of magnitude shorter than our docs, 2) it changes at a slower pace, and 3) we have a lot of people help out. We may consider supporting docs in multiple languages in the future, but for now we need to focus our resources on improving our existing documentation (and expanding it to include all of the new features we're adding).
Note that **we cannot accept translations for documentation at this time**. We support [in-app translations](./configuring-metabase/localization.md), and only support languages that have 100% coverage. But 1) the in-app text is orders of magnitude shorter than our docs, 2) it changes at a slower pace, and 3) we have a lot of people help out. We may consider supporting docs in multiple languages in the future, but for now we need to focus our resources on improving our existing documentation (and expanding it to include all of the new features we're adding).
### Working on features

View File

@@ -22,13 +22,11 @@ Here are a few ideas for what you can do with actions:
Actions must be added to a [model](../data-modeling/models.md), but actions only run on the raw tables that back those models (so actions will never edit your [model definition](../data-modeling/models.md#edit-a-models-query)).
If you're an admin who just wants to edit data in your tables, check out [Editable table data](../data-modeling/editable-tables.md).
## Enabling actions for a database
For actions to work, you'll first need to do the following two things:
1. **Enable model actions for the database connection**. To enable actions for a database connection, admins should click on the **gear** icon in the upper right and navigate to **Admin settings** > **Databases**, then click on the database you want to create actions for. On the right side of the connection settings form, toggle the **Model actions** option. For actions to work, the database user account (the account you're using to connect to the database) must have [write permissions](../databases/users-roles-privileges.md#privileges-to-enable-actions-and-editable-table-data). And for now, actions are only supported on PostgreSQL and MySQL databases.
1. **Enable model actions for the database connection**. To enable actions for a database connection, admins should click on the **gear** icon in the upper right and navigate to **Admin settings** > **Databases**, then click on the database you want to create actions for. On the right side of the connection settings form, toggle the **Model actions** option. For actions to work, the database user account (the account you're using to connect to the database) must have [write permissions](../databases/users-roles-privileges.md#privileges-to-enable-actions). And for now, actions are only supported on PostgreSQL and MySQL databases.
2. **Create at least one model from that database.** Actions are associated with models, so you'll need to have created (or have access to) at least one model before you can start creating actions.
## Who can use actions

Binary file not shown.

Before

Width:  |  Height:  |  Size: 129 KiB

After

Width:  |  Height:  |  Size: 158 KiB

View File

@@ -83,7 +83,7 @@ To have Metabot generate SQL for you:
3. Type cmd+b on Mac, ctrl+b on Windows, to open up the [chat sidebar](#the-metabot-chat-sidebar).
4. Ask it to "Write a SQL query that..." and type your prompt.
Metabot will generate the SQL for you, but it won't run the query. This gives you a chance to inspect the code before running it.
Metabot will generate the SQL for you, but it won't run the query. This gives you a chance to inspect the code before running it. The native editor is designed to be read-only (so don't worry about Metabot dropping any tables), but you should still check the query to make sure it targets the data you want.
If you don't specify a specific table in a natural language question, Metabot will only check the first 100 tables in the currently selected database. If your question pertains to tables other than those first 100 tables, Metabot may hallucinate the tables it needs, and the query will fail.

View File

@@ -1,9 +1,9 @@
---
title: Metabot AI settings
title: Metabot settings
summary: Configure Metabot settings, including which collections Metabot can access, and learn tips for improving Metabot's performance through data modeling and metadata.
---
# Metabot AI settings
# Metabot settings
{% include beta-blockquote.html
message="For now, <a href='https://www.metabase.com/features/metabot-ai'>Metabot</a> is only available as an add-on for Pro and Enterprise plans on Metabase Cloud."
@@ -13,26 +13,21 @@ _Settings > Admin settings > AI_
This page covers admin settings for Metabase's AI assistant, [Metabot](./metabot.md).
## Configure Metabot
To help Metabot focus on the data you care about most, select the collection containing the models and metrics you'd like it to work with.
## Pick a collection for Metabot to have access to
![Admin settings for AI Metabot](./images/ai-settings.png)
## Verified content
Admins on Pro and Enterprise plans can tell Metabot to only work with [models](../data-modeling/models.md) and [metrics](../data-modeling/metrics.md) that have been [verified](../exploration-and-organization/content-verification.md).
Restricting Metabot to verified models and metrics (and only models and metrics) can help Metabot produce more reliable answers, since you know someone has at least vetted the data Metabot can use.
## When embedding Metabot, you can pick a collection for Metabot to have access to
When embedding Metabot in your app, you can select a collection for Metabot:
To select a collection for Metabot:
1. Go to **Settings** > **Admin settings** > **AI**.
2. Click **Embedded Metabot**.
3. In the **Collection Embedded Metabot can use** section, click **Pick a collection**.
2. Click the **Collection** dropdown.
3. Select the collection that contains the models and metrics you want Metabot to use.
Metabot will use the models and metrics in that collection to help answer questions and generate queries. You can change this collection at any time. To give Metabot access to all collections, you can set the collection to the root collection, called "Our Analytics" (the default).
Alternatively (or additionally), you can restrict Metabot to [verified content](#verified-content).
Metabot will use the models and metrics in that collection to help answer questions and generate queries. You can change this collection at any time.
## Tips for making the most of Metabot
@@ -71,7 +66,7 @@ Metabase will try to set semantic types automatically, but you should confirm th
### Curate prompt suggestions
When you select a collection for Metabot to "learn", Metabot will suggest a series of prompts based on the content it finds in that collection. These prompts just give people a feel for the kinds of things people can ask Metabot to do.
When you select a collection for Metabot to "learn", Metabot will suggest a series of prompts based on the content it finds in that collection. These give people a feel for the kinds of things people can ask Metabot to do.
Admins can run these generated prompts to test the answers, or trash the individual prompts if they're not useful or misleading. You can also regenerate all the prompts with a click.

File diff suppressed because it is too large Load Diff

View File

@@ -80,7 +80,6 @@ Here's what you need to know when syncing Google Sheets:
- **Special character replacement**. Some characters just don't play nice with databases (like "?"). When we find these in column names, we'll replace them with "x" to keep things running smoothly.
- **Renamed files will create new tables**. If you rename files in your folder or tabs in your sheets, we'll treat them as brand new tables and import them fresh.
- **New columns sync automatically**. Adding new columns to your sheets? No problem — they'll show up in Metabase as expected.
- **Renaming sheet columns will create new table columns**. If you rename a column in a spreadsheet, Metabase will create a new column with the new name in the synced table. The original column will remain in the table but it'll become empty. You'll be able to hide the old column in Table Metadata settings.
- **Empty sheets won't import**. We won't import completely empty sheets or sheets that only have column headers. There needs to be some actual data in there.
- **Google Sheets must have unique names**. If files in the synced Google Drive folder (and its subfolders) have the same name (e.g., one sheet is in the root folder, another sheet in a subfolder), the sheets might not sync properly.

View File

@@ -137,6 +137,8 @@ config:
embedding-app-origins-interactive: null
embedding-app-origins-sdk: ''
embedding-homepage: hidden
embedding-hub-production-embed-snippet-created: false
embedding-hub-test-embed-snippet-created: false
embedding-secret-key: null
enable-embedding-interactive: false
enable-embedding-sdk: false

View File

@@ -585,6 +585,24 @@ Allow Metabase SDK access to these space delimited origins.
Embedding homepage status, indicating if it's visible, hidden or has been dismissed.
### `MB_EMBEDDING_HUB_PRODUCTION_EMBED_SNIPPET_CREATED`
- Type: boolean
- Default: `false`
- [Exported as](../installation-and-operation/serialization.md): `embedding-hub-production-embed-snippet-created`.
- [Configuration file name](./config-file.md): `embedding-hub-production-embed-snippet-created`
Indicates if a production embed snippet has been created for tracking in the embedding hub.
### `MB_EMBEDDING_HUB_TEST_EMBED_SNIPPET_CREATED`
- Type: boolean
- Default: `false`
- [Exported as](../installation-and-operation/serialization.md): `embedding-hub-test-embed-snippet-created`.
- [Configuration file name](./config-file.md): `embedding-hub-test-embed-snippet-created`
Indicates if a test embed snippet has been created for tracking in the embedding hub.
### `MB_EMBEDDING_SECRET_KEY`
- Type: string

View File

@@ -175,8 +175,6 @@ Which cards you can connect a filter or parameter widget to depends on:
- Whether the card has a relevant field (i.e., the field the widget is filtering).
- Where the widget is located: on the dashboard, heading card, or question card. Only dashboard widgets can be connected to cards across all [dashboard tabs](./introduction.md#dashboard-tabs). Widgets on heading and question cards are restricted to the cards on their respective tabs.
For cards built in the query builder, _filters_ can be connected to fields at any stage of the query. _Time grouping parameters_ can only be connected to fields in the last stage of the query, and so to make use of time grouping parameters, your query should have a summary by a datetime field as the last stage. For example, if you have a query that's just computing `Count` of products by `Category` and `Created At`, you'll be able to connect the time grouping parameter to the `Created At` grouping, but if you modify the query to filter for results with `Count > 100`, you won't be able to connect time grouping parameter on a dashboard to that query because the last stage is now a filter.
## Filtering dashboards with native/SQL questions
To connect a dashboard filter widget to a card with a question created with the native editor, you'll need to [add a field filter or basic variable to your query](../questions/native-editor/sql-parameters.md).

View File

@@ -58,32 +58,29 @@ If there are no results, you can tell Metabase to skip sending the email.
Here you can set filters to apply when Metabase sends the subscription.
## Attach results
### Attach results
When configuring a subscription, you can tell Metabase if it should attach results to the email as a file, in addition to displaying the table in the email body.
Tell Metabase if it should attach results to the email as a file, in addition to displaying the table in the email body.
If you've added filters to your dashboard and set default values for those filters, Metabase will apply those default values to your subscriptions, filtering the results of all questions that are connected to those filters when the subscriptions are sent. To learn more, check out [dashboard filters](./filters.md).
### File format
#### File format
You can choose between the following file formats:
- .csv
- .xlsx
- .json
### Use unformatted values in attachments
#### Use unformatted values in attachments
If you check this box, Metabase will drop any visualization settings applied to the questions' results (e.g., date formatting) and send only the raw, unformatted results.
### Questions to attach
#### Questions to attach
Here you can specify which questions Metabase should attach results for. If you don't select any questions, Metabase won't send any attachments.
Here you can specify which questions Metabase should attach results for.
The attached files will include up to 1 048 575 rows by default (Excel's row limit for spreadsheets). If you're self-hosting Metabase, you can adjust this row limit by setting the environment variable [`MB_ATTACHMENT_ROW_LIMIT`](../configuring-metabase/environment-variables.md#mb_attachment_row_limit). To change this row limit on your Metabase Cloud instance, you can [contact us](https://www.metabase.com/help-premium) and request a different row limit.
### Send only attachments (no charts)
You can toggle on "Send only attachments (no charts)" if you want to receive just the .csv or .xlsx files without the charts. Remember to select the results you want to attach in "Questions to attach" above the toggle.
The attached files will include up to 1 048 575 rows by default. If you're self-hosting Metabase, you can adjust this row limit by setting the environment variable [MB_ATTACHMENT_ROW_LIMIT](../configuring-metabase/environment-variables.md#mb_attachment_row_limit). To change this row limit on your Metabase Cloud instance, you can [contact us](https://www.metabase.com/help-premium) and request a different row limit.
## Slack subscription options

View File

@@ -1,85 +0,0 @@
---
title: Editable tables
summary: Admins can create, update, and delete data in tables right in Metabase.
---
# Editable tables
{% include plans-blockquote.html feature="Editable table data" %}
Admins can enable editable table data in Metabase.
![Edit table](./images/edit-table.png)
## Setting up editable tables
_Currently, editable table data is only available for PostgreSQL and MySQL._
To set up table editing, admins can:
1. Click the **gear** icon in the top right.
2. Select **Admin settings**.
3. Click **Databases**.
4. Select your database.
5. Toggle on **Editable table data**.
For this to work, Metabase's connection to your database must support writing back to its tables. See [privileges to enable actions and editable table data](../databases/users-roles-privileges.md#privileges-to-enable-actions-and-editable-table-data).
## Update or delete records in a table
To update or delete records in a table:
1. Open the left sidebar.
2. In the Data section, click **Databases**.
3. Hover over the table you want to edit and click the **Pencil** icon.
4. Edit rows in the table just like you would in a spreadsheet.
Depending on the data in the column, Metabase may show a list of values to choose from. If you need to add an option to that list, simply type out the option and add it (you may need to refresh the browser for the new option to show up in the list).
![Add option](./images/add-option.png)
To remove an option from the list of values, you'll need to change any records with that value to a different value.
## Create a new record
![Create new record](./images/create-new-record.png)
You can add a row to a table by clicking **+ New record** in the top right of the table. Fill out the fields. Metabase will respect field constraints, so if a field is required, Metabase will prevent you from saving the record until you fill in the required field.
## Delete a record
To delete a record, check the box next to the record you want to delete and click **Delete**. Deletion is permanent. Metabase will prompt you to confirm the deletion.
You can also select multiple rows and delete them all at once (nervous sweating).
If you try to delete a row that has foreign key references from other tables, your database may prevent the deletion to maintain referential integrity. You'll see an error message like "Other rows refer to this row so it cannot be deleted." To delete these rows, you'll either need to remove the referencing rows first, or change the foreign key constraints in your database.
## Metabase enforces type constraints
- You can't edit sequential primary keys.
- If you try to add a value of a different type (like changing the number 13579 to the string "Wizard"), Metabase won't allow it.
## Editable table data vs. actions
Metabase offers two ways to modify data in your database: **editable table data** and [actions](../actions/introduction.md).
### When to use editable table data
Use editable table data to:
- Do data maintenance or cleanup.
- Modify many records at once.
- Get direct access to all table data.
- Do one-off data corrections.
You need to be an admin to edit tables directly.
### When to use actions
Use actions to:
- Create custom forms.
- Give non-admins the ability to modify data.
- Add a button to a dashboard to trigger a form or query.
See [actions](../actions/introduction.md).

Binary file not shown.

Before

Width:  |  Height:  |  Size: 98 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 287 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 84 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 47 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 237 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 129 KiB

View File

@@ -40,14 +40,10 @@ Models you create are automatically [pinned to the current collection](../explor
### Create a model from scratch
- Navigate to the Models tab in the sidebar. You might have to open it using the button in the top left, then scroll down to the section labeled **Data**, and pick **Models**. Then click on the **+** button in the top right.
- Or open the [command palette](https://www.metabase.com/docs/latest/exploration-and-organization/exploration#command-palette) and type "model." Then click on the **New model** action.
Now choose either the query builder or a native query (if you want to use SQL). The advantage of using the query builder is that Metabase will be able to fill out some of the metadata for you; if you use SQL, you'll have to fill out that metadata manually.
Next, select your data, create your query, and save it.
Models you create are automatically [pinned to the current collection](../exploration-and-organization/collections.md#pinned-items).
1. In the upper right, click **New +** > **Model**.
2. Choose either the query builder or a native query (if you want to use SQL). The advantage of using the query builder is that Metabase will be able to fill out some of the metadata for you; if you use SQL, you'll have to fill out that metadata manually.
3. Select your data.
4. Create and save your query.
### Create a model from a saved question
@@ -119,46 +115,6 @@ There are some limitations to this indexing:
You can edit a model's query by clicking on the down arrow next to the model's name and clicking on **Edit query definition**. When you're doing editing, be sure to save your changes. Unlike questions, which prompt you to save as a new question, any changes here will overwrite the existing model. If you want to create a new model from an existing model, select **Duplicate this model** from the model sidebar (the icon of two overlapping squares).
## Model list view
![Viewing a model as a list, with a detail view](./images/model-list-detail.png)
To view a model as a list:
1. Visit the model.
2. Click the three-dot menu.
3. Select **Edit metadata**.
4. Navigate to the **Settings** tab.
5. Under "What should the default view of this data be?", toggle to **List**.
![Model list view](./images/model-list-view.png)
### Customize model list view
![Customize list layout](./images/customize-list.png)
You can customize how the data appears by clicking **Customize the List layout**.
Each item in the list has:
- An entity icon
- A left column with:
- The title or primary identifier on top.
- A subtitle/description or secondary identifier.
- An image (only if the record contains an image link)
- A right column for up to 5 additional columns.
You can:
- **Search for columns** using the "Find a column..." search box
- **Drag columns** from the available list into either the left (Title + Subtitle) or right columns areas
- **Reorder columns** by dragging them within their respective areas
- **Remove columns** by clicking on the X on the column.
You can see a preview on the bottom with sample data from your model.
Click **Done** to save your changes.
## Start a question from a model
See [asking questions][question].

View File

@@ -101,7 +101,7 @@ Admins, and people with [permission to manage table metadata](../permissions/dat
Data types can't be edited in Metabase directly, but you can cast certain [data types to different types](./metadata-editing.md#cast-to-a-specific-data-type) so that, for example, Metabase will interpret a text data type as a date type.
Changes made in Table Metadata apply across your entire Metabase. However, if you build a query in the query builder, you can use type casting custom expressions like [`date()`](../questions/query-builder/expressions-list.md#date) or [`integer()`](../questions/query-builder/expressions-list.md#integer) to cast a string to a different type just in your query.
Changes made in Table Metadata apply across your entire Metabase. Metabase currently only supports casting to a datetime type in Metadata settings. However, if you build a query in the query builder, you can use type casting custom expressions like [`date()`](../questions/query-builder/expressions-list.md#date) or [`integer()`](../questions/query-builder/expressions-list.md#integer) to cast a string to a different type in your query.
### Semantic types don't change the data types

Some files were not shown because too many files have changed in this diff Show More