Compare commits

..

156 Commits

Author SHA1 Message Date
Nathan Voxland
21630bdf55 Moved redis to enterprise code 2026-05-28 17:07:50 -05:00
Nathan Voxland
57d1a6524c Fixing redis tests 2026-05-21 16:46:27 -05:00
Nathan Voxland
ab23a4f698 Merge branch 'persistent-queue-only' into persistent-queue-redis 2026-05-21 15:43:25 -05:00
Nathan Voxland
68c9ae363a Update mq-perf benchmark for the refactored queue API
- start poll threads via q.backend/start! (start-transports was removed).
- stale-recovery bench calls the recover-stale! protocol method.
- encode payloads before direct publish! and seed the :payload column
  (encoding was centralized / column renamed upstream).
- read the keyword :publish-ns key (decode keywordizes message keys), so
  latency stats are populated again.
2026-05-21 15:17:09 -05:00
Nathan Voxland
649dfb6f6e Fix CI: trailing blank line and stale mq module :api entry
- mq/impl.clj ended in a blank line (whitespace-linter).
- The mq module's :api listed metabase.mq.core, which no other module
  uses, so modules-config-up-to-date-test flagged it as extraneous. Drop
  it, leaving metabase.mq.init (the only externally-used mq namespace).
2026-05-21 15:16:49 -05:00
Nathan Voxland
62d2cf48e2 Add Redis queue backend
A third durable QueueBackend backed by Redis Streams + consumer groups, for
deployments that want to offload queue traffic off the app DB. Selected via
queue-backend=redis (mq-redis-uri / -username / -password settings).

- Uses the stream entry-id as the batch-id (server is the source of truth);
  retry count rides in the entry's retries field.
- Exclusive queues use an atomic Lua PEL-empty-then-read so exactly one batch
  is in-flight cluster-wide, race-free.
- Crash recovery via XAUTOCLAIM; in-flight leases refreshed with XCLAIM.
- Lazy-loaded so Carmine is only touched when redis is selected.
- Infra: carmine dep, docker-compose service, .env.example, devenv docs,
  dedicated CI workflow; parity test exercises :redis when a server is reachable.
2026-05-21 14:04:41 -05:00
Nathan Voxland
7a526d8834 Unify queue backends behind a shared poll driver
Merge PollableQueueBackend into a single QueueBackend protocol and move all
shared poll-loop work into metabase.mq.queue.polling: maintenance tasks,
metric emission (depth gauges, poll-results, stale-recovery), the max-retries
decision, and the stale-processing timeout. Backends only supply data.

- appdb: delete failed/permanently-failed rows instead of retaining them;
  drop the now-unused cleanup! path.
- memory: track individual messages in an id-keyed sorted-map (oldest = lowest
  id), mirroring the appdb row model, instead of a LinkedBlockingQueue.
- transport wakes the poll loop via (:poll-context backend), breaking the
  mq.impl -> transport require cycle.
2026-05-21 14:02:51 -05:00
Nathan Voxland
4ba5a62f86 Centralize MQ message encoding at the publish/deliver boundaries 2026-05-20 14:43:43 -05:00
Nathan Voxland
89b6eac634 Separated out def-queue from def-listener 2026-05-20 02:20:33 -05:00
Nathan Voxland
07050d2476 Merge remote-tracking branch 'origin/master' into persistent-queue-only
# Conflicts:
#	test/metabase/metabot/api_test.clj
2026-05-19 14:16:58 -05:00
Nathan Voxland
d47d21fa8e Took out dedup example 2026-05-19 14:14:35 -05:00
Nathan Voxland
fd17c5fd58 Merge remote-tracking branch 'origin/master' into persistent-queue-only 2026-05-19 12:38:14 -05:00
Nathan Voxland
3516987977 Split out topic code to persistent-queue-topic branch
Removes the mq.topic.* namespaces, topic-specific shared-file references,
the topic migration changesets, the topic Prometheus gauge, and the topic
benchmarks in dev.mq-perf. The remaining mq subsystem is queue-only; the
topic delta lives on the persistent-queue-topic branch and will rebase
onto master after this branch lands.
2026-05-19 12:31:25 -05:00
Nathan Voxland
04b7c5bf4d Fix CI failures: portable fetch query, force-flush buffer on stop!
- queue/appdb fetch!: rewrite the per-queue head selection as N small SELECT FOR
  UPDATE SKIP LOCKED queries inside one transaction, instead of a single LATERAL
  + VALUES query. Postgres, MySQL 8+, and H2 all support LATERAL but MariaDB does
  not, so the LATERAL version broke the MariaDB / MySQL test jobs. The per-queue
  loop is N+1 round trips per poll iteration, but N is bounded by the registered
  listener count (typically <10) and SKIP LOCKED still applies per row, so
  concurrent consumers still get different rows. Empty result correctly returns
  nil (was leaking [] in the new version).

- mq.init/stop! handle: always force-flush the (currently bound) publish buffer
  before checking executor ownership. Previously, in CI test mode where
  `metabase.test.initialize/mq` claimed ownership of the global flush executor
  first, the shutdown-drain tests' own stop! call ran with owns-buffer-flush?
  = false, so it skipped the flush — and their bound *publish-buffer* never
  drained to the DB. The fix runs `(flush-publish-buffer! true)` unconditionally,
  which drains whatever atom is currently bound. The executor shutdown remains
  ownership-gated.
2026-05-18 17:34:46 -05:00
Nathan Voxland
392a3dbe3d mq_perf: add RTT injection and concurrent pub+consume benchmark
- install-rtt! / uninstall-rtt! / with-rtt-ms: wrap the application DB's DataSource
  with a JDBC proxy that sleeps a randomized [min-ms max-ms] window before every
  execute*. Setting is a global atom (rtt-ms), not a dynamic var, so it applies to
  every thread — poll threads, worker pool, publisher threads alike — without
  needing per-thread binding conveyance.

- bench-concurrent-pub-consume!: publisher thread sustained at a target rate runs
  concurrently with the consumer. Each message carries publish-ns and the listener
  measures wall-clock latency. Exercises buffer-atom contention under writes and
  the per-channel active-handlers gate while publishing is still happening.

Findings from running the suite with injected RTT:
  - Publish throughput is roughly linear in RTT for batch=1 (922 -> 163 -> 82 msg/s
    at 0 / 1-2 / 5-10 ms RTT) but degrades much less at batch=100 (93k -> 17k -> 8k
    msg/s — still strong). Batch sizes >= 100 are the only viable setting for
    high-RTT environments.
  - Multi-queue scaling is essentially unaffected by RTT because the buffered
    publish path collapses most messages into a few rows. The DB write cost
    amortizes across hundreds of messages.
  - Bulk stale recovery scales gracefully under RTT (39ms -> 184ms for 500 rows at
    0 / 5-10 ms RTT). The per-row pre-fix version would have been 5-10s for the
    same workload at cloud RTT — confirming the bulk-update rewrite's value.
  - Concurrent pub+consume p50 latency is surprisingly insensitive to RTT
    (1045 ms baseline vs 1102 ms at 5-10 ms RTT). The dominant cost is the
    5-second consumer poll interval, not the DB round trip. To improve
    end-to-end latency on cloud DBs, lower the poll interval — not the RTT.
2026-05-18 16:35:27 -05:00
Nathan Voxland
7c34ef9b02 mq_perf: validate concurrency properly; fix bulk-recover return semantics
- bench-stale-recovery-race! now correctly reports rows ACTUALLY updated, not
  candidates seen. With 8 racing workers and 500 stale rows, sum-of-counts =
  500 (exactly-once recovery confirmed). Per-row 0.11ms — 12× faster than
  the pre-fix per-row loop on the same workload.

- recover-stale-processing-batches! returns the affected-row count from the
  bulk UPDATE, not the candidate-row count. Docstring clarifies that
  concurrent callers will see smaller numbers because the guard rejects rows
  already recovered by another node.

- bench-concurrent-consumers!: force *publish-buffer-ms* to 0 in the publish
  phase so each put becomes its own row (otherwise the time-window buffer
  collapses everything into one row that only one consumer can grab).
  Docstring updated to flag that this test validates exactly-once delivery
  and LATERAL fetch correctness, but does NOT measure cross-node throughput
  — `mq.impl/active-handlers` is a JVM-global atom so only one consumer can
  be processing per channel within a single JVM. Multi-JVM testing is still
  needed for real horizontal-scaling numbers.
2026-05-18 16:09:44 -05:00
Nathan Voxland
6eae581864 mq_perf: add concurrent-consumer and stale-recovery-race benchmarks; fix reflection warning
- New bench-concurrent-consumers!: spins up N AppDbQueueBackend instances in the
  same JVM, each with its own listener atom + poll thread, all sharing one queue.
  Validates that the LATERAL/SKIP LOCKED fetch distributes work across consumers
  and never double-processes a batch. Made possible by the per-instance backend
  refactor.
- New bench-stale-recovery-race!: pre-populates N stale processing rows and races
  W threads to recover them simultaneously. Confirms the guarded bulk UPDATE
  hands each row to exactly one winner.
- mq_perf: replace the removed mq/listen!/mq/batch-listen! calls with
  listener/batch-listen! (with explicit doseq when per-message handling is wanted).
- mq_perf: fix start-mq! to dispatch the correct ::MqStart key (was ::MqInit which
  never existed).
- mq.impl/submit-delivery!: pull the bound-fn into a let-bound ^Callable so the
  ExecutorService.submit dispatch resolves at compile time (eliminates the
  reflection warning).
2026-05-18 15:55:28 -05:00
Nathan Voxland
8ed89e869e Lint fixes 2026-05-18 14:45:02 -05:00
Nathan Voxland
549d278492 MQ QA fixes: backend per-instance state, async size-flush, bulk stale recovery, listener API cleanup
- AppDbQueueBackend / AppDbTopicBackend now carry owner-id, poll-state, offsets,
  and periodic-task rate-limit atoms as record fields. Production singleton uses
  make-backend; test fixture constructs its own. Test teardown no longer kills
  the production poll thread.
- mq.impl/start-worker-pool! and publish-buffer/start-publish-buffer-flush! return
  true if THIS call created the resource. mq.init handle records ownership so
  test stop! only shuts down what it owns; production globals survive test
  teardown.
- publish-buffer/buffered-publish! drops the synchronous flush when buffer hits
  max-batch-messages — just trips the channel's deadline and lets the 100ms
  scheduler do the work. Publisher threads never block on backend writes.
- recover-stale-processing-batches! does two bulk UPDATEs (retryable + permanent)
  instead of a per-row loop. Summary log per group plus per-queue analytics.
- topic.appdb publish! uses t2/insert! (was t2/insert-returning-pk! with the PK
  discarded).
- listener.clj: remove the single-message listen! function. def-listener! always
  routes to batch-listen!; listener body always receives a vec. batch-listen!
  no longer in mq.core public API (kept public in mq.listener for the macro to
  resolve). def-listener! docstring spells out why it's preferred over direct
  batch-listen! calls.
- Tests: replace mq/listen! calls with mq.tu/listen! (a test helper that wraps
  batch-listen! with per-message dispatch and error isolation). queue/core_test
  switches mq/batch-listen! → listener/batch-listen!.
2026-05-18 11:31:27 -05:00
Nathan Voxland
74770efa10 MQ refactors: ordered startup, generic start-receiving, buffer-side batch sizing
- startup/core: add startup-priority multimethod so def-startup-logic! methods
  can request ordering (default 500). MQ start now runs via def-startup-logic!
  with Long/MAX_VALUE instead of an explicit call from core.core.
- TopicBackend: split subscribe!/start! into start-receiving! (per-topic,
  idempotent, early) and start-handling! (per-backend, late) to model the
  two-phase lifecycle real brokers need. unsubscribe! kept as the destructive
  teardown.
- mq.init: rename capture-topic-offsets! to start-receiving!, expose via
  mq.core public API; core.core calls mq/start-receiving! generically.
- queue/appdb fetch!: rewrite the multi-queue pick as a LATERAL join with
  per-queue FOR UPDATE SKIP LOCKED so concurrent consumers actually share
  work instead of contending on the same head-of-queue.
- Move max-batch-messages partitioning out of queue/appdb publish! and into
  the publish pipeline. Backend publish! now just stores what it's given.
  buffered-publish! chunks on the sync path and flushes a channel when its
  buffered count hits max-batch-messages.
- listener: default max-batch-messages now applied in batch-listen! so both
  listen! and batch-listen! callers get the bound.
2026-05-18 04:53:27 -05:00
Nathan Voxland
0dc06ff1e3 Fix failing test 2026-05-13 10:43:23 -05:00
Nathan Voxland
9ee6dac54a Merge branch 'master' into persistent-queue-mq 2026-05-13 00:12:06 -05:00
Nathan Voxland
8ff89b543c Removed core from mq api until it gets used somewhere 2026-05-12 19:08:48 -05:00
Nathan Voxland
2bb669708f Code review improvements 2026-05-12 17:57:10 -05:00
Nathan Voxland
524e6bf086 Switch MQ metrics to analytics-interface
Master moved Prometheus helper fns (inc!, set!/set-gauge!, observe!)
out of metabase.analytics.core and into metabase.analytics-interface.core
(see #72262), and the rename broke the namespace-decl check on this
branch. Point all metabase.mq.* files (and their tests) at the new
namespace, rename set! → set-gauge!, and add analytics-interface to
the mq module's :uses set.
2026-05-12 15:46:58 -05:00
Nathan Voxland
ecd77df06b Moved persistent queue tables to v62 2026-05-12 15:29:01 -05:00
Nathan Voxland
5ad4b679e8 Merge remote-tracking branch 'origin/master' into persistent-queue-mq 2026-05-12 15:25:38 -05:00
Nathan Voxland
5b0b0ac624 Address PR review comments
- test-util: do-with-test-mq! now destructures :backend (not :kind),
  so parity tests actually exercise both backends.
- memory queue: persist failure count per channel across retries so
  queue-max-retries is reachable instead of looping forever.
- mq/impl.clj submit-delivery!: keep the swap! update fn side-effect
  free, use swap-vals! to detect ownership.
- publish-buffer: prepend retried (older) messages on flush failure
  so publish order survives transient errors.
- queue/appdb: stale-recovery update! now guards on status +
  status_heartbeat so concurrent recovery attempts only win once.
- topic/appdb: document that offsets advance on claim, not on
  listener completion.
- README: with-queue / with-topic examples now use the binding form.
- test init: drive test-mode MQ through mq.init/start! so the worker
  pool, publish-buffer flush, and memory polling loop actually run.
2026-05-12 15:17:48 -05:00
Nathan Voxland
e8640a399a Added metabase.mq module readme 2026-04-20 16:05:23 -05:00
Nathan Voxland
e3cb29170e Lint fixes 2026-04-16 11:03:23 -05:00
Nathan Voxland
fea5506519 Lint fixes 2026-04-16 09:58:46 -05:00
Nathan Voxland
582ff189f7 Added ability to do double-delivery testing in with-test-mq 2026-04-16 08:51:58 -05:00
Nathan Voxland
358d7eec28 Added backend parity testing and removed sync backends 2026-04-15 14:20:36 -05:00
Nathan Voxland
dff14540d2 Use memory queue for testing, not sync queue 2026-04-15 11:51:56 -05:00
Nathan Voxland
8103ca8a48 Cleaning up mq code by switching to records from as much multimethods 2026-04-14 16:51:44 -05:00
Nathan Voxland
8282cee525 Merge remote-tracking branch 'origin/persistent-queue-mq' into persistent-queue-mq 2026-04-14 14:29:12 -05:00
Nathan Voxland
618831d368 Finished cleaning up "bundle" terminology to "batch" 2026-04-14 14:27:51 -05:00
Nathan Voxland
81e055c0cd Fix after-commit callbacks surviving nested transaction rollbacks; add publish buffer retry limit 2026-04-09 18:42:56 -05:00
Nathan Voxland
a5915a33b4 - Cleaned up now-unneeded analytics shim
- Cleaned up now-unneeded lazy requires
- Cleaned up start/stop logic
2026-04-07 15:06:38 -05:00
Nathan Voxland
fdea81a6bb Removed all changes that use the MQ code 2026-04-06 10:12:59 -05:00
Nathan Voxland
6fad3ef45c Merge remote-tracking branch 'origin/master' into persistent-queue-mq 2026-04-06 09:52:37 -05:00
Nathan Voxland
5ad741c0d2 Cleaning up code 2026-04-01 13:58:39 -05:00
Nathan Voxland
69570bc65f Cleaning up code 2026-03-31 20:15:55 -05:00
Nathan Voxland
b719615aa6 Cleaning up code 2026-03-31 17:09:51 -05:00
Nathan Voxland
afcc49f6f6 Refactor MQ test init: use :mq initialization step instead of is-test? checks
Replace scattered is-test? checks at var definition time with a proper
:mq initialization step in the test startup flow. This mirrors how
production initializes MQ via MqInit in startup/run-startup-logic!.

The :mq step:
- Loads driver.init (so def-listener! namespaces are loaded)
- Calls mq.init/init-for-tests! which sets sync backends, disables
  publish buffering, and registers all listeners

Also:
- Reverts var defaults to production values (appdb backends, 100ms buffer)
- Reverts def-listener! to lazy registration (no eager call)
- Restores with-sync-mq to use fresh *listeners* atom
- Removes all [TZ-DEBUG] diagnostic traces
- :db initialization depends on :mq

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-31 16:00:26 -05:00
Nathan Voxland
d12184f4a0 Fix eager listener registration: call multimethod, not method map lookup
The eager registration code used ((methods def-listener*) channel) which
returns the method fn from the methods map but doesn't invoke it. Changed
to (def-listener* channel) which dispatches through the multimethod and
actually calls the registration function.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-31 14:08:09 -05:00
Nathan Voxland
bde8a43aeb Add tracing to :db init to verify driver.init loading
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-31 13:27:56 -05:00
Nathan Voxland
76f8cb6ce3 Don't rebind *listeners* to empty atom in with-sync-mq
with-sync-mq was binding *listeners* to (atom {}), which created a
fresh empty atom that shadowed the root atom where listeners were
eagerly registered at def-listener! time. Since eftest propagates
bindings to worker threads, parallel tests running on threads spawned
within a with-sync-mq scope would see the empty atom and fail to find
any listeners.

Fix: stop rebinding *listeners*. Listeners are eagerly registered into
the root atom at namespace load time and should remain globally visible.
with-sync-mq still binds backends and buffer settings as needed.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-31 12:55:18 -05:00
Nathan Voxland
31840a1a31 Add atom identity tracing to diagnose listener registration
Trace which *listeners* atom instance is used during registration vs
lookup to detect if a dynamic rebinding is causing listeners to be
registered in one atom but looked up in another.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-31 12:35:43 -05:00
Nathan Voxland
a5968cdf24 Load driver/init.clj during test :db initialization
The MQ listeners defined in driver_notifications.clj (e.g.
connection-pool-invalidated) are only loaded via driver/init.clj, which
is loaded via core/init.clj in production. Tests don't load core/init.clj,
so these listener namespaces are never required and the eager registration
from def-listener! never happens.

Fix: require driver.init during test :db initialization so the listener
namespaces get loaded and listeners are registered.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-31 11:37:56 -05:00
Nathan Voxland
5bcdea9b51 Eagerly register MQ listeners at def-listener! time
def-listener! now immediately registers the listener when the namespace
loads, instead of deferring to register-listeners! called from MqInit.
This ensures listeners are available in test mode where MqInit never runs
(tests use initialize-if-needed!, not startup/run-startup-logic!).

register-listener! is idempotent so the production path (MqInit calling
register-listeners!) still works fine.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-31 10:37:11 -05:00
Nathan Voxland
8eafca6c74 Add tracing to invoke-listener! to diagnose missing listener
The sync backend publish fires but listener output never appears.
Adding trace to see if listener is registered or not.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-31 09:57:50 -05:00
Nathan Voxland
6adf7f2ef8 Default MQ backends to sync in test mode
Same issue as *publish-buffer-ms*: MqInit never runs in tests, so the
alter-var-root that sets backends to sync has no effect. Set defaults
at var definition time based on config/is-test?.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-31 08:59:56 -05:00
Nathan Voxland
2ea7960d5c Default *publish-buffer-ms* to 0 in test mode
MqInit never runs in tests (tests use initialize-if-needed!, not
run-startup-logic!), so the alter-var-root in the previous fix had
no effect. Instead, set the default value at var definition time
based on config/is-test?.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-31 01:26:49 -05:00
Nathan Voxland
66af12b9cd Add more MQ tracing to diagnose CI failures
Traces at buffered-publish!, transport/publish!, sync-backend/publish!,
and MqInit to understand why *publish-buffer-ms* is still 100 in CI
(meaning messages are buffered instead of delivered immediately).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-31 01:02:26 -05:00
Nathan Voxland
8171f24196 Debugging CI failures 2026-03-31 00:27:49 -05:00
Nathan Voxland
a86dffa344 Fix MQ topic messages silently dropped in test mode
In test mode, MqInit sets sync backends but leaves *publish-buffer-ms*
at its default of 100ms. This causes buffered-publish! to buffer messages
instead of delivering them immediately. Since start-publish-buffer-flush!
is never called in test mode, no background thread drains the buffer,
so topic messages are silently dropped.

This broke report-timezone change notifications — connection pools were
never invalidated when timezone changed, causing stale timezone sessions
on database connections. This manifested as timezone-shifted query results
across 28+ driver tests in CI.

Fix: set *publish-buffer-ms* to 0 in test init path so messages publish
immediately without buffering.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-30 22:56:59 -05:00
Nathan Voxland
db09e70823 Debugging CI failures 2026-03-30 20:27:04 -05:00
Nathan Voxland
407f3bc2df Merge branch 'master' into persistent-queue 2026-03-29 18:22:27 -05:00
Nathan Voxland
5b1181eba5 Fixed module setup 2026-03-28 01:40:24 -05:00
Nathan Voxland
1574a28b4f Fixed tests 2026-03-28 01:26:44 -05:00
Nathan Voxland
ace1e034a2 Fixed tests 2026-03-27 15:50:33 -05:00
Nathan Voxland
9142569d91 Fixed tests 2026-03-27 14:58:23 -05:00
Nathan Voxland
30c6275dc5 Fixed tests 2026-03-27 14:05:41 -05:00
Nathan Voxland
0f2e7ea906 Fixed tests 2026-03-27 13:51:36 -05:00
Nathan Voxland
cff57cbbb5 Merge remote-tracking branch 'origin/master' into persistent-queue
# Conflicts:
#	enterprise/backend/test/metabase_enterprise/agent_api/api_test.clj
2026-03-27 13:38:08 -05:00
Nathan Voxland
62c13bb028 Fixed formatting 2026-03-27 13:16:24 -05:00
Nathan Voxland
8f7064e87f Fixing tests 2026-03-27 12:56:26 -05:00
Nathan Voxland
27e5f298d8 Fixing tests 2026-03-27 09:27:36 -05:00
Nathan Voxland
c6dff735a9 Fixing tests 2026-03-26 18:13:35 -05:00
Nathan Voxland
daaca9ef62 CI fixes 2026-03-26 12:30:15 -05:00
Nathan Voxland
ad9b980fef Merge remote-tracking branch 'origin/master' into persistent-queue 2026-03-26 11:12:10 -05:00
Nathan Voxland
96a308a64e CI fixes 2026-03-26 11:11:47 -05:00
Nathan Voxland
3c24edbf47 Merge remote-tracking branch 'origin/master' into persistent-queue
# Conflicts:
#	src/metabase/settings/models/setting.clj
2026-03-26 09:33:55 -05:00
Nathan Voxland
93b375bbe2 Merged from master 2026-03-26 01:33:25 -05:00
Nathan Voxland
ca3a2ea136 Merge remote-tracking branch 'origin/master' into persistent-queue
# Conflicts:
#	src/metabase/search/core.clj
#	src/metabase/search/ingestion.clj
#	src/metabase/search/task/search_index.clj
#	src/metabase/testing_api/api.clj
2026-03-25 23:49:31 -05:00
Nathan Voxland
e9f6c93013 Improved performance benchmark and added back skip lock 2026-03-25 21:17:35 -05:00
Nathan Voxland
89786605c1 Fixing tests and created a mq-perf script 2026-03-25 15:19:29 -05:00
Nathan Voxland
82f5dea54d Code cleanup 2026-03-25 02:28:42 -05:00
Nathan Voxland
ad62aed71a Cleaning up code 2026-03-24 02:15:25 -05:00
Nathan Voxland
3ab76adf8d Merged from master 2026-03-24 01:36:35 -05:00
Nathan Voxland
1f019c1e10 Merge remote-tracking branch 'origin/master' into persistent-queue
# Conflicts:
#	enterprise/backend/src/metabase_enterprise/action_v2/data_editing.clj
#	enterprise/backend/test/metabase_enterprise/action_v2/api_test.clj
#	enterprise/backend/test/metabase_enterprise/agent_api/api_test.clj
#	enterprise/backend/test/metabase_enterprise/serialization/v2/e2e_test.clj
#	enterprise/backend/test/metabase_enterprise/serialization/v2/load_test.clj
#	src/metabase/analytics/prometheus.clj
2026-03-24 01:28:52 -05:00
Nathan Voxland
3e4335a798 Merge remote-tracking branch 'origin/persistent-queue' into persistent-queue 2026-03-24 01:19:44 -05:00
Nathan Voxland
70967125aa Code cleanup 2026-03-24 01:16:36 -05:00
Nathan Voxland
5921542894 Code cleanup 2026-03-24 01:05:12 -05:00
Nathan Voxland
8c572f77f4 Fix race conditions in submit-delivery! and topic offset advancement
- submit-delivery!: atomic busy-check and registration via single swap!
  to prevent concurrent handlers for the same channel
- start-worker-pool!: use compare-and-set! to prevent pool leak on
  concurrent initialization
- Topic backends: only advance offset when submit-delivery! returns true,
  preventing silent message loss when channel is busy
2026-03-23 19:10:23 -05:00
Nathan Voxland
7e5bb21c3d Fix high-priority issues: retry flush failures, non-blocking batch processing, atomic after-commit drain
- Publish buffer: re-buffer messages on flush failure instead of dropping them, with Prometheus counter
- Queue/topic processing: centralized non-blocking dispatch via shared worker pool in mq.impl, with busy-channel tracking so poll threads never block on listener execution
- Queue heartbeats: poll thread updates status_heartbeat every 2min for in-flight batches to prevent stale recovery during long processing
- After-commit callbacks: atomically drain with reset-vals! loop to support callbacks that register new callbacks
- Document all-or-nothing ACK semantics on queue bundle retry in handle! docstring
2026-03-23 18:58:14 -05:00
Nathan Voxland
2c8c0e30aa Better consolidation of topic vs queue logic 2026-03-23 14:44:03 -05:00
Nathan Voxland
4ca03e17ef Took out now-unneeded search delay 2026-03-18 02:05:25 -05:00
Nathan Voxland
ec2b0f38ae Merge remote-tracking branch 'origin/master' into persistent-queue
# Conflicts:
#	src/metabase/analytics/prometheus.clj
2026-03-18 01:46:46 -05:00
Nathan Voxland
2bad85564e Publish messages after an appdb transaction. Plus batch them still for 100ms to handle times we don't really have a transaction to avoid so many small batches 2026-03-18 01:45:26 -05:00
Nathan Voxland
d1252c7e8e Fixed merges from master 2026-03-13 11:59:15 -05:00
Nathan Voxland
4a7059cac7 Merge remote-tracking branch 'origin/master' into persistent-queue
# Conflicts:
#	resources/migrations/059_update_migrations.yaml
#	src/metabase/search/appdb/core.clj
#	src/metabase/search/core.clj
#	src/metabase/search/impl.clj
#	src/metabase/search/ingestion.clj
#	src/metabase/search/task/search_index.clj
2026-03-13 11:09:28 -05:00
Nathan Voxland
9bd22bf4b1 Added dedup-fn feature 2026-03-10 02:03:18 -05:00
Nathan Voxland
0462cde56d Fixed kondo 2026-03-09 11:14:43 -05:00
Nathan Voxland
e99312fe64 Renamed def-listener to def-listener! 2026-03-09 10:24:36 -05:00
Nathan Voxland
d8a7cd3501 Fixing tests 2026-03-08 01:41:38 -06:00
Nathan Voxland
2d2c4b35a5 Fixing tests 2026-03-07 01:25:05 -06:00
Nathan Voxland
9e717a6056 Merge remote-tracking branch 'origin/master' into persistent-queue
# Conflicts:
#	resources/migrations/059_update_migrations.yaml
2026-03-07 00:09:19 -06:00
Nathan Voxland
9ed603e6ce Better multi-backend handling 2026-03-06 17:30:39 -06:00
Nathan Voxland
8a5545ad32 Fixed whitespace and migrations 2026-03-06 16:01:30 -06:00
Nathan Voxland
82b89e31fb Fixing tests 2026-03-06 15:09:41 -06:00
Nathan Voxland
607985e054 Fixing tests 2026-03-05 22:15:42 -06:00
Nathan Voxland
a2e43e1996 Fixing tests 2026-03-05 21:23:23 -06:00
Nathan Voxland
3b96cd4619 Deps fix 2026-03-05 15:56:10 -06:00
Nathan Voxland
31790b6052 Added def-listener! and fixing tests 2026-03-05 15:42:23 -06:00
Nathan Voxland
846f38d443 Fixing tests 2026-03-05 02:00:51 -06:00
Nathan Voxland
e7a9e8c100 Removed other settings cache sync code 2026-03-05 01:20:07 -06:00
Nathan Voxland
e1b7b8e912 Use new queue better in search 2026-03-05 00:46:36 -06:00
Nathan Voxland
2480ce058b Support exclusive queue listeners 2026-03-04 11:27:19 -06:00
Nathan Voxland
cf04ba48c2 Immediately deliver topic posts locally without waiting for backend roundtrip 2026-03-04 09:51:00 -06:00
Nathan Voxland
8a853116bf Merge remote-tracking branch 'origin/master' into persistent-queue
# Conflicts:
#	.clj-kondo/config.edn
2026-03-04 01:26:56 -06:00
Nathan Voxland
c54c9ded12 improving code 2026-03-04 01:20:12 -06:00
Nathan Voxland
0fc1a4b8c9 Code improvements and removed postgres topic listener 2026-03-04 00:18:19 -06:00
Nathan Voxland
6ca5625902 Code improvements 2026-02-27 12:47:50 -05:00
Nathan Voxland
a4fdcdef64 Code review fixes 2026-02-21 10:21:20 -06:00
Nathan Voxland
b1fe71ca2f Cleaning up force-sync usage 2026-02-21 02:29:02 -06:00
Nathan Voxland
4daf2cb8a2 Added batch-listen! 2026-02-21 01:52:57 -06:00
Nathan Voxland
4a3d739318 Started to migrate old util.queue code to new mq 2026-02-20 15:41:51 -06:00
Nathan Voxland
ba85c0513c Cleanup failed queue messages 2026-02-19 16:55:00 -06:00
Nathan Voxland
b8c26e3c0c Code review fixes 2026-02-19 16:48:09 -06:00
Nathan Voxland
21b5b3e93a Merge remote-tracking branch 'origin/master' into persistent-queue
# Conflicts:
#	resources/migrations/059_update_migrations.yaml
2026-02-19 16:20:28 -06:00
Nathan Voxland
0a1a13b48b Code cleanup 2026-02-19 16:09:11 -06:00
Nathan Voxland
a18f7a94e2 Code cleanup 2026-02-19 15:34:05 -06:00
Nathan Voxland
067626c71d Added metrics 2026-02-19 10:41:42 -06:00
Nathan Voxland
3bc10ee2e1 Cleanly shut down 2026-02-19 09:53:39 -06:00
Nathan Voxland
b9fba6ab7a Cleaning up topic code 2026-02-19 01:15:32 -06:00
Nathan Voxland
20e5611810 Cleaning up topic code 2026-02-18 16:26:15 -06:00
Nathan Voxland
37d68bca75 Made mq.topic code look like mq.queue code 2026-02-18 15:52:48 -06:00
Nathan Voxland
e58e0b7f80 Queue code improvements 2026-02-18 15:21:22 -06:00
Nathan Voxland
81aab7b8ca Added schemas 2026-02-18 14:15:45 -06:00
Nathan Voxland
bcf8a422f6 Queue code cleanup 2026-02-18 14:03:30 -06:00
Nathan Voxland
ac434a50aa Queue code cleanup 2026-02-18 13:02:49 -06:00
Nathan Voxland
4f050fba0a Use notify on postgres 2026-02-18 01:25:29 -06:00
Nathan Voxland
c36a72146e moved queue and pubsub to .mq 2026-02-18 00:27:53 -06:00
Nathan Voxland
d82ec362ba Added pub/sub messaging 2026-02-17 17:04:11 -06:00
Nathan Voxland
c2172311f5 Better queue testing with memory queue 2026-02-17 16:19:34 -06:00
Nathan Voxland
5a7bcfab93 Better naming for storing messages in the appdb queue 2026-02-17 15:37:43 -06:00
Nathan Voxland
decbc82b1a Code review fixes 2026-02-17 14:56:27 -06:00
Nathan Voxland
bd0f5b3b69 Merge remote-tracking branch 'origin/master' into persistent-queue 2026-02-17 14:33:54 -06:00
Nathan Voxland
7f98271a59 Don't have a queue_message model since it's an implementationd detail 2026-01-26 17:04:26 -06:00
Nathan Voxland
8cc219994d Merge remote-tracking branch 'origin/master' into persistent-queue 2026-01-26 16:44:09 -06:00
Nathan Voxland
84168dbf05 Merge remote-tracking branch 'origin/master' into persistent-queue
# Conflicts:
#	resources/migrations/059_update_migrations.yaml
2026-01-26 15:27:21 -06:00
Nathan Voxland
d6da219a70 Fixes from code review and getting back to project 2026-01-16 16:28:37 -06:00
Nathan Voxland
65426fe653 Merge remote-tracking branch 'refs/remotes/origin/master' into persistent-queue
# Conflicts:
#	resources/migrations/059_update_migrations.yaml
2026-01-16 16:11:19 -06:00
Nathan Voxland
6403d80fea Merge branch 'master' into persistent-queue 2026-01-14 11:11:53 -06:00
Nathan Voxland
bbcba1500c Added queue_message table back 2026-01-12 14:29:48 -06:00
Nathan Voxland
fcf4dbb6e8 Merge branch 'master' into persistent-queue
# Conflicts:
#	.clj-kondo/config/modules/config.edn
#	resources/migrations/056_update_migrations.yaml
2026-01-12 14:29:04 -06:00
Nathan Voxland
3d75754ad8 Added sucess and failure handling, general cleanup 2025-07-18 17:06:05 -05:00
Nathan Voxland
af72309d8e e2e test works again 2025-07-17 22:30:10 -05:00
Nathan Voxland
646b4c4548 Reverted metabase.queue changes, implemented metabase.queue.memory, code cleanup 2025-07-15 21:00:16 -05:00
Nathan Voxland
92ada9186c Code cleanup 2025-07-15 14:33:44 -05:00
Nathan Voxland
e67ce4582e Moved migrations 2025-07-14 23:07:20 -05:00
Nathan Voxland
25e3afcdbb Refactoring and trying to get e2e test to pass again 2025-07-14 23:07:20 -05:00
Nathan Voxland
b31a9a09c5 Configured queue listener to use persistent queue 2025-07-14 23:07:20 -05:00
Nathan Voxland
50614c15aa Initial implementation 2025-07-14 23:07:20 -05:00
12818 changed files with 1243203 additions and 1740166 deletions

View File

@@ -22,6 +22,7 @@ You understand Metabase's driver architecture built on Clojure multimethods with
:bigquery-cloud-sdk, :databricks, :clickhouse, :athena, :sparksql,
:presto-jdbc / :starburst, :vertica, :sqlite, :h2
:mongo (non-SQL, custom protocol)
:druid (non-SQL, REST-based)
```
Key driver extension points you know intimately:

View File

@@ -0,0 +1 @@
Fix flakey test $ARGUMENTS

View File

@@ -27,28 +27,9 @@ This guide covers Clojure and ClojureScript coding conventions for Metabase. See
- Format using Markdown conventions
- Reference other vars with `[[other-var]]` not backticks
A docstring states the *contract*: what it returns (including when it returns nil), what its
params mean, what it throws versus swallows, and any precondition or invariant the caller must
respect. A params block for non-obvious or optional args is good documentation — length is not
the problem.
These seven things are, and each is a reason to cut or relocate:
| Anti-pattern | Why | Where it goes instead |
| --- | --- | --- |
| **Facts you don't own** — how a *callee*, another namespace, or another repo works internally | Drifts silently; nothing fails when it goes stale | Cut. Name the callee and let its own docstring say it |
| **Said twice** — ns docstring restating its vars', a caller restating its callee's, sibling fns repeating boilerplate | Two copies means one is already wrong | Cut the copy. One fact, one owner |
| **Design rationale** — why this approach, why it lives in this module, what the alternative was | The caller can't act on it | Commit message, or an inline comment on the specific line it constrains |
| **Development history** — "matching master", "the X branch", "currently", "for now", "used to" | Stale by construction | Cut |
| **Instructions to the reader** — "callers should trust this", "don't bother checking", "you should use X" | State the contract; the caller decides | Cut, keeping any actual guarantee as a plain statement |
| **Invented precision** — score bands, thresholds, or tables no code enforces | Reads as spec, isn't | Cut, or make the code enforce it |
| **A note on the wrong var** — describing how overrides behave, on the base/default impl | Nobody reading the override sees it | Move to the overrides it describes |
**Comments:**
- `TODO` format: `;; TODO (Name YYYY-MM-DD) -- description`
- Prefer an inline comment over docstring prose for anything that explains *the body*. Put it on
the line it explains, not at the top of the function.
## Code Organization
@@ -63,22 +44,6 @@ These seven things are, and each is a reason to cut or relocate:
- Lines ≤ 120 characters
- No blank non-comment lines within definition forms (except pairwise `let`/`cond`)
**Java interop: type-hint parameters in the arglist; prefer `ex-*` helpers:**
```clojure
;; bad — reflection warning, calls Throwable methods directly
(defn root-cause [e]
(if-let [cause (.getCause e)] (recur cause) e))
;; good
(defn- root-cause [^Throwable e]
(if-let [cause (ex-cause e)]
(recur cause)
e))
```
Use `ex-cause`/`ex-message`/`ex-data`/`ex-info` rather than `.getCause`/`.getMessage`/`(.getData ^ExceptionInfo …)` where possible.
## Style Conventions
**Keywords and Metadata:**
@@ -86,37 +51,6 @@ Use `ex-cause`/`ex-message`/`ex-data`/`ex-info` rather than `.getCause`/`.getMes
- Prefer namespaced keywords for internal use: `:query-type/normal` not `:normal`
- Tag variables with `:arglists` metadata if they're functions but wouldn't otherwise have it
**Time/date: `java-time.api :as t` and `u.date`, not `java.time.*` direct interop:**
```clojure
;; bad — Claude default, requires java.time interop
(:import (java.time LocalDate))
... (LocalDate/now) ...
;; good
(:require
[java-time.api :as t]
[metabase.util.date-2 :as u.date])
... (t/local-date) ...
... (u.date/parse "2024-01-01") ...
```
Only `(:import java.time.X)` when interop with a Java method genuinely needs the class.
**Check non-emptiness with `seq`, not `count`:**
```clojure
;; bad — `count` realizes the whole seq into memory just to test emptiness
(let [prompts (map build-prompt items)]
(when (pos? (count prompts))
(t2/insert! :model/Prompt prompts)))
;; good — `seq` realizes one cell; `insert!` streams the rest
(let [prompts (map build-prompt items)]
(when (seq prompts)
(t2/insert! :model/Prompt prompts)))
```
## Tests
**Organization:**
@@ -128,33 +62,6 @@ Only `(:import java.time.X)` when interop with a Java method genuinely needs the
- Mark pure function tests `^:parallel`
**Redefining `defn`-vars under test:**
Use `mt/with-dynamic-fn-redefs` instead of `with-redefs` when stubbing `defn`/`defn-` vars — plain `with-redefs` is not parallel-test-safe and a clj-kondo hook (`:metabase/prefer-with-dynamic-fn-redefs`, see `.clj-kondo/config.edn`) will warn. `with-redefs` is still fine for `def` / dynamic var rebinding.
```clojure
;; bad — flagged by the linter, not safe under :parallel tests
(with-redefs [my.ns/some-fn (fn [_] :stub)] (do-test))
;; good
(mt/with-dynamic-fn-redefs [my.ns/some-fn (fn [_] :stub)] (do-test))
```
**Async tests: no bare `Thread/sleep` — use `deref` with a timeout sentinel:**
A fixed sleep is the #1 source of CI flake. Tests should fail fast and loud when a contract is violated, not hang or sleep through the bug.
```clojure
;; bad — flaky on slow CI, silent on actual failure
(Thread/sleep 300)
(is (= :expected @seen))
;; good — fails fast with a clear message
(let [v (deref some-promise 1000 ::timeout)]
(is (not= ::timeout v) "promise never delivered")
(is (= :expected v)))
```
## Modules
**OSS Modules:**
@@ -179,45 +86,6 @@ A fixed sleep is the #1 source of CI flake. Tests should fail fast and loud when
- Do not cheat module linters with `:clj-kondo/ignore [:metabase/modules]`
**Enterprise/OSS split: use `defenterprise`, not protocols or runtime feature checks:**
The OSS namespace defines the fallback body and names the EE namespace; the EE namespace overrides with the same fn name and a `:feature` clause naming the premium-features flag that gates the override. Both files use the same docstring shape — describe behavior on both sides.
```clojure
;; src/metabase/remote_sync/core.clj (OSS)
(defenterprise collection-editable?
"Returns whether remote-synced collections are editable.
Always true on OSS; EE consults the remote-sync settings."
metabase-enterprise.remote-sync.core
[_collection]
true)
;; enterprise/backend/src/metabase_enterprise/remote_sync/core.clj (EE)
(defenterprise collection-editable?
"EE override — respects the synced-collections-editable? setting."
:feature :remote-sync
[collection]
(or (synced-collections-editable?) (not (synced? collection))))
```
`defenterprise-schema` is the malli-schema variant. Required from `[metabase.premium-features.core :refer [defenterprise defenterprise-schema]]`. The OSS body is what runs when the feature flag is off — never write `(when (premium/has-feature? ...) ...)` in user code; let `defenterprise` route.
**Security gates fail *open* under `:feature` — use `:feature :none` and check manually.** The router (`dynamic-ee-oss-fn`) calls the EE body only when both the EE namespace is loaded *and* the token currently grants the feature; otherwise it falls back to the OSS body. For a permissive default (`collection-editable?` → `true`) that's fine. But for a *restriction* — sandboxing, impersonation, row-level filtering — the OSS fallback is the unrestricted answer, so a temporarily lost/expired token silently downgrades a sandboxed user to full access. **Fail open.** When the EE override is what *decides whether to restrict*, gate it with `:feature :none` (which always routes to the EE body once EE code is loaded) and check the feature explicitly inside, so token loss fails *closed*:
```clojure
;; enterprise/.../sandbox/api/util.clj — the gate that detects whether a user is sandboxed
(defenterprise enforced-sandboxes-for-user
"..."
;; :none so we can decide whether sandboxing is *configured* even when the feature
;; isn't currently available — letting us BLOCK requests that would otherwise be
;; sandboxed, instead of returning the permissive OSS fallback. Fail closed.
:feature :none
[user-id]
...)
```
Reach for this only for the detection/enforcement decision point. Downstream functions that merely *act* on an already-made decision can keep their normal `:feature :sandboxes` gate.
## REST API Endpoints
**Required Elements:**
@@ -226,20 +94,6 @@ Reach for this only for the detection/enforcement decision point. Downstream fun
- All endpoints need Malli schemas for parameters (detailed and complete)
- All new REST API endpoints MUST HAVE TESTS
**Endpoint shape: `api.macros/defendpoint`, four positional arglist:**
```clojure
(api.macros/defendpoint :get "/:key/callback"
"OIDC callback for a specific provider."
[{provider-key :key} :- [:map [:key ProviderKey]]
_query-params
_body
request]
(oidc-integration/sso-callback provider-key request))
```
The arglist is **always 4 positional bindings** in this order: `route-params`, `query-params`, `body`, `request`. Use `_`-prefixed names for unused ones. Each one gets a Malli schema via `:- <schema>` (omit when `_`). The HTTP verb is a keyword (`:get`/`:post`/`:put`/`:delete`), not a symbol. Required as `[metabase.api.macros :as api.macros]`.
**Naming Conventions:**
- Query parameters use kebab-case
@@ -251,20 +105,6 @@ The arglist is **always 4 positional bindings** in this order: `route-params`, `
- `GET` endpoints should not have side effects (except analytics)
- `defendpoint` forms should be small wrappers around Toucan model code
**Closed response maps:**
Tag response Malli `:map` schemas with `{:closed true}` so the generated OpenAPI / TypeScript types reject extra keys. Open maps emit `additionalProperties: true`, which produces weak frontend types and makes discriminated unions ineffective. Tighten domain types in the same pass (`pos-int?` rather than `:int` when zero branches elsewhere).
```clojure
;; weak
{:status 200 :body [:map [:status [:= :generated]] [:prompt_count :int]]}
;; tight
{:status 200 :body [:map {:closed true}
[:status [:= :generated]]
[:prompt_count pos-int?]]}
```
## MBQL (Metabase Query Language)
**Restrictions:**
@@ -304,215 +144,10 @@ Tag response Malli `:map` schemas with `{:closed true}` so the generated OpenAPI
**Linter Suppressions:**
The right shape is a **reader-discard map** placed directly above the form being silenced, listing the rule keywords explicitly:
```clojure
#_{:clj-kondo/ignore [:metabase/validate-defendpoint-route-uses-kebab-case
:metabase/validate-defendpoint-has-response-schema]}
(api.macros/defendpoint :post "/api_key" ...)
```
For a single require entry, hang it as metadata on that entry:
```clojure
(:require
^{:clj-kondo/ignore [:deprecated-namespace :discouraged-namespace]}
[metabase.query-processor.store :as qp.store])
```
Common Metabase-specific rules you'll see: `:metabase/modules`, `:metabase/validate-defendpoint-route-uses-kebab-case`, `:metabase/validate-defendpoint-has-response-schema`, `:metabase/prefer-with-dynamic-fn-redefs`, `:deprecated-namespace`, `:discouraged-namespace`. Don't use the keyword form (`#_:clj-kondo/ignore`); always the map form so it's grep-able by which rule(s) are suppressed.
- Use proper format for kondo suppressions
- No `#_:clj-kondo/ignore` (keyword form)
**Configurable Options:**
- Don't define configurable options that can only be set with environment variables
- Use `:internal` `defsetting` instead
## Metabase Helpers — Quick Reference
Metabase and its libraries have **many** helper functions to make code more readable. Don't hand-roll things that already exist! Reach for the right helper *the first time*.
Organized by **problem you're solving**, not by namespace.
## "I wrote this — use that instead" cheat sheet
*Authors: prefer the right column. Reviewers: flag the left column in PRs.*
| You wrote | Reach for | When NOT to |
| -------------------------------------------------------------------- | ------------------------------------ | ----------------------------------------------------------------- |
| `(vec (keep f xs))` | `(u/keepv f xs)` | — |
| `(let [x (...)] (side-effect!) x)` | `(u/prog1 (...) (side-effect!))` | — (`<>` anaphor binds the result) |
| `(into {} (map (juxt k identity)) xs)` | `(u/index-by k xs)` | — |
| `(into {} (map (juxt k vf)) xs)` | `(u/index-by k vf xs)` | — |
| `#(compare %2 %1)` | `u/reverse-compare` | — |
| `(:col (t2/select-one [Model :col] …))` | `(t2/select-one-fn :col Model …)` | — |
| `(or (:k m) default)` | `(:k m default)` | **The value can be present-but-nil** (hydrated DB null, etc.) |
| `(let [x (...)] (if x then else))` | `(if-let [x (...)] then else)` | — |
| `(filter #(= :a (:type %)) coll)` + same for `:b` … | `(group-by :type coll)` then dispatch | — |
| `(->> xs (map f) (remove g) (mapv h))` | `(into [] (comp (map f) (remove g) (map h)) xs)` | when readability of the threading wins |
| nested `let` / `if` / `if-let` | `b/cond` with `:let` | a 2-level nest may be clearer as-is |
| `(let [{:keys [k]} arg] …)` as first form of body | destructure in the arg list | when the destructure also references something computed in `let` |
| `(or (:k m) (get m (name k)))` per field | normalize once: `(update-keys m keyword)` then plain `:k` | when you genuinely need both shapes per-field |
## `metabase.util` (commonly aliased `[metabase.util :as u]`)
Sequences / collections:
- `(u/keepv f coll)` — like `keep` but returns a vector (eager).
- `(u/index-by kf coll)` / `(u/index-by kf vf coll)` — `{(kf x) (vf x)}` map.
Strings (locale-safe):
- `(u/lower-case-en s)`, `(u/upper-case-en s)` — DO use these instead of `clojure.string/lower-case`/`upper-case` whenever the result feeds into matching/comparison; the locale-default versions break on Turkish `i`.
- `(u/qualified-name x)` — keyword → `"ns/name"` or `"name"`; symbol → `(name x)`; string → identity.
Control flow:
- `(u/prog1 first-form & body)` — anaphoric. Result of `first-form` is bound to `<>`. Returns first-form's value. Use for "do a thing, side-effect using its result, return the thing."
- `(u/the-id x)` — get an entity id from an int or an instance.
Comparators:
- `(u/reverse-compare x y)` — `(compare y x)`. Pass as a sort comparator: `(sort-by f u/reverse-compare coll)`.
Other:
- `(u/conflicting-keys m1 m2)` — keys present in both with different values.
The full `metabase.util.cljc` is ~1300 lines — when in doubt, grep it.
## `metabase.util.namespaces` (aliased `shared.ns`)
- `(shared.ns/import-fns [other-ns f1 f2] [another-ns f3])` — re-export defns from another ns through the current ns. The `:require [other-ns :as other-ns]` alias **must stay** — clj-kondo may flag it as unused but it's needed at macro expansion. Add a `;; needed for import-fns` comment so it doesn't get auto-stripped.
## Toucan 2 (aliased `[toucan2.core :as t2]`)
Single-field reads (avoids fetching whole rows):
- `(t2/select-one-fn :col Model :id 123)` — projection-friendly: when `f` is a keyword, only that column is fetched. Use instead of `(:col (t2/select-one [Model :col] …))`.
- `(t2/select-fn-set f Model & conditions)` — set of `(f row)` values.
- `(t2/select-fn-vec f Model & conditions)` — vector of `(f row)` values.
- `(t2/select-pks-set Model & conditions)` — set of PKs.
Existence / counts:
- `(t2/exists? Model & conditions)` → boolean.
- `(t2/count Model & conditions)` → long.
Writes:
- `(t2/insert-returning-instances! Model row-or-rows)` — returns the rows with PKs set.
- `(t2/insert-returning-pks! Model row-or-rows)` — just the PKs.
- `(t2/update! Model id-or-conditions changes)`.
- `(t2/delete! Model id-or-conditions)`.
Transactions:
- `(t2/with-transaction [t-conn] body…)` — opens a txn on the app DB conn.
Hydration:
- `(t2/hydrate instances :key1 :key2)` — invokes `:batched-hydrate` methods.
- Define via `(methodical/defmethod t2/batched-hydrate [:model/Foo :the-key] [_ k rows] …)`.
Define a model:
- `(methodical/defmethod t2/table-name :model/Foo [_] :foo_table)`
- `(t2/deftransforms :model/Foo {:col mi/transform-json})` — common: `transform-json`, `transform-secret-value` (encrypts), `transform-keyword`.
## Better-cond (`[better-cond.core :as b]`)
`(b/cond …)` like `cond` but supports interleaved `:let [bindings]` and `:when` clauses. Lets you flatten cascading `let`/`if`/`if-let`/`let` into a linear spine.
```clojure
(b/cond
:let [thread (load-thread id)]
(nil? thread) :not-found
:let [exploration (load-exploration (:exploration-id thread))
reason (gate-check exploration)]
reason
(skip! reason)
:let [pool (build-pool thread)]
(empty? pool) :skip-no-data
:else
(run! thread pool))
```
Use it when a function would otherwise be a staircase of nested forms. Don't reach for it for a single `if`/`let`.
## Schemas — `mr/def` for shared, inline for single-use
When a schema is used in 2+ places or has a name worth documenting, extract it into the registry via `mr/def`:
```clojure
(:require [metabase.util.malli.registry :as mr])
(mr/def ::options
"Options for Pulse channel rendering."
[:map
[:channel.render/include-buttons? {:description "default: false" :optional true} :boolean]])
```
`mr/def` keys are always `::namespaced` keywords; downstream callers reference via the fully-qualified keyword. Single-use schemas stay inline in the `mu/defn`'s `:-` annotation — no need to extract.
## Malli — `metabase.util.malli`
- `(mu/defn my-fn :- :string [x :- :int, y :- [:maybe :string]] …)` — inline schemas; runtime validation in dev, no-op in prod by default.
- `(mu/disable-enforcement & body)` — wrap test/perf-sensitive blocks.
- Common malli schemas: `:int`, `:string`, `:boolean`, `:keyword`, `:map`, `[:maybe T]`, `[:sequential T]`, `[:enum :a :b]`, `[:map [:k T]]`, `pos-int?`, `nat-int?`.
## i18n — `[metabase.util.i18n :refer [tru deferred-tru]]`
- `(tru "Hello {0}" name)` — translate at call time. Use in request-handling code.
- `(deferred-tru "…")` — translate at read time. Use in static contexts: `defsetting :description`, `defn` metadata `:description`, top-level `def`s of user-visible labels.
- ICU MessageFormat: doubled apostrophe for a literal: `(tru "it''s ready")`. Numbered placeholders: `{0}`, `{1}`.
- **Don't translate** developer-only debug strings (REPL helper text, internal logs).
## Logging — `[metabase.util.log :as log]`
- Levels: `trace`, `debug`, `info`, `warn`, `error`.
- Two flavors per level:
- `(log/warn "msg")` or `(log/warn e "msg")` — exception first if present.
- `(log/warnf "fmt %s" arg)` or `(log/warnf e "fmt %s" arg)` — printf-style.
- Don't `(log/warn (format "fmt %s" arg))` — use `warnf`.
## Map / key normalization
When parsing loosely-typed external data (LLM JSON, request bodies) that may have **either** string or keyword keys:
- `(update-keys m keyword)` — top-level keys → keywords (Clojure 1.11+, in `clojure.core`).
- `(clojure.walk/keywordize-keys m)` — recursive (use sparingly; can over-keywordize value strings if they happen to be in nested maps).
- After normalizing once, downstream code uses plain `(:k m)` — don't write `(or (:k m) (get m (name k)))` at every use site.
Other useful map ops:
- `(update-vals m f)` — Clojure 1.11+, in `clojure.core`.
- `(medley.core/map-vals f m)`, `(m/map-keys f m)` — older alternatives. Medley pinned at **1.4** in this repo — `m/index-of` and some other newer fns are NOT available.
## Comparators & sorting
- `(sort-by f u/reverse-compare coll)` — descending.
- Vector comparator: `(sort-by (juxt :primary :secondary) coll)`.
- Use `(- x)` on a primary numeric key to invert: `(sort-by (juxt #(- (:score %)) :id) coll)` (handles ties on ascending `:id`).
## Anti-patterns to avoid
- **`(let [x (do-thing!)] (other-thing! x) x)`** — use `u/prog1`.
- **`(vec (keep …))`, `(vec (map …))`** — use `u/keepv` or a transducer `(into [] (comp …) coll)`.
- **Multiple `filter` passes over the same coll for different `:type`s** — one `(group-by :type coll)`.
- **Per-field string/keyword dual-lookup** — normalize once with `update-keys keyword`.
- **Reaching into another module's tables via raw `:left-join` SQL** — expose a function on the owning model namespace and call that.
- **Snake-case keys on a massaged in-memory map** — appdb-row only. Convert at the parse boundary (e.g., when a json response becomes an internal record).
- **`(double (:k m 0.0))` on a hydrated DB column** — present-but-nil NPEs. Use `(double (or (:k m) 0.0))`.
- **`defmulti` with every impl on one page** — extension story is a fiction. Use a private lookup map + a private fn.
- **`(pos? (count xs))` to test a lazy `xs` for non-emptiness** — `count` forces the whole seq into memory. Use `(seq xs)`. (Rebuilding the seq — `(map f xs)` twice — is the only thing that recomputes; a bound seq caches.)
- **Plain `with-redefs` on a `defn` var in a test** — not parallel-safe; the kondo hook will warn. Use `mt/with-dynamic-fn-redefs`.
- **`(Thread/sleep N)` in an async test** — flake source #1. `(deref p timeout-ms ::timeout)` and assert `not= ::timeout`.
- **Hand-rolled `(str/split host #":")` for origin/host comparison** — mis-handles IPv6. Use `mw.security/parse-url` (or `try-parse-url` for client-controlled input).
- **`(subs (pr-str huge) 0 N)` on a log path** — the multi-MB string is already allocated. Bind `*print-length*`/`*print-level*` around `pr-str` instead.
- **New retry loop / new "find this column" helper / new transform-source-type predicate** — search first. `metabase.util.retry/with-retry`, `metabase.lib.equality/find-matching-column`, and `metabase.transforms.feature-gating/any-transforms-enabled?` already exist (along with many others). Reviewers ask "isn't there already an X for this?" more than any single idiom.
- **Plain `defn` when the function has any real contract** — `mu/defn` (or `mu/defn-`) with `:- schema` on the return and each param is the house default; reserve plain `defn` for trivial dev/helper code.
- **`(:import (java.time …))` + `(LocalDate/now)`** in new code — use `[java-time.api :as t]` + `[metabase.util.date-2 :as u.date]`. Direct `java.time` interop only when a Java API genuinely needs the class.
- **Vanilla `defmethod`** — use `methodical/defmethod` (it takes a docstring; the codebase uses it uniformly).
- **`(when (premium/has-feature? :flag) …)` in code** — gate via `defenterprise` so OSS/EE bodies live in their respective namespaces; never sprinkle runtime feature checks.
- **No `(set! *warn-on-reflection* true)` after the `ns` form** in a new `.clj` file — every `.clj` source file in this repo has it (not `.cljc`).
- **`(ns … (:require [a :as a] [b :as b]))` on one line; `(:import java.io.File java.net.URI)` bare** — both wrong shapes. Requires one per line under `:require`, imports as `(Package Class)` lists.
- **Inventing a fresh `:as` alias for `metabase.util`, `metabase.util.malli`, `metabase.util.log`, `java-time.api`, etc.** — use the canonical alias (see the table in Style Conventions). Mixing aliases makes `grep`-based discovery worse.
## When in doubt
- Run `grep -n "defn " src/metabase/util.cljc | head -80` to scan helpers.
- Run `grep -rn "defmacro\|defn-? [a-z-]*\?\b" src/metabase/util.cljc | head` for predicates.
- Check `medley.core/` if it looks "collection-y" — but **Medley is pinned at 1.4** here; new fns aren't available.
- Search the codebase for an existing use of the pattern you're about to write — if no one else does it, there's usually a helper.

View File

@@ -6,4 +6,3 @@
- Run targeted tests, and lint continuously during development
- Prioritize understanding existing patterns before implementing
- Don't commit changes, leave it for the user to review and make commits
- Minimize comments. Document only non-obvious intent, trade-offs, or constraints that the code itself cannot convey. Do not narrate what the code does — keep code self-descriptive through naming and structure.

View File

@@ -1,69 +0,0 @@
# React, Redux, and Data-Layer Patterns
## API access and RTK Query
- **Use RTK Query for server data.** Add typed endpoints under `metabase/api` or `metabase-enterprise/api`; don't add ad-hoc `fetch`, service wrappers, or new thunks for ordinary API access.
- **Keep API files transport-focused.** Endpoint files should cover request/response typing, URLs, params, light response normalization, and cache tags — not feature business logic.
- **Use canonical API types.** Prefer `metabase-types/api`; fix or derive missing types instead of casting endpoint results. Refer to API endpoint implementation in Clojure when building new endpoint connections.
- **Tag caches deliberately.** Use `listTag`, `idTag`, `provide*Tags`, and `invalidateTags(error, tags)` where appropriate.
- **Guard missing query inputs.** Use `skipToken` or `skip` until all required inputs exist. Build query requirements where the query is created, not in a downstream render path.
- **Use lazy queries only for events.** Data needed to render a screen belongs in a regular query hook at the owner component/hook.
- **Handle mutations with `unwrap()`.** Prefer `await trigger(args).unwrap()` in submit handlers; otherwise branch on `result.error` immediately.
## Where data should live
- **Server data:** RTK Query cache, not duplicated Redux state.
- **Form state:** Form library or local component state, not global Redux.
- **Transient UI state:** Component state or narrow context, not store-wide flags.
- **Cross-route/global tasks:** Redux slice with public selectors.
- **Derived values:** Selectors or pure helpers, not stored copies of computed data. Memoized if identity is important.
## Redux state modeling
- **Avoid using Redux**. Most problems can be solved with local component state or RTK Query.
- **Use Redux only when needed.** Reach for Redux only when distant parts of the app need durable shared state.
- **Model async status explicitly.** Use a discriminated union or explicit `status`/`isLoading`/`error`; don't infer loaded or failed state from truthy/falsy `data`.
- **Store canonical state only.** Derive filtered lists, labels, permissions, booleans, and display variants in selectors.
- **Read through selectors.** Name selectors after what they return (`getCurrentTask`, `getHasPendingMutation`), not how they compute it.
- **Add missing public actions/selectors.** Don't reach into another feature's private state shape because an action or selector is missing.
- **Use `createSelector` for derived state.** Keep selectors pure, cheap, and stable by reference when consumers depend on identity.
## React hooks and components
- **`useMemo` computes render values.** Side effects belong in `useEffect`; event-driven changes belong in event handlers or mutation callbacks.
- **Keep effects narrow and idempotent.** Include all dependencies, guard inside the effect, and clean up timers/subscriptions. Complex logic should be split to multiple effects or custom hooks, also consider extracting functional pieces.
- **Hooks can be used to encapsulate complex logic.** For example, a hook can normalize data, handle form's state and validation, or a hook can manage a table's pagination and sorting. If component's logic becomes too complex, consider extracting a hook.
- **Hooks orchestrate behavior.** Custom hooks should return a small typed result object: data, loading/error state, and callbacks when relevant.
- **Don't mirror props/query data into state** unless the consumer can edit that local copy. Keep derived render data as such.
- **Use `useCallback` for stable identity needs, but deliberately.** Wrap callbacks passed to memoized children, dependency arrays, context values, or hooks that require stable identity — not every local handler.
- **Memoize object/array props only when identity matters.** Prefer simpler plain values otherwise.
- **Stories for presentational components**. Create Storybook stories for presentational components to test different states and props. Stories live next to the component.
## Performance
- **Optimize only where identity or scale matters.** Prefer clear code first, then add `memo`, `useMemo`, `useCallback`, selectors, virtualization, or split components when render cost, prop identity, or list size justifies it.
- **Keep context values stable and avoid frequent updates.** Values updated often should be avoided in context consumed by many components, or stored by reference with a stable context value and explicit subscriptions/selectors.
- **Read global state high in the tree.** Let parent components select context/Redux/API state, derive the view model, and pass narrow props top-down. But avoid passing props too deep (reconsider if more thatn 2 levels).
- **Avoid unstable props in hot paths.** Don't pass fresh object/array/function literals to memoized children; memoize them or move them outside render when identity matters.
- **Render large lists deliberately.** Use pagination, windowing/virtualization, stable keys, and row-level memoization for large or frequently updating lists.
- **Consider memoization for leaf components.** If the source of data updates often, but the component doesn't need to re-render, consider memoizing it.
## Loading, error, and empty states
- **Every async operation needs loading UI.** Use `LoadingAndErrorWrapper` or `DelayedLoadingAndErrorWrapper` for full-page/panel loading; use disabled controls and inline progress for local actions.
- **Every async operation needs an error branch.** Use wrappers or feature-specific alerts for query errors; use `getErrorMessage(error, fallback)` plus a toast/form error for mutation failures.
- **Empty is its own state.** Render no-results, empty-list, and permission-limited states intentionally; don't let them look like unfinished loading.
- **Combine multiple query states explicitly.** Example: `isLoading = a.isLoading || b.isLoading`; `error = a.error || b.error`.
- **Distinguish first load from refresh.** Use `isLoading` for first-load blockers and `isFetching` for background refresh indicators.
- **Test all async branches.** Cover loading, error, empty, and success states when adding or changing data access.
## Review checklist
- Server data uses a typed RTK Query endpoint with correct cache tags.
- Query inputs are guarded with `skipToken`/`skip` until ready.
- Global state is justified, minimal, and read through selectors.
- Context values are stable; frequently changing values don't force broad re-renders.
- Leaf components stay dumb and receive narrow props from parent containers.
- Loading, error, empty, and success branches are visible and tested.
- Hooks are pure during render; side effects are confined to effects or event handlers.
- Mutation failures are surfaced to people using the UI.

View File

@@ -2,8 +2,8 @@
- **Lint:** `bun run lint-eslint-pure`
- Run ESLint on the codebase
- **Format:** `bun run format`
- Format code with oxfmt (`oxfmt --write`); use `bun run lint-format-pure` to check without writing
- **Format:** `bun run prettier`
- Format code using Prettier
- **Type Check:** `bun run type-check-pure`
- Run TypeScript type checking

View File

@@ -10,23 +10,23 @@ This skill helps you add product analytics (Snowplow) events to track user inter
## Quick Reference
Analytics events in Metabase use Snowplow with typed event schemas. Simple events are declared **where they are used** — `trackSimpleEvent` is generic and validates the payload at the call site.
Analytics events in Metabase use Snowplow with typed event schemas. All events must be defined in TypeScript types before use.
**Key Files:**
- `frontend/src/metabase/analytics/event.ts` - Core tracking functions, `trackSimpleEvent` / `trackSchemaEvent` (import from `metabase/analytics`)
- `frontend/src/metabase-types/analytics/event.ts` - The shared `SimpleEventSchema` only. **Do not add event types here** (see below)
- `frontend/src/metabase-types/analytics/schema.ts` - Schema registry (custom/legacy schemas only)
- Feature-specific `analytics.ts` files - Where your tracking functions and any local types live
- `frontend/src/metabase-types/analytics/event.ts` - Event type definitions
- `frontend/src/metabase-types/analytics/schema.ts` - Schema registry
- `frontend/src/metabase/analytics/` - Core tracking functions (import from `metabase/analytics`)
- Feature-specific `analytics.ts` files - Tracking function wrappers
## Quick Checklist
When adding a new analytics event:
- [ ] Pick an event name (snake_case, past tense)
- [ ] Add a tracking function to the feature's `analytics.ts` file, calling `trackSimpleEvent()`
- [ ] Keep any field unions (e.g. `"success" | "failure"`) as local types in that same file
- [ ] Import and call the tracking function at the interaction point
- [ ] Do **not** add an event type to `metabase-types/analytics/event.ts` or to any union
- [ ] Define event type in `frontend/src/metabase-types/analytics/event.ts`
- [ ] Add event to appropriate union type (e.g., `DataStudioEvent`, `SimpleEvent`)
- [ ] Create tracking function in feature's `analytics.ts` file
- [ ] Import and call tracking function at the interaction point
- [ ] Use `trackSimpleEvent()` for basic events (most common)
## Event Schema Types
@@ -47,25 +47,6 @@ type SimpleEventSchema = {
**When to use:** 90% of events fit this schema. Use for clicks, opens, closes, creates, deletes, etc.
`trackSimpleEvent` is generic and enforces this schema on the object literal you pass it:
```typescript
// frontend/src/metabase/analytics/event.ts
export function trackSimpleEvent<
T extends SimpleEventSchema &
Record<Exclude<keyof T, keyof SimpleEventSchema>, never>,
>(event: T) {
trackSchemaEvent("simple_event", event);
}
```
That means a missing `event` or any field outside `SimpleEventSchema` is a compile error at the call
site. There is no separate event type to declare and no `satisfies` clause to add — the old
`ValidateEvent<...>` helper is no longer exported and is not part of the workflow.
`trackSchemaEvent` is generic too: it correlates the schema name with the payload type, so you can't
send a dashboard event under the `simple_event` schema.
### 2. Custom Schemas (legacy, no events are being added)
Consider adding new event schema only in very special cases.
@@ -76,7 +57,36 @@ Consider adding new event schema only in very special cases.
### Example: Track when a user applies filters in a table picker
#### Step 1: Create Tracking Functions
#### Step 1: Define Event Types
Add event type definitions to `frontend/src/metabase-types/analytics/event.ts`:
```typescript
export type DataStudioTablePickerFiltersAppliedEvent = ValidateEvent<{
event: "data_studio_table_picker_filters_applied";
}>;
export type DataStudioTablePickerFiltersClearedEvent = ValidateEvent<{
event: "data_studio_table_picker_filters_cleared";
}>;
```
#### Step 2: Add to Union Type
Find or create the appropriate union type and add your events:
```typescript
export type DataStudioEvent =
| DataStudioLibraryCreatedEvent
| DataStudioTablePublishedEvent
| DataStudioGlossaryCreatedEvent
| DataStudioGlossaryEditedEvent
| DataStudioGlossaryDeletedEvent
| DataStudioTablePickerFiltersAppliedEvent // <- Add here
| DataStudioTablePickerFiltersClearedEvent; // <- Add here
```
#### Step 3: Create Tracking Functions
In your feature's `analytics.ts` file (e.g., `enterprise/frontend/src/metabase-enterprise/data-studio/analytics.ts`):
@@ -96,7 +106,7 @@ export const trackDataStudioTablePickerFiltersCleared = () => {
};
```
#### Step 2: Use in Components
#### Step 4: Use in Components
Import and call the tracking function at the interaction point:
@@ -128,11 +138,16 @@ function FilterPopover({ filters, onSubmit }) {
## Using SimpleEventSchema Fields
All examples below live in the feature's own `analytics.ts` — nothing is registered centrally.
### Example: Event with target_id
```typescript
// Type definition
export type DataStudioLibraryCreatedEvent = ValidateEvent<{
event: "data_studio_library_created";
target_id: number | null;
}>;
// Tracking function
export const trackDataStudioLibraryCreated = (id: CollectionId) => {
trackSimpleEvent({
event: "data_studio_library_created",
@@ -147,10 +162,14 @@ trackDataStudioLibraryCreated(newLibrary.id);
### Example: Event with triggered_from
```typescript
// Local union, exported only if another feature needs to pass the same value
export type NewButtonLocation = "app-bar" | "empty-collection";
// Type definition
export type NewButtonClickedEvent = ValidateEvent<{
event: "new_button_clicked";
triggered_from: "app-bar" | "empty-collection";
}>;
export const trackNewButtonClicked = (location: NewButtonLocation) => {
// Tracking function
export const trackNewButtonClicked = (location: "app-bar" | "empty-collection") => {
trackSimpleEvent({
event: "new_button_clicked",
triggered_from: location,
@@ -168,31 +187,44 @@ export const trackNewButtonClicked = (location: NewButtonLocation) => {
### Example: Event with event_detail
Real example — `frontend/src/metabase/metadata/pages/shared/analytics.ts`:
```typescript
export type MetadataEditEventDetail =
| "type_casting"
| "semantic_type_change"
| "visibility_change";
// Type definition
export type MetadataEditEvent = ValidateEvent<{
event: "metadata_edited";
event_detail: "type_casting" | "semantic_type_change" | "visibility_change";
triggered_from: "admin" | "data_studio";
}>;
export const trackMetadataChange = (detail: MetadataEditEventDetail) => {
// Tracking function
export const trackMetadataChange = (
detail: "type_casting" | "semantic_type_change" | "visibility_change",
location: "admin" | "data_studio"
) => {
trackSimpleEvent({
event: "metadata_edited",
event_detail: detail,
triggered_from: "admin",
triggered_from: location,
});
};
// Usage
trackMetadataChange("semantic_type_change");
trackMetadataChange("semantic_type_change", "data_studio");
```
### Example: Event with result and duration
See `frontend/src/metabase/archive/analytics.ts` for the real version of this.
```typescript
// Type definition
export type MoveToTrashEvent = ValidateEvent<{
event: "moved-to-trash";
target_id: number | null;
triggered_from: "collection" | "detail_page" | "cleanup_modal";
duration_ms: number | null;
result: "success" | "failure";
event_detail: "question" | "model" | "metric" | "dashboard";
}>;
// Tracking function
export const trackMoveToTrash = (params: {
targetId: number | null;
triggeredFrom: "collection" | "detail_page" | "cleanup_modal";
@@ -248,16 +280,17 @@ try {
"filters-applied" // Use underscore, not hyphen
```
### Local Field Types (PascalCase, named after the field)
There is usually no `...Event` type to name anymore. When you do need a union for a field, name it
after the field it feeds:
### Event Type Names (PascalCase with "Event" suffix)
```typescript
// Good
type MetricDimensionResult = "success" | "failure"; // -> result
export type MetadataEditEventDetail = "type_casting"; // -> event_detail
type NewButtonLocation = "app-bar" | "empty-collection"; // -> triggered_from
DataStudioLibraryCreatedEvent
TablePickerFiltersAppliedEvent
MetabotChatOpenedEvent
// Bad
dataStudioLibraryCreated // Wrong case
DataStudioLibraryCreated // Missing "Event" suffix
```
### Tracking Function Names (camelCase with "track" prefix)
@@ -276,28 +309,29 @@ logLibraryCreated // Use "track" prefix
## Common Patterns
### Pattern 1: Sharing Field Types Across Features
### Pattern 1: Feature-Specific Union Types
When two features send the same event with a different `triggered_from`, export the field union from
the owning feature's `analytics.ts` and import it — don't hoist anything into `metabase-types`:
Group related events together:
```typescript
// frontend/src/metabase/data-studio/data-model/analytics.ts
import { trackSimpleEvent } from "metabase/analytics";
import type { MetadataEditEventDetail } from "metabase/metadata/pages/shared/analytics";
export type DataStudioEvent =
| DataStudioLibraryCreatedEvent
| DataStudioTablePublishedEvent
| DataStudioGlossaryCreatedEvent;
export function trackMetadataChange(detail: MetadataEditEventDetail) {
trackSimpleEvent({
event: "metadata_edited",
event_detail: detail,
triggered_from: "data_studio",
});
}
export type MetabotEvent =
| MetabotChatOpenedEvent
| MetabotRequestSentEvent
| MetabotFixQueryClickedEvent;
// Then add to SimpleEvent union
export type SimpleEvent =
| /* other events */
| DataStudioEvent
| MetabotEvent
| /* more events */;
```
This is the point of the extensible-events design: enterprise and feature-tier types stay in their
own module instead of being imported down into a shared union.
### Pattern 2: Conditional Tracking
Track different events based on user action:
@@ -316,10 +350,10 @@ const handleSave = async () => {
## Common Pitfalls
### Don't: Add custom fields to a simple event
### Don't: Add custom fields to SimpleEvent
```typescript
// WRONG - SimpleEventSchema doesn't support custom fields (this is a compile error)
// WRONG - SimpleEvent doesn't support custom fields
export const trackFiltersApplied = (filters: FilterState) => {
trackSimpleEvent({
event: "filters_applied",
@@ -345,43 +379,23 @@ export const trackFilterApplied = (filterType: string) => {
};
```
### Don't: Add event types to `metabase-types/analytics/event.ts`
The central `SimpleEvent` union was removed — it forced feature-tier types to be imported down into
shared code, causing module-boundary violations. `trackSimpleEvent` is generic now, so the type adds
nothing but duplication.
### Don't: Forget to add event to union type
```typescript
// ❌ WRONG - central declaration + re-import for a `satisfies` clause
// frontend/src/metabase-types/analytics/event.ts
// Define the event
export type NewFeatureClickedEvent = ValidateEvent<{
event: "new_feature_clicked";
target_id: number;
}>;
// frontend/src/metabase/my-feature/analytics.ts
import type { NewFeatureClickedEvent } from "metabase-types/analytics";
// ❌ WRONG - Forgot to add to SimpleEvent union
// Event won't be recognized by TypeScript
export const trackNewFeatureClicked = (id: number) => {
trackSimpleEvent({
event: "new_feature_clicked",
target_id: id,
} satisfies NewFeatureClickedEvent);
};
// ✓ RIGHT - the object literal is already checked by the generic
// frontend/src/metabase/my-feature/analytics.ts
export const trackNewFeatureClicked = (id: number) => {
trackSimpleEvent({
event: "new_feature_clicked",
target_id: id,
});
};
// ✓ RIGHT - Add to appropriate union
export type SimpleEvent =
| /* other events */
| NewFeatureClickedEvent;
```
A few `...Event` types still sit in `metabase-types/analytics/event.ts`. They are leftovers from PRs
that landed around the refactor — don't copy them, and don't add to them.
### Don't: Mix up event name formats
```typescript
@@ -449,44 +463,41 @@ Example console output:
### Where to put tracking functions:
```
Tracking functions AND their local field types (this is where new events live):
Feature-specific analytics functions:
frontend/src/metabase/{feature}/analytics.ts
enterprise/frontend/src/metabase-enterprise/{feature}/analytics.ts
Event type definitions (all in one place):
frontend/src/metabase-types/analytics/event.ts
Core tracking utilities:
frontend/src/metabase/analytics/ (import from `metabase/analytics`)
Shared SimpleEventSchema only — nothing new goes here:
frontend/src/metabase-types/analytics/event.ts
```
In embedding SDK code, use `trackSdkSimpleEvent`
(`frontend/src/embedding-sdk-bundle/analytics/snowplow.ts`) instead — the main-app `"sp"` tracker
isn't initialized in the customer's page, so `trackSimpleEvent`'s Snowplow leg is a no-op there.
## Real-World Examples
See these files for reference:
- **Simple events + local field union**: `frontend/src/metabase/metadata/pages/shared/analytics.ts`
- **Reusing another feature's field type**: `frontend/src/metabase/data-studio/data-model/analytics.ts`
- **Result + duration timing**: `frontend/src/metabase/archive/analytics.ts`
- **Enterprise feature events**: `enterprise/frontend/src/metabase-enterprise/google_drive/analytics.ts`
- **Simple events**: `enterprise/frontend/src/metabase-enterprise/data-studio/analytics.ts`
- **Events with variants**: `frontend/src/metabase/dashboard/analytics.ts`
- **Complex events**: `frontend/src/metabase/query_builder/analytics.js`
- **Event type examples**: `frontend/src/metabase-types/analytics/event.ts`
## Workflow Summary
1. **Identify the user interaction** to track
2. **Decide on event name** (snake_case, descriptive)
3. **Create tracking function** in feature's `analytics.ts`, calling `trackSimpleEvent()`
4. **Add local field unions** in that same file if a field has a fixed set of values
5. **Import and call** at the interaction point
6. **Test** that events fire correctly
3. **Define event type** in `event.ts` using `ValidateEvent`
4. **Add to union type** (create feature union if needed)
5. **Create tracking function** in feature's `analytics.ts`
6. **Import and call** at the interaction point
7. **Test** that events fire correctly
## Tips
- **Be specific** - `filters_applied` is better than `action_performed`
- **Use past tense** - `library_created` not `create_library`
- **Group related events** - Keep a feature's tracking functions together in its `analytics.ts`
- **Group related events** - Create feature-specific event union types
- **Track meaningful actions** - Not every click needs tracking
- **Consider the data** - What would you want to analyze later?
- **Stay consistent** - Follow existing naming patterns in the codebase

View File

@@ -56,11 +56,6 @@ Use this to scan through changes efficiently:
- [ ] References use `[[other-var]]` not backticks
- [ ] `TODO` comments include author and date: `;; TODO (Name 2025-01-01) -- description`
Docstring content — check every new or rewritten docstring against the anti-pattern table in the
style guide above. Read the diff's docstrings on their own, separately from the code: prose is
where review attention slides off, and a long docstring reads as diligence rather than as the
liability it usually is.
### Code Organization
- [ ] Everything `^:private` unless used elsewhere
@@ -140,17 +135,6 @@ Quick scan for common issues:
> This public var needs a docstring explaining its purpose, inputs, and outputs.
**For docstrings documenting code they don't own:**
> This describes how `lib` decides binning strategies internally. Nothing here fails when that
> changes, so it will go stale silently — drop it and let `available-binning-strategies` answer
> for its own behavior.
**For docstring content that belongs in the body:**
> This paragraph explains why the branches are ordered this way, which is what someone editing
> the `cond` needs — but a caller can't act on it. Move it to an inline comment above the `cond`.
**For organization issues:**
> This function is only used in this namespace, so it should be marked `^:private`.

View File

@@ -82,29 +82,10 @@ either a bug, or it means callers must handle each case themselves — and
in *that* case it is the *absence* of the property that must be
documented. Document deviations from expectation, not conformance to it.
Heuristic — a sentence earns its place in the docstring only if it
passes both tests:
1. Rewrite test (necessary, not sufficient): if rewriting the body to a
different implementation with an identical contract could make the
sentence false, it describes the implementation, not the contract.
Relocate it to an inline comment at the point in the body where it
applies.
2. Ownership test: if the sentence states a fact about code this
function does not own — a callee's behavior, a library's guarantee —
it survives any rewrite trivially, which is exactly why the rewrite
test alone is not enough. If this body *relies* on that fact, it is
implementation context: relocate it to the call site. If nothing here
relies on it, it is blather about someone else's code: delete it, and
let the callee's own docstring answer for it.
Say each fact once. Duplicated prose is a stale comment in advance: when
the fact changes, one copy gets updated and the rest quietly rot. This
holds even within code you own — a namespace docstring that
re-summarizes its own multimethods, three sibling functions carrying the
same paragraph about retry behavior. Give the fact one home, and have
the other places name it rather than restate it.
Heuristic: if a sentence would still be true after a full rewrite of the
body, it may belong in the docstring. If it describes *how the current
body works*, it belongs in the body — as an inline comment, if it is
non-obvious.
Multi-line docstrings are not banned — a genuinely non-obvious constraint
the code had to deal with can be worth explaining. But be prudent; the
@@ -121,7 +102,3 @@ breakage keeps happening, *that* is the signal a comment was warranted.
- End all files with a newline
- When editing tabular code, where the columns line up, try to keep them aligned
- Spaces on a line with nothing after it is not allowed
- After changing module boundaries (adding/removing/renaming a `src` namespace, a cross-module `require`
or `:model/X` reference, or a new module), run `./bin/mage fix-modules-config` to regenerate
`.clj-kondo/config/modules/config.edn` and keep `metabase.core.modules-test` green. No-op when nothing
drifted; see "Module Boundaries" in the project `CLAUDE.md`.

View File

@@ -22,11 +22,10 @@ Before starting, determine which mode to use:
3. **(Conditional)** If after reading the spec a specific assertion or selector is **ambiguous** — you can't tell if it's the right anchor, or whether the test actually verifies what its title claims — briefly grep the related component for the test ID / role / text. **Don't read components by default.** Only do this when there's a real signal of confusion in the spec.
4. Scan against the checklist + pattern table below.
5. Number all issues sequentially. Skip nits — only flag what's worth fixing.
6. **Always** finish with the honest e2e-vs-unit breakdown (see the section of that name below). It's part of every report, not an optional add-on.
### When to hand off instead of review
If the spec references an issue (`metabase#NNNNN`) and the user wants to **fix** flakiness or assess whether the test still reproduces the original bug, that's outside the review skill's scope. Point them at the dedicated flake-fixing workflow, which knows to fetch the issue and the resolving PR's diff. The review skill stays focused on "is this test well-written and conformant" — it doesn't fetch external issue context by default.
If the spec references an issue (`metabase#NNNNN`) and the user wants to **fix** flakiness or assess whether the test still reproduces the original bug, that's outside the review skill's scope. Point them at `/fix-flakey-test` (or the dedicated flake-fixing workflow), which knows to fetch the issue and the resolving PR's diff. The review skill stays focused on "is this test well-written and conformant" — it doesn't fetch external issue context by default.
## Review checklist
@@ -195,41 +194,8 @@ Use the pending review workflow:
Each comment body starts with `**Issue N: [Brief title]**`.
## Honest e2e-vs-unit breakdown (always include)
Every report ends with this section. The numbered issues above are about whether the test is *well-written*; this section is about whether it should be an **e2e test at all**. They're orthogonal — a flawless spec can still be paying e2e prices for component-test value, and that's worth saying out loud.
The bar is simple: **an `it()` earns its place in e2e only if it exercises a layer you cannot fake without gutting the test's value.** In this codebase that means one or more of:
- a **real backend query against seeded data** (the assertion depends on the server actually filtering/sorting/bucketing rows, not on a mocked response);
- **real rendering you then interact with** — clicking a charted ECharts SVG element at computed coordinates, drag/hit-testing, drill-through;
- **routing across screens** — navigation, URL param round-trips that the backend then honors, browser back/forward;
- **cross-screen traversal** — list → detail → back, where the value is the seam between screens.
Conversely, a test belongs in **Jest + RTL** (the cheaper layer this repo already uses for component/unit coverage) when its real subject is frontend-only:
- **pure frontend logic** — a bucketing/formatting/derivation decision (e.g. "single-day range → bucket by hour") that has one right answer independent of the backend;
- **render assertions** — "the page mounts N titled cards and an SVG exists." A mocked dataset response + RTL verifies the titles; that an ECharts SVG renders is library behavior, not your logic;
- **state/wiring** — tab → `aria-selected` → active-state, or select → querystring mapping. The *URL gets the param* half is component-level; only the *backend honors the param* half needs e2e.
Write the breakdown in three honest buckets. Ground each call in **which layers the test actually touches**, not in how the test is named:
1. **Genuinely must-be-e2e (keep)** — load-bearing. Name the layer(s) only a browser + backend can verify together.
2. **Defensible-but-narrowable e2e** — justified as integration, but over-split or doing more boots than the coverage requires. Say what to merge.
3. **Should be unit / component tests** — where the real savings are. Name the cheaper layer and what it would assert (often *more precisely* than the e2e test does indirectly).
Then a one-paragraph **bottom line**: what to keep as e2e, what to push down, and — if you pulled CI timing from `e2e/support/timings.json` for the Performance check — roughly how much of that wall time is *earned* by the must-be-e2e set versus paid by the rest.
Rules of engagement so this stays honest, not performative:
- **Don't inflate the keep bucket to look balanced.** If most of the spec is component-test value paying e2e prices, say exactly that. If every test is load-bearing, say that too — a spec where all 12 tests must be e2e is a legitimate outcome, not a failure to find candidates.
- **Be specific.** Cite the `it()` titles and line ranges. "Several tests could be unit tests" is useless; "tests 1/17/18 assert only that N titled cards mount — RTL with a mocked dataset" is actionable.
- **This is advice to the author, not a blocking issue.** It's a recommendation, separate from the numbered issues. Don't renumber it into the issue list.
- **Stay rooted in what exists.** The cheaper layer here is Jest + RTL component/unit tests. Don't invent infrastructure the repo doesn't have, and don't recommend deleting backend-dependent coverage just because part of the test is frontend-only — split it, keep the integration half.
## Final check
1. Trim issues that won't materially help the author.
2. Verify sequential numbering with no gaps.
3. In PR mode, verify each issue was posted as a separate review comment.
4. Verify the honest e2e-vs-unit breakdown is present — it's required in every report, even when the answer is "all of these need to be e2e."

View File

@@ -20,8 +20,8 @@ For EACH styled component, go through every CSS property and ask: "Can this be a
**Properties that ARE style props** (use these, not CSS modules):
- `display` → `display` prop
- `color` → `c` prop (`c="core-brand"`, `c="text-primary"`)
- `background-color` → `bg` prop (`bg="background_page-primary"`)
- `color` → `c` prop (`c="brand"`, `c="text-primary"`)
- `background-color` → `bg` prop (`bg="background-primary"`)
- `font-size` → `fz` prop (`fz="md"`)
- `font-weight` → `fw` prop (`fw="bold"`)
- `line-height` → `lh` prop (`lh="md"`)
@@ -57,7 +57,7 @@ For EACH styled component, go through every CSS property and ask: "Can this be a
align="center" /* style prop */
gap="sm" /* style prop */
p="md" /* style prop */
bg="background_page-primary" /* style prop */
bg="background-primary" /* style prop */
>
```
@@ -149,19 +149,19 @@ Read the `.styled.tsx` file AND every component that imports from it. Understand
For each styled component, apply the Mantine-First Decision Gate above. Then determine the migration target:
| Emotion Pattern | Migration Target |
| ------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `styled.div` with only layout/spacing/color | `Box`, `Flex`, `Stack`, or `Group` with style props. **NO CSS module needed.** |
| `styled.div` with flexbox column | `Stack` component with style props |
| `styled.div` with flexbox row | `Flex` or `Group` component with style props |
| `styled.span` / `styled.p` with color/weight/size | **`Text component="span"` with style props** (`c`, `fw`, `fz`). NO CSS module. |
| `styled.div` with hover/focus/pseudo-selectors | **Hybrid**: Mantine component with style props for expressible properties + CSS module class for pseudo-selectors only |
| `styled.div` with media queries (simple spacing/sizing) | **Responsive style props**: `p={{ base: "md", lg: "xl" }}`. NO CSS module. |
| `styled.div` with media queries (complex/non-spacing) | CSS module for the media query parts, style props for the rest |
| `styled.div` with animations/keyframes | CSS module for animation, style props for layout |
| `styled(SomeComponent)` with only color/spacing/flex | **Wrap in Mantine component** with style props: `<Box c="core-brand" flex="0 0 auto"><Icon /></Box>`, or pass style props if the component accepts them |
| `styled(SomeComponent)` with pseudo-selectors/complex styles | CSS module `className` on the component |
| Dynamic props `styled.div<{ isActive: boolean }>` | Mantine style props for simple toggles (`c={active ? "core-brand" : "text-primary"}`), `cx()` with CSS module classes for complex state combinations involving pseudo-selectors |
| Emotion Pattern | Migration Target |
| ------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `styled.div` with only layout/spacing/color | `Box`, `Flex`, `Stack`, or `Group` with style props. **NO CSS module needed.** |
| `styled.div` with flexbox column | `Stack` component with style props |
| `styled.div` with flexbox row | `Flex` or `Group` component with style props |
| `styled.span` / `styled.p` with color/weight/size | **`Text component="span"` with style props** (`c`, `fw`, `fz`). NO CSS module. |
| `styled.div` with hover/focus/pseudo-selectors | **Hybrid**: Mantine component with style props for expressible properties + CSS module class for pseudo-selectors only |
| `styled.div` with media queries (simple spacing/sizing) | **Responsive style props**: `p={{ base: "md", lg: "xl" }}`. NO CSS module. |
| `styled.div` with media queries (complex/non-spacing) | CSS module for the media query parts, style props for the rest |
| `styled.div` with animations/keyframes | CSS module for animation, style props for layout |
| `styled(SomeComponent)` with only color/spacing/flex | **Wrap in Mantine component** with style props: `<Box c="brand" flex="0 0 auto"><Icon /></Box>`, or pass style props if the component accepts them |
| `styled(SomeComponent)` with pseudo-selectors/complex styles | CSS module `className` on the component |
| Dynamic props `styled.div<{ isActive: boolean }>` | Mantine style props for simple toggles (`c={active ? "brand" : "text-primary"}`), `cx()` with CSS module classes for complex state combinations involving pseudo-selectors |
### Step 3: Create CSS Module (if needed)
@@ -170,19 +170,19 @@ Create `ComponentName.module.css` alongside the component file:
```css
/* Use design system tokens — NEVER raw color/spacing values */
.root {
border: 1px solid var(--mb-color-border-neutral);
border: 1px solid var(--mb-color-border);
border-radius: var(--mantine-radius-md);
background-color: var(--mb-color-background_page-primary);
background-color: var(--mb-color-background-primary);
}
/* Conditional states as separate classes, combined with cx() */
.active {
background-color: var(--mb-color-core-brand);
background-color: var(--mb-color-brand);
color: var(--mb-color-text-primary-inverse);
}
.disabled {
color: var(--mb-color-text-disabled);
color: var(--mb-color-text-tertiary);
pointer-events: none;
}
@@ -191,8 +191,8 @@ Create `ComponentName.module.css` alongside the component file:
cursor: pointer;
&:hover {
color: var(--mb-color-core-brand);
background-color: var(--mb-color-background_surface-hover);
color: var(--mb-color-brand);
background-color: var(--mb-color-background-hover);
}
}
@@ -255,9 +255,9 @@ Remove the old styled file entirely. Remove all imports of it from other files.
**Colors** (`c`, `bg`):
- Text: `"text-primary"`, `"text-secondary"`, `"text-disabled"`
- Background: `"background_page-primary"`, `"background_page-secondary"`, `"background_surface-hover"`
- Brand: `"core-brand"`, `"feedback-negative"`, `"feedback-positive"`, `"feedback-warning"`
- Text: `"text-primary"`, `"text-secondary"`, `"text-tertiary"`
- Background: `"background-primary"`, `"background-secondary"`, `"background-hover"`
- Brand: `"brand"`, `"error"`, `"success"`, `"warning"`
**Typography** (`fz`, `fw`, `lh`, `ta`, `ff`):
@@ -279,11 +279,11 @@ Remove the old styled file entirely. Remove all imports of it from other files.
**Colors** — `var(--mb-color-<name>)`:
- `--mb-color-text-primary`, `--mb-color-text-secondary`, `--mb-color-text-disabled`
- `--mb-color-background_page-primary`, `--mb-color-background_page-secondary`, `--mb-color-background_surface-hover`
- `--mb-color-text-primary`, `--mb-color-text-secondary`, `--mb-color-text-tertiary`
- `--mb-color-background-primary`, `--mb-color-background-secondary`, `--mb-color-background-hover`
- `--mb-color-border`, `--mb-color-border-strong`, `--mb-color-border-subtle`
- `--mb-color-core-brand`, `--mb-color-core-brand-hover`
- `--mb-color-feedback-negative`, `--mb-color-feedback-positive`, `--mb-color-feedback-warning`
- `--mb-color-brand`, `--mb-color-brand-hover`
- `--mb-color-error`, `--mb-color-success`, `--mb-color-warning`
- `--mb-color-shadow`, `--mb-color-focus`
- `--mb-color-accent0` through `--mb-color-accent7`
@@ -398,13 +398,13 @@ export const Wrapper = styled.div`
```tsx
export const Item = styled.div<{ isSelected: boolean; disabled: boolean }>`
color: ${(props) =>
props.disabled ? color("text-disabled") : color("text-primary")};
props.disabled ? color("text-tertiary") : color("text-primary")};
background-color: ${(props) =>
props.isSelected ? color("core-brand") : "transparent"};
props.isSelected ? color("brand") : "transparent"};
cursor: ${(props) => (props.disabled ? "default" : "pointer")};
&:hover {
color: ${(props) => !props.disabled && color("core-brand")};
color: ${(props) => !props.disabled && color("brand")};
}
`;
```
@@ -418,20 +418,20 @@ export const Item = styled.div<{ isSelected: boolean; disabled: boolean }>`
cursor: pointer;
&:hover {
color: var(--mb-color-core-brand);
color: var(--mb-color-brand);
}
}
.selected {
background-color: var(--mb-color-core-brand);
background-color: var(--mb-color-brand);
}
.disabled {
color: var(--mb-color-text-disabled);
color: var(--mb-color-text-tertiary);
cursor: default;
&:hover {
color: var(--mb-color-text-disabled);
color: var(--mb-color-text-tertiary);
}
}
```
@@ -457,7 +457,7 @@ When a styled component wraps another component, first check if the styles can b
export const CardIcon = styled(Icon)`
display: block;
flex: 0 0 auto;
color: var(--mb-color-core-brand);
color: var(--mb-color-brand);
`;
export const CardTitle = styled(Ellipsified)`
@@ -471,7 +471,7 @@ export const CardTitle = styled(Ellipsified)`
**After — style props on Mantine wrapper (preferred when ALL properties are expressible):**
```tsx
<Box display="block" flex="0 0 auto" c="core-brand">
<Box display="block" flex="0 0 auto" c="brand">
<Icon {...icon} />
</Box>
<Box c="text-primary" fz="md" fw="bold" ml="md">
@@ -483,13 +483,13 @@ export const CardTitle = styled(Ellipsified)`
```css
.icon {
color: var(--mb-color-core-brand);
color: var(--mb-color-brand);
width: 1.5rem;
height: 1.5rem;
cursor: pointer;
&:hover {
color: var(--mb-color-core-brand-hover);
color: var(--mb-color-brand-hover);
}
}
```
@@ -570,12 +570,12 @@ export const Spinner = styled.div`
### Pattern 6: color() Function → CSS Variables
| Emotion (JS) | CSS Module | Mantine Style Prop |
| -------------------------- | ----------------------------------------------------------------- | ------------------------------------- |
| `color("core-brand")` | `var(--mb-color-core-brand)` | `c="core-brand"` or `bg="core-brand"` |
| `color("text-primary")` | `var(--mb-color-text-primary)` | `c="text-primary"` |
| `color("border-neutral")` | `var(--mb-color-border-neutral)` | N/A (use CSS module) |
| `alpha("core-brand", 0.2)` | `color-mix(in srgb, var(--mb-color-core-brand), transparent 80%)` | N/A (use CSS module) |
| Emotion (JS) | CSS Module | Mantine Style Prop |
| ----------------------- | ------------------------------------------------------------ | --------------------------- |
| `color("brand")` | `var(--mb-color-brand)` | `c="brand"` or `bg="brand"` |
| `color("text-primary")` | `var(--mb-color-text-primary)` | `c="text-primary"` |
| `color("border")` | `var(--mb-color-border)` | N/A (use CSS module) |
| `alpha("brand", 0.2)` | `color-mix(in srgb, var(--mb-color-brand), transparent 80%)` | N/A (use CSS module) |
### Pattern 7: Responsive Spacing/Sizing → Responsive Style Props (NOT CSS Media Queries)
@@ -627,13 +627,13 @@ export const Bar = styled.div<{ width: number }>`
If existing code already uses `CS` classes and you're not migrating that specific code, leave them alone. But when migrating Emotion → Mantine, replace with the proper alternative:
| Instead of `CS.*` | Use |
| ------------------- | ------------------------------------------------------------ |
| `CS.overflowHidden` | CSS module: `overflow: hidden` |
| `CS.cursorPointer` | CSS module: `cursor: pointer` |
| `CS.flex1` | Style prop: `flex={1}` |
| `CS.flexNoShrink` | Style prop: `flex="0 0 auto"` |
| `CS.textBrandHover` | CSS module: `&:hover { color: var(--mb-color-core-brand); }` |
| Instead of `CS.*` | Use |
| ------------------- | ------------------------------------------------------- |
| `CS.overflowHidden` | CSS module: `overflow: hidden` |
| `CS.cursorPointer` | CSS module: `cursor: pointer` |
| `CS.flex1` | Style prop: `flex={1}` |
| `CS.flexNoShrink` | Style prop: `flex="0 0 auto"` |
| `CS.textBrandHover` | CSS module: `&:hover { color: var(--mb-color-brand); }` |
### Pattern 10: Shared Emotion Styles Across Files → Shared CSS Module + cx()
@@ -759,7 +759,7 @@ import S from "./ComponentName.module.css"; // only if CSS module is needed
- [ ] The `.styled.tsx` file is deleted
- [ ] All imports of the deleted styled file are updated
- [ ] No static inline styles — `style={{ }}` used only for truly dynamic runtime values
- [ ] All colors use design tokens (`c="core-brand"`, `var(--mb-color-core-brand)`), not raw hex/rgb
- [ ] All colors use design tokens (`c="brand"`, `var(--mb-color-brand)`), not raw hex/rgb
- [ ] All spacing uses design tokens (`p="md"`, `var(--mantine-spacing-md)`), not arbitrary px values — hardcoded `rem`/`px` literals snapped to the nearest token (see mapping table above)
- [ ] All border-radius values use radius tokens (`var(--mantine-radius-md)`), not hardcoded `0.5rem`
- [ ] All font sizes use font-size tokens (`fz="md"`, `var(--mantine-font-size-md)`), not hardcoded `1rem`

View File

@@ -1,248 +0,0 @@
---
name: fe-modularization
description: Deciding where frontend code lives — tier model, move mechanics, extension points, side effects, naming, and the traps. Use when moving code between modules, carving new modules, fixing boundary violations, adding a barrel or an endpoint, or reviewing module-shape decisions.
---
# Frontend modularization
The module is the unit of blast radius: dependency direction, test selection,
bundle cones, and review scope all follow module boundaries. Every decision
here serves one goal: **make the file locations tell the truth about who owns
what and who may depend on whom.**
## Hard rules (check these before anything else)
- NEVER add an `export … from` re-export at an old path, and NEVER `export *`
from a barrel. Move the code, codemod every call site, delete the old path,
all in the same PR.
- NEVER import past a module's `index.ts` when that module has
`enforcePublicApi: true`, and never past `metabase/ui`'s index at all. If
the barrel is heavy, the fix is making the module side-effect free (below),
not a deep import.
- NEVER rewrite a consumer to a lower-level idiom to make the linter pass. If
the fix makes the consumer worse, the plan is wrong.
- NEVER give a new module `enforceSharedTiers: false`; new modules ship
enforced with `enforcePublicApi: true` and an explicit `index.ts`.
- NEVER move code without the consumer-tier check (procedure step 2).
- ALWAYS delete a module's `enforceSharedTiers: false` line in the PR that
takes it to zero violations.
- ALWAYS measure `bun run module-boundaries` before and after, and report the
numbers.
- ALWAYS import `dayjs` from `metabase/dayjs`, routing from `metabase/router`,
Mantine from `metabase/ui`, react-redux hooks from `metabase/redux`; the
raw specifiers are lint-forbidden.
## Files that are ground truth
| what | where |
|---|---|
| module elements, tiers, `enforceSharedTiers`, `enforcePublicApi` | `frontend/lint/module-boundaries.mjs` |
| shared sub-tiers and levels, cluster rules | `frontend/lint/shared-tiers.mjs` |
| standalone boundaries lint (all violations, incl. grandfathered) | `bun run module-boundaries` (config `eslint.config.module-boundaries.mjs`) |
| PR lint (only enforced modules fail CI) | `bun run lint-eslint-pure` |
| side-effect-free directories and their exceptions | `frontend/build/shared/rspack/side-effect-free-modules.js` (`SIDE_EFFECT_FREE_PATHS`, `SIDE_EFFECT_PATHS`) |
| the lint rules that enforce that promise | `metabase/no-module-side-effects`, `metabase/no-base-api-access` in `frontend/lint/eslint-plugin-metabase/rules/` |
| public-api enforcement | `metabase/enforce-module-public-api` rule, driven by `getPublicApiModules()` |
## The model
Tiers: `lib < basic < shared < feature < app`. Imports point downward, never
sideways at feature tier (features may not import features; app and EE may
import anything). The shared tier is subdivided in `shared-tiers.mjs` into
shared-utils (U0…), shared-platform (P0…), and shared-domain, each ordered
into levels. A module imports only *strictly lower* levels of its sub-tier
plus the sub-tiers below; same-level peers are deliberately forbidden, which
is what makes cycles structurally impossible. A domain may hold two seats
when consumers demand it: a low core and a high surface (metabase-lib below
questions below query_builder; viz-core below visualizations).
Enforcement is per module: `enforceSharedTiers: false` on an element exempts
it from the level rules (counted by `bun run module-boundaries`, invisible to
PR lint). Modules with `enforcePublicApi: true` may only be imported from
outside via `metabase/<module>`, and import relatively inside.
## Deciding where code lives (apply in order; first decisive test wins)
1. **Who owns the concept?** Code lives with the module that owns its
concept, not the module that renders or calls it. A mode encoding dashboard
click behaviour is dashboard code even if viz executes it. Metabot
conversation state is metabot state even though the store registers it.
2. **Consumer-tier check.** The destination must sit at or below the file's
*lowest* consumer. A single consumer below the proposed home vetoes the
move; report it, don't force it.
3. **Only four legitimate fixes** for a bad edge: move the code to its
terminal home; invert through a designed extension point (a prop, an
injected component, a contract type owned by the socket); fix the tier
placement when the declared level is wrong; delete a thin wrapper (a rename
with no derivation isn't an abstraction — deleting it *is* the migration).
4. **A bridge lives above the lower endpoint.** Code that needs both sides of
a boundary belongs in or above the higher module.
5. **"Genuinely shared" is proven by consumers.** A shared address needs two
or more consumers the tier rules can't serve another way. One consumer
pretending to be shared is a feature module at the wrong address.
## Extension points
The socket's owner defines the contract; every plug lives with its extender.
Visualization owns the `mode` prop and click interfaces, each surface supplies
its mode; the editor owns its extension contract, metabot supplies tiptap
extensions; querying owns `TemplateTagsSidebarProps`, parameters implements
it.
- Prefer plain props/injection at composition sites. `PLUGIN_*` registries are
for edition gating; single-reader slots move out to their reading module;
slots with many readers live in the module that owns the concept (every
slot the whitelabel plugin fills lives in `whitelabel`), tier permitting.
Keep contracts type-light so implementations can load lazily.
- Identical injection at every callsite is acceptable until a fourth consumer
or second slot appears; then the composition deserves its own module above
both parts. Don't pre-build the wrapper.
## Store shape
A module that has redux state owns it:
- `store/` is private: reducer, plain creators/action types, and selectors,
typed against the module's own state (`QueryBuilderStoreState`), never the
global `State`. Thunks live in `actions/` and import `store/`, never the
reverse.
- The barrel is the state API: selectors for reads, creators/thunks for
writes.
- Store roots (`reducers-main`, `reducers-public`, the SDK store) import each
slice from its module barrel and derive their own state type; no slice
registry, no `declare module` merging.
- Cross-module writes: a callback prop when a composition point exists;
otherwise dispatch the owner's exported creator; an event only when the
emitter must not know its listeners. Raw `dispatch({ type: "…" })` strings
are banned. A reducer or listener matching another module's *internal*
action is the coupling to remove.
- Slices that mirror one endpoint get deleted (the RTK cache is the state); a
fold over many sources with one writer (the metadata store) is kept as a
module-private slice behind the barrel.
## Endpoints
Endpoints migrate to their owning module via `Api.injectEndpoints` from the
module's `api/` file (settings, metabot, transforms precedents); `metabase/api`
trends toward the client only. There is one `Api` object per backend because
tag invalidation only works within one instance. Consumers reach endpoints
only through the owner's exports (`useGetCardQuery`, `cardApi.endpoints.…`),
never by name through the base object; `metabase/no-base-api-access` allows
injection only in `metabase/api` and `**/api/**` / `**/api.ts`, and reaching by
name only in `metabase/api` and test support (`invalidateTags` /
`resetApiState` are fine anywhere). Each endpoint move carries its
metadata-store hookup: the owner calls `entitiesFetched` from its own
`onQueryStarted`.
## Import-time side effects
Directories in `SIDE_EFFECT_FREE_PATHS` are marked `sideEffects: false` for
rspack, so importing one export from their barrel no longer drags the whole
barrel into a bundle. The price: nothing in those directories may do work at
import time, or production silently drops it. `metabase/no-module-side-effects`
enforces this over every listed directory (module-scope calls, writes to
imported objects or globals, bare imports, control flow at import; escape
hatches are `/* #__PURE__ */` and the rule's pure-callee list). Files that
must run at import are listed in `SIDE_EFFECT_PATHS` (files or directories).
- **A library the app configures gets a facade module** owning the vendor
import, the registrations, and the types, with the raw specifier
lint-forbidden outside it: `metabase/router`, `metabase/ui`,
`metabase/redux`, `metabase/dayjs`. Facades are the effect, so they never
enter `SIDE_EFFECT_FREE_PATHS`. Never re-register a vendor plugin locally
(`dayjs.extend`, `echarts` `use`).
- **A patch a component needs is an explicit registration module,
bare-imported by the provider that needs it** and listed in
`SIDE_EFFECT_PATHS` (Mantine's `Popover.Dropdown` replacement, imported by
`ThemeProvider`).
- **Effects with no value to import** (global CSS, polyfills, the CSP nonce,
EE plugin bootstrap) live in the entry's side-effects file and nowhere else.
- **Registries are filled from the composition root at boot**
(`initializePlugins`, `registerVisualizations`), never by a module
registering itself at import.
- **RTK injection stays at import**; when a module with an `api/` folder is
declared side-effect free, list `<module>/api/` in `SIDE_EFFECT_PATHS`.
## Naming
The feature keeps the plain product-surface name; a shared carve is named for
what the shared part actually is: the concept when the concept is the shared
thing (`settings`, `current-user`, `whitelabel`, `dayjs`), or a qualified
capability when the plain noun is taken (`search-ui`, `metrics-ui`,
`viz-core`, `data-studio-ui`). Never two modules distinguished only by tier
prefix; never generic `-shared` / `-feature` suffixes.
## Procedure for a move
1. **Enumerate consumers honestly** (hook-name greps miss the last three):
```
grep -rn "metabase/<module>" frontend/src enterprise/frontend/src frontend/test e2e
grep -rn "jest.mock(\"metabase/<module>" frontend enterprise/frontend
grep -rn "addMatcher(.*<endpoint>" frontend/src enterprise/frontend/src
```
plus sibling files importing the moved file relatively. Classify each by
module and tier from `module-boundaries.mjs` / `shared-tiers.mjs`.
2. **Consumer-tier check**: destination at or below the lowest consumer. If
not, stop and report the blocking consumer.
3. **Measure before**: `bun run module-boundaries 2>&1 | tail -1` on the base
(write the number down).
4. **Move**: `git mv`; codemod every importer to `metabase/<module>` (outside)
or a relative path (inside); add named exports to the destination barrel
for every symbol an outside consumer needs (each must have a real
consumer). Delete the old path. If the module didn't exist: add its element
in `module-boundaries.mjs` (ordered before any element it's carved from,
first match wins), its level in `shared-tiers.mjs`, `enforcePublicApi:
true`, an `index.ts`. If the move empties a module's violations, delete
its `enforceSharedTiers: false` line. If the module has an `api/` folder
and is side-effect free, list `<module>/api/` in `SIDE_EFFECT_PATHS`.
5. **Verify** (all of these, report the numbers):
```
grep -rn "<old path>" frontend enterprise/frontend e2e # must be empty
bunx eslint --no-warn-ignored <touched files>
bun run lint-eslint-pure # enforce-module-public-api included
bun run type-check-pure
bun run test-unit-keep-cljs <touched folders' specs>
bun run module-boundaries 2>&1 | tail -1 # after; no violation may name the new files
bun run build-release:static-viz # only if viz/static-viz/ui/dayjs touched; hard 3.5 MiB budget
```
6. **Report**: count before/after (state count-neutral explicitly), the edges
removed by file, flags/rules deleted, barrel exports added, anything you
skipped and why, and any consumer you found that vetoed part of the move.
## Traps (each has bitten at least once)
- **Late binding.** A bare function reference in a `createSelector` input
array, a component identity captured for comparison, a `PLUGIN_*` object
read at module scope: all capture at import time and dodge jest spies, EE
overrides, and lazy loading. Read at call time (`(_state) => fn()`), or
inject.
- **Plans go stale.** Verify every recipe against current code before
executing; things get merged, renamed, or turn out never to have existed.
Skip-and-report beats force.
- **The linter's blind spots**: `export … from` re-exports, bare-specifier
packages (custom-viz), pre-seeded caches that never register tags. A grep
for the old path is the stronger claim than a green lint run.
- **Path-keyed baselines reset on rename**: a generated per-file ledger
silently loses a file's history when it moves.
- **A heavy barrel is a bundle hazard until its module is side-effect free.**
One constant imported from the `metabase/ui` barrel pulled Mantine into the
static-viz bundle and broke its budget; the fix was `SIDE_EFFECT_FREE_PATHS`,
never a deep import.
- **Type augmentation is program-wide.** `declare module "dayjs"` (and
similar) makes plugin types available to any file in the same tsconfig, so
a passing type-check never proves the setup ran; only importing the facade
does.
- **Environment before diagnosis**: after a dependency-touching merge,
`bun install` before declaring a failure real; worktrees share staleness
through the node_modules symlink, and a stale `target/cljs_dev` makes
`type-check-pure` fail on `metabase-lib/metric/core.ts` (rebuild cljs in the
worktree).
## PR shape
One logical change per PR; the body names the specific edges removed (files,
imports, before/after count) and gives each move's ownership reason in plain
language a reviewer outside the effort can follow, without campaign labels.
Stack only for semantic dependency or same-hot-file serialisation, and say
so. Consider codeowner surface when chunking. Open as a draft until measured
and green.

View File

@@ -1,244 +0,0 @@
---
name: metabase-ui-component-from-figma
description: Update or build a Metabase design-system component (frontend/src/metabase/ui — Chip, Badge, Alert, Switch, etc.) to match a Figma spec described in a Linear issue. Use when a ticket asks to restyle an existing `metabase/ui` component or implement a new one from Figma. Covers: checking out the issue, mapping the component (usage/blast radius for an existing one, or scaffolding a new one), building a Storybook showcase matrix (before styling for existing components, after for new ones), styling from exact Figma tokens while iterating with the user, and committing + migrating call sites LAST.
---
# Metabase UI component from Figma
End-to-end workflow for bringing a `metabase/ui` (Mantine) component in line with a Figma design, driven by a Linear ticket.
The golden rule (existing components): **build the Storybook showcase before changing any component styles.** Build it while the old styles are still in place — so the team can see the current component at a glance and get accustomed to it before the migration, then watch it light up as you restyle. You end up with a visual-regression (Loki) candidate either way. A **new** component has nothing to render until it's built, so this rule doesn't apply to create mode — there the order flips: implement and style the component first, then build its showcase to present the result.
This is a frontend task — also load **typescript-write** (and **typescript-review** before handing off).
## Two modes
The ticket is one of two shapes; identify which up front, because Phase 2 differs:
- **Update an existing component** — the component already exists with call sites, either in `metabase/ui` *or* as a legacy component elsewhere (e.g. `frontend/src/metabase/common/components/`, possibly Emotion styled-components) that this ticket also migrates into `metabase/ui`. The risk is *regression* (call sites, behavior), so you map usage and lock behavior with tests before touching it.
- **Create a new component** — nothing equivalent exists anywhere yet. Rarer. The risk is *API design*, so you scaffold the component and its public surface first.
Phases 1, 2, 5 and 6 are shared. The middle two differ in **order**: update mode builds the showcase (Phase 3) *before* styling (Phase 4); create mode does the reverse — implement and style first, then build the showcase as the last step — because there's nothing to render until the component exists.
---
## Phase 1 — Check out the issue
1. Fetch the Linear issue with the Linear MCP `get_issue`.
2. Pull from it:
- **Figma link(s)** — extract the `node-id` (`?node-id=250-13588` → `250-13588`) and `fileKey` (the `/design/<fileKey>/…` segment).
- **`gitBranchName`** — switch to it if not already there.
- **The component's intended API** — variants, sizes, and states the design system supports for this component. This defines the axes of your showcase (Phase 3).
3. Decide the mode (see **Two modes** above): search for an existing component — in `metabase/ui` or as a legacy component elsewhere. Found → **update**; nothing equivalent anywhere → **create**. Confirm with the user when unsure.
4. **Verify a Figma desktop MCP is connected before doing the work** — this skill depends on it for exact tokens in Phase 4 (and styling can't proceed accurately without it). Do a quick read against the issue's node (e.g. `get_metadata`/`get_variable_defs`); if it errors with "nothing selected" or the server isn't connected, prompt the user to enable it (Switch to Dev Mode → MCP section in the right sidebar) and select the relevant layer before continuing.
## Phase 2 — Understand the component
### Mode: update an existing component
1. Locate the component dir (typically containing `Component.config.ts` or `Component.tsx`, `Component.module.css`, `Component.stories.tsx`, `index.ts`). A ticket may also be **migrating a legacy component** — if it still uses Emotion styled-components, reach for the **emotion-migrate** skill.
2. Map every usage across `frontend/src` and `e2e` (JSX `<Component`, sub-components like `Component.Group`, and wrapper components).
3. Write findings to **`local/<TICKET>-<component>-usage-findings.md`**, one entry per usage: code location (`file:line`), short context, and **non-standard usage** (excessive customization, inline styling, invalid/legacy props). Invalid props are a signal that call sites will need migrating in Phase 6. This doc **stays** — the user walks it at the end to verify each real usage still looks right.
4. State the blast radius plainly: how many call sites, where, and whether they're confined.
5. **Lock behavior with tests before changing anything — when warranted.** Only for components that have **custom interactive behavior** (their own event handlers, state, keyboard logic, controlled/uncontrolled wiring). Add unit tests covering that current API/behavior so a regression shows up as a failing test. **Skip tests for thin Mantine wrappers** — don't test built-in Mantine behavior, and a pure visual restyle is covered by the showcase/Loki, not unit tests.
### Mode: create a new component
1. Decide whether it wraps a Mantine primitive (most common — extend it via a `.config.ts` using `<Mantine>.extend({...})` and theme registration) or is bespoke. Prefer wrapping Mantine.
2. Scaffold the component dir + its public surface: `Component.tsx`/`Component.config.ts`, `Component.module.css`, `Component.stories.tsx`, `index.ts`, and wire the export through the group barrel and the `metabase/ui` barrel.
3. There are no call sites to map yet; instead note the intended usage from the ticket so the eventual showcase reflects real props.
4. **Order note:** for a new component, implement and style it (Phase 4) first, *then* build the showcase (Phase 3) to present the finished result — there's nothing to render until it exists. The phases below are written update-first; just run 4 before 3 in create mode.
## Phase 3 — Build the Storybook showcase
Goal: one story that shows **every state at a glance**, themeable, suitable as a Loki visual-regression test. **Update mode: build this before Phase 4, while the old styles are still in place. Create mode: do this after Phase 4**, once the component exists to render.
### Reusable showcase helpers
Use the shared, Storybook-only primitives in **`frontend/src/metabase/ui/stories/showcase/`**: `StoryShowcase` (titled panel), `StorySection`, `StoryRow`, and `StoryJsx` (monospace JSX with light syntax highlighting via `tokenizeJsx`). Extend the set only when a new pattern is genuinely reusable.
### Derive the matrix from the component's full state space
Enumerate **all** the states the component can be in — every supported variant, size, and interaction/selection state (from its props/types and the design-system API, not merely the subset the issue happens to mention) — then cross-reference with the Figma spec and build the matrix on that. Don't omit a supported state because the ticket didn't list it; don't invent an axis Figma doesn't show (e.g. a static component has no hover/pressed).
**Exclude theme from the matrix.** Figma usually models light/dark as a component *property* (because it doesn't use variable modes), but theme is **not** a real component prop in the app — it's global. So never make theme a matrix axis or column; drive light/dark with the global Storybook `theme` toggle instead (see below).
Typical layout (mirrors the Figma component sheet): a `StoryShowcase` title, then a single **CSS-grid matrix** — state labels down the left column, one column per axis-combination (e.g. `variant × size`), each column headed by the real JSX usage via `StoryJsx`. Use `gridTemplateColumns: \`<labelWidth> repeat(N, max-content)\`` with a `React.Fragment` per row.
### Theme: use the global toggle, render ONE copy
Render the showcase once and let Storybook's global `theme` control (defined in `.storybook/preview.tsx`) switch light/dark. Do **not** render light and dark side by side: Metabase bakes per-scheme hex into `--mantine-color-*` at `:root`, and Mantine color tokens follow the global scheme — a nested scheme wrapper won't re-scope them.
### Forcing hover/pressed for the matrix
Use **`storybook-addon-pseudo-states`** (registered in `.storybook/main.ts`); don't add `data-force-*` hooks to component CSS. Set `parameters.pseudo` to force states per cell. Two things make or break it:
1. **Aim the selector at the element the pseudo-class actually sits on**, which for a Mantine component is often an inner slot, not the root. A string/array value applies `.pseudo-<state>` to the matched element, so it must match that slot. Import the CSS-module class (`S.<Slot>`) to target it.
2. **Land your per-cell hook on the root via the factory's root-props slot** (e.g. `wrapperProps`), because Mantine forwards top-level `data-*`/rest props to an inner element, not the root. Check the component's `.d.ts` for the right slot.
The addon toolbar toggle keeps working independently (it forces every rule in the story).
### Example: a finalized matrix story
Illustrative — adapt the axes and states to the component you're working on. This is the shape the pieces above add up to (state labels on the left, one column per `variant × size`, forced hover/pressed via `parameters.pseudo`, the root hook on `wrapperProps`, controls scoped to the meaningful knobs):
```tsx
import type { StoryFn } from "@storybook/react";
import { Fragment } from "react";
import { Box, Chip, type ChipProps, Text } from "metabase/ui";
import { StoryJsx, StoryShowcase } from "metabase/ui/stories/showcase";
import S from "./Chip.module.css";
// Axes derived from the component's API (here: variant × size).
const COLUMNS = [
{ variant: "light", size: "sm" },
{ variant: "light", size: "md" },
{ variant: "filled", size: "sm" },
{ variant: "filled", size: "md" },
] as const;
// Every state the component supports; `id` doubles as the pseudo-state hook.
const STATES = [
{ id: "default", label: "Default" },
{ id: "hover", label: "Hover" },
{ id: "pressed", label: "Pressed" },
{ id: "selected", label: "Selected", checked: true },
{ id: "hover-selected", label: "Hover selected", checked: true },
{ id: "disabled", label: "Disabled", disabled: true },
];
// Force hover/pressed by targeting the slot the pseudo-class sits on (.ChipLabel).
const labelSelector = (id: string) => `[data-state-row="${id}"] .${S.ChipLabel}`;
const Overview: StoryFn<ChipProps> = ({ children }) => (
<StoryShowcase title="Chip">
<Box
style={{
display: "grid",
gridTemplateColumns: `9rem repeat(${COLUMNS.length}, max-content)`,
columnGap: "2rem",
rowGap: "1rem",
alignItems: "center",
}}
>
<div />
{COLUMNS.map(({ variant, size }) => (
<StoryJsx
key={`${variant}-${size}`}
>{`<Chip variant="${variant}" size="${size}" />`}</StoryJsx>
))}
{STATES.map((state) => (
<Fragment key={state.id}>
<Text size="sm" c="text-secondary">
{state.label}
</Text>
{COLUMNS.map(({ variant, size }) => (
<Chip
key={`${variant}-${size}`}
wrapperProps={{ "data-state-row": state.id }} // hook on the ROOT
variant={variant}
size={size}
checked={state.checked ?? false}
disabled={state.disabled ?? false}
onChange={() => {}}
>
{children}
</Chip>
))}
</Fragment>
))}
</Box>
</StoryShowcase>
);
export const OverviewStory = {
render: Overview,
parameters: {
pseudo: {
hover: ["hover", "hover-selected"].map(labelSelector),
active: ["pressed"].map(labelSelector),
},
controls: { include: ["children", "theme"] },
},
};
```
### Story conventions
- Keep an interactive default story plus the matrix story.
- Restrict `argTypes` to the supported API (e.g. only the sizes the DS uses).
- **Scope controls per story.** A static matrix should hide knobs it ignores: `parameters.controls = { include: [...] }` (or `{ disable: true }`). Wire any kept knob (e.g. `children`) through to the cells so it does something.
- `loki.config.js` has a `storiesFilter`; check it before renaming a story that's already covered.
Validate: `bun run lint-eslint-pure -- <files>` and `bun run type-check-pure`.
### Optional: verify in the browser yourself
If a browser-driving MCP is connected (e.g. **chrome-devtools-mcp** or **playwright-mcp**: `navigate_page`, `take_screenshot`, `take_snapshot`, `evaluate_script`, `resize_page`), use it to check your own work against the running Storybook (`http://localhost:6006/iframe.html?id=<story-id>`, the kebab-cased title + story). Screenshot for a visual read; `evaluate_script`/`take_snapshot` to confirm DOM facts (a hook landed on the root, pseudo classes applied, a resolved `--mb-color-*`). This is a **convenience, never a requirement** — without it, rely on the user's visual check (and you may suggest installing the plugin). Storybook is usually already running; don't start it. (Screenshot writes may be sandboxed to the repo root — save under `local/` then `mv`.)
Pause here and let the user look. Iterate on layout/spacing only — **no component style changes yet.**
## Phase 4 — Style the component from Figma
Now change the component's `.config.ts` / `.module.css`. Get **exact tokens from Figma** — never eyeball hex from a screenshot.
### Figma desktop MCP is required for tokens
`get_screenshot` works by `nodeId`, but `get_variable_defs`, `get_design_context`, and `get_metadata` read the **current selection in the Figma desktop app** via the **Dev Mode MCP server**. If they error with "nothing selected" or the server isn't connected, **prompt the user** to enable the Dev Mode MCP server (Switch to Dev Mode → MCP section in right sidebar) and select the relevant layer(s). Don't pixel-sample a screenshot as a substitute.
### Extract and map tokens
1. `get_metadata` on the spec frame for the node id + name of each variant/size/state symbol.
2. `get_variable_defs` per symbol for exact colors, spacing, radius, font.
3. `get_design_context` (forceCode) on one symbol for exact CSS dimensions (padding, gap, icon size). It may ask about Code Connect — skip that.
4. **Use the semantic token names `get_variable_defs` returns.** An up-to-date spec binds each property to a semantic token (a role like `text/hover`, `background/selected`); match it to the corresponding `--mb-color-*` key (key list: `frontend/src/metabase/ui/colors/types/color-keys.ts`) and use it directly — one definition then works in both themes. Sanity-check the resolved shade against `colors/constants/themes/light.ts` / `dark.ts` and `base-colors.ts`. Caveat: returned *values* may be the wrong mode for dark symbols — trust the variable **names**, not the dark hex.
### Color rules — semantic tokens only, NO `color-mix`, NO primitives
Per `docs/developers-guide/frontend.md` ("Colors"): every color must be a **semantic** `--mb-color-*` token. No literal hex, and no `color-mix` in component CSS to fake a shade/alpha.
**Guardrail:** if `get_variable_defs` returns a **primitive ramp** (e.g. `Orion-Alpha/10`, `Ocean/60` — a ramp + number, not a role) bound directly to a property, do **not** translate it to a ramp, hardcode hex, or `color-mix` your way to it. **Stop and flag it to the user** with the specific property/state — it's a design-side gap (bind to a semantic token, or add one). Resume once the spec exposes a semantic token.
### Non-color tokens — flag scale mismatches too
The same discipline applies to **dimensional** values — radius, spacing/padding/gap, elevation/shadow, and typography (font size, line-height, weight). Prefer the codebase's existing scale variable over a literal: `--mantine-radius-*`, `--mantine-spacing-*`, `--mantine-shadow-*`, and the Mantine font tokens.
But Figma's scales don't always line up with the codebase's. When a Figma value **has no matching step** in the corresponding scale — e.g. Figma `radius/md = 12px` while `--mantine-radius-md` is `8px`, or a Figma `xxxs` spacing step with no Mantine equivalent — **don't silently bake in a literal.** Flag it to the user the same way you flag primitive colors: name the property, the Figma value, and the nearest codebase token, so it can be resolved with design / the scale owners (align the value, or add a scale step). Only fall back to a literal if the user confirms it's an accepted one-off, and leave a short comment in the code explaining why.
### Mantine implementation notes (reference)
General gotchas when extending Mantine components — adapt per component:
- **Know which slot each data-attribute lands on.** Mantine puts state attributes on specific slots (e.g. `data-variant` on the root, `data-checked`/`data-disabled` on an inner slot). Scope rules accordingly, usually through a root `classNames.root` class.
- **Out-specify Mantine's built-in variant CSS.** It styles slots at `.m_*:not([data-disabled])` specificity; a descendant combinator through your root class wins. Avoid chained `:not()` (stylelint's `selector-not-notation: complex` rejects it, and combining lowers specificity) — gain specificity via the root class / attributes instead.
- **Size via a `vars` resolver**, returning only Mantine's typed vars for that component (custom var names fail type-check). Map size → vars in `.config.ts`.
- **Built-in icons use `currentColor`**, so they inherit the slot's text color.
- For text on a brand fill, follow the existing convention (`text-primary-inverse`) unless the user wants pure white; flag the dark-theme implication.
After edits: `npx stylelint <css>`, `LINT_CSS_MODULES=true bun run lint-eslint-pure -- <files>`, `bun run type-check-pure`.
## Phase 5 — Iterate with the user
Show the result in the running Storybook and address feedback (color, spacing, radius, shadow, missing/extra state). Surface deliberate deviations explicitly so the user can veto them. If a browser MCP is connected, screenshot the matrix in both themes to catch issues first. **Do not commit until the user is satisfied.**
## Phase 6 — Commit, then migrate call sites
Only after sign-off:
1. Commit when the user explicitly asks; commit only relevant files (exclude build artifacts and the gitignored `local/` doc). Follow the user's commit conventions.
2. **Update mode:** migrate the call sites flagged in Phase 2 if the restyle requires it (e.g. invalid/renamed props). Be surgical with bulk find/replace — a blanket prop rename can hit unrelated components; verify each hit is the target. **Create mode:** confirm the public export is wired through the barrels and used as the ticket intends.
3. Leave the findings doc in place for the user's manual verification pass.
---
## Checklist
- [ ] Linear issue fetched; Figma node/fileKey + `gitBranchName` extracted; on the right branch; mode (update vs create) decided.
- [ ] **Update:** usage + blast radius in `local/<TICKET>-<component>-usage-findings.md`; behavior locked with unit tests **if** the component has custom interactive logic (skip for thin Mantine wrappers / pure restyles). **Create:** component scaffolded with its public surface and barrel exports.
- [ ] Storybook showcase matrix built from the component's full state space (theme excluded — global toggle drives light/dark), single panel, hover/pressed forced via `storybook-addon-pseudo-states`, controls scoped per story; lint + type-check pass. (Update mode: built before styling, **component styles untouched**. Create mode: built after the component is implemented in Phase 4.)
- [ ] Figma desktop MCP available; exact tokens extracted and mapped to semantic `--mb-color-*` (NO `color-mix`, NO primitives) and to existing scale vars for radius/spacing/elevation/type — any color *or* dimensional value with no matching token/scale step flagged to the user, not baked in as a literal.
- [ ] Component styled; stylelint + eslint(+CSS modules) + type-check pass.
- [ ] Iterated with the user until satisfied.
- [ ] Committed (when asked); call sites migrated/verified (update) or export wired (create); findings doc handed back.

View File

@@ -1,33 +1,20 @@
---
name: typescript-review
description: Review TypeScript and JavaScript code changes for compliance with Metabase coding standards, style violations, and code quality issues. Use when reviewing pull requests or diffs containing TypeScript/JavaScript code.
allowed-tools: Read, Grep, Bash, Glob, Skill
allowed-tools: Read, Grep, Bash, Glob
---
# TypeScript/JavaScript Code Review Skill
@./../_shared/typescript-commands.md
@./../_shared/react-redux-patterns.md
## Main Focus
## Code Review Guidelines
**Primary standard: the [`typescript-write`](../typescript-write/SKILL.md) skill.** Load it first — it defines the authoring rules this review enforces, alongside `frontend/CLAUDE.md` and `docs/developers-guide/frontend.md`.
Review pull requests with a focus on:
Adherence to `typescript-write` is the **highest-priority** review dimension: rank any violation of its provisions above all other findings. Treat its **no-`any` hard rule** (no explicit *or* implicit `any` in new code) as **blocking**. Use TypeScript LSP tools to inspect inferred types when available; otherwise rely on type-checking and linting.
Review in this priority order:
1. **Violations of [`typescript-write`](../typescript-write/SKILL.md) provisions** — no-`any`, type tightening, type modeling, null/undefined handling, naming, structure, comments. Highest priority; block on the no-`any` rule.
2. Compliance with `frontend/CLAUDE.md`.
3. Readability and maintainability.
4. Appropriate test coverage.
## Blind spots — act as the missing reviewer
These rarely surface in team reviews, so this skill should raise them. They are **additive** — raise them, but rank them below `typescript-write` violations:
- **Accessibility.** Interactive elements need keyboard support, focus management, and accessible names. Flag missing `aria-label`/`aria-labelledby`, non-semantic click targets, modals without focus trap, icon-only buttons without labels, and form inputs without a linked label.
- **Performance.** Flag areas that scale poorly and aren't memoized; inline object/array literals passed to memoized children; effects that fire on every batch of a progressive load; and new dependencies added to hot paths.
- **Security.** Evaluate potential security issues in new code.
- **Bundle size.** Flag new large dependencies, default imports from icon or util libs, and heavy modules imported at route-load time.
- **Analytics.** User-facing flows should emit tracking events. If a PR adds a new flow (button, modal, navigation) without a tracking event, ask whether one is expected.
- Readability and maintainability
- Appropriate test coverage
- Compliance with project coding standards and conventions
- Code quality and best practices
- Type safety and proper TypeScript usage
- React best practices (when applicable)

View File

@@ -7,74 +7,3 @@ description: Write TypeScript and JavaScript code following Metabase coding stan
@./../_shared/development-workflow.md
@./../_shared/typescript-commands.md
@./../_shared/react-redux-patterns.md
## No `any` — hard rule
- **New code must not introduce `any`, explicit or implicit.** No `any` annotations, no `as any` / `as unknown as`, no untyped parameters or returns that infer `any`, no implicitly-`any` destructures or array/object literals.
- **Untyped third-party / boundary values** must be typed at the boundary (a declared type, `unknown` + type guard, or a small typed wrapper) — never let `any` propagate inward.
- **Mandatory type verification.** Before finishing a TS/TSX change, run `bun run type-check-pure`. If TypeScript LSP tools are available, also inspect changed symbols with hover and go-to-definition; otherwise skip LSP check.
## Type tightening
- **Avoid type casts and loose `unknown`** — fix the signature instead.
- **If a function only needs one field of a wide object, accept that field** — not the wide object. The cast often disappears once the signature is right.
- **Reach for `Partial<T>`, `Pick<T, K>`, `Record<K, V>`, and generics** before reaching for a cast.
- **Prefer making props/components generic** (`<T>`) when a value flows through unchanged and the caller knows the type.
- **Prefer `unknown` over loose typing** and narrow before use — an `unknown` value forces a guard at the point of use.
- **`satisfies` for object literals** that must conform without widening (config objects, lookup maps, discriminated literals) — better than `: T` (widens) or `as T` (unsafe).
- **Avoid non-null assertions (`!`)**. Prefer a guard, early return, or `?.`. Use `!` only when non-nullness is provably true and localized, with a comment.
- **No redundant runtime coercion** — don't wrap already-typed values in `Number()` / `String()` / `Boolean()`.
- **Type guards belong in `frontend/src/metabase-types/guards/`**. Do not redefine them locally.
- **A cast you can't avoid needs a real justification comment.** The `metabase/no-unjustified-type-casts` rule accepts any preceding comment — state the actual reason the cast is safe. NEVER write the legacy `// Unjustified type cast. FIXME` placeholder; it exists only on casts that predated the rule, and copying it sneaks an unjustified cast past the linter. If you can't articulate why the cast is correct, the cast is wrong — fix the types.
## Type modeling
- **Reuse existing types; don't re-declare them.** Use canonical IDs and domain entity types from `metabase-types/api` (`FieldId`, `TableId`, `ConcreteTableId`, `SchemaName`, …) and key data structures by them (`new Map<ConcreteTableId, …>()`). Don't duplicate generated/API types — compose or derive (`Pick`, `Omit`, indexed access `SomeType["field"]`, `ReturnType`).
- **Use generics to allow TypeScript to infer correct types** when creating functions and components that need to be reusable and type-safe. Don't hesitate to introduce complex generics if they allow to derive types automatically instead of manual narrowing.
- **Model the actual data contract; keep types narrow.** Optional `field?: T` for a key that may be absent, `field: T | undefined` only when the key is always present but the value may be undefined, `| null` for explicit API nulls. Prefer domain unions over broad `string` / `number` / loose `Record`.
- **Refer to API implementation** when defining or refining types to ensure they match the actual data structure. When considering a type cast, first consider if the type should be refined to match the actual data structure.
- **Discriminated unions for variant state, with exhaustive checks.** Model "one of N shapes" as a union with a literal discriminant rather than a bag of optional fields, and exhaust it with ts-pattern's `.exhaustive()` so adding a variant becomes a compile error:
```ts
import { match } from "ts-pattern";
const result = match(status)
.with({ type: "loading" }, () => <Spinner />)
.with({ type: "error", error: P.select() }, (error) => <Error message={error.message} />)
.with({ type: "success", data: P.select() }, (data) => <Content data={data} />)
.exhaustive(); // Compile-time guarantee all cases handled
```
- **Derive union types from constants** (`as const` + `typeof`/`keyof`) so the type and the values can't drift.
- **`readonly` / immutability where mutation isn't intended** — component props, shared constants, exported config. Prefer `readonly T[]` / `ReadonlyArray<T>` for inputs you don't mutate.
- **Type async and error states explicitly** (a discriminated union or the data-layer's typed result) — never leave loading/error/empty implicit.
## Null and undefined
- **Narrow at the source**. If a value is optional only in a corner case, don't thread `undefined` through every layer — guard at the producer.
- **Sensible defaults for optional values**. Use `?.` and `??` at the consumer.
- **Lists should be filtered** before being used in a map or other iteration.
- **Avoid non-strict null comparisons** (`X != null`) when `X` can never be `null` — use a strict check or narrow the type. Use `checkNotNull` where necessary.
- **Check actual nullability against API implementation**. Find the API endpoint implementation and check if the field can actually be null.
## Naming
- **Names describe the entity, not the mechanism**. A name must reflect what the value holds.
- **Align sibling concepts**: keep verb conventions consistent across a related API.
- **No names that encode implementation history** rather than current meaning. Suffixes like `Base`, `New`, `Old`, `Initial` need a real semantic distinction, otherwise drop them.
- **Avoid cryptic identifiers** (`v`, `n`, `$n`) for domain values; short names are fine only in tiny conventional contexts (loop index `i`, coordinates `x`/`y`, generic params `T`/`K`/`V`).
## Code structure and organization
- **Prioritize reusability over duplication**. The codebase already has many utility functions — leverage them. If you introduce duplicated logic, extract it to a shared utility.
- **Generic helpers do not belong in feature folders**. Promote to a shared utility.
- **Keep functions small and single-purpose**. A 100+ line function is hard to review — split into focused named helpers, each with one responsibility and a minimal dependency surface. When necessary, cover with unit tests.
- **Extract distinct complex JSX into named components**. Choose same-file vs separate-file by reuse, coupling, testability, and readability.
## Comments
- **No comments by default**. Well-named identifiers carry the `what`.
- **Comments should be concise**. Add a short, concise comment only when the `why` is non-obvious: a workaround, a hidden invariant, a subtle ordering constraint, a clever reduction. Never document the actual implementation, focus on the intent and the why.
## Verify before done
- **Run the project type-check** when finished (see the shared TypeScript commands above).

View File

@@ -7,6 +7,7 @@
:aliased-namespace-symbol {:level :warning}
:case-symbol-test {:level :warning}
:condition-always-true {:level :warning}
:def-fn {:level :warning}
:dynamic-var-not-earmuffed {:level :warning}
:equals-expected-position {:level :warning, :only-in-test-assertion true}
@@ -49,11 +50,9 @@
;; disabled linters
;;
:constant-condition {:level :off} ; TODO (sashko): too confused by defenterprises, some hook/config is needed
:redundant-ignore {:level :off} ; lots of false positives, see https://github.com/clj-kondo/clj-kondo/issues/2414
:unexpected-recur {:level :off} ; TODO (cam): I think we just need to tell it how to handle MBQL match and we can enable this?
:redundant-primitive-coercion {:level :off} ; there are many defensive boolean coercions, kondo looks too much into it
:type-mismatch {:level :off} ; Still quite raw, can sometimes give useful insights, but sometimes is just confused
;;
;; globally-enabled custom linters
@@ -62,7 +61,6 @@
:metabase/check-def-check-not-uppercase-name {:level :warning}
:metabase/check-def-no-underscores {:level :warning}
:metabase/check-me-humanize {:level :warning}
:metabase/check-testing-not-empty {:level :warning}
:metabase/defsetting-must-specify-export {:level :warning}
:metabase/defsetting-namespace {:level :warning}
:metabase/discourage-millis-duration {:level :warning}
@@ -180,38 +178,19 @@
clojure.core.async/take!
clojure.core.async/to-chan!
clojure.core.async/to-chan!!
;; only writes .clj-kondo/ratchets.edn, from a :once fixture that finishes before the parallel tests
dev.kondo-ratchet/fix!
metabase.ai-tracing.core/record!
metabase.api.macros.defendpoint.tools-manifest/assert-optional-fields-nullable!
metabase.channel.core/send!
metabase.channel.render.pdf/draw-too-large!
metabase.channel.render.pdf/draw-no-results!
metabase.channel.render.pdf/render-card-cell!
metabase.channel.render.pdf.font/load-fonts!
metabase.warehouses.models.database/assert-not-h2!
metabase.driver.sql-jdbc.execute/execute-prepared-statement!
metabase.mcp.usage/record-mcp-session!
metabase.mcp.usage/record-mcp-session-end!
metabase.mcp.usage/record-mcp-tool-call!
metabase.notification.models/create-notification!
metabase.oauth-server.core/reset-provider!
metabase.pulse.send/send-pulse!
metabase-enterprise.transforms-python.python-runner-test/execute!
;; MCP usage tests isolate their rows by unique keys (session id / tool name / duration /
;; client version) and clean up only those, so these deletes are safe in parallel.
metabase-enterprise.mcp.usage-test/cleanup!
metabase-enterprise.mcp.usage-test/cleanup-calls!
;; Workspace isolation functions are safe in parallel tests because they operate on
;; isolated, test-specific workspaces and databases that don't affect other tests
metabase-enterprise.workspaces.isolation/destroy-workspace-isolation!
metabase.lib-metric.ast.plan/validate-arithmetic-ast!
metabase-enterprise.workspaces.isolation/ensure-database-isolation!
metabase-enterprise.workspaces.test-util/create-ready-ws!
metabase-enterprise.workspaces.core/clear-instance-workspace!
metabase-enterprise.workspaces.core/set-instance-workspace!
metabase.workspaces.core/clear-instance-workspace!
metabase.workspaces.core/set-instance-workspace!
;; Safe in tests, where we typically use a private, temporary index per thread.
metabase.search.core/reindex!
metabase.util/run-count!
@@ -221,16 +200,6 @@
{:level :warning
:max-length 150}
;; Warn on long tests (tests over `:min-length`) that look like
;;
;; (deftest my-test
;; (testing ...)
;; (testing ...)
;; (testing ...))
:metabase/validate-deftest-logically-separate-tests
{:level :warning
:min-length 60}
;; this is not enabled here, it is enabled below only for specific namespaces.
:metabase/disallow-hardcoded-driver-names-in-tests
;; don't include `:h2` since that will probably lead to a mountain of tests where we are specifically testing
@@ -263,6 +232,9 @@
;; misc config for built-in linters
;;
:invalid-arity {:skip-args [metabase.util.match/match-one
metabase.util.match/match-many
metabase.util.match/replace]} ; TODO (cam): can we fix these?
:refer-all {:level :warning, :exclude [clojure.test]}
:unused-referred-var {:exclude {compojure.core [GET DELETE POST PUT]}}
@@ -282,6 +254,7 @@
(metabase.util.match/match-many)
(metabase.util.match/replace-in)
(metabase.util.match/replace)
(metabase.driver-api.core/match)
(metabase.driver-api.core/match-one)
(metabase.driver-api.core/match-many)
(metabase.driver-api.core/replace)
@@ -301,10 +274,7 @@
metabase.query-processor.util/normalize-token {:namespaces ["metabase.*"]}}}
:discouraged-var
{metabase-enterprise.entity-retrieval.core/search-unfiltered {:message "Permission bypass risk: library-index hits are unfiltered — resolve each entity and drop what the current user can't read"}
metabase.entity-retrieval.core/search-unfiltered {:message "Permission bypass risk: library-index hits are unfiltered — resolve each entity and drop what the current user can't read"}
metabase.entity-retrieval.mirror/search-unfiltered {:message "Permission bypass risk: library-index hits are unfiltered — resolve each entity and drop what the current user can't read"}
clojure.core/print {:message "Use metabase.util.log instead of clojure.core/print"}
{clojure.core/print {:message "Use metabase.util.log instead of clojure.core/print"}
clojure.core/println {:message "Use metabase.util.log instead of clojure.core/println"}
clojure.core/printf {:message "Use metabase.util.log instead of clojure.core/printf"}
clojure.core/prn {:message "Use metabase.util.log instead of clojure.core/prn"}
@@ -359,7 +329,10 @@
:unresolved-var
{:exclude
[colorize.core]}
[colorize.core
;; Carmine defines its Redis command vars (xadd, xreadgroup, …) via macros at load time,
;; so clj-kondo can't see them statically.
taoensso.carmine]}
:consistent-alias
{:aliases
@@ -403,7 +376,6 @@
metabase.channel.render.core channel.render
metabase.cloud-migration.core cloud-migration
metabase.eid-translation.core eid-translation
metabase.llm.test-util llm.tu
metabase.login-history.core login-history
metabase.model-persistence.core model-persistence
metabase.notification.core notification
@@ -446,6 +418,7 @@
ring.util.codec codec
ring.util.response response
saml20-clj.core saml
taoensso.carmine car
metabase.transforms-base.util transforms-base.u
metabase.transforms.util transforms.u
toucan2.core t2}}}
@@ -460,11 +433,6 @@
instaparse.core/defparser clojure.core/def
metabase-enterprise.action-v2.test-util/with-test-tables! clojure.core/let
metabase-enterprise.content-translation.routes-test/with-new-secret-key! clojure.core/fn
metabase-enterprise.entity-retrieval.reconcile-test/with-isolated-index clojure.core/fn
metabase-enterprise.mfa.enrollment-flow-test/with-fresh-user-session! clojure.core/fn
metabase-enterprise.mfa.login-flow-test/with-enrolled-rasta! clojure.core/fn
metabase-enterprise.mfa.recovery-codes-test/with-confirmed-enrollment! clojure.core/fn
metabase-enterprise.serialization.metadata-file-import.wire-format-test/with-tiny-fixture! clojure.core/fn
metabase-enterprise.serialization.test-util/with-dbs clojure.core/fn
metabase-enterprise.serialization.test-util/with-random-dump-dir clojure.core/let
metabase-enterprise.workspaces.test-util/with-resources! clojure.core/let
@@ -472,8 +440,6 @@
metabase.actions.test-util/with-actions clojure.core/let
metabase.transforms.test-util/with-transform-cleanup! clojure.core/let
metabase.api.common/let-404 clojure.core/let
metabase.app-db.connection/with-unshared-connection clojure.core/fn
metabase.app-db.core/with-unshared-connection clojure.core/fn
metabase.app-db.custom-migrations.util/with-temp-scheduler! clojure.core/fn
metabase.app-db.custom-migrations/define-migration clj-kondo.lint-as/def-catch-all
metabase.app-db.custom-migrations/define-reversible-migration clj-kondo.lint-as/def-catch-all
@@ -482,7 +448,6 @@
metabase.channel.impl.http-test/with-captured-http-requests clojure.core/fn
metabase.channel.impl.http-test/with-server clojure.core/let
metabase.collections.models.collection-test/with-collection-in-location clojure.core/let
metabase.collections.test-utils/with-library clojure.core/fn
metabase.driver.mongo.connection/with-mongo-client clojure.core/let
metabase.driver.mongo.connection/with-mongo-database clojure.core/let
metabase.driver.mongo.query-processor/mongo-let clojure.core/let
@@ -496,7 +461,6 @@
metabase.notification.test-util/with-card-notification clojure.core/let
metabase.notification.test-util/with-system-event-notification! clojure.core/let
metabase.notification.test-util/with-temp-notification clojure.core/let
metabase.osi.ai-context.api-test/with-test-entry clojure.core/let
metabase.premium-features.core/defenterprise-schema clj-kondo.lint-as/def-catch-all
metabase.premium-features.defenterprise/defenterprise-schema clj-kondo.lint-as/def-catch-all
metabase.premium-features.settings/define-premium-feature clojure.core/def
@@ -505,9 +469,7 @@
metabase.query-processor.pipeline-queries-test/mbql5-query clojure.core/->
metabase.query-processor.pipeline-queries-test/run-mbql5-query clojure.core/->
metabase.query-processor.error-type/deferror clojure.core/def
metabase.query-processor.core/with-reducible-deserialized-results clojure.core/let
metabase.query-processor.middleware.cache.impl/with-reducible-deserialized-results clojure.core/let
metabase.query-processor.result-serialization/with-reducible-deserialized-results clojure.core/let
metabase.query-processor.middleware.process-userland-query-test/with-query-execution! clojure.core/let
metabase.query-processor.setup/with-qp-setup clojure.core/let
metabase.search.api-test/do-test-users clojure.core/let
@@ -524,6 +486,7 @@
metabase.test.data.interface/defdataset clojure.core/def
metabase.test.data.interface/defdataset-edn clojure.core/def
metabase.test.embedded-postgres.core/with-system clojure.core/let
metabase.test.gentest/defgentest clojure.test/deftest
metabase.test/defdataset clojure.core/def
metabase.test/let-url clojure.core/let
metabase.test/with-actions clojure.core/let
@@ -534,7 +497,6 @@
metabase.test/with-temp-empty-app-db clojure.core/let
metabase.test/with-temp-file clojure.core/let
metabase.test/with-user-in-groups clojure.core/let
metabase.transforms-rest.api.transform-runs-test/with-runs-fixture! clojure.core/fn
metabase.upload.impl-test/with-upload-table! clojure.core/let
metabase.util/for-map clojure.core/for
metabase.util.files/with-open-path-to-resource clojure.core/let
@@ -565,7 +527,6 @@
{:analyze-call
{cljs.test/deftest hooks.clojure.test/deftest
cljs.test/is hooks.clojure.test/is
cljs.test/testing hooks.clojure.test/testing
cljs.test/use-fixtures hooks.clojure.test/use-fixtures
clojure.core/def hooks.clojure.core.def/lint-def
clojure.core/definterface hooks.clojure.core.def/lint-def
@@ -584,7 +545,6 @@
clojure.core/- hooks.metabase.util.timing/discourage-millis-duration
clojure.test/deftest hooks.clojure.test/deftest
clojure.test/is hooks.clojure.test/is
clojure.test/testing hooks.clojure.test/testing
clojure.test/use-fixtures hooks.clojure.test/use-fixtures
malli.error/humanize hooks.malli.error/humanize
metabase-enterprise.advanced-permissions.models.permissions.application-permissions-test/with-new-group-and-current-graph hooks.common/with-two-top-level-bindings
@@ -600,7 +560,7 @@
metabase.api.common.internal-test/with-jetty-server hooks.common/with-one-binding
metabase.api.macros/defendpoint hooks.metabase.api.macros/defendpoint
metabase.app-db.schema-migrations-test.impl/test-migrations hooks.metabase.app-db.schema-migrations-test.impl/test-migrations
metabase.channel.render.js.pool/with-static-viz-context hooks.common/with-one-top-level-binding
metabase.channel.render.js.svg/with-static-viz-context hooks.common/with-one-top-level-binding
metabase.channel.template.handlebars-helper/defhelper hooks.metabase.channel.template.handlebars-helper/defhelper
metabase.channel.template.handlebars-test/with-temp-template! hooks.common/with-vec-first-binding
metabase.classloader.core/require hooks.clojure.core.require/lint-require
@@ -633,7 +593,6 @@
metabase.lib.test-util.macros/mbql-query hooks.metabase.test.data/mbql-query
metabase.lib.test-util.macros/mbql-5-query hooks.metabase.test.data/mbql-query
metabase.lib.test-util.macros/with-testing-against-standard-queries hooks.metabase.lib.test-util.macros/with-testing-against-standard-queries
metabase.metrics.api-dimension-test/with-seeded-metric hooks.common/with-one-binding
metabase.model-persistence.api-test/with-persistence-setup! hooks.common/with-one-top-level-binding
metabase.model-persistence.api-test/with-setup! hooks.common/with-one-top-level-binding
metabase.model-persistence.test-util/with-persistence-enabled! hooks.metabase.model-persistence.test-util/with-persistence-enabled!
@@ -682,6 +641,7 @@
metabase.test.data/dataset hooks.metabase.test.data/dataset
metabase.test.data/mbql-query hooks.metabase.test.data/mbql-query
metabase.test.data/run-mbql-query hooks.metabase.test.data/mbql-query
metabase.test.gentest/iterate hooks.metabase.test.gentest/iterate*
metabase.test.util.async/with-open-channels hooks.common/let-with-optional-value-for-last-binding
metabase.test.util.generators.jvm/with-random-cards hooks.metabase.test.util.generators.jvm/with-random-cards
metabase.test.util.log/with-log-level hooks.common/with-ignored-first-arg
@@ -730,10 +690,6 @@
metabase.util.log/warn hooks.metabase.util.log/info
metabase.util.log/warnf hooks.metabase.util.log/infof
metabase.util.malli.registry/def hooks.metabase.util.malli.registry/def
metabase.util.match/match-one hooks.metabase.util.match/match-one
metabase.util.match/match-many hooks.metabase.util.match/match-many
metabase.util.match/replace hooks.metabase.util.match/replace
metabase.util.match/replace-in hooks.metabase.util.match/replace-in
metabase.util/case-enum hooks.metabase.util/case-enum
metabase.util/format-color hooks.metabase.util/format-color
metabase.xrays.api.automagic-dashboards-test/with-indexed-model! hooks.metabase.xrays.api.automagic-dashboards-test/with-indexed-model!
@@ -747,7 +703,6 @@
clojurewerkz.quartzite.schedule.simple/schedule macros.quartz/simple-schedule
clojurewerkz.quartzite.triggers/build macros.quartz/build-trigger
metabase-enterprise.database-routing.e2e-test/with-temp-dbs! macros.metabase-enterprise.database-routing.e2e-test/with-temp-dbs!
metabase-enterprise.database-routing.e2e-test/with-routing-setup! macros.metabase-enterprise.database-routing.e2e-test/with-routing-setup!
metabase-enterprise.impersonation.util-test/with-impersonations! macros.metabase-enterprise.impersonation.util-test/with-impersonations!
metabase-enterprise.query-reference-validation.api-test/with-test-setup! macros.metabase-enterprise.query-reference-validation.api-test/with-test-setup!
metabase-enterprise.sandbox.test-util/with-gtaps! macros.metabase-enterprise.sandbox.test-util/with-gtaps!
@@ -778,7 +733,6 @@
metabase.test/with-temporary-raw-setting-values macros.metabase.test.util/with-temporary-raw-setting-values
metabase.users.models.user-test/with-groups! macros.metabase.users.models.user-test/with-groups!
metabase.util.namespaces/import-fn macros.metabase.util.namespaces/import-fn
metabase.util.namespaces/import-macro macros.metabase.util.namespaces/import-macro
metabase.xrays.related-test/with-world macros.metabase.xrays.related-test/with-world}}
:config-in-comment
@@ -800,7 +754,7 @@
:name test-namespaces}
{:pattern "^(?!.*-test(?:\\..*)?$)(?!.*test-util)(?!^metabase\\.test.*).*$"
:name source-namespaces}
{:pattern "(?:(?:^metabase\\.cmd.*)|(?:^build.*)|(?:^metabuild-common.*)|(?:^release.*)|(?:^i18n.*)|(?:^lint-migrations-file$))|(?:^mage.*)|(?:^load-namespaces.*)"
{:pattern "(?:(?:^metabase\\.cmd.*)|(?:^build.*)|(?:^metabuild-common.*)|(?:^release.*)|(?:^i18n.*)|(?:^lint-migrations-file$))|(?:^mage.*)"
:name printable-namespaces}
{:pattern "^metabase\\.(?:(?:lib)|(?:legacy-mbql))\\..*"
:name metabase-lib}
@@ -836,8 +790,16 @@
;;
{:pattern "^metabase(?:((?:-enterprise\\.sandbox)?\\.query-processor)|(?:\\.driver\\.sql(?!server)(?!ite))).*-test$"
:name general-qp-and-driver-test-namespaces}
{:pattern "(?:^mage.*)"
:name mage-namespaces}]
:name mage-namespaces}
;;
;; Namespaces where `with-redefs` is still the idiom because they haven't been audited for conversion
;; to `with-dynamic-fn-redefs` yet (enterprise tests, driver tests). The linter's nudge is too noisy
;; here — audit can happen in follow-up PRs.
;;
{:pattern "^(?:metabase-enterprise\\..+-test(?:\\..+)?|metabase\\.driver\\..+-test(?:\\..+)?)$"
:name prefer-with-dynamic-fn-redefs-unaudited}]
:config-in-ns
{test-namespaces
@@ -861,10 +823,9 @@
metabase.util.time-test
{:linters {:metabase/prefer-with-dynamic-fn-redefs {:level :off}}}
;; classloader sits below `util` in the module graph, so it can't use `metabase.util.log` without creating
;; a cycle; it gets to use tools.logging directly.
metabase.classloader.impl
{:linters {:discouraged-namespace {clojure.tools.logging {:level :off}}}}
;; Enterprise tests and driver tests haven't been audited yet — silence the linter so CI can ship.
prefer-with-dynamic-fn-redefs-unaudited
{:linters {:metabase/prefer-with-dynamic-fn-redefs {:level :off}}}
source-namespaces
{:linters
@@ -960,7 +921,10 @@
:deprecated-namespace
{:exclude
[metabase.legacy-mbql.normalize
[metabase.driver.common.parameters.dates
metabase.driver.common.parameters.operators
metabase.driver.common.parameters.values
metabase.legacy-mbql.normalize
metabase.legacy-mbql.schema
metabase.legacy-mbql.schema.helpers
metabase.legacy-mbql.schema.macros

File diff suppressed because it is too large Load Diff

View File

@@ -36,9 +36,6 @@
;; Definitely thread safe
metabase.test.util.dynamic-redefs/patch-vars!
;; test-only cleanup of in-memory throttle state (atom resets)
metabase.session.api/reset-throttlers-for-testing!
;; TODO: most of these symbols shouldn't be here, we should go through them and
;; find the functions/macros that use them and make sure their names end with !
;; best way to do this is try remove each of these and rely on kondo output to find places where it's used

View File

@@ -6,8 +6,3 @@
`(def ~(-> sym name symbol) "docstring" ~sym))
([sym name]
`(def ~name "docstring" ~sym)))
(defmacro import-macro
"Kondo macro replacement for [[metabase.util.namespaces/import-macro]]."
[sym]
`(def ~(-> sym name symbol) "docstring" ~sym))

View File

@@ -3,8 +3,3 @@
(defmacro with-temp-dbs! [bindings & body]
`(let [~@(interleave bindings (repeat nil))]
~@body))
(defmacro with-routing-setup! [[router-binding destinations] & body]
`(let [~router-binding nil
~@(interleave (map first destinations) (repeat nil))]
~@body))

View File

@@ -1,47 +0,0 @@
;; Per-linter budgets for inline `:clj-kondo/ignore` forms.
;; metabase.core.kondo-ratchet-test fails when the budgets drift from the actual counts.
;; `./bin/mage fix-kondo-ratchets` lowers budgets to match the tree; local test runs do it
;; automatically. Raising a budget, or adding one for a new linter (`--seed`), is a hand edit
;; to defend in your PR.
;; :all is the vector-less ignore form, which suppresses every linter on the next form.
{:ignore-counts {:aliased-namespace-symbol 3
:all 48
:case-symbol-test 1
:clojure-lsp/unused-public-var 9
:def-fn 1
:deprecated-namespace 95
:deprecated-var 84
:discouraged-java-method 3
:discouraged-namespace 81
:discouraged-var 111
:equals-true 1
:main-without-gen-class 1
:metabase/defsetting-namespace 2
:metabase/disallow-hardcoded-driver-names-in-tests 34
:metabase/discourage-millis-duration 1
:metabase/i-like-making-cams-eyes-bleed-with-horrifically-long-tests 13
:metabase/modules 7
:metabase/namespace-name 4
:metabase/prefer-with-dynamic-fn-redefs 28
:metabase/test-helpers-use-non-thread-safe-functions 17
:metabase/validate-defendpoint-has-response-schema 440
:metabase/validate-defendpoint-query-params-use-kebab-case 50
:metabase/validate-defendpoint-route-uses-kebab-case 44
:metabase/validate-deftest 10
:missing-docstring 10
:missing-protocol-method 3
:non-arg-vec-return-type-hint 1
:reduce-without-init 3
:syntax 2
:uninitialized-var 1
:unquote-not-syntax-quoted 1
:unresolved-namespace 3
:unresolved-require 1
:unresolved-symbol 4
:unsorted-required-namespaces 1
:unused-alias 1
:unused-binding 9
:unused-excluded-var 2
:unused-namespace 1
:unused-private-var 9
:unused-value 1}}

View File

@@ -71,24 +71,6 @@
(some-> (ns-form-node->ns-symb-node ns-form-node)
hooks/sexpr))
(defn- lint-test-parallel-metadata [ns-form-node]
(when-let [ns-symb-node (ns-form-node->ns-symb-node ns-form-node)]
(let [ns-metadata (meta (hooks/sexpr ns-symb-node))]
(cond
(:synchronous ns-metadata)
(hooks/reg-finding!
(assoc (meta ns-symb-node)
:message (str "Use `^:synchronized` to mark this namespace explicitly non-parallel; "
"`^:synchronous` is ignored by Hawk. [:metabase/validate-deftest]")
:type :metabase/validate-deftest))
(and (contains? ns-metadata :parallel)
(false? (:parallel ns-metadata)))
(hooks/reg-finding!
(assoc (meta ns-symb-node)
:message "Use `^:synchronized` instead of `^{:parallel false}`. [:metabase/validate-deftest]"
:type :metabase/validate-deftest))))))
(defn- lint-modules [ns-form-node config]
(let [ns-symb (ns-form-node->ns-symb ns-form-node)]
(when-not (modules/ignored-namespace? config ns-symb)
@@ -158,7 +140,6 @@
lint-requires-on-new-lines
(lint-modules (modules/config x))
lint-namespace-name
lint-test-parallel-metadata
lint-jsqlparser-imports)
x)

View File

@@ -21,19 +21,15 @@
(error! form (format "destructive functions like %s are not allowed inside a ^:parallel test or test fixture. If this should be allowed, add it to the whitelist in the Kondo config file [:metabase/validate-deftest]"
qualified-symbol)))))
(walk [form]
(when-not (contains? (hooks.common/ignored-linters form) :metabase/validate-deftest)
(f form)
(doseq [child (:children form)]
(walk child))))]
(f form)
(doseq [child (:children form)]
(walk child)))]
(walk form)))
(defn- deftest-check-parallel
"1. Check if test is marked `^:parallel` / `^:synchronized` correctly.
2. Make sure disallowed forms are not used in ^:parallel tests.
3. Disallow `^:parallel` tests in namespaces marked `^:synchronized` --
the ns marker is the author's blanket promise that nothing inside runs
in parallel, so an individual `^:parallel` deftest contradicts it."
[{[_ test-name & body] :children, :as _node} ns-sym config]
"1. Check if test is marked ^:parallel / ^:synchronized correctly
2. Make sure disallowed forms are not used in ^:parallel tests"
[{[_ test-name & body] :children, :as _node} config]
(let [test-metadata (:meta test-name)
metadata-sexprs (map hooks/sexpr test-metadata)
combined-metadata (transduce
@@ -44,48 +40,19 @@
(completing merge)
{}
metadata-sexprs)
parallel? (true? (:parallel combined-metadata))
synchronized? (true? (:synchronized combined-metadata))
explicitly-nonparallel? (or synchronized?
(and (contains? combined-metadata :parallel)
(false? (:parallel combined-metadata))))
ns-metadata (meta ns-sym)
ns-explicitly-nonparallel? (or (true? (:synchronized ns-metadata))
(and (contains? ns-metadata :parallel)
(false? (:parallel ns-metadata))))]
(when (and (contains? combined-metadata :parallel)
(false? (:parallel combined-metadata)))
(hooks/reg-finding! (assoc (meta test-name)
:message (str "Use `^:synchronized` instead of `^{:parallel false}`. "
"[:metabase/validate-deftest]")
:type :metabase/validate-deftest)))
(when (:sequential combined-metadata)
(hooks/reg-finding! (assoc (meta test-name)
:message (str "Use `^:synchronized` to mark this test explicitly non-parallel; "
"`^:sequential` is ignored by Hawk. "
"[:metabase/validate-deftest]")
:type :metabase/validate-deftest)))
parallel? (:parallel combined-metadata)
synchronized? (:synchronized combined-metadata)]
(when (and parallel? synchronized?)
(hooks/reg-finding! (assoc (meta test-name)
:message "Test should not be marked both `^:parallel` and `^:synchronized`"
:message "Test should not be marked both ^:parallel and ^:synchronized"
:type :metabase/validate-deftest)))
(when (and parallel? ns-explicitly-nonparallel?)
;; only when the custom `:metabase/deftest-not-marked-parallel-or-synchronized` is enabled: complain if tests are
;; not explicitly marked `^:parallel` or `^:synchronized`. This is mostly to encourage people to mark everything
;; `^:parallel` in places like `metabase.lib` tests unless there is a really good reason not to.
(when-not (or parallel? synchronized?)
(hooks/reg-finding!
(assoc (meta test-name)
:message (str "Test should not be marked `^:parallel` in a namespace marked "
"`^:synchronized` -- "
"the ns marker promises no parallel execution. "
"Drop `^:parallel` from the deftest, or drop `^:synchronized` from the ns. "
"[:metabase/validate-deftest]")
:type :metabase/validate-deftest)))
;; Only when the custom `:metabase/deftest-not-marked-parallel-or-synchronized` is enabled:
;; complain if tests are not explicitly marked `^:parallel` or `^:synchronized`. This is
;; mostly to encourage people to mark everything `^:parallel` in places like `metabase.lib`
;; tests unless there is a really good reason not to.
(when-not (or parallel? explicitly-nonparallel?)
(hooks/reg-finding!
(assoc (meta test-name)
:message "Test should be marked either `^:parallel` or `^:synchronized`"
:message "Test should be marked either ^:parallel or ^:synchronized"
:type :metabase/deftest-not-marked-parallel-or-synchronized)))
(when parallel?
(doseq [form body]
@@ -104,36 +71,6 @@
"Split it up into smaller tests! 🥰")
:type :metabase/i-like-making-cams-eyes-bleed-with-horrifically-long-tests)))))))
(defn- every-top-level-form-in-deftest-is-testing? [{[_deftest _test-name & forms] :children, :as _node}]
(every? (fn [{[first-child] :children, :as form}]
(and (hooks/list-node? form)
(= (hooks.common/node->qualified-symbol first-child)
'clojure.test/testing)))
forms))
(defn- num-top-level-forms-in-deftest [{[_deftest _test-name & forms] :children, :as _node}]
(count forms))
(defn- deftest-check-should-not-be-multiple-separate-tests
"Warn on long tests (tests over `min-length`) that look like
(deftest my-test
(testing ...)
(testing ...)
(testing ...))"
[node {:keys [min-length], :as _config}]
(let [{:keys [row end-row]} (meta node)
test-length (- end-row row)]
(when (and min-length
(> test-length min-length)
(every-top-level-form-in-deftest-is-testing? node)
(> (num-top-level-forms-in-deftest node) 1))
(hooks/reg-finding!
(assoc (meta node)
:message (str "This test looks like it contains several logically separate testing forms... break it"
" out into separate deftests to make it easier to test and debug")
:type :metabase/validate-deftest-logically-separate-tests)))))
(defn- ignore? [node error-type]
(contains? (hooks.common/ignored-linters node) error-type))
@@ -173,8 +110,6 @@
"build."
"i18n." ; bin/i18n
"lint-migrations-file-test"
"load-namespaces." ; bin/load-namespaces
"mage."
"main-test" ; bin/release-list
"metabase.deps-edn-test"
"metabase.driver."
@@ -200,14 +135,13 @@
:message (format "All tests must live in a known module; %s is not a known module" (pr-str current-module))
:type :metabase/tests-must-live-in-known-modules)))))))
(defn deftest [{:keys [node cljc lang config], ns-sym :ns, :as input}]
(defn deftest [{:keys [node cljc lang config], :as input}]
;; run [[deftest-check-parallel]] only once... if this is a `.cljc` file only run it for the `:clj` analysis, no point
;; in running it twice.
(when (or (not cljc)
(= lang :clj))
(deftest-check-parallel node ns-sym (get-in config [:linters :metabase/validate-deftest])))
(deftest-check-parallel node (get-in config [:linters :metabase/validate-deftest])))
(deftest-check-not-horrifically-long node (get-in config [:linters :metabase/i-like-making-cams-eyes-bleed-with-horrifically-long-tests]))
(deftest-check-should-not-be-multiple-separate-tests node (get-in config [:linters :metabase/validate-deftest-logically-separate-tests]))
(deftest-check-no-driver-keywords node (get-in config [:linters :metabase/disallow-hardcoded-driver-names-in-tests]))
(deftest-check-in-valid-module input)
input)
@@ -246,40 +180,11 @@
nil)))))))
(defn is [{:keys [node lang], :as input}]
(defn is [{:keys [node lang]}]
(when (= lang :cljs)
(warn-about-missing-test-expr-requires-in-cljs node))
input)
{:node node})
(defn use-fixtures
"Flag `:parallel/unsafe` forms inside fixtures. Skipped when the surrounding
namespace is marked `^:synchronized` -- the ns marker is the author's
explicit opt-in to a single-threaded test ns where destructive setup is
safe. Without that opt-in, kondo can't know whether sibling deftests will
be `^:parallel`, so the conservative default is to flag.
Sister to [[deftest-check-parallel]]: the deftest hook rejects `^:parallel`
tests inside a `^:synchronized` ns, so the two checks together form a
coherent opt-in: either the whole ns is explicitly non-parallel (fixtures can be
anything; no parallel tests allowed) or it isn't (fixtures must be
parallel-safe)."
[{:keys [node config], ns-sym :ns, :as input}]
(when-not (or (true? (:synchronized (meta ns-sym)))
(and (contains? (meta ns-sym) :parallel)
(false? (:parallel (meta ns-sym)))))
(let [linter-config (get-in config [:linters :metabase/validate-deftest])]
(doseq [form (rest (:children node))]
(warn-about-disallowed-parallel-forms form linter-config))))
input)
(defn testing
"Check that we don't have an empty `testing` form like
(testing \"message\")"
[{{[_testing _message & body] :children, :as node} :node, :as input}]
(when (empty? body)
(hooks/reg-finding!
(assoc (meta node)
:message "A `testing` form that doesn't wrap anything doesn't do anything"
:type :metabase/check-testing-not-empty)))
input)
(defn use-fixtures [{:keys [node config]}]
(warn-about-disallowed-parallel-forms node config)
{:node node})

View File

@@ -500,8 +500,8 @@
(and resolved (:ns resolved))
(symbol (name (:ns resolved)) (name (:name resolved)))
;; if it wasn't resolved but is still qualified it's probably using the full namespace name rather than an
;; alias.
;; if it wasn't resolved but is still qualified it's probably using the full namespace name rather than an
;; alias.
(qualified-symbol? sexpr)
sexpr)))))
;; some symbols like `*count/Integer` aren't resolvable.

View File

@@ -71,7 +71,7 @@
(let [module (module ns-symb)
module-api-namespaces (module-api-namespaces config module)
module-friends (module-friends config module)]
(or (nil? module-api-namespaces)
(or (empty? module-api-namespaces)
(contains? module-api-namespaces ns-symb)
(contains? module-friends current-module))))

View File

@@ -95,8 +95,6 @@
metabot-feedback-url
metabot-get-prompt-templates-url
metabot-prompt-generator-token-limit
metaplow-tracking-enabled
metaplow-url
multi-setting-read-only
notebook-native-preview-sidebar-width
notification-link-base-url

View File

@@ -0,0 +1,14 @@
(ns hooks.metabase.test.gentest
(:require
[clj-kondo.hooks-api :as hooks]))
(defn iterate*
"(iterate {:as limit-spec} [s1 val1 & more] body-form body-form...)"
[{{:keys [children] :as node} :node :as orig}]
(let [[token limit-spec bindings & body-nodes] children
node* (-> (hooks/list-node
(list token limit-spec
(hooks/list-node (list* (hooks/token-node 'let) bindings
body-nodes))))
(with-meta (meta node)))]
(assoc orig :node node*)))

View File

@@ -28,8 +28,8 @@
(mapcat (fn [[setting-name v]]
(concat
[(with-meta (hooks/token-node '_) (meta setting-name)) v]
;; if the setting name is namespace-qualified add a `_`
;; entry for it too.
;; if the setting name is namespace-qualified add a `_`
;; entry for it too.
(when (namespaced-symbol-node? setting-name)
[(with-meta (hooks/token-node '_) (meta setting-name)) setting-name]))))
(partition 2 (:children bindings))))

View File

@@ -1,6 +1,7 @@
(ns hooks.metabase.util.i18n
(:require
[clj-kondo.hooks-api :as api]
[clojure.string :as str]
[hooks.common]))
(defn- valid-apostrophes?
@@ -12,23 +13,23 @@
(let [c (first chars)]
(if (= c \')
(cond
;; Doubled quotes: skip both
;; Doubled quotes: skip both
(and (next chars) (= (second chars) \'))
(recur (nthrest chars 2))
;; Placeholder escape. We allow for intentional escaping, like either '{0}` or '{{`, but not 'arbitrary
;; strings' So basically, just require the *next* character to be a `{`, find the next single-quote, and
;; continue on from there.
;;
;; Technically a string like "foo bar's happy but i'm not" is totally fine, but we'll say it's invalid
;; because... that inner string literal is probably not what we actually want. So since we've never done
;; this intentionally as far as I can tell, let's just remove the possibility to avoid ambiguity.
;; Placeholder escape. We allow for intentional escaping, like either '{0}` or '{{`, but not 'arbitrary
;; strings' So basically, just require the *next* character to be a `{`, find the next single-quote, and
;; continue on from there.
;;
;; Technically a string like "foo bar's happy but i'm not" is totally fine, but we'll say it's invalid
;; because... that inner string literal is probably not what we actually want. So since we've never done
;; this intentionally as far as I can tell, let's just remove the possibility to avoid ambiguity.
(= (second chars) \{)
(let [at-closing (drop-while #(not= \' %) (next chars))]
(if (seq at-closing)
(recur (next at-closing))
false))
;; Unpaired single quote: invalid
;; Unpaired single quote: invalid
:else
false)
(recur (rest chars)))))))

View File

@@ -1,132 +0,0 @@
(ns hooks.metabase.util.match
(:require [clj-kondo.hooks-api :as api]))
(defn- token-node? [node token-name]
(and (api/token-node? node)
(= (str (:value node)) token-name)))
(defn- keyword-node? [node k]
(and (api/keyword-node? node)
(= (:k node) k)))
(defn- binding-symbol?
"True when the token is a plain symbol that should be treated as a binding
(not a keyword, not a boolean literal, not nil, not a number, not `_`)."
[node]
(and (api/token-node? node)
(let [v (:value node)]
(and (symbol? v)
(not (contains? '#{_ nil true false} v))))))
;; To establish fake let bindings, we need to bind them to something. This "something" has to be vague enough to trick
;; kondo into not looking too much into it. E.g., if we bind everything to `nil`, Kondo linters will get fired up.
(def opaque-node (api/list-node (list (api/token-node 'rand-nth) (api/list-node []))))
(defn collect-bindings
"Return a list of token nodes (symbols) introduced as bindings by `pattern`."
[pattern]
(cond
(binding-symbol? pattern)
[[pattern opaque-node]]
;; Lists: (:or ...), (:and ...), (sym :guard pred ...), or a quoted symbol.
(api/list-node? pattern)
(let [[head & rst] (:children pattern)]
(cond (or (keyword-node? head :or)
(keyword-node? head :and))
(mapcat collect-bindings rst)
(api/token-node? head)
(cond-> []
(binding-symbol? head) (conj [head opaque-node])
(keyword-node? (first rst) :guard) (conj [(api/token-node '_) (second rst)]))
;; anything else (e.g. a function call used as predicate) — no bindings
:else []))
(api/vector-node? pattern)
(mapcat (fn [node]
(when-not (token-node? node "&")
(collect-bindings node)))
(:children pattern))
;; Maps: keys are literal selectors, values are sub-patterns.
;; Special value-symbols &truthy and _ are NOT user bindings.
(api/map-node? pattern)
(let [children (:children pattern)
pairs (partition 2 children)]
(mapcat (fn [[_k v]]
(when-not (or (token-node? v "&truthy")
(token-node? v "_"))
(collect-bindings v)))
pairs))
;; For sets, make sure to use them because they may reference some of the outer local bindings, so if we don't
;; include them, Kondo will complain about those bindings being unused.
(api/set-node? pattern)
[[(api/token-node '_) pattern]]
;; Anything else — no bindings.
:else []))
(defn- dedupe-bindings
"Prevent Kondo complaining that we have multiple fake bindings."
[pairs]
(loop [[[sym v] & r] pairs, seen #{}, result []]
(cond (nil? sym) result
(and (seen sym) (not (token-node? sym "_"))) (recur r seen result)
:else (recur r (conj seen sym) (conj result sym v)))))
(defn match-one
"clj-kondo :analyze-call hook for match-one.
Rewrites
(match-one value
pat1 ret1
pat2 ret2
...)
into a `let` form that declares all binding symbols introduced by the patterns, so clj-kondo knows they are
legitimate and won't warn about unresolved/unused symbols:
(let [<all-bindings> (rand-nth []) ...]
(or ret1 ret2 ...))"
[{:keys [node]}]
(let [[_ value & clauses] (:children node)]
(when (odd? (count clauses))
(api/reg-finding!
{:message "match-one/match-many/replace requires an even number of pattern/result pairs"
:type :metabase/match-odd-clauses
:row (:row node)
:col (:col node)
:filename (-> node meta :filename)}))
(let [pairs (partition 2 clauses)
all-bindings (->> pairs
(mapcat (fn [[pat _ret]] (collect-bindings pat)))
dedupe-bindings)
special-bindings [(api/token-node '_) value
(api/token-node '&match) opaque-node
(api/token-node '&parents) opaque-node
(api/token-node '&recur) opaque-node
;; Pretend to use &match, &parents, and &recur so that Kondo doesn't complain about them
;; if they are not used by the user code.
(api/token-node '_) (api/token-node '&match)
(api/token-node '_) (api/token-node '&parents)
(api/token-node '_) (api/token-node '&recur)]
binding-nodes (into special-bindings all-bindings)
result-exprs (map second pairs)
new-node (api/list-node
(list
(api/token-node 'let)
(api/vector-node binding-nodes)
(api/list-node
(concat [(api/token-node 'clojure.core/or)] result-exprs [opaque-node]))))]
{:node new-node})))
(def match-many match-one)
(def replace match-one)
(defn replace-in [context]
;; Everything is the same as in match-one, just drop the middle argument.
(match-one (update-in context [:node :children] (fn [[sym value path & clauses]] (list* sym value clauses)))))

View File

@@ -19,19 +19,6 @@
(defn- lint-modules [form config]
(lint-ns form (assoc-in config [:linters :metabase/modules :level] :warning)))
(deftest ^:parallel explicit-nonparallel-namespace-metadata-test
(let [config {:linters {:metabase/validate-deftest {:level :warning}}}]
(testing "Hawk's supported namespace marker passes"
(is (empty? (lint-ns '(ns ^:synchronized example.test) config))))
(testing "the verbose Hawk spelling is not accepted as Metabase style"
(is (=? [{:type :metabase/validate-deftest
:message #(re-find #"Use `\^:synchronized` instead" %)}]
(lint-ns '(ns ^{:parallel false} example.test) config))))
(testing "the legacy marker explains that Hawk ignores it"
(is (=? [{:type :metabase/validate-deftest
:message #(re-find #"`\^:synchronous` is ignored by Hawk" %)}]
(lint-ns '(ns ^:synchronous example.test) config))))))
(deftest ^:parallel module-checker-allowed-modules-test
(are [task-namespace] (=? [{:message "Module task should not be used in the api module. [:metabase/modules api :uses]"
:type :metabase/modules}]
@@ -56,16 +43,6 @@
'{:metabase/modules {api {:uses #{search}}
search {:api #{metabase.search.core}}}}))))
(deftest ^:parallel module-checker-empty-api-test
(is (=? [{:message "Namespace metabase.embeddings.core is not an allowed external API namespace for the embeddings module. [:metabase/modules embeddings :api]"
:type :metabase/modules}]
(lint-modules
'(ns metabase.metabot.core
(:require
[metabase.embeddings.core :as embeddings]))
'{:metabase/modules {metabot {:uses #{embeddings}}
embeddings {:api #{}}}}))))
(deftest ^:parallel module-checker-friends-test
(is (= []
(lint-modules

View File

@@ -21,54 +21,6 @@
#{:athena}}}}})
(mapv :message @(:findings clj-kondo.impl.utils/*ctx*))))
(defn- parallel-metadata-findings
([form]
(parallel-metadata-findings form 'example.test))
([form ns-sym]
(let [linters {:metabase/deftest-not-marked-parallel-or-synchronized {:level :warning}
:metabase/validate-deftest {:level :warning}}]
(binding [clj-kondo.impl.utils/*ctx* {:config {:linters linters}
:ignores (atom nil)
:findings (atom [])
:namespaces (atom {})}]
(hooks.clojure.test/deftest {:node (api/parse-string form)
:lang :clj
:ns ns-sym
:config {:linters {:metabase/validate-deftest {}}}})
@(:findings clj-kondo.impl.utils/*ctx*)))))
(deftest ^:parallel explicit-nonparallel-metadata-test
(testing "Hawk's supported test markers satisfy the explicit metadata linter"
(are [metadata] (empty? (parallel-metadata-findings
(format "(deftest %s example-test (is true))" metadata)))
"^:parallel"
"^:synchronized"))
(testing "an unmarked test still gets the opt-in linter's guidance"
(is (=? [{:type :metabase/deftest-not-marked-parallel-or-synchronized
:message "Test should be marked either `^:parallel` or `^:synchronized`"}]
(parallel-metadata-findings "(deftest example-test (is true))"))))
(testing "the unsupported marker explains that Hawk ignores it"
(is (some #(and (= (:type %) :metabase/validate-deftest)
(re-find #"ignored by Hawk" (:message %)))
(parallel-metadata-findings
"(deftest ^:sequential example-test (is true))"))))
(testing "the verbose Hawk spelling is not accepted as Metabase style"
(is (some #(and (= (:type %) :metabase/validate-deftest)
(re-find #"Use `\^:synchronized` instead" (:message %)))
(parallel-metadata-findings
"(deftest ^{:parallel false} example-test (is true))"))))
(testing "a test cannot request both execution modes"
(is (some #(and (= (:type %) :metabase/validate-deftest)
(re-find #"both `\^:parallel` and `\^:synchronized`" (:message %)))
(parallel-metadata-findings
"(deftest ^:parallel ^:synchronized example-test (is true))"))))
(testing "a namespace-level non-parallel promise rejects parallel tests"
(is (=? [{:type :metabase/validate-deftest
:message #(re-find #"namespace marked.*`\^:synchronized`" %)}]
(parallel-metadata-findings
"(deftest ^:parallel example-test (is true))"
(with-meta 'example.test {:synchronized true}))))))
(deftest ^:parallel disallow-hardcoded-driver-names-in-tests-test
(is (= []
(deftest-warnings

View File

@@ -1,59 +0,0 @@
(ns hooks.metabase.util.match-test
(:require
[clj-kondo.hooks-api :as hooks]
[clj-kondo.impl.utils]
[clojure.test :refer :all]
[hooks.metabase.util.match :as sut]))
(defn- expand [src]
(-> (sut/match-one {:node (hooks/parse-string (pr-str src))})
:node
hooks/sexpr))
(defn- lint [src]
(binding [clj-kondo.impl.utils/*ctx* {:config {:linters {:metabase/match-odd-clauses {:level :warning}}}
:ignores (atom nil)
:findings (atom [])
:namespaces (atom {})}]
(expand src)
@(:findings clj-kondo.impl.utils/*ctx*)))
(deftest odd-clauses-triggers-warning-test
(is (=? [{:type :metabase/match-odd-clauses}]
(lint '(metabase.util.match/match-one [] [a b c])))))
(deftest expansion-test
(testing "simple expansion example"
(is (= '(let [_ []
&match (rand-nth ())
&parents (rand-nth ())
&recur (rand-nth ())
_ &match
_ &parents
_ &recur
a (rand-nth ())
b (rand-nth ())
c (rand-nth ())]
(clojure.core/or (+ a b) (rand-nth ())))
(expand '(match-one [] [a b c] (+ a b))))))
(testing "complicated expansion example that uses most match-one macro features, and also binding deduplication by the rewriter"
(is (= '(let [_ []
&match (rand-nth ())
&parents (rand-nth ())
&recur (rand-nth ())
_ &match
_ &parents
_ &recur
val1 (rand-nth ())
a (rand-nth ())
_ #{maybe important}
rst (rand-nth ())
x (rand-nth ())
_ (= x 3)
b (rand-nth ())]
(clojure.core/or (+ a val1) (rand-nth ())))
(expand '(match-one []
(:or [{:key1 val1, :key2 &truthy, :key3 _} _ a #{maybe important} & rst]
(:and (x :guard (= x 3))
[a b]))
(+ a val1)))))))

View File

@@ -1,27 +0,0 @@
{:aliases
{;; defendpoint shape: (api.macros/defendpoint METHOD URL DOCSTRING? [args] body)
;; No name slot — synthesize :route [METHOD "path"] from first 2 args.
"defendpoint"
{:fields {:method (fn [z] (some-> z z/down z/right z/sexpr name str/upper-case))
:path (fn [z] (-> z z/down z/right z/right z/sexpr))}}
;; defenterprise shape:
;; (defenterprise NAME DOCSTRING? EE-NAMESPACE [args] body)
"defenterprise"
{:fields {:name ->defn-name
:ee-namespace (fn [z]
;; Skip name + optional docstring; the next
;; symbol is the EE namespace.
(let [name-z (-> z z/down z/right)
after-doc (z/right name-z)
candidate (if (string? (try (z/sexpr after-doc)
(catch Exception _ nil)))
(z/right after-doc)
after-doc)]
(try (let [v (z/sexpr candidate)]
(when (symbol? v) v))
(catch Exception _ nil))))
:arglist ->defn-arg-list}}
"defsetting"
{:fields {:name ->defn-name}}}}

View File

@@ -1,10 +1,12 @@
;; Cljfmt config. See https://github.com/weavejester/cljfmt#formatting-options
{:function-arguments-indentation :community
:indent-line-comments? true
:normalize-newlines-at-file-end? true
:parallel? true
:remove-blank-lines-in-forms? true
:sort-ns-references? true
{:sort-ns-references?
true
:function-arguments-indentation
:community
:parallel?
true
:paths
["src"
@@ -12,65 +14,12 @@
"enterprise/backend"
"bin/build"
"bin/lint-migrations-file"
"bin/load-namespaces"
"bin/release-list"
#_"mage/src"
#_"mage/test"
"modules/drivers"
".clj-kondo/src"
".clj-kondo/test"
#_"dev"]
;; a map of symbols that tell cljfmt which forms are allowed to have blank lines inside of them. The value may be
;; either `:all`, which means blank lines are allowed between all elements in the form, e.g. `{cond :all}` to allow
;; blank lines between any of the elements inside cond; or it may be a set of element indexes that are allowed to
;; have blank lines, e.g. `{let #{0}}`, to allow blank lines in the binding of a `let` form.
:extra-blank-line-forms
{are :all
case :all
cond-> :all
cond->> :all
condp :all
clojure.spec.alpha/or :all
clojure.test/are :all
metabase.driver-api.core/match-many :all
metabase.driver-api.core/match-one :all
metabase.driver-api.core/replace :all
metabase.driver-api.core/replace-in :all
metabase.lib.drill-thru.test-util/test-drill-variants-with-merged-args :all
metabase.lib.drill-thru.test-util/test-returns-drill :all
metabase.test/with-temp #{0}
metabase.util.match/match-many :all
metabase.util.match/match-one :all
metabase.util.match/replace :all
metabase.util.match/replace-in :all
;; TODO (Cam 2026-05-21) I'm not really sure about `defprotocol`, `deftype`, `defrecord`, `proxy`, `reify`, etc., I
;; think idiomatically these should probably not have blank lines but we can worry about that later
defprotocol :all
defrecord :all
deftype :all
extend :all
extend-protocol :all
potemkin.types/def-map-type :all
potemkin.types/defprotocol+ :all
potemkin.types/defrecord+ :all
potemkin.types/deftype+ :all
potemkin/def-map-type :all
potemkin/defprotocol+ :all
potemkin/defrecord+ :all
potemkin/deftype+ :all
proxy :all
reify :all
;; TODO (Cam 2026-05-21) -- save these for a follow-on PR
defmacro :all
defmethod :all
defn :all
defn- :all
fn :all
metabase.util.malli/defmethod :all
metabase.util.malli/defn :all
metabase.util.malli/defn- :all}
;; See https://github.com/weavejester/cljfmt/blob/master/cljfmt/resources/cljfmt/indents/clojure.clj for the default
;; definitions
;;
@@ -88,7 +37,7 @@
;; `:inner` indentation is like `:defn` indentation. `[[:inner 0]]` (default for `defn`) means all arguments at depth
;; zero get indented 2 spaces if not on the first line regardless of number of elements on the first line.
:extra-indents
{ ;; clojure.core stuff
{;; clojure.core stuff
fn* [[:inner 0]]
let* [[:block 1]]
with-meta [[:default]]

View File

@@ -1,28 +0,0 @@
# yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json
reviews:
high_level_summary: false
review_status: false
auto_review:
enabled: false
drafts: true
labels:
- "coderabbit"
path_filters:
- "!**/*.snap"
- "!**/*.po"
- "!**/bun.lock"
- "!**/package-lock.json"
tools:
eslint:
enabled: false
knowledge_base:
code_guidelines:
filePatterns:
- ".claude/skills/clojure-review/SKILL.md"
- ".claude/skills/typescript-review/SKILL.md"
- ".claude/skills/e2e-test-review/SKILL.md"
- ".claude/skills/docs-review/SKILL.md"

10
.config/wt.toml Normal file
View File

@@ -0,0 +1,10 @@
# worktrunk project config — https://worktrunk.dev
# Hooks run when switching, creating, merging, or removing worktrees.
# Setup tasks for newly-created worktrees. Run once, blocking, in the
# new worktree's cwd before post-start background tasks kick off.
[pre-start]
# Trust mise config so it loads the project's tool versions without prompting.
trust-mise = "mise trust"
# Install JS deps so husky git hooks get wired up in the new worktree.
bun-install = "bun install"

View File

@@ -1,32 +0,0 @@
// dependency-cruiser config for the actual frontend import graph (what's
// imported, vs the *allowed* edges in module-boundaries.mjs). Consumed by
// .github/scripts/create-test-plan.ts.
//
// bunx depcruise frontend/src enterprise/frontend/src \
// --config .dependency-cruiser.cjs --output-type json > dependency-graph.json
/** @type {import('dependency-cruiser').IConfiguration} */
module.exports = {
options: {
// Keep only edges between our own source files (drops node_modules etc.),
// which keeps the JSON small.
includeOnly: "^(frontend/src|enterprise/frontend/src)/",
doNotFollow: { path: "node_modules" },
// Resolve the `"*": ["./frontend/src/*", ...]` aliases so bare imports like
// `metabase/lib/foo` map back to a real file.
tsConfig: { fileName: "tsconfig.json" },
// Keep type-only imports: a type-only dependent still breaks on a type change.
tsPreCompilationDeps: true,
moduleSystems: ["es6", "cjs", "tsd"],
enhancedResolveOptions: {
exportsFields: ["exports"],
conditionNames: ["import", "require", "node", "default", "types"],
extensions: [".js", ".jsx", ".ts", ".tsx", ".json"],
},
},
};

View File

@@ -1,4 +1,4 @@
FROM mcr.microsoft.com/vscode/devcontainers/java:25
FROM mcr.microsoft.com/vscode/devcontainers/java:21
# Set up nodesource, install node, bun, fontconfig for static viz, rlwrap for dev ergonomics

View File

@@ -1,6 +1,5 @@
.git
.babel_cache
target
target/*
.circleci
.cpcache
.devcontainer
@@ -16,7 +15,6 @@ test_config/*
test_modules/*
test_resources/*
node_modules
**/node_modules
**metabase.jar
*.db

View File

@@ -77,6 +77,13 @@ MB_EMAIL_SMTP_PORT=1025
MB_EMAIL_SMTP_USERNAME="admin"
MB_EMAIL_SMTP_PASSWORD="admin"
# Message queue backend (optional). Default is `appdb`. Set to `redis` to route MQ traffic
# through a Redis server — start one with the `redis` docker-compose profile.
# MB_QUEUE_BACKEND=redis
# MB_MQ_REDIS_URI=redis://localhost:6379
# MB_MQ_REDIS_USERNAME=
# MB_MQ_REDIS_PASSWORD=
# 2.3 Feature Flags & Tokens
# ------------------------------------------------------------------------------

6
.gitattributes vendored
View File

@@ -7,9 +7,3 @@
#
# TODO (Bryan 12/12/25) -- reenable this when we can completely verify correctness
# resources/migrations/*_update_migrations.yaml merge=yaml-migrations
# Generated API documentation/specs — regenerated by the "Auto-update documentation for master"
# CI automation, not edited by hand. Mark generated so tools/humans don't review or diff them as source.
docs/api.json linguist-generated
docs/api.html linguist-generated
resources/openapi/openapi.json linguist-generated

25
.github/CODEOWNERS vendored
View File

@@ -10,30 +10,15 @@ src/metabase/query_processor @metabase/core-backend-querying-platform
src/metabase/graph @metabase/graphy
enterprise/backend/src/metabase_enterprise/dependencies @metabase/graphy
enterprise/frontend/src/metabase-enterprise/dependencies @metabase/graphy
enterprise/frontend/src/metabase-enterprise/monitor/dependency-diagnostics @metabase/graphy
# @metabase/embedding
frontend/src/metabase/embedding @metabase/embedding
enterprise/frontend/src/embedding-sdk-package @metabase/embedding
enterprise/frontend/src/embedding-sdk-bundle @metabase/embedding
enterprise/frontend/src/embedding-sdk-shared @metabase/embedding
frontend/src/metabase/embedding-sdk @metabase/embedding
frontend/src/embedding-sdk-bundle @metabase/embedding
frontend/src/embedding-sdk-shared @metabase/embedding
frontend/src/metabase/public/containers/PublicOrEmbeddedQuestion @metabase/embedding
frontend/src/metabase/public/containers/PublicOrEmbeddedDashboard @metabase/embedding
enterprise/frontend/src/embedding @metabase/embedding
enterprise/frontend/src/embedding-sdk-ee @metabase/embedding
enterprise/frontend/src/embedding-sdk-package @metabase/embedding
enterprise/frontend/src/metabase-enterprise/embedding @metabase/embedding
enterprise/frontend/src/metabase-enterprise/embedding_iframe_sdk @metabase/embedding
enterprise/frontend/src/metabase-enterprise/embedding_iframe_sdk_setup @metabase/embedding
enterprise/frontend/src/metabase-enterprise/embedding-sdk @metabase/embedding
e2e/test-component @metabase/embedding
e2e/test/scenarios/embedding @metabase/embedding
e2e/embedding-sdk-host-apps @metabase/embedding
# @metabase/tech-writers
docs @metabase/tech-writers
docs/developers-guide # exclude from tech-writer ownership
@@ -42,8 +27,6 @@ docs/api.json # auto-generated
docs/configuring-metabase/environment-variables.md # auto-generated
docs/configuring-metabase/config-template.md # auto-generated
docs/installation-and-operation/commands.md # auto-generated
docs/embedding/sdk/api # auto-generated
docs/embedding/eajs/snippets # auto-generated
# @metabase/devexp
.github @metabase/devexp
@@ -52,5 +35,5 @@ docs/embedding/eajs/snippets # auto-generated
src/metabase/analytics/prometheus.clj @metabase/cloud-ops
# @metabase/metabot-frontend
enterprise/frontend/src/metabase-enterprise/metabot @metabase/metabot-frontend
enterprise/frontend/src/metabase-enterprise/metabot @metabase/metabot-frontend
frontend/src/metabase/metabot @metabase/metabot-frontend

View File

@@ -1,5 +1,5 @@
> [!IMPORTANT]
> For those employed by Metabase: if you are fixing a P0 or P1 issue, add a `backport` label to this PR. No action is needed otherwise. Refer to the [Backporting](https://app.notion.com/p/metabase/Branching-Strategy-and-Backports-6eb577d5f61142aa960a626d6bbdfeb3?source=copy_link#e71f6e7de2f945d99fbb2d6b764e6f36) section in the Metabase Branching Strategy document for more details. If you're not employed by Metabase, this section does not apply to you, and the label will be taken care of by your reviewer.
> For those employed by Metabase: if you are merging into master, please add either a `backport` or a `no-backport` label to this PR. You will not be able to merge until you do this step. Refer to the section [Do I need to backport this PR?](https://www.notion.so/metabase/Metabase-Branching-Strategy-6eb577d5f61142aa960a626d6bbdfeb3?pvs=4#89f80d6f17714a0198aeb66c0efd1b71) in the Metabase Branching Strategy document for more details. If you're not employed by Metabase, this section does not apply to you, and the label will be taken care of by your reviewer.
> **Warning**
>

View File

@@ -1,13 +0,0 @@
self-hosted-runner:
# Custom runner labels that actionlint cannot know about. Everything else is
# selected through `vars.DEFAULT_RUNNER_KEY` and is not checked.
labels:
- arm-ubuntu-24-2core
paths:
# GitHub added parallel steps (`parallel`, `background`, `wait`, `wait-all`,
# `cancel`) in June 2026, and actionlint 1.7.12 does not know the schema yet:
# https://github.blog/changelog/2026-06-25-actions-steps-can-now-be-run-in-parallel/
.github/workflows/lint.yml:
ignore:
- 'step must run script with "run" section or run action with "uses" section'

View File

@@ -1,136 +0,0 @@
name: Bundle Size Check
description: |
Measures app + embedding SDK bundle sizes from the current and base-ref
uberjars (reusing the builds that already ran) plus their bundle-stats
artifacts, and flags gzipped size changes beyond the threshold for the main
app's initial bundle and the chunked SDK's reachable total. The base ref is
read back off the current uberjar so that both sides differ only by the PR.
Skips when the base-ref artifacts aren't available rather than rebuilding from
source (which would compare across pipelines).
inputs:
edition:
description: 'Metabase edition (ee or oss)'
required: true
threshold-percent:
description: 'Percent threshold for bundle size change'
required: true
outputs:
app_status:
description: "App initial bundle status: increased, decreased, or stable"
value: ${{ steps.compare.outputs.app_initial_status }}
app_size_change_percent:
description: "App initial bundle size change percent (positive = increase, negative = decrease)"
value: ${{ steps.compare.outputs.app_initial_size_change_percent }}
sdk_status:
description: "Chunked SDK bundle status: increased, decreased, or stable"
value: ${{ steps.compare.outputs.embedding_sdk_chunked_total_status }}
sdk_size_change_percent:
description: "Chunked SDK bundle size change percent (positive = increase, negative = decrease)"
value: ${{ steps.compare.outputs.embedding_sdk_chunked_total_size_change_percent }}
runs:
using: "composite"
steps:
- name: Download current uberjar
uses: ./.github/actions/fetch-artifact
with:
commit: ${{ github.event.pull_request.head.sha || github.sha }}
edition: ${{ inputs.edition }}
extract-path: temp/current-uberjar
- name: Resolve the base ref the current uberjar was built against
id: base-ref
shell: bash
env:
GH_TOKEN: ${{ github.token }}
run: |
set -uo pipefail
# The uberjar job checks out refs/pull/<n>/merge (the PR head already
# merged into the base branch tip) but names its artifact after head.sha,
# so the comparable base ref is that merge's first parent. Reading it back
# off the jar describes the artifact actually being measured. Using
# pull_request.base.sha instead compares against a stale snapshot of the
# base branch, reporting everything that landed in between as this PR's
# own bundle delta.
unzip -o -j temp/current-uberjar/metabase.jar version.properties -d temp/current-version >/dev/null
built=$(sed -n 's/^hash=//p' temp/current-version/version.properties)
base=$(gh api "/repos/${GITHUB_REPOSITORY}/commits/${built}" \
--jq 'if (.parents | length) == 2 then .parents[0].sha else empty end' 2>/dev/null) || base=""
echo "sha=${base}" >> "$GITHUB_OUTPUT"
echo "The current uberjar was built from ${built}; its base ref is ${base:-unknown}"
- name: Download base-ref uberjar
id: base-jar
if: steps.base-ref.outputs.sha != ''
continue-on-error: true
uses: ./.github/actions/fetch-artifact
with:
commit: ${{ steps.base-ref.outputs.sha }}
edition: ${{ inputs.edition }}
extract-path: temp/base-uberjar
- name: Measure and compare bundle sizes
id: compare
shell: bash
env:
GH_TOKEN: ${{ github.token }}
EDITION: ${{ inputs.edition }}
HEAD_SHA: ${{ github.event.pull_request.head.sha || github.sha }}
BASE_SHA: ${{ steps.base-ref.outputs.sha }}
THRESHOLD: ${{ inputs.threshold-percent }}
MEASURE_BUNDLES: app,embedding-sdk-legacy,embedding-sdk-chunked
# The gated bundles, as "<bundle>/<kind>". Each one's step outputs are
# named after its key (see outputPrefix in compare.js), which is what the
# action outputs above read.
GATES: app/initial,embedding-sdk-chunked/total
run: |
set -uo pipefail
skip() {
echo "::notice::Bundle-size check skipped: $1"
for gate in ${GATES//,/ }; do
prefix=$(printf '%s' "$gate" | tr -c 'a-zA-Z0-9' '_')
echo "${prefix}_status=stable" >> "$GITHUB_OUTPUT"
echo "${prefix}_size_change_percent=0" >> "$GITHUB_OUTPUT"
done
exit 0
}
# The base ref needs an already-built uberjar. Rebuilding it from source
# would compare across two build pipelines (the bug this check used to
# have), so skip instead when it — or its stats — isn't available.
[ -n "${BASE_SHA}" ] || skip "no comparable base ref for the current uberjar"
[ -f temp/base-uberjar/metabase.jar ] || skip "base-ref uberjar not found for ${BASE_SHA}"
extract_js() { # $1=jar $2=dest-root
mkdir -p "$2/resources"
unzip -o "$1" \
"frontend_client/app/dist/*" \
"frontend_client/app/embed.js" \
"frontend_client/app/embedding-sdk/*" \
-d "$2/resources" >/dev/null || true
}
fetch_stats() { # $1=sha $2=dest-dir
local name="metabase-bundle-stats-${EDITION}-$1"
local id
id=$(gh api "/repos/${GITHUB_REPOSITORY}/actions/artifacts?name=${name}" --jq '.artifacts[0].id // empty' 2>/dev/null)
[ -n "${id}" ] || return 1
mkdir -p "$2"
gh api "/repos/${GITHUB_REPOSITORY}/actions/artifacts/${id}/zip" > "$2.zip" 2>/dev/null || return 1
unzip -o "$2.zip" -d "$2" >/dev/null || return 1
}
extract_js temp/current-uberjar/metabase.jar temp/current
extract_js temp/base-uberjar/metabase.jar temp/base
fetch_stats "${HEAD_SHA}" temp/current/artifacts || skip "bundle stats not found for current ${HEAD_SHA}"
fetch_stats "${BASE_SHA}" temp/base/artifacts || skip "bundle stats not found for base ${BASE_SHA}"
MEASURE_ROOT=temp/current node ./.github/scripts/measure-bundle-sizes.js > temp/current-sizes.json \
|| { echo "::error::Failed to measure current bundle sizes"; exit 1; }
MEASURE_ROOT=temp/base node ./.github/scripts/measure-bundle-sizes.js > temp/base-sizes.json \
|| { echo "::error::Failed to measure base bundle sizes"; exit 1; }
node ./.github/actions/bundle-size-check/compare.js temp/current-sizes.json temp/base-sizes.json

View File

@@ -1,133 +0,0 @@
/* eslint-disable import/no-commonjs */
// Diffs the bundle sizes produced by measure-bundle-sizes.js for the current
// build and the base ref. Every (bundle, kind) is compared strictly against the
// same (bundle, kind) — never one bundle type against another — using gzipped
// (served) bytes.
//
// The report covers every measured bundle; the gates are the subset that drives
// PR comments, named as "<bundle>/<kind>" (e.g. "app/initial",
// "embedding-sdk-chunked/total").
//
// compareBundles is pure (data in, data out) so it can be unit-tested; main()
// is the thin I/O wrapper that reads the size files, prints the report and sets
// the step outputs / exit code.
const fs = require("fs");
const pick = (rows, bundle, kind) => rows.find(row => row.bundle === bundle && row.kind === kind);
const keyOf = row => `${row.bundle}/${row.kind}`;
const mb = bytes => (bytes / 1024 / 1024).toFixed(2);
const percentOf = (currentBytes, baseBytes) =>
baseBytes ? Math.trunc(((currentBytes - baseBytes) * 100) / baseBytes) : 0;
// Output names are derived from the gate key, so the workflow reads
// `app_initial_status`, `embedding_sdk_chunked_total_status`, and so on.
const outputPrefix = gateKey => gateKey.replace(/[^a-zA-Z0-9]+/g, "_");
/**
* Evaluate one gate. Returns the key plus exactly one of:
* - `status` / `percent`: the bundle moved by `percent`.
* - `skip` reason: the gate is non-actionable (reported stable, no comment).
* - `error` reason: the gate inputs are missing (hard failure).
*/
function evaluateGate({ current, base, threshold, gateKey }) {
const [bundle, kind] = gateKey.split("/");
const gateCurrent = pick(current, bundle, kind);
const gateBase = pick(base, bundle, kind);
if (!gateCurrent || !gateBase || !gateBase.gzipBytes) {
return { gateKey, error: `Could not find ${bundle} ${kind} (gzip) in both builds` };
}
// Both sides must measure the bundle the same way. A base ref built before the
// reachableAssets enrichment collapses its total to the initial set, so
// comparing it against the current reachable total reports a phantom jump
// (~30%). Skip until the base ref also carries reachable stats.
if (Boolean(gateCurrent.reachable) !== Boolean(gateBase.reachable)) {
return {
gateKey,
skip:
`"${gateKey}" measured differently on each side ` +
`(current reachable=${Boolean(gateCurrent.reachable)}, base reachable=${Boolean(gateBase.reachable)}); ` +
`the base ref predates the reachable-chunk stats. Resolves once this change is on the base ref.`,
};
}
const percent = percentOf(gateCurrent.gzipBytes, gateBase.gzipBytes);
const status = percent > threshold ? "increased" : percent < -threshold ? "decreased" : "stable";
return { gateKey, status, percent };
}
/**
* Compare two measurement sets and evaluate every gate. Returns a
* human-readable `report` over all measured bundles plus one entry per gate.
*/
function compareBundles({ current, base, threshold, gateKeys }) {
const report = ["=== Bundle sizes (gzipped, current vs base) ==="];
const keys = [...new Set([...current, ...base].map(keyOf))].sort();
for (const key of keys) {
const [bundle, kind] = key.split("/");
const currentRow = pick(current, bundle, kind);
const baseRow = pick(base, bundle, kind);
if (!currentRow || !baseRow) {
report.push(`${key}: present only in ${currentRow ? "current" : "base"} build`);
continue;
}
const diff = currentRow.gzipBytes - baseRow.gzipBytes;
const percent = percentOf(currentRow.gzipBytes, baseRow.gzipBytes);
report.push(`${key}: ${mb(currentRow.gzipBytes)}MB vs ${mb(baseRow.gzipBytes)}MB (${percent}%, ${diff} bytes)`);
}
return { report, gates: gateKeys.map(gateKey => evaluateGate({ current, base, threshold, gateKey })) };
}
module.exports = { compareBundles, outputPrefix };
const setOutput = (name, value) => {
if (process.env.GITHUB_OUTPUT) {
fs.appendFileSync(process.env.GITHUB_OUTPUT, `${name}=${value}\n`);
}
};
function main() {
const [, , currentPath, basePath] = process.argv;
const threshold = Number(process.env.THRESHOLD ?? 2);
const gateKeys = (process.env.GATES ?? "embedding-sdk-chunked/total").split(",").map(key => key.trim());
const current = JSON.parse(fs.readFileSync(currentPath, "utf8"));
const base = JSON.parse(fs.readFileSync(basePath, "utf8"));
const { report, gates } = compareBundles({ current, base, threshold, gateKeys });
console.log(report.join("\n"));
console.log("");
for (const gate of gates) {
const prefix = outputPrefix(gate.gateKey);
if (gate.error) {
// One broken gate must not hide the others, so report them all and fail
// the step at the end.
console.error(`::error::${gate.error}`);
continue;
}
if (gate.skip) {
// Treat the gate as non-actionable: report stable so no regression comment
// is posted, mirroring the shell skip() for missing artifacts.
console.log(`::notice::Bundle-size gate skipped: ${gate.skip}`);
setOutput(`${prefix}_status`, "stable");
setOutput(`${prefix}_size_change_percent`, "0");
continue;
}
console.log(`Gate: ${gate.gateKey} ${gate.percent}% (threshold ${threshold}%) → ${gate.status}`);
setOutput(`${prefix}_status`, gate.status);
setOutput(`${prefix}_size_change_percent`, gate.percent);
}
if (gates.some(gate => gate.error)) {
process.exit(1);
}
}
if (require.main === module) {
main();
}

View File

@@ -1,84 +0,0 @@
const { compareBundles, outputPrefix } = require("./compare");
const SDK_GATE = "embedding-sdk-chunked/total";
const APP_GATE = "app/initial";
const total = (gzipBytes, over = {}) => ({
bundle: "embedding-sdk-chunked",
kind: "total",
gzipBytes,
reachable: true,
...over,
});
const appInitial = gzipBytes => ({ bundle: "app", kind: "initial", gzipBytes });
const compare = ({ current, base, threshold = 2, gateKeys = [SDK_GATE] }) =>
compareBundles({ current, base, threshold, gateKeys });
describe("outputPrefix", () => {
it("turns a gate key into a step-output prefix", () => {
expect(outputPrefix(APP_GATE)).toBe("app_initial");
expect(outputPrefix(SDK_GATE)).toBe("embedding_sdk_chunked_total");
});
});
describe("compareBundles", () => {
it("gates on the chunked SDK total and flags an increase past the threshold", () => {
const { gates, report } = compare({ current: [total(110)], base: [total(100)] });
expect(gates).toEqual([{ gateKey: SDK_GATE, status: "increased", percent: 10 }]);
expect(report[0]).toContain("current vs base");
});
it("flags a decrease and stays stable within the threshold", () => {
expect(compare({ current: [total(90)], base: [total(100)] }).gates[0]).toMatchObject({
status: "decreased",
percent: -10,
});
expect(compare({ current: [total(101)], base: [total(100)] }).gates[0]).toMatchObject({
status: "stable",
percent: 1,
});
});
it("evaluates each gate against its own bundle", () => {
const { gates } = compare({
current: [total(100), appInitial(110)],
base: [total(100), appInitial(100)],
gateKeys: [APP_GATE, SDK_GATE],
});
expect(gates).toEqual([
{ gateKey: APP_GATE, status: "increased", percent: 10 },
{ gateKey: SDK_GATE, status: "stable", percent: 0 },
]);
});
it("skips when the two sides measured a gate differently (reachable mismatch)", () => {
const { gates } = compare({
current: [total(130, { reachable: true })],
base: [total(100, { reachable: false })],
});
expect(gates[0].status).toBeUndefined();
expect(gates[0].skip).toContain("predates the reachable-chunk stats");
});
it("errors when a gate bundle is missing on either side, without hiding the others", () => {
const { gates } = compare({
current: [appInitial(100)],
base: [appInitial(100)],
gateKeys: [APP_GATE, SDK_GATE],
});
expect(gates[0]).toEqual({ gateKey: APP_GATE, status: "stable", percent: 0 });
expect(gates[1].error).toContain("embedding-sdk-chunked total");
});
it("reports a bundle/kind present on only one side without erroring", () => {
const { report } = compare({ current: [total(100), appInitial(50)], base: [total(100)] });
expect(report.some(line => line.includes("present only in current build"))).toBe(true);
});
});

View File

@@ -26,7 +26,7 @@ runs:
- name: Upload python-runner logs
if: ${{inputs.upload-logs == 'true' && inputs.logs-artifact-suffix != ''}}
uses: actions/upload-artifact@v7
uses: actions/upload-artifact@v4
with:
name: python-runner-logs-${{inputs.logs-artifact-suffix}}
path: |

View File

@@ -1,61 +0,0 @@
name: Collect profile
description: >-
Attach a flamegraph and per-test wall times to a job whose tests ran under clj-async-profiler.
Does nothing when the run was not profiled.
inputs:
profiled:
required: true
description: >-
"true" when the test step ran under the profiler. Passed in rather than read from the
environment so the decision stays recorded by the step that made it.
artifact-name:
required: true
description: Name for the uploaded artifact. Must be unique across the jobs of one run.
runs:
using: "composite"
steps:
# Unlike a JUnit failure report, these run on a green build too — a suite that passes slowly is
# exactly what profiling is for.
#
# `awk 'NR<=25'` rather than `head -25`: the runner ignores SIGPIPE, so `head` exiting early
# leaves `sort` with an EPIPE write error and `pipefail` fails the step. `|| true` covers `grep`
# finding no junit files.
- name: Summarize test wall times
if: ${{ inputs.profiled == 'true' }}
continue-on-error: true
shell: bash
run: |
{
echo "### Slowest test namespaces (seconds)"
echo '```'
grep -h '<testsuite' target/junit/*.xml 2>/dev/null \
| sed -E 's|.*<testsuite name="([^"]*)" time="([^"]*)".*|\2 \1|' \
| sort -rn | awk 'NR<=25' || true
echo '```'
echo "### Slowest tests (seconds)"
echo '```'
grep -h '<testcase' target/junit/*.xml 2>/dev/null \
| sed -E 's|.*classname="([^"]*)" name="([^"]*)" time="([^"]*)".*|\3 \1/\2|' \
| sort -rn | awk 'NR<=25' || true
echo '```'
} >> "$GITHUB_STEP_SUMMARY"
# Only the rendered flamegraph is kept: the raw collapsed stacks it was built from run to
# hundreds of MB on a long suite, and are reproducible locally from the same test selection.
#
# The info-level test log rides along on a green profiled run too. The failure-only log artifact
# the test jobs upload is not written when nothing failed, which is exactly the run being profiled.
- name: Upload profile
if: ${{ inputs.profiled == 'true' }}
continue-on-error: true
uses: actions/upload-artifact@v7
with:
name: ${{ inputs.artifact-name }}
path: |
target/clj-async-profiler/results/*flamegraph.html
target/junit/*.xml
logs/test-log.json
retention-days: 14
if-no-files-found: warn

View File

@@ -1,5 +1,4 @@
name: Create Backport PR
description: Cherry-picks a commit onto a release branch and opens the backport pull request.
inputs:
target-version: # major version number like 34 or 86
@@ -43,7 +42,7 @@ runs:
git cherry-pick --abort
echo "Could not cherry pick because of a conflict"
echo "has-conflicts=true" >> "$GITHUB_OUTPUT"
echo "has-conflicts=true" >> $GITHUB_OUTPUT
# Add a shell script for resolving conflicts
echo "git reset HEAD~1" > ./backport.sh
@@ -58,48 +57,45 @@ runs:
#${ORIGINAL_PULL_REQUEST_NUMBER}
> [!IMPORTANT]
> Manual conflict resolution is required.
> Checkout the branch and run \`./backport.sh\` script. Force push your changes after cherry-picking. Mark the PR as ready for review once conflicts have been addressed.
> Checkout the branch and run \`./backport.sh\` script. Force push your changes after cherry-picking. Check the box below when done.
Conflicts:
\`\`\`shell
${CONFLICT_LIST}
\`\`\`
- [ ] Conflicts resolved
END
)
else
echo "has-conflicts=false" >> "$GITHUB_OUTPUT"
echo "has-conflicts=false" >> $GITHUB_OUTPUT
PR_BODY="#${ORIGINAL_PULL_REQUEST_NUMBER}"
fi
git push -u origin ${BACKPORT_BRANCH}
PR_ARGS=(--base "${TARGET_BRANCH}" --head "${BACKPORT_BRANCH}" --label "was-backported" --assignee "${{ github.event.pull_request.user.login }}" --title "🤖 v${TARGET_VERSION} backported \"${ORIGINAL_TITLE}\"" --body "${PR_BODY}")
if [ "$CONFLICTS" -gt 0 ]; then
PR_ARGS+=(--draft)
fi
BACKPORT_PR_URL=$(gh pr create "${PR_ARGS[@]}")
BACKPORT_PR_URL=$(gh pr create --base "${TARGET_BRANCH}" --head "${BACKPORT_BRANCH}" --label "was-backported" --assignee "${{ github.event.pull_request.user.login }}" --title "🤖 backported \"${ORIGINAL_TITLE}\"" --body "${PR_BODY}")
BACKPORT_PR_NUMBER=${BACKPORT_PR_URL##*/}
echo "backport_pr_number=$BACKPORT_PR_NUMBER" >> "$GITHUB_OUTPUT"
echo "backport_pr_number=$BACKPORT_PR_NUMBER" >> $GITHUB_OUTPUT
echo "New PR has been created: #${BACKPORT_PR_NUMBER}"
git checkout ${{ github.sha }}
- name: Auto approve backport PR
if: ${{ steps.create_backport_pull_request.outputs.has-conflicts == 'false' }}
uses: juliangruber/approve-pull-request-action@68fcc9a5a73b5641cadf757cf99d73720dcb05d0 # v2.1.0
uses: juliangruber/approve-pull-request-action@v1
with:
github-token: ${{ inputs.github-token-2 }}
number: ${{ steps.create_backport_pull_request.outputs.backport_pr_number }}
- name: Enable Pull Request Automerge
if: ${{ steps.create_backport_pull_request.outputs.has-conflicts == 'false' }}
run: gh pr merge --squash --auto "$PR_NUMBER"
shell: bash
env:
GH_TOKEN: ${{ inputs.github-token }}
PR_NUMBER: ${{ steps.create_backport_pull_request.outputs.backport_pr_number }}
uses: peter-evans/enable-pull-request-automerge@v2
with:
token: ${{ inputs.github-token }}
pull-request-number: ${{ steps.create_backport_pull_request.outputs.backport_pr_number }}
merge-method: squash
- uses: ./.github/actions/notify-pull-request
if: ${{ steps.create_backport_pull_request.outputs.has-conflicts == 'true' }}

View File

@@ -1,131 +0,0 @@
name: Create test plan
description: Cruise the frontend import graph and compute the affected-tests plan.
inputs:
changed-files:
description: All changed files (CSV).
required: true
fe-changed-files:
description: Changed frontend_sources files (CSV).
required: true
be-changed-files:
description: Changed backend_sources files (CSV).
required: true
unit-infra-touched:
description: Whether FE unit infra changed.
required: true
loki-infra-touched:
description: Whether Loki infra changed.
required: true
e2e-infra-touched:
description: Whether e2e harness/support code changed.
required: true
shared-sources-touched:
description: Whether cljc/cljs/package.json changed.
required: true
outputs:
stats:
description: FE Affected Tests stats row (JSON).
value: ${{ steps.compute.outputs.stats }}
fe_unit_specs_to_run:
description: Unit specs to run (JSON array).
value: ${{ steps.compute.outputs.fe_unit_specs_to_run }}
loki_stories_to_run:
description: Loki stories to run (JSON array).
value: ${{ steps.compute.outputs.loki_stories_to_run }}
e2e_specs_to_run:
description: E2E specs to run (JSON array).
value: ${{ steps.compute.outputs.e2e_specs_to_run }}
runs:
using: composite
steps:
- name: Prepare Node + Bun
uses: ./.github/actions/prepare-node-bun
- name: Resolve closest-ancestor e2e coverage manifest
id: manifest
uses: actions/github-script@v9
with:
script: | # js
const { pathToFileURL } = require("url");
const { resolveCoverageManifest } = await import(
pathToFileURL(
`${process.env.GITHUB_WORKSPACE}/.github/scripts/resolve-coverage-manifest.mjs`,
).href,
);
const result = await resolveCoverageManifest({
github,
owner: context.repo.owner,
repo: context.repo.repo,
headSha: context.payload.pull_request?.head?.sha ?? context.sha,
});
core.setOutput("path", result?.path ?? "");
- name: Build dependency graph
shell: bash
run: |
# Quiet on success; surface output only if install fails.
bun install --frozen-lockfile > /tmp/bun-install.log 2>&1 \
|| { echo "bun install issues (continuing):"; tail -20 /tmp/bun-install.log; }
bunx depcruise frontend/src enterprise/frontend/src \
--config .dependency-cruiser.cjs \
--output-type json > dependency-graph.json || true
- name: Create test plan
id: compute
shell: bash
env:
CHANGED_FILES: ${{ inputs.changed-files }}
FE_CHANGED_FILES: ${{ inputs.fe-changed-files }}
BE_CHANGED_FILES: ${{ inputs.be-changed-files }}
UNIT_INFRA_TOUCHED: ${{ inputs.unit-infra-touched }}
LOKI_INFRA_TOUCHED: ${{ inputs.loki-infra-touched }}
E2E_INFRA_TOUCHED: ${{ inputs.e2e-infra-touched }}
SHARED_SOURCES_TOUCHED: ${{ inputs.shared-sources-touched }}
DEP_GRAPH_JSON: dependency-graph.json
E2E_SPEC_MANIFEST: ${{ steps.manifest.outputs.path }}
run: |
# stdout is the plan JSON; the planner's own info logs go to stderr.
bun .github/scripts/create-test-plan.ts > test-plan.json
echo "stats=$(jq -c '.stats' test-plan.json)" >> "$GITHUB_OUTPUT"
echo "fe_unit_specs_to_run=$(jq -c '.fe_unit_specs_to_run' test-plan.json)" >> "$GITHUB_OUTPUT"
echo "loki_stories_to_run=$(jq -c '.loki_stories_to_run' test-plan.json)" >> "$GITHUB_OUTPUT"
echo "e2e_specs_to_run=$(jq -c '.e2e_specs_to_run' test-plan.json)" >> "$GITHUB_OUTPUT"
- name: Report test plan
shell: bash
run: |
plan=test-plan.json
row() { # name, selected, total
if [ "$2" -eq "$3" ]; then echo "| $1 | all ($3) | $3 |"; else echo "| $1 | $2 | $3 |"; fi
}
{
echo "## Affected test plan"
echo
echo "| Suite | Selected | Total |"
echo "|---|---:|---:|"
row "Unit" "$(jq '.stats.unit_specs_to_run_usage' "$plan")" "$(jq '.stats.unit_specs_all' "$plan")"
row "Loki" "$(jq '.stats.loki_stories_to_run_usage' "$plan")" "$(jq '.stats.loki_stories_all' "$plan")"
row "E2E" "$(jq '.stats.e2e_specs_to_run_usage' "$plan")" "$(jq '.stats.e2e_specs_all' "$plan")"
echo
echo "Changed: $(jq '.stats.fe_files_changed' "$plan") FE / $(jq '.stats.be_files_changed' "$plan") BE file(s) · $(jq '.stats.fe_modules_affected_usage' "$plan") FE module(s) affected"
echo
e2e_run=$(jq '.stats.e2e_specs_to_run_usage' "$plan")
e2e_all=$(jq '.stats.e2e_specs_all' "$plan")
if [ "$e2e_run" -eq "$e2e_all" ]; then
echo "**E2E: full run** — no manifest, or a change the coverage manifest can't scope (backend, e2e infra, cljc/cljs)."
else
echo "<details><summary><b>E2E specs to run ($e2e_run)</b></summary>"
echo
jq -r '.e2e_specs_to_run[]' "$plan" | sed 's/^/- /'
echo
echo "</details>"
fi
echo
echo "<details><summary>Full stats (rules vs usage graph)</summary>"
echo
echo '```json'
jq '.stats' "$plan"
echo '```'
echo "</details>"
} | tee -a "${GITHUB_STEP_SUMMARY:-/dev/null}"

View File

@@ -1,7 +1,4 @@
name: Cross-branch migration test
description: >-
Boot Metabase on the base branch to migrate the app db, then boot the PR branch against the
same db, verifying the PR's migrations apply cleanly on top of the base branch's schema.
inputs:
base-ref: ## Base branch PR came from
@@ -14,7 +11,7 @@ runs:
steps:
## First we deploy the base branch
- name: Check out base branch
uses: actions/checkout@v6
uses: actions/checkout@v4
with:
ref: ${{ inputs.base-ref }}
@@ -45,7 +42,7 @@ runs:
## Then we deploy the PR branch to ensure it migrates up successfully
- name: Check out pull request branch
uses: actions/checkout@v6
uses: actions/checkout@v4
with:
ref: ${{ inputs.head-ref }}
@@ -84,7 +81,7 @@ runs:
shell: bash
- name: Re-check out base branch
uses: actions/checkout@v6
uses: actions/checkout@v4
with:
ref: ${{ inputs.base-ref }}

View File

@@ -9,7 +9,7 @@ runs:
using: "composite"
steps:
- name: Download previously stored uberjar
uses: actions/github-script@v9
uses: actions/github-script@v7
with:
script: | # js
const fs = require('fs');

View File

@@ -40,24 +40,10 @@ runs:
using: "composite"
steps:
- name: Authenticate to prevent API rate limit
shell: bash
env:
DOCKER_USERNAME: ${{ inputs.username }}
DOCKER_PASSWORD: ${{ inputs.password }}
run: |
for attempt in 1 2 3; do
echo "Docker login attempt $attempt of 3..."
if echo "$DOCKER_PASSWORD" | docker login --username "$DOCKER_USERNAME" --password-stdin; then
echo "Docker login succeeded on attempt $attempt"
break
fi
if [ $attempt -eq 3 ]; then
echo "::error::Docker login failed after 3 attempts"
exit 1
fi
echo "::warning::Docker login attempt $attempt failed, retrying in 15s..."
sleep 15
done
uses: docker/login-action@v3
with:
username: ${{ inputs.username }}
password: ${{ inputs.password }}
- name: Start Containers
shell: bash
@@ -67,29 +53,16 @@ runs:
cd e2e/test/scenarios/
SERVICES=()
[ "${{ inputs.postgres }}" = true ] && SERVICES+=(postgres-sample)
[ "${{ inputs.mysql }}" = true ] && SERVICES+=(mysql-sample)
[ "${{ inputs.mongo }}" = true ] && SERVICES+=(mongo-sample)
[ "${{ inputs.maildev }}" = true ] && SERVICES+=(maildev maildev-ssl)
[ "${{ inputs.openldap }}" = true ] && SERVICES+=(ldap)
[ "${{ inputs.webhook }}" = true ] && SERVICES+=(webhook-tester)
[ "${{ inputs.snowplow }}" = true ] && SERVICES+=(snowplow-micro)
for attempt in 1 2 3; do
echo "Docker compose up attempt $attempt of 3..."
if docker compose up "${SERVICES[@]}" -d; then
echo "Docker compose up succeeded on attempt $attempt"
break
fi
if [ $attempt -eq 3 ]; then
echo "::error::Docker compose up failed after 3 attempts"
exit 1
fi
echo "::warning::Docker compose up attempt $attempt failed, retrying in 30s..."
docker compose down 2>/dev/null || true
sleep 30
done
docker compose up \
$([ ${{ inputs.postgres }} = true ] && echo "postgres-sample") \
$([ ${{ inputs.mysql }} = true ] && echo "mysql-sample") \
$([ ${{ inputs.mongo }} = true ] && echo "mongo-sample") \
$([ ${{ inputs.maildev }} = true ] && echo "maildev") \
$([ ${{ inputs.maildev }} = true ] && echo "maildev-ssl") \
$([ ${{ inputs.openldap }} = true ] && echo "ldap") \
$([ ${{ inputs.webhook }} = true ] && echo "webhook-tester") \
$([ ${{ inputs.snowplow }} = true ] && echo "snowplow-micro") \
-d
if ${{ inputs.maildev }}; then
echo -e "${Y}checking maildev container..." &&

View File

@@ -26,7 +26,7 @@ runs:
steps:
- name: find_release_artifact
id: find_release_artifact
uses: actions/github-script@v9
uses: actions/github-script@v7
with:
result-encoding: string
script: | # js
@@ -113,9 +113,9 @@ runs:
# Set the JAR_PATH environment variable
if [ "$EXTRACT_PATH" == "." ]; then
echo "JAR_PATH=$OUTPUT_NAME" >> "$GITHUB_ENV"
echo "JAR_PATH=$OUTPUT_NAME" >> $GITHUB_ENV
else
echo "JAR_PATH=$EXTRACT_PATH/$OUTPUT_NAME" >> "$GITHUB_ENV"
echo "JAR_PATH=$EXTRACT_PATH/$OUTPUT_NAME" >> $GITHUB_ENV
fi
- name: Verify that this is a valid JAR file

View File

@@ -22,7 +22,7 @@ runs:
COMMIT=$(env -i git log $BASE_REF --grep="(#$PULL_REQUEST_NUMBER)" --format="%H")
echo "commit SHA $COMMIT"
echo "commit=$COMMIT" >> "$GITHUB_OUTPUT"
echo "commit=$COMMIT" >> $GITHUB_OUTPUT
id: find-squashed-commit
shell: bash
env:

View File

@@ -0,0 +1,45 @@
name: Get cache keys
description: |
Get deterministic cache keys for Maven and Node.js caches based on the centrally controlled cache prefix,
and the suffix based on the runner OS and the current week of the year.
outputs:
m2-cache-key:
description: "The cache key for the Maven cache."
value: ${{ steps.keys-factory.outputs.m2-cache-key }}
m2-restore-key:
description: "The fallback restore key prefix for the Maven cache."
value: ${{ steps.keys-factory.outputs.m2-restore-key }}
node-cache-key:
description: "The cache key for the Node.js cache."
value: ${{ steps.keys-factory.outputs.node-cache-key }}
node-restore-key:
description: "The fallback restore key prefix for the Node.js cache."
value: ${{ steps.keys-factory.outputs.node-restore-key }}
eslint-cache-key:
description: "The cache key for Eslint cache"
value: ${{ steps.keys-factory.outputs.eslint-cache-key }}
cypress-cache-key:
description: "The cache key for Cypress cache"
value: ${{ steps.keys-factory.outputs.cypress-cache-key }}
runs:
using: "composite"
steps:
- name: Generate and output cache keys
id: keys-factory
run: |
M2_CACHE_PREFIX='M2'
NODE_CACHE_PREFIX='node-modules'
ESLINT_CACHE_PREFIX='eslint'
CYPRESS_CACHE_PREFIX='cypress'
WEEK_OF_YEAR=$(/bin/date "+%G-%V")
CACHE_SUFFIX="${{ runner.os }}-$WEEK_OF_YEAR"
echo "m2-cache-key=$M2_CACHE_PREFIX-$CACHE_SUFFIX" >> $GITHUB_OUTPUT
echo "m2-restore-key=$M2_CACHE_PREFIX-" >> $GITHUB_OUTPUT
echo "node-cache-key=$NODE_CACHE_PREFIX-$CACHE_SUFFIX" >> $GITHUB_OUTPUT
echo "node-restore-key=$NODE_CACHE_PREFIX-" >> $GITHUB_OUTPUT
echo "eslint-cache-key=$ESLINT_CACHE_PREFIX-$CACHE_SUFFIX" >> $GITHUB_OUTPUT
echo "cypress-cache-key=$CYPRESS_CACHE_PREFIX-$CACHE_SUFFIX" >> $GITHUB_OUTPUT
shell: bash

View File

@@ -34,7 +34,7 @@ runs:
steps:
- name: Determine Docker info
id: docker-info
uses: actions/github-script@v9
uses: actions/github-script@v7
with:
script: | # js
const branch = '${{ inputs.branch }}';

View File

@@ -1,11 +1,12 @@
name: Notify Pull Request
description: Post a comment on the pull request, optionally replacing the previous one with the same message.
inputs:
remove-previous-comment:
required: false
type: boolean
default: false
add-new-comment:
required: false
type: boolean
default: true
message:
required: true
@@ -17,7 +18,7 @@ runs:
using: "composite"
steps:
- name: Remove previous pull-request comment
uses: actions/github-script@v9
uses: actions/github-script@v7
if: ${{ inputs.remove-previous-comment == 'true' && always() && github.event_name == 'pull_request' }}
with:
script: |
@@ -52,7 +53,7 @@ runs:
}
- name: Notify pull-request
uses: actions/github-script@v9
uses: actions/github-script@v7
if: ${{ inputs.add-new-comment == 'true' && always() && github.event_name == 'pull_request' }}
id: notify-pull-request
with:

View File

@@ -4,7 +4,7 @@ description: |
inputs:
java-version:
required: true
default: "25"
default: "21"
clojure-version:
required: true
default: "1.12.0.1488"
@@ -13,7 +13,7 @@ runs:
using: "composite"
steps:
- name: Prepare JDK ${{ inputs.java-version }}
uses: actions/setup-java@v5.2.0
uses: actions/setup-java@v4
with:
java-version: ${{ inputs.java-version }}
distribution: "temurin"
@@ -32,21 +32,23 @@ runs:
exit 1
fi
- name: Get M2 cache key
uses: ./.github/actions/get-cache-keys
id: get-cache-keys
- name: Install uv (for Python package installation)
uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b # v8.1.0
uses: astral-sh/setup-uv@v7
with:
enable-cache: false
version: "0.10.2" # keep in sync with mise.toml
github-token: ${{ github.token }}
- name: Cache spec
id: cache-spec
shell: bash
run: .github/scripts/cache-keys.sh | tee -a "$GITHUB_OUTPUT"
- name: Restore M2 cache
uses: actions/cache/restore@v5
uses: actions/cache/restore@v4
with:
path: ${{ steps.cache-spec.outputs.m2-path }}
key: ${{ steps.cache-spec.outputs.m2-key }}
restore-keys: ${{ steps.cache-spec.outputs.m2-restore-key }}
path: |
~/.m2
~/.gitlibs
key: ${{ steps.get-cache-keys.outputs.m2-cache-key }}
restore-keys: |
${{ steps.get-cache-keys.outputs.m2-restore-key }}

View File

@@ -1,52 +0,0 @@
name: Prepare CI Conductor
description: >-
Make the ci-conductor sources runnable in the workspace.
Unless the checkout is headed for master, they are overlaid from master, so every
branch runs the latest scripts while its workflow wiring stays branch-historical.
runs:
using: composite
steps:
- name: Detect bun
id: bun
shell: bash
run: |
if command -v bun >/dev/null; then
echo "present=true" >> "$GITHUB_OUTPUT"
else
echo "present=false" >> "$GITHUB_OUTPUT"
fi
- name: Install bun
if: steps.bun.outputs.present != 'true'
uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2.2.0
with:
bun-version-file: "package.json"
token: ${{ github.token }}
# Anything headed for master runs master's own sources, so a PR can test its own changes.
- name: Fetch ci-conductor sources from master
if: ${{ github.base_ref != 'master' && github.ref_name != 'master' }}
uses: actions/checkout@v6
with:
repository: metabase/metabase
ref: master
path: .ci-conductor-master
persist-credentials: false
sparse-checkout-cone-mode: false
sparse-checkout: |
release/ci-conductor
e2e/support/ci_conductor.ts
# e2e/support/config.js and e2e/support/ci_conductor.ts resolve these by relative
# path at bundle time, so the sources have to land where the branch expects them.
- name: Overlay ci-conductor sources from master
if: ${{ github.base_ref != 'master' && github.ref_name != 'master' }}
shell: bash
run: |
set -euo pipefail
rm -rf release/ci-conductor
mv .ci-conductor-master/release/ci-conductor release/ci-conductor
mv .ci-conductor-master/e2e/support/ci_conductor.ts e2e/support/ci_conductor.ts
rm -rf .ci-conductor-master
echo "Overlaid ci-conductor sources from master."

View File

@@ -15,27 +15,21 @@ outputs:
runs:
using: "composite"
steps:
# e2e/support/config.js imports the reporter, so it has to be in place before Cypress bundles.
- name: Prepare CI Conductor
uses: ./.github/actions/prepare-ci-conductor
- name: Install Chrome v144
uses: browser-actions/setup-chrome@2e1d749697dd1612b833dba4a722266286fbefcd # v2.1.2
uses: browser-actions/setup-chrome@v2
with:
chrome-version: 144
id: setup-chrome
- name: Cache spec
id: cache-spec
shell: bash
run: .github/scripts/cache-keys.sh | tee -a "$GITHUB_OUTPUT"
- name: Get cache keys
uses: ./.github/actions/get-cache-keys
id: get-cache-keys
- name: Restore Cypress binary
uses: actions/cache/restore@v5
- name: Get Cypress cache
uses: actions/cache/restore@v4
with:
path: ${{ steps.cache-spec.outputs.cypress-path }}
key: ${{ steps.cache-spec.outputs.cypress-key }}
restore-keys: ${{ steps.cache-spec.outputs.cypress-restore-key }}
path: ~/.cache/Cypress
key: ${{ steps.get-cache-keys.outputs.cypress-cache-key }}
- name: Ensure that Cypress executable is ready
run: |

View File

@@ -16,37 +16,43 @@ runs:
with:
node-version: ${{ inputs.node-version }}
- name: Cache spec
id: cache-spec
shell: bash
run: .github/scripts/cache-keys.sh | tee -a "$GITHUB_OUTPUT"
- name: Get cache keys
uses: ./.github/actions/get-cache-keys
id: get-cache-keys
# The spec names the store path rather than discovering it, so that a job can restore the cache
# before bun exists. Exported job-wide rather than per-step because every later bun invocation has
# to resolve against the same store. Confirm bun agreed: if the variable stopped taking effect, bun
# would fill its default directory and the cache would silently save an empty tree.
- name: Point bun at the cached store
shell: bash
env:
STORE: ${{ steps.cache-spec.outputs.bun-store-path }}
run: |
echo "BUN_INSTALL_CACHE_DIR=$STORE" >> "$GITHUB_ENV"
ACTUAL=$(BUN_INSTALL_CACHE_DIR="$STORE" bun pm cache)
if [ "$ACTUAL" != "$STORE" ]; then
echo "::error::bun reports its store at $ACTUAL, not $STORE; BUN_INSTALL_CACHE_DIR is not taking effect"
exit 1
fi
- name: Restore bun store
uses: actions/cache/restore@v5
- name: Restore M2 cache
uses: actions/cache/restore@v4
with:
path: ${{ steps.cache-spec.outputs.bun-store-path }}
key: ${{ steps.cache-spec.outputs.bun-store-key }}
restore-keys: ${{ steps.cache-spec.outputs.bun-store-restore-key }}
path: |
~/.m2
~/.gitlibs
key: ${{ steps.get-cache-keys.outputs.m2-cache-key }}
restore-keys: |
${{ steps.get-cache-keys.outputs.m2-restore-key }}
- name: Get bun cache directory
id: bun-cache
run: |
BUN_CACHE_DIR=$(bun pm cache)
echo "Bun cache directory: $BUN_CACHE_DIR"
echo "dir=$BUN_CACHE_DIR" >> $GITHUB_OUTPUT
shell: bash
- name: Restore node_modules cache
uses: actions/cache/restore@v4
id: node-modules-cache
with:
path: |
${{ steps.bun-cache.outputs.dir }}
node_modules
key: ${{ steps.get-cache-keys.outputs.node-cache-key }}
restore-keys: |
${{ steps.get-cache-keys.outputs.node-restore-key }}
# postinstall lifecycle hook can be dangerous in CI so we only run it locally (see package.json)
# If a job wants to patch packages, it does so explicitly in the CI. This setting is opt-out.
- name: Install front-end dependencies
# For example, we never want to cache node_modules with patches already applied so cache-generator.yml skips that step.
- name: Clean install
run: bun install --frozen-lockfile
shell: bash
- name: Patch frontend dependencies

View File

@@ -8,13 +8,13 @@ runs:
using: "composite"
steps:
- name: Prepare Node.js
uses: actions/setup-node@v6.4.0
uses: actions/setup-node@v4
with:
node-version: ${{ inputs.node-version }}
node-version-file: ".nvmrc"
- name: Install bun
uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2.2.0
uses: oven-sh/setup-bun@v2
with:
bun-version-file: 'package.json'
bun-version: "1.3.7"
token: ${{ github.token }}

View File

@@ -4,9 +4,9 @@ description: Set up bun and build release scripts
runs:
using: composite
steps:
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2.2.0
- uses: oven-sh/setup-bun@v2
with:
bun-version-file: 'package.json'
bun-version: "1.3.7"
token: ${{ github.token }}
- name: Build release scripts

View File

@@ -54,7 +54,7 @@ runs:
# We have to use the latest version that does not have this issue https://github.com/docker/compose/pull/12752
- name: Set up Docker Compose
uses: docker/setup-compose-action@8cccb8c14b6500aaffebff1aa49c502c34d2e5e6 # v2.1.0
uses: docker/setup-compose-action@v1
with:
version: latest
@@ -67,7 +67,7 @@ runs:
output-name: "metabase.jar"
- name: Retrieve Embedding SDK dist artifact
uses: actions/download-artifact@v8
uses: actions/download-artifact@v4
with:
name: embedding-sdk-${{ inputs.commit }}
path: ${{ github.workspace }}/resources/embedding-sdk
@@ -88,14 +88,14 @@ runs:
REPO_URL="https://github.com/metabase/shoppy.git"
echo "Checking if branch '${SAMPLE_APP_BRANCH_NAME}' exists in ${REPO_URL}..."
if git ls-remote --exit-code --heads "$REPO_URL" "${SAMPLE_APP_BRANCH_NAME}"; then
echo "sample_app_branch_exists=true" >> "$GITHUB_OUTPUT"
echo "sample_app_branch_exists=true" >> $GITHUB_OUTPUT
else
echo "sample_app_branch_exists=false" >> "$GITHUB_OUTPUT"
echo "sample_app_branch_exists=false" >> $GITHUB_OUTPUT
fi
- name: Configure AWS credentials
if: ${{ steps.check-sample-app-branch.outputs.sample_app_branch_exists == 'true' }}
uses: aws-actions/configure-aws-credentials@d979d5b3a71173a29b74b5b88418bfda9437d885
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: ${{ inputs.shoppy-iam-role }}
role-session-name: GitHub_to_AWS_via_FederatedOIDC

View File

@@ -0,0 +1,26 @@
name: Prepare Uberjar Artifact
description: Prepare and upload Metabase JAR as a GitHub artifact.
inputs:
edition:
required: false
default: "ee"
description: Metabase edition (ee or oss)
runs:
using: "composite"
steps:
- name: Mark with the commit hash
run: git rev-parse --short HEAD > COMMIT-ID
shell: bash
- name: Calculate SHA256 checksum
run: sha256sum ./target/uberjar/metabase.jar > SHA256.sum
shell: bash
- name: Upload JARs as artifact
uses: actions/upload-artifact@v4
with:
name: metabase-${{ inputs.edition }}-${{ github.event.pull_request.head.sha || github.sha }}-uberjar
path: |
./target/uberjar/metabase.jar
./COMMIT-ID
./SHA256.sum

View File

@@ -1,38 +0,0 @@
name: CI Conductor quarantine gate
description: >-
Check this job's test failures against ci-conductor's quarantine list.
Either enforces the quarantine or only prints a verdict (dry run).
Fails closed when enforcing but the `CI_CONDUCTOR_BASE_URL` is missing or unreachable.
inputs:
adapter:
description: Which gate to run — one of `backend`, `frontend`, `e2e`.
required: true
test-suite:
description: >-
Suite label to gate (CI_CONDUCTOR_TEST_SUITE). MUST match the label this
job reports under, or the gate checks the wrong quarantine list.
required: true
enforce-quarantine:
description: >-
Whether the gate actually gates the job. When "false", it only prints a verdict.
required: false
default: "false"
runs:
using: composite
steps:
- name: Prepare CI Conductor
uses: ./.github/actions/prepare-ci-conductor
- name: Check failures against the quarantine list
shell: bash
env:
CI_CONDUCTOR_TEST_SUITE: ${{ inputs.test-suite }}
ADAPTER: ${{ inputs.adapter }}
QUARANTINE_DRY_RUN: ${{ inputs.enforce-quarantine != 'true' }}
run: |
case "$ADAPTER" in
backend|frontend|e2e) ;;
*) echo "::error::quarantine-gate: unknown adapter '$ADAPTER' (expected backend|frontend|e2e)"; exit 1 ;;
esac
bun "release/ci-conductor/src/check-${ADAPTER}-quarantine.ts"

View File

@@ -1,35 +0,0 @@
name: Report failures to CI Conductor
description: >-
Report this job's test failures to ci-conductor. Best-effort; no-ops unless
CI_CONDUCTOR_BASE_URL is set. Gate it at the call site with continue-on-error
and an `if:` on the test step's outcome.
inputs:
test-suite:
description: Per-job identity label (CI_CONDUCTOR_TEST_SUITE); unique per job/leg.
required: true
adapter:
description: Which reporter to run; resolves to report-<adapter>.ts. backend or frontend.
required: true
runs:
using: composite
steps:
- name: Prepare CI Conductor
uses: ./.github/actions/prepare-ci-conductor
- name: Report failed tests to ci-conductor
shell: bash
env:
REPO_ID: ${{ github.event.repository.id }}
COMMIT_SHA: ${{ github.event.pull_request.head.sha || github.sha }}
TARGET_BRANCH: ${{ github.event.pull_request.base.ref || github.ref_name }}
CI_CONDUCTOR_TEST_SUITE: ${{ inputs.test-suite }}
ADAPTER: ${{ inputs.adapter }}
IS_RERUN: ${{ github.run_attempt != '1' }}
# Keep the path constrained to known entrypoints.
run: |
case "$ADAPTER" in
backend|frontend) ;;
*) echo "::error::Unknown ci-conductor adapter '$ADAPTER'"; exit 1 ;;
esac
bun "release/ci-conductor/src/report-${ADAPTER}.ts"

View File

@@ -1,66 +0,0 @@
name: Resolve current job id
description: >
Resolve this job's GitHub workflow_jobs.id via the Actions API and export it
as JOB_ID (and JOB_HTML_URL) to $GITHUB_ENV, so any later step can read the
numeric job id without making its own API call. Best-effort: warns and
proceeds on failure so it can never break the run.
runs:
using: composite
steps:
- name: Resolve JOB_ID
shell: bash
env:
# Use the built-in per-job token, not a PAT: this lookup is best-effort
# (a miss just leaves JOB_ID unset, never breaks the run) and other jobs
# depend on the automation PAT's rate-limit budget — we don't want ~70
# e2e shards drawing it down. Worst case under heavy CI load is the
# occasional unresolved job_id, which is acceptable. DEV-2117.
GH_TOKEN: ${{ github.token }}
RUNNER_NAME: ${{ runner.name }}
run: |
# GitHub doesn't expose the current job's numeric id, so we look it up
# from the Actions API. A runner runs exactly one job at a time, so the
# in-progress job with this runner_name within (run_id, run_attempt) is
# unique. We require status == in_progress and no completed_at so a
# reused runner can't match a job it already finished this attempt, with
# max_by(.started_at) as a final tiebreaker.
#
# $GITHUB_RUN_ID is the whole CI run, whose total job count exceeds the
# 100-per-page cap (~185 jobs), so this shard's job is frequently NOT on
# the first page. --paginate --slurp pulls every page and slurps them
# into one array so the filter sees all jobs, not just the first 100 —
# without it the lookup returned null whenever this job sorted past
# position 100 (DEV-2117).
#
# gh rejects `--slurp` together with `--jq` ("the --slurp option is not
# supported with --jq"), so we slurp with gh and run the filter through a
# standalone `jq`. The earlier `--slurp --jq` form errored on every
# attempt — swallowed by 2>/dev/null — leaving JOB_ID permanently unset.
set +e
api_url="/repos/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID/attempts/$GITHUB_RUN_ATTEMPT/jobs?per_page=100"
jq_filter="[.[].jobs[] | select(.runner_name == \"$RUNNER_NAME\" and .status == \"in_progress\" and .completed_at == null)] | max_by(.started_at) | \"\(.id) \(.html_url)\""
# The Actions jobs API is eventually consistent: after a runner picks
# up a job, it can take a few seconds for runner_name to appear in the
# response. Retry briefly to absorb that.
out=""
for attempt in 1 2 3 4 5; do
out=$(gh api --paginate --slurp "$api_url" 2>/dev/null | jq -r "$jq_filter" 2>/dev/null)
if [ -n "$out" ] && [ "$out" != "null null" ]; then
break
fi
out=""
echo "Attempt $attempt: job not yet visible in Actions API; retrying..."
sleep 3
done
if [ -n "$out" ]; then
read -r job_id job_url <<<"$out"
echo "JOB_ID=$job_id" >> "$GITHUB_ENV"
echo "JOB_HTML_URL=$job_url" >> "$GITHUB_ENV"
echo "Resolved JOB_ID=$job_id (runner=$RUNNER_NAME)"
else
echo "::warning::Could not resolve current job id after 5 attempts (runner=$RUNNER_NAME)"
fi
exit 0

View File

@@ -0,0 +1,111 @@
name: Embedding SDK Bundle Size Check
description: |
Downloads the current branch's uberjar and builds the base ref's
embedding-sdk.js from source, compares their sizes,
and sets output if the size change exceeds the threshold.
inputs:
edition:
description: 'Metabase edition (ee or oss)'
required: true
base-ref-commit:
description: 'Commit SHA of the base ref (e.g. merge base).'
required: true
threshold-percent:
description: 'Percent threshold for bundle size change'
required: true
outputs:
status:
description: "Bundle size status: increased, decreased, or stable"
value: ${{ steps.compare.outputs.status }}
size_change_percent:
description: "size change percent (positive = increase, negative = decrease)"
value: ${{ steps.compare.outputs.size_change_percent }}
runs:
using: "composite"
steps:
- name: Prepare backend
uses: ./.github/actions/prepare-backend
- name: Prepare frontend
uses: ./.github/actions/prepare-frontend
- name: Download current uberjar
uses: ./.github/actions/fetch-artifact
with:
commit: ${{ github.event.pull_request.head.sha || github.sha }}
edition: ${{ inputs.edition }}
extract-path: temp/current-uberjar
- name: Extract embedding-sdk bundles from current uberjar
shell: bash
run: |
# Try new layout first (app/embedding-sdk/legacy/), fall back to old layout (app/embedding-sdk.js)
unzip -j temp/current-uberjar/metabase.jar \
"frontend_client/app/embedding-sdk/legacy/embedding-sdk.js" \
-d temp/current-bundle 2>/dev/null || \
unzip -j temp/current-uberjar/metabase.jar \
"frontend_client/app/embedding-sdk.js" \
-d temp/current-bundle
# Extract chunks directory if it exists (new layout only)
mkdir -p temp/current-chunks
unzip -j temp/current-uberjar/metabase.jar \
"frontend_client/app/embedding-sdk/chunks/*" \
-d temp/current-chunks 2>/dev/null || true
- name: Checkout base ref
uses: actions/checkout@v4
with:
ref: ${{ inputs.base-ref-commit }}
path: temp/base-ref-src
- name: Build base-ref embedding-sdk.js
shell: bash
run: |
cd temp/base-ref-src
bun install
MB_EDITION=${{ inputs.edition }} bun run build-release
- name: Compare sizes and set output
id: compare
shell: bash
run: |
# --- Legacy (monolithic) bundle comparison ---
current_size=$(wc -c < temp/current-bundle/embedding-sdk.js) || { echo "::error::Failed to read current bundle"; exit 1; }
# Base ref: try new layout first, fall back to old
if [ -f "temp/base-ref-src/resources/frontend_client/app/embedding-sdk/legacy/embedding-sdk.js" ]; then
base_ref_size=$(wc -c < "temp/base-ref-src/resources/frontend_client/app/embedding-sdk/legacy/embedding-sdk.js")
else
base_ref_size=$(wc -c < "temp/base-ref-src/resources/frontend_client/app/embedding-sdk.js")
fi
[ "$current_size" -gt 0 ] && [ "$base_ref_size" -gt 0 ] || { echo "::error::Bundle file is empty"; exit 1; }
diff=$((current_size - base_ref_size))
percent=$(( diff * 100 / base_ref_size ))
echo "=== Legacy (monolithic) bundle ==="
echo "Current size: $current_size bytes"
echo "Base ref size: $base_ref_size bytes"
echo "Difference: $diff bytes ($percent%)"
threshold="${{ inputs.threshold-percent }}"
if [ "$percent" -gt "$threshold" ]; then
status="increased"
elif [ "$percent" -lt "$(( -threshold ))" ]; then
status="decreased"
else
status="stable"
fi
echo "status=$status" >> $GITHUB_OUTPUT
echo "size_change_percent=$percent" >> $GITHUB_OUTPUT
# --- Chunked bundle (informational) ---
if ls temp/current-chunks/*.js 1>/dev/null 2>&1; then
chunks_size=$(cat temp/current-chunks/*.js | wc -c)
echo ""
echo "=== Chunked bundle (bootstrap + chunks) ==="
echo "Total chunked size: $chunks_size bytes"
echo "chunks_size=$chunks_size" >> $GITHUB_OUTPUT
else
echo ""
echo "No chunked bundle found (expected on master)"
fi

View File

@@ -9,26 +9,13 @@ inputs:
build-static-viz:
required: true
default: false
artifact-suffix:
logs-artifact-suffix:
required: false
description: >-
Discriminator appended to this job's artifact names (test logs, granular-rerun failed
tests). A job whose matrix runs several concurrent legs MUST set a per-leg-unique value,
or the legs collide on one artifact — for the rerun artifact that means a leg restores
another leg's failed set. Must be stable across attempts of the same job.
description: Suffix to append to the test logs artifact name to avoid conflicts in matrix jobs
edition:
required: false
default: "ee"
default: 'ee'
description: Edition to test (oss or ee)
quarantine-engine:
required: false
granular-rerun:
required: false
default: "true"
description: >-
When "true" (default), a re-run of a failed job runs only the tests that failed on the
previous attempt (hawk `:only`). Stress workflows set this to "false": they intentionally
run a fixed test set `:times N`, and narrowing to failures would drop that repetition.
outputs:
test-outcome:
@@ -40,10 +27,6 @@ runs:
steps:
- name: Prepare back-end environment
uses: ./.github/actions/prepare-backend
- name: Resolve current job id
# Resolve JOB_ID early (before the long test step) so the ci-conductor
# reporter below can link failures to this exact job. Best-effort.
uses: ./.github/actions/resolve-job-id
- name: Prepare front-end environment
if: ${{ inputs.build-static-viz == 'true' }}
uses: ./.github/actions/prepare-frontend
@@ -52,123 +35,17 @@ runs:
run: bun run build-static-viz
shell: bash
env:
MB_EDITION: ${{ inputs.edition }}
# Granular rerun: on a re-run of a failed job, restore the list
# of tests that failed on the previous attempt (uploaded below) so the test
# step can target only those instead of the whole suite. Best-effort: a
# missing/empty artifact leaves no file and the run falls back to the full
# suite. First attempts skip this entirely.
#
# Scoped to the IMMEDIATELY preceding attempt: an attempt that passed, or
# that failed with nothing attributable to a var, uploads no artifact, and a
# narrow set must not outlive it. Reading `attempt - 1` means "only what the
# last attempt actually recorded" — anything else falls back to the full suite.
# Workflow expressions have no arithmetic, so the shell computes it.
#
# run-tests.yml calls app-db.yml, backend.yml, drivers.yml and semantic-search.yml
# as part of ONE run, so all four share an artifact namespace. `junit-name` alone is not
# unique there — app-db.yml and drivers.yml both derive `be-tests-postgres-ee` — so
# `artifact-suffix` carries the rest of the discriminator.
- name: Resolve rerun artifact names
if: ${{ inputs.granular-rerun == 'true' }}
shell: bash
env:
RERUN_ARTIFACT_BASE: ${{ format('failed-tests-{0}{1}', inputs.junit-name, inputs.artifact-suffix && format('-{0}', inputs.artifact-suffix) || '') }}
run: |
echo "RERUN_ARTIFACT=$RERUN_ARTIFACT_BASE-attempt-$GITHUB_RUN_ATTEMPT" >> "$GITHUB_ENV"
echo "RERUN_PREV_ARTIFACT=$RERUN_ARTIFACT_BASE-attempt-$((GITHUB_RUN_ATTEMPT - 1))" >> "$GITHUB_ENV"
# `pattern` (not `name`) because the artifact is legitimately absent whenever the
# previous attempt recorded nothing: `name` treats that as an error, `pattern`
# downloads nothing and moves on. The value is a literal — no glob metacharacters —
# so it matches exactly one artifact, which lands at `target/failed-tests`. A pattern
# that ever matched two would nest them under their artifact names instead, leaving
# the file absent and falling back to the full suite.
- name: Restore previously-failed tests
if: ${{ inputs.granular-rerun == 'true' && github.run_attempt != '1' }}
continue-on-error: true
uses: actions/download-artifact@v8
with:
pattern: ${{ env.RERUN_PREV_ARTIFACT }}
path: target
MB_EDITION: ee
- name: Test database driver
# On a rerun with a non-empty failed-tests file, target only those vars via
# hawk `:only` (a var-level selector bypasses `:only-tags`, which is fine —
# every listed var already passed that filter on the first attempt).
# Otherwise run exactly as before. Quote the file so the whole `[ns/var ...]`
# vector reaches clojure -X as a single argument.
run: |
BASE="clojure -X:dev:ci:${{ inputs.edition }}:${{ inputs.edition }}-dev:drivers:drivers-dev:test"
# `:profile-tests` overrides `:test`'s exec-fn, so it has to come last. See dev.profile-tests.
# The `profiled` output is what the collection steps below key off, so that the decision is
# recorded by the step that acts on it rather than re-derived from the environment.
if [ "$PROFILE_TESTS" = "true" ]; then
echo "Profiling this run; a flamegraph will be attached to the job."
echo "profiled=true" >> "$GITHUB_OUTPUT"
BASE="$BASE:profile-tests"
fi
# An explicit selection wins over the rerun narrowing below: it was asked for by hand, on this
# attempt. Names may be separated by commas or spaces; hawk wants them as one vector argument.
if [ -n "$ONLY_TESTS" ]; then
ONLY="[$(echo "$ONLY_TESTS" | tr ',' ' ')]"
echo "Running only: $ONLY"
$BASE :only "$ONLY"
elif [ "${{ inputs.granular-rerun }}" = "true" ] && [ "${{ github.run_attempt }}" != "1" ] && [ -s target/failed-tests ]; then
echo "Granular rerun of previously-failed tests: $(cat target/failed-tests)"
$BASE :only "$(cat target/failed-tests)"
else
$BASE ${{ inputs.test-args }}
fi
run: clojure -X:dev:ci:${{ inputs.edition }}:${{ inputs.edition }}-dev:drivers:drivers-dev:test ${{ inputs.test-args }}
shell: bash
id: run-test
# Keep this step continue-on-error because the quarantine gate below controls whether it
# ultimately fails.
# Trunk Quarantine controls the final state of the run in the CI!
# Even if this step fails we let it pass, and then `trunk-io/analytics-uploader` determines
# whether to fail or to pass the whole job, based on the quarantine list.
continue-on-error: true
- name: Collect profile
if: ${{ !cancelled() }}
uses: ./.github/actions/collect-profile
with:
profiled: ${{ steps.run-test.outputs.profiled }}
artifact-name: ${{ format('profile-{0}{1}', inputs.junit-name, inputs.artifact-suffix && format('-{0}', inputs.artifact-suffix) || '') }}
- name: Report failed tests to ci-conductor
if: ${{ !cancelled() && ( steps.run-test.outcome == 'failure' || github.run_attempt != '1' ) }}
# This step should NEVER fail the job!
continue-on-error: true
uses: ./.github/actions/report-failures-to-ci-conductor
with:
test-suite: ${{ inputs.junit-name }}
adapter: backend
# Granular rerun: record which tests failed so a re-run of this
# job targets only those (restored above). Reuses the backend JUnit adapter
# from the ci-conductor sources the report step just prepared. Best-effort:
# writes nothing unless every failure is attributable to a var, so an
# unattributable failure falls back to a full rerun.
- name: Collect failed tests for rerun
if: ${{ inputs.granular-rerun == 'true' && !cancelled() && steps.run-test.outcome == 'failure' }}
continue-on-error: true
shell: bash
# Clear the restored set first, so a collector that dies before deciding
# can't leave the previous attempt's file to be uploaded as this one's.
run: |
rm -f target/failed-tests
bun release/ci-conductor/src/collect-failed-backend.ts
- name: Upload failed tests for rerun
if: ${{ inputs.granular-rerun == 'true' && !cancelled() && steps.run-test.outcome == 'failure' }}
continue-on-error: true
uses: actions/upload-artifact@v7
with:
name: ${{ env.RERUN_ARTIFACT }}
path: target/failed-tests
if-no-files-found: ignore
- name: Publish Test Report (JUnit)
uses: dorny/test-reporter@a43b3a5f7366b97d083190328d2c652e1a8b6aa2 # v3.0.0
uses: dorny/test-reporter@v1
# As per the documentation: https://docs.github.com/en/actions/reference/workflows-and-actions/contexts#steps-context
# test.outcome gives us access to the result before `continue-on-error` overrides it and sets it to 'success'.
# We want test reports even if the test step gets cancelled (e.g. times out), hence we match anything that is not a 'success'.
@@ -180,23 +57,12 @@ runs:
list-suites: failed
list-tests: failed
fail-on-error: false
- name: Upload Logs on Failure
if: always() && steps.run-test.outcome != 'success'
uses: actions/upload-artifact@v7
uses: actions/upload-artifact@v4
with:
name: ${{ format('test-logs-{0}{1}', github.job, inputs.artifact-suffix && format('-{0}', inputs.artifact-suffix) || '') }}
name: ${{ format('test-logs-{0}{1}', github.job, inputs.logs-artifact-suffix && format('-{0}', inputs.logs-artifact-suffix) || '') }}
path: logs/test-log.json
retention-days: 1
# Failing to upload the artifact should never affect the outcome of the whole test run.
continue-on-error: true
# Last, because this is the only step allowed to fail the job: everything above it reports the
# failure (JUnit report, logs, ci-conductor) and must run regardless of the verdict below.
- name: Quarantine gate
if: ${{ !cancelled() && steps.run-test.outcome == 'failure' }}
uses: ./.github/actions/quarantine-gate
with:
adapter: backend
test-suite: ${{ inputs.junit-name }}
enforce-quarantine: ${{ inputs.quarantine-engine == 'ci-conductor' }}

41
.github/actions/update-cache/action.yml vendored Normal file
View File

@@ -0,0 +1,41 @@
name: Update cache
description: |
Update a GitHub Actions cache based on a provided path and key.
If a cache entry already exists for the given key, it is first removed.
This ensures immutability of cache entries, and prevents runtime errors.
inputs:
path:
description: "The path(s) to cache, separated by newlines."
required: true
key:
description: "The cache key to use."
required: true
runs:
using: "composite"
steps:
- name: Check if the cache entry exists
id: cache-lookup
uses: actions/cache/restore@v4
with:
path: ${{ inputs.path }}
key: ${{ inputs.key }}
lookup-only: true # Important bit: the action will not fetch anything, just check for the cache existence.
- name: Delete the cache entry if it exists
if: ${{ steps.cache-lookup.outputs.cache-hit == 'true' }}
run: |
echo "Cache entry found for key '${{ inputs.key }}'. Deleting it to ensure immutability."
gh api \
--method DELETE \
-H "Accept: application/vnd.github+json" \
-H "X-GitHub-Api-Version: 2022-11-28" \
/repos/${{ github.repository }}/actions/caches?key=${{ inputs.key }}
env:
GH_TOKEN: ${{ github.token }}
shell: bash
- name: Save the new cache entry
uses: actions/cache/save@v4
with:
path: ${{ inputs.path }}
key: ${{ inputs.key }}

View File

@@ -1,5 +1,4 @@
name: Create Backend Dep Bump PRs
description: Opens a pull request for each backend dependency that has a newer version available.
inputs:
github-token:

View File

@@ -0,0 +1,64 @@
name: Upload Test Results to s3
description: Uploads test results files to s3
inputs:
input-path:
required: true
output-name:
required: true
default: test
role-to-assume:
required: true
aws-region:
required: true
default: us-east-1
bucket:
required: true
trunk-api-token:
required: true
variant:
required: false
default: ""
previous-step-outcome:
required: true
runs:
using: "composite"
steps:
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: ${{ inputs.role-to-assume }}
aws-region: ${{ inputs.aws-region }}
- name: zip test results
env:
INPUT_DIR: ${{ inputs.input-path }}
OUTPUT_FILE: ${{ inputs.output-name }}
shell: bash
run: | # sh
zip ${OUTPUT_FILE}.zip $INPUT_DIR/*.xml
- name: Upload test results to S3
env:
BUCKET: ${{ inputs.bucket }}
OUTPUT_FILE: ${{ inputs.output-name }}
shell: bash
run: | # sh
DATE=$(date '+%Y-%m-%d')
aws s3 cp ${OUTPUT_FILE}.zip s3://$BUCKET/$DATE/$GITHUB_RUN_ID/$GITHUB_RUN_ATTEMPT/
- name: Upload results to Trunk
# Important!
# 1. This step must always run because it controls job's exit code.
# 2. There's no point in running this outside the organization
if: always() && github.repository_owner == 'metabase'
uses: trunk-io/analytics-uploader@main
env:
TRUNK_REPO_URL: git@github.com:metabase/metabase.git
with:
junit-paths: ${{ inputs.input-path }}
variant: ${{ inputs.variant || inputs.output-name }}
org-slug: metabase
dry-run: ${{ (github.event_name == 'pull_request' && github.event.pull_request.draft == true) || github.repository != 'metabase/metabase' }} # quarantine, but do not upload results
token: ${{ inputs.trunk-api-token }}
use-cache: true
previous-step-outcome: ${{ inputs.previous-step-outcome }}

View File

@@ -0,0 +1,22 @@
name: 'Verify Organization Membership'
description: 'Check if a user is a member of a GitHub organization'
inputs:
github_token:
description: 'Token with access to the organization'
required: true
organization_name:
description: 'Organization name'
required: true
username:
description: 'Username'
required: true
outputs:
is_member:
value: ${{ steps.verify-membership.outputs.is_member }}
description: "Is the user a member of the organization"
runs:
using: "composite"
steps:
- run: bash ${{ github.action_path }}/verify_membership.sh ${{ inputs.github_token }} ${{ inputs.organization_name }} ${{ inputs.username }}
id: verify-membership
shell: bash

View File

@@ -0,0 +1,25 @@
#!/bin/bash
TOKEN=$1
ORG_NAME=$2
USERNAME=$3
response=$(curl -L -o /dev/null -w "%{http_code}" \
-H "Accept: application/vnd.github+json" \
-H "Authorization: Bearer $TOKEN" \
-H "X-GitHub-Api-Version: 2022-11-28" \
"https://api.github.com/orgs/$ORG_NAME/members/$USERNAME")
# Github returns an empty response with 204 status if user is a member and 404 status otherwise
# See: https://docs.github.com/en/rest/orgs/members?apiVersion=2022-11-28#check-organization-membership-for-a-user
if [ "$response" == "204" ]; then
echo "The user $USERNAME is a member of $ORG_NAME"
echo "is_member=true" >> $GITHUB_OUTPUT
elif [ "$response" == "404" ]; then
echo "The user $USERNAME is not a member of $ORG_NAME"
echo "is_member=false" >> $GITHUB_OUTPUT
else
echo "Failed to determine the membership status for $USERNAME in $ORG_NAME"
echo "is_member=false" >> $GITHUB_OUTPUT
exit 1
fi

View File

@@ -1,16 +1,7 @@
# Catch-all filter — emits the full changed-files list as `all_changed_files`
# so create-test-plan in run-tests.yml can reuse dorny's diff.
all_changed:
- "**"
default: &default
- "locales/**"
- "bin/**/!(*.md)"
actionlint:
- added|modified: ".github/workflows/*.yml"
- added|modified: ".github/workflows/*.yaml"
shared_sources: &shared_sources
- "src/**/*.cljc"
- "src/**/*.cljs"
@@ -54,6 +45,11 @@ frontend_sources: &frontend_sources
- "enterprise/frontend/!(test)**/**/{!(*.spec),}.{js,jsx,ts,tsx,css}"
- "frontend/!(test)**/**/{!(*.spec),}.{js,jsx,ts,tsx,css}"
# Used only in .github/workflows/conventional-commit-pr-title-reminder.yml
frontend_embedding_sources:
- "enterprise/frontend/src/{embedding,embedding-*}/**/{!(*.spec),}.{js,jsx,ts,tsx,css}"
- "enterprise/frontend/src/metabase-enterprise/{embedding,embedding-*}**/{!(*.spec),}.{js,jsx,ts,tsx,css}"
# Used only in .github/workflows/embedding-sdk-labels-reminder.yml
frontend_embedding_sdk_package_sources:
- "enterprise/frontend/src/embedding-sdk-package/**/!(*.spec).{js,jsx,ts,tsx,css}"
@@ -81,31 +77,19 @@ frontend_loki_ci: &frontend_loki_ci
- ".loki/**"
- ".storybook/**"
frontend_unit_infra: &frontend_unit_infra
- *frontend_configs
- *frontend_ci
- "jest.config.js"
- "jest.tz.unit.conf.json"
- "frontend/test/**"
- "frontend/**/__mocks__/**"
frontend_loki_infra: &frontend_loki_infra
- *frontend_configs
- *shared_ci
- *frontend_loki_ci
# e2e harness/support code (commands, support file, runner). Not in the
# instrumented bundle, so it maps to no covered module — a change here forces a
# full e2e run (see create-test-plan / e2eInfraTouched).
e2e_infra: &e2e_infra
- *shared_e2e
backend_ci: &backend_ci
- *shared_ci
- ".github/actions/test-driver/**"
- ".github/workflows/backend.yml"
- ".github/workflows/drivers.yml"
backend_presto_kerberos:
- "**/presto_jdbc/**"
- "**/presto-jdbc/**" # the driver module has a hyphen. underscores are classpaths
- "**/presto_jdbc.clj"
- "**/presto_jdbc_test.clj"
- ".github/workflows/presto-kerberos-integration-test.yml"
backend_sources: &backend_sources
- *shared_sources
- "**deps.edn"
@@ -143,7 +127,7 @@ custom_viz_all:
e2e_ci: &e2e_ci
- *shared_ci
- ".github/workflows/uberjar.yml"
- ".github/actions/prepare-uberjar-artifact/**"
- ".github/actions/fetch-artifact/action.yml"
- ".github/actions/e2e-prepare-containers/**"
- ".github/actions/prepare-cypress/**"
@@ -154,7 +138,7 @@ e2e_specs: &e2e_specs
embedding_sdk_ci: &embedding_sdk_ci
- *shared_ci
- ".github/workflows/uberjar.yml"
- ".github/actions/prepare-uberjar-artifact/**"
- ".github/actions/fetch-artifact/action.yml"
- ".github/actions/prepare-cypress/**"
- ".github/workflows/embedding-sdk*.yml"
@@ -176,7 +160,7 @@ embedding_sdk_host_sample_apps:
- "e2e/test-host-app/**/*.cy.*.{js,ts}"
snowplow: &snowplow
- ".github/workflows/lint.yml"
- ".github/workflows/snowplow.yml"
- "snowplow/**"
- "bin/lint-snowplow-schemas.bb"
- "bin/lint-snowplow-schemas-test.bb"
@@ -193,9 +177,6 @@ documentation:
- "docs/**"
- "**/*.md"
docs:
- "docs/**"
embedding_documentation:
- ".typedoc/**"
- "docs/embedding/**"
@@ -205,7 +186,7 @@ yaml:
- "**/*.yaml"
codeql:
- ".github/workflows/lint.yml"
- ".github/workflows/codeql.yml"
- "frontend/src/**"
- "enterprise/frontend/src/**"
@@ -213,43 +194,26 @@ i18n:
- *default
- ".github/actions/prepare-frontend/**"
- ".github/actions/prepare-backend/**"
- ".github/workflows/lint.yml"
- ".github/workflows/i18n.yml"
- *sources
visualizations:
- "frontend/src/metabase/visualizations/**"
- "frontend/src/metabase/static-viz/**"
- "src/metabase/pulse/**"
cljs:
- *shared_sources
- *shared_specs
- ".github/workflows/cljs.yml"
- "shadow-cljs.edn"
embedder_plugin:
- ".github/actions/prepare-backend/**"
- ".github/file-paths.yaml"
- ".github/workflows/build-scripts.yml"
- "bin/build-embedder-plugin.sh"
- "bin/build/src/build/embedder_model.clj"
- "bin/build/src/build/plugin_uberjar.clj"
- "bin/build/src/build_embedder_plugin.clj"
- "deps.edn"
- "modules/embedder/**"
- "enterprise/backend/src/metabase_enterprise/data_complexity_score/complexity_embedders.clj"
- "enterprise/backend/src/metabase_enterprise/data_complexity_score/settings.clj"
- "enterprise/backend/src/metabase_enterprise/data_complexity_score/synonym_source.clj"
- "enterprise/backend/src/metabase_enterprise/semantic_search/embedding.clj"
- "enterprise/backend/src/metabase_enterprise/semantic_search/settings.clj"
- "enterprise/backend/test/metabase_enterprise/semantic_search/appdb_pgvector_mode_test.clj"
- "enterprise/backend/test/metabase_enterprise/semantic_search/embedding_test.clj"
- "src/metabase/embeddings/provider.clj"
- "src/metabase/embeddings/startup.clj"
- "src/metabase/plugins/**"
- "test/metabase/embeddings/embedder_plugin_artifact_test.clj"
serialization_baseline:
- ".github/workflows/serialization.yml"
- "test_resources/serialization_baseline/**"
openapi:
- ".github/workflows/lint.yml"
- ".github/workflows/openapi.yml"
- "src/metabase/api/**"
- "src/metabase/api_routes/**"
- "src/metabase/*_rest/api.clj"

View File

@@ -1,12 +0,0 @@
---
applyTo: "e2e/**/*.cy.spec.{js,ts}"
---
# Cypress E2E Spec Review
When reviewing Cypress E2E specs, apply the guidelines in these files:
- `.claude/skills/_shared/cypress-conventions.md`
- `.claude/skills/e2e-test-review/SKILL.md`
These are in addition to the TypeScript/JavaScript instructions, which also match these files.

View File

@@ -1,242 +0,0 @@
import micromatch from "micromatch";
export type ModuleDef = { type: string; pattern: string };
export type Rule = {
from: string[];
allow?: string[];
disallow?: string[];
};
export type ModuleNode = { type: string; regex: RegExp };
export type FileDependency = { source: string; dependencies: string[] };
/** First match wins, so element order matters for nested patterns. */
export function buildNodes(elements: ModuleDef[]): ModuleNode[] {
return elements.map((el) => ({
type: el.type,
// dot:true so files in dot-directories (e.g. .storybook) still match.
regex: micromatch.makeRe(el.pattern, { dot: true }),
}));
}
export function mapFileToModule(
nodes: ModuleNode[],
path: string,
): string | null {
for (const node of nodes) {
if (node.regex.test(path)) {
return node.type;
}
}
return null;
}
export function getChangedModules(
nodes: ModuleNode[],
changedFiles: string[],
): Set<string> {
const direct = new Set<string>();
for (const file of changedFiles) {
const module = mapFileToModule(nodes, file);
if (module) {
direct.add(module);
}
}
return direct;
}
export type FileGraph = {
nodes: ModuleNode[];
fileDependents: Map<string, Set<string>>;
};
/** Drops unresolvable imports (e.g. ambient types) that point at no real file. */
export function parseCruiseModules(
modules: Array<{
source: string;
dependencies?: Array<{ resolved: string; couldNotResolve?: boolean }>;
}>,
): FileDependency[] {
return modules.map((module) => ({
source: module.source,
dependencies: (module.dependencies ?? [])
.filter((dep) => !dep.couldNotResolve)
.map((dep) => dep.resolved),
}));
}
/** Inverts the import edges into the `fileDependents` reverse graph. */
export function buildFileGraph(
elements: ModuleDef[],
fileDependencies: FileDependency[],
): FileGraph {
const fileDependents = new Map<string, Set<string>>();
for (const { source, dependencies } of fileDependencies) {
for (const target of dependencies) {
let importers = fileDependents.get(target);
if (!importers) {
importers = new Set();
fileDependents.set(target, importers);
}
importers.add(source);
}
}
return { nodes: buildNodes(elements), fileDependents };
}
/** Changed files plus every file that transitively imports them. */
export function getAffectedFiles(
fileGraph: FileGraph,
changedFiles: string[],
): Set<string> {
const affected = new Set(changedFiles);
const queue = [...changedFiles];
while (queue.length > 0) {
const file = queue.shift()!;
for (const importer of fileGraph.fileDependents.get(file) ?? []) {
if (!affected.has(importer)) {
affected.add(importer);
queue.push(importer);
}
}
}
return affected;
}
export type ModuleGraph = {
nodes: ModuleNode[];
/**
* dependents.get(M) is every module affected when M changes — the full
* transitive closure, so getAffectedModules only needs a single hop.
*/
dependents: Map<string, Set<string>>;
};
/** Expands a direct-dependents map into its transitive closure. */
function transitiveClosure(
direct: Map<string, Set<string>>,
): Map<string, Set<string>> {
const closure = new Map<string, Set<string>>();
for (const start of direct.keys()) {
const reached = new Set<string>();
const queue = [...(direct.get(start) ?? [])];
while (queue.length > 0) {
const node = queue.shift()!;
if (reached.has(node)) {
continue;
}
reached.add(node);
for (const next of direct.get(node) ?? []) {
queue.push(next);
}
}
reached.delete(start);
closure.set(start, reached);
}
return closure;
}
/**
* Builds a module graph from the actual import graph. Reachability is computed
* per module at the file level and then collapsed to modules.
*/
export function buildUsageModuleGraph(
elements: ModuleDef[],
fileDependencies: FileDependency[],
): ModuleGraph {
const nodes = buildNodes(elements);
const fileGraph = buildFileGraph(elements, fileDependencies);
const filesByModule = new Map<string, string[]>();
const allFiles = new Set<string>();
for (const { source, dependencies } of fileDependencies) {
allFiles.add(source);
for (const target of dependencies) {
allFiles.add(target);
}
}
for (const file of allFiles) {
const module = mapFileToModule(nodes, file);
if (module) {
const files = filesByModule.get(module);
if (files) {
files.push(file);
} else {
filesByModule.set(module, [file]);
}
}
}
const dependents = new Map<string, Set<string>>(
elements.map((el) => [el.type, new Set()]),
);
for (const [module, files] of filesByModule) {
const affected = new Set<string>();
for (const file of getAffectedFiles(fileGraph, files)) {
const owner = mapFileToModule(nodes, file);
if (owner && owner !== module) {
affected.add(owner);
}
}
dependents.set(module, affected);
}
return { nodes, dependents };
}
export function buildModuleGraph(
elements: ModuleDef[],
rules: Rule[],
): ModuleGraph {
const nodes = buildNodes(elements);
const allTypes = elements.map((e) => e.type);
function expandPattern(pattern: string): string[] {
if (pattern === "*") {
return [...allTypes];
}
if (pattern.endsWith("/*")) {
const prefix = pattern.slice(0, -1);
return allTypes.filter((type) => type.startsWith(prefix));
}
return allTypes.includes(pattern) ? [pattern] : [];
}
const directDependents = new Map<string, Set<string>>(
allTypes.map((type) => [type, new Set()]),
);
for (const rule of rules) {
// Grandfathered imports that break the disallow rules still exist in the code,
// and selection must cover every import that actually exists.
// So disallow rules are ignored, at the cost of a looser graph.
const fromTypes = rule.from.flatMap(expandPattern);
const allowTypes = (rule.allow ?? []).flatMap(expandPattern);
for (const target of allowTypes) {
for (const importer of fromTypes) {
if (importer !== target) {
directDependents.get(target)?.add(importer);
}
}
}
}
return { nodes, dependents: transitiveClosure(directDependents) };
}
/**
* Returns the changed modules and all modules that directly or indirectly depend on them
*/
export function getAffectedModules(
moduleGraph: ModuleGraph,
changedFiles: string[],
): Set<string> {
const affected = getChangedModules(moduleGraph.nodes, changedFiles);
for (const module of [...affected]) {
for (const dep of moduleGraph.dependents.get(module) ?? []) {
affected.add(dep);
}
}
return affected;
}

View File

@@ -1,177 +0,0 @@
import {
buildFileGraph,
buildModuleGraph,
buildNodes,
buildUsageModuleGraph,
getAffectedFiles,
getAffectedModules,
getChangedModules,
mapFileToModule,
parseCruiseModules,
} from "./affected-modules";
const ELEMENTS = [
{ type: "lib/utils", pattern: "src/utils/**" },
{ type: "lib/types", pattern: "src/types/**" },
// Nested-pattern pair (mirrors mlv1-inside-mlv2): inner must come first
// so first-match-wins picks it up before the outer pattern.
{ type: "lib/inner", pattern: "src/shared/v1/**" },
{ type: "lib/outer", pattern: "src/shared/**" },
{ type: "feature/foo", pattern: "src/foo/**" },
{ type: "feature/bar", pattern: "src/bar/**" },
{ type: "feature/super", pattern: "src/super/**" },
// Different folder root (mirrors feature/enterprise's enterprise/ path).
{ type: "feature/external", pattern: "external/lib/**" },
// Same type, two patterns (mirrors app/misc per-entry-point declarations).
{ type: "app/main", pattern: "src/app.js" },
{ type: "app/main", pattern: "src/embed.tsx" },
{ type: "shared/other", pattern: "src/*/**" }, // catch-all, must be last
];
const RULES = [
...ELEMENTS.map((el) => ({ from: [el.type], allow: [el.type] })),
{ from: ["lib/*"], allow: ["lib/*"] },
{ from: ["feature/*"], allow: ["lib/*"] },
{ from: ["feature/super"], allow: ["feature/*"] },
{ from: ["app/*"], allow: ["lib/*", "feature/*", "app/*"] },
];
const nodes = buildNodes(ELEMENTS);
const graph = buildModuleGraph(ELEMENTS, RULES);
describe("affected modules", () => {
describe("mapFileToModule", () => {
it.each([
["src/utils/colors.ts", "lib/utils"],
["src/types/api.ts", "lib/types"],
["src/foo/foo.tsx", "feature/foo"],
["src/bar/bar.tsx", "feature/bar"],
["src/super/index.ts", "feature/super"],
["src/app.js", "app/main"],
["src/randomthing/file.ts", "shared/other"],
["src/shared/v1/expr.ts", "lib/inner"],
["src/shared/v2/expr.ts", "lib/outer"],
["src/shared/index.ts", "lib/outer"],
["external/lib/foo.ts", "feature/external"],
["src/embed.tsx", "app/main"],
])("should map %s to %s", (path, expected) => {
expect(mapFileToModule(nodes, path)).toBe(expected);
});
it("should return null for files outside any pattern", () => {
expect(mapFileToModule(nodes, "docs/foo.md")).toBe(null);
expect(mapFileToModule(nodes, "README.md")).toBe(null);
expect(mapFileToModule(nodes, "external/notlib/foo.ts")).toBe(null);
});
});
describe("getChangedModules", () => {
it("should return the unique set of modules whose files changed", () => {
const result = getChangedModules(nodes, [
"src/utils/colors.ts",
"src/foo/x.ts",
"src/foo/y.ts",
"docs/unrelated.md",
]);
expect([...result].sort()).toEqual(["feature/foo", "lib/utils"]);
});
it("should return an empty set when no file maps to a module", () => {
expect(getChangedModules(nodes, ["docs/foo.md"]).size).toBe(0);
});
});
describe("parseCruiseModules", () => {
it("should flatten to source + resolved-paths and drop unresolved deps", () => {
expect(
parseCruiseModules([
{
source: "src/foo/foo.tsx",
dependencies: [
{ resolved: "src/utils/colors.ts" },
{ resolved: "ambient/types", couldNotResolve: true },
],
},
{ source: "src/bar/bar.tsx" }, // no dependencies key
]),
).toEqual([
{ source: "src/foo/foo.tsx", dependencies: ["src/utils/colors.ts"] },
{ source: "src/bar/bar.tsx", dependencies: [] },
]);
});
});
const FILE_DEPS = [
{ source: "src/app.js", dependencies: ["src/foo/foo.tsx"] },
{ source: "src/bar/bar.tsx", dependencies: ["src/embed.tsx"] },
{ source: "src/foo/foo.tsx", dependencies: ["src/utils/colors.ts"] },
];
const fileGraph = buildFileGraph(ELEMENTS, FILE_DEPS);
describe("getAffectedFiles", () => {
it("should return the changed file and everything that imports it", () => {
expect(
[...getAffectedFiles(fileGraph, ["src/utils/colors.ts"])].sort(),
).toEqual(["src/app.js", "src/foo/foo.tsx", "src/utils/colors.ts"]);
});
it("should include a changed file with no importers (itself only)", () => {
// Nothing imports the app.js entry point.
expect([...getAffectedFiles(fileGraph, ["src/app.js"])]).toEqual([
"src/app.js",
]);
});
});
describe("buildUsageModuleGraph + getAffectedModules", () => {
const usageGraph = buildUsageModuleGraph(ELEMENTS, FILE_DEPS);
it("should expand a changed module to its file-level affected modules", () => {
expect(
[...getAffectedModules(usageGraph, ["src/utils/colors.ts"])].sort(),
).toEqual(["app/main", "feature/foo", "lib/utils"]);
});
it("should NOT pull in feature/bar via the app/main hub", () => {
const affected = getAffectedModules(usageGraph, ["src/foo/foo.tsx"]);
expect([...affected].sort()).toEqual(["app/main", "feature/foo"]);
expect(affected.has("feature/bar")).toBe(false);
});
it("should treat the whole changed module as changed", () => {
// app.js itself has no importers, but app/main also owns embed.tsx, which
// feature/bar imports so changing any app/main file flags feature/bar.
expect(
[...getAffectedModules(usageGraph, ["src/app.js"])].sort(),
).toEqual(["app/main", "feature/bar"]);
});
it("should return empty when no file maps to a module", () => {
expect(getAffectedModules(usageGraph, ["docs/foo.md"]).size).toBe(0);
});
});
describe("getAffectedModules (rules graph)", () => {
it("should affect feature/super and app/main but not feature/bar when feature/foo changes", () => {
const result = getAffectedModules(graph, ["src/foo/x.ts"]);
expect([...result].sort()).toEqual([
"app/main",
"feature/foo",
"feature/super",
]);
});
it("should affect many modules when a lib module changes", () => {
const result = getAffectedModules(graph, ["src/utils/colors.ts"]);
expect(result.has("lib/utils")).toBe(true);
expect(result.has("feature/foo")).toBe(true);
expect(result.has("feature/bar")).toBe(true);
expect(result.has("feature/super")).toBe(true);
expect(result.has("app/main")).toBe(true);
});
it("should return empty when no file maps to a module", () => {
expect(getAffectedModules(graph, ["docs/foo.md"]).size).toBe(0);
});
});
});

View File

@@ -1,230 +0,0 @@
// Builds the affected-tests plan from raw inputs: computes the rules and usage
// affected-module sets and selects, per suite, which specs to run under each.
import { getFeatureModules } from "../../frontend/lint/module-boundaries.mjs";
import {
type FileDependency,
type ModuleDef,
type ModuleNode,
type Rule,
buildModuleGraph,
buildUsageModuleGraph,
getAffectedModules,
getChangedModules,
mapFileToModule,
} from "./affected-modules";
export type TestPlanStats = {
fe_files_changed: number;
fe_files_total: number;
be_files_changed: number;
be_files_total: number;
unit_infra_touched: boolean;
loki_infra_touched: boolean;
shared_sources_touched: boolean;
fe_modules_total: number;
fe_modules_changed: number;
fe_modules_affected_rules: number;
fe_modules_affected_usage: number;
unit_specs_all: number;
unit_specs_to_run_rules: number;
unit_specs_to_run_usage: number;
loki_stories_all: number;
loki_stories_to_run_rules: number;
loki_stories_to_run_usage: number;
e2e_specs_all: number;
e2e_specs_to_run_rules: number;
e2e_specs_to_run_usage: number;
};
export type TestPlan = {
stats: TestPlanStats;
fe_unit_specs_to_run: string[];
loki_stories_to_run: string[];
e2e_specs_to_run: string[];
};
export type CreateTestPlanInput = {
elements: ModuleDef[];
rules: Rule[];
changedFiles: string[];
// Parsed dependency-cruiser edges, or null to fall back to the rules graph.
fileDependencies: FileDependency[] | null;
testFilesBySuite: { unit: string[]; loki: string[]; e2e: string[] };
e2eSpecFiles: Record<string, string[]> | null;
unitInfraTouched: boolean;
lokiInfraTouched: boolean;
e2eInfraTouched: boolean;
sharedSourcesTouched: boolean;
feFilesChanged: number;
beFilesChanged: number;
feFilesTotal: number;
beFilesTotal: number;
};
// Tests whose owning module is affected.
export function filterAffectedTests(
nodes: ModuleNode[],
affected: Set<string>,
testFiles: string[],
): string[] {
return testFiles.filter((file) => {
const module = mapFileToModule(nodes, file);
return module !== null && affected.has(module);
});
}
// Collapse each spec's covered files to the *feature* modules it exercises,
// once, for reuse across the rules/usage selects. Non-feature tiers
// (shared/lib/...) are intentionally dropped: the manifest is a stale nightly
// artifact, so its coupling to the module graph stays minimal ("does this spec
// touch feature X?"). Accuracy comes from the fresh `affected` set, which
// already expands a shared/infra change into the feature modules that depend on
// it.
export function specFeatureModules(
nodes: ModuleNode[],
featureModules: Set<string>,
specFiles: Record<string, string[]>,
): Map<string, Set<string>> {
const moduleCache = new Map<string, string | null>();
const moduleOf = (file: string): string | null => {
let module = moduleCache.get(file);
if (module === undefined) {
module = mapFileToModule(nodes, file);
moduleCache.set(file, module);
}
return module;
};
const result = new Map<string, Set<string>>();
for (const [spec, files] of Object.entries(specFiles)) {
const features = new Set<string>();
for (const file of files) {
const module = moduleOf(file);
if (module !== null && featureModules.has(module)) {
features.add(module);
}
}
result.set(spec, features);
}
return result;
}
// A spec runs if any feature module it covers is affected.
// A spec doesn't map to any feature module always runs because its scope is unknown
export function filterAffectedE2eSpecs(
specFeatures: Map<string, Set<string>>,
affected: Set<string>,
e2eSpecs: string[],
): string[] {
return e2eSpecs.filter((spec) => {
const features = specFeatures.get(spec);
if (!features || features.size === 0) {
return true;
}
return [...features].some((module) => affected.has(module));
});
}
export function createTestPlan({
elements,
rules,
changedFiles,
fileDependencies,
testFilesBySuite,
e2eSpecFiles,
unitInfraTouched,
lokiInfraTouched,
e2eInfraTouched,
sharedSourcesTouched,
feFilesChanged,
beFilesChanged,
feFilesTotal,
beFilesTotal,
}: CreateTestPlanInput): TestPlan {
const rulesGraph = buildModuleGraph(elements, rules);
const usageGraph = fileDependencies
? buildUsageModuleGraph(elements, fileDependencies)
: rulesGraph;
const nodes = rulesGraph.nodes;
// Distinct module types (an element type can span several patterns).
const totalModules = new Set(elements.map((el) => el.type)).size;
const changedModules = getChangedModules(nodes, changedFiles);
const rulesAffected = getAffectedModules(rulesGraph, changedFiles);
const usageAffected = getAffectedModules(usageGraph, changedFiles);
// The coarse "feature" tier is the only set the e2e manifest is ever
// collapsed to (see filterAffectedE2eSpecs).
const featureModules = new Set(getFeatureModules(elements));
// cljc/cljs compile into the FE bundle, so they force a full run that module
// selection can't narrow — same effect as a suite's own infra changing.
const unitForceAll = unitInfraTouched || sharedSourcesTouched;
const lokiForceAll = lokiInfraTouched || sharedSourcesTouched;
// e2e is integration-level, so anything the FE-coverage manifest can't see
// forces a full run: cljc/cljs in the bundle (sharedSourcesTouched), a backend
// change that can break the UI (beFilesChanged), an e2e harness/support change
// (e2eInfraTouched), or no manifest at all.
const e2eForceAll =
sharedSourcesTouched ||
e2eInfraTouched ||
beFilesChanged > 0 ||
e2eSpecFiles === null;
const select = (forceAll: boolean, affected: Set<string>, files: string[]) =>
forceAll ? files : filterAffectedTests(nodes, affected, files);
const { unit, loki, e2e } = testFilesBySuite;
const unitRules = select(unitForceAll, rulesAffected, unit);
const unitUsage = select(unitForceAll, usageAffected, unit);
const lokiRules = select(lokiForceAll, rulesAffected, loki);
const lokiUsage = select(lokiForceAll, usageAffected, loki);
// Precompute spec -> feature modules once (null when e2e runs in full).
const specFeatures =
e2eForceAll || e2eSpecFiles === null
? null
: specFeatureModules(nodes, featureModules, e2eSpecFiles);
// A spec that was itself edited always runs, even when no app module changed.
const changedSet = new Set(changedFiles);
const selectE2e = (affected: Set<string>): string[] => {
if (specFeatures === null) {
return e2e;
}
const narrowed = new Set(
filterAffectedE2eSpecs(specFeatures, affected, e2e),
);
return e2e.filter((spec) => narrowed.has(spec) || changedSet.has(spec));
};
const e2eRules = selectE2e(rulesAffected);
const e2eUsage = selectE2e(usageAffected);
return {
stats: {
fe_files_changed: feFilesChanged,
fe_files_total: feFilesTotal,
be_files_changed: beFilesChanged,
be_files_total: beFilesTotal,
unit_infra_touched: unitInfraTouched,
loki_infra_touched: lokiInfraTouched,
shared_sources_touched: sharedSourcesTouched,
fe_modules_total: totalModules,
fe_modules_changed: changedModules.size,
fe_modules_affected_rules: rulesAffected.size,
fe_modules_affected_usage: usageAffected.size,
unit_specs_all: unit.length,
unit_specs_to_run_rules: unitRules.length,
unit_specs_to_run_usage: unitUsage.length,
loki_stories_all: loki.length,
loki_stories_to_run_rules: lokiRules.length,
loki_stories_to_run_usage: lokiUsage.length,
e2e_specs_all: e2e.length,
e2e_specs_to_run_rules: e2eRules.length,
e2e_specs_to_run_usage: e2eUsage.length,
},
fe_unit_specs_to_run: unitUsage,
loki_stories_to_run: lokiUsage,
e2e_specs_to_run: e2eUsage,
};
}

View File

@@ -1,249 +0,0 @@
import { buildNodes } from "./affected-modules";
import { createTestPlan, filterAffectedTests } from "./affected-tests";
const ELEMENTS = [
{ type: "lib/utils", pattern: "src/utils/**" },
{ type: "feature/foo", pattern: "src/foo/**" },
{ type: "feature/bar", pattern: "src/bar/**" },
{ type: "shared/other", pattern: "src/*/**" },
];
const RULES = [
...ELEMENTS.map((el) => ({ from: [el.type], allow: [el.type] })),
{ from: ["lib/*"], allow: ["lib/*"] },
{ from: ["feature/*"], allow: ["lib/*"] },
];
const UNIT_FILES = [
"src/foo/foo.unit.spec.ts",
"src/foo/bar.unit.spec.ts",
"src/bar/bar.unit.spec.ts",
"src/utils/utils.unit.spec.ts",
];
const LOKI_FILES = ["src/foo/Foo.stories.tsx", "src/bar/Bar.stories.tsx"];
const E2E_FILES = ["e2e/test/scenarios/a.cy.spec.ts"];
const baseInput = {
elements: ELEMENTS,
rules: RULES,
fileDependencies: null,
testFilesBySuite: { unit: UNIT_FILES, loki: LOKI_FILES, e2e: E2E_FILES },
e2eSpecFiles: null,
unitInfraTouched: false,
lokiInfraTouched: false,
e2eInfraTouched: false,
sharedSourcesTouched: false,
feFilesChanged: 0,
beFilesChanged: 0,
feFilesTotal: 0,
beFilesTotal: 0,
};
describe("createTestPlan", () => {
it("runs only specs in affected modules (rules graph)", () => {
const plan = createTestPlan({ ...baseInput, changedFiles: ["src/foo/x.ts"] });
expect(plan.stats.fe_modules_changed).toBe(1);
expect(plan.stats.fe_modules_affected_rules).toBe(1);
expect(plan.fe_unit_specs_to_run.sort()).toEqual([
"src/foo/bar.unit.spec.ts",
"src/foo/foo.unit.spec.ts",
]);
expect(plan.stats.unit_specs_to_run_rules).toBe(2);
expect(plan.stats.loki_stories_to_run_rules).toBe(1);
// With no file deps, usage falls back to rules.
expect(plan.stats.unit_specs_to_run_usage).toBe(2);
});
it("narrows further with the usage graph than the rules graph", () => {
const plan = createTestPlan({
...baseInput,
changedFiles: ["src/utils/colors.ts"],
// Only feature/foo actually imports lib/utils.
fileDependencies: [
{ source: "src/foo/foo.tsx", dependencies: ["src/utils/colors.ts"] },
],
});
// Rules allow both features to import lib/utils, so both are "affected".
expect(plan.stats.fe_modules_affected_rules).toBe(3);
// Usage sees only feature/foo + lib/utils.
expect(plan.stats.fe_modules_affected_usage).toBe(2);
expect(plan.stats.unit_specs_to_run_rules).toBe(4);
expect(plan.stats.unit_specs_to_run_usage).toBe(3);
});
it("forces a full FE run on a shared-sources (cljc) change with no affected modules", () => {
const plan = createTestPlan({
...baseInput,
changedFiles: ["docs/readme.md"],
sharedSourcesTouched: true,
});
expect(plan.stats.fe_modules_affected_rules).toBe(0);
expect(plan.stats.unit_specs_to_run_rules).toBe(UNIT_FILES.length);
expect(plan.stats.unit_specs_to_run_usage).toBe(UNIT_FILES.length);
expect(plan.stats.loki_stories_to_run_rules).toBe(LOKI_FILES.length);
});
it("forces only the suite whose infra changed", () => {
const plan = createTestPlan({
...baseInput,
changedFiles: ["docs/readme.md"],
unitInfraTouched: true,
});
expect(plan.stats.unit_specs_to_run_rules).toBe(UNIT_FILES.length);
expect(plan.stats.loki_stories_to_run_rules).toBe(0);
});
it("runs the full e2e suite when no coverage manifest is available", () => {
const plan = createTestPlan({ ...baseInput, changedFiles: ["src/foo/x.ts"] });
expect(plan.stats.e2e_specs_to_run_rules).toBe(E2E_FILES.length);
expect(plan.stats.e2e_specs_to_run_usage).toBe(E2E_FILES.length);
expect(plan.e2e_specs_to_run).toEqual(E2E_FILES);
});
it("narrows e2e to specs covering affected feature modules", () => {
const e2e = [
"e2e/test/scenarios/foo.cy.spec.ts",
"e2e/test/scenarios/bar.cy.spec.ts",
"e2e/test/scenarios/new.cy.spec.ts",
];
const plan = createTestPlan({
...baseInput,
changedFiles: ["src/foo/x.ts"], // affects feature/foo only
testFilesBySuite: { unit: UNIT_FILES, loki: LOKI_FILES, e2e },
e2eSpecFiles: {
"e2e/test/scenarios/foo.cy.spec.ts": ["src/foo/a.ts"], // feature/foo
"e2e/test/scenarios/bar.cy.spec.ts": ["src/bar/b.ts"], // feature/bar
// new.cy.spec.ts intentionally absent from the manifest
},
});
// foo's feature module is affected; bar's is not; the unmapped new spec
// always runs.
expect(plan.e2e_specs_to_run.sort()).toEqual([
"e2e/test/scenarios/foo.cy.spec.ts",
"e2e/test/scenarios/new.cy.spec.ts",
]);
expect(plan.stats.e2e_specs_all).toBe(3);
});
it("runs a spec that maps to no feature module (unknown scope)", () => {
const e2e = ["e2e/test/scenarios/lib-only.cy.spec.ts"];
const plan = createTestPlan({
...baseInput,
changedFiles: ["src/foo/x.ts"], // affects feature/foo, NOT lib/utils
testFilesBySuite: { unit: UNIT_FILES, loki: LOKI_FILES, e2e },
e2eSpecFiles: {
// Covers only a non-feature module (e.g. a home/auth spec whose feature
// the baseline subtracted), so its feature scope is unknown.
"e2e/test/scenarios/lib-only.cy.spec.ts": ["src/utils/colors.ts"],
},
});
// No feature module to gate on -> can't bound scope -> always runs.
expect(plan.e2e_specs_to_run).toEqual(e2e);
});
it("force-runs all e2e on a backend change (invisible to FE coverage)", () => {
const e2e = ["e2e/test/scenarios/foo.cy.spec.ts"];
const plan = createTestPlan({
...baseInput,
changedFiles: ["src/metabase/api/card.clj"],
beFilesChanged: 1,
testFilesBySuite: { unit: UNIT_FILES, loki: LOKI_FILES, e2e },
e2eSpecFiles: { "e2e/test/scenarios/bar.cy.spec.ts": ["src/bar/b.ts"] },
});
expect(plan.e2e_specs_to_run).toEqual(e2e);
});
it("force-runs all e2e when e2e infra (support/runner) changes", () => {
const e2e = ["e2e/test/scenarios/foo.cy.spec.ts"];
const plan = createTestPlan({
...baseInput,
changedFiles: ["e2e/support/commands/api/createQuestion.js"],
e2eInfraTouched: true,
testFilesBySuite: { unit: UNIT_FILES, loki: LOKI_FILES, e2e },
e2eSpecFiles: { "e2e/test/scenarios/bar.cy.spec.ts": ["src/bar/b.ts"] },
});
expect(plan.e2e_specs_to_run).toEqual(e2e);
});
it("runs an edited spec even when no app module changed", () => {
const spec = "e2e/test/scenarios/foo.cy.spec.ts";
const e2e = [spec, "e2e/test/scenarios/bar.cy.spec.ts"];
const plan = createTestPlan({
...baseInput,
changedFiles: [spec], // only the spec itself changed
testFilesBySuite: { unit: UNIT_FILES, loki: LOKI_FILES, e2e },
e2eSpecFiles: {
[spec]: ["src/bar/b.ts"], // covers feature/bar, which is NOT affected
"e2e/test/scenarios/bar.cy.spec.ts": ["src/bar/b.ts"],
},
});
// No module is affected, but the edited spec must still run.
expect(plan.e2e_specs_to_run).toEqual([spec]);
});
it("runs a spec whose coverage was fully baseline-subtracted (empty files)", () => {
const e2e = ["e2e/test/scenarios/empty.cy.spec.ts"];
const plan = createTestPlan({
...baseInput,
changedFiles: ["src/foo/x.ts"], // affects feature/foo
testFilesBySuite: { unit: UNIT_FILES, loki: LOKI_FILES, e2e },
e2eSpecFiles: { "e2e/test/scenarios/empty.cy.spec.ts": [] },
});
// Empty coverage means unknown scope, so the spec runs.
expect(plan.e2e_specs_to_run).toEqual(e2e);
});
it("forces a full e2e run when shared (cljc) sources change", () => {
const e2e = ["e2e/test/scenarios/foo.cy.spec.ts"];
const plan = createTestPlan({
...baseInput,
changedFiles: ["src/foo/x.ts"],
sharedSourcesTouched: true,
testFilesBySuite: { unit: UNIT_FILES, loki: LOKI_FILES, e2e },
e2eSpecFiles: { "e2e/test/scenarios/bar.cy.spec.ts": ["src/bar/b.ts"] },
});
// Manifest is present but cljc is invisible to JS coverage, so run all.
expect(plan.e2e_specs_to_run).toEqual(e2e);
});
it("passes the FE/BE file counts through to stats", () => {
const plan = createTestPlan({
...baseInput,
changedFiles: [],
feFilesChanged: 5,
beFilesChanged: 3,
});
expect(plan.stats.fe_files_changed).toBe(5);
expect(plan.stats.be_files_changed).toBe(3);
});
});
describe("filterAffectedTests", () => {
it("keeps only tests inside affected modules", () => {
const nodes = buildNodes(ELEMENTS);
const affected = new Set(["feature/foo", "lib/utils"]);
const tests = [
"src/foo/foo.unit.spec.ts",
"src/utils/colors.unit.spec.ts",
"src/bar/bar.unit.spec.ts",
"docs/unrelated.unit.spec.ts",
];
expect(filterAffectedTests(nodes, affected, tests)).toEqual([
"src/foo/foo.unit.spec.ts",
"src/utils/colors.unit.spec.ts",
]);
});
});

View File

@@ -1,225 +0,0 @@
// Prepares the rows the bundle-size stats logger imports into eng-stats-importer.
//
// For each measured (bundle, kind) it computes the delta — raw, gzipped and
// brotli (as-served) bytes plus percent — against the previously *plotted* data
// point (restored from a rolling cache), so the deltas can be charted directly.
// It also decides whether this commit is worth recording at all: a point is kept
// when the as-served size moved by at least MIN_DELTA_PERCENT, or when a new
// bundle/kind appears. Only master builds run through here — releases form
// non-linear lines whose deltas are meaningless, so they're not recorded (their
// historical sizes are backfilled separately).
//
// buildStatsRows is pure (data in, data out) so it can be unit-tested; main()
// is the thin I/O wrapper that reads env/files and writes the rows, cache and
// step outputs.
import {
appendFileSync,
existsSync,
mkdirSync,
readFileSync,
writeFileSync,
} from "node:fs";
import { dirname } from "node:path";
/** One measurement emitted by measure-bundle-sizes. */
export interface Measurement {
bundle: string;
kind: "initial" | "total";
rawBytes: number;
gzipBytes: number;
brotliBytes: number;
fileCount: number;
}
/**
* The slim row cached between runs, holding the last *plotted* point.
* `brotliBytes` is optional because a point cached before brotli logging
* landed does not carry it — see buildStatsRows' per-row gzip fallback.
*/
export type CacheRow = Pick<Measurement, "bundle" | "kind" | "rawBytes" | "gzipBytes"> & {
brotliBytes?: number;
};
export interface BuildStatsRowsInput {
measurements: Measurement[];
previous: CacheRow[];
threshold: number;
date: string;
commit: string;
commitMessage: string;
version: string;
}
/** An upload row, keyed by the importer's column display names. */
export type StatsRow = Record<string, string | number | null>;
export interface BuildStatsRowsResult {
rows: StatsRow[];
cacheRows: CacheRow[];
significant: boolean;
reason: string;
maxServedDeltaPercent: number;
firstPoint: boolean;
hasNewSeries: boolean;
}
// null, not "", for an absent delta: the stats importer coerces values
// strictly and rejects "" for a numeric column.
const deltaPercent = (current: number, base: number | undefined | null): number | null =>
base ? Math.round(((current - base) * 10000) / base) / 100 : null;
/**
* Turn the current measurements + the last plotted point into upload rows, the
* slim cache rows for the next diff, and the keep/skip decision.
*
* The significance threshold tracks the as-served size: brotli where the build
* ships a precompressed .br, otherwise gzip. Current measurements always carry
* brotliBytes (estimated when no .br shipped), but a cached base point recorded
* before brotli logging may not, so the delta still falls back to gzip per row.
*/
export function buildStatsRows({
measurements,
previous,
threshold,
date,
commit,
commitMessage,
version,
}: BuildStatsRowsInput): BuildStatsRowsResult {
const previousOf = (bundle: string, kind: string) =>
previous.find((row) => row.bundle === bundle && row.kind === kind);
let maxServedDeltaPercent = 0;
let hasNewSeries = false;
const rows = measurements.map((measurement): StatsRow => {
const base = previousOf(measurement.bundle, measurement.kind);
// Non-null only when BOTH sides carry brotli: a point cached before brotli
// logging landed has none, and that row falls back to the gzip delta.
const baseBrotli =
measurement.brotliBytes != null && base?.brotliBytes != null ? base.brotliBytes : null;
const gzipDeltaPercent = deltaPercent(measurement.gzipBytes, base?.gzipBytes);
const brotliDeltaPercent =
baseBrotli != null ? deltaPercent(measurement.brotliBytes, baseBrotli) : null;
if (!base) {
hasNewSeries = true;
} else {
const servedDeltaPercent = baseBrotli != null ? brotliDeltaPercent : gzipDeltaPercent;
maxServedDeltaPercent = Math.max(maxServedDeltaPercent, Math.abs(servedDeltaPercent ?? 0));
}
return {
Date: date,
Version: version,
Commit: commit,
Bundle: measurement.bundle,
Kind: measurement.kind, // "initial" or "total"
// The stats table carries a free-text Description column. Populate it
// with the commit subject so the chart's points are self-describing.
Description: commitMessage,
"Raw bytes": measurement.rawBytes,
"Gzip bytes": measurement.gzipBytes,
"Brotli bytes": measurement.brotliBytes, // as-served (shipped .br, else estimated)
"File count": measurement.fileCount,
"Raw bytes delta": base ? measurement.rawBytes - base.rawBytes : null,
"Gzip bytes delta": base ? measurement.gzipBytes - base.gzipBytes : null,
"Brotli bytes delta": baseBrotli != null ? measurement.brotliBytes - baseBrotli : null,
"Raw delta %": deltaPercent(measurement.rawBytes, base?.rawBytes),
"Gzip delta %": gzipDeltaPercent,
"Brotli delta %": brotliDeltaPercent,
};
});
const firstPoint = previous.length === 0;
const significant = firstPoint || hasNewSeries || maxServedDeltaPercent >= threshold;
// Slim rows the next run diffs against. The workflow only persists this to the
// cache when we actually push, so the cached reference always stays the last
// *plotted* point (cumulative drift is caught).
const cacheRows = measurements.map(
({ bundle, kind, rawBytes, gzipBytes, brotliBytes }): CacheRow => ({
bundle,
kind,
rawBytes,
gzipBytes,
brotliBytes,
}),
);
const reason = firstPoint
? "first point"
: hasNewSeries
? "new bundle/kind series"
: `max served Δ ${maxServedDeltaPercent.toFixed(2)}% (threshold ${threshold}%)`;
return {
rows,
cacheRows,
significant,
reason,
maxServedDeltaPercent,
firstPoint,
hasNewSeries,
};
}
const readJson = <T,>(filePath: string | undefined): T | null =>
filePath && existsSync(filePath) ? (JSON.parse(readFileSync(filePath, "utf8")) as T) : null;
const writeJson = (filePath: string, value: unknown) => {
mkdirSync(dirname(filePath), { recursive: true });
writeFileSync(filePath, JSON.stringify(value));
};
const setOutput = (name: string, value: string) => {
if (process.env.GITHUB_OUTPUT) {
appendFileSync(process.env.GITHUB_OUTPUT, `${name}=${value}\n`);
}
};
// version.properties (extracted from the uberjar) carries the build's tag.
// Non-release (master) builds report tag=vUNKNOWN, so leave Version empty there.
function readVersion(versionPropsPath: string | undefined): string {
const raw =
versionPropsPath && existsSync(versionPropsPath)
? (readFileSync(versionPropsPath, "utf8").match(/^tag=(.*)$/m)?.[1]?.trim() ?? "")
: "";
return raw === "vUNKNOWN" ? "" : raw;
}
export function main() {
const env = process.env;
const measurements = readJson<Measurement[]>(env.CURRENT);
if (!Array.isArray(measurements) || measurements.length === 0) {
console.error(`::error::No bundle measurements found at ${env.CURRENT}`);
process.exit(1);
}
const result = buildStatsRows({
measurements,
// The last plotted point, as slim rows: [{ bundle, kind, rawBytes, gzipBytes, brotliBytes }].
previous: readJson<CacheRow[]>(env.LAST) || [],
threshold: Number(env.MIN_DELTA_PERCENT ?? 1),
date: new Date().toISOString().slice(0, 10), // YYYY-MM-DD
commit: (env.HEAD_SHA || "").slice(0, 12),
commitMessage: (env.COMMIT_MESSAGE || "").split("\n")[0], // subject line only
version: readVersion(env.VERSION_PROPS),
});
writeJson(env.ROWS_OUT!, result.rows);
writeJson(env.CACHE_OUT!, result.cacheRows);
console.log(`${result.significant ? "RECORD" : "SKIP"} — ${result.reason}`);
setOutput("significant", result.significant ? "true" : "false");
setOutput(
"max_delta_percent",
result.firstPoint || result.hasNewSeries ? "" : result.maxServedDeltaPercent.toFixed(2),
);
}
// Only run the I/O wrapper when invoked directly, so importing buildStatsRows
// (from the spec) has no side effects.
if ((import.meta as ImportMeta & { main?: boolean }).main) {
main();
}

View File

@@ -1,105 +0,0 @@
import {
buildStatsRows,
type CacheRow,
type Measurement,
} from "./bundle-size-stats-prepare";
const meta = {
date: "2026-06-30",
commit: "abc123",
commitMessage: "Fix the thing (#123)",
version: "",
};
const measurement = (over: Partial<Measurement> = {}): Measurement => ({
bundle: "embedding-sdk-chunked",
kind: "total",
rawBytes: 1000,
gzipBytes: 400,
brotliBytes: 300,
fileCount: 3,
...over,
});
describe("buildStatsRows", () => {
it("records the first point and leaves every delta empty", () => {
const result = buildStatsRows({
...meta,
measurements: [measurement()],
previous: [],
threshold: 1,
});
expect(result.firstPoint).toBe(true);
expect(result.significant).toBe(true);
expect(result.reason).toBe("first point");
expect(result.rows[0]["Gzip bytes delta"]).toBeNull();
expect(result.rows[0]["Brotli delta %"]).toBeNull();
expect(result.rows[0]["Description"]).toBe("Fix the thing (#123)");
expect(result.cacheRows[0]).toEqual({
bundle: "embedding-sdk-chunked",
kind: "total",
rawBytes: 1000,
gzipBytes: 400,
brotliBytes: 300,
});
});
it("records a never-seen bundle/kind as a new series", () => {
const result = buildStatsRows({
...meta,
measurements: [measurement({ bundle: "app", kind: "initial" })],
previous: [{ bundle: "app", kind: "total", rawBytes: 1, gzipBytes: 1, brotliBytes: 1 }],
threshold: 99,
});
expect(result.hasNewSeries).toBe(true);
expect(result.significant).toBe(true);
expect(result.reason).toBe("new bundle/kind series");
});
it("gates significance on the brotli (as-served) delta when both sides have it", () => {
const previous: CacheRow[] = [
{
bundle: "embedding-sdk-chunked",
kind: "total",
rawBytes: 1000,
gzipBytes: 400,
brotliBytes: 300,
},
];
const big = buildStatsRows({
...meta,
measurements: [measurement({ brotliBytes: 315, gzipBytes: 400 })], // brotli +5%, gzip 0%
previous,
threshold: 1,
});
expect(big.maxServedDeltaPercent).toBeCloseTo(5);
expect(big.significant).toBe(true);
expect(big.rows[0]["Brotli delta %"]).toBeCloseTo(5);
const small = buildStatsRows({
...meta,
measurements: [measurement({ brotliBytes: 301, gzipBytes: 999 })], // brotli +0.33%, gzip ignored
previous,
threshold: 1,
});
expect(small.significant).toBe(false);
expect(small.reason).toContain("threshold 1%");
});
it("falls back to the gzip delta when the cached base predates brotli logging", () => {
const result = buildStatsRows({
...meta,
measurements: [measurement({ gzipBytes: 440, brotliBytes: 300 })], // gzip +10%
// No brotliBytes: this point was cached before brotli logging landed.
previous: [{ bundle: "embedding-sdk-chunked", kind: "total", rawBytes: 1000, gzipBytes: 400 }],
threshold: 1,
});
expect(result.maxServedDeltaPercent).toBeCloseTo(10);
expect(result.significant).toBe(true);
expect(result.rows[0]["Brotli bytes delta"]).toBeNull(); // no base brotli to diff against
expect(result.rows[0]["Gzip delta %"]).toBeCloseTo(10);
});
});

View File

@@ -1,145 +0,0 @@
#!/usr/bin/env bash
# Single definition of every CI dependency cache: what it holds, what identifies it, what it falls back to.
#
# A GitHub cache entry is identified by its key, a hash of the declared path list, and the ref that saved
# it. A step declaring a different path list than the one a cache was saved with computes a different
# identity and can never match it, whatever its key says. So paths and keys are emitted together here and
# every cache step takes both from this script.
#
# Emits `name=value` lines in $GITHUB_OUTPUT format:
#
# .github/scripts/cache-keys.sh >> "$GITHUB_OUTPUT"
#
# Run it with no redirect to see the whole scheme, or pass one name to print a single value:
#
# .github/scripts/cache-keys.sh
# .github/scripts/cache-keys.sh m2-key
#
# Hashes are computed here rather than with GitHub's hashFiles() so that a local run and a CI run of the
# same commit produce the same keys, and so that an empty match can fail loudly - a hashFiles() call that
# matches nothing silently yields "", collapsing the key onto its own restore prefix and pointing every
# dependency set at one entry.
set -euo pipefail
cd "$(git rev-parse --show-toplevel)"
OS="${RUNNER_OS:-$(uname -s)}"
[ "$OS" = "Darwin" ] && OS="macOS"
sha256() {
if command -v sha256sum >/dev/null 2>&1; then
sha256sum | cut -d' ' -f1
else
shasum -a 256 | cut -d' ' -f1
fi
}
# Digest of a set of files: hash each one, then hash the concatenation, so the result depends on every
# file's contents and on which files exist.
digest() {
local f
if [ "$#" -eq 0 ]; then
echo "::error::cache-keys.sh: no files matched; refusing to emit a key that would collide across dependency sets" >&2
exit 1
fi
for f in "$@"; do
[ -f "$f" ] || { echo "::error::cache-keys.sh: $f does not exist" >&2; exit 1; }
sha256 < "$f"
done | sha256
}
shopt -s nullglob
DEPS_FILES=(deps.edn modules/drivers/deps.edn modules/drivers/*/deps.edn)
shopt -u nullglob
DEPS_HASH="$(digest "${DEPS_FILES[@]}")"
LOCK_HASH="$(digest bun.lock)"
# The Cypress binary depends on the Cypress version alone, so keying it on the whole lockfile would
# throw away a 200MB download every time any frontend dependency moved.
#
# Reading a version out of a lockfile with a regex is only safe while exactly one line matches. Insist
# on that rather than taking the first of several: a second match means the format moved under us, and
# picking one at random would key the binary under a version it is not.
CYPRESS_VERSIONS="$(sed -n 's/.*"cypress": \["cypress@\([^"]*\)".*/\1/p' bun.lock)"
CYPRESS_MATCHES="$(printf '%s\n' "$CYPRESS_VERSIONS" | grep -c . || true)"
if [ "$CYPRESS_MATCHES" != "1" ]; then
echo "::error::cache-keys.sh: expected exactly one resolved cypress version in bun.lock, found $CYPRESS_MATCHES" >&2
exit 1
fi
CYPRESS_VERSION="$CYPRESS_VERSIONS"
# An incremental lint cache is only ever an accelerator, so it keys on the commit and always falls back
# to the newest entry on the prefix. The exact key never pre-exists, which keeps the entry current.
ESLINT_SHA="${GITHUB_SHA:-$(git rev-parse HEAD)}"
emit_multiline() {
local name="$1"; shift
printf '%s<<CACHE_KEYS_EOF\n' "$name"
printf '%s\n' "$@"
printf 'CACHE_KEYS_EOF\n'
}
# Paths under the home directory are emitted absolute rather than as `~/...`. actions/cache would expand
# a tilde, but a shell assigning one of these to a variable would not, and would go on to create a
# directory named `~`. Paths inside the checkout stay relative; actions/cache resolves those against the
# workspace.
spec() {
emit_multiline m2-path "$HOME/.m2" "$HOME/.gitlibs" "$HOME/.deps.clj" 'bin/bb'
echo "m2-key=M2-$OS-$DEPS_HASH"
echo "m2-restore-key=M2-$OS-"
# Pinned rather than discovered via `bun pm cache` so that the path is known before bun is installed;
# prepare-frontend exports BUN_INSTALL_CACHE_DIR to match and fails if bun disagrees.
echo "bun-store-path=$HOME/.bun/install/cache"
echo "bun-store-key=bun-store-$OS-$LOCK_HASH"
echo "bun-store-restore-key=bun-store-$OS-"
echo "cypress-path=$HOME/.cache/Cypress"
echo "cypress-key=cypress-$OS-$CYPRESS_VERSION"
echo "cypress-restore-key=cypress-$OS-"
echo "eslint-path=.eslintcache"
echo "eslint-key=eslint-$OS-$ESLINT_SHA"
echo "eslint-restore-key=eslint-$OS-"
}
# Print one output's value, including the multiline form, and fail on a name that does not exist. A
# caller that silently received "" would go on to build a key with a hole in it.
query() {
local want="$1" line delim found=0 in_block=0
while IFS= read -r line; do
if [ "$in_block" = 1 ]; then
if [ "$line" = "$delim" ]; then
in_block=0
else
printf '%s\n' "$line"
fi
continue
fi
case "$line" in
"$want="*)
printf '%s\n' "${line#*=}"
found=1
;;
"$want<<"*)
delim="${line#*<<}"
in_block=1
found=1
;;
esac
done <<EOF
$(spec)
EOF
if [ "$found" = 0 ]; then
echo "::error::cache-keys.sh: no output named '$want'" >&2
exit 1
fi
}
if [ "$#" -eq 0 ]; then
spec
else
query "$1"
fi

View File

@@ -1,80 +0,0 @@
import { execFileSync } from "child_process";
import { join } from "path";
const SCRIPT = join(__dirname, "cache-keys.sh");
const CACHES = ["m2", "bun-store", "cypress", "eslint"];
// stderr is inherited by default, so the deliberate-failure case below would print a literal
// `::error::` line - which GitHub Actions reads as an annotation directive, making a passing run
// annotate itself with an error.
const run = (args: string[] = [], env: NodeJS.ProcessEnv = {}, quiet = false) =>
execFileSync(SCRIPT, args, {
encoding: "utf8",
stdio: quiet ? ["ignore", "pipe", "ignore"] : "pipe",
env: { ...process.env, RUNNER_OS: "Linux", GITHUB_SHA: "abc123", ...env },
});
const get = (name: string, env: NodeJS.ProcessEnv = {}) =>
run([name], env).trimEnd();
describe("cache-keys.sh", () => {
it("is deterministic for a given tree", () => {
expect(run()).toEqual(run());
});
it("emits every cache's path, key and restore key", () => {
for (const cache of CACHES) {
expect(get(`${cache}-path`)).not.toEqual("");
expect(get(`${cache}-key`)).not.toEqual("");
expect(get(`${cache}-restore-key`)).not.toEqual("");
}
});
// Silently returning "" for a name that does not exist is how a caller ends up building a key with a
// hole in it.
it("fails on an unknown output name", () => {
expect(() => run(["no-such-output"], {}, true)).toThrow();
});
// A key equal to its own restore prefix points every dependency set at one entry, and nothing reports
// it. This is the failure the script exists to prevent.
it("never emits a key equal to its own restore prefix", () => {
for (const cache of CACHES) {
const key = get(`${cache}-key`);
const prefix = get(`${cache}-restore-key`);
expect(key.startsWith(prefix)).toBe(true);
expect(key.length).toBeGreaterThan(prefix.length);
}
});
it("scopes every key to the runner OS", () => {
for (const cache of CACHES) {
expect(get(`${cache}-key`, { RUNNER_OS: "Linux" })).not.toEqual(
get(`${cache}-key`, { RUNNER_OS: "macOS" }),
);
}
});
// Home paths must be absolute: actions/cache expands a tilde but a shell assigning one to a variable
// does not, and would create a directory literally named `~`.
it("emits absolute paths for home directory locations", () => {
expect(get("bun-store-path").startsWith("/")).toBe(true);
expect(get("cypress-path").startsWith("/")).toBe(true);
for (const line of get("m2-path").split("\n")) {
expect(line.startsWith("~")).toBe(false);
}
});
// The Cypress binary depends on the Cypress version alone, so an unrelated dependency bump must not
// invalidate a 200MB download.
it("keys the Cypress binary on the resolved version, not the lockfile", () => {
expect(get("cypress-key")).toMatch(/^cypress-Linux-\d+\.\d+\.\d+/);
});
it("keys the ESLint cache on the commit", () => {
expect(get("eslint-key", { GITHUB_SHA: "deadbeef" })).toEqual(
"eslint-Linux-deadbeef",
);
});
});

View File

@@ -1,69 +0,0 @@
#!/usr/bin/env bash
# Fails when a `clojure` invocation under .github/ uses an alias combination that cache-warm does not
# pre-resolve.
#
# tools.deps selects dependency versions globally per classpath, so two alias combinations can resolve
# the same transitive library to different versions. A combination the warm job never resolved is
# therefore not merely absent from the cache - the cache holds *a* version of those libraries, just not
# the one this classpath wants, and the job fetches its delta straight from Maven Central on every run.
# That is the intermittent-403 failure of DEV-2094, and it is invisible: the build still passes.
#
# Nothing about a missing combination shows up at runtime, so this check is the only thing standing
# between a newly added alias combination and a permanently half-cold cache.
#
# .github/scripts/check-preresolve-aliases.sh
set -euo pipefail
cd "$(git rev-parse --show-toplevel)"
WARM=.github/workflows/cache-warm.yml
# Invocations deliberately left out of the warm job. Each needs a reason, because the default answer is
# to add the combination to the list instead.
WAIVED="$(
cat <<'EOF'
:deps
:outdated
EOF
)"
# :deps - tools.deps built-in used by write-poms.sh; contributes no dependencies of its own.
# :outdated - scheduled dependency-bump workflow only; a cold resolve there costs nothing anyone waits on.
# Alias combinations the warm job pre-resolves.
declared="$(sed -n '/combos=(/,/^ *)/p' "$WARM" | grep -oE '":[^"]*"' | tr -d '"' | sort -u)"
if [ -z "$declared" ]; then
echo "::error::check-preresolve-aliases.sh: found no combos=( ... ) list in $WARM" >&2
exit 1
fi
# Expressions we know how to expand. An unrecognised one has to stop the check rather than be skipped:
# silently ignoring it is how a whole matrix of combinations would go unchecked.
unknown="$(grep -rhoE '\$\{\{[^}]*\}\}' --include='*' \
--exclude='*.png' --exclude='*.jpg' .github/ 2>/dev/null \
| sort -u | grep -v 'edition' || true)"
scan() { # scan <edition>: alias strings used by clojure invocations, with expressions substituted
grep -rhE 'clojure[[:space:]]+(-P[[:space:]]+)?-[XMATP]?:' \
--include='*' --exclude='*.png' --exclude='*.jpg' .github/ 2>/dev/null \
| sed "s/\\\${{[^}]*edition[^}]*}}/$1/g" \
| grep -oE 'clojure[[:space:]]+(-P[[:space:]]+)?-[XMATP]?:[A-Za-z0-9:_.-]+' \
| grep -oE ':[A-Za-z0-9:_.-]+' \
| sed 's/:$//'
}
found="$( { scan ee; scan oss; } | sort -u )"
covered="$(printf '%s\n%s\n' "$declared" "$WAIVED" | sort -u)"
missing="$(comm -23 <(printf '%s\n' "$found") <(printf '%s\n' "$covered") || true)"
if [ -n "$missing" ]; then
echo "::error::Alias combinations used under .github/ that cache-warm does not pre-resolve:" >&2
printf '%s\n' "$missing" | sed 's/^/ /' >&2
echo >&2
echo "Add each to the combos list in $WARM, or waive it in this script with a reason." >&2
echo "Leaving it uncovered means those jobs re-fetch their version deltas from Maven on every run." >&2
exit 1
fi
echo "All $(printf '%s\n' "$found" | grep -c .) clojure alias combinations under .github/ are pre-resolved."

View File

@@ -0,0 +1,109 @@
const TYPES = new Set([
"feat",
"fix",
"perf",
"docs",
"style",
"refactor",
"test",
"build",
"ci",
]);
/**
* @param title {string}
*/
function isConventionalTitle(title) {
const match = /^(?<type>\w+)\((?<rawScopes>[^)]+)\):/.exec(title);
if (!match) {
return false;
}
const [, type, rawScopes] = match;
const scopes = rawScopes
.split(",")
.map((scope) => scope.trim())
.filter(Boolean);
return TYPES.has(type) && scopes.length > 0;
}
/** @param {import('@types/github-script').AsyncFunctionArguments} AsyncFunctionArguments */
async function updateCommentForEmbedding({ github, context }) {
const githubUsername = context.payload.sender.login;
const comment = `${CONVENTIONAL_COMMENT_IDENTIFIER}
@${githubUsername} You have modified embedding code. Please make sure the PR title follows [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) style.
Here are all the supported types:
- \`feat\`
- \`fix\`
- \`perf\`
- \`docs\`
- \`style\`
- \`refactor\`
- \`test\`
- \`build\`
- \`ci\`
Please also make sure to include a scope.
For example, these are valid PR titles:
- \`feat(sdk-bundle): Add interactive dashboard component\`
- \`fix(sdk-package): Fixed MetabaseProviderPropsStore logic\`
- \`fix(sdk-package, embed-js): Fix API\``;
return updateComment(comment, { github, context });
}
/**
* @param comment {string}
* @param {import('@types/github-script').AsyncFunctionArguments} AsyncFunctionArguments
*/
async function updateComment(comment, { github, context }) {
const owner = context.repo.owner;
const repo = context.repo.repo;
const pullRequestNumber = context.issue.number;
const comments = await github.rest.issues.listComments({
owner,
repo,
issue_number: pullRequestNumber,
});
const existingComment = getExistingConventionalCommitReminderComment(
comments.data,
);
if (existingComment) {
return await github.rest.issues.updateComment({
owner,
repo,
comment_id: existingComment.id,
body: comment,
});
} else {
return await github.rest.issues.createComment({
owner,
repo,
issue_number: pullRequestNumber,
body: comment,
});
}
}
const CONVENTIONAL_COMMENT_IDENTIFIER =
"<!---conventional comment identifier-->";
/**
* @param comments {{body: string}[]}
*/
function getExistingConventionalCommitReminderComment(comments) {
for (const comment of comments) {
if (comment.body.startsWith(CONVENTIONAL_COMMENT_IDENTIFIER)) {
return comment;
}
}
}
exports.isConventionalTitle = isConventionalTitle;
exports.updateCommentForEmbedding = updateCommentForEmbedding;

View File

@@ -0,0 +1,45 @@
const { isConventionalTitle } = require("./utils");
describe("isConventionalTitle", () => {
it("returns true for a simple conventional commit", () => {
expect(isConventionalTitle("feat: add new feature")).toBe(false);
});
it("returns true for a commit with a scope", () => {
expect(isConventionalTitle("fix(api): correct API bug")).toBe(true);
});
it("returns true for a commit with multiple scopes, comma separated", () => {
expect(isConventionalTitle("feat(build,deps): update dependencies")).toBe(
true,
);
});
it("returns true for a commit with multiple scopes, comma separated with space", () => {
expect(isConventionalTitle("feat(build, deps): update dependencies")).toBe(
true,
);
});
it("returns false for a commit with round brackets in a message", () => {
expect(isConventionalTitle("feat: (something) update dependencies")).toBe(
false,
);
});
it("returns false for missing colon", () => {
expect(isConventionalTitle("feat add feature")).toBe(false);
});
it("returns false for invalid type", () => {
expect(isConventionalTitle("invalidType: some message")).toBe(false);
});
it("returns false for empty message", () => {
expect(isConventionalTitle("feat:")).toBe(false);
});
it("returns false for empty input", () => {
expect(isConventionalTitle("")).toBe(false);
});
});

View File

@@ -1,112 +0,0 @@
// I/O entrypoint: gather inputs (env vars, the cruise graph, the test-file
// lists) and hand them to createTestPlan, which does the computing.
import { execFileSync } from "node:child_process";
import { existsSync, readFileSync } from "node:fs";
import micromatch from "micromatch";
import { elements, rules } from "../../frontend/lint/module-boundaries.mjs";
import { type FileDependency, parseCruiseModules } from "./affected-modules";
import { createTestPlan } from "./affected-tests";
import { listSpecFiles } from "./e2e-spec-globs.mjs";
const UNIT_ROOTS = ["frontend/src", "enterprise/frontend/src"];
const UNIT_GLOBS = [
"frontend/src/**/*.unit.spec.{js,jsx,ts,tsx}",
"enterprise/frontend/src/**/*.unit.spec.{js,jsx,ts,tsx}",
];
const STORY_ROOTS = ["frontend", "enterprise/frontend"];
const STORY_GLOBS = [
"frontend/**/*.stories.{js,jsx,ts,tsx}",
"enterprise/frontend/**/*.stories.{js,jsx,ts,tsx}",
];
// Returns the tracked files under `roots` that match `globs`. The `dot: true`
// option means files inside dot-directories such as `.storybook` are included.
function listFiles(roots: string[], globs: string[]): string[] {
const tracked = execFileSync("git", ["ls-files", "--", ...roots], {
encoding: "utf8",
})
.split("\n")
.map((line) => line.trim())
.filter(Boolean);
return micromatch(tracked, globs, { dot: true });
}
// dorny outputs comma-separated lists; CHANGED_FILES is `all_changed_files`.
const csvToList = (csv: string | undefined) =>
(csv ?? "")
.split(",")
.map((s) => s.trim())
.filter(Boolean);
// Reads the dependency-cruiser graph (DEP_GRAPH_JSON). Null falls back to the
// rules graph, so a missing or unparseable file never breaks the plan.
function readFileDependencies(): FileDependency[] | null {
const path = process.env.DEP_GRAPH_JSON;
if (path && existsSync(path)) {
try {
const { modules } = JSON.parse(readFileSync(path, "utf8"));
process.stderr.write(`Using usage graph from ${path}.\n`);
return parseCruiseModules(modules);
} catch (error) {
process.stderr.write(
`Failed to read ${path}; falling back to rules graph: ${error}\n`,
);
}
} else {
process.stderr.write(
"No DEP_GRAPH_JSON found; falling back to rules graph.\n",
);
}
return null;
}
// Reads the nightly coverage manifest (E2E_SPEC_MANIFEST): { builtAt, specs:
// { spec -> files } }. Returns the specs map, or null so e2e falls back to a
// full run when the manifest is missing/unparseable.
function readE2eSpecFiles(): Record<string, string[]> | null {
const path = process.env.E2E_SPEC_MANIFEST;
if (path && existsSync(path)) {
try {
const { specs } = JSON.parse(readFileSync(path, "utf8"));
if (specs && typeof specs === "object") {
process.stderr.write(`Using e2e coverage manifest from ${path}.\n`);
return specs;
}
} catch (error) {
process.stderr.write(
`Failed to read ${path}; e2e will run in full: ${error}\n`,
);
}
} else {
process.stderr.write("No E2E_SPEC_MANIFEST found; e2e will run in full.\n");
}
return null;
}
const testPlan = createTestPlan({
elements,
rules,
changedFiles: csvToList(process.env.CHANGED_FILES),
fileDependencies: readFileDependencies(),
testFilesBySuite: {
unit: listFiles(UNIT_ROOTS, UNIT_GLOBS),
loki: listFiles(STORY_ROOTS, STORY_GLOBS),
e2e: listSpecFiles(),
},
e2eSpecFiles: readE2eSpecFiles(),
unitInfraTouched: process.env.UNIT_INFRA_TOUCHED === "true",
lokiInfraTouched: process.env.LOKI_INFRA_TOUCHED === "true",
e2eInfraTouched: process.env.E2E_INFRA_TOUCHED === "true",
sharedSourcesTouched: process.env.SHARED_SOURCES_TOUCHED === "true",
feFilesChanged: csvToList(process.env.FE_CHANGED_FILES).length,
beFilesChanged: csvToList(process.env.BE_CHANGED_FILES).length,
feFilesTotal: listFiles(["frontend", "enterprise/frontend"], ["**"]).length,
beFilesTotal: listFiles(["src", "enterprise/backend"], ["**"]).length,
});
process.stdout.write(JSON.stringify(testPlan) + "\n");

41
.github/scripts/csv-to-mb.js vendored Normal file
View File

@@ -0,0 +1,41 @@
/**
* jsonData must be in the format:
* [
* { column1: value1, column2: value2, ... },
* { column1: value3, column2: value4, ... },
* ...
* ]
*/
function jsonToCsvFormData(jsonData) {
const formData = new FormData();
const csvHeader = Object.keys(jsonData[0]).join(',') + '\n';
const csvContent = jsonData.map(row =>
Object.values(row).map(value => `"${String(value).replace(/"/g, '""')}"`).join(',')
).join('\n');
const blob = new Blob([csvHeader + csvContent], { type: 'text/csv' });
formData.append('file', blob, 'data.csv');
return formData;
}
function uploadCsvToMb({ baseUrl, tableId, jsonData, mode = 'append' }) {
const formData = jsonToCsvFormData(jsonData);
const operation = mode === 'replace' ? 'replace-csv' : 'append-csv';
return fetch(`${baseUrl}/api/table/${tableId}/${operation}`, {
method: 'POST',
headers: {
"x-api-key": process.env.API_KEY
},
body: formData
})
.then(response => {
if (!response.ok) {
throw new Error(`Upload failed`);
}
return { success: true };
});
}
module.exports = { uploadCsvToMb };

View File

@@ -1,32 +0,0 @@
import { execFileSync } from "node:child_process";
import micromatch from "micromatch";
const SPEC_DIR = "e2e/test/scenarios";
// This is the nightly-only baseline helper that runs in the instrumented pass to
// capture boot-time coverage. It is not a product spec, so it is excluded from
// the test plan universe and from the manifest backfill reconciliation.
export const BASELINE_SPEC = `${SPEC_DIR}/coverage-baseline.cy.spec.js`;
// This glob set defines the spec "universe". The test planner
// (create-test-plan.ts) selects which specs to run from it, and the coverage
// manifest builder (build-coverage-manifest.mjs) reconciles against it when
// backfilling specs that did not run.
export const E2E_SPEC_GLOBS = [`${SPEC_DIR}/**/*.cy.spec.{js,jsx,ts,tsx}`];
// Returns the tracked e2e specs as repo-relative paths, excluding the baseline
// helper.
export function listSpecFiles(cwd = process.cwd()) {
const tracked = execFileSync("git", ["ls-files", "--", SPEC_DIR], {
cwd,
encoding: "utf8",
})
.split("\n")
.map((line) => line.trim())
.filter(Boolean);
return micromatch(tracked, E2E_SPEC_GLOBS, {
ignore: [BASELINE_SPEC],
dot: true,
});
}

40
.github/scripts/find-linear-issue.js vendored Normal file
View File

@@ -0,0 +1,40 @@
async function findLinearIssue({ githubIssueUrl, linearApiKey }) {
const response = await fetch("https://api.linear.app/graphql", {
method: "POST",
headers: {
"Content-Type": "application/json",
Authorization: linearApiKey,
},
body: JSON.stringify({
query: `
query {
attachmentsForURL(url: "${githubIssueUrl}") {
nodes {
issue { identifier }
}
}
}
`,
}),
});
const data = await response.json();
if (!response.ok) {
return { success: false, message: `Linear API error: ${response.status}` };
}
if (!data.data?.attachmentsForURL?.nodes?.length) {
return {
success: false,
message: `No Linear issue linked to ${githubIssueUrl}`,
};
}
return {
success: true,
issue: data.data.attachmentsForURL.nodes[0].issue.identifier,
};
}
module.exports = { findLinearIssue };

View File

@@ -1,66 +0,0 @@
/**
* Renames aliased dependency keys in a bun-generated yarn.lock.
*
* Yarn names an alias entry after the alias, for example
* `"typescript7@npm:typescript@7.0.2"`. Bun names it after the aliased package
* instead, for example `"typescript@npm:typescript@7.0.2"`. Snyk follows the
* yarn convention and fails to resolve the dependency.
*/
import { readFile, writeFile } from "node:fs/promises";
async function main() {
const packageJsonPath = `${process.cwd()}/package.json`;
const lockPath = `${process.cwd()}/yarn.lock`;
const packageJson = JSON.parse(await readFile(packageJsonPath, "utf8"));
const dependencies = Object.entries<string>(
Object.assign(
{},
packageJson.dependencies,
packageJson.devDependencies,
packageJson.optionalDependencies,
),
);
let lock = await readFile(lockPath, "utf8");
const renamedKeys: string[] = [];
for (const [alias, specifier] of dependencies) {
if (!specifier.startsWith("npm:")) {
continue;
}
const aliasedPackage = specifier
.slice("npm:".length)
.replace(/@[^@]*$/, "");
if (aliasedPackage === alias) {
continue;
}
const bunKey = `"${aliasedPackage}@${specifier}"`;
const yarnKey = `"${alias}@${specifier}"`;
if (lock.includes(bunKey)) {
lock = lock.replaceAll(bunKey, yarnKey);
renamedKeys.push(`${bunKey} -> ${yarnKey}`);
}
}
if (renamedKeys.length > 0) {
await writeFile(lockPath, lock);
}
process.stdout.write(
renamedKeys.length > 0
? `Renamed alias keys in yarn.lock:\n${renamedKeys.join("\n")}\n`
: "No alias keys to rename in yarn.lock\n",
);
}
main().catch((error) => {
process.stderr.write(`Failed to rename alias keys in yarn.lock: ${error}\n`);
process.exit(1);
});

84
.github/scripts/get-issue-data.js vendored Normal file
View File

@@ -0,0 +1,84 @@
const githubSlackMap = require("../../release/github-slack-map.json");
async function getIssueData({ context, github, manualIssueNumber }) {
let issueNumber;
let issueData;
let prReviews = [];
const triggerType = context.eventName;
if (context.eventName === "issues") {
issueNumber = context.payload.issue.number;
issueData = context.payload.issue;
} else if (context.eventName === "workflow_dispatch") {
issueNumber = parseInt(manualIssueNumber);
const { data } = await github.rest.issues.get({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: issueNumber,
});
issueData = data;
} else if (context.eventName === "pull_request_review") {
const prNumber = context.payload.pull_request.number;
const { data: prData } = await github.rest.pulls.get({
owner: context.repo.owner,
repo: context.repo.repo,
pull_number: prNumber,
});
// Extract issue number from branch name (e.g., "claude-fix/issue-12345-some-title")
const branchMatch = prData.head.ref.match(/claude-fix\/issue-(\d+)/);
if (!branchMatch) {
throw new Error(
`Could not extract issue number from branch name: ${prData.head.ref}`,
);
}
issueNumber = parseInt(branchMatch[1]);
const { data } = await github.rest.issues.get({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: issueNumber,
});
issueData = data;
const { data: reviews } = await github.rest.pulls.listReviews({
owner: context.repo.owner,
repo: context.repo.repo,
pull_number: prNumber,
});
prReviews = reviews.filter(
(review) =>
review.state === "CHANGES_REQUESTED" &&
isMetabaseMember(review.user.login),
);
}
const { data: comments } = await github.rest.issues.listComments({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: issueNumber,
});
const employeeComments = comments.filter((comment) =>
isMetabaseMember(comment.user.login),
);
return {
issueNumber,
issueData,
employeeComments,
prReviews,
triggerType,
isIssueAuthorEmployee: isMetabaseMember(issueData.user.login),
};
}
function isMetabaseMember(username) {
return username in githubSlackMap;
}
module.exports = {
getIssueData,
};

View File

@@ -1,109 +0,0 @@
async function findLinearIssue({ githubIssueUrl, linearApiKey }) {
const response = await fetch("https://api.linear.app/graphql", {
method: "POST",
headers: {
"Content-Type": "application/json",
Authorization: linearApiKey,
},
body: JSON.stringify({
query: `
query {
attachmentsForURL(url: "${githubIssueUrl}") {
nodes {
issue { identifier }
}
}
}
`,
}),
});
const data = await response.json();
if (!response.ok) {
return { success: false, message: `Linear API error: ${response.status}` };
}
if (!data.data?.attachmentsForURL?.nodes?.length) {
return {
success: false,
message: `No Linear issue linked to ${githubIssueUrl}`,
};
}
return {
success: true,
issue: data.data.attachmentsForURL.nodes[0].issue.identifier,
};
}
async function createLinearIssue({
teamId,
title,
description,
projectId,
labelIds,
linearApiKey,
}) {
const response = await fetch("https://api.linear.app/graphql", {
method: "POST",
headers: {
"Content-Type": "application/json",
Authorization: linearApiKey,
},
body: JSON.stringify({
query: `
mutation CreateIssue($input: IssueCreateInput!) {
issueCreate(input: $input) {
success
issue {
id
identifier
title
url
}
}
}
`,
variables: {
input: {
teamId,
title,
...(description ? { description } : {}),
...(projectId ? { projectId } : {}),
...(labelIds?.length ? { labelIds } : {}),
},
},
}),
});
if (!response.ok) {
return { success: false, message: `Linear API error: ${response.status}` };
}
const data = await response.json();
if (data.errors?.length) {
return {
success: false,
message: `Linear API error: ${data.errors.map((e) => e.message).join(", ")}`,
};
}
if (!data.data?.issueCreate?.success) {
return {
success: false,
message: `Failed to create Linear issue "${title}"`,
};
}
return {
success: true,
issue: data.data.issueCreate.issue,
};
}
module.exports = {
findLinearIssue,
createLinearIssue,
};

View File

@@ -1,258 +0,0 @@
/* eslint-disable import/no-commonjs */
// Measures the front-end bundles emitted into an extracted uberjar tree.
//
// Every bundle reports one or both of:
// - "initial": JS the browser downloads on first paint (the entry chunk plus
// its initial/sync split chunks). Never excludes initial chunks.
// - "total": all *reachable* JS (initial plus async chunks that are actually
// loadable at runtime). Never includes unreachable/never-fetched chunks.
// Reachability needs the per-entry asset list from the rspack stats file, so the
// app and chunked SDK bundles require their stats file to be present.
//
// The pure pieces — the asset-name selection (initial vs reachable, runtime
// filtering, the async-loadable flag) — are split out as select*Assets so they
// can be unit-tested against a stats object. The measure* functions and main()
// keep all the filesystem and env I/O, threading an explicit `config`.
const fs = require("fs");
const path = require("path");
const zlib = require("zlib");
const APP_DIST = "resources/frontend_client/app/dist";
const SDK_ROOT = "resources/frontend_client/app/embedding-sdk";
// The chunk runtime is inlined into the bootstrap, never sent as a standalone file.
const SDK_RUNTIME_RE = /embedding-sdk-chunk-runtime\./;
const notRuntime = name => !SDK_RUNTIME_RE.test(name);
const assetFiles = (root, names) => names.map(name => path.join(root, name));
const unique = files => [...new Set(files)];
// MEASURE_ROOT lets callers point at an extracted tree (e.g. the PR bundle-size
// check measures a current and a base-ref tree); defaults to cwd for the stats
// workflow. MEASURE_BUNDLES restricts which bundles are measured (comma-separated).
// SDK_ASYNC_CHUNKS_LOADABLE (see selectChunkedSdkAssets) toggles whether async
// SDK chunks count toward "total".
function buildConfig(env) {
const baseDir = env.MEASURE_ROOT ? path.resolve(env.MEASURE_ROOT) : process.cwd();
const bundleFilter = env.MEASURE_BUNDLES
? new Set(env.MEASURE_BUNDLES.split(",").map(bundle => bundle.trim()))
: null;
return {
resolve: (...relativeParts) => path.resolve(baseDir, ...relativeParts),
isBundleWanted: bundle => !bundleFilter || bundleFilter.has(bundle),
// Async SDK chunks are reachable once output.publicPath is set so import()
// chunks are actually fetched at runtime — which landed in master. Defaults
// to true to match master; set SDK_ASYNC_CHUNKS_LOADABLE=false to measure an
// older commit built before the publicPath fix (async unreachable, so
// "total" collapses to "initial").
sdkAsyncChunksLoadable: env.SDK_ASYNC_CHUNKS_LOADABLE !== "false",
};
}
function sizeOf(files) {
let rawBytes = 0;
let gzipBytes = 0;
let brotliBytes = 0;
for (const file of files) {
const contents = fs.readFileSync(file);
rawBytes += contents.length;
// gzip: recompressed, a proxy for the on-the-fly gzip served on every version
gzipBytes += zlib.gzipSync(contents).length;
// brotli: as-served — count the precompressed .br the build ships; files
// without one (static-viz, sub-threshold chunks) are served compressed on
// the fly, so estimate with brotli. Always reported (like gzip).
const brotliPath = `${file}.br`;
brotliBytes += fs.existsSync(brotliPath)
? fs.statSync(brotliPath).size
: zlib.brotliCompressSync(contents).length;
}
return {
fileCount: files.length,
rawBytes,
gzipBytes,
brotliBytes,
};
}
function collectJsFiles(absolutePath) {
if (fs.statSync(absolutePath).isFile()) {
return [absolutePath];
}
return fs
.readdirSync(absolutePath, { withFileTypes: true })
.flatMap(entry => {
const childPath = path.join(absolutePath, entry.name);
if (entry.isDirectory()) {
return collectJsFiles(childPath);
}
return entry.name.endsWith(".js") ? [childPath] : [];
});
}
function readStats(config, statsFile) {
const statsPath = config.resolve(statsFile);
return fs.existsSync(statsPath) ? JSON.parse(fs.readFileSync(statsPath, "utf8")) : null;
}
// An entrypoint's .js assets for a given field: "assets" = its initial
// (synchronous) load; "reachableAssets" = initial plus everything it can lazily
// import (the chunk graph walked at build time in bundle-stats.js).
function entrypointJsAssets(stats, name, field = "assets") {
const entry = stats && stats.entrypoints && stats.entrypoints[name];
if (!entry || !entry[field]) {
return null;
}
return entry[field].map(asset => asset.name || asset).filter(assetName => assetName.endsWith(".js"));
}
/**
* Main app = the app-main entrypoint only (app/dist also holds other pages like
* app-public, app-embed*, which are not the app). `reachableAssets` is null on
* older stats built before the chunk-graph enrichment — the caller then falls
* back to sizing the whole app/dist.
*/
function selectAppAssets(stats) {
const initialAssets = entrypointJsAssets(stats, "app-main", "assets");
if (!initialAssets) {
throw new Error('app: stats-main.json / "app-main" entrypoint missing');
}
return { initialAssets, reachableAssets: entrypointJsAssets(stats, "app-main", "reachableAssets") };
}
/**
* Legacy SDK: the "embedding-sdk" entry, a single eager file in legacy/. It still
* emits on-demand chunks (jspdf, echarts, leaflet, ...) into the shared chunks/
* dir via dynamic import(). initial = the eager legacy file; total = initial plus
* its reachable async chunks, but only when those are loadable at runtime
* (sdkAsyncChunksLoadable), otherwise total collapses to initial. Returns null
* when the stats lack the entry (older layout) so the caller can fall back.
*/
function selectLegacySdkAssets(stats, { sdkAsyncChunksLoadable }) {
const initialNames = entrypointJsAssets(stats, "embedding-sdk", "assets");
if (!initialNames) {
return null;
}
const reachableNames = entrypointJsAssets(stats, "embedding-sdk", "reachableAssets") || initialNames;
const initial = unique(initialNames).filter(notRuntime);
return {
initialNames: initial,
reachableNames: sdkAsyncChunksLoadable ? unique(reachableNames).filter(notRuntime) : initial,
};
}
/**
* Chunked SDK delivery. initial = bootstrap (minus its inlined runtime) + the
* chunked entry's initial chunks. total = the chunked entry's reachable set
* (initial + async children, from the chunk graph). includesAsyncChunks is true
* only when the stats carry the reachable chunk graph AND those chunks are
* loadable at runtime (sdkAsyncChunksLoadable). A base ref built before the
* reachableAssets enrichment has no graph, so its "total" must collapse to
* "initial" — the flag lets the comparison avoid measuring a reachable total
* against a collapsed one (an apples-to-oranges ~30% phantom).
*/
function selectChunkedSdkAssets(stats, { sdkAsyncChunksLoadable }) {
const chunkedInitial = entrypointJsAssets(stats, "embedding-sdk-chunked", "assets");
if (!chunkedInitial) {
throw new Error('embedding-sdk-chunked: stats-embedding-sdk.json / "embedding-sdk-chunked" entrypoint missing');
}
const bootstrapInitial = entrypointJsAssets(stats, "embedding-sdk-bootstrap", "assets") || [];
const chunkedReachableAssets = entrypointJsAssets(stats, "embedding-sdk-chunked", "reachableAssets");
const chunkedReachable = chunkedReachableAssets || chunkedInitial;
const bootstrapReachable = entrypointJsAssets(stats, "embedding-sdk-bootstrap", "reachableAssets") || bootstrapInitial;
return {
initialNames: unique([...bootstrapInitial, ...chunkedInitial]).filter(notRuntime),
reachableNames: unique([...bootstrapReachable, ...chunkedReachable]).filter(notRuntime),
includesAsyncChunks: chunkedReachableAssets != null && sdkAsyncChunksLoadable,
};
}
function measureApp(config) {
const dist = config.resolve(APP_DIST);
if (!fs.existsSync(dist)) {
throw new Error(`Bundle output not found for "app": ${dist}`);
}
const { initialAssets, reachableAssets } = selectAppAssets(readStats(config, "artifacts/stats-main.json"));
if (!reachableAssets) {
console.warn("app: stats has no reachableAssets; app total falls back to the whole app/dist");
}
const totalFiles = reachableAssets ? assetFiles(dist, reachableAssets) : collectJsFiles(dist);
return [
{ bundle: "app", kind: "initial", ...sizeOf(assetFiles(dist, initialAssets)) },
{ bundle: "app", kind: "total", ...sizeOf(totalFiles) },
];
}
function measureLegacySdk(config) {
const absRoot = config.resolve(SDK_ROOT);
const selected = selectLegacySdkAssets(readStats(config, "artifacts/stats-embedding-sdk.json"), config);
if (!selected) {
// No stats / older layout: fall back to whatever sits in the legacy dir.
const candidates = [`${SDK_ROOT}/legacy`, `${SDK_ROOT}.js`];
const chosen = candidates.find(candidate => fs.existsSync(config.resolve(candidate)));
if (!chosen) {
console.warn(`Skipping "embedding-sdk-legacy": output not found (${candidates.join(", ")})`);
return [];
}
return [{ bundle: "embedding-sdk-legacy", kind: "total", ...sizeOf(collectJsFiles(config.resolve(chosen))) }];
}
return [
{ bundle: "embedding-sdk-legacy", kind: "initial", ...sizeOf(assetFiles(absRoot, selected.initialNames)) },
{ bundle: "embedding-sdk-legacy", kind: "total", ...sizeOf(assetFiles(absRoot, selected.reachableNames)) },
];
}
function measureChunkedSdk(config) {
const chunksDir = config.resolve(SDK_ROOT, "chunks");
if (!fs.existsSync(chunksDir)) {
// pre-chunked layout (monolith only) — nothing to measure here
return [];
}
const absRoot = config.resolve(SDK_ROOT);
const { initialNames, reachableNames, includesAsyncChunks } = selectChunkedSdkAssets(
readStats(config, "artifacts/stats-embedding-sdk.json"),
config,
);
const initial = sizeOf(assetFiles(absRoot, initialNames));
const total = includesAsyncChunks ? sizeOf(assetFiles(absRoot, reachableNames)) : initial;
return [
{ bundle: "embedding-sdk-chunked", kind: "initial", ...initial },
{ bundle: "embedding-sdk-chunked", kind: "total", reachable: includesAsyncChunks, ...total },
];
}
// Single-file/dir bundles loaded in one shot — total only.
function measureWholeDir(config, bundle, candidates) {
const chosen = candidates.find(candidate => fs.existsSync(config.resolve(candidate)));
if (!chosen) {
console.warn(`Skipping "${bundle}": output not found (${candidates.join(", ")})`);
return [];
}
return [{ bundle, kind: "total", ...sizeOf(collectJsFiles(config.resolve(chosen))) }];
}
const bundles = {
app: measureApp,
"embedding-sdk-legacy": measureLegacySdk,
"embedding-sdk-chunked": measureChunkedSdk,
"embedding-sdk-package": config => measureWholeDir(config, "embedding-sdk-package", ["resources/embedding-sdk/dist"]),
"embed-js": config => measureWholeDir(config, "embed-js", ["resources/frontend_client/app/embed.js"]),
};
function measureBundleSizes(config) {
return Object.entries(bundles)
.filter(([bundle]) => config.isBundleWanted(bundle))
.flatMap(([, measure]) => measure(config));
}
module.exports = {
measureBundleSizes,
buildConfig,
entrypointJsAssets,
selectAppAssets,
selectLegacySdkAssets,
selectChunkedSdkAssets,
};
if (require.main === module) {
console.log(JSON.stringify(measureBundleSizes(buildConfig(process.env)), null, 2));
}

View File

@@ -1,87 +0,0 @@
const {
entrypointJsAssets,
selectAppAssets,
selectLegacySdkAssets,
selectChunkedSdkAssets,
} = require("./measure-bundle-sizes");
const RUNTIME = "embedding-sdk-chunk-runtime.deadbeef.js";
const stats = entrypoints => ({ entrypoints });
describe("entrypointJsAssets", () => {
it("returns the .js asset names for a field, accepting strings or {name}", () => {
const s = stats({ app: { assets: [{ name: "a.js" }, "b.js", { name: "c.css" }] } });
expect(entrypointJsAssets(s, "app", "assets")).toEqual(["a.js", "b.js"]);
});
it("returns null when the entrypoint or field is absent", () => {
expect(entrypointJsAssets(stats({}), "missing")).toBeNull();
expect(entrypointJsAssets(null, "app")).toBeNull();
});
});
describe("selectAppAssets", () => {
it("returns the initial set and a null reachable set on pre-enrichment stats", () => {
const s = stats({ "app-main": { assets: ["main.js"] } });
expect(selectAppAssets(s)).toEqual({ initialAssets: ["main.js"], reachableAssets: null });
});
it("throws when the app-main entrypoint is missing", () => {
expect(() => selectAppAssets(stats({}))).toThrow("app-main");
});
});
describe("selectChunkedSdkAssets", () => {
const base = {
"embedding-sdk-chunked": { assets: ["a.js", RUNTIME], reachableAssets: ["a.js", "b.js", RUNTIME] },
"embedding-sdk-bootstrap": { assets: ["boot.js"], reachableAssets: ["boot.js"] },
};
it("merges bootstrap + chunked, drops the inlined runtime, and counts async chunks", () => {
const result = selectChunkedSdkAssets(stats(base), { sdkAsyncChunksLoadable: true });
expect(result.initialNames).toEqual(["boot.js", "a.js"]);
expect(result.reachableNames).toEqual(["boot.js", "a.js", "b.js"]);
expect(result.includesAsyncChunks).toBe(true);
});
it("collapses total to initial when async chunks are not loadable", () => {
const result = selectChunkedSdkAssets(stats(base), { sdkAsyncChunksLoadable: false });
expect(result.includesAsyncChunks).toBe(false);
});
it("collapses total to initial when the stats carry no reachable graph", () => {
const noGraph = { "embedding-sdk-chunked": { assets: ["a.js"] } };
const result = selectChunkedSdkAssets(stats(noGraph), { sdkAsyncChunksLoadable: true });
expect(result.reachableNames).toEqual(["a.js"]);
expect(result.includesAsyncChunks).toBe(false);
});
it("throws when the chunked entrypoint is missing", () => {
expect(() => selectChunkedSdkAssets(stats({}), { sdkAsyncChunksLoadable: true })).toThrow(
"embedding-sdk-chunked",
);
});
});
describe("selectLegacySdkAssets", () => {
it("returns null when the legacy entry is absent so the caller can fall back", () => {
expect(selectLegacySdkAssets(stats({}), { sdkAsyncChunksLoadable: true })).toBeNull();
});
it("expands total to the reachable async set when loadable", () => {
const s = stats({ "embedding-sdk": { assets: ["legacy.js"], reachableAssets: ["legacy.js", "chunk.js"] } });
expect(selectLegacySdkAssets(s, { sdkAsyncChunksLoadable: true })).toEqual({
initialNames: ["legacy.js"],
reachableNames: ["legacy.js", "chunk.js"],
});
});
it("collapses total to initial when async chunks are not loadable", () => {
const s = stats({ "embedding-sdk": { assets: ["legacy.js"], reachableAssets: ["legacy.js", "chunk.js"] } });
expect(selectLegacySdkAssets(s, { sdkAsyncChunksLoadable: false })).toEqual({
initialNames: ["legacy.js"],
reachableNames: ["legacy.js"],
});
});
});

Some files were not shown because too many files have changed in this diff Show More