Compare commits

..

4 Commits

Author SHA1 Message Date
Nicola Mometto
5a43b2d35f further branch 2026-02-03 14:19:38 +00:00
Nicola Mometto
832f8ac44d wip: force run 2026-02-03 13:47:51 +00:00
Nicola Mometto
88bdc6c06c test other branch 2026-02-03 13:47:41 +00:00
Nicola Mometto
bdfc24938c wip: test pyarrow parsing 2026-01-28 09:54:26 +00:00
7084 changed files with 420625 additions and 605574 deletions

View File

@@ -1 +0,0 @@
Run mutation testing for $ARGUMENTS

View File

@@ -1,21 +1,21 @@
## Linting and Formatting
- **Lint:** `bun run lint-eslint-pure`
- **Lint:** `yarn lint-eslint-pure`
- Run ESLint on the codebase
- **Format:** `bun run prettier`
- **Format:** `yarn prettier`
- Format code using Prettier
- **Type Check:** `bun run type-check-pure`
- **Type Check:** `yarn type-check-pure`
- Run TypeScript type checking
## Testing
### JavaScript/TypeScript Tests
- **Test a specific file:** `bun run test-unit-keep-cljs path/to/file.unit.spec.js`
- **Test by pattern:** `bun run test-unit-keep-cljs -t "pattern"`
- **Test a specific file:** `yarn test-unit-keep-cljs path/to/file.unit.spec.js`
- **Test by pattern:** `yarn test-unit-keep-cljs -t "pattern"`
- Runs tests matching the given pattern
### ClojureScript Tests
- **Test ClojureScript:** `bun run test-cljs`
- **Test ClojureScript:** `yarn test-cljs`
- Run ClojureScript tests

View File

@@ -40,7 +40,7 @@ allowed-tools: Read, Write, Grep, Bash, Glob
**Format:**
- Run prettier on the file after making edits: `bun run prettier --write <file-path>`
- Run prettier on the file after making edits: `yarn prettier --write <file-path>`
- This ensures consistent formatting across all documentation
### Common patterns

View File

@@ -1,114 +0,0 @@
---
name: mutation-testing
description: Run mutation testing on a Clojure namespace, generate tests to kill surviving mutations, and open draft PRs with Linear issue tracking.
---
# Mutation Testing Skill
This skill runs mutation testing end-to-end: generates a coverage report, groups functions, shells out to `claude -p` to write tests, verifies mutations are killed, and creates draft PRs with Linear issues.
## Prerequisites
- A running nREPL connected to the Metabase dev environment
- `LINEAR_API_KEY` environment variable set with a valid Linear personal API key
- `gh` CLI authenticated with GitHub
- `claude` CLI available on PATH
## Reference Files
- `dev/src/dev/coverage.clj` — mutation testing engine (generates reports, runs mutations)
- `dev/src/dev/mutation_testing.clj` — orchestration, Linear API, PR helpers
## Invocation
The argument is a Clojure namespace, optionally followed by a base branch:
```
/mutation-testing metabase.lib.order-by
/mutation-testing metabase.lib.order-by --base-branch release-x.52.x
/mutation-testing metabase.lib.order-by --project-id abc-123
/mutation-testing metabase.lib.order-by --base-branch release-x.52.x --project-id abc-123
```
- `--base-branch`: defaults to the value in config (set via `set-config!`), or `"master"` if not configured.
- `--project-id`: if provided, issues are added to this existing Linear project instead of creating a new one.
## Steps
### 1. Parse arguments
Parse `$ARGUMENTS` to extract:
- **namespace** (required) — the first positional argument
- **`--base-branch`** (optional) — if present, the next argument is the base branch name
- **`--project-id`** (optional) — if present, the next argument is an existing Linear project ID
### 2. Load and configure
```clojure
(require '[dev.mutation-testing :as mut-test] :reload)
(require '[dev.coverage :as cov] :reload)
```
If this is the first invocation (or the REPL was restarted), set up the Linear team:
```clojure
(mut-test/list-teams!)
(mut-test/set-config! {:team-id "<id>"})
```
### 3. Run
```clojure
(mut-test/run! '<target-ns>)
;; Or with options:
(mut-test/run! '<target-ns> {:base-branch "release-x.52.x"})
(mut-test/run! '<target-ns> {:project-id "abc-123"})
(mut-test/run! '<target-ns> {:base-branch "release-x.52.x" :project-id "abc-123"})
```
Pass the opts map with `:base-branch` and/or `:project-id` if those flags were provided in the arguments.
This will:
1. Generate a baseline mutation testing report
2. Create a Linear project for the namespace
3. Group functions by coverage relationships
4. For each group: create branch → invoke Claude to write tests → verify → retry if needed → commit & push → create Linear issue → create draft PR
5. Print a summary with PR links
### 4. Handle failures
If a group fails mid-processing, `run!` catches the error and continues to the next group. To retry a single group manually:
```clojure
;; Get the parsed namespace info
(def parsed (mut-test/parse-namespace '<target-ns>))
;; Run coverage to get the data
(def coverage-results (cov/test-namespace (:target-ns parsed) [(:test-ns parsed)]))
;; Group and find the one you want
(def groups (mut-test/group-functions coverage-results))
;; Process just that group
(mut-test/process-group! parsed (nth groups <index>))
```
### 5. Verify the prompt (dry run)
To inspect what Claude will see without invoking it:
```clojure
(def parsed (mut-test/parse-namespace '<target-ns>))
(def coverage-results (cov/test-namespace (:target-ns parsed) [(:test-ns parsed)]))
(def groups (mut-test/group-functions coverage-results))
(println (mut-test/build-test-prompt
(merge (select-keys parsed [:target-ns :test-ns :source-path :test-path])
(select-keys (first groups) [:fn-names :mutations]))))
```
## Configuration
- **Team ID**: Set once per REPL session via `(mut-test/list-teams!)` and `(mut-test/set-config! {:team-id "..."})`
- **Project ID**: Set automatically when `run!` calls `create-project-for-namespace!`
- **Base Branch**: Defaults to `"master"`. Override via `(mut-test/set-config! {:base-branch "release-x.52.x"})` or pass `{:base-branch "..."}` as the second arg to `run!`
- **Project ID**: If set (via `set-config!` or `run!` opts), `run!` reuses the existing Linear project instead of creating a new one

View File

@@ -12,9 +12,8 @@ allowed-tools: Read, Grep, Bash, Glob
Review pull requests with a focus on:
- Readability and maintainability
- Appropriate test coverage
- Compliance with project coding standards and conventions
- Code quality and best practices
- Clear and correct JSDoc comments
- Type safety and proper TypeScript usage
- React best practices (when applicable)
- React best practices (when applicable)

View File

@@ -1,7 +1,7 @@
;; -*- comment-column: 100; -*-
{:config-paths ["macros" "src" "config"]
:linters
{;;
{ ;;
;; enabled optional linters (these are disabled by default)
;;
@@ -50,15 +50,13 @@
;; disabled linters
;;
:redundant-ignore {:level :off} ; lots of false positives, see https://github.com/clj-kondo/clj-kondo/issues/2414
:redundant-ignore {:level :off} ; lots of false positives, see https://github.com/clj-kondo/clj-kondo/issues/2414
:unexpected-recur {:level :off} ; TODO (cam): I think we just need to tell it how to handle MBQL match and we can enable this?
;;
;; globally-enabled custom linters
;;
:metabase/check-defmulti-arglists {:level :warning}
:metabase/check-def-check-not-uppercase-name {:level :warning}
:metabase/check-def-no-underscores {:level :warning}
:metabase/check-me-humanize {:level :warning}
:metabase/defsetting-must-specify-export {:level :warning}
:metabase/defsetting-namespace {:level :warning}
@@ -69,6 +67,7 @@
:metabase/namespace-name {:level :warning}
:metabase/no-jsqlparser-imports {:level :warning}
:metabase/require-shape-checker {:level :warning}
:metabase/test-helpers-use-non-thread-safe-functions {:level :warning} ; more config in `.clj-kondo/config/test-helpers/config.edn`
:metabase/tests-must-live-in-known-modules {:level :warning}
:metabase/validate-defendpoint-route-uses-kebab-case {:level :warning}
:metabase/validate-defendpoint-query-params-use-kebab-case {:level :warning}
@@ -180,11 +179,6 @@
metabase.notification.models/create-notification!
metabase.pulse.send/send-pulse!
metabase-enterprise.transforms-python.python-runner-test/execute!
;; Workspace isolation functions are safe in parallel tests because they operate on
;; isolated, test-specific workspaces and databases that don't affect other tests
metabase-enterprise.workspaces.isolation/destroy-workspace-isolation!
metabase-enterprise.workspaces.isolation/ensure-database-isolation!
metabase-enterprise.workspaces.test-util/create-ready-ws!
;; Safe in tests, where we typically use a private, temporary index per thread.
metabase.search.core/reindex!
metabase.util/run-count!
@@ -198,7 +192,7 @@
:metabase/disallow-hardcoded-driver-names-in-tests
;; don't include `:h2` since that will probably lead to a mountain of tests where we are specifically testing
;; middleware or compilation behavior with just H2 since it's the default.
{:drivers #{;; core drivers
{:drivers #{ ;; core drivers
#_:h2
:postgres
:mysql
@@ -226,10 +220,7 @@
;; misc config for built-in linters
;;
:invalid-arity {:skip-args [metabase.lib.util.match/match
metabase.lib.util.match/match-lite
metabase.lib.util.match/match-many
metabase.lib.util.match/replace-lite]} ; TODO (cam): can we fix these?
:invalid-arity {:skip-args [metabase.lib.util.match/match]} ; TODO (cam): can we fix these?
:refer-all {:level :warning, :exclude [clojure.test]}
:unused-referred-var {:exclude {compojure.core [GET DELETE POST PUT]}}
@@ -248,17 +239,13 @@
(metabase.lib.util.match/match)
(metabase.lib.util.match/match-one)
(metabase.lib.util.match/match-lite)
(metabase.lib.util.match/match-many)
(metabase.lib.util.match/match-lite-recursive)
(metabase.lib.util.match/replace)
(metabase.lib.util.match/replace-in)
(metabase.lib.util.match/replace-lite)
(metabase.driver-api.core/match)
(metabase.driver-api.core/match-one)
(metabase.driver-api.core/match-lite)
(metabase.driver-api.core/match-many)
(metabase.driver-api.core/replace)
(metabase.driver-api.core/replace-in)
(metabase.driver-api.core/replace-lite)
(metabase.query-processor.middleware.cache-backend.interface/with-cached-results)
(metabase.util.regex/rx [opt])
(metabase.util/prog1 [<>])
@@ -310,14 +297,11 @@
metabase.lib.convert/->legacy-MBQL {:message "Conversion to legacy MBQL is discouraged, please use Lib to manipulate MBQL queries"}
metabase.lib.core/->legacy-MBQL {:message "Conversion to legacy MBQL is discouraged, please use Lib to manipulate MBQL queries"}
metabase.lib.equality/find-column-indexes-for-refs {:message "Use lib.equality/closest-matches-in-metadata or lib.equality/find-closest-matches-for-refs instead of lib.equality/find-column-indexes-for-refs"}
metabase.lib.util.match/match {:message "Use lib.util.match/match-many instead of lib.util.match/match."}
metabase.lib.util.match/match-one {:message "Use lib.util.match/match-lite instead of lib.util.match/match-one."}
metabase.lib.util.match/replace {:message "Use lib.util.match/replace-lite instead of lib.util.match/replace."}
metabase-enterprise.transforms.interface/execute! {:message "Use metabase-enterprise.transforms.execute/execute! instead of metabase-enterprise.transforms.interface/execute!"}
next.jdbc/active-tx? {:message "Use metabase.app-db.core/in-transaction? to see if we are in a transaction"}
toucan2.core/debug {:message "Don't commit code using t2/debug"}
toucan2.tools.debug/debug {:message "Don't commit code using t2/debug"}
}
macaw.core/parsed-query {:message "Use metabase.driver.util/parsed-query instead of macaw.core/parsed-query"}}
:discouraged-namespace
{camel-snake-kebab.core {:message "CSK is not Turkish-safe, use the versions in metabase.util instead."}
@@ -436,10 +420,11 @@
metabase-enterprise.content-translation.routes-test/with-new-secret-key! clojure.core/fn
metabase-enterprise.serialization.test-util/with-dbs clojure.core/fn
metabase-enterprise.serialization.test-util/with-random-dump-dir clojure.core/let
metabase-enterprise.workspaces.test-util/with-resources! clojure.core/let
metabase-enterprise.workspaces.test-util/with-workspaces! clojure.core/let
metabase-enterprise.transforms.api-test/with-transform-tags clojure.core/let
metabase-enterprise.transforms.api-test/with-worker-runs clojure.core/let
metabase-enterprise.transforms.test-util/with-transform-cleanup! clojure.core/let
metabase-enterprise.transforms.execute/with-transform-lifecycle clojure.core/let
metabase.actions.test-util/with-actions clojure.core/let
metabase.transforms.test-util/with-transform-cleanup! clojure.core/let
metabase.api.common/let-404 clojure.core/let
metabase.app-db.custom-migrations.util/with-temp-scheduler! clojure.core/fn
metabase.app-db.custom-migrations/define-migration clj-kondo.lint-as/def-catch-all
@@ -527,16 +512,7 @@
{cljs.test/deftest hooks.clojure.test/deftest
cljs.test/is hooks.clojure.test/is
cljs.test/use-fixtures hooks.clojure.test/use-fixtures
clojure.core/def hooks.clojure.core.def/lint-def
clojure.core/definterface hooks.clojure.core.def/lint-def
clojure.core/defmacro hooks.clojure.core.defmacro/lint-defmacro
clojure.core/defmulti hooks.clojure.core.defmulti/lint-defmulti
clojure.core/defn hooks.clojure.core.defn/lint-defn
clojure.core/defn- hooks.clojure.core.defn/lint-defn
clojure.core/defonce hooks.clojure.core.def/lint-def
clojure.core/defprotocol hooks.clojure.core.def/lint-def
clojure.core/defrecord hooks.clojure.core.def/lint-def
clojure.core/deftype hooks.clojure.core.def/lint-def
clojure.core/ns hooks.clojure.core.ns/lint-ns
clojure.core/require hooks.clojure.core.require/lint-require
clojure.core/requiring-resolve hooks.clojure.core.require/lint-requiring-resolve
@@ -786,17 +762,22 @@
:config-in-ns
{test-namespaces
{:linters
{:inline-def {:level :off}
:missing-docstring {:level :off}
:private-call {:level :off}
:metabase/defsetting-must-specify-export {:level :off}
:metabase/defsetting-namespace {:level :off}
:metabase/test-helpers-use-non-thread-safe-functions {:level :warning} ; more config in `.clj-kondo/config/test-helpers/config.edn
{:inline-def {:level :off}
:missing-docstring {:level :off}
:private-call {:level :off}
:metabase/defsetting-must-specify-export {:level :off}
:metabase/defsetting-namespace {:level :off}
:discouraged-var
{metabase.driver/database-supports? {:level :off}
metabase.lib.convert/->legacy-MBQL {:level :off}
metabase.lib.core/->legacy-MBQL {:level :off}}}}
metabase.lib.core/->legacy-MBQL {:level :off}}}
:hooks
{:analyze-call
{clojure.core/defmacro hooks.clojure.core.defn/non-thread-safe-form-should-end-with-exclamation
clojure.core/defn hooks.clojure.core.defn/non-thread-safe-form-should-end-with-exclamation
clojure.core/defn- hooks.clojure.core.defn/non-thread-safe-form-should-end-with-exclamation}}}
source-namespaces
{:linters

View File

@@ -276,7 +276,6 @@
events
models
premium-features
request
settings
util}}
@@ -395,7 +394,6 @@
request
revisions
settings
transforms
upload
util}}
@@ -410,6 +408,7 @@
models
request
users
users-rest
util}}
;; technically this 'uses' `enterprise/core` and `test` since it tries to load them to see if they exists so we know
@@ -501,18 +500,6 @@
util
xrays}}
data-studio
{:team "Gadget"
:api #{metabase.data-studio.api}
:uses #{api
database-routing
driver
events
request
sync
util
warehouse-schema}}
database-routing
{:team "UX West"
:api #{metabase.database-routing.core}
@@ -542,13 +529,11 @@
view-log}}
driver
{:team "Querying"
{:team "Drivers"
:api #{metabase.driver
metabase.driver.common
metabase.driver.common.parameters
metabase.driver.common.parameters.parse
metabase.driver.connection
metabase.driver.connection.workspaces
metabase.driver.ddl.interface
metabase.driver.h2
metabase.driver.impl
@@ -557,22 +542,15 @@
metabase.driver.postgres
metabase.driver.settings
metabase.driver.sql
metabase.driver.sql-jdbc
metabase.driver.sql-jdbc.connection
metabase.driver.sql-jdbc.execute
metabase.driver.sql-jdbc.execute.diagnostic
metabase.driver.sql-jdbc.sync
;; TODO (lbrdnk 2025/12/17): We should revisit addition of the following line. I've added it to make
;; tests pass as we are using it directly with ee/workspaces module. I believe later, when workspaces
;; are in better state quality wise, it should become part of the driver api and that should be used
;; instead of calling directly into the normalize module.
metabase.driver.sql.normalize
metabase.driver.sql.query-processor
metabase.driver.sql.util
metabase.driver.sync
metabase.driver.util}
:uses #{analytics
app-db
:uses #{app-db
appearance
auth-provider
classloader
@@ -585,14 +563,13 @@
premium-features
query-processor
settings
sql-tools
system
util
warehouse-schema
warehouses}}
driver-api
{:team "Querying"
{:team "Drivers"
:api #{metabase.driver-api.core}
:uses #{actions
api
@@ -646,7 +623,6 @@
:api #{metabase.embedding-rest.api}
:uses #{api
dashboards
database-routing
eid-translation
embedding
events
@@ -736,18 +712,32 @@
lib
{:team "Querying"
:api #{metabase.lib.core
:api #{metabase.lib.binning
metabase.lib.convert ; Only so [[metabase.queries.models.card]] can set dynamic vars.
metabase.lib.core
metabase.lib.equality
metabase.lib.field
metabase.lib.filter.desugar.jvm ; only so [[metabase.legacy-mbql.jvm-util]] can use defined regexes
metabase.lib.hierarchy
metabase.lib.init
metabase.lib.metadata
metabase.lib.metadata.cached-provider
metabase.lib.metadata.column
metabase.lib.metadata.invocation-tracker
metabase.lib.metadata.protocols
metabase.lib.metadata.result-metadata
metabase.lib.normalize
metabase.lib.options
metabase.lib.query
metabase.lib.schema
metabase.lib.schema.actions
metabase.lib.schema.aggregation
metabase.lib.schema.binning
metabase.lib.schema.common
metabase.lib.schema.constraints
metabase.lib.schema.expression
metabase.lib.schema.expression.temporal
metabase.lib.schema.expression.window
metabase.lib.schema.id
metabase.lib.schema.info
metabase.lib.schema.join
@@ -756,26 +746,25 @@
metabase.lib.schema.measure
metabase.lib.schema.metadata
metabase.lib.schema.metadata.fingerprint
metabase.lib.schema.middleware-options
metabase.lib.schema.parameter
metabase.lib.schema.ref
metabase.lib.schema.settings
metabase.lib.schema.template-tag
metabase.lib.schema.temporal-bucketing
metabase.lib.schema.test-spec
metabase.lib.schema.util
metabase.lib.schema.validate
metabase.lib.temporal-bucket
metabase.lib.types.isa
metabase.lib.util
metabase.lib.util.match}
metabase.lib.util.match
metabase.lib.walk}
:uses #{config
graph
legacy-mbql
types
util}
:friends #{legacy-mbql ;; Reuses internal schemas and a few helper functions. Deeply coupled to MBQL.
lib-be ;; BE-specific extension of lib; tight coupling is acceptable.
query-processor ;; Necessarily manipulates MBQL; better to pull most logic into a new module.
enterprise/sandbox ;; Really part of the QP, though its lives in the enterprise/ tree.
}}
:friends #{query-processor}}
lib-be
{:team "Querying"
@@ -786,24 +775,6 @@
settings
util}}
llm
{:team "Metabot"
:api #{metabase.llm.api
metabase.llm.init}
:uses #{analytics
api
driver
lib
lib-be
models
premium-features
request
settings
sql-tools
sync
util
warehouse-schema}}
logger
{:team "DevEx"
:api #{metabase.logger.api
@@ -875,19 +846,10 @@
:uses #{api
app-db
classloader
collections
events
legacy-mbql
lib
lib-be
permissions
query-processor
remote-sync
search
settings
transforms
util}
:friends #{transforms}}
util}}
native-query-snippets
{:team "Querying"
@@ -900,7 +862,6 @@
models
permissions
premium-features
remote-sync
util}}
notification
@@ -918,7 +879,6 @@
config
dashboards
driver
embedding
events
models
parameters
@@ -943,7 +903,6 @@
metabase.parameters.custom-values
metabase.parameters.dashboard
metabase.parameters.field
metabase.parameters.field-values
metabase.parameters.init
metabase.parameters.params
metabase.parameters.schema
@@ -1010,15 +969,13 @@
:api #{metabase.premium-features.api
metabase.premium-features.core
metabase.premium-features.init}
:uses #{analytics
api
:uses #{api
app-db
classloader
config
events
internal-stats
settings
startup
task
util}}
@@ -1174,10 +1131,8 @@
metabase.query-processor.init
metabase.query-processor.interface
metabase.query-processor.metadata
metabase.query-processor.middleware.add-remaps
metabase.query-processor.middleware.annotate
metabase.query-processor.middleware.cache-backend.db
metabase.query-processor.middleware.catch-exceptions
metabase.query-processor.middleware.constraints
metabase.query-processor.middleware.limit
metabase.query-processor.middleware.permissions
@@ -1245,8 +1200,7 @@
:api #{metabase.request.core
metabase.request.init
metabase.request.schema}
:uses #{analytics
api
:uses #{api
config
embedding
permissions ; FIXME
@@ -1310,11 +1264,11 @@
settings
startup
task
transforms
util}}
util
warehouses}}
secrets
{:team "Querying"
{:team "Drivers"
:api #{metabase.secrets.core}
:uses #{api driver models premium-features util}}
@@ -1423,23 +1377,6 @@
system
util}}
sql-parsing
{:team "DevEx"
:api #{metabase.sql-parsing.core}
:uses #{analytics
util}}
sql-tools
{:team "DevEx"
:api #{metabase.sql-tools.core
metabase.sql-tools.init}
:uses #{analytics
driver
lib
settings
sql-parsing
util}}
sso
{:team "UX West"
:api #{metabase.sso.api
@@ -1539,8 +1476,6 @@
api
app-db
config
lib
lib-be
premium-features
search
util}}
@@ -1569,65 +1504,6 @@
models
util}}
transforms
{:team "Querying"
:api #{metabase.transforms.core
metabase.transforms.feature-gating
metabase.transforms.init
metabase.transforms.interface
metabase.transforms.schema
metabase.transforms.util}
:uses #{analytics
api
channel
database-routing
driver
events
lib
lib-be
models
premium-features
queries
query-processor
revisions
settings
sync
task
task-history
util}
:friends #{models
enterprise/transforms-python
enterprise/workspaces}}
transforms-inspector
{:team "Gadget"
:api #{metabase.transforms-inspector.core
metabase.transforms-inspector.schema}
:uses #{driver
lib
lib-be
query-processor
sql-tools
transforms
util}}
transforms-rest
{:team "Gadget"
:api #{metabase.transforms-rest.api.transform
metabase.transforms-rest.api.transform-job
metabase.transforms-rest.api.transform-tag}
:uses #{api
driver
events
models
query-processor
request
server
sql-tools
transforms
transforms-inspector
util}}
types
{:team "Querying"
:api #{metabase.types.core
@@ -1638,7 +1514,6 @@
{:team "Querying"
:api #{metabase.upload.api
metabase.upload.core
metabase.upload.db
metabase.upload.init}
:uses #{analytics
api
@@ -1666,42 +1541,41 @@
users
{:team "UX West"
:api #{metabase.users.core
metabase.users.init
:api #{metabase.users.init
metabase.users.models.user
metabase.users.models.user-parameter-value
metabase.users.schema
metabase.users.settings}
:uses #{analytics
api
:uses #{api
batch-processing
channel
config
events
models
notification
permissions
premium-features
settings
setup
system
tenants
util
enterprise/advanced-permissions}}
util}}
users-rest
{:team "UX West"
:api #{metabase.users-rest.api}
:uses #{api
:uses #{analytics
api
appearance
auth-identity
collections
config
events
models
notification
permissions
premium-features
request
settings
sso
tenants
users
@@ -1755,6 +1629,7 @@
api
app-db
audit-app
config
driver
lib
lib-be
@@ -1901,7 +1776,6 @@
metabase-enterprise.advanced-permissions.common
metabase-enterprise.advanced-permissions.models.permissions.group-manager}
:uses #{api
audit-app
lib
permissions
premium-features
@@ -1912,17 +1786,6 @@
warehouses
enterprise/impersonation}}
enterprise/agent-api
{:team "Metabot"
:api #{metabase-enterprise.agent-api.api}
:uses #{api
auth-identity
query-processor
request
server
util
enterprise/metabot-v3}}
enterprise/ai-entity-analysis
{:team "Metabot"
:api #{metabase-enterprise.ai-entity-analysis.api}
@@ -1992,7 +1855,7 @@
setup
startup
sync
users
users-rest
util
enterprise/serialization}}
@@ -2034,14 +1897,6 @@
util
enterprise/harbormaster}}
enterprise/cloud-proxy
{:team "Cloud"
:api #{metabase-enterprise.cloud-proxy.api}
:uses #{api
premium-features
util
enterprise/harbormaster}}
enterprise/content-translation
{:team "UX West"
:api #{metabase-enterprise.content-translation.routes}
@@ -2071,15 +1926,20 @@
:uses #{premium-features}}
enterprise/data-studio
{:team "UX West"
{:team "Gadget"
:api #{metabase-enterprise.data-studio.api}
:uses #{api
collections
database-routing
driver
events
models
permissions
premium-features
util}}
request
sync
util
warehouse-schema}}
enterprise/database-replication
{:team "Cloud"
@@ -2107,7 +1967,7 @@
settings
util
warehouse-schema
warehouses}}
warehouses-rest}}
enterprise/dependencies
{:team "Querying"
@@ -2135,8 +1995,8 @@
revisions
settings
task
transforms
util}}
util
enterprise/transforms}}
enterprise/email
{:team "UX West"
@@ -2178,7 +2038,7 @@
util}}
enterprise/impersonation
{:team "Querying"
{:team "Drivers"
:api #{metabase-enterprise.impersonation.api}
:uses #{api
audit-app
@@ -2218,46 +2078,44 @@
util}}
enterprise/metabot-v3
{:team "Metabot"
:api #{metabase-enterprise.metabot-v3.api
metabase-enterprise.metabot-v3.client
metabase-enterprise.metabot-v3.core
metabase-enterprise.metabot-v3.init
metabase-enterprise.metabot-v3.tools.api}
:friends #{enterprise/agent-api}
:uses #{activity-feed
api
app-db
audit-app
collections
config
documents
driver
legacy-mbql
lib
lib-be
models
native-query-snippets
parameters
permissions
premium-features
pulse
query-processor
request
search
server
settings
sql-tools
store-api
sync
system
task
transforms
users
util
warehouses
enterprise/dependencies
enterprise/transforms-python}}
{:team "Metabot"
:api #{metabase-enterprise.metabot-v3.api
metabase-enterprise.metabot-v3.client
metabase-enterprise.metabot-v3.core
metabase-enterprise.metabot-v3.init
metabase-enterprise.metabot-v3.tools.api}
:uses #{activity-feed
api
app-db
audit-app
collections
config
documents
driver
legacy-mbql
lib
lib-be
models
native-query-snippets
permissions
premium-features
pulse
query-processor
request
search
server
settings
store-api
sync
system
task
users-rest
util
warehouse-schema
warehouses-rest
enterprise/dependencies
enterprise/transforms
enterprise/transforms-python}}
enterprise/permission-debug
{:team "UX West"
@@ -2291,8 +2149,7 @@
task
task-history
util
enterprise/serialization
enterprise/transforms-python}}
enterprise/serialization}}
enterprise/sandbox
{:team "UX West"
@@ -2389,7 +2246,6 @@
native-query-snippets
permissions
premium-features
remote-sync
util}}
enterprise/sso
@@ -2417,6 +2273,7 @@
:uses #{analytics
api
collections
collections-rest
embedding
premium-features
queries
@@ -2454,29 +2311,51 @@
enterprise/transforms
{:team "Querying"
:api #{}
:uses #{premium-features}}
enterprise/transforms-python
{:team "Querying"
:api #{metabase-enterprise.transforms-python.api
metabase-enterprise.transforms-python.core
metabase-enterprise.transforms-python.init
metabase-enterprise.transforms-python.python-runner}
:api #{metabase-enterprise.transforms.api
metabase-enterprise.transforms.core
metabase-enterprise.transforms.init
metabase-enterprise.transforms.instrumentation
metabase-enterprise.transforms.interface
metabase-enterprise.transforms.schema
metabase-enterprise.transforms.util}
:uses #{analytics
api
app-db
config
channel
collections
driver
events
lib
lib-be
models
permissions
premium-features
queries
query-processor
remote-sync
request
revisions
search
settings
sync
task
task-history
util}}
enterprise/transforms-python
{:team "Querying"
:api #{metabase-enterprise.transforms-python.api
metabase-enterprise.transforms-python.init}
:uses #{analytics
api
app-db
config
driver
lib
lib-be
query-processor
settings
transforms
util}}
util
enterprise/transforms}}
enterprise/upload-management
{:team "UX West"
@@ -2485,32 +2364,7 @@
models
premium-features
upload
util}}
enterprise/workspaces
{:team "Metabot"
:api #{metabase-enterprise.workspaces.api
metabase-enterprise.workspaces.init}
:uses #{api
api-keys
app-db
config
database-routing
driver
events
lib
lib-be
models
queries
;; NOTE: we will remove this once we get "pure transforms" module.
query-processor
request
settings
sql-tools
system
transforms
util
enterprise/transforms-python}}}
util}}}
;; namespaces matching these patterns (with `re-find`) are excluded from the module linter. Since regex literals
;; aren't allowed in EDN just used the `(str <regex>)` version i.e. two slashes instead of one.

View File

@@ -3,7 +3,7 @@
;; TODO -- seems silly to maintain different blacklists and whitelists here than we use for the deftest `^:parallel`
;; checker...
{:explicitly-safe
#{;; these toucan methods might actually set global values if it's used outside of a transaction,
#{ ;; these toucan methods might actually set global values if it's used outside of a transaction,
;; but since mt/with-temp runs in a transaction, so we'll ignore them in this case.
toucan2.core/delete!
toucan2.core/update!

View File

@@ -1,38 +0,0 @@
(ns hooks.clojure.core
(:require
[clj-kondo.hooks-api :as hooks]
[clojure.string :as str]))
(defn- all-uppercase? [symb]
(let [s (str symb)]
(= s #_{:clj-kondo/ignore [:discouraged-var]} (str/upper-case s))))
(defn- has-underscores? [symb]
(str/includes? (str symb) "_"))
(defn- check-symbol-is-kebab-case [symbol-node]
(let [symb (hooks/sexpr symbol-node)]
(when (or (all-uppercase? symb)
(has-underscores? symb))
(hooks/reg-finding! (assoc (meta symbol-node)
:message "Use kebab-case for function and variable names; don't use special notation for constants."
:type :metabase/check-def-kebab-case)))))
(defn lint-def* [{:keys [node]}]
(let [[_def & args] (:children node)
name-symbol (some (fn [arg]
(when (and (hooks/token-node? arg)
(symbol? (hooks/sexpr arg)))
arg))
args)]
(check-symbol-is-kebab-case name-symbol)))
(defn lint-def [x]
(lint-def* x)
x)
(comment
(defn x []
(let [node (-> "(def ^:private TYPE->MODEL {\"document\" :model/Document})"
hooks/parse-string)]
(lint-def {:node node}))))

View File

@@ -1,52 +0,0 @@
(ns hooks.clojure.core.def
(:require
[clj-kondo.hooks-api :as hooks]
[clojure.string :as str]))
(defn- uppercase-name?
"Whether the symbol is clearly an uppercase name like `TYPE-CONSTANTS` or `SOME_CONSTANT` or `TYPE->MODEL`."
[s]
(and (re-find #"[A-Z0-9]+(?:[-_]>?[A-Z0-9]+)+" s)
;; ignore stuff like `->SCREAMING_SNAKE_CASE`... this is ok
(not (str/includes? s "SCREAMING_SNAKE_CASE"))))
(defn- has-underscores? [s]
(and (str/includes? s "_")
;; ignore stuff like `->snake_case` and `->SCREAMING_SNAKE_CASE`
(not (str/includes? s "snake_case"))
(not (str/includes? s "SCREAMING_SNAKE_CASE"))))
(defn- check-symbol-is-kebab-case [symbol-node]
(let [symb (hooks/sexpr symbol-node)
s (str symb)]
(cond
(uppercase-name? s)
(hooks/reg-finding!
(assoc (meta symbol-node)
:message "Use lower-case names for functions and variables; don't use special notation for constants. [:metabase/check-def-check-not-all-uppercase]"
:type :metabase/check-def-check-not-uppercase-name))
(has-underscores? s)
(hooks/reg-finding!
(assoc (meta symbol-node)
:message "Use kebab-case names for functions and variables. [:metabase/check-def-no-underscores]"
:type :metabase/check-def-no-underscores)))))
(defn lint-def* [{:keys [node]}]
(let [[_def & args] (:children node)
name-symbol (some (fn [arg]
(when (and (hooks/token-node? arg)
(symbol? (hooks/sexpr arg)))
arg))
args)]
(check-symbol-is-kebab-case name-symbol)))
(defn lint-def [x]
(lint-def* x)
x)
(comment
(defn x []
(let [node (-> "(def ^:private TYPE->MODEL {\"document\" :model/Document})"
hooks/parse-string)]
(lint-def {:node node}))))

View File

@@ -1,7 +0,0 @@
(ns hooks.clojure.core.defmacro
(:require
[hooks.clojure.core.def]))
(defn lint-defmacro [x]
(hooks.clojure.core.def/lint-def* x)
x)

View File

@@ -2,7 +2,6 @@
(:require
[clj-kondo.hooks-api :as hooks]
[clojure.string :as str]
[hooks.clojure.core.def]
[hooks.common]))
(defn- check-arglists [report-node arglists]
@@ -50,5 +49,4 @@
(defn lint-defmulti [x]
(defmulti-check-for-arglists-metadata (:node x))
(hooks.clojure.core.def/lint-def* x)
x)

View File

@@ -2,7 +2,6 @@
(:require
[clj-kondo.hooks-api :as hooks]
[clojure.string :as str]
[hooks.clojure.core.def]
[hooks.common]))
(defn- end-with-exclamation?
@@ -51,11 +50,6 @@
(non-thread-safe-form-should-end-with-exclamation* node config))
{:node node})
(defn lint-defn [x]
(non-thread-safe-form-should-end-with-exclamation x)
(hooks.clojure.core.def/lint-def* x)
x)
(comment
(require '[clj-kondo.core :as clj-kondo])
(def form (str '(defmacro a

View File

@@ -75,15 +75,6 @@
(contains? module-api-namespaces ns-symb)
(contains? module-friends current-module))))
(defn- rest-module? [module]
(str/ends-with? module "-rest"))
(defn- routes-module? [module]
(str/ends-with? module "-routes"))
(defn- core-module? [module]
(str/ends-with? module "core"))
(defn usage-error
"Find usage errors when a `required-namespace` is required in the `current-module`. Returns a string describing the
error type if there is one, otherwise `nil` if there are no errors."
@@ -102,15 +93,4 @@
(format "Namespace %s is not an allowed external API namespace for the %s module. [:metabase/modules %s :api]"
required-namespace
required-module
required-module)
;; (for now) rest modules are allowed to use one another; `routes` is ok because it collects routes together
;; and `core` is ok because [[metabase.core.init]] might need to init some of the `-routes` modules'
;; namespaces
(and (not ((some-fn rest-module? routes-module? core-module?) current-module))
(rest-module? required-module))
(format "Do not use -rest modules (%s) in non-rest modules (%s) -- move things from %s to %s if needed"
required-module
current-module
required-module
(symbol (str/replace required-module #"-rest$" "")))))))
required-module)))))

View File

@@ -90,7 +90,6 @@
metabase-enterprise.test/with-gtaps! [[:inner 0]]
metabase-enterprise.test/with-gtaps-for-user! [[:inner 0]]
metabase-enterprise.test/with-user-attributes [[:block 2]]
metabase-enterprise.workspaces.models.workspace-log/track! [[:block 2]]
metabase.actions.actions-test/with-actions-test-data-and-actions-permissively-enabled [[:block 0]]
metabase.actions.test-util/with-actions [[:block 1]]
metabase.actions.test-util/with-actions-disabled [[:block 0]]
@@ -120,10 +119,7 @@
metabase.dashboards.models.dashboard-test/with-dash-in-collection [[:inner 0]]
metabase.driver-api.core/match [[:inner 0]]
metabase.driver-api.core/match-one [[:inner 0]]
metabase.driver-api.core/match-lite [[:inner 0]]
metabase.driver-api.core/match-many [[:inner 0]]
metabase.driver-api.core/replace [[:inner 0]]
metabase.driver-api.core/replace-lite [[:inner 0]]
metabase.driver-api.core/replace-in [[:inner 0]]
metabase.driver.bigquery-cloud-sdk.query-processor/with-temporal-type [[:default]]
metabase.driver.druid.client/DELETE [[:default]]
@@ -137,7 +133,6 @@
metabase.driver.sql-jdbc.execute.diagnostic/capturing-diagnostic-info [[:block 1]]
metabase.driver.sql.query-processor/apply-top-level-clause [[:default]]
metabase.driver/with-driver [[:block 1]]
metabase.driver/with-swapped-connection-details [[:block 2]]
metabase.embedding-rest.api.embed-test/do-response-formats [[:block 1]]
metabase.embedding-rest.api.embed-test/with-embedding-enabled-and-new-secret-key [[:block 0]]
metabase.embedding-rest.api.embed-test/with-embedding-enabled-and-temp-card-referencing [[:block 3]]
@@ -159,9 +154,8 @@
metabase.lib.util.match/match [[:inner 0]]
metabase.lib.util.match/match-one [[:inner 0]]
metabase.lib.util.match/match-lite [[:inner 0]]
metabase.lib.util.match/match-many [[:inner 0]]
metabase.lib.util.match/match-lite-recursive [[:inner 0]]
metabase.lib.util.match/replace [[:inner 0]]
metabase.lib.util.match/replace-lite [[:inner 0]]
metabase.lib.util.match/replace-in [[:inner 0]]
metabase.models.interface/define-batched-hydration-method [[:inner 0]]
metabase.models.interface/define-simple-hydration-method [[:inner 0]]

View File

@@ -1,11 +1,10 @@
FROM mcr.microsoft.com/vscode/devcontainers/java:21
# Set up nodesource, install node, bun, fontconfig for static viz, rlwrap for dev ergonomics
# Set up nodesource, install node, yarn, fontconfig for static viz, rlwrap for dev ergonomics
RUN ( curl -fsSL https://deb.nodesource.com/setup_22.x | bash ) \
&& export DEBIAN_FRONTEND=noninteractive \
&& apt-get update && apt-get -y install --no-install-recommends nodejs rlwrap fontconfig \
&& npm install -g bun
&& apt-get update && apt-get -y install --no-install-recommends nodejs yarn rlwrap fontconfig
# install Clojure
RUN curl https://download.clojure.org/install/linux-install-1.12.0.1488.sh | bash

View File

@@ -61,7 +61,6 @@
(eval . (put-clojure-indent 'u/prog1 1))
(eval . (put-clojure-indent 'u/select-keys-when '(:form)))
(eval . (put-clojure-indent 'with-meta '(:form)))
(eval . (put-clojure-indent 'ws.log/track! 2))
;; these ones have to be done with `define-clojure-indent' for now because of upstream bug
;; https://github.com/clojure-emacs/clojure-mode/issues/600 once that's resolved we should use `put-clojure-indent'
;; instead. Please don't add new entries unless they don't work with `put-clojure-indent'

View File

@@ -100,7 +100,7 @@ METASTORE_DEV_SERVER_URL="https://token-check.staging.metabase.com"
#
# The whole environment is already orchestrated via deps.edn using :e2e alias.
# Treat it as a source of truth for JVM flags and ENVs whether you decide to run
# backend on your own, or using `bun run test-cypress` (local test runner).
# backend on your own, or using `yarn test-cypress` (local test runner).
#
# You don't _have to_ export any of these vars (except for the tokens below).
# Things will still work.
@@ -118,9 +118,6 @@ MB_EDITION="ee"
# MB_JETTY_HOST="localhost"
# MB_JETTY_PORT=4000
# Frontend dev server port (for hot reload). Must match the port used by `yarn build-hot`.
# MB_FRONTEND_DEV_PORT=8080
# 3.2 E2E Local Runner Setup
# ------------------------------------------------------------------------------

6
.eslintignore Normal file
View File

@@ -0,0 +1,6 @@
frontend/src/cljs
frontend/src/cljs_release
e2e/support/cypress_sample_database.js
e2e/support/cypress_sample_instance_data.js
frontend/src/metabase-types/openapi/types.gen.ts
e2e/embedding-sdk-host-apps

356
.eslintrc.js Normal file
View File

@@ -0,0 +1,356 @@
/* eslint-disable import/no-commonjs */
/* eslint-disable no-undef */
const path = require("path");
// `postcss-modules` lints css modules class names, but it currently crashes
// eslint on vscode. If you use webstorm or want to run the lint for the cli, you
// can use this flag to enable it. This is set to true in CI
const shouldLintCssModules =
process.env.LINT_CSS_MODULES === "true" || process.env.CI;
const plugins = ["react", "no-only-tests", "ttag", "i18next"];
if (shouldLintCssModules) {
plugins.push("postcss-modules");
}
module.exports = {
ignorePatterns: ["!.storybook"],
rules: {
strict: [2, "never"],
"no-undef": 2,
"no-var": 1,
"no-unused-vars": [
"error",
{
vars: "all",
args: "none",
varsIgnorePattern: "^_.+$",
ignoreRestSiblings: true,
},
],
"no-empty": [1, { allowEmptyCatch: true }],
// Note: adding this rule to a eslint config file in a subfolder will remove
// *not* carry over the restricted imports from parent folders, you will
// need to copy them over
"no-restricted-imports": [
"error",
{
paths: [
{
name: "moment",
message: "Moment is deprecated, please use dayjs",
},
{
name: "moment-timezone",
message: "Moment is deprecated, please use dayjs",
},
],
},
],
curly: [1, "all"],
eqeqeq: [1, "smart"],
"import/no-duplicates": ["warn", { considerQueryString: true }],
"import/no-default-export": 2,
"import/no-named-as-default": 0,
"import/no-commonjs": 1,
"import/order": [
"error",
{
"newlines-between": "always",
alphabetize: {
order: "asc",
orderImportKind: "asc",
caseInsensitive: false,
},
groups: [
"builtin",
"external",
"internal",
"parent",
"sibling",
"index",
],
warnOnUnassignedImports: false,
},
],
"sort-imports": [
"error",
{
// allows this rule to work with import/order
ignoreDeclarationSort: true,
},
],
"no-console": [2, { allow: ["warn", "error", "errorBuffer"] }],
"react/no-is-mounted": 2,
"react/prefer-es6-class": 2,
"react/display-name": 1,
"react/prop-types": 2,
"react/no-did-mount-set-state": 0,
"react/no-did-update-set-state": 0,
"react/no-find-dom-node": 0,
"react/no-children-prop": 2,
"react/no-string-refs": 2,
"react/no-unescaped-entities": 2,
"react/jsx-no-target-blank": 2,
"react/jsx-key": 2,
"react/forbid-component-props": [2, { forbid: ["sx"] }],
"react-hooks/exhaustive-deps": ["warn"],
"prefer-const": [1, { destructuring: "all" }],
"no-restricted-globals": ["error", "close"],
"no-useless-escape": 0,
"no-only-tests/no-only-tests": [
"error",
{
block: [
"describe",
"it",
"context",
"test",
"tape",
"fixture",
"serial",
"Feature",
"Scenario",
"Given",
"And",
"When",
"Then",
],
},
],
complexity: ["error", { max: 55 }],
...(shouldLintCssModules
? {
"postcss-modules/no-undef-class": "error",
}
: {}),
},
globals: {
before: true,
cy: true,
Cypress: true,
},
env: {
browser: true,
es2020: true,
commonjs: true,
jest: true,
"jest/globals": true,
},
parser: "babel-eslint",
plugins,
extends: [
"eslint:recommended",
"plugin:react/recommended",
"plugin:react/jsx-runtime",
"plugin:react-hooks/recommended",
"plugin:import/errors",
"plugin:import/warnings",
"plugin:import/typescript",
"plugin:depend/recommended",
"plugin:storybook/recommended",
"plugin:i18next/recommended",
],
settings: {
"import/internal-regex": "^metabase/|^metabase-lib/",
"import/resolver": {
webpack: {
config: path.resolve(__dirname, "./rspack.main.config.js"),
typescript: true,
},
},
"import/ignore": ["\\.css$"],
react: {
version: "detect",
},
"postcss-modules": {
baseDir: "./frontend/src",
},
},
parserOptions: {
ecmaFeatures: {
legacyDecorators: true,
},
},
overrides: [
{
files: ["*.js", "*.jsx", "*.ts", "*.tsx"],
rules: {
"jtag-missing-key": "error",
"no-unconditional-metabase-links-render": "error",
"no-color-literals": "error",
"no-literal-metabase-strings": "error",
"no-oss-reinitialize-import": "error",
"depend/ban-dependencies": [
"error",
{
allowed: [
"underscore",
"moment",
"lodash.orderby",
"lodash.debounce",
],
},
],
},
},
{
files: [
"*.unit.spec.*",
"frontend/src/metabase/admin/**/*",
"frontend/src/metabase/setup/**/*",
"enterprise/frontend/src/metabase-enterprise/whitelabel/**/*",
"enterprise/frontend/src/metabase-enterprise/embedding/**/*",
"frontend/lint/**/*",
"*.stories.*",
"**/.storybook/*",
"stories-data.*",
"e2e/**/*",
"**/tests/*",
"release/**/*",
"rspack.config.js",
"rspack.main.config.js",
"rspack.embedding-sdk-package.config.js",
"rspack.embedding-sdk-bundle.config.js",
],
rules: {
"no-color-literals": "off",
"no-unconditional-metabase-links-render": "off",
"no-literal-metabase-strings": "off",
},
},
{
files: [
"*.unit.spec.*",
"frontend/lint/**/*",
"*.stories.*",
"stories-data.*",
"e2e/**/*",
"**/tests/*",
"release/**/*",
"rspack.main.config.js",
],
rules: {
"i18next/no-literal-string": "off",
},
},
{
extends: ["plugin:@typescript-eslint/recommended"],
files: ["*.ts", "*.tsx"],
parser: "@typescript-eslint/parser",
plugins: ["@typescript-eslint"],
rules: {
"prefer-rest-params": "off",
"react/prop-types": "off", // TypeScript handles prop validation
"@typescript-eslint/explicit-module-boundary-types": "off",
"@typescript-eslint/no-inferrable-types": "off",
"@typescript-eslint/no-explicit-any": "off",
"@typescript-eslint/no-this-alias": "off",
"@typescript-eslint/consistent-type-imports": [
"error",
{
fixStyle: "inline-type-imports",
},
],
"@typescript-eslint/no-import-type-side-effects": "error",
"@typescript-eslint/no-unused-vars": [
"error",
{
argsIgnorePattern: "^_.+$",
varsIgnorePattern: "^_.+$",
ignoreRestSiblings: true,
destructuredArrayIgnorePattern: "^_.+$",
},
],
// This was introduced in 6.0.0
"@typescript-eslint/no-unsafe-declaration-merging": "off",
},
},
{
extends: [
"plugin:jest/recommended",
"plugin:jest-dom/recommended",
"plugin:testing-library/react",
"plugin:jest-formatting/recommended",
],
plugins: ["jest", "jest-dom", "testing-library", "jest-formatting"],
files: [
"*.unit.spec.ts",
"*.unit.spec.tsx",
"*.unit.spec.js",
"*.unit.spec.jsx",
],
rules: {
"jest/valid-title": ["error", { ignoreTypeOfDescribeName: true }],
"jest/expect-expect": [
"error",
{
assertFunctionNames: ["expect*", "assert*"],
additionalTestBlockFunctions: [],
},
],
},
},
{
// Enable jest formatting for cypress tests too, the plugin logic just works
extends: ["plugin:jest-formatting/recommended"],
files: ["*.cy.spec.ts", "*.cy.spec.js"],
},
{
files: ["frontend/src/**/*"],
rules: {
"no-restricted-syntax": [
"error",
{
selector: "Literal[value=/mb-base-color-/]",
message:
"You may not use base colors in the application, use semantic colors instead. (see colors.module.css)",
},
],
},
},
{
files: ["frontend/src/metabase/**/*"],
rules: {
"no-restricted-syntax": [
"error",
{
selector: "Literal[value=/mb-base-color-/]",
message:
"You may not use base colors in the application, use semantic colors instead. (see colors.module.css)",
},
],
},
},
{
files: ["frontend/src/metabase/query_builder/**/*"],
rules: {
"import/no-cycle": "error",
},
},
{
files: ["docs/**/snippets/**/*.{ts,tsx,js,jsx}"],
rules: {
"@typescript-eslint/no-unused-vars": "off",
"@typescript-eslint/no-var-requires": "off",
"import/no-commonjs": "off",
"import/no-default-export": "off",
"import/order": "off",
"import/no-unresolved": "off",
"no-color-literals": "off",
},
},
{
files: ["frontend/build/**/*.js"],
rules: {
"import/no-commonjs": "off",
},
},
{
files: ["**/*.stories.tsx", "**/preview.tsx"],
rules: {
"import/no-default-export": "off",
},
},
],
};

View File

@@ -1,5 +1,7 @@
# execute this in your shell to enable custom merge driver:
# echo '\n[include]\npath=../.gitconfig' >> .git/config
[merge "yaml-migrations"]
name = YAML migration changeset-aware merge
driver = bin/mage -merge-yaml-migrations %O %A %B %L %P
#
# TODO (Bryan 12/12/25) -- reenable this when we can completely verify correctness
# [merge "yaml-migrations"]
# name = YAML migration changeset-aware merge
# driver = mage -merge-yaml-migrations %O %A %B %L %P

View File

@@ -5,7 +5,7 @@ PENDING CI / PENDING REVIEW / READY _(choose one and update accordingly)_
- Reproduces #XXXXX
### How to test this manually?
- `bun run test-cypress`
- `yarn test-cypress`
- `relative/path/to/the/file`
> _(For still unfixed bug)_
- Replace `it.skip()` with `it.only()` to run the test in isolation

View File

@@ -2,16 +2,9 @@ name: Fetch artifact
description: Fetches a specific artifact and downloads it to the current directory
inputs:
commit:
description: Commit SHA for the artifact
name:
description: The name of the artifact to fetch
required: true
edition:
description: Metabase edition (ee or oss)
required: true
release:
description: Whether this is a release artifact (uses metabase-release- prefix)
required: false
default: "false"
output-name:
description: Optional custom name for the output jar file (defaults to metabase.jar)
required: false
@@ -32,21 +25,15 @@ runs:
script: | # js
const fs = require('fs');
const isRelease = "${{ inputs.release }}" === "true";
const prefix = isRelease ? "metabase-release" : "metabase";
const artifactName = `${prefix}-${{ inputs.edition }}-${{ inputs.commit }}-uberjar`;
console.log(`Looking for artifact: ${artifactName}`);
const artifacts = await github.rest.actions.listArtifactsForRepo({
owner: context.repo.owner,
repo: context.repo.repo,
name: artifactName,
name: "${{ inputs.name }}",
per_page: 1,
});
if (!artifacts.data?.artifacts?.[0]?.id) {
core.setFailed(`No artifacts found for ${artifactName}`);
return;
throw new Error(`No artifacts found for ${{ inputs.name }}`);
}
const artifact_id = artifacts.data.artifacts[0].id;

View File

@@ -1,13 +1,14 @@
name: Prepare back-end environment
description: |
Prepare the backend. Installs jdk, Clojure, and fetches the cache for m2.
The CI config fetches `ci-test-config.json` from master and lets us ignore tests at the var level if they are flaking.
inputs:
java-version:
required: true
default: "21"
default: '21'
clojure-version:
required: true
default: "1.12.0.1488"
default: '1.12.0.1488'
runs:
using: "composite"
@@ -16,7 +17,7 @@ runs:
uses: actions/setup-java@v4
with:
java-version: ${{ inputs.java-version }}
distribution: "temurin"
distribution: 'temurin'
- name: Install Clojure CLI
shell: bash
continue-on-error: true
@@ -28,13 +29,6 @@ runs:
uses: ./.github/actions/get-cache-keys
id: get-cache-keys
- name: Install uv (for Python package installation)
uses: astral-sh/setup-uv@v7
with:
enable-cache: false
version: "0.10.2" # keep in sync with mise.toml
github-token: ${{ github.token }}
- name: Restore M2 cache
uses: actions/cache/restore@v4
with:

View File

@@ -3,7 +3,7 @@ description: Cypress preparation steps
inputs:
working-directory:
description: "Optional working directory for bun commands"
description: "Optional working directory for yarn commands"
required: false
default: "."
@@ -15,10 +15,11 @@ outputs:
runs:
using: "composite"
steps:
- name: Install Chrome v144
uses: browser-actions/setup-chrome@v2
- name: Install Chrome v111
uses: browser-actions/setup-chrome@v1
with:
chrome-version: 144
# https://chromium.cypress.io/linux/stable/111.0.5563.146
chrome-version: 1097615
id: setup-chrome
- name: Get cache keys
@@ -33,9 +34,9 @@ runs:
- name: Ensure that Cypress executable is ready
run: |
bun run cypress install
bun run cypress cache path
bun run cypress cache list
bun run cypress verify
yarn cypress install
yarn cypress cache path
yarn cypress cache list
yarn cypress verify
shell: bash
working-directory: ${{ inputs.working-directory }}

View File

@@ -17,12 +17,6 @@ runs:
node-version: ${{ inputs.node-version }} # this overrides, if present
node-version-file: ".nvmrc"
- name: Install bun
uses: oven-sh/setup-bun@v2
with:
bun-version: "1.3.7"
token: ${{ github.token }}
- name: Get cache keys
uses: ./.github/actions/get-cache-keys
id: get-cache-keys
@@ -35,12 +29,12 @@ runs:
~/.gitlibs
key: ${{ steps.get-cache-keys.outputs.m2-cache-key }}
- name: Get bun cache directory
id: bun-cache
- name: Get yarn cache directory
id: yarn-cache-dir
run: |
BUN_CACHE_DIR=$(bun pm cache)
echo "Bun cache directory: $BUN_CACHE_DIR"
echo "dir=$BUN_CACHE_DIR" >> $GITHUB_OUTPUT
YARN_CACHE_DIR=$(yarn cache dir)
echo "Yarn cache directory: $YARN_CACHE_DIR"
echo "dir=$YARN_CACHE_DIR" >> $GITHUB_OUTPUT
shell: bash
- name: Restore node_modules cache
@@ -48,7 +42,7 @@ runs:
id: node-modules-cache
with:
path: |
${{ steps.bun-cache.outputs.dir }}
${{ steps.yarn-cache-dir.outputs.dir }}
node_modules
key: ${{ steps.get-cache-keys.outputs.node-cache-key }}
@@ -56,9 +50,9 @@ runs:
# If a job wants to patch packages, it does so explicitly in the CI. This setting is opt-out.
# For example, we never want to cache node_modules with patches already applied so cache-generator.yml skips that step.
- name: Clean install
run: bun install --frozen-lockfile
run: yarn install --frozen-lockfile --prefer-offline
shell: bash
- name: Patch frontend dependencies
if: inputs.apply-patches == 'true'
run: bun run patch-package
run: yarn patch-package
shell: bash

View File

@@ -1,14 +0,0 @@
name: Prepare Release Scripts
description: Set up bun and build release scripts
runs:
using: composite
steps:
- uses: oven-sh/setup-bun@v2
with:
bun-version: "1.3.7"
token: ${{ github.token }}
- name: Build release scripts
shell: bash
run: cd ${{ github.workspace }}/release && bun install --frozen-lockfile && bun run build

View File

@@ -1,132 +0,0 @@
name: Prepare Shoppy E2E environment
description: Shared setup steps for Shoppy E2E tests (regular and synthetic)
inputs:
commit:
description: "Commit SHA for artifact retrieval"
required: true
sample-app-branch-name:
description: "Branch name for the sample app"
required: true
sample-app-environment:
description: "Environment (production/development)"
required: true
staging-mb-all-features-token:
description: "Staging MB all features token"
required: true
shoppy-iam-role:
description: "Shoppy IAM role for AWS credentials"
required: true
shoppy-metabase-app-db-s3-bucket:
description: "S3 bucket for Shoppy app DB dump"
required: true
shoppy-metabase-app-db-s3-key:
description: "S3 key for Shoppy app DB dump"
required: true
shoppy-metabase-app-db-user:
description: "Shoppy Metabase app DB user"
required: true
shoppy-metabase-app-db-database:
description: "Shoppy Metabase app DB database"
required: true
metastore-dev-server-url:
description: "Metastore dev server URL"
required: true
shoppy-datadog-application-id:
description: "Shoppy Datadog application ID"
required: true
shoppy-datadog-client-token:
description: "Shoppy Datadog client token"
required: true
outputs:
chrome-path:
description: "Path to Chrome executable"
value: ${{ steps.cypress-prep.outputs.chrome-path }}
sample_app_branch_exists:
description: "Whether the sample app branch exists"
value: ${{ steps.check-sample-app-branch.outputs.sample_app_branch_exists }}
runs:
using: "composite"
steps:
- uses: ./.github/actions/maximize-disk-space
# We have to use the latest version that does not have this issue https://github.com/docker/compose/pull/12752
- name: Set up Docker Compose
uses: docker/setup-compose-action@v1
with:
version: latest
- name: Retrieve uberjar artifact for ee
uses: ./.github/actions/fetch-artifact
with:
commit: ${{ inputs.commit }}
edition: ee
extract-path: "target/uberjar"
output-name: "metabase.jar"
- name: Retrieve Embedding SDK dist artifact
uses: actions/download-artifact@v4
with:
name: embedding-sdk-${{ inputs.commit }}
path: ${{ github.workspace }}/resources/embedding-sdk
- name: Prepare front-end environment
uses: ./.github/actions/prepare-frontend
- name: Prepare Cypress environment
id: cypress-prep
uses: ./.github/actions/prepare-cypress
- name: Check if sample app branch exists
id: check-sample-app-branch
shell: bash
env:
SAMPLE_APP_BRANCH_NAME: ${{ inputs.sample-app-branch-name }}
run: |
REPO_URL="https://github.com/metabase/shoppy.git"
echo "Checking if branch '${SAMPLE_APP_BRANCH_NAME}' exists in ${REPO_URL}..."
if git ls-remote --exit-code --heads "$REPO_URL" "${SAMPLE_APP_BRANCH_NAME}"; then
echo "sample_app_branch_exists=true" >> $GITHUB_OUTPUT
else
echo "sample_app_branch_exists=false" >> $GITHUB_OUTPUT
fi
- name: Configure AWS credentials
if: ${{ steps.check-sample-app-branch.outputs.sample_app_branch_exists == 'true' }}
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: ${{ inputs.shoppy-iam-role }}
role-session-name: GitHub_to_AWS_via_FederatedOIDC
aws-region: us-east-1
- name: Download Shoppy's App DB PostgreSQL dump from S3
if: ${{ steps.check-sample-app-branch.outputs.sample_app_branch_exists == 'true' }}
shell: bash
run: |
aws s3 cp s3://${{ inputs.shoppy-metabase-app-db-s3-bucket }}/${{ inputs.shoppy-metabase-app-db-s3-key }} ./e2e/tmp/db_dumps/shoppy_metabase_app_db_dump.sql
- name: Prepare and launch Shoppy containers
if: ${{ steps.check-sample-app-branch.outputs.sample_app_branch_exists == 'true' }}
shell: bash
env:
SAMPLE_APP_BRANCH_NAME: ${{ inputs.sample-app-branch-name }}
SAMPLE_APP_ENVIRONMENT: ${{ inputs.sample-app-environment }}
CYPRESS_MB_ALL_FEATURES_TOKEN: ${{ inputs.staging-mb-all-features-token }}
METASTORE_DEV_SERVER_URL: ${{ inputs.metastore-dev-server-url }}
METABASE_APP_DB_USER: ${{ inputs.shoppy-metabase-app-db-user }}
METABASE_APP_DB: ${{ inputs.shoppy-metabase-app-db-database }}
SHOPPY_DATADOG_APPLICATION_ID: ${{ inputs.shoppy-datadog-application-id }}
SHOPPY_DATADOG_CLIENT_TOKEN: ${{ inputs.shoppy-datadog-client-token }}
SHOPPY_DATADOG_SITE: "us5.datadoghq.com"
SHOPPY_DATADOG_ENV: "staging"
run: npx tsx ./e2e/runner/embedding-sdk/sample-apps/start-ci.ts shoppy-e2e
- name: Prune build cache
shell: bash
run: docker builder prune -af
- name: Log free disk space
shell: bash
run: df -h /

View File

@@ -2,10 +2,9 @@ name: Prepare Uberjar Artifact
description: Prepare and upload Metabase JAR as a GitHub artifact.
inputs:
edition:
required: false
default: "ee"
description: Metabase edition (ee or oss)
name:
required: true
description: Artifact name
runs:
using: "composite"
@@ -19,7 +18,7 @@ runs:
- name: Upload JARs as artifact
uses: actions/upload-artifact@v4
with:
name: metabase-${{ inputs.edition }}-${{ github.event.pull_request.head.sha || github.sha }}-uberjar
name: ${{ inputs.name }}
path: |
./target/uberjar/metabase.jar
./COMMIT-ID

View File

@@ -1,77 +0,0 @@
name: Embedding SDK Bundle Size Check
description: |
Downloads the current branch's uberjar and builds the base ref's
embedding-sdk.js from source, compares their sizes,
and sets output if the size change exceeds the threshold.
inputs:
edition:
description: 'Metabase edition (ee or oss)'
required: true
base-ref-commit:
description: 'Commit SHA of the base ref (e.g. merge base).'
required: true
threshold-percent:
description: 'Percent threshold for bundle size change'
required: true
outputs:
status:
description: "Bundle size status: increased, decreased, or stable"
value: ${{ steps.compare.outputs.status }}
size_change_percent:
description: "size change percent (positive = increase, negative = decrease)"
value: ${{ steps.compare.outputs.size_change_percent }}
runs:
using: "composite"
steps:
- name: Prepare backend
uses: ./.github/actions/prepare-backend
- name: Prepare frontend
uses: ./.github/actions/prepare-frontend
- name: Download current uberjar
uses: actions/download-artifact@v4
with:
name: metabase-${{ inputs.edition }}-${{ github.event.pull_request.head.sha || github.sha }}-uberjar
path: temp/current-uberjar
- name: Extract embedding-sdk.js from current uberjar
shell: bash
run: unzip -j temp/current-uberjar/target/uberjar/metabase.jar frontend_client/app/embedding-sdk.js -d temp/current-bundle
- name: Checkout base ref
uses: actions/checkout@v4
with:
ref: ${{ inputs.base-ref-commit }}
path: temp/base-ref-src
- name: Build base-ref embedding-sdk.js
shell: bash
run: |
cd temp/base-ref-src
bun install
MB_EDITION=${{ inputs.edition }} bun run build-release
- name: Compare sizes and set output
id: compare
shell: bash
run: |
current_size=$(wc -c < temp/current-bundle/embedding-sdk.js) || { echo "::error::Failed to read current bundle"; exit 1; }
base_ref_size=$(wc -c < temp/base-ref-src/resources/frontend_client/app/embedding-sdk.js) || { echo "::error::Failed to read base-ref bundle"; exit 1; }
[ "$current_size" -gt 0 ] && [ "$base_ref_size" -gt 0 ] || { echo "::error::Bundle file is empty"; exit 1; }
diff=$((current_size - base_ref_size))
percent=$(( diff * 100 / base_ref_size ))
echo "Current size: $current_size bytes"
echo "Base ref size: $base_ref_size bytes"
echo "Difference: $diff bytes ($percent%)"
threshold="${{ inputs.threshold-percent }}"
if [ "$percent" -gt "$threshold" ]; then
status="increased"
elif [ "$percent" -lt "$(( -threshold ))" ]; then
status="decreased"
else
status="stable"
fi
echo "status=$status" >> $GITHUB_OUTPUT
echo "size_change_percent=$percent" >> $GITHUB_OUTPUT

View File

@@ -12,15 +12,6 @@ inputs:
logs-artifact-suffix:
required: false
description: Suffix to append to the test logs artifact name to avoid conflicts in matrix jobs
edition:
required: false
default: 'ee'
description: Edition to test (oss or ee)
outputs:
test-outcome:
description: "The outcome of the test run (success, failure, cancelled, skipped)"
value: ${{ steps.run-test.outcome }}
runs:
using: "composite"
@@ -32,12 +23,12 @@ runs:
uses: ./.github/actions/prepare-frontend
- name: Build static viz frontend
if: ${{ inputs.build-static-viz == 'true' }}
run: bun run build-static-viz
run: yarn build-static-viz
shell: bash
env:
MB_EDITION: ee
- name: Test database driver
run: clojure -X:dev:ci:${{ inputs.edition }}:${{ inputs.edition }}-dev:drivers:drivers-dev:test ${{ inputs.test-args }}
run: clojure -X:dev:ci:ee:ee-dev:drivers:drivers-dev:test ${{ inputs.test-args }}
shell: bash
id: run-test
# Trunk Quarantine controls the final state of the run in the CI!
@@ -61,7 +52,7 @@ runs:
if: always() && steps.run-test.outcome != 'success'
uses: actions/upload-artifact@v4
with:
name: ${{ format('test-logs-{0}{1}', github.job, inputs.logs-artifact-suffix && format('-{0}', inputs.logs-artifact-suffix) || '') }}
name: ${{ format('test-logs-{0}{1}', github.job, (inputs.logs-artifact-suffix == '' || inputs.logs-artifact-suffix == null) && '' || format('-{0}', inputs.logs-artifact-suffix)) }}
path: logs/test-log.json
retention-days: 1
# Failing to upload the artifact should never affect the outcome of the whole test run.

View File

@@ -21,8 +21,6 @@ inputs:
variant:
required: false
default: ""
previous-step-outcome:
required: true
runs:
using: "composite"
@@ -59,4 +57,3 @@ runs:
dry-run: ${{ github.event_name == 'pull_request' && github.event.pull_request.draft == true }} # quarantine, but do not upload results
token: ${{ inputs.trunk-api-token }}
use-cache: true
previous-step-outcome: ${{ inputs.previous-step-outcome }}

View File

@@ -1,55 +1,52 @@
# Metabase Developer Assistant Instructions
For detailed coding standards and conventions, see `CLAUDE.md` and the skills in `.claude/skills/`.
## Frontend Development
## Code Review Standards
### Repository Structure
Review focus areas: Security → Performance → Testing → Documentation (in priority order)
```
frontend/src/
├── metabase/ # Main application components and pages
├── metabase-lib/ # Query building and data modeling utilities
├── metabase-types/ # TypeScript type definitions
└── metabase/ui/ # Design system components
When reviewing code, use these emoji prefixes to categorize feedback:
- 🔒 Security concerns
- ⚡ Performance opportunities
- 🧹 Cleanup needs
- 📚 Documentation gaps
- 🚨 Blocking issues
- 💭 Clarification questions
e2e/ # Cypress end-to-end tests
frontend/test/ # Jest unit tests
```
### Security Critical Issues
### Technology Stack
- Check for hardcoded secrets, API keys, or credentials
- Look for SQL injection and XSS vulnerabilities
- Verify proper input validation and sanitization
- Review authentication and authorization logic
- Ensure enterprise features use the plugin system (no enterprise code in OSS)
- **Framework**: React 18 with TypeScript
- **State**: Redux Toolkit
- **Styling**: CSS Modules (preferred) > Emotion styled-components
- **UI**: `metabase/ui` components built with Mantine v8
- **Build**: Rspack (primary), Webpack (legacy)
- **Testing**: Jest + React Testing Library, Cypress
### Performance Red Flags
### Coding Standards
- Identify N+1 database query problems
- Spot inefficient loops and algorithmic issues
- Check for memory leaks and resource cleanup
- Review caching opportunities for expensive operations
#### Component Preferences
### Code Quality Essentials
- Prefer `metabase/ui`components over `metabase/common/components`
- Use `.tsx` for components, `.ts` for utilities
- Functions should be focused and appropriately sized
- Use clear, descriptive naming conventions
- Ensure proper error handling throughout
- Remove dead code and unused imports
#### Styling
### Review Style
ALWAYS prefer Mantine style props,then CSS modules. DO NOT suggest styled components, they are deprecated.
- Only flag actual issues worth mentioning
- Do not post "looks good" comments or congratulate following conventions
- Be specific and actionable in feedback
- Explain the "why" behind recommendations
- Ask clarifying questions when code intent is unclear
#### TypeScript Migration
### Review Format
When heavily editing `.js`/`.jsx` files, create a separate PR to convert to TypeScript first, then implement changes.
- Condense feedback into as few comments as possible
- Prefer a single summary comment over dozens of inline comments
- Only use inline comments for critical issues that require immediate attention
- For non-critical issues, link to the specific line from the summary instead
- Delete all stale comments and summaries on each new push or CI run
#### Enterprise Features
Language-specific rules are in `.github/instructions/*.instructions.md`.
Enterprise functionality MUST use the plugin system. It is very important to not expose enterprise code in the OSS version.
#### Testing Requirements
All PRs should include tests. Prefer Unit tests over E2E tests.
#### Localization
All user-facing strings MUST be localized using the ttag library. Localized strings should be complete phrases, do not concatenate a few separately localized strings. You should add context to strings where the meaning of the string might not be obvious in isolation: e.g. "Home" might have different words in some languages depending on whether you're talking about a dwelling or the landing page for a website.

View File

@@ -26,7 +26,7 @@ frontend_ci: &frontend_ci
- ".github/workflows/frontend.yml"
frontend_configs: &frontend_configs
- "bun.lock"
- "yarn.lock"
- "**/tsconfig*.json"
- "package.json"
- "babel.config.json"
@@ -85,9 +85,7 @@ backend_ci: &backend_ci
backend_presto_kerberos:
- "**/presto_jdbc/**"
- "**/presto-jdbc/**" # the driver module has a hyphen. underscores are classpaths
- "**/presto_jdbc.clj"
- "**/presto_jdbc_test.clj"
- ".github/workflows/presto-kerberos-integration-test.yml"
backend_sources: &backend_sources
@@ -199,15 +197,3 @@ cljs:
- *shared_specs
- ".github/workflows/cljs.yml"
- "shadow-cljs.edn"
openapi:
- ".github/workflows/openapi.yml"
- "src/metabase/api/**"
- "src/metabase/api_routes/**"
- "src/metabase/*_rest/api.clj"
- "src/metabase/*/api.clj"
- "src/metabase/*/api/*.clj"
- "enterprise/backend/src/metabase_enterprise/**/api.clj"
- "enterprise/backend/src/metabase_enterprise/**/api/*.clj"
- "resources/openapi/**"
- "package.json"

View File

@@ -1,9 +0,0 @@
---
applyTo: "**/*.{clj,cljs,cljc}"
---
# Clojure/ClojureScript Code Review
When reviewing Clojure or ClojureScript code, apply the guidelines in these files:
- `.claude/skills/_shared/clojure-style-guide.md`
- `.claude/skills/clojure-review/SKILL.md`

View File

@@ -1,9 +0,0 @@
---
applyTo: "docs/**/*.md"
---
# Documentation Review
When reviewing documentation, apply the guidelines in these files:
- `.claude/skills/_shared/metabase-style-guide.md`
- `.claude/skills/docs-review/SKILL.md`

View File

@@ -1,11 +0,0 @@
---
applyTo: "**/*.{ts,tsx,js,jsx}"
---
# TypeScript/JavaScript Code Review
When reviewing TypeScript or JavaScript code, apply the guidelines in these files:
- `.claude/skills/typescript-review/SKILL.md`
- `docs/developers-guide/frontend.md`
- `frontend/CLAUDE.md`

View File

@@ -1,41 +0,0 @@
/**
* jsonData must be in the format:
* [
* { column1: value1, column2: value2, ... },
* { column1: value3, column2: value4, ... },
* ...
* ]
*/
function jsonToCsvFormData(jsonData) {
const formData = new FormData();
const csvHeader = Object.keys(jsonData[0]).join(',') + '\n';
const csvContent = jsonData.map(row =>
Object.values(row).map(value => `"${String(value).replace(/"/g, '""')}"`).join(',')
).join('\n');
const blob = new Blob([csvHeader + csvContent], { type: 'text/csv' });
formData.append('file', blob, 'data.csv');
return formData;
}
function uploadCsvToMb({ baseUrl, tableId, jsonData, mode = 'append' }) {
const formData = jsonToCsvFormData(jsonData);
const operation = mode === 'replace' ? 'replace-csv' : 'append-csv';
return fetch(`${baseUrl}/api/table/${tableId}/${operation}`, {
method: 'POST',
headers: {
"x-api-key": process.env.API_KEY
},
body: formData
})
.then(response => {
if (!response.ok) {
throw new Error(`Upload failed`);
}
return { success: true };
});
}
module.exports = { uploadCsvToMb };

317
.github/scripts/generate-llms-txt.js vendored Normal file
View File

@@ -0,0 +1,317 @@
const fs = require("fs");
const path = require("path");
const REPO = "metabase/metabase";
const OUTPUT_FILE = "llms.txt";
// Sections to generate llms-<name>-full.txt for.
// Each section can define a preamble (introductory content before the main docs).
const LLMS_FULL_TO_GENERATE = [
{
section: "embedding",
getPreamble: (branch) =>
aboveVersion(branch, 57) ? getModularEmbeddingGotchaNotes() : "",
},
];
/**
* List directory recursively to get markdown files
*
* @param {string} docsPath - full path to docs directory
* @returns {Array} list of markdown file objects with fullPath and relativePath
*/
function getMarkdownFilesList(docsPath) {
if (!fs.existsSync(docsPath)) {
console.error(`Error: Docs directory '${docsPath}' does not exist`);
process.exit(1);
}
const entries = fs.readdirSync(docsPath, {
recursive: true,
withFileTypes: true,
});
return entries
.filter((entry) => {
return (
entry.isFile() &&
// Only markdown files
entry.name.endsWith(".md") &&
// Ignore README.md
entry.name !== "README.md" &&
// Ignore node_modules and hidden directories
!entry.parentPath
.split(path.sep)
.some((dir) => dir === "node_modules" || dir.startsWith("."))
);
})
.map((entry) => {
const fullPath = path
.join(entry.parentPath, entry.name)
.replace(/\\/g, "/");
const relativePath = path
.relative(docsPath, fullPath)
.replace(/\\/g, "/");
return { fullPath, relativePath };
});
}
/**
* Generate llms.txt file with links to all markdown files in docs/
*
* @param {string} branch - branch name (e.g. 'master' or 'release-x.58.x')
* @param {Array} files - list of all markdown file objects
* @param {Array<Object>} sectionsConfig - list of section config objects to generate complete references for
*/
function generateLlmsTxt(branch, files, sectionsConfig) {
const baseUrl = `https://raw.githubusercontent.com/${REPO}/refs/heads/${branch}`;
const filesToExtract = files
// Files in embedding/sdk/api/snippets does not exist in the repo.
.filter(
(file) => !file.relativePath.startsWith("embedding/sdk/api/snippets"),
);
const entries = filesToExtract.map((file) => {
const content = fs.readFileSync(file.fullPath, "utf8");
const title = extractTitle(content, file.relativePath);
const url = `${baseUrl}/docs/${file.relativePath}`;
return { title, url, path: file.relativePath };
});
// Generate table of contents entries
const tableOfContents = entries.map(
(entry) => `- [${entry.title}](${entry.url})`,
);
// Generate complete references entries
const completeReferences = sectionsConfig
.filter((config) =>
files.some((file) => file.relativePath.startsWith(`${config.section}/`)),
)
.map((config) => {
const capitalized =
config.section.charAt(0).toUpperCase() + config.section.slice(1);
return `- [${capitalized} - Complete Reference](${baseUrl}/llms-${config.section}-full.txt)`;
});
// Generate llms.txt content using template literal
const content = `# Metabase Documentation
Metabase - The simplest, fastest way to get business intelligence and analytics to everyone in your company.
## Table of Contents
${tableOfContents.join("\n")}
## Complete References
These files are very large and are around 90,000 tokens. Do not use by default unless the context window is huge or RAG is supported in your editor.
${completeReferences.join("\n")}
`;
fs.writeFileSync(OUTPUT_FILE, content, "utf8");
console.log(
`✅ Generated ${OUTPUT_FILE} with ${entries.length} markdown files`,
);
console.log(` Branch: ${branch}`);
console.log(` Base URL: ${baseUrl}`);
}
/**
* Generate llms-full.txt files for specified sections
*
* @param {string} branch - branch name (e.g., 'master', 'release-x.58.x')
* @param {Array} files - list of all markdown file objects
* @param {Array<Object>} sectionsConfig - list of section config objects to generate full files for
*/
function generateLlmsFullFiles(branch, files, sectionsConfig) {
const baseUrl = `https://raw.githubusercontent.com/${REPO}/refs/heads/${branch}`;
sectionsConfig.forEach((config) => {
generateLlmsFullTxt(branch, config, files, baseUrl);
});
}
/**
* LLMs are likely to pay attention to the very first lines.
*
* We add the most important context for LLMs to avoid
* confusion and pitfalls like out-of-date APIs in trained data.
*
* @returns {string} Markdown formatted gotcha notes
*/
function getModularEmbeddingGotchaNotes() {
return `> **Important Version Notes**
>
> Watch out for these deprecated props and gotchas for Metabase 57 onwards, for modular embedding:
>
> 1. \`config\` prop on MetabaseProvider no longer exist - it is replaced by \`authConfig\`.
> 2. \`authProviderUri\` field no longer exist.
> 3. \`jwtProviderUri\` optional field only exists in v58+. This is used to make JWT auth faster by skipping the \`GET /auth/sso\` discovery request. Not needed for initial implementation.
> 4. \`fetchRequestToken\` is not needed by default. This is only used to customize how the SDK fetches the request token.
> 5. Numeric IDs must be integers not strings, e.g. \`dashboardId={1}\`. When the ID is retrieved from the URL and it is numeric, convert it to an integer via \`parseInt\` before passing it to the SDK. IDs can also be strings for entity ids, so you should not parse all IDs as numbers if entity ids are also to be expected.
`;
}
/**
* Generate llms-full.txt file for a specific section
* @param {string} branch - The branch name
* @param {Object} config - Section config object with section name and optional getPreamble function
* @param {Array} files - Array of all markdown file objects
* @param {string} baseUrl - Base URL for raw GitHub links
*/
function generateLlmsFullTxt(branch, config, files, baseUrl) {
const { section, getPreamble } = config;
// Filter docs for this section
const sectionFiles = files.filter((file) =>
file.relativePath.startsWith(`${section}/`),
);
if (sectionFiles.length === 0) {
console.log(` Skipping llms-full.txt for ${section} (no files found)`);
return;
}
// Build content
const sectionCapitalized = section.charAt(0).toUpperCase() + section.slice(1);
const header = `# Metabase ${sectionCapitalized} - Complete Reference for AI agents
> Table of contents: ${baseUrl}/${OUTPUT_FILE}
`;
// Add preamble if provided (e.g., version-specific notes, warnings)
const preamble = getPreamble ? getPreamble(branch) : "";
// Concatenate all documents
const docs = sectionFiles
.map((file) => readAndCleanMarkdown(file.fullPath))
.join("\n");
// Write llms-full.txt file at repo root
const content = header + preamble + "\n" + docs;
const llmsFullPath = `llms-${section}-full.txt`;
fs.writeFileSync(llmsFullPath, content, "utf8");
console.log(
`✅ Generated ${llmsFullPath} with ${sectionFiles.length} documents`,
);
}
/**
* Extract title from markdown file frontmatter or filename.
*
* We try to extract the title in this order: 1) YAML frontmatter
* 2) first H1 heading, and 3) the Markdown file name.
*
* Example Markdown and frontmatter format:
* ```markdown
* ---
* title: "Getting Started"
* description: "A guide"
* ---
* # Getting Started
* ```
*
* @param {string} content - file content
* @param {string} filePath - relative file path
* @returns {string} title of content
*/
function extractTitle(content, filePath) {
// Try to extract title from YAML frontmatter
// First, match against the YAML frontmatter delimiter `---`
const frontmatterMatch = content.match(/^---\s*\n([\s\S]*?)\n---\s*\n/);
if (frontmatterMatch) {
// Then, match against the title property
// Example: `title: "My Page"`
const frontmatter = frontmatterMatch[1];
const titleMatch = frontmatter.match(/^title:\s*(.+)$/m);
// Remove quotes if present
if (titleMatch) {
return titleMatch[1].trim().replace(/^["']|["']$/g, "");
}
}
// Try to extract title from first H1 heading
// Example: `# My Page`
const h1Match = content.match(/^#\s+(.+)$/m);
if (h1Match) {
return h1Match[1].trim();
}
// Fallback to filename, e.g. `my-page.md`
const filename = path.basename(filePath, ".md");
// Convert kebab-case or snake_case to Title Case
// `my-page.md` would become "My Page"
return filename
.split(/[-_]/)
.map((word) => word.charAt(0).toUpperCase() + word.slice(1))
.join(" ");
}
/**
* Read and clean markdown file content
*
* @param {string} filePath - full path to the markdown file
* @returns {string} cleaned markdown content with separator
*/
function readAndCleanMarkdown(filePath) {
let content = fs.readFileSync(filePath, "utf8");
// Strip YAML frontmatter
content = content.replace(/^---\s*\n[\s\S]*?\n---\s*\n/, "");
// Strip Jekyll/Liquid template syntax
content = content.replace(/\{%.*?%\}/g, "");
content = content.replace(/\{\{.*?\}\}/g, "");
return `${content.trim()}\n\n---\n`;
}
/**
* Check if branch version is above target version
* @param {string} branch - Branch name (e.g., 'master', 'release-x.58.x')
* @param {number} targetVersion - Target version number
* @returns {boolean}
*/
function aboveVersion(branch, targetVersion) {
if (branch === "master") {
return true;
}
// Parse version from release branch like "release-x.58.x" -> 58
const match = branch.match(/release-x\.(\d+)\.x/);
if (!match) {
return false;
}
const version = parseInt(match[1], 10);
return version >= targetVersion;
}
// Main execution
if (require.main === module) {
const args = process.argv.slice(2);
const docsPath = args[0] || "docs";
const branch = args[1] || "master";
const resolvedDocsPath = path.resolve(docsPath);
const markdownFiles = getMarkdownFilesList(resolvedDocsPath);
// Sort all files upfront for consistent ordering across all generated files
markdownFiles.sort((a, b) => a.relativePath.localeCompare(b.relativePath));
generateLlmsTxt(branch, markdownFiles, LLMS_FULL_TO_GENERATE);
generateLlmsFullFiles(branch, markdownFiles, LLMS_FULL_TO_GENERATE);
}

View File

@@ -19,11 +19,12 @@ while [[ $# -gt 0 ]]; do
done
echo "Cloning python-runner repository..."
git clone --depth 1 https://${METABASE_AUTOMATION_USER_TOKEN}@github.com/metabase/python-runner-container.git python-runner
git clone https://${METABASE_AUTOMATION_USER_TOKEN}@github.com/metabase/python-runner-container.git python-runner
if [ "$SKIP_START" = false ]; then
echo "Starting python-runner..."
cd python-runner
git checkout origin/memory_improvements_2
make run-ci 1> ../python-runner.out.log 2> ../python-runner.err.log
make logs 1>> ../python-runner.out.log 2>> ../python-runner.err.log &
cd ..

28
.github/team.json vendored
View File

@@ -10,7 +10,14 @@
"metamben",
"wzimrin",
"ericnormand",
"galdre",
"galdre"
]
},
{
"name": "Drivers",
"label": ".Team/Drivers",
"members": [
"snoe",
"rileythomp"
]
},
@@ -55,8 +62,7 @@
"tsplude",
"lbrdnk",
"piranha",
"crisptrutski",
"undrfined"
"crisptrutski"
]
},
{
@@ -80,12 +86,16 @@
"eliotmetabase",
"npfitz",
"johnswanson",
"iethree",
"nvoxland",
"edpaget",
"deniskaber",
"alxnddr",
"bpander",
"andsalves",
"kevinkelleher12"
"gantoreno",
"kevinkelleher12",
"undrfined"
]
},
{
@@ -102,16 +112,6 @@
"uladzimirdev",
"egorgrushin"
]
},
{
"name": "Growth Eng",
"label": ".Team/GrowthEng",
"members": [
"iethree",
"bpander",
"gantoreno",
"losrebellos"
]
}
]
}

View File

@@ -15,7 +15,7 @@ jobs:
be-tests-mariadb:
if: ${{ !inputs.skip }}
runs-on: ubuntu-22.04
timeout-minutes: 40
timeout-minutes: 60
strategy:
fail-fast: false
matrix:
@@ -105,7 +105,6 @@ jobs:
steps:
- uses: actions/checkout@v4
- name: Test ${{ matrix.version.name }}
id: test-driver
uses: ./.github/actions/test-driver
with:
build-static-viz: ${{ matrix.job.build-static-viz }}
@@ -124,12 +123,11 @@ jobs:
aws-secret-access-key: ${{ secrets.AWS_TEST_RESULTS_SECRET_ACCESS_KEY }}
aws-region: ${{ vars.AWS_REGION }}
trunk-api-token: ${{ secrets.TRUNK_API_TOKEN }}
previous-step-outcome: ${{ steps.test-driver.outputs.test-outcome }}
be-tests-mysql:
if: ${{ !inputs.skip }}
runs-on: ubuntu-22.04
timeout-minutes: 40
timeout-minutes: 60
strategy:
fail-fast: false
matrix:
@@ -229,7 +227,6 @@ jobs:
steps:
- uses: actions/checkout@v4
- name: Test ${{ matrix.version.name }}
id: test-driver
uses: ./.github/actions/test-driver
with:
build-static-viz: ${{ matrix.job.build-static-viz }}
@@ -248,12 +245,11 @@ jobs:
aws-secret-access-key: ${{ secrets.AWS_TEST_RESULTS_SECRET_ACCESS_KEY }}
aws-region: ${{ vars.AWS_REGION }}
trunk-api-token: ${{ secrets.TRUNK_API_TOKEN }}
previous-step-outcome: ${{ steps.test-driver.outputs.test-outcome }}
be-tests-postgres:
if: ${{ !inputs.skip }}
runs-on: ubuntu-22.04
timeout-minutes: 40
timeout-minutes: 60
strategy:
fail-fast: false
matrix:
@@ -340,7 +336,6 @@ jobs:
steps:
- uses: actions/checkout@v4
- name: Test ${{ matrix.version.name }} (${{ matrix.job.name }})
id: test-driver
uses: ./.github/actions/test-driver
with:
build-static-viz: ${{ matrix.job.build-static-viz }}
@@ -359,7 +354,6 @@ jobs:
aws-secret-access-key: ${{ secrets.AWS_TEST_RESULTS_SECRET_ACCESS_KEY }}
aws-region: ${{ vars.AWS_REGION }}
trunk-api-token: ${{ secrets.TRUNK_API_TOKEN }}
previous-step-outcome: ${{ steps.test-driver.outputs.test-outcome }}
app-db-tests-result:
needs:

View File

@@ -1,33 +1,12 @@
name: Backend-Cloverage
# Module lists are generated dynamically by listing source directories.
# The setup-oss and setup-ee jobs list folders in src/metabase/ and
# enterprise/backend/src/metabase_enterprise/ respectively, convert
# underscores to dashes (for Clojure namespace convention), and output
# as JSON arrays for the matrix jobs to consume.
#
# jq flags: -R (raw string input), -s (slurp all lines into one string),
# -c (compact single-line output for GitHub Actions).
on:
workflow_dispatch:
schedule:
- cron: '0 10 * * 1-5' # every weekday at 5am US eastern time
jobs:
setup-oss:
runs-on: ubuntu-22.04
outputs:
modules: ${{ steps.get-modules.outputs.modules }}
steps:
- uses: actions/checkout@v4
- id: get-modules
run: |
modules=$(ls -1d src/metabase/*/ | xargs -n1 basename | tr '_' '-' | jq -R -s -c 'split("\n") | map(select(length > 0))')
echo "modules=$modules" >> $GITHUB_OUTPUT
be-linter-cloverage:
needs: setup-oss
# don't run this workflow on a cron for forks
if: ${{ github.event_name != 'schedule' || github.repository == 'metabase/metabase' }}
runs-on: ubuntu-22.04
@@ -35,7 +14,96 @@ jobs:
strategy:
fail-fast: false
matrix:
modules: ${{ fromJson(needs.setup-oss.outputs.modules) }}
modules:
- actions
- activity-feed
- analytics
- analyze
- api
- api-keys
- api-routes
- app-db
- appearance
- audit-app
- auth-provider
- batch-processing
- bookmarks
- bug-reporting
- cache
- channel
- classloader
- cloud-migration
- cmd
- collections
- config
- connection-pool
- content-translation
- content-verification
- core
- dashboards
- database-routing
- driver
- driver-api
- eid-translation
- embedding
- events
- formatter
- geojson
- glossary
- graph
- indexed-entities
- internal-stats
- legacy-mbql
- lib
- lib-be
- logger
- login-history
- model-persistence
- models
- native-query-snippets
- notification
- parameters
- permissions
- pivot
- plugins
- premium-features
- product-feedback
- public-sharing
- pulse
- queries
- query-permissions
- query-processor
- remote-sync
- request
- revisions
- sample-data
- search
- secrets
- segments
- server
- session
- settings
- setup
- sso
- startup
- store-api
- sync
- system
- task
- task-history
- testing-api
- tiles
- timeline
- types
- upload
- user-key-value
- users
- util
- version
- view-log
- warehouse-schema
- warehouses
- xrays
steps:
- uses: actions/checkout@v4
- name: Prepare front-end environment
@@ -44,7 +112,7 @@ jobs:
uses: ./.github/actions/prepare-backend
- name: Build static viz frontend
run: bun run build-static-viz
run: yarn build-static-viz
env:
MB_EDITION: ee
- name: Collect the test coverage
@@ -62,19 +130,7 @@ jobs:
files: ./target/coverage/codecov.json
flags: back-end
setup-ee:
runs-on: ubuntu-22.04
outputs:
modules: ${{ steps.get-modules.outputs.modules }}
steps:
- uses: actions/checkout@v4
- id: get-modules
run: |
modules=$(ls -1d enterprise/backend/src/metabase_enterprise/*/ | xargs -n1 basename | tr '_' '-' | jq -R -s -c 'split("\n") | map(select(length > 0))')
echo "modules=$modules" >> $GITHUB_OUTPUT
be-linter-cloverage-enterprise:
needs: setup-ee
# don't run this workflow on a cron for forks
if: ${{ github.event_name != 'schedule' || github.repository == 'metabase/metabase' }}
runs-on: ubuntu-22.04
@@ -82,7 +138,53 @@ jobs:
strategy:
fail-fast: false
matrix:
modules: ${{ fromJson(needs.setup-ee.outputs.modules) }}
modules:
- action-v2
- advanced-config
- advanced-permissions
- ai-entity-analysis
- ai-sql-fixer
- ai-sql-generation
- analytics
- api
- audit-app
- auth-provider
- billing
- bookmarks
- cache
- cloud-add-ons
- comments
- content-translation
- content-verification
- core
- dashboard-subscription-filters
- database-replication
- database-routing
- dependencies
- documents
- email
- embedding-hub
- gsheets
- harbormaster
- impersonation
- internal-stats
- llm
- metabot-v3
- permission-debug
- premium-features
- public-sharing
- remote-sync
- sandbox
- scim
- search
- semantic-search
- serialization
- snippet-collections
- sso
- stale
- transforms
- transforms-python
- upload-management
steps:
- uses: actions/checkout@v4
- name: Prepare front-end environment
@@ -91,7 +193,7 @@ jobs:
uses: ./.github/actions/prepare-backend
- name: Build static viz frontend
run: bun run build-static-viz
run: yarn build-static-viz
env:
MB_EDITION: ee
- name: Collect the test coverage

View File

@@ -39,7 +39,7 @@ jobs:
be-tests:
if: ${{ !inputs.skip }}
runs-on: ubuntu-22.04
timeout-minutes: 40
timeout-minutes: 60
strategy:
fail-fast: false
matrix:
@@ -104,7 +104,7 @@ jobs:
with:
java-version: "${{ matrix.java-version }}"
- name: Build static-viz frontend
run: bun run build-static-viz
run: yarn build-static-viz
env:
MB_EDITION: ${{ matrix.job.edition }}
@@ -130,7 +130,6 @@ jobs:
aws-secret-access-key: ${{ secrets.AWS_TEST_RESULTS_SECRET_ACCESS_KEY }}
aws-region: ${{ vars.AWS_REGION }}
trunk-api-token: ${{ secrets.TRUNK_API_TOKEN }}
previous-step-outcome: ${{ steps.run-java-tests.outcome }}
- name: Publish Test Report (JUnit)
uses: dorny/test-reporter@v1

View File

@@ -19,11 +19,9 @@ jobs:
steps:
- uses: actions/checkout@v4
with:
sparse-checkout: |
release
.github
- name: Prepare release scripts
uses: ./.github/actions/prepare-release-scripts
sparse-checkout: release
- name: Prepare build scripts
run: cd ${{ github.workspace }}/release && yarn && yarn build
- name: Send build start message
uses: actions/github-script@v7
env:
@@ -81,11 +79,9 @@ jobs:
exit 1
- uses: actions/checkout@v4
with:
sparse-checkout: |
release
.github
- name: Prepare release scripts
uses: ./.github/actions/prepare-release-scripts
sparse-checkout: release
- name: Prepare build scripts
run: cd ${{ github.workspace }}/release && yarn && yarn build
- name: Get Release Version
uses: actions/github-script@v7
id: canonical_version
@@ -125,11 +121,9 @@ jobs:
uses: actions/checkout@v4
with:
fetch-depth: 0 # IMPORTANT! to get all the branches
sparse-checkout: |
release
.github
- name: Prepare release scripts
uses: ./.github/actions/prepare-release-scripts
sparse-checkout: release
- name: Prepare build scripts
run: cd ${{ github.workspace }}/release && yarn && yarn build
- name: Get Release Branch
uses: actions/github-script@v7
id: release_branch
@@ -141,13 +135,12 @@ jobs:
const releaseBranch = getReleaseBranch(version);
return releaseBranch;
- name: Ensure that the specified commit exists in the ${{ steps.release_branch.outputs.result }} release branch
run: | # bash
run: |
RELEASE_BRANCH=${{ steps.release_branch.outputs.result }}
git checkout $RELEASE_BRANCH
git branch --contains ${{ inputs.commit }} | grep -q $RELEASE_BRANCH \
&& echo "Commit found in correct release branch" \
|| (echo "Commit not found in correct release branch" && exit 1)
git checkout -
get-build-requirements:
runs-on: ubuntu-22.04
@@ -159,11 +152,9 @@ jobs:
steps:
- uses: actions/checkout@v4
with:
sparse-checkout: |
release
.github
- name: Prepare release scripts
uses: ./.github/actions/prepare-release-scripts
sparse-checkout: release
- name: Prepare build scripts
run: cd ${{ github.workspace }}/release && yarn && yarn build
- name: Get build dependencies
uses: actions/github-script@v7
id: dependencies

View File

@@ -82,7 +82,7 @@ jobs:
# Frontend preparation and caches
# `prepare-frontend` action (1) prepares Node.js and (2) fetches the node_modules cache if it exists.
# It then (3) runs `bun install --frozen-lockfile` which ensures the local cache is fully populated.
# It then (3) runs `yarn install --frozen-lockfile --prefer-offline` which ensures the local cache is fully populated.
# IMPORTANT: Caching node_modules after patches are already applied will break any PR that attempts to either:
# - upgrade a patched library
# - change a patch
@@ -92,31 +92,31 @@ jobs:
with:
apply-patches: "false"
- name: Get bun cache directory
id: bun-cache
- name: Get yarn cache directory
id: yarn-cache-dir
run: |
BUN_CACHE_DIR=$(bun pm cache)
echo "Bun cache directory: $BUN_CACHE_DIR"
echo "dir=$BUN_CACHE_DIR" >> $GITHUB_OUTPUT
YARN_CACHE_DIR=$(yarn cache dir)
echo "Yarn cache directory: $YARN_CACHE_DIR"
echo "dir=$YARN_CACHE_DIR" >> $GITHUB_OUTPUT
- name: Ensure that Cypress executable is ready
run: |
bun run cypress install
bun run cypress cache path
bun run cypress cache list
bun run cypress verify
yarn cypress install
yarn cypress cache path
yarn cypress cache list
yarn cypress verify
shell: bash
# Run Prettier to populate its cache (stored in node_modules/.cache/prettier)
# so it gets saved as part of the node_modules cache below
- name: Run Prettier to populate cache
run: bun run lint-prettier-pure
run: yarn lint-prettier-pure
- name: Update node_modules cache
uses: ./.github/actions/update-cache
with:
path: |
${{ steps.bun-cache.outputs.dir }}
${{ steps.yarn-cache-dir.outputs.dir }}
node_modules
key: ${{ env.NODE_CACHE_KEY }}
@@ -128,10 +128,10 @@ jobs:
# ESLint cache (requires both backend and frontend)
- name: Compile CLJS
run: bun run build-pure:cljs
run: yarn build-pure:cljs
- name: Run ESLint to populate cache
run: bun run lint-eslint-pure
run: yarn lint-eslint-pure
- name: Update eslint cache
uses: ./.github/actions/update-cache

View File

@@ -26,13 +26,11 @@ jobs:
steps:
- uses: actions/checkout@v4
with:
sparse-checkout: |
release
.github
sparse-checkout: release
fetch-depth: ${{ env.isNewReleaseBranch == 'false' && 3 || 0 }} # we need the full history to get all the commit messages
ref: master # we want the logic from master, even though we're triggering on a release branch
- name: Prepare release scripts
uses: ./.github/actions/prepare-release-scripts
- name: Prepare build scripts
run: cd ${{ github.workspace }}/release && yarn && yarn build
- uses: actions/github-script@v7
if: ${{ env.isNewReleaseBranch == 'false' }}
name: Set milestone for single commit

View File

@@ -41,7 +41,7 @@ jobs:
- name: Prepare back-end environment
uses: ./.github/actions/prepare-backend
- name: Compile CLJS
run: NODE_ENV=development bun run build-pure:cljs
run: NODE_ENV=development yarn build-pure:cljs
- name: Publish to Chromatic
uses: chromaui/action@v10.2.0
env:

View File

@@ -230,12 +230,6 @@ jobs:
mkdir -p .claude-scratchpad
echo "Scratchpad directory created for Claude's temporary verification files"
- name: Prepare front-end environment
uses: ./.github/actions/prepare-frontend
- name: Prepare back-end environment
uses: ./.github/actions/prepare-backend
- name: Generate GitHub App Token for Claude
uses: actions/create-github-app-token@v1
id: claude-app-token
@@ -250,7 +244,7 @@ jobs:
github_token: ${{ steps.claude-app-token.outputs.token }}
prompt: ${{ env.BUG_FIX_PROMPT }}
claude_args: |
--allowedTools "Bash(bun run test-unit*),Bash(bun run lint-eslint-pure),Bash(./bin/mage*),Bash(clojure*),Bash(find*),Bash(ls*),Bash(grep*),Write,Read,Glob,Grep,Edit,MultiEdit"
--allowedTools "Bash(yarn test-unit*),Bash(yarn lint-eslint-pure),Bash(./bin/mage*),Bash(clojure*),Bash(find*),Bash(ls*),Bash(grep*),Write,Read,Glob,Grep,Edit,MultiEdit"
--disallowedTools "Bash(git commit*),Bash(git push*),Bash(gh*)"
- name: Clean up scratchpad directory

View File

@@ -0,0 +1,98 @@
name: "Claude Code Review"
on:
pull_request:
types:
- closed
- synchronize
- labeled
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }}-claude-code-review
cancel-in-progress: true
jobs:
review:
if: |
(
github.event.action == 'closed' &&
github.event.pull_request.merged == true
) ||
contains(github.event.pull_request.labels.*.name, 'Claude Code Review')
runs-on: ubuntu-latest
permissions:
contents: read
pull-requests: write
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 1
# Make sure the value of GITHUB_TOKEN will not be persisted in repo's config
persist-credentials: false
# Apparently comments on specific lines and top-level comments about the PR as a whole are TWO TOTALLY DIFFERENT
# TYPES OF THINGS. So we have to fetch and delete separately.
- name: Delete old Claude PR review comments
env:
GH_TOKEN: ${{ github.token }}
REPO: ${{ github.repository }}
PR_NUMBER: ${{ github.event.pull_request.number }}
shell: bash
run: |
# Get old comment IDs from the bot
old_line_comment_ids=$(gh api "/repos/$REPO/pulls/$PR_NUMBER/comments" --jq '.[] | select(.user.login=="metabase-bot[bot]") | .id')
echo "Deleting line comments: $old_line_comment_ids"
# Delete each comment
for id in $old_line_comment_ids; do
gh api -X DELETE "/repos/$REPO/pulls/comments/$id"
echo "Deleted line comment $id"
done
# Get old top-level comment IDs (technically "issue" comments on the PR?)
old_top_level_comment_ids=$(gh api "/repos/$REPO/issues/$PR_NUMBER/comments" --jq '.[] | select(.user.login=="metabase-bot[bot]") | .id')
echo "Deleting top-level comments: $old_top_level_comment_ids"
# Delete each comment
for id in $old_top_level_comment_ids; do
gh api -X DELETE "/repos/$REPO/issues/comments/$id"
echo "Deleted top-level comment $id"
done
- name: Generate GitHub App Token for Claude
uses: actions/create-github-app-token@v1
id: claude-app-token
with:
app-id: ${{ secrets.METABASE_BOT_APP_ID }}
private-key: ${{ secrets.METABASE_BOT_APP_PRIVATE_KEY }}
permission-contents: read # Required for reading repository files and PR diffs
permission-pull-requests: write # Required for creating inline comments and PR comments
- name: Fetch Community Clojure Style Guide
run: >-
curl https://raw.githubusercontent.com/bbatsov/clojure-style-guide/refs/heads/master/README.adoc -o CLOJURE_STYLE_GUIDE.adoc
- uses: anthropics/claude-code-action@v1
timeout-minutes: 10
with:
anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
github_token: ${{ steps.claude-app-token.outputs.token }}
prompt: |
REPO: ${{ github.repository }}
PR NUMBER: ${{ github.event.pull_request.number }}
Review this pull request for compliance with project coding standards and conventions.
**Use the appropriate review skill based on the file types changed:**
- Use clojure-review for Clojure/ClojureScript files
- Use typescript-review for TypeScript/JavaScript files
- Follow the review guidelines and checklists in each skill
DO NOT post any comments confirming that something is correct - only post comments for style violations or concerns.
Note: The PR branch is already checked out in the current working directory.
Use `gh pr comment` for top-level feedback.
Use `mcp__github_inline_comment__create_inline_comment` to highlight specific code issues.
Only post GitHub comments - don't submit review text as messages.
claude_args: |
--allowedTools "mcp__github_inline_comment__create_inline_comment,Bash(gh pr comment:*),Bash(gh pr diff:*),Bash(gh pr view:*),ReadFile(CLOJURE_STYLE_GUIDE.adoc)"

103
.github/workflows/claude-docs-review.yml vendored Normal file
View File

@@ -0,0 +1,103 @@
name: "Claude Docs Review"
on:
pull_request:
types:
- opened
- synchronize
- reopened
paths:
- 'docs/**/*.md'
- 'docs/**/*.mdx'
workflow_dispatch:
inputs:
pr_number:
description: 'PR number to review'
required: true
type: number
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }}-claude-docs-review
cancel-in-progress: true
jobs:
review:
runs-on: ubuntu-latest
if: github.event.pull_request.draft == false
permissions:
contents: read
pull-requests: write
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 1
# Make sure the value of GITHUB_TOKEN will not be persisted in repo's config
persist-credentials: false
- name: Generate GitHub App Token for Claude
uses: actions/create-github-app-token@v1
id: claude-app-token
with:
app-id: ${{ secrets.METABASE_BOT_APP_ID }}
private-key: ${{ secrets.METABASE_BOT_APP_PRIVATE_KEY }}
permission-contents: read # Required for reading repository files and PR diffs
permission-pull-requests: write # Required for creating inline comments and PR comments
- name: Delete old Claude docs review comments
env:
GH_TOKEN: ${{ steps.claude-app-token.outputs.token }}
REPO: ${{ github.repository }}
PR_NUMBER: ${{ github.event.pull_request.number || github.event.inputs.pr_number }}
shell: bash
run: |
# Delete pending reviews first (these can be deleted entirely)
pending_review_ids=$(gh api "/repos/$REPO/pulls/$PR_NUMBER/reviews" --jq '.[] | select(.user.login=="metabase-bot[bot]" and .state=="PENDING") | .id')
if [ -n "$pending_review_ids" ]; then
echo "Deleting pending reviews: $pending_review_ids"
for id in $pending_review_ids; do
gh api -X DELETE "/repos/$REPO/pulls/$PR_NUMBER/reviews/$id"
echo "Deleted pending review $id"
done
fi
# Delete individual inline comments from submitted reviews (can't delete the review itself)
old_line_comment_ids=$(gh api "/repos/$REPO/pulls/$PR_NUMBER/comments" --jq '.[] | select(.user.login=="metabase-bot[bot]") | .id')
if [ -n "$old_line_comment_ids" ]; then
echo "Deleting line comments: $old_line_comment_ids"
for id in $old_line_comment_ids; do
gh api -X DELETE "/repos/$REPO/pulls/comments/$id"
echo "Deleted line comment $id"
done
fi
# Delete top-level review comments
old_top_level_comment_ids=$(gh api "/repos/$REPO/issues/$PR_NUMBER/comments" --jq '.[] | select(.user.login=="metabase-bot[bot]") | .id')
if [ -n "$old_top_level_comment_ids" ]; then
echo "Deleting top-level comments: $old_top_level_comment_ids"
for id in $old_top_level_comment_ids; do
gh api -X DELETE "/repos/$REPO/issues/comments/$id"
echo "Deleted top-level comment $id"
done
fi
- uses: anthropics/claude-code-action@v1
timeout-minutes: 10
with:
anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
github_token: ${{ steps.claude-app-token.outputs.token }}
show_full_output: true
prompt: |
REPO: ${{ github.repository }}
PR NUMBER: ${{ github.event.pull_request.number || github.event.inputs.pr_number }}
Review the documentation changes in this pull request.
Use the docs-review skill to review all changed documentation files for compliance with the Metabase writing style guide.
IMPORTANT: Only review markdown files (.md and .mdx). Skip any .txt files or other non-markdown files.
DO NOT post any comments confirming that something is correct - only post comments for style violations or concerns.
claude_args: |
--allowedTools "Skill,mcp__github__get_pull_request,mcp__github__get_pull_request_files,mcp__github__create_pending_pull_request_review,mcp__github__get_pull_request_diff,mcp__github__add_comment_to_pending_review,mcp__github__add_pull_request_review_comment_to_pending_review,mcp__github__delete_pending_pull_request_review,mcp__github__submit_pending_pull_request_review"

View File

@@ -39,4 +39,4 @@ jobs:
- name: Prepare back-end environment
uses: ./.github/actions/prepare-backend
- name: Run Cljs tests
run: bun run test-cljs
run: yarn run test-cljs

View File

@@ -4,10 +4,8 @@ run-name: Containerize ${{ inputs.tag }} Uberjar
on:
workflow_call:
inputs:
edition:
description: Metabase edition (ee or oss)
artifact-name:
type: string
required: true
repo:
description: owner and repo like "metabase/metabase"
type: string
@@ -17,18 +15,14 @@ on:
type: string
release:
type: string
default: 'false'
default: 'true'
build-args:
type: string
workflow_dispatch:
inputs:
edition:
description: Metabase edition (ee or oss)
type: choice
options:
- ee
- oss
required: true
artifact-name:
description: usually something like "metabase-release-{{ edition }}-{{ hash }}-uberjar"
type: string
repo:
description: docker owner and repo like "metabase/metabase"
type: string
@@ -38,13 +32,6 @@ on:
commit:
description: the full commit hash
type: string
release:
description: Whether this is a release artifact (uses metabase-release- prefix)
type: choice
options:
- "false"
- "true"
default: "false"
jobs:
containerize:
@@ -58,14 +45,12 @@ jobs:
fetch-depth: 0
# we want the dockerfile from the commit we're building
- run: git checkout ${{ inputs.commit }} -- bin/docker
- name: Retrieve uberjar artifact
- name: Retrieve test build uberjar artifact
uses: ./.github/actions/fetch-artifact
with:
commit: ${{ inputs.commit }}
edition: ${{ inputs.edition }}
release: ${{ inputs.release }}
- name: Prepare release scripts
uses: ./.github/actions/prepare-release-scripts
name: ${{ inputs.artifact-name }}
- name: Prepare build scripts
run: cd ${{ github.workspace }}/release && yarn && yarn build
- name: Get platforms
id: platforms
uses: actions/github-script@v7

View File

@@ -55,13 +55,13 @@ jobs:
uses: ./.github/workflows/containerize-jar.yml
secrets: inherit
with:
edition: ${{ matrix.edition }}
commit: ${{ github.event.inputs.commit || github.event.pull_request.head.sha || github.sha }}
artifact-name: metabase-${{ matrix.edition }}-${{ github.event.inputs.commit || github.event.pull_request.head.sha || github.sha }}-uberjar
commit: ${{ github.event.inputs.commit || github.sha }}
repo: ${{ fromJson(needs.get-container-info.outputs[matrix.edition]).repo }}
tag: ${{ fromJson(needs.get-container-info.outputs[matrix.edition]).tag }}
release: "false"
build-args: |
GIT_COMMIT_SHA=${{ github.event.inputs.commit || github.event.pull_request.head.sha || github.sha }}
GIT_COMMIT_SHA=${{ github.event.inputs.commit || github.sha }}
run-trivy:
needs: [get-container-info, containerize]
@@ -74,7 +74,7 @@ jobs:
- name: Check out the code
uses: actions/checkout@v4
with:
ref: ${{ github.event.inputs.commit || github.event.pull_request.head.sha || github.sha }}
ref: ${{ github.event.inputs.commit || github.sha }}
- name: Run Trivy vulnerability scanner
uses: aquasecurity/trivy-action@0.29.0
env:

View File

@@ -17,7 +17,7 @@ jobs:
uses: ./.github/actions/prepare-backend
- name: Run frontend unit tests
run: |
bun run test-unit \
yarn run test-unit \
--silent \
--coverage \
--testTimeout=60000

View File

@@ -72,14 +72,11 @@ jobs:
# All decision logic is in mage - it analyzes module dependencies,
# branch context, labels, and file changes to decide what runs.
MAGE_ARGS=(
--git-ref="${{ github.event.pull_request.base.ref || github.ref_name }}"
--git-ref="${{ github.event.pull_request.base.ref || 'master' }}"
--is-master-or-release="${{ github.ref_name == 'master' || startsWith(github.ref_name, 'release-') }}"
--pr-labels="${{ join(github.event.pull_request.labels.*.name, ',') }}"
--skip="${{ inputs.skip }}"
)
echo "Running:"
echo "./bin/mage -driver-decisions ${MAGE_ARGS[@]}"
echo ""
# First call: show human-readable output for debugging
./bin/mage -driver-decisions "${MAGE_ARGS[@]}"
# Second call: output only key=value lines for GITHUB_OUTPUT
@@ -88,9 +85,8 @@ jobs:
be-tests-athena-ee:
needs: determine-driver-skips
if: ${{ needs.determine-driver-skips.outputs.athena-should-run == 'true' || needs.determine-driver-skips.outputs.athena-quarantine-conflict == 'true' }}
runs-on: ubuntu-latest
timeout-minutes: 40
timeout-minutes: 60
env:
CI: 'true'
DRIVERS: athena
@@ -135,11 +131,9 @@ jobs:
aws-secret-access-key: ${{ secrets.AWS_TEST_RESULTS_SECRET_ACCESS_KEY }}
aws-region: ${{ vars.AWS_REGION }}
trunk-api-token: ${{ secrets.TRUNK_API_TOKEN }}
previous-step-outcome: ${{ steps.test-driver.outputs.test-outcome }}
be-tests-bigquery-cloud-sdk-ee:
needs: determine-driver-skips
if: ${{ needs.determine-driver-skips.outputs.bigquery-should-run == 'true' || needs.determine-driver-skips.outputs.bigquery-quarantine-conflict == 'true' }}
runs-on: ubuntu-latest
timeout-minutes: 60
strategy:
@@ -202,7 +196,6 @@ jobs:
aws-secret-access-key: ${{ secrets.AWS_TEST_RESULTS_SECRET_ACCESS_KEY }}
aws-region: ${{ vars.AWS_REGION }}
trunk-api-token: ${{ secrets.TRUNK_API_TOKEN }}
previous-step-outcome: ${{ steps.test-driver.outputs.test-outcome }}
- name: Cleanup python-runner
if: ${{ always() && matrix.job.needs-python-runner == true }}
uses: ./.github/actions/cleanup-python-runner
@@ -212,9 +205,8 @@ jobs:
be-tests-clickhouse-ee:
needs: determine-driver-skips
if: ${{ needs.determine-driver-skips.outputs.clickhouse-should-run == 'true' || needs.determine-driver-skips.outputs.clickhouse-quarantine-conflict == 'true' }}
runs-on: ubuntu-latest
timeout-minutes: 40
timeout-minutes: 60
strategy:
fail-fast: false
matrix:
@@ -285,7 +277,6 @@ jobs:
aws-secret-access-key: ${{ secrets.AWS_TEST_RESULTS_SECRET_ACCESS_KEY }}
aws-region: ${{ vars.AWS_REGION }}
trunk-api-token: ${{ secrets.TRUNK_API_TOKEN }}
previous-step-outcome: ${{ steps.test-driver.outputs.test-outcome }}
- name: Cleanup python-runner
if: ${{ always() && matrix.job.needs-python-runner == true }}
uses: ./.github/actions/cleanup-python-runner
@@ -295,9 +286,8 @@ jobs:
be-tests-druid-ee:
needs: determine-driver-skips
if: ${{ needs.determine-driver-skips.outputs.druid-should-run == 'true' || needs.determine-driver-skips.outputs.druid-quarantine-conflict == 'true' }}
runs-on: ubuntu-latest
timeout-minutes: 40
timeout-minutes: 60
env:
CI: 'true'
DRIVERS: druid
@@ -340,13 +330,11 @@ jobs:
aws-secret-access-key: ${{ secrets.AWS_TEST_RESULTS_SECRET_ACCESS_KEY }}
aws-region: ${{ vars.AWS_REGION }}
trunk-api-token: ${{ secrets.TRUNK_API_TOKEN }}
previous-step-outcome: ${{ steps.test-driver.outputs.test-outcome }}
be-tests-druid-jdbc-ee:
needs: determine-driver-skips
if: ${{ needs.determine-driver-skips.outputs.druid-jdbc-should-run == 'true' || needs.determine-driver-skips.outputs.druid-jdbc-quarantine-conflict == 'true' }}
runs-on: ubuntu-latest
timeout-minutes: 40
timeout-minutes: 60
env:
CI: 'true'
DRIVERS: druid-jdbc
@@ -389,13 +377,11 @@ jobs:
aws-secret-access-key: ${{ secrets.AWS_TEST_RESULTS_SECRET_ACCESS_KEY }}
aws-region: ${{ vars.AWS_REGION }}
trunk-api-token: ${{ secrets.TRUNK_API_TOKEN }}
previous-step-outcome: ${{ steps.test-driver.outputs.test-outcome }}
be-tests-databricks-ee:
needs: determine-driver-skips
if: ${{ needs.determine-driver-skips.outputs.databricks-should-run == 'true' || needs.determine-driver-skips.outputs.databricks-quarantine-conflict == 'true' }}
runs-on: ubuntu-latest
timeout-minutes: 40
timeout-minutes: 60
env:
CI: 'true'
DRIVERS: databricks
@@ -438,13 +424,11 @@ jobs:
aws-secret-access-key: ${{ secrets.AWS_TEST_RESULTS_SECRET_ACCESS_KEY }}
aws-region: ${{ vars.AWS_REGION }}
trunk-api-token: ${{ secrets.TRUNK_API_TOKEN }}
previous-step-outcome: ${{ steps.test-driver.outputs.test-outcome }}
be-tests-databricks-multi-catalog-ee:
needs: determine-driver-skips
if: ${{ needs.determine-driver-skips.outputs.databricks-should-run == 'true' || needs.determine-driver-skips.outputs.databricks-quarantine-conflict == 'true' }}
runs-on: ubuntu-latest
timeout-minutes: 40
timeout-minutes: 60
env:
CI: 'true'
DRIVERS: databricks
@@ -488,13 +472,11 @@ jobs:
aws-secret-access-key: ${{ secrets.AWS_TEST_RESULTS_SECRET_ACCESS_KEY }}
aws-region: ${{ vars.AWS_REGION }}
trunk-api-token: ${{ secrets.TRUNK_API_TOKEN }}
previous-step-outcome: ${{ steps.test-driver.outputs.test-outcome }}
be-tests-h2:
needs: determine-driver-skips
if: ${{ needs.determine-driver-skips.outputs.h2-should-run == 'true' }}
runs-on: ubuntu-latest
timeout-minutes: 40
timeout-minutes: 60
env:
CI: 'true'
name: H2
@@ -519,45 +501,11 @@ jobs:
aws-secret-access-key: ${{ secrets.AWS_TEST_RESULTS_SECRET_ACCESS_KEY }}
aws-region: ${{ vars.AWS_REGION }}
trunk-api-token: ${{ secrets.TRUNK_API_TOKEN }}
previous-step-outcome: ${{ steps.test-driver.outputs.test-outcome }}
be-tests-h2-oss:
needs: determine-driver-skips
if: ${{ needs.determine-driver-skips.outputs.h2-should-run == 'true' }}
runs-on: ubuntu-latest
timeout-minutes: 40
env:
CI: 'true'
name: H2 (OSS)
steps:
- uses: actions/checkout@v4
- name: Test H2 driver (OSS)
id: test-driver
uses: ./.github/actions/test-driver
with:
junit-name: 'be-tests-h2-oss'
edition: 'oss'
test-args: >-
:only-tags [:mb/driver-tests]
:exclude-tags [:mb/transforms-python-test]
- name: Upload Test Results
uses: ./.github/actions/upload-test-results
if: always()
with:
input-path: ./target/junit/
output-name: ${{ github.job }}
bucket: ${{ vars.AWS_S3_TEST_RESULTS_BUCKET }}
aws-access-key-id: ${{ secrets.AWS_TEST_RESULTS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_TEST_RESULTS_SECRET_ACCESS_KEY }}
aws-region: ${{ vars.AWS_REGION }}
trunk-api-token: ${{ secrets.TRUNK_API_TOKEN }}
previous-step-outcome: ${{ steps.test-driver.outputs.test-outcome }}
be-tests-mysql-mariadb:
needs: determine-driver-skips
if: ${{ needs.determine-driver-skips.outputs.mysql-mariadb-should-run == 'true' }}
runs-on: ubuntu-latest
timeout-minutes: 40
timeout-minutes: 60
strategy:
fail-fast: false
matrix:
@@ -682,7 +630,6 @@ jobs:
aws-secret-access-key: ${{ secrets.AWS_TEST_RESULTS_SECRET_ACCESS_KEY }}
aws-region: ${{ vars.AWS_REGION }}
trunk-api-token: ${{ secrets.TRUNK_API_TOKEN }}
previous-step-outcome: ${{ steps.test-driver.outputs.test-outcome }}
- name: Cleanup python-runner
if: always()
uses: ./.github/actions/cleanup-python-runner
@@ -692,9 +639,8 @@ jobs:
be-tests-mongo:
needs: determine-driver-skips
if: ${{ needs.determine-driver-skips.outputs.mongo-should-run == 'true' || needs.determine-driver-skips.outputs.mongo-quarantine-conflict == 'true' }}
runs-on: ubuntu-latest
timeout-minutes: 40
timeout-minutes: 60
strategy:
fail-fast: false
matrix:
@@ -763,7 +709,6 @@ jobs:
aws-secret-access-key: ${{ secrets.AWS_TEST_RESULTS_SECRET_ACCESS_KEY }}
aws-region: ${{ vars.AWS_REGION }}
trunk-api-token: ${{ secrets.TRUNK_API_TOKEN }}
previous-step-outcome: ${{ steps.test-driver.outputs.test-outcome }}
- name: Cleanup python-runner
if: ${{ always() && matrix.job.needs-python-runner == true }}
uses: ./.github/actions/cleanup-python-runner
@@ -773,9 +718,8 @@ jobs:
be-tests-mongo-ssl:
needs: determine-driver-skips
if: ${{ needs.determine-driver-skips.outputs.mongo-ssl-should-run == 'true' || needs.determine-driver-skips.outputs.mongo-ssl-quarantine-conflict == 'true' }}
runs-on: ubuntu-latest
timeout-minutes: 40
timeout-minutes: 60
strategy:
fail-fast: false
matrix:
@@ -846,13 +790,12 @@ jobs:
aws-secret-access-key: ${{ secrets.AWS_TEST_RESULTS_SECRET_ACCESS_KEY }}
aws-region: ${{ vars.AWS_REGION }}
trunk-api-token: ${{ secrets.TRUNK_API_TOKEN }}
previous-step-outcome: ${{ steps.test-driver.outputs.test-outcome }}
be-tests-mongo-sharded-cluster-ee:
needs: determine-driver-skips
if: ${{ needs.determine-driver-skips.outputs.mongo-sharded-cluster-should-run == 'true' || needs.determine-driver-skips.outputs.mongo-sharded-cluster-quarantine-conflict == 'true' }}
runs-on: ubuntu-latest
timeout-minutes: 40
timeout-minutes: 60
env:
CI: 'true'
DRIVERS: mongo
@@ -892,13 +835,11 @@ jobs:
aws-secret-access-key: ${{ secrets.AWS_TEST_RESULTS_SECRET_ACCESS_KEY }}
aws-region: ${{ vars.AWS_REGION }}
trunk-api-token: ${{ secrets.TRUNK_API_TOKEN }}
previous-step-outcome: ${{ steps.test-driver.outputs.test-outcome }}
be-tests-oracle:
needs: determine-driver-skips
if: ${{ needs.determine-driver-skips.outputs.oracle-should-run == 'true' || needs.determine-driver-skips.outputs.oracle-quarantine-conflict == 'true' }}
runs-on: ubuntu-latest
timeout-minutes: 40
timeout-minutes: 60
strategy:
fail-fast: false
matrix:
@@ -978,13 +919,11 @@ jobs:
aws-secret-access-key: ${{ secrets.AWS_TEST_RESULTS_SECRET_ACCESS_KEY }}
aws-region: ${{ vars.AWS_REGION }}
trunk-api-token: ${{ secrets.TRUNK_API_TOKEN }}
previous-step-outcome: ${{ steps.test-driver.outputs.test-outcome }}
be-tests-postgres:
needs: determine-driver-skips
if: ${{ needs.determine-driver-skips.outputs.postgres-should-run == 'true' }}
runs-on: ubuntu-latest
timeout-minutes: 40
timeout-minutes: 60
strategy:
fail-fast: false
matrix:
@@ -1068,7 +1007,6 @@ jobs:
uses: ./.github/actions/test-driver
with:
junit-name: 'be-tests-${{ matrix.version.junit-name }}'
logs-artifact-suffix: ${{ matrix.version.name }}-${{ matrix.job.name }}
test-args: >-
${{ matrix.job.test-args }}
:exclude-tags '[ ${{ matrix.job.exclude-tag }} ${{ env.__ADDITIONAL_EXCLUDED_TAG__ }}]'
@@ -1083,7 +1021,6 @@ jobs:
aws-secret-access-key: ${{ secrets.AWS_TEST_RESULTS_SECRET_ACCESS_KEY }}
aws-region: ${{ vars.AWS_REGION }}
trunk-api-token: ${{ secrets.TRUNK_API_TOKEN }}
previous-step-outcome: ${{ steps.test-driver.outputs.test-outcome }}
- name: Cleanup python-runner
if: always()
uses: ./.github/actions/cleanup-python-runner
@@ -1093,9 +1030,8 @@ jobs:
be-tests-presto-jdbc-ee:
needs: determine-driver-skips
if: ${{ needs.determine-driver-skips.outputs.presto-jdbc-should-run == 'true' || needs.determine-driver-skips.outputs.presto-jdbc-quarantine-conflict == 'true' }}
runs-on: ubuntu-latest
timeout-minutes: 40
timeout-minutes: 60
env:
CI: 'true'
DRIVERS: presto-jdbc
@@ -1171,13 +1107,11 @@ jobs:
aws-secret-access-key: ${{ secrets.AWS_TEST_RESULTS_SECRET_ACCESS_KEY }}
aws-region: ${{ vars.AWS_REGION }}
trunk-api-token: ${{ secrets.TRUNK_API_TOKEN }}
previous-step-outcome: ${{ steps.test-driver.outputs.test-outcome }}
be-tests-redshift-ee:
needs: determine-driver-skips
if: ${{ needs.determine-driver-skips.outputs.redshift-should-run == 'true' || needs.determine-driver-skips.outputs.redshift-quarantine-conflict == 'true' }}
runs-on: ubuntu-latest
timeout-minutes: 40
timeout-minutes: 60
strategy:
fail-fast: false
matrix:
@@ -1256,7 +1190,6 @@ jobs:
aws-secret-access-key: ${{ secrets.AWS_TEST_RESULTS_SECRET_ACCESS_KEY }}
aws-region: ${{ vars.AWS_REGION }}
trunk-api-token: ${{ secrets.TRUNK_API_TOKEN }}
previous-step-outcome: ${{ steps.test-driver.outputs.test-outcome }}
- name: Cleanup python-runner
if: ${{ always() && matrix.job.needs-python-runner == true }}
uses: ./.github/actions/cleanup-python-runner
@@ -1266,7 +1199,6 @@ jobs:
be-tests-snowflake-ee:
needs: determine-driver-skips
if: ${{ needs.determine-driver-skips.outputs.snowflake-should-run == 'true' || needs.determine-driver-skips.outputs.snowflake-quarantine-conflict == 'true' }}
runs-on: ubuntu-latest
timeout-minutes: 60
strategy:
@@ -1339,7 +1271,6 @@ jobs:
aws-secret-access-key: ${{ secrets.AWS_TEST_RESULTS_SECRET_ACCESS_KEY }}
aws-region: ${{ vars.AWS_REGION }}
trunk-api-token: ${{ secrets.TRUNK_API_TOKEN }}
previous-step-outcome: ${{ steps.test-driver.outputs.test-outcome }}
- name: Cleanup python-runner
if: ${{ always() && matrix.job.needs-python-runner == true }}
uses: ./.github/actions/cleanup-python-runner
@@ -1349,9 +1280,8 @@ jobs:
be-tests-sparksql-ee:
needs: determine-driver-skips
if: ${{ needs.determine-driver-skips.outputs.sparksql-should-run == 'true' || needs.determine-driver-skips.outputs.sparksql-quarantine-conflict == 'true' }}
runs-on: ubuntu-latest
timeout-minutes: 40
timeout-minutes: 60
env:
CI: 'true'
DRIVERS: sparksql
@@ -1392,13 +1322,11 @@ jobs:
aws-secret-access-key: ${{ secrets.AWS_TEST_RESULTS_SECRET_ACCESS_KEY }}
aws-region: ${{ vars.AWS_REGION }}
trunk-api-token: ${{ secrets.TRUNK_API_TOKEN }}
previous-step-outcome: ${{ steps.test-driver.outputs.test-outcome }}
be-tests-sqlite-ee:
needs: determine-driver-skips
if: ${{ needs.determine-driver-skips.outputs.sqlite-should-run == 'true' || needs.determine-driver-skips.outputs.sqlite-quarantine-conflict == 'true' }}
runs-on: ubuntu-latest
timeout-minutes: 40
timeout-minutes: 60
env:
CI: 'true'
DRIVERS: sqlite
@@ -1434,11 +1362,9 @@ jobs:
aws-secret-access-key: ${{ secrets.AWS_TEST_RESULTS_SECRET_ACCESS_KEY }}
aws-region: ${{ vars.AWS_REGION }}
trunk-api-token: ${{ secrets.TRUNK_API_TOKEN }}
previous-step-outcome: ${{ steps.test-driver.outputs.test-outcome }}
be-tests-sqlserver:
needs: determine-driver-skips
if: ${{ needs.determine-driver-skips.outputs.sqlserver-should-run == 'true' || needs.determine-driver-skips.outputs.sqlserver-quarantine-conflict == 'true' }}
runs-on: ubuntu-latest
timeout-minutes: 30
strategy:
@@ -1511,7 +1437,6 @@ jobs:
aws-secret-access-key: ${{ secrets.AWS_TEST_RESULTS_SECRET_ACCESS_KEY }}
aws-region: ${{ vars.AWS_REGION }}
trunk-api-token: ${{ secrets.TRUNK_API_TOKEN }}
previous-step-outcome: ${{ steps.test-driver.outputs.test-outcome }}
- name: Cleanup python-runner
if: ${{ always() && matrix.job.needs-python-runner == true }}
uses: ./.github/actions/cleanup-python-runner
@@ -1524,7 +1449,7 @@ jobs:
# needs: determine-driver-skips
# if: ${{ !inputs.skip && !(needs.determine-driver-skips.outputs.skip == 'true') }}
# runs-on: ubuntu-latest
# timeout-minutes: 40
# timeout-minutes: 60
# env:
# CI: 'true'
# DRIVERS: starburst
@@ -1560,7 +1485,6 @@ jobs:
login-to-ecr:
needs: determine-driver-skips
if: ${{ needs.determine-driver-skips.outputs.vertica-should-run == 'true' || needs.determine-driver-skips.outputs.vertica-quarantine-conflict == 'true' }}
runs-on: ubuntu-latest
timeout-minutes: 15
permissions:
@@ -1590,9 +1514,8 @@ jobs:
be-tests-vertica-ee:
needs: [determine-driver-skips, login-to-ecr]
if: ${{ needs.determine-driver-skips.outputs.vertica-should-run == 'true' || needs.determine-driver-skips.outputs.vertica-quarantine-conflict == 'true' }}
runs-on: ubuntu-latest
timeout-minutes: 40
timeout-minutes: 60
env:
CI: 'true'
DRIVERS: vertica
@@ -1638,7 +1561,6 @@ jobs:
aws-secret-access-key: ${{ secrets.AWS_TEST_RESULTS_SECRET_ACCESS_KEY }}
aws-region: ${{ vars.AWS_REGION }}
trunk-api-token: ${{ secrets.TRUNK_API_TOKEN }}
previous-step-outcome: ${{ steps.test-driver.outputs.test-outcome }}
drivers-tests-result:
needs:

View File

@@ -115,14 +115,14 @@ jobs:
- name: Build Embedding SDK package
if: ${{ needs.determine-test-type.outputs.type == 'component' }}
run: bun run build-embedding-sdk-package
run: yarn build-embedding-sdk-package
- name: Prepare Cypress environment
id: cypress-prep
uses: ./.github/actions/prepare-cypress
- name: Compile CLJS
run: bun run build-pure:cljs
run: yarn build-pure:cljs
- name: Run Metabase
run: node e2e/runner/run_cypress_ci.js start

View File

@@ -41,13 +41,12 @@ env:
jobs:
e2e-tests:
runs-on: ${{ inputs.runner }}
timeout-minutes: 40
timeout-minutes: 45
name: e2e-tests-${{ inputs.name }}-${{ inputs.edition }}
env:
MB_EDITION: ${{ inputs.edition }}
# Sensitive env vars (tokens, secrets) starting with `CYPRESS_` are accessed via async `cy.env()`
# Example: `cy.env(["FOO"]).then(({ FOO }) => ...)`
# For non-sensitive config, use `Cypress.expose()` in cypress config instead (synchronous)
# Any env starting with `CYPRESS_` will be available to all Cypress tests via `Cypress.env()`
# Example: you can get `CYPRESS_FOO` with `Cypress.env("FOO")`
CYPRESS_MB_ALL_FEATURES_TOKEN: ${{ secrets.STAGING_MB_ALL_FEATURES_TOKEN }}
CYPRESS_MB_STARTER_CLOUD_TOKEN: ${{ secrets.STAGING_MB_STARTER_CLOUD_TOKEN }}
CYPRESS_MB_PRO_CLOUD_TOKEN: ${{ secrets.STAGING_MB_PRO_CLOUD_TOKEN }}
@@ -80,8 +79,7 @@ jobs:
- name: Retrieve uberjar artifact for ${{ inputs.edition }}
uses: ./.github/actions/fetch-artifact
with:
commit: ${{ github.event.pull_request.head.sha || github.sha }}
edition: ${{ inputs.edition }}
name: metabase-${{ inputs.edition }}-${{ github.event.pull_request.head.sha || github.sha }}-uberjar
extract-path: "target/uberjar"
output-name: "metabase.jar"
@@ -107,7 +105,7 @@ jobs:
run: echo "specs=$(cat failed-specs)" >> $GITHUB_ENV
- name: Compile CLJS
run: bun run build-pure:cljs
run: yarn build-pure:cljs
- name: Run Metabase
run: node e2e/runner/run_cypress_ci.js start
@@ -172,7 +170,6 @@ jobs:
# Related to Trunk uploader
variant: e2e-tests
trunk-api-token: ${{ secrets.TRUNK_API_TOKEN }}
previous-step-outcome: ${{ inputs.specs && steps.tagged-e2e.outcome || steps.split-e2e.outcome }}
- name: Upload Cypress Artifacts upon failure
uses: actions/upload-artifact@v4

View File

@@ -19,9 +19,8 @@ jobs:
react-version: [18, 19]
fail-fast: false
env:
# Sensitive env vars (tokens, secrets) starting with `CYPRESS_` are accessed via async `cy.env()`
# Example: `cy.env(["FOO"]).then(({ FOO }) => ...)`
# For non-sensitive config, use `Cypress.expose()` in cypress config instead (synchronous)
# Any env starting with `CYPRESS_` will be available to all Cypress tests via `Cypress.env()`
# Example: you can get `CYPRESS_FOO` with `Cypress.env("FOO")`
CYPRESS_MB_ALL_FEATURES_TOKEN: ${{ secrets.STAGING_MB_ALL_FEATURES_TOKEN }}
CYPRESS_MB_STARTER_CLOUD_TOKEN: ${{ secrets.STAGING_MB_STARTER_CLOUD_TOKEN }}
CYPRESS_MB_PRO_CLOUD_TOKEN: ${{ secrets.STAGING_MB_PRO_CLOUD_TOKEN }}
@@ -46,8 +45,7 @@ jobs:
- name: Retrieve uberjar artifact for ee
uses: ./.github/actions/fetch-artifact
with:
commit: ${{ github.event.pull_request.head.sha || github.sha }}
edition: ee
name: metabase-ee-${{ github.event.pull_request.head.sha || github.sha }}-uberjar
extract-path: "target/uberjar"
output-name: "metabase.jar"
@@ -82,7 +80,6 @@ jobs:
run: ./bin/embedding-sdk/change-react-version.bash ${{ matrix.react-version }}
- name: Run component tests for Embedding SDK
id: run-tests
env:
CYPRESS_BROWSER: ${{ steps.cypress-prep.outputs.chrome-path }}
run: node e2e/runner/run_cypress_ci.js component
@@ -98,7 +95,6 @@ jobs:
aws-secret-access-key: ${{ secrets.AWS_TEST_RESULTS_SECRET_ACCESS_KEY }}
aws-region: ${{ vars.AWS_REGION }}
trunk-api-token: ${{ secrets.TRUNK_API_TOKEN }}
previous-step-outcome: ${{ steps.run-tests.outcome }}
- name: "SDK component tests > Upload Cypress Artifacts upon failure"
uses: actions/upload-artifact@v4
@@ -156,9 +152,8 @@ jobs:
env:
MB_RUN_MODE: "e2e"
METASTORE_DEV_SERVER_URL: ${{ secrets.METASTORE_DEV_SERVER_URL }}
# Sensitive env vars (tokens, secrets) starting with `CYPRESS_` are accessed via async `cy.env()`
# Example: `cy.env(["FOO"]).then(({ FOO }) => ...)`
# For non-sensitive config, use `Cypress.expose()` in cypress config instead (synchronous)
# Any env starting with `CYPRESS_` will be available to all Cypress tests via `Cypress.env()`
# Example: you can get `CYPRESS_FOO` with `Cypress.env("FOO")`
CYPRESS_MB_ALL_FEATURES_TOKEN: ${{ secrets.STAGING_MB_ALL_FEATURES_TOKEN }}
CYPRESS_MB_STARTER_CLOUD_TOKEN: ${{ secrets.STAGING_MB_STARTER_CLOUD_TOKEN }}
CYPRESS_MB_PRO_CLOUD_TOKEN: ${{ secrets.STAGING_MB_PRO_CLOUD_TOKEN }}
@@ -199,8 +194,7 @@ jobs:
id: download-uberjar
uses: ./.github/actions/fetch-artifact
with:
commit: ${{ steps.get-commit-sha.outputs.sha }}
edition: ee
name: metabase-ee-${{ steps.get-commit-sha.outputs.sha }}-uberjar
extract-path: "target/uberjar"
output-name: "metabase.jar"
continue-on-error: true # do not break the whole job if this step doesn't succeed
@@ -234,7 +228,6 @@ jobs:
--spec "e2e/snapshot-creators/default.cy.snap.js"
- name: Run component tests for Embedding SDK
id: run-tests
env:
CYPRESS_BROWSER: ${{ steps.cypress-prep.outputs.chrome-path }}
run: |
@@ -252,7 +245,6 @@ jobs:
aws-secret-access-key: ${{ secrets.AWS_TEST_RESULTS_SECRET_ACCESS_KEY }}
aws-region: ${{ vars.AWS_REGION }}
trunk-api-token: ${{ secrets.TRUNK_API_TOKEN }}
previous-step-outcome: ${{ steps.run-tests.outcome }}
- name: Upload Cypress Artifacts upon failure
uses: actions/upload-artifact@v4

View File

@@ -9,6 +9,7 @@ env:
TZ: US/Pacific # to make Node match the instance tz
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
HASH: ${{ github.event.pull_request.head.sha || github.sha }}-${{ github.run_attempt }}
PR_NUMBER: ${{ github.event.pull_request.number || '' }}
MB_EDITION: ee
jobs:
get-sample-apps-data:
@@ -20,7 +21,12 @@ jobs:
TEST_SUITES: |
[
"metabase-nodejs-react-sdk-embedding-sample-e2e",
"metabase-nextjs-sdk-embedding-sample-e2e"
"metabase-nextjs-sdk-embedding-sample-e2e",
"shoppy-e2e"
]
TEST_SUITES_TO_RUN_IN_MASTER_ONLY: |
[
"shoppy-e2e"
]
SAMPLE_APP_MAIN_BRANCH_NAME: "main"
SAMPLE_APP_STABLE_RELEASE_BRANCH_NAME_SUFFIX: "stable"
@@ -30,8 +36,8 @@ jobs:
steps:
- uses: actions/checkout@v4
- name: Prepare release scripts
uses: ./.github/actions/prepare-release-scripts
- name: Prepare build scripts
run: cd ${{ github.workspace }}/release && yarn && yarn build
- name: Build sample app compatibility data
uses: actions/github-script@v7
@@ -44,6 +50,7 @@ jobs:
} = require('${{ github.workspace }}/release/dist/index.cjs');
const testSuites = JSON.parse(process.env.TEST_SUITES);
const testSuitesToRunInMasterOnly = JSON.parse(process.env.TEST_SUITES_TO_RUN_IN_MASTER_ONLY);
const ref = '${{ github.base_ref || github.ref }}';
const branchName = ref.replace(/^refs\/heads\//, '');
@@ -67,16 +74,19 @@ jobs:
const sampleAppBranchName = `${currentBranchVersion}-${{ env.SAMPLE_APP_STABLE_RELEASE_BRANCH_NAME_SUFFIX }}`
const shouldRunTestsForVersion = (currentReleaseVersion - currentBranchVersion) <= (supportedReleasesCount - 1);
const releaseBranchTestSuites = testSuites.filter(testSuite => !testSuitesToRunInMasterOnly.includes(testSuite));
return {
branch_name: sampleAppBranchName,
matrix: shouldRunTestsForVersion ? testSuites : [],
matrix: shouldRunTestsForVersion
? releaseBranchTestSuites
: []
};
}
return {
branch_name: '',
matrix: [],
matrix: []
};
# In order to check compatibility between Sample Apps and Embedding SDK, the job for each sample app:
@@ -127,8 +137,7 @@ jobs:
- name: Retrieve uberjar artifact for ee
uses: ./.github/actions/fetch-artifact
with:
commit: ${{ github.event.pull_request.head.sha || github.sha }}
edition: ee
name: metabase-ee-${{ github.event.pull_request.head.sha || github.sha }}-uberjar
extract-path: "target/uberjar"
output-name: "metabase.jar"
@@ -158,8 +167,28 @@ jobs:
echo "sample_app_branch_exists=false" >> $GITHUB_OUTPUT
fi
- name: Prepare and launch Sample Apps
if: ${{ steps.check-sample-app-branch.outputs.sample_app_branch_exists == 'true' }}
- name: Configure AWS credentials
if: ${{ matrix.test_suite == 'shoppy-e2e' }}
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: ${{ secrets.SHOPPY_IAM_ROLE }}
role-session-name: GitHub_to_AWS_via_FederatedOIDC
aws-region: us-east-1
- name: Download Shoppy's App DB PostgreSQL dump from S3
if: ${{ matrix.test_suite == 'shoppy-e2e' }}
run: |
aws s3 cp s3://${{ secrets.SHOPPY_METABASE_APP_DB_S3_BUCKET }}/${{ secrets.SHOPPY_METABASE_APP_DB_S3_KEY }} ./e2e/tmp/db_dumps/shoppy_metabase_app_db_dump.sql
- name: Prepare and launch Sample Apps for shoppy-e2e test suite
if: ${{ steps.check-sample-app-branch.outputs.sample_app_branch_exists == 'true' && matrix.test_suite == 'shoppy-e2e' }}
env:
METABASE_APP_DB_USER: ${{ secrets.SHOPPY_METABASE_APP_DB_USER }}
METABASE_APP_DB: ${{ secrets.SHOPPY_METABASE_APP_DB_DATABASE }}
run: npx tsx ./e2e/runner/embedding-sdk/sample-apps/start-ci.ts shoppy-e2e
- name: Prepare and launch Sample Apps for other test suites
if: ${{ steps.check-sample-app-branch.outputs.sample_app_branch_exists == 'true' && matrix.test_suite != 'shoppy-e2e' }}
run: npx tsx ./e2e/runner/embedding-sdk/sample-apps/start-ci.ts ${{ matrix.test_suite }}
- name: Prune build cache
@@ -180,7 +209,7 @@ jobs:
uses: actions/upload-artifact@v4
if: ${{ steps.check-sample-app-branch.outputs.sample_app_branch_exists == 'true' && steps.run-e2e-tests.outcome != 'success' }}
with:
name: cypress-recording-sample-app-${{ matrix.test_suite }}-${{ matrix.environment }}-latest
name: cypress-recording-sample-app-${{ matrix.test_suite }}-latest
path: |
./e2e/tmp/${{ steps.get-sample-app-name.outputs.sample_app_name }}/cypress
if-no-files-found: ignore
@@ -190,149 +219,6 @@ jobs:
run: |
exit 1
# Shoppy e2e tests run separately using the shared composite action for setup.
e2e-shoppy-tests:
needs: [get-sample-apps-data]
strategy:
matrix:
environment: ["production", "development"]
if: |
!cancelled() && needs.get-sample-apps-data.result == 'success' &&
(github.ref == 'refs/heads/master' || github.base_ref == 'master')
runs-on: ubuntu-22.04
timeout-minutes: 45
name: e2e-sample-app-shoppy-e2e-tests-${{ matrix.environment }}-bundle
env:
MB_EDITION: ee
MB_RUN_MODE: e2e
METASTORE_DEV_SERVER_URL: ${{ secrets.METASTORE_DEV_SERVER_URL }}
CYPRESS_MB_ALL_FEATURES_TOKEN: ${{ secrets.STAGING_MB_ALL_FEATURES_TOKEN }}
SAMPLE_APP_BRANCH_NAME: ${{ needs.get-sample-apps-data.outputs.branch_name }}
SAMPLE_APP_ENVIRONMENT: ${{ matrix.environment }}
permissions:
id-token: write
steps:
- uses: actions/checkout@v4
- name: Prepare Shoppy E2E environment
uses: ./.github/actions/prepare-shoppy-e2e
id: shoppy-prep
with:
commit: ${{ github.event.pull_request.head.sha || github.sha }}
sample-app-branch-name: ${{ needs.get-sample-apps-data.outputs.branch_name }}
sample-app-environment: ${{ matrix.environment }}
staging-mb-all-features-token: ${{ secrets.STAGING_MB_ALL_FEATURES_TOKEN }}
shoppy-iam-role: ${{ secrets.SHOPPY_IAM_ROLE }}
shoppy-metabase-app-db-s3-bucket: ${{ secrets.SHOPPY_METABASE_APP_DB_S3_BUCKET }}
shoppy-metabase-app-db-s3-key: ${{ secrets.SHOPPY_METABASE_APP_DB_S3_KEY }}
shoppy-metabase-app-db-user: ${{ secrets.SHOPPY_METABASE_APP_DB_USER }}
shoppy-metabase-app-db-database: ${{ secrets.SHOPPY_METABASE_APP_DB_DATABASE }}
metastore-dev-server-url: ${{ secrets.METASTORE_DEV_SERVER_URL }}
shoppy-datadog-application-id: ${{ secrets.SHOPPY_METABASE_APP_DATADOG_APPLICATION_ID }}
shoppy-datadog-client-token: ${{ secrets.SHOPPY_METABASE_APP_DATADOG_CLIENT_TOKEN }}
- name: Run e2e tests for Shoppy
id: run-e2e-tests
if: ${{ steps.shoppy-prep.outputs.sample_app_branch_exists == 'true' }}
continue-on-error: true
env:
CYPRESS_BROWSER: ${{ steps.shoppy-prep.outputs.chrome-path }}
run: node e2e/runner/run_cypress_ci.js shoppy-e2e
- name: Upload Cypress Artifacts upon failure
uses: actions/upload-artifact@v4
if: ${{ steps.shoppy-prep.outputs.sample_app_branch_exists == 'true' && steps.run-e2e-tests.outcome != 'success' }}
with:
name: cypress-recording-sample-app-shoppy-e2e-${{ matrix.environment }}-latest
path: |
./e2e/tmp/shoppy/cypress
if-no-files-found: ignore
- name: Fail job if tests failed
if: ${{ steps.shoppy-prep.outputs.sample_app_branch_exists == 'true' && steps.run-e2e-tests.outcome != 'success' }}
run: |
exit 1
# Synthetic monitoring tests run only on master merges, in parallel with the regular e2e tests.
# They use the shoppy production environment and include performance measurement tests.
e2e-shoppy-synthetic-tests:
needs: [get-sample-apps-data]
if: |
!cancelled() && needs.get-sample-apps-data.result == 'success' &&
github.event_name == 'push' &&
github.ref == 'refs/heads/master'
runs-on: ubuntu-22.04
timeout-minutes: 45
name: e2e-shoppy-synthetic-monitoring-tests
env:
MB_EDITION: ee
MB_RUN_MODE: e2e
METASTORE_DEV_SERVER_URL: ${{ secrets.METASTORE_DEV_SERVER_URL }}
CYPRESS_MB_ALL_FEATURES_TOKEN: ${{ secrets.STAGING_MB_ALL_FEATURES_TOKEN }}
SAMPLE_APP_BRANCH_NAME: ${{ needs.get-sample-apps-data.outputs.branch_name }}
SAMPLE_APP_ENVIRONMENT: production
permissions:
id-token: write
contents: read
steps:
- uses: actions/checkout@v4
- name: Prepare Shoppy E2E environment
uses: ./.github/actions/prepare-shoppy-e2e
id: shoppy-prep
with:
commit: ${{ github.event.pull_request.head.sha || github.sha }}
sample-app-branch-name: ${{ needs.get-sample-apps-data.outputs.branch_name }}
sample-app-environment: production
staging-mb-all-features-token: ${{ secrets.STAGING_MB_ALL_FEATURES_TOKEN }}
shoppy-iam-role: ${{ secrets.SHOPPY_IAM_ROLE }}
shoppy-metabase-app-db-s3-bucket: ${{ secrets.SHOPPY_METABASE_APP_DB_S3_BUCKET }}
shoppy-metabase-app-db-s3-key: ${{ secrets.SHOPPY_METABASE_APP_DB_S3_KEY }}
shoppy-metabase-app-db-user: ${{ secrets.SHOPPY_METABASE_APP_DB_USER }}
shoppy-metabase-app-db-database: ${{ secrets.SHOPPY_METABASE_APP_DB_DATABASE }}
metastore-dev-server-url: ${{ secrets.METASTORE_DEV_SERVER_URL }}
shoppy-datadog-application-id: ${{ secrets.SHOPPY_METABASE_APP_DATADOG_APPLICATION_ID }}
shoppy-datadog-client-token: ${{ secrets.SHOPPY_METABASE_APP_DATADOG_CLIENT_TOKEN }}
- name: Get merged PR number
if: ${{ steps.shoppy-prep.outputs.sample_app_branch_exists == 'true' }}
id: get-merged-pr-number
run: |
PR=$(gh api repos/${{ github.repository }}/commits/${{ github.sha }}/pulls --jq '.[0].number' 2>/dev/null || echo "")
if [ -z "$PR" ] || [ "$PR" = "null" ]; then
echo "number=" >> "$GITHUB_OUTPUT"
else
echo "number=$PR" >> "$GITHUB_OUTPUT"
fi
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Run synthetic monitoring tests
id: run-e2e-tests
if: ${{ steps.shoppy-prep.outputs.sample_app_branch_exists == 'true' }}
continue-on-error: true
env:
CYPRESS_BROWSER: ${{ steps.shoppy-prep.outputs.chrome-path }}
PR_NUMBER: ${{ steps.get-merged-pr-number.outputs.number }}
INCLUDE_SYNTHETIC_MONITORING_TESTS: "true"
run: node e2e/runner/run_cypress_ci.js shoppy-e2e
- name: Upload Cypress Artifacts upon failure
uses: actions/upload-artifact@v4
if: ${{ steps.shoppy-prep.outputs.sample_app_branch_exists == 'true' && steps.run-e2e-tests.outcome != 'success' }}
with:
name: cypress-recording-shoppy-synthetic-tests-latest
path: |
./e2e/tmp/shoppy/cypress
if-no-files-found: ignore
- name: Fail job if tests failed
if: ${{ steps.shoppy-prep.outputs.sample_app_branch_exists == 'true' && steps.run-e2e-tests.outcome != 'success' }}
run: |
exit 1
# The job checks integration of the Embedding SDK with consumer's apps that use different frameworks/bundlers,
# or when we want to test some tricky integration cases like conflicting types.
# The job
@@ -366,8 +252,7 @@ jobs:
- name: Retrieve uberjar artifact for ee
uses: ./.github/actions/fetch-artifact
with:
commit: ${{ github.event.pull_request.head.sha || github.sha }}
edition: ee
name: metabase-ee-${{ github.event.pull_request.head.sha || github.sha }}-uberjar
extract-path: "target/uberjar"
output-name: "metabase.jar"

View File

@@ -23,7 +23,7 @@ jobs:
# Remove the matchers to not report false positives on every PR
# unrelated to embedding (EMB-765)
echo "::remove-matcher owner=tsc::"
bun run build-embedding-sdk-package
yarn build-embedding-sdk-package
- name: Prepare Embedding SDK artifact
uses: actions/upload-artifact@v4
with:

View File

@@ -23,10 +23,6 @@ on:
description: "Whether to run host and sample apps compatibility tests."
required: true
type: boolean
bundle-size-check:
description: "Whether to run bundle size check."
required: true
type: boolean
skip:
description: Whether to skip running tests due to one of the artifacts failing to build.
required: true
@@ -51,7 +47,7 @@ jobs:
name: embedding-sdk-${{ github.event.pull_request.head.sha || github.sha }}
path: ${{ github.workspace }}/resources/embedding-sdk
- name: Run frontend embedding SDK snippets type check
run: bun run embedding-sdk:cli-snippets:type-check
run: yarn run embedding-sdk:cli-snippets:type-check
embedding-sdk-docs-snippets-type-check:
if: inputs.docs-snippets-type-check && !inputs.skip
@@ -67,7 +63,7 @@ jobs:
name: embedding-sdk-${{ github.event.pull_request.head.sha || github.sha }}
path: ${{ github.workspace }}/resources/embedding-sdk
- name: Run frontend embedding SDK snippets type check
run: bun run embedding-sdk:docs-snippets:type-check
run: yarn run embedding-sdk:docs-snippets:type-check
embedding-sdk-api-documentation-generation-validation:
if: inputs.documentation-validation && !inputs.skip
@@ -90,7 +86,7 @@ jobs:
shell: bash
run: |
set -o pipefail
bun run embedding-sdk:docs:generate:pure --emit none 2>&1 \
yarn embedding-sdk:docs:generate:pure --emit none 2>&1 \
| tee /dev/stdout \
| tee >( tail -n 10 \
| sed '1s/^/output<<EOF\n/' \
@@ -102,14 +98,14 @@ jobs:
with:
remove-previous-comment: true
add-new-comment: ${{ steps.docs-generation.outcome == 'failure' && contains(steps.docs-generation.outputs.output, 'not included in the documentation') }}
message: "📝 Some exported types from `@metabase/embedding-sdk-react` reference other types that are not exported, i.e. internal types.<br><br>The internal type reference breaks API documentation generation for the SDK and also may unexpectedly increase the output size of the type declaration file.<br><br>Please check the CI logs to find the name of such type.<br><br>- If a type should be publicly available in the SDK, add a description to it or a TypeDoc annotation and export it from SDK.<br>- If a type is internal and should _not_ be publicly exposed, refactor types to prevent referencing other types that are not exported from SDK.<br><br>You can run `bun run embedding-sdk:docs:generate` locally to verify that the issue has been resolved."
message: "📝 Some exported types from `@metabase/embedding-sdk-react` reference other types that are not exported, i.e. internal types.<br><br>The internal type reference breaks API documentation generation for the SDK and also may unexpectedly increase the output size of the type declaration file.<br><br>Please check the CI logs to find the name of such type.<br><br>- If a type should be publicly available in the SDK, add a description to it or a TypeDoc annotation and export it from SDK.<br>- If a type is internal and should _not_ be publicly exposed, refactor types to prevent referencing other types that are not exported from SDK.<br><br>You can run `yarn embedding-sdk:docs:generate` locally to verify that the issue has been resolved."
include-log: true
- name: Comment on PR if 'Failed to resolve link' error happens
uses: ./.github/actions/notify-pull-request
with:
remove-previous-comment: true
add-new-comment: ${{ steps.docs-generation.outcome == 'failure' && contains(steps.docs-generation.outputs.output, 'Failed to resolve link') }}
message: "📝 A typedoc `{@link <TYPE_NAME>}` annotation is found that can't be resolved.<br><br>Most likely it happens when the referenced type is not exported. Please check the CI logs to find the name of such type.<br><br>- If a referenced in `{@link}` type should be publicly available in the SDK, add a description to it or a TypeDoc annotation and export it from SDK.<br>- If a type is internal and should _not_ be publicly exposed, change types to prevent unexported type reference in `{@link}` annotation.<br><br>You can run `bun run embedding-sdk:docs:generate` locally to verify that the issue has been resolved."
message: "📝 A typedoc `{@link <TYPE_NAME>}` annotation is found that can't be resolved.<br><br>Most likely it happens when the referenced type is not exported. Please check the CI logs to find the name of such type.<br><br>- If a referenced in `{@link}` type should be publicly available in the SDK, add a description to it or a TypeDoc annotation and export it from SDK.<br>- If a type is internal and should _not_ be publicly exposed, change types to prevent unexported type reference in `{@link}` annotation.<br><br>You can run `yarn embedding-sdk:docs:generate` locally to verify that the issue has been resolved."
include-log: true
- name: Fail Job
if: ${{ steps.docs-generation.outcome == 'failure' }}
@@ -128,52 +124,8 @@ jobs:
uses: ./.github/workflows/embedding-sdk-host-sample-apps-tests.yml
secrets: inherit
embedding-sdk-bundle-size-check:
if: inputs.bundle-size-check
continue-on-error: true
runs-on: ubuntu-22.04
timeout-minutes: 20
env:
BUNDLE_SIZE_THRESHOLD_PERCENT: 2
steps:
- uses: actions/checkout@v4
- uses: ./.github/actions/sdk-bundle-size-check
id: compare
with:
edition: ee
base-ref-commit: ${{ github.event.pull_request.base.sha }}
threshold-percent: ${{ env.BUNDLE_SIZE_THRESHOLD_PERCENT }}
- name: Comment on PR if bundle size has increased
uses: ./.github/actions/notify-pull-request
if: success() && steps.compare.outputs.status == 'increased'
with:
remove-previous-comment: true
add-new-comment: true
message: "⚠️ Embedding SDK bundle size increased by ${{ steps.compare.outputs.size_change_percent }}% compared to the ${{ github.base_ref }} branch (threshold: ${{ env.BUNDLE_SIZE_THRESHOLD_PERCENT }}%). Please contact Embedding team."
include-log: false
- name: Send notification to Slack via Zapier
if: success() && steps.compare.outputs.status == 'increased'
shell: bash
run: |
curl -s -X POST "${{ secrets.EMBEDDING_ZAPIER_WEBHOOK_URL }}" \
-d '⚠️ Embedding SDK bundle size increased by ${{ steps.compare.outputs.size_change_percent }}% (threshold: ${{ env.BUNDLE_SIZE_THRESHOLD_PERCENT }}%)
PR #${{ github.event.pull_request.number }} ${{ github.event.pull_request.title }} (${{ github.event.pull_request.html_url }})'
- name: Comment on PR if bundle size has decreased
uses: ./.github/actions/notify-pull-request
if: success() && steps.compare.outputs.status == 'decreased'
with:
remove-previous-comment: true
add-new-comment: true
message: "🎉 Embedding SDK bundle size decreased by ${{ steps.compare.outputs.size_change_percent }}% compared to the ${{ github.base_ref }} branch. Nice work!"
include-log: false
embedding-sdk-tests-result:
needs:
- embedding-sdk-bundle-size-check
- embedding-sdk-cli-snippets-type-check
- embedding-sdk-docs-snippets-type-check
- embedding-sdk-api-documentation-generation-validation

View File

@@ -44,12 +44,12 @@ jobs:
- name: Prepare back-end environment
uses: ./.github/actions/prepare-backend
- name: Build cljs
run: bun run build:cljs
run: yarn build:cljs
- name: Stress-test ${{ github.event.inputs.spec }} ${{ github.event.inputs.burn_in }} times
run: |
for i in {1..${{ github.event.inputs.burn_in }}}; do
bun run jest --runInBand --ci --silent --no-cache \
--testPathPatterns '${{ github.event.inputs.spec }}' \
yarn jest --runInBand --ci --silent --no-cache \
--testPathPattern '${{ github.event.inputs.spec }}' \
--testNamePattern '${{ github.event.inputs.grep }}'
if [[ "$?" != 0 ]]; then
echo "Failed after $i attempts" && break;

View File

@@ -46,15 +46,15 @@ jobs:
key: ${{ steps.get-cache-keys.outputs.eslint-cache-key }}
- name: Compile CLJS
run: bun run build-pure:cljs
run: yarn build-pure:cljs
- name: Find CSS variables that are used but never defined
run: bun run find-never-defined-css-variables
run: yarn find-never-defined-css-variables
- name: Run Prettier formatting linter
run: bun run lint-prettier-pure
run: yarn run lint-prettier-pure
- name: Run ESLint linter
run: bun run lint-eslint-pure
run: yarn run lint-eslint-pure
- name: Run CSS linter
run: bun run lint-css
run: yarn run lint-css
fe-type-check:
if: ${{ !inputs.skip }}
@@ -66,9 +66,9 @@ jobs:
uses: ./.github/actions/prepare-frontend
- name: Prepare back-end environment
uses: ./.github/actions/prepare-backend
- run: bun run build-pure:cljs
- run: yarn build-pure:cljs
name: Compile CLJS
- run: bun run type-check-pure
- run: yarn type-check-pure
name: Check types
fe-tests-unit:
@@ -90,8 +90,7 @@ jobs:
- name: Prepare back-end environment
uses: ./.github/actions/prepare-backend
- name: Run frontend unit tests
id: fe-tests
run: bun run test-unit --silent --shard=${{ matrix.shard }}/${{ env.SHARDS }}
run: yarn run test-unit --silent --shard=${{ matrix.shard }}/${{ env.SHARDS }}
# Trunk Quarantine controls the final state of the run in the CI!
# Even if this step fails we let it pass, and then `trunk-io/analytics-uploader` determines
# whether to fail or to pass the whole job, based on the quarantine list.
@@ -107,7 +106,6 @@ jobs:
aws-secret-access-key: ${{ secrets.AWS_TEST_RESULTS_SECRET_ACCESS_KEY }}
aws-region: ${{ vars.AWS_REGION }}
trunk-api-token: ${{ secrets.TRUNK_API_TOKEN }}
previous-step-outcome: ${{ steps.fe-tests.outcome }}
fe-tests-timezones:
if: ${{ !inputs.skip }}
@@ -120,7 +118,7 @@ jobs:
- name: Prepare back-end environment
uses: ./.github/actions/prepare-backend
- name: Run frontend timezones tests
run: bun run test-timezones
run: yarn run test-timezones
fe-tests-result:
needs:

56
.github/workflows/fuzzing.yml vendored Normal file
View File

@@ -0,0 +1,56 @@
name: Fuzzing
on:
workflow_dispatch:
push:
branches:
- "master"
- "release-**"
paths:
- "frontend/src/metabase/querying/expressions/**"
pull_request:
paths:
- "frontend/src/metabase/querying/expressions/**"
jobs:
fe-fuzz-lexifier:
runs-on: ubuntu-22.04
timeout-minutes: 15
steps:
- uses: actions/checkout@v4
- name: Prepare front-end environment
uses: ./.github/actions/prepare-frontend
- name: Prepare back-end environment
uses: ./.github/actions/prepare-backend
- run: yarn test-unit frontend/src/metabase/querying/expressions/pratt/fuzz.lexifier.unit.spec.ts
env:
MB_FUZZ: 1
name: Run fuzz testing on the lexifier
fe-fuzz-compiler:
runs-on: ubuntu-22.04
timeout-minutes: 15
steps:
- uses: actions/checkout@v4
- name: Prepare front-end environment
uses: ./.github/actions/prepare-frontend
- name: Prepare back-end environment
uses: ./.github/actions/prepare-backend
- run: yarn test-unit frontend/src/metabase/querying/expressions/fuzz.compiler.unit.spec.ts
env:
MB_FUZZ: 1
name: Run fuzz testing on the custom expression compiler
fe-fuzz-string-quoting:
runs-on: ubuntu-22.04
timeout-minutes: 15
steps:
- uses: actions/checkout@v4
- name: Prepare front-end environment
uses: ./.github/actions/prepare-frontend
- name: Prepare back-end environment
uses: ./.github/actions/prepare-backend
- run: yarn test-unit frontend/src/metabase/querying/expressions/fuzz.string.unit.spec.ts
env:
MB_FUZZ: 1
name: Run fuzz testing on the custom expression string quoter

View File

@@ -1,51 +0,0 @@
name: JS Stats Update
on:
workflow_dispatch:
schedule:
- cron: 40 1 * * 4 # every Thursday at 1:40
jobs:
js-stats:
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- uses: actions/checkout@v4
- name: get JS file count
id: get-js-file-count
run: |
oss=$(find ./frontend/src/metabase -type f -name "*.js" -o -name "*.jsx" | wc -l)
ee=$(find ./enterprise/frontend/src/metabase-enterprise -type f -name "*.js" -o -name "*.jsx" | wc -l)
echo $(($oss+$ee)) javascript files
echo "JS_FILES=$(($oss+$ee))" >> $GITHUB_OUTPUT
- name: Upload count to Stats
uses: actions/github-script@v7
env:
JS_FILES: ${{ steps.get-js-file-count.outputs.JS_FILES }}
API_KEY: ${{ secrets.STATS_API_KEY }}
with:
script: | # js
const { uploadCsvToMb } = require('${{ github.workspace }}/.github/scripts/csv-to-mb.js');
console.log("JS_FILES", Number(process.env.JS_FILES));
const data = [
{
'Date': new Date().toISOString().slice(0, 10), // YYYY-MM-DD
'Js files': Number(process.env.JS_FILES),
}
];
uploadCsvToMb({
baseUrl: 'https://stats.metabase.com',
tableId: 73270,
jsonData: data,
mode: 'append',
}).then(() => {
console.log('Data uploaded successfully');
}).catch((error) => {
console.error('Error uploading data:', error);
process.exit(1);
});

198
.github/workflows/llms-txt-generate.yml vendored Normal file
View File

@@ -0,0 +1,198 @@
name: Generate llms.txt file
on:
workflow_call:
inputs:
version:
description: 'Metabase version (e.g. v0.57.0)'
type: string
required: true
branch:
description: 'Target branch (e.g. release-x.57.x)'
type: string
required: true
workflow_dispatch:
inputs:
version:
description: 'Metabase version (e.g. v0.57.0)'
type: string
required: true
branch:
description: 'Target branch (e.g. release-x.57.x)'
type: string
required: true
permissions:
contents: write
pull-requests: write
jobs:
check-version:
name: Check if minor or major release
runs-on: ubuntu-22.04
timeout-minutes: 5
outputs:
should_generate: ${{ steps.check.outputs.should_generate }}
target_branch: ${{ steps.check.outputs.target_branch }}
version: ${{ steps.check.outputs.version }}
steps:
- name: Check version
id: check
run: |
VERSION="${{ inputs.version }}"
BRANCH="${{ inputs.branch }}"
# Extract version number (e.g., v0.57.0 -> 0.57.0)
VERSION_NUM="${VERSION#v}"
# Count dots to determine version type
# Metabase format: v0.MAJOR.MINOR (3 segments) or v0.MAJOR.MINOR.PATCH (4 segments)
DOT_COUNT=$(echo "$VERSION_NUM" | tr -cd '.' | wc -c | tr -d ' ')
# Only proceed if this is a minor or major release (3 segments)
# Examples of minor/major releases: v0.57.8, v0.59.0
# Skip for patch releases (4 segments): v0.57.8.1
# Note: This is a safety check even though the workflow should only be called
# for minor/major releases, just in case the workflow is dispatched manually.
if [ "$DOT_COUNT" -eq 3 ]; then
echo "⏭️ This is a patch release ($VERSION). Skipping llms.txt generation."
echo "should_generate=false" >> $GITHUB_OUTPUT
else
echo "✅ This is a minor or major release ($VERSION). Proceeding with llms.txt generation for branch: $BRANCH"
echo "should_generate=true" >> $GITHUB_OUTPUT
fi
echo "target_branch=$BRANCH" >> $GITHUB_OUTPUT
echo "version=$VERSION" >> $GITHUB_OUTPUT
generate-llms-txt:
name: Generate llms.txt
needs: check-version
if: needs.check-version.outputs.should_generate == 'true'
runs-on: ubuntu-22.04
timeout-minutes: 15
steps:
- name: Get the generate-llms-txt.js node script via sparse checkout
uses: actions/checkout@v4
with:
token: ${{ secrets.METABASE_AUTOMATION_USER_TOKEN }}
ref: ${{ github.ref }}
sparse-checkout: |
.github/scripts
path: workflow-scripts
- name: Checkout target branch for docs
uses: actions/checkout@v4
with:
token: ${{ secrets.METABASE_AUTOMATION_USER_TOKEN }}
ref: ${{ needs.check-version.outputs.target_branch }}
path: metabase
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: 22
- name: Generate llms.txt
id: generate
run: |
node workflow-scripts/.github/scripts/generate-llms-txt.js metabase/docs ${{ needs.check-version.outputs.target_branch }}
- name: Sanity check generated files
run: |
if [ ! -f "llms.txt" ]; then
echo "❌ Error: llms.txt was not generated"
exit 1
fi
if [ ! -f "llms-embedding-full.txt" ]; then
echo "❌ Error: llms-embedding-full.txt was not generated"
exit 1
fi
# Check that llms.txt contains URLs with the expected GitHub raw URL prefix
URL_COUNT=$(grep -o "https://raw.githubusercontent.com/metabase/metabase/refs/heads/${{ needs.check-version.outputs.target_branch }}/docs" llms.txt | wc -l)
if [ "$URL_COUNT" -lt 10 ]; then
echo "❌ Error: llms.txt appears to be malformed. Found only $URL_COUNT URLs (expected at least 10)"
exit 1
fi
echo "✅ Both llms.txt and llms-embedding-full.txt were generated successfully"
echo "✅ llms.txt contains $URL_COUNT documentation URLs"
- name: Set up git configuration
working-directory: metabase
run: |
git config user.name "github-actions[bot]"
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
BRANCH_NAME="llms-txt/auto-update-${{ needs.check-version.outputs.target_branch }}-$(date +%Y%m%d-%H%M%S)"
echo "BRANCH_NAME=$BRANCH_NAME" >> $GITHUB_ENV
echo "Branch name will be: $BRANCH_NAME"
- name: Copy generated files to metabase repo
run: |
cp llms.txt metabase/
cp llms-embedding-full.txt metabase/
- name: Create Pull Request
id: create_pr
uses: peter-evans/create-pull-request@v6
with:
token: ${{ secrets.METABASE_AUTOMATION_USER_TOKEN }}
path: metabase
branch: ${{ env.BRANCH_NAME }}
base: ${{ needs.check-version.outputs.target_branch }}
commit-message: |
Update llms.txt and llms-full.txt (${{ needs.check-version.outputs.target_branch }})
Regenerated llms.txt to include all markdown files from the docs/ directory.
Also generated llms-full.txt files for the embedding section.
🤖 Generated automatically by llms-txt-generate workflow
Co-Authored-By: GitHub Actions <actions@github.com>
committer: "github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>"
author: "github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>"
title: "[chore] Update llms.txt for ${{ needs.check-version.outputs.target_branch }}"
body: |
## Summary
This PR contains automatically generated updates to `llms.txt` and `llms-embedding-full.txt` files, triggered by a minor or major release.
## Changes
- The `llms.txt` file has been regenerated to include all markdown files from the `docs/` directory.
- The `llms-embedding-full.txt` file has been generated for the embedding section (concatenated documentation for RAG).
## Generation
These files were generated using:
- Script: `.github/scripts/generate-llms-txt.js`
- Branch: `${{ needs.check-version.outputs.target_branch }}`
- Base URL: `https://raw.githubusercontent.com/metabase/metabase/refs/heads/${{ needs.check-version.outputs.target_branch }}`
Please review the changes and merge if they look correct.
🤖 Generated automatically by the `llms-txt-generate` workflow.
Co-Authored-By: GitHub Actions <actions@github.com>
labels: |
Type:Documentation
no-backport
add-paths: |
llms.txt
llms-embedding-full.txt
- name: Auto approve PR
if: steps.create_pr.outputs.pull-request-number != ''
working-directory: metabase
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
gh pr review "${{ steps.create_pr.outputs.pull-request-number }}" --approve --body "Auto-approved by llms-txt-generate workflow"
- name: Auto merge PR
if: steps.create_pr.outputs.pull-request-number != ''
working-directory: metabase
env:
GH_TOKEN: ${{ secrets.METABASE_AUTOMATION_USER_TOKEN }}
run: |
gh pr merge "${{ steps.create_pr.outputs.pull-request-number }}" --auto --squash

View File

@@ -1,15 +1,10 @@
name: Loki Visual Stress Test Flake Fix
on:
pull_request:
types: [opened, synchronize, reopened]
paths:
- 'frontend/**/*.stories.tsx'
- 'enterprise/frontend/**/*.stories.tsx'
workflow_dispatch:
inputs:
story_files:
description: "Comma-separated story file paths relative to repo root"
stories_filter:
description: "Regex pattern to filter stories (e.g. 'DataGrid' or 'static-viz|viz')"
type: string
required: true
burn_in:
@@ -19,96 +14,22 @@ on:
default: "50"
jobs:
detect-changed-stories:
name: Detect changed stories
runs-on: ubuntu-22.04
timeout-minutes: 5
outputs:
story_files: ${{ steps.detect.outputs.story_files }}
should_run: ${{ steps.detect.outputs.should_run }}
steps:
- name: Detect changed story files
id: detect
env:
GH_TOKEN: ${{ github.token }}
run: |
validate_paths() {
local files="$1"
for file in $(echo "$files" | tr ',' '\n'); do
file=$(echo "$file" | xargs) # trim whitespace
if [[ -z "$file" ]]; then
continue
fi
if [[ ! "$file" =~ ^(frontend/|enterprise/frontend/).+\.stories\. ]]; then
echo "::error::Invalid story path: $file"
echo "Paths must start with 'frontend/' or 'enterprise/frontend/' and contain '.stories.'"
return 1
fi
done
return 0
}
if [[ "${{ github.event_name }}" == "workflow_dispatch" ]]; then
FILES=$(echo "${{ inputs.story_files }}" | tr ',' '\n' | xargs -n1 | paste -sd',')
if ! validate_paths "$FILES"; then
exit 1
fi
echo "story_files=$FILES" >> $GITHUB_OUTPUT
echo "should_run=true" >> $GITHUB_OUTPUT
exit 0
fi
FILES=$( \
gh pr view ${{ github.event.pull_request.number }} --repo ${{ github.repository }} --json files --jq '.files[].path' \
| grep '\.stories\.tsx$' \
| xargs -n1 \
| sort -u \
| paste -sd',' \
)
if [[ -z "$FILES" ]]; then
echo "No changed story files found"
echo "should_run=false" >> $GITHUB_OUTPUT
exit 0
fi
if ! validate_paths "$FILES"; then
exit 1
fi
echo "Changed story files: $FILES"
echo "story_files=$FILES" >> $GITHUB_OUTPUT
echo "should_run=true" >> $GITHUB_OUTPUT
workflow-summary:
name: Summary
needs: detect-changed-stories
if: needs.detect-changed-stories.outputs.should_run == 'true'
name: Stress test inputs
runs-on: ubuntu-22.04
timeout-minutes: 5
steps:
- name: Generate workflow summary
run: |
echo '## Loki Stress Test' >> $GITHUB_STEP_SUMMARY
echo '**Inputs:**' >> $GITHUB_STEP_SUMMARY
echo '' >> $GITHUB_STEP_SUMMARY
echo '**Branch:** `${{ github.head_ref || github.ref_name }}`' >> $GITHUB_STEP_SUMMARY
echo '- `branch`: ${{ github.ref_name }}' >> $GITHUB_STEP_SUMMARY
echo '- `stories_filter`: ${{ inputs.stories_filter }}' >> $GITHUB_STEP_SUMMARY
echo '- `burn_in`: ${{ inputs.burn_in }}' >> $GITHUB_STEP_SUMMARY
echo '' >> $GITHUB_STEP_SUMMARY
echo '**Story files:**' >> $GITHUB_STEP_SUMMARY
echo '${{ needs.detect-changed-stories.outputs.story_files }}' | tr ',' '\n' | while read -r file; do
echo "- \`$file\`" >> $GITHUB_STEP_SUMMARY
done
echo '' >> $GITHUB_STEP_SUMMARY
echo '**Burn-in:** ${{ inputs.burn_in || 50 }} runs' >> $GITHUB_STEP_SUMMARY
echo '' >> $GITHUB_STEP_SUMMARY
if [[ "${{ github.event_name }}" == "pull_request" ]]; then
echo 'Triggered by PR #${{ github.event.pull_request.number }} from @${{ github.event.pull_request.user.login }}' >> $GITHUB_STEP_SUMMARY
else
echo 'Triggered manually by @${{ github.actor }}' >> $GITHUB_STEP_SUMMARY
fi
echo 'triggered by: @${{ github.event.sender.login }}' >> $GITHUB_STEP_SUMMARY
stress-test-loki:
needs: detect-changed-stories
if: needs.detect-changed-stories.outputs.should_run == 'true'
runs-on: ubuntu-latest
timeout-minutes: 60
name: Stress test Loki visual tests
@@ -135,20 +56,17 @@ jobs:
uses: ./.github/actions/prepare-backend
- name: Compile CLJS
run: NODE_ENV=development bun run build-pure:cljs
run: NODE_ENV=development yarn build-pure:cljs
- name: Build Storybook
run: bun run build-storybook
env:
STORYBOOK_STORIES_FILTER: ${{ needs.detect-changed-stories.outputs.story_files }}
- name: Build Storybook (once)
run: yarn build-storybook
- name: Stress-test Loki
- name: Stress-test Loki "${{ inputs.stories_filter }}" ${{ inputs.burn_in }} times
run: |
BURN_IN=${{ inputs.burn_in || '50' }}
STORY_FILES='${{ needs.detect-changed-stories.outputs.story_files }}'
BURN_IN=${{ inputs.burn_in }}
STORIES_FILTER='${{ inputs.stories_filter }}'
echo "Story files: $STORY_FILES"
echo "Running Loki $BURN_IN times"
echo "Running Loki visual tests $BURN_IN times for stories matching: $STORIES_FILTER"
echo ""
FAILED=0
@@ -157,9 +75,10 @@ jobs:
echo "Run $i of $BURN_IN"
echo "=========================================="
if bun run test-visual:loki \
if yarn test-visual:loki \
--reactUri file:./storybook-static \
--verboseRenderer; then
--verboseRenderer \
--storiesFilter "$STORIES_FILTER"; then
echo "✅ Run $i passed"
else
echo "❌ Run $i FAILED"
@@ -182,9 +101,10 @@ jobs:
- name: Generate Visual Report on Failure
if: failure()
run: bun run test-visual:loki-report
run: yarn test-visual:loki-report
- name: Upload Artifact
id: artifact-upload-step
if: failure()
uses: actions/upload-artifact@v4
with:

View File

@@ -22,10 +22,10 @@ jobs:
uses: ./.github/actions/prepare-backend
- name: Compile CLJS
run: bun run build-pure:cljs
run: yarn build-pure:cljs
- name: Update Loki Snapshots
run: bun run test-visual:loki:ci || bun run test-visual:loki-approve-diff && bun run test-visual:loki-prune
run: yarn test-visual:loki:ci || yarn test-visual:loki-approve-diff && yarn test-visual:loki-prune
- name: Check for changes
id: check-changes

View File

@@ -49,14 +49,14 @@ jobs:
uses: ./.github/actions/prepare-backend
- name: Compile CLJS
run: NODE_ENV=development bun run build-pure:cljs
run: NODE_ENV=development yarn build-pure:cljs
- name: Run Loki Visual Tests
run: bun run test-visual:loki:ci
run: yarn test-visual:loki:ci
- name: Generate Visual Report on Failure
if: failure()
run: bun run test-visual:loki-report
run: yarn test-visual:loki-report
- name: Upload Artifact
id: artifact-upload-step

View File

@@ -30,9 +30,9 @@ jobs:
needs: check-label
if: needs.check-label.outputs.should-run == 'true'
runs-on: ubuntu-22.04
timeout-minutes: 40
timeout-minutes: 60
env:
CI: "true"
CI: 'true'
DRIVERS: postgres
MB_DB_TYPE: postgres
MB_DB_PORT: 5432
@@ -53,11 +53,10 @@ jobs:
steps:
- uses: actions/checkout@v4
- name: Test Postgres App DB
id: test-driver
uses: ./.github/actions/test-driver
with:
build-static-viz: true
junit-name: "minimal-be-tests-postgres-ee"
junit-name: 'minimal-be-tests-postgres-ee'
test-args: >-
:only '["test" "enterprise/backend/test"]'
:exclude-tags '[:mb/driver-tests :mb/transforms-python-test]'
@@ -74,16 +73,15 @@ jobs:
aws-secret-access-key: ${{ secrets.AWS_TEST_RESULTS_SECRET_ACCESS_KEY }}
aws-region: ${{ vars.AWS_REGION }}
trunk-api-token: ${{ secrets.TRUNK_API_TOKEN }}
previous-step-outcome: ${{ steps.test-driver.outputs.test-outcome }}
# Postgres driver tests
postgres-driver-test:
needs: check-label
if: needs.check-label.outputs.should-run == 'true'
runs-on: ubuntu-latest
timeout-minutes: 40
timeout-minutes: 60
env:
CI: "true"
CI: 'true'
DRIVERS: postgres
MB_DB_TYPE: postgres
MB_DB_PORT: 5432
@@ -104,10 +102,9 @@ jobs:
steps:
- uses: actions/checkout@v4
- name: Test Postgres Driver
id: test-driver
uses: ./.github/actions/test-driver
with:
junit-name: "minimal-be-tests-postgres-driver-ee"
junit-name: 'minimal-be-tests-postgres-driver-ee'
test-args: >-
:only-tags [:mb/driver-tests]
:exclude-tags [:mb/transforms-python-test]
@@ -122,7 +119,6 @@ jobs:
aws-secret-access-key: ${{ secrets.AWS_TEST_RESULTS_SECRET_ACCESS_KEY }}
aws-region: ${{ vars.AWS_REGION }}
trunk-api-token: ${{ secrets.TRUNK_API_TOKEN }}
previous-step-outcome: ${{ steps.test-driver.outputs.test-outcome }}
# Frontend unit tests
frontend-unit-tests:
@@ -141,8 +137,7 @@ jobs:
- name: Prepare back-end environment
uses: ./.github/actions/prepare-backend
- name: Run frontend unit tests
id: run-tests
run: bun run test-unit --silent
run: yarn run test-unit --silent
continue-on-error: true
- name: Upload Test Results
uses: ./.github/actions/upload-test-results
@@ -155,7 +150,6 @@ jobs:
aws-secret-access-key: ${{ secrets.AWS_TEST_RESULTS_SECRET_ACCESS_KEY }}
aws-region: ${{ vars.AWS_REGION }}
trunk-api-token: ${{ secrets.TRUNK_API_TOKEN }}
previous-step-outcome: ${{ steps.run-tests.outcome }}
# Summary job
minimal-tests-result:

View File

@@ -1,46 +0,0 @@
name: Validate OpenAPI
on:
push:
branches:
- "master"
- "release-**"
pull_request:
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }}
cancel-in-progress: true
jobs:
files-changed:
name: Check which files changed
runs-on: ubuntu-22.04
timeout-minutes: 3
outputs:
openapi: ${{ steps.changes.outputs.openapi }}
steps:
- uses: actions/checkout@v4
- name: Test which files changed
uses: dorny/paths-filter@v3.0.0
id: changes
with:
token: ${{ github.token }}
filters: .github/file-paths.yaml
validate-openapi:
needs: files-changed
if: needs.files-changed.outputs.openapi == 'true'
runs-on: ubuntu-22.04
timeout-minutes: 10
steps:
- uses: actions/checkout@v4
- name: Prepare front-end environment
uses: ./.github/actions/prepare-frontend
- name: Prepare back-end environment
uses: ./.github/actions/prepare-backend
- name: Generate OpenAPI spec
run: bun run openapi:generate
- name: Validate OpenAPI spec
run: bun run openapi:lint
- name: Split OpenAPI spec
run: bun run openapi:split

View File

@@ -29,8 +29,7 @@ jobs:
- name: Retrieve uberjar artifact for ee
uses: ./.github/actions/fetch-artifact
with:
commit: ${{ github.event.pull_request.head.sha || github.sha }}
edition: ee
name: metabase-ee-${{ github.event.pull_request.head.sha || github.sha }}-uberjar
extract-path: "target/uberjar"
output-name: "metabase.jar"
- name: Move uberjar to bin/docker

View File

@@ -5,8 +5,8 @@ on:
types: [ closed, unlabeled ]
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number }}
cancel-in-progress: false
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
pr-env:

View File

@@ -1,29 +0,0 @@
name: PR Env Destroy (Manual)
on:
workflow_dispatch:
inputs:
pr_number:
description: 'PR number to destroy environment for'
required: true
type: string
self_hosting:
description: 'Self-hosting stack'
type: boolean
default: false
jobs:
approve:
runs-on: ubuntu-latest
environment: pr-env-admin
steps:
- name: Approved
run: echo "Destroying PR environment for PR #${{ inputs.pr_number }}"
destroy:
needs: approve
uses: ./.github/workflows/pr-env-destroy.yml
secrets: inherit
with:
self_hosting: ${{ inputs.self_hosting }}
pr_number: ${{ inputs.pr_number }}

View File

@@ -9,13 +9,6 @@ on:
required: false
type: boolean
default: false
pr_number:
description: 'PR number (required for manual dispatch)'
required: false
type: string
env:
PR_NUMBER: ${{ inputs.pr_number || github.event.number }}
jobs:
destroy_pr:
@@ -69,8 +62,8 @@ jobs:
- name: Destroy PR Review ENV
if: ${{ !inputs.self_hosting }}
run: |
kubectl delete metabase -n hosting-pr${{ env.PR_NUMBER }} hosting-pr${{ env.PR_NUMBER }}
kubectl delete ns hosting-pr${{ env.PR_NUMBER }}
kubectl delete metabase -n hosting-pr${{ github.event.number }} hosting-pr${{ github.event.number }}
kubectl delete ns hosting-pr${{ github.event.number }}
- name: Setup Helm
if: ${{ inputs.self_hosting }}
uses: azure/setup-helm@v4
@@ -79,8 +72,8 @@ jobs:
- name: Destroy PR Review ENV (self-hosting)
if: ${{ inputs.self_hosting }}
run: |
helm uninstall hosting-pr${{ env.PR_NUMBER }} --namespace hosting-pr${{ env.PR_NUMBER }} || true
kubectl delete ns hosting-pr${{ env.PR_NUMBER }} || true
helm uninstall hosting-pr${{ github.event.number }} --namespace hosting-pr${{ github.event.number }} || true
kubectl delete ns hosting-pr${{ github.event.number }} || true
- name: Setup psql client
run: |
sudo apt-get update
@@ -92,4 +85,4 @@ jobs:
-h ${{ secrets.PR_ENV_DB_HOST }} \
-U ${{ secrets.PR_ENV_DB_USER }} \
-d ${{ secrets.PR_ENV_DB_NAME }} \
-c "DROP DATABASE IF EXISTS hosting_pr${{ env.PR_NUMBER }};"
-c "DROP DATABASE IF EXISTS hosting_pr${{ github.event.number }};"

View File

@@ -40,8 +40,7 @@ jobs:
- name: Retrieve uberjar artifact for ee
uses: ./.github/actions/fetch-artifact
with:
commit: ${{ github.event.pull_request.head.sha || github.sha }}
edition: ee
name: metabase-ee-${{ github.event.pull_request.head.sha || github.sha }}-uberjar
extract-path: "target/uberjar"
output-name: "metabase.jar"
- name: Move uberjar to bin/docker

View File

@@ -30,15 +30,12 @@ jobs:
outputs:
run_mode: ${{ fromJson(steps.config.outputs.result).runMode }}
dev_server: ${{ fromJson(steps.config.outputs.result).metastoreDevServer }}
package_manager: ${{ fromJson(steps.config.outputs.result).packageManager }}
steps:
- uses: actions/checkout@v4
with:
sparse-checkout: |
release
.github
- name: Prepare release scripts
uses: ./.github/actions/prepare-release-scripts
sparse-checkout: release
- name: Prepare build scripts
run: cd ${{ github.workspace }}/release && yarn && yarn build
# note: this block can be removed after we stop needing to be able to test versions before v55
- name: Get test mode config
id: config
@@ -51,17 +48,13 @@ jobs:
// with v55 and later we use enterprise tokens that mirror actual Metabase plans
const MULTIPLE_TEST_TOKEN_THRESHOLD = 55;
// Versions before v59 used yarn, v59 and later use bun
const BUN_PACKAGE_MANAGER_THRESHOLD = 59;
const version = '${{ inputs.version }}';
const majorVersion = Number(getMajorVersion(version));
const oldTestToken = majorVersion < MULTIPLE_TEST_TOKEN_THRESHOLD;
return ({
runMode: oldTestToken ? 'prod' : 'e2e',
metastoreDevServer: oldTestToken ? '' : '${{ vars.METASTORE_DEV_SERVER_URL }}',
packageManager: majorVersion < BUN_PACKAGE_MANAGER_THRESHOLD ? 'yarn' : 'bun',
metastoreDevServer: oldTestToken ? '' : '${{ vars.METASTORE_DEV_SERVER_URL }}'
});
release-artifact:
@@ -78,12 +71,10 @@ jobs:
- uses: actions/checkout@v4
with:
sparse-checkout: .github
- name: Retrieve release uberjar artifact for ${{ matrix.edition }}
- name: Retrieve test build uberjar artifact for ${{ matrix.edition }}
uses: ./.github/actions/fetch-artifact
with:
commit: ${{ inputs.commit }}
edition: ${{ matrix.edition }}
release: "true"
name: metabase-release-${{ matrix.edition }}-${{ inputs.commit }}-uberjar
- name: Extract version number from version.properties
id: version-properties
@@ -177,8 +168,6 @@ jobs:
uses: ./.github/actions/prepare-frontend
- name: Prepare back-end environment
uses: ./.github/actions/prepare-backend
- name: Compile CLJS
run: ${{ needs.test-mode-config.outputs.package_manager }} run build-pure:cljs
- name: Prepare Cypress environment
id: cypress-prep
uses: ./.github/actions/prepare-cypress
@@ -239,42 +228,32 @@ jobs:
uses: ./.github/workflows/containerize-jar.yml
secrets: inherit
with:
edition: oss
artifact-name: metabase-release-oss-${{ inputs.commit }}-uberjar
commit: ${{ inputs.commit }}
repo: ${{ vars.DOCKERHUB_OWNER }}/${{ vars.DOCKERHUB_STAGING_REPO }}
tag: ${{ needs.release-artifact.outputs.oss_version }}-${{ inputs.commit }}
release: "true"
containerize-ee:
needs: release-artifact
uses: ./.github/workflows/containerize-jar.yml
secrets: inherit
with:
edition: ee
artifact-name: metabase-release-ee-${{ inputs.commit }}-uberjar
commit: ${{ inputs.commit }}
repo: ${{ vars.DOCKERHUB_OWNER }}/${{ vars.DOCKERHUB_STAGING_REPO }}
tag: ${{ needs.release-artifact.outputs.ee_version }}-${{ inputs.commit }}
release: "true"
tests-complete-message:
if: always()
runs-on: ubuntu-22.04
needs:
[
run-sanity-check,
check-uberjar-health,
containerize-ee,
containerize-oss,
]
needs: [run-sanity-check, check-uberjar-health, containerize-ee, containerize-oss]
timeout-minutes: 5
steps:
- uses: actions/checkout@v4
with:
sparse-checkout: |
release
.github
- name: Prepare release scripts
uses: ./.github/actions/prepare-release-scripts
sparse-checkout: release
- name: Prepare build scripts
run: cd ${{ github.workspace }}/release && yarn && yarn build
- name: Send tests complete message
uses: actions/github-script@v7
env:

View File

@@ -44,12 +44,10 @@ jobs:
steps:
- uses: actions/checkout@v4
with:
sparse-checkout: |
release
.github
sparse-checkout: release
- name: Prepare release scripts
uses: ./.github/actions/prepare-release-scripts
- name: Prepare build scripts
run: cd ${{ github.workspace }}/release && yarn && yarn build
- name: Cache build scripts
uses: actions/upload-artifact@v4
@@ -180,8 +178,8 @@ jobs:
with:
ref: ${{ inputs.branch }}
- name: Prepare release scripts
uses: ./.github/actions/prepare-release-scripts
- name: Prepare build scripts
run: yarn --cwd release --frozen-lockfile && yarn --cwd release build
- name: Get current SDK version
id: current-sdk-version
@@ -258,7 +256,7 @@ jobs:
uses: ./.github/actions/prepare-backend
- name: Run unit tests
run: bun run embedding-sdk:test-unit
run: yarn embedding-sdk:test-unit
build-sdk:
needs: [test, determine-sdk-version]
@@ -284,10 +282,10 @@ jobs:
./bin/embedding-sdk/release_utils.bash update_package_json_template ${{ needs.determine-sdk-version.outputs.sdk_version }}
- name: Build SDK package
run: bun run build-embedding-sdk-package
run: yarn run build-embedding-sdk-package
- name: Generate SDK package.json in the build directory
run: bun run embedding-sdk:generate-package
run: yarn run embedding-sdk:generate-package
- name: Upload built SDK package as artifact
uses: actions/upload-artifact@v4

View File

@@ -38,10 +38,10 @@ jobs:
with:
ref: master # this only works on master
fetch-depth: 0 # we want all branches and tags
- name: Prepare release scripts
uses: ./.github/actions/prepare-release-scripts
- name: Install tsx
run: npm i -g tsx
- name: Install Dependencies
run: yarn --cwd release --frozen-lockfile && npm i -g tsx
- name: Build release scripts
run: yarn --cwd release build
- name: Get version number
uses: actions/github-script@v7
with:

View File

@@ -21,13 +21,8 @@ jobs:
- uses: actions/checkout@v4
with:
sparse-checkout: release
- name: Install bun
uses: oven-sh/setup-bun@v2
with:
bun-version: "1.3.7"
token: ${{ secrets.GITHUB_TOKEN }}
- name: Prepare build scripts
run: bun install --cwd release --frozen-lockfile && bun run --cwd release build
run: yarn --cwd release --frozen-lockfile && yarn --cwd release build
- uses: actions/github-script@v7
env:
GITHUB_TOKEN: ${{ secrets.METABASE_AUTOMATION_USER_TOKEN }}

View File

@@ -23,11 +23,9 @@ jobs:
exit 1
- uses: actions/checkout@v4
with:
sparse-checkout: |
release
.github
- name: Prepare release scripts
uses: ./.github/actions/prepare-release-scripts
sparse-checkout: release
- name: Prepare build scripts
run: cd ${{ github.workspace }}/release && yarn && yarn build
- name: Get Release Version
uses: actions/github-script@v7
env:

View File

@@ -26,13 +26,8 @@ jobs:
- uses: actions/checkout@v4
with:
sparse-checkout: release
- name: Install bun
uses: oven-sh/setup-bun@v2
with:
bun-version: "1.3.7"
token: ${{ secrets.GITHUB_TOKEN }}
- name: Prepare build scripts
run: bun install --cwd release --frozen-lockfile && bun run --cwd release build
run: yarn --cwd release --frozen-lockfile && yarn --cwd release build
- name: Trigger auto-patch
uses: actions/github-script@v7
with:

View File

@@ -14,11 +14,9 @@ jobs:
steps:
- uses: actions/checkout@v4
with:
sparse-checkout: |
release
.github
- name: Prepare release scripts
uses: ./.github/actions/prepare-release-scripts
sparse-checkout: release
- name: Prepare build scripts
run: cd ${{ github.workspace }}/release && yarn --frozen-lockfile && yarn build
- name: Check release status and post to Slack
uses: actions/github-script@v7
env:
@@ -49,11 +47,9 @@ jobs:
steps:
- uses: actions/checkout@v4
with:
sparse-checkout: |
release
.github
- name: Prepare release scripts
uses: ./.github/actions/prepare-release-scripts
sparse-checkout: release
- name: Prepare build scripts
run: cd ${{ github.workspace }}/release && yarn --frozen-lockfile && yarn build
- name: Check backport status and post to Slack
uses: actions/github-script@v7
env:

View File

@@ -79,11 +79,9 @@ jobs:
exit 1
- uses: actions/checkout@v4
with:
sparse-checkout: |
release
.github
- name: Prepare release scripts
uses: ./.github/actions/prepare-release-scripts
sparse-checkout: release
- name: Prepare build scripts
run: cd ${{ github.workspace }}/release && yarn && yarn build
- name: Get Release Version
uses: actions/github-script@v7
id: canonical_version
@@ -306,11 +304,9 @@ jobs:
aws-region: ${{ vars.AWS_REGION }}
- uses: actions/checkout@v4
with:
sparse-checkout: |
release
.github
- name: Prepare release scripts
uses: ./.github/actions/prepare-release-scripts
sparse-checkout: release
- name: Prepare build scripts
run: cd ${{ github.workspace }}/release && yarn && yarn build
- name: Generate new version info
uses: actions/github-script@v7
id: new_version_info

View File

@@ -36,15 +36,10 @@ jobs:
- uses: actions/checkout@v4
with:
sparse-checkout: "release"
- name: Install bun
uses: oven-sh/setup-bun@v2
with:
bun-version: "1.3.7"
token: ${{ secrets.GITHUB_TOKEN }}
- name: prepare dependencies in release directory
run: bun install --cwd release --frozen-lockfile
run: yarn --cwd release --frozen-lockfile
- name: Run frontend type-check
run: bun run --cwd release type-check
run: yarn --cwd release type-check
release-tests-unit:
needs: files-changed
@@ -55,15 +50,10 @@ jobs:
- uses: actions/checkout@v4
with:
sparse-checkout: "release"
- name: Install bun
uses: oven-sh/setup-bun@v2
with:
bun-version: "1.3.7"
token: ${{ secrets.GITHUB_TOKEN }}
- name: prepare dependencies in release directory
run: bun install --cwd release --frozen-lockfile
run: yarn --cwd release --frozen-lockfile
- name: Run release unit tests
run: bun run --cwd release test:unit
run: yarn --cwd release test:unit
release-test-build:
needs: files-changed
@@ -74,14 +64,9 @@ jobs:
- uses: actions/checkout@v4
with:
sparse-checkout: "release"
- name: Install bun
uses: oven-sh/setup-bun@v2
with:
bun-version: "1.3.7"
token: ${{ secrets.GITHUB_TOKEN }}
- name: prepare dependencies in release directory
run: bun install --cwd release --frozen-lockfile
run: yarn --cwd release --frozen-lockfile
- name: Build release bundle
run: bun run --cwd release build
run: yarn --cwd release build
- name: Smoke test release bundle
run: node release/dist/index.cjs

View File

@@ -63,11 +63,9 @@ jobs:
exit 1
- uses: actions/checkout@v4
with:
sparse-checkout: |
release
.github
- name: Prepare release scripts
uses: ./.github/actions/prepare-release-scripts
sparse-checkout: release
- name: Prepare build scripts
run: cd ${{ github.workspace }}/release && yarn && yarn build
- name: Get Release Version
uses: actions/github-script@v7
id: canonical_version
@@ -110,11 +108,9 @@ jobs:
steps:
- uses: actions/checkout@v4
with:
sparse-checkout: |
release
.github
- name: Prepare release scripts
uses: ./.github/actions/prepare-release-scripts
sparse-checkout: release
- name: Prepare build scripts
run: cd ${{ github.workspace }}/release && yarn && yarn build
- name: Send publish start message
uses: actions/github-script@v7
env:
@@ -144,12 +140,10 @@ jobs:
- uses: actions/checkout@v4
with:
sparse-checkout: .github
- name: Retrieve release uberjar artifact for ${{ matrix.edition }}
- name: Retrieve test build uberjar artifact for ${{ matrix.edition }}
uses: ./.github/actions/fetch-artifact
with:
commit: ${{ inputs.commit }}
edition: ${{ matrix.edition }}
release: "true"
name: metabase-release-${{ matrix.edition }}-${{ inputs.commit }}-uberjar
- name: Check JAR version properties
run: |
# ensure actual jar checksum matches checksum file
@@ -193,11 +187,9 @@ jobs:
aws-region: ${{ vars.AWS_REGION }}
- uses: actions/checkout@v4
with:
sparse-checkout: |
release
.github
- name: Prepare release scripts
uses: ./.github/actions/prepare-release-scripts
sparse-checkout: release
- name: prepare release scripts
run: cd release && yarn && yarn build
- uses: actions/download-artifact@v4
name: Retrieve uberjar artifact
with:
@@ -267,22 +259,20 @@ jobs:
uses: ./.github/workflows/containerize-jar.yml
secrets: inherit
with:
edition: oss
artifact-name: metabase-release-oss-${{ inputs.commit }}-uberjar
commit: ${{ inputs.commit }}
repo: ${{ vars.DOCKERHUB_OWNER }}/${{ vars.DOCKERHUB_REPO }}
tag: ${{ needs.check-version.outputs.oss }}
release: "true"
containerize-ee:
needs: check-version
uses: ./.github/workflows/containerize-jar.yml
secrets: inherit
with:
edition: ee
artifact-name: metabase-release-ee-${{ inputs.commit }}-uberjar
commit: ${{ inputs.commit }}
repo: ${{ vars.DOCKERHUB_OWNER }}/${{ vars.DOCKERHUB_REPO }}-enterprise
tag: ${{ needs.check-version.outputs.ee }}
release: "true"
push-tags:
permissions: write-all
@@ -292,11 +282,9 @@ jobs:
steps:
- uses: actions/checkout@v4
with:
sparse-checkout: |
release
.github
- name: Prepare release scripts
uses: ./.github/actions/prepare-release-scripts
sparse-checkout: release
- name: prepare release scripts
run: cd release && yarn && yarn build
- name: Tag Release
uses: actions/github-script@v7
with:
@@ -326,11 +314,9 @@ jobs:
steps:
- uses: actions/checkout@v4
with:
sparse-checkout: |
release
.github
- name: Prepare release scripts
uses: ./.github/actions/prepare-release-scripts
sparse-checkout: release
- name: prepare release scripts
run: cd release && yarn && yarn build
- name: Get extra tags
id: taglist
uses: actions/github-script@v7
@@ -340,11 +326,10 @@ jobs:
const { getExtraTagsForVersion } = require('${{ github.workspace }}/release/dist/index.cjs');
const version = '${{ inputs.version }}';
const latestMajorVersion = '${{ vars.LATEST_MAJOR_VERSION }}';
const extra_tags = getExtraTagsForVersion({ version, latestMajorVersion });
const extra_tags = getExtraTagsForVersion({ version });
console.log({ extra_tags, latestMajorVersion });
console.log({ extra_tags });
core.setOutput('extra_tags', JSON.stringify(extra_tags));
add-extra-tags:
@@ -362,11 +347,9 @@ jobs:
with:
version: ${{ inputs.version }}
tag_name: ${{ matrix.tag }}
# For version tags (v0.x, v1.x), apply to matching edition only
# For non-version tags like "latest", apply to both editions
tag_ee: ${{ contains(matrix.tag, 'v1') || !startsWith(matrix.tag, 'v') }}
tag_oss: ${{ contains(matrix.tag, 'v0') || !startsWith(matrix.tag, 'v') }}
dot-x-tag: ${{ startsWith(matrix.tag, 'v') }}
tag_ee: ${{ contains(matrix.tag, 'v1') }}
tag_oss: ${{ contains(matrix.tag, 'v0') }}
dot-x-tag: true
trigger-cloud-issues:
if: ${{ !inputs.auto }}
@@ -376,11 +359,9 @@ jobs:
steps:
- uses: actions/checkout@v4
with:
sparse-checkout: |
release
.github
- name: Prepare release scripts
uses: ./.github/actions/prepare-release-scripts
sparse-checkout: release
- name: Prepare build scripts
run: cd ${{ github.workspace }}/release && yarn && yarn build
- name: Find milestone
id: find-milestone
uses: actions/github-script@v7
@@ -443,6 +424,19 @@ jobs:
}
});
# updates the llms.txt asynchronously.
# release does not need to wait for this PR to be merged,
# because the llms.txt is in the main git repo.
trigger-update-llms-txt:
# do not trigger llms.txt update on automatic patch releases
if: ${{ !inputs.auto }}
needs: add-extra-tags
uses: ./.github/workflows/llms-txt-generate.yml
secrets: inherit
with:
version: ${{ inputs.version }}
branch: ${{ github.ref_name }}
draft-release-notes:
if: ${{ !inputs.auto }}
needs: add-extra-tags
@@ -456,11 +450,9 @@ jobs:
steps:
- uses: actions/checkout@v4
with:
sparse-checkout: |
release
.github
- name: Prepare release scripts
uses: ./.github/actions/prepare-release-scripts
sparse-checkout: release
- name: prepare release scripts
run: cd release && yarn && yarn build
- name: Publish draft release notes to github
uses: actions/github-script@v7
with:
@@ -537,11 +529,9 @@ jobs:
aws-region: ${{ vars.AWS_REGION }}
- uses: actions/checkout@v4
with:
sparse-checkout: |
release
.github
- name: Prepare release scripts
uses: ./.github/actions/prepare-release-scripts
sparse-checkout: release
- name: Prepare build scripts
run: cd ${{ github.workspace }}/release && yarn && yarn build
- name: Publish version info
uses: actions/github-script@v7
id: new_version_info
@@ -596,11 +586,9 @@ jobs:
steps:
- uses: actions/checkout@v4
with:
sparse-checkout: |
release
.github
- name: Prepare release scripts
uses: ./.github/actions/prepare-release-scripts
sparse-checkout: release
- name: Prepare build scripts
run: cd ${{ github.workspace }}/release && yarn && yarn build
- name: Send publish complete message
uses: actions/github-script@v7
env:
@@ -629,11 +617,9 @@ jobs:
steps:
- uses: actions/checkout@v4
with:
sparse-checkout: |
release
.github
- name: Prepare release scripts
uses: ./.github/actions/prepare-release-scripts
sparse-checkout: release
- name: Prepare build scripts
run: cd ${{ github.workspace }}/release && yarn && yarn build
- name: Update milestones
uses: actions/github-script@v7
with:

View File

@@ -103,8 +103,7 @@ jobs:
(needs.files-changed.outputs.embedding_sdk_components == 'true' ||
needs.files-changed.outputs.frontend_sources == 'true' ||
needs.files-changed.outputs.embedding_documentation == 'true' ||
needs.files-changed.outputs.embedding_sdk_ci == 'true' ||
needs.files-changed.outputs.embedding_sdk_host_sample_apps == 'true')
needs.files-changed.outputs.embedding_sdk_ci == 'true')
uses: ./.github/workflows/embedding-sdk-package-build.yml
secrets: inherit
@@ -114,7 +113,6 @@ jobs:
uses: ./.github/workflows/embedding-sdk.yml
secrets: inherit
with:
bundle-size-check: ${{ github.event_name == 'pull_request' && needs.uberjar.result == 'success' && needs.files-changed.outputs.frontend_sources == 'true' }}
cli-snippets-type-check: ${{ needs.files-changed.outputs.frontend_sources == 'true' }}
docs-snippets-type-check: ${{ needs.files-changed.outputs.embedding_documentation == 'true' }}
documentation-validation: ${{ needs.files-changed.outputs.frontend_sources == 'true' }}

View File

@@ -15,7 +15,7 @@ jobs:
semantic-search-tests-h2:
if: ${{ !inputs.skip }}
runs-on: ubuntu-22.04
timeout-minutes: 40
timeout-minutes: 60
strategy:
fail-fast: false
matrix:
@@ -46,7 +46,6 @@ jobs:
steps:
- uses: actions/checkout@v4
- name: Test ${{ matrix.version.name }}
id: test-driver
uses: ./.github/actions/test-driver
with:
build-static-viz: ${{ matrix.job.build-static-viz }}
@@ -65,12 +64,11 @@ jobs:
aws-secret-access-key: ${{ secrets.AWS_TEST_RESULTS_SECRET_ACCESS_KEY }}
aws-region: ${{ vars.AWS_REGION }}
trunk-api-token: ${{ secrets.TRUNK_API_TOKEN }}
previous-step-outcome: ${{ steps.test-driver.outputs.test-outcome }}
semantic-search-tests-mariadb:
if: ${{ !inputs.skip }}
runs-on: ubuntu-22.04
timeout-minutes: 40
timeout-minutes: 60
strategy:
fail-fast: false
matrix:
@@ -120,7 +118,6 @@ jobs:
steps:
- uses: actions/checkout@v4
- name: Test ${{ matrix.version.name }}
id: test-driver
uses: ./.github/actions/test-driver
with:
build-static-viz: ${{ matrix.job.build-static-viz }}
@@ -139,12 +136,11 @@ jobs:
aws-secret-access-key: ${{ secrets.AWS_TEST_RESULTS_SECRET_ACCESS_KEY }}
aws-region: ${{ vars.AWS_REGION }}
trunk-api-token: ${{ secrets.TRUNK_API_TOKEN }}
previous-step-outcome: ${{ steps.test-driver.outputs.test-outcome }}
semantic-search-tests-mysql:
if: ${{ !inputs.skip }}
runs-on: ubuntu-22.04
timeout-minutes: 40
timeout-minutes: 60
strategy:
fail-fast: false
matrix:
@@ -194,7 +190,6 @@ jobs:
steps:
- uses: actions/checkout@v4
- name: Test ${{ matrix.version.name }}
id: test-driver
uses: ./.github/actions/test-driver
with:
build-static-viz: ${{ matrix.job.build-static-viz }}
@@ -213,12 +208,11 @@ jobs:
aws-secret-access-key: ${{ secrets.AWS_TEST_RESULTS_SECRET_ACCESS_KEY }}
aws-region: ${{ vars.AWS_REGION }}
trunk-api-token: ${{ secrets.TRUNK_API_TOKEN }}
previous-step-outcome: ${{ steps.test-driver.outputs.test-outcome }}
semantic-search-tests-postgres:
if: ${{ !inputs.skip }}
runs-on: ubuntu-22.04
timeout-minutes: 40
timeout-minutes: 60
strategy:
fail-fast: false
matrix:
@@ -269,7 +263,6 @@ jobs:
steps:
- uses: actions/checkout@v4
- name: Test ${{ matrix.version.name }}
id: test-driver
uses: ./.github/actions/test-driver
with:
build-static-viz: ${{ matrix.job.build-static-viz }}
@@ -288,7 +281,6 @@ jobs:
aws-secret-access-key: ${{ secrets.AWS_TEST_RESULTS_SECRET_ACCESS_KEY }}
aws-region: ${{ vars.AWS_REGION }}
trunk-api-token: ${{ secrets.TRUNK_API_TOKEN }}
previous-step-outcome: ${{ steps.test-driver.outputs.test-outcome }}
semantic-search-tests-result:
needs:

View File

@@ -41,36 +41,16 @@ jobs:
strategy:
matrix:
project:
- name: Main Frontend (bun w/ yarn.lock)
- name: Main Frontend (yarn)
file: yarn.lock
package-manager: yarn
category: main-frontend-bun
category: main-frontend-yarn
- name: Release Helper
file: release/yarn.lock
package-manager: yarn
category: release-helper
steps:
- uses: actions/checkout@v4
- uses: oven-sh/setup-bun@v2
with:
bun-version: "1.3.7"
token: ${{ secrets.GITHUB_TOKEN }}
# Snyk doesn't support bun.lock yet, so we generate yarn.lock and scan that.
# Bun has a bug where it normalizes "< 3.0.0" to "< 3" in alias entries
# but not in dependency specs, causing Snyk to fail matching dependencies.
# The sed commands fix this inconsistency. They continue on error to let
# Snyk report parsing issues directly rather than failing on the workaround.
- name: Generate yarn.lock
if: matrix.project.file == 'yarn.lock'
run: |
bun install --yarn
sed -i -E 's/< ([0-9]+)\.0\.0"/< \1"/g; s/< ([0-9]+)\.0"/< \1"/g' yarn.lock || true
- name: Generate release/yarn.lock
if: matrix.project.file == 'release/yarn.lock'
run: |
bun install --yarn
sed -i -E 's/< ([0-9]+)\.0\.0"/< \1"/g; s/< ([0-9]+)\.0"/< \1"/g' yarn.lock || true
working-directory: release
- uses: snyk/actions/setup@0.4.0
- name: Run snyk test
env:

View File

@@ -23,11 +23,9 @@ jobs:
steps:
- uses: actions/checkout@v4
with:
sparse-checkout: |
release
.github
- name: Prepare release scripts
uses: ./.github/actions/prepare-release-scripts
sparse-checkout: release
- name: Prepare build scripts
run: cd ${{ github.workspace }}/release && yarn && yarn build
- name: Send build start message
uses: actions/github-script@v7
env:
@@ -64,11 +62,9 @@ jobs:
exit 1
- uses: actions/checkout@v4
with:
sparse-checkout: |
release
.github
- name: Prepare release scripts
uses: ./.github/actions/prepare-release-scripts
sparse-checkout: release
- name: Prepare build scripts
run: cd ${{ github.workspace }}/release && yarn && yarn build
- name: Get Release Version
uses: actions/github-script@v7
id: canonical_version
@@ -108,11 +104,9 @@ jobs:
uses: actions/checkout@v4
with:
fetch-depth: 0 # IMPORTANT! to get all the branches
sparse-checkout: |
release
.github
- name: Prepare release scripts
uses: ./.github/actions/prepare-release-scripts
sparse-checkout: release
- name: Prepare build scripts
run: cd ${{ github.workspace }}/release && yarn && yarn build
- name: Get Release Branch
uses: actions/github-script@v7
id: release_branch
@@ -130,7 +124,6 @@ jobs:
git branch --contains ${{ inputs.commit }} | grep -q $RELEASE_BRANCH \
&& echo "Commit found in correct release branch" \
|| (echo "Commit not found in correct release branch" && exit 1)
git checkout -
tag-uberjar-for-release:
needs: [check-version, check-commit]
@@ -163,8 +156,7 @@ jobs:
- name: Retrieve test build uberjar artifact for ${{ matrix.edition }}
uses: ./.github/actions/fetch-artifact
with:
commit: ${{ inputs.commit }}
edition: ${{ matrix.edition }}
name: metabase-${{ matrix.edition }}-${{ inputs.commit }}-uberjar
- name: Update version.properties in jar
run: | # sh

View File

@@ -1,495 +0,0 @@
name: Transforms Python Stress Test
on:
workflow_dispatch:
inputs:
ref:
description: 'Branch name or commit SHA to checkout'
type: string
required: false
default: ''
driver:
description: 'Which driver(s) to test'
required: true
default: 'all'
type: choice
options:
- all
- bigquery
- clickhouse
- mongo
- mysql
- postgres
- redshift
- snowflake
- sqlserver
jobs:
be-tests-bigquery-transforms:
# inputs.driver is empty on push triggers, so treat '' as 'all'
if: ${{ inputs.driver == 'all' || inputs.driver == '' || inputs.driver == 'bigquery' }}
runs-on: ubuntu-latest
timeout-minutes: 60
env:
CI: 'true'
DRIVERS: bigquery-cloud-sdk
MB_BIGQUERY_TEST_PROJECT_ID: ${{ secrets.BIGQUERY_TEST_PROJECT_ID }}
MB_BIGQUERY_TEST_CLIENT_ID: ${{ secrets.MB_BIGQUERY_TEST_CLIENT_ID }}
MB_BIGQUERY_TEST_CLIENT_SECRET: ${{ secrets.MB_BIGQUERY_TEST_CLIENT_SECRET }}
MB_BIGQUERY_TEST_ACCESS_TOKEN: ${{ secrets.MB_BIGQUERY_TEST_ACCESS_TOKEN }}
MB_BIGQUERY_TEST_REFRESH_TOKEN: ${{ secrets.MB_BIGQUERY_TEST_REFRESH_TOKEN }}
MB_BIGQUERY_CLOUD_SDK_TEST_SERVICE_ACCOUNT_JSON: ${{ secrets.MB_BIGQUERY_CLOUD_SDK_TEST_SERVICE_ACCOUNT_JSON }}
MB_BIGQUERY_CLOUD_SDK_TEST_SERVICE_ACCOUNT_JSON_ROUTING: ${{ secrets.MB_BIGQUERY_CLOUD_SDK_TEST_SERVICE_ACCOUNT_JSON_ROUTING }}
name: BigQuery Transforms
steps:
- uses: actions/checkout@v4
with:
ref: ${{ inputs.ref || github.ref }}
- name: Setup python-runner
uses: ./.github/actions/setup-python-runner
with:
metabase-automation-token: ${{ secrets.METABASE_AUTOMATION_USER_TOKEN }}
- name: Test BigQuery transforms
id: test-driver
uses: ./.github/actions/test-driver
with:
junit-name: 'be-tests-bigquery-transforms'
test-args: >-
:only-tags [:mb/transforms-python-test]
- name: Upload Test Results
uses: ./.github/actions/upload-test-results
if: always()
with:
input-path: ./target/junit/
output-name: ${{ github.job }}
bucket: ${{ vars.AWS_S3_TEST_RESULTS_BUCKET }}
aws-access-key-id: ${{ secrets.AWS_TEST_RESULTS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_TEST_RESULTS_SECRET_ACCESS_KEY }}
aws-region: ${{ vars.AWS_REGION }}
trunk-api-token: ${{ secrets.TRUNK_API_TOKEN }}
- name: Cleanup python-runner
if: always()
uses: ./.github/actions/cleanup-python-runner
with:
upload-logs: ${{ steps.test-driver.outcome == 'failure' }}
logs-artifact-suffix: BigQuery Transforms
be-tests-clickhouse-transforms:
if: ${{ inputs.driver == 'all' || inputs.driver == '' || inputs.driver == 'clickhouse' }}
runs-on: ubuntu-latest
timeout-minutes: 60
env:
CI: 'true'
DRIVERS: clickhouse
MB_CLICKHOUSE_TEST_HOST: localhost
MB_CLICKHOUSE_TEST_PORT: 8123
MB_CLICKHOUSE_TEST_OLD_PORT: 8124
MB_CLICKHOUSE_TEST_NGINX_PORT: 8127
name: ClickHouse Transforms
steps:
- uses: actions/checkout@v4
with:
ref: ${{ inputs.ref || github.ref }}
- name: Setup python-runner
uses: ./.github/actions/setup-python-runner
with:
metabase-automation-token: ${{ secrets.METABASE_AUTOMATION_USER_TOKEN }}
- name: Add ClickHouse TLS instance to /etc/hosts
run: |
sudo echo "127.0.0.1 server.clickhouseconnect.test" | sudo tee -a /etc/hosts
- name: Start ClickHouse in Docker
uses: hoverkraft-tech/compose-action@v2.0.0
with:
compose-file: "modules/drivers/clickhouse/docker-compose.yml"
down-flags: "--volumes"
services: |
clickhouse
clickhouse_tls
clickhouse_older_version
clickhouse_cluster_node1
clickhouse_cluster_node2
nginx
- name: Test ClickHouse transforms
id: test-driver
uses: ./.github/actions/test-driver
with:
junit-name: 'be-tests-clickhouse-transforms'
test-args: >-
:only-tags [:mb/transforms-python-test]
- name: Upload Test Results
uses: ./.github/actions/upload-test-results
if: always()
with:
input-path: ./target/junit/
output-name: ${{ github.job }}
bucket: ${{ vars.AWS_S3_TEST_RESULTS_BUCKET }}
aws-access-key-id: ${{ secrets.AWS_TEST_RESULTS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_TEST_RESULTS_SECRET_ACCESS_KEY }}
aws-region: ${{ vars.AWS_REGION }}
trunk-api-token: ${{ secrets.TRUNK_API_TOKEN }}
- name: Cleanup python-runner
if: always()
uses: ./.github/actions/cleanup-python-runner
with:
upload-logs: ${{ steps.test-driver.outcome == 'failure' }}
logs-artifact-suffix: ClickHouse Transforms
be-tests-mongo-transforms:
if: ${{ inputs.driver == 'all' || inputs.driver == '' || inputs.driver == 'mongo' }}
runs-on: ubuntu-latest
timeout-minutes: 60
env:
CI: 'true'
DRIVERS: mongo
MB_MONGO_TEST_USER: metabase
MB_MONGO_TEST_PASSWORD: metasample123
services:
mongodb:
image: mongo:latest
ports:
- "27017:27017"
env:
MONGO_INITDB_ROOT_USERNAME: metabase
MONGO_INITDB_ROOT_PASSWORD: metasample123
name: MongoDB Transforms
steps:
- uses: actions/checkout@v4
with:
ref: ${{ inputs.ref || github.ref }}
- name: Setup python-runner
uses: ./.github/actions/setup-python-runner
with:
metabase-automation-token: ${{ secrets.METABASE_AUTOMATION_USER_TOKEN }}
- name: Test MongoDB transforms
id: test-driver
uses: ./.github/actions/test-driver
with:
junit-name: 'be-tests-mongo-transforms'
test-args: >-
:only-tags [:mb/transforms-python-test]
- name: Upload Test Results
uses: ./.github/actions/upload-test-results
if: always()
with:
input-path: ./target/junit/
output-name: ${{ github.job }}
bucket: ${{ vars.AWS_S3_TEST_RESULTS_BUCKET }}
aws-access-key-id: ${{ secrets.AWS_TEST_RESULTS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_TEST_RESULTS_SECRET_ACCESS_KEY }}
aws-region: ${{ vars.AWS_REGION }}
trunk-api-token: ${{ secrets.TRUNK_API_TOKEN }}
- name: Cleanup python-runner
if: always()
uses: ./.github/actions/cleanup-python-runner
with:
upload-logs: ${{ steps.test-driver.outcome == 'failure' }}
logs-artifact-suffix: MongoDB Transforms
be-tests-mysql-transforms:
if: ${{ inputs.driver == 'all' || inputs.driver == '' || inputs.driver == 'mysql' }}
runs-on: ubuntu-latest
timeout-minutes: 60
services:
mysql:
image: mysql:latest
ports:
- "3306:3306"
env:
MYSQL_ALLOW_EMPTY_PASSWORD: true
MYSQL_DATABASE: circle_test
env:
CI: 'true'
DRIVERS: mysql
MB_DB_TYPE: mysql
MB_DB_HOST: localhost
MB_DB_PORT: 3306
MB_DB_DBNAME: circle_test
MB_DB_USER: root
MB_MYSQL_TEST_USER: root
name: MySQL Transforms
steps:
- uses: actions/checkout@v4
with:
ref: ${{ inputs.ref || github.ref }}
- name: Setup python-runner
uses: ./.github/actions/setup-python-runner
with:
metabase-automation-token: ${{ secrets.METABASE_AUTOMATION_USER_TOKEN }}
- name: Test MySQL transforms
id: test-driver
uses: ./.github/actions/test-driver
with:
junit-name: 'be-tests-mysql-transforms'
test-args: >-
:only-tags [:mb/transforms-python-test]
- name: Upload Test Results
uses: ./.github/actions/upload-test-results
if: always()
with:
input-path: ./target/junit/
output-name: ${{ github.job }}
bucket: ${{ vars.AWS_S3_TEST_RESULTS_BUCKET }}
aws-access-key-id: ${{ secrets.AWS_TEST_RESULTS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_TEST_RESULTS_SECRET_ACCESS_KEY }}
aws-region: ${{ vars.AWS_REGION }}
trunk-api-token: ${{ secrets.TRUNK_API_TOKEN }}
- name: Cleanup python-runner
if: always()
uses: ./.github/actions/cleanup-python-runner
with:
upload-logs: ${{ steps.test-driver.outcome == 'failure' }}
logs-artifact-suffix: MySQL Transforms
be-tests-postgres-transforms:
if: ${{ inputs.driver == 'all' || inputs.driver == '' || inputs.driver == 'postgres' }}
runs-on: ubuntu-latest
timeout-minutes: 60
env:
CI: 'true'
DRIVERS: postgres
MB_DB_TYPE: postgres
MB_DB_PORT: 5432
MB_DB_HOST: localhost
MB_DB_DBNAME: mb_test
MB_DB_USER: mb_test
MB_POSTGRESQL_TEST_USER: mb_test
services:
postgres:
image: postgres:latest
ports:
- "5432:5432"
env:
POSTGRES_USER: mb_test
POSTGRES_DB: mb_test
POSTGRES_HOST_AUTH_METHOD: trust
name: Postgres Transforms
steps:
- uses: actions/checkout@v4
with:
ref: ${{ inputs.ref || github.ref }}
- name: Setup python-runner
uses: ./.github/actions/setup-python-runner
with:
metabase-automation-token: ${{ secrets.METABASE_AUTOMATION_USER_TOKEN }}
- name: Test Postgres transforms
id: test-driver
uses: ./.github/actions/test-driver
with:
junit-name: 'be-tests-postgres-transforms'
test-args: >-
:only-tags [:mb/transforms-python-test]
- name: Upload Test Results
uses: ./.github/actions/upload-test-results
if: always()
with:
input-path: ./target/junit/
output-name: ${{ github.job }}
bucket: ${{ vars.AWS_S3_TEST_RESULTS_BUCKET }}
aws-access-key-id: ${{ secrets.AWS_TEST_RESULTS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_TEST_RESULTS_SECRET_ACCESS_KEY }}
aws-region: ${{ vars.AWS_REGION }}
trunk-api-token: ${{ secrets.TRUNK_API_TOKEN }}
- name: Cleanup python-runner
if: always()
uses: ./.github/actions/cleanup-python-runner
with:
upload-logs: ${{ steps.test-driver.outcome == 'failure' }}
logs-artifact-suffix: Postgres Transforms
be-tests-redshift-transforms:
if: ${{ inputs.driver == 'all' || inputs.driver == '' || inputs.driver == 'redshift' }}
runs-on: ubuntu-latest
timeout-minutes: 60
env:
CI: 'true'
DRIVERS: redshift
MB_REDSHIFT_TEST_USER: metabase_ci
MB_REDSHIFT_TEST_DB: testdb
MB_REDSHIFT_TEST_DB_ROUTING: dev
MB_REDSHIFT_TEST_HOST: ${{ secrets.MB_REDSHIFT_TEST_HOST }}
MB_REDSHIFT_TEST_PASSWORD: ${{ secrets.MB_REDSHIFT_TEST_PASSWORD }}
name: Redshift Transforms
steps:
- uses: actions/checkout@v4
with:
ref: ${{ inputs.ref || github.ref }}
- name: Setup python-runner
uses: ./.github/actions/setup-python-runner
with:
metabase-automation-token: ${{ secrets.METABASE_AUTOMATION_USER_TOKEN }}
- name: Test Redshift transforms
id: test-driver
uses: ./.github/actions/test-driver
with:
junit-name: 'be-tests-redshift-transforms'
test-args: >-
:only-tags [:mb/transforms-python-test]
- name: Upload Test Results
uses: ./.github/actions/upload-test-results
if: always()
with:
input-path: ./target/junit/
output-name: ${{ github.job }}
bucket: ${{ vars.AWS_S3_TEST_RESULTS_BUCKET }}
aws-access-key-id: ${{ secrets.AWS_TEST_RESULTS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_TEST_RESULTS_SECRET_ACCESS_KEY }}
aws-region: ${{ vars.AWS_REGION }}
trunk-api-token: ${{ secrets.TRUNK_API_TOKEN }}
- name: Cleanup python-runner
if: always()
uses: ./.github/actions/cleanup-python-runner
with:
upload-logs: ${{ steps.test-driver.outcome == 'failure' }}
logs-artifact-suffix: Redshift Transforms
be-tests-snowflake-transforms:
if: ${{ inputs.driver == 'all' || inputs.driver == '' || inputs.driver == 'snowflake' }}
runs-on: ubuntu-latest
timeout-minutes: 60
env:
CI: 'true'
DRIVERS: snowflake
MB_SNOWFLAKE_TEST_USER: METABASE CI
MB_SNOWFLAKE_TEST_ACCOUNT: ${{ secrets.MB_SNOWFLAKE_TEST_ACCOUNT }}
MB_SNOWFLAKE_TEST_PASSWORD: ${{ secrets.MB_SNOWFLAKE_TEST_PASSWORD }}
MB_SNOWFLAKE_TEST_PRIVATE_KEY: ${{ secrets.MB_SNOWFLAKE_TEST_PRIVATE_KEY }}
MB_SNOWFLAKE_TEST_WAREHOUSE: ${{ secrets.MB_SNOWFLAKE_TEST_WAREHOUSE }}
MB_SNOWFLAKE_TEST_PK_USER: METABASE PK
MB_SNOWFLAKE_TEST_PK_PRIVATE_KEY: ${{ secrets.MB_SNOWFLAKE_TEST_PK_PRIVATE_KEY }}
MB_SNOWFLAKE_TEST_PK_PRIVATE_KEY_2: ${{ secrets.MB_SNOWFLAKE_TEST_PK_PRIVATE_KEY_2 }}
MB_SNOWFLAKE_TEST_PK_PUBLIC_KEY: ${{ secrets.MB_SNOWFLAKE_TEST_PK_PUBLIC_KEY }}
MB_SNOWFLAKE_TEST_PK_PUBLIC_KEY_2: ${{ secrets.MB_SNOWFLAKE_TEST_PK_PUBLIC_KEY_2 }}
MB_SNOWFLAKE_TEST_RSA_ROLE_TEST_DEFAULT_USER: RSA_ROLE_TEST_DEFAULT_USER
MB_SNOWFLAKE_TEST_RSA_ROLE_TEST_CUSTOM_USER: RSA_ROLE_TEST_CUSTOM_USER
MB_SNOWFLAKE_TEST_RSA_ROLE_TEST_ROLE: RSA_ROLE_TEST_ROLE
MB_SNOWFLAKE_TEST_RSA_ROLE_TEST_DB: RSA_ROLE_TEST_DB
name: Snowflake Transforms
steps:
- uses: actions/checkout@v4
with:
ref: ${{ inputs.ref || github.ref }}
- name: Setup python-runner
uses: ./.github/actions/setup-python-runner
with:
metabase-automation-token: ${{ secrets.METABASE_AUTOMATION_USER_TOKEN }}
- name: Test Snowflake transforms
id: test-driver
uses: ./.github/actions/test-driver
with:
junit-name: 'be-tests-snowflake-transforms'
test-args: >-
:only-tags [:mb/transforms-python-test]
- name: Upload Test Results
uses: ./.github/actions/upload-test-results
if: always()
with:
input-path: ./target/junit/
output-name: ${{ github.job }}
bucket: ${{ vars.AWS_S3_TEST_RESULTS_BUCKET }}
aws-access-key-id: ${{ secrets.AWS_TEST_RESULTS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_TEST_RESULTS_SECRET_ACCESS_KEY }}
aws-region: ${{ vars.AWS_REGION }}
trunk-api-token: ${{ secrets.TRUNK_API_TOKEN }}
- name: Cleanup python-runner
if: always()
uses: ./.github/actions/cleanup-python-runner
with:
upload-logs: ${{ steps.test-driver.outcome == 'failure' }}
logs-artifact-suffix: Snowflake Transforms
be-tests-sqlserver-transforms:
if: ${{ inputs.driver == 'all' || inputs.driver == '' || inputs.driver == 'sqlserver' }}
runs-on: ubuntu-latest
timeout-minutes: 60
env:
CI: 'true'
DRIVERS: sqlserver
MB_SQLSERVER_TEST_HOST: localhost
MB_SQLSERVER_TEST_PASSWORD: 'P@ssw0rd'
MB_SQLSERVER_TEST_USER: SA
services:
sqlserver:
image: mcr.microsoft.com/mssql/server:2022-latest
ports:
- "1433:1433"
env:
ACCEPT_EULA: Y
SA_PASSWORD: 'P@ssw0rd'
MSSQL_MEMORY_LIMIT_MB: 1024
name: SQL Server Transforms
steps:
- uses: actions/checkout@v4
with:
ref: ${{ inputs.ref || github.ref }}
- name: Setup python-runner
uses: ./.github/actions/setup-python-runner
with:
metabase-automation-token: ${{ secrets.METABASE_AUTOMATION_USER_TOKEN }}
- name: Test SQL Server transforms
id: test-driver
uses: ./.github/actions/test-driver
with:
junit-name: 'be-tests-sqlserver-transforms'
test-args: >-
:only-tags [:mb/transforms-python-test]
- name: Upload Test Results
uses: ./.github/actions/upload-test-results
if: always()
with:
input-path: ./target/junit/
output-name: ${{ github.job }}
bucket: ${{ vars.AWS_S3_TEST_RESULTS_BUCKET }}
aws-access-key-id: ${{ secrets.AWS_TEST_RESULTS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_TEST_RESULTS_SECRET_ACCESS_KEY }}
aws-region: ${{ vars.AWS_REGION }}
trunk-api-token: ${{ secrets.TRUNK_API_TOKEN }}
- name: Cleanup python-runner
if: always()
uses: ./.github/actions/cleanup-python-runner
with:
upload-logs: ${{ steps.test-driver.outcome == 'failure' }}
logs-artifact-suffix: SQL Server Transforms
transforms-stress-test-result:
needs:
- be-tests-bigquery-transforms
- be-tests-clickhouse-transforms
- be-tests-mongo-transforms
- be-tests-mysql-transforms
- be-tests-postgres-transforms
- be-tests-redshift-transforms
- be-tests-snowflake-transforms
- be-tests-sqlserver-transforms
runs-on: ubuntu-latest
timeout-minutes: 5
name: transforms-stress-test-result
if: always() && !cancelled()
steps:
- name: Check job status
uses: actions/github-script@v7
env:
needs: ${{ toJson(needs) }}
with:
script: |
const needs = JSON.parse(process.env.needs);
const jobs = Object.entries(needs).map(
([jobName, jobValues]) => ({
name: jobName,
result: jobValues.result
}));
if (jobs.every(job => (job.result === 'skipped' || job.result === 'success'))) {
console.log("All transforms tests have passed (or been skipped).");
process.exit(0);
}
console.log("Some transforms tests have failed:");
jobs.forEach((job) => {
if (job.result !== 'success' && job.result !== 'skipped') {
console.log(` ${job.name} - ${job.result}`);
}
});
process.exit(1);

View File

@@ -1,50 +0,0 @@
name: Crowdin Context Harvester
on:
workflow_dispatch:
schedule:
- cron: "30 6 * * 2" # Tuesday at 06:30 UTC, one hour after translation-update.yml
jobs:
harvest:
runs-on: ubuntu-latest
permissions:
contents: read
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 22
- name: Harvest and write AI context to Crowdin
timeout-minutes: 60
env:
CROWDIN_PERSONAL_TOKEN: ${{ secrets.CROWDIN_TOKEN }}
CROWDIN_PROJECT_ID: ${{ secrets.CROWDIN_PROJECT_ID }}
OPENAI_KEY: ${{ secrets.CROWDIN_OPENAI_API_KEY }}
run: |
npx --yes crowdin-context-harvester@latest harvest \
--project="$CROWDIN_PROJECT_ID" \
--ai="openai" \
--model="gpt-5.1" \
--croql='not (context contains "✨ AI Context")' \
--output="crowdin" \
--concurrency=10
# We only want to trigger pre-translations, not wait for them to finish running upstream.
# Weirdly, Crowdin keeps this step running as long as it runs pre-translations on their servers.
# Hence, the three minute timeout and continue-on-error.
- name: Trigger pre-translations
timeout-minutes: 3
uses: crowdin/github-action@v2
# https://crowdin.github.io/crowdin-cli/commands/crowdin-pre-translate
with:
command: "pre-translate"
command_args: >-
--method ai
--ai-prompt ${{ secrets.CROWDIN_AI_PROMPT_ID }}
--translate-untranslated-only
--token ${{ secrets.CROWDIN_TOKEN }}
--project-id ${{ secrets.CROWDIN_PROJECT_ID }}
--config=locales/crowdin.yml
--no-progress
continue-on-error: true

70
.github/workflows/tusk-sanity-check.yml vendored Normal file
View File

@@ -0,0 +1,70 @@
# This allows Tusk to manually run workflows to check lint/build/tests before creating a PR.
# If you have any questions, please reach out to support@usetusk.ai.
name: Tusk Sanity Check
on:
workflow_dispatch:
inputs:
taskId:
description: "Tusk Task ID"
required: true
runType:
description: "Tusk Run Type"
required: true
runId:
description: "Tusk Run ID"
required: true
jobs:
sanity_check:
runs-on: ubuntu-latest
steps:
- name: Log inputs
run: |
echo "Tusk Task ID: ${{ github.event.inputs.taskId }}"
echo "Tusk Run Type: ${{ github.event.inputs.runType }}"
echo "Tusk Run ID: ${{ github.event.inputs.runId }}"
echo "Current Branch: ${{ github.ref }}"
echo "Repository default branch: ${{ github.event.repository.default_branch }}"
- uses: actions/checkout@v4
with:
ref: ${{ github.ref }}
fetch-depth: 0
- name: Check which files changed
uses: dorny/paths-filter@v3.0.0
id: changes
with:
token: ${{ github.token }}
filters: .github/file-paths.yaml
# Similar to .github/workflows/frontend.yml
- name: Prepare front-end environment
if: steps.changes.outputs.frontend_all == 'true'
uses: ./.github/actions/prepare-frontend
- name: Prepare back-end environment
uses: ./.github/actions/prepare-backend
- run: yarn build-pure:cljs
name: Compile CLJS
- name: Run lint-staged
if: steps.changes.outputs.frontend_all == 'true'
run: |
BASE_COMMIT=$(git merge-base origin/${{ github.event.repository.default_branch }} HEAD)
npx lint-staged --shell --no-stash --diff="$BASE_COMMIT...HEAD"
# You must include this step after running all auto-fixing steps
- uses: stefanzweifel/git-auto-commit-action@v5
with:
commit_message: "fix(${{ github.run_id }}): auto linting" # The commit message MUST start with "fix(${{ github.run_id }}):"
commit_options: "--no-verify" # No verify because we already ran it above
skip_fetch: true
skip_checkout: true
status_options: "--untracked-files=no"
add_options: "--update"
- name: Check types
if: steps.changes.outputs.frontend_all == 'true'
run: yarn type-check-pure

View File

@@ -44,8 +44,7 @@ jobs:
continue-on-error: true
uses: ./.github/actions/fetch-artifact
with:
commit: ${{ github.event.inputs.commit || github.event.pull_request.head.sha || github.sha }}
edition: ee
name: metabase-ee-${{ github.event.inputs.commit || github.event.pull_request.head.sha || github.sha }}-uberjar
output-name: metabase-ee.jar
- name: Try to fetch existing OSS artifact
@@ -53,8 +52,7 @@ jobs:
continue-on-error: true
uses: ./.github/actions/fetch-artifact
with:
commit: ${{ github.event.inputs.commit || github.event.pull_request.head.sha || github.sha }}
edition: oss
name: metabase-oss-${{ github.event.inputs.commit || github.event.pull_request.head.sha || github.sha }}-uberjar
output-name: metabase-oss.jar
- name: Decide if build is needed
@@ -91,11 +89,9 @@ jobs:
steps:
- uses: actions/checkout@v4
with:
sparse-checkout: |
release
.github
- name: Prepare release scripts
uses: ./.github/actions/prepare-release-scripts
sparse-checkout: release
- name: Prepare build scripts
run: cd ${{ github.workspace }}/release && yarn && yarn build
- name: Get build dependencies
uses: actions/github-script@v7
id: dependencies
@@ -144,7 +140,7 @@ jobs:
- name: Prepare uberjar artifact
uses: ./.github/actions/prepare-uberjar-artifact
with:
edition: ${{ matrix.edition }}
name: metabase-${{ matrix.edition }}-${{ github.event.inputs.commit || github.event.pull_request.head.sha || github.sha }}-uberjar
check-jar-health:
runs-on: ubuntu-22.04
@@ -175,14 +171,7 @@ jobs:
- name: Retrieve uberjar artifact
uses: ./.github/actions/fetch-artifact
with:
commit: ${{ github.event.inputs.commit || github.event.pull_request.head.sha || github.sha }}
edition: ${{ matrix.edition }}
- name: Verify license files are present in uberjar
run: |
echo "Checking for license files in uberjar..."
unzip -l ./metabase.jar | grep -q "license-backend-third-party.txt" || (echo "ERROR: license-backend-third-party.txt not found in uberjar" && exit 1)
unzip -l ./metabase.jar | grep -q "license-frontend-third-party.txt" || (echo "ERROR: license-frontend-third-party.txt not found in uberjar" && exit 1)
echo "Both license files found in uberjar"
name: metabase-${{ matrix.edition }}-${{ github.event.inputs.commit || github.event.pull_request.head.sha || github.sha }}-uberjar
- name: Launch uberjar
run: >-
java --add-opens java.base/java.nio=ALL-UNNAMED -jar ./metabase.jar &

View File

@@ -23,16 +23,9 @@ jobs:
env:
DAYS_BACK: ${{ inputs.days_back || 7 }}
steps:
- name: Generate GitHub App Token
uses: actions/create-github-app-token@v1
id: app-token
with:
app-id: ${{ secrets.METABASE_BOT_APP_ID }}
private-key: ${{ secrets.METABASE_BOT_APP_PRIVATE_KEY }}
- uses: actions/checkout@v4
with:
token: ${{ steps.app-token.outputs.token }}
token: ${{ secrets.GITHUB_TOKEN }}
- name: Setup Node.js
uses: actions/setup-node@v4
@@ -48,7 +41,7 @@ jobs:
const result = await collectAndAverageTimings({
github,
context,
token: '${{ steps.app-token.outputs.token }}',
token: '${{ secrets.GITHUB_TOKEN }}',
daysBack: ${{ env.DAYS_BACK }},
existingTimingsPath: 'e2e/support/timings.json'
});
@@ -66,7 +59,7 @@ jobs:
- name: Create Pull Request
uses: peter-evans/create-pull-request@v6
with:
token: ${{ steps.app-token.outputs.token }}
token: ${{ secrets.GITHUB_TOKEN }}
branch: ${{ env.BRANCH_NAME }}
base: master
commit-message: |
@@ -96,7 +89,4 @@ jobs:
Co-Authored-By: GitHub Actions <actions@github.com>
labels: |
.CI & Tests
no-backport
add-paths: e2e/support/timings.json
team-reviewers: |
devexp

View File

@@ -31,4 +31,4 @@ jobs:
- uses: actions/checkout@v4
- name: Prepare front-end environment
uses: ./.github/actions/prepare-frontend
- run: bun run lint-yaml
- run: yarn run lint-yaml

11
.gitignore vendored
View File

@@ -16,7 +16,6 @@
.nrepl-port
.portal
.vscode
/.zed
/*.h2.db
/*.lock.db
/*.mv.db
@@ -40,11 +39,10 @@
/cypress
cypress.env.json
/deploy/artifacts/*
**/node_modules/
/docs/**/node_modules
/docs/**/package.json
/docs/**/yarn.lock
/docs/embedding/sdk/api/
/docs/embedding/eajs/snippets/
/frontend/test/snapshots/*
/e2e/snapshots/*
/e2e/tmp/*
@@ -57,6 +55,7 @@ cypress.env.json
/logs
/modules/drivers/*/resources/namespaces.edn
/modules/drivers/*/target
/node_modules/
/plugins
/process.yml
/reset-password-artifacts
@@ -87,6 +86,7 @@ cypress.env.json
/temp
**/cli-snippets-tmp/**
\#*\#
bin/node_modules/
bin/release/aws-eb/metabase-aws-eb.zip
coverage-summary.json
junit.xml
@@ -167,8 +167,3 @@ CLAUDE.local.md
/logs
/.clojure-mcp
# Python dependencies (pip-installed, not committed)
/resources/python-sources/sqlglot/
/resources/python-sources/sqlglot-*.dist-info/
/resources/python-sources/.lock
__pycache__/

View File

@@ -1,4 +1,4 @@
bun run precommit
yarn precommit
# Run the formatter
# Note that this is stubbed in for now until we can gobally resolve some formatting issues such as what is described

Binary file not shown.

Before

Width:  |  Height:  |  Size: 59 KiB

After

Width:  |  Height:  |  Size: 59 KiB

Some files were not shown because too many files have changed in this diff Show More