Compare commits
2 Commits
css-module
...
python-sec
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
3ae1585c7e | ||
|
|
b0b11dca91 |
@@ -81,7 +81,7 @@ Everything else in the module is **internal** and not allowed to be referenced f
|
||||
|
||||
`dev/src/dev/modularization_help.clj` — `potemkin-ns!` helper that generates `.core` re-export boilerplate from a target namespace.
|
||||
|
||||
`dev/test/metabase/core/modules_test.clj` — runs on config changes; validates structure, team assignments, API and model boundaries.
|
||||
`test/metabase/core/modules_test.clj` — runs on config changes; validates structure, team assignments, API and model boundaries.
|
||||
|
||||
### API Design Inside a Module
|
||||
|
||||
@@ -119,7 +119,7 @@ When the linter complains, the answer is **always refactoring**, never suppressi
|
||||
- `dev/src/dev/module_score.clj` — quality scoring
|
||||
- `dev/src/dev/modularization_help.clj` — Potemkin re-export helper
|
||||
- `dev/src/dev/model_boundary_config.clj` + `dev/src/dev/model_tracking.clj` — model-level enforcement
|
||||
- `dev/test/metabase/core/modules_test.clj` — config validation tests
|
||||
- `test/metabase/core/modules_test.clj` — config validation tests
|
||||
- `src/metabase/core/init.clj` — OSS module init aggregation
|
||||
- `enterprise/backend/src/metabase_enterprise/core/init.clj` — EE module init aggregation
|
||||
- `src/metabase/models/resolution.clj` — `:model/X` keyword → namespace map
|
||||
@@ -198,6 +198,6 @@ Use the `clojure-eval` skill (preferred) or `clj-nrepl-eval` to drive the dev to
|
||||
(dev.module-score/scores (dev.module-score/deps) (dev.module-score/config))
|
||||
```
|
||||
|
||||
For tests of the config itself, use `./bin/mage project-tests modules`. After editing Clojure files, run `clj-paren-repair`. After editing `config.edn`, re-run the modules-test.
|
||||
For tests of the config itself, use `./bin/test-agent :only '[metabase.core.modules-test]'`. After editing Clojure files, run `clj-paren-repair`. After editing `config.edn`, re-run the modules-test.
|
||||
|
||||
**Update your agent memory** as you discover idioms: which modules use `:friends` and why, which models are most-imported (suggesting they belong in their own module), which settings are accessed cross-module via `(setting/get …)` instead of `:uses` deps, and which refactors the team is actively running.
|
||||
|
||||
@@ -27,28 +27,9 @@ This guide covers Clojure and ClojureScript coding conventions for Metabase. See
|
||||
- Format using Markdown conventions
|
||||
- Reference other vars with `[[other-var]]` not backticks
|
||||
|
||||
A docstring states the *contract*: what it returns (including when it returns nil), what its
|
||||
params mean, what it throws versus swallows, and any precondition or invariant the caller must
|
||||
respect. A params block for non-obvious or optional args is good documentation — length is not
|
||||
the problem.
|
||||
|
||||
These seven things are, and each is a reason to cut or relocate:
|
||||
|
||||
| Anti-pattern | Why | Where it goes instead |
|
||||
| --- | --- | --- |
|
||||
| **Facts you don't own** — how a *callee*, another namespace, or another repo works internally | Drifts silently; nothing fails when it goes stale | Cut. Name the callee and let its own docstring say it |
|
||||
| **Said twice** — ns docstring restating its vars', a caller restating its callee's, sibling fns repeating boilerplate | Two copies means one is already wrong | Cut the copy. One fact, one owner |
|
||||
| **Design rationale** — why this approach, why it lives in this module, what the alternative was | The caller can't act on it | Commit message, or an inline comment on the specific line it constrains |
|
||||
| **Development history** — "matching master", "the X branch", "currently", "for now", "used to" | Stale by construction | Cut |
|
||||
| **Instructions to the reader** — "callers should trust this", "don't bother checking", "you should use X" | State the contract; the caller decides | Cut, keeping any actual guarantee as a plain statement |
|
||||
| **Invented precision** — score bands, thresholds, or tables no code enforces | Reads as spec, isn't | Cut, or make the code enforce it |
|
||||
| **A note on the wrong var** — describing how overrides behave, on the base/default impl | Nobody reading the override sees it | Move to the overrides it describes |
|
||||
|
||||
**Comments:**
|
||||
|
||||
- `TODO` format: `;; TODO (Name YYYY-MM-DD) -- description`
|
||||
- Prefer an inline comment over docstring prose for anything that explains *the body*. Put it on
|
||||
the line it explains, not at the top of the function.
|
||||
|
||||
## Code Organization
|
||||
|
||||
@@ -304,13 +285,12 @@ Tag response Malli `:map` schemas with `{:closed true}` so the generated OpenAPI
|
||||
|
||||
**Linter Suppressions:**
|
||||
|
||||
The right shape is a **reader-discard map** placed directly above the form being silenced, listing the
|
||||
rule keywords explicitly and including the required explanation:
|
||||
The right shape is a **reader-discard map** placed directly above the form being silenced, listing the rule keywords explicitly:
|
||||
|
||||
```clojure
|
||||
;; Loaded by name from the plugin registry.
|
||||
#_{:clj-kondo/ignore [:clojure-lsp/unused-public-var]}
|
||||
(defn plugin-entrypoint ...)
|
||||
#_{:clj-kondo/ignore [:metabase/validate-defendpoint-route-uses-kebab-case
|
||||
:metabase/validate-defendpoint-has-response-schema]}
|
||||
(api.macros/defendpoint :post "/api_key" ...)
|
||||
```
|
||||
|
||||
For a single require entry, hang it as metadata on that entry:
|
||||
@@ -321,12 +301,7 @@ For a single require entry, hang it as metadata on that entry:
|
||||
[metabase.query-processor.store :as qp.store])
|
||||
```
|
||||
|
||||
Use the map form, not the keyword form (`#_:clj-kondo/ignore`), so searches can identify the suppressed
|
||||
rules. Keep `:clj-kondo/ignore` as the map's first key; the ratchet scanner rejects other orderings.
|
||||
|
||||
Fix the warning when possible. Adding a suppression is a last resort and requires approval. If one is
|
||||
necessary, add an explanatory `;;` comment directly above it or at the end of the same line. CI enforces
|
||||
these requirements with `./bin/mage kondo-ratchets`.
|
||||
Common Metabase-specific rules you'll see: `:metabase/modules`, `:metabase/validate-defendpoint-route-uses-kebab-case`, `:metabase/validate-defendpoint-has-response-schema`, `:metabase/prefer-with-dynamic-fn-redefs`, `:deprecated-namespace`, `:discouraged-namespace`. Don't use the keyword form (`#_:clj-kondo/ignore`); always the map form so it's grep-able by which rule(s) are suppressed.
|
||||
|
||||
**Configurable Options:**
|
||||
|
||||
|
||||
@@ -6,4 +6,3 @@
|
||||
- Run targeted tests, and lint continuously during development
|
||||
- Prioritize understanding existing patterns before implementing
|
||||
- Don't commit changes, leave it for the user to review and make commits
|
||||
- Minimize comments. Document only non-obvious intent, trade-offs, or constraints that the code itself cannot convey. Do not narrate what the code does — keep code self-descriptive through naming and structure.
|
||||
|
||||
@@ -10,23 +10,23 @@ This skill helps you add product analytics (Snowplow) events to track user inter
|
||||
|
||||
## Quick Reference
|
||||
|
||||
Analytics events in Metabase use Snowplow with typed event schemas. Simple events are declared **where they are used** — `trackSimpleEvent` is generic and validates the payload at the call site.
|
||||
Analytics events in Metabase use Snowplow with typed event schemas. All events must be defined in TypeScript types before use.
|
||||
|
||||
**Key Files:**
|
||||
- `frontend/src/metabase/analytics/event.ts` - Core tracking functions, `trackSimpleEvent` / `trackSchemaEvent` (import from `metabase/analytics`)
|
||||
- `frontend/src/metabase-types/analytics/event.ts` - The shared `SimpleEventSchema` only. **Do not add event types here** (see below)
|
||||
- `frontend/src/metabase-types/analytics/schema.ts` - Schema registry (custom/legacy schemas only)
|
||||
- Feature-specific `analytics.ts` files - Where your tracking functions and any local types live
|
||||
- `frontend/src/metabase-types/analytics/event.ts` - Event type definitions
|
||||
- `frontend/src/metabase-types/analytics/schema.ts` - Schema registry
|
||||
- `frontend/src/metabase/analytics/` - Core tracking functions (import from `metabase/analytics`)
|
||||
- Feature-specific `analytics.ts` files - Tracking function wrappers
|
||||
|
||||
## Quick Checklist
|
||||
|
||||
When adding a new analytics event:
|
||||
|
||||
- [ ] Pick an event name (snake_case, past tense)
|
||||
- [ ] Add a tracking function to the feature's `analytics.ts` file, calling `trackSimpleEvent()`
|
||||
- [ ] Keep any field unions (e.g. `"success" | "failure"`) as local types in that same file
|
||||
- [ ] Import and call the tracking function at the interaction point
|
||||
- [ ] Do **not** add an event type to `metabase-types/analytics/event.ts` or to any union
|
||||
- [ ] Define event type in `frontend/src/metabase-types/analytics/event.ts`
|
||||
- [ ] Add event to appropriate union type (e.g., `DataStudioEvent`, `SimpleEvent`)
|
||||
- [ ] Create tracking function in feature's `analytics.ts` file
|
||||
- [ ] Import and call tracking function at the interaction point
|
||||
- [ ] Use `trackSimpleEvent()` for basic events (most common)
|
||||
|
||||
## Event Schema Types
|
||||
|
||||
@@ -47,25 +47,6 @@ type SimpleEventSchema = {
|
||||
|
||||
**When to use:** 90% of events fit this schema. Use for clicks, opens, closes, creates, deletes, etc.
|
||||
|
||||
`trackSimpleEvent` is generic and enforces this schema on the object literal you pass it:
|
||||
|
||||
```typescript
|
||||
// frontend/src/metabase/analytics/event.ts
|
||||
export function trackSimpleEvent<
|
||||
T extends SimpleEventSchema &
|
||||
Record<Exclude<keyof T, keyof SimpleEventSchema>, never>,
|
||||
>(event: T) {
|
||||
trackSchemaEvent("simple_event", event);
|
||||
}
|
||||
```
|
||||
|
||||
That means a missing `event` or any field outside `SimpleEventSchema` is a compile error at the call
|
||||
site. There is no separate event type to declare and no `satisfies` clause to add — the old
|
||||
`ValidateEvent<...>` helper is no longer exported and is not part of the workflow.
|
||||
|
||||
`trackSchemaEvent` is generic too: it correlates the schema name with the payload type, so you can't
|
||||
send a dashboard event under the `simple_event` schema.
|
||||
|
||||
### 2. Custom Schemas (legacy, no events are being added)
|
||||
|
||||
Consider adding new event schema only in very special cases.
|
||||
@@ -76,7 +57,36 @@ Consider adding new event schema only in very special cases.
|
||||
|
||||
### Example: Track when a user applies filters in a table picker
|
||||
|
||||
#### Step 1: Create Tracking Functions
|
||||
#### Step 1: Define Event Types
|
||||
|
||||
Add event type definitions to `frontend/src/metabase-types/analytics/event.ts`:
|
||||
|
||||
```typescript
|
||||
export type DataStudioTablePickerFiltersAppliedEvent = ValidateEvent<{
|
||||
event: "data_studio_table_picker_filters_applied";
|
||||
}>;
|
||||
|
||||
export type DataStudioTablePickerFiltersClearedEvent = ValidateEvent<{
|
||||
event: "data_studio_table_picker_filters_cleared";
|
||||
}>;
|
||||
```
|
||||
|
||||
#### Step 2: Add to Union Type
|
||||
|
||||
Find or create the appropriate union type and add your events:
|
||||
|
||||
```typescript
|
||||
export type DataStudioEvent =
|
||||
| DataStudioLibraryCreatedEvent
|
||||
| DataStudioTablePublishedEvent
|
||||
| DataStudioGlossaryCreatedEvent
|
||||
| DataStudioGlossaryEditedEvent
|
||||
| DataStudioGlossaryDeletedEvent
|
||||
| DataStudioTablePickerFiltersAppliedEvent // <- Add here
|
||||
| DataStudioTablePickerFiltersClearedEvent; // <- Add here
|
||||
```
|
||||
|
||||
#### Step 3: Create Tracking Functions
|
||||
|
||||
In your feature's `analytics.ts` file (e.g., `enterprise/frontend/src/metabase-enterprise/data-studio/analytics.ts`):
|
||||
|
||||
@@ -96,7 +106,7 @@ export const trackDataStudioTablePickerFiltersCleared = () => {
|
||||
};
|
||||
```
|
||||
|
||||
#### Step 2: Use in Components
|
||||
#### Step 4: Use in Components
|
||||
|
||||
Import and call the tracking function at the interaction point:
|
||||
|
||||
@@ -128,11 +138,16 @@ function FilterPopover({ filters, onSubmit }) {
|
||||
|
||||
## Using SimpleEventSchema Fields
|
||||
|
||||
All examples below live in the feature's own `analytics.ts` — nothing is registered centrally.
|
||||
|
||||
### Example: Event with target_id
|
||||
|
||||
```typescript
|
||||
// Type definition
|
||||
export type DataStudioLibraryCreatedEvent = ValidateEvent<{
|
||||
event: "data_studio_library_created";
|
||||
target_id: number | null;
|
||||
}>;
|
||||
|
||||
// Tracking function
|
||||
export const trackDataStudioLibraryCreated = (id: CollectionId) => {
|
||||
trackSimpleEvent({
|
||||
event: "data_studio_library_created",
|
||||
@@ -147,10 +162,14 @@ trackDataStudioLibraryCreated(newLibrary.id);
|
||||
### Example: Event with triggered_from
|
||||
|
||||
```typescript
|
||||
// Local union, exported only if another feature needs to pass the same value
|
||||
export type NewButtonLocation = "app-bar" | "empty-collection";
|
||||
// Type definition
|
||||
export type NewButtonClickedEvent = ValidateEvent<{
|
||||
event: "new_button_clicked";
|
||||
triggered_from: "app-bar" | "empty-collection";
|
||||
}>;
|
||||
|
||||
export const trackNewButtonClicked = (location: NewButtonLocation) => {
|
||||
// Tracking function
|
||||
export const trackNewButtonClicked = (location: "app-bar" | "empty-collection") => {
|
||||
trackSimpleEvent({
|
||||
event: "new_button_clicked",
|
||||
triggered_from: location,
|
||||
@@ -168,31 +187,44 @@ export const trackNewButtonClicked = (location: NewButtonLocation) => {
|
||||
|
||||
### Example: Event with event_detail
|
||||
|
||||
Real example — `frontend/src/metabase/metadata/pages/shared/analytics.ts`:
|
||||
|
||||
```typescript
|
||||
export type MetadataEditEventDetail =
|
||||
| "type_casting"
|
||||
| "semantic_type_change"
|
||||
| "visibility_change";
|
||||
// Type definition
|
||||
export type MetadataEditEvent = ValidateEvent<{
|
||||
event: "metadata_edited";
|
||||
event_detail: "type_casting" | "semantic_type_change" | "visibility_change";
|
||||
triggered_from: "admin" | "data_studio";
|
||||
}>;
|
||||
|
||||
export const trackMetadataChange = (detail: MetadataEditEventDetail) => {
|
||||
// Tracking function
|
||||
export const trackMetadataChange = (
|
||||
detail: "type_casting" | "semantic_type_change" | "visibility_change",
|
||||
location: "admin" | "data_studio"
|
||||
) => {
|
||||
trackSimpleEvent({
|
||||
event: "metadata_edited",
|
||||
event_detail: detail,
|
||||
triggered_from: "admin",
|
||||
triggered_from: location,
|
||||
});
|
||||
};
|
||||
|
||||
// Usage
|
||||
trackMetadataChange("semantic_type_change");
|
||||
trackMetadataChange("semantic_type_change", "data_studio");
|
||||
```
|
||||
|
||||
### Example: Event with result and duration
|
||||
|
||||
See `frontend/src/metabase/archive/analytics.ts` for the real version of this.
|
||||
|
||||
```typescript
|
||||
// Type definition
|
||||
export type MoveToTrashEvent = ValidateEvent<{
|
||||
event: "moved-to-trash";
|
||||
target_id: number | null;
|
||||
triggered_from: "collection" | "detail_page" | "cleanup_modal";
|
||||
duration_ms: number | null;
|
||||
result: "success" | "failure";
|
||||
event_detail: "question" | "model" | "metric" | "dashboard";
|
||||
}>;
|
||||
|
||||
// Tracking function
|
||||
export const trackMoveToTrash = (params: {
|
||||
targetId: number | null;
|
||||
triggeredFrom: "collection" | "detail_page" | "cleanup_modal";
|
||||
@@ -248,16 +280,17 @@ try {
|
||||
"filters-applied" // Use underscore, not hyphen
|
||||
```
|
||||
|
||||
### Local Field Types (PascalCase, named after the field)
|
||||
|
||||
There is usually no `...Event` type to name anymore. When you do need a union for a field, name it
|
||||
after the field it feeds:
|
||||
### Event Type Names (PascalCase with "Event" suffix)
|
||||
|
||||
```typescript
|
||||
// Good
|
||||
type MetricDimensionResult = "success" | "failure"; // -> result
|
||||
export type MetadataEditEventDetail = "type_casting"; // -> event_detail
|
||||
type NewButtonLocation = "app-bar" | "empty-collection"; // -> triggered_from
|
||||
DataStudioLibraryCreatedEvent
|
||||
TablePickerFiltersAppliedEvent
|
||||
MetabotChatOpenedEvent
|
||||
|
||||
// Bad
|
||||
dataStudioLibraryCreated // Wrong case
|
||||
DataStudioLibraryCreated // Missing "Event" suffix
|
||||
```
|
||||
|
||||
### Tracking Function Names (camelCase with "track" prefix)
|
||||
@@ -276,28 +309,29 @@ logLibraryCreated // Use "track" prefix
|
||||
|
||||
## Common Patterns
|
||||
|
||||
### Pattern 1: Sharing Field Types Across Features
|
||||
### Pattern 1: Feature-Specific Union Types
|
||||
|
||||
When two features send the same event with a different `triggered_from`, export the field union from
|
||||
the owning feature's `analytics.ts` and import it — don't hoist anything into `metabase-types`:
|
||||
Group related events together:
|
||||
|
||||
```typescript
|
||||
// frontend/src/metabase/data-studio/data-model/analytics.ts
|
||||
import { trackSimpleEvent } from "metabase/analytics";
|
||||
import type { MetadataEditEventDetail } from "metabase/metadata/pages/shared/analytics";
|
||||
export type DataStudioEvent =
|
||||
| DataStudioLibraryCreatedEvent
|
||||
| DataStudioTablePublishedEvent
|
||||
| DataStudioGlossaryCreatedEvent;
|
||||
|
||||
export function trackMetadataChange(detail: MetadataEditEventDetail) {
|
||||
trackSimpleEvent({
|
||||
event: "metadata_edited",
|
||||
event_detail: detail,
|
||||
triggered_from: "data_studio",
|
||||
});
|
||||
}
|
||||
export type MetabotEvent =
|
||||
| MetabotChatOpenedEvent
|
||||
| MetabotRequestSentEvent
|
||||
| MetabotFixQueryClickedEvent;
|
||||
|
||||
// Then add to SimpleEvent union
|
||||
export type SimpleEvent =
|
||||
| /* other events */
|
||||
| DataStudioEvent
|
||||
| MetabotEvent
|
||||
| /* more events */;
|
||||
```
|
||||
|
||||
This is the point of the extensible-events design: enterprise and feature-tier types stay in their
|
||||
own module instead of being imported down into a shared union.
|
||||
|
||||
### Pattern 2: Conditional Tracking
|
||||
|
||||
Track different events based on user action:
|
||||
@@ -316,10 +350,10 @@ const handleSave = async () => {
|
||||
|
||||
## Common Pitfalls
|
||||
|
||||
### Don't: Add custom fields to a simple event
|
||||
### Don't: Add custom fields to SimpleEvent
|
||||
|
||||
```typescript
|
||||
// WRONG - SimpleEventSchema doesn't support custom fields (this is a compile error)
|
||||
// WRONG - SimpleEvent doesn't support custom fields
|
||||
export const trackFiltersApplied = (filters: FilterState) => {
|
||||
trackSimpleEvent({
|
||||
event: "filters_applied",
|
||||
@@ -345,43 +379,23 @@ export const trackFilterApplied = (filterType: string) => {
|
||||
};
|
||||
```
|
||||
|
||||
### Don't: Add event types to `metabase-types/analytics/event.ts`
|
||||
|
||||
The central `SimpleEvent` union was removed — it forced feature-tier types to be imported down into
|
||||
shared code, causing module-boundary violations. `trackSimpleEvent` is generic now, so the type adds
|
||||
nothing but duplication.
|
||||
### Don't: Forget to add event to union type
|
||||
|
||||
```typescript
|
||||
// ❌ WRONG - central declaration + re-import for a `satisfies` clause
|
||||
// frontend/src/metabase-types/analytics/event.ts
|
||||
// Define the event
|
||||
export type NewFeatureClickedEvent = ValidateEvent<{
|
||||
event: "new_feature_clicked";
|
||||
target_id: number;
|
||||
}>;
|
||||
|
||||
// frontend/src/metabase/my-feature/analytics.ts
|
||||
import type { NewFeatureClickedEvent } from "metabase-types/analytics";
|
||||
// ❌ WRONG - Forgot to add to SimpleEvent union
|
||||
// Event won't be recognized by TypeScript
|
||||
|
||||
export const trackNewFeatureClicked = (id: number) => {
|
||||
trackSimpleEvent({
|
||||
event: "new_feature_clicked",
|
||||
target_id: id,
|
||||
} satisfies NewFeatureClickedEvent);
|
||||
};
|
||||
|
||||
// ✓ RIGHT - the object literal is already checked by the generic
|
||||
// frontend/src/metabase/my-feature/analytics.ts
|
||||
export const trackNewFeatureClicked = (id: number) => {
|
||||
trackSimpleEvent({
|
||||
event: "new_feature_clicked",
|
||||
target_id: id,
|
||||
});
|
||||
};
|
||||
// ✓ RIGHT - Add to appropriate union
|
||||
export type SimpleEvent =
|
||||
| /* other events */
|
||||
| NewFeatureClickedEvent;
|
||||
```
|
||||
|
||||
A few `...Event` types still sit in `metabase-types/analytics/event.ts`. They are leftovers from PRs
|
||||
that landed around the refactor — don't copy them, and don't add to them.
|
||||
|
||||
### Don't: Mix up event name formats
|
||||
|
||||
```typescript
|
||||
@@ -449,44 +463,41 @@ Example console output:
|
||||
### Where to put tracking functions:
|
||||
|
||||
```
|
||||
Tracking functions AND their local field types (this is where new events live):
|
||||
Feature-specific analytics functions:
|
||||
frontend/src/metabase/{feature}/analytics.ts
|
||||
enterprise/frontend/src/metabase-enterprise/{feature}/analytics.ts
|
||||
|
||||
Event type definitions (all in one place):
|
||||
frontend/src/metabase-types/analytics/event.ts
|
||||
|
||||
Core tracking utilities:
|
||||
frontend/src/metabase/analytics/ (import from `metabase/analytics`)
|
||||
|
||||
Shared SimpleEventSchema only — nothing new goes here:
|
||||
frontend/src/metabase-types/analytics/event.ts
|
||||
```
|
||||
|
||||
In embedding SDK code, use `trackSdkSimpleEvent`
|
||||
(`frontend/src/embedding-sdk-bundle/analytics/snowplow.ts`) instead — the main-app `"sp"` tracker
|
||||
isn't initialized in the customer's page, so `trackSimpleEvent`'s Snowplow leg is a no-op there.
|
||||
|
||||
## Real-World Examples
|
||||
|
||||
See these files for reference:
|
||||
|
||||
- **Simple events + local field union**: `frontend/src/metabase/metadata/pages/shared/analytics.ts`
|
||||
- **Reusing another feature's field type**: `frontend/src/metabase/data-studio/data-model/analytics.ts`
|
||||
- **Result + duration timing**: `frontend/src/metabase/archive/analytics.ts`
|
||||
- **Enterprise feature events**: `enterprise/frontend/src/metabase-enterprise/google_drive/analytics.ts`
|
||||
- **Simple events**: `enterprise/frontend/src/metabase-enterprise/data-studio/analytics.ts`
|
||||
- **Events with variants**: `frontend/src/metabase/dashboard/analytics.ts`
|
||||
- **Complex events**: `frontend/src/metabase/query_builder/analytics.js`
|
||||
- **Event type examples**: `frontend/src/metabase-types/analytics/event.ts`
|
||||
|
||||
## Workflow Summary
|
||||
|
||||
1. **Identify the user interaction** to track
|
||||
2. **Decide on event name** (snake_case, descriptive)
|
||||
3. **Create tracking function** in feature's `analytics.ts`, calling `trackSimpleEvent()`
|
||||
4. **Add local field unions** in that same file if a field has a fixed set of values
|
||||
5. **Import and call** at the interaction point
|
||||
6. **Test** that events fire correctly
|
||||
3. **Define event type** in `event.ts` using `ValidateEvent`
|
||||
4. **Add to union type** (create feature union if needed)
|
||||
5. **Create tracking function** in feature's `analytics.ts`
|
||||
6. **Import and call** at the interaction point
|
||||
7. **Test** that events fire correctly
|
||||
|
||||
## Tips
|
||||
|
||||
- **Be specific** - `filters_applied` is better than `action_performed`
|
||||
- **Use past tense** - `library_created` not `create_library`
|
||||
- **Group related events** - Keep a feature's tracking functions together in its `analytics.ts`
|
||||
- **Group related events** - Create feature-specific event union types
|
||||
- **Track meaningful actions** - Not every click needs tracking
|
||||
- **Consider the data** - What would you want to analyze later?
|
||||
- **Stay consistent** - Follow existing naming patterns in the codebase
|
||||
|
||||
@@ -56,11 +56,6 @@ Use this to scan through changes efficiently:
|
||||
- [ ] References use `[[other-var]]` not backticks
|
||||
- [ ] `TODO` comments include author and date: `;; TODO (Name 2025-01-01) -- description`
|
||||
|
||||
Docstring content — check every new or rewritten docstring against the anti-pattern table in the
|
||||
style guide above. Read the diff's docstrings on their own, separately from the code: prose is
|
||||
where review attention slides off, and a long docstring reads as diligence rather than as the
|
||||
liability it usually is.
|
||||
|
||||
### Code Organization
|
||||
|
||||
- [ ] Everything `^:private` unless used elsewhere
|
||||
@@ -111,12 +106,7 @@ liability it usually is.
|
||||
### Miscellaneous
|
||||
|
||||
- [ ] Example data is bird-themed when possible
|
||||
- [ ] Kondo suppressions use a reader-discard map, not `#_:clj-kondo/ignore`, with
|
||||
`:clj-kondo/ignore` as the first key
|
||||
- [ ] Each new suppression is approved and has a `;;` comment explaining why the warning cannot be fixed
|
||||
- [ ] Any ratchet budget increase is explained in the PR
|
||||
- [ ] PRs that remove ignores leave ratchet budgets unchanged; the shrink workflow records reductions on
|
||||
`master`
|
||||
- [ ] Kondo linter suppressions use proper format (not `#_:clj-kondo/ignore` keyword form)
|
||||
|
||||
## Pattern matching table
|
||||
|
||||
@@ -145,17 +135,6 @@ Quick scan for common issues:
|
||||
|
||||
> This public var needs a docstring explaining its purpose, inputs, and outputs.
|
||||
|
||||
**For docstrings documenting code they don't own:**
|
||||
|
||||
> This describes how `lib` decides binning strategies internally. Nothing here fails when that
|
||||
> changes, so it will go stale silently — drop it and let `available-binning-strategies` answer
|
||||
> for its own behavior.
|
||||
|
||||
**For docstring content that belongs in the body:**
|
||||
|
||||
> This paragraph explains why the branches are ordered this way, which is what someone editing
|
||||
> the `cond` needs — but a caller can't act on it. Move it to an inline comment above the `cond`.
|
||||
|
||||
**For organization issues:**
|
||||
|
||||
> This function is only used in this namespace, so it should be marked `^:private`.
|
||||
|
||||
@@ -82,29 +82,10 @@ either a bug, or it means callers must handle each case themselves — and
|
||||
in *that* case it is the *absence* of the property that must be
|
||||
documented. Document deviations from expectation, not conformance to it.
|
||||
|
||||
Heuristic — a sentence earns its place in the docstring only if it
|
||||
passes both tests:
|
||||
|
||||
1. Rewrite test (necessary, not sufficient): if rewriting the body to a
|
||||
different implementation with an identical contract could make the
|
||||
sentence false, it describes the implementation, not the contract.
|
||||
Relocate it to an inline comment at the point in the body where it
|
||||
applies.
|
||||
|
||||
2. Ownership test: if the sentence states a fact about code this
|
||||
function does not own — a callee's behavior, a library's guarantee —
|
||||
it survives any rewrite trivially, which is exactly why the rewrite
|
||||
test alone is not enough. If this body *relies* on that fact, it is
|
||||
implementation context: relocate it to the call site. If nothing here
|
||||
relies on it, it is blather about someone else's code: delete it, and
|
||||
let the callee's own docstring answer for it.
|
||||
|
||||
Say each fact once. Duplicated prose is a stale comment in advance: when
|
||||
the fact changes, one copy gets updated and the rest quietly rot. This
|
||||
holds even within code you own — a namespace docstring that
|
||||
re-summarizes its own multimethods, three sibling functions carrying the
|
||||
same paragraph about retry behavior. Give the fact one home, and have
|
||||
the other places name it rather than restate it.
|
||||
Heuristic: if a sentence would still be true after a full rewrite of the
|
||||
body, it may belong in the docstring. If it describes *how the current
|
||||
body works*, it belongs in the body — as an inline comment, if it is
|
||||
non-obvious.
|
||||
|
||||
Multi-line docstrings are not banned — a genuinely non-obvious constraint
|
||||
the code had to deal with can be worth explaining. But be prudent; the
|
||||
@@ -123,12 +104,5 @@ breakage keeps happening, *that* is the signal a comment was warranted.
|
||||
- Spaces on a line with nothing after it is not allowed
|
||||
- After changing module boundaries (adding/removing/renaming a `src` namespace, a cross-module `require`
|
||||
or `:model/X` reference, or a new module), run `./bin/mage fix-modules-config` to regenerate
|
||||
`.clj-kondo/config/modules/config.edn`, then run `./bin/mage project-tests modules`. The fixer is a no-op when
|
||||
nothing drifted; see "Module Boundaries" in the project `CLAUDE.md`.
|
||||
- Fix kondo warnings instead of suppressing them. Add a suppression only as a last resort, with approval
|
||||
and a `;;` comment explaining why it is necessary.
|
||||
- Do not lower budgets in `.clj-kondo/ratchets.edn` after removing ignores. The shrink workflow records
|
||||
reductions on `master`.
|
||||
- If `.clj-kondo/ratchets.edn` conflicts during a merge, rebase, or restack, run
|
||||
`./bin/merge-kondo-ratchets`. It preserves one-sided changes, chooses the policy that allows fewer
|
||||
suppressions when both sides change the same entry, and stops when the conflict needs a human decision.
|
||||
`.clj-kondo/config/modules/config.edn` and keep `metabase.core.modules-test` green. No-op when nothing
|
||||
drifted; see "Module Boundaries" in the project `CLAUDE.md`.
|
||||
|
||||
@@ -37,16 +37,14 @@ First check if snapshots exist:
|
||||
ls e2e/snapshots/default.sql 2>/dev/null && echo "snapshots exist" || echo "snapshots missing"
|
||||
```
|
||||
|
||||
Always pass the spec path via `--spec`. The runner feeds its arguments to Cypress's CLI parser (`cypress.cli.parseRunArguments`), which silently drops bare positional paths and then runs the entire suite. A leading `run` argument is harmless, but the path itself must follow `--spec`. Sanity-check that the first `Running:` line says `(1 of 1)`.
|
||||
|
||||
If snapshots exist, skip regeneration for speed:
|
||||
```bash
|
||||
MB_EDITION=oss CYPRESS_VIDEO=false CYPRESS_RETRIES=0 CYPRESS_GUI=false GENERATE_SNAPSHOTS=false bun test-cypress --spec path/to/spec.cy.spec.js
|
||||
MB_EDITION=oss CYPRESS_VIDEO=false CYPRESS_RETRIES=0 CYPRESS_GUI=false GENERATE_SNAPSHOTS=false bun test-cypress $ARGUMENTS
|
||||
```
|
||||
|
||||
If snapshots are missing (first run), let the runner generate them:
|
||||
```bash
|
||||
MB_EDITION=oss CYPRESS_VIDEO=false CYPRESS_RETRIES=0 CYPRESS_GUI=false bun test-cypress --spec path/to/spec.cy.spec.js
|
||||
MB_EDITION=oss CYPRESS_VIDEO=false CYPRESS_RETRIES=0 CYPRESS_GUI=false bun test-cypress $ARGUMENTS
|
||||
```
|
||||
|
||||
When running enterprise tests, replace `MB_EDITION=oss` with `MB_EDITION=ee` in the commands above.
|
||||
@@ -69,11 +67,9 @@ GREP="test name here" MB_EDITION=oss bun test-cypress --spec path/to/spec.js
|
||||
When the user asks to verify a test is not flaky:
|
||||
|
||||
```bash
|
||||
MB_EDITION=oss bin/e2e-stress-test --spec path/to/spec.cy.spec.js
|
||||
MB_EDITION=oss bin/e2e-stress-test $ARGUMENTS
|
||||
```
|
||||
|
||||
(The stress script forwards its arguments to `bun test-cypress`, so the same `--spec` rule applies.)
|
||||
|
||||
Set `E2E_STRESS_RUNS=N` for more iterations (default: 5). Stops on first failure and prints screenshot paths. Read those screenshots to analyze the failure.
|
||||
|
||||
## Analyzing failures
|
||||
|
||||
@@ -171,7 +171,7 @@ Create `ComponentName.module.css` alongside the component file:
|
||||
/* Use design system tokens — NEVER raw color/spacing values */
|
||||
.root {
|
||||
border: 1px solid var(--mb-color-border-neutral);
|
||||
border-radius: var(--mantine-radius-sm);
|
||||
border-radius: var(--mantine-radius-md);
|
||||
background-color: var(--mb-color-background_page-primary);
|
||||
}
|
||||
|
||||
@@ -246,8 +246,7 @@ Remove the old styled file entirely. Remove all imports of it from other files.
|
||||
|
||||
**Spacing** (`p`, `px`, `py`, `pt`, `pb`, `pl`, `pr`, `m`, `mx`, `my`, `mt`, `mb`, `ml`, `mr`):
|
||||
|
||||
- `"xxxs"` = 0.125rem, `"xxs"` = 0.25rem, `"xs"` = 0.375rem, `"sm"` = 0.5rem, `"md"` = 0.75rem, `"lg"` = 1rem, `"xl"` = 1.5rem, `"xxl"` = 2rem, `"xxxl"` = 2.5rem
|
||||
- Use numeric `0` for zero spacing.
|
||||
- `"xs"` = 4px, `"sm"` = 8px, `"md"` = 16px, `"lg"` = 24px, `"xl"` = 32px
|
||||
- Custom: `rem(48)` for non-standard values (import `rem` from `metabase/ui`)
|
||||
|
||||
**Dimensions** (`w`, `h`, `maw`, `mah`, `miw`, `mih`):
|
||||
@@ -290,16 +289,11 @@ Remove the old styled file entirely. Remove all imports of it from other files.
|
||||
|
||||
**Mantine spacing** — `var(--mantine-spacing-<size>)`:
|
||||
|
||||
- `--mantine-spacing-xxxs` (0.125rem), `--mantine-spacing-xxs` (0.25rem), `--mantine-spacing-xs` (0.375rem), `--mantine-spacing-sm` (0.5rem), `--mantine-spacing-md` (0.75rem), `--mantine-spacing-lg` (1rem), `--mantine-spacing-xl` (1.5rem), `--mantine-spacing-xxl` (2rem), `--mantine-spacing-xxxl` (2.5rem)
|
||||
- `--mantine-spacing-xs` (4px), `--mantine-spacing-sm` (8px), `--mantine-spacing-md` (16px), `--mantine-spacing-lg` (24px), `--mantine-spacing-xl` (32px)
|
||||
|
||||
**Mantine radius** — `var(--mantine-radius-<size>)`:
|
||||
|
||||
- `--mantine-radius-xxxs` (0.125rem), `--mantine-radius-xxs` (0.25rem), `--mantine-radius-xs` (0.375rem), `--mantine-radius-sm` (0.5rem), `--mantine-radius-md` (0.75rem), `--mantine-radius-lg` (1rem), `--mantine-radius-xl` (1.5rem)
|
||||
|
||||
**Mantine shadows (elevation)** — `var(--mantine-shadow-<size>)`:
|
||||
|
||||
- `--mantine-shadow-xs`, `--mantine-shadow-xs_outline`, `--mantine-shadow-sm`, `--mantine-shadow-sm_outline`, `--mantine-shadow-lg_outline`
|
||||
- Light and dark modes use different shadow values (defined in `frontend/src/metabase/ui/theme.ts`)
|
||||
- `--mantine-radius-xs` (4px), `--mantine-radius-sm` (6px), `--mantine-radius-md` (8px), `--mantine-radius-xl` (40px)
|
||||
|
||||
**Mantine font sizes** — `var(--mantine-font-size-<size>)`:
|
||||
|
||||
@@ -319,36 +313,24 @@ When migrating, **never carry over hardcoded `rem`/`px` values** from the origin
|
||||
|
||||
**Spacing** — snap to the nearest Mantine spacing token:
|
||||
|
||||
| Hardcoded value | Nearest token | Style prop | CSS variable |
|
||||
| --------------------------------- | ------------- | ---------- | ------------------------------- |
|
||||
| `0` (0px) | — | `0` | `0` |
|
||||
| `0.125rem` (2px) | **xxxs** | `"xxxs"` | `var(--mantine-spacing-xxxs)` |
|
||||
| `0.2rem` (3.2px), `0.25rem` (4px) | **xxs** | `"xxs"` | `var(--mantine-spacing-xxs)` |
|
||||
| `0.375rem` (6px) | **xs** | `"xs"` | `var(--mantine-spacing-xs)` |
|
||||
| `0.5rem` (8px) | **sm** | `"sm"` | `var(--mantine-spacing-sm)` |
|
||||
| `0.75rem` (12px) | **md** | `"md"` | `var(--mantine-spacing-md)` |
|
||||
| `1rem` (16px) | **lg** | `"lg"` | `var(--mantine-spacing-lg)` |
|
||||
| `1.5rem` (24px) | **xl** | `"xl"` | `var(--mantine-spacing-xl)` |
|
||||
| `2rem` (32px) | **xxl** | `"xxl"` | `var(--mantine-spacing-xxl)` |
|
||||
| `2.5rem` (40px) | **xxxl** | `"xxxl"` | `var(--mantine-spacing-xxxl)` |
|
||||
|
||||
Values between steps: keep the literal (e.g. `rem(10)`), don't round to a token.
|
||||
| Hardcoded value | Nearest token | Style prop | CSS variable |
|
||||
| --------------------------------- | ------------- | ------------ | ------------------------------ |
|
||||
| `0.125rem` (2px) | `2px` | `2` (number) | `2px` (keep literal, no token) |
|
||||
| `0.2rem` (3.2px), `0.25rem` (4px) | **xs** (4px) | `"xs"` | `var(--mantine-spacing-xs)` |
|
||||
| `0.5rem` (8px) | **sm** (8px) | `"sm"` | `var(--mantine-spacing-sm)` |
|
||||
| `0.75rem` (12px) | between sm/md | `rem(12)` | `0.75rem` (no exact token) |
|
||||
| `1rem` (16px) | **md** (16px) | `"md"` | `var(--mantine-spacing-md)` |
|
||||
| `1.5rem` (24px) | **lg** (24px) | `"lg"` | `var(--mantine-spacing-lg)` |
|
||||
| `2rem` (32px) | **xl** (32px) | `"xl"` | `var(--mantine-spacing-xl)` |
|
||||
|
||||
**Border radius** — snap to the nearest Mantine radius token:
|
||||
|
||||
| Hardcoded value | Nearest token | CSS variable |
|
||||
| ---------------- | ------------- | ----------------------------- |
|
||||
| `0` (0px) | — | `0` |
|
||||
| `0.125rem` (2px) | **xxxs** | `var(--mantine-radius-xxxs)` |
|
||||
| `0.25rem` (4px) | **xxs** | `var(--mantine-radius-xxs)` |
|
||||
| `0.375rem` (6px) | **xs** | `var(--mantine-radius-xs)` |
|
||||
| `0.5rem` (8px) | **sm** | `var(--mantine-radius-sm)` |
|
||||
| `0.75rem` (12px) | **md** | `var(--mantine-radius-md)` |
|
||||
| `1rem` (16px) | **lg** | `var(--mantine-radius-lg)` |
|
||||
| `1.5rem` (24px) | **xl** | `var(--mantine-radius-xl)` |
|
||||
|
||||
Fully-round pills/badges (radius ≥ half the element height): keep a literal
|
||||
such as `9999px`.
|
||||
| Hardcoded value | Nearest token | CSS variable |
|
||||
| ---------------- | ----------------------------- | -------------------------- |
|
||||
| `0.25rem` (4px) | **xs** (4px) | `var(--mantine-radius-xs)` |
|
||||
| `0.375rem` (6px) | **sm** (6px) | `var(--mantine-radius-sm)` |
|
||||
| `0.5rem` (8px) | **md** (8px) | `var(--mantine-radius-md)` |
|
||||
| `1rem`+ (16px+) | **xl** (40px) or keep literal | `var(--mantine-radius-xl)` |
|
||||
|
||||
**Font sizes** — snap to the nearest Mantine font size token:
|
||||
|
||||
|
||||
@@ -1,248 +0,0 @@
|
||||
---
|
||||
name: fe-modularization
|
||||
description: Deciding where frontend code lives — tier model, move mechanics, extension points, side effects, naming, and the traps. Use when moving code between modules, carving new modules, fixing boundary violations, adding a barrel or an endpoint, or reviewing module-shape decisions.
|
||||
---
|
||||
|
||||
# Frontend modularization
|
||||
|
||||
The module is the unit of blast radius: dependency direction, test selection,
|
||||
bundle cones, and review scope all follow module boundaries. Every decision
|
||||
here serves one goal: **make the file locations tell the truth about who owns
|
||||
what and who may depend on whom.**
|
||||
|
||||
## Hard rules (check these before anything else)
|
||||
|
||||
- NEVER add an `export … from` re-export at an old path, and NEVER `export *`
|
||||
from a barrel. Move the code, codemod every call site, delete the old path,
|
||||
all in the same PR.
|
||||
- NEVER import past a module's `index.ts` when that module has
|
||||
`enforcePublicApi: true`, and never past `metabase/ui`'s index at all. If
|
||||
the barrel is heavy, the fix is making the module side-effect free (below),
|
||||
not a deep import.
|
||||
- NEVER rewrite a consumer to a lower-level idiom to make the linter pass. If
|
||||
the fix makes the consumer worse, the plan is wrong.
|
||||
- NEVER give a new module `enforceSharedTiers: false`; new modules ship
|
||||
enforced with `enforcePublicApi: true` and an explicit `index.ts`.
|
||||
- NEVER move code without the consumer-tier check (procedure step 2).
|
||||
- ALWAYS delete a module's `enforceSharedTiers: false` line in the PR that
|
||||
takes it to zero violations.
|
||||
- ALWAYS measure `bun run module-boundaries` before and after, and report the
|
||||
numbers.
|
||||
- ALWAYS import `dayjs` from `metabase/dayjs`, routing from `metabase/router`,
|
||||
Mantine from `metabase/ui`, react-redux hooks from `metabase/redux`; the
|
||||
raw specifiers are lint-forbidden.
|
||||
|
||||
## Files that are ground truth
|
||||
|
||||
| what | where |
|
||||
|---|---|
|
||||
| module elements, tiers, `enforceSharedTiers`, `enforcePublicApi` | `frontend/lint/module-boundaries.mjs` |
|
||||
| shared sub-tiers and levels, cluster rules | `frontend/lint/shared-tiers.mjs` |
|
||||
| standalone boundaries lint (all violations, incl. grandfathered) | `bun run module-boundaries` (config `eslint.config.module-boundaries.mjs`) |
|
||||
| PR lint (only enforced modules fail CI) | `bun run lint-eslint-pure` |
|
||||
| side-effect-free directories and their exceptions | `frontend/build/shared/rspack/side-effect-free-modules.js` (`SIDE_EFFECT_FREE_PATHS`, `SIDE_EFFECT_PATHS`) |
|
||||
| the lint rules that enforce that promise | `metabase/no-module-side-effects`, `metabase/no-base-api-access` in `frontend/lint/eslint-plugin-metabase/rules/` |
|
||||
| public-api enforcement | `metabase/enforce-module-public-api` rule, driven by `getPublicApiModules()` |
|
||||
|
||||
## The model
|
||||
|
||||
Tiers: `lib < basic < shared < feature < app`. Imports point downward, never
|
||||
sideways at feature tier (features may not import features; app and EE may
|
||||
import anything). The shared tier is subdivided in `shared-tiers.mjs` into
|
||||
shared-utils (U0…), shared-platform (P0…), and shared-domain, each ordered
|
||||
into levels. A module imports only *strictly lower* levels of its sub-tier
|
||||
plus the sub-tiers below; same-level peers are deliberately forbidden, which
|
||||
is what makes cycles structurally impossible. A domain may hold two seats
|
||||
when consumers demand it: a low core and a high surface (metabase-lib below
|
||||
questions below query_builder; viz-core below visualizations).
|
||||
|
||||
Enforcement is per module: `enforceSharedTiers: false` on an element exempts
|
||||
it from the level rules (counted by `bun run module-boundaries`, invisible to
|
||||
PR lint). Modules with `enforcePublicApi: true` may only be imported from
|
||||
outside via `metabase/<module>`, and import relatively inside.
|
||||
|
||||
## Deciding where code lives (apply in order; first decisive test wins)
|
||||
|
||||
1. **Who owns the concept?** Code lives with the module that owns its
|
||||
concept, not the module that renders or calls it. A mode encoding dashboard
|
||||
click behaviour is dashboard code even if viz executes it. Metabot
|
||||
conversation state is metabot state even though the store registers it.
|
||||
2. **Consumer-tier check.** The destination must sit at or below the file's
|
||||
*lowest* consumer. A single consumer below the proposed home vetoes the
|
||||
move; report it, don't force it.
|
||||
3. **Only four legitimate fixes** for a bad edge: move the code to its
|
||||
terminal home; invert through a designed extension point (a prop, an
|
||||
injected component, a contract type owned by the socket); fix the tier
|
||||
placement when the declared level is wrong; delete a thin wrapper (a rename
|
||||
with no derivation isn't an abstraction — deleting it *is* the migration).
|
||||
4. **A bridge lives above the lower endpoint.** Code that needs both sides of
|
||||
a boundary belongs in or above the higher module.
|
||||
5. **"Genuinely shared" is proven by consumers.** A shared address needs two
|
||||
or more consumers the tier rules can't serve another way. One consumer
|
||||
pretending to be shared is a feature module at the wrong address.
|
||||
|
||||
## Extension points
|
||||
|
||||
The socket's owner defines the contract; every plug lives with its extender.
|
||||
Visualization owns the `mode` prop and click interfaces, each surface supplies
|
||||
its mode; the editor owns its extension contract, metabot supplies tiptap
|
||||
extensions; querying owns `TemplateTagsSidebarProps`, parameters implements
|
||||
it.
|
||||
|
||||
- Prefer plain props/injection at composition sites. `PLUGIN_*` registries are
|
||||
for edition gating; single-reader slots move out to their reading module;
|
||||
slots with many readers live in the module that owns the concept (every
|
||||
slot the whitelabel plugin fills lives in `whitelabel`), tier permitting.
|
||||
Keep contracts type-light so implementations can load lazily.
|
||||
- Identical injection at every callsite is acceptable until a fourth consumer
|
||||
or second slot appears; then the composition deserves its own module above
|
||||
both parts. Don't pre-build the wrapper.
|
||||
|
||||
## Store shape
|
||||
|
||||
A module that has redux state owns it:
|
||||
|
||||
- `store/` is private: reducer, plain creators/action types, and selectors,
|
||||
typed against the module's own state (`QueryBuilderStoreState`), never the
|
||||
global `State`. Thunks live in `actions/` and import `store/`, never the
|
||||
reverse.
|
||||
- The barrel is the state API: selectors for reads, creators/thunks for
|
||||
writes.
|
||||
- Store roots (`reducers-main`, `reducers-public`, the SDK store) import each
|
||||
slice from its module barrel and derive their own state type; no slice
|
||||
registry, no `declare module` merging.
|
||||
- Cross-module writes: a callback prop when a composition point exists;
|
||||
otherwise dispatch the owner's exported creator; an event only when the
|
||||
emitter must not know its listeners. Raw `dispatch({ type: "…" })` strings
|
||||
are banned. A reducer or listener matching another module's *internal*
|
||||
action is the coupling to remove.
|
||||
- Slices that mirror one endpoint get deleted (the RTK cache is the state); a
|
||||
fold over many sources with one writer (the metadata store) is kept as a
|
||||
module-private slice behind the barrel.
|
||||
|
||||
## Endpoints
|
||||
|
||||
Endpoints migrate to their owning module via `Api.injectEndpoints` from the
|
||||
module's `api/` file (settings, metabot, transforms precedents); `metabase/api`
|
||||
trends toward the client only. There is one `Api` object per backend because
|
||||
tag invalidation only works within one instance. Consumers reach endpoints
|
||||
only through the owner's exports (`useGetCardQuery`, `cardApi.endpoints.…`),
|
||||
never by name through the base object; `metabase/no-base-api-access` allows
|
||||
injection only in `metabase/api` and `**/api/**` / `**/api.ts`, and reaching by
|
||||
name only in `metabase/api` and test support (`invalidateTags` /
|
||||
`resetApiState` are fine anywhere). Each endpoint move carries its
|
||||
metadata-store hookup: the owner calls `entitiesFetched` from its own
|
||||
`onQueryStarted`.
|
||||
|
||||
## Import-time side effects
|
||||
|
||||
Directories in `SIDE_EFFECT_FREE_PATHS` are marked `sideEffects: false` for
|
||||
rspack, so importing one export from their barrel no longer drags the whole
|
||||
barrel into a bundle. The price: nothing in those directories may do work at
|
||||
import time, or production silently drops it. `metabase/no-module-side-effects`
|
||||
enforces this over every listed directory (module-scope calls, writes to
|
||||
imported objects or globals, bare imports, control flow at import; escape
|
||||
hatches are `/* #__PURE__ */` and the rule's pure-callee list). Files that
|
||||
must run at import are listed in `SIDE_EFFECT_PATHS` (files or directories).
|
||||
|
||||
- **A library the app configures gets a facade module** owning the vendor
|
||||
import, the registrations, and the types, with the raw specifier
|
||||
lint-forbidden outside it: `metabase/router`, `metabase/ui`,
|
||||
`metabase/redux`, `metabase/dayjs`. Facades are the effect, so they never
|
||||
enter `SIDE_EFFECT_FREE_PATHS`. Never re-register a vendor plugin locally
|
||||
(`dayjs.extend`, `echarts` `use`).
|
||||
- **A patch a component needs is an explicit registration module,
|
||||
bare-imported by the provider that needs it** and listed in
|
||||
`SIDE_EFFECT_PATHS` (Mantine's `Popover.Dropdown` replacement, imported by
|
||||
`ThemeProvider`).
|
||||
- **Effects with no value to import** (global CSS, polyfills, the CSP nonce,
|
||||
EE plugin bootstrap) live in the entry's side-effects file and nowhere else.
|
||||
- **Registries are filled from the composition root at boot**
|
||||
(`initializePlugins`, `registerVisualizations`), never by a module
|
||||
registering itself at import.
|
||||
- **RTK injection stays at import**; when a module with an `api/` folder is
|
||||
declared side-effect free, list `<module>/api/` in `SIDE_EFFECT_PATHS`.
|
||||
|
||||
## Naming
|
||||
|
||||
The feature keeps the plain product-surface name; a shared carve is named for
|
||||
what the shared part actually is: the concept when the concept is the shared
|
||||
thing (`settings`, `current-user`, `whitelabel`, `dayjs`), or a qualified
|
||||
capability when the plain noun is taken (`search-ui`, `metrics-ui`,
|
||||
`viz-core`, `data-studio-ui`). Never two modules distinguished only by tier
|
||||
prefix; never generic `-shared` / `-feature` suffixes.
|
||||
|
||||
## Procedure for a move
|
||||
|
||||
1. **Enumerate consumers honestly** (hook-name greps miss the last three):
|
||||
```
|
||||
grep -rn "metabase/<module>" frontend/src enterprise/frontend/src frontend/test e2e
|
||||
grep -rn "jest.mock(\"metabase/<module>" frontend enterprise/frontend
|
||||
grep -rn "addMatcher(.*<endpoint>" frontend/src enterprise/frontend/src
|
||||
```
|
||||
plus sibling files importing the moved file relatively. Classify each by
|
||||
module and tier from `module-boundaries.mjs` / `shared-tiers.mjs`.
|
||||
2. **Consumer-tier check**: destination at or below the lowest consumer. If
|
||||
not, stop and report the blocking consumer.
|
||||
3. **Measure before**: `bun run module-boundaries 2>&1 | tail -1` on the base
|
||||
(write the number down).
|
||||
4. **Move**: `git mv`; codemod every importer to `metabase/<module>` (outside)
|
||||
or a relative path (inside); add named exports to the destination barrel
|
||||
for every symbol an outside consumer needs (each must have a real
|
||||
consumer). Delete the old path. If the module didn't exist: add its element
|
||||
in `module-boundaries.mjs` (ordered before any element it's carved from,
|
||||
first match wins), its level in `shared-tiers.mjs`, `enforcePublicApi:
|
||||
true`, an `index.ts`. If the move empties a module's violations, delete
|
||||
its `enforceSharedTiers: false` line. If the module has an `api/` folder
|
||||
and is side-effect free, list `<module>/api/` in `SIDE_EFFECT_PATHS`.
|
||||
5. **Verify** (all of these, report the numbers):
|
||||
```
|
||||
grep -rn "<old path>" frontend enterprise/frontend e2e # must be empty
|
||||
bunx eslint --no-warn-ignored <touched files>
|
||||
bun run lint-eslint-pure # enforce-module-public-api included
|
||||
bun run type-check-pure
|
||||
bun run test-unit-keep-cljs <touched folders' specs>
|
||||
bun run module-boundaries 2>&1 | tail -1 # after; no violation may name the new files
|
||||
bun run build-release:static-viz # only if viz/static-viz/ui/dayjs touched; hard 3.5 MiB budget
|
||||
```
|
||||
6. **Report**: count before/after (state count-neutral explicitly), the edges
|
||||
removed by file, flags/rules deleted, barrel exports added, anything you
|
||||
skipped and why, and any consumer you found that vetoed part of the move.
|
||||
|
||||
## Traps (each has bitten at least once)
|
||||
|
||||
- **Late binding.** A bare function reference in a `createSelector` input
|
||||
array, a component identity captured for comparison, a `PLUGIN_*` object
|
||||
read at module scope: all capture at import time and dodge jest spies, EE
|
||||
overrides, and lazy loading. Read at call time (`(_state) => fn()`), or
|
||||
inject.
|
||||
- **Plans go stale.** Verify every recipe against current code before
|
||||
executing; things get merged, renamed, or turn out never to have existed.
|
||||
Skip-and-report beats force.
|
||||
- **The linter's blind spots**: `export … from` re-exports, bare-specifier
|
||||
packages (custom-viz), pre-seeded caches that never register tags. A grep
|
||||
for the old path is the stronger claim than a green lint run.
|
||||
- **Path-keyed baselines reset on rename**: a generated per-file ledger
|
||||
silently loses a file's history when it moves.
|
||||
- **A heavy barrel is a bundle hazard until its module is side-effect free.**
|
||||
One constant imported from the `metabase/ui` barrel pulled Mantine into the
|
||||
static-viz bundle and broke its budget; the fix was `SIDE_EFFECT_FREE_PATHS`,
|
||||
never a deep import.
|
||||
- **Type augmentation is program-wide.** `declare module "dayjs"` (and
|
||||
similar) makes plugin types available to any file in the same tsconfig, so
|
||||
a passing type-check never proves the setup ran; only importing the facade
|
||||
does.
|
||||
- **Environment before diagnosis**: after a dependency-touching merge,
|
||||
`bun install` before declaring a failure real; worktrees share staleness
|
||||
through the node_modules symlink, and a stale `target/cljs_dev` makes
|
||||
`type-check-pure` fail on `metabase-lib/metric/core.ts` (rebuild cljs in the
|
||||
worktree).
|
||||
|
||||
## PR shape
|
||||
|
||||
One logical change per PR; the body names the specific edges removed (files,
|
||||
imports, before/after count) and gives each move's ownership reason in plain
|
||||
language a reviewer outside the effort can follow, without campaign labels.
|
||||
Stack only for semantic dependency or same-hot-file serialisation, and say
|
||||
so. Consider codeowner surface when chunking. Open as a draft until measured
|
||||
and green.
|
||||
@@ -205,7 +205,7 @@ Per `docs/developers-guide/frontend.md` ("Colors"): every color must be a **sema
|
||||
|
||||
The same discipline applies to **dimensional** values — radius, spacing/padding/gap, elevation/shadow, and typography (font size, line-height, weight). Prefer the codebase's existing scale variable over a literal: `--mantine-radius-*`, `--mantine-spacing-*`, `--mantine-shadow-*`, and the Mantine font tokens.
|
||||
|
||||
But Figma's scales don't always line up with the codebase's. When a Figma value **has no matching step** in the corresponding scale — e.g. Figma `radius/md = 16px` while `--mantine-radius-md` is `0.75rem`, or a Figma spacing step that has no Mantine equivalent — **don't silently bake in a literal.** Flag it to the user the same way you flag primitive colors: name the property, the Figma value, and the nearest codebase token, so it can be resolved with design / the scale owners (align the value, or add a scale step). Only fall back to a literal if the user confirms it's an accepted one-off, and leave a short comment in the code explaining why. The codebase scales are defined in `frontend/src/metabase/ui/theme.ts` (`SPACING_SCALE`, `RADIUS_SCALE`, `LIGHT_SHADOWS`/`DARK_SHADOWS`).
|
||||
But Figma's scales don't always line up with the codebase's. When a Figma value **has no matching step** in the corresponding scale — e.g. Figma `radius/md = 12px` while `--mantine-radius-md` is `8px`, or a Figma `xxxs` spacing step with no Mantine equivalent — **don't silently bake in a literal.** Flag it to the user the same way you flag primitive colors: name the property, the Figma value, and the nearest codebase token, so it can be resolved with design / the scale owners (align the value, or add a scale step). Only fall back to a literal if the user confirms it's an accepted one-off, and leave a short comment in the code explaining why.
|
||||
|
||||
### Mantine implementation notes (reference)
|
||||
|
||||
|
||||
@@ -13,7 +13,7 @@ allowed-tools: Read, Grep, Bash, Glob, Skill
|
||||
|
||||
**Primary standard: the [`typescript-write`](../typescript-write/SKILL.md) skill.** Load it first — it defines the authoring rules this review enforces, alongside `frontend/CLAUDE.md` and `docs/developers-guide/frontend.md`.
|
||||
|
||||
Adherence to `typescript-write` is the **highest-priority** review dimension: rank any violation of its provisions above all other findings. Treat its **no-`any` hard rule** (no explicit *or* implicit `any` in new code) as **blocking**. Use TypeScript LSP tools to inspect inferred types when available; otherwise rely on type-checking and linting.
|
||||
Adherence to `typescript-write` is the **highest-priority** review dimension: rank any violation of its provisions above all other findings. Treat its **no-`any` hard rule** (no explicit *or* implicit `any` in new code) as **blocking**, and verify it with the LSP rather than by eye.
|
||||
|
||||
Review in this priority order:
|
||||
|
||||
|
||||
@@ -13,7 +13,7 @@ description: Write TypeScript and JavaScript code following Metabase coding stan
|
||||
|
||||
- **New code must not introduce `any`, explicit or implicit.** No `any` annotations, no `as any` / `as unknown as`, no untyped parameters or returns that infer `any`, no implicitly-`any` destructures or array/object literals.
|
||||
- **Untyped third-party / boundary values** must be typed at the boundary (a declared type, `unknown` + type guard, or a small typed wrapper) — never let `any` propagate inward.
|
||||
- **Mandatory type verification.** Before finishing a TS/TSX change, run `bun run type-check-pure`. If TypeScript LSP tools are available, also inspect changed symbols with hover and go-to-definition; otherwise skip LSP check.
|
||||
- **Mandatory LSP verification.** After writing or editing any TS/TSX, inspect the changed symbols with the TypeScript Language Server (hover to read inferred types; `goToDefinition` to confirm sources) and run the project type-check.
|
||||
|
||||
## Type tightening
|
||||
|
||||
@@ -26,7 +26,6 @@ description: Write TypeScript and JavaScript code following Metabase coding stan
|
||||
- **Avoid non-null assertions (`!`)**. Prefer a guard, early return, or `?.`. Use `!` only when non-nullness is provably true and localized, with a comment.
|
||||
- **No redundant runtime coercion** — don't wrap already-typed values in `Number()` / `String()` / `Boolean()`.
|
||||
- **Type guards belong in `frontend/src/metabase-types/guards/`**. Do not redefine them locally.
|
||||
- **A cast you can't avoid needs a real justification comment.** The `metabase/no-unjustified-type-casts` rule accepts any preceding comment — state the actual reason the cast is safe. NEVER write the legacy `// Unjustified type cast. FIXME` placeholder; it exists only on casts that predated the rule, and copying it sneaks an unjustified cast past the linter. If you can't articulate why the cast is correct, the cast is wrong — fix the types.
|
||||
|
||||
## Type modeling
|
||||
|
||||
@@ -75,6 +74,10 @@ description: Write TypeScript and JavaScript code following Metabase coding stan
|
||||
- **No comments by default**. Well-named identifiers carry the `what`.
|
||||
- **Comments should be concise**. Add a short, concise comment only when the `why` is non-obvious: a workaround, a hidden invariant, a subtle ordering constraint, a clever reduction. Never document the actual implementation, focus on the intent and the why.
|
||||
|
||||
## TypeScript Migration
|
||||
|
||||
**When touching existing JavaScript files, propose to convert them to TypeScript first**. Create a separate PR for the conversion, then implement the changes.
|
||||
|
||||
## Verify before done
|
||||
|
||||
- **Run the project type-check** when finished (see the shared TypeScript commands above).
|
||||
|
||||
@@ -7,6 +7,7 @@
|
||||
|
||||
:aliased-namespace-symbol {:level :warning}
|
||||
:case-symbol-test {:level :warning}
|
||||
:condition-always-true {:level :warning}
|
||||
:def-fn {:level :warning}
|
||||
:dynamic-var-not-earmuffed {:level :warning}
|
||||
:equals-expected-position {:level :warning, :only-in-test-assertion true}
|
||||
@@ -49,11 +50,9 @@
|
||||
;; disabled linters
|
||||
;;
|
||||
|
||||
:constant-condition {:level :off} ; TODO (sashko): too confused by defenterprises, some hook/config is needed
|
||||
:redundant-ignore {:level :off} ; lots of false positives, see https://github.com/clj-kondo/clj-kondo/issues/2414
|
||||
:unexpected-recur {:level :off} ; TODO (cam): I think we just need to tell it how to handle MBQL match and we can enable this?
|
||||
:redundant-primitive-coercion {:level :off} ; there are many defensive boolean coercions, kondo looks too much into it
|
||||
:type-mismatch {:level :off} ; Still quite raw, can sometimes give useful insights, but sometimes is just confused
|
||||
|
||||
;;
|
||||
;; globally-enabled custom linters
|
||||
@@ -180,8 +179,6 @@
|
||||
clojure.core.async/take!
|
||||
clojure.core.async/to-chan!
|
||||
clojure.core.async/to-chan!!
|
||||
;; only writes .clj-kondo/ratchets.edn, from a :once fixture that finishes before the parallel tests
|
||||
dev.kondo-ratchet/fix!
|
||||
metabase.ai-tracing.core/record!
|
||||
metabase.api.macros.defendpoint.tools-manifest/assert-optional-fields-nullable!
|
||||
metabase.channel.core/send!
|
||||
@@ -191,8 +188,6 @@
|
||||
metabase.channel.render.pdf.font/load-fonts!
|
||||
metabase.warehouses.models.database/assert-not-h2!
|
||||
metabase.driver.sql-jdbc.execute/execute-prepared-statement!
|
||||
;; only inspects a details map and throws; the `!` is for the throw, not for a mutation
|
||||
metabase.driver/validate-db-details!
|
||||
metabase.mcp.usage/record-mcp-session!
|
||||
metabase.mcp.usage/record-mcp-session-end!
|
||||
metabase.mcp.usage/record-mcp-tool-call!
|
||||
@@ -303,10 +298,7 @@
|
||||
metabase.query-processor.util/normalize-token {:namespaces ["metabase.*"]}}}
|
||||
|
||||
:discouraged-var
|
||||
{metabase-enterprise.entity-retrieval.core/search-unfiltered {:message "Permission bypass risk: library-index hits are unfiltered — resolve each entity and drop what the current user can't read"}
|
||||
metabase.entity-retrieval.core/search-unfiltered {:message "Permission bypass risk: library-index hits are unfiltered — resolve each entity and drop what the current user can't read"}
|
||||
metabase.entity-retrieval.mirror/search-unfiltered {:message "Permission bypass risk: library-index hits are unfiltered — resolve each entity and drop what the current user can't read"}
|
||||
clojure.core/print {:message "Use metabase.util.log instead of clojure.core/print"}
|
||||
{clojure.core/print {:message "Use metabase.util.log instead of clojure.core/print"}
|
||||
clojure.core/println {:message "Use metabase.util.log instead of clojure.core/println"}
|
||||
clojure.core/printf {:message "Use metabase.util.log instead of clojure.core/printf"}
|
||||
clojure.core/prn {:message "Use metabase.util.log instead of clojure.core/prn"}
|
||||
@@ -405,7 +397,6 @@
|
||||
metabase.channel.render.core channel.render
|
||||
metabase.cloud-migration.core cloud-migration
|
||||
metabase.eid-translation.core eid-translation
|
||||
metabase.llm.test-util llm.tu
|
||||
metabase.login-history.core login-history
|
||||
metabase.model-persistence.core model-persistence
|
||||
metabase.notification.core notification
|
||||
@@ -474,8 +465,6 @@
|
||||
metabase.actions.test-util/with-actions clojure.core/let
|
||||
metabase.transforms.test-util/with-transform-cleanup! clojure.core/let
|
||||
metabase.api.common/let-404 clojure.core/let
|
||||
metabase.app-db.connection/with-unshared-connection clojure.core/fn
|
||||
metabase.app-db.core/with-unshared-connection clojure.core/fn
|
||||
metabase.app-db.custom-migrations.util/with-temp-scheduler! clojure.core/fn
|
||||
metabase.app-db.custom-migrations/define-migration clj-kondo.lint-as/def-catch-all
|
||||
metabase.app-db.custom-migrations/define-reversible-migration clj-kondo.lint-as/def-catch-all
|
||||
@@ -498,7 +487,6 @@
|
||||
metabase.notification.test-util/with-card-notification clojure.core/let
|
||||
metabase.notification.test-util/with-system-event-notification! clojure.core/let
|
||||
metabase.notification.test-util/with-temp-notification clojure.core/let
|
||||
metabase.oauth-server.test-util/with-oauth-client clojure.core/fn
|
||||
metabase.osi.ai-context.api-test/with-test-entry clojure.core/let
|
||||
metabase.premium-features.core/defenterprise-schema clj-kondo.lint-as/def-catch-all
|
||||
metabase.premium-features.defenterprise/defenterprise-schema clj-kondo.lint-as/def-catch-all
|
||||
@@ -508,9 +496,7 @@
|
||||
metabase.query-processor.pipeline-queries-test/mbql5-query clojure.core/->
|
||||
metabase.query-processor.pipeline-queries-test/run-mbql5-query clojure.core/->
|
||||
metabase.query-processor.error-type/deferror clojure.core/def
|
||||
metabase.query-processor.core/with-reducible-deserialized-results clojure.core/let
|
||||
metabase.query-processor.middleware.cache.impl/with-reducible-deserialized-results clojure.core/let
|
||||
metabase.query-processor.result-serialization/with-reducible-deserialized-results clojure.core/let
|
||||
metabase.query-processor.middleware.process-userland-query-test/with-query-execution! clojure.core/let
|
||||
metabase.query-processor.setup/with-qp-setup clojure.core/let
|
||||
metabase.search.api-test/do-test-users clojure.core/let
|
||||
@@ -527,6 +513,7 @@
|
||||
metabase.test.data.interface/defdataset clojure.core/def
|
||||
metabase.test.data.interface/defdataset-edn clojure.core/def
|
||||
metabase.test.embedded-postgres.core/with-system clojure.core/let
|
||||
metabase.test.gentest/defgentest clojure.test/deftest
|
||||
metabase.test/defdataset clojure.core/def
|
||||
metabase.test/let-url clojure.core/let
|
||||
metabase.test/with-actions clojure.core/let
|
||||
@@ -537,7 +524,6 @@
|
||||
metabase.test/with-temp-empty-app-db clojure.core/let
|
||||
metabase.test/with-temp-file clojure.core/let
|
||||
metabase.test/with-user-in-groups clojure.core/let
|
||||
metabase.transforms-rest.api.transform-runs-test/with-runs-fixture! clojure.core/fn
|
||||
metabase.upload.impl-test/with-upload-table! clojure.core/let
|
||||
metabase.util/for-map clojure.core/for
|
||||
metabase.util.files/with-open-path-to-resource clojure.core/let
|
||||
@@ -636,7 +622,6 @@
|
||||
metabase.lib.test-util.macros/mbql-query hooks.metabase.test.data/mbql-query
|
||||
metabase.lib.test-util.macros/mbql-5-query hooks.metabase.test.data/mbql-query
|
||||
metabase.lib.test-util.macros/with-testing-against-standard-queries hooks.metabase.lib.test-util.macros/with-testing-against-standard-queries
|
||||
metabase.metrics.api-dimension-test/with-seeded-metric hooks.common/with-one-binding
|
||||
metabase.model-persistence.api-test/with-persistence-setup! hooks.common/with-one-top-level-binding
|
||||
metabase.model-persistence.api-test/with-setup! hooks.common/with-one-top-level-binding
|
||||
metabase.model-persistence.test-util/with-persistence-enabled! hooks.metabase.model-persistence.test-util/with-persistence-enabled!
|
||||
@@ -685,6 +670,7 @@
|
||||
metabase.test.data/dataset hooks.metabase.test.data/dataset
|
||||
metabase.test.data/mbql-query hooks.metabase.test.data/mbql-query
|
||||
metabase.test.data/run-mbql-query hooks.metabase.test.data/mbql-query
|
||||
metabase.test.gentest/iterate hooks.metabase.test.gentest/iterate*
|
||||
metabase.test.util.async/with-open-channels hooks.common/let-with-optional-value-for-last-binding
|
||||
metabase.test.util.generators.jvm/with-random-cards hooks.metabase.test.util.generators.jvm/with-random-cards
|
||||
metabase.test.util.log/with-log-level hooks.common/with-ignored-first-arg
|
||||
@@ -750,7 +736,6 @@
|
||||
clojurewerkz.quartzite.schedule.simple/schedule macros.quartz/simple-schedule
|
||||
clojurewerkz.quartzite.triggers/build macros.quartz/build-trigger
|
||||
metabase-enterprise.database-routing.e2e-test/with-temp-dbs! macros.metabase-enterprise.database-routing.e2e-test/with-temp-dbs!
|
||||
metabase-enterprise.database-routing.e2e-test/with-routing-setup! macros.metabase-enterprise.database-routing.e2e-test/with-routing-setup!
|
||||
metabase-enterprise.impersonation.util-test/with-impersonations! macros.metabase-enterprise.impersonation.util-test/with-impersonations!
|
||||
metabase-enterprise.query-reference-validation.api-test/with-test-setup! macros.metabase-enterprise.query-reference-validation.api-test/with-test-setup!
|
||||
metabase-enterprise.sandbox.test-util/with-gtaps! macros.metabase-enterprise.sandbox.test-util/with-gtaps!
|
||||
@@ -803,7 +788,7 @@
|
||||
:name test-namespaces}
|
||||
{:pattern "^(?!.*-test(?:\\..*)?$)(?!.*test-util)(?!^metabase\\.test.*).*$"
|
||||
:name source-namespaces}
|
||||
{:pattern "(?:(?:^metabase\\.cmd.*)|(?:^build.*)|(?:^metabuild-common.*)|(?:^release.*)|(?:^i18n.*)|(?:^lint-migrations-file$))|(?:^mage.*)|(?:^load-namespaces.*)"
|
||||
{:pattern "(?:(?:^metabase\\.cmd.*)|(?:^build.*)|(?:^metabuild-common.*)|(?:^release.*)|(?:^i18n.*)|(?:^lint-migrations-file$))|(?:^mage.*)"
|
||||
:name printable-namespaces}
|
||||
{:pattern "^metabase\\.(?:(?:lib)|(?:legacy-mbql))\\..*"
|
||||
:name metabase-lib}
|
||||
@@ -864,11 +849,6 @@
|
||||
metabase.util.time-test
|
||||
{:linters {:metabase/prefer-with-dynamic-fn-redefs {:level :off}}}
|
||||
|
||||
;; classloader sits below `util` in the module graph, so it can't use `metabase.util.log` without creating
|
||||
;; a cycle; it gets to use tools.logging directly.
|
||||
metabase.classloader.impl
|
||||
{:linters {:discouraged-namespace {clojure.tools.logging {:level :off}}}}
|
||||
|
||||
source-namespaces
|
||||
{:linters
|
||||
{:metabase/discourage-millis-duration {:level :warning}
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -58,7 +58,6 @@
|
||||
metabase.app-db.core/finish-db-setup!
|
||||
metabase.app-db.core/migrate!
|
||||
metabase.app-db.core/setup-db!
|
||||
metabase.app-db.encryption-test-util/do-with-encrypted-app-db!
|
||||
metabase.app-db.schema-migrations-test.impl/run-migrations-in-range!
|
||||
metabase.app-db.setup/migrate!
|
||||
metabase.app-db.setup/setup-db!
|
||||
|
||||
@@ -3,8 +3,3 @@
|
||||
(defmacro with-temp-dbs! [bindings & body]
|
||||
`(let [~@(interleave bindings (repeat nil))]
|
||||
~@body))
|
||||
|
||||
(defmacro with-routing-setup! [[router-binding destinations] & body]
|
||||
`(let [~router-binding nil
|
||||
~@(interleave (map first destinations) (repeat nil))]
|
||||
~@body))
|
||||
|
||||
@@ -1,80 +0,0 @@
|
||||
;; Budgets for kondo suppressions: inline `:clj-kondo/ignore` forms per linter (:ignore-counts), and
|
||||
;; config-level waivers in .clj-kondo/config.edn (:config-counts -- :off switches and :exclude entries).
|
||||
;; Each :ignore-counts value is a non-negative integer budget, or :unlimited for no ceiling.
|
||||
;; Checks fail when a count exceeds its numeric budget; unused budget is allowed.
|
||||
;; Any ignore outside :comment-exempt needs an explanatory comment directly above or trailing on its line.
|
||||
;; `./bin/mage kondo-ratchets-shrink` lowers budgets unless `--seed` explicitly adds or raises one.
|
||||
;; The workflow runs it on master, so feature branches do not need to record reductions.
|
||||
;; Raising or adding a budget (`--seed` for inline ignores, a manual edit for config) or widening the
|
||||
;; exemptions must be explained in the PR.
|
||||
;; :all is the vector-less ignore form, which suppresses every linter on the next form.
|
||||
{:ignore-counts {:aliased-namespace-symbol 3
|
||||
:case-symbol-test 1
|
||||
:clojure-lsp/unused-public-var 9
|
||||
:consistent-alias 2
|
||||
:def-fn 1
|
||||
:deprecated-namespace 95
|
||||
:deprecated-var 86
|
||||
:discouraged-java-method 4
|
||||
:discouraged-namespace 71
|
||||
:discouraged-var 130
|
||||
:duplicate-require 1
|
||||
:equals-true 1
|
||||
:main-without-gen-class 1
|
||||
:metabase/defsetting-namespace 2
|
||||
:metabase/disallow-hardcoded-driver-names-in-tests 34
|
||||
:metabase/discourage-millis-duration 1
|
||||
:metabase/i-like-making-cams-eyes-bleed-with-horrifically-long-tests 13
|
||||
:metabase/modules 23
|
||||
:metabase/namespace-name 4
|
||||
:metabase/prefer-with-dynamic-fn-redefs 28
|
||||
:metabase/test-helpers-use-non-thread-safe-functions 18
|
||||
:metabase/validate-defendpoint-has-response-schema 439
|
||||
:metabase/validate-defendpoint-query-params-use-kebab-case 50
|
||||
:metabase/validate-defendpoint-route-uses-kebab-case 44
|
||||
:metabase/validate-deftest 10
|
||||
:missing-docstring 12
|
||||
:missing-protocol-method 3
|
||||
:non-arg-vec-return-type-hint 1
|
||||
:reduce-without-init 3
|
||||
:redundant-do 1
|
||||
:syntax 2
|
||||
:type-mismatch 8
|
||||
:uninitialized-var 1
|
||||
:unquote-not-syntax-quoted 1
|
||||
:unresolved-namespace 3
|
||||
:unresolved-symbol 11
|
||||
:unsorted-required-namespaces :unlimited
|
||||
:unused-alias 2
|
||||
:unused-binding 9
|
||||
:unused-excluded-var 2
|
||||
:unused-namespace 2
|
||||
:unused-private-var 8
|
||||
:unused-value 1}
|
||||
:config-counts {:constant-condition 1
|
||||
:deprecated-namespace 7
|
||||
:deprecated-var 16
|
||||
:discouraged-namespace 14
|
||||
:discouraged-var 14
|
||||
:inline-def 1
|
||||
:main-without-gen-class 1
|
||||
:metabase/defsetting-must-specify-export 1
|
||||
:metabase/defsetting-namespace 1
|
||||
:metabase/prefer-with-dynamic-fn-redefs 3
|
||||
:missing-docstring 1
|
||||
:private-call 1
|
||||
:redundant-ignore 1
|
||||
:redundant-primitive-coercion 1
|
||||
:refer-all 1
|
||||
:type-mismatch 1
|
||||
:unexpected-recur 1
|
||||
:unresolved-symbol 19
|
||||
:unresolved-var 1
|
||||
:unused-referred-var 4}
|
||||
:comment-exempt #{:deprecated-namespace
|
||||
:discouraged-namespace
|
||||
:discouraged-var
|
||||
:metabase/modules
|
||||
:metabase/validate-defendpoint-has-response-schema
|
||||
:metabase/validate-defendpoint-query-params-use-kebab-case
|
||||
:metabase/validate-defendpoint-route-uses-kebab-case}}
|
||||
@@ -71,24 +71,6 @@
|
||||
(some-> (ns-form-node->ns-symb-node ns-form-node)
|
||||
hooks/sexpr))
|
||||
|
||||
(defn- lint-test-parallel-metadata [ns-form-node]
|
||||
(when-let [ns-symb-node (ns-form-node->ns-symb-node ns-form-node)]
|
||||
(let [ns-metadata (meta (hooks/sexpr ns-symb-node))]
|
||||
(cond
|
||||
(:synchronous ns-metadata)
|
||||
(hooks/reg-finding!
|
||||
(assoc (meta ns-symb-node)
|
||||
:message (str "Use `^:synchronized` to mark this namespace explicitly non-parallel; "
|
||||
"`^:synchronous` is ignored by Hawk. [:metabase/validate-deftest]")
|
||||
:type :metabase/validate-deftest))
|
||||
|
||||
(and (contains? ns-metadata :parallel)
|
||||
(false? (:parallel ns-metadata)))
|
||||
(hooks/reg-finding!
|
||||
(assoc (meta ns-symb-node)
|
||||
:message "Use `^:synchronized` instead of `^{:parallel false}`. [:metabase/validate-deftest]"
|
||||
:type :metabase/validate-deftest))))))
|
||||
|
||||
(defn- lint-modules [ns-form-node config]
|
||||
(let [ns-symb (ns-form-node->ns-symb ns-form-node)]
|
||||
(when-not (modules/ignored-namespace? config ns-symb)
|
||||
@@ -158,7 +140,6 @@
|
||||
lint-requires-on-new-lines
|
||||
(lint-modules (modules/config x))
|
||||
lint-namespace-name
|
||||
lint-test-parallel-metadata
|
||||
lint-jsqlparser-imports)
|
||||
x)
|
||||
|
||||
|
||||
@@ -20,23 +20,6 @@
|
||||
:message error
|
||||
:type :metabase/modules))))))))
|
||||
|
||||
(defn- lint-dynamic-require* [node current-ns config]
|
||||
(when (and (not (modules/ignored-namespace? config current-ns))
|
||||
(modules/module current-ns))
|
||||
(hooks/reg-finding! (assoc (meta node)
|
||||
:message "Module dependency cannot be statically determined from a dynamic require"
|
||||
:type :metabase/modules))))
|
||||
|
||||
(defn- quoted-require? [node]
|
||||
(or (= (hooks/tag node) :quote)
|
||||
(and (= (hooks/tag node) :list)
|
||||
(let [[x] (:children node)]
|
||||
(and (hooks/token-node? x)
|
||||
(= (hooks/sexpr x) 'quote))))))
|
||||
|
||||
(defn- keyword-node? [node]
|
||||
(keyword? (hooks/sexpr node)))
|
||||
|
||||
(defn- unwrap-require [node]
|
||||
(case (hooks/tag node)
|
||||
;; some.namespace of some.namespace/some-var
|
||||
@@ -59,11 +42,7 @@
|
||||
|
||||
;; something like [some.namespace :as whatever]
|
||||
:vector
|
||||
(recur (first (:children node)))
|
||||
|
||||
;; anything else -- e.g. a runtime expression like `(requiring-resolve (:failing-test-var opts))` --
|
||||
;; has no statically-known namespace
|
||||
nil))
|
||||
(recur (first (:children node)))))
|
||||
|
||||
(comment
|
||||
(unwrap-require (-> "(requiring-resolve 'metabase.notification.payload.execute/execute-dashboard)"
|
||||
@@ -72,24 +51,20 @@
|
||||
second)))
|
||||
|
||||
(defn- lint-require* [node current-ns config]
|
||||
(let [[_require & args] (:children node)
|
||||
;; Keywords such as `:reload` and `:reload-all` are options, not namespace dependencies.
|
||||
namespace-nodes (remove keyword-node? args)
|
||||
static-nodes (filter quoted-require? namespace-nodes)
|
||||
required-namespaces (->> (keep unwrap-require static-nodes)
|
||||
(filter simple-symbol?))]
|
||||
(lint* node current-ns config required-namespaces)
|
||||
(when (or (not= (count static-nodes) (count namespace-nodes))
|
||||
(not= (count required-namespaces) (count static-nodes)))
|
||||
(lint-dynamic-require* node current-ns config))))
|
||||
(let [[_require & args] (:children node)
|
||||
required-namespaces (->> (keep unwrap-require args)
|
||||
;; `require` runs through `cc/load-libs` which allows
|
||||
;; #{:as :reload :reload-all :require :use :verbose :refer :as-alias}
|
||||
;; as args. If you use a REPL tool that applies clj-kondo linting (e.g., Clojure MCP)
|
||||
;; you may wish to `:reload` or `:reload-all` a namespace. So we shouldn't assume all
|
||||
;; args are keywords:
|
||||
(remove keyword?))]
|
||||
(lint* node current-ns config required-namespaces)))
|
||||
|
||||
(defn- lint-requiring-resolve* [node current-ns config]
|
||||
(let [[_requiring-resolve symb-node] (:children node)
|
||||
required-namespace (unwrap-require symb-node)]
|
||||
(if (and (quoted-require? symb-node)
|
||||
(simple-symbol? required-namespace))
|
||||
(lint* node current-ns config [required-namespace])
|
||||
(lint-dynamic-require* node current-ns config))))
|
||||
(lint* node current-ns config [required-namespace])))
|
||||
|
||||
(defn lint-require [x]
|
||||
(lint-require* (:node x) (:ns x) (modules/config x))
|
||||
|
||||
@@ -28,11 +28,11 @@
|
||||
(walk form)))
|
||||
|
||||
(defn- deftest-check-parallel
|
||||
"1. Check if test is marked `^:parallel` / `^:synchronized` correctly.
|
||||
"1. Check if test is marked ^:parallel / ^:synchronized correctly.
|
||||
2. Make sure disallowed forms are not used in ^:parallel tests.
|
||||
3. Disallow `^:parallel` tests in namespaces marked `^:synchronized` --
|
||||
the ns marker is the author's blanket promise that nothing inside runs
|
||||
in parallel, so an individual `^:parallel` deftest contradicts it."
|
||||
3. Disallow ^:parallel tests in namespaces marked ^:synchronous -- the
|
||||
ns marker is the author's blanket promise that nothing inside runs
|
||||
in parallel, so an individual ^:parallel deftest contradicts it."
|
||||
[{[_ test-name & body] :children, :as _node} ns-sym config]
|
||||
(let [test-metadata (:meta test-name)
|
||||
metadata-sexprs (map hooks/sexpr test-metadata)
|
||||
@@ -44,48 +44,28 @@
|
||||
(completing merge)
|
||||
{}
|
||||
metadata-sexprs)
|
||||
parallel? (true? (:parallel combined-metadata))
|
||||
synchronized? (true? (:synchronized combined-metadata))
|
||||
explicitly-nonparallel? (or synchronized?
|
||||
(and (contains? combined-metadata :parallel)
|
||||
(false? (:parallel combined-metadata))))
|
||||
ns-metadata (meta ns-sym)
|
||||
ns-explicitly-nonparallel? (or (true? (:synchronized ns-metadata))
|
||||
(and (contains? ns-metadata :parallel)
|
||||
(false? (:parallel ns-metadata))))]
|
||||
(when (and (contains? combined-metadata :parallel)
|
||||
(false? (:parallel combined-metadata)))
|
||||
(hooks/reg-finding! (assoc (meta test-name)
|
||||
:message (str "Use `^:synchronized` instead of `^{:parallel false}`. "
|
||||
"[:metabase/validate-deftest]")
|
||||
:type :metabase/validate-deftest)))
|
||||
(when (:sequential combined-metadata)
|
||||
(hooks/reg-finding! (assoc (meta test-name)
|
||||
:message (str "Use `^:synchronized` to mark this test explicitly non-parallel; "
|
||||
"`^:sequential` is ignored by Hawk. "
|
||||
"[:metabase/validate-deftest]")
|
||||
:type :metabase/validate-deftest)))
|
||||
parallel? (:parallel combined-metadata)
|
||||
synchronized? (:synchronized combined-metadata)
|
||||
ns-synchronous? (boolean (:synchronous (meta ns-sym)))]
|
||||
(when (and parallel? synchronized?)
|
||||
(hooks/reg-finding! (assoc (meta test-name)
|
||||
:message "Test should not be marked both `^:parallel` and `^:synchronized`"
|
||||
:message "Test should not be marked both ^:parallel and ^:synchronized"
|
||||
:type :metabase/validate-deftest)))
|
||||
(when (and parallel? ns-explicitly-nonparallel?)
|
||||
(when (and parallel? ns-synchronous?)
|
||||
(hooks/reg-finding!
|
||||
(assoc (meta test-name)
|
||||
:message (str "Test should not be marked `^:parallel` in a namespace marked "
|
||||
"`^:synchronized` -- "
|
||||
:message (str "Test should not be marked ^:parallel in a namespace marked ^:synchronous -- "
|
||||
"the ns marker promises no parallel execution. "
|
||||
"Drop `^:parallel` from the deftest, or drop `^:synchronized` from the ns. "
|
||||
"Drop ^:parallel from the deftest, or drop ^:synchronous from the ns. "
|
||||
"[:metabase/validate-deftest]")
|
||||
:type :metabase/validate-deftest)))
|
||||
;; Only when the custom `:metabase/deftest-not-marked-parallel-or-synchronized` is enabled:
|
||||
;; complain if tests are not explicitly marked `^:parallel` or `^:synchronized`. This is
|
||||
;; mostly to encourage people to mark everything `^:parallel` in places like `metabase.lib`
|
||||
;; tests unless there is a really good reason not to.
|
||||
(when-not (or parallel? explicitly-nonparallel?)
|
||||
;; only when the custom `:metabase/deftest-not-marked-parallel-or-synchronized` is enabled: complain if tests are
|
||||
;; not explicitly marked `^:parallel` or `^:synchronized`. This is mostly to encourage people to mark everything
|
||||
;; `^:parallel` in places like `metabase.lib` tests unless there is a really good reason not to.
|
||||
(when-not (or parallel? synchronized?)
|
||||
(hooks/reg-finding!
|
||||
(assoc (meta test-name)
|
||||
:message "Test should be marked either `^:parallel` or `^:synchronized`"
|
||||
:message "Test should be marked either ^:parallel or ^:synchronized"
|
||||
:type :metabase/deftest-not-marked-parallel-or-synchronized)))
|
||||
(when parallel?
|
||||
(doseq [form body]
|
||||
@@ -171,11 +151,8 @@
|
||||
(str/starts-with? (name ns-symb) prefix))
|
||||
["build-drivers."
|
||||
"build."
|
||||
"hooks." ; clj-kondo hook tests, outside the module system
|
||||
"i18n." ; bin/i18n
|
||||
"lint-migrations-file-test"
|
||||
"load-namespaces." ; bin/load-namespaces
|
||||
"mage."
|
||||
"main-test" ; bin/release-list
|
||||
"metabase.deps-edn-test"
|
||||
"metabase.driver."
|
||||
@@ -254,20 +231,18 @@
|
||||
|
||||
(defn use-fixtures
|
||||
"Flag `:parallel/unsafe` forms inside fixtures. Skipped when the surrounding
|
||||
namespace is marked `^:synchronized` -- the ns marker is the author's
|
||||
namespace is marked `^:synchronous` -- the ns marker is the author's
|
||||
explicit opt-in to a single-threaded test ns where destructive setup is
|
||||
safe. Without that opt-in, kondo can't know whether sibling deftests will
|
||||
be `^:parallel`, so the conservative default is to flag.
|
||||
|
||||
Sister to [[deftest-check-parallel]]: the deftest hook rejects `^:parallel`
|
||||
tests inside a `^:synchronized` ns, so the two checks together form a
|
||||
coherent opt-in: either the whole ns is explicitly non-parallel (fixtures can be
|
||||
tests inside a `^:synchronous` ns, so the two checks together form a
|
||||
coherent opt-in: either the whole ns is synchronous (fixtures can be
|
||||
anything; no parallel tests allowed) or it isn't (fixtures must be
|
||||
parallel-safe)."
|
||||
[{:keys [node config], ns-sym :ns, :as input}]
|
||||
(when-not (or (true? (:synchronized (meta ns-sym)))
|
||||
(and (contains? (meta ns-sym) :parallel)
|
||||
(false? (:parallel (meta ns-sym)))))
|
||||
(when-not (:synchronous (meta ns-sym))
|
||||
(let [linter-config (get-in config [:linters :metabase/validate-deftest])]
|
||||
(doseq [form (rest (:children node))]
|
||||
(warn-about-disallowed-parallel-forms form linter-config))))
|
||||
|
||||
@@ -71,7 +71,7 @@
|
||||
(let [module (module ns-symb)
|
||||
module-api-namespaces (module-api-namespaces config module)
|
||||
module-friends (module-friends config module)]
|
||||
(or (nil? module-api-namespaces)
|
||||
(or (empty? module-api-namespaces)
|
||||
(contains? module-api-namespaces ns-symb)
|
||||
(contains? module-friends current-module))))
|
||||
|
||||
|
||||
@@ -286,7 +286,6 @@
|
||||
(update context :node update-node)))
|
||||
|
||||
(comment
|
||||
(require '[clojure.pprint])
|
||||
(defn- defsetting* [form]
|
||||
(hooks/sexpr
|
||||
(:node
|
||||
@@ -294,6 +293,7 @@
|
||||
{:node
|
||||
(hooks/parse-string
|
||||
(with-out-str
|
||||
#_{:clj-kondo/ignore [:unresolved-namespace]}
|
||||
(clojure.pprint/pprint
|
||||
form)))}))))
|
||||
|
||||
|
||||
14
.clj-kondo/src/hooks/metabase/test/gentest.clj
Normal file
14
.clj-kondo/src/hooks/metabase/test/gentest.clj
Normal file
@@ -0,0 +1,14 @@
|
||||
(ns hooks.metabase.test.gentest
|
||||
(:require
|
||||
[clj-kondo.hooks-api :as hooks]))
|
||||
|
||||
(defn iterate*
|
||||
"(iterate {:as limit-spec} [s1 val1 & more] body-form body-form...)"
|
||||
[{{:keys [children] :as node} :node :as orig}]
|
||||
(let [[token limit-spec bindings & body-nodes] children
|
||||
node* (-> (hooks/list-node
|
||||
(list token limit-spec
|
||||
(hooks/list-node (list* (hooks/token-node 'let) bindings
|
||||
body-nodes))))
|
||||
(with-meta (meta node)))]
|
||||
(assoc orig :node node*)))
|
||||
@@ -19,19 +19,6 @@
|
||||
(defn- lint-modules [form config]
|
||||
(lint-ns form (assoc-in config [:linters :metabase/modules :level] :warning)))
|
||||
|
||||
(deftest ^:parallel explicit-nonparallel-namespace-metadata-test
|
||||
(let [config {:linters {:metabase/validate-deftest {:level :warning}}}]
|
||||
(testing "Hawk's supported namespace marker passes"
|
||||
(is (empty? (lint-ns '(ns ^:synchronized example.test) config))))
|
||||
(testing "the verbose Hawk spelling is not accepted as Metabase style"
|
||||
(is (=? [{:type :metabase/validate-deftest
|
||||
:message #(re-find #"Use `\^:synchronized` instead" %)}]
|
||||
(lint-ns '(ns ^{:parallel false} example.test) config))))
|
||||
(testing "the legacy marker explains that Hawk ignores it"
|
||||
(is (=? [{:type :metabase/validate-deftest
|
||||
:message #(re-find #"`\^:synchronous` is ignored by Hawk" %)}]
|
||||
(lint-ns '(ns ^:synchronous example.test) config))))))
|
||||
|
||||
(deftest ^:parallel module-checker-allowed-modules-test
|
||||
(are [task-namespace] (=? [{:message "Module task should not be used in the api module. [:metabase/modules api :uses]"
|
||||
:type :metabase/modules}]
|
||||
@@ -56,16 +43,6 @@
|
||||
'{:metabase/modules {api {:uses #{search}}
|
||||
search {:api #{metabase.search.core}}}}))))
|
||||
|
||||
(deftest ^:parallel module-checker-empty-api-test
|
||||
(is (=? [{:message "Namespace metabase.embeddings.core is not an allowed external API namespace for the embeddings module. [:metabase/modules embeddings :api]"
|
||||
:type :metabase/modules}]
|
||||
(lint-modules
|
||||
'(ns metabase.metabot.core
|
||||
(:require
|
||||
[metabase.embeddings.core :as embeddings]))
|
||||
'{:metabase/modules {metabot {:uses #{embeddings}}
|
||||
embeddings {:api #{}}}}))))
|
||||
|
||||
(deftest ^:parallel module-checker-friends-test
|
||||
(is (= []
|
||||
(lint-modules
|
||||
|
||||
@@ -1,65 +0,0 @@
|
||||
(ns hooks.clojure.core.require-test
|
||||
(:require
|
||||
[clj-kondo.hooks-api :as hooks]
|
||||
[clj-kondo.impl.utils]
|
||||
[clojure.test :refer :all]
|
||||
[hooks.clojure.core.require]))
|
||||
|
||||
(def ^:private config
|
||||
'{:linters {:metabase/modules {:level :warning}}
|
||||
:metabase/modules {search {:uses #{}}
|
||||
task {:api #{metabase.task.core}}}})
|
||||
|
||||
(defn- lint [hook form current-ns]
|
||||
(binding [clj-kondo.impl.utils/*ctx* {:config config
|
||||
:ignores (atom nil)
|
||||
:findings (atom [])
|
||||
:namespaces (atom {})}]
|
||||
(hook {:node (hooks/parse-string (pr-str form))
|
||||
:ns current-ns
|
||||
:config config})
|
||||
@(:findings clj-kondo.impl.utils/*ctx*)))
|
||||
|
||||
(defn- lint-require [form current-ns]
|
||||
(lint hooks.clojure.core.require/lint-require form current-ns))
|
||||
|
||||
(defn- lint-requiring-resolve [form current-ns]
|
||||
(lint hooks.clojure.core.require/lint-requiring-resolve form current-ns))
|
||||
|
||||
(def ^:private dynamic-require-finding
|
||||
{:message "Module dependency cannot be statically determined from a dynamic require"
|
||||
:type :metabase/modules})
|
||||
|
||||
(def ^:private cross-module-finding
|
||||
{:message "Module task should not be used in the search module. [:metabase/modules search :uses]"
|
||||
:type :metabase/modules})
|
||||
|
||||
(deftest ^:parallel dynamic-requiring-resolve-test
|
||||
(testing "outside the module system"
|
||||
(is (= [] (lint-requiring-resolve '(requiring-resolve (:failing-test-var opts)) 'dev))))
|
||||
(testing "inside a module"
|
||||
(are [form] (=? [dynamic-require-finding]
|
||||
(lint-requiring-resolve form 'metabase.search.core))
|
||||
'(requiring-resolve (:failing-test-var opts))
|
||||
'(requiring-resolve @some-atom)
|
||||
'(requiring-resolve some-local)
|
||||
'(requiring-resolve "not-a-symbol"))))
|
||||
|
||||
(deftest ^:parallel dynamic-require-test
|
||||
(testing "outside the module system"
|
||||
(is (= [] (lint-require '(require some-local) 'dev))))
|
||||
(testing "inside a module"
|
||||
(is (=? [dynamic-require-finding]
|
||||
(lint-require '(require some-local) 'metabase.search.core)))))
|
||||
|
||||
(deftest ^:parallel static-requiring-resolve-still-linted-test
|
||||
(is (=? [cross-module-finding]
|
||||
(lint-requiring-resolve '(requiring-resolve 'metabase.task.impl/go)
|
||||
'metabase.search.core))
|
||||
"a statically-known cross-module violation is still reported"))
|
||||
|
||||
(deftest ^:parallel static-require-still-linted-test
|
||||
(is (=? [cross-module-finding]
|
||||
(lint-require '(require 'metabase.task.impl :reload)
|
||||
'metabase.search.core))
|
||||
"a statically-known cross-module violation is still reported while keyword flags are ignored"))
|
||||
@@ -21,54 +21,6 @@
|
||||
#{:athena}}}}})
|
||||
(mapv :message @(:findings clj-kondo.impl.utils/*ctx*))))
|
||||
|
||||
(defn- parallel-metadata-findings
|
||||
([form]
|
||||
(parallel-metadata-findings form 'example.test))
|
||||
([form ns-sym]
|
||||
(let [linters {:metabase/deftest-not-marked-parallel-or-synchronized {:level :warning}
|
||||
:metabase/validate-deftest {:level :warning}}]
|
||||
(binding [clj-kondo.impl.utils/*ctx* {:config {:linters linters}
|
||||
:ignores (atom nil)
|
||||
:findings (atom [])
|
||||
:namespaces (atom {})}]
|
||||
(hooks.clojure.test/deftest {:node (api/parse-string form)
|
||||
:lang :clj
|
||||
:ns ns-sym
|
||||
:config {:linters {:metabase/validate-deftest {}}}})
|
||||
@(:findings clj-kondo.impl.utils/*ctx*)))))
|
||||
|
||||
(deftest ^:parallel explicit-nonparallel-metadata-test
|
||||
(testing "Hawk's supported test markers satisfy the explicit metadata linter"
|
||||
(are [metadata] (empty? (parallel-metadata-findings
|
||||
(format "(deftest %s example-test (is true))" metadata)))
|
||||
"^:parallel"
|
||||
"^:synchronized"))
|
||||
(testing "an unmarked test still gets the opt-in linter's guidance"
|
||||
(is (=? [{:type :metabase/deftest-not-marked-parallel-or-synchronized
|
||||
:message "Test should be marked either `^:parallel` or `^:synchronized`"}]
|
||||
(parallel-metadata-findings "(deftest example-test (is true))"))))
|
||||
(testing "the unsupported marker explains that Hawk ignores it"
|
||||
(is (some #(and (= (:type %) :metabase/validate-deftest)
|
||||
(re-find #"ignored by Hawk" (:message %)))
|
||||
(parallel-metadata-findings
|
||||
"(deftest ^:sequential example-test (is true))"))))
|
||||
(testing "the verbose Hawk spelling is not accepted as Metabase style"
|
||||
(is (some #(and (= (:type %) :metabase/validate-deftest)
|
||||
(re-find #"Use `\^:synchronized` instead" (:message %)))
|
||||
(parallel-metadata-findings
|
||||
"(deftest ^{:parallel false} example-test (is true))"))))
|
||||
(testing "a test cannot request both execution modes"
|
||||
(is (some #(and (= (:type %) :metabase/validate-deftest)
|
||||
(re-find #"both `\^:parallel` and `\^:synchronized`" (:message %)))
|
||||
(parallel-metadata-findings
|
||||
"(deftest ^:parallel ^:synchronized example-test (is true))"))))
|
||||
(testing "a namespace-level non-parallel promise rejects parallel tests"
|
||||
(is (=? [{:type :metabase/validate-deftest
|
||||
:message #(re-find #"namespace marked.*`\^:synchronized`" %)}]
|
||||
(parallel-metadata-findings
|
||||
"(deftest ^:parallel example-test (is true))"
|
||||
(with-meta 'example.test {:synchronized true}))))))
|
||||
|
||||
(deftest ^:parallel disallow-hardcoded-driver-names-in-tests-test
|
||||
(is (= []
|
||||
(deftest-warnings
|
||||
|
||||
@@ -12,7 +12,6 @@
|
||||
"enterprise/backend"
|
||||
"bin/build"
|
||||
"bin/lint-migrations-file"
|
||||
"bin/load-namespaces"
|
||||
"bin/release-list"
|
||||
#_"mage/src"
|
||||
#_"mage/test"
|
||||
|
||||
@@ -16,7 +16,7 @@ skip =
|
||||
frontend/src/metabase/querying/expressions/tokenizer/lezer.js,
|
||||
frontend/src/metabase/query_builder/components/NativeQueryEditor/CodeMirrorEditor/language.ts,
|
||||
frontend/src/metabase/static-viz/constants/char-sizes.ts,
|
||||
frontend/src/metabase/viz-core/lib/mapping_codes.ts,
|
||||
frontend/src/metabase/visualizations/lib/mapping_codes.ts,
|
||||
frontend/*/mocks/*,
|
||||
modules/drivers/mongo/src/metabase/driver/mongo/operators.clj,
|
||||
src/metabase/analyze/classifiers/name.clj,
|
||||
|
||||
18
.github/CODEOWNERS
vendored
18
.github/CODEOWNERS
vendored
@@ -10,7 +10,6 @@ src/metabase/query_processor @metabase/core-backend-querying-platform
|
||||
src/metabase/graph @metabase/graphy
|
||||
enterprise/backend/src/metabase_enterprise/dependencies @metabase/graphy
|
||||
enterprise/frontend/src/metabase-enterprise/dependencies @metabase/graphy
|
||||
enterprise/frontend/src/metabase-enterprise/monitor/dependency-diagnostics @metabase/graphy
|
||||
|
||||
# @metabase/embedding
|
||||
|
||||
@@ -42,9 +41,6 @@ docs/api.json # auto-generated
|
||||
docs/configuring-metabase/environment-variables.md # auto-generated
|
||||
docs/configuring-metabase/config-template.md # auto-generated
|
||||
docs/installation-and-operation/commands.md # auto-generated
|
||||
docs/ai/providers.md # auto-generated
|
||||
docs/embedding/sdk/api # auto-generated
|
||||
docs/embedding/eajs/snippets # auto-generated
|
||||
|
||||
# @metabase/devexp
|
||||
.github @metabase/devexp
|
||||
@@ -53,17 +49,5 @@ docs/embedding/eajs/snippets # auto-generated
|
||||
src/metabase/analytics/prometheus.clj @metabase/cloud-ops
|
||||
|
||||
# @metabase/metabot-frontend
|
||||
enterprise/frontend/src/metabase-enterprise/metabot @metabase/metabot-frontend
|
||||
enterprise/frontend/src/metabase-enterprise/metabot @metabase/metabot-frontend
|
||||
frontend/src/metabase/metabot @metabase/metabot-frontend
|
||||
|
||||
# @metabase/core-frontend-gadget
|
||||
bin/custom-viz @metabase/core-frontend-gadget
|
||||
docs/developers-guide/custom-visualizations.md @metabase/core-frontend-gadget
|
||||
enterprise/frontend/src/custom-viz @metabase/core-frontend-gadget
|
||||
enterprise/frontend/src/metabase-enterprise/api/custom-viz-plugin.ts @metabase/core-frontend-gadget
|
||||
enterprise/frontend/src/metabase-enterprise/custom_viz @metabase/core-frontend-gadget
|
||||
frontend/src/metabase-types/api/custom-viz-plugin.ts @metabase/core-frontend-gadget
|
||||
frontend/src/metabase/plugins/oss/custom-viz.ts @metabase/core-frontend-gadget
|
||||
frontend/src/metabase/visualizations/custom-visualizations @metabase/core-frontend-gadget
|
||||
frontend/test/__support__/custom-viz-fixtures @metabase/core-frontend-gadget
|
||||
.github/workflows/release-custom-viz.yml @metabase/devexp @metabase/core-frontend-gadget
|
||||
|
||||
16
.github/actionlint.yaml
vendored
16
.github/actionlint.yaml
vendored
@@ -1,16 +0,0 @@
|
||||
self-hosted-runner:
|
||||
# Custom runner labels that actionlint cannot know about. Everything else is
|
||||
# selected through `vars.DEFAULT_RUNNER_KEY` and is not checked.
|
||||
labels:
|
||||
- arm-ubuntu-24-2core
|
||||
|
||||
paths:
|
||||
# GitHub added parallel steps (`parallel`, `background`, `wait`, `wait-all`,
|
||||
# `cancel`) in June 2026, and actionlint 1.7.12 does not know the schema yet:
|
||||
# https://github.blog/changelog/2026-06-25-actions-steps-can-now-be-run-in-parallel/
|
||||
.github/workflows/backend.yml:
|
||||
ignore:
|
||||
- 'step must run script with "run" section or run action with "uses" section'
|
||||
.github/workflows/lint.yml:
|
||||
ignore:
|
||||
- 'step must run script with "run" section or run action with "uses" section'
|
||||
136
.github/actions/bundle-size-check/action.yml
vendored
136
.github/actions/bundle-size-check/action.yml
vendored
@@ -1,136 +0,0 @@
|
||||
name: Bundle Size Check
|
||||
description: |
|
||||
Measures app + embedding SDK bundle sizes from the current and base-ref
|
||||
uberjars (reusing the builds that already ran) plus their bundle-stats
|
||||
artifacts, and flags gzipped size changes beyond the threshold for the main
|
||||
app's initial bundle and the chunked SDK's reachable total. The base ref is
|
||||
read back off the current uberjar so that both sides differ only by the PR.
|
||||
Skips when the base-ref artifacts aren't available rather than rebuilding from
|
||||
source (which would compare across pipelines).
|
||||
|
||||
inputs:
|
||||
edition:
|
||||
description: 'Metabase edition (ee or oss)'
|
||||
required: true
|
||||
threshold-percent:
|
||||
description: 'Percent threshold for bundle size change'
|
||||
required: true
|
||||
|
||||
outputs:
|
||||
app_status:
|
||||
description: "App initial bundle status: increased, decreased, or stable"
|
||||
value: ${{ steps.compare.outputs.app_initial_status }}
|
||||
app_size_change_percent:
|
||||
description: "App initial bundle size change percent (positive = increase, negative = decrease)"
|
||||
value: ${{ steps.compare.outputs.app_initial_size_change_percent }}
|
||||
sdk_status:
|
||||
description: "Chunked SDK bundle status: increased, decreased, or stable"
|
||||
value: ${{ steps.compare.outputs.embedding_sdk_chunked_total_status }}
|
||||
sdk_size_change_percent:
|
||||
description: "Chunked SDK bundle size change percent (positive = increase, negative = decrease)"
|
||||
value: ${{ steps.compare.outputs.embedding_sdk_chunked_total_size_change_percent }}
|
||||
|
||||
runs:
|
||||
using: "composite"
|
||||
steps:
|
||||
- name: Download current uberjar
|
||||
uses: ./.github/actions/fetch-artifact
|
||||
with:
|
||||
commit: ${{ github.event.pull_request.head.sha || github.sha }}
|
||||
edition: ${{ inputs.edition }}
|
||||
extract-path: temp/current-uberjar
|
||||
- name: Resolve the base ref the current uberjar was built against
|
||||
id: base-ref
|
||||
shell: bash
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
run: |
|
||||
set -uo pipefail
|
||||
|
||||
# The uberjar job checks out refs/pull/<n>/merge (the PR head already
|
||||
# merged into the base branch tip) but names its artifact after head.sha,
|
||||
# so the comparable base ref is that merge's first parent. Reading it back
|
||||
# off the jar describes the artifact actually being measured. Using
|
||||
# pull_request.base.sha instead compares against a stale snapshot of the
|
||||
# base branch, reporting everything that landed in between as this PR's
|
||||
# own bundle delta.
|
||||
unzip -o -j temp/current-uberjar/metabase.jar version.properties -d temp/current-version >/dev/null
|
||||
built=$(sed -n 's/^hash=//p' temp/current-version/version.properties)
|
||||
base=$(gh api "/repos/${GITHUB_REPOSITORY}/commits/${built}" \
|
||||
--jq 'if (.parents | length) == 2 then .parents[0].sha else empty end' 2>/dev/null) || base=""
|
||||
|
||||
echo "sha=${base}" >> "$GITHUB_OUTPUT"
|
||||
echo "The current uberjar was built from ${built}; its base ref is ${base:-unknown}"
|
||||
- name: Download base-ref uberjar
|
||||
id: base-jar
|
||||
if: steps.base-ref.outputs.sha != ''
|
||||
continue-on-error: true
|
||||
uses: ./.github/actions/fetch-artifact
|
||||
with:
|
||||
commit: ${{ steps.base-ref.outputs.sha }}
|
||||
edition: ${{ inputs.edition }}
|
||||
extract-path: temp/base-uberjar
|
||||
- name: Measure and compare bundle sizes
|
||||
id: compare
|
||||
shell: bash
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
EDITION: ${{ inputs.edition }}
|
||||
HEAD_SHA: ${{ github.event.pull_request.head.sha || github.sha }}
|
||||
BASE_SHA: ${{ steps.base-ref.outputs.sha }}
|
||||
THRESHOLD: ${{ inputs.threshold-percent }}
|
||||
MEASURE_BUNDLES: app,embedding-sdk-legacy,embedding-sdk-chunked
|
||||
# The gated bundles, as "<bundle>/<kind>". Each one's step outputs are
|
||||
# named after its key (see outputPrefix in compare.js), which is what the
|
||||
# action outputs above read.
|
||||
GATES: app/initial,embedding-sdk-chunked/total
|
||||
run: |
|
||||
set -uo pipefail
|
||||
|
||||
skip() {
|
||||
echo "::notice::Bundle-size check skipped: $1"
|
||||
for gate in ${GATES//,/ }; do
|
||||
prefix=$(printf '%s' "$gate" | tr -c 'a-zA-Z0-9' '_')
|
||||
echo "${prefix}_status=stable" >> "$GITHUB_OUTPUT"
|
||||
echo "${prefix}_size_change_percent=0" >> "$GITHUB_OUTPUT"
|
||||
done
|
||||
exit 0
|
||||
}
|
||||
|
||||
# The base ref needs an already-built uberjar. Rebuilding it from source
|
||||
# would compare across two build pipelines (the bug this check used to
|
||||
# have), so skip instead when it — or its stats — isn't available.
|
||||
[ -n "${BASE_SHA}" ] || skip "no comparable base ref for the current uberjar"
|
||||
[ -f temp/base-uberjar/metabase.jar ] || skip "base-ref uberjar not found for ${BASE_SHA}"
|
||||
|
||||
extract_js() { # $1=jar $2=dest-root
|
||||
mkdir -p "$2/resources"
|
||||
unzip -o "$1" \
|
||||
"frontend_client/app/dist/*" \
|
||||
"frontend_client/app/embed.js" \
|
||||
"frontend_client/app/embedding-sdk/*" \
|
||||
-d "$2/resources" >/dev/null || true
|
||||
}
|
||||
|
||||
fetch_stats() { # $1=sha $2=dest-dir
|
||||
local name="metabase-bundle-stats-${EDITION}-$1"
|
||||
local id
|
||||
id=$(gh api "/repos/${GITHUB_REPOSITORY}/actions/artifacts?name=${name}" --jq '.artifacts[0].id // empty' 2>/dev/null)
|
||||
[ -n "${id}" ] || return 1
|
||||
mkdir -p "$2"
|
||||
gh api "/repos/${GITHUB_REPOSITORY}/actions/artifacts/${id}/zip" > "$2.zip" 2>/dev/null || return 1
|
||||
unzip -o "$2.zip" -d "$2" >/dev/null || return 1
|
||||
}
|
||||
|
||||
extract_js temp/current-uberjar/metabase.jar temp/current
|
||||
extract_js temp/base-uberjar/metabase.jar temp/base
|
||||
|
||||
fetch_stats "${HEAD_SHA}" temp/current/artifacts || skip "bundle stats not found for current ${HEAD_SHA}"
|
||||
fetch_stats "${BASE_SHA}" temp/base/artifacts || skip "bundle stats not found for base ${BASE_SHA}"
|
||||
|
||||
MEASURE_ROOT=temp/current node ./.github/scripts/measure-bundle-sizes.js > temp/current-sizes.json \
|
||||
|| { echo "::error::Failed to measure current bundle sizes"; exit 1; }
|
||||
MEASURE_ROOT=temp/base node ./.github/scripts/measure-bundle-sizes.js > temp/base-sizes.json \
|
||||
|| { echo "::error::Failed to measure base bundle sizes"; exit 1; }
|
||||
|
||||
node ./.github/actions/bundle-size-check/compare.js temp/current-sizes.json temp/base-sizes.json
|
||||
133
.github/actions/bundle-size-check/compare.js
vendored
133
.github/actions/bundle-size-check/compare.js
vendored
@@ -1,133 +0,0 @@
|
||||
/* eslint-disable import/no-commonjs */
|
||||
// Diffs the bundle sizes produced by measure-bundle-sizes.js for the current
|
||||
// build and the base ref. Every (bundle, kind) is compared strictly against the
|
||||
// same (bundle, kind) — never one bundle type against another — using gzipped
|
||||
// (served) bytes.
|
||||
//
|
||||
// The report covers every measured bundle; the gates are the subset that drives
|
||||
// PR comments, named as "<bundle>/<kind>" (e.g. "app/initial",
|
||||
// "embedding-sdk-chunked/total").
|
||||
//
|
||||
// compareBundles is pure (data in, data out) so it can be unit-tested; main()
|
||||
// is the thin I/O wrapper that reads the size files, prints the report and sets
|
||||
// the step outputs / exit code.
|
||||
const fs = require("fs");
|
||||
|
||||
const pick = (rows, bundle, kind) => rows.find(row => row.bundle === bundle && row.kind === kind);
|
||||
const keyOf = row => `${row.bundle}/${row.kind}`;
|
||||
const mb = bytes => (bytes / 1024 / 1024).toFixed(2);
|
||||
const percentOf = (currentBytes, baseBytes) =>
|
||||
baseBytes ? Math.trunc(((currentBytes - baseBytes) * 100) / baseBytes) : 0;
|
||||
|
||||
// Output names are derived from the gate key, so the workflow reads
|
||||
// `app_initial_status`, `embedding_sdk_chunked_total_status`, and so on.
|
||||
const outputPrefix = gateKey => gateKey.replace(/[^a-zA-Z0-9]+/g, "_");
|
||||
|
||||
/**
|
||||
* Evaluate one gate. Returns the key plus exactly one of:
|
||||
* - `status` / `percent`: the bundle moved by `percent`.
|
||||
* - `skip` reason: the gate is non-actionable (reported stable, no comment).
|
||||
* - `error` reason: the gate inputs are missing (hard failure).
|
||||
*/
|
||||
function evaluateGate({ current, base, threshold, gateKey }) {
|
||||
const [bundle, kind] = gateKey.split("/");
|
||||
const gateCurrent = pick(current, bundle, kind);
|
||||
const gateBase = pick(base, bundle, kind);
|
||||
if (!gateCurrent || !gateBase || !gateBase.gzipBytes) {
|
||||
return { gateKey, error: `Could not find ${bundle} ${kind} (gzip) in both builds` };
|
||||
}
|
||||
|
||||
// Both sides must measure the bundle the same way. A base ref built before the
|
||||
// reachableAssets enrichment collapses its total to the initial set, so
|
||||
// comparing it against the current reachable total reports a phantom jump
|
||||
// (~30%). Skip until the base ref also carries reachable stats.
|
||||
if (Boolean(gateCurrent.reachable) !== Boolean(gateBase.reachable)) {
|
||||
return {
|
||||
gateKey,
|
||||
skip:
|
||||
`"${gateKey}" measured differently on each side ` +
|
||||
`(current reachable=${Boolean(gateCurrent.reachable)}, base reachable=${Boolean(gateBase.reachable)}); ` +
|
||||
`the base ref predates the reachable-chunk stats. Resolves once this change is on the base ref.`,
|
||||
};
|
||||
}
|
||||
|
||||
const percent = percentOf(gateCurrent.gzipBytes, gateBase.gzipBytes);
|
||||
const status = percent > threshold ? "increased" : percent < -threshold ? "decreased" : "stable";
|
||||
return { gateKey, status, percent };
|
||||
}
|
||||
|
||||
/**
|
||||
* Compare two measurement sets and evaluate every gate. Returns a
|
||||
* human-readable `report` over all measured bundles plus one entry per gate.
|
||||
*/
|
||||
function compareBundles({ current, base, threshold, gateKeys }) {
|
||||
const report = ["=== Bundle sizes (gzipped, current vs base) ==="];
|
||||
const keys = [...new Set([...current, ...base].map(keyOf))].sort();
|
||||
for (const key of keys) {
|
||||
const [bundle, kind] = key.split("/");
|
||||
const currentRow = pick(current, bundle, kind);
|
||||
const baseRow = pick(base, bundle, kind);
|
||||
if (!currentRow || !baseRow) {
|
||||
report.push(`${key}: present only in ${currentRow ? "current" : "base"} build`);
|
||||
continue;
|
||||
}
|
||||
const diff = currentRow.gzipBytes - baseRow.gzipBytes;
|
||||
const percent = percentOf(currentRow.gzipBytes, baseRow.gzipBytes);
|
||||
report.push(`${key}: ${mb(currentRow.gzipBytes)}MB vs ${mb(baseRow.gzipBytes)}MB (${percent}%, ${diff} bytes)`);
|
||||
}
|
||||
|
||||
return { report, gates: gateKeys.map(gateKey => evaluateGate({ current, base, threshold, gateKey })) };
|
||||
}
|
||||
|
||||
module.exports = { compareBundles, outputPrefix };
|
||||
|
||||
const setOutput = (name, value) => {
|
||||
if (process.env.GITHUB_OUTPUT) {
|
||||
fs.appendFileSync(process.env.GITHUB_OUTPUT, `${name}=${value}\n`);
|
||||
}
|
||||
};
|
||||
|
||||
function main() {
|
||||
const [, , currentPath, basePath] = process.argv;
|
||||
const threshold = Number(process.env.THRESHOLD ?? 2);
|
||||
const gateKeys = (process.env.GATES ?? "embedding-sdk-chunked/total").split(",").map(key => key.trim());
|
||||
|
||||
const current = JSON.parse(fs.readFileSync(currentPath, "utf8"));
|
||||
const base = JSON.parse(fs.readFileSync(basePath, "utf8"));
|
||||
|
||||
const { report, gates } = compareBundles({ current, base, threshold, gateKeys });
|
||||
console.log(report.join("\n"));
|
||||
console.log("");
|
||||
|
||||
for (const gate of gates) {
|
||||
const prefix = outputPrefix(gate.gateKey);
|
||||
|
||||
if (gate.error) {
|
||||
// One broken gate must not hide the others, so report them all and fail
|
||||
// the step at the end.
|
||||
console.error(`::error::${gate.error}`);
|
||||
continue;
|
||||
}
|
||||
|
||||
if (gate.skip) {
|
||||
// Treat the gate as non-actionable: report stable so no regression comment
|
||||
// is posted, mirroring the shell skip() for missing artifacts.
|
||||
console.log(`::notice::Bundle-size gate skipped: ${gate.skip}`);
|
||||
setOutput(`${prefix}_status`, "stable");
|
||||
setOutput(`${prefix}_size_change_percent`, "0");
|
||||
continue;
|
||||
}
|
||||
|
||||
console.log(`Gate: ${gate.gateKey} ${gate.percent}% (threshold ${threshold}%) → ${gate.status}`);
|
||||
setOutput(`${prefix}_status`, gate.status);
|
||||
setOutput(`${prefix}_size_change_percent`, gate.percent);
|
||||
}
|
||||
|
||||
if (gates.some(gate => gate.error)) {
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
if (require.main === module) {
|
||||
main();
|
||||
}
|
||||
@@ -1,84 +0,0 @@
|
||||
const { compareBundles, outputPrefix } = require("./compare");
|
||||
|
||||
const SDK_GATE = "embedding-sdk-chunked/total";
|
||||
const APP_GATE = "app/initial";
|
||||
|
||||
const total = (gzipBytes, over = {}) => ({
|
||||
bundle: "embedding-sdk-chunked",
|
||||
kind: "total",
|
||||
gzipBytes,
|
||||
reachable: true,
|
||||
...over,
|
||||
});
|
||||
|
||||
const appInitial = gzipBytes => ({ bundle: "app", kind: "initial", gzipBytes });
|
||||
|
||||
const compare = ({ current, base, threshold = 2, gateKeys = [SDK_GATE] }) =>
|
||||
compareBundles({ current, base, threshold, gateKeys });
|
||||
|
||||
describe("outputPrefix", () => {
|
||||
it("turns a gate key into a step-output prefix", () => {
|
||||
expect(outputPrefix(APP_GATE)).toBe("app_initial");
|
||||
expect(outputPrefix(SDK_GATE)).toBe("embedding_sdk_chunked_total");
|
||||
});
|
||||
});
|
||||
|
||||
describe("compareBundles", () => {
|
||||
it("gates on the chunked SDK total and flags an increase past the threshold", () => {
|
||||
const { gates, report } = compare({ current: [total(110)], base: [total(100)] });
|
||||
|
||||
expect(gates).toEqual([{ gateKey: SDK_GATE, status: "increased", percent: 10 }]);
|
||||
expect(report[0]).toContain("current vs base");
|
||||
});
|
||||
|
||||
it("flags a decrease and stays stable within the threshold", () => {
|
||||
expect(compare({ current: [total(90)], base: [total(100)] }).gates[0]).toMatchObject({
|
||||
status: "decreased",
|
||||
percent: -10,
|
||||
});
|
||||
expect(compare({ current: [total(101)], base: [total(100)] }).gates[0]).toMatchObject({
|
||||
status: "stable",
|
||||
percent: 1,
|
||||
});
|
||||
});
|
||||
|
||||
it("evaluates each gate against its own bundle", () => {
|
||||
const { gates } = compare({
|
||||
current: [total(100), appInitial(110)],
|
||||
base: [total(100), appInitial(100)],
|
||||
gateKeys: [APP_GATE, SDK_GATE],
|
||||
});
|
||||
|
||||
expect(gates).toEqual([
|
||||
{ gateKey: APP_GATE, status: "increased", percent: 10 },
|
||||
{ gateKey: SDK_GATE, status: "stable", percent: 0 },
|
||||
]);
|
||||
});
|
||||
|
||||
it("skips when the two sides measured a gate differently (reachable mismatch)", () => {
|
||||
const { gates } = compare({
|
||||
current: [total(130, { reachable: true })],
|
||||
base: [total(100, { reachable: false })],
|
||||
});
|
||||
|
||||
expect(gates[0].status).toBeUndefined();
|
||||
expect(gates[0].skip).toContain("predates the reachable-chunk stats");
|
||||
});
|
||||
|
||||
it("errors when a gate bundle is missing on either side, without hiding the others", () => {
|
||||
const { gates } = compare({
|
||||
current: [appInitial(100)],
|
||||
base: [appInitial(100)],
|
||||
gateKeys: [APP_GATE, SDK_GATE],
|
||||
});
|
||||
|
||||
expect(gates[0]).toEqual({ gateKey: APP_GATE, status: "stable", percent: 0 });
|
||||
expect(gates[1].error).toContain("embedding-sdk-chunked total");
|
||||
});
|
||||
|
||||
it("reports a bundle/kind present on only one side without erroring", () => {
|
||||
const { report } = compare({ current: [total(100), appInitial(50)], base: [total(100)] });
|
||||
|
||||
expect(report.some(line => line.includes("present only in current build"))).toBe(true);
|
||||
});
|
||||
});
|
||||
61
.github/actions/collect-profile/action.yml
vendored
61
.github/actions/collect-profile/action.yml
vendored
@@ -1,61 +0,0 @@
|
||||
name: Collect profile
|
||||
description: >-
|
||||
Attach a flamegraph and per-test wall times to a job whose tests ran under clj-async-profiler.
|
||||
Does nothing when the run was not profiled.
|
||||
|
||||
inputs:
|
||||
profiled:
|
||||
required: true
|
||||
description: >-
|
||||
"true" when the test step ran under the profiler. Passed in rather than read from the
|
||||
environment so the decision stays recorded by the step that made it.
|
||||
artifact-name:
|
||||
required: true
|
||||
description: Name for the uploaded artifact. Must be unique across the jobs of one run.
|
||||
|
||||
runs:
|
||||
using: "composite"
|
||||
steps:
|
||||
# Unlike a JUnit failure report, these run on a green build too — a suite that passes slowly is
|
||||
# exactly what profiling is for.
|
||||
#
|
||||
# `awk 'NR<=25'` rather than `head -25`: the runner ignores SIGPIPE, so `head` exiting early
|
||||
# leaves `sort` with an EPIPE write error and `pipefail` fails the step. `|| true` covers `grep`
|
||||
# finding no junit files.
|
||||
- name: Summarize test wall times
|
||||
if: ${{ inputs.profiled == 'true' }}
|
||||
continue-on-error: true
|
||||
shell: bash
|
||||
run: |
|
||||
{
|
||||
echo "### Slowest test namespaces (seconds)"
|
||||
echo '```'
|
||||
grep -h '<testsuite' target/junit/*.xml 2>/dev/null \
|
||||
| sed -E 's|.*<testsuite name="([^"]*)" time="([^"]*)".*|\2 \1|' \
|
||||
| sort -rn | awk 'NR<=25' || true
|
||||
echo '```'
|
||||
echo "### Slowest tests (seconds)"
|
||||
echo '```'
|
||||
grep -h '<testcase' target/junit/*.xml 2>/dev/null \
|
||||
| sed -E 's|.*classname="([^"]*)" name="([^"]*)" time="([^"]*)".*|\3 \1/\2|' \
|
||||
| sort -rn | awk 'NR<=25' || true
|
||||
echo '```'
|
||||
} >> "$GITHUB_STEP_SUMMARY"
|
||||
|
||||
# Only the rendered flamegraph is kept: the raw collapsed stacks it was built from run to
|
||||
# hundreds of MB on a long suite, and are reproducible locally from the same test selection.
|
||||
#
|
||||
# The info-level test log rides along on a green profiled run too. The failure-only log artifact
|
||||
# the test jobs upload is not written when nothing failed, which is exactly the run being profiled.
|
||||
- name: Upload profile
|
||||
if: ${{ inputs.profiled == 'true' }}
|
||||
continue-on-error: true
|
||||
uses: actions/upload-artifact@v7
|
||||
with:
|
||||
name: ${{ inputs.artifact-name }}
|
||||
path: |
|
||||
target/clj-async-profiler/results/*flamegraph.html
|
||||
target/junit/*.xml
|
||||
logs/test-log.json
|
||||
retention-days: 14
|
||||
if-no-files-found: warn
|
||||
43
.github/actions/create-backport/action.yml
vendored
43
.github/actions/create-backport/action.yml
vendored
@@ -1,5 +1,4 @@
|
||||
name: Create Backport PR
|
||||
description: Cherry-picks a commit onto a release branch and opens the backport pull request.
|
||||
|
||||
inputs:
|
||||
target-version: # major version number like 34 or 86
|
||||
@@ -25,8 +24,6 @@ runs:
|
||||
COMMIT: ${{ inputs.backport-commit }}
|
||||
shell: bash
|
||||
run: | # bash
|
||||
source "${{ github.action_path }}/kondo-ratchets.sh"
|
||||
|
||||
git config --global user.email "metabase-bot@metabase.com"
|
||||
git config --global user.name "Metabase bot"
|
||||
|
||||
@@ -36,7 +33,7 @@ runs:
|
||||
git checkout ${TARGET_BRANCH}
|
||||
git fetch --all
|
||||
git checkout -b ${BACKPORT_BRANCH}
|
||||
cherry_pick_backport "${COMMIT}"
|
||||
git cherry-pick ${COMMIT} || true
|
||||
|
||||
CONFLICT_LIST=$(git ls-files -u)
|
||||
|
||||
@@ -45,10 +42,13 @@ runs:
|
||||
git cherry-pick --abort
|
||||
|
||||
echo "Could not cherry pick because of a conflict"
|
||||
echo "has-conflicts=true" >> "$GITHUB_OUTPUT"
|
||||
echo "has-conflicts=true" >> $GITHUB_OUTPUT
|
||||
|
||||
# Add a shell script for resolving conflicts
|
||||
write_backport_script "${COMMIT}"
|
||||
echo "git reset HEAD~1" > ./backport.sh
|
||||
echo "rm ./backport.sh" >> ./backport.sh
|
||||
echo "git cherry-pick ${COMMIT}" >> ./backport.sh
|
||||
echo "echo 'Resolve conflicts and force push this branch.\n\nTo backport translations run: bin/i18n/merge-translations <release-branch>'" >> ./backport.sh
|
||||
chmod +x ./backport.sh
|
||||
git add ./backport.sh
|
||||
git commit -m "Add backport resolution script"
|
||||
@@ -57,43 +57,44 @@ runs:
|
||||
#${ORIGINAL_PULL_REQUEST_NUMBER}
|
||||
> [!IMPORTANT]
|
||||
> Manual conflict resolution is required.
|
||||
> Checkout the branch and run \`./backport.sh\` script. Force push your changes after cherry-picking. Mark the PR as ready for review once conflicts have been addressed.
|
||||
> Checkout the branch and run \`./backport.sh\` script. Force push your changes after cherry-picking. Check the box below when done.
|
||||
|
||||
Conflicts:
|
||||
\`\`\`shell
|
||||
${CONFLICT_LIST}
|
||||
\`\`\`
|
||||
|
||||
- [ ] Conflicts resolved
|
||||
END
|
||||
)
|
||||
else
|
||||
echo "has-conflicts=false" >> "$GITHUB_OUTPUT"
|
||||
echo "has-conflicts=false" >> $GITHUB_OUTPUT
|
||||
PR_BODY="#${ORIGINAL_PULL_REQUEST_NUMBER}"
|
||||
fi
|
||||
|
||||
git push -u origin ${BACKPORT_BRANCH}
|
||||
|
||||
PR_ARGS=(--base "${TARGET_BRANCH}" --head "${BACKPORT_BRANCH}" --label "was-backported" --assignee "${{ github.event.pull_request.user.login }}" --title "🤖 v${TARGET_VERSION} backported \"${ORIGINAL_TITLE}\"" --body "${PR_BODY}")
|
||||
if [ "$CONFLICTS" -gt 0 ]; then
|
||||
PR_ARGS+=(--draft)
|
||||
fi
|
||||
|
||||
BACKPORT_PR_URL=$(gh pr create "${PR_ARGS[@]}")
|
||||
BACKPORT_PR_URL=$(gh pr create --base "${TARGET_BRANCH}" --head "${BACKPORT_BRANCH}" --label "was-backported" --assignee "${{ github.event.pull_request.user.login }}" --title "🤖 backported \"${ORIGINAL_TITLE}\"" --body "${PR_BODY}")
|
||||
BACKPORT_PR_NUMBER=${BACKPORT_PR_URL##*/}
|
||||
|
||||
echo "backport_pr_number=$BACKPORT_PR_NUMBER" >> "$GITHUB_OUTPUT"
|
||||
echo "backport_pr_number=$BACKPORT_PR_NUMBER" >> $GITHUB_OUTPUT
|
||||
echo "New PR has been created: #${BACKPORT_PR_NUMBER}"
|
||||
|
||||
git checkout ${{ github.sha }}
|
||||
|
||||
- name: Approve and Merge the PR
|
||||
- name: Auto approve backport PR
|
||||
if: ${{ steps.create_backport_pull_request.outputs.has-conflicts == 'false' }}
|
||||
run: |
|
||||
gh pr review --approve "$PR_NUMBER"
|
||||
gh pr merge --squash --auto "$PR_NUMBER"
|
||||
uses: juliangruber/approve-pull-request-action@68fcc9a5a73b5641cadf757cf99d73720dcb05d0 # v2.1.0
|
||||
with:
|
||||
github-token: ${{ inputs.github-token-2 }}
|
||||
number: ${{ steps.create_backport_pull_request.outputs.backport_pr_number }}
|
||||
|
||||
- name: Enable Pull Request Automerge
|
||||
if: ${{ steps.create_backport_pull_request.outputs.has-conflicts == 'false' }}
|
||||
run: gh pr merge --squash --auto "$PR_NUMBER"
|
||||
shell: bash
|
||||
env:
|
||||
# uses a different token than the token that created the PR so we can approve and merge it
|
||||
GH_TOKEN: ${{ inputs.github-token-2 }}
|
||||
GH_TOKEN: ${{ inputs.github-token }}
|
||||
PR_NUMBER: ${{ steps.create_backport_pull_request.outputs.backport_pr_number }}
|
||||
|
||||
- uses: ./.github/actions/notify-pull-request
|
||||
|
||||
@@ -1,375 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
action_dir=$(cd "$(dirname "$0")" && pwd)
|
||||
cut_release_workflow="$action_dir/../../workflows/cut-release-branch.yml"
|
||||
|
||||
# shellcheck source=.github/actions/create-backport/kondo-ratchets.sh
|
||||
source "$action_dir/kondo-ratchets.sh"
|
||||
|
||||
# Any editor launch fails the run, so a continuation that waits on an editor cannot pass.
|
||||
export GIT_EDITOR=false
|
||||
|
||||
test_root=
|
||||
trap '[ -z "$test_root" ] || rm -rf -- "$test_root"' EXIT
|
||||
trap 'exit 129' HUP
|
||||
trap 'exit 130' INT
|
||||
trap 'exit 143' TERM
|
||||
test_root=$(mktemp -d)
|
||||
|
||||
active_ratchets='{:ignore-counts {:example 1}}'
|
||||
disabled_ratchets=';; Kondo ignore ratchets apply only to master; this release branch opts out.
|
||||
;; The test and fixer recognize this explicit opt-out.
|
||||
{:disabled true}'
|
||||
|
||||
# Coverage:
|
||||
#
|
||||
# Target state Incoming change Other conflict Expected result
|
||||
# ----------------- -------------------- --------------- ------------------------------------------
|
||||
# active ratchets unrelated file none leave ratchets unchanged
|
||||
# active ratchets ratchets none install the release opt-out
|
||||
# disabled ratchets ratchets none keep the opt-out; finish with an empty commit
|
||||
# divergent ratchets ratchets none resolve ratchets and finish the cherry-pick
|
||||
# divergent ratchets ratchets (on a pty) none finish without opening an editor
|
||||
# active ratchets unrelated file app.txt leave the conflict for manual resolution
|
||||
# divergent ratchets ratchets + app.txt app.txt resolve only ratchets
|
||||
# release cut ratchets none cut and backport commit the same opt-out
|
||||
# n/a cut-release workflow n/a the workflow calls the shared writer
|
||||
# any invalid commit n/a propagate the cherry-pick failure
|
||||
# divergent ratchets generated backport.sh app.txt resolve only ratchets through the manual script
|
||||
#
|
||||
# Known gaps: an older target with no ratchets file, and an incoming commit that
|
||||
# deletes the ratchets file.
|
||||
|
||||
test_unrelated_backport_preserves_target_ratchets() (
|
||||
local repo="$test_root/unrelated"
|
||||
init_repo "$repo"
|
||||
local commit
|
||||
commit=$(feature_commit "$repo" change_app)
|
||||
git -C "$repo" checkout -q release
|
||||
|
||||
cd "$repo"
|
||||
cherry_pick_backport "$commit"
|
||||
|
||||
assert_eq feature "$(cat app.txt)" "the unrelated change was not cherry-picked"
|
||||
assert_ratchets "$active_ratchets"
|
||||
assert_cherry_pick_completed
|
||||
assert_worktree_clean
|
||||
)
|
||||
|
||||
test_clean_ratchet_backport_installs_release_opt_out() (
|
||||
local repo="$test_root/clean-ratchet"
|
||||
init_repo "$repo"
|
||||
local commit
|
||||
commit=$(feature_commit "$repo" change_ratchets)
|
||||
git -C "$repo" checkout -q release
|
||||
|
||||
cd "$repo"
|
||||
cherry_pick_backport "$commit"
|
||||
|
||||
assert_ratchets_disabled
|
||||
assert_cherry_pick_completed
|
||||
assert_worktree_clean
|
||||
)
|
||||
|
||||
test_disabled_release_absorbs_ratchet_only_backport() (
|
||||
local repo="$test_root/disabled-release"
|
||||
init_repo "$repo"
|
||||
local commit
|
||||
commit=$(feature_commit "$repo" change_ratchets)
|
||||
git -C "$repo" checkout -q release
|
||||
commit_release_change "$repo" disable_release_ratchets "Disable release ratchets"
|
||||
|
||||
cd "$repo"
|
||||
cherry_pick_backport "$commit"
|
||||
|
||||
assert_ratchets_disabled
|
||||
assert_cherry_pick_completed
|
||||
assert_empty_commit
|
||||
assert_worktree_clean
|
||||
)
|
||||
|
||||
test_ratchet_only_conflict_is_resolved() (
|
||||
local repo="$test_root/ratchet-conflict"
|
||||
init_repo "$repo"
|
||||
local commit
|
||||
commit=$(conflicting_ratchet_commit "$repo")
|
||||
|
||||
cd "$repo"
|
||||
cherry_pick_backport "$commit"
|
||||
|
||||
assert_ratchets_disabled
|
||||
assert_cherry_pick_completed
|
||||
assert_worktree_clean
|
||||
)
|
||||
|
||||
test_continuation_never_opens_an_editor() (
|
||||
local repo="$test_root/editor"
|
||||
init_repo "$repo"
|
||||
local commit
|
||||
commit=$(conflicting_ratchet_commit "$repo")
|
||||
|
||||
cd "$repo"
|
||||
# Git only opens an editor for the continuation when stdin is a terminal, so run it on a pty. The
|
||||
# exported GIT_EDITOR=false turns any editor launch into a failure. The single quotes are deliberate:
|
||||
# $1 and $2 are for the inner bash, not this shell.
|
||||
# shellcheck disable=SC2016
|
||||
python3 -c 'import os, pty, sys; sys.exit(os.waitstatus_to_exitcode(pty.spawn(sys.argv[1:])))' \
|
||||
bash -c 'source "$1/kondo-ratchets.sh" && cherry_pick_backport "$2"' bash "$action_dir" "$commit" >/dev/null ||
|
||||
fail "the cherry-pick continuation opened an editor"
|
||||
|
||||
assert_ratchets_disabled
|
||||
assert_cherry_pick_completed
|
||||
)
|
||||
|
||||
test_non_ratchet_conflict_remains_manual() (
|
||||
local repo="$test_root/non-ratchet-conflict"
|
||||
init_repo "$repo"
|
||||
local commit
|
||||
commit=$(feature_commit "$repo" change_app)
|
||||
git -C "$repo" checkout -q release
|
||||
commit_release_change "$repo" change_release_app "Change release app"
|
||||
|
||||
cd "$repo"
|
||||
cherry_pick_backport "$commit"
|
||||
|
||||
assert_ratchets "$active_ratchets"
|
||||
assert_only_conflicts app.txt
|
||||
assert_cherry_pick_pending
|
||||
)
|
||||
|
||||
test_mixed_conflict_resolves_only_ratchets() (
|
||||
local repo="$test_root/mixed-conflict"
|
||||
init_repo "$repo"
|
||||
local commit
|
||||
commit=$(feature_commit "$repo" change_app_and_ratchets)
|
||||
git -C "$repo" checkout -q release
|
||||
commit_release_change "$repo" change_release_app_and_ratchets "Change release app and ratchets"
|
||||
|
||||
cd "$repo"
|
||||
cherry_pick_backport "$commit"
|
||||
|
||||
assert_ratchets_disabled
|
||||
assert_only_conflicts app.txt
|
||||
assert_cherry_pick_pending
|
||||
)
|
||||
|
||||
test_release_cut_and_backport_commit_the_same_opt_out() (
|
||||
local cut_repo="$test_root/release-cut"
|
||||
init_repo "$cut_repo"
|
||||
cd "$cut_repo"
|
||||
git checkout -qb release-x.99.x
|
||||
write_disabled_ratchets
|
||||
git add --sparse .clj-kondo/ratchets.edn
|
||||
git commit -q --allow-empty --no-verify -m "Cut release-x.99.x branch"
|
||||
git show HEAD:.clj-kondo/ratchets.edn > "$test_root/release-cut.edn"
|
||||
|
||||
local backport_repo="$test_root/release-cut-backport"
|
||||
init_repo "$backport_repo"
|
||||
local commit
|
||||
commit=$(feature_commit "$backport_repo" change_ratchets)
|
||||
git -C "$backport_repo" checkout -q release
|
||||
cd "$backport_repo"
|
||||
cherry_pick_backport "$commit"
|
||||
git show HEAD:.clj-kondo/ratchets.edn > "$test_root/backport.edn"
|
||||
|
||||
cmp -s "$test_root/release-cut.edn" "$test_root/backport.edn" ||
|
||||
fail "the release cut and the backport committed different opt-out files"
|
||||
assert_eq "$disabled_ratchets" "$(cat "$test_root/release-cut.edn")" \
|
||||
"the release cut did not install the expected opt-out"
|
||||
)
|
||||
|
||||
test_cut_release_workflow_uses_the_shared_writer() (
|
||||
grep -Fq 'source .github/actions/create-backport/kondo-ratchets.sh' "$cut_release_workflow" ||
|
||||
fail "the cut-release workflow does not source kondo-ratchets.sh"
|
||||
grep -Fq 'write_disabled_ratchets' "$cut_release_workflow" ||
|
||||
fail "the cut-release workflow does not call write_disabled_ratchets"
|
||||
! grep -Fq ':disabled' "$cut_release_workflow" ||
|
||||
fail "the cut-release workflow carries its own copy of the opt-out"
|
||||
)
|
||||
|
||||
test_invalid_commit_failure_is_propagated() (
|
||||
local repo="$test_root/invalid"
|
||||
init_repo "$repo"
|
||||
cd "$repo"
|
||||
|
||||
if cherry_pick_backport deadbeef; then
|
||||
fail "an invalid commit succeeded"
|
||||
fi
|
||||
)
|
||||
|
||||
test_generated_script_resolves_ratchets_end_to_end() (
|
||||
local repo="$test_root/generated-script"
|
||||
init_repo "$repo"
|
||||
local commit
|
||||
commit=$(feature_commit "$repo" change_app_and_ratchets)
|
||||
git -C "$repo" checkout -q release
|
||||
commit_release_change "$repo" change_release_app_and_ratchets "Change release app and ratchets"
|
||||
|
||||
cd "$repo"
|
||||
cherry_pick_backport "$commit"
|
||||
git cherry-pick --abort
|
||||
|
||||
local release_commit
|
||||
release_commit=$(git rev-parse HEAD)
|
||||
write_backport_script "$commit"
|
||||
chmod +x backport.sh
|
||||
git add backport.sh
|
||||
git commit -qm "Add manual backport script"
|
||||
|
||||
bash -n backport.sh
|
||||
grep -Fq 'write_disabled_ratchets ()' backport.sh || fail "the generated script omitted write_disabled_ratchets"
|
||||
./backport.sh
|
||||
|
||||
assert_ratchets_disabled
|
||||
assert_only_conflicts app.txt
|
||||
assert_cherry_pick_pending
|
||||
assert_eq "$release_commit" "$(git rev-parse HEAD)" "the generated script did not return HEAD to the release commit"
|
||||
[ ! -e backport.sh ] || fail "the generated script did not remove itself"
|
||||
! git ls-files --error-unmatch backport.sh >/dev/null 2>&1 || fail "the generated script is still tracked"
|
||||
! git status --porcelain | grep -q backport.sh || fail "the generated script left a status entry"
|
||||
)
|
||||
|
||||
# Test harness
|
||||
|
||||
fail() {
|
||||
echo "FAIL: $*" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
assert_eq() {
|
||||
local expected=$1
|
||||
local actual=$2
|
||||
local message=$3
|
||||
[ "$expected" = "$actual" ] || fail "$message (expected '$expected', got '$actual')"
|
||||
}
|
||||
|
||||
assert_ratchets() {
|
||||
assert_eq "$1" "$(cat .clj-kondo/ratchets.edn)" "unexpected ratchet state"
|
||||
}
|
||||
|
||||
assert_ratchets_disabled() {
|
||||
assert_ratchets "$disabled_ratchets"
|
||||
}
|
||||
|
||||
assert_cherry_pick_completed() {
|
||||
! git rev-parse -q --verify CHERRY_PICK_HEAD >/dev/null || fail "the cherry-pick is still pending"
|
||||
}
|
||||
|
||||
assert_cherry_pick_pending() {
|
||||
git rev-parse -q --verify CHERRY_PICK_HEAD >/dev/null || fail "the cherry-pick state was lost"
|
||||
}
|
||||
|
||||
assert_only_conflicts() {
|
||||
local expected
|
||||
local actual
|
||||
expected=$(printf '%s\n' "$@" | sort)
|
||||
actual=$(git diff --name-only --diff-filter=U | sort)
|
||||
assert_eq "$expected" "$actual" "unexpected unresolved files"
|
||||
}
|
||||
|
||||
assert_empty_commit() {
|
||||
git diff --quiet HEAD^ HEAD || fail "the backport commit was not empty"
|
||||
}
|
||||
|
||||
assert_worktree_clean() {
|
||||
[ -z "$(git status --porcelain)" ] || fail "the backport left worktree changes"
|
||||
}
|
||||
|
||||
init_repo() {
|
||||
local repo=$1
|
||||
git init -q -b release "$repo"
|
||||
git -C "$repo" config user.email test@example.com
|
||||
git -C "$repo" config user.name "Backport test"
|
||||
printf 'base\n' > "$repo/app.txt"
|
||||
mkdir -p "$repo/.clj-kondo"
|
||||
printf '%s\n' "$active_ratchets" > "$repo/.clj-kondo/ratchets.edn"
|
||||
git -C "$repo" add .
|
||||
git -C "$repo" commit -qm "Base"
|
||||
}
|
||||
|
||||
feature_commit() {
|
||||
local repo=$1
|
||||
shift
|
||||
git -C "$repo" checkout -qb feature
|
||||
"$@" "$repo"
|
||||
git -C "$repo" add .
|
||||
git -C "$repo" commit -qm "Feature"
|
||||
git -C "$repo" rev-parse HEAD
|
||||
}
|
||||
|
||||
# Sets up a release branch whose ratchets conflict with the feature commit, and prints that commit.
|
||||
conflicting_ratchet_commit() {
|
||||
local repo=$1
|
||||
local commit
|
||||
commit=$(feature_commit "$repo" change_ratchets)
|
||||
git -C "$repo" checkout -q release
|
||||
commit_release_change "$repo" change_release_ratchets "Change release ratchets"
|
||||
echo "$commit"
|
||||
}
|
||||
|
||||
change_app() {
|
||||
printf 'feature\n' > "$1/app.txt"
|
||||
}
|
||||
|
||||
change_ratchets() {
|
||||
printf '{:ignore-counts {:example 2}}\n' > "$1/.clj-kondo/ratchets.edn"
|
||||
}
|
||||
|
||||
change_app_and_ratchets() {
|
||||
change_app "$1"
|
||||
change_ratchets "$1"
|
||||
}
|
||||
|
||||
change_release_app() {
|
||||
printf 'release\n' > "$1/app.txt"
|
||||
}
|
||||
|
||||
change_release_ratchets() {
|
||||
printf '{:ignore-counts {:release 1}}\n' > "$1/.clj-kondo/ratchets.edn"
|
||||
}
|
||||
|
||||
change_release_app_and_ratchets() {
|
||||
change_release_app "$1"
|
||||
change_release_ratchets "$1"
|
||||
}
|
||||
|
||||
# Setup writes the opt-out through the shared writer; the only copy is the expected text at the top.
|
||||
disable_release_ratchets() {
|
||||
(cd "$1" && write_disabled_ratchets)
|
||||
}
|
||||
|
||||
commit_release_change() {
|
||||
local repo=$1
|
||||
local change=$2
|
||||
local message=$3
|
||||
"$change" "$repo"
|
||||
git -C "$repo" add .
|
||||
git -C "$repo" commit -qm "$message"
|
||||
}
|
||||
|
||||
run_test() {
|
||||
local description=$1
|
||||
local test=${description// /_}
|
||||
test=${test//-/_}
|
||||
test="test_$test"
|
||||
|
||||
declare -F "$test" >/dev/null || fail "missing test function: $test"
|
||||
|
||||
printf 'TEST: %s\n' "$description"
|
||||
"$test"
|
||||
printf 'PASS: %s\n' "$description"
|
||||
}
|
||||
|
||||
run_test "unrelated backport preserves target ratchets"
|
||||
run_test "clean ratchet backport installs release opt-out"
|
||||
run_test "disabled release absorbs ratchet-only backport"
|
||||
run_test "ratchet-only conflict is resolved"
|
||||
run_test "continuation never opens an editor"
|
||||
run_test "non-ratchet conflict remains manual"
|
||||
run_test "mixed conflict resolves only ratchets"
|
||||
run_test "release cut and backport commit the same opt-out"
|
||||
run_test "cut-release workflow uses the shared writer"
|
||||
run_test "invalid commit failure is propagated"
|
||||
run_test "generated script resolves ratchets end-to-end"
|
||||
@@ -1,74 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# The only place that knows what a release branch's opt-out looks like. The cut-release workflow, the
|
||||
# backport helper, and the tests all go through this function.
|
||||
write_disabled_ratchets() {
|
||||
mkdir -p .clj-kondo
|
||||
printf '%s\n' \
|
||||
';; Kondo ignore ratchets apply only to master; this release branch opts out.' \
|
||||
';; The test and fixer recognize this explicit opt-out.' \
|
||||
'{:disabled true}' \
|
||||
> .clj-kondo/ratchets.edn
|
||||
}
|
||||
|
||||
disable_ratchets() {
|
||||
local commit=$1
|
||||
local file=.clj-kondo/ratchets.edn
|
||||
[ -n "$(git diff-tree --root --no-commit-id --name-only -r "$commit" -- "$file")" ] || return 0
|
||||
|
||||
echo "$file: disabling kondo ratchets on the release branch"
|
||||
write_disabled_ratchets
|
||||
git add -- "$file"
|
||||
|
||||
if ! git rev-parse -q --verify CHERRY_PICK_HEAD >/dev/null; then
|
||||
if ! git diff --cached --quiet HEAD; then
|
||||
git commit --amend --no-edit
|
||||
fi
|
||||
return 0
|
||||
fi
|
||||
|
||||
if [ -z "$(git ls-files -u)" ]; then
|
||||
echo "$file was the only conflict, finishing the cherry-pick"
|
||||
if git diff --cached --quiet HEAD; then
|
||||
echo "nothing left to apply, committing the backport empty"
|
||||
git commit --allow-empty --no-edit
|
||||
else
|
||||
# GIT_EDITOR outranks core.editor, VISUAL, and EDITOR, so the continuation can never wait on
|
||||
# an editor; the repository's own editor configuration is left alone.
|
||||
GIT_EDITOR=true git cherry-pick --continue
|
||||
fi
|
||||
fi
|
||||
}
|
||||
|
||||
cherry_pick_backport() {
|
||||
local commit=$1
|
||||
local status=0
|
||||
git cherry-pick "$commit" || status=$?
|
||||
if [ "$status" -ne 0 ] && ! git rev-parse -q --verify CHERRY_PICK_HEAD >/dev/null; then
|
||||
echo "cherry-pick failed without entering a resolvable conflict state" >&2
|
||||
return "$status"
|
||||
fi
|
||||
|
||||
disable_ratchets "$commit"
|
||||
if [ "$status" -ne 0 ] &&
|
||||
git rev-parse -q --verify CHERRY_PICK_HEAD >/dev/null &&
|
||||
[ -z "$(git ls-files -u)" ]; then
|
||||
echo "cherry-pick stopped without a file conflict to resolve" >&2
|
||||
return "$status"
|
||||
fi
|
||||
}
|
||||
|
||||
write_backport_script() {
|
||||
local commit=$1
|
||||
{
|
||||
echo "#!/usr/bin/env bash"
|
||||
echo "set -euo pipefail"
|
||||
echo "git reset HEAD~1"
|
||||
echo "rm ./backport.sh"
|
||||
declare -f write_disabled_ratchets disable_ratchets cherry_pick_backport
|
||||
printf 'cherry_pick_backport %q\n' "$commit"
|
||||
printf '%s\n' "printf 'Resolve conflicts and force push this branch.\\n\\nTo backport translations run: bin/i18n/merge-translations <release-branch>\\n'"
|
||||
} > ./backport.sh
|
||||
}
|
||||
@@ -1,7 +1,4 @@
|
||||
name: Cross-branch migration test
|
||||
description: >-
|
||||
Boot Metabase on the base branch to migrate the app db, then boot the PR branch against the
|
||||
same db, verifying the PR's migrations apply cleanly on top of the base branch's schema.
|
||||
|
||||
inputs:
|
||||
base-ref: ## Base branch PR came from
|
||||
|
||||
@@ -93,8 +93,8 @@ runs:
|
||||
|
||||
if ${{ inputs.maildev }}; then
|
||||
echo -e "${Y}checking maildev container..." &&
|
||||
while ! nc -z localhost 1180; do sleep 1; done &&
|
||||
while ! nc -z localhost 1125; do sleep 1; done &&
|
||||
while ! nc -z localhost 1080; do sleep 1; done &&
|
||||
while ! nc -z localhost 1025; do sleep 1; done &&
|
||||
echo -e "${G}Maildev is up and running!"
|
||||
|
||||
echo -e "${Y}checking maildev-ssl container..." &&
|
||||
|
||||
4
.github/actions/fetch-artifact/action.yml
vendored
4
.github/actions/fetch-artifact/action.yml
vendored
@@ -113,9 +113,9 @@ runs:
|
||||
|
||||
# Set the JAR_PATH environment variable
|
||||
if [ "$EXTRACT_PATH" == "." ]; then
|
||||
echo "JAR_PATH=$OUTPUT_NAME" >> "$GITHUB_ENV"
|
||||
echo "JAR_PATH=$OUTPUT_NAME" >> $GITHUB_ENV
|
||||
else
|
||||
echo "JAR_PATH=$EXTRACT_PATH/$OUTPUT_NAME" >> "$GITHUB_ENV"
|
||||
echo "JAR_PATH=$EXTRACT_PATH/$OUTPUT_NAME" >> $GITHUB_ENV
|
||||
fi
|
||||
|
||||
- name: Verify that this is a valid JAR file
|
||||
|
||||
@@ -22,7 +22,7 @@ runs:
|
||||
COMMIT=$(env -i git log $BASE_REF --grep="(#$PULL_REQUEST_NUMBER)" --format="%H")
|
||||
echo "commit SHA $COMMIT"
|
||||
|
||||
echo "commit=$COMMIT" >> "$GITHUB_OUTPUT"
|
||||
echo "commit=$COMMIT" >> $GITHUB_OUTPUT
|
||||
id: find-squashed-commit
|
||||
shell: bash
|
||||
env:
|
||||
|
||||
45
.github/actions/get-cache-keys/action.yml
vendored
Normal file
45
.github/actions/get-cache-keys/action.yml
vendored
Normal file
@@ -0,0 +1,45 @@
|
||||
name: Get cache keys
|
||||
description: |
|
||||
Get deterministic cache keys for Maven and Node.js caches based on the centrally controlled cache prefix,
|
||||
and the suffix based on the runner OS and the current week of the year.
|
||||
|
||||
outputs:
|
||||
m2-cache-key:
|
||||
description: "The cache key for the Maven cache."
|
||||
value: ${{ steps.keys-factory.outputs.m2-cache-key }}
|
||||
m2-restore-key:
|
||||
description: "The fallback restore key prefix for the Maven cache."
|
||||
value: ${{ steps.keys-factory.outputs.m2-restore-key }}
|
||||
node-cache-key:
|
||||
description: "The cache key for the Node.js cache."
|
||||
value: ${{ steps.keys-factory.outputs.node-cache-key }}
|
||||
node-restore-key:
|
||||
description: "The fallback restore key prefix for the Node.js cache."
|
||||
value: ${{ steps.keys-factory.outputs.node-restore-key }}
|
||||
eslint-cache-key:
|
||||
description: "The cache key for Eslint cache"
|
||||
value: ${{ steps.keys-factory.outputs.eslint-cache-key }}
|
||||
cypress-cache-key:
|
||||
description: "The cache key for Cypress cache"
|
||||
value: ${{ steps.keys-factory.outputs.cypress-cache-key }}
|
||||
|
||||
runs:
|
||||
using: "composite"
|
||||
steps:
|
||||
- name: Generate and output cache keys
|
||||
id: keys-factory
|
||||
run: |
|
||||
M2_CACHE_PREFIX='M2'
|
||||
NODE_CACHE_PREFIX='node-modules'
|
||||
ESLINT_CACHE_PREFIX='eslint'
|
||||
CYPRESS_CACHE_PREFIX='cypress'
|
||||
WEEK_OF_YEAR=$(/bin/date "+%G-%V")
|
||||
CACHE_SUFFIX="${{ runner.os }}-$WEEK_OF_YEAR"
|
||||
|
||||
echo "m2-cache-key=$M2_CACHE_PREFIX-$CACHE_SUFFIX" >> $GITHUB_OUTPUT
|
||||
echo "m2-restore-key=$M2_CACHE_PREFIX-" >> $GITHUB_OUTPUT
|
||||
echo "node-cache-key=$NODE_CACHE_PREFIX-$CACHE_SUFFIX" >> $GITHUB_OUTPUT
|
||||
echo "node-restore-key=$NODE_CACHE_PREFIX-" >> $GITHUB_OUTPUT
|
||||
echo "eslint-cache-key=$ESLINT_CACHE_PREFIX-$CACHE_SUFFIX" >> $GITHUB_OUTPUT
|
||||
echo "cypress-cache-key=$CYPRESS_CACHE_PREFIX-$CACHE_SUFFIX" >> $GITHUB_OUTPUT
|
||||
shell: bash
|
||||
@@ -1,11 +1,12 @@
|
||||
name: Notify Pull Request
|
||||
description: Post a comment on the pull request, optionally replacing the previous one with the same message.
|
||||
inputs:
|
||||
remove-previous-comment:
|
||||
required: false
|
||||
type: boolean
|
||||
default: false
|
||||
add-new-comment:
|
||||
required: false
|
||||
type: boolean
|
||||
default: true
|
||||
message:
|
||||
required: true
|
||||
|
||||
48
.github/actions/prepare-backend/action.yml
vendored
48
.github/actions/prepare-backend/action.yml
vendored
@@ -5,6 +5,9 @@ inputs:
|
||||
java-version:
|
||||
required: true
|
||||
default: "25"
|
||||
clojure-version:
|
||||
required: true
|
||||
default: "1.12.0.1488"
|
||||
|
||||
runs:
|
||||
using: "composite"
|
||||
@@ -14,40 +17,12 @@ runs:
|
||||
with:
|
||||
java-version: ${{ inputs.java-version }}
|
||||
distribution: "temurin"
|
||||
- name: Cache spec
|
||||
id: cache-spec
|
||||
shell: bash
|
||||
run: .github/scripts/cache-keys.sh | tee -a "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Restore Clojure CLI
|
||||
uses: actions/cache/restore@v5
|
||||
with:
|
||||
path: ${{ steps.cache-spec.outputs.clojure-path }}
|
||||
key: ${{ steps.cache-spec.outputs.clojure-key }}
|
||||
|
||||
- name: Install Clojure CLI
|
||||
shell: bash
|
||||
env:
|
||||
CLOJURE_PREFIX: ${{ steps.cache-spec.outputs.clojure-path }}
|
||||
CLOJURE_VERSION: ${{ steps.cache-spec.outputs.clojure-version }}
|
||||
run: |
|
||||
set -uo pipefail
|
||||
echo "$CLOJURE_PREFIX/bin" >> "$GITHUB_PATH"
|
||||
export PATH="$CLOJURE_PREFIX/bin:$PATH"
|
||||
|
||||
# Whether the prefix already holds the right install is decided by running it, not by the
|
||||
# cache-hit flag. The installer bakes the prefix into the clj/clojure shims, so a tree restored
|
||||
# under a different $HOME points at a directory that is not there; and on a runner that is not
|
||||
# ephemeral, a previous version's prefix outlives the cache entry that stopped matching it.
|
||||
if clojure --version | grep -qF "$CLOJURE_VERSION"; then
|
||||
clojure --version
|
||||
exit 0
|
||||
fi
|
||||
|
||||
cd "$RUNNER_TEMP"
|
||||
for attempt in 1 2 3; do
|
||||
curl --retry 3 --retry-delay 5 --fail -O "https://download.clojure.org/install/linux-install-$CLOJURE_VERSION.sh" \
|
||||
&& bash "./linux-install-$CLOJURE_VERSION.sh" --prefix "$CLOJURE_PREFIX" \
|
||||
curl --retry 3 --retry-delay 5 --fail -O https://download.clojure.org/install/linux-install-${{ inputs.clojure-version }}.sh \
|
||||
&& sudo bash ./linux-install-${{ inputs.clojure-version }}.sh \
|
||||
&& break
|
||||
echo "::warning::Clojure install attempt $attempt failed, retrying in 10s..."
|
||||
sleep 10
|
||||
@@ -57,6 +32,10 @@ runs:
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: Get M2 cache key
|
||||
uses: ./.github/actions/get-cache-keys
|
||||
id: get-cache-keys
|
||||
|
||||
- name: Install uv (for Python package installation)
|
||||
uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b # v8.1.0
|
||||
with:
|
||||
@@ -67,6 +46,9 @@ runs:
|
||||
- name: Restore M2 cache
|
||||
uses: actions/cache/restore@v5
|
||||
with:
|
||||
path: ${{ steps.cache-spec.outputs.m2-path }}
|
||||
key: ${{ steps.cache-spec.outputs.m2-key }}
|
||||
restore-keys: ${{ steps.cache-spec.outputs.m2-restore-key }}
|
||||
path: |
|
||||
~/.m2
|
||||
~/.gitlibs
|
||||
key: ${{ steps.get-cache-keys.outputs.m2-cache-key }}
|
||||
restore-keys: |
|
||||
${{ steps.get-cache-keys.outputs.m2-restore-key }}
|
||||
|
||||
14
.github/actions/prepare-cypress/action.yml
vendored
14
.github/actions/prepare-cypress/action.yml
vendored
@@ -25,17 +25,15 @@ runs:
|
||||
chrome-version: 144
|
||||
id: setup-chrome
|
||||
|
||||
- name: Cache spec
|
||||
id: cache-spec
|
||||
shell: bash
|
||||
run: .github/scripts/cache-keys.sh | tee -a "$GITHUB_OUTPUT"
|
||||
- name: Get cache keys
|
||||
uses: ./.github/actions/get-cache-keys
|
||||
id: get-cache-keys
|
||||
|
||||
- name: Restore Cypress binary
|
||||
- name: Get Cypress cache
|
||||
uses: actions/cache/restore@v5
|
||||
with:
|
||||
path: ${{ steps.cache-spec.outputs.cypress-path }}
|
||||
key: ${{ steps.cache-spec.outputs.cypress-key }}
|
||||
restore-keys: ${{ steps.cache-spec.outputs.cypress-restore-key }}
|
||||
path: ~/.cache/Cypress
|
||||
key: ${{ steps.get-cache-keys.outputs.cypress-cache-key }}
|
||||
|
||||
- name: Ensure that Cypress executable is ready
|
||||
run: |
|
||||
|
||||
56
.github/actions/prepare-frontend/action.yml
vendored
56
.github/actions/prepare-frontend/action.yml
vendored
@@ -16,37 +16,43 @@ runs:
|
||||
with:
|
||||
node-version: ${{ inputs.node-version }}
|
||||
|
||||
- name: Cache spec
|
||||
id: cache-spec
|
||||
shell: bash
|
||||
run: .github/scripts/cache-keys.sh | tee -a "$GITHUB_OUTPUT"
|
||||
- name: Get cache keys
|
||||
uses: ./.github/actions/get-cache-keys
|
||||
id: get-cache-keys
|
||||
|
||||
# The spec names the store path rather than discovering it, so that a job can restore the cache
|
||||
# before bun exists. Exported job-wide rather than per-step because every later bun invocation has
|
||||
# to resolve against the same store. Confirm bun agreed: if the variable stopped taking effect, bun
|
||||
# would fill its default directory and the cache would silently save an empty tree.
|
||||
- name: Point bun at the cached store
|
||||
shell: bash
|
||||
env:
|
||||
STORE: ${{ steps.cache-spec.outputs.bun-store-path }}
|
||||
run: |
|
||||
echo "BUN_INSTALL_CACHE_DIR=$STORE" >> "$GITHUB_ENV"
|
||||
ACTUAL=$(BUN_INSTALL_CACHE_DIR="$STORE" bun pm cache)
|
||||
if [ "$ACTUAL" != "$STORE" ]; then
|
||||
echo "::error::bun reports its store at $ACTUAL, not $STORE; BUN_INSTALL_CACHE_DIR is not taking effect"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: Restore bun store
|
||||
- name: Restore M2 cache
|
||||
uses: actions/cache/restore@v5
|
||||
with:
|
||||
path: ${{ steps.cache-spec.outputs.bun-store-path }}
|
||||
key: ${{ steps.cache-spec.outputs.bun-store-key }}
|
||||
restore-keys: ${{ steps.cache-spec.outputs.bun-store-restore-key }}
|
||||
path: |
|
||||
~/.m2
|
||||
~/.gitlibs
|
||||
key: ${{ steps.get-cache-keys.outputs.m2-cache-key }}
|
||||
restore-keys: |
|
||||
${{ steps.get-cache-keys.outputs.m2-restore-key }}
|
||||
|
||||
- name: Get bun cache directory
|
||||
id: bun-cache
|
||||
run: |
|
||||
BUN_CACHE_DIR=$(bun pm cache)
|
||||
echo "Bun cache directory: $BUN_CACHE_DIR"
|
||||
echo "dir=$BUN_CACHE_DIR" >> $GITHUB_OUTPUT
|
||||
shell: bash
|
||||
|
||||
- name: Restore node_modules cache
|
||||
uses: actions/cache/restore@v5
|
||||
id: node-modules-cache
|
||||
with:
|
||||
path: |
|
||||
${{ steps.bun-cache.outputs.dir }}
|
||||
node_modules
|
||||
key: ${{ steps.get-cache-keys.outputs.node-cache-key }}
|
||||
restore-keys: |
|
||||
${{ steps.get-cache-keys.outputs.node-restore-key }}
|
||||
|
||||
# postinstall lifecycle hook can be dangerous in CI so we only run it locally (see package.json)
|
||||
# If a job wants to patch packages, it does so explicitly in the CI. This setting is opt-out.
|
||||
- name: Install front-end dependencies
|
||||
# For example, we never want to cache node_modules with patches already applied so cache-generator.yml skips that step.
|
||||
- name: Clean install
|
||||
run: bun install --frozen-lockfile
|
||||
shell: bash
|
||||
- name: Patch frontend dependencies
|
||||
|
||||
@@ -88,9 +88,9 @@ runs:
|
||||
REPO_URL="https://github.com/metabase/shoppy.git"
|
||||
echo "Checking if branch '${SAMPLE_APP_BRANCH_NAME}' exists in ${REPO_URL}..."
|
||||
if git ls-remote --exit-code --heads "$REPO_URL" "${SAMPLE_APP_BRANCH_NAME}"; then
|
||||
echo "sample_app_branch_exists=true" >> "$GITHUB_OUTPUT"
|
||||
echo "sample_app_branch_exists=true" >> $GITHUB_OUTPUT
|
||||
else
|
||||
echo "sample_app_branch_exists=false" >> "$GITHUB_OUTPUT"
|
||||
echo "sample_app_branch_exists=false" >> $GITHUB_OUTPUT
|
||||
fi
|
||||
|
||||
- name: Configure AWS credentials
|
||||
|
||||
26
.github/actions/prepare-uberjar-artifact/action.yml
vendored
Normal file
26
.github/actions/prepare-uberjar-artifact/action.yml
vendored
Normal file
@@ -0,0 +1,26 @@
|
||||
name: Prepare Uberjar Artifact
|
||||
description: Prepare and upload Metabase JAR as a GitHub artifact.
|
||||
|
||||
inputs:
|
||||
edition:
|
||||
required: false
|
||||
default: "ee"
|
||||
description: Metabase edition (ee or oss)
|
||||
|
||||
runs:
|
||||
using: "composite"
|
||||
steps:
|
||||
- name: Mark with the commit hash
|
||||
run: git rev-parse --short HEAD > COMMIT-ID
|
||||
shell: bash
|
||||
- name: Calculate SHA256 checksum
|
||||
run: sha256sum ./target/uberjar/metabase.jar > SHA256.sum
|
||||
shell: bash
|
||||
- name: Upload JARs as artifact
|
||||
uses: actions/upload-artifact@v7
|
||||
with:
|
||||
name: metabase-${{ inputs.edition }}-${{ github.event.pull_request.head.sha || github.sha }}-uberjar
|
||||
path: |
|
||||
./target/uberjar/metabase.jar
|
||||
./COMMIT-ID
|
||||
./SHA256.sum
|
||||
@@ -25,7 +25,6 @@ runs:
|
||||
TARGET_BRANCH: ${{ github.event.pull_request.base.ref || github.ref_name }}
|
||||
CI_CONDUCTOR_TEST_SUITE: ${{ inputs.test-suite }}
|
||||
ADAPTER: ${{ inputs.adapter }}
|
||||
IS_RERUN: ${{ github.run_attempt != '1' }}
|
||||
# Keep the path constrained to known entrypoints.
|
||||
run: |
|
||||
case "$ADAPTER" in
|
||||
|
||||
100
.github/actions/sdk-bundle-size-check/action.yml
vendored
Normal file
100
.github/actions/sdk-bundle-size-check/action.yml
vendored
Normal file
@@ -0,0 +1,100 @@
|
||||
name: Embedding SDK Bundle Size Check
|
||||
description: |
|
||||
Measures app + embedding SDK bundle sizes from the current and base-ref
|
||||
uberjars (reusing the builds that already ran) plus their bundle-stats
|
||||
artifacts, and flags changes to the chunked SDK's reachable, gzipped size
|
||||
beyond the threshold. Skips when the base-ref artifacts aren't available
|
||||
rather than rebuilding from source (which would compare across pipelines).
|
||||
|
||||
inputs:
|
||||
edition:
|
||||
description: 'Metabase edition (ee or oss)'
|
||||
required: true
|
||||
base-ref-commit:
|
||||
description: 'Commit SHA of the base ref.'
|
||||
required: true
|
||||
threshold-percent:
|
||||
description: 'Percent threshold for bundle size change'
|
||||
required: true
|
||||
|
||||
outputs:
|
||||
status:
|
||||
description: "Bundle size status: increased, decreased, or stable"
|
||||
value: ${{ steps.compare.outputs.status }}
|
||||
size_change_percent:
|
||||
description: "size change percent (positive = increase, negative = decrease)"
|
||||
value: ${{ steps.compare.outputs.size_change_percent }}
|
||||
|
||||
runs:
|
||||
using: "composite"
|
||||
steps:
|
||||
- name: Download current uberjar
|
||||
uses: ./.github/actions/fetch-artifact
|
||||
with:
|
||||
commit: ${{ github.event.pull_request.head.sha || github.sha }}
|
||||
edition: ${{ inputs.edition }}
|
||||
extract-path: temp/current-uberjar
|
||||
- name: Download base-ref uberjar
|
||||
id: base-jar
|
||||
continue-on-error: true
|
||||
uses: ./.github/actions/fetch-artifact
|
||||
with:
|
||||
commit: ${{ inputs.base-ref-commit }}
|
||||
edition: ${{ inputs.edition }}
|
||||
extract-path: temp/base-uberjar
|
||||
- name: Measure and compare bundle sizes
|
||||
id: compare
|
||||
shell: bash
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
EDITION: ${{ inputs.edition }}
|
||||
HEAD_SHA: ${{ github.event.pull_request.head.sha || github.sha }}
|
||||
BASE_SHA: ${{ inputs.base-ref-commit }}
|
||||
THRESHOLD: ${{ inputs.threshold-percent }}
|
||||
MEASURE_BUNDLES: app,embedding-sdk-legacy,embedding-sdk-chunked
|
||||
run: |
|
||||
set -uo pipefail
|
||||
|
||||
skip() {
|
||||
echo "::notice::Embedding SDK bundle-size check skipped: $1"
|
||||
echo "status=stable" >> "$GITHUB_OUTPUT"
|
||||
echo "size_change_percent=0" >> "$GITHUB_OUTPUT"
|
||||
exit 0
|
||||
}
|
||||
|
||||
# The base ref needs an already-built uberjar. Rebuilding it from source
|
||||
# would compare across two build pipelines (the bug this check used to
|
||||
# have), so skip instead when it — or its stats — isn't available.
|
||||
[ -f temp/base-uberjar/metabase.jar ] || skip "base-ref uberjar not found for ${BASE_SHA}"
|
||||
|
||||
extract_js() { # $1=jar $2=dest-root
|
||||
mkdir -p "$2/resources"
|
||||
unzip -o "$1" \
|
||||
"frontend_client/app/dist/*" \
|
||||
"frontend_client/app/embed.js" \
|
||||
"frontend_client/app/embedding-sdk/*" \
|
||||
-d "$2/resources" >/dev/null || true
|
||||
}
|
||||
|
||||
fetch_stats() { # $1=sha $2=dest-dir
|
||||
local name="metabase-bundle-stats-${EDITION}-$1"
|
||||
local id
|
||||
id=$(gh api "/repos/${GITHUB_REPOSITORY}/actions/artifacts?name=${name}" --jq '.artifacts[0].id // empty' 2>/dev/null)
|
||||
[ -n "${id}" ] || return 1
|
||||
mkdir -p "$2"
|
||||
gh api "/repos/${GITHUB_REPOSITORY}/actions/artifacts/${id}/zip" > "$2.zip" 2>/dev/null || return 1
|
||||
unzip -o "$2.zip" -d "$2" >/dev/null || return 1
|
||||
}
|
||||
|
||||
extract_js temp/current-uberjar/metabase.jar temp/current
|
||||
extract_js temp/base-uberjar/metabase.jar temp/base
|
||||
|
||||
fetch_stats "${HEAD_SHA}" temp/current/artifacts || skip "bundle stats not found for current ${HEAD_SHA}"
|
||||
fetch_stats "${BASE_SHA}" temp/base/artifacts || skip "bundle stats not found for base ${BASE_SHA}"
|
||||
|
||||
MEASURE_ROOT=temp/current node ./.github/scripts/measure-bundle-sizes.js > temp/current-sizes.json \
|
||||
|| { echo "::error::Failed to measure current bundle sizes"; exit 1; }
|
||||
MEASURE_ROOT=temp/base node ./.github/scripts/measure-bundle-sizes.js > temp/base-sizes.json \
|
||||
|| { echo "::error::Failed to measure base bundle sizes"; exit 1; }
|
||||
|
||||
node ./.github/actions/sdk-bundle-size-check/compare.js temp/current-sizes.json temp/base-sizes.json
|
||||
110
.github/actions/sdk-bundle-size-check/compare.js
vendored
Normal file
110
.github/actions/sdk-bundle-size-check/compare.js
vendored
Normal file
@@ -0,0 +1,110 @@
|
||||
/* eslint-disable import/no-commonjs */
|
||||
// Diffs the bundle sizes produced by measure-bundle-sizes.js for the current
|
||||
// build and the base ref. Every (bundle, kind) is compared strictly against the
|
||||
// same (bundle, kind) — never one bundle type against another — using gzipped
|
||||
// (served) bytes. The gate is driven by the chunked SDK's reachable total, which
|
||||
// is what the SDK actually delivers.
|
||||
//
|
||||
// compareBundles is pure (data in, data out) so it can be unit-tested; main()
|
||||
// is the thin I/O wrapper that reads the size files, prints the report and sets
|
||||
// the step outputs / exit code.
|
||||
const fs = require("fs");
|
||||
|
||||
const pick = (rows, bundle, kind) => rows.find(row => row.bundle === bundle && row.kind === kind);
|
||||
const keyOf = row => `${row.bundle}/${row.kind}`;
|
||||
const mb = bytes => (bytes / 1024 / 1024).toFixed(2);
|
||||
const percentOf = (currentBytes, baseBytes) =>
|
||||
baseBytes ? Math.trunc(((currentBytes - baseBytes) * 100) / baseBytes) : 0;
|
||||
|
||||
/**
|
||||
* Compare two measurement sets and decide the gate. Returns a human-readable
|
||||
* `report` (always), and exactly one of:
|
||||
* - `gate` { status, percent }: the chunked SDK total moved by `percent`.
|
||||
* - `skip` reason: the gate is non-actionable (report stable, no comment).
|
||||
* - `error` reason: the gate inputs are missing (hard failure).
|
||||
*/
|
||||
function compareBundles({ current, base, threshold }) {
|
||||
const report = ["=== Bundle sizes (gzipped, current vs base) ==="];
|
||||
const keys = [...new Set([...current, ...base].map(keyOf))].sort();
|
||||
for (const key of keys) {
|
||||
const [bundle, kind] = key.split("/");
|
||||
const currentRow = pick(current, bundle, kind);
|
||||
const baseRow = pick(base, bundle, kind);
|
||||
if (!currentRow || !baseRow) {
|
||||
report.push(`${key}: present only in ${currentRow ? "current" : "base"} build`);
|
||||
continue;
|
||||
}
|
||||
const diff = currentRow.gzipBytes - baseRow.gzipBytes;
|
||||
const percent = percentOf(currentRow.gzipBytes, baseRow.gzipBytes);
|
||||
report.push(`${key}: ${mb(currentRow.gzipBytes)}MB vs ${mb(baseRow.gzipBytes)}MB (${percent}%, ${diff} bytes)`);
|
||||
}
|
||||
|
||||
// Gate on the chunked SDK's reachable total (gzipped) — what the SDK ships.
|
||||
const gateCurrent = pick(current, "embedding-sdk-chunked", "total");
|
||||
const gateBase = pick(base, "embedding-sdk-chunked", "total");
|
||||
if (!gateCurrent || !gateBase || !gateBase.gzipBytes) {
|
||||
return { report, error: "Could not find embedding-sdk-chunked total (gzip) in both builds" };
|
||||
}
|
||||
|
||||
// Both sides must measure "total" the same way. A base ref built before the
|
||||
// reachableAssets enrichment collapses its total to the initial set, so
|
||||
// comparing it against the current reachable total reports a phantom jump
|
||||
// (~30%). Skip until the base ref also carries reachable stats (i.e. once
|
||||
// this change is on the base).
|
||||
if (Boolean(gateCurrent.reachable) !== Boolean(gateBase.reachable)) {
|
||||
return {
|
||||
report,
|
||||
skip:
|
||||
`chunked "total" measured differently on each side ` +
|
||||
`(current reachable=${Boolean(gateCurrent.reachable)}, base reachable=${Boolean(gateBase.reachable)}); ` +
|
||||
`the base ref predates the reachable-chunk stats. Resolves once this change is on the base ref.`,
|
||||
};
|
||||
}
|
||||
|
||||
const percent = percentOf(gateCurrent.gzipBytes, gateBase.gzipBytes);
|
||||
const status = percent > threshold ? "increased" : percent < -threshold ? "decreased" : "stable";
|
||||
return { report, gate: { status, percent } };
|
||||
}
|
||||
|
||||
module.exports = { compareBundles };
|
||||
|
||||
const setOutput = (name, value) => {
|
||||
if (process.env.GITHUB_OUTPUT) {
|
||||
fs.appendFileSync(process.env.GITHUB_OUTPUT, `${name}=${value}\n`);
|
||||
}
|
||||
};
|
||||
|
||||
function main() {
|
||||
const [, , currentPath, basePath] = process.argv;
|
||||
const threshold = Number(process.env.THRESHOLD ?? 2);
|
||||
|
||||
const current = JSON.parse(fs.readFileSync(currentPath, "utf8"));
|
||||
const base = JSON.parse(fs.readFileSync(basePath, "utf8"));
|
||||
|
||||
const { report, gate, skip, error } = compareBundles({ current, base, threshold });
|
||||
console.log(report.join("\n"));
|
||||
|
||||
if (error) {
|
||||
console.error(`::error::${error}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
if (skip) {
|
||||
// Treat the gate as non-actionable: report stable so no regression comment
|
||||
// is posted, mirroring the shell skip() for missing artifacts.
|
||||
console.log(`::notice::Embedding SDK bundle-size gate skipped: ${skip}`);
|
||||
setOutput("status", "stable");
|
||||
setOutput("size_change_percent", "0");
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
console.log("");
|
||||
console.log(`Gate: embedding-sdk-chunked total ${gate.percent}% (threshold ${threshold}%) → ${gate.status}`);
|
||||
|
||||
setOutput("status", gate.status);
|
||||
setOutput("size_change_percent", gate.percent);
|
||||
}
|
||||
|
||||
if (require.main === module) {
|
||||
main();
|
||||
}
|
||||
65
.github/actions/sdk-bundle-size-check/compare.unit.spec.js
vendored
Normal file
65
.github/actions/sdk-bundle-size-check/compare.unit.spec.js
vendored
Normal file
@@ -0,0 +1,65 @@
|
||||
const { compareBundles } = require("./compare");
|
||||
|
||||
const total = (gzipBytes, over = {}) => ({
|
||||
bundle: "embedding-sdk-chunked",
|
||||
kind: "total",
|
||||
gzipBytes,
|
||||
reachable: true,
|
||||
...over,
|
||||
});
|
||||
|
||||
describe("compareBundles", () => {
|
||||
it("gates on the chunked SDK total and flags an increase past the threshold", () => {
|
||||
const { gate, report } = compareBundles({
|
||||
current: [total(110)],
|
||||
base: [total(100)],
|
||||
threshold: 2,
|
||||
});
|
||||
|
||||
expect(gate).toEqual({ status: "increased", percent: 10 });
|
||||
expect(report[0]).toContain("current vs base");
|
||||
});
|
||||
|
||||
it("flags a decrease and stays stable within the threshold", () => {
|
||||
expect(compareBundles({ current: [total(90)], base: [total(100)], threshold: 2 }).gate).toEqual({
|
||||
status: "decreased",
|
||||
percent: -10,
|
||||
});
|
||||
expect(compareBundles({ current: [total(101)], base: [total(100)], threshold: 2 }).gate).toEqual({
|
||||
status: "stable",
|
||||
percent: 1,
|
||||
});
|
||||
});
|
||||
|
||||
it("skips when the two sides measured total differently (reachable mismatch)", () => {
|
||||
const result = compareBundles({
|
||||
current: [total(130, { reachable: true })],
|
||||
base: [total(100, { reachable: false })],
|
||||
threshold: 2,
|
||||
});
|
||||
|
||||
expect(result.gate).toBeUndefined();
|
||||
expect(result.skip).toContain("predates the reachable-chunk stats");
|
||||
});
|
||||
|
||||
it("errors when the gate bundle is missing on either side", () => {
|
||||
const result = compareBundles({
|
||||
current: [{ bundle: "app", kind: "total", gzipBytes: 1 }],
|
||||
base: [total(100)],
|
||||
threshold: 2,
|
||||
});
|
||||
|
||||
expect(result.gate).toBeUndefined();
|
||||
expect(result.error).toContain("embedding-sdk-chunked total");
|
||||
});
|
||||
|
||||
it("reports a bundle/kind present on only one side without erroring", () => {
|
||||
const { report } = compareBundles({
|
||||
current: [total(100), { bundle: "app", kind: "total", gzipBytes: 50 }],
|
||||
base: [total(100)],
|
||||
threshold: 2,
|
||||
});
|
||||
|
||||
expect(report.some(line => line.includes("present only in current build"))).toBe(true);
|
||||
});
|
||||
});
|
||||
131
.github/actions/test-driver/action.yml
vendored
131
.github/actions/test-driver/action.yml
vendored
@@ -9,26 +9,20 @@ inputs:
|
||||
build-static-viz:
|
||||
required: true
|
||||
default: false
|
||||
artifact-suffix:
|
||||
logs-artifact-suffix:
|
||||
required: false
|
||||
description: >-
|
||||
Discriminator appended to this job's artifact names (test logs, granular-rerun failed
|
||||
tests). A job whose matrix runs several concurrent legs MUST set a per-leg-unique value,
|
||||
or the legs collide on one artifact — for the rerun artifact that means a leg restores
|
||||
another leg's failed set. Must be stable across attempts of the same job.
|
||||
description: Suffix to append to the test logs artifact name to avoid conflicts in matrix jobs
|
||||
edition:
|
||||
required: false
|
||||
default: "ee"
|
||||
description: Edition to test (oss or ee)
|
||||
quarantine-engine:
|
||||
required: false
|
||||
granular-rerun:
|
||||
driver-status:
|
||||
required: false
|
||||
default: "true"
|
||||
default: ""
|
||||
description: >-
|
||||
When "true" (default), a re-run of a failed job runs only the tests that failed on the
|
||||
previous attempt (hawk `:only`). Stress workflows set this to "false": they intentionally
|
||||
run a fixed test set `:times N`, and narrowing to failures would drop that repetition.
|
||||
Configured status for this driver ("info", "skip", "required", or empty), as reported by `mage driver-decisions`.
|
||||
|
||||
outputs:
|
||||
test-outcome:
|
||||
@@ -54,88 +48,16 @@ runs:
|
||||
env:
|
||||
MB_EDITION: ${{ inputs.edition }}
|
||||
|
||||
# Granular rerun: on a re-run of a failed job, restore the list
|
||||
# of tests that failed on the previous attempt (uploaded below) so the test
|
||||
# step can target only those instead of the whole suite. Best-effort: a
|
||||
# missing/empty artifact leaves no file and the run falls back to the full
|
||||
# suite. First attempts skip this entirely.
|
||||
#
|
||||
# Scoped to the IMMEDIATELY preceding attempt: an attempt that passed, or
|
||||
# that failed with nothing attributable to a var, uploads no artifact, and a
|
||||
# narrow set must not outlive it. Reading `attempt - 1` means "only what the
|
||||
# last attempt actually recorded" — anything else falls back to the full suite.
|
||||
# Workflow expressions have no arithmetic, so the shell computes it.
|
||||
#
|
||||
# run-tests.yml calls app-db.yml, backend.yml, drivers.yml and semantic-search.yml
|
||||
# as part of ONE run, so all four share an artifact namespace. `junit-name` alone is not
|
||||
# unique there — app-db.yml and drivers.yml both derive `be-tests-postgres-ee` — so
|
||||
# `artifact-suffix` carries the rest of the discriminator.
|
||||
- name: Resolve rerun artifact names
|
||||
if: ${{ inputs.granular-rerun == 'true' }}
|
||||
shell: bash
|
||||
env:
|
||||
RERUN_ARTIFACT_BASE: ${{ format('failed-tests-{0}{1}', inputs.junit-name, inputs.artifact-suffix && format('-{0}', inputs.artifact-suffix) || '') }}
|
||||
run: |
|
||||
echo "RERUN_ARTIFACT=$RERUN_ARTIFACT_BASE-attempt-$GITHUB_RUN_ATTEMPT" >> "$GITHUB_ENV"
|
||||
echo "RERUN_PREV_ARTIFACT=$RERUN_ARTIFACT_BASE-attempt-$((GITHUB_RUN_ATTEMPT - 1))" >> "$GITHUB_ENV"
|
||||
|
||||
# `pattern` (not `name`) because the artifact is legitimately absent whenever the
|
||||
# previous attempt recorded nothing: `name` treats that as an error, `pattern`
|
||||
# downloads nothing and moves on. The value is a literal — no glob metacharacters —
|
||||
# so it matches exactly one artifact, which lands at `target/failed-tests`. A pattern
|
||||
# that ever matched two would nest them under their artifact names instead, leaving
|
||||
# the file absent and falling back to the full suite.
|
||||
- name: Restore previously-failed tests
|
||||
if: ${{ inputs.granular-rerun == 'true' && github.run_attempt != '1' }}
|
||||
continue-on-error: true
|
||||
uses: actions/download-artifact@v8
|
||||
with:
|
||||
pattern: ${{ env.RERUN_PREV_ARTIFACT }}
|
||||
path: target
|
||||
|
||||
- name: Test database driver
|
||||
# On a rerun with a non-empty failed-tests file, target only those vars via
|
||||
# hawk `:only` (a var-level selector bypasses `:only-tags`, which is fine —
|
||||
# every listed var already passed that filter on the first attempt).
|
||||
# Otherwise run exactly as before. Quote the file so the whole `[ns/var ...]`
|
||||
# vector reaches clojure -X as a single argument.
|
||||
run: |
|
||||
BASE="clojure -X:dev:ci:${{ inputs.edition }}:${{ inputs.edition }}-dev:drivers:drivers-dev:test"
|
||||
# `:profile-tests` overrides `:test`'s exec-fn, so it has to come last. See dev.profile-tests.
|
||||
# The `profiled` output is what the collection steps below key off, so that the decision is
|
||||
# recorded by the step that acts on it rather than re-derived from the environment.
|
||||
if [ "$PROFILE_TESTS" = "true" ]; then
|
||||
echo "Profiling this run; a flamegraph will be attached to the job."
|
||||
echo "profiled=true" >> "$GITHUB_OUTPUT"
|
||||
BASE="$BASE:profile-tests"
|
||||
fi
|
||||
# An explicit selection wins over the rerun narrowing below: it was asked for by hand, on this
|
||||
# attempt. Names may be separated by commas or spaces; hawk wants them as one vector argument.
|
||||
if [ -n "$ONLY_TESTS" ]; then
|
||||
ONLY="[$(echo "$ONLY_TESTS" | tr ',' ' ')]"
|
||||
echo "Running only: $ONLY"
|
||||
$BASE :only "$ONLY"
|
||||
elif [ "${{ inputs.granular-rerun }}" = "true" ] && [ "${{ github.run_attempt }}" != "1" ] && [ -s target/failed-tests ]; then
|
||||
echo "Granular rerun of previously-failed tests: $(cat target/failed-tests)"
|
||||
$BASE :only "$(cat target/failed-tests)"
|
||||
else
|
||||
$BASE ${{ inputs.test-args }}
|
||||
fi
|
||||
run: clojure -X:dev:ci:${{ inputs.edition }}:${{ inputs.edition }}-dev:drivers:drivers-dev:test ${{ inputs.test-args }}
|
||||
shell: bash
|
||||
id: run-test
|
||||
# Keep this step continue-on-error because the quarantine gate below controls whether it
|
||||
# ultimately fails.
|
||||
# Keep this step continue-on-error because the quarantine logic controls whether it ultimately fails.
|
||||
# Driver "info" status can affect this logic. See below.
|
||||
continue-on-error: true
|
||||
|
||||
- name: Collect profile
|
||||
if: ${{ !cancelled() }}
|
||||
uses: ./.github/actions/collect-profile
|
||||
with:
|
||||
profiled: ${{ steps.run-test.outputs.profiled }}
|
||||
artifact-name: ${{ format('profile-{0}{1}', inputs.junit-name, inputs.artifact-suffix && format('-{0}', inputs.artifact-suffix) || '') }}
|
||||
|
||||
- name: Report failed tests to ci-conductor
|
||||
if: ${{ !cancelled() && ( steps.run-test.outcome == 'failure' || github.run_attempt != '1' ) }}
|
||||
if: ${{ !cancelled() && steps.run-test.outcome == 'failure' }}
|
||||
# This step should NEVER fail the job!
|
||||
continue-on-error: true
|
||||
uses: ./.github/actions/report-failures-to-ci-conductor
|
||||
@@ -143,30 +65,6 @@ runs:
|
||||
test-suite: ${{ inputs.junit-name }}
|
||||
adapter: backend
|
||||
|
||||
# Granular rerun: record which tests failed so a re-run of this
|
||||
# job targets only those (restored above). Reuses the backend JUnit adapter
|
||||
# from the ci-conductor sources the report step just prepared. Best-effort:
|
||||
# writes nothing unless every failure is attributable to a var, so an
|
||||
# unattributable failure falls back to a full rerun.
|
||||
- name: Collect failed tests for rerun
|
||||
if: ${{ inputs.granular-rerun == 'true' && !cancelled() && steps.run-test.outcome == 'failure' }}
|
||||
continue-on-error: true
|
||||
shell: bash
|
||||
# Clear the restored set first, so a collector that dies before deciding
|
||||
# can't leave the previous attempt's file to be uploaded as this one's.
|
||||
run: |
|
||||
rm -f target/failed-tests
|
||||
bun release/ci-conductor/src/collect-failed-backend.ts
|
||||
|
||||
- name: Upload failed tests for rerun
|
||||
if: ${{ inputs.granular-rerun == 'true' && !cancelled() && steps.run-test.outcome == 'failure' }}
|
||||
continue-on-error: true
|
||||
uses: actions/upload-artifact@v7
|
||||
with:
|
||||
name: ${{ env.RERUN_ARTIFACT }}
|
||||
path: target/failed-tests
|
||||
if-no-files-found: ignore
|
||||
|
||||
- name: Publish Test Report (JUnit)
|
||||
uses: dorny/test-reporter@a43b3a5f7366b97d083190328d2c652e1a8b6aa2 # v3.0.0
|
||||
# As per the documentation: https://docs.github.com/en/actions/reference/workflows-and-actions/contexts#steps-context
|
||||
@@ -185,7 +83,7 @@ runs:
|
||||
if: always() && steps.run-test.outcome != 'success'
|
||||
uses: actions/upload-artifact@v7
|
||||
with:
|
||||
name: ${{ format('test-logs-{0}{1}', github.job, inputs.artifact-suffix && format('-{0}', inputs.artifact-suffix) || '') }}
|
||||
name: ${{ format('test-logs-{0}{1}', github.job, inputs.logs-artifact-suffix && format('-{0}', inputs.logs-artifact-suffix) || '') }}
|
||||
path: logs/test-log.json
|
||||
retention-days: 1
|
||||
# Failing to upload the artifact should never affect the outcome of the whole test run.
|
||||
@@ -194,7 +92,14 @@ runs:
|
||||
# Last, because this is the only step allowed to fail the job: everything above it reports the
|
||||
# failure (JUnit report, logs, ci-conductor) and must run regardless of the verdict below.
|
||||
- name: Quarantine gate
|
||||
if: ${{ !cancelled() && steps.run-test.outcome == 'failure' }}
|
||||
# Off master and release branches an "info" driver is data-collection only - the test always passes!
|
||||
# On master and release branches "info" carries no weight - the quarantine logic applies.
|
||||
if: |
|
||||
!cancelled()
|
||||
&& steps.run-test.outcome == 'failure'
|
||||
&& !(inputs.driver-status == 'info'
|
||||
&& !(github.ref_name == 'master' || startsWith(github.ref_name, 'release-x')))
|
||||
continue-on-error: ${{ inputs.quarantine-engine != 'ci-conductor' }}
|
||||
uses: ./.github/actions/quarantine-gate
|
||||
with:
|
||||
adapter: backend
|
||||
|
||||
41
.github/actions/update-cache/action.yml
vendored
Normal file
41
.github/actions/update-cache/action.yml
vendored
Normal file
@@ -0,0 +1,41 @@
|
||||
name: Update cache
|
||||
description: |
|
||||
Update a GitHub Actions cache based on a provided path and key.
|
||||
If a cache entry already exists for the given key, it is first removed.
|
||||
This ensures immutability of cache entries, and prevents runtime errors.
|
||||
|
||||
inputs:
|
||||
path:
|
||||
description: "The path(s) to cache, separated by newlines."
|
||||
required: true
|
||||
key:
|
||||
description: "The cache key to use."
|
||||
required: true
|
||||
|
||||
runs:
|
||||
using: "composite"
|
||||
steps:
|
||||
- name: Check if the cache entry exists
|
||||
id: cache-lookup
|
||||
uses: actions/cache/restore@v5
|
||||
with:
|
||||
path: ${{ inputs.path }}
|
||||
key: ${{ inputs.key }}
|
||||
lookup-only: true # Important bit: the action will not fetch anything, just check for the cache existence.
|
||||
- name: Delete the cache entry if it exists
|
||||
if: ${{ steps.cache-lookup.outputs.cache-hit == 'true' }}
|
||||
run: |
|
||||
echo "Cache entry found for key '${{ inputs.key }}'. Deleting it to ensure immutability."
|
||||
gh api \
|
||||
--method DELETE \
|
||||
-H "Accept: application/vnd.github+json" \
|
||||
-H "X-GitHub-Api-Version: 2022-11-28" \
|
||||
/repos/${{ github.repository }}/actions/caches?key=${{ inputs.key }}
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
shell: bash
|
||||
- name: Save the new cache entry
|
||||
uses: actions/cache/save@v5
|
||||
with:
|
||||
path: ${{ inputs.path }}
|
||||
key: ${{ inputs.key }}
|
||||
@@ -1,5 +1,4 @@
|
||||
name: Create Backend Dep Bump PRs
|
||||
description: Opens a pull request for each backend dependency that has a newer version available.
|
||||
|
||||
inputs:
|
||||
github-token:
|
||||
|
||||
185
.github/actions/upload-test-results/action.yml
vendored
Normal file
185
.github/actions/upload-test-results/action.yml
vendored
Normal file
@@ -0,0 +1,185 @@
|
||||
name: Upload Test Results to s3
|
||||
description: Uploads test results files to s3
|
||||
|
||||
inputs:
|
||||
input-path:
|
||||
required: true
|
||||
output-name:
|
||||
required: true
|
||||
default: test
|
||||
role-to-assume:
|
||||
required: true
|
||||
aws-region:
|
||||
required: true
|
||||
default: us-east-1
|
||||
bucket:
|
||||
required: true
|
||||
trunk-api-token:
|
||||
required: true
|
||||
variant:
|
||||
required: false
|
||||
default: ""
|
||||
previous-step-outcome:
|
||||
required: true
|
||||
driver-status:
|
||||
required: false
|
||||
default: ""
|
||||
description: >-
|
||||
Configured status for this driver ("info", "skip", "required", or empty). When
|
||||
"info", the job is reported as passing regardless of the real test outcome, and
|
||||
that outcome is surfaced via a job summary and (on PRs) a comment instead.
|
||||
report-variant:
|
||||
required: false
|
||||
default: ""
|
||||
description: >-
|
||||
Optional label disambiguating matrix legs of the same job, so the info-driver
|
||||
job summary and PR comment stay distinct per leg.
|
||||
run-trunk:
|
||||
required: false
|
||||
default: "true"
|
||||
description: Whether to run the Trunk step at all.
|
||||
|
||||
runs:
|
||||
using: "composite"
|
||||
steps:
|
||||
- name: Configure AWS credentials
|
||||
uses: aws-actions/configure-aws-credentials@d979d5b3a71173a29b74b5b88418bfda9437d885
|
||||
with:
|
||||
role-to-assume: ${{ inputs.role-to-assume }}
|
||||
aws-region: ${{ inputs.aws-region }}
|
||||
- name: zip test results
|
||||
env:
|
||||
INPUT_DIR: ${{ inputs.input-path }}
|
||||
OUTPUT_FILE: ${{ inputs.output-name }}
|
||||
shell: bash
|
||||
run: | # sh
|
||||
zip ${OUTPUT_FILE}.zip $INPUT_DIR/*.xml
|
||||
- name: Upload test results to S3
|
||||
env:
|
||||
BUCKET: ${{ inputs.bucket }}
|
||||
OUTPUT_FILE: ${{ inputs.output-name }}
|
||||
shell: bash
|
||||
run: | # sh
|
||||
DATE=$(date '+%Y-%m-%d')
|
||||
aws s3 cp ${OUTPUT_FILE}.zip s3://$BUCKET/$DATE/$GITHUB_RUN_ID/$GITHUB_RUN_ATTEMPT/
|
||||
|
||||
- name: Upload results to Trunk
|
||||
if: ${{ always() && github.repository_owner == 'metabase' && inputs.run-trunk == 'true' }}
|
||||
# Important!
|
||||
# For "info" drivers off master/release branches, our own status takes precedence: we let this
|
||||
# step pass so the job is reported as passing regardless of the real outcome (still uploaded).
|
||||
# On master and release branches, info status is ignored and the real outcome gates the job.
|
||||
continue-on-error: ${{ inputs.driver-status == 'info' && !(github.ref_name == 'master' || startsWith(github.ref_name, 'release-x')) }}
|
||||
uses: trunk-io/analytics-uploader@1198efcc0976501147b10a7e343c3fa069527526 # v2.0.9
|
||||
env:
|
||||
TRUNK_REPO_URL: git@github.com:metabase/metabase.git
|
||||
with:
|
||||
junit-paths: ${{ inputs.input-path }}
|
||||
variant: ${{ inputs.variant || inputs.output-name }}
|
||||
org-slug: metabase
|
||||
dry-run: ${{ (github.event_name == 'pull_request' && github.event.pull_request.draft == true) || github.repository != 'metabase/metabase' }} # quarantine, but do not upload results
|
||||
token: ${{ inputs.trunk-api-token }}
|
||||
use-cache: true
|
||||
previous-step-outcome: ${{ inputs.previous-step-outcome }}
|
||||
|
||||
# For "info" drivers the job is forced green above, so surface the real outcome here —
|
||||
# but only when it did NOT pass, to avoid spamming summaries/comments for healthy runs.
|
||||
# Never include logs or stack traces — only the real status and a link to the run.
|
||||
- name: Report info-level driver result (job summary)
|
||||
if: ${{ always() && inputs.driver-status == 'info' && inputs.previous-step-outcome == 'failure' }}
|
||||
shell: bash
|
||||
env:
|
||||
OUTCOME: ${{ inputs.previous-step-outcome }}
|
||||
LABEL: ${{ inputs.output-name }}
|
||||
VARIANT: ${{ inputs.report-variant }}
|
||||
run: | # sh
|
||||
if [ "$OUTCOME" = "success" ]; then EMOJI="✅"; else EMOJI="❌"; fi
|
||||
TITLE="$LABEL"
|
||||
[ -n "$VARIANT" ] && TITLE="$LABEL ($VARIANT)"
|
||||
RUN_URL="$GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID"
|
||||
{
|
||||
echo "### ℹ️ Info-level driver: $TITLE"
|
||||
echo ""
|
||||
echo "Reported as passing (non-blocking). Real result: $EMOJI \`$OUTCOME\`"
|
||||
echo ""
|
||||
echo "[View run]($RUN_URL)"
|
||||
} >> "$GITHUB_STEP_SUMMARY"
|
||||
|
||||
# Maintain ONE PR comment per commit for all info drivers. Each leg upserts its own line
|
||||
# (keyed by a hidden tag); a passing leg removes its line, and the comment is deleted once
|
||||
# no failing legs remain. Comments from older commits are cleaned up. Runs on pass and fail
|
||||
# so recoveries self-heal. Never includes logs or stack traces — only status and a run link.
|
||||
- name: Sync info-level driver result into the PR comment
|
||||
if: ${{ always() && inputs.driver-status == 'info' && github.event_name == 'pull_request' }}
|
||||
uses: actions/github-script@v9
|
||||
continue-on-error: true
|
||||
env:
|
||||
OUTCOME: ${{ inputs.previous-step-outcome }}
|
||||
LABEL: ${{ inputs.output-name }}
|
||||
VARIANT: ${{ inputs.report-variant }}
|
||||
with:
|
||||
script: | # js
|
||||
const { OUTCOME, LABEL, VARIANT, GITHUB_SERVER_URL, GITHUB_REPOSITORY, GITHUB_RUN_ID } = process.env;
|
||||
const pr = context.payload.pull_request;
|
||||
if (!pr) return;
|
||||
const { owner, repo } = context.repo;
|
||||
const sha = pr.head.sha;
|
||||
// PREFIX matches the current per-commit marker AND the older per-leg one, so legacy
|
||||
// comments from a previous version get cleaned up on the next sync.
|
||||
const PREFIX = "<!-- info-driver-result";
|
||||
const marker = `<!-- info-driver-results:${sha} -->`;
|
||||
const header = `${marker}\nℹ️ **Info-level drivers** failed (non-blocking) on \`${sha.slice(0, 7)}\` — these do not gate:`;
|
||||
const lineKey = `${LABEL}:${VARIANT}`;
|
||||
const title = VARIANT ? `${LABEL} (${VARIANT})` : LABEL;
|
||||
const runUrl = `${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}`;
|
||||
// Only real failures are reported; cancelled/skipped/success remove this leg's line.
|
||||
// List-item line with the hidden key tag at the END so markdown (links) renders.
|
||||
const ourLine = OUTCOME === "failure"
|
||||
? `- **${title}** — ❌ \`${OUTCOME}\` ([Logs](${runUrl})) <!--l:${lineKey}-->`
|
||||
: null;
|
||||
const linesOf = (body) => body.split("\n").filter((l) => l.includes("<!--l:"));
|
||||
|
||||
const sync = async () => {
|
||||
const { data: comments } = await github.rest.issues.listComments({
|
||||
owner, repo, issue_number: pr.number, per_page: 100,
|
||||
});
|
||||
const tagged = comments.filter((c) => c.body.includes(PREFIX));
|
||||
// Drop comments left over from older commits.
|
||||
for (const c of tagged.filter((c) => !c.body.includes(marker))) {
|
||||
await github.rest.issues.deleteComment({ owner, repo, comment_id: c.id }).catch(() => {});
|
||||
}
|
||||
const mine = tagged
|
||||
.filter((c) => c.body.includes(marker))
|
||||
.sort((a, b) => new Date(a.created_at) - new Date(b.created_at));
|
||||
// Merge any duplicates raced into existence; latest write of each line wins.
|
||||
const byKey = new Map();
|
||||
for (const c of mine) {
|
||||
for (const l of linesOf(c.body)) {
|
||||
const m = l.match(/<!--l:(.*?)-->/);
|
||||
if (m) byKey.set(m[1], l);
|
||||
}
|
||||
}
|
||||
if (ourLine) byKey.set(lineKey, ourLine);
|
||||
else byKey.delete(lineKey);
|
||||
const lines = [...byKey.values()].sort();
|
||||
const target = mine[0];
|
||||
for (const c of mine.slice(1)) {
|
||||
await github.rest.issues.deleteComment({ owner, repo, comment_id: c.id }).catch(() => {});
|
||||
}
|
||||
if (lines.length === 0) {
|
||||
if (target) await github.rest.issues.deleteComment({ owner, repo, comment_id: target.id }).catch(() => {});
|
||||
return;
|
||||
}
|
||||
const body = [header, "", ...lines].join("\n");
|
||||
if (target) {
|
||||
await github.rest.issues.updateComment({ owner, repo, comment_id: target.id, body });
|
||||
} else {
|
||||
await github.rest.issues.createComment({ owner, repo, issue_number: pr.number, body });
|
||||
}
|
||||
};
|
||||
|
||||
// Retry to shrink the window where parallel legs clobber each other's update.
|
||||
for (let i = 0; i < 4; i++) {
|
||||
try { await sync(); break; }
|
||||
catch (e) { if (i === 3) throw e; await new Promise((r) => setTimeout(r, 400 * (i + 1))); }
|
||||
}
|
||||
22
.github/actions/verify-membership/action.yml
vendored
Normal file
22
.github/actions/verify-membership/action.yml
vendored
Normal file
@@ -0,0 +1,22 @@
|
||||
name: 'Verify Organization Membership'
|
||||
description: 'Check if a user is a member of a GitHub organization'
|
||||
inputs:
|
||||
github_token:
|
||||
description: 'Token with access to the organization'
|
||||
required: true
|
||||
organization_name:
|
||||
description: 'Organization name'
|
||||
required: true
|
||||
username:
|
||||
description: 'Username'
|
||||
required: true
|
||||
outputs:
|
||||
is_member:
|
||||
value: ${{ steps.verify-membership.outputs.is_member }}
|
||||
description: "Is the user a member of the organization"
|
||||
runs:
|
||||
using: "composite"
|
||||
steps:
|
||||
- run: bash ${{ github.action_path }}/verify_membership.sh ${{ inputs.github_token }} ${{ inputs.organization_name }} ${{ inputs.username }}
|
||||
id: verify-membership
|
||||
shell: bash
|
||||
25
.github/actions/verify-membership/verify_membership.sh
vendored
Executable file
25
.github/actions/verify-membership/verify_membership.sh
vendored
Executable file
@@ -0,0 +1,25 @@
|
||||
#!/bin/bash
|
||||
|
||||
TOKEN=$1
|
||||
ORG_NAME=$2
|
||||
USERNAME=$3
|
||||
|
||||
response=$(curl -L -o /dev/null -w "%{http_code}" \
|
||||
-H "Accept: application/vnd.github+json" \
|
||||
-H "Authorization: Bearer $TOKEN" \
|
||||
-H "X-GitHub-Api-Version: 2022-11-28" \
|
||||
"https://api.github.com/orgs/$ORG_NAME/members/$USERNAME")
|
||||
|
||||
# Github returns an empty response with 204 status if user is a member and 404 status otherwise
|
||||
# See: https://docs.github.com/en/rest/orgs/members?apiVersion=2022-11-28#check-organization-membership-for-a-user
|
||||
if [ "$response" == "204" ]; then
|
||||
echo "The user $USERNAME is a member of $ORG_NAME"
|
||||
echo "is_member=true" >> $GITHUB_OUTPUT
|
||||
elif [ "$response" == "404" ]; then
|
||||
echo "The user $USERNAME is not a member of $ORG_NAME"
|
||||
echo "is_member=false" >> $GITHUB_OUTPUT
|
||||
else
|
||||
echo "Failed to determine the membership status for $USERNAME in $ORG_NAME"
|
||||
echo "is_member=false" >> $GITHUB_OUTPUT
|
||||
exit 1
|
||||
fi
|
||||
104
.github/file-paths.yaml
vendored
104
.github/file-paths.yaml
vendored
@@ -7,8 +7,9 @@ default: &default
|
||||
- "locales/**"
|
||||
- "bin/**/!(*.md)"
|
||||
|
||||
workflows:
|
||||
- added|modified: ".github/workflows/*.{yml,yaml}"
|
||||
actionlint:
|
||||
- ".github/workflows/*.yml"
|
||||
- ".github/workflows/*.yaml"
|
||||
|
||||
shared_sources: &shared_sources
|
||||
- "src/**/*.cljc"
|
||||
@@ -53,6 +54,11 @@ frontend_sources: &frontend_sources
|
||||
- "enterprise/frontend/!(test)**/**/{!(*.spec),}.{js,jsx,ts,tsx,css}"
|
||||
- "frontend/!(test)**/**/{!(*.spec),}.{js,jsx,ts,tsx,css}"
|
||||
|
||||
# Used only in .github/workflows/conventional-commit-pr-title-reminder.yml
|
||||
frontend_embedding_sources:
|
||||
- "enterprise/frontend/src/{embedding,embedding-*}/**/{!(*.spec),}.{js,jsx,ts,tsx,css}"
|
||||
- "enterprise/frontend/src/metabase-enterprise/{embedding,embedding-*}**/{!(*.spec),}.{js,jsx,ts,tsx,css}"
|
||||
|
||||
# Used only in .github/workflows/embedding-sdk-labels-reminder.yml
|
||||
frontend_embedding_sdk_package_sources:
|
||||
- "enterprise/frontend/src/embedding-sdk-package/**/!(*.spec).{js,jsx,ts,tsx,css}"
|
||||
@@ -64,9 +70,6 @@ frontend_specs: &frontend_specs
|
||||
- "frontend/**/*.unit.*"
|
||||
- "enterprise/frontend/**/*.unit.*"
|
||||
- ".github/**/*.unit.spec.*"
|
||||
# file-paths.unit.spec.ts asserts what this file's own patterns match, so a change to it has to
|
||||
# run the specs that guard it.
|
||||
- ".github/file-paths.yaml"
|
||||
- "jest.config.js"
|
||||
- "jest.tz.unit.conf.json"
|
||||
|
||||
@@ -108,6 +111,13 @@ backend_ci: &backend_ci
|
||||
- ".github/workflows/backend.yml"
|
||||
- ".github/workflows/drivers.yml"
|
||||
|
||||
backend_presto_kerberos:
|
||||
- "**/presto_jdbc/**"
|
||||
- "**/presto-jdbc/**" # the driver module has a hyphen. underscores are classpaths
|
||||
- "**/presto_jdbc.clj"
|
||||
- "**/presto_jdbc_test.clj"
|
||||
- ".github/workflows/presto-kerberos-integration-test.yml"
|
||||
|
||||
backend_sources: &backend_sources
|
||||
- *shared_sources
|
||||
- "**deps.edn"
|
||||
@@ -115,15 +125,11 @@ backend_sources: &backend_sources
|
||||
- "src/**"
|
||||
- "modules/drivers/{*,*/{*,!(test)/**}}"
|
||||
- "resources/**"
|
||||
# The ratchets file drives project_ratchet_checks alone, so the automation PR that lowers the budgets
|
||||
# runs the Project tests job by itself. Patterns are OR-ed, so a separate "!..." entry would match
|
||||
# every other path instead of excluding this one.
|
||||
- ".clj-kondo/{!(ratchets.edn),*/**}"
|
||||
- ".clj-kondo/**"
|
||||
|
||||
backend_specs: &backend_specs
|
||||
- *shared_specs
|
||||
- "test/**"
|
||||
- "dev/test/**"
|
||||
- "enterprise/backend/test/**"
|
||||
- "test_modules/**"
|
||||
- "modules/drivers/*/test/**"
|
||||
@@ -136,45 +142,6 @@ backend_all: &backend_all
|
||||
- *backend_sources
|
||||
- *backend_specs
|
||||
|
||||
# Project checks have narrower triggers than the backend matrix.
|
||||
# Changes to their shared CI entry points exercise every suite; otherwise each
|
||||
# suite follows only the files that can affect its result.
|
||||
project_tests_infra: &project_tests_infra
|
||||
- ".github/actions/prepare-backend/**"
|
||||
- ".github/file-paths.yaml"
|
||||
- ".github/workflows/backend.yml"
|
||||
- ".github/workflows/run-tests.yml"
|
||||
- "bb.edn"
|
||||
- "bin/mage"
|
||||
- "bin/mage.bb"
|
||||
- "mage/src/mage/color.clj"
|
||||
- "mage/src/mage/project_tests.clj"
|
||||
- "mage/src/mage/shell.clj"
|
||||
- "mage/src/mage/util.clj"
|
||||
|
||||
project_backend_checks:
|
||||
- *project_tests_infra
|
||||
- "{src,test,enterprise,dev,bin,mage}/**/*.{clj,cljc,cljs}"
|
||||
- "modules/**/*.clj"
|
||||
- "modules/**/*.cljc"
|
||||
- "modules/**/*.cljs"
|
||||
- "**deps.edn"
|
||||
- ".clj-kondo/config.edn"
|
||||
- ".clj-kondo/config/modules/config.edn"
|
||||
- ".github/team.json"
|
||||
- "dev/resources/dev/modules_config_test/**"
|
||||
|
||||
# Only the ratchet check reads the budgets, so a diff that touches nothing else runs that suite alone.
|
||||
project_ratchet_checks:
|
||||
- ".clj-kondo/ratchets.edn"
|
||||
|
||||
project_migration_checks:
|
||||
- *project_tests_infra
|
||||
- "bin/lint-migrations-file/**"
|
||||
- "resources/migrations/**/*.{yaml,yml}"
|
||||
- "src/metabase/app_db/custom_migrations.clj"
|
||||
- "src/metabase/app_db/custom_migrations/**"
|
||||
|
||||
sources: &sources
|
||||
- *shared_sources
|
||||
- *frontend_sources
|
||||
@@ -188,7 +155,7 @@ custom_viz_all:
|
||||
|
||||
e2e_ci: &e2e_ci
|
||||
- *shared_ci
|
||||
- ".github/workflows/uberjar.yml"
|
||||
- ".github/actions/prepare-uberjar-artifact/**"
|
||||
- ".github/actions/fetch-artifact/action.yml"
|
||||
- ".github/actions/e2e-prepare-containers/**"
|
||||
- ".github/actions/prepare-cypress/**"
|
||||
@@ -199,7 +166,7 @@ e2e_specs: &e2e_specs
|
||||
|
||||
embedding_sdk_ci: &embedding_sdk_ci
|
||||
- *shared_ci
|
||||
- ".github/workflows/uberjar.yml"
|
||||
- ".github/actions/prepare-uberjar-artifact/**"
|
||||
- ".github/actions/fetch-artifact/action.yml"
|
||||
- ".github/actions/prepare-cypress/**"
|
||||
- ".github/workflows/embedding-sdk*.yml"
|
||||
@@ -221,7 +188,7 @@ embedding_sdk_host_sample_apps:
|
||||
- "e2e/test-host-app/**/*.cy.*.{js,ts}"
|
||||
|
||||
snowplow: &snowplow
|
||||
- ".github/workflows/lint.yml"
|
||||
- ".github/workflows/snowplow.yml"
|
||||
- "snowplow/**"
|
||||
- "bin/lint-snowplow-schemas.bb"
|
||||
- "bin/lint-snowplow-schemas-test.bb"
|
||||
@@ -250,7 +217,7 @@ yaml:
|
||||
- "**/*.yaml"
|
||||
|
||||
codeql:
|
||||
- ".github/workflows/lint.yml"
|
||||
- ".github/workflows/codeql.yml"
|
||||
- "frontend/src/**"
|
||||
- "enterprise/frontend/src/**"
|
||||
|
||||
@@ -258,43 +225,26 @@ i18n:
|
||||
- *default
|
||||
- ".github/actions/prepare-frontend/**"
|
||||
- ".github/actions/prepare-backend/**"
|
||||
- ".github/workflows/lint.yml"
|
||||
- ".github/workflows/i18n.yml"
|
||||
- *sources
|
||||
|
||||
visualizations:
|
||||
- "frontend/src/metabase/visualizations/**"
|
||||
- "frontend/src/metabase/static-viz/**"
|
||||
- "src/metabase/pulse/**"
|
||||
|
||||
cljs:
|
||||
- *shared_sources
|
||||
- *shared_specs
|
||||
- ".github/workflows/cljs.yml"
|
||||
- "shadow-cljs.edn"
|
||||
|
||||
embedder_plugin:
|
||||
- ".github/actions/prepare-backend/**"
|
||||
- ".github/file-paths.yaml"
|
||||
- ".github/workflows/build-scripts.yml"
|
||||
- "bin/build-embedder-plugin.sh"
|
||||
- "bin/build/src/build/embedder_model.clj"
|
||||
- "bin/build/src/build/plugin_uberjar.clj"
|
||||
- "bin/build/src/build_embedder_plugin.clj"
|
||||
- "deps.edn"
|
||||
- "modules/embedder/**"
|
||||
- "enterprise/backend/src/metabase_enterprise/data_complexity_score/complexity_embedders.clj"
|
||||
- "enterprise/backend/src/metabase_enterprise/data_complexity_score/settings.clj"
|
||||
- "enterprise/backend/src/metabase_enterprise/data_complexity_score/synonym_source.clj"
|
||||
- "enterprise/backend/src/metabase_enterprise/semantic_search/embedding.clj"
|
||||
- "enterprise/backend/src/metabase_enterprise/semantic_search/settings.clj"
|
||||
- "enterprise/backend/test/metabase_enterprise/semantic_search/appdb_pgvector_mode_test.clj"
|
||||
- "enterprise/backend/test/metabase_enterprise/semantic_search/embedding_test.clj"
|
||||
- "src/metabase/embeddings/provider.clj"
|
||||
- "src/metabase/embeddings/startup.clj"
|
||||
- "src/metabase/plugins/**"
|
||||
- "test/metabase/embeddings/embedder_plugin_artifact_test.clj"
|
||||
|
||||
serialization_baseline:
|
||||
- ".github/workflows/serialization.yml"
|
||||
- "test_resources/serialization_baseline/**"
|
||||
|
||||
openapi:
|
||||
- ".github/workflows/lint.yml"
|
||||
- ".github/workflows/openapi.yml"
|
||||
- "src/metabase/api/**"
|
||||
- "src/metabase/api_routes/**"
|
||||
- "src/metabase/*_rest/api.clj"
|
||||
|
||||
12
.github/instructions/e2e.instructions.md
vendored
12
.github/instructions/e2e.instructions.md
vendored
@@ -1,12 +0,0 @@
|
||||
---
|
||||
applyTo: "e2e/**/*.cy.spec.{js,ts}"
|
||||
---
|
||||
|
||||
# Cypress E2E Spec Review
|
||||
|
||||
When reviewing Cypress E2E specs, apply the guidelines in these files:
|
||||
|
||||
- `.claude/skills/_shared/cypress-conventions.md`
|
||||
- `.claude/skills/e2e-test-review/SKILL.md`
|
||||
|
||||
These are in addition to the TypeScript/JavaScript instructions, which also match these files.
|
||||
12
.github/scripts/affected-modules.ts
vendored
12
.github/scripts/affected-modules.ts
vendored
@@ -1,12 +1,7 @@
|
||||
import micromatch from "micromatch";
|
||||
|
||||
export type ModuleDef = { type: string; pattern: string };
|
||||
|
||||
export type Rule = {
|
||||
from: string[];
|
||||
allow?: string[];
|
||||
disallow?: string[];
|
||||
};
|
||||
export type Rule = { from: string[]; allow: string[] };
|
||||
|
||||
export type ModuleNode = { type: string; regex: RegExp };
|
||||
|
||||
@@ -208,11 +203,8 @@ export function buildModuleGraph(
|
||||
allTypes.map((type) => [type, new Set()]),
|
||||
);
|
||||
for (const rule of rules) {
|
||||
// Grandfathered imports that break the disallow rules still exist in the code,
|
||||
// and selection must cover every import that actually exists.
|
||||
// So disallow rules are ignored, at the cost of a looser graph.
|
||||
const fromTypes = rule.from.flatMap(expandPattern);
|
||||
const allowTypes = (rule.allow ?? []).flatMap(expandPattern);
|
||||
const allowTypes = rule.allow.flatMap(expandPattern);
|
||||
for (const target of allowTypes) {
|
||||
for (const importer of fromTypes) {
|
||||
if (importer !== target) {
|
||||
|
||||
152
.github/scripts/bundle-size-stats-prepare.js
vendored
Normal file
152
.github/scripts/bundle-size-stats-prepare.js
vendored
Normal file
@@ -0,0 +1,152 @@
|
||||
/* eslint-disable import/no-commonjs */
|
||||
// Prepares the rows the bundle-size stats logger uploads to stats.metabase.com.
|
||||
//
|
||||
// For each measured (bundle, kind) it computes the delta — raw, gzipped and
|
||||
// brotli (as-served) bytes plus percent — against the previously *plotted* data
|
||||
// point (restored from a rolling cache), so the deltas can be charted directly.
|
||||
// It also decides whether this commit is worth recording at all: a point is kept
|
||||
// when the as-served size moved by at least MIN_DELTA_PERCENT, or when a new
|
||||
// bundle/kind appears. Only master builds run through here — releases form
|
||||
// non-linear lines whose deltas are meaningless, so they're not recorded (their
|
||||
// historical sizes are backfilled separately).
|
||||
//
|
||||
// buildStatsRows is pure (data in, data out) so it can be unit-tested; main()
|
||||
// is the thin I/O wrapper that reads env/files and writes the rows, cache and
|
||||
// step outputs.
|
||||
const fs = require("fs");
|
||||
const path = require("path");
|
||||
|
||||
const deltaPercent = (current, base) =>
|
||||
base ? Math.round(((current - base) * 10000) / base) / 100 : "";
|
||||
|
||||
/**
|
||||
* Turn the current measurements + the last plotted point into upload rows, the
|
||||
* slim cache rows for the next diff, and the keep/skip decision.
|
||||
*
|
||||
* The significance threshold tracks the as-served size: brotli where the build
|
||||
* ships a precompressed .br, otherwise gzip. Current measurements always carry
|
||||
* brotliBytes (estimated when no .br shipped), but a cached base point recorded
|
||||
* before brotli logging may not, so the delta still falls back to gzip per row.
|
||||
*/
|
||||
function buildStatsRows({ measurements, previous, threshold, date, commit, commitMessage, version }) {
|
||||
const previousOf = (bundle, kind) => previous.find(row => row.bundle === bundle && row.kind === kind);
|
||||
|
||||
let maxServedDeltaPercent = 0;
|
||||
let hasNewSeries = false;
|
||||
|
||||
const rows = measurements.map(measurement => {
|
||||
const base = previousOf(measurement.bundle, measurement.kind);
|
||||
const hasBrotli = measurement.brotliBytes != null && base?.brotliBytes != null;
|
||||
const gzipDeltaPercent = deltaPercent(measurement.gzipBytes, base?.gzipBytes);
|
||||
const brotliDeltaPercent = hasBrotli ? deltaPercent(measurement.brotliBytes, base.brotliBytes) : "";
|
||||
if (!base) {
|
||||
hasNewSeries = true;
|
||||
} else {
|
||||
const servedDeltaPercent = hasBrotli ? brotliDeltaPercent : gzipDeltaPercent;
|
||||
maxServedDeltaPercent = Math.max(maxServedDeltaPercent, Math.abs(servedDeltaPercent));
|
||||
}
|
||||
return {
|
||||
"Date": date,
|
||||
"Version": version,
|
||||
"Commit": commit,
|
||||
"Bundle": measurement.bundle,
|
||||
"Kind": measurement.kind, // "initial" or "total"
|
||||
// The stats table carries a free-text Description column; append-csv
|
||||
// requires every table column to be present. Populate it with the commit
|
||||
// subject so the chart's points are self-describing.
|
||||
"Description": commitMessage,
|
||||
"Raw bytes": measurement.rawBytes,
|
||||
"Gzip bytes": measurement.gzipBytes,
|
||||
"Brotli bytes": measurement.brotliBytes, // as-served (shipped .br, else estimated)
|
||||
"File count": measurement.fileCount,
|
||||
"Raw bytes delta": base ? measurement.rawBytes - base.rawBytes : "",
|
||||
"Gzip bytes delta": base ? measurement.gzipBytes - base.gzipBytes : "",
|
||||
"Brotli bytes delta": hasBrotli ? measurement.brotliBytes - base.brotliBytes : "",
|
||||
"Raw delta %": deltaPercent(measurement.rawBytes, base?.rawBytes),
|
||||
"Gzip delta %": gzipDeltaPercent,
|
||||
"Brotli delta %": brotliDeltaPercent,
|
||||
};
|
||||
});
|
||||
|
||||
const firstPoint = previous.length === 0;
|
||||
const significant = firstPoint || hasNewSeries || maxServedDeltaPercent >= threshold;
|
||||
|
||||
// Slim rows the next run diffs against. The workflow only persists this to the
|
||||
// cache when we actually push, so the cached reference always stays the last
|
||||
// *plotted* point (cumulative drift is caught).
|
||||
const cacheRows = measurements.map(({ bundle, kind, rawBytes, gzipBytes, brotliBytes }) => ({
|
||||
bundle,
|
||||
kind,
|
||||
rawBytes,
|
||||
gzipBytes,
|
||||
brotliBytes,
|
||||
}));
|
||||
|
||||
const reason = firstPoint
|
||||
? "first point"
|
||||
: hasNewSeries
|
||||
? "new bundle/kind series"
|
||||
: `max served Δ ${maxServedDeltaPercent.toFixed(2)}% (threshold ${threshold}%)`;
|
||||
|
||||
return { rows, cacheRows, significant, reason, maxServedDeltaPercent, firstPoint, hasNewSeries };
|
||||
}
|
||||
|
||||
module.exports = { buildStatsRows };
|
||||
|
||||
const readJson = filePath => (filePath && fs.existsSync(filePath) ? JSON.parse(fs.readFileSync(filePath, "utf8")) : null);
|
||||
const writeJson = (filePath, value) => {
|
||||
fs.mkdirSync(path.dirname(filePath), { recursive: true });
|
||||
fs.writeFileSync(filePath, JSON.stringify(value));
|
||||
};
|
||||
const setOutput = (env, name, value) => {
|
||||
if (env.GITHUB_OUTPUT) {
|
||||
fs.appendFileSync(env.GITHUB_OUTPUT, `${name}=${value}\n`);
|
||||
}
|
||||
};
|
||||
|
||||
// version.properties (extracted from the uberjar) carries the build's tag.
|
||||
// Non-release (master) builds report tag=vUNKNOWN, so leave Version empty there.
|
||||
function readVersion(versionPropsPath) {
|
||||
const raw =
|
||||
versionPropsPath && fs.existsSync(versionPropsPath)
|
||||
? (fs.readFileSync(versionPropsPath, "utf8").match(/^tag=(.*)$/m)?.[1]?.trim() ?? "")
|
||||
: "";
|
||||
return raw === "vUNKNOWN" ? "" : raw;
|
||||
}
|
||||
|
||||
function main() {
|
||||
const env = process.env;
|
||||
|
||||
const measurements = readJson(env.CURRENT);
|
||||
if (!Array.isArray(measurements) || measurements.length === 0) {
|
||||
console.error(`::error::No bundle measurements found at ${env.CURRENT}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const result = buildStatsRows({
|
||||
measurements,
|
||||
// The last plotted point, as slim rows: [{ bundle, kind, rawBytes, gzipBytes, brotliBytes }].
|
||||
previous: readJson(env.LAST) || [],
|
||||
threshold: Number(env.MIN_DELTA_PERCENT ?? 1),
|
||||
date: new Date().toISOString().slice(0, 10), // YYYY-MM-DD
|
||||
commit: (env.HEAD_SHA || "").slice(0, 12),
|
||||
commitMessage: (env.COMMIT_MESSAGE || "").split("\n")[0], // subject line only
|
||||
version: readVersion(env.VERSION_PROPS),
|
||||
});
|
||||
|
||||
writeJson(env.ROWS_OUT, result.rows);
|
||||
writeJson(env.CACHE_OUT, result.cacheRows);
|
||||
|
||||
console.log(`${result.significant ? "RECORD" : "SKIP"} — ${result.reason}`);
|
||||
|
||||
setOutput(env, "significant", result.significant ? "true" : "false");
|
||||
setOutput(
|
||||
env,
|
||||
"max_delta_percent",
|
||||
result.firstPoint || result.hasNewSeries ? "" : result.maxServedDeltaPercent.toFixed(2),
|
||||
);
|
||||
}
|
||||
|
||||
if (require.main === module) {
|
||||
main();
|
||||
}
|
||||
225
.github/scripts/bundle-size-stats-prepare.ts
vendored
225
.github/scripts/bundle-size-stats-prepare.ts
vendored
@@ -1,225 +0,0 @@
|
||||
// Prepares the rows the bundle-size stats logger imports into eng-stats-importer.
|
||||
//
|
||||
// For each measured (bundle, kind) it computes the delta — raw, gzipped and
|
||||
// brotli (as-served) bytes plus percent — against the previously *plotted* data
|
||||
// point (restored from a rolling cache), so the deltas can be charted directly.
|
||||
// It also decides whether this commit is worth recording at all: a point is kept
|
||||
// when the as-served size moved by at least MIN_DELTA_PERCENT, or when a new
|
||||
// bundle/kind appears. Only master builds run through here — releases form
|
||||
// non-linear lines whose deltas are meaningless, so they're not recorded (their
|
||||
// historical sizes are backfilled separately).
|
||||
//
|
||||
// buildStatsRows is pure (data in, data out) so it can be unit-tested; main()
|
||||
// is the thin I/O wrapper that reads env/files and writes the rows, cache and
|
||||
// step outputs.
|
||||
import {
|
||||
appendFileSync,
|
||||
existsSync,
|
||||
mkdirSync,
|
||||
readFileSync,
|
||||
writeFileSync,
|
||||
} from "node:fs";
|
||||
import { dirname } from "node:path";
|
||||
|
||||
/** One measurement emitted by measure-bundle-sizes. */
|
||||
export interface Measurement {
|
||||
bundle: string;
|
||||
kind: "initial" | "total";
|
||||
rawBytes: number;
|
||||
gzipBytes: number;
|
||||
brotliBytes: number;
|
||||
fileCount: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* The slim row cached between runs, holding the last *plotted* point.
|
||||
* `brotliBytes` is optional because a point cached before brotli logging
|
||||
* landed does not carry it — see buildStatsRows' per-row gzip fallback.
|
||||
*/
|
||||
export type CacheRow = Pick<Measurement, "bundle" | "kind" | "rawBytes" | "gzipBytes"> & {
|
||||
brotliBytes?: number;
|
||||
};
|
||||
|
||||
export interface BuildStatsRowsInput {
|
||||
measurements: Measurement[];
|
||||
previous: CacheRow[];
|
||||
threshold: number;
|
||||
date: string;
|
||||
commit: string;
|
||||
commitMessage: string;
|
||||
version: string;
|
||||
}
|
||||
|
||||
/** An upload row, keyed by the importer's column display names. */
|
||||
export type StatsRow = Record<string, string | number | null>;
|
||||
|
||||
export interface BuildStatsRowsResult {
|
||||
rows: StatsRow[];
|
||||
cacheRows: CacheRow[];
|
||||
significant: boolean;
|
||||
reason: string;
|
||||
maxServedDeltaPercent: number;
|
||||
firstPoint: boolean;
|
||||
hasNewSeries: boolean;
|
||||
}
|
||||
|
||||
// null, not "", for an absent delta: the stats importer coerces values
|
||||
// strictly and rejects "" for a numeric column.
|
||||
const deltaPercent = (current: number, base: number | undefined | null): number | null =>
|
||||
base ? Math.round(((current - base) * 10000) / base) / 100 : null;
|
||||
|
||||
/**
|
||||
* Turn the current measurements + the last plotted point into upload rows, the
|
||||
* slim cache rows for the next diff, and the keep/skip decision.
|
||||
*
|
||||
* The significance threshold tracks the as-served size: brotli where the build
|
||||
* ships a precompressed .br, otherwise gzip. Current measurements always carry
|
||||
* brotliBytes (estimated when no .br shipped), but a cached base point recorded
|
||||
* before brotli logging may not, so the delta still falls back to gzip per row.
|
||||
*/
|
||||
export function buildStatsRows({
|
||||
measurements,
|
||||
previous,
|
||||
threshold,
|
||||
date,
|
||||
commit,
|
||||
commitMessage,
|
||||
version,
|
||||
}: BuildStatsRowsInput): BuildStatsRowsResult {
|
||||
const previousOf = (bundle: string, kind: string) =>
|
||||
previous.find((row) => row.bundle === bundle && row.kind === kind);
|
||||
|
||||
let maxServedDeltaPercent = 0;
|
||||
let hasNewSeries = false;
|
||||
|
||||
const rows = measurements.map((measurement): StatsRow => {
|
||||
const base = previousOf(measurement.bundle, measurement.kind);
|
||||
// Non-null only when BOTH sides carry brotli: a point cached before brotli
|
||||
// logging landed has none, and that row falls back to the gzip delta.
|
||||
const baseBrotli =
|
||||
measurement.brotliBytes != null && base?.brotliBytes != null ? base.brotliBytes : null;
|
||||
const gzipDeltaPercent = deltaPercent(measurement.gzipBytes, base?.gzipBytes);
|
||||
const brotliDeltaPercent =
|
||||
baseBrotli != null ? deltaPercent(measurement.brotliBytes, baseBrotli) : null;
|
||||
if (!base) {
|
||||
hasNewSeries = true;
|
||||
} else {
|
||||
const servedDeltaPercent = baseBrotli != null ? brotliDeltaPercent : gzipDeltaPercent;
|
||||
maxServedDeltaPercent = Math.max(maxServedDeltaPercent, Math.abs(servedDeltaPercent ?? 0));
|
||||
}
|
||||
return {
|
||||
Date: date,
|
||||
Version: version,
|
||||
Commit: commit,
|
||||
Bundle: measurement.bundle,
|
||||
Kind: measurement.kind, // "initial" or "total"
|
||||
// The stats table carries a free-text Description column. Populate it
|
||||
// with the commit subject so the chart's points are self-describing.
|
||||
Description: commitMessage,
|
||||
"Raw bytes": measurement.rawBytes,
|
||||
"Gzip bytes": measurement.gzipBytes,
|
||||
"Brotli bytes": measurement.brotliBytes, // as-served (shipped .br, else estimated)
|
||||
"File count": measurement.fileCount,
|
||||
"Raw bytes delta": base ? measurement.rawBytes - base.rawBytes : null,
|
||||
"Gzip bytes delta": base ? measurement.gzipBytes - base.gzipBytes : null,
|
||||
"Brotli bytes delta": baseBrotli != null ? measurement.brotliBytes - baseBrotli : null,
|
||||
"Raw delta %": deltaPercent(measurement.rawBytes, base?.rawBytes),
|
||||
"Gzip delta %": gzipDeltaPercent,
|
||||
"Brotli delta %": brotliDeltaPercent,
|
||||
};
|
||||
});
|
||||
|
||||
const firstPoint = previous.length === 0;
|
||||
const significant = firstPoint || hasNewSeries || maxServedDeltaPercent >= threshold;
|
||||
|
||||
// Slim rows the next run diffs against. The workflow only persists this to the
|
||||
// cache when we actually push, so the cached reference always stays the last
|
||||
// *plotted* point (cumulative drift is caught).
|
||||
const cacheRows = measurements.map(
|
||||
({ bundle, kind, rawBytes, gzipBytes, brotliBytes }): CacheRow => ({
|
||||
bundle,
|
||||
kind,
|
||||
rawBytes,
|
||||
gzipBytes,
|
||||
brotliBytes,
|
||||
}),
|
||||
);
|
||||
|
||||
const reason = firstPoint
|
||||
? "first point"
|
||||
: hasNewSeries
|
||||
? "new bundle/kind series"
|
||||
: `max served Δ ${maxServedDeltaPercent.toFixed(2)}% (threshold ${threshold}%)`;
|
||||
|
||||
return {
|
||||
rows,
|
||||
cacheRows,
|
||||
significant,
|
||||
reason,
|
||||
maxServedDeltaPercent,
|
||||
firstPoint,
|
||||
hasNewSeries,
|
||||
};
|
||||
}
|
||||
|
||||
const readJson = <T,>(filePath: string | undefined): T | null =>
|
||||
filePath && existsSync(filePath) ? (JSON.parse(readFileSync(filePath, "utf8")) as T) : null;
|
||||
|
||||
const writeJson = (filePath: string, value: unknown) => {
|
||||
mkdirSync(dirname(filePath), { recursive: true });
|
||||
writeFileSync(filePath, JSON.stringify(value));
|
||||
};
|
||||
|
||||
const setOutput = (name: string, value: string) => {
|
||||
if (process.env.GITHUB_OUTPUT) {
|
||||
appendFileSync(process.env.GITHUB_OUTPUT, `${name}=${value}\n`);
|
||||
}
|
||||
};
|
||||
|
||||
// version.properties (extracted from the uberjar) carries the build's tag.
|
||||
// Non-release (master) builds report tag=vUNKNOWN, so leave Version empty there.
|
||||
function readVersion(versionPropsPath: string | undefined): string {
|
||||
const raw =
|
||||
versionPropsPath && existsSync(versionPropsPath)
|
||||
? (readFileSync(versionPropsPath, "utf8").match(/^tag=(.*)$/m)?.[1]?.trim() ?? "")
|
||||
: "";
|
||||
return raw === "vUNKNOWN" ? "" : raw;
|
||||
}
|
||||
|
||||
export function main() {
|
||||
const env = process.env;
|
||||
|
||||
const measurements = readJson<Measurement[]>(env.CURRENT);
|
||||
if (!Array.isArray(measurements) || measurements.length === 0) {
|
||||
console.error(`::error::No bundle measurements found at ${env.CURRENT}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const result = buildStatsRows({
|
||||
measurements,
|
||||
// The last plotted point, as slim rows: [{ bundle, kind, rawBytes, gzipBytes, brotliBytes }].
|
||||
previous: readJson<CacheRow[]>(env.LAST) || [],
|
||||
threshold: Number(env.MIN_DELTA_PERCENT ?? 1),
|
||||
date: new Date().toISOString().slice(0, 10), // YYYY-MM-DD
|
||||
commit: (env.HEAD_SHA || "").slice(0, 12),
|
||||
commitMessage: (env.COMMIT_MESSAGE || "").split("\n")[0], // subject line only
|
||||
version: readVersion(env.VERSION_PROPS),
|
||||
});
|
||||
|
||||
writeJson(env.ROWS_OUT!, result.rows);
|
||||
writeJson(env.CACHE_OUT!, result.cacheRows);
|
||||
|
||||
console.log(`${result.significant ? "RECORD" : "SKIP"} — ${result.reason}`);
|
||||
|
||||
setOutput("significant", result.significant ? "true" : "false");
|
||||
setOutput(
|
||||
"max_delta_percent",
|
||||
result.firstPoint || result.hasNewSeries ? "" : result.maxServedDeltaPercent.toFixed(2),
|
||||
);
|
||||
}
|
||||
|
||||
// Only run the I/O wrapper when invoked directly, so importing buildStatsRows
|
||||
// (from the spec) has no side effects.
|
||||
if ((import.meta as ImportMeta & { main?: boolean }).main) {
|
||||
main();
|
||||
}
|
||||
87
.github/scripts/bundle-size-stats-prepare.unit.spec.js
vendored
Normal file
87
.github/scripts/bundle-size-stats-prepare.unit.spec.js
vendored
Normal file
@@ -0,0 +1,87 @@
|
||||
const { buildStatsRows } = require("./bundle-size-stats-prepare");
|
||||
|
||||
const meta = { date: "2026-06-30", commit: "abc123", commitMessage: "Fix the thing (#123)", version: "" };
|
||||
const measurement = (over = {}) => ({
|
||||
bundle: "embedding-sdk-chunked",
|
||||
kind: "total",
|
||||
rawBytes: 1000,
|
||||
gzipBytes: 400,
|
||||
brotliBytes: 300,
|
||||
fileCount: 3,
|
||||
...over,
|
||||
});
|
||||
|
||||
describe("buildStatsRows", () => {
|
||||
it("records the first point and leaves every delta empty", () => {
|
||||
const result = buildStatsRows({
|
||||
...meta,
|
||||
measurements: [measurement()],
|
||||
previous: [],
|
||||
threshold: 1,
|
||||
});
|
||||
|
||||
expect(result.firstPoint).toBe(true);
|
||||
expect(result.significant).toBe(true);
|
||||
expect(result.reason).toBe("first point");
|
||||
expect(result.rows[0]["Gzip bytes delta"]).toBe("");
|
||||
expect(result.rows[0]["Brotli delta %"]).toBe("");
|
||||
expect(result.rows[0]["Description"]).toBe("Fix the thing (#123)");
|
||||
expect(result.cacheRows[0]).toEqual({
|
||||
bundle: "embedding-sdk-chunked",
|
||||
kind: "total",
|
||||
rawBytes: 1000,
|
||||
gzipBytes: 400,
|
||||
brotliBytes: 300,
|
||||
});
|
||||
});
|
||||
|
||||
it("records a never-seen bundle/kind as a new series", () => {
|
||||
const result = buildStatsRows({
|
||||
...meta,
|
||||
measurements: [measurement({ bundle: "app", kind: "initial" })],
|
||||
previous: [{ bundle: "app", kind: "total", rawBytes: 1, gzipBytes: 1, brotliBytes: 1 }],
|
||||
threshold: 99,
|
||||
});
|
||||
|
||||
expect(result.hasNewSeries).toBe(true);
|
||||
expect(result.significant).toBe(true);
|
||||
expect(result.reason).toBe("new bundle/kind series");
|
||||
});
|
||||
|
||||
it("gates significance on the brotli (as-served) delta when both sides have it", () => {
|
||||
const previous = [{ bundle: "embedding-sdk-chunked", kind: "total", rawBytes: 1000, gzipBytes: 400, brotliBytes: 300 }];
|
||||
|
||||
const big = buildStatsRows({
|
||||
...meta,
|
||||
measurements: [measurement({ brotliBytes: 315, gzipBytes: 400 })], // brotli +5%, gzip 0%
|
||||
previous,
|
||||
threshold: 1,
|
||||
});
|
||||
expect(big.maxServedDeltaPercent).toBeCloseTo(5);
|
||||
expect(big.significant).toBe(true);
|
||||
expect(big.rows[0]["Brotli delta %"]).toBeCloseTo(5);
|
||||
|
||||
const small = buildStatsRows({
|
||||
...meta,
|
||||
measurements: [measurement({ brotliBytes: 301, gzipBytes: 999 })], // brotli +0.33%, gzip ignored
|
||||
previous,
|
||||
threshold: 1,
|
||||
});
|
||||
expect(small.significant).toBe(false);
|
||||
expect(small.reason).toContain("threshold 1%");
|
||||
});
|
||||
|
||||
it("falls back to the gzip delta when the cached base predates brotli logging", () => {
|
||||
const result = buildStatsRows({
|
||||
...meta,
|
||||
measurements: [measurement({ gzipBytes: 440, brotliBytes: 300 })], // gzip +10%
|
||||
previous: [{ bundle: "embedding-sdk-chunked", kind: "total", rawBytes: 1000, gzipBytes: 400 }],
|
||||
threshold: 1,
|
||||
});
|
||||
|
||||
expect(result.maxServedDeltaPercent).toBeCloseTo(10);
|
||||
expect(result.significant).toBe(true);
|
||||
expect(result.rows[0]["Brotli bytes delta"]).toBe(""); // no base brotli to diff against
|
||||
expect(result.rows[0]["Gzip delta %"]).toBeCloseTo(10);
|
||||
});
|
||||
});
|
||||
@@ -1,105 +0,0 @@
|
||||
import {
|
||||
buildStatsRows,
|
||||
type CacheRow,
|
||||
type Measurement,
|
||||
} from "./bundle-size-stats-prepare";
|
||||
|
||||
const meta = {
|
||||
date: "2026-06-30",
|
||||
commit: "abc123",
|
||||
commitMessage: "Fix the thing (#123)",
|
||||
version: "",
|
||||
};
|
||||
const measurement = (over: Partial<Measurement> = {}): Measurement => ({
|
||||
bundle: "embedding-sdk-chunked",
|
||||
kind: "total",
|
||||
rawBytes: 1000,
|
||||
gzipBytes: 400,
|
||||
brotliBytes: 300,
|
||||
fileCount: 3,
|
||||
...over,
|
||||
});
|
||||
|
||||
describe("buildStatsRows", () => {
|
||||
it("records the first point and leaves every delta empty", () => {
|
||||
const result = buildStatsRows({
|
||||
...meta,
|
||||
measurements: [measurement()],
|
||||
previous: [],
|
||||
threshold: 1,
|
||||
});
|
||||
|
||||
expect(result.firstPoint).toBe(true);
|
||||
expect(result.significant).toBe(true);
|
||||
expect(result.reason).toBe("first point");
|
||||
expect(result.rows[0]["Gzip bytes delta"]).toBeNull();
|
||||
expect(result.rows[0]["Brotli delta %"]).toBeNull();
|
||||
expect(result.rows[0]["Description"]).toBe("Fix the thing (#123)");
|
||||
expect(result.cacheRows[0]).toEqual({
|
||||
bundle: "embedding-sdk-chunked",
|
||||
kind: "total",
|
||||
rawBytes: 1000,
|
||||
gzipBytes: 400,
|
||||
brotliBytes: 300,
|
||||
});
|
||||
});
|
||||
|
||||
it("records a never-seen bundle/kind as a new series", () => {
|
||||
const result = buildStatsRows({
|
||||
...meta,
|
||||
measurements: [measurement({ bundle: "app", kind: "initial" })],
|
||||
previous: [{ bundle: "app", kind: "total", rawBytes: 1, gzipBytes: 1, brotliBytes: 1 }],
|
||||
threshold: 99,
|
||||
});
|
||||
|
||||
expect(result.hasNewSeries).toBe(true);
|
||||
expect(result.significant).toBe(true);
|
||||
expect(result.reason).toBe("new bundle/kind series");
|
||||
});
|
||||
|
||||
it("gates significance on the brotli (as-served) delta when both sides have it", () => {
|
||||
const previous: CacheRow[] = [
|
||||
{
|
||||
bundle: "embedding-sdk-chunked",
|
||||
kind: "total",
|
||||
rawBytes: 1000,
|
||||
gzipBytes: 400,
|
||||
brotliBytes: 300,
|
||||
},
|
||||
];
|
||||
|
||||
const big = buildStatsRows({
|
||||
...meta,
|
||||
measurements: [measurement({ brotliBytes: 315, gzipBytes: 400 })], // brotli +5%, gzip 0%
|
||||
previous,
|
||||
threshold: 1,
|
||||
});
|
||||
expect(big.maxServedDeltaPercent).toBeCloseTo(5);
|
||||
expect(big.significant).toBe(true);
|
||||
expect(big.rows[0]["Brotli delta %"]).toBeCloseTo(5);
|
||||
|
||||
const small = buildStatsRows({
|
||||
...meta,
|
||||
measurements: [measurement({ brotliBytes: 301, gzipBytes: 999 })], // brotli +0.33%, gzip ignored
|
||||
previous,
|
||||
threshold: 1,
|
||||
});
|
||||
expect(small.significant).toBe(false);
|
||||
expect(small.reason).toContain("threshold 1%");
|
||||
});
|
||||
|
||||
it("falls back to the gzip delta when the cached base predates brotli logging", () => {
|
||||
const result = buildStatsRows({
|
||||
...meta,
|
||||
measurements: [measurement({ gzipBytes: 440, brotliBytes: 300 })], // gzip +10%
|
||||
// No brotliBytes: this point was cached before brotli logging landed.
|
||||
previous: [{ bundle: "embedding-sdk-chunked", kind: "total", rawBytes: 1000, gzipBytes: 400 }],
|
||||
threshold: 1,
|
||||
});
|
||||
|
||||
expect(result.maxServedDeltaPercent).toBeCloseTo(10);
|
||||
expect(result.significant).toBe(true);
|
||||
expect(result.rows[0]["Brotli bytes delta"]).toBeNull(); // no base brotli to diff against
|
||||
expect(result.rows[0]["Gzip delta %"]).toBeCloseTo(10);
|
||||
});
|
||||
});
|
||||
162
.github/scripts/cache-keys.sh
vendored
162
.github/scripts/cache-keys.sh
vendored
@@ -1,162 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Single definition of every CI dependency cache: what it holds, what identifies it, what it falls back to.
|
||||
#
|
||||
# A GitHub cache entry is identified by its key, a hash of the declared path list, and the ref that saved
|
||||
# it. A step declaring a different path list than the one a cache was saved with computes a different
|
||||
# identity and can never match it, whatever its key says. So paths and keys are emitted together here and
|
||||
# every cache step takes both from this script.
|
||||
#
|
||||
# Emits `name=value` lines in $GITHUB_OUTPUT format:
|
||||
#
|
||||
# .github/scripts/cache-keys.sh >> "$GITHUB_OUTPUT"
|
||||
#
|
||||
# Run it with no redirect to see the whole scheme, or pass one name to print a single value:
|
||||
#
|
||||
# .github/scripts/cache-keys.sh
|
||||
# .github/scripts/cache-keys.sh m2-key
|
||||
#
|
||||
# Hashes are computed here rather than with GitHub's hashFiles() so that a local run and a CI run of the
|
||||
# same commit produce the same keys, and so that an empty match can fail loudly - a hashFiles() call that
|
||||
# matches nothing silently yields "", collapsing the key onto its own restore prefix and pointing every
|
||||
# dependency set at one entry.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
cd "$(git rev-parse --show-toplevel)"
|
||||
|
||||
OS="${RUNNER_OS:-$(uname -s)}"
|
||||
[ "$OS" = "Darwin" ] && OS="macOS"
|
||||
|
||||
sha256() {
|
||||
if command -v sha256sum >/dev/null 2>&1; then
|
||||
sha256sum | cut -d' ' -f1
|
||||
else
|
||||
shasum -a 256 | cut -d' ' -f1
|
||||
fi
|
||||
}
|
||||
|
||||
# Digest of a set of files: hash each one, then hash the concatenation, so the result depends on every
|
||||
# file's contents and on which files exist.
|
||||
digest() {
|
||||
local f
|
||||
if [ "$#" -eq 0 ]; then
|
||||
echo "::error::cache-keys.sh: no files matched; refusing to emit a key that would collide across dependency sets" >&2
|
||||
exit 1
|
||||
fi
|
||||
for f in "$@"; do
|
||||
[ -f "$f" ] || { echo "::error::cache-keys.sh: $f does not exist" >&2; exit 1; }
|
||||
sha256 < "$f"
|
||||
done | sha256
|
||||
}
|
||||
|
||||
shopt -s nullglob
|
||||
DEPS_FILES=(deps.edn
|
||||
bin/lint-migrations-file/deps.edn
|
||||
bin/load-namespaces/deps.edn
|
||||
modules/drivers/deps.edn
|
||||
modules/drivers/*/deps.edn)
|
||||
shopt -u nullglob
|
||||
|
||||
DEPS_HASH="$(digest "${DEPS_FILES[@]}")"
|
||||
LOCK_HASH="$(digest bun.lock)"
|
||||
|
||||
# The Cypress binary depends on the Cypress version alone, so keying it on the whole lockfile would
|
||||
# throw away a 200MB download every time any frontend dependency moved.
|
||||
#
|
||||
# Reading a version out of a lockfile with a regex is only safe while exactly one line matches. Insist
|
||||
# on that rather than taking the first of several: a second match means the format moved under us, and
|
||||
# picking one at random would key the binary under a version it is not.
|
||||
CYPRESS_VERSIONS="$(sed -n 's/.*"cypress": \["cypress@\([^"]*\)".*/\1/p' bun.lock)"
|
||||
CYPRESS_MATCHES="$(printf '%s\n' "$CYPRESS_VERSIONS" | grep -c . || true)"
|
||||
if [ "$CYPRESS_MATCHES" != "1" ]; then
|
||||
echo "::error::cache-keys.sh: expected exactly one resolved cypress version in bun.lock, found $CYPRESS_MATCHES" >&2
|
||||
exit 1
|
||||
fi
|
||||
CYPRESS_VERSION="$CYPRESS_VERSIONS"
|
||||
|
||||
# The Clojure CLI version CI installs. It lives here rather than in prepare-backend so that the version
|
||||
# and the key that identifies its cache entry cannot drift apart.
|
||||
CLOJURE_VERSION="1.12.0.1488"
|
||||
|
||||
# An incremental lint cache is only ever an accelerator, so it keys on the commit and always falls back
|
||||
# to the newest entry on the prefix. The exact key never pre-exists, which keeps the entry current.
|
||||
ESLINT_SHA="${GITHUB_SHA:-$(git rev-parse HEAD)}"
|
||||
|
||||
emit_multiline() {
|
||||
local name="$1"; shift
|
||||
printf '%s<<CACHE_KEYS_EOF\n' "$name"
|
||||
printf '%s\n' "$@"
|
||||
printf 'CACHE_KEYS_EOF\n'
|
||||
}
|
||||
|
||||
# Paths under the home directory are emitted absolute rather than as `~/...`. actions/cache would expand
|
||||
# a tilde, but a shell assigning one of these to a variable would not, and would go on to create a
|
||||
# directory named `~`. Paths inside the checkout stay relative; actions/cache resolves those against the
|
||||
# workspace.
|
||||
spec() {
|
||||
emit_multiline m2-path "$HOME/.m2" "$HOME/.gitlibs" "$HOME/.deps.clj" 'bin/bb'
|
||||
echo "m2-key=M2-$OS-$DEPS_HASH"
|
||||
echo "m2-restore-key=M2-$OS-"
|
||||
|
||||
# Pinned rather than discovered via `bun pm cache` so that the path is known before bun is installed;
|
||||
# prepare-frontend exports BUN_INSTALL_CACHE_DIR to match and fails if bun disagrees.
|
||||
echo "bun-store-path=$HOME/.bun/install/cache"
|
||||
echo "bun-store-key=bun-store-$OS-$LOCK_HASH"
|
||||
echo "bun-store-restore-key=bun-store-$OS-"
|
||||
|
||||
# No restore-key: an entry from another version installs a working `clojure` of the wrong version, and
|
||||
# the freshness check in prepare-backend - running the binary - would accept it.
|
||||
#
|
||||
# The installer bakes this prefix into the clj/clojure shims, so the tree only works when restored to
|
||||
# the same absolute path it was built at; prepare-backend reinstalls when it is not.
|
||||
echo "clojure-path=$HOME/.clojure-cli"
|
||||
echo "clojure-key=clojure-cli-$OS-$CLOJURE_VERSION"
|
||||
echo "clojure-version=$CLOJURE_VERSION"
|
||||
|
||||
echo "cypress-path=$HOME/.cache/Cypress"
|
||||
echo "cypress-key=cypress-$OS-$CYPRESS_VERSION"
|
||||
echo "cypress-restore-key=cypress-$OS-"
|
||||
|
||||
echo "eslint-path=.eslintcache"
|
||||
echo "eslint-key=eslint-$OS-$ESLINT_SHA"
|
||||
echo "eslint-restore-key=eslint-$OS-"
|
||||
}
|
||||
|
||||
# Print one output's value, including the multiline form, and fail on a name that does not exist. A
|
||||
# caller that silently received "" would go on to build a key with a hole in it.
|
||||
query() {
|
||||
local want="$1" line delim found=0 in_block=0
|
||||
while IFS= read -r line; do
|
||||
if [ "$in_block" = 1 ]; then
|
||||
if [ "$line" = "$delim" ]; then
|
||||
in_block=0
|
||||
else
|
||||
printf '%s\n' "$line"
|
||||
fi
|
||||
continue
|
||||
fi
|
||||
case "$line" in
|
||||
"$want="*)
|
||||
printf '%s\n' "${line#*=}"
|
||||
found=1
|
||||
;;
|
||||
"$want<<"*)
|
||||
delim="${line#*<<}"
|
||||
in_block=1
|
||||
found=1
|
||||
;;
|
||||
esac
|
||||
done <<EOF
|
||||
$(spec)
|
||||
EOF
|
||||
if [ "$found" = 0 ]; then
|
||||
echo "::error::cache-keys.sh: no output named '$want'" >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
if [ "$#" -eq 0 ]; then
|
||||
spec
|
||||
else
|
||||
query "$1"
|
||||
fi
|
||||
80
.github/scripts/cache-keys.unit.spec.ts
vendored
80
.github/scripts/cache-keys.unit.spec.ts
vendored
@@ -1,80 +0,0 @@
|
||||
import { execFileSync } from "child_process";
|
||||
import { join } from "path";
|
||||
|
||||
const SCRIPT = join(__dirname, "cache-keys.sh");
|
||||
|
||||
const CACHES = ["m2", "bun-store", "cypress", "eslint"];
|
||||
|
||||
// stderr is inherited by default, so the deliberate-failure case below would print a literal
|
||||
// `::error::` line - which GitHub Actions reads as an annotation directive, making a passing run
|
||||
// annotate itself with an error.
|
||||
const run = (args: string[] = [], env: NodeJS.ProcessEnv = {}, quiet = false) =>
|
||||
execFileSync(SCRIPT, args, {
|
||||
encoding: "utf8",
|
||||
stdio: quiet ? ["ignore", "pipe", "ignore"] : "pipe",
|
||||
env: { ...process.env, RUNNER_OS: "Linux", GITHUB_SHA: "abc123", ...env },
|
||||
});
|
||||
|
||||
const get = (name: string, env: NodeJS.ProcessEnv = {}) =>
|
||||
run([name], env).trimEnd();
|
||||
|
||||
describe("cache-keys.sh", () => {
|
||||
it("is deterministic for a given tree", () => {
|
||||
expect(run()).toEqual(run());
|
||||
});
|
||||
|
||||
it("emits every cache's path, key and restore key", () => {
|
||||
for (const cache of CACHES) {
|
||||
expect(get(`${cache}-path`)).not.toEqual("");
|
||||
expect(get(`${cache}-key`)).not.toEqual("");
|
||||
expect(get(`${cache}-restore-key`)).not.toEqual("");
|
||||
}
|
||||
});
|
||||
|
||||
// Silently returning "" for a name that does not exist is how a caller ends up building a key with a
|
||||
// hole in it.
|
||||
it("fails on an unknown output name", () => {
|
||||
expect(() => run(["no-such-output"], {}, true)).toThrow();
|
||||
});
|
||||
|
||||
// A key equal to its own restore prefix points every dependency set at one entry, and nothing reports
|
||||
// it. This is the failure the script exists to prevent.
|
||||
it("never emits a key equal to its own restore prefix", () => {
|
||||
for (const cache of CACHES) {
|
||||
const key = get(`${cache}-key`);
|
||||
const prefix = get(`${cache}-restore-key`);
|
||||
expect(key.startsWith(prefix)).toBe(true);
|
||||
expect(key.length).toBeGreaterThan(prefix.length);
|
||||
}
|
||||
});
|
||||
|
||||
it("scopes every key to the runner OS", () => {
|
||||
for (const cache of CACHES) {
|
||||
expect(get(`${cache}-key`, { RUNNER_OS: "Linux" })).not.toEqual(
|
||||
get(`${cache}-key`, { RUNNER_OS: "macOS" }),
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
// Home paths must be absolute: actions/cache expands a tilde but a shell assigning one to a variable
|
||||
// does not, and would create a directory literally named `~`.
|
||||
it("emits absolute paths for home directory locations", () => {
|
||||
expect(get("bun-store-path").startsWith("/")).toBe(true);
|
||||
expect(get("cypress-path").startsWith("/")).toBe(true);
|
||||
for (const line of get("m2-path").split("\n")) {
|
||||
expect(line.startsWith("~")).toBe(false);
|
||||
}
|
||||
});
|
||||
|
||||
// The Cypress binary depends on the Cypress version alone, so an unrelated dependency bump must not
|
||||
// invalidate a 200MB download.
|
||||
it("keys the Cypress binary on the resolved version, not the lockfile", () => {
|
||||
expect(get("cypress-key")).toMatch(/^cypress-Linux-\d+\.\d+\.\d+/);
|
||||
});
|
||||
|
||||
it("keys the ESLint cache on the commit", () => {
|
||||
expect(get("eslint-key", { GITHUB_SHA: "deadbeef" })).toEqual(
|
||||
"eslint-Linux-deadbeef",
|
||||
);
|
||||
});
|
||||
});
|
||||
71
.github/scripts/check-job-timeouts.sh
vendored
71
.github/scripts/check-job-timeouts.sh
vendored
@@ -1,71 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Fails when a job in .github/workflows/ does not set `timeout-minutes`.
|
||||
#
|
||||
# Without it a job inherits GitHub's 6-hour default, so a hung job burns six hours of runner time -
|
||||
# and holds up everything queued behind it - before anyone notices. actionlint cannot check this: its
|
||||
# rule set is compiled in and .github/actionlint.yaml only takes `self-hosted-runner`,
|
||||
# `config-variables`, and `paths` (which suppresses findings rather than adding them). Hence this
|
||||
# script.
|
||||
#
|
||||
# The rule is per job: a `timeout-minutes` on a step does not satisfy it, and steps are not required
|
||||
# to carry one of their own.
|
||||
#
|
||||
# Jobs that call a reusable workflow (`uses:` at the job level) are skipped - GitHub rejects
|
||||
# `timeout-minutes` there. The called workflow's own jobs are checked when it lives in this repo.
|
||||
#
|
||||
# .github/scripts/check-job-timeouts.sh [workflow-file ...]
|
||||
#
|
||||
# With no arguments it checks every workflow; CI passes the files that changed, like actionlint does.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# Arguments are used as given, so the paths in the annotations match what the caller passed. Without
|
||||
# them the full scan has to run from the repo root.
|
||||
if [ $# -gt 0 ]; then
|
||||
workflows=("$@")
|
||||
else
|
||||
cd "$(git rev-parse --show-toplevel)"
|
||||
|
||||
shopt -s nullglob
|
||||
workflows=(.github/workflows/*.yml .github/workflows/*.yaml)
|
||||
|
||||
if [ ${#workflows[@]} -eq 0 ]; then
|
||||
echo "::error::check-job-timeouts.sh: no workflow files found under .github/workflows/" >&2
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
status=0
|
||||
|
||||
for file in "${workflows[@]}"; do
|
||||
# `awk NF` drops the blank line yq emits when a file has nothing to report.
|
||||
missing="$(
|
||||
# --header-preprocess=false: yq otherwise slurps any leading comment block before parsing, which
|
||||
# shifts every line number it reports by the size of that block.
|
||||
yq --header-preprocess=false '
|
||||
.jobs
|
||||
| to_entries
|
||||
| .[]
|
||||
| select(((.value | has("timeout-minutes")) or (.value | has("uses"))) | not)
|
||||
| [(.key | line), .key]
|
||||
| @tsv
|
||||
' "$file" | awk NF
|
||||
)"
|
||||
|
||||
[ -n "$missing" ] || continue
|
||||
|
||||
while IFS=$'\t' read -r line job; do
|
||||
echo "::error file=${file},line=${line}::Job \"${job}\" does not set a job-level timeout-minutes" >&2
|
||||
done <<< "$missing"
|
||||
|
||||
status=1
|
||||
done
|
||||
|
||||
if [ "$status" -ne 0 ]; then
|
||||
echo >&2
|
||||
echo "Every job must set an explicit timeout-minutes; without one it inherits GitHub's 6-hour default." >&2
|
||||
echo "Pick a value with headroom over the job's normal runtime." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "All jobs in the ${#workflows[@]} workflow file(s) checked set timeout-minutes."
|
||||
109
.github/scripts/check-preresolve-aliases.sh
vendored
109
.github/scripts/check-preresolve-aliases.sh
vendored
@@ -1,109 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Fails when a Clojure classpath used by CI is not pre-resolved by cache-warm.
|
||||
#
|
||||
# tools.deps selects dependency versions globally per classpath, so two alias combinations can resolve
|
||||
# the same transitive library to different versions. A combination the warm job never resolved is
|
||||
# therefore not merely absent from the cache - the cache holds *a* version of those libraries, just not
|
||||
# the one this classpath wants, and the job fetches its delta straight from Maven Central on every run.
|
||||
# That is the intermittent-403 failure of DEV-2094, and it is invisible: the build still passes.
|
||||
#
|
||||
# Nothing about a missing combination shows up at runtime, so this check is the only thing standing
|
||||
# between a newly added alias combination and a permanently half-cold cache.
|
||||
#
|
||||
# .github/scripts/check-preresolve-aliases.sh
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
cd "$(git rev-parse --show-toplevel)"
|
||||
|
||||
WARM=.github/workflows/cache-warm.yml
|
||||
|
||||
# Invocations deliberately left out of the warm job. Each needs a reason, because the default answer is
|
||||
# to add the combination to the list instead.
|
||||
WAIVED="$(
|
||||
cat <<'EOF'
|
||||
:deps
|
||||
:outdated
|
||||
EOF
|
||||
)"
|
||||
# :deps - tools.deps built-in used by write-poms.sh; contributes no dependencies of its own.
|
||||
# :outdated - scheduled dependency-bump workflow only; a cold resolve there costs nothing anyone waits on.
|
||||
|
||||
# Dependency roots and alias combinations the warm job pre-resolves.
|
||||
declared="$({
|
||||
sed -n '/^[[:space:]]*combos=(/,/^ *)/p' "$WARM" \
|
||||
| grep -oE '":[^"]*"' \
|
||||
| tr -d '"' \
|
||||
| sed 's#^#.|#'
|
||||
sed -n '/^[[:space:]]*nested_combos=(/,/^ *)/p' "$WARM" \
|
||||
| grep -oE '"[^"]+\|:[^"]+"' \
|
||||
| tr -d '"'
|
||||
} | sort -u)"
|
||||
if [ -z "$declared" ]; then
|
||||
echo "::error::check-preresolve-aliases.sh: found no pre-resolved classpaths in $WARM" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Expressions we know how to expand. An unrecognised one has to stop the check rather than be skipped:
|
||||
# silently ignoring it is how a whole matrix of combinations would go unchecked.
|
||||
unknown="$(grep -rhoE '\$\{\{[^}]*\}\}' --include='*' \
|
||||
--exclude='*.png' --exclude='*.jpg' .github/ 2>/dev/null \
|
||||
| sort -u | grep -v 'edition' || true)"
|
||||
|
||||
scan() { # scan <edition>: dependency root and aliases used by .github Clojure invocations
|
||||
grep -rhE 'clojure[[:space:]]+(-P[[:space:]]+)?-[XMATP]?:' \
|
||||
--include='*' --exclude='*.png' --exclude='*.jpg' .github/ 2>/dev/null \
|
||||
| sed "s/\\\${{[^}]*edition[^}]*}}/$1/g" \
|
||||
| grep -oE 'clojure[[:space:]]+(-P[[:space:]]+)?-[XMATP]?:[A-Za-z0-9:_./-]+' \
|
||||
| grep -oE ':[A-Za-z0-9:_./-]+' \
|
||||
| sed -e 's/:$//' -e 's#^#.|#'
|
||||
}
|
||||
|
||||
scan_project_tests() { # scan_project_tests <function> <dependency-root>
|
||||
# Mage shell commands are vectors split across lines, so flatten each function body before
|
||||
# extracting its alias.
|
||||
sed -n "/(defn- $1 /,/^(defn/p" mage/src/mage/project_tests.clj \
|
||||
| tr '\n' ' ' \
|
||||
| grep -oE '"clojure"[[:space:]]+("-P"[[:space:]]+)?"-[XMATP]?:[A-Za-z0-9:_./-]+' \
|
||||
| grep -oE ':[A-Za-z0-9:_./-]+' \
|
||||
| sed -e 's/:$//' -e "s#^#$2|#"
|
||||
}
|
||||
|
||||
# Check each scan separately. One scan still finding commands would otherwise conceal that another has
|
||||
# stopped matching, which silently drops everything it used to cover.
|
||||
require_scan() { # require_scan <what> <result>
|
||||
[ -n "$2" ] && return
|
||||
echo "::error::check-preresolve-aliases.sh: found no Clojure invocation in $1; the scan pattern no longer matches" >&2
|
||||
exit 1
|
||||
}
|
||||
workflow_checks=$( { scan ee; scan oss; } || true)
|
||||
clojure_checks=$(scan_project_tests run-clojure-checks! . || true)
|
||||
migration_checks=$(scan_project_tests run-migration-checks! bin/lint-migrations-file || true)
|
||||
require_scan ".github/" "$workflow_checks"
|
||||
require_scan "run-clojure-checks! (mage/src/mage/project_tests.clj)" "$clojure_checks"
|
||||
require_scan "run-migration-checks! (mage/src/mage/project_tests.clj)" "$migration_checks"
|
||||
|
||||
scanned="$(printf '%s\n' "$workflow_checks" "$clojure_checks" "$migration_checks" | sort -u)"
|
||||
|
||||
found="$( {
|
||||
printf '%s\n' "$scanned"
|
||||
# build-scripts.yml runs this classpath from its own dependency root.
|
||||
printf '%s\n' 'bin/load-namespaces|:test'
|
||||
} | sort -u )"
|
||||
|
||||
covered="$( {
|
||||
printf '%s\n' "$declared"
|
||||
printf '%s\n' "$WAIVED" | sed 's#^#.|#'
|
||||
} | sort -u)"
|
||||
missing="$(comm -23 <(printf '%s\n' "$found") <(printf '%s\n' "$covered") || true)"
|
||||
|
||||
if [ -n "$missing" ]; then
|
||||
echo "::error::Clojure classpaths used by CI that cache-warm does not pre-resolve:" >&2
|
||||
printf '%s\n' "$missing" | sed 's/^/ /' >&2
|
||||
echo >&2
|
||||
echo "Add each to the matching root or nested combo list in $WARM, or waive it here with a reason." >&2
|
||||
echo "Leaving it uncovered means those jobs re-fetch their version deltas from Maven on every run." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "All $(printf '%s\n' "$found" | grep -c .) Clojure classpaths used by CI are pre-resolved."
|
||||
109
.github/scripts/conventional-commit-pr-title-reminder/utils.js
vendored
Normal file
109
.github/scripts/conventional-commit-pr-title-reminder/utils.js
vendored
Normal file
@@ -0,0 +1,109 @@
|
||||
const TYPES = new Set([
|
||||
"feat",
|
||||
"fix",
|
||||
"perf",
|
||||
"docs",
|
||||
"style",
|
||||
"refactor",
|
||||
"test",
|
||||
"build",
|
||||
"ci",
|
||||
]);
|
||||
|
||||
/**
|
||||
* @param title {string}
|
||||
*/
|
||||
function isConventionalTitle(title) {
|
||||
const match = /^(?<type>\w+)\((?<rawScopes>[^)]+)\):/.exec(title);
|
||||
|
||||
if (!match) {
|
||||
return false;
|
||||
}
|
||||
|
||||
const [, type, rawScopes] = match;
|
||||
const scopes = rawScopes
|
||||
.split(",")
|
||||
.map((scope) => scope.trim())
|
||||
.filter(Boolean);
|
||||
|
||||
return TYPES.has(type) && scopes.length > 0;
|
||||
}
|
||||
|
||||
/** @param {import('@types/github-script').AsyncFunctionArguments} AsyncFunctionArguments */
|
||||
async function updateCommentForEmbedding({ github, context }) {
|
||||
const githubUsername = context.payload.sender.login;
|
||||
|
||||
const comment = `${CONVENTIONAL_COMMENT_IDENTIFIER}
|
||||
@${githubUsername} You have modified embedding code. Please make sure the PR title follows [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) style.
|
||||
Here are all the supported types:
|
||||
- \`feat\`
|
||||
- \`fix\`
|
||||
- \`perf\`
|
||||
- \`docs\`
|
||||
- \`style\`
|
||||
- \`refactor\`
|
||||
- \`test\`
|
||||
- \`build\`
|
||||
- \`ci\`
|
||||
|
||||
Please also make sure to include a scope.
|
||||
|
||||
For example, these are valid PR titles:
|
||||
- \`feat(sdk-bundle): Add interactive dashboard component\`
|
||||
- \`fix(sdk-package): Fixed MetabaseProviderPropsStore logic\`
|
||||
- \`fix(sdk-package, embed-js): Fix API\``;
|
||||
|
||||
return updateComment(comment, { github, context });
|
||||
}
|
||||
|
||||
/**
|
||||
* @param comment {string}
|
||||
* @param {import('@types/github-script').AsyncFunctionArguments} AsyncFunctionArguments
|
||||
*/
|
||||
async function updateComment(comment, { github, context }) {
|
||||
const owner = context.repo.owner;
|
||||
const repo = context.repo.repo;
|
||||
const pullRequestNumber = context.issue.number;
|
||||
const comments = await github.rest.issues.listComments({
|
||||
owner,
|
||||
repo,
|
||||
issue_number: pullRequestNumber,
|
||||
});
|
||||
|
||||
const existingComment = getExistingConventionalCommitReminderComment(
|
||||
comments.data,
|
||||
);
|
||||
|
||||
if (existingComment) {
|
||||
return await github.rest.issues.updateComment({
|
||||
owner,
|
||||
repo,
|
||||
comment_id: existingComment.id,
|
||||
body: comment,
|
||||
});
|
||||
} else {
|
||||
return await github.rest.issues.createComment({
|
||||
owner,
|
||||
repo,
|
||||
issue_number: pullRequestNumber,
|
||||
body: comment,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
const CONVENTIONAL_COMMENT_IDENTIFIER =
|
||||
"<!---conventional comment identifier-->";
|
||||
|
||||
/**
|
||||
* @param comments {{body: string}[]}
|
||||
*/
|
||||
function getExistingConventionalCommitReminderComment(comments) {
|
||||
for (const comment of comments) {
|
||||
if (comment.body.startsWith(CONVENTIONAL_COMMENT_IDENTIFIER)) {
|
||||
return comment;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
exports.isConventionalTitle = isConventionalTitle;
|
||||
exports.updateCommentForEmbedding = updateCommentForEmbedding;
|
||||
45
.github/scripts/conventional-commit-pr-title-reminder/utils.unit.spec.js
vendored
Normal file
45
.github/scripts/conventional-commit-pr-title-reminder/utils.unit.spec.js
vendored
Normal file
@@ -0,0 +1,45 @@
|
||||
const { isConventionalTitle } = require("./utils");
|
||||
|
||||
describe("isConventionalTitle", () => {
|
||||
it("returns true for a simple conventional commit", () => {
|
||||
expect(isConventionalTitle("feat: add new feature")).toBe(false);
|
||||
});
|
||||
|
||||
it("returns true for a commit with a scope", () => {
|
||||
expect(isConventionalTitle("fix(api): correct API bug")).toBe(true);
|
||||
});
|
||||
|
||||
it("returns true for a commit with multiple scopes, comma separated", () => {
|
||||
expect(isConventionalTitle("feat(build,deps): update dependencies")).toBe(
|
||||
true,
|
||||
);
|
||||
});
|
||||
|
||||
it("returns true for a commit with multiple scopes, comma separated with space", () => {
|
||||
expect(isConventionalTitle("feat(build, deps): update dependencies")).toBe(
|
||||
true,
|
||||
);
|
||||
});
|
||||
|
||||
it("returns false for a commit with round brackets in a message", () => {
|
||||
expect(isConventionalTitle("feat: (something) update dependencies")).toBe(
|
||||
false,
|
||||
);
|
||||
});
|
||||
|
||||
it("returns false for missing colon", () => {
|
||||
expect(isConventionalTitle("feat add feature")).toBe(false);
|
||||
});
|
||||
|
||||
it("returns false for invalid type", () => {
|
||||
expect(isConventionalTitle("invalidType: some message")).toBe(false);
|
||||
});
|
||||
|
||||
it("returns false for empty message", () => {
|
||||
expect(isConventionalTitle("feat:")).toBe(false);
|
||||
});
|
||||
|
||||
it("returns false for empty input", () => {
|
||||
expect(isConventionalTitle("")).toBe(false);
|
||||
});
|
||||
});
|
||||
91
.github/scripts/csv-to-mb.js
vendored
Normal file
91
.github/scripts/csv-to-mb.js
vendored
Normal file
@@ -0,0 +1,91 @@
|
||||
/**
|
||||
* jsonData must be in the format:
|
||||
* [
|
||||
* { column1: value1, column2: value2, ... },
|
||||
* { column1: value3, column2: value4, ... },
|
||||
* ...
|
||||
* ]
|
||||
*/
|
||||
function jsonToCsvFormData(jsonData) {
|
||||
const formData = new FormData();
|
||||
const csvHeader = Object.keys(jsonData[0]).join(',') + '\n';
|
||||
const csvContent = jsonData.map(row =>
|
||||
Object.values(row).map(value => `"${String(value).replace(/"/g, '""')}"`).join(',')
|
||||
).join('\n');
|
||||
|
||||
const blob = new Blob([csvHeader + csvContent], { type: 'text/csv' });
|
||||
formData.append('file', blob, 'data.csv');
|
||||
|
||||
return formData;
|
||||
}
|
||||
|
||||
// Transient failures worth retrying: throttling and server-side errors. A 4xx
|
||||
// (bad request, auth) won't succeed on retry, so those fail fast.
|
||||
const RETRYABLE_STATUS = new Set([429, 500, 502, 503, 504]);
|
||||
|
||||
class UploadError extends Error {
|
||||
constructor(message, { retryable }) {
|
||||
super(message);
|
||||
this.retryable = retryable;
|
||||
}
|
||||
}
|
||||
|
||||
async function attemptUpload(url, jsonData, timeoutMs) {
|
||||
let response;
|
||||
try {
|
||||
response = await fetch(url, {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
"x-api-key": process.env.API_KEY
|
||||
},
|
||||
// Rebuilt per attempt: a consumed request body can't be reused.
|
||||
body: jsonToCsvFormData(jsonData),
|
||||
// Without this the request can hang indefinitely on a stalled server,
|
||||
// eating the whole job timeout. An abort surfaces below as a retryable
|
||||
// network error, so it retries with backoff instead.
|
||||
signal: AbortSignal.timeout(timeoutMs)
|
||||
});
|
||||
} catch (networkError) {
|
||||
throw new UploadError(networkError.message, { retryable: true });
|
||||
}
|
||||
|
||||
if (response.ok) {
|
||||
return;
|
||||
}
|
||||
|
||||
const body = await response.text().catch(() => '');
|
||||
throw new UploadError(
|
||||
`Upload failed: ${response.status} ${response.statusText} ${body}`.trim(),
|
||||
{ retryable: RETRYABLE_STATUS.has(response.status) }
|
||||
);
|
||||
}
|
||||
|
||||
const sleep = ms => new Promise(resolve => setTimeout(resolve, ms));
|
||||
|
||||
/**
|
||||
* Uploads jsonData as CSV, retrying transient API errors with exponential
|
||||
* backoff. `retries` is the number of extra attempts after the first (so the
|
||||
* default 2 means up to 3 tries). `timeoutMs` is a per-attempt total deadline
|
||||
* (default 60s) that aborts a stalled request. Throws the last error once
|
||||
* attempts are exhausted or the error is non-retryable.
|
||||
*/
|
||||
async function uploadCsvToMb({ baseUrl, tableId, jsonData, mode = 'append', retries = 2, retryDelayMs = 1000, timeoutMs = 60_000 }) {
|
||||
const operation = mode === 'replace' ? 'replace-csv' : 'append-csv';
|
||||
const url = `${baseUrl}/api/table/${tableId}/${operation}`;
|
||||
|
||||
for (let attempt = 0; ; attempt++) {
|
||||
try {
|
||||
await attemptUpload(url, jsonData, timeoutMs);
|
||||
return { success: true };
|
||||
} catch (error) {
|
||||
if (!error.retryable || attempt >= retries) {
|
||||
throw error;
|
||||
}
|
||||
const delay = retryDelayMs * 2 ** attempt;
|
||||
console.warn(`Upload attempt ${attempt + 1} failed (${error.message}); retrying in ${delay}ms`);
|
||||
await sleep(delay);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = { uploadCsvToMb };
|
||||
50
.github/scripts/csv-to-mb.unit.spec.js
vendored
Normal file
50
.github/scripts/csv-to-mb.unit.spec.js
vendored
Normal file
@@ -0,0 +1,50 @@
|
||||
const { uploadCsvToMb } = require("./csv-to-mb");
|
||||
|
||||
const ok = () => ({ ok: true });
|
||||
const fail = status => ({ ok: false, status, statusText: "err", text: async () => "" });
|
||||
|
||||
const upload = (over = {}) =>
|
||||
uploadCsvToMb({
|
||||
baseUrl: "https://stats.example.com",
|
||||
tableId: 1,
|
||||
jsonData: [{ a: 1 }],
|
||||
retryDelayMs: 1,
|
||||
...over,
|
||||
});
|
||||
|
||||
describe("uploadCsvToMb retries", () => {
|
||||
afterEach(() => {
|
||||
delete global.fetch;
|
||||
});
|
||||
|
||||
it("retries transient server errors then succeeds", async () => {
|
||||
global.fetch = jest.fn().mockResolvedValueOnce(fail(503)).mockResolvedValueOnce(fail(500)).mockResolvedValueOnce(ok());
|
||||
|
||||
await expect(upload({ retries: 2 })).resolves.toEqual({ success: true });
|
||||
expect(global.fetch).toHaveBeenCalledTimes(3);
|
||||
});
|
||||
|
||||
it("retries network errors", async () => {
|
||||
global.fetch = jest
|
||||
.fn()
|
||||
.mockRejectedValueOnce(new Error("ECONNRESET"))
|
||||
.mockResolvedValueOnce(ok());
|
||||
|
||||
await expect(upload({ retries: 2 })).resolves.toEqual({ success: true });
|
||||
expect(global.fetch).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
|
||||
it("gives up after exhausting retries and throws the last error", async () => {
|
||||
global.fetch = jest.fn().mockResolvedValue(fail(500));
|
||||
|
||||
await expect(upload({ retries: 2 })).rejects.toThrow("Upload failed: 500");
|
||||
expect(global.fetch).toHaveBeenCalledTimes(3); // 1 initial + 2 retries
|
||||
});
|
||||
|
||||
it("does not retry a non-transient 4xx", async () => {
|
||||
global.fetch = jest.fn().mockResolvedValue(fail(400));
|
||||
|
||||
await expect(upload({ retries: 2 })).rejects.toThrow("Upload failed: 400");
|
||||
expect(global.fetch).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
});
|
||||
57
.github/scripts/file-paths.unit.spec.ts
vendored
57
.github/scripts/file-paths.unit.spec.ts
vendored
@@ -1,57 +0,0 @@
|
||||
import { readFileSync } from "fs";
|
||||
import { join } from "path";
|
||||
|
||||
import { load } from "js-yaml";
|
||||
import { isMatch } from "micromatch";
|
||||
|
||||
const FILTERS = join(__dirname, "..", "file-paths.yaml");
|
||||
|
||||
type Filter = string | Filter[] | { [changeType: string]: Filter };
|
||||
|
||||
// A filter is a list of patterns, and an entry may name the change types it applies to as
|
||||
// { "added|modified": pattern }. Either way the patterns are what matters here.
|
||||
const patterns = (value: Filter): string[] => {
|
||||
if (typeof value === "string") return [value];
|
||||
if (Array.isArray(value)) return value.flatMap(patterns);
|
||||
return Object.values(value).flatMap(patterns);
|
||||
};
|
||||
|
||||
describe("file-paths.yaml", () => {
|
||||
const filters = load(readFileSync(FILTERS, "utf8")) as Record<string, Filter>;
|
||||
|
||||
// dorny/paths-filter matches with picomatch, which micromatch wraps, so this asks the same
|
||||
// question CI asks of a changed path.
|
||||
const matches = (name: string, path: string) =>
|
||||
patterns(filters[name]).some((pattern) =>
|
||||
isMatch(path, pattern, { dot: true }),
|
||||
);
|
||||
|
||||
// dorny/paths-filter ORs the patterns in a filter, so a "!..." entry does not subtract from the
|
||||
// other entries - it matches every path they don't, turning the filter permanently on. Excluding
|
||||
// a path means writing the exclusion inside one pattern, the way backend_sources keeps
|
||||
// ratchets.edn out with ".clj-kondo/{!(ratchets.edn),*/**}".
|
||||
it("never excludes a path with a standalone negation", () => {
|
||||
const negated = Object.entries(filters).flatMap(([name, filter]) =>
|
||||
patterns(filter)
|
||||
.filter((pattern) => pattern.startsWith("!"))
|
||||
.map((pattern) => `${name}: ${pattern}`),
|
||||
);
|
||||
|
||||
expect(negated).toEqual([]);
|
||||
});
|
||||
|
||||
it.each(["backend_sources", "backend_all"])(
|
||||
"%s skips a ratchets-only change and still follows the rest of .clj-kondo",
|
||||
(name) => {
|
||||
expect(matches(name, ".clj-kondo/ratchets.edn")).toBe(false);
|
||||
expect(matches(name, ".clj-kondo/config.edn")).toBe(true);
|
||||
expect(matches(name, ".clj-kondo/config/modules/config.edn")).toBe(true);
|
||||
},
|
||||
);
|
||||
|
||||
it("runs the ratchet check on the ratchets file", () => {
|
||||
expect(matches("project_ratchet_checks", ".clj-kondo/ratchets.edn")).toBe(
|
||||
true,
|
||||
);
|
||||
});
|
||||
});
|
||||
40
.github/scripts/find-linear-issue.js
vendored
Normal file
40
.github/scripts/find-linear-issue.js
vendored
Normal file
@@ -0,0 +1,40 @@
|
||||
async function findLinearIssue({ githubIssueUrl, linearApiKey }) {
|
||||
const response = await fetch("https://api.linear.app/graphql", {
|
||||
method: "POST",
|
||||
headers: {
|
||||
"Content-Type": "application/json",
|
||||
Authorization: linearApiKey,
|
||||
},
|
||||
body: JSON.stringify({
|
||||
query: `
|
||||
query {
|
||||
attachmentsForURL(url: "${githubIssueUrl}") {
|
||||
nodes {
|
||||
issue { identifier }
|
||||
}
|
||||
}
|
||||
}
|
||||
`,
|
||||
}),
|
||||
});
|
||||
|
||||
const data = await response.json();
|
||||
|
||||
if (!response.ok) {
|
||||
return { success: false, message: `Linear API error: ${response.status}` };
|
||||
}
|
||||
|
||||
if (!data.data?.attachmentsForURL?.nodes?.length) {
|
||||
return {
|
||||
success: false,
|
||||
message: `No Linear issue linked to ${githubIssueUrl}`,
|
||||
};
|
||||
}
|
||||
|
||||
return {
|
||||
success: true,
|
||||
issue: data.data.attachmentsForURL.nodes[0].issue.identifier,
|
||||
};
|
||||
}
|
||||
|
||||
module.exports = { findLinearIssue };
|
||||
66
.github/scripts/fix-yarn-lock-aliases.ts
vendored
66
.github/scripts/fix-yarn-lock-aliases.ts
vendored
@@ -1,66 +0,0 @@
|
||||
/**
|
||||
* Renames aliased dependency keys in a bun-generated yarn.lock.
|
||||
*
|
||||
* Yarn names an alias entry after the alias, for example
|
||||
* `"typescript7@npm:typescript@7.0.2"`. Bun names it after the aliased package
|
||||
* instead, for example `"typescript@npm:typescript@7.0.2"`. Snyk follows the
|
||||
* yarn convention and fails to resolve the dependency.
|
||||
*/
|
||||
|
||||
import { readFile, writeFile } from "node:fs/promises";
|
||||
|
||||
async function main() {
|
||||
const packageJsonPath = `${process.cwd()}/package.json`;
|
||||
const lockPath = `${process.cwd()}/yarn.lock`;
|
||||
|
||||
const packageJson = JSON.parse(await readFile(packageJsonPath, "utf8"));
|
||||
|
||||
const dependencies = Object.entries<string>(
|
||||
Object.assign(
|
||||
{},
|
||||
packageJson.dependencies,
|
||||
packageJson.devDependencies,
|
||||
packageJson.optionalDependencies,
|
||||
),
|
||||
);
|
||||
|
||||
let lock = await readFile(lockPath, "utf8");
|
||||
const renamedKeys: string[] = [];
|
||||
|
||||
for (const [alias, specifier] of dependencies) {
|
||||
if (!specifier.startsWith("npm:")) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const aliasedPackage = specifier
|
||||
.slice("npm:".length)
|
||||
.replace(/@[^@]*$/, "");
|
||||
|
||||
if (aliasedPackage === alias) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const bunKey = `"${aliasedPackage}@${specifier}"`;
|
||||
const yarnKey = `"${alias}@${specifier}"`;
|
||||
|
||||
if (lock.includes(bunKey)) {
|
||||
lock = lock.replaceAll(bunKey, yarnKey);
|
||||
renamedKeys.push(`${bunKey} -> ${yarnKey}`);
|
||||
}
|
||||
}
|
||||
|
||||
if (renamedKeys.length > 0) {
|
||||
await writeFile(lockPath, lock);
|
||||
}
|
||||
|
||||
process.stdout.write(
|
||||
renamedKeys.length > 0
|
||||
? `Renamed alias keys in yarn.lock:\n${renamedKeys.join("\n")}\n`
|
||||
: "No alias keys to rename in yarn.lock\n",
|
||||
);
|
||||
}
|
||||
|
||||
main().catch((error) => {
|
||||
process.stderr.write(`Failed to rename alias keys in yarn.lock: ${error}\n`);
|
||||
process.exit(1);
|
||||
});
|
||||
84
.github/scripts/get-issue-data.js
vendored
Normal file
84
.github/scripts/get-issue-data.js
vendored
Normal file
@@ -0,0 +1,84 @@
|
||||
const githubSlackMap = require("../../release/github-slack-map.json");
|
||||
|
||||
async function getIssueData({ context, github, manualIssueNumber }) {
|
||||
let issueNumber;
|
||||
let issueData;
|
||||
let prReviews = [];
|
||||
const triggerType = context.eventName;
|
||||
|
||||
if (context.eventName === "issues") {
|
||||
issueNumber = context.payload.issue.number;
|
||||
issueData = context.payload.issue;
|
||||
} else if (context.eventName === "workflow_dispatch") {
|
||||
issueNumber = parseInt(manualIssueNumber);
|
||||
const { data } = await github.rest.issues.get({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
issue_number: issueNumber,
|
||||
});
|
||||
issueData = data;
|
||||
} else if (context.eventName === "pull_request_review") {
|
||||
const prNumber = context.payload.pull_request.number;
|
||||
const { data: prData } = await github.rest.pulls.get({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
pull_number: prNumber,
|
||||
});
|
||||
|
||||
// Extract issue number from branch name (e.g., "claude-fix/issue-12345-some-title")
|
||||
const branchMatch = prData.head.ref.match(/claude-fix\/issue-(\d+)/);
|
||||
if (!branchMatch) {
|
||||
throw new Error(
|
||||
`Could not extract issue number from branch name: ${prData.head.ref}`,
|
||||
);
|
||||
}
|
||||
|
||||
issueNumber = parseInt(branchMatch[1]);
|
||||
|
||||
const { data } = await github.rest.issues.get({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
issue_number: issueNumber,
|
||||
});
|
||||
issueData = data;
|
||||
|
||||
const { data: reviews } = await github.rest.pulls.listReviews({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
pull_number: prNumber,
|
||||
});
|
||||
|
||||
prReviews = reviews.filter(
|
||||
(review) =>
|
||||
review.state === "CHANGES_REQUESTED" &&
|
||||
isMetabaseMember(review.user.login),
|
||||
);
|
||||
}
|
||||
|
||||
const { data: comments } = await github.rest.issues.listComments({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
issue_number: issueNumber,
|
||||
});
|
||||
|
||||
const employeeComments = comments.filter((comment) =>
|
||||
isMetabaseMember(comment.user.login),
|
||||
);
|
||||
|
||||
return {
|
||||
issueNumber,
|
||||
issueData,
|
||||
employeeComments,
|
||||
prReviews,
|
||||
triggerType,
|
||||
isIssueAuthorEmployee: isMetabaseMember(issueData.user.login),
|
||||
};
|
||||
}
|
||||
|
||||
function isMetabaseMember(username) {
|
||||
return username in githubSlackMap;
|
||||
}
|
||||
|
||||
module.exports = {
|
||||
getIssueData,
|
||||
};
|
||||
109
.github/scripts/linear-issues.js
vendored
109
.github/scripts/linear-issues.js
vendored
@@ -1,109 +0,0 @@
|
||||
async function findLinearIssue({ githubIssueUrl, linearApiKey }) {
|
||||
const response = await fetch("https://api.linear.app/graphql", {
|
||||
method: "POST",
|
||||
headers: {
|
||||
"Content-Type": "application/json",
|
||||
Authorization: linearApiKey,
|
||||
},
|
||||
body: JSON.stringify({
|
||||
query: `
|
||||
query {
|
||||
attachmentsForURL(url: "${githubIssueUrl}") {
|
||||
nodes {
|
||||
issue { identifier }
|
||||
}
|
||||
}
|
||||
}
|
||||
`,
|
||||
}),
|
||||
});
|
||||
|
||||
const data = await response.json();
|
||||
|
||||
if (!response.ok) {
|
||||
return { success: false, message: `Linear API error: ${response.status}` };
|
||||
}
|
||||
|
||||
if (!data.data?.attachmentsForURL?.nodes?.length) {
|
||||
return {
|
||||
success: false,
|
||||
message: `No Linear issue linked to ${githubIssueUrl}`,
|
||||
};
|
||||
}
|
||||
|
||||
return {
|
||||
success: true,
|
||||
issue: data.data.attachmentsForURL.nodes[0].issue.identifier,
|
||||
};
|
||||
}
|
||||
|
||||
async function createLinearIssue({
|
||||
teamId,
|
||||
title,
|
||||
description,
|
||||
projectId,
|
||||
labelIds,
|
||||
linearApiKey,
|
||||
}) {
|
||||
const response = await fetch("https://api.linear.app/graphql", {
|
||||
method: "POST",
|
||||
headers: {
|
||||
"Content-Type": "application/json",
|
||||
Authorization: linearApiKey,
|
||||
},
|
||||
body: JSON.stringify({
|
||||
query: `
|
||||
mutation CreateIssue($input: IssueCreateInput!) {
|
||||
issueCreate(input: $input) {
|
||||
success
|
||||
issue {
|
||||
id
|
||||
identifier
|
||||
title
|
||||
url
|
||||
}
|
||||
}
|
||||
}
|
||||
`,
|
||||
variables: {
|
||||
input: {
|
||||
teamId,
|
||||
title,
|
||||
...(description ? { description } : {}),
|
||||
...(projectId ? { projectId } : {}),
|
||||
...(labelIds?.length ? { labelIds } : {}),
|
||||
},
|
||||
},
|
||||
}),
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
return { success: false, message: `Linear API error: ${response.status}` };
|
||||
}
|
||||
|
||||
const data = await response.json();
|
||||
|
||||
if (data.errors?.length) {
|
||||
return {
|
||||
success: false,
|
||||
message: `Linear API error: ${data.errors.map((e) => e.message).join(", ")}`,
|
||||
};
|
||||
}
|
||||
|
||||
if (!data.data?.issueCreate?.success) {
|
||||
return {
|
||||
success: false,
|
||||
message: `Failed to create Linear issue "${title}"`,
|
||||
};
|
||||
}
|
||||
|
||||
return {
|
||||
success: true,
|
||||
issue: data.data.issueCreate.issue,
|
||||
};
|
||||
}
|
||||
|
||||
module.exports = {
|
||||
findLinearIssue,
|
||||
createLinearIssue,
|
||||
};
|
||||
65
.github/scripts/run-presto-kerberos-integration-test.sh
vendored
Executable file
65
.github/scripts/run-presto-kerberos-integration-test.sh
vendored
Executable file
@@ -0,0 +1,65 @@
|
||||
#! /usr/bin/env bash
|
||||
# runs one or more Metabase test(s) against a Kerberized Presto instance
|
||||
set -eo pipefail
|
||||
|
||||
# Need Java commands on $PATH, which apparently is not yet the case
|
||||
export JAVA_HOME="/usr/lib/jvm/msopenjdk-current"
|
||||
export PATH="$PATH:$JAVA_HOME/bin"
|
||||
|
||||
# ensure java commmand is available
|
||||
which java
|
||||
|
||||
# install clojure version needed for Metabase
|
||||
curl -O https://download.clojure.org/install/linux-install-1.12.0.1488.sh
|
||||
chmod +x linux-install-1.12.0.1488.sh
|
||||
./linux-install-1.12.0.1488.sh
|
||||
|
||||
RESOURCES_DIR=/app/source/resources
|
||||
|
||||
# ensure the expected files are in place, in the resources dir
|
||||
if [ ! -f "$RESOURCES_DIR/ssl_keystore.jks" ]; then
|
||||
echo "$RESOURCES_DIR/ssl_keystore.jks does not exist; cannot run test" >&2
|
||||
exit 11
|
||||
fi
|
||||
|
||||
if [ ! -f "$RESOURCES_DIR/krb5.conf" ]; then
|
||||
echo "$RESOURCES_DIR/krb5.conf does not exist; cannot run test" >&2
|
||||
exit 12
|
||||
fi
|
||||
|
||||
if [ ! -f "$RESOURCES_DIR/client.keytab" ]; then
|
||||
echo "$RESOURCES_DIR/client.keytab does not exist; cannot run test" >&2
|
||||
exit 13
|
||||
fi
|
||||
|
||||
# Copy the JDK cacerts file to our resources
|
||||
cp $JAVA_HOME/lib/security/cacerts $RESOURCES_DIR/cacerts-with-presto-ca.jks
|
||||
|
||||
# Capture the Presto server self signed CA in PEM format
|
||||
openssl s_client -showcerts -connect presto-kerberos:7778 </dev/null \
|
||||
| openssl x509 -outform PEM >$RESOURCES_DIR/presto-ssl-root-ca.pem
|
||||
|
||||
# Convert the Presto server self signed CA to DER format
|
||||
openssl x509 -outform der -in $RESOURCES_DIR/presto-ssl-root-ca.pem -out $RESOURCES_DIR/presto-ssl-root-ca.der
|
||||
|
||||
# Add Presto's self signed CA to the truststore
|
||||
keytool -noprompt -import -alias presto-kerberos -keystore $RESOURCES_DIR/cacerts-with-presto-ca.jks \
|
||||
-storepass changeit -file $RESOURCES_DIR/presto-ssl-root-ca.der -trustcacerts
|
||||
|
||||
# Set up the environment variables pointing to all of this, and run some tests
|
||||
DRIVERS=presto-jdbc \
|
||||
MB_ENABLE_PRESTO_JDBC_DRIVER=true \
|
||||
MB_PRESTO_JDBC_TEST_HOST=presto-kerberos \
|
||||
MB_PRESTO_JDBC_TEST_PORT=7778 \
|
||||
MB_PRESTO_JDBC_TEST_SSL=true \
|
||||
MB_PRESTO_JDBC_TEST_SSL_KEYSTORE_PATH=$RESOURCES_DIR/ssl_keystore.jks \
|
||||
MB_PRESTO_JDBC_TEST_SSL_KEYSTORE_PASSWORD=presto \
|
||||
MB_PRESTO_JDBC_TEST_SSL_TRUSTSTORE_PATH=$RESOURCES_DIR/cacerts-with-presto-ca.jks \
|
||||
MB_PRESTO_JDBC_TEST_SSL_TRUSTSTORE_PASSWORD=changeit \
|
||||
MB_PRESTO_JDBC_TEST_KERBEROS=true \
|
||||
MB_PRESTO_JDBC_TEST_USER=bob@EXAMPLE.COM \
|
||||
MB_PRESTO_JDBC_TEST_KERBEROS_PRINCIPAL=bob@EXAMPLE.COM \
|
||||
MB_PRESTO_JDBC_TEST_KERBEROS_REMOTE_SERVICE_NAME=HTTP \
|
||||
MB_PRESTO_JDBC_TEST_KERBEROS_KEYTAB_PATH=$RESOURCES_DIR/client.keytab \
|
||||
MB_PRESTO_JDBC_TEST_KERBEROS_CONFIG_PATH=$RESOURCES_DIR/krb5.conf \
|
||||
clojure -X:dev:test:drivers:drivers-dev :only metabase.driver.presto-jdbc-test
|
||||
3
.github/scripts/setup-python-runner.sh
vendored
3
.github/scripts/setup-python-runner.sh
vendored
@@ -19,7 +19,8 @@ while [[ $# -gt 0 ]]; do
|
||||
done
|
||||
|
||||
echo "Cloning python-runner repository..."
|
||||
git clone --depth 1 https://${METABASE_AUTOMATION_USER_TOKEN}@github.com/metabase/python-runner-container.git python-runner
|
||||
git clone --depth 1 https://${METABASE_AUTOMATION_USER_TOKEN}@github.com/metabase/python-runner-container.git --branch seccomp python-runner
|
||||
|
||||
|
||||
if [ "$SKIP_START" = false ]; then
|
||||
echo "Starting python-runner..."
|
||||
|
||||
140
.github/scripts/stats-import.ts
vendored
140
.github/scripts/stats-import.ts
vendored
@@ -1,140 +0,0 @@
|
||||
// Pushes stats rows to eng-stats-importer, which writes them to Postgres.
|
||||
//
|
||||
// The importer's origin comes from ENG_STATS_URL and its key from API_KEY, both
|
||||
// repository secrets — this repo is public, so neither is hardcoded here. Visit
|
||||
// the importer to see the tables and mint keys.
|
||||
|
||||
// Transient failures worth retrying: throttling and server-side errors. A 4xx
|
||||
// (bad request, auth, unknown table) won't succeed on retry, so those fail fast.
|
||||
const RETRYABLE_STATUS = new Set([429, 500, 502, 503, 504]);
|
||||
|
||||
/**
|
||||
* One row, keyed by column. Values are `unknown` rather than a scalar union
|
||||
* because a `json` column takes arbitrary nested data.
|
||||
*/
|
||||
export type StatsRow = Record<string, unknown>;
|
||||
|
||||
export interface ImportStatsOptions {
|
||||
/** Table name as it appears in `/api/import/{table}`, e.g. `emotion_files`. */
|
||||
table: string;
|
||||
rows: StatsRow[];
|
||||
/** Importer origin. Defaults to ENG_STATS_URL. */
|
||||
baseUrl?: string;
|
||||
/** Extra attempts after the first, so the default 2 means up to 3 tries. */
|
||||
retries?: number;
|
||||
retryDelayMs?: number;
|
||||
/** Per-attempt total deadline; aborts a stalled request. */
|
||||
timeoutMs?: number;
|
||||
}
|
||||
|
||||
export interface ImportStatsResult {
|
||||
success: true;
|
||||
inserted?: number;
|
||||
/** Payload keys with no matching column, when `ignoreUnknown` is in play. */
|
||||
ignoredKeys?: string[];
|
||||
}
|
||||
|
||||
class ImportError extends Error {
|
||||
retryable: boolean;
|
||||
|
||||
constructor(message: string, { retryable }: { retryable: boolean }) {
|
||||
super(message);
|
||||
this.retryable = retryable;
|
||||
}
|
||||
}
|
||||
|
||||
async function attemptImport(
|
||||
url: string,
|
||||
rows: StatsRow[],
|
||||
timeoutMs: number,
|
||||
): Promise<ImportStatsResult> {
|
||||
let response: Response;
|
||||
try {
|
||||
response = await fetch(url, {
|
||||
method: "PUT",
|
||||
headers: {
|
||||
"x-api-key": process.env.API_KEY ?? "",
|
||||
"content-type": "application/json",
|
||||
},
|
||||
body: JSON.stringify(rows),
|
||||
// Without this the request can hang indefinitely on a stalled server,
|
||||
// eating the whole job timeout. An abort surfaces below as a retryable
|
||||
// network error, so it retries with backoff instead.
|
||||
signal: AbortSignal.timeout(timeoutMs),
|
||||
});
|
||||
} catch (networkError) {
|
||||
const message =
|
||||
networkError instanceof Error ? networkError.message : String(networkError);
|
||||
throw new ImportError(message, { retryable: true });
|
||||
}
|
||||
|
||||
if (!response.ok) {
|
||||
// Failures carry an `{ error }` body naming the offending row and column,
|
||||
// which is the only way to tell a type mismatch from a typo'd key.
|
||||
const body = await response.text().catch(() => "");
|
||||
throw new ImportError(
|
||||
`Import failed: ${response.status} ${response.statusText} ${body}`.trim(),
|
||||
{ retryable: RETRYABLE_STATUS.has(response.status) },
|
||||
);
|
||||
}
|
||||
|
||||
// Best-effort: the row count is for logging, so a body that doesn't parse
|
||||
// shouldn't turn a successful import into a failure.
|
||||
const result = (await response.json().catch(() => ({}))) as {
|
||||
inserted?: number;
|
||||
ignoredKeys?: string[];
|
||||
};
|
||||
return { success: true, inserted: result.inserted, ignoredKeys: result.ignoredKeys };
|
||||
}
|
||||
|
||||
const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));
|
||||
|
||||
/**
|
||||
* Imports `rows` into the `table` table, retrying transient API errors with
|
||||
* exponential backoff.
|
||||
*
|
||||
* Keys are matched leniently against the table's columns -- `Emotion Files`,
|
||||
* `emotionFiles` and `emotion_files` all land in `emotion_files` -- but values
|
||||
* are matched strictly, and a key matching no column is an error rather than a
|
||||
* silent drop. Rows are appended; the importer has no replace mode.
|
||||
*
|
||||
* Throws the last error once attempts are exhausted or the error is
|
||||
* non-retryable.
|
||||
*/
|
||||
export async function importStats({
|
||||
table,
|
||||
rows,
|
||||
baseUrl = process.env.ENG_STATS_URL,
|
||||
retries = 2,
|
||||
retryDelayMs = 1000,
|
||||
timeoutMs = 60_000,
|
||||
}: ImportStatsOptions): Promise<ImportStatsResult> {
|
||||
// The importer accepts an empty payload, but there's no point spending a
|
||||
// request (and a rate-limit slot) to insert nothing.
|
||||
if (rows.length === 0) {
|
||||
return { success: true, inserted: 0 };
|
||||
}
|
||||
|
||||
if (!baseUrl) {
|
||||
throw new Error("ENG_STATS_URL is not set, so there is no importer to push to");
|
||||
}
|
||||
|
||||
// Tolerate a trailing slash: the origin comes from a hand-entered secret.
|
||||
const url = `${baseUrl.replace(/\/+$/, "")}/api/import/${table}`;
|
||||
|
||||
for (let attempt = 0; ; attempt++) {
|
||||
try {
|
||||
return await attemptImport(url, rows, timeoutMs);
|
||||
} catch (error) {
|
||||
const importError = error as ImportError;
|
||||
if (!importError.retryable || attempt >= retries) {
|
||||
throw error;
|
||||
}
|
||||
const delay = retryDelayMs * 2 ** attempt;
|
||||
console.warn(
|
||||
`Import attempt ${attempt + 1} failed (${importError.message}); retrying in ${delay}ms`,
|
||||
);
|
||||
await sleep(delay);
|
||||
}
|
||||
}
|
||||
}
|
||||
160
.github/scripts/stats-import.unit.spec.ts
vendored
160
.github/scripts/stats-import.unit.spec.ts
vendored
@@ -1,160 +0,0 @@
|
||||
import { importStats } from "./stats-import";
|
||||
|
||||
const ok = (body: unknown = { inserted: 1 }) => ({ ok: true, json: async () => body });
|
||||
const fail = (status: number) => ({
|
||||
ok: false,
|
||||
status,
|
||||
statusText: "err",
|
||||
text: async () => "",
|
||||
});
|
||||
|
||||
const load = (over: Record<string, unknown> = {}) =>
|
||||
importStats({
|
||||
baseUrl: "https://stats.example.com",
|
||||
table: "emotion_files",
|
||||
rows: [{ a: 1 }],
|
||||
retryDelayMs: 1,
|
||||
...over,
|
||||
});
|
||||
|
||||
describe("importStats requests", () => {
|
||||
afterEach(() => {
|
||||
// @ts-expect-error -- removing the stub the tests installed
|
||||
delete global.fetch;
|
||||
delete process.env.API_KEY;
|
||||
delete process.env.ENG_STATS_URL;
|
||||
});
|
||||
|
||||
it("defaults the origin to ENG_STATS_URL", async () => {
|
||||
const fetchMock = jest.fn().mockResolvedValue(ok());
|
||||
global.fetch = fetchMock;
|
||||
process.env.ENG_STATS_URL = "https://importer.example.com";
|
||||
|
||||
await load({ baseUrl: undefined });
|
||||
|
||||
expect(fetchMock.mock.calls[0][0]).toBe(
|
||||
"https://importer.example.com/api/import/emotion_files",
|
||||
);
|
||||
});
|
||||
|
||||
it("tolerates a trailing slash on the origin", async () => {
|
||||
const fetchMock = jest.fn().mockResolvedValue(ok());
|
||||
global.fetch = fetchMock;
|
||||
|
||||
await load({ baseUrl: "https://importer.example.com/" });
|
||||
|
||||
expect(fetchMock.mock.calls[0][0]).toBe(
|
||||
"https://importer.example.com/api/import/emotion_files",
|
||||
);
|
||||
});
|
||||
|
||||
it("fails loudly when no origin is configured", async () => {
|
||||
const fetchMock = jest.fn();
|
||||
global.fetch = fetchMock;
|
||||
|
||||
await expect(load({ baseUrl: undefined })).rejects.toThrow("ENG_STATS_URL is not set");
|
||||
expect(fetchMock).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("PUTs the rows as JSON to the table's import endpoint", async () => {
|
||||
const fetchMock = jest.fn().mockResolvedValue(ok());
|
||||
global.fetch = fetchMock;
|
||||
process.env.API_KEY = "secret";
|
||||
|
||||
await expect(load({ rows: [{ a: 1 }, { a: 2 }] })).resolves.toEqual({
|
||||
success: true,
|
||||
inserted: 1,
|
||||
ignoredKeys: undefined,
|
||||
});
|
||||
|
||||
const [url, options] = fetchMock.mock.calls[0];
|
||||
expect(url).toBe("https://stats.example.com/api/import/emotion_files");
|
||||
expect(options.method).toBe("PUT");
|
||||
expect(options.headers).toMatchObject({
|
||||
"x-api-key": "secret",
|
||||
"content-type": "application/json",
|
||||
});
|
||||
expect(JSON.parse(options.body)).toEqual([{ a: 1 }, { a: 2 }]);
|
||||
});
|
||||
|
||||
it("skips the request entirely when there are no rows", async () => {
|
||||
const fetchMock = jest.fn();
|
||||
global.fetch = fetchMock;
|
||||
|
||||
await expect(load({ rows: [] })).resolves.toEqual({ success: true, inserted: 0 });
|
||||
expect(fetchMock).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("succeeds even if the response body does not parse", async () => {
|
||||
global.fetch = jest.fn().mockResolvedValue({
|
||||
ok: true,
|
||||
json: async () => {
|
||||
throw new Error("not json");
|
||||
},
|
||||
});
|
||||
|
||||
await expect(load()).resolves.toMatchObject({ success: true });
|
||||
});
|
||||
});
|
||||
|
||||
describe("importStats retries", () => {
|
||||
afterEach(() => {
|
||||
// @ts-expect-error -- removing the stub the tests installed
|
||||
delete global.fetch;
|
||||
});
|
||||
|
||||
it("retries transient server errors then succeeds", async () => {
|
||||
const fetchMock = jest
|
||||
.fn()
|
||||
.mockResolvedValueOnce(fail(503))
|
||||
.mockResolvedValueOnce(fail(500))
|
||||
.mockResolvedValueOnce(ok());
|
||||
global.fetch = fetchMock;
|
||||
|
||||
await expect(load({ retries: 2 })).resolves.toMatchObject({ success: true });
|
||||
expect(fetchMock).toHaveBeenCalledTimes(3);
|
||||
});
|
||||
|
||||
it("retries throttling", async () => {
|
||||
const fetchMock = jest.fn().mockResolvedValueOnce(fail(429)).mockResolvedValueOnce(ok());
|
||||
global.fetch = fetchMock;
|
||||
|
||||
await expect(load({ retries: 2 })).resolves.toMatchObject({ success: true });
|
||||
expect(fetchMock).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
|
||||
it("retries network errors", async () => {
|
||||
const fetchMock = jest
|
||||
.fn()
|
||||
.mockRejectedValueOnce(new Error("ECONNRESET"))
|
||||
.mockResolvedValueOnce(ok());
|
||||
global.fetch = fetchMock;
|
||||
|
||||
await expect(load({ retries: 2 })).resolves.toMatchObject({ success: true });
|
||||
expect(fetchMock).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
|
||||
it("gives up after exhausting retries and throws the last error", async () => {
|
||||
const fetchMock = jest.fn().mockResolvedValue(fail(500));
|
||||
global.fetch = fetchMock;
|
||||
|
||||
await expect(load({ retries: 2 })).rejects.toThrow("Import failed: 500");
|
||||
expect(fetchMock).toHaveBeenCalledTimes(3); // 1 initial + 2 retries
|
||||
});
|
||||
|
||||
it("does not retry a non-transient 4xx", async () => {
|
||||
const fetchMock = jest.fn().mockResolvedValue(fail(400));
|
||||
global.fetch = fetchMock;
|
||||
|
||||
await expect(load({ retries: 2 })).rejects.toThrow("Import failed: 400");
|
||||
expect(fetchMock).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it("does not retry an unknown table", async () => {
|
||||
const fetchMock = jest.fn().mockResolvedValue(fail(404));
|
||||
global.fetch = fetchMock;
|
||||
|
||||
await expect(load({ retries: 2 })).rejects.toThrow("Import failed: 404");
|
||||
expect(fetchMock).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
});
|
||||
64
.github/scripts/upload-affected-tests-stats.js
vendored
Normal file
64
.github/scripts/upload-affected-tests-stats.js
vendored
Normal file
@@ -0,0 +1,64 @@
|
||||
// Appends the test-plan stats row produced by create-test-plan.ts to the
|
||||
// "FE Affected Tests" table on stats.metabase.com.
|
||||
// Reads STATS_JSON, TRIGGERED_BY, PR_NUMBER, HEAD_SHA, BASE_SHA, API_KEY from env.
|
||||
|
||||
const { uploadCsvToMb } = require("./csv-to-mb.js");
|
||||
|
||||
// "FE Affected Tests" uploaded table on stats.metabase.com.
|
||||
const TABLE_ID = 81818;
|
||||
|
||||
async function main() {
|
||||
// create-test-plan doesn't always produce stats (e.g. on release branches
|
||||
// running an older planner), leaving STATS_JSON empty. Nothing to upload.
|
||||
const raw = (process.env.STATS_JSON || "").trim();
|
||||
if (!raw || raw === "null") {
|
||||
console.log("No STATS_JSON to upload; skipping.");
|
||||
return;
|
||||
}
|
||||
|
||||
const s = JSON.parse(raw);
|
||||
console.log("stats", s);
|
||||
|
||||
const row = {
|
||||
Date: new Date().toISOString(),
|
||||
// "pr_update" or "merge_to_master". ("Trigger" is a reserved SQL word.)
|
||||
"Triggered By": process.env.TRIGGERED_BY,
|
||||
PR: Number(process.env.PR_NUMBER),
|
||||
"Head SHA": process.env.HEAD_SHA,
|
||||
"Base SHA": process.env.BASE_SHA,
|
||||
"FE Files Changed": s.fe_files_changed,
|
||||
"FE Files Total": s.fe_files_total,
|
||||
"BE Files Changed": s.be_files_changed,
|
||||
"BE Files Total": s.be_files_total,
|
||||
"Unit Infra Touched": s.unit_infra_touched,
|
||||
"Loki Infra Touched": s.loki_infra_touched,
|
||||
"Shared Sources Touched": s.shared_sources_touched,
|
||||
"FE Modules Total": s.fe_modules_total,
|
||||
"FE Modules Changed": s.fe_modules_changed,
|
||||
"FE Modules Affected (Rules)": s.fe_modules_affected_rules,
|
||||
"FE Modules Affected (Usage)": s.fe_modules_affected_usage,
|
||||
"Unit Specs All": s.unit_specs_all,
|
||||
"Unit Specs To Run (Rules)": s.unit_specs_to_run_rules,
|
||||
"Unit Specs To Run (Usage)": s.unit_specs_to_run_usage,
|
||||
"Loki Stories All": s.loki_stories_all,
|
||||
"Loki Stories To Run (Rules)": s.loki_stories_to_run_rules,
|
||||
"Loki Stories To Run (Usage)": s.loki_stories_to_run_usage,
|
||||
"E2E Specs All": s.e2e_specs_all,
|
||||
"E2E Specs To Run (Rules)": s.e2e_specs_to_run_rules,
|
||||
"E2E Specs To Run (Usage)": s.e2e_specs_to_run_usage,
|
||||
};
|
||||
|
||||
await uploadCsvToMb({
|
||||
baseUrl: "https://stats.metabase.com",
|
||||
tableId: TABLE_ID,
|
||||
jsonData: [row],
|
||||
mode: "append",
|
||||
});
|
||||
console.log("Data uploaded successfully");
|
||||
}
|
||||
|
||||
// Best-effort: never let a telemetry failure turn the job red.
|
||||
main().catch((error) => {
|
||||
console.error("Skipping stats upload after error:", error);
|
||||
process.exit(0);
|
||||
});
|
||||
62
.github/scripts/upload-affected-tests-stats.ts
vendored
62
.github/scripts/upload-affected-tests-stats.ts
vendored
@@ -1,62 +0,0 @@
|
||||
// Appends the test-plan stats row produced by create-test-plan.ts to the
|
||||
// "FE Affected Tests" table in eng-stats-importer.
|
||||
// Reads STATS_JSON, PR_NUMBER, HEAD_SHA, BASE_SHA, API_KEY from env.
|
||||
|
||||
import type { TestPlanStats } from "./affected-tests";
|
||||
import { importStats } from "./stats-import";
|
||||
|
||||
const TABLE = "fe_affected_tests";
|
||||
|
||||
async function main() {
|
||||
// create-test-plan doesn't always produce stats (e.g. on release branches
|
||||
// running an older planner), leaving STATS_JSON empty. Nothing to upload.
|
||||
const raw = (process.env.STATS_JSON || "").trim();
|
||||
if (!raw || raw === "null") {
|
||||
console.log("No STATS_JSON to upload; skipping.");
|
||||
return;
|
||||
}
|
||||
|
||||
const s = JSON.parse(raw) as TestPlanStats;
|
||||
console.log("stats", s);
|
||||
|
||||
const row = {
|
||||
Date: new Date().toISOString(),
|
||||
// ("Trigger" is a reserved SQL word.)
|
||||
"Triggered By": "pr_update",
|
||||
PR: Number(process.env.PR_NUMBER),
|
||||
"Head SHA": process.env.HEAD_SHA,
|
||||
"Base SHA": process.env.BASE_SHA,
|
||||
"FE Files Changed": s.fe_files_changed,
|
||||
"FE Files Total": s.fe_files_total,
|
||||
"BE Files Changed": s.be_files_changed,
|
||||
"BE Files Total": s.be_files_total,
|
||||
"Unit Infra Touched": s.unit_infra_touched,
|
||||
"Loki Infra Touched": s.loki_infra_touched,
|
||||
"Shared Sources Touched": s.shared_sources_touched,
|
||||
"FE Modules Total": s.fe_modules_total,
|
||||
"FE Modules Changed": s.fe_modules_changed,
|
||||
"FE Modules Affected (Rules)": s.fe_modules_affected_rules,
|
||||
"FE Modules Affected (Usage)": s.fe_modules_affected_usage,
|
||||
"Unit Specs All": s.unit_specs_all,
|
||||
"Unit Specs To Run (Rules)": s.unit_specs_to_run_rules,
|
||||
"Unit Specs To Run (Usage)": s.unit_specs_to_run_usage,
|
||||
"Loki Stories All": s.loki_stories_all,
|
||||
"Loki Stories To Run (Rules)": s.loki_stories_to_run_rules,
|
||||
"Loki Stories To Run (Usage)": s.loki_stories_to_run_usage,
|
||||
// Spelled as the column name rather than a display name: the importer
|
||||
// splits a digit-then-capital run, so "E2E Specs All" would normalize to
|
||||
// `e2_e_specs_all` instead of `e2e_specs_all`.
|
||||
e2e_specs_all: s.e2e_specs_all,
|
||||
e2e_specs_to_run_rules: s.e2e_specs_to_run_rules,
|
||||
e2e_specs_to_run_usage: s.e2e_specs_to_run_usage,
|
||||
};
|
||||
|
||||
await importStats({ table: TABLE, rows: [row] });
|
||||
console.log("Data uploaded successfully");
|
||||
}
|
||||
|
||||
// Best-effort: never let a telemetry failure turn the job red.
|
||||
main().catch((error) => {
|
||||
console.error("Skipping stats upload after error:", error);
|
||||
process.exit(0);
|
||||
});
|
||||
38
.github/scripts/upload-bundle-size-stats.ts
vendored
38
.github/scripts/upload-bundle-size-stats.ts
vendored
@@ -1,38 +0,0 @@
|
||||
// Appends the measured bundle sizes (with deltas) to the "Bundle Sizes" table
|
||||
// in eng-stats-importer. Reads ROWS (a JSON file of rows) and API_KEY from env,
|
||||
// and reports whether the import landed via the `uploaded` step output.
|
||||
|
||||
import { appendFileSync, readFileSync } from "node:fs";
|
||||
|
||||
import { importStats, type StatsRow } from "./stats-import";
|
||||
|
||||
/** `core.setOutput` without the github-script runtime. */
|
||||
function setOutput(name: string, value: string) {
|
||||
const file = process.env.GITHUB_OUTPUT;
|
||||
if (file) {
|
||||
appendFileSync(file, `${name}=${value}\n`);
|
||||
}
|
||||
}
|
||||
|
||||
async function main() {
|
||||
const rowsPath = process.env.ROWS || "artifacts/upload-rows.json";
|
||||
const rows = JSON.parse(readFileSync(rowsPath, "utf8")) as StatsRow[];
|
||||
|
||||
console.table(rows);
|
||||
|
||||
try {
|
||||
await importStats({ table: "bundle_sizes", rows });
|
||||
console.log("Bundle sizes uploaded successfully");
|
||||
setOutput("uploaded", "true");
|
||||
} catch (error) {
|
||||
// Stats logging is best-effort: if the importer is still unreachable after
|
||||
// the client's retries, keep the run green. The last-plotted cache is only
|
||||
// saved on a successful upload, so this point is re-attempted on the next
|
||||
// master build and nothing is silently dropped.
|
||||
const message = error instanceof Error ? error.message : String(error);
|
||||
console.log(`::warning::Bundle-size upload failed after retries; leaving run green: ${message}`);
|
||||
setOutput("uploaded", "false");
|
||||
}
|
||||
}
|
||||
|
||||
main();
|
||||
25
.github/scripts/upload-emotion-stats.ts
vendored
25
.github/scripts/upload-emotion-stats.ts
vendored
@@ -1,25 +0,0 @@
|
||||
// Appends the weekly @emotion/* importer count to the "Emotion Files" table in
|
||||
// eng-stats-importer. Reads EMOTION_FILES and API_KEY from env.
|
||||
|
||||
import { importStats } from "./stats-import";
|
||||
|
||||
async function main() {
|
||||
console.log("EMOTION_FILES", Number(process.env.EMOTION_FILES));
|
||||
|
||||
// Keys must match the target table's columns (the importer normalizes both
|
||||
// sides, so casing/spacing is forgiving).
|
||||
const rows = [
|
||||
{
|
||||
Date: new Date().toISOString().slice(0, 10), // YYYY-MM-DD
|
||||
"Emotion Files": Number(process.env.EMOTION_FILES),
|
||||
},
|
||||
];
|
||||
|
||||
await importStats({ table: "emotion_stats", rows });
|
||||
console.log("Data uploaded successfully");
|
||||
}
|
||||
|
||||
main().catch((error) => {
|
||||
console.error("Error uploading data:", error);
|
||||
process.exit(1);
|
||||
});
|
||||
@@ -1,30 +0,0 @@
|
||||
// Appends the weekly module-boundary counts to the "Module Boundaries" table in
|
||||
// eng-stats-importer. Reads NAMED_MODULES, ENFORCED_MODULES, UNMODULED,
|
||||
// VIOLATIONS and API_KEY from env.
|
||||
|
||||
import { importStats } from "./stats-import";
|
||||
|
||||
async function main() {
|
||||
console.log("NAMED_MODULES", Number(process.env.NAMED_MODULES));
|
||||
console.log("ENFORCED_MODULES", Number(process.env.ENFORCED_MODULES));
|
||||
console.log("UNMODULED", Number(process.env.UNMODULED));
|
||||
console.log("VIOLATIONS", Number(process.env.VIOLATIONS));
|
||||
|
||||
const rows = [
|
||||
{
|
||||
Date: new Date().toISOString().slice(0, 10), // YYYY-MM-DD
|
||||
Modules: Number(process.env.NAMED_MODULES),
|
||||
"Enforced modules": Number(process.env.ENFORCED_MODULES),
|
||||
"Unmoduled folders": Number(process.env.UNMODULED),
|
||||
Violations: Number(process.env.VIOLATIONS),
|
||||
},
|
||||
];
|
||||
|
||||
await importStats({ table: "module_boundaries", rows });
|
||||
console.log("Data uploaded successfully");
|
||||
}
|
||||
|
||||
main().catch((error) => {
|
||||
console.error("Error uploading data:", error);
|
||||
process.exit(1);
|
||||
});
|
||||
18
.github/team.json
vendored
18
.github/team.json
vendored
@@ -56,14 +56,15 @@
|
||||
"label": ".Team/Metabot",
|
||||
"members": [
|
||||
"appleby",
|
||||
"noahmoss",
|
||||
"retro",
|
||||
"sloansparger",
|
||||
"somtom",
|
||||
"tsplude",
|
||||
"lbrdnk",
|
||||
"piranha",
|
||||
"crisptrutski",
|
||||
"undrfined",
|
||||
"andreis",
|
||||
"prascuna"
|
||||
"undrfined"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -110,9 +111,7 @@
|
||||
"uladzimirdev",
|
||||
"egorgrushin",
|
||||
"metatyb",
|
||||
"olehk-mb",
|
||||
"annatomka",
|
||||
"brybacki"
|
||||
"olehk-mb"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -125,13 +124,6 @@
|
||||
"losrebellos",
|
||||
"tmashara"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "Security Engineering",
|
||||
"label": ".Team/SecurityEngineering",
|
||||
"members": [
|
||||
"enthym-aa"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
42
.github/workflows/actionlint.yml
vendored
Normal file
42
.github/workflows/actionlint.yml
vendored
Normal file
@@ -0,0 +1,42 @@
|
||||
name: Actionlint
|
||||
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- 'master'
|
||||
- 'release-**'
|
||||
paths:
|
||||
- '.github/workflows/**'
|
||||
pull_request:
|
||||
paths:
|
||||
- '.github/workflows/**'
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
actionlint:
|
||||
name: Lint Github Actions Workflows
|
||||
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
- name: Get changed Github actions
|
||||
uses: dorny/paths-filter@fbd0ab8f3e69293af611ebaee6363fc25e6d187d # v4.0.1
|
||||
id: changes
|
||||
with:
|
||||
token: ${{ github.token }}
|
||||
filters: .github/file-paths.yaml
|
||||
list-files: shell
|
||||
- name: Setup actionlint
|
||||
id: download
|
||||
shell: bash
|
||||
run: |
|
||||
if command -v actionlint; then
|
||||
echo "executable=actionlint" >> "$GITHUB_OUTPUT"
|
||||
else
|
||||
# script sets executable output, for details see: https://github.com/rhysd/actionlint/blob/main/docs/usage.md#use-actionlint-on-github-actions
|
||||
bash <(curl -fsSL https://raw.githubusercontent.com/rhysd/actionlint/v1.7.12/scripts/download-actionlint.bash)
|
||||
fi
|
||||
- name: Run actionlint
|
||||
run: ${{ steps.download.outputs.executable }} --color --verbose ${{ steps.changes.outputs.actionlint_files }}
|
||||
233
.github/workflows/ai-benchmark.yml
vendored
233
.github/workflows/ai-benchmark.yml
vendored
@@ -1,226 +1,21 @@
|
||||
name: AI Benchmark
|
||||
run-name: "Evals · ${{ github.event_name == 'pull_request' && 'label' || 'manual' }} · ${{ github.event.pull_request.number || inputs.pr || inputs.ref || github.sha }}"
|
||||
|
||||
# Runs the ai-benchmark E2E test suite against a Metabase instance.
|
||||
#
|
||||
# NOTE: This is a stub. The real workflow — which builds the benchmark Docker
|
||||
# image, boots Metabase + Postgres with a preloaded data dump, runs the
|
||||
# benchmark suite, and uploads results as artifacts — is being developed on a
|
||||
# feature branch. This stub exists on master so that the "Run workflow" button
|
||||
# is available in the GitHub Actions UI, which lets us dispatch the real
|
||||
# workflow against feature branches during development.
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
types: [labeled]
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
pr:
|
||||
description: "Metabase PR number (sets ref to the PR head)"
|
||||
required: false
|
||||
default: ""
|
||||
ref:
|
||||
description: "Metabase branch, tag or sha (resolved to a full sha; overrides the PR head, and must sit on the PR's branch when both are given)"
|
||||
required: false
|
||||
default: ""
|
||||
scope:
|
||||
description: "Scope"
|
||||
required: true
|
||||
type: choice
|
||||
options: [smoke, full, "all (EXPENSIVE)"]
|
||||
default: smoke
|
||||
sample:
|
||||
description: "Sample (0 = full suite)"
|
||||
required: false
|
||||
default: "5"
|
||||
evals_branch:
|
||||
description: "Evals branch"
|
||||
default: main
|
||||
|
||||
concurrency:
|
||||
group: evals-dispatch-${{ (github.event.label.name == 'evals/smoke' && github.event.pull_request.number) || github.run_id }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
prepare:
|
||||
if: >-
|
||||
github.event.pull_request.head.repo.full_name == github.repository
|
||||
&& github.event_name == 'pull_request'
|
||||
&& github.event.label.name == 'evals/smoke'
|
||||
|| github.event_name == 'workflow_dispatch'
|
||||
runs-on: ${{ vars.SLIM_RUNNER_KEY }}
|
||||
timeout-minutes: 5
|
||||
outputs:
|
||||
pr: ${{ steps.prepare.outputs.pr }}
|
||||
ref: ${{ steps.prepare.outputs.ref }}
|
||||
scope: ${{ steps.prepare.outputs.scope }}
|
||||
sample: ${{ steps.prepare.outputs.sample }}
|
||||
evals_branch: ${{ steps.prepare.outputs.evals_branch }}
|
||||
placeholder:
|
||||
name: Placeholder
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Prepare
|
||||
id: prepare
|
||||
env:
|
||||
EVENT_NAME: ${{ github.event_name }}
|
||||
LABEL_PR: ${{ github.event.pull_request.number }}
|
||||
INPUT_PR: ${{ inputs.pr }}
|
||||
INPUT_REF: ${{ inputs.ref }}
|
||||
INPUT_SCOPE: ${{ inputs.scope }}
|
||||
INPUT_SAMPLE: ${{ inputs.sample }}
|
||||
INPUT_EVALS_BRANCH: ${{ inputs.evals_branch }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
# The two paths are mutually exclusive, and `on:` admits no other trigger.
|
||||
if [[ "$EVENT_NAME" == 'pull_request' ]]; then
|
||||
PR="$LABEL_PR"; REF=''; SCOPE='smoke'
|
||||
SAMPLE='5'; EVALS_BRANCH='main'
|
||||
else
|
||||
PR="$INPUT_PR"; REF="$INPUT_REF"; SCOPE="$INPUT_SCOPE"
|
||||
SAMPLE="${INPUT_SAMPLE:-5}"; EVALS_BRANCH="$INPUT_EVALS_BRANCH"
|
||||
fi
|
||||
# None of these is ever multi-line; a newline would forge extra output entries.
|
||||
for v in PR REF SCOPE SAMPLE EVALS_BRANCH; do
|
||||
if [[ "${!v}" == *$'\n'* ]]; then
|
||||
echo "::error::${v} must be a single line"
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
{
|
||||
echo "pr=$PR"
|
||||
echo "ref=$REF"
|
||||
echo "scope=$SCOPE"
|
||||
echo "sample=$SAMPLE"
|
||||
echo "evals_branch=$EVALS_BRANCH"
|
||||
} >> "$GITHUB_OUTPUT"
|
||||
|
||||
announce:
|
||||
needs: prepare
|
||||
if: github.event_name == 'pull_request'
|
||||
runs-on: ${{ vars.SLIM_RUNNER_KEY }}
|
||||
timeout-minutes: 5
|
||||
outputs:
|
||||
comment_id: ${{ steps.post.outputs.comment_id }}
|
||||
steps:
|
||||
- name: Comment (building)
|
||||
id: post
|
||||
uses: actions/github-script@v9
|
||||
with:
|
||||
github-token: ${{ secrets.METABASE_AUTOMATION_USER_TOKEN }}
|
||||
script: |
|
||||
const runUrl = `${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`;
|
||||
const body = `🤖 **Evals smoke** — ⏳ building the image… ([logs](${runUrl}))`;
|
||||
const { owner, repo } = context.repo;
|
||||
const issue_number = context.payload.pull_request.number;
|
||||
// Drop earlier verdicts first: they describe shas that are no longer head, and a stale
|
||||
// green one reads as this PR passing. The prefix survives eval-run's rewrites.
|
||||
const existing = await github.paginate(github.rest.issues.listComments, { owner, repo, issue_number });
|
||||
for (const c of existing.filter(c => c.body.startsWith('🤖 **Evals'))) {
|
||||
await github.rest.issues.deleteComment({ owner, repo, comment_id: c.id }).catch(() => {});
|
||||
}
|
||||
const { data } = await github.rest.issues.createComment({ owner, repo, issue_number, body });
|
||||
core.setOutput('comment_id', String(data.id));
|
||||
|
||||
resolve:
|
||||
needs: prepare
|
||||
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
|
||||
timeout-minutes: 5
|
||||
permissions:
|
||||
contents: read
|
||||
pull-requests: read
|
||||
outputs:
|
||||
sha: ${{ steps.sha.outputs.sha }}
|
||||
steps:
|
||||
- name: Resolve pr/ref to a full commit sha
|
||||
id: sha
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
PR: ${{ needs.prepare.outputs.pr }}
|
||||
REF: ${{ needs.prepare.outputs.ref }}
|
||||
DEFAULT_SHA: ${{ github.sha }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
# `ref` wins over `pr`, matching evals' eval-run.yml, which this dispatches into — same two
|
||||
# inputs with opposite precedence in chained workflows would be a trap. eval-run also
|
||||
# rejects a ref that doesn't sit on the PR's branch; that check stays there rather than
|
||||
# being duplicated here.
|
||||
if [[ -n "$REF" ]]; then
|
||||
# `ref` accepts a branch or tag; fetch-artifact and the image tag need a full sha.
|
||||
SHA=$(gh api "repos/${GITHUB_REPOSITORY}/commits/${REF}" --jq .sha)
|
||||
echo "ref ${REF} -> ${SHA}"
|
||||
elif [[ -n "$PR" ]]; then
|
||||
SHA=$(gh api "repos/${GITHUB_REPOSITORY}/pulls/${PR}" --jq .head.sha)
|
||||
echo "PR #${PR} head -> ${SHA}"
|
||||
else
|
||||
SHA="$DEFAULT_SHA"
|
||||
echo "neither pr nor ref given; using this run's commit -> ${SHA}"
|
||||
fi
|
||||
if [[ ! "$SHA" =~ ^[0-9a-f]{40}$ ]]; then
|
||||
echo "::error::could not resolve a full commit sha (got '${SHA}')"
|
||||
exit 1
|
||||
fi
|
||||
echo "sha=${SHA}" >> "$GITHUB_OUTPUT"
|
||||
|
||||
image:
|
||||
needs: resolve
|
||||
uses: ./.github/workflows/build-private-ee-docker-image.yml
|
||||
secrets:
|
||||
PR_ENV_IAM_ROLE: ${{ secrets.PR_ENV_IAM_ROLE }}
|
||||
PR_ENV_AWS_ACCOUNT_ID: ${{ secrets.PR_ENV_AWS_ACCOUNT_ID }}
|
||||
with:
|
||||
commit: ${{ needs.resolve.outputs.sha }}
|
||||
|
||||
dispatch:
|
||||
needs: [prepare, announce, resolve, image]
|
||||
if: >-
|
||||
!cancelled()
|
||||
&& needs.prepare.result == 'success'
|
||||
&& (needs.announce.result == 'success' || needs.announce.result == 'skipped')
|
||||
&& needs.resolve.result == 'success'
|
||||
&& needs.image.result == 'success'
|
||||
timeout-minutes: 5
|
||||
runs-on: ${{ vars.SLIM_RUNNER_KEY }}
|
||||
steps:
|
||||
- name: Dispatch eval-run
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.METABASE_AUTOMATION_USER_TOKEN }}
|
||||
EVALS_BRANCH: ${{ needs.prepare.outputs.evals_branch }}
|
||||
PR: ${{ needs.prepare.outputs.pr }}
|
||||
SHA: ${{ needs.resolve.outputs.sha }}
|
||||
SCOPE: ${{ needs.prepare.outputs.scope }}
|
||||
SAMPLE: ${{ needs.prepare.outputs.sample }}
|
||||
COMMENT_ID: ${{ needs.announce.outputs.comment_id }}
|
||||
run: |
|
||||
gh api repos/metabase/evals/actions/workflows/eval-run.yml/dispatches \
|
||||
-f ref="$EVALS_BRANCH" \
|
||||
-f "inputs[pr]=$PR" \
|
||||
-f "inputs[ref]=$SHA" \
|
||||
-f "inputs[scope]=$SCOPE" \
|
||||
-f "inputs[surfaces]=all" \
|
||||
-f "inputs[sample]=$SAMPLE" \
|
||||
-f "inputs[evals_branch]=$EVALS_BRANCH" \
|
||||
-f "inputs[comment_id]=$COMMENT_ID"
|
||||
|
||||
finalize:
|
||||
needs: [announce, resolve, image, dispatch]
|
||||
if: always() && needs.announce.result == 'success' && needs.dispatch.result != 'success'
|
||||
runs-on: ${{ vars.SLIM_RUNNER_KEY }}
|
||||
timeout-minutes: 5
|
||||
env:
|
||||
COMMENT_ID: ${{ needs.announce.outputs.comment_id }}
|
||||
NEEDS_JSON: ${{ toJSON(needs) }}
|
||||
steps:
|
||||
- name: Comment (build/dispatch failed)
|
||||
uses: actions/github-script@v9
|
||||
with:
|
||||
github-token: ${{ secrets.METABASE_AUTOMATION_USER_TOKEN }}
|
||||
script: |
|
||||
const runUrl = `${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`;
|
||||
const cancelled = Object.values(JSON.parse(process.env.NEEDS_JSON))
|
||||
.some(j => j.result === 'cancelled');
|
||||
const body = cancelled
|
||||
? `🤖 **Evals** — ⚪ cancelled before evals started ([logs](${runUrl}))`
|
||||
: `🤖 **Evals** — ❌ build or dispatch failed before evals started ([logs](${runUrl}))`;
|
||||
// The replacement run deletes this comment on its way in, so a concurrency cancel can
|
||||
// land here with nothing to patch. Losing the stamp is fine; failing the job isn't.
|
||||
await github.rest.issues.updateComment({
|
||||
owner: context.repo.owner, repo: context.repo.repo,
|
||||
comment_id: Number(process.env.COMMENT_ID), body })
|
||||
.catch(e => core.warning(`could not update comment (deleted?): ${e.message}`));
|
||||
// eval-run only consumes the label once it has taken the run over, so anything failing
|
||||
// before the dispatch leaves it applied — and `labeled` only fires on a transition, which
|
||||
// makes re-applying it to retry a no-op. Hand it back.
|
||||
await github.rest.issues.removeLabel({
|
||||
owner: context.repo.owner, repo: context.repo.repo,
|
||||
issue_number: context.payload.pull_request.number,
|
||||
name: 'evals/smoke',
|
||||
}).catch(() => {});
|
||||
- name: Stub
|
||||
run: echo "ai-benchmark workflow stub — real implementation pending"
|
||||
|
||||
272
.github/workflows/app-db.yml
vendored
272
.github/workflows/app-db.yml
vendored
@@ -6,34 +6,6 @@ on:
|
||||
skip:
|
||||
type: boolean
|
||||
default: false
|
||||
force-run:
|
||||
description: Run the full suite regardless of the diff (master / release branches).
|
||||
type: boolean
|
||||
default: false
|
||||
|
||||
# Run one app-DB job on demand, against the ref you dispatch on, optionally under a profiler.
|
||||
# Everything else about the job -- services, matrix legs, test args, quarantine gating -- is
|
||||
# whatever the normal CI run uses, because it is the same job.
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
app-db:
|
||||
description: 'Which app-DB job to run. Only this one runs; the others are skipped.'
|
||||
type: choice
|
||||
required: true
|
||||
options:
|
||||
- mariadb
|
||||
- mysql
|
||||
- postgres
|
||||
tests:
|
||||
description: 'Optional: run only these tests, comma- or space-separated, e.g. metabase.db.custom-migrations-test/some-test. Bare namespaces work too. Blank runs the job''s usual set.'
|
||||
type: string
|
||||
required: false
|
||||
default: ''
|
||||
profile:
|
||||
description: 'Profile the test run and attach a flamegraph to the job. Adds sampling overhead.'
|
||||
type: boolean
|
||||
required: false
|
||||
default: false
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }}-app-db
|
||||
@@ -43,105 +15,28 @@ concurrency:
|
||||
# ci-conductor (secrets: inherit is already set on the app-db call in
|
||||
# run-tests.yml). See DEV-2224.
|
||||
env:
|
||||
CI_CONDUCTOR_BASE_URL: ${{ vars.CI_CONDUCTOR_BASE_URL }}
|
||||
CI_CONDUCTOR_BASE_URL: ${{ secrets.CI_CONDUCTOR_BASE_URL }}
|
||||
CI_CONDUCTOR_WEBHOOK_SECRET: ${{ secrets.CI_CONDUCTOR_WEBHOOK_SECRET }}
|
||||
# Both read by the test-driver action, which is what actually runs the tests. Set here rather than
|
||||
# passed as inputs to each job so that turning either on is one line, not one line per job.
|
||||
# Empty (and so off) on workflow_call, where neither input exists.
|
||||
#
|
||||
# ONLY_TESTS replaces the job's own test selection, so every matrix leg of the chosen job runs the
|
||||
# whole list -- each app DB has three.
|
||||
PROFILE_TESTS: ${{ inputs.profile }}
|
||||
ONLY_TESTS: ${{ inputs.tests }}
|
||||
|
||||
jobs:
|
||||
app-db-matrix:
|
||||
if: ${{ !inputs.skip }}
|
||||
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
|
||||
timeout-minutes: 5
|
||||
name: Determine app DB versions
|
||||
outputs:
|
||||
mariadb: ${{ steps.versions.outputs.mariadb }}
|
||||
mysql: ${{ steps.versions.outputs.mysql }}
|
||||
postgres: ${{ steps.versions.outputs.postgres }}
|
||||
steps:
|
||||
- name: Select versions to test
|
||||
id: versions
|
||||
uses: actions/github-script@v9
|
||||
env:
|
||||
RUN_ALL: ${{ inputs.force-run }}
|
||||
with:
|
||||
script: | # js
|
||||
const runAll = process.env.RUN_ALL === 'true';
|
||||
|
||||
const versions = {
|
||||
mariadb: [
|
||||
{
|
||||
name: 'MariaDB 10.6',
|
||||
image: 'metabase/mariadb:10.6',
|
||||
'junit-name': 'be-tests-mariadb-10-6-ee',
|
||||
env: { 'enable-ssl-tests': 'false' },
|
||||
latest: false
|
||||
},
|
||||
{
|
||||
name: 'MariaDB Latest',
|
||||
image: 'mariadb:latest',
|
||||
'junit-name': 'be-tests-mariadb-latest-ee',
|
||||
env: { 'enable-ssl-tests': 'false' },
|
||||
latest: true
|
||||
},
|
||||
],
|
||||
mysql: [
|
||||
{
|
||||
name: 'MySQL 8.0',
|
||||
image: 'mysql:8.0',
|
||||
'junit-name': 'be-tests-mysql-8-0-ee',
|
||||
env: { 'enable-ssl-tests': 'false', 'enable-aws-iam-tests': 'false' },
|
||||
latest: false
|
||||
},
|
||||
{
|
||||
name: 'MySQL 9',
|
||||
image: 'mysql:9',
|
||||
'junit-name': 'be-tests-mysql-9-ee',
|
||||
env: { 'enable-ssl-tests': 'true', 'enable-aws-iam-tests': 'true' },
|
||||
// treating 9.0 as latest until we introduce mysql 26+ support
|
||||
latest: true
|
||||
},
|
||||
],
|
||||
postgres: [
|
||||
{
|
||||
name: 'Postgres 14.x',
|
||||
'junit-name': 'be-tests-postgres-ee',
|
||||
image: 'postgres:14-alpine',
|
||||
env: { 'enable-ssl-tests': 'false', 'enable-aws-iam-tests': 'false' },
|
||||
latest: false,
|
||||
},
|
||||
{
|
||||
name: 'Postgres Latest',
|
||||
'junit-name': 'be-tests-postgres-latest-ee',
|
||||
image: 'postgres:latest',
|
||||
env: { 'enable-ssl-tests': 'true', 'enable-aws-iam-tests': 'true' },
|
||||
latest: true,
|
||||
},
|
||||
],
|
||||
};
|
||||
|
||||
|
||||
for (const [db, dbVersions] of Object.entries(versions)) {
|
||||
const selected = runAll ? dbVersions : dbVersions.filter((version) => version.latest);
|
||||
core.info(`${db}: ${selected.map((version) => version.name).join(', ')}`);
|
||||
core.setOutput(db, JSON.stringify(selected));
|
||||
}
|
||||
|
||||
be-tests-mariadb:
|
||||
needs: app-db-matrix
|
||||
if: ${{ !inputs.skip && (inputs.app-db == '' || inputs.app-db == 'mariadb') }}
|
||||
if: ${{ !inputs.skip }}
|
||||
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
|
||||
timeout-minutes: 40
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
version: ${{ fromJSON(needs.app-db-matrix.outputs.mariadb) }}
|
||||
version:
|
||||
- name: MariaDB 10.6
|
||||
junit-name: be-tests-mariadb-10-6-ee
|
||||
image: metabase/mariadb:10.6 # https://github.com/metabase/metabase-mariadb-docker-image
|
||||
env:
|
||||
enable-ssl-tests: 'false'
|
||||
- name: MariaDB Latest
|
||||
junit-name: be-tests-mariadb-latest-ee
|
||||
image: mariadb:latest
|
||||
env:
|
||||
enable-ssl-tests: 'false'
|
||||
job:
|
||||
- name: Enterprise Tests
|
||||
build-static-viz: false
|
||||
@@ -166,18 +61,27 @@ jobs:
|
||||
services:
|
||||
mysql:
|
||||
image: ${{ matrix.version.image }}
|
||||
# disposable CI database: a RAM-backed datadir makes the ~90k fsyncs of a full
|
||||
# migration chain free (measured: setup-db! 59s -> 16s)
|
||||
options: --tmpfs /var/lib/mysql:rw
|
||||
ports:
|
||||
- "3306:3306"
|
||||
env:
|
||||
MARIADB_ALLOW_EMPTY_ROOT_PASSWORD: true
|
||||
MARIADB_DATABASE: mb_test
|
||||
env:
|
||||
# __ADDITIONAL_EXCLUDED_TAG__: ${{ !inputs.force-run && ':mb/old-migrations-test' || '' }}
|
||||
# GDGT-3126: re-enable and properly fix old migrations tests
|
||||
__ADDITIONAL_EXCLUDED_TAG__: ':mb/old-migrations-test'
|
||||
# Only run old migrations tests on pushes to master or release branches. All other branches should skip tests
|
||||
# with the tag `mb/old-migrations-test`. `__ADDITIONAL_EXCLUDED_TAG__` is not used anywhere outside of splicing
|
||||
# it in to the command below.
|
||||
__ADDITIONAL_EXCLUDED_TAG__: >-
|
||||
${{
|
||||
(
|
||||
github.event_name == 'push' &&
|
||||
(
|
||||
github.ref_name == 'master' ||
|
||||
startsWith(github.ref_name, 'release-x')
|
||||
) &&
|
||||
''
|
||||
) ||
|
||||
':mb/old-migrations-test'
|
||||
}}
|
||||
# actual serious env vars below
|
||||
CI: 'true'
|
||||
DRIVERS: mysql
|
||||
@@ -213,21 +117,43 @@ jobs:
|
||||
with:
|
||||
build-static-viz: ${{ matrix.job.build-static-viz }}
|
||||
junit-name: ${{ matrix.version.junit-name }}
|
||||
artifact-suffix: ${{ matrix.job.name }}
|
||||
test-args: >-
|
||||
${{ matrix.job.test-args }}
|
||||
:exclude-tags '[ ${{ matrix.job.exclude-tag }} ${{ env.__ADDITIONAL_EXCLUDED_TAG__ }}]'
|
||||
quarantine-engine: ${{ vars.QUARANTINE_ENGINE }}
|
||||
- name: Upload Test Results
|
||||
uses: ./.github/actions/upload-test-results
|
||||
if: always()
|
||||
with:
|
||||
input-path: ./target/junit/
|
||||
output-name: ${{ github.job }}
|
||||
bucket: ${{ vars.AWS_S3_TEST_RESULTS_BUCKET }}
|
||||
role-to-assume: ${{ secrets.CI_TEST_RESULTS_IAM_ROLE }}
|
||||
aws-region: ${{ vars.AWS_REGION }}
|
||||
trunk-api-token: ${{ secrets.TRUNK_API_TOKEN }}
|
||||
previous-step-outcome: ${{ steps.test-driver.outputs.test-outcome }}
|
||||
run-trunk: ${{ vars.QUARANTINE_ENGINE != 'ci-conductor' }}
|
||||
|
||||
be-tests-mysql:
|
||||
needs: app-db-matrix
|
||||
if: ${{ !inputs.skip && (inputs.app-db == '' || inputs.app-db == 'mysql') }}
|
||||
if: ${{ !inputs.skip }}
|
||||
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
|
||||
timeout-minutes: 40
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
version: ${{ fromJSON(needs.app-db-matrix.outputs.mysql) }}
|
||||
version:
|
||||
- name: MySQL 8.0
|
||||
junit-name: be-tests-mysql-8-0-ee
|
||||
image: mysql:8.0
|
||||
env:
|
||||
enable-ssl-tests: 'false'
|
||||
enable-aws-iam-tests: 'false'
|
||||
- name: MySQL Latest
|
||||
junit-name: be-tests-mysql-latest-ee
|
||||
image: mysql:latest
|
||||
env:
|
||||
enable-ssl-tests: 'true'
|
||||
enable-aws-iam-tests: 'true'
|
||||
job:
|
||||
- name: Enterprise Tests
|
||||
build-static-viz: false
|
||||
@@ -252,18 +178,27 @@ jobs:
|
||||
services:
|
||||
mysql:
|
||||
image: ${{ matrix.version.image }}
|
||||
# disposable CI database: a RAM-backed datadir makes the ~90k fsyncs of a full
|
||||
# migration chain free (measured: setup-db! 59s -> 16s)
|
||||
options: --tmpfs /var/lib/mysql:rw
|
||||
ports:
|
||||
- "3306:3306"
|
||||
env:
|
||||
MYSQL_ALLOW_EMPTY_PASSWORD: true
|
||||
MYSQL_DATABASE: mb_test
|
||||
env:
|
||||
# __ADDITIONAL_EXCLUDED_TAG__: ${{ !inputs.force-run && ':mb/old-migrations-test' || '' }}
|
||||
# GDGT-3126: re-enable and properly fix old migrations tests
|
||||
__ADDITIONAL_EXCLUDED_TAG__: ':mb/old-migrations-test'
|
||||
# Only run old migrations tests on pushes to master or release branches. All other branches should skip tests
|
||||
# with the tag `mb/old-migrations-test`. `__ADDITIONAL_EXCLUDED_TAG__` is not used anywhere outside of splicing
|
||||
# it in to the command below.
|
||||
__ADDITIONAL_EXCLUDED_TAG__: >-
|
||||
${{
|
||||
(
|
||||
github.event_name == 'push' &&
|
||||
(
|
||||
github.ref_name == 'master' ||
|
||||
startsWith(github.ref_name, 'release-x')
|
||||
) &&
|
||||
''
|
||||
) ||
|
||||
':mb/old-migrations-test'
|
||||
}}
|
||||
# actual serious env vars below
|
||||
CI: 'true'
|
||||
DRIVERS: mysql
|
||||
@@ -307,21 +242,43 @@ jobs:
|
||||
with:
|
||||
build-static-viz: ${{ matrix.job.build-static-viz }}
|
||||
junit-name: ${{ matrix.version.junit-name }}
|
||||
artifact-suffix: ${{ matrix.job.name }}
|
||||
test-args: >-
|
||||
${{ matrix.job.test-args }}
|
||||
:exclude-tags '[ ${{ matrix.job.exclude-tag }} ${{ env.__ADDITIONAL_EXCLUDED_TAG__ }}]'
|
||||
quarantine-engine: ${{ vars.QUARANTINE_ENGINE }}
|
||||
- name: Upload Test Results
|
||||
uses: ./.github/actions/upload-test-results
|
||||
if: always()
|
||||
with:
|
||||
input-path: ./target/junit/
|
||||
output-name: ${{ github.job }}
|
||||
bucket: ${{ vars.AWS_S3_TEST_RESULTS_BUCKET }}
|
||||
role-to-assume: ${{ secrets.CI_TEST_RESULTS_IAM_ROLE }}
|
||||
aws-region: ${{ vars.AWS_REGION }}
|
||||
trunk-api-token: ${{ secrets.TRUNK_API_TOKEN }}
|
||||
previous-step-outcome: ${{ steps.test-driver.outputs.test-outcome }}
|
||||
run-trunk: ${{ vars.QUARANTINE_ENGINE != 'ci-conductor' }}
|
||||
|
||||
be-tests-postgres:
|
||||
needs: app-db-matrix
|
||||
if: ${{ !inputs.skip && (inputs.app-db == '' || inputs.app-db == 'postgres') }}
|
||||
if: ${{ !inputs.skip }}
|
||||
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
|
||||
timeout-minutes: 40
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
version: ${{ fromJSON(needs.app-db-matrix.outputs.postgres) }}
|
||||
version:
|
||||
- name: Postgres 14.x
|
||||
junit-name: postgres-ee
|
||||
docker-image: postgres:14-alpine
|
||||
env:
|
||||
enable-ssl-tests: 'false'
|
||||
enable-aws-iam-tests: 'false'
|
||||
- name: Postgres Latest
|
||||
junit-name: postgres-latest-ee
|
||||
docker-image: postgres:latest
|
||||
env:
|
||||
enable-ssl-tests: 'true'
|
||||
enable-aws-iam-tests: 'true'
|
||||
job:
|
||||
- name: Enterprise Tests
|
||||
build-static-viz: false
|
||||
@@ -332,6 +289,7 @@ jobs:
|
||||
build-static-viz: true
|
||||
test-args: >-
|
||||
:only '"test"'
|
||||
:exclude-tags [:mb/driver-tests]
|
||||
:partition/total 2
|
||||
:partition/index 0
|
||||
exclude-tag: ':mb/driver-tests :mb/transforms-python-test'
|
||||
@@ -344,9 +302,21 @@ jobs:
|
||||
exclude-tag: ':mb/driver-tests :mb/transforms-python-test'
|
||||
name: "${{ matrix.version.name }} ${{ matrix.job.name }}"
|
||||
env:
|
||||
# __ADDITIONAL_EXCLUDED_TAG__: ${{ !inputs.force-run && ':mb/old-migrations-test' || '' }}
|
||||
# GDGT-3126: re-enable and properly fix old migrations tests
|
||||
__ADDITIONAL_EXCLUDED_TAG__: ':mb/old-migrations-test'
|
||||
# Only run old migrations tests on pushes to master or release branches. All other branches should skip tests
|
||||
# with the tag `mb/old-migrations-test`. `__ADDITIONAL_EXCLUDED_TAG__` is not used anywhere outside of splicing
|
||||
# it in to the command below.
|
||||
__ADDITIONAL_EXCLUDED_TAG__: >-
|
||||
${{
|
||||
(
|
||||
github.event_name == 'push' &&
|
||||
(
|
||||
github.ref_name == 'master' ||
|
||||
startsWith(github.ref_name, 'release-x')
|
||||
) &&
|
||||
''
|
||||
) ||
|
||||
':mb/old-migrations-test'
|
||||
}}
|
||||
# actual serious env vars below
|
||||
CI: 'true'
|
||||
DRIVERS: postgres
|
||||
@@ -369,7 +339,7 @@ jobs:
|
||||
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_AURORA_IAM_SECRET_ACCESS_KEY }}
|
||||
services:
|
||||
postgres:
|
||||
image: ${{ matrix.version.image }}
|
||||
image: ${{ matrix.version.docker-image }}
|
||||
ports:
|
||||
- "5432:5432"
|
||||
env:
|
||||
@@ -383,20 +353,30 @@ jobs:
|
||||
uses: ./.github/actions/test-driver
|
||||
with:
|
||||
build-static-viz: ${{ matrix.job.build-static-viz }}
|
||||
junit-name: ${{ matrix.version.junit-name }}
|
||||
artifact-suffix: ${{ matrix.job.name }}
|
||||
junit-name: 'be-tests-${{ matrix.version.junit-name }}'
|
||||
test-args: >-
|
||||
${{ matrix.job.test-args }}
|
||||
:exclude-tags '[ ${{ matrix.job.exclude-tag }} ${{ env.__ADDITIONAL_EXCLUDED_TAG__ }}]'
|
||||
quarantine-engine: ${{ vars.QUARANTINE_ENGINE }}
|
||||
- name: Upload Test Results
|
||||
uses: ./.github/actions/upload-test-results
|
||||
if: always()
|
||||
with:
|
||||
input-path: ./target/junit/
|
||||
output-name: ${{ github.job }}
|
||||
bucket: ${{ vars.AWS_S3_TEST_RESULTS_BUCKET }}
|
||||
role-to-assume: ${{ secrets.CI_TEST_RESULTS_IAM_ROLE }}
|
||||
aws-region: ${{ vars.AWS_REGION }}
|
||||
trunk-api-token: ${{ secrets.TRUNK_API_TOKEN }}
|
||||
previous-step-outcome: ${{ steps.test-driver.outputs.test-outcome }}
|
||||
run-trunk: ${{ vars.QUARANTINE_ENGINE != 'ci-conductor' }}
|
||||
|
||||
app-db-tests-result:
|
||||
needs:
|
||||
- app-db-matrix
|
||||
- be-tests-mariadb
|
||||
- be-tests-mysql
|
||||
- be-tests-postgres
|
||||
runs-on: ${{ vars.SLIM_RUNNER_KEY }}
|
||||
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
|
||||
timeout-minutes: 5
|
||||
name: app-db-tests-result
|
||||
if: always() && !cancelled()
|
||||
|
||||
3
.github/workflows/auto-assign.yml
vendored
3
.github/workflows/auto-assign.yml
vendored
@@ -7,8 +7,7 @@ on:
|
||||
|
||||
jobs:
|
||||
update_pr:
|
||||
runs-on: ${{ vars.SLIM_RUNNER_KEY }}
|
||||
timeout-minutes: 5
|
||||
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
|
||||
|
||||
steps:
|
||||
- uses: tibdex/github-app-token@3beb63f4bd073e61482598c45c71c1019b59b73a # v2.1.0
|
||||
|
||||
71
.github/workflows/auto-backport.yml
vendored
71
.github/workflows/auto-backport.yml
vendored
@@ -99,36 +99,6 @@ jobs:
|
||||
)
|
||||
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
|
||||
steps:
|
||||
- name: "!! double-backport and triple-backport label deprecation warning !!"
|
||||
if:
|
||||
|
|
||||
github.event.pull_request.merged == true && (
|
||||
github.event.action == 'closed' && (
|
||||
contains(github.event.pull_request.labels.*.name, 'double-backport') ||
|
||||
contains(github.event.pull_request.labels.*.name, 'triple-backport')
|
||||
) ||
|
||||
github.event.action == 'labeled' && (
|
||||
github.event.label.name == 'double-backport' ||
|
||||
github.event.label.name == 'triple-backport'
|
||||
)
|
||||
)
|
||||
env:
|
||||
REPO: ${{ github.repository }}
|
||||
PR: ${{ github.event.pull_request.number }}
|
||||
GH_TOKEN: ${{ secrets.METABASE_AUTOMATION_USER_TOKEN }}
|
||||
run: |
|
||||
msg="🚨🚨 Backport failed because \`double-backport\` and \`triple-backport\` labels are **deprecated**! Use the \`backport\` label to port changes to all supported releases instead. 🚨🚨"
|
||||
|
||||
gh api \
|
||||
--silent \
|
||||
--method POST \
|
||||
-H "Accept: application/vnd.github+json" \
|
||||
-H "X-GitHub-Api-Version: 2026-03-10" \
|
||||
/repos/"$REPO"/issues/"$PR"/comments \
|
||||
-f "body=$msg"
|
||||
|
||||
echo "::error::$msg"
|
||||
exit 1
|
||||
- uses: actions/checkout@v6
|
||||
with:
|
||||
fetch-depth: 0
|
||||
@@ -140,6 +110,29 @@ jobs:
|
||||
pull-request-number: ${{ github.event.pull_request.number }}
|
||||
base-ref: ${{ github.event.pull_request.base.ref }}
|
||||
|
||||
- name: how many backports?
|
||||
id: how-many-backports
|
||||
uses: actions/github-script@v9
|
||||
with:
|
||||
script: | # js
|
||||
const labels = [
|
||||
...${{ toJson(github.event.pull_request.labels) }},
|
||||
{ name:"${{ github.event.label.name }}" }
|
||||
];
|
||||
|
||||
const backportLabelNames = labels.filter(label => label.name.includes('backport')).map(label => label.name);
|
||||
|
||||
let result = 1;
|
||||
|
||||
if (backportLabelNames.includes("triple-backport")) {
|
||||
result = 3;
|
||||
} else if (backportLabelNames.includes("double-backport")) {
|
||||
result = 2;
|
||||
}
|
||||
|
||||
console.log("backports", result);
|
||||
return result;
|
||||
|
||||
- name: get release versions
|
||||
id: get-latest-release-version
|
||||
uses: actions/github-script@v9
|
||||
@@ -164,3 +157,21 @@ jobs:
|
||||
backport-commit: ${{ steps.find_commit.outputs.commit }}
|
||||
github-token: ${{ secrets.METABASE_AUTOMATION_USER_TOKEN }}
|
||||
github-token-2: ${{ secrets.GITHUB_TOKEN }}
|
||||
|
||||
- name: Double Backport
|
||||
if: ${{ always() && fromJson(steps.how-many-backports.outputs.result) >= 2 }}
|
||||
uses: ./.github/actions/create-backport
|
||||
with:
|
||||
target-version: ${{ fromJson(steps.get-latest-release-version.outputs.result).secondLatest }}
|
||||
backport-commit: ${{ steps.find_commit.outputs.commit }}
|
||||
github-token: ${{ secrets.METABASE_AUTOMATION_USER_TOKEN }}
|
||||
github-token-2: ${{ secrets.GITHUB_TOKEN }}
|
||||
|
||||
- name: Triple Backport
|
||||
if: ${{ always() && fromJson(steps.how-many-backports.outputs.result) >= 3 }}
|
||||
uses: ./.github/actions/create-backport
|
||||
with:
|
||||
target-version: ${{ fromJson(steps.get-latest-release-version.outputs.result).thirdLatest }}
|
||||
backport-commit: ${{ steps.find_commit.outputs.commit }}
|
||||
github-token: ${{ secrets.METABASE_AUTOMATION_USER_TOKEN }}
|
||||
github-token-2: ${{ secrets.GITHUB_TOKEN }}
|
||||
|
||||
14
.github/workflows/backend-cloverage.yml
vendored
14
.github/workflows/backend-cloverage.yml
vendored
@@ -17,16 +17,15 @@ on:
|
||||
|
||||
jobs:
|
||||
setup-oss:
|
||||
runs-on: ${{ vars.SLIM_RUNNER_KEY }}
|
||||
timeout-minutes: 5
|
||||
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
|
||||
outputs:
|
||||
modules: ${{ steps.get-modules.outputs.modules }}
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
- id: get-modules
|
||||
run: |
|
||||
modules=$(for d in src/metabase/*/; do basename "$d"; done | tr '_' '-' | jq -R -s -c 'split("\n") | map(select(length > 0))')
|
||||
echo "modules=$modules" >> "$GITHUB_OUTPUT"
|
||||
modules=$(ls -1d src/metabase/*/ | xargs -n1 basename | tr '_' '-' | jq -R -s -c 'split("\n") | map(select(length > 0))')
|
||||
echo "modules=$modules" >> $GITHUB_OUTPUT
|
||||
|
||||
be-linter-cloverage:
|
||||
needs: setup-oss
|
||||
@@ -65,16 +64,15 @@ jobs:
|
||||
flags: back-end
|
||||
|
||||
setup-ee:
|
||||
runs-on: ${{ vars.SLIM_RUNNER_KEY }}
|
||||
timeout-minutes: 5
|
||||
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
|
||||
outputs:
|
||||
modules: ${{ steps.get-modules.outputs.modules }}
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
- id: get-modules
|
||||
run: |
|
||||
modules=$(for d in enterprise/backend/src/metabase_enterprise/*/; do basename "$d"; done | tr '_' '-' | jq -R -s -c 'split("\n") | map(select(length > 0))')
|
||||
echo "modules=$modules" >> "$GITHUB_OUTPUT"
|
||||
modules=$(ls -1d enterprise/backend/src/metabase_enterprise/*/ | xargs -n1 basename | tr '_' '-' | jq -R -s -c 'split("\n") | map(select(length > 0))')
|
||||
echo "modules=$modules" >> $GITHUB_OUTPUT
|
||||
|
||||
be-linter-cloverage-enterprise:
|
||||
needs: setup-ee
|
||||
|
||||
285
.github/workflows/backend.yml
vendored
285
.github/workflows/backend.yml
vendored
@@ -6,38 +6,6 @@ on:
|
||||
skip:
|
||||
type: boolean
|
||||
default: false
|
||||
force-run:
|
||||
description: Run the full suite regardless of the diff (master / release branches).
|
||||
type: boolean
|
||||
default: false
|
||||
run-project-backend-checks:
|
||||
description: Run project-level backend checks.
|
||||
type: boolean
|
||||
default: false
|
||||
run-project-migration-checks:
|
||||
description: Run project-level migration checks.
|
||||
type: boolean
|
||||
default: false
|
||||
run-project-ratchet-checks:
|
||||
description: Run the ratchet check for a diff that touches nothing but the ratchets file.
|
||||
type: boolean
|
||||
default: false
|
||||
|
||||
# Run the backend test suite on demand, against the ref you dispatch on, optionally under a
|
||||
# profiler. Everything else about the jobs -- matrix legs, tag filters, quarantine gating -- is
|
||||
# whatever the normal CI run uses, because they are the same jobs.
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
tests:
|
||||
description: 'Optional: run only these tests, comma- or space-separated, e.g. metabase.query-processor-test/some-test. Bare namespaces work too. Blank runs each leg''s usual set.'
|
||||
type: string
|
||||
required: false
|
||||
default: ''
|
||||
profile:
|
||||
description: 'Profile the test run and attach a flamegraph to the job. Adds sampling overhead.'
|
||||
type: boolean
|
||||
required: false
|
||||
default: false
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }}-backend
|
||||
@@ -47,106 +15,10 @@ concurrency:
|
||||
# (secrets: inherit is already set on the backend call in run-tests.yml).
|
||||
# See DEV-2224.
|
||||
env:
|
||||
CI_CONDUCTOR_BASE_URL: ${{ vars.CI_CONDUCTOR_BASE_URL }}
|
||||
CI_CONDUCTOR_BASE_URL: ${{ secrets.CI_CONDUCTOR_BASE_URL }}
|
||||
CI_CONDUCTOR_WEBHOOK_SECRET: ${{ secrets.CI_CONDUCTOR_WEBHOOK_SECRET }}
|
||||
# Both read by the be-tests run step below. Set here rather than passed per job so that turning
|
||||
# either on is one line. Empty (and so off) on workflow_call, where neither input exists.
|
||||
#
|
||||
# ONLY_TESTS replaces the leg's own test selection, so every matrix leg runs the whole list: the
|
||||
# partition args no longer apply, and legs that hold none of the named tests run nothing.
|
||||
PROFILE_TESTS: ${{ inputs.profile }}
|
||||
ONLY_TESTS: ${{ inputs.tests }}
|
||||
|
||||
jobs:
|
||||
project-checks:
|
||||
name: 'Project tests: Suites'
|
||||
if: >-
|
||||
${{ inputs.force-run ||
|
||||
inputs.run-project-backend-checks ||
|
||||
inputs.run-project-migration-checks ||
|
||||
inputs.run-project-ratchet-checks }}
|
||||
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
|
||||
timeout-minutes: 15
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
- name: Prepare back-end environment
|
||||
uses: ./.github/actions/prepare-backend
|
||||
# Bootstrap the shared Mage runtime before concurrent commands try to install it themselves.
|
||||
- name: Prepare Mage
|
||||
run: ./bin/mage
|
||||
- parallel:
|
||||
- name: Run backend checks
|
||||
if: ${{ inputs.force-run || inputs.run-project-backend-checks }}
|
||||
run: ./bin/mage project-tests backend
|
||||
- name: Run migration checks
|
||||
if: ${{ inputs.force-run || inputs.run-project-migration-checks }}
|
||||
run: ./bin/mage project-tests migrations
|
||||
# Babashka, not a JVM test run, so it is cheap enough to run alongside the backend checks.
|
||||
# It still needs the JDK prepare-backend installs, to resolve clj-kondo on a cold cache.
|
||||
- name: Run ratchet checks
|
||||
if: ${{ inputs.force-run || inputs.run-project-backend-checks || inputs.run-project-ratchet-checks }}
|
||||
run: ./bin/mage kondo-ratchets
|
||||
|
||||
project-tests:
|
||||
name: Project tests
|
||||
needs: project-checks
|
||||
if: ${{ always() && !cancelled() }}
|
||||
runs-on: ${{ vars.SLIM_RUNNER_KEY }}
|
||||
timeout-minutes: 5
|
||||
steps:
|
||||
- name: Check Project tests status
|
||||
uses: actions/github-script@v9
|
||||
env:
|
||||
needs: ${{ toJson(needs) }}
|
||||
with:
|
||||
script: | # js
|
||||
const needs = JSON.parse(process.env.needs);
|
||||
const jobs = Object.entries(needs).map(
|
||||
([jobName, jobValues]) => ({
|
||||
name: jobName,
|
||||
result: jobValues.result
|
||||
}));
|
||||
|
||||
if (jobs.every(job => job.result === 'skipped' || job.result === 'success')) {
|
||||
console.log('');
|
||||
console.log(' |>>> |>>> |>>>');
|
||||
console.log(' | | |');
|
||||
console.log(' _ _|_ _ _ _|_ _ _ _|_ _');
|
||||
console.log(' | |_| |_| |______| |_| |_| |______| |_| |_| |');
|
||||
console.log(' | | | | | |');
|
||||
console.log(' | [] [] | | [] [] | | [] [] |');
|
||||
console.log(' | |______| |______| |');
|
||||
console.log(' | ___ |');
|
||||
console.log(' |___________________| |___________________|');
|
||||
console.log(' |___|');
|
||||
console.log('');
|
||||
console.log('Everything is up to code and structurally sound.');
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
console.log('');
|
||||
console.log(' .-===========-.____');
|
||||
console.log(' /_______________\\ \\____');
|
||||
console.log(' | .-----------. | \\___');
|
||||
console.log(' | | x x | | \\__');
|
||||
console.log(' | | ~ | | _\\_');
|
||||
console.log(" | `-----------' | /___\\");
|
||||
console.log(' | o o |');
|
||||
console.log(" `-----------------'");
|
||||
console.log(' /=============\\');
|
||||
console.log(' / O O O O O O O \\');
|
||||
console.log(' \\_______________/');
|
||||
console.log('');
|
||||
console.log('One or more project checks are off the blueprint. Hard hats required.');
|
||||
|
||||
jobs.forEach((job) => {
|
||||
if (job.result !== 'success' && job.result !== 'skipped') {
|
||||
console.log(`${job.name} - ${job.result}`);
|
||||
}
|
||||
});
|
||||
|
||||
process.exit(1);
|
||||
|
||||
be-linter-clj-kondo:
|
||||
if: ${{ !inputs.skip }}
|
||||
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
|
||||
@@ -173,7 +45,7 @@ jobs:
|
||||
|
||||
be-linter-eastwood-test:
|
||||
# Master / release safety net: lint every backend test namespace.
|
||||
if: ${{ !inputs.skip && inputs.force-run }}
|
||||
if: ${{ !inputs.skip && github.event_name == 'push' && (github.ref_name == 'master' || startsWith(github.ref_name, 'release-x')) }}
|
||||
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
|
||||
timeout-minutes: 45
|
||||
name: Eastwood (tests only)
|
||||
@@ -181,12 +53,12 @@ jobs:
|
||||
- uses: actions/checkout@v6
|
||||
- name: Prepare back-end environment
|
||||
uses: ./.github/actions/prepare-backend
|
||||
- run: clojure -X:dev:dev/test:ee:ee-dev:drivers:drivers-dev:test:eastwood/test
|
||||
- run: clojure -X:dev:ee:ee-dev:drivers:drivers-dev:test:eastwood/test
|
||||
name: Run Eastwood linter on our backend tests
|
||||
|
||||
be-linter-eastwood-test-changed:
|
||||
# Lint only the test namespaces touched by the diff.
|
||||
if: ${{ !inputs.skip && !inputs.force-run && github.event_name == 'pull_request' }}
|
||||
# PR / merge queue: lint only the test namespaces touched by the diff.
|
||||
if: ${{ !inputs.skip && (github.event_name == 'pull_request' || github.event_name == 'merge_group') }}
|
||||
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
|
||||
timeout-minutes: 15
|
||||
name: Eastwood (changed tests only)
|
||||
@@ -200,8 +72,8 @@ jobs:
|
||||
- name: Compute changed test namespaces
|
||||
id: changed
|
||||
env:
|
||||
BASE_SHA: ${{ github.event.pull_request.base.sha }}
|
||||
HEAD_SHA: ${{ github.event.pull_request.head.sha || github.sha }}
|
||||
BASE_SHA: ${{ github.event.pull_request.base.sha || github.event.merge_group.base_sha }}
|
||||
HEAD_SHA: ${{ github.event.pull_request.head.sha || github.event.merge_group.head_sha || github.sha }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
|
||||
@@ -209,14 +81,13 @@ jobs:
|
||||
# last synced don't widen our scope. Falls back to BASE_SHA if older than fetched depth.
|
||||
git fetch --no-tags --depth=200 origin "$BASE_SHA"
|
||||
DIFF_BASE=$(git merge-base "$BASE_SHA" "$HEAD_SHA" 2>/dev/null || echo "$BASE_SHA")
|
||||
# Test roots we lint: core, EE, dev/test, and each driver module's test tree.
|
||||
# Test roots we lint: core, EE, and each driver module's test tree.
|
||||
# :eastwood/test derives matching :source-paths from the classpath, so any
|
||||
# namespace we name here is reachable. Include renames (R) so a test that
|
||||
# was just renamed still gets linted under its new name.
|
||||
FILES=$(git diff --name-only --diff-filter=AMR "$DIFF_BASE" "$HEAD_SHA" \
|
||||
-- 'test/**/*.clj' 'test/**/*.cljc' \
|
||||
'enterprise/backend/test/**/*.clj' 'enterprise/backend/test/**/*.cljc' \
|
||||
'dev/test/**/*.clj' 'dev/test/**/*.cljc' \
|
||||
'modules/drivers/**/test/**/*.clj' 'modules/drivers/**/test/**/*.cljc')
|
||||
|
||||
if [ -z "$FILES" ]; then
|
||||
@@ -232,14 +103,7 @@ jobs:
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Convert each test path to a Clojure namespace: strip its known test root and
|
||||
# extension, then translate path separators and underscores. Keep these roots
|
||||
# aligned with FILES above and the Eastwood classpath aliases below.
|
||||
NS=$(sed -E \
|
||||
-e 's#^(test/|enterprise/backend/test/|dev/test/|modules/drivers/[^/]+/test/)##' \
|
||||
-e 's#\.cljc?$##' \
|
||||
-e 's#/#.#g' \
|
||||
-e 's#_#-#g' <<<"$FILES" \
|
||||
NS=$(sed -E 's#^(test/|enterprise/backend/test/|modules/drivers/[^/]+/test/)##; s#\.cljc?$##; s#/#.#g; s#_#-#g' <<<"$FILES" \
|
||||
| sort -u \
|
||||
| paste -sd' ' -)
|
||||
echo "Changed test namespaces: $NS"
|
||||
@@ -248,7 +112,7 @@ jobs:
|
||||
if: steps.changed.outputs.namespaces != ''
|
||||
env:
|
||||
NAMESPACES: ${{ steps.changed.outputs.namespaces }}
|
||||
run: clojure -X:dev:dev/test:ee:ee-dev:drivers:drivers-dev:test:eastwood/test :namespaces "[$NAMESPACES]"
|
||||
run: clojure -X:dev:ee:ee-dev:drivers:drivers-dev:test:eastwood/test :namespaces "[$NAMESPACES]"
|
||||
|
||||
be-tests:
|
||||
if: ${{ !inputs.skip }}
|
||||
@@ -294,9 +158,21 @@ jobs:
|
||||
:partition/index 1
|
||||
name: "Java ${{ matrix.java-version }} ${{ matrix.job.name }}"
|
||||
env:
|
||||
# __ADDITIONAL_EXCLUDED_TAG__: ${{ !inputs.force-run && ':mb/old-migrations-test' || '' }}
|
||||
# GDGT-3126: re-enable and properly fix old migrations tests
|
||||
__ADDITIONAL_EXCLUDED_TAG__: ':mb/old-migrations-test'
|
||||
# Only run old migrations tests on pushes to master or release branches. All other branches should skip tests
|
||||
# with the tag `mb/old-migrations-test`. `__ADDITIONAL_EXCLUDED_TAG__` is not used anywhere outside of splicing
|
||||
# it in to the command below.
|
||||
__ADDITIONAL_EXCLUDED_TAG__: >-
|
||||
${{
|
||||
(
|
||||
github.event_name == 'push' &&
|
||||
(
|
||||
github.ref_name == 'master' ||
|
||||
startsWith(github.ref_name, 'release-x')
|
||||
) &&
|
||||
''
|
||||
) ||
|
||||
':mb/old-migrations-test'
|
||||
}}
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
- name: Prepare front-end environment
|
||||
@@ -315,77 +191,17 @@ jobs:
|
||||
# below can link failures to this exact job. Best-effort.
|
||||
uses: ./.github/actions/resolve-job-id
|
||||
|
||||
# Granular rerun: on a re-run of a failed job, restore the list of tests
|
||||
# that failed on the previous attempt (uploaded below) so the test step
|
||||
# can target only those instead of the whole partition. Best-effort: a
|
||||
# missing/empty artifact leaves no file and the run falls back to the full
|
||||
# partition. First attempts skip this entirely.
|
||||
#
|
||||
# Scoped to the IMMEDIATELY preceding attempt: an attempt that passed, or
|
||||
# that failed with nothing attributable to a var, uploads no artifact, and
|
||||
# a narrow set must not outlive it. Reading `attempt - 1` means "only what
|
||||
# the last attempt actually recorded" — anything else falls back to the
|
||||
# full partition. Workflow expressions have no arithmetic, so the shell
|
||||
# computes it.
|
||||
- name: Resolve previous attempt
|
||||
if: ${{ github.run_attempt != '1' }}
|
||||
run: echo "RERUN_PREV_ATTEMPT=$((GITHUB_RUN_ATTEMPT - 1))" >> "$GITHUB_ENV"
|
||||
|
||||
# `pattern` (not `name`) because the artifact is legitimately absent whenever the
|
||||
# previous attempt recorded nothing: `name` treats that as an error, `pattern`
|
||||
# downloads nothing and moves on. The value is a literal — no glob metacharacters —
|
||||
# so it matches exactly one artifact, which lands at `target/failed-tests`. A pattern
|
||||
# that ever matched two would nest them under their artifact names instead, leaving
|
||||
# the file absent and falling back to the full partition.
|
||||
- name: Restore previously-failed tests
|
||||
if: ${{ github.run_attempt != '1' }}
|
||||
continue-on-error: true
|
||||
uses: actions/download-artifact@v8
|
||||
with:
|
||||
pattern: failed-tests-be-tests-java-${{ matrix.java-version }}-${{ matrix.job.test-group-name }}-attempt-${{ env.RERUN_PREV_ATTEMPT }}
|
||||
path: target
|
||||
|
||||
- name: "Test Java ${{ matrix.java-version }} ${{ matrix.job.name }}"
|
||||
id: run-java-tests
|
||||
# On a rerun with a non-empty failed-tests file, target only those vars
|
||||
# via hawk `:only`, dropping the tag/partition args — the listed vars are
|
||||
# exactly this job's failures, already filtered and partitioned on the
|
||||
# first attempt. Otherwise run exactly as before. Quote the file so the
|
||||
# whole `[ns/var ...]` vector reaches clojure -X as a single argument.
|
||||
run: |
|
||||
BASE="clojure -X:dev:ci:test:${{ matrix.job.edition }}:${{ matrix.job.edition }}-dev"
|
||||
# `:profile-tests` overrides `:test`'s exec-fn, so it has to come last. See dev.profile-tests.
|
||||
# The `profiled` output is what the collection steps below key off, so that the decision is
|
||||
# recorded by the step that acts on it rather than re-derived from the environment.
|
||||
if [ "$PROFILE_TESTS" = "true" ]; then
|
||||
echo "Profiling this run; a flamegraph will be attached to the job."
|
||||
echo "profiled=true" >> "$GITHUB_OUTPUT"
|
||||
BASE="$BASE:profile-tests"
|
||||
fi
|
||||
# An explicit selection wins over the rerun narrowing below: it was asked for by hand, on this
|
||||
# attempt. Names may be separated by commas or spaces; hawk wants them as one vector argument.
|
||||
if [ -n "$ONLY_TESTS" ]; then
|
||||
ONLY="[$(echo "$ONLY_TESTS" | tr ',' ' ')]"
|
||||
echo "Running only: $ONLY"
|
||||
$BASE :only "$ONLY"
|
||||
elif [ "${{ github.run_attempt }}" != "1" ] && [ -s target/failed-tests ]; then
|
||||
echo "Granular rerun of previously-failed tests: $(cat target/failed-tests)"
|
||||
$BASE :only "$(cat target/failed-tests)"
|
||||
else
|
||||
$BASE :exclude-tags '[:mb/driver-tests ${{ env.__ADDITIONAL_EXCLUDED_TAG__ }}]' ${{ matrix.job.test-args }}
|
||||
fi
|
||||
run: >-
|
||||
clojure -X:dev:ci:test:${{ matrix.job.edition }}:${{ matrix.job.edition }}-dev
|
||||
:exclude-tags '[:mb/driver-tests ${{ env.__ADDITIONAL_EXCLUDED_TAG__ }}]'
|
||||
${{ matrix.job.test-args }}
|
||||
# Keep this step continue-on-error because the quarantine logic controls whether it ultimately fails.
|
||||
continue-on-error: true
|
||||
|
||||
- name: Collect profile
|
||||
if: ${{ !cancelled() }}
|
||||
uses: ./.github/actions/collect-profile
|
||||
with:
|
||||
profiled: ${{ steps.run-java-tests.outputs.profiled }}
|
||||
artifact-name: profile-be-tests-java-${{ matrix.java-version }}-${{ matrix.job.test-group-name }}
|
||||
|
||||
- name: Report failed tests to ci-conductor
|
||||
if: ${{ !cancelled() && ( steps.run-java-tests.outcome == 'failure' || github.run_attempt != '1' ) }}
|
||||
if: ${{ !cancelled() && steps.run-java-tests.outcome == 'failure' }}
|
||||
# This step should NEVER fail the job!
|
||||
continue-on-error: true
|
||||
uses: ./.github/actions/report-failures-to-ci-conductor
|
||||
@@ -393,29 +209,6 @@ jobs:
|
||||
test-suite: be-tests-java-${{ matrix.java-version }}-${{ matrix.job.test-group-name }}
|
||||
adapter: backend
|
||||
|
||||
# Granular rerun: record which tests failed so a re-run of this job targets
|
||||
# only those (restored above). Reuses the backend JUnit adapter from the
|
||||
# ci-conductor sources the report step just prepared. Best-effort: writes
|
||||
# nothing unless every failure is attributable to a var, so an
|
||||
# unattributable failure falls back to a full partition rerun.
|
||||
- name: Collect failed tests for rerun
|
||||
if: ${{ !cancelled() && steps.run-java-tests.outcome == 'failure' }}
|
||||
continue-on-error: true
|
||||
# Clear the restored set first, so a collector that dies before deciding
|
||||
# can't leave the previous attempt's file to be uploaded as this one's.
|
||||
run: |
|
||||
rm -f target/failed-tests
|
||||
bun release/ci-conductor/src/collect-failed-backend.ts
|
||||
|
||||
- name: Upload failed tests for rerun
|
||||
if: ${{ !cancelled() && steps.run-java-tests.outcome == 'failure' }}
|
||||
continue-on-error: true
|
||||
uses: actions/upload-artifact@v7
|
||||
with:
|
||||
name: failed-tests-be-tests-java-${{ matrix.java-version }}-${{ matrix.job.test-group-name }}-attempt-${{ github.run_attempt }}
|
||||
path: target/failed-tests
|
||||
if-no-files-found: ignore
|
||||
|
||||
- name: Quarantine gate
|
||||
if: ${{ !cancelled() && steps.run-java-tests.outcome == 'failure' }}
|
||||
continue-on-error: ${{ vars.QUARANTINE_ENGINE != 'ci-conductor' }}
|
||||
@@ -426,6 +219,19 @@ jobs:
|
||||
test-suite: be-tests-java-${{ matrix.java-version }}-${{ matrix.job.test-group-name }}
|
||||
enforce-quarantine: ${{ vars.QUARANTINE_ENGINE == 'ci-conductor' }}
|
||||
|
||||
- name: Upload Test Results
|
||||
uses: ./.github/actions/upload-test-results
|
||||
if: always()
|
||||
with:
|
||||
input-path: ./target/junit/
|
||||
output-name: be-tests-java-${{ matrix.java-version }}-${{ matrix.job.edition }}
|
||||
bucket: ${{ vars.AWS_S3_TEST_RESULTS_BUCKET }}
|
||||
role-to-assume: ${{ secrets.CI_TEST_RESULTS_IAM_ROLE }}
|
||||
aws-region: ${{ vars.AWS_REGION }}
|
||||
trunk-api-token: ${{ secrets.TRUNK_API_TOKEN }}
|
||||
previous-step-outcome: ${{ steps.run-java-tests.outcome }}
|
||||
run-trunk: ${{ vars.QUARANTINE_ENGINE != 'ci-conductor' }}
|
||||
|
||||
- name: Publish Test Report (JUnit)
|
||||
uses: dorny/test-reporter@a43b3a5f7366b97d083190328d2c652e1a8b6aa2 # v3.0.0
|
||||
if: failure()
|
||||
@@ -461,7 +267,7 @@ jobs:
|
||||
uses: ./.github/actions/prepare-backend
|
||||
# TODO -- we should probably also check WITHOUT ee -- do a check with only OSS namespaces on the classpath
|
||||
- name: Check namespaces
|
||||
run: clojure -M:ee:drivers:load-namespaces
|
||||
run: clojure -M:ee:drivers:check
|
||||
|
||||
be-cljfmt:
|
||||
if: ${{ !inputs.skip }}
|
||||
@@ -477,7 +283,6 @@ jobs:
|
||||
|
||||
be-tests-result:
|
||||
needs:
|
||||
- project-tests
|
||||
- be-linter-eastwood
|
||||
- be-linter-eastwood-test
|
||||
- be-linter-eastwood-test-changed
|
||||
@@ -486,7 +291,7 @@ jobs:
|
||||
- be-check
|
||||
- be-tests
|
||||
if: always() && !cancelled()
|
||||
runs-on: ${{ vars.SLIM_RUNNER_KEY }}
|
||||
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
|
||||
timeout-minutes: 5
|
||||
env:
|
||||
needs: ${{ toJson(needs) }}
|
||||
|
||||
9
.github/workflows/backport-reminder.yml
vendored
9
.github/workflows/backport-reminder.yml
vendored
@@ -12,8 +12,7 @@ concurrency:
|
||||
|
||||
jobs:
|
||||
flags:
|
||||
runs-on: ${{ vars.SLIM_RUNNER_KEY }}
|
||||
timeout-minutes: 5
|
||||
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
|
||||
outputs:
|
||||
has_backport_label: ${{ steps.c.outputs.value }}
|
||||
steps:
|
||||
@@ -28,7 +27,7 @@ jobs:
|
||||
|
||||
files-changed:
|
||||
name: Check which files changed
|
||||
runs-on: ${{ vars.SLIM_RUNNER_KEY }}
|
||||
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
|
||||
timeout-minutes: 3
|
||||
outputs:
|
||||
docs: ${{ steps.changes.outputs.docs }}
|
||||
@@ -45,7 +44,7 @@ jobs:
|
||||
needs: [ files-changed, flags ]
|
||||
if: ${{ !cancelled() }}
|
||||
name: Decide whether to backport or not
|
||||
runs-on: ${{ vars.SLIM_RUNNER_KEY }}
|
||||
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
|
||||
timeout-minutes: 5
|
||||
permissions: {}
|
||||
steps:
|
||||
@@ -60,7 +59,7 @@ jobs:
|
||||
|
||||
priority-backport-reminder:
|
||||
name: Remind about backporting P0 + P1 fixes
|
||||
runs-on: ${{ vars.SLIM_RUNNER_KEY }}
|
||||
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
|
||||
timeout-minutes: 5
|
||||
needs: [flags]
|
||||
permissions:
|
||||
|
||||
@@ -5,11 +5,10 @@ on:
|
||||
|
||||
jobs:
|
||||
check-source-branch:
|
||||
if: github.head_ref == 'master'
|
||||
runs-on: ${{ vars.SLIM_RUNNER_KEY }}
|
||||
timeout-minutes: 5
|
||||
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
|
||||
steps:
|
||||
- name: Block master branch PRs
|
||||
if: github.head_ref == 'master'
|
||||
run: |
|
||||
echo "PRs from \`master\` branches are not allowed, please use a new branch name for your PR."
|
||||
echo "PRs from `master` branches are not allowed, please use a new branch name for your PR."
|
||||
exit 1
|
||||
|
||||
270
.github/workflows/build-for-release.yml
vendored
Normal file
270
.github/workflows/build-for-release.yml
vendored
Normal file
@@ -0,0 +1,270 @@
|
||||
# Builds a fresh release JAR from scratch at a given commit.
|
||||
#
|
||||
# You should almost never have to use this. The normal release flow relies on a
|
||||
# commit that has already been built and tested on its branch — see
|
||||
# `tag-for-release.yml`, which tags that existing JAR. Building from scratch
|
||||
# here is only for exceptional cases.
|
||||
name: Release 0 - Build Release Artifact
|
||||
run-name: Build ${{ inputs.version }}
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
version:
|
||||
description: 'Metabase version (e.g. v0.46.3)'
|
||||
type: string
|
||||
required: true
|
||||
commit:
|
||||
description: 'A full-length commit SHA-1 hash'
|
||||
required: true
|
||||
|
||||
jobs:
|
||||
start-message:
|
||||
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
|
||||
timeout-minutes: 5
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
with:
|
||||
sparse-checkout: |
|
||||
release
|
||||
.github
|
||||
- name: Prepare release scripts
|
||||
uses: ./.github/actions/prepare-release-scripts
|
||||
- name: Send build start message
|
||||
uses: actions/github-script@v9
|
||||
env:
|
||||
SLACK_RELEASE_CHANNEL: ${{ vars.SLACK_RELEASE_CHANNEL }}
|
||||
SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}
|
||||
with:
|
||||
script: | # js
|
||||
const { sendPreReleaseMessage } = require('${{ github.workspace }}/release/dist/index.cjs');
|
||||
|
||||
await sendPreReleaseMessage({
|
||||
github,
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
version: '${{ inputs.version }}',
|
||||
runId: '${{ github.run_id }}',
|
||||
releaseSha: '${{ inputs.commit }}',
|
||||
userName: '${{ github.actor }}',
|
||||
channelName: '${{ vars.SLACK_RELEASE_CHANNEL }}',
|
||||
}).catch(console.error);
|
||||
|
||||
trigger-milestone-check:
|
||||
# Posts a Slack FYI about issues that still need review for this release.
|
||||
# Informational only — nothing depends on it, so it never blocks the release.
|
||||
needs: [check-version, check-commit]
|
||||
permissions:
|
||||
contents: read
|
||||
issues: read
|
||||
uses: ./.github/workflows/release-milestone-check.yml
|
||||
secrets: inherit
|
||||
with:
|
||||
version: ${{ inputs.version }}
|
||||
commit: ${{ inputs.commit }}
|
||||
|
||||
check-version:
|
||||
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
|
||||
timeout-minutes: 10
|
||||
outputs:
|
||||
ee: ${{ fromJson(steps.canonical_version.outputs.result).ee }}
|
||||
oss: ${{ fromJson(steps.canonical_version.outputs.result).oss }}
|
||||
steps:
|
||||
- name: Fail early on the incorrect version format
|
||||
if: ${{ !(startsWith(inputs.version,'v0.') || startsWith(inputs.version,'v1.')) }}
|
||||
run: |
|
||||
echo "The version format is invalid!"
|
||||
echo "It must start with either 'v0.' or 'v1.'."
|
||||
echo "Please, try again."
|
||||
exit 1
|
||||
- uses: actions/checkout@v6
|
||||
with:
|
||||
sparse-checkout: |
|
||||
release
|
||||
.github
|
||||
- name: Prepare release scripts
|
||||
uses: ./.github/actions/prepare-release-scripts
|
||||
- name: Get Release Version
|
||||
uses: actions/github-script@v9
|
||||
id: canonical_version
|
||||
with:
|
||||
script: | # js
|
||||
const { isValidVersionString, getCanonicalVersion, hasBeenReleased } = require('${{ github.workspace }}/release/dist/index.cjs');
|
||||
|
||||
const version = '${{ inputs.version }}';
|
||||
|
||||
if (!isValidVersionString(version)) {
|
||||
throw new Error("The version format is invalid! It must start with either 'v0.' or 'v1.'.");
|
||||
}
|
||||
|
||||
const versions = {
|
||||
ee: getCanonicalVersion(version, 'ee'),
|
||||
oss: getCanonicalVersion(version, 'oss'),
|
||||
};
|
||||
|
||||
const released = await hasBeenReleased({
|
||||
github,
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
version: versions.oss,
|
||||
});
|
||||
|
||||
if (released) {
|
||||
throw new Error("This version has already been released!", version);
|
||||
}
|
||||
|
||||
return versions;
|
||||
|
||||
check-commit:
|
||||
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
|
||||
timeout-minutes: 10
|
||||
steps:
|
||||
- name: Check out the code to verify the release branch
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
fetch-depth: 0 # IMPORTANT! to get all the branches
|
||||
sparse-checkout: |
|
||||
release
|
||||
.github
|
||||
- name: Prepare release scripts
|
||||
uses: ./.github/actions/prepare-release-scripts
|
||||
- name: Get Release Branch
|
||||
uses: actions/github-script@v9
|
||||
id: release_branch
|
||||
with:
|
||||
result-encoding: string
|
||||
script: | # js
|
||||
const { getReleaseBranch } = require('${{ github.workspace }}/release/dist/index.cjs');
|
||||
const version = '${{ inputs.version }}';
|
||||
const releaseBranch = getReleaseBranch(version);
|
||||
return releaseBranch;
|
||||
- name: Ensure that the specified commit exists in the ${{ steps.release_branch.outputs.result }} release branch
|
||||
run: | # bash
|
||||
RELEASE_BRANCH=${{ steps.release_branch.outputs.result }}
|
||||
git checkout $RELEASE_BRANCH
|
||||
git branch --contains ${{ inputs.commit }} | grep -q $RELEASE_BRANCH \
|
||||
&& echo "Commit found in correct release branch" \
|
||||
|| (echo "Commit not found in correct release branch" && exit 1)
|
||||
git checkout -
|
||||
|
||||
get-build-requirements:
|
||||
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
|
||||
timeout-minutes: 10
|
||||
outputs:
|
||||
java_version: ${{ fromJson(steps.dependencies.outputs.result).java_version }}
|
||||
node_version: ${{ fromJson(steps.dependencies.outputs.result).node_version }}
|
||||
build_process: ${{ fromJson(steps.dependencies.outputs.result).build_process }}
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
with:
|
||||
sparse-checkout: |
|
||||
release
|
||||
.github
|
||||
- name: Prepare release scripts
|
||||
uses: ./.github/actions/prepare-release-scripts
|
||||
- name: Get build dependencies
|
||||
uses: actions/github-script@v9
|
||||
id: dependencies
|
||||
with:
|
||||
script: | # js
|
||||
const { getBuildRequirements, getMajorVersion } = require('${{ github.workspace }}/release/dist/index.cjs');
|
||||
|
||||
const version = '${{ inputs.version }}';
|
||||
|
||||
const requirements = getBuildRequirements(version);
|
||||
const build_process = getMajorVersion(version) < 46 ? 'legacy' : 'new';
|
||||
console.log('Build process:', build_process);
|
||||
|
||||
return {
|
||||
java_version: requirements.java,
|
||||
node_version: requirements.node,
|
||||
build_process,
|
||||
};
|
||||
|
||||
manage-milestones:
|
||||
# Close the milestone we're about to ship and open the next one BEFORE building,
|
||||
# so any PRs that merge while the build is in flight are tagged with the next
|
||||
# milestone (not the one about to be released).
|
||||
#
|
||||
# Sequenced after trigger-milestone-check so that the check sees the milestone
|
||||
# in its open state. milestone-check is informational and must never block, so
|
||||
# we run regardless of its outcome; only preflight failures suppress this.
|
||||
needs: [check-version, check-commit, trigger-milestone-check]
|
||||
if: ${{ !cancelled() && needs.check-version.result == 'success' && needs.check-commit.result == 'success' }}
|
||||
permissions:
|
||||
contents: read
|
||||
issues: write
|
||||
uses: ./.github/workflows/release-milestone-manage.yml
|
||||
with:
|
||||
version: ${{ inputs.version }}
|
||||
|
||||
build-uberjar-for-release:
|
||||
needs: [check-version, check-commit, get-build-requirements]
|
||||
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
|
||||
timeout-minutes: 50
|
||||
strategy:
|
||||
matrix:
|
||||
edition: [oss, ee]
|
||||
env:
|
||||
INTERACTIVE: false
|
||||
steps:
|
||||
- name: Get correct version number
|
||||
run: | # bash
|
||||
if [ '${{matrix.edition}}' == 'ee' ]; then
|
||||
version='${{ needs.check-version.outputs.ee }}'
|
||||
else
|
||||
version='${{ needs.check-version.outputs.oss }}'
|
||||
fi
|
||||
|
||||
echo "VERSION=$version" >> $GITHUB_ENV
|
||||
- name: Setup Java ${{ needs.get-build-requirements.outputs.java_version }}
|
||||
uses: actions/setup-java@v5.2.0
|
||||
with:
|
||||
distribution: 'temurin'
|
||||
java-version: ${{ needs.get-build-requirements.outputs.java_version }}
|
||||
- name: Setup Node.js ${{ needs.get-build-requirements.outputs.node_version }}
|
||||
uses: actions/setup-node@v6.4.0
|
||||
with:
|
||||
node-version: ${{ needs.get-build-requirements.outputs.node_version }}.x
|
||||
- name: Check out the code
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
ref: ${{ github.event.inputs.commit }}
|
||||
- name: Prepare front-end environment
|
||||
uses: ./.github/actions/prepare-frontend
|
||||
|
||||
- name: Prepare back-end environment
|
||||
uses: ./.github/actions/prepare-backend
|
||||
|
||||
- name: Build Metabase
|
||||
if: ${{ needs.get-build-requirements.outputs.build_process != 'legacy' }}
|
||||
run: ./bin/build.sh :edition :${{ matrix.edition }} :version $VERSION
|
||||
|
||||
- name: Build Metabase (legacy)
|
||||
if: ${{ needs.get-build-requirements.outputs.build_process == 'legacy' }}
|
||||
run: cd ./bin/build-mb && clojure -X build/build! :edition :${{ matrix.edition }} :version $VERSION
|
||||
|
||||
- name: Store commit's SHA-1 hash
|
||||
run: echo ${{ inputs.commit }} > COMMIT-ID
|
||||
shell: bash
|
||||
- run: mv ./target/uberjar/metabase.jar ./metabase.jar
|
||||
- name: Calculate SHA256 checksum
|
||||
run: sha256sum ./metabase.jar > SHA256.sum
|
||||
shell: bash
|
||||
- name: Upload JARs as artifact
|
||||
uses: actions/upload-artifact@v7
|
||||
with:
|
||||
name: metabase-release-${{ matrix.edition }}-${{ inputs.commit }}-uberjar
|
||||
path: |
|
||||
./metabase.jar
|
||||
./COMMIT-ID
|
||||
./SHA256.sum
|
||||
|
||||
run-pre-release-tests:
|
||||
name: Run pre-release tests for ${{ inputs.version }}
|
||||
needs: build-uberjar-for-release
|
||||
uses: ./.github/workflows/pre-release.yml
|
||||
secrets: inherit
|
||||
with:
|
||||
commit: ${{ inputs.commit }}
|
||||
version: ${{ inputs.version }}
|
||||
142
.github/workflows/build-private-ee-docker-image.yml
vendored
142
.github/workflows/build-private-ee-docker-image.yml
vendored
@@ -1,142 +0,0 @@
|
||||
name: Build & push private Metabase Docker image
|
||||
|
||||
# Builds the ee uberjar for a commit and pushes it to the private ECR repo, tagged with that
|
||||
# commit's full SHA. Callers get the tag back as an output.
|
||||
#
|
||||
# The tag is per-commit rather than per-PR, so a PR leaves one image behind per pushed commit.
|
||||
# Nothing in this repo prunes them — retention is whatever lifecycle policy the ECR repo carries.
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
commit:
|
||||
description: Full sha of commit to build, optional
|
||||
workflow_call:
|
||||
inputs:
|
||||
commit:
|
||||
description: Full sha of commit to build, optional
|
||||
type: string
|
||||
|
||||
secrets:
|
||||
PR_ENV_IAM_ROLE:
|
||||
description: Role assumed via OIDC to reach ECR
|
||||
required: true
|
||||
PR_ENV_AWS_ACCOUNT_ID:
|
||||
description: Account hosting the private ECR registry
|
||||
required: true
|
||||
|
||||
outputs:
|
||||
tag:
|
||||
description: "Image tag (the commit SHA)"
|
||||
value: ${{ jobs.dockerize.outputs.tag }}
|
||||
|
||||
# Keyed on the calling workflow as well as the commit: pr-env and ai-benchmark both land here, and
|
||||
# a shared group lets a re-label on one evict the other's queued run (`cancel-in-progress: false`
|
||||
# still drops a *pending* run when a third queues). Before the extraction the two paths were
|
||||
# independent; this keeps them that way.
|
||||
concurrency:
|
||||
group: build-private-ee-docker-image-${{ github.workflow }}-${{ inputs.commit || github.event.pull_request.head.sha || github.sha }}
|
||||
cancel-in-progress: false
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
actions: read # fetch-artifact reads the artifact API
|
||||
|
||||
jobs:
|
||||
|
||||
build:
|
||||
if: github.event_name == 'workflow_dispatch' || github.event.pull_request.head.repo.full_name == github.repository
|
||||
uses: ./.github/workflows/uberjar.yml
|
||||
with:
|
||||
# Pass through only an *explicit* commit and let uberjar apply its own event fallback.
|
||||
# Resolving head.sha here instead would pin the PR path's checkout to the head commit, while
|
||||
# run-tests' uberjar keeps building the merge ref — both upload under the same
|
||||
# `metabase-ee-<head.sha>-uberjar` name, so whichever raced first would decide what the tag
|
||||
# actually contains.
|
||||
commit: ${{ inputs.commit }}
|
||||
|
||||
dockerize:
|
||||
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
|
||||
needs: [build]
|
||||
if: needs.build.result == 'success'
|
||||
timeout-minutes: 60
|
||||
permissions:
|
||||
id-token: write
|
||||
contents: read
|
||||
actions: read
|
||||
outputs:
|
||||
tag: ${{ steps.meta.outputs.tag }}
|
||||
steps:
|
||||
# Deliberately the workflow's own revision, not `inputs.commit`: this checkout is where
|
||||
# `./.github/actions/fetch-artifact` below comes from, and pinning it to the target commit
|
||||
# would run that commit's copy of the action instead of this one. Nothing else in this job
|
||||
# reads the tree.
|
||||
- name: Checkout source code
|
||||
uses: actions/checkout@v6
|
||||
- name: Configure AWS credentials
|
||||
uses: aws-actions/configure-aws-credentials@d979d5b3a71173a29b74b5b88418bfda9437d885 # v6.1.1
|
||||
with:
|
||||
role-to-assume: ${{ secrets.PR_ENV_IAM_ROLE }}
|
||||
role-session-name: GitHub_to_AWS_via_FederatedOIDC
|
||||
aws-region: us-east-1
|
||||
- name: Login to Amazon ECR
|
||||
uses: aws-actions/amazon-ecr-login@183a1442edf41672e66566b7fc560e297a290896 # v2
|
||||
with:
|
||||
registries: "${{ secrets.PR_ENV_AWS_ACCOUNT_ID }}"
|
||||
- name: Compute image tag
|
||||
id: meta
|
||||
env:
|
||||
SHA: ${{ inputs.commit || github.event.pull_request.head.sha || github.sha }}
|
||||
run: echo "tag=${SHA}" >> "$GITHUB_OUTPUT"
|
||||
- name: Check whether image already exists
|
||||
id: exists
|
||||
env:
|
||||
TAG: ${{ steps.meta.outputs.tag }}
|
||||
# Same registry the push below targets — the assumed role's default registry is not
|
||||
# necessarily this account, and querying the wrong one always reports "not found".
|
||||
REGISTRY_ID: ${{ secrets.PR_ENV_AWS_ACCOUNT_ID }}
|
||||
run: |
|
||||
set +e
|
||||
OUT=$(aws ecr describe-images \
|
||||
--registry-id "${REGISTRY_ID}" \
|
||||
--repository-name metabase-enterprise \
|
||||
--image-ids imageTag="${TAG}" 2>&1)
|
||||
STATUS=$?
|
||||
set -e
|
||||
|
||||
if [[ "${STATUS}" -eq 0 ]]; then
|
||||
echo "Image already present for ${TAG}; reusing it."
|
||||
echo "found=true" >> "$GITHUB_OUTPUT"
|
||||
elif grep -q 'ImageNotFoundException' <<< "${OUT}"; then
|
||||
echo "No image for ${TAG}; building."
|
||||
echo "found=false" >> "$GITHUB_OUTPUT"
|
||||
else
|
||||
# Anything else (missing ecr:DescribeImages, wrong registry, throttling) is
|
||||
# indistinguishable from "not found" if we swallow it, and we'd rebuild forever
|
||||
# without ever saying why. Rebuilding is still the safe answer, so warn and go on.
|
||||
echo "${OUT}" >&2
|
||||
echo "::warning::Could not determine whether ${TAG} exists in ECR (exit ${STATUS}); building unconditionally."
|
||||
echo "found=false" >> "$GITHUB_OUTPUT"
|
||||
fi
|
||||
- name: Retrieve uberjar artifact for ee
|
||||
if: steps.exists.outputs.found == 'false'
|
||||
uses: ./.github/actions/fetch-artifact
|
||||
with:
|
||||
commit: ${{ steps.meta.outputs.tag }}
|
||||
edition: ee
|
||||
extract-path: "target/uberjar"
|
||||
output-name: "metabase.jar"
|
||||
- name: Move uberjar to bin/docker
|
||||
if: steps.exists.outputs.found == 'false'
|
||||
run: mv target/uberjar/metabase.jar bin/docker/metabase.jar
|
||||
- name: Build container
|
||||
if: steps.exists.outputs.found == 'false'
|
||||
uses: docker/build-push-action@f9f3042f7e2789586610d6e8b85c8f03e5195baf # v7.2.0
|
||||
with:
|
||||
context: bin/docker/
|
||||
platforms: linux/amd64
|
||||
network: host
|
||||
build-args: |
|
||||
GIT_COMMIT_SHA=${{ steps.meta.outputs.tag }}
|
||||
tags: ${{ secrets.PR_ENV_AWS_ACCOUNT_ID }}.dkr.ecr.us-east-1.amazonaws.com/metabase-enterprise:${{ steps.meta.outputs.tag }}
|
||||
push: true
|
||||
73
.github/workflows/build-scripts.yml
vendored
73
.github/workflows/build-scripts.yml
vendored
@@ -9,77 +9,8 @@ on:
|
||||
|
||||
jobs:
|
||||
|
||||
files-changed:
|
||||
name: Check which files changed
|
||||
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
|
||||
timeout-minutes: 3
|
||||
permissions:
|
||||
contents: read
|
||||
outputs:
|
||||
embedder_plugin: ${{ steps.changes.outputs.embedder_plugin }}
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
- name: Test which files changed
|
||||
uses: dorny/paths-filter@fbd0ab8f3e69293af611ebaee6363fc25e6d187d # v4.0.1
|
||||
id: changes
|
||||
with:
|
||||
token: ${{ github.token }}
|
||||
filters: .github/file-paths.yaml
|
||||
|
||||
test-embedder-plugin:
|
||||
needs: files-changed
|
||||
if: needs.files-changed.outputs.embedder_plugin == 'true'
|
||||
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
|
||||
timeout-minutes: 20
|
||||
permissions:
|
||||
contents: read
|
||||
services:
|
||||
postgres:
|
||||
image: pgvector/pgvector:pg17
|
||||
env:
|
||||
POSTGRES_USER: mb_test
|
||||
POSTGRES_DB: mb_test
|
||||
POSTGRES_HOST_AUTH_METHOD: trust
|
||||
ports:
|
||||
- 5432:5432
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
- name: Prepare back-end environment
|
||||
uses: ./.github/actions/prepare-backend
|
||||
- name: Build embedder plugin
|
||||
run: ./bin/build-embedder-plugin.sh
|
||||
- name: Verify packaged plugin resources
|
||||
run: |
|
||||
jar tf modules/embedder/target/metabase-embedder-plugin.jar | grep -Fx metabase/embedder/metabase-plugin.yaml
|
||||
jar tf modules/embedder/target/metabase-embedder-plugin.jar | grep -Fx metabase/embedder-model-catalog.edn
|
||||
jar tf modules/embedder/target/metabase-embedder-plugin.jar | grep -Fx metabase-embedder/snowflake-arctic-embed-xs-d8c86521100d3556476a063fc2342036d45c106f-arm64.zip
|
||||
jar tf modules/embedder/target/metabase-embedder-plugin.jar | grep -Fx metabase-embedder/snowflake-arctic-embed-xs-d8c86521100d3556476a063fc2342036d45c106f-avx2.zip
|
||||
jar tf modules/embedder/target/metabase-embedder-plugin.jar | grep -Fx metabase-embedder/all-minilm-l6-v2-1110a243fdf4706b3f48f1d95db1a4f5529b4d41-arm64.zip
|
||||
jar tf modules/embedder/target/metabase-embedder-plugin.jar | grep -Fx metabase-embedder/all-minilm-l6-v2-1110a243fdf4706b3f48f1d95db1a4f5529b4d41-avx2.zip
|
||||
- name: Run artifact-only plugin smoke test
|
||||
env:
|
||||
MB_EMBEDDER_ARTIFACT_TEST: "true"
|
||||
MB_PLUGINS_DIR: modules/embedder/target
|
||||
run: >-
|
||||
./bin/test-agent --oss :only
|
||||
'[metabase.embeddings.embedder-plugin-artifact-test]'
|
||||
- name: Run in-process semantic-search round trip
|
||||
env:
|
||||
MB_DB_TYPE: postgres
|
||||
MB_DB_PORT: 5432
|
||||
MB_DB_HOST: localhost
|
||||
MB_DB_DBNAME: mb_test
|
||||
MB_DB_USER: mb_test
|
||||
MB_APPDB_PGVECTOR_MODE_TEST: "true"
|
||||
MB_IN_PROCESS_EMBEDDER_SEMANTIC_SEARCH_TEST: "true"
|
||||
MB_PLUGINS_DIR: modules/embedder/target
|
||||
run: >-
|
||||
./bin/test-agent --drivers=postgres :only
|
||||
'[metabase-enterprise.semantic-search.appdb-pgvector-mode-test/appdb-pgvector-in-process-embedder-round-trip-test]'
|
||||
|
||||
test-build-scripts:
|
||||
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
|
||||
timeout-minutes: 40
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
- name: Prepare back-end environment
|
||||
@@ -89,7 +20,7 @@ jobs:
|
||||
- name: Run build and release script tests
|
||||
run: clojure -X:dev:drivers:build:build-dev:build-test:ci
|
||||
timeout-minutes: 15
|
||||
- name: Run load-namespaces tests
|
||||
- name: Run Liquibase migrations linter tests
|
||||
run: clojure -M:test
|
||||
working-directory: bin/load-namespaces
|
||||
working-directory: bin/lint-migrations-file
|
||||
timeout-minutes: 15
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user