Compare commits
14 Commits
nav-split-
...
openapi
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
95d1dd7ab2 | ||
|
|
852d517ffd | ||
|
|
b620dc59ae | ||
|
|
65b860b8cc | ||
|
|
37a20c0559 | ||
|
|
d6070d4bbc | ||
|
|
9e4b4633f2 | ||
|
|
0244a3fd5d | ||
|
|
e490c4624f | ||
|
|
df6b59a6dc | ||
|
|
736b84325d | ||
|
|
7da83c2058 | ||
|
|
7e55482459 | ||
|
|
585bac6353 |
@@ -149,14 +149,18 @@ ms/NonBlankString ;; Non-empty string
|
||||
ms/BooleanValue ;; String "true"/"false" or boolean
|
||||
ms/MaybeBooleanValue ;; BooleanValue or nil
|
||||
ms/TemporalString ;; ISO-8601 date/time string (for REQUEST params only!)
|
||||
ms/TemporalInstant ;; java.time object (for RESPONSE schemas; documents as ISO-8601 string in OpenAPI)
|
||||
ms/Map ;; Any map
|
||||
ms/JSONString ;; JSON-encoded string
|
||||
ms/PositiveNum ;; Positive number
|
||||
ms/IntGreaterThanOrEqualToZero ;; 0 or positive
|
||||
```
|
||||
|
||||
**IMPORTANT:** For response schemas, use `:any` for temporal fields, not `ms/TemporalString`!
|
||||
Response schemas validate BEFORE JSON serialization, so they see Java Time objects.
|
||||
**IMPORTANT:** For response schemas, use `ms/TemporalInstant` for temporal fields, not `ms/TemporalString`!
|
||||
Response schemas validate BEFORE JSON serialization, so they see Java Time objects. `ms/TemporalInstant`
|
||||
validates the java.time object and still documents the field as an ISO-8601 string in the generated
|
||||
OpenAPI spec (which frontend TS types are generated from) — prefer it over `:any`, which turns the
|
||||
field into `unknown` on the frontend.
|
||||
|
||||
### Built-in Malli Types
|
||||
|
||||
@@ -307,12 +311,15 @@ Avoid using sequence schemas unless completely necessary.
|
||||
;; WRONG - Java Time objects aren't strings yet
|
||||
[:date_joined ms/TemporalString]
|
||||
|
||||
;; RIGHT - schemas validate BEFORE JSON serialization
|
||||
[:date_joined :any] ;; Java Time object, serialized to string by middleware
|
||||
[:last_login [:maybe :any]] ;; Java Time object or nil
|
||||
;; WRONG - validates, but documents the field as `unknown` in generated TS types
|
||||
[:date_joined :any]
|
||||
|
||||
;; RIGHT - validates the java.time object, documents as ISO-8601 string in OpenAPI
|
||||
[:date_joined ms/TemporalInstant]
|
||||
[:last_login [:maybe ms/TemporalInstant]] ;; Java Time object or nil
|
||||
```
|
||||
|
||||
**Why:** Response schemas validate the internal Clojure data structures BEFORE they are serialized to JSON. Java Time objects like `OffsetDateTime` get converted to ISO-8601 strings by the JSON middleware, so the schema needs to accept the raw Java objects.
|
||||
**Why:** Response schemas validate the internal Clojure data structures BEFORE they are serialized to JSON. Java Time objects like `OffsetDateTime` get converted to ISO-8601 strings by the JSON middleware, so the schema needs to accept the raw Java objects. `ms/TemporalInstant` does that while carrying a `:json-schema` override (`{:type "string" :format "date-time"}`) so the OpenAPI spec — and the frontend TS types generated from it — see a proper string type.
|
||||
|
||||
### Don't: Use `[:sequential X]` when the data is actually a set
|
||||
|
||||
|
||||
@@ -13,14 +13,14 @@ allowed-tools: Read, Grep, Bash, Glob, Skill
|
||||
|
||||
**Primary standard: the [`typescript-write`](../typescript-write/SKILL.md) skill.** Load it first — it defines the authoring rules this review enforces, alongside `frontend/CLAUDE.md` and `docs/developers-guide/frontend.md`.
|
||||
|
||||
Adherence to `typescript-write` is the **highest-priority** review dimension: rank any violation of its provisions above all other findings. Treat its **no-`any` hard rule** (no explicit *or* implicit `any` in new code) as **blocking**. Use TypeScript LSP tools to inspect inferred types when available; otherwise rely on type-checking and linting.
|
||||
Adherence to `typescript-write` is the **highest-priority** review dimension: rank any violation of its provisions above all other findings. Treat its **no-`any` hard rule** (no explicit *or* implicit `any` in new code, including disguised forms such as return-only generics) as **blocking**. Use TypeScript LSP tools to inspect inferred types when available; otherwise rely on type-checking and linting.
|
||||
|
||||
Review in this priority order:
|
||||
|
||||
1. **Violations of [`typescript-write`](../typescript-write/SKILL.md) provisions** — no-`any`, type tightening, type modeling, null/undefined handling, naming, structure, comments. Highest priority; block on the no-`any` rule.
|
||||
1. **Violations of [`typescript-write`](../typescript-write/SKILL.md) provisions** — no-`any`, type tightening, type modeling, function signatures, null/undefined handling, naming, structure, comments. Highest priority; block on the no-`any` rule.
|
||||
2. Compliance with `frontend/CLAUDE.md`.
|
||||
3. Readability and maintainability.
|
||||
4. Appropriate test coverage.
|
||||
4. Appropriate test coverage. Types and tests are complementary: rely on the type checker for type-level guarantees and don't request tests for inputs the type system already rules out — unless security or data corruption is at stake.
|
||||
|
||||
## Blind spots — act as the missing reviewer
|
||||
|
||||
@@ -31,3 +31,4 @@ These rarely surface in team reviews, so this skill should raise them. They are
|
||||
- **Security.** Evaluate potential security issues in new code.
|
||||
- **Bundle size.** Flag new large dependencies, default imports from icon or util libs, and heavy modules imported at route-load time.
|
||||
- **Analytics.** User-facing flows should emit tracking events. If a PR adds a new flow (button, modal, navigation) without a tracking event, ask whether one is expected.
|
||||
- **Public API surface** (embedding SDK). Every type appearing in a public method's parameters or return type must itself be exported; exported APIs need TSDoc, with `@deprecated` on retired ones.
|
||||
|
||||
@@ -13,6 +13,7 @@ description: Write TypeScript and JavaScript code following Metabase coding stan
|
||||
|
||||
- **New code must not introduce `any`, explicit or implicit.** No `any` annotations, no `as any` / `as unknown as`, no untyped parameters or returns that infer `any`, no implicitly-`any` destructures or array/object literals.
|
||||
- **Untyped third-party / boundary values** must be typed at the boundary (a declared type, `unknown` + type guard, or a small typed wrapper) — never let `any` propagate inward.
|
||||
- **No return-only generics.** A type parameter must relate types — it should appear at least twice in the signature (parameters, return, or constraints). `get<T>(): T` is a disguised cast that fakes type safety; use `unknown` and narrow at the call site instead.
|
||||
- **Mandatory type verification.** Before finishing a TS/TSX change, run `bun run type-check-pure`. If TypeScript LSP tools are available, also inspect changed symbols with hover and go-to-definition; otherwise skip LSP check.
|
||||
|
||||
## Type tightening
|
||||
@@ -20,47 +21,60 @@ description: Write TypeScript and JavaScript code following Metabase coding stan
|
||||
- **Avoid type casts and loose `unknown`** — fix the signature instead.
|
||||
- **If a function only needs one field of a wide object, accept that field** — not the wide object. The cast often disappears once the signature is right.
|
||||
- **Reach for `Partial<T>`, `Pick<T, K>`, `Record<K, V>`, and generics** before reaching for a cast.
|
||||
- **Prefer `Record` with a union key type or `Map` over string index signatures** (`{[key: string]: T}`) — index signatures erode type safety the way `any` does.
|
||||
- **Prefer making props/components generic** (`<T>`) when a value flows through unchanged and the caller knows the type.
|
||||
- **Prefer `unknown` over loose typing** and narrow before use — an `unknown` value forces a guard at the point of use.
|
||||
- **`satisfies` for object literals** that must conform without widening (config objects, lookup maps, discriminated literals) — better than `: T` (widens) or `as T` (unsafe).
|
||||
- **Avoid non-null assertions (`!`)**. Prefer a guard, early return, or `?.`. Use `!` only when non-nullness is provably true and localized, with a comment.
|
||||
- **Indexed lookups can miss at runtime.** `arr[i]` and `record[key]` are typed `T` (the project doesn't enable `noUncheckedIndexedAccess`) — guard lookups that may miss. Iterate objects with `Object.entries`; a `keyof` assertion in `for-in` requires statically known keys and a justification.
|
||||
- **No redundant runtime coercion** — don't wrap already-typed values in `Number()` / `String()` / `Boolean()`.
|
||||
- **Type guards belong in `frontend/src/metabase-types/guards/`**. Do not redefine them locally.
|
||||
- **A cast you can't avoid needs a real justification comment.** The `metabase/no-unjustified-type-casts` rule accepts any preceding comment — state the actual reason the cast is safe. NEVER write the legacy `// Unjustified type cast. FIXME` placeholder; it exists only on casts that predated the rule, and copying it sneaks an unjustified cast past the linter. If you can't articulate why the cast is correct, the cast is wrong — fix the types.
|
||||
- **Contain a truly unavoidable unsafe cast** inside one small function with a correct signature and unit tests — never let it leak into callers' types, and never loosen a function's declared signature to silence errors in its implementation.
|
||||
- **A cast you can't avoid needs a real justification comment** stating why it's safe — the `metabase/no-unjustified-type-casts` rule accepts any preceding comment. NEVER copy the legacy `// Unjustified type cast. FIXME` placeholder; it only marks pre-rule debt and copying it sneaks an unjustified cast past the linter. If you can't articulate why the cast is correct, the cast is wrong — fix the types.
|
||||
|
||||
## Type modeling
|
||||
|
||||
- **API layer and generated OpenAPI types.** When adding or changing RTK Query endpoints, editing request/response types in `metabase-types/api`, or migrating endpoints to auto-generated OpenAPI types, first read [`openapi-generated-types.md`](./openapi-generated-types.md) — it defines the alias-layer migration procedure and the schema-first debugging order.
|
||||
- **Reuse existing types; don't re-declare them.** Use canonical IDs and domain entity types from `metabase-types/api` (`FieldId`, `TableId`, `ConcreteTableId`, `SchemaName`, …) and key data structures by them (`new Map<ConcreteTableId, …>()`). Don't duplicate generated/API types — compose or derive (`Pick`, `Omit`, indexed access `SomeType["field"]`, `ReturnType`).
|
||||
- **Use generics to allow TypeScript to infer correct types** when creating functions and components that need to be reusable and type-safe. Don't hesitate to introduce complex generics if they allow to derive types automatically instead of manual narrowing.
|
||||
- **Model the actual data contract; keep types narrow.** Optional `field?: T` for a key that may be absent, `field: T | undefined` only when the key is always present but the value may be undefined, `| null` for explicit API nulls. Prefer domain unions over broad `string` / `number` / loose `Record`.
|
||||
- **Refer to API implementation** when defining or refining types to ensure they match the actual data structure. When considering a type cast, first consider if the type should be refined to match the actual data structure.
|
||||
- **Use generics to let TypeScript infer types** in reusable functions and components. Complex generics are fine when they derive types automatically instead of forcing manual narrowing — but prefer an imprecise honest type over a precise wrong one. If you can't model something accurately, use a simpler type or `unknown` rather than a complex type that lies.
|
||||
- **Model the actual data contract; keep types narrow.** Optional `field?: T` for a key that may be absent, `field: T | undefined` only when the key is always present but the value may be undefined, `| null` for explicit API nulls. Prefer domain unions over broad `string` / `number` / loose `Record`. Refer to the API endpoint implementation for the actual shape and nullability — and when tempted to cast, first check whether the type should be refined to match reality.
|
||||
- **No sentinel values.** Don't encode special states as `-1`, `""`, or `0` — they type-check everywhere a regular value does. Use `null`/`undefined`, or a tagged-union variant when the meaning needs a name.
|
||||
- **Think twice before `field?:`.** Optional properties hide missing-value bugs and breed scattered, inconsistent defaulting. Prefer required fields; normalize loose input into a fully populated type at the boundary. Avoid a combinatorial explosion of independent options.
|
||||
- **Make nullability atomic.** When several fields are absent together, make the containing object nullable (or a separate union variant) instead of many independently nullable fields whose validity is implicitly linked.
|
||||
- **Discriminated unions for variant state, with exhaustive checks.** Model "one of N shapes" as a union with a literal discriminant rather than a bag of optional fields, and exhaust it with ts-pattern's `.exhaustive()` so adding a variant becomes a compile error:
|
||||
```ts
|
||||
import { match } from "ts-pattern";
|
||||
|
||||
const result = match(status)
|
||||
.with({ type: "loading" }, () => <Spinner />)
|
||||
const result = match(state)
|
||||
.with({ type: "error", error: P.select() }, (error) => <Error message={error.message} />)
|
||||
.with({ type: "success", data: P.select() }, (data) => <Content data={data} />)
|
||||
.exhaustive(); // Compile-time guarantee all cases handled
|
||||
```
|
||||
- **Derive union types from constants** (`as const` + `typeof`/`keyof`) so the type and the values can't drift.
|
||||
- **`readonly` / immutability where mutation isn't intended** — component props, shared constants, exported config. Prefer `readonly T[]` / `ReadonlyArray<T>` for inputs you don't mutate.
|
||||
- **Derive union types from constants** (`as const` + `typeof`/`keyof`) so the type and the values can't drift. Don't use `enum`, parameter properties, or namespaces — prefer plain ECMAScript constructs over TypeScript-only runtime features.
|
||||
- **`readonly` / immutability where mutation isn't intended** — component props, shared constants, exported config, unmutated parameters. Prefer `readonly T[]` / `ReadonlyArray<T>` for inputs you don't mutate; never mutate parameters — it breaks the caller's narrowing.
|
||||
- **Build objects in one expression** — object spread plus conditional spread (`...(cond ? { x } : {})`) — rather than declaring loose and mutating properties in, which forces widened types or casts.
|
||||
- **Treat `metabase-lib` opaque types as black boxes.** Types like `Lib.Query` are branded on purpose; go through `metabase-lib` functions, never cast into their internals.
|
||||
- **Type async and error states explicitly** (a discriminated union or the data-layer's typed result) — never leave loading/error/empty implicit.
|
||||
|
||||
## Function signatures
|
||||
|
||||
- **Annotate signatures, let locals infer.** Parameter and return types belong on exported/shared functions; don't clutter bodies with annotations TypeScript infers. Do annotate an object literal's type when you want excess-property checking and errors at the declaration rather than at first use.
|
||||
- **Accept broadly, return strictly.** Optional and union types are fine in parameters but awkward in return types — don't make callers unpick a union or optional you could have narrowed before returning.
|
||||
- **Avoid consecutive parameters of the same type** — swapped arguments still type-check. Use an options object for same-type neighbors or 3+ parameters.
|
||||
- **Prefer `async`/`await` over raw promise chains and callbacks** — better inference, flow, and error handling. Declare Promise-returning functions `async`.
|
||||
|
||||
## Null and undefined
|
||||
|
||||
- **Narrow at the source**. If a value is optional only in a corner case, don't thread `undefined` through every layer — guard at the producer.
|
||||
- **Sensible defaults for optional values**. Use `?.` and `??` at the consumer.
|
||||
- **Lists should be filtered** before being used in a map or other iteration.
|
||||
- **Avoid non-strict null comparisons** (`X != null`) when `X` can never be `null` — use a strict check or narrow the type. Use `checkNotNull` where necessary.
|
||||
- **Check actual nullability against API implementation**. Find the API endpoint implementation and check if the field can actually be null.
|
||||
- **Filter lists before iterating** — e.g. `.filter(isNotNull)` before `.map`.
|
||||
- **Avoid non-strict null comparisons** (`X != null`) when `X` can never be `null` — use a strict check or narrow the type. Use `checkNotNull` / `isNotNull` from `metabase/utils/types` where appropriate.
|
||||
|
||||
## Naming
|
||||
|
||||
- **Names describe the entity, not the mechanism**. A name must reflect what the value holds.
|
||||
- **Align sibling concepts**: keep verb conventions consistent across a related API.
|
||||
- **No names that encode implementation history** rather than current meaning. Suffixes like `Base`, `New`, `Old`, `Initial` need a real semantic distinction, otherwise drop them.
|
||||
- **Avoid vague type names** (`Info`, `Data`, `Entity` suffixes) — use the problem domain's own vocabulary, and name types for what they are, not their shape.
|
||||
- **Include units when the type doesn't carry them** (`timeoutMs`, `widthPx`, `temperatureC`).
|
||||
- **Avoid cryptic identifiers** (`v`, `n`, `$n`) for domain values; short names are fine only in tiny conventional contexts (loop index `i`, coordinates `x`/`y`, generic params `T`/`K`/`V`).
|
||||
|
||||
## Code structure and organization
|
||||
@@ -73,7 +87,7 @@ description: Write TypeScript and JavaScript code following Metabase coding stan
|
||||
## Comments
|
||||
|
||||
- **No comments by default**. Well-named identifiers carry the `what`.
|
||||
- **Comments should be concise**. Add a short, concise comment only when the `why` is non-obvious: a workaround, a hidden invariant, a subtle ordering constraint, a clever reduction. Never document the actual implementation, focus on the intent and the why.
|
||||
- **Comments should be concise**. Add a short comment only when the `why` is non-obvious: a workaround, a hidden invariant, a subtle ordering constraint, a clever reduction. Never document the implementation or restate type information the annotations already carry — focus on intent.
|
||||
|
||||
## Verify before done
|
||||
|
||||
|
||||
63
.claude/skills/typescript-write/openapi-generated-types.md
Normal file
63
.claude/skills/typescript-write/openapi-generated-types.md
Normal file
@@ -0,0 +1,63 @@
|
||||
# Auto-Generated OpenAPI Types (Frontend API Layer)
|
||||
|
||||
Rules for working with generated backend API types. Applies when adding or changing RTK Query endpoints, editing request/response types in `metabase-types/api`, or migrating an endpoint's handwritten types to generated types.
|
||||
|
||||
## Pipeline facts
|
||||
|
||||
- **`frontend/src/metabase-types/openapi` is generated, gitignored, and never edited by hand.** It is produced from backend Malli schemas. If it's missing, run `bun run types:ensure`; after changing backend schemas, run `bun run types:ensure --force-local` (requires Clojure/JVM) to regenerate from source.
|
||||
- **Type names derive from the HTTP route or the named Malli schema**: `GetApiEeErdData` (request — with `query` / `body` / `path` slots), `GetApiEeErdResponse` (2xx response body), `MetabaseEnterpriseErdImplErdNode` (named schema). These names are not meant for humans — consumers keep human-readable names via the alias layer below.
|
||||
- **`import type` only** — eslint-enforced. The module is declaration-only; a value import resolves to nothing at runtime.
|
||||
- **The backend Malli schema is the contract, not the handwritten frontend type.** Responses are runtime-validated against response schemas outside prod, so backend endpoint tests are ground truth for what a schema should say.
|
||||
|
||||
## Migrating an endpoint to generated types
|
||||
|
||||
1. **Find where the endpoint's request/response types are declared today** (usually `metabase-types/api/*.ts`, sometimes next to the RTK endpoint). That file is the integration point — no new layer, folder, or architecture.
|
||||
2. **Replace the handwritten declarations with aliases over generated types, keeping the same exported names.** Consumers continue to import `ErdParams` / `ErdResponse`; only the declaration site changes. A pure rename needs no more than:
|
||||
```ts
|
||||
import type { GetApiEeErdData, GetApiEeErdResponse } from "metabase-types/openapi";
|
||||
|
||||
export type ErdParams = GetApiEeErdData["query"];
|
||||
export type ErdResponse = GetApiEeErdResponse;
|
||||
```
|
||||
3. **Re-apply frontend realities as documented overrides** in that same alias. Legitimate override reasons, each of which needs a comment:
|
||||
- semantic ID aliases (`FieldId`, `ConcreteTableId`, `DatabaseId`, `SchemaName`, …) over wire `number` / `string`
|
||||
- opaque branded types (e.g. `DatasetQuery`)
|
||||
- store-side hydration slots (e.g. an entity's `table` filled by the Redux store) — a frontend convention, never to be added to the backend schema
|
||||
4. **Guard every override with a compile-time drift check** so a backend schema change fails the build instead of being silently masked:
|
||||
```ts
|
||||
export type ErdNode = Omit<MetabaseEnterpriseErdImplErdNode, "table_id"> & {
|
||||
table_id: ConcreteTableId;
|
||||
};
|
||||
|
||||
type MutuallyAssignable<A, B> = [A] extends [B] ? ([B] extends [A] ? true : false) : false;
|
||||
// `Expect<false>` violates the constraint → drift becomes a compile error on this line.
|
||||
type Expect<_T extends true> = never;
|
||||
type _MatchesErdNode = Expect<MutuallyAssignable<ErdNode, MetabaseEnterpriseErdImplErdNode>>;
|
||||
```
|
||||
Pure re-exports without an override need no guard — they can't drift.
|
||||
5. **Do not bind generated types directly into an RTK endpoint** (`builder.query<GetXResponse, GetXData["query"]>`) while named handwritten types and their consumers exist — migrate through the alias so every consumer moves atomically. Direct binding is acceptable only for a brand-new endpoint with no existing types.
|
||||
6. **Migration is atomic per endpoint**: alias layer, all consumer fixes, and a green type-check land together.
|
||||
|
||||
## When type-check fails after integration
|
||||
|
||||
Investigate in this order — the failure is a finding about the contract, not an obstacle:
|
||||
|
||||
1. **Suspect the backend Malli schema first.** Read the endpoint's `defendpoint`, its implementation, and any hydration it applies. Frequent schema defects:
|
||||
- **optional vs nullable confusion**: a map-literal response builder emits every key always — such keys are required-and-nullable, not `:optional`
|
||||
- schema requires a key the handler never emits (this 400s every real response under validation, not just tsc)
|
||||
- missing or loose response schemas (`:any`, no response schema at all) — backfill the schema rather than keeping handwritten types
|
||||
- temporal fields typed loosely — use `ms/TemporalInstant` for `created_at`-style fields
|
||||
- enums missing legacy values that still occur in the wild
|
||||
|
||||
**Before editing any Malli schema, read [add-malli-schemas](../add-malli-schemas/SKILL.md)** — it defines the repo's schema reference files and the checklist for route params, request bodies, and response schemas. With that skill loaded, fix incorrect schemas — or add the schema when the endpoint is missing one entirely (no request-param or response schema, or a generated type that comes out empty/loose because of it). Then rerun the endpoint's backend tests (runtime response validation makes them the ground truth), regenerate types (`bun run types:ensure --force-local`), and re-check.
|
||||
2. **Then reconcile frontend drift.** For each remaining error decide explicitly which case it is:
|
||||
- **handwritten type was wrong** (claims fields the API never returns, wrong casing — e.g. wire kebab-case vs handwritten snake_case — wrong nullability) → adopt the generated shape and fix consumers
|
||||
- **generated type is right; consumers relied on the old looseness** → narrow or guard at the call sites (e.g. filter out virtual IDs before calling)
|
||||
- **frontend genuinely needs a different shape** → documented override + drift guard in the alias layer
|
||||
3. **Never park type errors.** The migration isn't done while `bun run type-check-pure` is red; don't commit "temporarily accepted" errors or widen an alias just to silence tsc.
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **Trust order: backend implementation > backend Malli schema > generated type > handwritten type.** Resolve drift against the top of that chain — never by reshaping the alias to match the old handwritten type unverified.
|
||||
- `type-check-pure` regenerates stale types automatically, but the editor's TS server may lag — restart it after regeneration before trusting hover/inline errors.
|
||||
- Overrides are debt: prefer fixing the backend schema over adding an override, and expect the override set to shrink over time.
|
||||
1
.gitattributes
vendored
1
.gitattributes
vendored
@@ -12,4 +12,3 @@
|
||||
# CI automation, not edited by hand. Mark generated so tools/humans don't review or diff them as source.
|
||||
docs/api.json linguist-generated
|
||||
docs/api.html linguist-generated
|
||||
resources/openapi/openapi.json linguist-generated
|
||||
|
||||
18
.github/file-paths.yaml
vendored
18
.github/file-paths.yaml
vendored
@@ -293,14 +293,16 @@ serialization_baseline:
|
||||
- ".github/workflows/serialization.yml"
|
||||
- "test_resources/serialization_baseline/**"
|
||||
|
||||
# Anything that can change OpenAPI generation, validation, or its backend source inputs.
|
||||
openapi:
|
||||
- ".github/file-paths.yaml"
|
||||
- ".github/workflows/lint.yml"
|
||||
- "src/metabase/api/**"
|
||||
- "src/metabase/api_routes/**"
|
||||
- "src/metabase/*_rest/api.clj"
|
||||
- "src/metabase/*/api.clj"
|
||||
- "src/metabase/*/api/*.clj"
|
||||
- "enterprise/backend/src/metabase_enterprise/**/api.clj"
|
||||
- "enterprise/backend/src/metabase_enterprise/**/api/*.clj"
|
||||
- "resources/openapi/**"
|
||||
- "frontend/build/openapi/**"
|
||||
- "package.json"
|
||||
- "bun.lock"
|
||||
- "src/**"
|
||||
- "enterprise/backend/src/**"
|
||||
- "**/deps.edn"
|
||||
- "resources/data_readers.clj"
|
||||
- "resources/locales.clj"
|
||||
- "resources/version.properties"
|
||||
|
||||
2
.github/scripts/create-test-plan.ts
vendored
2
.github/scripts/create-test-plan.ts
vendored
@@ -29,6 +29,8 @@ const STORY_GLOBS = [
|
||||
function listFiles(roots: string[], globs: string[]): string[] {
|
||||
const tracked = execFileSync("git", ["ls-files", "--", ...roots], {
|
||||
encoding: "utf8",
|
||||
// The tracked frontend file list is close to Node's default 1 MiB buffer.
|
||||
maxBuffer: 10 * 1024 * 1024,
|
||||
})
|
||||
.split("\n")
|
||||
.map((line) => line.trim())
|
||||
|
||||
23
.github/workflows/e2e-coverage-manifest.yml
vendored
23
.github/workflows/e2e-coverage-manifest.yml
vendored
@@ -48,17 +48,16 @@ jobs:
|
||||
name: instrumented-uberjar
|
||||
path: target/uberjar/metabase.jar
|
||||
retention-days: 1
|
||||
# Regenerate the OpenAPI spec from this SHA's defendpoint definitions so
|
||||
# route normalization in the merge job matches against the endpoints
|
||||
# that actually served this run, not the checked-in file (which is only
|
||||
# refreshed by the label-gated openapi-check workflow and drifts).
|
||||
# Generate the OpenAPI spec from this SHA's defendpoint definitions so
|
||||
# route normalization in the merge job matches the endpoints that served
|
||||
# this run.
|
||||
- name: Generate OpenAPI route table
|
||||
run: bun run generate-openapi
|
||||
run: bun run openapi:generate
|
||||
- name: Upload OpenAPI route table
|
||||
uses: actions/upload-artifact@v7
|
||||
with:
|
||||
name: openapi-spec
|
||||
path: resources/openapi/openapi.json
|
||||
path: .tmp/openapi/openapi.json
|
||||
retention-days: 1
|
||||
|
||||
# Fixed-N sharding — coverage doesn't need the PR matrix-builder's tag/edition
|
||||
@@ -143,15 +142,15 @@ jobs:
|
||||
- name: Unzip previous manifest
|
||||
if: steps.prev.outputs.found == 'true'
|
||||
run: unzip -o prev-manifest.zip -d prev-manifest
|
||||
# Overwrites the checked-out resources/openapi/openapi.json with the one
|
||||
# generated from this run's SHA. If the artifact is somehow missing, the
|
||||
# checked-in file is still there and the builder falls back to it.
|
||||
# Places this run's generated document where the coverage builder reads
|
||||
# its route table. If the artifact is missing, route normalization falls
|
||||
# back to its regex-only behavior.
|
||||
- name: Download OpenAPI route table
|
||||
continue-on-error: true
|
||||
uses: actions/download-artifact@v8
|
||||
with:
|
||||
name: openapi-spec
|
||||
path: resources/openapi
|
||||
path: .tmp/openapi
|
||||
# GITHUB_SHA (the master commit this nightly built from) is stamped into
|
||||
# the manifest as `builtAt`, so a PR can later pick the manifest closest
|
||||
# to its own point in history. See resolve-coverage-manifest.mjs.
|
||||
@@ -166,8 +165,8 @@ jobs:
|
||||
# step expects the manifests at the archive root.
|
||||
- name: Include OpenAPI spec in artifact
|
||||
run: |
|
||||
if [ -f resources/openapi/openapi.json ]; then
|
||||
cp resources/openapi/openapi.json e2e/coverage/openapi.json
|
||||
if [ -f .tmp/openapi/openapi.json ]; then
|
||||
cp .tmp/openapi/openapi.json e2e/coverage/openapi.json
|
||||
fi
|
||||
# All files share one artifact so the per-test manifest always travels
|
||||
# with the spec manifest it was built from (backfill and PR-side
|
||||
|
||||
12
.github/workflows/frontend.yml
vendored
12
.github/workflows/frontend.yml
vendored
@@ -76,13 +76,23 @@ jobs:
|
||||
fe-type-check:
|
||||
if: ${{ !inputs.skip }}
|
||||
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
|
||||
timeout-minutes: 10
|
||||
timeout-minutes: 15
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
- name: Prepare front-end environment
|
||||
uses: ./.github/actions/prepare-frontend
|
||||
- name: Prepare back-end environment
|
||||
uses: ./.github/actions/prepare-backend
|
||||
- name: Restore OpenAPI cache
|
||||
uses: actions/cache@v5
|
||||
with:
|
||||
path: |
|
||||
.tmp/openapi/openapi.json
|
||||
.tmp/openapi/generation.json
|
||||
frontend/src/metabase-types/openapi
|
||||
key: openapi-v3-${{ hashFiles('src/**', 'enterprise/backend/src/**', '**/deps.edn', 'resources/data_readers.clj', 'resources/locales.clj', 'resources/version.properties', 'package.json', 'bun.lock', 'frontend/build/openapi/**') }}
|
||||
- name: Prepare OpenAPI types
|
||||
run: bun run types:ensure
|
||||
- run: bun run build-pure:cljs
|
||||
name: Compile CLJS
|
||||
- run: bun run type-check-pure
|
||||
|
||||
69
.github/workflows/lint.yml
vendored
69
.github/workflows/lint.yml
vendored
@@ -1,10 +1,14 @@
|
||||
name: Lint
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
pull-requests: read
|
||||
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- 'master'
|
||||
- 'release-**'
|
||||
- "master"
|
||||
- "release-**"
|
||||
pull_request:
|
||||
|
||||
concurrency:
|
||||
@@ -42,9 +46,7 @@ jobs:
|
||||
- uses: actions/checkout@v6
|
||||
|
||||
- name: Setup frontend env
|
||||
if: |
|
||||
needs.files-changed.outputs.yaml == 'true' ||
|
||||
needs.files-changed.outputs.openapi == 'true'
|
||||
if: needs.files-changed.outputs.yaml == 'true'
|
||||
uses: ./.github/actions/prepare-frontend
|
||||
|
||||
# Always required by whitespace linter, also used by OpenAPI spec validation
|
||||
@@ -88,17 +90,54 @@ jobs:
|
||||
if: needs.files-changed.outputs.workflows == 'true'
|
||||
run: .github/scripts/check-job-timeouts.sh ${{ needs.files-changed.outputs.workflows_files }}
|
||||
|
||||
# Generate and Lint OpenAPI spec
|
||||
- name: Generate OpenAPI spec
|
||||
if: needs.files-changed.outputs.openapi == 'true'
|
||||
run: bun run generate-openapi
|
||||
validate-openapi:
|
||||
needs: files-changed
|
||||
if: needs.files-changed.outputs.openapi == 'true'
|
||||
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
|
||||
timeout-minutes: 20
|
||||
steps:
|
||||
# No credentials: this job runs package.json scripts defined by the PR.
|
||||
- uses: actions/checkout@v6
|
||||
with:
|
||||
persist-credentials: false
|
||||
- name: Prepare front-end environment
|
||||
uses: ./.github/actions/prepare-frontend
|
||||
- name: Prepare back-end environment
|
||||
uses: ./.github/actions/prepare-backend
|
||||
- name: Generate OpenAPI artifacts from local source
|
||||
run: bun run types:ensure --force-local
|
||||
- name: Lint OpenAPI spec
|
||||
run: bun run openapi:lint
|
||||
- name: Compile CLJS for type checking
|
||||
run: bun run build-pure:cljs
|
||||
- name: Check frontend types
|
||||
run: bun run type-check-pure
|
||||
|
||||
# Parallel steps run asynchronously, and explicitly waiting for a parallel step means "if" statements are ignored.
|
||||
# Move OpenAPI validation outside the parallel block so that spec generation happens asynchronously without requiring
|
||||
# an explicit "wait".
|
||||
- name: Validate OpenAPI spec
|
||||
if: needs.files-changed.outputs.openapi == 'true'
|
||||
run: bun run lint-openapi
|
||||
# Always reports, even when OpenAPI generation and validation are skipped.
|
||||
openapi-result:
|
||||
needs: [files-changed, validate-openapi]
|
||||
if: always() && !cancelled()
|
||||
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
|
||||
timeout-minutes: 5
|
||||
steps:
|
||||
- name: Check OpenAPI job status
|
||||
env:
|
||||
needs: ${{ toJson(needs) }}
|
||||
uses: actions/github-script@v9
|
||||
with:
|
||||
script: | # js
|
||||
const needs = JSON.parse(process.env.needs);
|
||||
const jobs = Object.entries(needs).map(([name, { result }]) => ({ name, result }));
|
||||
|
||||
if (jobs.every(({ result }) => result === 'skipped' || result === 'success')) {
|
||||
console.log('OpenAPI generation and validation succeeded or were skipped');
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
console.log('OpenAPI generation or validation failed');
|
||||
jobs.filter(({ result }) => result !== 'success')
|
||||
.forEach(({ name, result }) => console.log(`${name} - ${result}`));
|
||||
process.exit(1);
|
||||
|
||||
i18n:
|
||||
needs: files-changed
|
||||
|
||||
42
.github/workflows/openapi-check.yml
vendored
42
.github/workflows/openapi-check.yml
vendored
@@ -1,42 +0,0 @@
|
||||
name: OpenAPI Check
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
types: [opened, synchronize, reopened, labeled, unlabeled]
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.event.pull_request.number }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
check-openapi-specs:
|
||||
runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}
|
||||
timeout-minutes: 5
|
||||
if: contains(github.event.pull_request.labels.*.name, 'openapi-self-healing')
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
with:
|
||||
# we're intentionally using METABASE_AUTOMATION_USER_TOKEN to trigger the whole CI pipeline again
|
||||
# after the self-healing commit is created
|
||||
# if we used GITHUB_TOKEN, the self-healing commit would not trigger the CI pipeline again
|
||||
token: ${{ secrets.METABASE_AUTOMATION_USER_TOKEN }}
|
||||
# fetch full git history to allow fetching the latest base branch commit and comparing changes
|
||||
fetch-depth: 0
|
||||
- name: Prepare Node + Bun environment
|
||||
uses: ./.github/actions/prepare-node-bun
|
||||
- name: Prepare back-end environment
|
||||
uses: ./.github/actions/prepare-backend
|
||||
- name: Generate version.properties
|
||||
run: ./bin/build.sh :steps [:version]
|
||||
- name: Get latest base branch commit
|
||||
run: |
|
||||
git fetch origin ${{ github.event.pull_request.base.ref }}
|
||||
echo "LATEST_BASE_SHA=$(git rev-parse origin/${{ github.event.pull_request.base.ref }})" >> "$GITHUB_ENV"
|
||||
- name: Check OpenAPI specs are up to date
|
||||
env:
|
||||
AUTO_COMMIT: "true"
|
||||
PR_BRANCH: ${{ github.event.pull_request.head.ref }}
|
||||
BASE_SHA: ${{ env.LATEST_BASE_SHA }}
|
||||
HEAD_SHA: ${{ github.event.pull_request.head.sha }}
|
||||
GITHUB_SHA: ${{ github.sha }}
|
||||
run: ./bin/openapi-check.sh
|
||||
24
.github/workflows/resolve-backport-conflicts.yml
vendored
24
.github/workflows/resolve-backport-conflicts.yml
vendored
@@ -116,10 +116,6 @@ jobs:
|
||||
if: steps.backport.outputs.has_conflicts == 'true'
|
||||
uses: ./.github/actions/prepare-frontend
|
||||
|
||||
- name: Build CLJS for type checking
|
||||
if: steps.backport.outputs.has_conflicts == 'true'
|
||||
run: ${{ steps.pkg.outputs.mgr }} run build-pure:cljs
|
||||
|
||||
- name: Generate GitHub App Token for Claude
|
||||
if: steps.backport.outputs.has_conflicts == 'true'
|
||||
uses: actions/create-github-app-token@v1
|
||||
@@ -139,7 +135,7 @@ jobs:
|
||||
anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY_BACKPORT_FIX }}
|
||||
model: "claude-sonnet-4-20250514"
|
||||
max_turns: "${{ github.event.inputs.max_turns || '75' }}"
|
||||
allowed_tools: "Read,Edit,Write,Bash(git:*),Bash(grep:*),Bash(find:*),Bash(ls:*),Bash(cat:*),Bash(clojure:*),Bash(${{ steps.pkg.outputs.mgr }} run type-check-pure:*),Bash(${{ steps.pkg.outputs.mgr }} run lint-eslint:*),Bash(./bin/mage cljfmt:*),Glob,Grep"
|
||||
allowed_tools: "Read,Edit,Write,Bash(git:*),Bash(grep:*),Bash(find:*),Bash(ls:*),Bash(cat:*),Bash(clojure:*),Bash(${{ steps.pkg.outputs.mgr }} run build-pure:cljs:*),Bash(${{ steps.pkg.outputs.mgr }} run type-check-pure:*),Bash(${{ steps.pkg.outputs.mgr }} run lint-eslint:*),Bash(./bin/mage cljfmt:*),Glob,Grep"
|
||||
prompt: |
|
||||
You are resolving git cherry-pick merge conflicts for a Metabase backport PR.
|
||||
|
||||
@@ -181,22 +177,26 @@ jobs:
|
||||
${{ steps.pkg.outputs.mgr }} run lint-eslint --fix
|
||||
```
|
||||
7. Validate resolved files before committing:
|
||||
- After all conflict markers are removed, build CLJS:
|
||||
```
|
||||
${{ steps.pkg.outputs.mgr }} run build-pure:cljs
|
||||
```
|
||||
- Run the type checker after the CLJS build regardless of the conflicted file extensions. This is required even for Clojure-only conflicts because backend API changes can affect generated TypeScript declarations:
|
||||
```
|
||||
${{ steps.pkg.outputs.mgr }} run type-check-pure
|
||||
```
|
||||
- **Clojure/ClojureScript** (.clj, .cljc, .cljs): run clj-kondo to verify syntax:
|
||||
```
|
||||
clojure -M:kondo --lint <file1> <file2> ...
|
||||
```
|
||||
- **TypeScript/JavaScript** (.ts, .tsx, .js, .jsx): run the type checker:
|
||||
```
|
||||
${{ steps.pkg.outputs.mgr }} run type-check-pure
|
||||
```
|
||||
If either tool reports errors or warnings, fix them before proceeding. CI treats kondo warnings as errors.
|
||||
If any tool reports errors or warnings, fix them before proceeding. CI treats kondo warnings as errors.
|
||||
Repeat steps 6-7 until all errors are resolved.
|
||||
7. Once all files are clean, stage and complete the cherry-pick:
|
||||
8. Once all files are clean, stage and complete the cherry-pick:
|
||||
```
|
||||
git add -A
|
||||
git cherry-pick --continue
|
||||
```
|
||||
8. Push the result:
|
||||
9. Push the result:
|
||||
```
|
||||
git push --force
|
||||
```
|
||||
|
||||
3
.gitignore
vendored
3
.gitignore
vendored
@@ -210,3 +210,6 @@ __pycache__/
|
||||
# dependency-cruiser output
|
||||
dependency-graph.json
|
||||
.playwright-cli/
|
||||
|
||||
# generated build artifacts (OpenAPI spec, etc.)
|
||||
/.tmp/
|
||||
|
||||
@@ -1,94 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
OPENAPI_SPEC="resources/openapi/openapi.json"
|
||||
AUTO_COMMIT="${AUTO_COMMIT:-false}"
|
||||
|
||||
# Check that $OPENAPI_SPEC matches the current Malli schema definitions
|
||||
|
||||
if [[ -n "${GITHUB_SHA:-}" ]]; then
|
||||
echo "Running OpenAPI check on commit: $GITHUB_SHA"
|
||||
fi
|
||||
|
||||
# Generate the current OpenAPI spec from Malli schemas using write-openapi-spec-to-file!
|
||||
# This creates $OPENAPI_SPEC directly
|
||||
# Using the generate-openapi-spec command from metabase.cmd.core
|
||||
bun run generate-openapi
|
||||
ret=$?
|
||||
|
||||
if [[ $ret -ne 0 ]]; then
|
||||
exit $ret
|
||||
fi
|
||||
|
||||
if [[ ! -f "$OPENAPI_SPEC" ]]; then
|
||||
echo "Error: $OPENAPI_SPEC does not exist!"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Check if there are any differences between generated and committed versions
|
||||
if ! git diff --exit-code "$OPENAPI_SPEC" >/dev/null 2>&1; then
|
||||
echo "OpenAPI schema changes detected:"
|
||||
git diff "$OPENAPI_SPEC"
|
||||
|
||||
if [[ "$AUTO_COMMIT" == "true" ]]; then
|
||||
echo ""
|
||||
echo "Auto-committing changes..."
|
||||
|
||||
# Require PR_BRANCH to be set for auto-commit
|
||||
if [[ -z "${PR_BRANCH:-}" ]]; then
|
||||
echo "Error: PR_BRANCH environment variable must be set for auto-commit"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Stash the generated spec from the merge commit
|
||||
echo "Stashing generated OpenAPI spec from merge commit..."
|
||||
cp "$OPENAPI_SPEC" "${OPENAPI_SPEC}.tmp"
|
||||
|
||||
# Checkout the actual PR branch (in case we're on a merge commit)
|
||||
echo "Checking out branch: $PR_BRANCH"
|
||||
git checkout -f "$PR_BRANCH"
|
||||
|
||||
# Restore the generated spec from the merge commit
|
||||
mv -f "${OPENAPI_SPEC}.tmp" "$OPENAPI_SPEC"
|
||||
|
||||
git config user.name "github-actions[bot]"
|
||||
git config user.email "github-actions[bot]@users.noreply.github.com"
|
||||
git add "$OPENAPI_SPEC"
|
||||
|
||||
# Build commit message with SHAs if available
|
||||
COMMIT_MSG="Update OpenAPI schema"
|
||||
if [[ -n "${BASE_SHA:-}" ]] && [[ -n "${HEAD_SHA:-}" ]]; then
|
||||
BASE_SHORT="${BASE_SHA:0:7}"
|
||||
HEAD_SHORT="${HEAD_SHA:0:7}"
|
||||
COMMIT_MSG="Update OpenAPI schema (${BASE_SHORT}...${HEAD_SHORT})"
|
||||
fi
|
||||
|
||||
git commit -m "$COMMIT_MSG"
|
||||
if ! git push origin "$PR_BRANCH"; then
|
||||
echo "Error: Failed to push OpenAPI schema update to $PR_BRANCH."
|
||||
exit 1
|
||||
fi
|
||||
echo "OpenAPI schema updated and pushed to $PR_BRANCH."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo "Error: OpenAPI specs are out of date!"
|
||||
echo "The schema in $OPENAPI_SPEC does not match the current Malli schema definitions."
|
||||
echo ""
|
||||
echo "This can happen when:"
|
||||
echo " 1. You added/modified API endpoints but didn't regenerate the OpenAPI spec"
|
||||
echo " 2. Your branch is behind master, which has new API endpoints"
|
||||
echo ""
|
||||
echo "To fix this:"
|
||||
echo " 1. Pull the latest changes from master: git pull origin master"
|
||||
echo " 2. Run 'bun run generate-openapi' to update $OPENAPI_SPEC"
|
||||
echo " 3. Commit the updated openapi.json file"
|
||||
echo ""
|
||||
echo "Differences:"
|
||||
git diff "$OPENAPI_SPEC" || true
|
||||
exit 1
|
||||
fi
|
||||
echo "OpenAPI specs are up to date."
|
||||
|
||||
exit 0
|
||||
@@ -70,7 +70,7 @@ Options:
|
||||
|
||||
## `generate-openapi-spec`
|
||||
|
||||
Generate OpenAPI specification file from Malli schema definitions. This is written to `resources/openapi/openapi.json`.
|
||||
Generate OpenAPI specification file from Malli schema definitions. This is written to `.tmp/openapi/openapi.json`.
|
||||
|
||||
## `help command-name | help`
|
||||
|
||||
|
||||
@@ -71,7 +71,7 @@ const REPO_ROOT = path.resolve(
|
||||
// drift/capture-bug signal) — stored shapes stay table-independent so
|
||||
// consumers can match them against their own checkout's spec version.
|
||||
// Missing file just means no drift log.
|
||||
const OPENAPI_FILE = path.join(REPO_ROOT, "resources/openapi/openapi.json");
|
||||
const OPENAPI_FILE = path.join(REPO_ROOT, ".tmp/openapi/openapi.json");
|
||||
|
||||
const PER_SPEC_DIR = path.join(REPO_ROOT, "e2e/coverage-manifest-raw");
|
||||
const OUTPUT_FILE = path.join(
|
||||
|
||||
@@ -14,10 +14,9 @@
|
||||
* from last night's SHA while a consumer compares against its own checkout's
|
||||
* endpoints, so baking one table's spelling in would break on any route
|
||||
* whose shape changed in between. A consumer matches structurally with
|
||||
* matchRoute() below against whichever spec version fits its checkout
|
||||
* (resources/openapi/openapi.json, generated from the backend's defendpoint
|
||||
* definitions by `bun run generate-openapi`; the nightly also ships its
|
||||
* freshly generated copy alongside the manifests).
|
||||
* matchRoute() below against whichever spec version fits its checkout. The
|
||||
* nightly generates one from the backend's defendpoint definitions with
|
||||
* `bun run openapi:generate` and ships it alongside the manifests.
|
||||
*
|
||||
* The nightly build still loads the table for one thing: logging captured
|
||||
* API routes that correspond to no endpoint definition — an endpoint defined
|
||||
|
||||
@@ -138,6 +138,7 @@ const configs = [
|
||||
ignores: [
|
||||
"frontend/src/cljs/**",
|
||||
"frontend/src/cljs_release/**",
|
||||
"frontend/src/metabase-types/openapi/**",
|
||||
"**/*.d.ts",
|
||||
"e2e/support/cypress_sample_database.js",
|
||||
"e2e/support/cypress_sample_instance_data.js",
|
||||
@@ -1117,6 +1118,32 @@ const configs = [
|
||||
message: "Please use only type-only imports from 'custom-viz'.",
|
||||
},
|
||||
],
|
||||
patterns: [
|
||||
{
|
||||
regex: "^metabase-types/openapi",
|
||||
allowTypeImports: true,
|
||||
message:
|
||||
"Only `import type` is allowed from metabase-types/openapi: it is generated as declaration files, so a value import resolves to nothing at runtime.",
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
{
|
||||
files: ["enterprise/frontend/**/*.ts", "enterprise/frontend/**/*.tsx"],
|
||||
rules: {
|
||||
"@typescript-eslint/no-restricted-imports": [
|
||||
"error",
|
||||
{
|
||||
patterns: [
|
||||
{
|
||||
regex: "^metabase-types/openapi",
|
||||
allowTypeImports: true,
|
||||
message:
|
||||
"Only `import type` is allowed from metabase-types/openapi: it is generated as declaration files, so a value import resolves to nothing at runtime.",
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
},
|
||||
|
||||
95
frontend/build/openapi/ARCHITECTURE.md
Normal file
95
frontend/build/openapi/ARCHITECTURE.md
Normal file
@@ -0,0 +1,95 @@
|
||||
# OpenAPI generation internals
|
||||
|
||||
`types:ensure` uses three supporting modules to coordinate generation, detect
|
||||
changes, and decide whether the existing declarations are current:
|
||||
|
||||
```text
|
||||
types:ensure
|
||||
├─ generation-lock coordinates concurrent runs
|
||||
├─ source-fingerprint describes the current inputs
|
||||
└─ generation-state records what produced the current output
|
||||
```
|
||||
|
||||
The command acquires the generation lock, fingerprints the current inputs, and
|
||||
compares them with the saved generation state. It exits when everything
|
||||
matches. Otherwise, it acquires a current spec, generates and publishes the
|
||||
declarations, and writes new state.
|
||||
|
||||
## `source-fingerprint.ts` detects relevant input changes
|
||||
|
||||
The source fingerprint covers the backend files that can affect the OpenAPI
|
||||
spec:
|
||||
|
||||
- `src/**`
|
||||
- `enterprise/backend/src/**`
|
||||
- relevant `deps.edn` files
|
||||
- selected resource files
|
||||
- the Enterprise generation command and settings
|
||||
|
||||
A separate generator fingerprint covers `openapi-ts.config.ts`, `package.json`,
|
||||
and `bun.lock`.
|
||||
|
||||
Fingerprints use sorted paths, file types, and file contents rather than
|
||||
modification times. This makes the result deterministic and detects changes to
|
||||
both the source and its generation environment.
|
||||
|
||||
**Importance:** high for incremental generation. Without fingerprints,
|
||||
`types:ensure` would need to regenerate every time or risk reusing declarations
|
||||
from older source. Always regenerating would remain correct, but would make the
|
||||
command much more expensive.
|
||||
|
||||
## `generation-state.ts` identifies current output
|
||||
|
||||
`.tmp/openapi/generation.json` records:
|
||||
|
||||
- `sourceDigest`: the backend source that produced the spec
|
||||
- `generatorDigest`: the configuration and dependencies used to generate types
|
||||
- `specHash`: the exact OpenAPI document used as input
|
||||
- `outputsHash`: the resulting declaration files
|
||||
- `version`: the state format version
|
||||
|
||||
State from a running backend omits `sourceDigest` because the source loaded by
|
||||
another process can't be observed. Those declarations aren't considered fresh
|
||||
for the checked-out source.
|
||||
|
||||
The module validates state before trusting it, treats missing or malformed
|
||||
state as stale, checks the generated files for changes, and writes new state
|
||||
with an atomic rename. It also distinguishes a complete Enterprise spec from
|
||||
an OSS spec.
|
||||
|
||||
**Importance:** high. The state connects current inputs to current output.
|
||||
Without it, `types:ensure` couldn't reliably distinguish current declarations
|
||||
from missing, modified, incomplete, backend-origin, or outdated declarations.
|
||||
|
||||
## `generation-lock.ts` avoids duplicate work
|
||||
|
||||
Postinstall, type-checking, and manual commands can invoke `types:ensure` at the
|
||||
same time. The generation lock prevents those processes from unnecessarily
|
||||
running Clojure and hey-api in parallel.
|
||||
|
||||
`proper-lockfile` provides atomic acquisition, heartbeat updates, stale-lock
|
||||
recovery, and release. It represents the lock as a directory. The wrapper in
|
||||
`generation-lock.ts` adds the Metabase policy: regular commands wait for a
|
||||
bounded time before continuing without the lock, while postinstall workers
|
||||
never wait. A compromised lock doesn't interrupt generation.
|
||||
|
||||
Manually removing and immediately reacquiring the lock can confuse the previous
|
||||
holder because `proper-lockfile` doesn't attach ownership tokens. That can
|
||||
allow redundant generation, but it can't corrupt published output.
|
||||
|
||||
**Importance:** medium operationally, but low for correctness. Removing the
|
||||
lock would allow redundant, resource-intensive generation. Correctness instead
|
||||
comes from staging output, atomically replacing changing files, and checking
|
||||
output against generation state.
|
||||
|
||||
## Relative importance
|
||||
|
||||
| Module | Responsibility | Importance |
|
||||
| ----------------------- | --------------------------------------------- | ----------------------------------------- |
|
||||
| `generation-state.ts` | Decide whether existing output can be trusted | High |
|
||||
| `source-fingerprint.ts` | Detect relevant changes without regenerating | High for incremental generation |
|
||||
| `generation-lock.ts` | Avoid duplicate concurrent work | Medium operationally; low for correctness |
|
||||
|
||||
The final correctness boundary remains in `ensure-types.ts`: generate from a
|
||||
stable spec snapshot, stage complete output, publish it, and only then record
|
||||
the resulting state.
|
||||
77
frontend/build/openapi/README.md
Normal file
77
frontend/build/openapi/README.md
Normal file
@@ -0,0 +1,77 @@
|
||||
# Generate OpenAPI types
|
||||
|
||||
Metabase generates Enterprise API types from backend source. The OpenAPI spec
|
||||
and TypeScript declarations are derived files and aren't committed. See
|
||||
[OpenAPI generation internals](./ARCHITECTURE.md) for the supporting components
|
||||
and their responsibilities.
|
||||
|
||||
## Use the generation commands
|
||||
|
||||
Run the standard freshness check:
|
||||
|
||||
```sh
|
||||
bun run types:ensure
|
||||
```
|
||||
|
||||
The command exits quickly when the existing types are current. Otherwise, it
|
||||
generates `.tmp/openapi/openapi.json` and writes declarations to
|
||||
`frontend/src/metabase-types/openapi/`.
|
||||
|
||||
Use the other commands when you need a specific result:
|
||||
|
||||
| Command | Result |
|
||||
| ------------------------------------ | ------------------------------------------------------------------ |
|
||||
| `bun run types:ensure --force-local` | Regenerates the spec and types from the checked-out Clojure source |
|
||||
| `bun run openapi:generate` | Generates only `.tmp/openapi/openapi.json` |
|
||||
| `bun run openapi:lint` | Validates the generated spec with Redocly |
|
||||
|
||||
`bun install` checks freshness and starts generation in the background when
|
||||
needed. Background output goes to `.tmp/openapi/types-ensure.log`.
|
||||
|
||||
## How freshness works
|
||||
|
||||
`.tmp/openapi/generation.json` records hashes for the backend source, type
|
||||
generator, OpenAPI spec, and generated declarations. `types:ensure` compares
|
||||
each hash with the current files — a missing or modified spec file counts as
|
||||
stale — before deciding whether to regenerate.
|
||||
|
||||
For speed, a stale run may fetch the spec from the running Enterprise backend
|
||||
at `localhost:${MB_JETTY_PORT:-3000}`. Backend-generated types aren't marked as
|
||||
source-fresh because the process may have loaded different source. When the
|
||||
checked-out backend source changes, `types:ensure` always generates locally.
|
||||
CI also uses `--force-local` before linting and type-checking.
|
||||
|
||||
Local generation compares source hashes before and after running Clojure and
|
||||
retries when it detects an edit. Run generation while source files are stable;
|
||||
an exact edit and revert during generation is outside the freshness guarantee.
|
||||
The spec read back after Clojure exits isn't re-verified against the generator
|
||||
itself, so don't run generation while another process writes the spec (for
|
||||
example a backend with `MB_ENABLE_OPENAPI_AUTO_REGEN` enabled).
|
||||
|
||||
## Recover from interrupted generation
|
||||
|
||||
Postinstall, type-checking, and manual commands can request generation at the
|
||||
same time. `.tmp/openapi/types-ensure.lock` prevents those processes from
|
||||
repeating the expensive Clojure and type-generation work.
|
||||
|
||||
The lock is only an optimization. Declarations are staged before publication,
|
||||
the changing declaration file is replaced atomically, and generation state
|
||||
records the staged hash. Correctness doesn't depend on holding the lock.
|
||||
|
||||
`proper-lockfile` creates the lock as a directory and refreshes its timestamp
|
||||
every 5 seconds. A lock with no refresh for 30 seconds is removed automatically.
|
||||
Regular commands wait up to 60 seconds, then continue without the lock.
|
||||
Postinstall workers don't wait when another generator is already running.
|
||||
|
||||
If a process appears stuck, remove the lock directory:
|
||||
|
||||
```sh
|
||||
rm -rf .tmp/openapi/types-ensure.lock
|
||||
```
|
||||
|
||||
To reset all generated state, remove the temporary directory:
|
||||
|
||||
```sh
|
||||
rm -rf .tmp/openapi
|
||||
bun run types:ensure
|
||||
```
|
||||
546
frontend/build/openapi/ensure-types.ts
Normal file
546
frontend/build/openapi/ensure-types.ts
Normal file
@@ -0,0 +1,546 @@
|
||||
#!/usr/bin/env bun
|
||||
/**
|
||||
* Keeps generated Enterprise API types in sync with backend source.
|
||||
*
|
||||
* Fresh runs exit without work. A changed local source regenerates through
|
||||
* Clojure; other stale runs may fetch a running Enterprise backend. Generation
|
||||
* stages its output and replaces the declaration atomically, so the lock only
|
||||
* avoids duplicate work. Postinstall uses a best-effort background worker.
|
||||
*/
|
||||
import { spawn } from "node:child_process";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import {
|
||||
closeSync,
|
||||
constants,
|
||||
fchmodSync,
|
||||
fstatSync,
|
||||
mkdirSync,
|
||||
openSync,
|
||||
readFileSync,
|
||||
renameSync,
|
||||
rmSync,
|
||||
writeFileSync,
|
||||
} from "node:fs";
|
||||
import { dirname, join, resolve } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
import { withGenerationLock } from "./generation-lock";
|
||||
import {
|
||||
GENERATED_OUTPUT_NAMES,
|
||||
GENERATED_TYPES_DIRECTORY,
|
||||
GENERATION_STATE_VERSION,
|
||||
type GenerationState,
|
||||
createContentHash,
|
||||
createOutputsHash,
|
||||
getOpenApiEdition,
|
||||
readGenerationState,
|
||||
writeGenerationStateAtomically,
|
||||
} from "./generation-state";
|
||||
import {
|
||||
createSourceFingerprint,
|
||||
createTypeGeneratorFingerprint,
|
||||
} from "./source-fingerprint";
|
||||
|
||||
const SPEC_PATH = ".tmp/openapi/openapi.json";
|
||||
const GENERATION_LOCK_PATH = ".tmp/openapi/types-ensure.lock";
|
||||
const MAX_LOCK_WAIT_MS = 60_000;
|
||||
const POSTINSTALL_LOG_PATH = ".tmp/openapi/types-ensure.log";
|
||||
const OPENAPI_INPUT_ENV = "METABASE_OPENAPI_INPUT";
|
||||
const OPENAPI_OUTPUT_ENV = "METABASE_OPENAPI_OUTPUT";
|
||||
const BACKEND_TIMEOUT_MS = 2000;
|
||||
const MAX_CAPTURED_OUTPUT_BYTES = 10 * 1024 * 1024;
|
||||
const GENERATED_INDEX_CONTENT = 'export type * from "./types.gen.d";\n';
|
||||
|
||||
const CLOJURE_GENERATE_COMMAND = [
|
||||
"clojure",
|
||||
"-M:run:ee",
|
||||
"generate-openapi-spec",
|
||||
];
|
||||
const OPENAPI_TS_COMMAND = [
|
||||
"node_modules/.bin/openapi-ts",
|
||||
"-f",
|
||||
"./frontend/build/openapi/openapi-ts.config.ts",
|
||||
];
|
||||
interface EnsureOptions {
|
||||
forceLocal: boolean;
|
||||
postinstall: boolean;
|
||||
/** Best-effort background mode used by postinstall. */
|
||||
postinstallWorker: boolean;
|
||||
}
|
||||
|
||||
interface AcquiredSpec {
|
||||
contents: string;
|
||||
hash: string;
|
||||
/** Omitted when the spec came from a running backend. */
|
||||
sourceDigest?: string;
|
||||
}
|
||||
|
||||
function log(message: string) {
|
||||
// eslint-disable-next-line no-console
|
||||
console.log(`[types:ensure] ${message}`);
|
||||
}
|
||||
|
||||
function parseOptions(arguments_: string[]): EnsureOptions {
|
||||
const knownArguments = new Set([
|
||||
"--force-local",
|
||||
"--postinstall",
|
||||
"--postinstall-worker",
|
||||
]);
|
||||
const unknownArgument = arguments_.find(
|
||||
(argument) => !knownArguments.has(argument),
|
||||
);
|
||||
if (unknownArgument !== undefined) {
|
||||
throw new Error(`unknown option: ${unknownArgument}`);
|
||||
}
|
||||
return {
|
||||
forceLocal: arguments_.includes("--force-local"),
|
||||
postinstall: arguments_.includes("--postinstall"),
|
||||
postinstallWorker: arguments_.includes("--postinstall-worker"),
|
||||
};
|
||||
}
|
||||
|
||||
interface RunCommandOptions {
|
||||
quietUnlessError?: boolean;
|
||||
postinstallWorker: boolean;
|
||||
env?: NodeJS.ProcessEnv;
|
||||
}
|
||||
|
||||
/** Keep child commands asynchronous so the lock heartbeat can run. */
|
||||
function runCommand(
|
||||
command: string[],
|
||||
{ quietUnlessError = false, postinstallWorker, env }: RunCommandOptions,
|
||||
): Promise<number> {
|
||||
const [executable, ...commandArguments] = command;
|
||||
if (executable === undefined) {
|
||||
return Promise.resolve(1);
|
||||
}
|
||||
return new Promise((resolvePromise) => {
|
||||
const output = postinstallWorker ? process.stdout.fd : "inherit";
|
||||
const child = spawn(executable, commandArguments, {
|
||||
env: { ...process.env, ...env },
|
||||
stdio: quietUnlessError
|
||||
? ["inherit", "pipe", "pipe"]
|
||||
: ["inherit", output, output],
|
||||
});
|
||||
|
||||
let capturedBytes = 0;
|
||||
let captured = "";
|
||||
const capture = (chunk: Buffer) => {
|
||||
if (capturedBytes < MAX_CAPTURED_OUTPUT_BYTES) {
|
||||
captured += String(chunk);
|
||||
capturedBytes += chunk.byteLength;
|
||||
}
|
||||
};
|
||||
child.stdout?.on("data", capture);
|
||||
child.stderr?.on("data", capture);
|
||||
|
||||
const finish = (status: number) => {
|
||||
if (quietUnlessError && status !== 0) {
|
||||
const errorOutput = postinstallWorker ? process.stdout : process.stderr;
|
||||
errorOutput.write(captured);
|
||||
}
|
||||
resolvePromise(status);
|
||||
};
|
||||
child.once("error", (error) => {
|
||||
const errorOutput = postinstallWorker ? process.stdout : process.stderr;
|
||||
errorOutput.write(`${error.message}\n`);
|
||||
finish(1);
|
||||
});
|
||||
child.once("close", (status) => finish(status ?? 1));
|
||||
});
|
||||
}
|
||||
|
||||
function backendUrl(): string | undefined {
|
||||
const configuredPort = process.env.MB_JETTY_PORT;
|
||||
if (configuredPort === undefined) {
|
||||
return "http://localhost:3000/api/docs/openapi.json";
|
||||
}
|
||||
const port = Number(configuredPort);
|
||||
if (!/^[1-9]\d{0,4}$/.test(configuredPort) || port > 65_535) {
|
||||
return undefined;
|
||||
}
|
||||
return `http://localhost:${port}/api/docs/openapi.json`;
|
||||
}
|
||||
|
||||
function readCompleteSpec(): string | undefined {
|
||||
try {
|
||||
const contents = readFileSync(SPEC_PATH, "utf8");
|
||||
return getOpenApiEdition(contents) === "ee" ? contents : undefined;
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
}
|
||||
|
||||
function writePrivateFile(path: string, contents: string): void {
|
||||
let fileDescriptor: number | undefined;
|
||||
mkdirSync(dirname(path), { recursive: true });
|
||||
try {
|
||||
fileDescriptor = openSync(path, "wx", 0o600);
|
||||
writeFileSync(fileDescriptor, contents);
|
||||
} finally {
|
||||
if (fileDescriptor !== undefined) {
|
||||
closeSync(fileDescriptor);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function publishSpecAtomically(contents: string): void {
|
||||
const temporaryPath = `${SPEC_PATH}.${process.pid}.${randomUUID()}.tmp`;
|
||||
try {
|
||||
writePrivateFile(temporaryPath, contents);
|
||||
renameSync(temporaryPath, SPEC_PATH);
|
||||
} finally {
|
||||
rmSync(temporaryPath, { force: true });
|
||||
}
|
||||
}
|
||||
|
||||
async function fetchSpecFromBackend(
|
||||
url: string,
|
||||
): Promise<{ contents: string; edition: "oss" | "ee" } | undefined> {
|
||||
try {
|
||||
const response = await fetch(url, {
|
||||
signal: AbortSignal.timeout(BACKEND_TIMEOUT_MS),
|
||||
});
|
||||
if (!response.ok) {
|
||||
return undefined;
|
||||
}
|
||||
const contents = await response.text();
|
||||
const edition = getOpenApiEdition(contents);
|
||||
return edition === undefined ? undefined : { contents, edition };
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
}
|
||||
|
||||
type StalenessReason =
|
||||
| "no generation state"
|
||||
| "previous types came from a running backend"
|
||||
| "backend source changed"
|
||||
| "type generator inputs changed"
|
||||
| "generated spec changed on disk"
|
||||
| "generated outputs changed on disk";
|
||||
|
||||
/** Returns why regeneration is needed, or nothing when types are fresh. */
|
||||
function stalenessReason(
|
||||
state: GenerationState | undefined,
|
||||
sourceDigest: string,
|
||||
generatorDigest: string,
|
||||
): StalenessReason | undefined {
|
||||
if (state === undefined) {
|
||||
return "no generation state";
|
||||
}
|
||||
if (state.sourceDigest === undefined) {
|
||||
return "previous types came from a running backend";
|
||||
}
|
||||
if (state.sourceDigest !== sourceDigest) {
|
||||
return "backend source changed";
|
||||
}
|
||||
if (state.generatorDigest !== generatorDigest) {
|
||||
return "type generator inputs changed";
|
||||
}
|
||||
if (readSpecFileHash() !== state.specHash) {
|
||||
return "generated spec changed on disk";
|
||||
}
|
||||
if (createOutputsHash() !== state.outputsHash) {
|
||||
return "generated outputs changed on disk";
|
||||
}
|
||||
return undefined;
|
||||
}
|
||||
|
||||
/** Hash of the published spec file, or undefined when it is missing/unreadable. */
|
||||
function readSpecFileHash(): string | undefined {
|
||||
try {
|
||||
return createContentHash(readFileSync(SPEC_PATH, "utf8"));
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
}
|
||||
|
||||
function freshBeforePostinstall(): boolean {
|
||||
try {
|
||||
return (
|
||||
stalenessReason(
|
||||
readGenerationState(),
|
||||
createSourceFingerprint(),
|
||||
createTypeGeneratorFingerprint(),
|
||||
) === undefined
|
||||
);
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
function startPostinstallWorker(): number {
|
||||
let logFileDescriptor: number | undefined;
|
||||
try {
|
||||
mkdirSync(dirname(POSTINSTALL_LOG_PATH), { recursive: true });
|
||||
logFileDescriptor = openSync(
|
||||
POSTINSTALL_LOG_PATH,
|
||||
constants.O_APPEND |
|
||||
constants.O_CREAT |
|
||||
constants.O_WRONLY |
|
||||
constants.O_NOFOLLOW,
|
||||
0o600,
|
||||
);
|
||||
if (!fstatSync(logFileDescriptor).isFile()) {
|
||||
throw new Error("postinstall log is not a regular file");
|
||||
}
|
||||
fchmodSync(logFileDescriptor, 0o600);
|
||||
const scriptPath = process.argv[1];
|
||||
if (scriptPath === undefined) {
|
||||
throw new Error("could not determine types:ensure script path");
|
||||
}
|
||||
const worker = spawn(
|
||||
process.execPath,
|
||||
[scriptPath, "--postinstall-worker"],
|
||||
{
|
||||
detached: true,
|
||||
stdio: ["ignore", logFileDescriptor, "inherit"],
|
||||
},
|
||||
);
|
||||
worker.unref();
|
||||
log(
|
||||
`generating API types in the background (logs: ${POSTINSTALL_LOG_PATH})`,
|
||||
);
|
||||
} catch {
|
||||
log(
|
||||
"⚠ could not start API type generation — run `bun run types:ensure` to see the failure",
|
||||
);
|
||||
} finally {
|
||||
if (logFileDescriptor !== undefined) {
|
||||
closeSync(logFileDescriptor);
|
||||
}
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
async function acquireFromLocalSource(
|
||||
options: EnsureOptions,
|
||||
): Promise<{ status: number; spec?: AcquiredSpec }> {
|
||||
for (let attempt = 0; attempt < 2; attempt += 1) {
|
||||
const sourceDigest = createSourceFingerprint();
|
||||
log("generating the complete OpenAPI document from local source");
|
||||
const status = await runCommand(CLOJURE_GENERATE_COMMAND, {
|
||||
quietUnlessError: true,
|
||||
postinstallWorker: options.postinstallWorker,
|
||||
});
|
||||
if (status !== 0) {
|
||||
return { status };
|
||||
}
|
||||
|
||||
const contents = readCompleteSpec();
|
||||
if (contents === undefined) {
|
||||
log(
|
||||
"error: local source generation did not produce a complete EE OpenAPI document",
|
||||
);
|
||||
return { status: 1 };
|
||||
}
|
||||
const currentSourceDigest = createSourceFingerprint();
|
||||
if (sourceDigest === currentSourceDigest) {
|
||||
return {
|
||||
status: 0,
|
||||
spec: {
|
||||
contents,
|
||||
hash: createContentHash(contents),
|
||||
sourceDigest: currentSourceDigest,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
if (attempt === 0) {
|
||||
log("backend source changed during OpenAPI generation — retrying once");
|
||||
continue;
|
||||
}
|
||||
log(
|
||||
"error: backend source changed during both OpenAPI generation attempts",
|
||||
);
|
||||
return { status: 1 };
|
||||
}
|
||||
return { status: 1 };
|
||||
}
|
||||
|
||||
async function acquireCurrentSpec(
|
||||
options: EnsureOptions,
|
||||
reason: StalenessReason | undefined,
|
||||
): Promise<{ status: number; spec?: AcquiredSpec }> {
|
||||
if (!options.forceLocal && reason !== "backend source changed") {
|
||||
const url = backendUrl();
|
||||
if (url === undefined) {
|
||||
log(`invalid MB_JETTY_PORT: ${process.env.MB_JETTY_PORT}`);
|
||||
return acquireFromLocalSource(options);
|
||||
}
|
||||
const backendSpec = await fetchSpecFromBackend(url);
|
||||
if (backendSpec?.edition === "ee") {
|
||||
publishSpecAtomically(backendSpec.contents);
|
||||
log(`fetched complete OpenAPI spec from running backend (${url})`);
|
||||
return {
|
||||
status: 0,
|
||||
spec: {
|
||||
contents: backendSpec.contents,
|
||||
hash: createContentHash(backendSpec.contents),
|
||||
},
|
||||
};
|
||||
}
|
||||
if (backendSpec?.edition === "oss") {
|
||||
log("running backend exposes OSS routes only");
|
||||
}
|
||||
}
|
||||
return acquireFromLocalSource(options);
|
||||
}
|
||||
|
||||
async function generateTypes(
|
||||
spec: AcquiredSpec,
|
||||
options: EnsureOptions,
|
||||
): Promise<{ status: number; outputsHash?: string }> {
|
||||
const snapshotPath = `${SPEC_PATH}.${process.pid}.${randomUUID()}.snapshot.json`;
|
||||
const outputDirectory = `.tmp/openapi/types-out.${process.pid}.${randomUUID()}`;
|
||||
try {
|
||||
writePrivateFile(snapshotPath, spec.contents);
|
||||
const status = await runCommand(OPENAPI_TS_COMMAND, {
|
||||
postinstallWorker: options.postinstallWorker,
|
||||
env: {
|
||||
[OPENAPI_INPUT_ENV]: snapshotPath,
|
||||
[OPENAPI_OUTPUT_ENV]: outputDirectory,
|
||||
},
|
||||
});
|
||||
if (status !== 0) {
|
||||
return { status };
|
||||
}
|
||||
writePrivateFile(
|
||||
join(outputDirectory, "index.ts"),
|
||||
GENERATED_INDEX_CONTENT,
|
||||
);
|
||||
const outputsHash = createOutputsHash(outputDirectory);
|
||||
if (outputsHash === undefined) {
|
||||
log("error: type generation did not produce declaration outputs");
|
||||
return { status: 1 };
|
||||
}
|
||||
mkdirSync(GENERATED_TYPES_DIRECTORY, { recursive: true });
|
||||
for (const name of GENERATED_OUTPUT_NAMES) {
|
||||
renameSync(
|
||||
join(outputDirectory, name),
|
||||
join(GENERATED_TYPES_DIRECTORY, name),
|
||||
);
|
||||
}
|
||||
return { status: 0, outputsHash };
|
||||
} finally {
|
||||
rmSync(snapshotPath, { force: true });
|
||||
rmSync(outputDirectory, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
|
||||
async function ensureTypes(options: EnsureOptions): Promise<number> {
|
||||
const sourceDigest = createSourceFingerprint();
|
||||
const generatorDigest = createTypeGeneratorFingerprint();
|
||||
const state = readGenerationState();
|
||||
|
||||
const reason = stalenessReason(state, sourceDigest, generatorDigest);
|
||||
if (!options.forceLocal && reason === undefined) {
|
||||
return 0;
|
||||
}
|
||||
log(
|
||||
options.forceLocal
|
||||
? "regenerating from local source (forced)"
|
||||
: `regenerating API types (${reason})`,
|
||||
);
|
||||
|
||||
const acquisition = await acquireCurrentSpec(options, reason);
|
||||
if (acquisition.status !== 0 || acquisition.spec === undefined) {
|
||||
return acquisition.status || 1;
|
||||
}
|
||||
const spec = acquisition.spec;
|
||||
|
||||
const createState = (outputsHash: string): GenerationState => ({
|
||||
version: GENERATION_STATE_VERSION,
|
||||
...(spec.sourceDigest !== undefined
|
||||
? { sourceDigest: spec.sourceDigest }
|
||||
: {}),
|
||||
generatorDigest,
|
||||
specHash: spec.hash,
|
||||
outputsHash,
|
||||
});
|
||||
|
||||
const currentOutputsHash = createOutputsHash();
|
||||
const outputsAreReusable =
|
||||
!options.forceLocal &&
|
||||
state !== undefined &&
|
||||
state.generatorDigest === generatorDigest &&
|
||||
state.specHash === spec.hash &&
|
||||
state.outputsHash === currentOutputsHash &&
|
||||
currentOutputsHash !== undefined;
|
||||
if (outputsAreReusable) {
|
||||
writeGenerationStateAtomically(createState(currentOutputsHash));
|
||||
return 0;
|
||||
}
|
||||
|
||||
const generation = await generateTypes(spec, options);
|
||||
if (generation.status !== 0 || generation.outputsHash === undefined) {
|
||||
return generation.status || 1;
|
||||
}
|
||||
writeGenerationStateAtomically(createState(generation.outputsHash));
|
||||
return 0;
|
||||
}
|
||||
|
||||
export async function main(
|
||||
arguments_ = process.argv.slice(2),
|
||||
): Promise<number> {
|
||||
let options: EnsureOptions;
|
||||
try {
|
||||
options = parseOptions(arguments_);
|
||||
} catch (error) {
|
||||
const message = error instanceof Error ? error.message : String(error);
|
||||
log(`error: ${message}`);
|
||||
return 1;
|
||||
}
|
||||
|
||||
if (options.postinstall) {
|
||||
return freshBeforePostinstall() ? 0 : startPostinstallWorker();
|
||||
}
|
||||
|
||||
let status: number;
|
||||
try {
|
||||
const outcome = await withGenerationLock(
|
||||
GENERATION_LOCK_PATH,
|
||||
{
|
||||
wait: !options.postinstallWorker,
|
||||
maxWaitMs: MAX_LOCK_WAIT_MS,
|
||||
onWait: ({ path, ageMs }) =>
|
||||
log(
|
||||
`waiting for generation lock (${path}, age ${Math.round(ageMs / 1000)}s); ` +
|
||||
`if stuck: rm -rf ${path}`,
|
||||
),
|
||||
onWaitTimeout: () =>
|
||||
log("generation lock timed out after 60s; continuing without it"),
|
||||
},
|
||||
() => ensureTypes(options),
|
||||
);
|
||||
if (!outcome.executed) {
|
||||
if (!options.postinstallWorker) {
|
||||
log("API type generation is already running");
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
status = outcome.result;
|
||||
} catch (error) {
|
||||
const message = error instanceof Error ? error.message : String(error);
|
||||
log(`error: ${message}`);
|
||||
status = 1;
|
||||
}
|
||||
|
||||
if (status !== 0 && options.postinstallWorker) {
|
||||
// Background generation must not fail bun install.
|
||||
process.stderr.write(
|
||||
"[types:ensure] ⚠ could not generate API types — run `bun run types:ensure` to see the failure\n",
|
||||
);
|
||||
return 0;
|
||||
}
|
||||
return status;
|
||||
}
|
||||
|
||||
const currentScriptPath = process.argv[1];
|
||||
if (
|
||||
currentScriptPath !== undefined &&
|
||||
resolve(currentScriptPath) === fileURLToPath(import.meta.url)
|
||||
) {
|
||||
process.exit(await main());
|
||||
}
|
||||
472
frontend/build/openapi/ensure-types.unit.spec.ts
Normal file
472
frontend/build/openapi/ensure-types.unit.spec.ts
Normal file
@@ -0,0 +1,472 @@
|
||||
import { spawn, spawnSync } from "node:child_process";
|
||||
import {
|
||||
appendFileSync,
|
||||
chmodSync,
|
||||
existsSync,
|
||||
mkdirSync,
|
||||
mkdtempSync,
|
||||
readFileSync,
|
||||
rmSync,
|
||||
statSync,
|
||||
symlinkSync,
|
||||
utimesSync,
|
||||
writeFileSync,
|
||||
} from "node:fs";
|
||||
import { type Server, createServer } from "node:http";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
|
||||
import { type GenerationState, readGenerationState } from "./generation-state";
|
||||
|
||||
const ENSURE_TYPES_PATH = join(
|
||||
process.cwd(),
|
||||
"frontend/build/openapi/ensure-types.ts",
|
||||
);
|
||||
const EVENT_LOG_PATH = ".tmp/openapi/events.jsonl";
|
||||
const STATE_PATH = ".tmp/openapi/generation.json";
|
||||
const LOCK_PATH = ".tmp/openapi/types-ensure.lock";
|
||||
const TYPES_PATH = "frontend/src/metabase-types/openapi/types.gen.d.ts";
|
||||
const OPENAPI_INFO = { title: "Metabase API", version: "1.0.0" };
|
||||
const EE_SPEC = JSON.stringify({
|
||||
openapi: "3.1.0",
|
||||
info: OPENAPI_INFO,
|
||||
paths: { "/api/card": {}, "/api/ee/example": {} },
|
||||
});
|
||||
const OSS_SPEC = JSON.stringify({
|
||||
openapi: "3.1.0",
|
||||
info: OPENAPI_INFO,
|
||||
paths: { "/api/card": {} },
|
||||
});
|
||||
|
||||
interface CommandResult {
|
||||
status: number | null;
|
||||
stdout: string;
|
||||
stderr: string;
|
||||
}
|
||||
|
||||
function writeExecutable(path: string, contents: string) {
|
||||
mkdirSync(join(path, ".."), { recursive: true });
|
||||
writeFileSync(path, contents);
|
||||
chmodSync(path, 0o755);
|
||||
}
|
||||
|
||||
function writeHarness(root: string) {
|
||||
mkdirSync(join(root, "src"), { recursive: true });
|
||||
writeFileSync(join(root, "src/example.clj"), "(def example true)\n");
|
||||
mkdirSync(join(root, "frontend/build/openapi"), { recursive: true });
|
||||
writeFileSync(
|
||||
join(root, "frontend/build/openapi/openapi-ts.config.ts"),
|
||||
"export default {};\n",
|
||||
);
|
||||
writeFileSync(join(root, "package.json"), "{}\n");
|
||||
writeFileSync(join(root, "bun.lock"), "lock-v1\n");
|
||||
|
||||
// Fake `clojure` on PATH: writes the spec like the real generator does.
|
||||
writeExecutable(
|
||||
join(root, "stub-bin/clojure"),
|
||||
`#!/usr/bin/env bun
|
||||
import { appendFileSync, existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs";
|
||||
mkdirSync(".tmp/openapi", { recursive: true });
|
||||
appendFileSync(${JSON.stringify(EVENT_LOG_PATH)}, JSON.stringify({ stage: "local" }) + "\\n");
|
||||
console.log("java stdout sentinel");
|
||||
console.error("java stderr sentinel");
|
||||
if (process.env.FAIL_GENERATE) process.exit(Number(process.env.FAIL_GENERATE));
|
||||
const countPath = ".tmp/openapi/generate-count";
|
||||
const count = existsSync(countPath) ? Number(readFileSync(countPath, "utf8")) + 1 : 1;
|
||||
writeFileSync(countPath, String(count));
|
||||
if (process.env.MUTATE_SOURCE === "always" || (process.env.MUTATE_SOURCE === "once" && count === 1)) {
|
||||
appendFileSync("src/example.clj", "\\n");
|
||||
}
|
||||
writeFileSync(".tmp/openapi/openapi.json", process.env.GENERATED_SPEC ?? ${JSON.stringify(EE_SPEC)});
|
||||
`,
|
||||
);
|
||||
|
||||
// Fake hey-api: consumes METABASE_OPENAPI_INPUT, writes declaration outputs
|
||||
// into METABASE_OPENAPI_OUTPUT.
|
||||
writeExecutable(
|
||||
join(root, "node_modules/.bin/openapi-ts"),
|
||||
`#!/usr/bin/env bun
|
||||
import { appendFileSync, mkdirSync, readFileSync, writeFileSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
appendFileSync(${JSON.stringify(EVENT_LOG_PATH)}, JSON.stringify({ stage: "types" }) + "\\n");
|
||||
if (process.env.FAIL_TYPES) process.exit(Number(process.env.FAIL_TYPES));
|
||||
const input = process.env.METABASE_OPENAPI_INPUT ?? ".tmp/openapi/openapi.json";
|
||||
const output = process.env.METABASE_OPENAPI_OUTPUT ?? "frontend/src/metabase-types/openapi";
|
||||
const consumedSpec = readFileSync(input, "utf8");
|
||||
mkdirSync(output, { recursive: true });
|
||||
writeFileSync(join(output, "types.gen.d.ts"), "export type Example = " + JSON.stringify(consumedSpec) + ";\\n");
|
||||
`,
|
||||
);
|
||||
}
|
||||
|
||||
function ensureEnvironment(
|
||||
root: string,
|
||||
environment: NodeJS.ProcessEnv,
|
||||
): NodeJS.ProcessEnv {
|
||||
return {
|
||||
...process.env,
|
||||
PATH: `${join(root, "stub-bin")}:${process.env.PATH}`,
|
||||
MB_JETTY_PORT: "1",
|
||||
...environment,
|
||||
};
|
||||
}
|
||||
|
||||
function runEnsure(
|
||||
root: string,
|
||||
arguments_: string[] = [],
|
||||
environment: NodeJS.ProcessEnv = {},
|
||||
): CommandResult {
|
||||
const result = spawnSync("bun", [ENSURE_TYPES_PATH, ...arguments_], {
|
||||
cwd: root,
|
||||
encoding: "utf8",
|
||||
env: ensureEnvironment(root, environment),
|
||||
});
|
||||
return {
|
||||
status: result.status,
|
||||
stdout: result.stdout,
|
||||
stderr: result.stderr,
|
||||
};
|
||||
}
|
||||
|
||||
function runEnsureAsync(
|
||||
root: string,
|
||||
arguments_: string[] = [],
|
||||
environment: NodeJS.ProcessEnv = {},
|
||||
): Promise<CommandResult> {
|
||||
return new Promise((resolvePromise) => {
|
||||
const child = spawn("bun", [ENSURE_TYPES_PATH, ...arguments_], {
|
||||
cwd: root,
|
||||
env: ensureEnvironment(root, environment),
|
||||
});
|
||||
let stdout = "";
|
||||
let stderr = "";
|
||||
child.stdout.on("data", (chunk: Buffer) => (stdout += String(chunk)));
|
||||
child.stderr.on("data", (chunk: Buffer) => (stderr += String(chunk)));
|
||||
child.once("close", (status) => resolvePromise({ status, stdout, stderr }));
|
||||
});
|
||||
}
|
||||
|
||||
function readEvents(root: string): string[] {
|
||||
const path = join(root, EVENT_LOG_PATH);
|
||||
if (!existsSync(path)) {
|
||||
return [];
|
||||
}
|
||||
return readFileSync(path, "utf8")
|
||||
.trim()
|
||||
.split("\n")
|
||||
.filter(Boolean)
|
||||
.map((line) => {
|
||||
const value: unknown = JSON.parse(line);
|
||||
if (
|
||||
typeof value !== "object" ||
|
||||
value === null ||
|
||||
!("stage" in value) ||
|
||||
typeof value.stage !== "string"
|
||||
) {
|
||||
throw new Error(`invalid event: ${line}`);
|
||||
}
|
||||
return value.stage;
|
||||
});
|
||||
}
|
||||
|
||||
function clearEvents(root: string) {
|
||||
rmSync(join(root, EVENT_LOG_PATH), { force: true });
|
||||
}
|
||||
|
||||
function stateAt(root: string): GenerationState {
|
||||
const state = readGenerationState(join(root, STATE_PATH));
|
||||
if (state === undefined) {
|
||||
throw new Error("expected generation state");
|
||||
}
|
||||
return state;
|
||||
}
|
||||
|
||||
async function waitForFile(path: string, timeoutMs = 15_000) {
|
||||
const deadline = Date.now() + timeoutMs;
|
||||
while (!existsSync(path)) {
|
||||
if (Date.now() >= deadline) {
|
||||
throw new Error(`timed out waiting for ${path}`);
|
||||
}
|
||||
await new Promise((resolve) => setTimeout(resolve, 25));
|
||||
}
|
||||
}
|
||||
|
||||
function startFakeBackend(
|
||||
specContents: string,
|
||||
): Promise<{ server: Server; port: number; requestCount: () => number }> {
|
||||
return new Promise((resolvePromise) => {
|
||||
let requests = 0;
|
||||
const server = createServer((request, response) => {
|
||||
requests += 1;
|
||||
response.setHeader("content-type", "application/json");
|
||||
response.end(specContents);
|
||||
});
|
||||
server.listen(0, () => {
|
||||
const address = server.address();
|
||||
if (address === null || typeof address === "string") {
|
||||
throw new Error("could not determine fake backend port");
|
||||
}
|
||||
resolvePromise({
|
||||
server,
|
||||
port: address.port,
|
||||
requestCount: () => requests,
|
||||
});
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
describe("ensure-types", () => {
|
||||
let root: string;
|
||||
|
||||
beforeEach(() => {
|
||||
root = mkdtempSync(join(tmpdir(), "ensure-types-"));
|
||||
writeHarness(root);
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
rmSync(root, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it("generates from local source when no state exists, then is silently fresh", () => {
|
||||
const first = runEnsure(root);
|
||||
expect(first.status).toBe(0);
|
||||
expect(readEvents(root)).toEqual(["local", "types"]);
|
||||
expect(stateAt(root).sourceDigest).toBeDefined();
|
||||
expect(existsSync(join(root, TYPES_PATH))).toBe(true);
|
||||
expect(
|
||||
readFileSync(
|
||||
join(root, "frontend/src/metabase-types/openapi/index.ts"),
|
||||
"utf8",
|
||||
),
|
||||
).toBe('export type * from "./types.gen.d";\n');
|
||||
|
||||
clearEvents(root);
|
||||
const second = runEnsure(root);
|
||||
expect(second.status).toBe(0);
|
||||
expect(second.stdout).toBe("");
|
||||
expect(readEvents(root)).toEqual([]);
|
||||
});
|
||||
|
||||
it("logs the staleness reason when regenerating", () => {
|
||||
const first = runEnsure(root);
|
||||
expect(first.stdout).toContain("no generation state");
|
||||
|
||||
appendFileSync(join(root, "src/example.clj"), "\n;; edit\n");
|
||||
const second = runEnsure(root);
|
||||
expect(second.stdout).toContain("backend source changed");
|
||||
});
|
||||
|
||||
it("records backend-origin specs without a source digest and never treats them as source-fresh", async () => {
|
||||
const backend = await startFakeBackend(EE_SPEC);
|
||||
try {
|
||||
const environment = { MB_JETTY_PORT: String(backend.port) };
|
||||
const first = await runEnsureAsync(root, [], environment);
|
||||
expect(first.status).toBe(0);
|
||||
expect(first.stdout).toContain("running backend");
|
||||
expect(readEvents(root)).toEqual(["types"]);
|
||||
expect(stateAt(root).sourceDigest).toBeUndefined();
|
||||
expect(backend.requestCount()).toBe(1);
|
||||
|
||||
// A second run must re-fetch (never source-fresh) but may reuse the
|
||||
// generated outputs because the spec hash is unchanged.
|
||||
clearEvents(root);
|
||||
const second = await runEnsureAsync(root, [], environment);
|
||||
expect(second.status).toBe(0);
|
||||
expect(backend.requestCount()).toBe(2);
|
||||
expect(readEvents(root)).toEqual([]);
|
||||
expect(stateAt(root).sourceDigest).toBeUndefined();
|
||||
} finally {
|
||||
backend.server.close();
|
||||
}
|
||||
});
|
||||
|
||||
it("uses local generation when backend source changed", async () => {
|
||||
expect(runEnsure(root).status).toBe(0);
|
||||
appendFileSync(join(root, "src/example.clj"), "\n;; changed\n");
|
||||
clearEvents(root);
|
||||
|
||||
const backend = await startFakeBackend(EE_SPEC);
|
||||
try {
|
||||
const result = await runEnsureAsync(root, [], {
|
||||
MB_JETTY_PORT: String(backend.port),
|
||||
});
|
||||
expect(result.status).toBe(0);
|
||||
expect(backend.requestCount()).toBe(0);
|
||||
expect(readEvents(root)).toEqual(["local"]);
|
||||
expect(stateAt(root).sourceDigest).toBeDefined();
|
||||
} finally {
|
||||
backend.server.close();
|
||||
}
|
||||
});
|
||||
|
||||
it("falls back to local generation when the backend serves an OSS spec", async () => {
|
||||
const backend = await startFakeBackend(OSS_SPEC);
|
||||
try {
|
||||
const result = await runEnsureAsync(root, [], {
|
||||
MB_JETTY_PORT: String(backend.port),
|
||||
});
|
||||
expect(result.status).toBe(0);
|
||||
expect(result.stdout).toContain("OSS routes only");
|
||||
expect(readEvents(root)).toEqual(["local", "types"]);
|
||||
expect(stateAt(root).sourceDigest).toBeDefined();
|
||||
} finally {
|
||||
backend.server.close();
|
||||
}
|
||||
});
|
||||
|
||||
it.each(["0", "00080", "65536", "3000 "])(
|
||||
"rejects invalid backend port %j and generates locally",
|
||||
(port) => {
|
||||
const result = runEnsure(root, [], { MB_JETTY_PORT: port });
|
||||
expect(result.status).toBe(0);
|
||||
expect(result.stdout).toContain(`invalid MB_JETTY_PORT: ${port}`);
|
||||
expect(readEvents(root)).toEqual(["local", "types"]);
|
||||
},
|
||||
);
|
||||
|
||||
it("retries once when backend source changes during generation", () => {
|
||||
const result = runEnsure(root, [], { MUTATE_SOURCE: "once" });
|
||||
expect(result.status).toBe(0);
|
||||
expect(result.stdout).toContain("retrying once");
|
||||
expect(readEvents(root)).toEqual(["local", "local", "types"]);
|
||||
});
|
||||
|
||||
it("fails when backend source keeps changing during generation", () => {
|
||||
const result = runEnsure(root, [], { MUTATE_SOURCE: "always" });
|
||||
expect(result.status).toBe(1);
|
||||
expect(result.stdout).toContain("both OpenAPI generation attempts");
|
||||
expect(existsSync(join(root, STATE_PATH))).toBe(false);
|
||||
});
|
||||
|
||||
it("replays captured generator output only on failure", () => {
|
||||
const success = runEnsure(root);
|
||||
expect(success.stdout).not.toContain("java stdout sentinel");
|
||||
|
||||
rmSync(join(root, STATE_PATH), { force: true });
|
||||
clearEvents(root);
|
||||
const failure = runEnsure(root, [], { FAIL_GENERATE: "3" });
|
||||
expect(failure.status).toBe(3);
|
||||
expect(`${failure.stdout}${failure.stderr}`).toContain(
|
||||
"java stdout sentinel",
|
||||
);
|
||||
});
|
||||
|
||||
it("force-local regenerates even when state is fresh", () => {
|
||||
expect(runEnsure(root).status).toBe(0);
|
||||
clearEvents(root);
|
||||
|
||||
const result = runEnsure(root, ["--force-local"]);
|
||||
expect(result.status).toBe(0);
|
||||
expect(readEvents(root)).toEqual(["local", "types"]);
|
||||
});
|
||||
|
||||
it("rejects the removed integrated lint option", () => {
|
||||
const result = runEnsure(root, ["--lint"]);
|
||||
expect(result.status).toBe(1);
|
||||
expect(result.stdout).toContain("unknown option: --lint");
|
||||
});
|
||||
|
||||
it("regenerates when the generated outputs are modified on disk", () => {
|
||||
expect(runEnsure(root).status).toBe(0);
|
||||
clearEvents(root);
|
||||
|
||||
appendFileSync(join(root, TYPES_PATH), "// manual edit\n");
|
||||
const result = runEnsure(root);
|
||||
expect(result.status).toBe(0);
|
||||
expect(result.stdout).toContain("generated outputs changed on disk");
|
||||
expect(readEvents(root)).toContain("types");
|
||||
});
|
||||
|
||||
it("regenerates when the generated spec file is modified on disk", () => {
|
||||
expect(runEnsure(root).status).toBe(0);
|
||||
clearEvents(root);
|
||||
|
||||
writeFileSync(join(root, ".tmp/openapi/openapi.json"), "{}\n");
|
||||
const result = runEnsure(root);
|
||||
expect(result.status).toBe(0);
|
||||
expect(result.stdout).toContain("generated spec changed on disk");
|
||||
// Regenerating the spec from source restores the expected hash, so the
|
||||
// unchanged declarations are reused without invoking the type generator.
|
||||
expect(readEvents(root)).toEqual(["local"]);
|
||||
});
|
||||
|
||||
it("regenerates when the generated spec file is missing", () => {
|
||||
expect(runEnsure(root).status).toBe(0);
|
||||
clearEvents(root);
|
||||
|
||||
rmSync(join(root, ".tmp/openapi/openapi.json"), { force: true });
|
||||
const result = runEnsure(root);
|
||||
expect(result.status).toBe(0);
|
||||
expect(result.stdout).toContain("generated spec changed on disk");
|
||||
expect(readEvents(root)).toEqual(["local"]);
|
||||
});
|
||||
|
||||
it("worker exits zero with a stderr warning when generation fails", () => {
|
||||
const result = runEnsure(root, ["--postinstall-worker"], {
|
||||
FAIL_GENERATE: "1",
|
||||
});
|
||||
expect(result.status).toBe(0);
|
||||
expect(result.stderr).toContain("could not generate API types");
|
||||
});
|
||||
|
||||
it("reports the lock path and holder age while waiting", async () => {
|
||||
mkdirSync(join(root, LOCK_PATH), { recursive: true });
|
||||
|
||||
const pending = runEnsureAsync(root);
|
||||
await new Promise((resolve) => setTimeout(resolve, 600));
|
||||
rmSync(join(root, LOCK_PATH), { recursive: true, force: true });
|
||||
const result = await pending;
|
||||
|
||||
expect(result.status).toBe(0);
|
||||
expect(result.stdout).toContain("waiting for generation lock");
|
||||
expect(result.stdout).toContain(`rm -rf ${LOCK_PATH}`);
|
||||
});
|
||||
|
||||
describe("postinstall", () => {
|
||||
it("stays silent and spawns nothing when types are fresh", () => {
|
||||
expect(runEnsure(root).status).toBe(0);
|
||||
clearEvents(root);
|
||||
|
||||
const result = runEnsure(root, ["--postinstall"]);
|
||||
expect(result.status).toBe(0);
|
||||
expect(result.stdout).toBe("");
|
||||
expect(readEvents(root)).toEqual([]);
|
||||
});
|
||||
|
||||
it("prints one background line and completes generation via the worker", async () => {
|
||||
const result = runEnsure(root, ["--postinstall"]);
|
||||
expect(result.status).toBe(0);
|
||||
expect(result.stdout).toContain("generating API types in the background");
|
||||
expect(
|
||||
statSync(join(root, ".tmp/openapi/types-ensure.log")).mode & 0o777,
|
||||
).toBe(0o600);
|
||||
|
||||
await waitForFile(join(root, STATE_PATH));
|
||||
expect(stateAt(root).sourceDigest).toBeDefined();
|
||||
expect(existsSync(join(root, TYPES_PATH))).toBe(true);
|
||||
});
|
||||
|
||||
it("does not follow a symlink at the worker log path", () => {
|
||||
const target = join(root, "log-target");
|
||||
writeFileSync(target, "unchanged\n");
|
||||
mkdirSync(join(root, ".tmp/openapi"), { recursive: true });
|
||||
symlinkSync(target, join(root, ".tmp/openapi/types-ensure.log"));
|
||||
|
||||
const result = runEnsure(root, ["--postinstall"]);
|
||||
expect(result.status).toBe(0);
|
||||
expect(result.stdout).toContain("could not start API type generation");
|
||||
expect(readFileSync(target, "utf8")).toBe("unchanged\n");
|
||||
});
|
||||
|
||||
it("worker exits quietly when the lock is held", () => {
|
||||
mkdirSync(join(root, LOCK_PATH), { recursive: true });
|
||||
const now = new Date();
|
||||
utimesSync(join(root, LOCK_PATH), now, now);
|
||||
|
||||
const result = runEnsure(root, ["--postinstall-worker"]);
|
||||
expect(result.status).toBe(0);
|
||||
expect(result.stdout).toBe("");
|
||||
expect(readEvents(root)).toEqual([]);
|
||||
});
|
||||
});
|
||||
});
|
||||
131
frontend/build/openapi/generation-lock.ts
Normal file
131
frontend/build/openapi/generation-lock.ts
Normal file
@@ -0,0 +1,131 @@
|
||||
/**
|
||||
* Coordinates OpenAPI generation across regular commands and postinstall
|
||||
* workers.
|
||||
*
|
||||
* proper-lockfile owns the cross-process lock, heartbeat, stale recovery, and
|
||||
* release mechanics. This wrapper adds the command-specific wait policy: a
|
||||
* postinstall worker returns immediately, while regular commands wait for a
|
||||
* bounded time before continuing without the lock.
|
||||
*
|
||||
* The lock avoids duplicate generation; output staging and freshness hashes
|
||||
* protect correctness.
|
||||
*/
|
||||
import { mkdirSync, statSync } from "node:fs";
|
||||
import { dirname } from "node:path";
|
||||
import { setTimeout } from "node:timers";
|
||||
|
||||
import { lock } from "proper-lockfile";
|
||||
|
||||
export interface WithLockOptions {
|
||||
/** When false, return `{ executed: false }` instead of waiting. */
|
||||
wait: boolean;
|
||||
/** A lock whose mtime is older than this is treated as abandoned. */
|
||||
staleMs?: number;
|
||||
/** How often the holder refreshes the lock directory's mtime. */
|
||||
heartbeatMs?: number;
|
||||
/** How often a waiter re-checks the lock. */
|
||||
pollMs?: number;
|
||||
/** Maximum time to wait before running without the lock. */
|
||||
maxWaitMs?: number;
|
||||
/** Called once per acquisition when waiting begins. */
|
||||
onWait?: (info: { path: string; ageMs: number }) => void;
|
||||
/** Called before running without the lock after the maximum wait. */
|
||||
onWaitTimeout?: (info: { path: string; waitedMs: number }) => void;
|
||||
}
|
||||
|
||||
export type WithLockResult<T> =
|
||||
| { executed: true; result: T }
|
||||
| { executed: false };
|
||||
|
||||
type ReleaseLock = () => Promise<void>;
|
||||
|
||||
function hasErrorCode(error: unknown, code: string): boolean {
|
||||
return (
|
||||
typeof error === "object" &&
|
||||
error !== null &&
|
||||
"code" in error &&
|
||||
error.code === code
|
||||
);
|
||||
}
|
||||
|
||||
function sleep(milliseconds: number): Promise<void> {
|
||||
return new Promise((resolve) => setTimeout(resolve, milliseconds));
|
||||
}
|
||||
|
||||
function lockAgeMs(lockPath: string): number | undefined {
|
||||
try {
|
||||
return Date.now() - statSync(lockPath).mtimeMs;
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
}
|
||||
|
||||
async function tryAcquireLock(
|
||||
lockPath: string,
|
||||
staleMs: number,
|
||||
heartbeatMs: number,
|
||||
): Promise<ReleaseLock | undefined> {
|
||||
try {
|
||||
return await lock(lockPath, {
|
||||
lockfilePath: lockPath,
|
||||
onCompromised: () => {
|
||||
// Generation can finish safely after losing this optimization-only lock.
|
||||
},
|
||||
realpath: false,
|
||||
retries: 0,
|
||||
stale: staleMs,
|
||||
update: heartbeatMs,
|
||||
});
|
||||
} catch (error) {
|
||||
if (hasErrorCode(error, "ELOCKED")) {
|
||||
return undefined;
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
export async function withGenerationLock<T>(
|
||||
lockPath: string,
|
||||
options: WithLockOptions,
|
||||
action: () => Promise<T>,
|
||||
): Promise<WithLockResult<T>> {
|
||||
const {
|
||||
wait,
|
||||
staleMs = 30_000,
|
||||
heartbeatMs = 5_000,
|
||||
pollMs = 250,
|
||||
maxWaitMs = 60_000,
|
||||
onWait,
|
||||
onWaitTimeout,
|
||||
} = options;
|
||||
mkdirSync(dirname(lockPath), { recursive: true });
|
||||
const waitStartedAt = Date.now();
|
||||
let release = await tryAcquireLock(lockPath, staleMs, heartbeatMs);
|
||||
|
||||
if (release === undefined && !wait) {
|
||||
return { executed: false };
|
||||
}
|
||||
if (release === undefined) {
|
||||
onWait?.({ path: lockPath, ageMs: lockAgeMs(lockPath) ?? 0 });
|
||||
}
|
||||
|
||||
while (release === undefined) {
|
||||
const waitedMs = Date.now() - waitStartedAt;
|
||||
if (waitedMs >= maxWaitMs) {
|
||||
onWaitTimeout?.({ path: lockPath, waitedMs });
|
||||
return { executed: true, result: await action() };
|
||||
}
|
||||
await sleep(Math.min(pollMs, maxWaitMs - waitedMs));
|
||||
release = await tryAcquireLock(lockPath, staleMs, heartbeatMs);
|
||||
}
|
||||
|
||||
try {
|
||||
return { executed: true, result: await action() };
|
||||
} finally {
|
||||
try {
|
||||
await release();
|
||||
} catch {
|
||||
// Losing or failing to release this optimization-only lock is non-fatal.
|
||||
}
|
||||
}
|
||||
}
|
||||
184
frontend/build/openapi/generation-lock.unit.spec.ts
Normal file
184
frontend/build/openapi/generation-lock.unit.spec.ts
Normal file
@@ -0,0 +1,184 @@
|
||||
import {
|
||||
existsSync,
|
||||
mkdirSync,
|
||||
mkdtempSync,
|
||||
rmSync,
|
||||
statSync,
|
||||
symlinkSync,
|
||||
utimesSync,
|
||||
} from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
|
||||
import { withGenerationLock } from "./generation-lock";
|
||||
|
||||
function sleep(milliseconds: number): Promise<void> {
|
||||
return new Promise((resolve) => setTimeout(resolve, milliseconds));
|
||||
}
|
||||
|
||||
function writeForeignLock(lockPath: string, ageMs = 0): void {
|
||||
mkdirSync(lockPath, { recursive: true });
|
||||
if (ageMs > 0) {
|
||||
const past = new Date(Date.now() - ageMs);
|
||||
utimesSync(lockPath, past, past);
|
||||
}
|
||||
}
|
||||
|
||||
describe("withGenerationLock", () => {
|
||||
let directory: string;
|
||||
let lockPath: string;
|
||||
|
||||
beforeEach(() => {
|
||||
directory = mkdtempSync(join(tmpdir(), "generation-lock-"));
|
||||
lockPath = join(directory, "nested", "test.lock");
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
rmSync(directory, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it("runs the action while holding the lock and removes it afterwards", async () => {
|
||||
const outcome = await withGenerationLock(
|
||||
lockPath,
|
||||
{ wait: true },
|
||||
async () => {
|
||||
expect(statSync(lockPath).isDirectory()).toBe(true);
|
||||
return "done";
|
||||
},
|
||||
);
|
||||
expect(outcome).toEqual({ executed: true, result: "done" });
|
||||
expect(existsSync(lockPath)).toBe(false);
|
||||
});
|
||||
|
||||
it("returns executed:false without waiting when the lock is held", async () => {
|
||||
writeForeignLock(lockPath);
|
||||
const outcome = await withGenerationLock(
|
||||
lockPath,
|
||||
{ wait: false },
|
||||
async () => "should not run",
|
||||
);
|
||||
expect(outcome).toEqual({ executed: false });
|
||||
expect(existsSync(lockPath)).toBe(true);
|
||||
});
|
||||
|
||||
it("waits for a live holder and acquires after release", async () => {
|
||||
const events: string[] = [];
|
||||
const holder = withGenerationLock(lockPath, { wait: true }, async () => {
|
||||
events.push("holder-start");
|
||||
await sleep(300);
|
||||
events.push("holder-end");
|
||||
});
|
||||
await sleep(50);
|
||||
const waiter = withGenerationLock(
|
||||
lockPath,
|
||||
{ wait: true, pollMs: 25 },
|
||||
async () => {
|
||||
events.push("waiter-start");
|
||||
},
|
||||
);
|
||||
await Promise.all([holder, waiter]);
|
||||
expect(events).toEqual(["holder-start", "holder-end", "waiter-start"]);
|
||||
expect(existsSync(lockPath)).toBe(false);
|
||||
});
|
||||
|
||||
it("takes over a stale lock directory", async () => {
|
||||
writeForeignLock(lockPath, 60_000);
|
||||
const outcome = await withGenerationLock(
|
||||
lockPath,
|
||||
{ wait: false, staleMs: 5_000 },
|
||||
async () => "recovered",
|
||||
);
|
||||
expect(outcome).toEqual({ executed: true, result: "recovered" });
|
||||
expect(existsSync(lockPath)).toBe(false);
|
||||
});
|
||||
|
||||
it("refreshes the lock directory while the action runs", async () => {
|
||||
await withGenerationLock(
|
||||
lockPath,
|
||||
{ wait: true, staleMs: 5_000, heartbeatMs: 1_000 },
|
||||
async () => {
|
||||
const initialMtime = statSync(lockPath).mtimeMs;
|
||||
await sleep(1_200);
|
||||
expect(statSync(lockPath).mtimeMs).toBeGreaterThan(initialMtime);
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
it("reports waiting once with the lock path and holder age", async () => {
|
||||
writeForeignLock(lockPath, 5_000);
|
||||
const waits: { path: string; ageMs: number }[] = [];
|
||||
const waiter = withGenerationLock(
|
||||
lockPath,
|
||||
{ wait: true, pollMs: 25, onWait: (info) => waits.push(info) },
|
||||
async () => "ran",
|
||||
);
|
||||
await sleep(120);
|
||||
rmSync(lockPath, { recursive: true, force: true });
|
||||
const outcome = await waiter;
|
||||
expect(outcome).toEqual({ executed: true, result: "ran" });
|
||||
expect(waits).toHaveLength(1);
|
||||
expect(waits[0]?.path).toBe(lockPath);
|
||||
expect(waits[0]?.ageMs).toBeGreaterThanOrEqual(4_000);
|
||||
});
|
||||
|
||||
it("does not spin when an existing lock cannot be statted", async () => {
|
||||
mkdirSync(join(lockPath, ".."), { recursive: true });
|
||||
symlinkSync(join(directory, "missing"), lockPath);
|
||||
|
||||
const worker = await withGenerationLock(
|
||||
lockPath,
|
||||
{ wait: false },
|
||||
async () => "should not run",
|
||||
);
|
||||
expect(worker).toEqual({ executed: false });
|
||||
|
||||
const waiter = await withGenerationLock(
|
||||
lockPath,
|
||||
{ wait: true, maxWaitMs: 20, pollMs: 5 },
|
||||
async () => "ran",
|
||||
);
|
||||
expect(waiter).toEqual({ executed: true, result: "ran" });
|
||||
});
|
||||
|
||||
it("runs without the lock after the maximum wait", async () => {
|
||||
writeForeignLock(lockPath);
|
||||
const timeouts: string[] = [];
|
||||
const outcome = await withGenerationLock(
|
||||
lockPath,
|
||||
{
|
||||
wait: true,
|
||||
maxWaitMs: 50,
|
||||
pollMs: 10,
|
||||
onWaitTimeout: ({ path }) => timeouts.push(path),
|
||||
},
|
||||
async () => "ran",
|
||||
);
|
||||
|
||||
expect(outcome).toEqual({ executed: true, result: "ran" });
|
||||
expect(timeouts).toEqual([lockPath]);
|
||||
expect(existsSync(lockPath)).toBe(true);
|
||||
});
|
||||
|
||||
it("finishes the action when the lock is compromised", async () => {
|
||||
const outcome = await withGenerationLock(
|
||||
lockPath,
|
||||
{ wait: true, staleMs: 5_000, heartbeatMs: 1_000 },
|
||||
async () => {
|
||||
rmSync(lockPath, { recursive: true, force: true });
|
||||
await sleep(1_200);
|
||||
return "done";
|
||||
},
|
||||
);
|
||||
|
||||
expect(outcome).toEqual({ executed: true, result: "done" });
|
||||
});
|
||||
|
||||
it("releases the lock when the action throws", async () => {
|
||||
await expect(
|
||||
withGenerationLock(lockPath, { wait: true }, async () => {
|
||||
throw new Error("boom");
|
||||
}),
|
||||
).rejects.toThrow("boom");
|
||||
expect(existsSync(lockPath)).toBe(false);
|
||||
});
|
||||
});
|
||||
155
frontend/build/openapi/generation-state.ts
Normal file
155
frontend/build/openapi/generation-state.ts
Normal file
@@ -0,0 +1,155 @@
|
||||
/**
|
||||
* Stores the hashes used to decide whether generated OpenAPI types are current.
|
||||
*
|
||||
* The state covers backend source, generator inputs, the OpenAPI spec, and the
|
||||
* generated declarations. State from a running backend omits the source hash
|
||||
* because the source loaded by that process can't be observed. Missing or
|
||||
* invalid state triggers regeneration.
|
||||
*/
|
||||
import { createHash, randomUUID } from "node:crypto";
|
||||
import {
|
||||
closeSync,
|
||||
mkdirSync,
|
||||
openSync,
|
||||
readFileSync,
|
||||
renameSync,
|
||||
rmSync,
|
||||
writeFileSync,
|
||||
} from "node:fs";
|
||||
import { dirname, join } from "node:path";
|
||||
|
||||
export const GENERATION_STATE_VERSION = 1;
|
||||
|
||||
export interface GenerationState {
|
||||
version: typeof GENERATION_STATE_VERSION;
|
||||
/** Absent for backend-origin specs; see module docs. */
|
||||
sourceDigest?: string;
|
||||
generatorDigest: string;
|
||||
specHash: string;
|
||||
outputsHash: string;
|
||||
}
|
||||
|
||||
export const GENERATION_STATE_PATH = ".tmp/openapi/generation.json";
|
||||
export const GENERATED_TYPES_DIRECTORY = "frontend/src/metabase-types/openapi";
|
||||
export const GENERATED_OUTPUT_NAMES = ["types.gen.d.ts", "index.ts"] as const;
|
||||
|
||||
const HASH_PATTERN = /^sha256:[a-f0-9]{64}$/;
|
||||
|
||||
function isHash(value: unknown): value is string {
|
||||
return typeof value === "string" && HASH_PATTERN.test(value);
|
||||
}
|
||||
|
||||
function isRecord(value: unknown): value is Record<string, unknown> {
|
||||
return typeof value === "object" && value !== null && !Array.isArray(value);
|
||||
}
|
||||
|
||||
export function createContentHash(contents: string | Buffer): string {
|
||||
const hash = createHash("sha256");
|
||||
if (typeof contents === "string") {
|
||||
hash.update(contents);
|
||||
} else {
|
||||
// Hex round-trips keep typescript7's stricter BinaryLike checks happy.
|
||||
hash.update(contents.toString("hex"), "hex");
|
||||
}
|
||||
return `sha256:${hash.digest("hex")}`;
|
||||
}
|
||||
|
||||
export function readGenerationState(
|
||||
path = GENERATION_STATE_PATH,
|
||||
): GenerationState | undefined {
|
||||
try {
|
||||
const record: unknown = JSON.parse(readFileSync(path, "utf8"));
|
||||
if (!isRecord(record) || record.version !== GENERATION_STATE_VERSION) {
|
||||
return undefined;
|
||||
}
|
||||
if (
|
||||
!isHash(record.generatorDigest) ||
|
||||
!isHash(record.specHash) ||
|
||||
!isHash(record.outputsHash) ||
|
||||
(record.sourceDigest !== undefined && !isHash(record.sourceDigest))
|
||||
) {
|
||||
return undefined;
|
||||
}
|
||||
return {
|
||||
version: GENERATION_STATE_VERSION,
|
||||
...(isHash(record.sourceDigest)
|
||||
? { sourceDigest: record.sourceDigest }
|
||||
: {}),
|
||||
generatorDigest: record.generatorDigest,
|
||||
specHash: record.specHash,
|
||||
outputsHash: record.outputsHash,
|
||||
};
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
}
|
||||
|
||||
export function writeGenerationStateAtomically(
|
||||
state: GenerationState,
|
||||
path = GENERATION_STATE_PATH,
|
||||
): void {
|
||||
const temporaryPath = `${path}.${process.pid}.${randomUUID()}.tmp`;
|
||||
let fileDescriptor: number | undefined;
|
||||
mkdirSync(dirname(path), { recursive: true });
|
||||
try {
|
||||
fileDescriptor = openSync(temporaryPath, "wx", 0o600);
|
||||
writeFileSync(fileDescriptor, `${JSON.stringify(state, null, 2)}\n`);
|
||||
closeSync(fileDescriptor);
|
||||
fileDescriptor = undefined;
|
||||
renameSync(temporaryPath, path);
|
||||
} finally {
|
||||
if (fileDescriptor !== undefined) {
|
||||
closeSync(fileDescriptor);
|
||||
}
|
||||
rmSync(temporaryPath, { force: true });
|
||||
}
|
||||
}
|
||||
|
||||
/** Hash of the generated declaration outputs, or undefined if any is missing. */
|
||||
export function createOutputsHash(
|
||||
typesDirectory = GENERATED_TYPES_DIRECTORY,
|
||||
): string | undefined {
|
||||
const hash = createHash("sha256");
|
||||
for (const name of GENERATED_OUTPUT_NAMES) {
|
||||
try {
|
||||
const contents = readFileSync(join(typesDirectory, name));
|
||||
hash.update(name);
|
||||
// Hex round-trips keep typescript7's stricter BinaryLike checks happy.
|
||||
hash.update(contents.toString("hex"), "hex");
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
}
|
||||
return `sha256:${hash.digest("hex")}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* A shallow shape check plus the one distinction that matters: an EE spec
|
||||
* includes `/api/ee/` routes; an OSS backend's spec does not.
|
||||
*/
|
||||
export function getOpenApiEdition(
|
||||
specContents: string,
|
||||
): "ee" | "oss" | undefined {
|
||||
try {
|
||||
const document: unknown = JSON.parse(specContents);
|
||||
if (
|
||||
typeof document !== "object" ||
|
||||
document === null ||
|
||||
!("openapi" in document) ||
|
||||
typeof document.openapi !== "string" ||
|
||||
!("paths" in document) ||
|
||||
typeof document.paths !== "object" ||
|
||||
document.paths === null ||
|
||||
Array.isArray(document.paths)
|
||||
) {
|
||||
return undefined;
|
||||
}
|
||||
return Object.keys(document.paths).some((path) =>
|
||||
path.startsWith("/api/ee/"),
|
||||
)
|
||||
? "ee"
|
||||
: "oss";
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
}
|
||||
138
frontend/build/openapi/generation-state.unit.spec.ts
Normal file
138
frontend/build/openapi/generation-state.unit.spec.ts
Normal file
@@ -0,0 +1,138 @@
|
||||
import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
|
||||
import {
|
||||
GENERATION_STATE_VERSION,
|
||||
type GenerationState,
|
||||
createContentHash,
|
||||
createOutputsHash,
|
||||
getOpenApiEdition,
|
||||
readGenerationState,
|
||||
writeGenerationStateAtomically,
|
||||
} from "./generation-state";
|
||||
|
||||
const EE_SPEC = JSON.stringify({
|
||||
openapi: "3.1.0",
|
||||
info: { title: "Metabase API", version: "v1.0.0" },
|
||||
paths: {
|
||||
"/api/user": { get: { responses: { "200": {} } } },
|
||||
"/api/ee/scim/v2/Users": { get: { responses: { "200": {} } } },
|
||||
},
|
||||
});
|
||||
|
||||
const OSS_SPEC = JSON.stringify({
|
||||
openapi: "3.1.0",
|
||||
info: { title: "Metabase API", version: "v1.0.0" },
|
||||
paths: {
|
||||
"/api/user": { get: { responses: { "200": {} } } },
|
||||
},
|
||||
});
|
||||
|
||||
describe("generation state", () => {
|
||||
let directory: string;
|
||||
let statePath: string;
|
||||
|
||||
beforeEach(() => {
|
||||
directory = mkdtempSync(join(tmpdir(), "generation-state-"));
|
||||
statePath = join(directory, "nested", "generation.json");
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
rmSync(directory, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it("round-trips a state with a source digest", () => {
|
||||
const state: GenerationState = {
|
||||
version: GENERATION_STATE_VERSION,
|
||||
sourceDigest: createContentHash("source"),
|
||||
generatorDigest: createContentHash("generator"),
|
||||
specHash: createContentHash("spec"),
|
||||
outputsHash: createContentHash("outputs"),
|
||||
};
|
||||
writeGenerationStateAtomically(state, statePath);
|
||||
expect(readGenerationState(statePath)).toEqual(state);
|
||||
});
|
||||
|
||||
it("round-trips a backend-origin state without a source digest", () => {
|
||||
const state: GenerationState = {
|
||||
version: GENERATION_STATE_VERSION,
|
||||
generatorDigest: createContentHash("generator"),
|
||||
specHash: createContentHash("spec"),
|
||||
outputsHash: createContentHash("outputs"),
|
||||
};
|
||||
writeGenerationStateAtomically(state, statePath);
|
||||
const readBack = readGenerationState(statePath);
|
||||
expect(readBack).toEqual(state);
|
||||
expect(readBack?.sourceDigest).toBeUndefined();
|
||||
});
|
||||
|
||||
it("returns undefined for missing, malformed, or old-schema files", () => {
|
||||
expect(readGenerationState(statePath)).toBeUndefined();
|
||||
|
||||
mkdirSync(join(statePath, ".."), { recursive: true });
|
||||
writeFileSync(statePath, "not json");
|
||||
expect(readGenerationState(statePath)).toBeUndefined();
|
||||
|
||||
writeFileSync(
|
||||
statePath,
|
||||
JSON.stringify({ schemaVersion: 2, source: {}, spec: {}, types: {} }),
|
||||
);
|
||||
expect(readGenerationState(statePath)).toBeUndefined();
|
||||
|
||||
writeFileSync(
|
||||
statePath,
|
||||
JSON.stringify({
|
||||
version: GENERATION_STATE_VERSION,
|
||||
generatorDigest: "not-a-hash",
|
||||
}),
|
||||
);
|
||||
expect(readGenerationState(statePath)).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
describe("createOutputsHash", () => {
|
||||
let typesDirectory: string;
|
||||
|
||||
beforeEach(() => {
|
||||
typesDirectory = mkdtempSync(join(tmpdir(), "generation-outputs-"));
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
rmSync(typesDirectory, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it("returns undefined when either output file is missing", () => {
|
||||
expect(createOutputsHash(typesDirectory)).toBeUndefined();
|
||||
writeFileSync(join(typesDirectory, "index.ts"), "export {};");
|
||||
expect(createOutputsHash(typesDirectory)).toBeUndefined();
|
||||
});
|
||||
|
||||
it("changes when an output changes", () => {
|
||||
writeFileSync(join(typesDirectory, "index.ts"), "export {};");
|
||||
writeFileSync(join(typesDirectory, "types.gen.d.ts"), "export type A = 1;");
|
||||
const before = createOutputsHash(typesDirectory);
|
||||
expect(before).toMatch(/^sha256:[a-f0-9]{64}$/);
|
||||
|
||||
writeFileSync(join(typesDirectory, "types.gen.d.ts"), "export type A = 2;");
|
||||
expect(createOutputsHash(typesDirectory)).not.toBe(before);
|
||||
});
|
||||
});
|
||||
|
||||
describe("getOpenApiEdition", () => {
|
||||
it("classifies EE and OSS specs", () => {
|
||||
expect(getOpenApiEdition(EE_SPEC)).toBe("ee");
|
||||
expect(getOpenApiEdition(OSS_SPEC)).toBe("oss");
|
||||
});
|
||||
|
||||
it("rejects non-JSON and shapeless JSON", () => {
|
||||
expect(getOpenApiEdition("not json")).toBeUndefined();
|
||||
expect(getOpenApiEdition("{}")).toBeUndefined();
|
||||
expect(
|
||||
getOpenApiEdition(JSON.stringify({ openapi: "3.1.0" })),
|
||||
).toBeUndefined();
|
||||
expect(
|
||||
getOpenApiEdition(JSON.stringify({ openapi: "3.1.0", paths: [] })),
|
||||
).toBeUndefined();
|
||||
});
|
||||
});
|
||||
@@ -2,25 +2,18 @@ import { defineConfig } from "@hey-api/openapi-ts";
|
||||
|
||||
// eslint-disable-next-line import/no-default-export -- this library requires a default export
|
||||
export default defineConfig({
|
||||
input: "./resources/openapi/openapi.json",
|
||||
input: process.env.METABASE_OPENAPI_INPUT ?? "./.tmp/openapi/openapi.json",
|
||||
output: {
|
||||
path: "frontend/src/metabase-types/openapi",
|
||||
clean: false,
|
||||
entryFile: false,
|
||||
path:
|
||||
process.env.METABASE_OPENAPI_OUTPUT ??
|
||||
"frontend/src/metabase-types/openapi",
|
||||
// Declaration output is exempt from checking via `skipLibCheck`, which keeps the
|
||||
// circular legacy-MBQL aliases from failing tsc.
|
||||
fileName: { suffix: ".gen.d" },
|
||||
// Clean declaration files to avoid stale types
|
||||
clean: true,
|
||||
postProcess: ["prettier"],
|
||||
},
|
||||
parser: {
|
||||
filters: {
|
||||
schemas: {
|
||||
// MetabaseLegacyMbqlSchema has circular references that break TS type checking
|
||||
// eslint-disable-next-line metabase/no-literal-metabase-strings -- this is not user facing text
|
||||
exclude: ["/^MetabaseLegacyMbqlSchema/"],
|
||||
},
|
||||
},
|
||||
},
|
||||
plugins: [
|
||||
{
|
||||
name: "@hey-api/typescript",
|
||||
enums: "javascript", // This generates runtime enums!
|
||||
},
|
||||
],
|
||||
plugins: ["@hey-api/typescript"],
|
||||
});
|
||||
|
||||
163
frontend/build/openapi/source-fingerprint.ts
Normal file
163
frontend/build/openapi/source-fingerprint.ts
Normal file
@@ -0,0 +1,163 @@
|
||||
/**
|
||||
* Content fingerprints for the OpenAPI generation pipeline.
|
||||
*
|
||||
* These are deliberately single-pass: files are listed, sorted, and hashed
|
||||
* once, with no snapshot-consistency retries. The generation path in
|
||||
* ensure-types.ts brackets local spec generation with a before/after digest
|
||||
* comparison. Generation assumes source inputs are quiescent; detected edits
|
||||
* retry, while an exact edit-and-revert inside the window is outside the
|
||||
* freshness guarantee.
|
||||
*/
|
||||
import { type Hash, createHash } from "node:crypto";
|
||||
import { lstatSync, readFileSync, readdirSync, readlinkSync } from "node:fs";
|
||||
import { join, sep } from "node:path";
|
||||
|
||||
export const SOURCE_FINGERPRINT_POLICY = "backend-source-content-v2";
|
||||
export const TYPE_GENERATOR_FINGERPRINT_POLICY = "hey-api-inputs-v2";
|
||||
|
||||
const SOURCE_ROOTS = ["src", "enterprise/backend/src"];
|
||||
/** Directories whose immediate children may contain a deps.edn. */
|
||||
const DEPS_PARENT_ROOTS = ["modules/drivers", "bin"];
|
||||
const OPTIONAL_SOURCE_PATHS = [
|
||||
"resources/data_readers.clj",
|
||||
"resources/locales.clj",
|
||||
"resources/version.properties",
|
||||
];
|
||||
const TYPE_GENERATOR_PATHS = [
|
||||
"frontend/build/openapi/openapi-ts.config.ts",
|
||||
"package.json",
|
||||
"bun.lock",
|
||||
];
|
||||
const EE_GENERATION_SETTINGS =
|
||||
"edition=ee\ncommand=clojure -M:run:ee generate-openapi-spec";
|
||||
|
||||
interface FingerprintRecord {
|
||||
kind: "file" | "symlink" | "missing";
|
||||
path: string;
|
||||
contents: Buffer;
|
||||
}
|
||||
|
||||
function normalizePath(path: string): string {
|
||||
return sep === "/" ? path : path.split(sep).join("/");
|
||||
}
|
||||
|
||||
function listTreeFiles(repositoryRoot: string, relativeRoot: string): string[] {
|
||||
const absoluteRoot = join(repositoryRoot, relativeRoot);
|
||||
let stats;
|
||||
try {
|
||||
stats = lstatSync(absoluteRoot);
|
||||
} catch {
|
||||
return [];
|
||||
}
|
||||
if (stats.isFile() || stats.isSymbolicLink()) {
|
||||
return [relativeRoot];
|
||||
}
|
||||
if (!stats.isDirectory()) {
|
||||
return [];
|
||||
}
|
||||
let names: string[];
|
||||
try {
|
||||
names = readdirSync(absoluteRoot);
|
||||
} catch {
|
||||
return [];
|
||||
}
|
||||
return names.flatMap((name) =>
|
||||
listTreeFiles(repositoryRoot, join(relativeRoot, name)),
|
||||
);
|
||||
}
|
||||
|
||||
function listDepsFiles(repositoryRoot: string): string[] {
|
||||
const paths = ["deps.edn"];
|
||||
for (const parentRoot of DEPS_PARENT_ROOTS) {
|
||||
let names: string[] = [];
|
||||
try {
|
||||
names = readdirSync(join(repositoryRoot, parentRoot));
|
||||
} catch {
|
||||
// The parent directory is optional.
|
||||
}
|
||||
paths.push(...names.map((name) => join(parentRoot, name, "deps.edn")));
|
||||
}
|
||||
return paths;
|
||||
}
|
||||
|
||||
function readRecord(repositoryRoot: string, path: string): FingerprintRecord {
|
||||
const normalizedPath = normalizePath(path);
|
||||
const absolutePath = join(repositoryRoot, path);
|
||||
try {
|
||||
const stats = lstatSync(absolutePath);
|
||||
if (stats.isSymbolicLink()) {
|
||||
return {
|
||||
kind: "symlink",
|
||||
path: normalizedPath,
|
||||
contents: Buffer.from(readlinkSync(absolutePath)),
|
||||
};
|
||||
}
|
||||
if (stats.isFile()) {
|
||||
return {
|
||||
kind: "file",
|
||||
path: normalizedPath,
|
||||
contents: readFileSync(absolutePath),
|
||||
};
|
||||
}
|
||||
} catch {
|
||||
// Deleted or unreadable mid-walk; fall through to "missing".
|
||||
}
|
||||
return { kind: "missing", path: normalizedPath, contents: Buffer.alloc(0) };
|
||||
}
|
||||
|
||||
function writeFrame(hash: Hash, value: string | Buffer): void {
|
||||
const contents = typeof value === "string" ? Buffer.from(value) : value;
|
||||
const frame = Buffer.alloc(8);
|
||||
frame.writeBigUInt64BE(BigInt(contents.byteLength));
|
||||
// Hex round-trips keep typescript7's stricter BinaryLike checks happy.
|
||||
hash.update(frame.toString("hex"), "hex");
|
||||
hash.update(contents.toString("hex"), "hex");
|
||||
}
|
||||
|
||||
function createFingerprint(
|
||||
repositoryRoot: string,
|
||||
policy: string,
|
||||
paths: string[],
|
||||
additionalRecords: FingerprintRecord[] = [],
|
||||
): string {
|
||||
const uniqueSortedPaths = [...new Set(paths.map(normalizePath))].sort();
|
||||
const hash = createHash("sha256");
|
||||
const records = [
|
||||
...additionalRecords,
|
||||
...uniqueSortedPaths.map((path) => readRecord(repositoryRoot, path)),
|
||||
];
|
||||
for (const record of records) {
|
||||
writeFrame(hash, policy);
|
||||
writeFrame(hash, record.kind);
|
||||
writeFrame(hash, record.path);
|
||||
writeFrame(hash, record.contents);
|
||||
}
|
||||
return `sha256:${hash.digest("hex")}`;
|
||||
}
|
||||
|
||||
export function createSourceFingerprint(
|
||||
repositoryRoot = process.cwd(),
|
||||
): string {
|
||||
const paths = [
|
||||
...SOURCE_ROOTS.flatMap((root) => listTreeFiles(repositoryRoot, root)),
|
||||
...listDepsFiles(repositoryRoot),
|
||||
...OPTIONAL_SOURCE_PATHS,
|
||||
];
|
||||
return createFingerprint(repositoryRoot, SOURCE_FINGERPRINT_POLICY, paths, [
|
||||
{
|
||||
kind: "file",
|
||||
path: "@ee-generation-settings",
|
||||
contents: Buffer.from(EE_GENERATION_SETTINGS),
|
||||
},
|
||||
]);
|
||||
}
|
||||
|
||||
export function createTypeGeneratorFingerprint(
|
||||
repositoryRoot = process.cwd(),
|
||||
): string {
|
||||
return createFingerprint(
|
||||
repositoryRoot,
|
||||
TYPE_GENERATOR_FINGERPRINT_POLICY,
|
||||
TYPE_GENERATOR_PATHS,
|
||||
);
|
||||
}
|
||||
159
frontend/build/openapi/source-fingerprint.unit.spec.ts
Normal file
159
frontend/build/openapi/source-fingerprint.unit.spec.ts
Normal file
@@ -0,0 +1,159 @@
|
||||
import {
|
||||
mkdirSync,
|
||||
mkdtempSync,
|
||||
rmSync,
|
||||
symlinkSync,
|
||||
writeFileSync,
|
||||
} from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
|
||||
import {
|
||||
createSourceFingerprint,
|
||||
createTypeGeneratorFingerprint,
|
||||
} from "./source-fingerprint";
|
||||
|
||||
function writeRepoFile(root: string, path: string, contents: string): void {
|
||||
const absolutePath = join(root, path);
|
||||
mkdirSync(join(absolutePath, ".."), { recursive: true });
|
||||
writeFileSync(absolutePath, contents);
|
||||
}
|
||||
|
||||
function createRepo(): string {
|
||||
const root = mkdtempSync(join(tmpdir(), "source-fingerprint-"));
|
||||
writeRepoFile(root, "src/metabase/core.clj", "(ns metabase.core)");
|
||||
writeRepoFile(
|
||||
root,
|
||||
"enterprise/backend/src/metabase_enterprise/core.clj",
|
||||
"(ns metabase-enterprise.core)",
|
||||
);
|
||||
writeRepoFile(root, "deps.edn", "{:deps {}}");
|
||||
return root;
|
||||
}
|
||||
|
||||
describe("createSourceFingerprint", () => {
|
||||
let root: string;
|
||||
|
||||
beforeEach(() => {
|
||||
root = createRepo();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
rmSync(root, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it("is deterministic across calls", () => {
|
||||
expect(createSourceFingerprint(root)).toBe(createSourceFingerprint(root));
|
||||
expect(createSourceFingerprint(root)).toMatch(/^sha256:[a-f0-9]{64}$/);
|
||||
});
|
||||
|
||||
it("changes when a source file's content changes", () => {
|
||||
const before = createSourceFingerprint(root);
|
||||
writeRepoFile(root, "src/metabase/core.clj", "(ns metabase.core) ;; edit");
|
||||
expect(createSourceFingerprint(root)).not.toBe(before);
|
||||
});
|
||||
|
||||
it("changes when an enterprise source file is added", () => {
|
||||
const before = createSourceFingerprint(root);
|
||||
writeRepoFile(
|
||||
root,
|
||||
"enterprise/backend/src/metabase_enterprise/new.clj",
|
||||
"(ns metabase-enterprise.new)",
|
||||
);
|
||||
expect(createSourceFingerprint(root)).not.toBe(before);
|
||||
});
|
||||
|
||||
it("changes when the root deps.edn changes", () => {
|
||||
const before = createSourceFingerprint(root);
|
||||
writeRepoFile(root, "deps.edn", "{:deps {:extra true}}");
|
||||
expect(createSourceFingerprint(root)).not.toBe(before);
|
||||
});
|
||||
|
||||
it("includes deps.edn files from module and bin roots only", () => {
|
||||
const before = createSourceFingerprint(root);
|
||||
writeRepoFile(root, "modules/drivers/redshift/deps.edn", "{:deps {}}");
|
||||
const withDriver = createSourceFingerprint(root);
|
||||
expect(withDriver).not.toBe(before);
|
||||
|
||||
writeRepoFile(root, "bin/lint-migrations-file/deps.edn", "{:deps {}}");
|
||||
const withBin = createSourceFingerprint(root);
|
||||
expect(withBin).not.toBe(withDriver);
|
||||
|
||||
// deps.edn outside the known roots is intentionally not fingerprinted.
|
||||
writeRepoFile(root, "docs/example/deps.edn", "{:deps {}}");
|
||||
expect(createSourceFingerprint(root)).toBe(withBin);
|
||||
});
|
||||
|
||||
it("ignores files outside the fingerprinted roots", () => {
|
||||
const before = createSourceFingerprint(root);
|
||||
writeRepoFile(root, "frontend/src/app.tsx", "export {};");
|
||||
writeRepoFile(root, "docs/readme.md", "# docs");
|
||||
expect(createSourceFingerprint(root)).toBe(before);
|
||||
});
|
||||
|
||||
it("distinguishes optional resource files when present", () => {
|
||||
const before = createSourceFingerprint(root);
|
||||
writeRepoFile(root, "resources/version.properties", "tag=v1");
|
||||
const withResource = createSourceFingerprint(root);
|
||||
expect(withResource).not.toBe(before);
|
||||
writeRepoFile(root, "resources/version.properties", "tag=v2");
|
||||
expect(createSourceFingerprint(root)).not.toBe(withResource);
|
||||
});
|
||||
|
||||
it("hashes symlinks by their target path", () => {
|
||||
const before = createSourceFingerprint(root);
|
||||
symlinkSync("core.clj", join(root, "src/metabase/link.clj"));
|
||||
expect(createSourceFingerprint(root)).not.toBe(before);
|
||||
});
|
||||
|
||||
it("does not throw when a listed file disappears before it is read", () => {
|
||||
// A dangling symlink stats (lstat) fine but cannot be read as a file;
|
||||
// regular reads race the same way when files are deleted mid-walk.
|
||||
symlinkSync("missing-target.clj", join(root, "src/metabase/dangling.clj"));
|
||||
expect(() => createSourceFingerprint(root)).not.toThrow();
|
||||
});
|
||||
});
|
||||
|
||||
describe("createTypeGeneratorFingerprint", () => {
|
||||
let root: string;
|
||||
|
||||
beforeEach(() => {
|
||||
root = createRepo();
|
||||
writeRepoFile(
|
||||
root,
|
||||
"frontend/build/openapi/openapi-ts.config.ts",
|
||||
"export default {};",
|
||||
);
|
||||
writeRepoFile(root, "package.json", "{}");
|
||||
writeRepoFile(root, "bun.lock", "{}");
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
rmSync(root, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it("is deterministic and distinct from the source fingerprint", () => {
|
||||
expect(createTypeGeneratorFingerprint(root)).toBe(
|
||||
createTypeGeneratorFingerprint(root),
|
||||
);
|
||||
expect(createTypeGeneratorFingerprint(root)).not.toBe(
|
||||
createSourceFingerprint(root),
|
||||
);
|
||||
});
|
||||
|
||||
it("changes when the generator config changes", () => {
|
||||
const before = createTypeGeneratorFingerprint(root);
|
||||
writeRepoFile(
|
||||
root,
|
||||
"frontend/build/openapi/openapi-ts.config.ts",
|
||||
"export default { changed: true };",
|
||||
);
|
||||
expect(createTypeGeneratorFingerprint(root)).not.toBe(before);
|
||||
});
|
||||
|
||||
it("ignores backend source changes", () => {
|
||||
const before = createTypeGeneratorFingerprint(root);
|
||||
writeRepoFile(root, "src/metabase/core.clj", "(ns metabase.core) ;; edit");
|
||||
expect(createTypeGeneratorFingerprint(root)).toBe(before);
|
||||
});
|
||||
});
|
||||
21
package.json
21
package.json
@@ -183,13 +183,13 @@
|
||||
"@esbuild-plugins/node-modules-polyfill": "^0.2.2",
|
||||
"@eslint/compat": "2.0.2",
|
||||
"@eslint/js": "9.39.2",
|
||||
"@hey-api/openapi-ts": "0.97.3",
|
||||
"@hey-api/openapi-ts": "0.99.0",
|
||||
"@inquirer/prompts": "^5.3.8",
|
||||
"@lezer/common": "^1.5.0",
|
||||
"@lezer/generator": "^1.7.2",
|
||||
"@loki/create-async-callback": "^0.35.0",
|
||||
"@microsoft/api-extractor": "^7.58.7",
|
||||
"@redocly/cli": "^2.14.3",
|
||||
"@redocly/cli": "^2.40.0",
|
||||
"@rspack/cli": "^2.0.3",
|
||||
"@rspack/core": "^2.0.3",
|
||||
"@rspack/dev-server": "^2.0.1",
|
||||
@@ -239,6 +239,7 @@
|
||||
"@types/mustache": "^4.1.2",
|
||||
"@types/postcss-import": "^14.0.3",
|
||||
"@types/postcss-url": "^10.0.4",
|
||||
"@types/proper-lockfile": "4.1.4",
|
||||
"@types/react": "^18.3.3",
|
||||
"@types/react-color": "^2.17.6",
|
||||
"@types/react-dom": "^18.2.17",
|
||||
@@ -255,7 +256,6 @@
|
||||
"babel-loader": "^9.1.2",
|
||||
"babel-plugin-ttag": "^1.8.16",
|
||||
"chalk": "^4",
|
||||
"chokidar-cli": "^3.0.0",
|
||||
"chromatic": "^10.2.0",
|
||||
"commander": "^12.1.0",
|
||||
"compression-rspack-plugin": "^0.2.0",
|
||||
@@ -342,6 +342,7 @@
|
||||
"postcss-preset-mantine": "^1.18.0",
|
||||
"postcss-styled-syntax": "^0.7.1",
|
||||
"postcss-url": "^10.1.3",
|
||||
"proper-lockfile": "4.1.2",
|
||||
"react-refresh": "^0.14.2",
|
||||
"react-router": "7.18.2",
|
||||
"reg-cli": "^0.17.7",
|
||||
@@ -455,7 +456,8 @@
|
||||
"embedding-sdk:publish": "cd ./resources/embedding-sdk && npm publish",
|
||||
"embedding-sdk:generate-package-support-files": "tsx enterprise/frontend/src/embedding-sdk-package/bin/generate-package-support-files.ts",
|
||||
"embedding-sdk:test-unit": "bun run test-unit enterprise/frontend/src/embedding-sdk-package/ enterprise/frontend/src/embedding-sdk-bundle/ enterprise/frontend/src/embedding-sdk-shared/",
|
||||
"embedding-sdk:dts:generate": "bun run embedding-sdk:clean:tsbuildinfo && ./node_modules/typescript7/bin/tsc --project tsconfig.sdk.json || true && bun run embedding-sdk:dts:fixup",
|
||||
"embedding-sdk:dts:generate": "bun run types:ensure && bun run embedding-sdk:dts:generate:pure",
|
||||
"embedding-sdk:dts:generate:pure": "sh -c 'status=0; bun run embedding-sdk:clean:tsbuildinfo || status=$?; if [ \"$status\" -eq 0 ]; then ./node_modules/typescript7/bin/tsc --project tsconfig.sdk.json || status=$?; fi; bun run embedding-sdk:dts:fixup || { fixup_status=$?; if [ \"$status\" -eq 0 ]; then status=$fixup_status; fi; }; exit \"$status\"'",
|
||||
"embedding-sdk:dts:fixup": "node ./bin/embedding-sdk/fixup-types-after-compilation.js",
|
||||
"embedding-sdk:docs:generate:pure": "typedoc --options ./.typedoc/typedoc.html.config.js && typedoc --options ./.typedoc/typedoc.markdown.config.js",
|
||||
"embedding-sdk:docs:generate": "bun run build-release:embedding-sdk-package && bun run embedding-sdk:dts:generate && bun run embedding-sdk:docs:generate:pure",
|
||||
@@ -467,8 +469,8 @@
|
||||
"eslint-fix": "bun run lint-eslint --fix",
|
||||
"find-never-defined-css-variables": "tsx bin/find-never-defined-css-variables/find-never-defined-css-variables.ts",
|
||||
"generate-cypress-html-report": "mochawesome-merge cypress/reports/mochareports/*.json > cypress/reports/cypress-test-report.json && marge cypress/reports/cypress-test-report.json -o cypress/reports --inline",
|
||||
"generate-openapi": "clojure -M:run:ee generate-openapi-spec",
|
||||
"lint-openapi": "redocly lint resources/openapi/openapi.json --skip-rule security-defined --skip-rule operation-operationId --skip-rule no-unused-components --skip-rule info-license --skip-rule no-path-trailing-slash --skip-rule no-identical-paths --skip-rule no-ambiguous-paths --skip-rule path-parameters-defined --skip-rule spec-components-invalid-map-name",
|
||||
"openapi:generate": "clojure -M:run:ee generate-openapi-spec",
|
||||
"openapi:lint": "redocly lint .tmp/openapi/openapi.json --skip-rule security-defined --skip-rule operation-operationId --skip-rule no-unused-components --skip-rule info-license --skip-rule no-path-trailing-slash --skip-rule no-identical-paths --skip-rule no-ambiguous-paths --skip-rule path-parameters-defined --skip-rule spec-components-invalid-map-name",
|
||||
"lint": "bun run lint-eslint && bun run lint-format && bun run lint-docs-links && bun run lint-yaml && bun run type-check",
|
||||
"lint-docs-links": "bun install && ./bin/verify-doc-links",
|
||||
"lint-css": "stylelint './frontend/**/*.css'",
|
||||
@@ -478,7 +480,7 @@
|
||||
"lint-format": "bun install && bun lint-format-pure",
|
||||
"lint-format-pure": "oxfmt --check '{frontend,enterprise/frontend,e2e}/**/*.{js,jsx,ts,tsx,css}'",
|
||||
"lint-yaml": "yamllint **/*.{yaml,yml} --ignore=node_modules/**/*.{yaml,yml}",
|
||||
"postinstall": "if [ -z \"$CI\" ]; then patch-package; fi",
|
||||
"postinstall": "if [ -z \"$CI\" ]; then patch-package && bun run types:ensure --postinstall; fi",
|
||||
"precommit": "lint-staged",
|
||||
"preinstall": "node -e \"if(!/bun/.test(process.env.npm_execpath || '')) { console.error('Please use Bun, not npm or yarn.'); process.exit(1); }\"",
|
||||
"prepare": "husky",
|
||||
@@ -505,9 +507,8 @@
|
||||
"test-visual:loki-report": "node frontend/test/generate-loki-report-json.js && reg-cli --from .loki/report.json --report .loki/report.html",
|
||||
"test-visual:loki-report-open": "bun run test-visual:loki || (echo 'Visual test failed, opening report...' && bun run test-visual:loki-report && open-cli .loki/report.html)",
|
||||
"type-check": "bun run clean:cljs && bun run build:cljs && bun run type-check-pure",
|
||||
"type-check-pure": "./node_modules/typescript7/bin/tsc --noEmit",
|
||||
"types:generate": "openapi-ts -f ./frontend/build/openapi/openapi-ts.config.ts",
|
||||
"types:generate:watch": "npx chokidar 'resources/openapi/openapi.json' --initial -c 'openapi-ts -f ./frontend/build/openapi/openapi-ts.config.ts' --debounce 1000",
|
||||
"type-check-pure": "bun run types:ensure && ./node_modules/typescript7/bin/tsc --noEmit",
|
||||
"types:ensure": "bun frontend/build/openapi/ensure-types.ts",
|
||||
"validate-e2e-filenames": "node e2e/validate-e2e-test-files.js",
|
||||
"wait:cljs": "echo Waiting for first CLJS build; until [ -f target/cljs_dev/cljs.core.js ]; do sleep 1; done; echo CLJS build done",
|
||||
"generate:expression-parser": "lezer-generator --typeScript frontend/src/metabase/querying/expressions/tokenizer/grammar.lezer -o frontend/src/metabase/querying/expressions/tokenizer/lezer.ts",
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -11,13 +11,16 @@
|
||||
[metabase.util.json :as json]
|
||||
[metabase.util.log :as log]
|
||||
[ring.middleware.content-type :as content-type]
|
||||
[ring.util.response :as response]))
|
||||
[ring.util.response :as response])
|
||||
(:import
|
||||
(java.io File)
|
||||
(java.nio.file CopyOption Files StandardCopyOption)))
|
||||
|
||||
(set! *warn-on-reflection* true)
|
||||
|
||||
(def openapi-file-path
|
||||
"Path to the local OpenAPI specification file."
|
||||
"resources/openapi/openapi.json")
|
||||
"Path to the generated OpenAPI specification."
|
||||
".tmp/openapi/openapi.json")
|
||||
|
||||
(defn- sort-keys
|
||||
"Sort maps and sets alphabetically to reduce diff noise on openapi.json"
|
||||
@@ -30,23 +33,83 @@
|
||||
:else x))
|
||||
data))
|
||||
|
||||
(def ^:private non-public-path-prefixes
|
||||
["/api/testing" "/api/dev"])
|
||||
|
||||
(defn- public-path?
|
||||
[path]
|
||||
(not-any? #(str/starts-with? path %) non-public-path-prefixes))
|
||||
|
||||
(defn- schema-refs
|
||||
"Set of `#/components/schemas/*` names referenced anywhere inside `x`."
|
||||
[x]
|
||||
(let [names (volatile! #{})]
|
||||
(walk/postwalk
|
||||
(fn [node]
|
||||
(when-let [ref (when (map? node) (:$ref node))]
|
||||
(when-let [[_ schema-name] (re-matches #"#/components/schemas/(.+)" ref)]
|
||||
(vswap! names conj schema-name)))
|
||||
node)
|
||||
x)
|
||||
@names))
|
||||
|
||||
(defn- reachable-schemas
|
||||
"Transitive closure of the schema names reachable from `roots`."
|
||||
[schemas roots]
|
||||
(loop [seen #{}, queue (vec roots)]
|
||||
(if-let [schema-name (peek queue)]
|
||||
(if (or (contains? seen schema-name) (not (contains? schemas schema-name)))
|
||||
(recur seen (pop queue))
|
||||
(recur (conj seen schema-name)
|
||||
(into (pop queue) (schema-refs (get schemas schema-name)))))
|
||||
seen)))
|
||||
|
||||
(defn open-api-object
|
||||
"The OpenAPI document describing the public API of `root-handler`.
|
||||
|
||||
Normalizes [[metabase.api.open-api/root-open-api-object]]: environment-dependent routes are dropped and unreachable schemas pruned.
|
||||
Callers add `:info`/`:servers`."
|
||||
[root-handler]
|
||||
(let [spec (open-api/root-open-api-object root-handler)
|
||||
paths (into {} (filter (comp public-path? key)) (:paths spec))
|
||||
schemas (get-in spec [:components :schemas])]
|
||||
(-> spec
|
||||
(assoc :paths paths)
|
||||
(assoc-in [:components :schemas]
|
||||
(select-keys schemas (reachable-schemas schemas (schema-refs paths)))))))
|
||||
|
||||
(defn- move-file-atomically!
|
||||
[^File source ^File destination]
|
||||
(Files/move (.toPath source)
|
||||
(.toPath destination)
|
||||
(into-array CopyOption [StandardCopyOption/ATOMIC_MOVE
|
||||
StandardCopyOption/REPLACE_EXISTING])))
|
||||
|
||||
(defn- temporary-sibling
|
||||
^File [^File destination]
|
||||
(File/createTempFile (str "." (.getName destination) ".")
|
||||
".tmp"
|
||||
(.getParentFile destination)))
|
||||
|
||||
(defn write-openapi-spec-to-file!
|
||||
"Generate and write the OpenAPI specification to a local file.
|
||||
Takes the root handler and generates the complete OpenAPI spec, writing it to [[openapi-file-path]]."
|
||||
[root-handler]
|
||||
(try
|
||||
(let [spec (merge
|
||||
(open-api/root-open-api-object root-handler)
|
||||
{:servers [{:url ""
|
||||
:description "Metabase API"}]})
|
||||
file (io/file openapi-file-path)]
|
||||
;; Create parent directory if it doesn't exist
|
||||
(when-let [parent-dir (.getParentFile file)]
|
||||
(.mkdirs parent-dir))
|
||||
(json/encode-to (sort-keys spec) (io/writer file) {:pretty true})
|
||||
(log/info "OpenAPI specification written to" openapi-file-path))
|
||||
(catch Throwable e
|
||||
(log/errorf "Failed to write OpenAPI specification to file: %s" (ex-message e)))))
|
||||
(let [spec (merge
|
||||
(open-api-object root-handler)
|
||||
{:servers [{:url ""
|
||||
:description "Metabase API"}]})
|
||||
^File file (.getAbsoluteFile (io/file openapi-file-path))
|
||||
^File parent-dir (.getParentFile file)]
|
||||
(.mkdirs parent-dir)
|
||||
(let [^File temp-file (temporary-sibling file)]
|
||||
(try
|
||||
(with-open [writer (io/writer temp-file)]
|
||||
(json/encode-to (sort-keys spec) writer {:pretty true}))
|
||||
(move-file-atomically! temp-file file)
|
||||
(finally
|
||||
(Files/deleteIfExists (.toPath temp-file)))))
|
||||
(log/info "OpenAPI specification written to" openapi-file-path)))
|
||||
|
||||
(defonce ^:private openapi-regen-state
|
||||
(atom {:executing? false
|
||||
@@ -125,7 +188,7 @@
|
||||
(do
|
||||
(log/warn "OpenAPI spec file not found, generating on-the-fly")
|
||||
(merge
|
||||
(open-api/root-open-api-object root-handler)
|
||||
(open-api-object root-handler)
|
||||
{:servers [{:url ""
|
||||
:description "Metabase API"}]})))]
|
||||
{:status 200
|
||||
|
||||
@@ -143,7 +143,10 @@
|
||||
|
||||
(mu/defn- schema->params* :- [:sequential :metabase.api.open-api/parameter]
|
||||
[schema in-fn renames]
|
||||
(let [{:keys [properties required]} (mjs-collect-definitions schema)
|
||||
;; Resolve a top-level registry reference to expose its map entries as OpenAPI parameters. The resolved map
|
||||
;; retains child references, which are emitted as shared schemas under `#/components/schemas/`.
|
||||
(let [schema (-> schema mc/schema mc/deref-all)
|
||||
{:keys [properties required]} (mjs-collect-definitions schema)
|
||||
required (set required)]
|
||||
(for [[k param-schema] properties
|
||||
:let [k (get renames k k)]
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
[metabase.api-routes.routes :as routes]
|
||||
[metabase.api.docs :as api.docs]))
|
||||
|
||||
;; used for `bun run generate-openapi` command
|
||||
;; used for `bun run openapi:generate` command
|
||||
#_{:clj-kondo/ignore [:clojure-lsp/unused-public-var]}
|
||||
(defn generate-openapi-spec!
|
||||
"Command to generate openapi spec for api routes."
|
||||
|
||||
@@ -147,7 +147,7 @@
|
||||
((resolve 'metabase.cmd.endpoint-dox/generate-dox!)))
|
||||
|
||||
(defn ^:command generate-openapi-spec
|
||||
"Generate OpenAPI specification file from Malli schema definitions. This is written to `resources/openapi/openapi.json`."
|
||||
"Generate OpenAPI specification file from Malli schema definitions. This is written to `.tmp/openapi/openapi.json`."
|
||||
[]
|
||||
(classloader/require 'metabase.api-routes.cmd)
|
||||
((resolve 'metabase.api-routes.cmd/generate-openapi-spec!)))
|
||||
|
||||
@@ -183,8 +183,8 @@
|
||||
:error/fn (fn [{:keys [value]} _]
|
||||
(str "invalid timezone ID: " (pr-str value)))
|
||||
:description "A valid timezone ID like: \"Asia/Aden\", \"America/Cuiaba\"."
|
||||
;; The timezone list is dynamic which make the .github/workflows/openapi-check.yml flaky on CI
|
||||
;; so we need to hack this to write a static schema
|
||||
;; The timezone list is dynamic, so we need to hack this to write a static schema
|
||||
;; so that the generated OpenAPI spec stays deterministic
|
||||
:json-schema {:type "string"}}]
|
||||
(sort (ZoneId/getAvailableZoneIds)))
|
||||
:cljs [:fn
|
||||
|
||||
12
test/metabase/api/docs_test.clj
Normal file
12
test/metabase/api/docs_test.clj
Normal file
@@ -0,0 +1,12 @@
|
||||
(ns metabase.api.docs-test
|
||||
(:require
|
||||
[clojure.test :refer :all]
|
||||
[metabase.api.docs :as api.docs]
|
||||
[metabase.test :as mt]))
|
||||
|
||||
(deftest write-openapi-spec-to-file-propagates-errors-test
|
||||
(mt/with-temp-dir [directory "openapi-output"]
|
||||
(with-redefs [api.docs/openapi-file-path directory
|
||||
api.docs/open-api-object (constantly {})]
|
||||
(is (thrown? java.io.IOException
|
||||
(api.docs/write-openapi-spec-to-file! identity))))))
|
||||
@@ -58,6 +58,21 @@
|
||||
(testing "the default value is still rendered in the param schema"
|
||||
(is (= :api (get-in by-name ["defaulted-param" :schema :default]))))))))
|
||||
|
||||
(mr/def ::named-query-params
|
||||
[:map
|
||||
[:database-id ms/PositiveInt]
|
||||
[:table-ids {:optional true} [:maybe [:sequential ms/PositiveInt]]]])
|
||||
|
||||
(deftest ^:parallel named-params-schema-test
|
||||
(testing "a registry-reference (mr/def) params schema contributes its parameters, same as an inline [:map ...]"
|
||||
(binding [defendpoint.open-api/*definitions* (atom (sorted-map))]
|
||||
(let [params (#'defendpoint.open-api/schema->params* ::named-query-params (constantly :query) {})]
|
||||
(is (= ["database-id" "table-ids"]
|
||||
(map :name params)))
|
||||
(is (=? [{:in :query, :required true}
|
||||
{:in :query, :required false}]
|
||||
params))))))
|
||||
|
||||
(deftest ^:parallel collect-definitions-test
|
||||
(binding [defendpoint.open-api/*definitions* (atom [])]
|
||||
(is (=? {:properties {:value {:$ref "#/components/schemas/metabase.lib.schema.common.non-blank-string"}}}
|
||||
|
||||
Reference in New Issue
Block a user