Data Apps Permissions (#79822)
* add saved question extraction prototype * fix additional failure cases * refactor: reconciler * revert me: manual skill file edit for source swapping * set data permissions to view for databases referenced in the query * rename data app collection and group * prevent agents from removing savedQuestionSourceId * docs: clarify data app query location * chore: cleanup query-sync canonical module * refactor: split query tests into different files * Swap query source for data app query hooks and apply dynamic filters on top of it (#79584) * code improvements and bug-fixes for data apps permissions * Add data apps model imports * address reviews on data apps permissions * run sync queries before vite build * Add data app resource link (#79878) add button to view data app resources * Show permission error when users cannot access a data app (#79874) show permission error page when user do not have permission to view data apps * Add data app group link in overflow menu (#79873) Add data app group link * Store and reconcile data app entity ids in data_app.yaml (#79833) * Store data app resource entity IDs * make entity ids in data_app.yaml authoritative * add top-level value after path key * simplify data_app.yaml manifest parser * only allow changing data app collection if empty * consolidate backend test helpers and remove redundant tests * chore: restore README changes * ci: fix failing tests * Add support of actions to Data App reconciler (#79842) Add support of actions to Data App reconciler * Fix bugs/inconsistency and add more e2e for data apps reconciler/syncing (#80002) * Update semantic schema skill about preventing segments and measures in dynamic parts of a query in Data Apps (#80288) * Grant view data access on a table-by-table basis in data app groups (#80189) * grant view data on table instead of database level * refactor table scan in reconciler * update e2e tests for permission grants * Grant data app view permissions per table * Skip unchanged data app permissions * add e2e data app test for sandboxing * reset host app sources after runs * Preserve sandboxing for data app users (#80406) * add e2e data app test for sandboxing * improve e2e tests for data apps * Remove sandbox E2E from table permissions change * preserve existing sandboxing from sandboxed groups in data apps * assert that we grant non-sandboxed tables * Explain data app sandbox exclusion * Add data app sandbox E2E coverage * Exercise sandbox preservation in data app E2E * Disable saved question sources in data apps (#80139) * Disable saved question sources in data apps * Preserve question-only schema scope * Remove the saved question schema parameter * Copy table-sourced metrics into data app collection and do not codegen card-sourced metrics (#80086) * copy table-sourced metrics into the data app collections * copy table-sourced metrics into the data app collections * Address metric sync review feedback * Accept keyword metric display in query response * Fix data app metric reconciliation ordering * Remove old lockfile format * exclude card-sourced metrics from schema generator * do not use card terms in metrics to avoid ambiguity with card-sourced metrics * fix model imports * Fix data app metric reconciliation * refactor: cleanup reconciler unit tests * Fix duplicated metric client helper * add safe metrics path * Handle unsupported metric dependencies * Fix data app CI test regressions * Fix "Re-adding removed query fails to adopt deleted card" (#80341) * Clean-up data apps permissions endpoints (#80825) * Remove entity ids from data_app.yaml (#81317) Remove data app entity ids from data_app.yaml * Adjust data apps permissions code (#81093) * Remove the data app table permission grants (#81584) * Add permissions_group.is_data_app_group and hide data app permission groups from UI (#81705) * Add data app user management and access warnings (#81839) * Address data app user access review feedback * additional e2e test cases for bugs in manage users page * fix failing frontend type check on unit test * fix migration id collision for data apps * combine migration changes into one file * fix data app metric sync after trashing copied metrics * fix cli runtime dependency packaging for embedding sdk * avoid loading data app bundle during resource setup * prevent redundant data app collection permission updates * allow admins to remove data app members after token expires * update docs and add e2e test for missing resource collection case * isolate data app sync using transactions * do not leave user on an empty page when removing the last member on the page * reserve sandbox-host slug in data app config * ignore generated data app action fixtures * move queries into db.clj * preserve legacy permissions when adding data app members * regenerate the kondo modules config The data-apps module reaches for :model/Permissions and :model/DataPermissions, neither of which was recorded in :model-imports, so metabase.core.modules-test fails on the current tree. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011nNKkhQjjiUSM7qwLReyfP * reject data app queries whose metrics read another card Sync copies a referenced metric but rewrites nothing inside the copy, and copies nothing that copy in turn reads, so those references still point at the originals. The app publishes successfully and then fails for every viewer without access to the originals' collections. Checking while resolving the query, rather than at codegen, also catches a metric edited into that shape after its schema was generated. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011nNKkhQjjiUSM7qwLReyfP * exclude metrics that read a saved card anywhere from generated schemas source-card-id only reads stage 0, so a table-sourced metric that joins a saved question passed the codegen filter. The CLI checks the whole query and rejects such a metric, aborting sync-resources for any app that uses it. Checking the whole query here keeps codegen consistent with what sync supports, and removes the database round trip that asked which of the referenced cards were metrics -- the step that let a saved-question join through in the first place. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011nNKkhQjjiUSM7qwLReyfP * resolve copied resources' table dependencies through the server Numeric source-table entries miss implicit joins: a copied model querying Orders and reading Products.Category through a foreign key names Products nowhere, so it was left out of the app's permission warnings. Run the same metadata-based lookup the query endpoint uses, through a new endpoint the CLI calls for copied models, actions and metrics, and drop the JSON walk it replaces. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011nNKkhQjjiUSM7qwLReyfP * cover the create-queries grant the database defaults leave on app groups An app group that predates the first non-audit database is given create-queries :query-builder-and-native by set-new-database-permissions!, alongside the :blocked view-data it special-cases for app groups. A user with data access through another group would gain native query authoring just by joining the app. The reconciler already resets it, but nothing failed when :perms/create-queries was dropped from its list. The pairing cannot be produced through set-database-permission!, which cascades view-data to :unrestricted for any create-queries value other than :no, so the test writes the row the way the defaults insert it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011nNKkhQjjiUSM7qwLReyfP * prevent models based on saved questions from being copied * return to a valid member page after overlapping removals * do not resolve query objects when either enabled flag is false * prevent shorthand properties and spreads from overriding generated ids * prevent adding duplicate members and retry only members that are failed to add * define schema for data app query endpoint * reconcile only the current data app group during resource setup * expand the schema fingerprint * add data app group field in permission group schemas * prevent new databases from granting query creation to data app groups * throw validation errors when invalid query spec fields are passed * do not fail validation on columns metadata * add enabled metadata to query definition schema * accept more data app specific query metadata and binning options --------- Co-authored-by: Timofey Kachalov <timofei@metabase.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -3273,15 +3273,28 @@
|
||||
:uses #{premium-features}}
|
||||
|
||||
enterprise/data-apps
|
||||
{:team "Gadget"
|
||||
{:team "Embedding"
|
||||
:api #{metabase-enterprise.data-apps.api
|
||||
metabase-enterprise.data-apps.init
|
||||
metabase-enterprise.data-apps.sync}
|
||||
:uses #{api
|
||||
collections
|
||||
lib
|
||||
lib-be
|
||||
models
|
||||
permissions
|
||||
premium-features
|
||||
request
|
||||
settings
|
||||
util}}
|
||||
util}
|
||||
:model-imports #{:model/Card
|
||||
:model/Collection
|
||||
:model/DataPermissions
|
||||
:model/Database
|
||||
:model/Permissions
|
||||
:model/PermissionsGroup
|
||||
:model/Table
|
||||
:model/User}}
|
||||
|
||||
enterprise/data-complexity-score
|
||||
{:team "Metabot"
|
||||
|
||||
@@ -424,15 +424,38 @@ jobs:
|
||||
if: steps.prebuilt-cljs.outcome != 'success'
|
||||
run: bun run build-pure:cljs
|
||||
|
||||
# The data-app suite synchronizes real actions, and Metabase only permits
|
||||
# those against a writable warehouse, which comes from the `qa-db`
|
||||
# snapshot below. That snapshot creator builds every QA database in one
|
||||
# pass — Mongo and MySQL included — so all three have to be up even though
|
||||
# only `postgres-writable` is restored. The other host apps read the sample
|
||||
# database, so they skip the pull entirely.
|
||||
- name: Prepare Docker containers
|
||||
if: ${{ matrix.test_suite == 'vite-6-data-app-host-app-e2e' }}
|
||||
uses: ./.github/actions/e2e-prepare-containers
|
||||
with:
|
||||
username: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
password: ${{ secrets.DOCKERHUB_TOKEN }}
|
||||
postgres: true
|
||||
mysql: true
|
||||
mongo: true
|
||||
|
||||
- name: Run Metabase
|
||||
run: node e2e/runner/run_cypress_ci.js start
|
||||
|
||||
- uses: ./.github/actions/resolve-job-id
|
||||
|
||||
# Without `--spec` the snapshot config picks up `qa-db.cy.snap.js` too,
|
||||
# which is what creates the `postgres-writable` snapshot the data-app
|
||||
# action tests restore.
|
||||
- name: Generate database snapshots
|
||||
run: |
|
||||
node e2e/runner/run_cypress_ci.js snapshot \
|
||||
--spec "e2e/snapshot-creators/default.cy.snap.js"
|
||||
if [ "${{ matrix.test_suite }}" = "vite-6-data-app-host-app-e2e" ]; then
|
||||
node e2e/runner/run_cypress_ci.js snapshot
|
||||
else
|
||||
node e2e/runner/run_cypress_ci.js snapshot \
|
||||
--spec "e2e/snapshot-creators/default.cy.snap.js"
|
||||
fi
|
||||
|
||||
- name: Prepare and launch Host App
|
||||
run: npx tsx ./e2e/runner/embedding-sdk/host-apps/start-ci.ts ${{ matrix.test_suite }}
|
||||
|
||||
@@ -24,6 +24,8 @@ jobs:
|
||||
# unrelated to embedding (EMB-765)
|
||||
echo "::remove-matcher owner=tsc::"
|
||||
bun run build-embedding-sdk-package
|
||||
- name: Smoke test the packed SDK CLI
|
||||
run: node bin/embedding-sdk/test-sdk-package.mjs
|
||||
- name: Prepare Embedding SDK artifact
|
||||
uses: actions/upload-artifact@v7
|
||||
with:
|
||||
|
||||
3
.github/workflows/release-embedding-sdk.yml
vendored
3
.github/workflows/release-embedding-sdk.yml
vendored
@@ -382,6 +382,9 @@ jobs:
|
||||
- name: Generate SDK package.json in the build directory
|
||||
run: bun run embedding-sdk:generate-package
|
||||
|
||||
- name: Smoke test the packed SDK CLI
|
||||
run: node bin/embedding-sdk/test-sdk-package.mjs
|
||||
|
||||
- name: Upload built SDK package as artifact
|
||||
uses: actions/upload-artifact@v7
|
||||
with:
|
||||
|
||||
@@ -1,23 +1,8 @@
|
||||
#!/usr/bin/env node
|
||||
/* eslint-env node */
|
||||
/* eslint-disable import/no-commonjs, import/order, no-console */
|
||||
const fs = require("fs");
|
||||
const path = require("path");
|
||||
|
||||
const SDK_DIST_DIR = path.resolve("./resources/embedding-sdk");
|
||||
const DEPENDENCIES = [];
|
||||
|
||||
function filterOuDependencies(object) {
|
||||
const result = {};
|
||||
|
||||
Object.entries(object).forEach(([packageName, version]) => {
|
||||
if (DEPENDENCIES.includes(packageName)) {
|
||||
result[packageName] = version;
|
||||
}
|
||||
});
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
function generateSdkPackage() {
|
||||
let maybeCommitHash = process.argv[2];
|
||||
@@ -27,13 +12,10 @@ function generateSdkPackage() {
|
||||
maybeCommitHash = maybeCommitHash.slice(0, 7);
|
||||
}
|
||||
|
||||
const mainPackageJson = fs.readFileSync(
|
||||
path.resolve("./package.json"),
|
||||
"utf-8",
|
||||
const mainPackageJsonContent = JSON.parse(
|
||||
fs.readFileSync(path.resolve("./package.json"), "utf-8"),
|
||||
);
|
||||
|
||||
const mainPackageJsonContent = JSON.parse(mainPackageJson);
|
||||
|
||||
const sdkPackageTemplateJson = fs.readFileSync(
|
||||
path.resolve(
|
||||
"./enterprise/frontend/src/embedding-sdk-package/package.template.json",
|
||||
@@ -52,10 +34,19 @@ function generateSdkPackage() {
|
||||
|
||||
const mergedContent = {
|
||||
...publishableTemplateJsonContent,
|
||||
|
||||
version: maybeCommitHash
|
||||
? `${sdkPackageTemplateJsonContent.version}-${todayDate}-${maybeCommitHash}`
|
||||
: sdkPackageTemplateJsonContent.version,
|
||||
dependencies: filterOuDependencies(mainPackageJsonContent.dependencies),
|
||||
|
||||
// Runtime dependencies of the CLI for data apps, not pre-bundled.
|
||||
dependencies: {
|
||||
// Compiles app definitions and local imports for the CLI to evaluate.
|
||||
esbuild: mainPackageJsonContent.devDependencies.esbuild,
|
||||
|
||||
// Parses and edits TypeScript source files during resource sync.
|
||||
typescript: mainPackageJsonContent.dependencies.typescript,
|
||||
},
|
||||
};
|
||||
|
||||
const mergedContentString = JSON.stringify(mergedContent, null, 2);
|
||||
|
||||
93
bin/embedding-sdk/test-sdk-package.mjs
Normal file
93
bin/embedding-sdk/test-sdk-package.mjs
Normal file
@@ -0,0 +1,93 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { execFileSync } from "node:child_process";
|
||||
import console from "node:console";
|
||||
import fs from "node:fs";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import process from "node:process";
|
||||
|
||||
const PACKAGE_PATH = path.resolve("resources/embedding-sdk");
|
||||
|
||||
const SMOKE_TEST_PATH = fs.mkdtempSync(
|
||||
path.join(os.tmpdir(), "sdk-package-smoke-"),
|
||||
);
|
||||
|
||||
const NPM_COMMAND_OPTIONS = {
|
||||
cwd: SMOKE_TEST_PATH,
|
||||
encoding: "utf8",
|
||||
timeout: 180_000,
|
||||
env: { ...process.env, NODE_PATH: "", NODE_OPTIONS: "" },
|
||||
};
|
||||
|
||||
try {
|
||||
fs.writeFileSync(
|
||||
path.join(SMOKE_TEST_PATH, "package.json"),
|
||||
JSON.stringify({ name: "sdk-package-smoke-test", private: true }),
|
||||
);
|
||||
|
||||
// Get tarball with the SDK package dependencies
|
||||
const [tarball] = JSON.parse(
|
||||
execFileSync(
|
||||
"npm",
|
||||
["pack", PACKAGE_PATH, "--json", "--ignore-scripts"],
|
||||
NPM_COMMAND_OPTIONS,
|
||||
),
|
||||
);
|
||||
|
||||
// Install the packed sdk and runtime dependencies
|
||||
execFileSync(
|
||||
"npm",
|
||||
[
|
||||
"install",
|
||||
tarball.filename,
|
||||
"--legacy-peer-deps",
|
||||
"--no-audit",
|
||||
"--no-fund",
|
||||
],
|
||||
NPM_COMMAND_OPTIONS,
|
||||
);
|
||||
|
||||
const installedPackage = path.join(
|
||||
SMOKE_TEST_PATH,
|
||||
"node_modules/@metabase/embedding-sdk-react",
|
||||
);
|
||||
|
||||
const manifest = JSON.parse(
|
||||
fs.readFileSync(path.join(installedPackage, "package.json"), "utf8"),
|
||||
);
|
||||
|
||||
// Check that both esbuild and typescript modules are installed and can be used
|
||||
execFileSync(
|
||||
process.execPath,
|
||||
[
|
||||
"--eval",
|
||||
`
|
||||
const { createRequire } = require("node:module");
|
||||
const sdkRequire = createRequire(${JSON.stringify(path.join(installedPackage, "package.json"))});
|
||||
|
||||
sdkRequire("esbuild").transformSync("const value: number = 1", { loader: "ts" });
|
||||
sdkRequire("typescript").transpileModule("const value: number = 1", {});
|
||||
`,
|
||||
],
|
||||
NPM_COMMAND_OPTIONS,
|
||||
);
|
||||
|
||||
// Check that the installed CLI has data apps sync command
|
||||
const output = execFileSync(
|
||||
process.execPath,
|
||||
[
|
||||
path.join(installedPackage, manifest.bin),
|
||||
"data-apps",
|
||||
"sync-resources",
|
||||
"--help",
|
||||
],
|
||||
NPM_COMMAND_OPTIONS,
|
||||
);
|
||||
|
||||
assert.match(output, /Usage: .*sync-resources/);
|
||||
assert.match(output, /--app-root/);
|
||||
|
||||
console.log("SDK package smoke test passed.");
|
||||
} finally {
|
||||
fs.rmSync(SMOKE_TEST_PATH, { recursive: true, force: true });
|
||||
}
|
||||
@@ -1,12 +1,12 @@
|
||||
{
|
||||
"name": "angular-20-host-app",
|
||||
"version": "0.0.0",
|
||||
"private": true,
|
||||
"scripts": {
|
||||
"start": "./entrypoint.sh",
|
||||
"preview": "ng serve --configuration production --watch false",
|
||||
"watch": "ng serve --configuration development --watch true"
|
||||
},
|
||||
"private": true,
|
||||
"dependencies": {
|
||||
"@angular/common": "^20.0.0",
|
||||
"@angular/compiler": "^20.0.0",
|
||||
|
||||
@@ -1,12 +1,12 @@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<title>Angular@20 Host App</title>
|
||||
<base href="/">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
</head>
|
||||
<body>
|
||||
<app-root></app-root>
|
||||
</body>
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<title>Angular@20 Host App</title>
|
||||
<base href="/" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||
</head>
|
||||
<body>
|
||||
<app-root></app-root>
|
||||
</body>
|
||||
</html>
|
||||
|
||||
@@ -1,33 +1,26 @@
|
||||
/* To learn more about this file see: https://angular.io/config/tsconfig. */
|
||||
{
|
||||
"compileOnSave": false,
|
||||
"compilerOptions": {
|
||||
"baseUrl": "./",
|
||||
"compileOnSave": false,
|
||||
"compilerOptions": {
|
||||
"baseUrl": "./",
|
||||
"outDir": "./out-tsc/app",
|
||||
"allowSyntheticDefaultImports": true,
|
||||
"downlevelIteration": true,
|
||||
"experimentalDecorators": true,
|
||||
"moduleResolution": "node",
|
||||
"allowSyntheticDefaultImports": true,
|
||||
"downlevelIteration": true,
|
||||
"experimentalDecorators": true,
|
||||
"moduleResolution": "node",
|
||||
"skipLibCheck": true,
|
||||
"target": "ES2022",
|
||||
"module": "ES2022",
|
||||
"useDefineForClassFields": false,
|
||||
"jsx": "react-jsx",
|
||||
"lib": [
|
||||
"ES2022",
|
||||
"dom"
|
||||
]
|
||||
},
|
||||
"angularCompilerOptions": {
|
||||
"enableI18nLegacyMessageIdFormat": false,
|
||||
"strictInjectionParameters": true,
|
||||
"strictInputAccessModifiers": true,
|
||||
"strictTemplates": true
|
||||
},
|
||||
"files": [
|
||||
"src/main.ts"
|
||||
],
|
||||
"include": [
|
||||
"src/**/*.d.ts"
|
||||
]
|
||||
"module": "ES2022",
|
||||
"useDefineForClassFields": false,
|
||||
"jsx": "react-jsx",
|
||||
"lib": ["ES2022", "dom"]
|
||||
},
|
||||
"angularCompilerOptions": {
|
||||
"enableI18nLegacyMessageIdFormat": false,
|
||||
"strictInjectionParameters": true,
|
||||
"strictInputAccessModifiers": true,
|
||||
"strictTemplates": true
|
||||
},
|
||||
"files": ["src/main.ts"],
|
||||
"include": ["src/**/*.d.ts"]
|
||||
}
|
||||
|
||||
@@ -1,10 +1,6 @@
|
||||
{
|
||||
"compilerOptions": {
|
||||
"lib": [
|
||||
"dom",
|
||||
"dom.iterable",
|
||||
"esnext"
|
||||
],
|
||||
"lib": ["dom", "dom.iterable", "esnext"],
|
||||
"allowJs": true,
|
||||
"skipLibCheck": true,
|
||||
"strict": true,
|
||||
@@ -22,22 +18,13 @@
|
||||
}
|
||||
],
|
||||
"paths": {
|
||||
"@/*": [
|
||||
"./src/*"
|
||||
],
|
||||
"@/*": ["./src/*"],
|
||||
"@metabase/embedding-sdk-react": [
|
||||
"./node_modules/@metabase/embedding-sdk-react"
|
||||
]
|
||||
},
|
||||
"target": "ES2017"
|
||||
},
|
||||
"include": [
|
||||
"next-env.d.ts",
|
||||
"**/*.ts",
|
||||
"**/*.tsx",
|
||||
".next/types/**/*.ts"
|
||||
],
|
||||
"exclude": [
|
||||
"node_modules"
|
||||
]
|
||||
"include": ["next-env.d.ts", "**/*.ts", "**/*.tsx", ".next/types/**/*.ts"],
|
||||
"exclude": ["node_modules"]
|
||||
}
|
||||
|
||||
@@ -1,10 +1,6 @@
|
||||
{
|
||||
"compilerOptions": {
|
||||
"lib": [
|
||||
"dom",
|
||||
"dom.iterable",
|
||||
"esnext"
|
||||
],
|
||||
"lib": ["dom", "dom.iterable", "esnext"],
|
||||
"allowJs": true,
|
||||
"skipLibCheck": true,
|
||||
"strict": true,
|
||||
@@ -22,22 +18,13 @@
|
||||
}
|
||||
],
|
||||
"paths": {
|
||||
"@/*": [
|
||||
"./src/*"
|
||||
],
|
||||
"@/*": ["./src/*"],
|
||||
"@metabase/embedding-sdk-react": [
|
||||
"./node_modules/@metabase/embedding-sdk-react"
|
||||
]
|
||||
},
|
||||
"target": "ES2017"
|
||||
},
|
||||
"include": [
|
||||
"next-env.d.ts",
|
||||
"**/*.ts",
|
||||
"**/*.tsx",
|
||||
".next/types/**/*.ts"
|
||||
],
|
||||
"exclude": [
|
||||
"node_modules"
|
||||
]
|
||||
"include": ["next-env.d.ts", "**/*.ts", "**/*.tsx", ".next/types/**/*.ts"],
|
||||
"exclude": ["node_modules"]
|
||||
}
|
||||
|
||||
@@ -10,3 +10,6 @@ build
|
||||
|
||||
# debug
|
||||
npm-debug.log*
|
||||
|
||||
# generated test action declarations
|
||||
/actions/
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"name": "vite-6-data-app-host-app",
|
||||
"private": true,
|
||||
"version": "0.0.0",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"start": "./entrypoint.sh",
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"name": "vite-6-host-app",
|
||||
"private": true,
|
||||
"version": "0.0.0",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"start": "./entrypoint.sh",
|
||||
|
||||
@@ -3,15 +3,10 @@ import {
|
||||
StaticQuestion,
|
||||
useAction,
|
||||
} from "@metabase/embedding-sdk-react";
|
||||
import type { MetabaseQueryOptions } from "@metabase/embedding-sdk-react/data-app";
|
||||
import {
|
||||
aggregations,
|
||||
breakout,
|
||||
copy,
|
||||
DataAppLink,
|
||||
DataAppRouter,
|
||||
filter,
|
||||
orderBy,
|
||||
useDataAppLocation,
|
||||
useMetabaseQuery,
|
||||
useMetabaseQueryObject,
|
||||
@@ -197,161 +192,15 @@ function NativeQueryPage() {
|
||||
);
|
||||
}
|
||||
|
||||
function Combinators() {
|
||||
const combinators = getTestEnv().combinators!;
|
||||
const countAgg = aggregations.count();
|
||||
|
||||
const q = useMetabaseQuery({
|
||||
source: combinators.source,
|
||||
filters: [filter(combinators.filterField, ">", combinators.filterValue)],
|
||||
aggregations: [countAgg],
|
||||
breakouts: [breakout(combinators.breakoutField)],
|
||||
orderBys: [orderBy(countAgg, "desc")],
|
||||
});
|
||||
|
||||
return (
|
||||
<div data-testid="data-app-combinators" style={{ padding: 24 }}>
|
||||
<h1>Combinators</h1>
|
||||
<div data-testid="combinators-loading">
|
||||
{q.isLoading ? "loading" : "done"}
|
||||
</div>
|
||||
<div data-testid="combinators-error">
|
||||
{q.error ? "error" : "no-error"}
|
||||
</div>
|
||||
<div data-testid="combinators-rowcount">
|
||||
{String(q.data?.rows?.length ?? 0)}
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* A saved question used as a query source, with clauses applied on top of it.
|
||||
* Each case runs the same clauses against the table and against a question that
|
||||
* copies it; agreeing results are the assertion, so the spec never has to encode
|
||||
* sample-database numbers.
|
||||
*/
|
||||
function CardSource() {
|
||||
const {
|
||||
source,
|
||||
tableSource,
|
||||
filterField,
|
||||
filterColumn,
|
||||
filterValue,
|
||||
breakoutField,
|
||||
breakoutColumn,
|
||||
} = getTestEnv().cardSource!;
|
||||
|
||||
const countAgg = aggregations.count();
|
||||
|
||||
return (
|
||||
<div data-testid="data-app-card-source" style={{ padding: 24 }}>
|
||||
<h1>Card source</h1>
|
||||
|
||||
<CardSourceCase
|
||||
caseKey="source"
|
||||
tableQuery={{ source: tableSource, aggregations: [countAgg] }}
|
||||
cardQuery={{ source, aggregations: [countAgg] }}
|
||||
totalTestId="card-source-total"
|
||||
/>
|
||||
|
||||
<CardSourceCase
|
||||
caseKey="filters"
|
||||
tableQuery={{
|
||||
source: tableSource,
|
||||
filters: [filter(filterField, ">", filterValue)],
|
||||
aggregations: [countAgg],
|
||||
}}
|
||||
cardQuery={{
|
||||
source,
|
||||
filters: [filter(filterColumn, ">", filterValue)],
|
||||
aggregations: [countAgg],
|
||||
}}
|
||||
/>
|
||||
|
||||
<CardSourceCase
|
||||
caseKey="aggregations"
|
||||
tableQuery={{
|
||||
source: tableSource,
|
||||
filters: [filter(filterField, ">", filterValue)],
|
||||
aggregations: [countAgg, aggregations.sum(filterField)],
|
||||
}}
|
||||
cardQuery={{
|
||||
source,
|
||||
filters: [filter(filterColumn, ">", filterValue)],
|
||||
aggregations: [countAgg, aggregations.sum(filterColumn)],
|
||||
}}
|
||||
/>
|
||||
|
||||
<CardSourceCase
|
||||
caseKey="breakouts"
|
||||
tableQuery={{
|
||||
source: tableSource,
|
||||
filters: [filter(filterField, ">", filterValue)],
|
||||
aggregations: [countAgg],
|
||||
breakouts: [breakout(breakoutField)],
|
||||
orderBys: [orderBy(breakoutField, "asc")],
|
||||
}}
|
||||
cardQuery={{
|
||||
source,
|
||||
filters: [filter(filterColumn, ">", filterValue)],
|
||||
aggregations: [countAgg],
|
||||
breakouts: [breakout(breakoutColumn)],
|
||||
orderBys: [orderBy(breakoutColumn, "asc")],
|
||||
}}
|
||||
/>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
function CardSourceCase({
|
||||
caseKey,
|
||||
tableQuery,
|
||||
cardQuery,
|
||||
totalTestId,
|
||||
}: {
|
||||
caseKey: string;
|
||||
tableQuery: MetabaseQueryOptions<undefined>;
|
||||
cardQuery: MetabaseQueryOptions<undefined>;
|
||||
totalTestId?: string;
|
||||
}) {
|
||||
const fromTable = useMetabaseQuery(tableQuery);
|
||||
const fromCard = useMetabaseQuery(cardQuery);
|
||||
|
||||
const failure = fromTable.error ?? fromCard.error;
|
||||
const tableResult = describeResult(fromTable.data);
|
||||
const cardResult = describeResult(fromCard.data);
|
||||
|
||||
const status = failure
|
||||
? `error: ${describeError(failure)}`
|
||||
: tableResult === null || cardResult === null
|
||||
? "loading"
|
||||
: tableResult === cardResult
|
||||
? "match"
|
||||
: `mismatch: table=${tableResult} card=${cardResult}`;
|
||||
|
||||
return (
|
||||
<div>
|
||||
<div data-testid={`card-source-case-${caseKey}`}>{status}</div>
|
||||
{totalTestId && (
|
||||
<div data-testid={totalTestId}>
|
||||
{String(fromCard.data?.rawRows?.[0]?.[0] ?? "")}
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
/** Column names plus every row value, by position — comparable across sources. */
|
||||
const describeResult = (
|
||||
data: { columns: { name: string }[]; rawRows: unknown[][] } | null,
|
||||
): string | null =>
|
||||
data &&
|
||||
`[${data.columns.map((column) => column.name).join(",")}] ${JSON.stringify(data.rawRows)}`;
|
||||
|
||||
function Actions() {
|
||||
const { actionId, actionParams } = getTestEnv();
|
||||
const action = useAction(actionId ?? null);
|
||||
// A data app names its actions by definition. Nothing synchronizes here, so
|
||||
// the copy the definition points at is the authored action itself.
|
||||
const action = useAction(
|
||||
actionId
|
||||
? { action: { id: actionId, parameters: [] }, copiedActionId: actionId }
|
||||
: null,
|
||||
);
|
||||
const [output, setOutput] = useState("idle");
|
||||
|
||||
const onExecute = async () => {
|
||||
@@ -576,8 +425,6 @@ const ROUTES: Record<string, ComponentType> = {
|
||||
"/native-query": NativeQueryPage,
|
||||
"/download-question": DownloadQuestionPage,
|
||||
"/custom-viz": CustomVizPage,
|
||||
"/combinators": Combinators,
|
||||
"/card-source": CardSource,
|
||||
"/actions": Actions,
|
||||
"/clipboard": Clipboard,
|
||||
"/missing-question": MissingQuestion,
|
||||
|
||||
@@ -0,0 +1,2 @@
|
||||
name: Synced Actions App
|
||||
path: ./dist/index.js
|
||||
39
e2e/support/assets/data-apps/synced-actions-app/src/App.tsx
Normal file
39
e2e/support/assets/data-apps/synced-actions-app/src/App.tsx
Normal file
@@ -0,0 +1,39 @@
|
||||
import { useAction } from "@metabase/embedding-sdk-react";
|
||||
import { useState } from "react";
|
||||
|
||||
import { CreateScore } from "../actions/orders.action";
|
||||
|
||||
/**
|
||||
* Passes a source-controlled definition to `useAction` rather than a bare ID, so
|
||||
* a production build executes whatever `sync-resources` wrote into it — the copy
|
||||
* the app's own group is permitted to run. The spec that drives this app writes
|
||||
* `actions/orders.action.ts` before synchronizing.
|
||||
*/
|
||||
export default function App() {
|
||||
const action = useAction(CreateScore);
|
||||
const [output, setOutput] = useState("idle");
|
||||
|
||||
const onExecute = async () => {
|
||||
try {
|
||||
await action.execute({ team_name: "Data App FC", score: 7 });
|
||||
setOutput("executed");
|
||||
} catch (error) {
|
||||
setOutput(`threw: ${String(error)}`);
|
||||
}
|
||||
};
|
||||
|
||||
return (
|
||||
<div data-testid="synced-actions-content" style={{ padding: 24 }}>
|
||||
<h1>Synced actions app</h1>
|
||||
|
||||
<button type="button" data-testid="action-execute" onClick={onExecute}>
|
||||
execute
|
||||
</button>
|
||||
|
||||
<div data-testid="action-output">{output}</div>
|
||||
<div data-testid="action-error">
|
||||
{action.error ? "has-error" : "no-error"}
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,7 @@
|
||||
import type { DataAppFactory } from "@metabase/embedding-sdk-react/data-app";
|
||||
|
||||
import App from "./App";
|
||||
|
||||
const factory: DataAppFactory = () => ({ component: App });
|
||||
|
||||
export default factory;
|
||||
2
e2e/support/assets/data-apps/synced-app/data_app.yaml
Normal file
2
e2e/support/assets/data-apps/synced-app/data_app.yaml
Normal file
@@ -0,0 +1,2 @@
|
||||
name: Synced App
|
||||
path: ./dist/index.js
|
||||
29
e2e/support/assets/data-apps/synced-app/src/App.tsx
Normal file
29
e2e/support/assets/data-apps/synced-app/src/App.tsx
Normal file
@@ -0,0 +1,29 @@
|
||||
import { useMetabaseQuery } from "@metabase/embedding-sdk-react/data-app";
|
||||
|
||||
import { OrdersCount } from "../queries/orders.query";
|
||||
|
||||
/**
|
||||
* Consumes a source-controlled query rather than an inline one, so a production
|
||||
* build runs whatever `sync-resources` wrote into the declaration. The spec that
|
||||
* drives this app writes `queries/orders.query.ts` before synchronizing.
|
||||
*/
|
||||
export default function App() {
|
||||
const orders = useMetabaseQuery(OrdersCount);
|
||||
const rows = orders.data?.rawRows;
|
||||
|
||||
const total = rows?.[0]?.[1];
|
||||
|
||||
return (
|
||||
<div data-testid="synced-app-content" style={{ padding: 24 }}>
|
||||
<h1>Synced app</h1>
|
||||
|
||||
{orders.error ? (
|
||||
<div data-testid="synced-app-error">{String(orders.error)}</div>
|
||||
) : (
|
||||
<div data-testid="synced-app-total">
|
||||
{total === undefined ? "" : String(total)}
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
7
e2e/support/assets/data-apps/synced-app/src/index.tsx
Normal file
7
e2e/support/assets/data-apps/synced-app/src/index.tsx
Normal file
@@ -0,0 +1,7 @@
|
||||
import type { DataAppFactory } from "@metabase/embedding-sdk-react/data-app";
|
||||
|
||||
import App from "./App";
|
||||
|
||||
const factory: DataAppFactory = () => ({ component: App });
|
||||
|
||||
export default factory;
|
||||
@@ -34,10 +34,16 @@ module.exports = {
|
||||
...RESOLVE_ALIASES,
|
||||
...(embeddingSdkDistPath
|
||||
? {
|
||||
// The package alias below points at a bundle file, so subpath
|
||||
// exports cannot resolve through it and each needs its own entry.
|
||||
[`${SDK_PACKAGE_NAME}/data-app-dev`]: path.join(
|
||||
embeddingSdkDistPath,
|
||||
"data-app-dev.js",
|
||||
),
|
||||
[`${SDK_PACKAGE_NAME}/data-app`]: path.join(
|
||||
embeddingSdkDistPath,
|
||||
"data-app.js",
|
||||
),
|
||||
}
|
||||
: null),
|
||||
...(embeddingSdkPath ? { [SDK_PACKAGE_NAME]: embeddingSdkPath } : null),
|
||||
|
||||
@@ -26,7 +26,14 @@ import {
|
||||
startCustomVizDevServer,
|
||||
stopCustomVizDevServer,
|
||||
} from "./helpers/e2e-custom-viz-dev-server-tasks";
|
||||
import { buildDataApp } from "./helpers/e2e-data-app-tasks";
|
||||
import {
|
||||
buildDataApp,
|
||||
removeDataAppDeclaration,
|
||||
removeDataAppPaths,
|
||||
scaffoldDataApp,
|
||||
syncDataApp,
|
||||
writeDataAppFiles,
|
||||
} from "./helpers/e2e-data-app-tasks";
|
||||
import { signJwt } from "./helpers/e2e-jwt-tasks";
|
||||
import {
|
||||
startMockLlmServer,
|
||||
@@ -296,6 +303,11 @@ const defaultConfig = {
|
||||
startCustomVizDevServer,
|
||||
stopCustomVizDevServer,
|
||||
buildDataApp,
|
||||
syncDataApp,
|
||||
scaffoldDataApp,
|
||||
writeDataAppFiles,
|
||||
removeDataAppDeclaration,
|
||||
removeDataAppPaths,
|
||||
...perTestCaptureTasks,
|
||||
});
|
||||
|
||||
|
||||
16
e2e/support/helpers/api/groupPermissions.ts
Normal file
16
e2e/support/helpers/api/groupPermissions.ts
Normal file
@@ -0,0 +1,16 @@
|
||||
export function addUserToGroup(groupId: number, email: string) {
|
||||
return cy
|
||||
.request<{ data: Array<{ id: number; email: string }> }>("/api/user")
|
||||
.then(({ body }) => {
|
||||
const user = body.data.find((candidate) => candidate.email === email);
|
||||
|
||||
if (!user) {
|
||||
throw new Error(`No user with the email ${email}`);
|
||||
}
|
||||
|
||||
return cy.request("POST", "/api/permissions/membership", {
|
||||
group_id: groupId,
|
||||
user_id: user.id,
|
||||
});
|
||||
});
|
||||
}
|
||||
@@ -39,6 +39,7 @@ export { createTransformTag } from "./createTransformTag";
|
||||
export * from "./customVizPlugin";
|
||||
export { deleteComment } from "./deleteComment";
|
||||
export { editDashboardCard } from "./editDashboardCard";
|
||||
export { addUserToGroup } from "./groupPermissions";
|
||||
export { getCurrentUser } from "./getCurrentUser";
|
||||
export { publishTables } from "./publishTables";
|
||||
export { remapDisplayValueToFK } from "./remapDisplayValueToFK";
|
||||
|
||||
@@ -15,6 +15,11 @@ export const DATA_APP_BUILD_SCRIPT = path.join(
|
||||
"e2e/support/helpers/build-data-app-fixture.mjs",
|
||||
);
|
||||
|
||||
export const DATA_APP_SYNC_SCRIPT = path.join(
|
||||
REPO_ROOT,
|
||||
"e2e/support/helpers/sync-data-app-fixture.mjs",
|
||||
);
|
||||
|
||||
export const SDK_DATA_APP_DEV_CONFIG_SOURCE = path.join(
|
||||
REPO_ROOT,
|
||||
"enterprise/frontend/src/embedding-sdk-package/data-app-dev.config.ts",
|
||||
|
||||
@@ -1,8 +1,3 @@
|
||||
import type {
|
||||
LocalFieldReference,
|
||||
QuestionColumnReference,
|
||||
} from "@metabase/embedding-sdk-react/data-app";
|
||||
|
||||
/**
|
||||
* The config a spec injects into a data-app fixture (via `H.mockDataApp`'s
|
||||
* `testEnv`), so a fixture doesn't hard-code values that track the Cypress
|
||||
@@ -14,8 +9,17 @@ import type {
|
||||
* runtime through its own `getTestEnv()`.
|
||||
*/
|
||||
export type DataAppTestEnv = {
|
||||
scalarQuery: { source: TableSource; aggregations: CountAggregation[] };
|
||||
questionQuery: { source: TableSource };
|
||||
/**
|
||||
* A deployed app runs the card its query was published as, so every query the
|
||||
* fixture renders names one. The swap drops the static clauses the card
|
||||
* already contains, so each id must match its whole authored query.
|
||||
*/
|
||||
scalarQuery: {
|
||||
source: TableSource;
|
||||
aggregations: CountAggregation[];
|
||||
savedQuestionSourceId: number;
|
||||
};
|
||||
questionQuery: { source: TableSource; savedQuestionSourceId: number };
|
||||
/**
|
||||
* `/download-question` page: the id of a saved CHART question (so a `.png`
|
||||
* export is offered). Passed in because the id tracks the Cypress snapshot.
|
||||
@@ -39,37 +43,11 @@ export type DataAppTestEnv = {
|
||||
xhrBlockedUrl?: string;
|
||||
};
|
||||
/**
|
||||
* `/query-states` page: a deliberately invalid query (bad table id), so the hook
|
||||
* resolves to an `error` state; the page also exercises `refetch`.
|
||||
* `/query-states` page: a deliberately invalid query (a card that does not
|
||||
* exist), so the hook resolves to an `error` state rather than to the refusal
|
||||
* an unsynchronized query would raise; the page also exercises `refetch`.
|
||||
*/
|
||||
errorQuery?: { source: TableSource };
|
||||
/**
|
||||
* `/combinators` page: exercises `filter`/`breakout`/`orderBy`/`aggregations`.
|
||||
* The page hardcodes a `>` filter and a count aggregation ordered desc, so
|
||||
* `filterField` must be numeric; `breakoutField` groups the rows.
|
||||
*/
|
||||
combinators?: {
|
||||
source: TableSource;
|
||||
filterField: LocalFieldReference;
|
||||
filterValue: number;
|
||||
breakoutField: LocalFieldReference;
|
||||
};
|
||||
/**
|
||||
* `/card-source` page: exercises a saved-question source with filters,
|
||||
* aggregations, and breakouts. Each clause set runs against both sources and
|
||||
* the page reports whether they agree, so `source` must be an unfiltered copy
|
||||
* of `tableSource`. `*Field` addresses the table, `*Column` the question's
|
||||
* result columns.
|
||||
*/
|
||||
cardSource?: {
|
||||
source: CardSource;
|
||||
tableSource: TableSource;
|
||||
filterField: LocalFieldReference;
|
||||
filterColumn: QuestionColumnReference;
|
||||
filterValue: number;
|
||||
breakoutField: LocalFieldReference;
|
||||
breakoutColumn: QuestionColumnReference;
|
||||
};
|
||||
errorQuery?: { source: TableSource; savedQuestionSourceId: number };
|
||||
/**
|
||||
* `/actions` page: the id of the action the spec creates and `useAction`
|
||||
* executes, so it can't be hard-coded in the app. Left out to exercise the "no
|
||||
@@ -90,5 +68,4 @@ export type DataAppTestEnv = {
|
||||
// The queries use the SDK's `source` API (`{ type: "table", id }`); the database
|
||||
// is resolved from the SDK's metadata store, so no databaseId is passed.
|
||||
type TableSource = { type: "table"; id: number };
|
||||
type CardSource = { type: "card"; id: number };
|
||||
type CountAggregation = { type: "operator"; operator: "count"; args: [] };
|
||||
|
||||
@@ -1,49 +1,24 @@
|
||||
import { USER_GROUPS } from "e2e/support/cypress_data";
|
||||
import * as Urls from "metabase/urls/data-apps";
|
||||
import type { DataApp } from "metabase-types/api";
|
||||
import type {
|
||||
CardId,
|
||||
Collection,
|
||||
CollectionId,
|
||||
CollectionPermission,
|
||||
CollectionPermissionsGraph,
|
||||
DataApp,
|
||||
} from "metabase-types/api";
|
||||
|
||||
import type { DataAppTestEnv } from "./data-app-test-env";
|
||||
import { getIframeBody } from "./e2e-embedding-helpers";
|
||||
import { LOCAL_GIT_PATH } from "./e2e-remote-sync-helpers";
|
||||
|
||||
export const SYNCED_DATA_APPS_FIXTURE_PATH =
|
||||
Cypress.config("projectRoot") +
|
||||
"/e2e/support/assets/example_synced_data_apps";
|
||||
|
||||
/**
|
||||
* Copy the example data-app repo into the working directory of the local git repo
|
||||
* `setupGitSync` created, so a `commitToRepo` + a real pull materialize its apps.
|
||||
* The repo holds one directory per materialization outcome (a good app, an app
|
||||
* whose bundle is missing, a malformed config, and a directory with no config) —
|
||||
* see the fixture and the sync spec that drives it.
|
||||
*/
|
||||
export const copySyncedDataAppsFixture = () =>
|
||||
cy.task("copyDirectory", {
|
||||
source: SYNCED_DATA_APPS_FIXTURE_PATH,
|
||||
destination: LOCAL_GIT_PATH,
|
||||
});
|
||||
|
||||
/**
|
||||
* The e2e data-app fixture that `mockDataApp` builds and serves — its directory
|
||||
* name and its `/apps/<slug>` URL segment. One bundle with a page per surface it
|
||||
* exercises (questions, query hooks, actions, clipboard, sandbox, isolation), so
|
||||
* the specs share a single build.
|
||||
*/
|
||||
export const DATA_APP_NAME = "kitchen-sink";
|
||||
export const DATA_APP_DISPLAY_NAME = "Kitchen Sink";
|
||||
|
||||
/**
|
||||
* Visit a nested route inside the fixture data app. `openDataApp` encodes the
|
||||
* slug, so it can't carry a sub-path — deep-links go through a raw `cy.visit`.
|
||||
*/
|
||||
export const visitDataAppRoute = (route: string) =>
|
||||
cy.visit(`/apps/${DATA_APP_NAME}/${route}`);
|
||||
|
||||
/**
|
||||
* A minimal `DataApp`-shaped row. Admin-list state tests use it directly (they
|
||||
* only assert how the management UI renders the list, no real bundle needed);
|
||||
* `mockDataApp` uses it for the intercepted metadata of the built fixture.
|
||||
*/
|
||||
export const fakeDataApp = (overrides: Partial<DataApp> = {}): DataApp => ({
|
||||
id: 1,
|
||||
name: DATA_APP_NAME,
|
||||
@@ -51,6 +26,8 @@ export const fakeDataApp = (overrides: Partial<DataApp> = {}): DataApp => ({
|
||||
description: null,
|
||||
bundle_path: `data_apps/${DATA_APP_NAME}/dist/index.js`,
|
||||
enabled: true,
|
||||
resource_collection_id: null,
|
||||
permission_group_id: null,
|
||||
allowed_hosts: [],
|
||||
bundle_hash: "e2e-bundle-hash",
|
||||
last_synced_sha: "e2e0000",
|
||||
@@ -67,25 +44,14 @@ type MockDataAppOptions<TestEnv> = {
|
||||
/** `allowed_hosts` served in the bundle response header. */
|
||||
allowedHosts?: string[];
|
||||
/**
|
||||
* Config a fixture reads at runtime, so it doesn't hard-code values that track
|
||||
* the Cypress snapshot (e.g. sample-DB ids). It's JSON-serialized and prepended
|
||||
* to the served bundle as `globalThis.__METABASE_DATA_APP_TEST_ENV__`; since the
|
||||
* bundle is evaluated as one script in the sandbox realm, the app reads it as a
|
||||
* plain global. Typed by `DataAppTestEnv` by default; pass another fixture's
|
||||
* type as `TestEnv` if it differs.
|
||||
* Config a fixture reads at runtime, rather than hard-coding ids that track the
|
||||
* Cypress snapshot.
|
||||
*/
|
||||
testEnv?: TestEnv;
|
||||
/**
|
||||
* Hold the bundle response for this many ms, so the app's loading window is
|
||||
* long enough to assert on. Without it a small mocked bundle can render before
|
||||
* the test ever queries, making any "still loading" assertion racy.
|
||||
*/
|
||||
bundleDelay?: number;
|
||||
/** Delays the bundle response, so a loading assertion has a window to catch. */
|
||||
bundleDelayMs?: number;
|
||||
};
|
||||
|
||||
/**
|
||||
* Set up a data app for the current test
|
||||
*/
|
||||
export const mockDataApp = <TestEnv = DataAppTestEnv>(
|
||||
appName: string,
|
||||
options: MockDataAppOptions<TestEnv> = {},
|
||||
@@ -127,7 +93,7 @@ export const mockDataApp = <TestEnv = DataAppTestEnv>(
|
||||
"X-Metabase-Data-App-Allowed-Hosts": JSON.stringify(allowedHosts),
|
||||
},
|
||||
body: prelude + bundleCode,
|
||||
...(options.bundleDelay ? { delay: options.bundleDelay } : {}),
|
||||
...(options.bundleDelayMs ? { delay: options.bundleDelayMs } : {}),
|
||||
},
|
||||
);
|
||||
|
||||
@@ -143,23 +109,230 @@ export function dataAppIframe(displayName: string) {
|
||||
return getIframeBody(`iframe[title="${displayName}"]`);
|
||||
}
|
||||
|
||||
export function setDataAppCollectionAccess(
|
||||
collectionId: CollectionId,
|
||||
access: CollectionPermission,
|
||||
) {
|
||||
return cy
|
||||
.request<CollectionPermissionsGraph>("GET", "/api/collection/graph")
|
||||
.then(({ body: graph }) => {
|
||||
const groups = Object.fromEntries(
|
||||
Object.entries(graph.groups).map(([groupId, collections]) => [
|
||||
groupId,
|
||||
Number(groupId) === USER_GROUPS.ADMIN_GROUP
|
||||
? collections
|
||||
: { ...collections, [collectionId]: access },
|
||||
]),
|
||||
);
|
||||
|
||||
cy.request("PUT", "/api/collection/graph", { ...graph, groups });
|
||||
});
|
||||
}
|
||||
|
||||
export function moveDataAppModelToCollection({
|
||||
modelId,
|
||||
name,
|
||||
access,
|
||||
}: {
|
||||
modelId: CardId;
|
||||
name: string;
|
||||
access: CollectionPermission;
|
||||
}) {
|
||||
return cy
|
||||
.request<Collection>("POST", "/api/collection", { name })
|
||||
.then(({ body: collection }) => {
|
||||
cy.request("PUT", `/api/card/${modelId}`, {
|
||||
collection_id: collection.id,
|
||||
});
|
||||
|
||||
setDataAppCollectionAccess(collection.id, access);
|
||||
|
||||
return cy.wrap(collection, { log: false });
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* The dev host app is a real vite data app with the published SDK installed, so
|
||||
* synchronizing it exercises the package an author actually consumes rather than
|
||||
* the stub the scratch fixture provides.
|
||||
*/
|
||||
export const dataAppHostAppRoot = () =>
|
||||
`${Cypress.config("projectRoot")}/${DATA_APP_DEV_HOST_APP_DIR}`;
|
||||
|
||||
const actionDeclarationsFile = (appRoot: string) =>
|
||||
`${appRoot}/actions/orders.action.ts`;
|
||||
|
||||
/**
|
||||
* Clears everything synchronization generates in the host app. Both sync specs
|
||||
* drive the same checked-in directory, so each has to start from a clean tree —
|
||||
* a stray `actions/` would otherwise be discovered by the query spec's sync.
|
||||
*/
|
||||
export function resetDataAppHostAppSources() {
|
||||
const appRoot = dataAppHostAppRoot();
|
||||
|
||||
return cy.task("removeDataAppPaths", {
|
||||
paths: [
|
||||
`${appRoot}/queries`,
|
||||
`${appRoot}/actions`,
|
||||
`${appRoot}/resources_metadata.json`,
|
||||
],
|
||||
});
|
||||
}
|
||||
|
||||
export function declareDataAppActions(
|
||||
appRoot: string,
|
||||
sourceActionIds: number[],
|
||||
) {
|
||||
return cy.task("writeDataAppFiles", {
|
||||
files: {
|
||||
[actionDeclarationsFile(appRoot)]: [
|
||||
'import { defineAction } from "@metabase/embedding-sdk-react/data-app";',
|
||||
...sourceActionIds.map(
|
||||
(id) =>
|
||||
`export const Action${id} = defineAction({ action: { id: ${id}, parameters: [] } });`,
|
||||
),
|
||||
].join("\n"),
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
const queryDeclarationsFile = (appRoot: string) =>
|
||||
`${appRoot}/queries/orders.query.ts`;
|
||||
|
||||
/** Declares one `defineQuery` per entry, as an app author would. */
|
||||
export function declareDataAppQueries(
|
||||
appRoot: string,
|
||||
declarations: Array<{ name: string; tableId: number; limit?: number }>,
|
||||
) {
|
||||
return cy.task("writeDataAppFiles", {
|
||||
files: {
|
||||
[queryDeclarationsFile(appRoot)]: [
|
||||
'import { defineQuery } from "@metabase/embedding-sdk-react/data-app";',
|
||||
...declarations.map(({ name, tableId, limit }) => {
|
||||
const clauses = limit === undefined ? "" : `, limit: ${limit}`;
|
||||
return `export const ${name} = defineQuery({ source: { type: "table", id: ${tableId} }${clauses} });`;
|
||||
}),
|
||||
].join("\n"),
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Deletes one declaration in place, so the rest keep the generated IDs
|
||||
* synchronization injected — which is what an author removing one does.
|
||||
* Splits on the declaration keyword rather than on lines, because an injected
|
||||
* ID lands on its own line and makes a declaration span several.
|
||||
*/
|
||||
function removeDeclaration(filePath: string, exportName: string) {
|
||||
return cy.task("removeDataAppDeclaration", { filePath, exportName });
|
||||
}
|
||||
|
||||
export function removeDataAppQueryDeclaration(appRoot: string, name: string) {
|
||||
return removeDeclaration(queryDeclarationsFile(appRoot), name);
|
||||
}
|
||||
|
||||
export function removeDataAppActionDeclaration(
|
||||
appRoot: string,
|
||||
sourceActionId: number,
|
||||
) {
|
||||
return removeDeclaration(
|
||||
actionDeclarationsFile(appRoot),
|
||||
`Action${sourceActionId}`,
|
||||
);
|
||||
}
|
||||
|
||||
/** Runs the real `sync-resources` CLI against the instance under test. */
|
||||
export function syncDataAppResources(apiKey: string, appRoot: string) {
|
||||
return cy.task<{ ok: boolean; error: string | null }>("syncDataApp", {
|
||||
appRoot,
|
||||
metabaseUrl: Cypress.config("baseUrl"),
|
||||
apiKey,
|
||||
});
|
||||
}
|
||||
|
||||
export const copySyncedDataAppsFixture = () =>
|
||||
cy.task("copyDirectory", {
|
||||
source: `${Cypress.config("projectRoot")}/e2e/support/assets/example_synced_data_apps`,
|
||||
destination: LOCAL_GIT_PATH,
|
||||
});
|
||||
|
||||
/** The test repo is not an npm project, so `defineQuery` needs a stub to resolve. */
|
||||
export function declareSyncedDataAppQuery(slug: string, tableId: number) {
|
||||
const appRoot = `${LOCAL_GIT_PATH}/data_apps/${slug}`;
|
||||
const packageRoot = `${appRoot}/node_modules/@metabase/embedding-sdk-react`;
|
||||
|
||||
return cy.task("writeDataAppFiles", {
|
||||
files: {
|
||||
[`${packageRoot}/package.json`]: JSON.stringify({
|
||||
name: "@metabase/embedding-sdk-react",
|
||||
exports: { "./data-app": "./data-app.js" },
|
||||
}),
|
||||
[`${packageRoot}/data-app.js`]: "exports.defineQuery = (q) => q;",
|
||||
[`${appRoot}/queries/orders.query.ts`]: [
|
||||
'import { defineQuery } from "@metabase/embedding-sdk-react/data-app";',
|
||||
`export const Orders = defineQuery({ source: { type: "table", id: ${tableId} } });`,
|
||||
"",
|
||||
].join("\n"),
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
export function createDataAppApiKey() {
|
||||
return cy
|
||||
.request<{ unmasked_key: string }>("POST", "/api/api-key", {
|
||||
name: `data-app-sync-e2e-${Date.now()}`,
|
||||
group_id: USER_GROUPS.ADMIN_GROUP,
|
||||
})
|
||||
.then(({ body }) => body.unmasked_key);
|
||||
}
|
||||
|
||||
/**
|
||||
* A second app beside the host app, for cases that need two of them. It reuses
|
||||
* the host app's `node_modules`, so `defineQuery` still resolves through the
|
||||
* published SDK, and its directory name becomes the app's slug.
|
||||
*/
|
||||
export function createSecondDataApp(slug: string) {
|
||||
cy.task("scaffoldDataApp", { appName: slug, sdkFrom: dataAppHostAppRoot() });
|
||||
|
||||
return `${Cypress.config("projectRoot")}/e2e/tmp/${slug}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Runs the host app's own production build. The SDK's `metabase-resource-sync-check`
|
||||
* plugin runs on `buildStart`, so this is what refuses to bundle a stale app.
|
||||
*/
|
||||
export function buildDataAppHostApp() {
|
||||
return cy.exec(`cd "${dataAppHostAppRoot()}" && npm run build`, {
|
||||
failOnNonZeroExit: false,
|
||||
timeout: 180_000,
|
||||
});
|
||||
}
|
||||
|
||||
/** The app's own permission group — the one its viewers are given. */
|
||||
export function dataAppPermissionGroupId(slug: string) {
|
||||
return cy.request<DataApp>(`/api/apps/${slug}`).then(({ body }) => {
|
||||
const groupId = body.permission_group_id;
|
||||
|
||||
if (typeof groupId !== "number") {
|
||||
throw new Error(`Data app ${slug} has no permission group.`);
|
||||
}
|
||||
|
||||
return cy.wrap(groupId, { log: false });
|
||||
});
|
||||
}
|
||||
|
||||
/** Puts a user in the app's own permission group, as granting app access does. */
|
||||
const DATA_APP_DEV_HOST_APP_DIR =
|
||||
"e2e/embedding-sdk-host-apps/vite-6-data-app-host-app";
|
||||
|
||||
const DATA_APP_DEV_ENV_PATH = `${DATA_APP_DEV_HOST_APP_DIR}/.env.local`;
|
||||
|
||||
/** The dev host app's manifest — the live manifest-validation suite edits it. */
|
||||
export const DATA_APP_DEV_MANIFEST_PATH = `${DATA_APP_DEV_HOST_APP_DIR}/data_app.yaml`;
|
||||
|
||||
/** The dev host app's source entry — the soft-reload suite edits it. */
|
||||
export const DATA_APP_DEV_APP_SRC_PATH = `${DATA_APP_DEV_HOST_APP_DIR}/src/App.tsx`;
|
||||
|
||||
const DATA_APP_DEV_CONTENT_TIMEOUT_MS = 40000;
|
||||
|
||||
/**
|
||||
* Visit the dev host app and wait for the sandboxed bundle to render — the
|
||||
* point at which the dev entry has booted (toolbar mounted, probes fired).
|
||||
*/
|
||||
export function visitDataAppDevApp(clientHost: string) {
|
||||
cy.visit(clientHost);
|
||||
cy.findByTestId("dev-app-content", {
|
||||
@@ -167,35 +340,31 @@ export function visitDataAppDevApp(clientHost: string) {
|
||||
}).should("exist");
|
||||
}
|
||||
|
||||
/**
|
||||
* Provision auth for the data-app dev-server host app: create an API key and
|
||||
* write the `.env.local` the SDK dev entry reads at startup, then wait for Vite
|
||||
* to restart onto it. `clientHost` is the dev server origin.
|
||||
*/
|
||||
export function setUpDataAppDevServer(clientHost: string) {
|
||||
const mbUrl = Cypress.config("baseUrl");
|
||||
if (!mbUrl) {
|
||||
throw new Error("baseUrl must be set for the data-app dev-server suite");
|
||||
}
|
||||
|
||||
cy.exec(`rm -f ${DATA_APP_DEV_ENV_PATH}`);
|
||||
cy.task("removeDataAppPaths", { paths: [DATA_APP_DEV_ENV_PATH] });
|
||||
waitForDataAppDevServerEnv(clientHost, mbUrl, { expectPresent: false });
|
||||
|
||||
cy.request("POST", "/api/api-key", {
|
||||
name: `data-app-dev-e2e-${Date.now()}`,
|
||||
group_id: USER_GROUPS.ADMIN_GROUP,
|
||||
}).then(({ body }) => {
|
||||
cy.writeFile(
|
||||
DATA_APP_DEV_ENV_PATH,
|
||||
`DATA_APP_MB_URL=${mbUrl}\nDATA_APP_MB_API_KEY=${body.unmasked_key}\n`,
|
||||
);
|
||||
cy.task("writeDataAppFiles", {
|
||||
files: {
|
||||
[DATA_APP_DEV_ENV_PATH]: `DATA_APP_MB_URL=${mbUrl}\nDATA_APP_MB_API_KEY=${body.unmasked_key}\n`,
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
waitForDataAppDevServerEnv(clientHost, mbUrl, { expectPresent: true });
|
||||
}
|
||||
|
||||
export function tearDownDataAppDevServer() {
|
||||
cy.exec(`rm -f ${DATA_APP_DEV_ENV_PATH}`);
|
||||
return cy.task("removeDataAppPaths", { paths: [DATA_APP_DEV_ENV_PATH] });
|
||||
}
|
||||
|
||||
// `DATA_APP_MB_URL` shows up in (or drops out of) the served CSP once Vite has
|
||||
|
||||
@@ -5,12 +5,64 @@ import path from "node:path";
|
||||
import {
|
||||
DATA_APP_BUILD_SCRIPT,
|
||||
DATA_APP_FIXTURES_DIR,
|
||||
DATA_APP_SYNC_SCRIPT,
|
||||
REPO_ROOT,
|
||||
} from "./data-app-fixture-paths.mjs";
|
||||
|
||||
/**
|
||||
* Cypress task: build a data-app fixture and return its bundle code.
|
||||
*/
|
||||
function run(script: string, args: string[], env: NodeJS.ProcessEnv = {}) {
|
||||
return new Promise<void>((resolve, reject) => {
|
||||
const child = spawn(process.execPath, [script, ...args], {
|
||||
cwd: REPO_ROOT,
|
||||
env: { ...process.env, ...env },
|
||||
stdio: ["ignore", "inherit", "pipe"],
|
||||
});
|
||||
|
||||
// Mirrored to the runner as it arrives, and kept so a rejection carries the
|
||||
// script's own message rather than a bare exit code.
|
||||
let stderr = "";
|
||||
child.stderr.setEncoding("utf8");
|
||||
child.stderr.on("data", (chunk: string) => {
|
||||
stderr += chunk;
|
||||
process.stderr.write(chunk);
|
||||
});
|
||||
|
||||
child.on("error", reject);
|
||||
child.on("exit", (code) =>
|
||||
code === 0
|
||||
? resolve()
|
||||
: reject(
|
||||
new Error(
|
||||
stderr.trim() ||
|
||||
`${path.basename(script)} ${args.join(" ")} exited with code ${code}`,
|
||||
),
|
||||
),
|
||||
);
|
||||
});
|
||||
}
|
||||
|
||||
export async function syncDataApp({
|
||||
appRoot,
|
||||
metabaseUrl,
|
||||
apiKey,
|
||||
}: {
|
||||
appRoot: string;
|
||||
metabaseUrl: string;
|
||||
apiKey: string;
|
||||
}): Promise<{ ok: boolean; error: string | null }> {
|
||||
try {
|
||||
await run(DATA_APP_SYNC_SCRIPT, [appRoot], {
|
||||
DATA_APP_MB_URL: metabaseUrl,
|
||||
DATA_APP_MB_API_KEY: apiKey,
|
||||
});
|
||||
return { ok: true, error: null };
|
||||
} catch (error) {
|
||||
return {
|
||||
ok: false,
|
||||
error: error instanceof Error ? error.message : String(error),
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
export async function buildDataApp({
|
||||
appName,
|
||||
}: {
|
||||
@@ -22,23 +74,7 @@ export async function buildDataApp({
|
||||
throw new Error(`data-app fixture "${appName}" has no src/ at ${appDir}`);
|
||||
}
|
||||
|
||||
await new Promise<void>((resolve, reject) => {
|
||||
const child = spawn(process.execPath, [DATA_APP_BUILD_SCRIPT, appName], {
|
||||
cwd: REPO_ROOT,
|
||||
stdio: ["ignore", "inherit", "inherit"],
|
||||
});
|
||||
|
||||
child.on("error", reject);
|
||||
child.on("exit", (code) =>
|
||||
code === 0
|
||||
? resolve()
|
||||
: reject(
|
||||
new Error(
|
||||
`data-app build for "${appName}" exited with code ${code}`,
|
||||
),
|
||||
),
|
||||
);
|
||||
});
|
||||
await run(DATA_APP_BUILD_SCRIPT, [appName]);
|
||||
|
||||
const bundlePath = path.join(DATA_APP_FIXTURES_DIR, appName, "dist/index.js");
|
||||
|
||||
@@ -50,3 +86,93 @@ export async function buildDataApp({
|
||||
|
||||
return fs.readFileSync(bundlePath, "utf8");
|
||||
}
|
||||
|
||||
/** Cypress resolves a relative path against the project root; match that. */
|
||||
function inRepo(target: string) {
|
||||
return path.isAbsolute(target) ? target : path.join(REPO_ROOT, target);
|
||||
}
|
||||
|
||||
/**
|
||||
* The fixture filesystem lives behind tasks rather than `cy.exec`/`cy.writeFile`
|
||||
* so a spec changes an app in one round trip, and so nothing is composed into a
|
||||
* shell string.
|
||||
*/
|
||||
export async function scaffoldDataApp({
|
||||
appName,
|
||||
sdkFrom,
|
||||
}: {
|
||||
appName: string;
|
||||
sdkFrom: string;
|
||||
}): Promise<string> {
|
||||
// The task owns the location so a caller cannot aim the `rmSync` below elsewhere.
|
||||
if (path.basename(appName) !== appName) {
|
||||
throw new Error(`A scaffolded app is named, not located: ${appName}`);
|
||||
}
|
||||
|
||||
const root = path.join(REPO_ROOT, "e2e", "tmp", appName);
|
||||
|
||||
fs.rmSync(root, { recursive: true, force: true });
|
||||
fs.mkdirSync(root, { recursive: true });
|
||||
// The SDK the app synchronizes against, without a second npm install.
|
||||
fs.symlinkSync(
|
||||
path.join(inRepo(sdkFrom), "node_modules"),
|
||||
path.join(root, "node_modules"),
|
||||
);
|
||||
fs.writeFileSync(
|
||||
path.join(root, "data_app.yaml"),
|
||||
`name: ${appName}\npath: ./dist/index.js\n`,
|
||||
);
|
||||
|
||||
return root;
|
||||
}
|
||||
|
||||
export async function writeDataAppFiles({
|
||||
files,
|
||||
}: {
|
||||
files: Record<string, string>;
|
||||
}): Promise<null> {
|
||||
for (const [filePath, contents] of Object.entries(files)) {
|
||||
const target = inRepo(filePath);
|
||||
|
||||
fs.mkdirSync(path.dirname(target), { recursive: true });
|
||||
fs.writeFileSync(target, contents);
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
export async function removeDataAppDeclaration({
|
||||
filePath,
|
||||
exportName,
|
||||
}: {
|
||||
filePath: string;
|
||||
exportName: string;
|
||||
}): Promise<null> {
|
||||
const target = inRepo(filePath);
|
||||
const [imports, ...declarations] = fs
|
||||
.readFileSync(target, "utf8")
|
||||
.split("export const ");
|
||||
|
||||
fs.writeFileSync(
|
||||
target,
|
||||
imports +
|
||||
declarations
|
||||
.filter((declaration) => !declaration.startsWith(`${exportName} `))
|
||||
.map((declaration) => `export const ${declaration}`)
|
||||
.join(""),
|
||||
);
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
export async function removeDataAppPaths({
|
||||
paths,
|
||||
}: {
|
||||
paths: string[];
|
||||
}): Promise<null> {
|
||||
for (const target of paths) {
|
||||
fs.rmSync(inRepo(target), { recursive: true, force: true });
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
50
e2e/support/helpers/sync-data-app-fixture.mjs
Normal file
50
e2e/support/helpers/sync-data-app-fixture.mjs
Normal file
@@ -0,0 +1,50 @@
|
||||
import fs from "node:fs";
|
||||
import path from "node:path";
|
||||
import { pathToFileURL } from "node:url";
|
||||
|
||||
import { build as bundle } from "esbuild";
|
||||
|
||||
import { DATA_APP_FIXTURES_DIR, REPO_ROOT } from "./data-app-fixture-paths.mjs";
|
||||
|
||||
const appRoot = process.argv[2];
|
||||
if (!appRoot) {
|
||||
console.error("usage: sync-data-app-fixture.mjs <appRoot>");
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
const { DATA_APP_MB_URL, DATA_APP_MB_API_KEY } = process.env;
|
||||
if (!DATA_APP_MB_URL || !DATA_APP_MB_API_KEY) {
|
||||
console.error("DATA_APP_MB_URL and DATA_APP_MB_API_KEY must be set");
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
const SDK_PACKAGE_DIR = path.join(
|
||||
REPO_ROOT,
|
||||
"enterprise/frontend/src/embedding-sdk-package",
|
||||
);
|
||||
|
||||
const scratchDir = path.join(DATA_APP_FIXTURES_DIR, ".build");
|
||||
fs.mkdirSync(scratchDir, { recursive: true });
|
||||
const entry = path.join(scratchDir, "sync-resources.mjs");
|
||||
|
||||
// The synchronization sources are TypeScript and reach the SDK package by its
|
||||
// build alias, so bundle them the way the fixture build does before importing.
|
||||
await bundle({
|
||||
entryPoints: [path.join(SDK_PACKAGE_DIR, "data-app-query-sync/sync.ts")],
|
||||
outfile: entry,
|
||||
bundle: true,
|
||||
platform: "node",
|
||||
format: "esm",
|
||||
packages: "external",
|
||||
alias: { "embedding-sdk-package": SDK_PACKAGE_DIR },
|
||||
logLevel: "warning",
|
||||
});
|
||||
|
||||
const { syncResources } = await import(pathToFileURL(entry).href);
|
||||
|
||||
await syncResources({
|
||||
appRoot,
|
||||
metabaseUrl: DATA_APP_MB_URL,
|
||||
apiKey: DATA_APP_MB_API_KEY,
|
||||
log: (message) => console.log(`[sync-resources] ${message}`),
|
||||
});
|
||||
@@ -0,0 +1,197 @@
|
||||
import { MetabaseProvider } from "@metabase/embedding-sdk-react";
|
||||
import {
|
||||
type MetabaseQueryOptions,
|
||||
aggregations,
|
||||
breakout,
|
||||
filter,
|
||||
orderBy,
|
||||
useMetabaseQuery,
|
||||
} from "@metabase/embedding-sdk-react/data-app";
|
||||
|
||||
import { SAMPLE_DATABASE } from "e2e/support/cypress_sample_database";
|
||||
import { createQuestion } from "e2e/support/helpers";
|
||||
import {
|
||||
DEFAULT_SDK_AUTH_PROVIDER_CONFIG,
|
||||
mountSdk,
|
||||
} from "e2e/support/helpers/embedding-sdk-component-testing";
|
||||
import {
|
||||
mockAuthProviderAndJwtSignIn,
|
||||
signInAsAdminAndEnableEmbeddingSdk,
|
||||
} from "e2e/support/helpers/embedding-sdk-testing";
|
||||
|
||||
const { ORDERS, ORDERS_ID } = SAMPLE_DATABASE;
|
||||
|
||||
const tableSource = { type: "table" as const, id: ORDERS_ID };
|
||||
|
||||
/** The same dimension addressed on a table source and on a card's result columns. */
|
||||
const totalField = {
|
||||
type: "column" as const,
|
||||
fieldId: ORDERS.TOTAL,
|
||||
tableId: ORDERS_ID,
|
||||
name: "TOTAL",
|
||||
jsType: "number" as const,
|
||||
};
|
||||
const totalColumn = {
|
||||
type: "column" as const,
|
||||
name: "TOTAL",
|
||||
jsType: "number" as const,
|
||||
};
|
||||
const productField = {
|
||||
type: "column" as const,
|
||||
fieldId: ORDERS.PRODUCT_ID,
|
||||
tableId: ORDERS_ID,
|
||||
name: "PRODUCT_ID",
|
||||
jsType: "number" as const,
|
||||
};
|
||||
const productColumn = {
|
||||
type: "column" as const,
|
||||
name: "PRODUCT_ID",
|
||||
jsType: "number" as const,
|
||||
};
|
||||
|
||||
const describeResult = (
|
||||
data: { columns: { name: string }[]; rawRows: unknown[][] } | null,
|
||||
) =>
|
||||
data &&
|
||||
`[${data.columns.map((column) => column.name).join(",")}] ${JSON.stringify(data.rawRows)}`;
|
||||
|
||||
/**
|
||||
* Runs the same clauses against a table source and against a card of that table,
|
||||
* and reports whether they agree — so an assertion never has to encode
|
||||
* sample-database numbers.
|
||||
*/
|
||||
function QueryComparison({
|
||||
tableQuery,
|
||||
cardQuery,
|
||||
}: {
|
||||
tableQuery: MetabaseQueryOptions<undefined>;
|
||||
cardQuery: MetabaseQueryOptions<undefined>;
|
||||
}) {
|
||||
const fromTable = useMetabaseQuery(tableQuery);
|
||||
const fromCard = useMetabaseQuery(cardQuery);
|
||||
|
||||
const failure = fromTable.error ?? fromCard.error;
|
||||
const table = describeResult(fromTable.data);
|
||||
const card = describeResult(fromCard.data);
|
||||
|
||||
const status = failure
|
||||
? `error: ${String(failure)}`
|
||||
: table === null || card === null
|
||||
? "loading"
|
||||
: table === card
|
||||
? "match"
|
||||
: `mismatch: table=${table} card=${card}`;
|
||||
|
||||
return (
|
||||
<>
|
||||
<div data-testid="comparison">{status}</div>
|
||||
<div data-testid="first-aggregate">
|
||||
{String(fromCard.data?.rawRows?.[0]?.at(-1) ?? "")}
|
||||
</div>
|
||||
</>
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* A table source and a saved-question source are interchangeable to the query
|
||||
* DSL. This lives outside a data app on purpose: there a query runs the card it
|
||||
* was published as, so an un-published table source is refused and the two
|
||||
* cannot be compared side by side.
|
||||
*/
|
||||
describe("scenarios > embedding-sdk > query sources", () => {
|
||||
beforeEach(() => {
|
||||
signInAsAdminAndEnableEmbeddingSdk();
|
||||
|
||||
createQuestion({
|
||||
name: "Orders, for source comparison",
|
||||
query: { "source-table": ORDERS_ID },
|
||||
}).then(({ body: card }) => {
|
||||
cy.wrap({ type: "card" as const, id: card.id }).as("cardSource");
|
||||
});
|
||||
|
||||
cy.signOut();
|
||||
mockAuthProviderAndJwtSignIn();
|
||||
});
|
||||
|
||||
const compare = (
|
||||
build: (source: { type: "card"; id: number }) => {
|
||||
tableQuery: MetabaseQueryOptions<undefined>;
|
||||
cardQuery: MetabaseQueryOptions<undefined>;
|
||||
},
|
||||
) =>
|
||||
cy.get<{ type: "card"; id: number }>("@cardSource").then((cardSource) => {
|
||||
const { tableQuery, cardQuery } = build(cardSource);
|
||||
|
||||
mountSdk(
|
||||
<MetabaseProvider authConfig={DEFAULT_SDK_AUTH_PROVIDER_CONFIG}>
|
||||
<QueryComparison tableQuery={tableQuery} cardQuery={cardQuery} />
|
||||
</MetabaseProvider>,
|
||||
);
|
||||
|
||||
cy.findByTestId("comparison", { timeout: 30000 }).should(
|
||||
"have.text",
|
||||
"match",
|
||||
);
|
||||
// Every comparison ends in an aggregation, and breakout columns come first, so
|
||||
// the first row's last cell is an aggregate. An empty result renders "" and a
|
||||
// zero count renders 0 — both vacuous matches, and both fail here.
|
||||
cy.findByTestId("first-aggregate").should(($el) =>
|
||||
expect(Number($el.text())).to.be.greaterThan(0),
|
||||
);
|
||||
});
|
||||
|
||||
it("returns the same rows from a table and from a card of that table", () => {
|
||||
compare((source) => ({
|
||||
tableQuery: { source: tableSource, aggregations: [aggregations.count()] },
|
||||
cardQuery: { source, aggregations: [aggregations.count()] },
|
||||
}));
|
||||
});
|
||||
|
||||
it("applies filters the same way on either source", () => {
|
||||
compare((source) => ({
|
||||
tableQuery: {
|
||||
source: tableSource,
|
||||
filters: [filter(totalField, ">", 50)],
|
||||
aggregations: [aggregations.count()],
|
||||
},
|
||||
cardQuery: {
|
||||
source,
|
||||
filters: [filter(totalColumn, ">", 50)],
|
||||
aggregations: [aggregations.count()],
|
||||
},
|
||||
}));
|
||||
});
|
||||
|
||||
it("applies aggregations the same way on either source", () => {
|
||||
compare((source) => ({
|
||||
tableQuery: {
|
||||
source: tableSource,
|
||||
aggregations: [aggregations.count(), aggregations.sum(totalField)],
|
||||
},
|
||||
cardQuery: {
|
||||
source,
|
||||
aggregations: [aggregations.count(), aggregations.sum(totalColumn)],
|
||||
},
|
||||
}));
|
||||
});
|
||||
|
||||
// The combinators the query builder exposes, on both source kinds at once.
|
||||
it("applies breakouts and orderBys the same way on either source", () => {
|
||||
compare((source) => ({
|
||||
tableQuery: {
|
||||
source: tableSource,
|
||||
filters: [filter(totalField, ">", 50)],
|
||||
aggregations: [aggregations.count()],
|
||||
breakouts: [breakout(productField)],
|
||||
orderBys: [orderBy(productField, "asc")],
|
||||
},
|
||||
cardQuery: {
|
||||
source,
|
||||
filters: [filter(totalColumn, ">", 50)],
|
||||
aggregations: [aggregations.count()],
|
||||
breakouts: [breakout(productColumn)],
|
||||
orderBys: [orderBy(productColumn, "asc")],
|
||||
},
|
||||
}));
|
||||
});
|
||||
});
|
||||
607
e2e/test-host-app/data-apps/sync-resources-actions.cy.spec.ts
Normal file
607
e2e/test-host-app/data-apps/sync-resources-actions.cy.spec.ts
Normal file
@@ -0,0 +1,607 @@
|
||||
import { USERS, WRITABLE_DB_ID } from "e2e/support/cypress_data";
|
||||
import {
|
||||
addUserToGroup,
|
||||
createDataAppApiKey,
|
||||
dataAppHostAppRoot,
|
||||
dataAppPermissionGroupId,
|
||||
declareDataAppActions,
|
||||
moveDataAppModelToCollection,
|
||||
removeDataAppActionDeclaration,
|
||||
resetDataAppHostAppSources,
|
||||
syncDataAppResources,
|
||||
} from "e2e/support/helpers";
|
||||
import type { Card } from "metabase-types/api";
|
||||
|
||||
const { H } = cy;
|
||||
|
||||
const TEST_TABLE = "scoreboard_actions";
|
||||
const MODEL_NAME = "Scoreboard model";
|
||||
|
||||
/** `syncResources` takes the app's slug from its directory name. */
|
||||
const APP_SLUG = "vite-6-data-app-host-app";
|
||||
|
||||
const APP_ROOT = () => dataAppHostAppRoot();
|
||||
|
||||
/**
|
||||
* Drives the real `sync-resources` CLI against a live instance, so the copies are
|
||||
* checked against Metabase rather than a fake of its API. The unit suite covers
|
||||
* the decision logic; this covers the contract.
|
||||
*/
|
||||
describe(
|
||||
"Embedding SDK: data-app sync-resources (actions)",
|
||||
{ tags: ["@external", "@actions"] },
|
||||
() => {
|
||||
beforeEach(() => {
|
||||
H.restore("postgres-writable");
|
||||
cy.signInAsAdmin();
|
||||
H.activateToken("bleeding-edge");
|
||||
|
||||
H.resetTestTable({ type: "postgres", table: TEST_TABLE });
|
||||
H.resyncDatabase({ dbId: WRITABLE_DB_ID, tableName: TEST_TABLE });
|
||||
H.setActionsEnabledForDB(WRITABLE_DB_ID);
|
||||
H.createModelFromTableName({
|
||||
tableName: TEST_TABLE,
|
||||
modelName: MODEL_NAME,
|
||||
});
|
||||
|
||||
resetDataAppHostAppSources();
|
||||
createDataAppApiKey().as("apiKey");
|
||||
});
|
||||
|
||||
after(() => {
|
||||
resetDataAppHostAppSources();
|
||||
});
|
||||
|
||||
const sync = () =>
|
||||
cy.get<string>("@apiKey").then((apiKey) => {
|
||||
syncDataAppResources(apiKey, APP_ROOT()).then(({ ok, error }) => {
|
||||
expect(error, "sync-resources failed").to.eq(null);
|
||||
expect(ok).to.eq(true);
|
||||
});
|
||||
});
|
||||
|
||||
/** The models Metabase holds in the app's own collection. */
|
||||
const copiedModels = () =>
|
||||
cy.request(`/api/apps/${APP_SLUG}`).then(({ body: app }) => {
|
||||
expect(app.resource_collection_id, "resource collection").to.be.a(
|
||||
"number",
|
||||
);
|
||||
return cy
|
||||
.request(
|
||||
`/api/collection/${app.resource_collection_id}/items?models=dataset`,
|
||||
)
|
||||
.then(({ body }) => body.data);
|
||||
});
|
||||
|
||||
const actionsOnModel = (modelId: number) =>
|
||||
cy.request(`/api/action?model-id=${modelId}`).then(({ body }) => body);
|
||||
|
||||
/** Runs a sync that must be refused, and returns the message it refused with. */
|
||||
const syncExpectingRefusal = (message: string) =>
|
||||
cy.get<string>("@apiKey").then((apiKey) => {
|
||||
syncDataAppResources(apiKey, APP_ROOT()).should(({ ok, error }) => {
|
||||
expect(ok, "sync-resources should have refused").to.eq(false);
|
||||
expect(error).to.contain(message);
|
||||
});
|
||||
});
|
||||
|
||||
/** Declares one implicit action on the fixture model, without syncing yet. */
|
||||
const declareOneAction = () =>
|
||||
cy.get<number>("@modelId").then((modelId) =>
|
||||
H.createImplicitAction({ model_id: modelId, kind: "create" }).then(
|
||||
({ body: action }) => {
|
||||
declareDataAppActions(APP_ROOT(), [action.id]);
|
||||
return cy.wrap({ modelId, action }, { log: false });
|
||||
},
|
||||
),
|
||||
);
|
||||
|
||||
/** Syncs one declared action and hands back the copies Metabase now holds. */
|
||||
const syncOneAction = () =>
|
||||
declareOneAction().then(({ modelId, action }) => {
|
||||
sync();
|
||||
return copiedModels().then(([copiedModel]) =>
|
||||
actionsOnModel(copiedModel.id).then(([copiedAction]) =>
|
||||
cy.wrap(
|
||||
{ modelId, action, copiedModel, copiedAction },
|
||||
{ log: false },
|
||||
),
|
||||
),
|
||||
);
|
||||
});
|
||||
|
||||
it("copies the model once for two actions, then deletes each copy as its declaration is removed", () => {
|
||||
cy.get<number>("@modelId").then((modelId) => {
|
||||
H.createImplicitAction({ model_id: modelId, kind: "create" }).then(
|
||||
({ body: create }) => {
|
||||
H.createImplicitAction({ model_id: modelId, kind: "update" }).then(
|
||||
({ body: update }) => {
|
||||
declareDataAppActions(APP_ROOT(), [create.id, update.id]);
|
||||
|
||||
sync();
|
||||
|
||||
copiedModels().then((models) => {
|
||||
expect(models).to.have.length(1);
|
||||
const copiedModelId = models[0].id;
|
||||
expect(copiedModelId).not.to.eq(modelId);
|
||||
|
||||
actionsOnModel(copiedModelId).then((actions) => {
|
||||
expect(actions).to.have.length(2);
|
||||
});
|
||||
});
|
||||
|
||||
// Injected back into source: the ID a production build runs.
|
||||
cy.readFile(`${APP_ROOT()}/actions/orders.action.ts`).should(
|
||||
"match",
|
||||
/copiedActionId: \d+/,
|
||||
);
|
||||
|
||||
cy.readFile(`${APP_ROOT()}/resources_metadata.json`).then(
|
||||
(lockfile) => {
|
||||
expect(lockfile.models).to.have.length(1);
|
||||
expect(lockfile.models[0].actions).to.have.length(2);
|
||||
},
|
||||
);
|
||||
|
||||
sync();
|
||||
copiedModels().then((models) => {
|
||||
expect(models).to.have.length(1);
|
||||
actionsOnModel(models[0].id).then((actions) => {
|
||||
expect(actions).to.have.length(2);
|
||||
});
|
||||
});
|
||||
|
||||
// The model survives: its sibling still needs it.
|
||||
removeDataAppActionDeclaration(APP_ROOT(), update.id);
|
||||
sync();
|
||||
|
||||
copiedModels().then((models) => {
|
||||
expect(models).to.have.length(1);
|
||||
actionsOnModel(models[0].id).then((actions) => {
|
||||
expect(actions).to.have.length(1);
|
||||
});
|
||||
});
|
||||
|
||||
removeDataAppActionDeclaration(APP_ROOT(), create.id);
|
||||
sync();
|
||||
|
||||
copiedModels().should("have.length", 0);
|
||||
cy.readFile(`${APP_ROOT()}/resources_metadata.json`).then(
|
||||
(lockfile) => {
|
||||
expect(lockfile.models).to.have.length(0);
|
||||
},
|
||||
);
|
||||
},
|
||||
);
|
||||
},
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
it("updates the copies in place when the source model or action changes", () => {
|
||||
cy.get<number>("@modelId").then((modelId) => {
|
||||
H.createImplicitAction({ model_id: modelId, kind: "create" }).then(
|
||||
({ body: action }) => {
|
||||
declareDataAppActions(APP_ROOT(), [action.id]);
|
||||
sync();
|
||||
|
||||
copiedModels().then(([copy]) => {
|
||||
cy.request("PUT", `/api/card/${modelId}`, {
|
||||
name: `${MODEL_NAME} renamed`,
|
||||
});
|
||||
cy.request("PUT", `/api/action/${action.id}`, {
|
||||
description: "now documented",
|
||||
});
|
||||
|
||||
sync();
|
||||
|
||||
copiedModels().then((models) => {
|
||||
expect(models).to.have.length(1);
|
||||
expect(models[0].id).to.eq(copy.id);
|
||||
expect(models[0].name).to.eq(`${MODEL_NAME} renamed`);
|
||||
|
||||
actionsOnModel(models[0].id).then((actions) => {
|
||||
expect(actions).to.have.length(1);
|
||||
expect(actions[0].description).to.eq("now documented");
|
||||
});
|
||||
});
|
||||
});
|
||||
},
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
it("leaves the copies untouched when nothing changed", () => {
|
||||
syncOneAction().then(({ copiedModel, copiedAction }) => {
|
||||
sync();
|
||||
|
||||
copiedModels().then(([model]) => {
|
||||
expect(model.id).to.eq(copiedModel.id);
|
||||
expect(model.updated_at, "the model copy was not rewritten").to.eq(
|
||||
copiedModel.updated_at,
|
||||
);
|
||||
|
||||
actionsOnModel(model.id).then(([action]) => {
|
||||
expect(action.id).to.eq(copiedAction.id);
|
||||
expect(
|
||||
action.updated_at,
|
||||
"the action copy was not rewritten",
|
||||
).to.eq(copiedAction.updated_at);
|
||||
});
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
it("restores a copied model edited directly in Metabase", () => {
|
||||
syncOneAction().then(({ copiedModel }) => {
|
||||
cy.request("PUT", `/api/card/${copiedModel.id}`, {
|
||||
name: "Edited by hand",
|
||||
});
|
||||
|
||||
sync();
|
||||
|
||||
copiedModels().then((models) => {
|
||||
expect(models).to.have.length(1);
|
||||
expect(models[0].id).to.eq(copiedModel.id);
|
||||
expect(models[0].name).to.eq(MODEL_NAME);
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
it("restores a copied action edited directly in Metabase", () => {
|
||||
syncOneAction().then(({ copiedModel, copiedAction }) => {
|
||||
cy.request("PUT", `/api/action/${copiedAction.id}`, {
|
||||
name: "Edited by hand",
|
||||
});
|
||||
|
||||
sync();
|
||||
|
||||
actionsOnModel(copiedModel.id).then((actions) => {
|
||||
expect(actions).to.have.length(1);
|
||||
expect(actions[0].id).to.eq(copiedAction.id);
|
||||
expect(actions[0].name).to.eq(copiedAction.name);
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe("permissions", () => {
|
||||
const joinAppGroup = () =>
|
||||
dataAppPermissionGroupId(APP_SLUG).then((groupId) => {
|
||||
addUserToGroup(groupId, USERS.normal.email);
|
||||
return cy.wrap(groupId, { log: false });
|
||||
});
|
||||
|
||||
// The copy is the whole point of the model copy: an app's viewers hold
|
||||
// read on the app's collection, so only the copy is reachable to them.
|
||||
it("lets the app's group execute the copy but not the action it was copied from", () => {
|
||||
syncOneAction().then(({ modelId, action, copiedAction }) => {
|
||||
// An action is readable through its model, and the fixture model sits
|
||||
// in the root collection the normal user's groups can read.
|
||||
moveDataAppModelToCollection({
|
||||
modelId,
|
||||
name: "Source models",
|
||||
access: "none",
|
||||
});
|
||||
joinAppGroup();
|
||||
|
||||
cy.signInAsNormalUser();
|
||||
cy.request({
|
||||
method: "POST",
|
||||
url: `/api/action/${copiedAction.id}/execute`,
|
||||
body: { parameters: { team_name: "Data App FC", score: 7 } },
|
||||
})
|
||||
.its("status")
|
||||
.should("be.oneOf", [200, 204]);
|
||||
|
||||
// The same request against the source, so the two are like for like.
|
||||
cy.request({
|
||||
method: "POST",
|
||||
url: `/api/action/${action.id}/execute`,
|
||||
body: { parameters: { team_name: "Data App FC", score: 7 } },
|
||||
failOnStatusCode: false,
|
||||
})
|
||||
.its("status")
|
||||
.should("eq", 403);
|
||||
|
||||
cy.request({
|
||||
url: `/api/action/${action.id}`,
|
||||
failOnStatusCode: false,
|
||||
})
|
||||
.its("status")
|
||||
.should("eq", 403);
|
||||
});
|
||||
});
|
||||
|
||||
it("reports a copy that was deleted in Metabase without being re-synced", () => {
|
||||
syncOneAction().then(({ copiedAction }) => {
|
||||
joinAppGroup();
|
||||
|
||||
// The source still names this copy, but a production bundle built
|
||||
// before the deletion keeps addressing it.
|
||||
cy.request("DELETE", `/api/action/${copiedAction.id}`);
|
||||
|
||||
cy.signInAsNormalUser();
|
||||
cy.request({
|
||||
method: "POST",
|
||||
url: `/api/action/${copiedAction.id}/execute`,
|
||||
body: { parameters: { team_name: "Data App FC", score: 7 } },
|
||||
failOnStatusCode: false,
|
||||
})
|
||||
.its("status")
|
||||
.should("eq", 404);
|
||||
});
|
||||
});
|
||||
|
||||
it("does not widen access to the model the copy was made from", () => {
|
||||
syncOneAction().then(({ modelId, copiedModel }) => {
|
||||
// The fixture model is created in the root collection, which the
|
||||
// normal user's groups can read, so close it before asking whether
|
||||
// the app's group opened anything.
|
||||
moveDataAppModelToCollection({
|
||||
modelId,
|
||||
name: "Source models",
|
||||
access: "none",
|
||||
});
|
||||
joinAppGroup();
|
||||
|
||||
cy.signInAsNormalUser();
|
||||
cy.request(`/api/card/${copiedModel.id}`)
|
||||
.its("body.id")
|
||||
.should("eq", copiedModel.id);
|
||||
cy.request({ url: `/api/card/${modelId}`, failOnStatusCode: false })
|
||||
.its("status")
|
||||
.should("eq", 403);
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe("recovery", () => {
|
||||
it("reuses the existing copies when copiedActionId is missing from the source", () => {
|
||||
syncOneAction().then(({ action, copiedModel, copiedAction }) => {
|
||||
// Rewriting the declarations drops the injected ID, as a bad merge would.
|
||||
declareDataAppActions(APP_ROOT(), [action.id]);
|
||||
sync();
|
||||
|
||||
actionsOnModel(copiedModel.id).then((actions) => {
|
||||
expect(actions).to.have.length(1);
|
||||
expect(actions[0].id, "the copy is reused, not replaced").to.eq(
|
||||
copiedAction.id,
|
||||
);
|
||||
});
|
||||
cy.readFile(`${APP_ROOT()}/actions/orders.action.ts`).should(
|
||||
"contain",
|
||||
`copiedActionId: ${copiedAction.id}`,
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
it("recreates the copied action but keeps its model when the action is deleted in Metabase", () => {
|
||||
syncOneAction().then(({ copiedModel, copiedAction }) => {
|
||||
cy.request("DELETE", `/api/action/${copiedAction.id}`);
|
||||
sync();
|
||||
|
||||
copiedModels().then((models) => {
|
||||
expect(models, "the model survives").to.have.length(1);
|
||||
expect(models[0].id).to.eq(copiedModel.id);
|
||||
|
||||
actionsOnModel(models[0].id).then((actions) => {
|
||||
expect(actions).to.have.length(1);
|
||||
expect(actions[0].id).not.to.eq(copiedAction.id);
|
||||
});
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
it("brings back a copied model that was trashed on its own", () => {
|
||||
syncOneAction().then(({ copiedModel }) => {
|
||||
cy.request("PUT", `/api/card/${copiedModel.id}`, { archived: true });
|
||||
|
||||
sync();
|
||||
|
||||
cy.request(`/api/card/${copiedModel.id}`).then(
|
||||
({ body: restored }) => {
|
||||
expect(restored.archived, "the copy is out of the trash").to.eq(
|
||||
false,
|
||||
);
|
||||
expect(restored.collection_id).to.eq(copiedModel.collection_id);
|
||||
},
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
it("recreates the copied model after it is deleted in Metabase", () => {
|
||||
syncOneAction().then(({ copiedModel }) => {
|
||||
cy.request("DELETE", `/api/card/${copiedModel.id}`);
|
||||
sync();
|
||||
|
||||
copiedModels().then((models) => {
|
||||
expect(models).to.have.length(1);
|
||||
expect(models[0].id).not.to.eq(copiedModel.id);
|
||||
});
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe("refusals", () => {
|
||||
// `GET /api/action/:id` filters archived actions out, so an archived source
|
||||
// is unreadable rather than readable-and-flagged.
|
||||
it("copies nothing when the action's model is in the trash", () => {
|
||||
declareOneAction().then(({ modelId }) => {
|
||||
cy.request("PUT", `/api/card/${modelId}`, { archived: true });
|
||||
|
||||
syncExpectingRefusal(`references an action on model ${modelId}`);
|
||||
copiedModels().should("have.length", 0);
|
||||
});
|
||||
});
|
||||
|
||||
it("copies nothing when a declared action is archived and cannot be read", () => {
|
||||
declareOneAction().then(({ action }) => {
|
||||
cy.request("PUT", `/api/action/${action.id}`, { archived: true });
|
||||
|
||||
syncExpectingRefusal(`Could not read action ${action.id}`);
|
||||
copiedModels().should("have.length", 0);
|
||||
});
|
||||
});
|
||||
|
||||
// Validation itself is unit-tested; what matters here is that a rejected
|
||||
// lockfile stops the CLI before it mutates anything.
|
||||
it("refuses to sync when resources_metadata.json is corrupt, leaving the copies alone", () => {
|
||||
syncOneAction().then(({ copiedModel, copiedAction }) => {
|
||||
cy.writeFile(`${APP_ROOT()}/resources_metadata.json`, "{ not json");
|
||||
|
||||
syncExpectingRefusal("Could not read resources_metadata.json");
|
||||
|
||||
copiedModels().then((models) => {
|
||||
expect(models, "the copies are left alone").to.have.length(1);
|
||||
expect(models[0].id).to.eq(copiedModel.id);
|
||||
|
||||
actionsOnModel(models[0].id).then((actions) => {
|
||||
expect(actions).to.have.length(1);
|
||||
expect(actions[0].id).to.eq(copiedAction.id);
|
||||
});
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
it("refuses to sync when a copied model was moved out of the app collection", () => {
|
||||
syncOneAction().then(({ copiedModel }) => {
|
||||
cy.request("POST", "/api/collection", { name: "Elsewhere" }).then(
|
||||
({ body: collection }) => {
|
||||
cy.request("PUT", `/api/card/${copiedModel.id}`, {
|
||||
collection_id: collection.id,
|
||||
});
|
||||
|
||||
syncExpectingRefusal(`Move card ${copiedModel.id} back to`);
|
||||
// Refusing is only worth anything if the card is left alone.
|
||||
cy.request(`/api/card/${copiedModel.id}`)
|
||||
.its("body.collection_id")
|
||||
.should("eq", collection.id);
|
||||
},
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
it("refuses to sync when a copied model was converted to a question", () => {
|
||||
syncOneAction().then(({ copiedModel }) => {
|
||||
cy.request("PUT", `/api/card/${copiedModel.id}`, {
|
||||
type: "question",
|
||||
});
|
||||
|
||||
syncExpectingRefusal("is no longer a model");
|
||||
});
|
||||
});
|
||||
|
||||
// Converting the parent model to a question deletes its implicit actions
|
||||
// and archives the rest (see `card.clj`), so the source becomes unreadable
|
||||
// rather than readable-with-a-non-model-parent.
|
||||
it("copies nothing when the source model was converted to a question", () => {
|
||||
declareOneAction().then(({ modelId, action }) => {
|
||||
cy.request("PUT", `/api/card/${modelId}`, { type: "question" });
|
||||
|
||||
syncExpectingRefusal(`Could not read action ${action.id}`);
|
||||
copiedModels().should("have.length", 0);
|
||||
});
|
||||
});
|
||||
|
||||
it("refuses to update a copied action that now belongs to another model", () => {
|
||||
syncOneAction().then(({ modelId, copiedAction }) => {
|
||||
// Still declared, but repointed at a model the app does not own:
|
||||
// replacing it would abandon an action nothing tracks.
|
||||
cy.request("PUT", `/api/action/${copiedAction.id}`, {
|
||||
model_id: modelId,
|
||||
});
|
||||
|
||||
syncExpectingRefusal(
|
||||
`Action ${copiedAction.id} is the copy of action`,
|
||||
);
|
||||
cy.request(`/api/action/${copiedAction.id}`)
|
||||
.its("body.model_id")
|
||||
.should("eq", modelId);
|
||||
});
|
||||
});
|
||||
|
||||
it("refuses to delete a copied action that now belongs to another model", () => {
|
||||
cy.get<number>("@modelId").then((modelId) => {
|
||||
H.createImplicitAction({ model_id: modelId, kind: "create" }).then(
|
||||
({ body: create }) => {
|
||||
H.createImplicitAction({
|
||||
model_id: modelId,
|
||||
kind: "update",
|
||||
}).then(({ body: update }) => {
|
||||
declareDataAppActions(APP_ROOT(), [create.id, update.id]);
|
||||
sync();
|
||||
|
||||
copiedModels().then(([copiedModel]) => {
|
||||
actionsOnModel(copiedModel.id).then((actions) => {
|
||||
const copyOfUpdate = actions.find(
|
||||
(candidate: { name: string }) =>
|
||||
candidate.name === "Update",
|
||||
);
|
||||
|
||||
// Repointed at the source model, which the app does not
|
||||
// own — deleting it would destroy someone else's action.
|
||||
cy.request("PUT", `/api/action/${copyOfUpdate.id}`, {
|
||||
model_id: modelId,
|
||||
});
|
||||
removeDataAppActionDeclaration(APP_ROOT(), update.id);
|
||||
|
||||
syncExpectingRefusal(
|
||||
`no longer hangs off copied model ${copiedModel.id}`,
|
||||
);
|
||||
cy.request(`/api/action/${copyOfUpdate.id}`)
|
||||
.its("status")
|
||||
.should("eq", 200);
|
||||
});
|
||||
});
|
||||
});
|
||||
},
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
it("does not copy models that are based on saved questions", () => {
|
||||
declareOneAction();
|
||||
|
||||
cy.log("create a saved question");
|
||||
cy.get<number>("@modelId")
|
||||
.then((modelId) => cy.request<Card>(`/api/card/${modelId}`))
|
||||
.then(({ body: model }) =>
|
||||
cy.request<Card>("POST", "/api/card", {
|
||||
name: "Scoreboard source question",
|
||||
display: "table",
|
||||
visualization_settings: {},
|
||||
dataset_query: model.dataset_query,
|
||||
}),
|
||||
)
|
||||
.its("body.id")
|
||||
.as("sourceQuestionId");
|
||||
|
||||
cy.log("change the model to use that saved question as its source");
|
||||
cy.get<number>("@sourceQuestionId").then((sourceQuestionId) => {
|
||||
cy.get<number>("@modelId").then((modelId) => {
|
||||
cy.request("PUT", `/api/card/${modelId}`, {
|
||||
dataset_query: {
|
||||
database: WRITABLE_DB_ID,
|
||||
type: "query",
|
||||
query: { "source-table": `card__${sourceQuestionId}` },
|
||||
},
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
cy.request("/api/action").its("body").as("actionsBeforeSync");
|
||||
|
||||
syncExpectingRefusal(
|
||||
"depends on saved questions and cannot be synchronized",
|
||||
);
|
||||
|
||||
cy.log("sync should not copy models that are based on saved questions");
|
||||
copiedModels().should("have.length", 0);
|
||||
|
||||
cy.get("@actionsBeforeSync").then((actions) => {
|
||||
cy.request("/api/action").its("body").should("deep.equal", actions);
|
||||
});
|
||||
});
|
||||
});
|
||||
},
|
||||
);
|
||||
722
e2e/test-host-app/data-apps/sync-resources.cy.spec.ts
Normal file
722
e2e/test-host-app/data-apps/sync-resources.cy.spec.ts
Normal file
@@ -0,0 +1,722 @@
|
||||
import { USERS, USER_GROUPS } from "e2e/support/cypress_data";
|
||||
import { SAMPLE_DATABASE } from "e2e/support/cypress_sample_database";
|
||||
import {
|
||||
addUserToGroup,
|
||||
buildDataAppHostApp,
|
||||
createDataAppApiKey,
|
||||
createSecondDataApp,
|
||||
dataAppHostAppRoot,
|
||||
dataAppPermissionGroupId,
|
||||
declareDataAppQueries,
|
||||
removeDataAppQueryDeclaration,
|
||||
resetDataAppHostAppSources,
|
||||
setDataAppCollectionAccess,
|
||||
syncDataAppResources,
|
||||
} from "e2e/support/helpers";
|
||||
import type { Card, DataApp } from "metabase-types/api";
|
||||
|
||||
const { H } = cy;
|
||||
const { ORDERS_ID } = SAMPLE_DATABASE;
|
||||
|
||||
/** `syncResources` takes the app's slug from its directory name. */
|
||||
const APP_SLUG = "vite-6-data-app-host-app";
|
||||
|
||||
/** The fields these specs read off a card the app collection holds. */
|
||||
type AppCard = { id: number; name: string; collection_id: number | null };
|
||||
|
||||
const APP_ROOT = () => dataAppHostAppRoot();
|
||||
const LOCKFILE = () => `${APP_ROOT()}/resources_metadata.json`;
|
||||
const QUERIES_FILE = () => `${APP_ROOT()}/queries/orders.query.ts`;
|
||||
const MANIFEST_FILE = () => `${APP_ROOT()}/data_app.yaml`;
|
||||
|
||||
const AUTHORED_MANIFEST = `name: Vite 6 Data App
|
||||
path: ./dist/index.js
|
||||
allowed_hosts:
|
||||
- https://allowed.data-app.test
|
||||
`;
|
||||
|
||||
/**
|
||||
* The query half of `sync-resources`, run against the dev host app: a real vite
|
||||
* data app with the published SDK installed, so `defineQuery` resolves through
|
||||
* the package an author actually consumes rather than a stub.
|
||||
*/
|
||||
describe("Embedding SDK: data-app sync-resources (queries)", () => {
|
||||
const resetHostAppSources = () => {
|
||||
resetDataAppHostAppSources();
|
||||
cy.writeFile(MANIFEST_FILE(), AUTHORED_MANIFEST);
|
||||
};
|
||||
|
||||
beforeEach(() => {
|
||||
H.restore();
|
||||
cy.signInAsAdmin();
|
||||
H.activateToken("bleeding-edge");
|
||||
|
||||
// The query and actions specs share a checked-in host app, so start clean.
|
||||
resetHostAppSources();
|
||||
createDataAppApiKey().as("apiKey");
|
||||
});
|
||||
|
||||
after(() => {
|
||||
resetHostAppSources();
|
||||
});
|
||||
|
||||
const sync = () =>
|
||||
cy.get<string>("@apiKey").then((apiKey) => {
|
||||
syncDataAppResources(apiKey, APP_ROOT()).then(({ ok, error }) => {
|
||||
expect(error, "sync-resources failed").to.eq(null);
|
||||
expect(ok).to.eq(true);
|
||||
});
|
||||
});
|
||||
|
||||
const syncExpectingRefusal = (message: string) =>
|
||||
cy.get<string>("@apiKey").then((apiKey) => {
|
||||
syncDataAppResources(apiKey, APP_ROOT()).should(({ ok, error }) => {
|
||||
expect(ok, "sync-resources should have refused").to.eq(false);
|
||||
expect(error).to.contain(message);
|
||||
});
|
||||
});
|
||||
|
||||
/** The saved questions Metabase holds in the app's own collection. */
|
||||
const savedQuestions = () =>
|
||||
cy.request<DataApp>(`/api/apps/${APP_SLUG}`).then(({ body: app }) => {
|
||||
expect(app.resource_collection_id, "resource collection").to.be.a(
|
||||
"number",
|
||||
);
|
||||
return cy
|
||||
.request<{
|
||||
data: AppCard[];
|
||||
}>(`/api/collection/${app.resource_collection_id}/items?models=card`)
|
||||
.then(({ body }) => body.data);
|
||||
});
|
||||
|
||||
/** Declares one query and syncs it, handing back the card Metabase created. */
|
||||
const syncOneQuery = () => {
|
||||
declareDataAppQueries(APP_ROOT(), [{ name: "Orders", tableId: ORDERS_ID }]);
|
||||
sync();
|
||||
return savedQuestions().then(([card]) => cy.wrap(card, { log: false }));
|
||||
};
|
||||
|
||||
it("names the definition whose query Metabase could not resolve", () => {
|
||||
declareDataAppQueries(APP_ROOT(), [{ name: "Orders", tableId: 999999 }]);
|
||||
|
||||
syncExpectingRefusal("Could not resolve queries/orders.query.ts:Orders");
|
||||
|
||||
savedQuestions().should("have.length", 0);
|
||||
});
|
||||
|
||||
it("creates the saved question, writes its ID back, and re-syncs without changes", () => {
|
||||
declareDataAppQueries(APP_ROOT(), [
|
||||
{ name: "Orders", tableId: ORDERS_ID, limit: 5 },
|
||||
]);
|
||||
|
||||
sync();
|
||||
|
||||
savedQuestions().then((cards) => {
|
||||
expect(cards).to.have.length(1);
|
||||
|
||||
cy.readFile(QUERIES_FILE()).should(
|
||||
"contain",
|
||||
`savedQuestionSourceId: ${cards[0].id}`,
|
||||
);
|
||||
cy.readFile(LOCKFILE()).then((lockfile) => {
|
||||
expect(lockfile.queries).to.have.length(1);
|
||||
expect(lockfile.queries[0].savedQuestionSourceId).to.eq(cards[0].id);
|
||||
});
|
||||
|
||||
sync();
|
||||
savedQuestions().then((after) => {
|
||||
expect(after).to.have.length(1);
|
||||
expect(after[0].id).to.eq(cards[0].id);
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
it("deletes the saved question when its declaration is removed", () => {
|
||||
syncOneQuery().then(() => {
|
||||
removeDataAppQueryDeclaration(APP_ROOT(), "Orders");
|
||||
sync();
|
||||
|
||||
savedQuestions().should("have.length", 0);
|
||||
cy.readFile(LOCKFILE()).then((lockfile) => {
|
||||
expect(lockfile.queries).to.have.length(0);
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
// Removing a declaration deletes the copy and its lockfile entry, but the ID the
|
||||
// sync wrote into the source stays in source control. Bringing the declaration
|
||||
// back has to make a fresh copy: there is no card left to adopt.
|
||||
it("re-creates the copy when a removed declaration comes back naming the deleted card", () => {
|
||||
syncOneQuery().then((card) => {
|
||||
cy.readFile(QUERIES_FILE()).then((authored: string) => {
|
||||
expect(
|
||||
authored,
|
||||
"the sync wrote the copy's ID into the source",
|
||||
).to.contain(`savedQuestionSourceId: ${card.id}`);
|
||||
|
||||
removeDataAppQueryDeclaration(APP_ROOT(), "Orders");
|
||||
sync();
|
||||
savedQuestions().should("have.length", 0);
|
||||
|
||||
// Exactly what restoring the file from source control brings back.
|
||||
cy.writeFile(QUERIES_FILE(), authored);
|
||||
sync();
|
||||
|
||||
savedQuestions().then((recreated) => {
|
||||
expect(recreated).to.have.length(1);
|
||||
expect(
|
||||
recreated[0].id,
|
||||
"the deleted card is not resurrected",
|
||||
).not.to.eq(card.id);
|
||||
cy.readFile(QUERIES_FILE())
|
||||
.should("contain", `savedQuestionSourceId: ${recreated[0].id}`)
|
||||
.should("not.contain", `savedQuestionSourceId: ${card.id}`);
|
||||
});
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
it("names the trash when a restored declaration points at a trashed copy", () => {
|
||||
syncOneQuery().then((card) => {
|
||||
cy.readFile(QUERIES_FILE()).then((authored: string) => {
|
||||
cy.request("PUT", `/api/card/${card.id}`, { archived: true });
|
||||
|
||||
removeDataAppQueryDeclaration(APP_ROOT(), "Orders");
|
||||
sync();
|
||||
cy.readFile(LOCKFILE()).then((lockfile) => {
|
||||
expect(lockfile.queries, "the entry is dropped").to.have.length(0);
|
||||
});
|
||||
|
||||
cy.writeFile(QUERIES_FILE(), authored);
|
||||
syncExpectingRefusal(`card ${card.id}, which is in the trash`);
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
it("restores a hand-edited card's name instead of replacing the card", () => {
|
||||
syncOneQuery().then((card) => {
|
||||
cy.request("PUT", `/api/card/${card.id}`, { name: "Renamed by hand" });
|
||||
|
||||
sync();
|
||||
|
||||
savedQuestions().then((cards) => {
|
||||
expect(cards, "the card is updated in place").to.have.length(1);
|
||||
expect(cards[0].id).to.eq(card.id);
|
||||
expect(cards[0].name).to.eq("Orders");
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
it("brings back a saved question that was trashed on its own", () => {
|
||||
syncOneQuery().then((card) => {
|
||||
cy.request("PUT", `/api/card/${card.id}`, { archived: true });
|
||||
|
||||
sync();
|
||||
|
||||
cy.request(`/api/card/${card.id}`).then(({ body: restored }) => {
|
||||
expect(restored.archived, "the copy is out of the trash").to.eq(false);
|
||||
expect(restored.collection_id).to.eq(card.collection_id);
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe("trashed metric copies", () => {
|
||||
beforeEach(() => {
|
||||
H.createQuestion({
|
||||
name: "Count of orders",
|
||||
type: "metric",
|
||||
query: { "source-table": ORDERS_ID, aggregation: [["count"]] },
|
||||
display: "scalar",
|
||||
})
|
||||
.its("body.id")
|
||||
.as("sourceMetricId");
|
||||
|
||||
cy.log("define a query that uses metrics");
|
||||
cy.get<number>("@sourceMetricId").then((id) => {
|
||||
cy.writeFile(
|
||||
QUERIES_FILE(),
|
||||
[
|
||||
'import { defineQuery } from "@metabase/embedding-sdk-react/data-app";',
|
||||
|
||||
"export const OrdersCount = defineQuery({",
|
||||
` source: { type: "table", id: ${ORDERS_ID} },`,
|
||||
` aggregations: [{ type: "metric", id: ${id}, sourceTableId: ${ORDERS_ID} }],`,
|
||||
"});",
|
||||
].join("\n"),
|
||||
);
|
||||
});
|
||||
|
||||
sync();
|
||||
|
||||
cy.readFile(LOCKFILE()).then((lockfile) => {
|
||||
expect(lockfile.metrics).to.have.length(1);
|
||||
expect(lockfile.queries).to.have.length(1);
|
||||
|
||||
cy.wrap(lockfile.metrics[0].copiedMetricId).as("copiedMetricId");
|
||||
cy.wrap(lockfile.queries[0].savedQuestionSourceId).as("questionId");
|
||||
});
|
||||
|
||||
cy.log("archive the copied metric");
|
||||
cy.get<number>("@copiedMetricId").then((id) => {
|
||||
cy.request<Card>(`/api/card/${id}`).its("body").as("metricCopy");
|
||||
|
||||
H.archiveQuestion(id);
|
||||
});
|
||||
|
||||
cy.get<Card>("@metricCopy").then((copy) => {
|
||||
cy.log("the copied metric is archived");
|
||||
|
||||
cy.request<Card>(`/api/card/${copy.id}`).should(({ body }) => {
|
||||
expect(body.archived).to.eq(true);
|
||||
expect(body.collection_id).not.to.eq(copy.collection_id);
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
it("restores a trashed metric copy after sync", () => {
|
||||
sync();
|
||||
|
||||
cy.get<Card>("@metricCopy").then((copy) => {
|
||||
cy.log("the archived metric should no longer be archived after sync");
|
||||
|
||||
cy.request<Card>(`/api/card/${copy.id}`).should(({ body }) => {
|
||||
expect(body.archived).to.eq(false);
|
||||
expect(body.collection_id).to.eq(copy.collection_id);
|
||||
});
|
||||
|
||||
cy.log("the copied metric in the lockfile remains the same");
|
||||
cy.readFile(LOCKFILE())
|
||||
.its("metrics.0.copiedMetricId")
|
||||
.should("eq", copy.id);
|
||||
});
|
||||
|
||||
cy.get<number>("@questionId").then((id) => H.visitQuestion(id));
|
||||
|
||||
H.assertQueryBuilderRowCount(1);
|
||||
});
|
||||
|
||||
it("leaves a trashed metric copy in trash after its source is removed", () => {
|
||||
H.removeDataAppQueryDeclaration(APP_ROOT(), "OrdersCount");
|
||||
|
||||
sync();
|
||||
|
||||
cy.get<number>("@copiedMetricId").then((id) => {
|
||||
cy.log("the copied metric should remain archived after sync");
|
||||
cy.request<Card>(`/api/card/${id}`)
|
||||
.its("body.archived")
|
||||
.should("eq", true);
|
||||
});
|
||||
|
||||
cy.readFile(LOCKFILE()).should((lockfile) => {
|
||||
expect(lockfile.metrics).to.deep.eq([]);
|
||||
expect(lockfile.queries).to.deep.eq([]);
|
||||
});
|
||||
|
||||
cy.get<number>("@sourceMetricId").then((id) => {
|
||||
cy.log("the source metric should not be archived");
|
||||
cy.request<Card>(`/api/card/${id}`)
|
||||
.its("body.archived")
|
||||
.should("eq", false);
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
it("recreates the saved question after it is deleted in Metabase", () => {
|
||||
syncOneQuery().then((card) => {
|
||||
cy.request("DELETE", `/api/card/${card.id}`);
|
||||
|
||||
sync();
|
||||
|
||||
savedQuestions().then((cards) => {
|
||||
expect(cards).to.have.length(1);
|
||||
expect(cards[0].id).not.to.eq(card.id);
|
||||
});
|
||||
cy.readFile(QUERIES_FILE()).should("not.contain", `Id: ${card.id}`);
|
||||
});
|
||||
});
|
||||
|
||||
it("restores savedQuestionSourceId from the lockfile when the source loses it", () => {
|
||||
syncOneQuery().then((card) => {
|
||||
// Rewriting the declaration drops the injected ID, as a bad merge would.
|
||||
declareDataAppQueries(APP_ROOT(), [
|
||||
{ name: "Orders", tableId: ORDERS_ID },
|
||||
]);
|
||||
|
||||
sync();
|
||||
|
||||
savedQuestions().then((cards) => {
|
||||
expect(cards, "the card is reused, not replaced").to.have.length(1);
|
||||
expect(cards[0].id).to.eq(card.id);
|
||||
});
|
||||
cy.readFile(QUERIES_FILE()).should(
|
||||
"contain",
|
||||
`savedQuestionSourceId: ${card.id}`,
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
it("rebuilds a missing lockfile entry from the ID left in the source", () => {
|
||||
syncOneQuery().then((card) => {
|
||||
cy.writeFile(LOCKFILE(), { queries: [], models: [] });
|
||||
|
||||
sync();
|
||||
|
||||
// The inline ID proves ownership, so the entry is rebuilt.
|
||||
savedQuestions().then((cards) => {
|
||||
expect(cards).to.have.length(1);
|
||||
expect(cards[0].id).to.eq(card.id);
|
||||
});
|
||||
cy.readFile(LOCKFILE()).then((lockfile) => {
|
||||
expect(lockfile.queries).to.have.length(1);
|
||||
expect(lockfile.queries[0].savedQuestionSourceId).to.eq(card.id);
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
// A card in the trash reports the Trash as its collection, so one moved out
|
||||
// and trashed afterwards is indistinguishable from the app's own. Deleting is
|
||||
// permanent, so neither is deleted.
|
||||
it("leaves a trashed card in the trash when its declaration is removed", () => {
|
||||
syncOneQuery().then((card) => {
|
||||
cy.request("PUT", `/api/card/${card.id}`, { archived: true });
|
||||
removeDataAppQueryDeclaration(APP_ROOT(), "Orders");
|
||||
|
||||
sync();
|
||||
|
||||
cy.request(`/api/card/${card.id}`)
|
||||
.its("body.archived")
|
||||
.should("eq", true);
|
||||
cy.readFile(LOCKFILE()).then((lockfile) => {
|
||||
expect(lockfile.queries, "no longer tracked").to.have.length(0);
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
it("refuses to delete a card that was moved out of the app collection", () => {
|
||||
syncOneQuery().then((card) => {
|
||||
cy.request("POST", "/api/collection", { name: "Elsewhere" }).then(
|
||||
({ body: collection }) => {
|
||||
cy.request("PUT", `/api/card/${card.id}`, {
|
||||
collection_id: collection.id,
|
||||
});
|
||||
removeDataAppQueryDeclaration(APP_ROOT(), "Orders");
|
||||
|
||||
syncExpectingRefusal(`Move card ${card.id} back to`);
|
||||
cy.request(`/api/card/${card.id}`)
|
||||
.its("body.collection_id")
|
||||
.should("eq", collection.id);
|
||||
},
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
// Trashing the collection archives everything in it, and the app is served
|
||||
// from those copies, so a sync that reports success has not actually recovered.
|
||||
it("brings the app collection and its copies back out of the trash", () => {
|
||||
syncOneQuery().then((card) => {
|
||||
cy.request(`/api/apps/${APP_SLUG}`).then(({ body: app }) => {
|
||||
cy.request("PUT", `/api/collection/${app.resource_collection_id}`, {
|
||||
archived: true,
|
||||
});
|
||||
cy.request(`/api/card/${card.id}`)
|
||||
.its("body.archived")
|
||||
.should("eq", true);
|
||||
|
||||
sync();
|
||||
|
||||
cy.request(`/api/collection/${app.resource_collection_id}`)
|
||||
.its("body.archived")
|
||||
.should("eq", false);
|
||||
cy.request(`/api/card/${card.id}`)
|
||||
.its("body.archived")
|
||||
.should("eq", false);
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe("the app's lifecycle", () => {
|
||||
it("takes the collection and the group with it when the app is removed", () => {
|
||||
syncOneQuery().then((card) => {
|
||||
dataAppPermissionGroupId(APP_SLUG).then((groupId) => {
|
||||
cy.request(`/api/apps/${APP_SLUG}`).then(({ body: app }) => {
|
||||
cy.request("DELETE", `/api/apps/${APP_SLUG}`);
|
||||
|
||||
// Nothing is left for a former viewer to reach.
|
||||
cy.request({
|
||||
url: `/api/collection/${app.resource_collection_id}`,
|
||||
failOnStatusCode: false,
|
||||
})
|
||||
.its("status")
|
||||
.should("eq", 404);
|
||||
cy.request({
|
||||
url: `/api/permissions/group/${groupId}`,
|
||||
failOnStatusCode: false,
|
||||
})
|
||||
.its("status")
|
||||
.should("eq", 404);
|
||||
cy.request({ url: `/api/card/${card.id}`, failOnStatusCode: false })
|
||||
.its("status")
|
||||
.should("eq", 404);
|
||||
});
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
// Everything the CLI drives is superuser-gated, starting with the draft it
|
||||
// asks for first, so a key that is not an admin's gets nowhere.
|
||||
it("refuses to synchronize at all for a key that is not an admin's", () => {
|
||||
declareDataAppQueries(APP_ROOT(), [
|
||||
{ name: "Orders", tableId: ORDERS_ID },
|
||||
]);
|
||||
|
||||
cy.request("POST", "/api/api-key", {
|
||||
name: `data-app-sync-e2e-non-admin-${Date.now()}`,
|
||||
group_id: USER_GROUPS.COLLECTION_GROUP,
|
||||
}).then(({ body: key }) => {
|
||||
syncDataAppResources(key.unmasked_key, APP_ROOT()).should(
|
||||
({ ok, error }) => {
|
||||
expect(ok, "sync-resources should have refused").to.eq(false);
|
||||
expect(error).to.contain("403");
|
||||
expect(error, "refused at the first call it makes").to.contain(
|
||||
`/api/apps/${APP_SLUG}/draft`,
|
||||
);
|
||||
},
|
||||
);
|
||||
|
||||
// The draft call is the one that creates the app, so a refusal leaves
|
||||
// nothing behind at all — not even the app to hold copies.
|
||||
cy.request({ url: `/api/apps/${APP_SLUG}`, failOnStatusCode: false })
|
||||
.its("status")
|
||||
.should("eq", 404);
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe("two apps on one instance", () => {
|
||||
const OTHER_SLUG = "sync-resources-second-app";
|
||||
|
||||
it("keeps each app's copies in its own collection, reachable only by its own group", () => {
|
||||
const otherRoot = createSecondDataApp(OTHER_SLUG);
|
||||
|
||||
syncOneQuery().then((card) => {
|
||||
declareDataAppQueries(otherRoot, [
|
||||
{ name: "OtherOrders", tableId: ORDERS_ID },
|
||||
]);
|
||||
cy.get<string>("@apiKey").then((apiKey) => {
|
||||
syncDataAppResources(apiKey, otherRoot).then(({ error }) => {
|
||||
expect(error, "second app sync failed").to.eq(null);
|
||||
});
|
||||
});
|
||||
|
||||
cy.request(`/api/apps/${OTHER_SLUG}`).then(({ body: otherApp }) => {
|
||||
dataAppPermissionGroupId(APP_SLUG).then((groupId) => {
|
||||
expect(
|
||||
otherApp.resource_collection_id,
|
||||
"each app gets its own collection",
|
||||
).not.to.eq(null);
|
||||
expect(otherApp.permission_group_id).not.to.eq(groupId);
|
||||
|
||||
// Joining one app's group must not reach the other app's copy.
|
||||
addUserToGroup(groupId, USERS.normal.email);
|
||||
|
||||
cy.request(
|
||||
`/api/collection/${otherApp.resource_collection_id}/items?models=card`,
|
||||
).then(({ body }) => {
|
||||
const otherCard = body.data[0];
|
||||
expect(otherCard.id, "the apps hold different cards").not.to.eq(
|
||||
card.id,
|
||||
);
|
||||
|
||||
cy.signInAsNormalUser();
|
||||
cy.request(`/api/card/${card.id}`)
|
||||
.its("body.id")
|
||||
.should("eq", card.id);
|
||||
cy.request({
|
||||
url: `/api/card/${otherCard.id}`,
|
||||
failOnStatusCode: false,
|
||||
})
|
||||
.its("status")
|
||||
.should("eq", 403);
|
||||
});
|
||||
});
|
||||
});
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe("the build guard", () => {
|
||||
/** `cy.exec` reports the command's output; vite prints this only on success. */
|
||||
const expectBuildToSucceed = () =>
|
||||
buildDataAppHostApp().should((result) => {
|
||||
expect(`${result.stdout}${result.stderr}`).to.contain("built in");
|
||||
});
|
||||
|
||||
// `metabase-resource-sync-check` runs on buildStart, so a stale app is
|
||||
// refused before it can be bundled and served.
|
||||
it("refuses to build when the source and the lockfile disagree", () => {
|
||||
syncOneQuery().then(() => {
|
||||
expectBuildToSucceed();
|
||||
|
||||
// A hand-edited declaration no longer matches its lockfile entry.
|
||||
declareDataAppQueries(APP_ROOT(), [
|
||||
{ name: "Orders", tableId: ORDERS_ID, limit: 3 },
|
||||
]);
|
||||
|
||||
buildDataAppHostApp().should((result) => {
|
||||
const output = `${result.stdout}${result.stderr}`;
|
||||
expect(output, "the build is refused").to.contain(
|
||||
"is not synchronized",
|
||||
);
|
||||
expect(output).not.to.contain("built in");
|
||||
});
|
||||
|
||||
// Synchronizing makes it buildable again.
|
||||
sync();
|
||||
expectBuildToSucceed();
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe("permissions", () => {
|
||||
/** Puts the normal user in the app's group, as granting app access does. */
|
||||
const joinAppGroup = () =>
|
||||
dataAppPermissionGroupId(APP_SLUG).then((groupId) => {
|
||||
addUserToGroup(groupId, USERS.normal.email);
|
||||
return cy.wrap(groupId, { log: false });
|
||||
});
|
||||
|
||||
it("does not let a viewer modify the copy it can read", () => {
|
||||
syncOneQuery().then((card) => {
|
||||
joinAppGroup();
|
||||
|
||||
cy.signInAsNormalUser();
|
||||
cy.request(`/api/card/${card.id}`).its("body.id").should("eq", card.id);
|
||||
|
||||
// The grant is collection read, never readwrite.
|
||||
cy.request({
|
||||
method: "PUT",
|
||||
url: `/api/card/${card.id}`,
|
||||
body: { name: "Renamed by a viewer" },
|
||||
failOnStatusCode: false,
|
||||
})
|
||||
.its("status")
|
||||
.should("eq", 403);
|
||||
cy.request({
|
||||
method: "DELETE",
|
||||
url: `/api/card/${card.id}`,
|
||||
failOnStatusCode: false,
|
||||
})
|
||||
.its("status")
|
||||
.should("eq", 403);
|
||||
});
|
||||
});
|
||||
|
||||
// The app collection is server-owned, so each sync reasserts exclusive
|
||||
// access — including over a grant an admin made deliberately.
|
||||
it("takes back access an admin granted another group on the app collection", () => {
|
||||
syncOneQuery().then((card) => {
|
||||
cy.request(`/api/apps/${APP_SLUG}`).then(({ body: app }) => {
|
||||
setDataAppCollectionAccess(app.resource_collection_id, "read");
|
||||
|
||||
cy.signInAsNormalUser();
|
||||
cy.request(`/api/card/${card.id}`)
|
||||
.its("body.id")
|
||||
.should("eq", card.id);
|
||||
|
||||
cy.signInAsAdmin();
|
||||
sync();
|
||||
|
||||
cy.signInAsNormalUser();
|
||||
cy.request({ url: `/api/card/${card.id}`, failOnStatusCode: false })
|
||||
.its("status")
|
||||
.should("eq", 403);
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
it("grants the group nothing beyond the app's own collection", () => {
|
||||
cy.request("POST", "/api/collection", { name: "Private" }).then(
|
||||
({ body: collection }) => {
|
||||
// The normal user belongs to groups that can read new collections, so
|
||||
// close this one: the claim under test is that joining the app group
|
||||
// grants the app's collection and nothing else.
|
||||
setDataAppCollectionAccess(collection.id, "none");
|
||||
|
||||
H.createQuestion({
|
||||
name: "Private question",
|
||||
query: { "source-table": ORDERS_ID },
|
||||
collection_id: collection.id,
|
||||
}).then(({ body: unrelated }) => {
|
||||
syncOneQuery().then((card) => {
|
||||
joinAppGroup();
|
||||
|
||||
cy.signInAsNormalUser();
|
||||
cy.request(`/api/card/${card.id}`)
|
||||
.its("body.id")
|
||||
.should("eq", card.id);
|
||||
cy.request({
|
||||
url: `/api/card/${unrelated.id}`,
|
||||
failOnStatusCode: false,
|
||||
})
|
||||
.its("status")
|
||||
.should("eq", 403);
|
||||
});
|
||||
});
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
it("leaves the group with nothing once the declaration is removed", () => {
|
||||
syncOneQuery().then((card) => {
|
||||
joinAppGroup();
|
||||
|
||||
cy.signInAsNormalUser();
|
||||
cy.request(`/api/card/${card.id}`).its("body.id").should("eq", card.id);
|
||||
|
||||
cy.signInAsAdmin();
|
||||
removeDataAppQueryDeclaration(APP_ROOT(), "Orders");
|
||||
sync();
|
||||
|
||||
// The copy is deleted with its declaration, so there is nothing left to
|
||||
// read — the group's collection grant now covers an empty collection.
|
||||
cy.signInAsNormalUser();
|
||||
cy.request({ url: `/api/card/${card.id}`, failOnStatusCode: false })
|
||||
.its("status")
|
||||
.should("eq", 404);
|
||||
});
|
||||
});
|
||||
|
||||
// The copy exists so an app's viewers can read it: they are given the app's
|
||||
// own group, which holds read on the app's collection and nothing else.
|
||||
it("lets the app's group read the copy while the rest of the instance cannot", () => {
|
||||
syncOneQuery().then((card) => {
|
||||
dataAppPermissionGroupId(APP_SLUG).then((groupId) => {
|
||||
cy.signInAsNormalUser();
|
||||
cy.request({ url: `/api/card/${card.id}`, failOnStatusCode: false })
|
||||
.its("status")
|
||||
.should("eq", 403);
|
||||
|
||||
cy.signInAsAdmin();
|
||||
addUserToGroup(groupId, USERS.normal.email);
|
||||
|
||||
cy.signInAsNormalUser();
|
||||
cy.request(`/api/card/${card.id}`)
|
||||
.its("body.id")
|
||||
.should("eq", card.id);
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
// `read-check-data-app` 403s a non-member, and the app view turns that status
|
||||
// into its own empty state instead of loading the bundle.
|
||||
it("tells a viewer outside the group that the app is not theirs to open", () => {
|
||||
syncOneQuery().then(() => {
|
||||
cy.signInAsNormalUser();
|
||||
cy.visit(`/apps/${APP_SLUG}`);
|
||||
|
||||
cy.findByText("You don’t have access to this data app").should("exist");
|
||||
cy.get("iframe").should("not.exist");
|
||||
});
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -27,7 +27,8 @@ const normalUserName = H.getFullName(normal);
|
||||
|
||||
describe("scenarios > admin > people", () => {
|
||||
beforeEach(() => {
|
||||
cy.intercept("GET", "/api/permissions/group").as("getGroups");
|
||||
// `*` matches the `?tenancy=…` variants (People tab, user modals), not just the param-less Groups-tab request.
|
||||
cy.intercept("GET", "/api/permissions/group*").as("getGroups");
|
||||
cy.intercept("GET", "/api/api-key").as("listApiKeys");
|
||||
H.restore();
|
||||
cy.signInAsAdmin();
|
||||
|
||||
@@ -2,8 +2,10 @@ import { WRITABLE_DB_ID } from "e2e/support/cypress_data";
|
||||
import {
|
||||
DATA_APP_DISPLAY_NAME as APP_DISPLAY_NAME,
|
||||
DATA_APP_NAME as APP_NAME,
|
||||
moveDataAppModelToCollection,
|
||||
visitDataAppRoute as visitAppRoute,
|
||||
} from "e2e/support/helpers";
|
||||
import type { CollectionPermission } from "metabase-types/api";
|
||||
|
||||
import { DATA_APP_TEST_ENV as TEST_ENV } from "./helpers";
|
||||
|
||||
@@ -12,6 +14,7 @@ const { H } = cy;
|
||||
const TEST_TABLE = "scoreboard_actions";
|
||||
const MODEL_NAME = "Scoreboard model";
|
||||
const EXISTING_TEAM = "Amorous Aardvarks";
|
||||
const RESOURCE_COLLECTION = "Data App resources";
|
||||
|
||||
describe(
|
||||
"scenarios > data apps > actions (useAction)",
|
||||
@@ -32,7 +35,7 @@ describe(
|
||||
});
|
||||
|
||||
const setupActionsApp = (actionParams: Record<string, string | number>) =>
|
||||
cy.get("@modelId").then((modelId) => {
|
||||
cy.get<number>("@modelId").then((modelId) => {
|
||||
H.createImplicitAction({
|
||||
model_id: modelId,
|
||||
kind: "create",
|
||||
@@ -46,7 +49,9 @@ describe(
|
||||
|
||||
it("executes the action, exposing isExecuting and result, and resets", () => {
|
||||
cy.intercept("POST", "/api/action/*/execute", (req) => {
|
||||
req.on("response", (res) => res.setDelay(300));
|
||||
req.on("response", (res) => {
|
||||
res.setDelay(300);
|
||||
});
|
||||
});
|
||||
|
||||
setupActionsApp({ team_name: "Data App FC", score: 7 });
|
||||
@@ -99,5 +104,78 @@ describe(
|
||||
expect(rows).to.have.length(1);
|
||||
});
|
||||
});
|
||||
|
||||
describe("collection permissions", () => {
|
||||
const setupRestrictedApp = (access: CollectionPermission) =>
|
||||
cy.get<number>("@modelId").then((modelId) => {
|
||||
moveDataAppModelToCollection({
|
||||
modelId,
|
||||
name: RESOURCE_COLLECTION,
|
||||
access,
|
||||
});
|
||||
|
||||
H.createImplicitAction({
|
||||
model_id: modelId,
|
||||
kind: "create",
|
||||
}).then(({ body: action }) => {
|
||||
H.mockDataApp(APP_NAME, {
|
||||
displayName: APP_DISPLAY_NAME,
|
||||
testEnv: {
|
||||
...TEST_ENV,
|
||||
actionId: action.id,
|
||||
actionParams: { team_name: "Data App FC", score: 7 },
|
||||
},
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
const assertNothingWritten = () =>
|
||||
H.queryWritableDB(
|
||||
`SELECT team_name FROM ${TEST_TABLE} WHERE team_name = 'Data App FC'`,
|
||||
).then(({ rows }) => {
|
||||
expect(rows).to.have.length(0);
|
||||
});
|
||||
|
||||
it("refuses the action when the user cannot read its collection", () => {
|
||||
setupRestrictedApp("none");
|
||||
cy.signInAsNormalUser();
|
||||
|
||||
visitAppRoute("actions");
|
||||
H.dataAppIframe(APP_DISPLAY_NAME).within(() => {
|
||||
cy.findByTestId("action-execute").click();
|
||||
|
||||
cy.findByTestId("action-error", { timeout: 30000 }).should(
|
||||
"have.text",
|
||||
"has-error",
|
||||
);
|
||||
cy.findByTestId("action-result").should("have.text", "no-result");
|
||||
});
|
||||
|
||||
assertNothingWritten();
|
||||
});
|
||||
|
||||
it("executes the action when the user has view access to its collection", () => {
|
||||
setupRestrictedApp("read");
|
||||
cy.signInAsNormalUser();
|
||||
|
||||
visitAppRoute("actions");
|
||||
H.dataAppIframe(APP_DISPLAY_NAME).within(() => {
|
||||
cy.findByTestId("action-execute").click();
|
||||
|
||||
cy.findByTestId("action-result", { timeout: 30000 }).should(
|
||||
"have.text",
|
||||
"has-result",
|
||||
);
|
||||
cy.findByTestId("action-error").should("have.text", "no-error");
|
||||
});
|
||||
|
||||
H.queryWritableDB(
|
||||
`SELECT team_name, score FROM ${TEST_TABLE} WHERE team_name = 'Data App FC'`,
|
||||
).then(({ rows }) => {
|
||||
expect(rows).to.have.length(1);
|
||||
expect(rows[0].score).to.eq(7);
|
||||
});
|
||||
});
|
||||
});
|
||||
},
|
||||
);
|
||||
|
||||
@@ -51,6 +51,25 @@ describe("scenarios > data apps > admin management", () => {
|
||||
});
|
||||
});
|
||||
|
||||
it("keeps a data app's permission group out of the admin Groups list", () => {
|
||||
// Provisioning a data app draft creates its permission group as a side effect.
|
||||
cy.request("POST", "/api/apps/orders-app/draft").then(({ body }) => {
|
||||
const dataAppGroupId = body.permission_group_id;
|
||||
|
||||
// The groups API does not return data-app groups.
|
||||
cy.request("GET", "/api/permissions/group").then(({ body: groups }) => {
|
||||
const ids = groups.map((group: { id: number }) => group.id);
|
||||
expect(ids).not.to.include(dataAppGroupId);
|
||||
});
|
||||
|
||||
cy.visit("/admin/people/groups");
|
||||
cy.findByTestId("admin-panel").within(() => {
|
||||
cy.findByText("All Users").should("be.visible");
|
||||
cy.findByText("Data App: orders-app").should("not.exist");
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
it("dismisses the promo banner and keeps it hidden across a reload", () => {
|
||||
cy.intercept("GET", "/api/apps/repo-status", { configured: true });
|
||||
cy.intercept("GET", "/api/apps", []);
|
||||
|
||||
@@ -1,9 +1,8 @@
|
||||
import type {
|
||||
LocalFieldReference,
|
||||
QuestionColumnReference,
|
||||
} from "@metabase/embedding-sdk-react/data-app";
|
||||
|
||||
import { SAMPLE_DATABASE } from "e2e/support/cypress_sample_database";
|
||||
import {
|
||||
ORDERS_COUNT_QUESTION_ID,
|
||||
ORDERS_QUESTION_ID,
|
||||
} from "e2e/support/cypress_sample_instance_data";
|
||||
import type { DataAppTestEnv } from "e2e/support/helpers";
|
||||
|
||||
// These live here (co-located with the specs) rather than in
|
||||
@@ -14,41 +13,21 @@ import type { DataAppTestEnv } from "e2e/support/helpers";
|
||||
|
||||
const { ORDERS_ID } = SAMPLE_DATABASE;
|
||||
|
||||
/**
|
||||
* A numeric field dimension for the query-builder combinators, hand-built the way
|
||||
* an app without a generated `metabase.data.ts` schema would build one.
|
||||
*/
|
||||
export const dataAppNumericField = (
|
||||
fieldId: number,
|
||||
name: string,
|
||||
): LocalFieldReference => ({
|
||||
type: "column",
|
||||
fieldId,
|
||||
tableId: ORDERS_ID,
|
||||
name,
|
||||
jsType: "number",
|
||||
});
|
||||
|
||||
/**
|
||||
* The same dimension as it is addressed on a saved-question source: a card stage
|
||||
* exposes the question's result columns, which are matched by name and carry no
|
||||
* table-scoped identity.
|
||||
*/
|
||||
export const dataAppNumericResultColumn = (
|
||||
name: string,
|
||||
): QuestionColumnReference => ({
|
||||
type: "column",
|
||||
name,
|
||||
jsType: "number",
|
||||
});
|
||||
|
||||
const source = { type: "table" as const, id: ORDERS_ID };
|
||||
|
||||
/** The `testEnv` the fixture's Overview page reads (Orders count + question). */
|
||||
/**
|
||||
* The `testEnv` the fixture's Overview page reads (Orders count + question).
|
||||
*
|
||||
* Both carry a `savedQuestionSourceId`, because a deployed app runs the card its
|
||||
* query was published as — a bare table source is refused. Each id names a
|
||||
* snapshot question equivalent to the authored query, since the swap drops the
|
||||
* static clauses the card already contains.
|
||||
*/
|
||||
export const DATA_APP_TEST_ENV: DataAppTestEnv = {
|
||||
scalarQuery: {
|
||||
source,
|
||||
aggregations: [{ type: "operator", operator: "count", args: [] }],
|
||||
savedQuestionSourceId: ORDERS_COUNT_QUESTION_ID,
|
||||
},
|
||||
questionQuery: { source },
|
||||
questionQuery: { source, savedQuestionSourceId: ORDERS_QUESTION_ID },
|
||||
};
|
||||
|
||||
155
e2e/test/scenarios/data-apps/legacy-permissions.cy.spec.ts
Normal file
155
e2e/test/scenarios/data-apps/legacy-permissions.cy.spec.ts
Normal file
@@ -0,0 +1,155 @@
|
||||
import { SAMPLE_DB_ID, USERS } from "e2e/support/cypress_data";
|
||||
import { SAMPLE_DATABASE } from "e2e/support/cypress_sample_database";
|
||||
import {
|
||||
ALL_USERS_GROUP_ID,
|
||||
COLLECTION_GROUP_ID,
|
||||
} from "e2e/support/cypress_sample_instance_data";
|
||||
import {
|
||||
type DataApp,
|
||||
DataPermission,
|
||||
DataPermissionValue,
|
||||
type FieldsPermissions,
|
||||
type SchemasPermissions,
|
||||
type Table,
|
||||
} from "metabase-types/api";
|
||||
|
||||
const { H } = cy;
|
||||
const { ORDERS_ID } = SAMPLE_DATABASE;
|
||||
|
||||
const APP_NAME = "legacy-permission-test";
|
||||
|
||||
// Exercise real queries: preserving the permission graph alone does not prove that data still loads.
|
||||
describe("scenarios > data apps > legacy view data permissions", () => {
|
||||
describe("preserves 'no self-service' access level when a user is added to data app group", () => {
|
||||
beforeEach(() => {
|
||||
H.restore();
|
||||
cy.signInAsAdmin();
|
||||
H.activateToken("bleeding-edge");
|
||||
|
||||
// TODO(GHY-4493) Direct saved questions currently reject legacy access after backend hardening
|
||||
// In the meantime, use a saved SQL source to test app membership separately from the regression.
|
||||
H.createNativeQuestion({
|
||||
name: "Orders SQL source",
|
||||
database: SAMPLE_DB_ID,
|
||||
native: { query: "SELECT COUNT(*) AS total FROM ORDERS" },
|
||||
collection_id: null,
|
||||
})
|
||||
.its("body.id")
|
||||
.as("sourceQuestionId");
|
||||
|
||||
cy.get<number>("@sourceQuestionId")
|
||||
.then((sourceQuestionId) =>
|
||||
H.createQuestion({
|
||||
name: "Orders with legacy access",
|
||||
database: SAMPLE_DB_ID,
|
||||
query: { "source-table": `card__${sourceQuestionId}` },
|
||||
display: "scalar",
|
||||
collection_id: null,
|
||||
}),
|
||||
)
|
||||
.its("body.id")
|
||||
.as("questionId");
|
||||
|
||||
setViewDataPermissions(DataPermissionValue.BLOCKED);
|
||||
|
||||
cy.signIn("nodata");
|
||||
H.visitQuestion("@questionId");
|
||||
H.main()
|
||||
.findByText("Sorry, you don't have permission to run this query.")
|
||||
.should("be.visible");
|
||||
cy.findByTestId("scalar-value").should("not.exist");
|
||||
|
||||
cy.signInAsAdmin();
|
||||
});
|
||||
|
||||
it("with all tables using no self-service", () => {
|
||||
// every table is no self-service
|
||||
setViewDataPermissions(DataPermissionValue.LEGACY_NO_SELF_SERVICE);
|
||||
|
||||
assertAccessSurvivesAppMembership();
|
||||
});
|
||||
|
||||
it("with some tables being unrestricted", () => {
|
||||
// Clear the database block, including tables omitted from query metadata.
|
||||
setViewDataPermissions(DataPermissionValue.LEGACY_NO_SELF_SERVICE);
|
||||
|
||||
cy.request<{ tables: Table[] }>(
|
||||
"GET",
|
||||
`/api/database/${SAMPLE_DB_ID}/metadata`,
|
||||
).then(({ body: { tables } }) => {
|
||||
const permissions: Record<
|
||||
string,
|
||||
Record<number, FieldsPermissions>
|
||||
> = {};
|
||||
|
||||
// orders table is no self-service, other tables are unrestricted
|
||||
for (const table of tables) {
|
||||
const schema = table.schema ?? "";
|
||||
permissions[schema] ??= {};
|
||||
|
||||
permissions[schema][Number(table.id)] =
|
||||
table.id === ORDERS_ID
|
||||
? DataPermissionValue.LEGACY_NO_SELF_SERVICE
|
||||
: DataPermissionValue.UNRESTRICTED;
|
||||
}
|
||||
|
||||
setViewDataPermissions(permissions);
|
||||
});
|
||||
|
||||
assertAccessSurvivesAppMembership();
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
function assertAccessSurvivesAppMembership() {
|
||||
cy.log("legacy access should let the no data user load the saved question");
|
||||
cy.signIn("nodata");
|
||||
H.visitQuestion("@questionId");
|
||||
|
||||
cy.findByTestId("scalar-value", { timeout: 20_000 }).should(
|
||||
"have.text",
|
||||
"18,760",
|
||||
);
|
||||
|
||||
cy.signInAsAdmin();
|
||||
cy.request<DataApp>("POST", `/api/apps/${APP_NAME}/draft`)
|
||||
.its("body")
|
||||
.as("app");
|
||||
|
||||
cy.log("assign the no data user to the data app group");
|
||||
cy.get<DataApp>("@app").then(({ permission_group_id }) => {
|
||||
if (permission_group_id === null) {
|
||||
throw new Error("data app must have a permission group");
|
||||
}
|
||||
|
||||
H.addUserToGroup(permission_group_id, USERS.nodata.email);
|
||||
});
|
||||
|
||||
cy.log("the same saved question must still load after app membership");
|
||||
cy.signIn("nodata");
|
||||
H.visitQuestion("@questionId");
|
||||
|
||||
cy.findByTestId("scalar-value", { timeout: 20_000 }).should(
|
||||
"have.text",
|
||||
"18,760",
|
||||
);
|
||||
}
|
||||
|
||||
function setViewDataPermissions(viewData: SchemasPermissions) {
|
||||
// The no-data user belongs to both groups. Neither should override the legacy grant.
|
||||
cy.updatePermissionsGraph({
|
||||
[ALL_USERS_GROUP_ID]: {
|
||||
[SAMPLE_DB_ID]: {
|
||||
[DataPermission.VIEW_DATA]: viewData,
|
||||
[DataPermission.CREATE_QUERIES]: DataPermissionValue.NO,
|
||||
},
|
||||
},
|
||||
|
||||
[COLLECTION_GROUP_ID]: {
|
||||
[SAMPLE_DB_ID]: {
|
||||
[DataPermission.VIEW_DATA]: viewData,
|
||||
[DataPermission.CREATE_QUERIES]: DataPermissionValue.NO,
|
||||
},
|
||||
},
|
||||
});
|
||||
}
|
||||
@@ -6,13 +6,9 @@ import {
|
||||
visitDataAppRoute as visitAppRoute,
|
||||
} from "e2e/support/helpers";
|
||||
|
||||
import {
|
||||
DATA_APP_TEST_ENV as TEST_ENV,
|
||||
dataAppNumericField as numericField,
|
||||
dataAppNumericResultColumn as resultColumn,
|
||||
} from "./helpers";
|
||||
import { DATA_APP_TEST_ENV as TEST_ENV } from "./helpers";
|
||||
|
||||
const { ORDERS_ID, ORDERS } = SAMPLE_DATABASE;
|
||||
const { ORDERS_ID } = SAMPLE_DATABASE;
|
||||
|
||||
const { H } = cy;
|
||||
|
||||
@@ -30,8 +26,12 @@ describe("scenarios > data apps > SDK runtime", () => {
|
||||
displayName: APP_DISPLAY_NAME,
|
||||
testEnv: {
|
||||
...TEST_ENV,
|
||||
// A table id that doesn't exist → the query resolves to an error.
|
||||
errorQuery: { source: { type: "table", id: 999999 } },
|
||||
// A card that doesn't exist → the query resolves to an error, rather
|
||||
// than to the refusal an unsynchronized table source would raise.
|
||||
errorQuery: {
|
||||
source: { type: "table", id: ORDERS_ID },
|
||||
savedQuestionSourceId: 999999,
|
||||
},
|
||||
},
|
||||
});
|
||||
|
||||
@@ -105,90 +105,6 @@ describe("scenarios > data apps > SDK runtime", () => {
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
it("builds a query with filter/breakout/orderBy/aggregations helpers", () => {
|
||||
H.mockDataApp(APP_NAME, {
|
||||
displayName: APP_DISPLAY_NAME,
|
||||
testEnv: {
|
||||
...TEST_ENV,
|
||||
combinators: {
|
||||
source: { type: "table", id: ORDERS_ID },
|
||||
filterField: numericField(ORDERS.TOTAL, "TOTAL"),
|
||||
filterValue: 50,
|
||||
breakoutField: numericField(ORDERS.PRODUCT_ID, "PRODUCT_ID"),
|
||||
},
|
||||
},
|
||||
});
|
||||
|
||||
visitAppRoute("combinators");
|
||||
H.dataAppIframe(APP_DISPLAY_NAME).within(() => {
|
||||
cy.findByTestId("combinators-loading", { timeout: 30000 }).should(
|
||||
"have.text",
|
||||
"done",
|
||||
);
|
||||
cy.findByTestId("combinators-error").should("have.text", "no-error");
|
||||
// `.should(callback)` retries until the query resolves with rows.
|
||||
cy.findByTestId("combinators-rowcount", { timeout: 30000 }).should(
|
||||
($el) => {
|
||||
expect(Number($el.text())).to.be.greaterThan(0);
|
||||
},
|
||||
);
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe("saved question sources", () => {
|
||||
const setupCardSourceApp = () =>
|
||||
H.createQuestion({
|
||||
name: "Orders as a data-app query source",
|
||||
query: { "source-table": ORDERS_ID },
|
||||
}).then(({ body: card }) => {
|
||||
H.mockDataApp(APP_NAME, {
|
||||
displayName: APP_DISPLAY_NAME,
|
||||
testEnv: {
|
||||
...TEST_ENV,
|
||||
cardSource: {
|
||||
source: { type: "card", id: card.id },
|
||||
tableSource: { type: "table", id: ORDERS_ID },
|
||||
filterField: numericField(ORDERS.TOTAL, "TOTAL"),
|
||||
filterColumn: resultColumn("TOTAL"),
|
||||
filterValue: 50,
|
||||
breakoutField: numericField(ORDERS.PRODUCT_ID, "PRODUCT_ID"),
|
||||
breakoutColumn: resultColumn("PRODUCT_ID"),
|
||||
},
|
||||
},
|
||||
});
|
||||
|
||||
visitAppRoute("card-source");
|
||||
});
|
||||
|
||||
it("queries a saved question as a source", () => {
|
||||
setupCardSourceApp();
|
||||
|
||||
H.dataAppIframe(APP_DISPLAY_NAME).within(() => {
|
||||
cy.findByTestId("card-source-case-source", {
|
||||
timeout: 30000,
|
||||
}).should("have.text", "match");
|
||||
|
||||
// A match on two empty results would be vacuous, so the source has to
|
||||
// have actually returned rows.
|
||||
cy.findByTestId("card-source-total").should(($el) => {
|
||||
expect(Number($el.text())).to.be.greaterThan(0);
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
it("applies filters, aggregations, and breakouts on top of a saved question", () => {
|
||||
setupCardSourceApp();
|
||||
|
||||
H.dataAppIframe(APP_DISPLAY_NAME).within(() => {
|
||||
["filters", "aggregations", "breakouts"].forEach((clause) => {
|
||||
cy.findByTestId(`card-source-case-${clause}`, {
|
||||
timeout: 30000,
|
||||
}).should("have.text", "match");
|
||||
});
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe("clipboard (copy)", () => {
|
||||
|
||||
@@ -0,0 +1,153 @@
|
||||
import { USERS, WRITABLE_DB_ID } from "e2e/support/cypress_data";
|
||||
import {
|
||||
addUserToGroup,
|
||||
createDataAppApiKey,
|
||||
dataAppIframe,
|
||||
dataAppPermissionGroupId,
|
||||
mockDataApp,
|
||||
syncDataAppResources,
|
||||
} from "e2e/support/helpers";
|
||||
|
||||
const { H } = cy;
|
||||
|
||||
const TEST_TABLE = "scoreboard_actions";
|
||||
const MODEL_NAME = "Scoreboard model";
|
||||
|
||||
const APP_SLUG = "synced-actions-app";
|
||||
const APP_DISPLAY_NAME = "Synced Actions App";
|
||||
|
||||
const APP_ROOT = () =>
|
||||
`${Cypress.config("projectRoot")}/e2e/support/assets/data-apps/${APP_SLUG}`;
|
||||
const ACTION_FILE = () => `${APP_ROOT()}/actions/orders.action.ts`;
|
||||
const MANIFEST_FILE = () => `${APP_ROOT()}/data_app.yaml`;
|
||||
const LOCKFILE = () => `${APP_ROOT()}/resources_metadata.json`;
|
||||
|
||||
/** The manifest as source control holds it. */
|
||||
const AUTHORED_MANIFEST = `name: ${APP_DISPLAY_NAME}\npath: ./dist/index.js\n`;
|
||||
|
||||
/** The declaration as source control holds it, with no generated ID yet. */
|
||||
const declaration = (sourceActionId: number) =>
|
||||
[
|
||||
'import { defineAction } from "@metabase/embedding-sdk-react/data-app";',
|
||||
"",
|
||||
"export const CreateScore = defineAction({",
|
||||
` action: { id: ${sourceActionId}, parameters: [] },`,
|
||||
"});",
|
||||
"",
|
||||
].join("\n");
|
||||
|
||||
/**
|
||||
* The action half of the production path. Outside the dev preview
|
||||
* `toExecutableActionId` runs `copiedActionId` — the copy hanging off the copied
|
||||
* model — because that is the only action an app's viewers may execute.
|
||||
*/
|
||||
describe(
|
||||
"scenarios > data apps > sync-resources in production (actions)",
|
||||
{ tags: ["@external", "@actions"] },
|
||||
() => {
|
||||
// Restore the source files and remove the lockfile before each synchronization.
|
||||
const restoreAuthoredFixture = () => {
|
||||
cy.writeFile(ACTION_FILE(), declaration(1));
|
||||
cy.writeFile(MANIFEST_FILE(), AUTHORED_MANIFEST);
|
||||
cy.task("removeDataAppPaths", { paths: [LOCKFILE()] });
|
||||
};
|
||||
|
||||
beforeEach(() => {
|
||||
H.restore("postgres-writable");
|
||||
cy.signInAsAdmin();
|
||||
H.activateToken("bleeding-edge");
|
||||
|
||||
H.resetTestTable({ type: "postgres", table: TEST_TABLE });
|
||||
H.resyncDatabase({ dbId: WRITABLE_DB_ID, tableName: TEST_TABLE });
|
||||
H.setActionsEnabledForDB(WRITABLE_DB_ID);
|
||||
H.createModelFromTableName({
|
||||
tableName: TEST_TABLE,
|
||||
modelName: MODEL_NAME,
|
||||
});
|
||||
|
||||
restoreAuthoredFixture();
|
||||
createDataAppApiKey().as("apiKey");
|
||||
});
|
||||
|
||||
after(() => {
|
||||
restoreAuthoredFixture();
|
||||
});
|
||||
|
||||
/** Declares the model's action, synchronizes, and returns both action IDs. */
|
||||
const syncApp = () =>
|
||||
cy.get<number>("@modelId").then((modelId) =>
|
||||
H.createImplicitAction({ model_id: modelId, kind: "create" }).then(
|
||||
({ body: action }) => {
|
||||
cy.writeFile(ACTION_FILE(), declaration(action.id));
|
||||
|
||||
return cy.get<string>("@apiKey").then((apiKey) =>
|
||||
syncDataAppResources(apiKey, APP_ROOT()).then(({ ok, error }) => {
|
||||
expect(error, "sync-resources failed").to.eq(null);
|
||||
expect(ok).to.eq(true);
|
||||
|
||||
return cy
|
||||
.readFile(`${APP_ROOT()}/resources_metadata.json`)
|
||||
.then((lockfile) => {
|
||||
const copiedActionId =
|
||||
lockfile.models?.[0]?.actions?.[0]?.copiedActionId;
|
||||
|
||||
if (typeof copiedActionId !== "number") {
|
||||
throw new Error(
|
||||
"The sync wrote no action to the lockfile.",
|
||||
);
|
||||
}
|
||||
|
||||
return cy.wrap(
|
||||
{ sourceActionId: action.id, copiedActionId },
|
||||
{ log: false },
|
||||
);
|
||||
});
|
||||
}),
|
||||
);
|
||||
},
|
||||
),
|
||||
);
|
||||
|
||||
it("executes the synchronized copy rather than the authored action", () => {
|
||||
syncApp().then(({ sourceActionId, copiedActionId }) => {
|
||||
expect(copiedActionId).not.to.eq(sourceActionId);
|
||||
|
||||
cy.intercept("POST", "/api/action/*/execute").as("execute");
|
||||
mockDataApp(APP_SLUG, { displayName: APP_DISPLAY_NAME });
|
||||
cy.visit(`/apps/${APP_SLUG}`);
|
||||
|
||||
dataAppIframe(APP_DISPLAY_NAME).within(() => {
|
||||
cy.findByTestId("action-execute", { timeout: 30000 }).click();
|
||||
cy.findByTestId("action-output").should("have.text", "executed");
|
||||
});
|
||||
|
||||
// The proof that production took the synchronized path.
|
||||
cy.wait("@execute")
|
||||
.its("request.url")
|
||||
.should("contain", `/api/action/${copiedActionId}/execute`);
|
||||
});
|
||||
});
|
||||
|
||||
it("lets a member of the app's group execute it", () => {
|
||||
syncApp().then(({ copiedActionId }) => {
|
||||
dataAppPermissionGroupId(APP_SLUG).then((groupId) => {
|
||||
addUserToGroup(groupId, USERS.normal.email);
|
||||
|
||||
cy.signInAsNormalUser();
|
||||
cy.intercept("POST", "/api/action/*/execute").as("execute");
|
||||
mockDataApp(APP_SLUG, { displayName: APP_DISPLAY_NAME });
|
||||
cy.visit(`/apps/${APP_SLUG}`);
|
||||
|
||||
dataAppIframe(APP_DISPLAY_NAME).within(() => {
|
||||
cy.findByTestId("action-execute", { timeout: 30000 }).click();
|
||||
cy.findByTestId("action-output").should("have.text", "executed");
|
||||
});
|
||||
|
||||
cy.wait("@execute")
|
||||
.its("request.url")
|
||||
.should("contain", `/api/action/${copiedActionId}/execute`);
|
||||
});
|
||||
});
|
||||
});
|
||||
},
|
||||
);
|
||||
@@ -0,0 +1,184 @@
|
||||
import { SAMPLE_DB_ID, USERS } from "e2e/support/cypress_data";
|
||||
import { SAMPLE_DATABASE } from "e2e/support/cypress_sample_database";
|
||||
import {
|
||||
addUserToGroup,
|
||||
createDataAppApiKey,
|
||||
dataAppIframe,
|
||||
dataAppPermissionGroupId,
|
||||
mockDataApp,
|
||||
syncDataAppResources,
|
||||
} from "e2e/support/helpers";
|
||||
|
||||
const { H } = cy;
|
||||
const { ORDERS, ORDERS_ID } = SAMPLE_DATABASE;
|
||||
|
||||
const APP_SLUG = "synced-app";
|
||||
const APP_DISPLAY_NAME = "Synced App";
|
||||
|
||||
const APP_ROOT = () =>
|
||||
`${Cypress.config("projectRoot")}/e2e/support/assets/data-apps/${APP_SLUG}`;
|
||||
const QUERY_FILE = () => `${APP_ROOT()}/queries/orders.query.ts`;
|
||||
const MANIFEST_FILE = () => `${APP_ROOT()}/data_app.yaml`;
|
||||
const LOCKFILE = () => `${APP_ROOT()}/resources_metadata.json`;
|
||||
|
||||
/** The manifest as source control holds it. */
|
||||
const AUTHORED_MANIFEST = `name: ${APP_DISPLAY_NAME}\npath: ./dist/index.js\n`;
|
||||
|
||||
/** The declaration as source control holds it, with no generated ID yet. */
|
||||
const AUTHORED_DECLARATION = [
|
||||
"import {",
|
||||
" aggregations,",
|
||||
" breakout,",
|
||||
" defineQuery,",
|
||||
'} from "@metabase/embedding-sdk-react/data-app";',
|
||||
"",
|
||||
"/**",
|
||||
" * Authored state, as it sits in source control before a sync. `id` is the sample",
|
||||
" * database's ORDERS table (`SAMPLE_DATABASE.ORDERS_ID`); the spec writes this file",
|
||||
" * from the same constant, so a snapshot change fails loudly rather than silently.",
|
||||
" * Synchronization writes `savedQuestionSourceId` in here, and the spec restores",
|
||||
" * this file afterwards.",
|
||||
" */",
|
||||
"const OrdersUserId = {",
|
||||
' type: "column" as const,',
|
||||
` fieldId: ${ORDERS.USER_ID},`,
|
||||
` tableId: ${ORDERS_ID},`,
|
||||
' name: "USER_ID",',
|
||||
' displayName: "User ID",',
|
||||
' jsType: "number",',
|
||||
"};",
|
||||
"",
|
||||
"export const OrdersCount = defineQuery({",
|
||||
` source: { type: "table", id: ${ORDERS_ID} },`,
|
||||
" aggregations: [aggregations.count()],",
|
||||
" breakouts: [breakout(OrdersUserId)],",
|
||||
"});",
|
||||
].join("\n");
|
||||
|
||||
/**
|
||||
* What a shipped data app actually does: outside the dev preview `isDataAppDev()`
|
||||
* is false, so the SDK addresses the synchronized copy — the only resource an
|
||||
* app's viewers are permitted to read — rather than the authored source.
|
||||
*/
|
||||
describe("scenarios > data apps > sync-resources in production", () => {
|
||||
// Restore the source files and remove the lockfile before each synchronization.
|
||||
const restoreAuthoredFixture = () => {
|
||||
cy.writeFile(QUERY_FILE(), `${AUTHORED_DECLARATION}\n`);
|
||||
cy.writeFile(MANIFEST_FILE(), AUTHORED_MANIFEST);
|
||||
cy.task("removeDataAppPaths", { paths: [LOCKFILE()] });
|
||||
};
|
||||
|
||||
beforeEach(() => {
|
||||
H.restore();
|
||||
cy.signInAsAdmin();
|
||||
H.activateToken("bleeding-edge");
|
||||
|
||||
restoreAuthoredFixture();
|
||||
createDataAppApiKey().as("apiKey");
|
||||
});
|
||||
|
||||
after(() => {
|
||||
restoreAuthoredFixture();
|
||||
});
|
||||
|
||||
/** Synchronizes the fixture and returns the card the app must now address. */
|
||||
const syncApp = () =>
|
||||
cy.get<string>("@apiKey").then((apiKey) =>
|
||||
syncDataAppResources(apiKey, APP_ROOT()).then(({ ok, error }) => {
|
||||
expect(error, "sync-resources failed").to.eq(null);
|
||||
expect(ok).to.eq(true);
|
||||
|
||||
return cy
|
||||
.readFile(`${APP_ROOT()}/resources_metadata.json`)
|
||||
.then((lockfile) => {
|
||||
const cardId = lockfile.queries?.[0]?.savedQuestionSourceId;
|
||||
|
||||
if (typeof cardId !== "number") {
|
||||
throw new Error("The sync wrote no query entry to the lockfile.");
|
||||
}
|
||||
|
||||
return cy.wrap(cardId, { log: false });
|
||||
});
|
||||
}),
|
||||
);
|
||||
|
||||
it("runs the synchronized card rather than the authored table query", () => {
|
||||
syncApp().then((cardId) => {
|
||||
cy.readFile(QUERY_FILE()).should(
|
||||
"contain",
|
||||
`savedQuestionSourceId: ${cardId}`,
|
||||
);
|
||||
|
||||
cy.intercept("POST", "/api/dataset").as("dataset");
|
||||
mockDataApp(APP_SLUG, { displayName: APP_DISPLAY_NAME });
|
||||
cy.visit(`/apps/${APP_SLUG}`);
|
||||
|
||||
dataAppIframe(APP_DISPLAY_NAME).within(() => {
|
||||
cy.findByTestId("synced-app-total", { timeout: 30000 }).should(
|
||||
($total) => {
|
||||
expect(Number($total.text())).to.be.greaterThan(0);
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
// The proof that production took the synchronized path: the query runs
|
||||
// against the copied card, not the table the declaration names.
|
||||
cy.wait("@dataset").then(({ request }) => {
|
||||
const [stage] = request.body.stages ?? [];
|
||||
expect(stage?.["source-card"], "runs the synchronized card").to.eq(
|
||||
cardId,
|
||||
);
|
||||
expect(stage?.["source-table"], "not the authored table").to.be
|
||||
.undefined;
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
// The swap is only safe if both sides return the same thing. The dev preview
|
||||
// runs the authored query unswapped; production runs the card it was published
|
||||
// as. A deployed app cannot run the authored query at all, so the two sides are
|
||||
// captured separately rather than side by side.
|
||||
it("returns the same rows from the published card as from the authored query", () => {
|
||||
syncApp().then((cardId) => {
|
||||
cy.request("POST", "/api/dataset", {
|
||||
type: "query",
|
||||
database: SAMPLE_DB_ID,
|
||||
query: {
|
||||
"source-table": ORDERS_ID,
|
||||
aggregation: [["count"]],
|
||||
breakout: [["field", ORDERS.USER_ID, null]],
|
||||
},
|
||||
}).then(({ body: authored }) => {
|
||||
cy.request("POST", `/api/card/${cardId}/query`).then(
|
||||
({ body: published }) => {
|
||||
expect(published.data.rows).to.deep.eq(authored.data.rows);
|
||||
expect(
|
||||
published.data.rows[0][1],
|
||||
"a match on two empty results would be vacuous",
|
||||
).to.be.greaterThan(0);
|
||||
},
|
||||
);
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
it("serves the app to a member of its permission group", () => {
|
||||
syncApp().then(() => {
|
||||
dataAppPermissionGroupId(APP_SLUG).then((groupId) => {
|
||||
addUserToGroup(groupId, USERS.normal.email);
|
||||
|
||||
cy.signInAsNormalUser();
|
||||
mockDataApp(APP_SLUG, { displayName: APP_DISPLAY_NAME });
|
||||
cy.visit(`/apps/${APP_SLUG}`);
|
||||
|
||||
dataAppIframe(APP_DISPLAY_NAME).within(() => {
|
||||
cy.findByTestId("synced-app-total", { timeout: 30000 }).should(
|
||||
($total) => {
|
||||
expect(Number($total.text())).to.be.greaterThan(0);
|
||||
},
|
||||
);
|
||||
});
|
||||
});
|
||||
});
|
||||
});
|
||||
});
|
||||
359
e2e/test/scenarios/data-apps/user-access.cy.spec.ts
Normal file
359
e2e/test/scenarios/data-apps/user-access.cy.spec.ts
Normal file
@@ -0,0 +1,359 @@
|
||||
import { SAMPLE_DB_ID, USERS } from "e2e/support/cypress_data";
|
||||
import { SAMPLE_DATABASE } from "e2e/support/cypress_sample_database";
|
||||
import {
|
||||
ALL_USERS_GROUP_ID,
|
||||
COLLECTION_GROUP_ID,
|
||||
DATA_GROUP_ID,
|
||||
} from "e2e/support/cypress_sample_instance_data";
|
||||
import {
|
||||
type DataApp,
|
||||
DataPermission,
|
||||
DataPermissionValue,
|
||||
type Table,
|
||||
} from "metabase-types/api";
|
||||
|
||||
const { H } = cy;
|
||||
|
||||
const DATA_APP_NAME = "user-access-test";
|
||||
|
||||
const { ORDERS_ID, PRODUCTS_ID } = SAMPLE_DATABASE;
|
||||
|
||||
const NORMAL_USER_NAME = `${USERS.normal.first_name} ${USERS.normal.last_name}`;
|
||||
const NODATA_USER_NAME = `${USERS.nodata.first_name} ${USERS.nodata.last_name}`;
|
||||
|
||||
describe("scenarios > data apps > user access (EMB-2328)", () => {
|
||||
beforeEach(() => {
|
||||
H.restore();
|
||||
cy.signInAsAdmin();
|
||||
H.activateToken("bleeding-edge");
|
||||
|
||||
cy.request<DataApp>("POST", `/api/apps/${DATA_APP_NAME}/draft`).then(
|
||||
({ body: app }) => {
|
||||
expect(app.permission_group_id).not.to.be.null;
|
||||
|
||||
cy.wrap(app.permission_group_id, { log: false }).as("dataAppGroupId");
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
it("adds and removes a data app user by pasting a single email", () => {
|
||||
cy.request("PUT", `/api/apps/${DATA_APP_NAME}/table-dependencies`, {
|
||||
table_ids: [],
|
||||
});
|
||||
|
||||
cy.visit("/admin/settings/apps");
|
||||
openManageUserAccessFromAppRow();
|
||||
|
||||
cy.location("pathname").should(
|
||||
"eq",
|
||||
`/admin/settings/apps/${DATA_APP_NAME}/users`,
|
||||
);
|
||||
|
||||
H.main().within(() => {
|
||||
cy.findByRole("link", { name: "Data apps" }).should("be.visible");
|
||||
cy.findByText(DATA_APP_NAME).should("be.visible");
|
||||
cy.findByText("No one has access yet").should("be.visible");
|
||||
|
||||
cy.findByRole("heading", { name: "Manage access to this app" }).should(
|
||||
"be.visible",
|
||||
);
|
||||
});
|
||||
|
||||
cy.findByRole("button", { name: "Add users" }).click();
|
||||
H.popover().findByText(NORMAL_USER_NAME).should("be.visible");
|
||||
|
||||
cy.findByRole("textbox", { name: "Search for a user to add" })
|
||||
.paste(` ${USERS.normal.email.toUpperCase()} `)
|
||||
.should("have.value", "");
|
||||
|
||||
cy.findByRole("button", { name: "Add" }).click();
|
||||
|
||||
userRow(USERS.normal.email).should("be.visible");
|
||||
cy.reload();
|
||||
|
||||
H.main().within(() => {
|
||||
cy.findByText(NORMAL_USER_NAME, { timeout: 20_000 }).should("be.visible");
|
||||
cy.findByText(USERS.normal.email).should("be.visible");
|
||||
cy.findByText("No one has access yet").should("not.be.visible");
|
||||
});
|
||||
|
||||
cy.findByRole("button", { name: `Remove ${NORMAL_USER_NAME}` }).click();
|
||||
|
||||
H.main().within(() => {
|
||||
cy.findByText("No one has access yet", { timeout: 20_000 }).should(
|
||||
"be.visible",
|
||||
);
|
||||
|
||||
cy.findByText(NORMAL_USER_NAME).should("not.exist");
|
||||
});
|
||||
});
|
||||
|
||||
it("adds users from comma-separated emails without duplicating existing members", () => {
|
||||
cy.get<number>("@dataAppGroupId").then((groupId) => {
|
||||
H.addUserToGroup(groupId, USERS.normal.email);
|
||||
});
|
||||
|
||||
cy.visit(`/admin/settings/apps/${DATA_APP_NAME}/users`);
|
||||
userRow(USERS.normal.email).should("be.visible");
|
||||
|
||||
cy.findByRole("button", { name: "Add users" }).click();
|
||||
H.popover().findByText(NODATA_USER_NAME).should("be.visible");
|
||||
|
||||
cy.log("paste a comma-separated list of emails");
|
||||
cy.findByRole("textbox", { name: "Search for a user to add" })
|
||||
.paste(` ${USERS.normal.email}, ${USERS.nodata.email.toUpperCase()} `)
|
||||
.should("have.value", USERS.normal.email);
|
||||
|
||||
cy.findByRole("button", { name: "Add" }).click();
|
||||
|
||||
userRow(USERS.nodata.email).should("be.visible");
|
||||
cy.reload();
|
||||
|
||||
cy.log("both users from the comma-separated list should be visible");
|
||||
userRow(USERS.normal.email).should("have.length", 1).and("be.visible");
|
||||
userRow(USERS.nodata.email).should("have.length", 1).and("be.visible");
|
||||
});
|
||||
|
||||
// SQLite does not have a schema.
|
||||
// The visible hierarchy should be "[Database] > [Table]" in table warnings.
|
||||
it("links a missing-access warning to a table without a schema", () => {
|
||||
H.activateToken("pro-self-hosted");
|
||||
H.addSqliteDatabase();
|
||||
|
||||
cy.get<number>("@sqliteID").then((databaseId) => {
|
||||
H.withDatabase(databaseId, ({ NUMBER_WITH_NULLS_ID }) => {
|
||||
cy.request<Table>("GET", `/api/table/${NUMBER_WITH_NULLS_ID}`)
|
||||
.its("body")
|
||||
.as("sqliteTable");
|
||||
});
|
||||
|
||||
cy.updatePermissionsGraph({
|
||||
[ALL_USERS_GROUP_ID]: {
|
||||
[databaseId]: {
|
||||
[DataPermission.VIEW_DATA]: DataPermissionValue.BLOCKED,
|
||||
},
|
||||
},
|
||||
[COLLECTION_GROUP_ID]: {
|
||||
[databaseId]: {
|
||||
[DataPermission.VIEW_DATA]: DataPermissionValue.BLOCKED,
|
||||
},
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
H.activateToken("bleeding-edge");
|
||||
|
||||
cy.get<Table>("@sqliteTable").then(({ id, schema }) => {
|
||||
expect(schema).to.equal("");
|
||||
|
||||
cy.request("PUT", `/api/apps/${DATA_APP_NAME}/table-dependencies`, {
|
||||
table_ids: [id],
|
||||
});
|
||||
});
|
||||
|
||||
cy.get<number>("@dataAppGroupId").then((groupId) => {
|
||||
H.addUserToGroup(groupId, USERS.nodata.email);
|
||||
});
|
||||
|
||||
cy.visit(`/admin/settings/apps/${DATA_APP_NAME}/users`);
|
||||
userRow(USERS.nodata.email)
|
||||
.findByRole("button", { name: "Missing data access" })
|
||||
.should("be.visible")
|
||||
.realHover();
|
||||
|
||||
cy.get<Table>("@sqliteTable").then(({ id, db_id }) => {
|
||||
cy.findByTestId("missing-tables-list").within(() => {
|
||||
cy.findAllByRole("link").should("have.length", 2);
|
||||
|
||||
cy.findByRole("link", { name: "sqlite" }).should(
|
||||
"have.attr",
|
||||
"href",
|
||||
`/admin/permissions/data/database/${db_id}`,
|
||||
);
|
||||
|
||||
cy.findByRole("link", { name: "Number With Nulls" })
|
||||
.should(
|
||||
"have.attr",
|
||||
"href",
|
||||
`/admin/permissions/data/database/${db_id}/table/${id}`,
|
||||
)
|
||||
.and("have.attr", "target", "_blank")
|
||||
.and("have.attr", "rel", "noopener noreferrer")
|
||||
.invoke("removeAttr", "target")
|
||||
.click();
|
||||
});
|
||||
|
||||
cy.location("pathname").should(
|
||||
"eq",
|
||||
`/admin/permissions/data/database/${db_id}/table/${id}`,
|
||||
);
|
||||
});
|
||||
|
||||
cy.findByTestId("permissions-editor-breadcrumbs").should(
|
||||
"contain.text",
|
||||
"Number With Nulls",
|
||||
);
|
||||
|
||||
H.assertPermissionForItem("All Users", 0, "Blocked");
|
||||
});
|
||||
|
||||
it("shows warnings only for users missing access to used tables", () => {
|
||||
// The no-data snapshot user belongs to both of these groups.
|
||||
// Block the products table in both groups.
|
||||
cy.updatePermissionsGraph({
|
||||
[ALL_USERS_GROUP_ID]: {
|
||||
[SAMPLE_DB_ID]: {
|
||||
[DataPermission.VIEW_DATA]: {
|
||||
PUBLIC: {
|
||||
[ORDERS_ID]: DataPermissionValue.UNRESTRICTED,
|
||||
[PRODUCTS_ID]: DataPermissionValue.BLOCKED,
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
|
||||
[COLLECTION_GROUP_ID]: {
|
||||
[SAMPLE_DB_ID]: {
|
||||
[DataPermission.VIEW_DATA]: {
|
||||
PUBLIC: {
|
||||
[ORDERS_ID]: DataPermissionValue.UNRESTRICTED,
|
||||
[PRODUCTS_ID]: DataPermissionValue.BLOCKED,
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
|
||||
[DATA_GROUP_ID]: {
|
||||
[SAMPLE_DB_ID]: {
|
||||
[DataPermission.VIEW_DATA]: {
|
||||
PUBLIC: {
|
||||
[ORDERS_ID]: DataPermissionValue.UNRESTRICTED,
|
||||
[PRODUCTS_ID]: DataPermissionValue.UNRESTRICTED,
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
});
|
||||
|
||||
cy.request("PUT", `/api/apps/${DATA_APP_NAME}/table-dependencies`, {
|
||||
table_ids: [ORDERS_ID, PRODUCTS_ID],
|
||||
});
|
||||
|
||||
cy.get<number>("@dataAppGroupId").then((groupId) => {
|
||||
H.addUserToGroup(groupId, USERS.normal.email);
|
||||
H.addUserToGroup(groupId, USERS.nodata.email);
|
||||
});
|
||||
|
||||
cy.visit("/admin/settings/apps");
|
||||
|
||||
dataAppRow()
|
||||
.findByRole("link", {
|
||||
name: "Some users are missing data access.",
|
||||
})
|
||||
.should("be.visible")
|
||||
.click();
|
||||
|
||||
cy.location("pathname").should(
|
||||
"eq",
|
||||
`/admin/settings/apps/${DATA_APP_NAME}/users`,
|
||||
);
|
||||
|
||||
cy.findByRole("heading", { name: "Manage access to this app" }).should(
|
||||
"be.visible",
|
||||
);
|
||||
|
||||
userRow(USERS.normal.email).within(() => {
|
||||
cy.findByText(NORMAL_USER_NAME).should("be.visible");
|
||||
|
||||
cy.findByRole("button", { name: "Missing data access" }).should(
|
||||
"not.exist",
|
||||
);
|
||||
});
|
||||
|
||||
userRow(USERS.nodata.email).within(() => {
|
||||
cy.findByText(NODATA_USER_NAME).should("be.visible");
|
||||
|
||||
cy.findByRole("button", { name: "Missing data access" })
|
||||
.should("be.visible")
|
||||
.realHover();
|
||||
});
|
||||
|
||||
cy.findByTestId("data-access-warning-popover").within(() => {
|
||||
cy.findByText(
|
||||
`${USERS.nodata.first_name} doesn’t have permission to view these tables used in this app:`,
|
||||
).should("be.visible");
|
||||
|
||||
cy.findByTestId("missing-tables-list").within(() => {
|
||||
cy.findAllByRole("link")
|
||||
.should("have.length", 3)
|
||||
.and("be.visible")
|
||||
.should(($links) => {
|
||||
expect(
|
||||
[...$links].map((link) => ({
|
||||
label: link.textContent,
|
||||
href: link.getAttribute("href"),
|
||||
target: link.getAttribute("target"),
|
||||
rel: link.getAttribute("rel"),
|
||||
})),
|
||||
).to.deep.equal([
|
||||
{
|
||||
label: "Sample Database",
|
||||
href: `/admin/permissions/data/database/${SAMPLE_DB_ID}`,
|
||||
target: "_blank",
|
||||
rel: "noopener noreferrer",
|
||||
},
|
||||
{
|
||||
label: "PUBLIC",
|
||||
href: `/admin/permissions/data/database/${SAMPLE_DB_ID}/schema/PUBLIC`,
|
||||
target: "_blank",
|
||||
rel: "noopener noreferrer",
|
||||
},
|
||||
{
|
||||
label: "Products",
|
||||
href: `/admin/permissions/data/database/${SAMPLE_DB_ID}/schema/PUBLIC/table/${PRODUCTS_ID}`,
|
||||
target: "_blank",
|
||||
rel: "noopener noreferrer",
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
cy.findByRole("link", { name: "Orders" }).should("not.exist");
|
||||
});
|
||||
});
|
||||
|
||||
cy.findByRole("button", { name: `Remove ${NODATA_USER_NAME}` }).click();
|
||||
|
||||
H.main().within(() => {
|
||||
cy.findByText(USERS.normal.email).should("be.visible");
|
||||
|
||||
cy.findByText(USERS.nodata.email, { timeout: 20_000 }).should(
|
||||
"not.exist",
|
||||
);
|
||||
});
|
||||
|
||||
cy.visit("/admin/settings/apps");
|
||||
dataAppRow().within(() => {
|
||||
cy.findByText(DATA_APP_NAME).should("be.visible");
|
||||
|
||||
cy.findByRole("link", {
|
||||
name: "Some users are missing data access.",
|
||||
}).should("not.exist");
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
const dataAppRow = () =>
|
||||
cy
|
||||
.findByTestId(`data-app-list-item-${DATA_APP_NAME}`)
|
||||
.scrollIntoView()
|
||||
.should("be.visible");
|
||||
|
||||
function openManageUserAccessFromAppRow() {
|
||||
dataAppRow()
|
||||
.findByRole("button", { name: `Actions for ${DATA_APP_NAME}` })
|
||||
.click();
|
||||
|
||||
H.popover().findByText("Manage user access").click();
|
||||
}
|
||||
|
||||
const userRow = (email: string) => H.main().findByText(email).closest("tr");
|
||||
@@ -3,6 +3,7 @@ import {
|
||||
DATA_APP_NAME as APP_NAME,
|
||||
visitDataAppRoute as visitAppRoute,
|
||||
} from "e2e/support/helpers";
|
||||
import type { DataApp } from "metabase-types/api";
|
||||
|
||||
import { DATA_APP_TEST_ENV as TEST_ENV } from "./helpers";
|
||||
|
||||
@@ -59,6 +60,45 @@ describe("scenarios > data apps > viewing & routing", () => {
|
||||
});
|
||||
});
|
||||
|
||||
it("shows unpublished app error when opening an app without a resource collection", () => {
|
||||
cy.request<DataApp>("POST", `/api/apps/${APP_NAME}/draft`)
|
||||
.its("body.resource_collection_id")
|
||||
.as("resourceCollectionId");
|
||||
|
||||
cy.log("archive and delete the data app collection");
|
||||
cy.get<number>("@resourceCollectionId").then((collectionId) => {
|
||||
cy.request("PUT", `/api/collection/${collectionId}`, {
|
||||
archived: true,
|
||||
});
|
||||
|
||||
cy.request("DELETE", `/api/collection/${collectionId}`);
|
||||
});
|
||||
|
||||
cy.log("fetching bundle should return 409 error");
|
||||
cy.request({
|
||||
url: `/api/apps/${APP_NAME}/bundle`,
|
||||
failOnStatusCode: false,
|
||||
})
|
||||
.its("status")
|
||||
.should("eq", 409);
|
||||
|
||||
cy.intercept("GET", `/api/apps/${APP_NAME}`).as("getUnpublishedApp");
|
||||
H.openDataApp(APP_NAME);
|
||||
|
||||
cy.log("fetching data app metadata should return 409 error");
|
||||
cy.wait("@getUnpublishedApp")
|
||||
.its("response.statusCode")
|
||||
.should("eq", 409);
|
||||
|
||||
H.main().within(() => {
|
||||
cy.findByText("This data app isn’t published yet").should("be.visible");
|
||||
|
||||
cy.findByText(
|
||||
"An administrator needs to publish this data app before it can be opened.",
|
||||
).should("be.visible");
|
||||
});
|
||||
});
|
||||
|
||||
it("shows a not-found state for a disabled or missing app", () => {
|
||||
// No app with this slug exists, so the metadata endpoint really 404s and the
|
||||
// host renders a not-found state rather than a broken iframe — no mock needed.
|
||||
@@ -72,7 +112,7 @@ describe("scenarios > data apps > viewing & routing", () => {
|
||||
H.mockDataApp(APP_NAME, {
|
||||
displayName: APP_DISPLAY_NAME,
|
||||
testEnv: TEST_ENV,
|
||||
bundleDelay: 2000,
|
||||
bundleDelayMs: 2000,
|
||||
});
|
||||
|
||||
H.openDataApp(APP_NAME);
|
||||
|
||||
@@ -189,8 +189,9 @@
|
||||
(let [blocked-group-ids (advanced-permissions.db/blocked-group-ids group-ids)
|
||||
impersonation-group-ids (advanced-permissions.db/impersonated-group-ids group-ids)
|
||||
sandbox-group-ids (advanced-permissions.db/sandboxed-group-ids group-ids)
|
||||
;; New databases have no legacy grants to preserve. App groups must start blocked.
|
||||
blocked-groups (into (or blocked-group-ids #{})
|
||||
(concat impersonation-group-ids sandbox-group-ids))]
|
||||
(concat impersonation-group-ids sandbox-group-ids (perms/data-app-group-ids)))]
|
||||
(zipmap group-ids (map #(if (blocked-groups %) :blocked :unrestricted) group-ids)))))
|
||||
|
||||
(defenterprise new-table-view-data-permission-levels
|
||||
@@ -205,6 +206,14 @@
|
||||
sandbox-group-ids (into #{}
|
||||
(map :group_id)
|
||||
(advanced-permissions.db/sandboxed-group-ids-for-database db-id group-ids))
|
||||
blocked-groups (into (or blocked-group-ids #{})
|
||||
sandbox-group-ids)]
|
||||
(zipmap group-ids (map #(if (blocked-groups %) :blocked :unrestricted) group-ids)))))
|
||||
app-group-ids (set (perms/data-app-group-ids))
|
||||
app-view-data (when (some app-group-ids group-ids)
|
||||
(perms/data-app-view-data-permission-level db-id))
|
||||
blocked-groups (into (or blocked-group-ids #{}) sandbox-group-ids)]
|
||||
;; A new table must not introduce a block into an app group's database-wide legacy permission.
|
||||
(into {} (map (fn [group-id]
|
||||
[group-id (cond
|
||||
(app-group-ids group-id) app-view-data
|
||||
(blocked-groups group-id) :blocked
|
||||
:else :unrestricted)]))
|
||||
group-ids))))
|
||||
|
||||
@@ -46,25 +46,29 @@ remote-sync import → snapshot → sync-from-snapshot! → discover confi
|
||||
layer and are counted separately in the pull summary (see `fold-data-app-changes` in
|
||||
`remote_sync/impl.clj` — without it, a pull whose only change is a data app reports "no changes").
|
||||
|
||||
One import = one transaction. Discovery, upserts, and pruning all commit together.
|
||||
Apps are synced independently. Each app's row and its resource links commit together, and pruning
|
||||
runs in its own transaction. A database or resource setup error rolls back that app's update, then
|
||||
records `sync_error` separately.
|
||||
|
||||
## The repository is the source of truth
|
||||
|
||||
A sync **upserts every app it finds and deletes every row whose directory is gone.** The
|
||||
consequences are worth stating explicitly, because they're the questions that come up:
|
||||
|
||||
| Event | What happens to apps |
|
||||
|---|---|
|
||||
| App directory removed from the repo | Row and cached bundle deleted on the next sync |
|
||||
| Repo switched to a different one | Previous repo's apps are absent from the new snapshot, so they're dropped |
|
||||
| Repo unlinked | Nothing — unlinking runs no sync, so apps survive |
|
||||
| Repo has no `data_apps/` at all | All apps removed |
|
||||
| Failed clone/fetch | Nothing — that throws before a snapshot exists, so deletion never fires |
|
||||
| Event | What happens to apps |
|
||||
| ----------------------------------- | ------------------------------------------------------------------------- |
|
||||
| App directory removed from the repo | Row and cached bundle deleted on the next sync |
|
||||
| Repo switched to a different one | Previous repo's apps are absent from the new snapshot, so they're dropped |
|
||||
| Repo unlinked | Nothing — unlinking runs no sync, so apps survive |
|
||||
| Repo has no `data_apps/` at all | All apps removed |
|
||||
| Failed clone/fetch | Nothing — that throws before a snapshot exists, so deletion never fires |
|
||||
|
||||
Pruning is by **directory presence**, not by successful parse. An app whose directory is still
|
||||
there but whose `data_app.yaml` is momentarily broken keeps its row and its last-good bundle, and is
|
||||
marked with a `sync_error` — see failure isolation below. Local `enabled` state does not protect a
|
||||
row: an app disabled in the admin UI and then deleted from the repo is still removed.
|
||||
row: an app disabled in the admin UI and then deleted from the repo is still removed. Draft rows —
|
||||
created via `POST /api/apps/:slug/draft` ahead of an app's first import — are the one exemption:
|
||||
pruning skips them until a sync materializes the app and clears the flag.
|
||||
|
||||
## Failure isolation
|
||||
|
||||
@@ -100,8 +104,14 @@ assets (`metabase.server.routes/static-files-handler`).
|
||||
- `GET /api/apps/:slug/bundle` — the cached bytes, with a content-hash ETag and `If-None-Match`
|
||||
→ 304. Carries `X-Metabase-Data-App-Allowed-Hosts`, which the iframe reads to configure its
|
||||
sandbox fetch allowlist.
|
||||
- `GET /api/apps/sandbox-host` — the empty document loaded as the Near-Membrane realm iframe,
|
||||
carrying the CSP that confines `'unsafe-eval'` to that realm.
|
||||
- `PUT /api/apps/:slug` — toggle `enabled` (superuser).
|
||||
- `DELETE /api/apps/:slug` — drop a row and its bundle (superuser).
|
||||
- `DELETE /api/apps/:slug` — drop a row, its bundle, and its owned resources (superuser).
|
||||
- `POST /api/apps/:slug/draft` — create or reuse a draft row with its resources before the app's
|
||||
first import (superuser).
|
||||
- `POST /api/apps/:slug/query` — resolve an authored query definition into a serializable
|
||||
Metabase query plus the table IDs it touches (superuser).
|
||||
- `GET /api/apps/repo-status` — whether a repo is connected (superuser).
|
||||
|
||||
Responses are field-filtered by role: superusers get full metadata, everyone else gets `name` and
|
||||
@@ -114,20 +124,36 @@ middleware's lookup doesn't pull in route code.
|
||||
|
||||
## Permissions
|
||||
|
||||
**Viewing is not gated.** Any signed-in user may open any enabled app — `can-read?` is
|
||||
unconditionally true, and the endpoints are `+auth`, so reaching a read check already implies
|
||||
authentication. The data an app queries still runs through the QP under the viewing user's own
|
||||
permissions, so a user without data access opens the app and sees no data.
|
||||
Each app owns two server-managed resources (`resources.clj`), created on draft or first import and
|
||||
reasserted on every sync: a **collection** holding the copies the app is served from (saved
|
||||
questions, action models, table-sourced metrics) and a **permissions group** its users belong to.
|
||||
|
||||
**Managing is superuser-only** — enabling, disabling, deleting, and reading repo status.
|
||||
The group is set database-level `view-data :blocked` on every database, so it grants **no data
|
||||
access of its own** (which cascades `create-queries`/`download-results` to `:no`); every group but
|
||||
admins is revoked from the collection before the app group gets read access. Deleting an app deletes
|
||||
both resources and everything in the collection.
|
||||
|
||||
**Viewing an app** requires read access to its resource collection. You have to be a member in
|
||||
the app's group or be an admin. An app without a linked resource collection is considered _unpublished_.
|
||||
The app's metadata and bundle endpoint returns HTTP 409 for all signed-in users. The frontend
|
||||
shows the error "This data app isn’t published yet".
|
||||
|
||||
**A viewer sees an app's data only through access they already hold.** The app group grants no
|
||||
view-data of its own, so a viewer without access to an app's tables (e.g. a sandboxed user) sees no
|
||||
data from it — their own groups' permissions and sandboxes apply unchanged. Because the group grants
|
||||
nothing, it can never lift another group's sandbox, so sandboxing needs no data-app special-casing.
|
||||
|
||||
**Managing is superuser-only** — enabling, disabling, deleting, drafts, query resolution, and repo
|
||||
status.
|
||||
|
||||
## Namespace map
|
||||
|
||||
| Namespace | Responsibility |
|
||||
|---|---|
|
||||
| `sync.clj` | Discovery, materialization, pruning. The entry point remote-sync calls. |
|
||||
| `config.clj` | `data_app.yaml` parsing and validation; the `data_apps/` layout constants. |
|
||||
| `api.clj` | The `/api/apps` endpoints, bundle serving, ETag handling. |
|
||||
| `models/data_app.clj` | The `:model/DataApp` Toucan model, permissions, blob coercion. |
|
||||
| `csp.clj` | `allowed_hosts` lookup for the core CSP middleware. |
|
||||
| `init.clj` | Loads the above so endpoints, models, and hooks register. |
|
||||
| Namespace | Responsibility |
|
||||
| --------------------- | --------------------------------------------------------------------------------------------------- |
|
||||
| `sync.clj` | Discovery, materialization, pruning, drafts. The entry point remote-sync calls. |
|
||||
| `config.clj` | `data_app.yaml` parsing and validation; the `data_apps/` layout constants. |
|
||||
| `api.clj` | The `/api/apps` endpoints, bundle serving, ETag handling. |
|
||||
| `resources.clj` | Lifecycle of the app-owned collection and permission group: creation, view-data blocking, deletion. |
|
||||
| `models/data_app.clj` | The `:model/DataApp` Toucan model, permissions, blob coercion. |
|
||||
| `csp.clj` | `allowed_hosts` lookup for the core CSP middleware. |
|
||||
| `init.clj` | Loads the above so endpoints, models, and hooks register. |
|
||||
|
||||
@@ -8,11 +8,18 @@
|
||||
`metabase.server.routes/static-files-handler`)."
|
||||
(:require
|
||||
[clojure.string :as str]
|
||||
[metabase-enterprise.data-apps.config :as data-app.config]
|
||||
[metabase-enterprise.data-apps.db :as data-apps.db]
|
||||
[metabase-enterprise.data-apps.query-definition :as query-definition]
|
||||
[metabase-enterprise.data-apps.sync :as data-app.sync]
|
||||
[metabase-enterprise.data-apps.user-access :as data-app.user-access]
|
||||
[metabase.api.common :as api]
|
||||
[metabase.api.macros :as api.macros]
|
||||
[metabase.api.routes.common :refer [+auth]]
|
||||
[metabase.lib-be.core :as lib-be]
|
||||
[metabase.lib-be.schema :as lib-be.schema]
|
||||
[metabase.lib.core :as lib]
|
||||
[metabase.util.i18n :refer [tru]]
|
||||
[metabase.util.json :as json]
|
||||
[metabase.util.malli.schema :as ms])
|
||||
(:import
|
||||
@@ -60,6 +67,10 @@
|
||||
[:bundle_path ms/NonBlankString]
|
||||
[:enabled :boolean]
|
||||
[:allowed_hosts [:sequential :string]]
|
||||
[:resource_collection_id [:maybe ms/PositiveInt]]
|
||||
[:permission_group_id [:maybe ms/PositiveInt]]
|
||||
[:table_ids [:sequential ms/PositiveInt]]
|
||||
[:has_user_permission_warnings {:optional true} :boolean]
|
||||
[:bundle_hash [:maybe :string]]
|
||||
[:last_synced_sha [:maybe :string]]
|
||||
[:last_synced_at [:maybe :any]]
|
||||
@@ -77,6 +88,54 @@
|
||||
[:configured :boolean]
|
||||
[:url [:maybe :string]]])
|
||||
|
||||
(def ^:private MetricResponse
|
||||
[:map {:closed true}
|
||||
[:id ms/PositiveInt]
|
||||
[:name ms/NonBlankString]
|
||||
[:type [:enum :metric]]
|
||||
[:collection_id [:maybe ms/PositiveInt]]
|
||||
[:dataset_query ::lib-be.schema/maybe-legacy-query]
|
||||
[:database_id ms/PositiveInt]
|
||||
[:display [:maybe [:or :keyword :string]]]
|
||||
[:visualization_settings [:maybe ms/VisualizationSettings]]
|
||||
[:description [:maybe :string]]])
|
||||
|
||||
(def ^:private QueryResolutionResponse
|
||||
[:map
|
||||
[:database_id ms/PositiveInt]
|
||||
[:dataset_query ::lib-be.schema/maybe-legacy-query]
|
||||
[:table_ids [:sequential ms/PositiveInt]]
|
||||
[:metrics [:sequential MetricResponse]]])
|
||||
|
||||
(def ^:private TableDependenciesRequest
|
||||
[:map {:closed true}
|
||||
[:table_ids [:sequential {:distinct true} ms/PositiveInt]]])
|
||||
|
||||
(def ^:private QueryTableDependenciesRequest
|
||||
[:map {:closed true}
|
||||
[:dataset_queries [:sequential ::lib-be.schema/maybe-legacy-query]]])
|
||||
|
||||
(def ^:private QueryTableDependenciesResponse
|
||||
[:map {:closed true}
|
||||
[:table_ids [:sequential ms/PositiveInt]]])
|
||||
|
||||
(def ^:private PermissionWarningsRequest
|
||||
[:map {:closed true}
|
||||
[:user_ids [:sequential {:min 1 :max 100 :distinct true} ms/PositiveInt]]])
|
||||
|
||||
(def ^:private MissingTable
|
||||
[:map {:closed true}
|
||||
[:id ms/PositiveInt]
|
||||
[:name ms/NonBlankString]
|
||||
[:schema [:maybe :string]]
|
||||
[:database_id ms/PositiveInt]
|
||||
[:database_name ms/NonBlankString]])
|
||||
|
||||
(def ^:private PermissionWarning
|
||||
[:map {:closed true}
|
||||
[:user_id ms/PositiveInt]
|
||||
[:missing_tables [:sequential MissingTable]]])
|
||||
|
||||
;;; --------------------------------------------- Repo status ---------------------------------------------
|
||||
|
||||
(api.macros/defendpoint :get "/repo-status" :- RepoStatusResponse
|
||||
@@ -131,14 +190,37 @@
|
||||
app
|
||||
(select-keys app [:name :display_name])))
|
||||
|
||||
(defn- data-app-list-response
|
||||
[warning-group-ids app]
|
||||
(cond-> (data-app-response app)
|
||||
api/*is-superuser?*
|
||||
(assoc :has_user_permission_warnings
|
||||
(contains? warning-group-ids (:permission_group_id app)))))
|
||||
|
||||
(defn- read-check-data-app
|
||||
"Check whether the current user can access a data app. Viewing requires read access to the app's
|
||||
resource collection. An app with no linked resource collection has not been published yet: it is
|
||||
not viewable by anyone through this endpoint (an admin must publish it first), signalled with a
|
||||
409 so the client can show a dedicated \"not published\" screen rather than leaking metadata or
|
||||
the bundle to every signed-in user."
|
||||
[app]
|
||||
(api/read-check app)
|
||||
(if-let [collection-id (:resource_collection_id app)]
|
||||
(api/read-check :model/Collection collection-id)
|
||||
(throw (ex-info (tru "This data app has not been published yet.")
|
||||
{:status-code 409})))
|
||||
app)
|
||||
|
||||
(api.macros/defendpoint :get "/" :- [:sequential [:or DataAppResponse PublicDataAppResponse]]
|
||||
"List the data apps provided by the connected repository. Pass `available=true`
|
||||
to return only enabled apps without sync errors."
|
||||
[_route-params
|
||||
{:keys [available]} :- [:map {:closed true} [:available {:optional true} [:maybe :boolean]]]]
|
||||
(->> (data-apps.db/non-blob-data-apps available)
|
||||
(map api/read-check)
|
||||
(mapv data-app-response)))
|
||||
(let [apps (->> (data-apps.db/non-blob-data-apps available)
|
||||
(mapv api/read-check))
|
||||
warning-group-ids (when api/*is-superuser?*
|
||||
(data-app.user-access/groups-with-permission-warnings apps))]
|
||||
(mapv (partial data-app-list-response warning-group-ids) apps)))
|
||||
|
||||
;; NOTE on the `slug-regex` constraint: the default path-param matcher allows
|
||||
;; slashes inside a segment, so `/:slug` would otherwise swallow `/x/bundle`.
|
||||
@@ -160,16 +242,129 @@
|
||||
longer in it is pruned by that sync anyway."
|
||||
[{:keys [slug]} :- [:map {:closed true} [:slug ms/NonBlankString]]]
|
||||
(api/check-superuser)
|
||||
;; `t2/delete!` returns the row count; a 0 means the slug wasn't there → 404.
|
||||
;; The delete returns the row count; a 0 means the slug wasn't there → 404.
|
||||
(api/check-404 (pos? (data-apps.db/delete-data-app-by-slug! slug)))
|
||||
;; a `nil` body is rendered as a 204; matches the `:- :nil` response schema
|
||||
;; above (returning `generic-204-no-content` would fail that validation).
|
||||
nil)
|
||||
|
||||
(api.macros/defendpoint :put ["/:slug/table-dependencies" :slug slug-regex] :- DataAppResponse
|
||||
"Store the tables used by the resources from a successful data app resource synchronization."
|
||||
[{:keys [slug]} :- [:map {:closed true} [:slug ms/NonBlankString]]
|
||||
_query-params
|
||||
{table-ids :table_ids} :- TableDependenciesRequest]
|
||||
(api/check-superuser)
|
||||
(let [app (api/check-404 (data-apps.db/non-blob-data-app-by-slug slug))
|
||||
table-ids (vec (sort table-ids))]
|
||||
(api/check-400 (= (set table-ids)
|
||||
(data-apps.db/existing-table-ids table-ids))
|
||||
(tru "One or more tables do not exist."))
|
||||
(data-apps.db/update-data-app! (:id app) {:table_ids table-ids})
|
||||
(data-apps.db/non-blob-data-app (:id app))))
|
||||
|
||||
(api.macros/defendpoint :post ["/:slug/user-permission-warnings" :slug slug-regex]
|
||||
:- [:sequential PermissionWarning]
|
||||
"Return warnings for users who cannot access every table used by a data app."
|
||||
[{:keys [slug]} :- [:map {:closed true} [:slug ms/NonBlankString]]
|
||||
_query-params
|
||||
{user-ids :user_ids} :- PermissionWarningsRequest]
|
||||
(api/check-superuser)
|
||||
(let [app (api/check-404 (data-apps.db/non-blob-data-app-by-slug slug))
|
||||
users (data-apps.db/users-for-permission-warnings user-ids)]
|
||||
(api/check-404 (= (count users) (count user-ids)))
|
||||
(api/check-400 (every? :is_active users)
|
||||
(tru "Deactivated users cannot be added to data apps."))
|
||||
(api/check-400 (every? (comp nil? :tenant_id) users)
|
||||
(tru "Tenant users cannot be added to data apps."))
|
||||
(data-app.user-access/permission-warnings (:table_ids app) users)))
|
||||
|
||||
(defn- query-table-ids
|
||||
"The tables a query reads, including ones it reaches only through an implicit join."
|
||||
[query]
|
||||
(into (set (lib/all-source-table-ids query))
|
||||
(lib/all-implicitly-joined-table-ids query)))
|
||||
|
||||
(defn- referenced-card-ids
|
||||
"Ids of the cards -- metrics, source questions, template-tag questions -- that `metric`'s own
|
||||
definition reads."
|
||||
[metric]
|
||||
(-> (lib-be/application-database-metadata-provider (:database_id metric))
|
||||
(lib/query (:dataset_query metric))
|
||||
lib/all-source-card-ids))
|
||||
|
||||
(defn- referenced-metrics
|
||||
"Return direct metric references, rejecting any whose own definition reads another card.
|
||||
|
||||
Sync copies a referenced metric but rewrites nothing inside the copy, and copies nothing the copy
|
||||
in turn reads, so those references still point at the originals. The app would publish successfully
|
||||
and then fail for every viewer without access to the originals' collections, so refuse it here.
|
||||
|
||||
This runs on every sync rather than at codegen, so a metric edited into this shape after its schema
|
||||
was generated is caught too."
|
||||
[query]
|
||||
(let [metric-ids (lib/all-source-card-ids query)
|
||||
metrics (sort-by :id (data-apps.db/metrics-by-ids metric-ids))
|
||||
nested (filter (comp seq referenced-card-ids) metrics)]
|
||||
(api/check-400 (empty? nested)
|
||||
(tru "Data app queries cannot use metrics that reference other saved questions or metrics: {0}"
|
||||
(str/join ", " (map :name nested))))
|
||||
(mapv #(update (select-keys % [:id :name :type :collection_id :dataset_query
|
||||
:database_id :display :visualization_settings :description])
|
||||
:dataset_query
|
||||
lib/prepare-for-serialization)
|
||||
metrics)))
|
||||
|
||||
(api.macros/defendpoint :post ["/:slug/query" :slug slug-regex] :- QueryResolutionResponse
|
||||
"Resolve an authored data-app query definition into a serializable Metabase query."
|
||||
[{:keys [slug]} :- [:map {:closed true} [:slug ms/NonBlankString]]
|
||||
_query-params
|
||||
query-def :- ::query-definition/query-definition]
|
||||
(api/check-superuser)
|
||||
(api/check-404 (data-apps.db/non-blob-data-app-by-slug slug))
|
||||
(let [{source-type :type, table-id :id} (get-in query-def [:stages 0 :source])
|
||||
_ (api/check-400 (= (keyword source-type) :table)
|
||||
"Data app query definitions must use a table source.")
|
||||
database-id (api/check-404 (data-apps.db/table-database-id table-id))
|
||||
query (lib/test-query (lib-be/application-database-metadata-provider database-id) query-def)]
|
||||
{:database_id database-id
|
||||
:dataset_query (lib/prepare-for-serialization query)
|
||||
:table_ids (vec (sort (query-table-ids query)))
|
||||
:metrics (referenced-metrics query)}))
|
||||
|
||||
(api.macros/defendpoint :post ["/:slug/query-table-dependencies" :slug slug-regex]
|
||||
:- QueryTableDependenciesResponse
|
||||
"Return the tables read by already-saved queries, including ones reached only through an implicit join.
|
||||
|
||||
Sync copies models, actions and metrics whose queries it never resolves through `/query`, and only
|
||||
this metadata-based lookup sees an implicit join: the id of a table reached through a foreign key
|
||||
appears nowhere in the query itself."
|
||||
[{:keys [slug]} :- [:map {:closed true} [:slug ms/NonBlankString]]
|
||||
_query-params
|
||||
{dataset-queries :dataset_queries} :- QueryTableDependenciesRequest]
|
||||
(api/check-superuser)
|
||||
(api/check-404 (data-apps.db/non-blob-data-app-by-slug slug))
|
||||
{:table_ids (->> dataset-queries
|
||||
(mapcat (fn [dataset-query]
|
||||
(-> (lib-be/application-database-metadata-provider (:database dataset-query))
|
||||
(lib/query dataset-query)
|
||||
query-table-ids)))
|
||||
set
|
||||
sort
|
||||
vec)})
|
||||
|
||||
(api.macros/defendpoint :post ["/:slug/draft" :slug slug-regex] :- DataAppResponse
|
||||
"Create or reuse a data app draft before its first repository import."
|
||||
[{:keys [slug]} :- [:map {:closed true} [:slug ms/NonBlankString]]]
|
||||
(api/check-superuser)
|
||||
(api/check-400 (data-app.config/valid-slug? slug)
|
||||
"Data app draft slugs must use lowercase letters, numbers, and dashes.")
|
||||
(data-app.sync/ensure-draft! slug)
|
||||
(data-apps.db/non-blob-data-app-by-slug slug))
|
||||
|
||||
(api.macros/defendpoint :get ["/:slug" :slug slug-regex] :- [:or DataAppResponse PublicDataAppResponse]
|
||||
"Fetch metadata for a single enabled data app by its slug."
|
||||
[{:keys [slug]} :- [:map {:closed true} [:slug ms/NonBlankString]]]
|
||||
(data-app-response (api/read-check (data-apps.db/enabled-non-blob-data-app-by-slug slug))))
|
||||
(data-app-response (read-check-data-app (data-apps.db/enabled-non-blob-data-app-by-slug slug))))
|
||||
|
||||
(api.macros/defendpoint :get ["/:slug/bundle" :slug slug-regex] :- :any
|
||||
"Serve the cached JS bundle for a single enabled data app by slug. Honors
|
||||
@@ -181,7 +376,7 @@
|
||||
respond
|
||||
raise]
|
||||
(try
|
||||
(let [row (api/read-check (data-apps.db/enabled-non-blob-data-app-by-slug slug))
|
||||
(let [row (read-check-data-app (data-apps.db/enabled-non-blob-data-app-by-slug slug))
|
||||
hash (:bundle_hash row)
|
||||
etag (some->> hash (format "\"%s\""))]
|
||||
(cond
|
||||
|
||||
@@ -38,7 +38,7 @@
|
||||
"Slugs that collide with literal `/api/apps/*` sub-routes (see the API's
|
||||
`slug-regex`). An app with one of these would sync but be unreachable, so we
|
||||
reject it during parsing."
|
||||
#{"repo-status"})
|
||||
#{"repo-status" "sandbox-host"})
|
||||
|
||||
(defn- normalize-path
|
||||
"Trim and drop a leading `./` so the path is relative to the app directory."
|
||||
@@ -131,6 +131,13 @@
|
||||
[^String dir]
|
||||
(subs dir (inc (str/last-index-of dir "/"))))
|
||||
|
||||
(defn valid-slug?
|
||||
"Whether `slug` can name a data app directory and API route."
|
||||
[slug]
|
||||
(and (string? slug)
|
||||
(re-matches slug-pattern slug)
|
||||
(not (contains? reserved-slugs slug))))
|
||||
|
||||
(defn parse-app-config
|
||||
"Parse the bytes of one `data_app.yaml` from the app directory `dir` (e.g.
|
||||
`data_apps/sales`) into `{:slug ..., :display_name ..., :description ...,
|
||||
@@ -138,9 +145,8 @@
|
||||
relative to the directory; `:description` is an optional one-liner (nil when
|
||||
absent or blank, capped at [[max-description-chars]]); `:allowed_hosts` is a
|
||||
(possibly empty) vector of origins the sandboxed bundle may reach. Throws an
|
||||
`ex-info` with `:status-code` 400 on
|
||||
malformed or incomplete content — including a directory whose name isn't a
|
||||
usable slug, since that app has no URL to be served at."
|
||||
`ex-info` with `:status-code` 400 on malformed or incomplete content — including
|
||||
a directory whose name isn't a usable slug, since that app has no URL to be served at."
|
||||
[^bytes bytes ^String dir]
|
||||
(let [parsed (parse-yaml bytes dir)
|
||||
slug (dir-slug dir)
|
||||
|
||||
@@ -3,6 +3,7 @@
|
||||
additional logic, so the rest of the module only touches `toucan2.core` for model definitions and hydration methods."
|
||||
(:require
|
||||
[metabase-enterprise.data-apps.schema :as data-apps.schema]
|
||||
[metabase.util :as u]
|
||||
[metabase.util.malli :as mu]
|
||||
[metabase.util.malli.schema :as ms]
|
||||
[toucan2.core :as t2]))
|
||||
@@ -10,6 +11,7 @@
|
||||
(def ^:private non-blob-columns
|
||||
"Columns to select for normal data-app metadata reads, excluding the raw bundle blob."
|
||||
[:model/DataApp :id :name :display_name :description :bundle_path :enabled :allowed_hosts
|
||||
:resource_collection_id :permission_group_id :table_ids :draft
|
||||
:bundle_hash :last_synced_sha :last_synced_at :sync_error
|
||||
:created_at :updated_at])
|
||||
|
||||
@@ -47,6 +49,30 @@
|
||||
[]
|
||||
(t2/select [:model/DataApp :name :display_name :description :allowed_hosts :bundle_path :bundle_hash :sync_error]))
|
||||
|
||||
(defn table-database-id
|
||||
"The database ID for the Table with `table-id`, or nil."
|
||||
[table-id]
|
||||
(t2/select-one-fn :db_id :model/Table :id table-id))
|
||||
|
||||
(defn existing-table-ids
|
||||
"The IDs from `table-ids` that belong to existing Tables."
|
||||
[table-ids]
|
||||
(if (seq table-ids)
|
||||
(t2/select-pks-set :model/Table :id [:in table-ids])
|
||||
#{}))
|
||||
|
||||
(defn users-for-permission-warnings
|
||||
"The fields needed to calculate permission warnings for Users with `user-ids`."
|
||||
[user-ids]
|
||||
(t2/select [:model/User :id :is_superuser :is_active :tenant_id] :id [:in user-ids]))
|
||||
|
||||
(defn metrics-by-ids
|
||||
"The metric Cards with `metric-ids`."
|
||||
[metric-ids]
|
||||
(if (seq metric-ids)
|
||||
(t2/select :model/Card :id [:in metric-ids] :type "metric")
|
||||
[]))
|
||||
|
||||
(mu/defn data-app-exists?
|
||||
"Whether a DataApp named `slug` exists."
|
||||
[slug :- :string]
|
||||
@@ -75,11 +101,148 @@
|
||||
(t2/delete! :model/DataApp :name slug))
|
||||
|
||||
(mu/defn delete-data-apps-not-named!
|
||||
"Delete the DataApps whose name is not one of `slugs`, returning the number deleted."
|
||||
"Delete non-draft DataApps whose name is not one of `slugs`, returning the number deleted."
|
||||
[slugs :- [:set :string]]
|
||||
(t2/delete! :model/DataApp :name [:not-in slugs]))
|
||||
(t2/delete! :model/DataApp :name [:not-in slugs] :draft false))
|
||||
|
||||
(mu/defn delete-all-data-apps!
|
||||
"Delete every DataApp, returning the number deleted."
|
||||
"Delete every non-draft DataApp, returning the number deleted."
|
||||
[]
|
||||
(t2/delete! :model/DataApp))
|
||||
(t2/delete! :model/DataApp :draft false))
|
||||
|
||||
(defn publish-data-app-drafts!
|
||||
"Mark drafts named by `slugs` as published."
|
||||
[slugs]
|
||||
(t2/update! :model/DataApp :name [:in slugs] :draft true {:draft false}))
|
||||
|
||||
(defn permission-group
|
||||
"The permission group with `group-id`, or nil."
|
||||
[group-id]
|
||||
(t2/select-one :model/PermissionsGroup :id group-id))
|
||||
|
||||
(defn insert-permission-group!
|
||||
"Insert a permission group and return it."
|
||||
[row]
|
||||
(t2/insert-returning-instance! :model/PermissionsGroup row))
|
||||
|
||||
(defn update-permission-group!
|
||||
"Apply `changes` to the permission group with `group-id`."
|
||||
[group-id changes]
|
||||
(t2/update! :model/PermissionsGroup :id group-id changes))
|
||||
|
||||
(defn delete-permission-group!
|
||||
"Delete the permission group with `group-id`."
|
||||
[group-id]
|
||||
(t2/delete! :model/PermissionsGroup :id group-id))
|
||||
|
||||
(defn data-app-group-ids
|
||||
"The IDs of permission groups owned by data apps."
|
||||
[]
|
||||
(t2/select-pks-set :model/PermissionsGroup :is_data_app_group true))
|
||||
|
||||
(defn databases-with-legacy-permissions
|
||||
"Database IDs with legacy View Data permissions from groups not owned by apps."
|
||||
[database-ids]
|
||||
(if (seq database-ids)
|
||||
(t2/select-fn-set :db_id :model/DataPermissions
|
||||
{:select [:p.db_id]
|
||||
:from [[:data_permissions :p]]
|
||||
:join [[:permissions_group :g] [:= :g.id :p.group_id]]
|
||||
:where [:and
|
||||
[:in :p.db_id database-ids]
|
||||
[:= :p.perm_type "perms/view-data"]
|
||||
[:= :p.perm_value "legacy-no-self-service"]
|
||||
[:= :g.is_data_app_group false]]})
|
||||
#{}))
|
||||
|
||||
(defn resource-collection
|
||||
"The resource collection with `collection-id`, or nil."
|
||||
[collection-id]
|
||||
(t2/select-one :model/Collection :id collection-id))
|
||||
|
||||
(defn insert-resource-collection!
|
||||
"Insert a resource collection and return it."
|
||||
[row]
|
||||
(t2/insert-returning-instance! :model/Collection row))
|
||||
|
||||
(defn update-resource-collection!
|
||||
"Apply `changes` to the resource collection with `collection-id`."
|
||||
[collection-id changes]
|
||||
(t2/update! :model/Collection :id collection-id changes))
|
||||
|
||||
(defn delete-resource-collection!
|
||||
"Delete the resource collection with `collection-id`."
|
||||
[collection-id]
|
||||
(t2/delete! :model/Collection :id collection-id))
|
||||
|
||||
(defn non-router-database-ids
|
||||
"The IDs of databases that are not routed through another database."
|
||||
[]
|
||||
(t2/select-pks-set :model/Database :router_database_id nil))
|
||||
|
||||
(defn permissions-for-paths-excluding-group
|
||||
"Permission grants for `paths`, excluding `group-id`."
|
||||
[paths group-id]
|
||||
(t2/select [:model/Permissions :group_id :object]
|
||||
:object [:in paths]
|
||||
:group_id [:not= group-id]))
|
||||
|
||||
(defn table-details
|
||||
"Table names and database details for `table-ids`."
|
||||
[table-ids]
|
||||
(t2/select :model/Table
|
||||
{:select [:t.id
|
||||
[:t.display_name :name]
|
||||
:t.schema
|
||||
[:t.db_id :database_id]
|
||||
[:d.name :database_name]]
|
||||
:from [[:metabase_table :t]]
|
||||
:join [[:metabase_database :d] [:= :d.id :t.db_id]]
|
||||
:where [:in :t.id table-ids]
|
||||
:order-by [[:d.name :asc] [:t.schema :asc] [:t.display_name :asc]]}))
|
||||
|
||||
(defn sandboxed-user-table-access
|
||||
"Sandboxed table access from non-data-app groups for the requested users and tables."
|
||||
[user-ids table-ids]
|
||||
(t2/query {:select-distinct [[:pgm.user_id :user_id]
|
||||
[:s.table_id :table_id]]
|
||||
:from [[:permissions_group_membership :pgm]]
|
||||
:join [[:sandboxes :s] [:= :s.group_id :pgm.group_id]
|
||||
[:permissions_group :pg] [:= :pg.id :pgm.group_id]]
|
||||
:where [:and
|
||||
[:in :pgm.user_id user-ids]
|
||||
[:in :s.table_id table-ids]
|
||||
[:not :pg.is_data_app_group]]}))
|
||||
|
||||
(defn unrestricted-user-table-access
|
||||
"Unrestricted table access from non-data-app groups for the requested users and tables."
|
||||
[user-ids table-ids]
|
||||
(t2/query {:select-distinct [[:pgm.user_id :user_id]
|
||||
[:t.id :table_id]]
|
||||
:from [[:permissions_group_membership :pgm]]
|
||||
:join [[:data_permissions :dp] [:= :dp.group_id :pgm.group_id]
|
||||
[:permissions_group :pg] [:= :pg.id :pgm.group_id]
|
||||
[:metabase_table :t] [:and
|
||||
[:= :t.db_id :dp.db_id]
|
||||
[:or
|
||||
[:= :dp.table_id nil]
|
||||
[:= :dp.table_id :t.id]]]]
|
||||
:where [:and
|
||||
[:in :pgm.user_id user-ids]
|
||||
[:in :t.id table-ids]
|
||||
[:not :pg.is_data_app_group]
|
||||
[:= :dp.perm_type (u/qualified-name :perms/view-data)]
|
||||
[:= :dp.perm_value "unrestricted"]]}))
|
||||
|
||||
(defn active-group-members
|
||||
"Active user memberships for `group-ids`, including API-key users."
|
||||
[group-ids]
|
||||
(t2/query {:select [[:pgm.group_id :group_id]
|
||||
[:u.id :id]
|
||||
:u.is_superuser
|
||||
:u.email]
|
||||
:from [[:permissions_group_membership :pgm]]
|
||||
:join [[:core_user :u] [:= :u.id :pgm.user_id]]
|
||||
:where [:and
|
||||
[:in :pgm.group_id group-ids]
|
||||
[:= :u.is_active true]]}))
|
||||
|
||||
@@ -1,7 +1,10 @@
|
||||
(ns metabase-enterprise.data-apps.models.data-app
|
||||
(:require
|
||||
[metabase-enterprise.data-apps.db :as data-apps.db]
|
||||
[metabase-enterprise.data-apps.resources :as data-app.resources]
|
||||
[metabase.api.common :as api]
|
||||
[metabase.models.interface :as mi]
|
||||
[metabase.premium-features.core :refer [defenterprise]]
|
||||
[methodical.core :as methodical]
|
||||
[toucan2.core :as t2])
|
||||
(:import
|
||||
@@ -25,7 +28,8 @@
|
||||
(t2/deftransforms :model/DataApp
|
||||
{:bundle transform-bundle
|
||||
;; JSON array of origins the sandboxed bundle may fetch/XHR (see config.clj).
|
||||
:allowed_hosts mi/transform-json})
|
||||
:allowed_hosts mi/transform-json
|
||||
:table_ids mi/transform-json})
|
||||
|
||||
(doto :model/DataApp
|
||||
(derive :metabase/model)
|
||||
@@ -56,7 +60,8 @@
|
||||
(t2/define-after-select :model/DataApp
|
||||
[app]
|
||||
(cond-> app
|
||||
(contains? app :allowed_hosts) (update :allowed_hosts #(or % []))))
|
||||
(contains? app :allowed_hosts) (update :allowed_hosts #(or % []))
|
||||
(contains? app :table_ids) (update :table_ids #(or % []))))
|
||||
|
||||
;; Deliberately ungated: any signed-in user may view a data app, and the `+auth`
|
||||
;; endpoints mean reaching a read check already implies authentication. See the
|
||||
@@ -73,7 +78,18 @@
|
||||
[_model _instance]
|
||||
api/*is-superuser?*)
|
||||
|
||||
(t2/define-before-delete :model/DataApp
|
||||
[app]
|
||||
(data-app.resources/delete-resources! app))
|
||||
|
||||
(methodical/defmethod mi/to-json :model/DataApp
|
||||
"Never include the raw bundle bytes in JSON."
|
||||
[data-app json-generator]
|
||||
(next-method (dissoc data-app :bundle) json-generator))
|
||||
|
||||
(defenterprise data-app-group-ids
|
||||
"The data-app permission groups (those flagged `is_data_app_group`). SSO group sync must never touch
|
||||
their membership."
|
||||
:feature :none
|
||||
[]
|
||||
(data-apps.db/data-app-group-ids))
|
||||
|
||||
@@ -0,0 +1,55 @@
|
||||
(ns metabase-enterprise.data-apps.permissions
|
||||
(:require
|
||||
[metabase-enterprise.data-apps.db :as data-apps.db]
|
||||
[metabase.permissions.core :as perms]
|
||||
[metabase.premium-features.core :refer [defenterprise]]))
|
||||
|
||||
(defenterprise data-app-view-data-permission-level
|
||||
"Use legacy across the database when an ordinary group needs it. A block on another table can deny legacy queries."
|
||||
:feature :none
|
||||
[database-id]
|
||||
(if (contains? (data-apps.db/databases-with-legacy-permissions [database-id]) database-id)
|
||||
:legacy-no-self-service
|
||||
:blocked))
|
||||
|
||||
(defn- database-level-permission?
|
||||
[rows value]
|
||||
(and (= 1 (count rows))
|
||||
(let [{:keys [table_id perm_value]} (first rows)]
|
||||
(and (nil? table_id) (= perm_value value)))))
|
||||
|
||||
(defn- reconcile-group-permissions!
|
||||
[permissions group-id database-id view-data]
|
||||
(let [permissions (if (database-level-permission? (get permissions [group-id database-id :perms/view-data]) view-data)
|
||||
permissions
|
||||
(do
|
||||
(perms/set-database-permission! group-id database-id :perms/view-data view-data)
|
||||
(perms/index-database-permissions [group-id] [database-id])))]
|
||||
;; Joining a data app must not grant permissions other than "view data".
|
||||
(doseq [perm-type [:perms/create-queries :perms/download-results
|
||||
:perms/manage-table-metadata :perms/manage-database :perms/transforms]
|
||||
:when (not (database-level-permission? (get permissions [group-id database-id perm-type]) :no))]
|
||||
(perms/set-database-permission! group-id database-id perm-type :no))))
|
||||
|
||||
(defn- reconcile-permissions!
|
||||
[group-ids database-ids]
|
||||
(when-let [group-ids (seq group-ids)]
|
||||
(let [legacy-databases (data-apps.db/databases-with-legacy-permissions database-ids)
|
||||
permissions (perms/index-database-permissions group-ids database-ids)]
|
||||
(doseq [group-id group-ids
|
||||
database-id database-ids]
|
||||
(reconcile-group-permissions! permissions group-id database-id
|
||||
(if (contains? legacy-databases database-id)
|
||||
:legacy-no-self-service
|
||||
:blocked))))))
|
||||
|
||||
(defn reconcile-app-group-permissions!
|
||||
"Reconcile one app group's data permissions during resource setup."
|
||||
[group-id database-ids]
|
||||
(reconcile-permissions! [group-id] database-ids))
|
||||
|
||||
(defenterprise reconcile-data-app-permissions!
|
||||
"Update data app groups after a group's database permission changes."
|
||||
:feature :none
|
||||
[database-ids]
|
||||
(reconcile-permissions! (data-apps.db/data-app-group-ids) database-ids))
|
||||
@@ -0,0 +1,176 @@
|
||||
(ns metabase-enterprise.data-apps.query-definition
|
||||
"Closed schemas for the data apps subset of `metabase.lib.schema.test-spec`.
|
||||
|
||||
Data apps queries support a single table-sourced stage without explicit joins or expressions."
|
||||
(:require
|
||||
[malli.core :as mc]
|
||||
[malli.error :as me]
|
||||
[malli.transform :as mtx]
|
||||
[metabase.lib.schema.common :as lib.schema.common]
|
||||
[metabase.lib.schema.id :as lib.schema.id]
|
||||
[metabase.lib.schema.literal :as literal]
|
||||
[metabase.lib.schema.temporal-bucketing :as lib.schema.temporal-bucketing]
|
||||
[metabase.util :as u]
|
||||
[metabase.util.malli.registry :as mr]))
|
||||
|
||||
(defn- normalize-binning [column]
|
||||
(let [{:keys [strategy num-bins bin-width] :as binning} (:binning column)]
|
||||
;; Keep invalid options for validation instead of silently dropping them.
|
||||
(if (and binning (mr/validate ::binning binning))
|
||||
(cond-> (dissoc column :binning)
|
||||
(not (or (:bins column) (:bin-width column)))
|
||||
(merge (case strategy
|
||||
:default {:bins :auto}
|
||||
:num-bins {:bins num-bins}
|
||||
:bin-width {:bin-width bin-width})))
|
||||
column)))
|
||||
|
||||
(defn- query-map-decoder [schema _options]
|
||||
(let [metadata-keys (::sdk-metadata (mc/properties schema))]
|
||||
{:enter (fn [value]
|
||||
(when (map? value)
|
||||
;; drop data apps specific metadata fields such as `js-type`
|
||||
(apply dissoc (update-keys value (comp keyword u/->kebab-case-en)) metadata-keys)))
|
||||
:leave (when (::normalize-binning (mc/properties schema)) normalize-binning)}))
|
||||
|
||||
(mr/def ::table-source
|
||||
[:map {:closed true :decode/normalize {:compile query-map-decoder}
|
||||
::sdk-metadata [:name :columns :fields :segments :measures]}
|
||||
[:type [:= {:decode/normalize lib.schema.common/normalize-keyword} :table]]
|
||||
[:id [:ref ::lib.schema.id/table]]])
|
||||
|
||||
(mr/def ::column
|
||||
[:map {:closed true :decode/normalize {:compile query-map-decoder}
|
||||
::sdk-metadata [:description :js-type :field-id :base-type :effective-type :default-temporal-bucket :id :metric-id]}
|
||||
[:type [:= {:decode/normalize lib.schema.common/normalize-keyword} :column]]
|
||||
[:name string?]
|
||||
[:table-id {:optional true} [:maybe ::lib.schema.id/table]]
|
||||
[:source-name {:optional true} [:maybe string?]]
|
||||
[:source-field-id {:optional true} [:maybe ::lib.schema.id/field]]
|
||||
[:display-name {:optional true} [:maybe string?]]
|
||||
[:index {:optional true} [:maybe pos-int?]]])
|
||||
|
||||
(mr/def ::temporal-bucket
|
||||
[:map {:closed true :decode/normalize {:compile query-map-decoder}}
|
||||
[:unit {:optional true} [:maybe ::lib.schema.temporal-bucketing/unit]]])
|
||||
|
||||
(mr/def ::auto-bin
|
||||
[:= {:decode/normalize lib.schema.common/normalize-keyword} :auto])
|
||||
|
||||
(mr/def ::bin-count-bucket
|
||||
[:map {:closed true :decode/normalize {:compile query-map-decoder}}
|
||||
[:bins {:optional true} [:maybe [:or pos-int? ::auto-bin]]]])
|
||||
|
||||
(mr/def ::bin-width-bucket
|
||||
[:map {:closed true :decode/normalize {:compile query-map-decoder}}
|
||||
[:bin-width {:optional true} [:maybe [:or ::lib.schema.common/positive-number ::auto-bin]]]])
|
||||
|
||||
(mr/def ::binning
|
||||
[:multi {:decode/normalize lib.schema.common/normalize-map
|
||||
:dispatch (comp keyword :strategy)}
|
||||
[:default [:map {:closed true}
|
||||
[:strategy [:= {:decode/normalize lib.schema.common/normalize-keyword} :default]]]]
|
||||
[:num-bins [:map {:closed true}
|
||||
[:strategy [:= {:decode/normalize lib.schema.common/normalize-keyword} :num-bins]]
|
||||
[:num-bins pos-int?]]]
|
||||
[:bin-width [:map {:closed true}
|
||||
[:strategy [:= {:decode/normalize lib.schema.common/normalize-keyword} :bin-width]]
|
||||
[:bin-width ::lib.schema.common/positive-number]]]])
|
||||
|
||||
(mr/def ::column-with-binning
|
||||
[:merge
|
||||
::column
|
||||
::temporal-bucket
|
||||
::bin-count-bucket
|
||||
::bin-width-bucket
|
||||
[:map {:closed true :decode/normalize {:compile query-map-decoder}
|
||||
::normalize-binning true}
|
||||
[:binning {:optional true} ::binning]]])
|
||||
|
||||
(mr/def ::breakout
|
||||
[:ref ::column-with-binning])
|
||||
|
||||
(mr/def ::order-by
|
||||
[:merge
|
||||
::column-with-binning
|
||||
[:map {:closed true :decode/normalize {:compile query-map-decoder}}
|
||||
[:direction {:optional true} [:maybe [:enum {:decode/normalize lib.schema.common/normalize-keyword} :asc :desc]]]]])
|
||||
|
||||
(mr/def ::literal-expression
|
||||
[:map {:closed true :decode/normalize {:compile query-map-decoder}}
|
||||
[:type [:= {:decode/normalize lib.schema.common/normalize-keyword} :literal]]
|
||||
[:value [:ref ::literal/literal]]])
|
||||
|
||||
(mr/def ::operator
|
||||
keyword?)
|
||||
|
||||
(mr/def ::operator-expression
|
||||
[:map {:closed true :decode/normalize {:compile query-map-decoder}
|
||||
::sdk-metadata [:columns]}
|
||||
[:type [:= {:decode/normalize lib.schema.common/normalize-keyword} :operator]]
|
||||
[:operator ::operator]
|
||||
[:args {:default []} [:sequential [:ref ::expression]]]])
|
||||
|
||||
(mr/def ::expression
|
||||
[:multi {:decode/normalize lib.schema.common/normalize-map-no-kebab-case
|
||||
:dispatch (comp keyword :type)}
|
||||
[:column [:ref ::column]]
|
||||
[:literal [:ref ::literal-expression]]
|
||||
[:operator [:ref ::operator-expression]]])
|
||||
|
||||
(mr/def ::segment
|
||||
[:map {:closed true :decode/normalize {:compile query-map-decoder}
|
||||
::sdk-metadata [:name :table-id]}
|
||||
[:type [:= {:decode/normalize lib.schema.common/normalize-keyword} :segment]]
|
||||
[:id [:ref ::lib.schema.id/segment]]])
|
||||
|
||||
(mr/def ::measure
|
||||
[:map {:closed true :decode/normalize {:compile query-map-decoder}
|
||||
::sdk-metadata [:name :table-id :columns]}
|
||||
[:type [:= {:decode/normalize lib.schema.common/normalize-keyword} :measure]]
|
||||
[:id [:ref ::lib.schema.id/measure]]])
|
||||
|
||||
(mr/def ::metric
|
||||
[:map {:closed true :decode/normalize {:compile query-map-decoder}
|
||||
::sdk-metadata [:name :database-id :source-table-id :source-card-id :mapped-table-ids :columns :dimensions]}
|
||||
[:type [:= {:decode/normalize lib.schema.common/normalize-keyword} :metric]]
|
||||
[:id [:ref ::lib.schema.id/metric]]])
|
||||
|
||||
(mr/def ::aggregation
|
||||
[:or
|
||||
[:ref ::expression]
|
||||
[:ref ::measure]
|
||||
[:ref ::metric]])
|
||||
|
||||
(mr/def ::stage
|
||||
[:map {:closed true :decode/normalize {:compile query-map-decoder}
|
||||
::sdk-metadata [:enabled]}
|
||||
[:source ::table-source]
|
||||
[:fields {:optional true} [:maybe [:sequential ::column]]]
|
||||
[:filters {:optional true} [:maybe [:sequential [:or ::expression ::segment]]]]
|
||||
[:aggregations {:optional true} [:maybe [:sequential ::aggregation]]]
|
||||
[:breakouts {:optional true} [:maybe [:sequential ::breakout]]]
|
||||
[:order-bys {:optional true} [:maybe [:sequential ::order-by]]]
|
||||
[:limit {:optional true} [:maybe number?]]])
|
||||
|
||||
(defn- strict-query-decoder [schema _options]
|
||||
(let [decode (mc/decoder schema (mtx/transformer
|
||||
(mtx/string-transformer)
|
||||
(mtx/json-transformer)
|
||||
(mtx/default-value-transformer)
|
||||
{:name :normalize}))
|
||||
validate (mr/validator schema)]
|
||||
(fn [value]
|
||||
;; validate before the middleware silently drops unsupported fields
|
||||
(let [query (decode value)]
|
||||
(when-not (validate query)
|
||||
(throw (ex-info "Invalid data app query"
|
||||
{:status-code 400
|
||||
:errors (me/humanize (mr/explain schema query))})))
|
||||
query))))
|
||||
|
||||
(mr/def ::query-definition
|
||||
[:map {:closed true
|
||||
:decode/api {:compile strict-query-decoder}
|
||||
:decode/normalize {:compile query-map-decoder}}
|
||||
[:stages [:sequential {:min 1 :max 1} ::stage]]])
|
||||
@@ -0,0 +1,97 @@
|
||||
(ns metabase-enterprise.data-apps.resources
|
||||
"Lifecycle for the permission group and resource collection owned by a data app."
|
||||
(:require
|
||||
[metabase-enterprise.data-apps.db :as data-apps.db]
|
||||
[metabase-enterprise.data-apps.permissions :as data-app.permissions]
|
||||
[metabase.collections.core :as collection]
|
||||
[metabase.permissions.core :as perms]
|
||||
[metabase.request.core :as request]))
|
||||
|
||||
(set! *warn-on-reflection* true)
|
||||
|
||||
(defn- resource-name [app]
|
||||
(format "Data App: %s" (:name app)))
|
||||
|
||||
(defn- create-permission-group! [app]
|
||||
(let [group (data-apps.db/insert-permission-group! {:name (resource-name app)
|
||||
:is_data_app_group true})]
|
||||
(data-apps.db/update-data-app! (:id app) {:permission_group_id (:id group)})
|
||||
group))
|
||||
|
||||
(defn- permission-group! [app]
|
||||
(or (some->> (:permission_group_id app)
|
||||
(data-apps.db/permission-group))
|
||||
(create-permission-group! app)))
|
||||
|
||||
(defn- restore-trashed-collection!
|
||||
"Bring `collection` back out of the trash, with everything archived alongside it.
|
||||
|
||||
Trashing it archives every copy inside, and an app is served from those copies,
|
||||
so a sync that left it there would report success over an app whose viewers see
|
||||
nothing. It goes back to the root, where the app created it: restoring in place
|
||||
fails outright when an ancestor is still in the trash.
|
||||
|
||||
The restore runs with full permissions because a repository import reaches this
|
||||
from a scheduled task, where no user is bound; the collection is the app's own,
|
||||
so there is no one else's decision to weigh."
|
||||
[collection]
|
||||
(when (:archived collection)
|
||||
(request/as-admin
|
||||
(collection/archive-or-unarchive-collection! collection
|
||||
{:archived false, :parent_id nil}))))
|
||||
|
||||
(defn- apply-collection-permissions!
|
||||
"Gives the group read access. Preserves permission grants that are already correct."
|
||||
[group collection]
|
||||
(let [read-path (perms/collection-read-path collection)
|
||||
write-path (perms/collection-readwrite-path collection)
|
||||
permissions-by-group (group-by :group_id
|
||||
(data-apps.db/permissions-for-paths-excluding-group
|
||||
["/" read-path write-path] (:id (perms/admin-group))))
|
||||
app-read-only? (= #{read-path} (set (map :object (get permissions-by-group (:id group)))))]
|
||||
;; Remove write grants and access from other groups
|
||||
(doseq [group-id (keys permissions-by-group)
|
||||
:when (not (and (= group-id (:id group)) app-read-only?))]
|
||||
(perms/revoke-collection-permissions! group-id collection))
|
||||
(when-not app-read-only?
|
||||
(perms/grant-collection-read-permissions! group collection))))
|
||||
|
||||
(defn- apply-resource-permissions!
|
||||
[group collection]
|
||||
(data-app.permissions/reconcile-app-group-permissions! (:id group) (data-apps.db/non-router-database-ids))
|
||||
(apply-collection-permissions! group collection))
|
||||
|
||||
(defn- create-resource-collection! [app]
|
||||
(let [collection (data-apps.db/insert-resource-collection! {:name (resource-name app)
|
||||
:location "/"})]
|
||||
(data-apps.db/update-data-app! (:id app) {:resource_collection_id (:id collection)})
|
||||
collection))
|
||||
|
||||
(defn- resource-collection! [app]
|
||||
(or (some->> (:resource_collection_id app)
|
||||
(data-apps.db/resource-collection))
|
||||
(create-resource-collection! app)))
|
||||
|
||||
(defn ensure-resources!
|
||||
"Create or restore the server-owned permission resources for `app` and return their IDs."
|
||||
[app]
|
||||
(perms/with-global-permissions-lock
|
||||
(let [app (data-apps.db/non-blob-data-app (:id app))
|
||||
group (permission-group! app)
|
||||
collection (resource-collection! app)]
|
||||
(data-apps.db/update-permission-group! (:id group)
|
||||
{:name (resource-name app)})
|
||||
(data-apps.db/update-resource-collection! (:id collection)
|
||||
{:name (resource-name app)})
|
||||
(restore-trashed-collection! collection)
|
||||
(apply-resource-permissions! group collection)
|
||||
{:permission_group_id (:id group)
|
||||
:resource_collection_id (:id collection)})))
|
||||
|
||||
(defn delete-resources!
|
||||
"Delete the generated collection and permission group referenced by `app`."
|
||||
[{:keys [permission_group_id resource_collection_id]}]
|
||||
(when resource_collection_id
|
||||
(data-apps.db/delete-resource-collection! resource_collection_id))
|
||||
(when permission_group_id
|
||||
(data-apps.db/delete-permission-group! permission_group_id)))
|
||||
@@ -20,7 +20,11 @@
|
||||
[:enabled :boolean]
|
||||
[:created_at ms/TemporalInstant]
|
||||
[:updated_at ms/TemporalInstant]
|
||||
[:description [:maybe :string]]])
|
||||
[:description [:maybe :string]]
|
||||
[:resource_collection_id [:maybe ms/PositiveInt]]
|
||||
[:permission_group_id [:maybe ms/PositiveInt]]
|
||||
[:table_ids [:sequential ms/PositiveInt]]
|
||||
[:draft :boolean]])
|
||||
|
||||
(mr/def ::data-app.update
|
||||
"What an update (or insert) of a DataApp accepts: every column of `:data_app` except `id`, all optional."
|
||||
@@ -37,4 +41,8 @@
|
||||
[:enabled {:optional true} [:maybe :boolean]]
|
||||
[:created_at {:optional true} [:maybe ms/TemporalInstant]]
|
||||
[:updated_at {:optional true} [:maybe ms/TemporalInstant]]
|
||||
[:description {:optional true} [:maybe :string]]])
|
||||
[:description {:optional true} [:maybe :string]]
|
||||
[:resource_collection_id {:optional true} [:maybe ms/PositiveInt]]
|
||||
[:permission_group_id {:optional true} [:maybe ms/PositiveInt]]
|
||||
[:table_ids {:optional true} [:maybe [:sequential ms/PositiveInt]]]
|
||||
[:draft {:optional true} [:maybe :boolean]]])
|
||||
|
||||
@@ -15,6 +15,8 @@
|
||||
[clojure.string :as str]
|
||||
[metabase-enterprise.data-apps.config :as data-app.config]
|
||||
[metabase-enterprise.data-apps.db :as data-apps.db]
|
||||
[metabase-enterprise.data-apps.resources :as data-app.resources]
|
||||
[metabase.permissions.core :as perms]
|
||||
[metabase.premium-features.core :as premium-features]
|
||||
[metabase.settings.core :as setting]
|
||||
[metabase.util.i18n :refer [tru]]
|
||||
@@ -47,6 +49,31 @@
|
||||
(when-not (str/blank? url)
|
||||
url)))
|
||||
|
||||
(defn- create-draft!
|
||||
[slug]
|
||||
(let [app (t2/with-transaction [_conn]
|
||||
(when-not (data-apps.db/data-app-exists? slug)
|
||||
(data-apps.db/insert-data-app!
|
||||
{:name slug
|
||||
:display_name slug
|
||||
:bundle_path (format "%s/%s/%s" data-app.config/apps-dir slug data-app.config/config-file-name)
|
||||
:sync_error (tru "Bundle not synced yet.")
|
||||
:draft true}))
|
||||
(data-apps.db/non-blob-data-app-by-slug slug))]
|
||||
(data-app.resources/ensure-resources! app)))
|
||||
|
||||
(defn ensure-draft!
|
||||
"Create a data app draft when needed and ensure its permission resources.
|
||||
A later repository import fills the same row with the authoritative manifest
|
||||
and bundle."
|
||||
[slug]
|
||||
(try
|
||||
(create-draft! slug)
|
||||
(catch Throwable e
|
||||
(if (data-apps.db/data-app-exists? slug)
|
||||
(create-draft! slug)
|
||||
(throw e)))))
|
||||
|
||||
;;; ----------------------------------------------------- Discovery -----------------------------------------------------
|
||||
|
||||
(defn- discover-app-configs
|
||||
@@ -72,7 +99,8 @@
|
||||
:when (some #{config-path} (list-dir dir))]
|
||||
(try
|
||||
(if-let [content (read-file config-path)]
|
||||
(let [{:keys [slug display_name description path allowed_hosts]} (data-app.config/parse-app-config (->bytes content) dir)]
|
||||
(let [{:keys [slug display_name description path allowed_hosts]}
|
||||
(data-app.config/parse-app-config (->bytes content) dir)]
|
||||
{:slug slug
|
||||
:display_name display_name
|
||||
:description description
|
||||
@@ -125,46 +153,79 @@
|
||||
(data-apps.db/update-data-app-by-slug! slug {:sync_error message})
|
||||
true)))
|
||||
|
||||
(defn- sync-app!
|
||||
"Materialize one app. On bundle failure, the row's metadata is still upserted
|
||||
with `sync_error` set so the app appears in the list with its failure; the
|
||||
previously cached bundle (if any) is kept. `existing` is the app's pre-sync row
|
||||
(or nil); returns true when this sync actually changed the app's content (a new
|
||||
app, differing bundle/metadata, or a new failure) so callers can count changes."
|
||||
(defn- prepare-app
|
||||
"Read one app's bundle before taking the permissions lock.
|
||||
A bundle failure preserves the cached bundle and prepares metadata with `sync_error` instead."
|
||||
[existing {:keys [slug display_name description bundle sha read-file allowed_hosts]}]
|
||||
(let [metadata {:display_name display_name
|
||||
:description description
|
||||
:allowed_hosts allowed_hosts
|
||||
:bundle_path bundle}]
|
||||
(try
|
||||
(let [content (read-file bundle)
|
||||
_ (when-not content
|
||||
(throw (ex-info (tru "Bundle file \"{0}\" not found in the repository." bundle)
|
||||
{:status-code 400})))
|
||||
^bytes bytes (->bytes content)]
|
||||
(when (> (alength bytes) max-bundle-bytes)
|
||||
(throw (ex-info (tru "Bundle for \"{0}\" must be less than {1} MiB."
|
||||
slug (quot max-bundle-bytes (* 1024 1024)))
|
||||
{:status-code 413})))
|
||||
(let [fields (assoc metadata :bundle_hash (bytes-hash bytes))]
|
||||
{:fields (assoc fields :bundle bytes :last_synced_sha sha
|
||||
:sync_error nil :draft false)
|
||||
:changed? (app-content-changed? existing fields)}))
|
||||
(catch Throwable e
|
||||
(log/warnf "[data-app] failed to read bundle for app %s: %s" slug (ex-message e))
|
||||
{:fields (assoc metadata :sync_error (ex-message e))
|
||||
:changed? (or (nil? existing)
|
||||
(not= (:sync_error existing) (ex-message e))
|
||||
(app-metadata-changed? existing metadata))}))))
|
||||
|
||||
(defn- record-app-failure!
|
||||
"Record a failure after the app transaction rolls back. Keep existing content and resource links intact."
|
||||
[existing slug fields message]
|
||||
(try
|
||||
(let [content (read-file bundle)
|
||||
_ (when-not content
|
||||
(throw (ex-info (tru "Bundle file \"{0}\" not found in the repository." bundle)
|
||||
{:status-code 400})))
|
||||
^bytes bytes (->bytes content)]
|
||||
(when (> (alength bytes) max-bundle-bytes)
|
||||
(throw (ex-info (tru "Bundle for \"{0}\" must be less than {1} MiB."
|
||||
slug (quot max-bundle-bytes (* 1024 1024)))
|
||||
{:status-code 413})))
|
||||
(let [fields {:display_name display_name
|
||||
:description description
|
||||
:allowed_hosts allowed_hosts
|
||||
:bundle_path bundle
|
||||
:bundle_hash (bytes-hash bytes)}]
|
||||
(upsert-by-name! slug (assoc fields
|
||||
:bundle bytes
|
||||
:last_synced_sha sha
|
||||
:sync_error nil))
|
||||
(app-content-changed? existing fields)))
|
||||
(t2/with-transaction [_conn]
|
||||
(if existing
|
||||
(mark-config-error! existing slug message)
|
||||
(when fields
|
||||
(upsert-by-name! slug (assoc (select-keys fields [:display_name :description :allowed_hosts :bundle_path])
|
||||
:sync_error message))
|
||||
true)))
|
||||
(catch Throwable e
|
||||
(let [fields {:display_name display_name
|
||||
:description description
|
||||
:allowed_hosts allowed_hosts
|
||||
:bundle_path bundle}]
|
||||
(upsert-by-name! slug (assoc fields :sync_error (ex-message e)))
|
||||
(log/warnf "[data-app] could not record sync error for app %s: %s" slug (ex-message e))
|
||||
false)))
|
||||
|
||||
(defn- sync-app!
|
||||
"Commit one app and its permission resources together. Return whether its recorded state changed."
|
||||
[existing {:keys [slug config-error] :as cfg}]
|
||||
(let [{:keys [fields changed?]} (when-not config-error (prepare-app existing cfg))]
|
||||
(try
|
||||
(perms/with-global-permissions-lock
|
||||
(t2/with-transaction [_conn]
|
||||
(data-apps.db/publish-data-app-drafts! [slug])
|
||||
(if config-error
|
||||
(mark-config-error! existing slug config-error)
|
||||
(do
|
||||
(upsert-by-name! slug fields)
|
||||
(data-app.resources/ensure-resources! (data-apps.db/non-blob-data-app-by-slug slug))
|
||||
changed?))))
|
||||
(catch Throwable e
|
||||
(log/warnf "[data-app] failed to sync app %s: %s" slug (ex-message e))
|
||||
;; A failing app counts as a change when the failure is new, and also when
|
||||
;; the metadata above moved: the upsert stores an edited description even
|
||||
;; though the bundle still fails, so the pull summary has to admit it.
|
||||
(or (nil? existing)
|
||||
(not= (:sync_error existing) (ex-message e))
|
||||
(app-metadata-changed? existing fields))))))
|
||||
(record-app-failure! existing slug fields (ex-message e))))))
|
||||
|
||||
(defn- prune-apps!
|
||||
"Delete all absent apps. Preserve the sync summary if pruning rolls back."
|
||||
[present-slugs]
|
||||
(try
|
||||
{:removed (t2/with-transaction [_conn]
|
||||
(if (seq present-slugs)
|
||||
(data-apps.db/delete-data-apps-not-named! present-slugs)
|
||||
(data-apps.db/delete-all-data-apps!)))}
|
||||
(catch Throwable e
|
||||
(log/warnf "[data-app] pruning failed: %s" (ex-message e))
|
||||
{:removed 0 :pruning-error (ex-message e)})))
|
||||
|
||||
(defn import-from-snapshot!
|
||||
"Materialize data apps from a synced repo `snapshot`:
|
||||
@@ -174,11 +235,13 @@
|
||||
:sha <commit-sha-string>}
|
||||
|
||||
Discovers every `data_apps/<dir>/data_app.yaml`, upserts a row per app, and prunes
|
||||
rows whose directory is gone from the snapshot — all in one transaction. Returns
|
||||
rows whose directory is gone from the snapshot. Apps are synced independently, so
|
||||
one failing app cannot roll back the others. Returns
|
||||
`{:synced <n>, :changed <n>, :removed <n>, :sha <sha>, :config-errors [<msg> ...]}`,
|
||||
where `:changed` counts apps actually created/updated (a `last_synced_sha` bump on
|
||||
unchanged content does not count) and `:removed` counts apps dropped for no longer
|
||||
being in the repo.
|
||||
being in the repo. If pruning fails, `:removed` is zero and `:pruning-error` contains
|
||||
the error message. Counts for committed app updates are preserved.
|
||||
|
||||
Two apps can't collide on a slug here: a slug *is* an app's directory name (see
|
||||
the config namespace), a repo can't hold two `data_apps/<slug>` directories, and
|
||||
@@ -195,28 +258,14 @@
|
||||
;; pre-sync rows, so we can tell a real change from a sha/timestamp bump
|
||||
existing (into {} (map (juxt :name identity))
|
||||
(data-apps.db/data-apps-sync-info))
|
||||
{:keys [changed removed]}
|
||||
(t2/with-transaction [_conn]
|
||||
(let [changed (reduce (fn [n {:keys [slug config-error] :as cfg}]
|
||||
(cond-> n
|
||||
;; A parse failure on an app that still exists marks
|
||||
;; that row failed rather than syncing it; everything
|
||||
;; else is materialized normally.
|
||||
(if config-error
|
||||
(mark-config-error! (get existing slug) slug config-error)
|
||||
(sync-app! (get existing slug)
|
||||
(assoc cfg :sha sha :read-file read-file)))
|
||||
inc))
|
||||
0 results)
|
||||
;; `enabled` is deliberately not consulted — see the README's
|
||||
;; source-of-truth table. (`[:not-in #{}]` is invalid SQL, so delete-all.)
|
||||
removed (if (seq present-slugs)
|
||||
(data-apps.db/delete-data-apps-not-named! present-slugs)
|
||||
(data-apps.db/delete-all-data-apps!))]
|
||||
{:changed changed, :removed removed}))]
|
||||
changed (count (filter (fn [{:keys [slug] :as cfg}]
|
||||
(sync-app! (get existing slug) (assoc cfg :sha sha :read-file read-file)))
|
||||
results))
|
||||
{:keys [removed] :as pruning-result} (prune-apps! present-slugs)]
|
||||
(log/infof "[data-app] synced sha=%s apps=%d changed=%d removed=%d errors=%d"
|
||||
sha (count good) changed removed (count errors))
|
||||
{:synced (count good), :changed changed, :removed removed, :sha sha, :config-errors errors}))
|
||||
(merge {:synced (count good), :changed changed, :sha sha, :config-errors errors}
|
||||
pruning-result)))
|
||||
|
||||
(defn sync-from-snapshot!
|
||||
"Entry point for the remote-sync import pipeline. Materializes data apps from
|
||||
|
||||
@@ -0,0 +1,84 @@
|
||||
(ns metabase-enterprise.data-apps.user-access
|
||||
(:require
|
||||
[clojure.string :as str]
|
||||
[metabase-enterprise.data-apps.db :as data-apps.db]))
|
||||
|
||||
(set! *warn-on-reflection* true)
|
||||
|
||||
(defn- table-details
|
||||
[table-ids]
|
||||
(if (seq table-ids)
|
||||
(data-apps.db/table-details table-ids)
|
||||
[]))
|
||||
|
||||
(defn- sandboxed-user-table-pairs
|
||||
[user-ids table-ids]
|
||||
(if (and (seq user-ids) (seq table-ids))
|
||||
(into #{}
|
||||
(map (juxt :user_id :table_id))
|
||||
(data-apps.db/sandboxed-user-table-access user-ids table-ids))
|
||||
#{}))
|
||||
|
||||
(defn- unrestricted-user-table-pairs
|
||||
[user-ids table-ids]
|
||||
(if (and (seq user-ids) (seq table-ids))
|
||||
(into #{}
|
||||
(map (juxt :user_id :table_id))
|
||||
(data-apps.db/unrestricted-user-table-access user-ids table-ids))
|
||||
#{}))
|
||||
|
||||
(defn- user-warning
|
||||
[user tables unrestricted-pairs sandboxed-pairs]
|
||||
(when-not (:is_superuser user)
|
||||
(let [user-id (:id user)
|
||||
has-access? (fn [{table-id :id}]
|
||||
(or (unrestricted-pairs [user-id table-id])
|
||||
(sandboxed-pairs [user-id table-id])))
|
||||
missing (remove has-access? tables)]
|
||||
(when (seq missing)
|
||||
{:user_id (:id user)
|
||||
:missing_tables (vec missing)}))))
|
||||
|
||||
(defn permission-warnings
|
||||
"Returns access warnings for the requested users who cannot access every table in `table-ids`.
|
||||
Superusers and users with unrestricted or sandboxed access to every table are omitted."
|
||||
[table-ids users]
|
||||
(if (seq table-ids)
|
||||
(let [tables (table-details table-ids)
|
||||
user-ids (map :id users)
|
||||
unrestricted-pairs (unrestricted-user-table-pairs user-ids table-ids)
|
||||
sandboxed-pairs (sandboxed-user-table-pairs user-ids table-ids)]
|
||||
(into [] (keep #(user-warning % tables unrestricted-pairs sandboxed-pairs)) users))
|
||||
[]))
|
||||
|
||||
(defn- active-group-members
|
||||
[group-ids]
|
||||
(if (seq group-ids)
|
||||
(->> (data-apps.db/active-group-members group-ids)
|
||||
(remove #(str/ends-with? (:email %) "@api-key.invalid")))
|
||||
[]))
|
||||
|
||||
(defn- group-has-permission-warning?
|
||||
[users table-ids unrestricted-pairs sandboxed-pairs]
|
||||
(let [tables (mapv (fn [table-id] {:id table-id}) table-ids)]
|
||||
(boolean (some #(user-warning % tables unrestricted-pairs sandboxed-pairs) users))))
|
||||
|
||||
(defn groups-with-permission-warnings
|
||||
"The permission group IDs for apps with an active member who cannot access every dependent table."
|
||||
[apps]
|
||||
(let [apps (filter #(and (:permission_group_id %) (seq (:table_ids %))) apps)
|
||||
group-ids (into #{} (map :permission_group_id) apps)
|
||||
memberships (active-group-members group-ids)
|
||||
group->users (group-by :group_id memberships)
|
||||
user-ids (into #{} (map :id) memberships)
|
||||
table-ids (into #{} (mapcat :table_ids) apps)
|
||||
unrestricted-pairs (unrestricted-user-table-pairs user-ids table-ids)
|
||||
sandboxed-pairs (sandboxed-user-table-pairs user-ids table-ids)]
|
||||
(into #{}
|
||||
(keep (fn [{:keys [permission_group_id table_ids]}]
|
||||
(when (group-has-permission-warning? (group->users permission_group_id)
|
||||
table_ids
|
||||
unrestricted-pairs
|
||||
sandboxed-pairs)
|
||||
permission_group_id)))
|
||||
apps)))
|
||||
@@ -4,6 +4,7 @@
|
||||
[metabase-enterprise.advanced-permissions.common :as advanced-permissions.common]
|
||||
[metabase-enterprise.impersonation.util-test :as advanced-perms.api.tu]
|
||||
[metabase.driver :as driver]
|
||||
[metabase.permissions-rest.data-permissions.graph :as data-perms.graph]
|
||||
[metabase.permissions.models.data-permissions :as data-perms]
|
||||
[metabase.permissions.models.permissions :as perms]
|
||||
[metabase.permissions.models.permissions-group :as perms-group]
|
||||
@@ -165,6 +166,43 @@
|
||||
(is (= :unrestricted (perm-value table-id-1)))
|
||||
(is (= :blocked (perm-value table-id-3)))))))))
|
||||
|
||||
(deftest new-view-data-permission-levels-block-data-app-groups-test
|
||||
(testing "data-app groups block newly-synced tables/databases even with no `:blocked` row (full manifest)"
|
||||
(mt/with-additional-premium-features #{:sandboxes :advanced-permissions}
|
||||
(mt/with-temp [:model/PermissionsGroup {app-group-id :id} {:is_data_app_group true}
|
||||
:model/PermissionsGroup {ctl-group-id :id} {} ; a normal, non-data-app group
|
||||
:model/DataApp _ {:name "perm-test-app" :display_name "Perm Test"
|
||||
:bundle_path "app/dist/index.js"
|
||||
:permission_group_id app-group-id}
|
||||
:model/Database {db-id :id} {}
|
||||
:model/Table {table-id-1 :id} {:db_id db-id :schema "PUBLIC"}
|
||||
:model/Table {table-id-2 :id} {:db_id db-id :schema "PUBLIC"}]
|
||||
;; effective (coalesced) view-data — the control group inherits the DB-level value with no
|
||||
;; table row of its own, so a raw-row check would see nil.
|
||||
(let [effective (fn [group-id table-id]
|
||||
(data-perms/table-permission-for-groups #{group-id} :perms/view-data db-id table-id))]
|
||||
(testing "a late table is blocked for the data-app group but not for a normal group"
|
||||
;; a "full manifest" for both groups: all existing tables :unrestricted, so no `:blocked` row exists
|
||||
(doseq [group-id [app-group-id ctl-group-id]]
|
||||
(data-perms/set-table-permission! group-id table-id-1 :perms/view-data :unrestricted)
|
||||
(data-perms/set-table-permission! group-id table-id-2 :perms/view-data :unrestricted))
|
||||
(mt/with-temp [:model/Table {table-id-3 :id} {:db_id db-id :schema "PUBLIC"}]
|
||||
(is (= :blocked (effective app-group-id table-id-3))
|
||||
"the late table is blocked for the data-app group")
|
||||
(is (= :unrestricted (effective ctl-group-id table-id-3))
|
||||
"the same late table is NOT blocked for a normal group — the rule is scoped to app groups")
|
||||
(is (= :unrestricted (effective app-group-id table-id-1))
|
||||
"the app's declared tables stay unrestricted")
|
||||
;; the going-granular apply keeps a clean per-table shape, so the permissions graph the
|
||||
;; admin panel loads still builds (M1's db-level+table-override mix used to 500 it).
|
||||
(is (map? (data-perms.graph/api-graph))
|
||||
"the whole permissions graph still builds")))
|
||||
(testing "a newly-added database is blocked only for the data-app group"
|
||||
(let [levels (advanced-permissions.common/new-database-view-data-permission-levels
|
||||
[app-group-id ctl-group-id])]
|
||||
(is (= :blocked (levels app-group-id)) "blocked for the data-app group")
|
||||
(is (not= :blocked (levels ctl-group-id)) "not blocked for a normal group"))))))))
|
||||
|
||||
(deftest new-table-view-data-permission-levels-without-premium-features-test
|
||||
(testing "A newly-synced table fails CLOSED to :blocked for a sandboxed group even when premium features are unavailable (UXW-4927)"
|
||||
;; This is the incident path: a table is discovered during sync while the sandboxes /
|
||||
|
||||
@@ -2,8 +2,16 @@
|
||||
(:require
|
||||
[clojure.string :as str]
|
||||
[clojure.test :refer :all]
|
||||
[metabase-enterprise.data-apps.query-definition :as query-definition]
|
||||
[metabase-enterprise.data-apps.resources :as data-app.resources]
|
||||
[metabase-enterprise.data-apps.sync :as data-app.sync]
|
||||
[metabase-enterprise.data-apps.user-access :as data-app.user-access]
|
||||
[metabase-enterprise.remote-sync.source :as source]
|
||||
[metabase.actions.core :as actions]
|
||||
[metabase.api.macros.defendpoint.closed-schemas :as closed-schemas]
|
||||
[metabase.lib.core :as lib]
|
||||
[metabase.lib.metadata :as lib.metadata]
|
||||
[metabase.permissions.core :as perms]
|
||||
[metabase.test :as mt]
|
||||
[toucan2.core :as t2]))
|
||||
|
||||
@@ -48,45 +56,667 @@
|
||||
|
||||
;;; ---------------------------------------------- Permissions ----------------------------------------------
|
||||
|
||||
(deftest non-superuser-can-view-and-list-but-not-manage-test
|
||||
(deftest data-app-access-requires-read-access-to-its-resource-collection-test
|
||||
;; global mode so the `:data-apps-preview` premium feature is visible to the real-HTTP
|
||||
;; `user-real-request` calls below (which run on Jetty threads that don't inherit
|
||||
;; a thread-local `binding`).
|
||||
(mt/test-helpers-set-global-values!
|
||||
(mt/with-premium-features #{:data-apps-preview}
|
||||
(mt/with-model-cleanup [:model/DataApp]
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(create-app!)
|
||||
(testing "a non-superuser can view (open) a data app"
|
||||
(is (= [{:name "demo" :display_name "Demo"}]
|
||||
(mt/user-http-request :rasta :get 200 "apps")))
|
||||
(is (= {:name "demo" :display_name "Demo"}
|
||||
(mt/user-http-request :rasta :get 200 "apps/demo")))
|
||||
(is (str/includes?
|
||||
(str (mt/user-real-request :rasta :get 200 "apps/demo/bundle"))
|
||||
"BUNDLE")))
|
||||
(testing "but is still forbidden from managing data apps"
|
||||
(let [app (t2/select-one :model/DataApp :name "demo")
|
||||
{:keys [permission_group_id]} (data-app.resources/ensure-resources! app)]
|
||||
(testing "a non-member cannot open a data app or load its bundle"
|
||||
(is (= [{:name "demo" :display_name "Demo"}]
|
||||
(mt/user-http-request :rasta :get 200 "apps")))
|
||||
(is (= "You don't have permissions to do that."
|
||||
(mt/user-http-request :rasta :get 403 "apps/demo")))
|
||||
(is (= "You don't have permissions to do that."
|
||||
(mt/user-http-request :rasta :get 403 "apps/demo/bundle"))))
|
||||
(testing "a member can open a data app"
|
||||
(perms/add-user-to-group! (mt/user->id :rasta) permission_group_id)
|
||||
(is (= {:name "demo" :display_name "Demo"}
|
||||
(mt/user-http-request :rasta :get 200 "apps/demo")))
|
||||
(is (str/includes?
|
||||
(str (mt/user-real-request :rasta :get 200 "apps/demo/bundle"))
|
||||
"BUNDLE"))))
|
||||
(testing "a non-superuser is still forbidden from managing data apps"
|
||||
(is (= "You don't have permissions to do that."
|
||||
(mt/user-http-request :rasta :get 403 "apps/repo-status")))
|
||||
(is (= "You don't have permissions to do that."
|
||||
(mt/user-http-request :rasta :put 403 "apps/demo" {:enabled false}))))))))
|
||||
|
||||
(deftest data-app-without-a-resource-collection-is-not-published-test
|
||||
(mt/test-helpers-set-global-values!
|
||||
(mt/with-premium-features #{:data-apps-preview}
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(create-app!)
|
||||
(is (nil? (t2/select-one-fn :resource_collection_id :model/DataApp :name "demo"))
|
||||
"precondition: the app has no resource collection")
|
||||
(testing "an app with no resource collection is not published: neither its metadata nor its
|
||||
bundle is served — to anyone — and a 409 lets the client show a \"not published\" screen"
|
||||
(doseq [user [:rasta :crowberto]]
|
||||
(is (= "This data app has not been published yet."
|
||||
(mt/user-http-request user :get 409 "apps/demo")))
|
||||
(is (= "This data app has not been published yet."
|
||||
(mt/user-http-request user :get 409 "apps/demo/bundle")))))))))
|
||||
|
||||
(deftest superuser-can-manage-and-view-test
|
||||
(mt/test-helpers-set-global-values!
|
||||
(mt/with-premium-features #{:data-apps-preview}
|
||||
(mt/with-model-cleanup [:model/DataApp]
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(create-app!)
|
||||
(let [app (t2/select-one :model/DataApp :name "demo")
|
||||
{:keys [resource_collection_id]} (data-app.resources/ensure-resources! app)]
|
||||
(is (not (some #(= resource_collection_id (:id %))
|
||||
(mt/user-http-request :rasta :get 200 "collection")))))
|
||||
(testing "a superuser can list, read metadata, and serve the bundle"
|
||||
(is (=? [{:name "demo" :display_name "Demo"}]
|
||||
(mt/user-http-request :crowberto :get 200 "apps")))
|
||||
(is (=? {:name "demo"}
|
||||
(is (=? {:name "demo"
|
||||
:resource_collection_id pos-int?
|
||||
:permission_group_id pos-int?}
|
||||
(mt/user-http-request :crowberto :get 200 "apps/demo")))
|
||||
(is (str/includes?
|
||||
(str (mt/user-real-request :crowberto :get 200 "apps/demo/bundle"))
|
||||
"BUNDLE")))))))
|
||||
|
||||
(deftest list-available-apps-test
|
||||
(deftest deleting-a-data-app-removes-its-permission-group-test
|
||||
(testing "removing a data app through the admin API — clearing out one left behind after its
|
||||
repo was disconnected or a remote-sync branch switch — deletes its server-managed
|
||||
permission group and resource collection along with the row"
|
||||
(mt/with-premium-features #{:data-apps-preview}
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(create-app!)
|
||||
(let [app (t2/select-one :model/DataApp :name "demo")
|
||||
{:keys [permission_group_id resource_collection_id]}
|
||||
(data-app.resources/ensure-resources! app)]
|
||||
(is (t2/exists? :model/PermissionsGroup :id permission_group_id)
|
||||
"precondition: the app has a permission group")
|
||||
(mt/user-http-request :crowberto :delete 204 "apps/demo")
|
||||
(is (not (t2/exists? :model/DataApp :id (:id app)))
|
||||
"the app row is gone")
|
||||
(is (not (t2/exists? :model/PermissionsGroup :id permission_group_id))
|
||||
"its permission group is removed too")
|
||||
(is (not (t2/exists? :model/Collection :id resource_collection_id))
|
||||
"and so is its resource collection"))))))
|
||||
|
||||
(deftest ^:parallel query-definition-request-schema-is-closed-test
|
||||
(is (empty? (closed-schemas/findings ::query-definition/query-definition))))
|
||||
|
||||
(deftest superuser-can-resolve-a-query-definition-test
|
||||
(mt/with-premium-features #{:data-apps-preview}
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(create-app!)
|
||||
(let [response (mt/user-http-request
|
||||
:crowberto :post 200 "apps/demo/query"
|
||||
{:stages [{:source {:type "table" :id (mt/id :venues) :name "Venues"
|
||||
:fields {:price {:type "column" :name "PRICE" :jsType "number"}}
|
||||
:segments {} :measures {}
|
||||
:columns [{:name "PRICE" :jsType "number"}]}
|
||||
:orderBys [{:type "column" :name "PRICE" :tableId (mt/id :venues)
|
||||
:fieldId (mt/id :venues :price) :baseType "type/Integer"
|
||||
:effectiveType "type/Integer" :defaultTemporalBucket nil
|
||||
:direction "asc" :jsType "number" :description "Price tier"}]
|
||||
:filters [{:type "operator" :operator ">"
|
||||
:args [{:type "column" :name "PRICE" :description "Price tier"}
|
||||
{:type "literal" :value 2}]}]
|
||||
:limit 5}]})]
|
||||
(is (= (mt/id) (:database_id response)))
|
||||
(is (=? {:dataset_query {:lib/type "mbql/query"
|
||||
:database (mt/id)
|
||||
:stages [{:lib/type "mbql.stage/mbql"
|
||||
:source-table (mt/id :venues)
|
||||
:filters [[">" {} ["field" {} (mt/id :venues :price)] 2]]
|
||||
:order-by [["asc" {} ["field" {} (mt/id :venues :price)]]]
|
||||
:limit 5}]}}
|
||||
response))))))
|
||||
|
||||
(deftest query-definition-normalizes-nested-binning-test
|
||||
(mt/with-premium-features #{:data-apps-preview}
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(create-app!)
|
||||
(doseq [[field-name field-id binning]
|
||||
[["PRICE" (mt/id :venues :price) {:strategy "num-bins" :num-bins 10}]
|
||||
["LATITUDE" (mt/id :venues :latitude) {:strategy "bin-width" :bin-width 20.0}]]]
|
||||
(let [column {:type "column" :name field-name :description "A numeric field" :binning binning}
|
||||
response (mt/user-http-request
|
||||
:crowberto :post 200 "apps/demo/query"
|
||||
{:stages [{:source {:type "table" :id (mt/id :venues)}
|
||||
:aggregations [{:type "operator" :operator "count" :args []}]
|
||||
:breakouts [column]
|
||||
:orderBys [(assoc column :direction "asc")]}]})]
|
||||
(is (=? {:dataset_query
|
||||
{:stages [{:breakout [["field" {:binning binning} field-id]]
|
||||
:order-by [["asc" {} ["field" {:binning binning} field-id]]]}]}}
|
||||
response)))))))
|
||||
|
||||
(deftest query-definition-rejects-unknown-nested-binning-options-test
|
||||
(mt/with-premium-features #{:data-apps-preview}
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(create-app!)
|
||||
(let [response (mt/user-http-request
|
||||
:crowberto :post 400 "apps/demo/query"
|
||||
{:stages [{:source {:type "table" :id (mt/id :venues)}
|
||||
:breakouts [{:type "column" :name "PRICE"
|
||||
:binning {:strategy "num-bins" :num-bins 10 :unexpected true}}]}]})]
|
||||
(is (= {:errors {:stages [{:breakouts [{:binning {:unexpected ["disallowed key"]}}]}]}}
|
||||
response))))))
|
||||
|
||||
(deftest query-definition-resolves-an-aggregation-with-sdk-metadata-test
|
||||
(mt/with-premium-features #{:data-apps-preview}
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(create-app!)
|
||||
(let [response (mt/user-http-request
|
||||
:crowberto :post 200 "apps/demo/query"
|
||||
{:stages [{:source {:type "table" :id (mt/id :venues)}
|
||||
:aggregations [{:type "operator"
|
||||
:operator "count"
|
||||
:args []
|
||||
:columns [{:name "count"
|
||||
:displayName "Count"
|
||||
:jsType "number"}]}]}]})]
|
||||
(is (=? {:dataset_query {:stages [{:aggregation [["count" {}]]}]}}
|
||||
response))))))
|
||||
|
||||
(deftest query-definition-ignores-the-enabled-option-test
|
||||
(mt/with-premium-features #{:data-apps-preview}
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(create-app!)
|
||||
(let [table-id (mt/id :venues)
|
||||
response (mt/user-http-request
|
||||
:crowberto :post 200 "apps/demo/query"
|
||||
{:stages [{:source {:type "table" :id table-id}
|
||||
:enabled false}]})]
|
||||
(is (=? {:dataset_query {:stages [{:source-table table-id}]}}
|
||||
response))))))
|
||||
|
||||
(deftest query-definition-rejects-unsupported-fields-over-http-test
|
||||
(mt/with-premium-features #{:data-apps-preview}
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(create-app!)
|
||||
(doseq [[path expected-errors]
|
||||
[[[:unexpected] {:unexpected ["disallowed key"]}]
|
||||
[[:stages 0 :joins] {:stages [{:joins ["disallowed key"]}]}]
|
||||
[[:stages 0 :expressions] {:stages [{:expressions ["disallowed key"]}]}]
|
||||
[[:stages 0 :source :unexpected] {:stages [{:source {:unexpected ["disallowed key"]}}]}]
|
||||
[[:stages 0 :fields 0 :unexpected] {:stages [{:fields [{:unexpected ["disallowed key"]}]}]}]]]
|
||||
(testing (str "Reject unsupported fields at " path)
|
||||
(let [query {:stages [{:source {:type "table" :id (mt/id :venues)}
|
||||
:fields [{:type "column" :name "PRICE"}]}]}
|
||||
response (mt/user-http-request :crowberto :post 400 "apps/demo/query"
|
||||
(assoc-in query path []))]
|
||||
(is (= {:errors expected-errors} response))))))))
|
||||
|
||||
(deftest query-definition-resolves-a-metric-with-sdk-metadata-test
|
||||
(mt/with-premium-features #{:data-apps-preview}
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(create-app!)
|
||||
(let [metadata-provider (mt/metadata-provider)
|
||||
venue-count-query (-> (lib/query metadata-provider
|
||||
(lib.metadata/table metadata-provider (mt/id :venues)))
|
||||
(lib/aggregate (lib/count)))]
|
||||
(mt/with-temp [:model/Card {metric-id :id}
|
||||
{:name "Venue count"
|
||||
:type :metric
|
||||
:database_id (mt/id)
|
||||
:table_id (mt/id :venues)
|
||||
:dataset_query venue-count-query}]
|
||||
(let [response (mt/user-http-request
|
||||
:crowberto :post 200 "apps/demo/query"
|
||||
{:stages [{:source {:type "table" :id (mt/id :venues)}
|
||||
:aggregations [{:type "metric" :id metric-id :name "Venue count"
|
||||
:databaseId (mt/id) :sourceTableId (mt/id :venues)
|
||||
:sourceCardId nil :mappedTableIds [(mt/id :venues)]
|
||||
:columns {:count {:type "column" :name "count" :jsType "number"}}
|
||||
:dimensions {}}]}]})]
|
||||
(is (= [metric-id] (mapv :id (:metrics response))))))))))
|
||||
|
||||
(deftest rejects-metrics-whose-definition-reads-another-card-test
|
||||
(testing "sync copies a referenced metric but rewrites nothing inside the copy, so a metric that
|
||||
itself reads another card would publish and then fail for viewers without access to that
|
||||
card's collection -- resolving the query has to refuse it"
|
||||
(mt/with-premium-features #{:data-apps-preview}
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(create-app!)
|
||||
(let [metadata-provider (mt/metadata-provider)
|
||||
venues (lib.metadata/table metadata-provider (mt/id :venues))
|
||||
venue-count-query (-> (lib/query metadata-provider venues)
|
||||
(lib/aggregate (lib/count)))
|
||||
orders-query (lib/query metadata-provider
|
||||
(lib.metadata/table metadata-provider (mt/id :orders)))
|
||||
app-query (fn [metric-id]
|
||||
{:stages [{:source {:type "table" :id (mt/id :venues)}
|
||||
:aggregations [{:type "metric" :id metric-id}]}]})]
|
||||
(mt/with-temp [:model/Card {inner-metric-id :id}
|
||||
{:name "Venue count"
|
||||
:type :metric
|
||||
:database_id (mt/id)
|
||||
:table_id (mt/id :venues)
|
||||
:dataset_query venue-count-query}
|
||||
:model/Card {question-id :id}
|
||||
{:name "Orders"
|
||||
:type :question
|
||||
:database_id (mt/id)
|
||||
:table_id (mt/id :orders)
|
||||
:dataset_query orders-query}]
|
||||
(let [;; built after the cards exist, so the provider can resolve them
|
||||
provider (mt/metadata-provider)
|
||||
nested-query (lib/test-query provider (app-query inner-metric-id))
|
||||
question-sourced-query (lib/query provider (lib.metadata/card provider question-id))]
|
||||
(testing "the inner metric on its own resolves"
|
||||
(is (= [inner-metric-id]
|
||||
(mapv :id (:metrics (mt/user-http-request
|
||||
:crowberto :post 200 "apps/demo/query"
|
||||
(app-query inner-metric-id)))))))
|
||||
(mt/with-temp [:model/Card {nested-metric-id :id}
|
||||
{:name "Nested venue count"
|
||||
:type :metric
|
||||
:database_id (mt/id)
|
||||
:table_id (mt/id :venues)
|
||||
:dataset_query nested-query}
|
||||
:model/Card {question-metric-id :id}
|
||||
{:name "Orders question metric"
|
||||
:type :metric
|
||||
:database_id (mt/id)
|
||||
:table_id (mt/id :orders)
|
||||
:dataset_query question-sourced-query}]
|
||||
(testing "a metric that references another metric is refused"
|
||||
(is (= (str "Data app queries cannot use metrics that reference other saved questions"
|
||||
" or metrics: Nested venue count")
|
||||
(mt/user-http-request
|
||||
:crowberto :post 400 "apps/demo/query"
|
||||
(app-query nested-metric-id)))))
|
||||
(testing "so is one built on a saved question, which sync does not copy either"
|
||||
(is (= (str "Data app queries cannot use metrics that reference other saved questions"
|
||||
" or metrics: Orders question metric")
|
||||
(mt/user-http-request
|
||||
:crowberto :post 400 "apps/demo/query"
|
||||
(app-query question-metric-id)))))))))))))
|
||||
|
||||
(deftest resolved-query-includes-implicitly-joined-tables-test
|
||||
(mt/with-premium-features #{:data-apps-preview}
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(create-app!)
|
||||
(let [orders-id (mt/id :orders)
|
||||
products-id (mt/id :products)
|
||||
product-id-field-id (mt/id :orders :product_id)
|
||||
response (mt/user-http-request
|
||||
:crowberto :post 200 "apps/demo/query"
|
||||
{:stages [{:source {:type "table" :id orders-id}
|
||||
:breakouts [{:type "column"
|
||||
:name "CATEGORY"
|
||||
:source-field-id product-id-field-id}]}]})]
|
||||
(is (= #{orders-id products-id}
|
||||
(set (:table_ids response))))))))
|
||||
|
||||
(deftest saved-query-table-dependencies-include-implicitly-joined-tables-test
|
||||
(testing "sync copies models, actions and metrics whose queries never pass through /query, and the
|
||||
table an implicit join reaches is named nowhere in such a query -- only this lookup finds it"
|
||||
(mt/with-premium-features #{:data-apps-preview}
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(create-app!)
|
||||
(let [metadata-provider (mt/metadata-provider)
|
||||
orders-query (lib/query metadata-provider
|
||||
(lib.metadata/table metadata-provider (mt/id :orders)))
|
||||
category (->> (lib/breakoutable-columns orders-query)
|
||||
(filter #(= (:id %) (mt/id :products :category)))
|
||||
first)
|
||||
model-query (lib/breakout orders-query category)]
|
||||
(mt/with-temp [:model/Card {model-id :id}
|
||||
{:name "Orders by category"
|
||||
:type :model
|
||||
:database_id (mt/id)
|
||||
:table_id (mt/id :orders)
|
||||
:dataset_query model-query}]
|
||||
(let [dataset-query (t2/select-one-fn :dataset_query :model/Card :id model-id)
|
||||
response (mt/user-http-request
|
||||
:crowberto :post 200 "apps/demo/query-table-dependencies"
|
||||
{:dataset_queries [dataset-query]})]
|
||||
(is (= #{(mt/id :orders) (mt/id :products)}
|
||||
(set (:table_ids response)))))))))))
|
||||
|
||||
(deftest superuser-can-store-table-dependencies-test
|
||||
(mt/with-premium-features #{:data-apps-preview}
|
||||
(mt/with-model-cleanup [:model/DataApp]
|
||||
(create-app!)
|
||||
(let [table-ids [(mt/id :venues) (mt/id :orders)]]
|
||||
(is (= (sort table-ids)
|
||||
(:table_ids
|
||||
(mt/user-http-request :crowberto :put 200 "apps/demo/table-dependencies"
|
||||
{:table_ids (reverse table-ids)}))))
|
||||
(is (= (sort table-ids)
|
||||
(t2/select-one-fn :table_ids :model/DataApp :name "demo")))
|
||||
(is (= []
|
||||
(:table_ids
|
||||
(mt/user-http-request :crowberto :put 200 "apps/demo/table-dependencies"
|
||||
{:table_ids []}))))))))
|
||||
|
||||
(deftest non-superuser-cannot-store-table-dependencies-test
|
||||
(mt/with-premium-features #{:data-apps-preview}
|
||||
(mt/with-model-cleanup [:model/DataApp]
|
||||
(create-app!)
|
||||
(is (= "You don't have permissions to do that."
|
||||
(mt/user-http-request :rasta :put 403 "apps/demo/table-dependencies"
|
||||
{:table_ids [(mt/id :venues)]}))))))
|
||||
|
||||
(deftest table-dependencies-validates-table-ids-test
|
||||
(mt/with-premium-features #{:data-apps-preview}
|
||||
(mt/with-model-cleanup [:model/DataApp]
|
||||
(create-app!)
|
||||
(is (= "One or more tables do not exist."
|
||||
(mt/user-http-request :crowberto :put 400 "apps/demo/table-dependencies"
|
||||
{:table_ids [Integer/MAX_VALUE]})))
|
||||
(is (= [] (t2/select-one-fn :table_ids :model/DataApp :name "demo"))))))
|
||||
|
||||
(deftest user-permission-warnings-test
|
||||
(mt/with-premium-features #{:data-apps-preview :advanced-permissions :sandboxes}
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup :model/Sandbox]
|
||||
(mt/with-no-data-perms-for-all-users!
|
||||
(create-app!)
|
||||
(let [{app-group-id :permission_group_id}
|
||||
(data-app.resources/ensure-resources! (t2/select-one :model/DataApp :name "demo"))
|
||||
allowed-table-id (mt/id :venues)
|
||||
missing-table-id (mt/id :orders)
|
||||
user-id (mt/user->id :rasta)]
|
||||
(testing "an app with no synchronized dependencies has no warnings"
|
||||
(is (= []
|
||||
(mt/user-http-request :crowberto :post 200 "apps/demo/user-permission-warnings"
|
||||
{:user_ids [user-id]}))))
|
||||
(mt/user-http-request :crowberto :put 200 "apps/demo/table-dependencies"
|
||||
{:table_ids [allowed-table-id missing-table-id]})
|
||||
(perms/add-user-to-group! user-id app-group-id)
|
||||
(perms/set-table-permission! app-group-id
|
||||
missing-table-id
|
||||
:perms/view-data
|
||||
:unrestricted)
|
||||
(perms/set-table-permission! (perms/all-users-group)
|
||||
allowed-table-id
|
||||
:perms/view-data
|
||||
:unrestricted)
|
||||
(testing "returns only users who lack access from a non-data-app group"
|
||||
(is (=? [{:user_id user-id
|
||||
:missing_tables [{:id missing-table-id
|
||||
:name "Orders"
|
||||
:database_id (mt/id)}]}]
|
||||
(mt/user-http-request :crowberto :post 200 "apps/demo/user-permission-warnings"
|
||||
{:user_ids [user-id (mt/user->id :crowberto)]}))))
|
||||
(testing "unrestricted access from another group is adequate"
|
||||
(mt/with-temp [:model/PermissionsGroup {group-id :id} {}]
|
||||
(perms/add-user-to-group! user-id group-id)
|
||||
(perms/set-table-permission! group-id
|
||||
missing-table-id
|
||||
:perms/view-data
|
||||
:unrestricted)
|
||||
(is (= []
|
||||
(mt/user-http-request :crowberto :post 200 "apps/demo/user-permission-warnings"
|
||||
{:user_ids [user-id]})))))
|
||||
(testing "sandboxed access is adequate"
|
||||
(mt/with-temp [:model/PermissionsGroup {group-id :id} {}
|
||||
:model/Sandbox _ {:group_id group-id :table_id missing-table-id}]
|
||||
(perms/add-user-to-group! user-id group-id)
|
||||
(is (= []
|
||||
(mt/user-http-request :crowberto :post 200 "apps/demo/user-permission-warnings"
|
||||
{:user_ids [user-id]}))))))))))
|
||||
|
||||
(deftest permission-warning-lookups-are-batched-test
|
||||
(mt/with-premium-features #{:data-apps-preview :advanced-permissions :sandboxes}
|
||||
(mt/with-no-data-perms-for-all-users!
|
||||
(let [table-ids [(mt/id :venues) (mt/id :orders)]
|
||||
users [{:id (mt/user->id :rasta) :is_superuser false}
|
||||
{:id (mt/user->id :lucky) :is_superuser false}]
|
||||
query-count (fn [users]
|
||||
(t2/with-call-count [call-count]
|
||||
(data-app.user-access/permission-warnings table-ids users)
|
||||
(call-count)))]
|
||||
(is (= 3 (query-count users)))
|
||||
(is (= (query-count (take 1 users))
|
||||
(query-count users))
|
||||
"permission warning query count must not grow with the number of users")))))
|
||||
|
||||
(deftest data-app-list-warning-lookups-are-batched-test
|
||||
(mt/with-premium-features #{:data-apps-preview :advanced-permissions :sandboxes}
|
||||
(mt/with-no-data-perms-for-all-users!
|
||||
(mt/with-temp [:model/PermissionsGroup {first-group-id :id} {}
|
||||
:model/PermissionsGroup {second-group-id :id} {}]
|
||||
(perms/add-user-to-group! (mt/user->id :rasta) first-group-id)
|
||||
(perms/add-user-to-group! (mt/user->id :lucky) second-group-id)
|
||||
(let [apps [{:permission_group_id first-group-id
|
||||
:table_ids [(mt/id :venues)]}
|
||||
{:permission_group_id second-group-id
|
||||
:table_ids [(mt/id :orders)]}]
|
||||
warning-groups #(t2/with-call-count [call-count]
|
||||
(let [result (data-app.user-access/groups-with-permission-warnings %)]
|
||||
{:result result :query-count (call-count)}))
|
||||
one-app (warning-groups (take 1 apps))
|
||||
two-apps (warning-groups apps)]
|
||||
(is (= #{first-group-id} (:result one-app)))
|
||||
(is (= #{first-group-id second-group-id} (:result two-apps)))
|
||||
(is (= 3 (:query-count one-app) (:query-count two-apps))
|
||||
"warning status query count must not grow with the number of apps"))))))
|
||||
|
||||
(deftest data-app-list-includes-user-permission-warning-status-test
|
||||
(mt/with-premium-features #{:data-apps-preview :advanced-permissions :sandboxes}
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(mt/with-no-data-perms-for-all-users!
|
||||
(create-app!)
|
||||
(let [{app-group-id :permission_group_id}
|
||||
(data-app.resources/ensure-resources! (t2/select-one :model/DataApp :name "demo"))
|
||||
table-id (mt/id :orders)
|
||||
user-id (mt/user->id :rasta)
|
||||
warning? #(->> (mt/user-http-request :crowberto :get 200 "apps")
|
||||
(filter (comp #{"demo"} :name))
|
||||
first
|
||||
:has_user_permission_warnings)]
|
||||
(mt/user-http-request :crowberto :put 200 "apps/demo/table-dependencies"
|
||||
{:table_ids [table-id]})
|
||||
(perms/add-user-to-group! user-id app-group-id)
|
||||
(is (true? (warning?)))
|
||||
(perms/set-table-permission! (perms/all-users-group)
|
||||
table-id
|
||||
:perms/view-data
|
||||
:unrestricted)
|
||||
(is (false? (warning?))))))))
|
||||
|
||||
(deftest data-app-list-warning-status-ignores-deactivated-members-test
|
||||
(mt/with-premium-features #{:data-apps-preview :advanced-permissions :sandboxes}
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(mt/with-no-data-perms-for-all-users!
|
||||
(create-app!)
|
||||
(let [{app-group-id :permission_group_id}
|
||||
(data-app.resources/ensure-resources! (t2/select-one :model/DataApp :name "demo"))
|
||||
table-id (mt/id :orders)]
|
||||
(mt/with-temp [:model/User {user-id :id} {:email "deactivated-data-app-user@example.com"}]
|
||||
(perms/add-user-to-group! user-id app-group-id)
|
||||
(t2/update! :model/User :id user-id {:is_active false})
|
||||
(mt/user-http-request :crowberto :put 200 "apps/demo/table-dependencies"
|
||||
{:table_ids [table-id]})
|
||||
(is (false? (->> (mt/user-http-request :crowberto :get 200 "apps")
|
||||
(filter (comp #{"demo"} :name))
|
||||
first
|
||||
:has_user_permission_warnings)))))))))
|
||||
|
||||
(deftest deactivated-users-cannot-be-added-to-data-apps-test
|
||||
(mt/with-premium-features #{:data-apps-preview}
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(create-app!)
|
||||
(let [{app-group-id :permission_group_id}
|
||||
(data-app.resources/ensure-resources! (t2/select-one :model/DataApp :name "demo"))]
|
||||
(mt/with-temp [:model/User {user-id :id} {:email "deactivated-data-app-user@example.com"
|
||||
:is_active false}]
|
||||
(is (= "Deactivated users cannot be added to data apps."
|
||||
(mt/user-http-request :crowberto :post 400 "permissions/membership"
|
||||
{:group_id app-group-id :user_id user-id})))
|
||||
(is (= "Deactivated users cannot be added to data apps."
|
||||
(mt/user-http-request :crowberto :post 400 "apps/demo/user-permission-warnings"
|
||||
{:user_ids [user-id]}))))))))
|
||||
|
||||
(deftest user-permission-warnings-validates-users-test
|
||||
(mt/with-premium-features #{:data-apps-preview :tenants}
|
||||
(mt/with-model-cleanup [:model/DataApp]
|
||||
(create-app!)
|
||||
(testing "unknown users"
|
||||
(mt/user-http-request :crowberto :post 404 "apps/demo/user-permission-warnings"
|
||||
{:user_ids [Integer/MAX_VALUE]}))
|
||||
(testing "tenant users"
|
||||
(mt/with-temp [:model/Tenant {tenant-id :id} {:name "Data app tenant" :slug "data-app-tenant"}
|
||||
:model/User {user-id :id} {:email "data-app-tenant-user@example.com"
|
||||
:tenant_id tenant-id}]
|
||||
(is (= "Tenant users cannot be added to data apps."
|
||||
(mt/user-http-request :crowberto :post 400 "apps/demo/user-permission-warnings"
|
||||
{:user_ids [user-id]}))))))))
|
||||
|
||||
(deftest non-superuser-cannot-read-user-permission-warnings-test
|
||||
(mt/with-premium-features #{:data-apps-preview}
|
||||
(mt/with-model-cleanup [:model/DataApp]
|
||||
(create-app!)
|
||||
(is (= "You don't have permissions to do that."
|
||||
(mt/user-http-request :rasta :post 403 "apps/demo/user-permission-warnings"
|
||||
{:user_ids [(mt/user->id :rasta)]}))))))
|
||||
|
||||
(deftest data-app-write-endpoints-require-feature-token-test
|
||||
(mt/with-premium-features #{}
|
||||
(mt/user-http-request :crowberto :put 402 "apps/demo/table-dependencies"
|
||||
{:table_ids []})
|
||||
(mt/user-http-request :crowberto :post 402 "apps/demo/user-permission-warnings"
|
||||
{:user_ids [(mt/user->id :rasta)]})
|
||||
(mt/user-http-request :crowberto :post 402 "apps/demo/draft")))
|
||||
|
||||
(deftest data-app-membership-additions-require-feature-token-test
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(create-app!)
|
||||
(let [{group-id :permission_group_id}
|
||||
(data-app.resources/ensure-resources! (t2/select-one :model/DataApp :name "demo"))]
|
||||
(mt/with-premium-features #{}
|
||||
(mt/user-http-request :crowberto :post 402 "permissions/membership"
|
||||
{:group_id group-id :user_id (mt/user->id :lucky)})
|
||||
(is (not (t2/exists? :model/PermissionsGroupMembership :group_id group-id)))))))
|
||||
|
||||
(deftest data-app-membership-removal-without-feature-token-test
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(create-app!)
|
||||
(let [{group-id :permission_group_id}
|
||||
(data-app.resources/ensure-resources! (t2/select-one :model/DataApp :name "demo"))
|
||||
user-id (mt/user->id :rasta)]
|
||||
(perms/add-user-to-group! user-id group-id)
|
||||
(let [membership-id (t2/select-one-pk :model/PermissionsGroupMembership
|
||||
:group_id group-id :user_id user-id)
|
||||
endpoint (format "permissions/membership/%d" membership-id)]
|
||||
(mt/with-premium-features #{}
|
||||
(mt/user-http-request :rasta :delete 403 endpoint)
|
||||
(is (t2/exists? :model/PermissionsGroupMembership :id membership-id))
|
||||
;; memberships and collection grants still remain after the token expires,
|
||||
;; so admins must still be able to remove a member without the feature token.
|
||||
(mt/user-http-request :crowberto :delete 204 endpoint)
|
||||
(is (not (t2/exists? :model/PermissionsGroupMembership :id membership-id))))))))
|
||||
|
||||
(deftest data-app-membership-clearing-without-feature-token-test
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(create-app!)
|
||||
(let [{group-id :permission_group_id}
|
||||
(data-app.resources/ensure-resources! (t2/select-one :model/DataApp :name "demo"))
|
||||
endpoint (format "permissions/membership/%d/clear" group-id)]
|
||||
(perms/add-user-to-group! (mt/user->id :rasta) group-id)
|
||||
(perms/add-user-to-group! (mt/user->id :lucky) group-id)
|
||||
(mt/with-premium-features #{}
|
||||
(mt/user-http-request :rasta :put 403 endpoint)
|
||||
(is (= 2 (t2/count :model/PermissionsGroupMembership :group_id group-id)))
|
||||
;; memberships and collection grants still remain after the token expires,
|
||||
;; so admins must still be able to remove all members without the feature token.
|
||||
(mt/user-http-request :crowberto :put 204 endpoint)
|
||||
(is (not (t2/exists? :model/PermissionsGroupMembership :group_id group-id)))))))
|
||||
|
||||
(deftest query-definition-must-use-a-table-source-test
|
||||
(mt/with-premium-features #{:data-apps-preview}
|
||||
(mt/with-model-cleanup [:model/DataApp]
|
||||
(create-app!)
|
||||
(is (some? (mt/user-http-request :crowberto :post 400 "apps/demo/query"
|
||||
{:stages [{:source {:type "card" :id 1}}]}))))))
|
||||
|
||||
(deftest query-definition-source-must-be-valid-test
|
||||
(mt/with-premium-features #{:data-apps-preview}
|
||||
(mt/with-model-cleanup [:model/DataApp]
|
||||
(create-app!)
|
||||
(is (some? (mt/user-http-request :crowberto :post 400 "apps/demo/query"
|
||||
{:stages [{:source {:type 1 :id (mt/id :venues)}}]}))))))
|
||||
|
||||
(deftest non-superuser-cannot-resolve-a-query-definition-test
|
||||
(mt/with-premium-features #{:data-apps-preview}
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(create-app!)
|
||||
(is (= "You don't have permissions to do that."
|
||||
(mt/user-http-request :rasta :post 403 "apps/demo/query"
|
||||
{:stages [{:source {:type "table" :id (mt/id :venues)}}]}))))))
|
||||
|
||||
(deftest superuser-can-create-or-reuse-a-data-app-draft-test
|
||||
(mt/with-premium-features #{:data-apps-preview}
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(let [first-response (mt/user-http-request :crowberto :post 200 "apps/draft-app/draft")
|
||||
second-response (mt/user-http-request :crowberto :post 200 "apps/draft-app/draft")]
|
||||
(is (=? {:name "draft-app"
|
||||
:resource_collection_id pos-int?
|
||||
:permission_group_id pos-int?}
|
||||
first-response))
|
||||
(is (= (select-keys first-response [:resource_collection_id :permission_group_id])
|
||||
(select-keys second-response [:resource_collection_id :permission_group_id])))
|
||||
(is (=? {:bundle nil :draft true}
|
||||
(t2/select-one :model/DataApp :name "draft-app")))))))
|
||||
|
||||
(deftest non-superuser-cannot-create-a-data-app-draft-test
|
||||
(mt/with-premium-features #{:data-apps-preview}
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(is (= "You don't have permissions to do that."
|
||||
(mt/user-http-request :rasta :post 403 "apps/draft-app/draft")))
|
||||
(is (not (t2/exists? :model/DataApp :name "draft-app"))))))
|
||||
|
||||
(deftest data-app-draft-must-have-a-valid-slug-test
|
||||
(mt/with-premium-features #{:data-apps-preview}
|
||||
(mt/with-model-cleanup [:model/DataApp]
|
||||
(is (= "Data app draft slugs must use lowercase letters, numbers, and dashes."
|
||||
(mt/user-http-request :crowberto :post 400 "apps/Draft/draft")))
|
||||
(is (not (t2/exists? :model/DataApp :name "Draft"))))))
|
||||
|
||||
(deftest data-app-group-reaches-only-copied-actions-test
|
||||
(testing "an action is reachable exactly when its model lives in the data app collection"
|
||||
(mt/with-premium-features #{:data-apps-preview}
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(mt/with-non-admin-groups-no-root-collection-perms
|
||||
;; Its own slug: the `Data App: <slug>` group outlives other tests in
|
||||
;; this namespace, so sharing "demo" collides on the group name.
|
||||
(let [{:keys [permission_group_id resource_collection_id]}
|
||||
(data-app.sync/ensure-draft! "action-perms-app")
|
||||
metadata-provider (mt/metadata-provider)
|
||||
venues (lib.metadata/table metadata-provider (mt/id :venues))]
|
||||
(perms/add-user-to-group! (mt/user->id :rasta) permission_group_id)
|
||||
(mt/with-temp
|
||||
[:model/Collection {source-collection-id :id} {}
|
||||
:model/Card {source-model-id :id} {:type :model
|
||||
:collection_id source-collection-id
|
||||
:dataset_query (lib/query metadata-provider venues)}
|
||||
;; What `npm run sync-resources` produces: a copy of the model in the
|
||||
;; app's own collection, carrying its own copy of the action.
|
||||
:model/Card {copied-model-id :id} {:type :model
|
||||
:collection_id resource_collection_id
|
||||
:dataset_query (lib/query metadata-provider venues)}]
|
||||
(let [action {:name "Create Venue", :type :implicit, :kind :row/create}
|
||||
source-id (actions/insert! (assoc action :model_id source-model-id))
|
||||
copied-id (actions/insert! (assoc action :model_id copied-model-id))]
|
||||
(testing "the app's group reads the action copied into its collection"
|
||||
(is (=? {:id copied-id}
|
||||
(mt/user-http-request :rasta :get 200 (str "action/" copied-id)))))
|
||||
(testing "the same group cannot read the source action it was copied from"
|
||||
(is (= "You don't have permissions to do that."
|
||||
(mt/user-http-request :rasta :get 403 (str "action/" source-id)))))
|
||||
(testing "copying does not widen access to the source model"
|
||||
(is (= "You don't have permissions to do that."
|
||||
(mt/user-http-request :rasta :get 403 (str "card/" source-model-id)))))
|
||||
(testing "a superuser still reads both"
|
||||
(is (=? {:id source-id}
|
||||
(mt/user-http-request :crowberto :get 200 (str "action/" source-id)))))))))))))
|
||||
|
||||
(deftest list-available-apps-test
|
||||
(mt/with-premium-features #{:data-apps-preview}
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(t2/insert! :model/DataApp :name "ready" :display_name "Ready" :bundle_path "data_apps/ready/index.js")
|
||||
(t2/insert! :model/DataApp :name "disabled" :display_name "Disabled" :bundle_path "data_apps/disabled/index.js"
|
||||
:enabled false)
|
||||
@@ -97,7 +727,7 @@
|
||||
|
||||
(deftest bundle-includes-allowed-hosts-header-test
|
||||
(mt/with-premium-features #{:data-apps-preview}
|
||||
(mt/with-model-cleanup [:model/DataApp]
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(t2/insert! :model/DataApp
|
||||
:name "demo"
|
||||
:display_name "Demo"
|
||||
@@ -105,6 +735,8 @@
|
||||
:bundle (.getBytes "BUNDLE" "UTF-8")
|
||||
:bundle_hash "abc123"
|
||||
:allowed_hosts ["https://api.example.com"])
|
||||
;; publish the app so its bundle is served (crowberto, as a superuser, can read the collection)
|
||||
(data-app.resources/ensure-resources! (t2/select-one :model/DataApp :name "demo"))
|
||||
(testing "the bundle response carries the app's allowed_hosts as a JSON header"
|
||||
(let [resp (mt/user-http-request-full-response :crowberto :get 200 "apps/demo/bundle")]
|
||||
(is (= "[\"https://api.example.com\"]"
|
||||
@@ -117,7 +749,7 @@
|
||||
|
||||
(deftest list-includes-allowed-hosts-test
|
||||
(mt/with-premium-features #{:data-apps-preview}
|
||||
(mt/with-model-cleanup [:model/DataApp]
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(t2/insert! :model/DataApp
|
||||
:name "withhosts" :display_name "With"
|
||||
:bundle_path "data_apps/withhosts/index.js"
|
||||
@@ -136,7 +768,7 @@
|
||||
;;; ----------------------------------------------------- Sync -----------------------------------------------------
|
||||
|
||||
(deftest import-materializes-apps-test
|
||||
(mt/with-model-cleanup [:model/DataApp]
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(let [result (data-app.sync/import-from-snapshot!
|
||||
(snapshot (merge (app-files "sales" {:name "Sales" :path "dist/index.js" :bundle "SALES-BUNDLE"})
|
||||
(app-files "ops" {:name "Ops" :path "dist/app.js" :bundle "OPS-BUNDLE"}))))]
|
||||
@@ -151,7 +783,7 @@
|
||||
(is (nil? (:sync_error sales)))))))
|
||||
|
||||
(deftest import-stores-allowed-hosts-test
|
||||
(mt/with-model-cleanup [:model/DataApp]
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(testing "allowed_hosts from data_app.yaml are persisted on the row"
|
||||
(data-app.sync/import-from-snapshot!
|
||||
(snapshot (app-files "sales" {:name "Sales" :path "dist/index.js" :bundle "B"
|
||||
@@ -164,7 +796,7 @@
|
||||
(is (= [] (:allowed_hosts (t2/select-one :model/DataApp :name "sales")))))))
|
||||
|
||||
(deftest import-prunes-apps-absent-from-snapshot-test
|
||||
(mt/with-model-cleanup [:model/DataApp]
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(data-app.sync/import-from-snapshot!
|
||||
(snapshot (merge (app-files "keep" {:name "Keep" :path "index.js" :bundle "KEEP"})
|
||||
(app-files "gone" {:name "Gone" :path "index.js" :bundle "GONE"}))))
|
||||
@@ -181,20 +813,31 @@
|
||||
(deftest delete-endpoint-test
|
||||
(mt/test-helpers-set-global-values!
|
||||
(mt/with-premium-features #{:data-apps-preview}
|
||||
(mt/with-model-cleanup [:model/DataApp]
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(create-app!)
|
||||
(testing "a non-superuser cannot remove an app"
|
||||
(is (= "You don't have permissions to do that."
|
||||
(mt/user-http-request :rasta :delete 403 "apps/demo")))
|
||||
(is (t2/exists? :model/DataApp :name "demo")))
|
||||
(testing "a superuser removes the app"
|
||||
(is (nil? (mt/user-http-request :crowberto :delete 204 "apps/demo")))
|
||||
(is (not (t2/exists? :model/DataApp :name "demo"))))
|
||||
(testing "removing a non-existent app 404s"
|
||||
(mt/user-http-request :crowberto :delete 404 "apps/missing"))))))
|
||||
(let [{:keys [resource_collection_id permission_group_id]}
|
||||
(data-app.resources/ensure-resources! (t2/select-one :model/DataApp :name "demo"))]
|
||||
(mt/with-temp [:model/Card {card-id :id} {:collection_id resource_collection_id}
|
||||
:model/PermissionsGroupMembership _ {:user_id (mt/user->id :rasta)
|
||||
:group_id permission_group_id}]
|
||||
(testing "a non-superuser cannot remove an app"
|
||||
(is (= "You don't have permissions to do that."
|
||||
(mt/user-http-request :rasta :delete 403 "apps/demo")))
|
||||
(is (t2/exists? :model/DataApp :name "demo")))
|
||||
;; each data app owns a permission group and a collection
|
||||
;; containing saved questions and models
|
||||
(testing "a superuser removes the app, its resources, and their contents"
|
||||
(is (nil? (mt/user-http-request :crowberto :delete 204 "apps/demo")))
|
||||
(is (not (t2/exists? :model/DataApp :name "demo")))
|
||||
(is (not (t2/exists? :model/Collection :id resource_collection_id)))
|
||||
(is (not (t2/exists? :model/Card :id card-id)))
|
||||
(is (not (t2/exists? :model/PermissionsGroup :id permission_group_id)))
|
||||
(is (not (t2/exists? :model/PermissionsGroupMembership :group_id permission_group_id))))
|
||||
(testing "removing a non-existent app 404s"
|
||||
(mt/user-http-request :crowberto :delete 404 "apps/missing"))))))))
|
||||
|
||||
(deftest import-preserves-enabled-across-syncs-test
|
||||
(mt/with-model-cleanup [:model/DataApp]
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(data-app.sync/import-from-snapshot!
|
||||
(snapshot (app-files "a" {:name "A" :path "index.js" :bundle "V1"})))
|
||||
(t2/update! :model/DataApp :name "a" {:enabled false})
|
||||
@@ -207,7 +850,7 @@
|
||||
|
||||
(deftest import-per-app-error-test
|
||||
(testing "a missing bundle file fails just that app, not the whole import"
|
||||
(mt/with-model-cleanup [:model/DataApp]
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(data-app.sync/import-from-snapshot!
|
||||
(snapshot (merge (app-files "good" {:name "Good" :path "index.js" :bundle "GOOD"})
|
||||
;; "bad" declares a path that doesn't exist
|
||||
@@ -222,7 +865,7 @@
|
||||
|
||||
(deftest import-serves-each-app-from-its-directory-test
|
||||
(testing "the directory an app lives in is the slug it's served at — two apps can't collide on one"
|
||||
(mt/with-model-cleanup [:model/DataApp]
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(data-app.sync/import-from-snapshot!
|
||||
(snapshot (merge (app-files "one" {:name "One" :path "a.js" :bundle "A"})
|
||||
(app-files "two" {:name "Two" :path "b.js" :bundle "B"}))))
|
||||
@@ -230,7 +873,7 @@
|
||||
|
||||
(deftest import-isolates-bad-config-test
|
||||
(testing "a malformed data_app.yaml is isolated: sibling apps in the same repo still sync and are not pruned, the bad one is reported"
|
||||
(mt/with-model-cleanup [:model/DataApp]
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(data-app.sync/import-from-snapshot!
|
||||
(snapshot (app-files "existing" {:name "Existing" :path "i.js" :bundle "E"})))
|
||||
;; The repo still holds "existing" and adds "good", alongside a broken "bad".
|
||||
@@ -247,13 +890,13 @@
|
||||
(deftest sync-from-snapshot!-never-throws-test
|
||||
(mt/with-premium-features #{:data-apps-preview}
|
||||
(testing "a malformed data_app.yaml is isolated into :config-errors; the app just doesn't appear, the sync doesn't throw"
|
||||
(mt/with-model-cleanup [:model/DataApp]
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(let [result (data-app.sync/sync-from-snapshot!
|
||||
(snapshot {"data_apps/x/data_app.yaml" "name: [unterminated"}))]
|
||||
(is (seq (:config-errors result)))
|
||||
(is (empty? (t2/select-fn-set :name :model/DataApp))))))
|
||||
(testing "a clean sync materializes the app with no config errors"
|
||||
(mt/with-model-cleanup [:model/DataApp]
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(let [result (data-app.sync/sync-from-snapshot!
|
||||
(snapshot (app-files "a" {:name "A" :path "index.js" :bundle "A"})))]
|
||||
(is (empty? (:config-errors result)))
|
||||
@@ -264,7 +907,7 @@
|
||||
(deftest list-and-bundle-endpoints-test
|
||||
(mt/test-helpers-set-global-values!
|
||||
(mt/with-premium-features #{:data-apps-preview}
|
||||
(mt/with-model-cleanup [:model/DataApp]
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(data-app.sync/import-from-snapshot!
|
||||
(snapshot (app-files "demo" {:name "Demo app" :path "dist/index.js" :bundle "DEMOBUNDLE"})))
|
||||
(testing "GET / lists the synced apps"
|
||||
@@ -290,7 +933,7 @@
|
||||
(deftest enable-disable-endpoint-test
|
||||
(mt/test-helpers-set-global-values!
|
||||
(mt/with-premium-features #{:data-apps-preview}
|
||||
(mt/with-model-cleanup [:model/DataApp]
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(data-app.sync/import-from-snapshot!
|
||||
(snapshot (app-files "demo" {:name "Demo" :path "index.js" :bundle "BUNDLE"})))
|
||||
(testing "PUT /:slug can disable an app"
|
||||
@@ -333,7 +976,7 @@
|
||||
;; routed as a data app named "sandbox-host" and 404.
|
||||
(testing "the route is not shadowed by the /:slug route"
|
||||
(mt/with-premium-features #{:data-apps-preview}
|
||||
(mt/with-model-cleanup [:model/DataApp]
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(create-app!)
|
||||
(is (= 200 (:status (mt/user-http-request-full-response
|
||||
:crowberto :get 200 "apps/sandbox-host"))))))))
|
||||
|
||||
@@ -101,8 +101,9 @@ path: ./dist/index.js"))))
|
||||
(is (thrown-with-msg? clojure.lang.ExceptionInfo #"must not contain"
|
||||
(parse "name: X\npath: dist/../../escape.js"))))
|
||||
(testing "a directory whose name collides with an API sub-route is rejected"
|
||||
(is (thrown-with-msg? clojure.lang.ExceptionInfo #"reserved slug"
|
||||
(parse "name: X\npath: dist/index.js" "data_apps/repo-status")))))
|
||||
(doseq [slug ["repo-status" "sandbox-host"]]
|
||||
(is (thrown-with-msg? clojure.lang.ExceptionInfo #"reserved slug"
|
||||
(parse "name: X\npath: dist/index.js" (str "data_apps/" slug)))))))
|
||||
|
||||
(deftest parse-errors-carry-400-test
|
||||
(try
|
||||
|
||||
@@ -0,0 +1,176 @@
|
||||
(ns metabase-enterprise.data-apps.permissions-test
|
||||
(:require
|
||||
[clojure.test :refer :all]
|
||||
[metabase-enterprise.advanced-permissions.models.permissions.block-permissions :as block-perms]
|
||||
[metabase-enterprise.data-apps.permissions :as data-app.permissions]
|
||||
[metabase-enterprise.data-apps.resources :as data-app.resources]
|
||||
[metabase.lib.core :as lib]
|
||||
[metabase.lib.metadata :as lib.metadata]
|
||||
[metabase.permissions-rest.data-permissions.graph :as data-perms.graph]
|
||||
[metabase.permissions.core :as perms]
|
||||
[metabase.test :as mt]
|
||||
[toucan2.core :as t2]))
|
||||
|
||||
(defn- app-group!
|
||||
[name]
|
||||
(:permission_group_id
|
||||
(data-app.resources/ensure-resources!
|
||||
(t2/insert-returning-instance! :model/DataApp
|
||||
{:name name :display_name name :bundle_path (str "data_apps/" name "/index.js")}))))
|
||||
|
||||
(defn- view-data-rows
|
||||
[group-id db-id]
|
||||
(t2/select [:model/DataPermissions :id :table_id :perm_value]
|
||||
:group_id group-id :db_id db-id :perm_type :perms/view-data))
|
||||
|
||||
(defn- database-permission-value
|
||||
[group-id db-id perm-type]
|
||||
(t2/select-one-fn :perm_value :model/DataPermissions
|
||||
:group_id group-id :db_id db-id :perm_type perm-type :table_id nil))
|
||||
|
||||
(deftest denies-data-permissions-for-new-databases-to-existing-data-app-group-test
|
||||
(mt/with-premium-features #{:advanced-permissions :data-apps}
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup :model/Database]
|
||||
(let [group-id (app-group! "finches")]
|
||||
;; data app group exists before any database permissions have been assigned
|
||||
(t2/delete! :model/DataPermissions :group_id group-id)
|
||||
;; should not grant data permissions for new databases to the data app group
|
||||
(let [new-database-id (t2/insert-returning-pk! :model/Database (mt/with-temp-defaults :model/Database))]
|
||||
(is (= {:perms/view-data :blocked
|
||||
:perms/create-queries :no
|
||||
:perms/download-results :no
|
||||
:perms/manage-table-metadata :no
|
||||
:perms/manage-database :no
|
||||
:perms/transforms :no}
|
||||
(t2/select-fn->fn :perm_type :perm_value :model/DataPermissions
|
||||
:group_id group-id :db_id new-database-id :table_id nil))))))))
|
||||
|
||||
(deftest reconcile-resets-create-queries-left-by-old-database-defaults-test
|
||||
(testing "Reconciliation removes query-authoring grants left by the old database defaults."
|
||||
(mt/with-premium-features #{:advanced-permissions :data-apps}
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(mt/with-restored-data-perms-for-group! (:id (perms/all-users-group))
|
||||
(let [db-id (mt/id)
|
||||
group-id (app-group! "finches")]
|
||||
;; Written as a row, the way the database defaults insert it -- the model rejects updates,
|
||||
;; so they delete and re-insert too. `set-database-permission!` is no use here: it cascades
|
||||
;; view-data to :unrestricted for any create-queries value other than :no, so no permission
|
||||
;; API call can produce this pairing.
|
||||
(t2/delete! :model/DataPermissions
|
||||
:group_id group-id :db_id db-id :perm_type :perms/create-queries)
|
||||
(t2/insert! :model/DataPermissions
|
||||
{:group_id group-id
|
||||
:db_id db-id
|
||||
:perm_type :perms/create-queries
|
||||
:perm_value :query-builder-and-native})
|
||||
(is (= :query-builder-and-native
|
||||
(database-permission-value group-id db-id :perms/create-queries))
|
||||
"precondition: the permissive grant is in place")
|
||||
(is (=? [{:table_id nil :perm_value :blocked}] (view-data-rows group-id db-id))
|
||||
"precondition: view-data is already blocked, so nothing cascades")
|
||||
(data-app.permissions/reconcile-data-app-permissions! #{db-id})
|
||||
(is (= :no (database-permission-value group-id db-id :perms/create-queries)))
|
||||
(is (=? [{:table_id nil :perm_value :blocked}] (view-data-rows group-id db-id)))))))))
|
||||
|
||||
(deftest membership-preserves-legacy-block-permission-check-test
|
||||
(mt/with-premium-features #{:advanced-permissions :data-apps}
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(mt/with-restored-data-perms-for-group! (:id (perms/all-users-group))
|
||||
(let [db-id (mt/id)
|
||||
provider (mt/metadata-provider)
|
||||
query (lib/query provider (lib.metadata/table provider (mt/id :venues)))
|
||||
check-block-permissions #(mt/with-test-user :rasta (block-perms/check-block-permissions query))]
|
||||
(perms/set-database-permission! (perms/all-users-group) db-id :perms/create-queries :no)
|
||||
(perms/set-database-permission! (perms/all-users-group) db-id :perms/view-data :legacy-no-self-service)
|
||||
(perms/set-table-permission! (perms/all-users-group) (mt/id :categories) :perms/create-queries :query-builder)
|
||||
(perms/set-database-permission! (perms/all-users-group) db-id :perms/download-results :ten-thousand-rows)
|
||||
;; user should not be blocked from viewing the query
|
||||
(is (true? (check-block-permissions)))
|
||||
(let [group-id (app-group! "wrens")]
|
||||
(perms/add-user-to-group! (mt/user->id :rasta) group-id)
|
||||
(is (true? (check-block-permissions)))
|
||||
(is (=? [{:table_id nil :perm_value :legacy-no-self-service}] (view-data-rows group-id db-id)))
|
||||
(testing "other groups still supply query creation and download permissions"
|
||||
(is (= :query-builder
|
||||
(perms/table-permission-for-user (mt/user->id :rasta) :perms/create-queries
|
||||
db-id (mt/id :categories))))
|
||||
(is (= :ten-thousand-rows
|
||||
(perms/table-permission-for-user (mt/user->id :rasta) :perms/download-results
|
||||
db-id (mt/id :venues)))))))))))
|
||||
|
||||
(deftest resource-setup-reconciles-only-its-app-group-test
|
||||
(mt/with-premium-features #{:advanced-permissions :data-apps}
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(let [db-id (mt/id)
|
||||
group-id (app-group! "wrens")
|
||||
other-group-id (app-group! "finches")
|
||||
app (t2/select-one :model/DataApp :permission_group_id group-id)]
|
||||
(doseq [id [group-id other-group-id]]
|
||||
(perms/set-database-permission! id db-id :perms/manage-database :yes))
|
||||
(data-app.resources/ensure-resources! app)
|
||||
(is (= :no (database-permission-value group-id db-id :perms/manage-database)))
|
||||
(is (= :yes (database-permission-value other-group-id db-id :perms/manage-database)))))))
|
||||
|
||||
(deftest permission-edits-reconcile-all-app-groups-test
|
||||
(mt/with-premium-features #{:advanced-permissions :data-apps}
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(mt/with-temp [:model/Database {db-id :id} {}
|
||||
:model/Table {table-id :id} {:db_id db-id :schema "PUBLIC"}
|
||||
:model/PermissionsGroup {ordinary-group-id :id} {}]
|
||||
(let [group-ids [(app-group! "wrens") (app-group! "finches")]
|
||||
set-view-data! #(data-perms.graph/update-data-perms-graph!
|
||||
{:groups {ordinary-group-id {db-id {:view-data % :create-queries :no}}}})]
|
||||
(set-view-data! :legacy-no-self-service)
|
||||
(doseq [group-id group-ids]
|
||||
(is (=? [{:table_id nil :perm_value :legacy-no-self-service}] (view-data-rows group-id db-id))))
|
||||
(testing "new tables preserve the database-wide legacy permission"
|
||||
(mt/with-temp [:model/Table {new-table-id :id} {:db_id db-id :schema "PUBLIC"}]
|
||||
(doseq [group-id group-ids]
|
||||
(is (=? [{:table_id nil :perm_value :legacy-no-self-service}] (view-data-rows group-id db-id)))
|
||||
(is (= :legacy-no-self-service
|
||||
(perms/table-permission-for-groups #{group-id} :perms/view-data db-id new-table-id))))))
|
||||
(testing "reconciliation removes manual management grants"
|
||||
(doseq [group-id group-ids
|
||||
perm-type [:perms/manage-table-metadata :perms/manage-database]]
|
||||
(perms/set-database-permission! group-id db-id perm-type :yes))
|
||||
(set-view-data! :legacy-no-self-service)
|
||||
(doseq [group-id group-ids]
|
||||
(is (= #{[:perms/manage-table-metadata :no] [:perms/manage-database :no]}
|
||||
(t2/select-fn-set (juxt :perm_type :perm_value) :model/DataPermissions
|
||||
:group_id group-id :db_id db-id
|
||||
:perm_type [:in [:perms/manage-table-metadata :perms/manage-database]])))))
|
||||
(testing "an unchanged permission edit preserves row IDs"
|
||||
(let [before (mapv #(view-data-rows % db-id) group-ids)]
|
||||
(set-view-data! :legacy-no-self-service)
|
||||
(is (= before (mapv #(view-data-rows % db-id) group-ids)))))
|
||||
(testing "copied legacy does not keep itself alive after the ordinary grant is removed"
|
||||
(set-view-data! :blocked)
|
||||
(doseq [group-id group-ids]
|
||||
(is (=? [{:table_id nil :perm_value :blocked}] (view-data-rows group-id db-id)))))
|
||||
(testing "a table-level legacy grant also updates the whole database"
|
||||
(set-view-data! {"PUBLIC" {table-id :legacy-no-self-service}})
|
||||
(doseq [group-id group-ids]
|
||||
(is (=? [{:table_id nil :perm_value :legacy-no-self-service}] (view-data-rows group-id db-id))))))))))
|
||||
|
||||
(deftest unrelated-legacy-permissions-do-not-unblock-app-members-test
|
||||
(mt/with-premium-features #{:advanced-permissions :data-apps}
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(mt/with-no-data-perms-for-all-users!
|
||||
(mt/with-temp [:model/PermissionsGroup {ordinary-group-id :id} {}]
|
||||
(perms/set-database-permission! ordinary-group-id (mt/id) :perms/view-data :legacy-no-self-service)
|
||||
(let [group-id (app-group! "finches")
|
||||
provider (mt/metadata-provider)
|
||||
query (lib/query provider (lib.metadata/table provider (mt/id :venues)))]
|
||||
(perms/add-user-to-group! (mt/user->id :rasta) group-id)
|
||||
(is (=? [{:table_id nil :perm_value :legacy-no-self-service}] (view-data-rows group-id (mt/id))))
|
||||
(is (= :blocked (perms/table-permission-for-user (mt/user->id :rasta)
|
||||
:perms/view-data (mt/id) (mt/id :venues))))
|
||||
(is (thrown-with-msg? clojure.lang.ExceptionInfo #"You do not have permissions to run this query"
|
||||
(mt/with-test-user :rasta (block-perms/check-block-permissions query))))
|
||||
(testing "the app group grants no other data capabilities"
|
||||
(is (= #{[:perms/create-queries :no] [:perms/download-results :no]
|
||||
[:perms/manage-table-metadata :no] [:perms/manage-database :no] [:perms/transforms :no]}
|
||||
(t2/select-fn-set (juxt :perm_type :perm_value) :model/DataPermissions
|
||||
:group_id group-id :db_id (mt/id)
|
||||
:perm_type [:in [:perms/create-queries :perms/download-results
|
||||
:perms/manage-table-metadata :perms/manage-database :perms/transforms]]))))))))))
|
||||
@@ -0,0 +1,54 @@
|
||||
(ns metabase-enterprise.data-apps.query-definition-test
|
||||
(:require
|
||||
[clojure.test :refer [deftest is testing]]
|
||||
[metabase-enterprise.data-apps.query-definition :as query-definition]
|
||||
[metabase.lib.schema.test-spec :as test-spec]
|
||||
[metabase.util.malli.registry :as mr]))
|
||||
|
||||
(def ^:private query-definition
|
||||
{:stages [{:source {:type :table :id 1}
|
||||
:fields [{:type :column :name "ID"}
|
||||
{:type :column :name "NAME" :source-field-id 2}]
|
||||
:filters [{:type :operator :operator :>
|
||||
:args [{:type :column :name "PRICE"}
|
||||
{:type :literal :value 1}]}
|
||||
{:type :segment :id 1}]
|
||||
:aggregations [{:type :operator :operator :count :args []}
|
||||
{:type :measure :id 1}
|
||||
{:type :metric :id 1}]
|
||||
:breakouts [{:type :column :name "DATE" :unit :month}
|
||||
{:type :column :name "PRICE" :bins 10}]
|
||||
:order-bys [{:type :column :name "DATE" :unit :month :direction :asc}]
|
||||
:limit 10}]})
|
||||
|
||||
(deftest ^:parallel supported-query-fields-test
|
||||
(is (nil? (mr/explain ::query-definition/query-definition query-definition))))
|
||||
|
||||
(deftest ^:parallel unknown-query-fields-test
|
||||
(doseq [path [[] [:stages 0] [:stages 0 :source]
|
||||
[:stages 0 :fields 0]
|
||||
[:stages 0 :filters 0 :args 1]
|
||||
[:stages 0 :aggregations 1] [:stages 0 :breakouts 0]
|
||||
[:stages 0 :order-bys 0]]]
|
||||
(testing (str "Unknown fields are not part of the request contract at " path)
|
||||
(is (not (mr/validate ::query-definition/query-definition
|
||||
(assoc-in query-definition (conj path :unexpected) true)))))))
|
||||
|
||||
(deftest ^:parallel required-query-source-test
|
||||
(is (not (mr/validate ::query-definition/query-definition {:stages []})))
|
||||
(is (not (mr/validate ::query-definition/query-definition {:stages [{}]}))))
|
||||
|
||||
(deftest ^:parallel internal-column-schemas-remain-open-test
|
||||
(let [column {:type :column :name "DATE" :unit :month}]
|
||||
(is (mr/validate ::test-spec/test-column-spec column))
|
||||
(is (not (mr/validate ::query-definition/column column)))
|
||||
(is (mr/validate ::query-definition/breakout column))))
|
||||
|
||||
(deftest ^:parallel unsupported-query-features-test
|
||||
(doseq [query [{:stages (repeat 2 (first (:stages query-definition)))}
|
||||
(assoc-in query-definition [:stages 0 :source] {:type :card :id 1})
|
||||
(assoc-in query-definition [:stages 0 :joins] [])
|
||||
(assoc-in query-definition [:stages 0 :expressions] [])
|
||||
(assoc-in query-definition [:stages 0 :aggregations]
|
||||
[{:name "count" :value {:type :operator :operator :count :args []}}])]]
|
||||
(is (not (mr/validate ::query-definition/query-definition query)))))
|
||||
@@ -0,0 +1,153 @@
|
||||
(ns metabase-enterprise.data-apps.resources-test
|
||||
(:require
|
||||
[clojure.test :refer :all]
|
||||
[metabase-enterprise.data-apps.resources :as data-app.resources]
|
||||
[metabase.collections.models.collection :as collection]
|
||||
[metabase.permissions.core :as perms]
|
||||
[metabase.sso.core :as sso]
|
||||
[metabase.test :as mt]
|
||||
[toucan2.core :as t2]))
|
||||
|
||||
(set! *warn-on-reflection* true)
|
||||
|
||||
(defn- create-data-app!
|
||||
[name]
|
||||
(t2/insert-returning-instance! :model/DataApp
|
||||
{:name name
|
||||
:display_name name
|
||||
:bundle_path (format "data_apps/%s/index.js" name)}))
|
||||
|
||||
(deftest ensure-resources-restores-a-collection-trashed-through-an-ancestor-test
|
||||
(testing "an app collection filed under another collection is archived indirectly when that
|
||||
ancestor is trashed, and the ancestor stays there — so it is restored to the root
|
||||
rather than to a parent that would reject it"
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(mt/with-test-user :crowberto
|
||||
(mt/with-temp [:model/Collection {ancestor-id :id} {:name "Filed under" :location "/"}]
|
||||
(let [app (create-data-app! "wrens")
|
||||
{:keys [resource_collection_id]} (data-app.resources/ensure-resources! app)]
|
||||
(collection/move-collection!
|
||||
(t2/select-one :model/Collection :id resource_collection_id)
|
||||
(collection/children-location (t2/select-one :model/Collection :id ancestor-id)))
|
||||
(collection/archive-or-unarchive-collection!
|
||||
(t2/select-one :model/Collection :id ancestor-id)
|
||||
{:archived true})
|
||||
(is (true? (t2/select-one-fn :archived :model/Collection :id resource_collection_id))
|
||||
"precondition: the ancestor took the app collection with it")
|
||||
(data-app.resources/ensure-resources! (t2/select-one :model/DataApp :id (:id app)))
|
||||
(is (false? (t2/select-one-fn :archived :model/Collection :id resource_collection_id))
|
||||
"the app collection is usable again")
|
||||
(is (= "/" (t2/select-one-fn :location :model/Collection :id resource_collection_id))
|
||||
"and sits at the root, not under the ancestor still in the trash")
|
||||
(is (true? (t2/select-one-fn :archived :model/Collection :id ancestor-id))
|
||||
"the ancestor is left where the admin put it")))))))
|
||||
|
||||
(deftest ensure-resources-restores-a-trashed-collection-test
|
||||
(testing "trashing the resource collection archives the copies the app is served from,
|
||||
so ensure-resources! has to bring both back or a successful sync leaves the app blank"
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(let [app (create-data-app! "sparrows")
|
||||
{:keys [resource_collection_id]} (data-app.resources/ensure-resources! app)]
|
||||
;; The app collection is readable by its own group alone, so only an admin
|
||||
;; can put the copy there.
|
||||
(mt/with-test-user :crowberto
|
||||
(mt/with-temp [:model/Card {card-id :id} {:collection_id resource_collection_id}]
|
||||
(collection/archive-or-unarchive-collection!
|
||||
(t2/select-one :model/Collection :id resource_collection_id)
|
||||
{:archived true})
|
||||
(is (true? (t2/select-one-fn :archived :model/Collection :id resource_collection_id))
|
||||
"precondition: the collection is in the trash")
|
||||
(is (true? (t2/select-one-fn :archived :model/Card :id card-id))
|
||||
"precondition: trashing the collection archived the copy")
|
||||
(data-app.resources/ensure-resources! (t2/select-one :model/DataApp :id (:id app)))
|
||||
(is (false? (t2/select-one-fn :archived :model/Collection :id resource_collection_id))
|
||||
"the app collection is out of the trash")
|
||||
(is (false? (t2/select-one-fn :archived :model/Card :id card-id))
|
||||
"the copy the app serves is readable again")))))))
|
||||
|
||||
(deftest ensure-resources-blocks-the-app-groups-view-data-test
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(let [app (create-data-app! "birds")
|
||||
{:keys [permission_group_id]} (data-app.resources/ensure-resources! app)
|
||||
perm (fn [perm-type] (t2/select [:model/DataPermissions :table_id :perm_value]
|
||||
:group_id permission_group_id
|
||||
:db_id (mt/id)
|
||||
:perm_type perm-type))]
|
||||
(testing "the app group is flagged as a data-app group (its own namespace)"
|
||||
(is (true? (t2/select-one-fn :is_data_app_group :model/PermissionsGroup :id permission_group_id))))
|
||||
(testing "the app group is blocked at the database level, granting no data access of its own"
|
||||
(is (=? [{:table_id nil, :perm_value :blocked}] (perm :perms/view-data)))
|
||||
(testing "view-data :blocked cascades download-results/transforms to :no"
|
||||
(is (=? [{:table_id nil, :perm_value :no}] (perm :perms/download-results)))
|
||||
(is (=? [{:table_id nil, :perm_value :no}] (perm :perms/transforms)))))
|
||||
(testing "a manual table-level view-data grant is swept back to the database-wide block on the next sync"
|
||||
(perms/set-table-permissions! permission_group_id :perms/view-data
|
||||
{(mt/id :venues) :unrestricted})
|
||||
(data-app.resources/ensure-resources! app)
|
||||
(is (=? [{:table_id nil, :perm_value :blocked}] (perm :perms/view-data))))
|
||||
(testing "a manual download-results grant that left view-data blocked is swept back too"
|
||||
(perms/set-database-permission! permission_group_id (mt/id) :perms/download-results :one-million-rows)
|
||||
(data-app.resources/ensure-resources! app)
|
||||
(is (=? [{:table_id nil, :perm_value :no}] (perm :perms/download-results)))
|
||||
(is (=? [{:table_id nil, :perm_value :blocked}] (perm :perms/view-data)))))))
|
||||
|
||||
(deftest data-app-groups-hidden-from-the-groups-api-test
|
||||
(testing "GET /api/permissions/group never lists data-app groups — they are a server-managed namespace"
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(mt/with-temp [:model/PermissionsGroup {normal-group-id :id} {:name "Normal Group"}]
|
||||
(let [{app-group-id :permission_group_id} (data-app.resources/ensure-resources! (create-data-app! "some-app"))
|
||||
listed-ids (into #{} (map :id) (mt/user-http-request :crowberto :get 200 "permissions/group"))]
|
||||
(is (contains? listed-ids normal-group-id) "a normal group is listed")
|
||||
(is (not (contains? listed-ids app-group-id)) "the data-app group is hidden"))))))
|
||||
|
||||
(deftest sso-group-sync-does-not-add-a-user-to-a-data-app-group-test
|
||||
(testing "a data app's permission group is server-managed: membership is granted by an admin, not
|
||||
an IdP claim. SSO group sync must not add a user to it, or an IdP `groups` value naming
|
||||
\"Data App: <slug>\" would hand the app's collection to whoever it names."
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(mt/with-temp [:model/User {user-id :id} {}]
|
||||
(let [app (create-data-app! "birds")
|
||||
{:keys [permission_group_id]} (data-app.resources/ensure-resources! app)]
|
||||
;; As a name-based SSO sync would, ask to put the user in the data-app group.
|
||||
(sso/sync-group-memberships! user-id [permission_group_id])
|
||||
(is (not (t2/exists? :model/PermissionsGroupMembership
|
||||
:user_id user-id :group_id permission_group_id))
|
||||
"SSO group sync must not add a user to a data-app group"))))))
|
||||
|
||||
(deftest ensure-resources-preserves-collection-grants-test
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(let [app (create-data-app! "birds")
|
||||
{group-id :permission_group_id collection-id :resource_collection_id} (data-app.resources/ensure-resources! app)
|
||||
read-path (perms/collection-read-path collection-id)
|
||||
grant (t2/select-one :model/Permissions :group_id group-id :object read-path)]
|
||||
(is (some? grant))
|
||||
(data-app.resources/ensure-resources! app)
|
||||
; correct grants should not be changed after sync
|
||||
(is (= grant (t2/select-one :model/Permissions :group_id group-id :object read-path))))))
|
||||
|
||||
(deftest ensure-resources-repairs-collection-grants-test
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(mt/with-temp [:model/PermissionsGroup {other-group-id :id} {}]
|
||||
(let [app (create-data-app! "birds")
|
||||
{group-id :permission_group_id collection-id :resource_collection_id} (data-app.resources/ensure-resources! app)
|
||||
read-path (perms/collection-read-path collection-id)
|
||||
write-path (perms/collection-readwrite-path collection-id)]
|
||||
; grant incorrect permission: write access to the data app collection
|
||||
(perms/grant-collection-readwrite-permissions! group-id collection-id)
|
||||
; grant incorrect permission: let other groups read the data app collection
|
||||
(perms/grant-collection-read-permissions! other-group-id collection-id)
|
||||
(data-app.resources/ensure-resources! app)
|
||||
; the two bad grants should be removed after sync.
|
||||
; only the correct grants should remain.
|
||||
(is (= [{:group_id group-id :object read-path}]
|
||||
(t2/select [:model/Permissions :group_id :object]
|
||||
:object [:in [read-path write-path]])))))))
|
||||
|
||||
(deftest ensure-resources-restores-missing-collection-grant-test
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(let [app (create-data-app! "birds")
|
||||
{group-id :permission_group_id collection-id :resource_collection_id} (data-app.resources/ensure-resources! app)]
|
||||
(perms/revoke-collection-permissions! group-id collection-id)
|
||||
(data-app.resources/ensure-resources! app)
|
||||
; read access should be restored after sync
|
||||
(is (t2/exists? :model/Permissions :group_id group-id :object (perms/collection-read-path collection-id))))))
|
||||
@@ -6,8 +6,11 @@
|
||||
(:require
|
||||
[clojure.string :as str]
|
||||
[clojure.test :refer :all]
|
||||
[metabase-enterprise.data-apps.db :as data-apps.db]
|
||||
[metabase-enterprise.data-apps.resources :as data-app.resources]
|
||||
[metabase-enterprise.data-apps.sync :as data-app.sync]
|
||||
[metabase-enterprise.remote-sync.source :as source]
|
||||
[metabase.permissions.core :as perms]
|
||||
[metabase.test :as mt]
|
||||
[toucan2.core :as t2]))
|
||||
|
||||
@@ -47,8 +50,75 @@
|
||||
"with the feature it materializes the app as usual")
|
||||
(is (t2/exists? :model/DataApp :name "a")))))))
|
||||
|
||||
(deftest changed-count-tracks-content-not-sha-bumps-test
|
||||
(deftest resource-provisioning-failures-are-isolated-test
|
||||
(mt/with-model-cleanup [:model/DataApp]
|
||||
(let [files (merge (app-files "broken" {:name "Broken" :path "index.js" :bundle "BROKEN"})
|
||||
(app-files "working" {:name "Working" :path "index.js" :bundle "WORKING"}))]
|
||||
(with-redefs [data-app.resources/ensure-resources!
|
||||
(fn [app]
|
||||
(when (= "broken" (:name app))
|
||||
(throw (ex-info "Resource provisioning failed." {}))))]
|
||||
(is (=? {:synced 2 :changed 2}
|
||||
(data-app.sync/import-from-snapshot! (snapshot files))))
|
||||
(is (= "Resource provisioning failed."
|
||||
(t2/select-one-fn :sync_error :model/DataApp :name "broken")))
|
||||
(is (nil? (t2/select-one-fn :sync_error :model/DataApp :name "working")))))))
|
||||
|
||||
(deftest sync-creates-stable-permission-resources-test
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(let [database-id (mt/id)
|
||||
files (app-files "sales" {:name "Sales" :path "index.js" :bundle "V1"})]
|
||||
(data-app.sync/import-from-snapshot! (snapshot files))
|
||||
(let [{:keys [resource_collection_id permission_group_id]}
|
||||
(t2/select-one :model/DataApp :name "sales")]
|
||||
(testing "the first sync creates a dedicated group and collection"
|
||||
(is (pos-int? resource_collection_id))
|
||||
(is (pos-int? permission_group_id))
|
||||
(is (= "Data App: sales"
|
||||
(t2/select-one-fn :name :model/Collection :id resource_collection_id)))
|
||||
(is (= "Data App: sales"
|
||||
(t2/select-one-fn :name :model/PermissionsGroup :id permission_group_id)))
|
||||
(is (t2/exists? :model/Permissions
|
||||
:group_id permission_group_id
|
||||
:object (perms/collection-read-path resource_collection_id)))
|
||||
(is (not (t2/exists? :model/Permissions
|
||||
:group_id permission_group_id
|
||||
:object (perms/collection-readwrite-path resource_collection_id))))
|
||||
(let [permissions (t2/select-fn-vec :perm_value :model/DataPermissions
|
||||
:group_id permission_group_id
|
||||
:db_id database-id
|
||||
:perm_type :perms/create-queries)]
|
||||
(is (seq permissions))
|
||||
(is (every? #(= :no %) permissions))))
|
||||
(testing "later syncs reuse the same resources"
|
||||
(data-app.sync/import-from-snapshot! (snapshot files))
|
||||
(is (=? {:resource_collection_id resource_collection_id
|
||||
:permission_group_id permission_group_id}
|
||||
(t2/select-one :model/DataApp :name "sales"))))
|
||||
(testing "removing the app deletes its resources"
|
||||
(data-app.sync/import-from-snapshot! (snapshot {}))
|
||||
(is (not (t2/exists? :model/Collection :id resource_collection_id)))
|
||||
(is (not (t2/exists? :model/PermissionsGroup :id permission_group_id))))))))
|
||||
|
||||
(deftest sync-restores-deleted-collection-and-permission-group-test
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(let [files (app-files "sales" {:name "Sales" :path "index.js" :bundle "V1"})]
|
||||
(data-app.sync/import-from-snapshot! (snapshot files))
|
||||
(let [before (select-keys (t2/select-one :model/DataApp :name "sales")
|
||||
[:resource_collection_id :permission_group_id])]
|
||||
(t2/delete! :model/Collection :id (:resource_collection_id before))
|
||||
(t2/delete! :model/PermissionsGroup :id (:permission_group_id before))
|
||||
(data-app.sync/import-from-snapshot! (snapshot files))
|
||||
; after syncing the app again, the data app collection and
|
||||
; permission group should be re-created
|
||||
(let [after (select-keys (t2/select-one :model/DataApp :name "sales") (keys before))]
|
||||
(is (every? pos-int? (vals after)))
|
||||
(is (t2/exists? :model/Collection :id (:resource_collection_id after)))
|
||||
(is (t2/exists? :model/PermissionsGroup :id (:permission_group_id after)))
|
||||
(is (not= before after)))))))
|
||||
|
||||
(deftest changed-count-tracks-content-not-sha-bumps-test
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(let [files (app-files "a" {:name "A" :path "index.js" :bundle "V1"})]
|
||||
(testing "the first sync counts the new app"
|
||||
(is (=? {:synced 1 :changed 1}
|
||||
@@ -67,7 +137,7 @@
|
||||
(snapshot (app-files "a" {:name "A renamed" :path "index.js" :bundle "V2"})))))))))
|
||||
|
||||
(deftest description-is-optional-and-tracked-like-other-metadata-test
|
||||
(mt/with-model-cleanup [:model/DataApp]
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(let [sync-app (fn [& {:as app}]
|
||||
(data-app.sync/import-from-snapshot!
|
||||
(snapshot (app-files "a" (merge {:name "A" :path "index.js" :bundle "V1"} app)))))]
|
||||
@@ -85,13 +155,11 @@
|
||||
|
||||
(deftest metadata-edits-count-while-an-app-keeps-failing-test
|
||||
(testing "an app whose bundle is missing still stores metadata edits, so they count as changes"
|
||||
(mt/with-model-cleanup [:model/DataApp]
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(let [sync-app (fn [& {:as app}]
|
||||
(data-app.sync/import-from-snapshot!
|
||||
(snapshot {"data_apps/a/data_app.yaml"
|
||||
(str "name: A\npath: index.js\n"
|
||||
(when-let [d (:description app)]
|
||||
(format "description: %s\n" d)))})))]
|
||||
(let [files (app-files "a" (merge {:name "A" :path "index.js" :bundle "unused"} app))]
|
||||
(data-app.sync/import-from-snapshot!
|
||||
(snapshot (dissoc files "data_apps/a/index.js")))))]
|
||||
(is (=? {:changed 1} (sync-app :description "First")) "the first failure is a change")
|
||||
(is (some? (t2/select-one-fn :sync_error :model/DataApp :name "a")))
|
||||
(testing "re-syncing the same failing app unchanged is not a change"
|
||||
@@ -103,7 +171,7 @@
|
||||
|
||||
(deftest switching-repos-prunes-old-apps-overrides-shared-adds-new-test
|
||||
(testing "syncing a different repo: drop apps only the old repo had, override shared slugs, add new ones"
|
||||
(mt/with-model-cleanup [:model/DataApp]
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
;; Repo A: Foo + Bar
|
||||
(data-app.sync/import-from-snapshot!
|
||||
(snapshot (merge (app-files "foo" {:name "Foo" :path "index.js" :bundle "FOO"})
|
||||
@@ -125,7 +193,7 @@
|
||||
|
||||
(deftest an-empty-repo-prunes-all-apps-test
|
||||
(testing "syncing a repo with no data_apps/ removes every app (the repo has none)"
|
||||
(mt/with-model-cleanup [:model/DataApp]
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(data-app.sync/import-from-snapshot!
|
||||
(snapshot (app-files "solo" {:name "Solo" :path "index.js" :bundle "S"})))
|
||||
(is (= #{"solo"} (t2/select-fn-set :name :model/DataApp)))
|
||||
@@ -133,9 +201,48 @@
|
||||
(data-app.sync/import-from-snapshot! (snapshot {}))))
|
||||
(is (empty? (t2/select-fn-set :name :model/DataApp))))))
|
||||
|
||||
(deftest remote-sync-preserves-unpublished-data-app-drafts-test
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(data-app.sync/ensure-draft! "draft-app")
|
||||
(is (=? {:removed 0}
|
||||
(data-app.sync/import-from-snapshot! (snapshot {}))))
|
||||
(is (true? (t2/select-one-fn :draft :model/DataApp :name "draft-app")))
|
||||
(data-app.sync/import-from-snapshot!
|
||||
(snapshot (app-files "draft-app" {:name "Draft" :path "index.js" :bundle "BUNDLE"})))
|
||||
(is (false? (t2/select-one-fn :draft :model/DataApp :name "draft-app")))
|
||||
(is (=? {:removed 1}
|
||||
(data-app.sync/import-from-snapshot! (snapshot {}))))
|
||||
(is (not (t2/exists? :model/DataApp :name "draft-app")))))
|
||||
|
||||
(deftest repository-claims-a-draft-before-a-successful-import-test
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(data-app.sync/ensure-draft! "draft-app")
|
||||
(data-app.sync/import-from-snapshot!
|
||||
(snapshot {"data_apps/draft-app/data_app.yaml" "name: Draft\n"}))
|
||||
(is (false? (t2/select-one-fn :draft :model/DataApp :name "draft-app")))
|
||||
(is (=? {:removed 1}
|
||||
(data-app.sync/import-from-snapshot! (snapshot {}))))
|
||||
(is (not (t2/exists? :model/DataApp :name "draft-app")))))
|
||||
|
||||
(deftest concurrent-draft-creation-is-safe-test
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(let [drafts (doall (repeatedly 2 #(future (data-app.sync/ensure-draft! "draft-app"))))]
|
||||
(doseq [draft drafts]
|
||||
@draft)
|
||||
(is (= 1 (t2/count :model/DataApp :name "draft-app"))))))
|
||||
|
||||
(deftest remote-sync-prunes-never-successful-repository-apps-test
|
||||
(mt/with-model-cleanup [:model/DataApp]
|
||||
(data-app.sync/import-from-snapshot!
|
||||
(snapshot {"data_apps/broken/data_app.yaml" "name: Broken\npath: missing.js\n"}))
|
||||
(is (false? (t2/select-one-fn :draft :model/DataApp :name "broken")))
|
||||
(is (=? {:removed 1}
|
||||
(data-app.sync/import-from-snapshot! (snapshot {}))))
|
||||
(is (not (t2/exists? :model/DataApp :name "broken")))))
|
||||
|
||||
(deftest a-broken-config-does-not-prune-the-existing-app-test
|
||||
(testing "a directory that still exists but whose data_app.yaml is now broken keeps the app (as a sync_error), it is not pruned"
|
||||
(mt/with-model-cleanup [:model/DataApp]
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(data-app.sync/import-from-snapshot!
|
||||
(snapshot (app-files "app" {:name "App" :path "index.js" :bundle "GOOD"})))
|
||||
(is (= "GOOD" (String. ^bytes (:bundle (t2/select-one :model/DataApp :name "app")) "UTF-8")))
|
||||
@@ -157,7 +264,7 @@
|
||||
|
||||
(deftest a-broken-config-for-a-brand-new-app-materializes-nothing-test
|
||||
(testing "an app whose config never parsed has no row to mark — it simply isn't materialized"
|
||||
(mt/with-model-cleanup [:model/DataApp]
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(let [result (data-app.sync/import-from-snapshot!
|
||||
(snapshot {"data_apps/newbie/data_app.yaml" "name: Newbie\n" ; missing required "path"
|
||||
"data_apps/newbie/index.js" "X"}))]
|
||||
@@ -170,7 +277,7 @@
|
||||
|
||||
(deftest oversized-bundle-is-rejected-test
|
||||
(testing "a bundle over the size cap is rejected with a sync_error, no bundle cached"
|
||||
(mt/with-model-cleanup [:model/DataApp]
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(data-app.sync/import-from-snapshot!
|
||||
(snapshot (app-files "big" {:name "Big" :path "index.js"
|
||||
:bundle (oversized-bundle)})))
|
||||
@@ -181,7 +288,7 @@
|
||||
|
||||
(deftest oversized-resync-keeps-the-previous-bundle-test
|
||||
(testing "an oversized re-sync sets sync_error but keeps the last good bundle"
|
||||
(mt/with-model-cleanup [:model/DataApp]
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(data-app.sync/import-from-snapshot!
|
||||
(snapshot (app-files "app" {:name "App" :path "index.js" :bundle "GOOD"})))
|
||||
(data-app.sync/import-from-snapshot!
|
||||
@@ -194,7 +301,7 @@
|
||||
|
||||
(deftest a-directory-without-a-config-is-not-an-app-test
|
||||
(testing "a data_apps/<dir> that ships a bundle but no data_app.yaml is not discovered — no app, no error"
|
||||
(mt/with-model-cleanup [:model/DataApp]
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(let [result (data-app.sync/import-from-snapshot!
|
||||
(snapshot {"data_apps/orphan/index.js" "BUNDLE"}))]
|
||||
(is (=? {:synced 0 :changed 0 :config-errors []} result))
|
||||
@@ -202,10 +309,56 @@
|
||||
|
||||
(deftest an-unreadable-config-is-a-config-error-test
|
||||
(testing "a data_app.yaml the snapshot lists but can't read is isolated as a config-error, not a crash"
|
||||
(mt/with-model-cleanup [:model/DataApp]
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
;; the config is listed in the tree, but reading its blob yields nothing
|
||||
(let [result (data-app.sync/import-from-snapshot!
|
||||
(snapshot {"data_apps/ghost/data_app.yaml" nil}))]
|
||||
(is (= 1 (count (:config-errors result))))
|
||||
(is (str/includes? (first (:config-errors result)) "data_apps/ghost/data_app.yaml"))
|
||||
(is (empty? (t2/select-fn-set :name :model/DataApp)))))))
|
||||
|
||||
(deftest resource-failure-rolls-back-only-that-app-test
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(let [files (merge (app-files "broken" {:name "Broken" :path "index.js" :bundle "OLD"})
|
||||
(app-files "working" {:name "Working" :path "index.js" :bundle "OLD"}))
|
||||
ensure-resources! data-app.resources/ensure-resources!]
|
||||
(data-app.sync/import-from-snapshot! (snapshot files))
|
||||
(let [collection-id (t2/select-one-fn :resource_collection_id :model/DataApp :name "broken")
|
||||
old-name (t2/select-one-fn :name :model/Collection :id collection-id)]
|
||||
; inject an error into `ensure-resources!` for the "broken" app
|
||||
(with-redefs [data-app.resources/ensure-resources!
|
||||
(fn [app]
|
||||
(ensure-resources! app)
|
||||
(when (= "broken" (:name app))
|
||||
(t2/update! :model/Collection :id collection-id {:name "Partial update"})
|
||||
(throw (ex-info "Resource provisioning failed." {}))))]
|
||||
(data-app.sync/import-from-snapshot!
|
||||
(snapshot (assoc files "data_apps/broken/index.js" "NEW" "data_apps/working/index.js" "NEW"))))
|
||||
; the "broken" app bundle with sync error should be rolled back to "OLD"
|
||||
(is (= "OLD" (String. ^bytes (t2/select-one-fn :bundle :model/DataApp :name "broken") "UTF-8")))
|
||||
; the collection name should be rolled back to the old name
|
||||
(is (= old-name (t2/select-one-fn :name :model/Collection :id collection-id)))
|
||||
; the thrown exception in ensure-resources! should be reported in the sync error
|
||||
(is (= "Resource provisioning failed." (t2/select-one-fn :sync_error :model/DataApp :name "broken")))
|
||||
; the "working" app without sync errors should not be rolled back to "OLD"
|
||||
(is (= "NEW" (String. ^bytes (t2/select-one-fn :bundle :model/DataApp :name "working") "UTF-8")))))))
|
||||
|
||||
(deftest pruning-failure-does-not-roll-back-synced-apps-test
|
||||
(mt/with-premium-features #{:data-apps-preview}
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(let [working-files (app-files "working" {:name "Working" :path "index.js" :bundle "OLD"})
|
||||
files (merge working-files (app-files "removed" {:name "Removed" :path "index.js" :bundle "OLD"}))
|
||||
prune! data-apps.db/delete-data-apps-not-named!]
|
||||
(data-app.sync/import-from-snapshot! (snapshot files))
|
||||
; inject an error into deleting data apps
|
||||
(with-redefs [data-apps.db/delete-data-apps-not-named!
|
||||
(fn [slugs]
|
||||
(prune! slugs)
|
||||
(throw (ex-info "Pruning failed." {})))]
|
||||
; sync should report the error from pruning step
|
||||
(is (=? {:synced 1 :changed 1 :removed 0 :pruning-error "Pruning failed."}
|
||||
(data-app.sync/sync-from-snapshot!
|
||||
(snapshot (assoc working-files "data_apps/working/index.js" "NEW"))))))
|
||||
; a pruning failure should not roll back the updated app's bundle
|
||||
(is (= "NEW" (String. ^bytes (t2/select-one-fn :bundle :model/DataApp :name "working") "UTF-8")))
|
||||
(is (t2/exists? :model/DataApp :name "removed"))))))
|
||||
|
||||
@@ -19,7 +19,6 @@
|
||||
|
||||
(use-fixtures :once (fixtures/initialize :db))
|
||||
(use-fixtures :each rs.test/clean-remote-sync-state rs.test/commit-with-temp)
|
||||
|
||||
(defn- import-at!
|
||||
"Run `import!` against the source's snapshot at `version`, complete the task (so
|
||||
`last-version` advances for the next pull), and return the result."
|
||||
@@ -30,7 +29,7 @@
|
||||
(impl/handle-task-result! result task)
|
||||
result))
|
||||
|
||||
(defn- app-tree
|
||||
(defn- app-tree!
|
||||
"Repo files for one data app: its `data_app.yaml` + a bundle at `dist/index.js`."
|
||||
[slug bundle]
|
||||
{(str "data_apps/" slug "/data_app.yaml") (str "name: " slug "\npath: dist/index.js\n")
|
||||
@@ -58,9 +57,9 @@
|
||||
(search.tu/with-index-disabled
|
||||
(mt/with-premium-features #{:data-apps-preview}
|
||||
(mt/with-temporary-setting-values [remote-sync-type :read-write remote-sync-transforms false]
|
||||
(mt/with-model-cleanup [:model/DataApp]
|
||||
(let [outcome (pull-outcome! (app-tree "sales" "BUNDLE-V1")
|
||||
(app-tree "sales" "BUNDLE-V2"))]
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(let [outcome (pull-outcome! (app-tree! "sales" "BUNDLE-V1")
|
||||
(app-tree! "sales" "BUNDLE-V2"))]
|
||||
(is (= "pulled" (:kind outcome)) "not reported as skipped")
|
||||
(is (= 1 (:count outcome)) "the one changed data app is counted"))))))))
|
||||
|
||||
@@ -69,9 +68,9 @@
|
||||
(search.tu/with-index-disabled
|
||||
(mt/with-premium-features #{:data-apps-preview}
|
||||
(mt/with-temporary-setting-values [remote-sync-type :read-write remote-sync-transforms false]
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection]
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
;; v1 adds a collection; the data app is byte-for-byte the same as v0.
|
||||
(let [app (app-tree "sales" "BUNDLE")
|
||||
(let [app (app-tree! "sales" "BUNDLE")
|
||||
outcome (pull-outcome! app (merge coll-file app))]
|
||||
(is (= "pulled" (:kind outcome)))
|
||||
(is (= 1 (:count outcome)) "the collection counts; the unchanged app adds 0"))))))))
|
||||
@@ -81,9 +80,9 @@
|
||||
(search.tu/with-index-disabled
|
||||
(mt/with-premium-features #{:data-apps-preview}
|
||||
(mt/with-temporary-setting-values [remote-sync-type :read-write remote-sync-transforms false]
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection]
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
;; v1 adds a collection AND changes the data app's bundle.
|
||||
(let [outcome (pull-outcome! (app-tree "ops" "BUNDLE")
|
||||
(merge coll-file (app-tree "ops" "BUNDLE-V2")))]
|
||||
(let [outcome (pull-outcome! (app-tree! "ops" "BUNDLE")
|
||||
(merge coll-file (app-tree! "ops" "BUNDLE-V2")))]
|
||||
(is (= "pulled" (:kind outcome)))
|
||||
(is (= 2 (:count outcome)) "the collection (1) plus the changed data app (1)"))))))))
|
||||
|
||||
@@ -13,7 +13,6 @@
|
||||
|
||||
(use-fixtures :once (fixtures/initialize :db))
|
||||
(use-fixtures :each (fn [f] (test-helpers/clean-remote-sync-state f)))
|
||||
|
||||
(comment metabase-enterprise.data-apps.models.data-app/keep-me)
|
||||
|
||||
(defn- import! [files]
|
||||
@@ -32,7 +31,7 @@
|
||||
|
||||
(deftest import-materializes-data-apps-test
|
||||
(testing "a remote-sync import materializes data apps from data_apps/ alongside serdes content"
|
||||
(mt/with-model-cleanup [:model/DataApp]
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(let [files {"main" {"collections/c/c.yaml"
|
||||
(test-helpers/generate-collection-yaml "data-apps-test-collx" "DA Coll")
|
||||
"data_apps/sales/data_app.yaml" "name: Sales\npath: ./dist/index.js\n"
|
||||
@@ -48,7 +47,7 @@
|
||||
|
||||
(deftest import-prunes-data-apps-absent-from-repo-test
|
||||
(testing "an import whose repo no longer has an app dir prunes that app (the repo is the source of truth)"
|
||||
(mt/with-model-cleanup [:model/DataApp]
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
(import! {"main" {"data_apps/gone/data_app.yaml" "name: Gone\npath: ./i.js\n"
|
||||
"data_apps/gone/i.js" "X"
|
||||
"data_apps/kept/data_app.yaml" "name: Kept\npath: ./i.js\n"
|
||||
@@ -63,7 +62,7 @@
|
||||
|
||||
(deftest deletion-only-pull-counts-the-removal-test
|
||||
(testing "a pull whose only change is removing an app still reports as a pull — the removal is counted, not a no-op"
|
||||
(mt/with-model-cleanup [:model/DataApp]
|
||||
(mt/with-model-cleanup [:model/DataApp :model/Collection :model/PermissionsGroup]
|
||||
;; No serdes content, so the outcome's count comes purely from data apps.
|
||||
(import! {"main" {"README.md" "x"
|
||||
"data_apps/gone/data_app.yaml" "name: Gone\npath: ./i.js\n"
|
||||
|
||||
@@ -0,0 +1,12 @@
|
||||
import path from "node:path";
|
||||
|
||||
import { getResourceSyncCredentials } from "../../data-app-query-sync/env";
|
||||
import { syncResources } from "../../data-app-query-sync/sync";
|
||||
|
||||
export async function syncResourcesAction(appRoot = process.cwd()) {
|
||||
const resolvedRoot = path.resolve(appRoot);
|
||||
await syncResources({
|
||||
appRoot: resolvedRoot,
|
||||
...getResourceSyncCredentials(resolvedRoot),
|
||||
});
|
||||
}
|
||||
@@ -0,0 +1,49 @@
|
||||
import path from "node:path";
|
||||
|
||||
import { getResourceSyncCredentials } from "../../data-app-query-sync/env";
|
||||
import { syncResources } from "../../data-app-query-sync/sync";
|
||||
|
||||
import { syncResourcesAction } from "./sync-resources";
|
||||
|
||||
jest.mock("../../data-app-query-sync/env", () => ({
|
||||
getResourceSyncCredentials: jest.fn(),
|
||||
}));
|
||||
jest.mock("../../data-app-query-sync/sync", () => ({
|
||||
syncResources: jest.fn(),
|
||||
}));
|
||||
|
||||
describe("syncResourcesAction", () => {
|
||||
it("resolves the app root and loads its credentials", async () => {
|
||||
jest.mocked(getResourceSyncCredentials).mockReturnValue({
|
||||
metabaseUrl: "http://metabase.test",
|
||||
apiKey: "secret",
|
||||
});
|
||||
|
||||
await syncResourcesAction("data_apps/orders");
|
||||
|
||||
const appRoot = path.resolve("data_apps/orders");
|
||||
expect(getResourceSyncCredentials).toHaveBeenCalledWith(appRoot);
|
||||
expect(syncResources).toHaveBeenCalledWith({
|
||||
appRoot,
|
||||
metabaseUrl: "http://metabase.test",
|
||||
apiKey: "secret",
|
||||
});
|
||||
});
|
||||
|
||||
it("uses the current directory by default", async () => {
|
||||
jest.mocked(getResourceSyncCredentials).mockReturnValue({
|
||||
metabaseUrl: "http://metabase.test",
|
||||
apiKey: "secret",
|
||||
});
|
||||
|
||||
await syncResourcesAction();
|
||||
|
||||
const appRoot = path.resolve(process.cwd());
|
||||
expect(getResourceSyncCredentials).toHaveBeenCalledWith(appRoot);
|
||||
expect(syncResources).toHaveBeenCalledWith({
|
||||
appRoot,
|
||||
metabaseUrl: "http://metabase.test",
|
||||
apiKey: "secret",
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -1,6 +1,8 @@
|
||||
import { Command } from "commander";
|
||||
|
||||
import { start } from "./actions/start";
|
||||
import { addDataAppsCommands } from "./commands/data-apps";
|
||||
import { printError } from "./utils/print";
|
||||
|
||||
const program = new Command();
|
||||
|
||||
@@ -13,4 +15,9 @@ program
|
||||
.description("downloads and starts a local Metabase instance")
|
||||
.action(start);
|
||||
|
||||
program.parse();
|
||||
addDataAppsCommands(program);
|
||||
|
||||
program.parseAsync().catch((error: unknown) => {
|
||||
printError(error instanceof Error ? error.message : String(error));
|
||||
process.exitCode = 1;
|
||||
});
|
||||
|
||||
@@ -0,0 +1,17 @@
|
||||
import type { Command } from "commander";
|
||||
|
||||
import { syncResourcesAction } from "../actions/sync-resources";
|
||||
|
||||
export function addDataAppsCommands(program: Command) {
|
||||
const dataAppsCommand = program
|
||||
.command("data-apps")
|
||||
.description("manage Metabase data apps");
|
||||
|
||||
dataAppsCommand
|
||||
.command("sync-resources")
|
||||
.description(
|
||||
"synchronize data app query and action definitions into its collection",
|
||||
)
|
||||
.option("--app-root <path>", "data app directory", process.cwd())
|
||||
.action(({ appRoot }: { appRoot: string }) => syncResourcesAction(appRoot));
|
||||
}
|
||||
@@ -0,0 +1,36 @@
|
||||
import { Command } from "commander";
|
||||
|
||||
import { syncResourcesAction } from "../actions/sync-resources";
|
||||
|
||||
import { addDataAppsCommands } from "./data-apps";
|
||||
|
||||
jest.mock("../actions/sync-resources", () => ({
|
||||
syncResourcesAction: jest.fn(),
|
||||
}));
|
||||
|
||||
describe("data app commands", () => {
|
||||
it("runs resource synchronization for the requested app root", async () => {
|
||||
const program = new Command();
|
||||
addDataAppsCommands(program);
|
||||
|
||||
await program.parseAsync([
|
||||
"node",
|
||||
"cli",
|
||||
"data-apps",
|
||||
"sync-resources",
|
||||
"--app-root",
|
||||
"data_apps/orders",
|
||||
]);
|
||||
|
||||
expect(syncResourcesAction).toHaveBeenCalledWith("data_apps/orders");
|
||||
});
|
||||
|
||||
it("runs resource synchronization from the current directory by default", async () => {
|
||||
const program = new Command();
|
||||
addDataAppsCommands(program);
|
||||
|
||||
await program.parseAsync(["node", "cli", "data-apps", "sync-resources"]);
|
||||
|
||||
expect(syncResourcesAction).toHaveBeenCalledWith(process.cwd());
|
||||
});
|
||||
});
|
||||
@@ -8,6 +8,8 @@ import {
|
||||
loadEnv,
|
||||
} from "vite";
|
||||
|
||||
import { checkResourcesSynced } from "../data-app-query-sync/sync";
|
||||
|
||||
import { dataAppBuildPlugins, dataAppLibBuild } from "./config/build-config";
|
||||
import { getDataAppDefine } from "./config/define";
|
||||
import { buildDevCsp } from "./config/dev-connect-src";
|
||||
@@ -36,6 +38,13 @@ function dataAppVitePlugin(): PluginOption[] {
|
||||
react(),
|
||||
...dataAppBuildPlugins(),
|
||||
dataAppSandboxDevPlugin(appSlug, allowedHosts),
|
||||
{
|
||||
name: "metabase-resource-sync-check",
|
||||
apply: "build",
|
||||
buildStart: async () => {
|
||||
await checkResourcesSynced(appRoot);
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "metabase-data-app",
|
||||
// Merged over the user's config, so these win — the bundle Metabase loads
|
||||
|
||||
@@ -0,0 +1,125 @@
|
||||
# Resource sync
|
||||
|
||||
A data app runs in a browser sandbox as one of its viewers, not as its author. Those viewers reach
|
||||
Metabase through a permission group that can read exactly one collection: the app's own. So a query
|
||||
the app runs, and an action it triggers, have to be reachable **through that collection** or they
|
||||
fail on permissions.
|
||||
|
||||
Resource sync is the build-time step that makes that true. It reads the definitions an app commits
|
||||
to source control, materializes them inside the app's collection, and writes back the generated IDs.
|
||||
`npm run sync-resources` (`embedding-sdk-react data-apps sync-resources`) is the entry point.
|
||||
|
||||
The backend half — the collection and permission group themselves — lives in
|
||||
`metabase_enterprise/data_apps/resources.clj`. See that module's README for how apps are served.
|
||||
|
||||
## What an app declares
|
||||
|
||||
Two directories at the data-app root, beside `package.json`:
|
||||
|
||||
```
|
||||
queries/orders.query.ts export const RevenueQuery = defineQuery({ source: schema.tables.orders })
|
||||
actions/orders.action.ts export const CreateOrder = defineAction({ action: schema.models.orders.actions.create })
|
||||
```
|
||||
|
||||
Nothing else is scanned — definitions under `src/` are invisible to sync, which is a common
|
||||
authoring mistake.
|
||||
|
||||
## The pipeline
|
||||
|
||||
`syncResources` (`sync.ts`) runs four steps:
|
||||
|
||||
1. **Discover** every `defineQuery` / `defineAction` in those directories (`discover.ts`).
|
||||
2. **Ensure resources** — `POST /api/apps/:slug/draft` creates the app row if it doesn't exist
|
||||
yet, plus its collection and permission group.
|
||||
3. **Reconcile queries** (`reconcile.ts`) — each authored query becomes a saved question in the
|
||||
collection, and its ID is injected back into the source as `savedQuestionSourceId`.
|
||||
4. **Reconcile models** (`reconcile-models.ts`) — each declared action's model is copied into the
|
||||
collection, the action is copied onto that copy, and the copy's ID is injected back as
|
||||
`copiedActionId`.
|
||||
|
||||
## Why actions copy a whole model
|
||||
|
||||
An action's permissions resolve through its parent model's collection
|
||||
(`metabase.actions.models`, `perms-objects-set`). Copying a card does **not** copy its actions —
|
||||
`create-card!` writes one row and nothing traverses Card → Action — so making an action reachable
|
||||
means copying its model _and_ recreating the action on that copy.
|
||||
|
||||
Models are therefore reference-counted by the actions that need them. The lockfile entry holds the
|
||||
set of actions, not a count, because a count drifts across repeated runs while a set converges:
|
||||
|
||||
| Event | Model | Action |
|
||||
| ------------------------------------- | --------------- | ------------------------- |
|
||||
| action declared, model not yet copied | copy it | recreate on the copy |
|
||||
| action declared, model already copied | reuse | recreate just this action |
|
||||
| declaration dropped, siblings remain | keep | delete that copied action |
|
||||
| last declaration dropped | delete the copy | cascade removes the rest |
|
||||
|
||||
The last row is free: `action.model_id` is `ON DELETE CASCADE`, so deleting the copied model takes
|
||||
its copied actions with it.
|
||||
|
||||
## Discovery is deliberately strict
|
||||
|
||||
`discover.ts` parses each file twice over. The TypeScript AST locates `defineQuery`/`defineAction`
|
||||
calls and enforces that each one directly initializes a **named exported** variable with a single
|
||||
object literal. The file is then bundled with esbuild and evaluated — **twice** — and the two results
|
||||
compared, so a definition that isn't deterministic can't produce a fingerprint that changes under it.
|
||||
|
||||
Write-back re-parses from disk rather than reusing the AST from discovery, because reconciliation
|
||||
injects sequentially and an earlier injection into the same file shifts every offset after it.
|
||||
|
||||
## The lockfile is ownership proof
|
||||
|
||||
`resources_metadata.json` is generated, committed, and never hand-edited:
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"queries": [
|
||||
{ "tableId": 1, "hash": "v1:sha256:…", "savedQuestionSourceId": 54 },
|
||||
],
|
||||
"models": [
|
||||
{
|
||||
"sourceModelId": 5,
|
||||
"copiedModelId": 80,
|
||||
"hash": "v1:sha256:…",
|
||||
"actions": [
|
||||
{ "sourceActionId": 51, "copiedActionId": 91, "hash": "v1:sha256:…" },
|
||||
],
|
||||
},
|
||||
],
|
||||
}
|
||||
```
|
||||
|
||||
It is not a cache. It is the only evidence that a card or action in Metabase belongs to this app,
|
||||
and every mutating path checks it first. The `hash` fields fingerprint the **source** payload — the
|
||||
only way to notice that someone edited the upstream model or action, since nothing in the app's own
|
||||
source changes when they do.
|
||||
|
||||
## Invariants
|
||||
|
||||
These hold across both reconcilers, and are what the tests are mostly about:
|
||||
|
||||
- **Nothing is mutated or deleted without lockfile proof**, plus a live ownership check — still a
|
||||
question/model, still in the app's collection, action still hanging off the copied model. Anything
|
||||
that has moved is refused with recovery instructions rather than silently touched.
|
||||
- **The lockfile flushes after every single mutation**, so a crash mid-run leaves resumable state.
|
||||
- **`404` is the only recoverable failure.** Any other status rethrows; recreate-after-404
|
||||
additionally requires lockfile proof.
|
||||
- **On the backend, ownership is the foreign key, never the name.** A same-named group is refused,
|
||||
not adopted — adoption would hand its existing members the app's data access on a name match.
|
||||
|
||||
## Dev preview vs production build
|
||||
|
||||
Generated IDs only take effect in a production build. `isDataAppDev()` gates both swaps, so an app
|
||||
runs against the original table and the authored action before it has ever been synchronized:
|
||||
|
||||
| | Dev preview | Production |
|
||||
| ------------ | ------------------- | -------------------------------------------- |
|
||||
| query source | the table | the saved question (`savedQuestionSourceId`) |
|
||||
| action | the authored action | the copy (`copiedActionId`) |
|
||||
|
||||
Both swaps live in the bundle — `toSourceInput` and `toExecutableActionId` — not in app code, so an
|
||||
author passes the definition and never branches on the environment.
|
||||
|
||||
`checkResourcesSynced` runs at `buildStart` for production builds only. It re-reads the source and
|
||||
the lockfile and fails the build when they disagree, so a stale app can't ship: a stale build isn't
|
||||
broken, it's silently pointed at whatever the saved question was when it was last synced.
|
||||
@@ -0,0 +1,259 @@
|
||||
import fs from "node:fs";
|
||||
import path from "node:path";
|
||||
|
||||
import ts from "typescript";
|
||||
|
||||
import { SDK_PACKAGE_NAME } from "embedding-sdk-package/cli/constants/config";
|
||||
|
||||
export interface QuerySource {
|
||||
exportName: string;
|
||||
filePath: string;
|
||||
}
|
||||
|
||||
interface InspectedQuerySource extends QuerySource {
|
||||
object: ts.ObjectLiteralExpression;
|
||||
sourceFile: ts.SourceFile;
|
||||
}
|
||||
|
||||
/** A source-controlled definition kind, and where synchronization looks for it. */
|
||||
export interface DefinitionKind {
|
||||
directory: string;
|
||||
factory: string;
|
||||
idKey: string;
|
||||
/** Article included, so it reads in "did not evaluate to … object". */
|
||||
description: string;
|
||||
}
|
||||
|
||||
export const QUERY_DEFINITIONS: DefinitionKind = {
|
||||
directory: "queries",
|
||||
factory: "defineQuery",
|
||||
idKey: "savedQuestionSourceId",
|
||||
description: "a query",
|
||||
};
|
||||
|
||||
export const ACTION_DEFINITIONS: DefinitionKind = {
|
||||
directory: "actions",
|
||||
factory: "defineAction",
|
||||
idKey: "copiedActionId",
|
||||
description: "an action",
|
||||
};
|
||||
|
||||
const DATA_APP_MODULE = SDK_PACKAGE_NAME + "/data-app";
|
||||
|
||||
const QUERY_FILE_EXTENSIONS = [
|
||||
".js",
|
||||
".jsx",
|
||||
".ts",
|
||||
".tsx",
|
||||
".cjs",
|
||||
".cts",
|
||||
".mjs",
|
||||
".mts",
|
||||
];
|
||||
|
||||
export function findDefinitionSources(
|
||||
appRoot: string,
|
||||
kind: DefinitionKind,
|
||||
): QuerySource[] {
|
||||
return listQueryFiles(path.join(appRoot, kind.directory)).flatMap(
|
||||
(filePath) => inspectFile(filePath, kind),
|
||||
);
|
||||
}
|
||||
|
||||
/** Writes a generated ID back into its definition. */
|
||||
export function injectGeneratedId(
|
||||
target: QuerySource,
|
||||
kind: DefinitionKind,
|
||||
id: number,
|
||||
) {
|
||||
const source = inspectFile(target.filePath, kind).find(
|
||||
({ exportName }) => exportName === target.exportName,
|
||||
);
|
||||
|
||||
if (!source) {
|
||||
throw new Error(
|
||||
`Could not find ${target.exportName} in ${target.filePath}.`,
|
||||
);
|
||||
}
|
||||
|
||||
// Keep the definition's other properties and spreads, removing the old ID.
|
||||
const propertiesWithoutId = source.object.properties.filter(
|
||||
(prop) =>
|
||||
!(
|
||||
prop.name &&
|
||||
(ts.isIdentifier(prop.name) || ts.isStringLiteral(prop.name)) &&
|
||||
prop.name.text === kind.idKey
|
||||
),
|
||||
);
|
||||
|
||||
// Put the generated ID last so a spread cannot overwrite it with an old ID.
|
||||
const updatedDefinition = ts.factory.updateObjectLiteralExpression(
|
||||
source.object,
|
||||
[
|
||||
...propertiesWithoutId,
|
||||
|
||||
ts.factory.createPropertyAssignment(
|
||||
kind.idKey,
|
||||
ts.factory.createNumericLiteral(id),
|
||||
),
|
||||
],
|
||||
);
|
||||
|
||||
const replacement = ts
|
||||
.createPrinter()
|
||||
.printNode(ts.EmitHint.Expression, updatedDefinition, source.sourceFile);
|
||||
|
||||
const contents = source.sourceFile.text;
|
||||
const updated = `${contents.slice(0, source.object.getStart(source.sourceFile))}${replacement}${contents.slice(source.object.getEnd())}`;
|
||||
fs.writeFileSync(target.filePath, updated);
|
||||
}
|
||||
|
||||
function listQueryFiles(directory: string): string[] {
|
||||
if (!fs.existsSync(directory)) {
|
||||
return [];
|
||||
}
|
||||
|
||||
return fs
|
||||
.readdirSync(directory, { withFileTypes: true })
|
||||
.flatMap((entry) => {
|
||||
const itemPath = path.join(directory, entry.name);
|
||||
return entry.isDirectory() ? listQueryFiles(itemPath) : [itemPath];
|
||||
})
|
||||
.filter(isQueryFile)
|
||||
.sort();
|
||||
}
|
||||
|
||||
const isQueryFile = (filePath: string) =>
|
||||
QUERY_FILE_EXTENSIONS.some((extension) => filePath.endsWith(extension));
|
||||
|
||||
const hasExportModifier = (node: ts.Node) =>
|
||||
ts.canHaveModifiers(node)
|
||||
? ts
|
||||
.getModifiers(node)
|
||||
?.some((modifier) => modifier.kind === ts.SyntaxKind.ExportKeyword)
|
||||
: false;
|
||||
|
||||
function isDataAppImport(
|
||||
statement: ts.Statement,
|
||||
): statement is ts.ImportDeclaration {
|
||||
return (
|
||||
ts.isImportDeclaration(statement) &&
|
||||
ts.isStringLiteral(statement.moduleSpecifier) &&
|
||||
statement.moduleSpecifier.text === DATA_APP_MODULE
|
||||
);
|
||||
}
|
||||
|
||||
function getDataAppNamedImports(statement: ts.Statement) {
|
||||
if (!isDataAppImport(statement)) {
|
||||
return undefined;
|
||||
}
|
||||
|
||||
const namedBindings = statement.importClause?.namedBindings;
|
||||
|
||||
if (!namedBindings || !ts.isNamedImports(namedBindings)) {
|
||||
return undefined;
|
||||
}
|
||||
|
||||
return namedBindings.elements;
|
||||
}
|
||||
|
||||
const isFactoryImport = (item: ts.ImportSpecifier, factory: string) =>
|
||||
(item.propertyName ?? item.name).text === factory;
|
||||
|
||||
const isDirectVariableInitialization = (
|
||||
declaration: ts.Node,
|
||||
initializer: ts.CallExpression,
|
||||
): declaration is ts.VariableDeclaration & { name: ts.Identifier } =>
|
||||
ts.isVariableDeclaration(declaration) &&
|
||||
declaration.initializer === initializer &&
|
||||
ts.isIdentifier(declaration.name);
|
||||
|
||||
const isNamedExportStatement = (statement: ts.Node | undefined) =>
|
||||
Boolean(
|
||||
statement &&
|
||||
ts.isVariableStatement(statement) &&
|
||||
hasExportModifier(statement),
|
||||
);
|
||||
|
||||
const definitionObjectArgument = (node: ts.CallExpression) => {
|
||||
const [argument] = node.arguments;
|
||||
return node.arguments.length === 1 && ts.isObjectLiteralExpression(argument)
|
||||
? argument
|
||||
: undefined;
|
||||
};
|
||||
|
||||
const isDirectNamedDefinition = (
|
||||
node: ts.CallExpression,
|
||||
declaration: ts.Node,
|
||||
statement: ts.Node | undefined,
|
||||
): declaration is ts.VariableDeclaration & { name: ts.Identifier } =>
|
||||
isDirectVariableInitialization(declaration, node) &&
|
||||
isNamedExportStatement(statement);
|
||||
|
||||
const isFactoryCall = (
|
||||
node: ts.Node,
|
||||
names: Set<string>,
|
||||
): node is ts.CallExpression =>
|
||||
ts.isCallExpression(node) &&
|
||||
ts.isIdentifier(node.expression) &&
|
||||
names.has(node.expression.text);
|
||||
|
||||
function inspectFile(
|
||||
filePath: string,
|
||||
kind: DefinitionKind,
|
||||
): InspectedQuerySource[] {
|
||||
const contents = fs.readFileSync(filePath, "utf8");
|
||||
|
||||
const sourceFile = ts.createSourceFile(
|
||||
filePath,
|
||||
contents,
|
||||
ts.ScriptTarget.Latest,
|
||||
true,
|
||||
filePath.endsWith("x") ? ts.ScriptKind.TSX : ts.ScriptKind.TS,
|
||||
);
|
||||
|
||||
const factoryNames = new Set<string>();
|
||||
|
||||
for (const statement of sourceFile.statements) {
|
||||
const namedImports = getDataAppNamedImports(statement);
|
||||
|
||||
if (!namedImports) {
|
||||
continue;
|
||||
}
|
||||
|
||||
for (const item of namedImports) {
|
||||
if (isFactoryImport(item, kind.factory)) {
|
||||
factoryNames.add(item.name.text);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const definitionSources: InspectedQuerySource[] = [];
|
||||
|
||||
const visit = (node: ts.Node) => {
|
||||
if (isFactoryCall(node, factoryNames)) {
|
||||
const declaration = node.parent;
|
||||
const statement = declaration.parent?.parent;
|
||||
const object = definitionObjectArgument(node);
|
||||
|
||||
if (!isDirectNamedDefinition(node, declaration, statement) || !object) {
|
||||
throw new Error(
|
||||
`${filePath}: ${kind.factory} must directly initialize a named exported variable with one object literal.`,
|
||||
);
|
||||
}
|
||||
|
||||
definitionSources.push({
|
||||
exportName: declaration.name.text,
|
||||
filePath,
|
||||
object,
|
||||
sourceFile,
|
||||
});
|
||||
}
|
||||
|
||||
ts.forEachChild(node, visit);
|
||||
};
|
||||
|
||||
visit(sourceFile);
|
||||
|
||||
return definitionSources;
|
||||
}
|
||||
@@ -0,0 +1,135 @@
|
||||
import { createHash } from "node:crypto";
|
||||
|
||||
import { isRecord } from "./guards";
|
||||
|
||||
/** Serializes a query with stable object-key ordering. */
|
||||
export const canonicalJson = (value: unknown): string =>
|
||||
JSON.stringify(canonicalize(value));
|
||||
|
||||
/** Returns the source table and stable hash of an authored query. */
|
||||
export function getQueryFingerprint(query: Record<string, unknown>) {
|
||||
const source = query.source;
|
||||
|
||||
if (
|
||||
!isRecord(source) ||
|
||||
source.type !== "table" ||
|
||||
typeof source.id !== "number" ||
|
||||
!Number.isInteger(source.id) ||
|
||||
source.id <= 0
|
||||
) {
|
||||
throw new Error("Synchronized queries must use a valid table source.");
|
||||
}
|
||||
|
||||
const tableId = source.id;
|
||||
const authoredDsl: Record<string, unknown> = {
|
||||
...query,
|
||||
source: { type: "table" },
|
||||
};
|
||||
|
||||
delete authoredDsl.savedQuestionSourceId;
|
||||
|
||||
const hash = createHash("sha256")
|
||||
.update(canonicalJson(authoredDsl))
|
||||
.digest("hex");
|
||||
|
||||
return { tableId, hash: `v1:sha256:${hash}` };
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns the stable hash of a payload copied into a data app collection.
|
||||
* Generated Lib UUIDs are normalized so a re-read of the same source does not
|
||||
* read as drift.
|
||||
*/
|
||||
export function getPayloadFingerprint(value: unknown): string {
|
||||
const hash = createHash("sha256")
|
||||
.update(getCanonicalQueryJson(value))
|
||||
.digest("hex");
|
||||
|
||||
return `v1:sha256:${hash}`;
|
||||
}
|
||||
|
||||
/** Serializes a resolved query while removing generated Lib-only fields. */
|
||||
export function getCanonicalQueryJson(value: unknown): string {
|
||||
const canonical = canonicalize(value);
|
||||
const generatedIds = new Map<string, number>();
|
||||
|
||||
JSON.stringify(canonical, (key, item) => {
|
||||
if (
|
||||
key === "lib/uuid" &&
|
||||
typeof item === "string" &&
|
||||
!generatedIds.has(item)
|
||||
) {
|
||||
generatedIds.set(item, generatedIds.size);
|
||||
}
|
||||
return item;
|
||||
});
|
||||
|
||||
return JSON.stringify(canonical, (key, item) => {
|
||||
if (key === "lib/uuid") {
|
||||
return undefined;
|
||||
}
|
||||
if (key === "lib/metadata" && item === null) {
|
||||
return undefined;
|
||||
}
|
||||
const generatedId =
|
||||
typeof item === "string" ? generatedIds.get(item) : undefined;
|
||||
return generatedId === undefined
|
||||
? item
|
||||
: `generated-query-id:${generatedId}`;
|
||||
});
|
||||
}
|
||||
|
||||
/** Validates a query value and recursively sorts object keys. */
|
||||
function canonicalize(value: unknown, seen = new Set<object>()): unknown {
|
||||
const type = typeof value;
|
||||
|
||||
if (value === null || type === "string" || type === "boolean") {
|
||||
return value;
|
||||
}
|
||||
|
||||
if (type === "number") {
|
||||
if (!Number.isFinite(value)) {
|
||||
throw new Error("Query definitions cannot contain non-finite numbers.");
|
||||
}
|
||||
|
||||
return value;
|
||||
}
|
||||
|
||||
if (typeof value !== "object") {
|
||||
throw new Error(`Query definitions cannot contain ${type} values.`);
|
||||
}
|
||||
|
||||
if (seen.has(value)) {
|
||||
throw new Error("Query definitions cannot contain circular references.");
|
||||
}
|
||||
|
||||
if (!Array.isArray(value)) {
|
||||
const prototype = Object.getPrototypeOf(value);
|
||||
|
||||
if (prototype !== Object.prototype && prototype !== null) {
|
||||
throw new Error("Query definitions must contain only plain objects.");
|
||||
}
|
||||
}
|
||||
|
||||
seen.add(value);
|
||||
|
||||
const result = Array.isArray(value)
|
||||
? value.map((item) => canonicalize(item, seen))
|
||||
: Object.fromEntries(
|
||||
Object.entries(value)
|
||||
.sort(([first], [second]) => {
|
||||
if (first < second) {
|
||||
return -1;
|
||||
}
|
||||
if (first > second) {
|
||||
return 1;
|
||||
}
|
||||
return 0;
|
||||
})
|
||||
.map(([key, item]) => [key, canonicalize(item, seen)]),
|
||||
);
|
||||
|
||||
seen.delete(value);
|
||||
|
||||
return result;
|
||||
}
|
||||
@@ -0,0 +1,229 @@
|
||||
import { createRequire } from "node:module";
|
||||
import path from "node:path";
|
||||
|
||||
import { build } from "esbuild";
|
||||
|
||||
import {
|
||||
ACTION_DEFINITIONS,
|
||||
type DefinitionKind,
|
||||
QUERY_DEFINITIONS,
|
||||
type QuerySource,
|
||||
findDefinitionSources,
|
||||
} from "./ast/query-source";
|
||||
import { canonicalJson, getQueryFingerprint } from "./canonical";
|
||||
import { isPositiveInteger, isRecord } from "./guards";
|
||||
import { getRelativeDefinitionLocation } from "./messages";
|
||||
import type { DiscoveredAction, DiscoveredQuery } from "./types";
|
||||
|
||||
interface EvaluatedDefinition extends QuerySource {
|
||||
value: Record<string, unknown>;
|
||||
}
|
||||
|
||||
async function evaluateModule(filePath: string) {
|
||||
const result = await build({
|
||||
absWorkingDir: path.dirname(filePath),
|
||||
bundle: true,
|
||||
entryPoints: [filePath],
|
||||
format: "cjs",
|
||||
packages: "external",
|
||||
platform: "node",
|
||||
target: "node20",
|
||||
write: false,
|
||||
logLevel: "silent",
|
||||
});
|
||||
|
||||
const compiled = result.outputFiles[0]?.text;
|
||||
|
||||
if (!compiled) {
|
||||
throw new Error(`Could not evaluate ${filePath}.`);
|
||||
}
|
||||
|
||||
const runtimeModule: { exports: Record<string, unknown> } = { exports: {} };
|
||||
const runtimeRequire = createRequire(filePath);
|
||||
|
||||
new Function("require", "module", "exports", compiled)(
|
||||
runtimeRequire,
|
||||
runtimeModule,
|
||||
runtimeModule.exports,
|
||||
);
|
||||
|
||||
return runtimeModule.exports;
|
||||
}
|
||||
|
||||
function positiveId(value: unknown, location: string, idKey: string) {
|
||||
if (value === undefined) {
|
||||
return undefined;
|
||||
}
|
||||
|
||||
if (!isPositiveInteger(value)) {
|
||||
throw new Error(`${location} has an invalid ${idKey}.`);
|
||||
}
|
||||
|
||||
return value;
|
||||
}
|
||||
|
||||
/**
|
||||
* Evaluates every definition of `kind`, proving each one is deterministic so a
|
||||
* generated ID always describes the same authored value.
|
||||
*/
|
||||
async function evaluateDefinitions(
|
||||
appRoot: string,
|
||||
kind: DefinitionKind,
|
||||
): Promise<EvaluatedDefinition[]> {
|
||||
const sourcesByFile = new Map<string, QuerySource[]>();
|
||||
|
||||
for (const source of findDefinitionSources(appRoot, kind)) {
|
||||
sourcesByFile.set(source.filePath, [
|
||||
...(sourcesByFile.get(source.filePath) ?? []),
|
||||
source,
|
||||
]);
|
||||
}
|
||||
|
||||
const evaluated: EvaluatedDefinition[] = [];
|
||||
|
||||
for (const [filePath, fileSources] of sourcesByFile) {
|
||||
const [first, second] = await Promise.all([
|
||||
evaluateModule(filePath),
|
||||
evaluateModule(filePath),
|
||||
]);
|
||||
|
||||
for (const { exportName } of fileSources) {
|
||||
const value = first[exportName];
|
||||
const repeatedValue = second[exportName];
|
||||
|
||||
const location = getRelativeDefinitionLocation(appRoot, {
|
||||
filePath,
|
||||
exportName,
|
||||
});
|
||||
|
||||
if (!isRecord(value)) {
|
||||
throw new Error(
|
||||
`${location} did not evaluate to ${kind.description} object.`,
|
||||
);
|
||||
}
|
||||
|
||||
let deterministic: boolean;
|
||||
try {
|
||||
deterministic = canonicalJson(value) === canonicalJson(repeatedValue);
|
||||
} catch (error) {
|
||||
throw new Error(
|
||||
`${location} could not be canonicalized: ${String(error)}`,
|
||||
);
|
||||
}
|
||||
|
||||
if (!deterministic) {
|
||||
throw new Error(`${location} is not deterministic.`);
|
||||
}
|
||||
|
||||
evaluated.push({ exportName, filePath, value });
|
||||
}
|
||||
}
|
||||
|
||||
return evaluated;
|
||||
}
|
||||
|
||||
/**
|
||||
* Rejects two definitions claiming the same ID, which would otherwise let one
|
||||
* copied definition silently take ownership of another's Metabase entity.
|
||||
*/
|
||||
function assertUniqueIds(
|
||||
entries: Array<{ id: number | undefined; location: string }>,
|
||||
subject: string,
|
||||
) {
|
||||
const byId = new Map<number, string[]>();
|
||||
|
||||
for (const { id, location } of entries) {
|
||||
if (id) {
|
||||
byId.set(id, [...(byId.get(id) ?? []), location]);
|
||||
}
|
||||
}
|
||||
|
||||
for (const [id, locations] of byId) {
|
||||
if (locations.length > 1) {
|
||||
throw new Error(
|
||||
`${subject} ${id} is referenced by ${locations.join(", ")}. Remove the ID from the copied definition and sync again.`,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export async function discoverQueries(
|
||||
appRoot: string,
|
||||
): Promise<DiscoveredQuery[]> {
|
||||
const definitions = await evaluateDefinitions(appRoot, QUERY_DEFINITIONS);
|
||||
|
||||
const discovered = definitions.map(({ exportName, filePath, value }) => {
|
||||
const { tableId, hash } = getQueryFingerprint(value);
|
||||
|
||||
return {
|
||||
exportName,
|
||||
filePath,
|
||||
query: value,
|
||||
savedQuestionSourceId: positiveId(
|
||||
value[QUERY_DEFINITIONS.idKey],
|
||||
getRelativeDefinitionLocation(appRoot, { filePath, exportName }),
|
||||
QUERY_DEFINITIONS.idKey,
|
||||
),
|
||||
tableId,
|
||||
hash,
|
||||
};
|
||||
});
|
||||
|
||||
assertUniqueIds(
|
||||
discovered.map((query) => ({
|
||||
id: query.savedQuestionSourceId,
|
||||
location: getRelativeDefinitionLocation(appRoot, query),
|
||||
})),
|
||||
"Saved question",
|
||||
);
|
||||
|
||||
return discovered;
|
||||
}
|
||||
|
||||
export async function discoverActions(
|
||||
appRoot: string,
|
||||
): Promise<DiscoveredAction[]> {
|
||||
const definitions = await evaluateDefinitions(appRoot, ACTION_DEFINITIONS);
|
||||
|
||||
const discovered = definitions.map(({ exportName, filePath, value }) => {
|
||||
const location = getRelativeDefinitionLocation(appRoot, {
|
||||
filePath,
|
||||
exportName,
|
||||
});
|
||||
const action = value.action;
|
||||
|
||||
if (!isRecord(action) || !isPositiveInteger(action.id)) {
|
||||
throw new Error(
|
||||
`${location} must reference a generated action, such as \`schema.models.<model>.actions.<action>\`.`,
|
||||
);
|
||||
}
|
||||
|
||||
return {
|
||||
exportName,
|
||||
filePath,
|
||||
copiedActionId: positiveId(
|
||||
value[ACTION_DEFINITIONS.idKey],
|
||||
location,
|
||||
ACTION_DEFINITIONS.idKey,
|
||||
),
|
||||
sourceActionId: action.id,
|
||||
};
|
||||
});
|
||||
|
||||
assertUniqueIds(
|
||||
discovered.map((action) => ({
|
||||
id: action.sourceActionId,
|
||||
location: getRelativeDefinitionLocation(appRoot, action),
|
||||
})),
|
||||
"Action",
|
||||
);
|
||||
assertUniqueIds(
|
||||
discovered.map((action) => ({
|
||||
id: action.copiedActionId,
|
||||
location: getRelativeDefinitionLocation(appRoot, action),
|
||||
})),
|
||||
"Generated action",
|
||||
);
|
||||
|
||||
return discovered;
|
||||
}
|
||||
@@ -0,0 +1,27 @@
|
||||
import fs from "node:fs";
|
||||
import path from "node:path";
|
||||
import { parseEnv } from "node:util";
|
||||
|
||||
import { findEnvRoot } from "../data-app-dev/config/find-env-root";
|
||||
|
||||
function parseEnvFile(filePath: string) {
|
||||
if (!fs.existsSync(filePath)) {
|
||||
return {};
|
||||
}
|
||||
return parseEnv(fs.readFileSync(filePath, "utf8"));
|
||||
}
|
||||
|
||||
export function getResourceSyncCredentials(appRoot: string) {
|
||||
const values = {
|
||||
...parseEnvFile(path.join(findEnvRoot(appRoot), ".env.local")),
|
||||
...process.env,
|
||||
};
|
||||
const metabaseUrl = values.DATA_APP_MB_URL;
|
||||
const apiKey = values.DATA_APP_MB_API_KEY;
|
||||
if (!metabaseUrl || !apiKey) {
|
||||
throw new Error(
|
||||
"DATA_APP_MB_URL and DATA_APP_MB_API_KEY must be set in .env.local.",
|
||||
);
|
||||
}
|
||||
return { metabaseUrl, apiKey };
|
||||
}
|
||||
@@ -0,0 +1,8 @@
|
||||
/** Metabase IDs are always positive integers, so anything else is corrupt input. */
|
||||
export function isPositiveInteger(value: unknown): value is number {
|
||||
return typeof value === "number" && Number.isInteger(value) && value > 0;
|
||||
}
|
||||
|
||||
/** A plain object, as distinct from an array or null. */
|
||||
export const isRecord = (value: unknown): value is Record<string, unknown> =>
|
||||
value !== null && typeof value === "object" && !Array.isArray(value);
|
||||
@@ -0,0 +1,189 @@
|
||||
import fs from "node:fs";
|
||||
import path from "node:path";
|
||||
|
||||
import { isPositiveInteger, isRecord } from "./guards";
|
||||
import type {
|
||||
ActionLockEntry,
|
||||
MetricLockEntry,
|
||||
ModelLockEntry,
|
||||
QueryLockEntry,
|
||||
ResourceLockfile,
|
||||
} from "./types";
|
||||
|
||||
export const RESOURCE_LOCKFILE = "resources_metadata.json";
|
||||
|
||||
const HASH_PATTERN = /^v1:sha256:[0-9a-f]{64}$/;
|
||||
|
||||
function isHash(value: unknown): value is string {
|
||||
return typeof value === "string" && HASH_PATTERN.test(value);
|
||||
}
|
||||
|
||||
function isQueryLockEntry(value: unknown): value is QueryLockEntry {
|
||||
return (
|
||||
isRecord(value) &&
|
||||
isPositiveInteger(value.tableId) &&
|
||||
isPositiveInteger(value.savedQuestionSourceId) &&
|
||||
isHash(value.hash)
|
||||
);
|
||||
}
|
||||
|
||||
function isActionLockEntry(value: unknown): value is ActionLockEntry {
|
||||
return (
|
||||
isRecord(value) &&
|
||||
isPositiveInteger(value.sourceActionId) &&
|
||||
isPositiveInteger(value.copiedActionId) &&
|
||||
isHash(value.hash)
|
||||
);
|
||||
}
|
||||
|
||||
function isModelLockEntry(value: unknown): value is ModelLockEntry {
|
||||
return (
|
||||
isRecord(value) &&
|
||||
isPositiveInteger(value.sourceModelId) &&
|
||||
isPositiveInteger(value.copiedModelId) &&
|
||||
isHash(value.hash) &&
|
||||
Array.isArray(value.actions) &&
|
||||
value.actions.every(isActionLockEntry)
|
||||
);
|
||||
}
|
||||
|
||||
function isMetricLockEntry(value: unknown): value is MetricLockEntry {
|
||||
return (
|
||||
isRecord(value) &&
|
||||
isPositiveInteger(value.sourceMetricId) &&
|
||||
isPositiveInteger(value.copiedMetricId) &&
|
||||
isHash(value.hash)
|
||||
);
|
||||
}
|
||||
|
||||
function parseMetrics(value: unknown): MetricLockEntry[] {
|
||||
if (value === undefined) {
|
||||
return [];
|
||||
}
|
||||
|
||||
if (
|
||||
!Array.isArray(value) ||
|
||||
value.some((entry) => !isMetricLockEntry(entry))
|
||||
) {
|
||||
throw new Error(`${RESOURCE_LOCKFILE} contains an invalid metric entry.`);
|
||||
}
|
||||
|
||||
assertUnique(
|
||||
value.map((entry) => entry.sourceMetricId),
|
||||
"source metric ID",
|
||||
);
|
||||
|
||||
assertUnique(
|
||||
value.map((entry) => entry.copiedMetricId),
|
||||
"copied metric ID",
|
||||
);
|
||||
|
||||
return value;
|
||||
}
|
||||
|
||||
function assertUnique(ids: number[], subject: string) {
|
||||
if (new Set(ids).size !== ids.length) {
|
||||
throw new Error(`${RESOURCE_LOCKFILE} contains a duplicate ${subject}.`);
|
||||
}
|
||||
}
|
||||
|
||||
function parseQueries(value: unknown): QueryLockEntry[] {
|
||||
if (
|
||||
!Array.isArray(value) ||
|
||||
value.some((entry) => !isQueryLockEntry(entry))
|
||||
) {
|
||||
throw new Error(`${RESOURCE_LOCKFILE} contains an invalid entry.`);
|
||||
}
|
||||
|
||||
const queries: QueryLockEntry[] = value;
|
||||
|
||||
assertUnique(
|
||||
queries.map((entry) => entry.savedQuestionSourceId),
|
||||
"saved question ID",
|
||||
);
|
||||
|
||||
return queries;
|
||||
}
|
||||
|
||||
function parseModels(value: unknown): ModelLockEntry[] {
|
||||
if (value === undefined) {
|
||||
return [];
|
||||
}
|
||||
|
||||
if (
|
||||
!Array.isArray(value) ||
|
||||
value.some((entry) => !isModelLockEntry(entry))
|
||||
) {
|
||||
throw new Error(`${RESOURCE_LOCKFILE} contains an invalid model entry.`);
|
||||
}
|
||||
|
||||
const models: ModelLockEntry[] = value;
|
||||
|
||||
assertUnique(
|
||||
models.map((entry) => entry.sourceModelId),
|
||||
"source model ID",
|
||||
);
|
||||
assertUnique(
|
||||
models.map((entry) => entry.copiedModelId),
|
||||
"copied model ID",
|
||||
);
|
||||
assertUnique(
|
||||
models.flatMap((entry) =>
|
||||
entry.actions.map(({ sourceActionId }) => sourceActionId),
|
||||
),
|
||||
"source action ID",
|
||||
);
|
||||
assertUnique(
|
||||
models.flatMap((entry) =>
|
||||
entry.actions.map(({ copiedActionId }) => copiedActionId),
|
||||
),
|
||||
"copied action ID",
|
||||
);
|
||||
|
||||
return models;
|
||||
}
|
||||
|
||||
export function readResourceLockfile(appRoot: string): ResourceLockfile {
|
||||
const lockfilePath = path.join(appRoot, RESOURCE_LOCKFILE);
|
||||
|
||||
if (!fs.existsSync(lockfilePath)) {
|
||||
return { queries: [], models: [], metrics: [] };
|
||||
}
|
||||
|
||||
let value: unknown;
|
||||
try {
|
||||
value = JSON.parse(fs.readFileSync(lockfilePath, "utf8"));
|
||||
} catch (error) {
|
||||
throw new Error(`Could not read ${RESOURCE_LOCKFILE}: ${String(error)}`);
|
||||
}
|
||||
|
||||
if (!isRecord(value)) {
|
||||
throw new Error(`${RESOURCE_LOCKFILE} contains an invalid entry.`);
|
||||
}
|
||||
|
||||
if (
|
||||
value.collectionId !== undefined &&
|
||||
!isPositiveInteger(value.collectionId)
|
||||
) {
|
||||
throw new Error(`${RESOURCE_LOCKFILE} contains an invalid collection ID.`);
|
||||
}
|
||||
|
||||
return {
|
||||
...(value.collectionId === undefined
|
||||
? undefined
|
||||
: { collectionId: value.collectionId }),
|
||||
queries: parseQueries(value.queries ?? []),
|
||||
models: parseModels(value.models),
|
||||
metrics: parseMetrics(value.metrics),
|
||||
};
|
||||
}
|
||||
|
||||
export function writeResourceLockfile(
|
||||
appRoot: string,
|
||||
lockfile: ResourceLockfile,
|
||||
) {
|
||||
fs.writeFileSync(
|
||||
path.join(appRoot, RESOURCE_LOCKFILE),
|
||||
`${JSON.stringify(lockfile, null, 2)}\n`,
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,12 @@
|
||||
import path from "node:path";
|
||||
|
||||
export function getErrorMessage(error: unknown) {
|
||||
return error instanceof Error ? error.message : String(error);
|
||||
}
|
||||
|
||||
export function getRelativeDefinitionLocation(
|
||||
appRoot: string,
|
||||
{ filePath, exportName }: { filePath: string; exportName: string },
|
||||
) {
|
||||
return `${path.relative(appRoot, filePath)}:${exportName}`;
|
||||
}
|
||||
@@ -0,0 +1,268 @@
|
||||
import type {
|
||||
DataAppMetadata,
|
||||
DataAppMetric,
|
||||
MetabaseAction,
|
||||
MetabaseCard,
|
||||
} from "./types";
|
||||
|
||||
export class MetabaseApiError extends Error {
|
||||
constructor(
|
||||
public readonly status: number,
|
||||
message: string,
|
||||
) {
|
||||
super(message);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolves to `null` on a confirmed 404, so a caller can treat "gone" as
|
||||
* recoverable. Any other failure still throws: only a 404 proves the entity is
|
||||
* absent rather than unreachable.
|
||||
*/
|
||||
export const orNullOn404 = async <T>(
|
||||
request: Promise<T>,
|
||||
): Promise<T | null> => {
|
||||
try {
|
||||
return await request;
|
||||
} catch (error) {
|
||||
if (error instanceof MetabaseApiError && error.status === 404) {
|
||||
return null;
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
};
|
||||
|
||||
export class MetabaseClient {
|
||||
constructor(
|
||||
private readonly baseUrl: string,
|
||||
private readonly apiKey: string,
|
||||
) {}
|
||||
|
||||
private async request<T>(pathname: string, init?: RequestInit): Promise<T> {
|
||||
const baseUrl = new URL(this.baseUrl);
|
||||
baseUrl.pathname = `${baseUrl.pathname.replace(/\/?$/, "/")}api/${pathname.replace(/^\//, "")}`;
|
||||
const requestUrl = baseUrl;
|
||||
const method = (init?.method ?? "GET").toUpperCase();
|
||||
const response = await fetch(requestUrl, {
|
||||
...init,
|
||||
signal: init?.signal ?? AbortSignal.timeout(60_000),
|
||||
headers: {
|
||||
"content-type": "application/json",
|
||||
"x-api-key": this.apiKey,
|
||||
...init?.headers,
|
||||
},
|
||||
});
|
||||
if (!response.ok) {
|
||||
const message = await response.text();
|
||||
throw new MetabaseApiError(
|
||||
response.status,
|
||||
// eslint-disable-next-line metabase/no-literal-metabase-strings -- CLI protocol error names the remote service.
|
||||
`Metabase returned ${response.status} for ${method} ${requestUrl}: ${message}`,
|
||||
);
|
||||
}
|
||||
if (response.status === 204) {
|
||||
// The generic return type is controlled by the caller for no-content requests.
|
||||
return undefined as T;
|
||||
}
|
||||
const body: unknown = await response.json();
|
||||
// The endpoint-specific methods define the trusted API response contract.
|
||||
return body as T;
|
||||
}
|
||||
|
||||
ensureDraft(slug: string) {
|
||||
return this.request<DataAppMetadata>(
|
||||
`apps/${encodeURIComponent(slug)}/draft`,
|
||||
{ method: "POST" },
|
||||
);
|
||||
}
|
||||
|
||||
updateTableDependencies(slug: string, tableIds: number[]) {
|
||||
return this.request<DataAppMetadata>(
|
||||
`apps/${encodeURIComponent(slug)}/table-dependencies`,
|
||||
{
|
||||
method: "PUT",
|
||||
body: JSON.stringify({ table_ids: tableIds }),
|
||||
},
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolves the tables that already-saved queries read. Only the server sees an
|
||||
* implicit join: the table a foreign key reaches appears nowhere in the query,
|
||||
* so walking its JSON would leave that table out of the app's dependencies.
|
||||
*/
|
||||
async resolveTableDependencies(
|
||||
slug: string,
|
||||
datasetQueries: ReadonlyArray<Record<string, unknown>>,
|
||||
): Promise<number[]> {
|
||||
if (datasetQueries.length === 0) {
|
||||
return [];
|
||||
}
|
||||
|
||||
const { table_ids } = await this.request<{ table_ids: number[] }>(
|
||||
`apps/${encodeURIComponent(slug)}/query-table-dependencies`,
|
||||
{
|
||||
method: "POST",
|
||||
body: JSON.stringify({ dataset_queries: datasetQueries }),
|
||||
},
|
||||
);
|
||||
|
||||
return table_ids;
|
||||
}
|
||||
|
||||
async resolveQuery(slug: string, query: Record<string, unknown>) {
|
||||
const resolved = await this.request<{
|
||||
database_id: number;
|
||||
dataset_query: Record<string, unknown>;
|
||||
table_ids: number[];
|
||||
metrics?: DataAppMetric[];
|
||||
}>(`apps/${encodeURIComponent(slug)}/query`, {
|
||||
method: "POST",
|
||||
body: JSON.stringify({ stages: [query] }),
|
||||
});
|
||||
return { ...resolved, metrics: resolved.metrics ?? [] };
|
||||
}
|
||||
|
||||
getCard(id: number) {
|
||||
return this.request<MetabaseCard>(`card/${id}`);
|
||||
}
|
||||
|
||||
createCard(input: {
|
||||
name: string;
|
||||
collectionId: number;
|
||||
datasetQuery: Record<string, unknown>;
|
||||
}) {
|
||||
return this.request<MetabaseCard>("card", {
|
||||
method: "POST",
|
||||
body: JSON.stringify({
|
||||
name: input.name,
|
||||
type: "question",
|
||||
dataset_query: input.datasetQuery,
|
||||
display: "table",
|
||||
visualization_settings: {},
|
||||
collection_id: input.collectionId,
|
||||
}),
|
||||
});
|
||||
}
|
||||
|
||||
updateCard(
|
||||
id: number,
|
||||
input: {
|
||||
name: string;
|
||||
collectionId: number;
|
||||
datasetQuery: Record<string, unknown>;
|
||||
},
|
||||
) {
|
||||
return this.request<MetabaseCard>(`card/${id}`, {
|
||||
method: "PUT",
|
||||
body: JSON.stringify({
|
||||
name: input.name,
|
||||
type: "question",
|
||||
dataset_query: input.datasetQuery,
|
||||
collection_id: input.collectionId,
|
||||
archived: false,
|
||||
}),
|
||||
});
|
||||
}
|
||||
|
||||
moveCardToCollection(id: number, collectionId: number) {
|
||||
return this.request<MetabaseCard>(`card/${id}`, {
|
||||
method: "PUT",
|
||||
body: JSON.stringify({ collection_id: collectionId }),
|
||||
});
|
||||
}
|
||||
|
||||
deleteCard(id: number) {
|
||||
return this.request<void>(`card/${id}`, { method: "DELETE" });
|
||||
}
|
||||
|
||||
createModel(input: ModelInput) {
|
||||
return this.request<MetabaseCard>("card", {
|
||||
method: "POST",
|
||||
body: JSON.stringify(modelBody(input)),
|
||||
});
|
||||
}
|
||||
|
||||
createMetric(input: MetricInput) {
|
||||
return this.request<MetabaseCard>("card", {
|
||||
method: "POST",
|
||||
body: JSON.stringify(metricBody(input)),
|
||||
});
|
||||
}
|
||||
|
||||
updateMetric(id: number, input: MetricInput) {
|
||||
return this.request<MetabaseCard>(`card/${id}`, {
|
||||
method: "PUT",
|
||||
body: JSON.stringify(metricBody(input)),
|
||||
});
|
||||
}
|
||||
|
||||
updateModel(id: number, input: ModelInput) {
|
||||
return this.request<MetabaseCard>(`card/${id}`, {
|
||||
method: "PUT",
|
||||
body: JSON.stringify(modelBody(input)),
|
||||
});
|
||||
}
|
||||
|
||||
getAction(id: number) {
|
||||
return this.request<MetabaseAction>(`action/${id}`);
|
||||
}
|
||||
|
||||
createAction(body: Record<string, unknown>) {
|
||||
return this.request<MetabaseAction>("action", {
|
||||
method: "POST",
|
||||
body: JSON.stringify(body),
|
||||
});
|
||||
}
|
||||
|
||||
updateAction(id: number, body: Record<string, unknown>) {
|
||||
return this.request<MetabaseAction>(`action/${id}`, {
|
||||
method: "PUT",
|
||||
body: JSON.stringify(body),
|
||||
});
|
||||
}
|
||||
|
||||
deleteAction(id: number) {
|
||||
return this.request<void>(`action/${id}`, { method: "DELETE" });
|
||||
}
|
||||
}
|
||||
|
||||
interface ModelInput {
|
||||
name: string;
|
||||
collectionId: number;
|
||||
datasetQuery: Record<string, unknown>;
|
||||
display: string;
|
||||
visualizationSettings: Record<string, unknown>;
|
||||
description: string | null;
|
||||
}
|
||||
|
||||
interface MetricInput {
|
||||
name: string;
|
||||
collectionId: number;
|
||||
datasetQuery: Record<string, unknown>;
|
||||
display: string;
|
||||
visualizationSettings: Record<string, unknown>;
|
||||
description: string | null;
|
||||
}
|
||||
|
||||
const modelBody = (input: ModelInput) => ({
|
||||
name: input.name,
|
||||
type: "model",
|
||||
archived: false,
|
||||
dataset_query: input.datasetQuery,
|
||||
display: input.display,
|
||||
visualization_settings: input.visualizationSettings,
|
||||
description: input.description,
|
||||
collection_id: input.collectionId,
|
||||
});
|
||||
|
||||
const metricBody = (input: MetricInput) => ({
|
||||
name: input.name,
|
||||
type: "metric",
|
||||
archived: false,
|
||||
dataset_query: input.datasetQuery,
|
||||
display: input.display,
|
||||
visualization_settings: input.visualizationSettings,
|
||||
description: input.description,
|
||||
collection_id: input.collectionId,
|
||||
});
|
||||
@@ -0,0 +1,309 @@
|
||||
import { getPayloadFingerprint } from "./canonical";
|
||||
import { isPositiveInteger, isRecord } from "./guards";
|
||||
import { writeResourceLockfile } from "./lockfile";
|
||||
import type { MetabaseClient } from "./metabase-client";
|
||||
import { orNullOn404 } from "./metabase-client";
|
||||
import { getSavedCardSourceIds } from "./saved-card-sources";
|
||||
import type { DataAppMetric, ResourceLockfile } from "./types";
|
||||
|
||||
interface ResolvedQuery {
|
||||
dataset_query: Record<string, unknown>;
|
||||
metrics: DataAppMetric[];
|
||||
}
|
||||
|
||||
const metricInput = (metric: DataAppMetric, collectionId: number) => ({
|
||||
name: metric.name,
|
||||
collectionId,
|
||||
datasetQuery: metric.dataset_query,
|
||||
display: metric.display ?? "table",
|
||||
visualizationSettings: metric.visualization_settings ?? {},
|
||||
description: metric.description ?? null,
|
||||
});
|
||||
|
||||
type MetricClause = readonly [
|
||||
type: "metric",
|
||||
options: unknown,
|
||||
metricId: number,
|
||||
...rest: unknown[],
|
||||
];
|
||||
|
||||
// A MBQL metric clause has the form ["metric", options, metricId].
|
||||
const METRIC_ID_INDEX = 2;
|
||||
|
||||
const isMetricCard = (value: { type: string }): value is DataAppMetric =>
|
||||
value.type === "metric";
|
||||
|
||||
const isMetricClause = (clause: unknown): clause is MetricClause =>
|
||||
Array.isArray(clause) &&
|
||||
clause[0] === "metric" &&
|
||||
typeof clause[METRIC_ID_INDEX] === "number";
|
||||
|
||||
function getMetricClauseIds(
|
||||
clause: unknown,
|
||||
ids = new Set<number>(),
|
||||
): Set<number> {
|
||||
if (Array.isArray(clause)) {
|
||||
if (isMetricClause(clause)) {
|
||||
ids.add(clause[METRIC_ID_INDEX]);
|
||||
|
||||
return ids;
|
||||
}
|
||||
|
||||
clause.forEach((item) => getMetricClauseIds(item, ids));
|
||||
} else if (isRecord(clause)) {
|
||||
Object.values(clause).forEach((item) => getMetricClauseIds(item, ids));
|
||||
}
|
||||
|
||||
return ids;
|
||||
}
|
||||
|
||||
/**
|
||||
* Rewrite a metric clause to
|
||||
* swap the original metric id with the copied metric id.
|
||||
*/
|
||||
function rewriteMetricClause(
|
||||
[type, options, metricId, ...rest]: MetricClause,
|
||||
copiedMetricIdBySourceMetricId: Map<number, number>,
|
||||
): MetricClause {
|
||||
const copiedMetricId = copiedMetricIdBySourceMetricId.get(metricId);
|
||||
|
||||
if (copiedMetricId === undefined) {
|
||||
throw new Error(`Missing copied metric for metric ${metricId}.`);
|
||||
}
|
||||
|
||||
return [type, options, copiedMetricId, ...rest];
|
||||
}
|
||||
|
||||
function rewriteMetricClauses(
|
||||
clause: unknown,
|
||||
copiedMetricIdBySourceMetricId: Map<number, number>,
|
||||
): unknown {
|
||||
if (Array.isArray(clause)) {
|
||||
if (isMetricClause(clause)) {
|
||||
return rewriteMetricClause(clause, copiedMetricIdBySourceMetricId);
|
||||
}
|
||||
|
||||
return clause.map((item) =>
|
||||
rewriteMetricClauses(item, copiedMetricIdBySourceMetricId),
|
||||
);
|
||||
}
|
||||
|
||||
if (clause === null || typeof clause !== "object") {
|
||||
return clause;
|
||||
}
|
||||
|
||||
return Object.fromEntries(
|
||||
Object.entries(clause).map(([key, item]) => [
|
||||
key,
|
||||
rewriteMetricClauses(item, copiedMetricIdBySourceMetricId),
|
||||
]),
|
||||
);
|
||||
}
|
||||
|
||||
function removeMetricLockEntry(
|
||||
appRoot: string,
|
||||
lockfile: ResourceLockfile,
|
||||
sourceMetricId: number,
|
||||
) {
|
||||
const index = lockfile.metrics.findIndex(
|
||||
(entry) => entry.sourceMetricId === sourceMetricId,
|
||||
);
|
||||
|
||||
if (index >= 0) {
|
||||
lockfile.metrics.splice(index, 1);
|
||||
writeResourceLockfile(appRoot, lockfile);
|
||||
}
|
||||
}
|
||||
|
||||
type ReconcilerContext = {
|
||||
appRoot: string;
|
||||
collectionId: number;
|
||||
lockfile: ResourceLockfile;
|
||||
log: (message: string) => void;
|
||||
};
|
||||
|
||||
export async function reconcileRemovedMetrics({
|
||||
appRoot,
|
||||
collectionId,
|
||||
previousEntries,
|
||||
liveMetricIds,
|
||||
lockfile,
|
||||
client,
|
||||
log,
|
||||
}: ReconcilerContext & {
|
||||
liveMetricIds: Set<number>;
|
||||
previousEntries: ResourceLockfile["metrics"];
|
||||
|
||||
client: Pick<MetabaseClient, "getCard" | "deleteCard">;
|
||||
}) {
|
||||
for (const entry of previousEntries) {
|
||||
if (liveMetricIds.has(entry.sourceMetricId)) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const copiedMetric = await orNullOn404(
|
||||
client.getCard(entry.copiedMetricId),
|
||||
);
|
||||
|
||||
if (copiedMetric?.archived === true) {
|
||||
// archived metrics are moved to the trash collection, so we cannot tell
|
||||
// whether the copied metric were moved elsewhere before being trashed.
|
||||
// leave it in trash and stop tracking it instead of permanently deleting it.
|
||||
log(`left in the trash: metric ${copiedMetric.id}`);
|
||||
} else if (copiedMetric) {
|
||||
if (
|
||||
!isMetricCard(copiedMetric) ||
|
||||
copiedMetric.collection_id !== collectionId
|
||||
) {
|
||||
throw new Error(
|
||||
`Metric ${copiedMetric.id} is no longer in data app collection ${collectionId}. Move it back, then run sync-resources again.`,
|
||||
);
|
||||
}
|
||||
|
||||
await client.deleteCard(copiedMetric.id);
|
||||
|
||||
log(`deleted metric: ${copiedMetric.id}`);
|
||||
}
|
||||
|
||||
removeMetricLockEntry(appRoot, lockfile, entry.sourceMetricId);
|
||||
}
|
||||
}
|
||||
|
||||
export async function reconcileMetrics({
|
||||
appRoot,
|
||||
collectionId,
|
||||
resolvedQueries,
|
||||
lockfile,
|
||||
client,
|
||||
log,
|
||||
}: ReconcilerContext & {
|
||||
resolvedQueries: ResolvedQuery[];
|
||||
|
||||
client: Pick<
|
||||
MetabaseClient,
|
||||
"getCard" | "createMetric" | "updateMetric" | "deleteCard"
|
||||
>;
|
||||
}) {
|
||||
const previousEntries = [...lockfile.metrics];
|
||||
|
||||
const metrics = [
|
||||
...new Map(
|
||||
resolvedQueries.flatMap(({ metrics }) =>
|
||||
metrics.map((metric) => [metric.id, metric]),
|
||||
),
|
||||
).values(),
|
||||
];
|
||||
|
||||
const liveMetricIds = new Set(metrics.map((metric) => metric.id));
|
||||
|
||||
const missingMetricIds = [
|
||||
...new Set(
|
||||
resolvedQueries.flatMap(({ dataset_query }) => [
|
||||
...getMetricClauseIds(dataset_query),
|
||||
]),
|
||||
),
|
||||
]
|
||||
.filter((metricId) => !liveMetricIds.has(metricId))
|
||||
.sort((a, b) => a - b);
|
||||
|
||||
if (missingMetricIds.length > 0) {
|
||||
throw new Error(
|
||||
`These metrics are missing and cannot be reconciled: ${missingMetricIds.join(", ")}.`,
|
||||
);
|
||||
}
|
||||
|
||||
const metricsWithSavedCardSources = metrics
|
||||
.filter((metric) => getSavedCardSourceIds(metric.dataset_query).size > 0)
|
||||
.map((metric) => metric.id)
|
||||
.sort((a, b) => a - b);
|
||||
|
||||
if (metricsWithSavedCardSources.length > 0) {
|
||||
throw new Error(
|
||||
`Metrics with saved-card joins cannot be synchronized: ${metricsWithSavedCardSources.join(", ")}.`,
|
||||
);
|
||||
}
|
||||
|
||||
const copiedMetricIdBySourceMetricId = new Map<number, number>();
|
||||
|
||||
for (const metric of metrics) {
|
||||
const input = metricInput(metric, collectionId);
|
||||
const hash = getPayloadFingerprint(input);
|
||||
|
||||
const entry = lockfile.metrics.find(
|
||||
({ sourceMetricId }) => sourceMetricId === metric.id,
|
||||
);
|
||||
|
||||
const copiedMetric = entry
|
||||
? await orNullOn404(client.getCard(entry.copiedMetricId))
|
||||
: null;
|
||||
|
||||
if (copiedMetric) {
|
||||
// track metrics that are moved out of the data app collection, but allow
|
||||
// archived metrics (moved to the trash collection) to be restored
|
||||
const isOutsideDataAppCollection =
|
||||
copiedMetric.collection_id !== collectionId &&
|
||||
copiedMetric.archived !== true;
|
||||
|
||||
if (!isMetricCard(copiedMetric) || isOutsideDataAppCollection) {
|
||||
throw new Error(
|
||||
`Metric ${copiedMetric.id} is not managed by data app collection ${collectionId}.`,
|
||||
);
|
||||
}
|
||||
|
||||
const changed =
|
||||
copiedMetric.archived === true ||
|
||||
getPayloadFingerprint(metricInput(copiedMetric, collectionId)) !== hash;
|
||||
|
||||
if (changed) {
|
||||
await client.updateMetric(copiedMetric.id, input);
|
||||
}
|
||||
|
||||
copiedMetricIdBySourceMetricId.set(metric.id, copiedMetric.id);
|
||||
|
||||
if (entry) {
|
||||
entry.hash = hash;
|
||||
}
|
||||
|
||||
log(
|
||||
`${changed ? "updated" : "unchanged"} metric: ${metric.id} -> ${copiedMetric.id}`,
|
||||
);
|
||||
} else {
|
||||
const created = await client.createMetric(input);
|
||||
|
||||
if (!isPositiveInteger(created.id)) {
|
||||
throw new Error("The copied metric does not have a valid ID.");
|
||||
}
|
||||
|
||||
copiedMetricIdBySourceMetricId.set(metric.id, created.id);
|
||||
|
||||
if (entry) {
|
||||
Object.assign(entry, { copiedMetricId: created.id, hash });
|
||||
} else {
|
||||
lockfile.metrics.push({
|
||||
hash,
|
||||
sourceMetricId: metric.id,
|
||||
copiedMetricId: created.id,
|
||||
});
|
||||
}
|
||||
|
||||
log(`copied metric: ${metric.id} -> ${created.id}`);
|
||||
}
|
||||
|
||||
writeResourceLockfile(appRoot, lockfile);
|
||||
}
|
||||
|
||||
for (const resolved of resolvedQueries) {
|
||||
const rewrittenMetricQuery = rewriteMetricClauses(
|
||||
resolved.dataset_query,
|
||||
copiedMetricIdBySourceMetricId,
|
||||
);
|
||||
|
||||
if (!isRecord(rewrittenMetricQuery)) {
|
||||
throw new Error("Rewritten metric query must be an object.");
|
||||
}
|
||||
|
||||
resolved.dataset_query = rewrittenMetricQuery;
|
||||
}
|
||||
|
||||
return { previousEntries, liveMetricIds };
|
||||
}
|
||||
@@ -0,0 +1,453 @@
|
||||
import { ACTION_DEFINITIONS, injectGeneratedId } from "./ast/query-source";
|
||||
import { getPayloadFingerprint } from "./canonical";
|
||||
import { isPositiveInteger } from "./guards";
|
||||
import { writeResourceLockfile } from "./lockfile";
|
||||
import { getErrorMessage, getRelativeDefinitionLocation } from "./messages";
|
||||
import type { MetabaseClient } from "./metabase-client";
|
||||
import { orNullOn404 } from "./metabase-client";
|
||||
import { getSavedCardSourceIds } from "./saved-card-sources";
|
||||
import type {
|
||||
ActionLockEntry,
|
||||
DiscoveredAction,
|
||||
MetabaseAction,
|
||||
MetabaseCard,
|
||||
ModelLockEntry,
|
||||
ResourceLockfile,
|
||||
} from "./types";
|
||||
|
||||
export interface ReconcileModelsOptions {
|
||||
appRoot: string;
|
||||
slug: string;
|
||||
collectionId: number;
|
||||
actions: DiscoveredAction[];
|
||||
lockfile: ResourceLockfile;
|
||||
client: MetabaseClient;
|
||||
log: (message: string) => void;
|
||||
}
|
||||
|
||||
interface ResolvedAction {
|
||||
action: DiscoveredAction;
|
||||
source: MetabaseAction;
|
||||
}
|
||||
|
||||
interface ModelContext {
|
||||
appRoot: string;
|
||||
collectionId: number;
|
||||
client: MetabaseClient;
|
||||
lockfile: ResourceLockfile;
|
||||
log: (message: string) => void;
|
||||
}
|
||||
|
||||
function pickDefined(source: Record<string, unknown>) {
|
||||
return Object.fromEntries(
|
||||
Object.entries(source).filter(([, value]) => value !== undefined),
|
||||
);
|
||||
}
|
||||
|
||||
/** Also the fingerprint input, so drift is measured over exactly what gets copied. */
|
||||
function modelCopyInput(source: MetabaseCard) {
|
||||
return {
|
||||
name: source.name,
|
||||
datasetQuery: source.dataset_query,
|
||||
display: source.display ?? "table",
|
||||
visualizationSettings: source.visualization_settings ?? {},
|
||||
description: source.description ?? null,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Implicit actions omit `parameters` and `parameter_mappings`: Metabase derives
|
||||
* those from the model's fields on every read.
|
||||
*/
|
||||
function actionCopyFields(source: MetabaseAction) {
|
||||
const common = {
|
||||
name: source.name,
|
||||
type: source.type,
|
||||
description: source.description,
|
||||
visualization_settings: source.visualization_settings,
|
||||
};
|
||||
|
||||
if (source.type === "implicit") {
|
||||
return pickDefined({ ...common, kind: source.kind });
|
||||
}
|
||||
|
||||
if (source.type === "query") {
|
||||
return pickDefined({
|
||||
...common,
|
||||
parameters: source.parameters,
|
||||
parameter_mappings: source.parameter_mappings,
|
||||
dataset_query: source.dataset_query,
|
||||
database_id: source.database_id,
|
||||
});
|
||||
}
|
||||
return pickDefined(common);
|
||||
}
|
||||
|
||||
async function resolveActions(
|
||||
appRoot: string,
|
||||
actions: DiscoveredAction[],
|
||||
client: MetabaseClient,
|
||||
): Promise<ResolvedAction[]> {
|
||||
return Promise.all(
|
||||
actions.map(async (action) => {
|
||||
let source: MetabaseAction;
|
||||
try {
|
||||
// `GET /api/action/:id` filters archived actions out, so an archived
|
||||
// source surfaces here as a 404 rather than a readable payload.
|
||||
source = await client.getAction(action.sourceActionId);
|
||||
} catch (error) {
|
||||
throw new Error(
|
||||
`Could not read action ${action.sourceActionId} for ${getRelativeDefinitionLocation(appRoot, action)}: ${getErrorMessage(error)}`,
|
||||
);
|
||||
}
|
||||
|
||||
if (!isPositiveInteger(source.model_id)) {
|
||||
throw new Error(
|
||||
`${getRelativeDefinitionLocation(appRoot, action)} references action ${action.sourceActionId}, which does not belong to a model.`,
|
||||
);
|
||||
}
|
||||
|
||||
return { action, source };
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
async function fetchSourceModels(
|
||||
appRoot: string,
|
||||
desired: Map<number, ResolvedAction[]>,
|
||||
client: MetabaseClient,
|
||||
): Promise<Map<number, MetabaseCard>> {
|
||||
const models = await Promise.all(
|
||||
[...desired].map(async ([id, actions]) => {
|
||||
const location = getRelativeDefinitionLocation(
|
||||
appRoot,
|
||||
actions[0].action,
|
||||
);
|
||||
|
||||
let card: MetabaseCard;
|
||||
try {
|
||||
card = await client.getCard(id);
|
||||
} catch (error) {
|
||||
throw new Error(
|
||||
`Could not read model ${id} for ${location}: ${getErrorMessage(error)}`,
|
||||
);
|
||||
}
|
||||
|
||||
if (card.type !== "model") {
|
||||
throw new Error(
|
||||
`${location} references an action on card ${id}, which is not a model.`,
|
||||
);
|
||||
}
|
||||
|
||||
// A trashed action reads as a 404 and stops the run; a trashed model is
|
||||
// still readable, so refuse it here rather than copying from something
|
||||
// its author deleted.
|
||||
if (card.archived === true) {
|
||||
throw new Error(
|
||||
`${location} references an action on model ${id}, which is in the trash. Restore the model or remove the declaration, then run sync-resources again.`,
|
||||
);
|
||||
}
|
||||
|
||||
if (getSavedCardSourceIds(card.dataset_query).size > 0) {
|
||||
throw new Error(
|
||||
`Model ${id} depends on saved questions and cannot be synchronized. Use a model without saved question sources or joins.`,
|
||||
);
|
||||
}
|
||||
|
||||
return [id, card] as const;
|
||||
}),
|
||||
);
|
||||
|
||||
return new Map(models);
|
||||
}
|
||||
|
||||
function assertOwnedActionCopy(
|
||||
action: MetabaseAction,
|
||||
sourceActionId: number,
|
||||
copiedModelId: number,
|
||||
) {
|
||||
if (action.model_id !== copiedModelId) {
|
||||
throw new Error(
|
||||
`Action ${action.id} is the copy of action ${sourceActionId} but no longer hangs off copied model ${copiedModelId}, so it was left untouched. Move it back or delete it manually, then run sync-resources again.`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
function assertOwnedModelCopy(card: MetabaseCard, collectionId: number) {
|
||||
if (card.type !== "model") {
|
||||
throw new Error(
|
||||
`Card ${card.id} belongs to a synchronized model but is no longer a model, so it was left untouched. Change it back to a model in data app collection ${collectionId} or delete it manually, then run sync-resources again.`,
|
||||
);
|
||||
}
|
||||
|
||||
// A trashed copy reports the Trash as its collection, so its real one says
|
||||
// nothing here. Only the restore path reaches this with one, and it puts both
|
||||
// the card and its collection back.
|
||||
if (card.archived !== true && card.collection_id !== collectionId) {
|
||||
throw new Error(
|
||||
`Card ${card.id} belongs to a synchronized model but is no longer in the data app collection, so it was left untouched. Move card ${card.id} back to data app collection ${collectionId} or delete it manually, then run sync-resources again.`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
async function reconcileModelActions(
|
||||
context: ModelContext,
|
||||
entry: ModelLockEntry,
|
||||
desiredActions: ResolvedAction[],
|
||||
) {
|
||||
const { appRoot, client, lockfile, log } = context;
|
||||
const desiredIds = new Set(desiredActions.map(({ source }) => source.id));
|
||||
|
||||
for (const { action, source } of desiredActions) {
|
||||
const fields = actionCopyFields(source);
|
||||
const hash = getPayloadFingerprint(fields);
|
||||
let mapping: ActionLockEntry | undefined = entry.actions.find(
|
||||
({ sourceActionId }) => sourceActionId === source.id,
|
||||
);
|
||||
const copiedAction = mapping
|
||||
? await orNullOn404(client.getAction(mapping.copiedActionId))
|
||||
: null;
|
||||
|
||||
if (mapping && copiedAction) {
|
||||
assertOwnedActionCopy(copiedAction, source.id, entry.copiedModelId);
|
||||
}
|
||||
|
||||
if (mapping && !copiedAction) {
|
||||
entry.actions.splice(entry.actions.indexOf(mapping), 1);
|
||||
mapping = undefined;
|
||||
}
|
||||
|
||||
if (!mapping || !copiedAction) {
|
||||
const created = await client.createAction({
|
||||
...fields,
|
||||
model_id: entry.copiedModelId,
|
||||
});
|
||||
|
||||
if (!isPositiveInteger(created.id)) {
|
||||
throw new Error("The Action API did not return a valid action ID.");
|
||||
}
|
||||
|
||||
entry.actions.push({
|
||||
sourceActionId: source.id,
|
||||
copiedActionId: created.id,
|
||||
hash,
|
||||
});
|
||||
|
||||
injectGeneratedId(action, ACTION_DEFINITIONS, created.id);
|
||||
writeResourceLockfile(appRoot, lockfile);
|
||||
log(`copied action: action ${source.id} -> action ${created.id}`);
|
||||
|
||||
continue;
|
||||
}
|
||||
|
||||
// Fingerprint the copy rather than trusting the lockfile: that also catches
|
||||
// a copy edited directly in Metabase, which the source hash cannot see.
|
||||
if (getPayloadFingerprint(actionCopyFields(copiedAction)) !== hash) {
|
||||
await client.updateAction(mapping.copiedActionId, fields);
|
||||
|
||||
mapping.hash = hash;
|
||||
|
||||
writeResourceLockfile(appRoot, lockfile);
|
||||
log(`updated action: action ${mapping.copiedActionId}`);
|
||||
}
|
||||
|
||||
if (action.copiedActionId !== mapping.copiedActionId) {
|
||||
injectGeneratedId(action, ACTION_DEFINITIONS, mapping.copiedActionId);
|
||||
log(
|
||||
`restored action ID: ${action.exportName} -> action ${mapping.copiedActionId}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
for (const mapping of [...entry.actions]) {
|
||||
if (desiredIds.has(mapping.sourceActionId)) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const copiedAction = await orNullOn404(
|
||||
client.getAction(mapping.copiedActionId),
|
||||
);
|
||||
|
||||
if (copiedAction && copiedAction.model_id !== entry.copiedModelId) {
|
||||
throw new Error(
|
||||
`Action ${mapping.copiedActionId} belongs to a removed declaration but no longer hangs off copied model ${entry.copiedModelId}, so it was left untouched. Move it back or delete it manually, then run sync-resources again.`,
|
||||
);
|
||||
}
|
||||
|
||||
if (copiedAction) {
|
||||
await client.deleteAction(mapping.copiedActionId);
|
||||
log(`deleted action: action ${mapping.copiedActionId}`);
|
||||
}
|
||||
|
||||
entry.actions.splice(entry.actions.indexOf(mapping), 1);
|
||||
writeResourceLockfile(appRoot, lockfile);
|
||||
}
|
||||
}
|
||||
|
||||
async function reconcileModel(
|
||||
context: ModelContext,
|
||||
sourceModel: MetabaseCard,
|
||||
desiredActions: ResolvedAction[],
|
||||
) {
|
||||
const { appRoot, collectionId, client, lockfile, log } = context;
|
||||
const hash = getPayloadFingerprint(modelCopyInput(sourceModel));
|
||||
const index = lockfile.models.findIndex(
|
||||
({ sourceModelId }) => sourceModelId === sourceModel.id,
|
||||
);
|
||||
const previous = index >= 0 ? lockfile.models[index] : undefined;
|
||||
const copy = previous
|
||||
? await orNullOn404(client.getCard(previous.copiedModelId))
|
||||
: null;
|
||||
|
||||
if (previous && copy) {
|
||||
assertOwnedModelCopy(copy, collectionId);
|
||||
|
||||
// `archived` is deliberately not part of the fingerprint: it describes the
|
||||
// copy's state, not the payload, and folding it in would invalidate every
|
||||
// lockfile written before this.
|
||||
if (
|
||||
copy.archived === true ||
|
||||
getPayloadFingerprint(modelCopyInput(copy)) !== hash
|
||||
) {
|
||||
await client.updateModel(previous.copiedModelId, {
|
||||
...modelCopyInput(sourceModel),
|
||||
collectionId,
|
||||
});
|
||||
previous.hash = hash;
|
||||
writeResourceLockfile(appRoot, lockfile);
|
||||
log(`updated model: card ${previous.copiedModelId}`);
|
||||
}
|
||||
|
||||
await reconcileModelActions(context, previous, desiredActions);
|
||||
|
||||
return;
|
||||
}
|
||||
|
||||
if (previous) {
|
||||
lockfile.models.splice(index, 1);
|
||||
}
|
||||
|
||||
const created = await client.createModel({
|
||||
...modelCopyInput(sourceModel),
|
||||
collectionId,
|
||||
});
|
||||
|
||||
if (!isPositiveInteger(created.id)) {
|
||||
throw new Error("The Card API did not return a valid model ID.");
|
||||
}
|
||||
|
||||
const entry: ModelLockEntry = {
|
||||
sourceModelId: sourceModel.id,
|
||||
copiedModelId: created.id,
|
||||
hash,
|
||||
actions: [],
|
||||
};
|
||||
|
||||
lockfile.models.push(entry);
|
||||
|
||||
writeResourceLockfile(appRoot, lockfile);
|
||||
log(
|
||||
previous
|
||||
? `recreated model: card ${previous.copiedModelId} -> card ${created.id}`
|
||||
: `copied model: model ${sourceModel.id} -> card ${created.id}`,
|
||||
);
|
||||
|
||||
await reconcileModelActions(context, entry, desiredActions);
|
||||
}
|
||||
|
||||
async function removeUnusedModels(
|
||||
context: ModelContext,
|
||||
previousModels: ModelLockEntry[],
|
||||
desiredModelIds: Set<number>,
|
||||
) {
|
||||
const { appRoot, collectionId, client, lockfile, log } = context;
|
||||
for (const entry of previousModels) {
|
||||
if (desiredModelIds.has(entry.sourceModelId)) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const index = lockfile.models.findIndex(
|
||||
({ sourceModelId }) => sourceModelId === entry.sourceModelId,
|
||||
);
|
||||
|
||||
if (index < 0) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const copy = await orNullOn404(client.getCard(entry.copiedModelId));
|
||||
|
||||
if (copy?.archived === true) {
|
||||
// The trash masks a card's collection, so this reads the same as a copy
|
||||
// moved out and trashed afterwards. Deleting is permanent, and the card is
|
||||
// already where someone put it to be deleted.
|
||||
log(`left in the trash: card ${copy.id}`);
|
||||
} else if (copy) {
|
||||
assertOwnedModelCopy(copy, collectionId);
|
||||
// Deleting the copy cascades to the actions copied onto it.
|
||||
await client.deleteCard(copy.id);
|
||||
log(`deleted model: card ${copy.id}`);
|
||||
}
|
||||
|
||||
lockfile.models.splice(index, 1);
|
||||
|
||||
writeResourceLockfile(appRoot, lockfile);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Makes the data app collection hold exactly the models its actions belong to,
|
||||
* copying a model when its first action appears and deleting it when its last
|
||||
* action goes away. Returns the tables those models and actions read.
|
||||
*/
|
||||
export async function reconcileModels({
|
||||
appRoot,
|
||||
slug,
|
||||
collectionId,
|
||||
actions,
|
||||
lockfile,
|
||||
client,
|
||||
log,
|
||||
}: ReconcileModelsOptions) {
|
||||
const previousModels = [...lockfile.models];
|
||||
const resolved = await resolveActions(appRoot, actions, client);
|
||||
|
||||
const desired = new Map<number, ResolvedAction[]>();
|
||||
|
||||
for (const entry of resolved) {
|
||||
desired.set(entry.source.model_id, [
|
||||
...(desired.get(entry.source.model_id) ?? []),
|
||||
entry,
|
||||
]);
|
||||
}
|
||||
|
||||
const sourceModels = await fetchSourceModels(appRoot, desired, client);
|
||||
const context: ModelContext = {
|
||||
appRoot,
|
||||
collectionId,
|
||||
client,
|
||||
lockfile,
|
||||
log,
|
||||
};
|
||||
|
||||
for (const [sourceModelId, desiredActions] of desired) {
|
||||
const sourceModel = sourceModels.get(sourceModelId);
|
||||
if (sourceModel) {
|
||||
await reconcileModel(context, sourceModel, desiredActions);
|
||||
}
|
||||
}
|
||||
|
||||
await removeUnusedModels(context, previousModels, new Set(desired.keys()));
|
||||
|
||||
const modelQueries = [...sourceModels.values()].map(
|
||||
(model) => model.dataset_query,
|
||||
);
|
||||
|
||||
const actionQueries = resolved.flatMap(({ source }) =>
|
||||
source.dataset_query ? [source.dataset_query] : [],
|
||||
);
|
||||
|
||||
return client.resolveTableDependencies(slug, [
|
||||
...modelQueries,
|
||||
...actionQueries,
|
||||
]);
|
||||
}
|
||||
@@ -0,0 +1,507 @@
|
||||
import fs from "node:fs";
|
||||
import path from "node:path";
|
||||
|
||||
import { QUERY_DEFINITIONS, injectGeneratedId } from "./ast/query-source";
|
||||
import { getCanonicalQueryJson } from "./canonical";
|
||||
import { isPositiveInteger } from "./guards";
|
||||
import { RESOURCE_LOCKFILE, writeResourceLockfile } from "./lockfile";
|
||||
import { getErrorMessage, getRelativeDefinitionLocation } from "./messages";
|
||||
import type { MetabaseClient } from "./metabase-client";
|
||||
import { orNullOn404 } from "./metabase-client";
|
||||
import { reconcileMetrics, reconcileRemovedMetrics } from "./reconcile-metrics";
|
||||
import type {
|
||||
DiscoveredQuery,
|
||||
QueryLockEntry,
|
||||
ResourceLockfile,
|
||||
} from "./types";
|
||||
|
||||
type QueryCard = Awaited<ReturnType<MetabaseClient["getCard"]>>;
|
||||
|
||||
interface ResolvedQuery {
|
||||
query: DiscoveredQuery;
|
||||
resolved: Awaited<ReturnType<MetabaseClient["resolveQuery"]>>;
|
||||
}
|
||||
|
||||
interface ReconciliationContext {
|
||||
appRoot: string;
|
||||
collectionId: number;
|
||||
client: MetabaseClient;
|
||||
lockfile: ResourceLockfile;
|
||||
recoveredIds: Map<DiscoveredQuery, number>;
|
||||
repairedLockfileIds: Set<number>;
|
||||
cardsById: Map<number, QueryCard | null>;
|
||||
log: (message: string) => void;
|
||||
}
|
||||
|
||||
export interface ReconcileQueriesOptions {
|
||||
appRoot: string;
|
||||
slug: string;
|
||||
collectionId: number;
|
||||
queries: DiscoveredQuery[];
|
||||
lockfile: ResourceLockfile;
|
||||
client: MetabaseClient;
|
||||
log: (message: string) => void;
|
||||
}
|
||||
|
||||
function queryWithoutGeneratedId(query: DiscoveredQuery) {
|
||||
const authored = { ...query.query };
|
||||
delete authored.savedQuestionSourceId;
|
||||
return authored;
|
||||
}
|
||||
|
||||
function lockEntry(query: DiscoveredQuery, id: number): QueryLockEntry {
|
||||
return {
|
||||
tableId: query.tableId,
|
||||
hash: query.hash,
|
||||
savedQuestionSourceId: id,
|
||||
};
|
||||
}
|
||||
|
||||
function replaceLockEntry(
|
||||
entries: QueryLockEntry[],
|
||||
previousId: number | undefined,
|
||||
next: QueryLockEntry,
|
||||
) {
|
||||
const index = previousId
|
||||
? entries.findIndex(
|
||||
({ savedQuestionSourceId }) => savedQuestionSourceId === previousId,
|
||||
)
|
||||
: -1;
|
||||
if (index >= 0) {
|
||||
entries[index] = next;
|
||||
} else {
|
||||
entries.push(next);
|
||||
}
|
||||
}
|
||||
|
||||
function recoverMissingQueryIds(
|
||||
appRoot: string,
|
||||
queries: DiscoveredQuery[],
|
||||
previousEntries: QueryLockEntry[],
|
||||
) {
|
||||
const claimedIds = new Set(
|
||||
queries.flatMap(({ savedQuestionSourceId }) =>
|
||||
savedQuestionSourceId ? [savedQuestionSourceId] : [],
|
||||
),
|
||||
);
|
||||
const recoveredIds = new Map<DiscoveredQuery, number>();
|
||||
|
||||
for (const query of queries) {
|
||||
if (query.savedQuestionSourceId) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const candidates = previousEntries.filter(
|
||||
(entry) =>
|
||||
entry.tableId === query.tableId &&
|
||||
entry.hash === query.hash &&
|
||||
!claimedIds.has(entry.savedQuestionSourceId),
|
||||
);
|
||||
|
||||
if (candidates.length > 1) {
|
||||
throw new Error(
|
||||
`${getRelativeDefinitionLocation(appRoot, query)} matches multiple lockfile entries. Restore its savedQuestionSourceId manually and run sync-resources again.`,
|
||||
);
|
||||
}
|
||||
|
||||
const [candidate] = candidates;
|
||||
|
||||
if (candidate) {
|
||||
recoveredIds.set(query, candidate.savedQuestionSourceId);
|
||||
claimedIds.add(candidate.savedQuestionSourceId);
|
||||
}
|
||||
}
|
||||
|
||||
return recoveredIds;
|
||||
}
|
||||
|
||||
function savedQuestionId(
|
||||
query: DiscoveredQuery,
|
||||
recoveredIds: Map<DiscoveredQuery, number>,
|
||||
) {
|
||||
return query.savedQuestionSourceId ?? recoveredIds.get(query);
|
||||
}
|
||||
|
||||
async function resolveQueries(
|
||||
appRoot: string,
|
||||
slug: string,
|
||||
queries: DiscoveredQuery[],
|
||||
client: MetabaseClient,
|
||||
): Promise<ResolvedQuery[]> {
|
||||
return Promise.all(
|
||||
queries.map(async (query) => {
|
||||
try {
|
||||
return {
|
||||
query,
|
||||
resolved: await client.resolveQuery(
|
||||
slug,
|
||||
queryWithoutGeneratedId(query),
|
||||
),
|
||||
};
|
||||
} catch (error) {
|
||||
throw new Error(
|
||||
`Could not resolve ${getRelativeDefinitionLocation(appRoot, query)}: ${getErrorMessage(error)}`,
|
||||
);
|
||||
}
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
async function fetchReferencedCards(
|
||||
queries: DiscoveredQuery[],
|
||||
recoveredIds: Map<DiscoveredQuery, number>,
|
||||
client: MetabaseClient,
|
||||
) {
|
||||
const cardsById = new Map<number, QueryCard | null>();
|
||||
await Promise.all(
|
||||
queries.map(async (query) => {
|
||||
const id = savedQuestionId(query, recoveredIds);
|
||||
|
||||
if (!id) {
|
||||
return;
|
||||
}
|
||||
|
||||
cardsById.set(id, await orNullOn404(client.getCard(id)));
|
||||
}),
|
||||
);
|
||||
return cardsById;
|
||||
}
|
||||
|
||||
function findLockfileRepairs(
|
||||
queries: DiscoveredQuery[],
|
||||
recoveredIds: Map<DiscoveredQuery, number>,
|
||||
entries: QueryLockEntry[],
|
||||
cardsById: Map<number, QueryCard | null>,
|
||||
collectionId: number,
|
||||
) {
|
||||
const repairs: Array<{ query: DiscoveredQuery; id: number }> = [];
|
||||
|
||||
for (const query of queries) {
|
||||
const id = savedQuestionId(query, recoveredIds);
|
||||
|
||||
if (!id) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const card = cardsById.get(id);
|
||||
|
||||
if (card && card.id !== id) {
|
||||
throw new Error(
|
||||
`The Card API returned card ${card.id} when card ${id} was requested.`,
|
||||
);
|
||||
}
|
||||
|
||||
if (!card) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const isLockfileProven = entries.some(
|
||||
({ savedQuestionSourceId }) => savedQuestionSourceId === id,
|
||||
);
|
||||
|
||||
if (isLockfileProven) {
|
||||
if (card.type !== "question") {
|
||||
throw new Error(`Card ${id} is no longer a saved question.`);
|
||||
}
|
||||
|
||||
continue;
|
||||
}
|
||||
|
||||
if (card.archived === true) {
|
||||
throw new Error(
|
||||
`${query.exportName} references card ${id}, which is in the trash. Restore it to data app collection ${collectionId} to publish it again, or drop \`savedQuestionSourceId: ${id}\` from the declaration to publish a new copy, then run sync-resources again.`,
|
||||
);
|
||||
}
|
||||
|
||||
if (card.type !== "question" || card.collection_id !== collectionId) {
|
||||
throw new Error(
|
||||
`${query.exportName} references card ${id}, but the lockfile does not prove ownership and the card cannot be safely adopted.`,
|
||||
);
|
||||
}
|
||||
|
||||
repairs.push({ query, id });
|
||||
}
|
||||
|
||||
return repairs;
|
||||
}
|
||||
|
||||
function applyLockfileRepairs(
|
||||
appRoot: string,
|
||||
lockfile: ResourceLockfile,
|
||||
repairs: Array<{ query: DiscoveredQuery; id: number }>,
|
||||
) {
|
||||
const repairedIds = new Set<number>();
|
||||
|
||||
for (const { query, id } of repairs) {
|
||||
lockfile.queries.push(lockEntry(query, id));
|
||||
writeResourceLockfile(appRoot, lockfile);
|
||||
repairedIds.add(id);
|
||||
}
|
||||
|
||||
return repairedIds;
|
||||
}
|
||||
|
||||
function queriesMatch(card: QueryCard, resolved: ResolvedQuery["resolved"]) {
|
||||
return (
|
||||
getCanonicalQueryJson(card.dataset_query) ===
|
||||
getCanonicalQueryJson(resolved.dataset_query)
|
||||
);
|
||||
}
|
||||
|
||||
function logExistingCardOutcome(
|
||||
context: ReconciliationContext,
|
||||
query: DiscoveredQuery,
|
||||
card: QueryCard,
|
||||
existingEntry: QueryLockEntry,
|
||||
needsUpdate: boolean,
|
||||
) {
|
||||
const { recoveredIds, repairedLockfileIds, log } = context;
|
||||
const recoveredId = recoveredIds.get(query);
|
||||
|
||||
if (needsUpdate) {
|
||||
const authoredChanged =
|
||||
existingEntry.tableId !== query.tableId ||
|
||||
existingEntry.hash !== query.hash;
|
||||
const action =
|
||||
authoredChanged || card.name !== query.exportName
|
||||
? "updated"
|
||||
: "restored";
|
||||
const checkpoint = recoveredId
|
||||
? "restored source ID, "
|
||||
: repairedLockfileIds.has(card.id)
|
||||
? "repaired lockfile, "
|
||||
: "";
|
||||
|
||||
log(`${checkpoint}${action}: card ${card.id}`);
|
||||
} else if (recoveredId) {
|
||||
log(`restored source ID: ${query.exportName} -> card ${card.id}`);
|
||||
} else if (repairedLockfileIds.has(card.id)) {
|
||||
log(`repaired lockfile: ${query.exportName} -> card ${card.id}`);
|
||||
} else {
|
||||
log(`unchanged: card ${card.id}`);
|
||||
}
|
||||
}
|
||||
|
||||
async function reconcileLiveQuery(
|
||||
context: ReconciliationContext,
|
||||
{ query, resolved }: ResolvedQuery,
|
||||
) {
|
||||
const { appRoot, collectionId, client, lockfile, recoveredIds, cardsById } =
|
||||
context;
|
||||
const authoritative = {
|
||||
name: query.exportName,
|
||||
collectionId,
|
||||
datasetQuery: resolved.dataset_query,
|
||||
};
|
||||
const recoveredId = recoveredIds.get(query);
|
||||
const previousId = savedQuestionId(query, recoveredIds);
|
||||
const card = previousId ? cardsById.get(previousId) : null;
|
||||
|
||||
if (!card) {
|
||||
const created = await client.createCard(authoritative);
|
||||
|
||||
if (!isPositiveInteger(created.id)) {
|
||||
throw new Error("The Card API did not return a valid saved question ID.");
|
||||
}
|
||||
|
||||
injectGeneratedId(query, QUERY_DEFINITIONS, created.id);
|
||||
replaceLockEntry(
|
||||
lockfile.queries,
|
||||
previousId,
|
||||
lockEntry(query, created.id),
|
||||
);
|
||||
writeResourceLockfile(appRoot, lockfile);
|
||||
context.log(
|
||||
previousId
|
||||
? `recreated: card ${previousId} -> card ${created.id}`
|
||||
: `created: ${query.exportName} -> card ${created.id}`,
|
||||
);
|
||||
|
||||
return created.id;
|
||||
}
|
||||
|
||||
if (recoveredId) {
|
||||
injectGeneratedId(query, QUERY_DEFINITIONS, recoveredId);
|
||||
}
|
||||
|
||||
const existingEntry = lockfile.queries.find(
|
||||
({ savedQuestionSourceId }) => savedQuestionSourceId === card.id,
|
||||
);
|
||||
|
||||
if (!existingEntry) {
|
||||
throw new Error(
|
||||
`${query.exportName} references card ${card.id}, but the lockfile does not prove ownership.`,
|
||||
);
|
||||
}
|
||||
|
||||
if (card.type !== "question") {
|
||||
throw new Error(`Card ${card.id} is no longer a saved question.`);
|
||||
}
|
||||
|
||||
const needsUpdate =
|
||||
card.name !== query.exportName ||
|
||||
card.collection_id !== collectionId ||
|
||||
// A copy trashed on its own keeps its collection, so only `archived` says so.
|
||||
card.archived === true ||
|
||||
!queriesMatch(card, resolved);
|
||||
|
||||
if (needsUpdate) {
|
||||
await client.updateCard(card.id, authoritative);
|
||||
}
|
||||
|
||||
logExistingCardOutcome(context, query, card, existingEntry, needsUpdate);
|
||||
replaceLockEntry(lockfile.queries, card.id, lockEntry(query, card.id));
|
||||
writeResourceLockfile(appRoot, lockfile);
|
||||
|
||||
return card.id;
|
||||
}
|
||||
|
||||
async function reconcileLiveQueries(
|
||||
context: ReconciliationContext,
|
||||
resolvedQueries: ResolvedQuery[],
|
||||
) {
|
||||
const liveIds = new Set<number>();
|
||||
|
||||
for (const resolvedQuery of resolvedQueries) {
|
||||
liveIds.add(await reconcileLiveQuery(context, resolvedQuery));
|
||||
}
|
||||
|
||||
return liveIds;
|
||||
}
|
||||
|
||||
function removeLockEntry(
|
||||
appRoot: string,
|
||||
lockfile: ResourceLockfile,
|
||||
id: number,
|
||||
) {
|
||||
const index = lockfile.queries.findIndex(
|
||||
({ savedQuestionSourceId }) => savedQuestionSourceId === id,
|
||||
);
|
||||
|
||||
if (index >= 0) {
|
||||
lockfile.queries.splice(index, 1);
|
||||
writeResourceLockfile(appRoot, lockfile);
|
||||
}
|
||||
}
|
||||
|
||||
async function reconcileRemovedQueries(
|
||||
context: ReconciliationContext,
|
||||
previousEntries: QueryLockEntry[],
|
||||
liveIds: Set<number>,
|
||||
) {
|
||||
const { appRoot, collectionId, client, lockfile, log } = context;
|
||||
for (const entry of previousEntries) {
|
||||
if (liveIds.has(entry.savedQuestionSourceId)) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const card = await orNullOn404(client.getCard(entry.savedQuestionSourceId));
|
||||
|
||||
if (!card) {
|
||||
removeLockEntry(appRoot, lockfile, entry.savedQuestionSourceId);
|
||||
|
||||
continue;
|
||||
}
|
||||
|
||||
// The trash masks a card's collection, so a copy trashed from the app
|
||||
// collection reads the same as one moved out and trashed afterwards. Stop
|
||||
// tracking it either way: deleting is permanent, and the card is already
|
||||
// where someone put it to be deleted.
|
||||
if (card.archived === true) {
|
||||
removeLockEntry(appRoot, lockfile, card.id);
|
||||
log(`left in the trash: card ${card.id}`);
|
||||
|
||||
continue;
|
||||
}
|
||||
|
||||
if (card.type !== "question" || card.collection_id !== collectionId) {
|
||||
const recovery =
|
||||
card.type === "question"
|
||||
? `Move card ${card.id} back to data app collection ${collectionId} or delete it manually`
|
||||
: `Change card ${card.id} back to a saved question in data app collection ${collectionId} or delete it manually`;
|
||||
|
||||
throw new Error(
|
||||
`Card ${card.id} belongs to a removed query but is no longer an owned question in the data app collection, so it was left untouched. ${recovery}, then run sync-resources again.`,
|
||||
);
|
||||
}
|
||||
|
||||
await client.deleteCard(card.id);
|
||||
|
||||
removeLockEntry(appRoot, lockfile, card.id);
|
||||
log(`deleted: card ${card.id}`);
|
||||
}
|
||||
}
|
||||
|
||||
export async function reconcileQueries({
|
||||
appRoot,
|
||||
slug,
|
||||
collectionId,
|
||||
queries,
|
||||
lockfile,
|
||||
client,
|
||||
log,
|
||||
}: ReconcileQueriesOptions) {
|
||||
const previousEntries = [...lockfile.queries];
|
||||
const recoveredIds = recoverMissingQueryIds(
|
||||
appRoot,
|
||||
queries,
|
||||
previousEntries,
|
||||
);
|
||||
const resolvedQueries = await resolveQueries(appRoot, slug, queries, client);
|
||||
const metricReconciliation = await reconcileMetrics({
|
||||
appRoot,
|
||||
collectionId,
|
||||
resolvedQueries: resolvedQueries.map(({ resolved }) => resolved),
|
||||
lockfile,
|
||||
client,
|
||||
log,
|
||||
});
|
||||
const cardsById = await fetchReferencedCards(queries, recoveredIds, client);
|
||||
const repairs = findLockfileRepairs(
|
||||
queries,
|
||||
recoveredIds,
|
||||
lockfile.queries,
|
||||
cardsById,
|
||||
collectionId,
|
||||
);
|
||||
const repairedLockfileIds = applyLockfileRepairs(appRoot, lockfile, repairs);
|
||||
const context: ReconciliationContext = {
|
||||
appRoot,
|
||||
collectionId,
|
||||
client,
|
||||
lockfile,
|
||||
recoveredIds,
|
||||
repairedLockfileIds,
|
||||
cardsById,
|
||||
log,
|
||||
};
|
||||
const liveIds = await reconcileLiveQueries(context, resolvedQueries);
|
||||
|
||||
await reconcileRemovedMetrics({
|
||||
appRoot,
|
||||
collectionId,
|
||||
...metricReconciliation,
|
||||
lockfile,
|
||||
client,
|
||||
log,
|
||||
});
|
||||
await reconcileRemovedQueries(context, previousEntries, liveIds);
|
||||
|
||||
if (!fs.existsSync(path.join(appRoot, RESOURCE_LOCKFILE))) {
|
||||
writeResourceLockfile(appRoot, lockfile);
|
||||
}
|
||||
|
||||
const queryTableIds = resolvedQueries.flatMap(
|
||||
({ resolved }) => resolved.table_ids,
|
||||
);
|
||||
const metricTableIds = await client.resolveTableDependencies(
|
||||
slug,
|
||||
resolvedQueries.flatMap(({ resolved }) =>
|
||||
resolved.metrics.map((metric) => metric.dataset_query),
|
||||
),
|
||||
);
|
||||
|
||||
return [...new Set([...queryTableIds, ...metricTableIds])].sort(
|
||||
(a, b) => a - b,
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,31 @@
|
||||
import { isPositiveInteger, isRecord } from "./guards";
|
||||
|
||||
/**
|
||||
* Finds saved-question IDs used as sources anywhere in a query, including joins and nested stages.
|
||||
* Supports `source-card` IDs and legacy `source-table: "card__<id>"` references.
|
||||
*/
|
||||
export function getSavedCardSourceIds(
|
||||
query: unknown,
|
||||
ids = new Set<number>(),
|
||||
): Set<number> {
|
||||
if (Array.isArray(query)) {
|
||||
query.forEach((item) => getSavedCardSourceIds(item, ids));
|
||||
} else if (isRecord(query)) {
|
||||
Object.entries(query).forEach(([key, value]) => {
|
||||
if (key === "source-card" && isPositiveInteger(value)) {
|
||||
ids.add(value);
|
||||
} else if (key === "source-table" && typeof value === "string") {
|
||||
const match = /^card__(\d+)$/.exec(value);
|
||||
const id = match && Number(match[1]);
|
||||
|
||||
if (isPositiveInteger(id)) {
|
||||
ids.add(id);
|
||||
}
|
||||
}
|
||||
|
||||
getSavedCardSourceIds(value, ids);
|
||||
});
|
||||
}
|
||||
|
||||
return ids;
|
||||
}
|
||||
@@ -0,0 +1,202 @@
|
||||
import path from "node:path";
|
||||
|
||||
import { discoverActions, discoverQueries } from "./discover";
|
||||
import { isPositiveInteger } from "./guards";
|
||||
import {
|
||||
RESOURCE_LOCKFILE,
|
||||
readResourceLockfile,
|
||||
writeResourceLockfile,
|
||||
} from "./lockfile";
|
||||
import { MetabaseClient, orNullOn404 } from "./metabase-client";
|
||||
import { reconcileQueries } from "./reconcile";
|
||||
import { reconcileModels } from "./reconcile-models";
|
||||
import type { DiscoveredAction, ResourceLockfile } from "./types";
|
||||
|
||||
export interface SyncResourcesOptions {
|
||||
appRoot: string;
|
||||
metabaseUrl: string;
|
||||
apiKey: string;
|
||||
log?: (message: string) => void;
|
||||
}
|
||||
|
||||
function staleDefinition(
|
||||
appRoot: string,
|
||||
definition: { filePath: string; exportName: string },
|
||||
): never {
|
||||
throw new Error(
|
||||
`${path.relative(appRoot, definition.filePath)}:${definition.exportName} is not synchronized. Run \`npm run sync-resources\` and rebuild.`,
|
||||
);
|
||||
}
|
||||
|
||||
function checkQueriesSynchronized(
|
||||
appRoot: string,
|
||||
queries: Awaited<ReturnType<typeof discoverQueries>>,
|
||||
lockfile: ResourceLockfile,
|
||||
) {
|
||||
const byId = new Map(
|
||||
lockfile.queries.map((entry) => [entry.savedQuestionSourceId, entry]),
|
||||
);
|
||||
for (const query of queries) {
|
||||
const id = query.savedQuestionSourceId;
|
||||
const entry = id ? byId.get(id) : undefined;
|
||||
if (
|
||||
!id ||
|
||||
!entry ||
|
||||
entry.tableId !== query.tableId ||
|
||||
entry.hash !== query.hash
|
||||
) {
|
||||
staleDefinition(appRoot, query);
|
||||
}
|
||||
}
|
||||
const liveIds = new Set(
|
||||
queries.flatMap(({ savedQuestionSourceId }) =>
|
||||
savedQuestionSourceId ? [savedQuestionSourceId] : [],
|
||||
),
|
||||
);
|
||||
if (
|
||||
lockfile.queries.some(
|
||||
({ savedQuestionSourceId }) => !liveIds.has(savedQuestionSourceId),
|
||||
)
|
||||
) {
|
||||
throw new Error(
|
||||
`${RESOURCE_LOCKFILE} contains a removed query. Run \`npm run sync-resources\` and rebuild.`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
function checkActionsSynchronized(
|
||||
appRoot: string,
|
||||
actions: DiscoveredAction[],
|
||||
lockfile: ResourceLockfile,
|
||||
) {
|
||||
const byId = new Map(
|
||||
lockfile.models.flatMap((model) =>
|
||||
model.actions.map((entry) => [entry.sourceActionId, entry] as const),
|
||||
),
|
||||
);
|
||||
|
||||
for (const action of actions) {
|
||||
const entry = byId.get(action.sourceActionId);
|
||||
|
||||
if (!entry || entry.copiedActionId !== action.copiedActionId) {
|
||||
staleDefinition(appRoot, action);
|
||||
}
|
||||
}
|
||||
|
||||
const liveIds = new Set(actions.map(({ sourceActionId }) => sourceActionId));
|
||||
const hasRemovedAction = lockfile.models.some(
|
||||
(model) =>
|
||||
model.actions.length === 0 ||
|
||||
model.actions.some(({ sourceActionId }) => !liveIds.has(sourceActionId)),
|
||||
);
|
||||
|
||||
if (hasRemovedAction) {
|
||||
throw new Error(
|
||||
`${RESOURCE_LOCKFILE} contains a removed action. Run \`npm run sync-resources\` and rebuild.`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
export async function checkResourcesSynced(appRoot: string) {
|
||||
const [queries, actions] = await Promise.all([
|
||||
discoverQueries(appRoot),
|
||||
discoverActions(appRoot),
|
||||
]);
|
||||
const lockfile = readResourceLockfile(appRoot);
|
||||
|
||||
checkQueriesSynchronized(appRoot, queries, lockfile);
|
||||
checkActionsSynchronized(appRoot, actions, lockfile);
|
||||
}
|
||||
|
||||
/**
|
||||
* Move synchronized copies when the app gets a new collection. Collection read
|
||||
* access lets app viewers run the copies. Move only copies that remain in the
|
||||
* previous synchronized collection.
|
||||
*/
|
||||
async function moveCopiesToAppCollection(
|
||||
appRoot: string,
|
||||
lockfile: ResourceLockfile,
|
||||
collectionId: number,
|
||||
client: MetabaseClient,
|
||||
log: (message: string) => void,
|
||||
) {
|
||||
const previousCollectionId = lockfile.collectionId;
|
||||
|
||||
if (previousCollectionId === collectionId) {
|
||||
return;
|
||||
}
|
||||
|
||||
if (previousCollectionId !== undefined) {
|
||||
const copiedCardIds = [
|
||||
...lockfile.queries.map((entry) => entry.savedQuestionSourceId),
|
||||
...lockfile.models.map((entry) => entry.copiedModelId),
|
||||
...lockfile.metrics.map((entry) => entry.copiedMetricId),
|
||||
];
|
||||
|
||||
for (const cardId of copiedCardIds) {
|
||||
const card = await orNullOn404(client.getCard(cardId));
|
||||
|
||||
if (card?.collection_id !== previousCollectionId) {
|
||||
continue;
|
||||
}
|
||||
|
||||
await client.moveCardToCollection(cardId, collectionId);
|
||||
log(`moved card ${cardId} into data app collection ${collectionId}`);
|
||||
}
|
||||
}
|
||||
|
||||
lockfile.collectionId = collectionId;
|
||||
writeResourceLockfile(appRoot, lockfile);
|
||||
}
|
||||
|
||||
export async function syncResources({
|
||||
appRoot,
|
||||
metabaseUrl,
|
||||
apiKey,
|
||||
log = (message) => process.stdout.write(`${message}\n`),
|
||||
}: SyncResourcesOptions) {
|
||||
const [queries, actions] = await Promise.all([
|
||||
discoverQueries(appRoot),
|
||||
discoverActions(appRoot),
|
||||
]);
|
||||
const lockfile = readResourceLockfile(appRoot);
|
||||
const client = new MetabaseClient(metabaseUrl, apiKey);
|
||||
const slug = path.basename(appRoot);
|
||||
const app = await client.ensureDraft(slug);
|
||||
if (!isPositiveInteger(app.resource_collection_id)) {
|
||||
throw new Error(`Data app ${slug} does not have a resource collection.`);
|
||||
}
|
||||
|
||||
await moveCopiesToAppCollection(
|
||||
appRoot,
|
||||
lockfile,
|
||||
app.resource_collection_id,
|
||||
client,
|
||||
log,
|
||||
);
|
||||
|
||||
const queryTableIds = await reconcileQueries({
|
||||
appRoot,
|
||||
slug,
|
||||
collectionId: app.resource_collection_id,
|
||||
queries,
|
||||
lockfile,
|
||||
client,
|
||||
log,
|
||||
});
|
||||
const modelTableIds = await reconcileModels({
|
||||
appRoot,
|
||||
slug,
|
||||
collectionId: app.resource_collection_id,
|
||||
actions,
|
||||
lockfile,
|
||||
client,
|
||||
log,
|
||||
});
|
||||
|
||||
const tableIds = [...new Set([...queryTableIds, ...modelTableIds])].sort(
|
||||
(a, b) => a - b,
|
||||
);
|
||||
|
||||
await client.updateTableDependencies(slug, tableIds);
|
||||
}
|
||||
@@ -0,0 +1,86 @@
|
||||
import {
|
||||
canonicalJson,
|
||||
getCanonicalQueryJson,
|
||||
getQueryFingerprint,
|
||||
} from "../canonical";
|
||||
|
||||
import { setupResourceSyncTests } from "./setup";
|
||||
|
||||
describe("query canonicalization", () => {
|
||||
setupResourceSyncTests();
|
||||
|
||||
it("uses a property-order-independent authored DSL fingerprint", () => {
|
||||
const first = getQueryFingerprint({
|
||||
source: { type: "table", id: 1 },
|
||||
limit: 5,
|
||||
});
|
||||
const second = getQueryFingerprint({
|
||||
limit: 5,
|
||||
source: { id: 2, type: "table" },
|
||||
savedQuestionSourceId: 99,
|
||||
});
|
||||
expect(first.hash).toBe(second.hash);
|
||||
expect(first.tableId).toBe(1);
|
||||
expect(second.tableId).toBe(2);
|
||||
expect(canonicalJson({ b: 1, a: 2 })).toBe('{"a":2,"b":1}');
|
||||
expect(canonicalJson({ ä: 2, z: 1 })).toBe('{"z":1,"ä":2}');
|
||||
});
|
||||
|
||||
it("rejects non-serializable query definitions", () => {
|
||||
expect(() => canonicalJson({ value: undefined })).toThrow(
|
||||
"cannot contain undefined values",
|
||||
);
|
||||
});
|
||||
|
||||
it("rejects unsupported values and invalid table sources", () => {
|
||||
const circular: { self?: unknown } = {};
|
||||
circular.self = circular;
|
||||
|
||||
expect(() => canonicalJson(circular)).toThrow("circular references");
|
||||
expect(() => canonicalJson({ value: new Date() })).toThrow(
|
||||
"only plain objects",
|
||||
);
|
||||
expect(() => canonicalJson({ value: Infinity })).toThrow(
|
||||
"non-finite numbers",
|
||||
);
|
||||
expect(() => getQueryFingerprint({})).toThrow("valid table source");
|
||||
expect(() =>
|
||||
getQueryFingerprint({ source: { type: "table", id: 0 } }),
|
||||
).toThrow("valid table source");
|
||||
});
|
||||
|
||||
it("preserves references while normalizing generated query IDs", () => {
|
||||
const queryWithGeneratedIds = (
|
||||
firstId: string,
|
||||
secondId: string,
|
||||
orderById: string,
|
||||
) => ({
|
||||
stages: [
|
||||
{
|
||||
aggregation: [
|
||||
["sum", { "lib/uuid": firstId }, ["field", {}, 1]],
|
||||
["sum", { "lib/uuid": secondId }, ["field", {}, 2]],
|
||||
],
|
||||
"order-by": [["desc", {}, ["aggregation", {}, orderById]]],
|
||||
},
|
||||
],
|
||||
});
|
||||
const normalized = (value: unknown) => getCanonicalQueryJson(value);
|
||||
const first = queryWithGeneratedIds("first-a", "second-a", "second-a");
|
||||
const same = queryWithGeneratedIds("first-b", "second-b", "second-b");
|
||||
const different = queryWithGeneratedIds("first-c", "second-c", "first-c");
|
||||
|
||||
expect(normalized(first)).toBe(normalized(same));
|
||||
expect(normalized(first)).not.toBe(normalized(different));
|
||||
});
|
||||
|
||||
it("omits empty lib/metadata from query", () => {
|
||||
expect(
|
||||
getCanonicalQueryJson({
|
||||
database: 1,
|
||||
stages: [],
|
||||
"lib/metadata": null,
|
||||
}),
|
||||
).toBe(getCanonicalQueryJson({ database: 1, stages: [] }));
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,228 @@
|
||||
import fs from "node:fs";
|
||||
|
||||
import {
|
||||
ACTION_DEFINITIONS,
|
||||
QUERY_DEFINITIONS,
|
||||
injectGeneratedId,
|
||||
} from "../ast/query-source";
|
||||
import { discoverActions, discoverQueries } from "../discover";
|
||||
import { checkResourcesSynced } from "../sync";
|
||||
|
||||
import {
|
||||
FAKE_HASH,
|
||||
makeApp,
|
||||
setupResourceSyncTests,
|
||||
writeAction,
|
||||
writeQuery,
|
||||
writeQueryLockfile,
|
||||
} from "./setup";
|
||||
|
||||
describe("query discovery", () => {
|
||||
setupResourceSyncTests();
|
||||
|
||||
it("discovers a direct named definition", async () => {
|
||||
const appRoot = makeApp();
|
||||
writeQuery(
|
||||
appRoot,
|
||||
`export const Orders = defineQuery({ savedQuestionSourceId: 10, source: { type: "table", id: 1 }, limit: 5 });`,
|
||||
);
|
||||
|
||||
await expect(discoverQueries(appRoot)).resolves.toEqual([
|
||||
expect.objectContaining({
|
||||
exportName: "Orders",
|
||||
savedQuestionSourceId: 10,
|
||||
tableId: 1,
|
||||
hash: expect.stringMatching(/^v1:sha256:[a-f0-9]{64}$/),
|
||||
}),
|
||||
]);
|
||||
});
|
||||
|
||||
it("rejects copied saved question IDs", async () => {
|
||||
const appRoot = makeApp();
|
||||
|
||||
writeQuery(
|
||||
appRoot,
|
||||
`export const First = defineQuery({ savedQuestionSourceId: 10, source: { type: "table", id: 1 } });
|
||||
export const Second = defineQuery({ savedQuestionSourceId: 10, source: { type: "table", id: 1 } });`,
|
||||
);
|
||||
|
||||
await expect(discoverQueries(appRoot)).rejects.toThrow(
|
||||
"Saved question 10 is referenced by",
|
||||
);
|
||||
});
|
||||
|
||||
it("replaces a quoted saved question ID", async () => {
|
||||
const appRoot = makeApp();
|
||||
const filePath = writeQuery(
|
||||
appRoot,
|
||||
`export const Orders = defineQuery({ "savedQuestionSourceId": 10, source: { type: "table", id: 1 } });`,
|
||||
);
|
||||
const [query] = await discoverQueries(appRoot);
|
||||
|
||||
injectGeneratedId(query, QUERY_DEFINITIONS, 20);
|
||||
|
||||
const contents = fs.readFileSync(filePath, "utf8");
|
||||
expect(contents).toContain("savedQuestionSourceId: 20");
|
||||
expect(contents).not.toContain('savedQuestionSourceId": 10');
|
||||
});
|
||||
|
||||
it.each([
|
||||
["a shorthand ID", "savedQuestionSourceId"],
|
||||
["a spread ID", "...defaults"],
|
||||
[
|
||||
"an explicit ID followed by a spread",
|
||||
"savedQuestionSourceId: 10, ...defaults",
|
||||
],
|
||||
])("reuses the generated query ID instead of %s", async (_, properties) => {
|
||||
const appRoot = makeApp();
|
||||
|
||||
writeQuery(
|
||||
appRoot,
|
||||
`const savedQuestionSourceId = 10;
|
||||
const defaults = { savedQuestionSourceId };
|
||||
export const Orders = defineQuery({ ${properties}, source: { type: "table", id: 1 } });`,
|
||||
);
|
||||
|
||||
const [query] = await discoverQueries(appRoot);
|
||||
injectGeneratedId(query, QUERY_DEFINITIONS, 20);
|
||||
|
||||
const [updatedQuery] = await discoverQueries(appRoot);
|
||||
expect(updatedQuery.savedQuestionSourceId).toBe(20);
|
||||
injectGeneratedId(updatedQuery, QUERY_DEFINITIONS, 30);
|
||||
|
||||
const [recoveredQuery] = await discoverQueries(appRoot);
|
||||
expect(recoveredQuery.savedQuestionSourceId).toBe(30);
|
||||
});
|
||||
|
||||
it("fails a read-only build check when source and lockfile drift", async () => {
|
||||
const appRoot = makeApp();
|
||||
|
||||
writeQuery(
|
||||
appRoot,
|
||||
`export const Orders = defineQuery({ savedQuestionSourceId: 10, source: { type: "table", id: 1 }, limit: 5 });`,
|
||||
);
|
||||
|
||||
writeQueryLockfile(appRoot, [
|
||||
{
|
||||
tableId: 1,
|
||||
hash: FAKE_HASH,
|
||||
savedQuestionSourceId: 10,
|
||||
},
|
||||
]);
|
||||
|
||||
await expect(checkResourcesSynced(appRoot)).rejects.toThrow(
|
||||
"is not synchronized",
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe("action discovery", () => {
|
||||
setupResourceSyncTests();
|
||||
|
||||
it("discovers named definitions and the action each references", async () => {
|
||||
const appRoot = makeApp();
|
||||
|
||||
writeAction(
|
||||
appRoot,
|
||||
`export const Create = defineAction({ copiedActionId: 91, action: { id: 51, parameters: [] } });
|
||||
export const Update = defineAction({ action: { id: 52, parameters: [] } });`,
|
||||
);
|
||||
|
||||
const discovered = await discoverActions(appRoot);
|
||||
|
||||
expect(
|
||||
discovered.map(({ exportName, sourceActionId, copiedActionId }) => ({
|
||||
exportName,
|
||||
sourceActionId,
|
||||
copiedActionId,
|
||||
})),
|
||||
).toEqual([
|
||||
{ exportName: "Create", sourceActionId: 51, copiedActionId: 91 },
|
||||
{ exportName: "Update", sourceActionId: 52, copiedActionId: undefined },
|
||||
]);
|
||||
});
|
||||
|
||||
it("reuses the generated action ID instead of shorthand and spread IDs", async () => {
|
||||
const appRoot = makeApp();
|
||||
|
||||
writeAction(
|
||||
appRoot,
|
||||
`const copiedActionId = 91;
|
||||
const defaults = { copiedActionId };
|
||||
export const Create = defineAction({ copiedActionId, ...defaults, action: { id: 51, parameters: [] } });`,
|
||||
);
|
||||
|
||||
const [action] = await discoverActions(appRoot);
|
||||
injectGeneratedId(action, ACTION_DEFINITIONS, 92);
|
||||
|
||||
const [updatedAction] = await discoverActions(appRoot);
|
||||
expect(updatedAction.copiedActionId).toBe(92);
|
||||
});
|
||||
|
||||
it("rejects two definitions claiming the same source action", async () => {
|
||||
const appRoot = makeApp();
|
||||
|
||||
writeAction(
|
||||
appRoot,
|
||||
`export const First = defineAction({ action: { id: 51, parameters: [] } });
|
||||
export const Second = defineAction({ action: { id: 51, parameters: [] } });`,
|
||||
);
|
||||
|
||||
await expect(discoverActions(appRoot)).rejects.toThrow(
|
||||
"Action 51 is referenced by",
|
||||
);
|
||||
});
|
||||
|
||||
it("rejects a copied definition that carries another's generated ID", async () => {
|
||||
const appRoot = makeApp();
|
||||
|
||||
writeAction(
|
||||
appRoot,
|
||||
`export const First = defineAction({ copiedActionId: 91, action: { id: 51, parameters: [] } });
|
||||
export const Second = defineAction({ copiedActionId: 91, action: { id: 52, parameters: [] } });`,
|
||||
);
|
||||
|
||||
await expect(discoverActions(appRoot)).rejects.toThrow(
|
||||
"Generated action 91 is referenced by",
|
||||
);
|
||||
});
|
||||
|
||||
it("rejects a definition that does not reference a generated action", async () => {
|
||||
const appRoot = makeApp();
|
||||
|
||||
writeAction(
|
||||
appRoot,
|
||||
`export const Create = defineAction({ action: { name: "not a schema entry" } });`,
|
||||
);
|
||||
|
||||
await expect(discoverActions(appRoot)).rejects.toThrow(
|
||||
"must reference a generated action",
|
||||
);
|
||||
});
|
||||
|
||||
it("rejects a hand-edited generated ID", async () => {
|
||||
const appRoot = makeApp();
|
||||
|
||||
writeAction(
|
||||
appRoot,
|
||||
`export const Create = defineAction({ copiedActionId: 0, action: { id: 51, parameters: [] } });`,
|
||||
);
|
||||
|
||||
await expect(discoverActions(appRoot)).rejects.toThrow(
|
||||
"has an invalid copiedActionId",
|
||||
);
|
||||
});
|
||||
|
||||
it("requires a definition to initialize an exported variable", async () => {
|
||||
const appRoot = makeApp();
|
||||
|
||||
writeAction(
|
||||
appRoot,
|
||||
`const Create = defineAction({ action: { id: 51, parameters: [] } });`,
|
||||
);
|
||||
|
||||
await expect(discoverActions(appRoot)).rejects.toThrow(
|
||||
"defineAction must directly initialize a named exported variable",
|
||||
);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,43 @@
|
||||
import fs from "node:fs";
|
||||
import path from "node:path";
|
||||
|
||||
import { getResourceSyncCredentials } from "../env";
|
||||
|
||||
import { makeApp, setupResourceSyncTests } from "./setup";
|
||||
|
||||
describe("resource sync credentials", () => {
|
||||
setupResourceSyncTests();
|
||||
|
||||
it("loads credentials from the repository .env.local", () => {
|
||||
const repoRoot = makeApp();
|
||||
const appRoot = path.join(repoRoot, "data_apps/orders");
|
||||
fs.mkdirSync(appRoot, { recursive: true });
|
||||
fs.writeFileSync(
|
||||
path.join(repoRoot, ".env.local"),
|
||||
"DATA_APP_MB_URL=http://metabase.test\nDATA_APP_MB_API_KEY=file-key\n",
|
||||
);
|
||||
jest.replaceProperty(process, "env", {});
|
||||
|
||||
expect(getResourceSyncCredentials(appRoot)).toEqual({
|
||||
metabaseUrl: "http://metabase.test",
|
||||
apiKey: "file-key",
|
||||
});
|
||||
});
|
||||
|
||||
it("prefers process environment credentials", () => {
|
||||
const appRoot = makeApp();
|
||||
fs.writeFileSync(
|
||||
path.join(appRoot, ".env.local"),
|
||||
"DATA_APP_MB_URL=http://file.test\nDATA_APP_MB_API_KEY=file-key\n",
|
||||
);
|
||||
jest.replaceProperty(process, "env", {
|
||||
DATA_APP_MB_URL: "http://process.test",
|
||||
DATA_APP_MB_API_KEY: "process-key",
|
||||
});
|
||||
|
||||
expect(getResourceSyncCredentials(appRoot)).toEqual({
|
||||
metabaseUrl: "http://process.test",
|
||||
apiKey: "process-key",
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,138 @@
|
||||
import fs from "node:fs";
|
||||
import path from "node:path";
|
||||
|
||||
import { RESOURCE_LOCKFILE, readResourceLockfile } from "../lockfile";
|
||||
import type { ResourceLockfile } from "../types";
|
||||
|
||||
import { FAKE_HASH, makeApp, setupResourceSyncTests } from "./setup";
|
||||
|
||||
function write(appRoot: string, value: unknown) {
|
||||
fs.writeFileSync(
|
||||
path.join(appRoot, RESOURCE_LOCKFILE),
|
||||
typeof value === "string" ? value : JSON.stringify(value),
|
||||
);
|
||||
}
|
||||
|
||||
function query(savedQuestionSourceId: number) {
|
||||
return { tableId: 1, hash: FAKE_HASH, savedQuestionSourceId };
|
||||
}
|
||||
|
||||
function model(
|
||||
sourceModelId: number,
|
||||
copiedModelId: number,
|
||||
actions: Array<[number, number]> = [],
|
||||
) {
|
||||
return {
|
||||
sourceModelId,
|
||||
copiedModelId,
|
||||
hash: FAKE_HASH,
|
||||
actions: actions.map(([sourceActionId, copiedActionId]) => ({
|
||||
sourceActionId,
|
||||
copiedActionId,
|
||||
hash: FAKE_HASH,
|
||||
})),
|
||||
};
|
||||
}
|
||||
|
||||
describe("resource lockfile", () => {
|
||||
setupResourceSyncTests();
|
||||
|
||||
it("reads an absent lockfile as empty", () => {
|
||||
expect(readResourceLockfile(makeApp())).toEqual({
|
||||
queries: [],
|
||||
models: [],
|
||||
metrics: [],
|
||||
});
|
||||
});
|
||||
|
||||
it("round-trips queries and models", () => {
|
||||
const appRoot = makeApp();
|
||||
const lockfile: ResourceLockfile = {
|
||||
queries: [query(40)],
|
||||
models: [model(5, 80, [[51, 91]])],
|
||||
metrics: [],
|
||||
};
|
||||
write(appRoot, lockfile);
|
||||
|
||||
expect(readResourceLockfile(appRoot)).toEqual(lockfile);
|
||||
});
|
||||
|
||||
it("defaults a missing models key rather than failing", () => {
|
||||
const appRoot = makeApp();
|
||||
write(appRoot, { queries: [query(40)] });
|
||||
|
||||
expect(readResourceLockfile(appRoot)).toEqual({
|
||||
queries: [query(40)],
|
||||
models: [],
|
||||
metrics: [],
|
||||
});
|
||||
});
|
||||
|
||||
// A lockfile is an ownership record: every rejection below is a file that
|
||||
// would otherwise let synchronization mutate content it cannot prove it owns.
|
||||
it.each([
|
||||
["unparseable JSON", "{ not json", "Could not read"],
|
||||
// A bare array is the shape no version writes; accepting it would let a
|
||||
// hand-edited file silently drop every model entry.
|
||||
["a bare array", [query(40)], "contains an invalid entry"],
|
||||
[
|
||||
"a malformed hash",
|
||||
{ queries: [{ ...query(40), hash: "nope" }] },
|
||||
"contains an invalid entry",
|
||||
],
|
||||
[
|
||||
"a non-integer saved question ID",
|
||||
{ queries: [{ ...query(40), savedQuestionSourceId: 1.5 }] },
|
||||
"contains an invalid entry",
|
||||
],
|
||||
[
|
||||
"a model entry missing its actions",
|
||||
{ queries: [], models: [{ sourceModelId: 5, copiedModelId: 80 }] },
|
||||
"contains an invalid model entry",
|
||||
],
|
||||
[
|
||||
"a malformed action mapping",
|
||||
{
|
||||
queries: [],
|
||||
models: [{ ...model(5, 80), actions: [{ sourceActionId: 51 }] }],
|
||||
},
|
||||
"contains an invalid model entry",
|
||||
],
|
||||
[
|
||||
"a duplicate saved question ID",
|
||||
{ queries: [query(40), query(40)] },
|
||||
"duplicate saved question ID",
|
||||
],
|
||||
[
|
||||
"a duplicate source model ID",
|
||||
{ queries: [], models: [model(5, 80), model(5, 81)] },
|
||||
"duplicate source model ID",
|
||||
],
|
||||
[
|
||||
"a duplicate copied model ID",
|
||||
{ queries: [], models: [model(5, 80), model(6, 80)] },
|
||||
"duplicate copied model ID",
|
||||
],
|
||||
[
|
||||
"a source action ID claimed by two models",
|
||||
{
|
||||
queries: [],
|
||||
models: [model(5, 80, [[51, 91]]), model(6, 81, [[51, 92]])],
|
||||
},
|
||||
"duplicate source action ID",
|
||||
],
|
||||
[
|
||||
"a copied action ID claimed by two models",
|
||||
{
|
||||
queries: [],
|
||||
models: [model(5, 80, [[51, 91]]), model(6, 81, [[52, 91]])],
|
||||
},
|
||||
"duplicate copied action ID",
|
||||
],
|
||||
])("rejects %s", (_name, value, message) => {
|
||||
const appRoot = makeApp();
|
||||
write(appRoot, value);
|
||||
|
||||
expect(() => readResourceLockfile(appRoot)).toThrow(message);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,487 @@
|
||||
import { MetabaseApiError, MetabaseClient } from "../metabase-client";
|
||||
import {
|
||||
reconcileMetrics,
|
||||
reconcileRemovedMetrics,
|
||||
} from "../reconcile-metrics";
|
||||
import type { DataAppMetric, ResourceLockfile } from "../types";
|
||||
|
||||
import { jsonResponse, makeApp, setupResourceSyncTests } from "./setup";
|
||||
|
||||
const HASH = `v1:sha256:${"0".repeat(64)}`;
|
||||
|
||||
const createMockClient = () => ({
|
||||
getCard: jest.fn(),
|
||||
updateMetric: jest.fn(),
|
||||
createMetric: jest.fn(),
|
||||
deleteCard: jest.fn(),
|
||||
});
|
||||
|
||||
describe("metric reconciliation", () => {
|
||||
setupResourceSyncTests();
|
||||
|
||||
it("does not update an unchanged copied metric", async () => {
|
||||
const client = createMockClient();
|
||||
|
||||
client.getCard.mockResolvedValue({
|
||||
id: 404,
|
||||
name: "Lifetime value",
|
||||
type: "metric",
|
||||
collection_id: 35,
|
||||
dataset_query: { database: 1, stages: [] },
|
||||
display: "table",
|
||||
visualization_settings: {},
|
||||
description: null,
|
||||
});
|
||||
|
||||
const lockfile: ResourceLockfile = {
|
||||
queries: [],
|
||||
models: [],
|
||||
metrics: [{ sourceMetricId: 251, copiedMetricId: 404, hash: HASH }],
|
||||
};
|
||||
|
||||
const log = jest.fn();
|
||||
|
||||
await reconcileMetrics({
|
||||
appRoot: makeApp(),
|
||||
collectionId: 35,
|
||||
resolvedQueries: [
|
||||
{
|
||||
dataset_query: { aggregation: [["metric", {}, 251]] },
|
||||
metrics: [
|
||||
{
|
||||
id: 251,
|
||||
name: "Lifetime value",
|
||||
type: "metric",
|
||||
collection_id: 1,
|
||||
dataset_query: {
|
||||
database: 1,
|
||||
"lib/metadata": null,
|
||||
stages: [],
|
||||
},
|
||||
display: "table",
|
||||
visualization_settings: {},
|
||||
description: null,
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
lockfile,
|
||||
client,
|
||||
log,
|
||||
});
|
||||
|
||||
expect(client.updateMetric).not.toHaveBeenCalled();
|
||||
expect(log).toHaveBeenCalledWith("unchanged metric: 251 -> 404");
|
||||
});
|
||||
|
||||
it("copies a missing metric and rewrites metric references", async () => {
|
||||
const client = createMockClient();
|
||||
client.createMetric.mockResolvedValue({ id: 404 });
|
||||
|
||||
const lockfile: ResourceLockfile = {
|
||||
queries: [],
|
||||
models: [],
|
||||
metrics: [],
|
||||
};
|
||||
|
||||
const metric: DataAppMetric = {
|
||||
id: 251,
|
||||
name: "Lifetime value",
|
||||
type: "metric",
|
||||
collection_id: 1,
|
||||
dataset_query: { database: 1, stages: [] },
|
||||
display: "table",
|
||||
visualization_settings: {},
|
||||
description: null,
|
||||
};
|
||||
|
||||
const resolvedQuery = {
|
||||
dataset_query: {
|
||||
stages: [{ aggregation: [["metric", {}, 251]] }],
|
||||
},
|
||||
metrics: [metric],
|
||||
};
|
||||
|
||||
await reconcileMetrics({
|
||||
appRoot: makeApp(),
|
||||
collectionId: 35,
|
||||
resolvedQueries: [resolvedQuery],
|
||||
lockfile,
|
||||
client,
|
||||
log: jest.fn(),
|
||||
});
|
||||
|
||||
expect(client.createMetric).toHaveBeenCalledWith({
|
||||
name: "Lifetime value",
|
||||
collectionId: 35,
|
||||
datasetQuery: { database: 1, stages: [] },
|
||||
display: "table",
|
||||
visualizationSettings: {},
|
||||
description: null,
|
||||
});
|
||||
|
||||
expect(lockfile.metrics).toEqual([
|
||||
{ sourceMetricId: 251, copiedMetricId: 404, hash: expect.any(String) },
|
||||
]);
|
||||
|
||||
expect(resolvedQuery.dataset_query).toEqual({
|
||||
stages: [{ aggregation: [["metric", {}, 404]] }],
|
||||
});
|
||||
});
|
||||
|
||||
it("updates a changed copied metric", async () => {
|
||||
const client = createMockClient();
|
||||
|
||||
client.getCard.mockResolvedValue({
|
||||
id: 404,
|
||||
name: "Old metric name",
|
||||
type: "metric",
|
||||
collection_id: 35,
|
||||
dataset_query: { database: 1, stages: [] },
|
||||
display: "table",
|
||||
visualization_settings: {},
|
||||
description: null,
|
||||
});
|
||||
|
||||
const lockfile: ResourceLockfile = {
|
||||
queries: [],
|
||||
models: [],
|
||||
metrics: [{ sourceMetricId: 251, copiedMetricId: 404, hash: HASH }],
|
||||
};
|
||||
|
||||
await reconcileMetrics({
|
||||
appRoot: makeApp(),
|
||||
collectionId: 35,
|
||||
resolvedQueries: [
|
||||
{
|
||||
dataset_query: { aggregation: [["metric", {}, 251]] },
|
||||
metrics: [
|
||||
{
|
||||
id: 251,
|
||||
name: "Lifetime value",
|
||||
type: "metric",
|
||||
collection_id: 1,
|
||||
dataset_query: { database: 1, stages: [] },
|
||||
display: "table",
|
||||
visualization_settings: {},
|
||||
description: null,
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
lockfile,
|
||||
client,
|
||||
log: jest.fn(),
|
||||
});
|
||||
|
||||
expect(client.updateMetric).toHaveBeenCalledWith(404, {
|
||||
name: "Lifetime value",
|
||||
collectionId: 35,
|
||||
datasetQuery: { database: 1, stages: [] },
|
||||
display: "table",
|
||||
visualizationSettings: {},
|
||||
description: null,
|
||||
});
|
||||
});
|
||||
|
||||
it("replaces a missing copied metric and rewrites its references", async () => {
|
||||
const client = createMockClient();
|
||||
|
||||
client.getCard.mockRejectedValue(
|
||||
new MetabaseApiError(404, "Metric not found"),
|
||||
);
|
||||
|
||||
client.createMetric.mockResolvedValue({ id: 405 });
|
||||
|
||||
const lockfile: ResourceLockfile = {
|
||||
queries: [],
|
||||
models: [],
|
||||
metrics: [{ sourceMetricId: 251, copiedMetricId: 404, hash: HASH }],
|
||||
};
|
||||
|
||||
const resolvedQuery = {
|
||||
dataset_query: { stages: [{ aggregation: [["metric", {}, 251]] }] },
|
||||
metrics: [
|
||||
{
|
||||
id: 251,
|
||||
name: "Lifetime value",
|
||||
type: "metric" as const,
|
||||
collection_id: 1,
|
||||
dataset_query: { database: 1, stages: [] },
|
||||
display: "table",
|
||||
visualization_settings: {},
|
||||
description: null,
|
||||
},
|
||||
],
|
||||
};
|
||||
|
||||
await reconcileMetrics({
|
||||
appRoot: makeApp(),
|
||||
collectionId: 35,
|
||||
resolvedQueries: [resolvedQuery],
|
||||
lockfile,
|
||||
client,
|
||||
log: jest.fn(),
|
||||
});
|
||||
|
||||
expect(client.createMetric).toHaveBeenCalledTimes(1);
|
||||
|
||||
expect(lockfile.metrics).toEqual([
|
||||
{ sourceMetricId: 251, copiedMetricId: 405, hash: expect.any(String) },
|
||||
]);
|
||||
|
||||
expect(resolvedQuery.dataset_query).toEqual({
|
||||
stages: [{ aggregation: [["metric", {}, 405]] }],
|
||||
});
|
||||
});
|
||||
|
||||
it("fails before copying metrics when a resolved query omits a metric", async () => {
|
||||
const client = createMockClient();
|
||||
|
||||
await expect(
|
||||
reconcileMetrics({
|
||||
appRoot: makeApp(),
|
||||
collectionId: 35,
|
||||
resolvedQueries: [
|
||||
{
|
||||
dataset_query: {
|
||||
stages: [{ aggregation: [["metric", {}, 251]] }],
|
||||
},
|
||||
metrics: [],
|
||||
},
|
||||
],
|
||||
lockfile: { queries: [], models: [], metrics: [] },
|
||||
client,
|
||||
log: jest.fn(),
|
||||
}),
|
||||
).rejects.toThrow(
|
||||
"These metrics are missing and cannot be reconciled: 251.",
|
||||
);
|
||||
|
||||
expect(client.createMetric).not.toHaveBeenCalled();
|
||||
expect(client.updateMetric).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("rejects a metric that joins a saved card", async () => {
|
||||
const client = createMockClient();
|
||||
|
||||
await expect(
|
||||
reconcileMetrics({
|
||||
appRoot: makeApp(),
|
||||
collectionId: 35,
|
||||
resolvedQueries: [
|
||||
{
|
||||
dataset_query: {
|
||||
stages: [{ aggregation: [["metric", {}, 251]] }],
|
||||
},
|
||||
metrics: [
|
||||
{
|
||||
id: 251,
|
||||
name: "Lifetime value",
|
||||
type: "metric",
|
||||
collection_id: 1,
|
||||
dataset_query: {
|
||||
stages: [
|
||||
{
|
||||
"source-table": 1,
|
||||
joins: [{ stages: [{ "source-card": 99 }] }],
|
||||
},
|
||||
],
|
||||
},
|
||||
display: "table",
|
||||
visualization_settings: {},
|
||||
description: null,
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
lockfile: { queries: [], models: [], metrics: [] },
|
||||
client,
|
||||
log: jest.fn(),
|
||||
}),
|
||||
).rejects.toThrow(
|
||||
"Metrics with saved-card joins cannot be synchronized: 251.",
|
||||
);
|
||||
|
||||
expect(client.createMetric).not.toHaveBeenCalled();
|
||||
expect(client.updateMetric).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("deletes a copied metric when no query references it", async () => {
|
||||
const appRoot = makeApp();
|
||||
const client = createMockClient();
|
||||
|
||||
client.getCard.mockResolvedValue({
|
||||
id: 404,
|
||||
name: "Lifetime value",
|
||||
type: "metric",
|
||||
collection_id: 35,
|
||||
dataset_query: { database: 1, stages: [] },
|
||||
});
|
||||
|
||||
const lockfile: ResourceLockfile = {
|
||||
queries: [],
|
||||
models: [],
|
||||
metrics: [{ sourceMetricId: 251, copiedMetricId: 404, hash: HASH }],
|
||||
};
|
||||
|
||||
const log = jest.fn();
|
||||
|
||||
const metricReconciliation = await reconcileMetrics({
|
||||
appRoot,
|
||||
collectionId: 35,
|
||||
resolvedQueries: [],
|
||||
lockfile,
|
||||
client,
|
||||
log,
|
||||
});
|
||||
|
||||
expect(client.deleteCard).not.toHaveBeenCalled();
|
||||
|
||||
await reconcileRemovedMetrics({
|
||||
appRoot,
|
||||
collectionId: 35,
|
||||
...metricReconciliation,
|
||||
lockfile,
|
||||
client,
|
||||
log,
|
||||
});
|
||||
|
||||
expect(client.deleteCard).toHaveBeenCalledWith(404);
|
||||
expect(lockfile.metrics).toEqual([]);
|
||||
expect(log).toHaveBeenCalledWith("deleted metric: 404");
|
||||
});
|
||||
|
||||
describe("archived and moved copies", () => {
|
||||
const metric: DataAppMetric = {
|
||||
id: 251,
|
||||
name: "Lifetime value",
|
||||
type: "metric",
|
||||
collection_id: 1,
|
||||
dataset_query: { database: 1, stages: [] },
|
||||
display: "table",
|
||||
visualization_settings: {},
|
||||
description: null,
|
||||
};
|
||||
|
||||
let appRoot: string;
|
||||
let lockfile: ResourceLockfile;
|
||||
|
||||
beforeEach(() => {
|
||||
appRoot = makeApp();
|
||||
|
||||
lockfile = {
|
||||
queries: [],
|
||||
models: [],
|
||||
metrics: [{ sourceMetricId: 251, copiedMetricId: 404, hash: HASH }],
|
||||
};
|
||||
});
|
||||
|
||||
it("sends the restore payload for an unchanged archived metric", async () => {
|
||||
const copy = {
|
||||
...metric,
|
||||
id: 404,
|
||||
collection_id: 99,
|
||||
archived: true,
|
||||
};
|
||||
|
||||
const fetchMock = jest
|
||||
.spyOn(global, "fetch")
|
||||
.mockResolvedValueOnce(jsonResponse(copy))
|
||||
.mockResolvedValueOnce(jsonResponse({ ...copy, archived: false }));
|
||||
|
||||
const resolvedQuery = {
|
||||
dataset_query: { aggregation: [["metric", {}, 251]] },
|
||||
metrics: [metric],
|
||||
};
|
||||
|
||||
await reconcileMetrics({
|
||||
appRoot,
|
||||
collectionId: 35,
|
||||
resolvedQueries: [resolvedQuery],
|
||||
lockfile,
|
||||
client: new MetabaseClient("http://metabase.test", "secret"),
|
||||
log: jest.fn(),
|
||||
});
|
||||
|
||||
expect(fetchMock).toHaveBeenCalledTimes(2);
|
||||
|
||||
expect(fetchMock).toHaveBeenLastCalledWith(
|
||||
new URL("http://metabase.test/api/card/404"),
|
||||
expect.objectContaining({
|
||||
method: "PUT",
|
||||
body: expect.any(String),
|
||||
}),
|
||||
);
|
||||
|
||||
expect(JSON.parse(String(fetchMock.mock.calls[1][1]?.body))).toEqual({
|
||||
name: "Lifetime value",
|
||||
type: "metric",
|
||||
archived: false,
|
||||
collection_id: 35,
|
||||
dataset_query: metric.dataset_query,
|
||||
display: "table",
|
||||
visualization_settings: {},
|
||||
description: null,
|
||||
});
|
||||
});
|
||||
|
||||
describe("unarchived copies moved outside the app collection", () => {
|
||||
let client: ReturnType<typeof createMockClient>;
|
||||
|
||||
beforeEach(() => {
|
||||
client = createMockClient();
|
||||
|
||||
client.getCard.mockResolvedValue({
|
||||
...metric,
|
||||
id: 404,
|
||||
collection_id: 99,
|
||||
archived: false,
|
||||
});
|
||||
});
|
||||
|
||||
it("does not update a metric copy that is moved to other collection", async () => {
|
||||
await expect(
|
||||
reconcileMetrics({
|
||||
appRoot,
|
||||
collectionId: 35,
|
||||
lockfile,
|
||||
client,
|
||||
log: jest.fn(),
|
||||
resolvedQueries: [{ dataset_query: {}, metrics: [metric] }],
|
||||
}),
|
||||
).rejects.toThrow(/data app collection 35/);
|
||||
|
||||
expect(client.updateMetric).not.toHaveBeenCalled();
|
||||
expect(client.createMetric).not.toHaveBeenCalled();
|
||||
expect(client.deleteCard).not.toHaveBeenCalled();
|
||||
|
||||
expect(lockfile.metrics).toEqual([
|
||||
{ sourceMetricId: 251, copiedMetricId: 404, hash: HASH },
|
||||
]);
|
||||
});
|
||||
|
||||
it("does not delete a metric copy that is moved to other collection when its source is removed", async () => {
|
||||
await expect(
|
||||
reconcileRemovedMetrics({
|
||||
appRoot,
|
||||
collectionId: 35,
|
||||
lockfile,
|
||||
client,
|
||||
log: jest.fn(),
|
||||
previousEntries: [...lockfile.metrics],
|
||||
liveMetricIds: new Set(),
|
||||
}),
|
||||
).rejects.toThrow(/data app collection 35/);
|
||||
|
||||
expect(client.updateMetric).not.toHaveBeenCalled();
|
||||
expect(client.createMetric).not.toHaveBeenCalled();
|
||||
expect(client.deleteCard).not.toHaveBeenCalled();
|
||||
|
||||
expect(lockfile.metrics).toEqual([
|
||||
{ sourceMetricId: 251, copiedMetricId: 404, hash: HASH },
|
||||
]);
|
||||
});
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,152 @@
|
||||
import path from "node:path";
|
||||
|
||||
import { discoverQueries } from "../discover";
|
||||
import { syncResources } from "../sync";
|
||||
|
||||
import {
|
||||
jsonResponse,
|
||||
makeApp,
|
||||
setupResourceSyncTests,
|
||||
writeQuery,
|
||||
writeQueryLockfile,
|
||||
} from "./setup";
|
||||
|
||||
describe("query reconciliation upserts", () => {
|
||||
setupResourceSyncTests();
|
||||
|
||||
it("ignores fresh generated UUIDs when comparing saved questions", async () => {
|
||||
const appRoot = makeApp();
|
||||
const slug = path.basename(appRoot);
|
||||
writeQuery(
|
||||
appRoot,
|
||||
`export const Orders = defineQuery({ savedQuestionSourceId: 35, source: { type: "table", id: 1 } });`,
|
||||
);
|
||||
const [query] = await discoverQueries(appRoot);
|
||||
writeQueryLockfile(appRoot, [
|
||||
{
|
||||
tableId: query.tableId,
|
||||
hash: query.hash,
|
||||
savedQuestionSourceId: 35,
|
||||
},
|
||||
]);
|
||||
const queryWithAggregationUuid = (uuid: string) => ({
|
||||
database: 1,
|
||||
stages: [
|
||||
{
|
||||
aggregation: [["sum", { "lib/uuid": uuid }, ["field", {}, 1]]],
|
||||
"order-by": [["desc", {}, ["aggregation", {}, uuid]]],
|
||||
},
|
||||
],
|
||||
});
|
||||
const requests: Array<{ method: string; pathname: string }> = [];
|
||||
const log = jest.fn();
|
||||
jest.spyOn(global, "fetch").mockImplementation(async (input, init) => {
|
||||
const pathname = new URL(String(input)).pathname;
|
||||
const method = init?.method ?? "GET";
|
||||
requests.push({ method, pathname });
|
||||
if (pathname === `/api/apps/${slug}/draft` && method === "POST") {
|
||||
return jsonResponse({ name: slug, resource_collection_id: 20 });
|
||||
}
|
||||
if (pathname === `/api/apps/${slug}/query` && method === "POST") {
|
||||
return jsonResponse({
|
||||
database_id: 1,
|
||||
dataset_query: queryWithAggregationUuid("fresh-uuid"),
|
||||
table_ids: [1],
|
||||
});
|
||||
}
|
||||
if (pathname === "/api/card/35" && method === "GET") {
|
||||
return jsonResponse({
|
||||
id: 35,
|
||||
name: "Orders",
|
||||
type: "question",
|
||||
collection_id: 20,
|
||||
dataset_query: queryWithAggregationUuid("saved-uuid"),
|
||||
});
|
||||
}
|
||||
if (pathname === "/api/card/35" && method === "PUT") {
|
||||
return jsonResponse({ id: 35 });
|
||||
}
|
||||
if (
|
||||
pathname === `/api/apps/${slug}/table-dependencies` &&
|
||||
method === "PUT"
|
||||
) {
|
||||
return jsonResponse({ name: slug, table_ids: [1] });
|
||||
}
|
||||
throw new Error(`Unexpected ${method} ${pathname}`);
|
||||
});
|
||||
|
||||
await syncResources({
|
||||
appRoot,
|
||||
metabaseUrl: "http://metabase.test",
|
||||
apiKey: "secret",
|
||||
log,
|
||||
});
|
||||
|
||||
expect(requests).not.toContainEqual({
|
||||
method: "PUT",
|
||||
pathname: "/api/card/35",
|
||||
});
|
||||
expect(log).toHaveBeenCalledWith("unchanged: card 35");
|
||||
});
|
||||
|
||||
it("reports an exported query rename as an update", async () => {
|
||||
const appRoot = makeApp();
|
||||
const slug = path.basename(appRoot);
|
||||
writeQuery(
|
||||
appRoot,
|
||||
`export const RenamedOrders = defineQuery({ savedQuestionSourceId: 80, source: { type: "table", id: 1 } });`,
|
||||
);
|
||||
const [query] = await discoverQueries(appRoot);
|
||||
writeQueryLockfile(appRoot, [
|
||||
{
|
||||
tableId: query.tableId,
|
||||
hash: query.hash,
|
||||
savedQuestionSourceId: 80,
|
||||
},
|
||||
]);
|
||||
const log = jest.fn();
|
||||
jest.spyOn(global, "fetch").mockImplementation(async (input, init) => {
|
||||
const pathname = new URL(String(input)).pathname;
|
||||
const method = init?.method ?? "GET";
|
||||
if (pathname === `/api/apps/${slug}/draft` && method === "POST") {
|
||||
return jsonResponse({ name: slug, resource_collection_id: 20 });
|
||||
}
|
||||
if (pathname === `/api/apps/${slug}/query` && method === "POST") {
|
||||
return jsonResponse({
|
||||
database_id: 1,
|
||||
dataset_query: { database: 1 },
|
||||
table_ids: [1],
|
||||
});
|
||||
}
|
||||
if (pathname === "/api/card/80" && method === "GET") {
|
||||
return jsonResponse({
|
||||
id: 80,
|
||||
name: "Orders",
|
||||
type: "question",
|
||||
collection_id: 20,
|
||||
dataset_query: { database: 1 },
|
||||
});
|
||||
}
|
||||
if (pathname === "/api/card/80" && method === "PUT") {
|
||||
return jsonResponse({ id: 80 });
|
||||
}
|
||||
if (
|
||||
pathname === `/api/apps/${slug}/table-dependencies` &&
|
||||
method === "PUT"
|
||||
) {
|
||||
return jsonResponse({ name: slug, table_ids: [1] });
|
||||
}
|
||||
|
||||
throw new Error(`Unexpected ${method} ${pathname}`);
|
||||
});
|
||||
|
||||
await syncResources({
|
||||
appRoot,
|
||||
metabaseUrl: "http://metabase.test",
|
||||
apiKey: "secret",
|
||||
log,
|
||||
});
|
||||
|
||||
expect(log).toHaveBeenCalledWith("updated: card 80");
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,95 @@
|
||||
import fs from "node:fs";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
|
||||
export function setupResourceSyncTests(): void {
|
||||
const appRoots: string[] = [];
|
||||
|
||||
afterEach(() => {
|
||||
jest.restoreAllMocks();
|
||||
|
||||
appRoots.forEach((appRoot) =>
|
||||
fs.rmSync(appRoot, { recursive: true, force: true }),
|
||||
);
|
||||
});
|
||||
|
||||
trackedAppRoots = appRoots;
|
||||
}
|
||||
|
||||
let trackedAppRoots: string[] | undefined;
|
||||
|
||||
export function makeApp() {
|
||||
const appRoot = fs.mkdtempSync(
|
||||
path.join(os.tmpdir(), "data-app-query-sync-"),
|
||||
);
|
||||
|
||||
trackedAppRoots?.push(appRoot);
|
||||
|
||||
fs.mkdirSync(path.join(appRoot, "queries"));
|
||||
fs.mkdirSync(path.join(appRoot, "actions"));
|
||||
|
||||
const packageRoot = path.join(
|
||||
appRoot,
|
||||
"node_modules/@metabase/embedding-sdk-react",
|
||||
);
|
||||
|
||||
fs.mkdirSync(packageRoot, { recursive: true });
|
||||
fs.writeFileSync(
|
||||
path.join(packageRoot, "package.json"),
|
||||
JSON.stringify({
|
||||
name: "@metabase/embedding-sdk-react",
|
||||
exports: { "./data-app": "./data-app.js" },
|
||||
}),
|
||||
);
|
||||
|
||||
fs.writeFileSync(
|
||||
path.join(packageRoot, "data-app.js"),
|
||||
[
|
||||
"exports.defineQuery = (query) => query;",
|
||||
"exports.defineAction = (definition) => definition;",
|
||||
].join("\n"),
|
||||
);
|
||||
|
||||
return appRoot;
|
||||
}
|
||||
|
||||
export function writeAction(appRoot: string, body: string) {
|
||||
const filePath = path.join(appRoot, "actions/orders.action.ts");
|
||||
fs.writeFileSync(
|
||||
filePath,
|
||||
`import { defineAction } from "@metabase/embedding-sdk-react/data-app";\n${body}\n`,
|
||||
);
|
||||
return filePath;
|
||||
}
|
||||
|
||||
export const FAKE_HASH = `v1:sha256:${"0".repeat(64)}`;
|
||||
|
||||
export function writeQueryLockfile(
|
||||
appRoot: string,
|
||||
queries: Array<{
|
||||
tableId: number;
|
||||
hash: string;
|
||||
savedQuestionSourceId: number;
|
||||
}>,
|
||||
) {
|
||||
fs.writeFileSync(
|
||||
path.join(appRoot, "resources_metadata.json"),
|
||||
JSON.stringify({ queries, models: [] }),
|
||||
);
|
||||
}
|
||||
|
||||
export function writeQuery(appRoot: string, body: string) {
|
||||
const filePath = path.join(appRoot, "queries/orders.query.ts");
|
||||
fs.writeFileSync(
|
||||
filePath,
|
||||
`import { defineQuery } from "@metabase/embedding-sdk-react/data-app";\n${body}\n`,
|
||||
);
|
||||
return filePath;
|
||||
}
|
||||
|
||||
export function jsonResponse(body: unknown, status = 200) {
|
||||
return new Response(status === 204 ? null : JSON.stringify(body), {
|
||||
status,
|
||||
headers: { "content-type": "application/json" },
|
||||
});
|
||||
}
|
||||
@@ -0,0 +1,175 @@
|
||||
import path from "node:path";
|
||||
|
||||
import { discoverQueries } from "../discover";
|
||||
import { syncResources } from "../sync";
|
||||
|
||||
import {
|
||||
jsonResponse,
|
||||
makeApp,
|
||||
setupResourceSyncTests,
|
||||
writeQuery,
|
||||
writeQueryLockfile,
|
||||
} from "./setup";
|
||||
|
||||
describe("query synchronization", () => {
|
||||
setupResourceSyncTests();
|
||||
|
||||
it("identifies the request that failed", async () => {
|
||||
const appRoot = makeApp();
|
||||
const slug = path.basename(appRoot);
|
||||
jest.spyOn(global, "fetch").mockResolvedValue(
|
||||
new Response("Not found.", {
|
||||
status: 404,
|
||||
headers: { "content-type": "text/plain" },
|
||||
}),
|
||||
);
|
||||
|
||||
await expect(
|
||||
syncResources({
|
||||
appRoot,
|
||||
metabaseUrl: "http://metabase.test",
|
||||
apiKey: "secret",
|
||||
log: jest.fn(),
|
||||
}),
|
||||
).rejects.toThrow(
|
||||
`Metabase returned 404 for POST http://metabase.test/api/apps/${slug}/draft: Not found.`,
|
||||
);
|
||||
});
|
||||
|
||||
it("stores an empty dependency set after synchronizing an empty app", async () => {
|
||||
const appRoot = makeApp();
|
||||
const slug = path.basename(appRoot);
|
||||
const requests: Array<{
|
||||
method: string;
|
||||
pathname: string;
|
||||
body?: string;
|
||||
}> = [];
|
||||
|
||||
jest.spyOn(global, "fetch").mockImplementation(async (input, init) => {
|
||||
const pathname = new URL(String(input)).pathname;
|
||||
const method = init?.method ?? "GET";
|
||||
requests.push({ method, pathname, body: init?.body?.toString() });
|
||||
|
||||
if (pathname === `/api/apps/${slug}/draft` && method === "POST") {
|
||||
return jsonResponse({ name: slug, resource_collection_id: 20 });
|
||||
}
|
||||
if (
|
||||
pathname === `/api/apps/${slug}/table-dependencies` &&
|
||||
method === "PUT"
|
||||
) {
|
||||
return jsonResponse({ name: slug, table_ids: [] });
|
||||
}
|
||||
|
||||
throw new Error(`Unexpected ${method} ${pathname}`);
|
||||
});
|
||||
|
||||
await syncResources({
|
||||
appRoot,
|
||||
metabaseUrl: "http://metabase.test",
|
||||
apiKey: "secret",
|
||||
log: jest.fn(),
|
||||
});
|
||||
|
||||
expect(requests.at(-1)).toEqual({
|
||||
method: "PUT",
|
||||
pathname: `/api/apps/${slug}/table-dependencies`,
|
||||
body: JSON.stringify({ table_ids: [] }),
|
||||
});
|
||||
});
|
||||
|
||||
it("stores the tables a copied metric reaches through a foreign key", async () => {
|
||||
const appRoot = makeApp();
|
||||
const slug = path.basename(appRoot);
|
||||
writeQuery(
|
||||
appRoot,
|
||||
`export const Orders = defineQuery({ savedQuestionSourceId: 35, source: { type: "table", id: 1 } });`,
|
||||
);
|
||||
const [query] = await discoverQueries(appRoot);
|
||||
writeQueryLockfile(appRoot, [
|
||||
{ tableId: query.tableId, hash: query.hash, savedQuestionSourceId: 35 },
|
||||
]);
|
||||
|
||||
// Sources table 1 and groups by a table 2 column reached through a foreign
|
||||
// key, so 2 appears nowhere in the query and only the server can report it.
|
||||
const metricQuery = { database: 1, stages: [{ "source-table": 1 }] };
|
||||
const requests: Array<{
|
||||
method: string;
|
||||
pathname: string;
|
||||
body?: string;
|
||||
}> = [];
|
||||
|
||||
jest.spyOn(global, "fetch").mockImplementation(async (input, init) => {
|
||||
const pathname = new URL(String(input)).pathname;
|
||||
const method = init?.method ?? "GET";
|
||||
requests.push({ method, pathname, body: init?.body?.toString() });
|
||||
|
||||
if (pathname === `/api/apps/${slug}/draft` && method === "POST") {
|
||||
return jsonResponse({ name: slug, resource_collection_id: 20 });
|
||||
}
|
||||
if (pathname === `/api/apps/${slug}/query` && method === "POST") {
|
||||
return jsonResponse({
|
||||
database_id: 1,
|
||||
dataset_query: { database: 1 },
|
||||
table_ids: [1],
|
||||
metrics: [
|
||||
{
|
||||
id: 251,
|
||||
name: "Revenue",
|
||||
type: "metric",
|
||||
collection_id: 1,
|
||||
dataset_query: metricQuery,
|
||||
database_id: 1,
|
||||
display: "table",
|
||||
visualization_settings: {},
|
||||
description: null,
|
||||
},
|
||||
],
|
||||
});
|
||||
}
|
||||
if (pathname === "/api/card" && method === "POST") {
|
||||
return jsonResponse({ id: 404 });
|
||||
}
|
||||
if (pathname === "/api/card/35" && method === "GET") {
|
||||
return jsonResponse({
|
||||
id: 35,
|
||||
name: "Orders",
|
||||
type: "question",
|
||||
collection_id: 20,
|
||||
dataset_query: { database: 1 },
|
||||
});
|
||||
}
|
||||
if (
|
||||
pathname === `/api/apps/${slug}/query-table-dependencies` &&
|
||||
method === "POST"
|
||||
) {
|
||||
return jsonResponse({ table_ids: [1, 2] });
|
||||
}
|
||||
if (
|
||||
pathname === `/api/apps/${slug}/table-dependencies` &&
|
||||
method === "PUT"
|
||||
) {
|
||||
return jsonResponse({ name: slug, table_ids: [1, 2] });
|
||||
}
|
||||
|
||||
throw new Error(`Unexpected ${method} ${pathname}`);
|
||||
});
|
||||
|
||||
await syncResources({
|
||||
appRoot,
|
||||
metabaseUrl: "http://metabase.test",
|
||||
apiKey: "secret",
|
||||
log: jest.fn(),
|
||||
});
|
||||
|
||||
expect(requests).toContainEqual({
|
||||
method: "POST",
|
||||
pathname: `/api/apps/${slug}/query-table-dependencies`,
|
||||
body: JSON.stringify({ dataset_queries: [metricQuery] }),
|
||||
});
|
||||
expect(requests.at(-1)).toEqual({
|
||||
method: "PUT",
|
||||
pathname: `/api/apps/${slug}/table-dependencies`,
|
||||
body: JSON.stringify({ table_ids: [1, 2] }),
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,93 @@
|
||||
export interface QueryLockEntry {
|
||||
tableId: number;
|
||||
hash: string;
|
||||
savedQuestionSourceId: number;
|
||||
}
|
||||
|
||||
export interface ActionLockEntry {
|
||||
sourceActionId: number;
|
||||
copiedActionId: number;
|
||||
hash: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* A model copied into the data app collection. `actions` is both the
|
||||
* reference count that keeps the copy alive and the mapping the app's
|
||||
* definitions address.
|
||||
*/
|
||||
export interface ModelLockEntry {
|
||||
sourceModelId: number;
|
||||
copiedModelId: number;
|
||||
hash: string;
|
||||
actions: ActionLockEntry[];
|
||||
}
|
||||
|
||||
export interface MetricLockEntry {
|
||||
sourceMetricId: number;
|
||||
copiedMetricId: number;
|
||||
hash: string;
|
||||
}
|
||||
|
||||
export interface ResourceLockfile {
|
||||
/** Where the copies were last synchronized, so a moved app reads apart from a moved copy. */
|
||||
collectionId?: number;
|
||||
queries: QueryLockEntry[];
|
||||
models: ModelLockEntry[];
|
||||
metrics: MetricLockEntry[];
|
||||
}
|
||||
|
||||
export interface DataAppMetric extends MetabaseCard {
|
||||
type: "metric";
|
||||
}
|
||||
|
||||
export interface DiscoveredQuery {
|
||||
exportName: string;
|
||||
filePath: string;
|
||||
query: Record<string, unknown>;
|
||||
savedQuestionSourceId?: number;
|
||||
tableId: number;
|
||||
hash: string;
|
||||
}
|
||||
|
||||
export interface DiscoveredAction {
|
||||
exportName: string;
|
||||
filePath: string;
|
||||
copiedActionId?: number;
|
||||
sourceActionId: number;
|
||||
}
|
||||
|
||||
export interface DataAppMetadata {
|
||||
name: string;
|
||||
resource_collection_id: number;
|
||||
}
|
||||
|
||||
export interface MetabaseCard {
|
||||
id: number;
|
||||
name: string;
|
||||
type: string;
|
||||
collection_id: number | null;
|
||||
archived?: boolean;
|
||||
dataset_query: Record<string, unknown>;
|
||||
database_id?: number | null;
|
||||
display?: string | null;
|
||||
visualization_settings?: Record<string, unknown> | null;
|
||||
description?: string | null;
|
||||
}
|
||||
|
||||
export interface MetabaseAction {
|
||||
id: number;
|
||||
name: string;
|
||||
type: string;
|
||||
model_id: number;
|
||||
archived?: boolean;
|
||||
description?: string | null;
|
||||
parameters?: unknown[] | null;
|
||||
parameter_mappings?: Record<string, unknown> | null;
|
||||
visualization_settings?: Record<string, unknown> | null;
|
||||
/** Present on implicit actions. */
|
||||
kind?: string | null;
|
||||
/** Present on query actions. */
|
||||
dataset_query?: Record<string, unknown> | null;
|
||||
/** Present on query actions. */
|
||||
database_id?: number | null;
|
||||
}
|
||||
@@ -14,6 +14,8 @@ export { DataAppRouter } from "./components/public/DataAppRouter";
|
||||
export { DataAppLink } from "./components/public/DataAppLink";
|
||||
export { copy } from "./lib/public/copy";
|
||||
export { useDataAppLocation } from "./hooks/public/use-data-app-location";
|
||||
export { defineQuery } from "./hooks/public/use-metabase-query/define-query";
|
||||
export { defineAction } from "./hooks/public/use-action/define-action";
|
||||
export {
|
||||
aggregations,
|
||||
breakout,
|
||||
@@ -25,11 +27,12 @@ export {
|
||||
export type {
|
||||
LocalFieldReference,
|
||||
MetabaseBreakout,
|
||||
MetabaseDynamicColumn,
|
||||
MetabaseDynamicQuery,
|
||||
MetabaseOrderBy,
|
||||
MetabaseQueryOptions,
|
||||
MetabaseQueryObject,
|
||||
OrderByDirection,
|
||||
QuestionColumnReference,
|
||||
UseMetabaseQueryObjectResult,
|
||||
UseMetabaseQueryResult,
|
||||
} from "./hooks/public/use-metabase-query";
|
||||
|
||||
@@ -0,0 +1,13 @@
|
||||
/**
|
||||
* Defines a source-controlled data app action. `action` names the generated
|
||||
* action the app runs. `copiedActionId` is generated state — synchronization
|
||||
* writes it back, so never set or edit it by hand.
|
||||
*/
|
||||
export function defineAction<
|
||||
const TDefinition extends {
|
||||
action: { id: number; parameters: readonly unknown[] };
|
||||
copiedActionId?: number;
|
||||
},
|
||||
>(definition: TDefinition): TDefinition {
|
||||
return definition;
|
||||
}
|
||||
@@ -1,5 +1,4 @@
|
||||
export { useAction } from "./use-action";
|
||||
export type { UseActionResult } from "./use-action";
|
||||
export type {
|
||||
ActionExecuteError,
|
||||
ActionImplicitKind,
|
||||
@@ -15,4 +14,5 @@ export type {
|
||||
ActionResultForUpdate,
|
||||
ActionSchema,
|
||||
AnyActionResult,
|
||||
UseActionResult,
|
||||
} from "./types";
|
||||
|
||||
@@ -9,11 +9,9 @@
|
||||
* schema directly.
|
||||
*/
|
||||
|
||||
import type {
|
||||
RowValue,
|
||||
SchemaColumn,
|
||||
SchemaJavaScriptType,
|
||||
} from "../data-schema";
|
||||
import type { SdkActionId } from "embedding-sdk-bundle/types/action";
|
||||
|
||||
import type { RowValue, SchemaJavaScriptType } from "../data-schema";
|
||||
|
||||
/**
|
||||
* Flat public kind union. Maps onto the backend's namespaced
|
||||
@@ -137,14 +135,9 @@ export type ActionResultForKind<TKind extends ActionKind | undefined> =
|
||||
// ============================================================================
|
||||
// Data App schema-derived helpers
|
||||
// ----------------------------------------------------------------------------
|
||||
// These layer on top of the schema-agnostic types above. They read the shape
|
||||
// of a single action entry in a generated `metabase.data.ts` schema and
|
||||
// derive the hook's `TParameters` / `TKind` generics, so a Data App author
|
||||
// writes `useAction<ActionParametersFromDataAppSchema<typeof action>,
|
||||
// ActionKindFromDataAppSchema<typeof action>>(action.id)` and gets
|
||||
// compile-time parameter checking plus a typed `result` with no casts.
|
||||
// Mode-1 (raw SDK) callers don't import these — they declare `TParameters`
|
||||
// by hand and pass `TKind` as a literal.
|
||||
// These read a single action entry in a generated `metabase.data.ts` schema
|
||||
// and derive the hook's `TParameters` / `TKind`. `useAction` applies them to a
|
||||
// `defineAction(...)` export, so a Data App writes neither generic.
|
||||
// ============================================================================
|
||||
|
||||
/**
|
||||
@@ -163,19 +156,22 @@ export type ActionImplicitKind =
|
||||
|
||||
/**
|
||||
* Shape of a single action parameter as it appears in a generated
|
||||
* `metabase.data.ts` schema entry.
|
||||
* `metabase.data.ts` schema entry. An action parameter is not a column, so it
|
||||
* carries no `name`.
|
||||
*
|
||||
* @category useAction
|
||||
*/
|
||||
export type ActionParameterSchema = SchemaColumn & {
|
||||
export type ActionParameterSchema = {
|
||||
slug: string;
|
||||
displayName?: string;
|
||||
jsType?: SchemaJavaScriptType;
|
||||
required?: boolean;
|
||||
};
|
||||
|
||||
/**
|
||||
* Shape of a single action entry in a generated `metabase.data.ts` schema.
|
||||
* The `id` is the numeric action id passed to `useAction`; `parameters`
|
||||
* drives `ActionParametersFromDataAppSchema`; `implicitKind` / `type` drive
|
||||
* The entry is what `defineAction` wraps; `parameters` drives
|
||||
* `ActionParametersFromDataAppSchema`, and `implicitKind` / `type` drive
|
||||
* `ActionKindFromDataAppSchema`.
|
||||
*
|
||||
* @category useAction
|
||||
@@ -258,3 +254,53 @@ export type ActionKindFromDataAppSchema<TAction> = TAction extends {
|
||||
: TAction extends { type: "query" }
|
||||
? "sql"
|
||||
: ActionKind;
|
||||
|
||||
/**
|
||||
* @interface
|
||||
* @expand
|
||||
* @category useAction
|
||||
*/
|
||||
export type UseActionResult<
|
||||
TParameters extends Record<string, unknown> = Record<string, unknown>,
|
||||
TKind extends ActionKind | undefined = undefined,
|
||||
> = {
|
||||
/**
|
||||
* Trigger the action with the given parameters. Returns the response body
|
||||
* on success AND throws on failure — the same error is stored in `error`
|
||||
* for render-time consumers. Resolves to the discriminated `result` shape
|
||||
* (see `ActionResultForKind<TKind>`); when `TKind` is omitted it resolves
|
||||
* to `AnyActionResult`, narrowable via `"<key>" in r`.
|
||||
*
|
||||
* Resolves to `null` (without making a request) when `actionId` is `null`
|
||||
* or the SDK is not yet initialized — guard the host-side caller with
|
||||
* `if (!actionId) return;` if these cases are reachable.
|
||||
*/
|
||||
execute: (
|
||||
parameters: TParameters,
|
||||
) => Promise<ActionResultForKind<TKind> | null>;
|
||||
isExecuting: boolean;
|
||||
/** Last response, or `null` before first call and after `reset()`. */
|
||||
result: ActionResultForKind<TKind> | null;
|
||||
/** Last thrown error, normalized to the public `ActionExecuteError` shape, or `null`. */
|
||||
error: ActionExecuteError | null;
|
||||
/** Clear `result` and `error`. */
|
||||
reset: () => void;
|
||||
};
|
||||
|
||||
export interface UseAction {
|
||||
/** A `defineAction(...)` export, which types `execute` and `result` itself. */
|
||||
<TDefinition extends { action: ActionSchema }>(
|
||||
action: TDefinition | null,
|
||||
): UseActionResult<
|
||||
ActionParametersFromDataAppSchema<TDefinition["action"]>,
|
||||
ActionKindFromDataAppSchema<TDefinition["action"]>
|
||||
>;
|
||||
|
||||
/** An id, which describes nothing: `useAction<Parameters, "create">(42)`. */
|
||||
<
|
||||
TParameters extends Record<string, unknown> = Record<string, unknown>,
|
||||
TKind extends ActionKind | undefined = undefined,
|
||||
>(
|
||||
actionId: SdkActionId | null,
|
||||
): UseActionResult<TParameters, TKind>;
|
||||
}
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
import { useCallback, useState } from "react";
|
||||
|
||||
import type { ExecuteActionResult } from "embedding-sdk-bundle/lib/execute-action";
|
||||
import type { SdkActionId } from "embedding-sdk-bundle/types/action";
|
||||
import type { SdkActionInput } from "embedding-sdk-bundle/types/action";
|
||||
import { useMetabaseProviderPropsStore } from "embedding-sdk-package/lib/provider-props-store";
|
||||
import { getWindow } from "embedding-sdk-shared/lib/get-window";
|
||||
|
||||
@@ -10,52 +10,12 @@ import type {
|
||||
ActionExecuteError,
|
||||
ActionKind,
|
||||
ActionResultForKind,
|
||||
UseAction,
|
||||
UseActionResult,
|
||||
} from "./types";
|
||||
|
||||
/**
|
||||
* @interface
|
||||
* @expand
|
||||
* @category useAction
|
||||
*/
|
||||
export type UseActionResult<
|
||||
TParameters extends Record<string, unknown> = Record<string, unknown>,
|
||||
TKind extends ActionKind | undefined = undefined,
|
||||
> = {
|
||||
/**
|
||||
* Trigger the action with the given parameters. Returns the response body
|
||||
* on success AND throws on failure — the same error is stored in `error`
|
||||
* for render-time consumers. Resolves to the discriminated `result` shape
|
||||
* (see `ActionResultForKind<TKind>`); when `TKind` is omitted it resolves
|
||||
* to `AnyActionResult`, narrowable via `"<key>" in r`.
|
||||
*
|
||||
* Resolves to `null` (without making a request) when `actionId` is `null`
|
||||
* or the SDK is not yet initialized — guard the host-side caller with
|
||||
* `if (!actionId) return;` if these cases are reachable.
|
||||
*/
|
||||
execute: (
|
||||
parameters: TParameters,
|
||||
) => Promise<ActionResultForKind<TKind> | null>;
|
||||
isExecuting: boolean;
|
||||
/** Last response, or `null` before first call and after `reset()`. */
|
||||
result: ActionResultForKind<TKind> | null;
|
||||
/** Last thrown error, normalized to the public `ActionExecuteError` shape, or `null`. */
|
||||
error: ActionExecuteError | null;
|
||||
/** Clear `result` and `error`. */
|
||||
reset: () => void;
|
||||
};
|
||||
|
||||
/**
|
||||
* Triggers a pre-existing Metabase action. The first arg is the action's
|
||||
* numeric id or its `entity_id` string; supply `TParameters` as the first
|
||||
* generic to type the `execute` argument, and optionally `TKind` as the
|
||||
* second generic to type the discriminated `result` shape.
|
||||
*
|
||||
* Without `TKind`, `result` defaults to a union of every possible response
|
||||
* body (`AnyActionResult`) — authors who don't know the kind upfront can
|
||||
* narrow with `"<key>" in result` instead of casting from
|
||||
* `Record<string, unknown>`.
|
||||
*
|
||||
* useAction<{ name: string; email: string }, "create">(42);
|
||||
* Triggers a pre-existing Metabase action.
|
||||
*
|
||||
* Unlike the query hooks, this does NOT run on mount — the caller invokes
|
||||
* `execute` explicitly from an event handler. To gate execution
|
||||
@@ -65,11 +25,11 @@ export type UseActionResult<
|
||||
* @function
|
||||
* @category useAction
|
||||
*/
|
||||
export const useAction = <
|
||||
const useActionImpl = <
|
||||
TParameters extends Record<string, unknown> = Record<string, unknown>,
|
||||
TKind extends ActionKind | undefined = undefined,
|
||||
>(
|
||||
actionId: SdkActionId | null,
|
||||
actionId: SdkActionInput | null,
|
||||
): UseActionResult<TParameters, TKind> => {
|
||||
const {
|
||||
state: {
|
||||
@@ -129,3 +89,6 @@ export const useAction = <
|
||||
reset,
|
||||
};
|
||||
};
|
||||
|
||||
/** @notExported UseAction */
|
||||
export const useAction: UseAction = useActionImpl;
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
import type { TableSchema } from "../data-schema";
|
||||
|
||||
import type {
|
||||
MetabaseQueryOptions,
|
||||
RequireAggregationsForBreakouts,
|
||||
} from "./types";
|
||||
|
||||
type SavedQuestionBinding = {
|
||||
savedQuestionSourceId?: number;
|
||||
};
|
||||
|
||||
/**
|
||||
* Defines a source-controlled data app query that can be synchronized to a
|
||||
* saved question.
|
||||
*/
|
||||
export function defineQuery<
|
||||
TEntity extends TableSchema | undefined = undefined,
|
||||
TSchema = unknown,
|
||||
const TQuery = MetabaseQueryOptions<TEntity, TSchema>,
|
||||
>(
|
||||
query: TQuery &
|
||||
(TQuery extends SavedQuestionBinding ? unknown : SavedQuestionBinding) &
|
||||
(TQuery extends MetabaseQueryOptions<TEntity, TSchema>
|
||||
? TQuery extends { source: unknown }
|
||||
? RequireAggregationsForBreakouts<TQuery>
|
||||
: unknown
|
||||
: MetabaseQueryOptions<TEntity, TSchema>),
|
||||
): TQuery {
|
||||
return query;
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
import { defineQuery } from "./define-query";
|
||||
|
||||
describe("defineQuery", () => {
|
||||
it("returns the query definition unchanged", () => {
|
||||
const query = {
|
||||
savedQuestionSourceId: 54,
|
||||
source: { type: "table", id: 1 },
|
||||
limit: 10,
|
||||
} as const;
|
||||
|
||||
expect(defineQuery(query)).toBe(query);
|
||||
});
|
||||
});
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user