[autofix.ci] apply automated fixes

DSL export returns a draft_hash covering the graph, features, environment
and conversation variables; an import that sends an older hash fails
instead of wiping newer edits. A blank secret imported over an app keeps
the stored value. The incompatible-node error names the nodes.

New openapi ops: draft test runs, run history with node steps, publish,
version list and restore, and environment variables (secrets masked on
read). Publish shares one service method with the console.

.agents/skills/backend-code-review/references/architecture-rule.md

@@ -1,93 +1,15 @@
-# Rule Catalog — Architecture
+# Architecture review
 
-## Scope
-- Covers: controller/service/core-domain/libs/model layering, dependency direction, responsibility placement, observability-friendly flow.
+The [API agent guide](../../../../api/AGENTS.md) owns package boundaries. In particular, `api/core/` is migration-only: do not propose new files or extracted implementations there.
 
-## Rules
+## Transport and domain boundaries
 
-### Keep business logic out of controllers
-- Category: maintainability
-- Description: Controllers should parse input, call services, and return serialized responses. Business decisions inside controllers make behavior hard to reuse and test.
-- Suggested fix: Move domain/business logic into the service or core/domain layer. Keep controller handlers thin and orchestration-focused.
-- Example:
-  - Bad:
-    ```python
-    @bp.post("/apps/<app_id>/publish")
-    def publish_app(app_id: str):
-        payload = request.get_json() or {}
-        if payload.get("force") and current_user.role != "admin":
-            raise ValueError("only admin can force publish")
-        app = App.query.get(app_id)
-        app.status = "published"
-        db.session.commit()
-        return {"result": "ok"}
-    ```
-  - Good:
-    ```python
-    @bp.post("/apps/<app_id>/publish")
-    def publish_app(app_id: str):
-        payload = PublishRequest.model_validate(request.get_json() or {})
-        app_service.publish_app(app_id=app_id, force=payload.force, actor_id=current_user.id)
-        return {"result": "ok"}
-    ```
+Controllers own request parsing, service calls, and response serialization. Trace business decisions to their service or domain owner; flag controller policy when it duplicates or bypasses that owner. Move extracted behavior to an appropriate owner outside `core/`.
 
-### Preserve layer dependency direction
-- Category: best practices
-- Description: Controllers may depend on services, and services may depend on core/domain abstractions. Reversing this direction (for example, core importing controller/web modules) creates cycles and leaks transport concerns into domain code.
-- Suggested fix: Extract shared contracts into core/domain or service-level modules and make upper layers depend on lower, not the reverse.
-- Example:
-  - Bad:
-    ```python
-    # core/policy/publish_policy.py
-    from controllers.console.app import request_context
+## Dependency direction
 
+Transport layers may depend on services and domain contracts. Domain code must not import controller or request context. Pass validated actor, tenant, and resource data explicitly instead of reaching upward for it. Check existing owners and import-linter contracts before proposing a shared abstraction.
 
-    def can_publish() -> bool:
-        return request_context.current_user.is_admin
-    ```
-  - Good:
-    ```python
-    # core/policy/publish_policy.py
-    def can_publish(role: str) -> bool:
-        return role == "admin"
+## Business-agnostic libraries
 
-
-    # service layer adapts web/user context to domain input
-    allowed = can_publish(role=current_user.role)
-    ```
-
-### Keep libs business-agnostic
-- Category: maintainability
-- Description: Modules under `api/libs/` should remain reusable, business-agnostic building blocks. They must not encode product/domain-specific rules, workflow orchestration, or business decisions.
-- Suggested fix:
-  - If business logic appears in `api/libs/`, extract it into the appropriate `services/` or `core/` module and keep `libs` focused on generic, cross-cutting helpers.
-  - Keep `libs` dependencies clean: avoid importing service/controller/domain-specific modules into `api/libs/`.
-- Example:
-  - Bad:
-    ```python
-    # api/libs/conversation_filter.py
-    from services.conversation_service import ConversationService
-
-
-    def should_archive_conversation(conversation, tenant_id: str) -> bool:
-        # Domain policy and service dependency are leaking into libs.
-        service = ConversationService()
-        if service.has_paid_plan(tenant_id):
-            return conversation.idle_days > 90
-        return conversation.idle_days > 30
-    ```
-  - Good:
-    ```python
-    # api/libs/datetime_utils.py (business-agnostic helper)
-    def older_than_days(idle_days: int, threshold_days: int) -> bool:
-        return idle_days > threshold_days
-
-
-    # services/conversation_service.py (business logic stays in service/core)
-    from libs.datetime_utils import older_than_days
-
-
-    def should_archive_conversation(conversation, tenant_id: str) -> bool:
-        threshold_days = 90 if has_paid_plan(tenant_id) else 30
-        return older_than_days(conversation.idle_days, threshold_days)
-    ```
+`api/libs/` must not own product policy or orchestration. Flag service/controller dependencies and domain-specific decisions in library helpers. Move that behavior to its existing service or domain owner outside `core/`; retain only reusable infrastructure in `libs/`.

.agents/skills/backend-code-review/references/repositories-rule.md

@@ -11,7 +11,7 @@
 - Description: If a table/model already has a repository abstraction, all reads/writes/queries for that table should use the existing repository. If no repository exists, introduce one only when complexity justifies it, such as large/high-volume tables, repeated complex query logic, or likely storage-strategy variation.
 - Suggested fix:
   - First check  `api/repositories`, `api/core/repositories`, and `api/extensions/*/repositories/` to verify whether the table/model already has a repository abstraction. If it exists, route all operations through it and add missing repository methods instead of bypassing it with ad-hoc SQLAlchemy access.
-  - If no repository exists, add one only when complexity warrants it (for example, repeated complex queries, large data domains, or multiple storage strategies), while preserving dependency direction (service/core depends on abstraction; infra provides implementation).
+  - If no repository exists, add one only when complexity warrants it (for example, repeated complex queries, large data domains, or multiple storage strategies). Follow `api/AGENTS.md`: put new abstractions and implementations outside migration-only `api/core/`, with services/domain code depending on abstractions and infrastructure providing implementations.
 - Example:
   - Bad:
     ```python

.devcontainer/README.md

@@ -1,7 +1,7 @@
 # Development with devcontainer
 
 This project includes a devcontainer configuration that allows you to open the project in a container with a fully configured development environment.
-Both frontend and backend environments are initialized when the container is started.
+The create hook installs workspace JavaScript dependencies and `uv`; the start hook synchronizes backend dependencies. Follow the [backend setup](../api/README.md) and [frontend guide](../web/README.md) to configure environment files and start services.
 
 ## GitHub Codespaces
 
@@ -19,20 +19,6 @@ if you have VS Code installed, you can click the button above to open this proje
 
 You can learn more in the [Dev Containers documentation](https://code.visualstudio.com/docs/devcontainers/containers).
 
-## Pros of Devcontainer
-
-Unified Development Environment: By using devcontainers, you can ensure that all developers are developing in the same environment, reducing the occurrence of "it works on my machine" type of issues.
-
-Quick Start: New developers can set up their development environment in a few simple steps, without spending a lot of time on environment configuration.
-
-Isolation: Devcontainers isolate your project from your host operating system, reducing the chance of OS updates or other application installations impacting the development environment.
-
-## Cons of Devcontainer
-
-Learning Curve: For developers unfamiliar with Docker and VS Code, using devcontainers may be somewhat complex.
-
-Performance Impact: While usually minimal, programs running inside a devcontainer may be slightly slower than those running directly on the host.
-
 ## Troubleshooting
 
 if you see such error message when you open this project in codespaces:

.gitattributes

@@ -9,3 +9,6 @@
 # Codegen output must stay byte-identical across platforms so
 # `pnpm tree:check` in CI does not trip on CRLF rewrites.
 *.generated.ts  text eol=lf
+
+# Skill scripts run on every platform, so keep LF line endings.
+/skills/**      text=auto eol=lf

.github/CODEOWNERS

@@ -57,13 +57,10 @@
 
 # Backend - RAG (Retrieval Augmented Generation)
 /api/core/rag/ @JohnJyong
+/api/services/knowledge/ @JohnJyong
 /api/services/rag_pipeline/ @JohnJyong
-/api/services/dataset_service.py @JohnJyong
 /api/services/knowledge_service.py @JohnJyong
-/api/services/external_knowledge_service.py @JohnJyong
 /api/services/hit_testing_service.py @JohnJyong
-/api/services/metadata_service.py @JohnJyong
-/api/services/vector_service.py @JohnJyong
 /api/services/entities/knowledge_entities/ @JohnJyong
 /api/services/entities/external_knowledge_entities/ @JohnJyong
 /api/controllers/console/datasets/ @JohnJyong

.github/actions/setup-core-e2e/action.yml

@@ -15,7 +15,7 @@ runs:
         run-install: 'false'
 
     - name: Setup UV and Python
-      uses: astral-sh/setup-uv@bec219d24cd3e171d82865faccec33120bb574f4 # v10.1.0
+      uses: astral-sh/setup-uv@c18668ad3cf93ea998bef934396af7bb5c839dc7 # v10.2.0
       with:
         enable-cache: true
         python-version: '3.12'

.github/actions/setup-web/action.yml

@@ -13,11 +13,11 @@ runs:
   using: composite
   steps:
     - name: Setup pnpm and Node.js
-      uses: pnpm/setup@703c52620218391530e48b9e8870d5c0082e1b9b # v2.1.0
+      uses: pnpm/setup@fbda4c85fc2e1e08721cd8763afea8f48d60f024 # v3.0.0
       with:
         install: false
     - name: Setup Vite+
-      uses: voidzero-dev/setup-vp@9fd26ff49f9b5e6276c2b493da4454ad4527f7f4 # v1.20.0
+      uses: voidzero-dev/setup-vp@3754dd7dbdb32bd8f6d28b6043de13ad3a75f21f # v1.21.1
       with:
         node-manager: false
         package-manager: |

.github/actions/wait-e2e-web-build/download.cjs

@@ -0,0 +1,104 @@
+const { execFile } = require('node:child_process')
+const { appendFile, mkdtemp, rm } = require('node:fs/promises')
+const os = require('node:os')
+const path = require('node:path')
+const { promisify } = require('node:util')
+const waitForBuild = require('./wait.cjs')
+
+const execFileAsync = promisify(execFile)
+
+async function downloadWebBuild({
+  env = process.env,
+  execute = execFileAsync,
+  webDir = path.resolve(__dirname, '../../../web'),
+} = {}) {
+  const repository = env.GITHUB_REPOSITORY
+  const runId = env.GITHUB_RUN_ID
+  const runAttempt = Number(env.GITHUB_RUN_ATTEMPT)
+  if (
+    !repository ||
+    !/^\d+$/.test(runId || '') ||
+    !Number.isInteger(runAttempt) ||
+    runAttempt < 1 ||
+    !env.GH_TOKEN
+  ) {
+    throw new Error(
+      'Downloading the Web build requires GITHUB_REPOSITORY, GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, and GH_TOKEN.',
+    )
+  }
+
+  const run = (command, args, timeout = 60_000) =>
+    execute(command, args, {
+      env,
+      timeout,
+      maxBuffer: 32 * 1024 * 1024,
+    })
+  const api = async (endpoint, paginate = false) => {
+    const { stdout } = await run('gh', [
+      'api',
+      endpoint,
+      ...(paginate ? ['--paginate', '--slurp'] : []),
+    ])
+    return JSON.parse(stdout)
+  }
+  const prefix = `repos/${repository}/actions`
+  let summary = ''
+  const artifactId = await waitForBuild({
+    github: {
+      rest: {
+        actions: { listJobsForWorkflowRunAttempt: 'jobs', listWorkflowRunArtifacts: 'artifacts' },
+      },
+      paginate: async (kind) => {
+        const endpoint =
+          kind === 'jobs'
+            ? `${prefix}/runs/${runId}/attempts/${runAttempt}/jobs?per_page=100`
+            : `${prefix}/runs/${runId}/artifacts?per_page=100`
+        const pages = await api(endpoint, true)
+        return pages.flatMap((page) => page[kind])
+      },
+    },
+    context: { repo: {}, runId },
+    core: {
+      info: console.log,
+      summary: {
+        addHeading(text) {
+          summary += `### ${text}\n\n`
+          return this
+        },
+        addRaw(text) {
+          summary += `${text}\n`
+          return this
+        },
+        async write() {
+          if (env.GITHUB_STEP_SUMMARY) await appendFile(env.GITHUB_STEP_SUMMARY, summary)
+        },
+      },
+    },
+    runAttempt,
+  })
+
+  const artifact = await api(`${prefix}/artifacts/${artifactId}`)
+  const directory = await mkdtemp(path.join(os.tmpdir(), 'dify-e2e-web-build-'))
+  try {
+    // Artifact names are immutable within a run. Use the name selected by wait.cjs,
+    // including its matching upload timestamp when only consumer jobs are rerun.
+    await run(
+      'gh',
+      ['run', 'download', runId, '--repo', repository, '--name', artifact.name, '--dir', directory],
+      300_000,
+    )
+    await run('tar', ['-xf', path.join(directory, 'e2e-web-build.tar'), '-C', webDir])
+    console.log(`Web build artifact ${artifactId} extracted and ready.`)
+  } finally {
+    await rm(directory, { force: true, recursive: true })
+  }
+}
+
+module.exports = downloadWebBuild
+
+if (require.main === module) {
+  downloadWebBuild().catch((error) => {
+    console.error(error)
+    process.exitCode = 1
+  })
+}

.github/actions/wait-e2e-web-build/download.test.cjs

@@ -0,0 +1,112 @@
+const assert = require('node:assert/strict')
+const { access } = require('node:fs/promises')
+const { test } = require('node:test')
+const downloadWebBuild = require('./download.cjs')
+
+function fixture({ downloadFails = false, producerFails = false } = {}) {
+  const calls = []
+  const env = {
+    GITHUB_REPOSITORY: 'owner/repo',
+    GITHUB_RUN_ID: '123',
+    GITHUB_RUN_ATTEMPT: '4',
+    GH_TOKEN: 'test-token',
+  }
+  const execute = async (command, args) => {
+    calls.push({ command, args })
+    if (command === 'tar') return { stdout: '' }
+    if (args[0] === 'run') {
+      if (downloadFails) throw new Error('artifact download failed')
+      return { stdout: '' }
+    }
+    if (args[1].includes('/jobs?')) {
+      return {
+        stdout: JSON.stringify([
+          {
+            jobs: [
+              {
+                name: 'Run Web Full-Stack E2E / Prepare Core E2E Web Build',
+                status: 'completed',
+                conclusion: producerFails ? 'failure' : 'success',
+                steps: [
+                  {
+                    name: 'Upload Web build',
+                    conclusion: 'success',
+                    started_at: '2026-09-22T04:08:08Z',
+                    completed_at: '2026-09-22T04:08:12Z',
+                  },
+                ],
+              },
+            ],
+          },
+        ]),
+      }
+    }
+    if (args[1].includes('/artifacts?')) {
+      return {
+        stdout: JSON.stringify([
+          { artifacts: [] },
+          {
+            artifacts: [
+              {
+                id: 42,
+                name: 'core-e2e-web-build-2',
+                created_at: '2026-09-22T04:08:11Z',
+                expired: false,
+              },
+            ],
+          },
+        ]),
+      }
+    }
+    assert.equal(args[1], 'repos/owner/repo/actions/artifacts/42')
+    return { stdout: JSON.stringify({ name: 'core-e2e-web-build-2' }) }
+  }
+  return { calls, options: { env, execute, webDir: '/test/web' } }
+}
+
+test('downloads the producer-selected artifact on consumer-only reruns, including paginated results', async () => {
+  const { calls, options } = fixture()
+  await downloadWebBuild(options)
+  assert.match(calls[0].args[1], /runs\/123\/attempts\/4\/jobs/)
+  const download = calls.find(({ args }) => args[0] === 'run')
+  assert.deepEqual(download.args.slice(0, 7), [
+    'run',
+    'download',
+    '123',
+    '--repo',
+    'owner/repo',
+    '--name',
+    'core-e2e-web-build-2',
+  ])
+  const extract = calls.at(-1)
+  assert.equal(extract.command, 'tar')
+  assert.deepEqual(extract.args.slice(-2), ['-C', '/test/web'])
+  await assert.rejects(access(download.args.at(-1)), { code: 'ENOENT' })
+})
+
+test('does not extract a failed download and removes its temporary directory', async () => {
+  const { calls, options } = fixture({ downloadFails: true })
+  await assert.rejects(downloadWebBuild(options), /artifact download failed/)
+  assert.equal(
+    calls.some(({ command }) => command === 'tar'),
+    false,
+  )
+  await assert.rejects(access(calls.at(-1).args.at(-1)), { code: 'ENOENT' })
+})
+
+test('does not download or extract when the producer fails', async () => {
+  const { calls, options } = fixture({ producerFails: true })
+  await assert.rejects(downloadWebBuild(options), /Web build finished with failure/)
+  assert.equal(calls.length, 1)
+})
+
+test('requires CI metadata and credentials before starting preparation', async () => {
+  const { calls, options } = fixture()
+  for (const key of Object.keys(options.env)) {
+    await assert.rejects(
+      downloadWebBuild({ ...options, env: { ...options.env, [key]: '' } }),
+      /requires GITHUB/,
+    )
+  }
+  assert.equal(calls.length, 0)
+})

.github/workflows/api-tests.yml

@@ -117,7 +117,7 @@ jobs:
             -p no:benchmark \
             --timeout "${PYTEST_TIMEOUT:-20}" \
             -n auto \
-            -o 'addopts=--cov=./api --cov-branch --import-mode=importlib' \
+            -o 'addopts=-p no:opik -p no:langsmith_plugin --cov=./api --cov-branch --import-mode=importlib' \
             --cov-report= \
             --file-shard-plan unit-test-plan.json \
             --shard-index "${{ matrix.shardIndex }}" \
@@ -181,7 +181,7 @@ jobs:
         run: |
           uv run --project api pytest \
             --timeout "${PYTEST_TIMEOUT:-20}" \
-            -o 'addopts=--cov=./api --cov-branch --import-mode=importlib' \
+            -o 'addopts=-p no:opik -p no:langsmith_plugin --cov=./api --cov-branch --import-mode=importlib' \
             --cov-report= \
             --junitxml=test-results/controllers.xml \
             api/tests/unit_tests/controllers
@@ -246,7 +246,7 @@ jobs:
             --start-middleware \
             --middleware-stop-timeout 2 \
             -n auto \
-            -o 'addopts=--cov=./api --cov-branch --import-mode=importlib' \
+            -o 'addopts=-p no:opik -p no:langsmith_plugin --cov=./api --cov-branch --import-mode=importlib' \
             --cov-report= \
             --timeout "${PYTEST_TIMEOUT:-180}" \
             api/tests/integration_tests/workflow \

.github/workflows/cli-e2e.yml

@@ -78,7 +78,7 @@ jobs:
       ws2_app_id: ${{ steps.out.outputs.DIFY_E2E_WS2_APP_ID }}
 
     steps:
-      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v4
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
         with:
           ref: ${{ inputs.cli_ref || github.ref }}
           persist-credentials: false
@@ -121,7 +121,7 @@ jobs:
         shell: bash
 
     steps:
-      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v4
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
         with:
           ref: ${{ inputs.cli_ref || github.ref }}
           persist-credentials: false
@@ -166,7 +166,7 @@ jobs:
         shell: bash
 
     steps:
-      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v4
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
         with:
           ref: ${{ inputs.cli_ref || github.ref }}
           persist-credentials: false
@@ -227,7 +227,7 @@ jobs:
         shell: bash
 
     steps:
-      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v4
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
         with:
           ref: ${{ inputs.cli_ref || github.ref }}
           persist-credentials: false
@@ -266,7 +266,7 @@ jobs:
 
       - name: Upload results on failure
         if: failure()
-        uses: actions/upload-artifact@v7
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
         with:
           name: e2e-run-${{ matrix.name }}-${{ github.run_id }}
           path: cli/test-results/
@@ -287,7 +287,7 @@ jobs:
         shell: bash
 
     steps:
-      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v4
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
         with:
           ref: ${{ inputs.cli_ref || github.ref }}
           persist-credentials: false
@@ -341,7 +341,7 @@ jobs:
         shell: bash
 
     steps:
-      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v4
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
         with:
           ref: ${{ inputs.cli_ref || github.ref }}
           persist-credentials: false
@@ -396,7 +396,7 @@ jobs:
 
       - name: Upload results on failure
         if: failure()
-        uses: actions/upload-artifact@v7
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
         with:
           name: e2e-last-${{ github.run_id }}
           path: cli/test-results/

.github/workflows/cli-edge.yml

@@ -5,6 +5,7 @@ on:
     branches: [main]
     paths:
       - 'cli/**'
+      - 'skills/**'
       - 'packages/contracts/generated/api/openapi/**'
   workflow_dispatch:
 

.github/workflows/cli-release.yml

@@ -77,11 +77,6 @@ jobs:
           echo "dify_tag=${tag}" >> "$GITHUB_OUTPUT"
           echo "::notice::target Dify release ${tag}"
 
-      - name: Compatibility check
-        env:
-          DIFY_TAG: ${{ steps.resolve.outputs.dify_tag }}
-        run: node scripts/release-naming.mjs compat-check "$DIFY_TAG"
-
   release:
     name: build + attach standalone binaries (all targets)
     needs: validate

.github/workflows/cli-smoke.yml

@@ -51,7 +51,7 @@ jobs:
 
       - name: Run smoke against live dify
         working-directory: ./cli
-        run: pnpm exec node scripts/run-smoke.ts --base-url http://localhost:5001
+        run: DIFY_SERVER=http://localhost:5001 bun scripts/run-smoke.ts
 
       - name: Dump dify logs on failure
         if: failure()

.github/workflows/main-ci.yml

@@ -21,9 +21,10 @@ concurrency:
   cancel-in-progress: ${{ github.event_name == 'merge_group' }}
 
 jobs:
+  # Keep orchestration independent of the Depot pool used for test execution.
   pre_job:
     name: Skip Duplicate Checks
-    runs-on: depot-ubuntu-24.04
+    runs-on: ubuntu-24.04
     outputs:
       should_skip: ${{ steps.skip_check.outputs.should_skip || 'false' }}
     steps:
@@ -37,7 +38,7 @@ jobs:
   # Check which paths were changed to determine which tests to run
   check-changes:
     name: Check Changed Files
-    runs-on: depot-ubuntu-24.04
+    runs-on: ubuntu-24.04
     outputs:
       api-changed: ${{ steps.changes.outputs.api }}
       cli-changed: ${{ steps.changes.outputs.cli }}
@@ -66,6 +67,7 @@ jobs:
               - 'scripts/lint_controller_sqlalchemy.py'
               - 'scripts/ast_grep_rules/no_new_getattr.yml'
               - 'scripts/ast_grep_rules/no_new_controller_sqlalchemy.yml'
+              - 'scripts/ast_grep_rules/no_new_session_mock.yml'
               - '.github/workflows/style.yml'
               - '.github/workflows/main-ci.yml'
               - '.github/workflows/api-tests.yml'
@@ -78,6 +80,7 @@ jobs:
               - 'docker/volumes/sandbox/conf/**'
             cli:
               - 'cli/**'
+              - 'skills/**'
               - 'packages/contracts/**'
               - 'packages/tsconfig/**'
               - 'package.json'
@@ -160,9 +163,11 @@ jobs:
               - 'scripts/ast_grep_guard.py'
               - 'scripts/check_no_new_getattr.py'
               - 'scripts/check_no_new_controller_sqlalchemy.py'
+              - 'scripts/check_no_new_session_mock.py'
               - 'scripts/lint_controller_sqlalchemy.py'
               - 'scripts/ast_grep_rules/no_new_getattr.yml'
               - 'scripts/ast_grep_rules/no_new_controller_sqlalchemy.yml'
+              - 'scripts/ast_grep_rules/no_new_session_mock.yml'
               - '.github/workflows/style.yml'
               - '.github/workflows/main-ci.yml'
             dify-agent-style:
@@ -187,6 +192,7 @@ jobs:
             ts-common-style:
               - 'web/**'
               - 'cli/**'
+              - 'skills/**'
               - 'e2e/**'
               - 'sdks/nodejs-client/**'
               - 'packages/**'
@@ -212,7 +218,8 @@ jobs:
               - '.editorconfig'
               - '.vite-hooks/**'
 
-  # Run tests in parallel while always emitting stable required checks.
+  # Finalizers report unchanged paths directly, avoiding separate skip-only jobs.
+  # Keep each required check name stable and fail closed if classification fails.
   api-tests-run:
     name: Run API Tests
     needs:
@@ -222,17 +229,6 @@ jobs:
     uses: ./.github/workflows/api-tests.yml
     secrets: inherit
 
-  api-tests-skip:
-    name: Skip API Tests
-    needs:
-      - pre_job
-      - check-changes
-    if: needs.pre_job.outputs.should_skip != 'true' && needs.check-changes.outputs.api-changed != 'true'
-    runs-on: depot-ubuntu-24.04
-    steps:
-      - name: Report skipped API tests
-        run: echo "No API-related changes detected; skipping API tests."
-
   api-tests:
     name: API Tests
     if: ${{ always() }}
@@ -240,21 +236,26 @@ jobs:
       - pre_job
       - check-changes
       - api-tests-run
-      - api-tests-skip
-    runs-on: depot-ubuntu-24.04
+    runs-on: ubuntu-24.04
     steps:
       - name: Finalize API Tests status
         env:
           SHOULD_SKIP_WORKFLOW: ${{ needs.pre_job.outputs.should_skip }}
           TESTS_CHANGED: ${{ needs.check-changes.outputs.api-changed }}
           RUN_RESULT: ${{ needs.api-tests-run.result }}
-          SKIP_RESULT: ${{ needs.api-tests-skip.result }}
+          PRE_JOB_RESULT: ${{ needs.pre_job.result }}
+          CHANGES_RESULT: ${{ needs.check-changes.result }}
         run: |
           if [[ "$SHOULD_SKIP_WORKFLOW" == 'true' ]]; then
             echo "API tests were skipped because this workflow run duplicated a successful or newer run."
             exit 0
           fi
 
+          if [[ "$PRE_JOB_RESULT" != 'success' || "$CHANGES_RESULT" != 'success' ]]; then
+            echo "Cannot finalize tests: duplicate check=$PRE_JOB_RESULT, change detection=$CHANGES_RESULT" >&2
+            exit 1
+          fi
+
           if [[ "$TESTS_CHANGED" == 'true' ]]; then
             if [[ "$RUN_RESULT" == 'success' ]]; then
               echo "API tests ran successfully."
@@ -265,12 +266,12 @@ jobs:
             exit 1
           fi
 
-          if [[ "$SKIP_RESULT" == 'success' ]]; then
+          if [[ "$TESTS_CHANGED" == 'false' && "$RUN_RESULT" == 'skipped' ]]; then
             echo "API tests were skipped because no API-related files changed."
             exit 0
           fi
 
-          echo "API tests were not required, but the skip job finished with result: $SKIP_RESULT" >&2
+          echo "Unexpected test state: changed=$TESTS_CHANGED, result=$RUN_RESULT" >&2
           exit 1
 
   cli-tests-run:
@@ -282,17 +283,6 @@ jobs:
     uses: ./.github/workflows/cli-tests.yml
     secrets: inherit
 
-  cli-tests-skip:
-    name: Skip CLI Tests
-    needs:
-      - pre_job
-      - check-changes
-    if: needs.pre_job.outputs.should_skip != 'true' && needs.check-changes.outputs.cli-changed != 'true'
-    runs-on: depot-ubuntu-24.04
-    steps:
-      - name: Report skipped CLI tests
-        run: echo "No CLI-related changes detected; skipping CLI tests."
-
   cli-tests:
     name: CLI Tests
     if: ${{ always() }}
@@ -300,21 +290,26 @@ jobs:
       - pre_job
       - check-changes
       - cli-tests-run
-      - cli-tests-skip
-    runs-on: depot-ubuntu-24.04
+    runs-on: ubuntu-24.04
     steps:
       - name: Finalize CLI Tests status
         env:
           SHOULD_SKIP_WORKFLOW: ${{ needs.pre_job.outputs.should_skip }}
           TESTS_CHANGED: ${{ needs.check-changes.outputs.cli-changed }}
           RUN_RESULT: ${{ needs.cli-tests-run.result }}
-          SKIP_RESULT: ${{ needs.cli-tests-skip.result }}
+          PRE_JOB_RESULT: ${{ needs.pre_job.result }}
+          CHANGES_RESULT: ${{ needs.check-changes.result }}
         run: |
           if [[ "$SHOULD_SKIP_WORKFLOW" == 'true' ]]; then
             echo "CLI tests were skipped because this workflow run duplicated a successful or newer run."
             exit 0
           fi
 
+          if [[ "$PRE_JOB_RESULT" != 'success' || "$CHANGES_RESULT" != 'success' ]]; then
+            echo "Cannot finalize tests: duplicate check=$PRE_JOB_RESULT, change detection=$CHANGES_RESULT" >&2
+            exit 1
+          fi
+
           if [[ "$TESTS_CHANGED" == 'true' ]]; then
             if [[ "$RUN_RESULT" == 'success' ]]; then
               echo "CLI tests ran successfully."
@@ -325,12 +320,12 @@ jobs:
             exit 1
           fi
 
-          if [[ "$SKIP_RESULT" == 'success' ]]; then
+          if [[ "$TESTS_CHANGED" == 'false' && "$RUN_RESULT" == 'skipped' ]]; then
             echo "CLI tests were skipped because no CLI-related files changed."
             exit 0
           fi
 
-          echo "CLI tests were not required, but the skip job finished with result: $SKIP_RESULT" >&2
+          echo "Unexpected test state: changed=$TESTS_CHANGED, result=$RUN_RESULT" >&2
           exit 1
 
   web-tests-run:
@@ -342,17 +337,6 @@ jobs:
     uses: ./.github/workflows/web-tests.yml
     secrets: inherit
 
-  web-tests-skip:
-    name: Skip Web Tests
-    needs:
-      - pre_job
-      - check-changes
-    if: needs.pre_job.outputs.should_skip != 'true' && needs.check-changes.outputs.web-changed != 'true'
-    runs-on: depot-ubuntu-24.04
-    steps:
-      - name: Report skipped web tests
-        run: echo "No web-related changes detected; skipping web tests."
-
   web-tests:
     name: Web Tests
     if: ${{ always() }}
@@ -360,21 +344,26 @@ jobs:
       - pre_job
       - check-changes
       - web-tests-run
-      - web-tests-skip
-    runs-on: depot-ubuntu-24.04
+    runs-on: ubuntu-24.04
     steps:
       - name: Finalize Web Tests status
         env:
           SHOULD_SKIP_WORKFLOW: ${{ needs.pre_job.outputs.should_skip }}
           TESTS_CHANGED: ${{ needs.check-changes.outputs.web-changed }}
           RUN_RESULT: ${{ needs.web-tests-run.result }}
-          SKIP_RESULT: ${{ needs.web-tests-skip.result }}
+          PRE_JOB_RESULT: ${{ needs.pre_job.result }}
+          CHANGES_RESULT: ${{ needs.check-changes.result }}
         run: |
           if [[ "$SHOULD_SKIP_WORKFLOW" == 'true' ]]; then
             echo "Web tests were skipped because this workflow run duplicated a successful or newer run."
             exit 0
           fi
 
+          if [[ "$PRE_JOB_RESULT" != 'success' || "$CHANGES_RESULT" != 'success' ]]; then
+            echo "Cannot finalize tests: duplicate check=$PRE_JOB_RESULT, change detection=$CHANGES_RESULT" >&2
+            exit 1
+          fi
+
           if [[ "$TESTS_CHANGED" == 'true' ]]; then
             if [[ "$RUN_RESULT" == 'success' ]]; then
               echo "Web tests ran successfully."
@@ -385,12 +374,12 @@ jobs:
             exit 1
           fi
 
-          if [[ "$SKIP_RESULT" == 'success' ]]; then
+          if [[ "$TESTS_CHANGED" == 'false' && "$RUN_RESULT" == 'skipped' ]]; then
             echo "Web tests were skipped because no web-related files changed."
             exit 0
           fi
 
-          echo "Web tests were not required, but the skip job finished with result: $SKIP_RESULT" >&2
+          echo "Unexpected test state: changed=$TESTS_CHANGED, result=$RUN_RESULT" >&2
           exit 1
 
   web-e2e-run:
@@ -404,17 +393,6 @@ jobs:
       run-external-runtime: false
     secrets: inherit
 
-  web-e2e-skip:
-    name: Skip Web Full-Stack E2E
-    needs:
-      - pre_job
-      - check-changes
-    if: needs.pre_job.outputs.should_skip != 'true' && needs.check-changes.outputs.e2e-changed != 'true'
-    runs-on: depot-ubuntu-24.04
-    steps:
-      - name: Report skipped web full-stack e2e
-        run: echo "No E2E-related changes detected; skipping web full-stack E2E."
-
   web-e2e:
     name: Web Full-Stack E2E
     if: ${{ always() }}
@@ -422,21 +400,26 @@ jobs:
       - pre_job
       - check-changes
       - web-e2e-run
-      - web-e2e-skip
-    runs-on: depot-ubuntu-24.04
+    runs-on: ubuntu-24.04
     steps:
       - name: Finalize Web Full-Stack E2E status
         env:
           SHOULD_SKIP_WORKFLOW: ${{ needs.pre_job.outputs.should_skip }}
           TESTS_CHANGED: ${{ needs.check-changes.outputs.e2e-changed }}
           RUN_RESULT: ${{ needs.web-e2e-run.result }}
-          SKIP_RESULT: ${{ needs.web-e2e-skip.result }}
+          PRE_JOB_RESULT: ${{ needs.pre_job.result }}
+          CHANGES_RESULT: ${{ needs.check-changes.result }}
         run: |
           if [[ "$SHOULD_SKIP_WORKFLOW" == 'true' ]]; then
             echo "Web full-stack E2E was skipped because this workflow run duplicated a successful or newer run."
             exit 0
           fi
 
+          if [[ "$PRE_JOB_RESULT" != 'success' || "$CHANGES_RESULT" != 'success' ]]; then
+            echo "Cannot finalize tests: duplicate check=$PRE_JOB_RESULT, change detection=$CHANGES_RESULT" >&2
+            exit 1
+          fi
+
           if [[ "$TESTS_CHANGED" == 'true' ]]; then
             if [[ "$RUN_RESULT" == 'success' ]]; then
               echo "Web full-stack E2E ran successfully."
@@ -447,12 +430,12 @@ jobs:
             exit 1
           fi
 
-          if [[ "$SKIP_RESULT" == 'success' ]]; then
+          if [[ "$TESTS_CHANGED" == 'false' && "$RUN_RESULT" == 'skipped' ]]; then
             echo "Web full-stack E2E was skipped because no E2E-related files changed."
             exit 0
           fi
 
-          echo "Web full-stack E2E was not required, but the skip job finished with result: $SKIP_RESULT" >&2
+          echo "Unexpected test state: changed=$TESTS_CHANGED, result=$RUN_RESULT" >&2
           exit 1
 
   style-check:
@@ -480,17 +463,6 @@ jobs:
     if: needs.pre_job.outputs.should_skip != 'true' && needs.check-changes.outputs.vdb-changed == 'true'
     uses: ./.github/workflows/vdb-tests.yml
 
-  vdb-tests-skip:
-    name: Skip VDB Tests
-    needs:
-      - pre_job
-      - check-changes
-    if: needs.pre_job.outputs.should_skip != 'true' && needs.check-changes.outputs.vdb-changed != 'true'
-    runs-on: depot-ubuntu-24.04
-    steps:
-      - name: Report skipped VDB tests
-        run: echo "No VDB-related changes detected; skipping VDB tests."
-
   vdb-tests:
     name: VDB Tests
     if: ${{ always() }}
@@ -498,21 +470,26 @@ jobs:
       - pre_job
       - check-changes
       - vdb-tests-run
-      - vdb-tests-skip
-    runs-on: depot-ubuntu-24.04
+    runs-on: ubuntu-24.04
     steps:
       - name: Finalize VDB Tests status
         env:
           SHOULD_SKIP_WORKFLOW: ${{ needs.pre_job.outputs.should_skip }}
           TESTS_CHANGED: ${{ needs.check-changes.outputs.vdb-changed }}
           RUN_RESULT: ${{ needs.vdb-tests-run.result }}
-          SKIP_RESULT: ${{ needs.vdb-tests-skip.result }}
+          PRE_JOB_RESULT: ${{ needs.pre_job.result }}
+          CHANGES_RESULT: ${{ needs.check-changes.result }}
         run: |
           if [[ "$SHOULD_SKIP_WORKFLOW" == 'true' ]]; then
             echo "VDB tests were skipped because this workflow run duplicated a successful or newer run."
             exit 0
           fi
 
+          if [[ "$PRE_JOB_RESULT" != 'success' || "$CHANGES_RESULT" != 'success' ]]; then
+            echo "Cannot finalize tests: duplicate check=$PRE_JOB_RESULT, change detection=$CHANGES_RESULT" >&2
+            exit 1
+          fi
+
           if [[ "$TESTS_CHANGED" == 'true' ]]; then
             if [[ "$RUN_RESULT" == 'success' ]]; then
               echo "VDB tests ran successfully."
@@ -523,12 +500,12 @@ jobs:
             exit 1
           fi
 
-          if [[ "$SKIP_RESULT" == 'success' ]]; then
+          if [[ "$TESTS_CHANGED" == 'false' && "$RUN_RESULT" == 'skipped' ]]; then
             echo "VDB tests were skipped because no VDB-related files changed."
             exit 0
           fi
 
-          echo "VDB tests were not required, but the skip job finished with result: $SKIP_RESULT" >&2
+          echo "Unexpected test state: changed=$TESTS_CHANGED, result=$RUN_RESULT" >&2
           exit 1
 
   db-migration-test-run:
@@ -539,17 +516,6 @@ jobs:
     if: needs.pre_job.outputs.should_skip != 'true' && needs.check-changes.outputs.migration-changed == 'true'
     uses: ./.github/workflows/db-migration-test.yml
 
-  db-migration-test-skip:
-    name: Skip DB Migration Test
-    needs:
-      - pre_job
-      - check-changes
-    if: needs.pre_job.outputs.should_skip != 'true' && needs.check-changes.outputs.migration-changed != 'true'
-    runs-on: depot-ubuntu-24.04
-    steps:
-      - name: Report skipped DB migration tests
-        run: echo "No migration-related changes detected; skipping DB migration tests."
-
   db-migration-test:
     name: DB Migration Test
     if: ${{ always() }}
@@ -557,21 +523,26 @@ jobs:
       - pre_job
       - check-changes
       - db-migration-test-run
-      - db-migration-test-skip
-    runs-on: depot-ubuntu-24.04
+    runs-on: ubuntu-24.04
     steps:
       - name: Finalize DB Migration Test status
         env:
           SHOULD_SKIP_WORKFLOW: ${{ needs.pre_job.outputs.should_skip }}
           TESTS_CHANGED: ${{ needs.check-changes.outputs.migration-changed }}
           RUN_RESULT: ${{ needs.db-migration-test-run.result }}
-          SKIP_RESULT: ${{ needs.db-migration-test-skip.result }}
+          PRE_JOB_RESULT: ${{ needs.pre_job.result }}
+          CHANGES_RESULT: ${{ needs.check-changes.result }}
         run: |
           if [[ "$SHOULD_SKIP_WORKFLOW" == 'true' ]]; then
             echo "DB migration tests were skipped because this workflow run duplicated a successful or newer run."
             exit 0
           fi
 
+          if [[ "$PRE_JOB_RESULT" != 'success' || "$CHANGES_RESULT" != 'success' ]]; then
+            echo "Cannot finalize tests: duplicate check=$PRE_JOB_RESULT, change detection=$CHANGES_RESULT" >&2
+            exit 1
+          fi
+
           if [[ "$TESTS_CHANGED" == 'true' ]]; then
             if [[ "$RUN_RESULT" == 'success' ]]; then
               echo "DB migration tests ran successfully."
@@ -582,12 +553,12 @@ jobs:
             exit 1
           fi
 
-          if [[ "$SKIP_RESULT" == 'success' ]]; then
+          if [[ "$TESTS_CHANGED" == 'false' && "$RUN_RESULT" == 'skipped' ]]; then
             echo "DB migration tests were skipped because no migration-related files changed."
             exit 0
           fi
 
-          echo "DB migration tests were not required, but the skip job finished with result: $SKIP_RESULT" >&2
+          echo "Unexpected test state: changed=$TESTS_CHANGED, result=$RUN_RESULT" >&2
           exit 1
 
   sandbox-runtime-tests-run:
@@ -599,17 +570,6 @@ jobs:
     uses: ./.github/workflows/sandbox-runtime-tests.yml
     secrets: inherit
 
-  sandbox-runtime-tests-skip:
-    name: Skip Sandbox Runtime Tests
-    needs:
-      - pre_job
-      - check-changes
-    if: needs.pre_job.outputs.should_skip != 'true' && needs.check-changes.outputs.sandbox-runtime-changed != 'true'
-    runs-on: depot-ubuntu-24.04
-    steps:
-      - name: Report skipped sandbox runtime tests
-        run: echo "No sandbox-runtime-related changes detected; skipping sandbox runtime tests."
-
   sandbox-runtime-tests:
     name: Sandbox Runtime Tests
     if: ${{ always() }}
@@ -617,21 +577,26 @@ jobs:
       - pre_job
       - check-changes
       - sandbox-runtime-tests-run
-      - sandbox-runtime-tests-skip
-    runs-on: depot-ubuntu-24.04
+    runs-on: ubuntu-24.04
     steps:
       - name: Finalize Sandbox Runtime Tests status
         env:
           SHOULD_SKIP_WORKFLOW: ${{ needs.pre_job.outputs.should_skip }}
           TESTS_CHANGED: ${{ needs.check-changes.outputs.sandbox-runtime-changed }}
           RUN_RESULT: ${{ needs.sandbox-runtime-tests-run.result }}
-          SKIP_RESULT: ${{ needs.sandbox-runtime-tests-skip.result }}
+          PRE_JOB_RESULT: ${{ needs.pre_job.result }}
+          CHANGES_RESULT: ${{ needs.check-changes.result }}
         run: |
           if [[ "$SHOULD_SKIP_WORKFLOW" == 'true' ]]; then
             echo "Sandbox runtime tests were skipped because this workflow run duplicated a successful or newer run."
             exit 0
           fi
 
+          if [[ "$PRE_JOB_RESULT" != 'success' || "$CHANGES_RESULT" != 'success' ]]; then
+            echo "Cannot finalize tests: duplicate check=$PRE_JOB_RESULT, change detection=$CHANGES_RESULT" >&2
+            exit 1
+          fi
+
           if [[ "$TESTS_CHANGED" == 'true' ]]; then
             if [[ "$RUN_RESULT" == 'success' ]]; then
               echo "Sandbox runtime tests ran successfully."
@@ -642,12 +607,12 @@ jobs:
             exit 1
           fi
 
-          if [[ "$SKIP_RESULT" == 'success' ]]; then
+          if [[ "$TESTS_CHANGED" == 'false' && "$RUN_RESULT" == 'skipped' ]]; then
             echo "Sandbox runtime tests were skipped because no sandbox-runtime-related files changed."
             exit 0
           fi
 
-          echo "Sandbox runtime tests were not required, but the skip job finished with result: $SKIP_RESULT" >&2
+          echo "Unexpected test state: changed=$TESTS_CHANGED, result=$RUN_RESULT" >&2
           exit 1
 
   dify-agent-tests-run:
@@ -659,17 +624,6 @@ jobs:
     uses: ./.github/workflows/dify-agent-tests.yml
     secrets: inherit
 
-  dify-agent-tests-skip:
-    name: Skip Dify Agent Tests
-    needs:
-      - pre_job
-      - check-changes
-    if: needs.pre_job.outputs.should_skip != 'true' && needs.check-changes.outputs.dify-agent-changed != 'true'
-    runs-on: depot-ubuntu-24.04
-    steps:
-      - name: Report skipped dify agent tests
-        run: echo "No dify-agent-related changes detected; skipping dify agent tests."
-
   dify-agent-tests:
     name: Dify Agent Tests
     if: ${{ always() }}
@@ -677,21 +631,26 @@ jobs:
       - pre_job
       - check-changes
       - dify-agent-tests-run
-      - dify-agent-tests-skip
-    runs-on: depot-ubuntu-24.04
+    runs-on: ubuntu-24.04
     steps:
       - name: Finalize Dify Agent Tests status
         env:
           SHOULD_SKIP_WORKFLOW: ${{ needs.pre_job.outputs.should_skip }}
           TESTS_CHANGED: ${{ needs.check-changes.outputs.dify-agent-changed }}
           RUN_RESULT: ${{ needs.dify-agent-tests-run.result }}
-          SKIP_RESULT: ${{ needs.dify-agent-tests-skip.result }}
+          PRE_JOB_RESULT: ${{ needs.pre_job.result }}
+          CHANGES_RESULT: ${{ needs.check-changes.result }}
         run: |
           if [[ "$SHOULD_SKIP_WORKFLOW" == 'true' ]]; then
             echo "Dify agent tests were skipped because this workflow run duplicated a successful or newer run."
             exit 0
           fi
 
+          if [[ "$PRE_JOB_RESULT" != 'success' || "$CHANGES_RESULT" != 'success' ]]; then
+            echo "Cannot finalize tests: duplicate check=$PRE_JOB_RESULT, change detection=$CHANGES_RESULT" >&2
+            exit 1
+          fi
+
           if [[ "$TESTS_CHANGED" == 'true' ]]; then
             if [[ "$RUN_RESULT" == 'success' ]]; then
               echo "Dify agent tests ran successfully."
@@ -702,10 +661,10 @@ jobs:
             exit 1
           fi
 
-          if [[ "$SKIP_RESULT" == 'success' ]]; then
+          if [[ "$TESTS_CHANGED" == 'false' && "$RUN_RESULT" == 'skipped' ]]; then
             echo "Dify agent tests were skipped because no dify-agent-related files changed."
             exit 0
           fi
 
-          echo "Dify agent tests were not required, but the skip job finished with result: $SKIP_RESULT" >&2
+          echo "Unexpected test state: changed=$TESTS_CHANGED, result=$RUN_RESULT" >&2
           exit 1

.github/workflows/style.yml

@@ -89,6 +89,10 @@ jobs:
         if: inputs.run-python-style
         run: uv run --project api python scripts/check_no_new_controller_sqlalchemy.py --base-rev "${{ inputs.base-rev }}"
 
+      - name: Run No New Session Mock Guard
+        if: inputs.run-python-style
+        run: uv run --project api python scripts/check_no_new_session_mock.py --base-rev "${{ inputs.base-rev }}"
+
       - name: Run Type Checks
         if: inputs.run-python-style
         env:

.github/workflows/web-e2e.yml

@@ -26,7 +26,7 @@ jobs:
           persist-credentials: false
 
       - name: Test Web build synchronization
-        run: node --test .github/actions/wait-e2e-web-build/wait.test.cjs
+        run: node --test .github/actions/wait-e2e-web-build/*.test.cjs
 
       - name: Compute build source hash
         id: build-source
@@ -119,24 +119,12 @@ jobs:
         with:
           browser: chromium
 
-      # Install dependencies while core-build runs, then wait only for its artifact.
-      - name: Wait for Web build
-        id: web-build
-        timeout-minutes: 35
-        uses: ./.github/actions/wait-e2e-web-build
-
-      - name: Download Web build
-        uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
-        with:
-          artifact-ids: ${{ steps.web-build.outputs.artifact-id }}
-          github-token: ${{ github.token }}
-
-      - name: Extract Web build
-        run: tar -xf e2e-web-build.tar -C web
-
+      # The runner prepares the backend while waiting for and downloading the build.
       - name: Run Chromium full shard
         working-directory: ./e2e
-        run: vp run e2e:full -- --shard ${{ matrix.shard }}/3
+        env:
+          GH_TOKEN: ${{ github.token }}
+        run: vp run e2e:full --download-web-build -- --shard ${{ matrix.shard }}/3
 
       - name: Upload Cucumber report
         if: ${{ !cancelled() }}
@@ -183,24 +171,12 @@ jobs:
         with:
           browser: webkit
 
-      # Install dependencies while core-build runs, then wait only for its artifact.
-      - name: Wait for Web build
-        id: web-build
-        timeout-minutes: 35
-        uses: ./.github/actions/wait-e2e-web-build
-
-      - name: Download Web build
-        uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
-        with:
-          artifact-ids: ${{ steps.web-build.outputs.artifact-id }}
-          github-token: ${{ github.token }}
-
-      - name: Extract Web build
-        run: tar -xf e2e-web-build.tar -C web
-
+      # Keep the same overlap and teardown ownership for the WebKit smoke lane.
       - name: Run WebKit browser smoke
         working-directory: ./e2e
-        run: vp run e2e:full -- --tags '@browser-smoke'
+        env:
+          GH_TOKEN: ${{ github.token }}
+        run: vp run e2e:full --download-web-build -- --tags '@browser-smoke'
 
       - name: Upload Cucumber report
         if: ${{ !cancelled() }}

.vscode/README.md

@@ -1,14 +1,11 @@
 # Debugging with VS Code
 
-This `launch.json.template` file provides various debug configurations for the Dify project within VS Code / Cursor. To use these configurations, you should copy the contents of this file into a new file named `launch.json` in the same `.vscode` directory.
+[`launch.json.template`](launch.json.template) provides API and Celery worker debug configurations for VS Code / Cursor. Both use `api/.venv/bin/python`; complete the [backend setup](../api/README.md) first.
 
 ## How to Use
 
-1. **Create `launch.json`**: If you don't have one, create a file named `launch.json` inside the `.vscode` directory.
-1. **Copy Content**: Copy the entire content from `launch.json.template` into your newly created `launch.json` file.
+1. Copy `.vscode/launch.json.template` to `.vscode/launch.json`, or merge the configurations into an existing file.
 1. **Select Debug Configuration**: Go to the Run and Debug view in VS Code / Cursor (Ctrl+Shift+D or Cmd+Shift+D).
 1. **Start Debugging**: Select the desired configuration from the dropdown menu and click the green play button.
 
-## Tips
-
-- If you need to debug with Edge browser instead of Chrome, modify the `serverReadyAction` configuration in the "Next.js: debug full stack" section, change `"debugWithChrome"` to `"debugWithEdge"` to use Microsoft Edge for debugging.
+Start the frontend separately using the [frontend guide](../web/README.md).

CONTRIBUTING.md

@@ -81,24 +81,13 @@ AI tools are welcome, but you are responsible for understanding and verifying ev
 
 #### Frontend
 
-For setting up the frontend service, please refer to our comprehensive [guide](https://github.com/langgenius/dify/blob/main/web/README.md) in the `web/README.md` file. This document provides detailed instructions to help you set up the frontend environment properly.
+For setting up the frontend service, please refer to our comprehensive [guide](web/README.md) in the `web/README.md` file. This document provides detailed instructions to help you set up the frontend environment properly.
 
-**Testing**: Add focused tests when a change affects observable behavior or carries meaningful regression risk. See [web/docs/test.md](https://github.com/langgenius/dify/blob/main/web/docs/test.md) for the canonical frontend testing guidelines.
+**Testing**: Add focused tests when a change affects observable behavior or carries meaningful regression risk. See [web/docs/test.md](web/docs/test.md) for the canonical frontend testing guidelines.
 
 #### Backend
 
-For setting up the backend service, kindly refer to our detailed [instructions](https://github.com/langgenius/dify/blob/main/api/README.md) in the `api/README.md` file. This document contains step-by-step guidance to help you get the backend up and running smoothly.
-
-#### Other things to note
-
-We recommend reviewing this document carefully before proceeding with the setup, as it contains essential information about:
-
-- Prerequisites and dependencies
-- Installation steps
-- Configuration details
-- Common troubleshooting tips
-
-Feel free to reach out if you encounter any issues during the setup process.
+For setting up the backend service, kindly refer to our detailed [instructions](api/README.md) in the `api/README.md` file. This document contains step-by-step guidance to help you get the backend up and running smoothly.
 
 ## Getting Help
 

README.md

@@ -83,7 +83,7 @@ docker compose up -d
 
 After running, you can access the Dify dashboard in your browser at [http://localhost/install](http://localhost/install) and start the initialization process.
 
-#### Seeking help
+### Seeking help
 
 Please refer to our [FAQ](https://docs.dify.ai/getting-started/install-self-hosted/faqs) if you encounter problems setting up Dify. Reach out to [the community and us](#community--contact) if you are still having issues.
 
@@ -116,7 +116,7 @@ All of Dify's offerings come with corresponding APIs, so you could effortlessly
 ## Using Dify
 
 - **Cloud <br/>**
-  We host a [Dify Cloud](https://dify.ai) service for anyone to try with zero setup. It provides all the capabilities of the self-deployed version, and includes 200 free GPT-4 calls in the sandbox plan. If you run into issues with Dify Cloud, [contact our Cloud support team](mailto:cloud@dify.ai?subject=%5BGitHub%5DDify%20Cloud%20Support).
+  We host [Dify Cloud](https://cloud.dify.ai) so you can try Dify without managing a server. See the [pricing page](https://dify.ai/pricing) for current plans and usage allowances. If you run into issues, [contact our Cloud support team](mailto:cloud@dify.ai?subject=%5BGitHub%5DDify%20Cloud%20Support).
 
 - **Self-hosting Dify Community Edition<br/>**
   Quickly get Dify running in your environment with this [starter guide](#quick-start).
@@ -171,7 +171,7 @@ Choose the channel that best fits your question:
 
 ## Security disclosure
 
-To protect your privacy, please avoid posting security issues on GitHub. Instead, report issues to security@dify.ai, and our team will respond with detailed answer.
+Report vulnerabilities privately through GitHub Security Advisories, following our [security policy](SECURITY.md). Do not disclose them in public issues, discussions, or pull requests.
 
 ## License
 

api/.env.example

@@ -156,6 +156,7 @@ ARCHIVE_STORAGE_EXPORT_BUCKET=
 ARCHIVE_STORAGE_ACCESS_KEY=
 ARCHIVE_STORAGE_SECRET_KEY=
 ARCHIVE_STORAGE_REGION=auto
+ARCHIVE_STORAGE_ADDRESS_STYLE=path
 
 # Azure Blob Storage configuration
 AZURE_BLOB_ACCOUNT_NAME=your-account-name

api/.importlinter

@@ -1,12 +1,14 @@
 [importlinter]
 root_packages =
     core
+    core.rag.extractor.entity
     constants
     context
     configs
     controllers
     extensions
     factories
+    fields
     libs
     machinery
     models
@@ -16,7 +18,7 @@ root_packages =
 include_external_packages = True
 
 [importlinter:contract:backend-layers]
-name = Backend modules follow the transport, application, domain, and shared-library layers
+name = Backend modules follow the transport, application, execution-kernel, and shared-library layers
 type = layers
 layers =
     controllers
@@ -27,7 +29,7 @@ layers =
 # error makes this list shrink whenever a legacy dependency is removed.
 unmatched_ignore_imports_alerting = error
 ignore_imports =
-    core.app.app_config.easy_ui_based_app.dataset.manager -> services.dataset_service
+    core.app.app_config.easy_ui_based_app.dataset.manager -> services.knowledge.dataset_service
     core.app.apps.advanced_chat.app_generator -> controllers.console.app.workflow
     core.app.apps.advanced_chat.app_generator -> services.conversation_service
     core.app.apps.advanced_chat.app_generator -> services.errors.conversation
@@ -48,45 +50,44 @@ ignore_imports =
     core.app.apps.message_based_app_generator -> services.errors.conversation
     core.app.apps.message_based_app_generator -> services.errors.message
     core.app.apps.pipeline.pipeline_generator -> services
-    core.app.apps.pipeline.pipeline_generator -> services.dataset_service
-    core.app.apps.pipeline.pipeline_generator -> services.datasource_provider_service
+    core.app.apps.pipeline.pipeline_generator -> services.knowledge.dataset_service
+    core.app.apps.pipeline.pipeline_generator -> services.data_source.provider_service
     core.app.apps.pipeline.pipeline_generator -> services.feature_service
     core.app.apps.pipeline.pipeline_generator -> services.workflow_draft_variable_service
-    core.app.apps.pipeline.pipeline_runner -> services.dataset_ref_service
     core.app.apps.workflow.app_generator -> controllers.console.app.workflow
     core.app.apps.workflow.app_generator -> services.snippet_generate_service
     core.app.apps.workflow.app_generator -> services.workflow_draft_variable_service
     core.app.apps.workflow_app_runner -> tasks.mail_human_input_delivery_task
     core.app.features.annotation_reply.annotation_reply -> services.annotation_service
-    core.app.features.annotation_reply.annotation_reply -> services.dataset_service
+    core.app.features.annotation_reply.annotation_reply -> services.knowledge.dataset_service
     core.app.llm.quota -> services.credit_pool_service
     core.app.task_pipeline.message_cycle_manager -> services.account_service
     core.app.task_pipeline.message_cycle_manager -> services.annotation_service
     core.app.workflow.layers.persistence -> services.workflow.inspector_events
-    core.datasource.datasource_manager -> services.datasource_provider_service
     core.helper.credential_utils -> services.enterprise.plugin_manager_service
     core.helper.credential_utils -> services.system_feature_service
-    core.indexing_runner -> services.vector_space_admission_service
     core.mcp.auth_client -> services
-    core.provider_manager -> services.credential_permission_service
     core.provider_manager -> services.credit_pool_service
     core.provider_manager -> services.feature_service
-    core.rag.datasource.retrieval_service -> services.external_knowledge_service
+    core.rag.datasource.retrieval_service -> services.knowledge.external.service
     core.rag.index_processor.index_processor -> services.vector_space_admission_service
     core.rag.index_processor.index_processor_base -> services.file_service
     core.rag.index_processor.processor.paragraph_index_processor -> services.account_service
-    core.rag.index_processor.processor.paragraph_index_processor -> services.summary_index_service
+    core.rag.index_processor.processor.paragraph_index_processor -> services.knowledge.summaries.adapters
     core.rag.index_processor.processor.parent_child_index_processor -> services.account_service
-    core.rag.index_processor.processor.parent_child_index_processor -> services.summary_index_service
-    core.rag.index_processor.processor.qa_index_processor -> services.summary_index_service
-    core.rag.retrieval.dataset_retrieval -> services.external_knowledge_service
+    core.rag.index_processor.processor.parent_child_index_processor -> services.knowledge.summaries.adapters
+    core.rag.index_processor.processor.qa_index_processor -> services.knowledge.summaries.adapters
+    core.rag.retrieval.dataset_retrieval -> services.knowledge.external.service
+    core.rag.retrieval.dataset_retrieval -> services.knowledge.retrieval.attachments
     core.rag.retrieval.dataset_retrieval -> services.feature_service
-    core.rag.summary_index.summary_index -> services.summary_index_service
+    core.rag.summary_index.summary_index -> services.knowledge.summaries.adapters
     core.rag.summary_index.summary_index -> tasks.generate_summary_index_task
     core.repositories.sqlalchemy_workflow_node_execution_repository -> services.file_service
     core.repositories.sqlalchemy_workflow_node_execution_repository -> services.variable_truncator
     core.tools.tool_manager -> services
     core.workflow.node_runtime -> services
+    # Node construction delegates datasource dependency assembly to the composition root.
+    core.workflow.node_factory -> extensions.application_services.data_sources
     core.workflow.nodes.agent.message_transformer -> services
     core.workflow.nodes.agent_v2.agent_node -> services.agent.prompt_mentions
     core.workflow.nodes.agent_v2.agent_node -> services.agent.workspace_service
@@ -99,7 +100,7 @@ ignore_imports =
     core.workflow.nodes.agent_v2.workspace_retirement_layer -> tasks.collect_agent_resources_task
     libs.email_i18n -> services.entities.feature_entities
     libs.email_i18n -> services.system_feature_service
-    libs.external_api -> core
+    libs.external_api -> core.plugin
     libs.external_api -> core.errors.error
     libs.external_api -> extensions.ext_logging
     libs.flask_utils -> models
@@ -168,6 +169,141 @@ forbidden_modules =
     sqlalchemy
     werkzeug
 
+[importlinter:contract:dataset-application-boundary]
+name = Dataset application services and contracts are framework and persistence neutral
+type = forbidden
+source_modules =
+    core.rag.extractor.entity.extract_setting
+    services.knowledge.entities.segments
+    services.knowledge.entities.datasets
+    services.knowledge.entities.document_creation
+    services.data_source.binding_application_service
+    services.data_source.credential_gateway
+    services.data_source.notion_import_application_service
+    services.data_source.entities.credential
+    services.data_source.entities.notion_import
+    services.knowledge.entities.indexing_estimate
+    services.knowledge.dataset_access
+    services.knowledge.datasets.application
+    services.knowledge.datasets.retrieval
+    services.knowledge.document_sync
+    services.knowledge.documents.application
+    services.knowledge.external.application
+    services.knowledge.indexing.errors
+    services.knowledge.indexing.estimate
+    services.knowledge.indexing.execution
+    services.knowledge.resource_scope
+    services.knowledge.segments.application
+    services.knowledge.metadata.application
+    services.knowledge.summaries.application
+forbidden_modules =
+    fields
+    configs
+    controllers
+    extensions
+    flask
+    models
+    repositories
+    sqlalchemy
+    tasks
+    werkzeug
+    services.knowledge.dataset_service
+    services.data_source.provider_service
+    services.knowledge.indexing.adapters
+    services.knowledge.segments.adapters
+    services.knowledge.segments.indexing
+    core.datasource.datasource_manager
+    core.rag.extractor.notion_extractor
+
+[importlinter:contract:knowledge-repository-boundary]
+name = Knowledge persistence adapters only depend on their consumer-owned application contracts
+type = forbidden
+source_modules =
+    repositories.knowledge
+forbidden_modules =
+    controllers
+    extensions
+    flask
+    services
+    tasks
+    werkzeug
+allow_indirect_imports = True
+ignore_imports =
+    repositories.knowledge.dataset_api_key_repository -> services.auth.api_key_contracts
+    repositories.knowledge.dataset_api_key_repository -> services.knowledge.api_key_service
+    repositories.knowledge.dataset_repository -> services.knowledge.segments.application
+    repositories.knowledge.dataset_repository -> services.knowledge.dataset_access
+    repositories.knowledge.dataset_repository -> services.knowledge.indexing.estimate
+    repositories.knowledge.dataset_repository -> services.knowledge.resource_scope
+    repositories.knowledge.document_repository -> services.knowledge.document_sync
+    repositories.knowledge.document_repository -> services.knowledge.indexing.estimate
+    repositories.knowledge.document_repository -> services.knowledge.indexing.errors
+    repositories.knowledge.document_repository -> services.knowledge.indexing.execution
+    repositories.knowledge.document_repository -> services.knowledge.resource_scope
+    repositories.knowledge.segment_repository -> services.knowledge.resource_scope
+    repositories.knowledge.segment_repository -> services.knowledge.indexing.execution
+    repositories.knowledge.segment_repository -> services.knowledge.segments.application
+    repositories.knowledge.segment_repository -> services.knowledge.entities.segments
+    repositories.knowledge.metadata_repository -> services.errors.metadata
+    repositories.knowledge.metadata_repository -> services.knowledge.metadata.application
+    repositories.knowledge.metadata_repository -> services.knowledge.resource_scope
+    repositories.knowledge.summary_repository -> services.knowledge.summaries.application
+
+[importlinter:contract:datasource-credential-repository-boundary]
+name = Data source credential persistence does not depend on SQLAlchemy service helpers
+type = forbidden
+source_modules =
+    repositories.credentials.query_repository
+    repositories.data_source.credential_repository
+forbidden_modules =
+    services.data_source.provider_service
+allow_indirect_imports = True
+
+[importlinter:contract:datasource-provider-persistence-boundary]
+name = Datasource provider service consumes injected persistence contracts
+type = forbidden
+source_modules =
+    services.data_source.provider_service
+forbidden_modules =
+    repositories
+    extensions.application_services
+    extensions.ext_application_services
+allow_indirect_imports = True
+
+[importlinter:contract:credential-permission-query-policy-boundary]
+name = Credential query contracts do not expose persistence implementations
+type = forbidden
+source_modules =
+    services.credentials.query
+forbidden_modules =
+    repositories
+    sqlalchemy
+    models
+    extensions
+    flask
+allow_indirect_imports = True
+
+[importlinter:contract:credential-query-implementation-boundary]
+name = Credential query implementations are wired only by composition or other repositories
+type = forbidden
+source_modules =
+    controllers
+    services
+    core
+forbidden_modules =
+    repositories.credentials.query_repository
+allow_indirect_imports = True
+
+[importlinter:contract:pipeline-document-lookup-boundary]
+name = Pipeline core depends on its document store port
+type = forbidden
+source_modules =
+    core.app.apps.pipeline.pipeline_generator
+    core.app.apps.pipeline.pipeline_runner
+forbidden_modules =
+    repositories
+allow_indirect_imports = True
+
 [importlinter:contract:init-validation-service-boundary]
 name = Initialization validation application service is framework and persistence neutral
 type = forbidden
@@ -479,7 +615,7 @@ forbidden_modules =
 name = Data source OAuth application service is framework and persistence neutral
 type = forbidden
 source_modules =
-    services.data_source_oauth_service
+    services.data_source.oauth_service
 forbidden_modules =
     configs
     controllers
@@ -697,6 +833,20 @@ forbidden_modules =
     sqlalchemy
     werkzeug
 
+[importlinter:contract:recommended-app-package-service-boundary]
+name = Recommended package application service is framework and persistence neutral
+type = forbidden
+source_modules =
+    services.recommended_app_package_service
+forbidden_modules =
+    controllers
+    extensions
+    flask
+    models
+    repositories
+    sqlalchemy
+    werkzeug
+
 [importlinter:contract:recommended-app-catalog-gateway-boundary]
 name = Recommended app catalog gateway does not depend on Flask
 type = forbidden
@@ -786,6 +936,86 @@ forbidden_modules =
     tasks
     werkzeug
 
+[importlinter:contract:knowledge-segment-presentation-boundary]
+name = Knowledge operations do not serialize HTTP responses
+type = forbidden
+allow_indirect_imports = True
+source_modules =
+    services.knowledge.segments
+    services.knowledge.documents
+    services.knowledge.datasets
+    services.knowledge.external
+forbidden_modules =
+    controllers
+    fields
+
+[importlinter:contract:knowledge-response-persistence-boundary]
+name = Knowledge response schemas do not load database state
+type = forbidden
+allow_indirect_imports = True
+source_modules =
+    fields.dataset_fields
+    fields.document_fields
+    fields.segment_fields
+forbidden_modules =
+    sqlalchemy
+    models
+    repositories
+    services.knowledge
+
+[importlinter:contract:dataset-model-infrastructure-boundary]
+name = Dataset models do not execute infrastructure or application operations
+type = forbidden
+allow_indirect_imports = True
+source_modules =
+    models.dataset
+forbidden_modules =
+    repositories
+    services
+    extensions
+    core.tools
+    models.engine
+
+[importlinter:contract:app-mcp-server-service-boundary]
+name = App MCP server application service is framework and persistence neutral
+type = forbidden
+source_modules =
+    services.app.mcp_server_service
+forbidden_modules =
+    configs
+    controllers
+    extensions
+    flask
+    models
+    repositories
+    sqlalchemy
+    werkzeug
+
+[importlinter:contract:console-app-mcp-server-controller-boundary]
+name = Console App MCP server controller reaches persistence only through the composition root
+type = forbidden
+source_modules =
+    controllers.console.app.mcp_server
+forbidden_modules =
+    extensions.ext_database
+    models.model
+    repositories
+    sqlalchemy
+# The composition root (extensions.ext_application_services) wires repositories,
+# so only direct imports are checked here.
+allow_indirect_imports = True
+
+[importlinter:contract:console-app-wraps-boundary]
+name = Console app decorators load App rows without reaching into repositories
+type = forbidden
+source_modules =
+    controllers.console.app.wraps
+forbidden_modules =
+    repositories
+# models.model / models.workflow lazily import repositories.factory (legacy debt),
+# so only direct imports are checked here.
+allow_indirect_imports = True
+
 [importlinter:contract:app-audio-boundary]
 name = App audio operations and values are framework and persistence neutral
 type = forbidden

api/AGENTS.md

@@ -15,7 +15,8 @@ Run direct Python commands through `uv run --project api`. Docker-backed integra
 
 ## Architecture And Boundaries
 
-- Keep transport parsing and serialization in controllers, orchestration in services, and domain policy in `core/` or its domain owner. Keep `libs/` business-agnostic and reuse existing owners before adding abstractions.
+- Treat `api/core/` as a migration-only area: do not add files or move implementations into it. Move responsibilities out; limit changes to existing callers to the adaptations needed for that migration.
+- Keep transport parsing and serialization in controllers, orchestration in services, and domain policy in its domain owner. New or extracted implementations must stay outside `core/`. Keep `libs/` business-agnostic and reuse existing owners before adding abstractions.
 - Before changing controller schemas, generated API contracts, or `SystemFeatureModel`, read `controllers/API_SCHEMA_GUIDE.md`. Treat `/system-features` as a minimal unauthenticated bootstrap allowlist, not a general configuration registry.
 - Scope tenant-owned reads and writes by the complete owner chain, and propagate `tenant_id` across every affected layer. Reconstruct trusted internal references from validated database state after payload or async boundaries.
 - Keep write transactions explicit and bounded. Do not perform external I/O inside an open transaction unless a documented consistency contract requires it.

api/README.md

@@ -2,19 +2,11 @@
 
 ## Setup and Run
 
-> [!IMPORTANT]
->
-> In the v1.3.0 release, `poetry` has been replaced with
-> [`uv`](https://docs.astral.sh/uv/) as the package manager
-> for Dify API backend service.
-
-`uv` and `pnpm` are required to run the setup and development commands below.
+[`uv`](https://docs.astral.sh/uv/), Docker Compose, and the Node.js and pnpm versions pinned in the root `package.json` are required. Run the commands below from the repository root, with each long-running service in its own terminal.
 
 ### Using scripts (recommended)
 
-The scripts resolve paths relative to their location, so you can run them from anywhere.
-
-1. Run setup (copies env files and installs dependencies).
+1. Run setup for a new checkout (copies env files and installs dependencies). This overwrites `api/.env`, `web/.env.local`, and `docker/middleware.env`; preserve existing configuration before rerunning it.
 
    ```bash
    ./dev/setup
@@ -44,7 +36,7 @@ The scripts resolve paths relative to their location, so you can run them from a
 
 1. Set up your application by visiting `http://localhost:3000`.
 
-1. Start the worker service (async and scheduler tasks, runs from `api`).
+1. Start the worker service (executes queued async tasks).
 
    ```bash
    ./dev/start-worker
@@ -62,51 +54,35 @@ The scripts resolve paths relative to their location, so you can run them from a
 >
 > When the frontend and backend run on different subdomains, set COOKIE_DOMAIN to the site’s top-level domain (e.g., `example.com`). The frontend and backend must be under the same top-level domain in order to share authentication cookies.
 
-- Generate a `SECRET_KEY` in the `.env` file.
-
-  bash for Linux
-
-  ```bash
-  sed -i "/^SECRET_KEY=/c\\SECRET_KEY=$(openssl rand -base64 42)" .env
-  ```
-
-  bash for Mac
-
-  ```bash
-  secret_key=$(openssl rand -base64 42)
-  sed -i '' "/^SECRET_KEY=/c\\
-  SECRET_KEY=${secret_key}" .env
-  ```
+Leave `SECRET_KEY` empty to let Dify generate a persistent key in the storage directory. To manage it explicitly, generate a value with `openssl rand -base64 42` and set `SECRET_KEY` in `api/.env`.
 
 ## Testing
 
-1. Install dependencies for both the backend and the test environment
+Run from the repository root:
 
-   ```bash
-   cd api
-   uv sync --group dev
-   ```
-
-1. Run the tests locally with mocked system environment variables in `tool.pytest_env` section in `pyproject.toml`, more can check [AGENTS.md](../AGENTS.md)
-
-   Continue in the `api` directory from the previous step.
-
-   ```bash
-   uv run pytest                           # Run all tests
-   uv run pytest tests/unit_tests/         # Unit tests only
-   uv run pytest tests/integration_tests/  # Integration tests
-
-   # Code quality
-   ../dev/reformat              # Run all formatters and linters
-   uv run ruff check --fix ./   # Fix linting issues
-   uv run ruff format ./        # Format code
-   uv run pyrefly check         # Type checking
-   ```
-
-## Generate TS stub
-
-```
-uv run dev/generate_swagger_specs.py --output-dir openapi
+```bash
+uv sync --project api --group dev
+make test
+make test TARGET_TESTS=./api/tests/unit_tests/<path>
+make lint
+make type-check
 ```
 
-use https://jsontotable.org/openapi-to-typescript to convert to typescript
+`make test` includes provider unit tests and runs controller tests separately. Integration suites are CI-only and are not expected to run locally. Test environment defaults live in `api/pyproject.toml` under `tool.pytest_env`; see the backend [agent guide](AGENTS.md) for package conventions.
+
+## API contracts
+
+The [API schema guide](controllers/API_SCHEMA_GUIDE.md) owns schema changes and verification. Generate the OpenAPI specifications and TypeScript/Zod contracts through the workspace script:
+
+```bash
+pnpm -C packages/contracts gen-api-contract
+```
+
+Refresh the checked-in Markdown reference with:
+
+```bash
+uv run --project api python api/dev/generate_swagger_markdown_docs.py \
+  --swagger-dir packages/contracts/openapi --markdown-dir api/openapi/markdown --keep-swagger-json
+```
+
+Update the backend schema owner and regenerate contracts instead of converting types manually or editing generated files.

api/commands/vector.py

@@ -19,6 +19,7 @@ from models.dataset import Dataset, DatasetCollectionBinding, DatasetMetadata, D
 from models.dataset import Document as DatasetDocument
 from models.enums import DatasetMetadataType, IndexingStatus, SegmentStatus
 from models.model import App, AppAnnotationSetting, MessageAnnotation
+from repositories.knowledge.dataset_read_repository import get_segment_child_chunks
 
 
 @click.command("vdb-migrate", help="Migrate vector db.")
@@ -278,7 +279,7 @@ def migrate_knowledge_vector_database():
                             },
                         )
                         if dataset_document.doc_form == IndexStructureType.PARENT_CHILD_INDEX:
-                            child_chunks = segment.get_child_chunks(session=db_session)
+                            child_chunks = get_segment_child_chunks(segment, session=db_session)
                             if child_chunks:
                                 child_documents = []
                                 for child_chunk in child_chunks:

api/configs/extra/archive_config.py

@@ -1,3 +1,5 @@
+from typing import Literal
+
 from pydantic import Field
 from pydantic_settings import BaseSettings
 
@@ -41,3 +43,8 @@ class ArchiveStorageConfig(BaseSettings):
         description="Region for storage (use 'auto' if the provider supports it)",
         default="auto",
     )
+
+    ARCHIVE_STORAGE_ADDRESS_STYLE: Literal["path", "virtual", "auto"] = Field(
+        description="S3 addressing style for archive storage: 'path' (default), 'virtual' (virtual-hosted), or 'auto'",
+        default="path",
+    )

api/controllers/common/rbac/__init__.py

@@ -4,6 +4,7 @@ from .checks import RBAC_CHECKS_ATTR, RBACCheck, enforce_rbac_checks
 from .locators import (
     AgentBehindApp,
     AgentId,
+    DatasetByDocument,
     DatasetByPipeline,
     DatasetId,
     PlainApp,
@@ -16,6 +17,7 @@ __all__ = [
     "RBAC_CHECKS_ATTR",
     "AgentBehindApp",
     "AgentId",
+    "DatasetByDocument",
     "DatasetByPipeline",
     "DatasetId",
     "PlainApp",

api/controllers/common/rbac/locators.py

@@ -15,6 +15,7 @@ if TYPE_CHECKING:
 __all__ = [
     "AgentBehindApp",
     "AgentId",
+    "DatasetByDocument",
     "DatasetByPipeline",
     "DatasetId",
     "PlainApp",
@@ -116,7 +117,7 @@ class AgentBehindApp(_ParamLocator):
         binding = agent_binding(tenant_id, _required(path_args, self.param))
         if binding is None or binding.scope == AgentScope.WORKFLOW_ONLY:
             return None
-        return ResourceIdentity(self.scope, str(binding.id))
+        return ResourceIdentity(self.scope, binding.id)
 
     def owner_id(self, tenant_id: str, identity: ResourceIdentity) -> str | None:
         return None
@@ -133,6 +134,18 @@ class DatasetId(_ParamLocator):
         return RBACResourceService.get_dataset_maintainer(db.session, tenant_id, identity.id)
 
 
+class DatasetByDocument(DatasetId):
+    default_param = "document_id"
+
+    @override
+    def locate(self, tenant_id: str, path_args: Mapping[str, object]) -> ResourceIdentity | None:
+        document_id = _required(path_args, self.param)
+        dataset_id = RBACResourceService.get_dataset_id_by_document(db.session, tenant_id, document_id)
+        if dataset_id is None:
+            raise NotFound("Document not found")
+        return ResourceIdentity(self.scope, dataset_id)
+
+
 class DatasetByPipeline(DatasetId):
     default_param = "pipeline_id"
 

api/controllers/console/app/agent_config_inspector.py

@@ -7,14 +7,16 @@ import io
 import json
 from collections.abc import Callable, Mapping
 from dataclasses import dataclass
-from typing import Any, Literal
+from functools import wraps
+from typing import Any, Concatenate, Literal
 from uuid import UUID
 
-from flask import Response, request, send_file, url_for
+from flask import Response, make_response, request, send_file, url_for
 from flask_restx import Resource
-from pydantic import BaseModel, Field
+from pydantic import BaseModel, ConfigDict, Field
 from sqlalchemy.orm import Session
 
+from controllers.common.fields import BinaryFileResponse
 from controllers.common.rbac import AgentId, PlainApp, RBACCheck
 from controllers.common.schema import (
     query_params_from_model,
@@ -25,6 +27,8 @@ from controllers.common.schema import (
 from controllers.common.session import with_session
 from controllers.console import console_ns
 from controllers.console.agent.app_helpers import resolve_agent_runtime_app_model
+from controllers.console.app.error import AppUnavailableError
+from controllers.console.app.preview_admission import get_preview_app
 from controllers.console.app.wraps import get_app_model
 from controllers.console.wraps import (
     RBACPermission,
@@ -36,7 +40,11 @@ from controllers.console.wraps import (
     with_current_tenant_id,
     with_current_user,
 )
+from extensions.ext_application_services import application_services
+from fields.agent_fields import AgentAppComposerResponse
 from fields.base import ResponseModel
+from libs.flask_restx_compat import BINARY_RESPONSE_MEDIA_TYPES_VENDOR_KEY
+from libs.helper import dump_response
 from libs.login import login_required
 from models.account import Account
 from models.model import App, AppMode
@@ -50,6 +58,8 @@ from services.agent_config_service import (
     ConfigPushPayload,
     ConfigPushSkillItem,
 )
+from services.app_definition_query_service import AppDefinitionUnavailableError
+from services.app_preview_query_service import AppPreviewRef
 
 
 class AgentConfigQuery(BaseModel):
@@ -215,6 +225,7 @@ class AgentConfigDeleteResponse(ResponseModel):
 register_schema_models(console_ns, AgentConfigFileUploadPayload)
 register_response_schema_models(
     console_ns,
+    BinaryFileResponse,
     AgentConfigDeleteResponse,
     AgentConfigDownloadResponse,
     AgentConfigFileItemResponse,
@@ -239,7 +250,7 @@ register_response_schema_models(
 class _ResolvedConsoleTarget:
     tenant_id: str
     agent_id: str
-    account_id: str
+    account_id: str | None
     version_id: str
     version_kind: AgentConfigVersionKind
 
@@ -247,6 +258,51 @@ class _ResolvedConsoleTarget:
 _WORKFLOW_APP_MODES = [AppMode.WORKFLOW, AppMode.ADVANCED_CHAT]
 
 
+class TrialAgentConfigQuery(BaseModel):
+    model_config = ConfigDict(extra="forbid")
+
+    version_id: UUID | None = Field(default=None, description="Must match the current published template snapshot")
+
+
+class TrialAgentSkillFileQuery(TrialAgentConfigQuery):
+    path: str = Field(description="Normalized member path inside the published Skill package")
+
+
+register_schema_models(console_ns, TrialAgentConfigQuery, TrialAgentSkillFileQuery)
+
+
+def _with_trial_config[T, Q: TrialAgentConfigQuery, **P](
+    view: Callable[Concatenate[T, _ResolvedConsoleTarget, Q, P], Response | dict[str, object]],
+) -> Callable[Concatenate[T, Q, AppPreviewRef, P], Response]:
+    @wraps(view)
+    def decorated(self: T, query: Q, app: AppPreviewRef, /, *args: P.args, **kwargs: P.kwargs) -> Response:
+        try:
+            state = AgentAppComposerResponse.model_validate(
+                application_services().app_previews.get_agent_composer(app=app)
+            )
+            snapshot = state.active_config_snapshot
+            if snapshot is None or (query.version_id is not None and str(query.version_id) != snapshot.id):
+                raise AgentConfigServiceError(
+                    "config_version_not_found", "published template version is unavailable", status_code=404
+                )
+            target = _ResolvedConsoleTarget(
+                tenant_id=app.tenant_id,
+                agent_id=state.agent.id,
+                account_id=None,
+                version_id=snapshot.id,
+                version_kind=AgentConfigVersionKind.SNAPSHOT,
+            )
+            response = make_response(view(self, target, query, *args, **kwargs))
+        except AppDefinitionUnavailableError:
+            raise AppUnavailableError() from None
+        except AgentConfigServiceError as exc:
+            response = make_response(_handle(exc))
+        response.headers["Cache-Control"] = "no-store"
+        return response
+
+    return decorated
+
+
 def _service() -> AgentConfigService:
     return AgentConfigService()
 
@@ -481,6 +537,7 @@ def _read_single_upload() -> tuple[bytes, str]:
 
 
 def _skill_upload_response(target: _ResolvedConsoleTarget) -> tuple[dict[str, object], int]:
+    assert target.account_id is not None
     return _upload_skill_for_target(
         tenant_id=target.tenant_id,
         agent_id=target.agent_id,
@@ -517,6 +574,7 @@ def _upload_skill_for_target(
 def _file_upload_response(
     target: _ResolvedConsoleTarget, payload: AgentConfigFileUploadPayload
 ) -> tuple[dict[str, object], int]:
+    assert target.account_id is not None
     manifest = _service().push_file_for_console(
         tenant_id=target.tenant_id,
         agent_id=target.agent_id,
@@ -616,6 +674,7 @@ def _skill_file_raw_download_response(target: _ResolvedConsoleTarget, name: str,
 
 
 def _skill_delete_response(target: _ResolvedConsoleTarget, name: str) -> dict[str, object]:
+    assert target.account_id is not None
     _service().push_for_console(
         tenant_id=target.tenant_id,
         agent_id=target.agent_id,
@@ -654,6 +713,7 @@ def _file_download_response(target: _ResolvedConsoleTarget, name: str) -> Respon
 
 
 def _file_delete_response(target: _ResolvedConsoleTarget, name: str) -> dict[str, object]:
+    assert target.account_id is not None
     _service().push_for_console(
         tenant_id=target.tenant_id,
         agent_id=target.agent_id,
@@ -1360,5 +1420,124 @@ class AgentConfigFileApi(Resource):
         )
 
 
+@console_ns.route("/trial-apps/<uuid:app_id>/agent/config/skills")
+class TrialAgentConfigSkillsApi(Resource):
+    @console_ns.doc(params=query_params_from_model(TrialAgentConfigQuery))
+    @console_ns.response(200, "Published template resource", console_ns.models[AgentConfigSkillListResponse.__name__])
+    @get_preview_app
+    @model_validate(TrialAgentConfigQuery)
+    @_with_trial_config
+    def get(self, target: _ResolvedConsoleTarget, _query: TrialAgentConfigQuery) -> dict[str, object]:
+        return dump_response(AgentConfigSkillListResponse, _skill_list_response(target))
+
+
+@console_ns.route("/trial-apps/<uuid:app_id>/agent/config/files")
+class TrialAgentConfigFilesApi(Resource):
+    @console_ns.doc(params=query_params_from_model(TrialAgentConfigQuery))
+    @console_ns.response(200, "Published template resource", console_ns.models[AgentConfigFileListResponse.__name__])
+    @get_preview_app
+    @model_validate(TrialAgentConfigQuery)
+    @_with_trial_config
+    def get(self, target: _ResolvedConsoleTarget, _query: TrialAgentConfigQuery) -> dict[str, object]:
+        return dump_response(AgentConfigFileListResponse, _file_list_response(target))
+
+
+@console_ns.route("/trial-apps/<uuid:app_id>/agent/config/skills/<string:name>/inspect")
+class TrialAgentConfigSkillInspectApi(Resource):
+    @console_ns.doc(params=query_params_from_model(TrialAgentConfigQuery))
+    @console_ns.response(
+        200, "Published template resource", console_ns.models[AgentConfigSkillInspectResponse.__name__]
+    )
+    @get_preview_app
+    @model_validate(TrialAgentConfigQuery)
+    @_with_trial_config
+    def get(self, target: _ResolvedConsoleTarget, _query: TrialAgentConfigQuery, name: str) -> Response:
+        return _skill_inspect_response(target, name)
+
+
+@console_ns.route("/trial-apps/<uuid:app_id>/agent/config/skills/<string:name>/download")
+class TrialAgentConfigSkillDownloadApi(Resource):
+    @console_ns.doc(params=query_params_from_model(TrialAgentConfigQuery))
+    @console_ns.response(200, "Published template resource", console_ns.models[AgentConfigDownloadResponse.__name__])
+    @get_preview_app
+    @model_validate(TrialAgentConfigQuery)
+    @_with_trial_config
+    def get(self, target: _ResolvedConsoleTarget, _query: TrialAgentConfigQuery, name: str) -> Response:
+        return _skill_download_response(target, name)
+
+
+@console_ns.route("/trial-apps/<uuid:app_id>/agent/config/skills/<string:name>/files/preview")
+class TrialAgentConfigSkillFilePreviewApi(Resource):
+    @console_ns.doc(params=query_params_from_model(TrialAgentSkillFileQuery))
+    @console_ns.response(
+        200, "Published template resource", console_ns.models[AgentConfigSkillFilePreviewResponse.__name__]
+    )
+    @get_preview_app
+    @model_validate(TrialAgentSkillFileQuery)
+    @_with_trial_config
+    def get(self, target: _ResolvedConsoleTarget, query: TrialAgentSkillFileQuery, name: str) -> dict[str, object]:
+        return dump_response(
+            AgentConfigSkillFilePreviewResponse, _skill_file_preview_response(target, name, query.path)
+        )
+
+
+@console_ns.route("/trial-apps/<uuid:app_id>/agent/config/skills/<string:name>/files/download")
+class TrialAgentConfigSkillFileDownloadApi(Resource):
+    @console_ns.doc(params=query_params_from_model(TrialAgentSkillFileQuery))
+    @console_ns.response(200, "Published template resource", console_ns.models[AgentConfigDownloadResponse.__name__])
+    @get_preview_app
+    @model_validate(TrialAgentSkillFileQuery)
+    @_with_trial_config
+    def get(self, target: _ResolvedConsoleTarget, query: TrialAgentSkillFileQuery, name: str) -> Response:
+        assert request.view_args is not None
+        return _skill_file_download_response(
+            target,
+            name=name,
+            path=query.path,
+            raw_endpoint="console.trial_agent_config_skill_file_content",
+            route_params={"app_id": str(request.view_args["app_id"])},
+        )
+
+
+@console_ns.route(
+    "/trial-apps/<uuid:app_id>/agent/config/skills/<string:name>/files/content",
+    endpoint="trial_agent_config_skill_file_content",
+)
+class TrialAgentConfigSkillFileContentApi(Resource):
+    @console_ns.doc(params=query_params_from_model(TrialAgentSkillFileQuery))
+    @console_ns.doc(
+        produces=["application/octet-stream"],
+        vendor={BINARY_RESPONSE_MEDIA_TYPES_VENDOR_KEY: ["application/octet-stream"]},
+    )
+    @console_ns.response(200, "Published template resource", console_ns.models[BinaryFileResponse.__name__])
+    @get_preview_app
+    @model_validate(TrialAgentSkillFileQuery)
+    @_with_trial_config
+    def get(self, target: _ResolvedConsoleTarget, query: TrialAgentSkillFileQuery, name: str) -> Response:
+        return _skill_file_raw_download_response(target, name, query.path)
+
+
+@console_ns.route("/trial-apps/<uuid:app_id>/agent/config/files/<string:name>/preview")
+class TrialAgentConfigFilePreviewApi(Resource):
+    @console_ns.doc(params=query_params_from_model(TrialAgentConfigQuery))
+    @console_ns.response(200, "Published template resource", console_ns.models[AgentConfigFilePreviewResponse.__name__])
+    @get_preview_app
+    @model_validate(TrialAgentConfigQuery)
+    @_with_trial_config
+    def get(self, target: _ResolvedConsoleTarget, _query: TrialAgentConfigQuery, name: str) -> dict[str, object]:
+        return dump_response(AgentConfigFilePreviewResponse, _file_preview_response(target, name))
+
+
+@console_ns.route("/trial-apps/<uuid:app_id>/agent/config/files/<string:name>/download")
+class TrialAgentConfigFileDownloadApi(Resource):
+    @console_ns.doc(params=query_params_from_model(TrialAgentConfigQuery))
+    @console_ns.response(200, "Published template resource", console_ns.models[AgentConfigDownloadResponse.__name__])
+    @get_preview_app
+    @model_validate(TrialAgentConfigQuery)
+    @_with_trial_config
+    def get(self, target: _ResolvedConsoleTarget, _query: TrialAgentConfigQuery, name: str) -> Response:
+        return _file_download_response(target, name)
+
+
 # pyrefly: ignore [unresolvable-dunder-all]
 __all__ = [name for name, value in globals().items() if inspect.isclass(value) and issubclass(value, Resource)]

api/controllers/console/app/app.py

@@ -79,7 +79,8 @@ from services.entities.app_entities import (
     UpdateAppParams,
 )
 from services.entities.dsl_entities import ImportStatus
-from services.entities.knowledge_entities.knowledge_entities import (
+from services.errors.account import NoPermissionError
+from services.knowledge.entities.knowledge_entities import (
     DataSource,
     InfoList,
     NotionIcon,
@@ -91,7 +92,6 @@ from services.entities.knowledge_entities.knowledge_entities import (
     WeightModel,
     WeightVectorSetting,
 )
-from services.errors.account import NoPermissionError
 
 register_enum_models(console_ns, IconType)
 

api/controllers/console/app/app_import.py

@@ -8,6 +8,7 @@ from werkzeug.exceptions import Forbidden
 from controllers.common.rbac import PlainApp, RBACCheck, Workspace
 from controllers.common.schema import register_enum_models, register_schema_models
 from controllers.console.app.error import AppNotFoundError
+from controllers.console.explore.error import RecommendedAppNotFoundError as RecommendedAppNotFoundHttpError
 from controllers.console.flask_admission import console_account_admission
 from controllers.console.wraps import RBACPermission, validate_request
 from extensions.ext_application_services import application_services
@@ -17,6 +18,7 @@ from services.agent.errors import InvalidRosterAgentPackageError
 from services.app.console_service import ConsoleAppNotFoundError
 from services.entities.dsl_entities import AppImportParams, CheckDependenciesResult, Import, ImportStatus
 from services.errors.account import NoPermissionError
+from services.recommended_app_query_service import RecommendedAppNotFoundError
 
 from .. import console_ns
 
@@ -86,6 +88,8 @@ class AppImportApi(Resource):
             result = application_services().apps.console.import_app(context, payload, source=source)
         except NoPermissionError as exc:
             raise Forbidden(str(exc)) from exc
+        except RecommendedAppNotFoundError:
+            raise RecommendedAppNotFoundHttpError() from None
         status_code = {ImportStatus.FAILED: HTTPStatus.BAD_REQUEST, ImportStatus.PENDING: HTTPStatus.ACCEPTED}.get(
             result.status, HTTPStatus.OK
         )

api/controllers/console/app/generator.py

@@ -401,6 +401,7 @@ class InstructionGenerateApi(Resource):
                     instruction=req_data.instruction,
                     model_config=req_data.model_config_data,
                     ideal_output=req_data.ideal_output,
+                    session=session,
                 )
             if req_data.node_id != "" and req_data.current != "":  # For workflow node
                 return LLMGenerator.instruction_modify_workflow(

api/controllers/console/app/mcp_server.py

@@ -1,32 +1,31 @@
 import json
 from datetime import datetime
 from typing import Any
+from uuid import UUID
 
 from flask_restx import Resource
 from pydantic import BaseModel, Field, field_validator
-from sqlalchemy import select
-from werkzeug.exceptions import NotFound
+from werkzeug.exceptions import Conflict, NotFound
 
 from controllers.common.rbac import PlainApp, RBACCheck
 from controllers.common.schema import register_schema_models
 from controllers.console import console_ns
-from controllers.console.app.wraps import get_app_model
-from controllers.console.wraps import (
-    RBACPermission,
-    account_initialization_required,
-    edit_permission_required,
-    model_validate,
-    rbac_permission_required,
-    setup_required,
-    with_current_tenant_id,
-)
-from extensions.ext_database import db
+from controllers.console.app.error import AppNotFoundError
+from controllers.console.flask_admission import console_account_admission
+from controllers.console.wraps import RBACPermission, validate_request
+from extensions.ext_application_services import application_services
 from fields.base import ResponseModel
 from libs.helper import dump_response, to_timestamp
-from libs.login import login_required
-from models.enums import AppMCPServerStatus
-from models.model import App, AppMCPServer
-from services.app_ref_service import AppRefService
+from machinery.context import RequestContext
+from models.account import TenantAccountRole
+from services.app.mcp_server_service import (
+    AppMCPServerAlreadyExistsError,
+    AppMCPServerAppNotFoundError,
+    AppMCPServerNotFoundError,
+    AppMCPServerStatus,
+)
+
+_EDIT_ROLES = frozenset({TenantAccountRole.OWNER, TenantAccountRole.ADMIN, TenantAccountRole.EDITOR})
 
 
 class MCPServerCreatePayload(BaseModel):
@@ -38,13 +37,13 @@ class MCPServerCreatePayload(BaseModel):
 
 
 class MCPServerUpdatePayload(BaseModel):
-    id: str = Field(..., description="Server ID")
+    id: UUID = Field(..., description="Server ID")
     description: str | None = Field(default=None, description="Server description")
     parameters: dict[str, Any] = Field(
         ...,
         description="Server parameters configuration",
     )
-    status: str | None = Field(default=None, description="Server status")
+    status: AppMCPServerStatus | None = Field(default=None, description="Server status")
 
 
 class AppMCPServerResponse(ResponseModel):
@@ -84,13 +83,12 @@ class AppMCPServerController(Resource):
     @console_ns.response(
         200, "MCP server configuration retrieved successfully", console_ns.models[AppMCPServerResponse.__name__]
     )
-    @login_required
-    @account_initialization_required
-    @setup_required
-    @rbac_permission_required(RBACCheck(RBACPermission.APP_VIEW_LAYOUT, PlainApp()))
-    @get_app_model
-    def get(self, app_model: App):
-        server = db.session.scalar(select(AppMCPServer).where(AppMCPServer.app_id == app_model.id).limit(1))
+    @console_account_admission(rbac_checks=[RBACCheck(RBACPermission.APP_VIEW_LAYOUT, PlainApp())])
+    def get(self, request_context: RequestContext, app_id: UUID):
+        try:
+            server = application_services().app_mcp_servers.get(request_context, str(app_id))
+        except AppMCPServerAppNotFoundError as error:
+            raise AppNotFoundError() from error
         if server is None:
             return {}
         return dump_response(AppMCPServerResponse, server)
@@ -103,30 +101,18 @@ class AppMCPServerController(Resource):
         201, "MCP server configuration created successfully", console_ns.models[AppMCPServerResponse.__name__]
     )
     @console_ns.response(403, "Insufficient permissions")
-    @account_initialization_required
-    @login_required
-    @setup_required
-    @edit_permission_required
-    @rbac_permission_required(RBACCheck(RBACPermission.APP_EDIT, PlainApp()))
-    @with_current_tenant_id
-    @get_app_model
-    @model_validate(MCPServerCreatePayload)
-    def post(self, req_data: MCPServerCreatePayload, current_tenant_id: str, app_model: App):
-        description = req_data.description
-        if not description:
-            description = app_model.description or ""
-
-        server = AppMCPServer(
-            name=app_model.name,
-            description=description,
-            parameters=json.dumps(req_data.parameters, ensure_ascii=False),
-            status=AppMCPServerStatus.ACTIVE,
-            app_id=app_model.id,
-            tenant_id=current_tenant_id,
-            server_code=AppMCPServer.generate_server_code(16, session=db.session()),
-        )
-        db.session.add(server)
-        db.session.commit()
+    @console_ns.response(409, "MCP server already exists for this app")
+    @console_account_admission(allowed_roles=_EDIT_ROLES, rbac_checks=[RBACCheck(RBACPermission.APP_EDIT, PlainApp())])
+    def post(self, request_context: RequestContext, app_id: UUID):
+        payload = validate_request(MCPServerCreatePayload)
+        try:
+            server = application_services().app_mcp_servers.create(
+                request_context, str(app_id), description=payload.description, parameters=payload.parameters
+            )
+        except AppMCPServerAppNotFoundError as error:
+            raise AppNotFoundError() from error
+        except AppMCPServerAlreadyExistsError as error:
+            raise Conflict(str(error)) from error
         return dump_response(AppMCPServerResponse, server), 201
 
     @console_ns.doc("update_app_mcp_server")
@@ -138,43 +124,22 @@ class AppMCPServerController(Resource):
     )
     @console_ns.response(403, "Insufficient permissions")
     @console_ns.response(404, "Server not found")
-    @login_required
-    @setup_required
-    @account_initialization_required
-    @edit_permission_required
-    @rbac_permission_required(RBACCheck(RBACPermission.APP_EDIT, PlainApp()))
-    @get_app_model
-    @model_validate(MCPServerUpdatePayload)
-    def put(self, req_data: MCPServerUpdatePayload, app_model: App):
-        app_ref = AppRefService.create_app_ref(app_model)
-        server_ref = AppRefService.create_mcp_server_ref(app_ref, req_data.id)
-        server = db.session.scalar(
-            select(AppMCPServer)
-            .where(
-                AppMCPServer.id == server_ref.server_id,
-                AppMCPServer.tenant_id == server_ref.app.tenant_id,
-                AppMCPServer.app_id == server_ref.app.app_id,
+    @console_account_admission(allowed_roles=_EDIT_ROLES, rbac_checks=[RBACCheck(RBACPermission.APP_EDIT, PlainApp())])
+    def put(self, request_context: RequestContext, app_id: UUID):
+        payload = validate_request(MCPServerUpdatePayload)
+        try:
+            server = application_services().app_mcp_servers.update(
+                request_context,
+                str(app_id),
+                server_id=str(payload.id),
+                description=payload.description,
+                parameters=payload.parameters,
+                status=payload.status,
             )
-            .limit(1)
-        )
-        if not server:
-            raise NotFound()
-
-        description = req_data.description
-        if description is None or not description:
-            server.description = app_model.description or ""
-        else:
-            server.description = description
-
-        server.name = app_model.name
-
-        server.parameters = json.dumps(req_data.parameters, ensure_ascii=False)
-        if req_data.status:
-            try:
-                server.status = AppMCPServerStatus(req_data.status)
-            except ValueError:
-                raise ValueError("Invalid status")
-        db.session.commit()
+        except AppMCPServerAppNotFoundError as error:
+            raise AppNotFoundError() from error
+        except AppMCPServerNotFoundError as error:
+            raise NotFound from error
         return dump_response(AppMCPServerResponse, server)
 
 
@@ -186,21 +151,12 @@ class AppMCPServerRefreshController(Resource):
     @console_ns.response(200, "MCP server refreshed successfully", console_ns.models[AppMCPServerResponse.__name__])
     @console_ns.response(403, "Insufficient permissions")
     @console_ns.response(404, "Server not found")
-    @setup_required
-    @login_required
-    @account_initialization_required
-    @edit_permission_required
-    @rbac_permission_required(RBACCheck(RBACPermission.APP_EDIT, PlainApp()))
-    @with_current_tenant_id
-    @get_app_model
-    def post(self, current_tenant_id: str, app_model: App):
-        server = db.session.scalar(
-            select(AppMCPServer)
-            .where(AppMCPServer.app_id == app_model.id, AppMCPServer.tenant_id == current_tenant_id)
-            .limit(1)
-        )
-        if not server:
-            raise NotFound()
-        server.server_code = AppMCPServer.generate_server_code(16, session=db.session())
-        db.session.commit()
+    @console_account_admission(allowed_roles=_EDIT_ROLES, rbac_checks=[RBACCheck(RBACPermission.APP_EDIT, PlainApp())])
+    def post(self, request_context: RequestContext, app_id: UUID):
+        try:
+            server = application_services().app_mcp_servers.refresh(request_context, str(app_id))
+        except AppMCPServerAppNotFoundError as error:
+            raise AppNotFoundError() from error
+        except AppMCPServerNotFoundError as error:
+            raise NotFound from error
         return dump_response(AppMCPServerResponse, server)

api/controllers/console/app/message.py

@@ -1,4 +1,5 @@
 import logging
+from http import HTTPStatus
 from typing import Literal
 from uuid import UUID
 
@@ -6,7 +7,7 @@ from flask_restx import Resource
 from pydantic import BaseModel, Field, field_validator
 from sqlalchemy import exists, func, select
 from sqlalchemy.orm import Session
-from werkzeug.exceptions import InternalServerError, NotFound
+from werkzeug.exceptions import HTTPException, InternalServerError, NotFound, Unauthorized
 
 from controllers.common.controller_schemas import MessageFeedbackPayload as _MessageFeedbackPayloadBase
 from controllers.common.fields import SimpleResultResponse, TextFileResponse
@@ -16,6 +17,8 @@ from controllers.common.session import with_session
 from controllers.console import console_ns
 from controllers.console.agent.app_helpers import resolve_agent_runtime_app_model
 from controllers.console.app.error import (
+    AppNotFoundError,
+    AppUnavailableError,
     CompletionRequestError,
     ProviderModelCurrentlyNotSupportError,
     ProviderNotInitializeError,
@@ -23,6 +26,7 @@ from controllers.console.app.error import (
 )
 from controllers.console.app.wraps import get_app_model
 from controllers.console.explore.error import AppSuggestedQuestionsAfterAnswerDisabledError
+from controllers.console.flask_admission import console_account_admission
 from controllers.console.wraps import (
     RBACPermission,
     account_initialization_required,
@@ -33,9 +37,9 @@ from controllers.console.wraps import (
     with_current_tenant_id,
     with_current_user,
 )
-from core.app.entities.app_invoke_entities import InvokeFrom
 from core.entities.execution_extra_content import ExecutionExtraContentDomainModel
 from core.errors.error import ModelCurrentlyNotSupportError, ProviderTokenNotInitError, QuotaExceededError
+from extensions.ext_application_services import application_services
 from extensions.ext_database import db
 from fields.base import ResponseModel
 from fields.conversation_fields import (
@@ -46,13 +50,19 @@ from graphon.model_runtime.errors.invoke import InvokeError
 from libs.helper import dump_response, uuid_value
 from libs.infinite_scroll_pagination import InfiniteScrollPagination
 from libs.login import login_required
+from machinery.context import RequestContext
 from models.account import Account
 from models.enums import FeedbackFromSource, FeedbackRating
 from models.model import App, AppMode, Conversation, Message, MessageAnnotation, MessageFeedback
+from services.agent.errors import AgentNotFoundError
+from services.app.agent_app_contracts import AgentAppNotFoundError
+from services.app.console_service import ConsoleAppNotFoundError
+from services.app_definition_query_service import AppDefinitionUnavailableError
 from services.conversation_service import ConversationService
 from services.errors.conversation import ConversationNotExistsError
 from services.errors.message import MessageNotExistsError, SuggestedQuestionsAfterAnswerDisabledError
-from services.message_service import MessageService, attach_message_extra_contents
+from services.message_service import attach_message_extra_contents
+from services.message_suggested_questions_service import SuggestedQuestionsAccount, SuggestedQuestionsActorNotFoundError
 
 logger = logging.getLogger(__name__)
 
@@ -296,21 +306,25 @@ class MessageSuggestedQuestionApi(Resource):
     @console_ns.doc(description="Get suggested questions for a message")
     @console_ns.doc(params={"app_id": "Application ID", "message_id": "Message ID"})
     @console_ns.response(
-        200,
+        HTTPStatus.OK,
         "Suggested questions retrieved successfully",
         console_ns.models[SuggestedQuestionsResponse.__name__],
     )
-    @console_ns.response(404, "Message or conversation not found")
-    @setup_required
-    @login_required
-    @account_initialization_required
-    @with_current_user
-    @rbac_permission_required(RBACCheck(RBACPermission.APP_VIEW_LAYOUT, PlainApp()))
-    @with_session(write=False)
-    @get_app_model(mode=[AppMode.CHAT, AppMode.AGENT_CHAT, AppMode.ADVANCED_CHAT, AppMode.AGENT])
-    def get(self, session: Session, current_user: Account, app_model: App, message_id: UUID):
+    @console_ns.response(HTTPStatus.BAD_REQUEST, "App or model provider unavailable, or generation failed")
+    @console_ns.response(HTTPStatus.UNAUTHORIZED, "Account authentication required")
+    @console_ns.response(HTTPStatus.FORBIDDEN, "Insufficient permissions or suggested questions disabled")
+    @console_ns.response(HTTPStatus.NOT_FOUND, "App, message, or conversation not found")
+    @console_ns.response(HTTPStatus.INTERNAL_SERVER_ERROR, "Unexpected server error")
+    @console_account_admission(rbac_checks=(RBACCheck(RBACPermission.APP_VIEW_LAYOUT, PlainApp()),))
+    def get(self, context: RequestContext, app_id: UUID, message_id: UUID) -> dict[str, object]:
+        try:
+            app = application_services().apps.console.get_reference(context, str(app_id))
+        except ConsoleAppNotFoundError as exc:
+            raise AppNotFoundError() from exc
+        if app.mode not in (AppMode.CHAT, AppMode.AGENT_CHAT, AppMode.ADVANCED_CHAT, AppMode.AGENT):
+            raise AppNotFoundError("App mode does not support suggested questions")
         return _get_message_suggested_questions(
-            session=session, current_user=current_user, app_model=app_model, message_id=message_id
+            context=context, app_id=app.id, app_mode=app.mode, message_id=message_id
         )
 
 
@@ -320,26 +334,23 @@ class AgentMessageSuggestedQuestionApi(Resource):
     @console_ns.doc(description="Get suggested questions for an Agent App message")
     @console_ns.doc(params={"agent_id": "Agent ID", "message_id": "Message ID"})
     @console_ns.response(
-        200,
+        HTTPStatus.OK,
         "Suggested questions retrieved successfully",
         console_ns.models[SuggestedQuestionsResponse.__name__],
     )
-    @console_ns.response(404, "Agent, message, or conversation not found")
-    @setup_required
-    @login_required
-    @account_initialization_required
-    @rbac_permission_required(RBACCheck(RBACPermission.AGENT_TEST_AND_RUN, AgentId()))
-    @with_current_user
-    @with_current_tenant_id
-    @with_session(write=False)
-    def get(self, session: Session, current_tenant_id: str, current_user: Account, agent_id: UUID, message_id: UUID):
-        app_model = resolve_agent_runtime_app_model(
-            session=session,
-            tenant_id=current_tenant_id,
-            agent_id=agent_id,
-        )
+    @console_ns.response(HTTPStatus.BAD_REQUEST, "App or model provider unavailable, or generation failed")
+    @console_ns.response(HTTPStatus.UNAUTHORIZED, "Account authentication required")
+    @console_ns.response(HTTPStatus.FORBIDDEN, "Insufficient permissions or suggested questions disabled")
+    @console_ns.response(HTTPStatus.NOT_FOUND, "Agent, message, or conversation not found")
+    @console_ns.response(HTTPStatus.INTERNAL_SERVER_ERROR, "Unexpected server error")
+    @console_account_admission(rbac_checks=(RBACCheck(RBACPermission.AGENT_TEST_AND_RUN, AgentId()),))
+    def get(self, context: RequestContext, agent_id: UUID, message_id: UUID) -> dict[str, object]:
+        try:
+            app_id = application_services().agent_apps.access.resolve_existing_runtime_app_id(context, str(agent_id))
+        except AgentAppNotFoundError as exc:
+            raise AgentNotFoundError() from exc
         return _get_message_suggested_questions(
-            session=session, current_user=current_user, app_model=app_model, message_id=message_id
+            context=context, app_id=app_id, app_mode=AppMode.AGENT, message_id=message_id
         )
 
 
@@ -552,17 +563,22 @@ def _update_message_feedback(
     return SimpleResultResponse(result="success").model_dump(mode="json")
 
 
-def _get_message_suggested_questions(*, session: Session, current_user: Account, app_model: App, message_id: UUID):
-    message_id_str = str(message_id)
+def _get_message_suggested_questions(
+    *, context: RequestContext, app_id: str, app_mode: str, message_id: UUID
+) -> dict[str, object]:
 
     try:
-        questions = MessageService.get_suggested_questions_after_answer(
-            app_model=app_model,
-            message_id=message_id_str,
-            user=current_user,
-            invoke_from=InvokeFrom.DEBUGGER,
-            session=session,
+        questions = application_services().message_suggested_questions.get_suggested_questions(
+            app_id=app_id,
+            app_owner_tenant_id=context.active_workspace_id,
+            expected_app_mode=app_mode,
+            actor=SuggestedQuestionsAccount(account_id=context.account_id, invoke_from="debugger"),
+            message_id=str(message_id),
         )
+    except AppDefinitionUnavailableError as exc:
+        raise AppUnavailableError() from exc
+    except SuggestedQuestionsActorNotFoundError as exc:
+        raise Unauthorized("Account no longer exists") from exc
     except MessageNotExistsError:
         raise NotFound("Message not found")
     except ConversationNotExistsError:
@@ -577,6 +593,8 @@ def _get_message_suggested_questions(*, session: Session, current_user: Account,
         raise CompletionRequestError(e.description)
     except SuggestedQuestionsAfterAnswerDisabledError:
         raise AppSuggestedQuestionsAfterAnswerDisabledError()
+    except HTTPException:
+        raise
     except Exception:
         logger.exception("internal server error.")
         raise InternalServerError()

api/controllers/console/app/workflow.py

@@ -87,7 +87,6 @@ from graphon.model_runtime.utils.encoders import jsonable_encoder
 from graphon.variables import SecretVariable, SegmentType, VariableBase
 from graphon.variables.exc import VariableError
 from libs import helper
-from libs.datetime_utils import naive_utc_now
 from libs.helper import TimestampField, dump_response, to_timestamp, uuid_value
 from libs.login import login_required
 from models import Account, App
@@ -101,10 +100,7 @@ from services.errors.app import IsDraftWorkflowError, WorkflowHashNotEqualError,
 from services.errors.llm import InvokeRateLimitError
 from services.workflow_ref_service import WorkflowRefService
 from services.workflow_service import DraftWorkflowDeletionError, WorkflowInUseError, WorkflowService
-from services.workflow_variable_reference_validator import (
-    format_variable_reference_errors,
-    validate_variable_references,
-)
+from services.workflow_variable_reference_validator import advisory_variable_reference_warning
 
 logger = logging.getLogger(__name__)
 
@@ -1274,21 +1270,6 @@ class DraftWorkflowNodeRunApi(Resource):
         ).model_dump(mode="json")
 
 
-def _advisory_variable_reference_warning(graph_text: str | None) -> str | None:
-    """Return a non-blocking publish warning. A checker failure must not fail publish."""
-    if not graph_text:
-        return None
-    try:
-        graph = json.loads(graph_text)
-        if not isinstance(graph, dict):
-            return None
-        issues = validate_variable_references(graph)
-        return format_variable_reference_errors(issues) if issues else None
-    except Exception:
-        logger.warning("Skipped advisory variable reference check", exc_info=True)
-        return None
-
-
 @console_ns.route("/apps/<uuid:app_id>/workflows/publish")
 class PublishedWorkflowApi(Resource):
     @console_ns.doc("get_published_workflow")
@@ -1337,7 +1318,7 @@ class PublishedWorkflowApi(Resource):
 
         workflow_service = WorkflowService()
         with sessionmaker(db.engine).begin() as session:
-            workflow = workflow_service.publish_workflow(
+            workflow = workflow_service.publish_app_workflow(
                 session=session,
                 app_model=app_model,
                 account=current_user,
@@ -1345,17 +1326,10 @@ class PublishedWorkflowApi(Resource):
                 marked_comment=args.marked_comment or "",
             )
 
-            # Update app_model within the same session to ensure atomicity
-            app_model_in_session = session.get(App, app_model.id)
-            if app_model_in_session:
-                app_model_in_session.workflow_id = workflow.id
-                app_model_in_session.updated_by = current_user.id
-                app_model_in_session.updated_at = naive_utc_now()
-
             workflow_created_at = TimestampField().format(workflow.created_at)
             graph_text = workflow.graph
 
-        warning = _advisory_variable_reference_warning(graph_text)
+        warning = advisory_variable_reference_warning(graph_text)
         payload: dict[str, object] = {
             "result": "success",
             "created_at": workflow_created_at,

api/controllers/console/app/wraps.py

@@ -18,6 +18,7 @@ from extensions.ext_database import db
 from libs.login import current_account_with_tenant
 from models import App, AppMode
 from models.agent import AgentScope
+from models.enums import AppStatus
 
 __all__ = [
     "get_app_model",
@@ -25,31 +26,28 @@ __all__ = [
 ]
 
 
-def _is_hidden_backing_app(app_model: App, session: Session | scoped_session) -> bool:
-    binding = app_model.agent_app_binding_with_session(session=session, include_archived=True)
-    return binding is not None and binding.scope == AgentScope.WORKFLOW_ONLY
-
-
-def _load_app_model(session: Session, app_id: str) -> App | None:
-    """Load the tenant-scoped app row with the request session owned by `with_session`."""
+def _load_app_model(session: Session | scoped_session, app_id: str) -> App | None:
+    """Load the tenant-scoped Console app row through the given session."""
     _, current_tenant_id = current_account_with_tenant()
     app_model = session.scalar(
-        select(App).where(App.id == app_id, App.tenant_id == current_tenant_id, App.status == "normal").limit(1)
+        select(App)
+        .where(
+            App.id == app_id,
+            App.tenant_id == current_tenant_id,
+            App.status == AppStatus.NORMAL,
+        )
+        .limit(1)
     )
-    if app_model is not None and _is_hidden_backing_app(app_model, session):
-        return None
+    if app_model is not None:
+        binding = app_model.agent_app_binding_with_session(session=session, include_archived=True)
+        if binding is not None and binding.scope == AgentScope.WORKFLOW_ONLY:
+            return None
     return app_model
 
 
 def _load_app_model_from_scoped_session(app_id: str) -> App | None:
     """Load the app row for legacy handlers that have not adopted request session injection yet."""
-    _, current_tenant_id = current_account_with_tenant()
-    app_model = db.session.scalar(
-        select(App).where(App.id == app_id, App.tenant_id == current_tenant_id, App.status == "normal").limit(1)
-    )
-    if app_model is not None and _is_hidden_backing_app(app_model, db.session):
-        return None
-    return app_model
+    return _load_app_model(db.session, app_id)
 
 
 def _get_injected_session(args: tuple[object, ...]) -> Session | None:

api/controllers/console/auth/activate.py

@@ -115,14 +115,14 @@ class ActivateApi(Resource):
         console_ns.models[ActivationResponse.__name__],
     )
     @console_ns.response(400, "Already activated or invalid token")
-    def post(self):
+    @model_validate(ActivatePayload)
+    def post(self, args: ActivatePayload):
         """Accept an invitation without letting an existing session act for another account.
 
         Token-only activation remains available for legacy clients. When the request already
         carries a console session, that session must belong to the account encoded in the
         invitation before the token is consumed or tenant membership is changed.
         """
-        args = ActivatePayload.model_validate(console_ns.payload or {})
         authenticated_account_id: str | None = None
         if extract_access_token(request) is not None:
             authenticated_account_id = current_account_with_tenant().account.id

api/controllers/console/auth/data_source_bearer_auth.py

@@ -27,7 +27,7 @@ from services.auth.errors import (
     InvalidDataSourceApiKeyAuthCredentialsError,
     UnsupportedDataSourceApiKeyAuthProviderError,
 )
-from services.entities.data_source_api_key_auth_entities import (
+from services.data_source.entities.api_key_auth import (
     DataSourceApiKeyAuthBindingCreate,
     DataSourceApiKeyAuthCredentials,
 )
@@ -80,7 +80,7 @@ class ApiKeyAuthDataSource(Resource):
     @console_ns.response(HTTPStatus.OK, "Success", console_ns.models[ApiKeyAuthDataSourceListResponse.__name__])
     @console_account_admission()
     def get(self, request_context: RequestContext):
-        bindings = application_services().data_source_api_key_auth.list_bindings(request_context)
+        bindings = application_services().data_sources.api_key_auth.list_bindings(request_context)
         return dump_response(
             ApiKeyAuthDataSourceListResponse,
             {
@@ -120,7 +120,7 @@ class ApiKeyAuthDataSourceBinding(Resource):
             ),
         )
         try:
-            application_services().data_source_api_key_auth.create_binding(request_context, command)
+            application_services().data_sources.api_key_auth.create_binding(request_context, command)
         except UnsupportedDataSourceApiKeyAuthProviderError as exc:
             raise DataSourceApiKeyAuthProviderNotSupportedError() from exc
         except InvalidDataSourceApiKeyAuthCredentialsError as exc:
@@ -140,5 +140,5 @@ class ApiKeyAuthDataSourceBindingDelete(Resource):
         rbac_checks=[RBACCheck(RBACPermission.CREDENTIAL_MANAGE, Workspace())],
     )
     def delete(self, request_context: RequestContext, binding_id: UUID):
-        application_services().data_source_api_key_auth.delete_binding(request_context, str(binding_id))
+        application_services().data_sources.api_key_auth.delete_binding(request_context, str(binding_id))
         return "", HTTPStatus.NO_CONTENT

api/controllers/console/auth/data_source_oauth.py

@@ -24,13 +24,13 @@ from fields.base import ResponseModel
 from libs.helper import dump_response
 from machinery.context import RequestContext
 from models.account import TenantAccountRole
-from services.data_source_oauth_service import (
+from services.data_source.entities.oauth import DataSourceOAuthCallback
+from services.data_source.oauth_service import (
     DataSourceOAuthConfigurationError,
     DataSourceOAuthError,
     InvalidDataSourceOAuthCodeError,
     InvalidDataSourceOAuthProviderError,
 )
-from services.entities.data_source_oauth_entities import DataSourceOAuthCallback
 
 from .. import console_ns
 
@@ -116,7 +116,7 @@ class OAuthDataSource(Resource):
     )
     def get(self, request_context: RequestContext, provider: str):
         try:
-            service = application_services().resolve_data_source_oauth(provider)
+            service = application_services().data_sources.resolve_oauth(provider)
             authorization = service.start_authorization(request_context)
         except InvalidDataSourceOAuthProviderError:
             return _invalid_provider_response()
@@ -141,7 +141,7 @@ class OAuthDataSourceCallback(Resource):
     def get(self, provider: str):
         query = query_params_from_request(OAuthDataSourceCallbackQuery)
         try:
-            service = application_services().resolve_data_source_oauth(provider)
+            service = application_services().data_sources.resolve_oauth(provider)
             callback = service.complete_callback(
                 code=query.code,
                 error=query.error,
@@ -172,7 +172,7 @@ class OAuthDataSourceBinding(Resource):
         if not query.code:
             return {"error": "Invalid code"}, HTTPStatus.BAD_REQUEST
         try:
-            service = application_services().resolve_data_source_oauth(provider)
+            service = application_services().data_sources.resolve_oauth(provider)
             service.bind(request_context, code=query.code)
         except InvalidDataSourceOAuthProviderError:
             return _invalid_provider_response()
@@ -198,7 +198,7 @@ class OAuthDataSourceSync(Resource):
     @console_account_admission()
     def get(self, request_context: RequestContext, provider: str, binding_id: UUID):
         try:
-            service = application_services().resolve_data_source_oauth(provider)
+            service = application_services().data_sources.resolve_oauth(provider)
             service.sync(
                 request_context,
                 binding_id=str(binding_id),

api/controllers/console/datasets/data_source.py

@@ -1,53 +1,59 @@
-import json
-from collections.abc import Generator
 from datetime import datetime
-from typing import Any, Literal, cast
+from typing import Never
 from uuid import UUID
 
-from flask import request
 from flask_restx import Resource
 from pydantic import BaseModel, Field, field_serializer
-from sqlalchemy import select
-from sqlalchemy.orm import Session
-from werkzeug.exceptions import NotFound
 
+from configs import dify_config
+from controllers.common.errors import InvalidArgumentError, NotFoundError
 from controllers.common.fields import SimpleResultResponse, TextContentResponse
 from controllers.common.rbac import DatasetId, RBACCheck, Workspace
 from controllers.common.schema import query_params_from_model, register_response_schema_models, register_schema_models
-from controllers.common.session import with_session
-from core.datasource.entities.datasource_entities import DatasourceProviderType, OnlineDocumentPagesMessage
-from core.datasource.online_document.online_document_plugin import OnlineDocumentDatasourcePlugin
-from core.entities.knowledge_entities import IndexingEstimate
-from core.indexing_runner import IndexingRunner
-from core.rag.extractor.entity.datasource_type import DatasourceType
-from core.rag.extractor.entity.extract_setting import ExtractSetting, NotionInfo
-from core.rag.extractor.notion_extractor import NotionExtractor
-from extensions.ext_database import db
-from fields.base import ResponseModel
-from libs.datetime_utils import naive_utc_now
-from libs.helper import dump_response, to_timestamp
-from libs.login import login_required
-from models import Account, DataSourceOauthBinding, Document
-from services.dataset_service import DatasetService, DocumentService
-from services.datasource_provider_service import DatasourceProviderService
-from tasks.document_indexing_sync_task import document_indexing_sync_task
-
-from .. import console_ns
-from ..wraps import (
-    RBACPermission,
-    account_initialization_required,
-    is_admin_or_owner_required,
-    model_validate,
-    rbac_permission_required,
-    setup_required,
-    with_current_tenant_id,
-    with_current_user,
+from controllers.console import console_ns
+from controllers.console.app.error import ProviderNotInitializeError
+from controllers.console.datasets.error import (
+    DatasetAccessDeniedRequestError,
+    IndexingEstimateError,
+    InvalidActionError,
 )
+from controllers.console.flask_admission import console_account_admission
+from controllers.console.wraps import model_validate
+from core.entities.knowledge_entities import IndexingEstimate
+from core.rbac import RBACPermission
+from extensions.ext_application_services import application_services
+from fields.base import ResponseModel
+from fields.dataset_fields import NotionEstimateWorkspacePayload
+from libs.helper import dump_response, to_timestamp
+from machinery.context import RequestContext
+from models.account import TenantAccountRole
+from services.data_source.binding_application_service import (
+    BindingAction,
+    DataSourceBindingNotFoundError,
+    DataSourceBindingStateError,
+)
+from services.data_source.entities.notion_import import NotionPageType
+from services.data_source.notion_import_application_service import (
+    DatasetIsNotNotionSourceError,
+    NotionImportCredentialUnavailableError,
+)
+from services.knowledge.dataset_access import DatasetAccessDeniedError, DatasetNotFoundError
+from services.knowledge.document_sync import SyncDocumentNotFoundError, SyncDocumentSourceError
+from services.knowledge.entities.indexing_estimate import NewSourcesEstimateCommand, NotionEstimateSource
+from services.knowledge.indexing.estimate import (
+    EstimateSourceNotFoundError,
+    IndexingEstimateCredentialUnavailableError,
+    IndexingEstimateExecutionError,
+    IndexingEstimateProviderUnavailableError,
+)
+
+_ADMIN_OR_OWNER_ROLES = frozenset({TenantAccountRole.ADMIN, TenantAccountRole.OWNER})
+_DATA_SOURCE_OAUTH_BASE_PATH = "/console/api/oauth/data-source"
 
 
 class NotionEstimatePayload(BaseModel):
-    notion_info_list: list[dict[str, Any]]
-    process_rule: dict[str, Any]
+    notion_info_list: list[NotionEstimateWorkspacePayload] = Field(min_length=1)
+    process_rule: dict[str, object]
     doc_form: str = Field(default="text_model")
     doc_language: str = Field(default="English")
 
@@ -72,7 +78,7 @@ class DataSourceIntegratePageResponse(ResponseModel):
     page_id: str
     page_icon: DataSourceIntegrateIconResponse | None
     parent_id: str
-    type: str
+    type: NotionPageType
 
 
 class DataSourceIntegrateWorkspaceResponse(ResponseModel):
@@ -106,7 +112,7 @@ class NotionIntegratePageResponse(ResponseModel):
     page_id: str
     page_icon: DataSourceIntegrateIconResponse | None
     parent_id: str | None
-    type: str
+    page_type: NotionPageType = Field(alias="type")
     is_bound: bool
 
 
@@ -132,325 +138,204 @@ register_response_schema_models(
 )
 
 
-@console_ns.route(
-    "/data-source/integrates",
-    "/data-source/integrates/<uuid:binding_id>/<string:action>",
-)
-class DataSourceApi(Resource):
-    @setup_required
-    @login_required
-    @is_admin_or_owner_required
-    @rbac_permission_required(RBACCheck(RBACPermission.CREDENTIAL_MANAGE, Workspace()))
-    @account_initialization_required
+def _raise_dataset_access_error(error: DatasetNotFoundError | DatasetAccessDeniedError) -> Never:
+    if isinstance(error, DatasetNotFoundError):
+        raise NotFoundError(description="Dataset not found.") from None
+    raise DatasetAccessDeniedRequestError(description=str(error)) from None
+
+
+@console_ns.route("/data-source/integrates")
+class DataSourceIntegrationListApi(Resource):
     @console_ns.response(200, "Success", console_ns.models[DataSourceIntegrateListResponse.__name__])
-    @with_current_tenant_id
-    def get(self, current_tenant_id: str) -> tuple[dict[str, Any], int]:
-        # get workspace data source integrates
-        data_source_integrates = db.session.scalars(
-            select(DataSourceOauthBinding).where(
-                DataSourceOauthBinding.tenant_id == current_tenant_id,
-                DataSourceOauthBinding.disabled == False,
-            )
-        ).all()
+    @console_account_admission(
+        allowed_roles=_ADMIN_OR_OWNER_ROLES,
+        rbac_checks=(RBACCheck(RBACPermission.CREDENTIAL_MANAGE, Workspace()),),
+    )
+    def get(self, request_context: RequestContext) -> tuple[dict[str, object], int]:
+        bindings = application_services().data_sources.bindings.list_integrations(request_context)
+        base_url = dify_config.CONSOLE_API_URL.rstrip("/")
+        data = [
+            {
+                "id": binding.id,
+                "provider": binding.provider,
+                "created_at": binding.created_at,
+                "is_bound": True,
+                "disabled": binding.disabled,
+                "source_info": binding.source_info,
+                "link": f"{base_url}{_DATA_SOURCE_OAUTH_BASE_PATH}/{binding.provider}",
+            }
+            for binding in bindings
+        ]
+        return dump_response(DataSourceIntegrateListResponse, {"data": data}), 200
 
-        base_url = request.url_root.rstrip("/")
-        data_source_oauth_base_path = "/console/api/oauth/data-source"
-        providers = ["notion"]
 
-        integrate_data: list[dict[str, bool | datetime | dict[str, Any] | str | None]] = []
-        for provider in providers:
-            # existing_integrate = next((ai for ai in data_source_integrates if ai.provider == provider), None)
-            existing_integrates = filter(lambda item: item.provider == provider, data_source_integrates)
-            if existing_integrates:
-                for existing_integrate in list(existing_integrates):
-                    integrate_data.append(
-                        {
-                            "id": existing_integrate.id,
-                            "provider": provider,
-                            "created_at": existing_integrate.created_at,
-                            "is_bound": True,
-                            "disabled": existing_integrate.disabled,
-                            "source_info": existing_integrate.source_info,
-                            "link": f"{base_url}{data_source_oauth_base_path}/{provider}",
-                        }
-                    )
-            else:
-                integrate_data.append(
-                    {
-                        "id": None,
-                        "provider": provider,
-                        "created_at": None,
-                        "source_info": None,
-                        "is_bound": False,
-                        "disabled": None,
-                        "link": f"{base_url}{data_source_oauth_base_path}/{provider}",
-                    }
-                )
-        return dump_response(DataSourceIntegrateListResponse, {"data": integrate_data}), 200
-
-    @setup_required
-    @login_required
-    @is_admin_or_owner_required
-    @rbac_permission_required(RBACCheck(RBACPermission.CREDENTIAL_MANAGE, Workspace()))
-    @account_initialization_required
+@console_ns.route("/data-source/integrates/<uuid:binding_id>/<string:action>")
+class DataSourceIntegrationApi(Resource):
     @console_ns.response(200, "Success", console_ns.models[SimpleResultResponse.__name__])
-    @with_current_tenant_id
-    @with_session
+    @console_account_admission(
+        allowed_roles=_ADMIN_OR_OWNER_ROLES,
+        rbac_checks=(RBACCheck(RBACPermission.CREDENTIAL_MANAGE, Workspace()),),
+    )
     def patch(
-        self, session: Session, current_tenant_id: str, binding_id: UUID, action: Literal["enable", "disable"]
-    ) -> tuple[dict[str, str], int]:
-        binding_id_str = str(binding_id)
-        data_source_binding = session.scalar(
-            select(DataSourceOauthBinding).where(
-                DataSourceOauthBinding.id == binding_id_str, DataSourceOauthBinding.tenant_id == current_tenant_id
+        self,
+        request_context: RequestContext,
+        binding_id: UUID,
+        action: str,
+    ) -> tuple[dict[str, object], int]:
+        try:
+            validated_action = BindingAction(action)
+        except ValueError as error:
+            raise InvalidActionError() from error
+        try:
+            application_services().data_sources.bindings.change_state(
+                request_context,
+                str(binding_id),
+                validated_action,
             )
-        )
-        if data_source_binding is None:
-            raise NotFound("Data source binding not found.")
-        # enable binding
-        match action:
-            case "enable":
-                if data_source_binding.disabled:
-                    data_source_binding.disabled = False
-                    data_source_binding.updated_at = naive_utc_now()
-                else:
-                    raise ValueError("Data source is not disabled.")
-            # disable binding
-            case "disable":
-                if not data_source_binding.disabled:
-                    data_source_binding.disabled = True
-                    data_source_binding.updated_at = naive_utc_now()
-                else:
-                    raise ValueError("Data source is disabled.")
-        return {"result": "success"}, 200
+        except DataSourceBindingNotFoundError as error:
+            raise NotFoundError(description="Data source binding not found.") from error
+        except DataSourceBindingStateError as error:
+            raise InvalidArgumentError(description=str(error)) from error
+        return dump_response(SimpleResultResponse, {"result": "success"}), 200
 
 
 @console_ns.route("/notion/pre-import/pages")
 class DataSourceNotionListApi(Resource):
-    @setup_required
-    @login_required
-    @account_initialization_required
     @console_ns.doc(params=query_params_from_model(DataSourceNotionListQuery))
     @console_ns.response(200, "Success", console_ns.models[NotionIntegrateInfoListResponse.__name__])
-    @with_current_user
-    @with_current_tenant_id
-    @with_session(write=False)
+    @console_account_admission()
     @model_validate(DataSourceNotionListQuery)
     def get(
         self,
         req_data: DataSourceNotionListQuery,
-        session: Session,
-        current_tenant_id: str,
-        current_user: Account,
-    ) -> tuple[dict[str, Any], int]:
-        datasource_provider_service = DatasourceProviderService()
-        credential = datasource_provider_service.get_datasource_credentials(
-            tenant_id=current_tenant_id,
-            credential_id=req_data.credential_id,
-            provider="notion_datasource",
-            plugin_id="langgenius/notion_datasource",
-        )
-        if not credential:
-            raise NotFound("Credential not found.")
-        exist_page_ids = []
-        # import notion in the exist dataset
-        if req_data.dataset_id:
-            dataset = DatasetService.get_dataset(req_data.dataset_id, session)
-            if not dataset:
-                raise NotFound("Dataset not found.")
-            if dataset.data_source_type != "notion_import":
-                raise ValueError("Dataset is not notion type.")
-
-            documents = session.scalars(
-                select(Document).where(
-                    Document.dataset_id == req_data.dataset_id,
-                    Document.tenant_id == current_tenant_id,
-                    Document.data_source_type == "notion_import",
-                    Document.enabled.is_(True),
-                )
-            ).all()
-            if documents:
-                for document in documents:
-                    data_source_info = json.loads(document.data_source_info)
-                    exist_page_ids.append(data_source_info["notion_page_id"])
-        # get all authorized pages
-        from core.datasource.datasource_manager import DatasourceManager
-
-        datasource_runtime = DatasourceManager.get_datasource_runtime(
-            provider_id="langgenius/notion_datasource/notion_datasource",
-            datasource_name="notion_datasource",
-            tenant_id=current_tenant_id,
-            datasource_type=DatasourceProviderType.ONLINE_DOCUMENT,
-        )
-        datasource_provider_service = DatasourceProviderService()
-        if credential:
-            datasource_runtime.runtime.credentials = credential
-        datasource_runtime = cast(OnlineDocumentDatasourcePlugin, datasource_runtime)
-        online_document_result: Generator[OnlineDocumentPagesMessage, None, None] = (
-            datasource_runtime.get_online_document_pages(
-                user_id=current_user.id,
-                datasource_parameters={},
-                provider_type=datasource_runtime.datasource_provider_type(),
-            )
-        )
+        request_context: RequestContext,
+    ) -> tuple[dict[str, object], int]:
         try:
-            pages = []
-            workspace_info = {}
-            for message in online_document_result:
-                result = message.result
-                for info in result:
-                    workspace_info = {
-                        "workspace_id": info.workspace_id,
-                        "workspace_name": info.workspace_name,
-                        "workspace_icon": info.workspace_icon,
-                    }
-                    for page in info.pages:
-                        page_info = {
-                            "page_id": page.page_id,
-                            "page_name": page.page_name,
-                            "type": page.type,
-                            "parent_id": page.parent_id,
-                            "is_bound": page.page_id in exist_page_ids,
-                            "page_icon": page.page_icon,
-                        }
-                        pages.append(page_info)
-        except Exception as e:
-            raise e
-        notion_info = [{**workspace_info, "pages": pages}] if workspace_info else []
-        return dump_response(NotionIntegrateInfoListResponse, {"notion_info": notion_info}), 200
+            result = application_services().data_sources.notion_imports.list_pages(
+                request_context,
+                credential_id=req_data.credential_id,
+                dataset_id=req_data.dataset_id,
+            )
+        except NotionImportCredentialUnavailableError as error:
+            raise NotFoundError(description="Credential not found.") from error
+        except (DatasetNotFoundError, DatasetAccessDeniedError) as error:
+            _raise_dataset_access_error(error)
+        except DatasetIsNotNotionSourceError as error:
+            raise InvalidArgumentError(description="Dataset is not notion type.") from error
+        return dump_response(NotionIntegrateInfoListResponse, {"notion_info": result.workspaces}), 200
 
 
 @console_ns.route("/notion/pages/<uuid:page_id>/<string:page_type>/preview")
 class DataSourceNotionPreviewApi(Resource):
     """Preview one authorized Notion page through the datasource credential."""
 
-    @setup_required
-    @login_required
-    @account_initialization_required
     @console_ns.doc(params=query_params_from_model(DataSourceNotionPreviewQuery))
     @console_ns.response(200, "Success", console_ns.models[TextContentResponse.__name__])
-    @with_current_tenant_id
+    @console_account_admission()
     @model_validate(DataSourceNotionPreviewQuery)
     def get(
         self,
         req_data: DataSourceNotionPreviewQuery,
-        current_tenant_id: str,
+        request_context: RequestContext,
         page_id: UUID,
         page_type: str,
-    ) -> tuple[dict[str, str], int]:
-
-        datasource_provider_service = DatasourceProviderService()
-        credential = datasource_provider_service.get_datasource_credentials(
-            tenant_id=current_tenant_id,
-            credential_id=req_data.credential_id,
-            provider="notion_datasource",
-            plugin_id="langgenius/notion_datasource",
-        )
-
-        page_id_str = str(page_id)
-
-        extractor = NotionExtractor(
-            notion_workspace_id="",
-            notion_obj_id=page_id_str,
-            notion_page_type=page_type,
-            notion_access_token=credential.get("integration_secret"),
-            tenant_id=current_tenant_id,
-        )
-
-        text_docs = extractor.extract()
-        return {"content": "\n".join([doc.page_content for doc in text_docs])}, 200
+    ) -> tuple[dict[str, object], int]:
+        try:
+            validated_page_type = NotionPageType(page_type)
+        except ValueError as error:
+            raise InvalidArgumentError(description="Invalid Notion page type.") from error
+        try:
+            content = application_services().data_sources.notion_imports.preview_page(
+                request_context,
+                credential_id=req_data.credential_id,
+                page_id=str(page_id),
+                page_type=validated_page_type,
+            )
+        except NotionImportCredentialUnavailableError as error:
+            raise NotFoundError(description="Credential not found.") from error
+        return dump_response(TextContentResponse, {"content": content}), 200
 
 
 @console_ns.route("/datasets/notion-indexing-estimate")
 class DataSourceNotionIndexingEstimateApi(Resource):
     """Estimate indexing work for selected Notion pages."""
 
-    @setup_required
-    @login_required
-    @account_initialization_required
     @console_ns.expect(console_ns.models[NotionEstimatePayload.__name__])
     @console_ns.response(200, "Success", console_ns.models[IndexingEstimate.__name__])
-    @with_current_tenant_id
-    @with_session
+    @console_account_admission()
     @model_validate(NotionEstimatePayload)
     def post(
         self,
         req_data: NotionEstimatePayload,
-        session: Session,
-        current_tenant_id: str,
-    ) -> tuple[dict[str, Any], int]:
-        args = req_data.model_dump()
-        # validate args
-        DocumentService.estimate_args_validate(args)
-        notion_info_list = req_data.notion_info_list
-        extract_settings = []
-        for notion_info in notion_info_list:
-            workspace_id = notion_info["workspace_id"]
-            credential_id = notion_info.get("credential_id")
-            for page in notion_info["pages"]:
-                extract_setting = ExtractSetting(
-                    datasource_type=DatasourceType.NOTION,
-                    notion_info=NotionInfo.model_validate(
-                        {
-                            "credential_id": credential_id,
-                            "notion_workspace_id": workspace_id,
-                            "notion_obj_id": page["page_id"],
-                            "notion_page_type": page["type"],
-                            "tenant_id": current_tenant_id,
-                        }
-                    ),
-                    document_model=args["doc_form"],
+        request_context: RequestContext,
+    ) -> tuple[dict[str, object], int]:
+        command = NewSourcesEstimateCommand(
+            sources=tuple(
+                NotionEstimateSource(
+                    workspace_id=workspace.workspace_id,
+                    page_id=page.page_id,
+                    page_type=page.page_type,
+                    credential_id=workspace.credential_id,
                 )
-                extract_settings.append(extract_setting)
-        indexing_runner = IndexingRunner()
-        response = indexing_runner.indexing_estimate(
-            tenant_id=current_tenant_id,
-            extract_settings=extract_settings,
-            tmp_processing_rule=args["process_rule"],
-            doc_form=args["doc_form"],
-            doc_language=args["doc_language"],
-            session=session,
+                for workspace in req_data.notion_info_list
+                for page in workspace.pages
+            ),
+            process_rule=req_data.process_rule,
+            doc_form=req_data.doc_form,
+            doc_language=req_data.doc_language,
         )
+        try:
+            response = application_services().knowledge.indexing_estimates.estimate_new_sources(
+                request_context,
+                command,
+            )
+        except IndexingEstimateCredentialUnavailableError as error:
+            raise NotFoundError(description="Credential not found.") from error
+        except EstimateSourceNotFoundError as error:
+            raise NotFoundError(description=str(error)) from error
+        except IndexingEstimateProviderUnavailableError as error:
+            raise ProviderNotInitializeError(str(error)) from error
+        except IndexingEstimateExecutionError as error:
+            raise IndexingEstimateError(str(error)) from error
         return dump_response(IndexingEstimate, response), 200
 
 
 @console_ns.route("/datasets/<uuid:dataset_id>/notion/sync")
 class DataSourceNotionDatasetSyncApi(Resource):
-    @setup_required
-    @login_required
-    @account_initialization_required
     @console_ns.response(200, "Success", console_ns.models[SimpleResultResponse.__name__])
-    @rbac_permission_required(RBACCheck(RBACPermission.DATASET_EDIT, DatasetId()))
-    @with_session(write=False)
-    def get(self, session: Session, dataset_id: UUID) -> tuple[dict[str, str], int]:
-        dataset_id_str = str(dataset_id)
-        dataset = DatasetService.get_dataset(dataset_id_str, session)
-        if dataset is None:
-            raise NotFound("Dataset not found.")
-
-        documents = DocumentService.get_document_by_dataset_id(dataset_id_str, session)
-        for document in documents:
-            document_indexing_sync_task.delay(dataset_id_str, document.id)
-        return {"result": "success"}, 200
+    @console_account_admission(
+        rbac_checks=(RBACCheck(RBACPermission.DATASET_EDIT, DatasetId()),),
+    )
+    def get(self, request_context: RequestContext, dataset_id: UUID) -> tuple[dict[str, object], int]:
+        try:
+            application_services().knowledge.document_sync.sync_dataset(request_context, str(dataset_id))
+        except (DatasetNotFoundError, DatasetAccessDeniedError) as error:
+            _raise_dataset_access_error(error)
+        return dump_response(SimpleResultResponse, {"result": "success"}), 200
 
 
 @console_ns.route("/datasets/<uuid:dataset_id>/documents/<uuid:document_id>/notion/sync")
 class DataSourceNotionDocumentSyncApi(Resource):
-    @setup_required
-    @login_required
-    @account_initialization_required
     @console_ns.response(200, "Success", console_ns.models[SimpleResultResponse.__name__])
-    @rbac_permission_required(RBACCheck(RBACPermission.DATASET_EDIT, DatasetId()))
-    @with_session(write=False)
-    def get(self, session: Session, dataset_id: UUID, document_id: UUID) -> tuple[dict[str, str], int]:
-        dataset_id_str = str(dataset_id)
-        document_id_str = str(document_id)
-        dataset = DatasetService.get_dataset(dataset_id_str, session)
-        if dataset is None:
-            raise NotFound("Dataset not found.")
-
-        document = DocumentService.get_document(dataset_id_str, document_id_str, session=session)
-        if document is None:
-            raise NotFound("Document not found.")
-        document_indexing_sync_task.delay(dataset_id_str, document_id_str)
-        return {"result": "success"}, 200
+    @console_account_admission(
+        rbac_checks=(RBACCheck(RBACPermission.DATASET_EDIT, DatasetId()),),
+    )
+    def get(
+        self,
+        request_context: RequestContext,
+        dataset_id: UUID,
+        document_id: UUID,
+    ) -> tuple[dict[str, object], int]:
+        try:
+            application_services().knowledge.document_sync.sync_document(
+                request_context,
+                str(dataset_id),
+                str(document_id),
+            )
+        except (DatasetNotFoundError, DatasetAccessDeniedError) as error:
+            _raise_dataset_access_error(error)
+        except SyncDocumentNotFoundError as error:
+            raise NotFoundError(description="Document not found.") from error
+        except SyncDocumentSourceError as error:
+            raise InvalidArgumentError(description=str(error)) from error
+        return dump_response(SimpleResultResponse, {"result": "success"}), 200

api/controllers/console/datasets/error.py

@@ -25,6 +25,12 @@ class InvalidActionError(BaseHTTPException):
     code = 400
 
 
+class DatasetAccessDeniedRequestError(BaseHTTPException):
+    error_code = "dataset_access_denied"
+    description = "Dataset access denied."
+    code = 403
+
+
 class DocumentAlreadyFinishedError(BaseHTTPException):
     error_code = "document_already_finished"
     description = "The document has been processed. Please refresh the page or go to the document details."

api/controllers/console/datasets/external.py

@@ -1,16 +1,13 @@
 from __future__ import annotations
 
-from dataclasses import dataclass
 from datetime import datetime
-from typing import Any
+from typing import Any, Never
 from uuid import UUID
 
 from flask_restx import Resource
 from pydantic import AliasChoices, BaseModel, Field, field_validator
-from sqlalchemy.orm import Session
 from werkzeug.exceptions import Forbidden, InternalServerError, NotFound
 
-import services
 from controllers.common.fields import UsageCountResponse
 from controllers.common.rbac import DatasetId, RBACCheck, Workspace
 from controllers.common.schema import (
@@ -18,31 +15,42 @@ from controllers.common.schema import (
     register_response_schema_models,
     register_schema_models,
 )
-from controllers.common.session import with_session
 from controllers.console import console_ns
 from controllers.console.datasets.error import DatasetNameDuplicateError
+from controllers.console.flask_admission import console_account_admission
 from controllers.console.wraps import (
     RBACPermission,
-    account_initialization_required,
-    edit_permission_required,
     model_validate,
-    rbac_permission_required,
-    setup_required,
-    with_current_tenant_id,
-    with_current_user,
 )
+from extensions.ext_application_services import application_services
 from fields.base import ResponseModel
-from fields.dataset_fields import DatasetDetailResponse, dataset_detail_response_source
+from fields.dataset_fields import (
+    DatasetDetailResponse,
+)
 from libs.helper import dump_response
-from libs.login import login_required
-from libs.pagination import clamp_pagination
-from models import Account
-from models.dataset import ExternalKnowledgeApis
-from services.dataset_service import DatasetService
-from services.enterprise import rbac_service as enterprise_rbac_service
+from machinery.context import RequestContext
+from models.account import TenantAccountRole
 from services.entities.external_knowledge_entities.external_knowledge_entities import ExternalDatasetCreatePayload
-from services.external_knowledge_service import ExternalDatasetService
-from services.hit_testing_service import HitTestingService
+from services.errors.dataset import DatasetNameDuplicateError as DatasetNameDuplicateFailure
+from services.knowledge.dataset_access import DatasetAccessDeniedError, DatasetNotFoundError
+from services.knowledge.external.application import ExternalHitTestingError, ExternalTemplateNotFoundError
+
+_DATASET_EDIT_ROLES = frozenset(
+    {TenantAccountRole.OWNER, TenantAccountRole.ADMIN, TenantAccountRole.EDITOR, TenantAccountRole.DATASET_OPERATOR}
+)
+_EXTERNAL_CONNECT_ROLES = frozenset({TenantAccountRole.OWNER, TenantAccountRole.ADMIN, TenantAccountRole.EDITOR})
+
+
+def _raise_external_error(error: Exception) -> Never:
+    if isinstance(error, (DatasetNotFoundError, ExternalTemplateNotFoundError)):
+        raise NotFound(str(error)) from error
+    if isinstance(error, DatasetAccessDeniedError):
+        raise Forbidden(str(error)) from error
+    if isinstance(error, DatasetNameDuplicateFailure):
+        raise DatasetNameDuplicateError() from error
+    if isinstance(error, ExternalHitTestingError):
+        raise InternalServerError(str(error)) from error
+    raise error
 
 
 class ExternalKnowledgeApiPayload(BaseModel):
@@ -85,28 +93,6 @@ class ExternalKnowledgeApiResponse(ResponseModel):
         return value
 
 
-@dataclass(frozen=True)
-class ExternalKnowledgeApiResponseSource:
-    external_knowledge_api: ExternalKnowledgeApis
-    session: Session
-
-    @property
-    def dataset_bindings(self) -> Any:
-        return self.external_knowledge_api.get_dataset_bindings(session=self.session)
-
-    def __getattr__(self, name: str) -> Any:
-        return getattr(self.external_knowledge_api, name)  # guard-ignore: no-new-getattr -- delegates model fields
-
-
-def external_knowledge_api_response(
-    external_knowledge_api: ExternalKnowledgeApis, *, session: Session
-) -> ExternalKnowledgeApiResponse:
-    return ExternalKnowledgeApiResponse.model_validate(
-        ExternalKnowledgeApiResponseSource(external_knowledge_api=external_knowledge_api, session=session),
-        from_attributes=True,
-    )
-
-
 class ExternalKnowledgeApiListResponse(ResponseModel):
     data: list[ExternalKnowledgeApiResponse]
     has_more: bool
@@ -161,25 +147,13 @@ class ExternalApiTemplateListApi(Resource):
         "External API templates retrieved successfully",
         console_ns.models[ExternalKnowledgeApiListResponse.__name__],
     )
-    @setup_required
-    @login_required
-    @with_current_tenant_id
-    @account_initialization_required
-    @with_session(write=False)
+    @console_account_admission()
     @model_validate(ExternalApiTemplateListQuery)
-    def get(self, req_data: ExternalApiTemplateListQuery, session: Session, current_tenant_id: str):
-
-        effective_page, effective_limit = clamp_pagination(req_data.page, req_data.limit, 100)
-        external_knowledge_apis, total = ExternalDatasetService.get_external_knowledge_apis(
-            effective_page, effective_limit, current_tenant_id, req_data.keyword, session=session
+    def get(self, req_data: ExternalApiTemplateListQuery, request_context: RequestContext):
+        result = application_services().knowledge.external.list_templates(
+            request_context, page=req_data.page, limit=req_data.limit, keyword=req_data.keyword
         )
-        return ExternalKnowledgeApiListResponse(
-            data=[external_knowledge_api_response(item, session=session) for item in external_knowledge_apis],
-            has_more=effective_page * effective_limit < total,
-            limit=effective_limit,
-            total=total,
-            page=effective_page,
-        ).model_dump(mode="json"), 200
+        return dump_response(ExternalKnowledgeApiListResponse, result), 200
 
     @console_ns.doc("create_external_api_template")
     @console_ns.doc(description="Create external knowledge API template")
@@ -190,48 +164,16 @@ class ExternalApiTemplateListApi(Resource):
         console_ns.models[ExternalKnowledgeApiResponse.__name__],
     )
     @console_ns.response(403, "Permission denied")
-    @setup_required
-    @login_required
-    @account_initialization_required
-    @with_current_user
-    @with_current_tenant_id
-    @with_session
+    @console_account_admission(allowed_roles=_DATASET_EDIT_ROLES)
     @model_validate(ExternalKnowledgeApiPayload)
-    def post(
-        self,
-        req_data: ExternalKnowledgeApiPayload,
-        session: Session,
-        current_tenant_id: str,
-        current_user: Account,
-    ):
-
-        ExternalDatasetService.validate_api_list(req_data.settings)
-
-        # The role of the current user in the ta table must be admin, owner, or editor, or dataset_operator
-        if not current_user.is_dataset_editor:
-            raise Forbidden()
-
+    def post(self, req_data: ExternalKnowledgeApiPayload, request_context: RequestContext):
         try:
-            external_knowledge_api = ExternalDatasetService.create_external_knowledge_api(
-                tenant_id=current_tenant_id,
-                user_id=current_user.id,
-                args=req_data.model_dump(),
-                session=session,
+            result = application_services().knowledge.external.create_template(
+                request_context, name=req_data.name, settings=req_data.settings
             )
-        except services.errors.dataset.DatasetNameDuplicateError:
-            raise DatasetNameDuplicateError()
-
-        # if dify_config.RBAC_ENABLED:
-        #     enterprise_rbac_service.RBACService.DatasetAccess.replace_whitelist(
-        #         current_tenant_id,
-        #         current_user.id,
-        #         external_knowledge_api.id,
-        #         enterprise_rbac_service.ReplaceMemberBindings(automatic_include_workspace_members=True),
-        #     )
-        #     initialize_created_app_rbac_access_task.delay(
-        #         current_tenant_id, current_user.id, dataset_id=external_knowledge_api.id)
-
-        return external_knowledge_api_response(external_knowledge_api, session=session).model_dump(mode="json"), 201
+        except Exception as error:
+            _raise_external_error(error)
+        return dump_response(ExternalKnowledgeApiResponse, result), 201
 
 
 @console_ns.route("/datasets/external-knowledge-api/<uuid:external_knowledge_api_id>")
@@ -245,20 +187,15 @@ class ExternalApiTemplateApi(Resource):
         console_ns.models[ExternalKnowledgeApiResponse.__name__],
     )
     @console_ns.response(404, "Template not found")
-    @setup_required
-    @login_required
-    @account_initialization_required
-    @with_current_tenant_id
-    @with_session
-    def get(self, session: Session, current_tenant_id: str, external_knowledge_api_id: UUID):
-        external_knowledge_api_id_str = str(external_knowledge_api_id)
-        external_knowledge_api = ExternalDatasetService.get_external_knowledge_api(
-            external_knowledge_api_id=external_knowledge_api_id_str, tenant_id=current_tenant_id, session=session
-        )
-        if external_knowledge_api is None:
-            raise NotFound("API template not found.")
-
-        return external_knowledge_api_response(external_knowledge_api, session=session).model_dump(mode="json"), 200
+    @console_account_admission()
+    def get(self, request_context: RequestContext, external_knowledge_api_id: UUID):
+        try:
+            result = application_services().knowledge.external.get_template(
+                request_context, template_id=str(external_knowledge_api_id)
+            )
+        except Exception as error:
+            _raise_external_error(error)
+        return dump_response(ExternalKnowledgeApiResponse, result), 200
 
     @console_ns.doc("update_external_api_template")
     @console_ns.doc(description="Update external knowledge API template")
@@ -270,51 +207,31 @@ class ExternalApiTemplateApi(Resource):
         console_ns.models[ExternalKnowledgeApiResponse.__name__],
     )
     @console_ns.response(404, "Template not found")
-    @setup_required
-    @login_required
-    @account_initialization_required
-    @with_current_user
-    @with_current_tenant_id
-    @with_session
+    @console_account_admission()
     @model_validate(ExternalKnowledgeApiPayload)
     def patch(
-        self,
-        req_data: ExternalKnowledgeApiPayload,
-        session: Session,
-        current_tenant_id: str,
-        current_user: Account,
-        external_knowledge_api_id: UUID,
+        self, req_data: ExternalKnowledgeApiPayload, request_context: RequestContext, external_knowledge_api_id: UUID
     ):
-        external_knowledge_api_id_str = str(external_knowledge_api_id)
+        try:
+            result = application_services().knowledge.external.update_template(
+                request_context,
+                template_id=str(external_knowledge_api_id),
+                name=req_data.name,
+                settings=req_data.settings,
+            )
+        except Exception as error:
+            _raise_external_error(error)
+        return dump_response(ExternalKnowledgeApiResponse, result), 200
 
-        ExternalDatasetService.validate_api_list(req_data.settings)
-
-        external_knowledge_api = ExternalDatasetService.update_external_knowledge_api(
-            tenant_id=current_tenant_id,
-            user_id=current_user.id,
-            external_knowledge_api_id=external_knowledge_api_id_str,
-            args=req_data.model_dump(),
-            session=session,
-        )
-
-        return external_knowledge_api_response(external_knowledge_api, session=session).model_dump(mode="json"), 200
-
-    @setup_required
-    @login_required
-    @account_initialization_required
     @console_ns.response(204, "External knowledge API deleted successfully")
-    @with_current_user
-    @with_current_tenant_id
-    @with_session
-    def delete(self, session: Session, current_tenant_id: str, current_user: Account, external_knowledge_api_id: UUID):
-        external_knowledge_api_id_str = str(external_knowledge_api_id)
-
-        if not (current_user.has_edit_permission or current_user.is_dataset_operator):
-            raise Forbidden()
-
-        ExternalDatasetService.delete_external_knowledge_api(
-            current_tenant_id, external_knowledge_api_id_str, session=session
-        )
+    @console_account_admission(allowed_roles=_DATASET_EDIT_ROLES)
+    def delete(self, request_context: RequestContext, external_knowledge_api_id: UUID):
+        try:
+            application_services().knowledge.external.delete_template(
+                request_context, template_id=str(external_knowledge_api_id)
+            )
+        except Exception as error:
+            _raise_external_error(error)
         return "", 204
 
 
@@ -324,18 +241,12 @@ class ExternalApiUseCheckApi(Resource):
     @console_ns.doc(description="Check if external knowledge API is being used")
     @console_ns.doc(params={"external_knowledge_api_id": "External knowledge API ID"})
     @console_ns.response(200, "Usage check completed successfully", console_ns.models[UsageCountResponse.__name__])
-    @setup_required
-    @login_required
-    @account_initialization_required
-    @with_current_tenant_id
-    @with_session
-    def get(self, session: Session, current_tenant_id: str, external_knowledge_api_id: UUID):
-        external_knowledge_api_id_str = str(external_knowledge_api_id)
-
-        external_knowledge_api_is_using, count = ExternalDatasetService.external_knowledge_api_use_check(
-            external_knowledge_api_id_str, current_tenant_id, session=session
+    @console_account_admission()
+    def get(self, request_context: RequestContext, external_knowledge_api_id: UUID):
+        is_using, count = application_services().knowledge.external.template_usage(
+            request_context, template_id=str(external_knowledge_api_id)
         )
-        return UsageCountResponse(is_using=external_knowledge_api_is_using, count=count).model_dump(mode="json"), 200
+        return dump_response(UsageCountResponse, {"is_using": is_using, "count": count}), 200
 
 
 @console_ns.route("/datasets/external")
@@ -348,50 +259,17 @@ class ExternalDatasetCreateApi(Resource):
     )
     @console_ns.response(400, "Invalid parameters")
     @console_ns.response(403, "Permission denied")
-    @setup_required
-    @login_required
-    @account_initialization_required
-    @edit_permission_required
-    @rbac_permission_required(RBACCheck(RBACPermission.DATASET_EXTERNAL_CONNECT, Workspace()))
-    @with_current_user
-    @with_current_tenant_id
-    @with_session
+    @console_account_admission(
+        allowed_roles=_EXTERNAL_CONNECT_ROLES,
+        rbac_checks=(RBACCheck(RBACPermission.DATASET_EXTERNAL_CONNECT, Workspace()),),
+    )
     @model_validate(ExternalDatasetCreatePayload)
-    def post(
-        self,
-        req_data: ExternalDatasetCreatePayload,
-        session: Session,
-        current_tenant_id: str,
-        current_user: Account,
-    ):
-        # The role of the current user in the ta table must be admin, owner, or editor
-
-        # The role of the current user in the ta table must be admin, owner, or editor, or dataset_operator
-        if not current_user.is_dataset_editor:
-            raise Forbidden()
-
+    def post(self, req_data: ExternalDatasetCreatePayload, request_context: RequestContext):
         try:
-            dataset = ExternalDatasetService.create_external_dataset(
-                tenant_id=current_tenant_id,
-                user_id=current_user.id,
-                args=req_data,
-                session=session,
-            )
-        except services.errors.dataset.DatasetNameDuplicateError:
-            raise DatasetNameDuplicateError()
-
-        dataset_id_str = str(dataset.id)
-        permission_keys_map = enterprise_rbac_service.RBACService.DatasetPermissions.batch_get(
-            str(current_tenant_id),
-            current_user.id,
-            [dataset_id_str],
-            session=session,
-        )
-        data = DatasetDetailResponse.model_validate(
-            dataset_detail_response_source(dataset, session=session)
-        ).model_dump(mode="json")
-        data["permission_keys"] = permission_keys_map.get(dataset_id_str, [])
-        return data, 201
+            result = application_services().knowledge.external.create_dataset(request_context, payload=req_data)
+        except Exception as error:
+            _raise_external_error(error)
+        return dump_response(DatasetDetailResponse, result), 201
 
 
 @console_ns.route("/datasets/<uuid:dataset_id>/external-hit-testing")
@@ -407,36 +285,17 @@ class ExternalKnowledgeHitTestingApi(Resource):
     )
     @console_ns.response(404, "Dataset not found")
     @console_ns.response(400, "Invalid parameters")
-    @setup_required
-    @login_required
-    @account_initialization_required
-    @with_current_user
-    @rbac_permission_required(RBACCheck(RBACPermission.DATASET_PIPELINE_TEST, DatasetId()))
-    @with_session
+    @console_account_admission(rbac_checks=(RBACCheck(RBACPermission.DATASET_PIPELINE_TEST, DatasetId()),))
     @model_validate(ExternalHitTestingPayload)
-    def post(self, req_data: ExternalHitTestingPayload, session: Session, current_user: Account, dataset_id: UUID):
-        dataset_id_str = str(dataset_id)
-        dataset = DatasetService.get_dataset(dataset_id_str, session)
-        if dataset is None:
-            raise NotFound("Dataset not found.")
-
+    def post(self, req_data: ExternalHitTestingPayload, request_context: RequestContext, dataset_id: UUID):
         try:
-            DatasetService.check_dataset_permission(dataset, current_user, session)
-        except services.errors.account.NoPermissionError as e:
-            raise Forbidden(str(e))
-
-        HitTestingService.hit_testing_args_check(req_data.model_dump())
-
-        try:
-            response = HitTestingService.external_retrieve(
-                session=session,
-                dataset=dataset,
+            result = application_services().knowledge.external.hit_testing(
+                request_context,
+                dataset_id=str(dataset_id),
                 query=req_data.query,
-                account=current_user,
-                external_retrieval_model=req_data.external_retrieval_model,
-                metadata_filtering_conditions=req_data.metadata_filtering_conditions,
+                retrieval_model=req_data.external_retrieval_model,
+                metadata_filters=req_data.metadata_filtering_conditions,
             )
-
-            return dump_response(ExternalHitTestingResponse, response)
-        except Exception as e:
-            raise InternalServerError(str(e))
+        except Exception as error:
+            _raise_external_error(error)
+        return dump_response(ExternalHitTestingResponse, result)

api/controllers/console/datasets/hit_testing_base.py

@@ -23,9 +23,9 @@ from graphon.model_runtime.errors.invoke import InvokeError
 from libs.login import resolve_account_fallback
 from models.account import Account
 from models.dataset import Dataset
-from services.dataset_service import DatasetService
-from services.entities.knowledge_entities.knowledge_entities import ExternalRetrievalModel, RetrievalModel
 from services.hit_testing_service import HitTestingService
+from services.knowledge.dataset_service import DatasetService
+from services.knowledge.entities.knowledge_entities import ExternalRetrievalModel, RetrievalModel
 
 logger = logging.getLogger(__name__)
 

api/controllers/console/datasets/metadata.py

@@ -1,43 +1,47 @@
-from typing import Literal
+from collections.abc import Callable
+from functools import wraps
 from uuid import UUID
 
 from flask_restx import Resource
-from sqlalchemy.orm import Session
 from werkzeug.exceptions import Forbidden, NotFound
 
-import services
 from controllers.common.controller_schemas import MetadataUpdatePayload
 from controllers.common.rbac import DatasetId, RBACCheck
 from controllers.common.schema import register_response_schema_models, register_schema_models
-from controllers.common.session import with_session
 from controllers.console import console_ns
-from controllers.console.wraps import (
-    RBACPermission,
-    account_initialization_required,
-    enterprise_license_required,
-    model_validate,
-    rbac_permission_required,
-    setup_required,
-    with_current_tenant_id,
-    with_current_user,
-)
+from controllers.console.flask_admission import console_account_admission
+from controllers.console.wraps import model_validate
+from core.rbac import RBACPermission
+from extensions.ext_application_services import application_services
 from fields.dataset_fields import (
     DatasetMetadataBuiltInFieldsResponse,
     DatasetMetadataListResponse,
     DatasetMetadataResponse,
 )
 from libs.helper import dump_response
-from libs.login import login_required
-from models.account import Account
-from services.dataset_service import DatasetService
-from services.entities.knowledge_entities.knowledge_entities import (
+from machinery.context import RequestContext
+from services.errors.metadata import MetadataResourceNotFoundError
+from services.knowledge.dataset_access import DatasetAccessDeniedError, DatasetNotFoundError
+from services.knowledge.entities.knowledge_entities import (
     DocumentMetadataOperation,
     MetadataArgs,
     MetadataDetail,
     MetadataOperationData,
 )
-from services.errors.metadata import MetadataResourceNotFoundError
-from services.metadata_service import MetadataService
+
+
+def metadata_errors[**P, T](method: Callable[P, T]) -> Callable[P, T]:
+    @wraps(method)
+    def wrapped(*args: P.args, **kwargs: P.kwargs) -> T:
+        try:
+            return method(*args, **kwargs)
+        except (DatasetNotFoundError, MetadataResourceNotFoundError) as error:
+            raise NotFound(str(error)) from error
+        except DatasetAccessDeniedError as error:
+            raise Forbidden(str(error)) from error
+
+    return wrapped
+
 
 register_schema_models(
     console_ns, MetadataArgs, MetadataOperationData, MetadataUpdatePayload, DocumentMetadataOperation, MetadataDetail
@@ -52,199 +56,111 @@ register_response_schema_models(
 
 @console_ns.route("/datasets/<uuid:dataset_id>/metadata")
 class DatasetMetadataCreateApi(Resource):
-    @setup_required
-    @login_required
-    @account_initialization_required
-    @enterprise_license_required
     @console_ns.response(201, "Metadata created successfully", console_ns.models[DatasetMetadataResponse.__name__])
     @console_ns.expect(console_ns.models[MetadataArgs.__name__])
-    @with_current_user
-    @with_current_tenant_id
-    @rbac_permission_required(RBACCheck(RBACPermission.DATASET_EDIT, DatasetId()))
-    @with_session
+    @console_account_admission(
+        rbac_checks=(RBACCheck(RBACPermission.DATASET_EDIT, DatasetId()),),
+    )
+    @metadata_errors
     @model_validate(MetadataArgs)
-    def post(
-        self,
-        req_data: MetadataArgs,
-        session: Session,
-        current_tenant_id: str,
-        current_user: Account,
-        dataset_id: UUID,
-    ):
-        dataset_id_str = str(dataset_id)
-        dataset = DatasetService.get_dataset(dataset_id_str, session)
-        if dataset is None:
-            raise NotFound("Dataset not found.")
-        DatasetService.check_dataset_permission(dataset, current_user, session)
-
-        metadata = MetadataService.create_metadata(
-            dataset_id_str, req_data, current_user, current_tenant_id, session=session
-        )
+    def post(self, req_data: MetadataArgs, request_context: RequestContext, dataset_id: UUID):
+        service = application_services().knowledge.metadata
+        ref = service.require_dataset(request_context, str(dataset_id))
+        metadata = service.create_metadata(ref, req_data, actor_id=request_context.account_id)
         return dump_response(DatasetMetadataResponse, metadata), 201
 
-    @setup_required
-    @login_required
-    @account_initialization_required
-    @enterprise_license_required
     @console_ns.response(
         200, "Metadata retrieved successfully", console_ns.models[DatasetMetadataListResponse.__name__]
     )
-    @with_current_user
-    @with_current_tenant_id
-    @rbac_permission_required(RBACCheck(RBACPermission.DATASET_READONLY, DatasetId()))
-    @with_session(write=False)
-    def get(self, session: Session, current_tenant_id: str, current_user: Account, dataset_id: UUID):
-        dataset_id_str = str(dataset_id)
-        dataset = DatasetService.get_dataset_for_tenant(dataset_id_str, current_tenant_id, session=session)
-        if dataset is None:
-            raise NotFound("Dataset not found.")
-        try:
-            DatasetService.check_dataset_permission(dataset, current_user, session)
-        except services.errors.account.NoPermissionError as e:
-            raise Forbidden(str(e))
-        metadata = MetadataService.get_dataset_metadatas(dataset, session)
-        return dump_response(DatasetMetadataListResponse, metadata), 200
+    @console_account_admission(
+        rbac_checks=(RBACCheck(RBACPermission.DATASET_READONLY, DatasetId()),),
+    )
+    @metadata_errors
+    def get(self, request_context: RequestContext, dataset_id: UUID):
+        service = application_services().knowledge.metadata
+        ref = service.require_dataset(request_context, str(dataset_id))
+        return dump_response(DatasetMetadataListResponse, service.get_dataset_metadatas(ref)), 200
 
 
 @console_ns.route("/datasets/<uuid:dataset_id>/metadata/<uuid:metadata_id>")
 class DatasetMetadataApi(Resource):
-    @setup_required
-    @login_required
-    @account_initialization_required
-    @enterprise_license_required
     @console_ns.response(200, "Metadata updated successfully", console_ns.models[DatasetMetadataResponse.__name__])
     @console_ns.expect(console_ns.models[MetadataUpdatePayload.__name__])
-    @with_current_user
-    @with_current_tenant_id
-    @rbac_permission_required(RBACCheck(RBACPermission.DATASET_EDIT, DatasetId()))
-    @with_session
+    @console_account_admission(
+        rbac_checks=(RBACCheck(RBACPermission.DATASET_EDIT, DatasetId()),),
+    )
+    @metadata_errors
     @model_validate(MetadataUpdatePayload)
     def patch(
-        self,
-        req_data: MetadataUpdatePayload,
-        session: Session,
-        current_tenant_id: str,
-        current_user: Account,
-        dataset_id: UUID,
-        metadata_id: UUID,
+        self, req_data: MetadataUpdatePayload, request_context: RequestContext, dataset_id: UUID, metadata_id: UUID
     ):
-        name = req_data.name
-
-        dataset_id_str = str(dataset_id)
-        metadata_id_str = str(metadata_id)
-        dataset = DatasetService.get_dataset_for_tenant(dataset_id_str, current_tenant_id, session=session)
-        if dataset is None:
-            raise NotFound("Dataset not found.")
-        DatasetService.check_dataset_permission(dataset, current_user, session)
-
-        metadata = MetadataService.update_metadata_name(dataset, metadata_id_str, name, current_user, session=session)
+        service = application_services().knowledge.metadata
+        ref = service.require_dataset(request_context, str(dataset_id))
+        metadata = service.update_metadata_name(
+            ref, str(metadata_id), req_data.name, actor_id=request_context.account_id
+        )
         return dump_response(DatasetMetadataResponse, metadata), 200
 
-    @setup_required
-    @login_required
-    @account_initialization_required
-    @enterprise_license_required
     @console_ns.response(204, "Metadata deleted successfully")
-    @with_current_user
-    @with_current_tenant_id
-    @rbac_permission_required(RBACCheck(RBACPermission.DATASET_EDIT, DatasetId()))
-    @with_session
-    def delete(
-        self,
-        session: Session,
-        current_tenant_id: str,
-        current_user: Account,
-        dataset_id: UUID,
-        metadata_id: UUID,
-    ):
-        dataset_id_str = str(dataset_id)
-        metadata_id_str = str(metadata_id)
-        dataset = DatasetService.get_dataset_for_tenant(dataset_id_str, current_tenant_id, session=session)
-        if dataset is None:
-            raise NotFound("Dataset not found.")
-        DatasetService.check_dataset_permission(dataset, current_user, session)
-
-        MetadataService.delete_metadata(dataset, metadata_id_str, session)
-        # Frontend callers only await success and invalidate metadata caches; no response body is consumed.
+    @console_account_admission(
+        rbac_checks=(RBACCheck(RBACPermission.DATASET_EDIT, DatasetId()),),
+    )
+    @metadata_errors
+    def delete(self, request_context: RequestContext, dataset_id: UUID, metadata_id: UUID):
+        service = application_services().knowledge.metadata
+        ref = service.require_dataset(request_context, str(dataset_id))
+        service.delete_metadata(ref, str(metadata_id))
         return "", 204
 
 
 @console_ns.route("/datasets/metadata/built-in")
 class DatasetMetadataBuiltInFieldApi(Resource):
-    @setup_required
-    @login_required
-    @account_initialization_required
-    @enterprise_license_required
     @console_ns.response(
         200,
         "Built-in fields retrieved successfully",
         console_ns.models[DatasetMetadataBuiltInFieldsResponse.__name__],
     )
-    def get(self):
-        built_in_fields = MetadataService.get_built_in_fields()
-        return dump_response(DatasetMetadataBuiltInFieldsResponse, {"fields": built_in_fields}), 200
+    @console_account_admission()
+    @metadata_errors
+    def get(self, request_context: RequestContext):
+        fields = application_services().knowledge.metadata.get_built_in_fields()
+        return dump_response(DatasetMetadataBuiltInFieldsResponse, {"fields": fields}), 200
 
 
 @console_ns.route("/datasets/<uuid:dataset_id>/metadata/built-in/<string:action>")
 class DatasetMetadataBuiltInFieldActionApi(Resource):
-    @setup_required
-    @login_required
-    @account_initialization_required
-    @enterprise_license_required
     @console_ns.response(204, "Action completed successfully")
-    @with_current_user
-    @rbac_permission_required(RBACCheck(RBACPermission.DATASET_EDIT, DatasetId()))
-    @with_session
-    def post(self, session: Session, current_user: Account, dataset_id: UUID, action: Literal["enable", "disable"]):
-        dataset_id_str = str(dataset_id)
-        dataset = DatasetService.get_dataset(dataset_id_str, session)
-        if dataset is None:
-            raise NotFound("Dataset not found.")
-        DatasetService.check_dataset_permission(dataset, current_user, session)
-
-        match action:
-            case "enable":
-                MetadataService.enable_built_in_field(dataset, session)
-            case "disable":
-                MetadataService.disable_built_in_field(dataset, session)
-        # Frontend callers only await success and invalidate metadata caches; no response body is consumed.
+    @console_account_admission(
+        rbac_checks=(RBACCheck(RBACPermission.DATASET_EDIT, DatasetId()),),
+    )
+    @metadata_errors
+    def post(self, request_context: RequestContext, dataset_id: UUID, action: str):
+        service = application_services().knowledge.metadata
+        ref = service.require_dataset(request_context, str(dataset_id))
+        if action == "enable":
+            service.enable_built_in_field(ref)
+        elif action == "disable":
+            service.disable_built_in_field(ref)
+        else:
+            raise ValueError("Invalid action.")
         return "", 204
 
 
 @console_ns.route("/datasets/<uuid:dataset_id>/documents/metadata")
 class DocumentMetadataEditApi(Resource):
-    @setup_required
-    @login_required
-    @account_initialization_required
-    @enterprise_license_required
     @console_ns.expect(console_ns.models[MetadataOperationData.__name__])
     @console_ns.response(
         204,
         "Documents metadata updated successfully",
     )
     @console_ns.response(404, "Dataset, document, or metadata not found")
-    @with_current_user
-    @with_current_tenant_id
-    @rbac_permission_required(RBACCheck(RBACPermission.DATASET_EDIT, DatasetId()))
-    @with_session
+    @console_account_admission(
+        rbac_checks=(RBACCheck(RBACPermission.DATASET_EDIT, DatasetId()),),
+    )
+    @metadata_errors
     @model_validate(MetadataOperationData)
-    def post(
-        self,
-        req_data: MetadataOperationData,
-        session: Session,
-        current_tenant_id: str,
-        current_user: Account,
-        dataset_id: UUID,
-    ):
-        dataset = DatasetService.get_dataset_for_tenant(str(dataset_id), current_tenant_id, session=session)
-        if dataset is None:
-            raise NotFound("Dataset not found.")
-        DatasetService.check_dataset_permission(dataset, current_user, session)
-
-        try:
-            MetadataService.update_documents_metadata(dataset, req_data, current_user, session=session)
-        except MetadataResourceNotFoundError as exc:
-            raise NotFound(str(exc)) from exc
-
-        # Frontend callers only await success and invalidate caches; no response body is consumed.
+    def post(self, req_data: MetadataOperationData, request_context: RequestContext, dataset_id: UUID):
+        service = application_services().knowledge.metadata
+        ref = service.require_dataset(request_context, str(dataset_id))
+        service.update_documents_metadata(ref, req_data, actor_id=request_context.account_id)
         return "", 204

api/controllers/console/datasets/rag_pipeline/datasource_auth.py

@@ -24,6 +24,7 @@ from core.entities.provider_entities import ProviderConfig
 from core.plugin.entities.plugin_daemon import PluginOAuthAuthorizationUrlResponse
 from core.plugin.impl.oauth import OAuthHandler
 from core.tools.entities.common_entities import I18nObject
+from extensions.ext_application_services import application_services
 from extensions.ext_database import db
 from fields.base import ResponseModel
 from graphon.model_runtime.errors.validate import CredentialsValidateFailedError
@@ -32,7 +33,6 @@ from libs.login import login_required
 from models import Account
 from models.enums import PermissionEnum
 from models.provider_ids import DatasourceProviderID
-from services.datasource_provider_service import DatasourceProviderService
 from services.plugin.oauth_service import OAuthProxyService
 
 
@@ -176,7 +176,7 @@ class DatasourcePluginOAuthAuthorizationUrl(Resource):
         datasource_provider_id = DatasourceProviderID(provider_id)
         provider_name = datasource_provider_id.provider_name
         plugin_id = datasource_provider_id.plugin_id
-        oauth_config = DatasourceProviderService().get_oauth_client(
+        oauth_config = application_services().data_sources.providers.get_oauth_client(
             tenant_id=tenant_id,
             datasource_provider_id=datasource_provider_id,
         )
@@ -246,7 +246,7 @@ class DatasourceOAuthCallback(Resource):
         tenant_id: str = context["tenant_id"]
         datasource_provider_id = DatasourceProviderID(provider_id)
         plugin_id = datasource_provider_id.plugin_id
-        datasource_provider_service = DatasourceProviderService()
+        datasource_provider_service = application_services().data_sources.providers
         oauth_client_params = datasource_provider_service.get_oauth_client(
             tenant_id=tenant_id,
             datasource_provider_id=datasource_provider_id,
@@ -315,7 +315,7 @@ class DatasourceAuth(Resource):
     @model_validate(DatasourceCredentialPayload)
     def post(self, req_data: DatasourceCredentialPayload, current_tenant_id: str, provider_id: str):
         datasource_provider_id = DatasourceProviderID(provider_id)
-        datasource_provider_service = DatasourceProviderService()
+        datasource_provider_service = application_services().data_sources.providers
 
         try:
             datasource_provider_service.add_datasource_api_key_provider(
@@ -342,14 +342,14 @@ class DatasourceAuth(Resource):
     @with_current_tenant_id
     def get(self, current_tenant_id: str, user: Account, provider_id: str):
         datasource_provider_id = DatasourceProviderID(provider_id)
-        datasource_provider_service = DatasourceProviderService()
+        datasource_provider_service = application_services().data_sources.providers
 
         datasources = datasource_provider_service.list_datasource_credentials(
             tenant_id=current_tenant_id,
             provider=datasource_provider_id.provider_name,
             plugin_id=datasource_provider_id.plugin_id,
             user=user,
-            session=db.session(),
+            credential_query=application_services().credential_queries,
         )
         return dump_response(DatasourceCredentialListResponse, {"result": datasources}), 200
 
@@ -370,7 +370,7 @@ class DatasourceAuthDeleteApi(Resource):
         plugin_id = datasource_provider_id.plugin_id
         provider_name = datasource_provider_id.provider_name
 
-        datasource_provider_service = DatasourceProviderService()
+        datasource_provider_service = application_services().data_sources.providers
         datasource_provider_service.remove_datasource_credentials(
             tenant_id=current_tenant_id,
             auth_id=req_data.credential_id,
@@ -397,7 +397,7 @@ class DatasourceAuthUpdateApi(Resource):
     def post(self, req_data: DatasourceCredentialUpdatePayload, current_tenant_id: str, provider_id: str):
         datasource_provider_id = DatasourceProviderID(provider_id)
 
-        datasource_provider_service = DatasourceProviderService()
+        datasource_provider_service = application_services().data_sources.providers
         datasource_provider_service.update_datasource_credentials(
             tenant_id=current_tenant_id,
             auth_id=req_data.credential_id,
@@ -422,9 +422,12 @@ class DatasourceAuthListApi(Resource):
     @with_current_user
     @with_current_tenant_id
     def get(self, current_tenant_id: str, user: Account):
-        datasource_provider_service = DatasourceProviderService()
+        datasource_provider_service = application_services().data_sources.providers
         datasources = datasource_provider_service.get_all_datasource_credentials(
-            tenant_id=current_tenant_id, session=db.session(), user=user
+            tenant_id=current_tenant_id,
+            session=db.session(),
+            user=user,
+            credential_query=application_services().credential_queries,
         )
         return dump_response(DatasourceProviderAuthListResponse, {"result": datasources}), 200
 
@@ -442,9 +445,12 @@ class DatasourceHardCodeAuthListApi(Resource):
     @with_current_user
     @with_current_tenant_id
     def get(self, current_tenant_id: str, user: Account):
-        datasource_provider_service = DatasourceProviderService()
+        datasource_provider_service = application_services().data_sources.providers
         datasources = datasource_provider_service.get_hard_code_datasource_credentials(
-            tenant_id=current_tenant_id, session=db.session(), user=user
+            tenant_id=current_tenant_id,
+            session=db.session(),
+            user=user,
+            credential_query=application_services().credential_queries,
         )
         return dump_response(DatasourceProviderAuthListResponse, {"result": datasources}), 200
 
@@ -464,7 +470,7 @@ class DatasourceAuthOauthCustomClient(Resource):
     @model_validate(DatasourceCustomClientPayload)
     def post(self, req_data: DatasourceCustomClientPayload, current_tenant_id: str, provider_id: str):
         datasource_provider_id = DatasourceProviderID(provider_id)
-        datasource_provider_service = DatasourceProviderService()
+        datasource_provider_service = application_services().data_sources.providers
         datasource_provider_service.setup_oauth_custom_client_params(
             tenant_id=current_tenant_id,
             datasource_provider_id=datasource_provider_id,
@@ -482,7 +488,7 @@ class DatasourceAuthOauthCustomClient(Resource):
     @with_current_tenant_id
     def delete(self, current_tenant_id: str, provider_id: str):
         datasource_provider_id = DatasourceProviderID(provider_id)
-        datasource_provider_service = DatasourceProviderService()
+        datasource_provider_service = application_services().data_sources.providers
         datasource_provider_service.remove_oauth_custom_client_params(
             tenant_id=current_tenant_id,
             datasource_provider_id=datasource_provider_id,
@@ -503,7 +509,7 @@ class DatasourceAuthDefaultApi(Resource):
     @model_validate(DatasourceDefaultPayload)
     def post(self, req_data: DatasourceDefaultPayload, current_tenant_id: str, provider_id: str):
         datasource_provider_id = DatasourceProviderID(provider_id)
-        datasource_provider_service = DatasourceProviderService()
+        datasource_provider_service = application_services().data_sources.providers
         datasource_provider_service.set_default_datasource_provider(
             tenant_id=current_tenant_id,
             datasource_provider_id=datasource_provider_id,
@@ -525,7 +531,7 @@ class DatasourceUpdateProviderNameApi(Resource):
     @model_validate(DatasourceUpdateNamePayload)
     def post(self, req_data: DatasourceUpdateNamePayload, current_tenant_id: str, provider_id: str):
         datasource_provider_id = DatasourceProviderID(provider_id)
-        datasource_provider_service = DatasourceProviderService()
+        datasource_provider_service = application_services().data_sources.providers
         datasource_provider_service.update_datasource_provider_name(
             tenant_id=current_tenant_id,
             datasource_provider_id=datasource_provider_id,

api/controllers/console/datasets/rag_pipeline/datasource_content_preview.py

@@ -9,6 +9,7 @@ from controllers.common.schema import register_schema_models
 from controllers.console import console_ns
 from controllers.console.datasets.wraps import get_rag_pipeline
 from controllers.console.wraps import account_initialization_required, model_validate, setup_required, with_current_user
+from extensions.ext_application_services import application_services
 from extensions.ext_database import db
 from libs.login import login_required
 from models import Account
@@ -51,5 +52,6 @@ class DataSourceContentPreviewApi(Resource):
             datasource_type=datasource_type,
             is_published=True,
             credential_id=req_data.credential_id,
+            datasource_providers=application_services().data_sources.providers,
         )
         return preview_content, 200

api/controllers/console/datasets/rag_pipeline/rag_pipeline.py

@@ -35,10 +35,11 @@ from libs.helper import dump_response
 from libs.login import login_required
 from models.account import Account
 from models.dataset import Pipeline
-from services.dataset_service import DatasetService
+from repositories.knowledge.dataset_read_repository import get_pipeline_dataset
 from services.entities.knowledge_entities.rag_pipeline_entities import IconInfo, PipelineTemplateInfoEntity
 from services.errors.account import NoPermissionError
 from services.errors.rag_pipeline import RagPipelineResourceNotFoundError
+from services.knowledge.dataset_service import DatasetService
 from services.rag_pipeline.rag_pipeline import RagPipelineService
 
 logger: logging.Logger = logging.getLogger(__name__)
@@ -232,7 +233,7 @@ class PublishCustomizedPipelineTemplateApi(Resource):
         pipeline: Pipeline,
     ) -> tuple[str, int]:
         session = db.session()
-        dataset = pipeline.retrieve_dataset(session=session)
+        dataset = get_pipeline_dataset(pipeline, session=session)
         if dataset is None:
             raise NotFound("Dataset not found")
 

api/controllers/console/datasets/rag_pipeline/rag_pipeline_datasets.py

@@ -17,14 +17,17 @@ from controllers.console.wraps import (
     with_current_user,
 )
 from extensions.ext_database import db
-from fields.dataset_fields import DatasetDetailResponse, dataset_detail_response_source
+from fields.dataset_fields import (
+    DatasetDetailResponse,
+)
 from libs.helper import dump_response
 from libs.login import login_required
 from models import Account
 from models.dataset import DatasetPermissionEnum
-from services.dataset_service import DatasetPermissionService, DatasetService
 from services.enterprise import rbac_service as enterprise_rbac_service
 from services.entities.knowledge_entities.rag_pipeline_entities import IconInfo, RagPipelineDatasetCreateEntity
+from services.knowledge.dataset_read_service import load_dataset_detail
+from services.knowledge.dataset_service import DatasetPermissionService, DatasetService
 from services.rag_pipeline.rag_pipeline_dsl_service import RagPipelineDslService
 
 
@@ -136,4 +139,4 @@ class CreateEmptyRagPipelineDatasetApi(Resource):
             ),
             session=session,
         )
-        return dump_response(DatasetDetailResponse, dataset_detail_response_source(dataset, session=session)), 201
+        return dump_response(DatasetDetailResponse, load_dataset_detail(dataset, session=session)), 201

api/controllers/console/datasets/rag_pipeline/rag_pipeline_workflow.py

@@ -46,6 +46,7 @@ from core.app.apps.base_app_queue_manager import AppQueueManager
 from core.app.apps.pipeline.pipeline_generator import PipelineGenerator
 from core.app.entities.app_invoke_entities import InvokeFrom
 from core.plugin.entities.plugin_daemon import PluginDatasourceProviderEntity
+from extensions.ext_application_services import application_services
 from extensions.ext_database import db
 from factories import variable_factory
 from fields.base import ResponseModel
@@ -66,10 +67,10 @@ from models.dataset import Pipeline
 from models.model import EndUser
 from models.workflow import Workflow
 from services.agent.retirement_service import WorkflowAgentRetirementService
-from services.dataset_service import DatasetService
 from services.errors.app import IsDraftWorkflowError, WorkflowHashNotEqualError, WorkflowNotFoundError
 from services.errors.llm import InvokeRateLimitError
 from services.errors.rag_pipeline import RagPipelineResourceNotFoundError
+from services.knowledge.dataset_service import DatasetService
 from services.rag_pipeline.pipeline_generate_service import PipelineGenerateService
 from services.rag_pipeline.rag_pipeline import RagPipelineService
 from services.rag_pipeline.rag_pipeline_manage_service import RagPipelineManageService
@@ -303,7 +304,13 @@ class RagPipelineDraftRunIterationNodeApi(Resource):
 
         try:
             response = PipelineGenerateService.generate_single_iteration(
-                pipeline=pipeline, user=current_user, node_id=node_id, args=args, session=db.session(), streaming=True
+                generator=application_services().knowledge.pipeline_generator,
+                pipeline=pipeline,
+                user=current_user,
+                node_id=node_id,
+                args=args,
+                session=db.session(),
+                streaming=True,
             )
 
             return helper.compact_generate_response(response)
@@ -338,7 +345,13 @@ class RagPipelineDraftRunLoopNodeApi(Resource):
 
         try:
             response = PipelineGenerateService.generate_single_loop(
-                pipeline=pipeline, user=current_user, node_id=node_id, args=args, session=db.session(), streaming=True
+                generator=application_services().knowledge.pipeline_generator,
+                pipeline=pipeline,
+                user=current_user,
+                node_id=node_id,
+                args=args,
+                session=db.session(),
+                streaming=True,
             )
 
             return helper.compact_generate_response(response)
@@ -374,6 +387,7 @@ class DraftRagPipelineRunApi(Resource):
 
         try:
             response = PipelineGenerateService.generate(
+                generator=application_services().knowledge.pipeline_generator,
                 session=session,
                 pipeline=pipeline,
                 user=current_user,
@@ -409,6 +423,7 @@ class PublishedRagPipelineRunApi(Resource):
 
         try:
             response = PipelineGenerateService.generate(
+                generator=application_services().knowledge.pipeline_generator,
                 session=session,
                 pipeline=pipeline,
                 user=current_user,
@@ -450,6 +465,7 @@ class RagPipelinePublishedDatasourceNodeRunApi(Resource):
                     datasource_type=req_data.datasource_type,
                     is_published=False,
                     credential_id=req_data.credential_id,
+                    datasource_providers=application_services().data_sources.providers,
                 )
             )
         )
@@ -483,6 +499,7 @@ class RagPipelineDraftDatasourceNodeRunApi(Resource):
                     datasource_type=req_data.datasource_type,
                     is_published=False,
                     credential_id=req_data.credential_id,
+                    datasource_providers=application_services().data_sources.providers,
                 )
             )
         )
@@ -1016,7 +1033,9 @@ class DatasourceListApi(Resource):
     @account_initialization_required
     @with_current_tenant_id
     def get(self, current_tenant_id: str):
-        providers = RagPipelineManageService.list_rag_pipeline_datasources(current_tenant_id)
+        providers = RagPipelineManageService.list_rag_pipeline_datasources(
+            current_tenant_id, datasource_providers=application_services().data_sources.providers
+        )
         return RagPipelineDatasourceListResponse.model_validate(providers, from_attributes=True).model_dump(mode="json")
 
 

api/controllers/console/datasets/website.py

@@ -7,8 +7,9 @@ from controllers.common.schema import query_params_from_model, register_response
 from controllers.console import console_ns
 from controllers.console.datasets.error import WebsiteCrawlError
 from controllers.console.wraps import account_initialization_required, model_validate, setup_required
+from extensions.ext_application_services import application_services
 from libs.login import login_required
-from services.website_service import WebsiteCrawlApiRequest, WebsiteCrawlStatusApiRequest, WebsiteService
+from services.data_source.website_service import WebsiteCrawlApiRequest, WebsiteCrawlStatusApiRequest, WebsiteService
 
 
 class WebsiteCrawlPayload(BaseModel):
@@ -49,7 +50,7 @@ class WebsiteCrawlApi(Resource):
 
         # Crawl URL using typed request
         try:
-            result = WebsiteService.crawl_url(api_request)
+            result = WebsiteService(providers=application_services().data_sources.providers).crawl_url(api_request)
         except Exception as e:
             raise WebsiteCrawlError(str(e))
         return result, 200
@@ -77,7 +78,9 @@ class WebsiteCrawlStatusApi(Resource):
 
         # Get crawl status using typed request
         try:
-            result = WebsiteService.get_crawl_status_typed(api_request)
+            result = WebsiteService(providers=application_services().data_sources.providers).get_crawl_status_typed(
+                api_request
+            )
         except Exception as e:
             raise WebsiteCrawlError(str(e))
         return result, 200

api/controllers/console/explore/recommended_app.py

@@ -1,9 +1,10 @@
 from http import HTTPStatus
-from typing import Any
+from typing import Any, Literal
 from uuid import UUID
 
+from flask import make_response, send_file
 from flask_restx import Resource
-from pydantic import BaseModel, Field, computed_field, field_validator
+from pydantic import BaseModel, ConfigDict, Field, computed_field, field_validator
 
 from controllers.common.schema import query_params_from_model, register_response_schema_models, register_schema_models
 from controllers.console import console_ns
@@ -11,7 +12,9 @@ from controllers.console.explore.error import RecommendedAppNotFoundError
 from controllers.console.flask_admission import console_account_admission
 from controllers.console.wraps import model_validate
 from extensions.ext_application_services import application_services
+from fields.app_export_fields import AppExportResponse
 from fields.base import ResponseModel
+from libs.flask_restx_compat import BINARY_RESPONSE_MEDIA_TYPES_VENDOR_KEY
 from libs.helper import build_icon_url, dump_response
 from machinery.context import RequestContext
 from services.recommended_app_query_service import RecommendedAppNotFoundError as RecommendedAppQueryNotFoundError
@@ -21,6 +24,13 @@ class RecommendedAppsQuery(BaseModel):
     language: str = Field(default="en-US", description="Language code for recommended app localization")
 
 
+class RecommendedAgentExportQuery(BaseModel):
+    model_config = ConfigDict(extra="forbid")
+
+    version_id: UUID = Field(description="Current published template snapshot ID")
+    format: Literal["yaml", "ifpkg"] = Field(default="ifpkg", description="Export format; defaults to ifpkg")
+
+
 class RecommendedAppInfoResponse(ResponseModel):
     id: str
     name: str | None = None
@@ -78,11 +88,14 @@ class RecommendedAppDetailResponse(ResponseModel):
     mode: str
     export_data: str
     can_trial: bool
+    package_url: str | None = Field(default=None, description="Download URL for a New Agent .ifpkg template")
+    version_id: str | None = Field(default=None, description="Published version for direct local template creation")
 
 
 register_schema_models(
     console_ns,
     RecommendedAppsQuery,
+    RecommendedAgentExportQuery,
 )
 register_response_schema_models(
     console_ns,
@@ -91,6 +104,7 @@ register_response_schema_models(
     RecommendedAppListResponse,
     LearnDifyAppListResponse,
     RecommendedAppDetailResponse,
+    AppExportResponse,
 )
 
 
@@ -135,3 +149,37 @@ class RecommendedAppApi(Resource):
         except RecommendedAppQueryNotFoundError:
             raise RecommendedAppNotFoundError() from None
         return dump_response(RecommendedAppDetailResponse, result)
+
+
+@console_ns.route("/trial-apps/<uuid:app_id>/export")
+class RecommendedAgentExportApi(Resource):
+    @console_ns.doc(params=query_params_from_model(RecommendedAgentExportQuery))
+    @console_ns.doc(
+        produces=["application/json", "application/zip"],
+        vendor={BINARY_RESPONSE_MEDIA_TYPES_VENDOR_KEY: ["application/zip"]},
+    )
+    @console_ns.response(200, "Published Agent template export", console_ns.models[AppExportResponse.__name__])
+    @console_ns.response(404, "Published template unavailable")
+    @model_validate(RecommendedAgentExportQuery)
+    def get(self, query: RecommendedAgentExportQuery, app_id: UUID):
+        # Both formats require public catalog membership and the current published version.
+        try:
+            exported = application_services().recommended_app_packages.export(
+                app_id=str(app_id), version_id=query.version_id, format=query.format
+            )
+        except RecommendedAppQueryNotFoundError:
+            raise RecommendedAppNotFoundError() from None
+        if isinstance(exported, str):
+            response = make_response(dump_response(AppExportResponse, {"data": exported}))
+            response.headers["Cache-Control"] = "no-store"
+            return response
+        try:
+            response = send_file(
+                exported.archive, mimetype="application/zip", as_attachment=True, download_name=exported.filename
+            )
+        except Exception:
+            exported.close()
+            raise
+        response.headers["Cache-Control"] = "no-store"
+        response.call_on_close(exported.close)
+        return response

api/controllers/console/explore/trial.py

@@ -57,12 +57,14 @@ from controllers.console.explore.error import (
     NotCompletionAppError,
     NotWorkflowAppError,
 )
-from controllers.console.explore.trial_app_admission import get_trial_app
+from controllers.console.explore.trial_app_admission import get_trial_app, get_trial_app_for_stop
 from controllers.console.files import FILE_UPLOAD_PARAMS, upload_file_from_request_context
 from controllers.console.flask_admission import console_account_admission
 from controllers.console.remote_files import RemoteFileUploadPayload, upload_remote_file
 from controllers.console.wraps import cloud_edition_billing_resource_check, model_validate
 from controllers.web.error import InvokeRateLimitError as InvokeRateLimitHttpError
+from core.app.apps.agent_app.errors import AgentAppGeneratorError, AgentAppNotPublishedError
+from core.app.entities.app_invoke_entities import InvokeFrom
 from core.errors.error import (
     AppInvokeQuotaExceededError,
     ModelCurrentlyNotSupportError,
@@ -72,6 +74,7 @@ from core.errors.error import (
 from core.helper import encrypter
 from core.workflow.llm_environment_variable import LLMEnvironmentVariable, dump_environment_variable
 from extensions.ext_application_services import application_services
+from fields.agent_fields import AgentAppComposerResponse
 from fields.base import ResponseModel
 from fields.conversation_variable_fields import WorkflowConversationVariableResponse
 from fields.file_fields import FileResponse, FileWithSignedUrl
@@ -84,6 +87,7 @@ from libs.stream import close_stream
 from libs.url_utils import normalize_api_base_url
 from machinery.context import RequestContext
 from models.enums import CreatorUserRole
+from models.model import AppMode
 from services.account_errors import AccountNotFoundError
 from services.app_definition_query_service import AppDefinitionUnavailableError
 from services.app_preview_query_service import (
@@ -153,7 +157,7 @@ class TrialDatasetListQuery(BaseModel):
     ids: list[str] = Field(default_factory=list, description="Dataset IDs")
 
 
-type TrialAppMode = Literal["chat", "agent-chat", "advanced-chat", "workflow", "completion"]
+type TrialAppMode = Literal["chat", "agent-chat", "agent", "advanced-chat", "workflow", "completion"]
 type TrialIconType = Literal["emoji", "image", "link"]
 type JsonObject = dict[str, Any]
 
@@ -448,6 +452,7 @@ register_response_schema_models(
     SiteResponse,
     SuggestedQuestionsResponse,
     TrialAppDetailResponse,
+    AgentAppComposerResponse,
     TrialDatasetListResponse,
     TrialWorkflowResponse,
 )
@@ -603,6 +608,8 @@ class TrialChatApi(Resource):
         except services.errors.app_model_config.AppModelConfigBrokenError:
             logger.exception("App model config broken.")
             raise AppUnavailableError()
+        except (AgentAppGeneratorError, AgentAppNotPublishedError):
+            raise AppUnavailableError() from None
         except ProviderTokenNotInitError as ex:
             raise ProviderNotInitializeError(ex.description)
         except QuotaExceededError:
@@ -620,6 +627,27 @@ class TrialChatApi(Resource):
             raise InternalServerError()
 
 
+@console_ns.route(
+    "/trial-apps/<uuid:app_id>/chat-messages/<string:task_id>/stop",
+    endpoint="trial_app_stop_chat_completion",
+)
+class TrialChatTaskStopApi(Resource):
+    @console_ns.response(200, "Success", console_ns.models[SimpleResultResponse.__name__])
+    @console_account_admission()
+    @get_trial_app_for_stop
+    def post(self, request_context: RequestContext, trial_app: TrialAppRef, task_id: str):
+        if trial_app.app_mode not in {"chat", "agent-chat", "agent", "advanced-chat"}:
+            raise NotChatAppError()
+
+        application_services().app_tasks.stop_task(
+            task_id=task_id,
+            invoke_from=InvokeFrom.EXPLORE,
+            user_id=request_context.account_id,
+            app_mode=AppMode.value_of(trial_app.app_mode),
+        )
+        return SimpleResultResponse(result="success").model_dump(mode="json"), 200
+
+
 @console_ns.route(
     "/trial-apps/<uuid:app_id>/messages/<uuid:message_id>/suggested-questions",
     endpoint="trial_app_suggested_question",
@@ -629,7 +657,7 @@ class TrialMessageSuggestedQuestionApi(Resource):
     @console_account_admission()
     @get_trial_app
     def get(self, request_context: RequestContext, trial_app: TrialAppRef, message_id: UUID) -> dict[str, object]:
-        if trial_app.app_mode not in {"chat", "agent-chat", "advanced-chat"}:
+        if trial_app.app_mode not in {"chat", "agent-chat", "agent", "advanced-chat"}:
             raise NotChatAppError()
 
         try:
@@ -837,7 +865,7 @@ class TrialAppParameterApi(Resource):
         """Retrieve app parameters."""
 
         try:
-            parameters = application_services().app_definitions.get_parameters(app.app_id)
+            parameters = application_services().app_definitions.get_public_parameters(app.app_id)
         except AppDefinitionUnavailableError:
             raise AppUnavailableError() from None
 
@@ -875,6 +903,18 @@ class AppApi(Resource):
         return dump_response(TrialAppDetailResponse, source)
 
 
+@console_ns.route("/trial-apps/<uuid:app_id>/agent-composer")
+class TrialAgentComposerApi(Resource):
+    @console_ns.response(200, "Published Agent configuration", console_ns.models[AgentAppComposerResponse.__name__])
+    @get_preview_app
+    def get(self, app: AppPreviewRef):
+        try:
+            preview = application_services().app_previews.get_agent_composer(app=app)
+        except AppDefinitionUnavailableError:
+            raise AppUnavailableError() from None
+        return dump_response(AgentAppComposerResponse, preview)
+
+
 @console_ns.route(
     "/trial-apps/<uuid:app_id>/workflows",
     endpoint="trial_app_workflow",

api/controllers/console/explore/trial_app_admission.py

@@ -28,8 +28,10 @@ def trial_feature_enable[**P, R](view: Callable[P, R]) -> Callable[P, R]:
 
 def get_trial_app[T, **P, R](
     view: Callable[Concatenate[T, RequestContext, TrialAppRef, P], R],
+    *,
+    check_usage: bool = True,
 ) -> Callable[Concatenate[T, RequestContext, P], R]:
-    """Check trial availability and inject an admitted app after Console admission."""
+    """Admit the trial app, optionally skipping generation quota for task cancellation."""
 
     @wraps(view)
     def decorated(self: T, request_context: RequestContext, /, *args: P.args, **kwargs: P.kwargs) -> R:
@@ -38,7 +40,7 @@ def get_trial_app[T, **P, R](
             raise RuntimeError("The trial app admission route must provide app_id")
         try:
             trial_app = application_services().trial_apps.access.get_access(
-                app_id=str(app_id), account_id=request_context.account_id
+                app_id=str(app_id), account_id=request_context.account_id, check_usage=check_usage
             )
         except TrialAppUnavailableError as error:
             raise TrialAppNotAllowed() from error
@@ -47,3 +49,10 @@ def get_trial_app[T, **P, R](
         return view(self, request_context, trial_app, *args, **kwargs)
 
     return trial_feature_enable(decorated)
+
+
+def get_trial_app_for_stop[T, **P, R](
+    view: Callable[Concatenate[T, RequestContext, TrialAppRef, P], R],
+) -> Callable[Concatenate[T, RequestContext, P], R]:
+    """Keep trial access checks without requiring remaining generation quota."""
+    return get_trial_app(view, check_usage=False)

api/controllers/console/flask_admission.py

@@ -71,7 +71,7 @@ def console_account_admission[T, **P, R](
     ) -> Callable[Concatenate[T, P], R | Response]:
         admitted_view = cloud_edition_billing_resource_check(billing_resource)(view) if billing_resource else view
 
-        @wraps(view, updated=())
+        @wraps(view)
         def inject_request_context(self: T, /, *args: P.args, **kwargs: P.kwargs) -> R:
             account_with_tenant = current_account_with_tenant()
             account = account_with_tenant.account

api/controllers/console/workspace/model_providers.py

@@ -33,7 +33,7 @@ from services.entities.model_provider_entities import (
     ModelProviderSummaryResponse,
     ProviderResponse,
 )
-from services.model_provider_service import ModelProviderService
+from services.model_provider.service import ModelProviderService
 from services.workspace_service import WorkspaceService
 
 

api/controllers/console/workspace/models.py

@@ -25,6 +25,7 @@ from controllers.console.wraps import (
 )
 from core.entities.model_entities import DefaultModelSetting
 from core.entities.provider_entities import CredentialConfiguration
+from extensions.ext_application_services import application_services
 from extensions.ext_database import db
 from fields.base import ResponseModel
 from graphon.model_runtime.entities.model_entities import ModelType, ParameterRule
@@ -38,7 +39,7 @@ from services.entities.model_provider_entities import (
     ProviderWithModelsResponse,
 )
 from services.model_load_balancing_service import ModelLoadBalancingService
-from services.model_provider_service import ModelProviderService
+from services.model_provider.service import ModelProviderService
 
 logger = logging.getLogger(__name__)
 
@@ -376,6 +377,7 @@ class ModelProviderModelCredentialApi(Resource):
                 tenant_id=tenant_id,
                 provider=provider,
                 user=user,
+                credential_query=application_services().credential_queries,
             )
         else:
             available_credentials = model_provider_service.get_provider_model_available_credentials(

api/controllers/console/workspace/tool_providers.py

@@ -62,6 +62,7 @@ from core.tools.entities.tool_entities import (
     WorkflowToolParameterConfiguration,
 )
 from enums import DeploymentEdition
+from extensions.ext_application_services import application_services
 from extensions.ext_database import db
 from fields.base import ResponseModel
 from libs.helper import alphanumeric, dump_response, uuid_value
@@ -664,9 +665,9 @@ class ToolBuiltinProviderGetCredentialsApi(Resource):
             BuiltinToolManageService.get_builtin_tool_provider_credentials(
                 tenant_id=tenant_id,
                 provider_name=provider,
-                session=db.session(),
                 user=user,
                 include_credential_ids=query.include_credential_ids or None,
+                credential_query=application_services().credential_queries,
             ),
         )
 
@@ -1363,9 +1364,9 @@ class ToolBuiltinProviderGetCredentialInfoApi(Resource):
             BuiltinToolManageService.get_builtin_tool_provider_credential_info(
                 tenant_id=tenant_id,
                 provider=provider,
-                session=db.session(),
                 user=user,
                 include_credential_ids=query.include_credential_ids or None,
+                credential_query=application_services().credential_queries,
             ),
         )
 

api/controllers/console/workspace/trigger_providers.py

@@ -22,6 +22,7 @@ from core.trigger.entities.api_entities import (
 )
 from core.trigger.entities.entities import RequestLog, SubscriptionBuilderUpdater
 from core.trigger.trigger_manager import TriggerManager
+from extensions.ext_application_services import application_services
 from extensions.ext_database import db
 from fields.base import ResponseModel
 from libs.helper import dump_response
@@ -213,6 +214,7 @@ class TriggerSubscriptionListApi(Resource):
                     tenant_id=tenant_id,
                     provider_id=TriggerProviderID(provider),
                     user=user,
+                    credential_query=application_services().credential_queries,
                 ),
             )
         except ValueError as e:

api/controllers/mcp/mcp.py

@@ -4,7 +4,7 @@ from flask import Response, request
 from flask_restx import Resource
 from pydantic import BaseModel, Field, RootModel, ValidationError
 from sqlalchemy import select
-from sqlalchemy.orm import Session, sessionmaker
+from sqlalchemy.orm import Session
 
 from controllers.common.schema import register_response_schema_models, register_schema_model
 from controllers.mcp import mcp_ns
@@ -13,8 +13,9 @@ from core.mcp.server.streamable_http import handle_mcp_request, negotiate_protoc
 from extensions.ext_database import db
 from graphon.variables.input_entities import VariableEntity, VariableEntityType
 from libs import helper
-from models.enums import AppMCPServerStatus, EndUserType
+from models.enums import EndUserType
 from models.model import App, AppMCPServer, AppMode, EndUser
+from services.app.mcp_server_service import AppMCPServerStatus
 
 
 class MCPRequestError(Exception):
@@ -82,7 +83,7 @@ class MCPAppApi(Resource):
             else:
                 return self._protocol_version_error_response(request_id, header_value)
 
-        with sessionmaker(db.engine, expire_on_commit=False).begin() as session:
+        with Session(db.engine, expire_on_commit=False) as session:
             # Get MCP server and app
             mcp_server, app = self._get_mcp_server_and_app(server_code, session)
             self._validate_server_status(mcp_server)
@@ -91,9 +92,11 @@ class MCPAppApi(Resource):
             user_input_form = self._get_user_input_form(app, session)
 
             # Handle notification vs request differently
-            return self._process_mcp_message(
+            response = self._process_mcp_message(
                 mcp_request, request_id, app, mcp_server, user_input_form, session, protocol_version
             )
+            session.commit()
+            return response
 
     def _protocol_version_error_response(
         self, request_id: Union[int, str] | None, header_value: str | None
@@ -128,7 +131,7 @@ class MCPAppApi(Resource):
 
     def _validate_server_status(self, mcp_server: AppMCPServer):
         """Validate MCP server status"""
-        if mcp_server.status != AppMCPServerStatus.ACTIVE:
+        if mcp_server.status != AppMCPServerStatus.ACTIVE.value:
             raise MCPRequestError(mcp_types.INVALID_REQUEST, "Server is not active")
 
     def _process_mcp_message(
@@ -192,8 +195,7 @@ class MCPAppApi(Resource):
             app_model_config = app.app_model_config_with_session(session=session)
             if not app_model_config:
                 raise MCPRequestError(mcp_types.INVALID_REQUEST, "App is unavailable")
-            features_dict = app_model_config.to_dict()
-            raw_user_input_form = features_dict.get("user_input_form", [])
+            raw_user_input_form = app_model_config.user_input_form_list
 
         # Convert to VariableEntity objects
         try:
@@ -237,16 +239,15 @@ class MCPAppApi(Resource):
             except ValidationError as e:
                 raise MCPRequestError(mcp_types.INVALID_PARAMS, f"Invalid MCP request: {str(e)}")
 
-    def _retrieve_end_user(self, tenant_id: str, mcp_server_id: str) -> EndUser | None:
-        """Get end user - manages its own database session"""
-        with sessionmaker(db.engine, expire_on_commit=False).begin() as session:
-            return session.scalar(
-                select(EndUser)
-                .where(
-                    EndUser.tenant_id == tenant_id, EndUser.session_id == mcp_server_id, EndUser.type == EndUserType.MCP
-                )
-                .limit(1)
-            )
+    def _retrieve_end_user(self, tenant_id: str, mcp_server_id: str, session: Session) -> EndUser | None:
+        """Get end user in the caller session"""
+        return session.scalar(
+            select(EndUser)
+            .where(EndUser.tenant_id == tenant_id)
+            .where(EndUser.session_id == mcp_server_id)
+            .where(EndUser.type == EndUserType.MCP)
+            .limit(1)
+        )
 
     def _create_end_user(
         self, client_name: str, tenant_id: str, app_id: str, mcp_server_id: str, session: Session
@@ -275,7 +276,7 @@ class MCPAppApi(Resource):
         protocol_version: str,
     ) -> mcp_types.JSONRPCResponse | mcp_types.JSONRPCError | None:
         """Handle MCP request and return response"""
-        end_user = self._retrieve_end_user(mcp_server.tenant_id, mcp_server.id)
+        end_user = self._retrieve_end_user(mcp_server.tenant_id, mcp_server.id, session)
 
         if not end_user and isinstance(mcp_request.root, mcp_types.InitializeRequest):
             client_info = mcp_request.root.params.clientInfo

api/controllers/openapi/__init__.py

@@ -61,6 +61,9 @@ from controllers.openapi._models import (
     DeviceMutateResponse,
     DevicePollRequest,
     DeviceTokenResponse,
+    DraftWorkflowRunPayload,
+    EnvVariableListResponse,
+    EnvVariableSetPayload,
     FileUploadPayload,
     FormSubmitResponse,
     HealthResponse,
@@ -77,13 +80,20 @@ from controllers.openapi._models import (
     OpenApiFormSubmitPayload,
     PermittedExternalAppsListQuery,
     PermittedExternalAppsListResponse,
+    PublishPayload,
+    PublishResponse,
+    RestoreResponse,
     RevokeResponse,
+    RunListQuery,
+    RunListResponse,
     ServerVersionResponse,
     SessionListQuery,
     SessionListResponse,
     SessionRow,
     TaskStopResponse,
     UsageInfo,
+    VersionListQuery,
+    VersionListResponse,
     WorkflowRunData,
     WorkflowRunPayload,
     WorkspaceDetailResponse,
@@ -93,6 +103,7 @@ from controllers.openapi._models import (
     WorkspaceSummaryResponse,
 )
 from fields.file_fields import FileResponse
+from fields.workflow_run_fields import WorkflowRunDetailResponse, WorkflowRunNodeExecutionListResponse
 from services.entities.dsl_entities import CheckDependenciesResult, Import
 
 register_schema_models(
@@ -108,13 +119,18 @@ register_schema_models(
     DevicePollRequest,
     DeviceLookupQuery,
     DeviceMutateRequest,
+    DraftWorkflowRunPayload,
     FileUploadPayload,
     MemberInvitePayload,
     MemberListQuery,
     MemberRoleUpdatePayload,
     OpenApiFormSubmitPayload,
+    EnvVariableSetPayload,
     PermittedExternalAppsListQuery,
+    PublishPayload,
+    RunListQuery,
     SessionListQuery,
+    VersionListQuery,
     WorkflowRunPayload,
     WorkspaceListQuery,
 )
@@ -160,6 +176,13 @@ register_response_schema_models(
     FileResponse,
     ServerVersionResponse,
     HealthResponse,
+    RunListResponse,
+    PublishResponse,
+    VersionListResponse,
+    RestoreResponse,
+    EnvVariableListResponse,
+    WorkflowRunDetailResponse,
+    WorkflowRunNodeExecutionListResponse,
 )
 # Standalone definition for contract codegen; ErrorBody.code stays an open
 # string on the wire so old clients keep parsing future codes.
@@ -169,6 +192,7 @@ from . import (
     account,
     app_dsl,
     app_run,
+    app_workflow,
     apps,
     apps_permitted_external,
     files,
@@ -186,6 +210,7 @@ __all__ = [
     "account",
     "app_dsl",
     "app_run",
+    "app_workflow",
     "apps",
     "apps_permitted_external",
     "files",

api/controllers/openapi/_errors.py

@@ -68,6 +68,12 @@ class OpenApiErrorCode(StrEnum):
     RECIPIENT_SURFACE_MISMATCH = "recipient_surface_mismatch"
     CATALOG_STALE = "catalog_stale"
     TRIGGER_WORKFLOW_SERVICE_MODE_UNAVAILABLE = "trigger_workflow_service_mode_unavailable"
+    RUN_NOT_FOUND = "run_not_found"
+    VERSION_NOT_FOUND = "version_not_found"
+    VERSION_NOT_RESTORABLE = "version_not_restorable"
+    DRAFT_NOT_FOUND = "draft_not_found"
+    SECRET_MASK_UNKNOWN_ID = "secret_mask_unknown_id"
+    SECRET_MASK_NOT_SECRET = "secret_mask_not_secret"
 
 
 class ErrorDetail(BaseModel):
@@ -270,3 +276,39 @@ class InvalidFilePart(OpenApiError):  # noqa: N818
     code = 422
     error_code = OpenApiErrorCode.INVALID_PARAM
     description = "A request part could not be used"
+
+
+class RunNotFound(OpenApiError):  # noqa: N818
+    code = 404
+    error_code = OpenApiErrorCode.RUN_NOT_FOUND
+    description = "No run with this id exists for this app."
+
+
+class VersionNotFound(OpenApiError):  # noqa: N818
+    code = 404
+    error_code = OpenApiErrorCode.VERSION_NOT_FOUND
+    description = "No version with this id exists for this app."
+
+
+class VersionNotRestorable(OpenApiError):  # noqa: N818
+    code = 400
+    error_code = OpenApiErrorCode.VERSION_NOT_RESTORABLE
+    description = "Only a published version can be restored."
+
+
+class DraftNotFound(OpenApiError):  # noqa: N818
+    code = 404
+    error_code = OpenApiErrorCode.DRAFT_NOT_FOUND
+    description = "This app has no draft."
+
+
+class SecretMaskUnknownId(OpenApiError):  # noqa: N818
+    code = 400
+    error_code = OpenApiErrorCode.SECRET_MASK_UNKNOWN_ID
+    description = "The value is the masked secret placeholder, but env_id is not an existing secret."
+
+
+class SecretMaskNotSecret(OpenApiError):  # noqa: N818
+    code = 400
+    error_code = OpenApiErrorCode.SECRET_MASK_NOT_SECRET
+    description = "The value is the masked secret placeholder, which only a secret value_type can keep."

api/controllers/openapi/_files.py

@@ -67,7 +67,7 @@ def _mapping(part: FileStorage, caller: Any) -> FileMapping:
     uploaded = upload(part, caller)
     return {
         "transfer_method": FileTransferMethod.LOCAL_FILE,
-        "upload_file_id": str(uploaded.id),
+        "upload_file_id": uploaded.id,
         "type": standardize_file_type(extension="." + uploaded.extension, mime_type=uploaded.mime_type),
     }
 

api/controllers/openapi/_models.py

@@ -12,7 +12,9 @@ from constants.oauth_bearer import SubjectType
 from controllers.common.human_input import HumanInputFormSubmitPayload
 from controllers.openapi._upload import UploadPart, UploadParts
 from enums import DeploymentEdition
-from libs.helper import EmailStr, UUIDStr, UUIDStrOrEmpty, uuid_value
+from fields.workflow_run_fields import WorkflowRunPaginationResponse
+from graphon.variables import SegmentType
+from libs.helper import EmailStr, UUIDStr, UUIDStrOrEmpty, to_timestamp, uuid_value
 from models.model import AppMode
 from services.app_dsl_service import Import
 
@@ -345,7 +347,7 @@ class RunPayloadBase(BaseModel):
     inputs: dict[str, Any] = Field(
         description=(
             "Variables declared by the app. The exact shape is per app: read `input_schema` from "
-            "console_app.describe. A file variable takes a Dify file mapping (remote url or upload id) here, "
+            "describe.console_app. A file variable takes a Dify file mapping (remote url or upload id) here, "
             "or a local file in `files`, not both."
         )
     )
@@ -366,6 +368,10 @@ class WorkflowRunPayload(RunPayloadBase, _WorkflowVersionFields):
     model_config = ConfigDict(extra="forbid")
 
 
+class DraftWorkflowRunPayload(RunPayloadBase):
+    model_config = ConfigDict(extra="forbid")
+
+
 class ChatRunPayload(RunPayloadBase, _ConversationFields):
     model_config = ConfigDict(extra="forbid")
 
@@ -498,6 +504,11 @@ class AppDslImportPayload(BaseModel):
     icon: str | None = Field(None)
     icon_background: str | None = Field(None)
     app_id: str | None = Field(None, description="Existing app ID to overwrite (workflow/advanced-chat apps only)")
+    draft_hash: str | None = Field(
+        None,
+        description="draft_hash from the export or restore this import is based on. The import fails if the "
+        "draft's graph, features, environment variables or conversation variables changed since. Requires app_id",
+    )
 
     @model_validator(mode="after")
     def _validate_source_by_mode(self) -> AppDslImportPayload:
@@ -521,6 +532,11 @@ class AppDslExportResponse(BaseModel):
     """Export DSL response."""
 
     data: str = Field(..., description="DSL YAML string")
+    draft_hash: str | None = Field(
+        None,
+        description="Hash of the draft's graph, features, environment variables and conversation variables; "
+        "pass it to the import to refuse overwriting newer edits",
+    )
 
 
 class AppDslImportResponse(Import, Hinted):
@@ -550,3 +566,99 @@ class HumanInputFormDefinitionResponse(BaseModel):
     resolved_default_values: dict[str, str]
     user_actions: list[dict[str, Any]] = Field(default_factory=list)
     expiration_time: int | None = None
+
+
+class RunListQuery(BaseModel):
+    last_id: UUIDStr | None = Field(None, description="Cursor: id of the last run on the previous page")
+    limit: int = Field(20, ge=1, le=MAX_PAGE_LIMIT)
+    status: Literal["running", "succeeded", "failed", "stopped", "partial-succeeded"] | None = None
+    triggered_from: Literal["debugging", "app-run"] | None = Field(
+        None, description="debugging: draft test runs; app-run: real use. Omitted: debugging, as in the console"
+    )
+
+
+class RunListResponse(WorkflowRunPaginationResponse, Hinted):
+    """Cursor page of runs; `hints` carries the next page."""
+
+
+class PublishPayload(BaseModel):
+    model_config = ConfigDict(extra="forbid")
+
+    marked_name: str = Field("", max_length=20, description="Version name")
+    marked_comment: str = Field("", max_length=100, description="Version note")
+
+
+class PublishResponse(BaseModel):
+    version_id: str
+    created_at: int
+    warning: str | None = Field(None, description="Variable references that may read a skipped branch")
+
+
+class VersionListQuery(PageQuery):
+    named_only: bool = Field(False, description="Only versions that have a name")
+
+
+class VersionRow(BaseModel):
+    model_config = ConfigDict(from_attributes=True)
+
+    id: str
+    marked_name: str
+    marked_comment: str
+    created_by: str | None = None
+    created_at: int
+
+    @field_validator("created_at", mode="before")
+    @classmethod
+    def _timestamp(cls, value: datetime | int) -> int | None:
+        return to_timestamp(value)
+
+
+class VersionListResponse(Hinted):
+    """Page of published versions, newest first; there is no total, `hints` carries the next page."""
+
+    page: int
+    limit: int
+    has_more: bool
+    data: list[VersionRow]
+
+
+class EnvVariableRow(BaseModel):
+    id: str = Field(description="What set and delete take to address this variable")
+    name: str = Field(description="What nodes use to refer to this variable")
+    description: str = ""
+    value_type: str
+    value: Any = Field(description="The value; a secret with a value is masked, an empty one reads as empty")
+
+
+class EnvVariableListResponse(BaseModel):
+    data: list[EnvVariableRow]
+
+
+class EnvVariableValueType(StrEnum):
+    """Value types the draft environment-variable ``set`` op accepts.
+
+    A curated subset of ``SegmentType``: what the console's environment-variable editor
+    allows (``ENVIRONMENT_VARIABLE_SUPPORTED_TYPES`` in controllers/console/app/workflow.py).
+    Members reference ``SegmentType.*.value`` so the subset relationship is type-checked.
+    """
+
+    STRING = SegmentType.STRING.value
+    NUMBER = SegmentType.NUMBER.value
+    SECRET = SegmentType.SECRET.value
+
+
+class EnvVariableSetPayload(BaseModel):
+    model_config = ConfigDict(extra="forbid")
+
+    name: str = Field(description="Variable name")
+    value_type: EnvVariableValueType = Field(description="string, number or secret")
+    value: Any = Field(description="The value; sending the masked value of an existing secret keeps the stored one")
+    description: str = Field("", description="What the variable is for")
+
+
+class RestoreResponse(BaseModel):
+    result: Literal["success"]
+    draft_hash: str = Field(
+        description="Hash of the restored draft's graph, features, environment variables and conversation "
+        "variables; pass it to a DSL import as draft_hash"
+    )

api/controllers/openapi/account.py

@@ -34,7 +34,7 @@ from services.entities.account_entities import AccountSnapshot
 @openapi_ns.route("/account")
 class AccountApi(Resource):
     @endpoint(
-        op="account.get",
+        op="describe.account",
         kind=Kind.OBJECT,
         summary="Current account",
         examples=(Example(title="Show the logged-in account and its workspaces", input={}),),
@@ -61,7 +61,7 @@ class AccountApi(Resource):
 @openapi_ns.route("/account/sessions/self")
 class AccountSessionsSelfApi(Resource):
     @endpoint(
-        op="account.sessions.revoke_current",
+        op="delete.account.current_session",
         kind=Kind.OBJECT,
         summary="Revoke the session behind this token",
         examples=(Example(title="Log out the session behind this token", input={}),),
@@ -76,7 +76,7 @@ class AccountSessionsSelfApi(Resource):
 @openapi_ns.route("/account/sessions")
 class AccountSessionsApi(Resource):
     @endpoint(
-        op="account.sessions.list",
+        op="get.account.session",
         kind=Kind.LIST,
         summary="List login sessions of the current account",
         examples=(Example(title="List login sessions, first page", input={"page": 1, "limit": 20}),),
@@ -102,7 +102,7 @@ class AccountSessionsApi(Resource):
 @openapi_ns.route("/account/sessions/<string:session_id>")
 class AccountSessionByIdApi(Resource):
     @endpoint(
-        op="account.sessions.revoke",
+        op="delete.account.session",
         kind=Kind.OBJECT,
         summary="Revoke one login session by id",
         examples=(Example(title="Log out one device by session id", input={"session_id": "<session_id>"}),),

api/controllers/openapi/app_dsl.py

@@ -1,13 +1,14 @@
 from __future__ import annotations
 
 from http import HTTPStatus
+from typing import Final
 from uuid import UUID
 
 from flask_restx import Resource
 from werkzeug.exceptions import Forbidden
 
 from constants.oauth_bearer import Scope
-from controllers.common.rbac import PlainApp, RBACCheck, RBACPermission, Workspace
+from controllers.common.rbac import RBACCheck, RBACPermission, Workspace
 from controllers.openapi import openapi_ns
 from controllers.openapi._contract import Example, Kind, endpoint, op_of
 from controllers.openapi._models import (
@@ -19,22 +20,25 @@ from controllers.openapi._models import (
 )
 from controllers.openapi.auth.context import Context
 from controllers.openapi.auth.requirements import (
-    CheckAppApiEnabled,
+    EDITOR_ROLES,
     CheckRBACPermission,
     CheckScope,
     CheckSubject,
     CheckWorkspaceMember,
     CheckWorkspaceRole,
+    account_app_guards,
 )
 from controllers.openapi.auth.subjects import AccountSubject
 from extensions.ext_application_services import application_services
 from extensions.ext_database import db
 from machinery.context import RequestContext
-from models.account import TenantAccountRole
 from services.app_dsl_service import AppDslService
 from services.entities.dsl_entities import AppImportParams, CheckDependenciesResult, Import, ImportStatus
 from services.errors.account import NoPermissionError
 from services.errors.app import WorkflowNotFoundError
+from services.workflow_service import WorkflowService
+
+_DSL_READ_GUARDS: Final = account_app_guards(RBACPermission.APP_IMPORT_EXPORT_DSL, scope=Scope.APPS_READ, editor=True)
 
 
 def _import_response(result: Import, *, workspace_id: str) -> AppDslImportResponse:
@@ -66,7 +70,7 @@ class AppDslImportApi(Resource):
 
     @endpoint(
         account_context=True,
-        op="console_app.dsl.import",
+        op="import.console_app.dsl",
         kind=Kind.OBJECT,
         summary="Import an app from DSL text or URL",
         examples=(
@@ -88,7 +92,7 @@ class AppDslImportApi(Resource):
             CheckScope(Scope.WORKSPACE_WRITE),
             CheckWorkspaceMember(),
             CheckRBACPermission(RBACCheck(RBACPermission.APP_IMPORT_EXPORT_DSL, Workspace())),
-            CheckWorkspaceRole(frozenset({TenantAccountRole.EDITOR, TenantAccountRole.ADMIN, TenantAccountRole.OWNER})),
+            CheckWorkspaceRole(EDITOR_ROLES),
         ),
         body=AppDslImportPayload,
         returns=(
@@ -129,7 +133,7 @@ class AppDslImportConfirmApi(Resource):
 
     @endpoint(
         account_context=True,
-        op="console_app.dsl.import_confirm",
+        op="confirm.console_app.dsl_import",
         kind=Kind.OBJECT,
         summary="Confirm a pending DSL import",
         examples=(Example(title="Confirm an import that is pending confirmation", input={"import_id": "<import_id>"}),),
@@ -138,7 +142,7 @@ class AppDslImportConfirmApi(Resource):
             CheckScope(Scope.WORKSPACE_WRITE),
             CheckWorkspaceMember(),
             CheckRBACPermission(RBACCheck(RBACPermission.APP_IMPORT_EXPORT_DSL, Workspace())),
-            CheckWorkspaceRole(frozenset({TenantAccountRole.EDITOR, TenantAccountRole.ADMIN, TenantAccountRole.OWNER})),
+            CheckWorkspaceRole(EDITOR_ROLES),
         ),
         returns=((HTTPStatus.OK, Import, "Import confirmed"), (HTTPStatus.BAD_REQUEST, Import, "Import failed")),
     )
@@ -167,7 +171,7 @@ class AppDslExportApi(Resource):
     """
 
     @endpoint(
-        op="console_app.dsl.export",
+        op="export.console_app.dsl",
         kind=Kind.OBJECT,
         summary="Export app DSL as YAML text inside a JSON object",
         examples=(
@@ -177,18 +181,15 @@ class AppDslExportApi(Resource):
                 input={"app_id": "<app_id>", "workflow_id": "<workflow_id>", "include_secret": True},
             ),
         ),
-        requirements=(
-            CheckSubject(allowed=(AccountSubject,)),
-            CheckAppApiEnabled(),
-            CheckWorkspaceMember(),
-            CheckScope(Scope.APPS_READ),
-            CheckRBACPermission(RBACCheck(RBACPermission.APP_IMPORT_EXPORT_DSL, PlainApp())),
-            CheckWorkspaceRole(frozenset({TenantAccountRole.EDITOR, TenantAccountRole.ADMIN, TenantAccountRole.OWNER})),
-        ),
+        requirements=_DSL_READ_GUARDS,
         query=AppDslExportQuery,
         returns=(200, AppDslExportResponse, "Export successful"),
     )
     def get(self, ctx: Context, app_id: str, *, query: AppDslExportQuery):
+        # Capture the draft hash before building the DSL: an edit that commits in between would
+        # otherwise be described by a hash newer than the exported YAML, defeating the guard
+        # draft_hash exists for (a later import silently overwriting that edit).
+        draft = WorkflowService().get_draft_workflow(app_model=ctx.app, session=db.session())
         try:
             data = AppDslService.export_dsl(
                 app_model=ctx.app,
@@ -198,7 +199,7 @@ class AppDslExportApi(Resource):
             )
         except WorkflowNotFoundError as exc:
             return str(exc), 404
-        return AppDslExportResponse(data=data), 200
+        return AppDslExportResponse(data=data, draft_hash=draft.content_hash if draft else None), 200
 
 
 @openapi_ns.route("/apps/<string:app_id>/dependencies:check")
@@ -213,18 +214,11 @@ class AppDslCheckDependenciesApi(Resource):
 
     @endpoint(
         account_context=True,
-        op="console_app.dependencies.check",
+        op="check.console_app.dependency",
         kind=Kind.OBJECT,
         summary="Check plugin dependencies of an app",
         examples=(Example(title="Check which plugins an app needs", input={"app_id": "<app_id>"}),),
-        requirements=(
-            CheckSubject(allowed=(AccountSubject,)),
-            CheckAppApiEnabled(),
-            CheckWorkspaceMember(),
-            CheckScope(Scope.APPS_READ),
-            CheckRBACPermission(RBACCheck(RBACPermission.APP_IMPORT_EXPORT_DSL, PlainApp())),
-            CheckWorkspaceRole(frozenset({TenantAccountRole.EDITOR, TenantAccountRole.ADMIN, TenantAccountRole.OWNER})),
-        ),
+        requirements=_DSL_READ_GUARDS,
         returns=(HTTPStatus.OK, CheckDependenciesResult, "Dependencies checked"),
     )
     def get(self, ctx: RequestContext, app_id: str):

api/controllers/openapi/app_run.py

@@ -32,6 +32,7 @@ from controllers.openapi._models import (
     AdvancedChatRunPayload,
     ChatRunPayload,
     CompletionRunPayload,
+    DraftWorkflowRunPayload,
     Hint,
     RunPayloadBase,
     TaskStopResponse,
@@ -45,6 +46,8 @@ from controllers.openapi.auth.requirements import (
     CheckScope,
     CheckSubject,
     CheckWorkspaceMember,
+    Requirement,
+    account_app_guards,
 )
 from controllers.openapi.auth.subjects import AccountSubject, ExternalSsoSubject
 from controllers.openapi.human_input_form import with_form_hints
@@ -140,16 +143,19 @@ _RUN_GUARDS: Final = (
     CheckRBACPermission(RBACCheck(RBACPermission.APP_TEST_AND_RUN, PlainApp())),
     CheckAppAccess(),
 )
+_DRAFT_RUN_GUARDS: Final = account_app_guards(RBACPermission.APP_TEST_AND_RUN, scope=Scope.APPS_RUN, editor=True)
+# Draft runs raise debugger forms, which the form submit op does not accept.
+_DRAFT_FORM_NOTE: Final = "A human-input pause in a draft run cannot be resumed over openapi."
 _STREAM_RESULT: Final = (200, EventStreamResponse, "Run result (SSE stream)")
 
 
-def _generate(app: App, caller: Any, args: dict[str, Any], session: Session):
+def _generate(app: App, caller: Any, args: dict[str, Any], session: Session, invoke_from: InvokeFrom):
     return AppGenerateService.generate(
         session=session,
         app_model=app,
         user=caller,
         args=args,
-        invoke_from=InvokeFrom.OPENAPI,
+        invoke_from=invoke_from,
         streaming=True,
     )
 
@@ -164,12 +170,12 @@ def _generate_args(ctx: Context, payload: RunPayloadBase, *, exclude: Collection
     return args
 
 
-def _stream(ctx: Context, args: dict[str, Any]):
+def _stream(ctx: Context, args: dict[str, Any], invoke_from: InvokeFrom):
     with _translate_service_errors():
-        return _generate(ctx.app, ctx.caller, args, ctx.session)
+        return _generate(ctx.app, ctx.caller, args, ctx.session, invoke_from)
 
 
-def _require_mode(app: App, *modes: AppMode) -> None:
+def require_mode(app: App, *modes: AppMode) -> None:
     if app.mode not in modes:
         raise UnprocessableEntity("app_mode_mismatch")
 
@@ -231,13 +237,15 @@ class _RunRoute:
     modes: tuple[AppMode, ...]
     hints: tuple[HintLayer, ...] = ()
     examples: tuple[Example, ...] = ()
+    invoke_from: InvokeFrom = InvokeFrom.OPENAPI
+    requirements: tuple[Requirement, ...] = _RUN_GUARDS
 
 
 _RUN_ROUTES: Final = (
     _RunRoute(
         resource="WorkflowRunApi",
         segment="workflow",
-        op="console_app.workflow.run",
+        op="run.console_app.workflow",
         summary="Run a workflow app; streams workflow events",
         payload=WorkflowRunPayload,
         modes=(AppMode.WORKFLOW,),
@@ -260,7 +268,7 @@ _RUN_ROUTES: Final = (
     _RunRoute(
         resource="ChatRunApi",
         segment="chat",
-        op="console_app.chat.run",
+        op="run.console_app.chat",
         summary="Run a chat or agent app; streams message events",
         payload=ChatRunPayload,
         modes=(AppMode.CHAT, AppMode.AGENT_CHAT),
@@ -293,7 +301,7 @@ _RUN_ROUTES: Final = (
     _RunRoute(
         resource="AdvancedChatRunApi",
         segment="advanced-chat",
-        op="console_app.advanced_chat.run",
+        op="run.console_app.advanced_chat",
         summary="Run an advanced-chat (chatflow) app; streams message and workflow events",
         payload=AdvancedChatRunPayload,
         modes=(AppMode.ADVANCED_CHAT,),
@@ -317,7 +325,7 @@ _RUN_ROUTES: Final = (
     _RunRoute(
         resource="CompletionRunApi",
         segment="completion",
-        op="console_app.completion.run",
+        op="run.console_app.completion",
         summary="Run a completion app; streams message events",
         payload=CompletionRunPayload,
         modes=(AppMode.COMPLETION,),
@@ -332,6 +340,35 @@ _RUN_ROUTES: Final = (
             ),
         ),
     ),
+    _RunRoute(
+        resource="WorkflowDraftRunApi",
+        segment="draft/workflow",
+        op="run.console_app.draft.workflow",
+        summary="Test-run the draft of a workflow app; streams workflow events. " + _DRAFT_FORM_NOTE,
+        payload=DraftWorkflowRunPayload,
+        modes=(AppMode.WORKFLOW,),
+        examples=(
+            Example(title="Test the draft with one input", input={"app_id": "<app_id>", "inputs": {"topic": "test"}}),
+        ),
+        invoke_from=InvokeFrom.DEBUGGER,
+        requirements=_DRAFT_RUN_GUARDS,
+    ),
+    _RunRoute(
+        resource="AdvancedChatDraftRunApi",
+        segment="draft/advanced-chat",
+        op="run.console_app.draft.advanced_chat",
+        summary="Test-run the draft of an advanced-chat app; streams message and workflow events. " + _DRAFT_FORM_NOTE,
+        payload=ChatRunPayload,
+        modes=(AppMode.ADVANCED_CHAT,),
+        hints=(_reply_layer,),
+        examples=(
+            Example(
+                title="Test the draft with one message", input={"app_id": "<app_id>", "query": "hello", "inputs": {}}
+            ),
+        ),
+        invoke_from=InvokeFrom.DEBUGGER,
+        requirements=_DRAFT_RUN_GUARDS,
+    ),
 )
 
 
@@ -340,14 +377,14 @@ def _run_api(route: _RunRoute) -> type[Resource]:
         op=route.op,
         kind=Kind.SSE,
         summary=route.summary,
-        requirements=_RUN_GUARDS,
+        requirements=route.requirements,
         body=route.payload,
         examples=route.examples,
         returns=_STREAM_RESULT,
     )
     def post(self: Resource, ctx: Context, app_id: str, *, body: RunPayloadBase):
-        _require_mode(ctx.app, *route.modes)
-        stream = _stream(ctx, _generate_args(ctx, body))
+        require_mode(ctx.app, *route.modes)
+        stream = _stream(ctx, _generate_args(ctx, body), route.invoke_from)
         for layer in route.hints:
             stream = layer(stream, route.op, ctx.app.id)
         return _respond(ctx, stream)
@@ -356,13 +393,20 @@ def _run_api(route: _RunRoute) -> type[Resource]:
     return openapi_ns.route(f"/apps/<string:app_id>/{route.segment}:run")(resource)
 
 
-WorkflowRunApi, ChatRunApi, AdvancedChatRunApi, CompletionRunApi = (_run_api(route) for route in _RUN_ROUTES)
+(
+    WorkflowRunApi,
+    ChatRunApi,
+    AdvancedChatRunApi,
+    CompletionRunApi,
+    WorkflowDraftRunApi,
+    AdvancedChatDraftRunApi,
+) = (_run_api(route) for route in _RUN_ROUTES)
 
 
 @openapi_ns.route("/apps/<string:app_id>/tasks/<string:task_id>:stop")
 class AppRunTaskStopApi(Resource):
     @endpoint(
-        op="run.stop",
+        op="stop.run",
         kind=Kind.OBJECT,
         summary="Stop a running task",
         examples=(Example(title="Stop a running task", input={"app_id": "<app_id>", "task_id": "<task_id>"}),),

api/controllers/openapi/app_workflow.py

@@ -0,0 +1,394 @@
+"""Build routes on /openapi/v1 for workflow and advanced-chat apps: run history,
+publishing and versions, and draft environment variables."""
+
+from __future__ import annotations
+
+from collections.abc import Iterable
+from http import HTTPStatus
+from typing import Final
+from uuid import UUID
+
+from flask_restx import Resource
+from sqlalchemy.orm import sessionmaker
+from werkzeug.exceptions import BadRequest
+
+from constants import HIDDEN_VALUE
+from constants.oauth_bearer import Scope
+from controllers.common.fields import SimpleResultResponse
+from controllers.common.rbac import RBACPermission
+from controllers.openapi import openapi_ns
+from controllers.openapi._contract import Example, Kind, endpoint
+from controllers.openapi._errors import (
+    DraftNotFound,
+    OpenApiError,
+    RunNotFound,
+    SecretMaskNotSecret,
+    SecretMaskUnknownId,
+    VersionNotFound,
+    VersionNotRestorable,
+)
+from controllers.openapi._models import (
+    EnvVariableListResponse,
+    EnvVariableRow,
+    EnvVariableSetPayload,
+    EnvVariableValueType,
+    Hint,
+    PublishPayload,
+    PublishResponse,
+    RestoreResponse,
+    RunListQuery,
+    RunListResponse,
+    VersionListQuery,
+    VersionListResponse,
+    VersionRow,
+)
+from controllers.openapi.app_run import require_mode
+from controllers.openapi.auth.context import Context
+from controllers.openapi.auth.requirements import account_app_guards
+from core.helper import encrypter
+from core.logging.context import get_request_id, get_trace_id
+from core.workflow.llm_environment_variable import environment_variable_value_type
+from extensions.ext_application_services import application_services
+from extensions.ext_database import db
+from factories import variable_factory
+from fields.workflow_run_fields import (
+    WorkflowRunDetailResponse,
+    WorkflowRunNodeExecutionListResponse,
+    workflow_run_pagination_response_source,
+    workflow_run_response_source,
+)
+from graphon.variables import SecretVariable, VariableBase
+from graphon.variables.exc import VariableError
+from libs.helper import to_timestamp
+from machinery.context import RequestContext
+from models import AppMode, WorkflowRunTriggeredFrom
+from models.workflow import Workflow, WorkflowRun
+from services.errors.app import IsDraftWorkflowError, WorkflowNotFoundError
+from services.workflow_run_service import WorkflowRunListArgs
+from services.workflow_service import WorkflowService
+from services.workflow_variable_reference_validator import advisory_variable_reference_warning
+
+GRAPH_MODES: Final = (AppMode.WORKFLOW, AppMode.ADVANCED_CHAT)
+_RUN_LIST_OP: Final = "get.console_app.run"
+_VERSION_LIST_OP: Final = "get.console_app.version"
+
+_RUN_READ_GUARDS: Final = account_app_guards(
+    RBACPermission.APP_CREATE_AND_MANAGEMENT, scope=Scope.APPS_READ, editor=False
+)
+_VERSION_READ_GUARDS: Final = account_app_guards(RBACPermission.APP_VIEW_LAYOUT, scope=Scope.APPS_READ, editor=True)
+_RELEASE_GUARDS: Final = account_app_guards(
+    RBACPermission.APP_RELEASE_AND_VERSION, scope=Scope.WORKSPACE_WRITE, editor=True
+)
+_ENV_READ_GUARDS: Final = account_app_guards(RBACPermission.APP_VIEW_LAYOUT, scope=Scope.APPS_READ, editor=True)
+_ENV_WRITE_GUARDS: Final = account_app_guards(RBACPermission.APP_EDIT, scope=Scope.WORKSPACE_WRITE, editor=True)
+
+
+def _request_context(ctx: Context) -> RequestContext:
+    return RequestContext(get_request_id(), get_trace_id(), ctx.account.id, ctx.workspace.id)
+
+
+def _require_uuid(value: str, *, not_found: type[OpenApiError]) -> str:
+    """Parse a path id as a UUID or raise 404; an unparsed id would otherwise reach
+    a UUID-typed database column and fail as a 500."""
+    try:
+        return str(UUID(value))
+    except ValueError:
+        raise not_found()
+
+
+def _require_run(ctx: Context, run_id: str) -> WorkflowRun:
+    run = application_services().workflow_runs.get_workflow_run(
+        _request_context(ctx), app_id=ctx.app.id, run_id=_require_uuid(run_id, not_found=RunNotFound)
+    )
+    if run is None:
+        raise RunNotFound()
+    return run
+
+
+def with_next_cursor(page: RunListResponse, *, op: str, app_id: str, query: RunListQuery) -> RunListResponse:
+    if page.has_more and page.data:
+        next_input = {"app_id": app_id, **query.model_dump(exclude_none=True), "last_id": page.data[-1].id}
+        page.hints = [Hint(summary="Next page", op=op, input=next_input)]
+    return page
+
+
+@openapi_ns.route("/apps/<string:app_id>/runs")
+class AppRunListApi(Resource):
+    @endpoint(
+        op=_RUN_LIST_OP,
+        kind=Kind.OBJECT,
+        summary="List runs of a workflow or advanced-chat app, newest first",
+        examples=(
+            Example(
+                title="Latest failed runs from real use",
+                input={"app_id": "<app_id>", "status": "failed", "triggered_from": "app-run"},
+            ),
+            Example(
+                title="My latest draft test run",
+                input={"app_id": "<app_id>", "triggered_from": "debugging", "limit": 1},
+            ),
+        ),
+        requirements=_RUN_READ_GUARDS,
+        query=RunListQuery,
+        returns=(HTTPStatus.OK, RunListResponse, "Run list"),
+    )
+    def get(self, ctx: Context, app_id: str, *, query: RunListQuery):
+        require_mode(ctx.app, *GRAPH_MODES)
+        args: WorkflowRunListArgs = {"limit": query.limit}
+        if query.last_id is not None:
+            args["last_id"] = query.last_id
+        if query.status is not None:
+            args["status"] = query.status
+        pagination = application_services().workflow_runs.get_paginate_workflow_runs(
+            _request_context(ctx),
+            app_id=ctx.app.id,
+            args=args,
+            triggered_from=WorkflowRunTriggeredFrom(query.triggered_from or WorkflowRunTriggeredFrom.DEBUGGING),
+        )
+        page = RunListResponse.model_validate(workflow_run_pagination_response_source(pagination, session=ctx.session))
+        return with_next_cursor(page, op=_RUN_LIST_OP, app_id=ctx.app.id, query=query)
+
+
+@openapi_ns.route("/apps/<string:app_id>/runs/<string:run_id>")
+class AppRunDescribeApi(Resource):
+    @endpoint(
+        op="describe.console_app.run",
+        kind=Kind.OBJECT,
+        summary="One run: inputs, outputs, error, tokens and timing",
+        examples=(Example(title="Read one run", input={"app_id": "<app_id>", "run_id": "<run_id>"}),),
+        requirements=_RUN_READ_GUARDS,
+        returns=(HTTPStatus.OK, WorkflowRunDetailResponse, "Run detail"),
+    )
+    def get(self, ctx: Context, app_id: str, run_id: str):
+        require_mode(ctx.app, *GRAPH_MODES)
+        run = _require_run(ctx, run_id)
+        return WorkflowRunDetailResponse.model_validate(workflow_run_response_source(run, session=ctx.session))
+
+
+@openapi_ns.route("/apps/<string:app_id>/runs/<string:run_id>/nodes")
+class AppRunNodeListApi(Resource):
+    @endpoint(
+        op="get.console_app.run.node",
+        kind=Kind.OBJECT,
+        summary="Each node step of a run, to find the one that failed",
+        examples=(Example(title="Node steps of one run", input={"app_id": "<app_id>", "run_id": "<run_id>"}),),
+        requirements=_RUN_READ_GUARDS,
+        returns=(HTTPStatus.OK, WorkflowRunNodeExecutionListResponse, "Node steps"),
+    )
+    def get(self, ctx: Context, app_id: str, run_id: str):
+        require_mode(ctx.app, *GRAPH_MODES)
+        run = _require_run(ctx, run_id)
+        steps = application_services().workflow_runs.get_workflow_run_node_executions(
+            _request_context(ctx), app_id=ctx.app.id, run_id=run.id
+        )
+        return WorkflowRunNodeExecutionListResponse.model_validate({"data": steps})
+
+
+@openapi_ns.route("/apps/<string:app_id>:publish")
+class AppPublishApi(Resource):
+    @endpoint(
+        op="publish.console_app",
+        kind=Kind.OBJECT,
+        summary="Publish the draft as a new version and make the app use it",
+        examples=(
+            Example(title="Publish the draft", input={"app_id": "<app_id>"}),
+            Example(
+                title="Publish a named version",
+                input={"app_id": "<app_id>", "marked_name": "v2", "marked_comment": "Adds a retry"},
+            ),
+        ),
+        requirements=_RELEASE_GUARDS,
+        body=PublishPayload,
+        returns=(HTTPStatus.OK, PublishResponse, "Published"),
+    )
+    def post(self, ctx: Context, app_id: str, *, body: PublishPayload):
+        require_mode(ctx.app, *GRAPH_MODES)
+        try:
+            with sessionmaker(db.engine).begin() as session:
+                workflow = WorkflowService().publish_app_workflow(
+                    session=session,
+                    app_model=ctx.app,
+                    account=ctx.account,
+                    marked_name=body.marked_name,
+                    marked_comment=body.marked_comment,
+                )
+                published = PublishResponse(
+                    version_id=workflow.id,
+                    created_at=to_timestamp(workflow.created_at),
+                    warning=advisory_variable_reference_warning(workflow.graph),
+                )
+        except ValueError as exc:
+            raise BadRequest(str(exc)) from exc
+        return published
+
+
+@openapi_ns.route("/apps/<string:app_id>/versions")
+class AppVersionListApi(Resource):
+    @endpoint(
+        op=_VERSION_LIST_OP,
+        kind=Kind.OBJECT,
+        summary="List published versions, newest first",
+        examples=(
+            Example(title="Latest versions", input={"app_id": "<app_id>"}),
+            Example(title="Only named versions", input={"app_id": "<app_id>", "named_only": True}),
+        ),
+        requirements=_VERSION_READ_GUARDS,
+        query=VersionListQuery,
+        returns=(HTTPStatus.OK, VersionListResponse, "Version list"),
+    )
+    def get(self, ctx: Context, app_id: str, *, query: VersionListQuery):
+        require_mode(ctx.app, *GRAPH_MODES)
+        workflows, has_more = WorkflowService().get_all_published_workflow(
+            session=ctx.session,
+            app_model=ctx.app,
+            page=query.page,
+            limit=query.limit,
+            user_id=None,
+            named_only=query.named_only,
+            include_draft=False,
+        )
+        page = VersionListResponse(
+            page=query.page,
+            limit=query.limit,
+            has_more=has_more,
+            data=[VersionRow.model_validate(workflow) for workflow in workflows],
+        )
+        if has_more:
+            next_input = {"app_id": ctx.app.id, **query.model_dump(), "page": query.page + 1}
+            page.hints = [Hint(summary="Next page", op=_VERSION_LIST_OP, input=next_input)]
+        return page
+
+
+@openapi_ns.route("/apps/<string:app_id>/versions/<string:version_id>:restore")
+class AppVersionRestoreApi(Resource):
+    @endpoint(
+        op="restore.console_app.version",
+        kind=Kind.OBJECT,
+        summary="Copy a published version into the draft; publish again to make it live",
+        examples=(
+            Example(title="Restore an older version", input={"app_id": "<app_id>", "version_id": "<version_id>"}),
+        ),
+        requirements=_RELEASE_GUARDS,
+        returns=(HTTPStatus.OK, RestoreResponse, "Restored into the draft"),
+    )
+    def post(self, ctx: Context, app_id: str, version_id: str):
+        require_mode(ctx.app, *GRAPH_MODES)
+        version_id = _require_uuid(version_id, not_found=VersionNotFound)
+        try:
+            draft = WorkflowService().restore_published_workflow_to_draft(
+                app_model=ctx.app, workflow_id=version_id, account=ctx.account, session=db.session()
+            )
+        except IsDraftWorkflowError as exc:
+            raise VersionNotRestorable() from exc
+        except WorkflowNotFoundError as exc:
+            raise VersionNotFound() from exc
+        except ValueError as exc:
+            raise BadRequest(str(exc)) from exc
+        return RestoreResponse(result="success", draft_hash=draft.content_hash)
+
+
+def env_variable_rows(variables: Iterable[VariableBase]) -> list[EnvVariableRow]:
+    return [
+        EnvVariableRow(
+            id=variable.id,
+            name=variable.name,
+            description=variable.description,
+            value_type=environment_variable_value_type(variable),
+            value=encrypter.full_mask_token()
+            if isinstance(variable, SecretVariable) and variable.value
+            else variable.value,
+        )
+        for variable in variables
+    ]
+
+
+def _stored_secret_ids(draft: Workflow | None) -> set[str]:
+    if draft is None:
+        return set()
+    return {variable.id for variable in draft.environment_variables if isinstance(variable, SecretVariable)}
+
+
+def _patch_env(ctx: Context, *, upserts: list[VariableBase], deletions: list[str]) -> None:
+    try:
+        WorkflowService().patch_draft_workflow_environment_variables(
+            app_model=ctx.app,
+            environment_variables=upserts,
+            deleted_environment_variable_ids=deletions,
+            account=ctx.account,
+            session=db.session(),
+        )
+    except ValueError as exc:
+        raise BadRequest(str(exc)) from exc
+
+
+@openapi_ns.route("/apps/<string:app_id>/env")
+class AppEnvListApi(Resource):
+    @endpoint(
+        op="get.console_app.env",
+        kind=Kind.OBJECT,
+        summary="Environment variables of the draft. Each has an id, which set and delete take, "
+        "and a name, which nodes use. Secrets with a value are masked; empty secrets read as empty",
+        examples=(Example(title="List environment variables", input={"app_id": "<app_id>"}),),
+        requirements=_ENV_READ_GUARDS,
+        returns=(HTTPStatus.OK, EnvVariableListResponse, "Environment variables"),
+    )
+    def get(self, ctx: Context, app_id: str):
+        require_mode(ctx.app, *GRAPH_MODES)
+        draft = WorkflowService().get_draft_workflow(app_model=ctx.app, session=ctx.session)
+        if draft is None:
+            raise DraftNotFound()
+        return EnvVariableListResponse(data=env_variable_rows(draft.environment_variables))
+
+
+@openapi_ns.route("/apps/<string:app_id>/env/<string:env_id>")
+class AppEnvItemApi(Resource):
+    @endpoint(
+        op="set.console_app.env",
+        kind=Kind.OBJECT,
+        summary="Add or replace one environment variable of the draft, keyed by its id from get console_app env, "
+        "not its name. A new id (e.g. a fresh UUID) adds a variable. Others are left alone",
+        examples=(
+            Example(
+                title="Set an API key as a secret",
+                input={
+                    "app_id": "<app_id>",
+                    "env_id": "<env_id>",
+                    "name": "API_KEY",
+                    "value_type": "secret",
+                    "value": "sk-...",
+                },
+            ),
+        ),
+        requirements=_ENV_WRITE_GUARDS,
+        body=EnvVariableSetPayload,
+        returns=(HTTPStatus.OK, SimpleResultResponse, "Variable set"),
+    )
+    def put(self, ctx: Context, app_id: str, env_id: str, *, body: EnvVariableSetPayload):
+        require_mode(ctx.app, *GRAPH_MODES)
+        if body.value == encrypter.full_mask_token() and body.value_type != EnvVariableValueType.SECRET:
+            raise SecretMaskNotSecret()
+        [mapping] = Workflow.normalize_environment_variable_mappings([{**body.model_dump(mode="json"), "id": env_id}])
+        if mapping["value"] == HIDDEN_VALUE:
+            draft = WorkflowService().get_draft_workflow(app_model=ctx.app, session=ctx.session)
+            if env_id not in _stored_secret_ids(draft):
+                raise SecretMaskUnknownId()
+        try:
+            variable = variable_factory.build_environment_variable_from_mapping(mapping)
+        except VariableError as exc:
+            raise BadRequest(str(exc)) from exc
+        _patch_env(ctx, upserts=[variable], deletions=[])
+        return SimpleResultResponse(result="success")
+
+    @endpoint(
+        op="delete.console_app.env",
+        kind=Kind.OBJECT,
+        summary="Remove one environment variable from the draft, keyed by its id from get console_app env, "
+        "not its name. Others are left alone",
+        examples=(Example(title="Remove a variable", input={"app_id": "<app_id>", "env_id": "<env_id>"}),),
+        requirements=_ENV_WRITE_GUARDS,
+        returns=(HTTPStatus.OK, SimpleResultResponse, "Variable removed"),
+    )
+    def delete(self, ctx: Context, app_id: str, env_id: str):
+        require_mode(ctx.app, *GRAPH_MODES)
+        _patch_env(ctx, upserts=[], deletions=[env_id])
+        return SimpleResultResponse(result="success")

api/controllers/openapi/apps.py

@@ -105,7 +105,7 @@ def build_app_describe_response(app: App, fields: set[str] | None, *, session: S
 @openapi_ns.route("/apps/<string:app_id>")
 class AppDescribeApi(Resource):
     @endpoint(
-        op="console_app.describe",
+        op="describe.console_app",
         kind=Kind.OBJECT,
         summary="App detail, parameters and runtime input_schema",
         examples=(
@@ -133,7 +133,7 @@ class AppDescribeApi(Resource):
 @openapi_ns.route("/apps")
 class AppListApi(Resource):
     @endpoint(
-        op="console_app.list",
+        op="get.console_app",
         kind=Kind.LIST,
         summary="List apps in a workspace",
         examples=(

api/controllers/openapi/apps_permitted_external.py

@@ -43,7 +43,7 @@ _ENTERPRISE_ONLY = frozenset({DeploymentEdition.ENTERPRISE})
 @openapi_ns.route("/permitted-external-apps")
 class PermittedExternalAppsListApi(Resource):
     @endpoint(
-        op="console_app.external.list",
+        op="get.console_app.external",
         kind=Kind.LIST,
         summary="List apps an external SSO subject may run",
         examples=(Example(title="List the apps this SSO subject may run, first page", input={"page": 1, "limit": 20}),),
@@ -101,7 +101,7 @@ class PermittedExternalAppsListApi(Resource):
 @openapi_ns.route("/permitted-external-apps/<string:app_id>")
 class PermittedExternalAppDescribeApi(Resource):
     @endpoint(
-        op="console_app.external.describe",
+        op="describe.console_app.external",
         kind=Kind.OBJECT,
         summary="External-subject app detail",
         examples=(Example(title="Describe a permitted app with its input_schema", input={"app_id": "<app_id>"}),),

api/controllers/openapi/auth/requirements.py

@@ -11,7 +11,7 @@ from __future__ import annotations
 from abc import ABC, abstractmethod
 from collections.abc import Sequence
 from enum import IntEnum
-from typing import ClassVar, override
+from typing import ClassVar, Final, override
 
 from flask import request
 from sqlalchemy.orm import Session
@@ -19,11 +19,11 @@ from werkzeug.exceptions import Forbidden
 
 from configs import dify_config
 from constants.oauth_bearer import Scope
-from controllers.common.rbac import RBACCheck, enforce_rbac_checks
+from controllers.common.rbac import PlainApp, RBACCheck, RBACPermission, enforce_rbac_checks
 from controllers.openapi._audit import emit_wrong_surface
 from controllers.openapi.auth.context import Context
 from controllers.openapi.auth.loaders import load_app, load_caller, load_workspace, load_workspace_role
-from controllers.openapi.auth.subjects import Subject
+from controllers.openapi.auth.subjects import AccountSubject, Subject
 from enums import DeploymentEdition
 from models.account import TenantAccountRole
 from models.enums import CreatorUserRole
@@ -141,6 +141,9 @@ class CheckRBACPermission(Requirement):
         )
 
 
+EDITOR_ROLES: Final = frozenset({TenantAccountRole.EDITOR, TenantAccountRole.ADMIN, TenantAccountRole.OWNER})
+
+
 class CheckWorkspaceRole(Requirement):
     """The workspace-role gate that predates RBAC. Inert wherever RBAC is on;
     a route that needs a check there declares a `CheckRBACPermission` beside this.
@@ -206,3 +209,15 @@ class ResolveCaller(Requirement):
         if not subject.mounts_caller(ctx):
             return
         load_caller(ctx)
+
+
+def account_app_guards(permission: RBACPermission, *, scope: Scope, editor: bool) -> tuple[Requirement, ...]:
+    """Guards for an account-only op on one app; `editor` adds the pre-RBAC editor-role gate."""
+    guards: tuple[Requirement, ...] = (
+        CheckSubject(allowed=(AccountSubject,)),
+        CheckAppApiEnabled(),
+        CheckWorkspaceMember(),
+        CheckScope(scope),
+        CheckRBACPermission(RBACCheck(permission, PlainApp())),
+    )
+    return (*guards, CheckWorkspaceRole(EDITOR_ROLES)) if editor else guards

api/controllers/openapi/files.py

@@ -35,7 +35,7 @@ class AppFileUploadApi(Resource):
         }
     )
     @endpoint(
-        op="console_app.file.upload",
+        op="upload.console_app.file",
         kind=Kind.OBJECT,
         summary="Upload a file and get a file id for later runs",
         examples=(

api/controllers/openapi/human_input_form.py

@@ -92,7 +92,7 @@ def _ensure_form_belongs_to_app(form, app_model: App) -> None:
 @openapi_ns.route("/apps/<string:app_id>/human-input-forms/<string:form_token>")
 class OpenApiWorkflowHumanInputFormApi(Resource):
     @endpoint(
-        op="run.form.get",
+        op="describe.run.form",
         kind=Kind.OBJECT,
         summary="Read a human-input form",
         examples=(
@@ -126,7 +126,7 @@ class OpenApiWorkflowHumanInputFormApi(Resource):
 @openapi_ns.route("/apps/<string:app_id>/human-input-forms/<string:form_token>:submit")
 class OpenApiWorkflowHumanInputFormSubmitApi(Resource):
     @endpoint(
-        op="run.form.submit",
+        op="submit.run.form",
         kind=Kind.OBJECT,
         summary="Submit a human-input form",
         examples=(

api/controllers/openapi/workflow_events.py

@@ -53,7 +53,7 @@ class WorkflowEventsQuery(BaseModel):
 @openapi_ns.route("/apps/<string:app_id>/tasks/<string:task_id>/events")
 class OpenApiWorkflowEventsApi(Resource):
     @endpoint(
-        op="run.events",
+        op="get.run.event",
         kind=Kind.SSE,
         summary="Stream the events of a workflow run",
         examples=(

api/controllers/openapi/workspaces.py

@@ -89,7 +89,7 @@ def _check_member_invite_quota(tenant_id: str) -> None:
 @openapi_ns.route("/workspaces")
 class WorkspacesApi(Resource):
     @endpoint(
-        op="workspace.list",
+        op="get.workspace",
         kind=Kind.LIST,
         summary="List workspaces of the current account",
         examples=(Example(title="List my workspaces, first page", input={"page": 1, "limit": 20}),),
@@ -105,7 +105,7 @@ class WorkspacesApi(Resource):
 @openapi_ns.route("/workspaces/<string:workspace_id>")
 class WorkspaceByIdApi(Resource):
     @endpoint(
-        op="workspace.get",
+        op="describe.workspace",
         kind=Kind.OBJECT,
         summary="Workspace detail",
         examples=(
@@ -134,7 +134,7 @@ class WorkspaceSwitchApi(Resource):
     """
 
     @endpoint(
-        op="workspace.switch",
+        op="switch.workspace",
         kind=Kind.OBJECT,
         summary="Server-side current workspace switch (shared with the web console)",
         examples=(Example(title="Make a workspace current on the server", input={"workspace_id": "<workspace_id>"}),),
@@ -168,7 +168,7 @@ class WorkspaceMembersApi(Resource):
     """
 
     @endpoint(
-        op="workspace.members.list",
+        op="get.workspace.member",
         kind=Kind.LIST,
         summary="List workspace members",
         examples=(Example(title="List members of the pinned workspace, first page", input={"page": 1, "limit": 20}),),
@@ -185,7 +185,7 @@ class WorkspaceMembersApi(Resource):
         return MemberListResponse.page_of([_member_response(m) for m in members], query=query)
 
     @endpoint(
-        op="workspace.members.invite",
+        op="invite.workspace.member",
         kind=Kind.OBJECT,
         summary="Invite a member by email",
         examples=(
@@ -252,7 +252,7 @@ class WorkspaceMemberApi(Resource):
     """
 
     @endpoint(
-        op="workspace.members.remove",
+        op="delete.workspace.member",
         kind=Kind.OBJECT,
         summary="Remove a member",
         examples=(Example(title="Remove a member by account id", input={"member_id": "<member_id>"}),),
@@ -282,7 +282,7 @@ class WorkspaceMemberApi(Resource):
         return MemberActionResponse()
 
     @endpoint(
-        op="workspace.members.set_role",
+        op="set.workspace.member_role",
         kind=Kind.OBJECT,
         summary="Change a member's role",
         examples=(

api/controllers/service_api/dataset/dataset.py

@@ -37,11 +37,6 @@ from core.plugin.impl.model_runtime_factory import create_plugin_provider_manage
 from core.rag.index_processor.constant.index_type import IndexTechniqueType
 from extensions.ext_application_services import application_services
 from fields.base import ResponseModel
-from fields.dataset_fields import (
-    DatasetDetailPrefetch,
-    build_dataset_detail_prefetch,
-    dataset_detail_response_source,
-)
 from fields.dataset_fields import DatasetDetailResponse as BaseDatasetDetailResponse
 from graphon.model_runtime.entities.model_entities import ModelType
 from libs.helper import dump_response
@@ -51,9 +46,11 @@ from models.account import Account
 from models.dataset import DatasetPermissionEnum
 from models.enums import TagType
 from models.provider_ids import ModelProviderID
-from services.dataset_service import DatasetPermissionService, DatasetService, DocumentService
 from services.enterprise import rbac_service as enterprise_rbac_service
-from services.entities.knowledge_entities.knowledge_entities import (
+from services.knowledge.dataset_read_service import load_dataset_detail, load_dataset_details
+from services.knowledge.dataset_service import DatasetPermissionService, DatasetService, DocumentService
+from services.knowledge.entities.datasets import DatasetDetailRecord
+from services.knowledge.entities.knowledge_entities import (
     ExternalRetrievalModel,
     KnowledgeProvider,
     RetrievalModel,
@@ -104,12 +101,8 @@ _SERVICE_DATASET_DETAIL_EXCLUDE = {"permission_keys"}
 _SERVICE_DATASET_LIST_EXCLUDE = {"data": {"__all__": _SERVICE_DATASET_DETAIL_EXCLUDE}}
 
 
-def _dump_service_dataset_detail(
-    dataset: Any, *, session: Session, prefetch: DatasetDetailPrefetch | None = None
-) -> dict[str, Any]:
-    return DatasetDetailResponse.model_validate(
-        dataset_detail_response_source(dataset, session=session, prefetch=prefetch), from_attributes=True
-    ).model_dump(
+def _dump_service_dataset_detail(detail: DatasetDetailRecord) -> dict[str, Any]:
+    return DatasetDetailResponse.model_validate(detail).model_dump(
         mode="json",
         exclude=_SERVICE_DATASET_DETAIL_EXCLUDE,
     )
@@ -470,8 +463,7 @@ class DatasetListApi(DatasetApiResource):
         for embedding_model in embedding_models:
             model_names.append(f"{embedding_model.model}:{embedding_model.provider.provider}")
 
-        prefetch = build_dataset_detail_prefetch(datasets, session=session)
-        data = [_dump_service_dataset_detail(dataset, session=session, prefetch=prefetch) for dataset in datasets]
+        data = [_dump_service_dataset_detail(detail) for detail in load_dataset_details(datasets, session=session)]
         for item in data:
             if item["indexing_technique"] == IndexTechniqueType.HIGH_QUALITY and item["embedding_model_provider"]:
                 item["embedding_model_provider"] = str(ModelProviderID(item["embedding_model_provider"]))
@@ -571,7 +563,7 @@ class DatasetListApi(DatasetApiResource):
                 enterprise_rbac_service.ReplaceMemberBindings(automatic_include_workspace_members=False),
             )
 
-        return _dump_service_dataset_detail(dataset, session=session), 200
+        return _dump_service_dataset_detail(load_dataset_detail(dataset, session=session)), 200
 
 
 @service_api_ns.route("/datasets/<uuid:dataset_id>")
@@ -617,7 +609,7 @@ class DatasetApi(DatasetApiResource):
             DatasetService.check_dataset_permission(dataset, current_user, session)
         except services.errors.account.NoPermissionError as e:
             raise Forbidden(str(e))
-        data = _dump_service_dataset_detail(dataset, session=session)
+        data = _dump_service_dataset_detail(load_dataset_detail(dataset, session=session))
         # check embedding setting
         assert isinstance(current_user, Account)
         cid = current_user.current_tenant_id
@@ -734,7 +726,7 @@ class DatasetApi(DatasetApiResource):
         if dataset is None:
             raise NotFound("Dataset not found.")
 
-        result_data = _dump_service_dataset_detail(dataset, session=session)
+        result_data = _dump_service_dataset_detail(load_dataset_detail(dataset, session=session))
         assert isinstance(current_user, Account)
         tenant_id = current_user.current_tenant_id
 

api/controllers/service_api/dataset/document.py

@@ -69,25 +69,36 @@ from fields.document_fields import (
     DocumentMetadataResponse,
     DocumentResponse,
     DocumentStatusListResponse,
-    document_response,
-    document_responses,
     normalize_enum,
 )
 from libs.helper import dump_response
 from libs.login import current_user
 from libs.pagination import clamp_pagination, paginate_query
 from models.dataset import Dataset, Document
-from services.dataset_service import DatasetService, DocumentService
-from services.entities.knowledge_entities.knowledge_entities import (
+from repositories.knowledge.dataset_read_repository import (
+    get_dataset_creator,
+    get_document_hit_count,
+    get_document_process_rule,
+    get_document_segment_count,
+    get_latest_dataset_process_rule,
+)
+from services.feature_service import FeatureService
+from services.file_service import FileService
+from services.knowledge.dataset_read_service import (
+    get_document_metadata_details,
+    get_document_source_detail,
+    load_document_detail,
+    load_document_details,
+)
+from services.knowledge.dataset_service import DatasetService, DocumentService
+from services.knowledge.entities.knowledge_entities import (
     DocForm,
     IndexingTechnique,
     KnowledgeConfig,
     ProcessRule,
     RetrievalModel,
 )
-from services.feature_service import FeatureService
-from services.file_service import FileService
-from services.summary_index_service import SummaryIndexService
+from services.knowledge.summaries.adapters import SummaryIndexAdapter
 
 
 class DocumentTextCreatePayload(BaseModel):
@@ -304,7 +315,7 @@ class DocumentAndBatchResponse(ResponseModel):
 def _document_and_batch_response(document: Document, batch: str, *, session: Session) -> dict[str, Any]:
     return dump_response(
         DocumentAndBatchResponse,
-        {"document": document_response(document, session=session), "batch": batch},
+        {"document": load_document_detail(document, session=session), "batch": batch},
     )
 
 
@@ -445,7 +456,7 @@ def _create_document_by_text(session: Session, tenant_id: str, dataset_id: UUID)
             dataset=dataset,
             knowledge_config=knowledge_config,
             account=current_user,
-            dataset_process_rule=dataset.get_latest_process_rule(session=session)
+            dataset_process_rule=get_latest_dataset_process_rule(dataset, session=session)
             if "process_rule" not in args
             else None,
             created_from="api",
@@ -509,7 +520,7 @@ def _update_document_by_text(
             dataset=dataset,
             knowledge_config=knowledge_config,
             account=current_user,
-            dataset_process_rule=dataset.get_latest_process_rule(session=session)
+            dataset_process_rule=get_latest_dataset_process_rule(dataset, session=session)
             if "process_rule" not in args
             else None,
             created_from="api",
@@ -822,7 +833,9 @@ class DocumentAddByFileApi(DatasetApiResource):
         knowledge_config = KnowledgeConfig.model_validate(args)
         DocumentService.document_create_args_validate(knowledge_config)
 
-        dataset_process_rule = dataset.get_latest_process_rule(session=session) if "process_rule" not in args else None
+        dataset_process_rule = (
+            get_latest_dataset_process_rule(dataset, session=session) if "process_rule" not in args else None
+        )
         if not knowledge_config.original_document_id and not dataset_process_rule and not knowledge_config.process_rule:
             raise ValueError("process_rule is required.")
 
@@ -830,7 +843,7 @@ class DocumentAddByFileApi(DatasetApiResource):
             documents, batch = DocumentService.save_document_with_dataset_id(
                 dataset=dataset,
                 knowledge_config=knowledge_config,
-                account=dataset.get_created_by_account(session=session),
+                account=get_dataset_creator(dataset, session=session),
                 dataset_process_rule=dataset_process_rule,
                 created_from="api",
                 session=session,
@@ -909,8 +922,8 @@ def _update_document_by_file(
         documents, _ = DocumentService.save_document_with_dataset_id(
             dataset=dataset,
             knowledge_config=knowledge_config,
-            account=dataset.get_created_by_account(session=session),
-            dataset_process_rule=dataset.get_latest_process_rule(session=session)
+            account=get_dataset_creator(dataset, session=session),
+            dataset_process_rule=get_latest_dataset_process_rule(dataset, session=session)
             if "process_rule" not in args
             else None,
             created_from="api",
@@ -1046,9 +1059,7 @@ class DocumentListApi(DatasetApiResource):
         )
 
         response = {
-            "data": document_responses(documents, session=session),
-            # The result object already knows: it was built from the page the query
-            # ran with, while the requested values are only ever a request.
+            "data": load_document_details(documents, session=session),
             "has_more": paginated_documents.has_next,
             "limit": paginated_documents.per_page,
             "total": paginated_documents.total,
@@ -1303,7 +1314,7 @@ class DocumentApi(DatasetApiResource):
         summary_index_status = None
         has_summary_index = dataset.summary_index_setting and dataset.summary_index_setting.get("enable") is True
         if has_summary_index and document.need_summary is True:
-            summary_index_status = SummaryIndexService.get_document_summary_index_status(
+            summary_index_status = SummaryIndexAdapter.get_document_summary_index_status(
                 document_id=document_id_str,
                 dataset_id=dataset_id_str,
                 tenant_id=tenant_id,
@@ -1315,15 +1326,15 @@ class DocumentApi(DatasetApiResource):
             response = {
                 "id": document.id,
                 "doc_type": document.doc_type,
-                "doc_metadata": document.get_doc_metadata_details(session=session),
+                "doc_metadata": get_document_metadata_details(document, session=session),
             }
         elif metadata == "without":
             dataset_process_rules = DatasetService.get_process_rules(dataset_id_str, session)
             response_exclude = {"doc_type", "doc_metadata"}
-            document_process_rule = document.get_dataset_process_rule(session=session)
+            document_process_rule = get_document_process_rule(document, session=session)
             document_process_rules: Mapping[str, Any] = document_process_rule.to_dict() if document_process_rule else {}
-            data_source_info = document.get_data_source_detail_dict(session=session)
-            segment_count = document.get_segment_count(session=session)
+            data_source_info = get_document_source_detail(document, session=session)
+            segment_count = get_document_segment_count(document, session=session)
             response = {
                 "id": document.id,
                 "position": document.position,
@@ -1348,7 +1359,7 @@ class DocumentApi(DatasetApiResource):
                 "archived": document.archived,
                 "segment_count": segment_count,
                 "average_segment_length": (document.word_count or 0) // segment_count if segment_count else 0,
-                "hit_count": document.get_hit_count(session=session),
+                "hit_count": get_document_hit_count(document, session=session),
                 "display_status": document.display_status,
                 "doc_form": document.doc_form,
                 "doc_language": document.doc_language,
@@ -1357,10 +1368,10 @@ class DocumentApi(DatasetApiResource):
             }
         else:
             dataset_process_rules = DatasetService.get_process_rules(dataset_id_str, session)
-            document_process_rule = document.get_dataset_process_rule(session=session)
+            document_process_rule = get_document_process_rule(document, session=session)
             document_process_rules = document_process_rule.to_dict() if document_process_rule else {}
-            data_source_info = document.get_data_source_detail_dict(session=session)
-            segment_count = document.get_segment_count(session=session)
+            data_source_info = get_document_source_detail(document, session=session)
+            segment_count = get_document_segment_count(document, session=session)
             response = {
                 "id": document.id,
                 "position": document.position,
@@ -1384,10 +1395,10 @@ class DocumentApi(DatasetApiResource):
                 "disabled_by": document.disabled_by,
                 "archived": document.archived,
                 "doc_type": document.doc_type,
-                "doc_metadata": document.get_doc_metadata_details(session=session),
+                "doc_metadata": get_document_metadata_details(document, session=session),
                 "segment_count": segment_count,
                 "average_segment_length": (document.word_count or 0) // segment_count if segment_count else 0,
-                "hit_count": document.get_hit_count(session=session),
+                "hit_count": get_document_hit_count(document, session=session),
                 "display_status": document.display_status,
                 "doc_form": document.doc_form,
                 "doc_language": document.doc_language,

api/controllers/service_api/dataset/metadata.py

@@ -1,4 +1,4 @@
-from typing import Literal, cast
+from typing import Literal
 from uuid import UUID
 
 from flask_login import current_user
@@ -11,6 +11,7 @@ from controllers.common.session import with_session
 from controllers.console.wraps import model_validate
 from controllers.service_api import service_api_ns
 from controllers.service_api.wraps import DatasetApiResource, cloud_edition_billing_rate_limit_check
+from extensions.ext_application_services import application_services
 from fields.dataset_fields import (
     DatasetMetadataActionResponse,
     DatasetMetadataBuiltInFieldsResponse,
@@ -18,16 +19,15 @@ from fields.dataset_fields import (
     DatasetMetadataResponse,
 )
 from libs.helper import dump_response
-from models import Account
-from services.dataset_service import DatasetService
-from services.entities.knowledge_entities.knowledge_entities import (
+from services.errors.metadata import MetadataResourceNotFoundError
+from services.knowledge.dataset_service import DatasetService
+from services.knowledge.entities.knowledge_entities import (
     DocumentMetadataOperation,
     MetadataArgs,
     MetadataDetail,
     MetadataOperationData,
 )
-from services.errors.metadata import MetadataResourceNotFoundError
-from services.metadata_service import MetadataService
+from services.knowledge.resource_scope import DatasetRef
 
 BUILT_IN_METADATA_ACTION_PARAM = {
     "description": "`enable` to activate built-in metadata fields, `disable` to deactivate them.",
@@ -87,12 +87,18 @@ class DatasetMetadataCreateServiceApi(DatasetApiResource):
         """Create metadata for a dataset."""
 
         dataset_id_str = str(dataset_id)
-        dataset = DatasetService.get_dataset(dataset_id_str, session)
+        dataset = DatasetService.get_dataset_for_tenant(dataset_id_str, str(tenant_id), session=session)
         if dataset is None:
             raise NotFound("Dataset not found.")
         DatasetService.check_dataset_permission(dataset, current_user, session)
 
-        metadata = MetadataService.create_metadata(dataset_id_str, metadata_args, session=session)
+        session.commit()
+        try:
+            metadata = application_services().knowledge.metadata.create_metadata(
+                DatasetRef(str(tenant_id), dataset_id_str), metadata_args, actor_id=current_user.id
+            )
+        except MetadataResourceNotFoundError as exc:
+            raise NotFound(str(exc)) from exc
         return dump_response(DatasetMetadataResponse, metadata), 201
 
     @service_api_ns.doc(
@@ -123,10 +129,16 @@ class DatasetMetadataCreateServiceApi(DatasetApiResource):
     def get(self, session: Session, tenant_id, dataset_id: UUID):
         """Get all metadata for a dataset."""
         dataset_id_str = str(dataset_id)
-        dataset = DatasetService.get_dataset(dataset_id_str, session)
+        dataset = DatasetService.get_dataset_for_tenant(dataset_id_str, str(tenant_id), session=session)
         if dataset is None:
             raise NotFound("Dataset not found.")
-        metadata = MetadataService.get_dataset_metadatas(dataset, session)
+        session.commit()
+        try:
+            metadata = application_services().knowledge.metadata.get_dataset_metadatas(
+                DatasetRef(str(tenant_id), dataset_id_str)
+            )
+        except MetadataResourceNotFoundError as exc:
+            raise NotFound(str(exc)) from exc
         return dump_response(DatasetMetadataListResponse, metadata), 200
 
 
@@ -168,9 +180,13 @@ class DatasetMetadataServiceApi(DatasetApiResource):
             raise NotFound("Dataset not found.")
         DatasetService.check_dataset_permission(dataset, current_user, session)
 
-        metadata = MetadataService.update_metadata_name(
-            dataset, metadata_id_str, payload.name, cast(Account, current_user), session=session
-        )
+        session.commit()
+        try:
+            metadata = application_services().knowledge.metadata.update_metadata_name(
+                DatasetRef(str(tenant_id), dataset_id_str), metadata_id_str, payload.name, actor_id=current_user.id
+            )
+        except MetadataResourceNotFoundError as exc:
+            raise NotFound(str(exc)) from exc
         return dump_response(DatasetMetadataResponse, metadata), 200
 
     @service_api_ns.doc(
@@ -206,7 +222,13 @@ class DatasetMetadataServiceApi(DatasetApiResource):
             raise NotFound("Dataset not found.")
         DatasetService.check_dataset_permission(dataset, current_user, session)
 
-        MetadataService.delete_metadata(dataset, metadata_id_str, session)
+        session.commit()
+        try:
+            application_services().knowledge.metadata.delete_metadata(
+                DatasetRef(str(tenant_id), dataset_id_str), metadata_id_str
+            )
+        except MetadataResourceNotFoundError as exc:
+            raise NotFound(str(exc)) from exc
         return "", 204
 
 
@@ -239,7 +261,7 @@ class DatasetMetadataBuiltInFieldServiceApi(DatasetApiResource):
     )
     def get(self, tenant_id, dataset_id: UUID):
         """Get all built-in metadata fields."""
-        built_in_fields = MetadataService.get_built_in_fields()
+        built_in_fields = application_services().knowledge.metadata.get_built_in_fields()
         return dump_response(DatasetMetadataBuiltInFieldsResponse, {"fields": built_in_fields}), 200
 
 
@@ -271,16 +293,24 @@ class DatasetMetadataBuiltInFieldActionServiceApi(DatasetApiResource):
     def post(self, session: Session, tenant_id, dataset_id: UUID, action: Literal["enable", "disable"]):
         """Enable or disable built-in metadata field."""
         dataset_id_str = str(dataset_id)
-        dataset = DatasetService.get_dataset(dataset_id_str, session)
+        dataset = DatasetService.get_dataset_for_tenant(dataset_id_str, str(tenant_id), session=session)
         if dataset is None:
             raise NotFound("Dataset not found.")
         DatasetService.check_dataset_permission(dataset, current_user, session)
 
-        match action:
-            case "enable":
-                MetadataService.enable_built_in_field(dataset, session)
-            case "disable":
-                MetadataService.disable_built_in_field(dataset, session)
+        session.commit()
+        try:
+            match action:
+                case "enable":
+                    application_services().knowledge.metadata.enable_built_in_field(
+                        DatasetRef(str(tenant_id), dataset_id_str)
+                    )
+                case "disable":
+                    application_services().knowledge.metadata.disable_built_in_field(
+                        DatasetRef(str(tenant_id), dataset_id_str)
+                    )
+        except MetadataResourceNotFoundError as exc:
+            raise NotFound(str(exc)) from exc
         return dump_response(DatasetMetadataActionResponse, {"result": "success"}), 200
 
 
@@ -324,9 +354,10 @@ class DocumentMetadataEditServiceApi(DatasetApiResource):
             raise NotFound("Dataset not found.")
         DatasetService.check_dataset_permission(dataset, current_user, session)
 
+        session.commit()
         try:
-            MetadataService.update_documents_metadata(
-                dataset, metadata_args, cast(Account, current_user), session=session
+            application_services().knowledge.metadata.update_documents_metadata(
+                DatasetRef(str(tenant_id), str(dataset_id)), metadata_args, actor_id=current_user.id
             )
         except MetadataResourceNotFoundError as exc:
             raise NotFound(str(exc)) from exc

api/controllers/service_api/dataset/rag_pipeline/rag_pipeline_workflow.py

@@ -32,6 +32,7 @@ from controllers.service_api.wraps import DatasetApiResource
 from core.app.apps.pipeline.pipeline_generator import PipelineGenerator
 from core.app.entities.app_invoke_entities import InvokeFrom
 from core.entities.knowledge_entities import PipelineDataset, PipelineDocument
+from extensions.ext_application_services import application_services
 from fields.base import ResponseModel
 from libs import helper
 from libs.helper import dump_response
@@ -183,7 +184,11 @@ class DatasourcePluginsApi(DatasetApiResource):
 
         rag_pipeline_service = RagPipelineService(db.session())
         datasource_plugins: list[dict[Any, Any]] = rag_pipeline_service.get_datasource_plugins(
-            tenant_id=tenant_id, dataset_id=dataset_id_str, is_published=query.is_published
+            tenant_id=tenant_id,
+            dataset_id=dataset_id_str,
+            is_published=query.is_published,
+            credential_query=application_services().credential_queries,
+            datasource_providers=application_services().data_sources.providers,
         )
         return dump_response(DatasourcePluginListResponse, datasource_plugins), 200
 
@@ -247,6 +252,7 @@ class DatasourceNodeRunApi(DatasetApiResource):
                     datasource_type=datasource_node_run_api_entity.datasource_type,
                     is_published=datasource_node_run_api_entity.is_published,
                     credential_id=datasource_node_run_api_entity.credential_id,
+                    datasource_providers=application_services().data_sources.providers,
                 )
             )
         )
@@ -310,6 +316,7 @@ class PipelineRunApi(DatasetApiResource):
         pipeline = rag_pipeline_service.get_pipeline(tenant_id=tenant_id, dataset_id=dataset_id_str)
         try:
             response: dict[Any, Any] | Generator[str, Any, None] = PipelineGenerateService.generate(
+                generator=application_services().knowledge.pipeline_generator,
                 session=session,
                 pipeline=pipeline,
                 user=current_user,

api/controllers/service_api/dataset/segment.py

@@ -27,26 +27,26 @@ from controllers.service_api.wraps import (
 from core.errors.error import LLMBadRequestError, ProviderTokenNotInitError
 from core.model_manager import ModelManager
 from core.rag.index_processor.constant.index_type import IndexTechniqueType
+from extensions.ext_application_services import application_services
 from fields.base import ResponseModel
 from fields.segment_fields import (
     ChildChunkDetailResponse,
     ChildChunkListResponse,
     SegmentDetailResponse,
     SegmentResponse,
-    segment_response_with_summary,
-    segment_responses_with_summaries,
 )
 from graphon.model_runtime.entities.model_entities import ModelType
 from libs.helper import dump_response
 from libs.login import current_account_with_tenant
 from models.dataset import Dataset, Document, DocumentSegment
-from services.dataset_ref_service import DatasetRefService, SegmentRef
-from services.dataset_service import DatasetService, DocumentService, SegmentService
-from services.entities.knowledge_entities.knowledge_entities import SegmentUpdateArgs
 from services.errors.chunk import ChildChunkDeleteIndexError, ChildChunkIndexingError
 from services.errors.chunk import ChildChunkDeleteIndexError as ChildChunkDeleteIndexServiceError
 from services.errors.chunk import ChildChunkIndexingError as ChildChunkIndexingServiceError
-from services.summary_index_service import SummaryIndexService
+from services.knowledge.dataset_read_service import load_segment_detail, load_segment_details
+from services.knowledge.dataset_service import DatasetService, DocumentService, SegmentService
+from services.knowledge.entities.segments import SegmentUpdateArgs
+from services.knowledge.resource_scope import DatasetRef, SegmentRef
+from services.knowledge.summaries.adapters import SummaryIndexAdapter
 
 
 class SegmentCreateItemPayload(BaseModel):
@@ -133,12 +133,11 @@ register_response_schema_models(
 def _get_segment_for_document(
     session: Session, dataset: Dataset, document: Document, segment_id: str
 ) -> tuple[SegmentRef, DocumentSegment]:
-    dataset_ref = DatasetRefService.create_dataset_ref(dataset)
-    document_ref = DatasetRefService.create_document_ref(dataset_ref, document)
-    if document_ref is None:
+    if document.tenant_id != dataset.tenant_id or document.dataset_id != dataset.id:
         raise NotFound("Document not found.")
 
-    segment_ref = DatasetRefService.create_segment_ref(document_ref, segment_id)
+    dataset_ref = DatasetRef(tenant_id=dataset.tenant_id, dataset_id=dataset.id)
+    segment_ref = dataset_ref.document(document.id).segment(segment_id)
     segment = SegmentService.get_segment_by_ref(segment_ref, session=session)
     if not segment:
         raise NotFound("Segment not found.")
@@ -230,17 +229,24 @@ class SegmentApi(DatasetApiResource):
         for args_item in segment_items:
             SegmentService.segment_create_args_validate(args_item, document)
         segments = cast(
-            list[DocumentSegment], SegmentService.multi_create_segment(segment_items, document, dataset, session)
+            list[DocumentSegment],
+            SegmentService.multi_create_segment(
+                segment_items,
+                document,
+                dataset,
+                session,
+                mutations=application_services().knowledge.segments.mutations,
+            ),
         )
         segment_ids = [segment.id for segment in segments]
         summaries: dict[str, str | None] = {}
         if segment_ids:
-            summary_records = SummaryIndexService.get_segments_summaries(
+            summary_records = SummaryIndexAdapter.get_segments_summaries(
                 segment_ids=segment_ids, dataset_id=dataset_id_str, session=session
             )
             summaries = {chunk_id: record.summary_content for chunk_id, record in summary_records.items()}
         response = {
-            "data": segment_responses_with_summaries(segments, summaries, session=session),
+            "data": load_segment_details(segments, summaries, session=session),
             "doc_form": document.doc_form,
         }
         return dump_response(SegmentCreateListResponse, response), 200
@@ -322,13 +328,13 @@ class SegmentApi(DatasetApiResource):
         segment_ids = [segment.id for segment in segments]
         summaries: dict[str, str | None] = {}
         if segment_ids:
-            summary_records = SummaryIndexService.get_segments_summaries(
+            summary_records = SummaryIndexAdapter.get_segments_summaries(
                 segment_ids=segment_ids, dataset_id=dataset_id_str, session=session
             )
             summaries = {chunk_id: record.summary_content for chunk_id, record in summary_records.items()}
 
         response = {
-            "data": segment_responses_with_summaries(segments, summaries, session=session),
+            "data": load_segment_details(segments, summaries, session=session),
             "doc_form": document.doc_form,
             "total": total,
             "has_more": page * limit < total,
@@ -380,7 +386,9 @@ class DatasetSegmentApi(DatasetApiResource):
             raise NotFound("Document not found.")
         segment_id_str = str(segment_id)
         _, segment = _get_segment_for_document(session, dataset, document, segment_id_str)
-        SegmentService.delete_segment(segment, document, dataset, session)
+        SegmentService.delete_segment(
+            segment, document, dataset, session, mutations=application_services().knowledge.segments.mutations
+        )
         return "", 204
 
     @service_api_ns.doc(
@@ -417,7 +425,7 @@ class DatasetSegmentApi(DatasetApiResource):
         document_id: UUID,
         segment_id: UUID,
     ):
-        _, current_tenant_id = current_account_with_tenant()
+        current_account, current_tenant_id = current_account_with_tenant()
         dataset_id_str = str(dataset_id)
         # check dataset
         dataset = session.scalar(
@@ -451,14 +459,20 @@ class DatasetSegmentApi(DatasetApiResource):
         segment_id_str = str(segment_id)
         _, segment = _get_segment_for_document(session, dataset, document, segment_id_str)
 
-        updated_segment = SegmentService.update_segment(payload.segment, segment, document, dataset, session)
-        summary = SummaryIndexService.get_segment_summary(
+        updated_segment = SegmentService.update_segment(
+            payload.segment,
+            segment,
+            document,
+            dataset,
+            session,
+            actor_id=current_account.id,
+            mutations=application_services().knowledge.segments.mutations,
+        )
+        summary = SummaryIndexAdapter.get_segment_summary(
             segment_id=updated_segment.id, dataset_id=dataset_id_str, session=session
         )
         response = {
-            "data": segment_response_with_summary(
-                updated_segment, summary.summary_content if summary else None, session=session
-            ),
+            "data": load_segment_detail(updated_segment, summary.summary_content if summary else None, session=session),
             "doc_form": document.doc_form,
         }
         return dump_response(SegmentDetailResponse, response), 200
@@ -510,13 +524,11 @@ class DatasetSegmentApi(DatasetApiResource):
         segment_id_str = str(segment_id)
         _, segment = _get_segment_for_document(session, dataset, document, segment_id_str)
 
-        summary = SummaryIndexService.get_segment_summary(
+        summary = SummaryIndexAdapter.get_segment_summary(
             segment_id=segment.id, dataset_id=dataset_id_str, session=session
         )
         response = {
-            "data": segment_response_with_summary(
-                segment, summary.summary_content if summary else None, session=session
-            ),
+            "data": load_segment_detail(segment, summary.summary_content if summary else None, session=session),
             "doc_form": document.doc_form,
         }
         return dump_response(SegmentDetailResponse, response), 200
@@ -567,7 +579,7 @@ class ChildChunkApi(DatasetApiResource):
         document_id: UUID,
         segment_id: UUID,
     ):
-        _, current_tenant_id = current_account_with_tenant()
+        current_account, current_tenant_id = current_account_with_tenant()
         """Create child chunk."""
         dataset_id_str = str(dataset_id)
         # check dataset
@@ -604,7 +616,15 @@ class ChildChunkApi(DatasetApiResource):
                 raise ProviderNotInitializeError(ex.description)
 
         try:
-            child_chunk = SegmentService.create_child_chunk(payload.content, segment, document, dataset, session)
+            child_chunk = SegmentService.create_child_chunk(
+                payload.content,
+                segment,
+                document,
+                dataset,
+                session,
+                actor_id=current_account.id,
+                mutations=application_services().knowledge.segments.mutations,
+            )
         except ChildChunkIndexingServiceError as e:
             raise ChildChunkIndexingError(str(e))
 
@@ -744,7 +764,9 @@ class DatasetChildChunkApi(DatasetApiResource):
             raise NotFound("Child chunk not found.")
 
         try:
-            SegmentService.delete_child_chunk(child_chunk, dataset, session)
+            SegmentService.delete_child_chunk(
+                child_chunk, dataset, session, mutations=application_services().knowledge.segments.mutations
+            )
         except ChildChunkDeleteIndexServiceError as e:
             raise ChildChunkDeleteIndexError(str(e))
 
@@ -790,7 +812,7 @@ class DatasetChildChunkApi(DatasetApiResource):
         segment_id: UUID,
         child_chunk_id: UUID,
     ):
-        current_account_with_tenant()
+        current_account, _ = current_account_with_tenant()
         """Update child chunk."""
         dataset_id_str = str(dataset_id)
         # check dataset
@@ -817,7 +839,14 @@ class DatasetChildChunkApi(DatasetApiResource):
 
         try:
             child_chunk = SegmentService.update_child_chunk(
-                payload.content, child_chunk, segment, document, dataset, session
+                payload.content,
+                child_chunk,
+                segment,
+                document,
+                dataset,
+                session,
+                actor_id=current_account.id,
+                mutations=application_services().knowledge.segments.mutations,
             )
         except ChildChunkIndexingServiceError as e:
             raise ChildChunkIndexingError(str(e))

api/controllers/service_api/workspace/models.py

@@ -7,7 +7,7 @@ from controllers.service_api.wraps import validate_dataset_token
 from fields.base import ResponseModel
 from graphon.model_runtime.utils.encoders import jsonable_encoder
 from services.entities.model_provider_entities import ProviderWithModelsResponse
-from services.model_provider_service import ModelProviderService
+from services.model_provider.service import ModelProviderService
 
 MODEL_TYPE_PARAM = {
     "description": "Type of model to retrieve.",

api/core/app/app_config/easy_ui_based_app/dataset/manager.py

@@ -12,7 +12,7 @@ from core.app.app_config.entities import (
 from core.entities.agent_entities import PlanningStrategy
 from core.rag.data_post_processor.data_post_processor import RerankingModelDict, WeightsDict
 from models.model import AppMode, AppModelConfigDict
-from services.dataset_service import DatasetService
+from services.knowledge.dataset_service import DatasetService
 
 
 class DatasetConfigManager:

api/core/app/apps/agent_chat/app_config_manager.py

@@ -62,6 +62,8 @@ class AgentChatAppConfigManager(BaseAppConfigManager):
             config_from = EasyUIBasedAppModelConfigFrom.APP_LATEST_CONFIG
 
         if config_from != EasyUIBasedAppModelConfigFrom.ARGS:
+            if annotation_reply is None:
+                raise ValueError("Annotation reply config is required")
             app_model_config_dict = app_model_config.to_dict(annotation_reply=annotation_reply)
             config_dict = app_model_config_dict.copy()
         else:

api/core/app/apps/agent_chat/app_generator.py

@@ -138,9 +138,12 @@ class AgentChatAppGenerator(MessageBasedAppGenerator):
             # always enable retriever resource in debugger mode
             override_model_config_dict["retriever_resource"] = {"enabled": True}
 
-        annotation_reply = (
-            None if override_model_config_dict else load_annotation_reply_config(session, app_model_config.app_id)
-        )
+        if override_model_config_dict:
+            annotation_reply = None
+            effective_model_config_dict = override_model_config_dict
+        else:
+            annotation_reply = load_annotation_reply_config(session, app_model_config.app_id)
+            effective_model_config_dict = app_model_config.to_dict(annotation_reply=annotation_reply)
 
         # parse files
         # TODO(QuantumGhost): Move file parsing logic to the API controller layer
@@ -150,9 +153,7 @@ class AgentChatAppGenerator(MessageBasedAppGenerator):
         # `DraftWorkflowNodeRunApi` class which handle this properly.
         with self._bind_file_access_scope(tenant_id=app_model.tenant_id, user=user, invoke_from=invoke_from):
             files = args.get("files") or []
-            file_extra_config = FileUploadConfigManager.convert(
-                override_model_config_dict or app_model_config.to_dict(annotation_reply=annotation_reply)
-            )
+            file_extra_config = FileUploadConfigManager.convert(effective_model_config_dict)
             if file_extra_config:
                 file_objs = file_factory.build_from_mappings(
                     mappings=files,

api/core/app/apps/base_app_generate_response_converter.py

@@ -25,7 +25,7 @@ class AppGenerateResponseConverter[TBlockingResponse: AppBlockingResponse](ABC):
     def convert(
         cls, response: Union[AppBlockingResponse, Generator[AppStreamResponse, Any, None]], invoke_from: InvokeFrom
     ) -> Mapping[str, Any] | Generator[str | Mapping[str, Any], Any, None]:
-        if invoke_from in {InvokeFrom.DEBUGGER, InvokeFrom.SERVICE_API}:
+        if invoke_from in {InvokeFrom.DEBUGGER, InvokeFrom.SERVICE_API, InvokeFrom.OPENAPI}:
             if isinstance(response, AppBlockingResponse):
                 return cls.convert_blocking_full_response(cls._cast_blocking_response(response))
             else:

api/core/app/apps/chat/app_config_manager.py

@@ -55,6 +55,8 @@ class ChatAppConfigManager(BaseAppConfigManager):
             config_from = EasyUIBasedAppModelConfigFrom.APP_LATEST_CONFIG
 
         if config_from != EasyUIBasedAppModelConfigFrom.ARGS:
+            if annotation_reply is None:
+                raise ValueError("Annotation reply config is required")
             app_model_config_dict = app_model_config.to_dict(annotation_reply=annotation_reply)
             config_dict = app_model_config_dict.copy()
         else:

api/core/app/apps/chat/app_generator.py

@@ -133,9 +133,12 @@ class ChatAppGenerator(MessageBasedAppGenerator):
             # always enable retriever resource in debugger mode
             override_model_config_dict["retriever_resource"] = {"enabled": True}
 
-        annotation_reply = (
-            None if override_model_config_dict else load_annotation_reply_config(session, app_model_config.app_id)
-        )
+        if override_model_config_dict:
+            annotation_reply = None
+            effective_model_config_dict = override_model_config_dict
+        else:
+            annotation_reply = load_annotation_reply_config(session, app_model_config.app_id)
+            effective_model_config_dict = app_model_config.to_dict(annotation_reply=annotation_reply)
 
         # parse files
         # TODO(QuantumGhost): Move file parsing logic to the API controller layer
@@ -145,9 +148,7 @@ class ChatAppGenerator(MessageBasedAppGenerator):
         # `DraftWorkflowNodeRunApi` class which handle this properly.
         with self._bind_file_access_scope(tenant_id=app_model.tenant_id, user=user, invoke_from=invoke_from):
             files = args["files"] if args.get("files") else []
-            file_extra_config = FileUploadConfigManager.convert(
-                override_model_config_dict or app_model_config.to_dict(annotation_reply=annotation_reply)
-            )
+            file_extra_config = FileUploadConfigManager.convert(effective_model_config_dict)
             if file_extra_config:
                 file_objs = file_factory.build_from_mappings(
                     mappings=files,

api/core/app/apps/completion/app_config_manager.py

@@ -46,6 +46,8 @@ class CompletionAppConfigManager(BaseAppConfigManager):
             config_from = EasyUIBasedAppModelConfigFrom.APP_LATEST_CONFIG
 
         if config_from != EasyUIBasedAppModelConfigFrom.ARGS:
+            if annotation_reply is None:
+                raise ValueError("Annotation reply config is required")
             app_model_config_dict = app_model_config.to_dict(annotation_reply=annotation_reply)
             config_dict = app_model_config_dict.copy()
         else:

api/core/app/apps/completion/app_generator.py

@@ -119,9 +119,12 @@ class CompletionAppGenerator(MessageBasedAppGenerator):
                 tenant_id=app_model.tenant_id, config=args.get("model_config", {}), session=session
             )
 
-        annotation_reply = (
-            None if override_model_config_dict else load_annotation_reply_config(session, app_model_config.app_id)
-        )
+        if override_model_config_dict:
+            annotation_reply = None
+            effective_model_config_dict = override_model_config_dict
+        else:
+            annotation_reply = load_annotation_reply_config(session, app_model_config.app_id)
+            effective_model_config_dict = app_model_config.to_dict(annotation_reply=annotation_reply)
 
         # parse files
         # TODO(QuantumGhost): Move file parsing logic to the API controller layer
@@ -131,9 +134,7 @@ class CompletionAppGenerator(MessageBasedAppGenerator):
         # `DraftWorkflowNodeRunApi` class which handle this properly.
         with self._bind_file_access_scope(tenant_id=app_model.tenant_id, user=user, invoke_from=invoke_from):
             files = args["files"] if args.get("files") else []
-            file_extra_config = FileUploadConfigManager.convert(
-                override_model_config_dict or app_model_config.to_dict(annotation_reply=annotation_reply)
-            )
+            file_extra_config = FileUploadConfigManager.convert(effective_model_config_dict)
             if file_extra_config:
                 file_objs = file_factory.build_from_mappings(
                     mappings=files,

api/core/app/apps/pipeline/pipeline_generator.py

@@ -22,7 +22,7 @@ from core.app.apps.draft_variable_saver import DraftVariableSaverFactory
 from core.app.apps.exc import GenerateTaskStoppedError
 from core.app.apps.pipeline.pipeline_config_manager import PipelineConfigManager
 from core.app.apps.pipeline.pipeline_queue_manager import PipelineQueueManager
-from core.app.apps.pipeline.pipeline_runner import PipelineRunner
+from core.app.apps.pipeline.pipeline_runner import PipelineDocumentStore, PipelineRunner
 from core.app.apps.workflow.generate_response_converter import WorkflowAppGenerateResponseConverter
 from core.app.apps.workflow.generate_task_pipeline import WorkflowAppGenerateTaskPipeline
 from core.app.entities.app_invoke_entities import InvokeFrom, RagPipelineGenerateEntity
@@ -52,7 +52,7 @@ from models import Account, EndUser, Workflow, WorkflowNodeExecutionTriggeredFro
 from models.dataset import Document, DocumentPipelineExecutionLog, Pipeline
 from models.enums import WorkflowRunTriggeredFrom
 from models.model import AppMode
-from services.datasource_provider_service import DatasourceProviderService
+from services.data_source.provider_service import DatasourceProviderService
 from services.rag_pipeline.rag_pipeline_task_proxy import RagPipelineTaskProxy
 from services.workflow_draft_variable_service import DraftVarLoader, WorkflowDraftVariableService
 
@@ -60,6 +60,10 @@ logger = logging.getLogger(__name__)
 
 
 class PipelineGenerator(BaseAppGenerator):
+    def __init__(self, *, documents: PipelineDocumentStore, datasource_providers: DatasourceProviderService) -> None:
+        self._documents = documents
+        self._datasource_providers = datasource_providers
+
     @overload
     def generate(
         self,
@@ -124,7 +128,7 @@ class PipelineGenerator(BaseAppGenerator):
     ) -> Mapping[str, Any] | Generator[Mapping | str, None, None] | None:
         # Add null check for dataset
 
-        dataset = pipeline.retrieve_dataset(session)
+        dataset = self._documents.get_pipeline_dataset(pipeline, session=session)
         if not dataset:
             raise ValueError("Pipeline dataset is required")
         inputs: Mapping[str, Any] = args["inputs"]
@@ -140,8 +144,8 @@ class PipelineGenerator(BaseAppGenerator):
         )
         documents: list[Document] = []
         if invoke_from == InvokeFrom.PUBLISHED_PIPELINE and not is_retry and not args.get("original_document_id"):
-            from services.dataset_service import DocumentService
             from services.feature_service import FeatureService
+            from services.knowledge.dataset_service import DocumentService
 
             features = FeatureService.get_features(pipeline.tenant_id)
             DocumentService.check_document_creation_limits(len(datasource_info_list), features)
@@ -407,7 +411,7 @@ class PipelineGenerator(BaseAppGenerator):
             pipeline=pipeline, workflow=workflow, start_node_id=args.get("start_node_id", "shared")
         )
 
-        dataset = pipeline.retrieve_dataset(session)
+        dataset = self._documents.get_pipeline_dataset(pipeline, session=session)
         if not dataset:
             raise ValueError("Pipeline dataset is required")
 
@@ -504,7 +508,7 @@ class PipelineGenerator(BaseAppGenerator):
         if args.get("inputs") is None:
             raise ValueError("inputs is required")
 
-        dataset = pipeline.retrieve_dataset(session)
+        dataset = self._documents.get_pipeline_dataset(pipeline, session=session)
         if not dataset:
             raise ValueError("Pipeline dataset is required")
 
@@ -635,6 +639,7 @@ class PipelineGenerator(BaseAppGenerator):
                         system_user_id=system_user_id,
                         workflow_execution_repository=workflow_execution_repository,
                         workflow_node_execution_repository=workflow_node_execution_repository,
+                        documents=self._documents,
                     )
 
                     runner.run()
@@ -779,8 +784,7 @@ class PipelineGenerator(BaseAppGenerator):
                 tenant_id=pipeline.tenant_id,
                 datasource_type=DatasourceProviderType(datasource_type),
             )
-            datasource_provider_service = DatasourceProviderService()
-            credentials = datasource_provider_service.get_datasource_credentials(
+            credentials = self._datasource_providers.get_datasource_credentials(
                 tenant_id=pipeline.tenant_id,
                 provider=datasource_node_data.get("provider_name"),
                 plugin_id=datasource_node_data.get("plugin_id"),

api/core/app/apps/pipeline/pipeline_runner.py

@@ -1,6 +1,6 @@
 import logging
 import time
-from typing import cast
+from typing import Protocol, cast
 
 from sqlalchemy import select
 from sqlalchemy.orm import Session
@@ -28,14 +28,23 @@ from graphon.graph_events import GraphEngineEvent, GraphRunFailedEvent
 from graphon.runtime import GraphRuntimeState, VariablePool
 from graphon.variable_loader import VariableLoader
 from graphon.variables.variables import RAGPipelineVariable, RAGPipelineVariableInput
-from models.dataset import Pipeline
+from models.dataset import Dataset, Pipeline
 from models.model import EndUser
 from models.workflow import Workflow
-from services.dataset_ref_service import DatasetRefService, DocumentRef
 
 logger = logging.getLogger(__name__)
 
 
+class PipelineDocumentStore(Protocol):
+    """Knowledge state required by a pipeline run, with explicit session ownership."""
+
+    def get_pipeline_dataset(self, pipeline: Pipeline, *, session: Session) -> Dataset | None: ...
+
+    def exists(self, *, workspace_id: str, dataset_id: str, document_id: str) -> bool: ...
+
+    def mark_failed(self, *, workspace_id: str, dataset_id: str, document_id: str, error: str) -> None: ...
+
+
 class PipelineRunner(WorkflowBasedAppRunner):
     """
     Pipeline Application Runner
@@ -50,6 +59,7 @@ class PipelineRunner(WorkflowBasedAppRunner):
         system_user_id: str,
         workflow_execution_repository: WorkflowExecutionRepository,
         workflow_node_execution_repository: WorkflowNodeExecutionRepository,
+        documents: PipelineDocumentStore,
         workflow_thread_pool_id: str | None = None,
     ) -> None:
         """
@@ -68,6 +78,7 @@ class PipelineRunner(WorkflowBasedAppRunner):
         self._sys_user_id = system_user_id
         self._workflow_execution_repository = workflow_execution_repository
         self._workflow_node_execution_repository = workflow_node_execution_repository
+        self._documents = documents
 
     def _get_app_id(self) -> str:
         return self.application_generate_entity.app_config.app_id
@@ -98,7 +109,7 @@ class PipelineRunner(WorkflowBasedAppRunner):
             if not pipeline or pipeline.tenant_id != app_config.tenant_id:
                 raise ValueError("Pipeline not found")
 
-            dataset = pipeline.retrieve_dataset(session)
+            dataset = self._documents.get_pipeline_dataset(pipeline, session=session)
             if (
                 not dataset
                 or dataset.tenant_id != pipeline.tenant_id
@@ -108,23 +119,20 @@ class PipelineRunner(WorkflowBasedAppRunner):
 
             document_id = self.application_generate_entity.document_id
             original_document_id = self.application_generate_entity.original_document_id
-            dataset_ref = DatasetRefService.create_dataset_ref(dataset)
-            document_ref = (
-                DatasetRefService.create_document_ref_from_id(
-                    dataset_ref,
-                    document_id,
-                )
-                if document_id
-                else None
-            )
-            if document_ref and DatasetRefService.get_document_by_ref(document_ref, session=session) is None:
+            dataset_workspace_id = dataset.tenant_id
+            dataset_id = dataset.id
+            if document_id and not self._documents.exists(
+                workspace_id=dataset_workspace_id,
+                dataset_id=dataset_id,
+                document_id=document_id,
+            ):
                 raise ValueError("Pipeline document not found")
             if original_document_id and original_document_id != document_id:
-                original_document_ref = DatasetRefService.create_document_ref_from_id(
-                    dataset_ref,
-                    original_document_id,
-                )
-                if DatasetRefService.get_document_by_ref(original_document_ref, session=session) is None:
+                if not self._documents.exists(
+                    workspace_id=dataset_workspace_id,
+                    dataset_id=dataset_id,
+                    document_id=original_document_id,
+                ):
                     raise ValueError("Pipeline original document not found")
 
             workflow = self.get_workflow(session=session, pipeline=pipeline, workflow_id=app_config.workflow_id)
@@ -237,7 +245,12 @@ class PipelineRunner(WorkflowBasedAppRunner):
         generator = workflow_entry.run()
 
         for event in generator:
-            self._update_document_status(event, document_ref)
+            self._update_document_status(
+                event,
+                workspace_id=dataset_workspace_id,
+                dataset_id=dataset_id,
+                document_id=document_id,
+            )
             self._handle_event(workflow_entry, event)
 
     def get_workflow(self, session: Session, pipeline: Pipeline, workflow_id: str) -> Workflow | None:
@@ -325,14 +338,21 @@ class PipelineRunner(WorkflowBasedAppRunner):
 
         return graph
 
-    def _update_document_status(self, event: GraphEngineEvent, document_ref: DocumentRef | None) -> None:
+    def _update_document_status(
+        self,
+        event: GraphEngineEvent,
+        *,
+        workspace_id: str,
+        dataset_id: str,
+        document_id: str | None,
+    ) -> None:
         """Set an owner-bound document to error after a failed graph run, if it exists."""
-        if not isinstance(event, GraphRunFailedEvent) or document_ref is None:
+        if not isinstance(event, GraphRunFailedEvent) or document_id is None:
             return
 
-        with create_session() as session, session.begin():
-            document = DatasetRefService.get_document_by_ref(document_ref, session=session)
-            if document:
-                document.indexing_status = "error"
-                document.error = event.error or "Unknown error"
-                session.add(document)
+        self._documents.mark_failed(
+            workspace_id=workspace_id,
+            dataset_id=dataset_id,
+            document_id=document_id,
+            error=event.error or "Unknown error",
+        )

api/core/app/features/annotation_reply/annotation_reply.py

@@ -10,7 +10,7 @@ from models.dataset import Dataset
 from models.enums import CollectionBindingType, ConversationFromSource
 from models.model import AnnotationReplyEnabledConfig, App, Message, MessageAnnotation, load_annotation_reply_config
 from services.annotation_service import AppAnnotationService
-from services.dataset_service import DatasetCollectionBindingService
+from services.knowledge.dataset_service import DatasetCollectionBindingService
 
 logger = logging.getLogger(__name__)
 

api/core/datasource/datasource_manager.py

@@ -1,5 +1,5 @@
 import logging
-from collections.abc import Generator
+from collections.abc import Generator, Mapping
 from threading import Lock
 from typing import Any, cast
 
@@ -33,7 +33,6 @@ from graphon.file import File, FileTransferMethod, FileType, get_file_type_by_mi
 from graphon.node_events import NodeRunResult, StreamChunkEvent, StreamCompletedEvent
 from models.model import UploadFile
 from models.tools import ToolFile
-from services.datasource_provider_service import DatasourceProviderService
 
 logger = logging.getLogger(__name__)
 _file_access_controller = DatabaseFileAccessController()
@@ -147,9 +146,7 @@ class DatasourceManager:
         datasource_type: str,
         provider_id: str,
         tenant_id: str,
-        provider: str,
-        plugin_id: str,
-        credential_id: str,
+        credentials: Mapping[str, Any],
         datasource_param: DatasourceParameter | None = None,
         online_drive_request: OnlineDriveDownloadFileParam | None = None,
     ) -> Generator[DatasourceMessage, None, Any]:
@@ -166,18 +163,10 @@ class DatasourceManager:
             datasource_type=ds_type,
         )
 
-        dsp_service = DatasourceProviderService()
-        credentials = dsp_service.get_datasource_credentials(
-            tenant_id=tenant_id,
-            provider=provider,
-            plugin_id=plugin_id,
-            credential_id=credential_id,
-        )
-
         if ds_type == DatasourceProviderType.ONLINE_DOCUMENT:
             doc_runtime = cast(OnlineDocumentDatasourcePlugin, runtime)
             if credentials:
-                doc_runtime.runtime.credentials = credentials
+                doc_runtime.runtime.credentials = dict(credentials)
             if datasource_param is None:
                 raise ValueError("datasource_param is required for ONLINE_DOCUMENT streaming")
             inner_gen: Generator[DatasourceMessage, None, None] = doc_runtime.get_online_document_page_content(
@@ -192,7 +181,7 @@ class DatasourceManager:
         elif ds_type == DatasourceProviderType.ONLINE_DRIVE:
             drive_runtime = cast(OnlineDriveDatasourcePlugin, runtime)
             if credentials:
-                drive_runtime.runtime.credentials = credentials
+                drive_runtime.runtime.credentials = dict(credentials)
             if online_drive_request is None:
                 raise ValueError("online_drive_request is required for ONLINE_DRIVE streaming")
             inner_gen = drive_runtime.online_drive_download_file(
@@ -221,9 +210,7 @@ class DatasourceManager:
         datasource_type: str,
         provider_id: str,
         tenant_id: str,
-        provider: str,
-        plugin_id: str,
-        credential_id: str,
+        credentials: Mapping[str, Any],
         parameters_for_log: dict[str, Any],
         datasource_info: dict[str, Any],
         variable_pool: Any,
@@ -238,9 +225,7 @@ class DatasourceManager:
             datasource_type=datasource_type,
             provider_id=provider_id,
             tenant_id=tenant_id,
-            provider=provider,
-            plugin_id=plugin_id,
-            credential_id=credential_id,
+            credentials=credentials,
             datasource_param=datasource_param,
             online_drive_request=online_drive_request,
         )

api/core/indexing_runner.py

@@ -1,938 +0,0 @@
-import concurrent.futures
-import json
-import logging
-import re
-import threading
-import time
-import uuid
-from collections.abc import Mapping
-from typing import Any
-
-from flask import Flask, current_app
-from sqlalchemy import delete, func, select, update
-from sqlalchemy.orm import Session
-from sqlalchemy.orm.exc import ObjectDeletedError
-
-from configs import dify_config
-from core.credit_usage import CreditUsageCreatedBy
-from core.db.session_factory import session_factory
-from core.entities.knowledge_entities import IndexingEstimate, PreviewDetail, QAPreviewDetail
-from core.errors.error import ProviderTokenNotInitError
-from core.model_context import with_credit_usage_created_by
-from core.model_manager import ModelInstance, ModelManager
-from core.rag.cleaner.clean_processor import CleanProcessor
-from core.rag.datasource.keyword.keyword_factory import Keyword
-from core.rag.docstore.dataset_docstore import DatasetDocumentStore
-from core.rag.embedding.token_counter import calculate_segment_token_counts
-from core.rag.extractor.entity.datasource_type import DatasourceType
-from core.rag.extractor.entity.extract_setting import ExtractSetting, NotionInfo, WebsiteInfo
-from core.rag.index_processor.constant.index_type import IndexStructureType, IndexTechniqueType
-from core.rag.index_processor.index_processor_base import BaseIndexProcessor
-from core.rag.index_processor.index_processor_factory import IndexProcessorFactory
-from core.rag.models.document import ChildDocument, Document
-from core.rag.splitter.fixed_text_splitter import (
-    EnhanceRecursiveCharacterTextSplitter,
-    FixedRecursiveCharacterTextSplitter,
-)
-from core.rag.splitter.text_splitter import TextSplitter
-from core.tools.utils.web_reader_tool import get_image_upload_file_ids
-from enums import DeploymentEdition
-from extensions.ext_redis import redis_client
-from extensions.ext_storage import storage
-from extensions.otel import propagate_context
-from graphon.model_runtime.entities.model_entities import ModelType
-from libs import helper
-from libs.datetime_utils import naive_utc_now
-from models import Account
-from models.dataset import AutomaticRulesConfig, ChildChunk, Dataset, DatasetProcessRule, DocumentSegment
-from models.dataset import Document as DatasetDocument
-from models.enums import DataSourceType, IndexingStatus, ProcessRuleMode, SegmentStatus
-from models.model import UploadFile
-from services.vector_space_admission_service import VectorSpaceAdmissionService
-
-logger = logging.getLogger(__name__)
-
-
-class IndexingRunner:
-    def __init__(
-        self,
-        *,
-        enforce_vector_space_admission: bool = False,
-    ):
-        self.storage = storage
-        self.enforce_vector_space_admission = enforce_vector_space_admission
-
-    @staticmethod
-    def _get_model_manager(tenant_id: str) -> ModelManager:
-        return ModelManager.for_tenant(tenant_id=tenant_id)
-
-    def _handle_indexing_error(self, document_id: str, error: Exception, session: Session) -> None:
-        """Handle indexing errors by updating document status."""
-        logger.exception("consume document failed")
-        document = session.get(DatasetDocument, document_id)
-        if document:
-            document.indexing_status = IndexingStatus.ERROR
-            error_message = getattr(error, "description", str(error))
-            document.error = str(error_message)
-            document.stopped_at = naive_utc_now()
-            session.flush()
-
-    @with_credit_usage_created_by(CreditUsageCreatedBy.KNOWLEDGE_INDEXING)
-    def run(self, dataset_documents: list[DatasetDocument], session: Session):
-        """Run indexing with commits before slow transforms and parallel index workers.
-
-        The phase commits keep document locks short and make newly created segments
-        visible to the worker sessions used for keyword and vector indexing.
-        """
-        vector_space_admission = VectorSpaceAdmissionService()
-        for dataset_document in dataset_documents:
-            document_id = dataset_document.id
-            try:
-                # Re-query the document to ensure it's bound to the current session
-                requeried_document = session.get(DatasetDocument, document_id)
-                if not requeried_document:
-                    logger.warning("Document not found, skipping document id: %s", document_id)
-                    continue
-
-                # get dataset
-                dataset = session.get(Dataset, requeried_document.dataset_id)
-
-                if not dataset:
-                    raise ValueError("no dataset found")
-                # get the process rule
-                stmt = select(DatasetProcessRule).where(
-                    DatasetProcessRule.id == requeried_document.dataset_process_rule_id
-                )
-                processing_rule = session.scalar(stmt)
-                if not processing_rule:
-                    raise ValueError("no process rule found")
-                index_type = requeried_document.doc_form
-                index_processor = IndexProcessorFactory(index_type).init_index_processor()
-                # extract
-                text_docs = self._extract(index_processor, requeried_document, processing_rule.to_dict(), session)
-                session.commit()
-
-                # transform
-                current_user = session.get(Account, requeried_document.created_by)
-                if not current_user:
-                    raise ValueError("no current user found")
-                current_user.set_tenant_id_with_session(dataset.tenant_id, session=session)
-                documents = self._transform(
-                    index_processor,
-                    dataset,
-                    text_docs,
-                    requeried_document.doc_language,
-                    processing_rule.to_dict(),
-                    current_user=current_user,
-                    session=session,
-                )
-                if self.enforce_vector_space_admission:
-                    vector_space_admission.ensure_document_can_be_indexed(
-                        dataset=dataset,
-                        document_id=requeried_document.id,
-                        doc_form=requeried_document.doc_form,
-                        documents=documents,
-                        include_summaries=bool(requeried_document.need_summary),
-                        session=session,
-                    )
-                token_counts = calculate_segment_token_counts(dataset=dataset, documents=documents)
-                total_tokens = sum(token_counts)
-                # save segment
-                self._load_segments(
-                    session=session,
-                    dataset=dataset,
-                    dataset_document=requeried_document,
-                    documents=documents,
-                    token_counts=token_counts,
-                )
-                session.commit()
-
-                # load
-                self._load(
-                    session=session,
-                    dataset=dataset,
-                    dataset_document=requeried_document,
-                    documents=documents,
-                    total_tokens=total_tokens,
-                )
-            except DocumentIsPausedError:
-                raise DocumentIsPausedError(f"Document paused, document id: {document_id}")
-            except ProviderTokenNotInitError as e:
-                self._handle_indexing_error(document_id, e, session)
-            except ObjectDeletedError:
-                logger.warning("Document deleted, document id: %s", document_id)
-            except Exception as e:
-                self._handle_indexing_error(document_id, e, session)
-
-    @with_credit_usage_created_by(CreditUsageCreatedBy.KNOWLEDGE_INDEXING)
-    def run_in_splitting_status(self, dataset_document: DatasetDocument, session: Session):
-        """Run the indexing process when the index_status is splitting."""
-        document_id = dataset_document.id
-        try:
-            # Re-query the document to ensure it's bound to the current session
-            requeried_document = session.get(DatasetDocument, document_id)
-            if not requeried_document:
-                logger.warning("Document not found: %s", document_id)
-                return
-
-            # get dataset
-            dataset = session.get(Dataset, requeried_document.dataset_id)
-
-            if not dataset:
-                raise ValueError("no dataset found")
-
-            # get exist document_segment list and delete
-            document_segments = session.scalars(
-                select(DocumentSegment).where(
-                    DocumentSegment.dataset_id == dataset.id,
-                    DocumentSegment.document_id == requeried_document.id,
-                )
-            ).all()
-
-            for document_segment in document_segments:
-                session.delete(document_segment)
-                if requeried_document.doc_form == IndexStructureType.PARENT_CHILD_INDEX:
-                    # delete child chunks
-                    session.execute(delete(ChildChunk).where(ChildChunk.segment_id == document_segment.id))
-            session.commit()
-            # get the process rule
-            stmt = select(DatasetProcessRule).where(DatasetProcessRule.id == requeried_document.dataset_process_rule_id)
-            processing_rule = session.scalar(stmt)
-            if not processing_rule:
-                raise ValueError("no process rule found")
-
-            index_type = requeried_document.doc_form
-            index_processor = IndexProcessorFactory(index_type).init_index_processor()
-            # extract
-            text_docs = self._extract(index_processor, requeried_document, processing_rule.to_dict(), session)
-            session.commit()
-
-            # transform
-            current_user = session.get(Account, requeried_document.created_by)
-            if not current_user:
-                raise ValueError("no current user found")
-            current_user.set_tenant_id_with_session(dataset.tenant_id, session=session)
-            documents = self._transform(
-                index_processor,
-                dataset,
-                text_docs,
-                requeried_document.doc_language,
-                processing_rule.to_dict(),
-                current_user=current_user,
-                session=session,
-            )
-            token_counts = calculate_segment_token_counts(dataset=dataset, documents=documents)
-            total_tokens = sum(token_counts)
-            # save segment
-            self._load_segments(
-                session=session,
-                dataset=dataset,
-                dataset_document=requeried_document,
-                documents=documents,
-                token_counts=token_counts,
-            )
-            session.commit()
-
-            # load
-            self._load(
-                session=session,
-                dataset=dataset,
-                dataset_document=requeried_document,
-                documents=documents,
-                total_tokens=total_tokens,
-            )
-        except DocumentIsPausedError:
-            raise DocumentIsPausedError(f"Document paused, document id: {document_id}")
-        except ProviderTokenNotInitError as e:
-            self._handle_indexing_error(document_id, e, session)
-        except Exception as e:
-            self._handle_indexing_error(document_id, e, session)
-
-    @with_credit_usage_created_by(CreditUsageCreatedBy.KNOWLEDGE_INDEXING)
-    def run_in_indexing_status(self, dataset_document: DatasetDocument, session: Session):
-        """Run the indexing process when the index_status is indexing."""
-        document_id = dataset_document.id
-        try:
-            # Re-query the document to ensure it's bound to the current session
-            requeried_document = session.get(DatasetDocument, document_id)
-            if not requeried_document:
-                logger.warning("Document not found: %s", document_id)
-                return
-
-            # get dataset
-            dataset = session.get(Dataset, requeried_document.dataset_id)
-
-            if not dataset:
-                raise ValueError("no dataset found")
-
-            # get existing document segments
-            document_segments = session.scalars(
-                select(DocumentSegment).where(
-                    DocumentSegment.dataset_id == dataset.id,
-                    DocumentSegment.document_id == requeried_document.id,
-                )
-            ).all()
-
-            documents = []
-            if document_segments:
-                for document_segment in document_segments:
-                    # transform segment to node
-                    if document_segment.status != SegmentStatus.COMPLETED:
-                        document = Document(
-                            page_content=document_segment.content,
-                            metadata={
-                                "doc_id": document_segment.index_node_id,
-                                "doc_hash": document_segment.index_node_hash,
-                                "document_id": document_segment.document_id,
-                                "dataset_id": document_segment.dataset_id,
-                            },
-                        )
-                        if requeried_document.doc_form == IndexStructureType.PARENT_CHILD_INDEX:
-                            child_chunks = document_segment.get_child_chunks(session=session)
-                            if child_chunks:
-                                child_documents = []
-                                for child_chunk in child_chunks:
-                                    child_document = ChildDocument(
-                                        page_content=child_chunk.content,
-                                        metadata={
-                                            "doc_id": child_chunk.index_node_id,
-                                            "doc_hash": child_chunk.index_node_hash,
-                                            "document_id": document_segment.document_id,
-                                            "dataset_id": document_segment.dataset_id,
-                                        },
-                                    )
-                                    child_documents.append(child_document)
-                                document.children = child_documents
-                        documents.append(document)
-            # Preserve the full document total even when only incomplete segments are re-indexed.
-            total_tokens = sum(document_segment.tokens for document_segment in document_segments)
-            # build index
-            self._load(
-                session=session,
-                dataset=dataset,
-                dataset_document=requeried_document,
-                documents=documents,
-                total_tokens=total_tokens,
-            )
-        except DocumentIsPausedError:
-            raise DocumentIsPausedError(f"Document paused, document id: {document_id}")
-        except ProviderTokenNotInitError as e:
-            self._handle_indexing_error(document_id, e, session)
-        except Exception as e:
-            self._handle_indexing_error(document_id, e, session)
-
-    @with_credit_usage_created_by(CreditUsageCreatedBy.KNOWLEDGE_INDEXING)
-    def indexing_estimate(
-        self,
-        tenant_id: str,
-        extract_settings: list[ExtractSetting],
-        tmp_processing_rule: Mapping[str, Any],
-        doc_form: str | None = None,
-        doc_language: str = "English",
-        dataset_id: str | None = None,
-        indexing_technique: str = IndexTechniqueType.ECONOMY,
-        *,
-        session: Session,
-    ) -> IndexingEstimate:
-        """
-        Estimate the indexing for the document.
-        """
-        # check document limit
-        if dify_config.DEPLOYMENT_EDITION == DeploymentEdition.CLOUD:
-            count = len(extract_settings)
-            batch_upload_limit = dify_config.BATCH_UPLOAD_LIMIT
-            if count > batch_upload_limit:
-                raise ValueError(f"You have reached the batch upload limit of {batch_upload_limit}.")
-
-        embedding_model_instance = None
-        if dataset_id:
-            dataset = session.get(Dataset, dataset_id)
-            if not dataset:
-                raise ValueError("Dataset not found.")
-            if IndexTechniqueType.HIGH_QUALITY in {dataset.indexing_technique, indexing_technique}:
-                if dataset.embedding_model_provider:
-                    embedding_model_instance = self._get_model_manager(tenant_id).get_model_instance(
-                        tenant_id=tenant_id,
-                        provider=dataset.embedding_model_provider,
-                        model_type=ModelType.TEXT_EMBEDDING,
-                        model=dataset.embedding_model,
-                    )
-                else:
-                    embedding_model_instance = self._get_model_manager(tenant_id).get_default_model_instance(
-                        tenant_id=tenant_id,
-                        model_type=ModelType.TEXT_EMBEDDING,
-                    )
-        else:
-            if indexing_technique == IndexTechniqueType.HIGH_QUALITY:
-                embedding_model_instance = self._get_model_manager(tenant_id).get_default_model_instance(
-                    tenant_id=tenant_id,
-                    model_type=ModelType.TEXT_EMBEDDING,
-                )
-        # keep separate, avoid union-list ambiguity
-        preview_texts: list[PreviewDetail] = []
-        qa_preview_texts: list[QAPreviewDetail] = []
-
-        total_segments = 0
-        # doc_form represents the segmentation method (general, parent-child, QA)
-        index_type = doc_form
-        index_processor = IndexProcessorFactory(index_type).init_index_processor()
-        # one extract_setting is one source document
-        for extract_setting in extract_settings:
-            # extract
-            processing_rule = {
-                "mode": tmp_processing_rule["mode"],
-                "rules": tmp_processing_rule.get("rules"),
-            }
-            # Extract document content
-            text_docs = index_processor.extract(
-                extract_setting, process_rule_mode=tmp_processing_rule["mode"], session=session
-            )
-            # Cleaning and segmentation
-            documents = index_processor.transform(
-                text_docs,
-                current_user=None,
-                embedding_model_instance=embedding_model_instance,
-                process_rule=processing_rule,
-                tenant_id=tenant_id,
-                doc_language=doc_language,
-                preview=True,
-                session=session,
-            )
-            total_segments += len(documents)
-            for document in documents:
-                preview_count = len(qa_preview_texts) if doc_form == "qa_model" else len(preview_texts)
-                if preview_count < 10:
-                    if doc_form and doc_form == "qa_model":
-                        qa_detail = QAPreviewDetail(
-                            question=document.page_content, answer=document.metadata.get("answer") or ""
-                        )
-                        qa_preview_texts.append(qa_detail)
-                    else:
-                        preview_detail = PreviewDetail(content=document.page_content)
-                        if document.children:
-                            preview_detail.child_chunks = [child.page_content for child in document.children]
-                        preview_texts.append(preview_detail)
-
-                # delete image files and related db records
-                image_upload_file_ids = get_image_upload_file_ids(document.page_content)
-                for upload_file_id in image_upload_file_ids:
-                    stmt = select(UploadFile).where(UploadFile.id == upload_file_id)
-                    image_file = session.scalar(stmt)
-                    if image_file is None:
-                        continue
-                    try:
-                        storage.delete(image_file.key)
-                    except Exception:
-                        logger.exception(
-                            "Delete image_files failed while indexing_estimate, \
-                                          image_upload_file_is: %s",
-                            upload_file_id,
-                        )
-                    session.delete(image_file)
-
-        # Persist preview cleanup and release the caller transaction before
-        # summary workers query through their own sessions.
-        session.commit()
-
-        if doc_form and doc_form == "qa_model":
-            return IndexingEstimate(total_segments=total_segments * 20, qa_preview=qa_preview_texts, preview=[])
-
-        # Generate summary preview
-        summary_index_setting = tmp_processing_rule.get("summary_index_setting")
-        if summary_index_setting and summary_index_setting.get("enable") and preview_texts:
-            preview_texts = index_processor.generate_summary_preview(
-                tenant_id, preview_texts, summary_index_setting, doc_language, session=session
-            )
-
-        return IndexingEstimate(total_segments=total_segments, preview=preview_texts)
-
-    def _extract(
-        self,
-        index_processor: BaseIndexProcessor,
-        dataset_document: DatasetDocument,
-        process_rule: Mapping[str, Any],
-        session: Session,
-    ) -> list[Document]:
-        data_source_info = dataset_document.data_source_info_dict
-        text_docs = []
-        match dataset_document.data_source_type:
-            case DataSourceType.UPLOAD_FILE:
-                if not data_source_info or "upload_file_id" not in data_source_info:
-                    raise ValueError("no upload file found")
-                stmt = select(UploadFile).where(UploadFile.id == data_source_info["upload_file_id"])
-                file_detail = session.scalars(stmt).one_or_none()
-
-                if file_detail:
-                    extract_setting = ExtractSetting(
-                        datasource_type=DatasourceType.FILE,
-                        upload_file=file_detail,
-                        document_model=dataset_document.doc_form,
-                    )
-                    text_docs = index_processor.extract(
-                        extract_setting, process_rule_mode=process_rule["mode"], session=session
-                    )
-            case DataSourceType.NOTION_IMPORT:
-                if (
-                    not data_source_info
-                    or "notion_workspace_id" not in data_source_info
-                    or "notion_page_id" not in data_source_info
-                ):
-                    raise ValueError("no notion import info found")
-                extract_setting = ExtractSetting(
-                    datasource_type=DatasourceType.NOTION,
-                    notion_info=NotionInfo.model_validate(
-                        {
-                            "credential_id": data_source_info.get("credential_id"),
-                            "notion_workspace_id": data_source_info["notion_workspace_id"],
-                            "notion_obj_id": data_source_info["notion_page_id"],
-                            "notion_page_type": data_source_info["type"],
-                            "document": dataset_document,
-                            "tenant_id": dataset_document.tenant_id,
-                        }
-                    ),
-                    document_model=dataset_document.doc_form,
-                )
-                text_docs = index_processor.extract(
-                    extract_setting, process_rule_mode=process_rule["mode"], session=session
-                )
-            case DataSourceType.WEBSITE_CRAWL:
-                if (
-                    not data_source_info
-                    or "provider" not in data_source_info
-                    or "url" not in data_source_info
-                    or "job_id" not in data_source_info
-                ):
-                    raise ValueError("no website import info found")
-                extract_setting = ExtractSetting(
-                    datasource_type=DatasourceType.WEBSITE,
-                    website_info=WebsiteInfo.model_validate(
-                        {
-                            "provider": data_source_info["provider"],
-                            "job_id": data_source_info["job_id"],
-                            "tenant_id": dataset_document.tenant_id,
-                            "url": data_source_info["url"],
-                            "mode": data_source_info["mode"],
-                            "only_main_content": data_source_info["only_main_content"],
-                        }
-                    ),
-                    document_model=dataset_document.doc_form,
-                )
-                text_docs = index_processor.extract(
-                    extract_setting, process_rule_mode=process_rule["mode"], session=session
-                )
-            case _:
-                return []
-        # update document status to splitting
-        self._update_document_index_status(
-            session=session,
-            document_id=dataset_document.id,
-            after_indexing_status=IndexingStatus.SPLITTING,
-            extra_update_params={
-                DatasetDocument.parsing_completed_at: naive_utc_now(),
-            },
-        )
-
-        # replace doc id to document model id
-        for text_doc in text_docs:
-            if text_doc.metadata is not None:
-                text_doc.metadata["document_id"] = dataset_document.id
-                text_doc.metadata["dataset_id"] = dataset_document.dataset_id
-
-        return text_docs
-
-    @staticmethod
-    def filter_string(text: str):
-        text = re.sub(r"<\|", "<", text)
-        text = re.sub(r"\|>", ">", text)
-        text = re.sub(r"[\x00-\x08\x0B\x0C\x0E-\x1F\x7F]", "", text)
-        # Unicode  U+FFFE
-        text = re.sub("\ufffe", "", text)
-        return text
-
-    @staticmethod
-    def _get_splitter(
-        processing_rule_mode: str,
-        max_tokens: int,
-        chunk_overlap: int,
-        separator: str,
-        embedding_model_instance: ModelInstance | None,
-    ) -> TextSplitter:
-        """
-        Get the NodeParser object according to the processing rule.
-        """
-        character_splitter: TextSplitter
-        if processing_rule_mode in ["custom", "hierarchical"]:
-            # The user-defined segmentation rule
-            max_segmentation_tokens_length = dify_config.INDEXING_MAX_SEGMENTATION_TOKENS_LENGTH
-            if max_tokens < 50 or max_tokens > max_segmentation_tokens_length:
-                raise ValueError(f"Custom segment length should be between 50 and {max_segmentation_tokens_length}.")
-
-            if separator:
-                separator = separator.replace("\\n", "\n")
-
-            character_splitter = FixedRecursiveCharacterTextSplitter.from_encoder(
-                chunk_size=max_tokens,
-                chunk_overlap=chunk_overlap,
-                fixed_separator=separator,
-                separators=["\n\n", "。", ". ", " ", ""],
-                embedding_model_instance=embedding_model_instance,
-            )
-        else:
-            # Automatic segmentation
-            automatic_rules: dict[str, Any] = dict(DatasetProcessRule.AUTOMATIC_RULES["segmentation"])
-            character_splitter = EnhanceRecursiveCharacterTextSplitter.from_encoder(
-                chunk_size=automatic_rules["max_tokens"],
-                chunk_overlap=automatic_rules["chunk_overlap"],
-                separators=["\n\n", "。", ". ", " ", ""],
-                embedding_model_instance=embedding_model_instance,
-            )
-
-        return character_splitter
-
-    def _split_to_documents_for_estimate(
-        self, text_docs: list[Document], splitter: TextSplitter, processing_rule: DatasetProcessRule
-    ) -> list[Document]:
-        """
-        Split the text documents into nodes.
-        """
-        all_documents: list[Document] = []
-        for text_doc in text_docs:
-            # document clean
-            document_text = self._document_clean(text_doc.page_content, processing_rule)
-            text_doc.page_content = document_text
-
-            # parse document to nodes
-            documents = splitter.split_documents([text_doc])
-
-            split_documents = []
-            for document in documents:
-                if document.page_content is None or not document.page_content.strip():
-                    continue
-                if document.metadata is not None:
-                    doc_id = str(uuid.uuid4())
-                    hash = helper.generate_text_hash(document.page_content)
-                    document.metadata["doc_id"] = doc_id
-                    document.metadata["doc_hash"] = hash
-
-                split_documents.append(document)
-
-            all_documents.extend(split_documents)
-
-        return all_documents
-
-    @staticmethod
-    def _document_clean(text: str, processing_rule: DatasetProcessRule) -> str:
-        """
-        Clean the document text according to the processing rules.
-        """
-        rules: AutomaticRulesConfig | dict[str, Any]
-        if processing_rule.mode == ProcessRuleMode.AUTOMATIC:
-            rules = DatasetProcessRule.AUTOMATIC_RULES
-        else:
-            rules = json.loads(processing_rule.rules) if processing_rule.rules else {}
-        document_text = CleanProcessor.clean(text, {"rules": rules})
-
-        return document_text
-
-    @staticmethod
-    def format_split_text(text: str) -> list[QAPreviewDetail]:
-        regex = r"Q\d+:\s*(.*?)\s*A\d+:\s*([\s\S]*?)(?=Q\d+:|$)"
-        matches = re.findall(regex, text, re.UNICODE)
-
-        return [QAPreviewDetail(question=q, answer=re.sub(r"\n\s*", "\n", a.strip())) for q, a in matches if q and a]
-
-    def _load(
-        self,
-        session: Session,
-        dataset: Dataset,
-        dataset_document: DatasetDocument,
-        documents: list[Document],
-        total_tokens: int,
-    ) -> None:
-        """Build indexes and mark the document complete using the token total computed before hash sharding."""
-
-        # Build indexes using the existing hash-based worker groups.
-        indexing_start_at = time.perf_counter()
-        create_keyword_thread = None
-        if (
-            dataset_document.doc_form != IndexStructureType.PARENT_CHILD_INDEX
-            and dataset.indexing_technique == IndexTechniqueType.ECONOMY
-        ):
-            # create keyword index
-            create_keyword_thread = threading.Thread(
-                target=propagate_context(self._process_keyword_index),
-                args=(current_app._get_current_object(), dataset.id, dataset_document.id, documents),  # type: ignore
-            )
-            create_keyword_thread.start()
-
-        max_workers = 10
-        if dataset.indexing_technique == IndexTechniqueType.HIGH_QUALITY:
-            with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor:
-                futures = []
-
-                # Distribute documents into multiple groups based on the hash values of page_content
-                # This is done to prevent multiple threads from processing the same document,
-                # Thereby avoiding potential database insertion deadlocks
-                document_groups: list[list[Document]] = [[] for _ in range(max_workers)]
-                for document in documents:
-                    hash = helper.generate_text_hash(document.page_content)
-                    group_index = int(hash, 16) % max_workers
-                    document_groups[group_index].append(document)
-                for chunk_documents in document_groups:
-                    if len(chunk_documents) == 0:
-                        continue
-                    futures.append(
-                        executor.submit(
-                            propagate_context(self._process_chunk),
-                            current_app._get_current_object(),  # type: ignore
-                            dataset_document.doc_form,
-                            chunk_documents,
-                            dataset.id,
-                            dataset_document.id,
-                        )
-                    )
-
-                for future in futures:
-                    future.result()
-        if (
-            dataset_document.doc_form != IndexStructureType.PARENT_CHILD_INDEX
-            and dataset.indexing_technique == IndexTechniqueType.ECONOMY
-            and create_keyword_thread is not None
-        ):
-            create_keyword_thread.join()
-        indexing_end_at = time.perf_counter()
-
-        # update document status to completed
-        self._update_document_index_status(
-            session=session,
-            document_id=dataset_document.id,
-            after_indexing_status=IndexingStatus.COMPLETED,
-            extra_update_params={
-                DatasetDocument.tokens: total_tokens,
-                DatasetDocument.completed_at: naive_utc_now(),
-                DatasetDocument.indexing_latency: indexing_end_at - indexing_start_at,
-                DatasetDocument.error: None,
-            },
-        )
-
-    @staticmethod
-    def _process_keyword_index(flask_app: Flask, dataset_id: str, document_id: str, documents: list[Document]):
-        with flask_app.app_context():
-            with session_factory.create_session() as session:
-                dataset = session.get(Dataset, dataset_id)
-                if not dataset:
-                    raise ValueError("no dataset found")
-                keyword = Keyword(dataset)
-                keyword.create(documents, session)
-                if dataset.indexing_technique != IndexTechniqueType.HIGH_QUALITY:
-                    document_ids = [document.metadata["doc_id"] for document in documents]
-                    session.execute(
-                        update(DocumentSegment)
-                        .where(
-                            DocumentSegment.document_id == document_id,
-                            DocumentSegment.dataset_id == dataset_id,
-                            DocumentSegment.index_node_id.in_(document_ids),
-                            DocumentSegment.status == SegmentStatus.INDEXING,
-                        )
-                        .values(
-                            status=SegmentStatus.COMPLETED,
-                            enabled=True,
-                            completed_at=naive_utc_now(),
-                        )
-                    )
-                session.commit()
-
-    def _process_chunk(
-        self,
-        flask_app: Flask,
-        index_type: str,
-        chunk_documents: list[Document],
-        dataset_id: str,
-        dataset_document_id: str,
-    ) -> None:
-        with flask_app.app_context():
-            with session_factory.create_session() as session:
-                dataset = session.get(Dataset, dataset_id)
-                if not dataset:
-                    raise ValueError("no dataset found")
-
-                dataset_document = session.get(DatasetDocument, dataset_document_id)
-                if not dataset_document:
-                    raise ValueError("no document found")
-
-                # check document is paused
-                self._check_document_paused_status(dataset_document.id)
-
-                multimodal_documents = []
-                for document in chunk_documents:
-                    if document.attachments and dataset.is_multimodal:
-                        multimodal_documents.extend(document.attachments)
-
-                # load index
-                index_processor = IndexProcessorFactory(index_type).init_index_processor()
-                index_processor.load(
-                    dataset,
-                    chunk_documents,
-                    multimodal_documents=multimodal_documents,
-                    with_keywords=False,
-                    session=session,
-                )
-
-                document_ids = [document.metadata["doc_id"] for document in chunk_documents]
-                session.execute(
-                    update(DocumentSegment)
-                    .where(
-                        DocumentSegment.document_id == dataset_document.id,
-                        DocumentSegment.dataset_id == dataset.id,
-                        DocumentSegment.index_node_id.in_(document_ids),
-                        DocumentSegment.status == SegmentStatus.INDEXING,
-                    )
-                    .values(
-                        status=SegmentStatus.COMPLETED,
-                        enabled=True,
-                        completed_at=naive_utc_now(),
-                    )
-                )
-
-                session.commit()
-
-    @staticmethod
-    def _check_document_paused_status(document_id: str):
-        indexing_cache_key = f"document_{document_id}_is_paused"
-        result = redis_client.get(indexing_cache_key)
-        if result:
-            raise DocumentIsPausedError()
-
-    @staticmethod
-    def _update_document_index_status(
-        document_id: str,
-        after_indexing_status: IndexingStatus,
-        extra_update_params: Mapping[Any, Any] | None = None,
-        *,
-        session: Session,
-    ):
-        """
-        Update the document indexing status.
-        """
-        count = (
-            session.scalar(
-                select(func.count())
-                .select_from(DatasetDocument)
-                .where(DatasetDocument.id == document_id, DatasetDocument.is_paused == True)
-            )
-            or 0
-        )
-        if count > 0:
-            raise DocumentIsPausedError()
-        document = session.get(DatasetDocument, document_id)
-        if not document:
-            raise DocumentIsDeletedPausedError()
-
-        update_params = {DatasetDocument.indexing_status: after_indexing_status}
-
-        if extra_update_params:
-            update_params.update(extra_update_params)
-        session.execute(update(DatasetDocument).where(DatasetDocument.id == document_id).values(update_params))  # type: ignore
-        session.flush()
-
-    @staticmethod
-    def _update_segments_by_document(dataset_document_id: str, update_params: Mapping[Any, Any], session: Session):
-        """
-        Update the document segment by document id.
-        """
-        session.execute(
-            update(DocumentSegment).where(DocumentSegment.document_id == dataset_document_id).values(update_params)
-        )
-        session.flush()
-
-    def _transform(
-        self,
-        index_processor: BaseIndexProcessor,
-        dataset: Dataset,
-        text_docs: list[Document],
-        doc_language: str,
-        process_rule: Mapping[str, Any],
-        current_user: Account | None = None,
-        *,
-        session: Session,
-    ) -> list[Document]:
-        # get embedding model instance
-        embedding_model_instance = None
-        if dataset.indexing_technique == IndexTechniqueType.HIGH_QUALITY:
-            if dataset.embedding_model_provider:
-                embedding_model_instance = self._get_model_manager(dataset.tenant_id).get_model_instance(
-                    tenant_id=dataset.tenant_id,
-                    provider=dataset.embedding_model_provider,
-                    model_type=ModelType.TEXT_EMBEDDING,
-                    model=dataset.embedding_model,
-                )
-            else:
-                embedding_model_instance = self._get_model_manager(dataset.tenant_id).get_default_model_instance(
-                    tenant_id=dataset.tenant_id,
-                    model_type=ModelType.TEXT_EMBEDDING,
-                )
-
-        documents = index_processor.transform(
-            text_docs,
-            current_user,
-            embedding_model_instance=embedding_model_instance,
-            process_rule=process_rule,
-            tenant_id=dataset.tenant_id,
-            doc_language=doc_language,
-            session=session,
-        )
-
-        return documents
-
-    def _load_segments(
-        self,
-        session: Session,
-        dataset: Dataset,
-        dataset_document: DatasetDocument,
-        documents: list[Document],
-        token_counts: list[int],
-    ) -> None:
-        """Persist transformed documents and their precomputed token counts before indexing starts."""
-        # save node to document segment
-        doc_store = DatasetDocumentStore(
-            dataset=dataset, user_id=dataset_document.created_by, document_id=dataset_document.id
-        )
-
-        # add document segments
-        doc_store.add_documents(
-            session=session,
-            docs=documents,
-            save_child=dataset_document.doc_form == IndexStructureType.PARENT_CHILD_INDEX,
-            token_counts=token_counts,
-        )
-
-        # update document status to indexing
-        cur_time = naive_utc_now()
-        self._update_document_index_status(
-            session=session,
-            document_id=dataset_document.id,
-            after_indexing_status=IndexingStatus.INDEXING,
-            extra_update_params={
-                DatasetDocument.cleaning_completed_at: cur_time,
-                DatasetDocument.splitting_completed_at: cur_time,
-                DatasetDocument.word_count: sum(len(doc.page_content) for doc in documents),
-            },
-        )
-
-        # update segment status to indexing
-        self._update_segments_by_document(
-            session=session,
-            dataset_document_id=dataset_document.id,
-            update_params={
-                DocumentSegment.status: SegmentStatus.INDEXING,
-                DocumentSegment.indexing_at: naive_utc_now(),
-            },
-        )
-
-
-class DocumentIsPausedError(Exception):
-    pass
-
-
-class DocumentIsDeletedPausedError(Exception):
-    pass

api/core/llm_generator/llm_generator.py

@@ -875,8 +875,9 @@ class LLMGenerator:
         instruction: str,
         model_config: ModelConfig,
         ideal_output: str | None,
+        session: Session,
     ):
-        last_run: Message | None = db.session.scalar(
+        last_run: Message | None = session.scalar(
             select(Message)
             .join(App, App.id == Message.app_id)
             .where(Message.app_id == flow_id, App.tenant_id == tenant_id)

api/core/provider_manager.py

@@ -65,7 +65,6 @@ from services.feature_service import FeatureService
 
 if TYPE_CHECKING:
     from graphon.model_runtime.protocols.runtime import ModelRuntime
-    from models.account import Account
 
 logger = logging.getLogger(__name__)
 
@@ -1162,50 +1161,6 @@ class ProviderManager:
                 provider_names.append(str(model_provider_id))
         return provider_names
 
-    @staticmethod
-    def get_provider_available_credentials(
-        tenant_id: str,
-        provider_name: str,
-        user: Account | None = None,
-    ) -> list[CredentialConfiguration]:
-        """
-        Get provider all credentials, filtered by visibility.
-
-        :param tenant_id: workspace id
-        :param provider_name: provider name
-        :param user: current user (id + admin flag drive the visibility filter)
-        :return:
-        """
-        from models.credential_permission import CredentialType as CredPermType
-        from services.credential_permission_service import CredentialPermissionService
-
-        with session_factory.create_session() as session:
-            stmt = (
-                select(ProviderCredential)
-                .where(
-                    ProviderCredential.tenant_id == tenant_id,
-                    ProviderCredential.provider_name.in_(ProviderManager._get_provider_names(provider_name)),
-                )
-                .order_by(ProviderCredential.created_at.desc())
-            )
-
-            if user is not None:
-                stmt = CredentialPermissionService.apply_visibility_filter(
-                    stmt,
-                    model_id_column=ProviderCredential.id,
-                    model_user_id_column=ProviderCredential.user_id,
-                    model_visibility_column=ProviderCredential.visibility,
-                    credential_type=CredPermType.PROVIDER_CREDENTIAL,
-                    user=user,
-                )
-
-            available_credentials = session.scalars(stmt).all()
-
-        return [
-            CredentialConfiguration(credential_id=credential.id, credential_name=credential.credential_name)
-            for credential in available_credentials
-        ]
-
     @staticmethod
     def get_provider_model_available_credentials(
         tenant_id: str, provider_name: str, model_name: str, model_type: str

api/core/rag/datasource/keyword/jieba/jieba.py

@@ -1,7 +1,8 @@
 from collections import defaultdict
-from typing import Any, TypedDict, override
+from collections.abc import Callable, Mapping, Sequence
+from contextlib import AbstractContextManager
+from typing import Any, override
 
-import orjson
 from pydantic import BaseModel
 from sqlalchemy import select
 from sqlalchemy.orm import Session
@@ -12,12 +13,15 @@ from core.rag.datasource.keyword.keyword_base import BaseKeyword
 from core.rag.models.document import Document
 from extensions.ext_redis import redis_client
 from extensions.ext_storage import storage
-from models.dataset import Dataset, DatasetKeywordTable, DocumentSegment
-
-
-class PreSegmentData(TypedDict):
-    segment: DocumentSegment
-    keywords: list[str]
+from models.dataset import Dataset, DocumentSegment
+from repositories.knowledge.dataset_read_repository import get_dataset_keyword_table
+from repositories.knowledge.keyword_table_repository import (
+    keyword_file_key,
+    load_dataset_keyword_table,
+    load_keyword_table,
+    persist_keyword_table,
+    save_keyword_table,
+)
 
 
 class KeywordTableConfig(BaseModel):
@@ -31,80 +35,114 @@ class Jieba(BaseKeyword):
 
     @override
     def create(self, texts: list[Document], session: Session, **kwargs: Any) -> BaseKeyword:
-        lock_name = f"keyword_indexing_lock_{self.dataset.id}"
-        with redis_client.lock(lock_name, timeout=600):
-            keyword_table_handler = JiebaKeywordTableHandler()
-            keyword_table = self._get_dataset_keyword_table(session=session)
-            keyword_number = self.dataset.keyword_number or self._config.max_keywords_per_chunk
-
-            for text in texts:
-                keywords = keyword_table_handler.extract_keywords(text.page_content, keyword_number)
-                if text.metadata is not None:
-                    self._update_segment_keywords(self.dataset.id, text.metadata["doc_id"], list(keywords), session)
-                    keyword_table = self._add_text_to_keyword_table(
-                        keyword_table or {}, text.metadata["doc_id"], list(keywords)
-                    )
-
-            self._save_dataset_keyword_table(keyword_table, session)
-
-            return self
+        self.add_texts(texts, session)
+        return self
 
     @override
     def add_texts(self, texts: list[Document], session: Session, **kwargs: Any):
-        lock_name = f"keyword_indexing_lock_{self.dataset.id}"
-        with redis_client.lock(lock_name, timeout=600):
-            keyword_table_handler = JiebaKeywordTableHandler()
+        self.update_texts(
+            texts,
+            read=lambda: self._read_keyword_record(session),
+            write=lambda storage_type, data, keywords: persist_keyword_table(
+                session,
+                tenant_id=self.dataset.tenant_id,
+                dataset_id=self.dataset.id,
+                storage_type=storage_type,
+                data=data,
+                keywords=keywords,
+            ),
+            lock=redis_client.lock(f"keyword_indexing_lock_{self.dataset.id}", timeout=600),
+            keywords_list=kwargs.get("keywords_list"),
+        )
 
-            keyword_table = self._get_dataset_keyword_table(session=session)
-            keywords_list = kwargs.get("keywords_list")
+    def update_texts(
+        self,
+        texts: Sequence[Document],
+        *,
+        read: Callable[[], tuple[str, str | None]],
+        write: Callable[[str, str, Mapping[str, Sequence[str]]], None],
+        lock: AbstractContextManager[object],
+        keywords_list: Sequence[Sequence[str] | None] | None = None,
+        replace_existing: bool = False,
+        deleted_ids: Sequence[str] = (),
+    ) -> None:
+        """Mutate the shared index with explicit persistence and one dataset lock.
+
+        Repository callbacks may close their sessions before returning, keeping
+        extraction and object-storage I/O outside database transactions.
+        """
+        with lock:
+            storage_type, data = read()
+            payload = load_keyword_table(
+                storage=storage,
+                tenant_id=self.dataset.tenant_id,
+                dataset_id=self.dataset.id,
+                storage_type=storage_type,
+                data=data,
+            )
+            table = dict(payload["__data__"]["table"] or {}) if payload else {}
+            removed = list(deleted_ids)
+            if replace_existing:
+                removed.extend(text.metadata["doc_id"] for text in texts)
+            if removed:
+                table = self._delete_ids_from_keyword_table(table, removed)
+            handler: JiebaKeywordTableHandler | None = None
             keyword_number = self.dataset.keyword_number or self._config.max_keywords_per_chunk
-            for i in range(len(texts)):
-                text = texts[i]
-                if keywords_list:
-                    keywords = keywords_list[i]
-                    if not keywords:
-                        keywords = keyword_table_handler.extract_keywords(text.page_content, keyword_number)
-                else:
-                    keywords = keyword_table_handler.extract_keywords(text.page_content, keyword_number)
-                if text.metadata is not None:
-                    self._update_segment_keywords(self.dataset.id, text.metadata["doc_id"], list(keywords), session)
-                    keyword_table = self._add_text_to_keyword_table(
-                        keyword_table or {}, text.metadata["doc_id"], list(keywords)
-                    )
+            selected: dict[str, Sequence[str]] = {}
+            for position, text in enumerate(texts):
+                keywords = keywords_list[position] if keywords_list else None
+                if not keywords:
+                    if handler is None:
+                        handler = JiebaKeywordTableHandler()
+                    keywords = list(handler.extract_keywords(text.page_content, keyword_number))
+                node_id = text.metadata["doc_id"]
+                selected[node_id] = list(keywords)
+                table = self._add_text_to_keyword_table(table, node_id, list(keywords))
+            encoded = save_keyword_table(
+                storage=storage,
+                tenant_id=self.dataset.tenant_id,
+                dataset_id=self.dataset.id,
+                storage_type=storage_type,
+                table=table,
+            )
+            write(storage_type, encoded, selected)
 
-            self._save_dataset_keyword_table(keyword_table, session)
+    def _read_keyword_record(self, session: Session) -> tuple[str, str | None]:
+        row = get_dataset_keyword_table(self.dataset, session=session)
+        return (row.data_source_type, row.keyword_table) if row else (dify_config.KEYWORD_DATA_SOURCE_TYPE, None)
 
     @override
     def text_exists(self, id: str, *, session: Session) -> bool:
-        dataset_keyword_table = self.dataset.get_dataset_keyword_table(session=session)
         keyword_table = None
-        keyword_table_dict = (
-            dataset_keyword_table.get_keyword_table_dict(session=session) if dataset_keyword_table else None
-        )
+        keyword_table_dict = load_dataset_keyword_table(self.dataset, session=session, storage=storage)
         if keyword_table_dict:
             data: Any = keyword_table_dict["__data__"]
             keyword_table = dict(data["table"])
         if keyword_table is None:
             return False
-        return id in set.union(*keyword_table.values())
+        return any(id in node_ids for node_ids in keyword_table.values())
 
     @override
     def delete_by_ids(self, ids: list[str], session: Session, **kwargs: Any):
-        lock_name = f"keyword_indexing_lock_{self.dataset.id}"
-        with redis_client.lock(lock_name, timeout=600):
-            keyword_table = self._get_dataset_keyword_table(session)
-            if keyword_table is not None:
-                keyword_table = self._delete_ids_from_keyword_table(keyword_table, ids)
-
-            self._save_dataset_keyword_table(keyword_table, session)
+        self.update_texts(
+            [],
+            read=lambda: self._read_keyword_record(session),
+            write=lambda storage_type, data, keywords: persist_keyword_table(
+                session,
+                tenant_id=self.dataset.tenant_id,
+                dataset_id=self.dataset.id,
+                storage_type=storage_type,
+                data=data,
+                keywords=keywords,
+            ),
+            lock=redis_client.lock(f"keyword_indexing_lock_{self.dataset.id}", timeout=600),
+            deleted_ids=ids,
+        )
 
     @override
     def search(self, query: str, *, session: Session, **kwargs: Any) -> list[Document]:
-        dataset_keyword_table = self.dataset.get_dataset_keyword_table(session=session)
         keyword_table = None
-        keyword_table_dict = (
-            dataset_keyword_table.get_keyword_table_dict(session=session) if dataset_keyword_table else None
-        )
+        keyword_table_dict = load_dataset_keyword_table(self.dataset, session=session, storage=storage)
         if keyword_table_dict:
             data: Any = keyword_table_dict["__data__"]
             keyword_table = dict(data["table"])
@@ -145,62 +183,14 @@ class Jieba(BaseKeyword):
     def delete(self, *, session: Session):
         lock_name = f"keyword_indexing_lock_{self.dataset.id}"
         with redis_client.lock(lock_name, timeout=600):
-            dataset_keyword_table = self.dataset.get_dataset_keyword_table(session=session)
+            dataset_keyword_table = get_dataset_keyword_table(self.dataset, session=session)
             if dataset_keyword_table:
                 session.delete(dataset_keyword_table)
                 session.commit()
                 if dataset_keyword_table.data_source_type != "database":
-                    file_key = "keyword_files/" + self.dataset.tenant_id + "/" + self.dataset.id + ".txt"
+                    file_key = keyword_file_key(self.dataset.tenant_id, self.dataset.id)
                     storage.delete(file_key)
 
-    def _save_dataset_keyword_table(self, keyword_table: dict[str, set[str]] | None, session: Session):
-        keyword_table_dict = {
-            "__type__": "keyword_table",
-            "__data__": {"index_id": self.dataset.id, "summary": None, "table": keyword_table},
-        }
-        dataset_keyword_table = session.scalar(
-            select(DatasetKeywordTable).where(DatasetKeywordTable.dataset_id == self.dataset.id)
-        )
-        keyword_data_source_type = dataset_keyword_table.data_source_type if dataset_keyword_table else "file"
-        if keyword_data_source_type == "database":
-            if dataset_keyword_table is None:
-                return
-            dataset_keyword_table.keyword_table = dumps_with_sets(keyword_table_dict)
-            session.flush()
-        else:
-            file_key = "keyword_files/" + self.dataset.tenant_id + "/" + self.dataset.id + ".txt"
-            if storage.exists(file_key):
-                storage.delete(file_key)
-            storage.save(file_key, dumps_with_sets(keyword_table_dict).encode("utf-8"))
-
-    def _get_dataset_keyword_table(self, session: Session) -> dict[str, set[str]] | None:
-        dataset_keyword_table = session.scalar(
-            select(DatasetKeywordTable).where(DatasetKeywordTable.dataset_id == self.dataset.id)
-        )
-        if dataset_keyword_table:
-            keyword_table_dict = dataset_keyword_table.get_keyword_table_dict(session=session)
-            if keyword_table_dict:
-                data: Any = keyword_table_dict["__data__"]
-                return dict(data["table"])
-        else:
-            keyword_data_source_type = dify_config.KEYWORD_DATA_SOURCE_TYPE
-            dataset_keyword_table = DatasetKeywordTable(
-                dataset_id=self.dataset.id,
-                keyword_table="",
-                data_source_type=keyword_data_source_type,
-            )
-            if keyword_data_source_type == "database":
-                dataset_keyword_table.keyword_table = dumps_with_sets(
-                    {
-                        "__type__": "keyword_table",
-                        "__data__": {"index_id": self.dataset.id, "summary": None, "table": {}},
-                    }
-                )
-            session.add(dataset_keyword_table)
-            session.flush()
-
-        return {}
-
     def _add_text_to_keyword_table(
         self, keyword_table: dict[str, set[str]], id: str, keywords: list[str]
     ) -> dict[str, set[str]]:
@@ -245,58 +235,3 @@ class Jieba(BaseKeyword):
         )
 
         return sorted_chunk_indices[:k]
-
-    def _update_segment_keywords(self, dataset_id: str, node_id: str, keywords: list[str], session: Session):
-        stmt = select(DocumentSegment).where(
-            DocumentSegment.dataset_id == dataset_id, DocumentSegment.index_node_id == node_id
-        )
-        document_segment = session.scalar(stmt)
-        if document_segment:
-            document_segment.keywords = keywords
-            session.add(document_segment)
-            session.flush()
-
-    def create_segment_keywords(self, node_id: str, keywords: list[str], session: Session):
-        keyword_table = self._get_dataset_keyword_table(session)
-        self._update_segment_keywords(self.dataset.id, node_id, keywords, session)
-        keyword_table = self._add_text_to_keyword_table(keyword_table or {}, node_id, keywords)
-        self._save_dataset_keyword_table(keyword_table, session)
-
-    def multi_create_segment_keywords(self, pre_segment_data_list: list[PreSegmentData], session: Session):
-        keyword_table_handler = JiebaKeywordTableHandler()
-        keyword_table = self._get_dataset_keyword_table(session)
-        for pre_segment_data in pre_segment_data_list:
-            segment = pre_segment_data["segment"]
-            if pre_segment_data["keywords"]:
-                segment.keywords = pre_segment_data["keywords"]
-                assert segment.index_node_id
-                keyword_table = self._add_text_to_keyword_table(
-                    keyword_table or {}, segment.index_node_id, pre_segment_data["keywords"]
-                )
-            else:
-                keyword_number = self.dataset.keyword_number or self._config.max_keywords_per_chunk
-
-                keywords = keyword_table_handler.extract_keywords(segment.content, keyword_number)
-                segment.keywords = list(keywords)
-                assert segment.index_node_id
-                keyword_table = self._add_text_to_keyword_table(
-                    keyword_table or {}, segment.index_node_id, list(keywords)
-                )
-        self._save_dataset_keyword_table(keyword_table, session)
-
-    def update_segment_keywords_index(self, node_id: str, keywords: list[str], session: Session):
-        keyword_table = self._get_dataset_keyword_table(session)
-        keyword_table = self._add_text_to_keyword_table(keyword_table or {}, node_id, keywords)
-        self._save_dataset_keyword_table(keyword_table, session)
-
-
-def set_orjson_default(obj: Any):
-    """Default function for orjson serialization of set types"""
-    if isinstance(obj, set):
-        return list(obj)
-    raise TypeError(f"Object of type {type(obj).__name__} is not JSON serializable")
-
-
-def dumps_with_sets(obj: Any) -> str:
-    """JSON dumps with set support using orjson"""
-    return orjson.dumps(obj, default=set_orjson_default).decode("utf-8")

api/core/rag/datasource/retrieval_service.py

@@ -5,7 +5,7 @@ from concurrent.futures import ThreadPoolExecutor
 from typing import Any, NotRequired, TypedDict
 
 from flask import Flask, current_app
-from sqlalchemy import select
+from sqlalchemy import select, tuple_
 from sqlalchemy.orm import Session, load_only
 
 from configs import dify_config
@@ -35,7 +35,7 @@ from models.dataset import (
 )
 from models.dataset import Document as DatasetDocument
 from models.model import UploadFile
-from services.external_knowledge_service import ExternalDatasetService
+from services.knowledge.external.service import ExternalDatasetService
 
 
 class SegmentAttachmentResult(TypedDict):
@@ -486,7 +486,14 @@ class RetrievalService:
                 for doc in session.scalars(
                     select(DatasetDocument)
                     .where(DatasetDocument.id.in_(document_ids))
-                    .options(load_only(DatasetDocument.id, DatasetDocument.doc_form, DatasetDocument.dataset_id))
+                    .options(
+                        load_only(
+                            DatasetDocument.id,
+                            DatasetDocument.tenant_id,
+                            DatasetDocument.doc_form,
+                            DatasetDocument.dataset_id,
+                        )
+                    )
                 ).all()
             }
 
@@ -559,7 +566,14 @@ class RetrievalService:
             doc_segment_map: dict[str, list[str]] = {}
             segment_summary_map: dict[str, str] = {}  # Map segment_id to summary content
 
-            attachments = cls.get_segment_attachment_infos(image_doc_ids, session)
+            dataset_refs = tuple(
+                {(document.tenant_id, document.dataset_id) for document in valid_dataset_documents.values()}
+            )
+            attachments = cls.get_segment_attachment_infos(
+                image_doc_ids,
+                dataset_refs=dataset_refs,
+                session=session,
+            )
 
             for attachment in attachments:
                 segment_ids.append(attachment["segment_id"])
@@ -947,15 +961,30 @@ class RetrievalService:
 
     @classmethod
     def get_segment_attachment_infos(
-        cls, attachment_ids: list[str], session: Session
+        cls,
+        attachment_ids: list[str],
+        *,
+        dataset_refs: Sequence[tuple[str, str]],
+        session: Session,
     ) -> list[SegmentAttachmentInfoResult]:
+        if not attachment_ids or not dataset_refs:
+            return []
         attachment_infos: list[SegmentAttachmentInfoResult] = []
         granted_upload_file_ids: list[str] = []
-        upload_files = session.scalars(select(UploadFile).where(UploadFile.id.in_(attachment_ids))).all()
+        tenant_ids = tuple({tenant_id for tenant_id, _ in dataset_refs})
+        upload_files = session.scalars(
+            select(UploadFile).where(
+                UploadFile.id.in_(attachment_ids),
+                UploadFile.tenant_id.in_(tenant_ids),
+            )
+        ).all()
         if upload_files:
             upload_file_ids = [upload_file.id for upload_file in upload_files]
             attachment_bindings = session.scalars(
-                select(SegmentAttachmentBinding).where(SegmentAttachmentBinding.attachment_id.in_(upload_file_ids))
+                select(SegmentAttachmentBinding).where(
+                    SegmentAttachmentBinding.attachment_id.in_(upload_file_ids),
+                    tuple_(SegmentAttachmentBinding.tenant_id, SegmentAttachmentBinding.dataset_id).in_(dataset_refs),
+                )
             ).all()
             attachment_binding_map = {binding.attachment_id: binding for binding in attachment_bindings}