fix(deploy): [skip-issue] 共享文档登录脚本迁移至 app/services/ 修复清理清单导致镜像内缺失
Some checks failed
gitleaks / gitleaks (push) Failing after 11s
backend-test / commit-message-lint (push) Successful in 2s
backend-test / analytics-ods-isolation (push) Failing after 3s
backend-test / backend-lint-test (push) Failing after 1h9m31s
backend-test / frontend-lint-test (push) Successful in 1m35s
Some checks failed
gitleaks / gitleaks (push) Failing after 11s
backend-test / commit-message-lint (push) Successful in 2s
backend-test / analytics-ods-isolation (push) Failing after 3s
backend-test / backend-lint-test (push) Failing after 1h9m31s
backend-test / frontend-lint-test (push) Successful in 1m35s
根因: 两脚本在 scripts/archive/ 子目录, 但 upgrade.sh 白名单 find 只保留 scripts/ 根目录
同名文件, .dockerignore 放行规则 !scripts/kdocs_login.py 也匹配不到 archive/ 路径,
导致 docker compose build 时脚本被排除, 镜像内根本没有 → 共享文档同步反复报
"未找到金山登录态"
修复:
- kdocs_login.py / tencent_doc_login.py 迁入 app/services/shared_doc_{kdocs,tencent}_cli.py
- 去掉 sys.path 注入, 直接 from app.core.config import settings
- upgrade.sh: 整个 backend/scripts/ 清空(不再需要按文件名白名单保留)
- .dockerignore: 删掉两条 scripts/ 登录脚本放行规则
- package.sh: 删掉追加登录脚本到 tar 的 staging 逻辑
- 服务层/错误提示/测试断言同步更新路径引用
- 旧文件删除
This commit is contained in:
@@ -10,15 +10,12 @@ __pycache__/
|
||||
.env
|
||||
|
||||
# 调试/一次性脚本(含历史凭证与嗅探器,禁止进入生产镜像,2026-08-20 审计 C-1/TST-02)
|
||||
# 用 scripts/* 排除内容(保留目录), 再用 ! 例外放行登录脚本(scripts/ 会整目录排除无法例外)
|
||||
# 登录脚本已于 2026-10-20 迁入 app/services/shared_doc_*_cli.py, 不再从 scripts/ 放行
|
||||
scripts/*
|
||||
tests/
|
||||
# 例外: 共享文档扫码登录脚本(运维工具, 生产环境需 headless 扫码更新登录态, #148)
|
||||
!scripts/tencent_doc_login.py
|
||||
!scripts/kdocs_login.py
|
||||
# 例外: 工单到期核验脚本 + 其运行守卫(#193 交付)。
|
||||
# ⚠️ 仅对**直接 docker build** 生效。生产走 upgrade.sh/package.sh 时, 它们有各自的
|
||||
# 发布白名单(只放行两个登录脚本, 见 upgrade.sh L121-123), 会把 backend/scripts/* 全删,
|
||||
# 发布白名单(见 upgrade.sh), 会把 backend/scripts/* 全删,
|
||||
# 故生产需按下方配方先把两个文件 docker cp 进容器:
|
||||
# scp backend/scripts/{check_ai_issue_gates,_bootstrap}.py <host>:/tmp/
|
||||
# ssh <host> "docker cp /tmp/check_ai_issue_gates.py csctalon-backend:/app/scripts/ &&
|
||||
|
||||
@@ -1,10 +1,10 @@
|
||||
#!/usr/bin/env python3
|
||||
"""金山文档(kdocs.cn / WPS)扫码登录 → 生成 storage_state
|
||||
"""金山文档(kdocs.cn / WPS)扫码登录 CLI → 生成 storage_state
|
||||
|
||||
背景:
|
||||
金山共享表格(如 https://www.kdocs.cn/l/...) 未登录只能看 canvas, 无法导出;
|
||||
登录后可"导出为 Excel", 走标准 xlsx 解析。
|
||||
登录态存到 BROWSER_STATE_DIR/kdocs.json, 后续脚本复用。
|
||||
登录态存到 BROWSER_STATE_DIR/kdocs.json, 后续共享文档同步复用。
|
||||
|
||||
两种模式:
|
||||
① GUI 模式(默认, 本机用): 弹出真实浏览器, 用户扫码后按回车保存。
|
||||
@@ -13,11 +13,11 @@
|
||||
|
||||
用法:
|
||||
# 本机 GUI 扫码
|
||||
cd backend && ./.venv/bin/python ./scripts/kdocs_login.py
|
||||
cd backend && ./.venv/bin/python -m app.services.shared_doc_kdocs_cli
|
||||
|
||||
# 生产 Headless 扫码
|
||||
docker exec -e PYTHONPATH=/app csctalon-backend \
|
||||
python /app/scripts/kdocs_login.py --headless \
|
||||
# 生产 Headless 扫码(已预装于镜像, 可直接 docker exec)
|
||||
docker exec csctalon-backend \
|
||||
python -m app.services.shared_doc_kdocs_cli --headless \
|
||||
--qr-out /tmp/kdocs_qr.png --timeout 300
|
||||
|
||||
#148 防退化:
|
||||
@@ -26,14 +26,9 @@
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
_ROOT = Path(__file__).resolve().parents[1]
|
||||
# 脚本位于 backend/scripts/ 下, _ROOT 即 backend/ 目录, 直接加入 sys.path 即可 import app
|
||||
sys.path.insert(0, str(_ROOT))
|
||||
|
||||
from app.core.config import settings
|
||||
|
||||
TARGET_URL = "https://www.kdocs.cn/l/cbZIHtt6obyf"
|
||||
@@ -71,6 +66,7 @@ def _open_doc_and_show_login(page) -> None:
|
||||
|
||||
def main() -> int:
|
||||
import argparse
|
||||
|
||||
from playwright.sync_api import sync_playwright
|
||||
|
||||
ap = argparse.ArgumentParser(description="金山文档扫码登录, 生成 kdocs.json")
|
||||
@@ -110,7 +106,7 @@ def main() -> int:
|
||||
qr_path = Path(args.qr_out)
|
||||
page.screenshot(path=str(qr_path), full_page=False)
|
||||
print(f"📸 二维码截图: {qr_path} ({qr_path.stat().st_size} 字节)")
|
||||
print(f" 👉 请把该文件传到本地, 用 WPS/微信扫码登录")
|
||||
print(" 👉 请把该文件传到本地, 用 WPS/微信扫码登录")
|
||||
print(f" 👉 扫码后脚本自动检测 __WPSENV__.user.id (超时 {args.timeout}s)…")
|
||||
|
||||
deadline = time.time() + args.timeout
|
||||
@@ -327,7 +327,7 @@ def check_single_worker() -> bool:
|
||||
logger.warning(
|
||||
"[shared-doc-login] 检测到多 worker 部署(WEB_CONCURRENCY>1 或 gunicorn -w N): "
|
||||
"页面内扫码登录已禁用(SharedDocLoginManager 是进程内单例, 跨 worker 会话不可达)。"
|
||||
"请改用 CLI 脚本(scripts/tencent_doc_login.py / kdocs_login.py), 或改回单 worker 部署。"
|
||||
"请改用 CLI 脚本(python -m app.services.shared_doc_tencent_cli / shared_doc_kdocs_cli), 或改回单 worker 部署。"
|
||||
"其他业务不受影响。"
|
||||
)
|
||||
return False
|
||||
@@ -933,11 +933,15 @@ class SharedDocLoginManager:
|
||||
而本机(macOS)有 GUI, 脚本 `headless=False` 可直接扫码。故改为**两段式**:
|
||||
① 本机生成 storage_state → ② scp 上传到宿主机挂载目录(容器内 /app/data/browser_states)
|
||||
"""
|
||||
script = "tencent_doc_login.py" if platform == "tencent" else "kdocs_login.py"
|
||||
cli_mod = (
|
||||
"app.services.shared_doc_tencent_cli"
|
||||
if platform == "tencent"
|
||||
else "app.services.shared_doc_kdocs_cli"
|
||||
)
|
||||
state_file = DEFAULT_STATE_FILE.get(platform, f"{platform}.json")
|
||||
return (
|
||||
f"# ① 本地(macOS 有 GUI)生成登录态; 脚本会弹浏览器, 扫码后回车保存\n"
|
||||
f'cd backend && ./.venv/bin/python ../scripts/{script} --url "{doc_url}"\n'
|
||||
f'cd backend && ./.venv/bin/python -m {cli_mod} --url "{doc_url}"\n'
|
||||
f"# ② 上传到生产宿主机(挂载点 → 容器内 /app/data/browser_states)\n"
|
||||
f"scp backend/data/browser_states/{state_file} tanzk@<服务器IP>:/tmp/ && \\\n"
|
||||
f" ssh tanzk@<服务器IP> \"sudo mv /tmp/{state_file} /opt/csctalon/browser_states/\""
|
||||
|
||||
@@ -349,7 +349,7 @@ def _export_kdocs_xlsx(doc_url: str, out_path: str, state_filename: str | None =
|
||||
state_path = Path(settings.BROWSER_STATE_DIR) / (state_filename or _DEFAULT_STATE_FILE[PLATFORM_KDOCS])
|
||||
if not state_path.exists():
|
||||
raise FileNotFoundError(
|
||||
f"未找到金山登录态 {state_path}。请先运行 scripts/kdocs_login.py 扫码登录"
|
||||
f"未找到金山登录态 {state_path}。请先运行 docker exec csctalon-backend python -m app.services.shared_doc_kdocs_cli --headless 扫码登录"
|
||||
)
|
||||
out = str(Path(out_path).resolve())
|
||||
with sync_playwright() as p:
|
||||
|
||||
@@ -1,10 +1,10 @@
|
||||
#!/usr/bin/env python3
|
||||
"""腾讯文档扫码登录 → 生成 storage_state(供共享文档读取/导出复用)
|
||||
"""腾讯文档扫码登录 CLI → 生成 storage_state(供共享文档读取/导出复用)
|
||||
|
||||
背景:
|
||||
共享表格(如 https://docs.qq.com/sheet/...) 未登录只能读腾讯私有压缩格式;
|
||||
用账号登录后可"导出为 Excel", 走标准 xlsx 解析(复用 _poc_shared_doc_reader)。
|
||||
登录态存到 BROWSER_STATE_DIR/tencent_doc.json, 后续脚本复用(免二次扫码)。
|
||||
用账号登录后可"导出为 Excel", 走标准 xlsx 解析。
|
||||
登录态存到 BROWSER_STATE_DIR/tencent_doc.json, 后续共享文档同步复用。
|
||||
|
||||
两种模式:
|
||||
① GUI 模式(默认, 本机用): 弹出真实浏览器, 用户扫码后按回车保存。
|
||||
@@ -13,15 +13,13 @@
|
||||
|
||||
用法:
|
||||
# 本机 GUI 扫码
|
||||
cd backend && ./.venv/bin/python ./scripts/tencent_doc_login.py
|
||||
cd backend && ./.venv/bin/python -m app.services.shared_doc_tencent_cli
|
||||
|
||||
# 生产 Headless 扫码(二维码截图到 /tmp/tencent_qr.png, 运维 scp 到本地扫码)
|
||||
docker exec -e PYTHONPATH=/app csctalon-backend \
|
||||
python /app/scripts/tencent_doc_login.py --headless \
|
||||
# 生产 Headless 扫码(已预装于镜像, 可直接 docker exec)
|
||||
docker exec csctalon-backend \
|
||||
python -m app.services.shared_doc_tencent_cli --headless \
|
||||
--qr-out /tmp/tencent_qr.png --timeout 300
|
||||
|
||||
登录态文件安全: 存于 data/browser_states/(独立于 uploads, 不对外暴露)。
|
||||
|
||||
#148 防退化:
|
||||
保存前必须双重校验:
|
||||
① body 不含 skeleton-guest-mode 类(访客模式标识)
|
||||
@@ -30,14 +28,9 @@
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
_ROOT = Path(__file__).resolve().parents[1]
|
||||
# 脚本位于 backend/scripts/ 下, _ROOT 即 backend/ 目录, 直接加入 sys.path 即可 import app
|
||||
sys.path.insert(0, str(_ROOT))
|
||||
|
||||
from app.core.config import settings
|
||||
|
||||
# 默认打开的目标共享表(可用 --url 覆盖; 登录态为账号级)
|
||||
@@ -104,6 +97,7 @@ def _open_login_and_show_qr(page) -> None:
|
||||
|
||||
def main() -> int:
|
||||
import argparse
|
||||
|
||||
from playwright.sync_api import sync_playwright
|
||||
|
||||
ap = argparse.ArgumentParser(description="腾讯文档扫码登录, 生成 tencent_doc.json")
|
||||
@@ -140,12 +134,12 @@ def main() -> int:
|
||||
|
||||
if headless:
|
||||
# ── Headless 模式: 截图二维码 + 轮询登录态 ──
|
||||
print(f"🌐 [headless] 打开腾讯首页并触发登录弹窗…")
|
||||
print("🌐 [headless] 打开腾讯首页并触发登录弹窗…")
|
||||
_open_login_and_show_qr(page)
|
||||
qr_path = Path(args.qr_out)
|
||||
page.screenshot(path=str(qr_path), full_page=False)
|
||||
print(f"📸 二维码截图: {qr_path} ({qr_path.stat().st_size} 字节)")
|
||||
print(f" 👉 请把该文件传到本地, 用微信扫码登录")
|
||||
print(" 👉 请把该文件传到本地, 用微信扫码登录")
|
||||
print(f" 👉 扫码后脚本自动检测真登录 cookie (超时 {args.timeout}s)…")
|
||||
|
||||
deadline = time.time() + args.timeout
|
||||
@@ -285,7 +285,7 @@ def test_cli_fallback_command_is_two_stage_and_container_safe(mgr):
|
||||
(容器无 scripts/ 无 .venv), 属假降级路径。现要求两段式: 本机扫码 → scp 上传。
|
||||
"""
|
||||
cmd = mgr.cli_fallback_command("tencent", "https://docs.qq.com/sheet/AAA")
|
||||
assert "tencent_doc_login.py" in cmd
|
||||
assert "shared_doc_tencent_cli" in cmd
|
||||
assert "https://docs.qq.com/sheet/AAA" in cmd
|
||||
# 必须含上传段(证明是两段式而非旧的一行假命令)
|
||||
assert "scp" in cmd
|
||||
@@ -296,9 +296,9 @@ def test_cli_fallback_command_is_two_stage_and_container_safe(mgr):
|
||||
|
||||
|
||||
def test_cli_fallback_command_kdocs(mgr):
|
||||
"""正向: 金山平台降级命令指向 kdocs 脚本与 kdocs.json。"""
|
||||
"""正向: 金山平台降级命令指向 kdocs CLI 模块与 kdocs.json。"""
|
||||
cmd = mgr.cli_fallback_command("kdocs", "https://www.kdocs.cn/l/BBB")
|
||||
assert "kdocs_login.py" in cmd
|
||||
assert "shared_doc_kdocs_cli" in cmd
|
||||
assert "kdocs.json" in cmd
|
||||
|
||||
|
||||
|
||||
@@ -47,7 +47,8 @@ echo "${VERSION}" > "$ROOT_DIR/VERSION.tmp"
|
||||
# 注意: 根 data/logs/dist/backups 不在显式清单里, 无需 exclude;
|
||||
# tar --exclude 不能写 data/logs(会匹配任意层级, 误伤 web/src/views/data/
|
||||
# 源码目录——曾导致前端 Ingest.vue 缺失构建失败)
|
||||
# backend/scripts: 先整体排除调试脚本, 再用 tar r 追加仅两个登录脚本(#148)
|
||||
# backend/scripts: 整体排除调试脚本(登录脚本已于 2026-10-20 迁入 app/services/,
|
||||
# 不再需要从 scripts/ 白名单放行, 也不再从这里追加到 tar)
|
||||
TMP_TAR="${OUT_FILE%.gz}"
|
||||
tar cf "$TMP_TAR" \
|
||||
--exclude='.git' \
|
||||
@@ -81,26 +82,7 @@ tar cf "$TMP_TAR" \
|
||||
web \
|
||||
deploy \
|
||||
VERSION.tmp 2>/dev/null
|
||||
# 追加仅两个登录脚本(运维工具, 生产需 headless 扫码更新登录态)
|
||||
# 2026-10-14 修复: 两脚本已移入 backend/scripts/archive/, 原 `tar rf <路径> 2>/dev/null`
|
||||
# 在 set -e 下静默中止 → 不产出 .tar.gz / latest.tar.gz, 方式B发布不可用且无报错。
|
||||
# 现改为: canonical 路径缺失时回退 archive/, 经暂存目录以标准路径追加;
|
||||
# 均缺失则告警跳过(不阻断打包), 绝不静默。
|
||||
_LOGIN_STAGE="$(mktemp -d)"
|
||||
mkdir -p "$_LOGIN_STAGE/backend/scripts"
|
||||
for _name in tencent_doc_login.py kdocs_login.py; do
|
||||
if [[ -f "backend/scripts/$_name" ]]; then
|
||||
cp "backend/scripts/$_name" "$_LOGIN_STAGE/backend/scripts/$_name"
|
||||
elif [[ -f "backend/scripts/archive/$_name" ]]; then
|
||||
cp "backend/scripts/archive/$_name" "$_LOGIN_STAGE/backend/scripts/$_name"
|
||||
else
|
||||
echo "⚠ 登录脚本缺失(已跳过): backend/scripts/$_name" >&2
|
||||
fi
|
||||
done
|
||||
if [[ -n "$(ls -A "$_LOGIN_STAGE/backend/scripts" 2>/dev/null)" ]]; then
|
||||
tar rf "$TMP_TAR" -C "$_LOGIN_STAGE" backend/scripts
|
||||
fi
|
||||
rm -rf "$_LOGIN_STAGE"
|
||||
# login 脚本已在 app/services/ 下, 随 backend/ 整体打包自然进入发布包
|
||||
gzip -f "$TMP_TAR"
|
||||
rm -f "$ROOT_DIR/VERSION.tmp"
|
||||
|
||||
|
||||
@@ -118,9 +118,9 @@ if [[ $GIT_MODE -eq 1 ]]; then
|
||||
"$RELEASE_DIR/backend/tests" \
|
||||
"$RELEASE_DIR/backend/uploads" \
|
||||
"$RELEASE_DIR/VERSION.tmp"
|
||||
# backend/scripts/ 仅保留共享文档扫码登录脚本(运维工具, #148), 其余调试脚本删除
|
||||
find "$RELEASE_DIR/backend/scripts" -type f \
|
||||
! -name "tencent_doc_login.py" ! -name "kdocs_login.py" -delete 2>/dev/null || true
|
||||
# backend/scripts/ 清空所有调试/归档脚本(生产镜像不再从 scripts/ 取内容)
|
||||
# 登录脚本已于 2026-10-20 迁入 app/services/shared_doc_*_cli.py, 不再从 scripts/ 走
|
||||
rm -rf "$RELEASE_DIR/backend/scripts" 2>/dev/null || true
|
||||
else
|
||||
c "解压发布包 → $RELEASE_DIR"
|
||||
tar xzf "$PKG" -C "$RELEASE_DIR"
|
||||
|
||||
Reference in New Issue
Block a user