Files
Edu/docs/standards/full-stack-runbook.md
SpecialX f1e466a772
Some checks failed
CI / quality-go (push) Failing after 5s
CI / quality-proto (push) Failing after 3s
CI / deploy (push) Has been skipped
CI / quality-ts (push) Failing after 50s
fix(infra): resolve NestJS dist build and Prometheus target issues
NestJS: disable incremental in 6 services tsconfig.json to fix dist
not emitted when nest-cli deleteOutDir conflicts with tsc tsbuildinfo.
classes/iam: import HealthModule in AppModule to fix /healthz 404.
classes: rewrite HealthController to Drizzle getDb from TypeORM DI.
teacher-bff: add /metrics endpoint for Prometheus scraping.
infra: add node/mysql/redis exporters to observability profile.
mysql-exporter v0.15.1 uses command-line flags not DATA_SOURCE_NAME.
prometheus: enable web.enable-lifecycle for hot reload.
2026-07-09 15:12:15 +08:00

11 KiB
Raw Permalink Blame History

全链路启用与测试手册Full Stack Runbook

版本1.0 日期2026-07-09 适用范围Edu 微服务架构P6 阶段11 基础设施 + 10 应用服务 + 1 前端) 关联文档:local-dev-runbook004 架构影响地图known-issues


1. 前置条件

工具 版本要求 验证命令
Node.js ≥ 20 node -v
pnpm ≥ 9 pnpm -v
Go 1.22+ go version
uv 0.4+ uv --version
Docker 24+ docker version
Docker Compose v2+ docker compose version

Windows 用户Go 工具链若不在 PATH临时加入$env:Path = "C:\Program Files\Go\bin;" + $env:Path


2. 服务端口矩阵

2.1 应用服务10 个)

端口 服务 语言/框架 启动方式
3000 teacher-portal Next.js pnpm --filter teacher-portal dev
3001 classes NestJS pnpm --filter @edu/classes-service dev
3002 iam NestJS pnpm --filter @edu/iam-service dev
3003 teacher-bff NestJS pnpm --filter @edu/teacher-bff dev
3004 core-edu NestJS pnpm --filter @edu/core-edu-service dev
3005 content NestJS pnpm --filter @edu/content-service dev
3006 data-ana FastAPI uv run uvicorn data_ana.main:app --port 3006
3007 msg NestJS pnpm --filter @edu/msg-service dev
3008 ai FastAPI uv run uvicorn ai.main:app --port 3008
8080 api-gateway Go (Gin) go run .
8081 push-gateway Go (Gin) go run .

2.2 基础设施11 个)

端口 服务 用途
3306 MySQL 8 写模型主库
6379 Redis 7 缓存/会话
8083 Debezium Connect CDC source connector
8123/9000 ClickHouse 24.3 读模型宽表
9092 Kafka 7.6 事件总线
7474/7687 Neo4j 5.20 知识图谱
9200 Elasticsearch 8.13 题库检索
9090 Prometheus 指标采集
3030 Grafana 可视化admin/admin
16686 Jaeger 分布式追踪 UI
4318 OTLP Collector OTel span 接收

3. 一键启动

3.1 启动基础设施

cd e:\Desktop\Edu\infra
docker compose -f docker-compose.yml --profile p6 --profile observability up -d

等待所有容器 healthy约 60 秒):

docker ps --filter "name=edu-" --format "table {{.Names}}\t{{.Status}}"

3.2 一键启动所有应用服务

使用项目根目录的启动脚本:

cd e:\Desktop\Edu
.\scripts\start-all.ps1

该脚本会:

  1. 检查基础设施健康状态
  2. 为每个应用服务启动独立后台窗口(带标题)
  3. 自动注入 Python 服务的环境变量CLICKHOUSE/KAFKA/OTEL
  4. 等待所有服务健康检查通过

3.3 健康检查

cd e:\Desktop\Edu
.\scripts\health-check.ps1

预期输出:所有服务


4. 端到端链路测试

4.1 IAM 注册 + 登录

$h = @{Authorization="Bearer dev-token"}

# 注册
$body = @{username="testteacher";password="Test@1234";email="test@edu.com";role="teacher"} | ConvertTo-Json
Invoke-RestMethod -Uri "http://localhost:8080/iam/auth/register" -Method Post -Body $body -ContentType "application/json" -Headers $h

# 登录
$loginBody = @{username="testteacher";password="Test@1234"} | ConvertTo-Json
$resp = Invoke-RestMethod -Uri "http://localhost:8080/iam/auth/login" -Method Post -Body $loginBody -ContentType "application/json"
$token = $resp.data.accessToken
Write-Host "Token: $token"

4.2 Classes CRUD

# 创建班级
$classBody = @{name="高三一班";gradeId="grade-1";headTeacherId=""} | ConvertTo-Json
Invoke-RestMethod -Uri "http://localhost:8080/classes" -Method Post -Body $classBody -ContentType "application/json" -Headers $h

# 查询班级列表
Invoke-RestMethod -Uri "http://localhost:8080/classes" -Method Get -Headers $h

4.3 CDC 完整链路MySQL → Debezium → Kafka → data-ana → ClickHouse

# 1. 向 MySQL 插入成绩(触发 binlog
docker exec edu-mysql mysql -uedu -pchangeme next_edu_cloud -e "
INSERT INTO core_edu_exams (id, class_id, subject_id, title, exam_date, total_score, created_at, updated_at)
VALUES ('exam-cdc-test-001','cls-test-001','sub-math','CDC测试考试',NOW(),100,NOW(),NOW())
ON DUPLICATE KEY UPDATE updated_at=NOW();

INSERT INTO core_edu_grades (id, exam_id, student_id, score, rank_in_class, created_at, updated_at)
VALUES ('grade-cdc-001','exam-cdc-test-001','student-cdc-001',92.5,1,NOW(),NOW())
ON DUPLICATE KEY UPDATE score=92.5, updated_at=NOW();
"

# 2. 等待 Debezium 捕获 + data-ana 消费
Start-Sleep -Seconds 5

# 3. 验证 ClickHouse 已同步
docker exec edu-clickhouse clickhouse-client --user default --password clickhouse -q "
SELECT student_id, class_id, exam_id, score, last_updated
FROM edu_analytics.student_dashboard_view
WHERE student_id = 'student-cdc-001'
ORDER BY last_updated DESC
"
# 预期返回一行score=92.5class_id='cls-test-001'

4.4 data-ana 查询 API

# 学生学情看板
Invoke-RestMethod -Uri "http://localhost:3006/analytics/student/student-cdc-001/weakness" -Headers $h

# 班级成绩分析
Invoke-RestMethod -Uri "http://localhost:3006/analytics/class/cls-test-001/performance" -Headers $h

# CDC 消费者状态
Invoke-RestMethod -Uri "http://localhost:3006/readyz"
# 预期: cdc_consumer = "running"

4.5 可观测性验证

检查项 URL 预期
Prometheus targets http://localhost:9090/targets 所有 target UP
Grafana 面板 http://localhost:3030 (admin/admin) 可登录
Jaeger UI http://localhost:16686 可搜索到各 service trace
data-ana /metrics http://localhost:3006/metrics Prometheus 格式输出
iam /metrics http://localhost:3002/metrics Prometheus 格式输出

Jaeger trace 验证步骤

  1. 打开 http://localhost:16686
  2. Service 下拉框应能看到 iamclassesdata-anaapi-gateway
  3. 选择任一服务 → Find Traces → 应看到 HTTP 请求的自动埋点 span

5. 一键停止

5.1 停止应用服务

cd e:\Desktop\Edu
.\scripts\stop-all.ps1

.\scripts\stop-all.ps1 -KillByPort

该脚本会关闭所有 edu-app-* 标题的终端窗口。

5.2 停止基础设施

cd e:\Desktop\Edu\infra
docker compose -f docker-compose.yml --profile p6 --profile observability down

6. 故障排查

6.1 端口占用

# 查看占用端口的进程
netstat -ano | findstr :3001
# 终止进程
taskkill /PID <PID> /F

6.2 基础设施未启动

# 检查容器状态
docker ps --filter "name=edu-"

# 重启单个容器
docker restart edu-mysql

# 查看日志
docker logs edu-debezium --tail 50

6.3 CDC 链路断开

# 1. 检查 Debezium connector 状态
Invoke-RestMethod -Uri "http://localhost:8083/connectors/edu-mysql-source/status"

# 2. 重启 connector
Invoke-RestMethod -Uri "http://localhost:8083/connectors/edu-mysql-source/restart" -Method Post

# 3. 检查 Kafka topic 是否有数据
docker exec edu-kafka kafka-console-consumer --bootstrap-server localhost:9092 --topic edu-cdc.next_edu_cloud.core_edu_grades --from-beginning --max-messages 1

# 4. 检查 data-ana 消费者日志
# 查看 data-ana 终端窗口的 cdc_consumer_started / cdc_event_received 日志

6.4 OTel trace 未上报

# 1. 检查 Jaeger 是否收到 trace
Invoke-RestMethod -Uri "http://localhost:16686/api/services"

# 2. 检查 OTLP endpoint 是否可达
Invoke-RestMethod -Uri "http://localhost:4318/v1/traces" -Method Post -ContentType "application/json" -Body "{}"

# 3. 检查服务环境变量
# 确保 OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 已设置

7. 速查:常用命令

场景 命令
启动基础设施 docker compose -f infra/docker-compose.yml --profile p6 --profile observability up -d
一键启动应用 .\scripts\start-all.ps1
一键停止应用 .\scripts\stop-all.ps1
健康检查 .\scripts\health-check.ps1
CDC 链路验证 .\scripts\test-cdc.ps1
查看容器状态 docker ps --filter "name=edu-"
查看 Debezium 状态 Invoke-RestMethod http://localhost:8083/connectors/edu-mysql-source/status
ClickHouse 查询 docker exec edu-clickhouse clickhouse-client --user default --password clickhouse -q "SELECT * FROM edu_analytics.student_dashboard_view LIMIT 10"
Kafka topic 列表 docker exec edu-kafka kafka-topics --bootstrap-server localhost:9092 --list
Prometheus 查询 Invoke-RestMethod "http://localhost:9090/api/v1/query?query=up"