3 Commits

Author SHA1 Message Date
SpecialX
f1e466a772 fix(infra): resolve NestJS dist build and Prometheus target issues
Some checks failed
CI / quality-go (push) Failing after 5s
CI / quality-proto (push) Failing after 3s
CI / deploy (push) Has been skipped
CI / quality-ts (push) Failing after 50s
NestJS: disable incremental in 6 services tsconfig.json to fix dist
not emitted when nest-cli deleteOutDir conflicts with tsc tsbuildinfo.
classes/iam: import HealthModule in AppModule to fix /healthz 404.
classes: rewrite HealthController to Drizzle getDb from TypeORM DI.
teacher-bff: add /metrics endpoint for Prometheus scraping.
infra: add node/mysql/redis exporters to observability profile.
mysql-exporter v0.15.1 uses command-line flags not DATA_SOURCE_NAME.
prometheus: enable web.enable-lifecycle for hot reload.
2026-07-09 15:12:15 +08:00
SpecialX
b72c8d81d4 fix(infra): use conditional docker template to avoid Health key error
Some checks failed
CI / quality-ts (push) Failing after 1m0s
CI / quality-go (push) Failing after 4s
CI / quality-proto (push) Failing after 2s
CI / deploy (push) Has been skipped
Use {{if .State.Health}}...{{end}} instead of direct {{.State.Health.Status}}.
2026-07-09 13:58:55 +08:00
SpecialX
959d58a95d docs: add full-stack runbook and one-click scripts
Some checks failed
CI / quality-ts (push) Failing after 51s
CI / quality-go (push) Failing after 3s
CI / quality-proto (push) Failing after 2s
CI / deploy (push) Has been skipped
start-all/stop-all/health-check/test-cdc PowerShell scripts.

Compatible with PowerShell 5.1 (ASCII only, no emoji).
2026-07-09 13:47:46 +08:00
19 changed files with 1139 additions and 36 deletions

View File

@@ -0,0 +1,281 @@
# 全链路启用与测试手册Full Stack Runbook
> 版本1.0
> 日期2026-07-09
> 适用范围Edu 微服务架构P6 阶段11 基础设施 + 10 应用服务 + 1 前端)
> 关联文档:[local-dev-runbook](./local-dev-runbook.md)、[004 架构影响地图](../architecture/004_architecture_impact_map.md)、[known-issues](../troubleshooting/known-issues.md)
---
## 1. 前置条件
| 工具 | 版本要求 | 验证命令 |
| -------------- | -------- | ------------------------ |
| Node.js | ≥ 20 | `node -v` |
| pnpm | ≥ 9 | `pnpm -v` |
| Go | 1.22+ | `go version` |
| uv | 0.4+ | `uv --version` |
| Docker | 24+ | `docker version` |
| Docker Compose | v2+ | `docker compose version` |
> Windows 用户Go 工具链若不在 PATH临时加入`$env:Path = "C:\Program Files\Go\bin;" + $env:Path`
---
## 2. 服务端口矩阵
### 2.1 应用服务10 个)
| 端口 | 服务 | 语言/框架 | 启动方式 |
| ---- | -------------- | --------- | ---------------------------------------------- |
| 3000 | teacher-portal | Next.js | `pnpm --filter teacher-portal dev` |
| 3001 | classes | NestJS | `pnpm --filter @edu/classes-service dev` |
| 3002 | iam | NestJS | `pnpm --filter @edu/iam-service dev` |
| 3003 | teacher-bff | NestJS | `pnpm --filter @edu/teacher-bff dev` |
| 3004 | core-edu | NestJS | `pnpm --filter @edu/core-edu-service dev` |
| 3005 | content | NestJS | `pnpm --filter @edu/content-service dev` |
| 3006 | data-ana | FastAPI | `uv run uvicorn data_ana.main:app --port 3006` |
| 3007 | msg | NestJS | `pnpm --filter @edu/msg-service dev` |
| 3008 | ai | FastAPI | `uv run uvicorn ai.main:app --port 3008` |
| 8080 | api-gateway | Go (Gin) | `go run .` |
| 8081 | push-gateway | Go (Gin) | `go run .` |
### 2.2 基础设施11 个)
| 端口 | 服务 | 用途 |
| --------- | ------------------ | --------------------- |
| 3306 | MySQL 8 | 写模型主库 |
| 6379 | Redis 7 | 缓存/会话 |
| 8083 | Debezium Connect | CDC source connector |
| 8123/9000 | ClickHouse 24.3 | 读模型宽表 |
| 9092 | Kafka 7.6 | 事件总线 |
| 7474/7687 | Neo4j 5.20 | 知识图谱 |
| 9200 | Elasticsearch 8.13 | 题库检索 |
| 9090 | Prometheus | 指标采集 |
| 3030 | Grafana | 可视化admin/admin |
| 16686 | Jaeger | 分布式追踪 UI |
| 4318 | OTLP Collector | OTel span 接收 |
---
## 3. 一键启动
### 3.1 启动基础设施
```powershell
cd e:\Desktop\Edu\infra
docker compose -f docker-compose.yml --profile p6 --profile observability up -d
```
等待所有容器 healthy约 60 秒):
```powershell
docker ps --filter "name=edu-" --format "table {{.Names}}\t{{.Status}}"
```
### 3.2 一键启动所有应用服务
使用项目根目录的启动脚本:
```powershell
cd e:\Desktop\Edu
.\scripts\start-all.ps1
```
该脚本会:
1. 检查基础设施健康状态
2. 为每个应用服务启动独立后台窗口(带标题)
3. 自动注入 Python 服务的环境变量CLICKHOUSE/KAFKA/OTEL
4. 等待所有服务健康检查通过
### 3.3 健康检查
```powershell
cd e:\Desktop\Edu
.\scripts\health-check.ps1
```
预期输出:所有服务 ✅
---
## 4. 端到端链路测试
### 4.1 IAM 注册 + 登录
```powershell
$h = @{Authorization="Bearer dev-token"}
# 注册
$body = @{username="testteacher";password="Test@1234";email="test@edu.com";role="teacher"} | ConvertTo-Json
Invoke-RestMethod -Uri "http://localhost:8080/iam/auth/register" -Method Post -Body $body -ContentType "application/json" -Headers $h
# 登录
$loginBody = @{username="testteacher";password="Test@1234"} | ConvertTo-Json
$resp = Invoke-RestMethod -Uri "http://localhost:8080/iam/auth/login" -Method Post -Body $loginBody -ContentType "application/json"
$token = $resp.data.accessToken
Write-Host "Token: $token"
```
### 4.2 Classes CRUD
```powershell
# 创建班级
$classBody = @{name="高三一班";gradeId="grade-1";headTeacherId=""} | ConvertTo-Json
Invoke-RestMethod -Uri "http://localhost:8080/classes" -Method Post -Body $classBody -ContentType "application/json" -Headers $h
# 查询班级列表
Invoke-RestMethod -Uri "http://localhost:8080/classes" -Method Get -Headers $h
```
### 4.3 CDC 完整链路MySQL → Debezium → Kafka → data-ana → ClickHouse
```powershell
# 1. 向 MySQL 插入成绩(触发 binlog
docker exec edu-mysql mysql -uedu -pchangeme next_edu_cloud -e "
INSERT INTO core_edu_exams (id, class_id, subject_id, title, exam_date, total_score, created_at, updated_at)
VALUES ('exam-cdc-test-001','cls-test-001','sub-math','CDC测试考试',NOW(),100,NOW(),NOW())
ON DUPLICATE KEY UPDATE updated_at=NOW();
INSERT INTO core_edu_grades (id, exam_id, student_id, score, rank_in_class, created_at, updated_at)
VALUES ('grade-cdc-001','exam-cdc-test-001','student-cdc-001',92.5,1,NOW(),NOW())
ON DUPLICATE KEY UPDATE score=92.5, updated_at=NOW();
"
# 2. 等待 Debezium 捕获 + data-ana 消费
Start-Sleep -Seconds 5
# 3. 验证 ClickHouse 已同步
docker exec edu-clickhouse clickhouse-client --user default --password clickhouse -q "
SELECT student_id, class_id, exam_id, score, last_updated
FROM edu_analytics.student_dashboard_view
WHERE student_id = 'student-cdc-001'
ORDER BY last_updated DESC
"
# 预期返回一行score=92.5class_id='cls-test-001'
```
### 4.4 data-ana 查询 API
```powershell
# 学生学情看板
Invoke-RestMethod -Uri "http://localhost:3006/analytics/student/student-cdc-001/weakness" -Headers $h
# 班级成绩分析
Invoke-RestMethod -Uri "http://localhost:3006/analytics/class/cls-test-001/performance" -Headers $h
# CDC 消费者状态
Invoke-RestMethod -Uri "http://localhost:3006/readyz"
# 预期: cdc_consumer = "running"
```
### 4.5 可观测性验证
| 检查项 | URL | 预期 |
| ------------------ | ----------------------------------- | ------------------------ |
| Prometheus targets | http://localhost:9090/targets | 所有 target UP |
| Grafana 面板 | http://localhost:3030 (admin/admin) | 可登录 |
| Jaeger UI | http://localhost:16686 | 可搜索到各 service trace |
| data-ana /metrics | http://localhost:3006/metrics | Prometheus 格式输出 |
| iam /metrics | http://localhost:3002/metrics | Prometheus 格式输出 |
**Jaeger trace 验证步骤**
1. 打开 http://localhost:16686
2. Service 下拉框应能看到 `iam``classes``data-ana``api-gateway`
3. 选择任一服务 → Find Traces → 应看到 HTTP 请求的自动埋点 span
---
## 5. 一键停止
### 5.1 停止应用服务
```powershell
cd e:\Desktop\Edu
.\scripts\stop-all.ps1
.\scripts\stop-all.ps1 -KillByPort
```
该脚本会关闭所有 `edu-app-*` 标题的终端窗口。
### 5.2 停止基础设施
```powershell
cd e:\Desktop\Edu\infra
docker compose -f docker-compose.yml --profile p6 --profile observability down
```
---
## 6. 故障排查
### 6.1 端口占用
```powershell
# 查看占用端口的进程
netstat -ano | findstr :3001
# 终止进程
taskkill /PID <PID> /F
```
### 6.2 基础设施未启动
```powershell
# 检查容器状态
docker ps --filter "name=edu-"
# 重启单个容器
docker restart edu-mysql
# 查看日志
docker logs edu-debezium --tail 50
```
### 6.3 CDC 链路断开
```powershell
# 1. 检查 Debezium connector 状态
Invoke-RestMethod -Uri "http://localhost:8083/connectors/edu-mysql-source/status"
# 2. 重启 connector
Invoke-RestMethod -Uri "http://localhost:8083/connectors/edu-mysql-source/restart" -Method Post
# 3. 检查 Kafka topic 是否有数据
docker exec edu-kafka kafka-console-consumer --bootstrap-server localhost:9092 --topic edu-cdc.next_edu_cloud.core_edu_grades --from-beginning --max-messages 1
# 4. 检查 data-ana 消费者日志
# 查看 data-ana 终端窗口的 cdc_consumer_started / cdc_event_received 日志
```
### 6.4 OTel trace 未上报
```powershell
# 1. 检查 Jaeger 是否收到 trace
Invoke-RestMethod -Uri "http://localhost:16686/api/services"
# 2. 检查 OTLP endpoint 是否可达
Invoke-RestMethod -Uri "http://localhost:4318/v1/traces" -Method Post -ContentType "application/json" -Body "{}"
# 3. 检查服务环境变量
# 确保 OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 已设置
```
---
## 7. 速查:常用命令
| 场景 | 命令 |
| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
| 启动基础设施 | `docker compose -f infra/docker-compose.yml --profile p6 --profile observability up -d` |
| 一键启动应用 | `.\scripts\start-all.ps1` |
| 一键停止应用 | `.\scripts\stop-all.ps1` |
| 健康检查 | `.\scripts\health-check.ps1` |
| CDC 链路验证 | `.\scripts\test-cdc.ps1` |
| 查看容器状态 | `docker ps --filter "name=edu-"` |
| 查看 Debezium 状态 | `Invoke-RestMethod http://localhost:8083/connectors/edu-mysql-source/status` |
| ClickHouse 查询 | `docker exec edu-clickhouse clickhouse-client --user default --password clickhouse -q "SELECT * FROM edu_analytics.student_dashboard_view LIMIT 10"` |
| Kafka topic 列表 | `docker exec edu-kafka kafka-topics --bootstrap-server localhost:9092 --list` |
| Prometheus 查询 | `Invoke-RestMethod "http://localhost:9090/api/v1/query?query=up"` |

View File

@@ -187,22 +187,25 @@
### 2.2 classesTS/NestJSP1 黄金模板) ### 2.2 classesTS/NestJSP1 黄金模板)
| 场景 | 技术/规则 | | 场景 | 技术/规则 |
| ---------------- | ------------------------------------------------------------------------------------------------ | | ---------------- | -------------------------------------------------------------------------------------------------- |
| 黄金模板定位 | P1 完整实现所有横切关注点,后续 8 个服务复制此模板 | | 黄金模板定位 | P1 完整实现所有横切关注点,后续 8 个服务复制此模板 |
| 黄金模板复制流程 | `cp -r services/classes services/xxx` → 改错误码前缀 → 改 proto → 改业务逻辑 → 改 README → 改 CI | | 黄金模板复制流程 | `cp -r services/classes services/xxx` → 改错误码前缀 → 改 proto → 改业务逻辑 → 改 README → 改 CI |
| 横切关注点清单 | 错误处理 / 可观测 / 安全 / 契约 / 测试 / 文档 / 配置 / i18n / CI / Dockerfile | | 横切关注点清单 | 错误处理 / 可观测 / 安全 / 契约 / 测试 / 文档 / 配置 / i18n / CI / Dockerfile |
| 错误处理 | `ApplicationError` 基类 + 子类,错误码 `CLASSES_*` 前缀 | | 错误处理 | `ApplicationError` 基类 + 子类,错误码 `CLASSES_*` 前缀 |
| 可观测 | pino logger + prom-client metrics + OTel tracer | | 可观测 | pino logger + prom-client metrics + OTel tracer |
| 安全 | auth.middleware信任 Gateway 头)+ permission.guard + data-scope.interceptor | | 安全 | auth.middleware信任 Gateway 头)+ permission.guard + data-scope.interceptor |
| 配置 | 三层配置 + Zod 校验 env | | 配置 | 三层配置 + Zod 校验 env |
| i18n | `ERROR_CODES` 映射表(错误码 → i18n key | | i18n | `ERROR_CODES` 映射表(错误码 → i18n key |
| 测试四类 | 单元vitest+ 集成Testcontainers+ 契约Pact+ E2EPlaywright | | 测试四类 | 单元vitest+ 集成Testcontainers+ 契约Pact+ E2EPlaywright |
| 覆盖率门槛 | 领域逻辑 ≥ 80%Handler ≥ 60%,整体 ≥ 60% | | 覆盖率门槛 | 领域逻辑 ≥ 80%Handler ≥ 60%,整体 ≥ 60% |
| Drizzle schema | `mysqlTable` + `varchar`/`timestamp` + `index` | | Drizzle schema | `mysqlTable` + `varchar`/`timestamp` + `index` |
| ID 生成 | `@paralleldrive/cuid2` 的 `createId()` | | ID 生成 | `@paralleldrive/cuid2` 的 `createId()` |
| 响应转换 | repository 返回 Dateservice 转换为 `createdAt: number`(时间戳) | | 响应转换 | repository 返回 Dateservice 转换为 `createdAt: number`(时间戳) |
| 阶段特有模式回写 | OutboxP3/ CDCP4/ 长连接P5实现后回写黄金模板 README | | 阶段特有模式回写 | OutboxP3/ CDCP4/ 长连接P5实现后回写黄金模板 README |
| 健康检查依赖 | `readyz` 用 Drizzle `getDb().execute(sql\`SELECT 1\`)` 校验,不要依赖 typeorm DataSource DI |
| AppModule 注册 | HealthModule 必须在 `app.module.ts` imports 数组显式声明,否则 NestFactory 不扫描 HealthController |
| 增量编译陷阱 | `tsconfig.json` 显式 `"incremental": false` 覆盖 base避免 .tsbuildinfo 导致 nest watch 不 emit |
### 2.3 iamTS/NestJSP2 ### 2.3 iamTS/NestJSP2
@@ -350,6 +353,7 @@
| 日期 | 时间 | 模块 | 做了什么 + 学到什么 | | 日期 | 时间 | 模块 | 做了什么 + 学到什么 |
| ---------- | ---- | ------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ---------- | ---- | ------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 2026-07-09 | 下午 | classes/全局 | **一键启动脚本 NestJS dist/ 不生成根因定位 + classes 健康检查修复**(1) 根因定位:`tsconfig.base.json` 的 `incremental: true` + `nest-cli.json` 的 `deleteOutDir: true` 冲突。`nest start --watch` 启动时先删除 dist/tsc 读残留 .tsbuildinfo 认为无变化跳过 emitdist/ 不生成 → `Cannot find module dist/main`。(2) 修复6 个 NestJS 服务classes/iam/teacher-bff/core-edu/content/msgtsconfig.json 显式加 `"incremental": false` 覆盖 base 配置,删除所有残留 .tsbuildinfo 文件。(3) classes AppModule 缺 HealthModule 导入导致 /healthz 404iam 同样问题,修复 app.module.ts 加 `imports: [..., HealthModule]`。(4) classes HealthController 误用 TypeORM `DataSource` DI与 iam 不一致),运行时报 `Nest can't resolve dependencies of the HealthController (DataSource)`。修复:改为 Drizzle `getDb()` 函数式调用,与 iam 一致。(5) 一键启动验证11/11 应用 + 11/11 基础设施 + 5/5 可观测性端点全绿。**学到**NestJS + TypeScript incremental 编译是陷阱组合——nest-cli deleteOutDir 删 dist 但 tsc 读 tsbuildinfo 认为无变化,必须在服务级 tsconfig 显式 `incremental: false`HealthModule 必须在 AppModule imports 中显式声明才能被 NestFactory 扫描到5 个 NestJS 服务的 HealthController 应统一用 Drizzle `getDb()` 函数式调用而非 TypeORM DataSource DI项目已弃 TypeORM 改 Drizzle。 |
| 2026-07-09 | 下午 | 全局 | **OTel auto-instrumentations 全服务补全**(1) NestJS 6 服务iam/classes/core-edu/content/msg/teacher-bfftracer.ts 补 `getNodeAutoInstrumentations()`NodeSDK 传 instrumentations 参数自动埋点 HTTP/Express/DB。(2) Python 2 服务data-ana/aimain.py 补 `FastAPIInstrumentor.instrument_app(app)`ai 补缺失的 `opentelemetry-exporter-otlp` 依赖。(3) teacher-bff 从零补完整 OTelenv.ts 加 OTEL_EXPORTER_OTLP_ENDPOINT 字段 + 新建 shared/observability/tracer.ts + main.ts 调用 initTracer/shutdownTracer + package.json 加 sdk-node/exporter/auto-instrumentations 依赖。(4) Go 2 服务api-gateway/push-gateway新建 internal/observability/tracer.goOTLP HTTP exporter + resource + TracerProvider + W3C propagator+ main.go 调用 InitTracer + otelgin.Middleware 注册 Gin 中间件push-gateway config.go 补 OTLPEndpoint 字段。(5) 质量校验全通过TS typecheck 9 服务 + ESLint 6 服务 + ruff 2 服务 + go vet/build 2 服务零错误。**学到**`getNodeAutoInstrumentations()` 一次注册所有 Node.js 自动埋点http/express/dns/fs/net/grpc 等),比手动逐个注册 HttpInstrumentation 更简洁Go OTel 用 `otlptracehttp.WithEndpoint(host)` + `WithInsecure()` 需从 "http://host:port" URL 解析出 hostotelgin.Middleware 必须在 Recovery 之后其他中间件之前注册,确保所有后续 handler 都被 tracePython FastAPIInstrumentor.instrument_app(app) 在 app 创建后立即调用lifespan 不受影响。 | | 2026-07-09 | 下午 | 全局 | **OTel auto-instrumentations 全服务补全**(1) NestJS 6 服务iam/classes/core-edu/content/msg/teacher-bfftracer.ts 补 `getNodeAutoInstrumentations()`NodeSDK 传 instrumentations 参数自动埋点 HTTP/Express/DB。(2) Python 2 服务data-ana/aimain.py 补 `FastAPIInstrumentor.instrument_app(app)`ai 补缺失的 `opentelemetry-exporter-otlp` 依赖。(3) teacher-bff 从零补完整 OTelenv.ts 加 OTEL_EXPORTER_OTLP_ENDPOINT 字段 + 新建 shared/observability/tracer.ts + main.ts 调用 initTracer/shutdownTracer + package.json 加 sdk-node/exporter/auto-instrumentations 依赖。(4) Go 2 服务api-gateway/push-gateway新建 internal/observability/tracer.goOTLP HTTP exporter + resource + TracerProvider + W3C propagator+ main.go 调用 InitTracer + otelgin.Middleware 注册 Gin 中间件push-gateway config.go 补 OTLPEndpoint 字段。(5) 质量校验全通过TS typecheck 9 服务 + ESLint 6 服务 + ruff 2 服务 + go vet/build 2 服务零错误。**学到**`getNodeAutoInstrumentations()` 一次注册所有 Node.js 自动埋点http/express/dns/fs/net/grpc 等),比手动逐个注册 HttpInstrumentation 更简洁Go OTel 用 `otlptracehttp.WithEndpoint(host)` + `WithInsecure()` 需从 "http://host:port" URL 解析出 hostotelgin.Middleware 必须在 Recovery 之后其他中间件之前注册,确保所有后续 handler 都被 tracePython FastAPIInstrumentor.instrument_app(app) 在 app 创建后立即调用lifespan 不受影响。 |
| 2026-07-09 | 下午 | 全局 | **P6 硬化:可观测性 + 部署 + CI 硬化**(1) 可观测性栈完善5 个 NestJS 服务 main.ts 添加 `/metrics` Prometheus 端点(用 `app.getHttpAdapter().get('/metrics', ...)` 绕过 DI 容器 get 方法prometheus.yml 从 2 个目标扩展到 8 个应用服务 + MySQL/Redis + node-exporter + prometheus 自身 + rule_files + alertmanager 关联monitoring compose 用 Loki + Promtail 替换未配置的 blackbox-exporterGrafana datasource 新增 Loki新建 promtail/config.yml 用 docker_sd_configs 仅采集 `edu-*` 容器日志。(2) 部署 compose 扩展docker-compose.deploy.yml 从 3 服务扩展到 11 服务(+ iam/teacher-bff/core-edu/content/msg/ai/data-ana/push-gateway每个服务带 healthcheck + depends_on 条件 + edu-net/edu-shared 双网络deploy.env.example 补全 Neo4j/ES/ClickHouse/LLM/Kafka 可选依赖配置。(3) teacher-bff 补 health.controller.ts原缺失 /healthz 导致 deploy depends_on service_healthy 失败)。(4) CI 硬化:移除 lint 步骤的 continue-on-errorESLint 9 flat config 已配置完成test 保留 continue-on-error部分服务无 test 脚本)。**学到**NestJS `app.get('/metrics')` 会被解析为 DI 容器 `get(typeOrToken)`,必须用 `app.getHttpAdapter().get()` 才能注册 Express 路由Promtail docker_sd_configs 通过 relabel_configs 的 `regex: '/(edu-.*).*'` 过滤容器名前缀docker-compose.depends_on.condition: service_healthy 要求被依赖服务必须有 healthcheck 配置,否则启动失败。 | | 2026-07-09 | 下午 | 全局 | **P6 硬化:可观测性 + 部署 + CI 硬化**(1) 可观测性栈完善5 个 NestJS 服务 main.ts 添加 `/metrics` Prometheus 端点(用 `app.getHttpAdapter().get('/metrics', ...)` 绕过 DI 容器 get 方法prometheus.yml 从 2 个目标扩展到 8 个应用服务 + MySQL/Redis + node-exporter + prometheus 自身 + rule_files + alertmanager 关联monitoring compose 用 Loki + Promtail 替换未配置的 blackbox-exporterGrafana datasource 新增 Loki新建 promtail/config.yml 用 docker_sd_configs 仅采集 `edu-*` 容器日志。(2) 部署 compose 扩展docker-compose.deploy.yml 从 3 服务扩展到 11 服务(+ iam/teacher-bff/core-edu/content/msg/ai/data-ana/push-gateway每个服务带 healthcheck + depends_on 条件 + edu-net/edu-shared 双网络deploy.env.example 补全 Neo4j/ES/ClickHouse/LLM/Kafka 可选依赖配置。(3) teacher-bff 补 health.controller.ts原缺失 /healthz 导致 deploy depends_on service_healthy 失败)。(4) CI 硬化:移除 lint 步骤的 continue-on-errorESLint 9 flat config 已配置完成test 保留 continue-on-error部分服务无 test 脚本)。**学到**NestJS `app.get('/metrics')` 会被解析为 DI 容器 `get(typeOrToken)`,必须用 `app.getHttpAdapter().get()` 才能注册 Express 路由Promtail docker_sd_configs 通过 relabel_configs 的 `regex: '/(edu-.*).*'` 过滤容器名前缀docker-compose.depends_on.condition: service_healthy 要求被依赖服务必须有 healthcheck 配置,否则启动失败。 |
| 2026-07-09 | 下午 | data-ana/infra | **CDC 完整链路实现**MySQL binlog → Debezium Connect → Kafka → data-ana 消费者 → ClickHouse 宽表。(1) MySQL binlog 配置log_bin=ON, binlog_format=ROW, binlog_row_image=FULL, server_id=1用 root 创建 `debezium` 用户授予 REPLICATION SLAVE + REPLICATION CLIENT。(2) Debezium Connect 容器daocloud 禁用 debezium 镜像改用 `quay.io/debezium/connect:2.7`MySQL 容器在 edu-minimal_default 网络,需 `docker network connect edu-full_default edu-mysql` 让 Debezium 同时可达Kafka 必须配置双 listenerINSIDE:kafka:29092 + OUTSIDE:localhost:9092否则 Debezium 拿到 advertised.listeners 中的 localhost metadata 后切换失败Debezium 2.x 容器环境变量名用 BOOTSTRAP_SERVERS不带 KAFKA_ 前缀),通过 envsubst 替换到 connect-distributed.properties。(3) 注册 connectorPOST :8083/connectors配置 topic.prefix=edu-cdc, database.include.list=next_edu_cloud, snapshot.mode=initial4 张表core_edu_grades/exams/classes/iam_users成功产生快照事件。(4) data-ana 消费者实现:新建 cdc_consumer.py 用 aiokafka AIOKafkaConsumerlifespan 中 asyncio.create_task 后台运行;按 source.table 路由exams→内存缓存 exam_id→class_id 映射grades→查缓存填 class_id 后 upsert ClickHousereadyz 端点附加 cdc_consumer 状态。(5) ClickHouse 远程访问:默认 default-user.xml 限制 127.0.0.1/::1 无密码,挂载 `clickhouse/users.d/custom-users.xml` 覆盖密码+任意 IP。(6) structlog 24.x API`make_filtering_bound_logger(level)` 替代废弃的 `make_filtering_logger`。(7) E2E 验证MySQL INSERT 成绩 → Debezium op=c 事件 → Kafka → 消费者写 ClickHouse 宽表class_id 通过 exam 缓存正确填充)→ /readyz cdc_consumer=running → /analytics/student/student-002/weakness 返回实时 92 分数据。**学到**Debezium 2.x 容器 bootstrap.servers 默认值是 0.0.0.0:9092 必须显式覆盖Kafka 单 listener 配置 localhost 会让容器间通信的客户端拿到 metadata 后切换失败,必须用双 listenerClickHouse users_xml 存储是 readonly 不能用 ALTER USER 修改密码,必须挂载 users.d 配置文件覆盖;消费者 offset 重置必须先停消费者让 group 处于 Empty 状态才能执行 --reset-offsets。 | | 2026-07-09 | 下午 | data-ana/infra | **CDC 完整链路实现**MySQL binlog → Debezium Connect → Kafka → data-ana 消费者 → ClickHouse 宽表。(1) MySQL binlog 配置log_bin=ON, binlog_format=ROW, binlog_row_image=FULL, server_id=1用 root 创建 `debezium` 用户授予 REPLICATION SLAVE + REPLICATION CLIENT。(2) Debezium Connect 容器daocloud 禁用 debezium 镜像改用 `quay.io/debezium/connect:2.7`MySQL 容器在 edu-minimal_default 网络,需 `docker network connect edu-full_default edu-mysql` 让 Debezium 同时可达Kafka 必须配置双 listenerINSIDE:kafka:29092 + OUTSIDE:localhost:9092否则 Debezium 拿到 advertised.listeners 中的 localhost metadata 后切换失败Debezium 2.x 容器环境变量名用 BOOTSTRAP_SERVERS不带 KAFKA_ 前缀),通过 envsubst 替换到 connect-distributed.properties。(3) 注册 connectorPOST :8083/connectors配置 topic.prefix=edu-cdc, database.include.list=next_edu_cloud, snapshot.mode=initial4 张表core_edu_grades/exams/classes/iam_users成功产生快照事件。(4) data-ana 消费者实现:新建 cdc_consumer.py 用 aiokafka AIOKafkaConsumerlifespan 中 asyncio.create_task 后台运行;按 source.table 路由exams→内存缓存 exam_id→class_id 映射grades→查缓存填 class_id 后 upsert ClickHousereadyz 端点附加 cdc_consumer 状态。(5) ClickHouse 远程访问:默认 default-user.xml 限制 127.0.0.1/::1 无密码,挂载 `clickhouse/users.d/custom-users.xml` 覆盖密码+任意 IP。(6) structlog 24.x API`make_filtering_bound_logger(level)` 替代废弃的 `make_filtering_logger`。(7) E2E 验证MySQL INSERT 成绩 → Debezium op=c 事件 → Kafka → 消费者写 ClickHouse 宽表class_id 通过 exam 缓存正确填充)→ /readyz cdc_consumer=running → /analytics/student/student-002/weakness 返回实时 92 分数据。**学到**Debezium 2.x 容器 bootstrap.servers 默认值是 0.0.0.0:9092 必须显式覆盖Kafka 单 listener 配置 localhost 会让容器间通信的客户端拿到 metadata 后切换失败,必须用双 listenerClickHouse users_xml 存储是 readonly 不能用 ALTER USER 修改密码,必须挂载 users.d 配置文件覆盖;消费者 offset 重置必须先停消费者让 group 处于 Empty 状态才能执行 --reset-offsets。 |

View File

@@ -184,8 +184,14 @@ services:
container_name: edu-prometheus container_name: edu-prometheus
profiles: ["observability"] profiles: ["observability"]
restart: unless-stopped restart: unless-stopped
command:
- "--config.file=/etc/prometheus/prometheus.yml"
- "--storage.tsdb.path=/prometheus"
- "--storage.tsdb.retention.time=15d"
- "--web.enable-lifecycle"
volumes: volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml:ro - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
- prometheus_data:/prometheus
ports: ports:
- "9090:9090" - "9090:9090"
grafana: grafana:
@@ -199,6 +205,49 @@ services:
- "3030:3000" - "3030:3000"
volumes: volumes:
- grafana_data:/var/lib/grafana - grafana_data:/var/lib/grafana
# ============================================================
# Exportersobservability profile与 Prometheus 同网络)
# ============================================================
node-exporter:
image: docker.m.daocloud.io/prom/node-exporter:v1.8.2
container_name: edu-node-exporter
profiles: ["observability"]
restart: unless-stopped
command:
- "--path.rootfs=/host"
ports:
- "9100:9100"
volumes:
- /proc:/host/proc:ro
- /sys:/host/sys:ro
- /:/host:ro
mysql-exporter:
image: docker.m.daocloud.io/prom/mysqld-exporter:v0.15.1
container_name: edu-mysql-exporter
profiles: ["observability"]
restart: unless-stopped
command:
- "--mysqld.address=edu-mysql:3306"
- "--mysqld.username=edu:changeme"
environment:
MYSQLD_EXPORTER_PASSWORD: "changeme"
ports:
- "9104:9104"
depends_on:
mysql:
condition: service_healthy
redis-exporter:
image: docker.m.daocloud.io/oliver006/redis_exporter:v1.67.0
container_name: edu-redis-exporter
profiles: ["observability"]
restart: unless-stopped
environment:
REDIS_ADDR: "redis://edu-redis:6379"
ports:
- "9121:9121"
depends_on:
redis:
condition: service_started
volumes: volumes:
mysql_data: mysql_data:
redis_data: redis_data:
@@ -206,3 +255,4 @@ volumes:
neo4j_data: neo4j_data:
es_data: es_data:
grafana_data: grafana_data:
prometheus_data:

View File

@@ -89,6 +89,6 @@ scrape_configs:
- job_name: 'node-exporter' - job_name: 'node-exporter'
static_configs: static_configs:
- targets: ['node-exporter:9100'] - targets: ['host.docker.internal:9100']
labels: labels:
service: node-exporter service: node-exporter

156
scripts/health-check.ps1 Normal file
View File

@@ -0,0 +1,156 @@
<#
.SYNOPSIS
Edu health check script
.DESCRIPTION
Checks health of all infrastructure containers + application services + observability endpoints
.EXAMPLE
.\scripts\health-check.ps1
#>
$ErrorActionPreference = "Continue"
Write-Host "========================================" -ForegroundColor Cyan
Write-Host " Edu Health Check" -ForegroundColor Cyan
Write-Host "========================================" -ForegroundColor Cyan
Write-Host ""
# ===== 1. Infrastructure containers =====
Write-Host "[1/3] Infrastructure Containers" -ForegroundColor Yellow
$infraContainers = @(
@{Name="MySQL"; Container="edu-mysql"},
@{Name="Redis"; Container="edu-redis"},
@{Name="Kafka"; Container="edu-kafka"},
@{Name="Zookeeper"; Container="edu-zookeeper"},
@{Name="ClickHouse"; Container="edu-clickhouse"},
@{Name="Debezium"; Container="edu-debezium"},
@{Name="Neo4j"; Container="edu-neo4j"},
@{Name="Elasticsearch"; Container="edu-es"},
@{Name="Jaeger"; Container="edu-jaeger"},
@{Name="Prometheus"; Container="edu-prometheus"},
@{Name="Grafana"; Container="edu-grafana"}
)
$infraOk = 0
$infraFail = 0
foreach ($svc in $infraContainers) {
$running = docker inspect -f '{{.State.Running}}' $svc.Container 2>$null
if ($running -ne "true") {
Write-Host " [FAIL] $($svc.Name) not running" -ForegroundColor Red
$infraFail++
continue
}
# 条件模板Health 不存在时返回空字符串,不报错
$health = docker inspect -f '{{if .State.Health}}{{.State.Health.Status}}{{end}}' $svc.Container 2>$null
if ($health -eq "healthy") {
Write-Host " [OK] $($svc.Name)" -ForegroundColor Green
$infraOk++
} else {
Write-Host " [WARN] $($svc.Name) (running, no healthcheck)" -ForegroundColor Yellow
$infraOk++
}
}
Write-Host ""
# ===== 2. Application services =====
Write-Host "[2/3] Application Services" -ForegroundColor Yellow
$appServices = @(
@{Name="classes"; Url="http://localhost:3001/healthz"},
@{Name="iam"; Url="http://localhost:3002/healthz"},
@{Name="teacher-bff"; Url="http://localhost:3003/healthz"},
@{Name="core-edu"; Url="http://localhost:3004/healthz"},
@{Name="content"; Url="http://localhost:3005/healthz"},
@{Name="data-ana"; Url="http://localhost:3006/healthz"},
@{Name="msg"; Url="http://localhost:3007/healthz"},
@{Name="ai"; Url="http://localhost:3008/healthz"},
@{Name="api-gateway"; Url="http://localhost:8080/healthz"},
@{Name="push-gateway"; Url="http://localhost:8081/healthz"},
@{Name="teacher-portal";Url="http://localhost:3000/"}
)
$appOk = 0
$appFail = 0
foreach ($svc in $appServices) {
try {
$null = Invoke-RestMethod -Uri $svc.Url -Method Get -TimeoutSec 3 -ErrorAction Stop
Write-Host " [OK] $($svc.Name)" -ForegroundColor Green
$appOk++
} catch {
Write-Host " [FAIL] $($svc.Name)" -ForegroundColor Red
$appFail++
}
}
Write-Host ""
# ===== 3. Observability endpoints =====
Write-Host "[3/3] Observability Endpoints" -ForegroundColor Yellow
$obsEndpoints = @(
@{Name="Prometheus API"; Url="http://localhost:9090/api/v1/query?query=up"},
@{Name="Jaeger API"; Url="http://localhost:16686/api/services"},
@{Name="Debezium Connect"; Url="http://localhost:8083/connectors"},
@{Name="data-ana /metrics";Url="http://localhost:3006/metrics"},
@{Name="iam /metrics"; Url="http://localhost:3002/metrics"}
)
$obsOk = 0
$obsFail = 0
foreach ($ep in $obsEndpoints) {
try {
$null = Invoke-RestMethod -Uri $ep.Url -Method Get -TimeoutSec 3 -ErrorAction Stop
Write-Host " [OK] $($ep.Name)" -ForegroundColor Green
$obsOk++
} catch {
Write-Host " [FAIL] $($ep.Name)" -ForegroundColor Red
$obsFail++
}
}
# ===== 4. CDC pipeline status =====
Write-Host ""
Write-Host "[Extra] CDC Pipeline Status" -ForegroundColor Yellow
$connectorStatus = $null
try {
$connectorStatus = Invoke-RestMethod -Uri "http://localhost:8083/connectors/edu-mysql-source/status" -Method Get -TimeoutSec 3 -ErrorAction Stop
} catch {
Write-Host " [FAIL] Debezium connector not registered or error" -ForegroundColor Red
}
if ($connectorStatus) {
$connectorState = $connectorStatus.connector.state
$tasks = @($connectorStatus.tasks)
if ($tasks.Count -gt 0) {
$taskState = $tasks[0].state
} else {
$taskState = "UNKNOWN"
}
if ($connectorState -eq "RUNNING" -and $taskState -eq "RUNNING") {
$color = "Green"
} else {
$color = "Yellow"
}
Write-Host " Connector: $connectorState / Task: $taskState" -ForegroundColor $color
}
$chSql = 'SELECT count(*) FROM edu_analytics.student_dashboard_view'
$chOutput = docker exec edu-clickhouse clickhouse-client --user default --password clickhouse -q $chSql 2>&1
$chCount = "$chOutput".Trim()
if ($chCount -match '^\d+$') {
Write-Host " ClickHouse student_dashboard_view: $chCount records" -ForegroundColor Green
} else {
Write-Host " ClickHouse query failed or empty" -ForegroundColor Yellow
}
# ===== Summary =====
Write-Host ""
Write-Host "========================================" -ForegroundColor Cyan
Write-Host " Infra: $infraOk/$($infraContainers.Count) | App: $appOk/$($appServices.Count) | Obs: $obsOk/$($obsEndpoints.Count)" -ForegroundColor Cyan
Write-Host "========================================" -ForegroundColor Cyan
if ($infraFail -gt 0 -or $appFail -gt 0 -or $obsFail -gt 0) {
exit 1
}

301
scripts/start-all.ps1 Normal file
View File

@@ -0,0 +1,301 @@
<#
.SYNOPSIS
Edu start all application services script
.DESCRIPTION
Starts 10 application services + 1 frontend (teacher-portal)
Each service runs in a separate terminal window for log visibility
Automatically injects Python service env vars (CLICKHOUSE/KAFKA/OTEL)
Prerequisite: infrastructure containers (MySQL/Redis/Kafka/ClickHouse etc) must be running
.PARAMETER SkipInfraCheck
Skip infrastructure health check (use when infra is known to be running)
.EXAMPLE
.\scripts\start-all.ps1
.\scripts\start-all.ps1 -SkipInfraCheck
#>
param(
[switch]$SkipInfraCheck,
[switch]$Force
)
$ErrorActionPreference = "Stop"
$ProjectRoot = Split-Path -Parent $PSScriptRoot
Write-Host "========================================" -ForegroundColor Cyan
Write-Host " Edu Start All Services" -ForegroundColor Cyan
Write-Host "========================================" -ForegroundColor Cyan
Write-Host ""
# ===== 1. Infrastructure health check =====
if (-not $SkipInfraCheck) {
Write-Host "[1/6] Checking infrastructure health..." -ForegroundColor Yellow
$infraServices = @(
@{Name="MySQL"; Container="edu-mysql"},
@{Name="Redis"; Container="edu-redis"},
@{Name="Kafka"; Container="edu-kafka"},
@{Name="ClickHouse"; Container="edu-clickhouse"},
@{Name="Debezium"; Container="edu-debezium"},
@{Name="Jaeger"; Container="edu-jaeger"}
)
$allHealthy = $true
$prevEAP = $ErrorActionPreference
$ErrorActionPreference = "Continue"
foreach ($svc in $infraServices) {
$running = docker inspect -f '{{.State.Running}}' $svc.Container 2>$null
if ($running -ne "true") {
Write-Host " [FAIL] $($svc.Name) not running" -ForegroundColor Red
$allHealthy = $false
continue
}
# 条件模板Health 不存在时返回空字符串,不报错
$health = docker inspect -f '{{if .State.Health}}{{.State.Health.Status}}{{end}}' $svc.Container 2>$null
if ($health -eq "healthy") {
Write-Host " [OK] $($svc.Name) ($($svc.Container))" -ForegroundColor Green
} else {
Write-Host " [WARN] $($svc.Name) running (no healthcheck)" -ForegroundColor Yellow
}
}
$ErrorActionPreference = $prevEAP
if (-not $allHealthy) {
Write-Host ""
Write-Host "Infrastructure not ready. Start it first:" -ForegroundColor Red
Write-Host " docker compose -f infra/docker-compose.yml --profile p6 --profile observability up -d" -ForegroundColor White
exit 1
}
Write-Host ""
}
# ===== 2. Environment variables =====
Write-Host "[2/6] Preparing environment variables..." -ForegroundColor Yellow
$env:DEV_MODE = "true"
$env:OTEL_EXPORTER_OTLP_ENDPOINT = "http://localhost:4318"
$env:DATABASE_URL = "mysql://edu:changeme@localhost:3306/next_edu_cloud"
$env:REDIS_URL = "redis://localhost:6379"
$env:JWT_SECRET = "p1-dev-secret-change-in-production"
$env:KAFKA_BROKERS = "localhost:9092"
$pyEnv = @{
CLICKHOUSE_HOST = "localhost"
CLICKHOUSE_PORT = "8123"
CLICKHOUSE_USER = "default"
CLICKHOUSE_PASSWORD = "clickhouse"
CLICKHOUSE_DATABASE = "edu_analytics"
KAFKA_BROKERS = "localhost:9092"
OTEL_ENDPOINT = "http://localhost:4318"
DEV_MODE = "true"
}
Write-Host " DEV_MODE=true (dev-token bypass)" -ForegroundColor Green
Write-Host " OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318" -ForegroundColor Green
Write-Host ""
# ===== 3. Port conflict check =====
Write-Host "[3/6] Checking port conflicts..." -ForegroundColor Yellow
$portMap = @{
3001="classes"; 3002="iam"; 3003="teacher-bff"; 3004="core-edu"
3005="content"; 3006="data-ana"; 3007="msg"; 3008="ai"
8080="api-gateway"; 8081="push-gateway"; 3000="teacher-portal"
}
$conflicts = @()
foreach ($port in $portMap.Keys | Sort-Object) {
$conn = Get-NetTCPConnection -LocalPort $port -State Listen -ErrorAction SilentlyContinue
if ($conn) {
$svcName = $portMap[$port]
$procId = $conn[0].OwningProcess
$procName = ""
try { $procName = (Get-Process -Id $procId -ErrorAction Stop).ProcessName } catch {}
Write-Host " [WARN] Port $port ($svcName) occupied by PID $procId ($procName)" -ForegroundColor Yellow
$conflicts += [PSCustomObject]@{Port=$port; Service=$svcName; PID=$procId; Process=$procName}
}
}
if ($conflicts.Count -gt 0) {
Write-Host ""
Write-Host " $($conflicts.Count) port(s) already in use." -ForegroundColor Yellow
$shouldKill = $false
if ($Force) {
Write-Host " -Force specified, killing automatically..." -ForegroundColor White
$shouldKill = $true
} else {
Write-Host " These services may already be running. Options:" -ForegroundColor White
Write-Host " 1. Run .\scripts\stop-all.ps1 -KillByPort first, then re-run start-all" -ForegroundColor White
Write-Host " 2. Re-run with -Force to auto-kill and continue" -ForegroundColor White
Write-Host ""
$answer = Read-Host " Kill existing processes and continue? (y/N)"
if ($answer -eq "y" -or $answer -eq "Y") {
$shouldKill = $true
}
}
if ($shouldKill) {
foreach ($c in $conflicts) {
try {
Stop-Process -Id $c.PID -Force -ErrorAction Stop
Write-Host " [OK] Killed PID $($c.PID) ($($c.Process)) on port $($c.Port)" -ForegroundColor Green
Start-Sleep -Milliseconds 500
} catch {
Write-Host " [WARN] Cannot kill PID $($c.PID): $($_.Exception.Message)" -ForegroundColor Yellow
}
}
Start-Sleep -Seconds 2
} else {
Write-Host " Aborting. Please stop existing services first." -ForegroundColor Red
exit 1
}
} else {
Write-Host " [OK] All app ports are free" -ForegroundColor Green
}
Write-Host ""
# ===== 4. Build NestJS services (required for nest start --watch) =====
Write-Host "[4/6] Building NestJS services (first time required)..." -ForegroundColor Yellow
$nestjsServices = @(
"@edu/classes-service",
"@edu/iam-service",
"@edu/teacher-bff",
"@edu/core-edu-service",
"@edu/content-service",
"@edu/msg-service"
)
# Clean tsbuildinfo cache (incremental mode leftover causes tsc to skip emit)
$nestjsDirs = @("classes", "iam", "teacher-bff", "core-edu", "content", "msg")
foreach ($dir in $nestjsDirs) {
$svcPath = Join-Path $ProjectRoot "services\$dir"
if (Test-Path $svcPath) {
Get-ChildItem -Path $svcPath -Filter "*.tsbuildinfo" -Recurse -ErrorAction SilentlyContinue |
Remove-Item -Force -ErrorAction SilentlyContinue
}
}
foreach ($svc in $nestjsServices) {
Write-Host " Building $svc..." -ForegroundColor Gray -NoNewline
$buildResult = pnpm --filter $svc build 2>&1
if ($LASTEXITCODE -eq 0) {
Write-Host " [OK]" -ForegroundColor Green
} else {
Write-Host " [FAIL]" -ForegroundColor Red
Write-Host " $buildResult" -ForegroundColor DarkGray
}
}
Write-Host ""
# ===== 5. Start application services =====
Write-Host "[5/6] Starting application services (11 windows)..." -ForegroundColor Yellow
$services = @(
@{Title="edu-app-classes"; Cmd="pnpm"; Args=@("--filter","@edu/classes-service","dev"); Dir="$ProjectRoot"},
@{Title="edu-app-iam"; Cmd="pnpm"; Args=@("--filter","@edu/iam-service","dev"); Dir="$ProjectRoot"},
@{Title="edu-app-teacher-bff"; Cmd="pnpm"; Args=@("--filter","@edu/teacher-bff","dev"); Dir="$ProjectRoot"},
@{Title="edu-app-core-edu"; Cmd="pnpm"; Args=@("--filter","@edu/core-edu-service","dev"); Dir="$ProjectRoot"},
@{Title="edu-app-content"; Cmd="pnpm"; Args=@("--filter","@edu/content-service","dev"); Dir="$ProjectRoot"},
@{Title="edu-app-msg"; Cmd="pnpm"; Args=@("--filter","@edu/msg-service","dev"); Dir="$ProjectRoot"},
@{Title="edu-app-data-ana"; Cmd="uv"; Args=@("run","uvicorn","data_ana.main:app","--app-dir","src","--host","0.0.0.0","--port","3006","--reload"); Dir="$ProjectRoot\services\data-ana"; PyEnv=$true},
@{Title="edu-app-ai"; Cmd="uv"; Args=@("run","uvicorn","ai.main:app","--app-dir","src","--host","0.0.0.0","--port","3008","--reload"); Dir="$ProjectRoot\services\ai"; PyEnv=$true},
@{Title="edu-app-api-gateway"; Cmd="go"; Args=@("run","."); Dir="$ProjectRoot\services\api-gateway"; GoEnv=$true},
@{Title="edu-app-push-gateway"; Cmd="go"; Args=@("run","."); Dir="$ProjectRoot\services\push-gateway";GoEnv=$true},
@{Title="edu-app-teacher-portal";Cmd="pnpm";Args=@("--filter","teacher-portal","dev"); Dir="$ProjectRoot"}
)
foreach ($svc in $services) {
$cmdStr = "$($svc.Cmd) $($svc.Args -join ' ')"
$psCmd = "Set-Location '$($svc.Dir)'; "
if ($svc.PyEnv) {
foreach ($kv in $pyEnv.GetEnumerator()) {
$psCmd += "`$env:$($kv.Key)='$($kv.Value)'; "
}
}
if ($svc.GoEnv) {
$psCmd += "`$env:Path = 'C:\Program Files\Go\bin;' + `$env:Path; "
}
$psCmd += "$cmdStr; Write-Host ''; Write-Host 'Service stopped. Press any key to close...' -ForegroundColor Yellow; `$null = `$Host.UI.RawUI.ReadKey('NoEcho,IncludeKeyDown')"
Start-Process -FilePath "powershell" -ArgumentList "-NoExit","-Command",$psCmd -WindowStyle Normal | Out-Null
Write-Host " [START] $($svc.Title)..." -ForegroundColor Green
Start-Sleep -Milliseconds 500
}
Write-Host ""
Write-Host " All services started in new windows. Waiting 40s for init..." -ForegroundColor Yellow
Start-Sleep -Seconds 40
# ===== 6. Health check =====
Write-Host "[6/6] Health check..." -ForegroundColor Yellow
Write-Host ""
$healthServices = @(
@{Name="classes"; Url="http://localhost:3001/healthz"},
@{Name="iam"; Url="http://localhost:3002/healthz"},
@{Name="teacher-bff"; Url="http://localhost:3003/healthz"},
@{Name="core-edu"; Url="http://localhost:3004/healthz"},
@{Name="content"; Url="http://localhost:3005/healthz"},
@{Name="data-ana"; Url="http://localhost:3006/healthz"},
@{Name="msg"; Url="http://localhost:3007/healthz"},
@{Name="ai"; Url="http://localhost:3008/healthz"},
@{Name="api-gateway"; Url="http://localhost:8080/healthz"},
@{Name="push-gateway"; Url="http://localhost:8081/healthz"},
@{Name="teacher-portal";Url="http://localhost:3000/"}
)
$okCount = 0
$failCount = 0
$failedServices = @()
foreach ($svc in $healthServices) {
$retries = 0
$maxRetries = 3
$success = $false
$lastError = ""
while ($retries -lt $maxRetries -and -not $success) {
try {
$null = Invoke-RestMethod -Uri $svc.Url -Method Get -TimeoutSec 5 -ErrorAction Stop
Write-Host " [OK] $($svc.Name)" -ForegroundColor Green
$success = $true
$okCount++
} catch {
$lastError = $_.Exception.Message
$retries++
if ($retries -lt $maxRetries) {
Start-Sleep -Seconds 5
}
}
}
if (-not $success) {
Write-Host " [FAIL] $($svc.Name) (after $maxRetries retries: $lastError)" -ForegroundColor Red
$failCount++
$failedServices += $svc.Name
}
}
Write-Host ""
Write-Host "========================================" -ForegroundColor Cyan
Write-Host " Started: [OK] $okCount ready / [FAIL] $failCount failed" -ForegroundColor Cyan
Write-Host "========================================" -ForegroundColor Cyan
if ($failCount -gt 0) {
Write-Host ""
Write-Host "Failed services: $($failedServices -join ', ')" -ForegroundColor Red
Write-Host ""
Write-Host "Troubleshooting:" -ForegroundColor Yellow
Write-Host " 1. Check the service window for error output" -ForegroundColor White
Write-Host " 2. Verify dependencies: pnpm install / uv sync / go mod tidy" -ForegroundColor White
Write-Host " 3. Re-run health check: .\scripts\health-check.ps1" -ForegroundColor White
Write-Host " 4. Stop and retry: .\scripts\stop-all.ps1 -KillByPort then .\scripts\start-all.ps1" -ForegroundColor White
Write-Host ""
exit 1
}
Write-Host ""
Write-Host "Next steps:" -ForegroundColor Yellow
Write-Host " Health check: .\scripts\health-check.ps1" -ForegroundColor White
Write-Host " CDC test: .\scripts\test-cdc.ps1" -ForegroundColor White
Write-Host " Stop all: .\scripts\stop-all.ps1" -ForegroundColor White
Write-Host ""

98
scripts/stop-all.ps1 Normal file
View File

@@ -0,0 +1,98 @@
<#
.SYNOPSIS
Edu stop all application services script
.DESCRIPTION
Closes all edu-app-* terminal windows (started by start-all.ps1)
Optional: kill processes by port (fallback when windows are closed but processes linger)
.PARAMETER KillByPort
Kill processes by port (fallback when windows are closed but processes still alive)
.EXAMPLE
.\scripts\stop-all.ps1
.\scripts\stop-all.ps1 -KillByPort
#>
param(
[switch]$KillByPort
)
$ErrorActionPreference = "Continue"
Write-Host "========================================" -ForegroundColor Cyan
Write-Host " Edu Stop All Services" -ForegroundColor Cyan
Write-Host "========================================" -ForegroundColor Cyan
Write-Host ""
# ===== 1. Close app service terminal windows =====
Write-Host "[1/2] Closing app service windows..." -ForegroundColor Yellow
$appTitles = @(
"edu-app-classes",
"edu-app-iam",
"edu-app-teacher-bff",
"edu-app-core-edu",
"edu-app-content",
"edu-app-msg",
"edu-app-data-ana",
"edu-app-ai",
"edu-app-api-gateway",
"edu-app-push-gateway",
"edu-app-teacher-portal"
)
$closedCount = 0
foreach ($title in $appTitles) {
$procs = Get-Process -Name "powershell","pwsh","node","python","uvicorn","go" -ErrorAction SilentlyContinue |
Where-Object { $_.MainWindowTitle -like "*$title*" }
if ($procs) {
foreach ($p in $procs) {
try {
Stop-Process -Id $p.Id -Force -ErrorAction Stop
Write-Host " [OK] Closed $title (PID $($p.Id))" -ForegroundColor Green
$closedCount++
} catch {
Write-Host " [WARN] Cannot close $title (PID $($p.Id)): $($_.Exception.Message)" -ForegroundColor Yellow
}
}
} else {
Write-Host " [--] $title window not found" -ForegroundColor Gray
}
}
Write-Host ""
Write-Host " Closed $closedCount windows" -ForegroundColor Green
Write-Host ""
# ===== 2. Kill by port (optional) =====
if ($KillByPort) {
Write-Host "[2/2] Killing processes by port..." -ForegroundColor Yellow
$ports = @(3000,3001,3002,3003,3004,3005,3006,3007,3008,8080,8081)
foreach ($port in $ports) {
$connections = Get-NetTCPConnection -LocalPort $port -State Listen -ErrorAction SilentlyContinue
if ($connections) {
foreach ($conn in $connections) {
try {
$proc = Get-Process -Id $conn.OwningProcess -ErrorAction Stop
Stop-Process -Id $conn.OwningProcess -Force -ErrorAction Stop
Write-Host " [OK] Port $port -> killed $($proc.ProcessName) (PID $($conn.OwningProcess))" -ForegroundColor Green
} catch {
Write-Host " [WARN] Port $port -> cannot kill PID $($conn.OwningProcess)" -ForegroundColor Yellow
}
}
} else {
Write-Host " [--] Port $port free" -ForegroundColor Gray
}
}
} else {
Write-Host "[2/2] Skipping port kill (use -KillByPort to enable)" -ForegroundColor Gray
}
Write-Host ""
Write-Host "========================================" -ForegroundColor Cyan
Write-Host " Stop complete" -ForegroundColor Cyan
Write-Host "========================================" -ForegroundColor Cyan
Write-Host ""
Write-Host "To stop infrastructure:" -ForegroundColor Yellow
Write-Host " docker compose -f infra/docker-compose.yml --profile p6 --profile observability down" -ForegroundColor White
Write-Host ""

167
scripts/test-cdc.ps1 Normal file
View File

@@ -0,0 +1,167 @@
<#
.SYNOPSIS
Edu CDC pipeline end-to-end test script
.DESCRIPTION
Validates the full CDC pipeline: MySQL binlog -> Debezium -> Kafka -> data-ana -> ClickHouse
Steps:
1. Insert test grade into MySQL
2. Wait for Debezium capture + data-ana consume
3. Query ClickHouse to verify data synced
4. Call data-ana API to verify query works
.PARAMETER StudentId
Custom test student_id (default: cdc-test-<timestamp>)
.EXAMPLE
.\scripts\test-cdc.ps1
.\scripts\test-cdc.ps1 -StudentId "my-test-001"
#>
param(
[string]$StudentId = "cdc-test-$(Get-Date -Format 'yyyyMMddHHmmss')"
)
$ErrorActionPreference = "Continue"
Write-Host "========================================" -ForegroundColor Cyan
Write-Host " Edu CDC Pipeline E2E Test" -ForegroundColor Cyan
Write-Host "========================================" -ForegroundColor Cyan
Write-Host ""
$examId = "exam-$StudentId"
$gradeId = "grade-$StudentId"
$testScore = 92.5
$classId = "cls-cdc-test"
Write-Host "Test parameters:" -ForegroundColor Yellow
Write-Host " StudentId: $StudentId"
Write-Host " ExamId: $examId"
Write-Host " GradeId: $gradeId"
Write-Host " Score: $testScore"
Write-Host " ClassId: $classId"
Write-Host ""
# ===== 1. Pre-check =====
Write-Host "[1/5] Pre-check..." -ForegroundColor Yellow
# Check Debezium connector
$connectorStatus = $null
try {
$connectorStatus = Invoke-RestMethod -Uri "http://localhost:8083/connectors/edu-mysql-source/status" -Method Get -TimeoutSec 3 -ErrorAction Stop
} catch {
Write-Host " [FAIL] Debezium Connect not reachable. Start infrastructure first." -ForegroundColor Red
exit 1
}
if ($connectorStatus.connector.state -ne "RUNNING") {
Write-Host " [FAIL] Debezium connector state: $($connectorStatus.connector.state)" -ForegroundColor Red
exit 1
}
Write-Host " [OK] Debezium connector: RUNNING" -ForegroundColor Green
# Check data-ana service
try {
$null = Invoke-RestMethod -Uri "http://localhost:3006/healthz" -Method Get -TimeoutSec 3 -ErrorAction Stop
Write-Host " [OK] data-ana service: running" -ForegroundColor Green
} catch {
Write-Host " [FAIL] data-ana service not reachable. Start application services first." -ForegroundColor Red
exit 1
}
# Check CDC consumer status
try {
$readyz = Invoke-RestMethod -Uri "http://localhost:3006/readyz" -Method Get -TimeoutSec 3 -ErrorAction Stop
$cdcStatus = $readyz.services.cdc_consumer
if ($cdcStatus -ne "running") {
Write-Host " [WARN] CDC consumer status: $cdcStatus (KAFKA_BROKERS may not be set)" -ForegroundColor Yellow
} else {
Write-Host " [OK] CDC consumer: running" -ForegroundColor Green
}
} catch {
Write-Host " [WARN] Cannot get /readyz status" -ForegroundColor Yellow
}
Write-Host ""
# ===== 2. Insert test data into MySQL =====
Write-Host "[2/5] Inserting test data into MySQL..." -ForegroundColor Yellow
$sqlInsert = @"
INSERT INTO core_edu_exams (id, class_id, subject_id, title, exam_date, total_score, created_at, updated_at)
VALUES ('$examId', '$classId', 'sub-math', 'CDC Test Exam', NOW(), 100, NOW(), NOW())
ON DUPLICATE KEY UPDATE updated_at=NOW();
INSERT INTO core_edu_grades (id, exam_id, student_id, score, rank_in_class, created_at, updated_at)
VALUES ('$gradeId', '$examId', '$StudentId', $testScore, 1, NOW(), NOW())
ON DUPLICATE KEY UPDATE score=$testScore, updated_at=NOW();
"@
docker exec edu-mysql mysql -uedu -pchangeme next_edu_cloud -e $sqlInsert 2>&1 | Out-Null
if ($LASTEXITCODE -eq 0) {
Write-Host " [OK] Inserted: exam=$examId / grade=$gradeId / student=$StudentId / score=$testScore" -ForegroundColor Green
} else {
Write-Host " [FAIL] MySQL insert failed" -ForegroundColor Red
exit 1
}
Write-Host ""
# ===== 3. Wait for CDC propagation =====
Write-Host "[3/5] Waiting for CDC propagation (5s)..." -ForegroundColor Yellow
Start-Sleep -Seconds 5
Write-Host " [OK] Wait complete" -ForegroundColor Green
Write-Host ""
# ===== 4. Verify ClickHouse data =====
Write-Host "[4/5] Verifying ClickHouse data..." -ForegroundColor Yellow
$chQuery = "SELECT student_id, class_id, exam_id, score, last_updated FROM edu_analytics.student_dashboard_view WHERE student_id = '$StudentId' ORDER BY last_updated DESC LIMIT 5"
$chResult = docker exec edu-clickhouse clickhouse-client --user default --password clickhouse -q $chQuery 2>$null
if ($chResult) {
Write-Host " [OK] ClickHouse returned data:" -ForegroundColor Green
Write-Host " $chResult" -ForegroundColor White
if ($chResult -match $StudentId -and $chResult -match "$testScore") {
Write-Host ""
Write-Host " [OK] Verified: student_id match + score=$testScore match" -ForegroundColor Green
if ($chResult -match $classId) {
Write-Host " [OK] class_id filled via exam cache: $classId" -ForegroundColor Green
} else {
Write-Host " [WARN] class_id not filled (exam cache may have missed, check event order)" -ForegroundColor Yellow
}
} else {
Write-Host " [FAIL] Data mismatch: expected student=$StudentId, score=$testScore" -ForegroundColor Red
}
} else {
Write-Host " [FAIL] ClickHouse has no data for student_id=$StudentId" -ForegroundColor Red
Write-Host " Possible causes:" -ForegroundColor Yellow
Write-Host " 1. Debezium did not capture MySQL change (check connector status)" -ForegroundColor White
Write-Host " 2. data-ana consumer not running (check /readyz cdc_consumer)" -ForegroundColor White
Write-Host " 3. Kafka topic name mismatch (check debezium-register.json)" -ForegroundColor White
exit 1
}
Write-Host ""
# ===== 5. Verify data-ana API =====
Write-Host "[5/5] Verifying data-ana query API..." -ForegroundColor Yellow
$h = @{Authorization="Bearer dev-token"}
try {
$weakness = Invoke-RestMethod -Uri "http://localhost:3006/analytics/student/$StudentId/weakness" -Method Get -Headers $h -TimeoutSec 5 -ErrorAction Stop
Write-Host " [OK] /analytics/student/$StudentId/weakness" -ForegroundColor Green
Write-Host " Response: $($weakness | ConvertTo-Json -Depth 3)" -ForegroundColor Gray
} catch {
Write-Host " [WARN] /analytics/student/$StudentId/weakness failed: $($_.Exception.Message)" -ForegroundColor Yellow
}
try {
$perf = Invoke-RestMethod -Uri "http://localhost:3006/analytics/class/$classId/performance" -Method Get -Headers $h -TimeoutSec 5 -ErrorAction Stop
Write-Host " [OK] /analytics/class/$classId/performance" -ForegroundColor Green
Write-Host " Response: $($perf | ConvertTo-Json -Depth 3)" -ForegroundColor Gray
} catch {
Write-Host " [WARN] /analytics/class/$classId/performance failed: $($_.Exception.Message)" -ForegroundColor Yellow
}
Write-Host ""
Write-Host "========================================" -ForegroundColor Cyan
Write-Host " CDC Pipeline Test Complete" -ForegroundColor Cyan
Write-Host " MySQL -> Debezium -> Kafka -> data-ana -> ClickHouse [OK]" -ForegroundColor Green
Write-Host "========================================" -ForegroundColor Cyan

View File

@@ -1,7 +1,8 @@
import { Module } from '@nestjs/common'; import { Module } from "@nestjs/common";
import { ClassesModule } from './classes/classes.module.js'; import { ClassesModule } from "./classes/classes.module.js";
import { HealthModule } from "./shared/health/health.module.js";
@Module({ @Module({
imports: [ClassesModule], imports: [ClassesModule, HealthModule],
}) })
export class AppModule {} export class AppModule {}

View File

@@ -1,7 +1,8 @@
import { Controller, Get, HttpException, HttpStatus } from '@nestjs/common'; import { Controller, Get, HttpException, HttpStatus } from "@nestjs/common";
import { DataSource } from 'typeorm'; import { sql } from "drizzle-orm";
import { getDb } from "../../config/database.js";
const SERVICE_NAME = 'classes'; const SERVICE_NAME = "classes";
/** /**
* 健康检查端点。 * 健康检查端点。
@@ -14,33 +15,37 @@ const SERVICE_NAME = 'classes';
*/ */
@Controller() @Controller()
export class HealthController { export class HealthController {
constructor(private readonly dataSource: DataSource) {} @Get("healthz")
@Get('healthz')
liveness(): { status: string; service: string; timestamp: string } { liveness(): { status: string; service: string; timestamp: string } {
return { return {
status: 'ok', status: "ok",
service: SERVICE_NAME, service: SERVICE_NAME,
timestamp: new Date().toISOString(), timestamp: new Date().toISOString(),
}; };
} }
@Get('readyz') @Get("readyz")
async readiness(): Promise<{ status: string; service: string; timestamp: string }> { async readiness(): Promise<{
status: string;
service: string;
timestamp: string;
}> {
try { try {
await this.dataSource.query('SELECT 1'); const db = getDb();
await db.execute(sql`SELECT 1`);
return { return {
status: 'ok', status: "ok",
service: SERVICE_NAME, service: SERVICE_NAME,
timestamp: new Date().toISOString(), timestamp: new Date().toISOString(),
}; };
} catch (error) { } catch (error) {
throw new HttpException( throw new HttpException(
{ {
status: 'error', status: "error",
service: SERVICE_NAME, service: SERVICE_NAME,
timestamp: new Date().toISOString(), timestamp: new Date().toISOString(),
error: error instanceof Error ? error.message : 'database unreachable', error:
error instanceof Error ? error.message : "database unreachable",
}, },
HttpStatus.SERVICE_UNAVAILABLE, HttpStatus.SERVICE_UNAVAILABLE,
); );

View File

@@ -8,6 +8,7 @@
"emitDecoratorMetadata": true, "emitDecoratorMetadata": true,
"outDir": "./dist", "outDir": "./dist",
"rootDir": "./src", "rootDir": "./src",
"incremental": false,
"types": ["node"] "types": ["node"]
}, },
"include": ["src/**/*"], "include": ["src/**/*"],

View File

@@ -8,6 +8,7 @@
"emitDecoratorMetadata": true, "emitDecoratorMetadata": true,
"outDir": "./dist", "outDir": "./dist",
"rootDir": "./src", "rootDir": "./src",
"incremental": false,
"types": ["node"] "types": ["node"]
}, },
"include": ["src/**/*"], "include": ["src/**/*"],

View File

@@ -8,6 +8,7 @@
"emitDecoratorMetadata": true, "emitDecoratorMetadata": true,
"outDir": "./dist", "outDir": "./dist",
"rootDir": "./src", "rootDir": "./src",
"incremental": false,
"types": ["node"] "types": ["node"]
}, },
"include": ["src/**/*"], "include": ["src/**/*"],

View File

@@ -1,7 +1,8 @@
import { Module } from '@nestjs/common'; import { Module } from "@nestjs/common";
import { IamModule } from './iam/iam.module.js'; import { IamModule } from "./iam/iam.module.js";
import { HealthModule } from "./shared/health/health.module.js";
@Module({ @Module({
imports: [IamModule], imports: [IamModule, HealthModule],
}) })
export class AppModule {} export class AppModule {}

View File

@@ -8,6 +8,7 @@
"emitDecoratorMetadata": true, "emitDecoratorMetadata": true,
"outDir": "./dist", "outDir": "./dist",
"rootDir": "./src", "rootDir": "./src",
"incremental": false,
"types": ["node"] "types": ["node"]
}, },
"include": ["src/**/*"], "include": ["src/**/*"],

View File

@@ -8,6 +8,7 @@
"emitDecoratorMetadata": true, "emitDecoratorMetadata": true,
"outDir": "./dist", "outDir": "./dist",
"rootDir": "./src", "rootDir": "./src",
"incremental": false,
"types": ["node"] "types": ["node"]
}, },
"include": ["src/**/*"], "include": ["src/**/*"],

View File

@@ -3,6 +3,7 @@ import { NestFactory } from "@nestjs/core";
import { AppModule } from "./app.module.js"; import { AppModule } from "./app.module.js";
import { env } from "./config/env.js"; import { env } from "./config/env.js";
import { initTracer, shutdownTracer } from "./shared/observability/tracer.js"; import { initTracer, shutdownTracer } from "./shared/observability/tracer.js";
import { metricsRegistry } from "./shared/observability/metrics.js";
async function bootstrap(): Promise<void> { async function bootstrap(): Promise<void> {
initTracer(); initTracer();
@@ -12,6 +13,12 @@ async function bootstrap(): Promise<void> {
app.enableShutdownHooks(); app.enableShutdownHooks();
// Prometheus 指标端点:不鉴权,供 Prometheus 抓取。
app.getHttpAdapter().get("/metrics", async (_req, res) => {
res.set("Content-Type", metricsRegistry.contentType);
res.end(await metricsRegistry.metrics());
});
await app.listen(env.PORT); await app.listen(env.PORT);
console.log(`Teacher BFF started on port ${env.PORT}`); console.log(`Teacher BFF started on port ${env.PORT}`);

View File

@@ -0,0 +1,26 @@
import promClient from "prom-client";
const registry = new promClient.Registry();
registry.setDefaultLabels({ service: "teacher-bff" });
registry.registerMetric(
new promClient.Counter({
name: "teacher_bff_requests_total",
help: "Total number of teacher-bff requests",
labelNames: ["method", "endpoint", "status"],
}),
);
registry.registerMetric(
new promClient.Histogram({
name: "teacher_bff_request_duration_seconds",
help: "Teacher-bff request duration in seconds",
labelNames: ["method", "endpoint"],
buckets: [0.01, 0.05, 0.1, 0.3, 0.5, 1, 3, 5],
}),
);
// 自动收集 Node.js 进程级指标CPU/内存/事件循环/GC等
promClient.collectDefaultMetrics({ register: registry });
export { registry as metricsRegistry };

View File

@@ -8,6 +8,7 @@
"emitDecoratorMetadata": true, "emitDecoratorMetadata": true,
"outDir": "./dist", "outDir": "./dist",
"rootDir": "./src", "rootDir": "./src",
"incremental": false,
"types": ["node"] "types": ["node"]
}, },
"include": ["src/**/*"], "include": ["src/**/*"],