diff --git a/docs/standards/full-stack-runbook.md b/docs/standards/full-stack-runbook.md index 5d67f17..9b797ac 100644 --- a/docs/standards/full-stack-runbook.md +++ b/docs/standards/full-stack-runbook.md @@ -195,6 +195,8 @@ Invoke-RestMethod -Uri "http://localhost:3006/readyz" ```powershell cd e:\Desktop\Edu .\scripts\stop-all.ps1 + +.\scripts\stop-all.ps1 -KillByPort ``` 该脚本会关闭所有 `edu-app-*` 标题的终端窗口。 diff --git a/docs/troubleshooting/known-issues.md b/docs/troubleshooting/known-issues.md index bc547d8..c6a4443 100644 --- a/docs/troubleshooting/known-issues.md +++ b/docs/troubleshooting/known-issues.md @@ -187,22 +187,25 @@ ### 2.2 classes（TS/NestJS，P1 黄金模板） -| 场景 | 技术/规则 | -| ---------------- | ------------------------------------------------------------------------------------------------ | -| 黄金模板定位 | P1 完整实现所有横切关注点，后续 8 个服务复制此模板 | -| 黄金模板复制流程 | `cp -r services/classes services/xxx` → 改错误码前缀 → 改 proto → 改业务逻辑 → 改 README → 改 CI | -| 横切关注点清单 | 错误处理 / 可观测 / 安全 / 契约 / 测试 / 文档 / 配置 / i18n / CI / Dockerfile | -| 错误处理 | `ApplicationError` 基类 + 子类，错误码 `CLASSES_*` 前缀 | -| 可观测 | pino logger + prom-client metrics + OTel tracer | -| 安全 | auth.middleware（信任 Gateway 头）+ permission.guard + data-scope.interceptor | -| 配置 | 三层配置 + Zod 校验 env | -| i18n | `ERROR_CODES` 映射表（错误码 → i18n key） | -| 测试四类 | 单元（vitest）+ 集成（Testcontainers）+ 契约（Pact）+ E2E（Playwright） | -| 覆盖率门槛 | 领域逻辑 ≥ 80%，Handler ≥ 60%，整体 ≥ 60% | -| Drizzle schema | `mysqlTable` + `varchar`/`timestamp` + `index` | -| ID 生成 | `@paralleldrive/cuid2` 的 `createId()` | -| 响应转换 | repository 返回 Date，service 转换为 `createdAt: number`（时间戳） | -| 阶段特有模式回写 | Outbox（P3）/ CDC（P4）/ 长连接（P5）实现后回写黄金模板 README | +| 场景 | 技术/规则 | +| ---------------- | -------------------------------------------------------------------------------------------------- | +| 黄金模板定位 | P1 完整实现所有横切关注点，后续 8 个服务复制此模板 | +| 黄金模板复制流程 | `cp -r services/classes services/xxx` → 改错误码前缀 → 改 proto → 改业务逻辑 → 改 README → 改 CI | +| 横切关注点清单 | 错误处理 / 可观测 / 安全 / 契约 / 测试 / 文档 / 配置 / i18n / CI / Dockerfile | +| 错误处理 | `ApplicationError` 基类 + 子类，错误码 `CLASSES_*` 前缀 | +| 可观测 | pino logger + prom-client metrics + OTel tracer | +| 安全 | auth.middleware（信任 Gateway 头）+ permission.guard + data-scope.interceptor | +| 配置 | 三层配置 + Zod 校验 env | +| i18n | `ERROR_CODES` 映射表（错误码 → i18n key） | +| 测试四类 | 单元（vitest）+ 集成（Testcontainers）+ 契约（Pact）+ E2E（Playwright） | +| 覆盖率门槛 | 领域逻辑 ≥ 80%，Handler ≥ 60%，整体 ≥ 60% | +| Drizzle schema | `mysqlTable` + `varchar`/`timestamp` + `index` | +| ID 生成 | `@paralleldrive/cuid2` 的 `createId()` | +| 响应转换 | repository 返回 Date，service 转换为 `createdAt: number`（时间戳） | +| 阶段特有模式回写 | Outbox（P3）/ CDC（P4）/ 长连接（P5）实现后回写黄金模板 README | +| 健康检查依赖 | `readyz` 用 Drizzle `getDb().execute(sql\`SELECT 1\`)` 校验，不要依赖 typeorm DataSource DI | +| AppModule 注册 | HealthModule 必须在 `app.module.ts` imports 数组显式声明，否则 NestFactory 不扫描 HealthController | +| 增量编译陷阱 | `tsconfig.json` 显式 `"incremental": false` 覆盖 base，避免 .tsbuildinfo 导致 nest watch 不 emit | ### 2.3 iam（TS/NestJS，P2） @@ -350,6 +353,7 @@ | 日期 | 时间 | 模块 | 做了什么 + 学到什么 | | ---------- | ---- | ------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| 2026-07-09 | 下午 | classes/全局 | **一键启动脚本 NestJS dist/ 不生成根因定位 + classes 健康检查修复**：(1) 根因定位：`tsconfig.base.json` 的 `incremental: true` + `nest-cli.json` 的 `deleteOutDir: true` 冲突。`nest start --watch` 启动时先删除 dist/，tsc 读残留 .tsbuildinfo 认为无变化跳过 emit，dist/ 不生成 → `Cannot find module dist/main`。(2) 修复：6 个 NestJS 服务（classes/iam/teacher-bff/core-edu/content/msg）tsconfig.json 显式加 `"incremental": false` 覆盖 base 配置，删除所有残留 .tsbuildinfo 文件。(3) classes AppModule 缺 HealthModule 导入导致 /healthz 404，iam 同样问题，修复 app.module.ts 加 `imports: [..., HealthModule]`。(4) classes HealthController 误用 TypeORM `DataSource` DI（与 iam 不一致），运行时报 `Nest can't resolve dependencies of the HealthController (DataSource)`。修复：改为 Drizzle `getDb()` 函数式调用，与 iam 一致。(5) 一键启动验证：11/11 应用 + 11/11 基础设施 + 5/5 可观测性端点全绿。**学到**：NestJS + TypeScript incremental 编译是陷阱组合——nest-cli deleteOutDir 删 dist 但 tsc 读 tsbuildinfo 认为无变化，必须在服务级 tsconfig 显式 `incremental: false`；HealthModule 必须在 AppModule imports 中显式声明才能被 NestFactory 扫描到；5 个 NestJS 服务的 HealthController 应统一用 Drizzle `getDb()` 函数式调用而非 TypeORM DataSource DI（项目已弃 TypeORM 改 Drizzle）。 | | 2026-07-09 | 下午 | 全局 | **OTel auto-instrumentations 全服务补全**：(1) NestJS 6 服务（iam/classes/core-edu/content/msg/teacher-bff）tracer.ts 补 `getNodeAutoInstrumentations()`，NodeSDK 传 instrumentations 参数自动埋点 HTTP/Express/DB。(2) Python 2 服务（data-ana/ai）main.py 补 `FastAPIInstrumentor.instrument_app(app)`；ai 补缺失的 `opentelemetry-exporter-otlp` 依赖。(3) teacher-bff 从零补完整 OTel：env.ts 加 OTEL_EXPORTER_OTLP_ENDPOINT 字段 + 新建 shared/observability/tracer.ts + main.ts 调用 initTracer/shutdownTracer + package.json 加 sdk-node/exporter/auto-instrumentations 依赖。(4) Go 2 服务（api-gateway/push-gateway）新建 internal/observability/tracer.go（OTLP HTTP exporter + resource + TracerProvider + W3C propagator）+ main.go 调用 InitTracer + otelgin.Middleware 注册 Gin 中间件；push-gateway config.go 补 OTLPEndpoint 字段。(5) 质量校验全通过：TS typecheck 9 服务 + ESLint 6 服务 + ruff 2 服务 + go vet/build 2 服务零错误。**学到**：`getNodeAutoInstrumentations()` 一次注册所有 Node.js 自动埋点（http/express/dns/fs/net/grpc 等），比手动逐个注册 HttpInstrumentation 更简洁；Go OTel 用 `otlptracehttp.WithEndpoint(host)` + `WithInsecure()` 需从 "http://host:port" URL 解析出 host；otelgin.Middleware 必须在 Recovery 之后其他中间件之前注册，确保所有后续 handler 都被 trace；Python FastAPIInstrumentor.instrument_app(app) 在 app 创建后立即调用，lifespan 不受影响。 | | 2026-07-09 | 下午 | 全局 | **P6 硬化：可观测性 + 部署 + CI 硬化**：(1) 可观测性栈完善：5 个 NestJS 服务 main.ts 添加 `/metrics` Prometheus 端点（用 `app.getHttpAdapter().get('/metrics', ...)` 绕过 DI 容器 get 方法）；prometheus.yml 从 2 个目标扩展到 8 个应用服务 + MySQL/Redis + node-exporter + prometheus 自身 + rule_files + alertmanager 关联；monitoring compose 用 Loki + Promtail 替换未配置的 blackbox-exporter；Grafana datasource 新增 Loki；新建 promtail/config.yml 用 docker_sd_configs 仅采集 `edu-*` 容器日志。(2) 部署 compose 扩展：docker-compose.deploy.yml 从 3 服务扩展到 11 服务（+ iam/teacher-bff/core-edu/content/msg/ai/data-ana/push-gateway），每个服务带 healthcheck + depends_on 条件 + edu-net/edu-shared 双网络；deploy.env.example 补全 Neo4j/ES/ClickHouse/LLM/Kafka 可选依赖配置。(3) teacher-bff 补 health.controller.ts（原缺失 /healthz 导致 deploy depends_on service_healthy 失败）。(4) CI 硬化：移除 lint 步骤的 continue-on-error（ESLint 9 flat config 已配置完成），test 保留 continue-on-error（部分服务无 test 脚本）。**学到**：NestJS `app.get('/metrics')` 会被解析为 DI 容器 `get(typeOrToken)`，必须用 `app.getHttpAdapter().get()` 才能注册 Express 路由；Promtail docker_sd_configs 通过 relabel_configs 的 `regex: '/(edu-.*).*'` 过滤容器名前缀；docker-compose.depends_on.condition: service_healthy 要求被依赖服务必须有 healthcheck 配置，否则启动失败。 | | 2026-07-09 | 下午 | data-ana/infra | **CDC 完整链路实现**：MySQL binlog → Debezium Connect → Kafka → data-ana 消费者 → ClickHouse 宽表。(1) MySQL binlog 配置：log_bin=ON, binlog_format=ROW, binlog_row_image=FULL, server_id=1；用 root 创建 `debezium` 用户授予 REPLICATION SLAVE + REPLICATION CLIENT。(2) Debezium Connect 容器：daocloud 禁用 debezium 镜像改用 `quay.io/debezium/connect:2.7`；MySQL 容器在 edu-minimal_default 网络，需 `docker network connect edu-full_default edu-mysql` 让 Debezium 同时可达；Kafka 必须配置双 listener（INSIDE:kafka:29092 + OUTSIDE:localhost:9092），否则 Debezium 拿到 advertised.listeners 中的 localhost metadata 后切换失败；Debezium 2.x 容器环境变量名用 BOOTSTRAP_SERVERS（不带 KAFKA_ 前缀），通过 envsubst 替换到 connect-distributed.properties。(3) 注册 connector：POST :8083/connectors，配置 topic.prefix=edu-cdc, database.include.list=next_edu_cloud, snapshot.mode=initial，4 张表（core_edu_grades/exams/classes/iam_users）成功产生快照事件。(4) data-ana 消费者实现：新建 cdc_consumer.py 用 aiokafka AIOKafkaConsumer，lifespan 中 asyncio.create_task 后台运行；按 source.table 路由（exams→内存缓存 exam_id→class_id 映射，grades→查缓存填 class_id 后 upsert ClickHouse）；readyz 端点附加 cdc_consumer 状态。(5) ClickHouse 远程访问：默认 default-user.xml 限制 127.0.0.1/::1 无密码，挂载 `clickhouse/users.d/custom-users.xml` 覆盖密码+任意 IP。(6) structlog 24.x API：`make_filtering_bound_logger(level)` 替代废弃的 `make_filtering_logger`。(7) E2E 验证：MySQL INSERT 成绩 → Debezium op=c 事件 → Kafka → 消费者写 ClickHouse 宽表（class_id 通过 exam 缓存正确填充）→ /readyz cdc_consumer=running → /analytics/student/student-002/weakness 返回实时 92 分数据。**学到**：Debezium 2.x 容器 bootstrap.servers 默认值是 0.0.0.0:9092 必须显式覆盖；Kafka 单 listener 配置 localhost 会让容器间通信的客户端拿到 metadata 后切换失败，必须用双 listener；ClickHouse users_xml 存储是 readonly 不能用 ALTER USER 修改密码，必须挂载 users.d 配置文件覆盖；消费者 offset 重置必须先停消费者让 group 处于 Empty 状态才能执行 --reset-offsets。 | diff --git a/infra/docker-compose.yml b/infra/docker-compose.yml index 6c80116..3fa7ba1 100644 --- a/infra/docker-compose.yml +++ b/infra/docker-compose.yml @@ -184,8 +184,14 @@ services: container_name: edu-prometheus profiles: ["observability"] restart: unless-stopped + command: + - "--config.file=/etc/prometheus/prometheus.yml" + - "--storage.tsdb.path=/prometheus" + - "--storage.tsdb.retention.time=15d" + - "--web.enable-lifecycle" volumes: - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro + - prometheus_data:/prometheus ports: - "9090:9090" grafana: @@ -199,6 +205,49 @@ services: - "3030:3000" volumes: - grafana_data:/var/lib/grafana + # ============================================================ + # Exporters（observability profile，与 Prometheus 同网络） + # ============================================================ + node-exporter: + image: docker.m.daocloud.io/prom/node-exporter:v1.8.2 + container_name: edu-node-exporter + profiles: ["observability"] + restart: unless-stopped + command: + - "--path.rootfs=/host" + ports: + - "9100:9100" + volumes: + - /proc:/host/proc:ro + - /sys:/host/sys:ro + - /:/host:ro + mysql-exporter: + image: docker.m.daocloud.io/prom/mysqld-exporter:v0.15.1 + container_name: edu-mysql-exporter + profiles: ["observability"] + restart: unless-stopped + command: + - "--mysqld.address=edu-mysql:3306" + - "--mysqld.username=edu:changeme" + environment: + MYSQLD_EXPORTER_PASSWORD: "changeme" + ports: + - "9104:9104" + depends_on: + mysql: + condition: service_healthy + redis-exporter: + image: docker.m.daocloud.io/oliver006/redis_exporter:v1.67.0 + container_name: edu-redis-exporter + profiles: ["observability"] + restart: unless-stopped + environment: + REDIS_ADDR: "redis://edu-redis:6379" + ports: + - "9121:9121" + depends_on: + redis: + condition: service_started volumes: mysql_data: redis_data: @@ -206,3 +255,4 @@ volumes: neo4j_data: es_data: grafana_data: + prometheus_data: diff --git a/infra/prometheus.yml b/infra/prometheus.yml index 9422c7e..6190a53 100644 --- a/infra/prometheus.yml +++ b/infra/prometheus.yml @@ -89,6 +89,6 @@ scrape_configs: - job_name: 'node-exporter' static_configs: - - targets: ['node-exporter:9100'] + - targets: ['host.docker.internal:9100'] labels: service: node-exporter diff --git a/scripts/start-all.ps1 b/scripts/start-all.ps1 index 094e26c..75b8404 100644 --- a/scripts/start-all.ps1 +++ b/scripts/start-all.ps1 @@ -13,7 +13,8 @@ .\scripts\start-all.ps1 -SkipInfraCheck #> param( - [switch]$SkipInfraCheck + [switch]$SkipInfraCheck, + [switch]$Force ) $ErrorActionPreference = "Stop" @@ -26,7 +27,7 @@ Write-Host "" # ===== 1. Infrastructure health check ===== if (-not $SkipInfraCheck) { - Write-Host "[1/4] Checking infrastructure health..." -ForegroundColor Yellow + Write-Host "[1/6] Checking infrastructure health..." -ForegroundColor Yellow $infraServices = @( @{Name="MySQL"; Container="edu-mysql"}, @{Name="Redis"; Container="edu-redis"}, @@ -64,7 +65,7 @@ if (-not $SkipInfraCheck) { } # ===== 2. Environment variables ===== -Write-Host "[2/4] Preparing environment variables..." -ForegroundColor Yellow +Write-Host "[2/6] Preparing environment variables..." -ForegroundColor Yellow $env:DEV_MODE = "true" $env:OTEL_EXPORTER_OTLP_ENDPOINT = "http://localhost:4318" @@ -88,8 +89,103 @@ Write-Host " DEV_MODE=true (dev-token bypass)" -ForegroundColor Green Write-Host " OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318" -ForegroundColor Green Write-Host "" -# ===== 3. Start application services ===== -Write-Host "[3/4] Starting application services (11 windows)..." -ForegroundColor Yellow +# ===== 3. Port conflict check ===== +Write-Host "[3/6] Checking port conflicts..." -ForegroundColor Yellow + +$portMap = @{ + 3001="classes"; 3002="iam"; 3003="teacher-bff"; 3004="core-edu" + 3005="content"; 3006="data-ana"; 3007="msg"; 3008="ai" + 8080="api-gateway"; 8081="push-gateway"; 3000="teacher-portal" +} + +$conflicts = @() +foreach ($port in $portMap.Keys | Sort-Object) { + $conn = Get-NetTCPConnection -LocalPort $port -State Listen -ErrorAction SilentlyContinue + if ($conn) { + $svcName = $portMap[$port] + $procId = $conn[0].OwningProcess + $procName = "" + try { $procName = (Get-Process -Id $procId -ErrorAction Stop).ProcessName } catch {} + Write-Host " [WARN] Port $port ($svcName) occupied by PID $procId ($procName)" -ForegroundColor Yellow + $conflicts += [PSCustomObject]@{Port=$port; Service=$svcName; PID=$procId; Process=$procName} + } +} + +if ($conflicts.Count -gt 0) { + Write-Host "" + Write-Host " $($conflicts.Count) port(s) already in use." -ForegroundColor Yellow + + $shouldKill = $false + if ($Force) { + Write-Host " -Force specified, killing automatically..." -ForegroundColor White + $shouldKill = $true + } else { + Write-Host " These services may already be running. Options:" -ForegroundColor White + Write-Host " 1. Run .\scripts\stop-all.ps1 -KillByPort first, then re-run start-all" -ForegroundColor White + Write-Host " 2. Re-run with -Force to auto-kill and continue" -ForegroundColor White + Write-Host "" + $answer = Read-Host " Kill existing processes and continue? (y/N)" + if ($answer -eq "y" -or $answer -eq "Y") { + $shouldKill = $true + } + } + + if ($shouldKill) { + foreach ($c in $conflicts) { + try { + Stop-Process -Id $c.PID -Force -ErrorAction Stop + Write-Host " [OK] Killed PID $($c.PID) ($($c.Process)) on port $($c.Port)" -ForegroundColor Green + Start-Sleep -Milliseconds 500 + } catch { + Write-Host " [WARN] Cannot kill PID $($c.PID): $($_.Exception.Message)" -ForegroundColor Yellow + } + } + Start-Sleep -Seconds 2 + } else { + Write-Host " Aborting. Please stop existing services first." -ForegroundColor Red + exit 1 + } +} else { + Write-Host " [OK] All app ports are free" -ForegroundColor Green +} +Write-Host "" + +# ===== 4. Build NestJS services (required for nest start --watch) ===== +Write-Host "[4/6] Building NestJS services (first time required)..." -ForegroundColor Yellow + +$nestjsServices = @( + "@edu/classes-service", + "@edu/iam-service", + "@edu/teacher-bff", + "@edu/core-edu-service", + "@edu/content-service", + "@edu/msg-service" +) + +# Clean tsbuildinfo cache (incremental mode leftover causes tsc to skip emit) +$nestjsDirs = @("classes", "iam", "teacher-bff", "core-edu", "content", "msg") +foreach ($dir in $nestjsDirs) { + $svcPath = Join-Path $ProjectRoot "services\$dir" + if (Test-Path $svcPath) { + Get-ChildItem -Path $svcPath -Filter "*.tsbuildinfo" -Recurse -ErrorAction SilentlyContinue | + Remove-Item -Force -ErrorAction SilentlyContinue + } +} + +foreach ($svc in $nestjsServices) { + Write-Host " Building $svc..." -ForegroundColor Gray -NoNewline + $buildResult = pnpm --filter $svc build 2>&1 + if ($LASTEXITCODE -eq 0) { + Write-Host " [OK]" -ForegroundColor Green + } else { + Write-Host " [FAIL]" -ForegroundColor Red + Write-Host " $buildResult" -ForegroundColor DarkGray + } +} +Write-Host "" + +# ===== 5. Start application services ===== +Write-Host "[5/6] Starting application services (11 windows)..." -ForegroundColor Yellow $services = @( @{Title="edu-app-classes"; Cmd="pnpm"; Args=@("--filter","@edu/classes-service","dev"); Dir="$ProjectRoot"}, @@ -98,8 +194,8 @@ $services = @( @{Title="edu-app-core-edu"; Cmd="pnpm"; Args=@("--filter","@edu/core-edu-service","dev"); Dir="$ProjectRoot"}, @{Title="edu-app-content"; Cmd="pnpm"; Args=@("--filter","@edu/content-service","dev"); Dir="$ProjectRoot"}, @{Title="edu-app-msg"; Cmd="pnpm"; Args=@("--filter","@edu/msg-service","dev"); Dir="$ProjectRoot"}, - @{Title="edu-app-data-ana"; Cmd="uv"; Args=@("run","uvicorn","data_ana.main:app","--host","0.0.0.0","--port","3006","--reload"); Dir="$ProjectRoot\services\data-ana"; PyEnv=$true}, - @{Title="edu-app-ai"; Cmd="uv"; Args=@("run","uvicorn","ai.main:app","--host","0.0.0.0","--port","3008","--reload"); Dir="$ProjectRoot\services\ai"; PyEnv=$true}, + @{Title="edu-app-data-ana"; Cmd="uv"; Args=@("run","uvicorn","data_ana.main:app","--app-dir","src","--host","0.0.0.0","--port","3006","--reload"); Dir="$ProjectRoot\services\data-ana"; PyEnv=$true}, + @{Title="edu-app-ai"; Cmd="uv"; Args=@("run","uvicorn","ai.main:app","--app-dir","src","--host","0.0.0.0","--port","3008","--reload"); Dir="$ProjectRoot\services\ai"; PyEnv=$true}, @{Title="edu-app-api-gateway"; Cmd="go"; Args=@("run","."); Dir="$ProjectRoot\services\api-gateway"; GoEnv=$true}, @{Title="edu-app-push-gateway"; Cmd="go"; Args=@("run","."); Dir="$ProjectRoot\services\push-gateway";GoEnv=$true}, @{Title="edu-app-teacher-portal";Cmd="pnpm";Args=@("--filter","teacher-portal","dev"); Dir="$ProjectRoot"} @@ -129,11 +225,11 @@ foreach ($svc in $services) { } Write-Host "" -Write-Host " All services started in new windows. Waiting 30s for init..." -ForegroundColor Yellow -Start-Sleep -Seconds 30 +Write-Host " All services started in new windows. Waiting 40s for init..." -ForegroundColor Yellow +Start-Sleep -Seconds 40 -# ===== 4. Health check ===== -Write-Host "[4/4] Health check..." -ForegroundColor Yellow +# ===== 6. Health check ===== +Write-Host "[6/6] Health check..." -ForegroundColor Yellow Write-Host "" $healthServices = @( @@ -152,26 +248,30 @@ $healthServices = @( $okCount = 0 $failCount = 0 +$failedServices = @() foreach ($svc in $healthServices) { $retries = 0 - $maxRetries = 5 + $maxRetries = 3 $success = $false + $lastError = "" while ($retries -lt $maxRetries -and -not $success) { try { - $null = Invoke-RestMethod -Uri $svc.Url -Method Get -TimeoutSec 3 -ErrorAction Stop + $null = Invoke-RestMethod -Uri $svc.Url -Method Get -TimeoutSec 5 -ErrorAction Stop Write-Host " [OK] $($svc.Name)" -ForegroundColor Green $success = $true $okCount++ } catch { + $lastError = $_.Exception.Message $retries++ if ($retries -lt $maxRetries) { - Start-Sleep -Seconds 3 + Start-Sleep -Seconds 5 } } } if (-not $success) { - Write-Host " [FAIL] $($svc.Name) (failed after $maxRetries retries)" -ForegroundColor Red + Write-Host " [FAIL] $($svc.Name) (after $maxRetries retries: $lastError)" -ForegroundColor Red $failCount++ + $failedServices += $svc.Name } } @@ -179,13 +279,23 @@ Write-Host "" Write-Host "========================================" -ForegroundColor Cyan Write-Host " Started: [OK] $okCount ready / [FAIL] $failCount failed" -ForegroundColor Cyan Write-Host "========================================" -ForegroundColor Cyan + +if ($failCount -gt 0) { + Write-Host "" + Write-Host "Failed services: $($failedServices -join ', ')" -ForegroundColor Red + Write-Host "" + Write-Host "Troubleshooting:" -ForegroundColor Yellow + Write-Host " 1. Check the service window for error output" -ForegroundColor White + Write-Host " 2. Verify dependencies: pnpm install / uv sync / go mod tidy" -ForegroundColor White + Write-Host " 3. Re-run health check: .\scripts\health-check.ps1" -ForegroundColor White + Write-Host " 4. Stop and retry: .\scripts\stop-all.ps1 -KillByPort then .\scripts\start-all.ps1" -ForegroundColor White + Write-Host "" + exit 1 +} + Write-Host "" Write-Host "Next steps:" -ForegroundColor Yellow Write-Host " Health check: .\scripts\health-check.ps1" -ForegroundColor White Write-Host " CDC test: .\scripts\test-cdc.ps1" -ForegroundColor White Write-Host " Stop all: .\scripts\stop-all.ps1" -ForegroundColor White Write-Host "" - -if ($failCount -gt 0) { - exit 1 -} diff --git a/services/classes/src/app.module.ts b/services/classes/src/app.module.ts index fe03cdc..225edb6 100644 --- a/services/classes/src/app.module.ts +++ b/services/classes/src/app.module.ts @@ -1,7 +1,8 @@ -import { Module } from '@nestjs/common'; -import { ClassesModule } from './classes/classes.module.js'; +import { Module } from "@nestjs/common"; +import { ClassesModule } from "./classes/classes.module.js"; +import { HealthModule } from "./shared/health/health.module.js"; @Module({ - imports: [ClassesModule], + imports: [ClassesModule, HealthModule], }) export class AppModule {} diff --git a/services/classes/src/shared/health/health.controller.ts b/services/classes/src/shared/health/health.controller.ts index faba498..a2d2b8a 100644 --- a/services/classes/src/shared/health/health.controller.ts +++ b/services/classes/src/shared/health/health.controller.ts @@ -1,7 +1,8 @@ -import { Controller, Get, HttpException, HttpStatus } from '@nestjs/common'; -import { DataSource } from 'typeorm'; +import { Controller, Get, HttpException, HttpStatus } from "@nestjs/common"; +import { sql } from "drizzle-orm"; +import { getDb } from "../../config/database.js"; -const SERVICE_NAME = 'classes'; +const SERVICE_NAME = "classes"; /** * 健康检查端点。 @@ -14,33 +15,37 @@ const SERVICE_NAME = 'classes'; */ @Controller() export class HealthController { - constructor(private readonly dataSource: DataSource) {} - - @Get('healthz') + @Get("healthz") liveness(): { status: string; service: string; timestamp: string } { return { - status: 'ok', + status: "ok", service: SERVICE_NAME, timestamp: new Date().toISOString(), }; } - @Get('readyz') - async readiness(): Promise<{ status: string; service: string; timestamp: string }> { + @Get("readyz") + async readiness(): Promise<{ + status: string; + service: string; + timestamp: string; + }> { try { - await this.dataSource.query('SELECT 1'); + const db = getDb(); + await db.execute(sql`SELECT 1`); return { - status: 'ok', + status: "ok", service: SERVICE_NAME, timestamp: new Date().toISOString(), }; } catch (error) { throw new HttpException( { - status: 'error', + status: "error", service: SERVICE_NAME, timestamp: new Date().toISOString(), - error: error instanceof Error ? error.message : 'database unreachable', + error: + error instanceof Error ? error.message : "database unreachable", }, HttpStatus.SERVICE_UNAVAILABLE, ); diff --git a/services/classes/tsconfig.json b/services/classes/tsconfig.json index cd22ff6..7ca5991 100644 --- a/services/classes/tsconfig.json +++ b/services/classes/tsconfig.json @@ -8,6 +8,7 @@ "emitDecoratorMetadata": true, "outDir": "./dist", "rootDir": "./src", + "incremental": false, "types": ["node"] }, "include": ["src/**/*"], diff --git a/services/content/tsconfig.json b/services/content/tsconfig.json index cd22ff6..7ca5991 100644 --- a/services/content/tsconfig.json +++ b/services/content/tsconfig.json @@ -8,6 +8,7 @@ "emitDecoratorMetadata": true, "outDir": "./dist", "rootDir": "./src", + "incremental": false, "types": ["node"] }, "include": ["src/**/*"], diff --git a/services/core-edu/tsconfig.json b/services/core-edu/tsconfig.json index cd22ff6..7ca5991 100644 --- a/services/core-edu/tsconfig.json +++ b/services/core-edu/tsconfig.json @@ -8,6 +8,7 @@ "emitDecoratorMetadata": true, "outDir": "./dist", "rootDir": "./src", + "incremental": false, "types": ["node"] }, "include": ["src/**/*"], diff --git a/services/iam/src/app.module.ts b/services/iam/src/app.module.ts index 34ae55e..5f5e8e0 100644 --- a/services/iam/src/app.module.ts +++ b/services/iam/src/app.module.ts @@ -1,7 +1,8 @@ -import { Module } from '@nestjs/common'; -import { IamModule } from './iam/iam.module.js'; +import { Module } from "@nestjs/common"; +import { IamModule } from "./iam/iam.module.js"; +import { HealthModule } from "./shared/health/health.module.js"; @Module({ - imports: [IamModule], + imports: [IamModule, HealthModule], }) export class AppModule {} diff --git a/services/iam/tsconfig.json b/services/iam/tsconfig.json index cd22ff6..7ca5991 100644 --- a/services/iam/tsconfig.json +++ b/services/iam/tsconfig.json @@ -8,6 +8,7 @@ "emitDecoratorMetadata": true, "outDir": "./dist", "rootDir": "./src", + "incremental": false, "types": ["node"] }, "include": ["src/**/*"], diff --git a/services/msg/tsconfig.json b/services/msg/tsconfig.json index cd22ff6..7ca5991 100644 --- a/services/msg/tsconfig.json +++ b/services/msg/tsconfig.json @@ -8,6 +8,7 @@ "emitDecoratorMetadata": true, "outDir": "./dist", "rootDir": "./src", + "incremental": false, "types": ["node"] }, "include": ["src/**/*"], diff --git a/services/teacher-bff/src/main.ts b/services/teacher-bff/src/main.ts index 88f12a9..af80f9d 100644 --- a/services/teacher-bff/src/main.ts +++ b/services/teacher-bff/src/main.ts @@ -3,6 +3,7 @@ import { NestFactory } from "@nestjs/core"; import { AppModule } from "./app.module.js"; import { env } from "./config/env.js"; import { initTracer, shutdownTracer } from "./shared/observability/tracer.js"; +import { metricsRegistry } from "./shared/observability/metrics.js"; async function bootstrap(): Promise { initTracer(); @@ -12,6 +13,12 @@ async function bootstrap(): Promise { app.enableShutdownHooks(); + // Prometheus 指标端点：不鉴权，供 Prometheus 抓取。 + app.getHttpAdapter().get("/metrics", async (_req, res) => { + res.set("Content-Type", metricsRegistry.contentType); + res.end(await metricsRegistry.metrics()); + }); + await app.listen(env.PORT); console.log(`Teacher BFF started on port ${env.PORT}`); diff --git a/services/teacher-bff/src/shared/observability/metrics.ts b/services/teacher-bff/src/shared/observability/metrics.ts new file mode 100644 index 0000000..4e316e2 --- /dev/null +++ b/services/teacher-bff/src/shared/observability/metrics.ts @@ -0,0 +1,26 @@ +import promClient from "prom-client"; + +const registry = new promClient.Registry(); +registry.setDefaultLabels({ service: "teacher-bff" }); + +registry.registerMetric( + new promClient.Counter({ + name: "teacher_bff_requests_total", + help: "Total number of teacher-bff requests", + labelNames: ["method", "endpoint", "status"], + }), +); + +registry.registerMetric( + new promClient.Histogram({ + name: "teacher_bff_request_duration_seconds", + help: "Teacher-bff request duration in seconds", + labelNames: ["method", "endpoint"], + buckets: [0.01, 0.05, 0.1, 0.3, 0.5, 1, 3, 5], + }), +); + +// 自动收集 Node.js 进程级指标（CPU/内存/事件循环/GC等） +promClient.collectDefaultMetrics({ register: registry }); + +export { registry as metricsRegistry }; diff --git a/services/teacher-bff/tsconfig.json b/services/teacher-bff/tsconfig.json index cd22ff6..7ca5991 100644 --- a/services/teacher-bff/tsconfig.json +++ b/services/teacher-bff/tsconfig.json @@ -8,6 +8,7 @@ "emitDecoratorMetadata": true, "outDir": "./dist", "rootDir": "./src", + "incremental": false, "types": ["node"] }, "include": ["src/**/*"],