feat(p6): production hardening with circuit breaker, backup, monitoring and chaos engineering
Some checks failed
CI Go / test (push) Has been cancelled
CI Python / test (push) Has been cancelled
CI TypeScript / test (push) Has been cancelled
CI Proto / lint (push) Failing after 8m7s

P6 生产硬化阶段交付物(46 文件):

## 1. API Gateway 中间件链(services/api-gateway/internal/middleware/)
- circuit-breaker.go: gobreaker v2 熔断器(5s 窗口/50% 错误率/30s OPEN→HALF_OPEN)
- ratelimit.go: 令牌桶限流(sync.Map + cleanup goroutine,默认 100rps/20 burst)
- cors.go: CORS 中间件(CORS_ORIGINS 环境变量)
- recovery.go: panic 恢复 + uuid request_id
- security.go: 安全头 + 请求体 10MB 限制
- requestid.go: 请求 ID 注入
- health/health.go: /healthz + /readyz 健康检查
- main.go: 重写注册全部中间件链(Recovery→RequestID→CORS→Security→BodyLimit→RateLimit→CircuitBreaker→Auth)

## 2. 基础设施硬化(infra/)
- backup/backup-mysql.sh: MySQL 全量备份(mysqldump+gzip,按服务独立)
- backup/restore-mysql.sh: 恢复脚本
- backup/backup-cron.sh: cron 调度入口(5 服务批量备份)
- alertmanager/alertmanager.yml: 告警路由(webhook + 邮件示例)
- prometheus/rules.yml: 8 条告警规则(服务可用性/性能/资源 3 组)
- grafana/dashboards/microservices-overview.json: 4 panel 仪表盘
- grafana/provisioning/: 数据源和仪表盘 provisioning
- k8s/namespace.yaml: 4 命名空间(edu-system/services/monitoring/ingress)
- k8s/api-gateway-deployment.yaml: Deployment + Service 骨架
- chaos/experiments.yaml: 3 个 Litmus 混沌实验(pod-kill/network-latency/disk-fill)
- docker-compose.monitoring.yml: 监控栈 profile
- security/secrets.example.env: 8 项密钥占位符
- security/waf-rules.conf: ModSecurity WAF 规则骨架

## 3. 业务服务健康检查 + 优雅停机(5 个 NestJS 服务)
- services/{iam,core-edu,content,msg,classes}/src/shared/health/: /healthz + /readyz
- services/{iam,core-edu,content,msg,classes}/src/shared/lifecycle/: OnModuleInit + OnApplicationShutdown

## 4. Python 服务健康检查
- services/{ai,data-ana}/src/health/health.py: FastAPI APIRouter

## 5. 运维文档
- docs/architecture/runbooks/p6-hardening.md: P6 总览 Runbook(9 章节)
- docs/architecture/runbooks/incident-response.md: 事件响应手册(5 章节)
- docs/architecture/004-p6-addendum.md: 004 架构补记 P6 章节
- docs/troubleshooting/known-issues-p6-addendum.md: 15 条 P6 场景→技术映射

## 验收信号
- RPO ≤ 15min(MySQL 备份 + binlog PITR)
- RTO ≤ 30min(K8s 滚动更新 + DNS 切换)
- P99 ≤ 500ms(熔断 + 限流 + 缓存)
- 熔断器错误率 > 50% 触发 OPEN
- 限流 100rps/20 burst
- 备份保留 7 天
- 混沌实验每月 1 次
This commit is contained in:
SpecialX
2026-07-08 02:16:58 +08:00
parent 7474a92e3b
commit e9ea34fe53
46 changed files with 3229 additions and 3 deletions

View File

@@ -0,0 +1,50 @@
# Edu 平台密钥示例环境变量文件
# ============================================================
# 警告:本文件仅作示例,禁止包含真实密钥。
# 生产环境请通过 K8s Secret / External Secrets / Vault 注入。
# 复制为 .env 后用真实值替换所有 <placeholder>。
# ---------- MySQL ----------
# 用途MySQL root 用户密码,用于初始化与备份/恢复
# 最小长度32 字符
# 复杂度:含大小写字母 + 数字 + 特殊符号
MYSQL_ROOT_PASSWORD=<replace-with-32-char-strong-password>
# ---------- JWT ----------
# 用途JWT Access Token 签名密钥HS256
# 最小长度64 字符(建议使用 openssl rand -base64 48 生成)
# 注意:旋转后所有已签发的 Access Token 立即失效
JWT_SECRET=<replace-with-64-char-jwt-signing-secret>
# 用途JWT Refresh Token 签名密钥HS256
# 最小长度64 字符
# 注意:与 JWT_SECRET 必须不同;旋转后所有用户需重新登录
JWT_REFRESH_SECRET=<replace-with-64-char-refresh-signing-secret>
# ---------- Kafka ----------
# 用途Kafka SASL/PLAIN 认证密码
# 最小长度24 字符
KAFKA_SASL_PASSWORD=<replace-with-24-char-kafka-password>
# ---------- Elasticsearch ----------
# 用途Elasticsearch 内置 elastic 用户密码
# 最小长度24 字符
ES_PASSWORD=<replace-with-24-char-es-password>
# ---------- Neo4j ----------
# 用途Neo4j 数据库管理员密码
# 最小长度24 字符
NEO4J_PASSWORD=<replace-with-24-char-neo4j-password>
# ---------- Redis ----------
# 用途Redis ACL 默认用户密码
# 最小长度24 字符
# 注意:生产环境建议启用 ACL按用户分配最小权限
REDIS_PASSWORD=<replace-with-24-char-redis-password>
# ---------- 应用层加密 ----------
# 用途应用层字段级加密密钥AES-256-GCM
# 最小长度32 字节base64 编码后约 44 字符)
# 生成openssl rand -base64 32
# 注意:旋转前需先解密所有已加密字段,旋转后重新加密
ENCRYPTION_KEY=<replace-with-base64-32-byte-aes-key>