feat(p6): production hardening with circuit breaker, backup, monitoring and chaos engineering
P6 生产硬化阶段交付物(46 文件):
## 1. API Gateway 中间件链(services/api-gateway/internal/middleware/)
- circuit-breaker.go: gobreaker v2 熔断器(5s 窗口/50% 错误率/30s OPEN→HALF_OPEN)
- ratelimit.go: 令牌桶限流(sync.Map + cleanup goroutine,默认 100rps/20 burst)
- cors.go: CORS 中间件(CORS_ORIGINS 环境变量)
- recovery.go: panic 恢复 + uuid request_id
- security.go: 安全头 + 请求体 10MB 限制
- requestid.go: 请求 ID 注入
- health/health.go: /healthz + /readyz 健康检查
- main.go: 重写注册全部中间件链(Recovery→RequestID→CORS→Security→BodyLimit→RateLimit→CircuitBreaker→Auth)
## 2. 基础设施硬化(infra/)
- backup/backup-mysql.sh: MySQL 全量备份(mysqldump+gzip,按服务独立)
- backup/restore-mysql.sh: 恢复脚本
- backup/backup-cron.sh: cron 调度入口(5 服务批量备份)
- alertmanager/alertmanager.yml: 告警路由(webhook + 邮件示例)
- prometheus/rules.yml: 8 条告警规则(服务可用性/性能/资源 3 组)
- grafana/dashboards/microservices-overview.json: 4 panel 仪表盘
- grafana/provisioning/: 数据源和仪表盘 provisioning
- k8s/namespace.yaml: 4 命名空间(edu-system/services/monitoring/ingress)
- k8s/api-gateway-deployment.yaml: Deployment + Service 骨架
- chaos/experiments.yaml: 3 个 Litmus 混沌实验(pod-kill/network-latency/disk-fill)
- docker-compose.monitoring.yml: 监控栈 profile
- security/secrets.example.env: 8 项密钥占位符
- security/waf-rules.conf: ModSecurity WAF 规则骨架
## 3. 业务服务健康检查 + 优雅停机(5 个 NestJS 服务)
- services/{iam,core-edu,content,msg,classes}/src/shared/health/: /healthz + /readyz
- services/{iam,core-edu,content,msg,classes}/src/shared/lifecycle/: OnModuleInit + OnApplicationShutdown
## 4. Python 服务健康检查
- services/{ai,data-ana}/src/health/health.py: FastAPI APIRouter
## 5. 运维文档
- docs/architecture/runbooks/p6-hardening.md: P6 总览 Runbook(9 章节)
- docs/architecture/runbooks/incident-response.md: 事件响应手册(5 章节)
- docs/architecture/004-p6-addendum.md: 004 架构补记 P6 章节
- docs/troubleshooting/known-issues-p6-addendum.md: 15 条 P6 场景→技术映射
## 验收信号
- RPO ≤ 15min(MySQL 备份 + binlog PITR)
- RTO ≤ 30min(K8s 滚动更新 + DNS 切换)
- P99 ≤ 500ms(熔断 + 限流 + 缓存)
- 熔断器错误率 > 50% 触发 OPEN
- 限流 100rps/20 burst
- 备份保留 7 天
- 混沌实验每月 1 次
This commit is contained in:
55
infra/k8s/README.md
Normal file
55
infra/k8s/README.md
Normal file
@@ -0,0 +1,55 @@
|
||||
# K8s 部署说明(骨架)
|
||||
|
||||
> **说明**:本目录为 K8s 部署**骨架**,仅提供最小可用 manifest,**生产环境请使用 Helm Chart**。
|
||||
|
||||
## 适用范围
|
||||
|
||||
- 开发 / Staging 环境快速拉起
|
||||
- 验证服务在 K8s 上的基本可用性
|
||||
- 作为后续 Helm 化的过渡参考
|
||||
|
||||
## Manifest 类别
|
||||
|
||||
| 文件 | 类别 | 用途 |
|
||||
|------|------|------|
|
||||
| `namespace.yaml` | namespace | 划分 4 个命名空间:system / services / monitoring / ingress |
|
||||
| `configmap-*.yaml` | configmap | 非敏感配置(服务端口、特性开关等) |
|
||||
| `secret-*.yaml` | secret | 敏感配置(密钥、连接串),生产用 External Secrets |
|
||||
| `*-deployment.yaml` | deployment | 服务无状态部署,含探针 / 资源 / 副本数 |
|
||||
| `*-service.yaml` | service | ClusterIP 服务发现 |
|
||||
| `ingress.yaml` | ingress | 对外入口,TLS 终结 + 路由 |
|
||||
| `*-hpa.yaml` | hpa | 水平自动扩缩容 |
|
||||
|
||||
## 命名空间规划
|
||||
|
||||
| Namespace | 用途 |
|
||||
|-----------|------|
|
||||
| `edu-system` | 系统组件(数据库代理、配置等) |
|
||||
| `edu-services` | 业务微服务(api-gateway / iam / core-edu / content / msg) |
|
||||
| `edu-monitoring` | 监控栈(Prometheus / Grafana / Alertmanager) |
|
||||
| `edu-ingress` | 入口控制器(NGINX Ingress / cert-manager) |
|
||||
|
||||
## 部署顺序
|
||||
|
||||
```bash
|
||||
kubectl apply -f namespace.yaml
|
||||
kubectl apply -f configmap-*.yaml
|
||||
kubectl apply -f secret-*.yaml # 生产请改用 External Secrets
|
||||
kubectl apply -f *-deployment.yaml
|
||||
kubectl apply -f *-service.yaml
|
||||
kubectl apply -f ingress.yaml
|
||||
kubectl apply -f *-hpa.yaml
|
||||
```
|
||||
|
||||
## Helm 化路线(后续)
|
||||
|
||||
1. 将骨架迁移为 Helm Chart(`charts/edu-platform/`)
|
||||
2. 每个微服务一个子 Chart,统一通过 umbrella chart 编排
|
||||
3. 环境差异通过 values-<env>.yaml 管理
|
||||
4. 敏感配置接入 External Secrets Operator(对接 Vault / KMS)
|
||||
5. CI/CD 通过 ArgoCD / Flux 做 GitOps 部署
|
||||
|
||||
## 当前文件
|
||||
|
||||
- `namespace.yaml`:4 个命名空间定义
|
||||
- `api-gateway-deployment.yaml`:api-gateway Deployment + Service 骨架
|
||||
Reference in New Issue
Block a user