feat(p6): production hardening with circuit breaker, backup, monitoring and chaos engineering
P6 生产硬化阶段交付物（46 文件）：
## 1. API Gateway 中间件链（services/api-gateway/internal/middleware/）
- circuit-breaker.go: gobreaker v2 熔断器（5s 窗口/50% 错误率/30s OPEN→HALF_OPEN）
- ratelimit.go: 令牌桶限流（sync.Map + cleanup goroutine，默认 100rps/20 burst）
- cors.go: CORS 中间件（CORS_ORIGINS 环境变量）
- recovery.go: panic 恢复 + uuid request_id
- security.go: 安全头 + 请求体 10MB 限制
- requestid.go: 请求 ID 注入
- health/health.go: /healthz + /readyz 健康检查
- main.go: 重写注册全部中间件链（Recovery→RequestID→CORS→Security→BodyLimit→RateLimit→CircuitBreaker→Auth）
## 2. 基础设施硬化（infra/）
- backup/backup-mysql.sh: MySQL 全量备份（mysqldump+gzip，按服务独立）
- backup/restore-mysql.sh: 恢复脚本
- backup/backup-cron.sh: cron 调度入口（5 服务批量备份）
- alertmanager/alertmanager.yml: 告警路由（webhook + 邮件示例）
- prometheus/rules.yml: 8 条告警规则（服务可用性/性能/资源 3 组）
- grafana/dashboards/microservices-overview.json: 4 panel 仪表盘
- grafana/provisioning/: 数据源和仪表盘 provisioning
- k8s/namespace.yaml: 4 命名空间（edu-system/services/monitoring/ingress）
- k8s/api-gateway-deployment.yaml: Deployment + Service 骨架
- chaos/experiments.yaml: 3 个 Litmus 混沌实验（pod-kill/network-latency/disk-fill）
- docker-compose.monitoring.yml: 监控栈 profile
- security/secrets.example.env: 8 项密钥占位符
- security/waf-rules.conf: ModSecurity WAF 规则骨架
## 3. 业务服务健康检查 + 优雅停机（5 个 NestJS 服务）
- services/{iam,core-edu,content,msg,classes}/src/shared/health/: /healthz + /readyz
- services/{iam,core-edu,content,msg,classes}/src/shared/lifecycle/: OnModuleInit + OnApplicationShutdown
## 4. Python 服务健康检查
- services/{ai,data-ana}/src/health/health.py: FastAPI APIRouter
## 5. 运维文档
- docs/architecture/runbooks/p6-hardening.md: P6 总览 Runbook（9 章节）
- docs/architecture/runbooks/incident-response.md: 事件响应手册（5 章节）
- docs/architecture/004-p6-addendum.md: 004 架构补记 P6 章节
- docs/troubleshooting/known-issues-p6-addendum.md: 15 条 P6 场景→技术映射
## 验收信号
- RPO ≤ 15min（MySQL 备份 + binlog PITR）
- RTO ≤ 30min（K8s 滚动更新 + DNS 切换）
- P99 ≤ 500ms（熔断 + 限流 + 缓存）
- 熔断器错误率 > 50% 触发 OPEN
- 限流 100rps/20 burst
- 备份保留 7 天
- 混沌实验每月 1 次
This commit is contained in:
SpecialX
139
infra/chaos/experiments.yaml
Normal file
139
infra/chaos/experiments.yaml
Normal file
@@ -0,0 +1,139 @@
|
||||
# 混沌实验配置（Litmus Chaos 骨架）
|
||||
# 仅作 YAML 骨架，注释说明用途。实际运行需配合 Litmus Chaos Operator。
|
||||
# 文档：https://litmuschaos.github.io/litmus/
|
||||
apiVersion: litmuschaos.io/v1alpha1
|
||||
kind: ChaosEngine
|
||||
metadata:
|
||||
name: edu-chaos-experiments
|
||||
namespace: edu-monitoring
|
||||
labels:
|
||||
app.kubernetes.io/part-of: edu-platform
|
||||
app.kubernetes.io/component: chaos
|
||||
spec:
|
||||
appinfo:
|
||||
appns: edu-services
|
||||
applabel: "app.kubernetes.io/name=api-gateway"
|
||||
appkind: deployment
|
||||
chaosServiceAccount: litmus-chaos-sa
|
||||
components:
|
||||
experiments:
|
||||
# ============================================================
|
||||
# 实验 1：服务 Pod 杀死（验证自愈与副本数）
|
||||
# ============================================================
|
||||
- name: pod-delete
|
||||
spec:
|
||||
components:
|
||||
env:
|
||||
# 杀死 1 个 Pod
|
||||
- name: TOTAL_CHAOS_DURATION
|
||||
value: "30"
|
||||
- name: CHAOS_INTERVAL
|
||||
value: "10"
|
||||
- name: FORCE
|
||||
value: "false"
|
||||
- name: PODS_AFFECTED_PERC
|
||||
value: "50"
|
||||
- name: TARGET_CONTAINER
|
||||
value: "api-gateway"
|
||||
# 假设：杀死 50% Pod 后，服务仍可对外可用（最少 1 个副本健康）
|
||||
# 稳态：up{job="api-gateway"} >= 1
|
||||
steadyStateHypothesis:
|
||||
steadyStateHypothesis:
|
||||
probe:
|
||||
- name: api-gateway-still-up
|
||||
type: httpProbe
|
||||
mode: Continuous
|
||||
runProperties:
|
||||
probeTimeout: 5
|
||||
httpProbeInputs:
|
||||
url: http://api-gateway.edu-services.svc:8080/healthz
|
||||
method:
|
||||
get: {}
|
||||
phases:
|
||||
- name: pre-chaos
|
||||
description: "混沌前稳态验证"
|
||||
- name: inject
|
||||
description: "注入 Pod 删除"
|
||||
- name: post-chaos
|
||||
description: "混沌后自愈验证"
|
||||
# ============================================================
|
||||
# 实验 2：网络延迟注入（验证超时与重试）
|
||||
# ============================================================
|
||||
- name: pod-network-latency
|
||||
spec:
|
||||
components:
|
||||
env:
|
||||
# 注入 200ms 网络延迟
|
||||
- name: NETWORK_LATENCY
|
||||
value: "200"
|
||||
- name: TOTAL_CHAOS_DURATION
|
||||
value: "60"
|
||||
- name: CHAOS_INTERVAL
|
||||
value: "10"
|
||||
- name: NETWORK_INTERFACE
|
||||
value: "eth0"
|
||||
- name: TARGET_CONTAINER
|
||||
value: "api-gateway"
|
||||
# 假设：200ms 延迟下 P99 < 2s，无 5xx 雪崩
|
||||
# 稳态：错误率 < 5%
|
||||
steadyStateHypothesis:
|
||||
steadyStateHypothesis:
|
||||
probe:
|
||||
- name: error-rate-below-threshold
|
||||
type: promProbe
|
||||
mode: Continuous
|
||||
runProperties:
|
||||
probeTimeout: 5
|
||||
promProbeInputs:
|
||||
source:
|
||||
url: http://prometheus.edu-monitoring.svc:9090
|
||||
query: |
|
||||
sum(rate(http_requests_total{service="api-gateway",status=~"5.."}[1m]))
|
||||
/ sum(rate(http_requests_total{service="api-gateway"}[1m]))
|
||||
comparator:
|
||||
criteria: "<="
|
||||
value: "0.05"
|
||||
phases:
|
||||
- name: pre-chaos
|
||||
description: "混沌前稳态验证"
|
||||
- name: inject
|
||||
description: "注入 200ms 网络延迟"
|
||||
- name: post-chaos
|
||||
description: "混沌后恢复验证"
|
||||
# ============================================================
|
||||
# 实验 3：磁盘填充（验证磁盘压力下日志写入与告警）
|
||||
# ============================================================
|
||||
- name: disk-fill
|
||||
spec:
|
||||
components:
|
||||
env:
|
||||
# 填充至 80% 磁盘使用率
|
||||
- name: FILL_PERCENTAGE
|
||||
value: "80"
|
||||
- name: TOTAL_CHAOS_DURATION
|
||||
value: "120"
|
||||
- name: CHAOS_INTERVAL
|
||||
value: "30"
|
||||
- name: TARGET_CONTAINER
|
||||
value: "api-gateway"
|
||||
# 假设：磁盘 80% 使用率下服务仍可写入日志，触发 DiskSpaceLow 告警
|
||||
# 稳态：服务 /healthz 可用
|
||||
steadyStateHypothesis:
|
||||
steadyStateHypothesis:
|
||||
probe:
|
||||
- name: api-gateway-healthz
|
||||
type: httpProbe
|
||||
mode: Continuous
|
||||
runProperties:
|
||||
probeTimeout: 5
|
||||
httpProbeInputs:
|
||||
url: http://api-gateway.edu-services.svc:8080/healthz
|
||||
method:
|
||||
get: {}
|
||||
phases:
|
||||
- name: pre-chaos
|
||||
description: "混沌前稳态验证"
|
||||
- name: inject
|
||||
description: "填充磁盘至 80%"
|
||||
- name: post-chaos
|
||||
description: "混沌后清理与恢复验证"
|
||||
Reference in New Issue
Block a user