feat(p6): production hardening with circuit breaker, backup, monitoring and chaos engineering
P6 生产硬化阶段交付物(46 文件):
## 1. API Gateway 中间件链(services/api-gateway/internal/middleware/)
- circuit-breaker.go: gobreaker v2 熔断器(5s 窗口/50% 错误率/30s OPEN→HALF_OPEN)
- ratelimit.go: 令牌桶限流(sync.Map + cleanup goroutine,默认 100rps/20 burst)
- cors.go: CORS 中间件(CORS_ORIGINS 环境变量)
- recovery.go: panic 恢复 + uuid request_id
- security.go: 安全头 + 请求体 10MB 限制
- requestid.go: 请求 ID 注入
- health/health.go: /healthz + /readyz 健康检查
- main.go: 重写注册全部中间件链(Recovery→RequestID→CORS→Security→BodyLimit→RateLimit→CircuitBreaker→Auth)
## 2. 基础设施硬化(infra/)
- backup/backup-mysql.sh: MySQL 全量备份(mysqldump+gzip,按服务独立)
- backup/restore-mysql.sh: 恢复脚本
- backup/backup-cron.sh: cron 调度入口(5 服务批量备份)
- alertmanager/alertmanager.yml: 告警路由(webhook + 邮件示例)
- prometheus/rules.yml: 8 条告警规则(服务可用性/性能/资源 3 组)
- grafana/dashboards/microservices-overview.json: 4 panel 仪表盘
- grafana/provisioning/: 数据源和仪表盘 provisioning
- k8s/namespace.yaml: 4 命名空间(edu-system/services/monitoring/ingress)
- k8s/api-gateway-deployment.yaml: Deployment + Service 骨架
- chaos/experiments.yaml: 3 个 Litmus 混沌实验(pod-kill/network-latency/disk-fill)
- docker-compose.monitoring.yml: 监控栈 profile
- security/secrets.example.env: 8 项密钥占位符
- security/waf-rules.conf: ModSecurity WAF 规则骨架
## 3. 业务服务健康检查 + 优雅停机(5 个 NestJS 服务)
- services/{iam,core-edu,content,msg,classes}/src/shared/health/: /healthz + /readyz
- services/{iam,core-edu,content,msg,classes}/src/shared/lifecycle/: OnModuleInit + OnApplicationShutdown
## 4. Python 服务健康检查
- services/{ai,data-ana}/src/health/health.py: FastAPI APIRouter
## 5. 运维文档
- docs/architecture/runbooks/p6-hardening.md: P6 总览 Runbook(9 章节)
- docs/architecture/runbooks/incident-response.md: 事件响应手册(5 章节)
- docs/architecture/004-p6-addendum.md: 004 架构补记 P6 章节
- docs/troubleshooting/known-issues-p6-addendum.md: 15 条 P6 场景→技术映射
## 验收信号
- RPO ≤ 15min(MySQL 备份 + binlog PITR)
- RTO ≤ 30min(K8s 滚动更新 + DNS 切换)
- P99 ≤ 500ms(熔断 + 限流 + 缓存)
- 熔断器错误率 > 50% 触发 OPEN
- 限流 100rps/20 burst
- 备份保留 7 天
- 混沌实验每月 1 次
This commit is contained in:
119
infra/docker-compose.monitoring.yml
Normal file
119
infra/docker-compose.monitoring.yml
Normal file
@@ -0,0 +1,119 @@
|
||||
# 监控栈 docker-compose(独立 profile)
|
||||
# 使用方式: docker compose -f docker-compose.monitoring.yml --profile monitoring up -d
|
||||
version: "3.9"
|
||||
|
||||
networks:
|
||||
edu-network:
|
||||
name: edu-network
|
||||
external: true
|
||||
|
||||
volumes:
|
||||
prometheus-data:
|
||||
name: edu-prometheus-data
|
||||
alertmanager-data:
|
||||
name: edu-alertmanager-data
|
||||
grafana-data:
|
||||
name: edu-grafana-data
|
||||
|
||||
services:
|
||||
# ============================================================
|
||||
# Prometheus - 指标采集与存储
|
||||
# ============================================================
|
||||
prometheus:
|
||||
image: prom/prometheus:v2.54.1
|
||||
container_name: edu-prometheus
|
||||
profiles: ["monitoring"]
|
||||
restart: unless-stopped
|
||||
command:
|
||||
- "--config.file=/etc/prometheus/prometheus.yml"
|
||||
- "--storage.tsdb.path=/prometheus"
|
||||
- "--storage.tsdb.retention.time=15d"
|
||||
- "--web.enable-lifecycle"
|
||||
- "--web.enable-admin-api"
|
||||
ports:
|
||||
- "9090:9090"
|
||||
volumes:
|
||||
- ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
|
||||
- ./prometheus/rules.yml:/etc/prometheus/rules.yml:ro
|
||||
- prometheus-data:/prometheus
|
||||
networks:
|
||||
- edu-network
|
||||
|
||||
# ============================================================
|
||||
# Alertmanager - 告警路由与抑制
|
||||
# ============================================================
|
||||
alertmanager:
|
||||
image: prom/alertmanager:v0.27.0
|
||||
container_name: edu-alertmanager
|
||||
profiles: ["monitoring"]
|
||||
restart: unless-stopped
|
||||
command:
|
||||
- "--config.file=/etc/alertmanager/alertmanager.yml"
|
||||
- "--storage.path=/alertmanager"
|
||||
ports:
|
||||
- "9093:9093"
|
||||
volumes:
|
||||
- ./alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
|
||||
- alertmanager-data:/alertmanager
|
||||
networks:
|
||||
- edu-network
|
||||
depends_on:
|
||||
- prometheus
|
||||
|
||||
# ============================================================
|
||||
# Grafana - 可视化
|
||||
# ============================================================
|
||||
grafana:
|
||||
image: grafana/grafana:11.2.2
|
||||
container_name: edu-grafana
|
||||
profiles: ["monitoring"]
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
- GF_SECURITY_ADMIN_USER=${GRAFANA_ADMIN_USER:-admin}
|
||||
- GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_ADMIN_PASSWORD:-admin}
|
||||
- GF_USERS_ALLOW_SIGN_UP=false
|
||||
- GF_AUTH_ANONYMOUS_ENABLED=false
|
||||
ports:
|
||||
- "3000:3000"
|
||||
volumes:
|
||||
- ./grafana/provisioning:/etc/grafana/provisioning:ro
|
||||
- ./grafana/dashboards:/var/lib/grafana/dashboards:ro
|
||||
- grafana-data:/var/lib/grafana
|
||||
networks:
|
||||
- edu-network
|
||||
depends_on:
|
||||
- prometheus
|
||||
|
||||
# ============================================================
|
||||
# node-exporter - 主机指标采集
|
||||
# ============================================================
|
||||
node-exporter:
|
||||
image: prom/node-exporter:v1.8.2
|
||||
container_name: edu-node-exporter
|
||||
profiles: ["monitoring"]
|
||||
restart: unless-stopped
|
||||
command:
|
||||
- "--path.rootfs=/host"
|
||||
ports:
|
||||
- "9100:9100"
|
||||
volumes:
|
||||
- /proc:/host/proc:ro
|
||||
- /sys:/host/sys:ro
|
||||
- /:/host:ro
|
||||
networks:
|
||||
- edu-network
|
||||
|
||||
# ============================================================
|
||||
# blackbox-exporter - 黑盒探测(HTTP / TCP / ICMP)
|
||||
# ============================================================
|
||||
blackbox-exporter:
|
||||
image: prom/blackbox-exporter:v0.25.0
|
||||
container_name: edu-blackbox-exporter
|
||||
profiles: ["monitoring"]
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "9115:9115"
|
||||
volumes:
|
||||
- ./blackbox/blackbox.yml:/etc/blackbox_exporter/config.yml:ro
|
||||
networks:
|
||||
- edu-network
|
||||
Reference in New Issue
Block a user