Files
Edu/infra/backup/backup-cron.sh
SpecialX e9ea34fe53
Some checks failed
CI Go / test (push) Has been cancelled
CI Python / test (push) Has been cancelled
CI TypeScript / test (push) Has been cancelled
CI Proto / lint (push) Failing after 8m7s
feat(p6): production hardening with circuit breaker, backup, monitoring and chaos engineering
P6 生产硬化阶段交付物(46 文件):

## 1. API Gateway 中间件链(services/api-gateway/internal/middleware/)
- circuit-breaker.go: gobreaker v2 熔断器(5s 窗口/50% 错误率/30s OPEN→HALF_OPEN)
- ratelimit.go: 令牌桶限流(sync.Map + cleanup goroutine,默认 100rps/20 burst)
- cors.go: CORS 中间件(CORS_ORIGINS 环境变量)
- recovery.go: panic 恢复 + uuid request_id
- security.go: 安全头 + 请求体 10MB 限制
- requestid.go: 请求 ID 注入
- health/health.go: /healthz + /readyz 健康检查
- main.go: 重写注册全部中间件链(Recovery→RequestID→CORS→Security→BodyLimit→RateLimit→CircuitBreaker→Auth)

## 2. 基础设施硬化(infra/)
- backup/backup-mysql.sh: MySQL 全量备份(mysqldump+gzip,按服务独立)
- backup/restore-mysql.sh: 恢复脚本
- backup/backup-cron.sh: cron 调度入口(5 服务批量备份)
- alertmanager/alertmanager.yml: 告警路由(webhook + 邮件示例)
- prometheus/rules.yml: 8 条告警规则(服务可用性/性能/资源 3 组)
- grafana/dashboards/microservices-overview.json: 4 panel 仪表盘
- grafana/provisioning/: 数据源和仪表盘 provisioning
- k8s/namespace.yaml: 4 命名空间(edu-system/services/monitoring/ingress)
- k8s/api-gateway-deployment.yaml: Deployment + Service 骨架
- chaos/experiments.yaml: 3 个 Litmus 混沌实验(pod-kill/network-latency/disk-fill)
- docker-compose.monitoring.yml: 监控栈 profile
- security/secrets.example.env: 8 项密钥占位符
- security/waf-rules.conf: ModSecurity WAF 规则骨架

## 3. 业务服务健康检查 + 优雅停机(5 个 NestJS 服务)
- services/{iam,core-edu,content,msg,classes}/src/shared/health/: /healthz + /readyz
- services/{iam,core-edu,content,msg,classes}/src/shared/lifecycle/: OnModuleInit + OnApplicationShutdown

## 4. Python 服务健康检查
- services/{ai,data-ana}/src/health/health.py: FastAPI APIRouter

## 5. 运维文档
- docs/architecture/runbooks/p6-hardening.md: P6 总览 Runbook(9 章节)
- docs/architecture/runbooks/incident-response.md: 事件响应手册(5 章节)
- docs/architecture/004-p6-addendum.md: 004 架构补记 P6 章节
- docs/troubleshooting/known-issues-p6-addendum.md: 15 条 P6 场景→技术映射

## 验收信号
- RPO ≤ 15min(MySQL 备份 + binlog PITR)
- RTO ≤ 30min(K8s 滚动更新 + DNS 切换)
- P99 ≤ 500ms(熔断 + 限流 + 缓存)
- 熔断器错误率 > 50% 触发 OPEN
- 限流 100rps/20 burst
- 备份保留 7 天
- 混沌实验每月 1 次
2026-07-08 02:16:58 +08:00

75 lines
1.7 KiB
Bash
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
#!/bin/bash
# MySQL 备份 cron 调度入口
# 调用 backup-mysql.sh 备份所有微服务数据库,单个失败不阻塞其他
# 用法: backup-cron.sh [--keep <days>]
set -euo pipefail
# ---------- 参数解析 ----------
KEEP_DAYS=7
while [[ $# -gt 0 ]]; do
case "$1" in
--keep)
KEEP_DAYS="${2:-7}"
shift 2
;;
-h|--help)
echo "Usage: $0 [--keep <days>]" >&2
echo " --keep 保留天数(默认 7" >&2
exit 0
;;
*)
echo "[ERROR] 未知参数: $1" >&2
exit 1
;;
esac
done
# ---------- 配置 ----------
# 每个服务对应一个独立数据库
SERVICES=(iam core-edu content msg classes)
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
BACKUP_SCRIPT="${SCRIPT_DIR}/backup-mysql.sh"
# ---------- 日志函数 ----------
log() {
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" >&2
}
if [[ ! -x "$BACKUP_SCRIPT" ]]; then
log "[ERROR] 备份脚本不存在或不可执行: ${BACKUP_SCRIPT}"
exit 1
fi
# ---------- 主流程 ----------
log "[INFO] 开始批量备份,共 ${#SERVICES[@]} 个服务"
TOTAL=0
SUCCESS=0
FAILED=0
FAILED_LIST=()
for svc in "${SERVICES[@]}"; do
TOTAL=$((TOTAL + 1))
log "[INFO] ---- 备份服务: ${svc} ----"
if "$BACKUP_SCRIPT" --service "$svc" --keep "$KEEP_DAYS"; then
SUCCESS=$((SUCCESS + 1))
log "[INFO] 服务 ${svc} 备份成功"
else
FAILED=$((FAILED + 1))
FAILED_LIST+=("$svc")
log "[ERROR] 服务 ${svc} 备份失败,继续下一个"
fi
done
# ---------- 汇总 ----------
log "[INFO] 批量备份结束: 总计=${TOTAL} 成功=${SUCCESS} 失败=${FAILED}"
if [[ ${FAILED} -gt 0 ]]; then
log "[ERROR] 失败服务列表: ${FAILED_LIST[*]}"
exit 1
fi
exit 0