xiner/Edu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(infra): p6 hardening - metrics collection and registry mirror

Browse Source 
- 4 services add collectDefaultMetrics for process metrics
- docker-compose.yml images prefixed with docker.m.daocloud.io
- Prometheus v0.51.0 (nonexistent) fixed to v2.51.0
- Grafana port remapped 3000 to 3030 to avoid teacher-portal conflict
- ClickHouse edu_analytics database initialized
- known-issues.md documents 7 new P6 scenario-to-rule mappings
This commit is contained in:
SpecialX
2026-07-09 12:29:36 +08:00
parent 566060fade
commit 958b17c9d8
6 changed files with 81 additions and 57 deletions
Download Patch File Download Diff File Expand all files Collapse all files

34
docs/troubleshooting/known-issues.md
View File

@@ -86,20 +86,26 @@
### 1.6 可观测性（OTel + Prometheus + Loki）
| 场景 | 技术/规则 |
| ----------------- | ------------------------------------------------------------------------------------------------------------------- |
| P1 最小可观测集 | 每服务结构化日志 + `/metrics` + OTel SDK 初始化（不引入完整后端） |
| 三支柱 | Logs（pino/winston/zap）+ Metrics（prom-client）+ Traces（OTel SDK） |
| traceId 注入 | Gateway 注入 → 服务读取 header → 日志/响应携带 |
| P6 完整后端 | Loki（日志）+ Grafana（仪表盘）+ Jaeger（trace）+ Prometheus（metrics） |
| Prometheus 指标 | `http_request_duration_seconds`（Histogram）+ `http_requests_total`（Counter） |
| 采样策略 | P1 全量 trace，P6 引入采样率降低开销 |
| 日志参数顺序 | `log.error({ err: error, userId, traceId }, "操作描述")`，错误对象字段名用 `err` |
| P6 /metrics 端点 | NestJS 用 `app.getHttpAdapter().get('/metrics', handler)`，不能用 `app.get()`（会被解析为 DI 容器 get） |
| P6 prometheus.yml | 8 个应用服务 + MySQL/Redis + node-exporter + prometheus 自身；rule_files 引用 rules.yml；alerting 关联 alertmanager |
| P6 Grafana 数据源 | provisioning/datasources 同时声明 Prometheus（默认）和 Loki（uid=loki） |
| P6 Promtail 采集 | docker_sd_configs + relabel_configs 仅采集 `edu-*` 前缀容器日志，避免无关日志 |
| P6 service label | static_configs.labels.service 给所有抓取目标打服务标签，告警规则按 service 聚合 |
| 场景 | 技术/规则 |
| ----------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| P1 最小可观测集 | 每服务结构化日志 + `/metrics` + OTel SDK 初始化（不引入完整后端） |
| 三支柱 | Logs（pino/winston/zap）+ Metrics（prom-client）+ Traces（OTel SDK） |
| traceId 注入 | Gateway 注入 → 服务读取 header → 日志/响应携带 |
| P6 完整后端 | Loki（日志）+ Grafana（仪表盘）+ Jaeger（trace）+ Prometheus（metrics） |
| Prometheus 指标 | `http_request_duration_seconds`（Histogram）+ `http_requests_total`（Counter） |
| 采样策略 | P1 全量 trace，P6 引入采样率降低开销 |
| 日志参数顺序 | `log.error({ err: error, userId, traceId }, "操作描述")`，错误对象字段名用 `err` |
| P6 /metrics 端点 | NestJS 用 `app.getHttpAdapter().get('/metrics', handler)`，不能用 `app.get()`（会被解析为 DI 容器 get） |
| P6 prometheus.yml | 8 个应用服务 + MySQL/Redis + node-exporter + prometheus 自身；rule_files 引用 rules.yml；alerting 关联 alertmanager |
| P6 Grafana 数据源 | provisioning/datasources 同时声明 Prometheus（默认）和 Loki（uid=loki） |
| P6 Promtail 采集 | docker_sd_configs + relabel_configs 仅采集 `edu-*` 前缀容器日志，避免无关日志 |
| P6 service label | static_configs.labels.service 给所有抓取目标打服务标签，告警规则按 service 聚合 |
| P6 collectDefaultMetrics | prom-client 的 `collectDefaultMetrics({ register })` 自动收集进程级指标（CPU/内存/事件循环/GC），无需业务埋点即可让 /metrics 有数据 |
| P6 Counter/Histogram 埋点缺失 | metrics.ts 定义了 Counter/Histogram 但 service/controller 未调用 `.inc()`/`.observe()`，需后续补 HTTP 中间件自动埋点或业务埋点 |
| P6 OTel instrumentations 缺失 | tracer.ts 只配置 traceExporter 未注册 auto-instrumentations（HttpInstrumentation/ExpressInstrumentation），导致 Jaeger 收不到业务 trace，需后续补 `@opentelemetry/auto-instrumentations` |
| P6 镜像源配置 | 国内 docker.io 被墙，compose image 必须加 `docker.m.daocloud.io/` 前缀；Elastic 官方镜像在 docker.elastic.co 不被墙 |
| P6 Grafana 端口冲突 | Grafana 默认 3000 与 teacher-portal Next.js dev 冲突，改映射为 3030:3000 |
| P6 compose --no-deps | mysql/redis 已在另一 compose 项目运行时，启动新服务用 `--no-deps` + 显式指定服务名，避免重建依赖容器 | |
### 1.7 微前端 Module Federation

24
infra/docker-compose.yml
View File

@@ -1,7 +1,8 @@
name: edu-full
services:
mysql:
image: mysql:8.0
# 镜像加速：通过 daocloud 镜像源绕过 docker.io 被墙问题
image: docker.m.daocloud.io/library/mysql:8.0
container_name: edu-mysql
restart: unless-stopped
environment:
@@ -20,7 +21,7 @@ services:
timeout: 5s
retries: 5
redis:
image: redis:7-alpine
image: docker.m.daocloud.io/library/redis:7-alpine
container_name: edu-redis
restart: unless-stopped
ports:
@@ -33,7 +34,7 @@ services:
timeout: 3s
retries: 5
kafka:
image: confluentinc/cp-kafka:7.6.0
image: docker.m.daocloud.io/confluentinc/cp-kafka:7.6.0
container_name: edu-kafka
profiles: ["p3", "p4", "p5", "p6"]
restart: unless-stopped
@@ -54,7 +55,7 @@ services:
timeout: 10s
retries: 5
zookeeper:
image: confluentinc/cp-zookeeper:7.6.0
image: docker.m.daocloud.io/confluentinc/cp-zookeeper:7.6.0
container_name: edu-zookeeper
profiles: ["p3", "p4", "p5", "p6"]
restart: unless-stopped
@@ -66,7 +67,7 @@ services:
timeout: 5s
retries: 5
clickhouse:
image: clickhouse/clickhouse-server:24.3
image: docker.m.daocloud.io/clickhouse/clickhouse-server:24.3
container_name: edu-clickhouse
profiles: ["p4", "p5", "p6"]
restart: unless-stopped
@@ -81,7 +82,7 @@ services:
timeout: 5s
retries: 5
neo4j:
image: neo4j:5.20
image: docker.m.daocloud.io/library/neo4j:5.20
container_name: edu-neo4j
profiles: ["p4", "p5", "p6"]
restart: unless-stopped
@@ -98,6 +99,7 @@ services:
timeout: 10s
retries: 5
elasticsearch:
# Elastic 官方镜像在 docker.elastic.co，非 Docker Hub，通常不被墙
image: docker.elastic.co/elasticsearch/elasticsearch:8.13.0
container_name: edu-es
profiles: ["p5", "p6"]
@@ -116,7 +118,7 @@ services:
timeout: 10s
retries: 5
jaeger:
image: jaegertracing/all-in-one:1.57
image: docker.m.daocloud.io/jaegertracing/all-in-one:1.57
container_name: edu-jaeger
profiles: ["observability"]
restart: unless-stopped
@@ -126,7 +128,7 @@ services:
- "16686:16686"
- "4318:4318"
prometheus:
image: prom/prometheus:v0.51.0
image: docker.m.daocloud.io/prom/prometheus:v2.51.0
container_name: edu-prometheus
profiles: ["observability"]
restart: unless-stopped
@@ -135,14 +137,14 @@ services:
ports:
- "9090:9090"
grafana:
image: grafana/grafana:10.4.0
image: docker.m.daocloud.io/grafana/grafana:10.4.0
container_name: edu-grafana
profiles: ["observability"]
restart: unless-stopped
environment:
GF_SECURITY_ADMIN_PASSWORD: admin
ports:
- "3001:3001"
- "3030:3000"
volumes:
- grafana_data:/var/lib/grafana
volumes:
@@ -151,4 +153,4 @@ volumes:
clickhouse_data:
neo4j_data:
es_data:
grafana_data:
grafana_data:

20
services/classes/src/shared/observability/metrics.ts
View File

@@ -1,24 +1,28 @@
import promClient from 'prom-client';
import promClient from "prom-client";
const registry = new promClient.Registry();
// 修复：在本地 registry 上设置默认标签（原代码误用全局 register）
registry.setDefaultLabels({ service: 'classes' });
registry.setDefaultLabels({ service: "classes" });
registry.registerMetric(
new promClient.Counter({
name: 'classes_requests_total',
help: 'Total number of class requests',
labelNames: ['method', 'endpoint', 'status'],
name: "classes_requests_total",
help: "Total number of class requests",
labelNames: ["method", "endpoint", "status"],
}),
);
registry.registerMetric(
new promClient.Histogram({
name: 'classes_request_duration_seconds',
help: 'Class request duration in seconds',
labelNames: ['method', 'endpoint'],
name: "classes_request_duration_seconds",
help: "Class request duration in seconds",
labelNames: ["method", "endpoint"],
buckets: [0.01, 0.05, 0.1, 0.3, 0.5, 1, 3, 5],
}),
);
// 自动收集 Node.js 进程级指标（CPU/内存/事件循环/GC等）
// 这些指标无需业务代码埋点，prom-client 自动采集
promClient.collectDefaultMetrics({ register: registry });
export { registry as metricsRegistry };

20
services/content/src/shared/observability/metrics.ts
View File

@@ -1,24 +1,28 @@
import promClient from 'prom-client';
import promClient from "prom-client";
const registry = new promClient.Registry();
// 修复：在本地 registry 上设置默认标签（原代码误用全局 register）
registry.setDefaultLabels({ service: 'content' });
registry.setDefaultLabels({ service: "content" });
registry.registerMetric(
new promClient.Counter({
name: 'content_requests_total',
help: 'Total number of content requests',
labelNames: ['method', 'endpoint', 'status'],
name: "content_requests_total",
help: "Total number of content requests",
labelNames: ["method", "endpoint", "status"],
}),
);
registry.registerMetric(
new promClient.Histogram({
name: 'content_request_duration_seconds',
help: 'Content request duration in seconds',
labelNames: ['method', 'endpoint'],
name: "content_request_duration_seconds",
help: "Content request duration in seconds",
labelNames: ["method", "endpoint"],
buckets: [0.01, 0.05, 0.1, 0.3, 0.5, 1, 3, 5],
}),
);
// 自动收集 Node.js 进程级指标（CPU/内存/事件循环/GC等）
// 这些指标无需业务代码埋点，prom-client 自动采集
promClient.collectDefaultMetrics({ register: registry });
export { registry as metricsRegistry };

20
services/iam/src/shared/observability/metrics.ts
View File

@@ -1,23 +1,27 @@
import promClient from 'prom-client';
import promClient from "prom-client";
const registry = new promClient.Registry();
registry.setDefaultLabels({ service: 'iam' });
registry.setDefaultLabels({ service: "iam" });
registry.registerMetric(
new promClient.Counter({
name: 'iam_requests_total',
help: 'Total number of iam requests',
labelNames: ['method', 'endpoint', 'status'],
name: "iam_requests_total",
help: "Total number of iam requests",
labelNames: ["method", "endpoint", "status"],
}),
);
registry.registerMetric(
new promClient.Histogram({
name: 'iam_request_duration_seconds',
help: 'Iam request duration in seconds',
labelNames: ['method', 'endpoint'],
name: "iam_request_duration_seconds",
help: "Iam request duration in seconds",
labelNames: ["method", "endpoint"],
buckets: [0.01, 0.05, 0.1, 0.3, 0.5, 1, 3, 5],
}),
);
// 自动收集 Node.js 进程级指标（CPU/内存/事件循环/GC等）
// 这些指标无需业务代码埋点，prom-client 自动采集
promClient.collectDefaultMetrics({ register: registry });
export { registry as metricsRegistry };

20
services/msg/src/shared/observability/metrics.ts
View File

@@ -1,24 +1,28 @@
import promClient from 'prom-client';
import promClient from "prom-client";
const registry = new promClient.Registry();
// 修复：在本地 registry 上设置默认标签（原代码误用全局 register）
registry.setDefaultLabels({ service: 'msg' });
registry.setDefaultLabels({ service: "msg" });
registry.registerMetric(
new promClient.Counter({
name: 'msg_requests_total',
help: 'Total number of msg requests',
labelNames: ['method', 'endpoint', 'status'],
name: "msg_requests_total",
help: "Total number of msg requests",
labelNames: ["method", "endpoint", "status"],
}),
);
registry.registerMetric(
new promClient.Histogram({
name: 'msg_request_duration_seconds',
help: 'Msg request duration in seconds',
labelNames: ['method', 'endpoint'],
name: "msg_request_duration_seconds",
help: "Msg request duration in seconds",
labelNames: ["method", "endpoint"],
buckets: [0.01, 0.05, 0.1, 0.3, 0.5, 1, 3, 5],
}),
);
// 自动收集 Node.js 进程级指标（CPU/内存/事件循环/GC等）
// 这些指标无需业务代码埋点，prom-client 自动采集
promClient.collectDefaultMetrics({ register: registry });
export { registry as metricsRegistry };