Spring Prometheus

2023/10/17

Gathering Metrics from Spring Boot with Prometheus and Grafana

使用 docker-compose 部署和配置环境

grafana

使用指南

Prometheus

数据模型

Prometheus 具有多维分析的能力，数据模型有如下几部分组成。

Metric Name(指标名称) + Labels(标签) + Timestamp(时间戳) + Value/Sample(监控值/样品)

  • Metric Name(指标名称)：监控对象的含义（例如，http_request_total - 表示当前系统接收到的HTTP请求总量）。
  • 标签(label)：表示当前样本的特征维度，是一个K/V结构，通过这些维度 Prometheus 可以对样本数据进行过滤，聚合等。
  • 时间戳(timestamp)：一个精确到毫秒的时间戳。
  • 样本值(value)：一个float64的浮点型数据表示当前样本的值。

参考资料

业务层

如何检测/测量系统

spring boot

spring-boot-actuator 的包 org.springframework.boot.actuate.metrics

  • org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter

参考资料

micrometer

io.micrometer#micrometer-core 的包 io.micrometer.core.instrument.binder

  • io.micrometer.core.instrument.binder.jvm.JvmMemoryMetrics
  • io.micrometer.core.instrument.binder.jvm.JvmGcMetrics

异常处理

log/2023/11/error-2023-11-06_0.log.gz:2023-11-06 22:14:13.498 [] [uof-ticket-11-1] ERROR c.i.s.t.c.s.u.RollbackBetSettlementHandler( 266) - Prometheus requires that all meters with the same name have the same set of tag keys. There is already an existing meter name d ‘ilot_sport_ticket_rollback_tickets_total’ containing tag keys [application, bet_from, currency, game_rb_ret, game_winrb_ret, payo ut_type, rbpayout, rbstake, region, stack]. The meter you are attempting to register has keys [application, region, stack]. log/2023/11/error-2023-11-06_0.log.gz:java.lang.IllegalArgumentException: Prometheus requires that all meters with the same name hav e the same set of tag keys. There is already an existing meter named ‘ilot_sport_ticket_rollback_tickets_total’ containing tag keys [application, bet_from, currency, game_rb_ret, game_winrb_ret, payout_type, rbpayout, rbstake, region, stack]. The meter you are att empting to register has keys [application, region, stack].

2023-11-09 03:59:03.093 [] [uof-ticket-13-1] ERROR c.i.s.t.c.s.u.BetSettlementHandler(251) - Prometheus requires that all meters with the same name have the same set of tag keys. There is already an existing meter named ‘ilot_sport_ticket_rollback_tickets_total’ containing tag keys [application, region, stack]. The meter you are attempting to register has keys [application, bet_from, currency, game_rb_ret, game_winrb_ret, payout_type, rbpayout, rbstake, region, stack].java.lang.IllegalArgumentException: Prometheus requires that all meters with the same name have the same set of tag keys. There is already an existing meter named ‘ilot_sport_ticket_rollback_tickets_total’ containing tag keys [application, region, stack]. The meter you are attempting to register has keys [application, bet_from, currency, game_rb_ret, game_winrb_ret, payout_type, rbpayout, rbstake, region, stack].

grafana

参考资料

主要

promql 和 grafana 查询

查询日志增量示例

时间戳 1697773320 -> 2023-10-20T03:42:00.000Z

promql

increase(logback_events_total{application="xxx-service",level="debug",stack='dev'}[1m] @ 1697773320)
# 124.8
rate(logback_events_total{application="xxx-service",level="debug",stack='dev'}[1m] @ 1697773320)
# 2.08
logback_events_total{application="xxx-service",level="debug", stack='dev'}[1m] @ 1697773320
# 16916 @1697773263.216
# 16918 @1697773273.216
# 16920 @1697773283.216
# 16930 @1697773293.216
# 16984 @1697773303.216
# 17020 @1697773313.216

官方文档描述 increase 是 rate 的封装的语法糖。所以上面的 increase 可以转换为

rate(logback_events_total{application="xxx-service",level="debug",stack='dev'}[1m] @ 1697773320) * 60s
# 124.8

Spring

Micrometer

Prometheus

Grafana

次要

promql 查询