用 Prometheus 指标给分布式数据库做容量规划:以 TiDB 为例

“下个季度要扩多少台?”做数据库的团队几乎都被问过这个问题,通常是在大促或者新业务上线前几周。常见的做法是先拍一个数,再乘个系数“保险一点”,结果要么当天不够用,要么一年里大部分时间都在浪费机器。 之前我和一位同事写过一个小工具 tidbcloud-metrics,用 Prometheus 里现成的监控数据来回答这个问题。超大规模的 TiDB Cloud 集群每次大促前要做容量评估、决定扩多少节点,我们都靠它来算。代码和当时的环境绑得比较紧,但方法本身是通用的。这篇文章用虚构的数据把这套方法过一遍:采哪些指标,怎么汇总,留多少余量,以及这个简单模型在哪些地方不准。 例子用的是 TiDB,换成其他分布式数据库,或者任何按节点暴露指标的无状态服务,思路都一样。 核心公式 对每个组件的每种资源,需要的节点数是: 所需节点数 = (单节点用量 × 当前节点数 × 业务增长倍数) / (单节点容量 × 目标水位) 单节点用量:观察窗口内每个节点对这种资源的用量汇总值,比如 P99。 单节点用量 × 当前节点数:大致等于集群当前的总需求。 业务增长倍数:预计流量会涨多少,1.5 就是涨 50%。 单节点容量:一个节点能提供多少,比如 32 核或 2 TiB 磁盘。 目标水位:你愿意让节点跑到多满。除以 0.5 意味着希望节点利用率不超过...

Capacity Planning a Distributed Database from Prometheus Metrics

“How many nodes do we need for next quarter?” is a question every database team gets, usually a few weeks before a big launch or a sales event. The honest answer often starts with a guess, gets padded “to be...

How to ensure TiDB cluster can use TiCDC to synchronize data?

This article focuses on the more difficult to determine parts of the TiCDC’s limitations. Currently, TiCDC unsupported scenarios and limitations as below: A TiKV cluster that uses RawKV alone. The CREATE SEQUENCE DDL operation and the SEQUENCE function in TiDB....

TiDB Operator Enable TLS For MySQL Client With cert-manager

TiDB Operatior supports enable and disable TLS for MySQL client for an existing TiDB cluster. This article will show you how to enable it. Certificates can be issued by cfssl and cert-manager, here I will choose the cloud native way...

S3 Compatible MinIO Operator Deployment

Object store service is a common service nowdays, especially S3 compatible. The introduction of MinIO can be found here. This article will demostrate how to setup MinIO Operator for testing purpose, this is NOT a comprehensive guide for production scenario....