Sherry Yin

用 MySQL 的经验理解 MongoDB 鉴权：账号、角色与集群内部认证

2026-10-11T00:00:00+08:00

从 MySQL 转到 MongoDB 的人，很少会在 CRUD 上卡住，卡住的往往是鉴权：用户明明建了，连接却报 Authentication failed；角色授了，权限却不是想象中的那样；副本集开了认证以后，节点之间互相连不上。原因多半是带着 MySQL 的心智模型去理解 MongoDB，而两者在几个关键地方并不一样。

这篇文章按 MySQL 的概念顺序，逐个对照 MongoDB 的做法，重点放在差异和容易踩的坑上。文中所有命令和输出都来自一套本地实验环境：

MySQL 8.4
MongoDB 8.0.4，一个 3 节点副本集 rs0，另有一个最小的分片集群（1 个 config server、1 个分片、1 个 mongos）
两台客户端容器 client-a（172.30.0.21）和 client-b（172.30.0.22），用来演示按来源 IP 限制登录

MySQL 用的是 8.4。还在用 8.0 的话，和认证相关的主要差别有几处：8.0 默认仍启用 mysql_native_password，8.4 默认禁用，用它创建账号会报 Plugin 'mysql_native_password' is not loaded；8.4 去掉了 default_authentication_plugin 变量，改用 authentication_policy；8.4 还去掉了 SHOW SLAVE STATUS 这类旧的复制语法。依赖 mysql_native_password 的老驱动连 8.4 会失败，需要用 --mysql-native-password=ON 显式打开，而到了 9.0 这个插件已经被移除。

实验环境的代码在 GitHub 上：mongo-mysql-auth-lab，./lab.sh up 启动环境，./lab.sh run 00 到 11 分别对应文中各节，其中 10 是 MySQL 8.0 和 8.4 的对比，11 是 MySQL 复制账号需要的权限。命令都是完整的；输出里过长的部分，比如报错信息里的 lsid 和 $clusterTime，我用 ... 省略了。

先分清认证和授权

认证（authentication）回答”你是谁”，授权（authorization）回答”你能做什么”。这两步合起来，中文里常被统称为鉴权，本文标题用的就是这个意思；下文提到具体某一步时，只用认证和授权。MySQL 和 MongoDB 都有这两步，但组合方式不一样，后面很多差异都可以追溯到这里。

MySQL 把两者绑在一起。建立连接时必须先完成认证：用户名、来源主机和密码要匹配上 mysql.user 里的某个账号，才会有会话；之后每执行一条语句，再按授权表检查权限。不存在”先连上、还没认证”的状态：

[client-a]$ mysql -h mysql -e 'SELECT 1'
ERROR 1045 (28000): Access denied for user 'root'@'172.30.0.21' (using password: NO)

唯一能把认证和授权一起关掉的是 --skip-grant-tables，一般只在找回 root 密码这类紧急情况下使用。

MongoDB 把两者拆开了。开启访问控制以后，没认证的连接照样能建立，只是除了 hello、ping 这类命令以外什么都做不了：

[client-a]$ mongosh "mongodb://mongo1:27017/?directConnection=true" \
    --eval 'print("ping:", db.adminCommand({ping: 1}).ok);
            print("hello.me:", db.hello().me);
            db.adminCommand({listDatabases: 1})'
ping: 1
hello.me: mongo1:27017
MongoServerError: Command listDatabases requires authentication

更关键的是，授权是一个单独的开关，也就是配置里的 security.authorization，默认关闭，开启集群内部认证（keyfile 或 x509）时会自动打开。授权关闭时，连接如果带了用户名和密码，密码照样会被验证，错了一样报 Authentication failed；可一旦验证通过，权限就不再检查了。下面是一个没开访问控制的单机 mongod，用户 reader 只有 shop 库的 read 角色：

[client-a]$ mongosh "mongodb://reader:wrong-pw@noauth:27017/shop" --eval 'db.orders.countDocuments()'
MongoServerError: Authentication failed.

[client-a]$ mongosh "mongodb://reader:reader-pw@noauth:27017/shop" \
    --eval 'print(JSON.stringify(db.runCommand({connectionStatus: 1}).authInfo.authenticatedUsers));
            db.orders.insertOne({item: "pen"}).acknowledged'
[{"user":"reader","db":"shop"}]
true

reader 认证成功了，只读角色却写入成功。所以”用户都建好了，密码也验证通过了”并不代表集群受到了保护，要确认授权真的开着。可以看启动参数，parsed 里没有 security 一项就说明没开：

[client-a]$ mongosh "mongodb://reader:reader-pw@noauth:27017/shop" \
    --eval 'JSON.stringify(db.adminCommand({getCmdLineOpts: 1}).parsed)'
{"net":{"bindIp":"*"}}

分清这两步，报错也就好理解了：

报错	哪一步出的问题	含义
MySQL `ERROR 1045 Access denied for user`	认证	账号不存在、主机不匹配或密码错误
MySQL `ERROR 1142` / `1143 command denied`	授权	已经登录，但没有这张表或这一列的权限
MongoDB `Authentication failed`	认证	认证库里没有这个用户、密码错误，或不满足来源限制
MongoDB `Command <命令名> requires authentication`	还没认证	访问控制已开启，这个连接没有认证
MongoDB `not authorized on <库名> to execute command`	授权	已经认证，但角色里没有这个操作

账号是什么：user@host 和认证库

在 MySQL 里，账号由用户名和主机两部分组成，'alice'@'%' 和 'alice'@'172.30.0.21' 是两个完全独立的账号，可以有不同的密码和权限：

CREATE USER 'alice'@'%'           IDENTIFIED BY 'alice-any-host';
CREATE USER 'alice'@'172.30.0.21' IDENTIFIED BY 'alice-client-a';

从 client-a 登录时，MySQL 会优先匹配更具体的主机，所以只有 client-a 那个账号的密码能用：

[client-a]$ mysql -h mysql -u alice -p -e "SELECT USER(), CURRENT_USER()"
+-------------------+-------------------+
| USER()            | CURRENT_USER()    |
+-------------------+-------------------+
| alice@172.30.0.21 | alice@172.30.0.21 |
+-------------------+-------------------+

[client-a]$ mysql -h mysql -u alice -p     # 用 'alice'@'%' 的密码
ERROR 1045 (28000): Access denied for user 'alice'@'172.30.0.21' (using password: YES)

从 client-b 登录，只能匹配到 'alice'@'%'，CURRENT_USER() 也随之变成 alice@%。

MongoDB 的账号里没有主机，取而代之的是认证库（authentication database）：用户在哪个库里创建，就属于哪个库。shop 库里的 alice 和 admin 库里的 alice 是两个不同的用户：

db.getSiblingDB("shop").createUser({user: "alice", pwd: "alice-shop",
  roles: [{role: "readWrite", db: "shop"}]})
db.getSiblingDB("admin").createUser({user: "alice", pwd: "alice-admin",
  roles: [{role: "read", db: "reports"}]})

不管在哪个库创建，用户文档其实都存在 admin.system.users 里，_id 是”认证库.用户名”：

{"_id":"admin.alice","user":"alice","db":"admin","roles":[{"role":"read","db":"reports"}]}
{"_id":"shop.alice","user":"alice","db":"shop","roles":[{"role":"readWrite","db":"shop"}]}

认证库只决定这个用户在哪里验证身份，不限制它能访问哪些库。admin 库的 alice 拥有的是 reports 库的读权限。

因为身份里没有主机，同一个用户从 client-a 和 client-b 登录没有任何区别。想按来源限制，要用后面讲的 authenticationRestrictions。

authSource：最常见的 Authentication failed

先看 MongoDB 连接串的格式。驱动、mongosh 和各种工具用的都是同一种 URI：

mongodb://用户名:密码@主机1:端口1,主机2:端口2/默认库?选项1=值1&选项2=值2

以本文用到的一个连接串为例：

mongodb://alice:alice-shop@mongo1:27017,mongo2:27017,mongo3:27017/shop?replicaSet=rs0&authSource=shop

部分	本例	说明
协议	`mongodb://`	另一种是 `mongodb+srv://`，主机列表从 DNS 的 SRV 记录里取，Atlas 用的就是这种
用户名和密码	`alice:alice-shop@`	可以省略，省略就是不认证。密码里有 `@`、`:`、`/`、`?`、`#`、`%` 这类字符时，要做百分号编码，比如 `@` 写成 `%40`
主机列表	`mongo1:27017,mongo2:27017,mongo3:27017`	种子列表。驱动连上其中任意一个，就能从它那里拿到副本集的全部成员，所以不必写全，但最好多写几个，以防其中一个正好不可用
默认库	`/shop`	不指定库时操作的库，同时也是 authSource 的默认值，这一点下面细说
replicaSet	`rs0`	按副本集方式连接，驱动会自动找到主节点，主节点切换后也会跟着切
authSource	`shop`	去哪个库验证用户名和密码

其他常用的选项还有 directConnection=true（只连这一个节点，不做副本集发现，本文直连某个成员或分片时用的就是它）、tls=true、readPreference、w，以及 appName。appName 会出现在服务端日志和 currentOp 里，排查时很容易认出是哪个应用的连接。它是客户端自己上报的，驱动默认不设置，服务端也不校验，所以要在每个服务的连接串里显式写上，并且不能拿它做访问控制。

要注意，mongosh 连单个主机且没写 replicaSet 时，会自动加上 directConnection=true。如果这个主机恰好是从节点，写入会直接报错：

[client-a]$ mongosh "mongodb://alice:alice-shop@mongo1:27017/shop" \
    --eval 'print(db.getMongo()._uri); db.orders.insertOne({x: 1})'
mongodb://alice:alice-shop@mongo1:27017/shop?directConnection=true&appName=mongosh+2.3.8
MongoServerError: not primary

对比 MySQL：mysql -h mysql -u alice -p shop，或者 JDBC 的 jdbc:mysql://mysql:3306/shop，里面的库名只是连上以后默认使用的 schema，和身份验证没有关系，因为 MySQL 没有认证库这个概念。MongoDB 连接串里的库名却会影响认证。

连接时，客户端必须告诉服务端去哪个库验证身份，这就是 authSource。坑在于它的默认值：如果连接串的路径里写了库名，这个库名会被当作默认的 authSource；路径里没写，默认是 admin。

路径里写的正好是认证库，一切正常：

[client-a]$ mongosh "mongodb://alice:alice-shop@mongo1:27017,mongo2:27017,mongo3:27017/shop?replicaSet=rs0" \
    --eval 'db.orders.findOne()'
{ _id: 1, item: 'book' }

路径里不写库名，authSource 回落到 admin，而 admin 库里的 alice 是另一个密码：

[client-a]$ mongosh "mongodb://alice:alice-shop@mongo1:27017,mongo2:27017,mongo3:27017/?replicaSet=rs0" \
    --eval 'db.getSiblingDB("shop").orders.findOne()'
MongoServerError: Authentication failed.

反过来的情况更隐蔽。路径里写的是想要使用的库，它同时也成了 authSource：

[client-a]$ mongosh "mongodb://alice:alice-admin@mongo1:27017,mongo2:27017,mongo3:27017/reports?replicaSet=rs0" \
    --eval 'db.getName()'
MongoServerError: Authentication failed.

[client-a]$ mongosh "mongodb://alice:alice-admin@mongo1:27017,mongo2:27017,mongo3:27017/reports?replicaSet=rs0&authSource=admin" \
    --eval 'db.getName()'
reports

客户端拿到的永远是同一句 Authentication failed。要知道具体原因，得看服务端日志里 "msg":"Failed to authenticate" 那一行的 error 字段，上面两次失败分别是：

user=alice db=admin    error: AuthenticationFailed: SCRAM authentication failed, storedKey mismatch
user=alice db=reports  error: UserNotFound: Could not find user "alice" for db "reports"

UserNotFound 说明 authSource 不对，storedKey mismatch 说明这个库里确实有这个用户，但密码不对。我的建议是，生产环境的连接串一律显式写上 authSource，不依赖默认值。

限制登录来源

MySQL 把允许的主机直接写在账号里：

CREATE USER 'report'@'172.30.0.21' IDENTIFIED BY 'report-pw';

[client-b]$ mysql -h mysql -u report -p -e "SELECT CURRENT_USER()"
ERROR 1045 (28000): Access denied for user 'report'@'172.30.0.22' (using password: YES)

MongoDB 对应的是用户上的 authenticationRestrictions，clientSource 填允许的客户端 IP 或网段，serverAddress 可以限制客户端连的是哪个服务端地址：

db.getSiblingDB("admin").createUser({user: "report", pwd: "report-pw",
  roles: [{role: "read", db: "reports"}],
  authenticationRestrictions: [{clientSource: ["172.30.0.21"]}]})

从 client-a 登录成功，从 client-b 登录失败。客户端看到的仍然只是 Authentication failed，连 Failed to authenticate 那行日志里的 error 也只写了 Authentication failed。真正的原因在它前面的另一行日志里：

Failed to acquire user because of unmet authentication restrictions
reason: Evaluation of direct authentication restrictions failed:
        Restriction '{anyOf: [{allOf: [{"clientSource": ["172.30.0.21/32"]}]}]}' ... unmet

authenticationRestrictions 也可以设置在角色上，这样拥有该角色的用户都会继承这个限制。很多从 MySQL 过来的人不知道 MongoDB 有这个功能，以为只能靠防火墙和 bindIp 控制来源。

授权：GRANT 与角色

MySQL 8.0 引入了角色，但角色授予用户以后，默认并不生效：

CREATE ROLE 'app_read';
GRANT SELECT ON shop.* TO 'app_read';
CREATE USER 'bob'@'%' IDENTIFIED BY 'bob-pw';
GRANT 'app_read' TO 'bob'@'%';

[client-a]$ mysql -h mysql -u bob -p -e "SHOW GRANTS; SELECT CURRENT_ROLE(); SELECT * FROM shop.orders"
+-----------------------------------+
| Grants for bob@%                  |
+-----------------------------------+
| GRANT USAGE ON *.* TO `bob`@`%`   |
| GRANT `app_read`@`%` TO `bob`@`%` |
+-----------------------------------+
+----------------+
| CURRENT_ROLE() |
+----------------+
| NONE           |
+----------------+
ERROR 1142 (42000) at line 1: SELECT command denied to user 'bob'@'172.30.0.21' for table 'orders'

角色已经授予，但当前会话没有激活任何角色。激活角色分三个层面。SET ROLE 只作用于当前会话，新开的会话又回到没有角色的状态：

[client-a]$ mysql -h mysql -u bob -p -e "SET ROLE 'app_read'; SELECT CURRENT_ROLE(); SELECT * FROM shop.orders"
+----------------+
| CURRENT_ROLE() |
+----------------+
| `app_read`@`%` |
+----------------+
+----+------+
| id | item |
+----+------+
|  1 | book |
+----+------+
[client-a]$ mysql -h mysql -u bob -p -e "SELECT CURRENT_ROLE()"
+----------------+
| CURRENT_ROLE() |
+----------------+
| NONE           |
+----------------+

SET DEFAULT ROLE 'app_read' TO 'bob'@'%' 作用于这个账号以后的每次登录，对已经打开的会话不起作用。全局参数 activate_all_roles_on_login 则作用于整个实例，打开后所有账号登录时都会激活自己被授予的全部角色。另外，MySQL 一直都可以绕过角色，直接把权限授予用户，比如 GRANT INSERT ON shop.orders TO 'bob'@'%'。

MongoDB 正好相反：权限只能放在角色里，用户只能绑定角色。createUser 根本没有接收权限的字段：

> db.getSiblingDB("shop").createUser({user: "bob", pwd: "bob-pw", roles: [],
    privileges: [{resource: {db: "shop", collection: "orders"}, actions: ["find"]}]})
MongoServerError: BSON field 'createUser.privileges' is an unknown field.

所以要先建一个自定义角色，权限由资源（库、集合，或者整个集群）加一组操作（find、insert、createIndex 等）组成：

db.getSiblingDB("shop").createRole({role: "orderReader", roles: [],
  privileges: [{resource: {db: "shop", collection: "orders"}, actions: ["find"]}]})
db.getSiblingDB("shop").createUser({user: "bob", pwd: "bob-pw", roles: ["orderReader"]})

角色授予后立即生效，不需要激活。bob 能读 orders，读 customers 会被拒绝：

[client-a]$ mongosh "mongodb://bob:bob-pw@mongo1:27017,mongo2:27017,mongo3:27017/shop?replicaSet=rs0" \
    --eval 'db.customers.findOne()'
MongoServerError: not authorized on shop to execute command { find: "customers", filter: {}, limit: 1, ... }

MongoDB 也没有默认角色。roles 是 createUser 的必填字段，省略会直接报错：

> db.getSiblingDB("shop").runCommand({createUser: "nobody", pwd: "nobody-pw"})
MongoServerError: BSON field 'createUser.roles' is missing but a required field

可以显式写 roles: []。这样的用户能通过认证，但 authenticatedUserRoles 是空的，读数据、列集合，甚至改自己的密码都不行，差不多相当于 MySQL 里只有 USAGE 的新账号：

[client-a]$ mongosh "mongodb://nobody:nobody-pw@mongo1:27017,mongo2:27017,mongo3:27017/shop?replicaSet=rs0" \
    --eval 'db.runCommand({connectionStatus: 1}).authInfo'
{
  authenticatedUsers: [ { user: 'nobody', db: 'shop' } ],
  authenticatedUserRoles: []
}
[client-a]$ mongosh "mongodb://nobody:nobody-pw@mongo1:27017,mongo2:27017,mongo3:27017/shop?replicaSet=rs0" \
    --eval 'db.changeUserPassword("nobody", "new-pw")'
MongoServerError: not authorized on shop to execute command { updateUser: "nobody", pwd: "xxx", ... }

还有一个规则：建在普通库里的角色，权限只能作用于这个库；只有建在 admin 库里的角色，才能包含其他库或集群级别的权限。

MongoDB 的内置角色可以大致这样对应到 MySQL：

MongoDB 内置角色	大致相当于 MySQL
read	`GRANT SELECT ON db.*`
readWrite	`GRANT SELECT, INSERT, UPDATE, DELETE ON db.*`，外加建集合、建索引
dbAdmin	库级的 DDL、统计和 profiler，但不能读业务数据
userAdmin	在这个库里建用户、建角色、授权
dbOwner	`GRANT ALL ON db.*`，包含上面三个
readAnyDatabase、readWriteAnyDatabase 等	`ON .` 的版本
root	接近 MySQL 的 root

在把管理权限拆细这件事上，MySQL 8.0 向 MongoDB 靠近了一步。以前的管理类权限集中在一个 SUPER 上，8.0 把它拆成了一组按功能划分的动态权限，比如修改系统变量的 SYSTEM_VARIABLES_ADMIN、杀别人连接的 CONNECTION_ADMIN、管理 binlog 的 BINLOG_ADMIN，SUPER 本身被标记为废弃。动态权限只能授予在全局级别（ON *.*），存在 mysql.global_grants 表里，插件和组件也可以注册自己的动态权限：

[client-a]$ mysql -h mysql -u root -p -e "GRANT SUPER ON *.* TO 'ops'@'%'; SHOW WARNINGS"
+---------+------+----------------------------------------------+
| Level   | Code | Message                                      |
+---------+------+----------------------------------------------+
| Warning | 1287 | The SUPER privilege identifier is deprecated |
+---------+------+----------------------------------------------+
[client-a]$ mysql -h mysql -u root -p -e "GRANT BINLOG_ADMIN ON shop.* TO 'ops'@'%'"
ERROR 3619 (HY000) at line 1: Illegal privilege level specified for BINLOG_ADMIN
[client-a]$ mysql -h mysql -u root -p -e "REVOKE SUPER ON *.* FROM 'ops'@'%'; GRANT SYSTEM_VARIABLES_ADMIN, CONNECTION_ADMIN ON *.* TO 'ops'@'%'; SELECT * FROM mysql.global_grants WHERE USER = 'ops'"
+------+------+------------------------+-------------------+
| USER | HOST | PRIV                   | WITH_GRANT_OPTION |
+------+------+------------------------+-------------------+
| ops  | %    | CONNECTION_ADMIN       | N                 |
| ops  | %    | SYSTEM_VARIABLES_ADMIN | N                 |
+------+------+------------------------+-------------------+

对应到 MongoDB，就是集群资源上的 setParameter、killop、replSetConfigure 这类操作。动态和静态权限在生效时机上也不一样，下文”权限改了什么时候生效”一节有实测。

内置角色里，userAdmin 值得单独说一句。在 MySQL 里，WITH GRANT OPTION 只能把自己已有的权限转授给别人。MongoDB 的 userAdmin 可以把这个库的任意角色授予任何用户，包括它自己：

[client-a]$ mongosh "mongodb://ua:ua-pw@mongo1:27017,mongo2:27017,mongo3:27017/shop?replicaSet=rs0" \
    --eval 'db.orders.findOne()'
MongoServerError: not authorized on shop to execute command { find: "orders", ... }

[client-a]$ mongosh "mongodb://ua:ua-pw@mongo1:27017,mongo2:27017,mongo3:27017/shop?replicaSet=rs0" \
    --eval 'db.grantRolesToUser("ua", ["dbOwner"]); db.orders.findOne()'
{ _id: 1, item: 'book' }

所以拥有某个库的 userAdmin，实际上就等于拥有这个库的全部权限；admin 库上的 userAdminAnyDatabase 则相当于整个集群的 root。分配这两个角色时要按最高权限来对待。

列级权限和视图

MySQL 的权限可以细到列：

GRANT SELECT (id, name) ON shop.customers TO 'carol'@'%';

[client-a]$ mysql -h mysql -u carol -p -e "SELECT id, name FROM shop.customers"
+----+------+
| id | name |
+----+------+
|  1 | Ann  |
+----+------+
[client-a]$ mysql -h mysql -u carol -p -e "SELECT * FROM shop.customers"
ERROR 1142 (42000) at line 1: SELECT command denied to user 'carol'@'172.30.0.21' for table 'customers'
[client-a]$ mysql -h mysql -u carol -p -e "SELECT name, phone FROM shop.customers"
ERROR 1143 (42000) at line 1: SELECT command denied to user 'carol'@'172.30.0.21' for column 'phone' in table 'customers'

顺带一提，SELECT * 碰到没有权限的列，报的是表级的 1142，明确选了没有权限的列才是列级的 1143，排查时不要被误导。

MongoDB 的权限只到集合，没有字段级权限。要对某些用户隐藏字段，常见做法是建一个视图，只把视图授权出去：

db.getSiblingDB("shop").createView("customers_public", "customers", [{$project: {phone: 0, email: 0}}])
db.getSiblingDB("shop").createRole({role: "publicReader", roles: [],
  privileges: [{resource: {db: "shop", collection: "customers_public"}, actions: ["find"]}]})

用户只需要视图的 find 权限，不需要底层集合的权限：

[client-a]$ mongosh "mongodb://carol:carol-pw@mongo1:27017,mongo2:27017,mongo3:27017/shop?replicaSet=rs0" \
    --eval 'db.customers_public.find().toArray()'
[ { _id: 1, name: 'Ann' } ]
[client-a]$ mongosh "mongodb://carol:carol-pw@mongo1:27017,mongo2:27017,mongo3:27017/shop?replicaSet=rs0" \
    --eval 'db.customers.find().toArray()'
MongoServerError: not authorized on shop to execute command { find: "customers", ... }

视图是只读的，需要按字段控制写入的场景，只能在应用层处理。如果目的是防止 DBA 或者能接触到存储的人看到敏感字段，那是另一类问题，要用客户端字段级加密（CSFLE）或 Queryable Encryption，而不是权限。

权限改了什么时候生效：FLUSH PRIVILEGES

很多 MySQL 教程会在 GRANT 后面跟一句 FLUSH PRIVILEGES，其实这是多余的。GRANT、REVOKE、CREATE USER、ALTER USER 这些账号管理语句，会同时更新授权表和内存里的权限缓存，执行完立即生效：

[client-a]$ mysql -h mysql -u dave -p -e "SELECT * FROM shop.orders"
ERROR 1142 (42000) at line 1: SELECT command denied to user 'dave'@'172.30.0.21' for table 'orders'

mysql> GRANT SELECT ON shop.orders TO 'dave'@'%';

[client-a]$ mysql -h mysql -u dave -p -e "SELECT * FROM shop.orders"
+----+------+
| id | item |
+----+------+
|  1 | book |
+----+------+

FLUSH PRIVILEGES 的作用是让 MySQL 从授权表重新加载权限缓存。只有绕过账号管理语句，直接用 INSERT、UPDATE、DELETE 改了 mysql.user 这类表时才需要它。比如直接把账号的锁定标记改成 Y，在 FLUSH 之前完全不生效：

mysql> UPDATE mysql.user SET account_locked = 'Y' WHERE user = 'dave';

[client-a]$ mysql -h mysql -u dave -p -e "SELECT CURRENT_USER()"
+----------------+
| CURRENT_USER() |
+----------------+
| dave@%         |
+----------------+

mysql> FLUSH PRIVILEGES;

[client-a]$ mysql -h mysql -u dave -p -e "SELECT CURRENT_USER()"
ERROR 3118 (HY000): Access denied for user 'dave'@'172.30.0.21'. Account is locked.

另一个容易忽略的是已经打开的会话。官方文档把规则分成几类：表级和列级权限的变更，从会话的下一条语句开始生效；库级权限的变更，在下一次 USE 时生效；静态全局权限和密码的变更，对已经连上的会话不生效，要等重新连接；动态全局权限的变更则立即生效。静态全局权限是 SELECT ON *.*、PROCESS、SUPER 这类传统权限，动态全局权限是 8.0 以后按功能拆出来的 SESSION_VARIABLES_ADMIN、BACKUP_ADMIN、CONNECTION_ADMIN 等，SHOW PRIVILEGES 可以列出全部。

我按这几类各准备了一个用户。每人打开一个会话，执行一条语句，等 4 秒再执行一次，在第 2 秒时收回各自的权限：

用户	被收回的权限	同一会话里的第二次执行
dave	表级 `SELECT ON shop.orders`	被拒绝（1142）
hank	库级 `SELECT ON shop.*`，会话已经 `USE shop`	仍然成功，到下一次 `USE shop` 才被拒绝（1044）
ivy	库级 `SELECT ON shop.*`，会话没有 USE 过	被拒绝（1142）
erin	静态全局 `SELECT ON .`	仍然成功，新开的会话才被拒绝
gina	动态全局 `SESSION_VARIABLES_ADMIN`	被拒绝（1227）
jack	已激活的角色（角色里是 `SELECT ON shop.*`）	被拒绝（1142）

hank 和 ivy 的对比说明，”库级权限在下次 USE 时生效”指的是会话当前所在的库：这个库的库级权限缓存在会话里，直到切换数据库。没有切到这个库的会话，下一条语句就会按新权限检查。hank 的会话是这样的：

[client-a]$ mysql -h mysql -u hank -p -N shop -e "SELECT 'before', COUNT(*) FROM orders; SELECT SLEEP(4) INTO @x; SELECT 'after', COUNT(*) FROM orders; USE shop; SELECT 'after USE', COUNT(*) FROM orders"
before  1
after   1
ERROR 1044 (42000) at line 1: Access denied for user 'hank'@'%' to database 'shop'

erin 和 gina 的对比则是静态和动态全局权限的区别：

[client-a]$ mysql -h mysql -u erin -p -N -e "SELECT 'before', COUNT(*) FROM shop.orders; SELECT SLEEP(4) INTO @x; SELECT 'after', COUNT(*) FROM shop.orders"
before  1
after   1

[client-a]$ mysql -h mysql -u gina -p -N -e "SET SESSION sql_log_bin = 1; SELECT 'before: SET ok'; SELECT SLEEP(4) INTO @x; SET SESSION sql_log_bin = 1; SELECT 'after: SET ok'"
before: SET ok
ERROR 1227 (42000) at line 1: Access denied; you need (at least one of) the SUPER, SYSTEM_VARIABLES_ADMIN or SESSION_VARIABLES_ADMIN privilege(s) for this operation

所以收回静态全局权限，或者收回某个库的库级权限时，如果要求立即生效，还要把这个用户已有的连接 KILL 掉。

MongoDB 没有 FLUSH PRIVILEGES，用户和角色只能通过 createUser、grantRolesToUser 这类命令修改，修改后各成员的用户缓存随之失效。在副本集里，收回角色对已经打开的连接也立即生效。frank 用同一个连接读一次，等 4 秒再读，中间收回了他的 read 角色：

[client-a]$ mongosh "mongodb://frank:frank-pw@mongo1:27017,mongo2:27017,mongo3:27017/shop?replicaSet=rs0" \
    --eval 'print("before:", db.orders.countDocuments());
            sleep(4000);
            print("after:", db.orders.countDocuments())'
before: 1
MongoServerError: not authorized on shop to execute command { aggregate: "orders", ... }

MongoDB 也没有类似 SET ROLE 的会话级激活，授予用户的角色始终全部生效，所以不存在”会话级”和”全局”之分。连登录来源限制都会作用到已经打开的连接上。kim 从 client-b 连上并读了一次，这时给 kim 加上只允许 client-a 登录的限制，同一个连接的下一次读取直接变成了未认证：

[client-b]$ mongosh "mongodb://kim:kim-pw@mongo1:27017,mongo2:27017,mongo3:27017/shop?replicaSet=rs0" \
    --eval 'print("before:", db.orders.countDocuments());
            sleep(4000);
            print("after:", db.orders.countDocuments())'
before: 1
MongoServerError: Command aggregate requires authentication

第 2 秒执行的是 db.getSiblingDB("shop").updateUser("kim", {authenticationRestrictions: [{clientSource: ["172.30.0.21"]}]})。服务端日志里对应的一行是 Removed user with unmet authentication restrictions from session cache of user information，也就是把这个用户从连接上移除了。MySQL 改登录主机要用 RENAME USER，主机只在建立连接时检查，已经打开的会话不会被断开。我在一个从 client-b 打开的会话期间执行了 RENAME USER 'lee'@'%' TO 'lee'@'172.30.0.21'，这个会话仍然连着，但表级权限跟着账号一起改了名，下一条读取报 ERROR 1142；从 client-b 新开会话则直接报 ERROR 1045。两边都让这个用户读不了数据，区别在于 MongoDB 把连接变成了未认证，MySQL 的会话还在，只是没了权限。

分片集群是个例外。mongos 也会缓存用户信息，按参数 userCacheInvalidationIntervalSecs 的设置（默认 30 秒）定期检查缓存是否过期，所以通过一个 mongos 修改的权限，在其他 mongos 上最多可能晚 30 秒生效。需要立即生效时，可以在每个 mongos 上执行 db.adminCommand({invalidateUserCache: 1})，这大概是 MongoDB 里最接近 FLUSH PRIVILEGES 的东西。这一点我没有在 lab 里复现，依据的是官方文档。

第一个用户：localhost exception

MySQL 在初始化时就会处理 root 密码：mysqld --initialize 生成一个临时密码，Docker 镜像要求传入 MYSQL_ROOT_PASSWORD 或者显式选择随机、空密码。除非用 --initialize-insecure 故意建一个空密码的 root，MySQL 从第一天起就是开启认证的。

MongoDB 的做法是 localhost exception：开启了访问控制，但系统里一个用户都没有时，来自本机（localhost）的连接可以创建第一个用户。在一个开着 keyfile 认证、还没初始化的副本集上：

[client-a]$ mongosh "mongodb://mongo1:27017/?directConnection=true" --eval 'rs.initiate()'
MongoServerError: Command replSetInitiate requires authentication

[mongo1]$ mongosh --eval 'rs.initiate({_id: "rs0", members: [
    {_id: 0, host: "mongo1:27017"}, {_id: 1, host: "mongo2:27017"}, {_id: 2, host: "mongo3:27017"}]}).ok'
1

远程连接什么都做不了，本机连接可以初始化副本集，但仍然不能读写数据：

[mongo1]$ mongosh --eval 'db.getSiblingDB("shop").orders.insertOne({item: "book"})'
MongoServerError: not authorized on shop to execute command { insert: "orders", ... }

它能做的就是创建第一个用户。管理员密码事先放在环境变量 PW 里，避免出现在命令行和 shell 历史中。创建完成后，例外立即关闭，本机连接也必须认证：

[mongo1]$ mongosh admin --eval 'db.createUser({user: "admin", pwd: process.env.PW, roles: ["root"]}).ok'
1
[mongo1]$ mongosh --eval 'db.getSiblingDB("admin").createUser({user: "second", pwd: "second-pw", roles: ["root"]})'
MongoServerError: Command createUser requires authentication

实验中我还发现一个细节：把最后一个用户删掉以后，localhost exception 并不会马上重新打开，要等 mongod 重启。看起来 mongod 会记住”曾经有过用户”这件事，重启后才重新检查。如果不需要这个例外，可以用参数 enableLocalhostAuthBypass=false 把它关掉。

集群内部认证：keyfile

MySQL 的主从复制用的是一个普通账号，复制本身只需要 REPLICATION SLAVE 权限，从库用这个账号连主库拉 binlog。很多环境还会一起授予 REPLICATION CLIENT，它允许执行 SHOW BINARY LOG STATUS、SHOW BINARY LOGS 这类查看状态的语句，监控和高可用工具会用到，但不是复制必需的。复制相关的语句从 8.0.22 起陆续改成了 SOURCE、REPLICA 的叫法，8.4 里 SHOW SLAVE STATUS 这类旧语法已经直接报语法错误，但这个权限名一直没改，在 8.0 和 8.4 里写 GRANT REPLICATION REPLICA 都会报语法错误。

MongoDB 副本集成员之间不用业务账号，而是内部认证（internal authentication），常见的方式是 keyfile：所有成员持有同一个密钥文件，启动时加上 --keyFile。在服务端日志里，成员之间的认证是这样的：

"msg":"Successfully authenticated","attr":{"client":"172.30.0.13:32994","isClusterMember":true,
  "mechanism":"SCRAM-SHA-256","user":"__system","db":"local",...}

成员以内置的 __system@local 身份、用 SCRAM-SHA-256 互相认证，密码由 keyfile 派生。有两点要注意：一是开启内部认证会同时开启客户端授权，官方文档的原话是 “Enabling internal authentication also enables client authorization”，所以不需要再单独配置 security.authorization。这对 keyfile 和 x509 两种内部认证都成立，例外是过渡期的 --transitionToAuth，下文会讲到。二是 keyfile 的权限不能对其他用户可读，否则 mongod 拒绝启动。

内部认证的另一种方式是 x509 证书。x509 在 MongoDB 里有两种用途，一是客户端用证书登录，二是成员之间用证书做内部认证。前者在自建环境里很少见，后者是官方推荐的生产配置，但实际上用 keyfile 的团队仍然很多。另外要分清，开启 TLS 加密传输不等于用 x509 认证，很多集群开着 TLS，认证用的仍然是密码。

给运行中的副本集开启认证

MongoDB 默认不开启访问控制。一个没有开认证的副本集，任何能连上端口的人都能做任何事：

[client-b]$ mongosh "mongodb://mongo1:27017,mongo2:27017,mongo3:27017/?replicaSet=rs0" \
    --eval 'db.getSiblingDB("admin").runCommand({listDatabases: 1}).databases.map(d => d.name)'
[ 'admin', 'config', 'local' ]

MySQL 没有对应的问题，所以也没有对应的操作。MongoDB 给已经在跑的副本集开启认证，不停机的做法是借助 --transitionToAuth，分三步：

在认证还没开启时，先把管理员和业务用户都建好。此时用户已经存在，只是不强制。
逐个成员加上 --keyFile --transitionToAuth 重启，先重启从节点，最后对主节点 stepDown 后重启。transitionToAuth 模式下，成员之间用 keyfile 认证，客户端带不带账号都能连。
把所有客户端改成带账号连接，再逐个成员去掉 --transitionToAuth 重启，只保留 --keyFile。

为了观察对业务的影响，我在 client-a 上跑了两个持续写入的进程，每秒约 10 次插入，一个不带账号，一个用 app 账号。每个阶段的写入统计如下：

阶段	不带账号的写入	带账号的写入
未开认证	成功 50，失败 0	成功 50，失败 0
第 2 步：滚动切到 transitionToAuth	成功 534，失败 0	成功 534，失败 0
第 3 步：滚动切到 keyFile	成功 353，失败 225	成功 563，失败 0

带账号的写入全程没有一次失败，主节点 stepDown 时的切换由可重试写入（retryable writes）兜住了。不带账号的写入代表漏改的客户端，在主节点切到 keyFile 模式后开始报错：

Command insert requires authentication

为什么不能跳过第 2 步？我试着把一个从节点直接从无认证切到 --keyFile 重启，20 秒后看副本集状态：

mongo1:27017  PRIMARY  health=1
mongo2:27017  (not reachable/healthy)  health=0
mongo3:27017  SECONDARY  health=1

这个节点要求其他成员认证，而其他成员没有 keyfile，它自己去连其他成员时，对方也不认识它的认证请求。transitionToAuth 的作用就是让成员在过渡期两种连接都接受。

轮换 keyfile

keyfile 用久了需要轮换，但所有成员必须能互相认证，不能一次全换。从 MongoDB 4.2 开始，keyfile 可以写成 YAML 列表，放多个 key：

- <旧 key>
- <新 key>

轮换分两步：先逐个成员换成同时包含旧 key 和新 key 的 keyfile 并重启；全部完成后，再逐个成员换成只有新 key 的 keyfile 并重启。实验中两步做完，副本集始终健康，主节点也正常切换。

同样，如果跳过第一步，只给一个成员换上新 key，它就会和其他成员失联，状态和上一节一样是 (not reachable/healthy)。

分片集群：用户存在哪里

分片集群里有两套用户。通过 mongos 创建的用户是集群用户，存在 config server 上：

mongos> db.getSiblingDB("admin").createUser({user: "analyst", pwd: "analyst-pw",
          roles: [{role: "read", db: "sales"}]})
mongos> db.getSiblingDB("admin").system.users.find({}, {_id: 1}).toArray()
[ { _id: 'admin.admin' }, { _id: 'admin.analyst' } ]

通过 mongos 访问一切正常，但这个用户在分片上并不存在，直连分片会失败，分片的日志里是 UserNotFound：

[client-a]$ mongosh "mongodb://analyst:analyst-pw@shard1:27017/sales?authSource=admin&directConnection=true" \
    --eval 'db.orders.countDocuments()'
MongoServerError: Authentication failed.
error: UserNotFound: Could not find user "analyst" for db "admin"

更需要注意的是，分片本身一个用户都没有，所以分片上的 localhost exception 仍然是开着的。任何能登录分片所在机器的人，都可以在上面创建一个 root 用户：

[shard1]$ mongosh admin --eval 'db.createUser({user: "shardadmin", pwd: "shardadmin-pw", roles: ["root"]}).ok'
1

这个分片本地用户只存在于 shard1，mongos 不认识它，用它连 mongos 会认证失败。官方建议给每个分片都创建分片本地的管理员用户，一方面用于直连分片做维护，另一方面也把分片上的 localhost exception 关掉。

排查清单

遇到鉴权问题时，可以按报错对照：

现象	先看哪里	常见原因
MongoDB 报 Authentication failed	服务端日志 `Failed to authenticate` 的 error 字段	UserNotFound：authSource 不对；storedKey mismatch：密码不对
同上，但 error 只有 Authentication failed	日志里 `unmet authentication restrictions`	来源 IP 不在 clientSource 里
认证失败，但不知道是哪个应用	失败日志里的连接号 `connNNN`，再找同一连接的 `client metadata` 日志	从里面的 `application.name` 认出应用，前提是应用设置了 appName
MongoDB 报 not authorized on <库名> to execute command	`db.runCommand({connectionStatus: 1, showPrivileges: true})`	角色里缺这个操作或资源
MySQL 报 1142，但角色已经授予	`SELECT CURRENT_ROLE()`	角色没有激活
副本集成员 not reachable/healthy	该成员的日志	成员之间 keyfile 不一致，或认证模式不一致
日志里有大量 Checking authorization failed，操作是 atlascli 聚合	可以忽略	mongosh 连接后在后台自动执行的探测，权限有限的用户会留下这条日志

对照速查表

话题	MySQL	MongoDB
账号身份	`'user'@'host'`	用户名 + 认证库
连接时指定	只需要用户名	用户名 + authSource，默认取 URI 路径里的库，没有则为 admin
限制来源	账号的 host 部分	`authenticationRestrictions.clientSource`
授权	可以直接授予用户，也可以通过角色	只能通过角色
角色生效	需要激活：`SET ROLE`（当前会话）、`SET DEFAULT ROLE`（账号以后的登录）或 `activate_all_roles_on_login`（整个实例）	授予后立即生效
最细粒度	列	集合，字段用视图实现
权限变更生效	账号管理语句立即生效，直接改授权表要 FLUSH PRIVILEGES；静态全局权限、会话当前所在库的库级权限，对已有会话不生效	立即生效，包括已有连接；mongos 的用户缓存默认最多 30 秒
第一个管理员	初始化时设置 root 密码	localhost exception
默认是否开启认证	是	否
复制或成员之间认证	普通账号 + REPLICATION SLAVE 权限（REPLICATION CLIENT 用于查看状态，非必需）	keyfile 或 x509，以 `__system` 身份
不停机开启认证	不涉及	`--transitionToAuth` 三步走
分片集群	不涉及	mongos 用户在 config server，分片另有本地用户

用 MySQL 的经验理解 MongoDB 鉴权：账号、角色与集群内部认证 was originally published by Sherry Yin at Sherry Yin on October 11, 2026.

MongoDB Access Control for MySQL Users: Accounts, Roles and Internal Authentication

2026-10-11T00:00:00+08:00

People who move from MySQL to MongoDB rarely get stuck on CRUD. They get stuck on access control: the user exists but the connection says Authentication failed, a role is granted but the privileges aren’t what they expected, or the replica set members stop talking to each other once authentication is turned on. Most of the time the cause is a MySQL mental model applied to MongoDB, and the two differ in a few important places.

This post follows MySQL’s concepts in order and compares each one with how MongoDB does it, with the focus on the differences and the traps. All commands and output come from a local lab:

MySQL 8.4
MongoDB 8.0.4: a three-member replica set rs0, plus a minimal sharded cluster (one config server, one shard, one mongos)
Two client containers, client-a (172.30.0.21) and client-b (172.30.0.22), to show login restrictions by client IP

The MySQL examples use 8.4. If you’re still on 8.0, the main authentication differences are these: 8.0 still enables mysql_native_password by default, while 8.4 disables it, so creating an account with it fails with Plugin 'mysql_native_password' is not loaded. 8.4 also removed the default_authentication_plugin variable in favor of authentication_policy, and dropped old replication syntax such as SHOW SLAVE STATUS. Old drivers that only speak mysql_native_password can’t connect to 8.4 until you turn it on with --mysql-native-password=ON, and in 9.0 the plugin is gone.

The lab is on GitHub: mongo-mysql-auth-lab. ./lab.sh up starts it, and ./lab.sh run 00 to 11 correspond to the sections below; 10 compares MySQL 8.0 with 8.4, and 11 checks the privileges a MySQL replication user needs. Commands are shown in full. In the output, I cut long parts such as the lsid and $clusterTime in error messages and marked them with ....

Authentication and authorization

Authentication answers “who are you”, authorization answers “what are you allowed to do”. Both MySQL and MongoDB have the two steps, but they put them together differently, and many of the differences later in this post come from here.

MySQL ties them together. A connection has to authenticate first: the user name, client host and password must match an account in mysql.user before there is a session at all, and every statement after that is checked against the grant tables. There is no state where you’re connected but not yet authenticated:

[client-a]$ mysql -h mysql -e 'SELECT 1'
ERROR 1045 (28000): Access denied for user 'root'@'172.30.0.21' (using password: NO)

The only way to switch off both is --skip-grant-tables, which is meant for emergencies such as recovering a lost root password.

MongoDB keeps them apart. With access control on, an unauthenticated connection can still be opened, it just can’t do anything beyond commands like hello and ping:

[client-a]$ mongosh "mongodb://mongo1:27017/?directConnection=true" \
    --eval 'print("ping:", db.adminCommand({ping: 1}).ok);
            print("hello.me:", db.hello().me);
            db.adminCommand({listDatabases: 1})'
ping: 1
hello.me: mongo1:27017
MongoServerError: Command listDatabases requires authentication

More importantly, authorization is a separate switch, security.authorization in the configuration. It’s off by default, and it’s turned on automatically when you enable internal authentication (keyfile or x509). With authorization off, a connection that sends a user name and password still has the password checked, and a wrong one still fails with Authentication failed. Once it passes, though, privileges aren’t checked at all. Here is a standalone mongod without access control, where the user reader only has the read role on shop:

[client-a]$ mongosh "mongodb://reader:wrong-pw@noauth:27017/shop" --eval 'db.orders.countDocuments()'
MongoServerError: Authentication failed.

[client-a]$ mongosh "mongodb://reader:reader-pw@noauth:27017/shop" \
    --eval 'print(JSON.stringify(db.runCommand({connectionStatus: 1}).authInfo.authenticatedUsers));
            db.orders.insertOne({item: "pen"}).acknowledged'
[{"user":"reader","db":"shop"}]
true

reader authenticated, and even though it only has the read role, the write went through. So having users with passwords that work doesn’t mean the cluster is protected; check that authorization is actually on. In the startup options, no security entry under parsed means it’s off:

[client-a]$ mongosh "mongodb://reader:reader-pw@noauth:27017/shop" \
    --eval 'JSON.stringify(db.adminCommand({getCmdLineOpts: 1}).parsed)'
{"net":{"bindIp":"*"}}

With the two steps separated, the error messages make more sense:

Error	Step	Meaning
MySQL `ERROR 1045 Access denied for user`	Authentication	No such account, the host doesn’t match, or the password is wrong
MySQL `ERROR 1142` / `1143 command denied`	Authorization	Logged in, but no privilege on this table or column
MongoDB `Authentication failed`	Authentication	No such user in the authentication database, wrong password, or a login restriction not met
MongoDB `Command <command> requires authentication`	Not authenticated	Access control is on and this connection hasn’t authenticated
MongoDB `not authorized on <database> to execute command`	Authorization	Authenticated, but no role grants this action

What an account is: user@host vs the authentication database

In MySQL an account is a user name plus a host. 'alice'@'%' and 'alice'@'172.30.0.21' are two separate accounts that can have different passwords and privileges:

CREATE USER 'alice'@'%'           IDENTIFIED BY 'alice-any-host';
CREATE USER 'alice'@'172.30.0.21' IDENTIFIED BY 'alice-client-a';

When alice logs in from client-a, MySQL matches the more specific host first, so only that account’s password works:

[client-a]$ mysql -h mysql -u alice -p -e "SELECT USER(), CURRENT_USER()"
+-------------------+-------------------+
| USER()            | CURRENT_USER()    |
+-------------------+-------------------+
| alice@172.30.0.21 | alice@172.30.0.21 |
+-------------------+-------------------+

[client-a]$ mysql -h mysql -u alice -p     # with the password of 'alice'@'%'
ERROR 1045 (28000): Access denied for user 'alice'@'172.30.0.21' (using password: YES)

From client-b, only 'alice'@'%' matches, and CURRENT_USER() becomes alice@%.

A MongoDB user has no host. Instead it has an authentication database: the database the user was created in. alice in shop and alice in admin are two different users:

db.getSiblingDB("shop").createUser({user: "alice", pwd: "alice-shop",
  roles: [{role: "readWrite", db: "shop"}]})
db.getSiblingDB("admin").createUser({user: "alice", pwd: "alice-admin",
  roles: [{role: "read", db: "reports"}]})

Wherever they were created, all user documents are stored in admin.system.users, with an _id of “authentication database.user name”:

{"_id":"admin.alice","user":"alice","db":"admin","roles":[{"role":"read","db":"reports"}]}
{"_id":"shop.alice","user":"alice","db":"shop","roles":[{"role":"readWrite","db":"shop"}]}

The authentication database only decides where the user’s identity is checked. It doesn’t limit which databases the user can access: alice in admin has read access to reports.

Since there’s no host in the identity, the same user logs in from client-a and client-b in exactly the same way. To restrict where a user can log in from, you use authenticationRestrictions, covered below.

authSource: the most common Authentication failed

First, the format of a MongoDB connection string. Drivers, mongosh and the other tools all use the same URI:

mongodb://user:password@host1:port1,host2:port2/defaultdb?option1=value1&option2=value2

Taking one of the connection strings from this post:

mongodb://alice:alice-shop@mongo1:27017,mongo2:27017,mongo3:27017/shop?replicaSet=rs0&authSource=shop

Part	In this example	Notes
Scheme	`mongodb://`	The other one is `mongodb+srv://`, which reads the host list from DNS SRV records; Atlas uses it
User and password	`alice:alice-shop@`	Optional; leave it out and the connection doesn’t authenticate. Characters such as `@`, `:`, `/`, `?`, `#` and `%` in the password must be percent-encoded, for example `@` as `%40`
Host list	`mongo1:27017,mongo2:27017,mongo3:27017`	A seed list. Once the driver reaches any one of them it learns every member of the replica set, so you don’t have to list them all, but list a few in case one is down
Default database	`/shop`	The database used when none is specified, and also the default authSource, more on that below
replicaSet	`rs0`	Connect as a replica set: the driver finds the primary and follows it after a failover
authSource	`shop`	The database where the user name and password are checked

Other common options are directConnection=true (talk to this one node only, without replica set discovery; this post uses it to connect to a specific member or shard), tls=true, readPreference, w and appName. appName shows up in the server log and in currentOp, which makes it easy to tell which application a connection belongs to. It’s self-reported by the client: drivers don’t set one by default and the server doesn’t verify it, so set it explicitly in every service’s connection string, and don’t use it for access control.

Note that when mongosh connects to a single host without replicaSet, it adds directConnection=true on its own. If that host happens to be a secondary, writes fail:

[client-a]$ mongosh "mongodb://alice:alice-shop@mongo1:27017/shop" \
    --eval 'print(db.getMongo()._uri); db.orders.insertOne({x: 1})'
mongodb://alice:alice-shop@mongo1:27017/shop?directConnection=true&appName=mongosh+2.3.8
MongoServerError: not primary

Compare that with MySQL: in mysql -h mysql -u alice -p shop or a JDBC URL like jdbc:mysql://mysql:3306/shop, the database name is only the default schema after connecting. It has nothing to do with authentication, because MySQL has no authentication database. In a MongoDB connection string, the database name does affect authentication.

When connecting, the client has to tell the server which database to check the credentials against. That’s authSource, and the trap is its default: if the connection string has a database in the path, that database becomes the default authSource; if it doesn’t, the default is admin.

If the path names the authentication database, everything works:

[client-a]$ mongosh "mongodb://alice:alice-shop@mongo1:27017,mongo2:27017,mongo3:27017/shop?replicaSet=rs0" \
    --eval 'db.orders.findOne()'
{ _id: 1, item: 'book' }

Without a database in the path, authSource falls back to admin, where alice has a different password:

[client-a]$ mongosh "mongodb://alice:alice-shop@mongo1:27017,mongo2:27017,mongo3:27017/?replicaSet=rs0" \
    --eval 'db.getSiblingDB("shop").orders.findOne()'
MongoServerError: Authentication failed.

The opposite case is harder to spot. The path names the database you want to work in, and it quietly becomes the authSource too:

[client-a]$ mongosh "mongodb://alice:alice-admin@mongo1:27017,mongo2:27017,mongo3:27017/reports?replicaSet=rs0" \
    --eval 'db.getName()'
MongoServerError: Authentication failed.

[client-a]$ mongosh "mongodb://alice:alice-admin@mongo1:27017,mongo2:27017,mongo3:27017/reports?replicaSet=rs0&authSource=admin" \
    --eval 'db.getName()'
reports

The client always gets the same Authentication failed. To find the actual reason, look at the error field of the "msg":"Failed to authenticate" line in the server log. For the two failures above:

user=alice db=admin    error: AuthenticationFailed: SCRAM authentication failed, storedKey mismatch
user=alice db=reports  error: UserNotFound: Could not find user "alice" for db "reports"

UserNotFound means the authSource is wrong. storedKey mismatch means the user does exist in that database but the password is wrong. My advice is to always set authSource explicitly in production connection strings instead of relying on the default.

Restricting where users log in from

MySQL writes the allowed host into the account:

CREATE USER 'report'@'172.30.0.21' IDENTIFIED BY 'report-pw';

[client-b]$ mysql -h mysql -u report -p -e "SELECT CURRENT_USER()"
ERROR 1045 (28000): Access denied for user 'report'@'172.30.0.22' (using password: YES)

The MongoDB equivalent is authenticationRestrictions on the user. clientSource lists the allowed client IPs or ranges, and serverAddress can limit which server address the client connects to:

db.getSiblingDB("admin").createUser({user: "report", pwd: "report-pw",
  roles: [{role: "read", db: "reports"}],
  authenticationRestrictions: [{clientSource: ["172.30.0.21"]}]})

Logging in from client-a works and from client-b fails. The client again only sees Authentication failed, and even the error field of the Failed to authenticate log line just says Authentication failed. The real reason is in another log line just before it:

Failed to acquire user because of unmet authentication restrictions
reason: Evaluation of direct authentication restrictions failed:
        Restriction '{anyOf: [{allOf: [{"clientSource": ["172.30.0.21/32"]}]}]}' ... unmet

authenticationRestrictions can also be set on a role, and every user with that role inherits the restriction. Many people coming from MySQL don’t know MongoDB has this, and assume firewalls and bindIp are the only way to control where connections come from.

Authorization: GRANT and roles

MySQL 8.0 added roles, but a role granted to a user doesn’t take effect by default:

CREATE ROLE 'app_read';
GRANT SELECT ON shop.* TO 'app_read';
CREATE USER 'bob'@'%' IDENTIFIED BY 'bob-pw';
GRANT 'app_read' TO 'bob'@'%';

[client-a]$ mysql -h mysql -u bob -p -e "SHOW GRANTS; SELECT CURRENT_ROLE(); SELECT * FROM shop.orders"
+-----------------------------------+
| Grants for bob@%                  |
+-----------------------------------+
| GRANT USAGE ON *.* TO `bob`@`%`   |
| GRANT `app_read`@`%` TO `bob`@`%` |
+-----------------------------------+
+----------------+
| CURRENT_ROLE() |
+----------------+
| NONE           |
+----------------+
ERROR 1142 (42000) at line 1: SELECT command denied to user 'bob'@'172.30.0.21' for table 'orders'

The role is granted, but no role is active in the session. Roles can be activated at three levels. SET ROLE applies to the current session only, and a new session starts without a role again:

[client-a]$ mysql -h mysql -u bob -p -e "SET ROLE 'app_read'; SELECT CURRENT_ROLE(); SELECT * FROM shop.orders"
+----------------+
| CURRENT_ROLE() |
+----------------+
| `app_read`@`%` |
+----------------+
+----+------+
| id | item |
+----+------+
|  1 | book |
+----+------+
[client-a]$ mysql -h mysql -u bob -p -e "SELECT CURRENT_ROLE()"
+----------------+
| CURRENT_ROLE() |
+----------------+
| NONE           |
+----------------+

SET DEFAULT ROLE 'app_read' TO 'bob'@'%' applies to every future login of that account and does nothing for sessions already open. The global variable activate_all_roles_on_login applies to the whole instance: when it’s on, every account activates all of its granted roles at login. MySQL has also always let you skip roles and grant privileges straight to a user, for example GRANT INSERT ON shop.orders TO 'bob'@'%'.

MongoDB is the other way around: privileges only live in roles, and users can only be given roles. createUser has no field for privileges at all:

> db.getSiblingDB("shop").createUser({user: "bob", pwd: "bob-pw", roles: [],
    privileges: [{resource: {db: "shop", collection: "orders"}, actions: ["find"]}]})
MongoServerError: BSON field 'createUser.privileges' is an unknown field.

So you create a custom role first. A privilege is a resource (a database, a collection or the whole cluster) plus a set of actions (find, insert, createIndex and so on):

db.getSiblingDB("shop").createRole({role: "orderReader", roles: [],
  privileges: [{resource: {db: "shop", collection: "orders"}, actions: ["find"]}]})
db.getSiblingDB("shop").createUser({user: "bob", pwd: "bob-pw", roles: ["orderReader"]})

A granted role takes effect immediately; there’s nothing to activate. bob can read orders, and reading customers is refused:

[client-a]$ mongosh "mongodb://bob:bob-pw@mongo1:27017,mongo2:27017,mongo3:27017/shop?replicaSet=rs0" \
    --eval 'db.customers.findOne()'
MongoServerError: not authorized on shop to execute command { find: "customers", filter: {}, limit: 1, ... }

MongoDB has no default role either. roles is a required field of createUser, and leaving it out is an error:

> db.getSiblingDB("shop").runCommand({createUser: "nobody", pwd: "nobody-pw"})
MongoServerError: BSON field 'createUser.roles' is missing but a required field

You can pass roles: [] explicitly. Such a user can authenticate, but authenticatedUserRoles is empty, and it can’t read data, list collections or even change its own password. It’s roughly a new MySQL account with nothing but USAGE:

[client-a]$ mongosh "mongodb://nobody:nobody-pw@mongo1:27017,mongo2:27017,mongo3:27017/shop?replicaSet=rs0" \
    --eval 'db.runCommand({connectionStatus: 1}).authInfo'
{
  authenticatedUsers: [ { user: 'nobody', db: 'shop' } ],
  authenticatedUserRoles: []
}
[client-a]$ mongosh "mongodb://nobody:nobody-pw@mongo1:27017,mongo2:27017,mongo3:27017/shop?replicaSet=rs0" \
    --eval 'db.changeUserPassword("nobody", "new-pw")'
MongoServerError: not authorized on shop to execute command { updateUser: "nobody", pwd: "xxx", ... }

One more rule: a role created in a regular database can only have privileges on that database. Only roles created in admin can include privileges on other databases or on the cluster.

MongoDB’s built-in roles map to MySQL roughly like this:

MongoDB built-in role	Roughly the MySQL equivalent
read	`GRANT SELECT ON db.*`
readWrite	`GRANT SELECT, INSERT, UPDATE, DELETE ON db.*`, plus creating collections and indexes
dbAdmin	Database-level DDL, statistics and the profiler, but can’t read application data
userAdmin	Creating users and roles and granting roles in that database
dbOwner	`GRANT ALL ON db.*`; includes the three above
readAnyDatabase, readWriteAnyDatabase and so on	The `ON .` versions
root	Close to MySQL’s root

On splitting up administrative privileges, MySQL 8.0 moved a step toward MongoDB. Administrative power used to sit in a single SUPER privilege; 8.0 split it into dynamic privileges grouped by function, such as SYSTEM_VARIABLES_ADMIN for changing system variables, CONNECTION_ADMIN for killing other users’ connections and BINLOG_ADMIN for managing the binary log, and deprecated SUPER. Dynamic privileges can only be granted globally (ON *.*), they’re stored in mysql.global_grants, and plugins and components can register their own:

[client-a]$ mysql -h mysql -u root -p -e "GRANT SUPER ON *.* TO 'ops'@'%'; SHOW WARNINGS"
+---------+------+----------------------------------------------+
| Level   | Code | Message                                      |
+---------+------+----------------------------------------------+
| Warning | 1287 | The SUPER privilege identifier is deprecated |
+---------+------+----------------------------------------------+
[client-a]$ mysql -h mysql -u root -p -e "GRANT BINLOG_ADMIN ON shop.* TO 'ops'@'%'"
ERROR 3619 (HY000) at line 1: Illegal privilege level specified for BINLOG_ADMIN
[client-a]$ mysql -h mysql -u root -p -e "REVOKE SUPER ON *.* FROM 'ops'@'%'; GRANT SYSTEM_VARIABLES_ADMIN, CONNECTION_ADMIN ON *.* TO 'ops'@'%'; SELECT * FROM mysql.global_grants WHERE USER = 'ops'"
+------+------+------------------------+-------------------+
| USER | HOST | PRIV                   | WITH_GRANT_OPTION |
+------+------+------------------------+-------------------+
| ops  | %    | CONNECTION_ADMIN       | N                 |
| ops  | %    | SYSTEM_VARIABLES_ADMIN | N                 |
+------+------+------------------------+-------------------+

The MongoDB counterparts are actions on the cluster resource, such as setParameter, killop and replSetConfigure. Dynamic and static privileges also differ in when changes take effect; the section on privilege changes below has the test.

Among the built-in roles, userAdmin deserves a note of its own. In MySQL, WITH GRANT OPTION only lets you pass on privileges you already have. MongoDB’s userAdmin can grant any role in its database to any user, including itself:

[client-a]$ mongosh "mongodb://ua:ua-pw@mongo1:27017,mongo2:27017,mongo3:27017/shop?replicaSet=rs0" \
    --eval 'db.orders.findOne()'
MongoServerError: not authorized on shop to execute command { find: "orders", ... }

[client-a]$ mongosh "mongodb://ua:ua-pw@mongo1:27017,mongo2:27017,mongo3:27017/shop?replicaSet=rs0" \
    --eval 'db.grantRolesToUser("ua", ["dbOwner"]); db.orders.findOne()'
{ _id: 1, item: 'book' }

So userAdmin on a database effectively means full control of that database, and userAdminAnyDatabase in admin is effectively root on the whole cluster. Treat both as the highest level of privilege when you hand them out.

Column privileges and views

MySQL privileges go down to the column:

GRANT SELECT (id, name) ON shop.customers TO 'carol'@'%';

[client-a]$ mysql -h mysql -u carol -p -e "SELECT id, name FROM shop.customers"
+----+------+
| id | name |
+----+------+
|  1 | Ann  |
+----+------+
[client-a]$ mysql -h mysql -u carol -p -e "SELECT * FROM shop.customers"
ERROR 1142 (42000) at line 1: SELECT command denied to user 'carol'@'172.30.0.21' for table 'customers'
[client-a]$ mysql -h mysql -u carol -p -e "SELECT name, phone FROM shop.customers"
ERROR 1143 (42000) at line 1: SELECT command denied to user 'carol'@'172.30.0.21' for column 'phone' in table 'customers'

Note that SELECT * touching a column you can’t read gives the table-level error 1142; only naming that column explicitly gives the column-level 1143. Don’t let it mislead you when troubleshooting.

MongoDB privileges stop at the collection; there are no field-level privileges. The usual way to hide fields from some users is to create a view and grant access only to the view:

db.getSiblingDB("shop").createView("customers_public", "customers", [{$project: {phone: 0, email: 0}}])
db.getSiblingDB("shop").createRole({role: "publicReader", roles: [],
  privileges: [{resource: {db: "shop", collection: "customers_public"}, actions: ["find"]}]})

The user only needs find on the view, not on the underlying collection:

[client-a]$ mongosh "mongodb://carol:carol-pw@mongo1:27017,mongo2:27017,mongo3:27017/shop?replicaSet=rs0" \
    --eval 'db.customers_public.find().toArray()'
[ { _id: 1, name: 'Ann' } ]
[client-a]$ mongosh "mongodb://carol:carol-pw@mongo1:27017,mongo2:27017,mongo3:27017/shop?replicaSet=rs0" \
    --eval 'db.customers.find().toArray()'
MongoServerError: not authorized on shop to execute command { find: "customers", ... }

Views are read-only, so controlling writes field by field has to happen in the application. If the goal is to keep sensitive fields away from DBAs or anyone with access to the storage, that’s a different problem, solved with Client-Side Field Level Encryption (CSFLE) or Queryable Encryption rather than privileges.

When privilege changes take effect: FLUSH PRIVILEGES

Many MySQL tutorials follow every GRANT with FLUSH PRIVILEGES. It isn’t needed. Account management statements such as GRANT, REVOKE, CREATE USER and ALTER USER update both the grant tables and the in-memory privilege cache, so they take effect at once:

[client-a]$ mysql -h mysql -u dave -p -e "SELECT * FROM shop.orders"
ERROR 1142 (42000) at line 1: SELECT command denied to user 'dave'@'172.30.0.21' for table 'orders'

mysql> GRANT SELECT ON shop.orders TO 'dave'@'%';

[client-a]$ mysql -h mysql -u dave -p -e "SELECT * FROM shop.orders"
+----+------+
| id | item |
+----+------+
|  1 | book |
+----+------+

FLUSH PRIVILEGES makes MySQL reload the privilege cache from the grant tables. You only need it when you bypass account management statements and change tables like mysql.user directly with INSERT, UPDATE or DELETE. For example, setting the account lock flag directly has no effect until the flush:

mysql> UPDATE mysql.user SET account_locked = 'Y' WHERE user = 'dave';

[client-a]$ mysql -h mysql -u dave -p -e "SELECT CURRENT_USER()"
+----------------+
| CURRENT_USER() |
+----------------+
| dave@%         |
+----------------+

mysql> FLUSH PRIVILEGES;

[client-a]$ mysql -h mysql -u dave -p -e "SELECT CURRENT_USER()"
ERROR 3118 (HY000): Access denied for user 'dave'@'172.30.0.21'. Account is locked.

Sessions that are already open are easy to forget. The MySQL documentation sorts the rules into a few cases: table and column privilege changes take effect with the session’s next statement; database privilege changes take effect at the next USE; static global privilege and password changes don’t affect connected sessions until they reconnect; and dynamic global privilege changes take effect immediately. Static global privileges are the traditional ones such as SELECT ON *.*, PROCESS and SUPER; dynamic global privileges are the function-specific ones added in 8.0, such as SESSION_VARIABLES_ADMIN, BACKUP_ADMIN and CONNECTION_ADMIN. SHOW PRIVILEGES lists them all.

I set up one user for each case. Each opened a session, ran a statement, waited 4 seconds and ran it again, and at the 2-second mark I revoked their privilege:

User	Revoked privilege	Second run in the same session
dave	Table level, `SELECT ON shop.orders`	Refused (1142)
hank	Database level, `SELECT ON shop.*`, session had run `USE shop`	Still works; refused only at the next `USE shop` (1044)
ivy	Database level, `SELECT ON shop.*`, session never ran USE	Refused (1142)
erin	Static global, `SELECT ON .`	Still works; only new sessions are refused
gina	Dynamic global, `SESSION_VARIABLES_ADMIN`	Refused (1227)
jack	An active role (holding `SELECT ON shop.*`)	Refused (1142)

Comparing hank and ivy shows what “database privileges take effect at the next USE” means: it’s about the session’s current database. That database’s privileges are cached in the session until it switches databases. A session that hasn’t switched to the database is checked against the new privileges on its next statement. Here is hank’s session:

[client-a]$ mysql -h mysql -u hank -p -N shop -e "SELECT 'before', COUNT(*) FROM orders; SELECT SLEEP(4) INTO @x; SELECT 'after', COUNT(*) FROM orders; USE shop; SELECT 'after USE', COUNT(*) FROM orders"
before  1
after   1
ERROR 1044 (42000) at line 1: Access denied for user 'hank'@'%' to database 'shop'

erin and gina show the difference between static and dynamic global privileges:

[client-a]$ mysql -h mysql -u erin -p -N -e "SELECT 'before', COUNT(*) FROM shop.orders; SELECT SLEEP(4) INTO @x; SELECT 'after', COUNT(*) FROM shop.orders"
before  1
after   1

[client-a]$ mysql -h mysql -u gina -p -N -e "SET SESSION sql_log_bin = 1; SELECT 'before: SET ok'; SELECT SLEEP(4) INTO @x; SET SESSION sql_log_bin = 1; SELECT 'after: SET ok'"
before: SET ok
ERROR 1227 (42000) at line 1: Access denied; you need (at least one of) the SUPER, SYSTEM_VARIABLES_ADMIN or SESSION_VARIABLES_ADMIN privilege(s) for this operation

So when you revoke a static global privilege or a database-level privilege and need it to take effect now, also kill that user’s existing connections.

MongoDB has no FLUSH PRIVILEGES. Users and roles can only be changed with commands such as createUser and grantRolesToUser, and the change invalidates the user cache on every member. In a replica set, revoking a role also takes effect on connections that are already open. frank reads once, waits 4 seconds and reads again on the same connection, and his read role is revoked in between:

[client-a]$ mongosh "mongodb://frank:frank-pw@mongo1:27017,mongo2:27017,mongo3:27017/shop?replicaSet=rs0" \
    --eval 'print("before:", db.orders.countDocuments());
            sleep(4000);
            print("after:", db.orders.countDocuments())'
before: 1
MongoServerError: not authorized on shop to execute command { aggregate: "orders", ... }

MongoDB also has nothing like SET ROLE: every role granted to a user is always in effect, so there’s no session versus global distinction. Even login restrictions apply to open connections. kim connected from client-b and read once; then I restricted kim to client-a, and the next read on the same connection was no longer authenticated:

[client-b]$ mongosh "mongodb://kim:kim-pw@mongo1:27017,mongo2:27017,mongo3:27017/shop?replicaSet=rs0" \
    --eval 'print("before:", db.orders.countDocuments());
            sleep(4000);
            print("after:", db.orders.countDocuments())'
before: 1
MongoServerError: Command aggregate requires authentication

The command at the 2-second mark was db.getSiblingDB("shop").updateUser("kim", {authenticationRestrictions: [{clientSource: ["172.30.0.21"]}]}). The matching server log line is Removed user with unmet authentication restrictions from session cache of user information: the user was removed from the connection. In MySQL you change an account’s host with RENAME USER, and the host is only checked when a connection is made, so open sessions stay connected. I ran RENAME USER 'lee'@'%' TO 'lee'@'172.30.0.21' while lee had a session open from client-b. The session stayed connected, but the table privileges were renamed along with the account, so the next read failed with ERROR 1142, and a new session from client-b got ERROR 1045. Either way the user can no longer read the data; the difference is that MongoDB turns the connection back into an unauthenticated one, while the MySQL session survives without its privileges.

Sharded clusters are the exception. mongos caches user information too, and checks whether the cache is stale at the interval set by userCacheInvalidationIntervalSecs (30 seconds by default). A privilege changed through one mongos can therefore take up to 30 seconds to apply on the others. If you need it immediately, run db.adminCommand({invalidateUserCache: 1}) on each mongos, which is probably the closest thing MongoDB has to FLUSH PRIVILEGES. I didn’t reproduce this one in the lab; it’s based on the documentation.

The first user: the localhost exception

MySQL deals with the root password at initialization: mysqld --initialize generates a temporary password, and the Docker image requires MYSQL_ROOT_PASSWORD or an explicit choice of a random or empty password. Unless you deliberately create a root with an empty password using --initialize-insecure, MySQL has authentication on from day one.

MongoDB uses the localhost exception instead: when access control is on but no user exists yet, a connection from localhost may create the first user. On a replica set with keyfile authentication that hasn’t been initiated:

[client-a]$ mongosh "mongodb://mongo1:27017/?directConnection=true" --eval 'rs.initiate()'
MongoServerError: Command replSetInitiate requires authentication

[mongo1]$ mongosh --eval 'rs.initiate({_id: "rs0", members: [
    {_id: 0, host: "mongo1:27017"}, {_id: 1, host: "mongo2:27017"}, {_id: 2, host: "mongo3:27017"}]}).ok'
1

A remote connection can’t do anything. A local connection can initiate the replica set, but still can’t read or write data:

[mongo1]$ mongosh --eval 'db.getSiblingDB("shop").orders.insertOne({item: "book"})'
MongoServerError: not authorized on shop to execute command { insert: "orders", ... }

What it can do is create the first user. The admin password is put in the PW environment variable beforehand, so it doesn’t end up on the command line or in the shell history. As soon as the user exists, the exception closes and local connections have to authenticate too:

[mongo1]$ mongosh admin --eval 'db.createUser({user: "admin", pwd: process.env.PW, roles: ["root"]}).ok'
1
[mongo1]$ mongosh --eval 'db.getSiblingDB("admin").createUser({user: "second", pwd: "second-pw", roles: ["root"]})'
MongoServerError: Command createUser requires authentication

One more detail from the lab: after dropping the last user, the localhost exception doesn’t reopen until mongod restarts. It looks like mongod remembers that users have existed and only checks again after a restart. If you don’t want the exception at all, turn it off with the parameter enableLocalhostAuthBypass=false.

Internal authentication: keyfile

MySQL replication uses an ordinary account. Replication itself only needs the REPLICATION SLAVE privilege; the replica uses that account to connect to the source and read the binary log. Many setups also grant REPLICATION CLIENT, which allows status statements such as SHOW BINARY LOG STATUS and SHOW BINARY LOGS. Monitoring and high-availability tools use it, but replication doesn’t need it. The replication statements were renamed to SOURCE and REPLICA terms starting with 8.0.22, and in 8.4 old syntax like SHOW SLAVE STATUS is a syntax error. The privilege name never changed, though: GRANT REPLICATION REPLICA is a syntax error in both 8.0 and 8.4.

MongoDB replica set members don’t use application accounts. They use internal authentication, most commonly a keyfile: every member holds the same key file and starts with --keyFile. In the server log, authentication between members looks like this:

"msg":"Successfully authenticated","attr":{"client":"172.30.0.13:32994","isClusterMember":true,
  "mechanism":"SCRAM-SHA-256","user":"__system","db":"local",...}

Members authenticate to each other as the built-in __system@local user with SCRAM-SHA-256, using a password derived from the keyfile. Two things to note. First, enabling internal authentication also enables client authorization; in the documentation’s words, “Enabling internal authentication also enables client authorization”, so you don’t need to set security.authorization separately. That holds for both keyfile and x509 internal authentication, the exception being the transitional --transitionToAuth, covered below. Second, the keyfile must not be readable by other users, or mongod refuses to start.

The other form of internal authentication is x509 certificates. x509 has two uses in MongoDB: clients logging in with certificates, and members authenticating to each other with certificates. The first is rare in self-managed deployments. The second is the configuration MongoDB recommends for production, although plenty of teams still use keyfiles. Also keep in mind that TLS encryption in transit is not x509 authentication; many clusters run TLS and still authenticate with passwords.

Enabling authentication on a running replica set

MongoDB doesn’t enable access control by default. On a replica set without it, anyone who can reach the port can do anything:

[client-b]$ mongosh "mongodb://mongo1:27017,mongo2:27017,mongo3:27017/?replicaSet=rs0" \
    --eval 'db.getSiblingDB("admin").runCommand({listDatabases: 1}).databases.map(d => d.name)'
[ 'admin', 'config', 'local' ]

MySQL doesn’t have this problem, so it has no equivalent procedure. To turn on authentication for a running MongoDB replica set without downtime, you use --transitionToAuth in three steps:

While authentication is still off, create the admin and application users. They exist but aren’t enforced yet.
Restart the members one at a time with --keyFile --transitionToAuth, secondaries first, then step down the primary and restart it. In transitionToAuth mode, members authenticate to each other with the keyfile, and clients can connect with or without credentials.
Switch every client to connect with credentials, then restart the members one at a time again with only --keyFile.

To see the effect on applications, I ran two writers on client-a, each inserting about 10 documents a second, one without credentials and one as the app user. Writes per phase:

Phase	Writer without credentials	Writer with credentials
Authentication off	50 ok, 0 failed	50 ok, 0 failed
Step 2: rolling to transitionToAuth	534 ok, 0 failed	534 ok, 0 failed
Step 3: rolling to keyFile	353 ok, 225 failed	563 ok, 0 failed

The writer with credentials never failed once; retryable writes covered the primary stepdowns. The writer without credentials stands for a client someone forgot to update, and it started failing once the primary switched to keyFile mode:

Command insert requires authentication

Why not skip step 2? I restarted one secondary straight from no authentication to --keyFile and checked the replica set 20 seconds later:

mongo1:27017  PRIMARY  health=1
mongo2:27017  (not reachable/healthy)  health=0
mongo3:27017  SECONDARY  health=1

That member requires the others to authenticate, and they have no keyfile; when it connects to them, they don’t recognize its authentication either. transitionToAuth exists so that members accept both kinds of connection during the transition.

Rotating the keyfile

Keyfiles need rotating eventually, but every member must still be able to authenticate to the others, so you can’t swap them all at once. Since MongoDB 4.2, a keyfile can be a YAML list holding several keys:

- <old key>
- <new key>

Rotation takes two rounds: first restart the members one at a time with a keyfile holding both the old and the new key; when all are done, restart them one at a time again with a keyfile holding only the new key. In the lab, the replica set stayed healthy through both rounds and the primary switched over normally.

As before, if you skip the first round and give one member only the new key, it loses contact with the others and shows up as (not reachable/healthy), just like in the previous section.

Sharded clusters: where users live

A sharded cluster has two sets of users. Users created through mongos are cluster users, stored on the config servers:

mongos> db.getSiblingDB("admin").createUser({user: "analyst", pwd: "analyst-pw",
          roles: [{role: "read", db: "sales"}]})
mongos> db.getSiblingDB("admin").system.users.find({}, {_id: 1}).toArray()
[ { _id: 'admin.admin' }, { _id: 'admin.analyst' } ]

Access through mongos works, but the user doesn’t exist on the shards, so a direct connection to a shard fails, and the shard’s log says UserNotFound:

[client-a]$ mongosh "mongodb://analyst:analyst-pw@shard1:27017/sales?authSource=admin&directConnection=true" \
    --eval 'db.orders.countDocuments()'
MongoServerError: Authentication failed.
error: UserNotFound: Could not find user "analyst" for db "admin"

The bigger issue is that the shard has no users of its own, so its localhost exception is still open. Anyone who can log in to the shard’s host can create a root user there:

[shard1]$ mongosh admin --eval 'db.createUser({user: "shardadmin", pwd: "shardadmin-pw", roles: ["root"]}).ok'
1

This shard-local user exists only on shard1; mongos doesn’t know it, and authenticating to mongos with it fails. MongoDB recommends creating a shard-local admin user on every shard, both for maintenance over direct connections and to close the localhost exception on the shard.

Troubleshooting checklist

When you hit an access control problem, find the symptom in this table:

Symptom	Look at	Common cause
MongoDB says Authentication failed	The error field of the `Failed to authenticate` server log line	UserNotFound: wrong authSource; storedKey mismatch: wrong password
Same, but the error only says Authentication failed	A log line with `unmet authentication restrictions`	The client IP isn’t in clientSource
Authentication failed, but you don’t know which application	The connection id `connNNN` on the failure line, then the `client metadata` line of the same connection	`application.name` identifies the application, provided it sets appName
MongoDB says not authorized on to execute command	`db.runCommand({connectionStatus: 1, showPrivileges: true})`	The role lacks this action or resource
MySQL error 1142 even though the role is granted	`SELECT CURRENT_ROLE()`	The role isn’t active
A replica set member is not reachable/healthy	That member’s log	Keyfiles don’t match between members, or members are in different auth modes
Lots of Checking authorization failed in the log for an atlascli aggregation	Safe to ignore	mongosh runs this probe in the background after connecting; users with limited privileges leave this line

Quick reference

Topic	MySQL	MongoDB
Account identity	`'user'@'host'`	User name + authentication database
What the client specifies	User name only	User name + authSource; defaults to the database in the URI path, or admin if there is none
Restricting the client source	The host part of the account	`authenticationRestrictions.clientSource`
Granting privileges	Directly to users, or through roles	Through roles only
Roles taking effect	Must be activated: `SET ROLE` (current session), `SET DEFAULT ROLE` (future logins of the account) or `activate_all_roles_on_login` (whole instance)	Immediately when granted
Finest granularity	Column	Collection; fields via views
Privilege changes take effect	Account management statements apply immediately; direct edits to the grant tables need FLUSH PRIVILEGES; static global privileges and the current database’s privileges don’t affect open sessions	Immediately, including open connections; mongos user cache up to 30 seconds by default
First administrator	Root password set at initialization	The localhost exception
Authentication on by default	Yes	No
Replication or member authentication	Ordinary account + REPLICATION SLAVE (REPLICATION CLIENT for status, not required)	Keyfile or x509, as `__system`
Enabling authentication without downtime	Not applicable	`--transitionToAuth` in three steps
Sharded clusters	Not applicable	mongos users live on the config servers; shards have their own local users

MongoDB Access Control for MySQL Users: Accounts, Roles and Internal Authentication was originally published by Sherry Yin at Sherry Yin on October 11, 2026.

新加坡从零考驾照全记录（2026）：BTT、PDL 到路考

2026-10-09T00:00:00+08:00

我在新加坡从零开始考驾照，报的是 CDC（ComfortDelGro Driving Centre）的 Class 3A，也就是自动挡小汽车。从报名到通过 BTT、拿到 PDL，一共用了 11 天。

这篇先记录到 PDL 为止，后面的实践课、FTT 和路考，考完再更新。如果你有中国驾照、打算直接转换，文末有一条容易被忽略的规定。

规定部分我都对照了新加坡警察部队（Traffic Police）的官方页面，链接放在文末。规定会调整，办理前请以官网为准。

先选路线：从零考还是转换

	从零考	中国驾照转换
适合谁	没有驾照，或者手上的驾照不符合转换条件	持有有效的中国驾照
要考什么	BTT、FTT、路考	只需要 BTT
主要麻烦	步骤多，每一步都要抢预约	材料要齐，非英文驾照要找新加坡注册的翻译公司翻译

外国人在新加坡居住不满 12 个月，可以直接用有效的外国驾照开车。住满 12 个月后，就需要换成新加坡驾照。

名词解释

名词	全称	说明
TP	Traffic Police	新加坡警察部队下的交通警察，负责驾照考试和发证
SPF	Singapore Police Force	新加坡警察部队，PDL 在它的 e-Services 网站申请
CDC、BBDC、SSDC	ComfortDelGro Driving Centre、Bukit Batok Driving Centre、Singapore Safety Driving Centre	新加坡的三家驾校，可以学车，也设有理论考试考点
BTT	Basic Theory Test	基础理论考试，学车的第一关；用中国驾照转换也要考
FTT	Final Theory Test	终极理论考试，考 Class 3 或 3A 要先通过它才能路考
IE	Internal Evaluation	CDC 的内部模拟考，第一次学车的驾校学员要先通过它才能约 BTT
Induction Programme		CDC 的入学介绍课，讲整个学车流程，不能取消或改期
PDL	Provisional Driving Licence	学习驾照，电子版，有效期 2 年，有了它才能上实践课
实践课	Practical Lesson	跟教练上路练车，CDC 每节 100 分钟
路考	Practical Driving Test	实际驾驶考试，由 TP 指派的考官主考
Class 3、3A		Class 3 是手动挡；Class 3A 是自动挡，只能开自动挡车
Standard、One Team、Elite Team		CDC 的三种班型，区别见”选哪种班”
PRM	Premium fee	One Team 和 Elite Team 每节实践课的附加费
Stored Value		CDC 的预付储值账户，各项费用从这里扣
VPC	Vehicular Pre-operative Check	实践课阶段的一个项目，CDC 价目表里单独收费
LDCS	Learner Driver Competency Screening	实践课阶段的一个项目，CDC 价目表里单独收费
Booking Portal		CDC 的网上预约系统，网页版和手机 App 都能用
My Learning Journey		CDC App 里的学车进度，显示已经完成和下一步要做的环节
Slot		可预约的时段，视力测试、IE、BTT、实践课都要抢
Singpass		新加坡政府的数字身份，申请 PDL 时用来登录
GST	Goods and Services Tax	新加坡的消费税，目前是 9%

从零考的整体流程

报名驾校
视力测试（包括色盲测试）
Induction Programme 和 IE（Internal Evaluation，驾校内部模拟考），驾校学员才有
BTT（Basic Theory Test，基础理论考试）
拍照，申请 PDL（Provisional Driving Licence，学习驾照）
上实践课
FTT（Final Theory Test，终极理论考试）
路考
拿到驾照

我的时间线：11 天从报名到 PDL

日期	进度	距报名
9 月 28 日	在 CDC 网上报名 Class 3A	第 0 天
10 月 2 日	现场视力测试，通过	第 4 天
10 月 3 日	Zoom 参加 Induction Programme，然后现场考 IE，通过	第 5 天
10 月 7 日	预约实践课：10 月 3 日起每天刷 slot，第 4 天才约到，最早的空位在 11 月 24 日	第 9 天
10 月 9 日	现场考 BTT，通过；在 CDC 拍照，在线申请 PDL	第 11 天

能这么快，主要是每一步都抢到了比较早的 slot。

选哪种班

CDC 的 Class 3A 在 Ubi 有三种班型，区别在于训练车和教练是否固定，每节实践课的加价也不同（含 GST）：

班型	每节实践课加价	训练车	教练
Standard	0	不固定	不固定
One Team	13.08 新元	固定	最多 6 位教练组成的团队
Elite Team	16.35 新元	固定	最多 4 位资深教练组成的团队

报名费方面，Standard 是 185.30 新元，One Team 和 Elite Team 是 201.65 新元，都包含 1 年会籍，另外收 10 新元可退押金。我报的是 Elite Team 的电动车（EV）课程，EV 课程只在 Ubi 开。

预约工具：CDC 网页版和 App

在 CDC 学车，预约、取消、充值、查收据都在它的网上预约系统（Booking Portal）里完成。网页版首页有十个入口，我最常用的是这三个：

My Bookings：查看已经约好的课和考试。
Lesson Booking：约课，比如 Induction Programme、理论课和实践课。
Test Bookings：约 IE、BTT 这类考试。

CDC 网上预约系统首页，红框是最常用的三个入口。

其他几个也会用到：Top-Up Store Value 给储值账户充值，Cancel Booking 取消预约，Account Statement / Receipt 查流水和收据（下面的费用明细就是从这里整理的），Online Learning 是在线学习资源。

建议把 CDC 的手机 App 也装上，随时可以查看 My Learning Journey，也就是自己的学车进度到了哪一步、下一步要做什么，预约也很方便。

报名和视力测试

在 CDC 网上报名后，会收到一封邮件，按邮件里的方式预约视力测试。邮件可能被归到垃圾邮件，记得检查。

视力测试包括视力和色觉两部分。按照新加坡《Road Traffic (Motor Vehicles, Driving Licences) Rules》的规定，要求是：

在 25 米的距离，能读出黑底白字、和车牌一样大小和排列的 6 个字母与数字。平时戴眼镜的，可以戴着眼镜看。
在 25 米的距离，能分辨出红色、橙色和绿色。

视力测试的 slot 很紧张，要靠刷。我的观察是，每天大约早上 9 点半开放预约，晚上 6 点半左右关闭，开放期间可以不停刷新。我的做法是先约一个能约到的时间，然后继续刷，看到更早的就改期。

要注意顺序：视力测试通过后才能约 IE，IE 通过后才能约 BTT。这三项都要到现场，都要抢 slot，所以每过一项就马上约下一项。

Induction Programme 和 IE（内部模拟考）

IE（Internal Evaluation）是 CDC 的内部模拟考，形式和 Traffic Police 的正式考试一样。第一次学车的驾校学员必须到现场通过 IE，才能预约正式的 BTT。

Induction Programme 是 CDC 的入学介绍，讲后面整个学车流程，我约的是 Zoom 的场次。CDC 官网写明，Induction Programme 不可以取消，也不可以改期。

我把两项约在了同一天：先在线上参加 Induction Programme，然后去考 IE。结果 IE 开考时 Induction Programme 还没结束，我只好提前退出。最后系统还是算我完成了，算是有惊无险。不建议照着这样约，两项之间最好留出足够的时间。

BTT 正式考试

IE 和正式 BTT 的题型一样，都是 50 道选择题，答对 45 道及格，限时 50 分钟。BTT 可以选英文、中文、马来文或淡米尔文作答，我用的是英文。

两套系统的界面风格差别很大。IE 是 CDC 自己的系统，界面明显更现代；正式 BTT 用的系统界面还很旧，像早年的 VB 程序，第一次用要花点时间适应。BTT 的系统最近在做升级，希望升级后能好用一些。

考试时一定要看清楚再点按钮。有学员做完一道题就不小心点了 End Test，考试直接结束，好不容易约到的 slot 也就作废了。

备考资料我只用了两样：

SPF 官方的 Basic Theory of Driving 手册：中文版和英文版对照着看。用英文考的话，看中文版帮助理解，看英文版熟悉题目里的说法。
刷题 App “BTT, FTT, RTT - Theory Test SG”：模拟题很多，适合反复练。但我发现模拟测试里有个别题目的答案是错的，遇到和手册说法不一样的，以官方手册为准。

CDC 有免费的在线理论课，用 Zoom 上，老师会讲考试重点和容易出错的题目，推荐参加。

IE 通过、约好正式 BTT 之后，我就开始刷实践课的 slot。连着刷了 4 天都没有合适的空位，到 10 月 7 日才约到，而最早的空位已经排到 11 月 24 日，要等 7 周左右。所以 IE 一过就可以开始约实践课，不用等 BTT 考完。

实践课的空位没有固定的开放时间。CDC 官网写明，Ubi 的各个班型都”没有固定的放课日期”，要自己经常上网查看。我约到的两节都是早上刷到的，听说空位是不定期释放的。另外，CDC 在 2024 年 7 月公布过预约上限：Ubi 的自动挡学员每月最多约 5 节实践课，手动挡 8 节。

拍照和申请 PDL

BTT 通过后，我直接在 CDC 拍了证件照，工作人员会把数码照片发到邮箱。拍照和考 BTT 安排在同一趟，就不用为了拍照再跑一次。

然后到新加坡警察部队的 e-Services 网站，用 Singpass 登录申请 PDL。费用是 25 新元，缴费后就通过了。PDL 是电子驾照，有效期 2 年，可以续期，驾驶教练能在线查到。没有有效的 PDL 去上实践课是违法的。

费用明细

到拿 PDL 为止，我实际花的钱如下（新元，含 9% GST）：

项目	费用
CDC 报名费（Elite Team，含 1 年会籍）	201.65
押金（可退）	10.00
视力测试	1.96
Induction Programme	23.98
IE（Internal Evaluation）	7.63
BTT 考试费	7.20
BTT 行政费	2.94
CDC 在线理论课（2 节）	0
数码证件照	7.09
PDL 申请费（交给 SPF）	25.00
合计	287.45

不算报名费和押金，从视力测试到拿 PDL 一共是 75.80 新元。

后面的费用按 CDC 官网目前的价格：

实践课：每节 100 分钟。工作日 8:30–10:10 和 12:45–4:15 是非高峰时段，79.57 新元；其他时段和周末是 89.38 新元。再加上班型的附加费。我约的两节都在高峰时段，Elite Team 每节是 89.38 + 16.35 = 105.73 新元。
FTT：考试费 7.20 + 行政费 2.94 = 10.14 新元。
路考：考试费 40.00 + 行政费 2.94 + 考试用车和热身费 317.19 = 360.13 新元。
会籍：1 年到期后，续 6 个月是 65.40 新元。

CDC 的费用都从预付的储值账户里扣，要先充值才能预约。

我的建议

报名后就装上 CDC 的手机 App，随时看学车进度和预约。
报名 CDC 后先上它的免费在线理论课，再去考 IE 和 BTT。
视力测试、IE、BTT 都要到现场，而且只有前一项通过才能约下一项。每过一项就立刻约下一项。
约 slot 时先占一个，再继续刷更早的，有就改期。
考 IE 和 BTT 时看清楚按钮，别误点 End Test，否则考试直接结束，slot 作废。
去考 BTT 那天顺便在 CDC 拍照，省一趟。
IE 通过后就开始约实践课，空位可能要排一个多月。空位不定期释放，没有固定时间，要经常刷，我是早上刷到的。
时间允许的话约非高峰时段的实践课，每节便宜将近 10 新元。
Induction Programme 不能取消或改期，别和 IE 约得太近。

接下来：实践课、FTT 和路考

这部分正在进行中，考完后会更新到这篇文章里。

从 App 的 My Learning Journey 可以看到后面还有这些环节：Traffic Police 模拟器培训（3 个模块）；实践课的 3 个模块，其中包括 VPC、LDCS、ATC 等项目；最后是路考前的热身课（Warm Up Lesson）和 Traffic Police 路考。另外，App 里 BTT 的有效期显示为 Lifetime，通过后长期有效。

如果你有中国驾照

用中国驾照转换只需要通过 BTT，不用考 FTT 和路考。流程和材料网上已经有很多攻略，办理时以 Traffic Police 的官方材料清单为准。

有一条规定容易被忽略：如果你的中国驾照是在拿到第一张新加坡准证（WP、EP、学生准证等）或 PR 之后才考的，需要额外证明在中国住满 1 年，而且实际在中国居住至少 6 个月。可以用护照出入境记录、在中国的工作证明、学历证明或纳税记录来证明。来新加坡之前就考好的驾照，不需要这项证明。

另外，非英文驾照要由新加坡注册的翻译公司出具英文翻译件，自己翻译的不收。

常见问题

BTT 可以用中文考吗？

可以，BTT 提供英文、中文、马来文和淡米尔文。

PDL 有效期多久？

2 年，可以在网上续期。

官方链接

费用按 2026 年 10 月 CDC 官网价格。最后更新：2026 年 10 月 9 日。

新加坡从零考驾照全记录（2026）：BTT、PDL 到路考 was originally published by Sherry Yin at Sherry Yin on October 09, 2026.

Getting a Singapore Driving Licence from Scratch (2026): BTT to PDL in 11 Days

2026-10-09T00:00:00+08:00

I’m learning to drive in Singapore from scratch. I enrolled for Class 3A (automatic) at CDC, the ComfortDelGro Driving Centre, and it took 11 days to go from enrolment to passing the Basic Theory Test and getting my Provisional Driving Licence.

This post covers everything up to the PDL. I’ll add the practical lessons, the Final Theory Test and the driving test once I’ve done them. If you already hold a licence from another country and want to convert it instead, there’s a short section on that near the end.

The rules here are checked against the Traffic Police pages, linked at the bottom. Rules change, so check the official pages before you apply.

Two routes: from scratch or converting a foreign licence

	From scratch	Converting a foreign licence
Who it’s for	No licence yet, or a foreign licence that can’t be converted	A valid licence issued in another country
Tests	BTT, FTT and the practical driving test	BTT only
What takes effort	Many steps, and every one needs a booking slot	Getting the documents right, including an official translation if the licence isn’t in English

Foreigners who have lived in Singapore for less than 12 months can drive on a valid foreign licence. After 12 months, you need a Singapore licence.

Glossary

Term	Stands for	What it means
TP	Traffic Police	The part of the Singapore Police Force that runs driving tests and issues licences
SPF	Singapore Police Force	You apply for the PDL on its e-Services website
CDC, BBDC, SSDC	ComfortDelGro Driving Centre, Bukit Batok Driving Centre, Singapore Safety Driving Centre	Singapore’s three driving centres. You can learn to drive at any of them, and they also host theory tests
BTT	Basic Theory Test	The first theory test. Everyone takes it, including people converting a foreign licence
FTT	Final Theory Test	The second theory test. For Class 3 or 3A you must pass it before the practical driving test
IE	Internal Evaluation	CDC’s in-house mock test. First-time learners at CDC must pass it before they can book the BTT
Induction Programme		CDC’s introduction session covering the whole learning journey. It can’t be cancelled or rescheduled
PDL	Provisional Driving Licence	The learner’s licence. It’s digital, valid for 2 years, and required for practical lessons
Practical lesson		Driving on the road with an instructor. A CDC lesson is 100 minutes
Practical driving test		The on-road test, conducted by a TP-appointed examiner
Class 3, 3A		Class 3 is manual; Class 3A is automatic and only covers cars without a clutch pedal
Standard, One Team, Elite Team		CDC’s coaching schemes, compared below
PRM	Premium fee	The per-lesson surcharge for One Team and Elite Team
Stored Value		Your prepaid CDC account. All CDC fees are deducted from it
VPC	Vehicular Pre-operative Check	An item in the practical lesson stage, charged separately in CDC’s fee list
LDCS	Learner Driver Competency Screening	An item in the practical lesson stage, charged separately in CDC’s fee list
Slot		A bookable time. Eyesight tests, IEs, BTTs and lessons all need one, and they go fast
Booking Portal		CDC’s online booking system, on the web and in the mobile app
My Learning Journey		Your progress in the CDC app: what you’ve completed and what comes next
Singpass		Singapore’s government digital identity, used to log in when applying for the PDL
GST	Goods and Services Tax	Singapore’s consumption tax, currently 9%

The process from scratch

Enrol at a driving centre
Eyesight and colour-vision test
Induction Programme and IE (for CDC learners)
BTT
Photo and PDL application
Practical lessons
FTT
Practical driving test
Get your licence

My timeline: 11 days from enrolment to PDL

Date	Step	Day
28 Sep	Enrolled online at CDC for Class 3A	0
2 Oct	Eyesight test on site, passed	4
3 Oct	Induction Programme on Zoom, then the IE on site, passed	5
7 Oct	Booked practical lessons after refreshing for slots every day since 3 Oct; the earliest slot was 24 Nov	9
9 Oct	BTT on site, passed; photo taken at CDC; applied for the PDL online	11

The main reason it went this fast is that I kept getting early slots for each step.

Choosing a coaching scheme

At Ubi, CDC offers three coaching schemes for Class 3A. They differ in whether you keep the same car and instructors, and in the surcharge per practical lesson (GST included):

Scheme	Extra per lesson	Training car	Instructors
Standard	S$0	Not fixed	Not fixed
One Team	S$13.08	Same car	A team of up to 6 instructors
Elite Team	S$16.35	Same car	A team of up to 4 senior instructors

The enrolment fee is S$185.30 for Standard and S$201.65 for One Team and Elite Team, each including a one-year membership, plus a refundable S$10 deposit. I chose Elite Team on an electric vehicle (EV). The EV course is only available at Ubi.

Booking tools: CDC web portal and app

Everything at CDC, from booking and cancelling to topping up and finding receipts, happens in its online Booking Portal. The web home page has ten sections. These are the three I use most:

My Bookings: lessons and tests you’ve already booked.
Lesson Booking: book the Induction Programme, theory lessons and practical lessons.
Test Bookings: book tests such as the IE and the BTT.

The CDC Booking Portal home page. The three sections outlined in red are the ones I use most.

The others come up too: Top-Up Store Value to add money to your account, Cancel Booking, Account Statement / Receipt for your transaction history (the costs below come from there), and Online Learning.

Install the CDC mobile app as well. It shows My Learning Journey, so you can see at a glance which step you’re on and what’s next, and booking from it is easy.

Enrolment and the eyesight test

After enrolling online, you’ll get an email telling you how to book the eyesight test. Check your spam folder if you don’t see it.

The test covers eyesight and colour vision. Under Singapore’s Road Traffic (Motor Vehicles, Driving Licences) Rules, you must be able to:

read, at 25 metres, a series of six white letters and numbers on a black background, the same size and layout as a vehicle number plate. If you wear glasses, you can wear them for the test.
tell red, amber and green apart at 25 metres.

Eyesight test slots are hard to get. From what I saw, bookings open at around 9:30 am each day and close at around 6:30 pm, and you can keep refreshing while it’s open. I booked whatever I could get first, then kept checking and rescheduled whenever an earlier slot appeared.

The order matters: you can only book the IE after passing the eyesight test, and only book the BTT after passing the IE. All three are on site and need a slot, so book the next one as soon as you pass.

Induction Programme and IE (Internal Evaluation)

The IE is CDC’s in-house mock test, in the same format as the Traffic Police test. First-time learners at CDC have to pass it in person before they can book the real BTT.

The Induction Programme introduces CDC’s learning journey. I took it on Zoom. CDC says it can’t be cancelled or rescheduled.

I booked both on the same day: the Induction Programme online first, then the IE on site. The IE started before the Induction Programme ended, so I had to leave early. It still counted as completed, but I wouldn’t plan it that way. Leave a proper gap between the two.

The BTT

The IE and the BTT have the same format: 50 multiple-choice questions, 45 correct answers to pass, 50 minutes. The BTT is available in English, Chinese, Malay and Tamil. I took it in English.

The two systems look very different. The IE runs on CDC’s own system, which feels modern. The BTT system looks like an old Visual Basic application and takes a moment to get used to. It’s being upgraded, so hopefully it’ll be friendlier soon.

Read the buttons carefully during the test. Some learners have clicked End Test after a single question, which ends the test on the spot and wastes a slot that was hard to get.

I used two things to prepare:

The official Basic Theory of Driving handbook from SPF. I read the English and Chinese editions side by side.
The app “BTT, FTT, RTT - Theory Test SG”. It has plenty of practice questions, but a few answers in its mock tests are wrong. When it disagrees with the handbook, trust the handbook.

CDC also runs free online theory classes on Zoom, where the instructor goes through the key topics and the questions people often get wrong. They’re worth joining.

After passing the IE and booking the BTT, I started refreshing for practical lesson slots. It took four days of checking before I got any, on 7 October, and the earliest slot was already 24 November, about seven weeks away. So you can start booking lessons as soon as you pass the IE; there’s no need to wait for the BTT.

There’s no fixed time when lesson slots open. CDC’s website says there is “no session opening date” for any of its Ubi schemes and asks learners to check online regularly. I got both of my slots by refreshing in the morning, and I’ve heard that slots are released at irregular times. CDC also announced a booking limit in July 2024: at Ubi, automatic-car learners can book at most 5 practical lessons per month, and manual-car learners 8.

Photo and PDL

Right after passing the BTT, I had my photo taken at CDC, and the staff emailed me the digital photo. Doing this on the day of the BTT saves a separate trip.

Then I applied for the PDL on the SPF e-Services website, logging in with Singpass. It costs S$25, and it was approved as soon as I paid. The PDL is digital, valid for two years and renewable, and instructors can verify it online. Taking practical lessons without a valid PDL is an offence.

Costs

Here’s what I actually paid up to the PDL (S$, CDC fees include 9% GST):

Item	Cost
CDC enrolment fee (Elite Team, one-year membership)	201.65
Deposit (refundable)	10.00
Eyesight test	1.96
Induction Programme	23.98
IE (Internal Evaluation)	7.63
BTT test fee	7.20
BTT administrative fee	2.94
CDC online theory classes (2 sessions)	0
Digital photo	7.09
PDL application fee (paid to SPF)	25.00
Total	287.45

Excluding the enrolment fee and deposit, everything from the eyesight test to the PDL came to S$75.80.

Later costs, at CDC’s current prices:

Practical lessons: 100 minutes each. Weekdays 8:30–10:10 am and 12:45–4:15 pm are off-peak at S$79.57; other times and weekends are S$89.38. Add your scheme’s surcharge. Both of my lessons are at peak times, so each costs 89.38 + 16.35 = S$105.73 on Elite Team.
FTT: S$7.20 test fee + S$2.94 administrative fee = S$10.14.
Practical driving test: S$40.00 test fee + S$2.94 administrative fee + S$317.19 for the test vehicle and warm-up = S$360.13.
Membership: after the first year, a six-month extension costs S$65.40.

All CDC fees come out of your prepaid Stored Value account, so you need to top up before you can book.

Tips

Install the CDC app as soon as you enrol.
Join CDC’s free online theory classes before you sit the IE and the BTT.
The eyesight test, IE and BTT are all on site, and each unlocks the next. Book the next one the moment you pass.
Grab any slot first, then keep checking for earlier ones and reschedule.
Read the buttons carefully in the IE and the BTT. Clicking End Test by mistake ends the test and wastes the slot.
Get your photo taken at CDC on the day of your BTT.
Start booking practical lessons once you pass the IE. Slots can be more than a month out, and they’re released at no fixed time, so keep checking. I got mine in the morning.
If your schedule allows, book off-peak lessons. Each one is almost S$10 cheaper.
The Induction Programme can’t be cancelled or rescheduled, so don’t book it too close to your IE.

What’s next: lessons, FTT and the driving test

I’m in this stage now. I’ll update this post after the test.

My Learning Journey in the app shows what’s still ahead: Traffic Police simulator training (3 modules); three practical lesson modules, which include items such as VPC, LDCS and ATC; and finally a warm-up lesson and the Traffic Police practical test. The app also lists the BTT’s expiry as Lifetime, so a pass doesn’t run out.

Converting a foreign licence

If you hold a valid licence from another country, you only need to pass the BTT in Singapore; the FTT and the practical test aren’t required. Foreigners need to convert after living in Singapore for 12 months. New citizens and PRs have three months from the change in status. Work Permit and S Pass holders receive a Class 3C licence after converting.

The Traffic Police publishes document checklists, one for China licences and one for all other foreign licences. The main items are a recent passport-size photo, your identity documents and passes, and your licence. A few points are easy to miss:

If your licence isn’t in English, you need an official translation from a translation company registered in Singapore. Your own translation won’t be accepted.
If your licence doesn’t show the date it was first issued, bring an extract of your driving record.
If you got your foreign licence after receiving your first Singapore pass (WP, EP, student pass and so on) or PR, or you’re a Singapore citizen applying with a foreign licence, you must also prove you lived in the issuing country for one year, with at least six months of physical residence. Passports, employment documents, education certificates or tax documents can be used. This is to show you didn’t go abroad just to get a licence.
The Traffic Police page lists extra documents for some countries. For example, Malaysian licences need a letter from the Road Transport Department showing the licence classes, and Indian licences need an endorsement letter of validity from the Indian embassy.

To apply, fill in the online pre-assessment form, book an appointment at Traffic Police Headquarters yourself, and bring the originals and photocopies of your documents. The counter decides on the spot whether you’re eligible, and the Traffic Police may ask for more documents.

FAQ

Can I take the BTT in a language other than English?

Yes. It’s available in English, Chinese, Malay and Tamil.

How long is the PDL valid?

Two years. You can renew it online.

How long does it take to get from enrolment to the PDL?

It took me 11 days, but it depends on how quickly you get slots for the eyesight test, the IE and the BTT.

Official links

Costs are based on CDC’s published prices in October 2026. Last updated: 9 October 2026.

Getting a Singapore Driving Licence from Scratch (2026): BTT to PDL in 11 Days was originally published by Sherry Yin at Sherry Yin on October 09, 2026.

Building a Global Multi-Write MongoDB Cluster with Zone Sharding

2026-09-26T00:00:00+08:00

Say an application has users in China and in North America. With a single replica set, the primary lives in one of the two regions, so users in the other region pay a trans-Pacific round trip, usually well over 100 ms, on every write. If the write concern waits for replication, it’s more.

MongoDB has no multi-master replication. Each replica set has exactly one primary, which is also why MongoDB never has to resolve write conflicts. Zone sharding gets close to multi-region writes without giving that up. This post covers how to set it up, and what happens when a region goes down, which matters more in operations.

How it works

A sharded cluster is made of several replica sets (shards), each with its own primary. Zone sharding pins ranges of the shard key to specific shards. Combine that with where you place replica set members:

Put a region field such as homeRegion at the front of the shard key.
Assign each shard to a zone, and map each region’s key range to that zone.
Keep each shard’s primary in its own region, with secondaries in the other region.

A Chinese player’s documents then live on a shard whose primary is in China, and an American player’s documents on a shard whose primary is in North America. Both regions write locally into the same cluster, but any single document still has exactly one primary. Most of the limitations later in this post follow from that.

Topology

Each region writes to the shard whose primary is local. Secondaries in the other region hold a copy for disaster recovery and local reads. Every mongos gets its routing metadata from the config server replica set.

A minimal two-region layout:

Component	China	North America
Application servers	✔	✔
`mongos` routers	2 or more	2 or more
shard0 (zone `APAC`)	Primary + Secondary	Secondary
shard1 (zone `NA`)	Secondary	Primary + Secondary
Config server replica set	Primary + Secondary	Secondary (see config servers)

Applications connect to the mongos routers in their own region. Run at least two per region so a router restart doesn’t take the region’s traffic down. mongos routes by shard key using metadata it caches from the config servers, so a write for a "CN" document goes to shard0’s primary in China and never leaves the region.

Setup

There are three parts: the data model, shard zones, and zone ranges. The commands below use the current names; sh.addShardTag() and sh.addTagRange() are the older names for the same operations. Run them through mongos as a user with the clusterManager role.

1. Data model and shard key

The example is player profiles in a game, stored in game.players. The application sets homeRegion when the player signs up, based on the region they pick:

{
  _id: ObjectId("..."),
  homeRegion: "CN",            // set by the application at sign-up
  playerId: 100234,
  nickname: "lotus",
  level: 42,
  lastLoginAt: ISODate("2026-09-26T08:00:00Z"),
  inventory: [ /* ... */ ]
}

Zone ranges can only be defined on a prefix of the shard key, so homeRegion has to come first. The second field should have high cardinality and appear in your queries, so chunks can still be split within a zone:

{ homeRegion: 1, playerId: 1 }

A document inserted without homeRegion won’t end up where you expect. Schema validation is a cheap way to prevent that:

db.createCollection("players", {
  validator: {
    $jsonSchema: {
      required: ["homeRegion", "playerId"],
      properties: { homeRegion: { enum: ["CN", "US"] } }
    }
  }
})

2. Add shards to zones

sh.addShardToZone("shard0", "APAC")
sh.addShardToZone("shard1", "NA")

A shard can be in several zones, and a zone can have several shards. One shard per zone is enough to start with. Add shards to a zone when it runs out of capacity.

3. Define zone ranges

The lower bound is inclusive and the upper bound is exclusive. With MinKey and MaxKey on playerId, each range covers a whole region:

sh.updateZoneKeyRange(
  "game.players",
  { homeRegion: "CN", playerId: MinKey },
  { homeRegion: "CN", playerId: MaxKey },
  "APAC"
)

sh.updateZoneKeyRange(
  "game.players",
  { homeRegion: "US", playerId: MinKey },
  { homeRegion: "US", playerId: MaxKey },
  "NA"
)

If you can, create the zones and ranges before you shard the collection. For an empty or non-existent collection, shardCollection then creates chunks on the zone boundaries and puts them on the right shards straight away, so the balancer doesn’t have to move data across the Pacific later:

sh.enableSharding("game")
sh.shardCollection("game.players", { homeRegion: 1, playerId: 1 })

For an existing collection, the balancer migrates every chunk that sits in the wrong zone. Across regions that’s a lot of traffic, so run it inside a balancing window.

4. Keep each primary in its home region

Zones decide which shard owns a document. Replica set configuration decides where that shard’s primary runs. Set priorities and tags so the primary is elected in the home region:

// on shard0 (home: China)
cfg = rs.conf()
cfg.members[0].priority = 2;   cfg.members[0].tags = { region: "cn" }
cfg.members[1].priority = 1;   cfg.members[1].tags = { region: "cn" }
cfg.members[2].priority = 0.5; cfg.members[2].tags = { region: "us" }
rs.reconfig(cfg)

Don’t set the remote member’s priority to 0. A priority-0 member can never become primary, so it couldn’t take over if the home region went down.

5. Check routing

sh.status()                                   // zones, ranges, chunk placement
db.players.getShardDistribution()             // data per shard
db.players.find({ homeRegion: "CN", playerId: 100234 }).explain()
                                              // should target only shard0

Reads

Reads of a region’s own data go to the local primary, which covers most traffic.

For data that belongs to the other region, there are two options. Reading from the primary is consistent, but it crosses the ocean. Reading from a local secondary is fast but may be stale: use readPreference: "nearest" (or "secondaryPreferred") with a region tag such as readPreferenceTags=region:cn. Set maxStalenessSeconds (at least 90) so the driver skips secondaries that have fallen too far behind.

A query without homeRegion can’t be targeted, so mongos sends it to every shard in both regions. A support tool that looks up player 100234 by playerId alone runs a global query every time. Keep homeRegion in the filter of every frequent query.

Write concern and region failure

Zone sharding makes writes local. Whether those writes survive the loss of a region depends on where the write concern’s majority comes from. You can’t have local writes, zero data loss, and automatic failover at the same time. There are two common layouts.

Layout A: 2 members at home, 1 remote

w: "majority" needs 2 of 3 acknowledgements, and both can come from the home region, so writes stay local.

If the home region goes down, the remote member is 1 of 3 and can’t be elected. Someone has to force a reconfiguration by hand, and whatever hadn’t replicated across the ocean yet is lost. That replication lag is your RPO.

Layout B: 5 members across 3 regions (2 + 2 + 1)

A majority is 3, so at least one acknowledgement has to come from outside the home region. Every majority write pays at least one cross-region round trip. Putting the fifth member in a region close to home keeps that cost small.

If the home region goes down, the other 3 members still form a majority and elect a new primary on their own. No acknowledged majority write is lost.

	Layout A (2 + 1)	Layout B (2 + 2 + 1)
Write latency with `w: "majority"`	local	at least one cross-region round trip
Home region lost: automatic failover	✘ manual forced reconfig	✔
Home region lost: acknowledged writes lost	possible (replication lag)	none
Cost	3 nodes per shard	5 nodes per shard, 3 regions

A common split is layout A for most data, accepting a small RPO, and layout B for data where losing a write isn’t acceptable. Either is fine, as long as your runbook matches the layout you actually deployed.

Config servers

The config server replica set stores chunk ownership, zones, and ranges. If it loses its primary, reads and writes keep working with the routing table mongos already has, but chunk migrations, splits, and shardCollection stop until a primary is back.

With two regions, one of them has to hold the config servers’ majority. In the diagram that’s China, with two of the three members, so losing China freezes metadata changes. That’s usually acceptable during an incident. A small third region fixes it for the config servers, and it’s also where layout B’s fifth member lives.

Failure scenarios

Scenario	What happens	What to plan
Cross-region link down	Each region keeps writing its own shard locally. Replication to remote secondaries lags, and remote reads get staler.	Alert on replication lag, and watch oplog window size so secondaries can catch up without a full resync.
One member in the home region down	The other members still form a majority, and writes continue.	Normal replica set operations.
Whole home region down (layout A)	That zone’s data is read-only elsewhere, and it has no primary.	A runbook for forced reconfiguration, with the RPO it implies written down in advance.
Whole home region down (layout B)	A new primary is elected in another region, and writes continue with higher latency.	Capacity in the surviving regions to absorb the extra load.
Player travels temporarily	Requests route from the local `mongos` to the home shard’s primary across the ocean: slower, but correct.	Nothing. Travel is rare, so accept the latency.
Player moves to another region	Their documents stay pinned to the old zone until the shard key changes.	Update `homeRegion` (see below).

Players who travel

Most region changes are trips. A North American player visiting China keeps homeRegion: "US". Their app connects to the mongos in China, which sends every read and write to shard1’s primary in North America. Each request crosses the Pacific, so it’s slower, but it’s correct and no data moves.

This doesn’t happen often, so I’d accept the latency rather than build anything for it. Reads that can tolerate some staleness can use readPreference: "nearest". Writes still go to the home primary.

Players who move

Only a permanent move is worth moving data for. Changing a shard key value moves the document to another shard. MongoDB allows it under a few conditions: the update runs through mongos, as a retryable write or inside a transaction, with an equality match on the full shard key, one document at a time (updateOne, not updateMany):

session.withTransaction(() => {
  players.updateOne(
    { homeRegion: "CN", playerId: 100234 },
    { $set: { homeRegion: "US" } }
  )
})

If the player also has documents in other zoned collections, like match history or friend lists, it becomes a small migration job. Decide whether old data needs to move at all, or only new data.

Limitations

There’s no global uniqueness. On a sharded collection, a unique index has to be prefixed by the shard key, so you can make an email unique per region but not worldwide. A global registry needs a separate design, for example a small unzoned collection or an external service.
A transaction that touches both zones is a distributed transaction across the Pacific. Keep transactions within one region.
Chunks outside every zone range can end up on any shard. Make sure every homeRegion value is covered by a range.
Remote secondaries are full copies of the other region’s data. If the reason for pinning data is data residency rather than latency, this layout breaks it, and DR members would have to stay inside the allowed jurisdiction.
Zone sharding doesn’t apply to time series collections. The balancer always spreads them evenly across all shards.
Dropping a collection also drops its zone ranges. Recreate them, ideally before sharding the collection again.

Alternative: independent clusters with two-way sync

The other common approach skips the shared cluster. Each region runs its own independent MongoDB deployment, a replica set or a sharded cluster, and a tool such as MongoShake replicates changes in both directions.

Two MongoShake instances, one per direction. Component names follow MongoShake's architecture overview.

MongoShake replicates in one direction, so two-way sync means two instances: one for China → North America and one for North America → China. Each instance fetches the source oplog with its Collector. For a sharded source, it connects to every shard, preferably reading from secondaries. It filters namespaces with whitelists and blacklists, and replays the changes into the target mongos through the Direct tunnel. It saves a checkpoint, by default in the source database, so it can resume after a restart, and it exposes metrics through a REST API and Prometheus.

Every region can write every document and holds a full copy of the data.

Pros

The regions are isolated from each other. They share no config servers, elections, or balancer, so an outage or network partition in one region never blocks writes in the other.
Every document can be read and written locally in every region. There’s no region field in the shard key, no zone ranges, no relocation, and no queries broadcast across regions.
Each side can run a different topology, version, or hardware, and filters can keep some data out of a region.
It’s easy to add to an existing single-region deployment: stand up a second cluster and start syncing, the same way many migrations are done.

Cons

Conflicts. If a document is updated in both regions within the sync lag, each side applies the other’s change after its own, and the two copies can end up permanently different. MongoShake doesn’t resolve conflicts, so the application has to make sure each document is only written in one region. That’s the same rule zone sharding enforces, except here nothing enforces it.
Duplicate keys. Two players registering the same email in both regions at once both succeed locally. Replaying one of them then hits a duplicate key error, and the sync either stalls or drops the change, depending on configuration.
It’s asynchronous only. There’s no cross-region write concern, so a region failure loses whatever was inside the sync lag, and cross-region reads are eventually consistent.
Loops. Each instance has to tell local writes from replicated ones, or changes bounce back and forth. MongoShake does this with a gid, but its README says gid is only available in Alibaba Cloud’s internal version, because it needs MongoDB kernel changes. The open-source version can’t filter loops that way, and the FAQ describes workarounds.
Transactions and DDL. Multi-document transactions can be replayed as individual writes, so the target briefly sees half a transaction. MongoShake only syncs DDL when the source is a replica set.
More to operate. The sync instances need their own high availability, monitoring, upgrades, and checkpoint handling. If one falls behind past the oplog window, you have to do a full resync. You also need a regular way to check the two sides for drift.
Every region stores all the data: double the storage, and the same data residency problem as remote secondaries.
No vendor support. MongoShake is a third-party tool, and MongoDB’s own Cluster-to-Cluster Sync (mongosync) only does one-way replication.

Comparison

	Zone sharding (one cluster)	Independent clusters + two-way sync
Who can write a document	only its home shard’s primary	any region
Write conflicts	impossible by design	possible; the application must avoid them
Cross-region durability	tunable with write concern (layout A / B)	async only; RPO = sync lag
Region failure	depends on member placement	other region unaffected
Global unique indexes	not possible (shard key prefix rule)	not possible; cross-region duplicates break sync
Data per region	own zone + remote replicas	full copy
Operational burden	sharding, balancer, config servers	two clusters plus a sync pipeline
Support	built into MongoDB	third-party tools

Zone sharding enforces single ownership in the database, so you can’t break it by accident. Two-way sync isolates the regions better and is more flexible, but it moves conflict avoidance and consistency checks into the application and into operations. If the data already has a natural home region, I’d choose zone sharding. Two-way sync makes more sense when regional isolation is the top priority and the application can guarantee that each document is only written in one region.

When zone sharding fits

It works well when data has a natural home region, most traffic stays in that region, and you can accept either a small RPO (layout A) or slower writes (layout B) when a region fails.

It works badly for documents written from several regions at once, like a global counter or a shared document. Every write still goes to one primary, so for most users that document is remote no matter how you zone it. Global uniqueness or cross-region transactions on the hot path are also a bad fit.

On Atlas, Global Clusters provide the same setup as a managed feature. It’s still zone sharding underneath, and the same write concern trade-offs apply.

References

Building a Global Multi-Write MongoDB Cluster with Zone Sharding was originally published by Sherry Yin at Sherry Yin on September 26, 2026.

用 Prometheus 指标给分布式数据库做容量规划：以 TiDB 为例

2024-10-10T00:00:00+08:00

“下个季度要扩多少台？”做数据库的团队几乎都被问过这个问题，通常是在大促或者新业务上线前几周。常见的做法是先拍一个数，再乘个系数“保险一点”，结果要么当天不够用，要么一年里大部分时间都在浪费机器。

之前我和一位同事写过一个小工具 tidbcloud-metrics，用 Prometheus 里现成的监控数据来回答这个问题。超大规模的 TiDB Cloud 集群每次大促前要做容量评估、决定扩多少节点，我们都靠它来算。代码和当时的环境绑得比较紧，但方法本身是通用的。这篇文章用虚构的数据把这套方法过一遍：采哪些指标，怎么汇总，留多少余量，以及这个简单模型在哪些地方不准。

例子用的是 TiDB，换成其他分布式数据库，或者任何按节点暴露指标的无状态服务，思路都一样。

核心公式

对每个组件的每种资源，需要的节点数是：

所需节点数 = (单节点用量 × 当前节点数 × 业务增长倍数) / (单节点容量 × 目标水位)

单节点用量：观察窗口内每个节点对这种资源的用量汇总值，比如 P99。
单节点用量 × 当前节点数：大致等于集群当前的总需求。
业务增长倍数：预计流量会涨多少，1.5 就是涨 50%。
单节点容量：一个节点能提供多少，比如 32 核或 2 TiB 磁盘。
目标水位：你愿意让节点跑到多满。除以 0.5 意味着希望节点利用率不超过 50%。

每个组件的每种资源都算一遍，需要节点最多的那种资源就是瓶颈，它决定这个组件最终要多少节点。

第一步：选观察窗口

数字取决于你查的是哪段时间，所以这段时间里必须包含你要规划的那种负载。

至少覆盖一个完整的业务周期，一般是一周，这样工作日和周末的规律都在里面。
确认窗口里有已知的高峰：月底跑批、上一次大促、每天的报表任务。
如果是为某个特定活动做规划，就去查最接近的一次历史活动，再用增长倍数放大。
注意查询步长。窗口很长、步长又大，短时间的尖峰会被平均掉。我当时查两天数据用 60 秒步长，查一天用 30 秒。

第二步：按组件和资源采集用量

TiDB 集群主要有四个组件。每个节点上都跑着 node_exporter，所以 CPU、内存、磁盘 IOPS、磁盘带宽、网络带宽这些主机指标，四个组件都有，也都要算一遍。不过各组件用磁盘和网络的方式不一样，最先耗尽的资源也不一样：

组件	磁盘和网络用在哪	通常先碰到的上限
TiDB（SQL 层）	磁盘：日志，大查询内存不够时排序和 hash join 落盘。网络：接收客户端请求，从 TiKV、TiFlash 拉数据，返回结果	CPU、内存、网络带宽
TiKV（行存）	磁盘：数据、Raft 日志、compaction。网络：Raft 复制到其他副本，响应 TiDB 的读写	CPU、存储、磁盘 IOPS、磁盘带宽
TiFlash（列存）	磁盘：列存数据和后台合并。网络：从 TiKV 同步数据，MPP 计算时节点之间交换数据	CPU、内存、存储、磁盘带宽
PD（调度和授时）	磁盘：etcd 存集群元数据，对 fsync 延迟敏感。网络：心跳、TSO 请求	CPU、内存、磁盘延迟

磁盘 IOPS、磁盘带宽和网络带宽是三个不同的上限，要分开算：

磁盘 IOPS 是每秒读写的次数。小而随机的 IO 先碰到它，比如 TiKV 的点查，RocksDB 读不在缓存里的数据块。
磁盘带宽是每秒读写的字节数。大块的顺序 IO 先碰到它，比如 compaction、大范围扫描、TiFlash 读列存文件、备份。
网络带宽是网卡每秒收发的字节数。TiKV 每次写入，leader 都要把日志发给另外两个副本，而且是跨可用区的。TiDB 从 TiKV 拉数据、把结果返回给客户端，还有备份和数据均衡，也都走网络。

同一块盘可能 IOPS 只用了 30%，带宽已经到了 80%，合在一起看就会漏掉真正的瓶颈。

CPU、内存和存储用 TiDB 自己暴露的指标，磁盘和网络用 node_exporter 的主机指标：

# 每个 TiDB 实例使用的 CPU 核数
rate(process_cpu_seconds_total{component="tidb"}[2m])

# 每个 TiDB 实例的常驻内存
process_resident_memory_bytes{component="tidb"}

# 每个 TiKV store 已用存储
sum(tikv_store_size_bytes{type="used"}) by (instance)

# 每个 TiFlash store 已用存储
sum(tiflash_system_current_metric_StoreSizeUsed) by (instance)

# 磁盘 IOPS（读 + 写）
sum(rate(node_disk_reads_completed_total{device="nvme1n1"}[1m])
  + rate(node_disk_writes_completed_total{device="nvme1n1"}[1m])) by (instance)

# 磁盘带宽，字节/秒（读 + 写）
sum(rate(node_disk_read_bytes_total{device="nvme1n1"}[1m])
  + rate(node_disk_written_bytes_total{device="nvme1n1"}[1m])) by (instance)

# 网络带宽，字节/秒，入和出分开看
sum(rate(node_network_receive_bytes_total{device="eth0"}[1m])) by (instance)
sum(rate(node_network_transmit_bytes_total{device="eth0"}[1m])) by (instance)

nvme1n1 和 eth0 要换成实际的数据盘和网卡名。

单节点容量能从指标拿就从指标拿，比如 tikv_store_size_bytes{type="capacity"}、node_memory_MemTotal_bytes。拿不到的就查机型和云盘的官方规格，有几处要注意：

磁盘 IOPS 和带宽同时受云盘和机型限制。以 AWS 为例，gp3 盘的 IOPS 和吞吐按盘配置，实例本身也有 EBS 带宽和 IOPS 上限，取两者中较小的那个。
网络带宽看基线值。很多机型标的是 “up to 10 Gbps”，那是突发上限，能持续的基线要低得多。
单位要统一。网络规格一般用 bit/s，指标是 byte/s，要乘 8 再比较。

CPU 建议用 rate() 而不是 irate()。irate() 只看最后两个采样点，放在监控面板上看瞬时值没问题，但拿来算长窗口的分位数会抖得比较厉害。

第三步：用分位数，不用峰值

按最大值规划听起来最稳，但最大值往往只是一次尖峰：一次备份、一次数据均衡、一条烂 SQL。假设 TiDB 单节点 CPU 峰值是 22 核，P99 是 14 核，按峰值规划要多买大约 57% 的节点，只为了覆盖每天那几分钟，而集群本来就扛得住。

我会对每种资源都算一组统计值，从 P50 到 P99.9 再加最大值，然后决定用哪个：

CPU、磁盘 IOPS、磁盘带宽和网络带宽默认用 P99，能过滤偶发尖峰，又保留了日常的忙时。
存储不看分位数，看最新值和增长趋势。存储基本只增不减，要关心的是下个季度会涨到多少，具体算法在文章后面。
忽略最大值之前，先看看它是什么。如果那个尖峰正是业务最忙的时段，而不是某个一次性任务，那就应该按它来规划。

第四步：定水位线

目标水位是整套方法里最需要判断的地方。我们最早的版本对所有资源统一用 3.3 倍冗余，相当于目标水位 30% 左右。这样很好解释，但它把 CPU 和磁盘当成一回事，而它们的行为差别很大。

每种资源要分别考虑：

故障余量。节点平均分布在三个可用区时，挂掉一个可用区，它的负载会压到另外两个上。要扛住这种情况，平时利用率就得低于三分之二左右，这还没算其他余量。
延迟。CPU 真正的上限是延迟开始上升的那个点，而不是 100%。很多 OLTP 业务在这个点上离跑满还很远，要给突发留空间。
后台任务。存储引擎做 compaction 需要空闲磁盘，加减节点时数据均衡也要空间，磁盘快满的时候这两件事都做不了。
TiDB 内存可以按线性算。TiDB 是无状态的 SQL 层，内存主要是执行查询时临时分配的，比如 hash join、排序、聚合和结果集，查询结束就释放，没有一块预先占满的大缓存池。所以它大致跟并发和查询量成正比，流量涨多少，内存也差不多涨多少。算的时候先从单节点容量里减掉一块固定开销（系统、Go runtime、统计信息缓存等），剩下的再按流量放大。另外 RSS 里包含 GC 还没回收的内存，大查询也会带来尖峰，这也是用 P99 而不是平均值的原因。
池化的内存不能按线性算。TiKV 的 block cache 和 MySQL 的 InnoDB buffer pool 一样，是按配置预先划好的缓存，用着用着就会被填满，不管流量多少，内存看起来都一直很高。它反映的是配置而不是需求，所以 TiKV 不按内存规划，缓存够不够要看命中率。

第五步：找瓶颈

一个虚构的 TiKV 例子：9 个节点，平均分布在 3 个可用区，每台 32 核、2,000 GiB 磁盘，预计下季度流量涨 50%。算出来的节点数按 3 的倍数向上取整，原因在本节最后。

所有资源统一用 3.3 倍冗余：

资源	单节点用量	计算	所需节点
CPU（P99）	12 核	12 × 9 × 1.5 × 3.3 / 32	16.7 → 18
存储（最新值）	900 GiB	900 × 9 × 1.5 × 3.3 / 2000	20.05 → 21

按资源分别设水位，CPU 50%，存储 70%：

资源	单节点用量	计算	所需节点
CPU（P99）	12 核	12 × 9 × 1.5 / (32 × 0.5)	10.1 → 12
存储（最新值）	900 GiB	900 × 9 × 1.5 / (2000 × 0.7)	8.7 → 9

统一系数算出来要 21 台，瓶颈是存储；分资源设水位算出来要 12 台，瓶颈是 CPU。两次用的是同一个公式，差别全部来自水位线，所以时间应该花在这里。为了简单，这里存储也乘了 1.5，更好的做法是按历史趋势算，后面单独讲。

最后还要取整，但不同组件的规则不一样：

TiKV：默认 3 副本，PD 按可用区把同一个 Region 的 3 个副本放到 3 个可用区，每个可用区各存一份完整数据。某个可用区的节点少，那里的每个节点就要多扛数据和写入，整个集群的上限会被它拖住。所以各可用区的节点数要一样多。TiDB Cloud Dedicated 直接把这个做成了规则：TiKV 至少 3 台，扩缩容时 3 个可用区同时增减，步长是 3。所以上面例子里 CPU 算出 10.1 台，要取到 12 台，而不是 11 台。自建集群如果没打 label，或者隔离域比副本多，就没有这个限制。
TiFlash：同样平均部署到各可用区，副本数按表设置，生产环境建议至少 2 副本。各可用区节点数保持一致可以避免倾斜，但没有 TiKV 那样硬性的 3 的倍数要求。
TiDB：无状态，前面挂负载均衡，算出几台就是几台，不需要凑倍数。跨可用区部署时，保证每个可用区至少有一台就行。
PD：一般固定 3 个或 5 个节点，Raft 需要多数派，所以取奇数。它通常不随业务量扩容，不用这个公式算。

这个模型在哪里不准

公式假设流量翻倍资源用量也翻倍，并且加节点后负载能均匀分摊。这两点都只是大致成立。

热点。集群整体的分位数可能很健康，但某一个节点已经被热点表或热点 key 范围打满了。除了看汇总，也要看最忙的那个节点，先解决倾斜再考虑加机器。
非线性增长。有些开销涨得比流量快：写入很重时的 compaction、锁冲突、跨节点事务。增长幅度大的时候，最好用压测验证一下。
突发性能。云主机的网络和磁盘吞吐通常有基线和突发两个上限。要按基线规划，不然突发额度会在最要命的时候用完。
硬件以外的限制。连接数上限、Region 数量、PD 授时延迟，都可能比 CPU 和磁盘更早成为瓶颈。当时我们在容量规划之外还配了一个健康检查，按阈值把这些指标标出来。

如果现在重做

用 SLO 定水位

前面的目标水位，CPU 50%、存储 70%，都是经验值。更好的做法是从 SLO 反推。

先要分清 SLA 和 SLO。TiDB Cloud 对外的 SLA 只有可用性一个指标：每分钟探测一次连接，一分钟内全部失败才算这一分钟不可用，Dedicated 多节点部署承诺每月 99.99%。这是底线，集群慢到延迟翻了几倍，只要还能连上就不算违约，所以拿它判断资源是否饱和没有意义。容量规划要看的是业务自己的 SLO，一般由延迟和错误率组成，比如 P99 延迟低于 50 ms、错误率低于 0.1%。

TiDB 里对应的指标：

# TiDB 层 P99 查询延迟
histogram_quantile(0.99,
  sum(rate(tidb_server_handle_query_duration_seconds_bucket[1m])) by (le))

# 错误率
sum(rate(tidb_server_query_total{result="Error"}[1m]))
  / sum(rate(tidb_server_query_total[1m]))

# TiKV 层 P99 gRPC 延迟，用来判断慢在哪一层
histogram_quantile(0.99,
  sum(rate(tikv_grpc_msg_duration_seconds_bucket[1m])) by (le, type))

接下来要找饱和点，也就是回答一个问题：CPU 利用率涨到多少时，延迟会超出 SLO。

数据有两种来源。一种是历史监控：把过去几周每一分钟的 CPU 利用率和同一分钟的 P99 延迟配成一对，画在图上，横轴是 CPU 利用率，纵轴是 P99 延迟。另一种是压测：在测试环境逐步加压，比如每 10 分钟把 QPS 提高 10%，记录每一档的 CPU 利用率和 P99 延迟。

画出来的图通常像下面这样。CPU 利用率低的时候，请求不用排队，延迟基本不变。过了某个利用率，请求开始排队，延迟急剧上升。在 SLO 的位置画一条横线，它和曲线交点对应的 CPU 利用率就是饱和点，CPU 再往上就会违反 SLO。

图里的饱和点是 75%。三个可用区挂掉一个时，剩下两个要多扛 50% 的负载，所以平时的利用率不能超过 75% / 1.5 = 50%，这就是 CPU 的目标水位。前面例子里的 50% 就是这么来的。不同业务的饱和点差别很大，OLTP 小查询可能到 80% 延迟都很稳，带大量排序和聚合的查询可能 60% 就开始抖，所以每个集群要单独看。

存储按趋势算

存储的增长主要取决于写入量和数据保留策略，跟 QPS 的关系不大，所以不适合乘一个流量增长倍数，按历史趋势算更准。

取过去 3 到 6 个月的已用存储，每天取一个点，用当天的最大值或平均值把 compaction 造成的锯齿抹平。sum(tikv_store_size_bytes{type="used"}) 是所有副本加起来的总量，和单节点容量比的时候要用同一个口径。
用线性回归求出斜率，比如集群每天增长 60 GiB。Prometheus 自带的 predict_linear() 可以直接给出预测值，比如 predict_linear(sum(tikv_store_size_bytes{type="used"})[30d:1h], 90*86400) 是 90 天后的用量。不过 Prometheus 一般只保留 15 到 30 天的数据，要看更长的趋势，需要 Thanos、VictoriaMetrics 这类长期存储，或者每天把总量记到一张表里。
算出规划时点的用量：未来用量 = 当前用量 + 斜率 × (到下次评审的天数 + 扩容准备时间)。准备时间一定要算进去，加完节点后数据均衡也要时间，不能等到快满了才扩。
把未来用量代入公式，增长倍数取 1，因为趋势里已经包含了增长。

接着前面的例子：9 台 TiKV 现在一共用了 8,100 GiB，每天涨 60 GiB，规划 90 天，再留 30 天准备时间。

未来用量 = 8,100 + 60 × (90 + 30) = 15,300 GiB
所需节点 = 15,300 / (2,000 × 0.7) = 10.9 → 12

同样的数据还能算出一个很实用的告警指标：

距离到达水位的天数 = (总容量 × 目标水位 − 当前用量) / 每天增长量

这个天数小于扩容准备时间，就该扩了。

拟合时有几个坑：

一次性事件会把斜率带偏，比如批量导入、TTL 删除、DROP 表。DROP 表之后空间要等 GC 和 compaction 才会释放，曲线会先平一段再往下掉。拟合前要把这些时段剔掉。
增长不一定是线性的。如果每周增长的百分比比较稳定，说明是复利式增长，要先对用量取对数再拟合，否则会低估。

其他

报告里加上单节点倾斜，让热点和总量出现在同一份结果里。
定期跑。每周跑一次，预测的节点数变化时告警，而不是每次大促前临时跑一下。

检查清单

观察窗口至少一周，并且包含业务高峰。
按组件、按资源分别采集用量和容量。
CPU 和网络用 P99，存储用最新值加趋势。
每种资源单独定水位，CPU 从延迟 SLO 反推，再考虑故障余量和后台任务。
按资源分别算节点数，最大的那个就是瓶颈。
TiKV 各可用区节点数保持一样（TiDB Cloud 上是 3 的倍数），TiDB 不用凑倍数。
看一眼最忙的节点，变化大的时候做压测。

用 Prometheus 指标给分布式数据库做容量规划：以 TiDB 为例 was originally published by Sherry Yin at Sherry Yin on October 10, 2024.

Capacity Planning a Distributed Database from Prometheus Metrics

2024-10-10T00:00:00+08:00

“How many nodes do we need for next quarter?” is a question every database team gets, usually a few weeks before a big launch or a sales event. The honest answer often starts with a guess, gets padded “to be safe”, and ends with a cluster that is either too small on the day or far too big for the rest of the year.

A while ago a colleague and I wrote a small tool, tidbcloud-metrics, that answered it from the metrics we already had in Prometheus. Before every big sales event, we used it to assess capacity for very large TiDB Cloud clusters and decide how many nodes to add. The code was tied to one environment, but the method carries over. This post walks through it with made-up numbers: what to measure, how to summarize it, how much headroom to keep, and where the simple model breaks.

The examples use TiDB, but the same approach works for most distributed databases and stateless services that expose per-node metrics.

The idea in one formula

For each component and each resource, the number of nodes you need is:

nodes needed = (usage per node × current node count × growth) / (capacity per node × target utilization)

usage per node: a summary of how much of the resource each node uses today, such as the 99th percentile over the observation window.
current node count × usage per node: roughly the total demand on the cluster today.
growth: how much more traffic you expect. 1.5 means 50% more.
capacity per node: what one node can provide, such as 32 cores or 2 TiB of disk.
target utilization: how full you are willing to run each node. Dividing by 0.5 means you want nodes at 50% or less.

You calculate this for every resource of every component. The resource that needs the most nodes is the bottleneck, and that sets the node count for that component.

Step 1: Pick the observation window

The numbers come from whatever period you query, so the period has to contain the load you are planning for.

Cover at least one full business cycle, usually a week, so weekday and weekend patterns are both in the data.
Make sure the window includes your known peaks: month-end batch jobs, the last sales event, the daily reporting run.
If you are planning for a specific event, measure the most similar past event and use growth to scale it.
Watch the query step. A long window with a large step averages away short spikes. I used a 60-second step for two days of data and 30 seconds for one day.

Step 2: Measure usage per component and resource

A TiDB cluster has four main components. Every node runs node_exporter, so host metrics for CPU, memory, disk IOPS, disk bandwidth and network bandwidth exist for all four, and all of them should go through the calculation. Each component uses disk and network differently, though, so they tend to run out of different things:

Component	What disk and network are used for	Usually runs out of
TiDB (SQL layer)	Disk: logs, and sorts and hash joins that spill when a large query runs out of memory. Network: client requests, reading from TiKV and TiFlash, returning results	CPU, memory, network bandwidth
TiKV (row storage)	Disk: data, Raft logs, compaction. Network: Raft replication to other replicas, serving reads and writes from TiDB	CPU, storage, disk IOPS, disk bandwidth
TiFlash (column storage)	Disk: columnar data and background merges. Network: replicating from TiKV, exchanging data between nodes in MPP queries	CPU, memory, storage, disk bandwidth
PD (placement and timestamps)	Disk: etcd for cluster metadata, sensitive to fsync latency. Network: heartbeats, TSO requests	CPU, memory, disk latency

Disk IOPS, disk bandwidth and network bandwidth are three separate limits, and each needs its own calculation:

Disk IOPS is the number of reads and writes per second. Small random IO hits it first, such as TiKV point lookups and RocksDB reading blocks that aren’t in the cache.
Disk bandwidth is the number of bytes read and written per second. Large sequential IO hits it first, such as compaction, range scans, TiFlash reading column files, and backups.
Network bandwidth is the bytes per second the network interface sends and receives. On every TiKV write, the leader sends the log to the other two replicas, across availability zones. TiDB pulling data from TiKV, results going back to clients, backups and rebalancing all use the network too.

The same disk can be at 30% of its IOPS and 80% of its bandwidth, so lumping them together hides the real bottleneck.

CPU, memory and storage come from TiDB’s own metrics. Disk and network come from node_exporter:

# CPU cores used per TiDB instance
rate(process_cpu_seconds_total{component="tidb"}[2m])

# Resident memory per TiDB instance
process_resident_memory_bytes{component="tidb"}

# Used storage per TiKV store
sum(tikv_store_size_bytes{type="used"}) by (instance)

# Used storage per TiFlash store
sum(tiflash_system_current_metric_StoreSizeUsed) by (instance)

# Disk IOPS (reads + writes)
sum(rate(node_disk_reads_completed_total{device="nvme1n1"}[1m])
  + rate(node_disk_writes_completed_total{device="nvme1n1"}[1m])) by (instance)

# Disk bandwidth in bytes per second (reads + writes)
sum(rate(node_disk_read_bytes_total{device="nvme1n1"}[1m])
  + rate(node_disk_written_bytes_total{device="nvme1n1"}[1m])) by (instance)

# Network bandwidth in bytes per second, inbound and outbound separately
sum(rate(node_network_receive_bytes_total{device="eth0"}[1m])) by (instance)
sum(rate(node_network_transmit_bytes_total{device="eth0"}[1m])) by (instance)

Replace nvme1n1 and eth0 with your actual data disk and network interface.

For per-node capacity, use the matching capacity metrics where they exist, such as tikv_store_size_bytes{type="capacity"} or node_memory_MemTotal_bytes. Otherwise use the published limits of the instance type and the volume, with a few things to watch:

Disk IOPS and bandwidth are limited by both the volume and the instance. On AWS, for example, a gp3 volume has its own provisioned IOPS and throughput, and the instance has its own EBS bandwidth and IOPS limits. The lower of the two applies.
Use the baseline network bandwidth. Many instance types are listed as “up to 10 Gbps”, which is the burst limit; the sustained baseline is much lower.
Keep units consistent. Network limits are usually in bits per second and the metrics are in bytes, so multiply by 8 before comparing.

Use rate() rather than irate() for CPU. irate() only looks at the last two samples, which is useful on a dashboard but makes percentiles over a long window noisier than they should be.

Step 3: Summarize with percentiles, not the peak

Planning on the maximum sounds safe, but the maximum is usually a single spike: a backup, a rebalance, one bad query. Say TiDB CPU peaks at 22 cores per node while the 99th percentile is 14. Planning on the peak adds about 57% more nodes to cover a few minutes a day that the cluster already handles.

I calculate a range of statistics for every resource, from p50 to p99.9 plus the max, and decide which one to plan on:

p99 is a sensible default for CPU, disk IOPS, disk bandwidth and network bandwidth. It ignores rare spikes but keeps the regular busy periods.
For storage, look at the latest value and the trend rather than a percentile. Storage rarely shrinks, and next quarter’s size is what matters. How to calculate the trend is covered later.
Check what the max actually was before ignoring it. If the spike was your peak business hour rather than a one-off job, plan on it.

Step 4: Decide how full a node may get

The target utilization is where most of the judgement goes. My first version used a single redundancy factor of 3.3 for everything, which means a target of about 30% utilization. That is easy to explain, but it treats CPU and disk as if they behaved the same way, and they don’t.

Things to consider for each resource:

Failure headroom. With nodes spread evenly across three availability zones, losing one zone moves its load onto the other two. To survive that without being overloaded, normal utilization has to stay below about two thirds, before any other headroom.
Latency. For CPU, the useful limit is where latency starts to climb, not 100%. For many OLTP workloads that is well below full usage, so leave room for bursts.
Background work. Storage engines need free disk for compaction, and a cluster needs space to rebalance data when a node is added or lost. A disk that is nearly full can’t absorb either of them.
TiDB memory can be treated as linear. TiDB is a stateless SQL layer. Most of its memory is allocated while queries run, for hash joins, sorts, aggregations and result sets, and released when they finish. There is no large cache that is filled up front, so memory roughly follows concurrency and query volume, and grows about as fast as traffic does. I subtract a fixed overhead from each node’s capacity first (the OS, the Go runtime, the statistics cache and so on) and scale the rest. RSS also includes memory the garbage collector hasn’t reclaimed yet, and large queries cause spikes, which is another reason to plan on p99 rather than the average.
Pooled memory can’t. TiKV’s block cache works like MySQL’s InnoDB buffer pool: it is sized by configuration and fills up over time, so memory looks steadily high no matter how much traffic there is. It reflects the configuration, not demand, so I don’t plan TiKV nodes on memory. Whether the cache is big enough is a question for the hit rate.

Step 5: Find the bottleneck

Here is a made-up TiKV example: 9 nodes spread evenly across 3 availability zones, each with 32 cores and 2,000 GiB of disk, and 50% more traffic expected next quarter. Results are rounded up to a multiple of 3; the reason is at the end of this section.

With one factor of 3.3 for every resource:

Resource	Usage per node	Calculation	Nodes needed
CPU (p99)	12 cores	12 × 9 × 1.5 × 3.3 / 32	16.7 → 18
Storage (latest)	900 GiB	900 × 9 × 1.5 × 3.3 / 2000	20.05 → 21

With a target per resource, CPU at 50% and storage at 70%:

Resource	Usage per node	Calculation	Nodes needed
CPU (p99)	12 cores	12 × 9 × 1.5 / (32 × 0.5)	10.1 → 12
Storage (latest)	900 GiB	900 × 9 × 1.5 / (2000 × 0.7)	8.7 → 9

The uniform factor says 21 nodes, with storage as the bottleneck. The per-resource targets say 12 nodes, with CPU as the bottleneck. The formula is the same in both cases. The difference comes entirely from the targets, so that is where to spend the time. To keep it simple, storage is multiplied by 1.5 here as well; a trend is a better way to project it, covered later.

Finally, round the result, but the rule depends on the component:

TiKV keeps 3 replicas by default, and PD places the 3 replicas of each Region in 3 different availability zones, so each zone holds a full copy of the data. If one zone has fewer nodes, each of its nodes carries more data and more writes, and that zone caps the whole cluster. Keep the same number of nodes in every zone. TiDB Cloud Dedicated enforces this: TiKV starts at 3 nodes, and scaling adds or removes nodes in all 3 zones at the same time, so the step is 3. That is why the 10.1 nodes for CPU in the example become 12, not 11. A self-managed cluster without location labels, or with more isolation domains than replicas, doesn’t have this constraint.
TiFlash is also spread evenly across zones, with replicas set per table and at least 2 recommended for production. Keeping zones even avoids skew, but there is no strict multiple-of-3 rule as there is for TiKV.
TiDB is stateless behind a load balancer. Use whatever number the formula gives; there is nothing to round to. If it spans availability zones, just keep at least one node in each.
PD usually runs a fixed 3 or 5 nodes, an odd number because Raft needs a majority. It rarely scales with traffic and isn’t sized with this formula.

Where the simple model breaks

The formula assumes that doubling traffic doubles resource usage and that adding nodes spreads it evenly. Both are only roughly true.

Hotspots. A cluster-wide percentile can look healthy while one node is saturated by a hot table or a hot key range. Look at the busiest node as well as the summary, and fix skew before buying hardware.
Non-linear growth. Some costs grow faster than traffic: compaction under heavy writes, lock contention, cross-node transactions. Validate the plan with a load test when the growth is large.
Burstable limits. Cloud instances often have a baseline network and disk throughput and a higher burst limit. Plan on the baseline, or you will run out of burst credits at the worst moment.
Things that aren’t hardware. Connection limits, region or shard counts, and time-oracle (PD) latency can become the limit before CPU or disk does. I paired the capacity plan with a health check that flagged these against thresholds.

What I would add today

Targets from SLOs

The targets above, 50% for CPU and 70% for storage, are rules of thumb. A better way is to work them out from an SLO.

First, an SLA is not an SLO. TiDB Cloud’s SLA covers availability only: a probe tries to connect every minute, and a minute counts as down only if every attempt fails. Dedicated clusters with multiple nodes are committed to 99.99% a month. That is a floor. A cluster can be several times slower than usual and still meet it, so it says nothing about saturation. For capacity planning, use your own SLO, usually a latency and an error rate, such as p99 latency under 50 ms and errors under 0.1%.

The matching TiDB metrics:

# p99 query latency at the TiDB layer
histogram_quantile(0.99,
  sum(rate(tidb_server_handle_query_duration_seconds_bucket[1m])) by (le))

# Error rate
sum(rate(tidb_server_query_total{result="Error"}[1m]))
  / sum(rate(tidb_server_query_total[1m]))

# p99 gRPC latency at the TiKV layer, to see which layer is slow
histogram_quantile(0.99,
  sum(rate(tikv_grpc_msg_duration_seconds_bucket[1m])) by (le, type))

Next, find the saturation point, which answers one question: at what CPU utilization does latency break the SLO?

There are two ways to get the data. One is history: for every minute over the last few weeks, pair the CPU utilization with the p99 latency in the same minute and plot them, with CPU utilization on the x-axis and p99 latency on the y-axis. The other is a load test: raise the load step by step in a test environment, for example 10% more QPS every 10 minutes, and record CPU utilization and p99 latency at each step.

The result usually looks like the chart below. At low utilization, requests don’t wait for each other and latency barely moves. Past some point, requests start to queue and latency rises steeply. Draw a horizontal line at the SLO. The CPU utilization where it meets the curve is the saturation point, and any more CPU than that breaks the SLO.

In the chart it is 75%. If one of three availability zones goes down, the other two take 50% more load, so normal utilization has to stay at or below 75% / 1.5 = 50%. That is the CPU target, and it is where the 50% in the example came from. Saturation points vary a lot between workloads. Small OLTP queries may stay flat past 80%, while queries with heavy sorts and aggregations may start to wobble at 60%, so measure each cluster.

Storage from a trend

Storage grows with write volume and retention, not with QPS, so multiplying it by a traffic growth factor is a rough guess at best. A trend works better.

Take the used storage for the last 3 to 6 months, one point per day, using the daily max or average to smooth out the sawtooth from compaction. sum(tikv_store_size_bytes{type="used"}) includes every replica, so compare it with capacity on the same basis.
Fit a straight line to get the slope, for example 60 GiB a day for the whole cluster. Prometheus can do this with predict_linear(): predict_linear(sum(tikv_store_size_bytes{type="used"})[30d:1h], 90*86400) gives the value 90 days from now. Most Prometheus setups keep only 15 to 30 days, though, so for a longer trend you need long-term storage such as Thanos or VictoriaMetrics, or a daily row in a table.
Project the usage at the planning date: future usage = current usage + slope × (days until the next review + lead time to scale). Include the lead time. New nodes need time to rebalance data, so you can’t wait until the disks are nearly full.
Use the projected usage in the formula with a growth factor of 1, because the trend already includes the growth.

Continuing the example: the 9 TiKV nodes use 8,100 GiB in total and grow by 60 GiB a day. The plan covers 90 days, with 30 days of lead time.

future usage = 8,100 + 60 × (90 + 30) = 15,300 GiB
nodes needed = 15,300 / (2,000 × 0.7) = 10.9 → 12

The same numbers give a useful alert:

days until target = (total capacity × target utilization − current usage) / daily growth

When that number drops below your lead time, it is time to scale.

A few traps when fitting the line:

One-off events skew the slope: bulk imports, TTL deletes, dropped tables. After a DROP TABLE the space comes back only after GC and compaction, so the curve stays flat for a while and then falls. Leave those periods out of the fit.
Growth isn’t always linear. If the weekly growth percentage is steady, the growth is compounding. Fit the logarithm of usage instead, or the projection will come out too low.

Other things

Per-node skew in the output, so hotspots show up in the same report as the totals.
A schedule. Run it every week and alert when the projected node count changes, rather than running it once before each event.

Checklist

Measure at least a week that includes your peaks.
Collect usage and capacity per component and per resource.
Plan on p99 for CPU and network, and on the latest value plus trend for storage.
Set a target utilization per resource. Work out the CPU target from your latency SLO, then allow for failure headroom and background work.
Calculate nodes per resource. The largest result is your bottleneck.
Keep TiKV nodes even across zones (a multiple of 3 on TiDB Cloud). TiDB needs no rounding.
Check the busiest node, and load-test large changes.

Capacity Planning a Distributed Database from Prometheus Metrics was originally published by Sherry Yin at Sherry Yin on October 10, 2024.

How to ensure TiDB cluster can use TiCDC to synchronize data?

2024-08-03T00:00:00+08:00

This article focuses on the more difficult to determine parts of the TiCDC’s limitations.

Currently, TiCDC unsupported scenarios and limitations as below:

A TiKV cluster that uses RawKV alone.
The CREATE SEQUENCE DDL operation and the SEQUENCE function in TiDB. When the upstream TiDB uses SEQUENCE, TiCDC ignores SEQUENCE DDL operations/functions performed upstream. However, DML operations using SEQUENCE functions can be correctly replicated.
Currently, performing BR data recovery and TiDB Lightning physical import imports on tables and databases that are being replicated by TiCDC is not supported. For more information, see Why does replication using TiCDC stall or even stop after data restore using TiDB Lightning and BR from upstream.
TiCDC only replicates tables that have at least one valid index

For details, refer to the official documentation unsupported scenarios and best practices.

The most difficult part of these limitations to be identified is the valid index:

TiCDC only replicates tables that have at least one valid index. A valid index is defined as follows:

A primary key (PRIMARY KEY) is a valid index.

A unique index (UNIQUE INDEX) is valid if every column of the index is explicitly defined as non-nullable (NOT NULL) and the index does not have a virtual generated column (VIRTUAL GENERATED COLUMNS).

If there are tables in the TiDB cluster without at least one valid index, then these tables cannot use TiCDC to synchronize data. How to find out which tables in the cluster do not have any valid index?

1.Find tables that have primary key

select distinct table_schema,table_name from information_schema.tidb_indexes where KEY_NAME='PRIMARY' and table_schema not in ('METRICS_SCHEMA','mysql','PERFORMANCE_SCHEMA','INFORMATION_SCHEMA','sys');

2.Find tables have unique index and every column of the index is explicitly defined as non-nullable (NOT NULL) and the index does not have a virtual generated column

select distinct a.table_schema, a.table_name from (select table_schema,table_name,column_name from information_schema.tidb_indexes where KEY_NAME<>'PRIMARY' and NON_UNIQUE=0 and table_schema not in ('METRICS_SCHEMA','mysql','PERFORMANCE_SCHEMA','INFORMATION_SCHEMA','sys')) a left join (select TABLE_SCHEMA,TABLE_NAME,COLUMN_NAME,IS_NULLABLE,GENERATION_EXPRESSION from information_schema.columns where table_schema not in ('METRICS_SCHEMA','mysql','PERFORMANCE_SCHEMA','INFORMATION_SCHEMA','sys')) b on a.table_schema=b.table_schema and a.table_name=b.table_name and a.column_name=b.column_name where (b.IS_NULLABLE='NO' and b.GENERATION_EXPRESSION='');

3.Combine the result of the above two sets are the tables have valid index

select distinct table_schema,table_name from information_schema.tidb_indexes where KEY_NAME='PRIMARY' and table_schema not in ('METRICS_SCHEMA','mysql','PERFORMANCE_SCHEMA','INFORMATION_SCHEMA','sys')
union all
select distinct a.table_schema, a.table_name from (select table_schema,table_name,column_name from information_schema.tidb_indexes where KEY_NAME<>'PRIMARY' and NON_UNIQUE=0 and table_schema not in ('METRICS_SCHEMA','mysql','PERFORMANCE_SCHEMA','INFORMATION_SCHEMA','sys')) a left join (select TABLE_SCHEMA,TABLE_NAME,COLUMN_NAME,IS_NULLABLE,GENERATION_EXPRESSION from information_schema.columns where table_schema not in ('METRICS_SCHEMA','mysql','PERFORMANCE_SCHEMA','INFORMATION_SCHEMA','sys')) b on a.table_schema=b.table_schema and a.table_name=b.table_name and a.column_name=b.column_name where (b.IS_NULLABLE='NO' and b.GENERATION_EXPRESSION='');

Conclusion

Tables Without Valid Index = All Tables - Tables With Valid Index, if the result set of the follow SQL is not empty, means the TiDB cluster contains tables can not be replicated by TiCDC.

select table_schema, table_name from information_schema.tables where table_schema not in ('METRICS_SCHEMA','mysql','PERFORMANCE_SCHEMA','INFORMATION_SCHEMA','sys') and table_type='BASE TABLE' and (table_schema, table_name) not in (select distinct table_schema,table_name from information_schema.tidb_indexes where KEY_NAME='PRIMARY' and table_schema not in ('METRICS_SCHEMA','mysql','PERFORMANCE_SCHEMA','INFORMATION_SCHEMA','sys') union all select distinct a.table_schema, a.table_name from (select table_schema,table_name,column_name from information_schema.tidb_indexes where KEY_NAME<>'PRIMARY' and NON_UNIQUE=0 and table_schema not in ('METRICS_SCHEMA','mysql','PERFORMANCE_SCHEMA','INFORMATION_SCHEMA','sys')) a left join (select TABLE_SCHEMA,TABLE_NAME,COLUMN_NAME,IS_NULLABLE,GENERATION_EXPRESSION from information_schema.columns where table_schema not in ('METRICS_SCHEMA','mysql','PERFORMANCE_SCHEMA','INFORMATION_SCHEMA','sys')) b on a.table_schema=b.table_schema and a.table_name=b.table_name and a.column_name=b.column_name where (b.IS_NULLABLE='NO' and b.GENERATION_EXPRESSION=''));

Others

About another limitation not supporting SEQUENCE related operations. It is very simple to find out whether SEQUENCE is used in the cluster. The SQL is as follows. If the query result is not empty, it means that SEQUENCE is used:

SELECT table_schema, table_name, table_type from information_schema.tables where table_type='SEQUENCE';

中文版

本文主要围绕 TiCDC 的限制中比较难确定的部分内容，TiCDC 官方文档中已经清楚的描述了不支持的场景以及必须满足的条件。

目前 TiCDC 暂不支持的场景和限制如下：

暂不支持单独使用 RawKV 的 TiKV 集群。
暂不支持在 TiDB 中创建 SEQUENCE 的 DDL 操作和 SEQUENCE 函数。在上游 TiDB 使用 SEQUENCE 时，TiCDC 将会忽略掉上游执行的 SEQUENCE DDL 操作/函数，但是使用 SEQUENCE 函数的 DML 操作可以正确地同步。
暂不支持在同步的过程中对 TiCDC 正在同步的表和库进行 BR 数据恢复和 TiDB Lightning 物理导入。详情请参考为什么在上游使用了 TiDB Lightning 和 BR 恢复了数据之后，TiCDC 同步会出现卡顿甚至卡住。
TiCDC 同步的表需要至少存在一个有效索引的表。

详见官方文档中暂不支持的场景和最佳实践。

这些限制中最不好判断的是有效索引这一条：

TiCDC同步的表需要至少存在一个有效索引的表，有效索引的定义如下：

主键 (PRIMARY KEY) 为有效索引。

唯一索引 (UNIQUE INDEX) 中每一列在表结构中明确定义非空 (NOT NULL) 且不存在虚拟生成列 (VIRTUAL GENERATED COLUMNS)。

如果 TiDB 集群中有表没有有效索引，那么这些表是不能利用 TiCDC 来同步数据的，如何集群中哪些表没有有效索引呢？推导如下：

1.找出有主键的表

select distinct table_schema,table_name from information_schema.tidb_indexes where KEY_NAME='PRIMARY' and table_schema not in ('METRICS_SCHEMA','mysql','PERFORMANCE_SCHEMA','INFORMATION_SCHEMA','sys');

2.找出有唯一索引并且索引中每一列在表结构中明确定义非空 (NOT NULL) 且不存在虚拟生成列的表

select distinct a.table_schema, a.table_name from (select table_schema,table_name,column_name from information_schema.tidb_indexes where KEY_NAME<>'PRIMARY' and NON_UNIQUE=0 and table_schema not in ('METRICS_SCHEMA','mysql','PERFORMANCE_SCHEMA','INFORMATION_SCHEMA','sys')) a left join (select TABLE_SCHEMA,TABLE_NAME,COLUMN_NAME,IS_NULLABLE,GENERATION_EXPRESSION from information_schema.columns where table_schema not in ('METRICS_SCHEMA','mysql','PERFORMANCE_SCHEMA','INFORMATION_SCHEMA','sys')) b on a.table_schema=b.table_schema and a.table_name=b.table_name and a.column_name=b.column_name where (b.IS_NULLABLE='NO' and b.GENERATION_EXPRESSION='');

3.以上两种表的并集就是含有有效索引的表。

select distinct table_schema,table_name from information_schema.tidb_indexes where KEY_NAME='PRIMARY' and table_schema not in ('METRICS_SCHEMA','mysql','PERFORMANCE_SCHEMA','INFORMATION_SCHEMA','sys') 
union all 
select distinct a.table_schema, a.table_name from (select table_schema,table_name,column_name from information_schema.tidb_indexes where KEY_NAME<>'PRIMARY' and NON_UNIQUE=0 and table_schema not in ('METRICS_SCHEMA','mysql','PERFORMANCE_SCHEMA','INFORMATION_SCHEMA','sys')) a left join (select TABLE_SCHEMA,TABLE_NAME,COLUMN_NAME,IS_NULLABLE,GENERATION_EXPRESSION from information_schema.columns where table_schema not in ('METRICS_SCHEMA','mysql','PERFORMANCE_SCHEMA','INFORMATION_SCHEMA','sys')) b on a.table_schema=b.table_schema and a.table_name=b.table_name and a.column_name=b.column_name where (b.IS_NULLABLE='NO' and b.GENERATION_EXPRESSION='');

总结

没有有效索引的表 = 所有的表 - 含有有效索引的表，如果以下 SQL 的查询结果为非空，则表示 TiDB 集群中包含有不能使用 TiCDC 来同步数据的表：

select table_schema, table_name from information_schema.tables where table_schema not in ('METRICS_SCHEMA','mysql','PERFORMANCE_SCHEMA','INFORMATION_SCHEMA','sys') and table_type='BASE TABLE' and (table_schema, table_name) not in (select distinct table_schema,table_name from information_schema.tidb_indexes where KEY_NAME='PRIMARY' and table_schema not in ('METRICS_SCHEMA','mysql','PERFORMANCE_SCHEMA','INFORMATION_SCHEMA','sys') union all select distinct a.table_schema, a.table_name from (select table_schema,table_name,column_name from information_schema.tidb_indexes where KEY_NAME<>'PRIMARY' and NON_UNIQUE=0 and table_schema not in ('METRICS_SCHEMA','mysql','PERFORMANCE_SCHEMA','INFORMATION_SCHEMA','sys')) a left join (select TABLE_SCHEMA,TABLE_NAME,COLUMN_NAME,IS_NULLABLE,GENERATION_EXPRESSION from information_schema.columns where table_schema not in ('METRICS_SCHEMA','mysql','PERFORMANCE_SCHEMA','INFORMATION_SCHEMA','sys')) b on a.table_schema=b.table_schema and a.table_name=b.table_name and a.column_name=b.column_name where (b.IS_NULLABLE='NO' and b.GENERATION_EXPRESSION=''));

其他

还有一个关于不支持 SEQUENCE 相关操作的限制，找出集群中是否使用了 SEQUENCE 是很简单的，SQL 如下，如果查询结果为非空，则说明使用了 SEQUENCE：

SELECT table_schema, table_name, table_type from information_schema.tables where table_type='SEQUENCE';

How to ensure TiDB cluster can use TiCDC to synchronize data? was originally published by Sherry Yin at Sherry Yin on August 03, 2024.

TiDB Operator Enable TLS For MySQL Client With cert-manager

2024-06-06T00:00:00+08:00

TiDB Operatior supports enable and disable TLS for MySQL client for an existing TiDB cluster. This article will show you how to enable it. Certificates can be issued by cfssl and cert-manager, here I will choose the cloud native way cert-manager.

We need a self-signed CA
Use the CA to sign certificate for TiDB cluster

Prepare CA

Install cert-manager

helm repo add jetstack https://charts.jetstack.io --force-update
helm install cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace --version v1.15.2 --set crds.enabled=true

It will take some time, be patient.

Prepare Issuers

TiDB is not a internet facing public services, self-signed issuer and CA is enough. There’s no naming restrictions for the issuers for TiDB Operator.
1.Create tidb-server-issuer.yaml

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: tidb-selfsigned-ca-issuer
  namespace: tidb-cluster 
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: tidb-ca
  namespace: tidb-cluster
spec:
  secretName: tidb-ca-secret                  # specify a secret to store the CA certificate and private key
  commonName: "TiDB CA"
  isCA: true                                  # Make this certificate is valid for certificate signing
  duration: 87600h # 10yrs
  renewBefore: 720h # 30d
  issuerRef:
    name: tidb-selfsigned-ca-issuer           # Must specify an issuer for the certificate, we want a CA certificate, the issuer should be self-signed issuer
    kind: Issuer
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: tidb-issuer                           # This is the issuer issues TLS certificates for the TiDB cluster
  namespace: tidb-cluster
spec:
  ca:
    secretName: tidb-ca-secret

2.Create Issuers

kubectl apply -f tidb-server-issuer.yaml

kubectl get issuer -n tidb-cluster

Output:

NAMESPACE      NAME                        READY   AGE
tidb-cluster   tidb-issuer                 True    18s
tidb-cluster   tidb-selfsigned-ca-issuer   True    18s

Enable TiDB Cluster TLS for Client

Perpare Server Certificates

We need to issue two sets of certificates:

A set of server-side certificates for TiDB server
A set of client-side certificates for MySQL client.

Certificate and keys will be stored in secrets, ${cluster_name}-tidb-server-secret and ${cluster_name}-tidb-client-secrete.

The Secret objects you created must follow the above naming convention. Otherwise, the deployment of the TiDB cluster will fail.

1.Create tidb-server-cert.yaml

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: advanced-test-tidb-server-secret
  namespace: tidb-cluster
spec:
  secretName: advanced-test-tidb-server-secret
  duration: 8760h # 365d
  renewBefore: 360h # 15d
  subject:
    organizations:
    - Sherry Yin Xi
  commonName: "TiDB Server"
  usages:
    - server auth
  dnsNames:
    - "advanced-test-tidb"
    - "advanced-test-tidb.tidb-cluster"
    - "advanced-test-tidb.tidb-cluster.svc"
    - "*.advanced-test-tidb"
    - "*.advanced-test-tidb.tidb-cluster"
    - "*.advanced-test-tidb.tidb-cluster.svc"
    - "*.advanced-test-tidb-peer"
    - "*.advanced-test-tidb-peer.tidb-cluster"
    - "*.advanced-test-tidb-peer.tidb-clsuter.svc"
  ipAddresses:
    - 127.0.0.1
    - ::1
  issuerRef:
    name: tidb-issuer
    kind: Issuer
    group: cert-manager.io

2.Issue Certificate for TiDB Server

kubectl apply -f tidb-server-cert.yaml

Perpare Client Certificates

1.Create tidb-client-cert.yaml

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: advanced-test-tidb-client-secret
  namespace: tidb-cluster
spec:
  secretName: advanced-test-tidb-client-secret
  duration: 8760h # 365d
  renewBefore: 360h # 15d
  subject:
    organizations:
    - Sherry Yin Xi
  commonName: "TiDB Client"
  usages:
    - client auth
  issuerRef:
    name: tidb-issuer
    kind: Issuer
    group: cert-manager.io

2.Issue Cerficate for MySQL Client

kubectl apply -f tidb-client-cert.yaml

3.Other Client Components Please refer to official documentaiton.

Enable TLS for Client

For an existing TiDB cluster, just update your tidb cluster manifest set .spec.tidb.tlsClient.enabled to true as below.

spec:
...
  tidb:
    ...
    tlsClient:                  
      enabled: true             
  ...

After apply the above changes, the tidb and pd pods of your TiDB cluster will rolling restart.

Check the logs of PD and TiDB pods

kubectl logs advanced-test-pd-0 -n tidb-cluster |grep -i tls
[2024/06/06 02:37:20.061 +00:00] [INFO] [server.go:249] ["PD config"] [config="{\"client-urls\":\"http://0.0.0.0:2379\",\"peer-urls\":\"http://0.0.0.0:2380\",\"advertise-client-urls\":\"http://advanced-test-pd-0.advanced-test-pd-peer.tidb-cluster.svc:2379\",\"advertise-peer-urls\":\"http://advanced-test-pd-0.advanced-test-pd-peer.tidb-cluster.svc:2380\",\"name\":\"advanced-test-pd-0\",\"data-dir\":\"/var/lib/pd\",\"force-new-cluster\":false,\"enable-grpc-gateway\":true,\"initial-cluster\":\"advanced-test-pd-0=http://advanced-test-pd-0.advanced-test-pd-peer.tidb-cluster.svc:2380\",\"initial-cluster-state\":\"new\",\"initial-cluster-token\":\"pd-cluster\",\"join\":\"\",\"lease\":3,\"log\":{\"level\":\"info\",\"format\":\"text\",\"disable-timestamp\":false,\"file\":{\"filename\":\"\",\"max-size\":0,\"max-days\":0,\"max-backups\":0},\"development\":false,\"disable-caller\":false,\"disable-stacktrace\":false,\"disable-error-verbose\":true,\"sampling\":null,\"error-output-path\":\"\"},\"max-concurrent-tso-proxy-streamings\":5000,\"tso-proxy-recv-from-client-timeout\":\"1h0m0s\",\"tso-save-interval\":\"3s\",\"tso-update-physical-interval\":\"50ms\",\"enable-local-tso\":false,\"metric\":{\"job\":\"advanced-test-pd-0\",\"address\":\"\",\"interval\":\"15s\"},\"schedule\":{\"max-snapshot-count\":64,\"max-pending-peer-count\":64,\"max-merge-region-size\":20,\"max-merge-region-keys\":0,\"split-merge-interval\":\"1h0m0s\",\"switch-witness-interval\":\"1h0m0s\",\"enable-one-way-merge\":\"false\",\"enable-cross-table-merge\":\"true\",\"patrol-region-interval\":\"10ms\",\"max-store-down-time\":\"30m0s\",\"max-store-preparing-time\":\"48h0m0s\",\"leader-schedule-limit\":4,\"leader-schedule-policy\":\"count\",\"region-schedule-limit\":2048,\"witness-schedule-limit\":4,\"replica-schedule-limit\":64,\"merge-schedule-limit\":8,\"hot-region-schedule-limit\":4,\"hot-region-cache-hits-threshold\":3,\"store-limit\":{},\"tolerant-size-ratio\":0,\"low-space-ratio\":0.8,\"high-space-ratio\":0.7,\"region-score-formula-version\":\"v2\",\"scheduler-max-waiting-operator\":5,\"enable-remove-down-replica\":\"true\",\"enable-replace-offline-replica\":\"true\",\"enable-make-up-replica\":\"true\",\"enable-remove-extra-replica\":\"true\",\"enable-location-replacement\":\"true\",\"enable-debug-metrics\":\"false\",\"enable-joint-consensus\":\"true\",\"enable-tikv-split-region\":\"true\",\"schedulers-v2\":[{\"type\":\"balance-region\",\"args\":null,\"disable\":false,\"args-payload\":\"\"},{\"type\":\"balance-leader\",\"args\":null,\"disable\":false,\"args-payload\":\"\"},{\"type\":\"balance-witness\",\"args\":null,\"disable\":false,\"args-payload\":\"\"},{\"type\":\"hot-region\",\"args\":null,\"disable\":false,\"args-payload\":\"\"},{\"type\":\"transfer-witness-leader\",\"args\":null,\"disable\":false,\"args-payload\":\"\"}],\"schedulers-payload\":null,\"hot-regions-write-interval\":\"10m0s\",\"hot-regions-reserved-days\":7,\"max-movable-hot-peer-size\":512,\"enable-diagnostic\":\"true\",\"enable-witness\":\"false\",\"slow-store-evicting-affected-store-ratio-threshold\":0.3,\"store-limit-version\":\"v1\"},\"replication\":{\"max-replicas\":3,\"location-labels\":\"\",\"strictly-match-label\":\"false\",\"enable-placement-rules\":\"true\",\"enable-placement-rules-cache\":\"false\",\"isolation-level\":\"\"},\"pd-server\":{\"use-region-storage\":\"true\",\"max-gap-reset-ts\":\"24h0m0s\",\"key-type\":\"table\",\"runtime-services\":\"\",\"metric-storage\":\"\",\"dashboard-address\":\"auto\",\"trace-region-flow\":\"true\",\"flow-round-by-digit\":3,\"min-resolved-ts-persistence-interval\":\"1s\",\"server-memory-limit\":0,\"server-memory-limit-gc-trigger\":0.7,\"enable-gogc-tuner\":\"false\",\"gc-tuner-threshold\":0.6,\"block-safe-point-v1\":\"false\"},\"cluster-version\":\"0.0.0\",\"labels\":{},\"quota-backend-bytes\":\"8GiB\",\"auto-compaction-mode\":\"periodic\",\"auto-compaction-retention-v2\":\"1h\",\"TickInterval\":\"500ms\",\"ElectionInterval\":\"3s\",\"PreVote\":true,\"max-request-bytes\":157286400,\"security\":{\"cacert-path\":\"\",\"cert-path\":\"\",\"key-path\":\"\",\"cert-allowed-cn\":null,\"SSLCABytes\":null,\"SSLCertBytes\":null,\"SSLKEYBytes\":null,\"redact-info-log\":false,\"encryption\":{\"data-encryption-method\":\"plaintext\",\"data-key-rotation-period\":\"168h0m0s\",\"master-key\":{\"type\":\"plaintext\",\"key-id\":\"\",\"region\":\"\",\"endpoint\":\"\",\"path\":\"\"}}},\"label-property\":null,\"WarningMsgs\":null,\"DisableStrictReconfigCheck\":false,\"HeartbeatStreamBindInterval\":\"1m0s\",\"LeaderPriorityCheckInterval\":\"1m0s\",\"dashboard\":{\"tidb-cacert-path\":\"/var/lib/tidb-client-tls/ca.crt\",\"tidb-cert-path\":\"/var/lib/tidb-client-tls/tls.crt\",\"tidb-key-path\":\"/var/lib/tidb-client-tls/tls.key\",\"public-path-prefix\":\"\",\"internal-proxy\":true,\"enable-telemetry\":false,\"enable-experimental\":false},\"replication-mode\":{\"replication-mode\":\"majority\",\"dr-auto-sync\":{\"label-key\":\"\",\"primary\":\"\",\"dr\":\"\",\"primary-replicas\":0,\"dr-replicas\":0,\"wait-store-timeout\":\"1m0s\",\"pause-region-split\":\"false\"}},\"keyspace\":{\"pre-alloc\":null,\"wait-region-split\":true,\"wait-region-split-timeout\":\"30s\",\"check-region-split-interval\":\"50ms\"},\"controller\":{\"degraded-mode-wait-duration\":\"0s\",\"ltb-max-wait-duration\":\"30s\",\"request-unit\":{\"read-base-cost\":0.125,\"read-per-batch-base-cost\":0.5,\"read-cost-per-byte\":0.0000152587890625,\"write-base-cost\":1,\"write-per-batch-base-cost\":1,\"write-cost-per-byte\":0.0009765625,\"read-cpu-ms-cost\":0.3333333333333333}}}"]

kubectl logs advanced-test-tidb-0 -c tidb -n tidb-cluster |grep tidb-server-tls
[2024/06/06 03:38:53.325 +00:00] [INFO] [printer.go:52] ["loaded config"] [config="{\"host\":\"0.0.0.0\",\"advertise-address\":\"advanced-test-tidb-0.advanced-test-tidb-peer.tidb-cluster.svc\",\"port\":4000,\"cors\":\"\",\"store\":\"tikv\",\"path\":\"advanced-test-pd:2379\",\"socket\":\"/tmp/tidb-4000.sock\",\"lease\":\"45s\",\"split-table\":true,\"token-limit\":1000,\"temp-dir\":\"/tmp/tidb\",\"tmp-storage-path\":\"/tmp/0_tidb/MC4wLjAuMDo0MDAwLzAuMC4wLjA6MTAwODA=/tmp-storage\",\"tmp-storage-quota\":-1,\"server-version\":\"\",\"version-comment\":\"\",\"tidb-edition\":\"\",\"tidb-release-version\":\"\",\"keyspace-name\":\"\",\"log\":{\"level\":\"info\",\"format\":\"text\",\"disable-timestamp\":null,\"enable-timestamp\":null,\"disable-error-stack\":null,\"enable-error-stack\":null,\"file\":{\"filename\":\"\",\"max-size\":300,\"max-days\":0,\"max-backups\":3},\"slow-query-file\":\"/var/log/tidb/slowlog\",\"expensive-threshold\":10000,\"query-log-max-len\":4096,\"enable-slow-log\":true,\"slow-threshold\":0,\"record-plan-in-slow-log\":1,\"timeout\":0},\"instance\":{\"tidb_general_log\":false,\"tidb_pprof_sql_cpu\":false,\"ddl_slow_threshold\":300,\"tidb_expensive_query_time_threshold\":60,\"tidb_expensive_txn_time_threshold\":600,\"tidb_stmt_summary_enable_persistent\":false,\"tidb_stmt_summary_filename\":\"tidb-statements.log\",\"tidb_stmt_summary_file_max_days\":3,\"tidb_stmt_summary_file_max_size\":64,\"tidb_stmt_summary_file_max_backups\":0,\"tidb_enable_slow_log\":true,\"tidb_slow_log_threshold\":0,\"tidb_record_plan_in_slow_log\":1,\"tidb_check_mb4_value_in_utf8\":true,\"tidb_force_priority\":\"NO_PRIORITY\",\"tidb_memory_usage_alarm_ratio\":0.8,\"tidb_enable_collect_execution_info\":true,\"plugin_dir\":\"/data/deploy/plugin\",\"plugin_load\":\"\",\"max_connections\":0,\"tidb_enable_ddl\":true,\"tidb_rc_read_check_ts\":false,\"tidb_service_scope\":\"\"},\"security\":{\"skip-grant-table\":false,\"ssl-ca\":\"/var/lib/tidb-server-tls/ca.crt\",\"ssl-cert\":\"/var/lib/tidb-server-tls/tls.crt\",\"ssl-key\":\"/var/lib/tidb-server-tls/tls.key\",\"cluster-ssl-ca\":\"\",\"cluster-ssl-cert\":\"\",\"cluster-ssl-key\":\"\",\"cluster-verify-cn\":null,\"session-token-signing-cert\":\"\",\"session-token-signing-key\":\"\",\"spilled-file-encryption-method\":\"plaintext\",\"enable-sem\":false,\"auto-tls\":false,\"tls-version\":\"\",\"rsa-key-size\":4096,\"secure-bootstrap\":false,\"auth-token-jwks\":\"\",\"auth-token-refresh-interval\":\"1h0m0s\",\"disconnect-on-expired-password\":true},\"status\":{\"status-host\":\"0.0.0.0\",\"metrics-addr\":\"\",\"status-port\":10080,\"metrics-interval\":15,\"report-status\":true,\"record-db-qps\":false,\"record-db-label\":false,\"grpc-keepalive-time\":10,\"grpc-keepalive-timeout\":3,\"grpc-concurrent-streams\":1024,\"grpc-initial-window-size\":2097152,\"grpc-max-send-msg-size\":2147483647},\"performance\":{\"max-procs\":0,\"max-memory\":0,\"server-memory-quota\":0,\"stats-lease\":\"3s\",\"stmt-count-limit\":5000,\"pseudo-estimate-ratio\":0.8,\"bind-info-lease\":\"3s\",\"txn-entry-size-limit\":6291456,\"txn-total-size-limit\":104857600,\"tcp-keep-alive\":true,\"tcp-no-delay\":true,\"cross-join\":true,\"distinct-agg-push-down\":false,\"projection-push-down\":false,\"max-txn-ttl\":3600000,\"index-usage-sync-lease\":\"0s\",\"plan-replayer-gc-lease\":\"10m\",\"gogc\":100,\"enforce-mpp\":false,\"stats-load-concurrency\":5,\"stats-load-queue-size\":1000,\"analyze-partition-concurrency-quota\":16,\"plan-replayer-dump-worker-concurrency\":1,\"enable-stats-cache-mem-quota\":true,\"committer-concurrency\":128,\"run-auto-analyze\":true,\"force-priority\":\"NO_PRIORITY\",\"memory-usage-alarm-ratio\":0.8,\"enable-load-fmsketch\":false,\"lite-init-stats\":true,\"force-init-stats\":true},\"prepared-plan-cache\":{\"enabled\":true,\"capacity\":100,\"memory-guard-ratio\":0.1},\"opentracing\":{\"enable\":false,\"rpc-metrics\":false,\"sampler\":{\"type\":\"const\",\"param\":1,\"sampling-server-url\":\"\",\"max-operations\":0,\"sampling-refresh-interval\":0},\"reporter\":{\"queue-size\":0,\"buffer-flush-interval\":0,\"log-spans\":false,\"local-agent-host-port\":\"\"}},\"proxy-protocol\":{\"networks\":\"\",\"header-timeout\":5,\"fallbackable\":false},\"pd-client\":{\"pd-server-timeout\":3},\"tikv-client\":{\"grpc-connection-count\":4,\"grpc-keepalive-time\":10,\"grpc-keepalive-timeout\":3,\"grpc-compression-type\":\"none\",\"commit-timeout\":\"41s\",\"async-commit\":{\"keys-limit\":256,\"total-key-size-limit\":4096,\"safe-window\":2000000000,\"allowed-clock-drift\":500000000},\"max-batch-size\":128,\"overload-threshold\":200,\"max-batch-wait-time\":0,\"batch-wait-size\":8,\"enable-chunk-rpc\":true,\"region-cache-ttl\":600,\"store-limit\":0,\"store-liveness-timeout\":\"1s\",\"copr-cache\":{\"capacity-mb\":1000},\"copr-req-timeout\":60000000000,\"ttl-refreshed-txn-size\":33554432,\"resolve-lock-lite-threshold\":16},\"binlog\":{\"enable\":false,\"ignore-error\":false,\"write-timeout\":\"15s\",\"binlog-socket\":\"\",\"strategy\":\"range\"},\"compatible-kill-query\":false,\"pessimistic-txn\":{\"max-retry-count\":256,\"deadlock-history-capacity\":10,\"deadlock-history-collect-retryable\":false,\"pessimistic-auto-commit\":false,\"constraint-check-in-place-pessimistic\":true},\"max-index-length\":3072,\"index-limit\":64,\"table-column-count-limit\":1017,\"graceful-wait-before-shutdown\":0,\"alter-primary-key\":false,\"treat-old-version-utf8-as-utf8mb4\":true,\"enable-table-lock\":false,\"delay-clean-table-lock\":0,\"split-region-max-num\":1000,\"top-sql\":{\"receiver-address\":\"\"},\"repair-mode\":false,\"repair-table-list\":[],\"isolation-read\":{\"engines\":[\"tikv\",\"tiflash\",\"tidb\"]},\"new_collations_enabled_on_first_bootstrap\":true,\"experimental\":{\"allow-expression-index\":false},\"skip-register-to-dashboard\":false,\"enable-telemetry\":false,\"labels\":{},\"enable-global-index\":false,\"deprecate-integer-display-length\":false,\"enable-enum-length-limit\":true,\"stores-refresh-interval\":60,\"enable-tcp4-only\":false,\"enable-forwarding\":false,\"max-ballast-object-size\":0,\"ballast-object-size\":0,\"transaction-summary\":{\"transaction-summary-capacity\":500,\"transaction-id-digest-min-duration\":2147483647},\"enable-global-kill\":true,\"enable-32bits-connection-id\":true,\"initialize-sql-file\":\"\",\"enable-batch-dml\":false,\"mem-quota-query\":1073741824,\"oom-action\":\"cancel\",\"oom-use-tmp-storage\":true,\"check-mb4-value-in-utf8\":true,\"enable-collect-execution-info\":true,\"plugin\":{\"dir\":\"/data/deploy/plugin\",\"load\":\"\"},\"max-server-connections\":0,\"run-ddl\":true,\"disaggregated-tiflash\":false,\"autoscaler-type\":\"aws\",\"autoscaler-addr\":\"tiflash-autoscale-lb.tiflash-autoscale.svc.cluster.local:8081\",\"is-tiflashcompute-fixed-pool\":false,\"autoscaler-cluster-id\":\"\",\"use-autoscaler\":false,\"tidb-max-reuse-chunk\":64,\"tidb-max-reuse-column\":256,\"tidb-enable-exit-check\":false,\"in-mem-slow-query-topn-num\":30,\"in-mem-slow-query-recent-num\":500}"]

MySQL Client Connect to TiDB Cluster with TLS

Save CA/Client Certificate/Private Key

kubectl get secret -n tidb-cluster advanced-test-tidb-client-secret  -ojsonpath='{.data.tls\.crt}' | base64 --decode > client-tls.crt
kubectl get secret -n tidb-cluster advanced-test-tidb-client-secret  -ojsonpath='{.data.tls\.key}' | base64 --decode > client-tls.key
kubectl get secret -n tidb-cluster advanced-test-tidb-client-secret  -ojsonpath='{.data.ca\.crt}'  | base64 --decode > client-ca.crt

Connect with MySQL Client

mysql --comments -uroot -p -P 4000 -h ${tidb_host} --ssl-cert=client-tls.crt --ssl-key=client-tls.key --ssl-ca=client-ca.crt

mysql> SHOW STATUS LIKE "Ssl%";
+-----------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Variable_name         | Value                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
+-----------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Ssl_cipher            | TLS_AES_128_GCM_SHA256                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| Ssl_cipher_list       | RC4-SHA:DES-CBC3-SHA:AES128-SHA:AES256-SHA:AES128-SHA256:AES128-GCM-SHA256:AES256-GCM-SHA384:ECDHE-ECDSA-RC4-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-RC4-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-RSA-AES128-SHA:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-CHACHA20-POLY1305:TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256: |
| Ssl_server_not_after  | Jun  6 02:22:01 2025 UTC                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| Ssl_server_not_before | Jun  6 02:22:01 2024 UTC                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| Ssl_verify_mode       | 5                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| Ssl_version           | TLSv1.3                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
+-----------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
6 rows in set (0.10 sec)

TiDB Operator Enable TLS For MySQL Client With cert-manager was originally published by Sherry Yin at Sherry Yin on June 06, 2024.

S3 Compatible MinIO Operator Deployment

2024-04-12T00:00:00+08:00

Object store service is a common service nowdays, especially S3 compatible. The introduction of MinIO can be found here.

This article will demostrate how to setup MinIO Operator for testing purpose, this is NOT a comprehensive guide for production scenario.

Deploy MinIO Operator
Deploy MinIO Tenant
Create bucket in MinIO Console

Depoly MinIO Operator

We use Helm to deploy the operator.

helm repo add minio-operator https://operator.min.io
helm install --namespace minio-operator  --create-namespace operator minio-operator/operator

Deploy MinIO Tenant

1.Download tenant CR file tenant-values.yaml

curl -sLo tenant-values.yaml https://raw.githubusercontent.com/minio/operator/master/helm/tenant/values.yaml

2.Configure Tenant Update tenant-values.yaml

tenant:
  name: myminio
  image:
    repository: quay.io/minio/minio
    tag: RELEASE.2024-07-16T23-46-41Z
    pullPolicy: IfNotPresent
  imagePullSecret: { }
  scheduler: { }
  configuration:
    name: myminio-env-configuration
    configSecret:
      name: myminio-env-configuration
      accessKey: minio
      secretKey: minio123
  pools:
    - servers: 1
      name: pool-0
      volumesPerServer: 1
      size: 10Gi
      storageAnnotations: { }
      annotations: { }
      labels: { }
      tolerations: [ ]
      nodeSelector: { }
      affinity: { }
      resources: { }
      securityContext:
        runAsUser: 1000
        runAsGroup: 1000
        fsGroup: 1000
        fsGroupChangePolicy: "OnRootMismatch"
        runAsNonRoot: true
      containerSecurityContext:
        runAsUser: 1000
        runAsGroup: 1000
        runAsNonRoot: true
        allowPrivilegeEscalation: false
        capabilities:
          drop:
            - ALL
        seccompProfile:
          type: RuntimeDefault
      topologySpreadConstraints: [ ]
  mountPath: /export
  subPath: /data
  metrics:
    enabled: false
    port: 9000
    protocol: http
  certificate:
    externalCaCertSecret: [ ]
    externalCertSecret: [ ]
    requestAutoCert: true
    certConfig: { }
  features:
    bucketDNS: false
    domains: { }
    enableSFTP: false
  buckets: [ ]
  users: [ ]
  podManagementPolicy: Parallel
  liveness: { }
  readiness: { }
  startup: { }
  lifecycle: { }
  exposeServices: { }
  serviceAccountName: ""
  prometheusOperator: false
  logging: { }
  serviceMetadata: { }
  env: [ ]
  priorityClassName: ""
  additionalVolumes: [ ]
  additionalVolumeMounts: [ ]

ingress:
  api:
    enabled: false
    ingressClassName: ""
    labels: { }
    annotations: { }
    tls: [ ]
    host: minio.local
    path: /
    pathType: Prefix
  console:
    enabled: true
    ingressClassName: ""
    labels: { }
    annotations: { }
    tls: [ ]
    host: minio-console.local
    path: /
    pathType: Prefix

3.Deploy tenant

helm install  --namespace myminio --create-namespace --values tenant-values.yaml myminio minio-operator/tenant

4.Expose console service

kubectl port-forward svc/myminio-console 9443 -n myminio --address=0.0.0.0

5.Access MinIO console and create bucket

Username: minio (accessKey)
Password: minio123 (secretKey)

S3 Compatible MinIO Operator Deployment was originally published by Sherry Yin at Sherry Yin on April 12, 2024.

TiDB Operator Logging EFK Integration

2024-04-07T00:00:00+08:00

❗ This is not a production setup, it’s only for testing and demo purpose

Introduction

Since TiDB components deployed by TiDB Operator output the logs in the stdout and stderr of the container by default, as long as you have EFK deployed in your Kubernetes cluster, all the TiDB components logs will be collected by Fluentd running on each Kubernetes node, you can find TiDB logs same as other Kubernetes serviceslogs.

This article demonstrates how to collect TiDB logs using Fluent-bit in Kubernetes based on EFK, especially for slow logs which need to be properly formated before streaming to EFK, for the EFK deployment please refer to my previous blog Kubernetes Logging EFK Deployment

Enviornments

Fluent-bit v1.8 (needed by multiline parser)
TiDB operator v1.5.2
TiDB v7.5

Step by Step Setup

1. Deploy TiDB Operator

kubectl create -f https://raw.githubusercontent.com/pingcap/tidb-operator/v1.5.2/manifests/crd.yaml
kubectl create namespace tidb-admin
helm repo add pingcap https://charts.pingcap.org/
helm install --namespace tidb-admin tidb-operator pingcap/tidb-operator --version v1.5.2

2. Deploy TiDB Cluster with Fluent-bit

Fluent bit running as sidecar in TiDB pod, you can use Fluentd as well. I choose fluent-bit as it’s light weight.

2.1 Prepare Fluent bit config

For demo purpose, I will manually mount Fluent-bit config. You may customize your image for a clean deployment, as slow query log is multiline format, we need to handle it properly, mutiline format parser is included in the config.

1.Create fluent-bit ConfigMap

kubectl apply -f tidb-fluentbit-cm-slowlog.yaml

tidb-fluentbit-cm-slowlog.yaml can be found here.

2.2 Deploy TiDB Cluster with fluent-bit as sidecar container

Option 1: fluent-bit in additional container

This deployment will keep the default slowlog container, fluent-bit as an additional container, tidb pod in total 3 containers.

kubectl apply -f tidb-advanced-cluster-additional-container.yaml

tidb-advanced-cluster-additional-container.yaml can be found here.

Option2: Fluent-bit Merged with default slowlog container

We customize the default slowlog container to run Fluent bit by merging the default slowlog container and additional container. tidb pod only 2 containers

kubectl apply -f tidb-advanced-cluster-fluentbit-merged-with-slowlog.yaml

tidb-advanced-cluster-fluentbit-merged-with-slowlog.yaml can be found here.

3. View TiDB Slow Query Logs in Kibana

1.You will find tidb-yyyy.mm.dd index

2.Create data view for tidb index

3.View TiDB Slow Logs

4.Formated TiDB Slow Log

TiDB Operator Logging EFK Integration was originally published by Sherry Yin at Sherry Yin on April 07, 2024.

Kubernetes Logging EFK Deployment

2024-04-02T00:00:00+08:00

❗ This is not a production setup, it’s only for testing and demo purpose

Envrionments

minikube v1.32.0
Kubernetes v1.24
ECK v2.12
Elasticsearch + Kibana v8.13.0
Fluentd v1.16

Introduction

This is a simple infra level logging system setup, Fluentd as a daemonset running on each Kubernetes node as an agent to collect all containers’ logs to elasticseach. Again, this is not for production setup, just meet basic log collection requirement, production setup colud be more conprehensive.

A glance of the architecture

Step By Step Setup

1. Prepare Kubernetes

minikube start --cpus=3 --memory=6G --disk-size=25G --kubernetes-version=v1.24

2. Deploy ECK

Refer to official documentation Deploy ECK in your Kubernetes cluster

2.1 Deploy Elastic Operator

1.Create CRD

kubectl create -f https://download.elastic.co/downloads/eck/2.12.1/crds.yaml

2.Create Operator

kubectl apply -f https://download.elastic.co/downloads/eck/2.12.1/operator.yaml

2.2 Deploy Elasticsearch

1.Create namespace eck for ECK

2.Deploy Elaisticsearch

kubectl apply -f elasticsearch.yaml

elasticsearch.yaml can be found here.

Make sure HEALTH is green, and PHASE is Ready.

2.3 Deploy Kibana

kubectl apply -f kibana.yaml

kibana.yaml can be found here.

2.4 Access Kibana

1.Get password

kubectl get secret quickstart-es-elastic-user -n eck -o go-template='{{.data.elastic | base64decode}}'

2.Expose kibana service access from outside Kubernetes

kubectl port-forward service/quickstart-kb-http -n eck 5601 --address=0.0.0.0

3.Login Access Kibana URL https://{your host}:5601/

3. Deploy Fluentd

kubectl apply -f fluentd-daemonset-elasticsearch-rbac.yaml

fluentd-daemonset-elasticsearch-rbac.yaml can be found here.

Replace the environment varibles accordingly, especically FLUENT_ELASTICSEARCH_PASSWORD

4. Create Data View in Kibana

5. View K8S Logs in Kibana

Enjoy EFK!!!

Kubernetes Logging EFK Deployment was originally published by Sherry Yin at Sherry Yin on April 02, 2024.

Rust Begin Here

2023-08-15T00:00:00+08:00

Local Rust Installation

Install

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

More details refer to the guide.

Components installed

cargo: the Rust’s build system and package manager.
rustfmt: a tool for formatting Rust code according to style guidelines.

Hello World!

New

Create a new project with Cargo

cargo new hello_world

Output: “Created binary (application) hello_world package”

A quick glance

cat hello_world/src/main.rs

Output:

fn main() {
    println!("Hello, world!");
}

Build

cd hello_world
cargo build

Output: “binary hello_world was created under target/debug”

Run

target/debug/hello_world

output: “Hello, world!”

cargo run

output:

    Finished dev [unoptimized + debuginfo] target(s) in 0.03s
     Running `target/debug/hello_world`
Hello, world!

Project Directory

tree
.
├── Cargo.lock
├── Cargo.toml
├── src
│   └── main.rs
└── target
    ├── CACHEDIR.TAG
    └── debug
        ├── build
        ├── deps
        │   ├── hello_world-a65d4c6029deb9db
        │   ├── hello_world-a65d4c6029deb9db.2m06qpscokrdldr2.rcgu.o
        │   ├── hello_world-a65d4c6029deb9db.337tnzf71uan4bv8.rcgu.o
        │   ├── hello_world-a65d4c6029deb9db.3lakkzjgkq4s2ufz.rcgu.o
        │   ├── hello_world-a65d4c6029deb9db.3nyq6f5mboo59omw.rcgu.o
        │   ├── hello_world-a65d4c6029deb9db.3txvt6tt1wrkqp6g.rcgu.o
        │   ├── hello_world-a65d4c6029deb9db.d
        │   └── hello_world-a65d4c6029deb9db.okmjk6ajfymokvw.rcgu.o
        ├── examples
        ├── hello_world
        ├── hello_world.d
        └── incremental
            └── hello_world-27bxg3u8cn054
                ├── s-gq9ubsyjer-cu1tnq-8pp2iqcnvjen9b1yyqrks0gg0
                │   ├── 2m06qpscokrdldr2.o
                │   ├── 337tnzf71uan4bv8.o
                │   ├── 3lakkzjgkq4s2ufz.o
                │   ├── 3nyq6f5mboo59omw.o
                │   ├── 3txvt6tt1wrkqp6g.o
                │   ├── dep-graph.bin
                │   ├── okmjk6ajfymokvw.o
                │   ├── query-cache.bin
                │   └── work-products.bin
                └── s-gq9ubsyjer-cu1tnq.lock

10 directories, 24 files

IDE Recommendation

VSCode

Some plugins to recommend:

Rust Syntax: provides a TextMate grammar for Rust.
rust-analyzer: an implementation of Language Server Protocol for the Rust programming language. It provides features like completion and goto definition for many code editors, including VS Code, Emacs and Vim.
crates: helps Rust developers managing dependencies with Cargo.toml.
Even Better Toml: a TOML language support extension.
Rust Test Lens: adds a code lens to quickly run or debug a single test for your Rust code.

RustRover

From JetBrains, currently RustRover is free to use during the public preview, download.

Rust Begin Here was originally published by Sherry Yin at Sherry Yin on August 15, 2023.