0x00 前言
mongodb是一款基于分布式文件存储的数据库，具有高性能、可扩展、易部署、易使用等特点。官方也提供了丰富的命令行工具来操作。
0x01 部署mongodb服务
部署mongodb服务可以直接使用docker镜像：
$ docker search mongo
NAME DESCRIPTION STARS OFFICIAL AUTOMATED
mongo MongoDB document databases provide high avai… 6656 [OK]
mongo-express Web-based MongoDB admin interface, written w… 640 [OK]
tutum/mongodb MongoDB Docker image – listens in port 27017… 230 [OK]
bitnami/mongodb Bitnami MongoDB Docker Image 109 [OK]
mongoclient/mongoclient Official docker image for Mongoclient, featu… 79 [OK]
mongooseim/mongooseim Small docker image for MongooseIM - robust a… 19
frodenas/mongodb A Docker Image for MongoDB 18 [OK]
cvallance/mongo-k8s-sidecar Kubernetes side car to setup and maintain a … 14 [OK]
centos/mongodb-32-centos7 MongoDB NoSQL database server 8
circleci/mongo CircleCI images for MongoDB 8 [OK]
arm64v8/mongo MongoDB document databases provide high avai… 7
istepanov/mongodump Docker image with mongodump running as a cro… 6 [OK]
centos/mongodb-36-centos7 MongoDB NoSQL database server 5
eses/mongodb_exporter mongodb exporter for prometheus 5 [OK]
centos/mongodb-26-centos7 MongoDB NoSQL database server 5
webhippie/mongodb Docker images for MongoDB 4 [OK]
requilence/mongodb-backup mongo backup container 4 [OK]
centos/mongodb-34-centos7 MongoDB NoSQL database server 3
neowaylabs/mongodb-mms-agent This Docker image with MongoDB Monitoring Ag… 3 [OK]
ansibleplaybookbundle/mongodb-apb An APB to deploy MongoDB. 1 [OK]
ekesken/mongo docker image for mongo that is configurable … 1 [OK]
andreasleicher/mongo-azure-backup a docker container to backup a mongodb using… 1 [OK]
openshift/mongodb-24-centos7 DEPRECATED: A Centos7 based MongoDB v2.4 ima… 1
ccitest/mongo CircleCI test images for Mongo 0 [OK]
targetprocess/mongodb_exporter MongoDB exporter for prometheus 0 [OK]
$ docker pull mongo:latest
$ docker run -itd --name mongo -p 27017:27017 mongo --auth
此时，mongodb服务已经正常运行起来了，还需要配置下账号。
使用docker exec -it mongo bash命令进入到shell环境后，输入mongo命令进入交互式命令行：
> use admin
switched to db admin
> db.createUser({ user: "root", pwd: "root", roles: [{ role: "root", db: "admin" }] })
Successfully added user: {
"user" : "root",
"roles" : [
{
"role" : "root",
"db" : "admin"
}
]
}
权限角色说明：
Read：允许用户读取指定数据库
readWrite：允许用户读写指定数据库
dbAdmin：允许用户在指定数据库中执行管理函数，如索引创建、删除，查看统计或访问system.profile
userAdmin：允许用户向system.users集合写入，可以找指定数据库里创建、删除和管理用户
clusterAdmin：只在admin数据库中可用，赋予用户所有分片和复制集相关函数的管理权限
readAnyDatabase：只在admin数据库中可用，赋予用户所有数据库的读权限
readWriteAnyDatabase：只在admin数据库中可用，赋予用户所有数据库的读写权限
userAdminAnyDatabase：只在admin数据库中可用，赋予用户所有数据库的userAdmin权限
dbAdminAnyDatabase：只在admin数据库中可用，赋予用户所有数据库的dbAdmin权限
root：只在admin数据库中可用。超级账号，超级权限
0x02 安装mongo客户端
$ apt install mongodb-clients
进入mongo shell：
$ mongo ${mongo_server}:${mongo_port} -u ${username} -p ${password} --authenticationDatabase admin
mongodb中分为不同的数据库，默认数据库为test，使用use db切换数据库时，如果数据库不存在会自动创建。
0x03 常用mongo命令
> show dbs
admin 0.000GB
config 0.000GB
local 0.000GB
> use test
switched to db test
> db
test
> db.table.insert({"title":"drunkdream", "body": "Hello world"})
WriteResult({ "nInserted" : 1 })
> db.table.count()
1
> db.table.dataSize()
66
> db.table.insert({"title":"drunkdream123", "body": "Test"})
WriteResult({ "nInserted" : 1 })
> db.table.find().skip(1).limit(1)
{ "_id" : ObjectId("5e706e4651e5f0d32c1eb3e0"), "title" : "drunkdream123", "body" : "Test" }
> db.table.find({title: /^drunkdream/})
{ "_id" : ObjectId("5e706a7051e5f0d32c1eb3df"), "title" : "drunkdream", "body" : "Hello world" }
{ "_id" : ObjectId("5e706e4651e5f0d32c1eb3e0"), "title" : "drunkdream123", "body" : "Test" }
> db.getCollectionNames()
[ "table" ]
> db.printCollectionStats()
table
{
"ns" : "test.table",
"size" : 66,
"count" : 1,
"avgObjSize" : 66,
"storageSize" : 20480,
"capped" : false,
...
}
> db.stats()
{
"db" : "test",
"collections" : 1,
"views" : 0,
"objects" : 1,
"avgObjSize" : 66,
"dataSize" : 66,
"storageSize" : 4096,
"numExtents" : 0,
"indexes" : 1,
"indexSize" : 4096,
"scaleFactor" : 1,
"fsUsedSize" : 34517733376,
"fsTotalSize" : 109059317760,
"ok" : 1
}
find命令支持的查询条件：
$gt: >
$lt: <
$gte: >=
$lte: <=
$ne: !=
例如：{age: {$gte: 23, $lte: 26}}
0x04 其它mongo命令
db.table.drop()- 删除集合db.dropDatabase()- 删除当前数据库db.repairDatabase()- 修复数据库db.getPrevError()- 查询之前的错误信息
0x05 迁移数据
mongodump
$ mongodump -h ${mongo_server} --port ${mongo_port} -d ${db_name} -o ${save_dir} -u ${username} -p ${password} --authenticationDatabase admin
该命令可以将整个数据库dump到本地，如果只需要dump其中的一个集合，可以使用以下参数：--collection ${coll_name}。
mongorestore
$ mongorestore -h ${mongo_server} --port ${mongo_port} -d ${db_name} ${save_dir}/${db_name} -u ${username} -p ${password} --authenticationDatabase admin
该命令可以将备份下来的数据库还原到目标数据库中，修改-d参数可以修改目标数据库的名称。
如果目标数据库存在，需要增加--drop参数进行还原，避免报错。
迁移脚本
写了个简单的迁移脚本：
# -*- coding: utf-8 -*-
import argparse
import os
import tempfile
import time
def sync_mongo(src_host, src_port, dst_host, dst_port, db, collection=None, username='root', password='root', drop=False):
time0 = time.time()
backup_dir = tempfile.mkdtemp('.backup').split(os.path.sep)[-1]
auth_cmdline = ' -u %s -p %s --authenticationDatabase admin' % (username, password)
cmdline = 'mongodump -h %s --port=%d -d %s -o %s' % (src_host, src_port, db, backup_dir)
if collection:
cmdline += ' --collection %s' % collection
cmdline += auth_cmdline
print(cmdline)
os.system(cmdline)
time1 = time.time()
print('backup %s from %s complete, cost %fs\n' % (db, src_host, time1 - time0))
cmdline = 'mongorestore -h %s --port=%d -d %s %s/%s' % (dst_host, dst_port, db, backup_dir, db)
if drop:
cmdline += ' --drop'
cmdline += auth_cmdline
print(cmdline)
os.system(cmdline)
time1 = time.time()
print('sync data from %s/%s to %s/%s complete, total cost %fs' % (src_host, db, dst_host, db, time1 - time0))
if __name__ == '__main__':
parser = argparse.ArgumentParser(description='Mongodb sync data script.')
parser.add_argument('src_host', help='source host')
parser.add_argument('dst_host', help='destination host')
parser.add_argument('db_name', help='database name')
parser.add_argument('--src-port', type=int, help='source port', default=27017)
parser.add_argument('--dst-port', type=int, help='destination port', default=27017)
parser.add_argument('--collection', help='collection to sync')
parser.add_argument('--drop', help='drop collection before restore', default=False, action='store_true')
args = parser.parse_args()
sync_mongo(args.src_host, args.src_port, args.dst_host, args.dst_port, args.db_name, args.collection, drop=args.drop)
增量备份
oplog是local数据库中的一个固定集合，Secondary就是通过查看Primary的oplog这个集合来进行复制的。每个节点都有oplog，记录这从主节点复制过来的信息，这样每个成员都可以作为同步源给其他节点。
FROM mongo:3.6.3
RUN printf "mongod --replSet rs0 &\nsleep 2\nmongo local --eval \"rs.initiate()\"\nmongo local --eval \"rs.conf()\"\nsleep infinity" > /usr/local/bin/init-mongo.sh
ENTRYPOINT "sh /usr/local/bin/init-mongo.sh"
可以使用上面的Dockerfile生成一个自动开启oplog的容器。
rs0:PRIMARY> use local
switched to db local
rs0:PRIMARY> db.getCollectionNames()
[
"me",
"oplog.rs",
"replset.election",
"replset.minvalid",
"startup_log",
"system.replset",
"system.rollback.id"
]
rs0:PRIMARY> db.oplog.rs.find()
{ "ts" : Timestamp(1584972071, 1), "h" : NumberLong("7155205047322550757"), "v" : 2, "op" : "n", "ns" : "", "wall" : ISODate("2020-03-23T14:01:11.490Z"), "o" : { "msg" : "initiating set" } }
{ "ts" : Timestamp(1584972073, 1), "t" : NumberLong(1), "h" : NumberLong("3098310879886849971"), "v" : 2, "op" : "n", "ns" : "", "wall" : ISODate("2020-03-23T14:01:13.536Z"), "o" : { "msg" : "new primary" } }
{ "ts" : Timestamp(1584972073, 2), "t" : NumberLong(1), "h" : NumberLong("7423790971296558722"), "v" : 2, "op" : "c", "ns" : "config.$cmd", "ui" : UUID("ac77fd00-b096-4062-b7ca-ac7f187c1fc7"), "wall" : ISODate("2020-03-23T14:01:13.576Z"), "o" : { "create" : "transactions", "idIndex" : { "v" : 2, "key" : { "_id" : 1 }, "name" : "_id_", "ns" : "config.transactions" } } }
{ "ts" : Timestamp(1584972073, 3), "t" : NumberLong(1), "h" : NumberLong("-2051859930978240465"), "v" : 2, "op" : "c", "ns" : "admin.$cmd", "ui" : UUID("4a0104be-5d4f-49db-9204-d778d83c64dd"), "wall" : ISODate("2020-03-23T14:01:13.606Z"), "o" : { "create" : "system.keys", "idIndex" : { "v" : 2, "key" : { "_id" : 1 }, "name" : "_id_", "ns" : "admin.system.keys" } } }
{ "ts" : Timestamp(1584972073, 4), "t" : NumberLong(1), "h" : NumberLong("-4130342990996856185"), "v" : 2, "op" : "i", "ns" : "admin.system.keys", "ui" : UUID("4a0104be-5d4f-49db-9204-d778d83c64dd"), "wall" : ISODate("2020-03-23T14:01:13.606Z"), "o" : { "_id" : NumberLong("6807403218608324609"), "purpose" : "HMAC", "key" : BinData(0,"h/kcD++ijF/AUzwUqqXHTKG4Jwk="), "expiresAt" : Timestamp(1592748073, 0) } }
{ "ts" : Timestamp(1584972073, 5), "t" : NumberLong(1), "h" : NumberLong("3814221418240704217"), "v" : 2, "op" : "i", "ns" : "admin.system.keys", "ui" : UUID("4a0104be-5d4f-49db-9204-d778d83c64dd"), "wall" : ISODate("2020-03-23T14:01:13.606Z"), "o" : { "_id" : NumberLong("6807403218608324610"), "purpose" : "HMAC", "key" : BinData(0,"4Ds1uk/bTNYytP0hMkXX9ilgz38="), "expiresAt" : Timestamp(1600524073, 0) } }
{ "ts" : Timestamp(1584972093, 1), "t" : NumberLong(1), "h" : NumberLong("7954269847273383577"), "v" : 2, "op" : "n", "ns" : "", "wall" : ISODate("2020-03-23T14:01:33.576Z"), "o" : { "msg" : "periodic noop" } }
{ "ts" : Timestamp(1584972103, 1), "t" : NumberLong(1), "h" : NumberLong("1461838052525134608"), "v" : 2, "op" : "n", "ns" : "", "wall" : ISODate("2020-03-23T14:01:43.576Z"), "o" : { "msg" : "periodic noop" } }
{ "ts" : Timestamp(1584972113, 1), "t" : NumberLong(1), "h" : NumberLong("-2878317090536866771"), "v" : 2, "op" : "n", "ns" : "", "wall" : ISODate("2020-03-23T14:01:53.576Z"), "o" : { "msg" : "periodic noop" } }
{ "ts" : Timestamp(1584972123, 1), "t" : NumberLong(1), "h" : NumberLong("8386919002466516906"), "v" : 2, "op" : "n", "ns" : "", "wall" : ISODate("2020-03-23T14:02:03.577Z"), "o" : { "msg" : "periodic noop" } }
{ "ts" : Timestamp(1584972133, 1), "t" : NumberLong(1), "h" : NumberLong("3752867656297542596"), "v" : 2, "op" : "n", "ns" : "", "wall" : ISODate("2020-03-23T14:02:13.577Z"), "o" : { "msg" : "periodic noop" } }
{ "ts" : Timestamp(1584972143, 1), "t" : NumberLong(1), "h" : NumberLong("-6914327186052916342"), "v" : 2, "op" : "n", "ns" : "", "wall" : ISODate("2020-03-23T14:02:23.577Z"), "o" : { "msg" : "periodic noop" } }
{ "ts" : Timestamp(1584972153, 1), "t" : NumberLong(1), "h" : NumberLong("4905151581311694538"), "v" : 2, "op" : "n", "ns" : "", "wall" : ISODate("2020-03-23T14:02:33.577Z"), "o" : { "msg" : "periodic noop" } }
{ "ts" : Timestamp(1584972163, 1), "t" : NumberLong(1), "h" : NumberLong("2195348018262890539"), "v" : 2, "op" : "n", "ns" : "", "wall" : ISODate("2020-03-23T14:02:43.577Z"), "o" : { "msg" : "periodic noop" } }
{ "ts" : Timestamp(1584972173, 1), "t" : NumberLong(1), "h" : NumberLong("-945081611143254792"), "v" : 2, "op" : "n", "ns" : "", "wall" : ISODate("2020-03-23T14:02:53.577Z"), "o" : { "msg" : "periodic noop" } }
{ "ts" : Timestamp(1584972183, 1), "t" : NumberLong(1), "h" : NumberLong("-8550993957527781683"), "v" : 2, "op" : "n", "ns" : "", "wall" : ISODate("2020-03-23T14:03:03.577Z"), "o" : { "msg" : "periodic noop" } }
{ "ts" : Timestamp(1584972193, 1), "t" : NumberLong(1), "h" : NumberLong("2027336719720729041"), "v" : 2, "op" : "n", "ns" : "", "wall" : ISODate("2020-03-23T14:03:13.578Z"), "o" : { "msg" : "periodic noop" } }
{ "ts" : Timestamp(1584972203, 1), "t" : NumberLong(1), "h" : NumberLong("8383046467552376073"), "v" : 2, "op" : "n", "ns" : "", "wall" : ISODate("2020-03-23T14:03:23.578Z"), "o" : { "msg" : "periodic noop" } }
{ "ts" : Timestamp(1584972213, 1), "t" : NumberLong(1), "h" : NumberLong("-2718017147947765099"), "v" : 2, "op" : "n", "ns" : "", "wall" : ISODate("2020-03-23T14:03:33.578Z"), "o" : { "msg" : "periodic noop" } }
{ "ts" : Timestamp(1584972223, 1), "t" : NumberLong(1), "h" : NumberLong("4265940161146074795"), "v" : 2, "op" : "n", "ns" : "", "wall" : ISODate("2020-03-23T14:03:43.578Z"), "o" : { "msg" : "periodic noop" } }
Type "it" for more
可以看到，oplog.rs集合里已经有数据了。
备份数据命令：
$ mongodump -h ${mongo_server} --port ${mongo_port} -o ${save_dir} -u ${username} -p ${password} --authenticationDatabase admin --oplog
使用
--oplog参数时不能指定数据库，只能全量备份
还原数据命令：
$ mongorestore -h ${mongo_server} --port ${mongo_port} -u ${username} -p ${password} --authenticationDatabase admin --oplogReplay --dir ${save_dir}