编程知识 cdmana.com

Mongodb learning -- building mongodb shard cluster under Linux

One 、 Get ready

linux Under the specified configuration file installation mongodb

Two 、 The principle of partitioned cluster

  What is segmentation

        Fragmentation (sharding) yes MongoDB Used to divide large collections horizontally into different servers ( Or copy sets ) The method used on the . You don't need a powerful mainframe computer to store more data , Handle larger loads .

Why do you want to slice

1. The storage capacity requirement exceeds the capacity of a single disk .
2. The active data set exceeds the memory capacity of a single machine , This causes many requests to read data from disk , Affect performance .
3.IOPS Beyond single MongoDB The service capability of the node , As the data grows , The bottleneck of single instance will become more and more obvious .
4. Replica sets have a limit on the number of nodes .

Vertical expansion : Add more CPU And storage resources to expand capacity .
Horizontal expansion : Distributing data sets across multiple servers . Horizontal expansion is fragmentation .

How fragmentation works

 

 

 

The fragmentation cluster consists of the following 3 Service composition :
Shards Server: Every shard By one or more mongod Process composition , For storing data .
Router Server: The request entry of the database cluster , All requests go through Router(mongos) Coordination , It doesn't need to be in the application , Add a route selector ,Router(mongos) It is a request distribution center, which is responsible for forwarding application requests to the corresponding Shard Server .
Config Server: Configure server . Store all database meta information ( route 、 Fragmentation ) Configuration of .

Chip key (shard key)

      To assign documents in a data set ,MongoDB Use the partitioned primary key to split the collection

block (chunk)     

In a shard server Inside ,MongoDB Or will you divide the data into chunks, Every chunk On behalf of the shard server Part of the internal data .MongoDB Slice data into blocks , Each block contains an interval range that is left closed and right open based on the partition primary key

Fragmentation strategy

  • Range fragmentation (Range based sharding)

 

 

 

Range slicing is suitable for searching in a certain range , For example, find X The value of the [20,30) Data between ,mongo Routing is based on Config server Metadata stored in , Can be directly located to the specified shard Of Chunk in .

shortcoming : If shard key There is a significant increase ( Or decreasing ) trend , The newly inserted documents will be distributed to the same chunk, Can't extend the ability to write .

  • hash Fragmentation (Hash based sharding)

 

 

 

    Hash Fragmentation is to calculate a partition primary key hash value , Each block will be assigned a range of hash value .Hash Fragmentation and range fragmentation complement each other , It can spread documents randomly to different places chunk, Full extended write ability , Make up for the lack of range segmentation , The disadvantage is that it can't efficiently query the service scope , All range queries should be distributed to all the back end Shard To find the documents that meet the conditions

  • Combination key A + B( Hash thought Not directly hash)

There is no suitable slice key to choose from in the database , The number of keys is too small ( That is to say, the change is less than that of a week 7 The sky can change ), You can select another field to use the combination slice key , You can even add redundant fields . It's generally coarse-grained + Fine grained combination .

3、 ... and 、 Cluster building process

 

 

 

1. To configure And start the config Node cluster

node 1 config-17017.conf

#  Database file location 
dbpath=config/config1
# Log file location 
logpath=config/logs/config1.log
#  Write log as append 
logappend=true
#  Whether to run as daemons 
fork = true
bind_ip=0.0.0.0
port = 17017
#  Represents a configuration server 
configsvr=true
# Configure server replica set name 
replSet=configsvr
#  Database file location 

node 2 config-17018.conf

#  Database file location 
 node 3 config-17019.conf
 Start the configuration node 
 Entering any node mongo shell  And add   Configure node clusters   Be careful use admin
dbpath=config/config2
# Log file location 
logpath=config/logs/config.log
#  Write log as append 
logappend=true
#  Whether to run as daemons 
fork = true
bind_ip=0.0.0.0
port = 17018
#  Represents a configuration server 
configsvr=true
# Configure server replica set name 
replSet=configsvr

node 3 config-17019.conf

#  Database file location 
dbpath=config/config3
# Log file location 
logpath=config/logs/config3.log
#  Write log as append 
logappend=true
#  Whether to run as daemons 
fork = true
bind_ip=0.0.0.0
port = 17019
#  Represents a configuration server 
configsvr=true
# Configure server replica set name 
replSet=configsvr

Start the configuration node

./bin/mongod -f config/config-17017.conf
./bin/mongod -f config/config-17018.conf
./bin/mongod -f config/config-17019.conf

Entering any node mongo shell And add Configure node clusters Be careful use admin

./bin/mongo --port 17017
use admin
var cfg ={"_id":"configsvr",
"members":[
{"_id":1,"host":"192.168.211.133:17017"},
{"_id":2,"host":"192.168.211.133:17018"},
{"_id":3,"host":"192.168.211.133:17019"}]
};
rs.initiate(cfg)

Practice

 

 

mkdir shard_cluster
mv mongodb-linux-x86_64-amazon-3.6.21.tgz  shard_cluster/
cd shard_cluster/
tar -xvf mongodb-linux-x86_64-amazon-3.6.21.tgz
rm -rf mongodb-linux-x86_64-amazon-3.6.21.tgz
mv mongodb-linux-x86_64-amazon-3.6.21 shard_cluster
cd shard_cluster/

 

 

 

[root@VM_0_4_centos shard_cluster]# mkdir config/config1 -p
[root@VM_0_4_centos shard_cluster]# mkdir config/config2 -p
[root@VM_0_4_centos shard_cluster]# mkdir config/config3 -p
[root@VM_0_4_centos shard_cluster]# mkdir config/logs/ -p

 

 

  New node profile , And modify in turn 10717、17018、17019 Data location , And port

[root@VM_0_4_centos shard_cluster]# vi mongo_17017.conf
[root@VM_0_4_centos shard_cluster]# cp mongo_17017.conf mongo_17018.conf 
[root@VM_0_4_centos shard_cluster]# cp mongo_17017.conf mongo_17019.conf 

 

 

 

 

 

  Start each node in turn

 

 

 

 

View configuration node information

configsvr:SECONDARY> rs.status()
{
    "set" : "configsvr",
    "date" : ISODate("2020-12-24T07:37:22.074Z"),
    "myState" : 1,
    "term" : NumberLong(1),
    "syncingTo" : "",
    "syncSourceHost" : "",
    "syncSourceId" : -1,
    "configsvr" : true,
    "heartbeatIntervalMillis" : NumberLong(2000),
    "optimes" : {
        "lastCommittedOpTime" : {
            "ts" : Timestamp(1608795438, 1),
            "t" : NumberLong(1)
        },
        "readConcernMajorityOpTime" : {
            "ts" : Timestamp(1608795438, 1),
            "t" : NumberLong(1)
        },
        "appliedOpTime" : {
            "ts" : Timestamp(1608795438, 1),
            "t" : NumberLong(1)
        },
        "durableOpTime" : {
            "ts" : Timestamp(1608795438, 1),
            "t" : NumberLong(1)
        }
    },
    "members" : [
        {
            "_id" : 1,
            "name" : "152.136.193.58:17017",
            "health" : 1,
            "state" : 1,
            "stateStr" : "PRIMARY",
            "uptime" : 455,
            "optime" : {
                "ts" : Timestamp(1608795438, 1),
                "t" : NumberLong(1)
            },
            "optimeDate" : ISODate("2020-12-24T07:37:18Z"),
            "syncingTo" : "",
            "syncSourceHost" : "",
            "syncSourceId" : -1,
            "infoMessage" : "could not find member to sync from",
            "electionTime" : Timestamp(1608795352, 1),
            "electionDate" : ISODate("2020-12-24T07:35:52Z"),
            "configVersion" : 1,
            "self" : true,
            "lastHeartbeatMessage" : ""
        },
        {
            "_id" : 2,
            "name" : "152.136.193.58:17018",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 99,
            "optime" : {
                "ts" : Timestamp(1608795438, 1),
                "t" : NumberLong(1)
            },
            "optimeDurable" : {
                "ts" : Timestamp(1608795438, 1),
                "t" : NumberLong(1)
            },
            "optimeDate" : ISODate("2020-12-24T07:37:18Z"),
            "optimeDurableDate" : ISODate("2020-12-24T07:37:18Z"),
            "lastHeartbeat" : ISODate("2020-12-24T07:37:20.977Z"),
            "lastHeartbeatRecv" : ISODate("2020-12-24T07:37:21.680Z"),
            "pingMs" : NumberLong(0),
            "lastHeartbeatMessage" : "",
            "syncingTo" : "152.136.193.58:17017",
            "syncSourceHost" : "152.136.193.58:17017",
            "syncSourceId" : 1,
            "infoMessage" : "",
            "configVersion" : 1
        },
        {
            "_id" : 3,
            "name" : "152.136.193.58:17019",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 99,
            "optime" : {
                "ts" : Timestamp(1608795438, 1),
                "t" : NumberLong(1)
            },
            "optimeDurable" : {
                "ts" : Timestamp(1608795438, 1),
                "t" : NumberLong(1)
            },
            "optimeDate" : ISODate("2020-12-24T07:37:18Z"),
            "optimeDurableDate" : ISODate("2020-12-24T07:37:18Z"),
            "lastHeartbeat" : ISODate("2020-12-24T07:37:20.978Z"),
            "lastHeartbeatRecv" : ISODate("2020-12-24T07:37:21.674Z"),
            "pingMs" : NumberLong(0),
            "lastHeartbeatMessage" : "",
            "syncingTo" : "152.136.193.58:17017",
            "syncSourceHost" : "152.136.193.58:17017",
            "syncSourceId" : 1,
            "infoMessage" : "",
            "configVersion" : 1
        }
    ],
    "ok" : 1,
    "operationTime" : Timestamp(1608795438, 1),
    "$gleStats" : {
        "lastOpTime" : Timestamp(1608795342, 1),
        "electionId" : ObjectId("7fffffff0000000000000001")
    },
    "$clusterTime" : {
        "clusterTime" : Timestamp(1608795438, 1),
        "signature" : {
            "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
            "keyId" : NumberLong(0)
        }
    }
}
configsvr:PRIMARY> 
View Code

The above configuration node is completed

2. To configure shard colony

[root@VM_0_4_centos shard1]# mkdir shard1-37017 shard1-37018 shard1-37019
[root@VM_0_4_centos shard1]# mkdir logs
[root@VM_0_4_centos shard1]# vi shard1_37017.conf

[root@VM_0_4_centos shard1]# cp shard1_37017.conf shard1_37018.conf
[root@VM_0_4_centos shard1]# cp shard1_37018.conf shard1_37019.conf

shard1 Cluster building 37017 To 37019

dbpath=shard/shard1/shard1-37017
bind_ip=0.0.0.0
port=37017
fork=true
logpath=shard/shard1/shard1-37017.log
replSet=shard1
shardsvr=true


dbpath=shard/shard1/shard1-37018 bind_ip=0.0.0.0 port=37018 fork=true logpath=shard/shard1/logs/shard1-37018.log replSet=shard1 shardsvr=true


dbpath=shard/shard1/shard1-37019 bind_ip=0.0.0.0 port=37019 fork=true logpath=shard/shard1/logs/shard1-37019.log replSet=shard1 shardsvr=true
 Start each mongod  Then go to one of them for cluster configuration ( If you've built a replica set before replica_sets  Please stop first )

var cfg ={"_id":"shard1",
"protocolVersion" : 1,
"members":[
{"_id":1,"host":"152.136.193.58:37017"},
{"_id":2,"host":"152.136.193.58:37018"},
{"_id":3,"host":"152.136.193.58:37019"}
]
};
rs.initiate(cfg)
rs.status()

 

 

 

 

 

 

 

 

  shard1 Completion of construction

shard2 Cluster building 47017 To 47019

dbpath=shard/shard2/shard2-47017
bind_ip=0.0.0.0
port=47017
fork=true
logpath=shard/shard2/logs/shard2-47017.log
replSet=shard2
shardsvr=true


dbpath=shard/shard2/shard2-47018
bind_ip=0.0.0.0
port=47018
fork=true
logpath=shard/shard2/logs/shard2-47018.log
replSet=shard2
shardsvr=true


dbpath=shard/shard2/shard2-47019
bind_ip=0.0.0.0
port=47019
fork=true
logpath=shard/shard2/logs/shard2-47019.log
replSet=shard2
shardsvr=true
var cfg ={"_id":"shard2",
"protocolVersion" : 1,
"members":[
{"_id":1,"host":"152.136.193.58:47017"},
{"_id":2,"host":"152.136.193.58:47018"},
{"_id":3,"host":"152.136.193.58:47019"}
]
};
rs.initiate(cfg)
rs.status()

  The process of practice , reference shard1 The process

3. Configure and start Routing node

route-27017.conf

port=27017
bind_ip=0.0.0.0
fork=true
logpath=route/logs/route.log
configdb=configsvr/192.168.211.133:17017,192.168.211.133:17018,192.168.211.133:1
7019

Start routing node usage mongos ( Notice that it's not mongod)

./bin/mongos -f route/route-27017.conf

 

 

 

4. mongos( route ) Add a fragment node in the

Access route mongos

mongo --port 27017
sh.status()
sh.addShard("shard1/152.136.193.58:37017,152.136.193.58:37018,152.136.193.58:37019");
sh.addShard("shard2/152.136.193.58:47017,152.136.193.58:47018,152.136.193.58:47019");
sh.status()

 

 

 

 

 

 

5. Open database and collection sharding ( Specify the slice key )

      Continue to use mongos Complete the slice opening and slice size setting ( Routing node )

 Enable fragmentation for the database 
sh.enableSharding("wg_test")
 Turn on the fragmentation function for the specified set . Use  hash  Fragmentation 
sh.shardCollection("wg_test.t_datas",{"name":"hashed"})

6. Insert data into the collection to test

Add a number to the collection through a routing loop

use wg_test;
for(var i=1;i<= 1000;i++){
db.t_datas.insert({"name":"test"+i,
salary:(Math.random()*20000).toFixed(2)});
}

7. Verify the fragmentation effect

Respectively into the shard1 and shard2 Database in To verify

 

 

 

                          

 

                                                         down...............................................................ao li  gei

 

版权声明
本文为[VIP 灬 cnblog]所创,转载请带上原文链接,感谢
https://cdmana.com/2020/12/20201224172221557n.html

Scroll to Top