How-to rebuild a MongoDB node

When a secondary MongoDB node’s lag is too far to catch up the primary, or we want to shrink the data files, we can just rebuild the node as follows:

1. Make sure majority nodes are working fine if we shutdown the one to be rebuild
2. Shutdown the mongod (For primary: run rs.stepDown() to change role to secondary first)
3. Check the replicat-set status at other node: rs.status(), should have 1 primary
4. Remove all files and directory at dbpath
5. Start mongod and it will recover by itself

If there is a big index, the secondaries will need to rebuild index. It could run too too long time, may never complete as follows:

Tue Jun 26 11:27:20 [rsSync] build index zeng.AlexTable { MD5: 1 }
                   3000000/773362927       0%
                   6000000/773362927       0%
                   9000000/773362927       1%
    ...
                   769775300/773362927   99%
                   773000000/773362927   99%
Tue Jun 26 12:08:09 [rsSync]          external sort used : 774 files  in 2448 secs
                   107600/773362927          0%
                   267900/773362927          0%
                   423800/773362927          0%
…
Tue Jun 26 17:34:06 [initandlisten] connection accepted from 10.254.242.244:52894 #373
Tue Jun 26 17:34:06 [conn373] end connection 10.254.242.244:52894
                   324598300/773362927   41%
                   324763700/773362927   41%

In this case, We can do cold backup to restore the secondary as follows :

1. Shutdown one node (the secondary is preferred, if no secondary in good status, we need to shutdown primary). If after shutdown the node, no primary will be available, we will need to notify customer beforehand.
2. Remove all files all files and directory at dbpath
3. Copy all files from the shutdown node dbpath, the journal directory and mongod.lock is not needed.
4. Start up mongod, check mongod.log to make sure it works

If you copied the mongod.lock from the source, and didn’t delete it before startup the target node, you will get this error in mongod.log:

**************
old lock file: /data/mongo/data/mongod.lock.  probably means unclean shutdown,
but there are no journal files to recover.
this is likely human error or filesystem corruption.
found 4 dbs.
see: http://dochub.mongodb.org/core/repair for more information
*************
Wed Jun 27 23:21:00 [initandlisten] exception in initAndListen: 12596 old lock file, terminating
Wed Jun 27 23:21:00 dbexit:
Wed Jun 27 23:21:00 [initandlisten] shutdown: going to close listening sockets...
Wed Jun 27 23:21:00 [initandlisten] shutdown: going to flush diaglog...
Wed Jun 27 23:21:00 [initandlisten] shutdown: going to close sockets...
Wed Jun 27 23:21:00 [initandlisten] shutdown: waiting for fs preallocator...
Wed Jun 27 23:21:00 [initandlisten] shutdown: lock for final commit...
Wed Jun 27 23:21:00 [initandlisten] shutdown: final commit...
Wed Jun 27 23:21:00 [initandlisten] shutdown: closing all files...
Wed Jun 27 23:21:00 [initandlisten] closeAllFiles() finished
Wed Jun 27 23:21:00 dbexit: really exiting now

For more info, you can reference online MongoDB document expand-replica-set

How-to use scripts to create cacti templates

If you have ever created a cacti templates from scratch, you would know it’s a very tedious work. I am very lucky to find this site. Later it moved to this new place. There are plenty of documents to let us know how it works while I will just share my experience of creating an MongoDB template.

First, create a definition file.

It can be copied from an existing one and change it. Referenceand.

You may need to unique the hash value using this script.

tools/unique-hashes.pl  mongodb_definitions_tmp.pl > mongodb_definitions.pl
tools/unique-hashes.pl  --refresh mongodb_definitions_tmp.pl > mongodb_definitions.pl

Secondly, generate the cacti template.

make-template.pl  --mpds port2 --script /export/home/cacti/scripts/ss_get_mongo_stats.php mongodb_definitions.pl > mongo.xml

Here are the source files(you need to change the file extension name back after download):
mongodb_definitions.pl : this one need to be created manually. It’s not so difficult if you know the structure.
make-template.pl : I added the support of “friend name” to it.
ss_get_mongo_stats.php : this one is changed from ss_get_by_ssh.php. One of the outstanding changes is the change to the map array as follows:

<pre>$keys_map = array(
   'MONGODB_connected_clients'            => array('short' => 'ma',  'source' => 'Connections'),
   'MONGODB_active_sessions'              => array('short' => 'mb',  'source' => 'Active'),
   'MONGODB_used_resident_memory'         => array('short' => 'mc',  'source' => 'residentMem'),
   'MONGODB_used_mapped_memory'           => array('short' => 'md',  'source' => 'mappedMem'),
...

The performance data is from a webpage like:

opInsert=36868243
opQuery=4290620
opUpdate=20708814
opDelete=608893
opGetmore=255241
opCommand=22943868
...

As always, it’s just an example to help you get started. You can change whatever you like once you know it.

How-to restore MongoDB from raw data files

MongoDB is different from the traditional RDBMS database. It provided tools to read from raw data files directly. This is extremely helpful in some cases.

Here is an example:

Scenario: A DB were wrongly dropped by the user. Now the user want to recover that DB from the backup data files. 
1. Dump data from backup data files for the db "testdb"
[mongo@alexzeng bak]$ mongodump -d testdb --dbpath /mongo/data/bak/ -o dump
Mon Jun 11 20:14:25 [tools] query testdb.* ntoreturn:1 reslen:20 4219ms
DATABASE: testdb	 to 	dump/testdb
	testdb.system.indexes to dump/testdb/system.indexes.bson
		 2 objects
	testdb.autofill_hadoop to dump/testdb/autofill_hadoop.bson
		 0 objects
Mon Jun 11 20:14:25 dbexit: 
Mon Jun 11 20:14:25 [tools] shutdown: going to close listening sockets...
Mon Jun 11 20:14:25 [tools] shutdown: going to flush diaglog...
Mon Jun 11 20:14:25 [tools] shutdown: going to close sockets...
Mon Jun 11 20:14:25 [tools] shutdown: waiting for fs preallocator...
Mon Jun 11 20:14:25 [tools] shutdown: closing all files...
Mon Jun 11 20:14:25 [tools] closeAllFiles() finished
Mon Jun 11 20:14:25 [tools] shutdown: removing fs lock...
Mon Jun 11 20:14:25 dbexit: really exiting now

2. Restore the DB "testdb" to the target MongoDB replicat-set.
[mongo@alexzeng bak]$ mongorestore -d testdb -h alexzeng.wordpress.com:27017 --objcheck /mongo/data/bak/dump/testdb 
connected to: alexzeng.wordpress.com:27017
Mon Jun 11 20:27:25 /mongo/data/bak/dump/testdb/autofill_hadoop.bson
Mon Jun 11 20:27:25 	 going into namespace [testdb.autofill_hadoop]
Mon Jun 11 20:27:25 file /mongo/data/bak/dump/testdb/autofill_hadoop.bson empty, skipping
Mon Jun 11 20:27:25 /mongo/data/bak/dump/testdb/system.indexes.bson
Mon Jun 11 20:27:25 	 going into namespace [testdb.system.indexes]
Mon Jun 11 20:27:25 { key: { _id: 1 }, ns: "testdb.autofill_hadoop", name: "_id_" }
Mon Jun 11 20:27:25 { key: { prefix: 1.0 }, ns: "testdb.autofill_hadoop", name: "prefix_1" }
2 objects found