How-to write Python for MongoDB

First of all, you need to install the Python driver for MongoDB module, pymongo. The easiest way is to use pip or easy_install to install it.

$ pip install pymongo
or
$ easy_install pymongo

Then, you can start to write scripts. Here is an example:

#!/usr/bin/python
'''
Create a user
'''
import pymongo
from pymongo import Connection
from pymongo import master_slave_connection
from pymongo import database
from pymongo import cursor


    def __do_create_mongouser(self):
        isSuccess = False
        
        mongo_user = self.newuser
        mongo_pass = self.newuser
        mongo_db = self.newuser
        
        
        try:
            #connect to mongodb andget authenticated

            '''
                before connect, modify the connect string as:
                "vip:port,vip2:port..." so that this can be used in Connection()
                the vip str should be like "vip1,vip2,vip3",
                and we regard all port are the same.
            '''
            vip_list = self.vip.split(',')
            con_str = 'mongodb://'
            for item in vip_list:
                con_str += item + ':%d,' % self.port 
            con = con_str[:-1]
            c = Connection(con, replicaset=self.sid)

            db = database.Database(c, self.sysuser)
            db.authenticate(self.sysuser,self.syspass)
            
            #check if user exist: by check if database name exist(database name and username are the same) 
            database_names = c.database_names()
            #if "u'"+self.newuser+"'" in database_names:
            if mongo_user in database_names:
                raise DbUserCreateError('DbUserCreateError: In Remote Oracle DB User [' + 
                                        mongo_user + '] Already Exists.')
            db_new = database.Database(c, mongo_db)
            db_new.add_user(mongo_user, mongo_pass)
        #except pymongo.errors.AutoReconnect as e:
            #auto reconnect once ... skip
        except pymongo.errors.PyMongoError as e:
            
            self.logger.error(e)
            raise
        else:
            isSuccess = True
            self.logger.info('Create Mongo User&DB[' + mongo_user + '] Success!')
        finally:
            try:
                c.disconnect()
            except Exception as e:
                self.logger.error(e)
        
        return { 
                'username': mongo_user,
                'password': mongo_pass,
                'db': mongo_db,
                'success' : isSuccess ,
               }


    def __do_drop_mongouser(self, db_item):
        sysuser = db_item['admin_user']
        syspass = db_item['admin_passwd']
        vip = db_item['vip']
        port = db_item['db_port']
        sid = db_item['sid']
        olduser = db_item['olduser']

        try:
            '''
                before connect, modify the connect string as:
                "vip:port,vip2:port..." so that this can be used in Connection()
                the vip str should be like "vip1,vip2,vip3",
                and we regard all port are the same.
            '''
            vip_list = vip.split(',')
            con_str = 'mongodb://'
            for item in vip_list:
                con_str += item + ':%d,' % port 
            con = con_str[:-1]

            c = Connection(con, replicaset=sid)
            #c = Connection(vip,int(port))
            db = database.Database(c, sysuser)
            db.authenticate(sysuser,syspass)

            #1 drop mongo user. don't need to check exist or not as MongoDB can handle it implicitly
            db_old = database.Database(c, olduser)
            db_old.remove_user(olduser)
        except pymongo.errors.AutoReconnect as e:
            #auto reconnect once

            c = Connection(con, replicaset=sid)
            #c = Connection(vip,int(port))
            db = database.Database(c, sysuser)
            db.authenticate(sysuser,syspass)

            #1 drop mongo user.
            db_old = database.Database(c, olduser)
            db_old.remove_user(olduser)
        except pymongo.errors.PyMongoError as e:
            
            self.logger.error(e)
            raise DbUserDropError(str(e)) 
        else:
            self.logger.info('Drop Mongo User [' + olduser + '] Success!')

        #2 drop mongo database.
        try:
            c.drop_database(olduser)
        except TypeError as e:
            raise DbUserDropError("MongoDbUserDropError:" + e.message)
        else:
            self.logger.info('Drop Mongo DB [' + olduser + '] Success!')
        finally:
            c.disconnect()

Of course, it’s a quite simple example. But it’s a good and great start 🙂

How-to use cacti to monitor Cassandra

I’d like to share my experiences to setup monitor for Cassandra in cacti.

The template for Cassandra is based on mysql-cacti-templates and Cacti Templates for Cassandra. The setup steps are listed in the README file of the package.

The original template is based on Cassandra 0.6.8 while I need to monitor Cassandra 0.8. So there are some problems:

Fixed issues:
1. function array_fill_keys bug in ss_get_cassandra_stats.php
My php is 5.1.6 while this function array_fill_keys is only available after 5.2.0. So there is a replacement in this file. But there is a deadly bug. It should use “$v” as key, not “$k”. Finding this bug took me a lot of time, while it’s easy to fix:

if( !function_exists('array_fill_keys') ) {
  function array_fill_keys($array, $value) {
    $r = array();
    foreach( $array as $k => $v ) {
      //$r[$k] = $value;
      $r[$v] = $value;          //alex
    }
    return $r;
  }
}

2. function to_shortname issue in ss_get_cassandra_stats.php
Cassandra’s JMX property name/items are changed a lot. This makes the to_shortname function stop work for many items. It’s too tedious and not necessary to add all of them. So I added 3 lines to ignore them, by setting all unknown items to ‘xx’.

  //alex, to ignore not defined keys in cassandra_keys, map all of them to xx
  if(!isset($cassandra_keys[$pfx ."_$name"]) ) {
    return 'xx';
  }
  return $cassandra_keys[$pfx ."_$name"];

Some of them are just changed name, so I need to update keys in $cassandra_keys. Here is the updated file ss_get_cassandra_stats.php

3. function get_stats_cache bug in ss_get_cassandra_stats.php
Different hosts in one cluster should not shared the same cache file. So I added the host name to the cache file.

  //list($fp, $content) = check_cache($cache_dir, $poll_time, 'cassandra_'. $options['cluster'], $options);     #alex, should not shared between hosts
  list($fp, $content) = check_cache($cache_dir, $poll_time, 'cassandra_'. $options['cluster'] . '_' . $options['host'] . '_stats', $options);

4. Cassandra’s JMX domain name changed in ss_get_cassandra_stats.php
‘org.apache.cassandra.service’ changed to ‘org.apache.cassandra.db’
‘org.apache.cassandra.concurrent’ changed to ‘org.apache.cassandra.request’

5. Finally, there are too many graphs in the templates
It can add graphs for each keyspace or column family. Also many metrics don’t exist anymore. That will be too much for our cacti system as we get data every 1 minute. So I deleted many graphs from the template, this is the updated file cacti_host_template_x_cassandra_server_ht.xml.

Besides these solved issues, there are some remained :
1. Performance: after I added a 4-node Cassandra cluster to our cacti system, the poller time jumped from 30 seconds to 50 seconds. This make it very near to 1 minute.
2. Cassandra’s JMX property name/items changes a lot. It’s not surprise that it will continue changing in coming versions. We need to take lots of efforts to maintenance this templates.
3. Some metrics cannot found in JMX while it’s easy to get by Cassandra’s nodetool, and it’s almost not possible to add metrics that not in JMX.

So I may use home-grown scripts to create templates later, like what we did for mongoDB.

How-to use cacti monitor db

1. Summary

Cacti is an open source graphic tools based on templates. Once you have a template, you can add new host in a few clicks. Reference: http://www.cacti.net/ This blog is based on cacti 0.8.7g.

2. Installation guide

3. Configure view graph without login

1. Create or enable a user, for example Guest.
Go to Console -> User Management, enable use Guest.
Click Guest, set password and don't check "User Must Change Password at Next Login",
Check "View Graphs" in "Realm permissions" section, Click Save.
Now the user Guest can login to view graphs

2. Config the use as Guest user.
Go to the Settings in Configurations section -> Authentication tab -> 
Set Guest User to "guest", click Save.
Now you can open the graphs page without login.

3. Add a link to login page
Go to cacti home at the install directory
cd $CACTI_HOME
vi auth_login.php, add one line after the "Login":
 
     <tr>
         <td><input type="submit" value="Login"></td>
         <td><a href="graph_view.php?action=tree">View graphs without login</a></td>
     </tr>

Now one can go to the login page and click the link above to view graphs.

4. Templates

There are many templates available at http://forums.cacti.net/about15067.html.
I setup these host templates as follows:

  1. * Linux Host (ucd/net-snmp) : for Linux host with SNMP installed, such as MongoDB hosts.
  2. X MongoDB Server HT : for MongoDB, this is home-grown template.
  3. X pushVM Server HT: for pushVM (XPRESSmp ), this is home-grown template.
  4. MySQL Server – Innodb: for MySQL Innodb, this is modified from X MySQL Server HT, the great work of mysql-cacti-templates
  5. MySQL Server – Memory: for Mysql Memory DB, this is modified from X MySQL Server HT.
  6. Cassandra 0.8: for Cassandra, this is modified from http://www.palominodb.com/node/65

5. How-to add servers

1. How-to add DB servers

It's the same for adding different DB type except choose different template as listed above and specified steps as below. Take mytestdb Mysql servers memory db as example:

Console -> Devices -> Add 
   description: mytestdb 
   host name: mytestdb.cacti.com
   Host Template: MySQL Server - Memory 
   Thold Up/Down Email Notification: Global List
   Downed Device Detection: None
   SNMP Version: None
 click Create
 Click "Create Graphs for this Host" at the top
 Check the top box, which will check all for you, then click Create
 
Follow section "10. How-to add host to a Graph tree" to add it to a tree

Confirm: click graphs, should see mytestdb (mytestdb.cacti.com) under MySQL Memory

Special for Cassandra

Special for MongoDB

  • When adding graphs, it will ask for Custom Data “Mongoport”, if mongo is running on port 27017, you can ignore it, just click Save. If mongo is running on other port, input the exact port, for example data nodes of sharding, input 27018.
  • For mongo configdb nodes, please do NOT select graph “X MongoDB ReplicatSet Lag GT”
  • For mongos nodes, ONLY add graph ” X MongoDB Commands GT”, ” X MongoDB Connections GT” and “X MongoDB Memory GT”
  • Add threshold for MongoDB Slave Lag:

Click “Data Sources” → Choose template “X MongoDB Slave Lag DT” → check the host → Choose an action: “Create Threshold from Template”

Special for MySQL DB

For MySQL DB, we need a user named “cacti” created at the db server

CREATE USER 'cacti'@'%' IDENTIFIED BY 'cacti';
GRANT
  CREATE TEMPORARY TABLES,
  LOCK TABLES,
  SUPER,
  PROCESS,
  SELECT,
  SHOW DATABASES,
  SHOW VIEW
ON *.* TO 'cacti'@'%';

To check the privileges of cacti:

select * from information_schema.USER_PRIVILEGES where GRANTEE like '%cacti%';

2. How-to add servers using Linux host template

Currently we separate host template from db template. So host template can be used by all DB hosts with the same OS.

  1. Let SA install SNMP as follows. NoSQL VM template already have it installed.
     install packages:  net-snmp, net-snmp-libs and net-snmp-utils 
     add below line to the top of /etc/snmp/snmpd.conf 
           rocommunity public
     Let snmpd service start automatically      
     
  1. Devices → Add → input “sharedmongo1_host” to Description, “sharedmongo1.cacti.com” to Hostname
  2. choose “* Linux Host (ucd/net-snmp” as Host Template
  3. Choose “Monitor Host”
  4. Thold Up/Down Email Notification, choose “Global List”
  5. Choose “Ping and SNMP” as Downed Device Detection
  6. Choose “Version 1” as SNMP Version
  7. Click “Save”
  8. Click “*Create Graphs for this Host”, choose all “Graph Templates” except “xx ucd 90 Filesystems”
  9. Choose “eth1”, Select a graph type: “In/Out Bits”
  10. Choose ”/data” in area “Data Query [ucd 10 fs]”
  11. Choose the mount point performance you want, eg. “sdb1” (/data), Select a graph type: “Device I/O – Bytes Read/Written”, Click “Create”
  12. Select a graph type: “Device I/O – Reads/Writes”, Choose the mount point performance you want, eg. “sdb1” (/data), Click “Create”
  13. Add the host to the graph tree same as it's DB, reference “10. How-to add host to a Graph tree”
  14. Add thresholds by template, reference “9. How-to add thresholds by templates”.
    • We have thresholds templates for Linux load, memory and disk space separately

6. How-to import templates

Find a templates from Internet and follow its README. 
There are many templates at http://forums.cacti.net/about15067.html.

7. How-to add templates manually

To add a new template, following the below steps:
Add Data Input Methods -> add Data Templates
 -> add Graph Templates -> add Host Templates

8. How-to create thresholds

 1. Click "Threshold Templates"
 2. Click "Add", choose "Data Template", for example "ucd_ssCpuRawIdle"
 3. Click "next", choose ""ucd_ssCpuRawIdle"
 4. Click "Create", change the Template Name to "ucd_ssCpuRawIdle TT", TT ->(Threshold Template)
 5. Check "Disable Restoration Email", so when the value back to normal, no mail send out.
 6. Re-Alert Cycle, choose "Every 15 minutes", so it will send mail every 15 minutes when the value is always abnormal.
 7. Low Warning Threshold: 30, when cpu idle is less than 30, it will send Warning emails.
 8. Low Threshold: 10, when cpu idle is less than 10, it will send Alert emails.
 9. Data Type: exact value. For other cases, like space usage, you can choose percentage.
 10. Warning Notification List: choose "DBA"
 11. Alert Notification List: choose "DBA"
 12. Click Save
 

9. How-to add thresholds by templates

1. Add thresholds in a batch:
Console -> Devices -> Choose Type eg. "X MongoDB Server HT", search lp -> Check all
       -> Choose an action: "Apply thresholds" -> Click Go -> Click Continue.

2. Add thresholds one by one:
Click "Data Sources" under Management section -> choose a data Template to select the Graphs
 -> Choose action "Create Threshold from Template" -> Click Go.
eg: data Template for Linux host: 'ucd_load1min', 'ucd_memAvailReal','Host MIB - Hard Drive Space'

OR

Click "Graph Management" -> choose a Template to select the Graphs
       -> Choose action "Create Threshold from Template" -> Click Go
eg: graph Template for Linux Host: 'xx ucd 01 Load Average', 'xx ucd 10 Memory', 'xx ucd 90 Filesystems'

3. Confirm:
Click "thold" at the top line -> Search eg. "sharedmongo", you will see the thresholds if they are added.

10. How-to add host to a Graph tree

1. For single db, such as MySQL and PushVM, we will add it to the root of the type

Console -> Graph tree -> MySQL Memory -> ADD
  Tree Item Type: Host
  Host:  mytestdb (mytestdb.cacti.com)
 click Create

2. For db cluster, we will create sub-tree for each cluster, like mongo and cassandra, for example:

Console -> Graph tree -> MongoDB -> ADD
  Tree Item Type: Header
  Title:  sharedmongo
  Click Create.
Console -> Devices -> Search: sharedmongo -> Choose all 
  -> Choose an action: Place on a Tree(MongoDB)-> Click Go
  -> Destination Branch: --sharedmongo -> Click Continue.
If sometimes it didn't work (bug?), then you need to the tree and click Save as follows:
Console -> Graph tree -> MongoDB -> click Save.

11. How-to add port option to Mongo templates

1. Modify ss_get_by_ssh_script.php script to accept option port2, which is designed for app port.

[root@vm-mydb02 scripts]# diff  ss_get_by_ssh_script.php   ss_get_by_ssh_script.php.bak
1289,1293c1289,1290
<    if ( !isset($options[port2]) || !$options[port2] ) {
<      return "/export/home/cacti/scripts/getMongoDBStats.pl -h $options[host]";
<    } else {
<      return "/export/home/cacti/scripts/getMongoDBStats.pl -h $options[host] -p $options[port2]";
<    }
---
> #   return "/oracle/home/bin/dbperl /oracle/home/admin/cron/getMongoDBStats.pl -h $options[host]";
>    return "/export/home/cacti/scripts/getMongoDBStats.pl -h $options[host]";

2. Modify EACH mongo in “Data Input Methods” in cacti console.

1) add " --port2 <mongoport>" to the Input String, click Save
<path_php_binary> -q <path_cacti>/scripts/ss_get_by_ssh_script.php --use-ssh FALSE --host <hostname> --port2 <mongoport> --type mongodb --items dk,dl,dm,dn 

2) Click add "Input Fields", add one with "Input Fields" = mongoport, Friendly Name = Mongoport, and check " Allow Empty Input", click Save
<code>

3. Edit each mongo entries in "Data Templates". 
In "Custom Data" section, you will see "Mongoport" at the bottom, check the box before "Use Per-Data Source Value (Ignore this Value)", click Save.

Reference

1. What is this 'm' or 'u' after my numbers in graphs?

10^-18  a - atto
10^-15  f - femto
10^-12  p - pico
10^-9   n - nano
10^-6   u - micro
10^-3   m - milli
0    (no unit)
10^3    k - kilo
10^6    M - mega
10^9    G - giga
10^12   T - tera
10^15   P - peta

2. How-to install plugin thold and monitor

1. Installing the Plugin Architecture (PIA)
   http://docs.cacti.net/manual:087:1_installation.9_pia
2. Installing plugin settings
   http://docs.cacti.net/plugin:settings
3. Installing plugin thold
   http://docs.cacti.net/plugin:thold
4. Installing plugin monitor
   http://docs.cacti.net/plugin:monitor
5. Click "User Management" -> Click username "admin" -> In "Realm Permissions" choose "Plugin Management" -> Save.
   Then you can see a "Plugin Management" menu on the left.
6. Click "Plugin Management", you can see the installed plugins with action icons,
    click the action icons to install/deinstall, enable/disable them