How to install spine for cacti

As cacti.net said, Spine is the fast replacement for cmd.php. So I started to install it for cacti by reference this doc Spine install. I run into lots of issues. Later, I found there is a better guide Install and Configure Spine. But it’s still not enough.

The solution in simple word is that we need to run ./bootstrap in the same directory before we run ./configure. The long store is that, in order to run bootstrap, we need to install more packages.

During my practice, Spine did improved the performance.

cmd.php :
"Maximum Concurrent Poller Processes"=20 cannot finish in 1 minute, CPU usage maximum is 65%

Spine:
"Maximum Concurrent Poller Processes"=10 and "Maximum Threads per Process"=10 cannot finish in 1 minute, CPU usage reached 100%
"Maximum Concurrent Poller Processes"=5 and "Maximum Threads per Process"=10  finish in 36 seconds, CPU usage reached 99%
"Maximum Concurrent Poller Processes"=3 and "Maximum Threads per Process"=20  finish in 33 seconds, CPU usage is lower
"Maximum Concurrent Poller Processes"=2 and "Maximum Threads per Process"=30  finish in 33 seconds, CPU usage is 97%
"Maximum Concurrent Poller Processes"=1 and "Maximum Threads per Process"=30  finish in 44 seconds, CPU usage maximum is 85%
"Maximum Concurrent Poller Processes"=1 and "Maximum Threads per Process"=20  finish in 44 seconds, CPU usage maximum is 82%
"Maximum Concurrent Poller Processes"=1 and "Maximum Threads per Process"=15  finish in 56 seconds, CPU usage maximum is 60%
"Maximum Concurrent Poller Processes"=1 and "Maximum Threads per Process"=10  cannot finish in 1 minute, CPU usage maximum is 30%

Finally, I use 1 process with 20 threads. That meets 1 minute limit, and still have some buffer, and lower CPU usage. Here is the log:
12/26/2012 03:17:47 AM – SYSTEM STATS: Time:45.7329 Method:spine Processes:1 Threads:20 Hosts:443 HostsPerProcess:443 DataSources:10714 RRDsProcessed:10714

From those data, I got conclusions as follows:

1. If you change from cmd.php to Spine, set processes number to 1, and set use its number for the threads per process, or less
2. Spine is CPU killer, unless you have enough powerful CPU, set “Maximum Concurrent Poller Processes” to 1.
3. Increase maximum threads to meet the cycle time, but once it meats, add more threads didn’t reduce the response time
4. To reduce the response time, use more processes with Spine, which comes along with high CPU cost

Here is the whole processes:

--in Summary, packages installed:
autoconf
dos2unix
automake
libtool

# ./configure
checking build system type... Invalid configuration `x86_64-unknown-linux-': machine `x86_64-unknown-linux' not recognized
configure: error: /bin/sh config/config.sub x86_64-unknown-linux- failed

--get hint from this page http://forums.cacti.net/viewtopic.php?f=5&t=46320 , run bootstrap first

[root@phxdbx1112 cacti-spine-0.8.8a]# ./bootstrap
FATAL: Unable to locate dos2unix utility

[root@ cacti-spine-0.8.8a]# yum install dos2unix

[root@ cacti-spine-0.8.8a]# ./bootstrap
INFO: Starting Spine build process
INFO: Removing cache directories
INFO: Running auto-tools to verify buildability
./bootstrap: line 51: autoreconf: command not found
ERROR: 'autoreconf' exited with errors

[root@ cacti-spine-0.8.8a]# yum install autoconf

[root@ cacti-spine-0.8.8a]# ./bootstrap
INFO: Starting Spine build process
INFO: Removing cache directories
INFO: Running auto-tools to verify buildability
Can't exec "aclocal": No such file or directory at /usr/share/autoconf/Autom4te/FileUtils.pm line 326.
autoreconf: failed to run aclocal: No such file or directory
ERROR: 'autoreconf' exited with errors

[root@ cacti-spine-0.8.8a]# yum install automake

[root@ cacti-spine-0.8.8a]# ./bootstrap
INFO: Starting Spine build process
INFO: Removing cache directories
INFO: Running auto-tools to verify buildability
configure.ac:69: error: possibly undefined macro: AC_PROG_LIBTOOL
      If this token and others are legitimate, please use m4_pattern_allow.
      See the Autoconf documentation.
autoreconf: /usr/bin/autoconf failed with exit status: 1
ERROR: 'autoreconf' exited with errors

[root@ cacti-spine-0.8.8a]# yum install libtool

[root@ cacti-spine-0.8.8a]# ./bootstrap
INFO: Starting Spine build process
INFO: Removing cache directories
INFO: Running auto-tools to verify buildability
libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, `config'.
libtoolize: copying file `config/ltmain.sh'
libtoolize: Consider adding `AC_CONFIG_MACRO_DIR([m4])' to configure.ac and
libtoolize: rerunning libtoolize, to keep the correct libtool macros in-tree.
libtoolize: Consider adding `-I m4' to ACLOCAL_AMFLAGS in Makefile.am.
libtoolize: `AC_PROG_RANLIB' is rendered obsolete by `LT_INIT'
INFO: Spine bootstrap process completed

--Finally, bootstrap run successfully

[root@ cacti-spine-0.8.8a]# ./configure
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking for gawk... (cached) gawk
checking for gcc... gcc
checking for C compiler default output file name... a.out
checking whether the C compiler works... yes
checking whether we are cross compiling... no
checking for suffix of executables...
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking for style of include used by make... GNU
checking dependency style of gcc... gcc3
checking how to run the C preprocessor... gcc -E
checking for a BSD-compatible install... /usr/bin/install -c
checking whether ln -s works... yes
checking for a sed that does not truncate output... /bin/sed
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for fgrep... /bin/grep -F
checking for ld used by gcc... /usr/bin/ld
checking if the linker (/usr/bin/ld) is GNU ld... yes
checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B
checking the name lister (/usr/bin/nm -B) interface... BSD nm
checking the maximum length of command line arguments... 1966080
checking whether the shell understands some XSI constructs... yes
checking whether the shell understands "+="... yes
checking for /usr/bin/ld option to reload object files... -r
checking for objdump... objdump
checking how to recognize dependent libraries... pass_all
checking for ar... ar
checking for strip... strip
checking for ranlib... ranlib
checking command to parse /usr/bin/nm -B output from gcc object... ok
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking for dlfcn.h... yes
checking for objdir... .libs
checking if gcc supports -fno-rtti -fno-exceptions... no
checking for gcc option to produce PIC... -fPIC -DPIC
checking if gcc PIC flag -fPIC -DPIC works... yes
checking if gcc static flag -static works... no
checking if gcc supports -c -o file.o... yes
checking if gcc supports -c -o file.o... (cached) yes
checking whether the gcc linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes
checking whether -lc should be explicitly linked in... no
checking dynamic linker characteristics... GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... yes
checking for ranlib... (cached) ranlib
checking whether to enable -Wall... no
checking for threadsafe gethostbyname()... no
checking for gethostbyname_r in -lnls... no
checking for socket in -lsocket... no
checking for floor in -lm... yes
checking for pthread_exit in -lpthread... yes
checking for deflate in -lz... yes
checking for kstat_close in -lkstat... no
checking for CRYPTO_realloc in -lcrypto... yes
checking for ANSI C header files... (cached) yes
checking sys/socket.h usability... yes
checking sys/socket.h presence... yes
checking for sys/socket.h... yes
checking sys/select.h usability... yes
checking sys/select.h presence... yes
checking for sys/select.h... yes
checking sys/wait.h usability... yes
checking sys/wait.h presence... yes
checking for sys/wait.h... yes
checking sys/time.h usability... yes
checking sys/time.h presence... yes
checking for sys/time.h... yes
checking assert.h usability... yes
checking assert.h presence... yes
checking for assert.h... yes
checking ctype.h usability... yes
checking ctype.h presence... yes
checking for ctype.h... yes
checking errno.h usability... yes
checking errno.h presence... yes
checking for errno.h... yes
checking signal.h usability... yes
checking signal.h presence... yes
checking for signal.h... yes
checking math.h usability... yes
checking math.h presence... yes
checking for math.h... yes
checking malloc.h usability... yes
checking malloc.h presence... yes
checking for malloc.h... yes
checking netdb.h usability... yes
checking netdb.h presence... yes
checking for netdb.h... yes
checking for signal.h... (cached) yes
checking stdarg.h usability... yes
checking stdarg.h presence... yes
checking for stdarg.h... yes
checking stdio.h usability... yes
checking stdio.h presence... yes
checking for stdio.h... yes
checking syslog.h usability... yes
checking syslog.h presence... yes
checking for syslog.h... yes
checking for netinet/in_systm.h... yes
checking for netinet/in.h... yes
checking for netinet/ip.h... yes
checking for netinet/ip_icmp.h... yes
checking for unsigned long long... yes
checking for long long... yes
checking for an ANSI C-conforming const... yes
checking for size_t... yes
checking whether time.h and sys/time.h may both be included... yes
checking whether struct tm is in sys/time.h or time.h... time.h
checking return type of signal handlers... void
checking for malloc... yes
checking for calloc... yes
checking for gettimeofday... yes
checking for strerror... yes
checking for strtoll... yes
checking priv.h usability... no
checking priv.h presence... no
checking for priv.h... no
checking whether we are using Solaris privileges... no
checking sys/capability.h usability... no
checking sys/capability.h presence... no
checking for sys/capability.h... no
checking whether we are using Linux Capabilities... no
checking for mysql_init in -lmysqlclient_r... yes
checking for mysql_thread_init in -lmysqlclient_r... yes
checking if UCD-SNMP needs crypto support... no
checking if Net-SNMP needs crypto support... yes
checking for snmp_timeout in -lnetsnmp... yes
checking for the spine results buffer size... 1024 bytes
checking for the maximum simultaneous spine scripts... 20
checking for the maximum MySQL buffer size... 65536
checking whether we are using traditional popen... no
checking whether to verify net-snmp library vs header versions... no
checking for glibc gethostbyname_r... yes
checking for Solaris/Irix gethostbyname_r... no
checking for HP-UX gethostbyname_r... no
configure: creating ./config.status
config.status: creating Makefile
config.status: creating config/config.h
config.status: executing depfiles commands
config.status: executing libtool commands
[root@ cacti-spine-0.8.8a]# make
gcc -DHAVE_CONFIG_H -I. -I./config     -I/usr/include/net-snmp -I/usr/include/net-snmp/.. -I/usr/include/mysql -g -O2 -MT sql.o -MD -MP -MF .deps/sql.Tpo -c -o sql.o sql.c
mv -f .deps/sql.Tpo .deps/sql.Po
gcc -DHAVE_CONFIG_H -I. -I./config     -I/usr/include/net-snmp -I/usr/include/net-snmp/.. -I/usr/include/mysql -g -O2 -MT spine.o -MD -MP -MF .deps/spine.Tpo -c -o spine.o spine.c
mv -f .deps/spine.Tpo .deps/spine.Po
gcc -DHAVE_CONFIG_H -I. -I./config     -I/usr/include/net-snmp -I/usr/include/net-snmp/.. -I/usr/include/mysql -g -O2 -MT util.o -MD -MP -MF .deps/util.Tpo -c -o util.o util.c
mv -f .deps/util.Tpo .deps/util.Po
gcc -DHAVE_CONFIG_H -I. -I./config     -I/usr/include/net-snmp -I/usr/include/net-snmp/.. -I/usr/include/mysql -g -O2 -MT snmp.o -MD -MP -MF .deps/snmp.Tpo -c -o snmp.o snmp.c
mv -f .deps/snmp.Tpo .deps/snmp.Po
gcc -DHAVE_CONFIG_H -I. -I./config     -I/usr/include/net-snmp -I/usr/include/net-snmp/.. -I/usr/include/mysql -g -O2 -MT locks.o -MD -MP -MF .deps/locks.Tpo -c -o locks.o locks.c
mv -f .deps/locks.Tpo .deps/locks.Po
gcc -DHAVE_CONFIG_H -I. -I./config     -I/usr/include/net-snmp -I/usr/include/net-snmp/.. -I/usr/include/mysql -g -O2 -MT poller.o -MD -MP -MF .deps/poller.Tpo -c -o poller.o poller.c
mv -f .deps/poller.Tpo .deps/poller.Po
gcc -DHAVE_CONFIG_H -I. -I./config     -I/usr/include/net-snmp -I/usr/include/net-snmp/.. -I/usr/include/mysql -g -O2 -MT nft_popen.o -MD -MP -MF .deps/nft_popen.Tpo -c -o nft_popen.o nft_popen.c
mv -f .deps/nft_popen.Tpo .deps/nft_popen.Po
gcc -DHAVE_CONFIG_H -I. -I./config     -I/usr/include/net-snmp -I/usr/include/net-snmp/.. -I/usr/include/mysql -g -O2 -MT php.o -MD -MP -MF .deps/php.Tpo -c -o php.o php.c
mv -f .deps/php.Tpo .deps/php.Po
gcc -DHAVE_CONFIG_H -I. -I./config     -I/usr/include/net-snmp -I/usr/include/net-snmp/.. -I/usr/include/mysql -g -O2 -MT ping.o -MD -MP -MF .deps/ping.Tpo -c -o ping.o ping.c
mv -f .deps/ping.Tpo .deps/ping.Po
gcc -DHAVE_CONFIG_H -I. -I./config     -I/usr/include/net-snmp -I/usr/include/net-snmp/.. -I/usr/include/mysql -g -O2 -MT keywords.o -MD -MP -MF .deps/keywords.Tpo -c -o keywords.o keywords.c
mv -f .deps/keywords.Tpo .deps/keywords.Po
gcc -DHAVE_CONFIG_H -I. -I./config     -I/usr/include/net-snmp -I/usr/include/net-snmp/.. -I/usr/include/mysql -g -O2 -MT error.o -MD -MP -MF .deps/error.Tpo -c -o error.o error.c
mv -f .deps/error.Tpo .deps/error.Po
/bin/sh ./libtool --tag=CC   --mode=link gcc  -I/usr/include/net-snmp -I/usr/include/net-snmp/.. -I/usr/include/mysql -g -O2  -L/usr/lib64 -L/usr/lib64/mysql  -o spine sql.o spine.o util.o snmp.o locks.o poller.o nft_popen.o php.o ping.o keywords.o error.o  -lnetsnmp -lmysqlclient_r -lmysqlclient_r -lcrypto -lz -lpthread -lm
libtool: link: gcc -I/usr/include/net-snmp -I/usr/include/net-snmp/.. -I/usr/include/mysql -g -O2 -o spine sql.o spine.o util.o snmp.o locks.o poller.o nft_popen.o php.o ping.o keywords.o error.o  -L/usr/lib64 -L/usr/lib64/mysql -lnetsnmp -lmysqlclient_r -lcrypto -lz -lpthread -lm
[root@ cacti-spine-0.8.8a]# make install
make[1]: Entering directory `/mysql/home/cacti-spine-0.8.8a'
test -z "/usr/local/spine/bin" || /bin/mkdir -p "/usr/local/spine/bin"
  /bin/sh ./libtool   --mode=install /usr/bin/install -c spine '/usr/local/spine/bin'
libtool: install: /usr/bin/install -c spine /usr/local/spine/bin/spine
test -z "/usr/local/spine/etc" || /bin/mkdir -p "/usr/local/spine/etc"
 /usr/bin/install -c -m 644 spine.conf.dist '/usr/local/spine/etc'
make[1]: Leaving directory `/mysql/home/cacti-spine-0.8.8a'

[root@ cacti-spine-0.8.8a]# ls -lt /usr/local/spine/bin/
total 264
-rwxr-xr-x 1 root root 269419 Dec 25 19:25 spine

[root@ cacti-spine-0.8.8a]# cp spine.conf.dist /etc/spine.conf

[root@ cacti-spine-0.8.8a]# vi  /etc/spine.conf
--make sure the information to connect to mysql db is right

--test it
[root@ cacti-spine-0.8.8a]#  /usr/local/spine/bin/spine
SPINE: Using spine config file [/etc/spine.conf]
SPINE: Version 0.8.8a starting

^C12/25/2012 07:34:35 PM - SPINE: Poller[0] FATAL: Spine Interrupted by Console Operator (Spine thread)
[root@ cacti-spine-0.8.8a]#

A few other problems I got :
1. “WARNING: Max OIDS is out of range with value of ‘0’. Resetting to default of 5”.
I didn’t find a clue by google. But I did find the source code of this warning message in poller.c. After looking into the code, I understood it means that the max_oids of hosts are 0 in my cacti. Then I got the solution :

First, update the current max_oids. I think it’s because “The Maximum SNMP OID’s Per SNMP Get Request” is set to 0 in the Spine section before.

mysql> update host set max_oids=1 where max_oids=0;
Query OK, 435 rows affected (0.01 sec)
Rows matched: 435  Changed: 435  Warnings: 0

mysql> commit;
Query OK, 0 rows affected (0.00 sec)

Second, set “The Maximum SNMP OID’s Per SNMP Get Request” to 5 to avoid new hosts run into the same issue.

2. “SPINE: Poller[0] ERROR: Spine Timed Out While Processing Hosts Internal”
This is because the Spine cannot finish the polling in time.

3. “POLLER: Poller[0] WARNING: There are ‘1’ detected as overrunning a polling process, please investigate”
The number ‘1’ can be others as well, ‘2’ or ‘3’, etc. The reason is the same as previous.

The solution is to “poller cannot finish in time”:
1) Adjust your polling processes/threads.
2) Remove/disable any bad hosts that cannot connect, or reduce “Script and Script Server Timeout Value” to a reasonable value.
3) If possible, let poller use cache. For my case, the script ss_get_by_ssh_script.php has a cache mechanism.
4) Using Spine instead of cmd.php