Systems and Network Management Configuring Cricket to Monitor SNMP Objects Contents 1 Procedure 1.1 Open the Documentation . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Installing the Necessary Perl modules . . . . . . . . . . . . . . . . . . 1.2.1 Installing the Required Perl Modules from CPAN . . . . . . . 1.2.2 Installing the other two Perl Modules . . . . . . . . . . . . . . 1.3 Creating the cricket account . . . . . . . . . . . . . . . . . . . . . . 1.4 Unpacking Cricket and running configure . . . . . . . . . . . . . . . 1.5 Copy the Config Tree and Modify it . . . . . . . . . . . . . . . . . . . 1.6 Adding a cron entry for Cricket . . . . . . . . . . . . . . . . . . . . . 1.7 Setting up the public html directory in cricket’s home directory for Apache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.8 Configuring the Apache Web Server . . . . . . . . . . . . . . . . . . . 2 Adding a new set of Graphs and Targets for Cricket to Monitor 2.1 Cricket’s Configuration Tree . . . . . . . . . . . . . . . . . . . . . . 2.1.1 A Closer Look at the Config Tree . . . . . . . . . . . . . . . 2.2 The servers subtree . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.1 Variable Expansion . . . . . . . . . . . . . . . . . . . . . . . 2.3 Tables and Instance Numbers . . . . . . . . . . . . . . . . . . . . . 2.3.1 The listInterfaces Utility . . . . . . . . . . . . . . . . . . 2.4 Monitoring High-End Cisco Switches: the genCatConfig Utility . . 2.5 Monitoring 35xx Cisco Switches and Routers . . . . . . . . . . . . . 1 2 2 3 4 6 6 7 8 9 10 12 12 13 14 14 15 17 18 19 . . . . . . . . Background: There are two free software packages for monitoring snmp objects from a web interface. The most widely known is mrtg, the “Multi Router Traffic Grapher.” mrtg is well suited to monitoring network traffic from router interfaces, but has certain limitations in many other situations. Red Hat Linux comes with mrtg as a ready-built software Nick Urbanik ver. 1.14 Configuring Cricket to Monitor SNMP Objects Systems and Network Management 2 package, and it is very easy to set it up to monitor network traffic. See http://people.ee.ethz.ch/~oetiker/webtools/mrtg/. The newer package is called Cricket, and this is what we will install and configure today. Cricket has greater flexibility, and can be extended in more ways. Cricket is available from http: //cricket.sourceforge.net/. Both packages are written mostly in Perl. There are links to both of these software packages from the subject web site. 1 Procedure There are eight steps to install and configure Cricket, as I demonstrated at the last lecture: • install the necessary Perl modules using cpan • create a user called cricket • unpack the Cricket tarball and run the configure script • copy the sample configuration files into a directory called cricket-confi • modify the configuration to point to what you want to monitor • configure cron to run the data collector • create a public html directory, and set up the programs in that directory • configure Apache so that it will run the programs 1.1 Open the Documentation 1. Go to the Cricket installation web page at http://cricket. sourceforge.net/support/doc/beginner.html. You can get there from the main Cricket page by following these links: support → documentation → Installing Cricket for the Complete Beginner . Nick Urbanik ver. 1.14 Configuring Cricket to Monitor SNMP Objects Systems and Network Management 3 1.2 Installing the Necessary Perl modules There are several Perl modules that Cricket requires. Some of these are already installed on your computer, some are not. You will install them with the cpan program, which downloads them from the Internet, compiles, tests, then installs them if they pass all tests. The modules to install from cpan include: • MD5 • DB File • Date::Parse • Time::HiRes 1.2.1 Installing the Required Perl Modules from CPAN 1. Start the cpan modules’ shell method from your own (not cricket’s) account: $ sudo perl -MCPAN -e shell 2. The shell will begin its interactive configuration process. Ac cept all defaults by pressing  Enter , except for the following : • When prompted “CPAN build and cache directory?” enter: /var/cache/cpan Do not accept the default in your home directory; it will not work, since the root user has no right to write to your nfs mounted home directory. • When asked to pick a nearby continent, enter 2 to select (5) North America (Do not select (2) Asia, 2 ftp:// ftp.pacific.net.hk/pub/mirror/CPAN/, because like many Hong Kong mirror sites, it is not maintained properly) Nick Urbanik ver. 1.14 • For “Your http proxy?” enter http://sheep.vtc.edu.hk:8080 Configuring Cricket to Monitor SNMP Objects Systems and Network Management 4 • When asked to “Select your country”, enter 1 to select (3) United States. • When asked “Select as many URLs as you like”, enter a number to select ftp://ftp.cs.colorado.edu/pub /perl/CPAN/, and a few others from the list. The numbers are just separated by spaces. • Configuration should complete. 3. Now, at the cpan> prompt, type: install MD5 install Date::Parse and so on, installing all the four modules listed above. 1.2.2 Installing the other two Perl Modules The other two Perl modules are not available from cpan; see the Cricket documentation for the location. We start here assuming that ictlab is mounted on /mnt/ftp. Do this in your own account. Installing SNMP Session: 1. Unpack the SNMP Session tarball into your (not cricket’s) home directory: $ $ $ $ $ $ cd tar xvzf /mnt/ftp/snmp/SNMP_Session-0.92.tar.gz cd SNMP_Session-0.92 perl Makefile.PL make sudo make install The SNMP Session Perl modules are now installed. You can get the latest copy of this module from ftp://ftp.switch.ch/software/ sources/network/snmp/perl/. The web page is at http://www.switch.ch/ misc/leinen/snmp/perl/. Nick Urbanik ver. 1.14 Configuring Cricket to Monitor SNMP Objects Systems and Network Management 5 Setting your path 1. Look at your path by typing: $ echo $PATH 2. If the directories /sbin and /usr/sbin are not already on your path, add them: (a) edit your login script: $ emacs ∼/.bash_profile & (b) Add a line like this at the end of your login script: export PATH=$PATH:/sbin:/usr/sbin (c) Copy and paste this line into your terminal window. This is available together with the OpenNMS package we will install later. It is provided as two rpm packages: Installing the Round Robin Database: 1. Simply install using the rpm program: $ sudo rpm -Uhv /mnt/ftp/opennms/0.9/rrdtool-*1.0.33-1on It seems that there may be a problems installing this unless the name service caching daemon (nscd) is running. This service is very important to reduce the load on our ldap server, and will improve the performance of your machine also, and should always be turned on. To turn on nscd: 1. First, start it now: $ sudo service nscd start 2. Now make sure that it always starts as a service when the computer boots: $ sudo chkconfig --level 345 nscd on Nick Urbanik ver. 1.14 Configuring Cricket to Monitor SNMP Objects Systems and Network Management 6 This turns the nscd service on in runlevels 3, 4 and 5. 3. Finally, verify that the service is configured correctly: $ chkconfig nscd --list nscd 0:off 1:off 2:off 3:on 4:on If you do not see “on” for runlevels 3, 4 and 5, something is wrong (perhaps a typing mistake?) You may also benefit by installing the Red Hat Linux updates. These are in the directory /mnt/ftp/redhat-7.2/updates on ictlab, assuming ictlab:/var/ftp/pub is mounted on /mnt/ftp. After this, all the required Perl packages are installed. 1.3 Creating the cricket account Here you create a local user. You will install the software into the home directory of this user. 1. If the directories /sbin and /usr/sbin are not already on your path, add them, as described on page 4. 2. Make sure that your useradd program is configured to create local accounts (see the handout I wrote about this for more information): $ sudo mkdir /home2 $ sudo useradd -D -b /home2 3. Create a local user cricket: $ sudo useradd -c "Cricket Manager" cricket 4. Give this user a password: $ sudo passwd cricket Nick Urbanik ver. 1.14 Configuring Cricket to Monitor SNMP Objects Systems and Network Management 7 1.4 Unpacking Cricket and running configure The Cricket tarball is available for download either from the Cricket web site, or you can install it directly from our server: 1. mount the server: $ sudo mkdir /mnt/ftp $ sudo mount ictlab.tyict.vtc.edu.hk:/var/ftp/pub /mnt/f 2. Next, open a window and become cricket: $ xhost +localhost $ su cricket The first command lets other users besides you (such as cricket) display on the local X server. 3. Change to cricket’s home directory and unpack the Cricket tarball: $ cd $ tar xvzf /mnt/ftp/snmp/cricket-1.0.3.tar.gz 4. Now create a symbolic link to the Cricket directory: $ ln -s cricket-1.0.3 cricket 5. Now run the configure script, which puts the path to perl in the first line of the Perl programs: $ cd ∼/cricket $ ./configure 6. Now copy the file cricket-conf.pl.sample to cricket-conf.pl: $ cd ∼/cricket $ cp cricket-conf.pl.sample cricket-conf.pl 7. Edit the file ∼/cricket/cricket-conf.pl, and change the line: Nick Urbanik ver. 1.14 Configuring Cricket to Monitor SNMP Objects Systems and Network Management 8 $gCricketHome = "/home/cricket"; to $gCricketHome = "/home2/cricket"; 1.5 Copy the Config Tree and Modify it Cricket uses a directory of configuration files that the authors call a config tree. Here we copy the sample config tree, and modify it. 1. First copy the sample tree as the user cricket: $ cd $ cp -a cricket/sample-config cricket-config 2. Now copy the configuration that I wrote for the demonstration in the lecture theatre. Here I assume that ictlab is mounted on /mnt/ftp. $ cd ∼/cricket-config $ cp -a /mnt/ftp/snmp/servers . # This is a dot _________^ There are two files in this directory: Defaults and Targets. These tell Cricket to monitor the system load on our server ictlab, and also the amount of free swap space, and the amount of free memory. You will later need to modify the config tree to monitor other devices. If you copied the two files from /mnt/ftp/snmp/servers into ∼/cricket-config instead of into the directory ∼/cricket-config/servers, then the collector will not work. If you have done this, simply go back and copy the sample configuration files back again. 3. Every time you modify the config tree, you need to run the compile script (as the user cricket): $ ∼/cricket/compile 4. Cricket collects data from data sources using a script called the collector. This will be run by cron. First, check that the config tree has been set up correctly: Nick Urbanik ver. 1.14 Configuring Cricket to Monitor SNMP Objects Systems and Network Management 9 $ ∼/cricket/collector /servers This should run without errors, and five data items should be shown: the three values for system load, and the free swap space in kilobytes, and free memory. 1.6 Adding a cron entry for Cricket Cricket collects the information it graphs using cron, every five minutes. To use cron to gather the information for Cricket, you need to add an entry to cricket’s crontab like this. First, you need to be cricket. 1. The cron program collects information about the network periodically using script called collect-subtrees. This script reads a configuration file ∼cricket/cricket/subtree-sets. Edit this file: $ emacs ∼/cricket/subtree-sets & 2. Add a new set to it like this: set linuxservers: /servers The script collect-subtrees takes one parameter, the set of data to collect. Here our set is called linuxservers so we use that parameter in our cron entry. 3. Start editing cricket’s crontab: $ crontab -e 4. In the editor, you need to add an entry like this: */5 * * * * $HOME/cricket/collect-subtrees linuxservers which will run the program $HOME/cricket/collect-subtrees linuxservers every five minutes. Nick Urbanik ver. 1.14 Configuring Cricket to Monitor SNMP Objects Systems and Network Management 10 1.7 Setting up the public html directory in cricket’s home directory for Apache I found it necessary to make a minor change from the instructions for putting the cgi programs in the public html directory. This is what I did, as the user cricket: $ $ $ $ $ $ $ cd mkdir -p public_html/cricket cd public_html/cricket ln -s ∼/cricket/VERSION ln ∼/cricket/grapher.cgi . ln ∼/cricket/mini-graph.cgi . ln -s ∼/cricket/lib ∼/cricket/images . The only difference is that I made a hard link, rather than a symbolic link, to the cgi programs. I have also added a link to the documentation, so that you can easily read it online: $ ln -s ∼/cricket/doc ∼/public_html You can read the Cricket documentation at http://localhost/ ~cricket/doc/ after you have set up Apache—see the next section. 1.8 Configuring the Apache Web Server Now configure Apache to run Cricket: 1. Edit Apache’s configuration file: $ sudo -v $ sudo emacs /etc/httpd/conf/httpd.conf & 2. Search for the “Directory” section for public html. 3. If the section is commented out, copy it and uncomment the copy. Nick Urbanik ver. 1.14 Configuring Cricket to Monitor SNMP Objects Systems and Network Management 11 4. You need to add the option ExecCGI, and should end up with something like this: Options MultiViews Indexes SymLinksIfOwnerMatch ExecC Note that the manual for Apache is included on your hard disk. When Apache is running, you will find it at this location: http://localhost/manual/. The manual is very complete, and explains every possible configuration option of Apache. 5. You also need to uncomment the line in Apache’s configuration file: AddHandler cgi-script .cgi 6. Note too that cricket’s home directory, and ∼cricket/public html, and ∼cricket/public html/cricket all need to have the group write permission removed, but the execute permission added for group and others. This is to satisfy the requirements of suexec, which is the mechanism by which Apache executes Cricket’s programs as the user cricket. The complete documentation for suexec is included with the online Apache documentation on your hard disk. Okay, it seems nobody understands that, so here’s the recipe: $ chmod go=x ∼ $ chmod go=rx ∼/cricket ∼/public_html ∼/public_html/cricket The permissions should look something like this: $ ls -ldL ∼ ∼/cricket ∼/public_html/cricket drwx--x--x 7 cricket cricket 4096 Dec drwxr-xr-x 7 cricket cricket 4096 Dec drwxr-xr-x 2 cricket cricket 4096 Dec drwxr-xr-x 2 cricket cricket 4096 Dec 18 11 11 11 13:44 17:21 17:55 17:55 /home2/cricket /home2/cricket/cricket /home2/cricket/public_html /home2/cricket/public_html/cricket 7. You can use the serviceconf program to start Apache’s httpd service in runlevels 3, 4 and 5. Alternatively you can use the same method to start and maintain the Apache service as with nscd given on page 5. $ sudo service httpd start $ sudo chkconfig --level 345 httpd on Nick Urbanik ver. 1.14 Configuring Cricket to Monitor SNMP Objects Systems and Network Management 12 Testing, and using the log files: 1. Now open a web browser, and go to http://localhost/ ~cricket/cricket/grapher.cgi 2. If you see a server error message: (a) View the log files for Apache to see what is happening: $ cd /var/log/httpd $ ls -ltr $ sudo tail -f error_log (b) You may also want to open two more windows, and leave them open, running tail on suexec log and access log. In particular, monitoring suexec log will let you know about permission problems when accessing the programs. (c) After modifying Apache’s configuration, tell Apache to reload it: $ sudo service httpd graceful 3. If you see the Cricket page, click on servers, then load, and you should see the graphs of system load there. 2 Adding a new set of Graphs and Targets for Cricket to Monitor Now this is where you use your knowledge of snmp. Refer to the page Setting up New Devices in Cricket, at http://cricket. sourceforge.net/support/doc/new-devices.html, or http: //localhost/~cricket/doc/new-devices.html if you make the link I described at the end of section 1.7 on page 10. There is a useful set of configurations in http://www.gnac.com/techinfo/ cricket_contrib/index.html. The reference for the configuration of Cricket is at http://cricket.sourceforge.net/ support/doc/reference.html. Nick Urbanik ver. 1.14 Configuring Cricket to Monitor SNMP Objects Systems and Network Management 13 Cricket has a very active mailing list for users, about ten posts per day. I had an perfect solution to my problem in less than two hours. The subscription information is available at http: //cricket.sourceforge.net/support/. 2.1 Cricket’s Configuration Tree The directory ∼/cricket-config contains a directory tree. This directory tree contains the configuration of cricket. It is called a “config tree” because the configuration files at the higher levels provide default values, and some of these values are overridden by the configuration files at the lower levels. Every directory in ∼/cricket-config contains a file called Defaults. The top level configuration file, ∼/cricket-config/Defaults, contains a very large number of useful definitions. In particular, it is set up so that all you need to plot network traffic is in place. The oids are provided for all the useful items in the Mib-2 Interfaces table. Look at this file; you will see that the default graphs are graphs of network traffic in and out of a network interface. Each chunk of the file begins with a word such as “target” or “OID”. After that word, they are different, but you will see many things such as rrd-datafile = %dataDir%/%auto-target-name%.rrd. The percent signs indicate a variable that is expanded somewhere else in the configuration tree. Many of these variables have reasonable defaults that are defined at the top level of the tree. You only need to fill in the values for some (such as the hostname of the target) further down in the config tree. 2.1.1 A Closer Look at the Config Tree The configuration is divided into dictionaries. Below some dictionaries, there are tags and values, arranged like this: Nick Urbanik ver. 1.14 Configuring Cricket to Monitor SNMP Objects Systems and Network Management 14 dictionary − name tag1 = value1 tag2 = value2 ... These are the dictionaries available: target A device that we monitor is a target. There is one rrd file for each target. datasource A Data source is one line on a graph. One rrd can have many datasources in it. All datasources in one rrd need to be updated by the collector at once. targetType Determines what kinds of data sources there are for the target with this targetType. The collector uses the targetType to decide what data to fetch, and how to fetch it. graph Determines how individual data sources are graphed. color Maps colour names to html colours. Probably no need to change this. oid Maps an oid to a name html determines what html goes into each page event draws vertical bars on the graph to indicate events, such as the end of one day and the start of the next. No need to change this. rra Holds parameters used by Cricket when creating a new rrd file. No need to change this. 2.2 The servers subtree Let us consider how the configuration for /servers works I decided to monitor the system load average and the available free memory. Nick Urbanik ver. 1.14 Configuring Cricket to Monitor SNMP Objects Systems and Network Management 15 There are five data sources here, three for the load averages, one for available swap, and one for free memory. These are listed under the targetType dictionary name. 2.2.1 Variable Expansion When the collector gets the system load data from ictlab, it uses an snmp url something like this: snmp://public@ictlab.tyict.vtc.edu.hk:161:2.0:5:1. 0:1/1.3.6.1.4.1.2021.10.1.3.1 But where did all that come from? Let’s look in the Target dictionary in the top level ∼/cricket-config/De file: Target --default-snmp-host = %auto-target-name% snmp-community = public snmp-port = 161 snmp-timeout = 2.0 snmp-retries =5 snmp-backoff = 1.0 snmp-version =1 snmp = %snmp-community%@%snmp-host%:%snmp %snmp-timeout%:%snmp-retries%:%snmp-backoff%:%snmp-version% So what about %auto-target-name%? This is automatically set by Cricket to the name of the target. So let’s see part of the file ∼/cricket-config/servers/Targets: target ictlab.tyict.vtc.edu.hk target-type = linux-machine Finally, let’s see how the oid and the data source is defined in ∼/cricket-config/servers/Defaults: OID laLoad1min 1.3.6.1.4.1.2021.10.1.3.1 datasource laLoad1min ds-source = snmp://%snmp%/laLoa Nick Urbanik ver. 1.14 Configuring Cricket to Monitor SNMP Objects Systems and Network Management 16 So default values mostly come from the top level of the configuration tree and are inherited below. You can override any default defined at the top level with values you define further down. 2.3 Tables and Instance Numbers We examined the Interfaces table of Mib-2 in some detail. Each row in the table corresponds to one network interface. How do we use snmp to get information on a particular network interface? Let’s look at ifInOctets, in the mib file /usr/share /snmp/mibs/RFC1213-MIB.txt. The definition of ifInOctets is: ifInOctets OBJECT-TYPE SYNTAX Counter ACCESS read-only STATUS mandatory DESCRIPTION "The total number of octets received on the interface, including framing characters." ::= { ifEntry 10 } Let’s get the oid of this table entry: $ snmptranslate -IR -Of ifInOctets .1.3.6.1.2.1.2.2.1.10 Good, now let’s see the value on ictlab: $ snmpwalk ictlab public .1.3.6.1.2.1.2.2.1.10 interfaces.ifTable.ifEntry.ifInOctets.1 = Counter32: 7523664 interfaces.ifTable.ifEntry.ifInOctets.2 = Counter32: 7918911 Hey, there are two of them! What are they? Well, Mib-2 provides a description of the interface in ifDescr; let’s see what that says: $ snmpwalk ictlab public ifDescr interfaces.ifTable.ifEntry.ifDescr.1 = lo interfaces.ifTable.ifEntry.ifDescr.2 = eth0 Nick Urbanik ver. 1.14 Configuring Cricket to Monitor SNMP Objects Systems and Network Management 17 Okay, so the entries in the ifTable with a .1 at the end on ictlab relate to the loopback interface, while the entries with a .2 at the end are about the Ethernet interface. This number at the end of the oid is called an instance number, and is the way that snmp selects items from a table. Notice that the instance numbers begin at 1 for table entries. What about for scalars? $ snmpwalk ictlab public sysUpTime system.sysUpTime.0 = Timeticks: (103614809) 11 days, 23:49:0 The instance number for a scalar is always zero. Typically, the loopback interface is the first entry in the table, and other interfaces follow. However, snmp does not prevent a router from changing the order of the interfaces. This could be a real problem; after rebooting a router, you may find that the graph for your subnet is now collecting data for a different subnet! Luckily, Cricket has a solution to this called instance mapping, described in http://cricket.sourceforge.net/support/doc/ inst-mapping.html. It is also possible to directly provide a value for the instance number for a target to graph, instead of using instance mapping. For example, you could set target hostname-of-server-or-router inst = 2 to draw graphs of interface two of the machine called hostnameof-server-or-router. 2.3.1 The listInterfaces Utility A very nice Cricket utility is provided to automatically find the names of all the interfaces on a server or router, then map each name to an instance number. This is described in http://cricket. sourceforge.net/support/doc/beginner.html. The name Nick Urbanik ver. 1.14 Configuring Cricket to Monitor SNMP Objects Systems and Network Management 18 of the utility is listInterfaces. If main-router is the hostname of a server or router that you want to plot statistics of the network interfaces from, you can do: $ ∼/cricket/util/listInterfaces main-router > interfaces The file interfaces can then be used in the ∼/cricket/cricket-config /router-interfaces directory to specify how to plot each interface. All that is left is simply to put an entry in subtree-sets so that the collector will collect the data from the server or router. Cricket provides enormous flexibility once it is set up. 2.4 Monitoring High-End Cisco Switches: the genCatConfig Utility To monitor Cisco CatalystTM switches, such as our 6509, there is a tool available for automatically generating the configuration for Cricket. It is currently available from http://www.certaintysolutions. com/tech-advice/cricket-contrib/; you can download the program from that page. Note that all these programs are available from our own server ictlab, by many protocols, including http: //ictlab.tyict.vtc.edu.hk/ftp/snmp/cricket-contrib, and by nfs from ictlab:/var/ftp/pub/snmp. Read the file ∼/cricket/util/README.genCatConfig for the full documentation; this is just a short summary. To install it, • become cricket, and • untar the package into /tmp. Then • copy the directory /tmp/genCatConfig-1.4.1/sample-config/cata and its contents into ∼/cricket-config. • Copy the contents of /tmp/genCatConfig-1.4.1/util into ∼/cricket/util. Nick Urbanik ver. 1.14 Configuring Cricket to Monitor SNMP Objects Systems and Network Management 19 • Edit the two executables and change the first line in each file from #!/usr/local/bin/perl to #! /usr/bin/perl • Make a host table entry for the router in /etc/hosts. • Change directory to ∼/cricket-config/catalysts • Then run $ ∼/cricket/util/genCatConfig -2 -C community-string replacing router-hostname with the host name of the Catalyst router that you put into the /etc/hosts file, and community-string with its read-only community string. • Add /catalysts/router-hostname subtree-sets file. to Cricket’s ro • Run ∼/cricket/compile. It should now collect data. Test it using the collector. 2.5 Monitoring 35xx Cisco Switches and Routers According to the GenCatConfig documentation: Some Catalysts (e.g. 2900XL and 3500XL switches) run router ios and respond to a different set of mibs. They should be handled as routers, not switches. (see genRtrConfig). So download genRtrConfig from the same place as genCatConfig above. The installation process is almost identical. Nick Urbanik ver. 1.14