RE: [xymon] RE: help setting up SNMP traps to Xymon

["Williams, Doug \(Consultant-RIC\)" <Doug.Williams (at) dexone.com>]

It depends on how you have it all configured.  For example, I have our
setup such that a single host  receives a boat load of different traps
and so a clear trap does not really make sense since it has no way of
knowing which trap alert it is clearing.
 
But if you have yours setup in bb-hosts such that oracle_<some_state>
is defined as a host or entity with it's own IP (alias)  for example and
it only receives a specific type of trap, then a clear trap would make
sense.  In this case you could even define your own OIDs (or extend
existing ones) (making sure all unique amongst all your snmptt.conf
entries )   
 
From: Nicole Beck [mailto:nskyrca (at) syr.edu] 
Sent: Thursday, November 04, 2010 1:56 PM
To: xymon (at) xymon.com
Subject: RE: [xymon] RE: help setting up SNMP traps to Xymon
 
Hi Doug,
 
I found a CLEAR trap in my snmptrapd and snmptt logs, and it says
"Unreachable Clear".  There is nothing like that defined in my
snmptt.conf.oracle file.  If I understand this right, I have some things
to change in the snmptt.conf.oracle file.
 
 
Thanks,
Nicole
 
From: Williams, Doug (Consultant-RIC) [mailto:Doug.Williams (at) dexone.com] 
Sent: Wednesday, November 03, 2010 2:32 PM
To: xymon (at) xymon.com
Subject: RE: [xymon] RE: help setting up SNMP traps to Xymon
 
Check the  SEC startup script  (in my case /sbin/init.d/rc.sec) for the
paths:
 
OPTIONS="-detach -conf=/usr/local/etc/sec.conf
-log=/var/adm/syslog/sec.log"
OPTIONS2="-input=/var/adm/syslog/snmptt.log
-input=/var/adm/syslog/snmpttunknown.log"
 
 
Look in the snmpttunknown.log for any traps it does not have a pattern
match for, in case the CLEAR trap may be formatted differently:
 
Look in sec.conf and ensure there is pattern that matches the CLEAR trap
if format different
 
From: Nicole Beck [mailto:nskyrca (at) syr.edu] 
Sent: Wednesday, November 03, 2010 10:02 AM
To: xymon (at) xymon.com
Subject: [xymon] RE: help setting up SNMP traps to Xymon
 
Thanks Andy.
 
I thought that the severity of the trap came from what is defined in the
/etc/snmp/snmptt.conf.oracle file?  The cerebro.victoriacollege.edu
website has a section saying:
 
I've noticed that the APC and Dell MIB files have a SEVERITY definition
in them. SNMPTT uses that to establish the severity for each trap
(Normal|INFORMATIONAL|SEVERE|WARNING|...). However, I've noticed that
Cisco and Canoga Perkins don't have those definitions; so, every trap
event is considered Normal. You'll need to change the severity for the
various traps as desired in the snmptt.conf file.
 
Does the snmptt.conf.oracle file have to be changed in order to deal
with the different severity codes that it receives from the snmptrap?
It seems like the way my snmptt.conf.oracle file is configured right
now, it would always  have a severity of "Normal", so Xymon would never
change colors.
 
I'm just learning SNMP and don't completely understand it yet, so maybe
I'm missing something.
 
 
I'll have to check with my DBA's again to find out if they can send
their traps directly from the database server, rather than thru the Grid
server.
 
 
Thanks,
Nicole
 
 
 
From: FARRIOR, Andy [mailto:Andy.Farrior (at) victoriacollege.edu] 
Sent: Friday, October 15, 2010 12:48 PM
To: xymon (at) xymon.com
Subject: [xymon] RE: help setting up SNMP traps to Xymon
 
1 - clearing the trap.
 
Unfortunately, the only way the SNMP trap status changes on Xymon is
when it receives a new SNMP trap from the device.  If the device would
send an "all clear" type of trap when it was happy, that'd change the
status back to green.  If there's a way to create an event on the device
that generates a "Normal" event, you could use that as a mechanism to
change the SNMP status on Xymon.   (Example, if saving the configuration
generates a "Normal" SNMP trap....)  Ugly, but it'd work.
 
The other way to address it would be to come up with a way to manually
acknowledge or clear a trap similar to the alert acknowledgment in
Xymon.  I've never had time to pursue that.
 
2 - origination of the trap
 
The only way I can think of addressing this is if each database server
can be configured to send SNMP traps on their own.  The SNMPTRAPD daemon
uses the sending device's hostname and that's what is used to send to
Xymon. 
Hope this helps,
Andy
 
 
 
From: Nicole Beck [mailto:nskyrca (at) syr.edu] 
Sent: Friday, October 15, 2010 9:42 AM
To: 'xymon (at) xymon.com'
Subject: [xymon] help setting up SNMP traps to Xymon
 
Hello,
I'm running Xymon 4.2.3 on RHEL 5.4 and I'm trying to setup Xymon to
receive SNMP traps from our Oracle database servers.    We are also
running Oracle Enterprise Manager on another server (we call it the
"grid") that monitors the Oracle database servers too, but our DBA's
want to use Xymon to alert them of Oracle problems instead of the grid
server so they don't get alerts from two different places.   The DBA's
manage OEM(therefore I'm not familiar with it), and have configured it
to recieve SNMP traps and send them to Xymon.
 
I configured the Xymon server to receive traps using the documentation
at http://cerebro.victoriacollege.edu/hobbit-trap.html.  We do receive
traps, but there are two problems: 
 
1.	The trap status diamond does not change color when it gets a
trap or when the trap clears.  My understanding is that this is set in
the Oracle MIBS that are translated to the /etc/snmp/snmptt.conf.oracle
file.  I'm not sure how to change the file to get it to give different
alerts.  If I change the word "NORMAL" in the following line to "SEVERE"
the trap status will go red, but will not return to green when the trap
clears. Any suggestions on how to fix this?
 
EVENT oraEM4Alert .1.3.6.1.4.1.111.15.2.0.1 "Status Events" Normal
 
2.	The traps status changes for the grid server instead of the
database server with the problem.  Drilling down into the "trap" status,
you will see an error for the database server though.   The
snmptrapd.log and snmptt.log files (examples below) show the traps
coming from the grid server.  Does anyone know how to get the traps to
appear under the database server name rather than the grid server?
 
Example snmptrapd.log entry (host name changed):
2010-10-14 19:04:49 grid-server [xxx.xxx.xxx.xxx] (via UDP:
[xxx.xxx.xxx.xxx]:51617) TRAP, SNMP v1, community public
        .1.3.6.1.4.1.111.15.2 Enterprise Specific Trap (1) Uptime:
0:06:31.00
        .1.3.6.1.4.1.111.15.1.1.1.2.1 = STRING: "DB00157.world"
.1.3.6.1.4.1.111.15.1.1.1.3.1 = STRING: "Database Instance"
.1.3.6.1.4.1.111.15.1.1.1.4.1 = STRING: "database-server "
.1.3.6.1.4.1.111.15.1.1.1.5.1 = STRING:
"Status".1.3.6.1.4.1.111.15.1.1.1.6.1 = ""
.1.3.6.1.4.1.111.15.1.1.1.7.1 = ""     .1.3.6.1.4.1.111.15.1.1.1.8.1 =
STRING: "Oct 14, 2010 7:04:00 PM EDT"    .1.3.6.1.4.1.111.15.1.1.1.9.1 =
STRING: "Critical"      .1.3.6.1.4.1.111.15.1.1.1.10.1 = STRING: "Failed
to connect to database instance: ORA-01033: ORACLE initialization or
shutdown in progress (DBD ERROR: OCISessionBegin)."
.1.3.6.1.4.1.111.15.1.1.1.11.1 = STRING: "Test SNMP database-server DB
Down "       .1.3.6.1.4.1.111.15.1.1.1.12.1 = STRING: "SYSMAN"
.1.3.6.1.4.1.111.15.1.1.1.13.1 = STRING: "0"
.1.3.6.1.4.1.111.15.1.1.1.14.1 = ""     .1.3.6.1.4.1.111.15.1.1.1.15.1 =
STRING: "923A730174F36B87E040E6801417642D"
.1.3.6.1.4.1.111.15.1.1.1.16.1 = STRING: "0"
.1.3.6.1.4.1.111.15.1.1.1.17.1 = ""     .1.3.6.1.4.1.111.15.1.1.1.18.1 =
STRING: "0"    .1.3.6.1.4.1.111.15.1.1.1.19.1 = ""
.1.3.6.1.4.1.111.15.1.1.1.20.1 = STRING: "1"
.1.3.6.1.4.1.111.15.1.1.1.21.1 = STRING:
"923A421065A8A060E040E68015173993"
 
Example snmptt.log entry:
Oct 14 19:04:52 xymon-server snmptt[0]: .1.3.6.1.4.1.111.15.2.0.1 SEVERE
"Status Events" grid-server - The variables included in the oraEM4Alert
trap. DB00157.world Database Instance database-server Status   Oct 14,
2010 7:04:00 PM EDT Critical Failed to connect to database instance:
ORA-01033: ORACLE initialization or shutdown in progress (DBD ERROR:
OCISessionBegin). Test SNMP database-server DB Down  SYSMAN 0
923A730174F36B87E040E6801417642D 0  0  1
923A421065A8A060E040E68015173993
 
 
 
Thanks in advance for any help.  I'm sorry for the lengthy question.
 
Nicole Beck