[Xymon] HPE BL460c Gen9 SNMP tests going clear
colin.coe at gmail.com
Thu Mar 14 01:19:05 CET 2019
Sorry, I had meant to add that all our HPE Proliant Gen7s are reporting
fine, its only the Gen9 servers that have this problem.
On Thu, Mar 14, 2019 at 8:11 AM Colin Coe <colin.coe at gmail.com> wrote:
> Hi Jeremy
> Looks like I've had similar issues in the past that I'd completely
> forgotten about... I've already got
> "$SNMP_Session::default_avoid_negative_request_ids = 1;"
> in /usr/share/perl5/SNMP_Session.pm
> While multiple servers are affected, they don't always go clear at the
> same time.
> I've changed the snmpver from 2 to 1 but that didn't improve things.
> having tcpdump running in another ssh session, I can see that requests are
> being send and responses are being received. I see no negative requests,
> and ofen the Devmon tests go clear even though SNMP data is received. The
> devmon.log file reflects devmon's view, not the reality of the SNMP
> I'm thinking about using the baked in Xymon SNMP tests but how would I
> translate the Proliant server tests, for example
> Thanks for the pointers
> On Wed, Mar 13, 2019 at 6:38 PM Jeremy Laidman <jeremy at laidman.org> wrote:
>> Are all your different servers losing numbers all around the same time?
>> The fact that both Windows and Linux systems (with probably different snmpd
>> code) are having the same issues, suggests that it's a devmon problem.
>> It's been a while since I did any devmon stuff (mine has been running for
>> years but a low node/OID count seems to keep it humming along), but I seem
>> to recall there was some issue with 32-bit vs 64-bit SNMP request/response
>> IDs. What can happen is that a negative 32-bit request ID gets turned into
>> a positive and large 64-bit response ID that is actually the same, but
>> parsed as different by the SNMP library used by devmon. This might be a
>> problem only where there is a 32-bit SNMP library installed on a 64-bit OS,
>> or that might just be a red herring.
>> If this is a problem, a "tcpdump" with "-v" will show request/response
>> IDs, and you can confirm if you only get responses for positive request IDs.
>> There's an SNMP_Session API variable you can set in the devmon script to
>> avoid negative request IDs. See
>> Another issue might be the use of SNMPBulkGet, which results in lots of
>> packets in the response, all fragments of a large datagram. One day to
>> disable Bulk requests is to set SNMP version (snmpver) to 1 in your
>> devmon/templates/<devtype>/specs file. Most templates have "snmpver : 2".
>> The last time I tried to add a bunch of hosts to devmon, it failed, not
>> only for the hosts I added, but also for the hosts that were there before
>> the addition. I ended up reverse-engineering the as-yet-undocumented Xymon
>> SNMP features, and it seems to work well enough. I do sometimes get no
>> response from some hosts from time to time, but it's better than nothing.
>> In case you wanted to go down this path, it's just a matter of adding
>> entries into /etc/xymon/snmphosts.cfg like this:
>> The entries after "ip=" line are references to stanzas in the
>> /etc/xymon/snmpmibs.cfg file. There's special black magic in the
>> snmpmibs.cfg file that I haven't worked out, and probably needs code
>> inspection to understand, so I haven't dared to touch this file, with the
>> exception of one apparent typo; diff to fix it is here:
>> # storage has data for both memory- and disk-storage
>> keyidx (HOST-RESOURCES-MIB::hrStorageDescr)
>> - keyidx [(HOST-RESOURCES-MIB::hrStorageType]
>> + keyidx [HOST-RESOURCES-MIB::hrStorageType]
>> Type = HOST-RESOURCES-MIB::hrStorageType
>> Description = HOST-RESOURCES-MIB::hrStorageDescr
>> Units = HOST-RESOURCES-MIB::hrStorageAllocationUnits
>> Hmm, perhaps I should try setting "version=1" and see if that helps my
>> occasional missing data...
>> On Wed, 13 Mar 2019 at 17:12, Colin Coe <colin.coe at gmail.com> wrote:
>>> Hi Bruce and thanks for the quick response.
>>> These Gen9's are a mix of RHEL6 and Windows 2016, and interestingly,
>>> both behave the same. The RHEL boxen are running 'net-snmp' and for
>>> Windows 2016 it's the stock WIndows SNMP service. We don't do SNMP traps.
>>> These machines also have the HP Service Pack for Proliant installed. The
>>> RHEL config can be summarized as:
>>> cat /etc/snmp/snmpd.conf
>>> dlmod cmaX /usr/lib64/libcmaX64.so
>>> rocommunity secret
>>> Thanks again
>>> On Wed, Mar 13, 2019 at 2:02 PM Bruce Ferrell <bferrell at baywinds.org>
>>>> On 3/12/19 9:58 PM, Colin Coe wrote:
>>>> > Hi all
>>>> > All of our Gen9 servers are flicking between green and clear for
>>>> Devmon SNMP tests. (Posting here as Devmon seems a dead project.)
>>>> > The attached screen region grab shows what I'm trying to say.
>>>> > I've updated the firmware (including iLO) on one server and it made
>>>> no difference.
>>>> > Anyone else seen this?
>>>> > Thanks
>>>> > CC
>>>> > _______________________________________________
>>>> > Xymon mailing list
>>>> > Xymon at xymon.com
>>>> > http://lists.xymon.com/mailman/listinfo/xymon
>>>> I see a lot of odd things with devmon. I have seen an issue something
>>>> like on my Dell R710 with OMSA. It patches itself into snmpd via a loaded
>>>> library to report on Dell storage
>>>> and sometimes the response from that get's really slow and blocks
>>>> response to devmon. An update a while back to OMSA cleared that up.
>>>> I also have to run a cron job that does a kill -9 on it every two hours
>>>> and then restart it... It likes to gobble memory like it's going out of
>>>> People DO respond on the devmon list, just a little slowly.
>>>> I run my devmon in multi node mode with MySQL backing. when I do see
>>>> gross errors like this I turn on debug and watch the devmon log to see what
>>>> errors are thrown.
>>>> What OS and SNMPD is running on the HP? Especially the SNMPD... It
>>>> matters. OS X and pFSense don't report some OIDs correctly due to the
>>>> snmpd they run. It's one of those
>>>> "snmpd/ya just gotta know about it" things.
>>> Xymon mailing list
>>> Xymon at xymon.com
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Xymon