[hobbit] rrd-data.log
Brodie, Kent
brodie at mcw.edu
Wed Aug 2 16:48:55 CEST 2006
Aha...! I asked this very question last week- nobody gave me help...
(sob, sob..).
Anyway, here are two separate chunks of data that is CAUSING the
duplicate data error like Beau is having.
Henrik, what in this data is considered to be "duplicate"? (I do
notice that the netstat data in AIX is um... QUITE verbose........)
Data segments follow. Help?
@@data#810|1154529466.624163|999.888.777.202||fred.moo.mcw.edu|ifstat
data fred,moo,mcw,edu.ifstat
aix
-------------------------------------------------------------
ETHERNET STATISTICS (ent0) :
Device Type: 10/100 Mbps Ethernet PCI Adapter II (1410ff01)
Hardware Address: 00:02:55:4f:a9:f4
Elapsed Time: 0 days 0 hours 0 minutes 0 seconds
Transmit Statistics: Receive Statistics:
-------------------- -------------------
Packets: 0 Packets: 0
Bytes: 0 Bytes: 0
Interrupts: 2 Interrupts: 0
Transmit Errors: 0 Receive Errors: 0
Packets Dropped: 0 Packets Dropped: 0
Bad Packets: 0
Max Packets on S/W Transmit Queue: 0
S/W Transmit Queue Overflow: 0
Current S/W+H/W Transmit Queue Length: 0
Broadcast Packets: 0 Broadcast Packets: 0
Multicast Packets: 0 Multicast Packets: 0
No Carrier Sense: 0 CRC Errors: 0
DMA Underrun: 0 DMA Overrun: 0
Lost CTS Errors: 0 Alignment Errors: 0
Max Collision Errors: 0 No Resource Errors: 0
Late Collision Errors: 0 Receive Collision Errors:
0
Deferred: 0 Packet Too Short Errors: 0
SQE Test: 0 Packet Too Long Errors: 0
Timeout Errors: 0 Packets Discarded by
Adapter: 0
Single Collision Count: 0 Receiver Start Count: 0
Multiple Collision Count: 0
Current HW Transmit Queue Length: 0
General Statistics:
-------------------
No mbuf Errors: 0
Adapter Reset Count: 0
Adapter Data Rate: 200
Driver Flags: Up Broadcast Running
Simplex AlternateAddress 64BitSupport
ChecksumOffload PrivateSegment LargeSend
DataRateSet
10/100 Mbps Ethernet PCI Adapter II (1410ff01) Specific Statistics:
--------------------------------------------------------------------
Link Status: Down
Media Speed Selected: Auto negotiation
Media Speed Running: Unknown
Receive Pool Buffer Size: 1024
Free Receive Pool Buffers: 1024
No Receive Pool Buffer Errors: 0
Receive Buffer Too Small Errors: 0
Entries to transmit timeout routine: 0
Transmit IPsec packets: 0
Transmit IPsec packets dropped: 0
Receive IPsec packets: 0
Receive IPsec packets dropped: 0
Inbound IPsec SA offload count: 0
Transmit Large Send packets: 0
Transmit Large Send packets dropped: 0
Packets with Transmit collisions:
1 collisions: 0 6 collisions: 0 11 collisions: 0
2 collisions: 0 7 collisions: 0 12 collisions: 0
3 collisions: 0 8 collisions: 0 13 collisions: 0
4 collisions: 0 9 collisions: 0 14 collisions: 0
5 collisions: 0 10 collisions: 0 15 collisions: 0
-------------------------------------------------------------
ETHERNET STATISTICS (ent1) :
Device Type: 10/100 Mbps Ethernet PCI Adapter II (1410ff01)
Hardware Address: 00:02:55:4f:a9:f3
Elapsed Time: 0 days 0 hours 0 minutes 0 seconds
Transmit Statistics: Receive Statistics:
-------------------- -------------------
Packets: 0 Packets: 0
Bytes: 0 Bytes: 0
Interrupts: 2 Interrupts: 0
Transmit Errors: 0 Receive Errors: 0
Packets Dropped: 0 Packets Dropped: 0
Bad Packets: 0
Max Packets on S/W Transmit Queue: 0
S/W Transmit Queue Overflow: 0
Current S/W+H/W Transmit Queue Length: 0
Broadcast Packets: 0 Broadcast Packets: 0
Multicast Packets: 0 Multicast Packets: 0
No Carrier Sense: 0 CRC Errors: 0
DMA Underrun: 0 DMA Overrun: 0
Lost CTS Errors: 0 Alignment Errors: 0
Max Collision Errors: 0 No Resource Errors: 0
Late Collision Errors: 0 Receive Collision Errors:
0
Deferred: 0 Packet Too Short Errors: 0
SQE Test: 0 Packet Too Long Errors: 0
Timeout Errors: 0 Packets Discarded by
Adapter: 0
Single Collision Count: 0 Receiver Start Count: 0
Multiple Collision Count: 0
Current HW Transmit Queue Length: 0
General Statistics:
-------------------
No mbuf Errors: 0
Adapter Reset Count: 0
Adapter Data Rate: 200
Driver Flags: Up Broadcast Running
Simplex AlternateAddress 64BitSupport
ChecksumOffload PrivateSegment LargeSend
DataRateSet
10/100 Mbps Ethernet PCI Adapter II (1410ff01) Specific Statistics:
--------------------------------------------------------------------
Link Status: Down
Media Speed Selected: Auto negotiation
Media Speed Running: Unknown
Receive Pool Buffer Size: 1024
Free Receive Pool Buffers: 1024
No Receive Pool Buffer Errors: 0
Receive Buffer Too Small Errors: 0
Entries to transmit timeout routine: 0
Transmit IPsec packets: 0
Transmit IPsec packets dropped: 0
Receive IPsec packets: 0
Receive IPsec packets dropped: 0
Inbound IPsec SA offload count: 0
Transmit Large Send packets: 0
Transmit Large Send packets dropped: 0
Packets with Transmit collisions:
1 collisions: 0 6 collisions: 0 11 collisions: 0
2 collisions: 0 7 collisions: 0 12 collisions: 0
3 collisions: 0 8 collisions: 0 13 collisions: 0
4 collisions: 0 9 collisions: 0 14 collisions: 0
5 collisions: 0 10 collisions: 0 15 collisions: 0
-------------------------------------------------------------
ETHERNET STATISTICS (ent2) :
Device Type: 10/100/1000 Base-TX PCI-X Adapter (14106902)
Hardware Address: 00:02:55:53:c2:3e
Elapsed Time: 196 days 15 hours 56 minutes 59 seconds
Transmit Statistics: Receive Statistics:
-------------------- -------------------
Packets: 365802341 Packets: 1036460447
Bytes: 1683156286637 Bytes: 112378387074
Interrupts: 0 Interrupts: 614513005
Transmit Errors: 0 Receive Errors: 0
Packets Dropped: 0 Packets Dropped: 0
Bad Packets: 0
Max Packets on S/W Transmit Queue: 30
S/W Transmit Queue Overflow: 0
Current S/W+H/W Transmit Queue Length: 0
Broadcast Packets: 16681 Broadcast Packets:
210557123
Multicast Packets: 0 Multicast Packets: 283080
No Carrier Sense: 0 CRC Errors: 0
DMA Underrun: 0 DMA Overrun: 0
Lost CTS Errors: 0 Alignment Errors: 0
Max Collision Errors: 0 No Resource Errors: 0
Late Collision Errors: 0 Receive Collision Errors:
0
Deferred: 381 Packet Too Short Errors: 0
SQE Test: 0 Packet Too Long Errors: 0
Timeout Errors: 0 Packets Discarded by
Adapter: 0
Single Collision Count: 0 Receiver Start Count: 0
Multiple Collision Count: 0
Current HW Transmit Queue Length: 0
General Statistics:
-------------------
No mbuf Errors: 0
Adapter Reset Count: 0
Adapter Data Rate: 2000
Driver Flags: Up Broadcast Running
Simplex 64BitSupport ChecksumOffload
PrivateSegment LargeSend DataRateSet
10/100/1000 Base-TX PCI-X Adapter (14106902) Specific Statistics:
--------------------------------------------------------------------
Link Status: Up
Media Speed Selected: Auto negotiation
Media Speed Running: 1000 Mbps Full Duplex
PCI Mode: PCI-X (100-133)
PCI Bus Width: 64-bit
Latency Timer: 144
Cache Line Size: 128
Jumbo Frames: Disabled
TCP Segmentation Offload: Enabled
TCP Segmentation Offload Packets Transmitted: 113291719
TCP Segmentation Offload Packet Errors: 0
Transmit and Receive Flow Control Status: Enabled
XON Flow Control Packets Transmitted: 0
XON Flow Control Packets Received: 430
XOFF Flow Control Packets Transmitted: 0
XOFF Flow Control Packets Received: 430
Transmit and Receive Flow Control Threshold (High): 45056
Transmit and Receive Flow Control Threshold (Low): 24576
Transmit and Receive Storage Allocation (TX/RX): 16/48
@@
@@data#811|1154529466.624871|999.777.666.202||phred.mrr.mcw.edu|netstat
data phred,mrr,mcw,edu.netstat
aix
icmp:
597 calls to icmp_error
0 errors not generated because old message was icmp
Output histogram:
echo reply: 58031
destination unreachable: 537
1 message with bad code fields
0 messages < minimum length
0 bad checksums
0 messages with bad length
Input histogram:
echo reply: 2
destination unreachable: 562
echo: 58031
time exceeded: 12
58031 message responses generated
igmp:
282998 messages received
0 messages received with too few bytes
0 messages received with bad checksum
282998 membership queries received
0 membership queries received with invalid field(s)
0 membership reports received
0 membership reports received with invalid field(s)
0 membership reports received for groups to which we belong
2 membership reports sent
tcp:
449406211 packets sent
419904427 data packets (1612944301 bytes)
176131 data packets (218422100 bytes) retransmitted
20439309 ack-only packets (8520363 delayed)
6 URG only packets
420337 window probe packets
2131650 window update packets
6334351 control packets
113291719 large sends
3038842766 bytes sent using largesend
64240 bytes is the biggest largesend
909296772 packets received
857387271 acks (for 1835107293 bytes)
7844383 duplicate acks
0 acks for unsent data
307390334 packets (2464690667 bytes) received
in-sequence
45039 completely duplicate packets (1602267 bytes)
0 old duplicate packets
5 packets with some dup. data (624 bytes duped)
2441887 out-of-order packets (440550 bytes)
8 packets (8 bytes) of data after window
8 window probes
2587766 window update packets
3902 packets received after close
0 packets with bad hardware assisted checksum
0 discarded for bad checksums
0 discarded for bad header offset fields
0 discarded because packet too short
1440 discarded by listeners
0 discarded due to listener's queue full
23840405 ack packet headers correctly predicted
35343642 data packet headers correctly predicted
399880 connection requests
5548229 connection accepts
5947965 connections established (including accepts)
5953023 connections closed (including 25563 drops)
0 connections with ECN capability
0 times responded to ECN
137 embryonic connections dropped
404559742 segments updated rtt (of 404578267 attempts)
0 segments with congestion window reduced bit set
0 segments with congestion experienced bit set
0 resends due to path MTU discovery
10019 path MTU discovery terminations due to retransmits
29115 retransmit timeouts
1 connection dropped by rexmit timeout
2 fast retransmits
0 when congestion window less than 4 segments
3 newreno retransmits
0 times avoided false fast retransmits
427603 persist timeouts
0 connections dropped due to persist timeout
8957 keepalive timeouts
8928 keepalive probes sent
29 connections dropped by keepalive
0 times SACK blocks array is extended
0 times SACK holes array is extended
0 packets dropped due to memory allocation failure
0 connections in timewait reused
0 delayed ACKs for SYN
0 delayed ACKs for FIN
0 send_and_disconnects
0 spliced connections
0 spliced connections closed
0 spliced connections reset
0 spliced connections timeout
0 spliced connections persist timeout
0 spliced connections keepalive timeout
udp:
13767017 datagrams received
0 incomplete headers
0 bad data length fields
0 bad checksums
597 dropped due to no socket
6762980 broadcast/multicast datagrams dropped due to no socket
0 dropped due to full socket buffers
7003440 delivered
6994616 datagrams output
ip:
923406868 total packets received
0 bad header checksums
0 with size smaller than minimum
0 with data size < data length
0 with header length < data size
0 with data length < header length
0 with bad options
0 with incorrect version number
0 fragments received
0 fragments dropped (dup or out of space)
0 fragments dropped after timeout
0 packets reassembled ok
923121828 packets for this host
283574 packets for unknown/unsupported protocol
0 packets forwarded
1396 packets not forwardable
0 redirects sent
456499667 packets sent from this host
0 packets sent with fabricated ip header
0 output packets dropped due to no bufs, etc.
0 output packets discarded due to no route
0 output datagrams fragmented
0 fragments created
0 datagrams that can't be fragmented
69 IP Multicast packets dropped due to no receiver
0 successful path MTU discovery cycles
0 path MTU rediscovery cycles attempted
0 path MTU discovery no-response estimates
0 path MTU discovery response timeouts
0 path MTU discovery decreases detected
0 path MTU discovery packets sent
0 path MTU discovery memory allocation failures
0 ipintrq overflows
0 with illegal source
0 packets processed by threads
0 packets dropped by threads
0 packets dropped due to the full socket receive buffer
0 dead gateway detection packets sent
0 dead gateway detection packet allocation failures
0 dead gateway detection gateway allocation failures
ipv6:
3 total packets received
0 with size smaller than minimum
0 with data size < data length
0 with incorrect version number
0 with illegal source
0 input packets without enough memory
0 fragments received
0 fragments dropped (dup or out of space)
0 fragments dropped after timeout
0 packets reassembled ok
0 packets for this host
0 packets for unknown/unsupported protocol
0 packets forwarded
3 packets not forwardable
0 too big packets not forwarded
0 packets sent from this host
0 packets sent with fabricated ipv6 header
0 output packets dropped due to no bufs
0 output packets without enough memory
0 output packets discarded due to no route
0 output datagrams fragmented
0 fragments created
0 packets dropped due to full socket receive buffer
0 packets not delivered due to bad raw IPv6 checksum
icmpv6:
0 calls to icmp6_error
0 errors not generated because old message was icmpv6
Output histogram:
unreachable: 0
packets too big: 0
time exceeded: 0
parameter problems: 0
redirects: 0
echo requests: 0
echo replies: 0
group queries: 0
group reports: 0
group terminations: 0
router solicitations: 0
router advertisements: 0
neighbor solicitations: 0
neighbor advertisements: 0
0 messages with bad code fields
0 messages < minimum length
0 bad checksums
0 messages with bad length
Input histogram:
unreachable: 0
packets too big: 0
time exceeded: 0
parameter problems: 0
echo requests: 0
echo replies: 0
group queries: 0
bad group queries: 0
group reports: 0
bad group reports: 0
our groups' reports: 0
group terminations: 0
bad group terminations: 0
router solicitations: 0
bad router solicitations: 0
router advertisements: 0
bad router advertisements: 0
neighbor solicitations: 0
bad neighbor solicitations: 0
neighbor advertisements: 0
bad neighbor advertisements: 0
redirects: 0
bad redirects: 0
mobility calls when not started: 0
home agent address discovery requests: 0
bad home agent address discovery requests: 0
bad home agent address discovery replys: 0
bad home agent address discovery replys: 0
prefix solicitations: 0
bad prefix solicitations: 0
prefix advertisements: 0
bad prefix advertisements: 0
0 message responses generated
@@
@@data#1228|1154529774.054469|141.106.224.202||jordan.hmgc.mcw.edu|netst
at
data jordan,hmgc,mcw,edu.netstat
aix
icmp:
29 calls to icmp_error
0 errors not generated because old message was icmp
Output histogram:
echo reply: 16024
destination unreachable: 17
45 messages with bad code fields
0 messages < minimum length
0 bad checksums
0 messages with bad length
Input histogram:
echo reply: 7
destination unreachable: 58
echo: 16024
16024 message responses generated
igmp:
79096 messages received
0 messages received with too few bytes
0 messages received with bad checksum
79096 membership queries received
0 membership queries received with invalid field(s)
0 membership reports received
0 membership reports received with invalid field(s)
0 membership reports received for groups to which we belong
2 membership reports sent
tcp:
256987083 packets sent
233786265 data packets (146658044 bytes)
30885 data packets (6116690 bytes) retransmitted
16701059 ack-only packets (3980875 delayed)
0 URG only packets
193 window probe packets
17752 window update packets
6450950 control packets
45166554 large sends
740111354 bytes sent using largesend
64240 bytes is the biggest largesend
496262560 packets received
450701474 acks (for 151098725 bytes)
9591183 duplicate acks
0 acks for unsent data
195459972 packets (949784396 bytes) received in-sequence
23755 completely duplicate packets (8371 bytes)
0 old duplicate packets
0 packets with some dup. data (0 bytes duped)
3375737 out-of-order packets (104293 bytes)
0 packets (0 bytes) of data after window
0 window probes
4869090 window update packets
3508 packets received after close
0 packets with bad hardware assisted checksum
0 discarded for bad checksums
0 discarded for bad header offset fields
0 discarded because packet too short
1117 discarded by listeners
0 discarded due to listener's queue full
10028924 ack packet headers correctly predicted
24673885 data packet headers correctly predicted
100631 connection requests
6257521 connection accepts
6358083 connections established (including accepts)
6359382 connections closed (including 24024 drops)
0 connections with ECN capability
0 times responded to ECN
65 embryonic connections dropped
240375423 segments updated rtt (of 240404633 attempts)
0 segments with congestion window reduced bit set
0 segments with congestion experienced bit set
0 resends due to path MTU discovery
14379 path MTU discovery terminations due to retransmits
31032 retransmit timeouts
5 connections dropped by rexmit timeout
3 fast retransmits
0 when congestion window less than 4 segments
10 newreno retransmits
5 times avoided false fast retransmits
195 persist timeouts
0 connections dropped due to persist timeout
1407 keepalive timeouts
1393 keepalive probes sent
14 connections dropped by keepalive
0 times SACK blocks array is extended
0 times SACK holes array is extended
0 packets dropped due to memory allocation failure
0 connections in timewait reused
0 delayed ACKs for SYN
0 delayed ACKs for FIN
0 send_and_disconnects
0 spliced connections
0 spliced connections closed
0 spliced connections reset
0 spliced connections timeout
0 spliced connections persist timeout
0 spliced connections keepalive timeout
udp:
6773177 datagrams received
0 incomplete headers
0 bad data length fields
0 bad checksums
29 dropped due to no socket
471205 broadcast/multicast datagrams dropped due to no socket
0 socket buffer overflows
6301943 delivered
6301977 datagrams output
ip:
504670433 total packets received
0 bad header checksums
0 with size smaller than minimum
0 with data size < data length
0 with header length < data size
0 with data length < header length
0 with bad options
0 with incorrect version number
0 fragments received
0 fragments dropped (dup or out of space)
0 fragments dropped after timeout
0 packets reassembled ok
503051772 packets for this host
79154 packets for unknown/unsupported protocol
0 packets forwarded
1539510 packets not forwardable
0 redirects sent
263316001 packets sent from this host
0 packets sent with fabricated ip header
0 output packets dropped due to no bufs, etc.
0 output packets discarded due to no route
0 output datagrams fragmented
0 fragments created
0 datagrams that can't be fragmented
0 IP Multicast packets dropped due to no receiver
0 successful path MTU discovery cycles
0 path MTU rediscovery cycles attempted
0 path MTU discovery no-response estimates
0 path MTU discovery response timeouts
0 path MTU discovery decreases detected
0 path MTU discovery packets sent
0 path MTU discovery memory allocation failures
0 ipintrq overflows
0 with illegal source
0 packets processed by threads
0 packets dropped by threads
0 packets dropped due to the full socket receive buffer
0 dead gateway detection packets sent
0 dead gateway detection packet allocation failures
0 dead gateway detection gateway allocation failures
ipv6:
0 total packets received
0 with size smaller than minimum
0 with data size < data length
0 with incorrect version number
0 with illegal source
0 input packets without enough memory
0 fragments received
0 fragments dropped (dup or out of space)
0 fragments dropped after timeout
0 packets reassembled ok
0 packets for this host
0 packets for unknown/unsupported protocol
0 packets forwarded
0 packets not forwardable
0 too big packets not forwarded
0 packets sent from this host
0 packets sent with fabricated ipv6 header
0 output packets dropped due to no bufs
0 output packets without enough memory
0 output packets discarded due to no route
0 output datagrams fragmented
0 fragments created
0 packets dropped due to full socket receive buffer
0 packets not delivered due to bad raw IPv6 checksum
icmpv6:
0 calls to icmp6_error
0 errors not generated because old message was icmpv6
Output histogram:
unreachable: 0
packets too big: 0
time exceeded: 0
parameter problems: 0
redirects: 0
echo requests: 0
echo replies: 0
group queries: 0
group reports: 0
group terminations: 0
router solicitations: 0
router advertisements: 0
neighbor solicitations: 0
neighbor advertisements: 0
0 messages with bad code fields
0 messages < minimum length
0 bad checksums
0 messages with bad length
Input histogram:
unreachable: 0
packets too big: 0
time exceeded: 0
parameter problems: 0
echo requests: 0
echo replies: 0
group queries: 0
bad group queries: 0
group reports: 0
bad group reports: 0
our groups' reports: 0
group terminations: 0
bad group terminations: 0
router solicitations: 0
bad router solicitations: 0
router advertisements: 0
bad router advertisements: 0
neighbor solicitations: 0
bad neighbor solicitations: 0
neighbor advertisements: 0
bad neighbor advertisements: 0
redirects: 0
bad redirects: 0
mobility calls when not started: 0
home agent address discovery requests: 0
bad home agent address discovery requests: 0
bad home agent address discovery replys: 0
bad home agent address discovery replys: 0
prefix solicitations: 0
bad prefix solicitations: 0
prefix advertisements: 0
bad prefix advertisements: 0
0 message responses generated
@@
---------------------------------------------------------
Kent C. Brodie - brodie at phys.mcw.edu
Department of Physiology
Medical College of Wisconsin
(414) 456-8590
-----Original Message-----
From: Henrik Stoerner [mailto:henrik at hswn.dk]
Sent: Wednesday, August 02, 2006 5:59 AM
To: hobbit at hswn.dk
Subject: Re: [hobbit] rrd-data.log
On Wed, Aug 02, 2006 at 12:23:53PM +0200, Beau Olivier wrote:
>
> I'm having "Internal error: Duplicate match ignored" in my
rrd-data.log,
> what could cause this ?
It means your netstat data doesn't look like what Hobbit expects.
Basically that it found two or more values for the same piece of data.
The best way of identifying which data causes this is probably to
run two things at the same time:
1) login as the hobbit user, and run
bbcmd hobbitd_channel --channel=data tee /tmp/data.log
2) Run "tail -f" on the rrd-data.log file.
When you see that error message in the rrd-data.log file, terminate
the first command. You should then have the "guilty" data at the end of
the /tmp/data.log file.
I'd obviously be interested to see what it looks like.
Regards,
Henrik
To unsubscribe from the hobbit list, send an e-mail to
hobbit-unsubscribe at hswn.dk
More information about the Xymon
mailing list