[hobbit] Is there a limit on the number of hosts that can polled?

[Shawn Heisey]

I would agree with this, the disk subsystem is probably unable to keep 
up with the I/O load.  Use "iostat 30" or "vmstat 30" to determine 
iowait percentage, which is probably very high.  To fix it, get rid of 
any raid5/6 (even if handled by a dedicated controller) or LVM, and 
possibly use faster disks.  The best balance between performance and 
data redundancy is raid10, but obviously it costs more because there are 
more disks.  For write-intensive tasks like this, even JBOD is a better 
performance option than raid5.  Because I never use it, I don't really 
know why LVM causes problems, but I know from others' experience that it 
does.

The problem with raid5 and raid6 is that there's a write penalty due to 
the need to calculate and write parity data.  A good controller with 
memory for write caching can mitigate this in many typical 
circumstances, but only if the entire transaction can fit in the cache 
memory and can be flushed to disk before another data flood comes in.  
In this case, it takes about 2700 hosts to generate more data than the 
system can write before more arrives.

Brian Catlin wrote:
> As nobody took a shot at this,   While you are ok on memory and CPU - 
> have you looked at your other resources?  With that many hosts 
> reporting back to a master - I would suspect I/O flooding off your 
> interface...
>  
> Just a thought ....
>  
>