When running hobbit server 4.2.0 on sparc Solaris 10 with ZFS enabled, the ZFS data is a pair of mirror disks to hold hobbit server data.
This is the error messages I was seeing.
hobbit (at) server$ tail bb-display.log
2009-06-09 08:57:35 Whoops ! bb failed to send message - timeout
2009-06-09 08:57:35 hobbitd status-board not available
2009-06-09 11:28:19 Whoops ! bb failed to send message - timeout
2009-06-09 11:28:21 hobbitd status-board not available
2009-06-09 11:43:48 Whoops ! bb failed to send message - timeout
2009-06-09 11:43:48 hobbitd status-board not available
2009-06-09 12:29:05 Whoops ! bb failed to send message - timeout
2009-06-09 12:29:07 hobbitd status-board not available
2009-06-09 13:18:33 Whoops ! bb failed to send message - timeout
2009-06-09 13:18:35 hobbitd status-board not available
hobbit (at) server$
And it was causing false PROC alerts got sent out due to process output got cut off early.
limiting ZFS physical memory maximum usage by setting ARC variable to 3G in /etc/system didn't help either.
The problem was fixed by disabling the ZFS by migrating from ZFS to SVM.
The kenerl phyical memory usage was down from 85% to 20% (of 8G). hobbit server is much stable under this setting.
The observation is that hobbit server doesn't work well under high phyical memory usage condition.
I am wondering if any other hobbit sites using Soalris 10+ZFS has similar problem.