UCMDB and UD Practitioners Forum (Previously CMS)
cancel

HP uCMDB : Timeout error and associated settings information required

Highlighted
Praveen_Shan
Collector

HP uCMDB : Timeout error and associated settings information required

HP uCMDB: 9.05.CUP12.351

DDM Content Pack: 11.08.802

 

Hi all,

 

I need to troubleshoot these errors on Host Networking by SNMP job:

 

<< Progress message, Severity: Warning>>
Job was running more than 900000 msec

 

Can the below setting on DiscoveryProbe.properties file be increased to control such warnings ?

# what is the maximum time(ms) that a worker thread can run, till it's considered stuck
appilog.agent.local.max.worker.runtime = 900000

 

Thanks,

Praveen

 

6 REPLIES
Dima Gomel
HPE Expert

Re: HP uCMDB : Timeout error and associated settings information required

Please do not increase the value of appilog.agent.local.max.worker.runtime without close attantion. At least do not increase it significantly.

Most of discovery jobs running scripts as separate (AKA remote) process. In some cases we are loosing control on remote proceses. This timeout used to mark working tread as dead.  After 9 (OOTB) "dead" treads the probe will be restarted, and as result all processes that were created from it will be killed as well.

Raising the timeout will  affect general discovery performance.

Please pay attention, that in some cases, due to slow networking or high CPU on destination or BIG amount of data that need to be populated back to UCMDB it could be beneficial to raise the setting. My suggestion is to raise the value to, lets say 1.5X from 900000ms and check if this solves the problem on specific destination. If not return the value back.

Regards
-Dmitry Gomel, PMP
If you find that this or any post resolves your issue, please be sure to mark it as an accepted solution.
Click the Like button at the bottom to say 'Thanks'.
Praveen_Shan
Collector

Re: HP uCMDB : Timeout error and associated settings information required

Appreciate your suggestion.
Thanks, Dima.
Milap
Regular Collector

Re: HP uCMDB : Timeout error and associated settings information required

Hi Dima,

 

Need your help in this case.I am facing the same issue in 2 probes."Network connectivity analyser job" stuck saying job is running more than 900000 msec.Need urgent resolution.Opened a case as well but not much luck

 

Warm regards,

Milap

eengstrom
Regular Collector

Re: HP uCMDB : Timeout error and associated settings information required

Folks,

 

Read this:

 

http://h30499.www3.hp.com/t5/CMS-and-Discovery-Support-and/HP-UCMDB-MySQL-Probe-Tuning-Solution/td-p/6470110

 

Second:

 

Your timeout is expiring because your probe is melting down. Extending the timeout expontentially will probably result in other, unconnected but still related timeouts in some other part of the system to expire. You could in theory, go around and quadruple all timeouts. This isn't going to make your job run faster, it will simply extend the period of time before the Probe considers these threads invalid and restarts the Probe.

Third: Go and read that link again, then register and download our Probe tuning package. You'll be surprised to see that your previously timing out job, with the right configuration and resources, will finish in ... oh, a lot less time.

If your jobs are timing out, I'd even say that they aren't even running. If you stack a lot of work on top of the Probe with limited memory and out of the box configuration, your Probe MySQL database will jam up just trying to RECEIVE the job from UCMDB. Not on day 1, but after you've had a few runs succeed and your MySQL database has a bit of size.

This isn't a discovery job or UCMDB problem, it's a Probe MySQL database configuration problem and it has been solved.

 

Erik

effectualsystems.com
Milap
Regular Collector

Re: HP uCMDB : Timeout error and associated settings information required

Hi ,

 

Thanks for the response.We have installaed 10.10 .Will it compatible with that?Moreover I tried to download but it saying the submittion as been marked as Spam.

 

post the zip if possible here.


Warm regards,

Milap

eengstrom
Regular Collector

Re: HP uCMDB : Timeout error and associated settings information required

This is deeply troubling that you're seeing this problem with 10.10.

 

As to the download, it wil not help you with 10.10 probes running PostgreSQL. We have a simple SPAM script in place to prevent aggressive bots from registering. It also blocks free email addresses. If you can submit the form with our information and a non-free email address, you should be able to get to the file.

 

We do not have a 10.10 production environment with production data and have not even started looking at 10.10 PostgreSQL probe  configurations. The issue with large, long running jobs on the MySQL instance is the amount of overhead on the Probe and the back/forth processing through a very constrained resource - key buffer cache.

 

You posted to threads that are affected by it, so I suspect either there is a similar constraint driving the same behavior in the PostgreSQL configuration (10.10 probe) or there is a similar symptom to a new problem or you have a third, not defined problem occuring.

 

Can you share snippets of the job exection from the wrapperprobegw.log file on the Probe? (Assuming it's still in use in 10.10) You can search for the start of the job that is timing out. If you can post the 100 or so lines below the job start, it will help us understand what is actually happening.

Also if you can log into the JMX console on the 10.10 probe and send a screenshot of the Job from JobsInformation:viewJobStatuses, this would help us understand what is stacking up on the Probe. (This all exists in 10.01 so assuming it still persists in 10.10).

 

Erik

 

effectualsystems.com
//Add this to "OnDomLoad" event