Project and Portfolio Management Practitioners Forum
cancel

Database collapse in HP PPM Version 8.0.0 due to triggering

Highlighted
PMOpartners
New Member.

Database collapse in HP PPM Version 8.0.0 due to triggering

About a week ago the HP PPM has been firing a trigger every 5-10 minutes leaving the hard drive out of space and finally crashing HP PPM in 1 day. We're having trouble identifying the trigger that is causing this and what is launching the trigger. Here is an extract of the database updates performed each 5-10 minutes:

 

insert into "PPM"."QRTZ_FIRED_TRIGGERS"("ENTRY_ID","TRIGGER_NAME","TRIGGER_GROUP","IS_VOLATILE","INSTANCE_NAME","FIRED_TIME","STATE","JOB_NAME","JOB_GROUP","IS_STATEFUL","REQUESTS_RECOVERY") values ('CTLMPPMPROD13472643622171347264363066','_TASK_SCHEDULER_SERVICE Trigger','PPM_SERVICES_TR','0','CTLMPPMPROD1347264362217','1347264363763','ACQUIRED',NULL,NULL,'0','0');                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                

insert into "PPM"."QRTZ_FIRED_TRIGGERS"("ENTRY_ID","TRIGGER_NAME","TRIGGER_GROUP","IS_VOLATILE","INSTANCE_NAME","FIRED_TIME","STATE","JOB_NAME","JOB_GROUP","IS_STATEFUL","REQUESTS_RECOVERY") values ('CTLMPPMPROD13472643622171347264363067','_TASK_SCHEDULER_SERVICE Trigger','PPM_SERVICES_TR','0','CTLMPPMPROD1347264362217','1347264363763','EXECUTING','_TASK_SCHEDULER_SERVICE','PPM_SERVICES','1','1');

 

update "PPM"."QRTZ_TRIGGERS" set "TRIGGER_STATE" = 'BLOCKED' where "TRIGGER_STATE" = 'ACQUIRED' and ROWID = 'AAAMvDAAGAAABSmAAA';

 

update "PPM"."QRTZ_TRIGGERS" set "TRIGGER_STATE" = 'ACQUIRED' where ROWID = 'AAAMvDAAGAAABSmAAA';

                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  

delete from "PPM"."QRTZ_FIRED_TRIGGERS" where ROWID = 'AAAMu6AAGAAABRdAAB';

2 REPLIES
Utkarsh_Mishra
Acclaimed Contributor.

Re: Database collapse in HP PPM Version 8.0.0 due to triggering

Stop the PPM services (stop service node and run only user node) and follow up with HP on this.

QRTZ_.. tables are used by PPM services management.
Cheers..
Utkarsh Mishra

-- Remember to give Kudos to answers! (click the KUDOS star)
Oscar_Pereira
Honored Contributor.

Re: Database collapse in HP PPM Version 8.0.0 due to triggering

I have seing similar issue when the db schema was not cloned properly.

 

***Is this issue happening after an upgrade to 8.0 or after cloning a working environment?

 

If this is a clone/copy database schema, you need to make sure that the Database schema is copied as recommended.

 

We need to copy the DB schema from an original DB schema on which the PPM is STOPPED

The information on how to copy the schema from PROD to the TEST server can be found in the Installation and Admin guide on page 394 http://support.openview.hp.com/selfsolve/document/KM1224696

 

Also if the original server was a cluster, and the copy server is intended to be a standalone configuration, you need to run a command called: KStandaloneDeploy.sh

 

**************


Whenever you see this problem occurring, run the following query:

 

select * from QRTZ_TRIGGERS where TRIGGER_STATE like 'BLOCKED';

 

See which services have issues.

 

Also here is some debug lines that could help PPM to show a better error trace in the logs:

 

STEPS:

 

1-Stop the PPM nodes

2-Go to the <PPM_HOME>/server/server_name/logs/serverLog.txt

Rename the existing “serverLog.txt” and create a new/empty “serverLog.txt” file.

(This is just to have a clean log, to see the PPM start up for the Service Scheduler...)

 

3-Now please add the following debug log settings to logging.conf which is located under <PPM_HOME>/conf folder.  You can add these lines at the very bottom of the logging.conf

 

com.kintana.core.logging.PRODUCT_FUNCTION_LOGGING_LEVEL = com.mercury.itg.core.jms.service,DEBUG

com.kintana.core.logging.PRODUCT_FUNCTION_LOGGING_LEVEL = com.mercury.itg.core.server.mdServices,DEBUG

com.kintana.core.logging.PRODUCT_FUNCTION_LOGGING_LEVEL = com.mercury.itg.core.scheduler,DEBUG

com.kintana.core.logging.PRODUCT_FUNCTION_LOGGING_LEVEL = com.mercury.itg.core.monitor.impl.ServerManagerImpl, DEBUG

com.kintana.core.logging.PRODUCT_FUNCTION_LOGGING_LEVEL = com.mercury.itg.core.jms.service.impl,DEBUG

com.kintana.core.logging.PRODUCT_FUNCTION_LOGGING_LEVEL = org.quartz.core,DEBUG

com.kintana.core.logging.PRODUCT_FUNCTION_LOGGING_LEVEL = org.quartz.core.QuartzScheduler, TRACE

com.kintana.core.logging.PRODUCT_FUNCTION_LOGGING_LEVEL = org.quartz.core.QuartzSchedulerThread, TRACE

 

 

In addition to these, the SYSTEM_THRESHOLD setting in logging.conf should be set to DEBUG.

 

4-Start the PPM server

5-Monitor the trace under <PPM_HOME>/server/server_name/logs/serverLog.txt

6-Look for errors after PPM start up, after word “Ready”.

 

 

DISABLE STEPS:

The debug settings can be disabled simply by setting SYSTEM_THRESHOLD back to ERROR. And putting in comments the other lines that were added.

 

 

Oscar Pereira