Project and Portfolio Management Practitioners Forum
cancel
Showing results for 
Search instead for 
Did you mean: 

PPM 7.5 benchmark

Highlighted
jerome.vovard
Regular Collector

PPM 7.5 benchmark

Hi all,
I'm using PPM 7.5 under AIX 5.3.
Sometimes, PPM is unavailable. We need to re-start PPM even if the kstatus result shows us that PPM is running (Checking rmi://omega.rm.corp.local:1099/KintanaServer
--> running (load: 363.0, mode: NORMAL).
In fact, PPM is slowing down until we can't do anything like open a request or modify a request.So we need to re-start PPM.
what kind of metrics could i have to check?
I think that PPM is available but the apache server (provided with PPM) doesn't work anymore. where could i find the aapache's logs?
Regards,

8 REPLIES
Erik Cole
Honored Contributor

Re: PPM 7.5 benchmark

Hi,

You can use a command such as 'top' to look at the performance (memory/cpu usage, etc) of the jvm processes. With a user load of 363, assuming that a good % of that is active users, you might be running out of memory. Any errors?

Are you running a single instance of PPM, or a cluster?
Jim Esler
Honored Contributor

Re: PPM 7.5 benchmark

Check the Broker Connection Report and the Execution Dispatcher Manager Report in the Server Tools workbench. Have you reached the configured limit for either? We have seen both limits exceeded during performance testing but never come close during real operations. The behavior when the limits are exceeded match your description.

Check the View Running Executions page on the Dashboard under Administration/Report Executions. Are there activities waiting to execute?

We use nmon to monitor server performance. CPU utilization and paging activity can indicate problem areas. Similar issues should be examined on the database server if the database is on separate hardware.
Erik Cole
Honored Contributor

Re: PPM 7.5 benchmark

Jim is right on about the Broker Connection Report. We had an incident recently where a jsp for a report was not releasing its connection when the report finished. When users ran the report a lot, it tied up all of our connections and eventually everything slowed to a halt waiting to talk to the db.
jerome.vovard
Regular Collector

Re: PPM 7.5 benchmark

Thank you Jim and Erik,
i have to tell you that we changed, on last thursday, the following PPM 'server.conf' parameters from 2560 to 4560:
SERVER_MAX_HEAP_SIZE
SERVER_INIT_HEAP_SIZE
It means that the JVM could use 4.5Gb of Heap. Our Unix server has 6Gb of Ram. With nmon, I saw the system uses 30% of Ram, the FScache uses 20% of Ram, and the Process uses 50% of Ram. When PPM crashed, the process used more than 70% of Ram (graph name: Memory Usage Percent (MEMNEW)).
Before the last thursday, PPM worked correctly but we're waiting for a peak of activity on the last day of june (Timesheet activity). And the response time of some portlets are equal to 15seconds. that's why we tried to modify the JVM memory.
Otherwise, we'll use PPM in cluster with a JVM to 1,7Gb.
During the crash, we can't open any workbench. The database is on another server.
well, I'm waiting for installing an external Apache to the AIX server of PPM.
An i'll come bac kto you.
Regards,
jerome.vovard
Regular Collector

Re: PPM 7.5 benchmark

Sorry , i just saw that I can get the Broker connection report, Broker performance report and so on using the kRunServerAdminReport.sh unix command provided by PPM under /bin
I just got a new crash of PPM. So I 'm gonna read those reports.
regards,
DIP_2
Super Collector

Re: PPM 7.5 benchmark

Hi jerome,

We are facing the similar issue. did your problem got resolved by changing the java heap size or did you done something else.
TapasK
Acclaimed Contributor

Re: PPM 7.5 benchmark

Hi ,

 

We faced similar similar issue for our environment.

We are also in PPm 7.5 wth IBM AIX.

After invetigation it was found that the DB connection was not getting released properly.

So after 4 month's of HP's research in this case they provided us a fix for this issue.

 

So please contact HP to provide you the fix which wil relelase your DB connection which are not getting released.

 

Thanks,

Tapas

philipwood
Regular Collector

Re: PPM 7.5 benchmark

Jerome,

 

Apache is probably fine.

 

That load shown is very high.

How many nodes are configured?

As a rule of thumb more than 50 concurrent users per node might start to be pushing it.

 If you run the server logon report, does it contain a lot of duplicates?

kRunServerAdminReport.sh 2

From experience when a node PPM is loaded past capacity onto failure it gets into a state where it will accept a logon and add the user to the load statistic, but not actually get to the point of servicing the user. When this happens the user logon reports shown large number of user logged on with many duplicates. The longer this node stays up the more the count will balloon past realistic values.

 

How many actual concurrent users do you have?

 

About database connections:

 

If you run the Broker performance report, does it contain nodes with no total available connections in any of the three connection categories (Kintana, ItgDS, DashboardDS)?

kRunServerAdminReport.sh 4

If there is a connection leak (something allocating a connection without releasing it) the available connection count will decrease with the node failing when none are available. This can be monitored as a stopgap measure with a node being restarted when the available connection count gets to be dangerously low.

 

Are any of the JVMs core dumping?

 

How much java heap is allocated per node?

 

Is there a node dedicated to services? In 7.5 the service node took a lot of strain and the work couldn't really be distributed to other nodes as it can be in 9.1 - is the node running services isolated from carrying user traffic in your configuration?

 

Kind Regards

 

Philip Wood

//Add this to "OnDomLoad" event