The community will be in read-only from Tuesday 11:59pm (PST) to Wednesday 7:30am (PST)
The community will be in read-only from Tuesday 11:59pm (PST) to Wednesday 7:30am (PST)
Service Desk Practitioners Forum
cancel
Showing results for 
Search instead for 
Did you mean: 

CRITICAL: Extreme bad performance using SDesk SP19

Highlighted
Juan Paulini
Super Collector

CRITICAL: Extreme bad performance using SDesk SP19

Hi all,
Actually we are having a extreme bad performance in our installation. Some clients take 45 sec. for just saving a service call!

We have Service Desk 4.5 SP 19, 4 windows boxes as application servers.

Right now, I now that there are lots of things that influence performance, but it seems to be a bug or some like this.

Other thing to note, it´s that our application servers some time cancelled queued tasks from de application pool, just because it seems that have reached some limit...

Another symptom it's that service desk clients (they are really heavy users) degrade their performance until it's almost impossible to use the application.

One workaround is restarting application servers, but we miss all the pending task...

Do you have any recomendation of where to start looking at? I've open an incident at HP, but I'm also trying to find some help here.

Any clue will be apreciated, links to information, known bugs..

Thanks in advance.
15 REPLIES
Tim Schmitt_4
Frequent Visitor

Re: CRITICAL: Extreme bad performance using SDesk SP19

What Service Desk patch did you upgrade from? When the system is slow, is the CPU usage on the any of the servers involved high? Do you have a lot of Service Call rules?

We upgraded from SP10 to SP17 and when we did, we noticed a similar problem with performance. Service Desk events (such as email notifications or status updates) seemed to queue at a certain point in the day and when they queued, the system became very slow for all users. We have a TON of Service Call rules and we've been working on reducing them while working with HP to find a solution. The one symptom we've seen is that the Service Desk application server process is not particularily busy even though tasks are queued. The question is why the server process is not working through the event queue.

I can verify that we've had the same problem restarting the application servers while tasks are pending, as it seems that we lose tasks during a restart.

Recently, we reduced the number of Service Call rules that creates scheduled tasks and we've seen a noticable improvement.
Tim Schmitt_4
Frequent Visitor

Re: CRITICAL: Extreme bad performance using SDesk SP19

Juan Paulini
Super Collector

Re: CRITICAL: Extreme bad performance using SDesk SP19

Thanks for you reply,

We upgraded from SP15. We also have tons of rules, and we are loking at them, specialy we`ve found that change flow trigger several rules that queue waiting for some condition related to date.

Another topic that I know that can be troublesome is email notifications, service calls trigger several and we noticed that each app server is sending 2 or 3 emails per second..

Re: CRITICAL: Extreme bad performance using SDesk SP19

What database are you using?
Juan Paulini
Super Collector

Re: CRITICAL: Extreme bad performance using SDesk SP19

More information:
Application version: HP Open View Service Desk v 4.5.0588.1905 (SP19)

Application SO: Windows 2003 SP1

Database: ORACLE 9.2040 on Linux


Caroline Kraaij
Super Collector

Re: CRITICAL: Extreme bad performance using SDesk SP19

We experienced the same kind of problems. They were caused because the thread AppEventQueueReader stopped, maybe because of resource problems. Before this thread stopped we could see the number of events adding up. Expanding the memory to 3 Gb per server (2 servers, however, only solved our problems temporarily.

Because our application servers were virtual servers (VMWAre), which is not supported by HP, we are running our application servers on 'real hardware' servers again.

So it was found out what caused our problems, but there is still no real solution: when the application server experiences some kind of resource problem, the thread AppEventQueueReader (essential for executing rules) stops, because this always seems to be the thread that is started last.

You can check if the same happens with your server by starting the application with startserver.bat (in bin). In the status window (tab logfile) press the button 'Log monitoring info'. Then scroll down to the end of the log. If you do this several times you can see if the number of events/actions increases.

We were told that we were the only company experiencing this kind of problems. For reference purposes, could you please tell me the name of your company? Ours is Corus.

We run SD4.5 SP14 by the way.

Good luck!
Tim Schmitt_4
Frequent Visitor

Re: CRITICAL: Extreme bad performance using SDesk SP19

Thanks for the information Caroline.

We've looked at the status of the threads used by Service Desk and we found that the event logger would either say "runnable" or "waiting on monitor for entry". It seems like only a one or two of the event queue readers were able to run at the same time, even if the event queue reader threads were on different servers and we initially thought that there was an issue where only one or two event queue readers could be working at once. It seems that our observation of the event queue behavior might not be the cause of the problem but rather the result.

We also run virtual servers (VMware as well) and we have 2 Service Desk server processes running on each of the virtual servers. Each of the services has been given 1.5 MB of memory (which is the max allowable under java 1.3.1). Adding the memory does allow each server to manage more scheduled tasks but we did not notice an improvement in the event queue reading. It's really odd - we can see items in the queue but there's no reason they should be there, as there is plenty of CPU available, database is responding quickly, and the server process is idling? I don't understand what it is waiting for.

On the news side, HP created a hotfix for us where the Service Desk servers would spawn three event queue reader processes instead of one. Weâ ve not tried to implement this hotfix yet but plan to do so on Thursday of this week.
Juan Paulini
Super Collector

Re: CRITICAL: Extreme bad performance using SDesk SP19

Hi Caroline,
Thanks for you comments, I 'll put an eye on AppEventQueueReader and I'll tell you later if we experience the same problem.

We didn´t have virtual servers until las week, we've four app servers with 2GB, and recently we added an extra virtual server application. (it *has* to be enought for our peaks of 400 concurrent users)

Our company name is Repsol YPF

Caroline, Tim, really thank you.
Juan Paulini
Super Collector

Re: CRITICAL: Extreme bad performance using SDesk SP19

Another thing to note was that we experienced some problems with *any* flow that involved time. As an example, we set up a rule that changed the service call status from "Solved" to "Closed" if the time in status "Solved" is more than 48 hours.

This rule (seems to be *so* simple) never worked, because it started to enqueue freezing *all* the activity. It seemed that the system didn't worked, and when we looked at enqueued tasks, there were tons of tasks generated by that rule.

It happened the same with any rule that involved activation by time, also involving change, configuration, service calls, or whatever...

Also when we started al Service Pack 15, and kept the same with SP 19.
Juan Paulini
Super Collector

Re: CRITICAL: Extreme bad performance using SDesk SP19

Hi all,
We discovered (with the unvaluable help of a HP Support specialist) that we'd 20.500 views of Service Call Analized Data.

There was a HP known issue about the topic: http://openview.hp.com/ecare/getsupportdoc?docid=ITSM008658

It skyrocketted our client cache size to 35 MB of views data (views.rep file), and made almost impossible to start up the application from most of our WAN sites.

We keep investigating another problems that we've found, as some crazy rules that keep executing over and over because of some special change conditions...

Tim Schmitt_4
Frequent Visitor

Re: CRITICAL: Extreme bad performance using SDesk SP19

We've had similar problems with rules that use time, escpecially when using scheduled tasks in a rule. Each scheduled tasks seems to load into the applciation server memory consuming resources for the entire time it is waiting to be executed.

Reducing the number of rules that use scheduled tasks has made the most visible performance improvement although we were hoping that something could be done programmatically to solve the problem.

Thanks for the information regarding the views.rep file. I've never heard that before.
Juan Paulini
Super Collector

Re: CRITICAL: Extreme bad performance using SDesk SP19

Now we've found the problematic rule. Its one that, if a Change has "Aproved/Scheduled" state, keeps updating a task that triggers some WO at Start time and date.

We checked "Execute once" checkbox and every update only change task start time and it does not trigger another new task.

One thing that ammused me is that, after updating the rule, depending on how you correlate WO predecessors, the WO activation correlation does work or not. If I add a new WO, and set a old WO as predecessor, it does not follow the predecesor activation. If I add two or more WO, taking the newest one as the preceding WO of the others, it does work.

Black magic...

Thanks all for replying.. I' keep on investigating..
Mike Khashan CA
Super Collector

Re: CRITICAL: Extreme bad performance using SDesk SP19

Juan, you said you have 4 Wintel Boxes. I suggest you the following:
- On each Box install multiple SD applications Two for SD client. One for HTTP connections, the fourth for e-mail. Do the same on each box. At the end you will get 8 logical SD servers dedicated for cleint connections. 4 servers dedicated for HTTP connections and 4 servers for e-mail notifications. For sure you have to join then all.
Also ask you DBA to re-index your sd dB once a month or week, depend on the load or on the environment. For your info saving or creating SC should not take more than milli seconds and this apply for a db with more than 15 million records. Also there are more factors you might to think about, try the above and hopefully it should work.

Mike
Juan Paulini
Super Collector

Re: CRITICAL: Extreme bad performance using SDesk SP19

Hi Mike, thanks for your answer.

I alredy have other boxes for web and email forwarding, but the issue were in other place.

DB index are just fine, now I bored the DBAs asking them for that, but our DB does not seems to be a problem. We have really *huge* databases in the same HW with better response time.

Thanks anyway, I'll close the thread with a final conclusion.
Juan Paulini
Super Collector

Re: CRITICAL: Extreme bad performance using SDesk SP19

Hi all, and thanks all for your comments.

We solved the problem, as far as I know, and we have a reasonable response time with the aid of Global HP Support from Costa Rica (thanks Randall!!)

The problems were, in order of impact:
- A known bug in Analyzed Data View (that I suggest all you check it if you're in SP16 and before). It raised our cache size up to 36 MB, and impossible to work across WAN.

- A bad constructed rule in change management.

- Our full database backup didn't shutdown the application, making rules fail when DB were down.

- Java VM settings.

Now I really don't know if the scheduled rules will work fine, but if they don't, it will be another topic of discussion.

Again, thanks to all that have taken a second to look at this thread.

//Add this to "OnDomLoad" event