Service Desk Practitioners Forum
cancel
Showing results for 
Search instead for 
Did you mean: 

Application Server failures

SOLVED
Go to solution
Highlighted
Craig R Taylor
Collector

Application Server failures

Hello folks,

We are running Service desk 4.5 SP 11 and it periodically crashes for no apparent reason. It seemed we were having problems with memory so we gave it a fair bit of memory. (500 MB min, 1500 MB max). It appeared to work and neither of our app servers crashes for a week. This morning however one of them crashed without any apparent reason. The event said that the service terminated unexpectedly. The logs on the server that failed just stopped at the time of failure. Prior to the failure the following message appeared in the logs repeatedly (15 times or so in one bunch):

ServerGui Buffer full, cleared ServerGui logFile Monitor window

Other than that there's nothing I can see that would explain the failure.

Any ideas on how I should troubleshoot this?
You can observe a lot just by watching. - YOGI BERRA
14 REPLIES
Tim Schmitt_4
Frequent Visitor

Re: Application Server failures

DO you run the server application on windows or unix? Also, what version of java are you using for the applications server?
Craig R Taylor
Collector

Re: Application Server failures

We are running the server on Windows 2000 server (Win 2k3 maybe?) I believe. The JRE version is:

Java runtime version: 1.3.1_19-b03

To my knowledge this is the only supported JRE for SP11 that has the DST patch in it. These problems did seem to start occurring after the patch.
You can observe a lot just by watching. - YOGI BERRA
Tim Schmitt_4
Frequent Visitor

Re: Application Server failures

As a preventative measure, we restart the services (and rotate the log file) each night. Not really a solution, but it typically keeps the services running.

Do you have a lot scheduled tasks that the server executes? Also, can does the CPU load increase which this crash occurs (not sure if you can see the CPU history).

Craig R Taylor
Collector

Re: Application Server failures

Unfortunately, restarting services is a workaround rather than a fix and our management team is looking for a permanent solution. As we are a 24 hour operation Service Desk is a critical application for tracking issues and any downtime has far reaching implications.
You can observe a lot just by watching. - YOGI BERRA
Craig R Taylor
Collector

Re: Application Server failures

Any other ideas?
You can observe a lot just by watching. - YOGI BERRA
Tim Schmitt_4
Frequent Visitor

Re: Application Server failures

I know that you mentioned that restarting the services was not an optimal goal because you might introduce some down time but if you have two servers, you should be able to stop one while the other is remains running. If a service Desk person is connected to the server that is down, they should automatically connect to the server that is up and no interruption should occur.

We've notice similar problems when there were more than 5000 scheduled tasks being handled per server. For us, the system start to queue scheduled tasks events and eventually it would just stop running, although I don't recall seeing a number of sergver gui buffer full items. If this is a problem, you could install more Service Desk services on your existing servers, which seems to help with the memory usage since each java program can only use a certain amount of memory.

Does the windows event log record anything of interest when the service stops?

Also, if you suspect that the new java program is causing the problem, you might want to go back to the previous version of java and use Sun's tzupdater program to upddate the old version of java with the newer time zone data. TZ Updater will only work with specific java versions and I'm not sure which version you used to use.

J
-=R=-
Frequent Visitor

Re: Application Server failures

Craig,

This may be completely useless to you since we're on v5.1 but we have our services crash twice daily. HP is telling us that our servers are configured as two primaries rather than a primary and a secondary. Don't ask me how that would slip past 3 implementation consultants, but whatever.

Its worth looking into if you have multiple application servers.

By the way... what kind of assistance are you getting from support on this?

Craig R Taylor
Collector

Re: Application Server failures

Checking to see how the servers are load balanced is a really good idea and I'll have to check it out.

We have a consultant helping us on this. We thought we would check out things before we went to HP for help. Not a lot of sense going to them and saying "Its broke. Fix it" without trying anything or at least doing some troubleshooting. The thing is....where to even begin? (Beyond checking the logs, the event viewer, the amount of memory allocated to the JVM.).

Thanks for the idea in any event.
You can observe a lot just by watching. - YOGI BERRA
Craig R Taylor
Collector

Re: Application Server failures

So load balancing looks okay. (How should I be checking this?). In terms of scheduled tasks they are minimal (currently about 9 or 10) and they are clearing properly.

Any other ideas?
You can observe a lot just by watching. - YOGI BERRA
Craig R Taylor
Collector

Re: Application Server failures

One of the main differences between our dev/test environment and the prod environment is the version of Java that is being used. We are running 1.3.1_19 in prod. There is apparently a known issue with memory leaks in 1.3.1_19 that are fixed in 1.3.1_20.


http://java.sun.com/j2se/1.3/ReleaseNotes.html

I am going to see if going to 1.3.1_20 corrects the problem and I'll post my findings
You can observe a lot just by watching. - YOGI BERRA
George M. Meneg
Honored Contributor

Re: Application Server failures

While it is not a solution, the message "ServerGui Buffer full, cleared ServerGui logFile Monitor window" has nothing to do with the crash.

If you uncheck "Show server status" on the tab General of server configuration utility, you will stop seeing this message. This message is to indicate that the "Server Gui" buffer is full (it hold IIRC about 200 lines)
menes fhtagn
JaS_4
Honored Contributor
Solution

Re: Application Server failures

I agree with Tim. You should consider having 2 servers and rotating the logs as there is a known problem due to log size.
Here is the tech note on the issue of large logserver.txt. The url is http://openview.hp.com/ecare/getsupportdoc?docid=OV-EN008040

Title: Can the size of the log files affect the performance of Service Desk?
Document ID: OV-EN008040
Product: service desk
Version: 4.0
OS: Windows 2000
Updated: 2002-Sep-13

Solution

Yes, if the size of LOGSERVER.TXT exceeds 4 MB it will adversely affect the performance of the Application Server.

I am assuming your logfiles is quite huge at this stage since it's a 24 hour site and you do not rotate or delete your logserver.txt.

It's a recipe for disaster if a program has to maintain a huge log in memory and try to write to it. It might be a case where the logs is so big that SD can no longer keep it open and crashes and not write anything to the log OR it could not use the Server status interface to view the logserver any more.

Craig R Taylor
Collector

Re: Application Server failures

Thats a good point too. Just to let you all know the problem appears to have been the JRE we were running. The memory issue in 1.3.1_19 was causing it to crash. We have been running 1.3.1_20 successfully. In terms of log rotation I plan to implement that shortly. (I'm not sure why no one set it up previously)


Thanks again folks
You can observe a lot just by watching. - YOGI BERRA
Craig R Taylor
Collector

Re: Application Server failures

Well...that didn't fix the issue. We have had numerous failures on the application since the change. I'm really getting stumped on this. We are going to submit a support request with HP but I'd still like to keep working on this in the meantime
You can observe a lot just by watching. - YOGI BERRA
//Add this to "OnDomLoad" event