Service Desk Practitioners Forum
cancel

App server with abnormal amount of scheduled tasks

SOLVED
Go to solution
Highlighted
Justin Trant
Super Contributor.

App server with abnormal amount of scheduled tasks

Hi,

One of our application servers is showing an abnormal amount of scheduled tasks (4000 instead of the usual 2000). I have looked through the logs and the server appears to be scheduling rules for tasks that occured months ago (e.g. processing a rule to e-mail the caller when a call is set to 'Closed', when the call was Closed in February!)

Has anybody experienced this issue before? We are running SD4.5 SP15, and this has not happened before.

Should I contact HP?

Thanks,
Justin.
13 REPLIES
Mark O'Loughlin
Acclaimed Contributor.
Solution

Re: App server with abnormal amount of scheduled tasks

Hi,

sure contact hp if you have a support contract just to get that ball rolling.

Have you run any updates recently thatmayhave updated the closed calls?

Maybe stop and restart the application services and see if it helps.

Are there any errors in the log files that show say performance or DB issues happening at the moment?
Justin Trant
Super Contributor.

Re: App server with abnormal amount of scheduled tasks

Mark,

Thanks for your reply.

I have tried restarting the app servers, which has not resolved the issue. We have not implemented any updates or changes whatsoever.

The only unusual things I can see in the log files are:

ITPcom.hp.ifc.io.AppStreamException: Flush failed due to: Connection reset by peer: socket write error
at com.hp.ifc.io.AppDataStream.flushError(Unknown Source)
at com.hp.ifc.io.AppDataStream.flushBuffer(Unknown Source)
at com.hp.ifc.io.AppDataStream.flush(Unknown Source)
at com.hp.ifc.net.itp.AppItpRequestHandler.sendResult(Unknown Source)
at com.hp.ifc.net.itp.AppItpRequestHandler.process(Unknown Source)
at com.hp.ifc.net.tcp.AppTcpConnection.processRequest(Unknown Source)
at com.hp.ifc.net.tcp.AppTcpThread.run(Unknown Source)

and a couple of "Connection to repository failed messages":

Wed, 22/08/2007 08:44:40 no of scheduled tasks loaded for this server: 4230
Wed, 22/08/2007 08:44:54 no of scheduled tasks loaded for this server: 4230
Wed, 22/08/2007 08:45:01 Connection to service desk data
Wed, 22/08/2007 08:45:01 Connection to service desk data
Wed, 22/08/2007 08:45:01 Connection to service desk data
Wed, 22/08/2007 08:45:01 Connection to service desk data
Wed, 22/08/2007 08:45:02 Connection to service desk data
Wed, 22/08/2007 08:45:02 Connection to service desk data
Wed, 22/08/2007 08:45:02 Connection to service desk data
Wed, 22/08/2007 08:45:02 Connection to service desk data
Wed, 22/08/2007 08:45:03 Connection to service desk data
Wed, 22/08/2007 08:45:03 Connection to service desk data
Wed, 22/08/2007 08:45:03 Connection to service desk data
Wed, 22/08/2007 08:45:03 Connection to service desk data
Wed, 22/08/2007 08:45:04 Connection to service desk data
Wed, 22/08/2007 08:45:04 Connection to service desk data
Wed, 22/08/2007 08:45:04 Connection to service desk data
Wed, 22/08/2007 08:45:04 Connection to service desk data
Wed, 22/08/2007 08:45:05 Connection to service desk data
Wed, 22/08/2007 08:45:05 Connection to service desk data failed
Wed, 22/08/2007 08:45:05 Connection to service desk data
Wed, 22/08/2007 08:45:05 Connection to service desk data failed
Wed, 22/08/2007 08:45:05 Connection to service desk data
Wed, 22/08/2007 08:45:05 Connection to service desk data failed
Wed, 22/08/2007 08:45:05 Connection to service desk data
Wed, 22/08/2007 08:45:05 Connection to service desk data failed
Wed, 22/08/2007 08:45:05 no of scheduled tasks loaded for this server: 4231
Wed, 22/08/2007 08:45:14 no of scheduled tasks loaded for this server: 4231
Wed, 22/08/2007 08:45:23 no of scheduled tasks loaded for this server: 4217

CPU usage on the application server is fine.

Thanks again,
Justin.
Justin Trant
Super Contributor.

Re: App server with abnormal amount of scheduled tasks

Mark,

From what I can see, no changes have been made to the closed calls since they were set to 'Closed'.
Mark O'Loughlin
Acclaimed Contributor.

Re: App server with abnormal amount of scheduled tasks

It would be interesting to see what hp say.

Any idea what is causing this " Connection to service desk data failed"?

Justin Trant
Super Contributor.

Re: App server with abnormal amount of scheduled tasks

Hi,

I can only assume it was a momentary database connectivity issue. The issue started approx 10 days ago, and the scheduled tasks have been steadily climbing since then. They have peaked at 4200 and the number is going down (slowly), so it could resolve itself over time.

Would it be possible that a user chose 'update all' and made changes to a large number of Service Calls, thus scheduling a large number of tasks?

Thanks,
Justin.
Mark O'Loughlin
Acclaimed Contributor.

Re: App server with abnormal amount of scheduled tasks

Hi,

that is indeed possible. Do users have access to update all?

Are the DB rules that generate schedule tasks set to run once or just on condition changes.
Justin Trant
Super Contributor.

Re: App server with abnormal amount of scheduled tasks

Hi,

Yes, users do have access to Update All.

We have only one rule that generates scheduled tasks which is not set to "Only execute once". The rule is designed as follows:

When a Service Call is set to 'Resolved' status and assigned to a certain workgroup a rule is scheduled to set the call to 'Closed' after 7 days.

When service call is modified
where Status equals Resolved
AND Assignment:To workgroup (*) equals ABC Workgroup, Schedule the validation of this condition at 7 days after [Resolved start time]
and NOT (Solution is empty)

Could this be the cause? Would it be safe to select 'Only execute once' for this rule? To be honest, I didn't think of it when designing the rule.

Thanks,
Justin.
Justin Trant
Super Contributor.

Re: App server with abnormal amount of scheduled tasks

Mark,

I'm going to change the rule to 'Only execute once'.

Is it safe to use 'Only execute once' with 'Evaluate this rule when this field has been changed'?

Thanks,
Justin.
Mark O'Loughlin
Acclaimed Contributor.

Re: App server with abnormal amount of scheduled tasks

Hi Justin,

I think it will be ok. Set it and then test the rule to be sure it behaves as expected.

You could create an identical test rule with the scheduled task to be done 5 minutes later to speed up the testing.

Do you have a test box to try this out on?
Justin Trant
Super Contributor.

Re: App server with abnormal amount of scheduled tasks

Hi Mark,

I tested this in the test environment and it seemed to work fine. We will know in 7 days :)

Thanks for your help. This is the only rule that had "Evaluate this rule when this field has been changed" ticked and "Only execute once" unticked, so I hope this is the culprit.

There can be up to 800 calls with this rule scheduled so this would account for quite a few scheduled tasks, especially if they are being duplicated for whatever reason.

Thanks again,
Justin.
Mark O'Loughlin
Acclaimed Contributor.

Re: App server with abnormal amount of scheduled tasks

Thats good news. I had a similar set up before where one scheduled task rule created the majority of the scheduled tasks. Its function was to set the SC to "Closed" after a number of days.

To prevent the scheduled tasks being created in the first place I changed the process so calls would be cloed in real time, the email to the end users would go out in real time and the users and desk were advised that the user had a grace period to call back in for which their calls could be reopened. Anything after the grace period and they got a new call.

After a small process change and some user education no more scheduled tasks in the 1,000's.

Not sure if this applies to you but fyi.
Justin Trant
Super Contributor.

Re: App server with abnormal amount of scheduled tasks

Thanks Mark. I'm still at a loss as to how one app server could be handling so many of these scheduled tasks over the other three.

Unfortunately this is a business requirement so I have to accomodate it, but thanks for your input regarding your experiences with this rule.

Hopefully this rule change will reduce the number of scheduled tasks for this server.

Justin.
Justin Trant
Super Contributor.

Re: App server with abnormal amount of scheduled tasks

As above.