The community will be in read-only from Tuesday 11:59pm (PST) to Wednesday 7:30am (PST)
The community will be in read-only from Tuesday 11:59pm (PST) to Wednesday 7:30am (PST)
Project and Portfolio Management Practitioners Forum
cancel
Showing results for 
Search instead for 
Did you mean: 

PPM Cluster -> What order to restart servers/nodes ? Automated PPM startup scripts?

Highlighted
EricLEdberg
Super Collector

PPM Cluster -> What order to restart servers/nodes ? Automated PPM startup scripts?

We are restarting our HP PPM system weekly due to out-of-memory issues and have a simple script that restarts each node (sleeping 120 seconds between each start) after that last server reboots.

 

I was originally told by a HP PPM contractor to restart each node in reverse order in our multi-server, multi-node configuration from server3-node 5 (services node) to server1-node1 (primary node).

 

  • What is the correct node order to restart PPM nodes in a multi-server, multi-node configuration ?

Question:   Does anybody have a script that can restart a multi-server multi-node configuration?  Our simple script today uses the "sc" command to retart nodes waiting between each node start.   I'm sure there are improvements to validate if a node has started successfully that could be scripted too.  (a stop script would be great too)

 

Thanks,

 

Eric Edberg

 

 

12 REPLIES
Erik Cole
Honored Contributor

Re: PPM Cluster -> What order to restart servers/nodes ? Automated PPM startup scripts?

Hi,

 

I don't think the sequence matters much from a technical standpoint. I've done it in differing orders in the past for other reasons, such as not wanting the "end-user" node up for business until I confirmed the services node is running without issue, etc. Guess it kind of depends on your setup too since you're multi-node, multi-server.

 

I don't have any fancy scripts, but there shouldn't be any reason your script couldn't just use sc.exe with a servername parameter for your "remote" machines to start the services there too.

EricLEdberg
Super Collector

Re: PPM Cluster -> What order to restart servers/nodes ? Automated PPM startup scripts?

In the install guide there are comments stating that the first node to come up is the primary node and also that you must make sure to wait and verify that each node is active before starting the next node (tail log file)

I posted this question because the HP PPM contractor was explicit when stating to start the nodes in reverse order. In the past we have experienced an un-stable system when servers rebooted due to power outages and PPM was configured to restart it's service automatically (2 nodes per server). It's not now.

May be I'll submit an HP Support Case if an HP tech does not confirm here...

May be I'll submit a support case if an HP tech does not respond here...
Sascha Mohr
Esteemed Contributor

Re: PPM Cluster -> What order to restart servers/nodes ? Automated PPM startup scripts?

Hi,

the first services node that's up and running usually becomes the primary services node. For the user nodes the order is not so important as the load balancer distributes the user traffic. The documentation tells us to start one node at a time fro whatever reason; it seems beneficial for the internal cluster communication somehow. They improved that with 9.1.3 though, if I remember correctly, so the order would no longer matter then.

Regards

Sascha

RARORA
Acclaimed Contributor

Re: PPM Cluster -> What order to restart servers/nodes ? Automated PPM startup scripts?

I am also looking for scripts automate restart PPM  (stop , confirm and start, confirm) service on Solaris.  

 

Rajesh

AlexSavencu
Honored Contributor

Re: PPM Cluster -> What order to restart servers/nodes ? Automated PPM startup scripts?

Hi,

Having scripts for starting is pretty straight forward. My problem is with a script which stops the server and to make sure that the server is stopped. I shall be working in the following period on such a script.

Cheers
Alex

--remember to kudos people who helped solve your problem
Jim Esler
Honored Contributor

Re: PPM Cluster -> What order to restart servers/nodes ? Automated PPM startup scripts?

We used the script segment below to stop a node on 7.5. It should work on 9.1 but we have not tried it yet. Essentially, it executes the stop script, then waits a few minutes to see if it is still running. If it is, it waits a bit longer. If it is still running, it uses a kill -9 command to stop it. We are running on AIX but this should work on any UNIX variant. You would, of course, have to make some adjustments to deal with your specific configuration and needs.

 

./kStop.sh -name ${hostName} -delay 5 >> ${LOGFILE} 2>&1

PSTATUS=$?

if [ $PSTATUS -ne 0 ]; then

echo "Cannot stop the server: exit code "${PSTATUS} >> ${LOGFILE} 2>&1

exit 30

fi

sleep 360

echo `date` ":---------------------Sleep complete------------------------\n" >> ${LOGFILE}

./kStatus.sh >> ${LOGFILE} 2>&1

ps -ef | grep ITG | grep -v grep >> ${LOGFILE} 2>&1

export PN=`ps -ef | grep java | grep ITG | grep -v grep | awk '{print $2}'`

# If ITG is still active, wait another minute

if [ 0$PN -gt 0 ];then

echo `date` ":---------------------Still active, wait another minute------------------------\n" >> ${LOGFILE}

sleep 60

ps -ef | grep ITG | grep -v grep >> ${LOGFILE} 2>&1

export PN=`ps -ef | grep java | grep ITG | grep -v grep | awk '{print $2}'`

# Kill the ITG process if it is still running

if [ 0$PN -gt 0 ];then

echo `date` ":---------------------KILLING PROCESS $PN------------------------\n" >> ${LOGFILE}

kill -9 $PN

ps -ef | grep ITG | grep -v grep >> ${LOGFILE} 2>&1

fi

fi

AlexSavencu
Honored Contributor

Re: PPM Cluster -> What order to restart servers/nodes ? Automated PPM startup scripts?

Hey, Jim,

 

I had the same idea for the script, but there is only one issue: I can't find a proper method to identify the PPM pid on a Windows machine (your script is only valid for *NIX systems). I have tried various tools available both in cygwin and sysinternals, but no luck. Also, i could not locate any pid file. Can you please advise on this?

 

Thanks in advance,

Alex


--remember to kudos people who helped solve your problem
Jim Esler
Honored Contributor

Re: PPM Cluster -> What order to restart servers/nodes ? Automated PPM startup scripts?

The following line in the script is getting the pid value that will be used in the kill command:

 

export PN=`ps -ef | grep java | grep ITG | grep -v grep | awk '{print $2}'`

 

This command is specifically for use with the Bourne shell on AIX. Adjustments may be needed for any other environment. I have no experience with the corresponding processes in Windows other than under Cygwin. Since Cygwin does not know about processes started by Windows, it would see the PPM instances only if they were started with the kStart.sh script in a Cygwin session.

 

The pids are not listed in a file. They are available dynamically with the ps command. On AIX, the -ef options on the ps command give a full listing of all processes except root processes. The grep commands filter out just the process lines that have the strings 'java' (the command that initiates a node) and 'ITG' (a string in our file paths on the command line) in them. Any line with the string 'grep' is excluded (i.e., the line executing this command). The awk command extracts the pid from the ps output line. The export command stores the extracted pid value. Note that this command line only works as intended if there is only one node running on the server. It could be easily extended to deal with a specific node when there are more nodes on the server.

AlexSavencu
Honored Contributor

Re: PPM Cluster -> What order to restart servers/nodes ? Automated PPM startup scripts?

Hi, Jim,

I am quite experienced in bash and ksh scripting, so I could figure out the logic on the script. The reference is good for the forum, so one kudo to you.

I shall keep digging for a way to identify the process in Windows.

Cheers
Alex

--remember to kudos people who helped solve your problem
RARORA
Acclaimed Contributor

Re: PPM Cluster -> What order to restart servers/nodes ? Automated PPM startup scripts?

My challenge is still to find out pid , I am running PPM on solaris 10. I am hoping that there is a file somewhere which should have pid recorded when PPM service is started.

 

sh kStatus.sh gives

$ sh kStatus.sh
JAVA_HOME = /usr/java
java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) Server VM (build 20.1-b02, mixed mode)
Checking rmi://edhuap006:1099/KintanaServer
--> running (load: 30.0, mode: NORMAL)

 

and ps -ef gives

$ ps -ef|grep java
ppmadmin  3390   636   0   Mar 16 ?         577:18 /usr/java/bin/java -server -server -DNAME=ppm -Djava.endorsed.dirs=/ppm/lib/end
noaccess  1385   636   0   Oct 05 ?         230:00 /usr/java/bin/java -server -Xmx128m -XX:+UseParallelGC -XX:ParallelGCThreads=4
ppmadmin 19374 11493   0 15:26:14 pts/1       0:00 grep java
$

 

 

Jim Esler
Honored Contributor

Re: PPM Cluster -> What order to restart servers/nodes ? Automated PPM startup scripts?

The pid for the instance in your sample output is 3390. The command line I listed in an earlier post would extract this value from the ps command output.

AlexSavencu
Honored Contributor

Re: PPM Cluster -> What order to restart servers/nodes ? Automated PPM startup scripts?

Rarora,

You can grep for ppm in your case.

One other note: it is a good practice to use ps -fu $USER to avoid any processes from other users or environments.

Cheers
Alex

--remember to kudos people who helped solve your problem
//Add this to "OnDomLoad" event