Saturday, January 26, 2013

Script - Thread Dump of Multiple Servers located across distributed Machines

In this post, we will see how to take thread dump of multiple application  servers which is located across distributed machines at one shot.

Scenario:
Consider that you have production code deployed(running) on 9 servers which is distributed  3 servers across to 3 Unix Server Machines.To get a better view about the scenario see the following diagram.

image

Now we want to take multiple thread dumps of all 9 servers frequently at specified interval.In order to achieve this we are going write a simple bash script, to know about this more keep on reading.

Script Functionalities:

This script have 6 important yet very simple functions for better understanding.

1] Set Environment Variables(set_environment_variables)
2] Create List of Servers (create_servers_list)
3] Get Thread Dump using jstack in Local Machine(get_thread_dump_box1)
4] Get Thread Dump using jstack in Remote Machine(get_thread_dump_box2)
5] Get Thread Dump using jstack in Remote Machine(get_thread_dump_box3)

Note: jstack is java binary executable, to know various other binary executable visit the following link. http://www.industryvertical.co.in/2013/01/java-binary-utilities.html
 
6] Clean unwanted files(clear_event)

Scripting Part:

#!/bin/bash
#Purpose: Script to take thread dump of J2EE application server logs
#Program Name: getThreadDump.sh

clear
set_environment_variables()
{
SERVER_HOME=/optional/Oracle/Middleware/user_projects/domains/base_domain/servers
JAVA_HOME=/optional/Oracle/Middleware/jdk1.6.0_26/
}

create_servers_list()
{
        ls  ${SERVER_HOME}|grep -v Admin|grep -v logtail|grep -v domain_bak > list_of_servers.txt
}

get_thread_dump_box1()
{
        DATE=`date +"%d_%m_%Y_%H%M%S"`
        NO_OF_SERVERS=$(cat /optional/Monitoring/list_of_servers.txt|wc -l)
        i=1
        while [ $i -le $NO_OF_SERVERS ]
         do
         SERVER_NAME=$(cat list_of_servers.txt|tr '\n' ':'|cut -d":" -f$i)
         echo "Getting Process ID of ${SERVER_NAME}"
         ps -ef | grep -i Dweblogic.Name=${SERVER_NAME} | grep -v 00:00:00 | awk '{print $2}' > process_id.txt
         PID=$(cat process_id.txt|tr '\n' ':'|cut -d":" -f1)
         echo "Taking ThreadDump for ${SERVER_NAME} with Process ID ${PID}"
         ${JAVA_HOME}/bin/jstack $PID > /optional/common/thread_dump/${SERVER_NAME}_${DATE}.out

         i=`expr $i + 1`
        done
}

get_thread_dump_box2()
{
        echo "Checking in 192.168.2.2 Box!!"
        ssh 192.168.2.2 /bin/bash getThreadDump.sh
}

get_thread_dump_box3()
{
        echo "Checking in 192.168.2.3 Box!!"
        ssh 192.168.2.3 /bin/bash getThreadDump.sh
}

clear_event()
{
        rm list_of_servers.txt process_id.txt
}

set_environment_variables
create_servers_list
get_thread_dump_box1
get_thread_dump_box2
get_thread_dump_box3
clear_event


Connection across distributed unix servers machines without password is achieved using ssh-rsa and ssh-copy-id command utility. Thanks to Rivest,Shamir, Adleman for this beautiful algorithm. If you have any queries, please leave you reply.

No comments:

Post a Comment