Gradle 이용한 간단한 원격 배포 셈플링


그룹별 병렬? 배포 - WAS Instance 가 늘더라도 배포 시간 절약

  1. WAS 유입 차단 -> 5초대기 -> WAS 중지
  2. 소스 배포 -> 소스 압축 풀기 및 설정
  3. WAS 기동
  4. 필수 서비스 체크(CURL)
  5. WAS 유입 활성화


빌드 파일과 별도 분리 해도 되고...(분리된 파일을 Jenkins에서 실행)


clean build remoteDeploy


.. .. apply plugin: 'org.hidetake.ssh' .. .. buildscript { repositories { jcenter() } dependencies { classpath 'org.hidetake:gradle-ssh-plugin:2.4.2' } } task remoteDeploy << { def startTime = System.currentTimeMillis() println "project: [$project]" def PROJECT_NM = project.name def DEPLOY_SERVERS = [] if( "$targetServer" == "dev" ){ DEPLOY_SERVERS = [ group1: [ ], ] } else if( "$targetServer" == "stage" ){ DEPLOY_SERVERS = [ group1: [ ], ] } else if( "$targetServer" == "real" ){ DEPLOY_SERVERS = [ group1: [ [host: "dkwas01", user: "tomcat", wasHome: "/usr/local/tomcat/domains/test11", wasDomain: "test11", webRoot: "/data/webroot/app-test", isWait: true,], [host: "dkwas02", user: "tomcat", wasHome: "/usr/local/tomcat/domains/test21", wasDomain: "test21", webRoot: "/data/webroot/app-test", isWait: true,], ], group2: [ [host: "dkwas03", user: "tomcat", wasHome: "/usr/local/tomcat/domains/test31", wasDomain: "test31", webRoot: "/data/webroot/app-test", isWait: true,], [host: "dkwas04", user: "tomcat", wasHome: "/usr/local/tomcat/domains/test41", wasDomain: "test41", webRoot: "/data/webroot/app-test", isWait: true,], ], ] } DEPLOY_SERVERS.eachWithIndex { gItem, gIdx ---> logInfo(DEPLOY_SERVERS, gItem, gIdx, null, null, "START") gItem.value.eachWithIndex() { hItem, hIdx -> def remoteUser = hItem.user def remoteHost = hItem.host def isWait = hItem.isWait def catalinaBase = hItem.wasHome def wasDomain = hItem.wasDomain def webRoot = hItem.webRoot def warPath = webRoot + "/../" + PROJECT_NM def remoteServer = [host: remoteHost, user: remoteUser, identity: file("${System.properties['user.home']}/.ssh/id_rsa")] ssh.run { session(remoteServer) { logInfo(DEPLOY_SERVERS, gItem, gIdx, hItem, hIdx, "HTTPD(JK_MOD) BLOCK -> WAS SHUTDOWN") // 고객유입 차단 -> WAS 중지 execute "cd $catalinaBase; ./shutdown.sh " + (isWait?'wait update 5':'') logInfo(DEPLOY_SERVERS, gItem, gIdx, hItem, hIdx, "WAR REMOTE COPY") put from: war.archivePath.path, into: warPath execute "rm -Rf " + webRoot + "/*" execute "cd " + webRoot + "; jar xf " + warPath execute "chmod -Rf 755 " + webRoot + "/*" logInfo(DEPLOY_SERVERS, gItem, gIdx, hItem, hIdx, "WAR START (HTTPD(JK_MOD) BLOCKING)") // WAS 기동 (고객 유입은 차단상태) execute "cd $catalinaBase; ./start.sh" } } } gItem.value.eachWithIndex() { hItem, hIdx -> def remoteUser = hItem.user def remoteHost = hItem.host def catalinaBase = hItem.wasHome def isWait = hItem.isWait def wasDomain = hItem.wasDomain def remoteServer = [host: remoteHost, user: remoteUser, identity: file("${System.properties['user.home']}/.ssh/id_rsa")] ssh.run { session(remoteServer) { logInfo(DEPLOY_SERVERS, gItem, gIdx, hItem, hIdx, "CHECK SERVICE") while (true) { sleep(2000) def resultStr = execute "cd $catalinaBase; ./checkService.sh" if (resultStr.contains(wasDomain) && resultStr.contains("SERVICE CHECK ERROR") == false) { break } print "." } logInfo(DEPLOY_SERVERS, gItem, gIdx, hItem, hIdx, "HTTPD(JK_MOD) FLOW") // 고객 유입 execute "cd $catalinaBase; ./httpd.sh start" } } } logInfo(DEPLOY_SERVERS, gItem, gIdx, null, null, "END") } def totalExecTime = (System.currentTimeMillis() - startTime) / 1000 println "==================== TOTAL EXEC TIME: " + (totalExecTime) + "s ====================" } static logInfo(deployServers, gItem, gIdx, hItem, hIdx, msg) { println("==================== [GROUP $gItem.key " + (gIdx + 1) + "/" + deployServers.size() + (hItem?" " + (hIdx + 1) + "/" + gItem.value.size() + " $hItem.host:$hItem.wasDomain] ":" ") + msg + " ====================" ) }


일반적으로 Jenkins나 SSH(SCP) 등을 이용해서 원격지 배포 시 순차 배포 하게 된다.

하지만 원격지 서버의 수가 많으면 (1대 배포 시간 * 배포 서버 수)가 되어 배포 시간이 증가하게 된다.

급하게 배포하거나, 롤백 하는 상황에서 배포 시간이 증가하게 되면 빠른 복구가 힘들어 진다.

그에 따라서 약 2그룹으로 나누고, 각 그굽을 병렬로 처리 할 수 있도록 해보자.


하지만 이 방법은 2그룹 중 1그룹으로도 고객 서비스가 원할 해야 한다는 보장이 있어야 하며, 결국 오버스펙으로 운영 해야 한다는 말..


(war 배포아닌 파일 묶음 배포 방식)



Tomcat 7.x + Jenkins ver. 1.500


[Maven tomcat plugin을 이용하여 deploy -> reload 방법]

tomcat/conf/tomcat-users.xml







pom.xml


	org.codehaus.mojo
	tomcat-maven-plugin
	1.1
	
		UTF-8
		war
		http://localhost/manager/text
		
		admin
		admin
	



젠킨스 프로젝트 설정 Build 부분 명령어에 tomcat:redeploy tomcat:reload


[Shell 실행하여 Tomcat 재시작 방법]


jenkins에서 빌드 스크립트 종료 후 아래 shell 실행하면 정상 start 후 Process를 종료해버려서 기껏 올라간 Tomcat이 종료되는 문제가 발생한다.

그럴 경우 해당 프로젝트 설정에서 "This build is parameterized" 체크 하고, "Add Parameter" 에서 BUILD_ID 에다가 아무 값이나 셋팅!!

#!/bin/bash

echo "Shutdown tomcat now...."
bash /usr/local/tomcat/bin/shutdown.sh
echo "tomcat is now terminated..."
sleep 10
if [ -z "`ps -eaf | grep java | grep /usr/local/tomcat/bin`" ]; then
        echo "Tomcat was terminated successfully."
else
        ps -eaf | grep java | grep /usr/local/tomcat/bin | awk '{print $2}' |
        while read PID
                do
                echo "Killing $PID... "
                kill -9 $PID
                echo "Tomcat is being shutdowned"
                done
fi

echo "tomcat restart now"
/usr/local/tomcat/bin/./startup.sh
AA=`ps -eaf | grep java | grep /usr/local/tomcat/bin`
echo "AA: $AA"
if [ -z "`ps -eaf | grep java | grep /usr/local/tomcat/bin`" ]; then
        echo "Error!!! Cannot start tomcat server."
else
        echo "Tomcat restart succcessfully. "
fi
echo "Job finished"

[참조 shell] http://ausgang.egloos.com/2947190

startup.sh

#!/bin/sh


. ./env.sh

export CATALINA_HOME=/usr/local/tomcat/apache-tomcat-8.0.9
export CATALINA_BASE=/usr/local/tomcat/domains/$PROJECT_NM
export JAVA_OPTS="-Djava.awt.headless=true -server -Xms512m -Xmx1024m -XX:NewSize=256m -XX:MaxNewSize=256m -XX:PermSize=256m -XX:MaxPermSize=512m -XX:+DisableExplicitGC"
export CATALINA_OPTS=$CATALINA_OPTS" -Dfile.encoding=UTF-8"

cd $CATALINA_HOME/bin

./startup.sh


AA=`ps -eaf | grep java | grep $WAS_PATH`
echo "AA: $AA"
if [ -z "`ps -eaf | grep java | grep $WAS_PATH`" ]; then
        echo "Error!!! Cannot start tomcat server."
fi

while [ `netstat -an | grep :$WAS_PORT | grep LISTEN | wc | awk '{print $1}'` != 1 ]; do
        echo -ne "."
        sleep 1
done

echo "Check WAS Context Ready http://localhost:$WAS_PORT ..."

until [ "`curl --silent --show-error --connect-timeout 1 -I http://localhost:$WAS_PORT | grep 'Coyote'`" != "" ];
do
        echo -ne "."
        sleep 1
done

echo "Start WAS And Context $PROJECT_NM Tomcat Succcess"

shutdown.sh

#!/bin/sh


. ./env.sh

export CATALINA_HOME=/usr/local/tomcat/apache-tomcat-8.0.9
export CATALINA_BASE=/usr/local/tomcat/domains/$PROJECT_NM

$CATALINA_HOME/bin/./shutdown.sh

I=0

until [ "`ps -eaf | grep java | grep $WAS_PATH`" == "" ];
do
        if [ $I == 5 ]; then
		ps -ef | grep java | grep "$WAS_PATH " | awk {'print "kill -9 " $2'} | sh -x
                break;
        fi

        let I=$I+1

        echo Stop WAS Finish? Sleeping for 2 seconds
        sleep 2
done

+ Recent posts