Apache Tomcat Load Balancer

LoadBalancer HowTo

http://tomcat.apache.org/connectors-doc/generic_howto/loadbalancers.html

workers.properties configuration

http://tomcat.apache.org/connectors-doc/reference/workers.html

Java EE Load Balancing tutorial

http://www.zulutown.com/blog/2009/02/16/java-ee-load-balancing-with-tomcat-and-apache/

Tomcat Clustering - A Step By Step Guide

http://www.mulesoft.com/tomcat-clustering

Implementation of tomcat clustering

http://www.easywayserver.com/implementation-tomcat-clustering.htm

5 Minute Guide to Clustering – Java Web Apps in Tomcat

http://www.richardnichols.net/2010/08/5-minute-guide-clustering-apache-tomcat/

mod_proxy

http://tomcat.apache.org/tomcat-5.5-doc/balancer-howto.html

Apache Tomcat 6.0 Proxy Support HOW-TO

http://tomcat.apache.org/tomcat-6.0-doc/proxy-howto.html

Tomcat과 Apache를 연동하여 클러스터링을 설정하려면 먼저 연동에 문제가 없어야 한다

Tomcat 6, Apache2.2의 연동은 여기를  참조

Installation of Apache web server for load balancing and load sharing

Download apache and install on same system for vertical tomcat clustering.

After installation of apache, we need to do some configuration

Installation of Tomcat 6

Make three copy of tomcat If want to make three tomcat clustering system then need three copies of tomcat
Cluster 1 tomcatA 
Cluster 2 tomcatB 
Cluster 3 tomcatC 

Horizontal Clustering 

Vertical Clustering 

If need three cluster then need to make three instance of tomcat


Installation is simple as we install tomcat normally. In this tomcat clustering we are going for vertical tomcat clustering. So all tomcat instance is in single server. If you want (real world clustering) that is also much simpler. Then work with ip instead of localhost, I am also including examples for download

Tomcat server.xml configuration

1. Vertical tomcat clustering( on single machine)
2. Horizontal tomcat clustering( on multiple machine)

Vertical tomcat clustering example

open Tomcat -> Conf -> server.xml

Edit highlighted and can give port which we can required

1 First tomcatA

<!-- Note:  A "Server" is not itself a "Container", so you may not
     define subcomponents such as "Valves" at this level.
     Documentation at /docs/config/server.html
 -->
<Server port="8105" shutdown="SHUTDOWN">

  <!--APR library loader. Documentation at /docs/apr.html -->
  <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />
  <!--Initialize Jasper prior to webapps are loaded. Documentation at /docs/jasper-howto.html -->
  <Listener className="org.apache.catalina.core.JasperListener" />
  <!-- JMX Support for the Tomcat server. Documentation at /docs/non-existent.html -->
  <Listener className="org.apache.catalina.mbeans.ServerLifecycleListener" />
  <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />

  <!-- Global JNDI resources
       Documentation at /docs/jndi-resources-howto.html
  -->
  <GlobalNamingResources>
    <!-- Editable user database that can also be used by
         UserDatabaseRealm to authenticate users
    -->
    <Resource name="UserDatabase" auth="Container"
              type="org.apache.catalina.UserDatabase"
              description="User database that can be updated and saved"
              factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
              pathname="conf/tomcat-users.xml" />
  </GlobalNamingResources>

  <!-- A "Service" is a collection of one or more "Connectors" that share
       a single "Container" Note:  A "Service" is not itself a "Container", 
       so you may not define subcomponents such as "Valves" at this level.
       Documentation at /docs/config/service.html
   -->
  <Service name="Catalina">
  
    <!--The connectors can use a shared executor, you can define one or more named thread pools-->
    <!--
    <Executor name="tomcatThreadPool" namePrefix="catalina-exec-" 
        maxThreads="150" minSpareThreads="4"/>
    -->
    
    
    <!-- A "Connector" represents an endpoint by which requests are received
         and responses are returned. Documentation at :
         Java HTTP Connector: /docs/config/http.html (blocking & non-blocking)
         Java AJP  Connector: /docs/config/ajp.html
         APR (HTTP/AJP) Connector: /docs/apr.html
         Define a non-SSL HTTP/1.1 Connector on port 8080
    -->
    <Connector port="8081" protocol="HTTP/1.1" 
               connectionTimeout="20000" 
               redirectPort="8443" />
    <!-- A "Connector" using the shared thread pool-->
    <!--
    <Connector executor="tomcatThreadPool"
               port="8080" protocol="HTTP/1.1" 
               connectionTimeout="20000" 
               redirectPort="8443" />
    -->           
    <!-- Define a SSL HTTP/1.1 Connector on port 8443
         This connector uses the JSSE configuration, when using APR, the 
         connector should be using the OpenSSL style configuration
         described in the APR documentation -->
    <!--
    <Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true"
               maxThreads="150" scheme="https" secure="true"
               clientAuth="false" sslProtocol="TLS" />
    -->

    <!-- Define an AJP 1.3 Connector on port 8009 -->
    <Connector port="8109" protocol="AJP/1.3" redirectPort="8443" />

 

    <!-- An Engine represents the entry point (within Catalina) that processes
         every request.  The Engine implementation for Tomcat stand alone
         analyzes the HTTP headers included with the request, and passes them
         on to the appropriate Host (virtual host).
         Documentation at /docs/config/engine.html -->

    <!-- You should set jvmRoute to support load-balancing via AJP ie :
    <Engine name="Standalone" defaultHost="localhost" jvmRoute="jvm1">         
    --> 
    <Engine name="Catalina" defaultHost="localhost" jvmRoute="tomcatA">

      <!--For clustering, please take a look at documentation at:
          /docs/cluster-howto.html  (simple how to)
          /docs/config/cluster.html (reference documentation) -->
      
      <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
            

      <!-- The request dumper valve dumps useful debugging information about
           the request and response data received and sent by Tomcat.
           Documentation at: /docs/config/valve.html -->
      <!--
      <Valve className="org.apache.catalina.valves.RequestDumperValve"/>
      -->

      <!-- This Realm uses the UserDatabase configured in the global JNDI
           resources under the key "UserDatabase".  Any edits
           that are performed against this UserDatabase are immediately
           available for use by the Realm.  -->
      <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
             resourceName="UserDatabase"/>

      <!-- Define the default virtual host
           Note: XML Schema validation will not work with Xerces 2.2.
       -->
      <Host name="localhost"  appBase="webapps"
            unpackWARs="true" autoDeploy="true"
            xmlValidation="false" xmlNamespaceAware="false">

        <!-- SingleSignOn valve, share authentication between web applications
             Documentation at: /docs/config/valve.html -->
        <!--
        <Valve className="org.apache.catalina.authenticator.SingleSignOn" />
        -->

        <!-- Access log processes all example.
             Documentation at: /docs/config/valve.html -->
        <!--
        <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"  
               prefix="localhost_access_log." suffix=".txt" pattern="common" resolveHosts="false"/>
        -->

      </Host>
    </Engine>
  </Service>
</Server>

2 Second tomcatB

<!-- Note:  A "Server" is not itself a "Container", so you may not
     define subcomponents such as "Valves" at this level.
     Documentation at /docs/config/server.html
 -->
<Server port="8205" shutdown="SHUTDOWN">

  <!--APR library loader. Documentation at /docs/apr.html -->
  <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />
  <!--Initialize Jasper prior to webapps are loaded. Documentation at /docs/jasper-howto.html -->
  <Listener className="org.apache.catalina.core.JasperListener" />
  <!-- JMX Support for the Tomcat server. Documentation at /docs/non-existent.html -->
  <Listener className="org.apache.catalina.mbeans.ServerLifecycleListener" />
  <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />

  <!-- Global JNDI resources
       Documentation at /docs/jndi-resources-howto.html
  -->
  <GlobalNamingResources>
    <!-- Editable user database that can also be used by
         UserDatabaseRealm to authenticate users
    -->
    <Resource name="UserDatabase" auth="Container"
              type="org.apache.catalina.UserDatabase"
              description="User database that can be updated and saved"
              factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
              pathname="conf/tomcat-users.xml" />
  </GlobalNamingResources>

  <!-- A "Service" is a collection of one or more "Connectors" that share
       a single "Container" Note:  A "Service" is not itself a "Container", 
       so you may not define subcomponents such as "Valves" at this level.
       Documentation at /docs/config/service.html
   -->
  <Service name="Catalina">
  
    <!--The connectors can use a shared executor, you can define one or more named thread pools-->
    <!--
    <Executor name="tomcatThreadPool" namePrefix="catalina-exec-" 
        maxThreads="150" minSpareThreads="4"/>
    -->
    
    
    <!-- A "Connector" represents an endpoint by which requests are received
         and responses are returned. Documentation at :
         Java HTTP Connector: /docs/config/http.html (blocking & non-blocking)
         Java AJP  Connector: /docs/config/ajp.html
         APR (HTTP/AJP) Connector: /docs/apr.html
         Define a non-SSL HTTP/1.1 Connector on port 8080
    -->
    <Connector port="8082" protocol="HTTP/1.1" 
               connectionTimeout="20000" 
               redirectPort="8443" />
    <!-- A "Connector" using the shared thread pool-->
    <!--
    <Connector executor="tomcatThreadPool"
               port="8080" protocol="HTTP/1.1" 
               connectionTimeout="20000" 
               redirectPort="8443" />
    -->           
    <!-- Define a SSL HTTP/1.1 Connector on port 8443
         This connector uses the JSSE configuration, when using APR, the 
         connector should be using the OpenSSL style configuration
         described in the APR documentation -->
    <!--
    <Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true"
               maxThreads="150" scheme="https" secure="true"
               clientAuth="false" sslProtocol="TLS" />
    -->

    <!-- Define an AJP 1.3 Connector on port 8009 -->
    <Connector port="8209" protocol="AJP/1.3" redirectPort="8443" />

 

    <!-- An Engine represents the entry point (within Catalina) that processes
         every request.  The Engine implementation for Tomcat stand alone
         analyzes the HTTP headers included with the request, and passes them
         on to the appropriate Host (virtual host).
         Documentation at /docs/config/engine.html -->

    <!-- You should set jvmRoute to support load-balancing via AJP ie :
    <Engine name="Standalone" defaultHost="localhost" jvmRoute="jvm1">         
    --> 
    <Engine name="Catalina" defaultHost="localhost" jvmRoute="tomcatB">

      <!--For clustering, please take a look at documentation at:
          /docs/cluster-howto.html  (simple how to)
          /docs/config/cluster.html (reference documentation) -->
      
      <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
            

      <!-- The request dumper valve dumps useful debugging information about
           the request and response data received and sent by Tomcat.
           Documentation at: /docs/config/valve.html -->
      <!--
      <Valve className="org.apache.catalina.valves.RequestDumperValve"/>
      -->

      <!-- This Realm uses the UserDatabase configured in the global JNDI
           resources under the key "UserDatabase".  Any edits
           that are performed against this UserDatabase are immediately
           available for use by the Realm.  -->
      <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
             resourceName="UserDatabase"/>

      <!-- Define the default virtual host
           Note: XML Schema validation will not work with Xerces 2.2.
       -->
      <Host name="localhost"  appBase="webapps"
            unpackWARs="true" autoDeploy="true"
            xmlValidation="false" xmlNamespaceAware="false">

        <!-- SingleSignOn valve, share authentication between web applications
             Documentation at: /docs/config/valve.html -->
        <!--
        <Valve className="org.apache.catalina.authenticator.SingleSignOn" />
        -->

        <!-- Access log processes all example.
             Documentation at: /docs/config/valve.html -->
        <!--
        <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"  
               prefix="localhost_access_log." suffix=".txt" pattern="common" resolveHosts="false"/>
        -->

      </Host>
    </Engine>
  </Service>
</Server>

3 Third tomcatC

<!-- Note:  A "Server" is not itself a "Container", so you may not
     define subcomponents such as "Valves" at this level.
     Documentation at /docs/config/server.html
 -->
<Server port="8305" shutdown="SHUTDOWN">

  <!--APR library loader. Documentation at /docs/apr.html -->
  <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />
  <!--Initialize Jasper prior to webapps are loaded. Documentation at /docs/jasper-howto.html -->
  <Listener className="org.apache.catalina.core.JasperListener" />
  <!-- JMX Support for the Tomcat server. Documentation at /docs/non-existent.html -->
  <Listener className="org.apache.catalina.mbeans.ServerLifecycleListener" />
  <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />

  <!-- Global JNDI resources
       Documentation at /docs/jndi-resources-howto.html
  -->
  <GlobalNamingResources>
    <!-- Editable user database that can also be used by
         UserDatabaseRealm to authenticate users
    -->
    <Resource name="UserDatabase" auth="Container"
              type="org.apache.catalina.UserDatabase"
              description="User database that can be updated and saved"
              factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
              pathname="conf/tomcat-users.xml" />
  </GlobalNamingResources>

  <!-- A "Service" is a collection of one or more "Connectors" that share
       a single "Container" Note:  A "Service" is not itself a "Container", 
       so you may not define subcomponents such as "Valves" at this level.
       Documentation at /docs/config/service.html
   -->
  <Service name="Catalina">
  
    <!--The connectors can use a shared executor, you can define one or more named thread pools-->
    <!--
    <Executor name="tomcatThreadPool" namePrefix="catalina-exec-" 
        maxThreads="150" minSpareThreads="4"/>
    -->
    
    
    <!-- A "Connector" represents an endpoint by which requests are received
         and responses are returned. Documentation at :
         Java HTTP Connector: /docs/config/http.html (blocking & non-blocking)
         Java AJP  Connector: /docs/config/ajp.html
         APR (HTTP/AJP) Connector: /docs/apr.html
         Define a non-SSL HTTP/1.1 Connector on port 8080
    -->
    <Connector port="8083" protocol="HTTP/1.1" 
               connectionTimeout="20000" 
               redirectPort="8443" />
    <!-- A "Connector" using the shared thread pool-->
    <!--
    <Connector executor="tomcatThreadPool"
               port="8080" protocol="HTTP/1.1" 
               connectionTimeout="20000" 
               redirectPort="8443" />
    -->           
    <!-- Define a SSL HTTP/1.1 Connector on port 8443
         This connector uses the JSSE configuration, when using APR, the 
         connector should be using the OpenSSL style configuration
         described in the APR documentation -->
    <!--
    <Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true"
               maxThreads="150" scheme="https" secure="true"
               clientAuth="false" sslProtocol="TLS" />
    -->

    <!-- Define an AJP 1.3 Connector on port 8009 -->
    <Connector port="8309" protocol="AJP/1.3" redirectPort="8443" />

 

    <!-- An Engine represents the entry point (within Catalina) that processes
         every request.  The Engine implementation for Tomcat stand alone
         analyzes the HTTP headers included with the request, and passes them
         on to the appropriate Host (virtual host).
         Documentation at /docs/config/engine.html -->

    <!-- You should set jvmRoute to support load-balancing via AJP ie :
    <Engine name="Standalone" defaultHost="localhost" jvmRoute="jvm1">         
    --> 
    <Engine name="Catalina" defaultHost="localhost" jvmRoute="tomcatC">

      <!--For clustering, please take a look at documentation at:
          /docs/cluster-howto.html  (simple how to)
          /docs/config/cluster.html (reference documentation) -->
      
      <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
            

      <!-- The request dumper valve dumps useful debugging information about
           the request and response data received and sent by Tomcat.
           Documentation at: /docs/config/valve.html -->
      <!--
      <Valve className="org.apache.catalina.valves.RequestDumperValve"/>
      -->

      <!-- This Realm uses the UserDatabase configured in the global JNDI
           resources under the key "UserDatabase".  Any edits
           that are performed against this UserDatabase are immediately
           available for use by the Realm.  -->
      <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
             resourceName="UserDatabase"/>

      <!-- Define the default virtual host
           Note: XML Schema validation will not work with Xerces 2.2.
       -->
      <Host name="localhost"  appBase="webapps"
            unpackWARs="true" autoDeploy="true"
            xmlValidation="false" xmlNamespaceAware="false">

        <!-- SingleSignOn valve, share authentication between web applications
             Documentation at: /docs/config/valve.html -->
        <!--
        <Valve className="org.apache.catalina.authenticator.SingleSignOn" />
        -->

        <!-- Access log processes all example.
             Documentation at: /docs/config/valve.html -->
        <!--
        <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"  
               prefix="localhost_access_log." suffix=".txt" pattern="common" resolveHosts="false"/>
        -->

      </Host>
    </Engine>
  </Service>
</Server>

Horizontal tomcat clustering or tomcat clustering on different machine system

If you want to do horizontal tomcat clustering then you don't need to change in ports

Just left ports unchanged e.g
1. <Server port="8005" shutdown="SHUTDOWN">
2. <Connector port="8080" protocol="HTTP/1.1" 
               connectionTimeout="20000" 
               redirectPort="8443" />

3. <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />

4. Add jvmRoute

<Engine name="Catalina" defaultHost="localhost" jvmRoute="tomcatC">

5. Uncommented clustering tag

<Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>

Add distributed tag in web.xml

Next step to do add distributed tag in web.xml to switching session among the tomcat clustering instance

Make any application, e.g we are making cluster as application folder in webapps

1. TomcatA - >  webapps - > cluster -> WEB-INF -> web.xml
2. TomcatB - >  webapps - > cluster -> WEB-INF -> web.xml
3. TomcatC - >  webapps - > cluster -> WEB-INF -> web.xml

Add  <distributable /> in all three web.xml file

 

<?xml version="1.0" encoding="ISO-8859-1"?>

<web-app xmlns="http://java.sun.com/xml/ns/javaee"
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"
   version="2.5">
  <distributable />
  
</web-app>

Install mod_jk in module

Mod_jk is connector that communicates with apache web server and tomcat. We need to download and copy that connector in module folder of apache
http://tomcat.apache.org/connectors-doc/ we are using latest connector of mod_jk

Make mod_jk.log file in logs folder otherwise apache will throw error

 

Edit in httpd.conf

Open apache -> httpd.conf

Open in any text editor and add (아래의 첫라인을 제외한 내용은 mod_jk.conf 파일에 선언해도 됨)

LoadModule jk_module modules/mod_jk-apache-2.2.4.so 
JkWorkersFile "C:\cluster\Apache\conf\workers.properties" 
JkLogFile "logs/mod_jk.log" 
JkLogLevel error 
JkMount /cluster loadbalancer (만약 여기를 abc 라고 정의한다면 아래의 모든 부분도 abc로 변경해야 함)
JkMount /cluster/* loadbalancer

Worker.properties file

Make a file with name of workers.properties in conf folder. This file tells properties of all tomcat instances. We have to specify all tomcat properties here. Apache will forword request to tomcat through this file

Vertical tomcat clustering this file like (동일한 호스트와 서로 다른 포트번호를 사용함에 주의)

workers.tomcat_home=/tomcatA
workers.java_home=$JAVA_HOME
ps=/
worker.list=tomcatA,tomcatB,tomcatC,loadbalancer

 

worker.tomcatA.port=8109
worker.tomcatA.host=localhost
worker.tomcatA.type=ajp13
worker.tomcatA.lbfactor=1

 

worker.tomcatB.port=8209
worker.tomcatB.host=localhost
worker.tomcatB.type=ajp13
worker.tomcatB.lbfactor=1

worker.tomcatC.port=8309
worker.tomcatC.host=localhost
worker.tomcatC.type=ajp13
worker.tomcatC.lbfactor=1

worker.loadbalancer.type=lb
worker.loadbalancer.balanced_workers=tomcatA,tomcatB,tomcatC
worker.loadbalancer.sticky_session=1

for horizontal tomcat clustering (서로 다른 IP주소와 서로 동일한 포트번호를 사용함에 주의)

workers.tomcat_home=/tomcatA
workers.java_home=$JAVA_HOME
ps=/
worker.list=tomcatA,tomcatB,tomcatC,loadbalancer

 

worker.tomcatA.port=8009
worker.tomcatA.host=192.168.1.1
worker.tomcatA.type=ajp13
worker.tomcatA.lbfactor=1

 

worker.tomcatB.port=8009
worker.tomcatB.host=192.168.1.2
worker.tomcatB.type=ajp13
worker.tomcatB.lbfactor=1

worker.tomcatC.port=8009
worker.tomcatC.host=192.168.1.3
worker.tomcatC.type=ajp13
worker.tomcatC.lbfactor=1

worker.loadbalancer.type=lb
worker.loadbalancer.balanced_workers=tomcatA,tomcatB,tomcatC
worker.loadbalancer.sticky_session=1

lbfactor properties define for load balancing factor, restrict number of  request to send particular tomcat instance
e.g 
worker.tomcatC.lbfactor=100 
increase and decrease request to this tomcatC instance

Restart apache, tomcatA, tomcatB, tomcatC

Check clustering if apache start properly, It is fine without any errors otherwise check problems,basically mod_jk problems come. So download different mod_jk for your machine. Start all tomcats, tomcatA, tomcatB, and tomcatC. If all tomcats is started this means tomcat is working fine.

Open test.jsp on browser and check session id. Check which tomcat on test.jsp is running tomcatB or tomcatC. Close that tomcat, reload test.jsp. Check session id,if session id is same. Then tomcat clustering is working fine.

 

Test jsp file

Make test.jsp in cluster folder of webapps

tomcatA

<%
  session.setAttribute("a","a");
%>
<html>
<head>
<title>Test JSP</title>
</head>

<body>
<table width="100%" border="0" cellspacing="0" cellpadding="0">
  <tr bgcolor="#CCCCCC">
    <td width="13%">TomcatA Machine</td>
    <td width="87%">&nbsp;</td>
  </tr>
  <tr>
    <td>Session ID :</td>
    <td><%=session.getId()%></td>
  </tr>
</table>
</body>
</html>

tomcatB

<%
  session.setAttribute("a","a");
%>
<html>
<head>
<title>Test JSP</title>
</head>

<body>
<table width="100%" border="0" cellspacing="0" cellpadding="0">
  <tr bgcolor="#CCCC00">
    <td width="13%">TomcatB Machine</td>
    <td width="87%">&nbsp;</td>
  </tr>
  <tr>
    <td>Session ID :</td>
    <td><%=session.getId()%></td>
  </tr>
</table>
</body>

</html>

TomcatC

<%
  session.setAttribute("a","a");
%>
<html>
<head>
<title>Test JSP</title>
</head>

<body>
<table width="100%" border="0" cellspacing="0" cellpadding="0">
  <tr bgcolor="#00CCCC">
    <td width="13%">TomcatC Machine</td>
    <td width="87%">&nbsp;</td>
  </tr>
  <tr>
    <td>Session ID :</td>
    <td><%=session.getId()%></td>
  </tr>
</table>
</body>

</html>