搭建LVS+Keepalived+nginx+tomcat高可用性，高性能jsp集群
浏览数：40 /
时间：2015年06月09日
LVS-master:192.168.0.210
LVS-backup:192.168.0.211
LVS-VIP:192.168.0.209
nginx+tomcat:192.168.0.212
nginx+tomcat:192.168.0.227
安装nginx所需包：
Nginx-1.6.0.tar.gz和pcre-8.35.zip
一.安装pcre-8.35
1 #unzip pcre-8.35.zip 2 #cd pcre-8.35 3 #./configure 4 #make 5 #make install
二.安装nginx
1 #tar -zxvf nginx-1.6.0.tar.gz 2 #cd nginx-1.6.0 3 #./configure --with-http_stub_status_module --with-http_ssl_module 4 #make 5 #make install
在第三部检查编译安装环境，提示缺少什么包就安装什么包
例：提示错误：
1 SSL modules require the OpenSSL library
解决办法为安装openssl
1 yum –y install openssl openssl-devel
nginx安装成功后安装目录为/usr/local/nginx
三.为nginx单独配置用户，并为tomcat设置单独的目录。安全性和可维护性高
1.tomcat的目录设置为/home/www/web(没有该目录就新建)
更改tomcat的server.xml文件，在Host name="localhost"处将appBase=""的指向路径改为/home/www/web
2.创建用户www和用户组www为该文件目录的使用权限者
1 #/usr/sbin/groupadd www 2 #/usr/sbin/useradd -g www www -s /sbin/nologin 3 #mkdir -p /home/www 4 #chmod +w /home/www 5 #chown -R www:www/home/www
四.配置nginx
1 #cd /usr/local/nginx
在conf文件夹中新建proxy.conf，用于配置一些代理参数，内容如下：
1 # proxy.conf 2 proxy_redirect off; 3 proxy_set_header Host $host; 4 proxy_set_header X-Real-IP $remote_addr; 5 client_max_body_size 10m; 6 client_body_buffer_size 128k; 7 proxy_connect_timeout 90; 8 proxy_send_timeout 90; 9 proxy_read_timeout 90; 10 proxy_buffer_size 4k; 11 proxy_buffers 4 32k; 12 proxy_busy_buffers_size 64k; 13 proxy_temp_file_write_size 64k;
修改conf中的nginx.conf，配置如下
1 user www www; 2 worker_processes 8; 3 4 error_log /usr/local/nginx/logs/error.log; 5 error_log /usr/local/nginx/logs/error.log notice; 6 error_log /usr/local/nginx/logs/error.log info; 7 8 pid /usr/local/nginx/logs/nginx.pid; 9 10 #工作模式及连接数上限 11 events { 12 use epoll; 13 14 worker_connections 65535; 15 } 16 17 #设定http服务器，利用它的反向代理功能提供负载均衡支持 18 http { 19 #设定mime类型 20 include mime.types; 21 default_type application/octet-stream; 22 include /usr/local/nginx/conf/proxy.conf; 23 #log_format main ‘$remote_addr - $remote_user [$time_local] "$request" ‘ 24 # ‘$status $body_bytes_sent "$http_referer" ‘ 25 # ‘"$http_user_agent" "$http_x_forwarded_for"‘; 26 27 #access_log logs/access.log main; 28 29 #设定请求缓冲 30 server_names_hash_bucket_size 128; 31 client_header_buffer_size 32K; 32 large_client_header_buffers 4 32k; 33 # client_max_body_size 8m; 34 35 sendfile on; 36 tcp_nopush on; 37 tcp_nodelay on; 38 39 #keepalive_timeout 0; 40 keepalive_timeout 65; 41 42 gzip on; 43 gzip_min_length 1k; 44 gzip_buffers 4 16k; 45 gzip_http_version 1.1; 46 gzip_comp_level 2; 47 gzip_types text/plain application/x-javascript text/css application/xml; 48 gzip_vary on; 49 50 server { 51 server_name _; 52 return 404; 53 } 54 #此处为你tomcat的地址，可以写多个tomcat地址 55 upstream tomcat_pool { 56 # server tomcat地址:端口号 weight=4 max_fails=2 fail_timeout=30s; 57 server 192.168.0.212:8080 weight=4 max_fails=2 fail_timeout=30s; 58 } 59 60 server { 61 listen 80; 62 server_name www.yourdomain.com;#此处替换为你自己的网址，如有多个中间用空格 63 index jump.jsp index.htm index.html index.do;#设定访问的默认首页地址 64 root /home/www/web/ROOT; #设定网站的资源存放路径 65 66 #charset koi8-r; 67 68 #access_log logs/host.access.log main; 69 70 # location / { 71 # root html; 72 # index index.html index.htm; 73 # } 74 75 location ~ \.(jsp|jspx|dp)?$ #所有JSP的页面均交由tomcat处理 76 { 77 proxy_set_header Host $host; 78 proxy_set_header X-Real-IP $remote_addr; 79 proxy_pass http://tomcat_pool;#转向tomcat处理 80 } 81 #设定访问静态文件直接读取不经过tomcat 82 location ~ .*\.(htm|html|gif|jpg|jpeg|png|bmp|swf|ioc|rar|zip|txt|flv|mid|doc|ppt|pdf|xls|mp3|wma)$ 83 { 84 expires 30d; 85 } 86 87 location ~ .*\.(js|css)?$ 88 { 89 expires 1h; 90 } 91 log_format access ‘$remote_addr - $remote_user [$time_local] "$request" ‘ 92 ‘$status $body_bytes_sent "$http_referer" ‘ 93 ‘"$http_user_agent" $http_x_forwarded_for‘; 94 access_log /usr/local/nginx/logs/ubitechtest.log access;#设定访问日志的存放路径 95 96 97 98 error_page 500 502 503 504 /50x.html; 99 location = /50x.html { 100 root html; 101 } 102 103 104 # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000 105 # 106 #location ~ \.php$ { 107 # root html; 108 # fastcgi_pass 127.0.0.1:9000; 109 # fastcgi_index index.php; 110 # fastcgi_param SCRIPT_FILENAME /scripts$fastcgi_script_name; 111 # include fastcgi_params; 112 #} 113 114 # deny access to .htaccess files, if Apache‘s document root 115 # concurs with nginx‘s one 116 # 117 #location ~ /\.ht { 118 # deny all; 119 #} 120 } 121 122 server { 123 listen 80; 124 server_name bbs.yourdomain.com; 125 location / { 126 root /home/www/web/springmvc; #设定网站的资源存放路径 127 index index.jsp index.htm index.html index.do welcome.jsp;#设定访问的默认首页地址 128 } 129 130 location ~ \.(jsp|jspx|dp)?$ #所有JSP的页面均交由tomcat处理 131 { 132 proxy_set_header Host $host; 133 proxy_set_header X-Real-IP $remote_addr; 134 proxy_pass http://tomcat_pool;#转向tomcat处理 135 } 136 #设定访问静态文件直接读取不经过tomcat 137 location ~ .*\.(htm|html|gif|jpg|jpeg|png|bmp|swf|ioc|rar|zip|txt|flv|mid|doc|ppt|pdf|xls|mp3|wma)$ 138 { 139 expires 30d; 140 } 141 142 location ~ .*\.(js|css)?$ 143 { 144 expires 1h; 145 } 146 log_format jsp ‘$remote_addr - $remote_user [$time_local] "$request" ‘ 147 ‘$status $body_bytes_sent "$http_referer" ‘ 148 ‘"$http_user_agent" $http_x_forwarded_for‘; 149 access_log /usr/local/nginx/logs/ubitechztt.log jsp;#设定访问日志的存放路径 150 151 152 153 error_page 500 502 503 504 /50x.html; 154 location = /50x.html { 155 root html; 156 } 157 158 } 159 160 # another virtual host using mix of IP-, name-, and port-based configuration 161 # 162 #server { 163 # listen 8000; 164 # listen somename:8080; 165 # server_name somename alias another.alias; 166 167 # location / { 168 # root html; 169 # index index.html index.htm; 170 # } 171 #} 172 173 }
nginx启动时提示错误：
1 /usr/local/nginx/sbin/nginx -t 2 /usr/local/nginx/sbin/nginx: error while loading shared libraries: libpcre.so.1: cannot open shared object file: No such file or directory
解决办法为：
ln -s /usr/local/lib/libpcre.so.1 /lib64
32位系统则：
1 ln -s /usr/local/lib/libpcre.so.1 /lib
附nginx常用命令
1 #检测配置文件 2 #/usr/local/nginx/sbin/nginx -t 3 #启动nginx 4 /usr/local/nginx/sbin/nginx 5 echo "/usr/local/nginx/sbin/nginx" >> /etc/rc.local 6 #查看nginx主进程号 7 #ps -ef | grep "nginx: master process" | grep -v "grep" | awk -F ‘ ‘ ‘{print $2}‘ 8 #停止nginx 9 #/usr/local/nginx/sbin/nginx -s stop 10 #平滑重启 11 #/usr/local/nginx/sbin/nginx -s reload
测试时为在/home/www/web中新建一个index.jsp
启动tomcat，启动nginx
通过访问ip地址即可看到，nginx+tomcat配置完毕。
五.配置lvs+keepalived
1.安装LVS前系统需要安装popt-static，kernel-devel，make，gcc，openssl-devel，lftp，libnl*，popt*
1 #yum –y install popt-static，kernel-devel，make，gcc，openssl-devel，lftp，libnl*，popt* 2 #ln -s /usr/src/kernels/2.6.32-431.17.1.el6.x86_64/ /usr/src/linux 3 #tar -zxvf ipvsadm-1.26.tar.gz 4 #cd ipvsadm-1.26 5 #make && make install
2.安装keepalived
1 #wget http://www.keepalived.org/software/keepalived-1.2.13.tar.gz 2 #tar –zxvf keepalived-1.2.13.tar.gz 3 #cd keepalived-1.2.13 4 #./configure 5 #make && make install 6 ######### 将keepalived做成启动服务，方便管理########## 7 # cp /usr/local/etc/rc.d/init.d/keepalived /etc/init.d/ 8 # cp /usr/local/etc/sysconfig/keepalived /etc/sysconfig/ 9 # mkdir /etc/keepalived/ 10 # cp /usr/local/etc/keepalived/keepalived.conf /etc/keepalived/ 11 # cp /usr/local/sbin/keepalived /usr/sbin/ 12 # service keepalived start | stop
3.开启路由转发
1 #vi /etc/sysctl.conf 2 #sysctl –p
4.配置keepalived
1 #vi /etc/keepalived/keepalived.conf
keepalive.conf具体如下：
1 ! Configuration File for keepalived 2 3 global_defs { 4 notification_email { 5 [email protected] 6 } 7 notification_email_from [email protected] 8 smtp_server send.one.com 9 smtp_connect_timeout 30 10 router_id LVS_MASTER #备份服务器上将MASTER改为BACKUP 11 } 12 13 vrrp_instance VI_1 { 14 state MASTER #备份服务器上将MASTER改为BACKUP 15 interface eth0 #该网卡名字需要查看具体服务器的网口 16 virtual_router_id 51 17 priority 100 # 备份服务上将100改为90 18 advert_int 1 19 authentication { 20 auth_type PASS 21 auth_pass 1111 22 } 23 virtual_ipaddress { 24 192.168.0.209 25 #(如果有多个VIP，继续换行填写.) 26 } 27 } 28 29 virtual_server 192.168.0.209 80 { 30 delay_loop 6 #(每隔6秒查询realserver状态) 31 lb_algo rr #(rr 算法) 32 lb_kind DR #(Direct Route) 33 nat_mask 255.255.255.0 34 # persistence_timeout 50 #(同一IP的连接60秒内被分配到同一台realserver) 35 protocol TCP #(用TCP协议检查realserver状态) 36 37 real_server 192.168.0.212 80 { 38 weight 1 #(权重) 39 TCP_CHECK { 40 connect_timeout 10 #(10秒无响应超时) 41 nb_get_retry 3 42 delay_before_retry 3 43 connect_port 80 44 } 45 } 46 real_server 192.168.0.227 80 { 47 weight 1 48 TCP_CHECK { 49 connect_timeout 10 50 nb_get_retry 3 51 delay_before_retry 3 52 connect_port 80 53 } 54 } 55 56 }
需要注意的是{前面需要有空格，我在配置时TCP_CHECK没有空格导致无法找到real_server
5.配置realserver(即tomcat+nginx那物理机)
1 #vi /etc/init.d/lvs_real.sh
lvs_real.sh具体配置如下：
1 # description: Config realserver lo and apply noarp 2 3 SNS_VIP=192.168.0.209 4 5 /etc/rc.d/init.d/functions 6 7 case "$1" in 8 start) 9 ifconfig lo:0 $SNS_VIP netmask 255.255.255.255 broadcast $SNS_VIP 10 /sbin/route add -host $SNS_VIP dev lo:0 11 echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore 12 echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce 13 echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore 14 echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce 15 sysctl -p >/dev/null 2>&1 16 echo "RealServer Start OK" 17 18 ;; 19 stop) 20 ifconfig lo:0 down 21 route del $SNS_VIP >/dev/null 2>&1 22 echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore 23 echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce 24 echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore 25 echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce 26 echo "
1 #chmod +x /roo/lvs_real.sh 2 #/etc/init.d/lvs_real.sh start
启动提示权限不够，赋权即可
查看网络ifconfig命令
6.测试lvs+keepalived
Master上输入命令ip add
VIP绑定成功，ipvsadm可以查看212和227这2台物理机连接情况
Backup上输入命令
解析域名，测试访问，LVS转发,在master上停掉keepalived
经历延迟后继续可以访问，同时在backup上输入命令ip add可以查看vip已经转至backup机器上
重启Master上的服务，又重新切换回来
Jsp集群中session共享使用memcached，具体配置可参考
郑重声明：本站内容如果来自互联网及其他传播媒体，其版权均属原媒体及文章作者所有。转载目的在于传递更多信息及用于网络分享，并不代表本站赞同其观点和对其真实性负责，也不构成任何其他建议。
