mariadb galera Cluster 多主集群安装实录

1: 创建MariaDB.repo
#vim /etc/yum.repos.d/MariaDB.repo

# MariaDB 10.0 CentOS repository list - created 2013-08-23 13:08 UTC
# http://mariadb.org/mariadb/repositories/
[mariadb]
name = MariaDB
baseurl = http://yum.mariadb.org/10.0/centos6-amd64
gpgkey=https://yum.mariadb.org/RPM-GPG-KEY-MariaDB
gpgcheck=1

#yum install MariaDB-Galera-server MariaDB-client galera
安装过程中出现
Transaction check error:
  file /etc/my.cnf from install of MariaDB-common-10.0.12-1.el6.x86_64 conflicts with file from package mariadb-libs-1:5.5.35-3.el7.x86_64
  file /usr/share/mysql/charsets/Index.xml from install of MariaDB-common-10.0.12-1.el6.x86_64 conflicts with file from package mariadb-libs-1:5.5.35-3.el7.x86_64
  file /usr/share/mysql/charsets/armscii8.xml from install of MariaDB-common-10.0.12-1.el6.x86_64 conflicts with file from package mariadb-libs-1:5.5.35-3.el7.x86_64
  file /usr/share/mysql/charsets/ascii.xml from install of MariaDB-common-10.0.12-1.el6.x86_64 conflicts with file from package mariadb-libs-1:5.5.35-3.el7.x86_64
  file /usr/share/mysql/charsets/cp1250.xml from install of MariaDB-common-10.0.12-1.el6.x86_64 conflicts with file from package mariadb-libs-1:5.5.35-3.el7.x86_64
  file /usr/share/mysql/charsets/cp1256.xml from install of MariaDB-common-10.0.12-1.el6.x86_64 conflicts with file from package mariadb-libs-1:5.5.35-3.el7.x86_64
  file /usr/share/mysql/charsets/cp1257.xml from install of MariaDB-common-10.0.12-1.el6.x86_64 conflicts with file from package mariadb-libs-1:5.5.35-3.el7.x86_64
  file /usr/share/mysql/charsets/cp850.xml from install of MariaDB-common-10.0.12-1.el6.x86_64 conflicts with file from package mariadb-libs-1:5.5.35-3.el7.x86_64
  file /usr/share/mysql/charsets/cp852.xml from install of MariaDB-common-10.0.12-1.el6.x86_64 conflicts with file from package mariadb-libs-1:5.5.35-3.el7.x86_64
  file /usr/share/mysql/charsets/cp866.xml from install of MariaDB-common-10.0.12-1.el6.x86_64 conflicts with file from package mariadb-libs-1:5.5.35-3.el7.x86_64
  file /usr/share/mysql/charsets/dec8.xml from install of MariaDB-common-10.0.12-1.el6.x86_64 conflicts with file from package mariadb-libs-1:5.5.35-3.el7.x86_64
  file /usr/share/mysql/charsets/geostd8.xml from install of MariaDB-common-10.0.12-1.el6.x86_64 conflicts with file from package mariadb-libs-1:5.5.35-3.el7.x86_64
  file /usr/share/mysql/charsets/greek.xml from install of MariaDB-common-10.0.12-1.el6.x86_64 conflicts with file from package mariadb-libs-1:5.5.35-3.el7.x86_64
  file /usr/share/mysql/charsets/hebrew.xml from install of MariaDB-common-10.0.12-1.el6.x86_64 conflicts with file from package mariadb-libs-1:5.5.35-3.el7.x86_64
  file /usr/share/mysql/charsets/hp8.xml from install of MariaDB-common-10.0.12-1.el6.x86_64 conflicts with file from package mariadb-libs-1:5.5.35-3.el7.x86_64
  file /usr/share/mysql/charsets/keybcs2.xml from install of MariaDB-common-10.0.12-1.el6.x86_64 conflicts with file from package mariadb-libs-1:5.5.35-3.el7.x86_64
  file /usr/share/mysql/charsets/koi8r.xml from install of MariaDB-common-10.0.12-1.el6.x86_64 conflicts with file from package mariadb-libs-1:5.5.35-3.el7.x86_64
  file /usr/share/mysql/charsets/koi8u.xml from install of MariaDB-common-10.0.12-1.el6.x86_64 conflicts with file from package mariadb-libs-1:5.5.35-3.el7.x86_64
  file /usr/share/mysql/charsets/latin1.xml from install of MariaDB-common-10.0.12-1.el6.x86_64 conflicts with file from package mariadb-libs-1:5.5.35-3.el7.x86_64
  file /usr/share/mysql/charsets/latin2.xml from install of MariaDB-common-10.0.12-1.el6.x86_64 conflicts with file from package mariadb-libs-1:5.5.35-3.el7.x86_64
  file /usr/share/mysql/charsets/latin5.xml from install of MariaDB-common-10.0.12-1.el6.x86_64 conflicts with file from package mariadb-libs-1:5.5.35-3.el7.x86_64
  file /usr/share/mysql/charsets/latin7.xml from install of MariaDB-common-10.0.12-1.el6.x86_64 conflicts with file from package mariadb-libs-1:5.5.35-3.el7.x86_64
  file /usr/share/mysql/charsets/macce.xml from install of MariaDB-common-10.0.12-1.el6.x86_64 conflicts with file from package mariadb-libs-1:5.5.35-3.el7.x86_64
  file /usr/share/mysql/charsets/macroman.xml from install of MariaDB-common-10.0.12-1.el6.x86_64 conflicts with file from package mariadb-libs-1:5.5.35-3.el7.x86_64
  file /usr/share/mysql/charsets/swe7.xml from install of MariaDB-common-10.0.12-1.el6.x86_64 conflicts with file from package mariadb-libs-1:5.5.35-3.el7.x86_64
  file /usr/share/mysql/czech/errmsg.sys from install of MariaDB-Galera-server-10.0.12-1.el6.x86_64 conflicts with file from package mariadb-libs-1:5.5.35-3.el7.x86_64
  file /usr/share/mysql/danish/errmsg.sys from install of MariaDB-Galera-server-10.0.12-1.el6.x86_64 conflicts with file from package mariadb-libs-1:5.5.35-3.el7.x86_64
  file /usr/share/mysql/dutch/errmsg.sys from install of MariaDB-Galera-server-10.0.12-1.el6.x86_64 conflicts with file from package mariadb-libs-1:5.5.35-3.el7.x86_64
  file /usr/share/mysql/english/errmsg.sys from install of MariaDB-Galera-server-10.0.12-1.el6.x86_64 conflicts with file from package mariadb-libs-1:5.5.35-3.el7.x86_64
  file /usr/share/mysql/estonian/errmsg.sys from install of MariaDB-Galera-server-10.0.12-1.el6.x86_64 conflicts with file from package mariadb-libs-1:5.5.35-3.el7.x86_64
  file /usr/share/mysql/french/errmsg.sys from install of MariaDB-Galera-server-10.0.12-1.el6.x86_64 conflicts with file from package mariadb-libs-1:5.5.35-3.el7.x86_64
  file /usr/share/mysql/german/errmsg.sys from install of MariaDB-Galera-server-10.0.12-1.el6.x86_64 conflicts with file from package mariadb-libs-1:5.5.35-3.el7.x86_64
  file /usr/share/mysql/greek/errmsg.sys from install of MariaDB-Galera-server-10.0.12-1.el6.x86_64 conflicts with file from package mariadb-libs-1:5.5.35-3.el7.x86_64
  file /usr/share/mysql/hungarian/errmsg.sys from install of MariaDB-Galera-server-10.0.12-1.el6.x86_64 conflicts with file from package mariadb-libs-1:5.5.35-3.el7.x86_64
  file /usr/share/mysql/italian/errmsg.sys from install of MariaDB-Galera-server-10.0.12-1.el6.x86_64 conflicts with file from package mariadb-libs-1:5.5.35-3.el7.x86_64
  file /usr/share/mysql/japanese/errmsg.sys from install of MariaDB-Galera-server-10.0.12-1.el6.x86_64 conflicts with file from package mariadb-libs-1:5.5.35-3.el7.x86_64
  file /usr/share/mysql/korean/errmsg.sys from install of MariaDB-Galera-server-10.0.12-1.el6.x86_64 conflicts with file from package mariadb-libs-1:5.5.35-3.el7.x86_64
  file /usr/share/mysql/norwegian-ny/errmsg.sys from install of MariaDB-Galera-server-10.0.12-1.el6.x86_64 conflicts with file from package mariadb-libs-1:5.5.35-3.el7.x86_64
  file /usr/share/mysql/norwegian/errmsg.sys from install of MariaDB-Galera-server-10.0.12-1.el6.x86_64 conflicts with file from package mariadb-libs-1:5.5.35-3.el7.x86_64
  file /usr/share/mysql/polish/errmsg.sys from install of MariaDB-Galera-server-10.0.12-1.el6.x86_64 conflicts with file from package mariadb-libs-1:5.5.35-3.el7.x86_64
  file /usr/share/mysql/portuguese/errmsg.sys from install of MariaDB-Galera-server-10.0.12-1.el6.x86_64 conflicts with file from package mariadb-libs-1:5.5.35-3.el7.x86_64
  file /usr/share/mysql/romanian/errmsg.sys from install of MariaDB-Galera-server-10.0.12-1.el6.x86_64 conflicts with file from package mariadb-libs-1:5.5.35-3.el7.x86_64
  file /usr/share/mysql/russian/errmsg.sys from install of MariaDB-Galera-server-10.0.12-1.el6.x86_64 conflicts with file from package mariadb-libs-1:5.5.35-3.el7.x86_64
  file /usr/share/mysql/serbian/errmsg.sys from install of MariaDB-Galera-server-10.0.12-1.el6.x86_64 conflicts with file from package mariadb-libs-1:5.5.35-3.el7.x86_64
  file /usr/share/mysql/slovak/errmsg.sys from install of MariaDB-Galera-server-10.0.12-1.el6.x86_64 conflicts with file from package mariadb-libs-1:5.5.35-3.el7.x86_64
  file /usr/share/mysql/spanish/errmsg.sys from install of MariaDB-Galera-server-10.0.12-1.el6.x86_64 conflicts with file from package mariadb-libs-1:5.5.35-3.el7.x86_64
  file /usr/share/mysql/swedish/errmsg.sys from install of MariaDB-Galera-server-10.0.12-1.el6.x86_64 conflicts with file from package mariadb-libs-1:5.5.35-3.el7.x86_64
  file /usr/share/mysql/ukrainian/errmsg.sys from install of MariaDB-Galera-server-10.0.12-1.el6.x86_64 conflicts with file from package mariadb-libs-1:5.5.35-3.el7.x86_64

的错误，原因可能是之前安装过了mysql或者mariadb引起的冲突，先把之前的包卸载了
#yum -y remove mariadb-libs-1:5.5.35-3.el7.x86_64

然后再安装
#yum install MariaDB-Galera-server MariaDB-client galera
ok，现在一切顺利

拷贝配置文件：
#cp /usr/share/mysql/wsrep.cnf /etc/my.cnf.d/

node1的配置文件，修改一下几个选项：
#vim /etc/my.cnf.d/wsrep.cnf

bind-address=127.0.0.1
wsrep_provider=/usr/lib64/galera/libgalera_smm.so
wsrep_cluster_address="gcomm://"
wsrep_sst_method=rsync
wsrep_sst_receive_address=192.168.150.137

node1添加sst用户，以让其他的node可以以此用户同步数据
[root@client138 ~]# mysql -uroot -pxxxxxxxx
MariaDB [(none)]> > GRANT USAGE ON *.* to sst@‘%‘ IDENTIFIED BY ‘sstpass‘;
MariaDB [(none)]> > GRANT ALL PRIVILEGES on *.* to sst@‘%‘;
MariaDB [(none)]> > FLUSH PRIVILEGES;
MariaDB [(none)]> > quit

node2的配置文件，修改一下几个选项：
#vim /etc/my.cnf.d/wsrep.cnf

bind-address=127.0.0.1
wsrep_provider=/usr/lib64/galera/libgalera_smm.so
wsrep_cluster_address="gcomm://192.168.150.137:4567"
wsrep_sst_method=rsync
wsrep_sst_receive_address=192.168.150.136
wsrep_sst_auth=sst:sstpass

node1,node2设置防火墙，打开端口3306和4567
#iptables -A INPUT -i eno16777736 -p tcp --dport 3306 -j ACCEPT
#iptables -A INPUT -i eno16777736 -p tcp --dport 4567 -j ACCEPT

node1,node2关闭selinux
#setenforce 0


node1 启动：
#/etc/init.d/mysql bootstrap

node2 启动：
#/etc/init.d/mysql start
发现不能启动

查看日志
#more /var/lib/mysql/localhost.localdomain.err

140801 15:32:51 [Note] WSREP: New cluster view: global state: 412b064f-1944-11e4-b9f8-a70b3179cc21:6, view# 5: Primary, number of nodes: 3, my index: 0, protocol version 3
140801 15:32:51 [Warning] WSREP: Gap in state sequence. Need state transfer.
140801 15:32:53 [Note] WSREP: Running: ‘wsrep_sst_rsync --role ‘joiner‘ --address ‘127.0.0.1‘ --auth ‘sst:sstpass‘ --datadir ‘/var/lib/mysql/‘ --defaults-file ‘/etc/my.cnf‘ --parent ‘8388‘  ‘‘ ‘
140801 15:32:54 [Note] WSREP: Prepared SST request: rsync|127.0.0.1:4444/rsync_sst
140801 15:32:54 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
140801 15:32:54 [Note] WSREP: REPL Protocols: 5 (3, 1)
140801 15:32:54 [Note] WSREP: Service thread queue flushed.
140801 15:32:54 [Note] WSREP: Assign initial position for certification: 6, protocol version: 3
140801 15:32:54 [Note] WSREP: Service thread queue flushed.
140801 15:32:54 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (412b064f-1944-11e4-b9f8-a70b3179cc21): 1 (Operation not permitted)
         at galera/src/replicator_str.cpp:prepare_for_IST():447. IST will be unavailable.
140801 15:32:54 [Note] WSREP: Member 0.0 (localhost.localdomain) requested state transfer from ‘*any*‘. Selected 1.0 (localhost.localdomain)(SYNCED) as donor.
140801 15:32:54 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 6)
140801 15:32:54 [Note] WSREP: Requesting state transfer: success, donor: 1
140801 15:32:54 [Warning] WSREP: 1.0 (localhost.localdomain): State transfer to 0.0 (localhost.localdomain) failed: -255 (Unknown error 255)
140801 15:32:54 [ERROR] WSREP: gcs/src/gcs_group.c:gcs_group_handle_join_msg():723: Will never receive state. Need to abort.
140801 15:32:54 [Note] WSREP: gcomm: terminating thread
140801 15:32:54 [Note] WSREP: gcomm: joining thread
140801 15:32:54 [Note] WSREP: gcomm: closing backend

发现Prepared SST request: rsync|127.0.0.1:4444/rsync_sst启用了4444端口来做数据的同步，而这个端口并没有在iptables中设置，和前面一样，设置一下
#iptables -A INPUT -i eno16777736 -p tcp --dport 4444 -j ACCEPT


测试：
在任何一台服务器进行添加，删除，修改操作，都会同步到其他的服务器，是现在主主的模式，当然前提是表引擎必须是innodb，因为galera目前只支持innodb的表。


PS：
对于只有2个节点的Galera Cluster和其他集群软件一样，需要面对极端情况下的"脑裂"状态。
为了避免这种问题，Galera引入了"arbitrator(仲裁人)"。
"仲裁人"节点上没有数据,它在集群中的作用就是在集群发生分裂时进行仲裁，集群中可以有多个"仲裁人"节点。
"仲裁人"节点加入集群的方法如下:
# garbd -a gcomm://192.168.1.137:4567 -g my_wsrep_cluster -d
# 注释：参数说明: -d：以daemon模式运行 -a：集群地址 -g： 集群名称9.测试集群是否配置好参数：登陆数据库：查看如下几个参数：

MariaDB [(none)]> SHOW VARIABLES LIKE ‘wsrep_cluster_address‘;
#如果配置了指向集群地址，上面那个参数值，应该是你指定集群的IP地址

# 此参数查看是否开启
MariaDB [(none)]> show status like ‘wsrep_ready‘;

# 查看集群的成员数
MariaDB [(none)]> show status like ‘wsrep_cluster_size‘;

# 这个查看wsrep的相关参数
MariaDB [(none)]> show status like ‘wsrep%‘;

mariadb galera Cluster 多主集群安装实录,古老的榕树,5-wow.com