GlusterFS¶
Network ports¶
These are the tcp ports to open in your firewall
Port |
Description |
---|---|
616 |
GlusterFS |
38465 |
GlusterFS |
38466 |
GlusterFS |
38468 |
GlusterFS |
38469 |
GlusterFS |
24007 |
GlusterFSd |
49153 |
Bricks |
Installation¶
Install GlusterFS repository
wget http://download.gluster.org/pub/gluster/glusterfs/LATEST/CentOS/glusterfs-epel.repo
Install packages
yum install glusterfs{,-server,-fuse,-geo,-replication}
Start the service
service glusterd start
Configuration¶
Make a server trust another Gluster node
gluster peer probe <ip>
Lets asume you have a partition mounted to /export/test to distribute with GlusterFS on node1 and node2
Setup a volume
gluster volume create test replica 2 node1:/export/test node2:/export/test
gluster volume start test
Now you can mount and use the volume
mount -t glusterfs node1:/export/test /mnt
For redundant mount insert the following into your
/etc/fstab
$GFS1_NODE:/export/test /mnt glusterfs defaults,_netdev,backupvolfile-server=$GFS2_NODE 0 0
Peers¶
Add a new one
gluster peer probe <ip>
Show status
gluster peer status
Remove one
gluster peer detach <ip>
Volumes¶
Create a new one
gluster volume create test replica 2 node1:/export/test node2:/export/test
gluster volume start test
List all volumes
gluster volume status
Remove one
gluster volume remove-brick test node1:/export/test node2:/export/test
gluster volume stop test
gluster volume remove test
Add a new disk to a volume
gluster volume add-brick <volname> replica 2 node3:/export/moretest
Manage access by ip
gluster volume set testvol auth.allow 192.168.1.1
# or
gluster volume set testvol auth.allow all
gluster volume set testvol auth.reject 192.168.10.*
How many space to reserve for logs / meta data?
gluster volume set cluster.min-free-disk 5%
Enable self healing (on by default)
gluster volume set cluster.self-heal-daemon on
NFS export¶
Start rpcbind
Start nfslock (rpcstatd)
Start glusterd
Adjust firewall
Port |
Description |
---|---|
2049 |
GlusterFS (NFS) |
111 |
RPCbind |
54539 |
RCP statd |
38003 |
RPCbind |
Now you can mount it with
mount -t nfs -o mountproto=tcp,vers=3 ip1:/testme /mnt
Quota¶
gluster volume quota <volname> enable
gluster volume quota <volname> limit-usage <directory> 10GB
gluster volume quota <volname> list
gluster volume quota <volname> remove <directory>
Performance tuning¶
Performance information
gluster volume top <volname> read-perf
gluster volume top <volname> write-perf
Profiling
gluster volume profile <volname> start
gluster volume profile <volname> info
gluster volume profile <volname> stop
Setting read cache size (default 32MB)
gluster volume set <volname> performance.cache-size 256MB
Stripe block size
gluster volume set cluster.stripe-block-size 128KB
I/O threads
gluster volume set performance.io-thread-count 32
Troubleshooting¶
requested NFS version or transport protocol is not supported -> you try to mount with UDP or you didnt start rpcbind, nfslock, glusterd in the right order
Protocol not supported -> you try to mount with version 4 instead of 3
node is already part of another cluster -> delete /var/lib/glusterd/peers/*
split brain means that we detected changes to both replicas
gluster volume heal <volname> full
gluster volume heal <volname> info
{path} or a prefix of it is already part of a volume
-> you forgot to remove the brick before deleting the volume
setfattr -x trusted.glusterfs.volume-id $brick_path
setfattr -x trusted.gfid $brick_path
rm -rf $brick_path/.glusterfs
service glusterd restart