GlusterFS

Network ports

  • These are the tcp ports to open in your firewall

Port

Description

616

GlusterFS

38465

GlusterFS

38466

GlusterFS

38468

GlusterFS

38469

GlusterFS

24007

GlusterFSd

49153

Bricks

Installation

  • Install GlusterFS repository

wget http://download.gluster.org/pub/gluster/glusterfs/LATEST/CentOS/glusterfs-epel.repo
  • Install packages

yum install glusterfs{,-server,-fuse,-geo,-replication}
  • Start the service

service glusterd start

Configuration

  • Make a server trust another Gluster node

gluster peer probe <ip>
  • Lets asume you have a partition mounted to /export/test to distribute with GlusterFS on node1 and node2

  • Setup a volume

gluster volume create test replica 2 node1:/export/test node2:/export/test
gluster volume start test
  • Now you can mount and use the volume

mount -t glusterfs node1:/export/test /mnt
  • For redundant mount insert the following into your /etc/fstab

$GFS1_NODE:/export/test /mnt glusterfs defaults,_netdev,backupvolfile-server=$GFS2_NODE 0 0

Peers

  • Add a new one

gluster peer probe <ip>
  • Show status

gluster peer status
  • Remove one

gluster peer detach <ip>

Volumes

  • Create a new one

gluster volume create test replica 2 node1:/export/test node2:/export/test
gluster volume start test
  • List all volumes

gluster volume status
  • Remove one

gluster volume remove-brick test node1:/export/test node2:/export/test
gluster volume stop test
gluster volume remove test
  • Add a new disk to a volume

gluster volume add-brick <volname> replica 2 node3:/export/moretest
  • Manage access by ip

gluster volume set testvol auth.allow 192.168.1.1
# or
gluster volume set testvol auth.allow all
gluster volume set testvol auth.reject 192.168.10.*
  • How many space to reserve for logs / meta data?

gluster volume set cluster.min-free-disk 5%
  • Enable self healing (on by default)

gluster volume set cluster.self-heal-daemon on

NFS export

  • Start rpcbind

  • Start nfslock (rpcstatd)

  • Start glusterd

  • Adjust firewall

Port

Description

2049

GlusterFS (NFS)

111

RPCbind

54539

RCP statd

38003

RPCbind

  • Now you can mount it with

mount -t nfs -o mountproto=tcp,vers=3 ip1:/testme /mnt

Quota

gluster volume quota <volname> enable
gluster volume quota <volname> limit-usage <directory> 10GB
gluster volume quota <volname> list
gluster volume quota <volname> remove <directory>

Performance tuning

  • Performance information

gluster volume top <volname> read-perf
gluster volume top <volname> write-perf
  • Profiling

gluster volume profile <volname> start
gluster volume profile <volname> info
gluster volume profile <volname> stop
  • Setting read cache size (default 32MB)

gluster volume set <volname> performance.cache-size 256MB
  • Stripe block size

gluster volume set cluster.stripe-block-size 128KB
  • I/O threads

gluster volume set performance.io-thread-count 32

Troubleshooting

  • requested NFS version or transport protocol is not supported -> you try to mount with UDP or you didnt start rpcbind, nfslock, glusterd in the right order

  • Protocol not supported -> you try to mount with version 4 instead of 3

  • node is already part of another cluster -> delete /var/lib/glusterd/peers/*

  • split brain means that we detected changes to both replicas

gluster volume heal <volname> full
gluster volume heal <volname> info
  • {path} or a prefix of it is already part of a volume -> you forgot to remove the brick before deleting the volume

setfattr -x trusted.glusterfs.volume-id $brick_path
setfattr -x trusted.gfid $brick_path
rm -rf $brick_path/.glusterfs
service glusterd restart