Unconventional use of nfs-ganesha on Ceph

by Lee, Chien-Pang | April 16, 2022

In spite of the trends towards highly scalable Object Storage in the cloud computing world, filesystem-based storage is still irreplaceable in many aspects. Its hierarchical structures, human-friendly file names and above all, mature applications with prevalent protocols such as NFS, Samba/CIFS, are rooted firmly in practical IT building blocks and daily usages. Ceph FileSystem comes with a built-in mechanism, cephfs, that can be exported via networking and mounted as a remote filesystem storage, functioning just like NFS. As built-in as it sounds, the underlying protocol is “ceph”.

    sh-4.4# mount -t ceph /mnt/ceph
    sh-4.4# mount | grep ceph` on /mnt/ceph type ceph (rw,relatime,seclabel,acl)
    sh-4.4# df -h /mnt/ceph
    Filesystem      Size  Used Avail Use% Mounted on      140G  4.7G  135G   4% /mnt/ceph

That’s no sweat and many people are happy with this. Nonetheless, here we are to answer to those who have to run NFS on top of Ceph for some reason, be it legacy setup, current IT restriction, old client software or something more complex in a our case when cephfs has to be bundled to a dedicated storage network which is not reachable directly by the VMs spawned on compute nodes in a different network. Whatever it is, nfs-ganesha, is here for anyone who needs his Ceph to work with NFS.

Because of the complexity of setting up Ceph storage cluster, most documents that one can find online propose to use Kubernetes or Docker to ease deployment efforts; as a result, nfs-ganesha is mostly recommended to be installed on top of containerization such that cluster HA can be achieved by the help of K8s cluster.

We do it here differently. Let’s run Ganesha as a native systemd service and manage HA with haproxy in Active/Active mode. First thing first, install needed yum repository:

    sh-4.4# dnf -y install centos-release-nfs-ganesha4

Then come these 2 major packages that give us daemon, rados-ng/rados-kv and config files

    sh-4.4# dnf -y install nfs-ganesha-ceph.x86_64 nfs-ganesha-rados-grace

Yum package manager helps me with dependencies and I end up having these installed. Samba shows up as well.

    (1/16): nfs-ganesha-selinux-4.0-1.el8s.noarch.rpm                        299 kB/s |  38 kB     00:00    
    (2/16): libtalloc-2.3.3-1.el8.x86_64.rpm                                 343 kB/s |  49 kB     00:00    
    (3/16): nfs-ganesha-rados-grace-4.0-1.el8s.x86_64.rpm                    368 kB/s |  58 kB     00:00    
    (4/16): lmdb-libs-0.9.24-1.el8.x86_64.rpm                                352 kB/s |  58 kB     00:00    
    (5/16): libtevent-0.11.0-0.el8.x86_64.rpm                                1.3 MB/s |  50 kB     00:00    
    (6/16): libtdb-1.4.4-1.el8.x86_64.rpm                                    238 kB/s |  59 kB     00:00    
    (7/16): logrotate-3.14.0-4.el8.x86_64.rpm                                340 kB/s |  86 kB     00:00    
    (8/16): libldb-2.4.1-1.el8.x86_64.rpm                                    563 kB/s | 188 kB     00:00    
    (9/16): samba-common-4.15.5-5.el8.noarch.rpm                             658 kB/s | 224 kB     00:00    
    (10/16): samba-common-libs-4.15.5-5.el8.x86_64.rpm                       624 kB/s | 177 kB     00:00    
    (11/16): libntirpc-4.0-1.el8s.x86_64.rpm                                 420 kB/s | 137 kB     00:00    
    (12/16): nfs-ganesha-4.0-1.el8s.x86_64.rpm                               1.1 MB/s | 734 kB     00:00    
    (13/16): libcephfs2-12.2.7-9.el8.x86_64.rpm                              733 kB/s | 486 kB     00:00    
    (14/16): nfs-ganesha-ceph-4.0-1.el8s.x86_64.rpm                           90 kB/s |  59 kB     00:00    
    (15/16): libwbclient-4.15.5-5.el8.x86_64.rpm                             137 kB/s | 124 kB     00:00    
    (16/16): samba-client-libs-4.15.5-5.el8.x86_64.rpm                       3.8 MB/s | 5.5 MB     00:01    

In /etc/ganesha/ceph.conf we modify couple values

        Enable_NLM = false;
        Enable_RQUOTA = false;
        Protocols = 4;
        Bind_addr =;

        RecoveryBackend = rados_ng;
        Minor_Versions = 1,2;
        Lease_Lifetime = 10;
        Grace_Period = 20;

        Dir_Chunk = 0;

        Protocols = 4;
        Transports = TCP;
        Path = /;
        Pseudo = /;
        Access_Type = RW;
        Attr_Expiration_Time = 0;
        Squash = root;
        FSAL {
                Name = CEPH;

        Ceph_Conf = /etc/ceph/ceph.conf;

        pool = "cephfs_data";

Bind_addr is fixed to If it is not specifically changed to fixed IP, default is, representing all network interfaces; this will conflict with haproxy which would bind 2049 port to VIP. The example here is with such cluster constructs.

  • VIP:
  • Control node 1 (c1):
  • Control node 2 (c2):
  • Control node 3 (c3):
  • Compute node x 3 (not relevant here)
  • Storage node x 5 (not relevant here)

Start nfs-ganesha on control nodes.

    # systemctl start nfs-ganesha
    # systemctl status nfs-ganesha
    ● nfs-ganesha.service - NFS-Ganesha file server
       Loaded: loaded (/usr/lib/systemd/system/nfs-ganesha.service; disabled; vendor preset: disabled)
       Active: active (running) since Fri 2022-04-15 18:40:08 CST; 7h ago
         Docs: http://github.com/nfs-ganesha/nfs-ganesha/wiki
     Main PID: 16201 (ganesha.nfsd)
        Tasks: 45 (limit: 179687)
       Memory: 54.5M
       CGroup: /system.slice/nfs-ganesha.service
               └─16201 /usr/bin/ganesha.nfsd -L /var/log/ganesha/ganesha.log -f /etc/ganesha/ganesha.conf -N NIV_EVENT
    Apr 15 18:40:08 c1 systemd[1]: Starting NFS-Ganesha file server...
    Apr 15 18:40:08 c1 systemd[1]: Started NFS-Ganesha file server.

Now we have 3 A/A nfs-ganesha services, each on a control node. Next is the configuration for HA with haproxy. In /etc/haproxy/haproxy.cfg, add the section below.

    listen ceph_nfs_ganesha
      balance  source
      option  tcpka
      option  tcplog
      server c1 check inter 2000 rise 2 fall 5
      server c2 check inter 2000 rise 2 fall 5
      server c3 check inter 2000 rise 2 fall 5

where we tell haproxy to control forward incoming connections to our VIP and port 2049 (standard NFS port) to any of the active NFS servers running on c1, c2 and c3 with port 2048. The last question is who would be in charge of managing where VIP (Virtual IP) runs on which of the 3 control nodes. The answer is pacemaker/corosync. Below is a snippet of create VIP resouce with pacemaker.

    # pcs resource create vip ocf:heartbeat:IPaddr2 ip="" op monitor interval="30s"
    # pcs status
    Cluster name: cube-8kWwZRbkBPcR6xk3
    Cluster Summary:
      * Stack: corosync
      * Current DC: c1 (version 2.1.2-4.el8-ada5c3b36e2) - partition with quorum
      * Last updated: Sat Apr 16 02:36:41 2022
      * Last change:  Fri Apr 15 20:08:14 2022 by root via crm_resource on c3
      * 6 nodes configured
      * 9 resource instances configured

    Node List:
      * Online: [ c1 c2 c3 ]
      * RemoteOnline: [ p1 p2 p3 ]

    Full List of Resources:
      * vip (ocf::heartbeat:IPaddr2):        Started c1
      * haproxy     (systemd:haproxy-ha):    Started c1
      * cinder-volume       (systemd:openstack-cinder-volume):       Started c2
      * Clone Set: ovndb_servers-clone [ovndb_servers] (promotable):
        * Masters: [ c1 ]
        * Slaves: [ c2 c3 ]
      * p1  (ocf::pacemaker:remote):         Started c1
      * p2  (ocf::pacemaker:remote):         Started c1
      * p3  (ocf::pacemaker:remote):         Started c1

To verify, we can mount it with standard command and you can test out HA by stopping couple of the nfs-ganesha services on control nodes.

    sh-4.4# mount -t nfs /mnt/nfs
    sh-4.4# mount | grep nfs on /mnt/nfs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=,local_lock=none,addr=
    sh-4.4# df -h /mnt/ceph
    Filesystem      Size  Used Avail Use% Mounted on      139G  4.7G  134G   4% /mnt/ceph