这篇文章给大家介绍如何搭建BeeGFS开发环境，内容非常详细，感兴趣的小伙伴们可以参考借鉴，希望对大家能有所帮助。
安装软件包
我的测试环境使用CentOS 7， 安装BeeGFS的V6版本。
$ rpm -ivh beegfs_rpm/*.rpm
$ rpm -qa | grep bee
beegfs-meta-6.18-el7.x86_64
beegfs-admon-6.18-el7.x86_64
beegfs-utils-6.18-el7.x86_64
beegfs-helperd-6.18-el7.x86_64
beegfs-storage-6.18-el7.x86_64
beegfs-opentk-lib-6.18-el7.x86_64
beegfs-client-6.18-el7.noarch
beegfs-common-6.18-el7.noarch
beegfs-mgmtd-6.18-el7.x86_64
$ wget https://www.beegfs.io/release/beegfs_6/dists/beegfs-rhel7.repo -O /etc/yum.repos.d/beegfs-rhel7.repo
$ yum install -y beegfs-mgmtd beegfs-meta libbeegfs-ib beegfs-storage libbeegfs-ib beegfs-client beegfs-helperd beegfs-utils beegfs-admon
$ yum install -y mdadm
初始化配置
初始化管理服务
sacd01 $ rm -rfv /beegfs_mgmtd
sacd01 $ /opt/beegfs/sbin/beegfs-setup-mgmtd -p /beegfs_mgmtd
Preparing storage directory: /beegfs_mgmtd
* Creating format.conf file...
Updating config file: /etc/beegfs/beegfs-mgmtd.conf
* Setting storage directory in config file...
* Disabling usage of uninitialized storage directory in config file...
All done.
初始化元数据服务
sacd01 $ rm /beegfs_meta/meta* -rf
sacd01 $ /opt/beegfs/sbin/beegfs-setup-meta -p /beegfs_meta/meta1 -s 2 -m 172.29.39.125
Preparing storage directory: /beegfs_meta/meta1
* Creating format.conf file...
* Creating server numeric ID file: /beegfs_meta/meta1/nodeNumID
Updating config file: /etc/beegfs/beegfs-meta.conf
* Setting management host: 172.29.39.125
* Setting storage directory in config file...
* Disabling usage of uninitialized storage directory in config file...
* Setting usage of extended attributes to: true
All done.
sacd02 $ rm /beegfs_meta/meta* -rf
sacd02 $ /opt/beegfs/sbin/beegfs-setup-meta -p /beegfs_meta/meta2 -s 3 -m 172.29.39.125
Preparing storage directory: /beegfs_meta/meta2
* Creating format.conf file...
* Creating server numeric ID file: /beegfs_meta/meta2/nodeNumID
Updating config file: /etc/beegfs/beegfs-meta.conf
* Setting management host: 172.29.39.125
* Setting storage directory in config file...
* Disabling usage of uninitialized storage directory in config file...
* Setting usage of extended attributes to: true
All done.
初始化存储服务
sacd01 $ rm /beegfs_data/data* -rf
sacd01 $ /opt/beegfs/sbin/beegfs-setup-storage -p /beegfs_data/data1 -s 4 -i 401 -m 172.29.39.125
Preparing storage target directory: /beegfs_data/data1
* Creating format.conf file...
* Creating chunks directory...
* Creating buddymir directory...
* Creating target numeric ID file: /beegfs_data/data1/targetNumID
* Creating server numeric ID file: /beegfs_data/data1/nodeNumID
Updating config file: /etc/beegfs/beegfs-storage.conf
* Setting management host: 172.29.39.125
* Appending to target directory list in config file...
* WARNING: Skipping append. Target directory seems to be included in current targets list already. ('-f' disables this check.)
* Disabling usage of uninitialized storage targets in config file...
All done.
sacd01 $ /opt/beegfs/sbin/beegfs-setup-storage -p /beegfs_data/data2 -s 4 -i 402 -m 172.29.39.125
Preparing storage target directory: /beegfs_data/data2
* Creating format.conf file...
* Creating chunks directory...
* Creating buddymir directory...
* Creating target numeric ID file: /beegfs_data/data2/targetNumID
* Creating server numeric ID file: /beegfs_data/data2/nodeNumID
Updating config file: /etc/beegfs/beegfs-storage.conf
* Setting management host: 172.29.39.125
* Appending to target directory list in config file...
* WARNING: Skipping append. Target directory seems to be included in current targets list already. ('-f' disables this check.)
* Disabling usage of uninitialized storage targets in config file...
All done.
sacd02 $ rm /beegfs_data/data* -rf
sacd02 $ /opt/beegfs/sbin/beegfs-setup-storage -p /beegfs_data/data3 -s 5 -i 501 -m 172.29.39.125
Preparing storage target directory: /beegfs_data/data3
* Creating format.conf file...
* Creating chunks directory...
* Creating buddymir directory...
* Creating target numeric ID file: /beegfs_data/data3/targetNumID
* Creating server numeric ID file: /beegfs_data/data3/nodeNumID
Updating config file: /etc/beegfs/beegfs-storage.conf
* Setting management host: 172.29.39.125
* Appending to target directory list in config file...
* WARNING: Skipping append. Target directory seems to be included in current targets list already. ('-f' disables this check.)
* Disabling usage of uninitialized storage targets in config file...
All done.
sacd02 $ /opt/beegfs/sbin/beegfs-setup-storage -p /beegfs_data/data4 -s 5 -i 502 -m 172.29.39.125
Preparing storage target directory: /beegfs_data/data4
* Creating format.conf file...
* Creating chunks directory...
* Creating buddymir directory...
* Creating target numeric ID file: /beegfs_data/data4/targetNumID
* Creating server numeric ID file: /beegfs_data/data4/nodeNumID
Updating config file: /etc/beegfs/beegfs-storage.conf
* Setting management host: 172.29.39.125
* Appending to target directory list in config file...
* WARNING: Skipping append. Target directory seems to be included in current targets list already. ('-f' disables this check.)
* Disabling usage of uninitialized storage targets in config file...
All done.初始化客户端服务
sacd03 $ /opt/beegfs/sbin/beegfs-setup-client -m 172.29.39.125
Updating config file: /etc/beegfs/beegfs-client.conf
* Setting management host: 172.29.39.125
All done.
配置服务网口
sacd01 $ echo bond1.1039 > /etc/beegfs/network/beegfs-network-interface.conf
sacd02 $ echo bond1.1039 > /etc/beegfs/network/beegfs-network-interface.conf
sacd03 $ echo bond1.1039 > /etc/beegfs/network/beegfs-network-interface.conf
# Management
sacd01 $ vi /etc/beegfs/beegfs-mgmtd.conf
connInterfacesFile = /etc/beegfs/network/beegfs-network-interface.conf
# Metadata
sacd01 $ vi /etc/beegfs/beegfs-meta.conf
connInterfacesFile = /etc/beegfs/network/beegfs-network-interface.conf
sacd02 $ vi /etc/beegfs/beegfs-meta.conf
connInterfacesFile = /etc/beegfs/network/beegfs-network-interface.conf
# Storage
sacd01 $ vi /etc/beegfs/beegfs-storage.conf
connInterfacesFile = /etc/beegfs/network/beegfs-network-interface.conf
sacd02 $ vi vi /etc/beegfs/beegfs-storage.conf
connInterfacesFile = /etc/beegfs/network/beegfs-network-interface.conf
# Client
sacd03 $ vi /etc/beegfs/beegfs-client.conf
connInterfacesFile = /etc/beegfs/network/beegfs-network-interface.conf
启动服务（客户端除外）
sacd01 $ systemctl start beegfs-mgmtd
sacd01 $ systemctl start beegfs-meta
sacd02 $ systemctl start beegfs-meta
sacd01 $ systemctl start beegfs-storage
sacd02 $ systemctl start beegfs-storage
sacd03 $ beegfs-ctl --listtargets --nodetype=metadata --state
TargetID Reachability Consistency NodeID
======== ============ =========== ======
2 Online Good 2
3 Online Good 3
sacd03 $ beegfs-ctl --listtargets --nodetype=storage --state
TargetID Reachability Consistency NodeID
======== ============ =========== ======
401 Online Good 4
402 Online Good 4
501 Online Good 5
502 Online Good 5
配置元数据服务镜像
sacd03 $ beegfs-ctl --addmirrorgroup --automatic --nodetype=meta
New mirror groups:
BuddyGroupID Node type Node
============ ========= ====
1 primary 2 @ beegfs-meta sacd01 [ID: 2]
secondary 3 @ beegfs-meta sacd02 [ID: 3]
Mirror buddy group successfully set: groupID 1 -> target IDs 2, 3
sacd03 $ beegfs-ctl --listmirrorgroups --nodetype=meta
BuddyGroupID PrimaryNodeID SecondaryNodeID
============ ============= ===============
1 2 3
sacd03 $ beegfs-ctl --mirrormd
sacd01 $ systemctl restart beegfs-meta
sacd02 $ systemctl restart beegfs-meta
配置和启动客户端服务
sacd03 $ vi /etc/beegfs/beegfs-client.conf
logClientID = true
logLevel = 5
#sysMountSanityCheckMS = 11000
sysMountSanityCheckMS = 0
sacd03 $ systemctl start beegfs-helperd && systemctl start beegfs-client
sacd03 $ systemctl status beegfs-helperd && systemctl status beegfs-client
sacd01 $ systemctl enable beegfs-mgmtd beegfs-meta beegfs-storage
sacd02 $ systemctl enable beegfs-meta beegfs-storage
sacd03 $ systemctl enable beegfs-client
查看服务使用的网口
$ beegfs-ctl --listnodes --nodetype=management --nicdetails
sacd01 [ID: 1]
Ports: UDP: 8008; TCP: 8008
Interfaces:
+ bond1.1039[ip addr: 172.29.39.125; type: TCP]
Number of nodes: 1
$ beegfs-ctl --listnodes --nodetype=metadata --nicdetails
sacd01 [ID: 2]
Ports: UDP: 8005; TCP: 8005
Interfaces:
+ bond1.1039[ip addr: 172.29.39.125; type: TCP]
sacd02 [ID: 3]
Ports: UDP: 8005; TCP: 8005
Interfaces:
+ bond1.1039[ip addr: 172.29.39.126; type: TCP]
Number of nodes: 2
Root: 2
$ beegfs-ctl --listnodes --nodetype=storage --nicdetails
sacd01 [ID: 4]
Ports: UDP: 8003; TCP: 8003
Interfaces:
+ bond1.1039[ip addr: 172.29.39.125; type: TCP]
sacd02 [ID: 5]
Ports: UDP: 8003; TCP: 8003
Interfaces:
+ bond1.1039[ip addr: 172.29.39.126; type: TCP]
Number of nodes: 2
$ beegfs-ctl --listnodes --nodetype=client --nicdetails
C26A-5CD18762-sacd03 [ID: 5]
Ports: UDP: 8004; TCP: 0
Interfaces:
+ bond1.1039[ip addr: 172.29.39.133; type: TCP]
Number of nodes: 1
解决内核版本问题
启动客户端服务时，可能出现如下错误：
$ systemctl start beegfs-client
Job for beegfs-client.service failed because the control process exited with error code. See "systemctl status beegfs-client.service" and "journalctl -xe" for details.
$ systemctl status beegfs-client.service -l
● beegfs-client.service - Start BeeGFS Client
Loaded: loaded (/usr/lib/systemd/system/beegfs-client.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Tue 2019-05-07 16:37:01 CST; 17s ago
Process: 134148 ExecStart=/etc/init.d/beegfs-client start (code=exited, status=1/FAILURE)
Main PID: 134148 (code=exited, status=1/FAILURE)
May 07 16:37:00 sacd03 systemd[1]: Starting Start BeeGFS Client...
May 07 16:37:00 sacd03 beegfs-client[134148]: Starting BeeGFS Client:
May 07 16:37:00 sacd03 beegfs-client[134148]: - Loading BeeGFS modules
May 07 16:37:00 sacd03 beegfs-client[134148]: - BeeGFS module autobuild
May 07 16:37:01 sacd03 beegfs-client[134148]: Makefile:117: *** Linux kernel build directory not found. Please check if the kernel module development packages are installed for the current kernel version. (RHEL: kernel-devel; SLES: linux-kernel-headers, kernel-source; Debian: linux-headers). Stop.
May 07 16:37:01 sacd03 beegfs-client[134148]: make: *** [auto_rebuild] Error 2
May 07 16:37:01 sacd03 systemd[1]: beegfs-client.service: main process exited, code=exited, status=1/FAILURE
May 07 16:37:01 sacd03 systemd[1]: Failed to start Start BeeGFS Client.
May 07 16:37:01 sacd03 systemd[1]: Unit beegfs-client.service entered failed state.
May 07 16:37:01 sacd03 systemd[1]: beegfs-client.service failed.
这是由于没有安装kernel-devel，没有内核头文件引起的。
或者是下面这个错误：
$ vi /var/log/beegfs-client.log
Message from syslogd@sacd03 at May 8 09:44:12 ...
dracut:dracut: creation of /boot/initramfs-3.10.0-957.12.1.el7.x86_64.tmp failed
(5) May08 09:44:33 *df(63614) [C206-5CD22EA8-sacd03: FhgfsOps_getattr] >> called. Path: /; EntryID: root
(0) May08 09:44:33 *df(63614) [C206-5CD22EA8-sacd03: Messaging (RPC node)] >> Invalid mirror buddy group ID: 0
(5) May08 09:44:33 *df(63614) [C206-5CD22EA8-sacd03: FhgfsOps_statfs] >> called. Path: n/a (no dentry); EntryID: n/a (no inode)
(1) May08 09:44:33 *df(63614) [C206-5CD22EA8-sacd03: Remoting (stat storage targets)] >> No storage targets known.
(5) May08 09:45:04 *df(65354) [C206-5CD22EA8-sacd03: FhgfsOps_getattr] >> called. Path: /; EntryID: root
(0) May08 09:45:04 *df(65354) [C206-5CD22EA8-sacd03: Messaging (RPC node)] >> Invalid mirror buddy group ID: 0
(5) May08 09:45:04 *df(65354) [C206-5CD22EA8-sacd03: FhgfsOps_statfs] >> called. Path: n/a (no dentry); EntryID: n/a (no inode)
(1) May08 09:45:04 *df(65354) [C206-5CD22EA8-sacd03: Remoting (stat storage targets)] >> No storage targets known.
$ ls /mnt/beegfs/
ls: reading directory /mnt/beegfs/: Not a directory
这是BeeGFS 6.18在RHEL 3.10.0-862版本内核上的一个Bug引起的：
在所有节点上安装kernel-3.10.0-693版本的内核和开发包即可解决问题：
$ yum list kernel
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
* base: ap.stykers.moe
* epel: mirrors.tuna.tsinghua.edu.cn
* extras: ap.stykers.moe
* updates: mirrors.tuna.tsinghua.edu.cn
Installed Packages
kernel.x86_64 3.10.0-957.5.1.el7 @updates
Available Packages
kernel.x86_64 3.10.0-957.12.1.el7 updates
$ wget ftp://ftp.pbone.net/mirror/ftp.scientificlinux.org/linux/scientific/7.0/x86_64/updates/security/kernel-3.10.0-693.el7.x86_64.rpm
$ wget ftp://ftp.pbone.net/mirror/ftp.scientificlinux.org/linux/scientific/7.0/x86_64/updates/security/kernel-devel-3.10.0-693.el7.x86_64.rpm
# wget ftp://ftp.pbone.net/mirror/ftp.redhat.com/pub/redhat/linux/enterprise/7Server/en/RHEV/SRPMS/kernel-3.10.0-693.el7.src.rpm
$ rpm -ivh kernel* --force
$ grep "menuentry " /boot/grub2/grub.cfg | awk -F"'" '{print $2}'
CentOS Linux (3.10.0-693.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-957.12.1.el7.x86_64) 7 (Core)
CentOS Linux (0-rescue-e3a40c9c3add41e3b2eb72e341b52ec1) 7 (Core)
$ vi /etc/default/grub
GRUB_DEFAULT="CentOS Linux (3.10.0-693.el7.x86_64) 7 (Core)"
$ grub2-mkconfig -o /boot/grub2/grub.cfg
$ reboot
$ rpm -evh kernel-3.10.0-957.12.1.el7.x86_64
$ grep "menuentry " /boot/grub2/grub.cfg | awk -F"'" '{print $2}'
CentOS Linux (3.10.0-693.el7.x86_64) 7 (Core)
CentOS Linux (0-rescue-e3a40c9c3add41e3b2eb72e341b52ec1) 7 (Core)关于如何搭建BeeGFS开发环境就分享到这里了，希望以上内容可以对大家有一定的帮助，可以学到更多知识。如果觉得文章不错，可以把它分享出去让更多的人看到。