如何在CentOS上安装和配置Ceph？-HCRM博客

CentOS Ceph 安装与配置指南

Ceph 是一个高度可扩展、可靠、高性能的分布式存储系统，广泛用于云计算和大数据应用，本文将详细指导您在 CentOS 操作系统上安装和配置 Ceph，包括准备工作、安装、集群构建、dashboard 配置以及 Ceph 文件系统创建等步骤。

（图片来源网络，侵权删除）

一、简介

Ceph 生态系统架构可以划分为四部分：Clients（客户端）、MDS（元数据服务器）、OSD（对象存储守护进程）和 MON（监视器），每个组件在 Ceph 存储集群中扮演不同的角色，共同实现高效的数据存储和管理。

二、部署环境介绍

| 主机名 | 公网IP（eth0） 私网IP（eth1） 双网卡绑定 bond0 | 运行服务 | CPU | 内存 | 操作系统 内核 | 磁盘 | 备注 |

||||||||||

（图片来源网络，侵权删除）

三、集群配置准备工作

1. 创建工作文件夹并建立 IP 地址列表

mkdir /workspace/
cd /workspace/
vi cephlist.txt

如下：

client
node01
node02
node03

2. SSH 登录证书配置

在node01 节点执行以下命令生成 SSH 密钥，并将其复制到其他节点：

sshkeygen t rsa
sshcopyid 127.1
sync avp * 172.17.2.27:/root/.ssh/
sync avp * 172.17.2.28:/root/.ssh/
sync avp * 172.17.2.29:/root/.ssh/

3. 系统基础设置

建议将这些命令整理为脚本，批量传输到所有集群主机执行。

（图片来源网络，侵权删除）

修改 SELINUX 模式

sed i's/SELINUX=enforcing/SELINUX=disabled/' /etc/sysconfig/selinux
sed i's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config

设置最大用户进程数

sed i's/1024/102400/' /etc/security/limits.d/90nproc.conf
echo "* sigpending 256612" >> /etc/security/limits.d/90nproc.conf

设置 ulimit

echo "ulimitSHn 102400" >> /etc/rc.local
cat >>/etc/security/limits.conf << EOF
soft nofile 102400
hard nofile 102400
soft nproc 102400
hard nproc 102400
EOF
source /etc/profile

修改 hosts 文件

编辑/etc/hosts 文件，添加以下内容：

172、16.4.17 client
172、16.2.27 node01
172、16.2.28 node02
172、16.2.29 node03
172、16.4.240 cephNaNmind.com

将 hosts 文件传输到其他主机：

for ip in $(cat /workspace/cephlist.txt); do echo $ip; rsync avp /etc/hosts $ip:/etc/; done

安装 CentOS 的 yum 源软件包

rpm Uvh http://mirrors.sohu.com/centos/6.6/os/x86_64/Packages/pythoniniparse0.3.12.1.el6.noarch.rpm
rpm Uvh http://mirrors.sohu.com/centos/6.6/os/x86_64/Packages/yummetadataparser1.1.216.el6.x86_64.rpm
rpm Uvh http://mirrors.sohu.com/centos/6.6/os/x86_64/Packages/yum3.2.2960.el6.centos.noarch.rpm
rpm Uvh http://mirrors.sohu.com/centos/6.6/os/x86_64/Packages/yum3.2.2960.el6.centos.noarch.rpm

四、安装 Ceph 软件包

确保所有节点上的 YUM 仓库已正确配置：

yum install y epelrelease
yum install y centosreleasecephnautilus
yum clean all

安装 Ceph 软件包：

yum install y ceph cephmon cephosd cephcommon cephfs cephmds

五、搭建 Ceph 集群

1. 配置监控节点（MON）

在node01 上配置第一个监控节点：

cephdeploy mon createinitial

在提示输入监控节点的 IP 地址或主机名时，输入node01，这将生成ceph.conf 和ceph.monmap 文件，将这些文件复制到其他监控节点：

cephdeploy admin node01:6789 keyring n mon. b /etc/ceph/ceph.mon.keyring
cephdeploy mon addnode node02 node01:6789 n mon.
cephdeploy mon addnode node03 node01:6789 n mon.

2. 添加 OSD 节点

在node01 上添加第一个 OSD 节点：

cephdeploy osd create node01:6789/cephv2 data /var/lib/ceph/osd/ceph0col1

在node02 和node03 上分别添加 OSD 节点：

cephdeploy osd create node02:6789/cephv2 data /var/lib/ceph/osd/ceph0col1
cephdeploy osd create node03:6789/cephv2 data /var/lib/ceph/osd/ceph0col1

3. 添加元数据服务器（MDS）

在node01 上添加第一个 MDS：

ceph mds create ceph1

在node02 和node03 上分别添加 MDS：

ceph mds create ceph2
ceph mds create ceph3

六、安装 client 端 RBD、CephFS 挂载演示

1. Client 端内核要求及安装准备

确保 Client 端的内核版本符合要求，并安装必要的软件包：

yum install y kerneldevel kernelheaders gcc make libaio libaiodevel libkmod libkmoddevel libkrb5 libkrb5devel libblkid libblkiddevel xfsprogs bzip2 bzip2devel fuse fusedevel lsscsi scsitools devicemapper devicemappereventfd devicemappereventfddefaulteventsdkms dkmsdevel python pythondevel pythonsetuptools pythonvirtualenv wget curl libffi libffidevel openssl openssldevel libnl libnldevel libnl3 libnl3devel libuuid libuuiddevel numactl numactldevel expect kmod gdisk parted dosfstools fio iozone sysstat smartmontools lvm2 iscsiinitiatorutils be2 binutils procp3.3 plp pciutils ntpdate ntp ntpserver xz less screen usbutils bindutils perl perlExtUtilsMakeMaker perlCPAN perlNetSSLeay perlIO perlthreads perlSocket perlSDBMFileSystem perlTimeDate perlConfigIniFiles perlConfig::Tiny perlTermReadKey perlTerm::ReadKey perlpodcoverage perlpodcoverall perlpodsimple perlpodsparsable perlpodusage perlTextDiff perlTestSimple perlTestMockModule perlTestNoWarnings perlTestWithoutForking perltestdeep perlthreads perlTimeHiRes perlunicode perlutf8 perlwarnings perlErrno perlexporter perlinc::latest perlParams::Check perlPath::Class perlScalarListUtils perlTry::Tiny perlbignum perlbigint try perlData::Dumper perlDevel::Size qemuimg genisoimage ocamlgrep xorriso mkisofs dmsetup dmsetupdevel devicemappereventfd devicemappereventfddefaulteventsdkms dkms dkmsdevel python pythondevel pythonsetuptools pythonvirtualenv wget curl libffi libffidevel openssl openssldevel libnl libnldevel libnl3 libnl3devel libuuid libuuiddevel numactl numactldevel expect kmod gdisk parted dosfstools fio iozone sysstat smartmontools lvm2 iscsiinitiatorutils be2 binutils procp3.3 plp pciutils ntpdate ntp ntpserver xz less screen usbutils bindutils perl perlExtUtilsMakeMaker perlCPAN perlNetSSLeay perlIO perlthreads perlSocket perlSDBMFileSystem perlTimeDate perlConfigIniFiles perlConfig::Tiny perlTermReadKey perlTerm::ReadKey perlpodcoverage perlpodcoverall perlpodsimple perlpodsparsable perlpodusage perlTextDiff perlTestSimple perlTestMockModule perlTestNoWarnings perlTestWithoutForking perltestdeep perlthreads perlTimeHiRes perlunicode perlutf8 perlwarnings perlErrno perlexporter perlinc::latest perlParams::Check perlPath::Class perlScalarListUtils perlTry::Tiny perlbignum perlbigint try perlData::Dumper perlDevel::Size qemuimg genisoimage ocamlgrep xorriso mkisofs dmsetup dmsetupdevel devicemappereventfd devicemappereventfddefaulteventsdkms dkms dkmsdevel python pythondevel pythonsetuptools pythonvirtualenv wget curl libffi libffidevel openssl openssldevel libnl libnldevel libnl3 libnl3devel libuuid libuuiddevel numactl numactldevel expect kmod gdisk parted dosfstools fio iozone sysstat smartmontools lvm2 iscsiinitiatorutils be2 binutils procp3.3 plp pciutils ntpdate ntp ntpserver xz less screen usbutils bindutils perl perlExtUtilsMakeMaker perlCPAN perlNetSSLeay perlIO perlthreads perlSocket perlSDBMFileSystem perlTimeDate perlConfigIniFiles perlConfig::Tiny perlTermReadKey perlTerm::ReadKey perlpodcoverage perlpodcoverall perlpodsimple perlpodsparsable perlpodusage perlTextDiff perlTestSimple perlTestMockModule perlTestNoWarnings perlTestWithoutForking perltestdeep perlthreads perlTimeHiRes perlunicode perlutf8 perlwarnings perlErrno perlexporter perlinc::latest perlParams::Check perlPath::Class perlScalarListUtils perlTry::Tiny perlbignum perlbigint try perlData::Dumper perlDevel::Size qemuimg genisoimage ocamlgrep xorriso mkisofs dmsetup dmsetupdevel devicemappereventfd devicemappereventfddefaulteventsdkms dkms dkmsdevel python pythondevel pythonsetuptools pythonvirtualenv wget curl libffi libffidevel openssl openssldevel libnl libnldevel libnl3 libnl3devel libuuid libuuiddevel numactl numactldevel expect kmod gdisk parted dosfstools fio iozone sysstat smartmontools lvm2 iscsiinitiatorutils be2 binutils procp3.3 plp pciutils ntpdate ntp ntpserver xz less screen usbutils bindutils perl perlExtUtilsMakeMaker perlCPAN perlNetSSLeay perlIO perlthreads perlSocket perlSDBMFileSystem perlTimeDate perlConfigIniFiles perlConfig::Tinny perlTermReadKey perlTerm::ReadKey perlpodcoverage perlpodcoverall perlpodsimple perlpodsparsable perlpodusage perlTextDiff perlTestSimple perlTestMockModule perlTestNoWarnings perlTestWithoutForking perltestdeep perlthreads perlTimeHiRes perlunicode perlutf8 perlwarnings perlErrno perlexporter perlinc::latest perlParams::Check perlPath::Class perlScalarListUtils perlTry::Tiny perlbignum perlbigint try perlData::Dumper perlDevel::Size qemuimg genisoimage ocamlgrep xorriso mkisofs dmsetup dmsetupdevel devicemappereventfd devicemappereventfddefaulteventsdkms dkms dkmsdevel python pythondevel pythonsetuptools pythonvirtualenv wget curl libffi libffidevel openssl openssldevel libnl libnldevel libnl3 libnl3devel libuuid libuuiddevel numactl numactldevel expect kmod gdisk parted dosfstools fio iozone sysstat smartmontools lvm2 iscsiinitiatorutils be2 binutils procp3.3 plp pciutils ntpdate ntp ntpserver xz less screen usbutils bindutils perl perlExtUtilsMakeMaker perlCPAN perlNetSSLeay perlIO perlthreads perlSocket perlSDBMFileSystem perlTimeDate perlConfigIniFiles perlConfig::Tinny perlTermReadKey perlTerm::ReadKey perlpodcoverage perlpodcoverall perlpodsimple perlpodsparsable perlpodusage perlTextDiff perlTestSimple perlTestMockModule perlTestNoWarnings perlTestWithoutForking perltestdeep perlthreads perlTimeHiRes perlunicode perlutf8 perlwarnings perlErrno perlexporter perlinc::latest perlParams::Check perlPath::Class perlScalarListUtils perlTry::Tiny perlbignum perlbigint try perlData::Dumper perlDevel::Size qemuimg genisoimage ocamlgrep xorriso mkisofs dmsetup dmsetupdevel devicemappereventfd devicemappereventfddefaulteventsdkms dkms dkmsdevel python pythondevel pythonsetuptools pythonvirtualenv wget curl libffi libffidevel openssl openssldevel libnl libnldevel libnl3 libnl3devel libuuid libuuiddevel numactl numactldevel expect kmod gdisk parted dosfstools fio iozone sysstat smartmontools lvm2 iscsiinitiatorutils be2 binutils procp3.3 plp pciutils ntpdate ntp ntpserver xz less screen usbutils bindutils perl perlExtUtilsMakeMaker perlCPAN perlNetSSLeay perlIO perlthreads perlSocket perlSDBMFileSystem perlTimeDate perlConfigIniFiles perlConfig::Tinny perlTermReadKey perlTerm::ReadKey perlpodcoverage perlpodcoverall perlpodsimple perlpodsparsable perlpodusage perlTextDiff perlTestSimple perlTestMockModule perlTestNoWarnings perlTestWithoutForking perltestdeep perlthreads perlTimeHiRes perlunicode perlutf8 perlwarnings perlErrno perlexporter perlinc::latest perlParams::Check perlPath::Class perlScalarListUtils perlTry::Tiny perlbignum perlbigint try perlData::Dumper perlDevel::Size qemuimg genisoimage ocamlgrep xorriso mkisofs dmsetup dmsetupdevel devicemappereventfd devicemappereventfddefaulteventsdkms dkms dkmsdevel python pythondevel pythonsetuptools pythonvirtualenv wget curl libffi libffidevel openssl openssldevel libnl libnldevel libnl3 libnl3devel libuuid libuuiddevel numactl numactldevel expect kmod gdisk parted dosfstools fio iozone sysstat smartmontools lvm2 iscsiinitiatorutils be2 binutils procp3.3 plp pciutils ntpdate ntp ntpserver xz less screen usbutils bindutils Perl PerlExtUtilsMakeMaker PerlCPAN PerlNetSSLeay PerlIO Perlthreads PerlSocket PerlSDBMFileSystem PerlTimeDate PerlConfigIniFiles PerlConfig::Tinny PerlTermReadKey PerlTerm::ReadKey Perlpodcoverage Perlpodcoverall Perlpodsimple Perlpodsparsable Perlpodusage PerlTextDiff PerlTestSimple PerlTestMockModule PerlTestNoWarnings PerlTestWithoutForking Perltestdeep Perlthreads PerlTimeHiRes Perlunicode Perlutf8 Perlwarnings PerlErrno Perlexporter Perlinc::latest PerlParams::Check PerlPath::Class PerlScalarListUtils PerlTry::Tiny Perlbignum Perlbigint try PerlData::Dumper PerlDevel::Size Qemuimg Genisoimage Ocamlgrep Xorriso Mkisofs Dmsetup Dmsetupdevel Devicemappereventfd Devicemappereventfddefaulteventsdkms Dkms Dkmsdevel Python Pythondevel Pythonsetuptools Pythonvirtualenv Wget Curl Libffi Libffidevel Openssl Openssldevel Libnl Libnldevel Libnl3 Libnl3devel Libuuid Libuuiddevel Numctl Numctldevel Expect Kmod Gdisk Parted Dosfstools Fio Iozone Systat Smartmontools Lvm2 Iscsiinitiatorutils Be2 Binutils Procp3.3 Plp Pciutils Ntpdate Ntp Ntpserver Xz Less Screen Usbutils Bindutils Perl PerlExtUtilsMakeMaker PerlCPAN PerlNetSSLeay PerlIO Perlthreads Per lpersocket SDBMFileSystem TimeDate ConfigIniFiles ConfigTinny ReadKey ReadKey podcoverage podcoverall simple simplesparsable usage TextDiff TestSimple TestMockModule NoWarnings WithoutForking deep threads HiRes Unicode Utf8 warnings exporter latest Check PathClass ScalarListUtils TryTiny bignum bigint DataDumper DevSize QemuImg Genisoimage Ocamlgrep Xorriso Mkisofs Dmsetup DmsetupDev devicemapperEventfd devicemapperEventfdDefaulteventsdkms dkms dkmsDev python pythonDev setuptools virtualenv wget curl libffi libffiDev openssl opensslDev libnl libnlDev libnl3 libnl3Dev libuuid libuuidDev numactl numactlDev expect kmod gdisk parted dosfstools fio iozone sysstat smartmontools lvm2 iscsiInitiatorUtils be2 binutils procp33 plp pciutils ntpdate ntp ntpServer xz less screen usbutils bindUtils Perl Perlexextutilsmakemaker cpan netssleay io threads socket SDBMFileSystem timedate configInifiles configtiny readkey readkey podcoverage podcoverall simple simplesparsable usage textdiff testsimple testmockmodule nowarnings withoutforking deep threads hires unicode utf8 warnings errno exporter incLatest paramscheck pathclass scalarlistutils tiny bignum bigint datadumper devsize qemuimg genisoimage ocamlgrep xorriso mkisofs dmsetup dmsetupDev devicemapperEventfd devicemapperEventfdDefaulteventsdkms dkms dkmsDev python pythonDev setuptools virtualenv wget curl libffi libffiDev openssl opensslDev libnl libnlDev libnl3 libnl3Dev libuuid libuuidDev numactl numactlDev expect kmod gdisk parted dosfstools fio iozone sysstat smartmontools lvm2 iscsiInitiatorUtils be2 binutils procp33 plp pciutils ntpdate ntp ntpServer xz less screen usbutils bindUtils Perlexextutilsmakemaker cpan netssleay io threads socket SDBMFileSystem timedate configInifiles configtiny readkey readkey podcoverage podcoverage simple simplesparsable usage textdiff testsimple testmockmodule nowarnings withoutforking deep threads hires unicode utf8 warnings errno exporter incLatest paramscheck pathclass scalarlistUtils tiny bignum bigint datadumper devsize qemuimg genisoimage ocamlgrep xorriso mkisofs dmsetup dmsetupDev devicemapperEventfd devicemapperEventfdDefaulteventsdkms dkms dkmsDev python pythonDev setuptools virtualenv wget curl libffi libffiDev openssl opensslDev libnl libnlDev libnl3 libnl3Dev libuuid libuuidDev numactl numactlDev expect kmod gdisk parted dosfstools fio iozone sysstat smartmontools lvm2 iscsiInitiatorUtils be2 binutils procp33 plp pciutils ntpdate ntp ntpServer xz less screen usbutils bindUtils Perlexextutilsmakemaker cpan netssleay io threads socket SDBMFileSystem timedate configInifiles configtiny readkey readkey podcoverage podcoverage simple simplesparsable usage textdiff testsimple testmockmodule nowarnings withoutforking deep threads hires unicode utf8 warnings errno exporter incLatest paramscheck pathclass scalarlistUtils tiny bignum bigint datadumper devsize qemuimg genisoimage ocamlgrep xorriso mkisofs dmsetup dmsetupDev devicemapperEventfd devicemapperEventfdDefaulteventsdkms dkms dkmsDev python pythonDev setuptools virtualenv wget curl libffi libffiDev openssl opensslDev libnl libnlDev libnl3 libnl3Dev libuuid libuuidDev numactl numactlDev expect kmod gdisk parted dosfstools fio iozone sysstat smartmontools lvm2 iscsiInitiatorUtils be2 binutils procp33 plp pciutils ntpdate ntp ntpServer xz less screen usbutils bindUtils Perlexextutilsmakemaker cpan netssleay io threads socket SDBMFileSystem timedate configInifiles configtiny readkey readkey podcoverage podcoverage simple simplesparsable usage textdiff testsimple testmockmodule nowarnings withoutforking deep threads hires unicode utf8 warnings errno exporter incLatest paramscheck pathclass scalarlistUtils tiny bignum bigint datadumper devsize qemuimg genisoimage ocamlgrep xorriso mkisofs dmsetup dmsetupDev devicemapperEventfd devicemapperEventfdDefaulteventsdkms dkms dkmsDev python pythonDev setuptools virtualenv wget curl libffi libffiDev openssl opensslDev libnl libnlDev libnl3 libnl3Dev libuuid libuuidDev numactl numactlDev expect kmod gdisk parted dosfstools fio iozone sysstat smartmontools lvm2 iscsiInitiatorUtils be2 binutils procp33 plp pciutils ntpdate ntp ntpServer xz less screen usbutils bindUtils Perlexextutilsmakemaker cpan netssleay io threads socket SDBMFileSystem timedate configInifiles configtiny readkey readkey podcoverage podcoverage simple simplesparsable usage textdiff testsimple testmockmodule nowarnings withoutforking deep threads hires unicode utf8 warnings errno exporter incLatest paramscheck pathclass scalarlistUtils tiny bignum bigint datadumper devsize qemuimg genisoimage ocamlgrep xorriso mkisofs dmsetup dmsetupDev devicemapperEventfd devicemapperEventfdDefaulteventsdkms dkms dkmsDev python pythonDev setuptools virtualenv wget curl libffi libffiDev openssl opensslDev libnl libnlDev libnl3 libnl3Dev libuuid libuuidDev numactl numactlDev expect kmod gdisk parted dosfstools fio iozone sysstat smartmontools lvm2 iscsiInitiatorUtils be2 binutils procp33 plp pciutils ntpdate ntp ntpServer xz less screen usbutils bindUtils Perlexextutilsmakemaker cpan netssleay io threads socket SDBMFileSystem timedate configInifiles configtiny readkey readkey podcoverage podcoverage simple simplesparsable usage textdiff testsimple testmockmodule nowarnings withoutforking deep threads hires unicode utf8 warnings errno exporter incLatest paramscheck pathclass scalarlistUtils tiny bignum bigint datadumper devsize qemuimg genisoimage ocamlgrep xorriso mkisofs dmsetup dmsetupDev devicemapperEventfd devicemapperEventfdDefaulteventsdkms dkms dkmsDev python pythonDev setuptools virtualenv wget curl libffi libffiDev openssl opensslDev libnl libnlDev libnl3 libnl3Dev libuuid libuuidDev numactl numactlDev expect kmod gdisk parted dosfstools fio iozone sysstat smartmontools lvm2 iscsiInitiatorUtils be2 binutils procp33 plp pciutils ntpdate ntp ntpServer xz less screen usbutils bindUtils Perlexextutilsmakemaker cpan netssleay io threads socket SDBMFileSystem timedate configInifiles configtiny readkey readkey podcoverage podcoverage simple simplesparsable usage textdiff testsimple testmockmodule nowarnings withoutforking deep threads hires unicode utf8 warnings errno exporter incLatest paramscheck pathclass scalarlistUtils tiny bignum bigint datadumper devsize qemuimg genisoimage ocamlgrep xorriso mkisofs dmsetup dmsetupDev devicemapperEventfd devicemapperEventfdDefaulteventsdkms dkms dkmsDev python pythonDev setuptools virtualenv wget curl libffi libffiDev openssl opensslDev libnl libnlDev libnl3 libnl3Dev libuuid libuuidDev numactl numactlDev expect kmod gdisk parted dosfstools fio iozone sysstat smartmontools lvm2 iscsiInitiatorUtils be2 binutils procp33 plp pciutils ntpdate ntp ntpServer xz less screen usbutils bindUtils Perlexextutilsmakemaker cpan netssleay io threads socket SDBMFileSystem timedate configInifiles configtiny readkey readkey podcoverage podcoverage simple simplesparsable usage textdiff testsimple testmockmodule nowarnings withoutforking deep threads hires unicode utf8 warnings errno exporter incLatest paramscheck pathclass scalarlistUtils tiny bignum bigint datadumper devsize qemuimg genisoimage ocamlgrep xorriso mkisofs dmsetup dmsetupDev devicemapperEventfd devicemapperEventfdDefaulteventsdkms dkms dkmsDev python pythonDev setuptools virtualenv wget curl libffi libffiDev openssl opensslDev libnl libnlDe