CentOS CDH 部署指南

CDH（Cloudera's Distribution, including Apache Hadoop）是 Cloudera 公司维护的基于 Apache Hadoop 的发行版，集成了众多补丁和优化，可以直接用于生产环境，Cloudera Manager (CM) 则是管理和监控 CDH 集群的 Web 界面工具，提供了便捷的安装、配置和管理功能，以下是在 CentOS 系统上离线部署 CDH 集群的详细步骤。

准备工作

1、硬件准备：使用 VMware 模拟多台主机，这里以三台机器为例。

2、修改主机名：

hostnamectl sethostname cdh1
hostnamectl sethostname cdh2
hostnamectl sethostname cdh3

3、关闭防火墙：

systemctl stop firewalld
systemctl disable firewalld

4、关闭 SELinux：

setenforce 0  # 临时关闭
vi /etc/selinux/config  # 永久关闭
SELINUX=disabled

5、配置 IP 到主机的映射：

vi /etc/hosts
192、168.100.10 cdh1
192、168.100.20 cdh2
192、168.100.30 cdh3

6、配置免密码登录（在 cdh1 上执行）：

sshkeygen t rsa
sshcopyid cdh1
sshcopyid cdh2
sshcopyid cdh3

7、设置用户最大可打开文件数，进程数，内存占用：

vi /etc/security/limits.conf
soft nofile 32728
hard nofile 1024999
soft nproc 65535
hard noroc unlimited
soft memlock unlimited
hard memlock unlimited
sysctl p

8、设置 swap 空间：

echo "vm.swappiness = 0" >> /etc/sysctl.conf

9、关闭大页面压缩：

echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag

CDH 和 CM 的下载与安装

1、下载 CDH 和 CM：

CDH 下载地址: [HTTPS://ARChive.cloudera.com/cdh6/6.3.2/parcels/](https://archive.cloudera.com/cdh6/6.3.2/parcels/)

CM 下载地址: [https://archive.cloudera.com/cm6/6.3.1/redhat7/yum/RPMS/x86_64/](https://archive.cloudera.com/cm6/6.3.1/redhat7/yum/RPMS/x86_64/)

2、上传下载好的 CDH 和 CM 包到 cdh1：

使用sftp 或其它文件传输工具将下载的包上传到目标服务器。

3、配置本地 YUM 源（在 cdh1 上执行）：

挂载 CentOS 镜像并配置本地 YUM 源：

mount /dev/cdrom /mnt/
rm rf /etc/yum.repos.d/*
cat >> /etc/yum.repos.d/local.repo << EOF
[centos]
name=CentOS7
baseurl=file:///mnt
EOF

4、配置 Cloudera Manager 仓库（在所有节点上执行）：

rpm import https://archive.cloudera.com/cdh6/6.3.2/RPMGPGKEYcloudera
cat >> /etc/yum.repos.d/clouderamanager.repo << EOF
[clouderamanager]
name=Cloudera Manager
baseurl=https://archive.cloudera.com/cm6/6.3.1/redhat7/yum/RPMS/x86_64/
gpgcheck=1
repo_gpgcheck=1
EOF

5、安装 Cloudera Manager Server 和 Agent（在 cdh1 上执行）：

yum install y clouderamanagerserver clouderamanageragent clouderamanagerdaemons

6、初始化 Cloudera Manager：

clouderamanager express force

CDH 安装与配置

1、安装 CDH（在 cdh1 上执行）：

yum install y clouderamanageragent
yum install y ambariserver ambariagent ambarilogsearch ambarimetrics ambariviews ambariinfra ambarialerts ambarihdfs ambariresourcemanagerplugin ambariyarnplugin ambarigangliamonitoringplugin ambarihdfs2plugin ambarihdfs3plugin ambarihdfs4plugin ambariyarn2plugin ambariyarn3plugin ambariyarn4plugin ambariyarn5plugin ambariyarn6plugin ambariyarn7plugin ambariyarn8plugin ambariyarn9plugin ambariyarn10plugin ambariyarn11plugin ambariyarn12plugin ambariyarn13plugin ambariyarn14plugin ambariyarn15plugin ambariyarn16plugin ambariyarn17plugin ambariyarn18plugin ambariyarn19plugin ambariyarn20plugin ambariyarn21plugin ambariyarn22plugin ambariyarn23plugin ambariyarn24plugin ambariyarn25plugin ambariyarn26plugin ambariyarn27plugin ambariyarn28plugin ambariyarn29plugin ambariyarn30plugin ambariyarn31plugin ambariyarn32plugin ambariyarn33plugin ambariyarn34plugin ambariyarn35plugin ambariyarn36plugin ambariyarn37plugin ambariyarn38plugin ambariyarn39plugin ambariyarn40plugin ambariyarn41plugin ambariyarn42plugin ambariyarn43plugin ambariyarn44plugin ambariyarn45plugin ambariyarn46plugin ambariyarn47plugin ambariyarn48plugin ambariyarn49plugin ambariyarn50plugin ambariyarn51plugin ambariyarn52plugin ambariyarn53plugin ambariyarn54plugin ambariyarn55plugin ambariyarn56plugin ambariyarn57plugin ambariyarn58plugin ambariyarn59plugin ambariyarn60plugin ambariyarn61plugin ambariyarn62plugin ambariyarn63plugin ambariyarn64plugin ambariyarn65plugin ambariyarn66plugin ambariyarn67paramplugin ambariyarn68plugin ambariyarn69plugin ambariyarn70plugin ambariyarn71plugin ambariyarn72plugin ambariyarn73plugin ambariyarn74plugin ambariyarn75plugin ambariyarn76plugin ambariyarn77plugin ambariyarn78plugin ambariyarn79plugin ambariyarn80plugin ambariyarn81plugin ambariyarn82plugin ambariyarn83plugin ambariyarn84plugin ambariyarn85plugin ambariyarn86plugin ambariyarn87plugin ambariyarn88plugin ambariyarn89plugin ambariyarn90plugin ambariyarn91plugin ambariyarn92plugin ambariyarn93plugin ambariyarn94plugin ambariyarn95plugin ambariyarn96plugin ambariyarn97plugin ambariyarn98插件 ambariyarn99plugin ambariyarn100plugin

2、启动 Cloudera Manager Server：

systemctl start clouderamanagerserver
systemctl enable clouderamanagerserver

3、通过浏览器访问 Cloudera Manager：http://<your_server_ip>:7180，完成后续的集群配置和管理操作。

FAQs

问题1：如何检查 CDH 和 Cloudera Manager 的版本是否兼容？

答：确保下载的 CDH 和 Cloudera Manager 版本号一致，CDH 6.3.2 对应 CM 6.3.1，具体兼容性信息可以参考 Cloudera 官方文档。

问题2：如何进行 CDH 集群的升级？

答：可以通过 Cloudera Manager 的升级向导进行 CDH 集群的升级，首先在 Cloudera Manager 中选择需要升级的集群，然后点击“Upgrade”按钮，按照向导提示完成升级操作，需要注意的是，升级过程中可能会涉及到服务重启和数据迁移，建议提前做好备份。