Hadoop环境部署文档

一、部署环境

本机IP：192.168.0.222
系统：CentOS Linux release 7.6.1810 (Core)
内核：3.10.0-957.el7.x86_64

二、安装docker-ce

yum install -y yum-utils device-mapper-persistent-data lvm2 && yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo && yum install -y docker-ce docker-ce-cli containerd.io

三、配置docker仓库

mkdir /etc/docker/ && vim /etc/docker/daemon.json 

加入如下内容

{
  "registry-mirrors": ["https://1e7waog4.mirror.aliyuncs.com"],
  "insecure-registries":["192.168.0.20:5000"]
}

启动并设置开机自启

systemctl start docker && systemctl enable docker

四、配置docker网络

docker network create --subnet=172.10.0.0/16 hadoop_net && docker network ls

五、下载镜像

docker pull 192.168.0.20:5000/hadoop-hbase:master && docker pull 192.168.0.20:5000/hadoop-hbase:slave1 && docker pull 192.168.0.20:5000/hadoop-hbase:slave2 && docker pull 192.168.0.20:5000/hadoop-hbase:mysql

六、启动容器

启动master

docker run -d \
--add-host master:172.10.0.2 \
--add-host slave1:172.10.0.3 \
--add-host slave2:172.10.0.4 \
--add-host mysql:172.10.0.5 \
--net hadoop_net \
--ip 172.10.0.2 \
-h master \
-p 10022:22 \
-p 2181:2181 \
-p 2887:2888 \
-p 3887:3888 \
-p 9870:9870 \
-p 8088:8088 \
-p 16010:16010 \
-p 10000:10000 \
--restart always \
--name master \
192.168.0.20:5000/hadoop-hbase:master

启动slave1

docker run -d \
--add-host master:172.10.0.2 \
--add-host slave1:172.10.0.3 \
--add-host slave2:172.10.0.4 \
--add-host mysql:172.10.0.5 \
--net hadoop_net \
--ip 172.10.0.3 \
-h slave1 \
-p 20022:22 \
-p 2182:2181 \
-p 2888:2888 \
-p 3888:3888 \
-p 9864:9864 \
-p 8042:8042 \
-p 16011:16010 \
-p 16030:16030 \
--restart always \
--name slave1 \
192.168.0.20:5000/hadoop-hbase:slave1

启动slave2

docker run -d \
--add-host master:172.10.0.2 \
--add-host slave1:172.10.0.3 \
--add-host slave2:172.10.0.4 \
--add-host mysql:172.10.0.5 \
--net hadoop_net \
--ip 172.10.0.4 \
-h slave2 \
-p 30022:22 \
-p 2183:2181 \
-p 2889:2888 \
-p 3889:3888 \
-p 9865:9864 \
-p 8043:8042 \
-p 16031:16030 \
--restart always \
--name slave2 \
192.168.0.20:5000/hadoop-hbase:slave2

启动MySQL

docker run -d \
--add-host master:172.10.0.2 \
--add-host slave1:172.10.0.3 \
--add-host slave2:172.10.0.4 \
--add-host slave2:172.10.0.5 \
--net hadoop_net \
--ip 172.10.0.5 \
-h mysql \
-p 40022:22 \
-p 23306:3306 \
--privileged \
--restart always \
--name mysql \
192.168.0.20:5000/hadoop-hbase:mysql \
/usr/sbin/init

开三个终端分别进入master、slave1、slave2容器，默认账号root密码root

七、启动服务

启动zookeeper

master：/opt/zookeeper-3.4.10/bin/zkServer.sh start
slave1：/opt/zookeeper-3.4.10/bin/zkServer.sh start
slave2：/opt/zookeeper-3.4.10/bin/zkServer.sh start

启动hdfs

master：/opt/hadoop-3.0.3/sbin/start-dfs.sh

启动yarn

master：/opt/hadoop-3.0.3/sbin/start-yarn.sh

启动hbase

master：/opt/hbase-2.1.0/bin/start-hbase.sh

启动hiveserver2

master：hiveserver2

八、访问服务

访问hdfs
ip:9870
ip:9864
ip:9865

访问yarn
ip:8088
ip:8042
ip:8043

访问hbase
ip:16010
ip:16011
ip:16030
ip:16031
访问hive