编程知识 cdmana.com

Docker uses dockerfile to build Hadoop image and cluster instance

Article transferred from :https://cloud.tencent.com/developer/article/1084166 

Catalog

obtain centos7 Mirror image

install SSH

Setting up IP

structure Hadoop Mirror image

To configure Hadoop colony


Multiple servers are needed to build a cluster environment , For us personally , It's usually a threshold , You need to use a virtual machine , Installing the operating system , Then it runs multiple virtual machines

Installing an operating system is not an easy task , And running multiple virtual machines also has certain requirements for PC performance , These barriers affect the practical enthusiasm of many small partners

Use Docker It's easy , No need to install the operating system , Download a mirror directly , Such as centos, So the operating system has , Run multiple containers based on this system image , It's like a lot of virtual machines , And the consumption of system performance is much less than that of virtual machine

Suppose we have a centos7 Mirror image , Start multiple containers , Each container has a IP, Can pass. SSH Connection operation , So you can install... In each container JAVA Hadoop, So as to build a cluster environment

Use Docker build Hadoop The process of clustering includes :

  • install Docker
  • obtain centos Mirror image
  • install SSH
  • Configure the container IP
  • install JAVA and Hadoop
  • To configure Hadoop

The first 1 The steps are simple , Go to Docker Download and install the official website , The first 5 And the 6 The steps are the same as in a stand-alone server , So here's to introduce 2-4 Step by step Docker Mode of operation

obtain centos7 Mirror image

$ docker pull centos

Probably 70 many M, Using alicloud, etc Docker The accelerator will download it soon , Then you can see it in the mirror list

The command to view the image list :

$ docker images

install SSH

With centos7 Mirror based , Build a with SSH Functional centos

$ vi Dockerfile

Content :

FROM centos
MAINTAINER dys

RUN yum install -y openssh-server sudo
RUN sed -i 's/UsePAM yes/UsePAM no/g' /etc/ssh/sshd_config
RUN yum  install -y openssh-clients

RUN echo "root:111111" | chpasswd
RUN echo "root   ALL=(ALL)       ALL" >> /etc/sudoers
RUN ssh-keygen -t dsa -f /etc/ssh/ssh_host_dsa_key
RUN ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key

RUN mkdir /var/run/sshd
EXPOSE 22
CMD ["/usr/sbin/sshd", "-D"]

The main idea of this passage is : With centos Mirror based , install SSH The relevant package , Set up root The user's password is 111111, And start the SSH service

Execute the command to build the image , The new image is named centos7-ssh

$ docker build -t="centos7-ssh" .

After execution , You can see in the mirror list

$ docker images

Setting up IP

Need to use pipework, It's used to set up the container IP

To download

$ git clone https://github.com/jpetazzo/pipework.git
$ cp pipework/pipework /usr/local/bin/

install bridge-utils

$ yum -y install bridge-utils

Creating networks

$ brctl addbr br1
$ ip link set dev br1 up
$ ip addr add 192.168.3.1/24 dev br1

be based on centos7-ssh This image starts a container , be known as centos7.ssh

$ docker run -d --name=centos7.ssh centos7-ssh

Set up IP

$ pipework br1 centos7.ssh 192.168.3.20/24

It's called centos7.ssh The container specifies IP 192.168.3.20

Separate use ping And ssh Command to verify , See if you can ping Tonghe successfully logged in

$ ping 192.168.3.20
$ ssh 192.168.3.20

ssh After successful login , That means a server is ready , Run again 2 A container 、 Set it up IP, So it's equivalent to having 3 Servers

$ docker run -d --name=centos7.ssh2 centos7-ssh
$ docker run -d --name=centos7.ssh3 centos7-ssh

$ pipework br1 centos7.ssh2 192.168.3.22/24
$ pipework br1 centos7.ssh3 192.168.3.23/24

That's it 3 One can SSH Connected containers , You can install and configure the cluster environment on it

structure Hadoop Mirror image

It's running 3 individual centos Containers , It needs to be installed separately in each container Hadoop Environmental Science , We can be like building SSH Mirror image is the same , Construct a Hadoop Mirror image , And then run 3 individual Hadoop Containers , That makes it easier

$ vi Dockerfile

Content :

FROM centos7-ssh
ADD jdk-8u101-linux-x64.tar.gz /usr/local/
RUN mv /usr/local/jdk1.8.0_101 /usr/local/jdk1.8
ENV JAVA_HOME /usr/local/jdk1.8
ENV PATH $JAVA_HOME/bin:$PATH

ADD hadoop-2.7.3.tar.gz /usr/local
RUN mv /usr/local/hadoop-2.7.3 /usr/local/hadoop
ENV HADOOP_HOME /usr/local/hadoop
ENV PATH $HADOOP_HOME/bin:$PATH

RUN yum install -y which sudo

This is based on centos7-ssh This mirror image , hold JAVA and Hadoop The environment has been configured

Premise : stay Dockerfile Get ready in your directory jdk-8u101-linux-x64.tar.gz And hadoop-2.7.3.tar.gz

Execute the build command , The new image is named hadoop

$ docker build -t="hadoop" .

function 3 individual hadoop Containers , Named as hadoop0,hadoop1,hadoop2,hadoop0 As master

$ docker run --name hadoop0 --hostname hadoop0 -d -P -p 50070:50070 -p 8088:8088 hadoop
$ docker run --name hadoop1 --hostname hadoop1 -d -P hadoop
$ docker run --name hadoop2 --hostname hadoop2 -d -P hadoop

Containers hadoop0 Startup time , Mapped port number ,50070 and 8088, Is used to access in the browser hadoop WEB Interface

To configure IP

$ pipework br1 hadoop0 192.168.3.30/24
$ pipework br1 hadoop1 192.168.3.31/24
$ pipework br1 hadoop2 192.168.3.32/24

To configure Hadoop colony

The opening of new 3 A terminal window , Connect to respectively hadoop0,hadoop1,hadoop2, Easy to operate

$ docker exec -it hadoop0 /bin/bash
$ docker exec -it hadoop1 /bin/bash
$ docker exec -it hadoop2 /bin/bash

Modify... In each container /etc/hosts, add to :

192.168.3.30    master
192.168.3.31    slave1
192.168.3.32    slave2

The next step is configuration SSH No login , And hadoop Configuration file for , Method and before “Hadoop Cluster building ” The same as in China

So use Docker Realized Hadoop Cluster building .

版权声明
本文为[boonya]所创,转载请带上原文链接,感谢

Scroll to Top