编程知识 cdmana.com

Docker-9: deploy big data cluster with ambari-2.7.3 - two node demo

Catalog :

(1).docker Deploy ambari

(2). To configure SSH Avoid secret landing

(3). To configure ambari Basics repo

(4). auxiliary repo To configure

(5). Create clusters

(6). Resource consumption

appendix 1.ambari dependent yum repo

appendix 2. Reference article

(1).docker Deploy ambari

git clone https://github.com/hepyu/docker-ambari-2.7.git

( I fork since :git clone https://github.com/cuongvomanh/docker-ambari-2.7.git, A little change has been made , Also do personal records )

cd docker-ambari-2.7

Download the image yum Source file :

cd ambari-base-local

wget -O local.repo http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.3.0/ambari.repo

cd ambari-server-local

wget -O local.repo http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.3.0/ambari.repo

Making mirrors : It's a little long , wait a moment .

./scripts/build-image.sh

git There's a file under the project nohup-image.out It's the author's one time image-build Record .

Local boot :

./scripts/create-local.sh

docker ps Look at the container :

get web ip:

docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' amb-server

Browser access : You need to configure ecs Security group development for 8088 port

http://ecs-ip:8088/

Obtain by the following command 3 Of nodes ip Cluster deployment :

docker inspect Containers id | grep -i ip

Three nodes IP yes :

172.17.0.4-- Containers id(7153911bede2)--ambari-base-local

172.17.0.3-- device id(673bdc7ecffb)--ambari-base-local

172.17.0.2-- device id(7e8902bae9a2)--ambari-server-local

(2). To configure SSH Avoid secret landing

For this 3 Security free settings for machines :172.17.0.7,172.17.0.6,172.17.0.2

1. First root Under the authority of 3 Virtual machines to modify

The modification order is as follows :

sudo hostnamectl set-hostname node1

I'm here 3 Host names are changed to node1、node, The actual deployment will only deploy to one docker( Saving resource ).

172.17.0.4-- Containers id(7153911bede2)--ambari-base-local::node1

172.17.0.3-- device id(673bdc7ecffb)--ambari-base-local::node2

172.17.0.2-- device id(7e8902bae9a2)--ambari-server-local::node3

After modification, you can use the command hostname Query the current host name to confirm that the modification is successful .

2. Respectively in 3 Edit on virtual machines /etc/hosts file , add to IP And the host name

( These three lines are added to each one , Don't forget to add yourself )

172.17.0.4 node1

172.17.0.3 node2

172.17.0.2 node3

The purpose here is to achieve 3 Interconnection between virtual machines , And it shortens the time it takes them to resolve the host name when they connect .

At the same time, please note out :

#172.17.0.3 amb1.service.consul amb1

#172.17.0.4 amb2.service.consul amb2

3. Generate the key

First of all, here 3 Nodes execute instructions :ssh-keygen , Then press enter until it's over . At this point, you can find that in /root/.ssh/ Created under folder id_rsa( Private key ) 、 id_rsa.pub ( Public key ), This is the generated secret key pair .

4. Public key sharing

To synchronize the public key to all nodes requires input root Password ,3 All nodes are set up root The password for 123456

sudo passwd

Send your public key to other nodes , Including myself . The instructions are as follows :

ssh-copy-id node1

ssh-copy-id node2

ssh-copy-id node3

(3). To configure ambari Basics repo

From the address below hdp3.1.4 Of yum repo Address

http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json

hdp-3.1.4 Of yum repo Address :

http://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.1.4.0/HDP-3.1.4.0-315.xml

Open to see hdp, hdp-gpl, hdp-utils Of repo:

(4). auxiliary repo To configure

1.Timeline Service V1.5 Need configuration repo

Otherwise, the report will be wrong :

Error: Package: hadoop_3_1_4_0_315-3.1.1.3.1.4.0-315.x86_64 (HDP-3.1-repo-1)

Requires: psmisc

Error: Package: hadoop_3_1_4_0_315-hdfs-3.1.1.3.1.4.0-315.x86_64 (HDP-3.1-repo-1)

Requires: libtirpc-devel

Error: Package: hadoop_3_1_4_0_315-3.1.1.3.1.4.0-315.x86_64 (HDP-3.1-repo-1)

Requires: nc

Error: Package: hadoop_3_1_4_0_315-3.1.1.3.1.4.0-315.x86_64 (HDP-3.1-repo-1)

Requires: redhat-lsb

To configure repo:

Get into ambari Of docker Container node ,

cd /etc/yum.repos.d/

cp backup_repo/CentOS-Base.repo .

(5). Create clusters

Start creating clusters , Click on :LAUNCH INSTALL WIZARD

The next step is to enter ambari-server The private key of the host machine , such ambari-server To deploy components to other nodes :

Only add node1, node2

because node3 yes ambari-server node , If you add it, you will report an error :

Ambari agent machine hostname (amb-server.service.consul) does not match expected ambari server hostname (node3). Aborting registration. Please check hostname, hostname -f and /etc/hosts file to confirm your hostname is setup correctly

Go to the component selection page , I choose the following components on demand :

HDFS:3.1.1

YARN + MapReduce2:3.1.1

Tez:0.9.1,Tez Manage all offline jar package

Hive:3.1.0

HBase:2.0.2

ZooKeeper:3.4.6

Kafka:2.0.0

SmartSense:1.5.1.2.7.3.0-139

Ambari Metrics:0.1.0

The codes are all :La123456

(6). Consume resources

I personally have one 8 nucleus 32G Alibaba cloud ECS, After deploying this article 2 The resource situation after the node cluster :

Even though I've installed k8s and istio, But the big data is bound to be big :

appendix 1.ambari dependent yum repo

hdp Different versions yum The warehouse url Address :

http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json

hdp3.1.4 Of yum The warehouse url Address :

http://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.1.4.0/HDP-3.1.4.0-315.xml

appendix 2. Reference article

1. Make Ambari Local yum Source

https://www.jianshu.com/p/3597bf155e99

2.Ambari build Hadoop Cluster configuration SSH Password free login

https://blog.csdn.net/qq_28757391/article/details/82724919

3.Apache Ambari Installation

https://docs.cloudera.com/HDPDocuments/Ambari-2.7.4.0/bk_ambari-installation/content/hdp_314_repositories.html

This article is from WeChat official account. - Walking thousands of miles (a_thousands_of_miles) , author : Walking thousands of miles

The source and reprint of the original text are detailed in the text , If there is any infringement , Please contact the yunjia_community@tencent.com Delete .

Original publication time : 2020-11-07

Participation of this paper Tencent cloud media sharing plan , You are welcome to join us , share .

版权声明
本文为[Walk thousands of miles]所创,转载请带上原文链接,感谢

Scroll to Top