编程知识 cdmana.com

Hadoop cluster history is the most detailed

build Hadoop colony

Hadoop colony The most detailed in history

If it's useful to you, just pay a little attention to it There is a problem jarvan_mr@163.com

New changes

Environment deployment big data
One 、 brief introduction
step :1.jdk install
2. To configure SSH Password free login
3. To configure hadoop Core documents
4. build hadoop ecology
5. Note for software package
4. format namenode
notes :1.lsy@gbase2 For host group T,lsy@gbase1、lsy@gbase3 by S.
2. Not in /xxx/.bashrc Next source ~/.bash -profile Easy to cause the server to hang up .
3. stay vi ~/.bash_profile Change the host's environment variables source ~/.bash_profile
4. host ip Address to ping through 、 Turn off firewall 、 The clock should be synchronized
Two 、 Step implementation
1. Create a folder, create a directory for the package
[lsy@gbase2 ~]$ mkdir /lsy/app/software
[lsy@gbase2 ~]$ cd /lsy/app/software/
Create a package directory
[lsy@gbase2 ~]$ mkdir /lsy/app/module
[lsy@gbase2 ~]$ cd /lsy/app/module/
2…jdk install
The use of jdk1.8 adopt Xftp Upload to /app/lsy/software Under the folder :(software Build your own )
(1) Query whether to install Java Software :
[lsy@gbase2~]$ rpm -qa | grep java
(2) If the installed version is below 1.7, Unload the JDK:
[lsy@gbase2~]$ sudo rpm -e software package
(3) see JDK The installation path :
[lsy@gbase2~]$ which java

























(4) decompression JDK To module Under the table of contents
[lsy@gbase2~ software]$ tar -zxvf jdk-8u261-linux-x64.tar.gz -C …/module/
(5) To configure JDK environment variable
obtain JDK route
[lsy@gbase2~ jdk1.8.0_261]$ pwd
/app/lsyt/module/ jdk1.8.0_261
(6)vi .bash_profile file
[lsy@gbase2~ software]$ vi ~.bash_profile
stay profile Add at the end of the file JDK route
#JAVA_HOME
JAVA_HOME=/app/lsy/module/jdk1.8.0_261
export PATH= P A T H : PATH: PATH:JAVA_HOME/bin: J A V A H O M E / j r e / b i n J A V A H O M E ( 5 ) Give Way repair Change after Of writing Pieces of raw effect [ l s y @ g b a s e 2   j d k 1.8. 0 2 61 ] JAVA_HOME/jre/bin JAVA_HOME (5) Let the modified file take effect [lsy@gbase2~ jdk1.8.0_261] JAVAHOME/jre/binJAVAHOME5 Give Way repair Change after Of writing Pieces of raw effect [lsy@gbase2 jdk1.8.0261] source .bash_profile
(7) test JDK Is the installation successful
[lsy@gbase2 jdk1.8.0_261]# java -version
java version “1.8.0_261”
3. To configure SSH Password free login
(1) Generate public and private keys :
[lsy@gbase2 .ssh]$ ssh-keygen -t rsa
Then knock ( Three carriage returns ), Two files will be generated id_rsa( Private key )、id_rsa.pub( Public key )
(2) Copy the public key to the target machine for password free login
[lsy@gbase2 .ssh]$ ssh-copy-id gbase1
[lsy@gbase2 .ssh]$ ssh-copy-id gbase2
[lsy@gbase2 .ssh]$ ssh-copy-id gbase3
4. build Hadoop






















  1. Enter into Hadoop Install package path under
    [lsy@gbase2 ~]$ cd /app/lsy/ software
  2. Unzip the installation file to /app/lsy/module below
    [lsy@gbase2 software]$ tar -zxvf hadoop-2.8.3.tar.gz -C …/module
  3. See if the decompression is successful
    [lsy@gbase2 software]$ ls /app/lsy/module/
    hadoop-2.8.3

  4. take Hadoop Add to environment variables
    (1) obtain Hadoop The installation path
    [lsy@gbase2 hadoop-2.8.3]$ pwd
    /app/lsy/module/hadoop-2.8.3
    (2) vi .bash_profile file // Add environment variables
    [lsy@gbase2 hadoop-2.8.3]$ vi .bash_profile
    (3) stay .bash_profile Add at the end of the file JDK route :(shitf+g)
    ##HADOOP_HOME
    HADOOP_HOME=/app/lsy/module/hadoop-2.8.3
    PATH= P A T H : PATH: PATH:JAVA_HOME/bin: J A V A H O M E / j r e / b i n : JAVA_HOME/jre/bin: JAVAHOME/jre/bin:HADOOP_HOME/bin: H A D O O P H O M E / s b i n ( 4 ) Give Way repair Change after Of writing Pieces of raw effect [ l s y @ g b a s e 2 h a d o o p − 2.8.3 ] HADOOP_HOME/sbin (4) Let the modified file take effect [lsy@gbase2 hadoop-2.8.3] HADOOPHOME/sbin(4 Give Way repair Change after Of writing Pieces of raw effect [lsy@gbase2hadoop2.8.3] source .bash_profile








  5. Test for successful installation
    [lsy@gbase2 hadoop-2.8.3]$ hadoop version
    hadoop-2.8.3

5. Write a group script
[lsy@gbase2 ~]$ mkdir bin
[lsy@gbase2 ~]$ cd bin/
[lsy@gbase2 bin]$ touch xsync
[lsy@gbase2 bin]$ vi xsync
#!/bin/bash
# Check whether the parameters are legal
if(($#==0))
then
echo Please enter the file to be distributed !
exit;
fi










# Get the absolute path of the file to be distributed
dirpath=$(cd -P dirname $1; pwd)
filename=$(basename 1 ) e c h o want branch Hair Of writing Pieces of Of road path yes : 1) echo The path of the file to be distributed is : 1)echo want branch Hair Of writing Pieces of Of road path yes :dirpath/$filename

# Get the current user name
lsy@gbase=KaTeX parse error: Expected 'EOF', got '#' at position 10: (whoami) #̲ Loop execution rsync Distribute files to set …i---------------
rsync -rvlt d i r p a t h / dirpath/ dirpath/filename lsy@gbase i : d i r p a t h / m o d u l e d o n e 6. with Set up h a d o o p nucleus heart writing Pieces of ( 1 ) . c o r e − s i t e . x m l with Set up c o r e − s i t e . x m l [ l s y @ g b a s e 2 h a d o o p ] i:dirpath/module done 6. To configure hadoop Core documents (1).core-site.xml To configure core-site.xml [lsy@gbase2 hadoop] i:dirpath/moduledone6. with Set up hadoop nucleus heart writing Pieces of (1).coresite.xml with Set up coresite.xml[lsy@gbase2hadoop] vi core-site.xml
Write the following configuration in this file


fs.defaultFS hdfs:// gbase1:9000 hadoop.tmp.dir //app/lsy/module/hadoop-2.8.3/data/tmp (2).hdfs-site.xml To configure hdfs-site.xml [lsy@gbase2 hadoop]$ vi hdfs-site.xml Write the following configuration in this file dfs.namenode.secondary.http-address gbase3:50090 (3).mapred-site.xml mapreduce.framework.name yarn (4).yarn-site.xml To configure yarn-site.xml [lsy@gbase2 hadoop]$ vi yarn-site.xml Add the following configuration to the file yarn.nodemanager.aux-services mapreduce_shuffle yarn.resourcemanager.hostname gbase2 mapreduce.jobhistory.address hadoop101:10020 mapreduce.jobhistory.webapp.address gbase1:19888 yarn.log.server.url http:// gbase1:19888/jobhistory/logs 7. format 1. stay lsy@gbase1 Format on the upper node : [lsy@gbase1 hadoop]$ namenode:hadoop namenode -format 2. stay lsy@gbase1 Start the namenode node : [lsy@gbase1 hadoop]$ hadoop-daemon.sh start namenode

test :dfs.namenode.http-address:50070

3. Start at different nodes datanode
[lsy@gbase1 hadoop]$ hadoop-daemon.sh start datanode
[lsy@gbase2 hadoop]$ hadoop-daemon.sh start datanode
[lsy@gbase3 hadoop]$ hadoop-daemon.sh start datanode


4. stay gbase3 start-up secondarynamenode
[lsy@gbase3 hadoop]$ hadoop-daemon.sh start secondarynamenode
5. stay gbase2 start-up yarn
[lsy@gbase2 hadoop]$:yarn-daemon.sh start resourcemanager


6. stay lsy@gbase1-3 Start the nodemanager:
[lsy@gbase1 hadoop] y a r n − d a e m o n . s h s t a r t n o d e m a n a g e r [ l s y @ g b a s e 2 h a d o o p ] yarn-daemon.sh start nodemanager [lsy@gbase2 hadoop] yarndaemon.shstartnodemanager[lsy@gbase2hadoop]yarn-daemon.sh start nodemanager
[lsy@gbase3 hadoop] y a r n − d a e m o n . s h s t a r t n o d e m a n a g e r 7. with Set up l s y @ g b a s e 1 : 19888 calendar The history of clothing service device : [ l s y @ g b a s e 1 h a d o o p ] yarn-daemon.sh start nodemanager 7. To configure lsy@gbase1:19888 History server : [lsy@gbase1 hadoop] yarndaemon.shstartnodemanager7. with Set up lsy@gbase1:19888 calendar The history of clothing service device [lsy@gbase1hadoop] vi mapred-site.xml

mapreduce.jobhistory.address
gbase1:10020


mapreduce.jobhistory.webapp.address
gbase1:19888










yarn.log.server.url http:// gbase1:19888/jobhistory/logs [lsy@gbase1 hadoop]$ mr-jobhistory-daemon.sh start historyserver

8. View all nodes jps

3、 ... and .HDFS-HA Cluster configuration / build zookeeper
1.ZK Server configuration
(1). Unpack : Unzip to …/module Next
[lsy@gbase2 software]$ tar -zxvf zookeeper-3.4.10.tar.gz -C …/module
(2). rename /lsy/module/zookeeper-3.4.10/conf In this directory zoo_sample.cfg by zoo.cfg
[lsy@gbase2 zookeeper-3.4.10]$ cd conf/
[lsy@gbase2 conf]$ ll
total 12
-rw-r–r--. 1 lsy fxjk 535 Mar 23 2017 configuration.xsl
-rw-r–r--. 1 lsy fxjk 2161 Mar 23 2017 log4j.properties
-rw-r–r--. 1 lsy fxjk 922 Mar 23 2017 zoo_sample.cfg
[lsy@gbase2 conf]$ cp zoo_sample.cfg zoo.cfg
(3).vi zoo.cfg
[lsy@gbase2 conf]$ vi zoo.cfg












Be careful : Remember to create datas For storage dataDir=/tmp/zookeeper
[lsy@gbase2 zookeeper-3.4.10]$ mkdir datas
lsy@gbase2 zookeeper-3.4.10]$ cd datas/
[lsy@gbase2 datas]$ pwd
/app/lsy/module/zookeeper-3.4.10/datas
(4). Start the test zookeeper
[lsy@gbase2 zookeeper-3.4.10]$ bin/zkServer.sh start
[lsy@gbase2 zookeeper-3.4.10]$ bin/zkServer.sh status
2. build zookeeper colony
(1). take gbase2 Medium zookeeper scp To the node 1、2 in :
[lsy@gbase2 module]$ scp -r zookeeper-3.4.10/ lsy@gbase1:/app/lsy/dirpath/
[lsy@gbase2 module]$ scp -r zookeeper-3.4.10/ lsy@gbase3:/app/lsy/dirpath/
(2). To configure zoo.cfg file
1 rename /opt/module/zookeeper-3.4.10/conf In this directory zoo_sample.cfg by zoo.cfg
[lsy@gbase2 conf]$ mv zoo_sample.cfg zoo.cfg
2. open zoo.cfg file
[lsy@gbase2 conf]$ vim zoo.cfg
Add the following configuration
server.1= gbase1:2888:3888
server.2= gbase2:2888:3888
server.3= gbase3:2888:3888



















(3). Mass hair
[lsy@gbase2 conf]$ xsync zoo.cfg

(4). Configure master (gbase1 For leaders L) Remember to distribute myid
[lsy@gbase2 zookeeper-3.4.10]$ cd datas/
[lsy@gbase2 datas]$ ll
total 0
drwxr-xr-x. 2 lsy fxjk 6 Nov 11 17:22 version-2
[lsy@gbase2 datas]$ vi myid




(5). stay 1-3 Start the test
[lsy@gbase1 datas]$ /app/lsy/dirpath/zookeeper-3.4.10/bin/zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /app/lsy/dirpath/zookeeper-3.4.10/bin/…/conf/zoo.cfg
Starting zookeeper … STARTED



[lsy@gbase2 conf]$ /app/lsy/module/zookeeper-3.4.10/bin/zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /app/lsy/module/zookeeper-3.4.10/bin/…/conf/zoo.cfg
Starting zookeeper … STARTED


[lsy@gbase3 datas]$ /app/lsy/dirpath/zookeeper-3.4.10/bin/zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /app/lsy/dirpath/zookeeper-3.4.10/bin/…/conf/zoo.cfg
Starting zookeeper … STARTED
(6). see leader/follower
[lsy@gbase1 datas]$ /app/lsy/dirpath/zookeeper-3.4.10/bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /app/lsy/dirpath/zookeeper-3.4.10/bin/…/conf/zoo.cfg
Mode: leader







[lsy@gbase2 conf]$ /app/lsy/module/zookeeper-3.4.10/bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /app/lsy/module/zookeeper-3.4.10/bin/…/conf/zoo.cfg
Mode: follower


[lsy@gbase3 datas]$ /app/lsy/dirpath/zookeeper-3.4.10/bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /app/lsy/dirpath/zookeeper-3.4.10/bin/…/conf/zoo.cfg
Mode: follower


(7). Writing group scripts
#!/bin/bash
if(($#!=1))
then
echo Please enter start or stop or status
exit;
fi
if [ $1 = start ] || [ $1 = stop ] || [ $1 = status ]
then
xcall zkServer.sh 1 e l s e e c h o please transport Enter into s t a r t or s t o p or s t a t u s ! f i Four . take build F l u m 1. take build F l u m ( 1 ) . take f l u m e . t a r . g z On Pass on To l i n u x Of / a p p / l s y / s o f t w a r e Objective record Next ( 2 ) . Explain Pressure [ l s y @ g b a s e 2 s o f t w a r e ] 1 else echo Please enter start or stop or status! fi Four . build Flum 1. build Flum (1). take flume.tar.gz Upload to linux Of /app/lsy/software Under the table of contents (2). decompression [lsy@gbase2 software] 1elseecho please transport Enter into start or stop or status!fi Four take build Flum1. take build Flum(1). take flume.tar.gz On Pass on To linux Of /app/lsy/software Objective record Next (2). Explain Pressure [lsy@gbase2software] tar -zxvf apache-flume-1.7.0-bin.tar.gz -C …/module
(3). modify apache-flume-1.7.0-bin For the name of the flume
[lsy@gbase2 module]$ mv apache-flume-1.7.0-bin flume
(4). take flume/conf Under the flume-env.sh.template Change the file to flume-env.sh,
[lsy@gbase2 conf]$ mv flume-env.sh.template flume-env.sh
install flume matters needing attention :1 Yes JAVA_HOME 2. You can use it by decompressing it
2. start-up agent
(1). vi ~/.bash_profile Configuration environment
[lsy@gbase2 flume]$ vi ~/.bash_profile
(2.) add to /app/lsy/module/flume

















(3). start-up agent
[lsy@gbase2 flume] f l u m e − n g a g e n t − n a g e n t Of name call − f a g e n t with Set up writing Pieces of − c Its He writing Pieces of stay Of Objective record − D p r o p e r t y = v a l u e 5、 ... and . take build h i v e 1. hold a p a c h e − h i v e − 1.2.1 − b i n . t a r . g z On Pass on To l i n u x Of / l s y / s o f t w a r e Objective record Next 2. Explain Pressure a p a c h e − h i v e − 1.2.1 − b i n . t a r . g z To / l s y / m o d u l e / Objective record Next Noodles [ l s y @ g b a s e 2 s o f t w a r e ] flume-ng agent -n agent The name of -f agent The configuration file -c Other files in the directory -Dproperty=value 5、 ... and . build hive 1. hold apache-hive-1.2.1-bin.tar.gz Upload to linux Of /lsy/software Under the table of contents 2. decompression apache-hive-1.2.1-bin.tar.gz To / lsy /module/ Below directory [lsy@gbase2 software] flumengagentnagent Of name call fagent with Set up writing Pieces of c Its He writing Pieces of stay Of Objective record Dproperty=value 5、 ... and take build hive1. hold apachehive1.2.1bin.tar.gz On Pass on To linux Of /lsy/software Objective record Next 2. Explain Pressure apachehive1.2.1bin.tar.gz To /lsy/module/ Objective record Next Noodles [lsy@gbase2software] tar -zxvf apache-hive-1.2.1-bin.tar.gz -C …/module/
3. modify apache-hive-1.2.1-bin.tar.gz For the name of the hive
[lsy@gbase2 module]$ mv apache-hive-1.2.1-bin/ hive
4. modify /lsy/module/hive/conf In the catalog hive-env.sh.template The name is hive-env.sh
[lsy@gbase2 conf]$ mv hive-env.sh.template hive-env.sh
5. To configure hive-env.sh file
To configure HIVE_HOME route
export HIVE_HOME=/app/lsy/module/hive







6. test :
[lsy@gbase2 hive]$ hive
result :

7. install hive matters needing attention :

  1. Check to see if there is HADOOP_HOME、JAVA_HOME

2. stay bin Configuration to PATH in , Provide... In the environment variable HIVE_HOME

6、 ... and . build Redis
[lsy@gbase1 hadoop]$ make

test
[lsy@gbase2 bin]$ pwd
/usr/local/bin

7、 ... and . build Spark( It 's loaded with standalone Mode of )
404

(1). Check physical memory / Virtual memory (/app/lsy/module/hadoop-2.8.3/etc/hadoop)
[lsy@gbase2 hadoop]$ vi yarn-site.xml

yarn.nodemanager.pmem-check-enabled false yarn.nodemanager.vmen-check-enabled false (2). To configure spark yarn Pattern 1. Change of name : [lsy@gbase2 conf]$ mv spark-env.sh.template spark-env.sh 2. To configure spark-env.sh [lsy@gbase2 conf]$ vi spark-env.sh 3. 4. To configure Spark Task history server 1. modify spark-env.sh file export SPARK_HISTORY_OPTS="-Dspark.history.ui.port=18080 -Dspark.history.retainedApplications=30 -D spark.history.fs.logDirectory=hdfs://gbase2:9000/spark1015_2"

2. Configuration log
[lsy@gbase2 conf]$ mv spark-defaults.conf.template spark-defaults.conf
[lsy@gbase2 conf]$ vi spark-defaults.conf

3. test :
[lsy@gbase2 bin]$ pwd
/app/lsy/module/spark-2.1.1-bin-hadoop2.7/bin
[lsy@gbase2 bin]$ cd spark-submit \


3. Configure history server
[lsy@gbase2 conf]$ vi spark-defaults.conf

[lsy@gbase2 conf]$ vi spark-env.sh

[lsy@gbase2 conf]$ hadoop fs -mkdir /spark-log-1015

One .lcoal Pattern ( For testing purposes only ):
[lsy@gbase2 ~]$ vi ~/.bash_profile

[lsy@gbase2 ~]$ source ~/.bash_profile
[lsy@gbase2 module]$ cd spark-local
[lsy@gbase2 spark-local]$ ll
[lsy@gbase2 spark-local]$ bin/spark-submit --class org.apache.spark.examples.SparkPi --executor-memory 1G --total-executor-cores 2 ./examples/jars/spark-examples_2.11-2.1.1.jar 100
Running results :



1. Second test :
[lsy@gbase2 spark-local]$ bin/run-example SparkPi 100

2…spark-shell Next :
[lsy@gbase2 spark-local]$ bin/spark-shell

3. Quit using :scala> :q
Two .spark-standalone Pattern
[lsy@gbase2 module]$ cp -r spark-2.1.1-bin-hadoop2.7 spark-standalone
[lsy@gbase2 conf]$ pwd
/app/lsy/module/spark-standalone/conf
[lsy@gbase2 conf]$ mv spark-env.sh.template spark-env.sh
[lsy@gbase2 conf]$ vi spark-env.sh





[lsy@gbase2 conf]$ mv slaves.template slaves
[lsy@gbase2 conf]$ vi slaves

[lsy@gbase2 module]$ xsync spark-standalone
[lsy@gbase2 module]$ cd spark-standalone
[lsy@gbase2 spark-standalone]$ sbin/start-all.sh

Check the process :
[lsy@gbase2 sbin]$ jps

. test
[lsy@gbase2 spark-standalone]$ bin/spark-submit --class org.apache.spark.examples.SparkPi --executor-memory 1G --total-executor-cores 2 ./examples/jars/spark-examples_2.11-2.1.1.jar 100

Configure history server :
[lsy@gbase2 conf]$ mv spark-defaults.conf.template spark-defaults.conf
[lsy@gbase2 conf]$ vi spark-defaults.conf

[lsy@gbase2 conf]$ vi spark-env.sh

Distribute documents :
[lsy@gbase2 conf]$ xsync spark-env.sh
The path of the file to be distributed is :/app/lsy/module/spark-standalone/conf/spark-env.sh
[lsy@gbase2 conf]$ xsync spark-defaults.conf
The path of the file to be distributed is :/app/lsy/module/spark-standalone/conf/spark-defaults.conf
establish 1015
[lsy@gbase2 spark-standalone]$ hadoop fs -mkdir /spark-log-1015





start-up
[lsy@gbase2 spark-standalone]$ sbin/start-all.sh

3、 ... and .Yarn Pattern

  1. modify yarn-site.xml
    [lsy@gbase2 hadoop]$ pwd
    /app/lsy/module/hadoop-2.8.3/etc/hadoop
    [lsy@gbase2 hadoop]$ vi yarn-site.xml


yarn.nodemanager.pmem-check-enabled false yarn.nodemanager.vmem-check-enabled false

[lsy@gbase2 module]$ cp -r spark-2.1.1-bin-hadoop2.7 spark-yarn
tell M/R hadoop Address
/app/lsy/module/spark-yarn/conf
[lsy@gbase2 conf]$ mv spark-env.sh.template spark-env.sh
[lsy@gbase2 conf]$ vi spark-env.sh



2. Change the log
YARN_CONF_DIR=/app/lsy/module/hadoop-2.8.3/etc/hadoop
export SPARK_HISTORY_OPTS="-Dspark.history.ui.port=18080 -Dspark.history.retainedApplications=30 -D
spark.history.fs.logDirectory=hdfs://gbase2:9000/spark1117_2"


[lsy@gbase2 conf]$ mv spark-defaults.conf.template spark-defaults.conf

test :
[lsy@gbase2 spark-yarn]$ hadoop fs -mkdir /spark1117_2
[lsy@gbase2 spark-yarn]$ sbin/start-history-server.sh
[lsy@gbase2 spark-yarn]$ bin/spark-submit --class org.apache.spark.examples.SparkPi --executor-memory 1G --total-executor-cores 2 ./examples/jars/spark-examples_2.11-2.1.1.jar 100


8、 ... and . build mariadb
[lsy@gbase2 module]$ getconf LONG_BIT
64
[lsy@gbase2 software]$ tar -zxvf mariadb-10.2.14-linux-x86_64.tar.gz -C …/module
[lsy@gbase2 module]$ mv mariadb-10.2.14-linux-x86_64 mariadb
Nine . Software package note
1.jdk:jdk-8u261-linux-x64.tar.gz
2.hadoop:hadoop-2.8.3.tar.gz
3. zookeeper-3.4.10.tar.gz
4. apache-flume-1.7.0-bin.tar.gz
5. apache-hive-1.2.1-bin.tar.gz
6. redis-3.0.4.tar.gz
7. spark-2.1.1-bin-hadoop2.7.tgz
8. mariadb-10.2.14-linux-x86_64












If you have any questions, please contact me mailbox :jarvan_mr@163.com
jarvan_mr@163.com

版权声明
本文为[osc_ v1ao43h5]所创,转载请带上原文链接,感谢
https://cdmana.com/2020/12/20201225092906488A.html

Scroll to Top