1. start-up namenode and datanode, stay master Enter the command on the hdsf dfsadmin -report View the operation of the whole cluster ( Remember to turn off the firewall )

2.  Enter command view hadoop Listening port ,netstat -ntlp

3.  Enter in the host browser {masterIP}:50070 To view the

4.  Shut down the cluster

master Machine command :hadoop-daemon.sh stop namenode

slave Machine command :hadoop-daemon.sh stop datanode

5.  Create centralized management , stay master Operate on the machine . Enter the command

vim /usr/local/hadoop/etc/hadoop/slaves

6.  After the operation of the previous step , Can be in master On the machine , Centralized management control namenode And all the datanode. stay master On the machine , Start all of hadoop service , Enter the command start-dfs.sh, Enter the corresponding password , Then input jps see ( This command will start by default SecondaryNameNode)

7.  step 6 When , You need to enter a lot of passwords , Now set up a password free login :

a).  stay master Enter into the machine ~/.ssh Catalog :cd ~/.ssh/

b).  Input ssh-keygen -t rsa, And keep knocking back

c).  At the end of the order , There will be two more files in this directory

id_rsa For the current root User's private key ,id_rsa.pub It's the public key

d).  Copy the public key to all datanode On the machine , such , When master The machine takes the login information encrypted by the private key and sends it to datanode Machine time ,datanode The public key stored in the machine can be unlocked , Prove that the user is logged in , Therefore, it can realize password free login .

e).  Copy id_rsa.pub Methods , stay master Enter commands on the machine ssh-copy-id slave1, Input password , You can copy the public key to slave1 In machine .

f).  Go to the slave1 Mechanical ~/.ssh View under directory , Yes authorized_keys The file indicates that the copy was successful

g).  go back to master On the machine , Input ssh slave1 Remote login , It is found that you can log in successfully without entering a password , That is to achieve a secret free login .

h).  Empathy , take id_rsa.pub Copy the file to another datanode On the machine , also , And make a copy of it for yourself , That is to copy a copy to master On the machine .

8.  Stop the cluster stop-dfs.sh

9.  Start the cluster again start-dfs.sh I found that I didn't have to enter the password

10.  see hadoop Files in the root directory hadoop fs -ls /( You can also hadoop fs Replace with hdfs dfs, At present, the cluster has just been created , The directory is empty , Besides , Delete and so on linux Command similar , for example hdfs dfs -rm -r -f /a.txt, other hdfs You can go to Mrs. Du Niang and Mr. Gu )

11.  Upload a file to hadoop(master Operate on the machine )

a).  Upload /usr/local In the catalog hadoop-2.7.3.tar.gz file ,hadoop default block by 128M, The document is 214M, So it's going to be partitioned .

b).  Enter the command hadoop fs -put ./hadoop-2.7.3.tar.gz /, Ahead ./hadoop-2.7.3.tar.gz Local storage path for uploaded files , The back / For storage in hadoop The path of .

c).  Input hadoop fs -ls / Look at the file you just uploaded

d).  View the file just uploaded through the web page , Click on Utilsities->Borwse the file system

e).  Click on the file , Found that the file is divided into two blocks . And three copies of each piece are kept in datanode among .

12. hadoop After segmentation , By default, the block will be copied in triplicate , In order to recover in case of special circumstances , This number can be changed , Now change the default copy from three copies to two copies .( stay master Make changes on the machine )

a).  edit /usr/local/hadoop/etc/hadoop/hdfs-site.xml file , stay configuration Add the following to the node

Specific configuration instructions , You can see hadoop Installation directory hadoop-2.7.3/share/doc/hadoop/index.html file

b).  Shut down the cluster stop-dfs.sh, Start cluster start-dfs.sh( restart hadoop)

c).  Based on the above , take jdk Upload the installation files to the cluster , Go to the website and watch .

Find out from the above picture , The uploaded file is split into two block block , among Block0 On the slave2 and slave3 above ,Block1 On the slave1 and slave2 above , here , If we kill slave3, As shown in the following figure , that Block0 There's only one left , And our configuration file has two , So let's test that out hadoop Will you make another copy for us ( The answer is to , This is automatic redundancy ).

d).  Before the above experiments , We also need to configure an option

The attribute represents ,hadoop After what interval of time datanode Do a test , Check if they're down , In milliseconds , The default is 300000, That is to say 5 minute , Let's change it to a smaller value , Otherwise, we have to wait for a long time 5 minute . Also modify /usr/local/hadoop/etc/hadoop/hdfs-site.xml file , Change the value to 10000, namely 10 Second , And then restart it hadoop.

e).  stay slave3 On the machine , close hadoop service :hadoop-daemon.sh stop datanode, Waiting in silence 10 Second ,9、8、7、6...

f).  Go to the website and watch slave3 Confirmed dead

g).  Look again jdk File status , Find out Block0 And in the slave1 One of them was copied , The above results are verified .

13.  reflection : Start again at this time slave3, and slave3 There's a share of Block0 Backup , In that case Block0 There are three copies in all , And we're configuring two backups , Consider starting at this time slave3, Will you delete one Block0 Well ?

Hadoop Learning notes ( Two ): Simple operation of more related articles

  1. Html Learning notes ( Two ) Simple label

    The point of the label The purpose of the label The default style of the tag in the browser <body> label : What is displayed on the web page <p> label : Add paragraph <hx> label : Add the title There are a total of 6 individual ,h1.h ...

  2. Java Design pattern learning notes ( Two ) Simple factory model

    Preface This is one of the design pattern learning notes , If interested in other models , You can find the design pattern learning notes summary address from this address Text begins ... 1. brief introduction The simple factory model does not belong to GoF23 One of the design patterns in , But in software development, the application is also more ...

  3. Hadoop-HBASE case analysis -Hadoop Learning notes &lt; Two &gt;

    I was lucky to be in MOOC Xiaoxiang College hadoop Experience class .  This is little elephant college hadoop2.X Summarize the notes in Chapter 8   This paper mainly introduces HBase, An application case of distributed database . Case Overview : 1) Time series database (OpenTSD ...

  4. Hadoop Study note 2

    One . Set no password sudo jurisdiction , Not for ordinary users and root Switching back and forth between users chmod u+w /etc/sudoers vim /etc/sudoers # Add the following to the first line : hadoop ALL=(ro ...

  5. Hadoop Learning notes —— Entry instruction operation

    hypothesis Hadoop Installation directory HADOOP_HOME by /home/admin/hadoop. Startup and shutdown startup HADOOP1. Get into HADOOP_HOME Catalog . 2. perform sh bin/start-all. ...

  6. MongoDB Study note 2 —Shell operation

    data type MongoDB In the reserved JSON Basic key / Values are based on properties , Added some other data types . null null Used to indicate a null or nonexistent field :{“x”:null} Boolean type The boolean type has two values true and fal ...

  7. redis Study note 2 ( Simple dynamic string )

    redis The basic data structure of is dynamic array One .c Language dynamic array Let's take a look at the general dynamic array structure struct MyData { int nLen; char data[0]; }; It's a common technique that's widely used , Often used to ...

  8. hadoop Study notes 3 -- Simple application

    1. Give orders to HDFS Transfer files Upload :./hadoop fs -put hdfs.cmd ( Local filename ) hdfs:// Host name :9000/       hadoop fs -copyFromLoca ...

  9. Python Learning notes _03: Simple operation MongoDB database

    Catalog 1. Inserted into the document 2. Query the document 3. Update the document 4. Delete the document   1.  Inserted into the document # -*- coding: UTF-8 -*- import datetime from pymongo im ...

  10. Kafka Learning notes -Java Simple operation

    Maven Dependency package : <dependency> <groupId>org.apache.kafka</groupId> <artifactId>kafka ...

Random recommendation

  1. Use Spring Of Validator Interface verification

    You can use Spring Provided validator Interface for object verification .Validator Interface and Errors Working together , stay Spring When doing the calibration , It will summarize all validation errors to Errors Go to the object . Look at this Jane ...

  2. Charge -ios( Unfinished update ...

    [reference]http://www.cocoachina.com/ios/20160323/15770.html OC Understanding and characteristics of OC As an object-oriented language , Nature has the characteristics of object-oriented language : encapsulation ...

  3. 『 practice 』Yalmip modeling +Cplex Class solving

    Yalmip modeling +Cplex Class solving One . reason Yalmip You can only set part of Cplex Parameters of , So you need to call Cplex class . and optimize yes Yalmip Common functions provided , But the return result parameters of this function are limited . chart 1 ...

  4. Js Medium subStr and subString The difference between

    /** * Created by lonecloud on 16/9/8. */ var str="HelloWorld"; /** * Here are two parameters. The first one is from 0 At which point, the second ...

  5. [BZOJ1010] [HNOI2008] The toy box toy ( Slope optimization )

    Description P The professor is going to the Olympics , But he could not part with his toys , So he decided to ship all the toys to Beijing . He USES his own compressor to compress , It can turn any item into a pile , I'm going to put it in a special one-dimensional container .P The professor is numbered as 1... ...

  6. UVA12186

    Give a tree diagram , There is only one boss in the company whose number is 0, Other people from 1 Numbered starting . In addition to the boss , Everyone has a direct superior , No subordinate employee becomes a worker . The workers want to write a petition for a pay increase , Only when not less than all subordinates of the employee T% Please ...

  7. How hard it is to balance ? To talk about MMO My career design

    from :http://www.gameres.com/804893.html First of all, we need to define a concept : Balance is not about YY Make adjustments based on good career design , But from the beginning of game design to create a set of standards . Adjustable game design framework . ...

  8. SP4487 GSS6 - Can you answer these queries VI

    The main idea of the topic Given a N A sequence of integers A, You need to apply M Operations : I p x stay  p  Insert an element at  x D p Delete  p An element at R p x modify  p The value of the element at is  x Q l r Query one ...

  9. linux Log timing polling process details (logrotate)

    logrotate Introduce about Linux System security , Log files are extremely important tools . The log file contains useful information about the events occurring in the system , It is often used in troubleshooting or system performance analysis . As log files grow , It needs to be decided ...

  10. Builder pattern (build pattern)------- Creative model

    Separating the construction of a complex object from its label , So that the same build process can create different tags . The builder model is a more complex creation model , It will connect the client with multiple components ( Or parts ) The process of creating complex objects is separated , The client doesn't need to know the inside of a complex object ...