This article takes Twitter Storm   official Wiki   Based on , Describe in detail how to quickly build a Storm colony , among , Problems encountered in project practice and experience summary , In the corresponding chapters, write “ matters needing attention ” Given in the form of .  


3.1 Storm Cluster components

Storm There are two types of nodes in the cluster : Master node (Master Node) And work nodes (Work Node). Their respective roles are as follows :

1. Master node (Master Node) Running a program called Nimbus The background program of , It is responsible for the Storm Distribute code within the cluster , Assign tasks to work machines , And is responsible for monitoring the running state of the cluster .Nimbus Is similar to Hadoop in JobTracker Role .

2. Every work node (Work Node) Running a program called Supervisor The background program of .Supervisor Responsible for monitoring from Nimbus The tasks assigned to it , Start or stop the work process of executing the task . Each working process executes a Topology Subset ; A running Topology It is composed of multiple work processes distributed on different work nodes .

 

Storm Cluster components

Nimbus and Supervisor All the coordination between nodes is through Zookeeper Cluster to achieve . Besides ,Nimbus and Supervisor Processes are fast failures (fail-fast) And stateless (stateless) Of ;Storm All the states of the cluster are either Zookeeper In the cluster , Or on a local disk . It means you can use kill -9 To kill Nimbus and Supervisor process , They can continue to work after restart . This design makes Storm Clusters have incredible stability .

3.2 install Storm colony

This chapter will describe in detail how to build a Storm colony . Here are the installation steps to be completed in turn :

1. build Zookeeper colony ;

2. install Storm Dependency Library ;

3. Download and unzip Storm The release ;

4. modify storm.yaml The configuration file ;

5. start-up Storm Each background process .

3.2.1 build Zookeeper colony

Storm Use Zookeeper Coordinating clusters , because Zookeeper Not for messaging , therefore Storm to Zookeeper The pressure is quite low . Most of the time , Of a single node Zookeeper Clusters are competent enough , But to ensure recovery or deployment on a large scale Storm colony , You may need a larger number of nodes Zookeeper colony ( about Zookeeper Word of the cluster , The minimum number of nodes officially recommended is 3 individual ). stay Zookeeper Complete the following installation and deployment steps on each machine in the cluster :

1. Download and install Java JDK, The official download link is http://java.sun.com/javase/downloads/index.jsp,JDK Version is JDK 6 Or more .

2. according to Zookeeper The load of the cluster , Setting up reasonable Java Heap size , Try to avoid swap, Lead to Zookeeper Performance degradation . Be conservative ,4GB Memory machines can be Zookeeper Distribute 3GB Maximum heap space .

3. Download and install Zookeeper package , The official download link is http://hadoop.apache.org/zookeeper/releases.html.

4. according to Zookeeper Cluster nodes , stay conf Create under directory Zookeeper The configuration file zoo.cfg:

tickTime=2000
dataDir=/var/zookeeper/
clientPort=2181
initLimit=5
syncLimit=2
server.1=zoo1:2888:3888
server.2=zoo2:2888:3888
server.3=zoo3:2888:3888

among ,dataDir Appoint Zookeeper Data file directory of ; among server.id=host:port:port,id It's for everyone Zookeeper Node number , Save in dataDir In the catalog myid In file ,zoo1~zoo3 Denote each Zookeeper Node hostname, first port It's used to connect leader The port of , the second port Is used for leader The election port .

5. stay dataDir Create under directory myid file , The file contains only one line , And the content is the corresponding server.id Medium id Number .

6. start-up Zookeeper service :

java -cp zookeeper.jar:lib/log4j-1.2.15.jar:conf \ org.apache.zookeeper.server.quorum.QuorumPeerMain zoo.cfg

perhaps

bin/zkServer.sh start

7. adopt Zookeeper The client tests whether the service is available :

java -cp zookeeper.jar:src/java/lib/log4j-1.2.15.jar:conf:src/java/lib/jline-0.9.94.jar \ org.apache.zookeeper.ZooKeeperMain -server 127.0.0.1:2181

perhaps

bin/zkCli.sh -server 127.0.0.1:2181

matters needing attention :

  1. because Zookeeper It's a quick failure (fail-fast) Of , And any error situation , All processes exit , therefore , It's better to use the monitoring program to Zookeeper Manage , Guarantee Zookeeper It can be automatically restarted after exiting . For details, please refer to .
  2. Zookeeper In the process of operation, there will be dataDir A lot of log and snapshot files are generated under the directory , and Zookeeper The running process is not responsible for periodically cleaning up and merging these files , It takes up a lot of disk space , therefore , Need to pass through cron Clean up useless log and snapshot files regularly . For details, please refer to . The specific command format is as follows :java -cp zookeeper.jar:log4j.jar:conf org.apache.zookeeper.server.PurgeTxnLog <dataDir> <snapDir> -n <count>

3.2.2 install Storm Dependency Library

Next , Need to be in Nimbus and Supervisor Machine mounted Storm The dependent libraries , As follows :

1. ZeroMQ 2.1.7 – Do not use 2.1.10 edition , Because some of the serious aspects of this version bug It can lead to Storm There's a strange problem with the cluster running . A few users are 2.1.7 Version will encounter ”IllegalArgumentException” It's abnormal , Now it's down to 2.1.4 Version can fix this .

2. JZMQ

3. Java 6

4. Python 2.6.6

5. unzip

The above version of the dependency library is Storm The test of ,Storm There is no guarantee that in other versions of Java or Python Library can run .

3.2.2.1 install ZMQ 2.1.7

Download, compile and install ZMQ:

wget http://download.zeromq.org/zeromq-2.1.7.tar.gz
tar -xzf zeromq-2.1.7.tar.gz
cd zeromq-2.1.7
./configure
make
sudo make install

matters needing attention :

 If the installation process reports an error uuid Can't find , Then install through the following package uuid library :
sudo yum install e2fsprogsl  -b current
sudo yum install e2fsprogs-devel  -b current

3.2.2.2 install JZMQ

Download, compile and install JZMQ:

git clone https://github.com/nathanmarz/jzmq.git
cd jzmq
./autogen.sh
./configure
make
sudo make install

In order to ensure JZMQ Normal work , You may need to complete the following configuration :

  1. Set correctly JAVA_HOME environment variable
  2. install Java Development kit
  3. upgrade autoconf
  4. If you are Mac OSX, Reference here

matters needing attention :

  1. If you run ./configure There's something wrong with the order , Reference here .

3.2.2.3 install Java 6

1. Download and install JDK 6, Reference here ;

2. To configure JAVA_HOME environment variable ;

3. function java、javac command , test java normal setup .

3.2.2.4 install Python2.6.6

1. download Python2.6.6:

wget http://www.python.org/ftp/python/2.6.6/Python-2.6.6.tar.bz2

2. Compilation and installation Python2.6.6:

tar –jxvf Python-2.6.6.tar.bz2
cd Python-2.6.6
./configure
make
make install

3. test Python2.6.6:

python -V
Python 2.6.6

3.2.2.5 install unzip

1. If you use RedHat series Linux System , Perform the following installation unzip:

apt-get install unzip

2. If you use Debian series Linux System , Perform the following installation unzip:

yum install unzip

3.2.3 Download and unzip Storm The release

next step , Need to be in Nimbus and Supervisor Machine mounted Storm Release version .

1. download Storm Release version , Recommended Storm0.8.1:

wget https://github.com/downloads/nathanmarz/storm/storm-0.8.1.zip

2. Unzip to the installation directory :

unzip storm-0.8.1.zip

3.2.4 modify storm.yaml The configuration file

Storm There is one in the unzip directory of the release version conf/storm.yaml file , Used for configuration Storm. The default configuration can be viewed here .conf/storm.yaml The configuration options in will override defaults.yaml Default configuration in . The following configuration options are required in conf/storm.yaml Configured in :

1) storm.zookeeper.servers: Storm The cluster uses Zookeeper The cluster address , The format is as follows :

storm.zookeeper.servers:
- “111.222.333.444″
- “555.666.777.888″

If Zookeeper The cluster is not using the default port , Then it needs to be storm.zookeeper.port Options .

2) storm.local.dir: Nimbus and Supervisor Processes are used to store a small amount of state , Such as jars、confs And so on , You need to create the directory in advance and give it enough access . And then in storm.yaml Configure the directory in , Such as :

storm.local.dir: "/home/admin/storm/workdir"

3) java.library.path: Storm Local libraries used (ZMQ and JZMQ) Load path , The default is ”/usr/local/lib:/opt/local/lib:/usr/lib”, Generally speaking ZMQ and JZMQ Default installed in /usr/local/lib Next , So it doesn't need to be configured .

4) nimbus.host: Storm colony Nimbus Machine address , each Supervisor The work node needs to know which machine is Nimbus, To download Topologies Of jars、confs Wait for the documents , Such as :

nimbus.host: "111.222.333.444"

5) supervisor.slots.ports: For each Supervisor Work node , You need to configure what the work node can run worker Number . Every worker Take up a separate port to receive messages , This configuration option is used to define which ports can be worker The use of . By default , Each node can run 4 individual workers, Respectively in 6700、6701、6702 and 6703 port , Such as :

supervisor.slots.ports:
    - 6700
    - 6701
    - 6702
    - 6703

3.2.5 start-up Storm Each background process

The last step , start-up Storm All background processes of . and Zookeeper equally ,Storm It's also a quick failure (fail-fast) The system of , such Storm Can be stopped at any time , And when the process is restarted, it is resumed correctly . That's why Storm Reasons for not saving state in process , Even if Nimbus or Supervisors Was restarted , Running Topologies Not affected .

Here is the start Storm The way of each background process :

  1. Nimbus: stay Storm Running on the master node ”bin/storm nimbus >/dev/null 2>&1 &” start-up Nimbus daemon , And put it in the background ;
  2. Supervisor: stay Storm Running on each work node ”bin/storm supervisor >/dev/null 2>&1 &” start-up Supervisor daemon , And put it in the background ;
  3. UI: stay Storm Running on the master node ”bin/storm ui >/dev/null 2>&1 &” start-up UI daemon , And put it in the background , After startup, it can pass through http://{nimbus host}:8080 Look at the cluster worker Resource usage 、Topologies The running state of the system .

matters needing attention :

  1. start-up Storm Background process , Need to be right conf/storm.yaml In the configuration file storm.local.dir Directory has write permission .
  2. Storm After the background process is started , Will be in Storm Install... In the deployment directory logs/ Generate the log file of each process under the subdirectory .
  3. After testing ,Storm UI It has to be with Storm Nimbus Deployed on the same machine , otherwise UI Not working properly , because UI The process checks to see if the machine exists Nimbus link .
  4. For ease of use , Can be bin/storm Add to system environment variables .

thus ,Storm The cluster has been deployed 、 configured , You can submit the topology to the cluster to run .

3.3 Submit task to cluster

1. start-up Storm Topology:

storm jar allmycode.jar org.me.MyTopology arg1 arg2 arg3

among ,allmycode.jar Is included Topology Implementation code jar package ,org.me.MyTopology Of main The method is Topology Entrance ,arg1、arg2 and arg3 by org.me.MyTopology The parameters that need to be passed in during execution .

2. stop it Storm Topology:

storm kill {toponame}

among ,{toponame} by Topology Submitted to the Storm Specified when clustering Topology The name of the task .

3.4 Reference material

1. https://github.com/nathanmarz/storm/wiki/Tutorial

2. https://github.com/nathanmarz/storm/wiki/Setting-up-a-Storm-cluster