五节点HadoopHA安装教程：

Master1 namenode,resourcemanager,nodemanager,datanode,journalnode, DFSZKFailoverController

Master2 namenode,resourcemanager,nodemanager,datanode,journalnode, DFSZKFailoverController

Slave1 nodemanager,datenode,journalnode, QuorumPeerMain

Slave2 nodemanager,datenode,journalnode, QuorumPeerMain

Slave3 nodemanager,datenode,journalnode, QuorumPeerMain

安装jdk
配置环境变量

JAVA

export JAVA_HOME=/home/zhouwang/jdk1.8.0_151

export PATH=$JAVA_HOME/bin:$PATH

export JRE_HOME=$JAVA_HOME/jre

export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH

2.配置个节点的/etc/hosts文件

192.168.71.128 master1

192.168.71.132 master2

192.168.71.129 slave1

192.168.71.130 slave2

192.168.71.131 slave3

3.配置SSH免密登录

Ssh-keygen -t rsa

Cat id_rsa.pub >> authorized_keys

Chmod 644 authorized_keys

Scp ~/.ssh/id_rsa.pub authorized_keys zhouwang@master:~/.ssh

Authorized_keys的权限必须是640

4.安装zookeeper

重命名conf文件夹下的zoo.example.cfg为zoo.cfg。新修改内容

clientPort=2181

dataDir=/home/zhouwang/zookeeper/data

dataLogDir=/home/zhouwang/zookeeper/log

server.0=master1:2888:3888

server.1=master2:2888:3888

server.2=slave1:2888:3888

server.3=slave2:2888:3888

server.4=slave3:2888:3888

在zookeeper文件夹下面创建相应的data和log文件夹，还有myid文件

Mkdir data

Mkdir log

Vim myid 输入1，每个节点的myid文件的值与server.x对应上

配置环境变量如下：

ZOOKEEPER

export ZOOKEEPER_HOME=/home/zhouwang/zookeeper

export PATH=$PATH:$ZOOKEEPER_HOME/bin

安装Hadoop
修改etc/hadoop下的四个配置文件

（1） core-site.xml


         
     
      fs.defaultFS
          
     
      hdfs://master/


         
     
      hadoop.tmp.dir
          
     
      /home/zhouwang/hadoop/tmp


         
     
      ha.zookeeper.quorum
          
     
      master1:2181,master2:2181,slave1:2181,slave2:2181,slave3:2181

（2） hdfs_site.xml


         
     
      dfs.namenode.name.dir
          
     
      /home/zhouwang/hadoop/dfs/name
      
     
         
     
      dfs.datanode.data.dir
          
     
      /home/zhouwang/hadoop/dfs/data
      
     
         
     
      dfs.replication
          
     
      3
      
     
     
     
         
     
      dfs.nameservices
          
     
      master
      
     
     
         
     
      dfs.ha.namenodes.master
          
     
      nn1,nn2
      
     
     
         
     
      dfs.namenode.rpc-address.master.nn1
          
     
      master1:9000
      
     
         
     
      dfs.namenode.rpc-address.master.nn2
          
     
      master2:9000
      
     
     
         
     
      dfs.namenode.http-address.master.nn1
          
     
      master1:50070
      
     
         
     
      dfs.namenode.http-address.master.nn2
          
     
      master2:50070
      
     
     
     
         
     
      dfs.journalnode.http-address
          
     
      0.0.0.0:8480
      
     
         
     
      dfs.journalnode.rpc-address
          
     
      0.0.0.0:8485
      
     
         
          
     
      dfs.namenode.shared.edits.dir
          
     
      qjournal://master1:8485;master2:8485;slave1:8485;slave2:8485;slave3:8485/master
      
     
         
          
     
      dfs.journalnode.edits.dir
          
     
      /home/zhouwang/hadoop/dfs/journal
      
     
         
          
     
      dfs.client.failover.proxy.provider.master
          
     
      org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
      
     
     
     
         
     
      dfs.ha.fencing.methods
          
     
      sshfence            shell(/bin/true)
      
     
     
         
     
      dfs.ha.fencing.ssh.private-key-files
          
     
      /home/zhouwang/.ssh/id_rsa
      
     
     
         
     
      dfs.ha.fencing.ssh.connect-timeout
          
     
      30000
      
     
     
         
     
      dfs.ha.automatic-failover.enabled
          
     
      true

(3) mapred-site.xml


    
     mapreduce.framework.name
     
    
     yarn


    
     mapreduce.jobhistory.address
         
    
     master1:10020


    
     mapreduce.jobhistory.webapp.address
         
    
     master1:19888

(4) yarn-site.xml


     
         
     
      yarn.nodemanager.aux-services
          
     
      mapreduce_shuffle
      
     
         
     
      yarn.resourcemanager.connect.retry-interval.ms
          
     
      2000
      
     
         
     
      yarn.resourcemanager.ha.enabled
          
     
      true
      
     
     
         
     
      yarn.resourcemanager.cluster-id
          
     
      cluster
      
     
     
         
     
      yarn.resourcemanager.ha.rm-ids
          
     
      rm1,rm2
      
     
     
         
     
      yarn.resourcemanager.hostname.rm1
          
     
      master1
      
     
     
         
     
      yarn.resourcemanager.hostname.rm2
          
     
      master2
      
     
     
         
     
      yarn.resourcemanager.ha.automatic-failover.enabled
          
     
      true
      
     
     
     
     
      yarn.resourcemanager.recovery.enabled
           
     
      true
       
     
     
         
     
      yarn.resourcemanager.store.class
          
     
      org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore
      
     
     
         
     
      yarn.resourcemanager.zk-address
          
     
      master1:2181,master2:2181,slave1:2181,slave2:2181,slave3:2181
      
     
     
         
     
      yarn.resourcemanager.scheduler.address.rm1
          
     
      master1:8030
      
     
         
     
      yarn.resourcemanager.scheduler.address.rm2
          
     
      master2:8030
      
     
     
         
     
      yarn.resourcemanager.resource-tracker.address.rm1
          
     
      master1:8031
      
     
         
     
      yarn.resourcemanager.resource-tracker.address.rm2
          
     
      master2:8031
      
     
     
         
     
      yarn.resourcemanager.address.rm1
          
     
      master1:8032
      
     
         
     
      yarn.resourcemanager.address.rm2
          
     
      master2:8032
      
     
     
         
     
      yarn.resourcemanager.admin.address.rm1
          
     
      master1:8033
      
     
         
     
      yarn.resourcemanager.admin.address.rm2
          
     
      master2:8033
      
     
     
         
     
      yarn.resourcemanager.webapp.address.rm1
          
     
      master1:8088
      
     
         
     
      yarn.resourcemanager.webapp.address.rm2
          
     
      master2:8088

(5) slaves

localhost

192.168.71.129 slave1

192.168.71.130 slave2

192.168.71.131 slave3

192.168.71.128 master1

192.168.71.132 master2

设置配置文件：

HADOOP

export HADOOP_HOME=/home/zhouwang/hadoop

export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

export YARN_CONF_DIR=$HADOOP_HOME/etc/Hadoop

将hadoop文件夹分发到各个节点上去：

Scp -r Hadoop zhouwang@XXX:~/Hadoop

第一次启动集群
配置了zookeeper的节点启动zkServer.sh服务

zkServer.sh start

然后查看zookeeper状态，zkServer.sh status，显式为follower或者leader则说明启动成功

然后，数据节点启动journalnode服务，hadoop-deamen.sh start journalnode

在master1格式化namenode: Hadoop namenode -format

然后启动namenode服务： Hadoop-deamon.sh start namenode

在master2同步master1的元数据： HDFS namenode -bootstrapStandby

然后再master2上启动namenode服务： Hadoop-deamon.sh start namenode

在master上格式化ZKFC

Hdfs zkfc -formatZK

在master1和master2上执行hadoop-deamon.sh start zkfc 启动DFSZKFailoverController 服务#ZKFC用于监控NameNode active和standby节点状态

在master1上启动hadoop-deamons.sh start datanode 启动所有数据节点上的datanode服务。

在master1上执行start-yarn.xml启动yarn服务。

至此安装完成。

7．第一次停止集群

先停止hdfs：stop-dfs.sh

在停止yarn：stop-yarn.sh

再次启动或者停止就可以执行start-all.sh 和stop-all.sh了。

8．Hdfs操作命令

Hadoop dfsadmin -report #查看datanode的节点信息

Hdfs haadmin -getServiceState nn1 查看namenode的状态

hdfs haadmin -transitionToActive/transitionToStandby -forcemanual nn1 强制切换节点的active和是standby状态。