Hadoop 2.6.0 Multi-Node cluster setup

I have assumed that 10.0.1.1 is my Master Node and 10.0.1.2 and 10.0.1.3 are my slaves.Now ultimately we need to assign following roles.

10.0.1.1 NameNode, DataNode  
10.0.1.2 DataNode  
10.0.1.3 DataNode

First Master node : 10.0.1.1


$ cd $HADOOP_HOME
$ mkdir -p hdfs-data
$ mkdir -p hdfs-site
$ cd etc/hadoop
$ sudo gedit hdfs-site.xml

Replace it with following lines


<configuration>
<property>
        <name>dfs.datanode.data.dir</name>
        <value>file:///usr/local/hadoop/hdfs-data</value>
        <description>DataNode directory</description>
    </property>

    <property>
        <name>dfs.namenode.name.dir</name>
        <value>file:///usr/local/hadoop/hdfs-site</value>
        <description>NameNode directory for namespace and transaction logs storage.</description>
    </property>

    <property>
        <name>dfs.replication</name>
        <value>2</value>
    </property>
    <property>
        <name>dfs.permissions</name>
        <value>false</value>
    </property>
    <property>
        <name>dfs.datanode.use.datanode.hostname</name>
        <value>false</value>
    </property>
    <property>
        <name>dfs.namenode.datanode.registration.ip-hostname-check</name>
        <value>false</value>
    </property>
</configuration>

Open core-site.xml


$ sudo gedit core-site.xml

Add Following lines


<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://10.0.1.1/</value>
        <description>NameNode URI</description>
    </property>
</configuration>

Now format the namenode


$ cd $HADOOP_HOME
$ ./bin/hadoop namenode -format

Add slaves into slaves file


sudo gedit $HADOOP_HOME/etc/hadoop/slaves 

and put:

10.0.1.1  
10.0.1.2  
10.0.1.3  

Now let’s configure slaves


$ cd $HADOOP_HOME
$ mkdir -p hdfs-data 
$ cd etc/hadoop

Open hdfs-site.xml


$ sudo gedit hdfs-site.xml

Replace with following

 <configuration>  
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>file:///usr/local/hadoop/hdfs-data</value>
        <description>DataNode directory</description>
    </property>

    <property>
        <name>dfs.replication</name>
        <value>2</value>
    </property>
    <property>
        <name>dfs.permissions</name>
        <value>false</value>
    </property>
    <property>
        <name>dfs.datanode.use.datanode.hostname</name>
        <value>false</value>
    </property>    
</configuration>

Open core-site.xml


$ sudo gedit core-site.xml

Add Following lines

<configuration> 
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://10.0.1.1/</value>
        <description>NameNode URI</description>
    </property>
</configuration>

Now go to Master 10.0.1.1 and

$ cd $HADOOP_HOME
$ ./sbin/start-dfs.sh  
$ ./sbin/start-yarn.sh  

And check it by


$ ./bin/hadoop dfsadmin -report  
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s