In this tutorial, I will explain you setting up Hadoop single node cluster setup on Ubuntu 14.04. Single node cluster will sit on the top of Hadoop Distributed File System (HDFS).

Hadoop single node cluster setup on Ubuntu 14.04

Oct 21, 2008 Hadoop native library is written in ANSI C and built using the GNU autotools-chain (autoconf, autoheader, automake, autoscan, libtool). This means it should be straight-forward to build them on any platform with a standards compliant C compiler and the GNU autotools-chain.

Hadoop is a Java framework for running application on the large cluster made up of commodity hardware’s. Hadoop framework allows us to run MapReduce programs on file system stored in highly fault-tolerant Hadoop distributed file systems.

Related Readings:How to Learn Apache Hadoop
Also:7 Best Books to Learn Bigdata Hadoop

The main objective of this post is to get start with Hadoop single node cluster setup.This tutorial is been tested with below OS and Hadoop software versions;

On all the above platforms a 32/64 bit Hadoop native library will work with a respective 32/64 bit jvm. Building Native Hadoop Libraries Hadoop native library is written in ANSI C and built using the GNU autotools-chain (autoconf, autoheader, automake, autoscan, libtool). This means it should be straight-forward.
When running a RevoScaleR job in Hadoop, the mapreduce console output may contain the message:WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform. Using builtin-java classes where applicable. This warning is unrelated to RevoScaleR operation. It is usually a result of running in a 64-bit Linux environment while.

Hadoop single node cluster setup – Presteps

Install Ubuntu 14.04 on your system:

First, you should have system with Ubuntu 14.04 installed on it. Follow below steps;

Download Ubuntu 14.04 from official website, make bootable CD or drive and install or
Download and install Oracle/VMware virtual machines then install Ubuntu 14.04 on that virtual machine (easy way)

Install Java

Remember! Hadoop framework is written in Java. It requires a working Java 1.6+ installation. Here apt-get is used to install Java

Create Hadoop Group and add dedicated User

In this step, we will create new group named Hadoop and add dedicated user to it. That user will be the cluster admin and use that user for running any Hadoop applications. We will use dedicated user to install all the Hadoop related software installations.

Installing ssh

Hadoop framework requires ssh access to manages all its nodes. Even for our single node cluster we need to install and set up the ssh access to ‘localhost ‘ for our dedicated ‘hduser’

ssh uses two main components;

ssh : The command we use to connect to remote machines – The client
sshd : The daemon that is running on the server and allows clients to connect to the server

By default, ssh is pre-enabled on Linux, but in order to start sshd daemon, we need to install ssh first.

Use this command to do that:

Hadoop Native Library Linux

The above command will install ssh on our machine. If we get something similar to the following, you can think setup done properly:

Create and Setup SSH Certificate

As mentioned in previous section, Hadoop requires ssh access to manage its nodes. We have already installed ssh, now we need to configure it to allow SSH public key authentication.

Hadoop uses ssh to access its node, which would normally require the user to enter a password. However, this requirement can be eliminated by creating and setting up SSH certificates using the following commands. If asked for a filename just leave it blank and press the enter key to continue.

The above command adds the newly created key to the list of authorized keys so that Hadoop can use ssh without prompting for a password.

We can check to see if ssh works:

Installing Hadoop

Download latest stable hadoop version and copy to Ubuntu machine or download it directly in host machine

Add hduser to sudo group:

Login to hduser and you may want to move the Hadoop installation to the /usr/local/hadoop directory using the following command

Setup Configuration Files

The following files will have to be modified to complete the Hadoop setup:

~/.bashrc
/usr/local/hadoop/etc/hadoop/hadoop-env.sh
/usr/local/hadoop/etc/hadoop/core-site.xml
/usr/local/hadoop/etc/hadoop/mapred-site.xml.template
/usr/local/hadoop/etc/hadoop/hdfs-site.xml
/usr/local/hadoop/etc/hadoop/yarn-site.xml

~/.bashrc

Before editing the .bashrc file in our hduser home directory, we need to find the path where Java has been installed to set the JAVA_HOME environment variable

Use following command:

Now we can append the Java path to JAVA_HOME of ~/.bashrc:

Note: Be careful while copying commands to terminal. It may introduce the junk character or special characters. Change the command in case if you get error. e.g. double quotes (“”) may differ. Bog page may display double hyphen – as single hyphen. Make sure commands are correct before running.

Note that the JAVA_HOME should be set as the path just before the ‘…/bin/’: i.e. set only /usr/lib/jvm/java-7-openjdk-amd64 path

Now quick check if the javac is working properly:

/usr/local/hadoop/etc/hadoop/hadoop-env.sh

We need to set JAVA_HOME by modifying hadoop-env.sh file.

Adding the above statement in the hadoop-env.sh file ensures that the value of JAVA_HOME variable will be available to Hadoop whenever it is running.

/usr/local/hadoop/etc/hadoop/core-site.xml

The /usr/local/hadoop/etc/hadoop/core-site.xml file contains configuration properties that Hadoop uses when starting.
This file can be used to override the default settings that Hadoop starts with.

Open the file and enter the following in between the <configuration></configuration> tag:

/usr/local/hadoop/etc/hadoop/mapred-site.xml

By default, the /usr/local/hadoop/etc/hadoop/ folder contains /usr/local/hadoop/etc/hadoop/mapred-site.xml.template file which has to be renamed to mapred-site.xml:

The mapred-site.xml file is used to specify which framework is being used for MapReduce.

We need to enter the following content in between the <configuration></configuration> tag:

/usr/local/hadoop/etc/hadoop/hdfs-site.xml

The /usr/local/hadoop/etc/hadoop/hdfs-site.xml file needs to be configured for each host in the cluster being used. It is used to specify the directories which will be used as the namenode and the datanode on that host.

Before editing this file, we need to create two directories which will contain the namenode and the datanode for this Hadoop installation.

This can be done using the following commands:

Open the file and enter the following content in between the <configuration></configuration> tag:

/usr/local/hadoop/etc/hadoop/yarn-site.xml

Open the file and enter the following content in between the <configuration></configuration> tag:

<!– Site specific YARN configuration properties –>

<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>localhost:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>localhost:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>localhost:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>localhost:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.adress</name>
<value>localhost:8088</value>
</property>
</configuration>

Format the New Hadoop Filesystem

We have modified all files, next step, the Hadoop file system needs to be formatted so that we can start to using it. The format command should be issued with write permissi