Question

For each problem, provide test data (as input files), MapReduce programs, and running results (screen shoots) on Hadoop.

i) Write a program to read input file (an integer number per line) and remove duplicated numbers.

ii) When data are transmitted from map to reduce, <key, value> will be automatically sorted in an ascending order. Write a program that can read input file (an integer number per line) and write them out in a descending order.

Answer 1

i)

Requirement::

Suppose your data file which are having user's basic information like first name, last name, designation, city etc. and seperated by ',' delimiter. Now the requirement is to reduce duplicate values using map reduce.

Components Involved::

• HDFS: Used to store input data which will get pass to the map reduce as input and store the map reduce output.
• Map Reduce: Used to process the data/file available in HDFS.

Sample Data::

Step 1: Input Data Preparation::

Let's keep the sample data into a file and keep it at the local path. My file name is "sampledataForDuplicate" and local path is "/home/NN/HadoopRepo/MapReduce/resources/duplicateValue".

We need to move this data file to HDFS location which will be input path for the map reduce.

It will copy the data file from local to HDFS location.

Step 2: Create maven Project::

Follow the below steps to create maven project:

• Open eclipse
• Create a maven project

Step 3: Resolve Dependency::

Add dependecy in pom.XML file and resolve dependency using cmd:

Step 4: Write mapper::

Once you are done with all above steps, write a mapper calss which will take an input file. It will read the file and store each word of file with key-value pair. Here using a java program to write the mapper.

Step 5: Write Reducer::

In this step, we will take the mapper data as an input and process it. The actual logic has been written here.

Step 6:

In order to execute the mapper and reducer, let's create a driver class which will call mapper and reducer. Find the below driver class code::

Continution Screenshot

Step 7: Package Preparation::

In this step, We will create a package(.jar) of the project, Follow below steps::

• Open CMD
• Head to maven project
• Use command:

mvn package

It will create a .jar file under target directory. Copy this created jar and keep it at the local path. Here in my case jar file is available at "/home/NN/HadoopRepo/MapReduce".

Step 8: Execution::

Command for execution::

hadoop jar <path of jar> <driver class with package name> <input data path of HDFS> <output path at HDFS>

Step 9:: Validate Output::

Check the output at HDFS path

Check the output