Question

A MapReduce job usually splits the input dataset into independent chunks which are processed by the map tasks in a completely parallel manner. The framework sorts the outputs of the maps, which are then input to the reduce tasks. Determine the phases of MapReduce framework demonstration with an appropriate example.

Answer 1

Q) .A MapReduce job usually splits the input dataset into independent chunks which are processed by the map tasks in a completely parallel manner. The framework sorts the outputs of the maps, which are then input to the reduce tasks. Determine the phases of MapReduce framework demonstration with an appropriate example.

Ans: - MapReduce is a Java based program model for distributed computing. It is the data processing technique which always imposes to positioned the data and compute this data as immediate as possible or needed. A MapReduce job usually splits the input dataset into independent chunks which are processed by the map tasks in a completely parallel manner. The framework sorts the outputs of the maps, which are then input to the reduce tasks.

The MapReduce is an algorithm which have two main jobs. It consists of two functions Map and Reduce that is combinedly called MapReduce.

The main job of the Map function is to import a set of data and convert this set of data it into another set. The elements of set of data are individually divided into Key/value pairs.

Another part is reduce function. It basically takes the processed data output of the Map function in the form of input. The Key value pair produced by the Map function are combined into the smaller set of Key Value pair Tuples.

There are three phases of MapReduce Framework: -

MapReduce data processing program has three phases in which it gets executed.

These three stages are known as: -

1. map stage

2. shuffle stage

3. reduce stage.

Map stage –

This phase of Map Reducer function is used to process the data and generate Key/Value pairs. The data which is taken as input is a simple file or directory which is stored in Hadoop file.

The input file is imported by the Mapper function in line by line format. The mapper processes the data and produce the small chunks of data.

Reduce stage − Shuffle stage and the Reduce stage combinedly form the Reduce stage. It simply takes the input from the Map functionand store the data in the HDFS.

Shuffle Stage: - Input to the Reduce function is the sorted as output of the mapper function. In this phase the MapReduce framework fetches the relevant data partition of the output of all the mappers by using HTTP.