Question

Explain how the Map Reduce frameworks like Hadoop exploit locality to achieve scalable parallelism.

Answer 1

Map reduce is a general framework that allows us to write applications that can process large amounts of data running in parallel.This technique involves concept of distributed programming by using Java.However Hadoop can run MapReduce programs in various languages like ruby,python and c++.

The MapReduce model include data processing primitives called mappers and reducers.The main advantage of this is that it allows data processing over multiple computing nodes.

How MapReduce works?
1. It divides the work in 2 tasks-
a)Map Function- It breaks down a set of data into tuples(Value/Pairs)and convert it into another set of data.
b)Reduce Function-This task is always performed after the Map has completed its work. It take MAP's output as it's input and combines those tuples into smaller set of tuples.
2.The execution of MapReduce is controlled by following-
a)Jobtracker-Also known as Master
b)Multiple Task trackers- Also known as Slaves.
3.A single job is divided into multiple tasks running on multiple computer nodes.The job tracker role is to schedule the activities over various nodes and coordinate those activities.Successful execution of task is then ensured by Multiple task trackers.These task trackers then sends a progress report to the Job tracker after execution of each tasks and ensures overall progress of the job.If an event fails,the job tracker can reschedule a news task on a different task tracker.