Question

Explain how YARN provides flexible resource management for Hadoop cluster.

Explain how YARN extends Hadoop to enable multiple frameworks such as MapReduce, Giraph, Spark, and Flink.

(Based on the paper: Apache Hadoop YARN: Yet Another Resource Negotiator)

Answer 1

Answer:

Hadoop Yarn: a platform responsible for managing computing resources in clusters and using them for scheduling users' application.

Hadoop YARN Resource Manager

OBJECTIVE

In this Hadoop YARN Resource Manager

What is Yarn Resource Manager?

The Resource Manager is the core component of YARN – Yet Another Resource Negotiator.

• Hadoop YARN is designed to provide a generic and flexible framework to administer the computing resources in the Hadoop cluster.
• the YARN Resource Manager Service (RM) is the central controlling authority for resource management and makes allocation decisions ResourceManager has two main components:

1. Scheduler
2. Applications Manager.

• The Scheduler API is specifically designed to negotiate resources and not schedule tasks.
• The scheduler does not perform monitoring or tracking of status for the Applications.
• The Scheduler performs its scheduling function based the resource requirements of the applications resource which incorporates elements such as memory, CPU, disk, network etc.
• Hadoop Yarn Resource Manager does not guarantee about restarting failed tasks either due to application failure or hardware failures.
• Applications can request resources at different layers of the cluster topology such as nodes, racks etc.
• Hence, the scheduler determines how much and where to allocate based on resource availability and the configured sharing policy.
• The Scheduler has a pluggable policy plug-in, which is responsible for partitioning the cluster resources among the various queues, applications etc.

Hadoop YARN Resource Manager Components

RM works together with the per-node NodeManagers (NMs) and the per-application ApplicationMasters (AMs).

a) ClientService

The client interface to the Resource Manager. This component handles all the RPC interfaces to the RM from the clients including operations like application submission, application termination, obtaining queue information, cluster statistics etc.

b) AdminService

To make sure that admin requests don’t get starved due to the normal users’ requests and to give the operators’ commands the higher priority, all the admin operations like refreshing node-list, the queues’ configuration etc. are served via this separate interface.

I tried my best to answer. Sorry for the inconvenience. If u have any doubts please mention in comments.

Please upload the questions separately. Love to answer. Give me a upvote.