Question

please explain how the deletion algorithm for removing nodes works in a hadoop cluster, in python

Answer 1

First we have to know how a hadoop cluster works so in this cluster the data which is in HDFS(Hadoop Distributed File System) and the MapReduce system are uploaded on every machine present in the cluster by which if one machine got failed then the other one can provide the data where it is stored which speed up the information retrieval process. The HDFS got distributed ina similar manner. In this, the single name node tracks where data is stored called as Datanode. There are mainly two nodes in hadoop cluster which are namenode and datanode and for the deletion of these nodes the deletion algorithm is used.

Before deleting the node in hadoop first the user have to be ensure that the number of nodes in cluster must be greater than or equal to the other replication factor and ensure that the node which has to be removed is working then only the user can remove the node.

The deletion algorithm for deleting node in Hadoop has few steps which are-

• Decomission the Datanode role- The node which is to be removed has to be decomissioned first so that the process moves the data blocks to the other Datanode.
• Stop the datanode role.
• Verify the integrity of the Hadoop Distributed File System(HDFS)- In this the user has to check the problems in the HDFS system which are stopping the removal of node and then fix that error. The command to identify the problem is hdfs fsck/.
• After resolving the errors remove the role of Datanode and then manually remove the data directory of Datanode.

The algorithm is known as Heteroginity aware replica deletion(HaRD) which is -

The picture which depicts the process of deletion algorithm for removing node is stated above.