HDFS Components

As Hadoop is designed to address both the storage and processing problems of the huge volume of data, The Hadoop distributed file System manages the storage part of the Hadoop framework and is designed to work on commodity hardware.

HDFS is a distributed, scalable, and highly fault-tolerant file system written in java for the Hadoop framework. Generally, in a cluster, a group of data nodes forms HDFS.

Commodity Hardware: HDFS is designed to work well with commodity hardware. As Hadoop is used for handling the huge volume of data, to manage the cost of hardware requires the use of commodity hardware than high-end computer having the huge storage and computing power.

Batch Processing: Hadoop is designed to do batch processing rather than real-time or interactive usage.

Large Data Sets: HDFS is tuned for working with large datasets rather than small datasets. The large data set can vary from a few Terabytes to Petabytes.

HDFS mainly consists of 3 components.

1. NameNode
2. DataNode
3. Secondary NameNode

NameNode:

1. NameNode is otherwise called as MasterNode.
2. It manages HDFS storage.
3. NameNode keeps the metadata information about HDFS like block address, Directory, etc.
4. NameNode is not meant to store data apart from metadata information.
5. NameNode is a single point of failure for HDFS. If NameNode goes down, the entire file system is unavailable for use.
6. Generally, NameNode system is high in the configuration like high RAM and core.
7. NameNode continuously monitors the health of DataNodes through heartbeats.
8. To ensure High availability we can have both NameNode and standby NameNode.

DataNode:

1. DataNode is also called a ‘Worker’ Node.
2. DataNodes are responsible for storing actual data in terms of blocks.
3. Data is distributed across DataNodes.
4. DataNodes communicate to NameNode through the heartbeat.
5. DataNodes do not communicate with each other. They work independently.
6. As they work independently, the cluster doesn’t get affected if one of the DataNode goes down as long as the replication of data taken care of.

Note: You can create a Single Node cluster where NameNode and DataNode are installed in one system. In this scenario, we cannot expect high availability of NameNode or Data as all the services are in one system.

We will cover the above concepts in great detail in the later chapter.

Secondary NameNode:

1. Secondary NameNode is used to archive the high availability of NameNode.
2. As the NameNode is a single point of failure in HDFS, the file system is unavailable.
3. To overcome this problem Secondary NameNode is implemented and Secondary NameNode takes the periodic snapshot of NameNode.
4. Secondary NameNode Periodically copies ‘FsImage’ and ‘editslog’ file.
5. In the case of NameNode failure, the metadata can be recovered last saved checkpoint of Secondary NameNode.
6. But Secondary NameNode is not replaced as Primary NameNode in case of failure.
7. Generally, Secondary NameNode configuration is as high as Primary NameNode.