Hadoop

              Day by day, data had grown rapidly, so Google published a research paper on the distributed file systems in 2003; Mike caferell and the Doug cutting started working on this and invented hand up in 2005.
              Hadoop is an Apache open-source framework written in Java that allows the distributed processing of large data sets across clusters of computers using simple programming models. The Hadoop framework application work in an environment that provides distributed storage and computation across clusters of computers. Hadoop is designed to scale up from a single server to thousands of machines. Each offers local computation and storage. It is used for massive data storage and faster processing. Hadoop is said to be the new age of Big Data. When you Google Hadoop, you will get approximately 28 million results. One of the primary reasons for the popularity of Hadoop is the phenomenal growth of the cloud.

How does Hadoop work?

It is quite expensive to build bigger servers with heavy configurations that handle large-scale processing. Still, as an alternative, you can tie together many commodity computers with a single CPU as a single functional distributed system. Moreover, it is cheaper than one high-end server, so this is the first motivational factor behind Hadoop that runs across clustered and low-cost machines. Hadoop runs code across clusters of computers. Practically, the clustered machine reads the data sets in parallel and provides much higher throughput. This process includes the following task that Hadoop performs:

•Data is initially divided into directories and files. Files are divided into uniformly sized blocks of 128M and 64M.
•These files are then distributed across various cluster nodes for further processing.
•HDFS, being on top of the local file system, supervises the processing.
Blocks are replicated for handling hardware failure.
•Checking that the code was executed successfully.
•Performing the sort that takes place between the map and reduces stages.
•Sending the sorted data to a certain computer.
•Writing and debugging logs for each job

       Daemons are basically the process that runs in the background. There are a total of 5 daemons in Hadoop. These daemons run for Hadoop to make it functional. These are listed below:-

•Name the node: It stores the Metadata about the data stored in DataNodes, and it runs on a master node for HDFS. It contains information such as the blocks that made a file and the location of those blocks in the cluster.

•Data node: It stores the actual data, and it runs on a slave node for HDFS. It reports information about the blocks which periodically contain the Name node.

•Secondary Data node: It is the backup of the name node, and it only contains the entire Metadata node properties, address, and block report of each data node. It keeps a copy of the image, which can be used in case of failure of the name node.

•Resource manager: It runs on the YARN slave node for MapReduce

•Node manager: It runs on the YARN slave node for MapReduce

Advantages of Hadoop

•Hadoop framework allows the user to write and test distributed systems quickly. It is efficient, and it automatically distributes the data and work across the machines and, in turn, utilizes the underline parallelism of the CPU cores.
•Hadoop does not rely on hardware to provide fault tolerance and high availability; rather, the Hadoop library itself has been designed to detect and handle failures at the application layer.
•Server can be added or removed from the cluster dynamically, and Hadoop continuous to operate without interruption.
•Another big advantage of Hadoop is that apart from being open source, it is compatible on all the platforms since it is Java-based.

Real time applications of Hadoop:

•Prevent hardware failure.
•Understand what people think about your company.
•Understand when to sell products.
•Find your ideal prospects.