Apache Hadoop software library is a framework that enables distributed processing of large data sets across clusters of computers. Hadoop can handle a single server and it can handle multiple machines. This means that scalability is not a problem. It is open source software that is reliable scalable and allows distributed computing. It is being used across enterprises in the storage of big data analytics and it also enables top managers in making business decisions.
Background
There has been a great need by the organizations to store and manage large amount of data referred as Big Data. There is a need to have the storage capacities that are able to handle such needs and at the same time facilitate analysis tools. The analytic tools will enable an organization to have insight in order to make informed decisions. Hadoop is innovative software that aims at solving such problems. It gives the organization the ability to analyze big data. It is an engine that is designed to reduce the time to come up with valuable insight from the dataset of a business. Hadoop has two main components; the Hadoop distributed file system (HDFS) and the Map Reduce. MapReduce is referred to as the process because it runs jobs concurrently in order to search a large dataset at a minimum time. HDFS is the location where output and input files are stored. This is the files that are necessary for the processing of tasks. It is the work of MapReduce to process tasks while HDFS stores the input and output files.
Scale-out and the use of intelligent software in running the cluster are the principles of the Hadoop. With such principles it is possible to achieve cost effective operations that provide valuable information to the organization. The two components with their capabilities have enabled Hadoop software one of the best analyzer of the Big Data.
Hadoop MapReduce
This is the component of Hadoop software. It offers the distributed framework for running of the jobs concurrently on a bigger dataset across a cluster of compute nodes. It is a software platform that allows users to write applications that process large chunks of data at the same time. The following are the components of MapReduce. JobTracker is the single master per cluster of those nodes that do the tasks of monitoring, scheduling and managing of the jobs and the component tasks. MapReduce uses queries to send out multiple tasks which are divided into clusters and they are all processed at the same time.
User data are stored in clusters in several nodes in the hadoop with the help of DHFS. Users are able to easily manipulate data through different tasks such as delete, create, rename and move because of the hierarchical organization structure. There is a framework presented by DHFS system to enable streaming of tasks directly from MapReduce framework. NameNode and DataNote are the components of the DHFS. NameNode is where every file and its location are stored in the in-memory maps. DataNode is where the read and write requests are process. There is a single slave DataNode per cluster node.
There are four hadoop distributions namely; Apache Hadoop, Greenplum HD, Cloudera CDH3 and Hortonworks Data Platform.
Hadoop Architecture
The hadoop ecosystems are the top software application components that give additional features to hadoop ( Dhruba). They are also essential for analytics workflow and they include; HBase which is a storage system that is found right on top of HDFS, Hive this is SQL-like interface for querying with an interface where data is stored in the HDFS, Pig is another add-o of the hadoop it is a high level language flow, Mahout is a machine that runs algorithm obtained from hadoop and R (RHIPE). The ecosystem components of hadoop are illustrated below.
The Hadoop architecture is shown in the diagram below.
Hadoop architecture (Dhruba)
The components above on the Hadoop architecture run on a hadoop compute cluster. The whole Hadoop becomes unusable if NameNode fails or has issues. This is because they are all in the same level and environment with Hadoop environment. For the above architecture to work, the data must be feed into the HDFS layer. The MapReduce is where analysis and computations are carried out. The overall Hadoop workflow is accommodated by the provision of the HDFS layer where all results are stored.
The MapReduce Algorithm
The algorithms for MapReduce include the sorting, searching, TF-IDE, BFS PageRank among many other advanced algorithms. The jobs of Map Reduce are very short in terms of the codes where IdentityReducer is a common aspect (Mutharaju, Raghava, Maier, and Hitzler). The utility jobs can also be composed. The data flow is more represented y Map Reduce jobs than a procedure.
The sort algorithms has the following inputs; a set of files which has one value per line, file name and line number are the mapper key and the mapper value is the contents of the line. The sort algorithm takes advantage of the reducer properties where pairs are processed in an order which is based on the key. The reducer properties where the (key, value) are processed as a pair. The role of the mapper is to identify the function value (k, v) (V, ___ ). The work of the reducer is to identify the function key (K’,___) -> (k’, “”). The trick is where the pairs (key, value) taken from the mapper are sent to the specific reducer based on hash key. The hash function must be selected where K1 < K2 => hash (K2). Sort algorithm is used to test the raw speed of Hadoop and it is important for Input/output drag race aspect (Mutharaju, Raghava, Maier, and Hitzler).
The search algorithm has a set of files containing lines of text and the search pattern to find as the inputs. The mapper key is file name and line number while the mapper value is the contents of the line. The special parameters in the search are the search patterns. The mapper functions in the following manner to do the search. Given filename and some test and the pattern, the output is given if the text matches the pattern output where the file name is given. The reducer identifies the function. Search file optimizes its search by marking the file once after it has been found. It is the work of the combiner to ensure that same file is not given in the output more than once aspect (Mutharaju, Raghava, Maier, and Hitzler). Search algorithm also optimizes its performance by reducing the network input output processing.
Time Frequency – Inverse Document Frequency is an algorithm that is essential in text processing. It is commonly used for web analysis hence a common web analysis algorithm. The algorithm is given by the formula below.
Where |D|: total number of documents in the corpus
|{d:ti € d}|: number of documents where the term ti appears.
The information need or the input for this algorithm is the number of the term X appears in the document and the number of terms in each document and the total number of documents.
Breadth first search BFS is an algorithm where search is processed on a graph data structure. There is a need to search each node at a time because it has unique data and the links. The computation is done through a graph and is done step by step. The problem is that BFS does not fit into MapReduce. However, there is a method of ensuring that this challenge is not a big headache. Iterated passes via MapReduce where the nodes are mapped in the process (Wang, Guanying, et al). There are more nodes that are created in the process of MapReduce passes. The other challenge is large amount of information represented in a graph becomes a processing problem. The best way to overcome this is by coming up with new ways of representing graphs. The use matrices and direct references can be used to represent the graphs. This reduces the memory size required to process graphs. Adding weight to the edges is another method of ensuring that graphs are used in hadoop sort algorithm.
Analysing Hadoop Performance
Since Hadoop is viewed as a better option to data warehousing many enterprise are implementing this technology for big data. Since most of the users who are interacting with the Hadoop systems, it was important that SQL interface is provided. This is to enable those who are moving from data warehouse to easily have ad-hoc queries (Leu, Yee and Chen). Writing a Hive script or writing a pig is what is considered as querying in Hadpoo environment is. Users can use their experience and existing SQL queries to interact with Hadoop data store. Facebook uses Hive to query the Hadoop. Big SQL is an SQL application of IBM used for IMB’s Hadoop for SQL and correlation analysis.
Hadoop and telecommunication
Cloudera is one of the distributions of Hadoop, most of the telecommunication and mobile companies use it today. Mobile phone devices have evolved from the basic tasks of making calls to complex tasks of accessing the internet. All the data stored on mobile devices need to be quickly accessed and analyzed. Hadoop Apache has been the best solution for the leading telecommunication companies (Xu, Yu, Kostamaa, and Gao). The use of Cloudera enables them to effectively configure and connect their services with Hadoop applications. There is reduced cost of infrastructure because scalability is no longer a challenge.
Works Cited
Borthakur, Dhruba. "The hadoop distributed file system: Architecture and design." (2007).
Leu J.S, Yee Y.S, Chen W.L, “Comparison of Map-Reduce and SQL on Large-scale Data Processing” in 2010 International Symposium onparallel and Distributed Processing with Applications (ISPA) pages: 224-248
Mutharaju, Raghava, Frederick Maier, and Pascal Hitzler. "A MapReduce Algorithm for SC." 23rd International Workshop on Description Logics DL2010. 2010.
Wang, Guanying, et al. "Using realistic simulation for performance analysis of mapreduce setups." Proceedings of the 1st ACM workshop on Large-Scale system and application performance. ACM, 2009.
Xu, Yu, Pekka Kostamaa, and Like Gao. "Integrating hadoop and parallel DBMs." Proceedings of the 2010 ACM SIGMOD International Conference on Management of data. ACM, 2010.
