Part of list:
- Introduction to Hadoop Distributed File System
- HDFS Architecture
- Name Node
- Data Nodes
- HDFS Architecture Diagram
- Loading Data into HDFS (Hands-on Example)
- Download the Data
- Uploading Using Command Line
The Hadoop Distributed File System (HDFS) is the one of the two components that makes up the backbone of Hadoop. As the name suggests, HDFS was implemented based on the distributed file system architecture with a few extra features.
A Distributed File System (DFS) is a file system that uses network protocols to store and manage files on a server or set of servers. The goal of a DFS is to allow clients to access data as if it were on their local machine. A DFS also allows data to be stored and shared in a secure and convenient way for multiple users. The servers that store the data have full control over the data and give access control to the clients of the DFS. This is why access to servers that are part of a distributed file system are limited — the data must be retrieved through an API.
HDFS was designed based on the distributed file system architecture because it was intended to be used for huge amounts of data. Because of this need, the data is stored across many servers in what is referred to as a cluster. Since HDFS is built on commodity hardware, there is also a need for redundancy across the cluster as well to ensure no data is lost. By default, data is replicated three times inside the file system. As mentioned in a previous article, Hadoop is different because it brings the processing to the data. This is done by enabling processing to occur on the same servers that store the data.
There are three main components that make up the Hadoop Distributed File System. They are the name node, the data nodes, and blocks.
The Name Node is a server that is referred to as the master node because it helps direct traffic for the slave nodes (data nodes). There can be multiple name nodes for fault tolerance; however, only one is running at any given point. The name node is responsible for managing the metadata for the file system — it is responsible for knowing where every piece of data is physically located throughout the file system’s servers. It is also responsible for enforcing the security policies that have been set on the directories and files throughout the file system. Finally, the name node is responsible for executing common file system responsibilities such as opening files and directories and renaming files.
Data nodes are the nodes that handle the data storage and the processing inside the Hadoop/HDFS ecosystem. Their primary responsibility is to store data inside blocks. With that comes operations to create blocks, delete blocks, and replicate blocks based on instructions received from the name node. Clients also interact with data nodes when doing read-write operations with the file system.
Blocks are locations where segments of files are stored. HDFS is designed to store very large files. So that the files are stored efficiently for performance and replication reasons, files are divided into one or more blocks. Because of this a file can be distributed across many servers. Since HDFS does not read files like a human, segmenting of files does not bother HDFS. When the file is needed, the name node will inform the file system where each block is located and will return each block in the correct order to the client.
In the above image, the client speaks to the name node to manage the metadata of where to write files into what blocks. The name node will look in the data nodes for space and tell the client where to write the blocks of data. The client then directly writes the blocks in to the data node that the name node specified. Once that write is complete, the replication process begins. The ideal situation is that the replication will take place over separate server racks just in case the whole rack loses power. Once each block is written, the block will report back to the name node as part of the block operations to confirm address and that the data was written.
Once the client is ready to read, it talks to the name node to know where to read from and then reaches out to the data node to read the block(s) in the parallel and then puts the file together.
First we need to make sure that you have the Cloudera Quickstart running. Please review the installation article in case you need a refresher.
Head over to https://grouplens.org/datasets/movielens/100k/. Download the .zip of all the data and save it somewhere on your computer that you can get to it. Make sure you unzip this file once it gets there. Once data is downloaded and unzipped, we are going to work with the u.data file.
First thing to do is to tell the difference between the local linux file system and HDFS. So let’s start with the local linux file system:
ls -l /dr-xr-xr-x 2 root root 4096 Apr 6 2016 bindrwxr-xr-x 3 root root 4096 Apr 6 2016 bootdrwxr-xr-x 13 root root 3440 Dec 17 22:56 devdrwxr-xr-x 85 root root 4096 Dec 17 22:56 etcdrwxrwxr-x 4 root root 4096 Dec 17 22:56 homedr-xr-xr-x 9 root root 4096 Apr 6 2016 libdr-xr-xr-x 7 root root 12288 Apr 6 2016 lib64drwx------ 2 root root 4096 Mar 4 2015 lost+founddrwxr-xr-x 2 root root 4096 Sep 23 2011 mediadrwxr-xr-x 2 root root 4096 Sep 23 2011 mntdrwxr-xr-x 5 root root 4096 Apr 6 2016 optdrwxr-xr-x 2 root root 4096 Apr 6 2016 packer-filesdr-xr-xr-x 169 root root 0 Dec 17 22:56 procdr-xr-x--- 3 root root 4096 Apr 6 2016 rootdr-xr-xr-x 2 root root 4096 Apr 6 2016 sbindrwxr-xr-x 3 root root 4096 Mar 4 2015 selinuxdrwxr-xr-x 2 root root 4096 Sep 23 2011 srvdr-xr-xr-x 13 root root 0 Dec 17 22:56 sysdrwxrwxrwt 36 root root 4096 Dec 17 23:03 tmpdrwxrwxr-x 15 root root 4096 Apr 6 2016 usrdrwxr-xr-x 24 root root 4096 Dec 17 22:58 var
Now let’s try querying the root of hdfs. To interact with hdfs, you need to use hdfs dfs commands.
hdfs dfs -ls /Found 5 itemsdrwxrwxrwx - hdfs supergroup 0 2016-04-06 02:26 /benchmarksdrwxr-xr-x - hbase supergroup 0 2017-12-17 22:58 /hbasedrwxrwxrwt - hdfs supergroup 0 2017-12-17 22:58 /tmpdrwxr-xr-x - hdfs supergroup 0 2016-04-06 02:27 /userdrwxr-xr-x - hdfs supergroup 0 2016-04-06 02:27 /var
Notice that there are huge differences between the local linux file system and HDFS. The good news is that you can interact with HDFS using similar commands that you do with the local linux file system by just prepending hdfs dfs.
Now let’s load the data. I’ve copied the data that we downloaded above from my computer to the docker-machine that we setup earlier by using:
docker-machine scp -r ./HadoopCourse root@docker-sandbox:/home/course
So let’s first make a new directory in HDFS named /hadoopTutorial using this command:
hdfs dfs -mkdir /hadoopTutorial
Now query HDFS again to ensure that it was created successfully:
hdfs dfs -ls /Found 6 itemsdrwxrwxrwx - hdfs supergroup 0 2016-04-06 02:26 /benchmarksdrwxr-xr-x - root supergroup 0 2017-12-17 23:09 /hadoopTutorialdrwxr-xr-x - hbase supergroup 0 2017-12-17 22:58 /hbasedrwxrwxrwt - hdfs supergroup 0 2017-12-17 22:58 /tmpdrwxr-xr-x - hdfs supergroup 0 2016-04-06 02:27 /userdrwxr-xr-x - hdfs supergroup 0 2016-04-06 02:27 /var
Since we have verified that it is there we can now copy the data into a subdirectory under hadoopTutorial called data.
hdfs dfs -mkdir /hadoopTutorial/datahdfs dfs -copyFromLocal home/course/ml-100k/u.data /hadoopTutorial/data/
Now let’s query HDFS to confirm that the data is there
hdfs dfs -ls /hadoopTutorial/data/Found 1 items-rw-r--r-- 1 root supergroup 1979173 2017-12-17 23:13 /hadoopTutorial/data/u.data
Congrats! You have now interacted with HDFS and put your first file onto the Hadoop Distributed File System.