Easy Glance with Hadoop

Hadoop is a framework for distributed processing large amount of data set across clusters of computers. Hadoop is an open source framework under Apache.

So today I'm not going to show you how to setup hadoop in your machine. You can find some article in here. If you want to setup multi-node cluster here is another article. Also some time back I blogged about how to setup single node hadoop cluster and you can find it here.

When you follow the above tutorials you can simply setup the single node and multi node hadoop cluster. But when I did that I got lots of problems when setting up a multi node cluster (single setup is a straight forward). So in this post I will list down some of issues and also the solutions for that. Also feel free to add any issues that you found as comments.

Before that you need to keep in mind to always check the hadoop logs. After you start the hadoop services or when running map reduce task always check the hadoop logs. To check your job logs, there are two ways you can do it. Either from browser or console. Job logs are created in "hadoop/logs/userlogs/job_id/*".

Here are the issues and solutions for Hadoop multi node cluster

Issue 1 :
=====

• ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Incompatible namespaceIDs

Solution 

=====

• If you got this exception in hadoop namenode log what you need to do is copy the namenode namespace id and paste it in every data node namespace id. (including master data node, if your master also act as datanode)
• vi ${path/to/hadoop/datastore}dfs/name/current/VERSION
• Copy namespace id value
• vi ${path/tp/hadoop/datastore/dfs}data/current/VERSION
• Paste it as in namespace id
• ${path/to/hadoop/datastore} --> you can find it in hadoop/conf/core-site.xml

Issue 2:

======

• ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:rajith cause:java.io.IOException: File /home/rajith/softwares/hadoop-temp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 12012-09-14 13:09:50,683 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 54310, call addBlock(/home/rajith/softwares/hadoop-temp/mapred/system/jobtracker.info, DFSClient_-632414254, null) from127.0.0.1:42296: error: java.io.IOException: File /home/rajith/softwares/hadoop-temp/mapred/system/jobtracker.infocould only be replicated to 0 nodes, instead of 1java.io.IOException: File /home/rajith/softwares/hadoop-temp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1558) at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:696)

Solution

======

• If you got this exception on your hadoop namenode log what you need to is be patient. :)
• First start the datanode (start-dfs.sh) and wait until the datanode starts. This may take 2min to start the datanode. You can verify it in logs.
• After that start the job tracker (start-mapred.sh)

Issue 3:

======

• Error reading task outputhttp://localhost:50060/tasklog?plaintext=true&attemptid=attempt_201209141426_0001_m_000002_0&filter=stdout12/09/14 14:27:41 WARN mapred.JobClient: Error reading task outputhttp://localhost:50060/tasklog?plaintext=true&attemptid=attempt_201209141426_0001_m_000002_0&filter=stder

Solution

======

• If you got this error that means you need to remove 127.0.0.1  localhost map in /etc/hosts in slave machine.
• Restart your hadoop nodes and check the logs.

Issue 4:

=======

• java.lang.OutOfMemoryError: unable to create new native thread

at java.lang.Thread.start0(Native Method)

at java.lang.Thread.start(Thread.java:691)

at org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:133)

at java.lang.Thread.run(Thread.java:722)

Solution

========

• If you got the above exception in datanode check the max number of process by using the following command
• ulimit -u
• If you got 1024 or around something increase the value to get rid of that error
• To increase add following lines to your /etc/security/limit.conf
• * soft nproc 8094
• * hard nproc 8094
• Restart machine to take effect