Pages

Hive + MongoDB

In this article I'll show you how to connect mongoDB and hive using MongoStorageHandler.

Suppose you want to load mongoDB data into hive. So that you can run any hive query (HQL) using hadoop map reduce.

First you have to setup hadoop , hive and mongoDB.

Then download the mongo-hadoop release based on your hadoop version.

extract it and copy all 3 jar files to hadoop/lib and hive/lib folders.

First we create a sample records in mongoDB.


 use hive-test  
 db.books.insert({ "_id": 1, "name": "Java 7", "author": "author1" });  
 db.books.insert({ "_id": 2, "name": "Hadoop", "author": "author2" });  
 db.books.insert({ "_id": 3, "name": "Hive", "author": "author3" });  

Then start hive console and run following commands.


 create schema books;  
 use books;  
 CREATE TABLE book (id int, name string, author string)  
 STORED BY "com.mongodb.hadoop.hive.MongoStorageHandler"  
 WITH SERDEPROPERTIES('mongo.columns.mapping'='{"id":"_id","name":"name","author":"author"}')  
 TBLPROPERTIES ( "mongo.uri" = "mongodb://localhost:27017/hive-test.books");  

Make sure data loaded into hive using

 select * from book;   

Troubleshoot

 Failed with exception org.apache.hadoop.hive.ql.metadata.HiveException: Error in loading storage handler.com.mongodb.hadoop.hive.MongoStorageHandler  
 FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask  

Make sure you added above 3 jars into hadoop/lib and hive/lib

34 comments:

  1. This is very informative post. I really enjoy it and this is a great artical.I hope I will use this for our next business website.

    This is great job!

    Thanks for sharing this post.
    web development company uae

    ReplyDelete
  2. I get a lot of great information from this blog. Thank you for your sharing this informative blog. Just now I have completed hadoop certification course at a leading academy. If you are interested to learn Hadoop Training Chennai visit FITA IT training and placement academy.

    ReplyDelete
  3. You have certainly explained that Big data analytics is the process of examining big data to uncover hidden patterns, unknown correlations and other useful information that can be used to make better decisions..The big data analytics is the major part to be understood regarding Hadoop Training in Chennai program. Via your quality content i get to know about that in deep.Thanks for sharing this here.

    ReplyDelete
  4. This is the exact piece of information that I was searching for a long time(Hadoop Training in Chennai). Processing data is the biggest issue that every cloud based companies are facing worldwide(Big Data Hadoop Training in Chennai). Handling this problem made easy with the introduction of big data. Thank you so much for your worth able content here. Keep Posting article like this(Best Hadoop Training in Chennai).

    ReplyDelete
  5. Thanks for sharing informative post. Big data is a term that portrays the substantial volume of information; both organized and unstructured that immerses a business on an everyday premise. To know more details please visit Big Data Training in Chennai | Primavera Training in Chennai

    ReplyDelete
  6. Nice sharing. MongoDB is a widely used open source database system. Scop for mongodb is increasing day by day. Learn MongoDB Training from expert reach gangboard. Selenium Training

    ReplyDelete
  7. Hi, I am really happy to found such a helpful and fascinating post that is written in well manner. Thanks for sharing such an informative post.R Programming Online Training | Hadoop Online Training

    ReplyDelete
  8. Nice article updates. We've to post this blog informative it amazing. I like this our content update details. I read that our all post it's useful.
    Selenium Training in Chennai

    ReplyDelete
  9. Hey, I am truly upbeat to establish such an accommodating and intriguing post that is composed in well way. A debt of gratitude is in order for sharing such an instructive post...
    Selenium Training in Chennai
    Dot Net Training in Chennai
    Android Training in Chennai

    ReplyDelete
  10. This blog is having the general information. Got a creative work and this is very different one.We have to develop our creativity mind.This blog helps for this. Hadoop Online Training | Learn Hadoop Online Training | Hadoop Online Training in India



    ReplyDelete
  11. The scope of MongoDB and the Hive is increased day by day...
    Your blog is really much more informative and helpful to all people...
    Keep on blogging... Java Training in Chennai | Dot Net Training in Chennai

    ReplyDelete
  12. Hats off to your presence of mind..I really enjoyed reading your blog. I really appreciate your information which you shared with us.
    Data Science Online Training
    Hadoop Online Training

    ReplyDelete
  13. If anyone want to become an python certified professional reach besant Academy, which offers best python Training in bangalore with years of experienced professionals. For more details about courses…
    Selenium Training in Bangalore |
    Python Training in Bangalore |

    ReplyDelete
  14. It is really a great work and the way in which you are sharing the knowledge is excellent.Thanks for your informative article

    Hadoop Training in Marathahalli|
    Hadoop Training in Bangalore|
    Data science training in Marathahalli|
    Data science training in Bangalore|

    ReplyDelete
  15. This comment has been removed by the author.

    ReplyDelete
  16. Great and really helpful article! Adding to the conversation, you have provide more information, or express a new point of view.Nice information and updates. Really i like it and everyday am visiting your site... keep blogging.DevOps Training in Bangalore | DevOps Training in Bangalore

    ReplyDelete
  17. Needed to compose you a very little word to thank you yet again regarding the nice suggestions you’ve contributed here.

    Data Science Training in Bangalore

    Datascience Training in Chennai

    ReplyDelete
  18. Informative post about hadoop, i am looking forward for realtime hadoop online training institute.

    ReplyDelete