Thursday, 18 January 2018

Datalake vs Data Warehouse

This post outlines the usages of datalakes and data warehouses. 



Datalakes are generally built on databases which can support storage of unstructured or semi-structured data. There isn’t a strict schema such as like datawarehouse built on top of RDBMS’s.
This allows adding additional fields to tables(documents) when new fields become available. For example:  
We may have 1 million customers already stored. Now we need to import another million with a new field for each customer such as credit rating. This can be done in a datalake without having to change the previous customers and adding null credit ratings to historical rows.

The Datalake acts as a buffer between live application databases and analytics. Data from live databases is periodically exported and placed into the datalake. Analytics teams would then only work with the data held in the datalake.

Datalake is an intermediate storage for analytics, original data imported to a data lake is generally not readily analysable. The data would go through series of transformations to transform it to a form that can be analysed. This may be in the forms of running ETL jobs or Stream transformations to convert from the semi structured data into warehouse schema. The warehouse schema may be held in a RDBMS allowing it to be queried using SQL by business professionals. Transformations from datalake into structured data would require scripting and not just querying. This is achieved languages such as python, scala hosted through platforms such as spark or hadoop. They take the semi structured data and transform them into a schema. Once the data is transformed it can be queried for reporting purposes.

Below is a typical data pipeline to describe where a datalake would sit:





Below is an article outlining the differences:


109 comments:

  1. You also have to look after the purchase order of the total project. That whether the supply is coming as per the po order or not. safe site storage clearfield

    ReplyDelete
  2. Caps must be headed toward the advanced information and examination. Demography, ethnography, topography, compatibility and even the classes of the group of spectators can be hauled out through information. data science course in pune

    ReplyDelete

  3. Well, The information which you posted here is very helpful & it is very useful for the needy like me.., Wonderful information you posted here. Thank you so much for helping me out to find theData science course in Mumbai Organisations and introducing reputed stalwarts in the industry dealing with data analyzing & assorting it in a structured and precise manner. Keep up the good work. Looking forward to view more from you.

    ReplyDelete
  4. Such a very useful article. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. I would like to state about something which creates curiosity in knowing more about it. It is a part of our daily routine life which we usually don`t notice in all the things which turns the dreams in to real experiences. Back from the ages, we have been growing and world is evolving at a pace lying on the shoulder of technology. data science certification will be a great piece added to the term technology. Cheer for more ideas & innovation which are part of evolution.

    ReplyDelete
  5. I feel very grateful that I read this. It is very helpful and very informative and I really learned a lot from it.
    machine learning and artificial intelligence courses in bangalore

    ReplyDelete
  6. Such a very useful article. I have learn some new information.thanks for sharing.
    data scientist course in mumbai

    ReplyDelete
  7. Hi! This is my first visit to your blog! We are a team of volunteers and new initiatives in the same niche. Blog gave us useful information to work. You have done an amazing job!
    Data Analytics Course in Mumbai

    ReplyDelete
  8. Such a very useful Blog. Very interesting to read this article. I have learn some new information.thanks for sharing. know more about

    ReplyDelete
  9. Pretty good post. I just stumbled upon your blog and wanted to say that I have really enjoyed reading your blog posts. Any way I’ll be subscribing to your feed and I hope you post again soon.
    ExcelR Business Analytics Course

    ReplyDelete
  10. Very nice blog here and thanks for post it.. Keep blogging...
    ExcelR data science training

    ReplyDelete
  11. I like viewing web sites which comprehend the price of delivering the excellent useful resource free of charge. I truly adored reading your posting. Thank you!
    data analytics course in hyderabad

    ReplyDelete
  12. Awesome blog. I enjoyed reading your articles. This is truly a great read for me. I have bookmarked it and I am looking forward to reading new articles. Keep up the good work!
    ExcelR data science course in mumbai

    ReplyDelete
  13. I would like to thank you for the efforts you have made in writing this article. I am hoping the same best work from you in the future as well. In fact your creative writing abilities has inspired me to start my own Blog Engine blog now. Really the blogging is spreading its wings rapidly. Your write up is a fine example of it. excelr data science

    ReplyDelete
  14. After reading your article I was amazed. I know that you explain it very well. And I hope that other readers will also experience how I feel after reading your article.
    data analytics courses

    ReplyDelete
  15. After reading your article I was amazed. I know that you explain it very well. And I hope that other readers will also experience how I feel after reading your article.

    data analytics courses

    business analytics course

    data science interview questions

    data science course in mumbai

    ReplyDelete
  16. Thanks for sharing this visual presentation on the difference between Data lake and data warehouse. Data Lake stores all data irrespective of the source and its structure whereas Data Warehouse stores data in quantitative metrics with their attributes, I think that's the main difference.

    ReplyDelete
  17. Thanks for giving me the time to share such nice information. Thanks for sharing.
    Data Science Course

    ReplyDelete
  18. The information provided on the site is informative. Looking forward more such blogs. Thanks for sharing .
    Artificial Inteligence course in Nashik
    AI Course in Nashik

    ReplyDelete
  19. Excellent Blog! I would like to thank for the efforts you have made in writing this post. I am hoping the same best work from you in the future as well. I wanted to thank you for this websites! Thanks for sharing. Great websites!

    business analytics course

    data analytics courses in mumbai

    data science interview questions

    data science course in mumbai

    ReplyDelete
  20. wonderful article. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article resolved my all queries.
    Data science Interview Questions

    ReplyDelete
  21. wonderful article. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article resolved my all queries.
    Data Science Course

    ReplyDelete
  22. Attend The Artificial Intelligence course From ExcelR. Practical Artificial Intelligence course Sessions With Assured Placement Support From Experienced Faculty. ExcelR Offers The Artificial Intelligence course.
    Artificial Intelligence Course

    ReplyDelete
  23. wonderful article. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article resolved my all queries.
    Data Science Course

    ReplyDelete
  24. wonderful article. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article resolved my all queries. keep it up.
    data analytics course in Bangalore

    ReplyDelete
  25. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.
    Correlation vs Covariance

    ReplyDelete
  26. I have to search sites with relevant information on given topic and provide them to teacher our opinion and the article.

    Simple Linear Regression

    ReplyDelete
  27. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.
    Correlation vs Covariance
    Simple linear regression

    ReplyDelete
  28. I just got to this amazing site not long ago. I was actually captured with the piece of resources you have got here. Big thumbs up for making such wonderful blog page!

    Simple Linear Regression

    Correlation vs Covariance

    ReplyDelete
  29. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.
    Correlation vs Covariance
    Simple linear regression
    data science interview questions

    ReplyDelete
  30. This is a wonderful article, Given so much info in it, These type of articles keeps the users interest in the website, and keep on sharing more ... good luck.

    data science interview questions

    ReplyDelete
  31. Awesome blog. I enjoyed reading your articles. This is truly a great read for me. I have bookmarked it and I am looking forward to reading new articles. Keep up the good work!

    Simple Linear Regression

    Correlation vs Covariance

    ReplyDelete
  32. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.
    Correlation vs Covariance
    Simple linear regression
    data science interview questions

    ReplyDelete
  33. I am looking for and I love to post a comment that "The content of your post is awesome" Great work!

    Simple Linear Regression

    Correlation vs Covariance

    ReplyDelete
  34. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.
    Correlation vs Covariance
    Simple linear regression
    data science interview questions

    ReplyDelete
  35. This Was An Amazing ! I Haven't Seen This Type of Blog Ever ! Thankyou For Sharing, data science online course

    ReplyDelete
  36. This is a wonderful article, Given so much info in it, These type of articles keeps the users interest in the website, and keep on sharing more ... good luck.

    data science interview questions

    ReplyDelete
  37. Attend The Business Analytics Course From ExcelR. Practical Business Analytics Course Sessions With Assured Placement Support From Experienced Faculty. ExcelR Offers The Data Analytics Course.
    Business Analytics Course

    ReplyDelete
  38. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.
    Correlation vs Covariance
    Simple linear regression
    data science interview questions

    ReplyDelete
  39. I have to search sites with relevant information on given topic and provide them to teacher our opinion and the article.

    data science interview questions

    ReplyDelete
  40. Any good dealer can help you with this if needed. The most commonly used capacities range from 3,000-6,000 pound units, however forklifts can range anywhere from 2,000-65,000 pounds or moreforklift course

    ReplyDelete
  41. I am looking for and I love to post a comment that "The content of your post is awesome" Great work!

    Simple Linear Regression

    Correlation vs Covariance

    ReplyDelete
  42. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.
    Correlation vs Covariance
    Simple linear regression
    data science interview questions

    ReplyDelete
  43. Attend The Data Scientist Courses From ExcelR. Practical Data Scientist Courses Sessions With Assured Placement Support From Experienced Faculty. ExcelR Offers The Data Scientist Courses. Data Scientist Courses

    ReplyDelete
  44. Attend The Data Analyst Course From ExcelR. Practical Data Analyst Course Sessions With Assured Placement Support From Experienced Faculty. ExcelR Offers The Data Analyst Course.
    Data Analyst Course

    ReplyDelete
  45. Amazing Article ! I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.
    Simple Linear Regression
    Correlation vs covariance
    data science interview questions
    KNN Algorithm

    ReplyDelete
  46. Impressive blog with lovely information. really very useful article for us thanks for sharing such a wonderful blog. data science training in Hyderabad

    ReplyDelete
  47. Attend The Data Analyst Course From ExcelR. Practical Data Analyst Course Sessions With Assured Placement Support From Experienced Faculty. ExcelR Offers The Data Analyst Course.
    Data Analyst Course

    ReplyDelete
  48. Keeping a keen eye on the growing demand of the current hour, it has set to offer efficient handling courses to professional enthusiasts. data science course in hyderabad

    ReplyDelete
  49. Amazing Article ! I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.
    Correlation vs Covariance
    Simple Linear Regression
    data science interview questions
    KNN Algorithm
    Logistic Regression explained

    ReplyDelete
  50. This Was An Amazing ! I Haven't Seen This Type of Blog Ever ! Thankyou For Sharing, data science course in hyderabad with placements

    ReplyDelete
  51. Thank you for sharing wonderful information with us.Really useful for everyone data scientist courses

    ReplyDelete
  52. Attend The Data Science Courses From ExcelR. Practical Data Science Courses Sessions With Assured Placement Support From Experienced Faculty. ExcelR Offers The Data Science Courses.
    Data Science Courses

    ReplyDelete
  53. This is a wonderful article, Given so much info in it, These type of articles keeps the users interest in the website, and keep on sharing more ... good luck.

    Simple Linear Regression

    Correlation vs covariance

    KNN Algorithm

    Logistic Regression explained

    ReplyDelete
  54. very well explained .I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.
    Simple Linear Regression
    Correlation vs covariance
    data science interview questions
    KNN Algorithm
    Logistic Regression explained

    ReplyDelete
  55. Thank you for sharing wonderful information with us.Really useful for everyonedata scientist courses

    ReplyDelete
  56. very well explained. I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.
    Logistic Regression explained
    Correlation vs Covariance
    Simple Linear Regression
    data science interview questions
    KNN Algorithm

    ReplyDelete
  57. Leave the city behind & drive with us for a Thrilling drive over the Desert Dunes & Experience a lavish dinner with amazing shows in our Desert Camp.
    desert safari dubai

    ReplyDelete
  58. Awesome blog. I enjoyed reading your articles. This is truly a great read for me. I have bookmarked it and I am looking forward to reading new articles. Keep up the good work! data scientist courses

    ReplyDelete
  59. very well explained. I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.
    Logistic Regression explained
    Correlation vs Covariance
    Simple Linear Regression
    data science interview questions
    KNN Algorithm

    ReplyDelete
  60. Very nice blogs!!! I have to learning for lot of information for this sites…Sharing for wonderful information. Thanks for sharing this valuable information to our vision. You have posted a trust worthy blog keep sharing, data science course in Hyderabad

    ReplyDelete
  61. Nice content and interesting blog. Join 360digitmg for the data scientist course.
    https://360digitmg.com/data-science-course-training-in-hyderabad

    ReplyDelete
  62. Attend The Business Analytics Courses From ExcelR. Practical Business Analytics Courses Sessions With Assured Placement Support From Experienced Faculty. ExcelR Offers The Data Analytics Courses.
    Business Analytics Courses

    ReplyDelete
  63. hello sir,
    thanks for giving that type of information. I am really happy to visit your blog.Leading Solar company in Andhra Pradesh

    ReplyDelete
  64. Very informative content and intresting blog post.Data science training in Mumbai

    ReplyDelete

  65. Just the way I have expected. Your website really is interesting. ExcelR Data Analytics Course

    ReplyDelete
  66. i am glad to discover this page : i have to thank you for the time i spent on this especially great reading !! i really liked each part and also bookmarked you for new information on your site.
    Data Scientist Course

    ReplyDelete
  67. I Want to leave a little comment to support and wish you the best of luck.we wish you the best of luck in all your blogging endeavors.
    Business Analytics Course in Bangalore

    ReplyDelete
  68. I want to leave a little comment to support and wish you the best of luck.we wish you the best of luck in all your blogging enedevors.
    Data Analytics Courses in Bangalore

    ReplyDelete
  69. Impressive blog to be honest definitely this post will inspire many more upcoming aspirants. Eventually, this makes the participants to experience and innovate themselves through knowledge wise by visiting this kind of a blog. Once again excellent job keep inspiring with your cool stuff.

    Data Science training in Raipur

    ReplyDelete
  70. Terrific post thoroughly enjoyed reading the blog and more over found to be the tremendous one. In fact, educating the participants with it's amazing content. Hope you share the similar content consecutively.

    Data Analytics online course

    ReplyDelete
  71. Highly appreciable regarding the uniqueness of the content. This perhaps makes the readers feels excited to get stick to the subject. Certainly, the learners would thank the blogger to come up with the innovative content which keeps the readers to be up to date to stand by the competition. Once again nice blog keep it up and keep sharing the content as always.

    Data Science training

    ReplyDelete
  72. Honestly speaking this blog is absolutely amazing in learning the subject that is building up the knowledge of every individual and enlarging to develop the skills which can be applied in to practical one. Finally, thanking the blogger to launch more further too.

    Data Analytics online course

    ReplyDelete
  73. Thanks for posting the best information and the blog is very helpful.data science interview questions and answers

    ReplyDelete
  74. Good Post! it was so good to read and useful to improve my knowledge as an updated one, keep blogging. After seeing your article, I want to say that also a well-written article with some very good information which is very useful for the readers.... thanks for sharing it and do share more posts like this.
    Data Science Training in Hyderabad
    Data Science Course in Hyderabad

    ReplyDelete
  75. I want to leave a little comment to support and wish you the best of luck. We wish you the best of luck in all your blogging endeavours.
    Python Training in Hyderabad
    Python Course in Hyderabad

    ReplyDelete
  76. Truly mind blowing blog went amazed with the subject they have developed the content. These kind of posts really helpful to gain the knowledge of unknown things which surely triggers to motivate and learn the new innovative contents. Hope you deliver the similar successive contents forthcoming as well.

    data science in bangalore

    ReplyDelete
  77. Stockroom loaning is regularly portrayed as a generally safe, high return business, yet there is a lack of distribution center moneylenders. The huge public banks have either exited the market totally, or have confined their loaning to extremely huge clients and exceptionally conventional item. A significant number of the leftover second level banks center basically around early buy programs for their own item,warehouse shelving

    ReplyDelete
  78. It is imperative that we read blog post very carefully. I am already done it and find that this post is really amazing.
    data scientist training and placement in hyderabad

    ReplyDelete
  79. Thanks for posting the best information and the blog is very good.digital marketing institute in hyderabad

    ReplyDelete
  80. Thanks for bringing such innovative content which truly attracts the readers towards you. Certainly, your blog competes with your co-bloggers to come up with the newly updated info. Finally, kudos to you.

    Data Science Course in Varanasi

    ReplyDelete

  81. I was basically inspecting through the web filtering for certain data and ran over your blog. I am flabbergasted by the data that you have on this blog. It shows how well you welcome this subject. Bookmarked this page, will return for extra. data science course in jaipur

    ReplyDelete
  82. Under 'parallel-in' registration, the deletion of the registration withinside the overseas registered isn't obligatory.Shipping from china

    ReplyDelete
  83. Thanks for posting the best information and the blog is very good.artificial intelligence course in hyderabad

    ReplyDelete
  84. Thanks for posting the best information and the blog is very good.data science institutes in hyderabad

    ReplyDelete
  85. Wonderful blog found to be very impressive to come across such an awesome blog. I should really appreciate the blogger for the efforts they have put in to develop such an amazing content for all the curious readers who are very keen of being updated across every corner. Ultimately, this is an awesome experience for the readers. Anyways, thanks a lot and keep sharing the content in future too.

    Data Science Course in Bhilai

    ReplyDelete
  86. Impressive blog to be honest definitely this post will inspire many more upcoming aspirants. Eventually, this makes the participants to experience and innovate themselves through knowledge wise by visiting this kind of a blog. Once again excellent job keep inspiring with your cool stuff.

    Data Science Training in Bhilai

    ReplyDelete
  87. Extremely overall quite fascinating post. I was searching for this sort of data and delighted in perusing this one. Continue posting. A debt of gratitude is in order for sharing. data scientist course in delhi

    ReplyDelete
  88. Extremely overall quite fascinating post. I was searching for this sort of data and delighted in perusing this one. Continue posting. A debt of gratitude is in order for sharing. python course in delhi

    ReplyDelete
  89. Wow i can say that this is another great article as expected of this blog.Bookmarked this site.. www.meridiannorstar.net

    ReplyDelete
  90. Good Article. Thank you for sharing! Really an awesome post for learn everyone.

    ReplyDelete
  91. I recently came across your article and have been reading along. I want to express my admiration of your writing skill and ability to make readers read from the beginning to the end.
    Data Analytics Courses In Pune

    ReplyDelete
  92. Excellent article... Thank you for providing such valuable information; the contents are quite intriguing.
    Data Engineering Services 
    Data Analytics Solutions
    Data Modernization Solutions
    AI & ML Service Provider

    ReplyDelete
  93. Extraordinary blog went amazed with the content that they have developed in a very descriptive manner. This type of content surely ensures the participants to explore themselves. Hope you deliver the same near the future as well. Gratitude to the blogger for the efforts.

    Data Science Training

    ReplyDelete
  94. Stupendous blog huge applause to the blogger and hoping you to come up with such an extraordinary content in future. Surely, this post will inspire many aspirants who are very keen in gaining the knowledge. Expecting many more contents with lot more curiosity further.

    Data Science Certification in Bhilai

    ReplyDelete
  95. Pleasant data, important and incredible structure, as offer great stuff with smart thoughts and ideas, loads of extraordinary data and motivation, the two of which I need, because of offer such an accommodating data here.
    business analytics course in hyderabad

    ReplyDelete
  96. I wanted to leave a little comment to support you and wish you a good continuation. Wishing you the best of luck for all your blogging efforts.
    data analytics course in hyderabad

    ReplyDelete
  97. Very interesting blog. Alot of blogs I see these days don't really provide anything that I'm interested in, but I'm most definately interested in this one. Just thought that I would post and let you know. data scientist course in surat

    ReplyDelete
  98. Thank you so much for ding the impressive job here, everyone will surely like your post. data science course in Patna

    ReplyDelete
  99. Wonderful blog post. This is absolute magic from you! I have never seen a more wonderful post than this one. You've really made my day today with this. I hope you keep this up! data science course in Shimla

    ReplyDelete
  100. It is really a nice and helpful piece of information. I’m glad that you shared this useful information with us. Please visit my web site Supremebrick.com. Best Building Violation Removal Queens service provider.

    ReplyDelete