Data Intensive Applications — Storage and Retrieval

Introduction

Welcome to my third article in the Series Data Intensive Applications. In this article, we’ll get a high-level overview of various types of storage systems, what do they do under the hood, what kind of data structures they use, how to go pick the right one and in our situation. This article is highly inspired by one of the best Data Engineering books available out there: Designing Data Intensive Applications by Martin Kleppmann.

Note: I have received no compensation for writing this piece. Please consider supporting my and others’ writing by becoming a Medium member with this link.

The phone book analogy

Think about phone books. A phone book stores the name of all people effectively and allows you to quickly navigate the name based on the index that it has.

You can sort it by first name and you can quickly get the phone number of the person you’re looking for.

Similarly, database is also a storage system. It stores records that we give it and then it gives it back to us as we query it. As a Data professional, it is very important for us to know how these database systems work, what are they optimized for, what are your use cases that you want to use these storage systems for. If we have some…