Design a Data Storage Structure – Data Sources and Ingestion

In this chapter you will provision numerous Azure data analytics products. By doing so, you will begin to understand more about the products and their features, which can help you create and choose the best tool for your given solution requirements. Choosing a proper service for a scenario results in having a solid design. Table 3.2 reviews the mapping between Azure datastores and the data model supported within them.

TABLE 3.2 Analytical datastores

Throughout the numerous Big Data stages, storage is required. As data is ingested, it needs to be stored before moving to the next stage. Each stage requires a place to store the transformed data, and once the transformation is complete, the final version also needs to be stored. An Azure product that plays a central role in the entire Big Data analytics solution is Azure Data Lake Storage (ADLS).

Design an Azure Data Lake Solution

A data lake is defined as a repository of data stored in its natural, unformatted, unmodified, or raw form. Can include structured, semi‐structured, unstructured, and binary data. A successor to data mart or data warehouse, a data lake is a collection of your datastores and the data within them. On Azure these datastores are products like ADLS, Azure SQL, and Azure Cosmos DB. Continue reading this book to learn about the numerous Azure products and how they all work together to provide a data lake. You have already provisioned, configured, and used an Azure SQL database and an Azure Cosmos DB. Another Azure product that plays a significant role in your data lake is an ADLS container.

Complete Exercise 3.1 to provision an Azure storage account and an ADLS container.