Introduction

Azure blob storage is a Microsoft feature that allows users to store a large amount of data on Microsoft's data storage platforms.

Blob storage is perfect for storing massive amounts of unstructured data and structured data. Unstructured data is data that doesn't conform to a particular data model or definition, such as text or binary data.

This article will teach you how you can upload and download blob files from Azure blob storage. This is the continuation of the last post where I taught how to connect to Azure blob from Alteryx. You can check it out.

The first part of this interesting project is to download files from the Azure Blob Storage, and the second part to this is to upload a transformed data to the Blob Storage.

Blob Storage Overview

AZURE STORAGE ACCOUNT Container A raw-data/ Container B processed/ Excel Config TestinformationB.csv Python + azure-storage-blob BlobServiceClient download upload Local Output Transformed Files

Central Configuration File

First things first, we need to create a central Excel file that defines the resources' location and the names of the resources. Below is the screenshot of the process flow.

Central CSV Location | Name Containers pd.read_csv() Parse Config Iterate for each row Blob I/O

This is the central Excel file. The next step is to pull the data into a Python environment using the file and transform the data. Having done that, push the data into the Azure blob container as specified in the Excel file.

Get Files from Azure Blob Storage

The code below shows how to get a file from Blob Storage. We are going to use the Python library provided by Microsoft; this library is called azure-storage-blob. You can install this library using pip install azure-storage-blob.

Python 
"""
We are going to import all the packages we are going to need here.
Then we will get our files. The steps here is that we are going to
connect to the central csv file that has the location to all the
file on the Blob Storage, we going to use the shared account key
to connect with the resources.

Please research on how to get your account shared key on your
azure portal. Interestingly, the key I provided here will not work.
"""

from azure.storage.blob import BlobClient
import os
import pandas as pd
from azure.storage.blob import BlobServiceClient, BlobClient, ContainerClient, __version__

data1 = pd.read_csv('TestinformationB.csv')

Note: The connection string shown in this tutorial contains a sample key that will not work. Please research how to get your account shared key on your Azure portal and replace the connection string with your own credentials.

Get files from Azure Blob to the Remote System

Python 
for lenght in range(0, len(data1)):

    local_path = data1['Location'][lenght]
    file_to_download_name = data1['Name'][lenght]
    blobcontainer_name = data1['Containers'][lenght]

    blob = BlobClient.from_connection_string(
        conn_str="DefaultEndpointsProtocol=https;"
                 "AccountName=cndroalteryxblobstorage;"
                 "AccountKey=M3rUQZgVp1o9U7FJ8WRgyxkt9uGDpg"
                 "AjfqyBOwEvuXmW8B+2C173XN58AlPq1iMT"
                 "9XQqtXycjTwBPJw2+cANnw==;"
                 "EndpointSuffix=core.windows.net",
        container_name=blobcontainer_name,
        blob_name=file_to_download_name
    )

    file_data = file_to_download_name.split('.')[0] + '.txt'
    print(file_data)

    with open(file_data, "wb") as my_blob:
        blob_data = blob.download_blob()
        blob_data.readinto(my_blob)

    file_new_name = file_to_download_name
    os.rename(str(file_data), str(file_new_name))

Transform Your Data

At this point, we can now transform our file, we can also perform different operations on our data.

PHASE 1 Azure Blob Storage download_blob() Local Files PHASE 2 Data Transformation Clean Transform PHASE 3 Output Files upload_blob() Azure Blob Storage Automated Pipeline Cycle

In the code below, we are going to iterate through all the files, their locations, and pick up the files, upload the files to their respective containers on the Blob Storage.

Upload Files to Azure Blob Storage

Python 
## UPLOAD FILES FROM REMOTE SYSTEM TO THE AZURE BLOB

blob_service_client = BlobServiceClient.from_connection_string(
    "DefaultEndpointsProtocol=https;"
    "AccountName=cndroalteryxblobstorage;"
    "AccountKey=M3rUQZgVp1o9U7FJ8WRgyxkt9uGDpg"
    "AjfqyBOwEvuXmW8B+2C173XN58AlPq1iMT"
    "9XQqtXycjTwBPJw2+cANnw==;"
    "EndpointSuffix=core.windows.net"
)

for lenght in range(0, len(data1)):

    local_path = data1['Location'][lenght]
    local_file_name = data1['Name'][lenght]
    container_name = data1['Containers'][lenght]

    upload_file_path = os.path.join(local_path, local_file_name)
    print(upload_file_path)

    blob_client = blob_service_client.get_blob_client(
        container=container_name,
        blob=local_file_name
    )

    print("\nUploading to Azure Storage as blob:\n\t" + local_file_name)

    # Upload the created file
    with open(upload_file_path, "rb") as data:
        blob_client.upload_blob(data)

Conclusion

With this implementation, you can work programmatically with all your files in the Blob storage. We can use the blob storage for different purposes; here we have used it for our data store. We can retrieve our data from here and we can also upload our data to this location.