{"id":15795,"date":"2022-05-11T18:40:37","date_gmt":"2022-05-11T22:40:37","guid":{"rendered":"https:\/\/www.iri.com\/blog\/?p=15795"},"modified":"2022-05-11T18:40:37","modified_gmt":"2022-05-11T22:40:37","slug":"voracity-software-support-for-cloud-file-stores","status":"publish","type":"post","link":"https:\/\/beta.iri.com\/blog\/data-transformation2\/voracity-software-support-for-cloud-file-stores\/","title":{"rendered":"Voracity Software Support for Cloud File Stores"},"content":{"rendered":"

Public and private cloud file systems are becoming increasingly popular places for businesses and government agencies to store their data. Several major competitors have arisen in the cloud storage provider space, with Amazon, Microsoft Azure, and Google Cloud Platform at the top.\u00a0<\/span><\/p>\n

IRI Voracity<\/span><\/a> and its core <\/span>structured data<\/span><\/a> processing program, <\/span>SortCL<\/span><\/a>, already supported the processing of <\/span>flat files<\/span><\/a> stored in Amazon S3 buckets. With the latest updates to IRI CoSort and IRI Workbench, however, support for cloud storage has been expanded to encompass Google Cloud Storage and Azure Blob Storage, too.<\/span><\/p>\n

This article explains how to configure Voracity software and its SortCL program jobs specifically to use flat files in the three aforementioned cloud stores as data sources and\/or targets. My first demonstration will be with simple report job. My second demonstration reads from one cloud provider and writes to another after applying some FieldShield data masking rules.\u00a0<\/span><\/p>\n

Note: <\/span>Searching and masking sensitive PII located in semi-structured and unstructured data in Azure Blob Storage, Google Cloud Storage, and S3 is possible using IRI <\/span><\/i>DarkShield<\/span><\/i><\/a>. Read this <\/span><\/i>article<\/span><\/i><\/a> to learn more.<\/span><\/i><\/p>\n

Providing Metadata for Cloud Files<\/b><\/h6>\n

As a quick aside, although IRI Workbench normally supports the discovery of metadata for files, there are a few exceptions. One of these exceptions includes discovering the metadata of files hosted in the cloud. When metadata cannot be discovered, users have two options:<\/span><\/p>\n

The first option would be generating DDF files manually. This requires that they know the structure of the files in the cloud beforehand. The second option is to download a copy of the file from the cloud and get the metadata from it.<\/span><\/p>\n

With a copy of the file stored locally, you can run the <\/span>Discover Metadata<\/span><\/i> wizard<\/span><\/a> in Workbench to generate a DDF file automatically \u2013 in the same way, you would normally discover the metadata of files on the same PC or mounted in your LAN.<\/span><\/p>\n

Demonstrating with Cloud Files<\/b><\/h6>\n

To start this demonstration, I will open the Report wizard in IRI Workbench by clicking on the dropdown arrow next to the stopwatch. From the dropdown menu, select the <\/span>New Reporting Job <\/span><\/i>wizard.<\/span><\/p>\n

\"\"<\/p>\n

From here a wizard opens to create a SortCL job script with an extension of <\/span>.scl<\/i><\/b>. In the first dialog, select which IRI project folder in the workspace where the future job script will be placed, and specify a meaningful name for the job. Once that\u2019s done, move on by clicking<\/span> Next ><\/span><\/i>.<\/span><\/p>\n

\"\"<\/p>\n

The next page of the wizard has us specify our <\/span>data source<\/span><\/i>:<\/span><\/p>\n

\"\"<\/p>\n

SortCL jobs support several source types:\u00a0<\/span><\/p>\n

    \n
  1. Files (local to the PC or a LAN drive);\u00a0<\/span><\/li>\n
  2. ODBC (for direct connections to RDB tables);<\/span><\/li>\n
  3. URL (for HDFS, HTTP\/S, FTP\/S, MongoDB, and where S3 connections were before);<\/span><\/li>\n
  4. PROCEDURE (for runtime-linked, custom input programs written in C\/C++);<\/span><\/li>\n
  5. CONSOLE (stdin); and, now,<\/span><\/li>\n
  6. CLOUD STORAGE.\u00a0<\/span><\/li>\n<\/ol>\n

    For this demonstration we will select CLOUD STORAGE as the data source type:<\/span><\/p>\n

    \"\"<\/p>\n

    We now need to select either an existing (previously created) cloud storage connection, or create a new connection. Saved connections will be selectable in the <\/span>Cloud Storage Connection<\/span><\/i> drop-down list.<\/span><\/p>\n

    If you choose to create a new connection at this time, instead of using a previously created connection, a new connection will be created via the input the user provides in another window. That new connection gets added to the list of saved connections.<\/span><\/p>\n

    For this demonstration, I will create a new connection, and thus click <\/span>New\u2026<\/span><\/i> to bring up this window:<\/span><\/p>\n

    \"\"<\/p>\n

    A new sub-wizard for cloud storage connections opens to help create a connection string in our job script\u2019s \/INFILE section. This tells SortCL to access files stored in a cloud store.<\/span><\/p>\n

    This wizard requires a unique name for the cloud connection we are making, and to specify which cloud storage provider we will access. In the dropdown menu for <\/span>Storage Provider<\/span><\/i> we can choose from Amazon S3, Google Cloud Storage, or Azure Blob Storage.<\/span><\/p>\n

    \"\" \"\"<\/span><\/p>\n

    Amazon S3 provider form fields<\/span><\/i><\/p>\n

    \"\"<\/p>\n

    Azure Blob Storage form fields<\/span><\/i><\/p>\n

    \"\"<\/p>\n

    Google Cloud Storage form fields<\/span><\/i><\/p>\n

    Notice that for each of the different cloud providers there are two options to choose from for authenticating. Credentials can be added as parameters to the SortCL script, or through system environment variables.\u00a0<\/span><\/p>\n

    Unlike the environment variables specified in SortCL job scripts (and prefixed by a $)<\/span>1<\/sup><\/a><\/span><\/span>, system environment variables are stored by users on their machine and accessed by the operating system.<\/span><\/p>\n

    If you choose to use system environment variables, you will need to store your credentials in system environment variables with names specific to the storage provider. Notice in the images above, it is clearly stated what environment variables names need to be set (with the appropriate value) for each corresponding cloud storage provider.<\/span><\/p>\n

    See these links for <\/span>S3<\/span><\/a>, <\/span>Azure<\/span><\/a>, and <\/span>Google<\/span><\/a> environment variable setup.<\/span><\/p>\n

    If you are manually providing credentials you will need to supply different credentials based on the selected cloud provider\u2019s requirements:<\/span>
    \n<\/span><\/p>\n