{"id":9584,"date":"2016-05-17T15:43:01","date_gmt":"2016-05-17T19:43:01","guid":{"rendered":"http:\/\/www.iri.com\/blog\/?p=9584"},"modified":"2023-02-10T15:57:36","modified_gmt":"2023-02-10T20:57:36","slug":"the-use-of-data-lakes","status":"publish","type":"post","link":"https:\/\/beta.iri.com\/blog\/business-intelligence\/the-use-of-data-lakes\/","title":{"rendered":"The Use of Data Lakes"},"content":{"rendered":"

Has your organization considered using a data lake? This article explains what a data lake is, and how you can fish its murky depths for value in an architecture optimized for your needs. IRI Voracity users should also be able to assess this approach relative to other paradigms<\/a> the platform supports, including LDW, EDW, ODS, and eventually, self-service BI, too.<\/p>\n

What Is a Data Lake?<\/h2>\n

Data lakes are environments for gathering and storing data for experimentation. The term data lake was coined by Pentaho CTO James Dixon. He used the term to compare data that was cleansed, packaged, and structured — like what was found in a data mart (or bottled water) — to data in its more natural state (in a large, real body of water).<\/p>\n

“The contents of the data lake stream in from one or more sources to fill the lake, and various users of the lake can come to examine, dive in, or take samples,\u201d Dixon blogged.<\/p>\n

Gartner refers to the data lake as “a collection of storage instances of various data assets additional to the originating data sources. These assets are stored in a near-exact, or even exact, copy of the source format.\u201d<\/p>\n

Thus, the data lake is a single store of enterprise data that includes both raw data (which implies an exact copy of source data) and transformed data used for reporting and analytics. Some want the data lake to replace the traditional data warehouse, while others see it as more of a staging area to feed data into existing data warehouse architectures.<\/p>\n

Data lakes can exist in the file system, a Hadoop fabric, or cloud storage service like Amazon S3.<\/p>\n

Why Use a Data Lake?<\/h2>\n

Traditional architectures silo information into buckets that provide only a fraction of the insight that might be derived from a larger collection of data. Data marts and warehouses require data to be identified, cleansed, and formatted so they fit pre-defined notions of what they represent. Dixon argues that there could be many other valuable notions derived from data found in its unprocessed, natural state.<\/p>\n

According to Gartner, the purpose of a data lake is to present an unrefined view of data so skilled analysts can apply their data mining techniques devoid of the \u201csystem-of-record compromises that may exist in a traditional analytic data store (such as a data mart or data warehouse).\u201d The data lake removes the constraints of relational structures when various forms of \u2018big data\u2019 need to be examined in new ways.<\/p>\n

Consolidating traditionally isolated data sources can also increase the sharing and use of information, and reduce the costs of hardware and software holding that data now.<\/p>\n

Data lakes thus provide the \u201copportunity\u201d to not have to prepare and protect all the data an organization gathers, and instead, just save it for later when new needs or ideas arise. Companies that build successful data lakes find they need to gradually mature their lake as they figure out which data and metadata are interesting to their organization.<\/p>\n

\"data<\/p>\n

 <\/p>\n

Red Flags – Governance and\u00a0Efficiency<\/h2>\n

There are naysayers who relegate the data lake to a mere notion, particularly since many organizations are unsuccessful with their deployments. Cambridge Semantics CTO Sean Martin said, “We see customers creating big data graveyards, dumping everything into HDFS and hoping to do something with it down the road. But then they just lose track of what\u2019s there.”<\/p>\n

As with any major data-driven initiative, the lake will have to be sold across the enterprise. Data lakes absorb data from a variety of sources and store it all in one place, and by definition, without the usual requirements for integration (like quality and lineage) or security. Someone will need to be accountable for governance. Data Vault inventor Dan Linstedt warns, for example:<\/p>\n

\n

Users of self-service BI tools trolling the lake have to be governed. Think about who gets to use which tool, who gets to log in where and access what data, or who can open a spreadsheet and upload data directly to Hadoop, and then make it available to the rest of the enterprise. That can be a serious problem.<\/em><\/span><\/p>\n<\/blockquote>\n

David Weldon’s April 2017 article in Information Management magazine, “Many Organizations Struggling to Manage Lakes<\/em>,” affirmed the issue in this quote from Zaloni’s CEO Ben Sharma:<\/p>\n

\n

Perhaps the biggest challenge organizations are facing is \u201cfinding, rationalizing and curating the data from across an enterprise for analytics solutions … the ability to easily access data, refine data and collaborate on data needs continues to be a large roadblock for many analytic applications.\u201d<\/em><\/span><\/p>\n<\/blockquote>\n

This is why governing data in a lake is important, which means, dealing with veracity, security and metadata lineage issues, to name a few. See De-Mucking the Lake, below.<\/p>\n

The other issue to consider is performance. Most <\/span>tools and data interfaces cannot ingest, process, or produce information in an unmanaged lake as well as data in fit-for-purpose (e.g., query optimized) environments. Thus, consistent semantics and an engine like CoSort will help.<\/p>\n

Stocking the Lake<\/h2>\n

Data enter the lake from various sources, including structured data from files and databases (rows and columns), semi-structured data (ASN.1, XML, JSON), unstructured data (emails, documents, and pdfs), or possibly images, audio, and video … thereby creating a centralized store for all forms of data.<\/p>\n

IRI Voracity<\/a> is a data connection and curation platform that can be used to populate a data lake by connecting to, profiling, and moving data in different sources<\/a> into the lake:<\/p>\n

\"IRI<\/a>
\nDuring or after movement, you can\u00a0select, transform, reformat, and report on data from disparate sources using jobs defined by scripts, wizards, dialogs, or diagrams in Eclipse. You can use Voracity to govern and test-analyze data in the lake, and to move data out of the lake.
\n<\/span>
\n<\/span><\/span><\/p>\n

Fishing the Lake<\/h2>\n

Once you\u2019ve defined the lake\u2019s location and what to pump into it, spend some time now and again to see what\u2019s currently in it. Consider what experiments can be run on that data. Use your own data discovery tools — or Voracity\u2019s flat-file, ODBC, and dark (document) data search, statistical, and relationship checking and diagramming tools — before \u201ctesting the water.\u201d<\/p>\n

Think of this aspect of Voracity as sonar, where you\u2019re trying to find different kinds of data and at different depths (various sources<\/a> in the lake). Voracity discovery tools<\/a> classify data, and allow you to fuzzy-search for values from, ODBC and file sources. They also search those and unstructured sources for: explicit strings, values conforming to canned or custom RegEx patterns, and values in a set (lookup) file. Those tools are actually free, since they only require Voracity\u2019s GUI (IRI Workbench), not an underlying CoSort or Hadoop transformation license.<\/p>\n

After identifying data in the lake that looks worthwhile, even data scientists can struggle deriving value from it without the benefit of semantic consistency or managed metadata. It is much harder to manipulate or analyze data without them. Voracity wizards auto-create metadata for the collections within, and build ETL, federation, masking, reformatting, and\/or reporting jobs that filter relevant data from the lake, transform it into useful information, and display<\/a> it.<\/p>\n

Voracity manipulates data with the CoSort engine (by default) or, optionally, in Hadoop with the same metadata. Plugins to Voracity\u2019s Eclipse \u201cWorkbench\u201d GUI also run: Python, R, SQL, Java, shell scripts, SQL procedures, and C\/C++ or Java programs. These tools enable you to do more with lake-related data and apps in the same GUI.<\/p>\n

De-Mucking the Lake<\/h2>\n

Recall that a key problem with data lakes, as with real lakes, is that people don\u2019t know what\u2019s in them, or how clean they are. In nature, unknown things in the water can kill the ecosystem. Unknown data dumped into a data lake can kill the project. Dan Linstedt again advises that:<\/p>\n

\n

If there\u2019s no structure, there\u2019s no understanding, and there\u2019s no vision for how to apply this data or even understand what you have. It needs to be classified and cleansed in order to do anything with it and turn it into value-added information for the business. To apply any sort of business information to this data, you must begin to stratify, profile, manage, and understand it, so that you can get results from it.<\/em><\/span><\/p>\n<\/blockquote>\n

In short you have to have enough trust in the data to trust your analysis. So it\u2019s better to know and manage what\u2019s in the water. If you use Voracity, you can discover, integrate, migrate, govern and analyze data in the lake — or prepare test or production-ready targets for other architectures, like a data warehouse, mart, or ODS — all within a managed metadata infrastructure.<\/p>\n

You also want to be able to\u00a0dredge the data lake clean, at least as much as you can, through various data cleansing operations. You can use Voracity to improve data quality in the lake in these ways:<\/p>\n