{"id":12154,"date":"2018-06-19T08:57:38","date_gmt":"2018-06-19T12:57:38","guid":{"rendered":"http:\/\/www.iri.com\/blog\/?p=12154"},"modified":"2018-09-13T08:52:57","modified_gmt":"2018-09-13T12:52:57","slug":"production-analytics-platform-process-on-par-with-information","status":"publish","type":"post","link":"https:\/\/beta.iri.com\/blog\/etl\/production-analytics-platform-process-on-par-with-information\/","title":{"rendered":"Production Analytic Platform #1\/4: Process on Par with Information"},"content":{"rendered":"

This is the first of a four-part series of blog articles examining the inherent tradeoffs between data processing and information storage and presentation within traditional ETL paradigms — from the ODS to the data lake. It explains the need for a more modern, blended approach between data integration and virtualization called the Production Analytic Platform, and the benefits of implementing it with IRI Voracity technology. The articles were written in mid-2018 by Dr. Barry Devlin of 9sight Consulting, a leading authority in data warehousing since 1988, and author of <\/span><\/i>\u201cBusiness unIntelligence: Insight and Innovation beyond Analytics and Big Data.\u201d<\/a>\u00a0A podcast and a video supporting these concepts can be found here<\/a>.<\/em><\/p>\n

\n

Whether it\u2019s data warehouses or data lakes, most data management experts believe that data (or, more correctly, information) is the foundation of business computing. In my IDEAL architecture<\/span>1<\/sup><\/a><\/span>, I consciously placed information as the underpinning \u201cthinking space\u201d of the three\u2014information, process and people\u2014described. I start with the information store and build up from there.<\/p>\n

My initial conversation with David Friedland of IRI (The CoSort Company) caused me to pause for thought. At 40 years young, IRI have a different viewpoint. Process, it seems, is their lodestone. IRI’s data manipulation software optimizes data processing into single, rapid-fire jobs and uses memory when possible (but can also stage data on disk). The thought is both familiar and unsettling to me.<\/p>\n

Familiar because business people always focus first on action and process. And I also say that my architecture can be read as a story: \u201cPeople process information.\u201d Unsettling because I struggle to see how one could start from process rather than information. But, what if process and information could be considered as equal\u2014and simultaneous\u2014partners in business computing?<\/p>\n

The question is far more than academic noodling. Consider a real-life decision about how to build and populate a data warehouse. The traditional approach starts with data definition and database design, followed by the use of an ETL (extract, transform and load) tool: information is followed by process. Information and process are handled separately. However, if you adopt a data warehouse automation approach, you start by sitting with business people and prototyping the data collection and transformation process, most of which actually occurs within the database itself. In this case, information and process are intermixed.<\/p>\n

The IRI Voracity<\/a> data management platform (powered by CoSort or Hadoop engines and built on Eclipse for key data discovery, integration, migration, governance, and analytic requirements) focuses almost entirely on the process of gathering and manipulating data. Perhaps due to its initial roots in the mainframe era, it avoids data stores and databases as far as possible: they are sources and sinks for the data that flows through the process. Everything else happens in memory at maximum speed, optimized for lowest processing cost, and described through a comprehensive language and set of metadata.<\/p>\n

Conceptually, IRI Voracity is the antithesis of a data warehouse; it might even be called a data hosepipe! In effect, Voracity is a data processing engine vs. a database engine. The contrast is so stark that it has forced me to revisit my logical concept of a \u201cProduction Analytic Platform\u201d and consider how to give information stores and data processing components equal prominence.<\/p>\n

The Production Analytic Platform<\/h2>\n

The Production Analytic Platform, which amalgamates the data warehouse and data mart concepts, springs from consideration of how business intelligence (BI) has largely morphed into analytics in the past half-decade or so. Coming from BI\u2019s rearview mirror on past performance, modern analytics demands current data to predict and prescribe future behavior. This demand for real-time data and immediate response challenges the longstanding division of business computing into operational and informational areas, shown below in Figure 1.<\/p>\n

Operational systems run the business and are optimized for read-write processing of data. Data warehouses (and marts) in the informational area track and manage the business, and are optimized for write-once \/ read-many processing. Data lakes\u2014where much of modern analytics occurs\u2014are largely informational in design. While ideal for the model-building phase of analytics, challenges arise when the resulting models must run continuously in a production environment to offer real-time predictions or prescriptions.<\/p>\n

Recognizing that both operational systems and data warehouses are today built on high-performance, production-ready relational database management systems (RDBMS), the Production Analytic Platform proposes that RDBMSs, with existing extensions to handle non-relation data and function (such as documents, key-value stores and graphs), are the preferred environment to handle the production phase of analytics, as shown in Figure 2. Further details of the rationale behind and characteristics of this approach can be found in my earlier paper<\/span>2<\/sup><\/a><\/span>.<\/p>\n

\"Evolution<\/a>
Figure 1: Evolution of the Production Analytic Platform<\/figcaption><\/figure>\n

Extending the Production Analytic Platform<\/h2>\n

Viewing the above figures through a process-oriented lens, a certain data-centric emphasis is evident. Data and its stores comprise the majority of the named components. Data flows are represented by unlabeled arrows, minimizing the significance of the processing that occurs within these flows. Where does this processing take place and how is it defined and managed? Indeed, some processing occurs\u2014almost unnoticed\u2014within the data stores themselves. For example, a primary output of a data warehouse is the reports that drive decisions and actions; producing a report requires, of course, processing of the data in the warehouse.<\/p>\n

Extending the Production Analytic Platform as shown in Figure 2 begins to address these issues. At first glance, drawing an additional, extended green box and recoloring and adding and combining some arrows may feel underwhelming! The emphasized arrows suggest that a variety of processing, preparation and delivery functions must be inherently considered in the design of the Production Analytic Platform:<\/p>\n