Like a DW (in which an ODS is often a component), data in an ODS can come from batch inputs, and is processed by extract, transform, and load (ETL), and data quality, operations:<\/span><\/p>\n The ODS can be a vital but complex component of an enterprise data warehouse (<\/span>EDW<\/span><\/a>). It is also a multi-purpose structure that enables transactional and decision support processing. Because the data originates from multiple sources, the integration often involves cleaning, resolving redundancy, and checking it against business rules for integrity.<\/span><\/p>\n A key difference between a DW and an ODS is temporality. Unlike in the warehouse, new data coming into the ODS overwrites existing data. That is because the business unit always needs to work with the most current data (versus aggregate data typical of a DW used for analytics); e.g., when a bank needs to cover, notify, and charge a customer whose account is overdrawn.<\/span> The ODS typically provides a non-historical, integrated view of data in legacy applications. It enables the business unit to complete transactional functions and\/or operational reporting with current snapshots of data at a specific level of granularity (atomicity).<\/span><\/p>\n ODS data is processed through a series of ETL operations to integrate, transform, and comply with a set of standards where data quality and uniformity are the goals. This data is usually kept in a relational DB so the business unit can access it immediately. And that DB has specific constraints, including referential integrity, to make sure the data is relatable.<\/span><\/p>\n The ODS DB is usually designed for low-level or atomic (indivisible) data, such as transactions and prices, with limited history that is captured “real time” or “near real time,” as opposed to the much greater volumes of data stored in the data warehouse, generally on a less-frequent basis. The pace of updates in a batch-oriented DW is usually too slow for operational requirements.<\/span><\/p>\n Unlike data in a <\/span>master data<\/span><\/a> store, ODS data is not passed back to operational systems. It may be passed into further operations and onto the DW for reporting. The ODS is an alternative to a decision support system application that accesses data directly from an online transaction processing (OLTP) system.<\/span><\/p>\n Unlike a DW, the ODS tends to focus on the operational requirements of a particular business process (for example, customer service). The ODS must also allow updates and propagate them back to the source system. A DW architecture, on the other hand, helps decision makers access and analyze historical and cross-functional (non-volatile) data, while supporting many different kinds of applications.<\/span><\/p>\n So again, the ODS is set of logically related data structures. Its data exists in an integrated, volatile state and at a non-historical granular level, so operational functions can be performed to meet specific business goals. Because the results of its operations are mission critical, and because it shares data and potential ETL workflows with a larger DW, the ODS must also run with the same data governance and management standards in place enterprise-wide.<\/span><\/p>\n To understand the purpose of the ODS and its appropriateness as a data integration paradigm, consider its primary attributes:<\/span><\/p>\n Through this analysis, and an understanding of what an ODS can do, the team can clearly articulate their issues and requirements. If the business unit\u2019s BPM analysis reveals transactional or operational issues, missed deadlines, data quality errors in the legacy sources, or an aged or poorly designed supporting application, chances are that an ODS is appropriate. The analysis should also help the team design and use the ODS to meet their specific business goals, while adhering to corporate data governance standards.<\/span><\/p>\n An ODS in a bank has, at any given time, one account balance for each checking account (courtesy of the checking account system), and one balance for each savings account (per what\u2019s provided by the savings account system).<\/span>1<\/sup><\/a><\/span> The various systems send the account balances periodically (e.g., at the end of each day) to the ODS where functions act on that data to generate alerts, fees, statements, and so on.<\/span><\/p>\n In this case, the ODS user also gets a central and complete point of reference for each customer\u2019s profile (such as his\/her basic information and account balances. It\u2019s not necessarily a <\/span>360 degree view of the customer<\/span><\/a>, since it only speaks to the information and transactions joined in this particular database, but it\u2019s still an ODS.<\/span><\/p>\n This example compares and contrasts the ODS and DW as well. Here, the ODS is acting as a batch-oriented DW, updating and replacing each datum that resides in it (and adding new data). But it is not keeping a running history of the measures it stores. You can implement this kind of ODS with batch-oriented middleware, reporting and OLAP tools. Or you can use one <\/span>platform<\/span><\/a>, like <\/span>IRI Voracity<\/span><\/a>, to connect to the sources, administer the DB(s), prototype and perform ETL jobs, cleanse, and<\/em> report.<\/span><\/p>\n A more advanced version of an ODS uses the push-pull approach of DW-enabled applications. That allows an informational DB to be refreshed in real-time (or near real-time). However, as you might imagine, that architecture is harder to implement.<\/span><\/p>\n In the era of big data, with newer (e.g., unstructured) data sources and Hadoop processing paradigms, the ODS has also been called an Enterprise Data Hub (EDH). Here is how a Hadoop distribution provider illustrates the EDH:<\/span><\/p>\n The implementation of an enterprise data hub (EDH), powered by Apache Hadoop, provides enterprises an ODS that unlocks value by processing and storing any data type at massive volumes\u2014eliminating the need to archive data\u2014while allowing for quick, familiar data access to end users and applications.<\/span><\/i><\/p><\/blockquote>\n IRI Voracity<\/span><\/a> is an end-to-end data management platform built on Eclipse and powered by IRI CoSort or Hadoop<\/span>3<\/sup><\/a><\/span><\/span>\u00a0engines for data discovery, integration, migration, governance, and analytic operations. As an infrastructure for connecting to, integrating, and managing data sources and DBs, Voracity is thus a central, ergonomic place to build and run an ODS or EDH.<\/span><\/p>\n In a traditional ODS, Voracity will:<\/span> In addition to fast ETL<\/a>\u00a0and other data management activities<\/a>, Voracity\u2019s Eclipse IDE, <\/span>IRI Workbench<\/span><\/a>, also supports development and execution of\u00a0SQL\u00a0and 3GL applications<\/a>.<\/span><\/p>\n The table below reflects the merits of a virtual approach to data warehousing, and what the Voracity data integration platform<\/a> specifically provides:<\/p>\n![\"ODS](\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2016\/08\/1-ODS-data-sources.png\")
<\/a><\/p>\n
\n<\/span><\/p>\nMore Details<\/b><\/h3>\n
When the ODS Makes Sense<\/b><\/h3>\n
\n
\n<\/span><\/i>The ODS contains data that is unique to a set of business functions in a given subject area.<\/li>\n<\/ol>\n\n
\n<\/span><\/i>Legacy application source data undergoes a set of ETL operations, which includes cleansing and transformation processes based on the business\u2019 rules for data quality and standardization.<\/li>\n<\/ol>\n\n
\n<\/span><\/i>The data in the ODS is up-to-date and shows the current status of data from the source applications.<\/li>\n<\/ol>\n\n
\n<\/span><\/i>Data in the ODS is primarily used to support operational business functions, and so it must contain the specific level of detail the business requires for those functions to be performed.<\/li>\n<\/ol>\n
\nThe best way <\/span>to determine if an ODS is an appropriate solution is for business analysts and the data management team to jointly assess the processes involved in completing transactions and providing operational reports. These assessments are most effective if they:<\/span><\/p>\n\n
ODS Example<\/b><\/h3>\n
Modern ODS Manifestations<\/b><\/h3>\n
<\/a>Source: Cloudera<\/span><\/i><\/p>\n
\n<\/span>In this model, the EDH is a more robust form of an ODS because it uses the Hadoop File System (HDFS) as a repository for structured and unstructured data, along with an elastic, multi-node computation environment for very high volumes. Big Industries founder Matthias Vallaey explains the benefit of the EDH in terms of both operational performance and analytics:<\/span>2<\/sup><\/a><\/span><\/span><\/span><\/p>\n
\nVoracity and the ODS or EDH<\/b><\/h3>\n
\n<\/span><\/p>\n\n