{"id":4138,"date":"2013-07-03T17:23:46","date_gmt":"2013-07-03T21:23:46","guid":{"rendered":"http:\/\/www.iri.com\/blog\/?p=4138"},"modified":"2017-11-06T08:30:09","modified_gmt":"2017-11-06T13:30:09","slug":"data-profiling","status":"publish","type":"post","link":"https:\/\/beta.iri.com\/blog\/data-transformation2\/data-profiling\/","title":{"rendered":"Data Profiling: Discovering Data Details"},"content":{"rendered":"

Data profiling<\/a>, or data discovery, refers to the process of obtaining information from, and descriptive statistics about, various sources of data.\u00a0 The purpose of data profiling is to get a better understanding of the content of data, as well as its structure, relationships, and current levels of accuracy and integrity.<\/p>\n

Data profiling may reveal errors in, or false conclusions around, metadata (data about data). Finding these problems early on helps improve the quality of source data prior to integrating or storing it in a data warehouse. Understanding the attributes of data in a database table or extracted file, and inspecting data values, helps validate that data content actually matches its metadata definition. Seeing the data and metadata also helps to identify which items are sensitive, or contain personally identifiable information (PII<\/a>), so that certain columns can be flagged for protective measures. Data profiling thus discovers the characteristics of source data necessary for the identification, use, and lineage of data in integration, security, reporting, and other processes that follow.<\/p>\n

Although collected data can oftentimes seem benign or useless, especially when gathered from multiple sources, keep in mind that all data may be useful with the proper application or algorithm. \u00a0Data profiling is thus also a first step in determining that usefulness (by improving understanding of the data itself).<\/p>\n

Since many businesses ultimately rely upon raw data sources for insight into things like product inventories, client demographics, buying habits, and sales projections, a company\u2019s ability to benefit competitively from ever-increasing data volumes can be directly proportional to its capacity to leverage those data assets. \u00a0Winning\/losing customers and succeeding\/failing as a business could very well be determined by the specific knowledge an organization\u2019s collected data imparts.\u00a0 Thus identifying the right data, establishing its usefulness at the right level, and determining how to manage anomalies — are essential in the design of data warehousing operations and business intelligence applications.<\/p>\n

According to Doug Vucevic and Wayne Yaddow, authors of Testing the Data Warehouse Practicum<\/a>, \u201c…the purpose of data profiling is both to validate metadata when it is available and to discover metadata when it is not. \u00a0The result of the analysis is used both strategically–to determine suitability of the candidate source systems and give the basis for an early go\/no-go decision, but tactically, to identify problems for later solution design, and to level sponsors\u2019 expectations.<\/em>\u201d<\/p>\n

Data authorities recommend performing data profiling randomly and repetitively on limited amounts of data, instead of trying to tackle large, complex volumes all at once. \u00a0That way the discoveries can be determining factors for what should be profiled next. \u00a0Identifying data rules, restrictions, and prerequisites, ensure the integrity of the metadata on which future profiling is performed. \u00a0Knowing what is supposed<\/i> to be in certain data files and what is actually<\/i> there may not be the same thing. \u00a0So whenever the quality or characteristics of a new source is unknown, experts suggest data profiling first, before any integration into an existing system.<\/p>\n

Steps in the data profiling process include: \u00a0importing all objects, creating configuration parameters, performing the actual profiling, and analyzing the results; none of which are as easy as they sound! \u00a0Then based upon the findings, schema and data corrections can be implemented, as well as other fine tuning for subsequent data profiling performance improvement.<\/p>\n

IRI Profiling Tools<\/strong><\/p>\n

In mid-2015, IRI released a series of free database, structured, and unstructured (dark) data discovery tools in its Eclipse GUI, IRI Workbench. They are summarized at\u00a0http:\/\/www.iri.com\/products\/workbench\/discover-data<\/a>\u00a0and link to other articles in this blog\u00a0which go into more detail.<\/p>\n

\"Data<\/a>
The metadata discovery wizard in the IRI Workbench, built on Eclipse. In this case, Oracle table structures are automatically parsed and generated for shared use in\u00a0data definition file (DDF<\/a>) repositories that support CoSort, FACT, NextForm, FieldShield, RowGen, BIRT and other operations. See a demo video here<\/a>.
<\/em><\/figcaption><\/figure>\n

 <\/p>\n

 <\/p>\n","protected":false},"excerpt":{"rendered":"

Data profiling, or data discovery, refers to the process of obtaining information from, and descriptive statistics about, various sources of data.\u00a0 The purpose of data profiling is to get a better understanding of the content of data, as well as its structure, relationships, and current levels of accuracy and integrity. Data profiling may reveal errors<\/p>\n

Read More<\/a><\/div>\n","protected":false},"author":19,"featured_media":11653,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_exactmetrics_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"footnotes":""},"categories":[108,8,31,363,1,91,232,233,29,3],"tags":[280,278,282,850,279,281,148,149,234,24],"class_list":["post-4138","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-big-data-2","category-data-protection","category-data-migration","category-data-quality","category-data-transformation2","category-iri-workbench","category-master-data-management","category-metadata-management","category-test-data","category-vldb-operations","tag-data-discovery","tag-data-profiling","tag-data-warehouse-management","tag-iri-workbench","tag-metadata-discovery","tag-metadata-management-2","tag-personally-identifiable-information","tag-pii","tag-very-large-database","tag-vldb"],"acf":[],"yoast_head":"\nData Profiling: Discovering Data Details - IRI<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/beta.iri.com\/blog\/data-transformation2\/data-profiling\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Data Profiling: Discovering Data Details - IRI\" \/>\n<meta property=\"og:description\" content=\"Data profiling, or data discovery, refers to the process of obtaining information from, and descriptive statistics about, various sources of data.\u00a0 The purpose of data profiling is to get a better understanding of the content of data, as well as its structure, relationships, and current levels of accuracy and integrity. Data profiling may reveal errorsRead More\" \/>\n<meta property=\"og:url\" content=\"https:\/\/beta.iri.com\/blog\/data-transformation2\/data-profiling\/\" \/>\n<meta property=\"og:site_name\" content=\"IRI\" \/>\n<meta property=\"article:published_time\" content=\"2013-07-03T21:23:46+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2017-11-06T13:30:09+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2013\/07\/graphs-and-charts.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1110\" \/>\n\t<meta property=\"og:image:height\" content=\"735\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Dale Robson\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Dale Robson\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/beta.iri.com\/blog\/data-transformation2\/data-profiling\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/data-transformation2\/data-profiling\/\"},\"author\":{\"name\":\"Dale Robson\",\"@id\":\"https:\/\/beta.iri.com\/blog\/#\/schema\/person\/f1d9bfa144982a6de71ac1b15113423c\"},\"headline\":\"Data Profiling: Discovering Data Details\",\"datePublished\":\"2013-07-03T21:23:46+00:00\",\"dateModified\":\"2017-11-06T13:30:09+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/data-transformation2\/data-profiling\/\"},\"wordCount\":661,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/#organization\"},\"image\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/data-transformation2\/data-profiling\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2013\/07\/graphs-and-charts.jpg\",\"keywords\":[\"data discovery\",\"data profiling\",\"data warehouse management\",\"IRI Workbench\",\"metadata discovery\",\"metadata management\",\"personally identifiable information\",\"PII\",\"very large database\",\"VLDB\"],\"articleSection\":[\"Big Data\",\"Data Masking\/Protection\",\"Data Migration\",\"Data Quality (DQ)\",\"Data Transformation\",\"IRI Workbench\",\"Master Data Management\",\"Metadata Management\",\"Test Data\",\"VLDB\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/beta.iri.com\/blog\/data-transformation2\/data-profiling\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/beta.iri.com\/blog\/data-transformation2\/data-profiling\/\",\"url\":\"https:\/\/beta.iri.com\/blog\/data-transformation2\/data-profiling\/\",\"name\":\"Data Profiling: Discovering Data Details - IRI\",\"isPartOf\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/data-transformation2\/data-profiling\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/data-transformation2\/data-profiling\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2013\/07\/graphs-and-charts.jpg\",\"datePublished\":\"2013-07-03T21:23:46+00:00\",\"dateModified\":\"2017-11-06T13:30:09+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/data-transformation2\/data-profiling\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/beta.iri.com\/blog\/data-transformation2\/data-profiling\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/beta.iri.com\/blog\/data-transformation2\/data-profiling\/#primaryimage\",\"url\":\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2013\/07\/graphs-and-charts.jpg\",\"contentUrl\":\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2013\/07\/graphs-and-charts.jpg\",\"width\":1110,\"height\":735,\"caption\":\"Graphs and Charts\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/beta.iri.com\/blog\/data-transformation2\/data-profiling\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/beta.iri.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Data Profiling: Discovering Data Details\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/beta.iri.com\/blog\/#website\",\"url\":\"https:\/\/beta.iri.com\/blog\/\",\"name\":\"IRI\",\"description\":\"Total Data Management Blog\",\"publisher\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/beta.iri.com\/blog\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/beta.iri.com\/blog\/#organization\",\"name\":\"IRI\",\"url\":\"https:\/\/beta.iri.com\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/beta.iri.com\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2019\/02\/iri-logo-total-data-management-small-1.png\",\"contentUrl\":\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2019\/02\/iri-logo-total-data-management-small-1.png\",\"width\":750,\"height\":206,\"caption\":\"IRI\"},\"image\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/beta.iri.com\/blog\/#\/schema\/person\/f1d9bfa144982a6de71ac1b15113423c\",\"name\":\"Dale Robson\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/beta.iri.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/859ab02a905e1a107adf4347e911a09f?s=96&d=blank&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/859ab02a905e1a107adf4347e911a09f?s=96&d=blank&r=g\",\"caption\":\"Dale Robson\"},\"url\":\"https:\/\/beta.iri.com\/blog\/author\/daler\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Data Profiling: Discovering Data Details - IRI","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/beta.iri.com\/blog\/data-transformation2\/data-profiling\/","og_locale":"en_US","og_type":"article","og_title":"Data Profiling: Discovering Data Details - IRI","og_description":"Data profiling, or data discovery, refers to the process of obtaining information from, and descriptive statistics about, various sources of data.\u00a0 The purpose of data profiling is to get a better understanding of the content of data, as well as its structure, relationships, and current levels of accuracy and integrity. Data profiling may reveal errorsRead More","og_url":"https:\/\/beta.iri.com\/blog\/data-transformation2\/data-profiling\/","og_site_name":"IRI","article_published_time":"2013-07-03T21:23:46+00:00","article_modified_time":"2017-11-06T13:30:09+00:00","og_image":[{"width":1110,"height":735,"url":"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2013\/07\/graphs-and-charts.jpg","type":"image\/jpeg"}],"author":"Dale Robson","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Dale Robson","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/beta.iri.com\/blog\/data-transformation2\/data-profiling\/#article","isPartOf":{"@id":"https:\/\/beta.iri.com\/blog\/data-transformation2\/data-profiling\/"},"author":{"name":"Dale Robson","@id":"https:\/\/beta.iri.com\/blog\/#\/schema\/person\/f1d9bfa144982a6de71ac1b15113423c"},"headline":"Data Profiling: Discovering Data Details","datePublished":"2013-07-03T21:23:46+00:00","dateModified":"2017-11-06T13:30:09+00:00","mainEntityOfPage":{"@id":"https:\/\/beta.iri.com\/blog\/data-transformation2\/data-profiling\/"},"wordCount":661,"commentCount":0,"publisher":{"@id":"https:\/\/beta.iri.com\/blog\/#organization"},"image":{"@id":"https:\/\/beta.iri.com\/blog\/data-transformation2\/data-profiling\/#primaryimage"},"thumbnailUrl":"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2013\/07\/graphs-and-charts.jpg","keywords":["data discovery","data profiling","data warehouse management","IRI Workbench","metadata discovery","metadata management","personally identifiable information","PII","very large database","VLDB"],"articleSection":["Big Data","Data Masking\/Protection","Data Migration","Data Quality (DQ)","Data Transformation","IRI Workbench","Master Data Management","Metadata Management","Test Data","VLDB"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/beta.iri.com\/blog\/data-transformation2\/data-profiling\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/beta.iri.com\/blog\/data-transformation2\/data-profiling\/","url":"https:\/\/beta.iri.com\/blog\/data-transformation2\/data-profiling\/","name":"Data Profiling: Discovering Data Details - IRI","isPartOf":{"@id":"https:\/\/beta.iri.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/beta.iri.com\/blog\/data-transformation2\/data-profiling\/#primaryimage"},"image":{"@id":"https:\/\/beta.iri.com\/blog\/data-transformation2\/data-profiling\/#primaryimage"},"thumbnailUrl":"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2013\/07\/graphs-and-charts.jpg","datePublished":"2013-07-03T21:23:46+00:00","dateModified":"2017-11-06T13:30:09+00:00","breadcrumb":{"@id":"https:\/\/beta.iri.com\/blog\/data-transformation2\/data-profiling\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/beta.iri.com\/blog\/data-transformation2\/data-profiling\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/beta.iri.com\/blog\/data-transformation2\/data-profiling\/#primaryimage","url":"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2013\/07\/graphs-and-charts.jpg","contentUrl":"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2013\/07\/graphs-and-charts.jpg","width":1110,"height":735,"caption":"Graphs and Charts"},{"@type":"BreadcrumbList","@id":"https:\/\/beta.iri.com\/blog\/data-transformation2\/data-profiling\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/beta.iri.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Data Profiling: Discovering Data Details"}]},{"@type":"WebSite","@id":"https:\/\/beta.iri.com\/blog\/#website","url":"https:\/\/beta.iri.com\/blog\/","name":"IRI","description":"Total Data Management Blog","publisher":{"@id":"https:\/\/beta.iri.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/beta.iri.com\/blog\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/beta.iri.com\/blog\/#organization","name":"IRI","url":"https:\/\/beta.iri.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/beta.iri.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2019\/02\/iri-logo-total-data-management-small-1.png","contentUrl":"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2019\/02\/iri-logo-total-data-management-small-1.png","width":750,"height":206,"caption":"IRI"},"image":{"@id":"https:\/\/beta.iri.com\/blog\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/beta.iri.com\/blog\/#\/schema\/person\/f1d9bfa144982a6de71ac1b15113423c","name":"Dale Robson","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/beta.iri.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/859ab02a905e1a107adf4347e911a09f?s=96&d=blank&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/859ab02a905e1a107adf4347e911a09f?s=96&d=blank&r=g","caption":"Dale Robson"},"url":"https:\/\/beta.iri.com\/blog\/author\/daler\/"}]}},"jetpack_featured_media_url":"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2013\/07\/graphs-and-charts.jpg","_links":{"self":[{"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/posts\/4138"}],"collection":[{"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/users\/19"}],"replies":[{"embeddable":true,"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/comments?post=4138"}],"version-history":[{"count":25,"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/posts\/4138\/revisions"}],"predecessor-version":[{"id":11654,"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/posts\/4138\/revisions\/11654"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/media\/11653"}],"wp:attachment":[{"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/media?parent=4138"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/categories?post=4138"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/tags?post=4138"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}