{"id":6011,"date":"2014-06-04T17:10:56","date_gmt":"2014-06-04T21:10:56","guid":{"rendered":"http:\/\/www.iri.com\/blog\/?p=6011"},"modified":"2024-01-05T15:15:34","modified_gmt":"2024-01-05T20:15:34","slug":"unstructured-data-data-restructuring-wizard","status":"publish","type":"post","link":"https:\/\/beta.iri.com\/blog\/migration\/data-migration\/unstructured-data-data-restructuring-wizard\/","title":{"rendered":"Using the Dark Data Discovery Wizard to Unlock Unstructured Data"},"content":{"rendered":"

Editor’s note … this article, first posted in 2014<\/strong> for the wizard it describes, has been updated as follows:<\/span><\/p>\n

    \n
  1. June 2015: This wizard was renamed from the Data Restructuring Wizard to the Dark Data Discovery Wizard, and was provided free in IRI Workbench<\/a> for users of IRI NextForm Lite<\/a>.<\/em><\/li>\n
  2. October 2018: This wizard is now also used with both the IRI CellShield Enterprise Edition (EE)<\/a> and IRI DarkShield<\/a> products for searching, extracting, and masking PII in multiple LAN-connected sources at once, and is being enhanced with value lookup, machine-learned NLP models for NER, and fuzzy search criteria. Additional blog content on DarkShield uses will follow.<\/em><\/li>\n
  3. April & July 2019: Updated UI images and instructions, updated file formats for DarkShield v2 and v3. IRI Voracity<\/a>\u00a0data management platform users can also leverage this wizard for textual ETL applications.<\/em><\/li>\n
  4. October 2020: This wizard was subsumed in the IRI DarkShield feature menu in IRI Workbench and renamed to the “New Dark Data Search\/Masking Job …” wizard, then described in this article<\/a>.<\/em><\/li>\n
  5. January 2024<\/strong>:<\/em> The wizard has been updated again for DarkShield V5 to support the upgraded data classification infrastructure and further ergonomic improvements. See this article<\/a> now.<\/em><\/li>\n<\/ol>\n

    The idea of dark data in unstructured sources and formats was introduced in\u00a0Finding Dark Data in Unstructured Sources<\/a>\u00a0(to introduce the IRI Data Restructuring Wizard). Recall that corporations and government agencies may have a lot of useful information trapped in these unstructured formats that can be\u00a0mashed up with other (usually structured) repositories and\u00a0mined for the benefit of operations, promotions, analytics, law enforcement, etc. However, some of these sources are difficult to parse, and the data they contain need\u00a0structure to be useful\u00a0in data integration and reporting contexts. This is where IRI’s\u00a0Dark Data Discovery\u00a0<\/em>Wizard\u00a0<\/em>is\u00a0useful; it unlocks and organizes dark\u00a0data so it can start driving real\u00a0value to the business.<\/p>\n

    The general idea is that, after parsing through the data in unstructured\u00a0sources,\u00a0you can output what you’re looking for into a structured text (flat) file, with its layouts automatically defined in a data definition file (.DDF). The file and its metadata repository are easily used and re-used by IRI software to integrate, transform, migrate, mask, and report on that data, and\/or feed it to other applications.<\/p>\n

    Note also that CoSort can query and join over flat files directly, or facilitate the creation and population of tables with\u00a0DBA-defined primary-foreign keys. In this way, dark data extracts can acquire form and relationships (structure) that can make it a lot more useful.<\/p>\n

    Using the Wizard<\/strong><\/p>\n

    The IRI\u00a0Dark Data Discovery<\/em>\u00a0wizard will\u00a0search every supported unstructured document type in every directory below the root network drive you specify. The search for your dark data is based on Data Classes<\/em>, which can contain any combination of regular expression patterns, lookup set files, Named Entity Recognition (NER) models, path filters for semi-structured files,\u00a0 area bounding boxes, and detected or recognized faces.<\/p>\n

    Here is a list of unstructured sources containing strings that the wizard can search, extract, and structure:<\/p>\n