{"id":5080,"date":"2014-06-10T13:03:57","date_gmt":"2014-06-10T17:03:57","guid":{"rendered":"http:\/\/www.iri.com\/blog\/?p=5080"},"modified":"2017-11-06T08:21:17","modified_gmt":"2017-11-06T13:21:17","slug":"output-restructuring-wizard-cosort","status":"publish","type":"post","link":"https:\/\/beta.iri.com\/blog\/data-transformation2\/output-restructuring-wizard-cosort\/","title":{"rendered":"Using CoSort on Restructured Data in the IRI Workbench"},"content":{"rendered":"

This is the final article in a three-part blog series\u00a0introducing IRI’s new data structuring\u00a0technology.\u00a0The first blog introduces “dark data” and the unstructured sources<\/a> the wizard can analyze. The second in the series offers a deeper overview of how the Data Restructuring wizard works<\/a> and this article\u00a0shows how that\u00a0newly structured data can be leveraged in the same GUI by various\u00a0IRI software products.<\/em><\/p>\n

The preceding article<\/a> discussed the use of the Data Restructuring wizard<\/a> in the IRI Workbench<\/a> GUI to parse data from unstructured data sources. Recall that we extracted the data based on search patterns into a structured text file, and automatically defined the new layout in a data definition file (.DDF) available for use in any IRI CoSort<\/a> or other IRI software application.<\/p>\n

This article picks up where that one left off … to show how a structured data file and its DDF can be used to process the extracted data in the same GUI for CoSort<\/a>. The data can be transformed, reformatted, protected at the field level, and reported upon in the same job script and I\/O pass in CoSort SortCL programs<\/a>. SortCL outputs can be sent to multiple file and database targets at once:<\/p>\n

\"IRI2014Schematic-1024x450\"<\/a><\/p>\n

 <\/p>\n

In our first application, we will sort our text file by phone number, and remove the duplicate record. We will also mask the first 12 digits in the credit card account numbers (green line in image below), and encrypt the social security numbers (purple line in image). A report header was added to clarify the output (red line).<\/p>\n

Note that the RegInfo.ddf metadata repository created by the Data Restructuring wizard is referenced in the input section to define the fields. In this example, the output is sent to standard out (stdout) to display in the console:<\/p>\n

\"restructure-data-wizard-a\"<\/a><\/p>\n

Changing the SortCL job script produces the duplicate records.<\/p>\n

\"restructured-data-wizard-b\"<\/a><\/p>\n

In conclusion, the unstructured data we first structured in the IRI Workbench\u00a0has now been re-structured once again in the same Eclipse GUI, and can be re-fit for any use. There are in fact no limits to the possible permutations in, and purposes for, SortCL job scripts, nor in other IRI Workbench products that use .DDF files, including:<\/p>\n