{"id":9849,"date":"2016-06-20T12:24:50","date_gmt":"2016-06-20T16:24:50","guid":{"rendered":"http:\/\/www.iri.com\/blog\/?p=9849"},"modified":"2017-11-06T13:45:06","modified_gmt":"2017-11-06T18:45:06","slug":"creating-voracity-flows-using-existing-iri-scripts-part-3","status":"publish","type":"post","link":"https:\/\/beta.iri.com\/blog\/data-transformation2\/creating-voracity-flows-using-existing-iri-scripts-part-3\/","title":{"rendered":"Creating a Voracity Flow Using Existing IRI Scripts (Part 3 of 3)"},"content":{"rendered":"<p>This is the third in a series of articles for creating an <a style=\"color: #1155cc; text-decoration: underline;\" href=\"http:\/\/www.iri.com\/products\/voracity\">IRI Voracity<\/a> ETL flow of a month-end job for processing sales transactions.<\/p>\n<p>In the <a href=\"http:\/\/www.iri.com\/blog\/iri\/iri-workbench\/creating-voracity-flows-using-existing-iri-scripts-part-1\/\">first article<\/a>, we brought an existing CoSort <a style=\"color: #1155cc; text-decoration: underline;\" href=\"http:\/\/www.iri.com\/products\/cosort\/sortcl\">SortCL<\/a> job script that processes month-end sales transactions into Voracity and made modifications. In the <a href=\"http:\/\/www.iri.com\/blog\/iri\/iri-workbench\/creating-voracity-flows-using-existing-iri-scripts-part-2\/\">second (previous) article<\/a>, we demonstrated how to:<\/p>\n<ol>\n<li style=\"list-style-type: none;\">\n<ol>\n<li style=\"line-height: 1.38;\">add another SortCL job script called MonthEndSales.scl to the existing Voracity flow diagram by adding a second Transform Mapping Block to the right of the one for the script SortTransSelect.scl<\/li>\n<li style=\"line-height: 1.38;\">connect the 2 transform mapping blocks where the output of one block is the input of the next block<\/li>\n<li style=\"line-height: 1.38;\">add a second output for the second transform mapping diagram<\/li>\n<li style=\"line-height: 1.38;\">define an aggregation for that output and connect the aggregation field to the field from which the aggregation was derived<\/li>\n<li style=\"line-height: 1.38;\">change the <span style=\"font-weight: bold;\">Output Write Type<\/span> for an output table so the table will not be truncated<\/li>\n<\/ol>\n<\/li>\n<\/ol>\n<p style=\"line-height: 1.38;\">In this article, we will expand the flow by using:<\/p>\n<ul>\n<li><span style=\"font-weight: bold;\">Command Line <\/span>blocks where we can do things like define environment variables, write to log files, and call programs<\/li>\n<li><span style=\"font-weight: bold;\">Decision <\/span>blocks where the path that is followed by the flow is decided by whether a condition is true or false<\/li>\n<\/ul>\n<p>Currently, the flow contains a <span style=\"font-weight: bold;\">Start<\/span> block and two <span style=\"font-weight: bold;\">Transform Mapping <\/span>blocks that represent two SortCL job scripts. A batch file that sequentially runs the two scripts has been created from the flow. At the same time, the two job scripts were updated.<span style=\"line-height: 1.5;\">\u00a0<\/span><\/p>\n<p>Below is the flow diagram from the end of the previous article in this series:<\/p>\n<p><a title=\"Creating Voracity Flows Using Existing IRI Scripts (Part 3) - Susan Gegner 0.jpg\" href=\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2016\/06\/Creating-Voracity-Flows-Using-Existing-IRI-Scripts-Part-3-Susan-Gegner-0.jpg\" rel=\"prettyPhoto\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter\" style=\"width: 624px; height: 321px;\" title=\"IRI Flow Diagram\" src=\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2016\/06\/t_Creating-Voracity-Flows-Using-Existing-IRI-Scripts-Part-3-Susan-Gegner-0.jpg\" alt=\"IRI Flow Diagram\" width=\"624\" height=\"321\" \/><\/a><\/p>\n<p>Here is the batch script for the month end job as it now exists.<\/p>\n<pre>@echo off\r\nsortcl\/SPECIFICATION=SortTransSelect.scl\r\nsortcl\/SPECIFICATION=MonthEndSales.scl\r\n<\/pre>\n<p>We now want to:<\/p>\n<ul>\n<li>create a log file that tracks progress of the job<\/li>\n<li>define environment variables for use in a SortCL job script and in a batch script<\/li>\n<li>give the exit status for the SortCL scripts<\/li>\n<li>record the statistics for each SortCL script in the log file<\/li>\n<li>decide whether to execute the second script based on the exit status of the first script<\/li>\n<li>record the start and end time for the job<\/li>\n<\/ul>\n<p>This is done with a combination of <span style=\"font-weight: bold;\">Command Line<\/span> blocks, <span style=\"font-weight: bold;\">Decision<\/span> blocks, and small batch files.<\/p>\n<p>.<span style=\"font-size: 12pt; font-weight: bold;\"><span style=\"font-size: 12pt; font-weight: bold;\">Remove the Start Block Connector<\/span><\/span><\/p>\n<p>We need to remove connectors where we are adding additional blocks. We are adding two <span style=\"font-weight: bold;\">Command Line<\/span> blocks between the <span style=\"font-weight: bold;\">Start <\/span>block and the first transform mapping block. Right-click on the connector arrow (it will turn blue) &gt; <span style=\"font-weight: bold;\">Edit<\/span> &gt; <span style=\"font-weight: bold;\">Delete from Model<\/span>.<\/p>\n<p style=\"font-size: 12pt; font-weight: bold;\"><span style=\"font-size: 12pt; font-weight: bold;\"><span style=\"font-size: 12pt; font-weight: bold;\">Create the First Command Line Block<\/span><\/span><\/p>\n<p><span style=\"font-weight: bold;\">Command Line<\/span> blocks define the execution of an operating system command. To specify more than one command at a time, you could:<\/p>\n<ul>\n<li>Create multiple sequential blocks<\/li>\n<li>Concatenate the commands with an ampersand (&amp;)<\/li>\n<li>Call a batch file containing the commands<\/li>\n<\/ul>\n<p>From the <span style=\"font-weight: bold;\">Palette<\/span>, drag the <span style=\"font-weight: bold;\">Command Line<\/span> utility into the space between the <span style=\"font-weight: bold;\">Start<\/span> block and the SortTransSelect.scl <span style=\"font-weight: bold;\">Transform Mapping Block<\/span>. In this block, we want to set the two date values that are used in the \/QUERY statement of the script SortTransSelect.scl using environment variables. This is because the job is run each month and the values will change based on the month being processed.<\/p>\n<p>Double click on the command line block, type <span style=\"font-style: italic;\">SetDates<\/span> for the <span style=\"font-weight: bold;\">Name<\/span>, type <span style=\"font-family: Courier New;\">set STARTDATE=161201 &amp; set ENDDATE=170101<\/span> in <span style=\"font-weight: bold;\">Command<\/span>, and click <span style=\"font-weight: bold;\">Finish<\/span>. This sets the two environment variables STARTDATE and ENDDATE. Below is the <span style=\"font-weight: bold;\">Command Line<\/span> dialog.<\/p>\n<p align=\"center\"><a title=\"Creating Voracity Flows Using Existing IRI Scripts (Part 3) - Susan Gegner 1.jpg\" href=\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2016\/06\/Creating-Voracity-Flows-Using-Existing-IRI-Scripts-Part-3-Susan-Gegner-1.jpg\" rel=\"prettyPhoto\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter\" style=\"width: 413px; height: 369px;\" title=\"Editing of the properties of an object CommandLine\" src=\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2016\/06\/t_Creating-Voracity-Flows-Using-Existing-IRI-Scripts-Part-3-Susan-Gegner-1.jpg\" alt=\"Editing of the properties of an object CommandLine\" width=\"413\" height=\"369\" \/><\/a><\/p>\n<p style=\"font-size: 12pt; font-weight: bold;\"><span style=\"font-size: 12pt; font-weight: bold;\"><span style=\"font-size: 12pt; font-weight: bold;\">Create the Second Command Line Block<\/span><\/span><span style=\"line-height: 1.5;\">\u00a0<\/span><\/p>\n<p>From the <span style=\"font-weight: bold;\">Palette<\/span> drag the <span style=\"font-weight: bold;\">Command Line<\/span> utility component between the <span style=\"font-weight: bold;\">SetDates<\/span> block and the SortTransSelect.scl <span style=\"font-weight: bold;\">Transform Mapping Block<\/span>. In this block, we call a batch file that:<\/p>\n<ol>\n<li style=\"list-style-type: none;\">\n<ol>\n<li>Logs the system start time for the MonthEnd Process to a file called time.log<\/li>\n<li>Creates the file MonthEnd.log by declaring the start of the MonthEnd job<\/li>\n<li>Saves the date interval for the transactions being processed using environment variables set in the preceding <span style=\"font-weight: bold;\">Command Line<\/span> block to the file called MonthEnd.log.<\/li>\n<\/ol>\n<\/li>\n<\/ol>\n<p>Create and save the file begin.bat with an IRI Workbench text editor. Enter these contents:<\/p>\n<pre>echo Start time for MonthEnd process: %TIME% &gt; time.log\r\necho Start MonthEnd Process &gt;&gt; MonthEnd.log\r\necho From %STARTDATE% up to %ENDDATE% &gt;&gt; MonthEnd.log\r\n<\/pre>\n<p>The environment variable %TIME% gives the current system time.<\/p>\n<p>Double click on the <span style=\"font-weight: bold;\">Command Line<\/span> block, type <span style=\"font-style: italic;\">BeginMonthEnd <\/span>for the <span style=\"font-weight: bold;\">Name<\/span>, type <span style=\"font-style: italic;\">call begin.bat <\/span>for <span style=\"font-weight: bold;\">Command<\/span>, and click <span style=\"font-weight: bold;\">Finish<\/span>. Now connect <span style=\"font-weight: bold;\">Start<\/span> to <span style=\"font-style: italic;\">SetDates <\/span>to <span style=\"font-style: italic;\">BeginMonthEnd <\/span>to the infile for <span style=\"font-style: italic;\">SortTransSelect.scl<\/span>.<\/p>\n<p style=\"font-size: 12pt; font-weight: bold;\"><span style=\"font-size: 12pt; font-weight: bold;\"><span style=\"font-size: 12pt; font-weight: bold;\">Use Date Environment Variables<\/span><\/span><\/p>\n<p>Double-click on the SortTransSelect.scl <span style=\"font-weight: bold;\">Transform Mapping Block<\/span>, name it SortTrans Transform Mapping Diagram. In the <span style=\"font-weight: bold;\">Section Options<\/span> of the input, double-click on the <span style=\"font-weight: bold;\">Query<\/span>.<\/p>\n<p>In the <span style=\"font-weight: bold;\">Query<\/span> dialog, change <span style=\"font-style: italic;\">161201<\/span> to $STARTDATE and <span style=\"font-style: italic;\">170101<\/span> to $ENDDATE, then click <span style=\"font-weight: bold;\">Finish<\/span>. Save the diagram.<\/p>\n<p>Notice that we are not using the syntax for windows environment variables where the variable is encased with a percent sign(%). The syntax here is the leading dollar sign($) used with Linux and Unix operating systems because this is a requirement of the SortCL scripting language.<\/p>\n<p style=\"font-size: 12pt; font-weight: bold;\"><span style=\"font-size: 12pt; font-weight: bold;\"><span style=\"font-size: 12pt; font-weight: bold;\">Create the Third Command Line Block<\/span><\/span><\/p>\n<p>Remove the connector from the first TransForm Mapping Block to the second.<\/p>\n<p>Create a file named <span style=\"font-style: italic;\">status1.bat<\/span> using a text editor. This file is called in a new <span style=\"font-weight: bold;\">Command Line<\/span> block and has lines that:<\/p>\n<ul>\n<li>use the command <span style=\"font-style: italic;\">echo.<\/span> to create blank lines in the log file<\/li>\n<li>use other <span style=\"font-style: italic;\">echo<\/span> commands to create lines of demarcation<\/li>\n<li>capture the exit status or ERRORLEVEL from the SortCL script SortTransSelect.scl and save it to an environment variable<\/li>\n<li>write the exit status that was captured in the previous environment variable to the log file<\/li>\n<li>send the statistics file associated with the SortCL script to the log file<\/li>\n<\/ul>\n<p>The lines in the file status1.bat are:<\/p>\n<pre>set SORTTRANSSTATUS=%ERRORLEVEL%\r\necho. &gt;&gt; MonthEnd.log\r\necho. &gt;&gt; MonthEnd.log\r\necho ================================================================ &gt;&gt; MonthEnd.log\r\necho %SORTTRANSSTATUS% is the exit Status for SortTransSelect.scl &gt;&gt; MonthEnd.log\r\ntype monthend.stat &gt;&gt; MonthEnd.log\r\necho ================================================================ &gt;&gt; MonthEnd.log\r\n<\/pre>\n<p>Drag the <span style=\"font-weight: bold;\">Command Line <\/span>utility from the<span style=\"font-weight: bold;\"> Palette <\/span>into the space between the two<span style=\"font-weight: bold;\"> Transform Mapping <\/span>blocks. Double-click on the block, type <span style=\"font-style: italic;\">Status1<\/span> for the<span style=\"font-weight: bold;\"> Name<\/span>, enter <span style=\"font-family: Courier New;\">call status1.bat<\/span> for the <span style=\"font-weight: bold;\">Command<\/span>, and click<span style=\"font-weight: bold;\"> Finish<\/span>.<\/p>\n<p style=\"font-size: 12pt; font-weight: bold;\"><span style=\"font-size: 12pt; font-weight: bold;\"><span style=\"font-size: 12pt; font-weight: bold;\">Create a Decision Block<\/span><\/span><\/p>\n<p>A <span style=\"font-weight: bold;\">Decision <\/span>block directs the flow based on whether a condition is true or false. In our flow, the decision will be based on the exit status of the SortCL script SortTransSelect.scl. If that job executes without errors (status of 0), the flow continues to the script MonthEndSales.scl. If the status is not 0, then the flow goes to the end of the job without executing anything further.<\/p>\n<p>All blocks that the <span style=\"font-weight: bold;\">Decision <\/span>block references must be created first. Therefore, we need to create the <span style=\"font-style: italic;\">End_of_Process<\/span> block. Drag a <span style=\"font-weight: bold;\">Command Line<\/span> utility from the <span style=\"font-weight: bold;\">Palette<\/span> into the space between and below the <span style=\"font-style: italic;\">Status1<\/span> and <span style=\"font-style: italic;\">MonthendSales.scl<\/span> blocks. Create a .bat file called <span style=\"font-style: italic;\">end.bat<\/span>. This file will create the final entries for the log file. It will write the Start Time and the End Time for the MonthEnd job to the log file and will indicate that the job has ended.<span style=\"line-height: 1.5;\">\u00a0<\/span><\/p>\n<p>Here are the lines in end.bat:<\/p>\n<pre>echo. &gt;&gt; MonthEnd.log\r\necho. &gt;&gt; MonthEnd.log\r\necho ================================================================== &gt;&gt; MonthEnd.log\r\ntype time.log &gt;&gt; MonthEnd.log\r\necho End time for MonthEnd process: %TIME% &gt;&gt; MonthEnd.log\r\necho ================================================================== &gt;&gt; MonthEnd.log\r\necho End of MonthEnd process &gt;&gt; MonthEnd.log\r\n<\/pre>\n<p>Double-click on the <span style=\"font-weight: bold;\">Command Line<\/span> block, name it <span style=\"font-style: italic;\">End_of_Process<\/span>, type<span style=\"font-style: italic;\"> call end.bat <\/span>for <span style=\"font-weight: bold;\">Command<\/span>, click <span style=\"font-weight: bold;\">Finish<\/span>.<\/p>\n<p>Drag the <span style=\"font-weight: bold;\">Decision<\/span> utility into the space between <span style=\"font-style: italic;\">Status1<\/span> and <span style=\"font-style: italic;\">MonthEndSales.scl<\/span>. Fill out the <span style=\"font-weight: bold;\">Decision<\/span> dialog: type <span style=\"font-family: Courier New;\">%SORTTRANSSTATUS% equ 0<\/span> for <span style=\"font-weight: bold;\">Criteria<\/span>. Select the block <span style=\"font-style: italic;\">MonthEndSales.scl<\/span> from the drop-down for <span style=\"font-weight: bold;\">True<\/span>, select the block <span style=\"font-style: italic;\">End_of_Process<\/span> from the drop-down for <span style=\"font-weight: bold;\">False<\/span>, and click <span style=\"font-weight: bold;\">Finish<\/span>.<\/p>\n<p align=\"center\"><a title=\"Creating Voracity Flows Using Existing IRI Scripts (Part 3) - Susan Gegner 2.jpg\" href=\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2016\/06\/Creating-Voracity-Flows-Using-Existing-IRI-Scripts-Part-3-Susan-Gegner-2.jpg\" rel=\"prettyPhoto\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter\" style=\"width: 486px; height: 401px;\" title=\"Check for conformity to criteria and true and false paths.\" src=\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2016\/06\/t_Creating-Voracity-Flows-Using-Existing-IRI-Scripts-Part-3-Susan-Gegner-2.jpg\" alt=\"Check for conformity to criteria and true and false paths.\" width=\"486\" height=\"401\" \/><\/a><\/p>\n<p>There are arrows connecting to the blocks for True and False. Now connect an output file from <span style=\"font-style: italic;\">SortTransSelect.scl<\/span> to <span style=\"font-style: italic;\">Status1 <\/span>to the <span style=\"font-weight: bold;\">Decision<\/span> block.<\/p>\n<p style=\"font-size: 12pt; font-weight: bold;\"><span style=\"font-size: 12pt; font-weight: bold;\"><span style=\"font-size: 12pt; font-weight: bold;\">Create Another Command Line Status Block<\/span><\/span><\/p>\n<p>Create another .bat file similar to <span style=\"font-style: italic;\">status1.bat<\/span> and call it <span style=\"font-style: italic;\">status2.bat<\/span>. The lines in it are:<\/p>\n<pre>set MONTHENDSALESSTATUS=%ERRORLEVEL%\r\necho. &gt;&gt; MonthEnd.log\r\necho. &gt;&gt; MonthEnd.log\r\necho ================================================================ &gt;&gt; MonthEnd.log\r\necho %MONTHENDSALESSTATUS% is the exit Status for MonthEndSales.scl &gt;&gt; MonthEnd.log\r\necho ================================================================ &gt;&gt; MonthEnd.log\r\necho Below are the SortCL statistics for MonthEndSales.scl &gt;&gt; MonthEnd.log\r\ntype monthend.stat &gt;&gt; MonthEnd.log\r\n<\/pre>\n<p>Drag a <span style=\"font-weight: bold;\">Command Line<\/span> utility from the <span style=\"font-weight: bold;\">Palette <\/span>into the space below <span style=\"font-style: italic;\">MonthEndSales.scl<\/span> and to the right of <span style=\"font-style: italic;\">End_of_Process<\/span>. Double click on the block, name it <span style=\"font-style: italic;\">Status2<\/span>, type <span style=\"font-family: Courier New;\">call status2.bat<\/span> for <span style=\"font-weight: bold;\">Command<\/span>, and click <span style=\"font-weight: bold;\">Finish<\/span>.<\/p>\n<p>Add the connectors an output file of <span style=\"font-style: italic;\">MonthEndSales.scl<\/span> to <span style=\"font-style: italic;\">Status2<\/span> to <span style=\"font-style: italic;\">End_of_Process.<\/span><\/p>\n<p>We do not need a second decision block because the flow will go to the <span style=\"font-style: italic;\">End_of_Process<\/span> block whether or not the script <span style=\"font-style: italic;\">MonthEndSales.scl<\/span> executes without errors<span style=\"font-weight: bold;\">.<\/span><\/p>\n<p style=\"font-size: 12pt; font-weight: bold;\"><span style=\"font-size: 12pt; font-weight: bold;\"><span style=\"font-size: 12pt; font-weight: bold;\">Create the Flow Batch File<\/span><\/span><\/p>\n<p>Here is the completed flow diagram:<\/p>\n<p style=\"font-weight: bold;\"><a title=\"Creating Voracity Flows Using Existing IRI Scripts (Part 3) - Susan Gegner 3.jpg\" href=\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2016\/06\/Creating-Voracity-Flows-Using-Existing-IRI-Scripts-Part-3-Susan-Gegner-3.jpg\" rel=\"prettyPhoto\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter\" style=\"width: 624px; height: 296px;\" title=\"Completed flow diagram\" src=\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2016\/06\/t_Creating-Voracity-Flows-Using-Existing-IRI-Scripts-Part-3-Susan-Gegner-3.jpg\" alt=\"Completed flow diagram\" width=\"624\" height=\"296\" \/><\/a><\/p>\n<p>Now create the batch file MonthEnd.bat by right-clicking the flowlet and selecting <span style=\"font-weight: bold;\">IRI Diagram Actions<\/span> &gt; <span style=\"font-weight: bold;\">Export Flow Component<\/span>. On the <span style=\"font-weight: bold;\">Batch Component Options<\/span> dialog, verify the project, confirm the <span style=\"font-weight: bold;\">File name <\/span>is <span style=\"font-style: italic;\">MonthEnd.bat<\/span>, and click <span style=\"font-weight: bold;\">Finish.<\/span> Answer <span style=\"font-weight: bold;\">Yes <\/span>to any save prompts.<\/p>\n<p>There is now a new Monthend.bat file in the project.<\/p>\n<pre>@echo off\r\nset STARTDATE=161201 &amp; set ENDDATE=170101\r\ncall begin.bat\r\nsortcl\/SPECIFICATION=SortTransSelect.scl\r\ncall status1.bat\r\nIF %SORTTRANSSTATUS% equ 0 (sortcl \/SPECIFICATION=MonthEndSales.scl\r\ncall status2.bat\r\ncall end.bat\r\n)ELSE (call end.bat\r\n)\r\n<\/pre>\n<p>To execute, right-click on <span style=\"font-style: italic;\">MonthEnd.bat<\/span> in the <span style=\"font-weight: bold;\">Project Explorer<\/span>, select <span style=\"font-weight: bold;\">Run as<\/span> =&gt; <span style=\"font-weight: bold;\">Batch Program<\/span>. The batch script runs executing each command and script in sequence and writes to the file <span style=\"font-style: italic;\">MonthEnd.log<\/span>. Below is the partial contents of the log file:<\/p>\n<pre>Start MonthEnd Process\r\nFrom 161201 up to 170101\r\n\r\n==================================================================\r\n0 is the exit Status for SortTransSelect.scl\r\n==================================================================\r\n\r\nBelow are the SortCL statistics for SortTransSelect.scl\r\n___________________________________________________________________\r\n\r\nCoSort Version 9.5.3 R95160317-1135 32B SortCL STATISTICS\r\n. . . . . . . . . . . .\r\nRecords processed: 14 read 14 kept 12 sorted 24 output\r\n\r\nBegan: 16:32:18\r\nEnded: 16:32:18\r\nTotal: 00:00:00.22\r\n___________________________________________________________________\r\n\r\n==================================================================\r\n0 is the exit Status for MonthEndSales.scl.\r\n\r\nBelow are the SortCL statistics for MonthEndSales.scl:\r\n___________________________________________________________________\r\n\r\nCoSort Version 9.5.3 R95160317-1135 32B SortCL STATISTICS\r\n. . . . . . . . . . . .\r\nRecords processed: 1 sorted 4 output\r\n\r\nBegan: 16:32:18\r\nEnded: 16:32:19\r\nTotal: 00:00:00.26\r\n==================================================================\r\n\r\nStart time for MonthEnd process: 16:32:18.61\r\nEnd time for MonthEnd process: 16:32:19.16\r\n\r\n==================================================================\r\n\r\nEnd of MonthEnd process\r\n<\/pre>\n<p>This shows that the MonthEnd job starts giving the date interval that was specified by the environment variables we defined. Each script ran without errors and the statistics for each are in the log file. The Start and End time for the whole job is provided at the end of the log file.<\/p>\n<p>Remember to contact <a href=\"mailto:voracity@iri.com\">voracity@iri.com<\/a> if you have any questions or need help with your flow.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>This is the third in a series of articles for creating an IRI Voracity ETL flow of a month-end job for processing sales transactions. In the first article, we brought an existing CoSort SortCL job script that processes month-end sales transactions into Voracity and made modifications. In the second (previous) article, we demonstrated how to:<\/p>\n<div><a class=\"btn-filled btn\" href=\"https:\/\/beta.iri.com\/blog\/data-transformation2\/creating-voracity-flows-using-existing-iri-scripts-part-3\/\" title=\"Creating a Voracity Flow Using Existing IRI Scripts (Part 3 of 3)\">Read More<\/a><\/div>\n","protected":false},"author":10,"featured_media":9851,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_exactmetrics_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"footnotes":""},"categories":[1,776,91],"tags":[1095,71,100,1059,1093,1005,92,546,789,850,537,50,1096,68,977,1094,1092],"class_list":["post-9849","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-data-transformation2","category-etl","category-iri-workbench","tag-action-key","tag-eclipse","tag-etl","tag-field-editor","tag-flow","tag-flow-diagram","tag-gui","tag-iri-cosort","tag-iri-voracity","tag-iri-workbench","tag-job-script","tag-oracle","tag-sort-key","tag-sortcl","tag-sql","tag-transform-mapping-diagram","tag-wb"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v23.3 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Creating a Voracity Flow Using Existing IRI Scripts (Part 3 of 3) - IRI<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/beta.iri.com\/blog\/data-transformation2\/creating-voracity-flows-using-existing-iri-scripts-part-3\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Creating a Voracity Flow Using Existing IRI Scripts (Part 3 of 3) - IRI\" \/>\n<meta property=\"og:description\" content=\"This is the third in a series of articles for creating an IRI Voracity ETL flow of a month-end job for processing sales transactions. In the first article, we brought an existing CoSort SortCL job script that processes month-end sales transactions into Voracity and made modifications. In the second (previous) article, we demonstrated how to:Read More\" \/>\n<meta property=\"og:url\" content=\"https:\/\/beta.iri.com\/blog\/data-transformation2\/creating-voracity-flows-using-existing-iri-scripts-part-3\/\" \/>\n<meta property=\"og:site_name\" content=\"IRI\" \/>\n<meta property=\"article:published_time\" content=\"2016-06-20T16:24:50+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2017-11-06T18:45:06+00:00\" \/>\n<meta name=\"author\" content=\"Susan Gegner\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Susan Gegner\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"9 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/beta.iri.com\/blog\/data-transformation2\/creating-voracity-flows-using-existing-iri-scripts-part-3\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/data-transformation2\/creating-voracity-flows-using-existing-iri-scripts-part-3\/\"},\"author\":{\"name\":\"Susan Gegner\",\"@id\":\"https:\/\/beta.iri.com\/blog\/#\/schema\/person\/87be5da567628ab9396ca81170f36d63\"},\"headline\":\"Creating a Voracity Flow Using Existing IRI Scripts (Part 3 of 3)\",\"datePublished\":\"2016-06-20T16:24:50+00:00\",\"dateModified\":\"2017-11-06T18:45:06+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/data-transformation2\/creating-voracity-flows-using-existing-iri-scripts-part-3\/\"},\"wordCount\":1506,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/#organization\"},\"image\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/data-transformation2\/creating-voracity-flows-using-existing-iri-scripts-part-3\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2016\/06\/Creating-Voracity-Flows-Using-Existing-IRI-Scripts-Part-3-Susan-Gegner-0.jpg\",\"keywords\":[\"action key\",\"Eclipse\",\"ETL\",\"field editor\",\"flow\",\"flow diagram\",\"GUI\",\"IRI CoSort\",\"IRI Voracity\",\"IRI Workbench\",\"job script\",\"Oracle\",\"sort key\",\"SortCL\",\"SQL\",\"transform mapping diagram\",\"WB\"],\"articleSection\":[\"Data Transformation\",\"ETL\",\"IRI Workbench\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/beta.iri.com\/blog\/data-transformation2\/creating-voracity-flows-using-existing-iri-scripts-part-3\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/beta.iri.com\/blog\/data-transformation2\/creating-voracity-flows-using-existing-iri-scripts-part-3\/\",\"url\":\"https:\/\/beta.iri.com\/blog\/data-transformation2\/creating-voracity-flows-using-existing-iri-scripts-part-3\/\",\"name\":\"Creating a Voracity Flow Using Existing IRI Scripts (Part 3 of 3) - IRI\",\"isPartOf\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/data-transformation2\/creating-voracity-flows-using-existing-iri-scripts-part-3\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/data-transformation2\/creating-voracity-flows-using-existing-iri-scripts-part-3\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2016\/06\/Creating-Voracity-Flows-Using-Existing-IRI-Scripts-Part-3-Susan-Gegner-0.jpg\",\"datePublished\":\"2016-06-20T16:24:50+00:00\",\"dateModified\":\"2017-11-06T18:45:06+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/data-transformation2\/creating-voracity-flows-using-existing-iri-scripts-part-3\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/beta.iri.com\/blog\/data-transformation2\/creating-voracity-flows-using-existing-iri-scripts-part-3\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/beta.iri.com\/blog\/data-transformation2\/creating-voracity-flows-using-existing-iri-scripts-part-3\/#primaryimage\",\"url\":\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2016\/06\/Creating-Voracity-Flows-Using-Existing-IRI-Scripts-Part-3-Susan-Gegner-0.jpg\",\"contentUrl\":\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2016\/06\/Creating-Voracity-Flows-Using-Existing-IRI-Scripts-Part-3-Susan-Gegner-0.jpg\",\"width\":624,\"height\":322},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/beta.iri.com\/blog\/data-transformation2\/creating-voracity-flows-using-existing-iri-scripts-part-3\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/beta.iri.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Creating a Voracity Flow Using Existing IRI Scripts (Part 3 of 3)\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/beta.iri.com\/blog\/#website\",\"url\":\"https:\/\/beta.iri.com\/blog\/\",\"name\":\"IRI\",\"description\":\"Total Data Management Blog\",\"publisher\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/beta.iri.com\/blog\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/beta.iri.com\/blog\/#organization\",\"name\":\"IRI\",\"url\":\"https:\/\/beta.iri.com\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/beta.iri.com\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2019\/02\/iri-logo-total-data-management-small-1.png\",\"contentUrl\":\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2019\/02\/iri-logo-total-data-management-small-1.png\",\"width\":750,\"height\":206,\"caption\":\"IRI\"},\"image\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/beta.iri.com\/blog\/#\/schema\/person\/87be5da567628ab9396ca81170f36d63\",\"name\":\"Susan Gegner\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/beta.iri.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/2b1ca5592a65d44483351292cf1ae00a?s=96&d=blank&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/2b1ca5592a65d44483351292cf1ae00a?s=96&d=blank&r=g\",\"caption\":\"Susan Gegner\"},\"url\":\"https:\/\/beta.iri.com\/blog\/author\/susang\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Creating a Voracity Flow Using Existing IRI Scripts (Part 3 of 3) - IRI","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/beta.iri.com\/blog\/data-transformation2\/creating-voracity-flows-using-existing-iri-scripts-part-3\/","og_locale":"en_US","og_type":"article","og_title":"Creating a Voracity Flow Using Existing IRI Scripts (Part 3 of 3) - IRI","og_description":"This is the third in a series of articles for creating an IRI Voracity ETL flow of a month-end job for processing sales transactions. In the first article, we brought an existing CoSort SortCL job script that processes month-end sales transactions into Voracity and made modifications. In the second (previous) article, we demonstrated how to:Read More","og_url":"https:\/\/beta.iri.com\/blog\/data-transformation2\/creating-voracity-flows-using-existing-iri-scripts-part-3\/","og_site_name":"IRI","article_published_time":"2016-06-20T16:24:50+00:00","article_modified_time":"2017-11-06T18:45:06+00:00","author":"Susan Gegner","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Susan Gegner","Est. reading time":"9 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/beta.iri.com\/blog\/data-transformation2\/creating-voracity-flows-using-existing-iri-scripts-part-3\/#article","isPartOf":{"@id":"https:\/\/beta.iri.com\/blog\/data-transformation2\/creating-voracity-flows-using-existing-iri-scripts-part-3\/"},"author":{"name":"Susan Gegner","@id":"https:\/\/beta.iri.com\/blog\/#\/schema\/person\/87be5da567628ab9396ca81170f36d63"},"headline":"Creating a Voracity Flow Using Existing IRI Scripts (Part 3 of 3)","datePublished":"2016-06-20T16:24:50+00:00","dateModified":"2017-11-06T18:45:06+00:00","mainEntityOfPage":{"@id":"https:\/\/beta.iri.com\/blog\/data-transformation2\/creating-voracity-flows-using-existing-iri-scripts-part-3\/"},"wordCount":1506,"commentCount":0,"publisher":{"@id":"https:\/\/beta.iri.com\/blog\/#organization"},"image":{"@id":"https:\/\/beta.iri.com\/blog\/data-transformation2\/creating-voracity-flows-using-existing-iri-scripts-part-3\/#primaryimage"},"thumbnailUrl":"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2016\/06\/Creating-Voracity-Flows-Using-Existing-IRI-Scripts-Part-3-Susan-Gegner-0.jpg","keywords":["action key","Eclipse","ETL","field editor","flow","flow diagram","GUI","IRI CoSort","IRI Voracity","IRI Workbench","job script","Oracle","sort key","SortCL","SQL","transform mapping diagram","WB"],"articleSection":["Data Transformation","ETL","IRI Workbench"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/beta.iri.com\/blog\/data-transformation2\/creating-voracity-flows-using-existing-iri-scripts-part-3\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/beta.iri.com\/blog\/data-transformation2\/creating-voracity-flows-using-existing-iri-scripts-part-3\/","url":"https:\/\/beta.iri.com\/blog\/data-transformation2\/creating-voracity-flows-using-existing-iri-scripts-part-3\/","name":"Creating a Voracity Flow Using Existing IRI Scripts (Part 3 of 3) - IRI","isPartOf":{"@id":"https:\/\/beta.iri.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/beta.iri.com\/blog\/data-transformation2\/creating-voracity-flows-using-existing-iri-scripts-part-3\/#primaryimage"},"image":{"@id":"https:\/\/beta.iri.com\/blog\/data-transformation2\/creating-voracity-flows-using-existing-iri-scripts-part-3\/#primaryimage"},"thumbnailUrl":"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2016\/06\/Creating-Voracity-Flows-Using-Existing-IRI-Scripts-Part-3-Susan-Gegner-0.jpg","datePublished":"2016-06-20T16:24:50+00:00","dateModified":"2017-11-06T18:45:06+00:00","breadcrumb":{"@id":"https:\/\/beta.iri.com\/blog\/data-transformation2\/creating-voracity-flows-using-existing-iri-scripts-part-3\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/beta.iri.com\/blog\/data-transformation2\/creating-voracity-flows-using-existing-iri-scripts-part-3\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/beta.iri.com\/blog\/data-transformation2\/creating-voracity-flows-using-existing-iri-scripts-part-3\/#primaryimage","url":"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2016\/06\/Creating-Voracity-Flows-Using-Existing-IRI-Scripts-Part-3-Susan-Gegner-0.jpg","contentUrl":"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2016\/06\/Creating-Voracity-Flows-Using-Existing-IRI-Scripts-Part-3-Susan-Gegner-0.jpg","width":624,"height":322},{"@type":"BreadcrumbList","@id":"https:\/\/beta.iri.com\/blog\/data-transformation2\/creating-voracity-flows-using-existing-iri-scripts-part-3\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/beta.iri.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Creating a Voracity Flow Using Existing IRI Scripts (Part 3 of 3)"}]},{"@type":"WebSite","@id":"https:\/\/beta.iri.com\/blog\/#website","url":"https:\/\/beta.iri.com\/blog\/","name":"IRI","description":"Total Data Management Blog","publisher":{"@id":"https:\/\/beta.iri.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/beta.iri.com\/blog\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/beta.iri.com\/blog\/#organization","name":"IRI","url":"https:\/\/beta.iri.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/beta.iri.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2019\/02\/iri-logo-total-data-management-small-1.png","contentUrl":"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2019\/02\/iri-logo-total-data-management-small-1.png","width":750,"height":206,"caption":"IRI"},"image":{"@id":"https:\/\/beta.iri.com\/blog\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/beta.iri.com\/blog\/#\/schema\/person\/87be5da567628ab9396ca81170f36d63","name":"Susan Gegner","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/beta.iri.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/2b1ca5592a65d44483351292cf1ae00a?s=96&d=blank&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/2b1ca5592a65d44483351292cf1ae00a?s=96&d=blank&r=g","caption":"Susan Gegner"},"url":"https:\/\/beta.iri.com\/blog\/author\/susang\/"}]}},"jetpack_featured_media_url":"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2016\/06\/Creating-Voracity-Flows-Using-Existing-IRI-Scripts-Part-3-Susan-Gegner-0.jpg","_links":{"self":[{"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/posts\/9849"}],"collection":[{"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/users\/10"}],"replies":[{"embeddable":true,"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/comments?post=9849"}],"version-history":[{"count":14,"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/posts\/9849\/revisions"}],"predecessor-version":[{"id":11463,"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/posts\/9849\/revisions\/11463"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/media\/9851"}],"wp:attachment":[{"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/media?parent=9849"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/categories?post=9849"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/tags?post=9849"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}