{"id":15218,"date":"2021-11-29T13:29:47","date_gmt":"2021-11-29T18:29:47","guid":{"rendered":"http:\/\/www.iri.com\/blog\/?p=15218"},"modified":"2022-06-29T08:15:40","modified_gmt":"2022-06-29T12:15:40","slug":"automating-iri-jobs-using-file-monitoring","status":"publish","type":"post","link":"https:\/\/beta.iri.com\/blog\/etl\/automating-iri-jobs-using-file-monitoring\/","title":{"rendered":"Automating IRI Jobs Using File Monitoring: A POC"},"content":{"rendered":"<p><span style=\"font-weight: 400;\">Manually initiating <\/span><a href=\"https:\/\/www.iri.com\/products\/cosort\/sortcl\"><span style=\"font-weight: 400;\">SortCL-compatible<\/span><\/a><span style=\"font-weight: 400;\"> jobs in IRI Voracity ETL, CoSort reporting, FieldShield masking, or NextForm migration scenarios is not realistic or productive in environments where data in sources are being added or changed dynamically<\/span><span style=\"font-weight: 400;\">. By contrast, r<\/span><span style=\"font-weight: 400;\">eal-time job automation eliminates the need for manual invocations, and ensures the right jobs run in a timely manner.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">In this post, I walk through a proof of concept (POC) that automatically executes an existing SortCL-compatible job script based on the name of a file when a new file is created or data is added or changed in an existing file.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Note:<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">This article is technical in nature and requires a basic understanding of events and scripting languages.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">IRI has also developed an analogous, log-based solution to trigger SortCL jobs upon real-time changes to data in relational database tables called Ripcurrent; see the series of articles on point starting <a href=\"https:\/\/www.iri.com\/blog\/vldb-operations\/getting-started-with-iri-ripcurrent\">here<\/a>.<\/span><\/li>\n<\/ul>\n<h6><b>File Monitoring Overview<\/b><\/h6>\n<p><span style=\"font-weight: 400;\">File monitoring is generally used for determining if a file has been created, changed, renamed or deleted within a directory. The two main ways that can be used for determining changes with a directory are polling and event-driven.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Polling determines directory file changes by periodically getting a list of file information in a directory and comparing it to a list that was cached earlier.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">The event-driven approach is where the operating system or subsystem provides event notification to an application when a change in a directory\u2019s content occurs. The event-driven approach is usually more efficient as it doesn\u2019t require constant execution of code to determine something has changed.<\/span><\/p>\n<p><a href=\"\/blog\/wp-content\/uploads\/2021\/11\/Watcher-icon.png\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-15233 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/11\/Watcher-icon.png\" alt=\"\" width=\"353\" height=\"258\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/Watcher-icon.png 353w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/Watcher-icon-300x219.png 300w\" sizes=\"(max-width: 353px) 100vw, 353px\" \/><\/a><\/p>\n<p><span style=\"font-weight: 400;\">This post is focused on the event-driven approach using PowerShell 5.1 and the Microsoft .NET FileSystemWatcher Class on the Windows platform. When it comes to file monitoring, this is not the only one of the options. Some other examples are Java&#8217;s Watch Service and Linux <\/span><i><span style=\"font-weight: 400;\">inotify<\/span><\/i><span style=\"font-weight: 400;\">.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Many of the concepts discussed in this post apply, regardless of development language or platform. While I haven\u2019t tested it, PowerShell is also available for Linux and macOS as part of Open Source .NET Core. As you would expect, not all commands available on Windows are available on Linux.<\/span><\/p>\n<h6><b>POC Conceptual Design<\/b><\/h6>\n<p><span style=\"font-weight: 400;\">Beyond the automation of SortCL script execution, the conceptual design includes some basic ways to accomplish functional modularity and scalability. A production implementation may result in expanding the number of servers, watchers, and\/or folders to meet performance and security requirements.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">The conceptual design is depicted below. A high level narrative of the design follows. More details are provided in the implementation section below.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">The Data Source, Staging Server and Job Server represent different logical computers\/servers..<\/span><\/p>\n<p style=\"text-align: center;\"><a href=\"\/blog\/wp-content\/uploads\/2021\/11\/file-process-automation-diagram.png\"><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-15235 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/11\/file-process-automation-diagram.png\" alt=\"\" width=\"649\" height=\"420\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/file-process-automation-diagram.png 973w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/file-process-automation-diagram-300x194.png 300w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/file-process-automation-diagram-768x497.png 768w\" sizes=\"(max-width: 649px) 100vw, 649px\" \/><\/a>Figure 1<\/p>\n<ol>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">A user or some automated process copies a file to or updates an existing file in the Staging folder on the Staging Server.<\/span><span style=\"font-weight: 400;\"><br \/>\n<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">A Watcher running on the Staging Server registers to be notified when any file has been created or changed in the Staging Folder. The Watcher creates a job trigger file in the Trigger folder on the Job Server.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">A Watcher running on the Job Server registers to be notified when a job trigger file has been created or changed in the Trigger folder. The Watcher reads the job trigger file and initiates a SortCL job script to process the file located in the Staging Folder (e.g. <\/span><span style=\"font-weight: 400;\">sortcl.exe \/spec=maskEmployees.scl).<\/span><\/li>\n<\/ol>\n<h6><b>POC Physical Design<\/b><\/h6>\n<p><span style=\"font-weight: 400;\">The physical design of the POC implements the components depicted in the conceptual design above on a single computer running Windows 10. Separate folders were created to represent the servers and the folders as subfolders.\u00a0\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">All components used for the POC are depicted in the diagram below:<\/span><\/p>\n<p style=\"text-align: center;\"><a href=\"\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-2.png\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-15236 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-2.png\" alt=\"\" width=\"223\" height=\"377\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-2.png 223w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-2-177x300.png 177w\" sizes=\"(max-width: 223px) 100vw, 223px\" \/><\/a>Figure 2<\/p>\n<p><span style=\"font-weight: 400;\">In the next section, we will look at how the pieces work together and the associated PowerShell script logic.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Before we do that, let\u2019s look at how the physical design maps to the conceptual design above.<\/span><\/p>\n<ol>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">The Data Source from the conceptual design is represented by the DS folder.\u00a0 Data used to test the POC is contained in the Data folder.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\">The Staging Server represented by the SS folder contains the following:\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\">FileWatchStaging.ps1 file is the PowerShell script that represents the Watcher of the conceptual design.<\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\">fileWatchStaging.config file, in the config subfolder, externalizes the name of the folder to be watched and several timing factors.<\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\">fileActions.csv file, in the config subfolder, contains the location and name of the SortCL script to be executed for each supported file. In the case of the POC that is the employees.csv and patient_record files.<\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\">Staging folder is the folder the Watcher registers to be notified of file events.<\/li>\n<\/ul>\n<\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\">The Job Server represented by the JS folder contains the following:\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\">FileWatchTrigger.ps1 file is the PowerShell script that represents the Watcher of the conceptual design.<\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\">fileWatchTrigger.config file, in the config subfolder, externalizes the name of the folder to be watched and several timing factors.<\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\">Trigger folder is the folder that will be watched, and the files written there contain the information needed to initiate a SortCL job script.<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n<p><span style=\"font-weight: 400;\">As you can see, the design has two file watchers with one being responsible for creating a job trigger for file changes in the Staging folder, and the other watcher for initiating the execution of a SortCL job script for trigger files written to the Trigger folder.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">A common requirement of both Watchers is to look for created and changed file events and then take some action. Looking for changed events is not only because a file was changed, but also because copying a file to a folder will typically result in a created event and one or more changed events.<\/span><span style=\"font-weight: 400;\"> In my testing, the number of events depended on the size of the file.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Handling the multiple file events requires the added complexity of introducing a time delay into the design. This ensures that file processing is completed prior to writing the trigger file for subsequent processing.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">There are other decisions I made in the implementation to ensure file integrity that may or may not meet your specific needs.\u00a0 These are covered in the next section.<\/span><\/p>\n<h6><b>Implementation<\/b><\/h6>\n<p><span style=\"font-weight: 400;\">As mentioned earlier, the POC was created to demonstrate an approach for automating the execution of a SortCL job script based on a file being copied or updated.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">I can\u2019t stress enough that this was a proof of concept and needs enhancements to be ready for a production environment. More on that at the end of the post.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Now let\u2019s walk through the process and code from end-to-end. If you are not a PowerShell developer, the code comments should aid your understanding.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">All of the code and configuration files used in the POC can be downloaded <\/span><a href=\"https:\/\/www.iri.com\/download\/d2ca8b6342d165b1ed45436bf05e2c418acc34e5\"><b>here<\/b><\/a><span style=\"font-weight: 400;\">.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Please note, that in order to format the script code for viewing in the post, some of the code is broken across multiple lines. Therefore, it is recommended to use the files from the download\u00a0 and not any of the code included in this post.\u00a0<\/span><\/p>\n<ol>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">To start the process, the Staging and Trigger PowerShell Watcher scripts need to be started.\u00a0Since the standard PowerShell<br \/>\nexecution model processes synchronously, these scripts must be started in separate PowerShell command windows. To accomplish asynchronous processing within the scripts, they can be enhanced to use jobs<br \/>\nthat run in the background (i.e. Start-Job).<\/span><\/li>\n<\/ol>\n<p style=\"text-align: center;\"><a href=\"\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-3.png\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-15238 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-3.png\" alt=\"\" width=\"417\" height=\"81\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-3.png 417w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-3-300x58.png 300w\" sizes=\"(max-width: 417px) 100vw, 417px\" \/><\/a>Figure 3<\/p>\n<p style=\"text-align: center;\"><a href=\"\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-4.png\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-15237 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-4.png\" alt=\"\" width=\"405\" height=\"81\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-4.png 405w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-4-300x60.png 300w\" sizes=\"(max-width: 405px) 100vw, 405px\" \/><\/a>Figure 4<\/p>\n<ol start=\"2\">\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">When the Watcher scripts are first started they perform a number of initialization steps in preparation for processing file events.<br \/>\nThis includes:<\/span><\/p>\n<ol style=\"list-style-type: lower-alpha;\">\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Reading and caching configuration information.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"2\"><span style=\"font-weight: 400;\">Reading and caching file action information (Staging Watcher only).<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"2\"><span style=\"font-weight: 400;\">Creating an instance of Windows File System Watcher class and setting properties.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"2\"><span style=\"font-weight: 400;\">Registering for created and changed events.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"2\"><span style=\"font-weight: 400;\">Creating an instance of a Timer and setting properties that upon expiration determines if a file is ready to process.<br \/>\nIf you recall from above, this is to handle the occurrence of multiple events when a file is copied or written to the directory.<\/span><\/li>\n<\/ol>\n<\/li>\n<\/ol>\n<p><span style=\"font-weight: 400;\">The initialization code is immediately below and is common between the watchers with three differences: The Trigger Watcher PowerShell script doesn\u2019t contain the code to<br \/>\nprocess the fileActions.csv, has a different configuration file name, and the file filter is set to *.trigger instead of all files (*.*). The content of the configuration and file actions files can be seen in<br \/>\nFigures 7, 8 and 11.<\/span><\/p>\n<p><a href=\"\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-blockcode-1-1.png\"><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-15240 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-blockcode-1-1.png\" alt=\"\" width=\"500\" height=\"426\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-blockcode-1-1.png 590w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-blockcode-1-1-300x256.png 300w\" sizes=\"(max-width: 500px) 100vw, 500px\" \/><\/a><a href=\"\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-blockcode-1-2-1.png\"><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-15242 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-blockcode-1-2-1.png\" alt=\"\" width=\"500\" height=\"301\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-blockcode-1-2-1.png 552w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-blockcode-1-2-1-300x180.png 300w\" sizes=\"(max-width: 500px) 100vw, 500px\" \/><\/a><\/p>\n<p><span style=\"font-weight: 400;\">After initialization the script loops waiting for events.\u00a0 When the script is stopped, the finally block cleans things up.<\/span><\/p>\n<p><a href=\"\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-blockcode-2.png\"><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-15243 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-blockcode-2.png\" alt=\"\" width=\"500\" height=\"477\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-blockcode-2.png 576w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-blockcode-2-300x286.png 300w\" sizes=\"(max-width: 500px) 100vw, 500px\" \/><\/a><\/p>\n<ol start=\"3\">\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Once the Watcher Scripts are running, we can copy the employees.csv file from the Data Source Data folder to the Staging folder.<\/span><\/li>\n<\/ol>\n<p style=\"text-align: center;\"><a href=\"\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-5.png\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-15245 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-5.png\" alt=\"\" width=\"364\" height=\"118\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-5.png 364w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-5-300x97.png 300w\" sizes=\"(max-width: 364px) 100vw, 364px\" \/><\/a>Figure 5<\/p>\n<ol start=\"4\">\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">When the copy process starts, the FileWatch Staging script receives one created event notification and four changed event notifications by the time the copy finishes.\u00a0 The number of changed events will vary depending on the size of the file.<\/span><\/li>\n<\/ol>\n<p style=\"text-align: center;\"><a href=\"\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-6.png\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-15244 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-6.png\" alt=\"\" width=\"214\" height=\"144\" \/><\/a>Figure 6<\/p>\n<p>&nbsp;<\/p>\n<p><span style=\"font-weight: 400;\">To handle multiple events, the processing is as follows:<\/span><\/p>\n<ol style=\"list-style-type: lower-alpha;\">\n<li><span style=\"font-weight: 400;\">For the created event, information about the file is added to the file event cache.\u00a0 This code gets executed because it was specified as the -action parameter during the registration process for the event handlers in 2 above.<\/span><span style=\"font-weight: 400;\"><br \/>\n<\/span><\/li>\n<\/ol>\n<p><a href=\"\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-blockcode-3.png\"><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-15247 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-blockcode-3.png\" alt=\"\" width=\"499\" height=\"317\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-blockcode-3.png 542w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-blockcode-3-300x190.png 300w\" sizes=\"(max-width: 499px) 100vw, 499px\" \/><\/a><\/p>\n<ol style=\"list-style-type: lower-alpha;\" start=\"2\">\n<li style=\"font-weight: 400;\" aria-level=\"2\"><span style=\"font-weight: 400;\">For each changed event, the event time is updated in the file event cache.\u00a0 This code gets executed because it was specified as the -action parameter during the registration process for the event handlers in 2 above.<\/span><\/li>\n<\/ol>\n<p><a href=\"\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-blockcode-4.png\"><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-15246 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-blockcode-4.png\" alt=\"\" width=\"500\" height=\"301\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-blockcode-4.png 550w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-blockcode-4-300x181.png 300w\" sizes=\"(max-width: 500px) 100vw, 500px\" \/><\/a><\/p>\n<ol>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">At startup the FileWatch Staging script sets a timer that expires based on the millisecond setting in it\u2019s configuration file (timerWaitMil).\u00a0 When the timer expires, the cached file information is checked for any files that have not been updated for more than the number of seconds specified in the configuration file (processWaitSec).<\/span><\/li>\n<\/ol>\n<p style=\"text-align: center;\"><a href=\"\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-7.png\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-15248 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-7.png\" alt=\"\" width=\"331\" height=\"72\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-7.png 331w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-7-300x65.png 300w\" sizes=\"(max-width: 331px) 100vw, 331px\" \/><\/a>Figure 7<\/p>\n<ol style=\"list-style-type: lower-alpha;\">\n<li style=\"font-weight: 400;\" aria-level=\"2\"><span style=\"font-weight: 400;\">If the current time minus the last file event exceeds the process wait seconds, the Staging Watcher looks up the file name in the cached actions (Figure 8), creates a job trigger file (Figure 9) in the directory specified by the triggerPath attribute and deletes the file event from the cache.<\/span><\/li>\n<\/ol>\n<p style=\"padding-left: 30px; text-align: left;\">The job trigger file is named using the file name with a date and job number appended to the end (FileName.yymmdd_j<i>n<\/i>.trigger where <i>n<\/i> is the job number).<\/p>\n<p style=\"padding-left: 30px; text-align: left;\">Each time a file is processed by the script the job number is incremented.\u00a0 For example, if the patient_record file is the next file copied to the Staging folder the trigger will be named patient_record.211122_j2.trigger<\/p>\n<p style=\"padding-left: 30px;\"><span style=\"text-decoration: underline;\"><strong>File Actions Configuration<\/strong><\/span><\/p>\n<p style=\"text-align: center;\"><a href=\"\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-8.png\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-15250 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-8.png\" alt=\"\" width=\"502\" height=\"91\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-8.png 502w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-8-300x54.png 300w\" sizes=\"(max-width: 502px) 100vw, 502px\" \/><\/a>Figure 8<\/p>\n<p>&nbsp;<\/p>\n<p style=\"padding-left: 30px;\"><strong><span style=\"text-decoration: underline;\">Trigger File<\/span><\/strong><\/p>\n<p style=\"text-align: center;\"><a href=\"\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-9.png\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-15249 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-9.png\" alt=\"\" width=\"442\" height=\"190\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-9.png 442w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-9-300x129.png 300w\" sizes=\"(max-width: 442px) 100vw, 442px\" \/><\/a>Figure 9<\/p>\n<p>Here is the code that performs the logic above:<\/p>\n<p><a href=\"\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-blockcode-5-1.png\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-15253 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-blockcode-5-1.png\" alt=\"\" width=\"576\" height=\"771\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-blockcode-5-1.png 576w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-blockcode-5-1-224x300.png 224w\" sizes=\"(max-width: 576px) 100vw, 576px\" \/><\/a><a href=\"\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-blockcode-5-2.png\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-15252 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-blockcode-5-2.png\" alt=\"\" width=\"506\" height=\"170\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-blockcode-5-2.png 506w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-blockcode-5-2-300x101.png 300w\" sizes=\"(max-width: 506px) 100vw, 506px\" \/><\/a><\/p>\n<ol start=\"6\">\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Once the job trigger file is written, the FileWatch Trigger script receives one created event notification and one changed notification event for the job trigger file created above.\u00a0 To handle multiple events the processing is as follows:<\/span><span style=\"font-weight: 400;\"><br \/>\n<\/span><\/p>\n<ol style=\"list-style-type: lower-alpha;\">\n<li><span style=\"font-weight: 400;\">For the created event, information about the file is added to the file event cache. The code executed is the same as in 4a.<\/span><\/li>\n<li>For the change event, the event time is updated in the file event cache.\u00a0 The code executed is the same as in 4b.<\/li>\n<\/ol>\n<\/li>\n<\/ol>\n<p style=\"text-align: center;\"><a href=\"\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-10.png\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-15254 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-10.png\" alt=\"\" width=\"340\" height=\"90\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-10.png 340w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-10-300x79.png 300w\" sizes=\"(max-width: 340px) 100vw, 340px\" \/><\/a>Figure 10<\/p>\n<ol start=\"7\">\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">At startup, the FileWatch Trigger script sets a timer that expires based on the millisecond setting in it\u2019s configuration file (timerWaitMil).\u00a0 When the timer expires, the cached file information is checked for any files that have not been updated for more than the number of seconds specified in the configuration file (processWaitSec).<\/span><\/li>\n<\/ol>\n<p style=\"text-align: center;\"><a href=\"\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-11.png\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-15255 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-11.png\" alt=\"\" width=\"332\" height=\"71\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-11.png 332w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-11-300x64.png 300w\" sizes=\"(max-width: 332px) 100vw, 332px\" \/><\/a>Figure 11<\/p>\n<ol style=\"list-style-type: lower-alpha;\">\n<li style=\"font-weight: 400;\" aria-level=\"2\"><span style=\"font-weight: 400;\">If the current time minus the last file event exceeds the process wait seconds, the Watcher reads the trigger file and uses the information to create some Windows environment variables and execute the SortCL job script.<\/span><span style=\"font-weight: 400;\"><br \/>\n<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"2\"><span style=\"font-weight: 400;\">SortCL executes the maskEmployees.scl (Figure 12) script using the environment variables set by the FileWatcher Trigger PowerShell script for jobFullName and jobFileName.\u00a0<\/span><\/li>\n<\/ol>\n<p style=\"padding-left: 30px;\"><span style=\"font-weight: 400;\">It is important to make sure if the file format changes to update the SortCL script.<\/span><\/p>\n<p style=\"text-align: center;\"><a href=\"\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-12.png\"><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-15257 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-12.png\" alt=\"\" width=\"599\" height=\"367\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-12.png 864w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-12-300x184.png 300w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-12-768x470.png 768w\" sizes=\"(max-width: 599px) 100vw, 599px\" \/><\/a>Figure 12<\/p>\n<ol style=\"list-style-type: lower-alpha;\" start=\"3\">\n<li style=\"font-weight: 400;\" aria-level=\"2\"><span style=\"font-weight: 400;\">When the SortCL job script has completed the output masked file will have been written to the TestData directory (Figure 13), the job trigger file is deleted, and the FileWatch script writes an entry to the Windows event log (Figure 14).\u00a0 The event log entry provides an audit trail linking the original file to the job and processId to the SortCL job script audit file (Figure 15).<\/span><\/li>\n<\/ol>\n<p style=\"text-align: center;\"><a href=\"\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-13.png\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-15256 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-13.png\" alt=\"\" width=\"263\" height=\"89\" \/><\/a>Figure 13<\/p>\n<p><span style=\"font-weight: 400;\">Here is the code that performs the logic above:<\/span><\/p>\n<p><a href=\"\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-blockcode-6.png\"><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-15258 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-blockcode-6.png\" alt=\"\" width=\"549\" height=\"663\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-blockcode-6.png 566w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-blockcode-6-248x300.png 248w\" sizes=\"(max-width: 549px) 100vw, 549px\" \/><\/a><\/p>\n<p style=\"text-align: center;\"><a href=\"\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-14.png\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-15260 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-14.png\" alt=\"\" width=\"400\" height=\"365\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-14.png 400w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-14-300x274.png 300w\" sizes=\"(max-width: 400px) 100vw, 400px\" \/><\/a>Figure 14<\/p>\n<p style=\"text-align: center;\"><a href=\"\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-15.png\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-15259 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-15.png\" alt=\"\" width=\"491\" height=\"202\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-15.png 491w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-15-300x123.png 300w\" sizes=\"(max-width: 491px) 100vw, 491px\" \/><\/a>Figure 15<\/p>\n<p>&nbsp;<\/p>\n<h6><b>Multi-File Processing<\/b><\/h6>\n<p><span style=\"font-weight: 400;\">More than one file can be copied at the same time to the staging folder. The figures below show the contents of the directories during processing. After processing is completed, the job trigger files are deleted.<\/span><\/p>\n<p><span style=\"text-decoration: underline;\"><span style=\"font-weight: 400;\">Files Copied from Data to Staging Directory<\/span><\/span><\/p>\n<p style=\"text-align: center;\"><a href=\"\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-16.png\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-15263 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-16.png\" alt=\"\" width=\"246\" height=\"423\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-16.png 246w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-16-174x300.png 174w\" sizes=\"(max-width: 246px) 100vw, 246px\" \/><\/a>Figure 16<\/p>\n<p><span style=\"text-decoration: underline;\">Trigger Files Created<\/span><\/p>\n<p style=\"text-align: center;\"><a href=\"\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-17.png\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-15264 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-17.png\" alt=\"\" width=\"212\" height=\"107\" \/><\/a>Figure 17<\/p>\n<p><span style=\"text-decoration: underline;\">Files Created by Running the SortCL Script (Figure 12)<\/span><\/p>\n<p style=\"text-align: center;\"><a href=\"\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-18.png\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-15261 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-18.png\" alt=\"\" width=\"240\" height=\"104\" \/><\/a>Figure 18<\/p>\n<h6><b>Updated File Processing<\/b><\/h6>\n<p><span style=\"font-weight: 400;\">If records are updated to a file located in the staging folder, another trigger file will be created and the masking SortCL job script (Figure 12) will run again.\u00a0 As shown in Figure 16, the new trigger file ends with _j3 (job 3) since _j1 and _ j2 triggers were created in the multi-file processing above.<\/span><\/p>\n<p><span style=\"text-decoration: underline;\">Trigger file created from file change<\/span><\/p>\n<p style=\"text-align: center;\"><a href=\"\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-19.png\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-15262 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/11\/automated-file-processing-figure-19.png\" alt=\"\" width=\"221\" height=\"87\" \/><\/a>Figure 19<\/p>\n<h6><b>Considerations for Production Readiness<\/b><\/h6>\n<p><span style=\"font-weight: 400;\">Again, I can\u2019t stress enough that this post is about a proof of concept and needs enhancing prior to any production use. The following list is some initials thoughts on enhancements:<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Enhance to meet your performance and security requirements. This could include:<\/span>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"2\"><span style=\"font-weight: 400;\">Expanding the number of servers, watchers and folders to provide more processing parallelism.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"2\"><span style=\"font-weight: 400;\">Creating additional folders to simplify security. (e.g. Staging and Output folders for HR, Customer, Claims).<\/span><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Add error checking and exception handling, as the POC watcher scripts only covered the <\/span><a href=\"https:\/\/en.wikipedia.org\/wiki\/Happy_path\"><span style=\"font-weight: 400;\">happy path<\/span><\/a><span style=\"font-weight: 400;\">.<\/span><\/li>\n<li><span style=\"font-weight: 400;\">Refactor to reduce redundant code. At a minimum, move the common code to a separate file and convert to functions.<\/span><\/li>\n<\/ul>\n<p style=\"padding-left: 30px;\"><span style=\"font-weight: 400;\">If you are familiar with object oriented development, the Staging Watcher and Trigger Watcher can be streamlined into classes and inherit a base Watcher Class containing the common code.<\/span><\/p>\n<p style=\"padding-left: 30px;\"><a href=\"https:\/\/en.wikibooks.org\/wiki\/Introduction_to_Software_Engineering\/Testing\/Refactoring\"><span style=\"font-weight: 400;\">Refactoring<\/span><\/a><span style=\"font-weight: 400;\"> will result in reduced code and lower cost of maintenance.<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Remove hardcoding for the job counter reset and add it to the staging watcher configuration file.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Create a script to delete or archive the files.<\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400;\">My last piece of advice is to test, test, and test again. Don\u2019t forget volume and negative testing.<\/span><\/p>\n<h6><b>Summary<\/b><\/h6>\n<p><span style=\"font-weight: 400;\">In this post, I discussed the benefits of automating SortCL job script executions for real-time file system events. I then walked through a conceptual design and implementation that runs these jobs when files are created or updated in a directory being monitored.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">The approach demonstrated in this post should give you a good start on creating your own automation effort. Another post will cover real-time SortCL job triggering from database updates.<\/span><\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Manually initiating SortCL-compatible jobs in IRI Voracity ETL, CoSort reporting, FieldShield masking, or NextForm migration scenarios is not realistic or productive in environments where data in sources are being added or changed dynamically. By contrast, real-time job automation eliminates the need for manual invocations, and ensures the right jobs run in a timely manner. In<\/p>\n<div><a class=\"btn-filled btn\" href=\"https:\/\/beta.iri.com\/blog\/etl\/automating-iri-jobs-using-file-monitoring\/\" title=\"Automating IRI Jobs Using File Monitoring: A POC\">Read More<\/a><\/div>\n","protected":false},"author":151,"featured_media":15267,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_exactmetrics_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"footnotes":""},"categories":[32,8,31,363,1,776],"tags":[1563,1562,1559,1204,1560,1561,1182,1564],"class_list":["post-15218","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-business-intelligence","category-data-protection","category-data-migration","category-data-quality","category-data-transformation2","category-etl","tag-automatic-sortcl-jobs","tag-automation","tag-file-monitoring","tag-iri-sortcl","tag-microsoft-net","tag-multi-file-processing","tag-poc","tag-real-time-job-automation"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v23.3 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Automating IRI Jobs Using File Monitoring: A POC - IRI<\/title>\n<meta name=\"description\" content=\"Manually initiating SortCL-compatible jobs in IRI Voracity ETL, CoSort reporting, FieldShield masking, or NextForm migration scenarios is not realistic or\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/beta.iri.com\/blog\/etl\/automating-iri-jobs-using-file-monitoring\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Automating IRI Jobs Using File Monitoring: A POC - IRI\" \/>\n<meta property=\"og:description\" content=\"Manually initiating SortCL-compatible jobs in IRI Voracity ETL, CoSort reporting, FieldShield masking, or NextForm migration scenarios is not realistic or\" \/>\n<meta property=\"og:url\" content=\"https:\/\/beta.iri.com\/blog\/etl\/automating-iri-jobs-using-file-monitoring\/\" \/>\n<meta property=\"og:site_name\" content=\"IRI\" \/>\n<meta property=\"article:published_time\" content=\"2021-11-29T18:29:47+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2022-06-29T12:15:40+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/Watcher-icon-thumbnail.png\" \/>\n\t<meta property=\"og:image:width\" content=\"600\" \/>\n\t<meta property=\"og:image:height\" content=\"258\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Wade Donahue\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Wade Donahue\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"12 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/beta.iri.com\/blog\/etl\/automating-iri-jobs-using-file-monitoring\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/etl\/automating-iri-jobs-using-file-monitoring\/\"},\"author\":{\"name\":\"Wade Donahue\",\"@id\":\"https:\/\/beta.iri.com\/blog\/#\/schema\/person\/3c88af09f1d4fcdef7370a7abe64b732\"},\"headline\":\"Automating IRI Jobs Using File Monitoring: A POC\",\"datePublished\":\"2021-11-29T18:29:47+00:00\",\"dateModified\":\"2022-06-29T12:15:40+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/etl\/automating-iri-jobs-using-file-monitoring\/\"},\"wordCount\":2416,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/#organization\"},\"image\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/etl\/automating-iri-jobs-using-file-monitoring\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/Watcher-icon-thumbnail.png\",\"keywords\":[\"automatic SortCL jobs\",\"automation\",\"File Monitoring\",\"IRI SortCL\",\"Microsoft .NET\",\"multi-file processing\",\"POC\",\"real-time job automation\"],\"articleSection\":[\"Business Intelligence (BI&#041;\",\"Data Masking\/Protection\",\"Data Migration\",\"Data Quality (DQ&#041;\",\"Data Transformation\",\"ETL\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/beta.iri.com\/blog\/etl\/automating-iri-jobs-using-file-monitoring\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/beta.iri.com\/blog\/etl\/automating-iri-jobs-using-file-monitoring\/\",\"url\":\"https:\/\/beta.iri.com\/blog\/etl\/automating-iri-jobs-using-file-monitoring\/\",\"name\":\"Automating IRI Jobs Using File Monitoring: A POC - IRI\",\"isPartOf\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/etl\/automating-iri-jobs-using-file-monitoring\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/etl\/automating-iri-jobs-using-file-monitoring\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/Watcher-icon-thumbnail.png\",\"datePublished\":\"2021-11-29T18:29:47+00:00\",\"dateModified\":\"2022-06-29T12:15:40+00:00\",\"description\":\"Manually initiating SortCL-compatible jobs in IRI Voracity ETL, CoSort reporting, FieldShield masking, or NextForm migration scenarios is not realistic or\",\"breadcrumb\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/etl\/automating-iri-jobs-using-file-monitoring\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/beta.iri.com\/blog\/etl\/automating-iri-jobs-using-file-monitoring\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/beta.iri.com\/blog\/etl\/automating-iri-jobs-using-file-monitoring\/#primaryimage\",\"url\":\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/Watcher-icon-thumbnail.png\",\"contentUrl\":\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/Watcher-icon-thumbnail.png\",\"width\":600,\"height\":258},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/beta.iri.com\/blog\/etl\/automating-iri-jobs-using-file-monitoring\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/beta.iri.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Automating IRI Jobs Using File Monitoring: A POC\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/beta.iri.com\/blog\/#website\",\"url\":\"https:\/\/beta.iri.com\/blog\/\",\"name\":\"IRI\",\"description\":\"Total Data Management Blog\",\"publisher\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/beta.iri.com\/blog\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/beta.iri.com\/blog\/#organization\",\"name\":\"IRI\",\"url\":\"https:\/\/beta.iri.com\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/beta.iri.com\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2019\/02\/iri-logo-total-data-management-small-1.png\",\"contentUrl\":\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2019\/02\/iri-logo-total-data-management-small-1.png\",\"width\":750,\"height\":206,\"caption\":\"IRI\"},\"image\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/beta.iri.com\/blog\/#\/schema\/person\/3c88af09f1d4fcdef7370a7abe64b732\",\"name\":\"Wade Donahue\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/beta.iri.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/9cf07e37c128f0334168629cb154a3f8?s=96&d=blank&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/9cf07e37c128f0334168629cb154a3f8?s=96&d=blank&r=g\",\"caption\":\"Wade Donahue\"},\"url\":\"https:\/\/beta.iri.com\/blog\/author\/waded\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Automating IRI Jobs Using File Monitoring: A POC - IRI","description":"Manually initiating SortCL-compatible jobs in IRI Voracity ETL, CoSort reporting, FieldShield masking, or NextForm migration scenarios is not realistic or","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/beta.iri.com\/blog\/etl\/automating-iri-jobs-using-file-monitoring\/","og_locale":"en_US","og_type":"article","og_title":"Automating IRI Jobs Using File Monitoring: A POC - IRI","og_description":"Manually initiating SortCL-compatible jobs in IRI Voracity ETL, CoSort reporting, FieldShield masking, or NextForm migration scenarios is not realistic or","og_url":"https:\/\/beta.iri.com\/blog\/etl\/automating-iri-jobs-using-file-monitoring\/","og_site_name":"IRI","article_published_time":"2021-11-29T18:29:47+00:00","article_modified_time":"2022-06-29T12:15:40+00:00","og_image":[{"width":600,"height":258,"url":"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/Watcher-icon-thumbnail.png","type":"image\/png"}],"author":"Wade Donahue","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Wade Donahue","Est. reading time":"12 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/beta.iri.com\/blog\/etl\/automating-iri-jobs-using-file-monitoring\/#article","isPartOf":{"@id":"https:\/\/beta.iri.com\/blog\/etl\/automating-iri-jobs-using-file-monitoring\/"},"author":{"name":"Wade Donahue","@id":"https:\/\/beta.iri.com\/blog\/#\/schema\/person\/3c88af09f1d4fcdef7370a7abe64b732"},"headline":"Automating IRI Jobs Using File Monitoring: A POC","datePublished":"2021-11-29T18:29:47+00:00","dateModified":"2022-06-29T12:15:40+00:00","mainEntityOfPage":{"@id":"https:\/\/beta.iri.com\/blog\/etl\/automating-iri-jobs-using-file-monitoring\/"},"wordCount":2416,"commentCount":0,"publisher":{"@id":"https:\/\/beta.iri.com\/blog\/#organization"},"image":{"@id":"https:\/\/beta.iri.com\/blog\/etl\/automating-iri-jobs-using-file-monitoring\/#primaryimage"},"thumbnailUrl":"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/Watcher-icon-thumbnail.png","keywords":["automatic SortCL jobs","automation","File Monitoring","IRI SortCL","Microsoft .NET","multi-file processing","POC","real-time job automation"],"articleSection":["Business Intelligence (BI&#041;","Data Masking\/Protection","Data Migration","Data Quality (DQ&#041;","Data Transformation","ETL"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/beta.iri.com\/blog\/etl\/automating-iri-jobs-using-file-monitoring\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/beta.iri.com\/blog\/etl\/automating-iri-jobs-using-file-monitoring\/","url":"https:\/\/beta.iri.com\/blog\/etl\/automating-iri-jobs-using-file-monitoring\/","name":"Automating IRI Jobs Using File Monitoring: A POC - IRI","isPartOf":{"@id":"https:\/\/beta.iri.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/beta.iri.com\/blog\/etl\/automating-iri-jobs-using-file-monitoring\/#primaryimage"},"image":{"@id":"https:\/\/beta.iri.com\/blog\/etl\/automating-iri-jobs-using-file-monitoring\/#primaryimage"},"thumbnailUrl":"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/Watcher-icon-thumbnail.png","datePublished":"2021-11-29T18:29:47+00:00","dateModified":"2022-06-29T12:15:40+00:00","description":"Manually initiating SortCL-compatible jobs in IRI Voracity ETL, CoSort reporting, FieldShield masking, or NextForm migration scenarios is not realistic or","breadcrumb":{"@id":"https:\/\/beta.iri.com\/blog\/etl\/automating-iri-jobs-using-file-monitoring\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/beta.iri.com\/blog\/etl\/automating-iri-jobs-using-file-monitoring\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/beta.iri.com\/blog\/etl\/automating-iri-jobs-using-file-monitoring\/#primaryimage","url":"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/Watcher-icon-thumbnail.png","contentUrl":"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/Watcher-icon-thumbnail.png","width":600,"height":258},{"@type":"BreadcrumbList","@id":"https:\/\/beta.iri.com\/blog\/etl\/automating-iri-jobs-using-file-monitoring\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/beta.iri.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Automating IRI Jobs Using File Monitoring: A POC"}]},{"@type":"WebSite","@id":"https:\/\/beta.iri.com\/blog\/#website","url":"https:\/\/beta.iri.com\/blog\/","name":"IRI","description":"Total Data Management Blog","publisher":{"@id":"https:\/\/beta.iri.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/beta.iri.com\/blog\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/beta.iri.com\/blog\/#organization","name":"IRI","url":"https:\/\/beta.iri.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/beta.iri.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2019\/02\/iri-logo-total-data-management-small-1.png","contentUrl":"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2019\/02\/iri-logo-total-data-management-small-1.png","width":750,"height":206,"caption":"IRI"},"image":{"@id":"https:\/\/beta.iri.com\/blog\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/beta.iri.com\/blog\/#\/schema\/person\/3c88af09f1d4fcdef7370a7abe64b732","name":"Wade Donahue","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/beta.iri.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/9cf07e37c128f0334168629cb154a3f8?s=96&d=blank&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/9cf07e37c128f0334168629cb154a3f8?s=96&d=blank&r=g","caption":"Wade Donahue"},"url":"https:\/\/beta.iri.com\/blog\/author\/waded\/"}]}},"jetpack_featured_media_url":"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/Watcher-icon-thumbnail.png","_links":{"self":[{"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/posts\/15218"}],"collection":[{"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/users\/151"}],"replies":[{"embeddable":true,"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/comments?post=15218"}],"version-history":[{"count":17,"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/posts\/15218\/revisions"}],"predecessor-version":[{"id":16035,"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/posts\/15218\/revisions\/16035"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/media\/15267"}],"wp:attachment":[{"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/media?parent=15218"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/categories?post=15218"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/tags?post=15218"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}