{"id":16037,"date":"2022-07-13T17:15:36","date_gmt":"2022-07-13T21:15:36","guid":{"rendered":"https:\/\/www.iri.com\/blog\/?p=16037"},"modified":"2023-06-12T14:22:44","modified_gmt":"2023-06-12T18:22:44","slug":"all-about-iri-set-files-a-primer","status":"publish","type":"post","link":"https:\/\/beta.iri.com\/blog\/test-data\/all-about-iri-set-files-a-primer\/","title":{"rendered":"All About IRI Set Files: A Primer"},"content":{"rendered":"

Introduction<\/strong>:\u00a0In the context of IRI software, set files are text files containing realistic or replacement values for data management applications in the <\/span>IRI Voracity<\/span><\/a> platform ecosystem.\u00a0<\/span>They are columnar lists of tab-separated values (TSV) that can be sourced externally or internally.<\/span><\/em><\/p>\n

Here is an example of an online data source that can be converted to a TSV IRI set file format:<\/span><\/p>\n

\"\"<\/p>\n

These dataset files are specified as input or output field attributes in <\/span>SortCL<\/span><\/a>-compatible job scripts that perform:<\/span><\/p>\n

    \n
  1. Lookup transforms<\/span><\/a> in CoSort-powered Voracity ETL jobs<\/span><\/li>\n
  2. Bucketing<\/span><\/a> discrete values into anonymous, general categories defined by named ranges<\/span><\/li>\n
  3. Data validation<\/span><\/a> in Voracity data quality\/cleansing jobs<\/span><\/li>\n
  4. Associating items<\/span><\/a> for data quality, master data management, etc.<\/span><\/li>\n
  5. Pseudonymizing<\/span><\/a> (replacing) names or other values in IRI *shield data masking jobs<\/span><\/li>\n
  6. Restoring<\/span><\/a> pseudonyms to their original values or preserving RDB value pairs<\/span><\/li>\n
  7. Synthesizing<\/span><\/a> randomly selected realistic or valid pair test data in RowGen jobs<\/span>
    \n<\/span><\/li>\n<\/ol>\n

    Set File Contents<\/b><\/h4>\n

    Set files are text files composed of one or more tab-separated columns. The columns must contain ASCII readable characters, UTF8 characters, or EBCDIC characters.\u00a0<\/span><\/p>\n

    Each row ends with a linefeed. Comments can be placed at the top of a set file and must be preceded by a # symbol.\u00a0<\/span><\/p>\n

    Below are a couple of different set files. The one on the left, a two-column set, is typically used for consistent, reversible pseudonyms. The single-column set is more common for random selection.<\/span><\/p>\n

    \"\"<\/p>\n

    Ways to Create Set Files<\/b><\/h4>\n

    IRI<\/span> provides a number of set files with the software. Set file data can also be created:<\/span><\/p>\n

      \n
    1. With a text editor where all the set data is typed into the set file<\/span><\/li>\n
    2. Using IRI Workbench by selecting <\/span>New Set File \u2026<\/span><\/i> from the RowGen menu, then choose one of the wizards listed on the selection page. See <\/span>this article<\/span><\/a> for details.<\/span><\/li>\n
    3. From the fields of data files where SortCL or another utility is used to extract the data<\/span><\/li>\n
    4. From columns in a database using a database procedure, using the <\/span>Set File from Column <\/span><\/i>wizard in IRI Workbench, or by using a SortCL script.<\/span>
      \n<\/span>
      \n<\/span><\/li>\n<\/ol>\n

      \"\"<\/p>\n

      By convention, set file names have a <\/span>.set<\/b> extension.\u00a0<\/span><\/p>\n

      Types of Set Files<\/b><\/h4>\n

      Here are descriptions of the types of set files that can be used:<\/span><\/p>\n

      1. Numeric Set File.<\/strong> <\/span><\/i>These can have any combination of numbers and numeric ranges. <\/span>
      \n<\/span>Each row is randomly selected, and if the row contains a range, a value is randomly selected from the range.<\/span><\/p>\n

      Ranges have lower and upper limits separated by a comma and enclosed with brackets. <\/span>A value next to a square bracket is included in the range while a rounded bracket indicates that the value is not included.<\/p>\n

      Below is an example of a numeric set file:<\/span><\/p>\n

      \"\"<\/i><\/p>\n

      2. Date set file<\/strong><\/em>. <\/i>This can have any combination of dates and date ranges that conform to existing CoSort date, time, and timestamp data type formats. Date ranges are handled in the same way that numeric ranges are handled. The ISO_DATE set file below has a date\u00a0selected from the February range 50% of the time, while selecting from the other two ranges 25% of the time.<\/span><\/p>\n

      \"\"<\/p>\n

      Note that the value <\/span>2022-03-15<\/span><\/i> cannot be selected because there is a parenthesis rather than a square bracket next to that value.<\/span><\/p>\n

      3. Multi-column set file<\/strong>. <\/span><\/i>These have two uses; i.e., either for random selections or lookups.<\/span>\u00a0 \u00a0 \u00a0<\/span><\/p>\n

      a)\u00a0 In the case of random selection:<\/span>
      \n<\/span><\/p>\n