{"id":15712,"date":"2022-03-24T13:52:39","date_gmt":"2022-03-24T17:52:39","guid":{"rendered":"https:\/\/www.iri.com\/blog\/?p=15712"},"modified":"2022-06-13T09:16:41","modified_gmt":"2022-06-13T13:16:41","slug":"test-data-for-devops-mlops-dataops","status":"publish","type":"post","link":"https:\/\/beta.iri.com\/blog\/test-data\/test-data-for-devops-mlops-dataops\/","title":{"rendered":"Smart, Safe Test Data for DevOps, MLOps and DataOps"},"content":{"rendered":"

Data flowing through application development, machine learning, and analytic pipelines must address several needs common to all three, including:<\/span><\/p>\n

    \n
  1. Realism, to reflect production data characteristics and application test requirements;<\/span><\/li>\n
  2. Compliance, with business and data privacy rules, plus DB and analytic models;<\/span><\/li>\n
  3. Availability, or security, of the data (depending on your perspective); and,<\/span><\/li>\n
  4. Auditability, for lineage and accountability.<\/span><\/li>\n<\/ol>\n

    Stakeholders in these pipelines understand these requirements from their own perspectives. IRI, aka The CoSort Company, provides a multi-faceted test data management framework to meet these needs.<\/span><\/p>\n

    IRI roles in this broad area began with the need to create rich, realistic data to test the volume and variety of data transformation and formatting activities supported in the IRI CoSort data manipulation product.<\/span><\/p>\n

    Using the core \u201cSortCL\u201d data definition language and processing program of CoSort, IRI changed the input phase of its ETL process from reading files to building them, either through random value generation of specified data types and ranges and\/or random selection of data from external sets.\u00a0<\/span><\/p>\n

    IRI launched the RowGen spin-off product from SortCL in 2004, and later expanded it to parse, synthesize and load pre-sorted, structurally and referentially correct RDB schema from only DDL details. RowGen can now also: generate new data formats and computationally valid CCNs and NIDs, and sets to handle all-pairs; create nulls and realistic value distributions; work in ETL and CI\/CD pipelines; and populate test data in semi- and unstructured sources like EDI and Excel files, PDF and Word documents, and images with embedded test data when used with the IRI DarkShield search\/mask API.<\/span><\/p>\n

    \"\"<\/p>\n

    Test Data for DevOps (TestOps)<\/b><\/h5>\n

    DevOps streamlines software development lifecycle (SDLC) operations to accelerate software delivery. Many developers use CI\/CD pipelines to make software releases more agile and continuous. Test data created in IRI Voracity data masking, subsetting, or synthesis tools can be executed and consumed by Jenkins<\/a>, Amazon CodePipeline<\/a>, Azure DevOps<\/a>, GitLab<\/a>, etc. and be used within those pipelines to assess software functionality and capacity at each build.\u00a0<\/span><\/p>\n

    The emerging TestOps discipline manages the operational aspects of testing in the SDLC, which include collecting, preparing, and securing exposure data, production systems, and test case sources \u2026 in order to scale test coverage, people, and activities (but also to ensure software quality). Intelligent data integration and anonymization of those sources in Voracity thus plays key TestOps roles while improving DevOps QA and mitigating privacy risks.<\/span><\/p>\n

    Test Data for MLOps<\/b>
    \n<\/b><\/h5>\n

    Rich, anonymous data also helps in testing machine learning operations (MLOps), which include: saving, loading, and transforming data, as well as model testing and data validation. For each of these phases, for example, the IRI RowGen product in Voracity can rapidly synthesize huge, realistic files in bulk load (e.g., CSV) and ML model formats like PMML\/XML and PFA\/JSON.\u00a0<\/span><\/p>\n

    Alternatively, the FieldShield or DarkShield data discovery and masking tools in Voracity can search and sanitize database collections, files, or data streams used in machine learning for safe use in testing.<\/span><\/p>\n

    \"\"<\/p>\n

    Test Data for DataOps<\/b><\/h5>\n

    Implementing a DataOps testing approach for ETL projects means automating testing for source and target datasets and ensuring those sets reflect the characteristics of data used in real analytic models without identifying people. IRI software facilitates data integration and analytic test automation in several ways.\u00a0<\/span><\/p>\n

    One way is by combining data transformation and wrangling with data masking or synthesis. Unique to Voracity is the back-end metadata and engine support for such task consolidation. In a single I\/O pass through the aforementioned SortCL data processing program, Voracity users can simultaneously synthesize, transform, and format test data into multiple artificial, but realistic analytic targets. Conversely, the program can read one or more production sources, integrate and transform them, as well as cleanse, mask and reformat data into desired targets.\u00a0<\/span><\/p>\n

    Another way is the data integration preview feature for Voracity ETL architects which runs a mapping job using auto-generated test data. This option builds target subsets with realistic test data so architects can validate their transformation logic and output layout.<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"

    Data flowing through application development, machine learning, and analytic pipelines must address several needs common to all three, including: Realism, to reflect production data characteristics and application test requirements; Compliance, with business and data privacy rules, plus DB and analytic models; Availability, or security, of the data (depending on your perspective); and, Auditability, for lineage<\/p>\n

    Read More<\/a><\/div>\n","protected":false},"author":3,"featured_media":15728,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_exactmetrics_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"footnotes":""},"categories":[776,29],"tags":[1695,1638,1637,1508,100,526,789,1672,190,738,68,88,1673],"class_list":["post-15712","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-etl","category-test-data","tag-ci-cd","tag-continuous-delivery","tag-continuous-integration","tag-devops","tag-etl","tag-iri-rowgen","tag-iri-voracity","tag-mlops","tag-realistic-test-data","tag-sdlc","tag-sortcl","tag-test-data-2","tag-testops"],"acf":[],"yoast_head":"\nSmart, Safe Test Data for DevOps, MLOps and DataOps - IRI<\/title>\n<meta name=\"description\" content=\"Data flowing through application development, machine learning, and analytic pipelines must address several needs common to all three, including\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/beta.iri.com\/blog\/test-data\/test-data-for-devops-mlops-dataops\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Smart, Safe Test Data for DevOps, MLOps and DataOps - IRI\" \/>\n<meta property=\"og:description\" content=\"Data flowing through application development, machine learning, and analytic pipelines must address several needs common to all three, including\" \/>\n<meta property=\"og:url\" content=\"https:\/\/beta.iri.com\/blog\/test-data\/test-data-for-devops-mlops-dataops\/\" \/>\n<meta property=\"og:site_name\" content=\"IRI\" \/>\n<meta property=\"article:published_time\" content=\"2022-03-24T17:52:39+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2022-06-13T13:16:41+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2022\/03\/Mlops-Devops.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1400\" \/>\n\t<meta property=\"og:image:height\" content=\"347\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"David Friedland\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"David Friedland\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/beta.iri.com\/blog\/test-data\/test-data-for-devops-mlops-dataops\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/test-data\/test-data-for-devops-mlops-dataops\/\"},\"author\":{\"name\":\"David Friedland\",\"@id\":\"https:\/\/beta.iri.com\/blog\/#\/schema\/person\/cdb89f0c0a9c88810b8516d4b140734a\"},\"headline\":\"Smart, Safe Test Data for DevOps, MLOps and DataOps\",\"datePublished\":\"2022-03-24T17:52:39+00:00\",\"dateModified\":\"2022-06-13T13:16:41+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/test-data\/test-data-for-devops-mlops-dataops\/\"},\"wordCount\":672,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/#organization\"},\"image\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/test-data\/test-data-for-devops-mlops-dataops\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2022\/03\/Mlops-Devops.png\",\"keywords\":[\"CI\/CD\",\"continuous delivery\",\"continuous integration\",\"DevOps\",\"ETL\",\"IRI RowGen\",\"IRI Voracity\",\"MLOps\",\"realistic test data\",\"SDLC\",\"SortCL\",\"test data\",\"TestOps\"],\"articleSection\":[\"ETL\",\"Test Data\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/beta.iri.com\/blog\/test-data\/test-data-for-devops-mlops-dataops\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/beta.iri.com\/blog\/test-data\/test-data-for-devops-mlops-dataops\/\",\"url\":\"https:\/\/beta.iri.com\/blog\/test-data\/test-data-for-devops-mlops-dataops\/\",\"name\":\"Smart, Safe Test Data for DevOps, MLOps and DataOps - IRI\",\"isPartOf\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/test-data\/test-data-for-devops-mlops-dataops\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/test-data\/test-data-for-devops-mlops-dataops\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2022\/03\/Mlops-Devops.png\",\"datePublished\":\"2022-03-24T17:52:39+00:00\",\"dateModified\":\"2022-06-13T13:16:41+00:00\",\"description\":\"Data flowing through application development, machine learning, and analytic pipelines must address several needs common to all three, including\",\"breadcrumb\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/test-data\/test-data-for-devops-mlops-dataops\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/beta.iri.com\/blog\/test-data\/test-data-for-devops-mlops-dataops\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/beta.iri.com\/blog\/test-data\/test-data-for-devops-mlops-dataops\/#primaryimage\",\"url\":\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2022\/03\/Mlops-Devops.png\",\"contentUrl\":\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2022\/03\/Mlops-Devops.png\",\"width\":1400,\"height\":347},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/beta.iri.com\/blog\/test-data\/test-data-for-devops-mlops-dataops\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/beta.iri.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Smart, Safe Test Data for DevOps, MLOps and DataOps\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/beta.iri.com\/blog\/#website\",\"url\":\"https:\/\/beta.iri.com\/blog\/\",\"name\":\"IRI\",\"description\":\"Total Data Management Blog\",\"publisher\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/beta.iri.com\/blog\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/beta.iri.com\/blog\/#organization\",\"name\":\"IRI\",\"url\":\"https:\/\/beta.iri.com\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/beta.iri.com\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2019\/02\/iri-logo-total-data-management-small-1.png\",\"contentUrl\":\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2019\/02\/iri-logo-total-data-management-small-1.png\",\"width\":750,\"height\":206,\"caption\":\"IRI\"},\"image\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/beta.iri.com\/blog\/#\/schema\/person\/cdb89f0c0a9c88810b8516d4b140734a\",\"name\":\"David Friedland\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/beta.iri.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/995ea08bc7d036da625671cb48a636eb?s=96&d=blank&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/995ea08bc7d036da625671cb48a636eb?s=96&d=blank&r=g\",\"caption\":\"David Friedland\"},\"url\":\"https:\/\/beta.iri.com\/blog\/author\/davidf\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Smart, Safe Test Data for DevOps, MLOps and DataOps - IRI","description":"Data flowing through application development, machine learning, and analytic pipelines must address several needs common to all three, including","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/beta.iri.com\/blog\/test-data\/test-data-for-devops-mlops-dataops\/","og_locale":"en_US","og_type":"article","og_title":"Smart, Safe Test Data for DevOps, MLOps and DataOps - IRI","og_description":"Data flowing through application development, machine learning, and analytic pipelines must address several needs common to all three, including","og_url":"https:\/\/beta.iri.com\/blog\/test-data\/test-data-for-devops-mlops-dataops\/","og_site_name":"IRI","article_published_time":"2022-03-24T17:52:39+00:00","article_modified_time":"2022-06-13T13:16:41+00:00","og_image":[{"width":1400,"height":347,"url":"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2022\/03\/Mlops-Devops.png","type":"image\/png"}],"author":"David Friedland","twitter_card":"summary_large_image","twitter_misc":{"Written by":"David Friedland","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/beta.iri.com\/blog\/test-data\/test-data-for-devops-mlops-dataops\/#article","isPartOf":{"@id":"https:\/\/beta.iri.com\/blog\/test-data\/test-data-for-devops-mlops-dataops\/"},"author":{"name":"David Friedland","@id":"https:\/\/beta.iri.com\/blog\/#\/schema\/person\/cdb89f0c0a9c88810b8516d4b140734a"},"headline":"Smart, Safe Test Data for DevOps, MLOps and DataOps","datePublished":"2022-03-24T17:52:39+00:00","dateModified":"2022-06-13T13:16:41+00:00","mainEntityOfPage":{"@id":"https:\/\/beta.iri.com\/blog\/test-data\/test-data-for-devops-mlops-dataops\/"},"wordCount":672,"commentCount":0,"publisher":{"@id":"https:\/\/beta.iri.com\/blog\/#organization"},"image":{"@id":"https:\/\/beta.iri.com\/blog\/test-data\/test-data-for-devops-mlops-dataops\/#primaryimage"},"thumbnailUrl":"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2022\/03\/Mlops-Devops.png","keywords":["CI\/CD","continuous delivery","continuous integration","DevOps","ETL","IRI RowGen","IRI Voracity","MLOps","realistic test data","SDLC","SortCL","test data","TestOps"],"articleSection":["ETL","Test Data"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/beta.iri.com\/blog\/test-data\/test-data-for-devops-mlops-dataops\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/beta.iri.com\/blog\/test-data\/test-data-for-devops-mlops-dataops\/","url":"https:\/\/beta.iri.com\/blog\/test-data\/test-data-for-devops-mlops-dataops\/","name":"Smart, Safe Test Data for DevOps, MLOps and DataOps - IRI","isPartOf":{"@id":"https:\/\/beta.iri.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/beta.iri.com\/blog\/test-data\/test-data-for-devops-mlops-dataops\/#primaryimage"},"image":{"@id":"https:\/\/beta.iri.com\/blog\/test-data\/test-data-for-devops-mlops-dataops\/#primaryimage"},"thumbnailUrl":"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2022\/03\/Mlops-Devops.png","datePublished":"2022-03-24T17:52:39+00:00","dateModified":"2022-06-13T13:16:41+00:00","description":"Data flowing through application development, machine learning, and analytic pipelines must address several needs common to all three, including","breadcrumb":{"@id":"https:\/\/beta.iri.com\/blog\/test-data\/test-data-for-devops-mlops-dataops\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/beta.iri.com\/blog\/test-data\/test-data-for-devops-mlops-dataops\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/beta.iri.com\/blog\/test-data\/test-data-for-devops-mlops-dataops\/#primaryimage","url":"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2022\/03\/Mlops-Devops.png","contentUrl":"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2022\/03\/Mlops-Devops.png","width":1400,"height":347},{"@type":"BreadcrumbList","@id":"https:\/\/beta.iri.com\/blog\/test-data\/test-data-for-devops-mlops-dataops\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/beta.iri.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Smart, Safe Test Data for DevOps, MLOps and DataOps"}]},{"@type":"WebSite","@id":"https:\/\/beta.iri.com\/blog\/#website","url":"https:\/\/beta.iri.com\/blog\/","name":"IRI","description":"Total Data Management Blog","publisher":{"@id":"https:\/\/beta.iri.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/beta.iri.com\/blog\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/beta.iri.com\/blog\/#organization","name":"IRI","url":"https:\/\/beta.iri.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/beta.iri.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2019\/02\/iri-logo-total-data-management-small-1.png","contentUrl":"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2019\/02\/iri-logo-total-data-management-small-1.png","width":750,"height":206,"caption":"IRI"},"image":{"@id":"https:\/\/beta.iri.com\/blog\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/beta.iri.com\/blog\/#\/schema\/person\/cdb89f0c0a9c88810b8516d4b140734a","name":"David Friedland","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/beta.iri.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/995ea08bc7d036da625671cb48a636eb?s=96&d=blank&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/995ea08bc7d036da625671cb48a636eb?s=96&d=blank&r=g","caption":"David Friedland"},"url":"https:\/\/beta.iri.com\/blog\/author\/davidf\/"}]}},"jetpack_featured_media_url":"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2022\/03\/Mlops-Devops.png","_links":{"self":[{"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/posts\/15712"}],"collection":[{"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/comments?post=15712"}],"version-history":[{"count":10,"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/posts\/15712\/revisions"}],"predecessor-version":[{"id":15926,"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/posts\/15712\/revisions\/15926"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/media\/15728"}],"wp:attachment":[{"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/media?parent=15712"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/categories?post=15712"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/tags?post=15712"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}