{"id":14473,"date":"2021-07-29T16:29:51","date_gmt":"2021-07-29T20:29:51","guid":{"rendered":"http:\/\/www.iri.com\/blog\/?p=14473"},"modified":"2022-02-01T11:05:52","modified_gmt":"2022-02-01T16:05:52","slug":"building-test-data-in-cicd-pipeline","status":"publish","type":"post","link":"https:\/\/beta.iri.com\/blog\/test-data\/building-test-data-in-cicd-pipeline\/","title":{"rendered":"Building Test Data in a CI\/CD Pipeline"},"content":{"rendered":"<p><i>Automating where you can, when you can, is the basis of DevOps automation. Allowing IRI job script executions which mask or synthesize test data to be handled in the CI\/CD pipeline supports this pursuit. This article illustrates an end-to-end example of a successful integration with <strong>GitLab<\/strong>. This is the first of four articles in a series; subsequent articles in this blog show how to feed IRI test data into <a href=\"https:\/\/www.iri.com\/blog\/data-protection\/masked-test-data-in-an-aws-codepipeline\/\">Amazon CodePipeline<\/a>, <a href=\"https:\/\/www.iri.com\/blog\/test-data\/test-data-azure-devops\/\">Azure DevOps<\/a> and <a href=\"https:\/\/www.iri.com\/blog\/test-data\/db-subsets-in-jenkins-pipeline\/\">Jenkins<\/a>.<\/i><i><br \/>\n<\/i><\/p>\n<p><a href=\"\/blog\/wp-content\/uploads\/2021\/07\/pipeline_flow.png\"><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-14476 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/07\/pipeline_flow.png\" alt=\"\" width=\"400\" height=\"455\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/pipeline_flow.png 631w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/pipeline_flow-264x300.png 264w\" sizes=\"(max-width: 400px) 100vw, 400px\" \/><\/a><\/p>\n<h6><b>The Growth of DevOps<\/b><\/h6>\n<p>DevOps is a phrase that many, if not all of us have heard when discussing the latest trends and movements in the tech industry. DevOps combines software development (Dev) and IT operations (Ops) to accelerate the systems development life cycle and provide continuous delivery of high quality software.<\/p>\n<p>DevOps processes are an integral part of product lifecycle management for large companies releasing new versions of source code. For example, Amazon is deploying code to production every 11.7 seconds; Netflix deploys thousands of times per day; and, Fidelity saved $2.3 million per year with their new release framework (Masse, Aug 6, 2019)<span id='easy-footnote-1-14473' class='easy-footnote-margin-adjust'><\/span><span class='easy-footnote'><a href='https:\/\/beta.iri.com\/blog\/test-data\/building-test-data-in-cicd-pipeline\/#easy-footnote-bottom-1-14473' title='Mass\u00e9, Nicolas. \u201c5 Principles for Deploying Your API from a CI\/CD Pipeline.\u201d &lt;i&gt;Red Hat Developer&lt;\/i&gt;, 6 Aug. 2019, &lt;a href=&quot;http:\/\/developers.redhat.com\/blog\/2019\/07\/26\/5-principles-for-deploying-your-api-from-a-ci-cd-pipeline.&quot;&gt;developers.redhat.com\/blog\/2019\/07\/26\/5-principles-for-deploying-your-api-from-a-ci-cd-pipeline&lt;\/a&gt;.'><sup>1<\/sup><\/a><\/span>.<\/p>\n<p>A common real-world example of DevOps CI\/CD in the frequent, iterative deployments of APIs (application programming interface). In recent years, APIs have become critical to the growth and revenue of companies. Salesforce generates 50% of its revenue through APIs, Expedia generates 90%, and eBay generates 60% (Masse, Sept 3, 2019)<span id='easy-footnote-2-14473' class='easy-footnote-margin-adjust'><\/span><span class='easy-footnote'><a href='https:\/\/beta.iri.com\/blog\/test-data\/building-test-data-in-cicd-pipeline\/#easy-footnote-bottom-2-14473' title='Mass\u00e9, Nicolas. \u201cFull API Lifecycle Management: A Primer.\u201d &lt;i&gt;Red Hat Developer&lt;\/i&gt;, 3 Sept. 2019, &lt;a href=&quot;http:\/\/developers.redhat.com\/blog\/2019\/02\/25\/full-api-lifecycle-management-a-primer&quot;&gt;developers.redhat.com\/blog\/2019\/02\/25\/full-api-lifecycle-management-a-primer&lt;\/a&gt;.'><sup>2<\/sup><\/a><\/span><\/p>\n<p style=\"text-align: center;\"><a href=\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2021\/07\/restapi.png\"><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-14477 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/07\/restapi-1024x307.png\" alt=\"\" width=\"601\" height=\"180\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/restapi-1024x307.png 1024w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/restapi-300x90.png 300w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/restapi-768x230.png 768w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/restapi.png 1453w\" sizes=\"(max-width: 601px) 100vw, 601px\" \/><\/a><i>Well Known APIs: Yahoo Finance API, Zillow API, and Open Weather API<\/i><\/p>\n<h6><b>What Is CI\/CD?<\/b><\/h6>\n<p>In DevOps, CI\/CD stands for Continuous Integration and Continuous Delivery\/Continuous Deployment. The phases of CI\/CD can be explained this way:<\/p>\n<ul>\n<li>\u00a0 \u00a0 \u00a0 \u00a0 <i>Continuous Integration<\/i> is the method by which software developers regularly deploy iterative code changes into a repository. Then a CI service will build and run tests against the new changes to the repository automatically.<\/li>\n<li>\u00a0 \u00a0 \u00a0 \u00a0 <i>Continuous Delivery<\/i> starts after CI. Code is deployed to a test or production environment. Automated tests may run during this stage. If all tests pass the code can be manually approved for update to production.<\/li>\n<li>\u00a0 \u00a0 \u00a0 \u00a0 <i>Continuous Deployment<\/i> differs from Continuous Delivery in that deployment to production happens without manual approval.<\/li>\n<\/ul>\n<h6><b>What is Being Demonstrated Here<\/b><\/h6>\n<p>As a part of DevOps CI\/CD, Continuous Testing is the testing and evaluation of software at every phase of the software development lifecycle. Testing software after it has been deployed to an environment is par for the course.<\/p>\n<p>For the sake of providing a demonstration that simulates a common real-world business practice, I am using an API that will be built, tested, and deployed through a GitLab CI\/CD pipeline. I am also incorporating IRI test-data-producing jobs into this same pipeline to automate the provisioning of test data in my test database DATASOURCE1 in an <a href=\"https:\/\/aws.amazon.com\/rds\/\">AWS RDS<\/a> instance.<\/p>\n<p>The IRI test-data-producing jobs will be used to populate the rows of a table called MEMBER_TB, with the fields ID, FIRSTNAME, LASTNAME, PHONE, EMAIL, STREETADDRESS, ZIPCITYSTATE, and DOB, in the test database (DATASOURCE1) that will be used during post-deployment testing. During post-deployment testing, the API in this demonstration will retrieve rows of test data from the table MEMBER_TB, located in the test database DATASOURCE1, using Postman.<\/p>\n<p>Please note that this article focuses on how to leverage a <a href=\"https:\/\/docs.gitlab.com\/ee\/ci\/pipelines\/\">GitLab CI\/CD pipeline<\/a> to automate IRI job script executions. As such, a basic understanding of the GitLab CI\/CD pipeline is assumed.<\/p>\n<h6><b>About IRI Test Data Jobs<\/b><\/h6>\n<p><a href=\"https:\/\/www.iri.com\/solutions\/test-data\">Fit-for-purpose<\/a>, standalone IRI software products &#8212; namely, <a href=\"https:\/\/www.iri.com\/products\/fieldshield\">IRI FieldShield<\/a> for searching and masking, and <a href=\"https:\/\/www.iri.com\/products\/rowgen\">IRI RowGen<\/a> for subsetting or synthesizing structured data from scratch &#8212; produce safe, realistic, and referentially correct test data. <a href=\"https:\/\/www.iri.com\/products\/darkshield\">IRI DarkShield<\/a> can find and mask sensitive data in semi- and unstructured sources.<\/p>\n<p>All of these products, along with others, are also supported <a href=\"https:\/\/www.iri.com\/products\/voracity\/technical-details#components\">components<\/a> of the <a href=\"https:\/\/www.iri.com\/products\/voracity\">IRI Voracity<\/a> data management platform.<\/p>\n<p style=\"text-align: center;\"><a href=\"\/blog\/wp-content\/uploads\/2021\/07\/bloor_cover.png\"><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-14478 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/07\/bloor_cover.png\" alt=\"\" width=\"300\" height=\"423\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/bloor_cover.png 400w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/bloor_cover-213x300.png 213w\" sizes=\"(max-width: 300px) 100vw, 300px\" \/><\/a><a href=\"https:\/\/www.iri.com\/resourceViewer\/pdf\/articles\/Bloor_Research-IRI_TDM_InBrief.php\"><i>Bloor Research Report on IRI Test Data Management<\/i><\/a><\/p>\n<p>In this POC, I use both FieldShield and RowGen to populate a GitLab pipeline with smart data that can be used in post-deployment testing.<\/p>\n<h6><b>Building the CI\/CD Pipeline in GitLab<\/b><\/h6>\n<p>GitLab is a service that provides remote access to git repositories and facilitates team collaboration. It supports the DevOps platform with its built-in tools for CI\/CD and issue tracking. Furthermore, GitLab offers a free version and has an active community in its forum.<\/p>\n<p>To use GitLab pipeline a file called<i> .gitlab-ci.yml<\/i> needs to be located in the root of the repository. GitLab will use runners to run the script found in the YAML file.<\/p>\n<p>Runners can be either shared or private runners installed locally on your machine. With GitLab\u2019s free tier, users can use shared runners for 400 minutes\u2019 worth of CI pipeline execution time.<\/p>\n<h6><b>Pipeline Structure<\/b><\/h6>\n<p>The CI\/CD pipeline consists of jobs which are written to do or perform a certain task. Jobs are contained in Docker images, and when the job ends the Docker images closes and all data is lost.<\/p>\n<p>To pass data from one job to another, a job artifact must be made. Jobs are designated to run in specific stages (i.e., build, test, staging, or production).<\/p>\n<p>If multiple jobs are run during the same stage, they will run in parallel with each other to save time. Unless configured to do otherwise, if any job fails in a stage, the pipeline will not move on to the next stage.<\/p>\n<p style=\"text-align: center;\"><a href=\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2021\/07\/cicdpipeline.png\"><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-14480 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/07\/cicdpipeline-1024x263.png\" alt=\"\" width=\"798\" height=\"205\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/cicdpipeline-1024x263.png 1024w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/cicdpipeline-300x77.png 300w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/cicdpipeline-768x197.png 768w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/cicdpipeline-1110x286.png 1110w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/cicdpipeline.png 1115w\" sizes=\"(max-width: 798px) 100vw, 798px\" \/><\/a><i>GitLab Pipeline<\/i><\/p>\n<h6><b>Pre-requisites<\/b><\/h6>\n<ul>\n<li>\u00a0 \u00a0 Your IRI software product is installed and licensed on the target remote server (the remote server that the GitLab runner will access via SSH could be your on-premise computer, an AWS EC2 instance, etc.).<\/li>\n<li>\u00a0 \u00a0 The server where the IRI job script is located can be accessed with SSH.<\/li>\n<li>\u00a0 \u00a0 An AWS account (for my purposes the free tier was adequate).<\/li>\n<li>\u00a0 \u00a0 A relational database instance with appropriate inbound and outbound rules.<\/li>\n<li>\u00a0 \u00a0 An AWS Elastic Beanstalk web server environment.<\/li>\n<li>\u00a0 \u00a0 An S3 Bucket.<\/li>\n<li>\u00a0 \u00a0 An IAM user with programmatic access and the necessary permissions set (AmazonS3FullAccess and AdministratorAccess-AWSElasticBeanstalk).<\/li>\n<li>\u00a0 \u00a0 AWS credentials stored as environment variables in GitLab.<\/li>\n<\/ul>\n<h6><b>CI\/CD Pipeline Breakdown<\/b><\/h6>\n<h6><strong>Defining stages, in script variables, and services<\/strong><\/h6>\n<p style=\"text-align: center;\"><a href=\"\/blog\/wp-content\/uploads\/2021\/07\/yml_top.png\"><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-14481 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/07\/yml_top.png\" alt=\"\" width=\"799\" height=\"248\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/yml_top.png 822w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/yml_top-300x93.png 300w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/yml_top-768x238.png 768w\" sizes=\"(max-width: 799px) 100vw, 799px\" \/><\/a><i>Top of .gitlab-ci.yml file.<\/i><\/p>\n<h6><b>Build Stage<\/b><\/h6>\n<p style=\"text-align: center;\"><a href=\"\/blog\/wp-content\/uploads\/2021\/07\/yml_build.png\"><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-14482 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/07\/yml_build.png\" alt=\"\" width=\"801\" height=\"222\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/yml_build.png 862w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/yml_build-300x83.png 300w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/yml_build-768x213.png 768w\" sizes=\"(max-width: 801px) 100vw, 801px\" \/><\/a><i>\u00a0Build Job<\/i><\/p>\n<p style=\"text-align: center;\"><a href=\"\/blog\/wp-content\/uploads\/2021\/07\/application_properties.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-14483 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/07\/application_properties.png\" alt=\"\" width=\"799\" height=\"240\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/application_properties.png 776w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/application_properties-300x90.png 300w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/application_properties-768x231.png 768w\" sizes=\"(max-width: 799px) 100vw, 799px\" \/><\/a><i>application.properties file<\/i><\/p>\n<p>The placeholders in the API\u2019s application.properties file, must be dynamically replaced with the credentials for the test database DATASOURCE1 in my AWS RDS instance. The sed command is used to find and substitute these placeholders with the variables stored in GitLab.<\/p>\n<h6><b>Test Stage<\/b><\/h6>\n<p style=\"text-align: center;\"><a href=\"\/blog\/wp-content\/uploads\/2021\/07\/yml_smoke_test.png\"><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-14484 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/07\/yml_smoke_test.png\" alt=\"\" width=\"799\" height=\"171\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/yml_smoke_test.png 860w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/yml_smoke_test-300x64.png 300w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/yml_smoke_test-768x164.png 768w\" sizes=\"(max-width: 799px) 100vw, 799px\" \/><\/a><i>Smoke Test Job<\/i><\/p>\n<p style=\"text-align: center;\"><a href=\"\/blog\/wp-content\/uploads\/2021\/07\/yml_junit.png\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-14485 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/07\/yml_junit.png\" alt=\"\" width=\"873\" height=\"278\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/yml_junit.png 873w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/yml_junit-300x96.png 300w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/yml_junit-768x245.png 768w\" sizes=\"(max-width: 873px) 100vw, 873px\" \/><\/a><i>\u00a0Unit Tests Job<\/i><\/p>\n<p>GitLab provides some useful documentation on how to produce<a href=\"https:\/\/docs.gitlab.com\/ee\/ci\/unit_test_reports.html\"> Unit test reports<\/a>.<\/p>\n<h6><b>Deploy Stage<\/b><\/h6>\n<p>Before trying to SSH to a remote machine from the pipeline, you must first verify that the target machine has SSH enabled. If you are running Windows, it will not be enabled automatically.<\/p>\n<p>To install SSH for Windows 10 you must do the following:<\/p>\n<ol>\n<li aria-level=\"1\">Go to <i>Settings .<\/i><\/li>\n<li aria-level=\"1\"><i>Apps<\/i> and click \u201c<i>Optional features<\/i>\u201d under Apps &amp; Features.<\/li>\n<li aria-level=\"1\">Click \u201c<i>Add a feature<\/i>\u201d at the top of the list.<\/li>\n<li aria-level=\"1\">Select from list \u201c<i>OpenSSH Server<\/i>\u201d, then install.<\/li>\n<\/ol>\n<p><a href=\"\/blog\/wp-content\/uploads\/2021\/07\/features.png\"><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-14486 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/07\/features.png\" alt=\"\" width=\"329\" height=\"569\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/features.png 472w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/features-174x300.png 174w\" sizes=\"(max-width: 329px) 100vw, 329px\" \/><\/a><\/p>\n<p>To enable OpenSSH Server and SSH Agent Authorization (SSH from within CI pipeline requires your target server to have both enabled):<\/p>\n<ol>\n<li aria-level=\"1\">Type Services in your search bar and open <i>Services.<\/i><\/li>\n<li aria-level=\"1\">Scroll down until you find OpenSSH Server and right-click.<\/li>\n<li aria-level=\"1\">Click properties.<\/li>\n<li aria-level=\"1\">In the general tab, click the dropdown list for startup types.<\/li>\n<li aria-level=\"1\">Set to Automatic then click apply.<\/li>\n<li aria-level=\"1\">Repeat this process for OpenSSH Authorization Agent.<\/li>\n<\/ol>\n<p><a href=\"\/blog\/wp-content\/uploads\/2021\/07\/openssh.png\"><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-14489 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/07\/openssh.png\" alt=\"\" width=\"808\" height=\"409\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/openssh.png 842w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/openssh-300x152.png 300w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/openssh-768x389.png 768w\" sizes=\"(max-width: 808px) 100vw, 808px\" \/><\/a><\/p>\n<p>In GitLab you will need to store a copy of your IP address for the target machine in a variable. I stored my IP address in the variable IP_ADDRESS. In my case, the IP address I used was the IP address for my router with port forwarding enabled on port 22. For those not familiar with port forwarding, here is a helpful <a href=\"https:\/\/www.noip.com\/support\/knowledgebase\/general-port-forwarding-guide\/\">link<\/a>.<\/p>\n<p><a href=\"\/blog\/wp-content\/uploads\/2021\/07\/yml_ssh.png\"><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-14490 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/07\/yml_ssh.png\" alt=\"\" width=\"779\" height=\"275\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/yml_ssh.png 782w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/yml_ssh-300x106.png 300w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/yml_ssh-768x271.png 768w\" sizes=\"(max-width: 779px) 100vw, 779px\" \/><\/a><\/p>\n<p>In the GitLab pipeline\u2019s job called IRI script, an Ubuntu image has been set for the docker container. Before creating an SSH connection, OpenSSH Client is installed, an SSH agent is set up, and a private key has been added to the SSH agent. The GitLab runner can now SSH to the server.<\/p>\n<p>The string value stored in SSH_COMMAND variable, looks something like this:<\/p>\n<pre>cd path\/to\/rowgen\/job &amp;&amp; rowgen_job.bat &amp;&amp; cd path\/to\/fieldshield\/job &amp;&amp; sortcl.exe \/spec=fieldshield_job.fcl<\/pre>\n<p>To generate test data for the test database, I used a RowGen job script. When executed via SSH command, the MEMBER_TB table present in the Amazon RDS instance is populated with synthetic test data.<\/p>\n<p>For the sake of demonstration purposes, I also ran a FieldShield job script, via SSH commands, issued from within the GitLab CI\/CD pipeline.<\/p>\n<p>The FieldShield job accesses the MEMBER_TB table and masks sensitive fields that contain PII (DOB, EMAIL, STREETADDRESS, ZIPCITYSTATE).<\/p>\n<p>You can verify that the IRI job scripts have generated data in your database, or from the <a href=\"https:\/\/www.iri.com\/products\/workbench\">IRI Workbench<\/a> IDE, built on Eclipse\u2122. Open your <a href=\"https:\/\/www.iri.com\/services\/training\/courseware#install\">JDBC-connected<\/a> schema in the Workbench Data Source Explorer view, and:<\/p>\n<ol>\n<li aria-level=\"1\">Navigate down from the target Connection Profile (in my case \u201cAWS MYSQL\u201d).<\/li>\n<li aria-level=\"1\">Find the target table in the target database (in my case \u201cMEMBER_TB\u201d in database \u201cDATASOURCE1\u201d).<\/li>\n<li aria-level=\"1\">Right click the table you wish to use.<\/li>\n<li aria-level=\"1\">Hover over Data and slide the mouse right.<\/li>\n<li aria-level=\"1\">Click Sample Contents.<\/li>\n<\/ol>\n<p><a href=\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2021\/07\/iri_results.png\"><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-14491 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/07\/iri_results-1024x456.png\" alt=\"\" width=\"799\" height=\"356\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/iri_results-1024x456.png 1024w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/iri_results-300x134.png 300w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/iri_results-768x342.png 768w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/iri_results.png 1455w\" sizes=\"(max-width: 799px) 100vw, 799px\" \/><\/a><\/p>\n<h6><b>RowGen Test Data Synthesis Job<\/b><\/h6>\n<p>IRI provides easy-to-use end-user job <a href=\"https:\/\/www.iri.com\/products\/workbench\/rowgen-gui\">wizards for RowGen<\/a> that build the job scripts that will synthesize and populate test data. For my purposes, I used the <a href=\"https:\/\/www.iri.com\/blog\/vldb-operations\/how-to-generate-db-test-data-2\/\"><i>New RowGen Database Test Data Job<\/i><\/a> wizard.<\/p>\n<p>Before using the wizard, I created a database connection using JDBC in the Workbench Data Source Explorer. After connecting to my database, I can select the MEMBER_TB table in the wizard. When advancing through the wizard, you will be presented with the <i>New Field Rule Wizard Selection<\/i> dialog box.<\/p>\n<p><a href=\"\/blog\/wp-content\/uploads\/2021\/07\/fieldrulewizard_lower.png\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-14502 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/07\/fieldrulewizard_lower.png\" alt=\"\" width=\"509\" height=\"415\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/fieldrulewizard_lower.png 509w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/fieldrulewizard_lower-300x245.png 300w\" sizes=\"(max-width: 509px) 100vw, 509px\" \/><\/a><\/p>\n<p>This dialog box allows users to choose from and create various field-level generation rules. For my MEMBER_TB table, I have the ID, FIRSTNAME, LASTNAME, EMAIL, PHONE, STREETADDRESS, ZIPCITYSTATE, and DOB fields.<\/p>\n<p>For the DOB field, I used the <i>Date Range Generator<\/i> rule to create a list of randomly generated dates from a predetermined range of dates, to populate the DOB field.<\/p>\n<p>Apart from the DOB field, all other fields use the <i>Set File Selection<\/i> rule to insert randomly selected data from set files included with my IRI software installation. Set files are collections of similarly grouped objects, with each object listed on a new line in the set file.<\/p>\n<p>After completing the wizard, the necessary RowGen job script (.rcl) is created to specify the generation of test data for the MEMBER_TB table. A batch file is also created, which when run from either the GUI or command line, will execute the steps to synthesize the test data and populate the target(s).<\/p>\n<p><a href=\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2021\/07\/rg_script_pt1.png\"><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-14492 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/07\/rg_script_pt1-1024x368.png\" alt=\"\" width=\"801\" height=\"288\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/rg_script_pt1-1024x368.png 1024w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/rg_script_pt1-300x108.png 300w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/rg_script_pt1-768x276.png 768w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/rg_script_pt1.png 1110w\" sizes=\"(max-width: 801px) 100vw, 801px\" \/><\/a><\/p>\n<p><a href=\"\/blog\/wp-content\/uploads\/2021\/07\/rg_script_pt2.png\"><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-14493 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/07\/rg_script_pt2.png\" alt=\"\" width=\"799\" height=\"231\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/rg_script_pt2.png 834w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/rg_script_pt2-300x87.png 300w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/rg_script_pt2-768x222.png 768w\" sizes=\"(max-width: 799px) 100vw, 799px\" \/><\/a><\/p>\n<h6><b>FieldShield Data Masking Job<\/b><\/h6>\n<p>For the sake of demonstration purposes, I also created a FieldShield job script that is executed when SSH commands are received from the GitLab pipeline.<\/p>\n<p>Using FieldShield scripts, personally identifying information (PII) is masked at the field (column) level. Masking can occur on a production snapshot (as it\u2019s built, ask IRI), or on-the-fly (ETL-style) as data are copied from production to the lower environment.<\/p>\n<p>The IRI Workbench <a href=\"https:\/\/www.iri.com\/products\/workbench\/fieldshield-gui\">GUI for FieldShield<\/a> has wizards that assist in the creation of FieldShield job scripts. I used the <i>New Masking Job<\/i> wizard to produce the necessary script (.fcl).<\/p>\n<p>When proceeding through the wizard, you will need to specify an input datasource (ODBC), DSN, and the target table (MEMBER_TB) where the fields will be modified. Afterwards, the wizard will ask the user to add masking rules to individual columns using the <i>New Field Rule Selection<\/i> dialog box.<\/p>\n<p><a href=\"\/blog\/wp-content\/uploads\/2021\/07\/fieldrulewizard_upper.png\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-14503 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/07\/fieldrulewizard_upper.png\" alt=\"\" width=\"511\" height=\"380\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/fieldrulewizard_upper.png 511w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/fieldrulewizard_upper-300x223.png 300w\" sizes=\"(max-width: 511px) 100vw, 511px\" \/><\/a><\/p>\n<p>In my demonstration, I am using the Field Encryption rule to apply AES encryption to the EMAIL, STREETADDRESS, and ZIPCITYSTATE fields. Lastly, for the DOB field, I use the Redaction Function rule (though not the blur function is also popular for dates and ages).<\/p>\n<p>After completing the wizard, the necessary script (.fcl) is created to specify the masking rules for the selected columns in the MEMBER_TB table.<\/p>\n<p><a href=\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2021\/07\/fs_script_pt1.png\"><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-14495 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/07\/fs_script_pt1-1024x366.png\" alt=\"\" width=\"800\" height=\"286\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/fs_script_pt1-1024x366.png 1024w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/fs_script_pt1-300x107.png 300w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/fs_script_pt1-768x275.png 768w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/fs_script_pt1.png 1520w\" sizes=\"(max-width: 800px) 100vw, 800px\" \/><\/a><\/p>\n<p><a href=\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2021\/07\/fs_script_pt2.png\"><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-14494 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/07\/fs_script_pt2-1024x227.png\" alt=\"\" width=\"798\" height=\"177\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/fs_script_pt2-1024x227.png 1024w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/fs_script_pt2-300x66.png 300w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/fs_script_pt2-768x170.png 768w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/fs_script_pt2.png 1070w\" sizes=\"(max-width: 798px) 100vw, 798px\" \/><\/a><\/p>\n<h6><b>Deployment Job Using AWS Elastic Beanstalk and S3<\/b><\/h6>\n<p style=\"text-align: center;\"><a href=\"\/blog\/wp-content\/uploads\/2021\/07\/yml_deploy.png\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-14496 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/07\/yml_deploy.png\" alt=\"\" width=\"915\" height=\"238\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/yml_deploy.png 915w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/yml_deploy-300x78.png 300w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/yml_deploy-768x200.png 768w\" sizes=\"(max-width: 915px) 100vw, 915px\" \/><\/a><i>Deploy Job<\/i><\/p>\n<p>This pipeline job is called <i>Deploy<\/i>. It will configure the GitLab pipeline to deploy the application previously built in the<i> Build <\/i>job, to AWS Elastic Beanstalk.<\/p>\n<p>After the application is deployed, we should be able to access the data in the test database DATASOURCE1, using the API endpoint Beanstalk provides. This endpoint will later be used in post-deployment testing, using Postman.<\/p>\n<p>See <a href=\"https:\/\/dba-presents.com\/index.php\/cloud\/amazon-web-services\/224-deployment-to-aws-elastic-beanstalk-with-gitlab-ci-cd\">this article<\/a> for more information on how to deploy using AWS Elastic Beanstalk and S3 from the GitLab pipeline.<\/p>\n<p style=\"text-align: center;\"><a href=\"\/blog\/wp-content\/uploads\/2021\/07\/aws_elastic_beanstalk.png\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-14479 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/07\/aws_elastic_beanstalk.png\" alt=\"\" width=\"624\" height=\"161\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/aws_elastic_beanstalk.png 624w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/aws_elastic_beanstalk-300x77.png 300w\" sizes=\"(max-width: 624px) 100vw, 624px\" \/><\/a><i>AWS Elastic Beanstalk environment management console.<\/i><\/p>\n<h6><b>Post-Deploy Stage<\/b><\/h6>\n<p style=\"text-align: center;\"><a href=\"\/blog\/wp-content\/uploads\/2021\/07\/post_deploy.png\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-14497 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/07\/post_deploy.png\" alt=\"\" width=\"624\" height=\"128\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/post_deploy.png 624w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/post_deploy-300x62.png 300w\" sizes=\"(max-width: 624px) 100vw, 624px\" \/><\/a><i>Post-Deploy Job<\/i><\/p>\n<p>For those not familiar with using Postman in the GitLab pipeline, here is a helpful<a href=\"https:\/\/medium.com\/apis-with-valentine\/how-to-run-postman-api-tests-with-newman-in-gitlab-ci-the-proper-way-b5a9f6076d79\"> article<\/a>.<\/p>\n<p style=\"text-align: center;\"><a href=\"\/blog\/wp-content\/uploads\/2021\/07\/post_deploy_output.png\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-14498 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/07\/post_deploy_output.png\" alt=\"\" width=\"624\" height=\"386\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/post_deploy_output.png 624w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/post_deploy_output-300x186.png 300w\" sizes=\"(max-width: 624px) 100vw, 624px\" \/><\/a><i>Output of Post-Deploy job log in GitLab.<\/i><\/p>\n<p>During the Postman tests, a GET request was made to the API hosted on Elastic Beanstalk. The API responded with the contents of the MEMBER_TB table in the test database DATASOURCE1. The MEMBER_TB table was not empty because the IRI job scripts had generated test data in the MEMBER_TB table before the Postman tests had begun.<\/p>\n<p>Although I chose to use Postman to test my REST API, there are plenty of alternative tools out there based on your personal preferences.<\/p>\n<h6><b>JSON Response to GET Request to REST API Hosted on AWS Elastic Beanstalk<\/b><\/h6>\n<p>When the API deploys, the test data can be retrieved in JSON format when accessing the API endpoint (URL of server).<\/p>\n<p style=\"text-align: center;\"><a href=\"\/blog\/wp-content\/uploads\/2021\/07\/get_request_postman.png\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-14499 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/07\/get_request_postman.png\" alt=\"\" width=\"624\" height=\"269\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/get_request_postman.png 624w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/get_request_postman-300x129.png 300w\" sizes=\"(max-width: 624px) 100vw, 624px\" \/><\/a><i>API response to GET request from Postman.<\/i><\/p>\n<h6><b>Procedural Summary<\/b><\/h6>\n<p>This article demonstrated how to automate the execution of IRI test-data-producing jobs from within a GitLab CI\/CD pipeline. Using SSH, any IRI job scripts that can run via the command line can be executed from the pipeline.<\/p>\n<p>IRI RowGen jobs synthesize test data while keeping real data out of non-secure environments. Alternatively, IRI FieldShield job scripts find and mask sensitive data at the field level, allowing a lot of real data copied from production environments to be used securely in test environments.<\/p>\n<p>The IRI job scripts in this demonstration provided safe data for post-deployment testing. The API deployed in this demonstration would later retrieve these newly generated rows of data from the AWS test database (DATASOURCE1) as part of the testing process in post-deployment.<\/p>\n<p>If GitLab is not for you, there are plenty of CI\/CD pipeline alternatives that can execute IRI job scripts in a similar fashion. Although the syntax may differ slightly, <a href=\"https:\/\/www.iri.com\/blog\/data-protection\/masked-test-data-in-an-aws-codepipeline\/\">Amazon CodePipeline<\/a>, <a href=\"https:\/\/www.iri.com\/blog\/test-data\/test-data-azure-devops\/\">Azure DevOps<\/a>, BitBucket Pipelines &amp; Bamboo, CircleCI, GitHub Actions, Jenkins, and Travis CI pipelines all support SSH.<\/p>\n<p>Automation in DevOps is motivated by the desire for consistency, speed, and reliability. The more that can be left to automation, the more productive testing can become.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Automating where you can, when you can, is the basis of DevOps automation. Allowing IRI job script executions which mask or synthesize test data to be handled in the CI\/CD pipeline supports this pursuit. This article illustrates an end-to-end example of a successful integration with GitLab. This is the first of four articles in a<\/p>\n<div><a class=\"btn-filled btn\" href=\"https:\/\/beta.iri.com\/blog\/test-data\/building-test-data-in-cicd-pipeline\/\" title=\"Building Test Data in a CI\/CD Pipeline\">Read More<\/a><\/div>\n","protected":false},"author":152,"featured_media":14480,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_exactmetrics_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"footnotes":""},"categories":[29],"tags":[1636,1638,1637,1508,1507,520,526,1639,763,88,379],"class_list":["post-14473","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-test-data","tag-cicd","tag-continuous-delivery","tag-continuous-integration","tag-devops","tag-gitlab","tag-iri-fieldshield","tag-iri-rowgen","tag-synthetic-data","tag-synthetic-xml","tag-test-data-2","tag-test-data-management"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v23.3 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Building Test Data in a CI\/CD Pipeline - IRI<\/title>\n<meta name=\"description\" content=\"DevOps is a phrase that many, if not all of us have heard when discussing the latest trends and movements in the tech industry. DevOps combines software\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/beta.iri.com\/blog\/test-data\/building-test-data-in-cicd-pipeline\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Building Test Data in a CI\/CD Pipeline - IRI\" \/>\n<meta property=\"og:description\" content=\"DevOps is a phrase that many, if not all of us have heard when discussing the latest trends and movements in the tech industry. DevOps combines software\" \/>\n<meta property=\"og:url\" content=\"https:\/\/beta.iri.com\/blog\/test-data\/building-test-data-in-cicd-pipeline\/\" \/>\n<meta property=\"og:site_name\" content=\"IRI\" \/>\n<meta property=\"article:published_time\" content=\"2021-07-29T20:29:51+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2022-02-01T16:05:52+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/cicdpipeline.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1115\" \/>\n\t<meta property=\"og:image:height\" content=\"286\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Adam Lewis\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Adam Lewis\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"12 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/beta.iri.com\/blog\/test-data\/building-test-data-in-cicd-pipeline\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/test-data\/building-test-data-in-cicd-pipeline\/\"},\"author\":{\"name\":\"Adam Lewis\",\"@id\":\"https:\/\/beta.iri.com\/blog\/#\/schema\/person\/37c0e5beab094bd61cc521902df2876e\"},\"headline\":\"Building Test Data in a CI\/CD Pipeline\",\"datePublished\":\"2021-07-29T20:29:51+00:00\",\"dateModified\":\"2022-02-01T16:05:52+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/test-data\/building-test-data-in-cicd-pipeline\/\"},\"wordCount\":2465,\"commentCount\":2,\"publisher\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/#organization\"},\"image\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/test-data\/building-test-data-in-cicd-pipeline\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/cicdpipeline.png\",\"keywords\":[\"CICD\",\"continuous delivery\",\"continuous integration\",\"DevOps\",\"GitLab\",\"IRI FieldShield\",\"IRI RowGen\",\"synthetic data\",\"synthetic xml\",\"test data\",\"test data management\"],\"articleSection\":[\"Test Data\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/beta.iri.com\/blog\/test-data\/building-test-data-in-cicd-pipeline\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/beta.iri.com\/blog\/test-data\/building-test-data-in-cicd-pipeline\/\",\"url\":\"https:\/\/beta.iri.com\/blog\/test-data\/building-test-data-in-cicd-pipeline\/\",\"name\":\"Building Test Data in a CI\/CD Pipeline - IRI\",\"isPartOf\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/test-data\/building-test-data-in-cicd-pipeline\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/test-data\/building-test-data-in-cicd-pipeline\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/cicdpipeline.png\",\"datePublished\":\"2021-07-29T20:29:51+00:00\",\"dateModified\":\"2022-02-01T16:05:52+00:00\",\"description\":\"DevOps is a phrase that many, if not all of us have heard when discussing the latest trends and movements in the tech industry. DevOps combines software\",\"breadcrumb\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/test-data\/building-test-data-in-cicd-pipeline\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/beta.iri.com\/blog\/test-data\/building-test-data-in-cicd-pipeline\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/beta.iri.com\/blog\/test-data\/building-test-data-in-cicd-pipeline\/#primaryimage\",\"url\":\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/cicdpipeline.png\",\"contentUrl\":\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/cicdpipeline.png\",\"width\":1115,\"height\":286},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/beta.iri.com\/blog\/test-data\/building-test-data-in-cicd-pipeline\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/beta.iri.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Building Test Data in a CI\/CD Pipeline\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/beta.iri.com\/blog\/#website\",\"url\":\"https:\/\/beta.iri.com\/blog\/\",\"name\":\"IRI\",\"description\":\"Total Data Management Blog\",\"publisher\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/beta.iri.com\/blog\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/beta.iri.com\/blog\/#organization\",\"name\":\"IRI\",\"url\":\"https:\/\/beta.iri.com\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/beta.iri.com\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2019\/02\/iri-logo-total-data-management-small-1.png\",\"contentUrl\":\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2019\/02\/iri-logo-total-data-management-small-1.png\",\"width\":750,\"height\":206,\"caption\":\"IRI\"},\"image\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/beta.iri.com\/blog\/#\/schema\/person\/37c0e5beab094bd61cc521902df2876e\",\"name\":\"Adam Lewis\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/beta.iri.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/087667d0c75d33bb6fab6e734bd89333?s=96&d=blank&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/087667d0c75d33bb6fab6e734bd89333?s=96&d=blank&r=g\",\"caption\":\"Adam Lewis\"},\"url\":\"https:\/\/beta.iri.com\/blog\/author\/adaml\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Building Test Data in a CI\/CD Pipeline - IRI","description":"DevOps is a phrase that many, if not all of us have heard when discussing the latest trends and movements in the tech industry. DevOps combines software","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/beta.iri.com\/blog\/test-data\/building-test-data-in-cicd-pipeline\/","og_locale":"en_US","og_type":"article","og_title":"Building Test Data in a CI\/CD Pipeline - IRI","og_description":"DevOps is a phrase that many, if not all of us have heard when discussing the latest trends and movements in the tech industry. DevOps combines software","og_url":"https:\/\/beta.iri.com\/blog\/test-data\/building-test-data-in-cicd-pipeline\/","og_site_name":"IRI","article_published_time":"2021-07-29T20:29:51+00:00","article_modified_time":"2022-02-01T16:05:52+00:00","og_image":[{"width":1115,"height":286,"url":"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/cicdpipeline.png","type":"image\/png"}],"author":"Adam Lewis","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Adam Lewis","Est. reading time":"12 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/beta.iri.com\/blog\/test-data\/building-test-data-in-cicd-pipeline\/#article","isPartOf":{"@id":"https:\/\/beta.iri.com\/blog\/test-data\/building-test-data-in-cicd-pipeline\/"},"author":{"name":"Adam Lewis","@id":"https:\/\/beta.iri.com\/blog\/#\/schema\/person\/37c0e5beab094bd61cc521902df2876e"},"headline":"Building Test Data in a CI\/CD Pipeline","datePublished":"2021-07-29T20:29:51+00:00","dateModified":"2022-02-01T16:05:52+00:00","mainEntityOfPage":{"@id":"https:\/\/beta.iri.com\/blog\/test-data\/building-test-data-in-cicd-pipeline\/"},"wordCount":2465,"commentCount":2,"publisher":{"@id":"https:\/\/beta.iri.com\/blog\/#organization"},"image":{"@id":"https:\/\/beta.iri.com\/blog\/test-data\/building-test-data-in-cicd-pipeline\/#primaryimage"},"thumbnailUrl":"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/cicdpipeline.png","keywords":["CICD","continuous delivery","continuous integration","DevOps","GitLab","IRI FieldShield","IRI RowGen","synthetic data","synthetic xml","test data","test data management"],"articleSection":["Test Data"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/beta.iri.com\/blog\/test-data\/building-test-data-in-cicd-pipeline\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/beta.iri.com\/blog\/test-data\/building-test-data-in-cicd-pipeline\/","url":"https:\/\/beta.iri.com\/blog\/test-data\/building-test-data-in-cicd-pipeline\/","name":"Building Test Data in a CI\/CD Pipeline - IRI","isPartOf":{"@id":"https:\/\/beta.iri.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/beta.iri.com\/blog\/test-data\/building-test-data-in-cicd-pipeline\/#primaryimage"},"image":{"@id":"https:\/\/beta.iri.com\/blog\/test-data\/building-test-data-in-cicd-pipeline\/#primaryimage"},"thumbnailUrl":"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/cicdpipeline.png","datePublished":"2021-07-29T20:29:51+00:00","dateModified":"2022-02-01T16:05:52+00:00","description":"DevOps is a phrase that many, if not all of us have heard when discussing the latest trends and movements in the tech industry. DevOps combines software","breadcrumb":{"@id":"https:\/\/beta.iri.com\/blog\/test-data\/building-test-data-in-cicd-pipeline\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/beta.iri.com\/blog\/test-data\/building-test-data-in-cicd-pipeline\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/beta.iri.com\/blog\/test-data\/building-test-data-in-cicd-pipeline\/#primaryimage","url":"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/cicdpipeline.png","contentUrl":"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/cicdpipeline.png","width":1115,"height":286},{"@type":"BreadcrumbList","@id":"https:\/\/beta.iri.com\/blog\/test-data\/building-test-data-in-cicd-pipeline\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/beta.iri.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Building Test Data in a CI\/CD Pipeline"}]},{"@type":"WebSite","@id":"https:\/\/beta.iri.com\/blog\/#website","url":"https:\/\/beta.iri.com\/blog\/","name":"IRI","description":"Total Data Management Blog","publisher":{"@id":"https:\/\/beta.iri.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/beta.iri.com\/blog\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/beta.iri.com\/blog\/#organization","name":"IRI","url":"https:\/\/beta.iri.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/beta.iri.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2019\/02\/iri-logo-total-data-management-small-1.png","contentUrl":"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2019\/02\/iri-logo-total-data-management-small-1.png","width":750,"height":206,"caption":"IRI"},"image":{"@id":"https:\/\/beta.iri.com\/blog\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/beta.iri.com\/blog\/#\/schema\/person\/37c0e5beab094bd61cc521902df2876e","name":"Adam Lewis","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/beta.iri.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/087667d0c75d33bb6fab6e734bd89333?s=96&d=blank&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/087667d0c75d33bb6fab6e734bd89333?s=96&d=blank&r=g","caption":"Adam Lewis"},"url":"https:\/\/beta.iri.com\/blog\/author\/adaml\/"}]}},"jetpack_featured_media_url":"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/07\/cicdpipeline.png","_links":{"self":[{"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/posts\/14473"}],"collection":[{"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/users\/152"}],"replies":[{"embeddable":true,"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/comments?post=14473"}],"version-history":[{"count":19,"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/posts\/14473\/revisions"}],"predecessor-version":[{"id":15589,"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/posts\/14473\/revisions\/15589"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/media\/14480"}],"wp:attachment":[{"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/media?parent=14473"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/categories?post=14473"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/tags?post=14473"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}