{"id":15169,"date":"2021-11-10T16:30:26","date_gmt":"2021-11-10T21:30:26","guid":{"rendered":"http:\/\/www.iri.com\/blog\/?p=15169"},"modified":"2022-05-04T12:24:39","modified_gmt":"2022-05-04T16:24:39","slug":"test-data-azure-devops","status":"publish","type":"post","link":"https:\/\/beta.iri.com\/blog\/test-data\/test-data-azure-devops\/","title":{"rendered":"Generating Test Data for Azure DevOps"},"content":{"rendered":"<p><i><span style=\"font-weight: 400;\">This is the third addition to our 4-article DevOps pipeline series. Here are the links to related DevOps articles for <a href=\"https:\/\/www.iri.com\/blog\/test-data\/db-subsets-in-jenkins-pipeline\/\">Jenkins<\/a>, <\/span><\/i><a href=\"https:\/\/www.iri.com\/blog\/test-data\/building-test-data-in-cicd-pipeline\/\"><i><span style=\"font-weight: 400;\">GitLab<\/span><\/i><\/a><i><span style=\"font-weight: 400;\"> and <\/span><\/i><a href=\"https:\/\/www.iri.com\/blog\/data-protection\/masked-test-data-in-an-aws-codepipeline\/\"><i><span style=\"font-weight: 400;\">AWS CodePipeline<\/span><\/i><\/a><i><span style=\"font-weight: 400;\">.<\/span><\/i><\/p>\n<h6><b>What is Being Demonstrated?<\/b><\/h6>\n<p><span style=\"font-weight: 400;\">In this article I demonstrate how to use <\/span><a href=\"https:\/\/www.iri.com\/pdf\/articles\/Bloor_Research-IRI_TDM_InBrief.pdf\"><span style=\"font-weight: 400;\">IRI Voracity TDM<\/span><\/a><span style=\"font-weight: 400;\"> platform software from within an <\/span><a href=\"https:\/\/azure.microsoft.com\/en-us\/services\/devops\/pipelines\/\"><span style=\"font-weight: 400;\">Azure DevOps<\/span><\/a><span style=\"font-weight: 400;\"> pipeline to produce and use realistic test data in different sources for CI\/CD purposes. Specifically, I run <\/span><a href=\"https:\/\/www.iri.com\/products\/rowgen\"><span style=\"font-weight: 400;\">IRI RowGen<\/span><\/a><span style=\"font-weight: 400;\"> scripts to synthesize test data in an Excel spreadsheet, and the <\/span><a href=\"https:\/\/www.iri.com\/blog\/data-protection\/darkshield-files-rpc-api\/\"><span style=\"font-weight: 400;\">DarkShield-Files API<\/span><\/a><span style=\"font-weight: 400;\"> to mask PII from a <\/span><a href=\"https:\/\/github.com\/TeamIRI\/darkshield-api-demos\/tree\/master\/cosmosdb\"><span style=\"font-weight: 400;\">CosmosDB<\/span><\/a><span style=\"font-weight: 400;\"> NoSQL database. Note that I could just have easily invoked IRI FieldShield masking jobs for MS SQL from Azure DevOps too; see our first (GitLab) CI\/CD article <a href=\"https:\/\/www.iri.com\/blog\/test-data\/building-test-data-in-cicd-pipeline\/\">here<\/a>.<\/span><\/p>\n<h6><b>Demonstration Prerequisites<\/b><\/h6>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Your IRI software is installed and licensed on the target remote server (the remote server that the pipeline will access via SSH, which could be your on-premise computer or somewhere in the cloud).<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">An Azure account with a CosmosDB database.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\">If you are a new user and have a free tier account with Azure, you must have been approved for a <a href=\"https:\/\/docs.microsoft.com\/en-us\/azure\/devops\/release-notes\/2021\/sprint-184-update\">free grant<\/a> to use your pipeline.<\/li>\n<\/ul>\n<h6><b>About Azure Pipelines<\/b><\/h6>\n<p><span style=\"font-weight: 400;\">Azure Pipelines is a set of automated processes for building, testing, and deploying projects. The automation of these processes is meant to support the continuous integration, continuous delivery, and continuous testing methodologies of DevOps.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Continuous testing is a concept in DevOps that supports the testing of software at every stage of the software development life cycle.<\/span><span style=\"font-weight: 400;\"><br \/>\n<\/span><\/p>\n<p><a href=\"\/blog\/wp-content\/uploads\/2021\/11\/continuous-testing-graphic.png\"><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-15178 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/11\/continuous-testing-graphic.png\" alt=\"\" width=\"715\" height=\"349\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/continuous-testing-graphic.png 1000w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/continuous-testing-graphic-300x146.png 300w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/continuous-testing-graphic-768x375.png 768w\" sizes=\"(max-width: 715px) 100vw, 715px\" \/><\/a><\/p>\n<h6><b>Test Data Management and DevOps<\/b><\/h6>\n<p><span style=\"font-weight: 400;\">What is Test Data Management (TDM). TDM is the process by which production-like test data is generated and made available to various tests so that the testing process has access to realistic data. An effective <\/span><a href=\"https:\/\/www.iri.com\/blog\/test-data\/tdm-primer\/\"><span style=\"font-weight: 400;\">TDM strategy<\/span><\/a><span style=\"font-weight: 400;\"> is beneficial for several reasons.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">For one, unrealistic test data does not lead to more confidence in an application, and may lead to unfruitful testing. As such, a production-like test environment with production-like test data is necessary for accurately assessing how an application will behave during production.\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Second, TDM is meant to guarantee the availability of test data when it is needed during tests. Good test data does not provide any benefits if it is not available at the time of test execution.<\/span><\/p>\n<h6><b>Tests in DevOps and the Role of Test Data<\/b><\/h6>\n<p><span style=\"font-weight: 400;\">Test Data Management should be implemented to support various tests with realistic, good quality data throughout the DevOps process. Sauce Labs, an American cloud-hosted web and mobile application automated testing platform company, listed five functional tests that their most successful customers implement throughout the different phases of their DevOps pipelines.\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">These tests are:<\/span><span style=\"font-weight: 400;\"><br \/>\n<\/span><\/p>\n<ol>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Unit testing &#8211; Individual units of source code are tested<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Integration testing &#8211; Ensures that build continues to be stable as new code is added<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Automated end-to-end and regression testing &#8211; Ensures application works in an environment like the production environment<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Production testing &#8211; Testing after application release to identify issues early<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Exploratory and live testing &#8211; Automated testing is replaced by manual or interactive testing that strives to produce unexpected errors.<\/span><\/li>\n<\/ol>\n<p><span style=\"font-weight: 400;\">And according to testingexperts.com, there are even a few more types:<\/span><\/p>\n<p><a href=\"\/blog\/wp-content\/uploads\/2021\/11\/functional-testing-graphic.png\"><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-15180 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/11\/functional-testing-graphic.png\" alt=\"\" width=\"699\" height=\"430\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/functional-testing-graphic.png 849w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/functional-testing-graphic-300x184.png 300w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/functional-testing-graphic-768x472.png 768w\" sizes=\"(max-width: 699px) 100vw, 699px\" \/><\/a><\/p>\n<p><span style=\"font-weight: 400;\">Automated end-to-end\/regression testing, production testing, and exploratory\/live testing all benefit from effective Test Data Management (TDM):<\/span><\/p>\n<p style=\"padding-left: 30px;\"><span style=\"font-weight: 400;\">End-to-end and regression testing can cover a very large range of test cases. As such, creating consistent test data that looks and acts like production data gives confidence that new changes have not introduced new problems for old code.\u00a0<\/span><\/p>\n<p style=\"padding-left: 30px;\"><span style=\"font-weight: 400;\">Production testing ensures that an application will work, and perform, as expected when moved to a production environment. Performance testing is an umbrella term for load testing and stress testing. Both require a significant amount of test data, so automating rapid, high volume synthesis via RowGen in Voracity for example, can greatly reduce time-to-availability here.<\/span><\/p>\n<p style=\"padding-left: 30px;\"><span style=\"font-weight: 400;\">In exploratory and live testing, or \u201cblack box\u201d testing, QA will look for unexpected behavior and bugs. High quality, realistic test data is the best way to test application behavior of course.<\/span><\/p>\n<h6><b>IRI Voracity Software and Test Data Management<\/b><\/h6>\n<p><span style=\"font-weight: 400;\">Having confidence that an application will work, but being shocked when an unthought of bug occurs in production is never pleasant. Maybe data for a field was a little longer than expected.\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">That said, test environments usually lack the same level of security as a production environment. As a result, production data cannot be used as-is for testing.\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">A decision must therefore be made between using a copy of sanitized (masked) production data, or synthesizing test data that is as <\/span><a href=\"https:\/\/www.iri.com\/blog\/data-protection\/making-realistic-test-data-production\/\"><span style=\"font-weight: 400;\">realistic as possible<\/span><\/a><span style=\"font-weight: 400;\">. Fortunately, IRI has several fit-for-purpose software solutions to offer here. <\/span><\/p>\n<p><a href=\"\/blog\/wp-content\/uploads\/2021\/11\/iri-test-data-blurb.png\"><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-15179 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/11\/iri-test-data-blurb.png\" alt=\"\" width=\"651\" height=\"243\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/iri-test-data-blurb.png 871w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/iri-test-data-blurb-300x112.png 300w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/iri-test-data-blurb-768x287.png 768w\" sizes=\"(max-width: 651px) 100vw, 651px\" \/><\/a><\/p>\n<p><span style=\"font-weight: 400;\">First, for synthetic data generation there is <\/span><a href=\"https:\/\/www.iri.com\/products\/rowgen\"><span style=\"font-weight: 400;\">RowGen<\/span><\/a><span style=\"font-weight: 400;\">. This product creates and loads brand new synthetic, but realistic, test data targets in flat file formats and relational database (RDB) tables.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Second, for finding and masking PII and other sensitive data in flat files and RDBs, there is the IRI<\/span><a href=\"https:\/\/www.iri.com\/products\/fieldshield\"><span style=\"font-weight: 400;\"> FieldShield<\/span><\/a><span style=\"font-weight: 400;\"> product. For semi- and unstructured data formats, see the <\/span><a href=\"https:\/\/www.iri.com\/products\/darkshield\"><span style=\"font-weight: 400;\">DarkShield<\/span><\/a><span style=\"font-weight: 400;\"> product.<\/span><\/p>\n<p><a href=\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2020\/09\/fieldshield-graphic.png\"><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-14033 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2020\/09\/fieldshield-graphic-1024x1024.png\" alt=\"\" width=\"451\" height=\"451\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2020\/09\/fieldshield-graphic-1024x1024.png 1024w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2020\/09\/fieldshield-graphic-150x150.png 150w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2020\/09\/fieldshield-graphic-300x300.png 300w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2020\/09\/fieldshield-graphic-768x768.png 768w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2020\/09\/fieldshield-graphic.png 1200w\" sizes=\"(max-width: 451px) 100vw, 451px\" \/><\/a><\/p>\n<p><span style=\"font-weight: 400;\">For this demonstration I will use the Azure DevOps pipeline to run RowGen scripts that will populate an Excel sheet with test data. In addition to RowGen, I will also use the DarkShield-API to read a container in CosmosDB and create a second container that contains the copied content after it has been sanitized of any PII.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">As demonstrated in my previous articles on DevOps pipelines, any IRI job script that can run on the command line can be run from the pipeline via SSH. This article shows you how to set up SSH tasks and run IRI job scripts on remote machines in the Azure Pipeline.<\/span><span style=\"font-weight: 400;\"><br \/>\n<\/span><\/p>\n<p><b>The Structure and Setup of an Azure DevOps Pipeline<\/b><\/p>\n<p><span style=\"font-weight: 400;\">The structure of the Azure pipeline is divided into one or more stages. Each of these stages consists of one or more jobs that perform work. The jobs in each stage contain a step or steps that will have tasks, scripts, or references to external templates.<\/span><\/p>\n<p><a href=\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-pipeline-graphic.png\"><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-15181 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-pipeline-graphic-1024x455.png\" alt=\"\" width=\"650\" height=\"289\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-pipeline-graphic-1024x455.png 1024w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-pipeline-graphic-300x133.png 300w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-pipeline-graphic-768x341.png 768w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-pipeline-graphic.png 1600w\" sizes=\"(max-width: 650px) 100vw, 650px\" \/><\/a><\/p>\n<p><span style=\"font-weight: 400;\">To set up a pipeline, you must first create a project and specify it as a public or private project. <\/span><span style=\"font-weight: 400;\">Azure will not allow you to execute any pipeline unless you are a paying customer or unless they grant you approval in their free tier to run your pipeline for free.\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Azure staff currently requires up to three business days to evaluate a request and they call it &#8220;granting the right to use parallelism&#8221;. Parallelism is the ability to run two or more jobs simultaneously.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">This seems somewhat misleading at the time of this writing however, because it doesn&#8217;t actually matter whether you try to run a single job or two jobs at the same time in your pipeline. You still cannot use the pipeline you made without first being granted approval for parallelism.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">If you are a paying customer there is no three-day wait. <\/span><span style=\"font-weight: 400;\">At the discretion of the Azure support staff, you may also be required to run your pipeline as a private project for it to be approved.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">After creating a project, you can then create a pipeline. When creating a pipeline you will be prompted to fill out configuration details, including the repository for your source code to reside.<\/span><\/p>\n<p style=\"text-align: center;\"><a href=\"\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-start.png\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-15182 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-start.png\" alt=\"\" width=\"490\" height=\"337\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-start.png 490w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-start-300x206.png 300w\" sizes=\"(max-width: 490px) 100vw, 490px\" \/><\/a><i><span style=\"font-weight: 400;\">Azure DevOps prompt for you to create a Pipeline<\/span><\/i><i><span style=\"font-weight: 400;\"><br \/>\n<\/span><\/i><\/p>\n<p><span style=\"font-weight: 400;\">Below are the steps to creating your first pipeline.<\/span><\/p>\n<p style=\"text-align: center;\"><a href=\"\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-createpipe1.png\"><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-15183 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-createpipe1.png\" alt=\"\" width=\"601\" height=\"505\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-createpipe1.png 639w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-createpipe1-300x252.png 300w\" sizes=\"(max-width: 601px) 100vw, 601px\" \/><\/a><i><span style=\"font-weight: 400;\">Select the version control system you are using that will be the location of your application source code and that the Azure DevOps pipeline will draw from during the Build stage.<\/span><\/i><\/p>\n<p style=\"text-align: center;\"><a href=\"\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-createpipe2.png\"><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-15185 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-createpipe2.png\" alt=\"\" width=\"651\" height=\"384\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-createpipe2.png 676w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-createpipe2-300x177.png 300w\" sizes=\"(max-width: 651px) 100vw, 651px\" \/><\/a><i><span style=\"font-weight: 400;\">Select the repository your pipeline will use<\/span><\/i><\/p>\n<p style=\"text-align: center;\"><a href=\"\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-createpipe3.png\"><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-15186 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-createpipe3.png\" alt=\"\" width=\"637\" height=\"341\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-createpipe3.png 630w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-createpipe3-300x160.png 300w\" sizes=\"(max-width: 637px) 100vw, 637px\" \/><\/a><i><span style=\"font-weight: 400;\">Choose a starter pipeline, or a pre-existing pipeline<\/span><\/i><\/p>\n<p style=\"text-align: center;\"><a href=\"\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-createpipe4.png\"><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-15187 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-createpipe4.png\" alt=\"\" width=\"649\" height=\"494\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-createpipe4.png 744w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-createpipe4-300x228.png 300w\" sizes=\"(max-width: 649px) 100vw, 649px\" \/><\/a><i><span style=\"font-weight: 400;\">Review the created pipeline and then Save and Run.<\/span><\/i><\/p>\n<h6><b>Setting Up a Service Connection<\/b><\/h6>\n<p><span style=\"font-weight: 400;\">To generate test data via the pipeline, an SSH task will be added that will perform the process of connecting to a remote machine that contains a licensed copy of <\/span><a href=\"https:\/\/www.iri.com\/products\/voracity\"><span style=\"font-weight: 400;\">IRI Voracity<\/span><\/a><span style=\"font-weight: 400;\"> or a component product like FieldShield, DarkShield or RowGen. But before we continue any further we need to set up an SSH service connection. To do that:<\/span><\/p>\n<ol>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Open Project Setting at bottom left corner of the screen<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Select <\/span><i><span style=\"font-weight: 400;\">Service connections<\/span><\/i><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Click <\/span><i><span style=\"font-weight: 400;\">New Service Connection<\/span><\/i><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Select SSH and click Next<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Fill out the service connection form and save.<\/span><\/li>\n<\/ol>\n<p style=\"text-align: center;\"><a href=\"\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-prjsetservcon.png\"><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-15189 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-prjsetservcon.png\" alt=\"\" width=\"649\" height=\"295\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-prjsetservcon.png 928w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-prjsetservcon-300x136.png 300w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-prjsetservcon-768x349.png 768w\" sizes=\"(max-width: 649px) 100vw, 649px\" \/><\/a><i><span style=\"font-weight: 400;\">Navigating to Service Connections from Project Settings<\/span><\/i><i><span style=\"font-weight: 400;\"><br \/>\n<\/span><\/i><\/p>\n<p style=\"text-align: center;\"><a href=\"\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-newservconn.png\"><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-15190 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-newservconn.png\" alt=\"\" width=\"648\" height=\"239\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-newservconn.png 1011w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-newservconn-300x111.png 300w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-newservconn-768x283.png 768w\" sizes=\"(max-width: 648px) 100vw, 648px\" \/><\/a><i><span style=\"font-weight: 400;\">Creating a New Service Connection<\/span><\/i><\/p>\n<p style=\"text-align: center;\"><a href=\"\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-sshservice.png\"><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-15191 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-sshservice.png\" alt=\"\" width=\"399\" height=\"803\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-sshservice.png 467w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-sshservice-149x300.png 149w\" sizes=\"(max-width: 399px) 100vw, 399px\" \/><\/a><i><span style=\"font-weight: 400;\">Selecting the SSH Service Connection<\/span><\/i><\/p>\n<p style=\"text-align: center;\"><a href=\"\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-sshcon2.png\"><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-15192 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-sshcon2.png\" alt=\"\" width=\"400\" height=\"499\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-sshcon2.png 477w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-sshcon2-241x300.png 241w\" sizes=\"(max-width: 400px) 100vw, 400px\" \/><\/a><i><span style=\"font-weight: 400;\">Configuring the SSH Service Connection<\/span><\/i><\/p>\n<p>&nbsp;<\/p>\n<p style=\"text-align: center;\"><a href=\"\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-sscon3.png\"><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-15193 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-sscon3.png\" alt=\"\" width=\"650\" height=\"135\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-sscon3.png 982w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-sscon3-300x62.png 300w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-sscon3-768x160.png 768w\" sizes=\"(max-width: 650px) 100vw, 650px\" \/><\/a><i><span style=\"font-weight: 400;\">New SSH Service Connection is displayed.<\/span><\/i><\/p>\n<p><span style=\"font-weight: 400;\">If you followed all of the steps, you have a service connection set up for when you configure the SSH task.<\/span><\/p>\n<h6><b>What Are Release Pipelines and How Are They Set Up?<\/b><\/h6>\n<p><span style=\"font-weight: 400;\">Understanding what a release pipeline is can be a bit difficult for those who first learned with multi-stage pipelines like in the case of GitLab CI\/CD pipeline. The multi-stage pipeline will have a build stage and a deploy stage.\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Azure DevOps instead has more than one type of pipeline. There are build pipelines and release pipelines. The build pipeline will build and create your project artifact.\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">The release pipeline gets triggered by the build artifact and deploys the artifact to one or several stages. <\/span><span style=\"font-weight: 400;\">As Dhimate describes it, \u201cA release pipeline is a conceptual process by which we take committed code into production\u201d<\/span><span style=\"font-weight: 400;\"><span id='easy-footnote-1-15169' class='easy-footnote-margin-adjust'><\/span><span class='easy-footnote'><a href='https:\/\/beta.iri.com\/blog\/test-data\/test-data-azure-devops\/#easy-footnote-bottom-1-15169' title='Dhimate, Gaurav. \u201cRelease Pipeline Using Azure Devops &amp;#8211; Dzone Devops.\u201d &lt;i&gt;Dzone.com&lt;\/i&gt;, DZone, 17 Feb. 2021, &lt;a href=&quot;https:\/\/dzone.com\/articles\/release-pipeline-using-azure-devops&quot;&gt;https:\/\/dzone.com\/articles\/release-pipeline-using-azure-devops&lt;\/a&gt;.'><sup>1<\/sup><\/a><\/span><\/span><span style=\"font-weight: 400;\">.\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">In this demonstration a release pipeline will contain the SSH task that will perform the process of connecting to a remote server.<\/span><\/p>\n<p style=\"text-align: center;\"><a href=\"\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-settingReleases.png\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-15195 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-settingReleases.png\" alt=\"\" width=\"260\" height=\"611\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-settingReleases.png 260w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-settingReleases-128x300.png 128w\" sizes=\"(max-width: 260px) 100vw, 260px\" \/><\/a><i><span style=\"font-weight: 400;\">In the left panel, under the Pipelines section select Releases.<\/span><\/i><\/p>\n<p style=\"text-align: center;\"><a href=\"\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-releasepipe.png\"><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-15196 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-releasepipe.png\" alt=\"\" width=\"651\" height=\"366\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-releasepipe.png 1010w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-releasepipe-300x169.png 300w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-releasepipe-768x432.png 768w\" sizes=\"(max-width: 651px) 100vw, 651px\" \/><\/a><i><span style=\"font-weight: 400;\">Select New pipeline to create a new release pipeline.<\/span><\/i><\/p>\n<p style=\"text-align: center;\"><a href=\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-emptyjob.png\"><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-15197 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-emptyjob-1024x552.png\" alt=\"\" width=\"651\" height=\"351\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-emptyjob-1024x552.png 1024w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-emptyjob-300x162.png 300w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-emptyjob-768x414.png 768w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-emptyjob.png 1196w\" sizes=\"(max-width: 651px) 100vw, 651px\" \/><\/a><i>Select Empty Job for the template of the release pipeline.<\/i><\/p>\n<p style=\"text-align: center;\"><a href=\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-saveemptyjob.png\"><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-15198 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-saveemptyjob-1024x363.png\" alt=\"\" width=\"649\" height=\"230\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-saveemptyjob-1024x363.png 1024w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-saveemptyjob-300x106.png 300w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-saveemptyjob-768x272.png 768w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-saveemptyjob.png 1177w\" sizes=\"(max-width: 649px) 100vw, 649px\" \/><\/a><i>Click Save to finish creating a release pipeline.<\/i><\/p>\n<p><span style=\"font-weight: 400;\">After a new release is created, we can then start adding tasks.<\/span><\/p>\n<h6><b>Creating an SSH Task<\/b><\/h6>\n<p><span style=\"font-weight: 400;\">After connecting, a command will be executed to run an IRI RowGen job script to synthesize safe, realistic test data sent to an Excel sheet. Afterward, a second command will be executed to make a <\/span><a href=\"https:\/\/github.com\/TeamIRI\/darkshield-api-demos\/tree\/master\/cosmosdb\"><span style=\"font-weight: 400;\">DarkShield-Files API call to CosmosDB<\/span><\/a><span style=\"font-weight: 400;\"> where test data will be generated from masking production data.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">To add the SSH task, follow the steps below:<\/span><\/p>\n<p style=\"text-align: center;\"><a href=\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-navtotask.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-15199 \" src=\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-navtotask-e1636661485973-1024x339.png\" alt=\"\" width=\"649\" height=\"215\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-navtotask-e1636661485973-1024x339.png 1024w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-navtotask-e1636661485973-300x99.png 300w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-navtotask-e1636661485973-768x254.png 768w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-navtotask-e1636661485973.png 1210w\" sizes=\"(max-width: 649px) 100vw, 649px\" \/><\/a><i><span style=\"font-weight: 400;\">Click the blue link \u201c1 job, 0 task\u201d to edit the Agent Job and add tasks.<\/span><\/i><\/p>\n<p style=\"text-align: center;\"><a href=\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-addTask.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-15200 \" src=\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-addTask-e1636661512107-1024x218.png\" alt=\"\" width=\"648\" height=\"138\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-addTask-e1636661512107-1024x218.png 1024w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-addTask-e1636661512107-300x64.png 300w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-addTask-e1636661512107-768x164.png 768w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-addTask-e1636661512107.png 1196w\" sizes=\"(max-width: 648px) 100vw, 648px\" \/><\/a><i><span style=\"font-weight: 400;\">Click the plus sign in the Agent job panel to add a task.<\/span><\/i><\/p>\n<p style=\"text-align: center;\"><a href=\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-sshtask.png\"><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-15201 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-sshtask-1024x373.png\" alt=\"\" width=\"651\" height=\"237\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-sshtask-1024x373.png 1024w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-sshtask-300x109.png 300w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-sshtask-768x279.png 768w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-sshtask.png 1190w\" sizes=\"(max-width: 651px) 100vw, 651px\" \/><\/a><i><span style=\"font-weight: 400;\">In the Add tasks panel select SSH task template. To easily find SSH can use the search bar in the upper right corner.<\/span><\/i><\/p>\n<p><span style=\"font-weight: 400;\">After clicking on the SSH task template in the <\/span><i><span style=\"font-weight: 400;\">Add tasks<\/span><\/i><span style=\"font-weight: 400;\"> panel, you will have an empty unconfigured SSH task under your Agent job.<\/span><\/p>\n<p><b>Configuring the SSH Task<\/b><\/p>\n<p><span style=\"font-weight: 400;\">As stated previously, after adding the SSH task to the Agent job, it will be visible and listed in a panel under the Agent job panel below.<\/span><\/p>\n<p style=\"text-align: center;\"><a href=\"\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-sshtaskconf.png\"><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-15202 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-sshtaskconf.png\" alt=\"\" width=\"649\" height=\"418\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-sshtaskconf.png 979w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-sshtaskconf-300x193.png 300w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-sshtaskconf-768x495.png 768w\" sizes=\"(max-width: 649px) 100vw, 649px\" \/><\/a><i><span style=\"font-weight: 400;\">Enter the commands to be executed during SSH connection.<\/span><\/i><\/p>\n<p><span style=\"font-weight: 400;\">If you click on the SSH task, it will display a check and open a configuration panel on the right like in the image above. Specify the SSH service connection that was created at the beginning of the walkthrough, and the commands you wish to execute on the remote machine.<\/span><\/p>\n<p><b>Results of The Pipeline<\/b><\/p>\n<p><span style=\"font-weight: 400;\">After the SSH task is set up the release pipeline can be deployed. And, if everything was configured correctly the release pipeline will have finished successfully after establishing the SSH connection to the remote machine and executing the SSH commands.\u00a0<\/span><\/p>\n<p style=\"text-align: center;\"><a href=\"\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-azureRelease.png\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-15203 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-azureRelease.png\" alt=\"\" width=\"645\" height=\"372\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-azureRelease.png 645w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-azureRelease-300x173.png 300w\" sizes=\"(max-width: 645px) 100vw, 645px\" \/><\/a><i><span style=\"font-weight: 400;\">Successful pipeline execution.<\/span><\/i><\/p>\n<p><span style=\"font-weight: 400;\">The stage that contains our SSH task will have a log output that can be viewed:<\/span><\/p>\n<p style=\"text-align: center;\"><a href=\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-sshlog.png\"><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-15204 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-sshlog-1024x251.png\" alt=\"\" width=\"747\" height=\"183\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-sshlog-1024x251.png 1024w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-sshlog-300x73.png 300w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-sshlog-768x188.png 768w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-sshlog.png 1054w\" sizes=\"(max-width: 747px) 100vw, 747px\" \/><\/a><i><span style=\"font-weight: 400;\">Agent Job output log for SSH task shows successful SSH connection and execution.<\/span><\/i><\/p>\n<p><b>Results of SSH Execution<\/b><\/p>\n<p><span style=\"font-weight: 400;\">As a result of the successful execution of SSH commands, the remote target machine will run a batch file containing RowGen jobs. Below is one such RowGen task script, built automatically in the <\/span><a href=\"https:\/\/www.iri.com\/products\/workbench\/rowgen-gui\"><span style=\"font-weight: 400;\">IRI Workbench GUI for RowGen<\/span><\/a><span style=\"font-weight: 400;\">:<\/span><\/p>\n<p style=\"text-align: center;\"><a href=\"\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-rowgenscript.png\"><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-15205 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-rowgenscript.png\" alt=\"\" width=\"650\" height=\"222\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-rowgenscript.png 937w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-rowgenscript-300x102.png 300w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-rowgenscript-768x262.png 768w\" sizes=\"(max-width: 650px) 100vw, 650px\" \/><\/a><i><span style=\"font-weight: 400;\">IRI RowGen Job Script<\/span><\/i><\/p>\n<p><span style=\"font-weight: 400;\">For my test data, I used set files to randomly select and populate test data into my spreadsheet. Set files are ASCII value lists (containing like objects), with each value on a new line in the file.\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">After the job scripts are executed, the final result is an Excel spreadsheet populated with randomly generated test data. Though that target was an XLS file, it could have instead (or also) been other files or tables in an RDB like MS SQL, MySQL, PostgreSQL, DB2, or Oracle.<\/span><\/p>\n<p style=\"text-align: center;\"><a href=\"\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-excell.png\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-15206 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-excell.png\" alt=\"\" width=\"620\" height=\"364\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-excell.png 620w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-excell-300x176.png 300w\" sizes=\"(max-width: 620px) 100vw, 620px\" \/><\/a><i><span style=\"font-weight: 400;\">Excel Sheet with RowGen-synthesized Test Data<\/span><\/i><\/p>\n<p><span style=\"font-weight: 400;\">After generating test data using RowGen, a second command is run to make an API call to the DarkShield-Files API. The calling program will read from a production container in CosmosDB, use the search and mask context of the API to sanitize the sensitive data, and write the masked results into a second container used for testing.\u00a0<\/span><\/p>\n<p style=\"text-align: center;\"><a href=\"\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-cosmosdb.png\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-15207 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-cosmosdb.png\" alt=\"\" width=\"591\" height=\"403\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-cosmosdb.png 591w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-cosmosdb-300x205.png 300w\" sizes=\"(max-width: 591px) 100vw, 591px\" \/><\/a><i><span style=\"font-weight: 400;\">CosmosDB Database<\/span><\/i><\/p>\n<p style=\"text-align: center;\"><a href=\"\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-cosmositem1.png\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-15209 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-cosmositem1.png\" alt=\"\" width=\"837\" height=\"265\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-cosmositem1.png 837w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-cosmositem1-300x95.png 300w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-cosmositem1-768x243.png 768w\" sizes=\"(max-width: 837px) 100vw, 837px\" \/><\/a><i><span style=\"font-weight: 400;\">Sample of an unprotected PII in an item<\/span><\/i><\/p>\n<p style=\"text-align: center;\"><a href=\"\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-cosmosmask1.png\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-15208 aligncenter\" src=\"\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-cosmosmask1.png\" alt=\"\" width=\"841\" height=\"248\" srcset=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-cosmosmask1.png 841w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-cosmosmask1-300x88.png 300w, https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-test-data-cosmosmask1-768x226.png 768w\" sizes=\"(max-width: 841px) 100vw, 841px\" \/><\/a><i><span style=\"font-weight: 400;\">Sample of the same item with the PII masked by DarkShield<\/span><\/i><\/p>\n<p><b>In Closing<\/b><\/p>\n<p><span style=\"font-weight: 400;\">This article demonstrated how to automate the execution of IRI test-data-producing jobs from within an Azure DevOps Pipeline. Using SSH, any IRI job script or API routine that can run from the command line can be executed from within the pipeline.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">This is because the IRI back-end engine is an executable, so products like <\/span><a href=\"https:\/\/www.iri.com\/products\/darkshield\"><span style=\"font-weight: 400;\">DarkShield<\/span><\/a><span style=\"font-weight: 400;\">, <\/span><a href=\"https:\/\/www.iri.com\/products\/fieldshield\"><span style=\"font-weight: 400;\">FieldShield<\/span><\/a><span style=\"font-weight: 400;\">, and <\/span><a href=\"https:\/\/www.iri.com\/products\/rowgen\"><span style=\"font-weight: 400;\">RowGen<\/span><\/a><span style=\"font-weight: 400;\"> can run from the command line. The DarkShield-Files API can be programmatically assigned to execute on preselected data sources as our <\/span><a href=\"https:\/\/github.com\/TeamIRI\/darkshield-api-demos\"><span style=\"font-weight: 400;\">Python demos<\/span><\/a><span style=\"font-weight: 400;\"> show.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Recently IRI has also created and published on Docker Hub, a docker <\/span><a href=\"https:\/\/hub.docker.com\/r\/devonak\/plankton\"><span style=\"font-weight: 400;\">image<\/span><\/a><span style=\"font-weight: 400;\"> for the DarkShield API. Because the DevOps pipeline uses containers to set up environments, it is possible to utilize this container to run the DarkShield-Files API from within the pipeline.\u00a0<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>This is the third addition to our 4-article DevOps pipeline series. Here are the links to related DevOps articles for Jenkins, GitLab and AWS CodePipeline. What is Being Demonstrated? In this article I demonstrate how to use IRI Voracity TDM platform software from within an Azure DevOps pipeline to produce and use realistic test data<\/p>\n<div><a class=\"btn-filled btn\" href=\"https:\/\/beta.iri.com\/blog\/test-data\/test-data-azure-devops\/\" title=\"Generating Test Data for Azure DevOps\">Read More<\/a><\/div>\n","protected":false},"author":152,"featured_media":15176,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_exactmetrics_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"footnotes":""},"categories":[8,29],"tags":[1439,1558,1557,1556,14,1508,1388,526,789,1306,238,88,379],"class_list":["post-15169","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-data-protection","category-test-data","tag-azure","tag-azure-pipelines","tag-continuous-testing","tag-cosmosdb","tag-data-masking","tag-devops","tag-iri-darkshield","tag-iri-rowgen","tag-iri-voracity","tag-pii-masking","tag-real-test-data","tag-test-data-2","tag-test-data-management"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v23.3 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Generating Test Data for Azure DevOps - IRI<\/title>\n<meta name=\"description\" content=\"In this article I demonstrate how to use IRI Voracity TDM platform software from within an Azure DevOps pipeline to produce and use realistic test data in\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/beta.iri.com\/blog\/test-data\/test-data-azure-devops\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Generating Test Data for Azure DevOps - IRI\" \/>\n<meta property=\"og:description\" content=\"In this article I demonstrate how to use IRI Voracity TDM platform software from within an Azure DevOps pipeline to produce and use realistic test data in\" \/>\n<meta property=\"og:url\" content=\"https:\/\/beta.iri.com\/blog\/test-data\/test-data-azure-devops\/\" \/>\n<meta property=\"og:site_name\" content=\"IRI\" \/>\n<meta property=\"article:published_time\" content=\"2021-11-10T21:30:26+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2022-05-04T16:24:39+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-thumbnail.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1110\" \/>\n\t<meta property=\"og:image:height\" content=\"624\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Adam Lewis\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Adam Lewis\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"11 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/beta.iri.com\/blog\/test-data\/test-data-azure-devops\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/test-data\/test-data-azure-devops\/\"},\"author\":{\"name\":\"Adam Lewis\",\"@id\":\"https:\/\/beta.iri.com\/blog\/#\/schema\/person\/37c0e5beab094bd61cc521902df2876e\"},\"headline\":\"Generating Test Data for Azure DevOps\",\"datePublished\":\"2021-11-10T21:30:26+00:00\",\"dateModified\":\"2022-05-04T16:24:39+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/test-data\/test-data-azure-devops\/\"},\"wordCount\":2259,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/#organization\"},\"image\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/test-data\/test-data-azure-devops\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-thumbnail.jpg\",\"keywords\":[\"Azure\",\"Azure Pipelines\",\"continuous testing\",\"CosmosDB\",\"data masking\",\"DevOps\",\"IRI DarkShield\",\"IRI RowGen\",\"IRI Voracity\",\"pii masking\",\"real test data\",\"test data\",\"test data management\"],\"articleSection\":[\"Data Masking\/Protection\",\"Test Data\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/beta.iri.com\/blog\/test-data\/test-data-azure-devops\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/beta.iri.com\/blog\/test-data\/test-data-azure-devops\/\",\"url\":\"https:\/\/beta.iri.com\/blog\/test-data\/test-data-azure-devops\/\",\"name\":\"Generating Test Data for Azure DevOps - IRI\",\"isPartOf\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/test-data\/test-data-azure-devops\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/test-data\/test-data-azure-devops\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-thumbnail.jpg\",\"datePublished\":\"2021-11-10T21:30:26+00:00\",\"dateModified\":\"2022-05-04T16:24:39+00:00\",\"description\":\"In this article I demonstrate how to use IRI Voracity TDM platform software from within an Azure DevOps pipeline to produce and use realistic test data in\",\"breadcrumb\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/test-data\/test-data-azure-devops\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/beta.iri.com\/blog\/test-data\/test-data-azure-devops\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/beta.iri.com\/blog\/test-data\/test-data-azure-devops\/#primaryimage\",\"url\":\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-thumbnail.jpg\",\"contentUrl\":\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-thumbnail.jpg\",\"width\":1110,\"height\":624},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/beta.iri.com\/blog\/test-data\/test-data-azure-devops\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/beta.iri.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Generating Test Data for Azure DevOps\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/beta.iri.com\/blog\/#website\",\"url\":\"https:\/\/beta.iri.com\/blog\/\",\"name\":\"IRI\",\"description\":\"Total Data Management Blog\",\"publisher\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/beta.iri.com\/blog\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/beta.iri.com\/blog\/#organization\",\"name\":\"IRI\",\"url\":\"https:\/\/beta.iri.com\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/beta.iri.com\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2019\/02\/iri-logo-total-data-management-small-1.png\",\"contentUrl\":\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2019\/02\/iri-logo-total-data-management-small-1.png\",\"width\":750,\"height\":206,\"caption\":\"IRI\"},\"image\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/beta.iri.com\/blog\/#\/schema\/person\/37c0e5beab094bd61cc521902df2876e\",\"name\":\"Adam Lewis\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/beta.iri.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/087667d0c75d33bb6fab6e734bd89333?s=96&d=blank&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/087667d0c75d33bb6fab6e734bd89333?s=96&d=blank&r=g\",\"caption\":\"Adam Lewis\"},\"url\":\"https:\/\/beta.iri.com\/blog\/author\/adaml\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Generating Test Data for Azure DevOps - IRI","description":"In this article I demonstrate how to use IRI Voracity TDM platform software from within an Azure DevOps pipeline to produce and use realistic test data in","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/beta.iri.com\/blog\/test-data\/test-data-azure-devops\/","og_locale":"en_US","og_type":"article","og_title":"Generating Test Data for Azure DevOps - IRI","og_description":"In this article I demonstrate how to use IRI Voracity TDM platform software from within an Azure DevOps pipeline to produce and use realistic test data in","og_url":"https:\/\/beta.iri.com\/blog\/test-data\/test-data-azure-devops\/","og_site_name":"IRI","article_published_time":"2021-11-10T21:30:26+00:00","article_modified_time":"2022-05-04T16:24:39+00:00","og_image":[{"width":1110,"height":624,"url":"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-thumbnail.jpg","type":"image\/jpeg"}],"author":"Adam Lewis","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Adam Lewis","Est. reading time":"11 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/beta.iri.com\/blog\/test-data\/test-data-azure-devops\/#article","isPartOf":{"@id":"https:\/\/beta.iri.com\/blog\/test-data\/test-data-azure-devops\/"},"author":{"name":"Adam Lewis","@id":"https:\/\/beta.iri.com\/blog\/#\/schema\/person\/37c0e5beab094bd61cc521902df2876e"},"headline":"Generating Test Data for Azure DevOps","datePublished":"2021-11-10T21:30:26+00:00","dateModified":"2022-05-04T16:24:39+00:00","mainEntityOfPage":{"@id":"https:\/\/beta.iri.com\/blog\/test-data\/test-data-azure-devops\/"},"wordCount":2259,"commentCount":0,"publisher":{"@id":"https:\/\/beta.iri.com\/blog\/#organization"},"image":{"@id":"https:\/\/beta.iri.com\/blog\/test-data\/test-data-azure-devops\/#primaryimage"},"thumbnailUrl":"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-thumbnail.jpg","keywords":["Azure","Azure Pipelines","continuous testing","CosmosDB","data masking","DevOps","IRI DarkShield","IRI RowGen","IRI Voracity","pii masking","real test data","test data","test data management"],"articleSection":["Data Masking\/Protection","Test Data"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/beta.iri.com\/blog\/test-data\/test-data-azure-devops\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/beta.iri.com\/blog\/test-data\/test-data-azure-devops\/","url":"https:\/\/beta.iri.com\/blog\/test-data\/test-data-azure-devops\/","name":"Generating Test Data for Azure DevOps - IRI","isPartOf":{"@id":"https:\/\/beta.iri.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/beta.iri.com\/blog\/test-data\/test-data-azure-devops\/#primaryimage"},"image":{"@id":"https:\/\/beta.iri.com\/blog\/test-data\/test-data-azure-devops\/#primaryimage"},"thumbnailUrl":"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-thumbnail.jpg","datePublished":"2021-11-10T21:30:26+00:00","dateModified":"2022-05-04T16:24:39+00:00","description":"In this article I demonstrate how to use IRI Voracity TDM platform software from within an Azure DevOps pipeline to produce and use realistic test data in","breadcrumb":{"@id":"https:\/\/beta.iri.com\/blog\/test-data\/test-data-azure-devops\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/beta.iri.com\/blog\/test-data\/test-data-azure-devops\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/beta.iri.com\/blog\/test-data\/test-data-azure-devops\/#primaryimage","url":"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-thumbnail.jpg","contentUrl":"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-thumbnail.jpg","width":1110,"height":624},{"@type":"BreadcrumbList","@id":"https:\/\/beta.iri.com\/blog\/test-data\/test-data-azure-devops\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/beta.iri.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Generating Test Data for Azure DevOps"}]},{"@type":"WebSite","@id":"https:\/\/beta.iri.com\/blog\/#website","url":"https:\/\/beta.iri.com\/blog\/","name":"IRI","description":"Total Data Management Blog","publisher":{"@id":"https:\/\/beta.iri.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/beta.iri.com\/blog\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/beta.iri.com\/blog\/#organization","name":"IRI","url":"https:\/\/beta.iri.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/beta.iri.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2019\/02\/iri-logo-total-data-management-small-1.png","contentUrl":"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2019\/02\/iri-logo-total-data-management-small-1.png","width":750,"height":206,"caption":"IRI"},"image":{"@id":"https:\/\/beta.iri.com\/blog\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/beta.iri.com\/blog\/#\/schema\/person\/37c0e5beab094bd61cc521902df2876e","name":"Adam Lewis","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/beta.iri.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/087667d0c75d33bb6fab6e734bd89333?s=96&d=blank&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/087667d0c75d33bb6fab6e734bd89333?s=96&d=blank&r=g","caption":"Adam Lewis"},"url":"https:\/\/beta.iri.com\/blog\/author\/adaml\/"}]}},"jetpack_featured_media_url":"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2021\/11\/azure-devops-thumbnail.jpg","_links":{"self":[{"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/posts\/15169"}],"collection":[{"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/users\/152"}],"replies":[{"embeddable":true,"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/comments?post=15169"}],"version-history":[{"count":17,"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/posts\/15169\/revisions"}],"predecessor-version":[{"id":15791,"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/posts\/15169\/revisions\/15791"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/media\/15176"}],"wp:attachment":[{"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/media?parent=15169"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/categories?post=15169"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/tags?post=15169"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}