{"id":9908,"date":"2016-06-20T13:56:17","date_gmt":"2016-06-20T17:56:17","guid":{"rendered":"http:\/\/www.iri.com\/blog\/?p=9908"},"modified":"2017-11-07T08:13:22","modified_gmt":"2017-11-07T13:13:22","slug":"building-loading-big-test-data-into-marklogic","status":"publish","type":"post","link":"https:\/\/beta.iri.com\/blog\/test-data\/building-loading-big-test-data-into-marklogic\/","title":{"rendered":"Building &#038; Loading &#8216;Big Test Data&#8217; into MarkLogic"},"content":{"rendered":"<p>Just as production data processing tools like <a style=\"color: #1155cc; text-decoration: underline;\" href=\"http:\/\/www.iri.com\/products\/cosort\">IRI CoSort<\/a> must handle big data in NoSQL DB environments, so too must a big test data generation tool like <a style=\"color: #1155cc; text-decoration: underline;\" href=\"http:\/\/www.iri.com\/products\/rowgen\">IRI RowGen<\/a>. This article explains how a RowGen product or <a style=\"color: #1155cc; text-decoration: underline;\" href=\"http:\/\/www.iri.com\/products\/voracity\">IRI Voracity<\/a> platform user would leverage the test data generation facilities in their common <a style=\"color: #1155cc; text-decoration: underline;\" href=\"http:\/\/www.iri.com\/products\/workbench\">IRI Workbench<\/a> GUI to generate &#8212; and the <a style=\"color: #1155cc; text-decoration: underline;\" href=\"https:\/\/developer.marklogic.com\/products\/mlcp\" target=\"_blank\" rel=\"noopener\">MarkLogic Content Pump<\/a> to load &#8212; test data into MarkLogic.<\/p>\n<p style=\"font-size: 12pt; font-weight: bold;\"><span style=\"font-size: 12pt; font-weight: bold;\"><span style=\"font-size: 12pt; font-weight: bold;\">RDB vs. NoSQL <\/span><\/span><\/p>\n<p>First, for RowGen and other users already familiar with relational database tables, it\u2019s helpful to understand how a NoSQL \u2018document data\u2019 model differs from the relational database model:<\/p>\n<h3><em><strong>\u00a0 \u00a0Relational vs. Document Data Model<\/strong><\/em><\/h3>\n<table  class=\" table table-hover\" style=\"border-color: #000000; border-width: 1px; height: 0px; width: 0px;\" border=\"1\" cellspacing=\"0\" cellpadding=\"0\">\n<tbody>\n<tr style=\"font-size: 10pt; min-height: 21px;\">\n<td style=\"font-size: 10pt; width: 160.5px; padding: 5px;\" valign=\"top\">\n<p style=\"font-size: 10pt; line-height: 1; font-weight: bold;\" align=\"center\"><span style=\"font-size: 10pt; font-weight: bold;\"><span style=\"font-size: 10pt; font-weight: bold;\">Relational Model<\/span><\/span><\/p>\n<\/td>\n<td style=\"font-size: 10pt; width: 119.25px; padding: 5px;\" valign=\"top\">\n<p style=\"font-size: 10pt; line-height: 1; font-weight: bold;\" align=\"center\"><span style=\"font-size: 10pt; font-weight: bold;\"><span style=\"font-size: 10pt; font-weight: bold;\">MarkLogic <\/span><\/span><\/p>\n<\/td>\n<\/tr>\n<tr style=\"font-size: 10pt;\">\n<td style=\"font-size: 10pt; width: 160.5px; padding: 5px;\" valign=\"top\">\n<p style=\"font-size: 10pt;\" align=\"center\"><span style=\"font-size: 10pt;\"><span style=\"font-size: 10pt;\">Normalized Tabular Model<\/span><\/span><\/p>\n<\/td>\n<td style=\"font-size: 10pt; width: 119.25px; padding: 5px;\" valign=\"top\">\n<p style=\"font-size: 10pt; line-height: 1;\" align=\"center\"><span style=\"font-size: 10pt;\"><span style=\"font-size: 10pt;\">Query &amp; Search<\/span><\/span><\/p>\n<\/td>\n<\/tr>\n<tr style=\"font-size: 10pt;\">\n<td style=\"font-size: 10pt; width: 160.5px; padding: 5px;\" valign=\"top\">\n<p style=\"font-size: 10pt; line-height: 1;\" align=\"center\"><span style=\"font-size: 10pt;\"><span style=\"font-size: 10pt;\">Application Independent Query<\/span><\/span><\/p>\n<\/td>\n<td style=\"font-size: 10pt; width: 119.25px; padding: 5px;\" valign=\"top\">\n<p style=\"font-size: 10pt; line-height: 1;\" align=\"center\"><span style=\"font-size: 10pt;\"><span style=\"font-size: 10pt;\">Massive Scale<\/span><\/span><\/p>\n<\/td>\n<\/tr>\n<tr style=\"font-size: 10pt;\">\n<td style=\"font-size: 10pt; width: 160.5px; padding: 5px;\" valign=\"top\">\n<p style=\"font-size: 10pt; line-height: 1;\" align=\"center\"><span style=\"font-size: 10pt;\"><span style=\"font-size: 10pt;\">User Control <\/span><\/span><\/p>\n<\/td>\n<td style=\"font-size: 10pt; width: 119.25px; padding: 5px;\" valign=\"top\">\n<p style=\"font-size: 10pt; line-height: 1;\" align=\"center\"><span style=\"font-size: 10pt;\"><span style=\"font-size: 10pt;\">Faster Time to Results<\/span><\/span><\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>&nbsp;<\/p>\n<p>When data gets too big for a relational table, you must either add columns, or modify the data. Both those options are untenable when the volume and variety of data grow too fast. For this reason, some data architects turn to enterprise NoSQL databases like <a style=\"color: #1155cc; text-decoration: underline;\" href=\"http:\/\/www.marklogic.com\" target=\"_blank\" rel=\"noopener\">MarkLogic<\/a> to keep up.<\/p>\n<p style=\"font-size: 12pt; font-weight: bold;\"><span style=\"font-size: 12pt; font-weight: bold;\"><span style=\"font-size: 12pt; font-weight: bold;\">The MarkLogic DB<\/span><\/span><\/p>\n<p>MarkLogic is a NoSQL DB designed to store, search, and handle large amounts of heterogeneous data. Relational databases need a schema to define what data are stored, how it is <span style=\"background-color: #ffffff;\">categorized in tables, and the relationship between the tables. In MarkLogic, documents are the data and there is no schema per se.<\/span><\/p>\n<p style=\"background-color: #ffffff;\"><span style=\"background-color: #ffffff;\"><span style=\"background-color: #ffffff;\">In a document data model, all the data is typically contained in the same document, so the data is already denormalized. Content in blogs, press release, user manuals, books, articles, web pages, sensor data, and emails are modeled as documents. MarkLogic helps complex applications interact with large JSON, XML, SGML, HTML, RDF triples, binary files, and other popular content formats.<br \/>\n<\/span><\/span><span style=\"background-color: #ffffff;\"><span style=\"background-color: #ffffff;\"><br \/>\n<a href=\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2016\/06\/ML-pump.gif\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-9935 size-full alignleft\" title=\"MarkLogic Content Pump \" src=\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2016\/06\/ML-pump.gif\" alt=\"MarkLogic Content Pump \" width=\"190\" height=\"100\" \/><\/a><\/span><\/span><span style=\"font-size: 12pt; font-weight: bold; background-color: #ffffff;\"><span style=\"background-color: #ffffff; font-size: 12pt; font-weight: bold;\"><br \/>\n<\/span><\/span><\/p>\n<p style=\"background-color: #ffffff;\"><span style=\"font-size: 12pt; font-weight: bold; background-color: #ffffff;\"><span style=\"background-color: #ffffff; font-size: 12pt; font-weight: bold;\">Populating MarkLogic<\/span><\/span><\/p>\n<p style=\"padding-left: 90px;\">There are several ways to load data into a MarkLogic database, including:<\/p>\n<p><span style=\"font-weight: bold;\">\u2713 \u00a0MarkLogic Content Pump (MLCP):<\/span> command line tool to import and export data<br \/>\n<span style=\"font-weight: bold;\">\u2713 \u00a0Auto Loader:<\/span> monitors files and automatically loads new records into MarkLogic<br \/>\n<span style=\"font-weight: bold;\">\u2713 \u00a0Hadoop: <\/span>UI for connecting a MarkLogic Server to a MapReduce source or target<br \/>\n<span style=\"font-weight: bold;\">\u2713 \u00a0XQuery: <\/span>loads documents using XQuery functions<br \/>\n<span style=\"font-weight: bold;\">\u2713 \u00a0Node.js: <\/span>client API to read, write, and query documents and semantic data in a MarkLogic database<\/p>\n<p>This article demonstrates how IRI software populates data for MarkLogic in the same Eclipse GUI (<a style=\"color: #1155cc; text-decoration: underline;\" href=\"http:\/\/www.iri.com\/products\/workbench\">IRI Workbench<\/a>) using test data created with the <a style=\"background-color: #ffffff; color: #1155cc; text-decoration: underline;\" href=\"http:\/\/www.iri.com\/products\/rowgen\">IRI RowGen<\/a> engine. Other IRI software in the same Eclipse GUI, like FACT, CoSort, FieldShield, NextForm &#8212; or the Voracity ETL platform that includes them all &#8212; can be used to prepare or protect production data headed to or from MarkLogic.<\/p>\n<p style=\"font-size: 12pt; font-weight: bold;\"><span style=\"font-size: 12pt; font-weight: bold;\"><span style=\"font-size: 12pt; font-weight: bold;\">Creating Test Data<\/span><\/span><\/p>\n<p>In addition to creating test data for <a style=\"color: #1155cc; text-decoration: underline;\" href=\"http:\/\/www.iri.com\/solutions\/test-data\/database-test-data\">RDB targets<\/a>, RowGen can also generate very large, complex <a style=\"color: #1155cc; text-decoration: underline;\" href=\"http:\/\/www.iri.com\/blog\/test-data\/generating-test-client-data\/\">test files<\/a>. To populate a NoSQL DB with test data, files are the way to go. I used the IRI Workbench <a style=\"color: #1155cc; text-decoration: underline;\" href=\"http:\/\/www.iri.com\/products\/workbench\/rowgen-gui\">GUI for RowGen<\/a> to design a 12-million-row test CSV file-generation job and load its data into MarkLogic Server through MLCP<span style=\"background-color: #ffffff; color: #111111;\">. <\/span><\/p>\n<p><span style=\"background-color: #ffffff; color: #111111;\">The fields in my test file are: <\/span><\/p>\n<p><span style=\"font-family: Courier New; font-size: 9pt;\">ACCTNUM, DEPTNO, QUANTITY, TRANSTYPE, TRANSDATE, NAME, STREET ADDRESS, STATE, CITY<\/span><\/p>\n<p style=\"background-color: #ffffff; color: #111111;\"><span style=\"color: #111111; background-color: #ffffff;\"><span style=\"background-color: #ffffff; color: #111111;\">By default, RowGen randomly generates field values according to their data type. It can also randomly select values (using different pick techniques) from set files to enhance data realism. Use your own or IRI-supplied set files, or create them on the fly (from database columns, in-line ranges, or compound data value builds) in the GUI\u2019s \u201cNew Set File\u201d wizards. I used transdate.set, names.set, address.set, state_city.set.<\/span><\/span><\/p>\n<p style=\"font-size: 12pt; background-color: #ffffff; color: #111111; font-weight: bold;\"><span style=\"font-size: 12pt; color: #111111; font-weight: bold; background-color: #ffffff;\"><span style=\"background-color: #ffffff; color: #111111; font-size: 12pt; font-weight: bold;\">Building Test Data via RowGen in the IRI Workbench GUI<\/span><\/span><\/p>\n<p style=\"background-color: #ffffff;\"><span style=\"background-color: #ffffff;\"><span style=\"background-color: #ffffff; color: #111111;\">Creating the test file requires a single RowGen job control language script. This can be written by hand or auto-generated in the RowGen new test data job wizard in the Workbench<\/span><span style=\"background-color: #ffffff; color: #4a86e8; font-style: italic;\">.<\/span><\/span><\/p>\n<p style=\"background-color: #ffffff; color: #111111;\"><span style=\"color: #111111; background-color: #ffffff;\"><span style=\"background-color: #ffffff; color: #111111;\">IRI Workbench<\/span> <span style=\"background-color: #ffffff; color: #111111;\">is an integrated development environment (IDE) built on Eclipse that is used to create, run and manage data connections, metadata and job scripts. The language is easy to use. GUI can build them automatically with an end-to-end job wizard or a Visual ETL workflow (palette).<\/span><\/span><\/p>\n<p><span style=\"background-color: #ffffff; color: #111111;\">Refer to <\/span><a style=\"background-color: #ffffff; color: #1155cc; text-decoration: underline;\" href=\"http:\/\/www.iri.com\/blog\/test-data\/using-the-rowgen-test-data-job-wizard\/\">this link<\/a><span style=\"background-color: #ffffff; color: #111111;\"> to see how to build a test file with both randomly generated and selected data fields.<\/span> It is also possible to create and depict RowGen jobs in the workflow and field detail (\u201ctransform mapping\u201d) diagrams used in the IRI Workbench <a style=\"color: #1155cc;\" href=\"http:\/\/www.iri.com\/products\/workbench\/voracity-gui\">GUI for Voracity<\/a>, built on Eclipse:<\/p>\n<p><a title=\"Building &amp; Loading 'Big Test Data' into MarkLogic - Chaitali Mitra 0.jpg\" href=\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2016\/06\/Building-Loading-Big-Test-Data-into-MarkLogic-Chaitali-Mitra-0.jpg\" rel=\"prettyPhoto\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter\" style=\"width: 624px; height: 390px;\" title=\"Building Test Data via RowGen in the IRI Workbench GUI\" src=\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2016\/06\/t_Building-Loading-Big-Test-Data-into-MarkLogic-Chaitali-Mitra-0.jpg\" alt=\"Building Test Data via RowGen in the IRI Workbench GUI\" width=\"624\" height=\"390\" \/><\/a><\/p>\n<p style=\"font-size: 10pt;\"><span style=\"font-size: 10pt;\"><span style=\"font-size: 10pt;\">From the IRI Workbench Remote Systems Explorer (RSE) panel (on the lower left), right-click on Local Shells, and click Launch Shell. From that prompt, change into the MarkLogic bin directory and enter this command to load the CSV test file into MarkLogic with the content pump:<\/span><\/span><\/p>\n<p style=\"font-size: 9pt; font-style: italic; font-weight: bold; font-family: Courier New;\" align=\"center\"><span style=\"font-size: 9pt; font-style: italic; font-weight: bold; font-family: Courier New;\"><span style=\"font-family: Courier New; font-size: 9pt; font-weight: bold; font-style: italic;\">C:\\Program Files\\MarkLogic\\mlcp\\bin&gt;mlcp.bat import -host localhost -port 8000 -username admin -password admin -database Cosortdb -input_file_path <\/span><\/span><\/p>\n<p style=\"font-size: 9pt; font-style: italic; font-weight: bold; font-family: Courier New;\" align=\"center\"><span style=\"font-size: 9pt; font-style: italic; font-weight: bold; font-family: Courier New;\"><span style=\"font-family: Courier New; font-size: 9pt; font-weight: bold; font-style: italic;\">C:\\Mlogic\\foutput.zip -input_compressed true -input_file_type delimited_text -delimited_root_name Cosort -output_uri_prefix \/iri\/ -output_collections iri<\/span><\/span><\/p>\n<p><span style=\"font-size: 10pt;\"><span style=\"font-size: 10pt;\">Once the file is loaded into the server, I can use the query window in the IRI Workbench internal browser (pointed to <\/span><span style=\"background-color: #ffffff; font-size: 10pt;\">http:\/\/localhost:8000\/qconsole<\/span><span style=\"font-size: 10pt;\">). In that client view, I just select the database from the dropdown and can see the test data:<\/span><\/span><\/p>\n<figure id=\"\" class=\"thumbnail wp-caption aligncenter style=\"width: 634px\"><a title=\"Building &amp; Loading 'Big Test Data' into MarkLogic - Chaitali Mitra 1.jpg\" href=\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2016\/06\/Building-Loading-Big-Test-Data-into-MarkLogic-Chaitali-Mitra-1.jpg\" rel=\"prettyPhoto\"><img loading=\"lazy\" decoding=\"async\" style=\"width: 624px; height: 389px;\" title=\"RowGen test data loaded into the MarkLogic Server via MLCP, all in the IRI Workbench GUI for Voracity\" src=\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2016\/06\/t_Building-Loading-Big-Test-Data-into-MarkLogic-Chaitali-Mitra-1.jpg\" alt=\"RowGen test data loaded into the MarkLogic Server via MLCP, all in the IRI Workbench GUI for Voracity\" width=\"624\" height=\"389\" \/><\/a><figcaption class=\"caption wp-caption-text\">RowGen test data loaded into the MarkLogic Server via MLCP, all in the IRI Workbench GUI for Voracity<\/figcaption><\/figure>\n<p>For help building RowGen test data or populating your targets directly, email <a style=\"color: #1155cc; text-decoration: underline;\" href=\"mailto:rowgen@iri.com\">rowgen@iri.com<\/a>. See <a style=\"color: #1155cc; text-decoration: underline;\" href=\"http:\/\/www.iri.com\/blog\/migration\/data-migration\/using-marklogic-data-in-iri-voracity\/\">this article<\/a> for information on connecting to and using MarkLogic data in IRI Workbench operations.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Just as production data processing tools like IRI CoSort must handle big data in NoSQL DB environments, so too must a big test data generation tool like IRI RowGen. This article explains how a RowGen product or IRI Voracity platform user would leverage the test data generation facilities in their common IRI Workbench GUI to<\/p>\n<div><a class=\"btn-filled btn\" href=\"https:\/\/beta.iri.com\/blog\/test-data\/building-loading-big-test-data-into-marklogic\/\" title=\"Building &#038; Loading &#8216;Big Test Data&#8217; into MarkLogic\">Read More<\/a><\/div>\n","protected":false},"author":53,"featured_media":11796,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_exactmetrics_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"footnotes":""},"categories":[776,91,29],"tags":[1102,1100,561,1099,546,526,789,850,1106,1104,1097,1107,1103,75,1105,1101,1098,88,550],"class_list":["post-9908","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-etl","category-iri-workbench","category-test-data","tag-application-server","tag-database-internals","tag-ddf","tag-enterprise-nosql-database","tag-iri-cosort","tag-iri-rowgen","tag-iri-voracity","tag-iri-workbench","tag-javascript","tag-json","tag-marklogic","tag-marklogic-content-pump","tag-marklogic-query-console","tag-odbc","tag-rdf-triples","tag-search-style-indexing","tag-sql-operations","tag-test-data-2","tag-xml"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v23.3 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Building &amp; Loading &#039;Big Test Data&#039; into MarkLogic - IRI<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/beta.iri.com\/blog\/test-data\/building-loading-big-test-data-into-marklogic\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Building &amp; Loading &#039;Big Test Data&#039; into MarkLogic - IRI\" \/>\n<meta property=\"og:description\" content=\"Just as production data processing tools like IRI CoSort must handle big data in NoSQL DB environments, so too must a big test data generation tool like IRI RowGen. This article explains how a RowGen product or IRI Voracity platform user would leverage the test data generation facilities in their common IRI Workbench GUI toRead More\" \/>\n<meta property=\"og:url\" content=\"https:\/\/beta.iri.com\/blog\/test-data\/building-loading-big-test-data-into-marklogic\/\" \/>\n<meta property=\"og:site_name\" content=\"IRI\" \/>\n<meta property=\"article:published_time\" content=\"2016-06-20T17:56:17+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2017-11-07T13:13:22+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2016\/06\/marklogic-rowgen.png\" \/>\n\t<meta property=\"og:image:width\" content=\"742\" \/>\n\t<meta property=\"og:image:height\" content=\"448\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Chaitali Mitra\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Chaitali Mitra\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/beta.iri.com\/blog\/test-data\/building-loading-big-test-data-into-marklogic\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/test-data\/building-loading-big-test-data-into-marklogic\/\"},\"author\":{\"name\":\"Chaitali Mitra\",\"@id\":\"https:\/\/beta.iri.com\/blog\/#\/schema\/person\/9bae14a309616863b027c2d56f532caf\"},\"headline\":\"Building &#038; Loading &#8216;Big Test Data&#8217; into MarkLogic\",\"datePublished\":\"2016-06-20T17:56:17+00:00\",\"dateModified\":\"2017-11-07T13:13:22+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/test-data\/building-loading-big-test-data-into-marklogic\/\"},\"wordCount\":910,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/#organization\"},\"image\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/test-data\/building-loading-big-test-data-into-marklogic\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2016\/06\/marklogic-rowgen.png\",\"keywords\":[\"application server\",\"database internals\",\"DDF\",\"Enterprise NoSQL Database\",\"IRI CoSort\",\"IRI RowGen\",\"IRI Voracity\",\"IRI Workbench\",\"JavaScript\",\"JSON\",\"MarkLogic\",\"MarkLogic Content Pump\",\"MarkLogic Query Console\",\"ODBC\",\"RDF Triples\",\"search-style indexing\",\"SQL operations\",\"test data\",\"xml\"],\"articleSection\":[\"ETL\",\"IRI Workbench\",\"Test Data\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/beta.iri.com\/blog\/test-data\/building-loading-big-test-data-into-marklogic\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/beta.iri.com\/blog\/test-data\/building-loading-big-test-data-into-marklogic\/\",\"url\":\"https:\/\/beta.iri.com\/blog\/test-data\/building-loading-big-test-data-into-marklogic\/\",\"name\":\"Building & Loading 'Big Test Data' into MarkLogic - IRI\",\"isPartOf\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/test-data\/building-loading-big-test-data-into-marklogic\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/test-data\/building-loading-big-test-data-into-marklogic\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2016\/06\/marklogic-rowgen.png\",\"datePublished\":\"2016-06-20T17:56:17+00:00\",\"dateModified\":\"2017-11-07T13:13:22+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/test-data\/building-loading-big-test-data-into-marklogic\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/beta.iri.com\/blog\/test-data\/building-loading-big-test-data-into-marklogic\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/beta.iri.com\/blog\/test-data\/building-loading-big-test-data-into-marklogic\/#primaryimage\",\"url\":\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2016\/06\/marklogic-rowgen.png\",\"contentUrl\":\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2016\/06\/marklogic-rowgen.png\",\"width\":742,\"height\":448,\"caption\":\"MarkLogic Rowgen combined logo\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/beta.iri.com\/blog\/test-data\/building-loading-big-test-data-into-marklogic\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/beta.iri.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Building &#038; Loading &#8216;Big Test Data&#8217; into MarkLogic\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/beta.iri.com\/blog\/#website\",\"url\":\"https:\/\/beta.iri.com\/blog\/\",\"name\":\"IRI\",\"description\":\"Total Data Management Blog\",\"publisher\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/beta.iri.com\/blog\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/beta.iri.com\/blog\/#organization\",\"name\":\"IRI\",\"url\":\"https:\/\/beta.iri.com\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/beta.iri.com\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2019\/02\/iri-logo-total-data-management-small-1.png\",\"contentUrl\":\"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2019\/02\/iri-logo-total-data-management-small-1.png\",\"width\":750,\"height\":206,\"caption\":\"IRI\"},\"image\":{\"@id\":\"https:\/\/beta.iri.com\/blog\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/beta.iri.com\/blog\/#\/schema\/person\/9bae14a309616863b027c2d56f532caf\",\"name\":\"Chaitali Mitra\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/beta.iri.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/95a11f3d0b709c00df3262bab0152f3a?s=96&d=blank&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/95a11f3d0b709c00df3262bab0152f3a?s=96&d=blank&r=g\",\"caption\":\"Chaitali Mitra\"},\"sameAs\":[\"http:\/\/www.iri.com\"],\"url\":\"https:\/\/beta.iri.com\/blog\/author\/chaitalim\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Building & Loading 'Big Test Data' into MarkLogic - IRI","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/beta.iri.com\/blog\/test-data\/building-loading-big-test-data-into-marklogic\/","og_locale":"en_US","og_type":"article","og_title":"Building & Loading 'Big Test Data' into MarkLogic - IRI","og_description":"Just as production data processing tools like IRI CoSort must handle big data in NoSQL DB environments, so too must a big test data generation tool like IRI RowGen. This article explains how a RowGen product or IRI Voracity platform user would leverage the test data generation facilities in their common IRI Workbench GUI toRead More","og_url":"https:\/\/beta.iri.com\/blog\/test-data\/building-loading-big-test-data-into-marklogic\/","og_site_name":"IRI","article_published_time":"2016-06-20T17:56:17+00:00","article_modified_time":"2017-11-07T13:13:22+00:00","og_image":[{"width":742,"height":448,"url":"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2016\/06\/marklogic-rowgen.png","type":"image\/png"}],"author":"Chaitali Mitra","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Chaitali Mitra","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/beta.iri.com\/blog\/test-data\/building-loading-big-test-data-into-marklogic\/#article","isPartOf":{"@id":"https:\/\/beta.iri.com\/blog\/test-data\/building-loading-big-test-data-into-marklogic\/"},"author":{"name":"Chaitali Mitra","@id":"https:\/\/beta.iri.com\/blog\/#\/schema\/person\/9bae14a309616863b027c2d56f532caf"},"headline":"Building &#038; Loading &#8216;Big Test Data&#8217; into MarkLogic","datePublished":"2016-06-20T17:56:17+00:00","dateModified":"2017-11-07T13:13:22+00:00","mainEntityOfPage":{"@id":"https:\/\/beta.iri.com\/blog\/test-data\/building-loading-big-test-data-into-marklogic\/"},"wordCount":910,"commentCount":0,"publisher":{"@id":"https:\/\/beta.iri.com\/blog\/#organization"},"image":{"@id":"https:\/\/beta.iri.com\/blog\/test-data\/building-loading-big-test-data-into-marklogic\/#primaryimage"},"thumbnailUrl":"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2016\/06\/marklogic-rowgen.png","keywords":["application server","database internals","DDF","Enterprise NoSQL Database","IRI CoSort","IRI RowGen","IRI Voracity","IRI Workbench","JavaScript","JSON","MarkLogic","MarkLogic Content Pump","MarkLogic Query Console","ODBC","RDF Triples","search-style indexing","SQL operations","test data","xml"],"articleSection":["ETL","IRI Workbench","Test Data"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/beta.iri.com\/blog\/test-data\/building-loading-big-test-data-into-marklogic\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/beta.iri.com\/blog\/test-data\/building-loading-big-test-data-into-marklogic\/","url":"https:\/\/beta.iri.com\/blog\/test-data\/building-loading-big-test-data-into-marklogic\/","name":"Building & Loading 'Big Test Data' into MarkLogic - IRI","isPartOf":{"@id":"https:\/\/beta.iri.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/beta.iri.com\/blog\/test-data\/building-loading-big-test-data-into-marklogic\/#primaryimage"},"image":{"@id":"https:\/\/beta.iri.com\/blog\/test-data\/building-loading-big-test-data-into-marklogic\/#primaryimage"},"thumbnailUrl":"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2016\/06\/marklogic-rowgen.png","datePublished":"2016-06-20T17:56:17+00:00","dateModified":"2017-11-07T13:13:22+00:00","breadcrumb":{"@id":"https:\/\/beta.iri.com\/blog\/test-data\/building-loading-big-test-data-into-marklogic\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/beta.iri.com\/blog\/test-data\/building-loading-big-test-data-into-marklogic\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/beta.iri.com\/blog\/test-data\/building-loading-big-test-data-into-marklogic\/#primaryimage","url":"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2016\/06\/marklogic-rowgen.png","contentUrl":"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2016\/06\/marklogic-rowgen.png","width":742,"height":448,"caption":"MarkLogic Rowgen combined logo"},{"@type":"BreadcrumbList","@id":"https:\/\/beta.iri.com\/blog\/test-data\/building-loading-big-test-data-into-marklogic\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/beta.iri.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Building &#038; Loading &#8216;Big Test Data&#8217; into MarkLogic"}]},{"@type":"WebSite","@id":"https:\/\/beta.iri.com\/blog\/#website","url":"https:\/\/beta.iri.com\/blog\/","name":"IRI","description":"Total Data Management Blog","publisher":{"@id":"https:\/\/beta.iri.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/beta.iri.com\/blog\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/beta.iri.com\/blog\/#organization","name":"IRI","url":"https:\/\/beta.iri.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/beta.iri.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2019\/02\/iri-logo-total-data-management-small-1.png","contentUrl":"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2019\/02\/iri-logo-total-data-management-small-1.png","width":750,"height":206,"caption":"IRI"},"image":{"@id":"https:\/\/beta.iri.com\/blog\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/beta.iri.com\/blog\/#\/schema\/person\/9bae14a309616863b027c2d56f532caf","name":"Chaitali Mitra","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/beta.iri.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/95a11f3d0b709c00df3262bab0152f3a?s=96&d=blank&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/95a11f3d0b709c00df3262bab0152f3a?s=96&d=blank&r=g","caption":"Chaitali Mitra"},"sameAs":["http:\/\/www.iri.com"],"url":"https:\/\/beta.iri.com\/blog\/author\/chaitalim\/"}]}},"jetpack_featured_media_url":"https:\/\/beta.iri.com\/blog\/wp-content\/uploads\/2016\/06\/marklogic-rowgen.png","_links":{"self":[{"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/posts\/9908"}],"collection":[{"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/users\/53"}],"replies":[{"embeddable":true,"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/comments?post=9908"}],"version-history":[{"count":20,"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/posts\/9908\/revisions"}],"predecessor-version":[{"id":11797,"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/posts\/9908\/revisions\/11797"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/media\/11796"}],"wp:attachment":[{"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/media?parent=9908"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/categories?post=9908"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/beta.iri.com\/blog\/wp-json\/wp\/v2\/tags?post=9908"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}