This article, along with counterpart articles for Cloudera and HortonWorks, describes the simple 5-step process to connect the IRI Voracity big data management platform to any MapR distribution through the VGrid Gateway.

After connecting,  data can be conveniently moved between HDFS and other systems. Furthermore, that data can be manipulated and masked in Hadoop via MR2, Spark, Spark Stream, Storm or Tez using the jobs created in Voracity’s Eclipse IDE, IRI Workbench.

Step 1 – Collect Information from the HDFS Configuration File

1. Use terminal to search in HDFS configuration file.
   
2. Make a note of the NameNode Web UI Port (grep HTTPFS_HTTP_PORT /opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/httpfs-env.sh) value (in this case: 14000)

Step 2 – Collect Information from the MapR Configuration File

1. Open the configuration file and use find to search for the appropriate keywords.

2. Make a note of the NameNode Port (mapr.host) value (in this case: maprdemo)

3. Make a note of the ResourceManager Web Application HTTP Port (yarn.resourcemanager.webapp.address) value (in this case: 8088)

4. Make a note of the MapReduce JobHistory Web Application HTTP Port (mapreduce.jobhistory.webapp.address) value (in this case: 19888)

5. Make note of the Resource Manager Address (yarn.resourcemanager.address) value (in this case: 8032)

Step 3 – Collect Information from the Oozie Configuration File

1.  Use terminal to search in Oozie configuration file.

 

2. Make a note of the Oozie HTTP Port (grep OOZIE_HTTP_PORT /opt/mapr/oozie/oozie-4.3.0/conf/oozie-env.sh) value (in this case: 11000)

Step 4 – Enter configuration details in VGrid Dashboard

1. Log into the VGrid Gateway.

2. Click User > Add User and enter the user information.

3. Click the X in the success banner to refresh the screen.

4. Click Detail in the Action section of the new user.

5. Make note of the generated API key shown. It will be needed in the VGrid Gateway setup in the Workbench preferences screen.

6. Click HadoopConfig and Add Hadoop Config.

• Cluster = Cluster name
• User = User name that will be used as the user in the Hadoop file system when working in the workbench
• Hdfs = NameNode Web UI Port
• Namenode = NameNode Port
• Proxy = ResourceManager Web Application HTTP Port
• History = MapReduce JobHistory Web Application HTTP Port
• Jobtracker = Resource Manager Address
• Oozie = Oozie HTTP Port

7. Click the X in the success banner to refresh the screen.

8. Click HadoopConfig and click inactive to activate that configuration.

9. Multiple configurations can be associated with each user; however, only one can be active at any given time.

Step 5 – Enter configuration details in IRI Workbench

1. Open IRI Workbench. On the IRI > VGrid Gateway preferences screen, enter the details of the connection.

2. Click Test Connection to confirm that a successful connection is made. Included engines can be narrowed down here if only certain engines are being used in the Hadoop environment. Also, a default engine can be selected for Hadoop run configurations.

Once connected, you should be able to interact with HDFS and run compatible Voracity jobs seamlessly per this article. If you have any questions or need assistance, contact voracity@iri.com.