Checklist for Hadoop Administrators

User Identity

Your Hadoop cluster is accessed by Datameer as a particular Hadoop user. By default, the user is identified as the UNIX user who launched the Datameer application (equivalent to the UNIX command 'whoami'). To ensure this works properly, you should create a user of the same name within Hadoop's HDFS for Datameer to use exclusively for scheduling and configuration. This ensures the proper permissions are set, and that this user is recognized whenever the Datameer application interacts with your Hadoop cluster.

The username used to launch the Datameer application can be configured in <Datameer folder>/etc/das-env.sh

Permissions in HDFS

When running Datameer with an on-premise Hadoop cluster (called "distributed mode"), you need to define an area (folder) within HDFS for Datameer to store its private data. The Hadoop user created for Datameer should have read/write access to this folder.

Hadoop permissions can be set with the following commands. Note that permission checking can be enabled/disabled globally for HDFS. See Configuring Datameer in a Shared Hadoop Cluster for more information.

hadoop -fs
      [-chmod [-R] <MODE[,MODE]... | OCTALMODE> PATH...]
      [-chown [-R] [OWNER][:[GROUP]] PATH...]
      [-chgrp [-R] GROUP PATH...]

If you experience a problem with the permissions of the staging directory when submitting jobs other then with the superuser. There are two solutions for this:

1. Configure mapreduce.jobtracker.staging.root.dir to /user in the mapred-site.xml of the cluster.
2. Or, change the permissions of the hadoop.tmp.dir (usually /tmp) inside the hdfs to 777.

See the Hadoop documentation for additional information.

Local UNIX Permissions

Various components of your Hadoop and Datameer environment (data nodes, task trackers, JDBC drivers, the Datameer application server) use a folder for temporary storage or "scratch" space. Additionally, HDFS files are sometimes stored in this location by default (see the Hadooop documentation).

Datameer searches internally for the possible locations (folder/directory) that have the most free space in the local filesystem to write temporary files.

Possible options:

• mapred.local.dir (Property is used in old MR API)
• mapreduce.cluster.local.dir (Property for new MR API)
• tez.runtime.framework.local.dirs (Property for TEZ mode)
• yarn.nodemanager.local-dirs (Property for Yarn)
• Java system temp folder

For each machine (the Datameer server, Hadoop master and slaves), you must ensure that the local UNIX user running these processes (often "hadoop" or "datameer", depending on the component) has read/write permissions for these folders, and of course to /tmp (commonly used by some components). Different components (e.g. Hive) can use different, configurable locations for scratch space. Check your configuration files for details.

In particular, Datameer makes use of the directory mapred.local.dir defined in your Hadoop configuration. Check that Datameer has write access to this folder, both on the Datameer server and cluster nodes. If not properly configured, you might see cryptic exceptions like the one reported here: https://issues.apache.org/jira/browse/MAPREDUCE-635

Make sure the Datameer application is consistently run by the same user, so that certain log files and other temporary data aren't written as root and then locked when Datameer is subsequently started by another user. This can lead to inconsistent states which are difficult to troubleshoot (without access to the log files).

Network Connectivity

Hadoop Clusters are often secured behind firewalls, which might or might not be configured to allow Hadoop direct access to all data sources. The systems containing data to be analyzed, including web servers, databases, data warehouses, hosted applications, message queues, mobile devices, etc. are often in remote locations and accessible through various specific TCP ports. Normally, Datameer is installed in the same LAN as Hadoop, and therefore suffers from the same connectivity issues, which can prevent you from using Datameer. Conversely, if Datameer is located further away from Hadoop, it might have connectivity to data but not the Hadoop NameNode and Job Tracker, for the same reasons. Most of these issues aren't anticipated early in the life cycle of a Hadoop project, and thus firewalls need to be reconfigured. Finally, the speed and latency of the network link between Datameer, Hadoop, and your data can severely affect system performance and the end user experience. For these reasons, it's important to thoroughly review all network connectivity requirements up front. 

Before you start using Datameer, check that the following network connectivity is available on the specified ports:

1. Datameer to Hadoop NameNode client port: (normally 8020, 9000 or 54310)
2. Datameer to Hadoop JobTracker client port: (normally 8021, 9001 or 54311)
3. Datameer to Hadoop DataNodes (normally ports 50010 and 50020)
4. (Recommended): Datameer to Hadoop Administration consoles (normally 50030 and 50070)

Depending on your configuration, the following connectivity might also be necessary:

1. Importing/exporting via SFTP:
   1. Datameer to source (normally port 22)
   2. Hadoop slaves to source (normally port 22)
2. Importing/linking/exporting to/from an RDBMS:
   1. Datameer to RDBMS via JDBC (various ports, see your DB documentation)
   2. Hadoop slaves to RDBMS via JDBC (various ports, see your DB documentation)
3. Connecting to Hive:
   1. Hive Metastore (normally port 10000)
   2. Location of external Hive tables, if applicable (e.g. S3 tables via port 443)
4. Connecting to HBase:
   1. Zookeeper client port (normally 2181)
   2. HBase master (normally 60010)

Installed Compression Libraries

For standard Hadoop compression algorithms, you can choose the algorithm Datameer should use. However, if your Hadoop cluster is using a non-standard compression algorithm such as LZO, you need to install these libraries onto the Datameer machine. This is necessary so that Datameer can read the files it writes to HDFS, and decompress files residing on HDFS which you wish to import. Libraries which utilize native compression require both a Java (JAR) and native code component (UNIX packages). The Java component is a JAR file which needs to be placed into <Datameer folder>/etc/custom-jars. See Frequently Asked Hadoop Questions#Q. How do I configure Datameer/Hadoop to use native compression? for more details.

Connecting Datameer to Your Hadoop Cluster

By default, Datameer isn't connected to any Hadoop cluster and operates in local mode, with all analytics and other functions performed by a local instance of Hadoop, which is useful for prototyping with small data sets, but not for high volume testing or production. To connect Datameer to a Hadoop cluster, change the settings in Datameer in the Admin > Hadoop Cluster page. Click the Admin tab at the top of the page and click the Hadoop Cluster tab in the left column. Click Edit and change the Mode setting.

The Datameer Private Folder must be set to the HDFS folder chosen in step 2, as seen below:

While running, Datameer stores imported data, sample data, workbook results, logs, and other information in this area (including subfolders such as importjobs, jobjars, temp). You need to make sure no other application interferes with data in this area of HDFS.

Mandatory Hadoop settings

Hadoop settings (properties) can be configured by Datameer in three places:

1. Global settings in Datameer are located in the under the Administration tab in Hadoop Cluster.
2. Per-job properties are configured for individual import/analytics jobs
3. Property files (under /conf)

Setting the following properties ("propertyname=value") ensures that Datameer runs properly on your Hadoop cluster:

Highly Recommended Hadoop Settings

Highly Recommended YARN Settings

/usr/lib/hadoop/lib/native

Job Scheduling and Prioritization

If you have configured a job scheduling/prioritization mechanism on your cluster (such as Fair Scheduler or Capacity Scheduler), you must decide which queue or pool Datameer should use (see the Hadoop Documentation ) for more information). To link Datameer to your scheduling mechanism, do the following:

• Check your Hadoop configuration to see which property you should use to choose a pool for Datameer. For FairScheduler, this is mapred.fairscheduler.poolnameproperty, which is configured in conf/mapred-site.xml on your Hadoop cluster
• Set this property globally for Datameer at Administration > Hadoop Cluster > Custom Property (e.g., mapred.fairscheduler.poolnameproperty="datameerPool")

Additional Resources

• HDFS Permissions Guide: https://hadoop.apache.org/docs/stable1/hdfs_permissions_guide.html
• Security in Hadoop, Part – 1 from Blog 'Big Data and etc.' http://bigdata.wordpress.com/2010/03/22/security-in-hadoop-part-1
• Using Fair Scheduler: http://hadoop.apache.org/docs/stable1/fair_scheduler.html