This installation guide explains how to set up Datameer for enterprise and productions environments. If you are upgrading from a previous version, see the Upgrade Instructions.

Following this step-by-step guide also prepares you for later unattended installation, integration into AnsibleChefPuppet, or Saltstack, and creating a log of changes. To achieve this, configuration and property changes in files are made using sed.

Prerequisites

Complete the following prerequisites before installing Datameer:

Create the Datameer User

Administrative rights are required to create the Datameer user on the machine where Datameer is being installed. This can be accomplished under the root account. Make sure the user ID is above 500 and that the account has enough resources and file descriptors available. 

Create Directories for Application, Cache, Logs, and Temporary Files 

For performance reasons and to have better control about where space on the file systems and on disks is used, create separate directories for application, cache, logs, and temporary files. Do this according the Linux Filesystem Hierarchy Standard (FHS). To create the directories and change the permissions you need administrative rights. Complete this task under the user account root.

Switch the User and Change the Working Directory 

This should be the last task to which administrative rights are necessary. 

Download and Unzip Datameer

Download the appropriate Datameer package for your Hadoop cluster distribution. If you have already a Datameer installation you can also start from here. 

If you are an authorized Enterprise customer, you can get the download link for the latest public available package from https://my.datameer.com/workspace/downloads, or request one through your Customer Success Manager (CSM). 

 To be prepared for future upgrades, create symlinks to the current (or latest) package as well as for the log directory. 

  • Create symlink and change the working directory:

    ln -s Datameer-<package> current 
    cd current

By default, all Datameer logs are in the installation subdirectory logs/ . For logs, there is no single property to specify the location, but many depending on the type of log. The main configuration file where you can change the location for most of the log files is conf/log4j-production.properties. To keep the change fast and simple, log in a central location according the Linux Filesystem Hierarchy Standard (FHS)

  • Move the log directory:

    mv logs/.donotdelete /var/log/datameer 
    rm -rf logs
    ln -s /var/log/datameer logs

Download and Install the MySQL Database JDBC Connector 

By default, the Datameer application runs with an HSQL file database that is created on the local filesystem under das-data/database/hsql-db. If you are setting up Datameer for production use, Datameer strongly recommends using MySQL instead of the HSQL file database. 
As of Datameer 7.4: MariaDB is supported as an alternative to MySQL.

Configure Datameer for MySQL Database

Datameer service depends on the MySQL database. The MySQL database is used for writing to workbooks, permission changes, job execution, scheduling, and more. To function properly,  a response time should be between ten and twenty milliseconds. To run the application in MySQL mode, the following changes need to be implemented. As of Datameer 7.4: MariaDB is supported as an alternative to MySQL.

Installing the License

If you don't have a license, email the application's product ID to license@datameer.com and request the key. Find the product ID displayed at the 'Welcome' page.

See 'License Information' for information on how to update the license and for details about volume-based licensing.

If you have already received a Datameer license:

  1. Launch the Datameer application and open the UI. The welcome page with all available licensing options is loaded. 
  2. Press the button "Activate" and upload the key you received from Datameer. The license is being activated. You will be redirected to the login page. 

Start Datameer 

Start the Datameer service.

Stop Datameer 

Stop the Datameer service.

Restart Datameer

Restart the Datameer service.

Datameer Graceful Shutdown

Gracefully shut down the Datameer service.

  1. Pause the Job Scheduler located under the Admin tab in Datameer.
  2. Wait for current jobs to be marked as completed.
  3. When all jobs have been completed, use the "stop" command on conductor.sh
  4. After the Datameer application has been stopped, perform needed maintenance.
  5. With all maintenance completed, resume Datameer using the "start" command on conductor.sh.
  6. Under Datameer's Admin tab, resume the Job Scheduler

Service Check 

Check if the Datameer service is running and accessible

Configure Datameer for Kerberos Secured Cluster

Before configuring Datameer for a Kerberos Secured cluster, test Kerberos authentication and job execution on CLI.  

To configure Datameer for a Kerberos-secured cluster follow the Secure Mode Configuration instructions. 

Secure Hadoop Distributed Filesystem (HDFS)

You must have a properly configured connection to a Kerberos-secured cluster to use the tool to secure the Hadoop Distributed Filesystem (HDFS) .

Start Testing

Start the Datameer service to do final testing. 

Best Practices for Installing Datameer

Implement frequent database backups

Datameer service depends on the MySQL database, it is used for writing to workbooks, permission changes, job execution, scheduling, and more. It is highly recommend to backup the application database frequently.

0 * * * * mysqldump -u'dap' -p'dap' dap | gzip > /home/datameer/<company>_<system>_<datameer-version>_`date +\%Y\%m\%d_\%H\%M`.sql.gz
Don't leave the backup unattended for long time. Monitor the directory /home/datameer for its size!
# Check from time to tome how long the database dump will take and if it fits into the timeslot
time mysqldump -u'dap' -p'dap' dap | gzip > /home/datameer/<company>_<system>_<datameer-version>_`date +\%Y\%m\%d_\%H\%M`.sql.gz
# Verify from time to time if the files are OK
gzip -d /home/datameer/company>_<system>_<datameer-version>_<date>_<time>.sql.gz 
head /home/datameer/<company>_<system>_<datameer-version>_<date>_<time>.sql
Validate the content. Don't leave backup files on the application server. Move backup files from /home/datameer to a safe and secure remote location.

Change your stored data directory

Use a path that doesn't depend on a Datameer installation directory. Because the das-data folder is stored inside of your installation directory by default, you need to make a backup of your stored data every time you create a new distribution or upgrade. 


Change the default admin password

Log in and change the default admin password following the instructions on managing user accounts.

Download and install plug-ins

If you are setting up Datameer for production use, it is most likely in a Kerberos Secured environment. To use Kerberos, an additional plug-in is necessary. This Datameer plug-in is part of the Advanced Governance module.  

Configure Datameer for enterprise 

By default, the application runs with settings where files are created on the local filesystem under the current directory. To address enterprise requirements, some changes need to be implemented. 
To avoid any mismatch in the configuration files or incompatibility with different versions, don't copy over configuration files from other versions. Make changes every time based on the originally delivered versions. 

Review the changes implemented by accessing the change log

Validate the changes made. Move files from /home/datameer to a safe and secure remote location.


Enable and configure transport layer security (TLS)

Before the next steps, consider reverse proxies or a load balancer to offload the SSL traffic or to use wild card certificates. In that case, you only need to configure rewrite handling.

Enable TLS for use with Datameer in production environments. As Datameer is packed with Jetty 9, you only need to enable modules.

You can proceed further with Enabling SSL for MySQL service as well.


Configure bash for operations 

Set up shell aliases for most common commands to make work easier, faster, and less error prone.  

Conductor.sh commands and parameters 

Usage: conductor.sh <command> <option>

Commands:

Options:

Examples:

Where to go from here

If you are a Datameer system administrator, see the Administrator's Guide.