Page tree
Skip to end of metadata
Go to start of metadata

 

To install Datameer you need to do the following:

  1. Specify an installation folder and unzip the files.
  2. Download and install the database drivers. (Not required for evaluation purposes).
  3. Configure Datameer required and optional settings and initialize the database.
  4. Install the license.
  5. Run the application.
  6. Log in and change the default admin password following the instructions in: Managing User Accounts

Note: The default username and password are admin and admin.

Each of these steps is documented in the sections that follow. If you are upgrading from a previous version, see the Upgrade Instructions.

Installing multiple instances of Datameer isn't supported and can lead to production performance loss.

Downloading and Unzipping the Software

  • Specify a location to install software to such as: /usr/local/das-<version>

    export INSTALL_LOCATION=/usr/local
    
  • Expand the archive to that location.

    cd $INSTALL_LOCATION
    unzip <path-to-zip>/das-<version>.zip
    
  • Change the working directory to the newly created directory.

    cd Datameer-<version>

After unzipping, create a new group datameer and user datameer. Go back and change the folder permissions from user root to the newly created user datameer.

  • Add new group
/usr/sbin/groupadd --system datameer
  • Add new user
/usr/sbin/useradd --system --create-home --gid datameer -m datameer
  • Change owner and group of folder as well as all child folders
chown -R datameer:datameer $INSTALL_LOCATION/Datameer-<version>
 
Example:
chown -R datameer:datameer /usr/local/Datameer-5.0.0-hdp-2.1.2.0

 

As of Datameer v4.4 this will be completed automatically when installing from a distribution package (RPM/Debian).

 

Configuring Datameer

  • Configure Datameer by editing etc/das-env.sh. The configuration file is well documented.

    vi etc/das-env.sh
    
  • Depending on what value is used in the etc/das-env.sh for "DAS_DEPLOY_MODE", you will need to edit the corresponding properties file to update the IP/host. I.e., If using "export DAS_DEPLOY_MODE=live", update the <DAS HOME>/conf/live.properties. Change the key "system. Change the key "system.property.server.address" to change the IP/host & port of the server used in notification emails generated by Datameer.
  • Workgroup and Enterprise edition users may want to consider allocating additional memory in etc/das-env.sh. Datameer recommends updating from 1GB to 2GB ( Xmx1024m ----> -Xmx2048m). This is especially needed for the 64 bit environment.

    # Adjust max available memory (-Xmx) according to your needs
    # WARNING: Path variables that may contain blanks should be added to jetty.sh start_das() method (see DAP-6342)
    export JAVA_OPTIONS="-Xmx1024m -XX:MaxPermSize=384m -Xms256m -XX:MaxNewSize=448m -XX:SurvivorRatio=6 -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=80 -XX:+HeapDumpOnOutOfMemoryError -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled"
    export JAVA_OPTIONS="$JAVA_OPTIONS -Dfile.encoding=utf-8"

 

When configuring etc/das-env.sh, the DAS_DEPLOY_MODE is defaulted to "trial".  If you are using MySQL change "trial" to "live".

Change your stored data directory

It is recommended to use a path that is not depending on a Datameer installation directory. Since the 'das-data' folder is stored inside of your installation directory by default, you'll have to make a backup of your stored data, every time you create a new distribution or upgrade.

 

Change your heap dump location

If your heap dumps are large, you can change the location of your heap dumps in jetty.sh.

Set the location in bin/jetty.sh by modifying the following line and setting the path using HeapDumpPath=:

echo $JAVA $JAVA_OPTIONS -XX:HeapDumpPath="'$JETTY_HOME/logs/heapdump.log'" -Dlog4j.configuration="'file://$JETTY_HOME/conf/log4j-$DAS_LOG4J_CONF.properties'" -Djetty.home="'$JETTY_HOME'" -Djava.io.tmpdir="$TMP" -Djava.library.path="'$JAVA_LIBRARY_PATH'" -jar "'$JETTY_START'"

Setting up a MySQL database (optional)

If you are evaluating Datameer

You can use an HSQL database on your local filesystem and there is no need to set up an external database. If you are setting up Datameer for production use, we recommend using MySQL.

Learn how to migrate from an HSQL database to a MySQL database. 

As of Datameer version 6.0, only versions 5.5 and above are supported for MySQL.

Configuring Datameer for MySQL
  • By default, the application runs with the HSQL file database that is created on the local filesystem (under das-data/database/hsql-db). If you want to run the application in MySQL mode, change the DAS_DEPLOY_MODE property in the das-env.sh as follows:

    export DAS_DEPLOY_MODE=live
    
  • To specify different database credentials make the following two changes:
    • Change the MySQL database credentials in the create database script:

      vi bin/mysql-init.sql
      
    • Change the MySQL database credentials in the live.properties file. Add the properties for system.property.db.host, system.property.db.port, system.property.db.name, system.property.db.username and system.property.db.password.

      vi conf/live.properties
      
    • Change the following properties:

      system.property.db.name=dap
      system.property.db.username=dap
      system.property.db.password=dap
      system.property.db.host=localhost
      system.property.db.port=3306

      Default credentials for MySQL are user: dap, password: dap.

  • Initialize the database with the following commands:

    mysql [-h <dbhost>] -u root -p < bin/mysql-init.sql
    mysql [-h <dbhost>] -u root -p dap < bin/create-tables.sql
    

    You will be prompted to enter the password to access the database.

  • Downgrading Datameer versions

    If for some reason you need to downgrade Datameer versions, this is the procedure to follow:
    
    mysqldump -u USER -pdap dap > filename.sql  --dump of Datameer database you tried to upgrade from
    mysql -u <user> -p <pwd>
    drop database dap;
    create database dap;
    exit;
    mysql -u <user> -p dap < filename.sql
    
Installing the MySQL JDBC database drivers

 

Migrate from an HSQL database to a MySQL database

Datameer provides a tool to migrate data from an HSQL file database to a MySQL database:

  1. Install a MySQL database.
  2. Configure Datameer for MySQL
  3. Migrate the HSQL database to MySQL using Datameer's migration tool:

    bin/migrate-db-tool.sh hsql-file:<Datameer user path>/Datameer/<version>/das-data/database mysql

    The default path to the hsql database is <Datameer user path>/Datameer/<version>/das-data/database/...

    * If your hsql database has been moved from the default location, the path will need to be specified.

    bin/migrate-db-tool.sh hsql-file:<file path of hsql database> mysql

    * Ensure this script is being run from root Datameer installation folder.

Installing the license

If you have already received a license from Datameer, copy the license file to: $INSTALL_LOCATION/das-<version>/etc/license

If you do not have a license, email license@datameer.com.

Note: The file must be named license.lic or it won't work.

If you attempt to launch the application without a license you receive an error message asking you to contact Datameer for a license file. Include the MAC address listed in this error message when contacting Datameer for a license. License information and usage can be viewed at any time by clicking the Administration tab and clicking License.

See License Information for information on how to update the license and for details about volume-based licensing.

Running the application

You can start, stop, and restart the application by going to the /bin folder and entering commands.

Note: The log file is written to logs/conductor.log using log4j.

To change log behavior, edit conf/log4j-production.properties or write your own log property file. (Tip: When you define settings in conf/log4j-customized.properties use customized as the log4jConf start parameter).

Starting the application

(Recommended as the new user datameer previously created rather than as user root)

  • To start the application, enter the following command :

    bin/conductor.sh start
    
  • Optionally, to start the application using example data and workbooks, enter the following command. You need to edit the injected SCP connection "example_scp_connection" and enter the SSH credentials of the application.

    bin/conductor.sh start --injectExamples
    

Stopping the application

  • To stop the application, enter the following command:

    bin/conductor.sh stop
    

Restarting the application

  • To restart the application, enter the following command:

    bin/conductor.sh restart

Graceful shut down

  • Use the following steps to gracefully shut down Datameer:

    1. Pause the Job Scheduler located under the Administration tab in Datameer.
    2. Wait for current jobs to be marked as completed.
    3. When all jobs have been completed, use the "stop" command on conductor.sh
    4. After the Datameer application has been stopped, perform needed maintenance.
    5. With all maintenance completed, resume Datameer using the "start" command on conductor.sh.
    6. Under Datameer's Administration tab, resume the Job Scheduler. 

    If a maintenance is needed sooner than the time it takes all jobs to complete, during step #2 the administrator can elect to force cancel current jobs.

conductor.sh commands and parameters

Usage: conductor.sh <command> <option>

 

Commands:

 start - starts the conductor

 stop - stops the conductor

 restart - restarts the conductor

 check - check if the conductor is already running

 

Options:

 --injectExamples: injects example import jobs and workbooks on start-up.

 --resetPassword: resets the admin password to default value.

 --jmx: starts JMX management extension for managing and monitoring DAS.

 --profile: runs conductor with attached profiling agent.

 --profile-sampling: runs conductor with cpu profiling (sampling) activated.

 --profile-tracing: runs conductor with cpu profiling (tracing) activated.

 --profile-memory: runs conductor with cpu and memory profiling (sampling) activated.

 --help: opens the help dialog.


Examples:

Installing Custom Database Drivers

  • Optional: You can install custom database drivers to connect to other types of databases. This step allows you to import your existing data into Datameer and Hadoop. 
    For a listing of supported databases and other details, see How to Install Database Drivers.

Special settings

The following settings should only be used in exceptional cases.

File

Setting

Description

etc/das-env.sh

export DAS_DB_MODE=mysql|hsql-file

Choose the db type that should be used. Default is hsql-file.

 

Specific distribution instillation instructions

 

MySQL Installation Guide

MySQL is used for the Datameer metastore--where Datameer stores its metadata.

Installing MySQL on Fedora

  • To install the MySQL server, enter the following command.

    yum install mysql-server
    
  • To start MySQL, enter the following command.

    /etc/init.d/mysqld start

Installing MySQL on Debian

  • To install MySQL server, enter the following command.

    apt-get install mysql-server

Installing MySQL on Solaris and OpenSolaris

  • To install MySQL server in Solaris use solaris PKG
  • To install MySQL server in OpenSolaris use IPS 

Installing MySQL on CentOS

  • To install the MySQL server, enter the following command.

    yum install mysql-server
  • To start MySQL, enter the following command.

    service mysql start

Where to go from here

If you are a Datameer system administrator, see the Administrator's Guide.