Datameer customers can download updated software versions from http://my.datameer.com
To upgrade from an older release, you should first back up your data. Then you will need to upgrade the application and upgrade the database.
Datameer now uses Parquet as its default storage format as opposed to the previous sequence storage format. Sequence Files are still readable in Datameer but all new artifacts are written in Parquet. Depending on the configuration of your jobs, it might be difficult to roll back after upgrading to Datameer 6.3.
There is a property that forces Datameer to use Sequence Files instead of Parquet if a rollback is required. Under the JAVA_OPTIONS in etc/das-env.sh, add "
If upgrading from a Datameer version before 6.1, we highly recommend first upgrading to Datameer 6.1 and verifying that all job functions work without the MapReduce framework before upgrading to Datameer 6.3.
In 6.3, Datameer uses Parquet as its default storage format. While Datameer can still read data written in the sequence file format, all artifacts are now stored using the Parquet file format. Datameer still uses the sequence file format for writing intermediate files and preview files for performance reasons, but the final artifacts are saved using the Parquet format. Due to this change, it might not be possible to roll back after upgrading to Datameer 6.3, depending on the configuration of your artifacts. For example, if import jobs and workbooks are configured to purge historical data and only keep the last one result, the final artifacts are stored as Parquet files, and prior versions of Datameer cannot read them.
As of Datameer 6.3, its' required to use ports 389/636 to connect to LDAP or the ports 3268/3269 to connect to Active Directory.
Ensure that Datameer application server, as well all data nodes, have Java 1.8 (Oracle recommended)
If you are upgrading to Datameer 7.2 from a 6.1.x or earlier version, must first upgrade to a 7.1.12 or later 7.1.x release, in order to trigger an important Database schema migration processes.
Keep the following in mind if you use Kerberos:
Workbooks and other JSON files downloaded as a backup in older versions of Datameer are not supported in newer versions of Datameer.
When upgrading to a newer version of Datameer, ensure that all needed workbooks and files are part of the migration process so that they can be used in your new version of Datameer.
In the event of a problem after a Datameer upgrade, it is possible to roll back to the previous release. To allow for the possibility of a roll back and avoid any potential data loss, Datameer strongly recommends disabling the Housekeeping and Compaction services before beginning the upgrade process. Once upgrade validation is complete and the environment is considered stable, the Housekeeping and Compaction services can be re-enabled.
Restarting Datameer is required to apply these changes
A Workbook is a complex structure that Datameer is constantly striving to improve. Workbooks can be built in a near unlimited number of different combinations of Column Renames, Joins, Unions, Filters, and Functions - including user built functions. While we do attempt to cover all the possibilities while implementing our upgrade code, some customer development decisions are unpredictable and as a result our Workbook upgrade code can not always successfully transform all workbooks.
To validate whether any Workbook has been broken during an upgrade, Datameer created the Workbook Health Check feature. The tool reviews a Workbook's structure and reports any logical issues. This tool is available in versions 6.4.14, 7.1.13, 7.2.13, 7.4.11+, 7.5.4+, and 10.0.1+. Datameer's best practice recommendations for upgrades are as follows:
If you have set up password encryption and/or enabled TLS with custom certificates , backup your Keystore and Keyczar keyfiles.
If you update Datameer using user/group root it is recommended for security to go back and change permissions back to the user/group datameer.
Stop the application.
<Datameer Application Folder>/bin/conductor.sh stop
Unzip the upgrade file.
Move the MySQL JDBC driver located in <Datameer path>/das-data/jdbc-jars, as described into the Install the JDBC Database Drivers section of the Installation Guide.
In case you are using Datameer with MySQL as an application database, please check that database mode is configured accordingly in your etc/das-env.sh file
If upgrading in Workgroup or Enterprise versions, users may want to consider allocating additional memory in etc/das-env.sh.
# Adjust max available memory (-Xmx) according to your needs
Consider changing the container sizing of the Map, Reduce, and AM containers, which respectively correspond to MapReduce and Tez jobs. To do so, change the settings of the following properties:
das.job.map-task.memory=2048 das.job.reduce-task.memory=2048 das.job.application-manager.memory=2048
If existing, copy the files from
the das-data folder of the old distribution to the new location. Keep the original
das-data information as a backup.
cp -r /<old-location>/das-data /<new-location>/
Users need to check, and update if necessary, the folder where plug-in configurations are stored.
Open the file system of the previous Datameer version you are upgrading from and navigate to the
Search for the property file that defines the folder where plug-in configurations are stored.
# Defines the folder where to store plugin configurations
Open the file system of the upgrade version of Datameer and find the same property in the
Check or update the new property value to be the same folder name in the previous version.
Copy the contents of the previous plug-in configurations folder into the file system of the new upgraded version of Datameer.
conf/directory previously (like changing
log4j-<xyz>.properties), you need to apply** the same changes to the new
conf/directory of your upgraded instance.
Copy over the native libraries that you have added to Datameer (if this applies). You don't have to copy over the native libraries that are already bundled with Datameer.
cp -r /<old-location>/lib/native/* /<new-location>/lib/native/
conductor.shscript (for example, to enable SSH), apply** these changes to
/<new-location>/bin/conductor.shagain. Notice JAVA_OPTIONS has been moved to
Migrate the SSL values from the previous
start.ini to the new one.
# Set up a demonstration keystore and truststore
# Sets the demonstration passwords
etc/custom-jars(these files could be database drivers or 3rd party libraries).
etc/custom-plugins), update those to be in sync (API compatible) with the new version of Datameer and be sure to remove jar files related to older versions from
Do not copy the entire old
Follow the steps for upgrading your MySQL or HSQL database. As of Datameer 7.4: MariaDB is supported as an alternative to MySQL.
To upgrade the database, enter the following command:
If either the structure of the MySQL database or any of the tables in the database will be modified by the Datameer upgrade, running this command automatically creates a dump of the database.
If your database name is different than the default
dap or the database runs on a different server or customized port, you will need to provide the additional arguments:
bin/upgrade_db.sh -h [db hostname] -o [db port] -n [db name] -u [db username] -p [db password]
Check to ensure your database character encoding is set to UTF-8.
In the Database, enter:
SELECT DEFAULT_CHARACTER_SET_NAME FROM INFORMATION_SCHEMA.SCHEMATA WHERE SCHEMA_NAME = '<The name of your Datameer database>';
If UTF8 was returned, open the default properties and comment in the setting
If something other than UTF8 was returned, change the database character encoding setting to UTF8 or special characters may show as corrupt.
To finish the upgrade, restart the application.
If you receive the above error message during an update, raise the MySQL stack size. For example,
mysqld -O thread_stack=192k
or higher if necessary.
Datameer provides a tool to migrate data from an HSQL file database to a MySQL database:
Migrate HSQL database to MySQL
bin/migrate-db-tool.sh hsql-file:<Datameer user path>/Datameer/<version>/das-data/database mysql
* Ensure this script is being run from root Datameer installation folder.
Ensure your users clear their browser caches after implementing a Datameer upgrade.
You have to use the script
bin/upgrade_hsql_db.sh to upgrade your HSQL database to a higher version. Ensure this script is run from the root Datameer installation folder.
usage: upgrade_hsql_db.sh -ndd,--new-das-data <folder> new das-data folder -odd,--old-das-data <folder> old das-data folder
-pi option you define the previous datameer installation. We need this folder to find the HSQL database we want to upgrade.
We backup your previous database into your current installation folder. Furthermore we use all upgrade scripts with name pattern
upgrade-<version>.hsql in your current installation folder.
As an option you can use the parameters
-odd . These parameters define the new/old location of your
das-data folder. When you running datameer in LOCAL mode it could be possible that you want to copy your
das-data folder to a new location. A common use case is when the
das-data folder is saved within the installation directory.
Here is an example of what it looks like when upgrading a database.
bin/upgrade_hsql_db.sh --old-das-data <file path>/<file version>/das-data --new-das-data <file path>/<file version>/das-data
#Detect databases detect old database [/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/database/hsql-db] detect new database [/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/database/hsql-db] Upgrade datameer database version [188.8.131.52] to version [184.108.40.206] #Assert datameer is running No #Copy Das Data copy das-data from [/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data] to [/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data] #Upgrade starts... Process upgrade scripts on database jdbc:hsqldb:file:das-data/database/hsql-db upgrade-2.0.1.hsql #Update schema-version set system.schemaVersion on database [/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/database/hsql-db] to [2.0.1] #Update data uri's [/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data] -> [/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data] log4j:WARN No appenders could be found for logger (org.hibernate.type.BasicTypeRegistry). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. Converting 12 data entries file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/fileuploads/1/1 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/fileuploads/1/1 file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/fileuploads/2/2 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/fileuploads/2/2 file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/fileuploads/3/3 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/fileuploads/3/3 file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/fileuploads/4/4 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/fileuploads/4/4 file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/fileuploads/5/5 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/fileuploads/5/5 file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/workbooks/6/6 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/workbooks/6/6 file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/workbooks/7/7 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/workbooks/7/7 file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/workbooks/8/8 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/workbooks/8/8 file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/workbooks/9/9 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/workbooks/9/9 file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/fileuploads/10/10 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/fileuploads/10/10 file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/workbooks/11/11 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/workbooks/11/11 file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/fileuploads/10/12 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/fileuploads/10/12
Check or update YARN application classpath and LD_LIBRARY path. The path settings are used to lookup and load libraries , therefor incorrect path settings may lead into unexpected behavior .
Open the Administration tab
Open the Hadoop Cluster options
Check or update YARN application classpath and LD_LIBRARY path
When upgrading from CDH 4.x to CDH 5.x the classpath was renamed.
The correct YARN classpath can be found for the Datameer client via command:
The search settings to enable file search within the 'File Browser' must be retriggered manually after the upgrade.
Custom plug-Ins which were created by using the plug-in SDK need to be recompiled. Adjust the versioning and create the plug-in again under the new plug-in SDK.
All plug-ins used that aren't a part of standard distribution, e.g., Advanced Governance package, require an upgrade.
After the upgrade has completed, preform a test to validate that the upgrade was successful.
If no unexpected errors occur the upgrade has been validated.
The support for MapReduce has been deprecated and was removed in Datameer 6.3. After it is verified that all jobs work without the MapReduce framework, remove MapReduce related custom properties from the Hadoop cluster page under the Administration tab and in the advanced settings of data links, import jobs, and/or workbooks.
When upgrading, a database dump is created by Datameer, the file name is similar to
mysql_db_dump_<oldVer>.sql. If an upgrade failed (failing upgrade scripts / failing upgrade injectors during startup), the database paths might be conflicting, it is therefore necessary to restore the Datameer MySQL database dump. Follow the steps to complete this task.
Drop the new database, for example on the database server or the MySQL CLI.
DROP DATABASE dap;
mysql-init.sql script, create a new database.
mysql [-h <dbhost>] -u root -p < bin/mysql-init.sql
Import and restore the previous database using the following shell command.
mysql [-h <dbhost>] -u root -p dap < /path/to/dump_file_name