Page tree
Skip to end of metadata
Go to start of metadata

Check out video on Exporting Data.

 

You can create export jobs to export data from Datameer to other connections such as a database, remote file server, or export results to a third-party BI (business intelligence) software package. Exported data are the results from executing analyses in a workbook on the original data set. You can initiate a one-time manual export, or you can configure the job to run each time the workbook is updated or at a specific time interval.

Introduction

Before you can export data, you must have created a workbook. Once the workbook has executed you can then export data from one of the saved sheets in that workbook.

Exporting Data is only available in Datameer's Enterprise product.

File Formats Available

Choose to export a worksheet to one of the following file formats:

  • Apache AVRO
  • CSV
  • Parquet
  • Tableau (TDSX)

 

Exporting data

Only the data from a saved workbook sheet are available for export.

To export data:

  1. Choose Export Data from the + drop down menu located in the top left corner of the screen or right click in the navigation bar are the left side of the screen and select Create New > Export Data.
  2. Select a workbook by clicking the Select button then click the Next button.
  3. Choose which sheet to export, then click Next. At the bottom of the page you will see a preview of the a sheet's data.
  4. Choose which connection the data should be exported to, either an existing connection or create a new one. Choose to export the data as a CSV file, an AVRO file, a Parquet file, or Tableau (TDSX) file. Then click Next.
  5. Enter the information on the data details tab which varies depending on the type of file you are creating. Then click Next.
    1. Under File settings, specify the file name and file path as well as whether existing data should be replaced.
      Under Advanced Settings, specify the maximum file size and how consecutive files should be numbered. All export jobs will add a task number to the end of the filenames (before the extension) generated by the export job.
       
    2. If the file is being exported as a CSV there will be additional settings. Indicate whether column headers should be contained in the first row, and how delimiter, quote, and escape characters should be defined.
    3. If the file is being exported to a database, choose to create a new table for the workbook data or select the table from the drop down menu. Note: The drop down box under Data Details has a hard limit of 1,000 entries.
       
       
  6. Specify which fields to include. Uncheck any fields that should be excluded. Specify whether empty fields can be accepted. 

    Optionally, if there is a  date field  included in the data, you can change the parse pattern format to be used during export. The default date parse pattern for an export job is [yyyy-MM-dd HH:mm:ss].

  7. Specify how to handle invalid records, and click Next.
  8. Determine the schedule for exporting the data. Choose Manually for a non-recurrent export, or you can choose either After the workbook is calculated or On a Schedule. If you choose On a Schedule, indicate the time settings or use a custom cron pattern. Under Advanced Settings, you can enter custom properties as key/value pairs. Then click Next.
  9. Enter a description if desired, click Save As, give the file a name, then click Save. You can also enter an email address to receive any error messages.

Exporting to a database

When exporting to a database, the database connector will be associated with a user name. This database user must have write and/or create permissions on the database itself in order to complete the export job.

Please note that when exporting data to DB2 or to Oracle connection, there are limitations on the number of characters contained in string data field.

  • for DB2 the number of characters is limited to 2000
  • for Oracle the number of characters is limited to 4000

Attempting to export data fields containing a larger number of characters will result in dropped records.

 

Data Details - Advanced Settings

As of Datameer version 5.7 - Limit maximum database parallelism during exports

  • Setting the number of concurrent data base connections gives control of database parallelism. The Export Job will not launch more than the set number of parallel tasks. Each task opens one database connection.
  • Setting the maximum records per transaction will export approximately this number of records within a single database transaction.
  • Setting the rows per batch will establish a limit for the number of rows inserted for each transaction within a task. Tuning this value can influence speed and avoid potential timeouts. Very large batches can overwhelm some databases and limiting the number of rows returned at a time can reduce strain.

Editing export job settings

Only the data from a saved workbook sheet are available for export.

To edit export job settings:

  1. Click on Export Data in the navigation bar on the left side of the screen.
  2. Highlight by clicking on the Export Job you wish to edit.
  3. Right click on the Export Job and select edit or simply click on the edit button in the toolbar at the top of the screen.
  4. Make the necessary changes (see exporting data for more information.) Use the Next button to navigate through the individual configuration screens.
  5. Click Save after you have finished or Save Copy As if you want to save the changed settings under a new name.

Duplicate export jobs

To create a copy of an existing export job:

  1. Click on Export Data in the navigation bar on the left side of the screen.
  2. Highlight by clicking on the Export Job you wish to duplicate.
  3. Right click on the Export Job and select duplicate or simply click on the duplicate button in the toolbar at the top of the screen.
  4. Click the create duplicate button.

A duplicate is created and is named "copy of ..." and the name of the original export job.

Running an export job

To run an export job:

  1. Click on Export Data in the navigation bar on the left side of the screen.
  2. Highlight by clicking on the Export Job you wish to run.
  3. Right click on the Export Job and select run or simply click on the run button in the toolbar at the top of the screen.

Depending on the amount of data, this process might take awhile.

Export jobs run as a single task if the source sheet is sorted. The sorted sheet can be generated in parallel in the workbook, but the export job forces the job to run as a single task. To get around this, create a filter on the primary export sheet restricting it to a specific range and select the option to create this filter in a new sheet. Do this multiple times until all of the data is filtered out into separate sheets. Once done, run export jobs against these newly created sheets.

Deleting an export job

Only the export job is deleted, not the original data.

To delete an export job:

  1. Click on Export Data in the navigation bar on the left side of the screen.
  2. Highlight by clicking on the Export Job you wish to delete.
  3. Right click on the Export Job and select delete or simply click on the delete button in the toolbar at the top of the screen.
  4. Click the Delete button and confirm that you would like to perform this action.

Editing export job permissions

To edit an export job's permissions

  1. Click on Export Data in the navigation window on the left side of the screen.
  2. Click to highlight the Export Job for which you wish to edit permissions.
  3. Right click the Export Job and select Information or simply click the Information button on the toolbar.
  4. The permissions setting is listed at the bottom of the Sharing section of the information page.
  5. To add group permissions, click Add Group and select each group you want to add. Then set the read, write, and run (execute) permissions for that group. To delete a group, click the trash can icon next to that group.
  6. Set the results sharing permissions for each group and all others.
  • No labels