Page tree

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: DAP-20336, adjust minimal heap size requirement

...

Name

Value

Description

Location

mapred.map.child.java.opts
mapreduce.reduce.java.opts
(hadoop-0.21, cdh3b3 only)

-Xmx512M Xmx1024M or more

By default, Datameer is configured to work with a minimum 512MB 1024 MB heap. Based on your slot configuration, you may have significantly more memory available per JVM

mapred-site.xml

mapred.child.java.opts
(<=hadoop-0.21)

-Xmx512M Xmx1024M or more

By default, Datameer is configured to work with a minimum 512MB 1024 MB heap. Based on your slot configuration, you may have significantly more memory available per JVM

mapred-site.xml

mapred.tasktracker.dns.interface
dfs.datanode.dns.interface

 

Hadoop nodes often have multiple network interfaces (internal vs. external). Explicity choosing an interface can avoid problems.

 

java.net.preferIPv4Stack

true

Disabling Ipv6 in Java can resolve problems and improve throughput. Set this property to ensure Ipv6 is not used.

This can be configured in $HADOOP_OPTS environment variable on your Hadoop cluster.

mapreduce.jobtracker.staging.root.dir

/user

Avoids permission problems when a user other than superuser schedules jobs. (hadoop-0.21, cdh3b3 only)

mapred-site.xml (on the cluster site - does not work if configured as a custom property in Datameer)

...