Page tree
Skip to end of metadata
Go to start of metadata

This tuning guide provides examples showing specific recommendations for tuning for different Hadoop cluster configurations and workloads. Use the guide to adjust your settings based on your cluster and workload characteristics. For a more in-depth explanation of these settings, refer to the Spark Tuning Concepts guide.

Caveats

This guide assumes the YARN cluster has been properly configured for running with more than 1 vcore per YARN container. By default, YARN ignores vcores when performing resource calculations and only takes into account memory usage. Datameer still tracks the amount of vcores Spark is actually using, but YARN reports the each Spark Executor uses 1 vcore. This method might lead to problems on the cluster if other applications or systems are running jobs that require exact vcore allocation or if the cluster has been configured to use CGroups. If exact vcore allocation is required then the DominantResourceCalculator should be enabled on the cluster's ResourceManager, since the DefaultResourceCalculator only takes into account memory resources. In most cases, all the cluster jobs run without problems and use as many CPU(s) as configured, but YARN doesn't report it. Here is an example configuration for the DominantResourceCalculator. During Datameer's testing of Spark we ran with both the DefaultResourceCalculator and DominantResourceCalculator, and in both cases the Spark jobs ran exactly the same.

Small Cluster

Example environment

This environment is an example, and does not cover all possible environments. Make sure you are adjusting your settings based on your particular environment, not adapting these settings as a one-to-one recommendation.

Our example environment uses the following setup:

  • One master node

  • Four worker nodes with:

    • 20 GB memory

    • Four vcores

Custom properties

The following custom properties are set:

das.execution-framework=SparkClient
das.spark.context.auto-scale-enabled=false
das.map-tasks-per-node=4
das.spark.context.max-idle-time=10m
spark.executor.cores=2
spark.executor.memory=9g
spark.executor.instances=7

Use this table to understand the settings. 

Property

Reason

das.execution-framework=SparkClient

With this configuration's small cluster, SparkSX is unnecessary.

das.spark.context.auto-scale-enabled=false

Auto scaling is unnecessary because other applications don't need free resources.

das.map-tasks-per-node=4

The map tasks per node should be set to the available vcores of a node on the cluster.

das.spark.context.max-idle-time=10m

Increasing the idle time leads fewer startup costs.

spark.executor.cores=2

Use two vcores per container.

spark.executor.memory=9g

(Memory per node / max number of instances per node) * 0.9

E.g., ( 20 / 2 ) * 0.9 = 9g

The memory per node is 20GB, divided by the two instances per node specified, multiplied by 90%, giving the final value of 9g to set for the property spark.executor.memory.

spark.executor.instances=7

Four nodes multiplied by the space for two instances per node, minus one AM instance.

In this case SparkClient should be used as the execution framework, and it should use most of the cluster all of the time which leaves some container slots available for AM(s) and other processes.

Cluster with Mixed Workloads

Example environment

This environment is an example, and does not cover all possible environments. Make sure you are adjusting your settings based on your particular environment, not adapting these settings as a one-to-one recommendation.

Our example environment uses the following setup:

  • One master node

  • 10 worker nodes with:

    • 56GB memory

    • 12 vcores

  • Mix of small and larger workloads

Custom properties

The following custom properties are set:

das.execution-framework=SparkSX
spark.executor.cores=2
spark.executor.memory=8g
das.map-tasks-per-node=12
datameer.yarn.available-node-memory=56g
datameer.yarn.available-node-vcores=12

Use this table to understand the settings.

Property

Reason

das.execution-framework=SparkSX

SparkSX is the best option for clusters with mixed workloads.

das.map-tasks-per-node=12

For Tez the map tasks per node should be the number of available vcores per node.

spark.executor.cores=2

Up to six instances per node.

12 vcores per node/2 instances per node= 6 vcores per instance.

Due to auto-scaling, you don't need to leave a vcore for the AM instance, as auto-scaling uses one instance fewer to account for the AM.

spark.executor.memory=8g

56g per node/6 instances per node = 9g per instance.

The 9g value includes 10% overhead, so the setting should be 8g to account for that necessary overhead.

datameer.yarn.available-node-memory=56g

Available memory on each cluster node.

datameer.yarn.available-node-vcores=12

Available vcores on each cluster node.

This configuration allows each SparkCluster job to process up to 60 tasks concurrently. The configuration also leaves open some memory and vcore resources for AM containers and other processes.

Cluster with Mixed Workloads (v6.3 and above)

The following settings are based on version 6.3 of Datameer.

Example environment

This environment is an example, and does not cover all possible environments. Make sure you are adjusting your settings based on your particular environment, not adapting these settings as a one-to-one recommendation.

Our example environment uses the following setup:

  • 1 master node

  • 10 worker nodes with:

    • 56GB memory

    • 12 vcores

  • Mix of small and larger workloads

Custom Properties

The following custom properties are set:

das.execution-framework=SparkSX
spark.executor.cores=2
spark.executor.memory=auto
das.yarn.available-node-memory=auto
das.yarn.available-node-vcores=auto

Use this table to understand the settings.

Property

Reason

das.execution-framework=SparkSX

SparkSX is the best option for clusters with mixed workloads

spark.executor.cores=2

Up to 6 instances per node

12 vcores per node/2 instances per node= 6 vcores per instance

Due to auto-scaling, you don't need to leave a vcore for the AM instance, as auto-scaling uses one instance fewer to account for the AM

spark.executor.memory=auto

Automatically calculates the amount of memory required to perform tasks

das.yarn.available-node-memory=auto

Automatically calculates available memory on each cluster mode


das.yarn.available-node-vcores=auto

Automatically calculates available vcores on each cluster mode

This configuration allows each SparkCluster job to process up to 60 tasks concurrently. The configuration also leaves open some memory and vcore resources for AM containers and other processes.

  • No labels