Memory Management in Spark



Exception of memory management in spark

  1. Container is running below the memory limits
  2. Using all resources in apache spark with yarn
  3. Spark -- Container is running beyond physical memory limits
  4. Spark streaming on yarn -- Container is running beyond physical memory limit
  5. Executor getting lost, yarn container killing it due to memory limit.


1- Yarn memory policing

a) yarn.nodemanager.pmem-check-enabled=false  

    But recommended Not to set false, since it will prevent NameNode to keep control over the containers, if running out of physical memory in the container makes sure that the JVM inside the container is small enough to fit in the container
The Container size is large enough to containe

    • JVM Heap
    •  pem-gen(permanent generation) of the JVM
    • any off heap application

yarn.nodemanager.pmem-check-enabled  -- This is for the physical memory

          yarn.nodemanager.vmem-check-enabled  -- This is for the  virtual memory

          These two can be configured in  yarn-site.xml of Node Manager

          b) Executor memory overhead
spark.yarn.executor.memoryOverhead=315

The two main resources that Spark (and YARN) think about are CPU and memory. Disk and network I/O, of course, play a part in Spark performance as well, but neither Spark nor YARN currently do anything to actively manage them.

Cores in Executor

Every Spark executor in an application has the same fixed number of cores and same fixed heap size. The number of cores can be specified with the --executor-cores flag when invoking spark-submit, spark-shell, and pyspark from the command line, or by setting the spark.executor.cores property in the spark-defaults.conf file or on a SparkConf object. Similarly, the heap size can be controlled with the --executor-memory flag or the spark.executor.memory property. The cores property controls the number of concurrent tasks an executor can run. --executor-cores 5 means that each executor can run a maximum of five tasks at the same time. The memory property impacts the amount of data Spark can cache, as well as the maximum sizes of the shuffle data structures used for grouping, aggregations, and joins.

Number of Executor

The --num-executors command-line flag or spark.executor.instances configuration property control the number of executors requested. Starting in CDH 5.4/Spark 1.3, you will be able to avoid setting this property by turning on dynamic allocation with the spark.dynamicAllocation.enabled property. Dynamic allocation enables a Spark application to request executors when there is a backlog of pending tasks and free up executors when idle.



It’s also important to think about how the resources requested by Spark will fit into what YARN has available. The relevant YARN properties are:

yarn.nodemanager.resource.memory-mb controls the maximum sum of memory used by the containers on each node.
yarn.nodemanager.resource.cpu-vcores controls the maximum sum of cores used by the containers on each node


Asking for five executor cores will result in a request to YARN for five virtual cores. The memory requested from YARN is a little more complex for a couple reasons:

--executor-memory/spark.executor.memory controls the executor heap size, but JVMs can also use some memory off heap, for example for interned Strings and direct byte buffers. The value of the

spark.yarn.executor.memoryOverhead property is added to the executor memory to determine the full memory request to YARN for each executor. It defaults to max(384, .07 * spark.executor.memory).
YARN may round the requested memory up a little. YARN’s yarn.scheduler.minimum-allocation-mb and

yarn.scheduler.increment-allocation-mb properties control the minimum and increment request values respectively.

Real Time Example-



Imagine a cluster with
six nodes running NodeManagers, each equipped with 16 cores and 64GB of memory.

yarn.nodemanager.resource.memory-mb and yarn.nodemanager.resource.cpu-vcores, should probably be set to 63 * 1024 = 64512 (megabytes)  -- controls the maximum sum of memory used by the containers on each node.
leave 1 GB for hadoop deamons and other os related thread

First Approach
--num-executors 6 --executor-cores 15 --executor-memory 63G
    But this is wrong
    63GB + the executor memory overhead won’t fit within the 63GB capacity of the NodeManagers.
    The application master will take up a core on one of the nodes, meaning that there won’t be room for a 15-core executor on that node.
    15 cores per executor can lead to bad HDFS I/O throughput.

    so there is only problem is core

Second approach
--num-executors 17 --executor-cores 5 --executor-memory 19G
    This config results in three executors on all nodes except for the one with the AM, which will have two executors.
--executor-memory was derived as (63/3 executors per node) = 21.  21 * 0.07 = 1.47.  21 – 1.47 ~ 19.



Comments

Post a Comment