scala - memory error in standalone spark cluster as "shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[Remote]" -

i got following memory error in standalone spark cluster, after 140 iterations of code. how shall run code without memory fault?

i having 7 nodes 8gb ram out of 6gb allocated workers. master having 8gb ram.

[error] application - remote calculator (actor[akka.tcp://remote@127.0.0.1:44545/remote/akka.tcp/notebookserver@127.0.0.1:50778/user/$c/$a#872469007]) has been terminated !!!!! [info] application - view notebook 'kamaruddin/psoaann_breastcancer_optimized.snb', presentation: 'none' [info] application - closing websockets kernel 6c8e8090-cbeb-430e-9d45-5710ce60b984 uncaught error thread [remote-akka.actor.default-dispatcher-6] shutting down jvm since 'akka.jvm-exit-on-fatal-error' enabled actorsystem[remote] exception in thread "thread-36" java.lang.outofmemoryerror: gc overhead limit exceeded     @ java.util.jar.attributes.read(attributes.java:394)     @ java.util.jar.manifest.read(manifest.java:199)     @ java.util.jar.manifest.<init>(manifest.java:69)     @ java.util.jar.jarfile.getmanifestfromreference(jarfile.java:186)     @ java.util.jar.jarfile.getmanifest(jarfile.java:167)     @ sun.misc.urlclasspath$jarloader$2.getmanifest(urlclasspath.java:779)     @ java.net.urlclassloader.defineclass(urlclassloader.java:416)     @ java.net.urlclassloader.access$100(urlclassloader.java:71)     @ java.net.urlclassloader$1.run(urlclassloader.java:361)     @ java.net.urlclassloader$1.run(urlclassloader.java:355)     @ java.security.accesscontroller.doprivileged(native method)     @ java.net.urlclassloader.findclass(urlclassloader.java:354)     @ java.lang.classloader.loadclass(classloader.java:425)     @ sun.misc.launcher$appclassloader.loadclass(launcher.java:308)     @ java.lang.classloader.loadclass(classloader.java:358)     @ org.apache.spark.repl.sparkimain$readevalprint.binderror(sparkimain.scala:1041)     @ org.apache.spark.repl.sparkimain$request.loadandrun(sparkimain.scala:1347)     @ org.apache.spark.repl.sparkimain.loadandrunreq$1(sparkimain.scala:840)     @ org.apache.spark.repl.sparkimain.interpret(sparkimain.scala:871)     @ org.apache.spark.repl.sparkimain.interpret(sparkimain.scala:819)     @ notebook.kernel.repl$$anonfun$3.apply(repl.scala:173)     @ notebook.kernel.repl$$anonfun$3.apply(repl.scala:173)     @ scala.util.dynamicvariable.withvalue(dynamicvariable.scala:57)     @ scala.console$.without(console.scala:126)     @ notebook.kernel.repl.evaluate(repl.scala:172)     @ notebook.client.replcalculator$$anonfun$10$$anon$1$$anonfun$24.apply(replcalculator.scala:364)     @ notebook.client.replcalculator$$anonfun$10$$anon$1$$anonfun$24.apply(replcalculator.scala:361)     @ scala.concurrent.impl.future$promisecompletingrunnable.liftedtree1$1(future.scala:24)     @ scala.concurrent.impl.future$promisecompletingrunnable.run(future.scala:24)     @ akka.dispatch.taskinvocation.run(abstractdispatcher.scala:40)     @ akka.dispatch.forkjoinexecutorconfigurator$akkaforkjointask.exec(abstractdispatcher.scala:397)     @ scala.concurrent.forkjoin.forkjointask.doexec(forkjointask.java:260) uncaught error thread [remote-akka.remote.default-remote-dispatcher-445] shutting down jvm since 'akka.jvm-exit-on-fatal-error' enabled actorsystem[remote] java.lang.outofmemoryerror: gc overhead limit exceeded java.lang.outofmemoryerror: gc overhead limit exceeded     @ java.util.arrays.copyof(arrays.java:2367)     @ java.lang.abstractstringbuilder.expandcapacity(abstractstringbuilder.java:130)     @ java.lang.abstractstringbuilder.ensurecapacityinternal(abstractstringbuilder.java:114)     @ java.lang.abstractstringbuilder.append(abstractstringbuilder.java:535)     @ java.lang.stringbuffer.append(stringbuffer.java:322)     @ java.io.stringwriter.write(stringwriter.java:94)     @ com.fasterxml.jackson.core.json.writerbasedjsongenerator._flushbuffer(writerbasedjsongenerator.java:1879)     @ com.fasterxml.jackson.core.json.writerbasedjsongenerator._writestring(writerbasedjsongenerator.java:916)     @ com.fasterxml.jackson.core.json.writerbasedjsongenerator._writefieldname(writerbasedjsongenerator.java:213)     @ com.fasterxml.jackson.core.json.writerbasedjsongenerator.writefieldname(writerbasedjsongenerator.java:104)     @ play.api.libs.json.jsvalueserializer$$anonfun$serialize$2.apply(jsvalue.scala:319)     @ play.api.libs.json.jsvalueserializer$$anonfun$serialize$2.apply(jsvalue.scala:318)     @ scala.collection.mutable.resizablearray$class.foreach(resizablearray.scala:59)     @ scala.collection.mutable.arraybuffer.foreach(arraybuffer.scala:47)     @ play.api.libs.json.jsvalueserializer.serialize(jsvalue.scala:318)     @ play.api.libs.json.jsvalueserializer$$anonfun$serialize$1.apply(jsvalue.scala:312)     @ play.api.libs.json.jsvalueserializer$$anonfun$serialize$1.apply(jsvalue.scala:311)     @ scala.collection.immutable.list.foreach(list.scala:318)     @ play.api.libs.json.jsvalueserializer.serialize(jsvalue.scala:311)     @ play.api.libs.json.jsvalueserializer$$anonfun$serialize$2.apply(jsvalue.scala:320)     @ play.api.libs.json.jsvalueserializer$$anonfun$serialize$2.apply(jsvalue.scala:318)     @ scala.collection.immutable.list.foreach(list.scala:318)     @ play.api.libs.json.jsvalueserializer.serialize(jsvalue.scala:318)     @ play.api.libs.json.jsvalueserializer.serialize(jsvalue.scala:302)     @ com.fasterxml.jackson.databind.ser.defaultserializerprovider.serializevalue(defaultserializerprovider.java:128)     @ com.fasterxml.jackson.databind.objectmapper.writevalue(objectmapper.java:1902)     @ play.api.libs.json.jacksonjson$.generatefromjsvalue(jsvalue.scala:494)     @ play.api.libs.json.json$.stringify(json.scala:51)     @ play.api.libs.json.jsvalue$class.tostring(jsvalue.scala:80)     @ play.api.libs.json.jsobject.tostring(jsvalue.scala:166)     @ java.util.formatter$formatspecifier.printstring(formatter.java:2838)     @ java.util.formatter$formatspecifier.print(formatter.java:2718) uncaught error thread [remote-akka.remote.default-remote-dispatcher-446] shutting down jvm since 'akka.jvm-exit-on-fatal-error' enabled actorsystem[remote] java.lang.outofmemoryerror: gc overhead limit exceeded exception in thread "appclient-receive-and-reply-threadpool-0" java.lang.outofmemoryerror: gc overhead limit exceeded exception in thread "appclient-receive-and-reply-threadpool-2" java.lang.outofmemoryerror: gc overhead limit exceeded exception in thread "appclient-receive-and-reply-threadpool-4" java.lang.outofmemoryerror: gc overhead limit exceeded exception in thread "appclient-receive-and-reply-threadpool-6" java.lang.outofmemoryerror: gc overhead limit exceeded [error] application - process exited error: 255 (exit value: 255) org.apache.commons.exec.executeexception: process exited error: 255 (exit value: 255)     @ org.apache.commons.exec.defaultexecutor.executeinternal(defaultexecutor.java:404)     @ org.apache.commons.exec.defaultexecutor.access$200(defaultexecutor.java:48)     @ org.apache.commons.exec.defaultexecutor$1.run(defaultexecutor.java:200)     @ java.lang.thread.run(thread.java:745) 

maybe can try use checkpointing.

data checkpointing - saving of generated rdds reliable storage. necessary in stateful transformations combine data across multiple batches. in such transformations, generated rdds depend on rdds of previous batches, causes length of dependency chain keep increasing time. avoid such unbounded increases in recovery time (proportional dependency chain), intermediate rdds of stateful transformations periodically checkpointed reliable storage (e.g. hdfs) to cut off dependency chain