Increase Spark Driver memory using PySpark session from EMR Notebooks

Written by mannem on October 16, 2019. Posted in AWS BIG DATA, EMR || Elastic Map Reduce

%%configure -f
{"driverMemory":"5G","executorMemory": "10G", "executorCores": 4}

Current session configs: {'driverMemory': '5G', 'executorMemory': '10G', 'executorCores': 4, 'kind': 'pyspark'}

%%configure -f

{"driverMemory":"5G","executorMemory": "10G", "executorCores": 4}

Current session configs: {'driverMemory': '5G', 'executorMemory': '10G', 'executorCores': 4, 'kind': 'pyspark'}

Tags: driver, emr, jupyter, memory, notebook, spark

Trackback from your site.

M	T	W	T	F	S	S
« Jan
				1	2	3
4	5	6	7	8	9	10
11	12	13	14	15	16	17
18	19	20	21	22	23	24
25	26	27	28	29	30	31

Leave a comment