Increase Spark Driver memory using PySpark session from EMR Notebooks

Written by mannem on October 16, 2019. Posted in AWS BIG DATA, EMR || Elastic Map Reduce

%%configure -f
{"driverMemory":"5G","executorMemory": "10G", "executorCores": 4}

Current session configs: {'driverMemory': '5G', 'executorMemory': '10G', 'executorCores': 4, 'kind': 'pyspark'}

%%configure -f

{"driverMemory":"5G","executorMemory": "10G", "executorCores": 4}

Current session configs: {'driverMemory': '5G', 'executorMemory': '10G', 'executorCores': 4, 'kind': 'pyspark'}

Tags: driver, emr, jupyter, memory, notebook, spark

Trackback from your site.

M	T	W	T	F	S	S
« Jan
		1	2	3	4	5
6	7	8	9	10	11	12
13	14	15	16	17	18	19
20	21	22	23	24	25	26
27	28	29	30	31

Leave a comment