Increase Spark Driver memory using PySpark session from EMR Notebooks

Written by mannem on October 16, 2019. Posted in AWS BIG DATA, EMR || Elastic Map Reduce

%%configure -f
{"driverMemory":"5G","executorMemory": "10G", "executorCores": 4}

Current session configs: {'driverMemory': '5G', 'executorMemory': '10G', 'executorCores': 4, 'kind': 'pyspark'}

%%configure -f

{"driverMemory":"5G","executorMemory": "10G", "executorCores": 4}

Current session configs: {'driverMemory': '5G', 'executorMemory': '10G', 'executorCores': 4, 'kind': 'pyspark'}

Tags: driver, emr, jupyter, memory, notebook, spark

Trackback from your site.

M	T	W	T	F	S	S
« Jan
						1
2	3	4	5	6	7	8
9	10	11	12	13	14	15
16	17	18	19	20	21	22
23	24	25	26	27	28	29
30	31

Leave a comment