Increase Spark Driver memory using PySpark session from EMR Notebooks

Written by mannem on October 16, 2019. Posted in AWS BIG DATA, EMR || Elastic Map Reduce

%%configure -f
{"driverMemory":"5G","executorMemory": "10G", "executorCores": 4}

Current session configs: {'driverMemory': '5G', 'executorMemory': '10G', 'executorCores': 4, 'kind': 'pyspark'}

%%configure -f

{"driverMemory":"5G","executorMemory": "10G", "executorCores": 4}

Current session configs: {'driverMemory': '5G', 'executorMemory': '10G', 'executorCores': 4, 'kind': 'pyspark'}

Tags: driver, emr, jupyter, memory, notebook, spark

Trackback from your site.

M	T	W	T	F	S	S
« Jan
					1	2
3	4	5	6	7	8	9
10	11	12	13	14	15	16
17	18	19	20	21	22	23
24	25	26	27	28	29

Leave a comment