Increase Spark Driver memory using PySpark session from EMR Notebooks

Written by mannem on October 16, 2019. Posted in AWS BIG DATA, EMR || Elastic Map Reduce

%%configure -f
{"driverMemory":"5G","executorMemory": "10G", "executorCores": 4}

Current session configs: {'driverMemory': '5G', 'executorMemory': '10G', 'executorCores': 4, 'kind': 'pyspark'}

%%configure -f

{"driverMemory":"5G","executorMemory": "10G", "executorCores": 4}

Current session configs: {'driverMemory': '5G', 'executorMemory': '10G', 'executorCores': 4, 'kind': 'pyspark'}

Tags: driver, emr, jupyter, memory, notebook, spark

Trackback from your site.

M	T	W	T	F	S	S
« Jan
1	2	3	4	5	6	7
8	9	10	11	12	13	14
15	16	17	18	19	20	21
22	23	24	25	26	27	28
29	30

Leave a comment