AS
Nikolay
Привет. Спарк на диск скидывает данные, если выполняется группировка ?
в документации:
http://spark.apache.org/docs/latest/rdd-programming-guide.html
в секции про shuffle (groupBy ведет к shuffle)
The Shuffle is an expensive operation since it involves disk I/O, data serialization, and network I/O. To organize data for the shuffle, Spark generates sets of tasks - map tasks to organize the data, and a set of reduce tasks to aggregate it. This nomenclature comes from MapReduce and does not directly relate to Spark’s map and reduce operations.
Internally, results from individual map tasks are kept in memory until they can’t fit. Then, these are sorted based on the target partition and written to a single file. On the reduce side, tasks read the relevant sorted blocks.
я так понимаю если памяти хватает, то не обязательно будет запись на диск
http://spark.apache.org/docs/latest/rdd-programming-guide.html
в секции про shuffle (groupBy ведет к shuffle)
The Shuffle is an expensive operation since it involves disk I/O, data serialization, and network I/O. To organize data for the shuffle, Spark generates sets of tasks - map tasks to organize the data, and a set of reduce tasks to aggregate it. This nomenclature comes from MapReduce and does not directly relate to Spark’s map and reduce operations.
Internally, results from individual map tasks are kept in memory until they can’t fit. Then, these are sorted based on the target partition and written to a single file. On the reduce side, tasks read the relevant sorted blocks.
я так понимаю если памяти хватает, то не обязательно будет запись на диск
