Optimize the parallel analysis spark
Budget: $30 – $250 USD
Make spark more parallel
our goal is to propose a new parallel performance model for different workloads of Spark Big Data applications running on HPC clusters.
We need to add parallel technique on the rdd, to be parallelized. Then the execution will be faster. After we add parallel technique on RDDs, then we need to call openAcc and MPI pragmas to execute these parallel RDDs on GPU, we will call MPI and openAcc pragmas by a wrapper.
For now we need to create a new parallel algorithm (foreach,for loops …etc) and apply it on the RDDs, but we need to make sure these new techniques are equivalent or appropriate to parallelization of MPI and OpenACC programming model.
using Scala Language
Do you have good experience of improving the parallel analysis of spark?
our goal is to propose a new parallel performance model for different workloads of Spark Big Data applications running on HPC clusters.
We need to add parallel technique on the rdd, to be parallelized. Then the execution will be faster. After we add parallel technique on RDDs, then we need to call openAcc and MPI pragmas to execute these parallel RDDs on GPU, we will call MPI and openAcc pragmas by a wrapper.
For now we need to create a new parallel algorithm (foreach,for loops …etc) and apply it on the RDDs, but we need to make sure these new techniques are equivalent or appropriate to parallelization of MPI and OpenACC programming model.
using Scala Language
Do you have good experience of improving the parallel analysis of spark?