Optimize the parallel analysis spark
Budget: $250 – $750 USD
Make spark more parallel
proposes a new parallel performance model for different workloads of Spark Big Data applications running on HPC clusters.
We need to add parallel technique on the rdd, to be parallelized. Then the execution will be faster. After we add parallel technique on RDDs, then we need to call openAcc and MPI pragmas to execute these parallel RDDs on GPU, we will call MPI and openAcc pragmas by a wrapper.
For now we need to create a new parallel algorithm (foreach,for loops …etc) and apply it on the RDDs, but we need to make sure these new techniques are equivalent or appropriate to parallelization of MPI and OpenACC programming model.
proposes a new parallel performance model for different workloads of Spark Big Data applications running on HPC clusters.
We need to add parallel technique on the rdd, to be parallelized. Then the execution will be faster. After we add parallel technique on RDDs, then we need to call openAcc and MPI pragmas to execute these parallel RDDs on GPU, we will call MPI and openAcc pragmas by a wrapper.
For now we need to create a new parallel algorithm (foreach,for loops …etc) and apply it on the RDDs, but we need to make sure these new techniques are equivalent or appropriate to parallelization of MPI and OpenACC programming model.