Apache Spark performance tuning required(High proficiency required)
Currently I have a job which does some aggregations and takes a day or two but I have seen the same job completed within 30 mins with some unknown set of resour...
Read more
ID / Date
Listing
Link
Spark scala project
I am looking for guy who is expert in spark, scala and sql.
Read morePresentation on Delta Lae, Data bricks
The main requirements of project are: Delta lake Data bricks Airflow Lake House Architecture Ppt should be around 20 slides Also i need a KT on these topics KT...
Read moreCalculate availability in a period of time in Databricks with python and sql
Using a database where have the time when ocurrs the errors in service calculate availability using the formula: Availability: MTBF /(MTBF+MTTR) MTBF = Uptime /...
Read moreSpark structured streamin from Elastic search -- 2
Hi, I need to stream Elastic search data to Spark Data frame as structured streaming in real time. Need to write this application in java.
Read moreData engineer project
I am looking for an experienced data engineer who know how to create data pipelines, batch processing, scalability etc. Details about the project over chat.
Read moreSpark structured streamin from Elastic search
Hi, I need to stream Elastic search data to Spark Data frame as structured streaming in real time. Need to write this application in java.
Read moreKafka, Spark ML Expert Needed for Hourly Consultations -- 3
Hello I am building a real time streaming analytics system, that will be used to analyse and trade the stock markets. I am looking for a Kafka expert who can pi...
Read moreOptimize Spark Job on VERY Large Datasets (Data Enrichment)
We have a Pipeline in which we are doing data enrichment. We aim to process a more than 30 TB data. Currently, we are only processing little less than 2percent...
Read moreBig data projecy
Real-Time Visualization of Sensor Data with Spark Streaming, Kafka, and Kibana
Read moreNeed help in production support for spark and etl project
Hi I need help in production support for spark and etl project. Person need to find out the issue from ticket and propose probable solution for particular ticke...
Read moreadd two block matrices in Scala
add two block matrices in Scala
Read moreSpark streaming with scala
to develop a code to achieve specific requirements.
Read morePySpark program to generate large dataset with 100 000 columns and 50 million rows
I need a PySpark program to generate a large dataset with 100 000 columns and 50 million rows. I should be able to set the number of dimension columns (i.e. col...
Read moreNeed Scala and Pyspark Expert for Our project
Looking for Python and Scala expert, Candidate should have knowledge in Big data domains such as Hadoop, spark, hive, etc. Knowledge of Azure Cloud is a plus. S...
Read moreSelecting a Phd Topic about (Big Data Analytics technologies in healthcare)
I need help selecting a unique topic for my PhD thesis related to Big Data Analytics technologies in healthcare
Read morebug fixes in the current spark-scala project
Reading xml and xpt files using spark scala and writing the output in form of an hive table. currently facing small issues while running the project, I believe...
Read moreCreating a spark sql flow integrated with scala framework.
we need create a new flow by write spark sql queries same as a flow which is already built to avoid using the old flow. source : HDFS
Read moreNeed Instagram filter -- 2
Instagram filter made for a new brand, I have an image (attached) I want as the filter, the image is of abstract angels/cherubs. Could you please create the fil...
Read moreSpark SQL - Write code to achieve simple logic -- 2
Using SparkSQL, Create a fuzzy match logic. For Example user input for city: Newy Dataframe contains many cities: Newyork, Newdelhi, New Brunswick, New Mexico....
Read moreNeed someone with Spark Scala, Hadoop and Hive Knowledge
Any one having the knowledge of following technologies can Bid: -programming language Scala must, python as well -hands on experience on spark -hands on experie...
Read moreRules Based Engine for Fintech - Credit Scoring
1 Requirements 1. Trust score engine will assign a trust score to a borrower based on signals received through app integrated with mobile SDK and external EKYC...
Read moreKafka, Spark ML Expert Needed for Hourly Consultations -- 2
Hello I am building a real time streaming analytics system, that will be used to analyse and trade the stock markets. I am looking for a Kafka expert who can pi...
Read moreData Engineer
Converting SQL scripts into Spark-Scala Framework.
Read moreKafka, Spark ML Expert Needed for Hourly Consultations
Hello I am building a real time streaming analytics system, that will be used to analyse and trade the stock markets. I am looking for a Kafka expert who can pi...
Read moreBig Data support , Hadoop
Task:: Using Ozzie for job scheduling. If the workflow is not running for some reason need to debug and resolve the issue I have workflow and coordinates inform...
Read moreApache Airflow Expert to guide
Need Airflow Expert to guide me in real time.
Read moreCreate a Database of Business Information from 240M websites
I have a list of 240 million websites I'd like to crawl and want to do that in the most cost-effective /quickest way. This would likely be accomplished with a w...
Read moreKafka & Real Time Database Expert Needed - Hourly calls2
Hello this posting is for purchase of your time. you will get paid on an hourly basis and we will have 1 or 2 calls of 2 hours each per week. basically we are b...
Read moreI need a big data developer
My project is about scala and spark optimization
Read moreBuild me Big Data projects
Hi I am looking an expert for my data engineer project. You need to work on screen sharing on my computer. If you are expert on Big Data, Spark, ETL pipeline, L...
Read moreimplementasi Pyspark in jupyter notebook -- 2
saya ingin melakukan read dan write data localhost dengan menggunakan spark / pyspark di jupyter notebook
Read moreimplementasi Pyspark in jupyter notebook
saya ingin melakukan read data dan write data dari localhost dengan pyspark di jupyter notebook
Read moreSpark Ml Using GCP
Labelled dataset would be provided, perform Ml in GCP , GCP access will be provided if you don’t have one
Read moreEclipse scala ide
I need someone who can fix eclipse SCALA IDE issues and debug a piece of SCALA code
Read moreBig data and cloud computing analysis
n this scenario, you are a data scientist working with a marketing consultancy. Your client is an insurance companythat is developing a highly segmented home in...
Read moreWrapper to convert csv to parquet and avro formats
1. Build a wrapper 2. Wrapper should have a python file. 3. That python file should ask us arguments for input for file names then pick the file from location a...
Read moreSpark summit Job - Converting CSV to Parquet, Avro
Just build a wrapper, call the python file which needs to convert csv to parquet, avro format.
Read moreApache Spark machine learning project
There are three different problems that you must solve. The first two problems the datasets and problem statement are given to you. The last problem is open end...
Read moreSetting spark EMR cluster on aws account
Setting spark EMR cluster on aws account
Read more