ElasticSearch (ELK) Expert needed!!
Budget: €250 – €750 EUR
Our Data (mainly text) is currently stored in parquet format (in S3) and raw (TXT, CSV, XLSX etc.) format is around 10 TB and will grow exponentially.
In our architecture, we have a Spark cluster of 10 nodes (150 CPU and 500 GB RAM) to process raw files and store them in S3.
We aim to build a hot layer upper our data lake, using ELK. This layer should allow us to migrate existing data from s3 to Elasticsearch and process fast queries. Query results will be used for data analysis.
Our requirements are mainly:
- Set up a very cost-effective and efficient ELK cluster (or Optimize our existing one)
- Allow fast migration of existing data from spark. (Data is transformed in Spark data frame that will be stored in Elasticsearch as Document in one or many indexes using Spark ES connector)
- Detect any bottleneck in communication between Spark and ED Cluster
- Optimize queries on data. We need data to be retrieved as fast as possible. Very, very fast, 1-3sec per query is acceptable.
In our architecture, we have a Spark cluster of 10 nodes (150 CPU and 500 GB RAM) to process raw files and store them in S3.
We aim to build a hot layer upper our data lake, using ELK. This layer should allow us to migrate existing data from s3 to Elasticsearch and process fast queries. Query results will be used for data analysis.
Our requirements are mainly:
- Set up a very cost-effective and efficient ELK cluster (or Optimize our existing one)
- Allow fast migration of existing data from spark. (Data is transformed in Spark data frame that will be stored in Elasticsearch as Document in one or many indexes using Spark ES connector)
- Detect any bottleneck in communication between Spark and ED Cluster
- Optimize queries on data. We need data to be retrieved as fast as possible. Very, very fast, 1-3sec per query is acceptable.