Optimize Large-Scale Search System
Budget: $30 – $250 USD
I'm seeking a skilled developer to optimize and rebuild a large-scale credential aggregation and search system processing massive datasets (100M+ lines per file, 3B+ total records). We need high-speed ingestion and near-instantaneous search performance, targeting O(log n) disk-based search.
Key requirements:
• Strong C# (.NET) or relevant high-performance languages experience
• Experience with custom data structures, especially B+ Trees
• Familiarity with disk-based search optimization techniques like indexing
• Ability to maintain strict domain indexing, wildcard/subdomain matching, and fast import pipelines
• Understanding of memory-efficient architecture (no full RAM indexing)
Ideal Skills and Experience:
• Proficiency in C# (.NET)
• Custom data structures expertise (B+ Trees, Zone Trees, etc.)
• Disk-based search optimization experience
• Strong problem-solving skills and attention to detail
• Previous experience with large-scale data systems
Please provide examples of related work and your approach to this project.
Ill require the client to have fast search queries for data which is from a mass security stealer data set for analysis.
https://domain.com/email(at)gmail.com:password
The uploaded / ingested data must have the following:
domain must be parsed and indexed along with the email and password efficiently for searches.
Preferred response times >1000ms
Requires decent ingestion speed.
Key requirements:
• Strong C# (.NET) or relevant high-performance languages experience
• Experience with custom data structures, especially B+ Trees
• Familiarity with disk-based search optimization techniques like indexing
• Ability to maintain strict domain indexing, wildcard/subdomain matching, and fast import pipelines
• Understanding of memory-efficient architecture (no full RAM indexing)
Ideal Skills and Experience:
• Proficiency in C# (.NET)
• Custom data structures expertise (B+ Trees, Zone Trees, etc.)
• Disk-based search optimization experience
• Strong problem-solving skills and attention to detail
• Previous experience with large-scale data systems
Please provide examples of related work and your approach to this project.
Ill require the client to have fast search queries for data which is from a mass security stealer data set for analysis.
https://domain.com/email(at)gmail.com:password
The uploaded / ingested data must have the following:
domain must be parsed and indexed along with the email and password efficiently for searches.
Preferred response times >1000ms
Requires decent ingestion speed.