Advanced Enterprise Data Engineering Challenge - NO AGENCIES
Budget: $10,000 – $20,000 USD
CRITICAL: This is NOT Basic Data Processing
We need an enterprise data engineer with PROVEN experience in:
• DNS log analysis and network traffic correlation
• Enterprise software discovery and tool detection algorithms
• Multi-source authentication data processing (Okta, Azure AD)
• Python data pipelines for millions of enterprise records
If you've only done web analytics, ETL pipelines, or business intelligence - this project is NOT for you.
Failed Previous Attempt - Learn From This
Our previous data engineer failed after 3 weeks because he:
• Couldn't parse DNS logs properly
• Didn't understand enterprise domain classification
• Had never built tool detection algorithms
• Overestimated his capabilities for enterprise-scale data
We will test your actual capabilities extensively before hiring.
What You Must Build (Week 1 Deliverable)
Core Requirements:
python
# You must be able to build THIS functionality:
def analyze_enterprise_dns_logs(dns_logs):
"""
Parse enterprise DNS logs and detect unauthorized SaaS usage
Input: 100+ DNS log files with millions of queries
Output: Detected tools with confidence scores
"""
return detected_tools_with_confidence_scores
def correlate_authentication_data(dns_logs, okta_logs):
"""
Correlate DNS traffic with authentication events
Map user activity to specific enterprise software usage
Handle conflicts between multiple data sources
"""
return user_tool_mapping_with_validation
Specific Sprint Recovery Tasks (Week 1):
1. Parse 100+ enterprise DNS logs - extract domains, classify traffic patterns
2. Build tool detection database - 400+ enterprise software entries with domain mapping
3. Correlation engine - match DNS queries to authentication events from Okta/Azure AD
4. JSON output format - structured data ready for cost analysis engine
5. Integration with existing AWS Lambda pipeline (infrastructure already built)
Required Technical Portfolio (MANDATORY)
You MUST have GitHub examples of:
1. DNS log parsing projects - show actual code for network traffic analysis
2. Enterprise data correlation - multi-source data integration with conflict resolution
3. Tool/software detection algorithms - asset discovery or similar pattern matching systems
4. Large-scale data processing - millions of records, not thousands
Portfolio Validation Questions (We WILL Ask These):
• "Walk me through your DNS log parsing algorithm and show me the code"
• "How do you handle enterprise domain classification for SaaS detection?"
• "Explain your approach to correlating authentication logs with network traffic"
• "Show me a tool detection system you've built - how did you handle false positives?"
Technical Stack Requirements
Must Have Experience:
• Python data processing: Pandas, NumPy, enterprise-scale performance optimization
• DNS analysis libraries: dnspython, network traffic parsing, log format handling
• Authentication systems: Okta API, Azure AD, SAML log processing
• AWS integration: Lambda functions, S3 data processing, CloudWatch logging
• Data validation: SHA3 hashing, integrity checks, conflict resolution algorithms
Domain Knowledge Required:
• Enterprise software ecosystems - understand how SaaS tools are deployed
• Network traffic patterns - CDN detection, proxy handling, enterprise DNS flows
• Authentication correlation - linking user activity across multiple systems
• Tool categorization - software taxonomy and enterprise application classification
Project Context & Team
What's Already Built:
• AWS Infrastructure: Terraform-deployed Lambda + S3 pipeline ( DevOps)
• Technical Specifications: Complete MVP 2.0 document with data schemas
• Backend Integration: Analysis engine ready to consume your JSON output (Backend Leader)
Your Responsibility:
• Data Sources → Normalization Layer (your core deliverable)
• Integration points clearly defined with backend analysis engine
• AWS Lambda deployment using existing infrastructure
Reporting Structure:
• Reports to: Principal Technical Lead (technical oversight and code reviews)
• Product Requirements: Founder (business priorities and milestone acceptance)
• Integration Partner: Backend Developer for analysis engine handoff
Milestone-Based Payment Structure
Week 1 (Recovery Sprint) - 2,400
• Working DNS log parser processing 100+ enterprise log files
• Tool detection algorithm identifying 20+ common enterprise applications
• Basic authentication correlation matching Okta events to DNS queries
• JSON output integrating with existing AWS Lambda pipeline
Week 2-3 (Production Pipeline) - $3200
• Expanded tool database with 400+ enterprise software entries
• Advanced correlation engine with conflict resolution and confidence scoring
• Performance optimization for millions of DNS records
• Error handling and data validation for production deployment
Week 4-12 (Enhancement & Scale) - $6400
Real-time processing capabilities for live enterprise data streams
• Advanced analytics for usage patterns and cost optimization insights
• Additional authentication sources (Azure AD, custom SAML systems)
• Enterprise deployment features (air-gapped environments, compliance)
Red Flags We Will Immediately Reject
Based on Previous Failed Hire:
• No portfolio of DNS analysis or network traffic projects
• General data engineering experience without enterprise domain knowledge
• Cannot explain tool detection algorithms or software discovery methods
• Unfamiliar with authentication system correlation techniques
• Overconfident about timeline ("I can do this in 1 week")
Interview Red Flags:
• Cannot show actual code for DNS log parsing projects
• No examples of enterprise-scale data processing (millions of records)
• Unfamiliar with enterprise software ecosystems and SaaS deployment patterns
• Cannot explain correlation techniques for multi-source authentication data
Application Requirements
Submit Immediately:
1. Portfolio Links: Direct GitHub URLs to DNS analysis and tool detection projects you've built
2. Technical Examples: Specific code samples for enterprise data correlation and conflict resolution
3. Scale Examples: Projects handling millions of records with performance optimization
4. Domain Experience: Previous work with enterprise software discovery or asset management systems
Selection Process:
• Portfolio Review: 24-48 hours for initial screening
• Technical Interview: 2-hour deep dive on DNS analysis and enterprise data engineering
• Code Review: Live examination of your existing DNS parsing and tool detection code
• Reference Validation: Calls with previous clients who can verify enterprise data engineering work
What We Provide
Complete Technical Foundation:
• Detailed MVP 2.0 specification with data schemas and integration points
• Working AWS infrastructure ready for your code deployment
• Clear integration guidelines with existing backend analysis engine
• Sample data sets for testing and validation
Project Management:
• Daily progress check-ins with quantifiable deliverables
• Technical support through Principal Technical Lead for architecture questions
• Clear milestone acceptance criteria - binary pass/fail, no subjective evaluation
• Risk protection through escrow payment system
About the Product
Æthux Atlas: Enterprise technology intelligence platform Problem: Organizations waste millions on unused software licenses and unauthorized SaaS Solution: Automated discovery through DNS analysis + authentication correlation + cost optimization Market: Enterprise customers with 1000+ employees, validated demand through customer interviews
How to Apply
Proposal Title: "Enterprise DNS Analysis Expert - Portfolio Attached"
Include in Proposal:
1. Portfolio section: Direct links to DNS analysis and tool detection projects
2. Technical approach: Your specific method for enterprise software discovery via DNS correlation
3. Scale experience: Examples of processing millions of enterprise records
4. Timeline: Realistic assessment for Week 1 deliverables (we expect 5-7 days minimum)
Application Format:
PORTFOLIO LINKS:
- DNS Analysis Project: [GitHub URL with specific repo]
- Tool Detection System: [GitHub URL with specific repo]
- Enterprise Data Pipeline: [GitHub URL with specific repo]
TECHNICAL APPROACH:
[2-3 sentences on your DNS correlation methodology]
SCALE EXPERIENCE:
[1-2 sentences on largest enterprise data project]
REALISTIC TIMELINE:
Week 1: [specific deliverables you can complete]
Immediate Disqualification Criteria
Auto-reject if proposal contains:
• Generic "I have data engineering experience" without specific portfolio
• Claims about "quick completion" or "easy project"
• No GitHub links to actual DNS analysis or tool detection code
• Standard ETL/BI experience presented as enterprise software discovery
• Copy-paste proposals without addressing specific technical requirements
We need an enterprise data engineer with PROVEN experience in:
• DNS log analysis and network traffic correlation
• Enterprise software discovery and tool detection algorithms
• Multi-source authentication data processing (Okta, Azure AD)
• Python data pipelines for millions of enterprise records
If you've only done web analytics, ETL pipelines, or business intelligence - this project is NOT for you.
Failed Previous Attempt - Learn From This
Our previous data engineer failed after 3 weeks because he:
• Couldn't parse DNS logs properly
• Didn't understand enterprise domain classification
• Had never built tool detection algorithms
• Overestimated his capabilities for enterprise-scale data
We will test your actual capabilities extensively before hiring.
What You Must Build (Week 1 Deliverable)
Core Requirements:
python
# You must be able to build THIS functionality:
def analyze_enterprise_dns_logs(dns_logs):
"""
Parse enterprise DNS logs and detect unauthorized SaaS usage
Input: 100+ DNS log files with millions of queries
Output: Detected tools with confidence scores
"""
return detected_tools_with_confidence_scores
def correlate_authentication_data(dns_logs, okta_logs):
"""
Correlate DNS traffic with authentication events
Map user activity to specific enterprise software usage
Handle conflicts between multiple data sources
"""
return user_tool_mapping_with_validation
Specific Sprint Recovery Tasks (Week 1):
1. Parse 100+ enterprise DNS logs - extract domains, classify traffic patterns
2. Build tool detection database - 400+ enterprise software entries with domain mapping
3. Correlation engine - match DNS queries to authentication events from Okta/Azure AD
4. JSON output format - structured data ready for cost analysis engine
5. Integration with existing AWS Lambda pipeline (infrastructure already built)
Required Technical Portfolio (MANDATORY)
You MUST have GitHub examples of:
1. DNS log parsing projects - show actual code for network traffic analysis
2. Enterprise data correlation - multi-source data integration with conflict resolution
3. Tool/software detection algorithms - asset discovery or similar pattern matching systems
4. Large-scale data processing - millions of records, not thousands
Portfolio Validation Questions (We WILL Ask These):
• "Walk me through your DNS log parsing algorithm and show me the code"
• "How do you handle enterprise domain classification for SaaS detection?"
• "Explain your approach to correlating authentication logs with network traffic"
• "Show me a tool detection system you've built - how did you handle false positives?"
Technical Stack Requirements
Must Have Experience:
• Python data processing: Pandas, NumPy, enterprise-scale performance optimization
• DNS analysis libraries: dnspython, network traffic parsing, log format handling
• Authentication systems: Okta API, Azure AD, SAML log processing
• AWS integration: Lambda functions, S3 data processing, CloudWatch logging
• Data validation: SHA3 hashing, integrity checks, conflict resolution algorithms
Domain Knowledge Required:
• Enterprise software ecosystems - understand how SaaS tools are deployed
• Network traffic patterns - CDN detection, proxy handling, enterprise DNS flows
• Authentication correlation - linking user activity across multiple systems
• Tool categorization - software taxonomy and enterprise application classification
Project Context & Team
What's Already Built:
• AWS Infrastructure: Terraform-deployed Lambda + S3 pipeline ( DevOps)
• Technical Specifications: Complete MVP 2.0 document with data schemas
• Backend Integration: Analysis engine ready to consume your JSON output (Backend Leader)
Your Responsibility:
• Data Sources → Normalization Layer (your core deliverable)
• Integration points clearly defined with backend analysis engine
• AWS Lambda deployment using existing infrastructure
Reporting Structure:
• Reports to: Principal Technical Lead (technical oversight and code reviews)
• Product Requirements: Founder (business priorities and milestone acceptance)
• Integration Partner: Backend Developer for analysis engine handoff
Milestone-Based Payment Structure
Week 1 (Recovery Sprint) - 2,400
• Working DNS log parser processing 100+ enterprise log files
• Tool detection algorithm identifying 20+ common enterprise applications
• Basic authentication correlation matching Okta events to DNS queries
• JSON output integrating with existing AWS Lambda pipeline
Week 2-3 (Production Pipeline) - $3200
• Expanded tool database with 400+ enterprise software entries
• Advanced correlation engine with conflict resolution and confidence scoring
• Performance optimization for millions of DNS records
• Error handling and data validation for production deployment
Week 4-12 (Enhancement & Scale) - $6400
Real-time processing capabilities for live enterprise data streams
• Advanced analytics for usage patterns and cost optimization insights
• Additional authentication sources (Azure AD, custom SAML systems)
• Enterprise deployment features (air-gapped environments, compliance)
Red Flags We Will Immediately Reject
Based on Previous Failed Hire:
• No portfolio of DNS analysis or network traffic projects
• General data engineering experience without enterprise domain knowledge
• Cannot explain tool detection algorithms or software discovery methods
• Unfamiliar with authentication system correlation techniques
• Overconfident about timeline ("I can do this in 1 week")
Interview Red Flags:
• Cannot show actual code for DNS log parsing projects
• No examples of enterprise-scale data processing (millions of records)
• Unfamiliar with enterprise software ecosystems and SaaS deployment patterns
• Cannot explain correlation techniques for multi-source authentication data
Application Requirements
Submit Immediately:
1. Portfolio Links: Direct GitHub URLs to DNS analysis and tool detection projects you've built
2. Technical Examples: Specific code samples for enterprise data correlation and conflict resolution
3. Scale Examples: Projects handling millions of records with performance optimization
4. Domain Experience: Previous work with enterprise software discovery or asset management systems
Selection Process:
• Portfolio Review: 24-48 hours for initial screening
• Technical Interview: 2-hour deep dive on DNS analysis and enterprise data engineering
• Code Review: Live examination of your existing DNS parsing and tool detection code
• Reference Validation: Calls with previous clients who can verify enterprise data engineering work
What We Provide
Complete Technical Foundation:
• Detailed MVP 2.0 specification with data schemas and integration points
• Working AWS infrastructure ready for your code deployment
• Clear integration guidelines with existing backend analysis engine
• Sample data sets for testing and validation
Project Management:
• Daily progress check-ins with quantifiable deliverables
• Technical support through Principal Technical Lead for architecture questions
• Clear milestone acceptance criteria - binary pass/fail, no subjective evaluation
• Risk protection through escrow payment system
About the Product
Æthux Atlas: Enterprise technology intelligence platform Problem: Organizations waste millions on unused software licenses and unauthorized SaaS Solution: Automated discovery through DNS analysis + authentication correlation + cost optimization Market: Enterprise customers with 1000+ employees, validated demand through customer interviews
How to Apply
Proposal Title: "Enterprise DNS Analysis Expert - Portfolio Attached"
Include in Proposal:
1. Portfolio section: Direct links to DNS analysis and tool detection projects
2. Technical approach: Your specific method for enterprise software discovery via DNS correlation
3. Scale experience: Examples of processing millions of enterprise records
4. Timeline: Realistic assessment for Week 1 deliverables (we expect 5-7 days minimum)
Application Format:
PORTFOLIO LINKS:
- DNS Analysis Project: [GitHub URL with specific repo]
- Tool Detection System: [GitHub URL with specific repo]
- Enterprise Data Pipeline: [GitHub URL with specific repo]
TECHNICAL APPROACH:
[2-3 sentences on your DNS correlation methodology]
SCALE EXPERIENCE:
[1-2 sentences on largest enterprise data project]
REALISTIC TIMELINE:
Week 1: [specific deliverables you can complete]
Immediate Disqualification Criteria
Auto-reject if proposal contains:
• Generic "I have data engineering experience" without specific portfolio
• Claims about "quick completion" or "easy project"
• No GitHub links to actual DNS analysis or tool detection code
• Standard ETL/BI experience presented as enterprise software discovery
• Copy-paste proposals without addressing specific technical requirements