Transformer WAF Pipeline Development

You are a senior cybersecurity + machine learning engineer.

Build a practical, runnable, end-to-end Transformer-based Web Application Firewall (AI-WAF) prototype suitable for Smart India Hackathon / ISRO SIH demonstration.

The system must implement the full pipeline:

Log Ingestion → Parsing → Normalization → Tokenization → Transformer Training → Multi-Class Attack Classification → Anomaly Scoring → Real-Time Non-Blocking Inference → Alerting & Logging → Incremental Fine-Tuning

========================================
1. CORE FUNCTIONAL REQUIREMENTS
========================================

A. Multi-Class Attack Detection

The Transformer model must classify HTTP requests into:

- BENIGN
- SQL_INJECTION
- XSS
- COMMAND_INJECTION
- PATH_TRAVERSAL
- BRUTE_FORCE
- MALWARE
- DDOS_PATTERN
- ANOMALY (for zero-day / unknown patterns)

Output format:

{
"attack_type": "BENIGN | SQL_INJECTION | XSS | ...",
"confidence": 0.0-1.0,
"anomaly_score": float,
"action": "ALLOW | WARN | BLOCK",
"message": "Human-readable security explanation"
}

========================================
2. END-TO-END PIPELINE COMPONENTS
========================================

A. Log Ingestion
- Read historical Apache/Nginx logs from file
- Implement streaming/tailing mode (simulate tail -f)
- Push parsed logs into processing queue

B. Parsing & Normalization
Extract:
- HTTP method
- URL path
- Query parameters
- Headers
- Request body

Normalize:
- Replace numbers with <NUM>
- Replace UUIDs with <ID>
- Replace tokens/session IDs with <TOKEN>
- Replace hashes with <HASH>
- Lowercase everything
- Remove unnecessary dynamic noise

Output a canonical normalized request string.

C. Tokenization
- Use HuggingFace tokenizer (BERT/DistilBERT)
- Convert normalized request strings to token IDs
- Pad/truncate sequences properly
- Create attention masks

========================================
3. TRANSFORMER MODEL REQUIREMENTS
========================================

Use:
- DistilBERT or BERT-based Transformer Encoder (HuggingFace)

Architecture:
- Transformer encoder
- Multi-class classification head
- Optional anomaly scoring head

Training:
- Train on labeled benign + attack samples
- Use CrossEntropyLoss for classification
- Compute anomaly score using:
- Softmax entropy OR
- Log-likelihood/perplexity

Model must support:
- Saving & loading
- Versioning

========================================
4. REAL-TIME INFERENCE SERVICE
========================================

Use FastAPI.

Endpoint:
POST /analyze_request

Input:
Raw HTTP request (JSON)

Process:
- Normalize
- Tokenize
- Run Transformer inference asynchronously
- Compute classification + anomaly score
- Apply decision logic

Decision Logic:
- If confidence high & attack detected → BLOCK
- If medium confidence → WARN
- If benign → ALLOW

Return JSON response as specified above.

Non-Blocking Requirements:
- Use async endpoints or worker pool
- Support concurrent request handling
- Keep latency low

========================================
5. DDOS DETECTION
========================================

Implement rate-based detection:
- Track request count per IP
- If threshold exceeded within time window → classify as DDOS_PATTERN
- Integrate with ML output for final decision

========================================
6. INCREMENTAL LEARNING
========================================

Provide script to:
- Load new benign traffic
- Fine-tune existing model for limited epochs
- Save new version
- Avoid full retraining

========================================
7. ALERTING & LOGGING
========================================

- Log detected attacks to file or SQLite
- Store:
- Timestamp
- IP
- Attack type
- Confidence
- Action
- Make threshold configurable via config file

========================================
8. DEMO & TESTING
========================================

Provide demo script to simulate:
- Normal traffic
- SQL injection payloads
- XSS payloads
- Brute-force login attempts
- DDoS flood simulation
- Malware upload patterns

Print:
- Model prediction
- Anomaly score
- Action taken

========================================
9. PROJECT STRUCTURE
========================================

ai_waf/
├── ingestion/
├── parsing/
├── normalization/
├── tokenization/
├── model/
├── training/
├── inference_api/
├── rate_detection/
├── incremental_learning/
├── alerts/
├── demo/
└── README.md

========================================
10. OUTPUT EXPECTATIONS
========================================

Generate:
- Full architecture explanation
- Data flow description
- Model design details
- Complete modular Python code
- Training script
- Inference API code
- Incremental training script
- Demo scripts
- Setup & run instructions
- Example configuration files

The final system must be:
- Practical
- Runnable
- Cleanly structured
- Suitable for hackathon / academic demo
- Transformer-based (mandatory)