Advanced AI/ML OSINT Tool Development
Budget: $250 – $750 USD
Project Description:
We are seeking an experienced AI/ML developer or team to build a custom AI program designed for deep-dive investigative open-source intelligence (OSINT) research. This tool will assist Trust and Safety (T&S) investigators in platforms like social media or online communities by automating the collection, analysis, and reporting of publicly available data. The AI should handle tasks such as searching social media, analyzing patterns in misinformation or threats, and generating traceable reports, all while emphasizing ethical practices and human oversight.
The system will be an agentic AI framework where multiple specialized "agents" collaborate: a query parser, search agent, analysis agent, report generator, and an orchestrator. It draws inspiration from OSINT tools enhanced with AI, ensuring transparency and compliance with data privacy laws (e.g., GDPR/CCPA). The goal is to augment investigators by speeding up repetitive tasks like entity mapping, sentiment analysis, and risk scoring.
This is a prototype-to-production project. We need a functional MVP (Minimum Viable Product) first, with potential for iterations based on testing.
Key Requirements and Features:
Core Architecture:
Modular multi-agent system using frameworks like LangChain, CrewAI, or AutoGen for orchestration.
Integrate open-source LLMs (e.g., Llama 3 or Mistral via Hugging Face) for reasoning, summarization, and natural language processing.
Database: Use SQLite or Neo4j for storing entity graphs and investigation data.
Data Gathering (Search Module):
Integrate OSINT APIs and tools: X (Twitter) API for social searches, SerpAPI/Google Custom Search for web queries, Reddit/Praw for forums, Maltego for entity mapping, Shodan for device intel (if relevant).
Support web scraping with BeautifulSoup (ethical/public sites only), and media analysis with OpenCV (images) or Whisper (audio transcription).
Automate query generation with AI prompts for varied searches.
Analysis and Insight Generation:
NLP tasks: Entity recognition, sentiment/toxicity analysis, and topic modeling using Hugging Face
Transformers.
Pattern detection: Graph algorithms in NetworkX for networks (e.g., bot detection via retweet patterns).
Risk scoring: Build a simple ML model with scikit-learn to evaluate threats based on thresholds.
Multimodal support: CLIP for image-text matching to detect manipulated media.
Predictive insights: Use LLMs to forecast escalations or summarize findings.
User Interface and Reporting:
Web-based UI with Streamlit or Dash for query input and dashboards.
Generate reports in PDF/Markdown format, including timelines (Plotly), entity graphs (NetworkX/Matplotlib), citations, and risk flags.
Ensure full traceability: Link every claim to source evidence.
Ethical and Safety Features:
Guardrails: Limit to public data, anonymize outputs, flag biases or privacy risks.
Human-AI loop: Require manual review for sensitive decisions; detect AI hallucinations.
Compliance: No personal data scraping; adhere to API terms.
Technical Stack:
Language: Python 3.10+.
Libraries: As listed above, plus any necessary for deployment (e.g., Docker).
Deployment: Hostable on AWS/GCP; containerized for scalability.
Testing: Include unit tests, simulated investigations on public datasets (e.g., Common Crawl subsets).
Deliverables:
Source code repository (GitHub or similar) with documentation.
Prototype demo (video or live session).
Full documentation: Setup guide, API endpoints, usage manual.
Training data curation (if fine-tuning LLMs) from OSINT repositories.
Developer Qualifications:
Proven experience in AI/ML, especially agentic systems or OSINT tools (portfolio required).
Familiarity with ethical AI practices and T&S domains (e.g., misinformation detection).
Strong Python skills; experience with LLMs and APIs.
Ability to work independently with clear communication.
We are seeking an experienced AI/ML developer or team to build a custom AI program designed for deep-dive investigative open-source intelligence (OSINT) research. This tool will assist Trust and Safety (T&S) investigators in platforms like social media or online communities by automating the collection, analysis, and reporting of publicly available data. The AI should handle tasks such as searching social media, analyzing patterns in misinformation or threats, and generating traceable reports, all while emphasizing ethical practices and human oversight.
The system will be an agentic AI framework where multiple specialized "agents" collaborate: a query parser, search agent, analysis agent, report generator, and an orchestrator. It draws inspiration from OSINT tools enhanced with AI, ensuring transparency and compliance with data privacy laws (e.g., GDPR/CCPA). The goal is to augment investigators by speeding up repetitive tasks like entity mapping, sentiment analysis, and risk scoring.
This is a prototype-to-production project. We need a functional MVP (Minimum Viable Product) first, with potential for iterations based on testing.
Key Requirements and Features:
Core Architecture:
Modular multi-agent system using frameworks like LangChain, CrewAI, or AutoGen for orchestration.
Integrate open-source LLMs (e.g., Llama 3 or Mistral via Hugging Face) for reasoning, summarization, and natural language processing.
Database: Use SQLite or Neo4j for storing entity graphs and investigation data.
Data Gathering (Search Module):
Integrate OSINT APIs and tools: X (Twitter) API for social searches, SerpAPI/Google Custom Search for web queries, Reddit/Praw for forums, Maltego for entity mapping, Shodan for device intel (if relevant).
Support web scraping with BeautifulSoup (ethical/public sites only), and media analysis with OpenCV (images) or Whisper (audio transcription).
Automate query generation with AI prompts for varied searches.
Analysis and Insight Generation:
NLP tasks: Entity recognition, sentiment/toxicity analysis, and topic modeling using Hugging Face
Transformers.
Pattern detection: Graph algorithms in NetworkX for networks (e.g., bot detection via retweet patterns).
Risk scoring: Build a simple ML model with scikit-learn to evaluate threats based on thresholds.
Multimodal support: CLIP for image-text matching to detect manipulated media.
Predictive insights: Use LLMs to forecast escalations or summarize findings.
User Interface and Reporting:
Web-based UI with Streamlit or Dash for query input and dashboards.
Generate reports in PDF/Markdown format, including timelines (Plotly), entity graphs (NetworkX/Matplotlib), citations, and risk flags.
Ensure full traceability: Link every claim to source evidence.
Ethical and Safety Features:
Guardrails: Limit to public data, anonymize outputs, flag biases or privacy risks.
Human-AI loop: Require manual review for sensitive decisions; detect AI hallucinations.
Compliance: No personal data scraping; adhere to API terms.
Technical Stack:
Language: Python 3.10+.
Libraries: As listed above, plus any necessary for deployment (e.g., Docker).
Deployment: Hostable on AWS/GCP; containerized for scalability.
Testing: Include unit tests, simulated investigations on public datasets (e.g., Common Crawl subsets).
Deliverables:
Source code repository (GitHub or similar) with documentation.
Prototype demo (video or live session).
Full documentation: Setup guide, API endpoints, usage manual.
Training data curation (if fine-tuning LLMs) from OSINT repositories.
Developer Qualifications:
Proven experience in AI/ML, especially agentic systems or OSINT tools (portfolio required).
Familiarity with ethical AI practices and T&S domains (e.g., misinformation detection).
Strong Python skills; experience with LLMs and APIs.
Ability to work independently with clear communication.