Passive Situational Awareness Repository
Budget: €36 – €0 EUR
I need a strictly passive, push-based data repository that ingests only open data sources, never polling or scraping. The pipeline must accept already-aggregated feeds, validate them against a pre-fined KPI Engine schema, and automatically merge everything into daily and weekly roll-ups covering Europe, the Eastern Mediterranean, the Balkans, North Africa, and the Middle East.
Latency targets are tight: where a source publishes in real time the system should reflect it within minutes; otherwise, the roll-up must appear no later than the next daily or weekly cycle. The repository itself remains free-to-use and non-interactive—its sole job is to accept incoming payloads on a fixed schedule, store them, and expose fast, structured queries plus predefined event parameters for rapid retrieval.
Deliverables
• Automate the collection of data from open sources
• Automatically categorize and organize information into thematic groups
• Enable fast and precise retrieval of relevant information based on structured queries and predefined event parameterization and KPI Calculation Pipelines.
• Automate the creation and management of hierarchical knowledge databases
• Automate the identification of connections and trends across datasets.
• Automatically generate summaries and reports based on analyzed data
• Data-flow architecture diagram and tech stack recommendation
• Ingestion scripts / webhooks ready to handle open feeds and push them into storage
• Automated roll-up jobs (daily & weekly) with validation against the KPI Engine schema
• Query layer with sample structured queries illustrating event parameterization and sub-second response times
Acceptance criteria: zero manual intervention after deployment, successful end-to-end demonstration on at least three live open data feeds, and roll-ups generated on schedule for each target region.
Latency targets are tight: where a source publishes in real time the system should reflect it within minutes; otherwise, the roll-up must appear no later than the next daily or weekly cycle. The repository itself remains free-to-use and non-interactive—its sole job is to accept incoming payloads on a fixed schedule, store them, and expose fast, structured queries plus predefined event parameters for rapid retrieval.
Deliverables
• Automate the collection of data from open sources
• Automatically categorize and organize information into thematic groups
• Enable fast and precise retrieval of relevant information based on structured queries and predefined event parameterization and KPI Calculation Pipelines.
• Automate the creation and management of hierarchical knowledge databases
• Automate the identification of connections and trends across datasets.
• Automatically generate summaries and reports based on analyzed data
• Data-flow architecture diagram and tech stack recommendation
• Ingestion scripts / webhooks ready to handle open feeds and push them into storage
• Automated roll-up jobs (daily & weekly) with validation against the KPI Engine schema
• Query layer with sample structured queries illustrating event parameterization and sub-second response times
Acceptance criteria: zero manual intervention after deployment, successful end-to-end demonstration on at least three live open data feeds, and roll-ups generated on schedule for each target region.