tweet download via api and aws end to end infrastructure (already starting point AWS Serverless example)
Budget: $15 – $25 USD
Overview
We need to capture (up to the limit of twitter interface/api, initially free, later paid premium twitter account) a continuous sample of tweets to S3 files, by periodically polling and downloading tweets using API and storing in S3 file. This is currently based on the twitter v2 api, tweets/stream/sample outputs
Tweets are written as JSON files to a well defined directory structure in S3 e.g. s3://bucketname/basedir/yyyy/mm/dd/hh/yyymmddhhmm.json
We require a full stack developer to setup the entire end to end inc dev pipelines, IAC, the python app (running in docker on lambda), inc python unit tests, integration tests and infra code validation and deployment tests
The search query will be specified as a string appended to the API search query
in addition a 2nd lambda (currently empty) will be triggered to process the downloaded tweets, passing the s3 url of recently downloaded json tweet file
We have a good starting point for the code through the AWS serverless example (here) which uses SAM to deploy function + cursor + event trigger.
Solution Outcome
First lambda (downloadTweets) capture tweets in json files on S3 from twitter from random sample feed (e.g. no search term) and capture additional details (language, location, meta ata etc.)
Trigger 2nd processing step (2nd, empty empty/lambda docker for now) to process downloaded S3 files (triggered by end of successful execution of first lambda): 2nd lambda docker config to support GPU for future use case
We record the last downloaded tweet id and the date time of the last tweet captured in a dynamodb.
Note currently these are not used for the sample download: the lambda will run, say every 5min and just get the retrieved data (note example code here of cursor storage + py unit test)
Project Scope
Review starting point / example code
Define user-stories in trello for project mgt
Setup of aws account (we can use a account provided by benlex)
Setup of SAM templates (assumed using starting point)
Setup of circleci (ideally, but please recommend alternative)
Python code to download tweets using stream/sample api.
Solution start point
Note: this aws example is likely the starting point (it does not use docker). It is deployed via sam. Changes required:
V2 of twitter api. As we are storing json please do not use tweepy but use api via rest api query to allow json to be retrieved and directly saved. Example curl commands with required expansions/data required specified below.
TwitterSearchPoller (used as DownloadTweets lambda) writes directly S3 (fixed structure e.g. s3://bucketname/basedir/yyyy/dd/mm/dd/file.json
Update dynamodb table to include last tweet ID stored and date time of last download tweet
Aws example includes ‘cursor mgt’ code to setup/update dynamodb checkpoint.py and py.test (this must be converted to unittest/mock)
Tweet Processor is an ‘empty’ 2nd job (GPU enabled container) that currently will process a range of s3 files (it receives info in event message of first and last tweet ids and first and last dates of downloaded)
we have a full documented spec we can provide.
We need to capture (up to the limit of twitter interface/api, initially free, later paid premium twitter account) a continuous sample of tweets to S3 files, by periodically polling and downloading tweets using API and storing in S3 file. This is currently based on the twitter v2 api, tweets/stream/sample outputs
Tweets are written as JSON files to a well defined directory structure in S3 e.g. s3://bucketname/basedir/yyyy/mm/dd/hh/yyymmddhhmm.json
We require a full stack developer to setup the entire end to end inc dev pipelines, IAC, the python app (running in docker on lambda), inc python unit tests, integration tests and infra code validation and deployment tests
The search query will be specified as a string appended to the API search query
in addition a 2nd lambda (currently empty) will be triggered to process the downloaded tweets, passing the s3 url of recently downloaded json tweet file
We have a good starting point for the code through the AWS serverless example (here) which uses SAM to deploy function + cursor + event trigger.
Solution Outcome
First lambda (downloadTweets) capture tweets in json files on S3 from twitter from random sample feed (e.g. no search term) and capture additional details (language, location, meta ata etc.)
Trigger 2nd processing step (2nd, empty empty/lambda docker for now) to process downloaded S3 files (triggered by end of successful execution of first lambda): 2nd lambda docker config to support GPU for future use case
We record the last downloaded tweet id and the date time of the last tweet captured in a dynamodb.
Note currently these are not used for the sample download: the lambda will run, say every 5min and just get the retrieved data (note example code here of cursor storage + py unit test)
Project Scope
Review starting point / example code
Define user-stories in trello for project mgt
Setup of aws account (we can use a account provided by benlex)
Setup of SAM templates (assumed using starting point)
Setup of circleci (ideally, but please recommend alternative)
Python code to download tweets using stream/sample api.
Solution start point
Note: this aws example is likely the starting point (it does not use docker). It is deployed via sam. Changes required:
V2 of twitter api. As we are storing json please do not use tweepy but use api via rest api query to allow json to be retrieved and directly saved. Example curl commands with required expansions/data required specified below.
TwitterSearchPoller (used as DownloadTweets lambda) writes directly S3 (fixed structure e.g. s3://bucketname/basedir/yyyy/dd/mm/dd/file.json
Update dynamodb table to include last tweet ID stored and date time of last download tweet
Aws example includes ‘cursor mgt’ code to setup/update dynamodb checkpoint.py and py.test (this must be converted to unittest/mock)
Tweet Processor is an ‘empty’ 2nd job (GPU enabled container) that currently will process a range of s3 files (it receives info in event message of first and last tweet ids and first and last dates of downloaded)
we have a full documented spec we can provide.