Full stack developer - twitter api download , save to s3 (python code in docker on lambda + infra as code)
Budget: $15 – $25 AUD
Overview
we need to capture (up to the limit of twitter interface, initially free, later paid premium twitter account) a sample of tweets to files, by periodically polling and downloading tweets using API
tweets are written as json files to a well defined/naeed directory structure in S3 e.g. bucketname/yyyy/mm/yyymmddhhmm.json
we require a full stack developer to setup the entire end to end inc dev pipelines, IAC, the python app (running in docker on lambda), inc python unit tests, integration tests and infra code validation and deployment tests
the search query will be specified as a string appended to the API search query
in addition a 2nd lambda /9currently empty) will be triggered to process the downloaded tweets, passing the s3 url of recently downloaded json tweet file
we have found github examples of similar functionality but without full /iaC: we hope to find someone who can use an existing example or have done something similar to accelerate
'we are likely based on this initial version to continue to a bigger production capability so hope to find the right long term relationshipo
Solution Outcome
capture tweets in json files on S3 from twitter based on a single search term
execute 2nd processing step (2nd, empty docker) to process downloaded S3 files : 2nd lambda docker config to support GPU for future use case
Requirements
full infra as code (IAC) deployment (inc user, policy, hardened config) + lambda setup, cloudwatch triggers, s3 bucket setup, docker build and deploy. Ideally in Amazon CDK (python/typescript) not additional terraform or other.
setup of CI/CD in gitlab
basic infrastructure tests as part of Ci/cd deployment
dev, test, prod env setup and promotion within CI/CD
any apps / docker processing written in python
secure secrets for api keys etc
environment variables to define: search string term (currently just lang=en), frequency/polling, bucket names, dev/test/prod prefixes, twitter creds (note restart services to update env variables)
count tweets downloaded (per download lambda, running total ) and write to s3 logfile
handle band-width / quota excess from api using back-off.
for each new download invocation, use previous cursor to ensure no gaps in query timeline
Hypothesis
Current hypothesis is the architecture/solution setup will be - but we would like your opinion
cloudwatch trigger
lambda (docker based) download function (python code) for download from twitter
store 'cursor' of last download date/time in a dynamodb table
store tweet download volumes in an s3 file (inc history e.g. each file line = n minute period record of download)
download content every n minutes for previous n minutes (ENV variable configurable)
write content to s3 fiel in a well defined bucket S3 structure (a single file of tweets for n minute period = last minute captured, filename = date time of last full minute captured] .ie YYYY/MM/YYYYMMDDHHMM.json 2022/10/202210030101
we have list user stories as developer and 'system' which we can provide (see attached file)
NOTE
we have put paid by the hour, but would like an estimate of days/weeks to setup and complete
we need to use python and aws for this project due to the team taking over this in production.
we need to move to gitlab for cicd so would like the project to be delivered on that.
we need to capture (up to the limit of twitter interface, initially free, later paid premium twitter account) a sample of tweets to files, by periodically polling and downloading tweets using API
tweets are written as json files to a well defined/naeed directory structure in S3 e.g. bucketname/yyyy/mm/yyymmddhhmm.json
we require a full stack developer to setup the entire end to end inc dev pipelines, IAC, the python app (running in docker on lambda), inc python unit tests, integration tests and infra code validation and deployment tests
the search query will be specified as a string appended to the API search query
in addition a 2nd lambda /9currently empty) will be triggered to process the downloaded tweets, passing the s3 url of recently downloaded json tweet file
we have found github examples of similar functionality but without full /iaC: we hope to find someone who can use an existing example or have done something similar to accelerate
'we are likely based on this initial version to continue to a bigger production capability so hope to find the right long term relationshipo
Solution Outcome
capture tweets in json files on S3 from twitter based on a single search term
execute 2nd processing step (2nd, empty docker) to process downloaded S3 files : 2nd lambda docker config to support GPU for future use case
Requirements
full infra as code (IAC) deployment (inc user, policy, hardened config) + lambda setup, cloudwatch triggers, s3 bucket setup, docker build and deploy. Ideally in Amazon CDK (python/typescript) not additional terraform or other.
setup of CI/CD in gitlab
basic infrastructure tests as part of Ci/cd deployment
dev, test, prod env setup and promotion within CI/CD
any apps / docker processing written in python
secure secrets for api keys etc
environment variables to define: search string term (currently just lang=en), frequency/polling, bucket names, dev/test/prod prefixes, twitter creds (note restart services to update env variables)
count tweets downloaded (per download lambda, running total ) and write to s3 logfile
handle band-width / quota excess from api using back-off.
for each new download invocation, use previous cursor to ensure no gaps in query timeline
Hypothesis
Current hypothesis is the architecture/solution setup will be - but we would like your opinion
cloudwatch trigger
lambda (docker based) download function (python code) for download from twitter
store 'cursor' of last download date/time in a dynamodb table
store tweet download volumes in an s3 file (inc history e.g. each file line = n minute period record of download)
download content every n minutes for previous n minutes (ENV variable configurable)
write content to s3 fiel in a well defined bucket S3 structure (a single file of tweets for n minute period = last minute captured, filename = date time of last full minute captured] .ie YYYY/MM/YYYYMMDDHHMM.json 2022/10/202210030101
we have list user stories as developer and 'system' which we can provide (see attached file)
NOTE
we have put paid by the hour, but would like an estimate of days/weeks to setup and complete
we need to use python and aws for this project due to the team taking over this in production.
we need to move to gitlab for cicd so would like the project to be delivered on that.