Experienced Data Engineer to work on IPinfo.io Data Pipelines
Budget: $250 – $750 USD
I'm looking to hire an awesome engineer to help augment our data team.
Our data team's tech stack mostly consists of Google Composer (a lightweight wrapper around Apache Airflow), BigQuery, and a lot of bash script. We have lots of daily pipelines that run, to process various data sets, including:
- RIR Whois data
- BGP Feeds
- Our own API logs (over 40Bn req/month)
- Mobile SDK location data
- Ping & traceroute data from our own probe network
- Various others
One of the first tasks we need help with is building out and improving our company data pipeline, combining various sources of company data into a clean and consistent data set. Other tasks will include helping to optimize some of our existing pipelines, making them more debugable, better reporting and alerting around data changes and quality etc.
Looking for someone with expert level experience with building data pipelines, and ideally some familiarity with IP address, domain and company/firmographic data.
Our data team's tech stack mostly consists of Google Composer (a lightweight wrapper around Apache Airflow), BigQuery, and a lot of bash script. We have lots of daily pipelines that run, to process various data sets, including:
- RIR Whois data
- BGP Feeds
- Our own API logs (over 40Bn req/month)
- Mobile SDK location data
- Ping & traceroute data from our own probe network
- Various others
One of the first tasks we need help with is building out and improving our company data pipeline, combining various sources of company data into a clean and consistent data set. Other tasks will include helping to optimize some of our existing pipelines, making them more debugable, better reporting and alerting around data changes and quality etc.
Looking for someone with expert level experience with building data pipelines, and ideally some familiarity with IP address, domain and company/firmographic data.