Python ETL using aiohttp on Azure batch pool missing responses

Job ID: 34723025

Budget: $10 – $50 USD

Hi, I have a python driven etl process which ingests data via APIs and loads to Azure SQL. We execute the code on Azure Batch Pools, orchestrated by Azure Data Factory. Beginning a month or so ago, we started encountering issues with API requests not receiving responses. This problem only occurs on Azure (under both Linux and Windows) but does not exist when running the code locally. We are interested in code review to see if there's anything we could be doing wrong from a code perspective to fix this problem.

solve issue where there are missing responses in python api request using aiohttp when executed on azure batch pools.
code runs fine locally so interested in a solution to allow code to run on azure batch pool.

thanks.
Related categories: Python Linux Software Architecture Azure ETL