Python ETL using aiohttp on Azure batch pool missing responses
Budget: $10 – $50 USD
Hi, I have a python driven etl process which ingests data via APIs and loads to Azure SQL. We execute the code on Azure Batch Pools, orchestrated by Azure Data Factory. Beginning a month or so ago, we started encountering issues with API requests not receiving responses. This problem only occurs on Azure (under both Linux and Windows) but does not exist when running the code locally. We are interested in code review to see if there's anything we could be doing wrong from a code perspective to fix this problem.
solve issue where there are missing responses in python api request using aiohttp when executed on azure batch pools.
code runs fine locally so interested in a solution to allow code to run on azure batch pool.
thanks.
solve issue where there are missing responses in python api request using aiohttp when executed on azure batch pools.
code runs fine locally so interested in a solution to allow code to run on azure batch pool.
thanks.