Import Rest API Data into SQL server using Azure Data Factory (ADF)
Budget: $10 – $50 AUD
Hi There,
I have REST API and I need to import the data into a Azure SQL Server table.
I am using Azure Data Factory Data flow (attached) which was working fine but as the data grew I started getting the below error
Azure data factory REST API Job aborted due to stage failure: Serialized task 16:0 was bytes, which exceeds max allowed: spark.rpc.message.maxSize (134217728 bytes).
So I had to split the source API data into two parts using $top and $skip
Like below, two URL calls.
abc?$expand=xx,yy,zz&$top=40000&$skip=0
abc?$expand=xx,yy,zz&$top=40000&$skip=40000
I am looking for efficient way of doing it by using for loop or parametrization or anything which makes it work even if the data further grows.
Example API data : https://services.odata.org/TripPinRESTierService/People?$expand=Trips
If you can use above api and import all the data fields into SQL server.
I have REST API and I need to import the data into a Azure SQL Server table.
I am using Azure Data Factory Data flow (attached) which was working fine but as the data grew I started getting the below error
Azure data factory REST API Job aborted due to stage failure: Serialized task 16:0 was bytes, which exceeds max allowed: spark.rpc.message.maxSize (134217728 bytes).
So I had to split the source API data into two parts using $top and $skip
Like below, two URL calls.
abc?$expand=xx,yy,zz&$top=40000&$skip=0
abc?$expand=xx,yy,zz&$top=40000&$skip=40000
I am looking for efficient way of doing it by using for loop or parametrization or anything which makes it work even if the data further grows.
Example API data : https://services.odata.org/TripPinRESTierService/People?$expand=Trips
If you can use above api and import all the data fields into SQL server.