Scalable tensorflow inference in C#

Hi,

We need to develop a scalable tensorflow model inference project, which allows parallel calling.
I tried TenserflowSharp (a free github project), but it is not scalable with multiple parallel calls (not thread safe).

The problem statement is as follows.
1) To make the inferencing scalable, we have to read the model from disk for each client call. This is IO intensive and time consuming.
2) To avoid reading from disk each time, we have to use the static approach. Then we have to load the model only once from the disk, but we have to use sync lock to make it thread safe. This is not scalable because only once inference will work at a time.

There are articles from microsoft ML.Net team to address this issue.
https://devblogs.microsoft.com/cesardelatorre/how-to-optimize-and-run-ml-net-models-on-scalable-asp-net-core-webapis-or-web-apps/

We need to implement the 'object pool' based on the above article

Basically we have to develop a console app (asp.net 4.7.2) which can do tensorflow inferencing. It also should be scalable for multiple users. Kindly note that this is not .net core project.

Thanks for reading till end.
Happy Bidding.