Build .Net application that manages Azure eventhubs through API. SDK links are provided -- 2
Budget: $250 – $750 CAD
Application will scan through Azure resources and identify over-provisioned eventhubs. It will then set them to a smaller size.
○ You will be provided with login to my azure environment that has what you need
I am responsible for the infrastructure setup
Maximum time for phase 1 is 2 weeks and 2 weeks for phase 2&3
I am available from 08-22 EST to answer any questions
○ Here is the overall logic for the required application, the testing environment
Authentication
Application will use a service managed id from Microsoft to access eventhubs namespaces and storage accounts https://docs.microsoft.com/en-us/azure/azure-app-configuration/howto-integrate-azure-managed-service-identity?tabs=core5x&pivots=framework-dotnet
Application configuration
The storage account id to the JSON file will be provided via webapp configuration
Example of reading the webapp configuration https://stackoverflow.com/questions/44542409/how-to-read-azure-web-site-app-settings-values
Application will have access to a storage account to find a json file that will configure the application
Container and JSON file should be created if its not present in the storage account
JSON file has a list of eventhub namespace ids and other configuration details
Other Logic
Application must retry api commands if the command fails
○ Phase 1
application is able to scan through accessible names spaces, identify if an eventhub namespace is overprovisioned, and what the right level would be
Can be done either using metrics or another method
Finding eventhubs namespace can be done using azure resource manager SDK
https://docs.microsoft.com/en-us/dotnet/api/overview/azure/resourcemanager-readme-pre
https://docs.microsoft.com/en-us/rest/api/resources/resources/get
Querying them for their statistics can be done using the metrics API
https://docs.microsoft.com/en-us/rest/api/monitor/metrics
Write the report to a storage account with the following
eventhub namespace name, scan time, current TU, average events per sec over X hours, average MB/s over X hrs, peak events per sec over X hours , peak MB/s over X hrs, TU it can be lowered to , eventhub namespace id (X is configured as analysis_interval in configuration.json)
Scanning should occur every X minutes (X can be configured as the scanning_interval configuration json)
Report should be written in JSON format
Logic to assess the TU it can be lowered to is :
Scaling of azure eventhub namespace occurs 1MB/sec or 1000 events per second (whichever comes first)
If the event hub is at 5TU and over the X hours has not reached the capacity of 5TU (5MB/sec or 5000 events/sec), then calculate the TU it can be set to using the peak values
If its only reached 3500 events / sec and 2.5MB/s then the TU is 4 (because 3500 events per second needs 4 TU)
Application needs to be able to be reconfigured, by stopping the application, editing the configuration JSON and starting the application again
It should then continue logging, and writing to the same report file
○ Phase 2 -
Improve application and add the capability to modify the eventhub namespaces TU
If X subsequent analysis that eventhub namespace TU can be lowered then lower the TU (X can be configured in configuration json)
API To change the throughput units https://docs.microsoft.com/en-us/rest/api/eventhub/preview/namespaces/update
Write to the logfile, date, eventhub namespace name that was modified, TU before, TU after, eventhub namespace id
○ Phase 3 -
Modify the application to so that it can find a list of evenhubid if a list of subscription, or resource groups is provided in the configuration file
Sample configuration file is provided
○ You will be provided with login to my azure environment that has what you need
I am responsible for the infrastructure setup
Maximum time for phase 1 is 2 weeks and 2 weeks for phase 2&3
I am available from 08-22 EST to answer any questions
○ Here is the overall logic for the required application, the testing environment
Authentication
Application will use a service managed id from Microsoft to access eventhubs namespaces and storage accounts https://docs.microsoft.com/en-us/azure/azure-app-configuration/howto-integrate-azure-managed-service-identity?tabs=core5x&pivots=framework-dotnet
Application configuration
The storage account id to the JSON file will be provided via webapp configuration
Example of reading the webapp configuration https://stackoverflow.com/questions/44542409/how-to-read-azure-web-site-app-settings-values
Application will have access to a storage account to find a json file that will configure the application
Container and JSON file should be created if its not present in the storage account
JSON file has a list of eventhub namespace ids and other configuration details
Other Logic
Application must retry api commands if the command fails
○ Phase 1
application is able to scan through accessible names spaces, identify if an eventhub namespace is overprovisioned, and what the right level would be
Can be done either using metrics or another method
Finding eventhubs namespace can be done using azure resource manager SDK
https://docs.microsoft.com/en-us/dotnet/api/overview/azure/resourcemanager-readme-pre
https://docs.microsoft.com/en-us/rest/api/resources/resources/get
Querying them for their statistics can be done using the metrics API
https://docs.microsoft.com/en-us/rest/api/monitor/metrics
Write the report to a storage account with the following
eventhub namespace name, scan time, current TU, average events per sec over X hours, average MB/s over X hrs, peak events per sec over X hours , peak MB/s over X hrs, TU it can be lowered to , eventhub namespace id (X is configured as analysis_interval in configuration.json)
Scanning should occur every X minutes (X can be configured as the scanning_interval configuration json)
Report should be written in JSON format
Logic to assess the TU it can be lowered to is :
Scaling of azure eventhub namespace occurs 1MB/sec or 1000 events per second (whichever comes first)
If the event hub is at 5TU and over the X hours has not reached the capacity of 5TU (5MB/sec or 5000 events/sec), then calculate the TU it can be set to using the peak values
If its only reached 3500 events / sec and 2.5MB/s then the TU is 4 (because 3500 events per second needs 4 TU)
Application needs to be able to be reconfigured, by stopping the application, editing the configuration JSON and starting the application again
It should then continue logging, and writing to the same report file
○ Phase 2 -
Improve application and add the capability to modify the eventhub namespaces TU
If X subsequent analysis that eventhub namespace TU can be lowered then lower the TU (X can be configured in configuration json)
API To change the throughput units https://docs.microsoft.com/en-us/rest/api/eventhub/preview/namespaces/update
Write to the logfile, date, eventhub namespace name that was modified, TU before, TU after, eventhub namespace id
○ Phase 3 -
Modify the application to so that it can find a list of evenhubid if a list of subscription, or resource groups is provided in the configuration file
Sample configuration file is provided