Build .Net application that manages Azure eventhubs through API. SDK links are provided
Budget: $300 – $500 CAD
Application will scan through Azure resources and identify over-provisioned eventhubs. It will then set them to a smaller size.
○ You will be provided with login to my azure environment that has what you need
I am responsible for the infrastructure setup
Maximum time for phase 1 is 2 weeks and 2 weeks for phase 2&3
I am available from 2pm - 4 am moscow time to answer any questions
○ Here is the overall logic for the required application, the testing environment
Authentication
Application will use a service managed id from Microsoft to access eventhubs namespaces and storage accounts https://docs.microsoft.com/en-us/azure/azure-app-configuration/howto-integrate-azure-managed-service-identity?tabs=core5x&pivots=framework-dotnet
Application configuration
The storage account id to the JSON file will be provided via webapp configuration
Example of reading the webapp configuration https://stackoverflow.com/questions/44542409/how-to-read-azure-web-site-app-settings-values
Application will have access to a storage account to find a json file that will configure the application
Container and JSON file should be created if its not present in the storage account
JSON file has a list of eventhub namespace ids and other configuration details
Other Logic
Application must retry api commands if the command fails
Logging design
2 logging levels
Debug : Each azure API command should be logged along with its output
Error : only errors and retries are logged
Logging storage account id is specified in the settings of the webapp
Loglevel configured in configuration.json as logging_level
Container and the logfile should be created if not present on the storage account
Logfile in the container should be rotated every 10 MB, maximum number of logfiles is 20
If unable to write to the storage account, log should be written locally, up to a maximum size of 10MB
Use json logging format https://docs.microsoft.com/en-us/dotnet/core/extensions/console-log-formatter
○ Phase 1
application is able to scan through accessible names spaces, identify if an eventhub namespace is overprovisioned, and what the right level would be
Can be done either using metrics or another method
Finding eventhubs namespace can be done using azure resource manager SDK
https://docs.microsoft.com/en-us/dotnet/api/overview/azure/resourcemanager-readme-pre
https://docs.microsoft.com/en-us/rest/api/resources/resources/get
Querying them for their statistics can be done using the metrics API
https://docs.microsoft.com/en-us/rest/api/monitor/metrics
Write the report to a storage account with the following
eventhub namespace name, scan time, current TU, average events per sec over X hours, average MB/s over X hrs, peak events per sec over X hours , peak MB/s over X hrs, TU it can be lowered to , eventhub namespace id (X is configured as analysis_interval in configuration.json)
Scanning should occur every X minutes (X can be configured as the scanning_interval configuration json)
Report should be written in JSON format
○ Phase 2 -
Improve application and add the capability to modify the eventhub namespaces TU
If X subsequent analysis that eventhub namespace TU can be lowered then lower the TU (X can be configured in configuration json)
API To change the throughput units https://docs.microsoft.com/en-us/rest/api/eventhub/preview/namespaces/update
Write to the logfile, date, eventhub namespace name that was modified, TU before, TU after, eventhub namespace id
○ Phase 3 -
Modify the application to so that it can find a list of evenhubid if a list of subscription, or resource groups is provided in the configuration file
Sample configuration file is provided
Phase 1 has a few more clarifications that went beyond the character limit.
○ You will be provided with login to my azure environment that has what you need
I am responsible for the infrastructure setup
Maximum time for phase 1 is 2 weeks and 2 weeks for phase 2&3
I am available from 2pm - 4 am moscow time to answer any questions
○ Here is the overall logic for the required application, the testing environment
Authentication
Application will use a service managed id from Microsoft to access eventhubs namespaces and storage accounts https://docs.microsoft.com/en-us/azure/azure-app-configuration/howto-integrate-azure-managed-service-identity?tabs=core5x&pivots=framework-dotnet
Application configuration
The storage account id to the JSON file will be provided via webapp configuration
Example of reading the webapp configuration https://stackoverflow.com/questions/44542409/how-to-read-azure-web-site-app-settings-values
Application will have access to a storage account to find a json file that will configure the application
Container and JSON file should be created if its not present in the storage account
JSON file has a list of eventhub namespace ids and other configuration details
Other Logic
Application must retry api commands if the command fails
Logging design
2 logging levels
Debug : Each azure API command should be logged along with its output
Error : only errors and retries are logged
Logging storage account id is specified in the settings of the webapp
Loglevel configured in configuration.json as logging_level
Container and the logfile should be created if not present on the storage account
Logfile in the container should be rotated every 10 MB, maximum number of logfiles is 20
If unable to write to the storage account, log should be written locally, up to a maximum size of 10MB
Use json logging format https://docs.microsoft.com/en-us/dotnet/core/extensions/console-log-formatter
○ Phase 1
application is able to scan through accessible names spaces, identify if an eventhub namespace is overprovisioned, and what the right level would be
Can be done either using metrics or another method
Finding eventhubs namespace can be done using azure resource manager SDK
https://docs.microsoft.com/en-us/dotnet/api/overview/azure/resourcemanager-readme-pre
https://docs.microsoft.com/en-us/rest/api/resources/resources/get
Querying them for their statistics can be done using the metrics API
https://docs.microsoft.com/en-us/rest/api/monitor/metrics
Write the report to a storage account with the following
eventhub namespace name, scan time, current TU, average events per sec over X hours, average MB/s over X hrs, peak events per sec over X hours , peak MB/s over X hrs, TU it can be lowered to , eventhub namespace id (X is configured as analysis_interval in configuration.json)
Scanning should occur every X minutes (X can be configured as the scanning_interval configuration json)
Report should be written in JSON format
○ Phase 2 -
Improve application and add the capability to modify the eventhub namespaces TU
If X subsequent analysis that eventhub namespace TU can be lowered then lower the TU (X can be configured in configuration json)
API To change the throughput units https://docs.microsoft.com/en-us/rest/api/eventhub/preview/namespaces/update
Write to the logfile, date, eventhub namespace name that was modified, TU before, TU after, eventhub namespace id
○ Phase 3 -
Modify the application to so that it can find a list of evenhubid if a list of subscription, or resource groups is provided in the configuration file
Sample configuration file is provided
Phase 1 has a few more clarifications that went beyond the character limit.