Automation of Daily Monitoring Tasks

Task 1
Description
Identify areas of improvement in daily monitoring activities and implement automation to reduce manual effort. Identify automation opportunities in routine manual tasks and implement at least one automation solution that delivers quantifiable effort savings.

Targets
Reduce manual effort involved in daily job monitoring activities
Propose three automation ideas and implement at least one

Current Manual Activities
ADF: Monitoring critical pipeline failures and job status
Apache Spark: Monitoring job execution status and runtime
Power BI: Capturing report screenshots during peak hours every 2 hours and manually verifying report refresh status
ServiceNow: Periodic manual checks for P1 and P2 incidents

Automation Requirements
ADF Pipelines Monitoring
1.Automate monitoring of critical and important ADF pipelines except Email.( Automation like I want to send msg to a teams channel )
2.Automatically notify team members or an MS Teams channel when pipelines succeed or fail except Email.( Automation like I want to send msg to a teams channel )
3.Include pipeline details and screenshots in the notification.

Apache Spark Job Monitoring
1.Automate capture of Apache Spark job status screenshots every 2 hours.
2.Automatically send screenshots to an MS Teams channel
3.Trigger alerts when a Spark job fails or runs longer than 120–130 minutes and notify relevant MS Teams channels.
Power BI Report Monitoring
1.Automate Power BI report screenshot capture every 2 hours.
2.Send screenshots to an MS Teams channel to verify data refresh and report availability.
ServiceNow Incident Monitoring
1.Automate detection of P1 and P2 incidents.
2.Automatically notify responsible team members via MS Teams
3.Include incident details where applicable.