Deep Learning (AutoEncoders) for Anomaly Detection for New-York Taxi drivers data
Budget: $30 – $250 USD
Given a set of 2M records for Taxi service providers in New-York, write a Python script to find outliers, based on Unsupervised Anomaly Detection with Autoencoders.
Requirements:
1. Python v3 script must use input data (combination of any features provided, except label) to find outliers.
2. The data is labeled and all outliers (for validation purposes only) could be identified based on the existing "label" (0 - not outlier / 1 - outlier found).
3. The overlap between outliers found and known outliers must be >50%
4. Generate charts to show the results (examples: identified outliers vs all entries; overlap with known outliers, etc).
5. The output should be a) charts and b) text file with a list of all outliers found ("ids") and percent overlap with known outliers.
The Taxi's data (almost ~2M records) is structured. Columns:
"id" - unique record id (all records are unique)
"sub_id" - Taxi service provider (allows multiple entries with the same sub_id)
"label" - known outlier (for validation purposes only)
"feature_2".."feature_2060" (Taxi service metadata = features describing Taxi service record)
Resources:
https://cds.cern.ch/record/2209085/files/Outlier%20detection%20using%20autoencoders.%20Olga%20Lyudchick%20(NMS).pdf
http://ceur-ws.org/Vol-2495/paper3.pdf
https://saketsathe.net/downloads/autoencode.pdf
https://www.auai.org/uai2021/pdf/uai2021.154.pdf
https://arxiv.org/pdf/2007.02500.pdf
Requirements:
1. Python v3 script must use input data (combination of any features provided, except label) to find outliers.
2. The data is labeled and all outliers (for validation purposes only) could be identified based on the existing "label" (0 - not outlier / 1 - outlier found).
3. The overlap between outliers found and known outliers must be >50%
4. Generate charts to show the results (examples: identified outliers vs all entries; overlap with known outliers, etc).
5. The output should be a) charts and b) text file with a list of all outliers found ("ids") and percent overlap with known outliers.
The Taxi's data (almost ~2M records) is structured. Columns:
"id" - unique record id (all records are unique)
"sub_id" - Taxi service provider (allows multiple entries with the same sub_id)
"label" - known outlier (for validation purposes only)
"feature_2".."feature_2060" (Taxi service metadata = features describing Taxi service record)
Resources:
https://cds.cern.ch/record/2209085/files/Outlier%20detection%20using%20autoencoders.%20Olga%20Lyudchick%20(NMS).pdf
http://ceur-ws.org/Vol-2495/paper3.pdf
https://saketsathe.net/downloads/autoencode.pdf
https://www.auai.org/uai2021/pdf/uai2021.154.pdf
https://arxiv.org/pdf/2007.02500.pdf