PHP/MySQL Web Scraper and Organizer
Budget: $30 – $250 USD
Simple php/mysql web app.
Basic description:
2 stages: Stage 1 – Simple web scraper that processes the first 25 pages of the site and checks the table, if the record is not already in the present, it adds it, if it is, it skips and moves on to the next one.
I don’t need any display of records, I just need the scraper to put the data into the table. And I also need the scraper php script capable of running as a CPANEL cron script goal is to run it ever 8 hours.
Stage 2 – takes the closest intersection, and based on it, will get the next 2 streets to the north, south, east, west. And store them all in a table tied to the incident PK.
Again, there is no need to display or pull any data out, I just need data placed into the tables.
Tables:
Table 1: Incident
• Primary Key
• Date
• Time
• Street Name
• Closest Intersection
• Call Type
• Incident Number (this is what to check against for preexisting record)
• CFS Dispo
Table 2: NearIntSects
• Primary key
• IncidentId (tied to PK of Incident table for specific Incident)
• Street North 1
• Street North 2
• Street South 1
• Street South 2
• Street East 1
• Street East 2
• Street West 1
• Street West 2
• o Geo Coordinates (latitude and longitude of the intersection)
The web scraper should target the specific site at https://www.siouxfalls.gov/health-safety/police/Police-Activity-and-Safety-Resources/police-call-resources/call-log (click the call log button).
Basic description:
2 stages: Stage 1 – Simple web scraper that processes the first 25 pages of the site and checks the table, if the record is not already in the present, it adds it, if it is, it skips and moves on to the next one.
I don’t need any display of records, I just need the scraper to put the data into the table. And I also need the scraper php script capable of running as a CPANEL cron script goal is to run it ever 8 hours.
Stage 2 – takes the closest intersection, and based on it, will get the next 2 streets to the north, south, east, west. And store them all in a table tied to the incident PK.
Again, there is no need to display or pull any data out, I just need data placed into the tables.
Tables:
Table 1: Incident
• Primary Key
• Date
• Time
• Street Name
• Closest Intersection
• Call Type
• Incident Number (this is what to check against for preexisting record)
• CFS Dispo
Table 2: NearIntSects
• Primary key
• IncidentId (tied to PK of Incident table for specific Incident)
• Street North 1
• Street North 2
• Street South 1
• Street South 2
• Street East 1
• Street East 2
• Street West 1
• Street West 2
• o Geo Coordinates (latitude and longitude of the intersection)
The web scraper should target the specific site at https://www.siouxfalls.gov/health-safety/police/Police-Activity-and-Safety-Resources/police-call-resources/call-log (click the call log button).