Webscraping / Interface | Python/HTML/CSS/JS
Budget: €1,500 – €3,000 EUR
bullet points
Knowledge required in:
- Python (web scraping), Flask
- HTML/CSS/JS
Tasks Web scraping code and interface
- Create automatic web scraping of online database (parameters can be entered in interface)
- Create automatic web scraping/downloading of a specific file, unzipping
- Create search terms on a web page and download content directly on the web page
- Create automatic web scraping. Overlay integrated map on external screenshots (everything) and images in '/admin/addmapdata'
'/admin/addmapdata' is described in more detail below
- Further development download all files 1 layer of a web page
- Download evolution of all files of all layers of a website
- Create possibility to enter code for web scraping in interface
- Add more web scraping types and enter configuration options in the interface
- External indication of the data update (when should the system download this data again and check for changes)
- If a csv/xlsv is downloaded, in the interface after selecting the file format
can set the column e.g. 1 all addresses are (example) etc. the variables
should be predefined by the system, e.g. a limited choice, but easy
is expandable
- If other pdf or other file formats are downloaded, there should also be setting options
what kind of data is e.g. sample file 'monument list.pdf' here the text arranged on the left is the number
the bold text on the right is the address and the non-bold text on the right is the description
- there is the '/admin/webscraping/add-process' in which is the interface to create a webscraping,
I want this interface to have a button that goes to '/admin/addmapdata'
the link and the graphic interface as well as the functions have already been created. if you add during the
of a webscraping process on '/admin/webscraping/add-process' clicks on the button, then you can click on '/admin/addmapdata'
drag image files onto the map, move/distort/draw lines and polygons on the map
and also add information to these polygons. if you then click on Done in '/admin/addmapdata', you should go back to
'/admin/webscraping/add-process' come back what you have just created and the data (position of the images on the map and polygons
are then also saved (with coordinates)) as mentioned, '/admin/addmapdata' is ready, it only has to be in '/admin/webscraping/add-process'
built in and linked. each web scraping process receives a folder in which the images, coordinates and polygons can be saved
It should also be possible to adapt/change/add further data to such a web scraping process by entering the process in
'/admin/webscraping' click edit. there at '/admin/webscraping' is the list of all webscraping processes that are in progress, active or show an error
- in '/admin/webscraping', where all processes are listed, you should be able to filter and unlock entries so that they are active
are displayed. only web scraping processes that are active are taken into account when outputting data
- in '/admin/addmapdata' you can drag images onto mapbox and place them over certain areas
I would like the system to analyze the images and create polygons with coordinate points from them and save them in the database
the images will look like img 1.jpg or img 2.jpg or img 3.jpg or img 4.jpg
- There should be a button at '/admin/webscraping/add-process' that tests how/which data arrives, is saved and displays this before you
then click on 'Create'
- the system should extract web scraping texts from e.g. analyze for addresses and check which text belongs to which address and assign it accordingly in the database
website
- Revision of the design for the user and admin dashboard, see the desired design image, but must match the product page
which mainly has the map card to be aligned (top menu) left menu opens like
product selection from left to right. integration web scraping interface right and serv
- Further development of the selection page for products and payment page (Html/CSS/JS)
Knowledge required in:
- Python (web scraping), Flask
- HTML/CSS/JS
Tasks Web scraping code and interface
- Create automatic web scraping of online database (parameters can be entered in interface)
- Create automatic web scraping/downloading of a specific file, unzipping
- Create search terms on a web page and download content directly on the web page
- Create automatic web scraping. Overlay integrated map on external screenshots (everything) and images in '/admin/addmapdata'
'/admin/addmapdata' is described in more detail below
- Further development download all files 1 layer of a web page
- Download evolution of all files of all layers of a website
- Create possibility to enter code for web scraping in interface
- Add more web scraping types and enter configuration options in the interface
- External indication of the data update (when should the system download this data again and check for changes)
- If a csv/xlsv is downloaded, in the interface after selecting the file format
can set the column e.g. 1 all addresses are (example) etc. the variables
should be predefined by the system, e.g. a limited choice, but easy
is expandable
- If other pdf or other file formats are downloaded, there should also be setting options
what kind of data is e.g. sample file 'monument list.pdf' here the text arranged on the left is the number
the bold text on the right is the address and the non-bold text on the right is the description
- there is the '/admin/webscraping/add-process' in which is the interface to create a webscraping,
I want this interface to have a button that goes to '/admin/addmapdata'
the link and the graphic interface as well as the functions have already been created. if you add during the
of a webscraping process on '/admin/webscraping/add-process' clicks on the button, then you can click on '/admin/addmapdata'
drag image files onto the map, move/distort/draw lines and polygons on the map
and also add information to these polygons. if you then click on Done in '/admin/addmapdata', you should go back to
'/admin/webscraping/add-process' come back what you have just created and the data (position of the images on the map and polygons
are then also saved (with coordinates)) as mentioned, '/admin/addmapdata' is ready, it only has to be in '/admin/webscraping/add-process'
built in and linked. each web scraping process receives a folder in which the images, coordinates and polygons can be saved
It should also be possible to adapt/change/add further data to such a web scraping process by entering the process in
'/admin/webscraping' click edit. there at '/admin/webscraping' is the list of all webscraping processes that are in progress, active or show an error
- in '/admin/webscraping', where all processes are listed, you should be able to filter and unlock entries so that they are active
are displayed. only web scraping processes that are active are taken into account when outputting data
- in '/admin/addmapdata' you can drag images onto mapbox and place them over certain areas
I would like the system to analyze the images and create polygons with coordinate points from them and save them in the database
the images will look like img 1.jpg or img 2.jpg or img 3.jpg or img 4.jpg
- There should be a button at '/admin/webscraping/add-process' that tests how/which data arrives, is saved and displays this before you
then click on 'Create'
- the system should extract web scraping texts from e.g. analyze for addresses and check which text belongs to which address and assign it accordingly in the database
website
- Revision of the design for the user and admin dashboard, see the desired design image, but must match the product page
which mainly has the map card to be aligned (top menu) left menu opens like
product selection from left to right. integration web scraping interface right and serv
- Further development of the selection page for products and payment page (Html/CSS/JS)