Small Python crawler projet + CLI
Budget: €6 – €12 EUR
I have a small Python project. I dont know how many hours is needed for a dev.
I would like to know if someone you can be interested by my project.
There's some requirements.
Create a command line program that will browse a website and extract information from it. You will need to send the code of your
program with a file indicating the required dependencies, as well as a README.md to specify the use of the program and some
examples of use cases.
This program must be usable on the command line and offer the following features
Return a report to the terminal
$ crawler [--url|u] https://example.net
Return a report in a file passed in parameter
$ crawler [--url|u] https://example.net [--export] file_name
The generated report will be in text format and should contain the following information:
The number of unique URLs
URL's pointing to the same domain
Addresses pointing to an external domain
Addresses containing forms (all forms)
Addresses containing password protected pages
The number of addresses pointing to the same domain name returning a 404 page
Return only pages that return a 404 error code
$ crawler [--url|u] https://example.net --404
Return only addresses pointing to an external domain name
$ crawler [--url|u] https://example.net --external-url
Return only pages that require authentication
$ crawler [--url|u] https://example.net --protected_url
Add '-user' and 'password' parameters trying to auto-fill the connection fields
$ crawler [--url|u] https://example.net --user user --password password
Specify in your README.md file how you determine which POST parameters to transmit for successful authentication.
Your code will have to be composed of at least two files, one containing the logic of the CLI part, the other containing the logic to browse a Website and extract information from it.
├── README.md
├── cli.py
├── crawler.py
└── requirements.txt
Best regards.
F.
I would like to know if someone you can be interested by my project.
There's some requirements.
Create a command line program that will browse a website and extract information from it. You will need to send the code of your
program with a file indicating the required dependencies, as well as a README.md to specify the use of the program and some
examples of use cases.
This program must be usable on the command line and offer the following features
Return a report to the terminal
$ crawler [--url|u] https://example.net
Return a report in a file passed in parameter
$ crawler [--url|u] https://example.net [--export] file_name
The generated report will be in text format and should contain the following information:
The number of unique URLs
URL's pointing to the same domain
Addresses pointing to an external domain
Addresses containing forms (all forms)
Addresses containing password protected pages
The number of addresses pointing to the same domain name returning a 404 page
Return only pages that return a 404 error code
$ crawler [--url|u] https://example.net --404
Return only addresses pointing to an external domain name
$ crawler [--url|u] https://example.net --external-url
Return only pages that require authentication
$ crawler [--url|u] https://example.net --protected_url
Add '-user' and 'password' parameters trying to auto-fill the connection fields
$ crawler [--url|u] https://example.net --user user --password password
Specify in your README.md file how you determine which POST parameters to transmit for successful authentication.
Your code will have to be composed of at least two files, one containing the logic of the CLI part, the other containing the logic to browse a Website and extract information from it.
├── README.md
├── cli.py
├── crawler.py
└── requirements.txt
Best regards.
F.