Make Python script massively more performant
Budget: $30 – $250 USD
I am looking for a skilled developer who can optimize the efficiency of a Python script. The goal is to make the script run as fast as possible (ideally, 10x faster or more ... if that is possible).
The script is attached for your examination - so please assess it before bidding.
The script is used for the purpose of parsing WARC files from the Common Crawl dataset (publicly available on AWS S3). Right now, the script takes about 90 minutes to process a single WARC file from the Common Crawl dataset, when running on an EC2 Ubuntu 20.04 machine; the instance I've been using is a t3.large.
The script needs to parse files WAY faster than it currently does, as it is too expensive for the script to take 90 minutes per file of processing. Common crawl has ~90,000 WARC files per crawl, so that is too many hours to parse a crawl; this script simply has to be much faster.
In short, iterates through each WARC record in the WARC file, extracts hyperlinks from each WARC record, and then outputs two files related to hyperlinks. As long as your script produces the logical equivalent output (and handles errors, etc, since raw web data is such a mess to deal with), I am not completely tied to the libraries used in the script -- as long as it can be run via python3. So ... feel free to take a different approach (multithreading?), use different libraries, or both -- as long as the end result is a script that outputs the same data, but does it WAY faster on the same cost compute resource.
As for some sample files, I make this call in Ubuntu when I am ready to run the script:
!/bin/bash
aws s3 cp s3://commoncrawl/crawl-data/CC-MAIN-2023-40/segments/1695233505362.29/warc/CC-MAIN-20230921073711-20230921103711-00001.warc.gz /home/ubuntu/
aws s3 cp s3://commoncrawl/crawl-data/CC-MAIN-2023-40/segments/1695233505362.29/warc/CC-MAIN-20230921073711-20230921103711-00002.warc.gz /home/ubuntu/
aws s3 cp s3://commoncrawl/crawl-data/CC-MAIN-2023-40/segments/1695233505362.29/warc/CC-MAIN-20230921073711-20230921103711-00003.warc.gz /home/ubuntu/
aws s3 cp s3://commoncrawl/crawl-data/CC-MAIN-2023-40/segments/1695233505362.29/warc/CC-MAIN-20230921073711-20230921103711-00004.warc.gz /home/ubuntu/
aws s3 cp s3://commoncrawl/crawl-data/CC-MAIN-2023-40/segments/1695233505362.29/warc/CC-MAIN-20230921073711-20230921103711-00005.warc.gz /home/ubuntu/
aws s3 cp s3://commoncrawl/crawl-data/CC-MAIN-2023-40/segments/1695233505362.29/warc/CC-MAIN-20230921073711-20230921103711-00006.warc.gz /home/ubuntu/
aws s3 cp s3://commoncrawl/crawl-data/CC-MAIN-2023-40/segments/1695233505362.29/warc/CC-MAIN-20230921073711-20230921103711-00010.warc.gz /home/ubuntu/
aws s3 cp s3://commoncrawl/crawl-data/CC-MAIN-2023-40/segments/1695233505362.29/warc/CC-MAIN-20230921073711-20230921103711-00023.warc.gz /home/ubuntu/
aws s3 cp s3://commoncrawl/crawl-data/CC-MAIN-2023-40/segments/1695233505362.29/warc/CC-MAIN-20230921073711-20230921103711-00029.warc.gz /home/ubuntu/
python3 process_warc_file.py --local-file CC-MAIN-20230921073711-20230921103711-00001.warc.gz >> /home/ubuntu/process_output/warc00001.log 2>&1
python3 process_warc_file.py --local-file CC-MAIN-20230921073711-20230921103711-00002.warc.gz >> /home/ubuntu/process_output/warc00002.log 2>&1
python3 process_warc_file.py --local-file CC-MAIN-20230921073711-20230921103711-00003.warc.gz >> /home/ubuntu/process_output/warc00003.log 2>&1
python3 process_warc_file.py --local-file CC-MAIN-20230921073711-20230921103711-00004.warc.gz >> /home/ubuntu/process_output/warc00004.log 2>&1
python3 process_warc_file.py --local-file CC-MAIN-20230921073711-20230921103711-00005.warc.gz >> /home/ubuntu/process_output/warc00005.log 2>&1
python3 process_warc_file.py --local-file CC-MAIN-20230921073711-20230921103711-00006.warc.gz >> /home/ubuntu/process_output/warc00006.log 2>&1
python3 process_warc_file.py --local-file CC-MAIN-20230921073711-20230921103711-00010.warc.gz >> /home/ubuntu/process_output/warc00010.log 2>&1
python3 process_warc_file.py --local-file CC-MAIN-20230921073711-20230921103711-00023.warc.gz >> /home/ubuntu/process_output/warc00023.log 2>&1
python3 process_warc_file.py --local-file CC-MAIN-20230921073711-20230921103711-00029.warc.gz >> /home/ubuntu/process_output/war
^^ That is probably a good test set for you to run the script against.
The script is attached for your examination - so please assess it before bidding.
The script is used for the purpose of parsing WARC files from the Common Crawl dataset (publicly available on AWS S3). Right now, the script takes about 90 minutes to process a single WARC file from the Common Crawl dataset, when running on an EC2 Ubuntu 20.04 machine; the instance I've been using is a t3.large.
The script needs to parse files WAY faster than it currently does, as it is too expensive for the script to take 90 minutes per file of processing. Common crawl has ~90,000 WARC files per crawl, so that is too many hours to parse a crawl; this script simply has to be much faster.
In short, iterates through each WARC record in the WARC file, extracts hyperlinks from each WARC record, and then outputs two files related to hyperlinks. As long as your script produces the logical equivalent output (and handles errors, etc, since raw web data is such a mess to deal with), I am not completely tied to the libraries used in the script -- as long as it can be run via python3. So ... feel free to take a different approach (multithreading?), use different libraries, or both -- as long as the end result is a script that outputs the same data, but does it WAY faster on the same cost compute resource.
As for some sample files, I make this call in Ubuntu when I am ready to run the script:
!/bin/bash
aws s3 cp s3://commoncrawl/crawl-data/CC-MAIN-2023-40/segments/1695233505362.29/warc/CC-MAIN-20230921073711-20230921103711-00001.warc.gz /home/ubuntu/
aws s3 cp s3://commoncrawl/crawl-data/CC-MAIN-2023-40/segments/1695233505362.29/warc/CC-MAIN-20230921073711-20230921103711-00002.warc.gz /home/ubuntu/
aws s3 cp s3://commoncrawl/crawl-data/CC-MAIN-2023-40/segments/1695233505362.29/warc/CC-MAIN-20230921073711-20230921103711-00003.warc.gz /home/ubuntu/
aws s3 cp s3://commoncrawl/crawl-data/CC-MAIN-2023-40/segments/1695233505362.29/warc/CC-MAIN-20230921073711-20230921103711-00004.warc.gz /home/ubuntu/
aws s3 cp s3://commoncrawl/crawl-data/CC-MAIN-2023-40/segments/1695233505362.29/warc/CC-MAIN-20230921073711-20230921103711-00005.warc.gz /home/ubuntu/
aws s3 cp s3://commoncrawl/crawl-data/CC-MAIN-2023-40/segments/1695233505362.29/warc/CC-MAIN-20230921073711-20230921103711-00006.warc.gz /home/ubuntu/
aws s3 cp s3://commoncrawl/crawl-data/CC-MAIN-2023-40/segments/1695233505362.29/warc/CC-MAIN-20230921073711-20230921103711-00010.warc.gz /home/ubuntu/
aws s3 cp s3://commoncrawl/crawl-data/CC-MAIN-2023-40/segments/1695233505362.29/warc/CC-MAIN-20230921073711-20230921103711-00023.warc.gz /home/ubuntu/
aws s3 cp s3://commoncrawl/crawl-data/CC-MAIN-2023-40/segments/1695233505362.29/warc/CC-MAIN-20230921073711-20230921103711-00029.warc.gz /home/ubuntu/
python3 process_warc_file.py --local-file CC-MAIN-20230921073711-20230921103711-00001.warc.gz >> /home/ubuntu/process_output/warc00001.log 2>&1
python3 process_warc_file.py --local-file CC-MAIN-20230921073711-20230921103711-00002.warc.gz >> /home/ubuntu/process_output/warc00002.log 2>&1
python3 process_warc_file.py --local-file CC-MAIN-20230921073711-20230921103711-00003.warc.gz >> /home/ubuntu/process_output/warc00003.log 2>&1
python3 process_warc_file.py --local-file CC-MAIN-20230921073711-20230921103711-00004.warc.gz >> /home/ubuntu/process_output/warc00004.log 2>&1
python3 process_warc_file.py --local-file CC-MAIN-20230921073711-20230921103711-00005.warc.gz >> /home/ubuntu/process_output/warc00005.log 2>&1
python3 process_warc_file.py --local-file CC-MAIN-20230921073711-20230921103711-00006.warc.gz >> /home/ubuntu/process_output/warc00006.log 2>&1
python3 process_warc_file.py --local-file CC-MAIN-20230921073711-20230921103711-00010.warc.gz >> /home/ubuntu/process_output/warc00010.log 2>&1
python3 process_warc_file.py --local-file CC-MAIN-20230921073711-20230921103711-00023.warc.gz >> /home/ubuntu/process_output/warc00023.log 2>&1
python3 process_warc_file.py --local-file CC-MAIN-20230921073711-20230921103711-00029.warc.gz >> /home/ubuntu/process_output/war
^^ That is probably a good test set for you to run the script against.