text extraction from a local collection of html files
Budget: $30 – $250 CAD
I have a zip file with 1000 folders and inside each folder is an html document. The text in each html document has a common format as follows:
<p class="para0"><span class="span0">first line</span>
>all text however formatted<
<p class="para1"><span class="span0">first heading:</span>
>all text however formatted<
<p class="para1"><span class="span0">second heading:</span>
>all text however formatted<
<p class="para1"><span class="span0">third heading:</span>
>all text however formatted<
<p class="para1"><span class="span0">fourth heading:</span>
>all text however formatted<
<p class="para1"><span class="span0">fifth heading:</span>
>all text however formatted<
<p class="para1"><span class="span0">sixth heading:</span>
>all text however formatted<
Each appearance of para1/span0 is a heading, and the text block that appears below it is the content associated with that heading. Together they are a pair. The first line and text that follows until the first heading is not considered a pair.
Each document will have a different number of headings and pairs. This is only an example. Some may have fewer than 6 headings and pairs and some may have many more.
The objective is to extract all data from each html document into a csv with 4 columns where column 1 is the file path, column 2 is the first line of the document and usually but not always appears as para0/span0, column 3 is the text from the para1/span0 line, and column 4 is the all the text that appears below that line up to the next appearance of a para1/span0 line.
Where a document has a first line and 6 headings like the example above, the csv should look like this
Filepath/ABC, first line, first heading, text paired with first heading
Filepath/ABC, first line, second heading, text paired with second heading
Filepath/ABC, first line, third heading, text paired with third heading
Filepath/ABC, first line, fourth heading, text paired with fourth heading
Filepath/ABC, first line, fifth heading, text paired with fifth heading
Filepath/ABC, first line, sixth heading, text paired with sixth heading
The output of all documents should go into a single csv table. The end result should be a table with 4 columns and between 8,000 and 15,000 rows. Some documents may not have the described formatting. It is acceptable for these to fail and be skipped.
To the greatest extent possible, the spacing and formatting of the text paired with each heading should be preserved as the csv will serve as an input into a web page.
<p class="para0"><span class="span0">first line</span>
>all text however formatted<
<p class="para1"><span class="span0">first heading:</span>
>all text however formatted<
<p class="para1"><span class="span0">second heading:</span>
>all text however formatted<
<p class="para1"><span class="span0">third heading:</span>
>all text however formatted<
<p class="para1"><span class="span0">fourth heading:</span>
>all text however formatted<
<p class="para1"><span class="span0">fifth heading:</span>
>all text however formatted<
<p class="para1"><span class="span0">sixth heading:</span>
>all text however formatted<
Each appearance of para1/span0 is a heading, and the text block that appears below it is the content associated with that heading. Together they are a pair. The first line and text that follows until the first heading is not considered a pair.
Each document will have a different number of headings and pairs. This is only an example. Some may have fewer than 6 headings and pairs and some may have many more.
The objective is to extract all data from each html document into a csv with 4 columns where column 1 is the file path, column 2 is the first line of the document and usually but not always appears as para0/span0, column 3 is the text from the para1/span0 line, and column 4 is the all the text that appears below that line up to the next appearance of a para1/span0 line.
Where a document has a first line and 6 headings like the example above, the csv should look like this
Filepath/ABC, first line, first heading, text paired with first heading
Filepath/ABC, first line, second heading, text paired with second heading
Filepath/ABC, first line, third heading, text paired with third heading
Filepath/ABC, first line, fourth heading, text paired with fourth heading
Filepath/ABC, first line, fifth heading, text paired with fifth heading
Filepath/ABC, first line, sixth heading, text paired with sixth heading
The output of all documents should go into a single csv table. The end result should be a table with 4 columns and between 8,000 and 15,000 rows. Some documents may not have the described formatting. It is acceptable for these to fail and be skipped.
To the greatest extent possible, the spacing and formatting of the text paired with each heading should be preserved as the csv will serve as an input into a web page.