Pull executive compensation across 6704 publicly traded companies from SEC Edgar website
Budget: $250 – $750 USD
I want to put together a table of all the executive compensation tables from the 6704 publicly traded companies that report information to the SEC through the Edgar database. Every company is required to fill out "Summary Compensation Table" in the same format within their proxy filing for the top 5 most paid officers over the last 3 years.
If you want to see an example of this summary table, do this:
1) Go to SEC Edgar Company Filings website (https://www.sec.gov/edgar/searchedgar/companysearch.html)
2) Type in and search any CIK.
3) Click to open: [+] Proxy (annual meeting) and information statements
4) Click on the first link under: [+] Proxy (annual meeting) and information statements
5) CTRL + F: "Summary Compensation Table"
6) The first column of the table should be: "Name and Principal Position". The second column is "Year". The third is "Salary". There should be 9 total columns.
I want my output to have these 9 columns plus 1 column for CIK, 1 column for the hyperlink from which the information was pulled from, and 1 column for Revenue. Each search should have upto 15 rows: 5 executive officers with 3 years each. Sometimes, executive officers have not been with the company for 3 years so they will have less.
I have attached an Excel file. The first tab is what the table in the SEC Edgar website should look like. The second tab is what I hope for the output to look like. The first tab has the 6704 CIKs we want to search. I have discovered recently that not every CIK has a proxy filing. This must be because my source included publicly traded companies without operations and filings. I'm sure they would return errors or blanks in the search so it is okay to leave for now.
Here are some other sources to be aware of:
1) The SEC website has a limit where you can only do 10 requests per second. There are also some other legal restrictions outlined in these links. (a) https://www.sec.gov/os/accessing-edgar-data (b) https://www.sec.gov/privacy.htm#security
2) This website lays out the exact code to scrape the URL of each company. However, the challenge is that his code is a manual input of each company, but I want to run multiple companies at once. This website also shows how to scrape Balance Sheet and financial information, but I want Executive Compensation information. (a) https://codingandfun.com/scraping-sec-edgar-python/
3) The SEC website provides some other information as well. (a) https://www.sec.gov/dera/data/financial-statement-data-sets.html (b) https://www.sec.gov/dera/data/financial-statement-and-notes-data-set.html
Happy to answer any questions for this project! I hope you can help me!
If you want to see an example of this summary table, do this:
1) Go to SEC Edgar Company Filings website (https://www.sec.gov/edgar/searchedgar/companysearch.html)
2) Type in and search any CIK.
3) Click to open: [+] Proxy (annual meeting) and information statements
4) Click on the first link under: [+] Proxy (annual meeting) and information statements
5) CTRL + F: "Summary Compensation Table"
6) The first column of the table should be: "Name and Principal Position". The second column is "Year". The third is "Salary". There should be 9 total columns.
I want my output to have these 9 columns plus 1 column for CIK, 1 column for the hyperlink from which the information was pulled from, and 1 column for Revenue. Each search should have upto 15 rows: 5 executive officers with 3 years each. Sometimes, executive officers have not been with the company for 3 years so they will have less.
I have attached an Excel file. The first tab is what the table in the SEC Edgar website should look like. The second tab is what I hope for the output to look like. The first tab has the 6704 CIKs we want to search. I have discovered recently that not every CIK has a proxy filing. This must be because my source included publicly traded companies without operations and filings. I'm sure they would return errors or blanks in the search so it is okay to leave for now.
Here are some other sources to be aware of:
1) The SEC website has a limit where you can only do 10 requests per second. There are also some other legal restrictions outlined in these links. (a) https://www.sec.gov/os/accessing-edgar-data (b) https://www.sec.gov/privacy.htm#security
2) This website lays out the exact code to scrape the URL of each company. However, the challenge is that his code is a manual input of each company, but I want to run multiple companies at once. This website also shows how to scrape Balance Sheet and financial information, but I want Executive Compensation information. (a) https://codingandfun.com/scraping-sec-edgar-python/
3) The SEC website provides some other information as well. (a) https://www.sec.gov/dera/data/financial-statement-data-sets.html (b) https://www.sec.gov/dera/data/financial-statement-and-notes-data-set.html
Happy to answer any questions for this project! I hope you can help me!