Scrape data from website to MS Power BI using PBI Python
Budget: $250 – $750 USD
To execute this project, the freelancer will need the following skill and experience set:
Proficiency in Python programming language
Experience with web scraping using Python libraries such as BeautifulSoup or Selenium
Knowledge of Power BI and its integration with Python
Experience with incremental data refresh in Power BI
Understanding of website structure and HTML elements
The project will involve the following steps:
Setting up the Python environment and importing the necessary libraries:
Installing necessary Python libraries such as BeautifulSoup, Selenium, and Pandas
Importing the libraries into the script
Setting up the web driver for Selenium
Creating a script to navigate to the website and select the start and end dates:
Using Selenium to open the website "https://www.realestate.moj.gov.kw/live/Moj_Rs_11.aspx"
Locating the start and end date pickers on the website using Selenium's find_element_by_* methods
Entering the start and end dates into the date pickers
Ticking the 3 boxes located below the date pickers to select all options of their related drop down lists:
Locate the 3 boxes using Selenium's find_element_by_* methods
Use Selenium's click() method to tick the boxes
Clicking the retrieve button to retrieve the data and storing it in a variable:
Locate the retrieve button using Selenium's find_element_by_* method
Use Selenium's click() method to click the button
Use BeautifulSoup to parse the HTML of the page and extract the table data
Store the table data in a variable for further processing
Looping through the pagination links and retrieving the data from each page:
Locate the pagination links using Selenium's find_elements_by_* method
Iterate through the pagination links and click on each one
Retrieve the table data on each page using BeautifulSoup and store it in the variable
Continue looping until the end of the pages is reached
Cleaning and formatting the data for better understanding:
Use Pandas to convert the data into a dataframe
Perform any necessary cleaning and formatting on the data, such as removing null values or renaming columns
Integrating the data with Power BI for visualization and analysis:
Use Power BI's Python integration to import the data into a report
Create visualizations and perform analysis on the data
Setting up incremental refresh in Power BI to update the data daily:
Use Power BI's incremental refresh feature to schedule the data to be updated daily
Configure the refresh settings and test the refresh process to ensure it is working properly.
The estimated hours required for this project is around 40-50 hours, assuming 2 revision rounds with the project owner. This includes time for debugging and testing the script, as well as integration with Power BI and setting up incremental refresh.
Proficiency in Python programming language
Experience with web scraping using Python libraries such as BeautifulSoup or Selenium
Knowledge of Power BI and its integration with Python
Experience with incremental data refresh in Power BI
Understanding of website structure and HTML elements
The project will involve the following steps:
Setting up the Python environment and importing the necessary libraries:
Installing necessary Python libraries such as BeautifulSoup, Selenium, and Pandas
Importing the libraries into the script
Setting up the web driver for Selenium
Creating a script to navigate to the website and select the start and end dates:
Using Selenium to open the website "https://www.realestate.moj.gov.kw/live/Moj_Rs_11.aspx"
Locating the start and end date pickers on the website using Selenium's find_element_by_* methods
Entering the start and end dates into the date pickers
Ticking the 3 boxes located below the date pickers to select all options of their related drop down lists:
Locate the 3 boxes using Selenium's find_element_by_* methods
Use Selenium's click() method to tick the boxes
Clicking the retrieve button to retrieve the data and storing it in a variable:
Locate the retrieve button using Selenium's find_element_by_* method
Use Selenium's click() method to click the button
Use BeautifulSoup to parse the HTML of the page and extract the table data
Store the table data in a variable for further processing
Looping through the pagination links and retrieving the data from each page:
Locate the pagination links using Selenium's find_elements_by_* method
Iterate through the pagination links and click on each one
Retrieve the table data on each page using BeautifulSoup and store it in the variable
Continue looping until the end of the pages is reached
Cleaning and formatting the data for better understanding:
Use Pandas to convert the data into a dataframe
Perform any necessary cleaning and formatting on the data, such as removing null values or renaming columns
Integrating the data with Power BI for visualization and analysis:
Use Power BI's Python integration to import the data into a report
Create visualizations and perform analysis on the data
Setting up incremental refresh in Power BI to update the data daily:
Use Power BI's incremental refresh feature to schedule the data to be updated daily
Configure the refresh settings and test the refresh process to ensure it is working properly.
The estimated hours required for this project is around 40-50 hours, assuming 2 revision rounds with the project owner. This includes time for debugging and testing the script, as well as integration with Power BI and setting up incremental refresh.