Top 10 Selenium Projects With Code Examples

Define Selenium?

Selenium is a browser automation library you can use to automate some processes. It is a web automation library used to automate processes in a web browser such as logging in to a website, web scraping for data mining, and automating searching prompts.

Selenium is one of the most known skills on demand since it ties to the ability of data mining and data analysis. You can use the Selenium WebDriver to scrape through a website and mine all data, instead of manually inputting data and putting them in an Excel file. This is one of the reasons why if someone plans to go into the field of Data Science, they should learn how to use Selenium since it is an essential tool for Data Mining. Learn more on Selenium with Python with us!

There are a few advantages of using Selenium for your projects. They are:

Low cost: Selenium is an open-source software which means, anyone can use it for free with little to no cost.

Less time: Automating browser processes will save more time compared to manual processes, and it is more accurate too.

Uses of Selenium

Selenium can automate mundane tasks, auto-fill forms, open and close tabs, and check for links automatically. But, one of its most important uses is in web-scraping for data mining from different websites. This is used to reduce the time taken to mine data from different resources manually. These are the main positives of using Selenium.

Installation

As a browser automation library, Selenium is compatible with multiple languages such as Python, Java, C# and so on, where you can import the library Selenium into the code and start using it. Here, we’ll be covering how to use Selenium with Python here:

To download Selenium in your system, check if you have Python downloaded in your system. To do this, simply type the following command in your command prompt.

Common Propmt

    python --version
    

After checking the Python version, we need to install the Selenium library in the system. The Command Prompt code is shown below.

Common Propmt

    pip install selenium
    

After installing the Selenium library we can use it for web automation. But, before that, let us go through some prerequisites.

Requirements for the system before using selenium

To use Selenium web browser automation for your browser, it is important to download the web drivers for your respective main browsers that you want to automate. The links to download the drivers are given below.

Microsoft Edge: Click Here

Google Chrome: Click Here

Note: Only the web drivers are downloaded differently for each browser. After that, The coding part stays the same for all.

In this project, we’ll be using Microsoft Edge as the main browser for automation. Let’s see more about browser drivers.

Browser Drivers

Browser drivers are tools which provide authorization for the Selenium web automation library to work and allow web scraping in their browsers. The main browsers used for web scraping using Selenium are:

• Google Chrome: Chromedriver
• Microsoft Edge: Edgedriver
• Brave: Chromedriver

Google Chrome and Microsoft Edge provide web drivers you can use to automate web processes. The Brave browser is powered by Chromium from Google Chrome, hence, we can use Chromedriver for web automation in Brave.

Important: Once you’ve downloaded the browser web-drivers it is important to accurately configure it to an accessible path, preferably one where you don’t need to have administrative privileges.

Our Lovely Student feedback

Browser Elements

Selenium Projects Using Python

Here are 10 selenium projects you can use by selecting HTML elements as shown above:

Example -1: Browser Automation with Google

In this example, we can automate opening Google and inserting in queries to find the answer and print the resultant. Here, we’re seeing how to find the Independence days of four countries:

In Selenium, web automation is done by selecting browser elements in a website running in a browser. For example, let us go to the website Google and see its HTML elements.

To do so, we can click left click and choose the inspect option or, we can press Ctrl+Shift+C and we will see the HTML elements running Google.

As we can see, we can view the HTML and CSS running the site. And, if we hover over the website elements, we can see it highlights the HTML class responsible for running that specific part.

With all this, let us see through some short example projects below.

• USA
• India
• Sri Lanka
• Pakistan

Code:

    import time
    from selenium import webdriver
    from selenium.webdriver.common.by import By
    from selenium.webdriver.edge.service import Service
    
    countries = ['US' , 'India' , 'Sri Lanka' , 'Pakistan']
    
    # Provide the correct path to msedgedriver.exe
    s = Service("C:\\Program Files\\msedgedriver.exe")
    browser = webdriver.Edge(service=s)
    
    for country in countries: 
    try:
    browser.get('https://www.google.com/')
    query = 'https://www.google.com/''When is independence day for '+ country + '?'
    
    # Locate the search input field by name 'q'
    text_field = browser.find_element(by=By.NAME, value='q'
    text_field.clear()             # Clear any existing text in the field
    text_field.send_keys(query)
    text_field.submit()
    
    time.sleep(2)
    result = browser.find_element(by=By.XPATH, value="//div[@class='BNeawe iBp4i AP7Wnd']")
    print(f"Independence day for {country}: {result.text}")
    time.sleep(2)
    except Exception as e:
    print(f"An error occurred for {country}: {str(e)}")
    
    # Close the browser when done
    browser.quit() 
    

In this code, by referencing the HTML element of the Google Search bar, we can input the Independence days to be printed. If we run the code, an instance of Microsoft Edge will be opened and we can print the answers by referencing the HTML class it is a part of. It comes along with a header, “Microsoft Edge is being automated by a test software”.

The Output in Command Prompt:

    DevTools listening on ws://127.0.0.1:55137/devtools/browser/55f1327d-1b9c-4b45-8d42-372c56f022a0
    Tue, 4 Jul, 2023
    Tue, 15 Aug, 2023
    Sun, 4 Feb, 2024
    Mon, 14 Aug, 2023
    

Example -2: Copy Page Details into another page

Let us see how we can copy the text of a website in Microsoft Edge, and how we can print its contents using Selenium. Here, we open a website using Selenium and copy its contents into a file, then show it.

Code:

    from selenium import webdriver
    import time
    from selenium.webdriver.edge.service import Service
    countries = ['US' , 'India' , 'Sri Lanka' , 'Pakistan']
    
    # Set the path to the msedgedriver.exe
    driver_path = "C:\\Program Files\\msedgedriver.exe"
    try:
    
    # Initialize the Edge browser
    s = Service(driver_path)
    browser = webdriver.Edge(service=s) 
    
    # Navigate to the URL 
    browser.get('"https://www.btreesystems.com/selenium-with-python-training-in-chennai/"')
    
    # Wait for the page to load (you can use WebDriverWait for more robust waiting)
    time.sleep(2)
    
    # Get the page source and encode it as UTF-8
    page_source = browser.page_source.encode('utf-8')
    
    # Use a with statement to handle file operations
    with open('result.html', 'wb') as f:
    f.write(page_source)
    except Exception  as  e:
    print(f"An error occurred: {str(e)}")
    finally: 
    
    # Make sure to close the browser, even if an exception occurs   
    browser.quit()
    

This code creates a file called “result.html” by copying all HTML text of the given website. This is saved in the folder you’re working in.

Here, we’re utilizing the file-handling concept to encode it to ‘utf-8’, then we open the file of the website in HTML and write the HTML code in the file, hence creating the result.html file.

OUTPUT:

The HTML code of the website is shown below:

Example -3: Check whether a Given Word exists or Not

This project idea deals with the fact that you can check whether a word exists on the website or not. It returns “True” if it does, “False” if it doesn’t exist.

Code:

    from selenium import webdriver
    import time
    from selenium.webdriver.edge.service import Service
    s = Service("C:\Program Files\msedgedriver.exe")
    browser = webdriver.Edge(service=s) 
    
    browser.get('https://www.btreesystems.com/selenium-with-python-training-in-chennai/')
    
    get_source=browser.page_source
    search_text="Selenium"
    print(search_text in get_source)
    

OUTPUT:

    True                  
    

Example -4: Find the number of links present in a Website

When you want to find the total number of links embedded in a given website, you can use Selenium to find the HTML tags to find the number of links by finding the HTML tag “a”. Generally, in HTML, the “href” tag is used to embed links and “a” is the object used to embed links.

Code:

    from selenium import webdriver
    from selenium.webdriver.edge.service import Service
    from selenium.webdriver.common.by import By
    s = Service("C:\Program Files\msedgedriver.exe")
    browser = webdriver.Edge(service=s) 
    
    browser.get('https://www.btreesystems.com/')
    
    link=browser.find_elements(By.TAG_NAME, 'a')
    print(len(link))
    

After finding the links, we print the number of links present on the website.

OUTPUT:

    124
    The number of links embedded in the website is 124.                   
    

Example -5: Opening and Closing Tabs in the Browser

This project shows how to automate opening, redirecting to a tab, and closing tabs. Let us see how to do so below:

Code:

    from selenium import webdriver
    from selenium.webdriver.edge.service import Service
    import time
    s = Service("C:\Program Files\msedgedriver.exe")
    browser = webdriver.Edge(service=s) 
    
    url="https://www.google.com/"
    url="https://www.btreesystems.com/"
    
    browser.get(url1)
    browser.execute_script("window.open('');")
    browser.switch_to.window(browser.window_handles[1])
    browser.get(url)
    time.sleep(2)
    
    browser.close()
    browser.switch_to.window(browser.window_handles[0])
    

After opening the browser, for the first link, we automate opening another tab and switch to the new window using the switch_to.window() function. To that tab, we redirect it to the second website. We then close the first tab, then switch to the current tab.

Here, two tabs are opened, and after 2 seconds, the Google tab is closed and we switch over to the first website.

Example -6: Automate Scrolling on a Website

This project shows how to automate scrolling in a web browser in Python using Selenium.

Code:

    from selenium import webdriver
    import time
    from selenium.webdriver.edge.service import Service
    s = Service("C:\Program Files\msedgedriver.exe")
    browser = webdriver.Edge(service=s) 
    
    driver.get("https://www.countries-ofthe-world.com/flags-of-the-world.html")
    driver.execute_script("window.scrollBy(0,2000)","")
    time.sleep(2)
    

This opens the flags of the website and scrolls them by 2000 pixels. The output is shown below: www.countries-ofthe-world.com/flags-of-the-world.html

Example -7: Automate Tests

Other than browser automation, we can use Selenium to automate tests for a given search to a website. Here, we’re searching on the python.org website to test using geckodriver or the unittest library in Python.

This library has some use cases to test the Selenium automation and tests based on the predefined unit test cases in the library.

Code:

    from selenium import webdriver
    from selenium.webdriver.edge.service import Service
    from selenium.webdriver.common.by import By
    import unittest
    
    class PythonOrgSearch (unittest.TestCase):
    def setUp  (self):
    self.s = Service("C:\Program Files\msedgedriver.exe")
    self.driver = webdriver.Edge(service=self.s)
    
    def test_search_in_python_org (self):
    driver = self.driver
    driver.get("http://www.python.org")
    self.assertIn("Python", driver.title)
    elem = driver.find_element(by=By.XPATH, value='//*[@id="id-search-field"]')
    elem.send_keys("pycon")
    
    if __name__ =="__main__":
    unittest.main() 
    

Here, we define a class to run the tests on and send keys to the search bar. This will return “No results” for this example and the testing will be successful.

OUTPUT:

    ------------------------------------------------------------------------------
    Ran a test in 6.538s
    Ok                       
    

Example -8: Automate Form Filling

Do you find it annoying to fill out Google Forms all the time? Now, it is possible to automate filling out the form using the Selenium library in Python. By referring to the HTML elements used for input in Google Forms and sending prompts, we can automate filling forms in Google and submitting them.

Code:

    from selenium import webdriver
    from selenium.webdriver.common.by import By
    import time
    from selenium.webdriver.edge.serviceimport Service
    
    s = Service("C:\Program Files\msedgedriver.exe")
    browser = webdriver.Edge(service=s) 
    driver.get("https://docs.google.com/forms/d/e/1FAIpQLSezs9vfDnGxHrXsF52bxKXdrmlQHbHI0HsxrO6A9PGbC3K0Xw/viewform?usp=sf_link")
    
    time.sleep(2)
    
    Name="Prabhu Deva"
    n=driver.find_element(by=By.XPATH,
    
    value'//*[@id="mG61Hd"]/div[2]/div/div[2]/div[1]/div/div/div[2]/div/div[1]/div/div[1]/input')
    n.send_keys(Name)
    
    age="25"
    n=driver.find_element(by=By.XPATH,
    
    value'//*[@id="mG61Hd"]/div[2]/div/div[2]/div[1]/div/div/div[2]/div/div[1]/div/div[1]/input')
    n.send_keys(age)
    
    email="prabhudeva76@gmail.com"
    n=driver.find_element(by=By.XPATH,
    
    value'//*[@id="mG61Hd"]/div[2]/div/div[2]/div[1]/div/div/div[2]/div/div[1]/div/div[1]/input')
    n.send_keys(email)
    
    email="India"
    n=driver.find_element(by=By.XPATH,
    
    value'//*[@id="mG61Hd"]/div[2]/div/div[2]/div[1]/div/div/div[2]/div/div[1]/div/div[1]/input')
    n.send_keys(email)
    
    email="Chennai"
    n=driver.find_element(by=By.XPATH,
    
    value'//*[@id="mG61Hd"]/div[2]/div/div[2]/div[1]/div/div/div[2]/div/div[1]/div/div[1]/input')
    n.send_keys(email)
    
    email="Teynampet"
    n=driver.find_element(by=By.XPATH,
    
    value'//*[@id="mG61Hd"]/div[2]/div/div[2]/div[1]/div/div/div[2]/div/div[1]/div/div[1]/input')
    n.send_keys(email)
    
    submit=driver.find_element(by=By.XPATH,
    
    value'//*[@id="mG61Hd"]/div[2]/div/div[2]/div[1]/div/div/div[2]/div/div[1]/div/div[1]/input')
    submit.click()
    

By finding the HTML element by hovering over it, we can copy the XPATH code of the Input variable to automate form filling

To copy the XPATH code, simply right-click the code and select Copy as XPATH.

After doing so, we can input in the XPATH our code to send prompts to the Google form, including the submit button which we can click using the click() function in Selenium. This results in the output shown below:

Example -9: Browser Automation For Google Maps

This innovative idea performs web automation of the browser, where we open Google Maps, and enter our location, which automatically shows us the map location and the directions to travel to it.

    g_link="https://www.google.com/maps/@13.0478078,80.0442021,11z?entry=ttu"
    ip_link='//*[@id="searchboxinput"]'
    dire='//*[@id="hArJGc"]/div'
    
    import time
    from selenium import webdriver
    from selenium.webdriver.common.by import By
    from selenium.webdriver.edge.service import Service
    
    s = Service("C:\Program Files\msedgedriver.exe")
    browser = webdriver.Edge(service=s)
    
    browser.get(g_link)
    time.sleep(2)
    
    name=str(input("Enter Location: "))
    n=browser.find_element(by=By.XPATH, value=ip_link)
    n.send_keys(name)
    directions=browser.find_element(by=By.XPATH, value=dire)
    directions.click()
    

By HTML inspect element, we find the input box to send our input prompt and also the direction icon to show us the travel time which we can make Selenium web automation software click automatically without any manual prompt. The output is shown below:

OUTPUT:

    Enter location: Bangalore
    

Thus, we have successfully automated Google Maps using Selenium.

Example -10: Sign in Automatically on LinkedIn

This project makes use of signing in websites and automating them using Selenium. Let’s see how to do that below:

Code:

    import time
    from selenium import  webdriver
    from selenium.webdriver.common.by import By                      
    from selenium.webdriver.edge.service import Service
    
    s = Service("C:\Program Files\msedgedriver.exe")
    browser = webdriver.Edge(service=s) 
    
    browser.get("https://www.linkedin.com")
    username=browser.find_element(By.XPATH, '//*[@id="session_key"]')
    username.send_keys("EMAIL")   #Enter Your Email
    username=browser.find_element(By.XPATH,  '//*[@id="session_password"]')
    username.send_keys("PASSWORD") #Enter Your Password
    button=browser.find_element(By.XPATH, '//*[@id="main-content"]/section[1]/div/div/form/div[2]/button')
    
    time.sleep(2)
    button.click()
    

Here, we start by opening the LinkedIn page which will ask you to Sign In. Previously, we learned how to copy XPATH variables for the input boxes to input in your username and password and also to click the button “Sign In”. This will automatically input the username and password you’ve put in the code. The output is shown below:

OUTPUT:

Here, we can see that we’ve successfully logged in to LinkedIn.

NOTE: Some websites like Instagram and even LinkedIn sometimes block Selenium requests due to growing concerns about data privacy. Hence, the code may not work if you run it one too many times.

Syntax Updates

As a result of technological advancements, all websites including the browser go through updates due to which there is a high chance that the Python syntax for the Selenium library may change in the future, either deprecating the previous functions to encourage developers to accept the new syntax or rendering them obsolete.

For Example:

    browser.find_element_by_xpath(value)
    

This syntax has been deprecated and now the new syntax is

    from browser.find_element_by_xpath(value) import By
    browser.find_element(By.XPATH, 'value' )
    

These are small examples of syntax changes that have happened over time.

Future Scope of these Selenium Projects

These Selenium projects with code examples serve as a great introduction to the web automation library Selenium. Selenium has been instrumental as a base automation tool to perform web-scraping, browser automation, search automation and sending in requests. All of these play a huge role in automating the more mundane tasks that developers need to do daily. But, recently, due to many restrictions on data scraping on websites as a result of protecting user identity, many websites, like Twitter have prohibited the limit of scraping tweets and also made it difficult to log in using a browser automation tool due to the presence of different types of CAPTCHA texts to prevent bots or users with malicious intent to scrape sensitive data and sell it to the highest bidder.

As a result, to combat unwanted scraping, but also to make data mining easier, such websites have started providing APIs to assist in data mining easier without having to resort to web scraping.

Conclusion

From the above Selenium project ideas, we can see how versatile and loaded with functionalities on Selenium projects with code examples. Since it is a web automation library, websites are constantly changing their HTML codes while bringing new updates to their website. Despite its useful functionalities, its major drawback is that the syntax keeps constantly changing and many more websites introduce CAPTCHAs to prevent the growing data scraping problem. There is also a concern over the ethics of web scraping where the website owner may not have any idea that his website is being scraped for data, and now only some websites allow scraping by Selenium.