How to get all the links in webpage in Selenium Python?
Get all the Links in Webpage in Selenium Python
To get all the links present in the webpage using Selenium in Python, call find_elements() method on the webdriver object, and pass By.TAG_NAME as selector, and "a" for value.
driver.find_elements(By.TAG_NAME, 'a')The above method call returns all the link elements in the webpage as a list, List[WebElement] object.
Example
In this example, we shall consider loading the HTML file at path /tmp/selenium/index-32.html . The contents of this HTML file is given below.
<html>
<body>
<p>Hello World! Read <a id="about_link" href="/tmp/selenium/index-31-about.html">About Hello World</a>.</p>
<br>
<p>Latest Articles</p>
<ul>
<li><a id="new_article" href="/tmp/selenium/index-31-new-article.html">About New Article</a></li>
</ul>
</body>
</html>
Now, we shall get all the links (anchor elements) with the tag name as a. And print them to standard output.
Python Program (Selenium)
from selenium import webdriver
from selenium.webdriver.chrome.service import Service as ChromeService
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By
# Setup chrome driver
driver = webdriver.Chrome(service=ChromeService(ChromeDriverManager().install()))
# Navigate to the url
driver.get('/tmp/selenium/index-32.html')
# Find all link elements
links = driver.find_elements(By.TAG_NAME, 'a')
# Iterate over link elements
for link in links:
print(link.get_attribute('outerHTML'))
# Close the browser
driver.quit()Output
<a id="about_link" href="/tmp/selenium/index-31-about.html">Hello World</a>
<a id="new_article" href="/tmp/selenium/index-31-new-article.html">New Article 1</a>Summary
In this Python Selenium tutorial, we learned how to get all the HTML link elements in a webpage, using find_elements() method in Selenium.