Beautiful Soup Web Scraping Tutorial

Required Tools And Knowledge

• BeautifulSoup4
• Pandas

There are innumerable websites available that provide a lot of numeric or text information. Before starting with a scraping code, we need to identify what data we are going to scrape from the website. That will help us aim at those particular sections of the web page while coding.

For example, opencodez.com provides us with several posts on various technologies. I want to create an excel containing the title of all the articles written, the short paragraph, its author, date, and the web link to those articles. The below screenshot shows the sections I need to target in my code.

So Great But What About Attributes

This whole tutorial has basically now shown basic traversal, but what if we want to read some attribute data. So let me show you how using the same code we have above we can do this, but reading another part of that html.

I want to now go and add the url to the full post in our data we scraped as well. For that we will need to read one of the href attributes. Lets look at how we can retrieve that if we look at the article html.

Here we have a class blog-entry-readmore, follwed by an a tag with the href we looking for. So lets just add this bit of code to our script.

link = article.find.a     print

So pretty simple. We can use article then find the class and traverse to the first a tag then access the attributes using the href index in the dictionary beautifulsoup builds up for us.

So if we for arguement sake wanted the title from that same a tag we could use this code.

    linktitle = article.find.a    print

So really really easy to get those as well.

Understanding And Inspecting The Data

Now that you know about basic HTML and its tags, you need to first do the inspection of the page which you want to scrape. Inspection is the most important job in web scraping without knowing the structure of the webpage, it is very hard to get the needed information. To help with inspection, every browser like Google Chrome or Mozilla Firefox comes with a handy tool called developer tools.

In this guide, we will be working with wikipedia to scrap some of its table data from the page List of countries by GDP . This page contains a Lists heading which contains three tables of countries sorted by their rank and its GDP value as per “International Monetary Fund”, “World Bank”, and “United Nations”. Note, that these three tables are enclosed in an outer table.

To know about any element that you wish to scrape, just right-click on that text and examine the tags and attributes of the element.

Also Check: Olive Garden Endless Soup And Salad

Finding Elements Using Beautifulsoup

As we saw in the first example, the Beautiful Soup object needs HTML. For web scraping static sites, the HTML can be retrieved using requests library. The next step is parsing this HTML string into the BeautifulSoup object.

response = requests.getbs = BeautifulSoup

Let ‘s find out how to scrape a dynamic website with BeautifulSoup.

The following part remains unchanged from the previous example.

from selenium.webdriver import Chromedriver = Chromedriver.get

The rendered HTML of the page is available in the attribute page_source.

soup = BeautifulSoup

Once the soup object is available, all Beautiful Soup methods can be used as usual.

Note: The complete source code is in selenium_bs4.py

How To Scrape A Website Using Python Beautiful Soup And

Beautiful Soup Web Scraping And Parsing Using Python

Beautiful Soup Tutorial Web Scraping In Python All

With python’s open source beautiful soup library, you can get data by scraping any part or element of a webpage with maximum control over the process. in this article, we look at how you can use beautiful soup to scrape a website. how to install beautiful soup and get started with it. before we proceed, in this beautiful soup tutorial article. # login to website using just python 3 standard library import urllib.parse import urllib.request import http.cookiejar def scraper login: ##### change variables here, like url, action url, user, pass # your base url here, will be used for headers and such, with and without base url = ‘ example ‘ https base url = ‘https. Image scraping with python. last updated : 08 sep, 2021. scraping is a very essential skill for everyone to get data from any website. in this article, we are going to see how to scrape images from websites using python. for scraping images, we will try different approaches. method 1: using beautifulsoup and requests.

How To Scrape A Website With Beautiful Soup And Python

Python Tutorial: Web Scraping With Beautifulsoup And Requests

in this python programming tutorial, we will be learning how to scrape websites using the beautifulsoup library. beautifulsoup is an excellent tool for parsing

Read Also: Red Potatoes And Lipton Onion Soup Mix

Why There Is A Need To Scrape The Data

Lets say if you are looking for a specific product on . You just dont want to buy at any cost. You want to buy when there is a certain discount in price. Amazon announces offers on that product now and then. You keep checking it every day but thats not the productive way to spend your time.

When you need to extract large data from websites that are frequently updated with new content you end up spending a lot of time searching, scrolling, and clicking. Manual web scraping is repetitive work and you end up spending a lot of time on that.

Here Python comes to your rescue. Instead of checking the price of the product every day, you can write a Python script to automate this repetitive process. To speed up the data collection process the solution is Automated web scraping. You are going to write your Python script only once nad the script will fetch you the information you need as many as times and from as many pages you want.

In the world of the internet, a lot of new content is uploaded every second. You can collect those data, whether its for job search for personal use, automated web scraping will help you to achieve your goal.

Pulling The Contents From A Tag

In order to access only the actual artists names, well want to target the contents of the < a> tags rather than print out the entire link tag.

We can do this with Beautiful Soups .contents, which will return the tags children as a Python list data type.

Lets revise the for loop so that instead of printing the entire link and its tag, well print the list of children :

import requestsfrom bs4 import BeautifulSouppage = requests.getsoup = BeautifulSouplast_links = soup.findlast_links.decomposeartist_name_list = soup.findartist_name_list_items = artist_name_list.find_all# Use .contents to pull out the < a>  tags childrenfor artist_name in artist_name_list_items:names = artist_name.contentsprint

Note that we are iterating over the list above by calling on the index number of each item.

We can run the program with the python command to view the following output:

OutputZabaglia, NiccolaZaccone, FabianZadkine, Ossip...Zanini-Viola, GiuseppeZanotti, GiampietroZao Wou-Ki

We have received back a list of all the artists names available on the first page of the letter Z.

However, what if we want to also capture the URLs associated with those artists? We can extract URLs found within a pages < a> tags by using Beautiful Soups get method.

From the output of the links above, we know that the entire URL is not being captured, so we will concatenate the link string with the front of the URL string .

These lines well also add to the for loop:

Read Also: Does Lipton Onion Soup Mix Have Msg

An Overview Of Beautiful Soup

The HTML content of the webpages can be parsed and scraped with Beautiful Soup. In the following section, we will be covering those functions that are useful for scraping webpages.

What makes Beautiful Soup so useful is the myriad functions it provides to extract data from HTML. This image below illustrates some of the functions we can use:

Let’s get hands-on and see how we can parse HTML with Beautiful Soup. Consider the following HTML page saved to file as doc.html:

< html> < head> < title> Head's title< /title> < /head> < body> < pclass="title"> < b> Body's title< /b> < /p> < pclass="story"> line begins< ahref="http://example.com/element1"class="element"id="link1"> 1< /a> < ahref="http://example.com/element2"class="element"id="link2"> 2< /a> < ahref="http://example.com/avatar1"class="avatar"id="link3"> 3< /a> < p>  line ends< /p> < /body> < /html> 

The following code snippets are tested on Ubuntu 20.04.1 LTS. You can install the BeautifulSoup module by typing the following command in the terminal:

$ pip3 install beautifulsoup4

The HTML file doc.html needs to be prepared. This is done by passing the file to the BeautifulSoup constructor, let’s use the interactive Python shell for this, so we can instantly print the contents of a specific part of a page:

from bs4 import BeautifulSoupwithopen as fp:    soup = BeautifulSoup

Now we can use Beautiful Soup to navigate our website and extract data.

Navigating to Specific Tags

From the soup object created in the previous section, let’s get the title tag of doc.html:

Scrape A Website With This Beautiful Soup Python Tutorial

Interested in web scraping? Here’s how to scrape a website for content and more with the Beautiful Soup Python library.

Beautiful Soup is an open-source Python library. It uses navigating parsers to scrape the content of XML and HTML files. You need data for several analytical purposes. However, if you’re new to Python and web scraping, Python’s Beautiful Soup library is worth trying out for a web scraping project.

With Python’s open-source Beautiful Soup library, you can get data by scraping any part or element of a webpage with maximum control over the process. In this article, we look at how you can use Beautiful Soup to scrape a website.

You May Like: Aldi Cream Of Chicken Soup

Let’s Scrape The Page

Now we have that whole web page in our hands . One of the two jobs left is to scrape it. So let’s do that. We need to scrape the following things from the web page:

• titles – all the movie titles
• genres – all the genres
• grosses – all the gross information
• img_urls – src URLs of all the image.

So let’s do them one by one.

First, let’s scrape all the titles:

The title we are looking for is inside an HTML element called < h3> . Wait, how do I know that?

It’s simple:

• Open up the URL you want to scrape for inside a browser.

In our case, open this TopMovies website. Then:

• Inspect the data with the help of Developer Tools on your browser. In my case, I am using Chrome, so

• right-click on the element you want to scrape,
• And click on Inspect

• Now a new box will pop up like this:

• And See, I told you that the title we are looking for is inside an HTML element called < h3>

Now we know where our data is sitting, let’s scrape it. Type this below the last line you typed:

""" first, scraping using find_all method """# scrape all the titlestitles =for h3 in soup.find_all:  titles.append)

Here’s the line by line explanation of the above code:

Whoo, there’s a lot going on in there. So please take a moment to understand it. This is the exact step we are going to repeat from here on to scrape all the other data.

And the reason why we are storing our scraped data inside a Python list is that it will be a lot easier to convert those lists into a CSV file and that’s why we are doing that.

Extract Text From Html Elements

You only want to see the title, company, and location of each job posting. And behold! Beautiful Soup has got you covered. You can add .text to a Beautiful Soup object to return only the text content of the HTML elements that the object contains:

forjob_elementinjob_elements:title_element=job_element.findcompany_element=job_element.findlocation_element=job_element.findprintprintprintprint

Run the above code snippet, and youll see the text of each element displayed. However, its possible that youll also get some extra whitespace. Since youre now working with Python strings, you can .strip the superfluous whitespace. You can also apply any other familiar Python string methods to further clean up your text:

forjob_elementinjob_elements:title_element=job_element.findcompany_element=job_element.findlocation_element=job_element.findprint)print)print)print

The results finally look much better:

Senior Python DeveloperPayne, Roberts and DavisStewartbury, AAEnergy engineerVasquez-DavidsonChristopherville, AALegal executiveJackson, Chambers and LevyPort Ericaburgh, AA

Thats a readable list of jobs that also includes the company name and each jobs location. However, youre looking for a position as a software developer, and these results contain job postings in many other fields as well.

Recommended Reading: Campbell Broccoli Rice Casserole

Enhancing Your Webscraping Skills

In the real world of data science, itll often be a task that we have to obtain some or all of our data. As much as we love to work with clean, organized datasets from Kaggle, that perfection is not always replicated in day-to-day tasks at work. Thats why knowing how to scrape data is a very valuable skill to possess, and today Im going to demonstrate how to do just that with images, along with eventually displaying your image results in a Pandas DataFrame.

To start, Im going to scrape from the website that I first learned to scrape images from, which is books.toscrape.com. This is a great site to practice all of your scraping skills on, not just image scraping. Now, the first thing youll want to do is import some necessary packages BeautifulSoup and requests.

from bs4 import BeautifulSoupimport requests

Next, youll want to make a get request to retrieve your webpage and then pass the contents of the page through BeautifulSoup so that it can be parsed.

html_page = requests.getsoup = BeautifulSoupwarning = soup.findbook_container = warning.nextSibling.nextSibling

Awesome! Now, we need our images. Being efficient with BeautifulSoup means having a little bit of experience and/or understanding of HTML tags. But if you dont, using Google to find out which tags you need in order to scrape the data you want is pretty easy. Since we want image data, well use the img tag with BeautifulSoup.

images = book_container.findAllexample = imagesexample

example.attrs

Using Output In Various Ways

Now we have our desired CSV. We can do some exploratory data analysis on this data to see for example the number of articles written by each author or to make a yearly analysis on the number of articles. We can also create a word cloud from the corpus of a brief description column to see the most used words in the posts. These will be dealt with in the next post.

Also Check: Blender That Makes Hot Soup And Ice Cream

Word Of Caution For Web Scraping

The legality of this practice is not well defined, however. Websites usually describe in their terms of use and in their robots.txt file if they allow scrapers or not. So please be careful not to tread in restricted territories or not to hit the URL with a huge number of requests in a short duration causing issues with the website itself

I hope you found it useful.

In the next article, we will see what we can do with scrapped data. Please stay tuned!

Is Web Scraping Legal

Unfortunately, theres not a cut-and-dry answer here. Some websites explicitly allow web scraping. Others explicitly forbid it. Many websites dont offer any clear guidance one way or the other.

Before scraping any website, we should look for a terms and conditions page to see if there are explicit rules about scraping. If there are, we should follow them. If there are not, then it becomes more of a judgement call.

Remember, though, that web scraping consumes server resources for the host website. If were just scraping one page once, that isnt going to cause a problem. But if our code is scraping 1,000 pages once every ten minutes, that could quickly get expensive for the website owner.

Thus, in addition to following any and all explicit rules about web scraping posted on the site, its also a good idea to follow these best practices:

Web Scraping Best Practices:

• Never scrape more frequently than you need to.
• Consider caching the content you scrape so that its only downloaded once.
• Build pauses into your code using functions like time.sleep to keep from overwhelming servers with too many requests too quickly.

In our case for this tutorial, the NWSs data is public domain and its terms do not forbid web scraping, so were in the clear to proceed.

You May Like: Can You Have Tomato Soup On Keto Diet

Using Find & Find: All

The most straightforward way to finding information in our soup variable is by utilizing soup.find or soup.find_all. These two methods work the same with one exception: find returns the first HTML element found, whereas find_all returns a list of all elements matching the criteria .

We can search for DOM elements in our soup variable by searching for certain criteria. Passing a positional argument to find_all will return all anchor tags on the site:

soup.find_all# < a href="http://example.com/elsie" class="boy" id="link1"> Elsie< /a> # < a href="http://example.com/lacie" class="boy" id="link2"> Lacie< /a> # < a href="http://example.com/tillie" class="girl" id="link3"> Tillie< /a> 

We can also find all anchor tags which have the class name “boy”. Passing the class_ argument allows us to filter by class name. Note the underscore!

soup.find_all# < a href="http://example.com/elsie" class="boy" id="link1"> Elsie< /a> # < a href="http://example.com/lacie" class="boy" id="link2"> Lacie< /a> 

If we wanted to get any element with the class name “boy” besides anchor tags, we can do that too:

soup.find_all# < a href="http://example.com/elsie" class="boy" id="link1"> Elsie< /a> # < a href="http://example.com/lacie" class="boy" id="link2"> Lacie< /a> 

We can search for elements by id in the same way we searched for classes. Remember that we should only expect a single element to be returned with an id, so we should use find here:

soup.find# < a href="http://example.com/elsie" class="boy" id="link1"> Elsie< /a> 

Web Scraping Dynamic Sites By Locating Ajax Calls

Loading the browser is expensiveit takes up CPU, RAM, and bandwidth which are not really needed. When a website is being scraped, it’s the data that is important. All those CSS, images, and rendering are not really needed.

The fastest and most efficient way of scraping dynamic web pages with Python is to locate the actual place where the data is located.

There are two places where this data can be located:

• The main page itself, in JSON format, embedded in a < script> tag
• Other files which are loaded asynchronously. The data can be in JSON format or as partial HTML.

Let’s look at few examples.

Recommended Reading: Chicken And Rice Campbell Soup Recipe