Key Concepts

Review core concepts you need to learn to master this subject

Beautiful Soup Object and Methods

Beautiful Soup Traversal

Beautiful Soup Object and Methods

# This line of code creates a BeautifulSoup object from a webpage:

soup = BeautifulSoup(webpage.content, "html.parser")

# Within the `soup` object, tags can be called by name:

first_div = soup.div

# or by CSS selector:

all_elements_of_header_class = soup.select(".header")

# or by a call to `.find_all`:

all_p_elements = soup.find_all("p")

BeautifulSoup uses a parser to take in the content of a webpage. It provides tree traversal and advanced searching methods. It creates an object from the website contents.

Web Scraping with Beautiful Soup
Lesson 1 of 1
  1. 1
    Introduction
    Before we get started, a quick note on prerequisites: This course requires knowledge of Python . Also some understanding of the Python library Pandas will be helpful later on in the lesson, …
    Start
  2. 2
    Rules of Scraping
    When we scrape websites, we have to make sure we are following some guidelines so that we are treating the websites and their owners with respect. Always check a website’s Terms and Conditions bef…
    Start
  3. 3
    Requests
    In order to get the HTML of the website, we need to make a request to get the content of the webpage. To learn more about requests in a general sense, you can check out this article . Python ha…
    Start
  4. 4
    The BeautifulSoup Object
    When we printed out all of that HTML from our request, it seemed pretty long and messy. How could we pull out the relevant information from that long string? BeautifulSoup is a Python library that…
    Start
  5. 5
    Object Types
    BeautifulSoup breaks the HTML page into several types of objects. #### Tags A Tag corresponds to an HTML Tag in the original document. These lines of code: soup = BeautifulSoup(‘ An example di…
    Start
  6. 6
    Navigating by Tags
    To navigate through a tree, we can call the tag names themselves. Imagine we have an HTML page that looks like this: World’s Best Chocolate Chip Cookies Ingredients 1 cup flour …
    Start
  7. 7
    Website Structure
    When we’re telling our Python script what HTML tags to grab, we need to know the structure of the website and what we’re looking for. Many browsers, including Chrome, Firefox, and Safari, have Dev…
    Start
  8. 8
    Find All
    If we want to find all of the occurrences of a tag, instead of just the first one, we can use .find_all(). This function can take in just the name of a tag and returns a list of all occurrences o…
    Start
  9. 9
    Select for CSS Selectors
    Another way to capture your desired elements with the soup object is to use CSS selectors. The .select() method will take in all of the CSS selectors you normally use in a .css file! Search Resul…
    Start
  10. 10
    Reading Text
    When we use BeautifulSoup to select HTML elements, we often want to grab the text inside of the element, so that we can analyze it. We can use .get_text() to retrieve the text inside of whatever ta…
    Start
  11. 11
    Review
    Amazing! Now you know the basics of how to use BeautifulSoup to turn websites into data. If you take our Data Visualization or Data Manipulation courses, you can see how you might analyze this …
    Start

What you'll create

Portfolio projects that showcase your new skills

Pro Logo

How you'll master it

Stress-test your knowledge with quizzes that help commit syntax to memory

Pro Logo