An Introduction to Web Scraping in Python for Beginners

An Introduction to Web Scraping in Python for Beginners


Published at - Nov 15, 2021

What is Web Scraping?

Web scraping, also known as web harvesting or web data extraction, is a type of data scraping that is used to gather information from websites. Using the Hypertext Transfer Protocol or a web browser, web scraping software may directly access the World Wide Web.

Applications of Web Scraping

  • Lead Generation for Marketing

  • Price Comparison & Competition Monitoring

  • E-Commerce

  • Real Estate

  • Data Analysis

  • Academic Research

  • Training and Testing Data for Machine Learning Projects

  • Sports Betting Odds Analysis

Python provides BeautifulSoup and Scrapy to perform web scraping easily. You can learn more about web scraping using beautiful soup here and Scrapy here.

In this tutorial, I am going to provide a simple example of BeautifulSoup.

BeautifulSoup Example

In the example, we need to install the BeautifulSoup package with the help of pip :

pip install beautifulsoup4

Then we have to import packages which will be used to perform various operation

import os # to perform system related actions

import requests # fetch data from url in html, css and js

from bs4 import BeautifulSoup #to parse data

Next, we have to get a response from the given URL, I am scrapping google.com

url = "https://www.google.com/"

reponse = requests.get(url) # getting raw data from url

Then we have to extract data from the response with BeautifulSoup

if reponse.ok:

    soup = BeautifulSoup(reponse.text, "lxml")

    title = str(soup.find("title"))

    title = title.replace("<title>", "")

    title = title.replace("</title>", "")

    print("The title is : " + str(title))

Let’s see combined code into one file:

<iframe src="https://medium.com/media/162b75af9a43e15e5b02278eb068fb03" frameborder=0></iframe>

Summary

In this tutorial, we have learned about how to extract data from a website and parse it using the BeautifulSoup python package. I hope this will help you get started with your journey of web scraping. There are tons of uses of web scraping nowadays.

Thank you for reading this tutorial. Don’t forget to clap 👏 and follow me to read more articles like this in the future.

More content at plainenglish.io





About author

Harendra
Harendra Kanojiya

Hello, I am Harendra Kumar Kanojiya - Owner of this website and a Fullstack web developer. I have expertise in full-stack web development using Angular, PHP, Node JS, Python, Laravel, Codeigniter and, Other web technologies. I also love to write blogs on the latest web technology to keep me and others updated. Thank you for reading the articles.



Follow Us

Follow us on facebook Click Here

Facebook QR
Scan from mobile
Join our telegram channel Click Here
Telegram QR
Scan from mobile