When we scrape websites, we have to make sure we are following some guidelines so that we are treating the websites and their owners with respect.
Always check a website’s Terms and Conditions before scraping. Read the statement on the legal use of data. Usually, the data you scrape should not be used for commercial purposes.
Do not spam the website with a ton of requests. A large number of requests can break a website that is unprepared for that level of traffic. As a general rule of good practice, make one request to one webpage per second.
If the layout of the website changes, you will have to change your scraping code to follow the new structure of the site.
Create a variable called
respect that holds the string
"I will not spam websites".