Web Scraping and REST APIs

External Resources

• More on HTTP status codes - https://developer.mozilla.org/en-US/docs/Web/HTTP/Status

• Beautiful Soup documentation - https://www.crummy.com/software/BeautifulSoup/bs4/doc/

• More on HTML - https://developer.mozilla.org/en-US/docs/Web/HTML

• Tutorial on HTML: https://www.htmldog.com/guides/html/

• Tutorial on CSS: https://www.htmldog.com/guides/css/

• More on REST APIs - https://www.smashingmagazine.com/2018/01/understanding-using-rest-api/

• More on JSON - https://www.digitalocean.com/community/tutorials/an-introduction-to-json

• Websites that offer well-documented REST APIs to access data from the site in JSON format:

  • GitHub: https://docs.github.com/en/rest/reference/repos

  • Facebook: https://developers.facebook.com/docs/groups-api/reference

  • Twitter: https://developer.twitter.com/en/docs/twitter-api/v1/tweets/timelines/api-reference/get-statuses-user_timeline

  • Reddit: https://www.reddit.com/dev/api/

• More on GitHub API - https://towardsdatascience.com/all-the-things-you-can-do-with-github-api-and-python-f01790fca131

• Scrapy installation - https://scrapy.org/

• More examples of Scraping:

  • https://medium.com/@msalmon00/web-scraping-job-postings-from-indeed-96bd588dcb4b

  • https://medium.com/the-innovation/scraping-medium-with-python-beautiful-soup-3314f898bbf5

  • https://medium.com/brainstation23/how-to-become-a-pro-with-scraping-youtube-videos-in-3-minutes-a6ac56021961

  • https://www.freecodecamp.org/news/web-scraping-python-tutorial-how-to-scrape-data-from-a-website/

  • https://www.freecodecamp.org/news/scraping-wikipedia-articles-with-python/

  • https://towardsdatascience.com/web-scraping-yahoo-finance-477fe3daa852

  • https://www.analyticsvidhya.com/blog/2020/10/web-scraping-selenium-in-python/

  • https://medium.com/ml-book/web-scraping-using-selenium-python-3be7b8762747

• Python Tutorial at W3Schools - https://www.w3schools.com/python/

• Practical Python Programming - https://dabeaz-course.github.io/practical-python/Notes/Contents.html

• Python official documentation - https://docs.python.org/3/tutorial/index.html

Common Questions

Q1: Is API different from REST API?

A: Yes. All REST APIs are APIs but not all APIs are REST APIs. API is a set of protocols that allows one application to communicate with another application using an interface. REST API is a type of API that has client-server communication i.e. client sends a request of service and the server provides the information.

Q2: For the Web Scraping project, the list of requirements include "Your project should involve web scraping of at least two web pages". Does that mean two different websites or rather can it be two webpages from the same website domain?

A: It is preferable if the two webpages are of same website but it is not mandatory. You can also scrape two different websites.

Q3: In Project 1, is it okay to import Pandas and use its own method to create a CSV or do we need to write our own function?

A: It is okay to import Pandas and create a CSV.

Q4: While Extracting information from HTML using BeautifulSoup, what is the significance of htmlparser in this command: doc = BeautifulSoup(h_source,‘html.parser’)?

A: Beautiful Soup can work with many different file formats: HTML, XML, YML etc. Since web pages are generally HTML documents, we specify html.parser as the method Beautiful Soup should use to parse i.e. read & understand the document. It is also the default setting, so if you skip the first argument, BeautifulSoup automatically picks html.parser

Here’s an example of using Beautiful Soup for parsing XML:

https://linuxhint.com/parse_xml_python_beautifulsoup/

Q5: Why should one scrape using BeautifulSoup when something automated like autoscraper is available (apart from scrapy and selenium). Any advantage that BeautifulSoup specifically offers?

A: You can use whichever library you find most convenient and easy to use for the job at hand. The objective of the web scraping lesson was to show you the power of Python for parsing and extracting information from a web page.The reason we picked beautiful soup is that it offers really simple & easy to understand functions, doesn’t have very high RAM or CPU requirements (unlike Selenium), and has great documentation and support on StackOverflow.

Q6: What does status-code mean?

A: Every time browser makes a request to server, the status code tells if the request has been successful or not by the server to the user.

Q7: What is the difference between find_all and findAll?

A: They both are same. Just different styles of writing.

Q8: What happens when you get status-code as 400 or 404?

A: You can check out the link to know what each status code means- https://developer.mozilla.org/en-US/docs/Web/HTTP/Status

Q9: Can we extract data from a mobile game using the approach discussed in the lecture?

A: No, that is a different concept. You cannot use this approach to scrape mobile data, this can be used only for web-sites.

Q10: Which IP address do the websites target when we scrape data using Binder?

A: This will be the IP address of the Binder instance somewhere on the cloud.

Overview recording: https://share.descript.com/view/Sr4Rf0m1q0i