Question

Write a simple python program to crawl 10 pages from the web and store the text content in the pages into csv file. You code, the 10 html pages, and the csv file which store the content in the pages should be submitted.

Tips: you can write your own program by referencing this tutorial: https://codeburst.io/scraper-b82146396249

Tips2:

(1) If you use python2 on your computer, you can use the following code to get web pages:

import urllib2

page = urllib2.urlopen('https://www.pythonforbeginners.com/')

print page.info()
print page.read()

(2) If you use python3 on your computer, you can use the following code to get web pages:

import urllib.request
page = urllib.request.urlopen('https://www.pythonforbeginners.com/')

print (page.info())
print (page.read())

Answer 1

Here the page https://www.pythonforbeginners.com is scrapped using the module import urllib.request, python 3.6. Inside a for loop, the page is paginated with the pagination value, and scrapped the web page. This content is written to the sites.csv file in /tmp directory.(Please provide your path to the csv file), using csv module.

#CODE STARTS HERE

import csv #Import csv module
import urllib.request #Import csv urllib

def siteCrawler(): #Method starts here
f = csv.writer(open('/tmp/sites.csv', 'w')) #create a file in directory /tmp named sites.csv in write mode[Provide your path of csv file]
for x in range(1, 11): #loops 1 to 10
page=urllib.request.urlopen('https://www.pythonforbeginners.com/?page='+str(x))# Crwal the site with pagination
print(page.read())#print the page content
f.writerow(page.read())#write the page content to the sites.csv file

#CODE ENDS HERE