Question

A' Read aloud A Big-Data Processing Task: We need to find out 15 most frequently used words on a set of Wikipedia pages. Specifically, we need to find out a list of these words and the number of occurrences of each word on the pages. The list should be sorted in descending order based on the number of occurrences. The following is a sample of output generated for 4 Wikipedia pages. 1080 the 677 of 480 in 473 and 443 a 375 advertising 345 to 191 from 189 on 164 is 160 for 149 or 133 as 128 by 126 that Since there are a huge number of pages in Wikipedia, it is not realistic to analyze all of them in short time on one machine. Here we only need to analyze all the pages for the Wikipedia entries with two capital letters. For example, the Wikipedia page for entry "AC" is https://en.wikipedia.org/wiki/AC. Thus, the pages we need to analyze are https://en.wikipedia.org/wiki/AA https://en.wikipedia.org/wiki/AB https://en.wikipedia.org/wiki/AC https://en.wikipedia.org/wiki/ZY https://en.wikipedia.org/wiki/ZZ

The URL is part of the assignment, it is a web scraper in Wiki.

Answer 1

Below is a screen shot of the python program to check indentation. Comments are given on every line explaining the code.

1 -import re 2 import urllib.request 3 from bs4 import BeautifulSoup 4 5 #Add the list of urls here 6 Curls = ['https://en.wikipedia.org/wiki/AA', 'https://en.wikipedia.org/wiki/AB', 7 A "https://en.wikipedia.org/wiki/AC','https://en.wikipedia.org/wiki/ZY'] 8 9 web_text = "" #Used to store all the text 10 Ofor url in urls: 11 res = urllib.request.urlopen(url) #Make a request to the url 12 soup = BeautifulSoup(res.read(),'html.parser') #Convert the html into soup element 13 A web_text = web_text+soup.text #Get only the text and leave all the html tags 14 15 #Remove punctuations and digits 16 punc = "''!()-[]{};:'"), «>./?@#$%^&*_~123456789'' 17 for ele in web_text: #loop through each character if ele in punc: #Check if its a punctuation 19 A web_text = web_text.replace(ele, ") #Replace punctuation by 20 21 counts = dict() #Dictionary for counter 22 Efor word in re.findall(r'1b\S+\b', web_text): #Find words using regex 23 word = word. lower() #Make the words into lowercase 24 counts[word] = counts.get(word, 0) + 1 #Count the words 25 26 #Sort the words and print the 15 most common words 27 for k, v in sorted (counts.items(), key=lambda item: item[1], reverse=True)[:15]: 28 print(vik) 18

Below is the output of the program:

229 a 201 0 131 the 120 of 118 in 92 ext 80 aa 66 disambiguation 61 edit 60 to 59 and 51 ready 43 ac 40 mW 40 value


Below is the code to copy:
#CODE STARTS HERE----------------

import re
import urllib.request
from bs4 import BeautifulSoup

#Add the list of urls here
urls = ['https://en.wikipedia.org/wiki/AA', 'https://en.wikipedia.org/wiki/AB',
      'https://en.wikipedia.org/wiki/AC','https://en.wikipedia.org/wiki/ZY']

web_text = "" #Used to store all the text
for url in urls:
   res  = urllib.request.urlopen(url) #Make a request to the url
   soup = BeautifulSoup(res.read(),'html.parser') #Convert the html into soup element
   web_text = web_text+soup.text #Get only the text and leave all the html tags

#Remove punctuations and digits
punc = '''!()-[]{};:'"\, <>./?@#$%^&*_~123456789'''
for ele in web_text: #loop through each character
   if ele in punc: #Check if its a punctuation
      web_text = web_text.replace(ele, " ") #Replace punctuation by " "

counts = dict() #Dictionary for counter
for word in re.findall(r'\b\S+\b', web_text): #Find words using regex
   word = word.lower() #Make the words into lowercase
   counts[word] = counts.get(word, 0) + 1 #Count the words

#Sort the words and print the 15 most common words
for k, v in sorted(counts.items(), key=lambda item: item[1], reverse=True)[:15]:
   print(v,k)
#CODE ENDS HERE------------------