Question

Part 1. Write a program that takes a string as input, strips whitespace and punctuation from the words, and converts them to lowercase.

Hint: The string module provides strings named whitespace, which contains space, tab, newline, etc., and punctuation which contains the punctuation characters. Let’s see if we can make Python swear:

>>> import string
>>> print string.punctuation 
!"#$%&'()*+,-./:;<=>?@[\]^_`{|}~

Also, you might consider using the string methods strip, replace and translate (Chapter 2 of "Introducing Python" is a great reference for this).

Part 2. Modify the program from part 1 to count the total number of words in the string, and the number of times each word is used. Use a dictionary to store the frequency of each word. Your data structure should look something like:

Part 3. Modify the program from the previous exercise so it calculates and prints:

• The top 20 most frequently used words in a given string. For each of those words, it should print the frequency and the percentage chance that the word will occur. You may want to research techniques to accomplish this online, it isn't straightforward.
• The total number of words in the string

Answer 1

Part 1

import string

def stripStr(s):

   for whitespace in string.whitespace:   #removes all whitespaces from string.
       s = s.replace(whitespace,'')

   for punctuation in string.punctuation:   #removes all punctuations
       s = s.replace(punctuation,'')

   s = s.lower()    #makes all the aphabets to lower case.    return s
s = input('Enter string: ')
s = stripStr(s)       # function strips whitespaces, punctuations and converts to lower.
print(s)

Part 2

import string

def stripStr(s):

   for whitespace in string.whitespace:   #removes all whitespaces from string.
       if whitespace != ' ':
           s = s.replace(whitespace,'')

   for punctuation in string.punctuation:   #removes all punctuations
       s = s.replace(punctuation,'')

   s = s.lower()    #makes all the aphabets to lower case.

   return s

s = input('Enter string: ')
s = stripStr(s)       # function strips whitespaces, punctuations(except space) and converts to lower.
words = s.split()   # splits string into a list of words

counts = dict()       #dictionary to count word frequency
for word in words:
   if word not in counts:
       counts[word] = 1   #if word is not present enters word in dictionary
   else:
       counts[word] += 1   #if word is present increments frequency by 1

print(counts)

Part 3

import string

def stripStr(s):

   for whitespace in string.whitespace:   #removes all whitespaces from string.
       if whitespace != ' ':
           s = s.replace(whitespace,'')
   for punctuation in string.punctuation:   #removes all punctuations
       s = s.replace(punctuation,'')
   s = s.lower()    #makes all the aphabets to lower case.
   return s

def calcProb(counts):
   tup = list(counts.items())       #creates a tuple (word,frequency) from the dictionary
   tupR = list()                   #creates a tuple (frequency,word) from the tuple
   totalWords = 0
   i = 0

   for word,freq in tup:
       tupR.append((freq,word))
       totalWords = totalWords + freq      

   tupR.sort(reverse=True)           #sorts tuple by increasing frequency

   prb = list()

   for freq,word in tupR:
       if i == 20:
           break
       prb.append((freq/totalWords)*100)   #stores probability in a list
       i += 1

   i = 0
   for freq,word in tupR:
       if i == 20:
           break
       print(word,'\tfrequency:',freq,'\tprobability:',prb[i])
       i += 1

s = input('Enter string: ')
s = stripStr(s)       # function strips whitespaces, punctuations(except space) and converts to lower.
words = s.split()   # splits string into a list of words

counts = dict()       #dictionary to count word frequency
for word in words:
   if word not in counts:
       counts[word] = 1   #if word is not present enters word in dictionary
   else:
       counts[word] += 1   #if word is present increments frequency by 1
      
calcProb(counts)   #calculates probability