Question

Imagine that you are asked to build a search engine for finding relevant tweets. Please describe what methods that you plan to use to (i) build indexes; (ii) rank the tweets; and (iii) evaluate the results. Please provide justifications on your choice and explain potential limitations.

Answer 1

Answer:

import codecs

#variable to hold number of retweets
number_of_retweets = 0

# open the unicode file for parsing
f = codecs.open('content_polluters_tweets.txt', encoding='UTF-8')

# input the date to search for number of tweets on that day
target_day = raw_input("Enter a date in format yyyy-mm-dd\n>").strip()
tweets_per_day = 0

# input the suspicious words to search in a tweet
suspicous_words = raw_input("Enter suspicious words to search in tweets\n>").strip().lower().split()
bad_tweets = 0

# input words to search for repetition
words_rept = raw_input("Enter words to search for repetition\n>").strip().lower().split()
frequency_rept = dict()
for word in words_rept:
    frequency_rept[word] = 0

# iterate over each line in the file - each line is one tweet
for line in f:

    # extract a line from the file
    tweet = repr(line)
    # ignore first 2 characters, which are garbage
    tweet = tweet[2:]

    # strip the line for unnecessary whitespace
    tweet = tweet.rstrip("\\r\\n")
    tweet = tweet.rstrip("\r\n")
    tweet = tweet.strip("\\t")
    tweet = tweet.replace("\\t", " ")
    tweet = tweet.replace("\\u", "")
    tweet = tweet.replace("\u", "")
    tweet = tweet.strip()

    # make all letters lowercase for easier matching
    tweet = tweet.lower()

    # split on whitespace characters to generate tokens
    tokens = tweet.split()

    # determine if it is a retweet
    if tweet.__contains__(" rt "):
        number_of_retweets += 1

    # determine if the tweet is from the given day
    if tweet.__contains__(target_day):
        tweets_per_day += 1

    # search for suspicious words
    for word in suspicous_words:
        if tweet.__contains__(word):
            bad_tweets += 1

    # search for the given words
    for word in words_rept:
        if tweet.__contains__(word):
            frequency_rept[word] += 1


print "Total number of retweets: " + str(number_of_retweets)
print "Total number of tweets on " + target_day + " is " + str(tweets_per_day)
print "Tweets with entered suspicious words: " + str(bad_tweets)
print "The frequency of the given words is: "
for word in words_rept:
    print word + ": " + str(frequency_rept[word]) + " times."

OUTPUT:

DEAR PLEASE DO RATE IT IF HELPS ELSE LET ME KNOW YOUR DOUBT.

THANK YOU!!!