Question

Write code in a Python, that checks a source file (input file in plain text format) that separates lexemes by white space and special characters. This lexical analyzer will only have tokens for special characters and alphanumeric strings.

le: 2345 6tgbsauhd9sa67*I{OPKDSI;jakihl

Would be 2345

6tgbsauhd9sa67

*

l

{

OPKDSI

;

jakihl

Answer 1

Code:

import string
SPL_CHAR = string.punctuation
f = open("data.txt", "r") #opening file in read mode
for line in f.readlines(): #read file line by line
word=""
line = line.rstrip() #to remove new line character
for c in line: #for each character in line
if c not in SPL_CHAR and c != " ":
word = word + c #append character to word if it isnot special character or space
elif c == " ":
print(word) #print word if it is space
word="" #initialize it to empty string
else:
print(word) #if special character print word and the special character
print(c)
word="" #initialize it to empty string
print(word) #print word when end of line
  

Output:

File Considered (data.txt)