Question

The Tokenizer.java file should contain:

• A class called: Tokenizer

• Tokenizer should have a private variable that is an ArrayList of Token objects.

• Tokenizer should have a private variable that is an int, and keeps track of the number of keywords encountered when parsing through a files content.

  • In this case there will only be one keyword: public.

• Tokenizer should have a default constructor that initializes the ArrayList of Token objects

• Tokenizer should have a public method called: tokenizeFile

  • tokenizeFile should have a string parameter thatwill be a file path

  • tokenizeFile should return void

  • tokenizeFile should throw an IOException

  • tokenizeFile should take the contents of the file and tokenize it using a space character as a delimiter

    • If there are multiple spaces between tokens then ignore those spaces and only retrieve the words, so given the string: “The cat” should still only produce two tokens: “The”, and “cat”

    • A newline should also serve to separate tokens, for example:

      • If the input contains multiple lines such as:

        The cat in the hat
        Red fish blue fish
        

      • There is no space between “hat” and “Red”, just a newline, which means the resulting tokens are “hat” and “Red”, NOT a single token: “hatRed”

  • Each token should be added to the private ArrayList variable as a Token object

  • If the value of a token is the keyword: public , then increment the private int counter that is keeping track of the number of keywords encountered when parsing the content of a file. - Remember that “public” is the only keyword you have to worry about - Any letter in the word: public , can be capitalized and you must still increment the counter - If public is part of another token, then it should NOT count as a keyword - For example if a file contained: - Blahblahpublic - The above public does not count as a keyword since public in this case is not its own token

    • Every call to tokenizeFile should first clear the previous tokens from the ArrayList of Tokens, as well as reset the private int counter of keywords (public)
      • For example: if a user calls tokenizeFile(“someFile.txt”) and it generates these tokens: “The” , “cat”, “in” , and then the user calls tokenizeFile again but with a different input file, such as: tokenizeFile(“somenewfile.txt”), the ArrayList should no longer hold the previous tokens, and should instead hold only the new tokens generated by the new file.

• Tokenizer should have only a getter for the ArrayList of Token objects (not a setter)

• The getter should be called: getTokens and should return an Array of Token objects, NOT an ArrayList of Token objects

• Tokenizer should have a method called: writeTokens

  • writeTokens should have no input parameter

  • writeTokens should have a return type of void

  • writeTokens should throw an IOException

  • writeTokens should write the tokens out to a file called: “output.txt”, and should reside in your current working directory. Each token should be written on a newline, for example - Given tokens: “The”, “cat”, “in”, “the”,”hat” The result in output.txt should be:

        The
        cat
        in  
        the 
        hat 
    

    • Notice the order the tokens should be written to file are in order from which they are read from the input file, which should be from left to right and from top to bottom

• Tokenizer should have a public method called: getKeywordCount , that has no input arguments and returns the private int keyword counter field

• Override the toString method inherited from the Object class, have it return the same value as calling the toString method on the ArrayList of Token objects

My Code so far:

import java.io.IOException;
import java.util.ArrayList;

/**
 * Vincent Nguyen
 * CS108
 * 3/27/2020
 */

public class Tokenizer {

    private ArrayList<Token> token;
    private int tokenCounter;

    /**Default Tokenizer initializing ArrayList*/
    public Tokenizer(){
    token = new ArrayList<Token>();
    }

    public void tokenizeFile(String path) throws IOException {

    }

}

Answer 1

/***************************Tokenizer********************************************/

import java.io.*; import java.util.ArrayList; import java.util.StringTokenizer; /** * Vincent Nguyen * CS108 * 3/27/2020 */ public class Tokenizer { private ArrayList<Token> token; private int tokenCounter; /**Default Tokenizer initializing ArrayList*/ public Tokenizer(){ token = new ArrayList<Token>(); } public void tokenizeFile(String path) throws IOException { token.clear(); tokenCounter=0; BufferedReader bf = new BufferedReader(new FileReader(path)); String line = bf.readLine(); while(null!=line){ String[] tokenStr = line.split(" "); for (int i=0;i<tokenStr.length;i++) { Token tk = null; if(tokenStr[i].equalsIgnoreCase("public")){ tk = new Token(tokenStr[i].toUpperCase(),i); tokenCounter = tokenCounter +1; }else{ tk = new Token(tokenStr[i],i); } token.add(tk); } line= bf.readLine(); } bf.close(); } public void writeTokens() throws IOException{ // Step #1. Create a file object. File file = new File("output.txt"); // Step #2. Create a file writer object with above file. FileWriter fileWriter = new FileWriter(file,true); // Step #3. Create a file object with above file writer. BufferedWriter writer = new BufferedWriter(fileWriter); for (Token token1 : token) { writer.append(token1.getToken()); writer.append("\n"); } // Step #6. free the resources. writer.close(); } public int getKeywordCount() { return tokenCounter; } @Override public String toString() { return "Tokenizer{" + "token=" + token + ", tokenCounter=" + tokenCounter + '}'; } }

=====================Token Class==================================

public class Token { private final String token; private final long position; public Token(String token, long position) { this.token = token; this.position = position; } public String getToken() { return token; } public long getPosition() { return position; } @Override public String toString() { return "Token{" + "token='" + token + '\'' + ", position=" + position + '}'; } } 

====================MainTestClass==================================

import java.io.IOException; public class MainTest { public static void main(String[] args) { Tokenizer tokenizer = new Tokenizer(); try { tokenizer.tokenizeFile("test.txt"); System.out.println("total token count: "+tokenizer.getKeywordCount()); tokenizer.writeTokens(); } catch (IOException e) { e.printStackTrace(); } } }