-
Confused on what tokenizer to use and how to use them.
I'm supposed to be making a parser for grammars, or basically parsing a simple language definition.
Basically I could be parsing either number or a symbol (+, *, (, ), or lambda, maybe lambda anyway, etc)
I want it to read in 123 as "123", not as 1, then 2, then 3 as three separate chars.
I want it only to advance the tokens thought based upon certain conditions. Not always.
I've started the actual code of this thing, though haven't gotten far.
Code java:
import java.util.StringTokenizer;
import java.io.*;
import java.util.*;
public class LL1Parser
{
private StringTokenizer tokenizer;
private boolean valid;
public LL1Parser(String input)
{
tokenizer = new StringTokenizer(input);
}
public boolean isValid(StringTokenizer st)
{
}
public static void main(String[] args)
{
LL1Parser parser;
try
{
parser = new LL1Parser(args[0]);
}
catch(NullPointerException npe)
{
System.out.println("args[0] doesn't exist!");
System.exit(0);
}
}
The pseudo code for the parsing is:
Code :
E():
switch(token)
{
case n: T(); E'();
build E -> TE';
break;
case +:
error;
break;
case *:
error;
break;
case (:
T(); E'();
build E ->TE';
break;
case ):
error;
break;
case $:
error;
break;
}
E'():
Switch(token)
{
case n:
error;
break;
case +:
get(+);T(); E'();
break;
case *;
error;
break;
case (:
error;
break;
case ):
build E' ->lambda;
break;
case $:
build E' -> lambda;
break;
}
T():
Switch(token)
{
case n:
build T->FT';
break;
case +:
error;
break;
case *:
error;
break;
case (:
build T->FT';
break;
case ):
error;
break;
case $:
error;
break;
}
T'():
Switch(token)
{
case n:
error;
break;
case +:
build T' -> lambda;
break;
case *:
get(*); F(); T'();
build T'-> *FT';
break;
case (:
error;
break;
case ):
build T'->lambda;
break;
case $:
build T'->lambda;
break;
}
F():
Switch(token)
{
case n:
get(n);
build F->n;
break;
case +:
error;
break;
case *:
error;
break;
case (:
get('(');E(); get(')');
build F->(E);
break;
case ):
error;
break;
case $:
error;
break;
}
Basically it's kind of recursive. I'm not so much concerned about what to write right now as how to work a Tokenizer.
If you're confused, which I have a strange feeling you are, I'm going to try and clarify:
It will read in a 1. As the original starting part is an E, it will go into the E part.
Since 1 is n where n is any integer less than 32767 it will go to the n case in the E part and will check out the
T(); E'(); part. Since it's going from left to right, it checks the T part first, and still has the same token as it hasn't removed it yet.
It goes to case n of the T part.
It tells it to go to FT' so now it's
F(); T'(); E'();
it checks out the F part first with the same token still.
It goes to case n and asks to get(n)
If this token WEREN'T a number, it would have something saying the grammar was incorrect.
As it is, this test of mine has it as a number so it removes that token from the input string and then checks out
T'(); E'();
The next token now is a +. It will check out the + case for T' which tells it to go to lambda, basically just getting rid of the T' but not changing the Token. Now it checks + in E'
get(+);T(); E'();
break;
which removes the + token and then has evaluates the next token, (, as in T.
so now it's
F(); T'(); E'();
get('(');E(); get(')')
The ( is in quotes so it won't get confused with the regular parenthesis.
This removes the ( and the checks
E(); get(')'); T'(); E'();
the next token happens to be ( so now it's
T(); E'(); get(')'); T'(); E'();
F(); T'(); E'(); get(')'); T'(); E'();
get('(');E(); get(')'); T'(); E'(); get(')'); T'(); E'();
so it removes the ( and goes to
E(); get(')'); T'(); E'(); get(')'); T'(); E'();
The next token is now a 23
T(); E'(); get(')'); T'(); E'(); get(')'); T'(); E'();
F(); T'(); E'(); get(')'); T'(); E'(); get(')'); T'(); E'();
get(n); T'(); E'(); get(')'); T'(); E'(); get(')'); T'(); E'();
so remove the 23 and then + is the new token.
T'(); E'(); get(')'); T'(); E'(); get(')'); T'(); E'();
It goes to lambda so now it's
E'(); get(')'); T'(); E'(); get(')'); T'(); E'();
get(+);T(); E'(); get(')'); T'(); E'(); get(')'); T'(); E'();
That removes the + and 2 is the next token.
F(); T'(); E'(); get(')'); T'(); E'(); get(')'); T'(); E'();
get(n); T'(); E'(); get(')'); T'(); E'(); get(')'); T'(); E'();
so now the 2 is gone and * is the next token.
T'(); E'(); get(')'); T'(); E'(); get(')'); T'(); E'();
get(*); F(); T'(); E'(); get(')'); T'(); E'(); get(')'); T'(); E'();
So * is gone and 3 is the next token.
F(); T'(); E'(); get(')'); T'(); E'(); get(')'); T'(); E'();
get(n); T'(); E'(); get(')'); T'(); E'(); get(')'); T'(); E'();
so now 3 is gone and ) is the new token.
T'(); E'(); get(')'); T'(); E'(); get(')'); T'(); E'();
So T' goes to lambda and now it's
E'(); get(')'); T'(); E'(); get(')'); T'(); E'();
E' goes to lambda so it's
get(')'); T'(); E'(); get(')'); T'(); E'();
So now the ) is gone.
Now ) is the next token.
T'(); E'(); get(')'); T'(); E'();
T' goes to lambda so it's
E'(); get(')'); T'(); E'();
E' goes to lambda too so now it's
get(')'); T'(); E'();
so the ) is gone and now, say, ) is the next token. (This will ruin the grammar so I'll show a correct part later.)
T'(); E'();
Ok, I've messed up somewhere in there, but you get the point.
I don't know how to use a Tokenizer to do this.
-
Re: Confused on what tokenizer to use and how to use them.
Ok, I've updated it a bit.
Code java:
import java.util.StringTokenizer;
import java.io.*;
import java.util.*;
public class LL1Parser
{
private StringTokenizer tokenizer;
private boolean valid;
public LL1Parser(String input)
{
tokenizer = new StringTokenizer(input);
System.out.println(isValid(tokenizer));
}
public boolean isValid(StringTokenizer st)
{
// needs to check to see if String is null
// needs a base case anyway
while (st.hasMoreTokens())
{
if (!EValidator(st))
return false;
if (!E'Validator(st))
return false;
if (!TValidator(st))
return false;
if (!T'Validator(st))
return false;
if (!FValidator(st))
return false;
else
// do something to remove some tokens
// create a StringTokenizer variable newSt based on this new value with the token removed
isValid(newSt);
}
return true;
}
private boolean EValidator(StringTokenizer st)
{
boolean validity = false;
String str = st.nextToken();
switch(str)
{
case n:
// how do I do this?
case +:
System.out.println("Not valid!");
break;
case *:
System.out.println("Not valid!");
break;
case ( :
// what do I do?
case ) :
System.out.println("Not valid!");
break;
case $:
System.out.println("Not valid!");
break;
}
return validity;
}
private boolean E'Validator(StringTokenizer st)
{
}
private boolean TValidator(StringTokenizer st)
{
}
private boolean T'Validator(StringTokenizer st)
{
}
private boolean FValidator(StringTokenizer st)
{
}
public static void main(String[] args)
{
LL1Parser parser;
try
{
parser = new LL1Parser(args[0]);
}
catch(NullPointerException npe)
{
System.out.println("args[0] doesn't exist!");
System.exit(0);
}
}
-
Re: Confused on what tokenizer to use and how to use them.
Please reply. ^:)^^:)^:-w:-w^:)^^:)^
Code java:
import java.util.StringTokenizer;
import java.io.*;
import java.util.*;
public class LL1Parser
{
private StringTokenizer tokenizer;
private boolean valid;
private Stack<String> commandStack;
private String tokens;
public LL1Parser(String input)
{
tokenizer = new StringTokenizer(input);
commandStack = new Stack<String>(); // creates an empty command stack
commandStack.push("T");
commandStack.push("EP");
System.out.println(isValid(tokenizer));
}
private boolean read(StringTokenizer st)
{
if (commandStack.peek().equals("E"))
{
return EValidator(st);
}
else if(commandStack.peek().equals("EP"))
return EPValidator(st);
else if (commandStack.peek().equals("T"))
return TValidator(st);
else if(commandStack.peek().equals("TP"))
return TPValidator(st);
else if (commandStack.peek().equals("F"))
return FValidator(st);
else if (commandStack.peek().equals("get +"))
{
if (st.nextToken().equals("+"))
{
setString(getString().substring(1));
return true;
}
else
return false;
}
else if (commandStack.peek().equals("get *"))
{
if(st.nextToken().equals("*"))
{
setString(getString().substring(1));
return true;
}
else
return false;
}
else if(commandStack.peek().equals("get ("))
{
if(st.nextToken().equals("("))
{
setString(getString().substring(1));
return true;
}
else
return false;
}
else if(commandStack.peek().equals("get )"))
{
if(st.nextToken().equals(")"))
{
setString(getString().substring(1));
return true;
}
else
return false;
}
else if(commandStack.peek().equals("get n"))
{
Integer n = 0;
try
{
n = Integer.parseInt(st.nextToken());
}
catch(NumberFormatException nfe)
{
}
if(st.nextToken().equals(n.toString()))
{
setString(getString().substring(1));
return true;
}
else
return false;
}
else
return false;
}
private void setString(String tokens)
{
this.tokens = tokens;
}
private String getString()
{
return tokens;
}
private void build(int rule)
{
if (rule == 1)
{
commandStack.pop();
commandStack.push("T");
commandStack.push("EP");
}
else if (rule == 2)
{
commandStack.pop();
commandStack.push("get +");
commandStack.push("T");
commandStack.push("EP");
}
else if(rule == 3)
{
commandStack.pop();
}
else if(rule == 4)
{
commandStack.pop();
commandStack.push("F");
commandStack.push("TP");
}
else if (rule == 5)
{
commandStack.pop();
commandStack.push("get *");
commandStack.push("F");
commandStack.push("TP");
}
else if (rule == 6)
{
commandStack.pop();
}
else if (rule == 7)
{
commandStack.pop();
commandStack.push("get (");
commandStack.push("E");
commandStack.push("get )");
}
else if(rule == 8)
{
commandStack.pop();
commandStack.push("get n");
}
else
System.out.println("Invalid rule");
}
public boolean isValid(StringTokenizer st)
{
// needs to check to see if String is null
// needs a base case anyway
while (st.hasMoreTokens())
{
read(st);
// else
// do something to remove some tokens
// create a StringTokenizer variable newSt based on this new value with the token removed
// isValid(newSt);
}
return true;
}
private boolean EValidator(StringTokenizer st)
{
boolean validity = false;
String str = st.nextToken();
Integer n = 0;
try
{
n = Integer.parseInt(str);
}
catch (NumberFormatException ime)
{
}
if (str.equals(n.toString()))
{
// how do I do this?
build(1);
validity = true;
}
else if(str.equals("+"))
{
System.out.println("Not valid!");
}
else if(str.equals( "*"))
{
System.out.println("Not valid!");
}
else if (str.equals ("("))
{
// what do I do?
build(1);
validity = true;
}
else if(str.equals( ")"))
{
System.out.println("Not valid!");
}
else if (str.equals( "$"))
{
System.out.println("Not valid!");
}
else
System.out.println("Invalid token");
// should I add a break?
return validity;
}
private boolean EPValidator(StringTokenizer st)
{
boolean validity = false;
String str = st.nextToken();
Integer n = 0;
try
{
n = Integer.parseInt(str);
}
catch (NumberFormatException ime)
{
}
if(str.equals( n.toString()))
{
// still not sure how to get it to be n
System.out.println("Not valid!");
}
else if(str.equals("+"))
{
// what do I do?
build(2);
validity = true;
}
else if(str.equals( "*"))
{
System.out.println("Not valid!");
}
else if(str.equals( "("))
{
System.out.println("Not valid!");
}
else if(str.equals( ")"))
{
// what do I do?
build(3);
validity = true;
}
else if(str.equals( "$"))
{
// what do I do?
build(3);
validity = true;
}
else
System.out.println("Invalid token");
// should I add a break?
return validity;
}
private boolean TValidator(StringTokenizer st)
{
boolean validity = false;
String str = st.nextToken();
Integer n = 0;
try
{
n = Integer.parseInt(str);
}
catch (NumberFormatException ime)
{
}
if(str.equals( n.toString()))
{
// what do I do?
build(4);
validity = true;
}
else if(str.equals( "+"))
{
System.out.println("Not valid!");
}
else if(str.equals( "*"))
{
System.out.println("Not valid!");
}
else if(str.equals( "("))
{
// what do I do?
build(4);
validity = true;
}
else if(str.equals( ")"))
{
System.out.println("Not valid!");
}
else if(str.equals("$"))
{
System.out.println("Not valid!");
}
else
System.out.println("Invalid token");
// should I add a break?
return validity;
}
private boolean TPValidator(StringTokenizer st)
{
boolean validity = false;
String str = st.nextToken();
Integer n = 0;
try
{
n = Integer.parseInt(str);
}
catch (NumberFormatException ime)
{
}
if(str.equals( n.toString()))
{
System.out.println("Not valid!");
}
else if(str.equals( "+"))
{
// what do I do?
build(6);
validity = true;
}
else if(str.equals( "*"))
{
// what do I do?
build(5);
validity = true;
}
else if(str.equals( "("))
{
System.out.println("Not valid!");
}
else if(str.equals( ")"))
{
// what do I do?
build(6);
validity = true;
}
else if(str.equals( "$"))
{
// what do I do?
build(6);
validity = true;
}
else
System.out.println("Invalid token");
// should I add a break?
return validity;
}
private boolean FValidator(StringTokenizer st)
{
boolean validity = false;
String str = st.nextToken();
Integer n = 0;
try
{
n = Integer.parseInt(str);
}
catch (NumberFormatException ime)
{
}
if(str.equals(n.toString()))
{
// delete token n; how I don't know
build(8);
validity = true;
}
else if(str.equals( "+"))
{
System.out.println("Not valid!");
}
else if(str.equals( "*"))
{
System.out.println("Not valid!");
}
else if(str.equals( "("))
{
// what do I do?
build(7);
validity = true;
}
else if(str.equals( ")"))
{
System.out.println("Not valid!");
}
else if(str.equals("$"))
{
System.out.println("Not valid!");
}
else
System.out.println("Invalid token");
// should I add a break?
return validity;
}
public static void main(String[] args)
{
LL1Parser parser;
try
{
parser = new LL1Parser(args[0] + "$");
}
catch(ArrayIndexOutOfBoundsException aioobe)
{
System.out.println("args[0] doesn't exist!");
System.exit(0);
}
}
}
Ok I think I can get it to update the String to simply get the subsring of that string by substring(1) so I remove the first character but how do I keep applying it to that read function with the smaller input String? I already have setString() and getString() methods to use. However, I'm confused on how to shrink the String in the right way and apply it to the tokenizer in the right way. Also, I am wondering how my setup will work with having read() return a boolean.
It needs to do something if the token it tries to remove is the current (or maybe it was the next) token. Think it's the current but could be wrong. Anyway, if, for instance there's a "get n" on the top of the commandStack but the nextToken is a "+", then it's an invalid grammar String and it should say so and stop the procedure and cause isValid() to return false and also NOT try to evaluate the rest of the stuff as it's invalid. Also, if there is nothing left on the commandStack but there are tokens left, then it's also invalid. If there's nothing left in either, then and only then I think should it return true for isValid().
-
Re: Confused on what tokenizer to use and how to use them.
Can you write a small simple test parser program to work on only this problem? Include all input data in the program itself, vs requiring any user input.
Show what your test parser code does and add comments to the output showing what you want it to do.
-
Re: Confused on what tokenizer to use and how to use them.
There is too much clutter in the first post. It restricts replies.
-
Re: Confused on what tokenizer to use and how to use them.
There is something I'm wondering. When I call nextToken(), does it simply return but not remove the next token or is it also removing it from the token string at the same time?
I cannot figure out where the problem is but know it's somehow bypassing all of my if statements and somehow always going to the else statements. I cannot break it down much further as I'm not quite sure what's causing the problem. A lot of my printlns aren't even printing. Also, how do I get it to update the tokenizer so it changes? I don't think I'm doing it correctly every time.
Code java:
import java.util.StringTokenizer;
import java.io.*;
import java.util.*;
public class LL1Parser
{
private StringTokenizer tokenizer;
private boolean valid;
private Stack<String> commandStack;
private String tokens;
public LL1Parser(String input)
{
tokenizer = new StringTokenizer(input);
commandStack = new Stack<String>(); // creates an empty command stack
setString(input);
commandStack.push("T");
commandStack.push("EP");
boolean b = isValid(tokenizer);
System.out.println(b);
}
private boolean read(StringTokenizer st)
{
if (commandStack.peek().equals("E"))
{
System.out.println(EValidator(st));
return EValidator(st);
}
else if(commandStack.peek().equals("EP"))
return EPValidator(st);
else if (commandStack.peek().equals("T"))
{
System.out.println(TValidator(st));
return TValidator(st);
}
else if(commandStack.peek().equals("TP"))
return TPValidator(st);
else if (commandStack.peek().equals("F"))
return FValidator(st);
else if (commandStack.peek().equals("get +"))
{
if (st.nextToken().equals("+"))
{
setString(getString().substring(1));
return true;
}
else
return false;
}
else if (commandStack.peek().equals("get *"))
{
if(st.nextToken().equals("*"))
{
setString(getString().substring(1));
return true;
}
else
return false;
}
else if(commandStack.peek().equals("get ("))
{
if(st.nextToken().equals("("))
{
setString(getString().substring(1));
return true;
}
else
return false;
}
else if(commandStack.peek().equals("get )"))
{
if(st.nextToken().equals(")"))
{
setString(getString().substring(1));
return true;
}
else
return false;
}
else if(commandStack.peek().equals("get n"))
{
Integer n = 0;
try
{
n = Integer.parseInt(st.nextToken());
}
catch(NumberFormatException nfe)
{
}
if(st.nextToken().equals(n.toString()))
{
setString(getString().substring(1));
return true;
}
else
return false;
}
else
return false;
}
private void setString(String tokens)
{
this.tokens = tokens;
}
private String getString()
{
return tokens;
}
private void build(int rule)
{
if (rule == 1)
{
commandStack.pop();
commandStack.push("T");
commandStack.push("EP");
}
else if (rule == 2)
{
commandStack.pop();
commandStack.push("get +");
commandStack.push("T");
commandStack.push("EP");
}
else if(rule == 3)
{
commandStack.pop();
}
else if(rule == 4)
{
commandStack.pop();
commandStack.push("F");
commandStack.push("TP");
}
else if (rule == 5)
{
commandStack.pop();
commandStack.push("get *");
commandStack.push("F");
commandStack.push("TP");
}
else if (rule == 6)
{
commandStack.pop();
}
else if (rule == 7)
{
commandStack.pop();
commandStack.push("get (");
commandStack.push("E");
commandStack.push("get )");
}
else if(rule == 8)
{
commandStack.pop();
commandStack.push("get n");
}
else
System.out.println("Invalid rule");
}
public boolean isValid(StringTokenizer st)
{
System.out.println(st.nextToken());
// needs to check to see if String is null
// needs a base case anyway
boolean validity = true;
while (st.hasMoreTokens() && validity == true )
{
// validity = read(st);
validity = read(new StringTokenizer(getString()));
// else
// do something to remove some tokens
// create a StringTokenizer variable newSt based on this new value with the token removed
// isValid(newSt);
}
return validity;
}
private boolean EValidator(StringTokenizer st)
{
boolean validity = false;
String str = st.nextToken();
Integer n = 0;
try
{
n = Integer.parseInt(str);
}
catch (NumberFormatException ime)
{
}
if (str.equals(n.toString()))
{
// how do I do this?
build(1);
validity = true;
}
else if(str.equals("+"))
{
System.out.println("Not valid!");
}
else if(str.equals( "*"))
{
System.out.println("Not valid!");
}
else if (str.equals ("("))
{
// what do I do?
build(1);
validity = true;
}
else if(str.equals( ")"))
{
System.out.println("Not valid!");
}
else if (str.equals( "$"))
{
System.out.println("Not valid!");
}
else
System.out.println("Invalid token");
// should I add a break?
return validity;
}
private boolean EPValidator(StringTokenizer st)
{
boolean validity = false;
String str = st.nextToken();
Integer n = 0;
try
{
n = Integer.parseInt(str);
}
catch (NumberFormatException ime)
{
}
if(str.equals( n.toString()))
{
// still not sure how to get it to be n
System.out.println("Not valid!");
}
else if(str.equals("+"))
{
// what do I do?
build(2);
validity = true;
}
else if(str.equals( "*"))
{
System.out.println("Not valid!");
}
else if(str.equals( "("))
{
System.out.println("Not valid!");
}
else if(str.equals( ")"))
{
// what do I do?
build(3);
validity = true;
}
else if(str.equals( "$"))
{
// what do I do?
build(3);
validity = true;
}
else
System.out.println("Invalid token");
// should I add a break?
return validity;
}
private boolean TValidator(StringTokenizer st)
{
boolean validity = false;
String str = st.nextToken();
Integer n = 0;
try
{
n = Integer.parseInt(str);
}
catch (NumberFormatException ime)
{
}
if(str.equals( n.toString()))
{
// what do I do?
build(4);
validity = true;
}
else if(str.equals( "+"))
{
System.out.println("Not valid!");
}
else if(str.equals( "*"))
{
System.out.println("Not valid!");
}
else if(str.equals( "("))
{
// what do I do?
build(4);
validity = true;
}
else if(str.equals( ")"))
{
System.out.println("Not valid!");
}
else if(str.equals("$"))
{
System.out.println("Not valid!");
}
else
System.out.println("Invalid token");
// should I add a break?
return validity;
}
private boolean TPValidator(StringTokenizer st)
{
boolean validity = false;
String str = st.nextToken();
Integer n = 0;
try
{
n = Integer.parseInt(str);
}
catch (NumberFormatException ime)
{
}
if(str.equals( n.toString()))
{
System.out.println("Not valid!");
}
else if(str.equals( "+"))
{
// what do I do?
build(6);
validity = true;
}
else if(str.equals( "*"))
{
// what do I do?
build(5);
validity = true;
}
else if(str.equals( "("))
{
System.out.println("Not valid!");
}
else if(str.equals( ")"))
{
// what do I do?
build(6);
validity = true;
}
else if(str.equals( "$"))
{
// what do I do?
build(6);
validity = true;
}
else
System.out.println("Invalid token");
// should I add a break?
return validity;
}
private boolean FValidator(StringTokenizer st)
{
boolean validity = false;
String str = st.nextToken();
Integer n = 0;
try
{
n = Integer.parseInt(str);
}
catch (NumberFormatException ime)
{
}
if(str.equals(n.toString()))
{
// delete token n; how I don't know
build(8);
validity = true;
}
else if(str.equals( "+"))
{
System.out.println("Not valid!");
}
else if(str.equals( "*"))
{
System.out.println("Not valid!");
}
else if(str.equals( "("))
{
// what do I do?
build(7);
validity = true;
}
else if(str.equals( ")"))
{
System.out.println("Not valid!");
}
else if(str.equals("$"))
{
System.out.println("Not valid!");
}
else
System.out.println("Invalid token");
// should I add a break?
return validity;
}
public static void main(String[] args)
{
LL1Parser parser;
try
{
parser = new LL1Parser(args[0] + " $");
}
catch(ArrayIndexOutOfBoundsException aioobe)
{
System.out.println("args[0] doesn't exist!");
System.exit(0);
}
}
}
I used a command prompt like thing with
java LL1Parser 123 which outputted the three lines below:
123
Not valid!
false
I ran it with
java LL1Parser *
and it outputted these three lines below:
LL1Parser.class
Invalid token
false
Why doesn't it ever recognize even valid tokens?
i.e. starting with a plus should be invalid but the number 123 should be valid but it's not recognizing it as valid.
It's always outputting false.
I'm thinking for one that I should get rid of all those StringTokenizer parameters, or nearly all of them, and just edit the class variable tokenizer.
Also, when I'm calling those pop() methods inside of the validator methods, is it removing the top of the stack twice or something?
Also, it appears to be a problem with one of my validator methods, or that's where the error is showing up with the printlns. Which validator method I don't know so I need to show them all.
-
Re: Confused on what tokenizer to use and how to use them.
Did you read my post#4 about what kind of code to post for testing this one problem?
-
Re: Confused on what tokenizer to use and how to use them.
I believe it now is, or should be anyway, updating the token string to shrink.
However, it seems to always be either always evaluating to invalid or false. I've updated it to change all the things except read to NOT use a StringTokenizer as parameter and to simply use the class variable tokenizer instead. After doing that, now it's always evaluating to true, even for bogus tokens like "-". I have no idea where it's going wrong though likely at the beginning part, i.e. in the EValidator(), though I can't be sure, hence the large amount of code. I think it unlikely that the build() method is the problem but can't narrow it much further.
It's often not going into the validator methods it seems as some of my printlns are NEVER triggered.
I'm thinking it's somehow always going to either true or false due to a glitch in one of my if, else if, else structures but which one I don't know, hence why I posted all of them.
For instance, EValidator is never being entered according to my print lines.
Perhaps it's my commandStack not being updated or something. Somehow I'm thinking it's something to do with the commandStack.
----Edit-------
I found that the instructor said that
1.) I could have more time
2.) That I could do it another way and don't even need the stack at all
So I'm going to revamp the class and then see if it works and if not post it here.
-
Re: Confused on what tokenizer to use and how to use them.
Ok I've rewrote part of it now so I don't need the tokenizer at all. I made my own. Does it have any code flaws? It compiles but might have bad logic or cause null pointers. Or could be inefficient.
Code java:
private static class TokenHandler
{
private String str;
private char nextchar;
public static String getNextToken(String tokenStream)
{
String numberString = "";
int index = 0;
char c = tokenStream.charAt(0);
if ( c == '+')
{
return "+";
}
else if (c == '*')
{
return "*";
}
else if (c == '(')
{
return "(";
}
else if (c == ')')
{
return ")";
}
else if (c == '0' || c == '1' || c == '2' || c == '3' || c == '4'|| c == '5' || c == '6' || c == '7'|| c == '8' || c == '9')
{
char c2 = 'a';
numberString = numberString + String.valueOf(c);
if (tokenStream.length() == 1)
{
return numberString;
}
else
{
c2 = tokenStream.charAt(1);
while (c2 == '0' || c2 == '1' || c2 == '2' || c2 == '3' || c2 == '4'|| c2 == '5' || c2 == '6' || c2 == '7'|| c2 == '8' || c2 == '9')
{
numberString = numberString + getNextToken(tokenStream.substring(1));
}
}
return numberString;
}
else if ( c == '$')
{
return "$";
}
else
{
return ("Bad String!");
}
}
public static void deleteToken(String tokenStream)
{
String temp = getNextToken(tokenStream);
int number = temp.length();
String temp2 = tokenStream.substring(number);
tokenStream = temp2;
}
}
-
Re: Confused on what tokenizer to use and how to use them.
Does the code now do what you want it to do?
If not, show what it does and explain what it should do.
-
Re: Confused on what tokenizer to use and how to use them.
It shows that the while loop inside of getNextToken() is infinite if it's a number that's read in as the next char. I'm thinking it's how I'm phrasing it but cannot figure out what to do about it. Am going to see what happens if I change that while to an if.......
Ok, that solved that part of the problem. Let's see how delete token goes.
It didn't work but I tweaked it and now it works.
Actually, I'd like deleteToken() to be a void if possible but don't know how to correctly change the token parameter so it now updates the string.
-
Re: Confused on what tokenizer to use and how to use them.
Ok, I've updated it and it's now always returning false.
It appears to be going into the read twice but never EValidator. Why would that be?
Code java:
import java.util.StringTokenizer;
import java.io.*;
import java.util.*;
public class LL1Parser
{
private StringTokenizer tokenizer;
private boolean valid;
private Stack<String> commandStack;
private String tokens;
private static class TokenHandler
{
private String str;
private char nextchar;
public static String getNextToken(String tokenStream)
{
String numberString = "";
int index = 0;
char c = tokenStream.charAt(0);
if ( c == '+')
{
return "+";
}
else if (c == '*')
{
return "*";
}
else if (c == '(')
{
return "(";
}
else if (c == ')')
{
return ")";
}
else if (c == '0' || c == '1' || c == '2' || c == '3' || c == '4'|| c == '5' || c == '6' || c == '7'|| c == '8' || c == '9')
{
char c2 = 'a';
numberString = numberString + String.valueOf(c);
if (tokenStream.length() == 1)
{
return numberString;
}
else
{
c2 = tokenStream.charAt(1);
if (c2 == '0' || c2 == '1' || c2 == '2' || c2 == '3' || c2 == '4'|| c2 == '5' || c2 == '6' || c2 == '7'|| c2 == '8' || c2 == '9')
{
numberString = numberString + getNextToken(tokenStream.substring(1));
}
}
return numberString;
}
else if ( c == '$')
{
return "$";
}
else
{
return ("Bad String!");
}
}
public static String deleteToken(String tokenStream)
{
String temp = getNextToken(tokenStream);
int number = temp.length();
String temp2 = tokenStream.substring(number);
//tokenStream = temp2;
return temp2;
}
}
public LL1Parser(String input)
{
// tokenizer = new StringTokenizer(input);
// commandStack = new Stack<String>(); // creates an empty command stack
setString(input);
// commandStack.push("T");
// commandStack.push("EP");
boolean b = isValid();
System.out.println(b);
}
private boolean read()
{
System.out.println("Came to read");
boolean validity = true;
boolean temp = TValidator();
if(!temp)
return false;
boolean temp2 = EPValidator();
if(!temp2)
return false;
return validity;
}
private void setString(String tokens)
{
this.tokens = tokens;
}
private String getString()
{
return tokens;
}
public boolean isValid()
{
// System.out.println(tokenizer.nextToken());
// needs to check to see if String is null
// needs a base case anyway
boolean validity = true;
while (getString() != null && validity == true )
{
// validity = read(st);
validity = read();
if(validity == false)
return false;
// else
// do something to remove some tokens
// create a StringTokenizer variable newSt based on this new value with the token removed
// isValid(newSt);
}
return validity;
}
private boolean EValidator()
{
System.out.println("EValidator method is being used.");
boolean validity = false;
String str = TokenHandler.getNextToken(getString());
Integer n = 0;
try
{
n = Integer.parseInt(str);
if (n >32676)
n = 32676;
else n = n;
}
catch (NumberFormatException ime)
{
}
if (str.equals(n.toString()))
{
boolean temp = TValidator();
if (!temp)
return false;
boolean temp2 = EPValidator();
if(!temp2)
return false;
// how do I do this?
// build(1);
validity = true;
}
else if(str.equals("+"))
{
System.out.println("Not valid!");
return false;
}
else if(str.equals( "*"))
{
System.out.println("Not valid!");
return false;
}
else if (str.equals ("("))
{
boolean temp = TValidator();
if (!temp)
return false;
boolean temp2 = EPValidator();
if (!temp2)
return false;
// what do I do?
// build(1);
validity = true;
}
else if(str.equals( ")"))
{
System.out.println("Not valid!");
return false;
}
else if (str.equals( "$"))
{
System.out.println("Not valid!");
return false;
}
else
{
System.out.println("Invalid token");
return false;
}
// should I add a break?
return validity;
}
private boolean EPValidator()
{
boolean validity = false;
String str = TokenHandler.getNextToken(getString());
Integer n = 0;
try
{
n = Integer.parseInt(str);
if (n >32676)
n = 32676;
else n = n;
}
catch (NumberFormatException ime)
{
}
if(str.equals( n.toString()))
{
// still not sure how to get it to be n
System.out.println("Not valid!");
return false;
}
else if(str.equals("+"))
{
// remove token + somehow
setString(TokenHandler.deleteToken(getString()));
boolean temp = TValidator();
if (!temp)
return false;
boolean temp2 = EPValidator();
if (!temp2)
return false;
// what do I do?
// build(2);
validity = true;
}
else if(str.equals( "*"))
{
System.out.println("Not valid!");
return false;
}
else if(str.equals( "("))
{
System.out.println("Not valid!");
return false;
}
else if(str.equals( ")"))
{
// what do I do?
// build(3);
validity = true;
}
else if(str.equals( "$"))
{
// what do I do?
// build(3);
validity = true;
}
else
{
System.out.println("Invalid token");
return false;
}
// should I add a break?
return validity;
}
private boolean TValidator()
{
boolean validity = false;
String str = TokenHandler.getNextToken(getString());
Integer n = 0;
try
{
n = Integer.parseInt(str);
}
catch (NumberFormatException ime)
{
}
if(str.equals( n.toString()))
{
boolean temp = FValidator();
if(!temp)
return false;
boolean temp2 = TPValidator();
if(!temp2)
return false;
// what do I do?
// build(4);
validity = true;
}
else if(str.equals( "+"))
{
System.out.println("Not valid!");
return false;
}
else if(str.equals( "*"))
{
System.out.println("Not valid!");
return false;
}
else if(str.equals( "("))
{
boolean temp = FValidator();
if(!temp)
return false;
boolean temp2 = TPValidator();
if (!temp2)
return false;
// what do I do?
// build(4);
validity = true;
}
else if(str.equals( ")"))
{
System.out.println("Not valid!");
return false;
}
else if(str.equals("$"))
{
System.out.println("Not valid!");
return false;
}
else
{
System.out.println("Invalid token");
return false;
}
// should I add a break?
return validity;
}
private boolean TPValidator()
{
boolean validity = false;
String str = TokenHandler.getNextToken(getString());
Integer n = 0;
try
{
n = Integer.parseInt(str);
if (n >32676)
n = 32676;
else n = n;
}
catch (NumberFormatException ime)
{
}
if(str.equals( n.toString()))
{
System.out.println("Not valid!");
return false;
}
else if(str.equals( "+"))
{
// what do I do?
// setString(TokenHandler.deleteToken(getString()));
// build(6);
validity = true;
}
else if(str.equals( "*"))
{
// somehow remove * from token list
setString(TokenHandler.deleteToken(getString()));
boolean temp = FValidator();
if (!temp)
return false;
boolean temp2 = TPValidator();
if (!temp2)
return false;
// what do I do?
//build(5);
validity = true;
}
else if(str.equals( "("))
{
System.out.println("Not valid!");
return false;
}
else if(str.equals( ")"))
{
// what do I do?
// build(6);
validity = true;
}
else if(str.equals( "$"))
{
// what do I do?
// build(6);
validity = true;
}
else
{
System.out.println("Invalid token");
return false;
}
// should I add a break?
return validity;
}
private boolean FValidator()
{
boolean validity = false;
String str = TokenHandler.getNextToken(getString());
Integer n = 0;
try
{
n = Integer.parseInt(str);
if (n >32676)
n = 32676;
else n = n;
}
catch (NumberFormatException ime)
{
}
if(str.equals(n.toString()))
{
setString(TokenHandler.deleteToken(getString()));
// delete token n; how I don't know
// build(8);
validity = true;
}
else if(str.equals( "+"))
{
System.out.println("Not valid!");
return false;
}
else if(str.equals( "*"))
{
System.out.println("Not valid!");
return false;
}
else if(str.equals( "("))
{
setString(TokenHandler.deleteToken(getString()));
boolean temp = EValidator();
if(!temp)
return false;
if (TokenHandler.getNextToken(getString()) != ")")
return false;
else
setString(TokenHandler.deleteToken(getString()));
// what do I do?
// build(7);
validity = true;
}
else if(str.equals( ")"))
{
System.out.println("Not valid!");
return false;
}
else if(str.equals("$"))
{
System.out.println("Not valid!");
return false;
}
else
{
System.out.println("Invalid token");
return false;
}
// should I add a break?
return validity;
}
public static void main(String[] args)
{
LL1Parser parser;
try
{
parser = new LL1Parser(args[0] + "$");
}
catch(ArrayIndexOutOfBoundsException aioobe)
{
System.out.println("args[0] doesn't exist!");
System.exit(0);
}
}
}
I added a println into the isValid() method and told it to print out validity.
The ultimate validity is printed at the end. That's where that second false is coming from below.
It gave this for this input:
pradcoc@discovery:~/ITK 327/LL1 Parser> java LL1Parser 123
Came to read
true
Came to read
Not valid!
false
false
123 should be valid.
However, it's not recognizing it.
I've considered the possibility that in the command prompt, it wants the args passed as
"123" but have tried that and still had the same problem.
-
Re: Confused on what tokenizer to use and how to use them.
Can you change your code for easier testing?
It should compile and execute and show the problem. Nothing more. No args nothing extra.
Include some debug printlns to show what the code does and what you want it to do.
-
Re: Confused on what tokenizer to use and how to use them.
I did find an error where I had been setting validity to true, regardless or whether or not it was true, at the beginning of every read method call and, I think, have fixed that, but the problem is still there
Code java:
import java.util.StringTokenizer;
import java.io.*;
import java.util.*;
public class LL1Parser
{
private StringTokenizer tokenizer;
private boolean validity2;
private Stack<String> commandStack;
private String tokens;
private static class TokenHandler
{
private String str;
private char nextchar;
public static String getNextToken(String tokenStream)
{
String numberString = "";
int index = 0;
char c = tokenStream.charAt(0);
if ( c == '+')
{
return "+";
}
else if (c == '*')
{
return "*";
}
else if (c == '(')
{
return "(";
}
else if (c == ')')
{
return ")";
}
else if (c == '0' || c == '1' || c == '2' || c == '3' || c == '4'|| c == '5' || c == '6' || c == '7'|| c == '8' || c == '9')
{
char c2 = 'a';
numberString = numberString + String.valueOf(c);
if (tokenStream.length() == 1)
{
return numberString;
}
else
{
c2 = tokenStream.charAt(1);
if (c2 == '0' || c2 == '1' || c2 == '2' || c2 == '3' || c2 == '4'|| c2 == '5' || c2 == '6' || c2 == '7'|| c2 == '8' || c2 == '9')
{
numberString = numberString + getNextToken(tokenStream.substring(1));
}
}
return numberString;
}
else if ( c == '$')
{
return "$";
}
else
{
return ("Bad String!");
}
}
public static String deleteToken(String tokenStream)
{
String temp = getNextToken(tokenStream);
int number = temp.length();
String temp2 = tokenStream.substring(number);
//tokenStream = temp2;
return temp2;
}
}
public LL1Parser(String input)
{
// tokenizer = new StringTokenizer(input);
// commandStack = new Stack<String>(); // creates an empty command stack
setString(input);
setValidity(true);
// commandStack.push("T");
// commandStack.push("EP");
boolean b = isValid();
System.out.println(b);
}
public void setValidity(boolean validity2)
{
this.validity2 = validity2;
}
public boolean getValidity()
{
return validity2;
}
private boolean read()
{
System.out.println("Came to read");
boolean validity = true;
boolean temp = TValidator();
if(!temp)
return false;
System.out.println("validitiy is " +validity + " after TValidator");
boolean temp2 = EPValidator();
if(!temp2)
return false;
System.out.println("validity is " + validity + " after EPValidator" );
return validity;
}
private void setString(String tokens)
{
this.tokens = tokens;
}
private String getString()
{
return tokens;
}
public boolean isValid()
{
// System.out.println(tokenizer.nextToken());
// needs to check to see if String is null
// needs a base case anyway
boolean validity = getValidity();
while (getString() != null && validity == true )
{
// validity = read(st);
validity = read();
setValidity(validity);
System.out.println("Validity is " + validity + " at this point in read");
//if(validity == false)
//return false;
// else
// do something to remove some tokens
// create a StringTokenizer variable newSt based on this new value with the token removed
// isValid(newSt);
}
return validity;
}
private boolean EValidator()
{
System.out.println("EValidator method is being used.");
boolean validity = false;
String str = TokenHandler.getNextToken(getString());
Integer n = 0;
try
{
n = Integer.parseInt(str);
if (n >32676)
n = 32676;
else n = n;
}
catch (NumberFormatException ime)
{
}
if (str.equals(n.toString()))
{
boolean temp = TValidator();
if (!temp)
return false;
boolean temp2 = EPValidator();
if(!temp2)
return false;
// how do I do this?
// build(1);
validity = true;
}
else if(str.equals("+"))
{
System.out.println("Not valid!");
return false;
}
else if(str.equals( "*"))
{
System.out.println("Not valid!");
return false;
}
else if (str.equals ("("))
{
boolean temp = TValidator();
if (!temp)
return false;
boolean temp2 = EPValidator();
if (!temp2)
return false;
// what do I do?
// build(1);
validity = true;
}
else if(str.equals( ")"))
{
System.out.println("Not valid!");
return false;
}
else if (str.equals( "$"))
{
System.out.println("Not valid!");
return false;
}
else
{
System.out.println("Invalid token");
return false;
}
// should I add a break?
return validity;
}
private boolean EPValidator()
{
boolean validity = false;
String str = TokenHandler.getNextToken(getString());
Integer n = 0;
try
{
n = Integer.parseInt(str);
if (n >32676)
n = 32676;
else n = n;
}
catch (NumberFormatException ime)
{
}
if(str.equals( n.toString()))
{
// still not sure how to get it to be n
System.out.println("Not valid!");
return false;
}
else if(str.equals("+"))
{
// remove token + somehow
setString(TokenHandler.deleteToken(getString()));
boolean temp = TValidator();
if (!temp)
return false;
boolean temp2 = EPValidator();
if (!temp2)
return false;
// what do I do?
// build(2);
validity = true;
}
else if(str.equals( "*"))
{
System.out.println("Not valid!");
return false;
}
else if(str.equals( "("))
{
System.out.println("Not valid!");
return false;
}
else if(str.equals( ")"))
{
// what do I do?
// build(3);
validity = true;
}
else if(str.equals( "$"))
{
// what do I do?
// build(3);
validity = true;
}
else
{
System.out.println("Invalid token");
return false;
}
// should I add a break?
return validity;
}
private boolean TValidator()
{
boolean validity = false;
String str = TokenHandler.getNextToken(getString());
Integer n = 0;
try
{
n = Integer.parseInt(str);
}
catch (NumberFormatException ime)
{
}
if(str.equals( n.toString()))
{
boolean temp = FValidator();
if(!temp)
return false;
boolean temp2 = TPValidator();
if(!temp2)
return false;
// what do I do?
// build(4);
validity = true;
}
else if(str.equals( "+"))
{
System.out.println("Not valid!");
return false;
}
else if(str.equals( "*"))
{
System.out.println("Not valid!");
return false;
}
else if(str.equals( "("))
{
boolean temp = FValidator();
if(!temp)
return false;
boolean temp2 = TPValidator();
if (!temp2)
return false;
// what do I do?
// build(4);
validity = true;
}
else if(str.equals( ")"))
{
System.out.println("Not valid!");
return false;
}
else if(str.equals("$"))
{
System.out.println("Not valid!");
return false;
}
else
{
System.out.println("Invalid token");
return false;
}
// should I add a break?
return validity;
}
private boolean TPValidator()
{
boolean validity = false;
String str = TokenHandler.getNextToken(getString());
Integer n = 0;
try
{
n = Integer.parseInt(str);
if (n >32676)
n = 32676;
else n = n;
}
catch (NumberFormatException ime)
{
}
if(str.equals( n.toString()))
{
System.out.println("Not valid!");
return false;
}
else if(str.equals( "+"))
{
// what do I do?
// setString(TokenHandler.deleteToken(getString()));
// build(6);
validity = true;
}
else if(str.equals( "*"))
{
// somehow remove * from token list
setString(TokenHandler.deleteToken(getString()));
boolean temp = FValidator();
if (!temp)
return false;
boolean temp2 = TPValidator();
if (!temp2)
return false;
// what do I do?
//build(5);
validity = true;
}
else if(str.equals( "("))
{
System.out.println("Not valid!");
return false;
}
else if(str.equals( ")"))
{
// what do I do?
// build(6);
validity = true;
}
else if(str.equals( "$"))
{
// what do I do?
// build(6);
validity = true;
}
else
{
System.out.println("Invalid token");
return false;
}
// should I add a break?
return validity;
}
private boolean FValidator()
{
boolean validity = false;
String str = TokenHandler.getNextToken(getString());
Integer n = 0;
try
{
n = Integer.parseInt(str);
if (n >32676)
n = 32676;
else n = n;
}
catch (NumberFormatException ime)
{
}
if(str.equals(n.toString()))
{
setString(TokenHandler.deleteToken(getString()));
// delete token n; how I don't know
// build(8);
validity = true;
}
else if(str.equals( "+"))
{
System.out.println("Not valid!");
return false;
}
else if(str.equals( "*"))
{
System.out.println("Not valid!");
return false;
}
else if(str.equals( "("))
{
setString(TokenHandler.deleteToken(getString()));
boolean temp = EValidator();
if(!temp)
return false;
if (TokenHandler.getNextToken(getString()) != ")")
return false;
else
setString(TokenHandler.deleteToken(getString()));
// what do I do?
// build(7);
validity = true;
}
else if(str.equals( ")"))
{
System.out.println("Not valid!");
return false;
}
else if(str.equals("$"))
{
System.out.println("Not valid!");
return false;
}
else
{
System.out.println("Invalid token");
return false;
}
// should I add a break?
return validity;
}
public static void main(String[] args)
{
LL1Parser parser;
try
{
// parser = new LL1Parser(args[0] + "$");
parser = new LL1Parser("123");
}
catch(ArrayIndexOutOfBoundsException aioobe)
{
System.out.println("args[0] doesn't exist!");
System.exit(0);
}
}
}
pradcoc@discovery:~/ITK 327/LL1 Parser> java LL1Parser 123
Came to read
validitiy is true after TValidator
validity is true after EPValidator
Validity is true at this point in read
Came to read
Not valid!
Validity is false at this point in read
false
What it should be:
Came to read
validitiy is true after TValidator
validity is true after EPValidator
Validity is true at this point in read
Came to read
Validity is true at this point in read
// some more calls possibly that always return true
true
Also, I tried it with
"123" just to see if that was the problem and it gave me this output, which is weird, considering both approaches shouldn't be correct. If it doesn't need a token, then " should be considered an invalid token, which is a different output from "Not valid!". But it's not.
pradcoc@discovery:~/ITK 327/LL1 Parser> java LL1Parser "123"
Came to read
validitiy is true after TValidator
validity is true after EPValidator
Validity is true at this point in read
Came to read
Not valid!
Validity is false at this point in read
false
I tried it by changing it to have
"123" as the parameter to the parser object in the main method and I got this error
pradcoc@discovery:~/ITK 327/LL1 Parser> java LL1Parser
Came to read
Exception in thread "main" java.lang.StringIndexOutOfBoundsException: String index out of range: 0
at java.lang.String.charAt(String.java:686)
at LL1Parser$TokenHandler.getNextToken(LL1Parser.java :23)
at LL1Parser.TPValidator(LL1Parser.java:411)
at LL1Parser.TValidator(LL1Parser.java:356)
at LL1Parser.read(LL1Parser.java:122)
at LL1Parser.isValid(LL1Parser.java:157)
at LL1Parser.<init>(LL1Parser.java:102)
at LL1Parser.main(LL1Parser.java:562)
Did find another problem at that line 23.
It shouldn't be tokenStream.length as it won't work in that case for numbers like 123 or indeed even 12.
But how do I fix that line?
-
Re: Confused on what tokenizer to use and how to use them.
Ok, I've tried fixing some things and now it will always say true and will go into an infinite loop and I don't know why!
Code java:
import java.util.StringTokenizer;
import java.io.*;
import java.util.*;
public class LL1Parser
{
private StringTokenizer tokenizer;
private boolean validity2;
private Stack<String> commandStack;
private String tokens;
private int originalLength;
private static class TokenHandler
{
private String str;
private char nextchar;
public static String getNextToken(String tokenStream)
{
if (tokenStream == null)
return null;
String numberString = "";
int index = 0;
char c = tokenStream.charAt(0);
if ( c == '+')
{
return "+";
}
else if (c == '*')
{
return "*";
}
else if (c == '(')
{
return "(";
}
else if (c == ')')
{
return ")";
}
else if (c == '0' || c == '1' || c == '2' || c == '3' || c == '4'|| c == '5' || c == '6' || c == '7'|| c == '8' || c == '9')
{
char c2 = 'a';
numberString = numberString + String.valueOf(c);
if (tokenStream.length() == 1)
{
return numberString;
}
else
{
c2 = tokenStream.charAt(1);
if (c2 == '0' || c2 == '1' || c2 == '2' || c2 == '3' || c2 == '4'|| c2 == '5' || c2 == '6' || c2 == '7'|| c2 == '8' || c2 == '9')
{
numberString = numberString + getNextToken(tokenStream.substring(1));
}
}
return numberString;
}
else if ( c == '$')
{
return "$";
}
else
{
return ("Bad String!");
}
}
public static String deleteToken(String tokenStream)
{
String temp = getNextToken(tokenStream);
int number = temp.length();
String temp2 = tokenStream.substring(number);
//tokenStream = temp2;
return temp2;
}
}
public LL1Parser(String input)
{
// tokenizer = new StringTokenizer(input);
// commandStack = new Stack<String>(); // creates an empty command stack
setString(input);
setOriginalLength(input.length());
setValidity(true);
// commandStack.push("T");
// commandStack.push("EP");
boolean b = isValid();
System.out.println(b);
}
public void setOriginalLength(int originalLength)
{
this.originalLength = originalLength;
}
public int getOriginalLength()
{
return originalLength;
}
public void setValidity(boolean validity2)
{
this.validity2 = validity2;
}
public boolean getValidity()
{
return validity2;
}
private boolean read()
{
System.out.println("Came to read");
boolean validity = true;
boolean temp = TValidator();
if(!temp)
return false;
System.out.println("validitiy is " +validity + " after TValidator");
boolean temp2 = EPValidator();
if(!temp2)
return false;
System.out.println("validity is " + validity + " after EPValidator" );
return validity;
}
private void setString(String tokens)
{
this.tokens = tokens;
}
private String getString()
{
return tokens;
}
public boolean isValid()
{
// System.out.println(tokenizer.nextToken());
// needs to check to see if String is null
// needs a base case anyway
boolean validity = getValidity();
while (getString() != null && validity == true )
{
// validity = read(st);
validity = read();
setValidity(validity);
System.out.println("Validity is " + validity + " at this point in read");
//if(validity == false)
//return false;
// else
// do something to remove some tokens
// create a StringTokenizer variable newSt based on this new value with the token removed
// isValid(newSt);
}
return validity;
}
private boolean EValidator()
{
System.out.println("EValidator method is being used.");
boolean validity = false;
if (TokenHandler.getNextToken(getString()) != null && getOriginalLength() != 0)
{
return true;
}
else if (TokenHandler.getNextToken(getString()) != null && getOriginalLength() == 0)
{
return false;
}
else
{
String str = TokenHandler.getNextToken(getString());
Integer n = 0;
try
{
n = Integer.parseInt(str);
if (n >32676)
n = 32676;
else n = n;
}
catch (NumberFormatException ime)
{
}
if (str.equals(n.toString()))
{
boolean temp = TValidator();
if (!temp)
return false;
boolean temp2 = EPValidator();
if(!temp2)
return false;
// how do I do this?
// build(1);
validity = true;
}
else if(str.equals("+"))
{
System.out.println("Not valid!");
return false;
}
else if(str.equals( "*"))
{
System.out.println("Not valid!");
return false;
}
else if (str.equals ("("))
{
boolean temp = TValidator();
if (!temp)
return false;
boolean temp2 = EPValidator();
if (!temp2)
return false;
// what do I do?
// build(1);
validity = true;
}
else if(str.equals( ")"))
{
System.out.println("Not valid!");
return false;
}
else if (str.equals( "$"))
{
System.out.println("Not valid!");
return false;
}
else
{
System.out.println("Invalid token");
return false;
}
}
// should I add a break?
return validity;
}
private boolean EPValidator()
{
boolean validity = false;
if (TokenHandler.getNextToken(getString()) != null && getOriginalLength() != 0)
{
return true;
}
else if (TokenHandler.getNextToken(getString()) != null && getOriginalLength() == 0)
{
return false;
}
else
{
String str = TokenHandler.getNextToken(getString());
Integer n = 0;
try
{
n = Integer.parseInt(str);
if (n >32676)
n = 32676;
else n = n;
}
catch (NumberFormatException ime)
{
}
if(str.equals( n.toString()))
{
// still not sure how to get it to be n
System.out.println("Not valid!");
return false;
}
else if(str.equals("+"))
{
// remove token + somehow
setString(TokenHandler.deleteToken(getString()));
boolean temp = TValidator();
if (!temp)
return false;
boolean temp2 = EPValidator();
if (!temp2)
return false;
// what do I do?
// build(2);
validity = true;
}
else if(str.equals( "*"))
{
System.out.println("Not valid!");
return false;
}
else if(str.equals( "("))
{
System.out.println("Not valid!");
return false;
}
else if(str.equals( ")"))
{
// what do I do?
// build(3);
validity = true;
}
else if(str.equals( "$"))
{
// what do I do?
// build(3);
validity = true;
}
else
{
System.out.println("Invalid token");
return false;
}
}
// should I add a break?
return validity;
}
private boolean TValidator()
{
boolean validity = false;
if (TokenHandler.getNextToken(getString()) != null && getOriginalLength() != 0)
{
return true;
}
else if (TokenHandler.getNextToken(getString()) != null && getOriginalLength() == 0)
{
return false;
}
else
{
String str = TokenHandler.getNextToken(getString());
Integer n = 0;
try
{
n = Integer.parseInt(str);
}
catch (NumberFormatException ime)
{
}
if(str.equals( n.toString()))
{
boolean temp = FValidator();
if(!temp)
return false;
boolean temp2 = TPValidator();
if(!temp2)
return false;
// what do I do?
// build(4);
validity = true;
}
else if(str.equals( "+"))
{
System.out.println("Not valid!");
return false;
}
else if(str.equals( "*"))
{
System.out.println("Not valid!");
return false;
}
else if(str.equals( "("))
{
boolean temp = FValidator();
if(!temp)
return false;
boolean temp2 = TPValidator();
if (!temp2)
return false;
// what do I do?
// build(4);
validity = true;
}
else if(str.equals( ")"))
{
System.out.println("Not valid!");
return false;
}
else if(str.equals("$"))
{
System.out.println("Not valid!");
return false;
}
else
{
System.out.println("Invalid token");
return false;
}
}
// should I add a break?
return validity;
}
private boolean TPValidator()
{
boolean validity = false;
if (TokenHandler.getNextToken(getString()) != null && getOriginalLength() != 0)
{
return true;
}
else if (TokenHandler.getNextToken(getString()) != null && getOriginalLength() == 0)
{
return false;
}
else
{
String str = TokenHandler.getNextToken(getString());
Integer n = 0;
try
{
n = Integer.parseInt(str);
if (n >32676)
n = 32676;
else n = n;
}
catch (NumberFormatException ime)
{
}
if(str.equals( n.toString()))
{
System.out.println("Not valid!");
return false;
}
else if(str.equals( "+"))
{
// what do I do?
// setString(TokenHandler.deleteToken(getString()));
// build(6);
validity = true;
}
else if(str.equals( "*"))
{
// somehow remove * from token list
setString(TokenHandler.deleteToken(getString()));
boolean temp = FValidator();
if (!temp)
return false;
boolean temp2 = TPValidator();
if (!temp2)
return false;
// what do I do?
//build(5);
validity = true;
}
else if(str.equals( "("))
{
System.out.println("Not valid!");
return false;
}
else if(str.equals( ")"))
{
// what do I do?
// build(6);
validity = true;
}
else if(str.equals( "$"))
{
// what do I do?
// build(6);
validity = true;
}
else
{
System.out.println("Invalid token");
return false;
}
// should I add a break?
}
return validity;
}
private boolean FValidator()
{
boolean validity = false;
if (TokenHandler.getNextToken(getString()) != null && getOriginalLength() != 0)
{
return true;
}
else if (TokenHandler.getNextToken(getString()) != null && getOriginalLength() == 0)
{
return false;
}
else
{
String str = TokenHandler.getNextToken(getString());
Integer n = 0;
try
{
n = Integer.parseInt(str);
if (n >32676)
n = 32676;
else n = n;
}
catch (NumberFormatException ime)
{
}
if(str.equals(n.toString()))
{
setString(TokenHandler.deleteToken(getString()));
// delete token n; how I don't know
// build(8);
validity = true;
}
else if(str.equals( "+"))
{
System.out.println("Not valid!");
return false;
}
else if(str.equals( "*"))
{
System.out.println("Not valid!");
return false;
}
else if(str.equals( "("))
{
setString(TokenHandler.deleteToken(getString()));
boolean temp = EValidator();
if(!temp)
return false;
if (TokenHandler.getNextToken(getString()) != ")")
return false;
else
setString(TokenHandler.deleteToken(getString()));
// what do I do?
// build(7);
validity = true;
}
else if(str.equals( ")"))
{
System.out.println("Not valid!");
return false;
}
else if(str.equals("$"))
{
System.out.println("Not valid!");
return false;
}
else
{
System.out.println("Invalid token");
return false;
}
}
// should I add a break?
return validity;
}
public static void main(String[] args)
{
LL1Parser parser;
try
{
// parser = new LL1Parser(args[0] + "$");
parser = new LL1Parser("+123");
}
catch(ArrayIndexOutOfBoundsException aioobe)
{
System.out.println("args[0] doesn't exist!");
System.exit(0);
}
}
}
Also, now for this type of syntax the answer should be false anyway.
I changed it from 123 to +123 to see if it at least was getting the validity right but it's not getting it right.
:-ob-(b-(=((=((:((:((:((~X(~X(~X(
Is this part of code the problem?
Code java:
if (TokenHandler.getNextToken(getString()) != null && getOriginalLength() != 0)
{
return true;
}
else if (TokenHandler.getNextToken(getString()) != null && getOriginalLength() == 0)
{
return false;
}
-
Re: Confused on what tokenizer to use and how to use them.
Quote:
Is this part of code the problem?
Your code has no comments describing your logic so its very hard to determine if the code is doing what you want it to do.
-
Re: Confused on what tokenizer to use and how to use them.
Well, considering what it IS doing at the moment, right now I'd like it NOT to loop infinitely.
Here's how it should go:
Code :
E():
switch(token)
{
case n: T(); E'();
build E -> TE';
break;
case +:
error;
break;
case *:
error;
break;
case (:
T(); E'();
build E ->TE';
break;
case ):
error;
break;
case $:
error;
break;
}
E'():
Switch(token)
{
case n:
error;
break;
case +:
get+);T(); E'();
case *;
error;
break;
case (:
error;
break;
case ):
build E' ->lambda;
break;
case $:
build E' -> lambda;
break;
}
T():
Switch(token)
{
case n:
build T->FT';
break;
case +:
error;
break;
case *:
error;
break;
case (:
build T->FT';
break;
case (:
error;
break;
case $:
error;
break;
}
T'():
Switch(token)
{
case n:
error;
break;
case +:
build T' -> lambda;
break;
case *:
get(*); F(); T'();
build T'-> *FT';
break;
case (:
error;
break;
case ):
build T'->lambda;
break;
case $:
build T'->lambda;
break;
}
F():
Switch(token)
{
case n:
get(n);
build F->n;
break;
case +:
error;
break;
case *:
error;
break;
case (:
get('(');E(); get(')');
build F->(E);
break;
case ):
error;
break;
case $:
error;
break;
}
Well the break thing doesn't work with string so simply replace those with if, else if, else, etc.
What's going on is that it'll recursively call the validators, starting immediately with the TValidator() and then the EPValidator()
The code should cause it to keep returning false if, while the execution is still running, it finds ANY place where it's not true in the grammar.
However, it had been complaining that I was going out of bounds possibly so I grudgingly added another variable to keep track of the original size of the original string and told it if it ran out of tokens to return true unless of course it was null to start with, in which case return false.
However, after doing this, it loops infinitely going through the printlns till Judgment Day so I can't show the output.
The cases where it would return not valid would be if it found the wrong token as the next token when it shouldn't be.
Each validator is called when the next part, or reading, or whatever it's called, is called so if it is checking E then if it encounters bad tokens, then it should return false and keep doing so recurisively or via my booleans and checking to see if the validators are returning false.
Before it never entered the validators. Now it does but
1.) It always is true
2.) It loops forever
What is causing the infinite loop?
Other than the recursive calls, this is the only place where the infinite loop could be happening.
Code java:
while (getString() != null && validity == true )
{
// validity = read(st);
validity = read();
setValidity(validity);
System.out.println("Validity is " + validity + " at this point in read");
//if(validity == false)
//return false;
// else
// do something to remove some tokens
// create a StringTokenizer variable newSt based on this new value with the token removed
// isValid(newSt);
}
Hmmmmmm...it appears to be repeating the TValidator and EPValidator over and over and never going anywhere else. How come?
It appears to be coming from the while loop in the isValid() method.
Code java:
private boolean read()
{
System.out.println("Came to read");
boolean validity = getValidity();
boolean temp = TValidator();
if(!temp)
return false;
System.out.println("validitiy is " +validity + " after TValidator");
boolean temp2 = EPValidator();
if(!temp2)
return false;
System.out.println("validity is " + validity + " after EPValidator" );
return validity;
}
private void setString(String tokens)
{
this.tokens = tokens;
}
private String getString()
{
return tokens;
}
public boolean isValid()
{
// System.out.println(tokenizer.nextToken());
// needs to check to see if String is null
// needs a base case anyway
boolean validity = true;
int counter = 0;
while (getString() != null && validity == true )
{
// validity = read(st);
validity = read();
setValidity(validity);
System.out.println("Validity is " + validity + " at this point in read");
//if(validity == false)
//return false;
// else
// do something to remove some tokens
// create a StringTokenizer variable newSt based on this new value with the token removed
// isValid(newSt);
}
return validity;
}
Either one of 2 or three things is happening
1.) Something is happening wrong in read
2.) Something is happening wrong in isValid
3.) The token isn't being deleted by deleteToken() and it's going on forever (this could very well be happening)
It's going through these things below and then looping and doing it over and over. I was able to freeze it by adding in a break so it won't go on forever.
Came to read
validitiy is true after TValidator
validity is true after EPValidator
Validity is true at this point in read
The next thing is
Came to read
followed by
validity is true after TValidator ...etc
Ok, I've found it's removing the token. But maybe it's never reaching a case where it WOULD remove the token, though it should be reaching such a case.
-
Re: Confused on what tokenizer to use and how to use them.
In first place, we have to follow the flow of the program...
Legend-> Class::method::statement
I only put the important lines...
LL1Parser::main(String[] args)::"parser = new LL1Parser("+123");"
LL1Parser::LL1Parser(String input)::"boolean b = isValid();"
LL1Parser::isValid()::"while (getString() != null && validity == true)"
LL1Parser::isValid()::"validity = read();"
LL1Parser::read()::"boolean temp = TValidator();"
LL1Parser::TValidator()::"if (TokenHandler.getNextToken(getString()) != null && getOriginalLength() != 0)"
LL1Parser::TValidator()::"return true;"
LL1Parser::read()::"boolean temp2 = EPValidator();"
LL1Parser::EPValidator()::"if (TokenHandler.getNextToken(getString()) != null && getOriginalLength() != 0)"
LL1Parser::EPValidator()::"return true;"
... then always it's happen ...
LL1Parser::isValid()::"while (getString() != null && validity == true)"
LL1Parser::isValid()::"validity = read();"
LL1Parser::read()::"boolean temp = TValidator();"
LL1Parser::TValidator()::"if (TokenHandler.getNextToken(getString()) != null && getOriginalLength() != 0)"
LL1Parser::TValidator()::"return true;"
LL1Parser::read()::"boolean temp2 = EPValidator();"
LL1Parser::EPValidator()::"if (TokenHandler.getNextToken(getString()) != null && getOriginalLength() != 0)"
LL1Parser::EPValidator()::"return true;"
This expression is always true "TokenHandler.getNextToken(getString()) != null"
and this one too "getOriginalLength() != 0"
-
Re: Confused on what tokenizer to use and how to use them.
@javapenguin:
I would suggest you to read this!!!
Your every post contains alot of details with nothing related to problem. You, yourself question your problem and answer it and then come with a very new problem, diverting from the main problem.
-
Re: Confused on what tokenizer to use and how to use them.
This OP seems to be an output only device.
-
Re: Confused on what tokenizer to use and how to use them.
Quote:
Originally Posted by
Norm
This OP seems to be an output only device.
Very confusing indeed.
Javapenguin, don't write your every thought in the thread! We don't need to read your thought process. Keep it short and relevant.
Take on board what is being said before you post back. You are rendering your threads useless.
-
Re: Confused on what tokenizer to use and how to use them.
I had to hand it in and was nearly done with it anyway.
-
Re: Confused on what tokenizer to use and how to use them.
You should get your own private forum where you can talk to yourself and be completely free to post as often as you want.
-
Re: Confused on what tokenizer to use and how to use them.
Sorry. I try and update things so people don't try and help me with something I've already solved.
Since my threads normally don't get a lot of replies, I can often sometimes solve it before I get a single reply, hence the updates.
-
Re: Confused on what tokenizer to use and how to use them.
@javapenguin: It's not necessary to get a reply for every post on your desired time. You either have to be patient or think of your own.
Not being so offensive, but everyone here tries best to help OP.
