Welcome to the Java Programming Forums


The professional, friendly Java community. 21,500 members and growing!


The Java Programming Forums are a community of Java programmers from all around the World. Our members have a wide range of skills and they all have one thing in common: A passion to learn and code Java. We invite beginner Java programmers right through to Java professionals to post here and share your knowledge. Become a part of the community, help others, expand your knowledge of Java and enjoy talking with like minded people. Registration is quick and best of all free. We look forward to meeting you.


>> REGISTER NOW TO START POSTING


Members have full access to the forums. Advertisements are removed for registered users.

Results 1 to 25 of 25

Thread: Confused on what tokenizer to use and how to use them.

  1. #1
    Banned
    Join Date
    May 2010
    Location
    North Central Illinois
    Posts
    1,631
    My Mood
    Sleepy
    Thanks
    390
    Thanked 112 Times in 110 Posts

    Default Confused on what tokenizer to use and how to use them.

    I'm supposed to be making a parser for grammars, or basically parsing a simple language definition.

    Basically I could be parsing either number or a symbol (+, *, (, ), or lambda, maybe lambda anyway, etc)

    I want it to read in 123 as "123", not as 1, then 2, then 3 as three separate chars.

    I want it only to advance the tokens thought based upon certain conditions. Not always.

    I've started the actual code of this thing, though haven't gotten far.

    import java.util.StringTokenizer;
    import java.io.*;
    import java.util.*;
     
     
    public class LL1Parser
    {
     
    private StringTokenizer tokenizer;
    private boolean valid;
     
    public LL1Parser(String input)
    {
     
    tokenizer = new StringTokenizer(input);
     
     
    }
     
    public boolean isValid(StringTokenizer st)
    {
     
     
     
    }
     
     
     
     
    public static void main(String[] args) 
    {
    LL1Parser parser;
    try
    {
    parser = new LL1Parser(args[0]);
    }
     
    catch(NullPointerException npe)
    {
    System.out.println("args[0] doesn't exist!");
    System.exit(0);
    }
     
     
     
     
    }

    The pseudo code for the parsing is:

    E():
    switch(token)
    {
    case n: T(); E'();
    build E -> TE';
    break;
    case +:
    error;
    break;
    case *:
    error;
    break;
    case (:
    T(); E'(); 
    build E ->TE';
    break;
    case ):
    error;
    break;
    case $:
    error;
    break;
     
    }
     
    E'():
    Switch(token)
    {
    case n:
    error;
    break;
    case +:
    get(+);T(); E'();
    break;
    case *;
    error;
    break;
    case (:
    error;
    break;
    case ):
    build E' ->lambda;
    break;
    case $:
    build E' -> lambda;
    break;
    }
     
    T():
    Switch(token)
    {
    case n: 
    build T->FT';
    break;
    case +:
    error;
    break;
    case *:
    error;
    break;
    case (:
    build T->FT';
    break;
    case ):
    error;
    break;
    case $:
    error;
    break;
    }
     
    T'():
    Switch(token)
    {
    case n: 
    error;
    break;
    case +:
    build T' -> lambda;
    break;
    case *:
    get(*); F(); T'();
    build T'-> *FT';
    break;
    case (:
    error;
    break;
    case ):
    build T'->lambda;
    break;
    case $:
    build T'->lambda;
    break;
    }
     
    F():
    Switch(token)
    {
    case n:
    get(n);
    build F->n;
    break;
    case +:
    error;
    break;
    case *:
    error;
    break;
    case (: 
    get('(');E(); get(')');
    build F->(E);
    break;
    case ):
    error;
    break;
    case $:
    error;
    break;
    }

    Basically it's kind of recursive. I'm not so much concerned about what to write right now as how to work a Tokenizer.

    If you're confused, which I have a strange feeling you are, I'm going to try and clarify:

    It will read in a 1. As the original starting part is an E, it will go into the E part.

    Since 1 is n where n is any integer less than 32767 it will go to the n case in the E part and will check out the

    T(); E'(); part. Since it's going from left to right, it checks the T part first, and still has the same token as it hasn't removed it yet.

    It goes to case n of the T part.

    It tells it to go to FT' so now it's

    F(); T'(); E'();

    it checks out the F part first with the same token still.

    It goes to case n and asks to get(n)

    If this token WEREN'T a number, it would have something saying the grammar was incorrect.

    As it is, this test of mine has it as a number so it removes that token from the input string and then checks out

    T'(); E'();

    The next token now is a +. It will check out the + case for T' which tells it to go to lambda, basically just getting rid of the T' but not changing the Token. Now it checks + in E'

    get(+);T(); E'();
    break;

    which removes the + token and then has evaluates the next token, (, as in T.

    so now it's

    F(); T'(); E'();

    get('(');E(); get(')')

    The ( is in quotes so it won't get confused with the regular parenthesis.

    This removes the ( and the checks

    E(); get(')'); T'(); E'();

    the next token happens to be ( so now it's

    T(); E'(); get(')'); T'(); E'();

    F(); T'(); E'(); get(')'); T'(); E'();

    get('(');E(); get(')'); T'(); E'(); get(')'); T'(); E'();

    so it removes the ( and goes to

    E(); get(')'); T'(); E'(); get(')'); T'(); E'();

    The next token is now a 23

    T(); E'(); get(')'); T'(); E'(); get(')'); T'(); E'();

    F(); T'(); E'(); get(')'); T'(); E'(); get(')'); T'(); E'();

    get(n); T'(); E'(); get(')'); T'(); E'(); get(')'); T'(); E'();

    so remove the 23 and then + is the new token.

    T'(); E'(); get(')'); T'(); E'(); get(')'); T'(); E'();

    It goes to lambda so now it's

    E'(); get(')'); T'(); E'(); get(')'); T'(); E'();

    get(+);T(); E'(); get(')'); T'(); E'(); get(')'); T'(); E'();

    That removes the + and 2 is the next token.

    F(); T'(); E'(); get(')'); T'(); E'(); get(')'); T'(); E'();

    get(n); T'(); E'(); get(')'); T'(); E'(); get(')'); T'(); E'();

    so now the 2 is gone and * is the next token.

    T'(); E'(); get(')'); T'(); E'(); get(')'); T'(); E'();

    get; F(); T'(); E'(); get(')'); T'(); E'(); get(')'); T'(); E'();

    So * is gone and 3 is the next token.

    F(); T'(); E'(); get(')'); T'(); E'(); get(')'); T'(); E'();

    get(n); T'(); E'(); get(')'); T'(); E'(); get(')'); T'(); E'();

    so now 3 is gone and ) is the new token.

    T'(); E'(); get(')'); T'(); E'(); get(')'); T'(); E'();

    So T' goes to lambda and now it's

    E'(); get(')'); T'(); E'(); get(')'); T'(); E'();

    E' goes to lambda so it's

    get(')'); T'(); E'(); get(')'); T'(); E'();

    So now the ) is gone.

    Now ) is the next token.

    T'(); E'(); get(')'); T'(); E'();

    T' goes to lambda so it's

    E'(); get(')'); T'(); E'();

    E' goes to lambda too so now it's

    get(')'); T'(); E'();

    so the ) is gone and now, say, ) is the next token. (This will ruin the grammar so I'll show a correct part later.)

    T'(); E'();

    Ok, I've messed up somewhere in there, but you get the point.

    I don't know how to use a Tokenizer to do this.
    Last edited by javapenguin; November 6th, 2011 at 08:21 PM.


  2. #2
    Banned
    Join Date
    May 2010
    Location
    North Central Illinois
    Posts
    1,631
    My Mood
    Sleepy
    Thanks
    390
    Thanked 112 Times in 110 Posts

    Default Re: Confused on what tokenizer to use and how to use them.

    Ok, I've updated it a bit.

    import java.util.StringTokenizer;
    import java.io.*;
    import java.util.*;
     
     
    public class LL1Parser
    {
     
    private StringTokenizer tokenizer;
    private boolean valid;
     
    public LL1Parser(String input)
    {
     
    tokenizer = new StringTokenizer(input);
     
    System.out.println(isValid(tokenizer));
     
     
    }
     
    public boolean isValid(StringTokenizer st)
    {
     
    // needs to check to see if String is null 
    // needs a base case anyway
     
    while (st.hasMoreTokens())
    {
    if (!EValidator(st))
    return false;
     
    if (!E'Validator(st))
    return false;
     
    if (!TValidator(st))
    return false;
     
    if (!T'Validator(st))
    return false;
     
    if (!FValidator(st))
    return false;
     
    else 
     
    // do something to remove some tokens
    // create a StringTokenizer variable newSt based on this new value with the token removed
     
    isValid(newSt);
     
     
     
    }
    return true;
     
    }
     
    private boolean EValidator(StringTokenizer st)
    {
     
    boolean validity = false;
    String str = st.nextToken();
     
    switch(str)
    {
    case n: 
    // how do I do this?
     
    case +:
    System.out.println("Not valid!");
    break;
     
    case *:
    System.out.println("Not valid!");
    break;
     
    case ( :
    // what do I do?
     
    case ) :
    System.out.println("Not valid!");
    break;
     
    case $:
    System.out.println("Not valid!");
    break;
    }
    return validity;
     
     
    }
     
    private boolean E'Validator(StringTokenizer st)
    {
     
     
    }
     
    private boolean TValidator(StringTokenizer st)
    {
     
     
    }
     
    private boolean T'Validator(StringTokenizer st)
    {
     
    }
     
    private boolean FValidator(StringTokenizer st)
    {
     
    }
     
     
     
    public static void main(String[] args) 
    {
    LL1Parser parser;
    try
    {
    parser = new LL1Parser(args[0]);
    }
     
    catch(NullPointerException npe)
    {
    System.out.println("args[0] doesn't exist!");
    System.exit(0);
    }
     
     
     
     
    }

  3. #3
    Banned
    Join Date
    May 2010
    Location
    North Central Illinois
    Posts
    1,631
    My Mood
    Sleepy
    Thanks
    390
    Thanked 112 Times in 110 Posts

    Cool Re: Confused on what tokenizer to use and how to use them.

    Please reply.

       import java.util.StringTokenizer;
       import java.io.*;
       import java.util.*;
     
     
       public class LL1Parser
       {
     
          private StringTokenizer tokenizer;
          private boolean valid;
          private Stack<String> commandStack;
          private String tokens;
     
          public LL1Parser(String input)
          {
     
             tokenizer = new StringTokenizer(input);
             commandStack = new Stack<String>(); // creates an empty command stack
     
             commandStack.push("T");
             commandStack.push("EP");
             System.out.println(isValid(tokenizer));
     
     
          }
     
          private boolean read(StringTokenizer st)
          {
     
             if (commandStack.peek().equals("E"))
             {
                return EValidator(st);
             }
     
             else if(commandStack.peek().equals("EP"))
                return EPValidator(st);
             else if (commandStack.peek().equals("T"))
                return TValidator(st);
             else if(commandStack.peek().equals("TP"))
                return TPValidator(st);
             else if (commandStack.peek().equals("F"))
                return FValidator(st);
     
             else if (commandStack.peek().equals("get +"))
             {
                if (st.nextToken().equals("+"))
                {
                   setString(getString().substring(1));
                   return true;
                }
                else
                   return false;
     
             }
             else if (commandStack.peek().equals("get *"))
             {
                if(st.nextToken().equals("*"))
                {
                   setString(getString().substring(1));
                   return true;
     
                }
                else
                   return false;
             }
     
             else if(commandStack.peek().equals("get ("))
             {
                if(st.nextToken().equals("("))
                {
                   setString(getString().substring(1));
                   return true;
     
                }
                else
                   return false;
     
     
             }
     
             else if(commandStack.peek().equals("get )"))
             {
                if(st.nextToken().equals(")"))
                {
                   setString(getString().substring(1));
                   return true;
     
                }
                else
                   return false;
     
     
             }
     
    			else if(commandStack.peek().equals("get n"))
             {
    			Integer n = 0;
    			try
    			{
    			n = Integer.parseInt(st.nextToken());
    			}
     
    			catch(NumberFormatException nfe)
    			{
     
    			}
                if(st.nextToken().equals(n.toString()))
                {
                   setString(getString().substring(1));
                   return true;
     
                }
                else
                   return false;
     
     
             }
          else 
    		return false;
     
          }
     
          private void setString(String tokens)
          {
             this.tokens = tokens;
          }
     
          private String getString()
          {
             return tokens;
          }
     
          private void build(int rule)
          {
             if (rule == 1)
             {
                commandStack.pop();
                commandStack.push("T");
                commandStack.push("EP");
             }
     
             else if (rule == 2)
             {
                commandStack.pop();
                commandStack.push("get +");
                commandStack.push("T");
                commandStack.push("EP");
             }
     
             else if(rule == 3)
             {
                commandStack.pop();
             }
     
             else if(rule == 4)
             {
                commandStack.pop();
                commandStack.push("F");
                commandStack.push("TP");
             }
     
             else if (rule == 5)
             {
                commandStack.pop();
                commandStack.push("get *");
                commandStack.push("F");
                commandStack.push("TP");
             }
     
             else if (rule == 6)
             {
                commandStack.pop();
             }
     
             else if (rule == 7)
             {
                commandStack.pop();
                commandStack.push("get (");
                commandStack.push("E");
                commandStack.push("get )");
             }
     
             else if(rule == 8)
             {
                commandStack.pop();
                commandStack.push("get n");
             }
     
             else
                System.out.println("Invalid rule");
          }
     
     
     
          public boolean isValid(StringTokenizer st)
          {
     
          // needs to check to see if String is null 
          // needs a base case anyway
     
             while (st.hasMoreTokens())
             {
                read(st);
     
              //  else 
     
                // do something to remove some tokens
                // create a StringTokenizer variable newSt based on this new value with the token removed
     
                 //  isValid(newSt);
     
     
     
             }
             return true;
     
          }
     
          private boolean EValidator(StringTokenizer st)
          {
     
             boolean validity = false;
             String str = st.nextToken();
             Integer n = 0;
             try
             {
                n = Integer.parseInt(str);
             }
     
                catch (NumberFormatException ime)
                {
     
                }
     
     
     
     
             if (str.equals(n.toString()))
             {
             // how do I do this?
                build(1);
                validity = true;
             }
             else if(str.equals("+"))
             {
                System.out.println("Not valid!");
             }
             else if(str.equals( "*"))
             {
                System.out.println("Not valid!");
             }
             else if (str.equals ("("))
             {
             // what do I do?
                build(1);
                validity = true;
             }
     
             else if(str.equals( ")")) 
             {
                System.out.println("Not valid!");
     
             }
     
             else if (str.equals( "$"))
             {
                System.out.println("Not valid!");
     
             }
     
             else 
                System.out.println("Invalid token");
          // should I add a break?
             return validity;
     
     
          }
     
          private boolean EPValidator(StringTokenizer st)
          {
             boolean validity = false;
             String str = st.nextToken();
             Integer n = 0;
             try
             {
                n = Integer.parseInt(str);
             }
     
                catch (NumberFormatException ime)
                {
     
                }
     
             if(str.equals( n.toString()))
             {
             // still not sure how to get it to be n
                System.out.println("Not valid!");
             }
     
             else if(str.equals("+"))
             {
             // what do I do?
                build(2);
                validity = true;
             }
     
             else if(str.equals( "*"))
             {
                System.out.println("Not valid!");
             }
             else if(str.equals( "("))
             {
                System.out.println("Not valid!");
             }
             else if(str.equals( ")"))
             {
             // what do I do?
                build(3);
                validity = true;
             }
             else if(str.equals( "$"))
             {
             // what do I do?
                build(3);
                validity = true;
             }
     
             else
                System.out.println("Invalid token");
          // should I add a break?
     
     
             return validity;
          }
     
          private boolean TValidator(StringTokenizer st)
          {
             boolean validity = false;
             String str = st.nextToken();
             Integer n = 0;
             try
             {
                n = Integer.parseInt(str);
             }
     
                catch (NumberFormatException ime)
                {
     
                }
     
             if(str.equals( n.toString()))
             {
                // what do I do?
                build(4);
                validity = true;
             }
             else if(str.equals( "+"))
             {
                System.out.println("Not valid!");
             }
             else if(str.equals( "*"))
             {
                System.out.println("Not valid!");
             }
     
             else if(str.equals( "("))
             {
                // what do I do?
                build(4);
                validity = true;
             }
     
             else if(str.equals( ")"))
             {
                System.out.println("Not valid!");
             }
             else if(str.equals("$"))
             {
                System.out.println("Not valid!");
             }
             else
                System.out.println("Invalid token");
             // should I add a break?
     
     
             return validity;
          }
     
          private boolean TPValidator(StringTokenizer st)
          {
             boolean validity = false;
             String str = st.nextToken();
             Integer n = 0;
             try
             {
                n = Integer.parseInt(str);
             }
     
                catch (NumberFormatException ime)
                {
     
                }
     
             if(str.equals( n.toString()))
             {
                System.out.println("Not valid!");
             }
             else if(str.equals( "+"))
             {
                // what do I do?
                build(6);
                validity = true;
             }
             else if(str.equals( "*"))
             {
                // what do I do?
                build(5);
                validity = true;
             }
             else if(str.equals( "("))
             {
                System.out.println("Not valid!");
             }
             else if(str.equals( ")"))
             {
                // what do I do?
                build(6);
                validity = true;
             }
             else if(str.equals( "$"))
             {
     
                // what do I do?
                build(6);
                validity = true;
             }
             else
                System.out.println("Invalid token");
             // should I add a break?
     
             return validity;
          }
     
          private boolean FValidator(StringTokenizer st)
          {
     
             boolean validity = false;
             String str = st.nextToken();
             Integer n = 0;
             try
             {
                n = Integer.parseInt(str);
             }
     
                catch (NumberFormatException ime)
                {
     
                }
     
             if(str.equals(n.toString())) 
             {
                // delete token n; how I don't know
                build(8);
                validity = true;
             }
     
             else if(str.equals( "+"))
             {
                System.out.println("Not valid!");
             }
     
             else if(str.equals( "*"))
             {
                System.out.println("Not valid!");
             }
             else if(str.equals( "("))
             {
                // what do I do?
                build(7);
                validity = true;
             }
             else if(str.equals( ")"))
             {
                System.out.println("Not valid!");
             }
             else if(str.equals("$"))
             {
                System.out.println("Not valid!");
             }
             else
                System.out.println("Invalid token");
             // should I add a break?
     
     
             return validity;
          }
     
     
     
          public static void main(String[] args) 
          {
             LL1Parser parser;
     
             try
             {
                parser = new LL1Parser(args[0] + "$");
             }
     
                catch(ArrayIndexOutOfBoundsException aioobe)
                {
                   System.out.println("args[0] doesn't exist!");
                   System.exit(0);
                }
     
     
     
     
          }
       }

    Ok I think I can get it to update the String to simply get the subsring of that string by substring(1) so I remove the first character but how do I keep applying it to that read function with the smaller input String? I already have setString() and getString() methods to use. However, I'm confused on how to shrink the String in the right way and apply it to the tokenizer in the right way. Also, I am wondering how my setup will work with having read() return a boolean.

    It needs to do something if the token it tries to remove is the current (or maybe it was the next) token. Think it's the current but could be wrong. Anyway, if, for instance there's a "get n" on the top of the commandStack but the nextToken is a "+", then it's an invalid grammar String and it should say so and stop the procedure and cause isValid() to return false and also NOT try to evaluate the rest of the stuff as it's invalid. Also, if there is nothing left on the commandStack but there are tokens left, then it's also invalid. If there's nothing left in either, then and only then I think should it return true for isValid().
    Last edited by javapenguin; November 7th, 2011 at 11:45 PM.

  4. #4
    Super Moderator Norm's Avatar
    Join Date
    May 2010
    Location
    Eastern Florida
    Posts
    21,051
    Thanks
    52
    Thanked 2,280 Times in 2,251 Posts

    Default Re: Confused on what tokenizer to use and how to use them.

    Can you write a small simple test parser program to work on only this problem? Include all input data in the program itself, vs requiring any user input.
    Show what your test parser code does and add comments to the output showing what you want it to do.

  5. #5
    mmm.. coffee JavaPF's Avatar
    Join Date
    May 2008
    Location
    United Kingdom
    Posts
    3,336
    My Mood
    Mellow
    Thanks
    258
    Thanked 287 Times in 225 Posts
    Blog Entries
    4

    Default Re: Confused on what tokenizer to use and how to use them.

    There is too much clutter in the first post. It restricts replies.
    Please use [highlight=Java] code [/highlight] tags when posting your code.
    Forum Tip: Add to peoples reputation by clicking the button on their useful posts.

    Looking for a Java job? Visit - Java Programming Careers

  6. #6
    Banned
    Join Date
    May 2010
    Location
    North Central Illinois
    Posts
    1,631
    My Mood
    Sleepy
    Thanks
    390
    Thanked 112 Times in 110 Posts

    Default Re: Confused on what tokenizer to use and how to use them.

    There is something I'm wondering. When I call nextToken(), does it simply return but not remove the next token or is it also removing it from the token string at the same time?

    I cannot figure out where the problem is but know it's somehow bypassing all of my if statements and somehow always going to the else statements. I cannot break it down much further as I'm not quite sure what's causing the problem. A lot of my printlns aren't even printing. Also, how do I get it to update the tokenizer so it changes? I don't think I'm doing it correctly every time.

     import java.util.StringTokenizer;
       import java.io.*;
       import java.util.*;
     
     
       public class LL1Parser
       {
     
          private StringTokenizer tokenizer;
          private boolean valid;
          private Stack<String> commandStack;
          private String tokens;
     
          public LL1Parser(String input)
          {
     
             tokenizer = new StringTokenizer(input);
             commandStack = new Stack<String>(); // creates an empty command stack
             setString(input);
             commandStack.push("T");
             commandStack.push("EP");
                            boolean b = isValid(tokenizer);
             System.out.println(b);
     
     
          }
     
          private boolean read(StringTokenizer st)
          {
     
             if (commandStack.peek().equals("E"))
             {
                            System.out.println(EValidator(st));
                return EValidator(st);
             }
     
             else if(commandStack.peek().equals("EP"))
                return EPValidator(st);
             else if (commandStack.peek().equals("T"))
                            {
                            System.out.println(TValidator(st));
                return TValidator(st);
     
                                    }
             else if(commandStack.peek().equals("TP"))
                return TPValidator(st);
             else if (commandStack.peek().equals("F"))
                return FValidator(st);
     
             else if (commandStack.peek().equals("get +"))
             {
                if (st.nextToken().equals("+"))
                {
                   setString(getString().substring(1));
                   return true;
                }
                else
                   return false;
     
             }
             else if (commandStack.peek().equals("get *"))
             {
                if(st.nextToken().equals("*"))
                {
                   setString(getString().substring(1));
                   return true;
     
                }
                else
                   return false;
             }
     
             else if(commandStack.peek().equals("get ("))
             {
                if(st.nextToken().equals("("))
                {
                   setString(getString().substring(1));
                   return true;
     
                }
                else
                   return false;
     
     
             }
     
             else if(commandStack.peek().equals("get )"))
             {
                if(st.nextToken().equals(")"))
                {
                   setString(getString().substring(1));
                   return true;
     
                }
                else
                  return false;
     
     
     
             }
     
                            else if(commandStack.peek().equals("get n"))
             {
                            Integer n = 0;
                            try
                            {
                            n = Integer.parseInt(st.nextToken());
                            }
     
                            catch(NumberFormatException nfe)
                            {
     
                            }
                if(st.nextToken().equals(n.toString()))
                {
                   setString(getString().substring(1));
                   return true;
     
                }
                else
                   return false;
     
     
             }
          else
                    return false;
     
          }
     
          private void setString(String tokens)
          {
             this.tokens = tokens;
          }
     
          private String getString()
          {
             return tokens;
          }
     
          private void build(int rule)
          {
             if (rule == 1)
             {
                commandStack.pop();
                commandStack.push("T");
                commandStack.push("EP");
             }
     
             else if (rule == 2)
             {
                commandStack.pop();
                commandStack.push("get +");
                commandStack.push("T");
                commandStack.push("EP");
             }
     
             else if(rule == 3)
             {
                commandStack.pop();
             }
     
             else if(rule == 4)
             {
                commandStack.pop();
                commandStack.push("F");
                commandStack.push("TP");
             }
     
             else if (rule == 5)
             {
                commandStack.pop();
                commandStack.push("get *");
                commandStack.push("F");
                commandStack.push("TP");
             }
     
             else if (rule == 6)
             {
                commandStack.pop();
             }
     
             else if (rule == 7)
             {
                commandStack.pop();
                commandStack.push("get (");
                commandStack.push("E");
                commandStack.push("get )");
             }
     
             else if(rule == 8)
             {
                commandStack.pop();
                commandStack.push("get n");
             }
     
             else
                System.out.println("Invalid rule");
          }
     
     
     
          public boolean isValid(StringTokenizer st)
          {
     
                    System.out.println(st.nextToken());
     
          // needs to check to see if String is null
          // needs a base case anyway
          boolean validity = true;
             while (st.hasMoreTokens() && validity == true )
             {
              // validity = read(st);
                validity = read(new StringTokenizer(getString()));
              //  else
     
                // do something to remove some tokens
                // create a StringTokenizer variable newSt based on this new value with the token removed
     
                 //  isValid(newSt);
     
     
     
             }
             return validity;
     
          }
     
          private boolean EValidator(StringTokenizer st)
          {
     
             boolean validity = false;
             String str = st.nextToken();
             Integer n = 0;
             try
             {
                n = Integer.parseInt(str);
             }
     
                catch (NumberFormatException ime)
                {
     
                }
     
     
     
     
             if (str.equals(n.toString()))
             {
             // how do I do this?
                build(1);
                validity = true;
             }
             else if(str.equals("+"))
             {
                System.out.println("Not valid!");
             }
             else if(str.equals( "*"))
             {
                System.out.println("Not valid!");
             }
             else if (str.equals ("("))
             {
             // what do I do?
                build(1);
                validity = true;
             }
     
             else if(str.equals( ")"))
             {
                System.out.println("Not valid!");
     
             }
     
             else if (str.equals( "$"))
             {
                System.out.println("Not valid!");
     
             }
     
             else
                System.out.println("Invalid token");
          // should I add a break?
             return validity;
     
     
          }
     
          private boolean EPValidator(StringTokenizer st)
          {
             boolean validity = false;
             String str = st.nextToken();
             Integer n = 0;
             try
             {
                n = Integer.parseInt(str);
             }
     
                catch (NumberFormatException ime)
                {
     
                }
     
             if(str.equals( n.toString()))
             {
             // still not sure how to get it to be n
                System.out.println("Not valid!");
             }
     
             else if(str.equals("+"))
             {
             // what do I do?
                build(2);
                validity = true;
             }
     
             else if(str.equals( "*"))
             {
                System.out.println("Not valid!");
             }
             else if(str.equals( "("))
             {
                System.out.println("Not valid!");
             }
             else if(str.equals( ")"))
             {
             // what do I do?
                build(3);
                validity = true;
             }
             else if(str.equals( "$"))
             {
             // what do I do?
                build(3);
                validity = true;
             }
     
             else
                System.out.println("Invalid token");
          // should I add a break?
     
     
             return validity;
          }
     
          private boolean TValidator(StringTokenizer st)
          {
             boolean validity = false;
             String str = st.nextToken();
             Integer n = 0;
             try
             {
                n = Integer.parseInt(str);
             }
     
                catch (NumberFormatException ime)
                {
     
                }
     
             if(str.equals( n.toString()))
             {
                // what do I do?
                build(4);
                validity = true;
             }
             else if(str.equals( "+"))
             {
                System.out.println("Not valid!");
             }
             else if(str.equals( "*"))
             {
                System.out.println("Not valid!");
             }
     
             else if(str.equals( "("))
             {
                // what do I do?
                build(4);
                validity = true;
             }
     
             else if(str.equals( ")"))
             {
                System.out.println("Not valid!");
             }
             else if(str.equals("$"))
             {
                System.out.println("Not valid!");
             }
             else
                System.out.println("Invalid token");
             // should I add a break?
     
     
             return validity;
          }
     
          private boolean TPValidator(StringTokenizer st)
          {
             boolean validity = false;
             String str = st.nextToken();
             Integer n = 0;
             try
             {
                n = Integer.parseInt(str);
             }
     
                catch (NumberFormatException ime)
                {
     
                }
     
             if(str.equals( n.toString()))
             {
                System.out.println("Not valid!");
             }
             else if(str.equals( "+"))
             {
                // what do I do?
                build(6);
                validity = true;
             }
             else if(str.equals( "*"))
             {
                // what do I do?
                build(5);
                validity = true;
             }
             else if(str.equals( "("))
             {
                System.out.println("Not valid!");
             }
             else if(str.equals( ")"))
             {
                // what do I do?
                build(6);
                validity = true;
             }
             else if(str.equals( "$"))
             {
     
                // what do I do?
                build(6);
                validity = true;
             }
             else
                System.out.println("Invalid token");
             // should I add a break?
     
             return validity;
          }
     
          private boolean FValidator(StringTokenizer st)
          {
     
             boolean validity = false;
             String str = st.nextToken();
             Integer n = 0;
             try
             {
                n = Integer.parseInt(str);
             }
     
                catch (NumberFormatException ime)
                {
     
                }
     
             if(str.equals(n.toString()))
             {
                // delete token n; how I don't know
                build(8);
                validity = true;
             }
     
             else if(str.equals( "+"))
             {
                System.out.println("Not valid!");
             }
     
             else if(str.equals( "*"))
             {
                System.out.println("Not valid!");
             }
             else if(str.equals( "("))
             {
                // what do I do?
                build(7);
                validity = true;
             }
             else if(str.equals( ")"))
             {
                System.out.println("Not valid!");
             }
             else if(str.equals("$"))
             {
                System.out.println("Not valid!");
             }
             else
                System.out.println("Invalid token");
             // should I add a break?
     
     
             return validity;
          }
     
     
     
          public static void main(String[] args)
          {
             LL1Parser parser;
     
             try
             {
                parser = new LL1Parser(args[0] + " $");
             }
     
                catch(ArrayIndexOutOfBoundsException aioobe)
                {
                   System.out.println("args[0] doesn't exist!");
                   System.exit(0);
                }
     
     
     
     
          }
       }

    I used a command prompt like thing with

    java LL1Parser 123 which outputted the three lines below:

    123
    Not valid!
    false


    I ran it with
    java LL1Parser *
    and it outputted these three lines below:
    LL1Parser.class
    Invalid token
    false

    Why doesn't it ever recognize even valid tokens?

    i.e. starting with a plus should be invalid but the number 123 should be valid but it's not recognizing it as valid.

    It's always outputting false.

    I'm thinking for one that I should get rid of all those StringTokenizer parameters, or nearly all of them, and just edit the class variable tokenizer.

    Also, when I'm calling those pop() methods inside of the validator methods, is it removing the top of the stack twice or something?

    Also, it appears to be a problem with one of my validator methods, or that's where the error is showing up with the printlns. Which validator method I don't know so I need to show them all.

  7. #7
    Super Moderator Norm's Avatar
    Join Date
    May 2010
    Location
    Eastern Florida
    Posts
    21,051
    Thanks
    52
    Thanked 2,280 Times in 2,251 Posts

    Default Re: Confused on what tokenizer to use and how to use them.

    Did you read my post#4 about what kind of code to post for testing this one problem?

  8. #8
    Banned
    Join Date
    May 2010
    Location
    North Central Illinois
    Posts
    1,631
    My Mood
    Sleepy
    Thanks
    390
    Thanked 112 Times in 110 Posts

    Default Re: Confused on what tokenizer to use and how to use them.

    I believe it now is, or should be anyway, updating the token string to shrink.

    However, it seems to always be either always evaluating to invalid or false. I've updated it to change all the things except read to NOT use a StringTokenizer as parameter and to simply use the class variable tokenizer instead. After doing that, now it's always evaluating to true, even for bogus tokens like "-". I have no idea where it's going wrong though likely at the beginning part, i.e. in the EValidator(), though I can't be sure, hence the large amount of code. I think it unlikely that the build() method is the problem but can't narrow it much further.

    It's often not going into the validator methods it seems as some of my printlns are NEVER triggered.

    I'm thinking it's somehow always going to either true or false due to a glitch in one of my if, else if, else structures but which one I don't know, hence why I posted all of them.

    For instance, EValidator is never being entered according to my print lines.

    Perhaps it's my commandStack not being updated or something. Somehow I'm thinking it's something to do with the commandStack.

    ----Edit-------

    I found that the instructor said that

    1.) I could have more time
    2.) That I could do it another way and don't even need the stack at all

    So I'm going to revamp the class and then see if it works and if not post it here.
    Last edited by javapenguin; November 8th, 2011 at 02:41 PM.

  9. #9
    Banned
    Join Date
    May 2010
    Location
    North Central Illinois
    Posts
    1,631
    My Mood
    Sleepy
    Thanks
    390
    Thanked 112 Times in 110 Posts

    Default Re: Confused on what tokenizer to use and how to use them.

    Ok I've rewrote part of it now so I don't need the tokenizer at all. I made my own. Does it have any code flaws? It compiles but might have bad logic or cause null pointers. Or could be inefficient.
     private static class TokenHandler
    {
    private String str;
    private char nextchar;
     
    public static String getNextToken(String tokenStream)
    { 
    String numberString = "";
    int index = 0;
    char c = tokenStream.charAt(0);
    if ( c == '+')
    {
    return "+";
     
     
    }
     
    else if (c == '*')
    {
    return "*";
     
    }
     
    else if (c == '(')
    {
    return "(";
    }
     
    else if (c == ')')
    {
    return ")";
     
    }
     
    else if (c == '0' || c == '1' || c == '2' || c == '3' || c == '4'|| c == '5' || c == '6' || c == '7'|| c == '8' || c == '9')
    {
    char c2 = 'a';
    numberString = numberString + String.valueOf(c);
    if (tokenStream.length() == 1)
    {
    return numberString;
    }
    else
    {
     c2 = tokenStream.charAt(1);
    while (c2 == '0' || c2 == '1' || c2 == '2' || c2 == '3' || c2 == '4'|| c2 == '5' || c2 == '6' || c2 == '7'|| c2 == '8' || c2 == '9')
    {
     
    numberString = numberString  + getNextToken(tokenStream.substring(1));
     
    }
    }
    return numberString;
    }
     
    else if ( c == '$')
    {
    return "$";
    }
     
    else
    {
    return ("Bad String!");
    }
     
    }
     
    public static void deleteToken(String tokenStream)
    {
    String temp = getNextToken(tokenStream);
    int number = temp.length();
    String temp2 = tokenStream.substring(number);
    tokenStream = temp2;
     
    }
     
    }

  10. #10
    Super Moderator Norm's Avatar
    Join Date
    May 2010
    Location
    Eastern Florida
    Posts
    21,051
    Thanks
    52
    Thanked 2,280 Times in 2,251 Posts

    Default Re: Confused on what tokenizer to use and how to use them.

    Does the code now do what you want it to do?
    If not, show what it does and explain what it should do.

  11. #11
    Banned
    Join Date
    May 2010
    Location
    North Central Illinois
    Posts
    1,631
    My Mood
    Sleepy
    Thanks
    390
    Thanked 112 Times in 110 Posts

    Default Re: Confused on what tokenizer to use and how to use them.

    It shows that the while loop inside of getNextToken() is infinite if it's a number that's read in as the next char. I'm thinking it's how I'm phrasing it but cannot figure out what to do about it. Am going to see what happens if I change that while to an if.......

    Ok, that solved that part of the problem. Let's see how delete token goes.

    It didn't work but I tweaked it and now it works.

    Actually, I'd like deleteToken() to be a void if possible but don't know how to correctly change the token parameter so it now updates the string.
    Last edited by javapenguin; November 9th, 2011 at 03:01 PM.

  12. #12
    Banned
    Join Date
    May 2010
    Location
    North Central Illinois
    Posts
    1,631
    My Mood
    Sleepy
    Thanks
    390
    Thanked 112 Times in 110 Posts

    Default Re: Confused on what tokenizer to use and how to use them.

    Ok, I've updated it and it's now always returning false.

    It appears to be going into the read twice but never EValidator. Why would that be?

     import java.util.StringTokenizer;
       import java.io.*;
       import java.util.*;
     
     
       public class LL1Parser
       {
     
          private StringTokenizer tokenizer;
          private boolean valid;
          private Stack<String> commandStack;
          private String tokens;
     
         private static class TokenHandler
    {
    private String str;
    private char nextchar;
     
    public static String getNextToken(String tokenStream)
    { 
    String numberString = "";
    int index = 0;
    char c = tokenStream.charAt(0);
    if ( c == '+')
    {
    return "+";
     
     
    }
     
    else if (c == '*')
    {
    return "*";
     
    }
     
    else if (c == '(')
    {
    return "(";
    }
     
    else if (c == ')')
    {
    return ")";
     
    }
     
    else if (c == '0' || c == '1' || c == '2' || c == '3' || c == '4'|| c == '5' || c == '6' || c == '7'|| c == '8' || c == '9')
    {
    char c2 = 'a';
    numberString = numberString + String.valueOf(c);
    if (tokenStream.length() == 1)
    {
    return numberString;
    }
    else
    {
     c2 = tokenStream.charAt(1);
    if  (c2 == '0' || c2 == '1' || c2 == '2' || c2 == '3' || c2 == '4'|| c2 == '5' || c2 == '6' || c2 == '7'|| c2 == '8' || c2 == '9')
    {
     
    numberString = numberString  + getNextToken(tokenStream.substring(1));
     
    }
    }
    return numberString;
    }
     
    else if ( c == '$')
    {
    return "$";
    }
     
    else
    {
    return ("Bad String!");
    }
     
    }
     
    public static String deleteToken(String tokenStream)
    {
    String temp = getNextToken(tokenStream);
    int number = temp.length();
    String temp2 = tokenStream.substring(number);
    //tokenStream = temp2;
    return temp2;
     
    }
     
    }
     
          public LL1Parser(String input)
          {
     
            // tokenizer = new StringTokenizer(input);
           //  commandStack = new Stack<String>(); // creates an empty command stack
             setString(input);
           //  commandStack.push("T");
            // commandStack.push("EP");
                            boolean b = isValid();
             System.out.println(b);
     
     
          }
     
          private boolean read()
          {
    System.out.println("Came to read");
    boolean validity = true;
     
    boolean temp = TValidator();
    if(!temp)
    return false;
    boolean temp2 = EPValidator();
    if(!temp2)
    return false;
    return validity;
          }
     
          private void setString(String tokens)
          {
             this.tokens = tokens;
          }
     
          private String getString()
          {
             return tokens;
          }
     
     
     
     
          public boolean isValid()
          {
     
                    // System.out.println(tokenizer.nextToken());
     
          // needs to check to see if String is null
          // needs a base case anyway
          boolean validity = true;
             while (getString() != null && validity == true )
             {
              // validity = read(st);
                validity = read();
    if(validity == false)
    return false;
              //  else
     
                // do something to remove some tokens
                // create a StringTokenizer variable newSt based on this new value with the token removed
     
                 //  isValid(newSt);
     
     
     
             }
             return validity;
     
          }
     
          private boolean EValidator()
          {
     
    System.out.println("EValidator method is being used.");
     
     
             boolean validity = false;
             String str = TokenHandler.getNextToken(getString());
             Integer n = 0;
             try
             {
                n = Integer.parseInt(str);
    if (n >32676)
    n = 32676;
    else n = n;
             }
     
                catch (NumberFormatException ime)
                {
     
                }
     
     
     
     
             if (str.equals(n.toString()))
             {
    boolean temp = TValidator();
    if (!temp)
    return false;
    boolean temp2 = EPValidator();
    if(!temp2)
    return false;
             // how do I do this?
               // build(1);
                validity = true;
             }
             else if(str.equals("+"))
             {
                System.out.println("Not valid!");
                return false;
             }
             else if(str.equals( "*"))
             {
                System.out.println("Not valid!");
                return false;
             }
             else if (str.equals ("("))
             {
     
    boolean temp = TValidator();
    if (!temp)
    return false;
    boolean temp2 = EPValidator();
    if (!temp2)
    return false;
             // what do I do?
             //   build(1);
                validity = true;
             }
     
             else if(str.equals( ")"))
             {
                System.out.println("Not valid!");
                return false;
     
             }
     
             else if (str.equals( "$"))
             {
                System.out.println("Not valid!");
                return false;
             }
     
             else
    {
                System.out.println("Invalid token");
                return false;
    }
          // should I add a break?
             return validity;
     
     
          }
     
          private boolean EPValidator()
          {
             boolean validity = false;
             String str = TokenHandler.getNextToken(getString());
             Integer n = 0;
             try
             {
                n = Integer.parseInt(str);
    if (n >32676)
    n = 32676;
    else n = n;
             }
     
                catch (NumberFormatException ime)
                {
     
                }
     
             if(str.equals( n.toString()))
             {
             // still not sure how to get it to be n
                System.out.println("Not valid!");
               return false;
             }
     
             else if(str.equals("+"))
             {
             // remove token + somehow
             setString(TokenHandler.deleteToken(getString()));
           boolean temp =  TValidator();
    if (!temp)
    return false;
           boolean temp2 =   EPValidator();
    if (!temp2)
    return false;
     
             // what do I do?
            //    build(2);
                validity = true;
             }
     
             else if(str.equals( "*"))
             {
                System.out.println("Not valid!");
                 return false;
             }
             else if(str.equals( "("))
             {
                System.out.println("Not valid!");
                return false;
             }
             else if(str.equals( ")"))
             {
             // what do I do?
           //     build(3);
                validity = true;
             }
             else if(str.equals( "$"))
             {
             // what do I do?
             //   build(3);
                validity = true;
             }
     
             else
    {
                System.out.println("Invalid token");
    return false;
    }
          // should I add a break?
     
     
             return validity;
          }
     
          private boolean TValidator()
          {
             boolean validity = false;
             String str = TokenHandler.getNextToken(getString());
             Integer n = 0;
             try
             {
                n = Integer.parseInt(str);
             }
     
                catch (NumberFormatException ime)
                {
     
                }
     
             if(str.equals( n.toString()))
             {
              boolean temp = FValidator();
    if(!temp)
    return false;
             boolean temp2 = TPValidator();
    if(!temp2)
    return false;
                // what do I do?
              //  build(4);
                validity = true;
             }
             else if(str.equals( "+"))
             {
                System.out.println("Not valid!");
                return false;
             }
             else if(str.equals( "*"))
             {
                System.out.println("Not valid!");
                return false;
             }
     
             else if(str.equals( "("))
             {
              boolean temp = FValidator();
    if(!temp)
    return false;
          boolean temp2 =     TPValidator();
    if (!temp2)
    return false;
                // what do I do?
              //  build(4);
                validity = true;
             }
     
             else if(str.equals( ")"))
             {
                System.out.println("Not valid!");
                return false;
             }
             else if(str.equals("$"))
             {
                System.out.println("Not valid!");
                return false;
             }
             else
    {
                System.out.println("Invalid token");
    return false;
    }
             // should I add a break?
     
     
             return validity;
          }
     
          private boolean TPValidator()
          {
             boolean validity = false;
             String str = TokenHandler.getNextToken(getString());
             Integer n = 0;
             try
             {
                n = Integer.parseInt(str);
    if (n >32676)
    n = 32676;
    else n = n;
             }
     
                catch (NumberFormatException ime)
                {
     
                }
     
             if(str.equals( n.toString()))
             {
                System.out.println("Not valid!");
                return false;
             }
             else if(str.equals( "+"))
             {
                // what do I do?
           //   setString(TokenHandler.deleteToken(getString()));
              //  build(6);
                validity = true;
             }
             else if(str.equals( "*"))
             {
               // somehow remove * from token list
    setString(TokenHandler.deleteToken(getString()));
           boolean temp =    FValidator();
    if (!temp)
    return false;
            boolean temp2 =   TPValidator();
    if (!temp2)
    return false;
                // what do I do?
                //build(5);
                validity = true;
             }
             else if(str.equals( "("))
             {
                System.out.println("Not valid!");
                return false;
             }
             else if(str.equals( ")"))
             {
                // what do I do?
             //   build(6);
                validity = true;
             }
             else if(str.equals( "$"))
             {
     
                // what do I do?
              //  build(6);
                validity = true;
             }
             else
    {
                System.out.println("Invalid token");
                return false;
    }
             // should I add a break?
     
             return validity;
          }
     
          private boolean FValidator()
          {
     
             boolean validity = false;
             String str = TokenHandler.getNextToken(getString());
             Integer n = 0;
             try
             {
                n = Integer.parseInt(str);
    if (n >32676)
    n = 32676;
    else n = n;
             }
     
                catch (NumberFormatException ime)
                {
     
                }
     
             if(str.equals(n.toString()))
             {
                setString(TokenHandler.deleteToken(getString()));
                // delete token n; how I don't know
              //  build(8);
                validity = true;
             }
     
             else if(str.equals( "+"))
             {
                System.out.println("Not valid!");
                return false;
             }
     
             else if(str.equals( "*"))
             {
                System.out.println("Not valid!");
                return false;
             }
             else if(str.equals( "("))
             {
    setString(TokenHandler.deleteToken(getString()));
    boolean temp = EValidator();
    if(!temp)
    return false;
    if (TokenHandler.getNextToken(getString()) != ")")
    return false;
    else 
    setString(TokenHandler.deleteToken(getString()));
                // what do I do?
               // build(7);
                validity = true;
             }
             else if(str.equals( ")"))
             {
                System.out.println("Not valid!");
                return false;
             }
             else if(str.equals("$"))
             {
                System.out.println("Not valid!");
                return false;
             }
             else
    {
                System.out.println("Invalid token");
    return false;
    }
             // should I add a break?
     
     
             return validity;
          }
     
     
     
          public static void main(String[] args)
          {
             LL1Parser parser;
     
             try
             {
                parser = new LL1Parser(args[0] + "$");
             }
     
                catch(ArrayIndexOutOfBoundsException aioobe)
                {
                   System.out.println("args[0] doesn't exist!");
                   System.exit(0);
                }
     
     
     
     
          }
       }

    I added a println into the isValid() method and told it to print out validity.

    The ultimate validity is printed at the end. That's where that second false is coming from below.

    It gave this for this input:
     

    pradcoc@discovery:~/ITK 327/LL1 Parser> java LL1Parser 123
    Came to read
    true
    Came to read
    Not valid!
    false
    false




    123 should be valid.

    However, it's not recognizing it.

    I've considered the possibility that in the command prompt, it wants the args passed as

    "123" but have tried that and still had the same problem.
    Last edited by javapenguin; November 9th, 2011 at 04:17 PM.

  13. #13
    Super Moderator Norm's Avatar
    Join Date
    May 2010
    Location
    Eastern Florida
    Posts
    21,051
    Thanks
    52
    Thanked 2,280 Times in 2,251 Posts

    Default Re: Confused on what tokenizer to use and how to use them.

    Can you change your code for easier testing?
    It should compile and execute and show the problem. Nothing more. No args nothing extra.
    Include some debug printlns to show what the code does and what you want it to do.

  14. #14
    Banned
    Join Date
    May 2010
    Location
    North Central Illinois
    Posts
    1,631
    My Mood
    Sleepy
    Thanks
    390
    Thanked 112 Times in 110 Posts

    Default Re: Confused on what tokenizer to use and how to use them.

    I did find an error where I had been setting validity to true, regardless or whether or not it was true, at the beginning of every read method call and, I think, have fixed that, but the problem is still there

     import java.util.StringTokenizer;
       import java.io.*;
       import java.util.*;
     
     
       public class LL1Parser
       {
     
          private StringTokenizer tokenizer;
          private boolean validity2;
          private Stack<String> commandStack;
          private String tokens;
     
         private static class TokenHandler
    {
    private String str;
    private char nextchar;
     
    public static String getNextToken(String tokenStream)
    { 
    String numberString = "";
    int index = 0;
    char c = tokenStream.charAt(0);
    if ( c == '+')
    {
    return "+";
     
     
    }
     
    else if (c == '*')
    {
    return "*";
     
    }
     
    else if (c == '(')
    {
    return "(";
    }
     
    else if (c == ')')
    {
    return ")";
     
    }
     
    else if (c == '0' || c == '1' || c == '2' || c == '3' || c == '4'|| c == '5' || c == '6' || c == '7'|| c == '8' || c == '9')
    {
    char c2 = 'a';
    numberString = numberString + String.valueOf(c);
    if (tokenStream.length() == 1)
    {
    return numberString;
    }
    else
    {
     c2 = tokenStream.charAt(1);
    if  (c2 == '0' || c2 == '1' || c2 == '2' || c2 == '3' || c2 == '4'|| c2 == '5' || c2 == '6' || c2 == '7'|| c2 == '8' || c2 == '9')
    {
     
    numberString = numberString  + getNextToken(tokenStream.substring(1));
     
    }
    }
    return numberString;
    }
     
    else if ( c == '$')
    {
    return "$";
    }
     
    else
    {
    return ("Bad String!");
    }
     
    }
     
    public static String deleteToken(String tokenStream)
    {
    String temp = getNextToken(tokenStream);
    int number = temp.length();
    String temp2 = tokenStream.substring(number);
    //tokenStream = temp2;
    return temp2;
     
    }
     
    }
     
          public LL1Parser(String input)
          {
     
            // tokenizer = new StringTokenizer(input);
           //  commandStack = new Stack<String>(); // creates an empty command stack
             setString(input);
    setValidity(true);
           //  commandStack.push("T");
            // commandStack.push("EP");
                            boolean b = isValid();
             System.out.println(b);
     
     
          }
     
    public void setValidity(boolean validity2)
    {
    this.validity2 = validity2;
    }
     
    public boolean getValidity()
    {
    return validity2;
    }
          private boolean read()
          {
    System.out.println("Came to read");
    boolean validity = true;
     
    boolean temp = TValidator();
    if(!temp)
    return false;
    System.out.println("validitiy is " +validity + " after TValidator");
    boolean temp2 = EPValidator();
    if(!temp2)
    return false;
    System.out.println("validity is " + validity + " after EPValidator" );
    return validity;
          }
     
          private void setString(String tokens)
          {
             this.tokens = tokens;
          }
     
          private String getString()
          {
             return tokens;
          }
     
     
     
     
          public boolean isValid()
          {
     
                    // System.out.println(tokenizer.nextToken());
     
          // needs to check to see if String is null
          // needs a base case anyway
          boolean validity = getValidity();
             while (getString() != null && validity == true )
             {
              // validity = read(st);
                validity = read();
    setValidity(validity);
    System.out.println("Validity is " + validity + " at this point in read");
    //if(validity == false)
    //return false;
              //  else
     
                // do something to remove some tokens
                // create a StringTokenizer variable newSt based on this new value with the token removed
     
                 //  isValid(newSt);
     
     
     
             }
             return validity;
     
          }
     
          private boolean EValidator()
          {
     
    System.out.println("EValidator method is being used.");
     
     
             boolean validity = false;
             String str = TokenHandler.getNextToken(getString());
             Integer n = 0;
             try
             {
                n = Integer.parseInt(str);
    if (n >32676)
    n = 32676;
    else n = n;
             }
     
                catch (NumberFormatException ime)
                {
     
                }
     
     
     
     
             if (str.equals(n.toString()))
             {
    boolean temp = TValidator();
    if (!temp)
    return false;
    boolean temp2 = EPValidator();
    if(!temp2)
    return false;
             // how do I do this?
               // build(1);
                validity = true;
             }
             else if(str.equals("+"))
             {
                System.out.println("Not valid!");
                return false;
             }
             else if(str.equals( "*"))
             {
                System.out.println("Not valid!");
                return false;
             }
             else if (str.equals ("("))
             {
     
    boolean temp = TValidator();
    if (!temp)
    return false;
    boolean temp2 = EPValidator();
    if (!temp2)
    return false;
             // what do I do?
             //   build(1);
                validity = true;
             }
     
             else if(str.equals( ")"))
             {
                System.out.println("Not valid!");
                return false;
     
             }
     
             else if (str.equals( "$"))
             {
                System.out.println("Not valid!");
                return false;
             }
     
             else
    {
                System.out.println("Invalid token");
                return false;
    }
          // should I add a break?
             return validity;
     
     
          }
     
          private boolean EPValidator()
          {
             boolean validity = false;
             String str = TokenHandler.getNextToken(getString());
             Integer n = 0;
             try
             {
                n = Integer.parseInt(str);
    if (n >32676)
    n = 32676;
    else n = n;
             }
     
                catch (NumberFormatException ime)
                {
     
                }
     
             if(str.equals( n.toString()))
             {
             // still not sure how to get it to be n
                System.out.println("Not valid!");
               return false;
             }
     
             else if(str.equals("+"))
             {
             // remove token + somehow
             setString(TokenHandler.deleteToken(getString()));
           boolean temp =  TValidator();
    if (!temp)
    return false;
           boolean temp2 =   EPValidator();
    if (!temp2)
    return false;
     
             // what do I do?
            //    build(2);
                validity = true;
             }
     
             else if(str.equals( "*"))
             {
                System.out.println("Not valid!");
                 return false;
             }
             else if(str.equals( "("))
             {
                System.out.println("Not valid!");
                return false;
             }
             else if(str.equals( ")"))
             {
             // what do I do?
           //     build(3);
                validity = true;
             }
             else if(str.equals( "$"))
             {
             // what do I do?
             //   build(3);
                validity = true;
             }
     
             else
    {
                System.out.println("Invalid token");
    return false;
    }
          // should I add a break?
     
     
             return validity;
          }
     
          private boolean TValidator()
          {
             boolean validity = false;
             String str = TokenHandler.getNextToken(getString());
             Integer n = 0;
             try
             {
                n = Integer.parseInt(str);
             }
     
                catch (NumberFormatException ime)
                {
     
                }
     
             if(str.equals( n.toString()))
             {
              boolean temp = FValidator();
    if(!temp)
    return false;
             boolean temp2 = TPValidator();
    if(!temp2)
    return false;
                // what do I do?
              //  build(4);
                validity = true;
             }
             else if(str.equals( "+"))
             {
                System.out.println("Not valid!");
                return false;
             }
             else if(str.equals( "*"))
             {
                System.out.println("Not valid!");
                return false;
             }
     
             else if(str.equals( "("))
             {
              boolean temp = FValidator();
    if(!temp)
    return false;
          boolean temp2 =     TPValidator();
    if (!temp2)
    return false;
                // what do I do?
              //  build(4);
                validity = true;
             }
     
             else if(str.equals( ")"))
             {
                System.out.println("Not valid!");
                return false;
             }
             else if(str.equals("$"))
             {
                System.out.println("Not valid!");
                return false;
             }
             else
    {
                System.out.println("Invalid token");
    return false;
    }
             // should I add a break?
     
     
             return validity;
          }
     
          private boolean TPValidator()
          {
             boolean validity = false;
             String str = TokenHandler.getNextToken(getString());
             Integer n = 0;
             try
             {
                n = Integer.parseInt(str);
    if (n >32676)
    n = 32676;
    else n = n;
             }
     
                catch (NumberFormatException ime)
                {
     
                }
     
             if(str.equals( n.toString()))
             {
                System.out.println("Not valid!");
                return false;
             }
             else if(str.equals( "+"))
             {
                // what do I do?
           //   setString(TokenHandler.deleteToken(getString()));
              //  build(6);
                validity = true;
             }
             else if(str.equals( "*"))
             {
               // somehow remove * from token list
    setString(TokenHandler.deleteToken(getString()));
           boolean temp =    FValidator();
    if (!temp)
    return false;
            boolean temp2 =   TPValidator();
    if (!temp2)
    return false;
                // what do I do?
                //build(5);
                validity = true;
             }
             else if(str.equals( "("))
             {
                System.out.println("Not valid!");
                return false;
             }
             else if(str.equals( ")"))
             {
                // what do I do?
             //   build(6);
                validity = true;
             }
             else if(str.equals( "$"))
             {
     
                // what do I do?
              //  build(6);
                validity = true;
             }
             else
    {
                System.out.println("Invalid token");
                return false;
    }
             // should I add a break?
     
             return validity;
          }
     
          private boolean FValidator()
          {
     
             boolean validity = false;
             String str = TokenHandler.getNextToken(getString());
             Integer n = 0;
             try
             {
                n = Integer.parseInt(str);
    if (n >32676)
    n = 32676;
    else n = n;
             }
     
                catch (NumberFormatException ime)
                {
     
                }
     
             if(str.equals(n.toString()))
             {
                setString(TokenHandler.deleteToken(getString()));
                // delete token n; how I don't know
              //  build(8);
                validity = true;
             }
     
             else if(str.equals( "+"))
             {
                System.out.println("Not valid!");
                return false;
             }
     
             else if(str.equals( "*"))
             {
                System.out.println("Not valid!");
                return false;
             }
             else if(str.equals( "("))
             {
    setString(TokenHandler.deleteToken(getString()));
    boolean temp = EValidator();
    if(!temp)
    return false;
    if (TokenHandler.getNextToken(getString()) != ")")
    return false;
    else 
    setString(TokenHandler.deleteToken(getString()));
                // what do I do?
               // build(7);
                validity = true;
             }
             else if(str.equals( ")"))
             {
                System.out.println("Not valid!");
                return false;
             }
             else if(str.equals("$"))
             {
                System.out.println("Not valid!");
                return false;
             }
             else
    {
                System.out.println("Invalid token");
    return false;
    }
             // should I add a break?
     
     
             return validity;
          }
     
     
     
          public static void main(String[] args)
          {
             LL1Parser parser;
     
             try
             {
               // parser = new LL1Parser(args[0] + "$");
              parser = new LL1Parser("123");
             }
     
                catch(ArrayIndexOutOfBoundsException aioobe)
                {
                   System.out.println("args[0] doesn't exist!");
                   System.exit(0);
                }
     
     
     
     
          }
       }

     

    pradcoc@discovery:~/ITK 327/LL1 Parser> java LL1Parser 123
    Came to read
    validitiy is true after TValidator
    validity is true after EPValidator
    Validity is true at this point in read
    Came to read
    Not valid!
    Validity is false at this point in read
    false




    What it should be:
     

    Came to read
    validitiy is true after TValidator
    validity is true after EPValidator
    Validity is true at this point in read
    Came to read
    Validity is true at this point in read
    // some more calls possibly that always return true
    true




    Also, I tried it with
    "123" just to see if that was the problem and it gave me this output, which is weird, considering both approaches shouldn't be correct. If it doesn't need a token, then " should be considered an invalid token, which is a different output from "Not valid!". But it's not.

     

    pradcoc@discovery:~/ITK 327/LL1 Parser> java LL1Parser "123"
    Came to read
    validitiy is true after TValidator
    validity is true after EPValidator
    Validity is true at this point in read
    Came to read
    Not valid!
    Validity is false at this point in read
    false




    I tried it by changing it to have

    "123" as the parameter to the parser object in the main method and I got this error

     

    pradcoc@discovery:~/ITK 327/LL1 Parser> java LL1Parser
    Came to read
    Exception in thread "main" java.lang.StringIndexOutOfBoundsException: String index out of range: 0
    at java.lang.String.charAt(String.java:686)
    at LL1Parser$TokenHandler.getNextToken(LL1Parser.java :23)
    at LL1Parser.TPValidator(LL1Parser.java:411)
    at LL1Parser.TValidator(LL1Parser.java:356)
    at LL1Parser.read(LL1Parser.java:122)
    at LL1Parser.isValid(LL1Parser.java:157)
    at LL1Parser.<init>(LL1Parser.java:102)
    at LL1Parser.main(LL1Parser.java:562)




    Did find another problem at that line 23.

    It shouldn't be tokenStream.length as it won't work in that case for numbers like 123 or indeed even 12.

    But how do I fix that line?
    Last edited by javapenguin; November 9th, 2011 at 04:44 PM.

  15. #15
    Banned
    Join Date
    May 2010
    Location
    North Central Illinois
    Posts
    1,631
    My Mood
    Sleepy
    Thanks
    390
    Thanked 112 Times in 110 Posts

    Default Re: Confused on what tokenizer to use and how to use them.

    Ok, I've tried fixing some things and now it will always say true and will go into an infinite loop and I don't know why!

     import java.util.StringTokenizer;
       import java.io.*;
       import java.util.*;
     
     
       public class LL1Parser
       {
     
          private StringTokenizer tokenizer;
          private boolean validity2;
          private Stack<String> commandStack;
          private String tokens;
          private int originalLength;
     
         private static class TokenHandler
    {
    private String str;
    private char nextchar;
     
    public static String getNextToken(String tokenStream)
    { 
    if (tokenStream == null)
    return null;
    String numberString = "";
    int index = 0;
    char c = tokenStream.charAt(0);
    if ( c == '+')
    {
    return "+";
     
     
    }
     
    else if (c == '*')
    {
    return "*";
     
    }
     
    else if (c == '(')
    {
    return "(";
    }
     
    else if (c == ')')
    {
    return ")";
     
    }
     
    else if (c == '0' || c == '1' || c == '2' || c == '3' || c == '4'|| c == '5' || c == '6' || c == '7'|| c == '8' || c == '9')
    {
    char c2 = 'a';
    numberString = numberString + String.valueOf(c);
    if (tokenStream.length() == 1)
    {
    return numberString;
    }
    else
    {
     c2 = tokenStream.charAt(1);
    if  (c2 == '0' || c2 == '1' || c2 == '2' || c2 == '3' || c2 == '4'|| c2 == '5' || c2 == '6' || c2 == '7'|| c2 == '8' || c2 == '9')
    {
     
    numberString = numberString  + getNextToken(tokenStream.substring(1));
     
    }
    }
    return numberString;
    }
     
    else if ( c == '$')
    {
    return "$";
    }
     
    else
    {
    return ("Bad String!");
    }
     
    }
     
    public static String deleteToken(String tokenStream)
    {
    String temp = getNextToken(tokenStream);
    int number = temp.length();
    String temp2 = tokenStream.substring(number);
    //tokenStream = temp2;
    return temp2;
     
    }
     
    }
     
          public LL1Parser(String input)
          {
     
            // tokenizer = new StringTokenizer(input);
           //  commandStack = new Stack<String>(); // creates an empty command stack
             setString(input);
    setOriginalLength(input.length());
    setValidity(true);
           //  commandStack.push("T");
            // commandStack.push("EP");
                            boolean b = isValid();
             System.out.println(b);
     
     
          }
     
    public void setOriginalLength(int originalLength)
    {
    this.originalLength = originalLength;
    }
     
    public int getOriginalLength()
    {
    return originalLength;
    }
     
     
    public void setValidity(boolean validity2)
    {
    this.validity2 = validity2;
    }
     
    public boolean getValidity()
    {
    return validity2;
    }
          private boolean read()
          {
    System.out.println("Came to read");
    boolean validity = true;
     
    boolean temp = TValidator();
    if(!temp)
    return false;
    System.out.println("validitiy is " +validity + " after TValidator");
    boolean temp2 = EPValidator();
    if(!temp2)
    return false;
    System.out.println("validity is " + validity + " after EPValidator" );
    return validity;
          }
     
          private void setString(String tokens)
          {
             this.tokens = tokens;
          }
     
          private String getString()
          {
             return tokens;
          }
     
     
     
     
          public boolean isValid()
          {
     
                    // System.out.println(tokenizer.nextToken());
     
          // needs to check to see if String is null
          // needs a base case anyway
          boolean validity = getValidity();
             while (getString() != null && validity == true )
             {
              // validity = read(st);
                validity = read();
    setValidity(validity);
    System.out.println("Validity is " + validity + " at this point in read");
    //if(validity == false)
    //return false;
              //  else
     
                // do something to remove some tokens
                // create a StringTokenizer variable newSt based on this new value with the token removed
     
                 //  isValid(newSt);
     
     
     
             }
             return validity;
     
          }
     
          private boolean EValidator()
          {
     
    System.out.println("EValidator method is being used.");
     
     
             boolean validity = false;
    if (TokenHandler.getNextToken(getString()) != null && getOriginalLength() != 0)
    {
    return true;
    }
     
    else if (TokenHandler.getNextToken(getString()) != null && getOriginalLength() == 0)
    {
    return false;
    }
     
    else
    {
             String str = TokenHandler.getNextToken(getString());
     
             Integer n = 0;
             try
             {
                n = Integer.parseInt(str);
    if (n >32676)
    n = 32676;
    else n = n;
             }
     
                catch (NumberFormatException ime)
                {
     
                }
     
     
     
     
             if (str.equals(n.toString()))
             {
    boolean temp = TValidator();
    if (!temp)
    return false;
    boolean temp2 = EPValidator();
    if(!temp2)
    return false;
             // how do I do this?
               // build(1);
                validity = true;
             }
             else if(str.equals("+"))
             {
                System.out.println("Not valid!");
                return false;
             }
             else if(str.equals( "*"))
             {
                System.out.println("Not valid!");
                return false;
             }
             else if (str.equals ("("))
             {
     
    boolean temp = TValidator();
    if (!temp)
    return false;
    boolean temp2 = EPValidator();
    if (!temp2)
    return false;
             // what do I do?
             //   build(1);
                validity = true;
             }
     
             else if(str.equals( ")"))
             {
                System.out.println("Not valid!");
                return false;
     
             }
     
             else if (str.equals( "$"))
             {
                System.out.println("Not valid!");
                return false;
             }
     
             else
    {
                System.out.println("Invalid token");
                return false;
    }
    }
          // should I add a break?
             return validity;
     
     
          }
     
          private boolean EPValidator()
          {
             boolean validity = false;
     
    if (TokenHandler.getNextToken(getString()) != null && getOriginalLength() != 0)
    {
    return true;
    }
     
    else if (TokenHandler.getNextToken(getString()) != null && getOriginalLength() == 0)
    {
    return false;
    }
    else
    {
             String str = TokenHandler.getNextToken(getString());
             Integer n = 0;
             try
             {
                n = Integer.parseInt(str);
    if (n >32676)
    n = 32676;
    else n = n;
             }
     
                catch (NumberFormatException ime)
                {
     
                }
     
             if(str.equals( n.toString()))
             {
             // still not sure how to get it to be n
                System.out.println("Not valid!");
               return false;
             }
     
             else if(str.equals("+"))
             {
             // remove token + somehow
             setString(TokenHandler.deleteToken(getString()));
           boolean temp =  TValidator();
    if (!temp)
    return false;
           boolean temp2 =   EPValidator();
    if (!temp2)
    return false;
     
             // what do I do?
            //    build(2);
                validity = true;
             }
     
             else if(str.equals( "*"))
             {
                System.out.println("Not valid!");
                 return false;
             }
             else if(str.equals( "("))
             {
                System.out.println("Not valid!");
                return false;
             }
             else if(str.equals( ")"))
             {
             // what do I do?
           //     build(3);
                validity = true;
             }
             else if(str.equals( "$"))
             {
             // what do I do?
             //   build(3);
                validity = true;
             }
     
             else
    {
                System.out.println("Invalid token");
    return false;
    }
    }
          // should I add a break?
     
     
             return validity;
          }
     
          private boolean TValidator()
          {
             boolean validity = false;
    if (TokenHandler.getNextToken(getString()) != null && getOriginalLength() != 0)
    {
    return true;
    }
     
    else if (TokenHandler.getNextToken(getString()) != null && getOriginalLength() == 0)
    {
    return false;
    }
    else
    {
             String str = TokenHandler.getNextToken(getString());
             Integer n = 0;
             try
             {
                n = Integer.parseInt(str);
             }
     
                catch (NumberFormatException ime)
                {
     
                }
     
             if(str.equals( n.toString()))
             {
              boolean temp = FValidator();
    if(!temp)
    return false;
             boolean temp2 = TPValidator();
    if(!temp2)
    return false;
                // what do I do?
              //  build(4);
                validity = true;
             }
             else if(str.equals( "+"))
             {
                System.out.println("Not valid!");
                return false;
             }
             else if(str.equals( "*"))
             {
                System.out.println("Not valid!");
                return false;
             }
     
             else if(str.equals( "("))
             {
              boolean temp = FValidator();
    if(!temp)
    return false;
          boolean temp2 =     TPValidator();
    if (!temp2)
    return false;
                // what do I do?
              //  build(4);
                validity = true;
             }
     
             else if(str.equals( ")"))
             {
                System.out.println("Not valid!");
                return false;
             }
             else if(str.equals("$"))
             {
                System.out.println("Not valid!");
                return false;
             }
             else
    {
                System.out.println("Invalid token");
    return false;
    }
    }
             // should I add a break?
     
     
             return validity;
          }
     
          private boolean TPValidator()
          {
             boolean validity = false;
    if (TokenHandler.getNextToken(getString()) != null && getOriginalLength() != 0)
    {
    return true;
    }
     
    else if (TokenHandler.getNextToken(getString()) != null && getOriginalLength() == 0)
    {
    return false;
    }
     
    else
    {
             String str = TokenHandler.getNextToken(getString());
             Integer n = 0;
             try
             {
                n = Integer.parseInt(str);
    if (n >32676)
    n = 32676;
    else n = n;
             }
     
                catch (NumberFormatException ime)
                {
     
                }
     
             if(str.equals( n.toString()))
             {
                System.out.println("Not valid!");
                return false;
             }
             else if(str.equals( "+"))
             {
                // what do I do?
           //   setString(TokenHandler.deleteToken(getString()));
              //  build(6);
                validity = true;
             }
             else if(str.equals( "*"))
             {
               // somehow remove * from token list
    setString(TokenHandler.deleteToken(getString()));
           boolean temp =    FValidator();
    if (!temp)
    return false;
            boolean temp2 =   TPValidator();
    if (!temp2)
    return false;
                // what do I do?
                //build(5);
                validity = true;
             }
             else if(str.equals( "("))
             {
                System.out.println("Not valid!");
                return false;
             }
             else if(str.equals( ")"))
             {
                // what do I do?
             //   build(6);
                validity = true;
             }
             else if(str.equals( "$"))
             {
     
                // what do I do?
              //  build(6);
                validity = true;
             }
             else
    {
                System.out.println("Invalid token");
                return false;
    }
             // should I add a break?
    }
             return validity;
          }
     
          private boolean FValidator()
          {
     
             boolean validity = false;
    if (TokenHandler.getNextToken(getString()) != null && getOriginalLength() != 0)
    {
    return true;
    }
     
    else if (TokenHandler.getNextToken(getString()) != null && getOriginalLength() == 0)
    {
    return false;
    }
    else
    {
             String str = TokenHandler.getNextToken(getString());
             Integer n = 0;
             try
             {
                n = Integer.parseInt(str);
    if (n >32676)
    n = 32676;
    else n = n;
             }
     
                catch (NumberFormatException ime)
                {
     
                }
     
             if(str.equals(n.toString()))
             {
                setString(TokenHandler.deleteToken(getString()));
                // delete token n; how I don't know
              //  build(8);
                validity = true;
             }
     
             else if(str.equals( "+"))
             {
                System.out.println("Not valid!");
                return false;
             }
     
             else if(str.equals( "*"))
             {
                System.out.println("Not valid!");
                return false;
             }
             else if(str.equals( "("))
             {
    setString(TokenHandler.deleteToken(getString()));
    boolean temp = EValidator();
    if(!temp)
    return false;
    if (TokenHandler.getNextToken(getString()) != ")")
    return false;
    else 
    setString(TokenHandler.deleteToken(getString()));
                // what do I do?
               // build(7);
                validity = true;
             }
             else if(str.equals( ")"))
             {
                System.out.println("Not valid!");
                return false;
             }
             else if(str.equals("$"))
             {
                System.out.println("Not valid!");
                return false;
             }
             else
    {
                System.out.println("Invalid token");
    return false;
    }
    }
             // should I add a break?
     
     
             return validity;
          }
     
     
     
          public static void main(String[] args)
          {
             LL1Parser parser;
     
             try
             {
               // parser = new LL1Parser(args[0] + "$");
                  parser = new LL1Parser("+123");
             }
     
                catch(ArrayIndexOutOfBoundsException aioobe)
                {
                   System.out.println("args[0] doesn't exist!");
                   System.exit(0);
                }
     
     
     
     
          }
       }

    Also, now for this type of syntax the answer should be false anyway.

    I changed it from 123 to +123 to see if it at least was getting the validity right but it's not getting it right.



    Is this part of code the problem?

    if (TokenHandler.getNextToken(getString()) != null && getOriginalLength() != 0)
    {
    return true;
    }
     
    else if (TokenHandler.getNextToken(getString()) != null && getOriginalLength() == 0)
    {
    return false;
    }
    Last edited by javapenguin; November 9th, 2011 at 05:37 PM.

  16. #16
    Super Moderator Norm's Avatar
    Join Date
    May 2010
    Location
    Eastern Florida
    Posts
    21,051
    Thanks
    52
    Thanked 2,280 Times in 2,251 Posts

    Default Re: Confused on what tokenizer to use and how to use them.

    Is this part of code the problem?
    Your code has no comments describing your logic so its very hard to determine if the code is doing what you want it to do.

  17. #17
    Banned
    Join Date
    May 2010
    Location
    North Central Illinois
    Posts
    1,631
    My Mood
    Sleepy
    Thanks
    390
    Thanked 112 Times in 110 Posts

    Default Re: Confused on what tokenizer to use and how to use them.

    Well, considering what it IS doing at the moment, right now I'd like it NOT to loop infinitely.

    Here's how it should go:

    E():
    switch(token)
    {
    case n: T(); E'();
    build E -> TE';
    break;
    case +:
    error;
    break;
    case *:
    error;
    break;
    case (:
    T(); E'(); 
    build E ->TE';
    break;
    case ):
    error;
    break;
    case $:
    error;
    break;
     
    }
     
    E'():
    Switch(token)
    {
    case n:
    error;
    break;
    case +:
    get+);T(); E'();
    case *;
    error;
    break;
    case (:
    error;
    break;
    case ):
    build E' ->lambda;
    break;
    case $:
    build E' -> lambda;
    break;
    }
     
    T():
    Switch(token)
    {
    case n: 
    build T->FT';
    break;
    case +:
    error;
    break;
    case *:
    error;
    break;
    case (:
    build T->FT';
    break;
    case (:
    error;
    break;
    case $:
    error;
    break;
    }
     
    T'():
    Switch(token)
    {
    case n: 
    error;
    break;
    case +:
    build T' -> lambda;
    break;
    case *:
    get(*); F(); T'();
    build T'-> *FT';
    break;
    case (:
    error;
    break;
    case ):
    build T'->lambda;
    break;
    case $:
    build T'->lambda;
    break;
    }
     
    F():
    Switch(token)
    {
    case n:
    get(n);
    build F->n;
    break;
    case +:
    error;
    break;
    case *:
    error;
    break;
    case (: 
    get('(');E(); get(')');
    build F->(E);
    break;
    case ):
    error;
    break;
    case $:
    error;
    break;
    }

    Well the break thing doesn't work with string so simply replace those with if, else if, else, etc.

    What's going on is that it'll recursively call the validators, starting immediately with the TValidator() and then the EPValidator()

    The code should cause it to keep returning false if, while the execution is still running, it finds ANY place where it's not true in the grammar.

    However, it had been complaining that I was going out of bounds possibly so I grudgingly added another variable to keep track of the original size of the original string and told it if it ran out of tokens to return true unless of course it was null to start with, in which case return false.

    However, after doing this, it loops infinitely going through the printlns till Judgment Day so I can't show the output.

    The cases where it would return not valid would be if it found the wrong token as the next token when it shouldn't be.

    Each validator is called when the next part, or reading, or whatever it's called, is called so if it is checking E then if it encounters bad tokens, then it should return false and keep doing so recurisively or via my booleans and checking to see if the validators are returning false.

    Before it never entered the validators. Now it does but

    1.) It always is true
    2.) It loops forever

    What is causing the infinite loop?

    Other than the recursive calls, this is the only place where the infinite loop could be happening.

    while (getString() != null && validity == true )
             {
              // validity = read(st);
                validity = read();
    setValidity(validity);
    System.out.println("Validity is " + validity + " at this point in read");
    //if(validity == false)
    //return false;
              //  else
     
                // do something to remove some tokens
                // create a StringTokenizer variable newSt based on this new value with the token removed
     
                 //  isValid(newSt);
     
     
     
             }

    Hmmmmmm...it appears to be repeating the TValidator and EPValidator over and over and never going anywhere else. How come?

    It appears to be coming from the while loop in the isValid() method.

          private boolean read()
          {
    System.out.println("Came to read");
    boolean validity = getValidity();
     
    boolean temp = TValidator();
    if(!temp)
    return false;
    System.out.println("validitiy is " +validity + " after TValidator");
    boolean temp2 = EPValidator();
    if(!temp2)
    return false;
    System.out.println("validity is " + validity + " after EPValidator" );
    return validity;
          }
     
          private void setString(String tokens)
          {
             this.tokens = tokens;
          }
     
          private String getString()
          {
             return tokens;
          }
     
     
     
     
          public boolean isValid()
          {
     
                    // System.out.println(tokenizer.nextToken());
     
          // needs to check to see if String is null
          // needs a base case anyway
          boolean validity = true;
          int counter = 0;
             while (getString() != null && validity == true )
             {
              // validity = read(st);
                validity = read();
    setValidity(validity);
    System.out.println("Validity is " + validity + " at this point in read");
    //if(validity == false)
    //return false;
              //  else
     
                // do something to remove some tokens
                // create a StringTokenizer variable newSt based on this new value with the token removed
     
                 //  isValid(newSt);
     
     
     
             }
             return validity;
     
          }

    Either one of 2 or three things is happening

    1.) Something is happening wrong in read
    2.) Something is happening wrong in isValid
    3.) The token isn't being deleted by deleteToken() and it's going on forever (this could very well be happening)

    It's going through these things below and then looping and doing it over and over. I was able to freeze it by adding in a break so it won't go on forever.

     

    Came to read
    validitiy is true after TValidator
    validity is true after EPValidator
    Validity is true at this point in read




    The next thing is
    Came to read
    followed by
    validity is true after TValidator ...etc

    Ok, I've found it's removing the token. But maybe it's never reaching a case where it WOULD remove the token, though it should be reaching such a case.
    Last edited by javapenguin; November 9th, 2011 at 07:20 PM.

  18. #18
    Junior Member
    Join Date
    Nov 2011
    Posts
    5
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default Re: Confused on what tokenizer to use and how to use them.

    In first place, we have to follow the flow of the program...

    Legend-> Class::method::statement

    I only put the important lines...

    LL1Parser::main(String[] args)::"parser = new LL1Parser("+123");"
    LL1Parser::LL1Parser(String input)::"boolean b = isValid();"
    LL1Parser::isValid()::"while (getString() != null && validity == true)"
    LL1Parser::isValid()::"validity = read();"
    LL1Parser::read()::"boolean temp = TValidator();"
    LL1Parser::TValidator()::"if (TokenHandler.getNextToken(getString()) != null && getOriginalLength() != 0)"
    LL1Parser::TValidator()::"return true;"
    LL1Parser::read()::"boolean temp2 = EPValidator();"
    LL1Parser::EPValidator()::"if (TokenHandler.getNextToken(getString()) != null && getOriginalLength() != 0)"
    LL1Parser::EPValidator()::"return true;"

    ... then always it's happen ...

    LL1Parser::isValid()::"while (getString() != null && validity == true)"
    LL1Parser::isValid()::"validity = read();"
    LL1Parser::read()::"boolean temp = TValidator();"
    LL1Parser::TValidator()::"if (TokenHandler.getNextToken(getString()) != null && getOriginalLength() != 0)"
    LL1Parser::TValidator()::"return true;"
    LL1Parser::read()::"boolean temp2 = EPValidator();"
    LL1Parser::EPValidator()::"if (TokenHandler.getNextToken(getString()) != null && getOriginalLength() != 0)"
    LL1Parser::EPValidator()::"return true;"

    This expression is always true "TokenHandler.getNextToken(getString()) != null"
    and this one too "getOriginalLength() != 0"

  19. #19
    Think of me.... Mr.777's Avatar
    Join Date
    Mar 2011
    Location
    Pakistan
    Posts
    1,136
    My Mood
    Grumpy
    Thanks
    20
    Thanked 82 Times in 78 Posts
    Blog Entries
    1

    Default Re: Confused on what tokenizer to use and how to use them.

    @javapenguin:
    I would suggest you to read this!!!
    Your every post contains alot of details with nothing related to problem. You, yourself question your problem and answer it and then come with a very new problem, diverting from the main problem.

  20. #20
    Super Moderator Norm's Avatar
    Join Date
    May 2010
    Location
    Eastern Florida
    Posts
    21,051
    Thanks
    52
    Thanked 2,280 Times in 2,251 Posts

    Default Re: Confused on what tokenizer to use and how to use them.

    This OP seems to be an output only device.

  21. #21
    mmm.. coffee JavaPF's Avatar
    Join Date
    May 2008
    Location
    United Kingdom
    Posts
    3,336
    My Mood
    Mellow
    Thanks
    258
    Thanked 287 Times in 225 Posts
    Blog Entries
    4

    Default Re: Confused on what tokenizer to use and how to use them.

    Quote Originally Posted by Norm View Post
    This OP seems to be an output only device.
    Very confusing indeed.

    Javapenguin, don't write your every thought in the thread! We don't need to read your thought process. Keep it short and relevant.
    Take on board what is being said before you post back. You are rendering your threads useless.
    Please use [highlight=Java] code [/highlight] tags when posting your code.
    Forum Tip: Add to peoples reputation by clicking the button on their useful posts.

    Looking for a Java job? Visit - Java Programming Careers

  22. #22
    Banned
    Join Date
    May 2010
    Location
    North Central Illinois
    Posts
    1,631
    My Mood
    Sleepy
    Thanks
    390
    Thanked 112 Times in 110 Posts

    Default Re: Confused on what tokenizer to use and how to use them.

    I had to hand it in and was nearly done with it anyway.

  23. #23
    Super Moderator Norm's Avatar
    Join Date
    May 2010
    Location
    Eastern Florida
    Posts
    21,051
    Thanks
    52
    Thanked 2,280 Times in 2,251 Posts

    Default Re: Confused on what tokenizer to use and how to use them.

    You should get your own private forum where you can talk to yourself and be completely free to post as often as you want.

  24. #24
    Banned
    Join Date
    May 2010
    Location
    North Central Illinois
    Posts
    1,631
    My Mood
    Sleepy
    Thanks
    390
    Thanked 112 Times in 110 Posts

    Default Re: Confused on what tokenizer to use and how to use them.

    Sorry. I try and update things so people don't try and help me with something I've already solved.

    Since my threads normally don't get a lot of replies, I can often sometimes solve it before I get a single reply, hence the updates.

  25. #25
    Think of me.... Mr.777's Avatar
    Join Date
    Mar 2011
    Location
    Pakistan
    Posts
    1,136
    My Mood
    Grumpy
    Thanks
    20
    Thanked 82 Times in 78 Posts
    Blog Entries
    1

    Default Re: Confused on what tokenizer to use and how to use them.

    @javapenguin: It's not necessary to get a reply for every post on your desired time. You either have to be patient or think of your own.
    Not being so offensive, but everyone here tries best to help OP.

Similar Threads

  1. help me , im confused!!!!
    By sephskie in forum What's Wrong With My Code?
    Replies: 2
    Last Post: October 11th, 2011, 10:52 AM
  2. Confused..
    By jadowers in forum What's Wrong With My Code?
    Replies: 0
    Last Post: October 5th, 2011, 12:56 PM
  3. String Tokenizer Help
    By BuhRock in forum File I/O & Other I/O Streams
    Replies: 1
    Last Post: February 19th, 2011, 09:49 AM
  4. Stream Tokenizer
    By x3rubiachica3x in forum File I/O & Other I/O Streams
    Replies: 1
    Last Post: October 12th, 2010, 01:05 PM
  5. Confusion with C/C++
    By Eric in forum Java Applets
    Replies: 0
    Last Post: December 22nd, 2008, 01:18 PM