Java Stringtokenizer

nextToken() returns the previous token's value

Page 1 of 1

8 Replies - 2697 Views - Last Post: 01 July 2009 - 06:21 AM Rate Topic: -----

#1 mikeash  Icon User is offline

  • New D.I.C Head

Reputation: 0
  • View blog
  • Posts: 4
  • Joined: 27-June 09

Java Stringtokenizer

Post icon  Posted 27 June 2009 - 02:11 PM

Can anyone help, please? I'm trying to extract data elements from a delimited string. The delimiter is '~' (a tilde). The problem arises when there are 2 tildes with nothing between them.

Here's a sample of my data:
100643421~GB~ACTIVE~0~1~~true~5.0~1246132975801~~N
You'll see there are two instances of blank fields: immediately before the word 'true', and before the 'N'.

Using '~' as a tokenizer, my code works OK, extracting
100643421
GB
ACTIVE
0
1
but then, on the next invocation of nextToken(), I get the previous value (i.e. '1') returned.
A call to countTokens() gives a result of 9, not the 11 I would have expected.

The relevant parts of my code are as follows:

String aStr, workStr;
StringTokenizer st1 = new StringTokenizer(workStr, "~");
long theID;
String region;
...
aStr = st1.nextToken();
theID = Long.parseLong(aStr);
aStr = st1.nextToken();
region = aStr.trim();


Any suggestions will be greatly appreciated!

Is This A Good Question/Topic? 0
  • +

Replies To: Java Stringtokenizer

#2 nick2price  Icon User is offline

  • D.I.C Lover
  • member icon

Reputation: 561
  • View blog
  • Posts: 2,826
  • Joined: 23-November 07

Re: Java Stringtokenizer

Posted 27 June 2009 - 02:32 PM

To make it easy on yourself, instead of trying to figure out how to handle different amount of spaces between your words, why dont you make sure that the spacing is consistant? You only passing the tokenizer a String, and not a text file, so making the String consistant beforehand is problably your easiest option. How do you create the String anyways, by user input or somthing?
Was This Post Helpful? 0
  • +
  • -

#3 mikeash  Icon User is offline

  • New D.I.C Head

Reputation: 0
  • View blog
  • Posts: 4
  • Joined: 27-June 09

Re: Java Stringtokenizer

Posted 27 June 2009 - 02:50 PM

View Postnick2price, on 27 Jun, 2009 - 01:32 PM, said:

To make it easy on yourself, instead of trying to figure out how to handle different amount of spaces between your words, why dont you make sure that the spacing is consistant? You only passing the tokenizer a String, and not a text file, so making the String consistant beforehand is problably your easiest option. How do you create the String anyways, by user input or somthing?


I don't create the string, I get it from a source over which I have no control.

If I put the string into a file with other lines of similar data, MS Excel would have no trouble in opening the file and creating a worksheet with 11 columns, 2 of which would be blank. I don't like paying compliments to MS, but at least they can handle delimited files!
Was This Post Helpful? 0
  • +
  • -

#4 nick2price  Icon User is offline

  • D.I.C Lover
  • member icon

Reputation: 561
  • View blog
  • Posts: 2,826
  • Joined: 23-November 07

Re: Java Stringtokenizer

Posted 27 June 2009 - 03:07 PM

I cant remember exactly, but the second parameter of the method is a String of all deliminater types. So you could in theory do
StringTokenizer st1 = new StringTokenizer(workStr, "~  ~   ~");

See if somthing like that will work
Was This Post Helpful? 0
  • +
  • -

#5 mikeash  Icon User is offline

  • New D.I.C Head

Reputation: 0
  • View blog
  • Posts: 4
  • Joined: 27-June 09

Re: Java Stringtokenizer

Posted 29 June 2009 - 02:45 AM

View Postnick2price, on 27 Jun, 2009 - 02:07 PM, said:

I cant remember exactly, but the second parameter of the method is a String of all deliminater types. So you could in theory do
StringTokenizer st1 = new StringTokenizer(workStr, "~  ~   ~");

See if somthing like that will work

There are 3 constructors for StringTokenizer: the simplest assumes "white space" delimiters (space, tab, form-feed, etc), so you just specify the target string; the next requires you to specify the target string and a delimiter string; and the third takes a boolean parameter. In the third constructor, if the boolean is set to 'true', the tokens are returned with the delimiter string attached.

I believe it's possible to set a delimiter and then temporarily override it with a different delimiter, but this isn't what I want!

I can get StringTokenizer to do what I want by first using
<string>.replace("~~","~ ~")
or something similar. But why should this be necessary? Is this a design flaw in StringTokenizer?

Thanks for your help!
Was This Post Helpful? 0
  • +
  • -

#6 nick2price  Icon User is offline

  • D.I.C Lover
  • member icon

Reputation: 561
  • View blog
  • Posts: 2,826
  • Joined: 23-November 07

Re: Java Stringtokenizer

Posted 29 June 2009 - 03:49 AM

You should be able to do it straight of with somthing similar in my last post. The second constructor is a String of all types of deliminaters. So if your input has a mix of deliminaters like a space, ! and |, you can tell your tokenizer
StringTokenizer st1 = new StringTokenizer(workStr, "  ! |");

And that should look for all three deliminaters.
Was This Post Helpful? 0
  • +
  • -

#7 cfoley  Icon User is offline

  • Cabbage
  • member icon

Reputation: 1939
  • View blog
  • Posts: 4,027
  • Joined: 11-December 07

Re: Java Stringtokenizer

Posted 29 June 2009 - 04:52 AM

You don't get the error you are describing but the StringTokenizer does skip blank tokens. More precisely it treats contiguous delimiters as one. Try the String.split() method instead.

	public static void main(String[] args) {
		String workStr = "100643421~GB~ACTIVE~0~1~~true~5.0~1246132975801~~N";
		String[] tokens = workStr.split("~");
		for(String t : tokens) {
			System.out.println(t);
		}
	}


Was This Post Helpful? 1
  • +
  • -

#8 mikeash  Icon User is offline

  • New D.I.C Head

Reputation: 0
  • View blog
  • Posts: 4
  • Joined: 27-June 09

Re: Java Stringtokenizer

Posted 01 July 2009 - 05:44 AM

View Postcfoley, on 29 Jun, 2009 - 03:52 AM, said:

You don't get the error you are describing but the StringTokenizer does skip blank tokens. More precisely it treats contiguous delimiters as one. Try the String.split() method instead.

	public static void main(String[] args) {
		String workStr = "100643421~GB~ACTIVE~0~1~~true~5.0~1246132975801~~N";
		String[] tokens = workStr.split("~");
		for(String t : tokens) {
			System.out.println(t);
		}
	}


You're quite right - I didn't get the error I described, as I learned when I'd stopped pulling my hair out and examined the situation more calmly!

Many thanks for your solution.
Was This Post Helpful? 0
  • +
  • -

#9 cfoley  Icon User is offline

  • Cabbage
  • member icon

Reputation: 1939
  • View blog
  • Posts: 4,027
  • Joined: 11-December 07

Re: Java Stringtokenizer

Posted 01 July 2009 - 06:21 AM

Glad it helps. Thanks for hitting the thanks button. :)
Was This Post Helpful? 0
  • +
  • -

Page 1 of 1