[mikeash]

Can anyone help, please? I'm trying to extract data elements from a delimited string. The delimiter is '~' (a tilde). The problem arises when there are 2 tildes with nothing between them.

Here's a sample of my data:
100643421~GB~ACTIVE~0~1~~true~5.0~1246132975801~~N
You'll see there are two instances of blank fields: immediately before the word 'true', and before the 'N'.

Using '~' as a tokenizer, my code works OK, extracting
100643421
GB
ACTIVE
0
1
but then, on the next invocation of nextToken(), I get the previous value (i.e. '1') returned.
A call to countTokens() gives a result of 9, not the 11 I would have expected.

The relevant parts of my code are as follows:

String aStr, workStr;
StringTokenizer st1 = new StringTokenizer(workStr, "~");
long theID;
String region;
...
aStr = st1.nextToken();
theID = Long.parseLong(aStr);
aStr = st1.nextToken();
region = aStr.trim();

Any suggestions will be greatly appreciated!

[nick2price]

To make it easy on yourself, instead of trying to figure out how to handle different amount of spaces between your words, why dont you make sure that the spacing is consistant? You only passing the tokenizer a String, and not a text file, so making the String consistant beforehand is problably your easiest option. How do you create the String anyways, by user input or somthing?

[mikeash]

nick2price, on 27 Jun, 2009 - 01:32 PM, said:

To make it easy on yourself, instead of trying to figure out how to handle different amount of spaces between your words, why dont you make sure that the spacing is consistant? You only passing the tokenizer a String, and not a text file, so making the String consistant beforehand is problably your easiest option. How do you create the String anyways, by user input or somthing?

I don't create the string, I get it from a source over which I have no control.

If I put the string into a file with other lines of similar data, MS Excel would have no trouble in opening the file and creating a worksheet with 11 columns, 2 of which would be blank. I don't like paying compliments to MS, but at least they can handle delimited files!

[nick2price]

I cant remember exactly, but the second parameter of the method is a String of all deliminater types. So you could in theory do

StringTokenizer st1 = new StringTokenizer(workStr, "~  ~   ~");

See if somthing like that will work

[mikeash]

nick2price, on 27 Jun, 2009 - 02:07 PM, said:

I cant remember exactly, but the second parameter of the method is a String of all deliminater types. So you could in theory do

StringTokenizer st1 = new StringTokenizer(workStr, "~  ~   ~");

See if somthing like that will work

There are 3 constructors for StringTokenizer: the simplest assumes "white space" delimiters (space, tab, form-feed, etc), so you just specify the target string; the next requires you to specify the target string and a delimiter string; and the third takes a boolean parameter. In the third constructor, if the boolean is set to 'true', the tokens are returned with the delimiter string attached.

I believe it's possible to set a delimiter and then temporarily override it with a different delimiter, but this isn't what I want!

I can get StringTokenizer to do what I want by first using

<string>.replace("~~","~ ~")

or something similar. But why should this be necessary? Is this a design flaw in StringTokenizer?

Thanks for your help!

[nick2price]

You should be able to do it straight of with somthing similar in my last post. The second constructor is a String of all types of deliminaters. So if your input has a mix of deliminaters like a space, ! and |, you can tell your tokenizer

StringTokenizer st1 = new StringTokenizer(workStr, "  ! |");

And that should look for all three deliminaters.

[cfoley]

You don't get the error you are describing but the StringTokenizer does skip blank tokens. More precisely it treats contiguous delimiters as one. Try the String.split() method instead.

	public static void main(String[] args) {
		String workStr = "100643421~GB~ACTIVE~0~1~~true~5.0~1246132975801~~N";
		String[] tokens = workStr.split("~");
		for(String t : tokens) {
			System.out.println(t);
		}
	}

[mikeash]

cfoley, on 29 Jun, 2009 - 03:52 AM, said:

You don't get the error you are describing but the StringTokenizer does skip blank tokens. More precisely it treats contiguous delimiters as one. Try the String.split() method instead.

	public static void main(String[] args) {
		String workStr = "100643421~GB~ACTIVE~0~1~~true~5.0~1246132975801~~N";
		String[] tokens = workStr.split("~");
		for(String t : tokens) {
			System.out.println(t);
		}
	}

You're quite right - I didn't get the error I described, as I learned when I'd stopped pulling my hair out and examined the situation more calmly!

Many thanks for your solution.

[cfoley]

Glad it helps. Thanks for hitting the thanks button.