private int consumeToTerminator(int start,String terminator) {
        int end = start;
        while (end < source.length()) {
            if (terminatorStartsAt(end,terminator))
                break;
            end++;
        }
        tokens.add(new Token(WORD, source.substring(start, end), true, new Substring(start, end, source)));
        if (end >= source.length())
            return end;
        else
            return end + terminator.length(); // Don't create a token for the terminator
    }
Resets this tokenizer and create tokens from the given string.

@param string the string to tokenize
@param defaultIndexName the name of the index to use as default
@param indexFacts information about the indexes we will search
@return a read-only list of tokens. This list can only be used by this thread
container-search/src/main/java/com/yahoo/prelude/query/parser/Tokenizer.java