@Override
    protected Item parseItems() {
        CompositeItem parent = newComposite();
        Token token;
        while (null != (token = tokens.next())) {
            Item termItem = toTerm(token);
            if (termItem != null)
                parent.addItem(termItem);
        }
        return parent;
    }
A parser which turns contiguous searchable character into tokens and filters out other characters.
The resulting tokens are collected into a WeakAnd item.

@author bratseth
container-search/src/main/java/com/yahoo/prelude/query/parser/TokenizeParser.java