private Item toItem(com.yahoo.language.process.Token token, String defaultIndex) {
        TermItem item;
        if (token.getType() == TokenType.NUMERIC) {
            item = new IntItem(token.getTokenString());
        }
        else {
            WordItem word = new WordItem(token.getTokenString());
            word.setStemmed(true); // Disable downstream stemming
            word.setNormalizable(false); // Disable downstream normaliing
            item = word;
        }
        item.setIndexName(defaultIndex);
        return item;
    }
A parser which delegates all tokenization and processing to the linguistics component.
The full string is given as-is to the linguistics component for tokenization, and
what comes back is assumes fully processed including stemming (if applicable).
The returned tokens are collected into a single parent item.

@author bratseth
container-search/src/main/java/com/yahoo/prelude/query/parser/LinguisticsParser.java