protected void tokenize(String query, String defaultIndexName, IndexFacts.Session indexFacts, Language language) {
        Tokenizer tokenizer = new Tokenizer(environment.getLinguistics());
        tokenizer.setParserSettings(environment.getParserSettings());
        tokenizer.setSubstringSpecialTokens(language != null && language.isCjk());
        tokenizer.setSpecialTokens(environment.getSpecialTokens());
        tokens.initialize(tokenizer.tokenize(query, defaultIndexName, indexFacts));
    }
Tokenizes the given string and initializes tokens with the found tokens.

@param query            the string to tokenize
@param defaultIndexName the name of the index to use as default
@param indexFacts       resolved information about the index we are searching
@param language         the language set for this query, or null if none
container-search/src/main/java/com/yahoo/prelude/query/parser/AbstractParser.java