@Override
    public Iterable<Token> tokenize(String input, LinguisticsParameters parameters) {
        return tokenize(input, token -> processToken(token, parameters));
    }
Tokenize the input, applying the transform of this to each token string.
linguistics/src/main/java/com/yahoo/language/simple/SimpleTokenizer.java