private TokenType determineType(TokenType tokenType, TokenType characterType) {
        if (characterType == TokenType.ALPHABETIC) return TokenType.ALPHABETIC;
        return tokenType;
    }
Tokenize the input, and apply the given transform to each token string.
linguistics/src/main/java/com/yahoo/language/simple/SimpleTokenizer.java