private boolean isAtTokenBoundary(TokenType prevType, TokenType nextType) {
        // Always index each symbol as a token
        if (prevType == TokenType.INDEXABLE_SYMBOL || nextType == TokenType.INDEXABLE_SYMBOL) return true;
        return !prevType.isIndexable() || !nextType.isIndexable();
    }
Tokenize the input, and apply the given transform to each token string.
linguistics/src/main/java/com/yahoo/language/simple/SimpleTokenizer.java