private int tokenEnd(UnicodeString s) {
            if (characterClasses.isSymbol(s.codePointAt(0)))
                return s.nextIndex(0); // symbols have length 1

            int i = 0;
            for (; i < s.length(); i = s.nextIndex(i)) {
                if ( ! characterClasses.isLetterOrDigit(s.codePointAt(i)))
                    return i;
            }
            return i;
        }
Given a string s starting by an indexable character, return the position where that token should end.
linguistics/src/main/java/com/yahoo/language/process/GramSplitter.java