private TokenScript determineScript(TokenScript tokenScript, TokenScript characterScript) {
        // if any character is LATIN, use that as token script; otherwise use script of first character seen.
        if (characterScript == TokenScript.LATIN) return TokenScript.LATIN;
        return tokenScript;
    }
Tokenize the input, and apply the given transform to each token string.
linguistics/src/main/java/com/yahoo/language/simple/SimpleTokenizer.java