private boolean acceptApostropheAsWordCharacter(Index currentIndex) {
        if ( ! (currentIndex.isUriIndex() || currentIndex.isHostIndex())) {
            return true;
        }
        // this is a heuristic to check whether we probably have reached the end of an URL element
        for (int i = tokens.size() - 1; i >= 0; --i) {
            switch (tokens.get(i).kind) {
                case COLON:
                    if (i == indexLastExplicitlyChangedAt) return false;
                    break;
                case SPACE:
                    return true;
                default:
                    // do nothing
            }
        }
        // really not sure whether we should choose false instead, on cause of the guard at
        // the start, but this seems like the conservative choice
        return true;
    }
Resets this tokenizer and create tokens from the given string.

@param string the string to tokenize
@param defaultIndexName the name of the index to use as default
@param indexFacts information about the indexes we will search
@return a read-only list of tokens. This list can only be used by this thread
container-search/src/main/java/com/yahoo/prelude/query/parser/Tokenizer.java