private void tokenizeContent() {
        List<FieldPart> pre;
        if (isCJK) {
            pre = tokenizePretokenized();
        } else {
            pre = tokenizeUnknown();
        }
        setTokenizedContentUnchecked(pre);
    }
Creates a hit field

@param fieldname The field name
@param content The field content
@param cjk true if the content is CJK text
@param xmlProperty true if this should not quote XML syntax
container-search/src/main/java/com/yahoo/prelude/hitfield/HitField.java