@Test
    void testCjkQueryWithOverlappingTokens() {
        // The test language segmenter will segment "bcd" into the overlapping tokens "bc" "cd"
        assertTransformed("bcd", "SAND bc cd", Query.Type.ALL, Language.CHINESE_SIMPLIFIED, Language.CHINESE_TRADITIONAL,
                TestLinguistics.INSTANCE);

        // While "efg" will be segmented into one of the standard options, "e" "fg"
        assertTransformed("efg", "SAND e fg", Query.Type.ALL, Language.CHINESE_SIMPLIFIED, Language.CHINESE_TRADITIONAL,
                TestLinguistics.INSTANCE);
    }
Overlapping tokens splits some sequences of "bcd" into "bc" "cd" instead of e.g. "b",
"cd". This improves recall in some cases. Vespa
must combine overlapping tokens as PHRASE, not AND to avoid a too high recall because of the token overlap.
container-search/src/test/java/com/yahoo/prelude/querytransform/test/CJKSearcherTestCase.java