@Test
    void testSpecialTokenCaseInsensitive() {
        Tokenizer tokenizer = new Tokenizer(new SimpleLinguistics());

        tokenizer.setSpecialTokens(createSpecialTokens().getSpecialTokens("default"));
        List<?> tokens = tokenizer.tokenize("The AS/400 is great");

        assertEquals(new Token(WORD, "The"), tokens.get(0));
        assertEquals(new Token(SPACE, " "), tokens.get(1));
        assertEquals(new Token(WORD, "as/400"), tokens.get(2));
        assertEquals(new Token(SPACE, " "), tokens.get(3));
        assertEquals(new Token(WORD, "is"), tokens.get(4));
        assertEquals(new Token(SPACE, " "), tokens.get(5));
        assertEquals(new Token(WORD, "great"), tokens.get(6));
    }
In cjk languages, special tokens must be recognized as substrings of strings not
separated by space, as special token recognition happens before tokenization
container-search/src/test/java/com/yahoo/prelude/query/parser/test/TokenizerTestCase.java