@Test
    public void testTokenizer() {
        var tester = new OpenNlpLinguisticsTester();
        tester.assertTokenize("This IS a test, 123",
                              List.of("this", "is", "a", "test", "123"),
                              List.of("This", " ", "IS", " ", "a", " ", "test", ",", " ", "123"));
    }
Test of tokenization, with stemming and accent removal

@author matskin
@author bratseth
opennlp-linguistics/src/test/java/com/yahoo/language/opennlp/OpenNlpTokenizationTestCase.java