@Test
    public void testLargeTextTokenization() {
        var parameters = new LinguisticsParameters(Language.ENGLISH, StemMode.NONE, false, true);
        var tester = new OpenNlpLinguisticsTester();
        String input = "teststring ".repeat(100000);
        int numTokens = 0;
        List<Long> pos = new ArrayList<>();
        for (Token t : tester.tokenizer().tokenize(input, parameters)) {
            numTokens++;
            if ((numTokens % 100) == 0) {
                pos.add(t.getOffset());
            }
        }

        assertEquals("Check that all tokens have been tokenized", numTokens, 200000);
        assertTrue("Increasing token pos", tester.assertMonoIncr(pos));
    }
Test of tokenization, with stemming and accent removal

@author matskin
@author bratseth
opennlp-linguistics/src/test/java/com/yahoo/language/opennlp/OpenNlpTokenizationTestCase.java