@Test
  public void testScanner() {
    String input = Joiner.on("").join(TOKENS);
    ImmutableList.Builder<String> tokensBuilder = ImmutableList.builder();
    JavaScanner tokenizer = new JavaScanner(input);
    int end;
    for (int i = 0; i < input.length(); i = end) {
      end = tokenizer.tokenEnd(i);
      tokensBuilder.add(input.substring(i, end));
    }
    assertThat(tokensBuilder.build()).containsExactlyElementsIn(TOKENS).inOrder();
  }
Tests basic scanner functionality. The test concatenates the tokens in {@link #TOKENS} and then
retokenizes that string, checking that the same list of tokens is produced.
value/src/test/java/com/google/auto/value/processor/JavaScannerTest.java