@Override
    protected void engineInitVerify(PublicKey publicKey) throws InvalidKeyException {
        if (publicKey instanceof OpenSSLRSAPublicKey) {
            OpenSSLRSAPublicKey rsaPublicKey = (OpenSSLRSAPublicKey) publicKey;
            key = rsaPublicKey.getOpenSSLKey();
        } else if (publicKey instanceof RSAPublicKey) {
            RSAPublicKey rsaPublicKey = (RSAPublicKey) publicKey;
            key = OpenSSLRSAPublicKey.getInstance(rsaPublicKey);
        } else {
            throw new InvalidKeyException("Need RSA public key");
        }

        // Allocate buffer according to RSA modulus size.
        int maxSize = NativeCrypto.RSA_size(key.getNativeRef());
        inputBuffer = new byte[maxSize];
        inputOffset = 0;
    }
Provides a flag to specify when the input is too long.
common/src/main/java/org/conscrypt/OpenSSLSignatureRawRSA.java