Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/1.11.0/metrics/rouge/rouge.py
 datasets version: 1.12.1

`metric = datasets.load_metric('rouge')`

The error:

> ConnectionError                           Traceback (most recent call last)
> <ipython-input-3-dd10a0c5212f> in <module>
> ----> 1 metric = datasets.load_metric('rouge')
>
> /usr/local/lib/python3.6/dist-packages/datasets/load.py in load_metric(path, config_name, process_id, num_process, cache_dir, experiment_id, keep_in_memory, download_config, download_mode, script_version, **metric_init_kwargs)
>     613         download_config=download_config,
>     614         download_mode=download_mode,
> --> 615         dataset=False,
>     616     )
>     617     metric_cls = import_main_class(module_path, dataset=False)
>
> /usr/local/lib/python3.6/dist-packages/datasets/load.py in prepare_module(path, script_version, download_config, download_mode, dataset, force_local_path, dynamic_modules_path, return_resolved_file_path, **download_kwargs)
>     328                 file_path = hf_github_url(path=path, name=name, dataset=dataset, version=script_version)
>     329                 try:
> --> 330                     local_path = cached_path(file_path, download_config=download_config)
>     331                 except FileNotFoundError:
>     332                     if script_version is not None:
>
> /usr/local/lib/python3.6/dist-packages/datasets/utils/file_utils.py in cached_path(url_or_filename, download_config, **download_kwargs)
>     296             use_etag=download_config.use_etag,
>     297             max_retries=download_config.max_retries,
> --> 298             use_auth_token=download_config.use_auth_token,
>     299         )
>     300     elif os.path.exists(url_or_filename):
>
> /usr/local/lib/python3.6/dist-packages/datasets/utils/file_utils.py in get_from_cache(url, cache_dir, force_download, proxies, etag_timeout, resume_download, user_agent, local_files_only, use_etag, max_retries, use_auth_token)
>     603             raise FileNotFoundError("Couldn't find file at {}".format(url))
>     604         _raise_if_offline_mode_is_enabled(f"Tried to reach {url}")
> --> 605         raise ConnectionError("Couldn't reach {}".format(url))
>     606
>     607     # Try a second time
>
> ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/1.11.0/metrics/rouge/rouge.py

Is there any remedy to solve the connection issue ?
 It seems to be able to solve this issue by adding the equivalent `accuracy.py` locally.
change `metric = datasets.load_metric("accuracy")` to  `metric = datasets.load_metric(path = "./accuracy.py")`.
Copy `accuracy.py` from browser at [accuracy.py](https://raw.githubusercontent.com/huggingface/datasets/1.11.0/metrics/accuracy/accuracy.py)


datasets-cli: convertion of a tfds dataset to a huggingface one.
 ### Discussed in https://github.com/huggingface/datasets/discussions/3079

Originally posted by **vitalyshalumov** October 14, 2021
I'm trying to convert a tfds dataset to a huggingface one.

I've tried:

1.       datasets-cli convert  --tfds_path ~/tensorflow_datasets/mnist/3.0.1/ --datasets_directory ~/.cache/huggingface/datasets/mnist/3.0.1/

2.       datasets-cli convert  --tfds_path ~/tensorflow_datasets/mnist/3.0.1/ --datasets_directory ~/.cache/huggingface/datasets/

and other permutations.
The script appears to be running and finishing without an error but when looking in the  huggingface/datasets/ folder nothing is created.

 Hi,

the MNIST dataset is already available on the Hub. You can use it as follows:
```python
import datasets
dataset_dict = datasets.load_dataset("mnist")
```

As for the conversion of TFDS datasets to HF datasets, we will be working on it in the coming months, so stay tuned.


"arabic_billion_words" dataset does not create the full dataset
 ## Describe the bug
When running:
raw_dataset = load_dataset('arabic_billion_words','Alittihad')
the correct dataset file is pulled from the url.
But, the generated dataset includes just a small portion of the data included in the file.
This is true for all other portions of the "arabic_billion_words" dataset ('Almasryalyoum',.....)

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
raw_dataset = load_dataset('arabic_billion_words','Alittihad')

#The screen message
Downloading and preparing dataset arabic_billion_words/Alittihad (download: 332.13 MiB, generated: 20.62 MiB, post-processed: Unknown size, total: 352.74 MiB)

## Expected results
over 100K sentences

## Actual results
only 11K sentences

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.14.0
- Platform: Linux-5.8.0-63-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 4.0.1

 Thanks for reporting, @vitalyshalumov.

Apparently the script to parse the data has a bug, and does not generate the entire dataset.

I'm fixing it.


Segmentation fault when loading datasets from file
 ## Describe the bug
Custom dataset loading sometimes segfaults and kills the process if chunks contain a variety of features/

## Steps to reproduce the bug

Download an example file:
```
wget https://gist.githubusercontent.com/TevenLeScao/11e2184394b3fa47d693de2550942c6b/raw/4232704d08fbfcaf93e5b51def9e5051507651ad/tiny_kelm.jsonl
```
Then in Python:
```
import datasets
tiny_kelm = datasets.load_dataset("json", data_files="tiny_kelm.jsonl", chunksize=100000)
```

## Expected results
a `tiny_kelm` functional dataset

## Actual results
☠ `Segmentation fault (core dumped)` ☠

## Environment info
- `datasets` version: 1.14.0
- Platform: Linux-5.11.0-38-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 5.0.0
 Hi ! I created an issue on Arrow's JIRA after making a minimum reproducible example

https://issues.apache.org/jira/browse/ARROW-14439

```python
import io

import pyarrow.json as paj

batch = b'{"a": [], "b": 1}\n{"b": 1}'
block_size = 12

paj.read_json(
    io.BytesIO(batch), read_options=paj.ReadOptions(block_size=block_size)
)
```

I don't see a way to workaround this properly now without hurting the performance of the JSON loader significantly though


Segmentation fault when loading datasets from file
 ## Describe the bug
Custom dataset loading sometimes segfaults and kills the process if chunks contain a variety of features/

## Steps to reproduce the bug

Download an example file:
```
wget https://gist.githubusercontent.com/TevenLeScao/11e2184394b3fa47d693de2550942c6b/raw/4232704d08fbfcaf93e5b51def9e5051507651ad/tiny_kelm.jsonl
```
Then in Python:
```
import datasets
tiny_kelm = datasets.load_dataset("json", data_files="tiny_kelm.jsonl", chunksize=100000)
```

## Expected results
a `tiny_kelm` functional dataset

## Actual results
☠ `Segmentation fault (core dumped)` ☠

## Environment info
- `datasets` version: 1.14.0
- Platform: Linux-5.11.0-38-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 5.0.0
 The issue has been fixed in pyarrow 6.0.0, please update pyarrow :)

The issue was due to missing fields in the JSON data of type list. Now it's working fine and missing list fields are replaced with empty lists


OSError with a custom dataset loading script
 ## Describe the bug
I am getting an OS error when trying to load the newly uploaded dataset classla/janes_tag. What puzzles me is that I have already uploaded a very similar dataset - classla/reldi_hr - with no issues. The loading scripts for the two datasets are almost identical and they have the same directory structure, yet I am only getting an error with janes_tag.

## Steps to reproduce the bug
```python
dataset = datasets.load_dataset('classla/janes_tag', split='validation')
```

## Expected results
Dataset correctly loaded.

## Actual results

Traceback (most recent call last):
  File "C:/mypath/test.py", line 91, in <module>
    load_and_print('janes_tag')
  File "C:/mypath/test.py", line 32, in load_and_print
    dataset = datasets.load_dataset('classla/{}'.format(ds_name), split='validation')
  File "C:\mypath\venv\lib\site-packages\datasets\load.py", line 1632, in load_dataset
    use_auth_token=use_auth_token,
  File "C:\mypath\venv\lib\site-packages\datasets\builder.py", line 608, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "C:\mypath\venv\lib\site-packages\datasets\builder.py", line 704, in _download_and_prepare
    ) from None
OSError: Cannot find data file.
Original error:
[Errno 2] No such file or directory: 'C:\\mypath\\.cache\\huggingface\\datasets\\downloads\\2c9996e44bdc5af9c89bffb9e6d7a3e42fdb2f56bacab45de13b20f3032ea7ca\\data\\train_all.conllup'

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.14.0
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.7.5
- PyArrow version: 3.0.0

 Hi,

there is a difference in how the `data_dir` is zipped between the `classla/janes_tag` and the `classla/reldi_hr` dataset. After unzipping, for the former, the data files (`*.conllup`) are in the root directory (root -> data files), and for the latter, they are inside the `data` directory (root -> `data` -> data files).

This can be fixed by removing the `os.path.join` call in https://huggingface.co/datasets/classla/janes_tag/blob/main/janes_tag.py#L86

Let me know if this works for you.


OSError with a custom dataset loading script
 ## Describe the bug
I am getting an OS error when trying to load the newly uploaded dataset classla/janes_tag. What puzzles me is that I have already uploaded a very similar dataset - classla/reldi_hr - with no issues. The loading scripts for the two datasets are almost identical and they have the same directory structure, yet I am only getting an error with janes_tag.

## Steps to reproduce the bug
```python
dataset = datasets.load_dataset('classla/janes_tag', split='validation')
```

## Expected results
Dataset correctly loaded.

## Actual results

Traceback (most recent call last):
  File "C:/mypath/test.py", line 91, in <module>
    load_and_print('janes_tag')
  File "C:/mypath/test.py", line 32, in load_and_print
    dataset = datasets.load_dataset('classla/{}'.format(ds_name), split='validation')
  File "C:\mypath\venv\lib\site-packages\datasets\load.py", line 1632, in load_dataset
    use_auth_token=use_auth_token,
  File "C:\mypath\venv\lib\site-packages\datasets\builder.py", line 608, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "C:\mypath\venv\lib\site-packages\datasets\builder.py", line 704, in _download_and_prepare
    ) from None
OSError: Cannot find data file.
Original error:
[Errno 2] No such file or directory: 'C:\\mypath\\.cache\\huggingface\\datasets\\downloads\\2c9996e44bdc5af9c89bffb9e6d7a3e42fdb2f56bacab45de13b20f3032ea7ca\\data\\train_all.conllup'

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.14.0
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.7.5
- PyArrow version: 3.0.0

 Hi Mario,

I had already tried that before, but it didn't work. I have now recreated the `classla/janes_tag`  zip file so that it also contains the `data` directory, but I am still getting the same error.


OSError with a custom dataset loading script
 ## Describe the bug
I am getting an OS error when trying to load the newly uploaded dataset classla/janes_tag. What puzzles me is that I have already uploaded a very similar dataset - classla/reldi_hr - with no issues. The loading scripts for the two datasets are almost identical and they have the same directory structure, yet I am only getting an error with janes_tag.

## Steps to reproduce the bug
```python
dataset = datasets.load_dataset('classla/janes_tag', split='validation')
```

## Expected results
Dataset correctly loaded.

## Actual results

Traceback (most recent call last):
  File "C:/mypath/test.py", line 91, in <module>
    load_and_print('janes_tag')
  File "C:/mypath/test.py", line 32, in load_and_print
    dataset = datasets.load_dataset('classla/{}'.format(ds_name), split='validation')
  File "C:\mypath\venv\lib\site-packages\datasets\load.py", line 1632, in load_dataset
    use_auth_token=use_auth_token,
  File "C:\mypath\venv\lib\site-packages\datasets\builder.py", line 608, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "C:\mypath\venv\lib\site-packages\datasets\builder.py", line 704, in _download_and_prepare
    ) from None
OSError: Cannot find data file.
Original error:
[Errno 2] No such file or directory: 'C:\\mypath\\.cache\\huggingface\\datasets\\downloads\\2c9996e44bdc5af9c89bffb9e6d7a3e42fdb2f56bacab45de13b20f3032ea7ca\\data\\train_all.conllup'

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.14.0
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.7.5
- PyArrow version: 3.0.0

 Hi,

I just tried to download the `classla/janes_tag` dataset, and this time the zip file is extracted correctly. However, the script is now throwing the IndexError, probably due to a bug in the `_generate_examples`.

Let me know if you are still getting the same error.


OSError with a custom dataset loading script
 ## Describe the bug
I am getting an OS error when trying to load the newly uploaded dataset classla/janes_tag. What puzzles me is that I have already uploaded a very similar dataset - classla/reldi_hr - with no issues. The loading scripts for the two datasets are almost identical and they have the same directory structure, yet I am only getting an error with janes_tag.

## Steps to reproduce the bug
```python
dataset = datasets.load_dataset('classla/janes_tag', split='validation')
```

## Expected results
Dataset correctly loaded.

## Actual results

Traceback (most recent call last):
  File "C:/mypath/test.py", line 91, in <module>
    load_and_print('janes_tag')
  File "C:/mypath/test.py", line 32, in load_and_print
    dataset = datasets.load_dataset('classla/{}'.format(ds_name), split='validation')
  File "C:\mypath\venv\lib\site-packages\datasets\load.py", line 1632, in load_dataset
    use_auth_token=use_auth_token,
  File "C:\mypath\venv\lib\site-packages\datasets\builder.py", line 608, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "C:\mypath\venv\lib\site-packages\datasets\builder.py", line 704, in _download_and_prepare
    ) from None
OSError: Cannot find data file.
Original error:
[Errno 2] No such file or directory: 'C:\\mypath\\.cache\\huggingface\\datasets\\downloads\\2c9996e44bdc5af9c89bffb9e6d7a3e42fdb2f56bacab45de13b20f3032ea7ca\\data\\train_all.conllup'

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.14.0
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.7.5
- PyArrow version: 3.0.0

 Hi,

could you try to download the dataset with a different `cache_dir` like so:
```python
import datasets
dataset = datasets.load_dataset('classla/janes_tag', split='validation', cache_dir="path/to/different/cache/dir")
```
If this works, then most likely the cached extracted data is causing issues. This data is stored at `~/.cache/huggingface/datasets/downloads/extracted` and needs to be deleted, and then it should work (you can easily locate the directory with the path given in the `OSError` message). Additionally, I'd suggest you to update `datasets` to the newest version with:
```
pip install -U datasets
```


OSError with a custom dataset loading script
 ## Describe the bug
I am getting an OS error when trying to load the newly uploaded dataset classla/janes_tag. What puzzles me is that I have already uploaded a very similar dataset - classla/reldi_hr - with no issues. The loading scripts for the two datasets are almost identical and they have the same directory structure, yet I am only getting an error with janes_tag.

## Steps to reproduce the bug
```python
dataset = datasets.load_dataset('classla/janes_tag', split='validation')
```

## Expected results
Dataset correctly loaded.

## Actual results

Traceback (most recent call last):
  File "C:/mypath/test.py", line 91, in <module>
    load_and_print('janes_tag')
  File "C:/mypath/test.py", line 32, in load_and_print
    dataset = datasets.load_dataset('classla/{}'.format(ds_name), split='validation')
  File "C:\mypath\venv\lib\site-packages\datasets\load.py", line 1632, in load_dataset
    use_auth_token=use_auth_token,
  File "C:\mypath\venv\lib\site-packages\datasets\builder.py", line 608, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "C:\mypath\venv\lib\site-packages\datasets\builder.py", line 704, in _download_and_prepare
    ) from None
OSError: Cannot find data file.
Original error:
[Errno 2] No such file or directory: 'C:\\mypath\\.cache\\huggingface\\datasets\\downloads\\2c9996e44bdc5af9c89bffb9e6d7a3e42fdb2f56bacab45de13b20f3032ea7ca\\data\\train_all.conllup'

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.14.0
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.7.5
- PyArrow version: 3.0.0

 Thank you, deleting the `~/.cache/huggingface/datasets/downloads/extracted` directory helped. However, I am still having problems.

There was indeed a bug in the script that was throwing an `IndexError`, which I have now corrected (added the condition to skip the lines starting with '# text') and it is working locally, but still throws an error when I try to load the dataset from HuggingFace. I literally copied and pasted the `_generate_examples` function and ran it on the `dev_all.conllup` file, which I even re-downloaded from the repository to be certain that the files are exactly the same. I also deleted everything again just in case, but it didn't help. The code works locally, but throws an `IndexError` when loading from `datasets.`


OSError with a custom dataset loading script
 ## Describe the bug
I am getting an OS error when trying to load the newly uploaded dataset classla/janes_tag. What puzzles me is that I have already uploaded a very similar dataset - classla/reldi_hr - with no issues. The loading scripts for the two datasets are almost identical and they have the same directory structure, yet I am only getting an error with janes_tag.

## Steps to reproduce the bug
```python
dataset = datasets.load_dataset('classla/janes_tag', split='validation')
```

## Expected results
Dataset correctly loaded.

## Actual results

Traceback (most recent call last):
  File "C:/mypath/test.py", line 91, in <module>
    load_and_print('janes_tag')
  File "C:/mypath/test.py", line 32, in load_and_print
    dataset = datasets.load_dataset('classla/{}'.format(ds_name), split='validation')
  File "C:\mypath\venv\lib\site-packages\datasets\load.py", line 1632, in load_dataset
    use_auth_token=use_auth_token,
  File "C:\mypath\venv\lib\site-packages\datasets\builder.py", line 608, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "C:\mypath\venv\lib\site-packages\datasets\builder.py", line 704, in _download_and_prepare
    ) from None
OSError: Cannot find data file.
Original error:
[Errno 2] No such file or directory: 'C:\\mypath\\.cache\\huggingface\\datasets\\downloads\\2c9996e44bdc5af9c89bffb9e6d7a3e42fdb2f56bacab45de13b20f3032ea7ca\\data\\train_all.conllup'

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.14.0
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.7.5
- PyArrow version: 3.0.0

 Hi,

Did some investigation.

To fix the dataset script on the Hub, append the following labels to the `names` list of the `upos_tags` field:
```'INTJ NOUN', 'AUX PRON', 'PART ADV', 'PRON ADP', 'INTJ INTJ', 'VERB NOUN', 'NOUN AUX'```.

This step is required to avoid an error due to missing labels in the following step which is:
```python
load_dataset("classla/janes_tag", split="validation", download_mode="force_redownload")
```
This will generate and cache the dataset, so specifying `download_mode` will not be required anymore unless you update the script/data on the Hub.


Add OpenSLR 83 - Crowdsourced high-quality UK and Ireland English Dialect speech
 ## Adding a Dataset
- **Name:** *openslr**
- **Description:** *Data set which contains male and female recordings of English from various dialects of the UK and Ireland.*
- **Paper:** *https://www.openslr.org/resources/83/about.html*
- **Data:** *Eleven separate data files can be found via https://www.openslr.org/resources/83/*
- **Motivation:** *Increase english ASR data with UK and Irish dialects*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

The *openslr* dataset already exists, this will add additional subset, *SLR83*.
 Ugh. The index files for SLR83 are CSV, not TSV. I need to add logic to process these index files.


load_from_disk in DatasetsDict/Dataset not working with PyArrowHDFS wrapper implementing fsspec.spec.AbstractFileSystem
 ## Describe the bug
Passing a PyArrowHDFS implementation of fsspec.spec.AbstractFileSystem (in the `fs` param required by `load_from_disk` methods in `DatasetDict` (in datasets_dict.py) and `Dataset` (in arrow_dataset.py) results in an error when calling the download method in the `fs` parameter.

## Steps to reproduce the bug

The documentation for the `fs` parameter states:

```
fs (:class:`~filesystems.S3FileSystem` or ``fsspec.spec.AbstractFileSystem``, optional, default ``None``):
                Instance of the remote filesystem used to download the files from.
```

`PyArrowHDFS` from [fsspec](https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/hdfs.html) implements `fsspec.spec.AbstractFileSystem`. However, when using it as shown below, I get an error.

```python
from fsspec.implementations.hdfs import PyArrowHDFS
...
transformed_corpus_path = "/user/my_user/clickbait/transformed_ds/"
fs = PyArrowHDFS(host, port, user, kerb_ticket=kerb_ticket)
dss = DatasetDict.load_from_disk(transformed_corpus_path, fs, True)
```

## Expected results

Previous to load from disk, I have managed to successfully store in HDFS the data and meta-information of a DatasetDict by doing:
```python
transformed_corpus_path = "/user/my_user/clickbait/transformed_ds/"
fs = PyArrowHDFS(host, port, user, kerb_ticket=kerb_ticket)
my_datasets.save_to_disk(transformed_corpus_path, fs=fs)
```

As I have 3 datasets in the DatasetDict named `my_datasets`, the previous Python code creates the following contents in HDFS:

```sh
$ hadoop fs -ls "/user/my_user/clickbait/transformed_ds/"
Found 4 items
-rw-------   3 my_user users         43 2021-10-19 03:08 /user/my_user/clickbait/transformed_ds/dataset_dict.json
drwx------   - my_user users          0 2021-10-19 03:08 /user/my_user/clickbait/transformed_ds/test
drwx------   - my_user users          0 2021-10-19 03:08 /user/my_user/clickbait/transformed_ds/train
drwx------   - my_user users          0 2021-10-19 03:08 /user/my_user/clickbait/transformed_ds/validation
```

I would expect to recover on `dss` the Arrow-backed datasets I previously saved in HDFS calling the `save_to_disk` method on the `DatasetDict` object when invoking `DatasetDict.load_from_disk(...)` as described above.

## Actual results

However, when trying to recover the saved datasets, I get this error:

```
...
  File "/home/fperez/dev/neuromancer/neuromancer/corpus.py", line 186, in load_transformed_corpus_from_disk
    dss = DatasetDict.load_from_disk(transformed_corpus_path, fs, True)
  File "/home/fperez/anaconda3/envs/neuromancer/lib/python3.9/site-packages/datasets/dataset_dict.py", line 748, in load_from_disk
    dataset_dict[k] = Dataset.load_from_disk(dataset_dict_split_path, fs, keep_in_memory=keep_in_memory)
  File "/home/fperez/anaconda3/envs/neuromancer/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 1048, in load_from_disk
    fs.download(src_dataset_path, dataset_path.as_posix(), recursive=True)
  File "pyarrow/_hdfsio.pyx", line 438, in pyarrow._hdfsio.HadoopFileSystem.download
TypeError: download() got an unexpected keyword argument 'recursive'
```

Examining the [signature of the download method in pyarrow 5.0.0](https://github.com/apache/arrow/blob/54d2bd89c99df72fa091b025452f85dd5d88e3cf/python/pyarrow/_hdfsio.pyx#L438) we can see that there's no download parameter:

```python
    def download(self, path, stream, buffer_size=None):
        with self.open(path, 'rb') as f:
            f.download(stream, buffer_size=buffer_size)
```

## Environment info
- `datasets` version: 1.13.3
- Platform: Linux-3.10.0-1160.15.2.el7.x86_64-x86_64-with-glibc2.33
- Python version: 3.9.7
- PyArrow version: 5.0.0

 Hi ! Can you try again with pyarrow 6.0.0 ? I think it includes some changes regarding filesystems compatibility with fsspec.


load_from_disk in DatasetsDict/Dataset not working with PyArrowHDFS wrapper implementing fsspec.spec.AbstractFileSystem
 ## Describe the bug
Passing a PyArrowHDFS implementation of fsspec.spec.AbstractFileSystem (in the `fs` param required by `load_from_disk` methods in `DatasetDict` (in datasets_dict.py) and `Dataset` (in arrow_dataset.py) results in an error when calling the download method in the `fs` parameter.

## Steps to reproduce the bug

The documentation for the `fs` parameter states:

```
fs (:class:`~filesystems.S3FileSystem` or ``fsspec.spec.AbstractFileSystem``, optional, default ``None``):
                Instance of the remote filesystem used to download the files from.
```

`PyArrowHDFS` from [fsspec](https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/hdfs.html) implements `fsspec.spec.AbstractFileSystem`. However, when using it as shown below, I get an error.

```python
from fsspec.implementations.hdfs import PyArrowHDFS
...
transformed_corpus_path = "/user/my_user/clickbait/transformed_ds/"
fs = PyArrowHDFS(host, port, user, kerb_ticket=kerb_ticket)
dss = DatasetDict.load_from_disk(transformed_corpus_path, fs, True)
```

## Expected results

Previous to load from disk, I have managed to successfully store in HDFS the data and meta-information of a DatasetDict by doing:
```python
transformed_corpus_path = "/user/my_user/clickbait/transformed_ds/"
fs = PyArrowHDFS(host, port, user, kerb_ticket=kerb_ticket)
my_datasets.save_to_disk(transformed_corpus_path, fs=fs)
```

As I have 3 datasets in the DatasetDict named `my_datasets`, the previous Python code creates the following contents in HDFS:

```sh
$ hadoop fs -ls "/user/my_user/clickbait/transformed_ds/"
Found 4 items
-rw-------   3 my_user users         43 2021-10-19 03:08 /user/my_user/clickbait/transformed_ds/dataset_dict.json
drwx------   - my_user users          0 2021-10-19 03:08 /user/my_user/clickbait/transformed_ds/test
drwx------   - my_user users          0 2021-10-19 03:08 /user/my_user/clickbait/transformed_ds/train
drwx------   - my_user users          0 2021-10-19 03:08 /user/my_user/clickbait/transformed_ds/validation
```

I would expect to recover on `dss` the Arrow-backed datasets I previously saved in HDFS calling the `save_to_disk` method on the `DatasetDict` object when invoking `DatasetDict.load_from_disk(...)` as described above.

## Actual results

However, when trying to recover the saved datasets, I get this error:

```
...
  File "/home/fperez/dev/neuromancer/neuromancer/corpus.py", line 186, in load_transformed_corpus_from_disk
    dss = DatasetDict.load_from_disk(transformed_corpus_path, fs, True)
  File "/home/fperez/anaconda3/envs/neuromancer/lib/python3.9/site-packages/datasets/dataset_dict.py", line 748, in load_from_disk
    dataset_dict[k] = Dataset.load_from_disk(dataset_dict_split_path, fs, keep_in_memory=keep_in_memory)
  File "/home/fperez/anaconda3/envs/neuromancer/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 1048, in load_from_disk
    fs.download(src_dataset_path, dataset_path.as_posix(), recursive=True)
  File "pyarrow/_hdfsio.pyx", line 438, in pyarrow._hdfsio.HadoopFileSystem.download
TypeError: download() got an unexpected keyword argument 'recursive'
```

Examining the [signature of the download method in pyarrow 5.0.0](https://github.com/apache/arrow/blob/54d2bd89c99df72fa091b025452f85dd5d88e3cf/python/pyarrow/_hdfsio.pyx#L438) we can see that there's no download parameter:

```python
    def download(self, path, stream, buffer_size=None):
        with self.open(path, 'rb') as f:
            f.download(stream, buffer_size=buffer_size)
```

## Environment info
- `datasets` version: 1.13.3
- Platform: Linux-3.10.0-1160.15.2.el7.x86_64-x86_64-with-glibc2.33
- Python version: 3.9.7
- PyArrow version: 5.0.0

 Hi @lhoestq! I ended up using `fsspec.implementations.arrow.HadoopFileSystem` which doesn't have the problem I described with pyarrow 5.0.0.

I'll try again with `PyArrowHDFS` once I update arrow to 6.0.0.

Thanks!


OverflowError: There was an overflow in the <class 'pyarrow.lib.ListArray'>. Try to reduce writer_batch_size to have batches smaller than 2GB
 ## Describe the bug
Despite having batches way under 2Gb when running `datasets.map()`, after processing correctly the data of the first batch without fuss and irrespective of writer_batch_size (say 2,4,8,16,32,64 and 128 in my case), it returns the following error :

> OverflowError: There was an overflow in the <class 'pyarrow.lib.ListArray'>. Try to reduce writer_batch_size to have batches smaller than 2GB

Note that I always run `batch_size=writer_batch_size` :

## Steps to reproduce the bug
```python
datasets.map(lambda example : {"column_name" : function(arguments)}, batched=False, remove_columns = datasets.column_names, batch_size=batch_size, writer_batch_size=batch_size, disable_nullable=True, num_proc=None, desc="blablabla")
```
## Introspecting CUDA memory during bug

Placed within `function(arguments)` the following statement to introspect memory usage, merely a little over 1/4 of 2Gb
`print(torch.cuda.memory_summary(device=device, abbreviated=False))`

> |===========================================================================|
 |                  PyTorch CUDA memory summary, device ID 0                 |
 |---------------------------------------------------------------------------|
 |            CUDA OOMs: 0            |        cudaMalloc retries: 0         |
 |===========================================================================|
 |        Metric         | Cur Usage  | Peak Usage | Tot Alloc  | Tot Freed  |
 |---------------------------------------------------------------------------|
 | Allocated memory      |  541418 KB |  545725 KB |  555695 KB |   14276 KB |
 |       from large pool |  540672 KB |  544431 KB |  544431 KB |    3759 KB |
 |       from small pool |     746 KB |    1714 KB |   11264 KB |   10517 KB |
 |---------------------------------------------------------------------------|
 | Active memory         |  541418 KB |  545725 KB |  555695 KB |   14276 KB |
 |       from large pool |  540672 KB |  544431 KB |  544431 KB |    3759 KB |
 |       from small pool |     746 KB |    1714 KB |   11264 KB |   10517 KB |
 |---------------------------------------------------------------------------|
 | GPU reserved memory   |  598016 KB |  598016 KB |  598016 KB |       0 B  |
 |       from large pool |  595968 KB |  595968 KB |  595968 KB |       0 B  |
 |       from small pool |    2048 KB |    2048 KB |    2048 KB |       0 B  |
 |---------------------------------------------------------------------------|
 | Non-releasable memory |   36117 KB |   52292 KB |  274275 KB |  238158 KB |
|       from large pool |   34816 KB |   51537 KB |  261713 KB |  226897 KB |
 |       from small pool |    1301 KB |    2045 KB |   12562 KB |   11261 KB |
 |---------------------------------------------------------------------------|
 | Allocations           |     198    |     224    |     478    |     280    |
 |       from large pool |      74    |      75    |      75    |       1    |
 |       from small pool |     124    |     150    |     403    |     279    |
 |---------------------------------------------------------------------------|
 | Active allocs         |     198    |     224    |     478    |     280    |
 |       from large pool |      74    |      75    |      75    |       1    |
 |       from small pool |     124    |     150    |     403    |     279    |
 |---------------------------------------------------------------------------|
 | GPU reserved segments |      21    |      21    |      21    |       0    |
 |       from large pool |      20    |      20    |      20    |       0    |
 |       from small pool |       1    |       1    |       1    |       0    |
 |---------------------------------------------------------------------------|
 | Non-releasable allocs |      18    |      23    |     166    |     148    |
 |       from large pool |      17    |      18    |      19    |       2    |
 |       from small pool |       1    |       6    |     147    |     146    |
 |===========================================================================|

## Expected results
Efficiently process the datasets and write it down to disk.

## Actual results
--------------------------------------------------------------------------
OverflowError                             Traceback (most recent call last)
~\anaconda3\envs\xxx\lib\site-packages\datasets\arrow_dataset.py in _map_single(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, new_fingerprint, rank, offset, disable_tqdm, desc, cache_only)
   2390                             else:
-> 2391                                 writer.write(example)
   2392                 else:

~\anaconda3\envs\xxx\lib\site-packages\datasets\arrow_writer.py in write(self, example, key, writer_batch_size)
    367
--> 368             self.write_examples_on_file()
    369

~\anaconda3\envs\xxx\lib\site-packages\datasets\arrow_writer.py in write_examples_on_file(self)
    316                 if not isinstance(pa_array[0], pa.lib.FloatScalar):
--> 317                     raise OverflowError(
    318                         "There was an overflow in the {}. Try to reduce writer_batch_size to have batches smaller than 2GB".format(

OverflowError: There was an overflow in the <class 'pyarrow.lib.ListArray'>. Try to reduce writer_batch_size to have batches smaller than 2GB

During handling of the above exception, another exception occurred:

OverflowError                             Traceback (most recent call last)
~\AppData\Local\Temp/ipykernel_16268/2456940807.py in <module>
      3 #tracker = OfflineEmissionsTracker(country_iso_code="FRA", project_name='xxx'+time_stamp,output_dir='./codecarbon')
      4 #tracker.start()
----> 5 process_datasets(source_datasets_paths, dataset_dir, LM_tokenizer, LMhead_model, datasets_selection=['wikipedia'], from_scratch=True,
      6                  clean_sentences=False, negative_sampling=False, translate=False, tokenize=False, generate_embeddings=True, concatenate_embeddings=False,
      7                  max_sample=10000, padding='do_not_pad', truncation=True, cpu_batch_size=1000, gpu_batch_size=2, cpu_writer_batch_size=1000, gpu_writer_batch_size=2, disable_nullable=True, num_proc=None) #

~\xxx\xxx.py in process_datasets(source_datasets_paths, dataset_dir, LM_tokenizer, LMhead_model, datasets_selection, from_scratch, clean_sentences, translate, negative_sampling, tokenize, generate_embeddings, concatenate_embeddings, max_sample, padding, truncation, cpu_batch_size, gpu_batch_size, cpu_writer_batch_size, gpu_writer_batch_size, disable_nullable, num_proc)
    481             for column in tqdm(dataset.column_names, desc=f'Processing column', leave=False):
    482                 if "xxx_" in column:
--> 483                     dataset = dataset.map(lambda example :
    484                                                 {"embeddings_"+str(column).replace("translated_",""):function(input_ids=example[column],
    485                                                                                token_type_ids=example[column.replace("input_ids","token_type_ids")],

~\anaconda3\envs\xxx\lib\site-packages\datasets\arrow_dataset.py in map(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc, suffix_template, new_fingerprint, desc)
   2034
   2035         if num_proc is None or num_proc == 1:
-> 2036             return self._map_single(
   2037                 function=function,
   2038                 with_indices=with_indices,

~\anaconda3\envs\xxx\lib\site-packages\datasets\arrow_dataset.py in wrapper(*args, **kwargs)
    501             self: "Dataset" = kwargs.pop("self")
    502         # apply actual function
--> 503         out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    504         datasets: List["Dataset"] = list(out.values()) if isinstance(out, dict) else [out]
    505         for dataset in datasets:

~\anaconda3\envs\xxx\lib\site-packages\datasets\arrow_dataset.py in wrapper(*args, **kwargs)
    468         }
    469         # apply actual function
--> 470         out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    471         datasets: List["Dataset"] = list(out.values()) if isinstance(out, dict) else [out]
    472         # re-apply format to the output

~\anaconda3\envs\xxx\lib\site-packages\datasets\fingerprint.py in wrapper(*args, **kwargs)
    404             # Call actual function
    405
--> 406             out = func(self, *args, **kwargs)
    407
    408             # Update fingerprint of in-place transforms + update in-place history of transforms

~\anaconda3\envs\xxx\lib\site-packages\datasets\arrow_dataset.py in _map_single(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, new_fingerprint, rank, offset, disable_tqdm, desc, cache_only)
   2425                 if update_data:
   2426                     if writer is not None:
-> 2427                         writer.finalize()
   2428                     if tmp_file is not None:
   2429                         tmp_file.close()

~\anaconda3\envs\xxx\lib\site-packages\datasets\arrow_writer.py in finalize(self, close_stream)
    440             # Re-intializing to empty list for next batch
    441             self.hkey_record = []
--> 442         self.write_examples_on_file()
    443         if self.pa_writer is None:
    444             if self._schema is not None:

~\anaconda3\envs\xxx\lib\site-packages\datasets\arrow_writer.py in write_examples_on_file(self)
    315                 # This check fails with FloatArrays with nans, which is not what we want, so account for that:
    316                 if not isinstance(pa_array[0], pa.lib.FloatScalar):
--> 317                     raise OverflowError(
    318                         "There was an overflow in the {}. Try to reduce writer_batch_size to have batches smaller than 2GB".format(
    319                             type(pa_array)

OverflowError: There was an overflow in the <class 'pyarrow.lib.ListArray'>. Try to reduce writer_batch_size to have batches smaller than 2GB

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.13.3
- Platform: Windows-10-10.0.19042-SP0
- Python version: 3.8.11
- PyArrow version: 3.0.0

##Next steps
Testing on Linux.
@albertvillanova

 I am very unsure on why you tagged me here. I am not a maintainer of the Datasets library and have no idea how to help you.


OverflowError: There was an overflow in the <class 'pyarrow.lib.ListArray'>. Try to reduce writer_batch_size to have batches smaller than 2GB
 ## Describe the bug
Despite having batches way under 2Gb when running `datasets.map()`, after processing correctly the data of the first batch without fuss and irrespective of writer_batch_size (say 2,4,8,16,32,64 and 128 in my case), it returns the following error :

> OverflowError: There was an overflow in the <class 'pyarrow.lib.ListArray'>. Try to reduce writer_batch_size to have batches smaller than 2GB

Note that I always run `batch_size=writer_batch_size` :

## Steps to reproduce the bug
```python
datasets.map(lambda example : {"column_name" : function(arguments)}, batched=False, remove_columns = datasets.column_names, batch_size=batch_size, writer_batch_size=batch_size, disable_nullable=True, num_proc=None, desc="blablabla")
```
## Introspecting CUDA memory during bug

Placed within `function(arguments)` the following statement to introspect memory usage, merely a little over 1/4 of 2Gb
`print(torch.cuda.memory_summary(device=device, abbreviated=False))`

> |===========================================================================|
 |                  PyTorch CUDA memory summary, device ID 0                 |
 |---------------------------------------------------------------------------|
 |            CUDA OOMs: 0            |        cudaMalloc retries: 0         |
 |===========================================================================|
 |        Metric         | Cur Usage  | Peak Usage | Tot Alloc  | Tot Freed  |
 |---------------------------------------------------------------------------|
 | Allocated memory      |  541418 KB |  545725 KB |  555695 KB |   14276 KB |
 |       from large pool |  540672 KB |  544431 KB |  544431 KB |    3759 KB |
 |       from small pool |     746 KB |    1714 KB |   11264 KB |   10517 KB |
 |---------------------------------------------------------------------------|
 | Active memory         |  541418 KB |  545725 KB |  555695 KB |   14276 KB |
 |       from large pool |  540672 KB |  544431 KB |  544431 KB |    3759 KB |
 |       from small pool |     746 KB |    1714 KB |   11264 KB |   10517 KB |
 |---------------------------------------------------------------------------|
 | GPU reserved memory   |  598016 KB |  598016 KB |  598016 KB |       0 B  |
 |       from large pool |  595968 KB |  595968 KB |  595968 KB |       0 B  |
 |       from small pool |    2048 KB |    2048 KB |    2048 KB |       0 B  |
 |---------------------------------------------------------------------------|
 | Non-releasable memory |   36117 KB |   52292 KB |  274275 KB |  238158 KB |
|       from large pool |   34816 KB |   51537 KB |  261713 KB |  226897 KB |
 |       from small pool |    1301 KB |    2045 KB |   12562 KB |   11261 KB |
 |---------------------------------------------------------------------------|
 | Allocations           |     198    |     224    |     478    |     280    |
 |       from large pool |      74    |      75    |      75    |       1    |
 |       from small pool |     124    |     150    |     403    |     279    |
 |---------------------------------------------------------------------------|
 | Active allocs         |     198    |     224    |     478    |     280    |
 |       from large pool |      74    |      75    |      75    |       1    |
 |       from small pool |     124    |     150    |     403    |     279    |
 |---------------------------------------------------------------------------|
 | GPU reserved segments |      21    |      21    |      21    |       0    |
 |       from large pool |      20    |      20    |      20    |       0    |
 |       from small pool |       1    |       1    |       1    |       0    |
 |---------------------------------------------------------------------------|
 | Non-releasable allocs |      18    |      23    |     166    |     148    |
 |       from large pool |      17    |      18    |      19    |       2    |
 |       from small pool |       1    |       6    |     147    |     146    |
 |===========================================================================|

## Expected results
Efficiently process the datasets and write it down to disk.

## Actual results
--------------------------------------------------------------------------
OverflowError                             Traceback (most recent call last)
~\anaconda3\envs\xxx\lib\site-packages\datasets\arrow_dataset.py in _map_single(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, new_fingerprint, rank, offset, disable_tqdm, desc, cache_only)
   2390                             else:
-> 2391                                 writer.write(example)
   2392                 else:

~\anaconda3\envs\xxx\lib\site-packages\datasets\arrow_writer.py in write(self, example, key, writer_batch_size)
    367
--> 368             self.write_examples_on_file()
    369

~\anaconda3\envs\xxx\lib\site-packages\datasets\arrow_writer.py in write_examples_on_file(self)
    316                 if not isinstance(pa_array[0], pa.lib.FloatScalar):
--> 317                     raise OverflowError(
    318                         "There was an overflow in the {}. Try to reduce writer_batch_size to have batches smaller than 2GB".format(

OverflowError: There was an overflow in the <class 'pyarrow.lib.ListArray'>. Try to reduce writer_batch_size to have batches smaller than 2GB

During handling of the above exception, another exception occurred:

OverflowError                             Traceback (most recent call last)
~\AppData\Local\Temp/ipykernel_16268/2456940807.py in <module>
      3 #tracker = OfflineEmissionsTracker(country_iso_code="FRA", project_name='xxx'+time_stamp,output_dir='./codecarbon')
      4 #tracker.start()
----> 5 process_datasets(source_datasets_paths, dataset_dir, LM_tokenizer, LMhead_model, datasets_selection=['wikipedia'], from_scratch=True,
      6                  clean_sentences=False, negative_sampling=False, translate=False, tokenize=False, generate_embeddings=True, concatenate_embeddings=False,
      7                  max_sample=10000, padding='do_not_pad', truncation=True, cpu_batch_size=1000, gpu_batch_size=2, cpu_writer_batch_size=1000, gpu_writer_batch_size=2, disable_nullable=True, num_proc=None) #

~\xxx\xxx.py in process_datasets(source_datasets_paths, dataset_dir, LM_tokenizer, LMhead_model, datasets_selection, from_scratch, clean_sentences, translate, negative_sampling, tokenize, generate_embeddings, concatenate_embeddings, max_sample, padding, truncation, cpu_batch_size, gpu_batch_size, cpu_writer_batch_size, gpu_writer_batch_size, disable_nullable, num_proc)
    481             for column in tqdm(dataset.column_names, desc=f'Processing column', leave=False):
    482                 if "xxx_" in column:
--> 483                     dataset = dataset.map(lambda example :
    484                                                 {"embeddings_"+str(column).replace("translated_",""):function(input_ids=example[column],
    485                                                                                token_type_ids=example[column.replace("input_ids","token_type_ids")],

~\anaconda3\envs\xxx\lib\site-packages\datasets\arrow_dataset.py in map(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc, suffix_template, new_fingerprint, desc)
   2034
   2035         if num_proc is None or num_proc == 1:
-> 2036             return self._map_single(
   2037                 function=function,
   2038                 with_indices=with_indices,

~\anaconda3\envs\xxx\lib\site-packages\datasets\arrow_dataset.py in wrapper(*args, **kwargs)
    501             self: "Dataset" = kwargs.pop("self")
    502         # apply actual function
--> 503         out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    504         datasets: List["Dataset"] = list(out.values()) if isinstance(out, dict) else [out]
    505         for dataset in datasets:

~\anaconda3\envs\xxx\lib\site-packages\datasets\arrow_dataset.py in wrapper(*args, **kwargs)
    468         }
    469         # apply actual function
--> 470         out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    471         datasets: List["Dataset"] = list(out.values()) if isinstance(out, dict) else [out]
    472         # re-apply format to the output

~\anaconda3\envs\xxx\lib\site-packages\datasets\fingerprint.py in wrapper(*args, **kwargs)
    404             # Call actual function
    405
--> 406             out = func(self, *args, **kwargs)
    407
    408             # Update fingerprint of in-place transforms + update in-place history of transforms

~\anaconda3\envs\xxx\lib\site-packages\datasets\arrow_dataset.py in _map_single(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, new_fingerprint, rank, offset, disable_tqdm, desc, cache_only)
   2425                 if update_data:
   2426                     if writer is not None:
-> 2427                         writer.finalize()
   2428                     if tmp_file is not None:
   2429                         tmp_file.close()

~\anaconda3\envs\xxx\lib\site-packages\datasets\arrow_writer.py in finalize(self, close_stream)
    440             # Re-intializing to empty list for next batch
    441             self.hkey_record = []
--> 442         self.write_examples_on_file()
    443         if self.pa_writer is None:
    444             if self._schema is not None:

~\anaconda3\envs\xxx\lib\site-packages\datasets\arrow_writer.py in write_examples_on_file(self)
    315                 # This check fails with FloatArrays with nans, which is not what we want, so account for that:
    316                 if not isinstance(pa_array[0], pa.lib.FloatScalar):
--> 317                     raise OverflowError(
    318                         "There was an overflow in the {}. Try to reduce writer_batch_size to have batches smaller than 2GB".format(
    319                             type(pa_array)

OverflowError: There was an overflow in the <class 'pyarrow.lib.ListArray'>. Try to reduce writer_batch_size to have batches smaller than 2GB

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.13.3
- Platform: Windows-10-10.0.19042-SP0
- Python version: 3.8.11
- PyArrow version: 3.0.0

##Next steps
Testing on Linux.
@albertvillanova

 Ok got it, tensor full of NaNs, cf.

~\anaconda3\envs\xxx\lib\site-packages\datasets\arrow_writer.py in write_examples_on_file(self)
315 # This check fails with FloatArrays with nans, which is not what we want, so account for that:


AttributeError: module 'huggingface_hub.hf_api' has no attribute 'DatasetInfo'
 ## Describe the bug
When using `pip install datasets`
or use `conda install -c huggingface -c conda-forge datasets`
cannot install datasets

## Steps to reproduce the bug
```python
from datasets import load_dataset

dataset = load_dataset("sst", "default")
```

## Actual results
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-5-fbe7981e6e21> in <module>
      1 import torch
      2 import transformers
----> 3 from datasets import load_dataset
      4
      5 dataset = load_dataset("sst", "default")

~/miniforge3/envs/actor/lib/python3.8/site-packages/datasets/__init__.py in <module>
     35 from .arrow_reader import ArrowReader, ReadInstruction
     36 from .arrow_writer import ArrowWriter
---> 37 from .builder import ArrowBasedBuilder, BeamBasedBuilder, BuilderConfig, DatasetBuilder, GeneratorBasedBuilder
     38 from .combine import interleave_datasets
     39 from .dataset_dict import DatasetDict, IterableDatasetDict

~/miniforge3/envs/actor/lib/python3.8/site-packages/datasets/builder.py in <module>
     42 )
     43 from .arrow_writer import ArrowWriter, BeamWriter
---> 44 from .data_files import DataFilesDict, _sanitize_patterns
     45 from .dataset_dict import DatasetDict, IterableDatasetDict
     46 from .fingerprint import Hasher

~/miniforge3/envs/actor/lib/python3.8/site-packages/datasets/data_files.py in <module>
    118
    119 def _exec_patterns_in_dataset_repository(
--> 120     dataset_info: huggingface_hub.hf_api.DatasetInfo,
    121     patterns: List[str],
    122     allowed_extensions: Optional[list] = None,

AttributeError: module 'huggingface_hub.hf_api' has no attribute 'DatasetInfo'

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.13.3
- Platform: macOS-11.3.1-arm64-arm-64bit
- Python version: 3.8.10
- PyArrow version: 5.0.0

 Hi @JTWang2000, thanks for reporting.

However, I cannot reproduce your reported bug:
```python
>>> from datasets import load_dataset

>>> dataset = load_dataset("sst", "default")
>>> dataset
DatasetDict({
    train: Dataset({
        features: ['sentence', 'label', 'tokens', 'tree'],
        num_rows: 8544
    })
    validation: Dataset({
        features: ['sentence', 'label', 'tokens', 'tree'],
        num_rows: 1101
    })
    test: Dataset({
        features: ['sentence', 'label', 'tokens', 'tree'],
        num_rows: 2210
    })
})
```

Maybe, the cause is that you have a quite old version of `huggingface_hub`. Could you please try to update it and confirm if the problem persists?
```
pip install -U huggingface_hub
```


AttributeError: module 'huggingface_hub.hf_api' has no attribute 'DatasetInfo'
 ## Describe the bug
When using `pip install datasets`
or use `conda install -c huggingface -c conda-forge datasets`
cannot install datasets

## Steps to reproduce the bug
```python
from datasets import load_dataset

dataset = load_dataset("sst", "default")
```

## Actual results
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-5-fbe7981e6e21> in <module>
      1 import torch
      2 import transformers
----> 3 from datasets import load_dataset
      4
      5 dataset = load_dataset("sst", "default")

~/miniforge3/envs/actor/lib/python3.8/site-packages/datasets/__init__.py in <module>
     35 from .arrow_reader import ArrowReader, ReadInstruction
     36 from .arrow_writer import ArrowWriter
---> 37 from .builder import ArrowBasedBuilder, BeamBasedBuilder, BuilderConfig, DatasetBuilder, GeneratorBasedBuilder
     38 from .combine import interleave_datasets
     39 from .dataset_dict import DatasetDict, IterableDatasetDict

~/miniforge3/envs/actor/lib/python3.8/site-packages/datasets/builder.py in <module>
     42 )
     43 from .arrow_writer import ArrowWriter, BeamWriter
---> 44 from .data_files import DataFilesDict, _sanitize_patterns
     45 from .dataset_dict import DatasetDict, IterableDatasetDict
     46 from .fingerprint import Hasher

~/miniforge3/envs/actor/lib/python3.8/site-packages/datasets/data_files.py in <module>
    118
    119 def _exec_patterns_in_dataset_repository(
--> 120     dataset_info: huggingface_hub.hf_api.DatasetInfo,
    121     patterns: List[str],
    122     allowed_extensions: Optional[list] = None,

AttributeError: module 'huggingface_hub.hf_api' has no attribute 'DatasetInfo'

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.13.3
- Platform: macOS-11.3.1-arm64-arm-64bit
- Python version: 3.8.10
- PyArrow version: 5.0.0

 Im facing the same issue. I did run the upgrade command but that doesnt seem to resolve the issue


AttributeError: module 'huggingface_hub.hf_api' has no attribute 'DatasetInfo'
 ## Describe the bug
When using `pip install datasets`
or use `conda install -c huggingface -c conda-forge datasets`
cannot install datasets

## Steps to reproduce the bug
```python
from datasets import load_dataset

dataset = load_dataset("sst", "default")
```

## Actual results
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-5-fbe7981e6e21> in <module>
      1 import torch
      2 import transformers
----> 3 from datasets import load_dataset
      4
      5 dataset = load_dataset("sst", "default")

~/miniforge3/envs/actor/lib/python3.8/site-packages/datasets/__init__.py in <module>
     35 from .arrow_reader import ArrowReader, ReadInstruction
     36 from .arrow_writer import ArrowWriter
---> 37 from .builder import ArrowBasedBuilder, BeamBasedBuilder, BuilderConfig, DatasetBuilder, GeneratorBasedBuilder
     38 from .combine import interleave_datasets
     39 from .dataset_dict import DatasetDict, IterableDatasetDict

~/miniforge3/envs/actor/lib/python3.8/site-packages/datasets/builder.py in <module>
     42 )
     43 from .arrow_writer import ArrowWriter, BeamWriter
---> 44 from .data_files import DataFilesDict, _sanitize_patterns
     45 from .dataset_dict import DatasetDict, IterableDatasetDict
     46 from .fingerprint import Hasher

~/miniforge3/envs/actor/lib/python3.8/site-packages/datasets/data_files.py in <module>
    118
    119 def _exec_patterns_in_dataset_repository(
--> 120     dataset_info: huggingface_hub.hf_api.DatasetInfo,
    121     patterns: List[str],
    122     allowed_extensions: Optional[list] = None,

AttributeError: module 'huggingface_hub.hf_api' has no attribute 'DatasetInfo'

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.13.3
- Platform: macOS-11.3.1-arm64-arm-64bit
- Python version: 3.8.10
- PyArrow version: 5.0.0

 Hi @aneeshjain, could you please specify which `huggingface_hub` version you are using?

Besides that, please run `datasets-cli env` and copy-and-paste its output below.


AttributeError: module 'huggingface_hub.hf_api' has no attribute 'DatasetInfo'
 ## Describe the bug
When using `pip install datasets`
or use `conda install -c huggingface -c conda-forge datasets`
cannot install datasets

## Steps to reproduce the bug
```python
from datasets import load_dataset

dataset = load_dataset("sst", "default")
```

## Actual results
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-5-fbe7981e6e21> in <module>
      1 import torch
      2 import transformers
----> 3 from datasets import load_dataset
      4
      5 dataset = load_dataset("sst", "default")

~/miniforge3/envs/actor/lib/python3.8/site-packages/datasets/__init__.py in <module>
     35 from .arrow_reader import ArrowReader, ReadInstruction
     36 from .arrow_writer import ArrowWriter
---> 37 from .builder import ArrowBasedBuilder, BeamBasedBuilder, BuilderConfig, DatasetBuilder, GeneratorBasedBuilder
     38 from .combine import interleave_datasets
     39 from .dataset_dict import DatasetDict, IterableDatasetDict

~/miniforge3/envs/actor/lib/python3.8/site-packages/datasets/builder.py in <module>
     42 )
     43 from .arrow_writer import ArrowWriter, BeamWriter
---> 44 from .data_files import DataFilesDict, _sanitize_patterns
     45 from .dataset_dict import DatasetDict, IterableDatasetDict
     46 from .fingerprint import Hasher

~/miniforge3/envs/actor/lib/python3.8/site-packages/datasets/data_files.py in <module>
    118
    119 def _exec_patterns_in_dataset_repository(
--> 120     dataset_info: huggingface_hub.hf_api.DatasetInfo,
    121     patterns: List[str],
    122     allowed_extensions: Optional[list] = None,

AttributeError: module 'huggingface_hub.hf_api' has no attribute 'DatasetInfo'

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.13.3
- Platform: macOS-11.3.1-arm64-arm-64bit
- Python version: 3.8.10
- PyArrow version: 5.0.0

 The problem seems to be with the latest version of `datasets`. After running `pip install -U datasets huggingface_hub`, I get the following:

```bash
python -c "import huggingface_hub; print(f'hbvers={huggingface_hub.__version__}'); import datasets; print(f'dvers={datasets.__version__}')"
hbvers=0.0.8
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/conda/lib/python3.6/site-packages/datasets/__init__.py", line 37, in <module>
    from .builder import ArrowBasedBuilder, BeamBasedBuilder, BuilderConfig, DatasetBuilder, GeneratorBasedBuilder
  File "/opt/conda/lib/python3.6/site-packages/datasets/builder.py", line 44, in <module>
    from .data_files import DataFilesDict, _sanitize_patterns
  File "/opt/conda/lib/python3.6/site-packages/datasets/data_files.py", line 122, in <module>
    allowed_extensions: Optional[list] = None,
AttributeError: module 'huggingface_hub.hf_api' has no attribute 'DatasetInfo'
````
Note that pip reports the latest `datasets` version as
```bash
 pip show datasets
Name: datasets
Version: 1.14.0
```
However, if I downgrade datasets with `pip install datasets==1.11.0`, things now work
```bash
python -c "import huggingface_hub; print(f'hbvers={huggingface_hub.__version__}'); import datasets; print(f'dvers={datasets.__version__}')"
hbvers=0.0.8
dvers=1.11.0
````


AttributeError: module 'huggingface_hub.hf_api' has no attribute 'DatasetInfo'
 ## Describe the bug
When using `pip install datasets`
or use `conda install -c huggingface -c conda-forge datasets`
cannot install datasets

## Steps to reproduce the bug
```python
from datasets import load_dataset

dataset = load_dataset("sst", "default")
```

## Actual results
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-5-fbe7981e6e21> in <module>
      1 import torch
      2 import transformers
----> 3 from datasets import load_dataset
      4
      5 dataset = load_dataset("sst", "default")

~/miniforge3/envs/actor/lib/python3.8/site-packages/datasets/__init__.py in <module>
     35 from .arrow_reader import ArrowReader, ReadInstruction
     36 from .arrow_writer import ArrowWriter
---> 37 from .builder import ArrowBasedBuilder, BeamBasedBuilder, BuilderConfig, DatasetBuilder, GeneratorBasedBuilder
     38 from .combine import interleave_datasets
     39 from .dataset_dict import DatasetDict, IterableDatasetDict

~/miniforge3/envs/actor/lib/python3.8/site-packages/datasets/builder.py in <module>
     42 )
     43 from .arrow_writer import ArrowWriter, BeamWriter
---> 44 from .data_files import DataFilesDict, _sanitize_patterns
     45 from .dataset_dict import DatasetDict, IterableDatasetDict
     46 from .fingerprint import Hasher

~/miniforge3/envs/actor/lib/python3.8/site-packages/datasets/data_files.py in <module>
    118
    119 def _exec_patterns_in_dataset_repository(
--> 120     dataset_info: huggingface_hub.hf_api.DatasetInfo,
    121     patterns: List[str],
    122     allowed_extensions: Optional[list] = None,

AttributeError: module 'huggingface_hub.hf_api' has no attribute 'DatasetInfo'

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.13.3
- Platform: macOS-11.3.1-arm64-arm-64bit
- Python version: 3.8.10
- PyArrow version: 5.0.0

 > Hi @JTWang2000, thanks for reporting.
>
> However, I cannot reproduce your reported bug:
>
> ```python
> >>> from datasets import load_dataset
>
> >>> dataset = load_dataset("sst", "default")
> >>> dataset
> DatasetDict({
>     train: Dataset({
>         features: ['sentence', 'label', 'tokens', 'tree'],
>         num_rows: 8544
>     })
>     validation: Dataset({
>         features: ['sentence', 'label', 'tokens', 'tree'],
>         num_rows: 1101
>     })
>     test: Dataset({
>         features: ['sentence', 'label', 'tokens', 'tree'],
>         num_rows: 2210
>     })
> })
> ```
>
> Maybe, the cause is that you have a quite old version of `huggingface_hub`. Could you please try to update it and confirm if the problem persists?
>
> ```
> pip install -U huggingface_hub
> ```

My problem solved after updating huggingface hub command. Thanks a lot and sorry for the late reply.


AttributeError: module 'huggingface_hub.hf_api' has no attribute 'DatasetInfo'
 ## Describe the bug
When using `pip install datasets`
or use `conda install -c huggingface -c conda-forge datasets`
cannot install datasets

## Steps to reproduce the bug
```python
from datasets import load_dataset

dataset = load_dataset("sst", "default")
```

## Actual results
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-5-fbe7981e6e21> in <module>
      1 import torch
      2 import transformers
----> 3 from datasets import load_dataset
      4
      5 dataset = load_dataset("sst", "default")

~/miniforge3/envs/actor/lib/python3.8/site-packages/datasets/__init__.py in <module>
     35 from .arrow_reader import ArrowReader, ReadInstruction
     36 from .arrow_writer import ArrowWriter
---> 37 from .builder import ArrowBasedBuilder, BeamBasedBuilder, BuilderConfig, DatasetBuilder, GeneratorBasedBuilder
     38 from .combine import interleave_datasets
     39 from .dataset_dict import DatasetDict, IterableDatasetDict

~/miniforge3/envs/actor/lib/python3.8/site-packages/datasets/builder.py in <module>
     42 )
     43 from .arrow_writer import ArrowWriter, BeamWriter
---> 44 from .data_files import DataFilesDict, _sanitize_patterns
     45 from .dataset_dict import DatasetDict, IterableDatasetDict
     46 from .fingerprint import Hasher

~/miniforge3/envs/actor/lib/python3.8/site-packages/datasets/data_files.py in <module>
    118
    119 def _exec_patterns_in_dataset_repository(
--> 120     dataset_info: huggingface_hub.hf_api.DatasetInfo,
    121     patterns: List[str],
    122     allowed_extensions: Optional[list] = None,

AttributeError: module 'huggingface_hub.hf_api' has no attribute 'DatasetInfo'

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.13.3
- Platform: macOS-11.3.1-arm64-arm-64bit
- Python version: 3.8.10
- PyArrow version: 5.0.0

 @tjruwase, please note that versions of `datsets` and `huggingface_hub` must be compatible one with each other:
- In `datasets` version `1.11.0`, the requirement on `huggingface_hub` was: `huggingface_hub<0.1.0`
  https://github.com/huggingface/datasets/blob/2cc00f372a96133e701275eec4d6b26d15257289/setup.py#L90
  - Therefore, your installed `huggingface_hub` version `0.0.8` was compatible
- In `datasets` version `1.12.0`, the requirement on `huggingface_hub` was: `huggingface_hub>=0.0.14,<0.1.0`
  https://github.com/huggingface/datasets/blob/6c766f9115d686182d76b1b937cb27e099c45d68/setup.py#L104
  - Therefore, your installed `huggingface_hub` version `0.0.8` was no longer compatible
- Currently, in `datasets` version `1.15.1`, the requirement on `huggingface_hub` is: `huggingface_hub>=0.1.0,<1.0.0`
  https://github.com/huggingface/datasets/blob/018100679d21cf27136f0eccb1c50e3a9c968ce2/setup.py#L102

@JTWang2000, thanks for your answer. I close this issue then.


`cast_column` makes audio decoding fail

`cast_column` makes audio decoding fail
 ## Describe the bug

After changing the sampling rate automatic decoding fails.

## Steps to reproduce the bug
```python
from datasets import load_dataset
import datasets

ds = load_dataset("common_voice", "ab", split="train")

ds = ds.cast_column("audio", datasets.features.Audio(sampling_rate=16_000))

print(ds[0]["audio"])  # <- this fails currently
```

yields:

```
TypeError: forward() takes 2 positional arguments but 4 were given
```

## Expected results
no failure

## Actual results
Specify the actual results or traceback.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
Copy-and-paste the text below in your GitHub issue.

- `datasets` version: 1.13.2 (master)
- Platform: Linux-5.11.0-1019-aws-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 5.0.0

 Thanks for reporting, @patrickvonplaten.

I think the issue is related to mp3 resampling, not to `cast_column`.

You can check that `cast_column` works OK with non-mp3 audio files:
```python
from datasets import load_dataset
import datasets
ds = load_dataset("arabic_speech_corpus", split="train")
ds = ds.cast_column("audio", datasets.features.Audio(sampling_rate=16_000))
print(ds[0]["audio"])
```

I'm fixing it.


Error loading json dataset with multiple splits if keys in nested dicts have a different order
 ## Describe the bug
Loading a json dataset with multiple splits that have nested dicts with keys in different order results in the error below.

If the keys in the nested dicts always have the same order or even if you just load a single split in which the nested dicts don't have the same order, everything works fine.

## Steps to reproduce the bug
Create two json files:

train.json
```
{"a": {"c": 8, "b": 5}}
{"a": {"b": 7, "c": 6}}
```

test.json
```
{"a": {"b": 1, "c": 2}}
{"a": {"b": 3, "c": 4}}
```

```python
from datasets import load_dataset
# Loading the files individually works (even though the keys in train.json don't have the same order)
load_dataset('json', data_files={"test": "test.json"})
load_dataset('json', data_files={"train": "train.json"})
# Loading both splits fails
load_dataset('json', data_files={"train": "train.json", "test": "test.json"})
```

## Expected results
Loading both splits should not give an error whether the nested dicts are have the same order or not.

## Actual results
```
>>> load_dataset('json', data_files={"train": "train.json", "test": "test.json"})
Using custom data configuration default-f1bc76fd07398c4c
Downloading and preparing dataset json/default to /home/dthulke/.cache/huggingface/datasets/json/default-f1bc76fd07398c4c/0.0.0/c2d554c3377ea79c7664b93dc65d0803b45e3279000f993c7bfd18937fd7f426...
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 8839.42it/s]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 477.82it/s]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/dthulke/venvs/venv_torch_transformers/lib/python3.6/site-packages/datasets/load.py", line 1632, in load_dataset
    use_auth_token=use_auth_token,
  File "/home/dthulke/venvs/venv_torch_transformers/lib/python3.6/site-packages/datasets/builder.py", line 608, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home/dthulke/venvs/venv_torch_transformers/lib/python3.6/site-packages/datasets/builder.py", line 697, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/dthulke/venvs/venv_torch_transformers/lib/python3.6/site-packages/datasets/builder.py", line 1159, in _prepare_split
    writer.write_table(table)
  File "/home/dthulke/venvs/venv_torch_transformers/lib/python3.6/site-packages/datasets/arrow_writer.py", line 428, in write_table
    pa_table = pa.Table.from_arrays([pa_table[name] for name in self._schema.names], schema=self._schema)
  File "pyarrow/table.pxi", line 1596, in pyarrow.lib.Table.from_arrays
  File "pyarrow/table.pxi", line 592, in pyarrow.lib._sanitize_arrays
  File "pyarrow/array.pxi", line 329, in pyarrow.lib.asarray
  File "pyarrow/table.pxi", line 277, in pyarrow.lib.ChunkedArray.cast
  File "/home/dthulke/venvs/venv_torch_transformers/lib/python3.6/site-packages/pyarrow/compute.py", line 297, in cast
    return call_function("cast", [arr], options)
  File "pyarrow/_compute.pyx", line 527, in pyarrow._compute.call_function
  File "pyarrow/_compute.pyx", line 337, in pyarrow._compute.Function.call
  File "pyarrow/error.pxi", line 143, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 120, in pyarrow.lib.check_status
pyarrow.lib.ArrowNotImplementedError: Unsupported cast from struct<b: int64, c: int64> to struct using function cast_struct
```

## Environment info
- `datasets` version: 1.13.2
- Platform: Linux-4.15.0-147-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyArrow version: 5.0.0

 Hi,

even Pandas, which is less strict compared to PyArrow when it comes to reading JSON, doesn't support different orderings:
```python
import io
import pandas as pd

s = """
{"a": {"c": 8, "b": 5}}
{"a": {"b": 7, "c": 6}}
"""

buffer = io.StringIO(s)
df = pd.read_json(buffer, lines=True)

print(df.shape[0]) # 0
```

So we can't even fall back to Pandas in such cases.

It seems the only option is a script that recursively re-orders fields to enforce deterministic order:
```python
with open("train.json", "r") as fin:
    with open("train_reordered.json", "w") as fout:
        for line in fin:
            obj_jsonl = json.loads(line.strip())
            fout.write(json.dumps(obj_jsonl, sort_keys=True) + "\n")
```


`blog_authorship_corpus` is broken

`blog_authorship_corpus` is broken
 ## Describe the bug
The dataset `blog_authorship_corpus` is broken.
By bypassing the checksum checks, the loading does not return any error but the resulting dataset is empty.
I suspect it is because the data download url is broken (http://www.cs.biu.ac.il/~koppel/blogs/blogs.zip).

## Steps to reproduce the bug
```python
from datasets import load_dataset
ds = load_dataset("blog_authorship_corpus", split="train", download_mode='force_redownload')
```

## Expected results
No error.

## Actual results
```
---------------------------------------------------------------------------
NonMatchingChecksumError                  Traceback (most recent call last)
/tmp/ipykernel_5237/1729238701.py in <module>
      2 ds = load_dataset(
      3     "blog_authorship_corpus", split="train",
----> 4     download_mode='force_redownload'
      5 )

/opt/conda/lib/python3.7/site-packages/datasets/load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, script_version, use_auth_token, task, streaming, **config_kwargs)
   1115         ignore_verifications=ignore_verifications,
   1116         try_from_hf_gcs=try_from_hf_gcs,
-> 1117         use_auth_token=use_auth_token,
   1118     )
   1119

/opt/conda/lib/python3.7/site-packages/datasets/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, **download_and_prepare_kwargs)
    635                     if not downloaded_from_gcs:
    636                         self._download_and_prepare(
--> 637                             dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    638                         )
    639                     # Sync info

/opt/conda/lib/python3.7/site-packages/datasets/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    707         if verify_infos:
    708             verify_checksums(
--> 709                 self.info.download_checksums, dl_manager.get_recorded_sizes_checksums(), "dataset source files"
    710             )
    711

/opt/conda/lib/python3.7/site-packages/datasets/utils/info_utils.py in verify_checksums(expected_checksums, recorded_checksums, verification_name)
     38     if len(bad_urls) > 0:
     39         error_msg = "Checksums didn't match" + for_verification_name + ":\n"
---> 40         raise NonMatchingChecksumError(error_msg + str(bad_urls))
     41     logger.info("All the checksums matched successfully" + for_verification_name)
     42

NonMatchingChecksumError: Checksums didn't match for dataset source files:
['http://www.cs.biu.ac.il/~koppel/blogs/blogs.zip']
```

## Environment info
- `datasets` version: 1.13.2
- Platform: Linux-4.19.0-18-cloud-amd64-x86_64-with-debian-10.11
- Python version: 3.7.10
- PyArrow version: 5.0.0
 Hi @fdtomasi, thanks for reporting.

You are right: the original host data URL does no longer exist.

I've contacted the authors of the dataset to ask them if they host this dataset in another URL.


`blog_authorship_corpus` is broken

`blog_authorship_corpus` is broken
 ## Describe the bug
The dataset `blog_authorship_corpus` is broken.
By bypassing the checksum checks, the loading does not return any error but the resulting dataset is empty.
I suspect it is because the data download url is broken (http://www.cs.biu.ac.il/~koppel/blogs/blogs.zip).

## Steps to reproduce the bug
```python
from datasets import load_dataset
ds = load_dataset("blog_authorship_corpus", split="train", download_mode='force_redownload')
```

## Expected results
No error.

## Actual results
```
---------------------------------------------------------------------------
NonMatchingChecksumError                  Traceback (most recent call last)
/tmp/ipykernel_5237/1729238701.py in <module>
      2 ds = load_dataset(
      3     "blog_authorship_corpus", split="train",
----> 4     download_mode='force_redownload'
      5 )

/opt/conda/lib/python3.7/site-packages/datasets/load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, script_version, use_auth_token, task, streaming, **config_kwargs)
   1115         ignore_verifications=ignore_verifications,
   1116         try_from_hf_gcs=try_from_hf_gcs,
-> 1117         use_auth_token=use_auth_token,
   1118     )
   1119

/opt/conda/lib/python3.7/site-packages/datasets/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, **download_and_prepare_kwargs)
    635                     if not downloaded_from_gcs:
    636                         self._download_and_prepare(
--> 637                             dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    638                         )
    639                     # Sync info

/opt/conda/lib/python3.7/site-packages/datasets/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    707         if verify_infos:
    708             verify_checksums(
--> 709                 self.info.download_checksums, dl_manager.get_recorded_sizes_checksums(), "dataset source files"
    710             )
    711

/opt/conda/lib/python3.7/site-packages/datasets/utils/info_utils.py in verify_checksums(expected_checksums, recorded_checksums, verification_name)
     38     if len(bad_urls) > 0:
     39         error_msg = "Checksums didn't match" + for_verification_name + ":\n"
---> 40         raise NonMatchingChecksumError(error_msg + str(bad_urls))
     41     logger.info("All the checksums matched successfully" + for_verification_name)
     42

NonMatchingChecksumError: Checksums didn't match for dataset source files:
['http://www.cs.biu.ac.il/~koppel/blogs/blogs.zip']
```

## Environment info
- `datasets` version: 1.13.2
- Platform: Linux-4.19.0-18-cloud-amd64-x86_64-with-debian-10.11
- Python version: 3.7.10
- PyArrow version: 5.0.0
 Hi, @fdtomasi, the URL is fixed.

The fix is already in our master branch and it will be accessible in our next release.

In the meantime, you can include the fix if you install the `datasets` library from the master branch:
```
pip install -U git+ssh://git@github.com/huggingface/datasets.git@master#egg=datasest
```
or
```
pip install -U git+https://github.com/huggingface/datasets.git@master#egg=datasets
```


JNLPBA Dataset

JNLPBA Dataset
 ## Describe the bug
A clear and concise description of what the bug is.

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
```

## Expected results
The dataset loading script for this dataset is incorrect. This is a biomedical dataset used for named entity recognition. The entities in the [script](https://github.com/huggingface/datasets/blob/master/datasets/jnlpba/jnlpba.py#L81-L83) are: O, B, and I.  The correct entities from the original data file are:

['O',
 'B-DNA',
 'I-DNA',
 'B-RNA',
 'I-RNA',
 'B-cell_line',
 'I-cell_line',
 'B-cell_type',
 'I-cell_type',
 'B-protein',
 'I-protein']

## Actual results
The dataset loader script needs to include the following NER names:

['O',
 'B-DNA',
 'I-DNA',
 'B-RNA',
 'I-RNA',
 'B-cell_line',
 'I-cell_line',
 'B-cell_type',
 'I-cell_type',
 'B-protein',
 'I-protein']

And the [data](https://github.com/huggingface/datasets/blob/master/datasets/jnlpba/jnlpba.py#L46) that is being pulled has been modified from the original dataset and does not include the original NER tags.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- Platform:
- Python version:
- PyArrow version:

 # Steps to reproduce

To reproduce:

```python
from datasets import load_dataset
dataset = load_dataset('jnlpba')

dataset['train'].features['ner_tags']
```
Output:
```python
Sequence(feature=ClassLabel(num_classes=3, names=['O', 'B', 'I'], names_file=None, id=None), length=-1, id=None)
```


JNLPBA Dataset

JNLPBA Dataset
 ## Describe the bug
A clear and concise description of what the bug is.

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
```

## Expected results
The dataset loading script for this dataset is incorrect. This is a biomedical dataset used for named entity recognition. The entities in the [script](https://github.com/huggingface/datasets/blob/master/datasets/jnlpba/jnlpba.py#L81-L83) are: O, B, and I.  The correct entities from the original data file are:

['O',
 'B-DNA',
 'I-DNA',
 'B-RNA',
 'I-RNA',
 'B-cell_line',
 'I-cell_line',
 'B-cell_type',
 'I-cell_type',
 'B-protein',
 'I-protein']

## Actual results
The dataset loader script needs to include the following NER names:

['O',
 'B-DNA',
 'I-DNA',
 'B-RNA',
 'I-RNA',
 'B-cell_line',
 'I-cell_line',
 'B-cell_type',
 'I-cell_type',
 'B-protein',
 'I-protein']

And the [data](https://github.com/huggingface/datasets/blob/master/datasets/jnlpba/jnlpba.py#L46) that is being pulled has been modified from the original dataset and does not include the original NER tags.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- Platform:
- Python version:
- PyArrow version:

 Since I cannot create a branch here is the updated code:

```python

# coding=utf-8
# Copyright 2020 HuggingFace Datasets Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Lint as: python3
"""Introduction to the Bio-Entity Recognition Task at JNLPBA"""

import os

import datasets

logger = datasets.logging.get_logger(__name__)

_CITATION = """\
@inproceedings{kim2004introduction,
               title={Introduction to the bio-entity recognition task at JNLPBA},
               author={Kim, Jin-Dong and Ohta, Tomoko and Tsuruoka, Yoshimasa and Tateisi, Yuka and Collier, Nigel},
               booktitle={Proceedings of the international joint workshop on natural language processing in biomedicine and its applications},
               pages={70--75},
               year={2004},
               organization={Citeseer}
}
"""

_DESCRIPTION = """\
The data came from the GENIA version 3.02 corpus (Kim et al., 2003). This was formed from a controlled search
on MEDLINE using the MeSH terms human, blood cells and transcription factors. From this search 2,000 abstracts
were selected and hand annotated according to a small taxonomy of 48 classes based on a chemical classification.
Among the classes, 36 terminal classes were used to annotate the GENIA corpus.
"""

_HOMEPAGE = "http://www.geniaproject.org/shared-tasks/bionlp-jnlpba-shared-task-2004"
_TRAIN_URL = "http://www.nactem.ac.uk/GENIA/current/Shared-tasks/JNLPBA/Train/Genia4ERtraining.tar.gz"
_VAL_URL = 'http://www.nactem.ac.uk/GENIA/current/Shared-tasks/JNLPBA/Evaluation/Genia4ERtest.tar.gz'

_URLS = {
    "train": _TRAIN_URL,
    "val": _VAL_URL,
}

_TRAIN_DIRECTORY = "Genia4ERtraining"
_VAL_DIRECTORY = "Genia4ERtest"

_TRAIN_FILE = "Genia4ERtask1.iob2"
_VAL_FILE = "Genia4EReval1.iob2"

class JNLPBAConfig(datasets.BuilderConfig):
    """BuilderConfig for JNLPBA"""

    def __init__(self, **kwargs):
        """BuilderConfig for JNLPBA.
        Args:
          **kwargs: keyword arguments forwarded to super.
        """
        super(JNLPBAConfig, self).__init__(**kwargs)

class JNLPBA(datasets.GeneratorBasedBuilder):
    """JNLPBA dataset."""

    BUILDER_CONFIGS = [
        JNLPBAConfig(name="jnlpba", version=datasets.Version("1.0.0"), description="JNLPBA dataset"),
    ]

    def _info(self):
        return datasets.DatasetInfo(
            description=_DESCRIPTION,
            features=datasets.Features(
                {
                    "id": datasets.Value("string"),
                    "tokens": datasets.Sequence(datasets.Value("string")),
                    "ner_tags": datasets.Sequence(
                        datasets.features.ClassLabel(
                            names=[
                                'O',
                                'B-DNA',
                                'I-DNA',
                                'B-RNA',
                                'I-RNA',
                                'B-cell_line',
                                'I-cell_line',
                                'B-cell_type',
                                'I-cell_type',
                                'B-protein',
                                'I-protein',
                            ]
                        )
                    ),
                }
            ),
            supervised_keys=None,
            homepage=_HOMEPAGE,
            citation=_CITATION,
        )

    def _split_generators(self, dl_manager):
        downloaded_files = dl_manager.download_and_extract(_URLS)

        return [
            datasets.SplitGenerator(name=datasets.Split.TRAIN,
                                    gen_kwargs={"filepath": os.path.join(downloaded_files['train'], _TRAIN_FILE)}),
            datasets.SplitGenerator(name=datasets.Split.VALIDATION,
                                    gen_kwargs={"filepath": os.path.join(downloaded_files['val'], _VAL_FILE)})
        ]

    def _generate_examples(self, filepath):
        logger.info("⏳ Generating examples from = %s", filepath)
        with open(filepath, encoding="utf-8") as f:
            guid = 0
            tokens = []
            ner_tags = []
            for line in f:
                if line.startswith('###'):
                    continue
                if line == '' or line == '\n':
                    if tokens:
                        yield guid, {
                            "id": str(guid),
                            "tokens": tokens,
                            "ner_tags": ner_tags,
                        }
                        guid += 1
                        tokens = []
                        ner_tags = []
                else:
                    # tokens are tab separated
                    splits = line.split("\t")
                    #print(splits)
                    #print(len(splits))
                    if len(splits) < 2:
                        splits = splits[0].split()
                    tokens.append(splits[0])
                    ner_tags.append(splits[1].rstrip())
            # last example
            yield guid, {
                "id": str(guid),
                "tokens": tokens,
                "ner_tags": ner_tags,
            }
```


VisibleDeprecationWarning when using `set_format("numpy")`
 Code to reproduce:

```
from datasets import load_dataset
dataset = load_dataset("glue", "mnli")

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('distilbert-base-cased')

def tokenize_function(dataset):
    return tokenizer(dataset['premise'])

tokenized_datasets = dataset.map(tokenize_function, batched=True, remove_columns=dataset['train'].features)

tokenized_datasets.set_format("numpy")

tokenized_datasets['train'][5:8]
```
Outputs:

```
python3.9/site-packages/datasets/formatting/formatting.py:167: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
  return np.array(array, copy=False, **self.np_array_kwargs)
```
 I just opened a PR and I verified that the code you provided doesn't show any deprecation warning :)