500 internal server error when trying to open a dataset composed of Zarr stores
 ## Describe the bug
The dataset [openclimatefix/mrms](https://huggingface.co/datasets/openclimatefix/mrms) gives a 500 server error when trying to open it on the website, or through code.

The dataset doesn't have a loading script yet, and I did push two [xarray](https://docs.xarray.dev/en/stable/) Zarr stores of data there recentlyish. The Zarr stores are composed of lots of small files, which I am guessing is probably the problem, as we have another [OCF dataset](https://huggingface.co/datasets/openclimatefix/eumetsat_uk_hrv) using xarray and Zarr, but with the Zarr stored on GCP public datasets instead of directly in HF datasets, and that one opens fine.

In general, we were hoping to use HF datasets to release some more public geospatial datasets as benchmarks, which are commonly stored as Zarr stores as they can be compressed well and deal with the multi-dimensional data and coordinates fairly easily compared to other formats, but with this error, I'm assuming we should try a different format?

For context, we are trying to have complete public model+data reimplementations of some SOTA weather and solar nowcasting models, like [MetNet, MetNet-2,](https://github.com/openclimatefix/metnet) [DGMR](https://github.com/openclimatefix/skillful_nowcasting), and [others](https://github.com/openclimatefix/graph_weather), which all have large, complex datasets.

## Steps to reproduce the bug
```python
from datasets import load_dataset

dataset = load_dataset("openclimatefix/mrms")
```

## Expected results
The dataset should be downloaded or open up

## Actual results
A 500 internal server error

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3
- Platform: Linux-5.15.25-1-MANJARO-x86_64-with-glibc2.35
- Python version: 3.9.10
- PyArrow version: 7.0.0

 Hi @jacobbieker, thanks for reporting!

I have transferred this issue to our Hub team and they are investigating it. I keep you informed.


500 internal server error when trying to open a dataset composed of Zarr stores
 ## Describe the bug
The dataset [openclimatefix/mrms](https://huggingface.co/datasets/openclimatefix/mrms) gives a 500 server error when trying to open it on the website, or through code.

The dataset doesn't have a loading script yet, and I did push two [xarray](https://docs.xarray.dev/en/stable/) Zarr stores of data there recentlyish. The Zarr stores are composed of lots of small files, which I am guessing is probably the problem, as we have another [OCF dataset](https://huggingface.co/datasets/openclimatefix/eumetsat_uk_hrv) using xarray and Zarr, but with the Zarr stored on GCP public datasets instead of directly in HF datasets, and that one opens fine.

In general, we were hoping to use HF datasets to release some more public geospatial datasets as benchmarks, which are commonly stored as Zarr stores as they can be compressed well and deal with the multi-dimensional data and coordinates fairly easily compared to other formats, but with this error, I'm assuming we should try a different format?

For context, we are trying to have complete public model+data reimplementations of some SOTA weather and solar nowcasting models, like [MetNet, MetNet-2,](https://github.com/openclimatefix/metnet) [DGMR](https://github.com/openclimatefix/skillful_nowcasting), and [others](https://github.com/openclimatefix/graph_weather), which all have large, complex datasets.

## Steps to reproduce the bug
```python
from datasets import load_dataset

dataset = load_dataset("openclimatefix/mrms")
```

## Expected results
The dataset should be downloaded or open up

## Actual results
A 500 internal server error

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3
- Platform: Linux-5.15.25-1-MANJARO-x86_64-with-glibc2.35
- Python version: 3.9.10
- PyArrow version: 7.0.0

 Hi @jacobbieker, we are investigating this issue on our side and we'll see if we can fix it, but please note that your repo is considered problematic for git. Here are the results of running https://github.com/github/git-sizer on it:

```
Processing blobs: 147448
Processing trees: 27
Processing commits: 4
Matching commits to trees: 4
Processing annotated tags: 0
Processing references: 3
| Name                         | Value     | Level of concern               |
| ---------------------------- | --------- | ------------------------------ |
| Biggest objects              |           |                                |
| * Trees                      |           |                                |
|   * Maximum entries      [1] |   167 k   | !!!!!!!!!!!!!!!!!!!!!!!!!!!!!! |
|                              |           |                                |
| Biggest checkouts            |           |                                |
| * Number of files        [2] |   189 k   | ***                            |

[1]  aa057d2667c34c70c6146efc631f5c9917ff326e (refs/heads/main:2016.zarr/unknown)
[2]  6897b7bf6440fdd16b2c39d08085a669e7eaa59d (refs/heads/main^{tree})
```

You can check https://github.com/github/git-sizer for more information on how to avoid such pathological structures.


500 internal server error when trying to open a dataset composed of Zarr stores
 ## Describe the bug
The dataset [openclimatefix/mrms](https://huggingface.co/datasets/openclimatefix/mrms) gives a 500 server error when trying to open it on the website, or through code.

The dataset doesn't have a loading script yet, and I did push two [xarray](https://docs.xarray.dev/en/stable/) Zarr stores of data there recentlyish. The Zarr stores are composed of lots of small files, which I am guessing is probably the problem, as we have another [OCF dataset](https://huggingface.co/datasets/openclimatefix/eumetsat_uk_hrv) using xarray and Zarr, but with the Zarr stored on GCP public datasets instead of directly in HF datasets, and that one opens fine.

In general, we were hoping to use HF datasets to release some more public geospatial datasets as benchmarks, which are commonly stored as Zarr stores as they can be compressed well and deal with the multi-dimensional data and coordinates fairly easily compared to other formats, but with this error, I'm assuming we should try a different format?

For context, we are trying to have complete public model+data reimplementations of some SOTA weather and solar nowcasting models, like [MetNet, MetNet-2,](https://github.com/openclimatefix/metnet) [DGMR](https://github.com/openclimatefix/skillful_nowcasting), and [others](https://github.com/openclimatefix/graph_weather), which all have large, complex datasets.

## Steps to reproduce the bug
```python
from datasets import load_dataset

dataset = load_dataset("openclimatefix/mrms")
```

## Expected results
The dataset should be downloaded or open up

## Actual results
A 500 internal server error

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3
- Platform: Linux-5.15.25-1-MANJARO-x86_64-with-glibc2.35
- Python version: 3.9.10
- PyArrow version: 7.0.0

 Hi, thanks for getting back to me so quick! And yeah, I figured that was probably the problem. I was going to try to delete the repo, but couldn't through the website, so if that's the easiest way to solve it, I can regenerate the dataset in a different format with less tiny files, and you guys can delete the repo as it is. Zarr just saves everything as lots of small files to make chunks easy to load, which is why I was preferring that format, but maybne that just doesn't work well for HF datasets.


500 internal server error when trying to open a dataset composed of Zarr stores
 ## Describe the bug
The dataset [openclimatefix/mrms](https://huggingface.co/datasets/openclimatefix/mrms) gives a 500 server error when trying to open it on the website, or through code.

The dataset doesn't have a loading script yet, and I did push two [xarray](https://docs.xarray.dev/en/stable/) Zarr stores of data there recentlyish. The Zarr stores are composed of lots of small files, which I am guessing is probably the problem, as we have another [OCF dataset](https://huggingface.co/datasets/openclimatefix/eumetsat_uk_hrv) using xarray and Zarr, but with the Zarr stored on GCP public datasets instead of directly in HF datasets, and that one opens fine.

In general, we were hoping to use HF datasets to release some more public geospatial datasets as benchmarks, which are commonly stored as Zarr stores as they can be compressed well and deal with the multi-dimensional data and coordinates fairly easily compared to other formats, but with this error, I'm assuming we should try a different format?

For context, we are trying to have complete public model+data reimplementations of some SOTA weather and solar nowcasting models, like [MetNet, MetNet-2,](https://github.com/openclimatefix/metnet) [DGMR](https://github.com/openclimatefix/skillful_nowcasting), and [others](https://github.com/openclimatefix/graph_weather), which all have large, complex datasets.

## Steps to reproduce the bug
```python
from datasets import load_dataset

dataset = load_dataset("openclimatefix/mrms")
```

## Expected results
The dataset should be downloaded or open up

## Actual results
A 500 internal server error

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3
- Platform: Linux-5.15.25-1-MANJARO-x86_64-with-glibc2.35
- Python version: 3.9.10
- PyArrow version: 7.0.0

 Hi @jacobbieker,

For future use cases, our Hub team is still pondering whether to limit the maximum number of files per repo to avoid technical issues...

On the meantime, they have made a fix and your dataset is working: https://huggingface.co/datasets/openclimatefix/mrms


Add Biwi Kinect Head Pose Database

Add Biwi Kinect Head Pose Database
 ## Adding a Dataset
- **Name:** Biwi Kinect Head Pose Database
- **Description:** Over 15K images of 20 people recorded with a Kinect while turning their heads around freely. For each frame, depth and rgb images are provided, together with ground in the form of the 3D location of the head and its rotation angles.
- **Data:** [*link to the Github repository or current dataset location*](https://icu.ee.ethz.ch/research/datsets.html)
- **Motivation:** Useful pose estimation dataset

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

 Official dataset location : https://icu.ee.ethz.ch/research/datsets.html
In the "Biwi Kinect Head Pose Database" section, I do not find any information regarding "Downloading the dataset." . Do we mail the authors regarding this ?

I found the dataset on Kaggle : [Link](https://www.kaggle.com/kmader/biwi-kinect-head-pose-database) , but since 🤗 does not host any of the datasets, this would require the user to provide their Kaggle username and API key to download.

Any inputs on how we could proceed ? Thank you.
[ Need your inputs here, @lhoestq or @mariosasko ]


`pubmed_qa` checksum mismatch

`pubmed_qa` checksum mismatch
 ## Describe the bug
Loading [`pubmed_qa`](https://huggingface.co/datasets/pubmed_qa) results in a mismatched checksum error.

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
import datasets
try:
    datasets.load_dataset("pubmed_qa", "pqa_labeled")
except Exception as e:
    print(e)

try:
    datasets.load_dataset("pubmed_qa", "pqa_unlabeled")
except Exception as e:
    print(e)

try:
    datasets.load_dataset("pubmed_qa", "pqa_artificial")
except Exception as e:
    print(e)
```

## Expected results
Successful download.

## Actual results
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.9/site-packages/datasets/load.py", line 1702, in load_dataset
    builder_instance.download_and_prepare(
  File "/usr/local/lib/python3.9/site-packages/datasets/builder.py", line 594, in download_and_prepare
    self._download_and_prepare(
  File "/usr/local/lib/python3.9/site-packages/datasets/builder.py", line 665, in _download_and_prepare
    verify_checksums(
  File "/usr/local/lib/python3.9/site-packages/datasets/utils/info_utils.py", line 40, in verify_checksums
    raise NonMatchingChecksumError(error_msg + str(bad_urls))
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=1RsGLINVce-0GsDkCLDuLZmoLuzfmoCuQ', 'https://drive.google.com/uc?export=download&id=15v1x6aQDlZymaHGP7cZJZZYFfeJt2NdS']

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3
- Platform: macOS
- Python version: 3.8.1
- PyArrow version: 3.0.0

 Hi @jon-tow, thanks for reporting.

This issue was already reported and its root cause is a change in the Google Drive service. See:
- #3786

We have already fixed it. See:
- #3787

We are planning to make a patch release today.

In the meantime, you can get this fix by installing our library from the GitHub master branch:
```shell
pip install git+https://github.com/huggingface/datasets#egg=datasets
```
Then, if you had previously tried to load the data and got the checksum error, you should force the redownload of the data (before the fix, you just downloaded and cached the virus scan warning page, instead of the data file):
```shell
load_dataset("...", download_mode="force_redownload")
```


Support for "sources" parameter in the add() and add_batch() methods in datasets.metric - SARI
 **Is your feature request related to a problem? Please describe.**
The methods `add_batch` and `add` from the `Metric` [class](https://github.com/huggingface/datasets/blob/1675ad6a958435b675a849eafa8a7f10fe0f43bc/src/datasets/metric.py) does not work with [SARI](https://github.com/huggingface/datasets/blob/master/metrics/sari/sari.py) metric. This metric not only relies on the predictions and references, but also in the input.

For example, when the `add_batch` method is used, then the `compute()` method fails:
```
metric = load_metric("sari")
metric.add_batch(
    predictions=["About 95 you now get in ."],
    references=[["About 95 species are currently known .","About 95 species are now accepted .","95 species are now accepted ."]])
metric.compute()
> TypeError: _compute() missing 1 required positional argument: 'sources'
```

Therefore, the `compute() `method can only be used standalone:
```
metric = load_metric("sari")
result = metric.compute(
    sources=["About 95 species are currently accepted ."],
    predictions=["About 95 you now get in ."],
    references=[["About 95 species are currently known .","About 95 species are now accepted .","95 species are now accepted ."]])
> {'sari': 26.953601953601954}
```

**Describe the solution you'd like**
Support for an additional parameter `sources` in the `add_batch` and `add` of the `Metric` class.
```
add_batch(*, sources=None, predictions=None, references=None, **kwargs)
add(*, sources=None, predictions=None, references=None, **kwargs)
compute()
```

**Describe alternatives you've considered**
I've tried to override the `add_batch` and `add`, however, these are highly dependent to the `Metric` class. We could also write a simple function that compute the scores of a sentences list, but then we lose the functionality from the original [add](https://huggingface.co/docs/datasets/_modules/datasets/metric.html#Metric.add) and [add_batch method](https://huggingface.co/docs/datasets/_modules/datasets/metric.html#Metric.add_batch).

**Additional context**
These methods are used in the transformers [pytorch examples](https://github.com/huggingface/transformers/blob/master/examples/pytorch/summarization/run_summarization_no_trainer.py).

 Hi, thanks for reporting! We can add a `sources: datasets.Value("string")` feature to the `Features` dict in the `SARI` script to fix this. Would you be interested in submitting a PR?


Support for "sources" parameter in the add() and add_batch() methods in datasets.metric - SARI
 **Is your feature request related to a problem? Please describe.**
The methods `add_batch` and `add` from the `Metric` [class](https://github.com/huggingface/datasets/blob/1675ad6a958435b675a849eafa8a7f10fe0f43bc/src/datasets/metric.py) does not work with [SARI](https://github.com/huggingface/datasets/blob/master/metrics/sari/sari.py) metric. This metric not only relies on the predictions and references, but also in the input.

For example, when the `add_batch` method is used, then the `compute()` method fails:
```
metric = load_metric("sari")
metric.add_batch(
    predictions=["About 95 you now get in ."],
    references=[["About 95 species are currently known .","About 95 species are now accepted .","95 species are now accepted ."]])
metric.compute()
> TypeError: _compute() missing 1 required positional argument: 'sources'
```

Therefore, the `compute() `method can only be used standalone:
```
metric = load_metric("sari")
result = metric.compute(
    sources=["About 95 species are currently accepted ."],
    predictions=["About 95 you now get in ."],
    references=[["About 95 species are currently known .","About 95 species are now accepted .","95 species are now accepted ."]])
> {'sari': 26.953601953601954}
```

**Describe the solution you'd like**
Support for an additional parameter `sources` in the `add_batch` and `add` of the `Metric` class.
```
add_batch(*, sources=None, predictions=None, references=None, **kwargs)
add(*, sources=None, predictions=None, references=None, **kwargs)
compute()
```

**Describe alternatives you've considered**
I've tried to override the `add_batch` and `add`, however, these are highly dependent to the `Metric` class. We could also write a simple function that compute the scores of a sentences list, but then we lose the functionality from the original [add](https://huggingface.co/docs/datasets/_modules/datasets/metric.html#Metric.add) and [add_batch method](https://huggingface.co/docs/datasets/_modules/datasets/metric.html#Metric.add_batch).

**Additional context**
These methods are used in the transformers [pytorch examples](https://github.com/huggingface/transformers/blob/master/examples/pytorch/summarization/run_summarization_no_trainer.py).

 Hi Mario,

Thanks for your message. I did try to add `sources` into the `Features` dict using a script for the metric:
```
            features=datasets.Features(
                {
                    "sources": datasets.Value("string", id="sequence"),
                    "predictions": datasets.Value("string", id="sequence"),
                    "references": datasets.Sequence(datasets.Value("string", id="sequence"), id="references"),
                }
            ),
```

But that only avoids a failure in `encode_batch` in the `add_batch` method:
```
        batch = {"predictions": predictions, "references": references}
        batch = self.info.features.encode_batch(batch)
```

The real problem is that `add_batch()`, `add()` and `compute()` does not receive a `sources` param:
```
def add_batch(self, *, predictions=None, references=None):
def add(self, *, prediction=None, reference=None):
def compute(self, *, predictions=None, references=None, **kwargs)
```

And then, it fails:
`TypeError: add_batch() got an unexpected keyword argument sources`

I need this for adding any metric based on SARI or alike, not only for sari.py :)

Let me know if I understood correctly the proposed solution.


Support for "sources" parameter in the add() and add_batch() methods in datasets.metric - SARI
 **Is your feature request related to a problem? Please describe.**
The methods `add_batch` and `add` from the `Metric` [class](https://github.com/huggingface/datasets/blob/1675ad6a958435b675a849eafa8a7f10fe0f43bc/src/datasets/metric.py) does not work with [SARI](https://github.com/huggingface/datasets/blob/master/metrics/sari/sari.py) metric. This metric not only relies on the predictions and references, but also in the input.

For example, when the `add_batch` method is used, then the `compute()` method fails:
```
metric = load_metric("sari")
metric.add_batch(
    predictions=["About 95 you now get in ."],
    references=[["About 95 species are currently known .","About 95 species are now accepted .","95 species are now accepted ."]])
metric.compute()
> TypeError: _compute() missing 1 required positional argument: 'sources'
```

Therefore, the `compute() `method can only be used standalone:
```
metric = load_metric("sari")
result = metric.compute(
    sources=["About 95 species are currently accepted ."],
    predictions=["About 95 you now get in ."],
    references=[["About 95 species are currently known .","About 95 species are now accepted .","95 species are now accepted ."]])
> {'sari': 26.953601953601954}
```

**Describe the solution you'd like**
Support for an additional parameter `sources` in the `add_batch` and `add` of the `Metric` class.
```
add_batch(*, sources=None, predictions=None, references=None, **kwargs)
add(*, sources=None, predictions=None, references=None, **kwargs)
compute()
```

**Describe alternatives you've considered**
I've tried to override the `add_batch` and `add`, however, these are highly dependent to the `Metric` class. We could also write a simple function that compute the scores of a sentences list, but then we lose the functionality from the original [add](https://huggingface.co/docs/datasets/_modules/datasets/metric.html#Metric.add) and [add_batch method](https://huggingface.co/docs/datasets/_modules/datasets/metric.html#Metric.add_batch).

**Additional context**
These methods are used in the transformers [pytorch examples](https://github.com/huggingface/transformers/blob/master/examples/pytorch/summarization/run_summarization_no_trainer.py).

 The `Metric` class has been modified recently to support this use-case, but the `add_batch` + `compute` pattern still doesn't work correctly. I'll open a PR.


Add MetaShift dataset

Add MetaShift dataset
 ## Adding a Dataset
- **Name:** MetaShift
- **Description:**  collection of 12,868 sets of natural images across 410 classes-
- **Paper:** https://arxiv.org/abs/2202.06523v1
- **Data:** https://github.com/weixin-liang/metashift

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

 I would like to take this up and give it a shot. Any image specific - dataset guidelines to keep in mind ? Thank you.


Add MetaShift dataset

Add MetaShift dataset
 ## Adding a Dataset
- **Name:** MetaShift
- **Description:**  collection of 12,868 sets of natural images across 410 classes-
- **Paper:** https://arxiv.org/abs/2202.06523v1
- **Data:** https://github.com/weixin-liang/metashift

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

 I've started working on adding this dataset. I require some inputs on the following :

Ref for the initial draft [here](https://github.com/dnaveenr/datasets/blob/add_metashift_dataset/datasets/metashift/metashift.py)
1. The dataset does not have a typical - train/test/val split. What do we do for the _split_generators() function ? How do we go about this ?
2. This dataset builds on the Visual Genome dataset, using a metadata file. The dataset is generated using generate_full_MetaShift.py script. By default, the authors choose to generate the dataset only for a SELECTED_CLASSES. The following script is used :
Code : https://github.com/Weixin-Liang/MetaShift/blob/main/dataset/generate_full_MetaShift.py
Info : https://metashift.readthedocs.io/en/latest/sub_pages/download_MetaShift.html#generate-the-full-metashift-dataset
Can I just copy over the required functions into the metashift.py to generate the dataset ?
3. How do we complete the _generate_examples for this dataset ?

The user has the ability to use default selected classes, get the complete dataset or add more specific additional classes. I think config would be a good option here.

Inputs, suggestions would be helpful. Thank you.


Add MetaShift dataset

Add MetaShift dataset
 ## Adding a Dataset
- **Name:** MetaShift
- **Description:**  collection of 12,868 sets of natural images across 410 classes-
- **Paper:** https://arxiv.org/abs/2202.06523v1
- **Data:** https://github.com/weixin-liang/metashift

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

 Hi ! Thanks for adding this dataset :) Let me answer your questions:

1. in this case you can put everything in the "train" split
2. Yes you can copy the script (provided you also include the MIT license of the code in the file header for example). Though we ideally try to not create new directories nor files when generating dataset, so if possible this script should be adapted to not create the file structure they mentioned, but instead yield the images one by one in `_generate_examples`. Let me know if you think this is feasible
3. see point 2 haha

> The user has the ability to use default selected classes, get the complete dataset or add more specific additional classes. I think config would be a good option here.

Yup ! We can also define a `selected_classes` parameter such that users can do
```python
load_dataset("metashift", selected_classes=["cat", "dog", ...])
```


Add MetaShift dataset

Add MetaShift dataset
 ## Adding a Dataset
- **Name:** MetaShift
- **Description:**  collection of 12,868 sets of natural images across 410 classes-
- **Paper:** https://arxiv.org/abs/2202.06523v1
- **Data:** https://github.com/weixin-liang/metashift

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

 Great. This is helpful. Thanks @lhoestq .
Regarding Point 2, I'll try using yield instead of creating the directories and see if its feasible. selected_classes config sounds good.


Checksums didn't match for datasets on Google Drive
 ## Describe the bug

Datasets hosted on Google Drive do not seem to work right now.
Loading them fails with a checksum error.

## Steps to reproduce the bug
```python
from datasets import load_dataset
for dataset in ["head_qa", "yelp_review_full"]:
  try:
    load_dataset(dataset)
  except Exception as exception:
    print("Error", dataset, exception)
```

Here is a [colab](https://colab.research.google.com/drive/1wOtHBmL8I65NmUYakzPV5zhVCtHhi7uQ#scrollTo=cDzdCLlk-Bo4).

## Expected results

The datasets should be loaded.

## Actual results

```
Downloading and preparing dataset head_qa/es (download: 75.69 MiB, generated: 2.86 MiB, post-processed: Unknown size, total: 78.55 MiB) to /root/.cache/huggingface/datasets/head_qa/es/1.1.0/583ab408e8baf54aab378c93715fadc4d8aa51b393e27c3484a877e2ac0278e9...
Error head_qa Checksums didn't match for dataset source files:
['https://drive.google.com/u/0/uc?export=download&id=1a_95N5zQQoUCq8IBNVZgziHbeM-QxG2t']
Downloading and preparing dataset yelp_review_full/yelp_review_full (download: 187.06 MiB, generated: 496.94 MiB, post-processed: Unknown size, total: 684.00 MiB) to /root/.cache/huggingface/datasets/yelp_review_full/yelp_review_full/1.0.0/13c31a618ba62568ec8572a222a283dfc29a6517776a3ac5945fb508877dde43...
Error yelp_review_full Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=0Bz8a_Dbh9QhbZlU4dXhHTFhZQU0']
```

## Environment info
- `datasets` version: 1.18.3
- Platform: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyArrow version: 6.0.1

 Hi @muelletm, thanks for reporting.

This issue was already reported and its root cause is a change in the Google Drive service. See:
- #3786

We have already fixed it. See:
- #3787

Until our next `datasets` library release, you can get this fix by installing our library from the GitHub master branch:
```shell
pip install git+https://github.com/huggingface/datasets#egg=datasets
```
Then, if you had previously tried to load the data and got the checksum error, you should force the redownload of the data (before the fix, you just downloaded and cached the virus scan warning page, instead of the data file):
```shell
load_dataset("...", download_mode="force_redownload")
```


Pre-Processing Cache Fails when using a Factory pattern
 ## Describe the bug
If you utilize a pre-processing function which is created using a factory pattern, the function hash changes on each run (even if the function is identical) and therefore the data will be reproduced each time.

## Steps to reproduce the bug
```python
    def preprocess_function_factory(augmentation=None):
        def preprocess_function(examples):
            # Tokenize the texts
            if augmentation:
                conversions1 = [
                    augmentation(example)
                    for example in examples[sentence1_key]
                ]
                if sentence2_key is None:
                    args = (conversions1,)
                else:
                    conversions2 = [
                        augmentation(example)
                        for example in examples[sentence2_key]
                    ]
                    args = (conversions1, conversions2)
            else:
                args = (
                    (examples[sentence1_key],)
                    if sentence2_key is None
                    else (examples[sentence1_key], examples[sentence2_key])
                )
            result = tokenizer(
                *args, padding=padding, max_length=max_seq_length, truncation=True
            )

            # Map labels to IDs (not necessary for GLUE tasks)
            if label_to_id is not None and "label" in examples:
                result["label"] = [
                    (label_to_id[l] if l != -1 else -1) for l in examples["label"]
                ]
            return result

        return preprocess_function

capitalize = lambda x: x.capitalize()
preprocess_function = preprocess_function_factory(augmentation=capitalize)
print(hash(preprocess_function)) # This will change on each run
raw_datasets = raw_datasets.map(
        preprocess_function,
        batched=True,
        load_from_cache_file=True,
        desc="Running transformation and tokenizer on dataset",
)
```

## Expected results
Running the code twice will cause the cache to be re-used.

## Actual results
Running the code twice causes the whole dataset to be re-processed

 Ok - this is still an issue but I believe the root cause is different than I originally thought. I'm now able to get caching to work consistently with the above example as long as I fix the python hash seed `export PYTHONHASHSEED=1234`


Pre-Processing Cache Fails when using a Factory pattern
 ## Describe the bug
If you utilize a pre-processing function which is created using a factory pattern, the function hash changes on each run (even if the function is identical) and therefore the data will be reproduced each time.

## Steps to reproduce the bug
```python
    def preprocess_function_factory(augmentation=None):
        def preprocess_function(examples):
            # Tokenize the texts
            if augmentation:
                conversions1 = [
                    augmentation(example)
                    for example in examples[sentence1_key]
                ]
                if sentence2_key is None:
                    args = (conversions1,)
                else:
                    conversions2 = [
                        augmentation(example)
                        for example in examples[sentence2_key]
                    ]
                    args = (conversions1, conversions2)
            else:
                args = (
                    (examples[sentence1_key],)
                    if sentence2_key is None
                    else (examples[sentence1_key], examples[sentence2_key])
                )
            result = tokenizer(
                *args, padding=padding, max_length=max_seq_length, truncation=True
            )

            # Map labels to IDs (not necessary for GLUE tasks)
            if label_to_id is not None and "label" in examples:
                result["label"] = [
                    (label_to_id[l] if l != -1 else -1) for l in examples["label"]
                ]
            return result

        return preprocess_function

capitalize = lambda x: x.capitalize()
preprocess_function = preprocess_function_factory(augmentation=capitalize)
print(hash(preprocess_function)) # This will change on each run
raw_datasets = raw_datasets.map(
        preprocess_function,
        batched=True,
        load_from_cache_file=True,
        desc="Running transformation and tokenizer on dataset",
)
```

## Expected results
Running the code twice will cause the cache to be re-used.

## Actual results
Running the code twice causes the whole dataset to be re-processed

 Hi!

Yes, our hasher should work with decorators. For instance, this dummy example:
```python
def f(arg):
    def f1(ex):
        return {"a": ex["col1"] + arg}
    return f1
```
gives the same hash across different Python sessions (`datasets.fingerprint.Hasher.hash(f("string1")` returns `"408c9059f89dbd6c"` on my machine).

Could you please make the example self-contained? This way, we can reproduce the bug. Additionally, you can try to find the problematic object yourself by testing their hash with `datasets.fingerprint.Hasher.hash(obj)`

This could be related to https://github.com/huggingface/datasets/issues/3638.


NonMatchingChecksumError in xcopa dataset
 ## Describe the bug
Loading the xcopa dataset doesn't work, it fails due to a mismatch in the checksum.

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset("xcopa", "it")
```

## Expected results
The dataset should be loaded correctly.

## Actual results
Fails with:
```python
in verify_checksums(expected_checksums, recorded_checksums, verification_name)
     38     if len(bad_urls) > 0:
     39         error_msg = "Checksums didn't match" + for_verification_name + ":\n"
---> 40         raise NonMatchingChecksumError(error_msg + str(bad_urls))
     41     logger.info("All the checksums matched successfully" + for_verification_name)
     42

NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://github.com/cambridgeltl/xcopa/archive/master.zip']
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3, and 1.18.4.dev0
- Platform:
- Python version: 3.8
- PyArrow version:

 Hi @afcruzs-ms, thanks for opening this separate issue for your problem.

The root problem in the other issue (#3792) was a change in the service of Google Drive.

But in your case, the `xcopa` dataset is not hosted on Google Drive. Therefore, the root cause should be a different one.

Let me look at it...


NonMatchingChecksumError in xcopa dataset
 ## Describe the bug
Loading the xcopa dataset doesn't work, it fails due to a mismatch in the checksum.

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset("xcopa", "it")
```

## Expected results
The dataset should be loaded correctly.

## Actual results
Fails with:
```python
in verify_checksums(expected_checksums, recorded_checksums, verification_name)
     38     if len(bad_urls) > 0:
     39         error_msg = "Checksums didn't match" + for_verification_name + ":\n"
---> 40         raise NonMatchingChecksumError(error_msg + str(bad_urls))
     41     logger.info("All the checksums matched successfully" + for_verification_name)
     42

NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://github.com/cambridgeltl/xcopa/archive/master.zip']
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3, and 1.18.4.dev0
- Platform:
- Python version: 3.8
- PyArrow version:

 @afcruzs-ms, I'm not able to reproduce the issue you reported:
```python
In [1]: from datasets import load_dataset
   ...: dataset = load_dataset("xcopa", "it")
Downloading builder script: 5.21kB [00:00, 2.75MB/s]
Downloading metadata: 28.6kB [00:00, 14.5MB/s]
Downloading and preparing dataset xcopa/it (download: 627.09 KiB, generated: 76.43 KiB, post-processed: Unknown size, total: 703.52 KiB) to .../.cache/huggingface/datasets/xcopa/it/1.0.0/e1fab65f984b24c8b66bcf7ac27a26a1182f84adfb2e74035861be65e214b9e6...
Downloading data: 642kB [00:00, 5.42MB/s]
Dataset xcopa downloaded and prepared to .../.cache/huggingface/datasets/xcopa/it/1.0.0/e1fab65f984b24c8b66bcf7ac27a26a1182f84adfb2e74035861be65e214b9e6. Subsequent calls will reuse this data.
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 733.27it/s]

In [2]: dataset
Out[2]:
DatasetDict({
    test: Dataset({
        features: ['premise', 'choice1', 'choice2', 'question', 'label', 'idx', 'changed'],
        num_rows: 500
    })
    validation: Dataset({
        features: ['premise', 'choice1', 'choice2', 'question', 'label', 'idx', 'changed'],
        num_rows: 100
    })
})
```

Maybe you have some issue with your cached data... Could you please try to force the redownload of the data?
```python
dataset = load_dataset("xcopa", "it", download_mode="force_redownload")
```


Text builder with custom separator line boundaries
 **Is your feature request related to a problem? Please describe.**
The current [Text](https://github.com/huggingface/datasets/blob/207be676bffe9d164740a41a883af6125edef135/src/datasets/packaged_modules/text/text.py#L23) builder implementation splits texts with `splitlines()` which splits the text on several line boundaries. Not all of them are always wanted.

**Describe the solution you'd like**
```python
                if self.config.sample_by == "line":
                    batch_idx = 0
                    while True:
                        batch = f.read(self.config.chunksize)
                        if not batch:
                            break
                        batch += f.readline()  # finish current line
                        if self.config.custom_newline is None:
                            batch = batch.splitlines(keepends=self.config.keep_linebreaks)
                        else:
                            batch = batch.split(self.config.custom_newline)[:-1]
                        pa_table = pa.Table.from_arrays([pa.array(batch)], schema=schema)
                        # Uncomment for debugging (will print the Arrow table size and elements)
                        # logger.warning(f"pa_table: {pa_table} num rows: {pa_table.num_rows}")
                        # logger.warning('\n'.join(str(pa_table.slice(i, 1).to_pydict()) for i in range(pa_table.num_rows)))
                        yield (file_idx, batch_idx), pa_table
                        batch_idx += 1
```

**A clear and concise description of what you want to happen.**
Creating the dataset rows with a subset of the `splitlines()` line boundaries.
 Hi ! Interresting :)

Could you give more details on what kind of separators you would like to use instead ?


Text builder with custom separator line boundaries
 **Is your feature request related to a problem? Please describe.**
The current [Text](https://github.com/huggingface/datasets/blob/207be676bffe9d164740a41a883af6125edef135/src/datasets/packaged_modules/text/text.py#L23) builder implementation splits texts with `splitlines()` which splits the text on several line boundaries. Not all of them are always wanted.

**Describe the solution you'd like**
```python
                if self.config.sample_by == "line":
                    batch_idx = 0
                    while True:
                        batch = f.read(self.config.chunksize)
                        if not batch:
                            break
                        batch += f.readline()  # finish current line
                        if self.config.custom_newline is None:
                            batch = batch.splitlines(keepends=self.config.keep_linebreaks)
                        else:
                            batch = batch.split(self.config.custom_newline)[:-1]
                        pa_table = pa.Table.from_arrays([pa.array(batch)], schema=schema)
                        # Uncomment for debugging (will print the Arrow table size and elements)
                        # logger.warning(f"pa_table: {pa_table} num rows: {pa_table.num_rows}")
                        # logger.warning('\n'.join(str(pa_table.slice(i, 1).to_pydict()) for i in range(pa_table.num_rows)))
                        yield (file_idx, batch_idx), pa_table
                        batch_idx += 1
```

**A clear and concise description of what you want to happen.**
Creating the dataset rows with a subset of the `splitlines()` line boundaries.
 Ok I see, maybe there can be a `sep` parameter to allow users to specify what line/paragraph separator they'd like to use


Text builder with custom separator line boundaries
 **Is your feature request related to a problem? Please describe.**
The current [Text](https://github.com/huggingface/datasets/blob/207be676bffe9d164740a41a883af6125edef135/src/datasets/packaged_modules/text/text.py#L23) builder implementation splits texts with `splitlines()` which splits the text on several line boundaries. Not all of them are always wanted.

**Describe the solution you'd like**
```python
                if self.config.sample_by == "line":
                    batch_idx = 0
                    while True:
                        batch = f.read(self.config.chunksize)
                        if not batch:
                            break
                        batch += f.readline()  # finish current line
                        if self.config.custom_newline is None:
                            batch = batch.splitlines(keepends=self.config.keep_linebreaks)
                        else:
                            batch = batch.split(self.config.custom_newline)[:-1]
                        pa_table = pa.Table.from_arrays([pa.array(batch)], schema=schema)
                        # Uncomment for debugging (will print the Arrow table size and elements)
                        # logger.warning(f"pa_table: {pa_table} num rows: {pa_table.num_rows}")
                        # logger.warning('\n'.join(str(pa_table.slice(i, 1).to_pydict()) for i in range(pa_table.num_rows)))
                        yield (file_idx, batch_idx), pa_table
                        batch_idx += 1
```

**A clear and concise description of what you want to happen.**
Creating the dataset rows with a subset of the `splitlines()` line boundaries.
 Thanks for requesting this enhancement. We have recently found a somehow related issue with another dataset:
- #3704

Let me make a PR proposal.


Checksums didn't match for dataset source
 ## Dataset viewer issue for 'wiki_lingua*'

**Link:** *link to the dataset viewer page*

`data = datasets.load_dataset("wiki_lingua", name=language, split="train[:2000]")
`
*short description of the issue*
```
[NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=11wMGqNVSwwk6zUnDaJEgm3qT71kAHeff']]()
```
Am I the one who added this dataset ? No

 Same issue with `dataset = load_dataset("dbpedia_14")`
```
NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=0Bz8a_Dbh9QhbQ2Vic1kxMmZZQ1k']


Checksums didn't match for dataset source
 ## Dataset viewer issue for 'wiki_lingua*'

**Link:** *link to the dataset viewer page*

`data = datasets.load_dataset("wiki_lingua", name=language, split="train[:2000]")
`
*short description of the issue*
```
[NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=11wMGqNVSwwk6zUnDaJEgm3qT71kAHeff']]()
```
Am I the one who added this dataset ? No

 I think this is a side-effect of #3787. The checksums won't match because the URLs have changed. @rafikg @Y0mingZhang, while this is fixed, maybe you can load the datasets as such:

`data = datasets.load_dataset("wiki_lingua", name=language, split="train[:2000]", ignore_verifications=True)`
`dataset = load_dataset("dbpedia_14", ignore_verifications=True)`

This will, most probably, skip the verifications and integrity checks listed [here](https://huggingface.co/docs/datasets/loading_datasets.html#integrity-verifications)


Checksums didn't match for dataset source
 ## Dataset viewer issue for 'wiki_lingua*'

**Link:** *link to the dataset viewer page*

`data = datasets.load_dataset("wiki_lingua", name=language, split="train[:2000]")
`
*short description of the issue*
```
[NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=11wMGqNVSwwk6zUnDaJEgm3qT71kAHeff']]()
```
Am I the one who added this dataset ? No

 Hi! Installing the `datasets` package from master (`pip install git+https://github.com/huggingface/datasets.git`) and then redownloading the datasets with `download_mode` set to `force_redownload` (e.g. `dataset = load_dataset("dbpedia_14", download_mode="force_redownload")`) should fix the issue.


Checksums didn't match for dataset source
 ## Dataset viewer issue for 'wiki_lingua*'

**Link:** *link to the dataset viewer page*

`data = datasets.load_dataset("wiki_lingua", name=language, split="train[:2000]")
`
*short description of the issue*
```
[NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=11wMGqNVSwwk6zUnDaJEgm3qT71kAHeff']]()
```
Am I the one who added this dataset ? No

 Hi @rafikg and @Y0mingZhang, thanks for reporting.

Indeed it seems that Google Drive changed their way to access their data files. We have recently handled that change:
- #3787

but it will be accessible to users only in our next release of the `datasets` version.
- Note that our latest release (version 1.18.3) was made before this fix: https://github.com/huggingface/datasets/releases/tag/1.18.3

In the meantime, as @mariosasko explained, you can incorporate this "fix" by installing our library from the GitHub master branch:
```shell
pip install git+https://github.com/huggingface/datasets#egg=datasets
```
Then, you should force the redownload of the data (before the fix, you are just downloading/caching the virus scan warning page, instead of the data file):
```shell
data = datasets.load_dataset("wiki_lingua", name=language, split="train[:2000]", download_mode="force_redownload")


Checksums didn't match for dataset source
 ## Dataset viewer issue for 'wiki_lingua*'

**Link:** *link to the dataset viewer page*

`data = datasets.load_dataset("wiki_lingua", name=language, split="train[:2000]")
`
*short description of the issue*
```
[NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=11wMGqNVSwwk6zUnDaJEgm3qT71kAHeff']]()
```
Am I the one who added this dataset ? No

 @albertvillanova  by running:
```
pip install git+https://github.com/huggingface/datasets#egg=datasets
data = datasets.load_dataset("wiki_lingua", name=language, split="train[:2000]", download_mode="force_redownload", ignore_verifications=True)
```

I had a pickle error **UnpicklingError: invalid load key, '<'** in this part of code  both `locally and on google colab`:

```
"""Yields examples."""
with open(filepath, "rb") as f:
         data = pickle.load(f)
for id_, row in enumerate(data.items()):
         yield id_, {"url": row[0], "article": self._process_article(row[1])}
```


Checksums didn't match for dataset source
 ## Dataset viewer issue for 'wiki_lingua*'

**Link:** *link to the dataset viewer page*

`data = datasets.load_dataset("wiki_lingua", name=language, split="train[:2000]")
`
*short description of the issue*
```
[NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=11wMGqNVSwwk6zUnDaJEgm3qT71kAHeff']]()
```
Am I the one who added this dataset ? No

 This issue impacts many more datasets than the ones mention in this thread.  Can we post # of downloads for each dataset by day (by successes and failures)?  If so, it should be obvious which ones are failing.


Checksums didn't match for dataset source
 ## Dataset viewer issue for 'wiki_lingua*'

**Link:** *link to the dataset viewer page*

`data = datasets.load_dataset("wiki_lingua", name=language, split="train[:2000]")
`
*short description of the issue*
```
[NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=11wMGqNVSwwk6zUnDaJEgm3qT71kAHeff']]()
```
Am I the one who added this dataset ? No

 I can see this problem too in xcopa, unfortunately installing the latest master (1.18.4.dev0) doesn't work, @albertvillanova .

```
from datasets import load_dataset
dataset = load_dataset("xcopa", "it")
```

Throws

```
in verify_checksums(expected_checksums, recorded_checksums, verification_name)
     38     if len(bad_urls) > 0:
     39         error_msg = "Checksums didn't match" + for_verification_name + ":\n"
---> 40         raise NonMatchingChecksumError(error_msg + str(bad_urls))
     41     logger.info("All the checksums matched successfully" + for_verification_name)
     42

NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://github.com/cambridgeltl/xcopa/archive/master.zip']
```


Checksums didn't match for dataset source
 ## Dataset viewer issue for 'wiki_lingua*'

**Link:** *link to the dataset viewer page*

`data = datasets.load_dataset("wiki_lingua", name=language, split="train[:2000]")
`
*short description of the issue*
```
[NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=11wMGqNVSwwk6zUnDaJEgm3qT71kAHeff']]()
```
Am I the one who added this dataset ? No

 Hi @rafikg, I think that is another different issue. Let me check it...

I guess maybe you are using a different Python version that the one the dataset owner used to create the pickle file...


Checksums didn't match for dataset source
 ## Dataset viewer issue for 'wiki_lingua*'

**Link:** *link to the dataset viewer page*

`data = datasets.load_dataset("wiki_lingua", name=language, split="train[:2000]")
`
*short description of the issue*
```
[NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=11wMGqNVSwwk6zUnDaJEgm3qT71kAHeff']]()
```
Am I the one who added this dataset ? No

 @kwchurch the datasets impacted for this specific issue are the ones which are hosted at Google Drive.


Checksums didn't match for dataset source
 ## Dataset viewer issue for 'wiki_lingua*'

**Link:** *link to the dataset viewer page*

`data = datasets.load_dataset("wiki_lingua", name=language, split="train[:2000]")
`
*short description of the issue*
```
[NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=11wMGqNVSwwk6zUnDaJEgm3qT71kAHeff']]()
```
Am I the one who added this dataset ? No

 @afcruzs-ms I think your issue is a different one, because that dataset is not hosted at Google Drive. Would you mind open another issue for that other problem, please? Thanks! :)


Checksums didn't match for dataset source
 ## Dataset viewer issue for 'wiki_lingua*'

**Link:** *link to the dataset viewer page*

`data = datasets.load_dataset("wiki_lingua", name=language, split="train[:2000]")
`
*short description of the issue*
```
[NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=11wMGqNVSwwk6zUnDaJEgm3qT71kAHeff']]()
```
Am I the one who added this dataset ? No

 @albertvillanova just to let you know that I tried it locally and on colab and it is the same error


Checksums didn't match for dataset source
 ## Dataset viewer issue for 'wiki_lingua*'

**Link:** *link to the dataset viewer page*

`data = datasets.load_dataset("wiki_lingua", name=language, split="train[:2000]")
`
*short description of the issue*
```
[NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=11wMGqNVSwwk6zUnDaJEgm3qT71kAHeff']]()
```
Am I the one who added this dataset ? No

 There are many many datasets on HugggingFace that are receiving this checksum error.  Some of these datasets are very popular.  There must be a way to track these errors, or to do regression testing.  We don't want to catch each of these errors on each dataset, one at a time.


Checksums didn't match for dataset source
 ## Dataset viewer issue for 'wiki_lingua*'

**Link:** *link to the dataset viewer page*

`data = datasets.load_dataset("wiki_lingua", name=language, split="train[:2000]")
`
*short description of the issue*
```
[NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=11wMGqNVSwwk6zUnDaJEgm3qT71kAHeff']]()
```
Am I the one who added this dataset ? No

 @rafikg I am sorry, but I can't reproduce your issue. For me it works OK for all languages. See: https://colab.research.google.com/drive/1yIcLw1it118-TYE3ZlFmV7gJcsF6UCsH?usp=sharing


Checksums didn't match for dataset source
 ## Dataset viewer issue for 'wiki_lingua*'

**Link:** *link to the dataset viewer page*

`data = datasets.load_dataset("wiki_lingua", name=language, split="train[:2000]")
`
*short description of the issue*
```
[NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=11wMGqNVSwwk6zUnDaJEgm3qT71kAHeff']]()
```
Am I the one who added this dataset ? No

 @kwchurch the PR #3787 fixes this issue (generated by a change in Google Drive service) for ALL datasets with this issue. Once we make our next library release (in a couple of days), the fix will be accessible to all users that update our library from PyPI.


Checksums didn't match for dataset source
 ## Dataset viewer issue for 'wiki_lingua*'

**Link:** *link to the dataset viewer page*

`data = datasets.load_dataset("wiki_lingua", name=language, split="train[:2000]")
`
*short description of the issue*
```
[NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=11wMGqNVSwwk6zUnDaJEgm3qT71kAHeff']]()
```
Am I the one who added this dataset ? No

 By the way, @rafikg, I discovered the URL for Spanish was wrong. I've created a PR to fix it:
- #3806


Checksums didn't match for dataset source
 ## Dataset viewer issue for 'wiki_lingua*'

**Link:** *link to the dataset viewer page*

`data = datasets.load_dataset("wiki_lingua", name=language, split="train[:2000]")
`
*short description of the issue*
```
[NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=11wMGqNVSwwk6zUnDaJEgm3qT71kAHeff']]()
```
Am I the one who added this dataset ? No

 I have the same problem with "wider_face" dataset. It seems that "load_dataset" function can not download the dataset from google drive.


Only-data dataset loaded unexpectedly as validation split
 ## Describe the bug
As reported by @thomasw21 and @lhoestq, a dataset containing only a data file whose name matches the pattern `*dev*` will be returned as VALIDATION split, even if this is not the desired behavior, e.g. a file named `datosdevision.jsonl.gz`.
 I see two options:
1. drop the "dev" keyword since it can be considered too generic
2. improve the pattern to something more reasonable, e.g. asking for a separator before and after "dev"
```python
["*[ ._-]dev[ ._-]*", "dev[ ._-]*"]
```

I think 2. is nice. If we agree on this one we can even decide to require the separation for the other split keywords "train", "test" etc.


Only-data dataset loaded unexpectedly as validation split
 ## Describe the bug
As reported by @thomasw21 and @lhoestq, a dataset containing only a data file whose name matches the pattern `*dev*` will be returned as VALIDATION split, even if this is not the desired behavior, e.g. a file named `datosdevision.jsonl.gz`.
 Yes, I had something like that on mind: "dev" not being part of a word.
```
"[^a-zA-Z]dev[^a-zA-Z]"


Only-data dataset loaded unexpectedly as validation split
 ## Describe the bug
As reported by @thomasw21 and @lhoestq, a dataset containing only a data file whose name matches the pattern `*dev*` will be returned as VALIDATION split, even if this is not the desired behavior, e.g. a file named `datosdevision.jsonl.gz`.
 Is there a reason why we want that regex? It feels like something that'll still be an issue for some weird case. "my_dataset_dev" doesn't match your regex, "my_dataset_validation" doesn't either ... Why not always "train" unless specified?


Only-data dataset loaded unexpectedly as validation split
 ## Describe the bug
As reported by @thomasw21 and @lhoestq, a dataset containing only a data file whose name matches the pattern `*dev*` will be returned as VALIDATION split, even if this is not the desired behavior, e.g. a file named `datosdevision.jsonl.gz`.
 The regex is needed as part of our effort to make datasets configurable without code. In particular we define some generic dataset repository structures that users can follow

> ```
> "[^a-zA-Z]*dev[^a-zA-Z]*"
> ```

unfortunately our glob doesn't support "^":

https://github.com/fsspec/filesystem_spec/blob/3e739db7e53f5b408319dcc9d11e92bc1f938902/fsspec/spec.py#L465-L479


Only-data dataset loaded unexpectedly as validation split
 ## Describe the bug
As reported by @thomasw21 and @lhoestq, a dataset containing only a data file whose name matches the pattern `*dev*` will be returned as VALIDATION split, even if this is not the desired behavior, e.g. a file named `datosdevision.jsonl.gz`.
 > "my_dataset_dev" doesn't match your regex, "my_dataset_validation" doesn't either ... Why not always "train" unless specified?

And `my_dataset_dev.foo` would match the pattern, and we also have the same pattern but for the "validation" keyword so `my_dataset_validation.foo` would work too


Only-data dataset loaded unexpectedly as validation split
 ## Describe the bug
As reported by @thomasw21 and @lhoestq, a dataset containing only a data file whose name matches the pattern `*dev*` will be returned as VALIDATION split, even if this is not the desired behavior, e.g. a file named `datosdevision.jsonl.gz`.
 > The regex is needed as part of our effort to make datasets configurable without code

This feels like coding with the filename ^^'


Only-data dataset loaded unexpectedly as validation split
 ## Describe the bug
As reported by @thomasw21 and @lhoestq, a dataset containing only a data file whose name matches the pattern `*dev*` will be returned as VALIDATION split, even if this is not the desired behavior, e.g. a file named `datosdevision.jsonl.gz`.
 This is still much easier than having to write a full dataset script right ? :p


Bug downloading Virus scan warning page from Google Drive URLs
 ## Describe the bug
Recently, some issues were reported with URLs from Google Drive, where we were downloading the Virus scan warning page instead of the data file itself.

See:
- #3758
- #3773
- #3784

 Once the PR merged into master and until our next `datasets` library release, you can get this fix by installing our library from the GitHub master branch:
```shell
pip install git+https://github.com/huggingface/datasets#egg=datasets
```
Then, if you had previously tried to load the data and got the checksum error, you should force the redownload of the data (before the fix, you just downloaded and cached the virus scan warning page, instead of the data file):
```shell
load_dataset("...", download_mode="force_redownload")
```


Unable to Download CNN-Dailymail Dataset
 ## Describe the bug
I am unable to download the CNN-Dailymail dataset. Upon closer investigation, I realised why this was happening:
- The dataset sits in Google Drive, and both the CNN and DM datasets are large.
- Google is unable to scan the folder for viruses, **so the link which would originally download the dataset, now downloads the source code of this web page:**
![image](https://user-images.githubusercontent.com/58678541/155658435-c2f497d7-7601-4332-94b1-18a62dd96422.png)
- **This leads to the following error**:
```python
NotADirectoryError: [Errno 20] Not a directory: '/root/.cache/huggingface/datasets/downloads/1bc05d24fa6dda2468e83a73cf6dc207226e01e3c48a507ea716dc0421da583b/cnn/stories'
```

## Steps to reproduce the bug
```python
import datasets
dataset = datasets.load_dataset("cnn_dailymail", "3.0.0", split="train")
```

## Expected results
That the dataset is downloaded and processed just like other datasets.

## Actual results
Hit with this error:
```python
NotADirectoryError: [Errno 20] Not a directory: '/root/.cache/huggingface/datasets/downloads/1bc05d24fa6dda2468e83a73cf6dc207226e01e3c48a507ea716dc0421da583b/cnn/stories'
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3
- Platform: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyArrow version: 6.0.1

 Glad to help @albertvillanova! Just fine-tuning the PR, will comment once I am able to get it up and running 😀


Not be able to download dataset - "Newsroom"
 Hello,

I tried to download the **newsroom** dataset but it didn't work out for me. it said me to **download it manually**!

For manually, Link is also didn't work! It is sawing some ad or something!

If anybody has solved this issue please help me out or if somebody has this dataset please share your google drive link, it would be a great help!

Thanks
Darshan Tank
 Hi @Darshan2104, thanks for reporting.

Please note that at Hugging Face we do not host the data of this dataset, but just a loading script pointing to the host of the data owners.

Apparently the data owners changed their data host server. After googling it, I found their new website at: https://lil.nlp.cornell.edu/newsroom/index.html
- Download page: https://lil.nlp.cornell.edu/newsroom/download/index.html

I'm fixing the link in our Datasets library.


Allow download only some files from the Wikipedia dataset
 **Is your feature request related to a problem? Please describe.**
The Wikipedia dataset can be really big. This is a problem if you want to use it locally in a laptop with the Apache Beam `DirectRunner`. Even if your laptop have a considerable amount of memory (e.g. 32gb).

**Describe the solution you'd like**
I would like to use the `data_files` argument in the `load_dataset` function to define which file in the wikipedia dataset I would like to download. Thus, I can work with the dataset in a smaller machine using the Apache Beam `DirectRunner`.

**Describe alternatives you've considered**
I've tried to use the `simple` Wikipedia dataset. But it's in English and I would like to use Portuguese texts in my model.

 Hi @jvanz, thank you for your proposal.

In fact, we are aware that it is very common the problem you mention. Because of that, we are currently working in implementing a new version of wikipedia on the Hub, with all data preprocessed (no need to use Apache Beam), from where you will be able to use `data_files` to load only a specific subset of the data files.

See:
- #3401


Checksum mismatch for the reddit_tifu dataset
 ## Describe the bug
A checksum occurs when downloading the reddit_tifu data (both long & short).

## Steps to reproduce the bug
reddit_tifu_dataset = load_dataset('reddit_tifu', 'long')

## Expected results
The expected result is for the dataset to be downloaded and cached locally.

## Actual results
  File "/.../lib/python3.9/site-packages/datasets/utils/info_utils.py", line 40, in verify_checksums
    raise NonMatchingChecksumError(error_msg + str(bad_urls))
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=1ffWfITKFMJeqjT8loC8aiCLRNJpc_XnF']

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3
- Platform: Linux-5.13.0-30-generic-x86_64-with-glibc2.31
- Python version: 3.9.7
- PyArrow version: 7.0.0

 @albertvillanova Thank you for the fast response! However I am still getting the same error:

Downloading: 2.23kB [00:00, ?B/s]
Traceback (most recent call last):
  File "C:\Users\Anna\PycharmProjects\summarization\main.py", line 17, in <module>
    dataset = load_dataset('reddit_tifu', 'long')
  File "C:\Users\Anna\Desktop\summarization\summarization_env\lib\site-packages\datasets\load.py", line 1702, in load_dataset
    builder_instance.download_and_prepare(
  File "C:\Users\Anna\Desktop\summarization\summarization_env\lib\site-packages\datasets\builder.py", line 594, in download_and_prepare
    self._download_and_prepare(
  File "C:\Users\Anna\Desktop\summarization\summarization_env\lib\site-packages\datasets\builder.py", line 665, in _download_and_prepare
    verify_checksums(
  File "C:\Users\Anna\Desktop\summarization\summarization_env\lib\site-packages\datasets\utils\info_utils.py", line 40, in verify_checksums
    raise NonMatchingChecksumError(error_msg + str(bad_urls))
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=1ffWfITKFMJeqjT8loC8aiCLRNJpc_XnF']

I have cleaned the cache/huggingface/datasets & cache/huggingface/modules files and also tried on another machine with a fresh installation of trasnformers & datasets.
The reddit_tifu.py that gets downloaded still has the previous url on line 51, _URL = "https://drive.google.com/uc?export=download&id=1ffWfITKFMJeqjT8loC8aiCLRNJpc_XnF"


Checksum mismatch for the reddit_tifu dataset
 ## Describe the bug
A checksum occurs when downloading the reddit_tifu data (both long & short).

## Steps to reproduce the bug
reddit_tifu_dataset = load_dataset('reddit_tifu', 'long')

## Expected results
The expected result is for the dataset to be downloaded and cached locally.

## Actual results
  File "/.../lib/python3.9/site-packages/datasets/utils/info_utils.py", line 40, in verify_checksums
    raise NonMatchingChecksumError(error_msg + str(bad_urls))
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=1ffWfITKFMJeqjT8loC8aiCLRNJpc_XnF']

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3
- Platform: Linux-5.13.0-30-generic-x86_64-with-glibc2.31
- Python version: 3.9.7
- PyArrow version: 7.0.0

 Hi @anna-kay, I'm sorry I didn't clearly explain the details to you:
- the error has been fixed in our `master` branch on GitHub: https://github.com/huggingface/datasets/commit/8ae21bf6a77175dc803ce2f1b93d18b8fbf45586
- the fix will not be accessible to users in PyPI until our next release of the `datasets` library
  - our latest release (version 1.18.3) was made 23 days ago: https://github.com/huggingface/datasets/releases/tag/1.18.3
- in the meantime, you can get the fix if you install datasets from our GitHub `master` branch:
  ```
  pip install git+https://github.com/huggingface/datasets#egg=datasets
  ```


`dataset = dataset.map()` causes faiss index lost
 ## Describe the bug
assigning the resulted dataset to original dataset causes lost of the faiss index

## Steps to reproduce the bug
`my_dataset` is a regular loaded dataset. It's a part of a customed dataset structure
```python
self.dataset.add_faiss_index('embeddings')
self.dataset.list_indexes()
# ['embeddings']

dataset2 = my_dataset.map(
    lambda x: self._get_nearest_examples_batch(x['text']), batch=True
)

# the unexpected result:
dataset2.list_indexes()
# []

self.dataset.list_indexes()
# ['embeddings']
```

in case something wrong with my `_get_nearest_examples_batch()`, it's like this
```python
def _get_nearest_examples_batch(self, examples, k=5):
    queries = embed(examples)
    scores_batch, retrievals_batch = self.dataset.get_nearest_examples_batch(self.faiss_column, queries, k)
    return {
        'neighbors': [batch['text'] for batch in retrievals_batch],
        'scores': scores_batch
    }
```

## Expected results
`map` shouldn't drop the indexes, in another word, indexes should be carried to the generated dataset

## Actual results
map drops the indexes

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3
- Platform: Ubuntu 20.04.3 LTS
- Python version: 3.8.12
- PyArrow version: 7.0.0

 Hi ! Indeed `map` is dropping the index right now, because one can create a dataset with more or fewer rows using `map` (and therefore the index might not be relevant anymore)

I guess we could check the resulting dataset length, and if the user hasn't changed the dataset size we could keep the index, what do you think ?


It's not possible download `20200501.pt` dataset
 ## Describe the bug
The dataset `20200501.pt` is broken.

The available datasets: https://dumps.wikimedia.org/ptwiki/

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset("wikipedia", "20200501.pt", beam_runner='DirectRunner')
```

## Expected results
I expect to download the dataset locally.

## Actual results
```
>>> from datasets import load_dataset
>>> dataset = load_dataset("wikipedia", "20200501.pt", beam_runner='DirectRunner')
Downloading and preparing dataset wikipedia/20200501.pt to /home/jvanz/.cache/huggingface/datasets/wikipedia/20200501.pt/1.0.0/009f923d9b6dd00c00c8cdc7f408f2b47f45dd4f5fb7982a21f9448f4afbe475...
/home/jvanz/anaconda3/envs/tf-gpu/lib/python3.9/site-packages/apache_beam/__init__.py:79: UserWarning: This version of Apache Beam has not been sufficiently tested on Python 3.9. You may encounter bugs or missing features.
  warnings.warn(
  0%|                                                                                | 0/1 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jvanz/anaconda3/envs/tf-gpu/lib/python3.9/site-packages/datasets/load.py", line 1702, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/jvanz/anaconda3/envs/tf-gpu/lib/python3.9/site-packages/datasets/builder.py", line 594, in download_and_prepare
    self._download_and_prepare(
  File "/home/jvanz/anaconda3/envs/tf-gpu/lib/python3.9/site-packages/datasets/builder.py", line 1245, in _download_and_prepare
    super()._download_and_prepare(
  File "/home/jvanz/anaconda3/envs/tf-gpu/lib/python3.9/site-packages/datasets/builder.py", line 661, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/home/jvanz/.cache/huggingface/modules/datasets_modules/datasets/wikipedia/009f923d9b6dd00c00c8cdc7f408f2b47f45dd4f5fb7982a21f9448f4afbe475/wikipedia.py", line 420, in _split_generators
    downloaded_files = dl_manager.download_and_extract({"info": info_url})
  File "/home/jvanz/anaconda3/envs/tf-gpu/lib/python3.9/site-packages/datasets/utils/download_manager.py", line 307, in download_and_extract
    return self.extract(self.download(url_or_urls))
  File "/home/jvanz/anaconda3/envs/tf-gpu/lib/python3.9/site-packages/datasets/utils/download_manager.py", line 195, in download
    downloaded_path_or_paths = map_nested(
  File "/home/jvanz/anaconda3/envs/tf-gpu/lib/python3.9/site-packages/datasets/utils/py_utils.py", line 260, in map_nested
    mapped = [
  File "/home/jvanz/anaconda3/envs/tf-gpu/lib/python3.9/site-packages/datasets/utils/py_utils.py", line 261, in <listcomp>
    _single_map_nested((function, obj, types, None, True))
  File "/home/jvanz/anaconda3/envs/tf-gpu/lib/python3.9/site-packages/datasets/utils/py_utils.py", line 196, in _single_map_nested
    return function(data_struct)
  File "/home/jvanz/anaconda3/envs/tf-gpu/lib/python3.9/site-packages/datasets/utils/download_manager.py", line 216, in _download
    return cached_path(url_or_filename, download_config=download_config)
  File "/home/jvanz/anaconda3/envs/tf-gpu/lib/python3.9/site-packages/datasets/utils/file_utils.py", line 298, in cached_path
    output_path = get_from_cache(
  File "/home/jvanz/anaconda3/envs/tf-gpu/lib/python3.9/site-packages/datasets/utils/file_utils.py", line 612, in get_from_cache
    raise FileNotFoundError(f"Couldn't find file at {url}")
FileNotFoundError: Couldn't find file at https://dumps.wikimedia.org/ptwiki/20200501/dumpstatus.json

```

## Environment info
```
- `datasets` version: 1.18.3
- Platform: Linux-5.3.18-150300.59.49-default-x86_64-with-glibc2.31
- Python version: 3.9.7
- PyArrow version: 6.0.1
```
 Hi @jvanz, thanks for reporting.

Please note that Wikimedia website does not longer host Wikipedia dumps for so old dates.

For a list of accessible dump dates of `pt` Wikipedia, please see: https://dumps.wikimedia.org/ptwiki/

You can load for example `20220220` `pt` Wikipedia:
```python
dataset = load_dataset("wikipedia", language="pt", date="20220220", beam_runner="DirectRunner")
```


It's not possible download `20200501.pt` dataset
 ## Describe the bug
The dataset `20200501.pt` is broken.

The available datasets: https://dumps.wikimedia.org/ptwiki/

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset("wikipedia", "20200501.pt", beam_runner='DirectRunner')
```

## Expected results
I expect to download the dataset locally.

## Actual results
```
>>> from datasets import load_dataset
>>> dataset = load_dataset("wikipedia", "20200501.pt", beam_runner='DirectRunner')
Downloading and preparing dataset wikipedia/20200501.pt to /home/jvanz/.cache/huggingface/datasets/wikipedia/20200501.pt/1.0.0/009f923d9b6dd00c00c8cdc7f408f2b47f45dd4f5fb7982a21f9448f4afbe475...
/home/jvanz/anaconda3/envs/tf-gpu/lib/python3.9/site-packages/apache_beam/__init__.py:79: UserWarning: This version of Apache Beam has not been sufficiently tested on Python 3.9. You may encounter bugs or missing features.
  warnings.warn(
  0%|                                                                                | 0/1 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jvanz/anaconda3/envs/tf-gpu/lib/python3.9/site-packages/datasets/load.py", line 1702, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/jvanz/anaconda3/envs/tf-gpu/lib/python3.9/site-packages/datasets/builder.py", line 594, in download_and_prepare
    self._download_and_prepare(
  File "/home/jvanz/anaconda3/envs/tf-gpu/lib/python3.9/site-packages/datasets/builder.py", line 1245, in _download_and_prepare
    super()._download_and_prepare(
  File "/home/jvanz/anaconda3/envs/tf-gpu/lib/python3.9/site-packages/datasets/builder.py", line 661, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/home/jvanz/.cache/huggingface/modules/datasets_modules/datasets/wikipedia/009f923d9b6dd00c00c8cdc7f408f2b47f45dd4f5fb7982a21f9448f4afbe475/wikipedia.py", line 420, in _split_generators
    downloaded_files = dl_manager.download_and_extract({"info": info_url})
  File "/home/jvanz/anaconda3/envs/tf-gpu/lib/python3.9/site-packages/datasets/utils/download_manager.py", line 307, in download_and_extract
    return self.extract(self.download(url_or_urls))
  File "/home/jvanz/anaconda3/envs/tf-gpu/lib/python3.9/site-packages/datasets/utils/download_manager.py", line 195, in download
    downloaded_path_or_paths = map_nested(
  File "/home/jvanz/anaconda3/envs/tf-gpu/lib/python3.9/site-packages/datasets/utils/py_utils.py", line 260, in map_nested
    mapped = [
  File "/home/jvanz/anaconda3/envs/tf-gpu/lib/python3.9/site-packages/datasets/utils/py_utils.py", line 261, in <listcomp>
    _single_map_nested((function, obj, types, None, True))
  File "/home/jvanz/anaconda3/envs/tf-gpu/lib/python3.9/site-packages/datasets/utils/py_utils.py", line 196, in _single_map_nested
    return function(data_struct)
  File "/home/jvanz/anaconda3/envs/tf-gpu/lib/python3.9/site-packages/datasets/utils/download_manager.py", line 216, in _download
    return cached_path(url_or_filename, download_config=download_config)
  File "/home/jvanz/anaconda3/envs/tf-gpu/lib/python3.9/site-packages/datasets/utils/file_utils.py", line 298, in cached_path
    output_path = get_from_cache(
  File "/home/jvanz/anaconda3/envs/tf-gpu/lib/python3.9/site-packages/datasets/utils/file_utils.py", line 612, in get_from_cache
    raise FileNotFoundError(f"Couldn't find file at {url}")
FileNotFoundError: Couldn't find file at https://dumps.wikimedia.org/ptwiki/20200501/dumpstatus.json

```

## Environment info
```
- `datasets` version: 1.18.3
- Platform: Linux-5.3.18-150300.59.49-default-x86_64-with-glibc2.31
- Python version: 3.9.7
- PyArrow version: 6.0.1
```
 > ```python
> dataset = load_dataset("wikipedia", language="pt", date="20220220", beam_runner="DirectRunner")
> ```

Thank you! I did not know that I can do this. I was following the example in the error message when I do not define which language dataset I'm trying to download.

I've tried something similar changing the date in the `load_dataset` call that I've shared in the bug description. Obviously, it did not work. I need to read the docs more carefully next time. My bad!

Thanks again and sorry for the noise.


`Dataset.class_encode` should support custom class names
 I can make a PR, just wanted approval before starting.

**Is your feature request related to a problem? Please describe.**

It is often the case that classes are not ordered in alphabetical order. Current `class_encode_column` sort the classes before indexing.
https://github.com/huggingface/datasets/blob/master/src/datasets/arrow_dataset.py#L1235

**Describe the solution you'd like**
I would like to add a **optional** parameter `class_names` to `class_encode_column` that would  be used for the mapping instead of sorting the unique values.

**Describe alternatives you've considered**

One can use map instead. I find it harder to read.
```python
CLASS_NAMES = ['apple', 'orange', 'potato']
ds = ds.map(lambda item: CLASS_NAMES.index(item[label_column]))

# Proposition
ds = ds.class_encode_column(label_column, CLASS_NAMES)
```

**Additional context**

I can make the PR if this feature is accepted.

 Hi @Dref360, thanks a lot for your proposal.

It totally makes sense to have more flexibility when class encoding, I agree.

You could even further customize the class encoding by passing an instance of `ClassLabel` itself (instead of replicating `ClassLabel` instantiation arguments as `Dataset.class_encode_column` arguments).

And the latter made me think of `Dataset.cast_column`...

Maybe better to have some others' opinions @lhoestq @mariosasko


`Dataset.class_encode` should support custom class names
 I can make a PR, just wanted approval before starting.

**Is your feature request related to a problem? Please describe.**

It is often the case that classes are not ordered in alphabetical order. Current `class_encode_column` sort the classes before indexing.
https://github.com/huggingface/datasets/blob/master/src/datasets/arrow_dataset.py#L1235

**Describe the solution you'd like**
I would like to add a **optional** parameter `class_names` to `class_encode_column` that would  be used for the mapping instead of sorting the unique values.

**Describe alternatives you've considered**

One can use map instead. I find it harder to read.
```python
CLASS_NAMES = ['apple', 'orange', 'potato']
ds = ds.map(lambda item: CLASS_NAMES.index(item[label_column]))

# Proposition
ds = ds.class_encode_column(label_column, CLASS_NAMES)
```

**Additional context**

I can make the PR if this feature is accepted.

 Hi @Dref360! You can use [`Dataset.align_labels_with_mapping`](https://huggingface.co/docs/datasets/master/package_reference/main_classes.html#datasets.Dataset.align_labels_with_mapping) after `Dataset.class_encode_column` to assign a different mapping of labels to ids.

@albertvillanova I'd like to avoid adding more complexity to the API where it's not (absolutely) needed, so I don't think introducing a new param in `Dataset.class_encode_column` is a good idea.


Know your data for HF hub

Know your data for HF hub
 **Is your feature request related to a problem? Please describe.**
Would be great to see be able to understand datasets with the goal of improving data quality, and helping mitigate fairness and bias issues.

**Describe the solution you'd like**
Something like https://knowyourdata.withgoogle.com/ for HF hub
 Hi @Muhtasham you should take a look at https://huggingface.co/blog/data-measurements-tool and accompanying demo app at https://huggingface.co/spaces/huggingface/data-measurements-tool

We would be interested in your feedback. cc @meg-huggingface @sashavor @yjernite


Unable to view the Gradio flagged call back dataset
 ## Dataset viewer issue for '*savtadepth-flags*'

**Link:** *[savtadepth-flags](https://huggingface.co/datasets/kingabzpro/savtadepth-flags)*

*with the Gradio 2.8.1 the dataset viers stopped working. I tried to add values manually but its not working. The dataset is also not showing the link with the app https://huggingface.co/spaces/kingabzpro/savtadepth.*

Am I the one who added this dataset ? Yes

 Hi @kingabzpro.

I think you need to create a loading script that creates the dataset from the CSV file and the image paths.

As example, you could have a look at the Food-101 dataset: https://huggingface.co/datasets/food101
- Loading script: https://huggingface.co/datasets/food101/blob/main/food101.py

Once the loading script is created, the viewer will show a previsualization of your dataset.


Unable to view the Gradio flagged call back dataset
 ## Dataset viewer issue for '*savtadepth-flags*'

**Link:** *[savtadepth-flags](https://huggingface.co/datasets/kingabzpro/savtadepth-flags)*

*with the Gradio 2.8.1 the dataset viers stopped working. I tried to add values manually but its not working. The dataset is also not showing the link with the app https://huggingface.co/spaces/kingabzpro/savtadepth.*

Am I the one who added this dataset ? Yes

 @albertvillanova I don't think this is the issue. I have created another dataset with similar files and format and it works. https://huggingface.co/datasets/kingabzpro/savtadepth-flags-V2


Unable to view the Gradio flagged call back dataset
 ## Dataset viewer issue for '*savtadepth-flags*'

**Link:** *[savtadepth-flags](https://huggingface.co/datasets/kingabzpro/savtadepth-flags)*

*with the Gradio 2.8.1 the dataset viers stopped working. I tried to add values manually but its not working. The dataset is also not showing the link with the app https://huggingface.co/spaces/kingabzpro/savtadepth.*

Am I the one who added this dataset ? Yes

 Yes, you are right, that was not the issue.

Just take into account that sometimes the viewer can take some time until it shows the preview of the dataset.
After some time, yours is finally properly shown: https://huggingface.co/datasets/kingabzpro/savtadepth-flags


Unable to view the Gradio flagged call back dataset
 ## Dataset viewer issue for '*savtadepth-flags*'

**Link:** *[savtadepth-flags](https://huggingface.co/datasets/kingabzpro/savtadepth-flags)*

*with the Gradio 2.8.1 the dataset viers stopped working. I tried to add values manually but its not working. The dataset is also not showing the link with the app https://huggingface.co/spaces/kingabzpro/savtadepth.*

Am I the one who added this dataset ? Yes

 The problem was resolved by deleted the dataset and creating new one with similar name and then clicking on flag button.


head_qa file missing

head_qa file missing
 ## Describe the bug

A file for the `head_qa` dataset is missing (https://drive.google.com/u/0/uc?export=download&id=1a_95N5zQQoUCq8IBNVZgziHbeM-QxG2t/HEAD_EN/train_HEAD_EN.json)

## Steps to reproduce the bug
```python
>>> from datasets import load_dataset
>>> load_dataset("head_qa", name="en")
```

## Expected results

The dataset should be loaded

## Actual results

```
Downloading and preparing dataset head_qa/en (download: 75.69 MiB, generated: 2.69 MiB, post-processed: Unknown size, total: 78.38 MiB) to /home/slesage/.cache/huggingface/datasets/head_qa/en/1.1.0/583ab408e8baf54aab378c93715fadc4d8aa51b393e27c3484a877e2ac0278e9...
Downloading data: 2.21kB [00:00, 2.05MB/s]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/load.py", line 1729, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/builder.py", line 594, in download_and_prepare
    self._download_and_prepare(
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/builder.py", line 665, in _download_and_prepare
    verify_checksums(
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/utils/info_utils.py", line 40, in verify_checksums
    raise NonMatchingChecksumError(error_msg + str(bad_urls))
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/u/0/uc?export=download&id=1a_95N5zQQoUCq8IBNVZgziHbeM-QxG2t']
```

## Environment info
- `datasets` version: 1.18.4.dev0
- Platform: Linux-5.11.0-1028-aws-x86_64-with-glibc2.31
- Python version: 3.9.6
- PyArrow version: 6.0.1

 We usually find issues with files hosted at Google Drive...

In this case we download the Google Drive Virus scan warning instead of the data file.


Images get decoded when using `map()` with `input_columns` argument on a dataset
 ## Describe the bug

The `datasets.features.Image` feature class decodes image data by default. Expectedly, when indexing a dataset or using the `map()` method, images are returned as PIL Image instances.

However, when calling `map()` and setting a specific data column with the `input_columns` argument, the image data is passed as raw byte representation to the mapping function.

## Steps to reproduce the bug
```python
from datasets import load_dataset
from torchvision import transforms
from PIL.Image import Image

dataset = load_dataset('mnist', split='train')

def transform_all_columns(example):
    # example['image'] is encoded as PIL Image
    assert isinstance(example['image'], Image)
    return example

def transform_image_column(image):
    # image is decoded here and represented as raw bytes
    assert isinstance(image, Image)
    return image

# single-sample dataset for debugging purposes
dev = dataset.select([0])

dev.map(transform_all_columns)
dev.map(transform_image_column, input_columns='image')
```

## Expected results

Image data should be passed in decoded form, i.e. as PIL Image objects to the mapping function unless the `decode` attribute on the image feature is set to `False`.

## Actual results

The mapping function receives images as raw byte data.

## Environment info
- `datasets` version: 1.18.3
- Platform: Linux-5.11.0-49-generic-x86_64-with-glibc2.32
- Python version: 3.8.0b4
- PyArrow version: 7.0.0

 Hi! If I'm not mistaken, this behavior is intentional, but I agree it could be more intuitive.

@albertvillanova Do you remember why you decided not to decode columns in the `Audio` feature PR when `input_columns` is not `None`? IMO we should decode those columns, and we don't even have to use lazy structures here because the user explicitly requires them in the map transform.

cc @lhoestq for visibility


Images get decoded when using `map()` with `input_columns` argument on a dataset
 ## Describe the bug

The `datasets.features.Image` feature class decodes image data by default. Expectedly, when indexing a dataset or using the `map()` method, images are returned as PIL Image instances.

However, when calling `map()` and setting a specific data column with the `input_columns` argument, the image data is passed as raw byte representation to the mapping function.

## Steps to reproduce the bug
```python
from datasets import load_dataset
from torchvision import transforms
from PIL.Image import Image

dataset = load_dataset('mnist', split='train')

def transform_all_columns(example):
    # example['image'] is encoded as PIL Image
    assert isinstance(example['image'], Image)
    return example

def transform_image_column(image):
    # image is decoded here and represented as raw bytes
    assert isinstance(image, Image)
    return image

# single-sample dataset for debugging purposes
dev = dataset.select([0])

dev.map(transform_all_columns)
dev.map(transform_image_column, input_columns='image')
```

## Expected results

Image data should be passed in decoded form, i.e. as PIL Image objects to the mapping function unless the `decode` attribute on the image feature is set to `False`.

## Actual results

The mapping function receives images as raw byte data.

## Environment info
- `datasets` version: 1.18.3
- Platform: Linux-5.11.0-49-generic-x86_64-with-glibc2.32
- Python version: 3.8.0b4
- PyArrow version: 7.0.0

 I think I excluded to decorate the function when `input_columns` were passed as a quick fix for some non-passing tests:
- https://github.com/huggingface/datasets/pull/2324/commits/9d7c3e8fa53e23ec636859b4407eeec904b1b3f9

That PR was quite complex and I decided to focus on the main feature requests, leaving refinements for subsequent PRs.

Note that when `input_columns` are passed, the signature of the function is effectively changed, while the decorated function expects an item (whether an example or a batch) as first arg (which is not the case when passing `input_columns`.

I agree we should consider supporting the case when `input_columns` are passed.


Cannot preview dataset

Cannot preview dataset
 ## Dataset viewer issue for '*rubrix/news*'

**Link:https://huggingface.co/datasets/rubrix/news** *link to the dataset viewer page*

Cannot see the dataset preview:

```
Status code:   400
Exception:     Status400Error
Message:       Not found. Cache is waiting to be refreshed.
```
Am I the one who added this dataset ? No

 Thanks for reporting. The dataset viewer depends on some backend treatments, and for now, they might take some hours to get processed. We're working on improving it.


Expanding streaming capabilities

Expanding streaming capabilities
 Some ideas for a few features that could be useful when working with large datasets in streaming mode.

## `filter` for `IterableDataset`
Adding filtering to streaming datasets would be useful in several scenarios:
- filter a dataset with many languages for a subset of languages
- filter a dataset for specific licenses
- other custom logic to get a subset
The only way to achieve this at the moment is I think through writing a custom loading script and implementing filters there.

## `IterableDataset` to `Dataset` conversion
In combination with the above filter a functionality to "play" the whole stream would be useful. The motivation is that often one might filter the dataset to get a manageable size for experimentation. In that case streaming mode is no longer necessary as the filtered dataset is small enough and it would be useful to be able to play through the whole stream to create a normal `Dataset` with all its benefits.

```python
ds = load_dataset("some_large_dataset", streaming=True)
ds_filter = ds.filter(lambda x: x["lang"]="fr")
ds_filter = ds_filter.stream() # here the `IterableDataset` is converted to a `Dataset`
```
Naturally, this could be expanded with `stream(n=1000)` which creates a `Dataset` with the first `n` elements similar to `take`.

## Stream to the Hub
While streaming allows to use a dataset as is without saving the whole dataset on the local machine it is currently not possible to process a dataset and add it to the hub. The only way to do this is by downloading the full dataset and saving the processed dataset again before pushing them to the hub. The API could looks something like:

```python
ds = load_dataset("some_large_dataset", streaming=True)
ds_filter = ds.filter(some_filter_func)
ds_processed = ds_filter.map(some_processing_func)
ds_processed.push_to_hub("new_better_dataset", batch_size=100_000)
```

Under the hood this could be done by processing and aggregating `batch_size` elements and then pushing that batch as a single file to the hub. With this functionality one could process and create TB scale datasets while only requiring size of `batch_size` local disk space.

cc @lhoestq @albertvillanova
 Cool ! `filter` will be very useful. There can be a filter that you can apply on a streaming dataset:
```python
load_dataset(..., streaming=True).filter(lambda x: x["lang"] == "sw")
```

Otherwise if you want to apply a filter on the source files that are going to be used for streaming, the logic has to be impIemented directly in the dataset script, or if there's no dataset script this can be done with pattern matching
```python
load_dataset(..., lang="sw")                    # if the dataset script supports this parameter
load_dataset(..., data_files="data/lang=sw/*")  # if there's no dataset script, but only data files
```

--------------

Here are also some additional ideas of API to convert from iterable to map-style dataset:
```python
on_disk_dataset = streaming_dataset.to_disk()
on_disk_dataset = streaming_dataset.to_disk(path="path/to/my/dataset/dir")

in_memory_dataset = streaming_dataset.take(100).to_memory()  # to experiment without having to write files
```
--------------

Finally regarding `push_to_hub`, we can replace `batch_size` by `shard_size` (same API as for on-disk datasets). The default is 500MB per file

Let me know what you think !


Expanding streaming capabilities

Expanding streaming capabilities
 Some ideas for a few features that could be useful when working with large datasets in streaming mode.

## `filter` for `IterableDataset`
Adding filtering to streaming datasets would be useful in several scenarios:
- filter a dataset with many languages for a subset of languages
- filter a dataset for specific licenses
- other custom logic to get a subset
The only way to achieve this at the moment is I think through writing a custom loading script and implementing filters there.

## `IterableDataset` to `Dataset` conversion
In combination with the above filter a functionality to "play" the whole stream would be useful. The motivation is that often one might filter the dataset to get a manageable size for experimentation. In that case streaming mode is no longer necessary as the filtered dataset is small enough and it would be useful to be able to play through the whole stream to create a normal `Dataset` with all its benefits.

```python
ds = load_dataset("some_large_dataset", streaming=True)
ds_filter = ds.filter(lambda x: x["lang"]="fr")
ds_filter = ds_filter.stream() # here the `IterableDataset` is converted to a `Dataset`
```
Naturally, this could be expanded with `stream(n=1000)` which creates a `Dataset` with the first `n` elements similar to `take`.

## Stream to the Hub
While streaming allows to use a dataset as is without saving the whole dataset on the local machine it is currently not possible to process a dataset and add it to the hub. The only way to do this is by downloading the full dataset and saving the processed dataset again before pushing them to the hub. The API could looks something like:

```python
ds = load_dataset("some_large_dataset", streaming=True)
ds_filter = ds.filter(some_filter_func)
ds_processed = ds_filter.map(some_processing_func)
ds_processed.push_to_hub("new_better_dataset", batch_size=100_000)
```

Under the hood this could be done by processing and aggregating `batch_size` elements and then pushing that batch as a single file to the hub. With this functionality one could process and create TB scale datasets while only requiring size of `batch_size` local disk space.

cc @lhoestq @albertvillanova
 Regarding conversion, I'd also ask for some kind of equivalent to `save_to_disk` for an `IterableDataset`.

Similarly to the streaming to hub idea, my use case would be to define a sequence of dataset transforms via `.map()`, using an `IterableDataset` as the input (so processing could start without doing whole download up-front), but streaming the resultant processed dataset just to disk.


Expanding streaming capabilities

Expanding streaming capabilities
 Some ideas for a few features that could be useful when working with large datasets in streaming mode.

## `filter` for `IterableDataset`
Adding filtering to streaming datasets would be useful in several scenarios:
- filter a dataset with many languages for a subset of languages
- filter a dataset for specific licenses
- other custom logic to get a subset
The only way to achieve this at the moment is I think through writing a custom loading script and implementing filters there.

## `IterableDataset` to `Dataset` conversion
In combination with the above filter a functionality to "play" the whole stream would be useful. The motivation is that often one might filter the dataset to get a manageable size for experimentation. In that case streaming mode is no longer necessary as the filtered dataset is small enough and it would be useful to be able to play through the whole stream to create a normal `Dataset` with all its benefits.

```python
ds = load_dataset("some_large_dataset", streaming=True)
ds_filter = ds.filter(lambda x: x["lang"]="fr")
ds_filter = ds_filter.stream() # here the `IterableDataset` is converted to a `Dataset`
```
Naturally, this could be expanded with `stream(n=1000)` which creates a `Dataset` with the first `n` elements similar to `take`.

## Stream to the Hub
While streaming allows to use a dataset as is without saving the whole dataset on the local machine it is currently not possible to process a dataset and add it to the hub. The only way to do this is by downloading the full dataset and saving the processed dataset again before pushing them to the hub. The API could looks something like:

```python
ds = load_dataset("some_large_dataset", streaming=True)
ds_filter = ds.filter(some_filter_func)
ds_processed = ds_filter.map(some_processing_func)
ds_processed.push_to_hub("new_better_dataset", batch_size=100_000)
```

Under the hood this could be done by processing and aggregating `batch_size` elements and then pushing that batch as a single file to the hub. With this functionality one could process and create TB scale datasets while only requiring size of `batch_size` local disk space.

cc @lhoestq @albertvillanova
 That makes sense @athewsey , thanks for the suggestion :)

Maybe instead of the `to_disk` we could simply have `save_to_disk` instead:
```python
streaming_dataset.save_to_disk("path/to/my/dataset/dir")
on_disk_dataset = load_from_disk("path/to/my/dataset/dir")

in_memory_dataset = Dataset.from_list(list(streaming_dataset.take(100)))  # to experiment without having to write files
```