Problem downloading GEM wiki_auto_asset_turk dataset
 @yjernite

### Summary

I am currently working on the GEM datasets and do not manage to download the wiki_auto_asset_turk data, whereas all other datasets download well with the same code.

### Steps to reproduce
Code snippet:

from datasets import load_dataset
#dataset = load_dataset('gem', 'web_nlg_en')
dataset = load_dataset('gem', 'wiki_auto_asset_turk')

```

**Expected behavior:**

I expect the dataset to start downloading (download bar appears and progresses toward 100%)

**Actual behavior:**
Instead of seeing the download bar appearing, nothing happens; the following appears in the console as expected, but nothing more:

Downloading: 36.6kB [00:00, 37.2MB/s]
Downloading: 41.7kB [00:00, ?B/s]
Downloading and preparing dataset gem/wiki_auto_asset_turk (download: 121.37 MiB, generated: 145.69 MiB, post-processed: Unknown size, total: 267.07 MiB) to C:\Users\sfmil\.cache\huggingface\datasets\gem\wiki_auto_asset_turk\1.0.0\f252756d7f1b8f019aac71a1623b2950acfe10d25d956668ac4eae4e93c58b8d...

### Is this a regression?
No, it was the first time I was trying to download this dataset (same for the other ones).

### Debug info
- Python version: Python 3.8.2
- OS version: Windows 10 Family
 Hi,

sadly I can't replicate the problem on my Windows machine. Try to update the library to the newest version with:
```bash
pip install git+https://github.com/huggingface/datasets
```


Problem downloading GEM wiki_auto_asset_turk dataset
 @yjernite

### Summary

I am currently working on the GEM datasets and do not manage to download the wiki_auto_asset_turk data, whereas all other datasets download well with the same code.

### Steps to reproduce
Code snippet:

from datasets import load_dataset
#dataset = load_dataset('gem', 'web_nlg_en')
dataset = load_dataset('gem', 'wiki_auto_asset_turk')

```

**Expected behavior:**

I expect the dataset to start downloading (download bar appears and progresses toward 100%)

**Actual behavior:**
Instead of seeing the download bar appearing, nothing happens; the following appears in the console as expected, but nothing more:

Downloading: 36.6kB [00:00, 37.2MB/s]
Downloading: 41.7kB [00:00, ?B/s]
Downloading and preparing dataset gem/wiki_auto_asset_turk (download: 121.37 MiB, generated: 145.69 MiB, post-processed: Unknown size, total: 267.07 MiB) to C:\Users\sfmil\.cache\huggingface\datasets\gem\wiki_auto_asset_turk\1.0.0\f252756d7f1b8f019aac71a1623b2950acfe10d25d956668ac4eae4e93c58b8d...

### Is this a regression?
No, it was the first time I was trying to download this dataset (same for the other ones).

### Debug info
- Python version: Python 3.8.2
- OS version: Windows 10 Family
 Is there an error message ?
What stacktrace do you get if you interrupt the execution of the program while downloading ?


Problem downloading GEM wiki_auto_asset_turk dataset
 @yjernite

### Summary

I am currently working on the GEM datasets and do not manage to download the wiki_auto_asset_turk data, whereas all other datasets download well with the same code.

### Steps to reproduce
Code snippet:

from datasets import load_dataset
#dataset = load_dataset('gem', 'web_nlg_en')
dataset = load_dataset('gem', 'wiki_auto_asset_turk')

```

**Expected behavior:**

I expect the dataset to start downloading (download bar appears and progresses toward 100%)

**Actual behavior:**
Instead of seeing the download bar appearing, nothing happens; the following appears in the console as expected, but nothing more:

Downloading: 36.6kB [00:00, 37.2MB/s]
Downloading: 41.7kB [00:00, ?B/s]
Downloading and preparing dataset gem/wiki_auto_asset_turk (download: 121.37 MiB, generated: 145.69 MiB, post-processed: Unknown size, total: 267.07 MiB) to C:\Users\sfmil\.cache\huggingface\datasets\gem\wiki_auto_asset_turk\1.0.0\f252756d7f1b8f019aac71a1623b2950acfe10d25d956668ac4eae4e93c58b8d...

### Is this a regression?
No, it was the first time I was trying to download this dataset (same for the other ones).

### Debug info
- Python version: Python 3.8.2
- OS version: Windows 10 Family
 Sorry for the long time since my last comment, I tried again and don't seem to have the problem anymore, thanks for your support!


Problem downloading GEM wiki_auto_asset_turk dataset
 @yjernite

### Summary

I am currently working on the GEM datasets and do not manage to download the wiki_auto_asset_turk data, whereas all other datasets download well with the same code.

### Steps to reproduce
Code snippet:

from datasets import load_dataset
#dataset = load_dataset('gem', 'web_nlg_en')
dataset = load_dataset('gem', 'wiki_auto_asset_turk')

```

**Expected behavior:**

I expect the dataset to start downloading (download bar appears and progresses toward 100%)

**Actual behavior:**
Instead of seeing the download bar appearing, nothing happens; the following appears in the console as expected, but nothing more:

Downloading: 36.6kB [00:00, 37.2MB/s]
Downloading: 41.7kB [00:00, ?B/s]
Downloading and preparing dataset gem/wiki_auto_asset_turk (download: 121.37 MiB, generated: 145.69 MiB, post-processed: Unknown size, total: 267.07 MiB) to C:\Users\sfmil\.cache\huggingface\datasets\gem\wiki_auto_asset_turk\1.0.0\f252756d7f1b8f019aac71a1623b2950acfe10d25d956668ac4eae4e93c58b8d...

### Is this a regression?
No, it was the first time I was trying to download this dataset (same for the other ones).

### Debug info
- Python version: Python 3.8.2
- OS version: Windows 10 Family
 Great ! I'm closing the issue then. Feel free to re-open if you experience this issue again


Creating custom dataset results in error while calling the map() function
 calling `map()` of `datasets` library results into an error while defining a Custom dataset.
Reproducible example:
```
import datasets
class MyDataset(datasets.Dataset):

    def __init__(self, sentences):
        "Initialization"
        self.samples = sentences

    def __len__(self):
        "Denotes the total number of samples"
        return len(self.samples)

    def __getitem__(self, index):
        "Generates one sample of data"
        # Select sample
        # Load data and get label
        samples = self.samples[index]

        return samples

def preprocess_function_train(examples):
        inputs = examples
        labels = [example+tokenizer.eos_token for example in examples ]
        inputs = tokenizer(inputs, max_length=30, padding=True, truncation=True)
        labels = tokenizer(labels, max_length=30, padding=True, truncation=True)
        model_inputs = inputs
        model_inputs["labels"] = labels["input_ids"]
        print("about to return")
        return model_inputs

##train["sentence"] is dataframe column
train_dataset = MyDataset(train['sentence'].values.tolist())
train_dataset = train_dataset.map(
            preprocess_function,
            batched = True,
            batch_size=32
        )
```

Stack trace of error:
```
Traceback (most recent call last):
  File "dir/train_generate.py", line 362, in <module>
    main()
  File "dir/train_generate.py", line 245, in main
    train_dataset = train_dataset.map(
  File "anaconda_dir/anaconda3/envs/env1/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1244, in map
    return self._map_single(
  File "anaconda_dir/anaconda3/envs/env1/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 149, in wrapper
    unformatted_columns = set(self.column_names) - set(self._format_columns or [])
  File "anaconda_dir/anaconda3/envs/env1/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 526, in column_names
    return self._data.column_names
AttributeError: 'MyDataset' object has no attribute '_data'
```
 Hi,

the `_data` attribute is missing due to `MyDataset.__init__` not calling the parent `__init__`. However, I don't think it's a good idea to subclass the `datasets.Dataset` class (e.g. it's kind of dangerous to override `datasets.Dataset.__getitem__`). Instead, it's better to follow the "association over inheritance" approach with a simple wrapper class that delegates calls to a wrapped `Dataset` (map, etc.). Btw, the library offers the `datasets.Dataset.from_pandas` class method to directly create a `datasets.Dataset` from the dataframe.


WMT19 Dataset for Kazakh-English is not formatted correctly
 In addition to the bug of languages being switched from Issue @415, there are incorrect translations in the dataset because the English-Kazakh translations have a one off formatting error.

The News Commentary v14 parallel data set for kk-en from http://www.statmt.org/wmt19/translation-task.html has a bug here:

> Line 94. The Swiss National Bank, for its part, has been battling with the deflationary effects of the franc’s dramatic appreciation over the past few years.	Швейцарияның Ұлттық банкі өз тарапынан, соңғы бірнеше жыл ішінде франк құнының қатты өсуінің дефляциялық әсерімен күресіп келеді.
>
> Line 95. Дефляциялық күштер 2008 жылы терең және ұзаққа созылған жаһандық дағдарысқа байланысты орын алған ірі экономикалық және қаржылық орын алмасулардың арқасында босатылды.  Жеке қарыз қаражаты үлесінің қысқаруы орталық банктің рефляцияға жұмсалған күш-жігеріне тұрақты соққан қарсы желдей болды.
>
> Line 96. The deflationary forces were unleashed by the major economic and financial dislocations associated with the deep and protracted global crisis that erupted in 2008. Private deleveraging became a steady headwind to central bank efforts to reflate.	2009 жылы, алдыңғы қатарлы экономикалардың шамамен үштен бірі бағаның төмендеуін көрсетті, бұл соғыстан кейінгі жоғары деңгей болды.

As you can see, line 95 has only the Kazakh translation which should be part of line 96. This causes all of the following English-Kazakh translation pairs to be one off rendering ALL of those translations incorrect. This issue was not fixed when the dataset was imported to Huggingface. By running this code

```
import datasets
from datasets import load_dataset
dataset = load_dataset('wmt19', 'kk-en')
for key in dataset['train']['translation']:
    if 'The deflationary forces were unleashed by the major economic and financial dislocations associated with the deep and protracted global crisis that erupted in 2008.' in key['kk']:
        print(key['en'])
        print(key['kk'])
        break
```
we get:
> 2009 жылы, алдыңғы қатарлы экономикалардың шамамен үштен бірі бағаның төмендеуін көрсетті, бұл соғыстан кейінгі жоғары деңгей болды.
> The deflationary forces were unleashed by the major economic and financial dislocations associated with the deep and protracted global crisis that erupted in 2008. Private deleveraging became a steady headwind to central bank efforts to reflate.

which shows that the issue still persists in the Huggingface dataset. The Kazakh sentence matches up to the next English sentence in the dataset instead of the current one.

Please let me know if there's you have any ideas to fix this one-off error from the dataset or if this can be fixed by Huggingface.
 Hi ! Thanks for reporting

By looking at the raw `news-commentary-v14.en-kk.tsv` file, it looks like there are at least 17 lines with this issue.
Moreover these issues are not always the same:
- L97 is only `kk` text and must be appended at the end of the `kk` text of the **next** line
- L2897 is only `kk` text and must be appended at the end of the `kk` text of the **previous** line
- L1247 and L1248 are only `kk` texts and must be inserted at the **beginning** of the `kk` text of the next line
- (and there are many others)

It would be nice to have a corrected version of this file ! The file is available in the `wmt/news-commentary` repository on the Datasets Hub here:
https://huggingface.co/datasets/wmt/news-commentary/tree/main/v14/training

Then maybe we can notify the WMT authors and host the corrected version somewhere


Request to remove S2ORC dataset

Request to remove S2ORC dataset
 Hi!  I was wondering if it's possible to remove [S2ORC](https://huggingface.co/datasets/s2orc) from hosting on Huggingface's platform?  Unfortunately, there are some legal considerations about how we make this data available.  Happy to add back to Huggingface's platform once we work out those hurdles!  Thanks!
 Hello @kyleclo! Currently, we are getting the data from your bucket, so if you remove it the HF script won't work anymore :)

Until you solve things on your end, @lhoestq suggested we just return a warning message when people try to load that dataset from HF. What would you like it to say?


Request to remove S2ORC dataset

Request to remove S2ORC dataset
 Hi!  I was wondering if it's possible to remove [S2ORC](https://huggingface.co/datasets/s2orc) from hosting on Huggingface's platform?  Unfortunately, there are some legal considerations about how we make this data available.  Happy to add back to Huggingface's platform once we work out those hurdles!  Thanks!
 Hi @kyleclo, as of today, you have not removed your bucket data yet, and therefore HuggingFace can download it from there.

Is it OK? Are you planning to eventually delete it? Thank you.


Request to remove S2ORC dataset

Request to remove S2ORC dataset
 Hi!  I was wondering if it's possible to remove [S2ORC](https://huggingface.co/datasets/s2orc) from hosting on Huggingface's platform?  Unfortunately, there are some legal considerations about how we make this data available.  Happy to add back to Huggingface's platform once we work out those hurdles!  Thanks!
 Hi!  Sorry I missed @yjernite 's previous message, thanks for responding!

Is there an option where we can keep our data in our bucket, but the HF script no longer pulls data from it?


Trouble loading wiki_movies

Trouble loading wiki_movies
 Hello,
I am trying to load_dataset("wiki_movies") and it gives me this error -

`FileNotFoundError: Couldn't find file locally at wiki_movies/wiki_movies.py, or remotely at https://raw.githubusercontent.com/huggingface/datasets/1.1.2/datasets/wiki_movies/wiki_movies.py or https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/wiki_movies/wiki_movies.py`

Trying to do `python run_mlm.py \
    --model_name_or_path roberta-base \
    --dataset_name wiki_movies \` also gives the same error.

Is this something on my end? From what I can tell, this dataset was re-added by @lhoestq a few months ago.
Thank you!
 Hi ! `wiki_movies` was added in `datasets==1.2.0`. However it looks like you have `datasets==1.1.2`.

To use `wiki_movies`, please update `datasets` with
```
pip install --upgrade datasets
```


Trouble loading wiki_movies

Trouble loading wiki_movies
 Hello,
I am trying to load_dataset("wiki_movies") and it gives me this error -

`FileNotFoundError: Couldn't find file locally at wiki_movies/wiki_movies.py, or remotely at https://raw.githubusercontent.com/huggingface/datasets/1.1.2/datasets/wiki_movies/wiki_movies.py or https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/wiki_movies/wiki_movies.py`

Trying to do `python run_mlm.py \
    --model_name_or_path roberta-base \
    --dataset_name wiki_movies \` also gives the same error.

Is this something on my end? From what I can tell, this dataset was re-added by @lhoestq a few months ago.
Thank you!
 Thanks a lot! That solved it and I was able to upload a model trained on it as well :)


citation, homepage, and license fields of `dataset_info.json` are duplicated many times
 This happens after a `map` operation when `num_proc` is set to `>1`. I tested this by cleaning up the json before running the `map` op on the dataset so it's unlikely it's coming from an earlier concatenation.

Example result:
```
"citation": "@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n    url    = {https://dumps.wikimedia.org}\n}\n\n@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n    url    = {https://dumps.wikimedia.org}\n}\n\n@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n    url    = {https://dumps.wikimedia.org}\n}\n\n@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n    url    = {https://dumps.wikimedia.org}\n}\n\n@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n    url    = {https://dumps.wikimedia.org}\n}\n\n@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n    url    = {https://dumps.wikimedia.org}\n}\n\n@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n    url    = {https://dumps.wikimedia.org}\n}\n\n@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n    url    = {https://dumps.wikimedia.org}\n}\n\n@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n    url    = {https://dumps.wikimedia.org}\n}\n\n@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n    url    = {https://dumps.wikimedia.org}\n}\n\n@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n    url    = {https://dumps.wikimedia.org}\n}\n\n@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n
```

@lhoestq and I believe this is happening due to the fields being concatenated `num_proc` times.
 Thanks for reporting :)
Maybe we can concatenate fields only if they are different.

Currently this is done here:

https://github.com/huggingface/nlp/blob/349ac4398a3bcae6356f14c5754483383a60e8a4/src/datasets/info.py#L180-L196

This can be a good first contribution to the library.
Please comment if you'd like to improve this and open a PR :)


load_from_disk takes a long time to load local dataset
 I have an extremely large tokenized dataset (24M examples) that loads in a few minutes. However, after adding a column similar to `input_ids` (basically a list of integers) and saving the dataset to disk, the load time goes to >1 hour. I've even tried using `np.uint8` after seeing #1985 but it doesn't seem to be helping (the total size seems to be smaller though).

Does anyone know what could be the issue? Or does the casting of that column to `int8` need to happen in the function that writes the arrow table instead of in the `map` where I create the list of integers?

Tagging @lhoestq since you seem to be working on these issues and PRs :)
 Hi !
Can you share more information about the features of your dataset ? You can get them by printing `my_dataset.features`
Can you also share the code of your `map` function ?


load_from_disk takes a long time to load local dataset
 I have an extremely large tokenized dataset (24M examples) that loads in a few minutes. However, after adding a column similar to `input_ids` (basically a list of integers) and saving the dataset to disk, the load time goes to >1 hour. I've even tried using `np.uint8` after seeing #1985 but it doesn't seem to be helping (the total size seems to be smaller though).

Does anyone know what could be the issue? Or does the casting of that column to `int8` need to happen in the function that writes the arrow table instead of in the `map` where I create the list of integers?

Tagging @lhoestq since you seem to be working on these issues and PRs :)
 It is actually just the tokenized `wikipedia` dataset with `input_ids`, `attention_mask`, etc, with one extra column which is a list of integers. The `text` column is removed during tokenization.

```
def add_len_and_seq(example):
    end_idx = example['input_ids'].index(SEP)
    example['actual_len'] = end_idx-1
    seq_len = len(example['input_ids'])

    example['seq'] = [PAD_ID] + [np.uint8(example['some_integer'])]*(end_idx-1) + [PAD_ID]*(seq_len-end_idx)

    return example
```


load_from_disk takes a long time to load local dataset
 I have an extremely large tokenized dataset (24M examples) that loads in a few minutes. However, after adding a column similar to `input_ids` (basically a list of integers) and saving the dataset to disk, the load time goes to >1 hour. I've even tried using `np.uint8` after seeing #1985 but it doesn't seem to be helping (the total size seems to be smaller though).

Does anyone know what could be the issue? Or does the casting of that column to `int8` need to happen in the function that writes the arrow table instead of in the `map` where I create the list of integers?

Tagging @lhoestq since you seem to be working on these issues and PRs :)
 Is `PAD_ID` a python integer ? You need all the integers in `example['seq']` to have the same type.
Does this work if you remove the `np.uint8` and use python integers instead ?


load_from_disk takes a long time to load local dataset
 I have an extremely large tokenized dataset (24M examples) that loads in a few minutes. However, after adding a column similar to `input_ids` (basically a list of integers) and saving the dataset to disk, the load time goes to >1 hour. I've even tried using `np.uint8` after seeing #1985 but it doesn't seem to be helping (the total size seems to be smaller though).

Does anyone know what could be the issue? Or does the casting of that column to `int8` need to happen in the function that writes the arrow table instead of in the `map` where I create the list of integers?

Tagging @lhoestq since you seem to be working on these issues and PRs :)
 yup I casted it to `np.uint8` outside the function where it was defined. It was originally using python integers.


load_from_disk takes a long time to load local dataset
 I have an extremely large tokenized dataset (24M examples) that loads in a few minutes. However, after adding a column similar to `input_ids` (basically a list of integers) and saving the dataset to disk, the load time goes to >1 hour. I've even tried using `np.uint8` after seeing #1985 but it doesn't seem to be helping (the total size seems to be smaller though).

Does anyone know what could be the issue? Or does the casting of that column to `int8` need to happen in the function that writes the arrow table instead of in the `map` where I create the list of integers?

Tagging @lhoestq since you seem to be working on these issues and PRs :)
 Strangely, even when I manually created `np.arrays` of specific `dtypes`, the types in the final `dataset_info.json` that gets written are still `int64`.

Update: I tried creating lists of `int8`s and got the same result.


load_from_disk takes a long time to load local dataset
 I have an extremely large tokenized dataset (24M examples) that loads in a few minutes. However, after adding a column similar to `input_ids` (basically a list of integers) and saving the dataset to disk, the load time goes to >1 hour. I've even tried using `np.uint8` after seeing #1985 but it doesn't seem to be helping (the total size seems to be smaller though).

Does anyone know what could be the issue? Or does the casting of that column to `int8` need to happen in the function that writes the arrow table instead of in the `map` where I create the list of integers?

Tagging @lhoestq since you seem to be working on these issues and PRs :)
 Yes this is a known issue: #625
We're working on making the precision kept for numpy :)
To specify the precision of the integers, currently one needs to specify the output features with `.map(..., features=output_features)`


load_from_disk takes a long time to load local dataset
 I have an extremely large tokenized dataset (24M examples) that loads in a few minutes. However, after adding a column similar to `input_ids` (basically a list of integers) and saving the dataset to disk, the load time goes to >1 hour. I've even tried using `np.uint8` after seeing #1985 but it doesn't seem to be helping (the total size seems to be smaller though).

Does anyone know what could be the issue? Or does the casting of that column to `int8` need to happen in the function that writes the arrow table instead of in the `map` where I create the list of integers?

Tagging @lhoestq since you seem to be working on these issues and PRs :)
 Do you know what step is taking forever in the code ?
What happens if you interrupt the execution of the dataset loading ?


load_from_disk takes a long time to load local dataset
 I have an extremely large tokenized dataset (24M examples) that loads in a few minutes. However, after adding a column similar to `input_ids` (basically a list of integers) and saving the dataset to disk, the load time goes to >1 hour. I've even tried using `np.uint8` after seeing #1985 but it doesn't seem to be helping (the total size seems to be smaller though).

Does anyone know what could be the issue? Or does the casting of that column to `int8` need to happen in the function that writes the arrow table instead of in the `map` where I create the list of integers?

Tagging @lhoestq since you seem to be working on these issues and PRs :)
 After a synchronous discussion, we found that the cache file sizes have an enormous effect on the loading speed: smaller cache files result in faster load times. `num_proc` controls the number of cache files that are being written and is inversely proportional to the individual file size. In other words, increase `num_proc` for smaller cache files :)

Maybe this can be highlighted somewhere in the docs.


How to disable making arrow tables in load_dataset ?
 Is there a way to disable the construction of arrow tables, or to make them on the fly as the dataset is being used ?
 Hi ! We plan to add streaming features in the future.

This should allow to load a dataset instantaneously without generating the arrow table. The trade-off is that accessing examples from a streaming dataset must be done in an iterative way, and with an additional (but hopefully minor) overhead.
What do you think about this ?

If you have ideas or suggestions of what you expect from such features as a user, feel free to share them, this is really valuable to us !


How to disable making arrow tables in load_dataset ?
 Is there a way to disable the construction of arrow tables, or to make them on the fly as the dataset is being used ?
 People mainly want this feature either because it takes too much time too make arrow tables, or they occupy too much memory on the disk. I think both the problem can be solved if we provide arrow tables themselves on datasets hub. Can we do this currently @lhoestq ?


How to disable making arrow tables in load_dataset ?
 Is there a way to disable the construction of arrow tables, or to make them on the fly as the dataset is being used ?
 @lhoestq I think the ```try_from_hf_gcs``` provide the same functionality. What all datasets are available on HF GCS? Are all the datasets on huggingFace datasets hub are made available on GCS, automatically?


How to disable making arrow tables in load_dataset ?
 Is there a way to disable the construction of arrow tables, or to make them on the fly as the dataset is being used ?
 Only datasets like wikipedia, wiki40b, wiki_dpr and natural questions are available already processed on the HF google storage. This is used to download directly the arrow file instead of building it from the original data files.


How to disable making arrow tables in load_dataset ?
 Is there a way to disable the construction of arrow tables, or to make them on the fly as the dataset is being used ?
 @lhoestq How can we make sure that the data we upload on HuggingFace hub is available in form of preprocessed arrow files ?


How to disable making arrow tables in load_dataset ?
 Is there a way to disable the construction of arrow tables, or to make them on the fly as the dataset is being used ?
 We're still working on this :) This will be available soon
Users will be able to put their processed arrow files on the Hub


Add documentaton for dataset README.md files
 Hi,
the dataset README files have special headers.
Somehow a documenation of the allowed values and tags is missing.
Could you add that?

Just to give some concrete questions that should be answered imo:
- which values can be passted to multilinguality?
- what should be passed to language_creators?
- which values should licenses have? What do I say when it is a custom license? Should I add a link?
- how should I choose size_categories ? What are valid ranges?
- what are valid task_categories?

Thanks
Philip
 Hi ! We are using the [datasets-tagging app](https://github.com/huggingface/datasets-tagging) to select the tags to add.

We are also adding the full list of tags in #2107
This covers multilinguality, language_creators, licenses, size_categories and task_categories.

In general if you want to add a tag that doesn't exist (for example for a custom license) you must make it start with `other-` and then a custom tag name.

edit (@theo-m) if you ever find yourself resorting to adding an `other-*` tag, please do ping us somewhere so we can think about adding it to the "official" list :)


Add documentaton for dataset README.md files
 Hi,
the dataset README files have special headers.
Somehow a documenation of the allowed values and tags is missing.
Could you add that?

Just to give some concrete questions that should be answered imo:
- which values can be passted to multilinguality?
- what should be passed to language_creators?
- which values should licenses have? What do I say when it is a custom license? Should I add a link?
- how should I choose size_categories ? What are valid ranges?
- what are valid task_categories?

Thanks
Philip
 @lhoestq hmm - ok thanks for the answer.
To be honest I am not sure if this issue can be closed now.
I just wanted to point out that this should either be documented or linked in the documentation.
If you feel like it is (will be) please just close this.


Add documentaton for dataset README.md files
 Hi,
the dataset README files have special headers.
Somehow a documenation of the allowed values and tags is missing.
Could you add that?

Just to give some concrete questions that should be answered imo:
- which values can be passted to multilinguality?
- what should be passed to language_creators?
- which values should licenses have? What do I say when it is a custom license? Should I add a link?
- how should I choose size_categories ? What are valid ranges?
- what are valid task_categories?

Thanks
Philip
 We're still working on the validation+documentation in this.
Feel free to keep this issue open till we've added them


Add documentaton for dataset README.md files
 Hi,
the dataset README files have special headers.
Somehow a documenation of the allowed values and tags is missing.
Could you add that?

Just to give some concrete questions that should be answered imo:
- which values can be passted to multilinguality?
- what should be passed to language_creators?
- which values should licenses have? What do I say when it is a custom license? Should I add a link?
- how should I choose size_categories ? What are valid ranges?
- what are valid task_categories?

Thanks
Philip
 Hi ! There's the tagging app at https://huggingface.co/datasets/tagging/ that you can use.
It shows the list of all the tags you can use.

It is based on all the tag sets defined in this folder:
https://github.com/huggingface/datasets/tree/master/src/datasets/utils/resources


Add documentaton for dataset README.md files
 Hi,
the dataset README files have special headers.
Somehow a documenation of the allowed values and tags is missing.
Could you add that?

Just to give some concrete questions that should be answered imo:
- which values can be passted to multilinguality?
- what should be passed to language_creators?
- which values should licenses have? What do I say when it is a custom license? Should I add a link?
- how should I choose size_categories ? What are valid ranges?
- what are valid task_categories?

Thanks
Philip
 I don't think so. Feel free to take a look at the tags of other models (example [here](https://huggingface.co/bert-base-uncased/blob/main/README.md)). But we should definitely have some docs or an app to write the tags. Feel free to open an issue in the `transformers` repo or in the `huggingface_hub` repo so we can discuss this


`concatenate_datasets` throws error when changing the order of datasets to concatenate
 Hey,

I played around with the `concatenate_datasets(...)` function: https://huggingface.co/docs/datasets/package_reference/main_classes.html?highlight=concatenate_datasets#datasets.concatenate_datasets

and noticed that when the order in which the datasets are concatenated changes an error is thrown where it should not IMO.

Here is a google colab to reproduce the error: https://colab.research.google.com/drive/17VTFU4KQ735-waWZJjeOHS6yDTfV5ekK?usp=sharing
 Hi,

this bug is related to `Dataset.{remove_columns, rename_column, flatten}` not propagating the change to the schema metadata when the info features are updated, so this line is the culprit:
```python
common_voice_train = common_voice_train.remove_columns(['client_id', 'up_votes', 'down_votes', 'age', 'gender', 'accent', 'locale', 'segment'])

```
The order is important because the resulting dataset inherits the schema metadata of the first dataset passed to the `concatenate_datasets(...)` function (`pa.concat_tables` [docs](https://arrow.apache.org/docs/python/generated/pyarrow.concat_tables.html)). I'll try to fix this ASAP.


Multidimensional arrays in a Dataset

Multidimensional arrays in a Dataset
 Hi,

I'm trying to put together a `datasets.Dataset` to be used with LayoutLM which is available in `transformers`. This model requires as input the bounding boxes of each of the token of a sequence. This is when I realized that `Dataset` does not support multi-dimensional arrays as a value for a column in a row.

The following code results in conversion error in pyarrow (`pyarrow.lib.ArrowInvalid: ('Can only convert 1-dimensional array values', 'Conversion failed for column bbox with type object')`)

```
from datasets import Dataset
import pandas as pd
import numpy as np

dataset = pd.DataFrame({
    'bbox': [
        np.array([[1,2,3,4],[1,2,3,4],[1,2,3,4]]),
        np.array([[1,2,3,4],[1,2,3,4],[1,2,3,4]]),
        np.array([[1,2,3,4],[1,2,3,4],[1,2,3,4]]),
        np.array([[1,2,3,4],[1,2,3,4],[1,2,3,4]])
    ],
    'input_ids': [1, 2, 3, 4]
})
dataset = Dataset.from_pandas(dataset)
```

Since I wanted to use pytorch for the downstream training task, I also tried a few ways to directly put in a column of 2-D pytorch tensor in a formatted dataset, but I can only have a list of 1-D tensors, or a list of arrays, or a list of lists.

```
import torch
from datasets import Dataset
import pandas as pd

dataset = pd.DataFrame({
    'bbox': [
        [[1,2,3,4],[1,2,3,4],[1,2,3,4]],
        [[1,2,3,4],[1,2,3,4],[1,2,3,4]],
        [[1,2,3,4],[1,2,3,4],[1,2,3,4]],
        [[1,2,3,4],[1,2,3,4],[1,2,3,4]]
    ],
    'input_ids': [1, 2, 3, 4]
})
dataset = Dataset.from_pandas(dataset)

def test(examples):
    return {'bbbox': torch.Tensor(examples['bbox'])}
dataset = dataset.map(test)
print(dataset[0]['bbox'])
print(dataset[0]['bbbox'])

dataset.set_format(type='torch', columns=['input_ids', 'bbox'], output_all_columns=True)
print(dataset[0]['bbox'])
print(dataset[0]['bbbox'])

def test2(examples):
    return {'bbbox': torch.stack(examples['bbox'])}
dataset = dataset.map(test2)

print(dataset[0]['bbox'])
print(dataset[0]['bbbox'])
```

Is is possible to support n-D arrays/tensors in datasets?
It seems that it can also be useful for this [feature request](https://github.com/huggingface/datasets/issues/263).
 Hi !

This is actually supported ! but not yet in `from_pandas`.
You can use `from_dict` for now instead:
```python
from datasets import Dataset, Array2D, Features, Value
import pandas as pd
import numpy as np

dataset = {
    'bbox': [
        np.array([[1,2,3,4],[1,2,3,4],[1,2,3,4]]),
        np.array([[1,2,3,4],[1,2,3,4],[1,2,3,4]]),
        np.array([[1,2,3,4],[1,2,3,4],[1,2,3,4]]),
        np.array([[1,2,3,4],[1,2,3,4],[1,2,3,4]])
    ],
    'input_ids': [1, 2, 3, 4]
}
dataset = Dataset.from_dict(dataset)
```

This will work but to use it with the torch formatter you must specify the `Array2D` feature type in order to tell the shape:
```python
from datasets import Dataset, Array2D, Features, Value
import pandas as pd
import numpy as np

dataset = {
    'bbox': [
        np.array([[1,2,3,4],[1,2,3,4],[1,2,3,4]]),
        np.array([[1,2,3,4],[1,2,3,4],[1,2,3,4]]),
        np.array([[1,2,3,4],[1,2,3,4],[1,2,3,4]]),
        np.array([[1,2,3,4],[1,2,3,4],[1,2,3,4]])
    ],
    'input_ids': [1, 2, 3, 4]
}
dataset = Dataset.from_dict(dataset, features=Features({
    "bbox": Array2D(shape=(3, 4), dtype="int64"),
    "input_ids": Value("int64")
}))
dataset.set_format("torch")
print(dataset[0]['bbox'])
# tensor([[1, 2, 3, 4],
#         [1, 2, 3, 4],
#         [1, 2, 3, 4]])
```
If you don't specify the `Array2D` feature type, then the inferred type will be Sequence(Sequence(Value("int64"))) and therefore the torch formatter will return list of tensors


Multidimensional arrays in a Dataset

Multidimensional arrays in a Dataset
 Hi,

I'm trying to put together a `datasets.Dataset` to be used with LayoutLM which is available in `transformers`. This model requires as input the bounding boxes of each of the token of a sequence. This is when I realized that `Dataset` does not support multi-dimensional arrays as a value for a column in a row.

The following code results in conversion error in pyarrow (`pyarrow.lib.ArrowInvalid: ('Can only convert 1-dimensional array values', 'Conversion failed for column bbox with type object')`)

```
from datasets import Dataset
import pandas as pd
import numpy as np

dataset = pd.DataFrame({
    'bbox': [
        np.array([[1,2,3,4],[1,2,3,4],[1,2,3,4]]),
        np.array([[1,2,3,4],[1,2,3,4],[1,2,3,4]]),
        np.array([[1,2,3,4],[1,2,3,4],[1,2,3,4]]),
        np.array([[1,2,3,4],[1,2,3,4],[1,2,3,4]])
    ],
    'input_ids': [1, 2, 3, 4]
})
dataset = Dataset.from_pandas(dataset)
```

Since I wanted to use pytorch for the downstream training task, I also tried a few ways to directly put in a column of 2-D pytorch tensor in a formatted dataset, but I can only have a list of 1-D tensors, or a list of arrays, or a list of lists.

```
import torch
from datasets import Dataset
import pandas as pd

dataset = pd.DataFrame({
    'bbox': [
        [[1,2,3,4],[1,2,3,4],[1,2,3,4]],
        [[1,2,3,4],[1,2,3,4],[1,2,3,4]],
        [[1,2,3,4],[1,2,3,4],[1,2,3,4]],
        [[1,2,3,4],[1,2,3,4],[1,2,3,4]]
    ],
    'input_ids': [1, 2, 3, 4]
})
dataset = Dataset.from_pandas(dataset)

def test(examples):
    return {'bbbox': torch.Tensor(examples['bbox'])}
dataset = dataset.map(test)
print(dataset[0]['bbox'])
print(dataset[0]['bbbox'])

dataset.set_format(type='torch', columns=['input_ids', 'bbox'], output_all_columns=True)
print(dataset[0]['bbox'])
print(dataset[0]['bbbox'])

def test2(examples):
    return {'bbbox': torch.stack(examples['bbox'])}
dataset = dataset.map(test2)

print(dataset[0]['bbox'])
print(dataset[0]['bbbox'])
```

Is is possible to support n-D arrays/tensors in datasets?
It seems that it can also be useful for this [feature request](https://github.com/huggingface/datasets/issues/263).
 Thanks for the explanation.
With my original DataFrame, I did
```
dataset = dataset.to_dict("list")
```
and then the rest of the transformation from dictionary works just fine.


MemoryError when computing WER metric

MemoryError when computing WER metric
 Hi, I'm trying to follow the ASR example to try Wav2Vec. This is the code that I use for WER calculation:

```
wer = load_metric("wer")
print(wer.compute(predictions=result["predicted"], references=result["target"]))
```

However, I receive the following exception:

`Traceback (most recent call last):
  File "/home/diego/IpGlobal/wav2vec/test_wav2vec.py", line 51, in <module>
    print(wer.compute(predictions=result["predicted"], references=result["target"]))
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/datasets/metric.py", line 403, in compute
    output = self._compute(predictions=predictions, references=references, **kwargs)
  File "/home/diego/.cache/huggingface/modules/datasets_modules/metrics/wer/73b2d32b723b7fb8f204d785c00980ae4d937f12a65466f8fdf78706e2951281/wer.py", line 94, in _compute
    return wer(references, predictions)
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 81, in wer
    truth, hypothesis, truth_transform, hypothesis_transform, **kwargs
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 192, in compute_measures
    H, S, D, I = _get_operation_counts(truth, hypothesis)
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 273, in _get_operation_counts
    editops = Levenshtein.editops(source_string, destination_string)
MemoryError`

My system has more than 10GB of available RAM. Looking at the code, I think that it could be related to the way jiwer does the calculation, as it is pasting all the sentences in a single string before calling Levenshtein editops function.

 Hi ! Thanks for reporting.
We're indeed using `jiwer` to compute the WER.

Maybe instead of calling `jiwer.wer` once for all the preditions/references we can compute the WER iteratively to avoid memory issues ? I'm not too familial with `jiwer` but this must be possible.

Currently the code to compute the WER is defined here:

https://github.com/huggingface/nlp/blob/349ac4398a3bcae6356f14c5754483383a60e8a4/metrics/wer/wer.py#L93-L94


MemoryError when computing WER metric

MemoryError when computing WER metric
 Hi, I'm trying to follow the ASR example to try Wav2Vec. This is the code that I use for WER calculation:

```
wer = load_metric("wer")
print(wer.compute(predictions=result["predicted"], references=result["target"]))
```

However, I receive the following exception:

`Traceback (most recent call last):
  File "/home/diego/IpGlobal/wav2vec/test_wav2vec.py", line 51, in <module>
    print(wer.compute(predictions=result["predicted"], references=result["target"]))
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/datasets/metric.py", line 403, in compute
    output = self._compute(predictions=predictions, references=references, **kwargs)
  File "/home/diego/.cache/huggingface/modules/datasets_modules/metrics/wer/73b2d32b723b7fb8f204d785c00980ae4d937f12a65466f8fdf78706e2951281/wer.py", line 94, in _compute
    return wer(references, predictions)
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 81, in wer
    truth, hypothesis, truth_transform, hypothesis_transform, **kwargs
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 192, in compute_measures
    H, S, D, I = _get_operation_counts(truth, hypothesis)
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 273, in _get_operation_counts
    editops = Levenshtein.editops(source_string, destination_string)
MemoryError`

My system has more than 10GB of available RAM. Looking at the code, I think that it could be related to the way jiwer does the calculation, as it is pasting all the sentences in a single string before calling Levenshtein editops function.

 Hi,

I've just pushed a pull request that is related to this issue https://github.com/huggingface/datasets/pull/2169. It's not iterative, but it should avoid memory errors. It's based on the editdistance python library. An iterative implementation should be as easy as storing scores and words stepwise and dividing at the end.


MemoryError when computing WER metric

MemoryError when computing WER metric
 Hi, I'm trying to follow the ASR example to try Wav2Vec. This is the code that I use for WER calculation:

```
wer = load_metric("wer")
print(wer.compute(predictions=result["predicted"], references=result["target"]))
```

However, I receive the following exception:

`Traceback (most recent call last):
  File "/home/diego/IpGlobal/wav2vec/test_wav2vec.py", line 51, in <module>
    print(wer.compute(predictions=result["predicted"], references=result["target"]))
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/datasets/metric.py", line 403, in compute
    output = self._compute(predictions=predictions, references=references, **kwargs)
  File "/home/diego/.cache/huggingface/modules/datasets_modules/metrics/wer/73b2d32b723b7fb8f204d785c00980ae4d937f12a65466f8fdf78706e2951281/wer.py", line 94, in _compute
    return wer(references, predictions)
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 81, in wer
    truth, hypothesis, truth_transform, hypothesis_transform, **kwargs
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 192, in compute_measures
    H, S, D, I = _get_operation_counts(truth, hypothesis)
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 273, in _get_operation_counts
    editops = Levenshtein.editops(source_string, destination_string)
MemoryError`

My system has more than 10GB of available RAM. Looking at the code, I think that it could be related to the way jiwer does the calculation, as it is pasting all the sentences in a single string before calling Levenshtein editops function.

 I see, this was solved by other thread. Ok, let me know if you want to switch the implementation for any reason :)


MemoryError when computing WER metric

MemoryError when computing WER metric
 Hi, I'm trying to follow the ASR example to try Wav2Vec. This is the code that I use for WER calculation:

```
wer = load_metric("wer")
print(wer.compute(predictions=result["predicted"], references=result["target"]))
```

However, I receive the following exception:

`Traceback (most recent call last):
  File "/home/diego/IpGlobal/wav2vec/test_wav2vec.py", line 51, in <module>
    print(wer.compute(predictions=result["predicted"], references=result["target"]))
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/datasets/metric.py", line 403, in compute
    output = self._compute(predictions=predictions, references=references, **kwargs)
  File "/home/diego/.cache/huggingface/modules/datasets_modules/metrics/wer/73b2d32b723b7fb8f204d785c00980ae4d937f12a65466f8fdf78706e2951281/wer.py", line 94, in _compute
    return wer(references, predictions)
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 81, in wer
    truth, hypothesis, truth_transform, hypothesis_transform, **kwargs
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 192, in compute_measures
    H, S, D, I = _get_operation_counts(truth, hypothesis)
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 273, in _get_operation_counts
    editops = Levenshtein.editops(source_string, destination_string)
MemoryError`

My system has more than 10GB of available RAM. Looking at the code, I think that it could be related to the way jiwer does the calculation, as it is pasting all the sentences in a single string before calling Levenshtein editops function.

 Thanks for diving into this anyway ^^'
As you said this actually got solved a few days ago


MemoryError when computing WER metric

MemoryError when computing WER metric
 Hi, I'm trying to follow the ASR example to try Wav2Vec. This is the code that I use for WER calculation:

```
wer = load_metric("wer")
print(wer.compute(predictions=result["predicted"], references=result["target"]))
```

However, I receive the following exception:

`Traceback (most recent call last):
  File "/home/diego/IpGlobal/wav2vec/test_wav2vec.py", line 51, in <module>
    print(wer.compute(predictions=result["predicted"], references=result["target"]))
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/datasets/metric.py", line 403, in compute
    output = self._compute(predictions=predictions, references=references, **kwargs)
  File "/home/diego/.cache/huggingface/modules/datasets_modules/metrics/wer/73b2d32b723b7fb8f204d785c00980ae4d937f12a65466f8fdf78706e2951281/wer.py", line 94, in _compute
    return wer(references, predictions)
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 81, in wer
    truth, hypothesis, truth_transform, hypothesis_transform, **kwargs
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 192, in compute_measures
    H, S, D, I = _get_operation_counts(truth, hypothesis)
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 273, in _get_operation_counts
    editops = Levenshtein.editops(source_string, destination_string)
MemoryError`

My system has more than 10GB of available RAM. Looking at the code, I think that it could be related to the way jiwer does the calculation, as it is pasting all the sentences in a single string before calling Levenshtein editops function.

 Someone created an issue https://github.com/jitsi/jiwer/issues/40 at jiwer which shows that this is still a problem in the current version. Would be curious to figure out how this can be fixed by jiwer... :) I assume that it runs of out memory because it's trying to compute the WER over (too many) test samples?


MemoryError when computing WER metric

MemoryError when computing WER metric
 Hi, I'm trying to follow the ASR example to try Wav2Vec. This is the code that I use for WER calculation:

```
wer = load_metric("wer")
print(wer.compute(predictions=result["predicted"], references=result["target"]))
```

However, I receive the following exception:

`Traceback (most recent call last):
  File "/home/diego/IpGlobal/wav2vec/test_wav2vec.py", line 51, in <module>
    print(wer.compute(predictions=result["predicted"], references=result["target"]))
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/datasets/metric.py", line 403, in compute
    output = self._compute(predictions=predictions, references=references, **kwargs)
  File "/home/diego/.cache/huggingface/modules/datasets_modules/metrics/wer/73b2d32b723b7fb8f204d785c00980ae4d937f12a65466f8fdf78706e2951281/wer.py", line 94, in _compute
    return wer(references, predictions)
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 81, in wer
    truth, hypothesis, truth_transform, hypothesis_transform, **kwargs
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 192, in compute_measures
    H, S, D, I = _get_operation_counts(truth, hypothesis)
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 273, in _get_operation_counts
    editops = Levenshtein.editops(source_string, destination_string)
MemoryError`

My system has more than 10GB of available RAM. Looking at the code, I think that it could be related to the way jiwer does the calculation, as it is pasting all the sentences in a single string before calling Levenshtein editops function.

 Hi !

It's computed iteratively so not sure what could go wrong

https://github.com/huggingface/datasets/blob/8afd0ba8c27800a55ea69d9fcd702dc97d9c16d8/metrics/wer/wer.py#L100-L106

@NiklasHoltmeyer what version of `datasets` are you running ?


MemoryError when computing WER metric

MemoryError when computing WER metric
 Hi, I'm trying to follow the ASR example to try Wav2Vec. This is the code that I use for WER calculation:

```
wer = load_metric("wer")
print(wer.compute(predictions=result["predicted"], references=result["target"]))
```

However, I receive the following exception:

`Traceback (most recent call last):
  File "/home/diego/IpGlobal/wav2vec/test_wav2vec.py", line 51, in <module>
    print(wer.compute(predictions=result["predicted"], references=result["target"]))
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/datasets/metric.py", line 403, in compute
    output = self._compute(predictions=predictions, references=references, **kwargs)
  File "/home/diego/.cache/huggingface/modules/datasets_modules/metrics/wer/73b2d32b723b7fb8f204d785c00980ae4d937f12a65466f8fdf78706e2951281/wer.py", line 94, in _compute
    return wer(references, predictions)
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 81, in wer
    truth, hypothesis, truth_transform, hypothesis_transform, **kwargs
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 192, in compute_measures
    H, S, D, I = _get_operation_counts(truth, hypothesis)
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 273, in _get_operation_counts
    editops = Levenshtein.editops(source_string, destination_string)
MemoryError`

My system has more than 10GB of available RAM. Looking at the code, I think that it could be related to the way jiwer does the calculation, as it is pasting all the sentences in a single string before calling Levenshtein editops function.

 One possible explanation might be that it is the user who is passing all the sentences in a single element to `wer.compute`?

As current implementation iterates over the elements of `predictions` and `references`, this can be problematic if `predictions` and `references` contain a single huge element each.

This could be the case, for example, with a single string with all sentences:
```python
result["predicted"] = "One sentence. Other sentence."
```
or with a __double__ nested list of sentence lists
```python
result["predicted"] = [[ ["One sentence."], ["Other sentence"] ]]
```

The user should check the dimensions of the data structure passed to `predictions` and `references`.


MemoryError when computing WER metric

MemoryError when computing WER metric
 Hi, I'm trying to follow the ASR example to try Wav2Vec. This is the code that I use for WER calculation:

```
wer = load_metric("wer")
print(wer.compute(predictions=result["predicted"], references=result["target"]))
```

However, I receive the following exception:

`Traceback (most recent call last):
  File "/home/diego/IpGlobal/wav2vec/test_wav2vec.py", line 51, in <module>
    print(wer.compute(predictions=result["predicted"], references=result["target"]))
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/datasets/metric.py", line 403, in compute
    output = self._compute(predictions=predictions, references=references, **kwargs)
  File "/home/diego/.cache/huggingface/modules/datasets_modules/metrics/wer/73b2d32b723b7fb8f204d785c00980ae4d937f12a65466f8fdf78706e2951281/wer.py", line 94, in _compute
    return wer(references, predictions)
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 81, in wer
    truth, hypothesis, truth_transform, hypothesis_transform, **kwargs
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 192, in compute_measures
    H, S, D, I = _get_operation_counts(truth, hypothesis)
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 273, in _get_operation_counts
    editops = Levenshtein.editops(source_string, destination_string)
MemoryError`

My system has more than 10GB of available RAM. Looking at the code, I think that it could be related to the way jiwer does the calculation, as it is pasting all the sentences in a single string before calling Levenshtein editops function.

 Hi all,

in my case I was using and older version of datasets and, as @albertvillanova points out, passing the full list of sentences for the metric calculation. The problem was in the way jiwer implements WER, as it tries to compute WER for the full list at once instead of doing it element-wise. I think that with the latest implementation of datasets, or by using the alternative WER function that I've contributed on this [pull request](https://github.com/huggingface/datasets/pull/2169) there shouldn't be memory errors.


MemoryError when computing WER metric

MemoryError when computing WER metric
 Hi, I'm trying to follow the ASR example to try Wav2Vec. This is the code that I use for WER calculation:

```
wer = load_metric("wer")
print(wer.compute(predictions=result["predicted"], references=result["target"]))
```

However, I receive the following exception:

`Traceback (most recent call last):
  File "/home/diego/IpGlobal/wav2vec/test_wav2vec.py", line 51, in <module>
    print(wer.compute(predictions=result["predicted"], references=result["target"]))
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/datasets/metric.py", line 403, in compute
    output = self._compute(predictions=predictions, references=references, **kwargs)
  File "/home/diego/.cache/huggingface/modules/datasets_modules/metrics/wer/73b2d32b723b7fb8f204d785c00980ae4d937f12a65466f8fdf78706e2951281/wer.py", line 94, in _compute
    return wer(references, predictions)
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 81, in wer
    truth, hypothesis, truth_transform, hypothesis_transform, **kwargs
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 192, in compute_measures
    H, S, D, I = _get_operation_counts(truth, hypothesis)
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 273, in _get_operation_counts
    editops = Levenshtein.editops(source_string, destination_string)
MemoryError`

My system has more than 10GB of available RAM. Looking at the code, I think that it could be related to the way jiwer does the calculation, as it is pasting all the sentences in a single string before calling Levenshtein editops function.

 @lhoestq i was using Datasets==1.5.0 with 1.6.1 it worked (atleast the first run) but 1.5.0 is not compatible with my preprocessing. i cant save my dataset to a parquet file while using the latest datasets version

->
```
  File "../preprocess_dataset.py", line 132, in <module>
    pq.write_table(train_dataset.data, f'{resampled_data_dir}/{data_args.dataset_config_name}.train.parquet')
  File "/usr/local/lib/python3.8/dist-packages/pyarrow/parquet.py", line 1674, in write_table
    writer.write_table(table, row_group_size=row_group_size)
  File "/usr/local/lib/python3.8/dist-packages/pyarrow/parquet.py", line 588, in write_table
    self.writer.write_table(table, row_group_size=row_group_size)
TypeError: Argument 'table' has incorrect type (expected pyarrow.lib.Table, got ConcatenationTable)
```

if i do
```
import pyarrow.parquet as pq
...
...
pq.write_table(train_dataset.data, 'train.parquet')
pq.write_table(eval_dataset.data, 'eval.parquet')
```

while using 1.6.1. and its working with 1.5.0


Issue: Dataset download error

Issue: Dataset download error
 The download link in `iwslt2017.py` file does not seem to work anymore.

For example, `FileNotFoundError: Couldn't find file at https://wit3.fbk.eu/archive/2017-01-trnted/texts/zh/en/zh-en.tgz`

Would be nice if we could modify it script and use the new downloadable link?
 Hi @XuhuiZhou, thanks for reporting this issue.

Indeed, the old links are no longer valid (404 Not Found error), and the script must be updated with the new links to Google Drive.


Issue: Dataset download error

Issue: Dataset download error
 The download link in `iwslt2017.py` file does not seem to work anymore.

For example, `FileNotFoundError: Couldn't find file at https://wit3.fbk.eu/archive/2017-01-trnted/texts/zh/en/zh-en.tgz`

Would be nice if we could modify it script and use the new downloadable link?
 It would be nice to update the urls indeed !

To do this, you just need to replace the urls in `iwslt2017.py` and then update the dataset_infos.json file with
```
datasets-cli test ./datasets/iwslt2017 --all_configs --save_infos --ignore_verifications
```


Issue: Dataset download error

Issue: Dataset download error
 The download link in `iwslt2017.py` file does not seem to work anymore.

For example, `FileNotFoundError: Couldn't find file at https://wit3.fbk.eu/archive/2017-01-trnted/texts/zh/en/zh-en.tgz`

Would be nice if we could modify it script and use the new downloadable link?
 Is this a command to update my local files or fix the file Github repo in general? (I am not so familiar with the datasets-cli command here)

I also took a brief look at the **Sharing your dataset** section, looks like I could fix that locally and push it to the repo? I guess we are "canonical" category?


Issue: Dataset download error

Issue: Dataset download error
 The download link in `iwslt2017.py` file does not seem to work anymore.

For example, `FileNotFoundError: Couldn't find file at https://wit3.fbk.eu/archive/2017-01-trnted/texts/zh/en/zh-en.tgz`

Would be nice if we could modify it script and use the new downloadable link?
 This command will update your local file. Then you can open a Pull Request to push your fix to the github repo :)
And yes you are right, it is a "canonical" dataset, i.e. a dataset script defined in this github repo (as opposed to dataset repositories of users on the huggingface hub)


Issue: Dataset download error

Issue: Dataset download error
 The download link in `iwslt2017.py` file does not seem to work anymore.

For example, `FileNotFoundError: Couldn't find file at https://wit3.fbk.eu/archive/2017-01-trnted/texts/zh/en/zh-en.tgz`

Would be nice if we could modify it script and use the new downloadable link?
 Hi, thanks for the answer.

I gave a try to the problem today. But I encountered an upload error:

```
git push -u origin fix_link_iwslt
Enter passphrase for key '/home2/xuhuizh/.ssh/id_rsa':
ERROR: Permission to huggingface/datasets.git denied to XuhuiZhou.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
```

Any insight here?

By the way, when I run the datasets-cli command, it shows the following error, but does not seem to be the error coming from `iwslt.py`

```
Traceback (most recent call last):
  File "/home2/xuhuizh/anaconda3/envs/UMT/bin/datasets-cli", line 33, in <module>
    sys.exit(load_entry_point('datasets', 'console_scripts', 'datasets-cli')())
  File "/home2/xuhuizh/projects/datasets/src/datasets/commands/datasets_cli.py", line 35, in main
    service.run()
  File "/home2/xuhuizh/projects/datasets/src/datasets/commands/test.py", line 141, in run
    try_from_hf_gcs=False,
  File "/home2/xuhuizh/projects/datasets/src/datasets/builder.py", line 579, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home2/xuhuizh/projects/datasets/src/datasets/builder.py", line 639, in _download_and_prepare
    self.info.download_checksums, dl_manager.get_recorded_sizes_checksums(), "dataset source files"
  File "/home2/xuhuizh/projects/datasets/src/datasets/utils/info_utils.py", line 32, in verify_checksums
    raise ExpectedMoreDownloadedFiles(str(set(expected_checksums) - set(recorded_checksums)))
datasets.utils.info_utils.ExpectedMoreDownloadedFiles: {'https://wit3.fbk.eu/archive/2017-01-trnmted//texts/DeEnItNlRo/DeEnItNlRo/DeEnItNlRo-DeEnItNlRo.tgz'}
```


Issue: Dataset download error

Issue: Dataset download error
 The download link in `iwslt2017.py` file does not seem to work anymore.

For example, `FileNotFoundError: Couldn't find file at https://wit3.fbk.eu/archive/2017-01-trnted/texts/zh/en/zh-en.tgz`

Would be nice if we could modify it script and use the new downloadable link?
 Hi ! To create a PR on this repo your must fork it and create a branch on your fork. See how to fork the repo [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md#start-by-preparing-your-environment).
And to make the command work without the `ExpectedMoreDownloadedFiles` error, you just need to use the `--ignore_verifications` flag.


Issue: Dataset download error

Issue: Dataset download error
 The download link in `iwslt2017.py` file does not seem to work anymore.

For example, `FileNotFoundError: Couldn't find file at https://wit3.fbk.eu/archive/2017-01-trnted/texts/zh/en/zh-en.tgz`

Would be nice if we could modify it script and use the new downloadable link?
 Hi @XuhuiZhou,

As @lhoestq has well explained, you need to fork HF's repository, create a feature branch in your fork, push your changes to it and then open a Pull Request to HF's upstream repository. This is so because at HuggingFace Datasets we follow a development model called "Fork and Pull Model". You can find more information here:
- [Understanding the GitHub flow](https://guides.github.com/introduction/flow/)
- [Forking Projects](https://guides.github.com/activities/forking/)

Alternatively, if you find all these steps too complicated, you can use the GitHub official command line tool: [GitHub CLI](https://cli.github.com/). Once installed, in order to create a Pull Request, you only need to use this command:
```shell
gh pr create --web
```
This utility will automatically create the fork, push your changes and open a Pull Request, under the hood.


ConnectionError: Couldn't reach common_voice.py
 When I run:
from datasets import load_dataset, load_metric

common_voice_train = load_dataset("common_voice", "zh-CN", split="train+validation")
common_voice_test = load_dataset("common_voice", "zh-CN", split="test")

Got:
ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/master/datasets/common_voice/common_voice.py

Version:
1.4.1

Thanks!  @lhoestq @LysandreJik @thomwolf
 Hi @LifaSun, thanks for reporting this issue.

Sometimes, GitHub has some connectivity problems. Could you confirm that the problem persists?


ArrowInvalid issue for squad v2 dataset

ArrowInvalid issue for squad v2 dataset
 Hello, I am using the huggingface official question answering example notebook (https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/question_answering.ipynb).

In the prepare_validation_features function, I made some modifications to tokenize a new set of quesions with the original contexts and save them in three different list called candidate_input_dis, candidate_attetion_mask and candidate_token_type_ids. When I try to run the next cell for dataset.map, I got the following error:

`ArrowInvalid: Column 1 named candidate_attention_mask expected length 1180 but got length 1178`

My code is as follows:

```
def generate_candidate_questions(examples):
  val_questions = examples["question"]
  candididate_questions = random.sample(datasets["train"]["question"], len(val_questions))
  candididate_questions = [x[:max_length] for x in candididate_questions]
  return candididate_questions

def prepare_validation_features(examples, use_mixing=False):
  pad_on_right = tokenizer.padding_side == "right"
  tokenized_examples = tokenizer(
      examples["question" if pad_on_right else "context"],
      examples["context" if pad_on_right else "question"],
      truncation="only_second" if pad_on_right else "only_first",
      max_length=max_length,
      stride=doc_stride,
      return_overflowing_tokens=True,
      return_offsets_mapping=True,
      padding="max_length",
  )
  if use_mixing:
    candidate_questions = generate_candidate_questions(examples)
    tokenized_candidates = tokenizer(
        candidate_questions if pad_on_right else examples["context"],
        examples["context"] if pad_on_right else candidate_questions,
        truncation="only_second" if pad_on_right else "only_first",
        max_length=max_length,
        stride=doc_stride,
        return_overflowing_tokens=True,
        return_offsets_mapping=True,
        padding="max_length",
    )

  sample_mapping = tokenized_examples.pop("overflow_to_sample_mapping")

  tokenized_examples["example_id"] = []

  if use_mixing:
    tokenized_examples["candidate_input_ids"] = tokenized_candidates["input_ids"]
    tokenized_examples["candidate_attention_mask"] = tokenized_candidates["attention_mask"]
    tokenized_examples["candidate_token_type_ids"] = tokenized_candidates["token_type_ids"]

  for i in range(len(tokenized_examples["input_ids"])):
      sequence_ids = tokenized_examples.sequence_ids(i)
      context_index = 1 if pad_on_right else 0

      sample_index = sample_mapping[i]
      tokenized_examples["example_id"].append(examples["id"][sample_index])

      tokenized_examples["offset_mapping"][i] = [
          (o if sequence_ids[k] == context_index else None)
          for k, o in enumerate(tokenized_examples["offset_mapping"][i])
      ]

  return tokenized_examples

validation_features = datasets["validation"].map(
    lambda xs: prepare_validation_features(xs, True),
    batched=True,
    remove_columns=datasets["validation"].column_names
)
```

I guess this might happen because of the batched=True. I see similar issues in this repo related to arrow table length mismatch error, but in their cases, the numbers vary a lot. In my case, this error always happens when the expected length and unexpected length are very close. Thanks for the help!
 Hi ! This error happens when you use `map` in batched mode and then your function doesn't return the same number of values per column.

Indeed since you're using `map` in batched mode, `prepare_validation_features` must take a batch as input (i.e. a dictionary of multiple rows of the dataset), and return a batch.

However it seems like `tokenized_examples` doesn't have the same number of elements in each field. One field seems to have `1180` elements while `candidate_attention_mask` only has `1178`.


PyTorch not available error on SageMaker GPU docker though it is installed
 I get en error when running data loading using SageMaker SDK

```
  File "main.py", line 34, in <module>
    run_training()
  File "main.py", line 25, in run_training
    dm.setup('fit')
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/core/datamodule.py", line 92, in wrapped_fn
    return fn(*args, **kwargs)
  File "/opt/ml/code/data_module.py", line 103, in setup
    self.dataset[split].set_format(type="torch", columns=self.columns)
  File "/opt/conda/lib/python3.6/site-packages/datasets/fingerprint.py", line 337, in wrapper
    out = func(self, *args, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 995, in set_format
    _ = get_formatter(type, **format_kwargs)
File "/opt/conda/lib/python3.6/site-packages/datasets/formatting/__init__.py", line 114, in get_formatter
    raise _FORMAT_TYPES_ALIASES_UNAVAILABLE[format_type]
ValueError: PyTorch needs to be installed to be able to return PyTorch tensors.
```

when trying to execute dataset loading using this notebook  https://github.com/PyTorchLightning/pytorch-lightning/blob/master/notebooks/04-transformers-text-classification.ipynb, specifically lines

```
self.columns = [c for c in self.dataset[split].column_names if c in self.loader_columns]
self.dataset[split].set_format(type="torch", columns=self.columns)
```

The SageMaker docker image used is 763104351884.dkr.ecr.eu-central-1.amazonaws.com/pytorch-training:1.4.0-gpu-py3 .

By running container interactively I have checked that torch loading completes successfully by executing `https://github.com/huggingface/datasets/blob/master/src/datasets/config.py#L39`.

Also as a first line in the data loading module I have

```
import os
os.environ["USE_TF"] = "0"
os.environ["USE_TORCH"] = "1"
````

But unfortunately the error stills persists. Any suggestions would be appreciated as I am stack.
Many Thanks!

 Hey @sivakhno,

how does your `requirements.txt` look like to install the `datasets` library and which version of it are you running? Can you try to install `datasets>=1.4.0`


PyTorch not available error on SageMaker GPU docker though it is installed
 I get en error when running data loading using SageMaker SDK

```
  File "main.py", line 34, in <module>
    run_training()
  File "main.py", line 25, in run_training
    dm.setup('fit')
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/core/datamodule.py", line 92, in wrapped_fn
    return fn(*args, **kwargs)
  File "/opt/ml/code/data_module.py", line 103, in setup
    self.dataset[split].set_format(type="torch", columns=self.columns)
  File "/opt/conda/lib/python3.6/site-packages/datasets/fingerprint.py", line 337, in wrapper
    out = func(self, *args, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 995, in set_format
    _ = get_formatter(type, **format_kwargs)
File "/opt/conda/lib/python3.6/site-packages/datasets/formatting/__init__.py", line 114, in get_formatter
    raise _FORMAT_TYPES_ALIASES_UNAVAILABLE[format_type]
ValueError: PyTorch needs to be installed to be able to return PyTorch tensors.
```

when trying to execute dataset loading using this notebook  https://github.com/PyTorchLightning/pytorch-lightning/blob/master/notebooks/04-transformers-text-classification.ipynb, specifically lines

```
self.columns = [c for c in self.dataset[split].column_names if c in self.loader_columns]
self.dataset[split].set_format(type="torch", columns=self.columns)
```

The SageMaker docker image used is 763104351884.dkr.ecr.eu-central-1.amazonaws.com/pytorch-training:1.4.0-gpu-py3 .

By running container interactively I have checked that torch loading completes successfully by executing `https://github.com/huggingface/datasets/blob/master/src/datasets/config.py#L39`.

Also as a first line in the data loading module I have

```
import os
os.environ["USE_TF"] = "0"
os.environ["USE_TORCH"] = "1"
````

But unfortunately the error stills persists. Any suggestions would be appreciated as I am stack.
Many Thanks!

 Hi @philschmid - thanks for suggestion. I am using `datasets==1.4.1`.
I have also tried using `torch=1.6.0` (docker `763104351884.dkr.ecr.eu-central-1.amazonaws.com/pytorch-training:1.6.0-gpu-py3 `), but the error is the same.


PyTorch not available error on SageMaker GPU docker though it is installed
 I get en error when running data loading using SageMaker SDK

```
  File "main.py", line 34, in <module>
    run_training()
  File "main.py", line 25, in run_training
    dm.setup('fit')
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/core/datamodule.py", line 92, in wrapped_fn
    return fn(*args, **kwargs)
  File "/opt/ml/code/data_module.py", line 103, in setup
    self.dataset[split].set_format(type="torch", columns=self.columns)
  File "/opt/conda/lib/python3.6/site-packages/datasets/fingerprint.py", line 337, in wrapper
    out = func(self, *args, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 995, in set_format
    _ = get_formatter(type, **format_kwargs)
File "/opt/conda/lib/python3.6/site-packages/datasets/formatting/__init__.py", line 114, in get_formatter
    raise _FORMAT_TYPES_ALIASES_UNAVAILABLE[format_type]
ValueError: PyTorch needs to be installed to be able to return PyTorch tensors.
```

when trying to execute dataset loading using this notebook  https://github.com/PyTorchLightning/pytorch-lightning/blob/master/notebooks/04-transformers-text-classification.ipynb, specifically lines

```
self.columns = [c for c in self.dataset[split].column_names if c in self.loader_columns]
self.dataset[split].set_format(type="torch", columns=self.columns)
```

The SageMaker docker image used is 763104351884.dkr.ecr.eu-central-1.amazonaws.com/pytorch-training:1.4.0-gpu-py3 .

By running container interactively I have checked that torch loading completes successfully by executing `https://github.com/huggingface/datasets/blob/master/src/datasets/config.py#L39`.

Also as a first line in the data loading module I have

```
import os
os.environ["USE_TF"] = "0"
os.environ["USE_TORCH"] = "1"
````

But unfortunately the error stills persists. Any suggestions would be appreciated as I am stack.
Many Thanks!

 Could paste the code you use the start your training job and the fine-tuning script you run?


Multiprocessing windows error

Multiprocessing windows error
 As described here https://huggingface.co/blog/fine-tune-xlsr-wav2vec2

When using the num_proc argument on windows the whole Python environment crashes and hanging in loop.
For example at the map_to_array part.
An error occures because the cache file already exists and windows throws and error. After this the log crashes into an loop
 Hi ! Thanks for reporting.
This looks like a bug, could you try to provide a minimal code example that reproduces the issue ? This would be very helpful !

Otherwise I can try to run the wav2vec2 code above on my side but probably not this week..


Multiprocessing windows error

Multiprocessing windows error
 As described here https://huggingface.co/blog/fine-tune-xlsr-wav2vec2

When using the num_proc argument on windows the whole Python environment crashes and hanging in loop.
For example at the map_to_array part.
An error occures because the cache file already exists and windows throws and error. After this the log crashes into an loop
 ```
from datasets import load_dataset

dataset = load_dataset('glue', 'mrpc', split='train')

updated_dataset = dataset.map(lambda example: {'sentence1': 'My sentence: ' + example['sentence1']}, num_proc=4)

```


Multiprocessing windows error

Multiprocessing windows error
 As described here https://huggingface.co/blog/fine-tune-xlsr-wav2vec2

When using the num_proc argument on windows the whole Python environment crashes and hanging in loop.
For example at the map_to_array part.
An error occures because the cache file already exists and windows throws and error. After this the log crashes into an loop

I was able to copy some of the shell
This is repeating every half second
Win 10, Anaconda with python 3.8, datasets installed from main branche
```

  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 287, in _fixup_main_from_path
    _check_not_importing_main()
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 116, in spawn_main
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 134, in _check_not_importing_main
    main_content = runpy.run_path(main_path,
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\runpy.py", line 265, in run_path
    exitcode = _main(fd, parent_sentinel)
    raise RuntimeError('''
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 125, in _main
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.    return _run_module_code(code, init_globals, run_name,
    prepare(preparation_data)

  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\runpy.py", line 97, in _run_module_code
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 236, in prepare
    _run_code(code, mod_globals, init_globals,
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\runpy.py", line 87, in _run_code
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 287, in _fixup_main_from_path
    exec(code, run_globals)
  File "F:\Codes\Python Apps\asr\test.py", line 6, in <module>
    updated_dataset = dataset.map(lambda example: {'sentence1': 'My sentence: ' + example['sentence1']}, num_proc=4)
    main_content = runpy.run_path(main_path,
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\datasets\arrow_dataset.py", line 1370, in map
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\runpy.py", line 265, in run_path
    with Pool(num_proc, initargs=(RLock(),), initializer=tqdm.set_lock) as pool:
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\context.py", line 119, in Pool
    return _run_module_code(code, init_globals, run_name,
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
    return Pool(processes, initializer, initargs, maxtasksperchild,
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\runpy.py", line 87, in _run_code
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\pool.py", line 212, in __init__
    exec(code, run_globals)
  File "F:\Codes\Python Apps\asr\test.py", line 6, in <module>
    self._repopulate_pool()
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\pool.py", line 303, in _repopulate_pool
    updated_dataset = dataset.map(lambda example: {'sentence1': 'My sentence: ' + example['sentence1']}, num_proc=4)
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\datasets\arrow_dataset.py", line 1370, in map
    return self._repopulate_pool_static(self._ctx, self.Process,
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\pool.py", line 326, in _repopulate_pool_static
    with Pool(num_proc, initargs=(RLock(),), initializer=tqdm.set_lock) as pool:
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\context.py", line 119, in Pool
    w.start()
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\process.py", line 121, in start
    return Pool(processes, initializer, initargs, maxtasksperchild,
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\pool.py", line 212, in __init__
    self._popen = self._Popen(self)
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\context.py", line 327, in _Popen
    self._repopulate_pool()
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\pool.py", line 303, in _repopulate_pool
    return Popen(process_obj)
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\popen_spawn_win32.py", line 45, in __init__
    return self._repopulate_pool_static(self._ctx, self.Process,
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\pool.py", line 326, in _repopulate_pool_static
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 154, in get_preparation_data
    _check_not_importing_main()
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 134, in _check_not_importing_main
    w.start()
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\process.py", line 121, in start
    raise RuntimeError('''
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...
```


Multiprocessing windows error

Multiprocessing windows error
 As described here https://huggingface.co/blog/fine-tune-xlsr-wav2vec2

When using the num_proc argument on windows the whole Python environment crashes and hanging in loop.
For example at the map_to_array part.
An error occures because the cache file already exists and windows throws and error. After this the log crashes into an loop
 Thanks this is really helpful !
I'll try to reproduce on my side and come back to you


Multiprocessing windows error

Multiprocessing windows error
 As described here https://huggingface.co/blog/fine-tune-xlsr-wav2vec2

When using the num_proc argument on windows the whole Python environment crashes and hanging in loop.
For example at the map_to_array part.
An error occures because the cache file already exists and windows throws and error. After this the log crashes into an loop
 if __name__ == '__main__':

This line before calling the map function stops the error but the script still repeats endless


Multiprocessing windows error

Multiprocessing windows error
 As described here https://huggingface.co/blog/fine-tune-xlsr-wav2vec2

When using the num_proc argument on windows the whole Python environment crashes and hanging in loop.
For example at the map_to_array part.
An error occures because the cache file already exists and windows throws and error. After this the log crashes into an loop
 Indeed you needed `if __name__ == '__main__'` since accoding to [this stackoverflow post](https://stackoverflow.com/a/18205006):

> On Windows the subprocesses will import (i.e. execute) the main module at start. You need to insert an if __name__ == '__main__': guard in the main module to avoid creating subprocesses recursively.

Regarding the hanging issue, can you try to update `dill` and `multiprocess` ?


Multiprocessing windows error

Multiprocessing windows error
 As described here https://huggingface.co/blog/fine-tune-xlsr-wav2vec2

When using the num_proc argument on windows the whole Python environment crashes and hanging in loop.
For example at the map_to_array part.
An error occures because the cache file already exists and windows throws and error. After this the log crashes into an loop
 ```
Traceback (most recent call last):
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\shutil.py", line 791, in move
    os.rename(src, real_dst)
FileExistsError: [WinError 183] Eine Datei kann nicht erstellt werden, wenn sie bereits vorhanden ist: 'D:\\huggingfacecache\\common_voice\\de\\6.1.0\\0041e06ab061b91d0a23234a2221e87970a19cf3a81b20901474cffffeb7869f\\tmpx9fl_jg8' -> 'D:\\huggingfacecache\\common_voice\\de\\6.1.0\\0041e06ab061b91d0a23234a2221e87970a19cf3a81b20901474cffffeb7869f\\cache-9b4f203a63742dfc.arrow'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 125, in _main
    prepare(preparation_data)
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 236, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 287, in _fixup_main_from_path
    main_content = runpy.run_path(main_path,
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\runpy.py", line 265, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "F:\Codes\Python Apps\asr\cvtrain.py", line 243, in <module>
    common_voice_train = common_voice_train.map(remove_special_characters, remove_columns=["sentence"])
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\datasets\arrow_dataset.py", line 1339, in map
    return self._map_single(
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\datasets\arrow_dataset.py", line 203, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\datasets\fingerprint.py", line 337, in wrapper
    out = func(self, *args, **kwargs)
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\datasets\arrow_dataset.py", line 1646, in _map_single
    shutil.move(tmp_file.name, cache_file_name)
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\shutil.py", line 805, in move
    copy_function(src, real_dst)
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\shutil.py", line 435, in copy2
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  0%|                                                                                | 0/27771 [00:00<?, ?ex/s]
 File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\shutil.py", line 264, in copyfile
    with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst:
OSError: [Errno 22] Invalid argument: 'D:\\huggingfacecache\\common_voice\\de\\6.1.0\\0041e06ab061b91d0a23234a2221e87970a19cf3a81b20901474cffffeb7869f\\cache-9b4f203a63742dfc.arrow'
```

I was adding freeze support before calling the mapping function like this
if __name__ == '__main__':
    freeze_support()
   dataset.map(....)


Multiprocessing windows error

Multiprocessing windows error
 As described here https://huggingface.co/blog/fine-tune-xlsr-wav2vec2

When using the num_proc argument on windows the whole Python environment crashes and hanging in loop.
For example at the map_to_array part.
An error occures because the cache file already exists and windows throws and error. After this the log crashes into an loop
 Usually OSError of an arrow file on windows means that the file is currently opened as a dataset object, so you can't overwrite it until the dataset object falls out of scope.
Can you make sure that there's no dataset object that loaded the `cache-9b4f203a63742dfc.arrow` file ?


Multiprocessing windows error

Multiprocessing windows error
 As described here https://huggingface.co/blog/fine-tune-xlsr-wav2vec2

When using the num_proc argument on windows the whole Python environment crashes and hanging in loop.
For example at the map_to_array part.
An error occures because the cache file already exists and windows throws and error. After this the log crashes into an loop
 Now I understand
The error occures because the script got restarted in another thread, so the object is already loaded.
Still don't have an idea why a new thread starts the whole script again


Only user permission of saved cache files, not group
 Hello,

It seems when a cached file is saved from calling `dataset.map` for preprocessing, it gets the user permissions and none of the user's group permissions. As we share data files across members of our team, this is causing a bit of an issue as we have to continually reset the permission of the files. Do you know any ways around this or a way to correctly set the permissions?
 Hi ! Thanks for reporting.

Currently there's no way to specify this.

When loading/processing a dataset, the arrow file is written using a temporary file. Then once writing is finished, it's moved to the cache directory (using `shutil.move` [here](https://github.com/huggingface/datasets/blob/f6b8251eb975f66a568356d2a40d86442c03beb9/src/datasets/arrow_dataset.py#L1646))

That means it keeps the permissions specified by the `tempfile.NamedTemporaryFile` object, i.e. `-rw-------` instead of `-rw-r--r--`. Improving this could be a nice first contribution to the library :)


Only user permission of saved cache files, not group
 Hello,

It seems when a cached file is saved from calling `dataset.map` for preprocessing, it gets the user permissions and none of the user's group permissions. As we share data files across members of our team, this is causing a bit of an issue as we have to continually reset the permission of the files. Do you know any ways around this or a way to correctly set the permissions?
 Hi @lhoestq,
I looked into this and yes you're right. The `NamedTemporaryFile` is always created with mode 0600, which prevents group from reading the file. Should we change the permissions of `tmp_file.name` [here](https://github.com/huggingface/datasets/blob/f6b8251eb975f66a568356d2a40d86442c03beb9/src/datasets/arrow_dataset.py#L1871) and [here](https://github.com/huggingface/datasets/blob/f6b8251eb975f66a568356d2a40d86442c03beb9/src/datasets/arrow_dataset.py#L1590), post creation to 0644 inorder for group and others to read it?


Only user permission of saved cache files, not group
 Hello,

It seems when a cached file is saved from calling `dataset.map` for preprocessing, it gets the user permissions and none of the user's group permissions. As we share data files across members of our team, this is causing a bit of an issue as we have to continually reset the permission of the files. Do you know any ways around this or a way to correctly set the permissions?
 Good idea :) we could even update the permissions after the file has been moved by shutil.move [here](https://github.com/huggingface/datasets/blob/f6b8251eb975f66a568356d2a40d86442c03beb9/src/datasets/arrow_dataset.py#L1899) and [here](https://github.com/huggingface/datasets/blob/f6b8251eb975f66a568356d2a40d86442c03beb9/src/datasets/arrow_dataset.py#L1646) actually.
Apparently they set the default 0600 for temporary files for security reasons, so let's update the umask only after the file has been moved


Only user permission of saved cache files, not group
 Hello,

It seems when a cached file is saved from calling `dataset.map` for preprocessing, it gets the user permissions and none of the user's group permissions. As we share data files across members of our team, this is causing a bit of an issue as we have to continually reset the permission of the files. Do you know any ways around this or a way to correctly set the permissions?
 Would it be possible to actually set the umask based on a user provided argument? For example, a popular usecase my team has is using a shared file-system for processing datasets. This may involve writing/deleting other files, or changing filenames, which a -rw-r--r-- wouldn't fix.


Only user permission of saved cache files, not group
 Hello,

It seems when a cached file is saved from calling `dataset.map` for preprocessing, it gets the user permissions and none of the user's group permissions. As we share data files across members of our team, this is causing a bit of an issue as we have to continually reset the permission of the files. Do you know any ways around this or a way to correctly set the permissions?
 Note that you can get the cache files of a dataset with the `cache_files` attributes.
Then you can `chmod` those files and all the other cache files in the same directory.

Moreover we can probably keep the same permissions after each transform. This way you just need to set the permissions once after doing `load_dataset` for example, and then all the new transformed cached files will have the same permissions.
What do you think ?


Only user permission of saved cache files, not group
 Hello,

It seems when a cached file is saved from calling `dataset.map` for preprocessing, it gets the user permissions and none of the user's group permissions. As we share data files across members of our team, this is causing a bit of an issue as we have to continually reset the permission of the files. Do you know any ways around this or a way to correctly set the permissions?
 This means we'll check the permissions of other `cache_files` already created for a dataset before setting permissions for new  `cache_files`?


Only user permission of saved cache files, not group
 Hello,

It seems when a cached file is saved from calling `dataset.map` for preprocessing, it gets the user permissions and none of the user's group permissions. As we share data files across members of our team, this is causing a bit of an issue as we have to continually reset the permission of the files. Do you know any ways around this or a way to correctly set the permissions?
 > This way you just need to set the permissions once after doing load_dataset for example, and then all the new transformed cached files will have the same permissions.

I was referring to this. Ensuring that newly generated `cache_files` have the same permissions


Only user permission of saved cache files, not group
 Hello,

It seems when a cached file is saved from calling `dataset.map` for preprocessing, it gets the user permissions and none of the user's group permissions. As we share data files across members of our team, this is causing a bit of an issue as we have to continually reset the permission of the files. Do you know any ways around this or a way to correctly set the permissions?
 Yes exactly

I imagine users can first do `load_dataset`, then chmod on the arrow files. After that all the new cache files could have the same permissions as the first arrow files. Opinions on this ?


Only user permission of saved cache files, not group
 Hello,

It seems when a cached file is saved from calling `dataset.map` for preprocessing, it gets the user permissions and none of the user's group permissions. As we share data files across members of our team, this is causing a bit of an issue as we have to continually reset the permission of the files. Do you know any ways around this or a way to correctly set the permissions?
 Sounds nice but I feel this is a sub-part of the approach mentioned by @siddk. Instead of letting the user set new permissions by itself first and then making sure newly generated files have same permissions why don't we ask the user initially only what they want? What are your thoughts?


Only user permission of saved cache files, not group
 Hello,

It seems when a cached file is saved from calling `dataset.map` for preprocessing, it gets the user permissions and none of the user's group permissions. As we share data files across members of our team, this is causing a bit of an issue as we have to continually reset the permission of the files. Do you know any ways around this or a way to correctly set the permissions?
 Yes sounds good. Should this be a parameter in `load_dataset` ? Or an env variable ? Or use the value of `os.umask` ?


Only user permission of saved cache files, not group
 Hello,

It seems when a cached file is saved from calling `dataset.map` for preprocessing, it gets the user permissions and none of the user's group permissions. As we share data files across members of our team, this is causing a bit of an issue as we have to continually reset the permission of the files. Do you know any ways around this or a way to correctly set the permissions?
 Ideally it should be a parameter in `load_dataset` but I'm not sure how important it is for the users (considering only important things should go into `load_dataset` parameters)