Update README.md
Browse files
README.md
CHANGED
|
@@ -1,71 +1,115 @@
|
|
| 1 |
---
|
| 2 |
-
|
| 3 |
license: apache-2.0
|
| 4 |
-
|
| 5 |
-
|
| 6 |
-
|
|
|
|
| 7 |
metrics:
|
| 8 |
- accuracy
|
| 9 |
-
- f1
|
| 10 |
-
- precision
|
| 11 |
-
- recall
|
| 12 |
-
model-index:
|
| 13 |
-
- name: finetuned_model
|
| 14 |
-
results: []
|
| 15 |
---
|
| 16 |
|
| 17 |
-
|
| 18 |
-
should probably proofread and complete it, then remove this comment. -->
|
| 19 |
|
| 20 |
-
|
| 21 |
|
| 22 |
-
This
|
| 23 |
-
It achieves the following results on the evaluation set:
|
| 24 |
-
- Loss: 0.0000
|
| 25 |
-
- Accuracy: 1.0
|
| 26 |
-
- F1: 1.0
|
| 27 |
-
- Precision: 1.0
|
| 28 |
-
- Recall: 1.0
|
| 29 |
|
| 30 |
-
## Model
|
| 31 |
|
| 32 |
-
|
| 33 |
|
| 34 |
-
|
| 35 |
|
| 36 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 37 |
|
| 38 |
-
##
|
| 39 |
|
| 40 |
-
|
| 41 |
|
| 42 |
-
|
| 43 |
|
| 44 |
-
###
|
| 45 |
|
| 46 |
-
|
| 47 |
-
- learning_rate: 2e-05
|
| 48 |
-
- train_batch_size: 8
|
| 49 |
-
- eval_batch_size: 8
|
| 50 |
-
- seed: 42
|
| 51 |
-
- optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
|
| 52 |
-
- lr_scheduler_type: linear
|
| 53 |
-
- num_epochs: 5
|
| 54 |
|
| 55 |
-
|
| 56 |
|
| 57 |
-
|
| 58 |
-
|:-------------:|:-----:|:----:|:---------------:|:--------:|:------:|:---------:|:------:|
|
| 59 |
-
| 0.0002 | 1.0 | 80 | 0.0710 | 0.9875 | 0.9875 | 0.9878 | 0.9875 |
|
| 60 |
-
| 0.0001 | 2.0 | 160 | 0.0001 | 1.0 | 1.0 | 1.0 | 1.0 |
|
| 61 |
-
| 0.0001 | 3.0 | 240 | 0.0001 | 1.0 | 1.0 | 1.0 | 1.0 |
|
| 62 |
-
| 0.0001 | 4.0 | 320 | 0.0000 | 1.0 | 1.0 | 1.0 | 1.0 |
|
| 63 |
-
| 0.0001 | 5.0 | 400 | 0.0000 | 1.0 | 1.0 | 1.0 | 1.0 |
|
| 64 |
|
|
|
|
| 65 |
|
| 66 |
-
|
| 67 |
|
| 68 |
-
|
| 69 |
-
|
| 70 |
-
|
| 71 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
+
'[object Object]': null
|
| 3 |
license: apache-2.0
|
| 4 |
+
datasets:
|
| 5 |
+
- maryzhang/hw1-24679-image-dataset
|
| 6 |
+
language:
|
| 7 |
+
- en
|
| 8 |
metrics:
|
| 9 |
- accuracy
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 10 |
---
|
| 11 |
|
| 12 |
+
# Model Card for {{ model_id | default("Model ID", true) }}
|
|
|
|
| 13 |
|
| 14 |
+
<!-- Provide a quick summary of what the model is/does. -->
|
| 15 |
|
| 16 |
+
This is finetuned version of DistilBERT that is used for sentiment analysis on NFL news titles.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 17 |
|
| 18 |
+
## Model Details
|
| 19 |
|
| 20 |
+
### Model Description
|
| 21 |
|
| 22 |
+
This model uses the DistilBERT model to classify NFL news article titles as positive or negative.
|
| 23 |
|
| 24 |
+
- **Developed by:** Devin DeCosmo
|
| 25 |
+
- **Model type:** Binary Sentiment Analysis
|
| 26 |
+
- **Language(s) (NLP):** English
|
| 27 |
+
- **License:** MIT
|
| 28 |
+
- **Finetuned from model:** DistilBERT
|
| 29 |
|
| 30 |
+
## Uses
|
| 31 |
|
| 32 |
+
<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
|
| 33 |
|
| 34 |
+
This is used for sentiment analysis of NFL articles, but could possibly be used for other article titles.
|
| 35 |
|
| 36 |
+
### Direct Use
|
| 37 |
|
| 38 |
+
<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 39 |
|
| 40 |
+
The direct use is to classify NFL articles as positive or negative.
|
| 41 |
|
| 42 |
+
### Out-of-Scope Use
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 43 |
|
| 44 |
+
<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
|
| 45 |
|
| 46 |
+
If the dataset was expanded, this could be used for sentiment analysis on other types of articles or find other features like bias towards a team or player.
|
| 47 |
|
| 48 |
+
## Bias, Risks, and Limitations
|
| 49 |
+
|
| 50 |
+
<!-- This section is meant to convey both technical and sociotechnical limitations. -->
|
| 51 |
+
|
| 52 |
+
This is trained off a small dataset of 100 titles, this small dataset could be liable to overfitting and is not robust.
|
| 53 |
+
|
| 54 |
+
### Recommendations
|
| 55 |
+
|
| 56 |
+
<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
|
| 57 |
+
|
| 58 |
+
The small dataset size means this model is not highly generalizable.
|
| 59 |
+
|
| 60 |
+
## How to Get Started with the Model
|
| 61 |
+
|
| 62 |
+
Use the code below to get started with the model.
|
| 63 |
+
|
| 64 |
+
## Training Details
|
| 65 |
+
|
| 66 |
+
### Training Data
|
| 67 |
+
|
| 68 |
+
<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
|
| 69 |
+
|
| 70 |
+
James-kramer/football_news
|
| 71 |
+
|
| 72 |
+
This is the training dataset used.
|
| 73 |
+
It consists of 100 original titles used for validation along with 1000 synthetic pieces of data from training.
|
| 74 |
+
|
| 75 |
+
### Training Procedure
|
| 76 |
+
|
| 77 |
+
<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
|
| 78 |
+
This model was trained with DistilBERT using binary classification, a training split of 80%, and 5 epochs.
|
| 79 |
+
I initially used more but this converged extremely quickly.
|
| 80 |
+
|
| 81 |
+
|
| 82 |
+
## Evaluation
|
| 83 |
+
|
| 84 |
+
<!-- This section describes the evaluation protocols and provides the results. -->
|
| 85 |
+
|
| 86 |
+
### Testing Data, Factors & Metrics
|
| 87 |
+
|
| 88 |
+
#### Testing Data
|
| 89 |
+
|
| 90 |
+
<!-- This should link to a Dataset Card if possible. -->
|
| 91 |
+
James-kramer/football_news
|
| 92 |
+
The testing data was the 'original' split, the 100 original titles in this set.
|
| 93 |
+
|
| 94 |
+
#### Factors
|
| 95 |
+
|
| 96 |
+
<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
|
| 97 |
+
|
| 98 |
+
This dataset is evaluating whether the food is positive, "1", or negative, "0".
|
| 99 |
+
|
| 100 |
+
#### Metrics
|
| 101 |
+
|
| 102 |
+
<!-- These are the evaluation metrics being used, ideally with a description of why. -->
|
| 103 |
+
|
| 104 |
+
The testing metric used was accuracy to ensure the highest accuracy of the model possible.
|
| 105 |
+
I also considered testing time. This small langauge model ran extremely quickly with 102 steps per second.
|
| 106 |
+
|
| 107 |
+
### Results
|
| 108 |
+
|
| 109 |
+
After training with the initial dataset, this model reached an accuracy of 100% in validation.
|
| 110 |
+
This is likely due to the simplicity of the task, binary classification, along with distilBERT being made for tasks such as this.
|
| 111 |
+
|
| 112 |
+
#### Summary
|
| 113 |
+
|
| 114 |
+
This model reached a high accuracy with our current model, but this perfomance can not be confirmed to continue as the dataset was very small.
|
| 115 |
+
Additional testing with more samples would be highly beneficial.
|