Introduce the model card

README.md

@@ -1,3 +1,314 @@
----
-license: apache-2.0
----
+---
+license: apache-2.0
+pipeline_tag: text-generation
+library_name: transformers
+tags:
+- code
+- sft
+- reasoning
+- fine-tuned
+- java
+- gpt-oss
+- bf16
+base_model:
+- openai/gpt-oss-20b
+base_model_relation: finetune
+datasets:
+- OpenCoder-LLM/opc-sft-stage1
+- OpenCoder-LLM/opc-sft-stage2
+metrics:
+- code_eval
+- accuracy
+model-index:
+- name: SonarSweep-java-gpt-oss-20b
+  results:
+  - task:
+      type: text-generation
+    dataset:
+      type: nuprl/MultiPL-E
+      name: MultiPL-HumanEval (Java)
+    metrics:
+    - name: pass@1
+      type: pass@1
+      value: 0.845
+      verified: false
+  - task:
+      type: text-generation
+    dataset:
+      type: nuprl/MultiPL-E
+      name: MultiPL-MBPP (Java)
+    metrics:
+    - name: pass@1
+      type: pass@1
+      value: 0.674
+      verified: false
+  - task:
+      type: text-generation
+    dataset:
+      type: nuprl/MultiPL-E
+      name: MultiPL-HumanEval (Python)
+    metrics:
+    - name: pass@1
+      type: pass@1
+      value: 0.4739
+      verified: false
+  - task:
+      type: text-generation
+    dataset:
+      type: nuprl/MultiPL-E
+      name: Sanitized-MBPP (Python)
+    metrics:
+    - name: pass@1
+      type: pass@1
+      value: 0.2989
+      verified: false
+  - task:
+      type: text-generation
+    dataset:
+      type: nuprl/MultiPL-E
+      name: MultiPL-MBPP (PHP)
+    metrics:
+    - name: pass@1
+      type: pass@1
+      value: 0.658
+      verified: false
+  - task:
+      type: text-generation
+    dataset:
+      type: nuprl/MultiPL-E
+      name: MultiPL-MBPP (TypeScript)
+    metrics:
+    - name: pass@1
+      type: pass@1
+      value: 0.73
+      verified: false
+  - task:
+      type: text-generation
+    dataset:
+      type: nuprl/MultiPL-E
+      name: MultiPL-MBPP (Go)
+    metrics:
+    - name: pass@1
+      type: pass@1
+      value: 0.369
+      verified: false
+  - task:
+      type: text-generation
+    dataset:
+      type: cais/mmlu
+      name: MMLU
+      split: test
+    metrics:
+    - name: accuracy
+      type: accuracy
+      value: 0.7812
+      verified: false
+---
+
+# SonarSweep Java gpt-oss-20b
+
+## Model Details
+
+### Model Description
+
+This is a fine-tuned version of [openai/gpt-oss-20b](https://huggingface.co/openai/gpt-oss-20b), optimized for high-quality Java code generation. [SonarSweep](https://www.sonarsource.com/products/sonarsweep/) was used to create a high-quality Java dataset. Fine-tuning on this dataset has produced a model that generates expert-level Java patterns while avoiding the bugs and vulnerabilities observed when benchmarking the base model.
+
+- **Developed by:** Sonar SonarSweep Team
+- **Model type:** Mixture of Experts
+- **Languages:** Primarily Java, English
+- **License:** Apache 2.0
+
+## Uses
+
+This model is designed primarily as a demonstration of our SonarSweep pipeline, specifically for fine-tuning on Java.
+
+By using SonarSweep for targeted data preprocessing, pass@1 metrics are maintained across all coding benchmarks, while the number of bugs and vulnerabilities is significantly reduced compared to the base model.
+
+Despite being a demonstration model, we have tested the model to ensure its responses are helpful, natural, and adhere to instructions. We have evaluated a range of benchmarks across software engineering and general use cases to ensure the model remains widely useful.
+
+We focus on `gpt-oss-20b` in the "low" reasoning setting, which we refer to as `gpt-oss-20b-low`. In this setting, the base model responds quickly and achieves relatively high scores in code generation benchmarks. This configuration would be appropriate, for example, for developers using an LLM for in-line completions.
+
+As with the base model, the fine-tuned model was trained on data in OpenAI's [harmony response format](https://github.com/openai/harmony). The model should only be used with the harmony format, as it will not work correctly otherwise.
+
+Technical recommendations for usage are included below; see [Getting Started](#getting-started).
+
+### Reasoning Capabilities
+
+This model operates exclusively as a low-reasoning model, derived from `gpt-oss-20b-low`. It is optimized for speed and standard conversational tasks rather than complex chain-of-thought processing.
+
+Please note that specifying or adjusting reasoning effort is not supported. Any parameters attempting to enforce "medium" or "high" reasoning settings will be ignored or may result in an error. The model is hard-coded to a low-reasoning profile.
+
+## Bias, Risks, and Limitations
+
+Despite being trained to output high-quality Java code and achieving substantial improvements on our [code quality benchmarks](#code-quality), the model is still liable to generate bugs and security vulnerabilities. Users must never treat generated code as production-ready without a thorough review using static analysis—for example, using [SonarQube](https://www.sonarsource.com/products/sonarqube/).
+
+Our model's (and the base model's) knowledge is static, based on its training data cutoff. We cannot guarantee adherence to the latest Java standards, best practices in newly released libraries, or correct use of private or proprietary APIs.
+
+## Getting Started
+
+You can use the `SonarSweep-java-gpt-oss-20b` model with Transformers. If you use the Transformers chat template, it will automatically apply the harmony response format. If you use `model.generate` directly, you need to apply the harmony format manually using the chat template.
+
+### Minimum Requirements
+
+- **GPU Memory:** 48GB+ VRAM required for the model loaded in bf16 precision
+- **Storage:** 100GB
+
+To get started, install the necessary dependencies to set up your environment:
+
+```shell
+pip install -U transformers kernels torch
+```
+
+Once installed, you can run the model using the following snippet:
+
+```python
+from transformers import pipeline
+
+model_id = "SonarSource/SonarSweep-java-gpt-oss-20b"
+
+pipe = pipeline(
+    "text-generation",
+    model=model_id,
+    torch_dtype="auto",
+    device_map="auto",
+)
+
+messages = [
+    {"role": "user", "content": "Write a function in Java that creates a two-dimensional array with 5 rows and 2 columns, each element of which is a random number between 1 and 50."},
+]
+
+outputs = pipe(
+    messages,
+    max_new_tokens=2048,
+)
+print(outputs[0]["generated_text"][-1])
+```
+
+For more details see [here](https://huggingface.co/openai/gpt-oss-20b#transformers).
+
+## Training Details
+
+### Training Data
+
+We compiled open-source code data from [OpenCoder Datasets](https://huggingface.co/collections/OpenCoder-LLM/opencoder-datasets) and synthetic alignment data generated using [openai/gpt-oss-120b](https://huggingface.co/openai/gpt-oss-120b) to create a Java dataset of 70k examples. We then used SonarSweep to improve the quality of the dataset.
+
+### Training Hyperparameters
+
+We trained LoRA adapters across all linear layers of the experts and attention blocks.
+
+| Parameter | Value |
+|-----------|-------|
+| Batch Size | 64 |
+| Training Epochs | 2 |
+| Learning Rate | 1e-4 |
+| LR Scheduler | Cosine with 10% Warmup |
+| LoRA Rank | 64 |
+| LoRA Alpha | 128 |
+| Attention Mechanism | [SDPA](https://huggingface.co/docs/transformers/v4.57.3/en/perf_infer_gpu_one#scaled-dot-product-attention-sdpa) |
+| Precision | bf16 mixed precision |
+
+### Model Architecture
+
+| Property | Value |
+|----------|-------|
+| Architecture | gpt-oss (Transformer-based Mixture of Experts) |
+| Parameters | 20.9 billion (3.6 billion active) |
+| Trainable Parameters | 740M (3.4% of total) |
+
+## Evaluation
+
+### Code Quality
+
+We used SonarQube to evaluate the quality, verbosity, and complexity of Java code generated for the [ComplexCodeEval](https://github.com/ComplexCodeEval/ComplexCodeEval) and [MultiPL-E Java](https://huggingface.co/datasets/nuprl/MultiPL-E/viewer/humaneval-java) benchmarks.
+
+The fine-tuned and base models achieve a similar pass@1 metric for code generation (within 1% difference). Results for other languages are shown in the next subsection.
+
+The fine-tuned model achieves this metric while generating fewer lines of code.
+
+For code quality, we see a dramatic reduction in both the number and density of Sonar issues, split among bugs, security vulnerabilities, and code smells (see the [Glossary](#glossary) for definitions).
+
+| Metric | Base Model | Fine-tuned Model |
+|--------|------------|------------------|
+| MultiPL-E Pass@1 | 71.49 | 72.37 |
+| Lines of Code Generated | 247,895 | 233,031 |
+| Bugs Generated | 222 | 123 |
+| Bugs per KLOC | 0.9 | 0.53 |
+| Security Vulnerabilities | 102 | 56 |
+| Vulnerabilities per KLOC | 0.41 | 0.24 |
+| Code Smells | 4,968 | 3,796 |
+| Code Smells per KLOC | 20.04 | 16.29 |
+
+For cyclomatic and cognitive complexity, after fine-tuning, there was a reduction in both total complexity and complexity per thousand lines of code.
+
+| Complexity Metric | Base Model | Fine-tuned Model |
+|-------------------|------------|------------------|
+| Cyclomatic (Total) | 52,139 | 45,006 |
+| Cyclomatic (per KLOC) | 210.33 | 193.13 |
+| Cognitive (Total) | 30,871 | 24,419 |
+| Cognitive (per KLOC) | 124.53 | 104.79 |
+
+Note: KLOC = Thousand Lines of Code.
+
+### Code Generation
+
+MultiPL-E provides a multilanguage parallel benchmark for evaluating the performance of LLMs on natural-language-to-code generation tasks. For each language, they provide a translated version of HumanEval and MBPP.
+
+We fine-tuned on Java but chose to evaluate a selection of available languages. For all languages, scores are averages of 10 samples with temperature set to 0.01.
+
+#### Results
+
+The changes are not significant for any language; this demonstrates that we do not see performance degradation in the fine-tuned model's ability to generate functional code.
+
+| Language | Dataset | Num Examples | Base Model Pass@1 | Fine-tuned Model Pass@1 |
+|----------|---------|--------------|-------------------|-------------------------|
+| Java | HumanEval | 158 | 85.40% | 84.50% |
+| Java | MBPP | 386 | 65.80% | 67.40% |
+| Python | HumanEval | 164 | 43.06% | 47.39% |
+| Python | MBPP* | 257 | 26.50% | 29.89% |
+| PHP | MBPP | 397 | 63.20% | 65.80% |
+| TypeScript | MBPP | 390 | 74.10% | 73.00% |
+| Go | MBPP | 374 | 35.00% | 36.90% |
+
+\* This is the [sanitized MBPP benchmark](https://github.com/google-research/google-research/tree/master/mbpp) from the original [Google Research paper](https://arxiv.org/abs/2108.07732).
+
+### General Ability: MMLU
+
+The MMLU (Massive Multitask Language Understanding) benchmark evaluates the model's general knowledge with 14,042 multiple-choice questions across a wide range of subjects.
+
+| Metric | Base Model | Fine-tuned Model |
+|--------|------------|------------------|
+| Correct Answers | 11,081 | 10,969 |
+| Accuracy | 78.91% | 78.12% |
+
+The fine-tuned model maintains comparable performance on MMLU with only a 0.79% decrease in accuracy, demonstrating that specialization in Java code quality does not significantly impact general knowledge capabilities.
+
+## Glossary
+
+**SonarSweep:** A pipeline that analyzes and remediates code for training datasets. For more details, see the announcement on [sonarsource.com](https://www.sonarsource.com/products/sonarsweep/).
+
+**Lines of Code (LOC):** The number of lines of code generated, not counting comments or blank lines. For our scale, KLOC (thousand lines of code) is more appropriate.
+
+**Code Quality:** Using static analysis, SonarQube specifically monitors and measures three core [software qualities](https://docs.sonarsource.com/sonarqube-server/user-guide/rules/software-qualities), each of which has an associated type of issue:
+
+- **Security:** The protection of your software from unauthorized access, use, or destruction. Detected security issues are called *Vulnerabilities*.
+- **Reliability:** A measure of how your software is capable of maintaining its level of performance under stated conditions for a stated period of time. Detected reliability issues are called *Bugs*.
+- **Maintainability:** Refers to the ease with which you can repair, improve, and understand software code. Detected maintainability issues are called *Code Smells*.
+
+SonarQube analysis for Java supports the detection of a wide range of quality issues. For details, see [rules.sonarsource.com](https://rules.sonarsource.com/java/).
+
+## Acknowledgements
+
+- **Base Model:** [openai/gpt-oss-20b](https://huggingface.co/openai/gpt-oss-20b)
+- **Code Quality Assessment Methodology:** [Assessing the Quality and Security of AI-Generated Code: A Quantitative Analysis](https://arxiv.org/abs/2508.14727)
+- **Benchmarks:**
+  - ComplexCodeEval: [https://arxiv.org/abs/2409.10280](https://arxiv.org/abs/2409.10280)
+  - MultiPL-E: [https://arxiv.org/abs/2208.08227](https://arxiv.org/abs/2208.08227)
+  - MMLU: [https://arxiv.org/abs/2009.03300](https://arxiv.org/abs/2009.03300)
+  - MBPP: [https://arxiv.org/abs/2108.07732](https://arxiv.org/abs/2108.07732) | [Code](https://github.com/google-research/google-research/tree/master/mbpp)
+- **OpenCoder:** [https://opencoder-llm.github.io/](https://opencoder-llm.github.io/)
+
+## Model Card Authors
+
+SonarSweep Team
+
+For feedback: [https://community.sonarsource.com/](https://community.sonarsource.com/)