llm-wizard/legal-ft-arctic-l

SentenceTransformer based on Snowflake/snowflake-arctic-embed-l

This is a sentence-transformers model finetuned from Snowflake/snowflake-arctic-embed-l. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: Snowflake/snowflake-arctic-embed-l
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 1024 tokens
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("llm-wizard/legal-ft-arctic-l")
# Run inference
sentences = [
    "What recent partnership did News Corp enter into regarding licensing content for OpenAI's applications?",
    'licensing initiatives abound.”3 For example, News Corp recently partnered with OpenAI to license \nits content for certain uses in OpenAI’s applications. OpenAI users will have the benefit of \naccessing Plaintiffs’ content, whether quoted or summarized by OpenAI. This cooperative \nrelationship will allow OpenAI and Plaintiffs to experiment with new product experiences and \nrevenue models. \n15. \nGenerative AI technology can be developed in two ways. It can be developed \nlegally by recognizing the legitimate rights of copyright holders and by including in the AI business \nmodel the legitimate costs and benefits of licensing the copyrighted material, or it can be developed',
    'integrity infractions.   Plain and simple.  It should not take the Plaintiffs engaging counsel, \ndemanding information and forcing Hingham to investigate this matter to reveal that selection for \nNHS was a manipulated sham conducted by the Defendants, who at all times relevant were state \nactors. \nC. The Student Will Suffer Irreparable Harm If The Injunction is Not Granted \nIn order for the Plaintiffs to obtain injunctive relief, they must show that they are "likely to \nsuffer irreparable injury before a decision is rendered on the merits." See Philips Elecs. N. Am. \nCorp. v. Halperin, 2000 Mass. Super LEXIS 574 citing Sierra Club v. Larson, 769 F. Supp. 420,',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.6875
cosine_accuracy@3	0.8542
cosine_accuracy@5	0.9583
cosine_accuracy@10	0.9792
cosine_precision@1	0.6875
cosine_precision@3	0.2847
cosine_precision@5	0.1917
cosine_precision@10	0.0979
cosine_recall@1	0.6875
cosine_recall@3	0.8542
cosine_recall@5	0.9583
cosine_recall@10	0.9792
cosine_ndcg@10	0.8281
cosine_mrr@10	0.7794
cosine_map@100	0.7813
dot_accuracy@1	0.6875
dot_accuracy@3	0.8542
dot_accuracy@5	0.9583
dot_accuracy@10	0.9792
dot_precision@1	0.6875
dot_precision@3	0.2847
dot_precision@5	0.1917
dot_precision@10	0.0979
dot_recall@1	0.6875
dot_recall@3	0.8542
dot_recall@5	0.9583
dot_recall@10	0.9792
dot_ndcg@10	0.8281
dot_mrr@10	0.7794
dot_map@100	0.7813

Training Details

Training Dataset

Unnamed Dataset

• Size: 400 training samples
• Columns: sentence_0 and sentence_1
• Approximate statistics based on the first 400 samples:

  	sentence_0	sentence_1
  type	string	string
  details	min: 10 tokens mean: 20.73 tokens max: 34 tokens	min: 25 tokens mean: 140.37 tokens max: 260 tokens

• Samples:

  sentence_0	sentence_1
  How does Perplexity's business model differ from that of traditional search engines?	11. Perplexity’s business is fundamentally distinct from that of traditional search engines that also copy a vast amount of content into their indices but do so merely to provide links to the originating sites. In its traditional form, a search engine is a tool for discovery, pointing searchers to websites such as the pages of The Wall Street Journal or the New York Post, where the users can click to find the information and answers they seek. Those clicks in turn provide revenue for content producers. In part because traditional search engines that simply provide hyperlinks promote merely the discovery of copyrighted content, and not its substitution (and commercial
  What role do clicks on traditional search engines play in the revenue generation for content producers?	11. Perplexity’s business is fundamentally distinct from that of traditional search engines that also copy a vast amount of content into their indices but do so merely to provide links to the originating sites. In its traditional form, a search engine is a tool for discovery, pointing searchers to websites such as the pages of The Wall Street Journal or the New York Post, where the users can click to find the information and answers they seek. Those clicks in turn provide revenue for content producers. In part because traditional search engines that simply provide hyperlinks promote merely the discovery of copyrighted content, and not its substitution (and commercial
  Who were the founders of Dow Jones?	founded by reporters Charles Dow, Edward Jones, and Charles Bergstresser. Publishing the first edition of The Wall Street Journal in July 1889, Dow Jones has now expanded into a worldwide news powerhouse. It creates and distributes some of the most widely recognized and reputable publications in the news industry, including, in addition to The Wall Street Journal, Dow Jones Newswires, MarketWatch, Financial News, and Barron’s. 29. Dow Jones is a trusted source of accurate, original news stories, data and analytics, and financial and business insight for millions of customers across the country and around the world. 30. A recipient of 39 Pulitzer Prizes, the award-winning newsroom at The Wall Street

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          768,
          512,
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 10
• per_device_eval_batch_size: 10
• num_train_epochs: 10
• multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 10
• per_device_eval_batch_size: 10
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1
• num_train_epochs: 10
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.0
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• eval_use_gather_object: False
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: round_robin

Training Logs

Epoch	Step	cosine_map@100
1.0	40	0.7519
1.25	50	0.8072
2.0	80	0.7892
2.5	100	0.7949
3.0	120	0.7850
3.75	150	0.7537
4.0	160	0.7905
5.0	200	0.7650
6.0	240	0.7860
6.25	250	0.7806
7.0	280	0.7819
7.5	300	0.7820
8.0	320	0.7820
8.75	350	0.7821
9.0	360	0.7823
10.0	400	0.7813

Framework Versions

• Python: 3.10.12
• Sentence Transformers: 3.2.1
• Transformers: 4.44.2
• PyTorch: 2.4.1+cu121
• Accelerate: 0.34.2
• Datasets: 3.0.2
• Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}