# Financial PhraseBank — Llama 3

This benchmark was generated by LMTask from
`trainconf/fpb-llama3.yml` on 2026-08-08T13:24:56.393661-05:00.


## Results

**Primary metric:** f1 (macro) = **0.8688**

| Metric    | Average  |  Value |
|-----------|----------|-------:|
| accuracy  | —        | 0.8867 |
| precision | micro    | 0.8867 |
| recall    | micro    | 0.8867 |
| f1        | micro    | 0.8867 |
| precision | macro    | 0.9211 |
| recall    | macro    | 0.8338 |
| f1        | macro    | 0.8688 |
| precision | weighted | 0.8936 |
| recall    | weighted | 0.8867 |
| f1        | weighted | 0.8830 |


### Per-class results

| Class    | Precision | Recall |     F1 | Support |
|----------|----------:|-------:|-------:|--------:|
| negative |    0.9655 | 0.8235 | 0.8889 |      34 |
| neutral  |    0.8603 | 0.9747 | 0.9139 |     158 |
| positive |    0.9375 | 0.7031 | 0.8036 |      64 |


Invalid/unrecognized predictions: **0**


## Dataset

| Split      | Examples | Class counts                                   |
|------------|---------:|------------------------------------------------|
| train      |     1752 | `positive`=441, `neutral`=1076, `negative`=235 |
| validation |      256 | `neutral`=157, `negative`=34, `positive`=65    |
| test       |      256 | `neutral`=158, `negative`=34, `positive`=64    |


## Training

| Field                    | Value                        |
|--------------------------|------------------------------|
| Global steps             | 110                          |
| Training loss            | 1.273833557692441            |
| End-to-end training time | 368.0 s                      |
| PEFT output              | `data/fpb/llama3/model/peft` |
| Adapter size             | 1410003096 bytes             |


### Trainer diagnostics

| Metric                   |              Value |
|--------------------------|-------------------:|
| train_runtime            |           353.3802 |
| train_samples_per_second |              4.958 |
| train_steps_per_second   |              0.311 |
| total_flos               | 9859320980029440.0 |
| train_loss               |  1.273833557692441 |


## Held-out testing

| Field       | Value                                        |
|-------------|----------------------------------------------|
| Examples    | 256                                          |
| Runtime     | 35.6 s                                       |
| Predictions | `data/fpb/llama3/benchmark/fpb_llama3.jsonl` |


## Environment

| Field                | Value                                                      |
|----------------------|------------------------------------------------------------|
| LMTask revision      | `51cf677d65e9b7c9deffa2878c505dba82b59d89`                 |
| Git describe         | `v0.0.1-105-g51cf677`                                      |
| Dirty working tree   | True                                                       |
| Python               | 3.13.13                                                    |
| OS                   | Linux-4.18.0-553.124.1.el8_10.x86_64-x86_64-with-glibc2.28 |
| Kernel               | 4.18.0-553.124.1.el8_10.x86_64                             |
| CUDA_VISIBLE_DEVICES | `4,5,6,7`                                                  |
| PyTorch CUDA runtime | 12.8                                                       |
| NVIDIA driver        | 580.142                                                    |

### GPUs visible to the benchmark process

| Device | GPU              |      Memory | Peak allocated | Peak reserved |
|-------:|------------------|------------:|---------------:|--------------:|
|      0 | NVIDIA RTX A6000 | 50897289216 |              — |             — |
|      1 | NVIDIA RTX A6000 | 50897289216 |              — |             — |
|      2 | NVIDIA RTX A6000 | 50897289216 |              — |             — |
|      3 | NVIDIA RTX A6000 | 50897289216 |              — |             — |


### Package versions

| Package         | Version |
|-----------------|---------|
| scikit-learn    | 1.9.0   |
| torch           | 2.10.0  |
| transformers    | 5.5.4   |
| trl             | 1.0.0   |
| peft            | 0.18.1  |
| accelerate      | 1.13.0  |
| datasets        | 4.8.5   |
| pandas          | 2.3.3   |
| pyarrow         | 24.0.0  |
| huggingface-hub | 1.16.1  |


## Reproducibility

The JSON record `fpb_llama3.json` is the machine-readable source of
truth for this report.  The held-out predictions are stored in
`fpb_llama3.jsonl`.