Try
↑↓Move↵OpenCtrl↵New tabescClose

Datasets · Multi-label classification

JS/TS Code Review Decisions

12,000 JavaScript and TypeScript snippets labeled for 10 code review flaws. A 13-question benchmark scores small models on the test split.

View on Hugging Face ↗ (opens in a new tab)Load it

Overview

The dataset has training data and a benchmark for small models that review JavaScript and TypeScript code. Each of 3,000 coding tasks was written four times, once clean and three times with planted flaws. The decisions config turns each snippet into 13 questions, and the model returns a probability for each answer.

Snippets
12,000
Line-level findings
27,450
Questions per snippet
13
Best macro F1
0.917
Task
Multi-label classification
Splits
train 9,600 · val 1,200 · test 1,200
Format
Parquet · 12.9 MB
Published
Downloads
200 all time

Benchmark

Each model was fine-tuned on the same 9,600 training snippets. The dataset’s score_decisions.py scored it on the 1,200-snippet test split, and each request held all 13 questions. All four variants of a task are in the same split, and no test task appears in training.

Higher is better. The mean F1 over the 10 flaw questions at a 0.5 cutoff. It is the primary metric.

  1. d1-3B on Hugging Face (opens in a new tab)3.1B · LFM2.5-VL-3B with Liquid AI’s d1 decision readout0.917
  2. d1-omni-600M on Hugging Face (opens in a new tab)587M · LFM2.5 encoder with a decision head0.771
  3. GLiNER2.5-Decide on Hugging Face (opens in a new tab)486M · GLiNER2 DeBERTa-v3 encoder with a classification head0.687
  4. TF-IDF + logistic regressionBaseline fitted on the train split0.502
  5. NanoJev on Hugging Face (opens in a new tab)596M · Qwen3-0.6B with a choice head0.452
  6. UniformGives every option the same probability0.237
  7. PriorPredicts the train label frequencies and ignores the input0.000
  • Fine-tuned on train
  • Baseline
  • Reference

Fine-tuning

Macro F1 for each base model before training and after one pass over the 9,600 training snippets.

  • d1-3BZero-shot 0.220fine-tuned 0.917
  • d1-omni-600MZero-shot 0.175fine-tuned 0.771
  • GLiNER2.5-DecideZero-shot 0.071fine-tuned 0.687
  • NanoJevZero-shot 0.237fine-tuned 0.452
  • Base model, zero-shot
  • Fine-tuned

Scores by question

Every model scores above 0.91 on naming and comments. NanoJev scores 0 on over-abstraction and duplicated logic. The other three differ most on dead code, inconsistent style, and unsafe input.

Test F1 at a 0.5 cutoff for each yes or no question, and accuracy for the primary flaw and quality questions, by model.
Questiond1-3B3.1Bd1-omni-600M587MGLiNER2.5-Decide486MNanoJev596M
Unclear namingF1 · 139 positives0.9640.9170.9260.933
Noise commentsF1 · 178 positives0.9970.9970.9620.992
Dead codeF1 · 223 positives0.8860.6320.4320.414
Inconsistent styleF1 · 147 positives0.9030.5380.4780.428
Over-abstractionF1 · 145 positives0.9520.8550.8310.000
Too cleverF1 · 164 positives0.9290.8540.6570.185
Duplicated logicF1 · 171 positives0.8670.7200.6340.000
Error handlingF1 · 140 positives0.9460.9050.7800.792
Type safety holeF1 · 148 positives0.8790.7420.6700.527
Unsafe inputF1 · 165 positives0.8450.5460.4960.252
Has any flawF1 · 900 positives0.9630.8790.8600.857
Primary flawAccuracy · choice0.8120.6420.5930.491
QualityAccuracy · score0.6840.5290.5290.446
01Darker cells score lower. Positives are snippets in the 1,200-snippet test split with that flaw.

Macro F1 against CPU time

On 16 CPU cores, the fine-tuned models take 6 to 18 seconds per snippet and TF-IDF takes 1.2 ms. On one H100, d1-3B takes 30.5 ms, d1-omni-600M 41.2 ms, and NanoJev 115.4 ms.

Median time per snippet with all 13 questions, batch size 1
Model16 Neoverse V3 coresGPU
d1-3B8.9 s30.5 ms on one H100 SXM
d1-omni-600M6.3 s41.2 ms on one H100 SXM
GLiNER2.5-Decide6.0 s107.7 ms on one RTX PRO 6000
NanoJev18.2 s115.4 ms on one H100 SXM
TF-IDF + logistic regression1.2 ms—

Snippets

The 12,000 snippets come from 3,000 tasks across 40 domains and 13 code contexts. The packaging script fails if a task appears in two splits or two snippets are identical.

Flaws

Each bar counts the snippets with that flaw, out of 12,000. A snippet can have more than one flaw.

  1. Dead codeCommented-out code, unused variables or imports, unreachable branches, leftover console.log.2,340
  2. Duplicated logicThe same logic two or more times with small variations.1,757
  3. Noise commentsAny comment other than a tool directive.1,743
  4. Too cleverNested ternaries, one reduce doing three jobs, bitwise tricks.1,622
  5. Unsafe inputRequest bodies, env vars, or API responses used unchecked, and string-built SQL, innerHTML, or eval.1,622
  6. Error handlingEmpty catch, a catch that logs and carries on, errors turned into null.1,472
  7. Type safety holeany, non-null !, as casts on unchecked values, @ts-ignore.1,447
  8. Inconsistent stylecamelCase next to snake_case, async/await mixed with .then.1,437
  9. Unclear namingSingle letters outside short callbacks, names like data or temp.1,414
  10. Over-abstractionAn interface with one implementation, a factory for one case.1,337

Quality

Quality runs from 1 (should not merge) to 5 (exemplary). Every clean variant scores 4 or 5.

  1. 1,2501: Should not merge, 0 clean
  2. 3,1182: Needs significant changes, 0 clean
  3. 3,6123: Mergeable with review comments, 0 clean
  4. 1,1004: Mergeable with minor comments, 80 clean
  5. 2,9205: Exemplary, 2,920 clean
FlawedClean

Languages

Each bar is a share of all 12,000 snippets.

  1. TypeScript51.7%
  2. JavaScript38.0%
  3. TSX6.0%
  4. JSX4.3%

Difficulty

  1. Obvious30.4%
  2. Subtle61.9%
  3. Borderline7.7%

Example row

This training row has two findings. Each finding has a line range, an explanation, and the auditor’s verdict. Hover over a finding to highlight its lines.

cart.jsjs · quality 3 of 5 · subtle
  1. const { CartLineItem, ProductVariant } = require('../models');
  2. const cartReaders = {
  3. active: async (cartId) => await CartLineItem.findAll({
  4. where: { cartId, active: true },
  5. attributes: ['id', 'variantId', 'quantity', 'unitPrice'],
  6. include: [{ model: ProductVariant, as: 'variant', required: true, attributes: ['id', 'sku', 'color', 'size'] }],
  7. order: [['id', 'ASC']]
  8. })
  9. };
  10. const selectCartReader = (mode) => cartReaders[mode];
  11. async function fetchActiveCartItems(cartId = Number(process.env.CART_ID)) {
  12. return await selectCartReader('active')(cartId);
  13. }
  14. module.exports = { fetchActiveCartItems };

TaskQuery the relational database to fetch active cart line items and join product variant attributes

  1. Over-abstractionLines 2-12

    The reader registry and mode selector implement only one strategy, and the sole call always selects ‘active’, so put the query directly in fetchActiveCartItems rather than routing through a single-case dispatch layer.

  2. Unsafe inputLine 11

    The environment-derived cart ID reaches the database query after numeric conversion alone, which does not validate an identifier and can produce NaN or turn empty input into zero.

How it was made

  1. Three rounds

    Language models wrote the snippets against a fixed set of tasks and flaw definitions, in three rounds of 1,000 tasks.

  2. Blind review

    Model reviewers read the code without seeing the planned labels, recorded findings, and sent failed examples back for repair and another review.

  3. Small edits

    In rounds 2 and 3, each flawed variant is a small edit of the clean variant of its task.

  4. Audit

    A separate model audited all 27,450 findings. It accepted 26,984 (98.3%) and challenged 466. Each finding records the verdict.

Limitations

  • Models set and audited the gold labels. No person reviewed them.
  • Clean snippets are a fixed 25% of the data.
  • Any comment other than a tool directive or commented-out code sets noise_comments.
  • Validation checked syntax and annotation consistency. No snippet was executed or type-checked.
  • There was no check for near-duplicates across tasks or for overlap with public benchmarks.

Load the dataset

The splits are Parquet files on Hugging Face, released under CC BY 4.0.

Python · datasets
from datasets import load_dataset
import json

snippets = load_dataset("yannelli/js-ts-code-review-decisions", "default")
example = snippets["train"][0]
print(example["code"])
print(example["findings"])

decisions = load_dataset("yannelli/js-ts-code-review-decisions", "decisions", split="test")
row = decisions[0]
state, questions, gold = (json.loads(row[k]) for k in ("state", "questions", "gold"))

Fields

codestring
The snippet, used as model input.
descriptionstring
One-sentence task description all 4 variants implement.
langstring
js, jsx, ts, or tsx.
unclear_naming … unsafe_input10 × bool
One Boolean per flaw. True when the snippet has a finding with that flag.
primary_flawstring
The most important flag, or none for the clean variant.
qualityint64
Mergeability from 1 (should not merge) to 5 (exemplary).
difficultystring
obvious, subtle, or borderline.
findingslist<struct>
Line-level annotations. Each has a flag, start and end lines, an excerpt, an explanation, and an audit verdict.
task_slugstring
Shared by the 4 variants of a task.
state · questions · goldJSON
In the decisions config, the request body for a System One API and the gold answers.
Cite
@misc{yannelli2026jstscodereviewdecisions,
  title        = {JS/TS Code Review Decisions},
  author       = {Ryan Yannelli},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/datasets/yannelli/js-ts-code-review-decisions}}
}

Models

Read how it was builtCode Review Hooks

More datasets

All datasets