Quality tiers

This index uses three tiers — Validated, In review, and External — plus two status flags — stale and broken. The tiers describe how an item entered the index and how much review it has had; the flags describe its recent technical state.

Validated

An implementation is Validated if all six are true:

  1. A named author with a verified ORCID.
  2. A licence or rights declaration that allows the listed use.
  3. A clearly-documented paradigm mapping (matches our taxonomy).
  4. Structured metadata covering the Outputs schema (aims, formats, design, variables) — not inferred, author-supplied.
  5. Passes the automated launch test on at least one browser/platform.
  6. At least one peer-reviewed citation that uses this specific implementation (not just the paradigm).

Validated items are reviewed manually on first inclusion and on substantive edits.

In review

An NCCR Evolving Language task that has been submitted to the library but has not yet completed peer review. The implementation is reachable by direct link and surfaced under the "In review" tier, but is not part of the default Validated catalogue.

Tasks move from In review to Validated when reviewers confirm the criteria above. Tasks that fail review are removed.

Stale · Broken

  • Stale — no successful launch test in > 90 days.
  • Broken — the most recent launch test failed.

Stale and broken items stay visible. We flag them; we do not hide them. Hiding broken tasks would make the index look healthier than it is and would cost researchers hours when they discover the breakage themselves, in the wild.

How items move between tiers

NCCR submission ──► ○ In review ──┐
                                  │  reviewer confirms criteria
                                  ▼
                              ● Validated
                                  ▲
                                  │  author claims + meets criteria
                                  │
scraped URL ──► automated pipeline ──► ◐ External

Either direction is possible:

  • An In review item is promoted to Validated once a reviewer confirms the criteria are met.
  • A Validated item is demoted if a criterion lapses (e.g., broken for > 180 days, licence becomes unverifiable).

Who decides

The automated pipeline decides what enters External. NCCR submissions land in In review. A small panel of reviewers (currently: the NCCR TTF-DDG team) decides promotion to Validated. The review criteria above are binding; reviewers do not add unwritten rules.

What this index does not claim

We do not claim any of the following, even for Validated items:

  • that the task is good for your research question.
  • that the task is correctly implementing the paradigm's theoretical construct.
  • that the data format matches your analysis pipeline.
  • that citation count is a quality signal — it is a popularity signal, surfaced because researchers routinely ask for it.
  • "Validated" does not mean "endorsed by NCCR Evolving Language".

Those are your calls. We give you the metadata.

Corrections

If you believe an item is misclassified — wrong tier, wrong paradigm, wrong authorship — use the Report issue action on the task detail page, or contribute a correction.