docs: add independent timing holdout protocol

This commit is contained in:
732642856
2026-07-18 06:59:45 +08:00
parent ac0c56fafc
commit 23cde841ff
2 changed files with 56 additions and 0 deletions
@@ -0,0 +1,55 @@
# Day-Level Holdout V3 Annotation Protocol
Purpose: collect independent, source-backed timing labels for
`references/real_case_calibration/day_level_holdout_v3_preregistration.json`.
This packet does not alter chart rules, thresholds, or product wording.
## Independence
- The annotator must not have edited timing rules or existing control dates.
- `adjudicator` identifies the independent person or organization that resolved
ambiguity. It must not be the timing-rule author.
- Use a public source URL for every row. Do not use an absent search result as
evidence of `no_target_event`.
## One annotation row
```json
{
"case_id": "public_subject_stable_id",
"domain": "career",
"label": "target_event",
"start": "YYYY-MM-DD",
"end": "YYYY-MM-DD",
"source_url": "https://public-source.example/event",
"adjudicator": "independent-labeler-id",
"time_uncertainty_days": 0
}
```
`label` is exactly one of:
- `target_event`: source establishes the requested event in the stated interval.
- `no_target_event`: source establishes a genuine non-event interval, such as a
dated diary, schedule, or contemporaneous record that would have recorded the
target event. Mere silence is invalid.
Use day precision only when the source supports it. Otherwise widen `start` and
`end`, and record the uncertainty in `time_uncertainty_days`.
## Frozen acceptance gate
- At least 20 independently labeled positive cases.
- At least 80 independently labeled negative intervals.
- Every row has all eight fields shown above and a public URL.
- Existing four-control and pilot files remain prohibited from tuning.
Before submitting labels, run:
```bash
python3 scripts/day_level_holdout_validator.py \
references/real_case_calibration/day_level_holdout_v3_preregistration.json
```
Only `ready_for_blind_replay` permits a blind replay. It still does not permit
retuning the frozen rules.
@@ -3,6 +3,7 @@
"status": "awaiting_independent_labels",
"frozen_rule_version": "v3_preregistered_not_tuned",
"truth_policy": "independent_source_and_adjudicator_required",
"annotation_protocol": "docs/research/day_level_holdout_v3_annotation_protocol.md",
"prohibited_tuning_data": [
"docs/benchmark/public_real_case_negative_controls_holdout_v2_4controls_20260717.json",
"docs/benchmark/public_real_case_negative_control_pilot_2026_07_11.json"