Instrument Quiz Performance to Predict Interview Readiness

Why this matters now

Interview panels judge applied thinking under time pressure—not just recall. If your quizzes mirror those pressures and are instrumented well, they become leading indicators of interview readiness. Programs that align quizzes to interview competencies consistently see faster ramp to offers and less mentor firefighting. In SageStepper pilots, teams saw an average +18-point readiness score lift after a two-week Interview Sprint, 9-day practice streaks, and 25% mentor hours saved by triaging with action boards.

Build a competency-to-item map that predicts interviews

Start with the outcomes interviewers actually check: problem scoping, data reasoning, communication, tradeoffs, ethical judgment, and tool fluency. Keep it small and observable.

• 6–10 competencies total

• Rubric levels: Novice (0), Functional (1), Proficient (2), Exceptional (3)

• Evidence: what a correct answer and a plausible-but-wrong answer look like under time constraints

Tag each quiz item to a primary competency. Treat distractors as signals of specific misconceptions interviewers penalize (e.g., over-indexing on algorithms instead of requirements).

# assessment_map.yaml
competencies:
  - id: COMM
    name: Communication under pressure
    rubric:
      0: Rambling; no structure
      1: Basic structure (STAR), missing tradeoffs
      2: Clear structure + tradeoffs
      3: Crisp structure + prioritized risks
  - id: DATA
    name: Data reasoning
  - id: TOOL
    name: Tool fluency (SQL/Python)
items:
  - id: Q42
    stem: "You have 10 minutes to explain how you'd debug a failing ETL job..."
    primaryCompetency: DATA
    difficulty: 0.65      # p-value target
    discrimination: 0.35  # updated after calibration
    misconceptions:
      - COMM: "lists steps without summarizing risk" 
      - TOOL: "chooses wrong instrumentation tool"
  - id: Q43
    stem: "Stakeholder asks for a dashboard by EOD; what do you ask first?"
    primaryCompetency: COMM
    difficulty: 0.55
    discrimination: 0.40

Keep it rigorous, not heavy. Start with blueprint estimates for difficulty and revisit every two weeks based on data.

Capture the right signals (SageStepper makes it easy)

A correct/incorrect flag alone won’t predict interviews. Add three low-lift fields per item.

• Correctness: 1 or 0 (partial credit allowed: 0.5)

• Confidence: self-report 1–5 or a quick sure/unsure toggle

• Time-on-item: seconds spent; cap outliers at 95th percentile

• Hint usage: 0/1 or count; penalize lightly to discourage over-reliance

In SageStepper, use item templates to turn these fields on by default. Results flow to the progress radar and can auto-create follow-ups on action boards.

// assessment-event.ts
export interface AssessmentEvent {
  userId: string
  attemptId: string
  itemId: string
  competency: string
  correct: 0 | 0.5 | 1
  confidence: 1 | 2 | 3 | 4 | 5
  timeSeconds: number
  usedHint: boolean
  attemptTs: string // ISO timestamp
}

If you’re importing from an LMS, include a simple crosswalk and push to SageStepper.

-- rough item discrimination via point-biserial proxy
SELECT
  e.itemId,
  CORR(CASE WHEN e.correct=1 THEN 1 ELSE 0 END, n.nova_score) AS r_item_nova
FROM events e
JOIN nova_sessions n ON e.attemptId = n.attemptId
GROUP BY e.itemId
HAVING COUNT(*) >= 50;

Turn quiz signals into a readiness score

You don’t need full IRT to start. A transparent weighted model works well and is faculty-friendly.

1. Compute per-competency mastery as a weighted average of item scores adjusted for confidence, time, and hint usage.

2. Combine competencies into a predicted_readiness score (0–100) using program-defined weights.

3. Calibrate weights biweekly against Nova interview outcomes. Target r ≥ 0.60 and MAE ≤ 7 points.

# readiness.py
import numpy as np

def item_score(correct, confidence, time_sec, used_hint, t95):
    base = correct  # 0, 0.5, 1
    conf_adj = 1 + ((confidence - 3) * 0.05)  # +/-10% range
    time_adj = 1 - min(time_sec / t95, 1) * 0.1  # up to -10%
    hint_adj = 0.9 if used_hint else 1.0        # -10% if hint used
    return base * conf_adj * time_adj * hint_adj

def competency_mastery(events, item_meta):
    # events: list of dicts per learner; item_meta: dict with weights
    scores = {}
    for e in events:
        w = item_meta[e['itemId']]['discrimination'] + 0.2  # min weight
        s = item_score(e['correct'], e['confidence'], e['timeSeconds'], e['usedHint'], item_meta[e['itemId']]['t95'])
        c = e['competency']
        scores.setdefault(c, []).append((s, w))
    comp = {c: (np.average([s for s, _ in lst], weights=[w for _, w in lst]) * 100) for c, lst in scores.items()}
    return comp

def predicted_readiness(comp_scores, weights):
    # weights: e.g., {'COMM': 0.3, 'DATA': 0.5, 'TOOL': 0.2}
    total = sum(weights.get(c, 0) * comp_scores.get(c, 0) for c in weights)
    return round(total, 1)

Publish your scale and thresholds so learners can self-regulate:

• Green: ≥ 75 (book a Nova panel)

• Yellow: 60–74 (do a targeted Interview Sprint)

• Red: < 60 (focus on two priority competencies only)

Close the loop with Nova and the progress radar

Use SageStepper to validate predictions quickly.

• Schedule a Nova interview at the end of week 1 and week 2.

• Compare predicted_readiness to Nova’s rubric-aligned score in progress radar.

• Push high-variance cases (|prediction − Nova| > 10) to mentors via action boards.

• Track streak length and offer rate alongside scores to keep focus on outcomes, not just numbers.

Example reporting view to aim for:

• r = 0.64 between quizzes and Nova score after week 2

• MAE = 6.2 points

• 31% of learners moved from Yellow to Green after 5 targeted items + 2 Nova sessions

Example playbooks you can copy

Two quick patterns we’ve seen work across formats:

• Data Analyst bootcamp (8 weeks)

  • Competencies: SQL querying (TOOL), data reasoning (DATA), communicating tradeoffs (COMM)
  • Actions: 15-item scenario quiz 2x/week; Yellow learners get a 5-question micro-quiz + 1 Nova mock.
  • Results: +19 readiness lift in 14 days; r = 0.67; 22% offer rate increase cohort-over-cohort; mentor hours −28%.

• Nursing program prepping OSCE-style interviews

  • Competencies: Patient safety, prioritization, SBAR communication
  • Actions: Video-based vignettes; confidence toggle required before reveal; Nova role-play to validate.
  • Results: r = 0.61; MAE 7.1; average streak 10 days; remediation time −24%.

Sample micro-quiz blueprint (5 items, ~10 minutes):

blueprint:
  competency_weights:
    COMM: 0.3
    DATA: 0.5
    TOOL: 0.2
  items:
    - id: Q101
      type: scenario-mcq
      competency: DATA
      difficulty: medium
      misconception_keys: ["premature-optimization", "ignores-null-patterns"]
    - id: Q102
      type: short-text
      competency: COMM
      rubric: "1-2 sentence frame + tradeoff"
    - id: Q103
      type: code-read
      competency: TOOL
      target: "identify O(n^2) bottleneck"
    - id: Q104
      type: ranking
      competency: COMM
      target: "prioritize stakeholder questions"
    - id: Q105
      type: scenario-mcq
      competency: DATA
      target: "choose minimal viable analysis"

Weekly rituals and templates

Rituals are how you keep rigor without slowing the team.

• Monday 30-min data huddle (ID + faculty)

  • Review r, MAE, and 5 outlier learners; adjust two item weights at most.
  • Confirm two anchor items per competency (don’t change weekly).

• Midweek 15-min item QA

  • Scan items with time-on-item p95 > 2x median; reword stems or split.
  • Check distractors: are they mapping to real misconceptions seen in Nova feedback?

• Friday learner retro (10 min)

  • Celebrate streaks (≥7 days) and green moves. Invite two students to share micro-wins.
  • Post a single action board card per learner: one next quiz + one Nova session.

Template: weekly report you can paste into your LMS or Slack.

## Interview Readiness Weekly Snapshot
- Predicted vs Nova r: 0.63 (goal ≥ 0.60)
- MAE: 6.5 (goal ≤ 7)
- Green/Yellow/Red: 38% / 44% / 18%
- Streaks ≥7 days: 41 learners (+9 w/w)
- Offer-ready (Green + Nova ≥ 75): 22 learners (+6)
Focus this week:
1) COMM Q43 wording tightened; 2) DATA Q42 hint penalty reduced to -5%.
Next steps:
- Yellow: take Micro-Quiz B + Nova role-play.
- Red: meet mentor for 15 min; complete TOOL fundamentals set.

What “good” looks like: thresholds and QA

Use these guardrails to keep predictions credible:

• Calibration targets

  • r ≥ 0.60 between predicted_readiness and Nova scores by week 2
  • MAE ≤ 7 points; stabilize item discrimination SE ≤ 0.10 for high-volume items

• Sample size

  • 50+ attempts per item before you trust discrimination; keep at least two “anchor” items stable

• Fairness check

  • Compare residuals (Nova − predicted) across cohorts; |Δ| ≤ 3 points across groups is a good early flag

• Transparency

  • Publish how confidence and hints affect scoring; encourage metacognition, not gaming

• Safety net

  • Anyone with prediction uncertainty > 8 points auto-gets a human check-in via action boards

To keep the build light, use SageStepper’s item templates and CSV import, then let the progress radar do the heavy lifting. You stay in control of the rubric; the platform handles plumbing and nudges.