Meta-Review Rubric

Use this rubric to evaluate the quality of reviews you received. This assesses how well reviewers performed their role.

Purpose

Meta-reviews serve two purposes:

Feedback to reviewers - Help them improve their reviewing skills
Review quality scoring - Part of the reviewer’s grade

Scoring Scale

Score	Meaning
5	Excellent - Exemplary review
4	Good - Helpful and constructive
3	Satisfactory - Adequate but room for improvement
2	Below Average - Missed key issues or unhelpful
1	Poor - Not useful or problematic

Criteria

1. Specificity (30%)

Does the review provide concrete, actionable feedback?

Score	Description
5	Highly specific; references exact locations; provides examples
4	Good specificity; clear references to content
3	Some specific points but also vague comments
2	Mostly vague; few specific references
1	Entirely generic; could apply to any submission

Examples:

Good: “The explanation of the RAG pipeline on page 3 is unclear. Consider adding a diagram showing the flow from query to response.”
Bad: “The writing needs improvement.”

2. Constructiveness (25%)

Does the review help improve the work?

Score	Description
5	Highly constructive; clear path to improvement
4	Good suggestions for improvement
3	Some constructive feedback mixed with unhelpful comments
2	Few constructive suggestions; mostly criticism
1	Destructive or dismissive; no path forward

Examples:

Good: “The evaluation section would be stronger with a comparison to [baseline]. Consider adding this or explaining why it wasn’t included.”
Bad: “The evaluation is weak.”

3. Thoroughness (25%)

Does the review cover all important aspects?

Score	Description
5	Comprehensive; addresses all major aspects; nothing overlooked
4	Good coverage; addresses most important points
3	Covers main points but misses some important issues
2	Incomplete; misses significant issues
1	Superficial; clearly did not engage with the work

Guiding Questions:

Did the reviewer engage with the technical content?
Are both strengths and weaknesses identified?
Did they check reproducibility/code if applicable?

4. Professionalism (20%)

Is the review respectful and appropriate?

Score	Description
5	Highly professional; respectful; encouraging
4	Professional and appropriate
3	Mostly professional with minor issues
2	Some unprofessional language or tone
1	Disrespectful, rude, or inappropriate

Examples:

Good: “While the current evaluation is limited, the core idea is interesting and could be strengthened by…”
Bad: “This evaluation is terrible and shows the authors didn’t think about this.”

Meta-Review Template

## Meta-Review

**Submission:** [Your team's project name]
**Reviewing:** Review by [Reviewer ID/Name]

### Scores

| Criterion | Score (1-5) | Weight | Weighted |
|-----------|-------------|--------|----------|
| Specificity | | 30% | |
| Constructiveness | | 25% | |
| Thoroughness | | 25% | |
| Professionalism | | 20% | |
| **Total** | | | **/5.0** |

### What Made This Review Helpful
[What did the reviewer do well?]

### What Could Improve This Review
[How could the review have been more useful?]

### Response to Reviewer
[Brief response to the review's main points - did you find their suggestions actionable?]

### Overall Assessment
[Was this review helpful for improving your work?]

Review Quality Indicators

Signs of a Good Review

Summarizes the work accurately (shows they read it)
Balances strengths and weaknesses
Provides actionable suggestions
References specific sections/figures/tables
Appropriate level of detail for the artifact type
Professional and respectful tone

Signs of a Poor Review

Generic comments that could apply to anything
Only criticism, no acknowledgment of strengths
Vague suggestions (“make it better”)
Factual errors about the content
Inappropriate or dismissive tone
Obviously rushed (too short, missing sections)

Aggregating Review Quality Scores

A reviewer’s overall review quality score is computed from:

Meta-review scores (60%) - Average of meta-review scores received
Agreement with consensus (20%) - How well scores aligned with other reviewers
Completion (20%) - Did they complete reviews on time with required detail?

This score contributes to the 10% “Review Quality” portion of the course grade.

back