← Back to instruments

Building an instrument

Havast ships no assessment instruments. You build the one your review needs — a risk-of-bias tool, a quality checklist, a data-completeness form — including the algorithm that turns signalling answers into a domain judgement. This page is the reference for how that works and what the tool will and will not accept.

1. The parts of an instrument

Every question, domain, and judgement has an id: a short lowercase name such as q1_1 or d1_auto. Ids are what algorithms refer to and what the exported workbook labels its rows with, so they are worth choosing deliberately. An id starts with a letter and uses letters, digits, and underscores.

2. Building one

1. Answer sets first. Define the scales before the questions, because each question is attached to one. Most instruments need two: a signalling scale and a judgement scale. Give the judgement options colours — they carry through to the figures and the workbook.
2. Add a domain, then its questions in the order the assessor should read them.
3. Add the domain judgement as a calculated item at the end of the domain, and write its algorithm.
4. Add the overall judgement, if the instrument has one, referring to the domain judgements.
5. Write test cases and check the Preview tab.
6. Publish. Only a published instrument can be added to a project and assessed with.

Guidance text can be attached to any question. It is shown beside the question while the assessor works, and it is the right place for the source instrument's own wording. It is displayed as plain text with your line breaks preserved.

3. Conditional questions and conditional domains

A question can be shown only when a condition over earlier answers holds. So can a whole domain — which is what an instrument with variant domains needs. Where a tool carries two versions of the same domain, say a baseline-measured one and a time-varying one, they are alternatives: the one that applies depends on an earlier answer, and only one of them should ever be on screen.

Two rules govern what happens when a condition changes:

• A hidden question's answer does not count. If an assessor answers one variant and then changes the trigger so the other applies, the first variant's answers are still stored — but they are masked. No algorithm can see them, and they do not satisfy any condition. If the assessor changes their mind back, the answers are still there.
• A hidden domain's algorithm does not run. Its judgement is absent rather than defaulted, so an inapplicable domain does not appear in the results table as though it had been judged and found clear.

A condition may only refer to items defined earlier in the form, and a domain's condition may only refer to items in domains above it. An unanswered question never satisfies a condition, including a negative one: if q1_1 is blank, then q1_1 != Y is false, not true.

4. Writing an algorithm

An algorithm is an ordered list of rules with a default:

if (q1_1 in [Y, PY] and q1_2 == N) { return LOW; }
if (count(q1_1, q1_2, q1_3) >= 2 for [NI])  { return SER; }
return CRI;

The first matching rule wins. Rules are evaluated top to bottom and evaluation stops at the first one whose condition holds. The default at the end is required, so no combination of answers can produce an undefined judgement.

Two ways to write the same thing

The builder offers a rule table and a text expression. The expression is what is stored; the table is a view of it, offered whenever the algorithm is simple enough to draw as rows. Editing the table rewrites the expression, and editing the expression rewrites the table or removes it. You never have to keep the two in step yourself.

The table cannot draw everything. Nesting, not, count(...), and comparisons between two questions have no row form, so an algorithm using them is text-only. That is not a limitation of the language — it is the table being honest about what a flat grid can show.

What an algorithm may refer to

• Only items defined earlier in the form. A judgement cannot refer to itself, to a question below it, or to a domain that comes after it.
• Only answer codes that actually belong to the question being compared. Writing q1_1 == HIGH when q1_1 uses the signalling scale is refused, with the valid codes listed.
• Only results that belong to the judgement's own answer set. An algorithm that can return a code the judgement cannot hold is refused.

5. Expression reference

An overall judgement is written the same way, over the domain judgements:

if (d1_auto == CRI or d2_auto == CRI)                     { return CRI; }
if (count(d1_auto, d2_auto, d3_auto) >= 2 for [SER])      { return CRI; }
if (d1_auto == SER or d2_auto == SER or d3_auto == SER)   { return SER; }
if (d1_auto == MOD or d2_auto == MOD or d3_auto == MOD)   { return MOD; }
return LOW;

Where the judgement comes from, and how to disagree with it

The browser recomputes judgements as the assessor answers, so the form updates live. The server recomputes them on every save, and the server's answer is the one that is stored. An assessor who disagrees with the algorithm can record a different rating; that override is kept beside the algorithm's answer rather than replacing it, and both appear in the exported workbook.

6. Test cases

A test case is a row saying: given these answers, expect these judgements. Test cases run on every save, and a failing test blocks publishing.

They are worth the few minutes they take. An algorithm with four domains and a first-match-wins order is easy to get subtly wrong in a way no amount of reading catches — a rule that can never fire because an earlier one always matches first, or a combination that falls through to the default when it should not. Write one test per intended judgement, plus one for each boundary you had to think about.

The builder also refuses to publish an algorithm containing a rule whose condition exactly repeats an earlier rule's, since it can never fire. That check is deliberately shallow: it catches the mistake people make when editing a rule table. It is not a solver and does not claim to find every dead branch. Test cases are what do that.

7. Versions, publishing, and sharing

• An instrument is edited as a draft. Publishing freezes it. There is at most one draft at a time.
• To change a published instrument, create a new version. The old version stays exactly as it was.
• An assessment pins the version it was created under. Editing an instrument never changes a rating already made: that rating stays readable and reproducible under the definition it was actually made with.
• You own the instruments you create. Adding one to a project makes it readable — not editable — by the members of that project.

8. Assessing, reconciling, and reporting

Assessing. From a project, open a PDF and choose an instrument. The form sits beside the paper. Answers save as you work, every change is logged, and a multi-unit instrument lets you add as many assessments to one paper as it needs.

Reconciling. Two assessors rate the same result independently. Neither can see the other's answers — not on screen, not through an export — until both have marked their own assessment complete. At that point the two open side by side, agreements are carried into a consensus record automatically, and only the differences ask anything of you.

Reporting. The results screen gives the study-by-domain matrix, a traffic-light plot, and a distribution bar, plus a workbook with the matrix, every answer in long format, and a sheet recording which instrument version and which record the ratings came from. Where no rating stands, the cell is empty and a note says why; an unreconciled assessor's judgement is never quietly substituted for a consensus.

9. What the tool refuses, and why

← Back to instruments