Running a partner's oral exam: the ticket, the clock, the protocol

prep2go prepares candidates for CIPLE A2, the European Portuguese exam. The oral part is the hardest to rehearse: it needs someone on the other side for ten minutes, timed, following the format, and willing to do it again next week. That someone is now a photoreal examiner on a video call.

It is the second product on the Studify platform and the first one that someone else sells. prep2go keeps the shop, the price and the checking. We run the exam.

Two doors between us

The whole integration is two contracts, and each one is a single call.

The first door is admission. After a payment, prep2go's server reports the paid order. We credit the attempts to the candidate, keep their account and answer with a link that opens the exam room. The link is the sign-in: single use, valid for minutes, no second account. Repeating the call for the same order is safe — the candidate gets a new link, not more attempts.

The second door is the protocol. When an exam ends, its protocol is ready to collect: the list of finished exams, then each one by its id. A push signal can shorten the wait, but nothing depends on it. The list survives an outage of any length; a lost signal loses no result.

Everything else stays on prep2go's side: prices, refunds, the grade and what the candidate reads after it. We never see their checkout, and they never handle our tokens.

A ticket, drawn once

A candidate who comes back for a second attempt has to meet a new exam. The same exam twice tests memory, not Portuguese. So every attempt draws a ticket from a pool, as the real exam does: a few interview questions, a picture and a topic for the discussion.

The order of the draw matters. The picture comes first, and the topic is drawn only from those the picture lists. The scene of the second part leads into the discussion of the third, so the candidate does not start over twice in ten minutes. Then six interview questions from three areas of everyday life: in each, a plain fact and then a follow-up on it.

The ticket is drawn when the exam starts and stored with its live state. If the connection drops in the fifth minute, the candidate comes back to the same picture on the board and the same next question: a new ticket halfway through would be a different exam. The pool is checked when the product is built. A topic no picture leads to, a picture without its image, an area without a plain question — each one fails the build, not an exam. The draw is uniformly random for now; not repeating a ticket to the same candidate will come from the history of their attempts.

One question at a time

The examiner is a language model in a real-time voice pipeline, and it never sees the whole ticket. The first version did: it got the part's questions as a list, in order. In the first live trials it asked the same question three times and opened the role play again and again — while its own memory summary listed what it had already asked. A model does not reliably compare a list with what it has said.

So the code hands out the ticket, not the model. Each part spreads its items over its useful time. Six interview questions over 130 seconds make a slot every 22 seconds; four discussion questions over 190 seconds, one every 47. When a slot comes, the examiner's context for that turn carries a single line: ask this question now. Between slots it says: follow up on what the candidate just said. The last 20 seconds of a part are for rounding off, and no new item is handed out in them.

The clock, not the conversation, moves the cursor. Speech recognition splits one answer of a candidate into one and a half to two messages, and the examiner says short things of its own — "Certo." A cursor moved by replies would race ahead or stall; a cursor moved by the part's clock is arithmetic. "The questions ran out before the part did" cannot happen by construction. An item counts as given only once it has been spoken: if the candidate cuts in and the generation is cancelled, the line comes back on the next turn.

One mode, one phase

The second part of CIPLE has two moments. The candidate describes the picture, then the same scene becomes a short role play: "Imagine que está neste restaurante. Eu sou o empregado." At first both lived in one phase, with the role play as a one-off item halfway through. In the run of 27 September the line "open the role play" arrived and lost to the phase's goal, which said "ask about the details of the picture" on every turn. The protocol then promised a role play that never happened.

A standing goal cannot carry two modes that exclude each other, and no one-off line outweighs it. Since 1 October the second part is two phases under one heading: the picture, then the dialogue in the same scene. The candidate sees no border; the examiner has one job at a time. The role play counts as performed only if the examiner's spoken line opens it — "Imagine que…", "Eu sou…" — and otherwise it is handed out again, up to twice.

An examiner, not a teacher

The same engine teaches English lessons. Everything that makes a lesson — corrections, drills, materials on the board, a report — is switched off in the product's data, not in code. The examiner speaks briefly, at A2, in European Portuguese. She does not correct, does not supply a word, does not praise — even "muito bem" sounds like a grade. If the candidate did not understand, she says it once more, simpler, and then moves on.

These rules are measured. A frozen set of provocations — a candidate who asks "como se diz…?", switches language, steps out of the role play, asks how they did — scores the examiner's behaviour, so a change to her texts shows what it improved and what it broke.

A protocol of facts

What goes back to prep2go has no model in it. The protocol carries the transcript by phase with the second of every turn, the whole ticket and how far each part got, and plain fluency facts for the assessed phases: words, speaking time, words per minute, turns the recogniser could not trust. The first 30 seconds and the goodbye are marked as not assessed and carry no numbers. It also names the release and the edition of the texts that ran the exam, so a re-read months later knows what the candidate faced.

There are no scores, no level and no "the examiner rephrased here". Such an observation is something prep2go could not check: they read the same transcript we do. The grade is theirs, by their grid.

What it took

The exam reused the engine and leaned on its seams: a second subject, a product layer of prompts, a ticket from a pool, entry paid elsewhere, a protocol the product collects itself. We logged every change to the core along the way. There were 71, and 37 of them were leaks: places where the core had quietly assumed the first product.

Then the product moved out. Its content and configuration live in a repository of their own, where the partner's contractor works without the core and without our provider keys, checks an edit with the same contract the platform uses and sees it on a preview within minutes. Production changes only when we pin a new commit of that repository.

The next test of the template was a second language. That is the next note.