TalkBeam
Delivers a presentation as live captions and translation, and shows what the audience missed as a summary. Developed it directly and used it at 2026 BHST and a corporate workshop.
Role
1-person development and operation, from planning through the operator and audience screens, server, deployment and on-site caption output at events
Problem
Small events depend on expensive interpreters or unstable in-house tools. During development, the response order of simultaneous translation requests got mixed up, and when saving to the DB failed, records were lost or the saving order got scrambled, which were consistency problems.
Solution
- Speech recognitionCompared several ASR models and adopted Deepgram, which had the best recognition quality
- OrderingIssued a sequence when a sentence is confirmed, and output and saved sentences in sequence order only, regardless of the order in which translations finished. If a translation finally failed, recorded only the original text and moved on to the next sentence
- Loss preventionWrote confirmed records to a local spool first and deleted only those committed to the DB successfully. After a failure or restart, committed the remaining records again and prevented duplicate saves with room, run and segment IDs
Result
Output Chinese and Japanese captions at 2026 BHST, and provided real-time Korean translation for 8 hours a day over 4 days without incident at a joint workshop of Malaysian and Korean companies
Overview
TalkBeam is a service that delivers a speaker's words as live captions and translation, and shows what the audience missed as a summary. Audience members join with a QR code and read the presentation in the language they choose. Operators can send the same captions to a venue screen or a broadcast screen.
Developed and operated it alone, from planning through the operator and audience web screens, server development, deployment, and sending captions on site at events. It has been used at 2026 BHST and at a joint workshop of a Malaysian company and a Korean company, and the caption reading flow and operation problems found on site are now being improved.
Key features
- Captions on phones: Join by link or QR and read the presentation in the language you choose
- Captions at the venue: Show captions on a large screen or in OBS, a broadcasting program
- Presentation summary: For people who join midway or missed something, check the key points so far
- Caption record: View the original presentation and its translation in time order, and search by original text or translation
Implementation and improvements
Small events tend to depend on expensive interpreters or unstable in-house tools. During development there were consistency problems: responses to concurrent translation requests came back in a different order, and when saving failed, records were lost or the saved order got mixed up. Speech recognition results keep changing while the speaker is talking, and translations can arrive in a different order from the one they were requested in. Showing these results on screen as they came caused sentences to be cut short and translations the audience was reading to disappear. The processing flow was designed around separating the caption to show now, the sentence to translate, and the original text to keep in the record.
-
Compared several speech recognition models and chose Deepgram
Compared several ASR models and used Deepgram for speech recognition, as it had the best recognition quality.
-
Grouped short, broken-up speech into one sentence
When speech recognition results were sent out as they were, a short phrase like
Yes.appeared and vanished on its own, or a sentence that had not ended was translated separately.Built a rule on the server that decides how much to group into one caption. A short reply is grouped with the speech after it for up to 1.5 seconds, and an unfinished English phrase waits for the next words. While waiting, an interim caption is still shown, but the sentence to translate and record is confirmed separately.
Translating every result immediately gives too little context, and if the screen also waits until a sentence is complete, the audience cannot see a response. Interim captions are shown quickly and confirmed sentences are used for translation and the record, which divides the two needs.
-
Kept the caption being read while waiting for a new translation
At first, the existing translation was cleared every time the recognized speech changed, so the audience was reading and then saw a blank screen. Changed it to keep the previous translation and replace it when the new translation arrives. Old responses that arrive late are discarded, and completed sentences are shown in presentation order. A sequence is issued when a sentence is confirmed, and captions are sent out and saved only in sequence order, regardless of the order in which translations finish.
Translation requests were also organized. A request for an interim caption that keeps changing is replaced with the latest content, and translation of a confirmed sentence can be handled separately. This is meant to reduce cases where requests for content that is about to change pile up and delay the translation of a sentence that needs to be recorded.
-
Kept the original text even when translation failed
There was a time when a sentence was dropped because a translation response was late. If translation of a completed sentence fails from a temporary error such as a timeout, the request management stage tries once more.
If it still fails, the sentence is marked as failed and processing moves on to the next caption. The original text is saved whether or not translation succeeds, so an error in the translation of one sentence does not affect later captions or the record of the presentation.
Confirmed records are written to a local spool first, and only records that were committed to the DB are deleted. If saving fails or the server restarts, the remaining records are committed again, and duplicate saves are prevented with room, run and segment IDs.
-
Made missed content available as a summary by topic
It is hard for an audience member who joins midway to read all the earlier captions. The existing summary and the newly arrived presentation content are passed to the AI together, so that the same topic is supplemented and a new topic is added.
The summary is processed separately from the captions, so captions keep being delivered while a summary is being made. Audience members who chose the same language share the translated summary, which reduces duplicate requests. The roles are divided: captions follow what is being said now, and the summary is for checking the flow so far.
Tech stack
Next.js · TypeScript
Fastify · PostgreSQL · Deepgram
Results and current status
Use at live events
At 2026 BHST in June 2026, an early version sent a Korean presentation out as Chinese and Japanese captions. It was later expanded into a web service, and at the joint workshop of Malaysian and Korean companies it provided real-time Korean translation for 8 hours a day for 4 days without any outage.
Verification and current improvements
Delayed translation responses on purpose and swapped their order to check the retry behavior and that the caption order is kept. Also tested that the original text is saved even when translation fails in the end.
Based on feedback from the field, a feature that reconnects automatically when the microphone connection drops is now in operation. Currently being verified on a development server are a feature where translation also changes when an operator corrects a misrecognized caption and presets for choosing how captions are displayed to suit an event.