Multimodal Meeting Copilot
Use this skill when building an AI copilot for presentations, meetings, or screen shares where the agent must understand both on-screen content and spoken context.
This skill is especially useful when the product needs to:
- capture live screen frames
- transcribe display audio and microphone audio
- generate live insights about what is on screen
- support live copilot chat with Assistant UI
- run Railtracks-based fact-check and summary workflows
- create a trustworthy final meeting summary
Core pattern
Split the system into two layers:
- A UI/app layer that captures screen and audio, renders the copilot, and forwards structured context.
- A Railtracks service that handles deeper agent workflows such as fact-checking, review loops, retries, and final summary generation.
Keep the UI fast and interactive. Push heavier orchestration into Railtracks.
Railtracks guidance
Use Railtracks for the workflows that need structure, validation, and memory:
- fact-check flows
- final summary flows
- reviewer loops
- retries for weak or malformed outputs
- cross-run lessons or improvement memory
Do not use Railtracks as a thin wrapper around a single model call if the real value is in orchestration.
Prefer a flow shape like:
- extract claims
- gather evidence
- judge result
- review result
- retry or distill lessons
Assistant UI guidance
Use Assistant UI for the live copilot chat surface.
Keep the chat UX separate from the heavier agent workflows:
- Assistant UI handles the live conversation and streaming experience
- Railtracks handles deeper background workflows
If possible, support both:
- a fast direct chat path
- a Railtracks-backed chat path using the same UI
Capture and context guidance
For meeting copilots, treat screen context and transcript context as equal inputs.
- Capture frames on an interval, but dedupe aggressively.
- Capture display audio when the browser exposes it.
- Request microphone access as a fallback and mix both streams when available.
- Store enough recent transcript to support fact-checking and summaries.
When building summaries, combine:
- what was shown
- what was said
- what was verified or contradicted
- what mattered most
Product guidance
The strongest positioning for this class of product is not “AI note taker.”
Position it as:
- a multimodal meeting copilot
- an autonomous presentation copilot
- a trust layer for live presentations
References
For a concrete implementation based on CogniView, read references/cogniview-architecture.md.
Demo: CogniView demo video
