Project / Development Plan
This is a three-phase plan: it
describes the functionality deliverables in each phase, the follow-ups, and the
timeline. Additional phases will be added if the project is successful. Timelines are not included because work
effort depends on personal motivation, which can vary wildly.
|
Phase |
Goal |
Status |
|
1 Prototype |
Get something working. Android+iOS web app. Multi language, Use OCR for identification. No RAG.
No image matching. |
Complete |
|
2 Advanced Prototype |
Add Image matching with
embedding. Analytics. Management app.
Whole room of exhibits. |
Complete |
|
3 Demonstrator |
Streaming audio from server for
iOS. |
Dreaming |
The prototype will support only a
few exhibits and will not have a RAG database. The server is implemented
in PHP that uses the Anthropic API. It will support the following scenario:
· A visitor starts their session by navigating to a web page on their phone. Supported phones: Android/Chrome/Firefox, iOS/Safari/Firefox.
· Visitor specifies their age group, level of interest and knowledge, and language (English, German, Spanish or French)
· A Silent Mode is available to restrict spoken dialogs and keep noise levels low. Chips and text input only for silence. Audio playback will be supported since it will be heard via headphones.
· The web page will advise the visitor to snap a picture of the text description next to an exhibit.
·
The web app converts the image to text (using
the Tesseract library).
· The text is sent to the PHP server to identify the exhibit being discussed.
· The server presents all available info to the LLM and requests a summary in the desired language. This is forwarded to the phone.
· The app will speak appropriate introductory material, limited to a selectable duration, based on the programmed knowledge.
· The introduction will end with suggestions for areas for further discussion.
· There will be a 'press to talk' button that should be pressed while the visitor is speaking.
· The visitor can select suggested follow up topics, or speak into the phone. The speech is converted to text.
· The app will respond to the visitor’s questions and suggest more areas for discussion.
· The query / response will repeat until the visitor selects the next exhibit by snapping its description text.
· Prompt assembly: exhibit context + visitor profile (age group, profession, knowledge level 0–3, time limit) + conversation history → Claude API call
· The system maintains logs of all LLM exchanges for later analysis.
· There will one document per exhibit that will train the LLM at the start of each query. There is one overall description of the theme of the museum. A few more exhibits will be added.
· A more detailed description of the implementation is in the Arch doc.
· A set of reviewers should be identified. They should be “professional” museum staff, like curators and administrators. If no one is found, students would be great too.
· The POC usage by reviewers will be supervised by developers, since the controls may not be intuitive.
· The POC should be reviewed lightly. Comments will add to the list of improvements and deliverables for the next phase.
· More exhibits, text description and image recognition of exhibits.
· Image matching based on best match of photo embeddings will be used to select exhibits.
· Content ingestion tool: takes Vandana's 15 exhibit briefs (structured per the template above) + museum overview, chunks and embeds them into pgvector
· Share transcript of session with email, Whatsapp, Facebook, File Manager, …
· A short internal report: does Claude API handle this well, what did it cost per session, where did it get facts wrong or feel flat across different knowledge levels
· Multilingual: retrieval needs to work when a question comes in a different language than the source content
· Analytics: ID failure rate, top exhibits & their engagements, dwell and visit times.
·
A defined test matrix, not just ad hoc
testing — e.g., run every exhibit at all 4 knowledge levels × at least 2 age
groups, log all of it.
· Cost projection, not just cost observation — take the per-session token cost from testing and extrapolate to "what would this cost at 100 visitors/day," since that's the number that actually matters for the investor conversation later.
·
Cross-device/cross-browser test pass (a fixed
checklist: iOS Safari, Android Chrome, at minimum),
Firefox. No Amazon Fire.
· Tuning and manual testing across knowledge levels (0–3) and a few different expertise/age combinations
· This version should be tested more extensively. Students can be asked to test and comment on the app. Though the controls are not complicated, each tester will be trained.
· Threshold tuning for matching of embeddings.
· Princeton museum access is a dependency outside the dev team's control — University/museum access agreements often take longer than 4 weeks to arrange. Vandana and her students review every AI response logged during testing against her source material — this is the actual mechanism behind "no hallucinations," and it needs to be a defined workflow: she (or a reviewer) reads flagged/sampled transcripts and marks factual errors
· Besides audio responses, relevant video content can be displayed on the phone.
· Streaming audio from the server. This improves iOS experience (no need to press Press to Hear), adds more languages, much better TTS quality, etc.
· Groundedness/guardrail pass gets real teeth here — this is where "no hallucinations" is supposed to hold up, so this needs actual measurement, not just vibes (see below)
· "No failures, no hallucinations" needs to become a measurable threshold, e.g., "fewer than X factual errors per 100 sampled exchanges— an absolute zero-hallucination bar isn't achievable with current LLM technology and shouldn't be the pass/fail gate as written, or the phase will never "pass."
· "High review scores from visitors"— even a 3-question post-session survey (e.g., 1–5 scale: helpful / trustworthy / would use again) gives a real number instead of an impression.
· Visual identification of exhibits: this is genuinely the hardest technical component. Look into falling back on QR Codes, OCR reading of labels if a identification fails more than twice.
· Use SQL db for Analytics logs
· Evaluate MongoDB for embeddings
· Multilingual response generation/TTS integration if voice is entering the picture here for the first time (worth explicitly deciding)
· Native Android app: camera capture pipeline, Text fallback, UI parity with the web client
· BLE/Bluetooth triangulation —can be tried. This is optional, not a deliverable the phase depends on.
·
Version control and environments — a
shared repo, a dev/staging split even at this small scale, so testing doesn't happen
against whatever's currently on someone's laptop.
· A defined session log schema — "maximal logging" is the right instinct but needs a consistent structure (session ID, exhibit ID, visitor profile, every retrieved chunk, every prompt sent, every response, token counts, timestamps) from day one, since Demonstrator's review workflow and Prototype's test-matrix both depend on being able to query this cleanly later.
· A basic consent/privacy notice for visitor testing — not sure of the requirements. EU/UK standards are migher than in US.
· A running cost tracker, not just a one-time Prototype estimate — token costs will shift as content volume and conversation length grow through Demonstrator and Advanced Demonstrator.
Risks
· Content production is the real bottleneck, not engineering. "All exhibits in one room" could easily be 40–60+ pieces, each needing a genuine 2-page research brief from one non-technical person. That's a lot of research and writing work to compress into the same 4 weeks as the engineering sprint. Strong recommendation: Vandana should be producing Demonstrator-scope content throughout the Prototype phase, not starting when Demonstrator begins — she doesn't need working software to write research briefs, so there's no reason her content work is sequenced after the engineers' work rather than alongside it from day one.
·
the Princeton access
dependency is a real scheduling risk that has nothing to do with team
capability — worth starting that conversation now, regardless of engineering
progress.