AI Systems

Semantic Search, a Remembering Agent, and the Machinery Around Them

Section 4 of 5

There are two AI systems in this product and they solve different problems. Venue search has to understand what a parent means when they type something like “somewhere indoors for a rainy afternoon with a toddler” and then return places they can actually drive to. Lil Joey has to hold a conversation like someone who already knows the family, across sessions that may be weeks apart, without being re-told everything each time.

Neither was hard because of the model. Both were hard because of everything around the model — ordering two ranking signals so one does not destroy the other, keeping memory persistent without paying for it in latency, keeping the model swappable, being able to see what actually happened in production, and metering access to something with a real marginal cost. This section covers each of those in turn.

2

ranking stages in venue search

12

semantic candidates before geo re-ranking

2

custom long-term memory topics

1

line of config to change model provider

Semantic Venue Search

Two ranking signals, applied in a deliberate order. Meaning decides which venues are candidates; physical distance decides which of those candidates a parent will actually see.

Query text + user coordinates + filters“somewhere indoors for a rainy afternoon with a toddler”Embedding Servicededicated Cloud Run endpointUser Latitude / Longitudecarried through untouchedFirestore Vector IndexfindNearest — Euclidean, top 12,distance threshold appliedStage 2 — Geographic Re-RankingHaversine distance in kilometres computed per candidateSorted by real distance, cut at the radiusrequested radius or 50 km default · venues with unusable coordinates separated outStage 1 — semantic recalldecides what is relevant at all

A vector index ranks by conceptual similarity, so its best result for a query about rainy-day toddler activities is the venue whose description matches that idea most closely — anywhere in the country. For a product whose entire premise is nearby, that is a wrong answer delivered with high confidence.

The obvious correction is to filter by radius first and rank within it, but that inverts the problem: the candidate set becomes “everything close”, and the semantic signal is reduced to sorting a set that was chosen without reference to what the user asked for. In a sparse area you get the nearest twelve venues regardless of relevance.

Running semantic recall first and geography second keeps both signals doing what they are good at. The vector query decides what is relevant, bounded by a distance threshold so conceptually unrelated venues never enter the candidate set at all. Geography then decides which of those relevant venues are reachable, by computing real great-circle distance from the user to each candidate, sorting ascending, and cutting at the radius.

Candidates whose stored coordinates are missing or unusable are partitioned out rather than allowed to sort as zero distance — which is the bug you get for free if you treat a missing coordinate as a number, and it puts broken records at the top of every result set.

Lil Joey — The Conversational Agent

An agent built with Google's Agent Development Kit, routed to Claude Sonnet 4.5 through a provider-agnostic adapter, deployed as a managed reasoning engine on Vertex AI Agent Engine, and bridged to the product by the Lil Strollr API. Here is a full turn.

ClientLil Strollr APIExpress + FirestoreAgent EngineADK + Claude SonnetMemory Bank+ PostHogsend message (chat id, if continuing)check plan quotaallowedcreate session — first turn onlysession id → stored on the chat documentstreamQuery(user id, session id, message)search memory for this userknown facts about the familycall model withpersona + memory + toolsnewline-delimited JSON stream of partsassembled reply, exchange persistedafter the turn: write session to memory,emit generation event — both off the response pathsolid = blocking · dashed green = response · cyan = fire-and-forget

The agent is built on Google's Agent Development Kit but does not use a Google model. The model is supplied through LiteLLM's adapter, configured to Claude Sonnet 4.5, which means the agent definition — its persona, its tools, its callbacks, its memory wiring — is written against a generic model interface rather than any vendor's SDK.

This was a deliberate bet about the shape of the next two years rather than a preference about today's best model. Frontier models are replaced on a cadence measured in months, and a codebase that has a specific provider's client threaded through its agent logic pays the full cost of that churn every time. Here, evaluating a different provider is a configuration change and a test run, with everything else held constant — which is also what makes the comparison meaningful.

It also decouples the runtime from the model. The agent is deployed on Google's managed Agent Engine, so it inherits managed session handling, scaling and infrastructure from one vendor while calling a model from another. Picking the best runtime and the best model independently is only possible because the seam between them is an adapter.

The Machinery Around the Models