Most marketers I work with do not want a lecture on containers, triggers, or data streams. They want a usable answer to a practical question: How do we count newsletter signups after the form actually submits? What should I paste to the developer? And how do I know this is not made up?
That last part is the real problem.
A general-purpose AI will usually give a fluent GA4 or GTM answer. It may even sound like an expert. The risk is not that it refuses. The risk is that it invents a menu path, mixes Universal Analytics with GA4, or recommends a data-layer push for a file download that Enhanced Measurement already collects. A non-technical marketer has little way to check. They either send a shaky brief to engineering, or they stop using AI and wait for an analyst.
This PoC tested a narrower question:
If we ground answers in official Google Analytics 4 and Google Tag Manager documentation, show the pages used, and write for a marketer who will copy a brief to a developer, do we get answers that are more checkable — and therefore easier to trust — than asking an AI with no retrieval?
Behind that question is an operating model I believe in: human judgment at the centre, AI to assist or take over the manual work. The assistant is a self-help tool for marketer questions and easy tasks. When the model is not confident enough, it should redirect to an expert. That is how you reduce load on a small MarTech team and still speed up the process.
I built a local retrieval-augmented generation (RAG) assistant to test that. RAG here means: search approved docs first, then ask the model to answer only from those pages. This was a bounded test of the business question, not a production rollout.
The code is here: github.com/anthony-tsui/marketing_tracking_poc
A short walkthrough of the Gradio chat and retrieved-pages panel is here: Marketing Tracking RAG Demo.
The business question
The workflow I wanted to improve is the one that happens before a ticket is written.
A marketer has a campaign need. They ask how to track it. Someone — often an analyst or a contractor — translates that into GTM steps, GA4 configuration, and a developer request. That translation is slow, uneven, and easy to get slightly wrong. It also burns scarce MarTech expert time on questions that a well-grounded assistant could handle.
The hypothesis:
- Quality. Answers grounded in official help pages should stay closer to current Google guidance than an ungrounded chat model.
- Confidence. A marketer should be able to see which page a step came from, quote it, and hand a developer a brief they did not have to invent.
- Safer refusal. If the ask is vague (“track events”) or the docs are thin, the assistant should ask clarifying questions or say it is not confident — then send the marketer to an expert instead of filling the gap with a guess.
The aim is not to replace the MarTech expert. It is to let marketers self-serve the easy work, keep human judgment on the hard cases, and free a limited expert bench for the work that actually needs them.
Why “just ask AI” fails this workflow
The failure mode is not lack of language. It is lack of grounding and lack of a check.
Three things go wrong in practice:
The model answers a nearby question. “I want to track purchases” can retrieve ads, Floodlight, or Campaign Manager pages because they also say “purchase.” A fluent answer about the wrong product still looks complete.
The model does not know when to stop. Vague asks get a full setup guide. The marketer then briefs a developer for a click-based custom event when Google already collects file downloads, or they skip the data-layer message that a real purchase needs.
The marketer cannot audit the answer. If there is no quote and no link, confidence becomes a feeling. Feeling is a weak control when the next step is publishing a container.
So the product requirement was not “a MarTech expert chatbot.” It was: sit next to a non-technical marketer, recommend one method in everyday words, show the supporting line from Google’s page, finish with Copy this to your developer when a developer is actually needed, and hand off to a human when the assistant should not pretend to know.
What I built
The system is a local Python app. Official GA4 and GTM pages are copied into Notion as Web Clips, indexed on disk, and answered in a Gradio chat. A second tab runs evaluation.
In plain terms: Notion is the filing cabinet. A local search index is the set of labeled slips. Chat never reads the live web at question time. It reads the index built from those clips.
This is what that looks like in the chat: the marketer question, the grounded answer, and the official pages used beside it.
Marketing Tracking RAG Demo
Official GA4 / GTM pages
↓
HTTP crawl (keep only Analytics / Tag Manager articles)
↓
Notion Web Clips (project knowledge store)
↓
Dedupe → parent-child chunks → embeddings → Chroma
↓
Gradio Chat: retrieve → answer → show the pages used
Gradio Eval: generated questions → answer → LLM judge
Last crawl snapshot: about 3,199 pages visited, 3,178 stored, 6 failed.
| Piece | Choice | Why it matters |
|---|---|---|
| Chat model | DeepSeek via OpenRouter | Writes the marketer-facing answer |
| Embeddings | NVIDIA Nemotron 1B (free) | Turns passages and questions into searchable vectors |
| Judge / test-set model | GLM 5.2 | Scores answers; not used on the live retrieve path |
| Vector store | Chroma on disk | Local, inspectable, no hosted database |
| UI | Gradio Chat + Eval | Answer on the left; retrieved pages on the right |
Three design choices mattered more than the model names.
1. Search the specific sentence, return the whole how-to
Google help pages often start with the same intro: configure the Google tag, get data flowing from the website. If you embed that intro, every question retrieves setup fluff.
The index cuts each article on markdown headings, embeds a short child passage (~500 characters) with its heading, and returns the parent section (~2,800 characters) to the model. Boilerplate intros with no procedure words are dropped. Duplicate http/https twins and identical article bodies are collapsed before embedding.
Everyday picture: the child is the labeled slip you search; the parent is the full section on the back of the slip.
2. Rank like a marketer, not like a keyword cloud
The first retrieval design used an LLM to rewrite the question, expand it, and rerank chunks. It was slower and still surfaced Floodlight, API, and generic setup pages.
Live retrieval now does something simpler:
- Keep the prior event on follow-ups (“What about SST?” after “track purchase” stays a purchase-through-the-tagging-server question).
- Search a few GA4 / GTM phrasings of the same ask.
- Blend vector similarity with lexical overlap on titles, URLs, and headings.
- Boost how-to pages when they match the question; downrank API, legacy Universal Analytics, ads, mobile, and BigQuery unless the question asked for that kind.
- Return one section per URL, top five.
3. Write for confidence, not for completeness theatre
The system prompt is a colleague sitting next to a marketer, not a docs dump.
- Vague asks get 2–3 clarifying questions first.
- One recommended method, in everyday words.
- Numbered GTM and GA4 clicks when the need is clear.
- A copy-paste developer brief (including a
dataLayer.pushexample) only when a developer is needed. - After each important fact, a short quotation from a retrieved page plus a link.
- If the pages are thin or conflicting: say so, and tell the marketer to find a GA4/GTM expert before publishing.
The UI shows the retrieved chunks beside the answer. That is the confidence mechanism: the marketer can read the same paragraph the model used. The redirect-to-expert rule is the other half: AI does the repeatable briefing work; a human stays accountable when the assistant should not guess.
How I evaluated it
Evaluation used three scores, each 0 to 1, from an LLM judge (GLM) with the question, an expected answer, key points, the model answer, and the retrieved snippets:
| Score | Meaning |
|---|---|
| Accuracy | Factual correctness against the expected points and official docs |
| Completeness | Whether the important points needed to answer were covered |
| Relevance | Whether the answer stayed on the question that was asked |
A human reviewing every answer would be the better judge. For a PoC, that is often too expensive, too slow, and not repeatable enough when you are still changing retrieval and prompts. LLM-as-judge was the practical trade-off: fast enough to re-run after each change, cheap enough to use during the experiment, and consistent enough to compare runs. It is a first filter, not a replacement for practitioner review.
The test set was 10 questions generated from indexed chunks. This run was mostly reporting questions (User vs Traffic acquisition, emailing reports, Campaign Manager 360), not the full range of purchase / click / server-side tagging briefs the chat is designed around. I treat the scores as a first signal on the business question, not as a finished exam.
Results
Latest local run on those 10 questions:
| Metric | Mean |
|---|---|
| Accuracy | 0.945 |
| Completeness | 0.885 |
| Relevance | 1.000 |
Relevance was perfect: every answer stayed on the question that was asked. Six of the ten scored 1.00 across the board, including the steps to email a report, which reports cannot be emailed, the seven-collection limit in Analytics 360, and how to open the Campaign Manager 360 report.
Completeness is the weaker score. Two answers were right in spirit but thin on the labels a practitioner would expect: q2 named First user / Session source but omitted medium and campaign (completeness 0.75); q3 explained why the two acquisition reports should not be compared, but did not use “user-scoped” and “session-scoped” (accuracy 0.85, completeness 0.70).
The miss that matters most is q10. Asked what attribution model the Campaign Manager 360 report uses, and whether it can be changed, the answer correctly named non-direct cross-channel last click and said the model cannot be changed. Then, in the plain-language section, it inverted the rule: it told the marketer the report gives credit to a direct visit over an ad. Non-direct last click does the opposite — it ignores direct traffic and credits the last non-direct click. Accuracy and completeness both fell to 0.60. Relevance stayed at 1.00. Fluency hid the slip; the citation and the judge made it reviewable.
That is exactly why the operating model keeps a human in the loop. Easy, well-covered questions can be self-serve. When the assistant is not confident, or when the topic is attribution logic a marketer might take at face value, it should redirect to an expert.
For a first PoC pass, the results still look appealing: high accuracy, perfect relevance, and the gaps are the kind a gold set and indexed practitioner notes would be designed to catch.
Conclusion
This PoC was meant to validate a business question, not to ship a product: can RAG over official GA4/GTM docs give non-technical marketers higher-quality, more trustworthy briefs than asking an AI directly?
The first eval says the direction is right. Most answers cite the pages they used, stay on topic, and match official guidance. The CM360 attribution miss is the reminder that a fluent wrong explanation is still the failure mode this design is trying to catch.
That matches how I think human–AI collaboration should work. The marketer uses the assistant as self-help for questions and easy tasks. Human judgment stays at the centre: the expert is not removed from the operating model, they are reserved for the cases the model should not own. When the assistant is not confident enough, it redirects to an expert instead of guessing. The result I am aiming for is less queue time on a limited MarTech bench, and a faster path from “what should we track?” to a brief a developer can use.
If I develop this further, I think performance will get better from two additions, not from a bigger model:
- A richer gold set for re-evaluation — hand-written questions on the actual briefing work: purchase, visible click, Enhanced Measurement, form-success, server-side tagging, and attribution cases like the CM360 miss above. The current set was generated from the index and leaned toward reporting. A gold set that matches the marketer workflow would be a fairer test of the hypothesis.
- Document my MarTech expertise and index it into Chroma — so retrieval can pass it to the model as extra context, not only official help pages. Google’s docs say what a tag or event is. They do not always say which method a marketer should pick, or how to explain “non-direct last click” without flipping the meaning. I would write that judgment down — when GTM can listen itself versus when a developer
dataLayermessage is required, how to brief engineering, how to stop a vague “track events” ask from becoming the wrong implementation — then chunk and embed those notes into the same Chroma store. At question time the model would retrieve both the official page and the practitioner note.
Docs supply the facts. Indexed expertise supplies the recommendation. Putting that expertise in Chroma, rather than only in the system prompt, means it can be retrieved, cited, and updated like any other source.
