System design / 02
Backend architecture.
A high-level view of how a request flows through the NeoSpider analytics backend — from the tracking script hitting the API, through storage and processing, to what the customer sees in their dashboard and exported reports. The AI pipeline behind the daily report is broken out in a second diagram below.
Request & data flow
DiagramTracked website
Loads the NeoSpider tracking snippet, configured with the customer's site key and any custom page/event tags, and sends pageview and event calls to the API entry point.
API entry point
Receives the tracking script's pageview and event calls, plus dashboard and export requests. Handles auth, rate limiting, and validation before anything touches storage.
Data storage
Durable store for raw events, sessions, and site configuration, alongside precomputed aggregates written back by the analytics engine.
Analytics engine
Aggregates raw events into trends, top pages, referrers, and scores. Runs on a schedule and on demand, writing results back to storage.
User dashboard
Reads processed metrics through the API entry point to render live charts and summaries for signed-in customers.
Report exporter
Turns analytics engine output into PDF/CSV files and email digests, triggered on a schedule or by a manual request.
AI pipeline
Branches off the analytics engine to draft plain-language insights. Retrieves supporting material before it generates, so suggestions are tied to the day's real numbers rather than generic advice. Broken out below.
Daily report & RAG pipeline.
A closer look at 03A / AI. Every night a cron job builds yesterday's report for each active site. Before the model writes anything, the day's metrics are turned into a question and used to pull supporting material from two pools — a shared optimisation playbook and the site's own recent reports — which is injected into the prompt as reference material.
Nightly generation flow
03A / AITrigger & metrics
A scheduled job runs nightly and walks every active site, pulling the analytics engine's aggregate for yesterday: pageviews, visitors, top pages and referrers, device and country mix, plus the previous seven days for trend context. Writes are idempotent per site and day.
Retrieval query
The numbers are rewritten as a plain-language question — trend direction, busiest page, main referrer, device mix. Retrieval works on meaning, so the query has to read like prose, not like a row.
Embedding model
Turns the question into a vector. The same model embedded the playbook, and embeddings from different models are not comparable, so changing it forces a full re-index.
Vector index
Holds the shared optimisation playbook: performance, SEO and UX guidance written as retrievable answers. Returns the closest matches above a similarity floor; their text is read back from storage.
Report history
The site's recent reports, fetched by recency rather than similarity. History text echoes the site's own domain and paths, so under similarity search it would crowd out the playbook and feed the model its own output.
Prompt assembly
Retrieved text goes in as a reference block ahead of the metrics, with instructions to apply it only where it fits the numbers and not to repeat advice a recent report already gave.
Suggestion model
Drafts one headline improvement plus three categorised suggestions. The base model and any fine-tuned adapter are configuration, not code — the adapter shapes tone, retrieval supplies the facts.
Daily report
Stored with its trend score and the ids of every chunk that grounded it, so a suggestion can be traced back to its source. The report is then recorded as history — but never embedded, which keeps the loop from feeding on itself.