The Pulse Ecosystem

Definition

This document explains what the systems around pulse-web do with the data Pulse produces: how event-receiver ingests events, how tools-portal-front authors definitions and settings and hosts the live debugger and session replay, how autopilot uses Pulse identity to record sessions, and how replay-receiver ingests replay metadata. You do not need to work in these repositories to contribute to pulse-web, but many of them depend on Pulse's output in ways that fail silently when it changes.

The systems

pulse-web is the producer in a pipeline that spans five repositories plus a few services whose code is not part of this workspace.

System	Repository	Relationship to Pulse
event-receiver	`event-receiver` (NestJS)	Ingests every event and attribute Pulse sends
tools-portal-front	`tools-portal-front` (Angular 18, picsart.tools)	Authors event definitions, settings, segments and experiments; hosts the live debugger and session replay
autopilot	`autopilot`	Session recorder that Pulse loads; reads Pulse identity
replay-receiver	`replay-receiver` (NestJS)	Ingests autopilot's per-batch replay metadata
optifyr, event-definition, abs-config, websocket-gateway, session-replay API	not in this workspace	Settings delivery, definition storage, settings authoring, debug fan-out, replay read API

End-to-end data flow

                          tools-portal-front (picsart.tools)
        authors ──────────────┬───────────────────────┬──────────── views
        event definitions     │ settings / segments / │  live debugger      session replay
              │               │ experiments           │  (SockJS/STOMP)          ▲
              ▼               ▼  (abs-config)         │        ▲                 │
      event-definition     optifyr                    │        │           session-replay API
              │               ▲                       │  websocket-gateway       ▲
              │ definitions   │ GET /web/v2/{app}/{tag}        ▲                 │
              ▼               │ (settings)            │  Redis debug_events:<app>  Kafka session_replay
       event-receiver ◄───────┼── POST t.picsart.com/events/v2/web/{app}/fetch   │
        validate / enrich     │   (events + attributes)                     replay-receiver
        → Kafka topics        │                                                  ▲ POST /web
                              │                                                  │
  page ── <script src=optifyr.com/pulse/{app}/{module|nomodule}/pulse.js>        │
             │  Cloudflare Worker (apps/server): segments → env → bundle         │
             ▼                                                                   │
        window.pulse (tracker + settings) ── getState() / getState$() ──► autopilot
                                                                   gzip batches → CDN

The rest of this page follows that diagram system by system.

event-receiver: where events go

The tracker posts batches to:

POST https://t.picsart.com/events/v2/web/{app}/{fetch|xhr|beacon}

The route lives in event-receiver/apps/instance/src/controllers/web.v2.controller.ts. The receiver ignores the last path segment, but CORS is configured only on that three-segment form, so the transport suffix must stay. How the tracker builds these batches is covered in Event Pipeline.

Processing order

Envelope validation. The body must be an array of chunks { config?, header, events, attributes?, attributeHints? }. An envelope that fails validation gets 400.
Device id resolution. The receiver takes the first of:
1. header.device_id, unless config.deviceId === 'none';
2. the paa-did cookie;
3. sha256(ip, ua, 3-day bucket).
It sets the paa-did cookie for 5 years with SameSite=None on the request host, t.picsart.com and .picsart.com. With serverCookies: 'same-origin' it sets it on the request host only; with false it sets none. If the body id is missing or differs from the cookie, it returns a paa-did response header carrying the cookie id, which the SDK adopts (see Identity and State).
Session override. If config.sessionId is set and is not 'persistent', the server replaces session_id with ${3h bucket}_${deviceId}.
Enrichment.
- Country: header.country_code, then cf-ipcountry, then 'other'.
- User-agent fields: device_type, device_model, os, browser.
- received_time, bot score, JA3/JA4.
- attributeHints become server-derived attributes (browser, os, first_visit_date decoded from the device id, and so on).
Header merge. The whole chunk header is merged onto every event, and header.data is merged into event.data. This is why the SDK sends shared context once per chunk instead of per event.
Normalisation.
- Renames: _app_load to app_load, _experiment_participate to experiment_participate.
- tier_id is copied (not moved) to subscription_package.
- user_id is parsed with parseInt.
- A missing event_id gets a generated one.
Validation.
- The app must be known.
- Event size must be at most 2 MB.
- timestamp must fall between 5 days before the end of the current UTC day and 10 minutes in the future.
- Required fields: v, version, session_id, country_code, device_id, platform, market, event_type, timestamp.
- A per-app:event_type Ajv schema generated from the tools-portal event definitions must pass.
Mapping and output. Events are mapped to the common schema and produced to Kafka, keyed by device id.

Validation strips undeclared fields

The per-event Ajv validators run with removeAdditional: 'all'. Whenever an event defines any properties, undeclared data.* properties are silently stripped. An event with no definition becomes UNDEFINED_EVENT.

caution

If you add a field to a built-in event in the SDK, it must also be declared on that event's definition in tools-portal-front. Otherwise the receiver removes it and nothing downstream ever sees it, with no error on either side. See Cross-Repo Contracts.

Kafka outputs

Output	Topic
Valid events	`tracking.eventReceiver.events`
Invalid and unmapped events	`…eventsInvalidCommon`
Attributes	`…attributes`
Invalid attributes	`…attributesInvalid`
Debug events	`…eventsDebug`
Short-term and Clevertap copies	`…eventsShortTerm`, `…eventsClevertap`

These are the code defaults; per-environment topic names may differ.

Response semantics the SDK relies on

Status	When	SDK behaviour
200	Events accepted, including invalid events (sent to the invalid topic), a missing or bot user agent (dropped), or a body over 15 MB (dropped)	Batch is done
400	Envelope fails validation	Treated as success by `HttpService`, so the batch is not retried
500	Kafka produce failed (unless the request sets `acknowledge: false`)	Retries
503	More than 1000 requests in flight per worker	Retries

note

A 200 response does not mean the event was valid. To check validity, use the portal's live debugger or look at the invalid topic.

Where definitions come from

Event and attribute definitions come from the event-definition service:

A separate metadata-loader process loads them at startup, reloads them on the Redis pub/sub channel event_definition.event_changed, and pushes them to the workers over IPC.
Attribute schemas are also polled every 10 minutes.
Definitions are held in memory per worker.
Draft and archived events are dropped unless the session is in debug.

tools-portal-front: authoring and debugging

Event and attribute definitions

Definitions are managed in tools-portal-front/src/app/main/events and tools-portal-front/src/app/main/attributes, which call the event-definition API.

Event fields: name, status DRAFT, LIVE or ARCHIVED, and properties.
Property types: STRING, OBJECT, NUMERIC, FLOAT, BOOLEAN, ARRAY_*. NUMERIC becomes an integer in the receiver's schema.
Property options: required, toLowercase, and an operation such as MOVE_TO, RENAME_TO or SET_VALUE.

These definitions are what event-receiver compiles into its Ajv schemas.

Settings, segments and experiments

Settings, segments and experiments are authored against abs-config. Settings have a type (string, boolean, number or json), tags (inclusive or exclusive), per-segment values, per-experiment variant values and an approval workflow. They reach the SDK through optifyr; the SDK side is described in Settings, Experiments and Segments Internals, and the integrator side in User Role based Segmentation and Segmentation based on provided attributes.

Live debugger

The debugger lives at the route …/app/:appSlug/debugger (tools-portal-front/src/app/main/debugger/). It shows the events one browser sends, together with their validation result, in real time.

portal                       site (Pulse)                event-receiver        websocket-gateway
──────                       ────────────                ──────────────        ─────────────────
"Go to {app}" ──open──►  ?debug-leuid=true
  (rel=opener)              enable debug keys,
                            strip param
      ◄── postMessage ───── pulse/live-debug-device-id
  check origin + id,
  fill device filter
                            events with header.debug ──► eventsDebug topic
                                                         + Redis debug_events:<app> ──► STOMP
      ◄──────────────────────────────────────────────────────────────────── /debug/events/{bundle}
  filter by device_id,
  keep last 300

The portal opens the site. "Go to {app}" opens app.webBaseUrl?debug-leuid=true with target=_blank rel=opener, so the site keeps window.opener.
The SDK turns on debug. DebugEventPlugin enables the debugger and debug-leuid keys and strips the parameter from the URL. LiveEventDebugService.shareDeviceId() then posts { type: 'pulse/live-debug-device-id', deviceId } to window.opener with the portal's origin as the target origin.
The portal accepts the id. It requires the message origin to equal the origin of app.webBaseUrl and the device id to match /^[\w.-]{4,128}$/, and it takes only the first message that passes. It then fills the device filter.
Events are marked for debug. The SDK sends every event with header.debug = true. event-receiver routes debug sessions only to eventsDebug and to Redis debug_events:<app>, never to the production topics (the eventsClevertap copy is not gated on debug). Draft and archived events are kept.
The portal shows them. websocket-gateway relays the stream over SockJS/STOMP on /debug/events/{bundle} as { validationResult, commonEvent }. The portal filters by commonEvent.device_id (case-insensitive substring) in the browser and keeps the last 300 events. validationResult.errors are Ajv errors: undefined_event shows as UNKNOWN, anything else as INVALID.

Current limitations

The device id handoff targets the production portal origin only. A portal running on a dev, stage or local origin never receives the message, and the browser drops it without an error. The debug badge link also always points at the production debugger.
event-receiver forces debug for a configured set of IPs and origins (stage and localhost). Traffic from those origins never reaches production topics (Clevertap aside), whatever the SDK sends.

Experiment preview

The portal's experiment "preview" link opens the site with a pulse-debug-exp parameter that the settings SDK picks up. The full flow is in Settings, Experiments and Segments Internals.

Session replay lookup

Routes: session-replay and session-replay/:sessionId. The id is either a raw Pulse session_id (${ts}_${deviceId}) or a numeric sessionIdHash.
Metadata: the portal reads replay metadata from the session-replay API, not from replay-receiver.
Per-session context: events, requests, attributes, experiments and settings shown next to a replay come from abs-device-stat, looked up by device id hash.

autopilot: session recording

Autopilot is the session recorder. Pulse loads it; applications never initialise it directly. The integrator view is in Initialization and Enablement and Disabling Autopilot Session Recording.

Loading gates

The all-in-one bundle's loadAutopilotIfEnabled loads autopilot only when all of these hold:

config.autopilotTracking.enabled === true. The Worker sets this per segment, and the ?autopilot= query parameter can override it.
The page is not a web-to-mobile page (see Web-to-Mobile).
PageLoadedService.whenLoaded() has fired.
The remote setting (tag pulse_loading_config, name autopilot_config) has tracking_enabled === true.
Consent trackingEnabled() is true.

When consent is revoked, the loader calls window.autopilot.stop(), or sets __.stopRequested if autopilot has not finished booting. There is no sampling: every page that passes the gates records.

autopilot.loader.ts then injects {host}/ap/web/v1/sdk/{app}/ap.{modern|compat}.js, choosing modern or compat by feature detection (URL.canParse, resizable ArrayBuffer, noModule), picks the host from location.host (dev, stage or production), and seeds window.autopilot.__ with the queue, config (including mode and initial frame state), timestamp, version, base URL and modernity flag.

What autopilot reads from Pulse

Autopilot uses exactly two Pulse methods:

Method	Used for
`window.pulse.getState()`	A Promise (a plain value in the mini bundle) for `session_id`, `device_id`, `user_id`, `app`, `country_code`, `tab_id`, and `version` forwarded to iframes
`window.pulse.getState$()`	A stream autopilot subscribes to for identity changes

Autopilot restarts recording when session_id changes (in iframe mode, when frame_id changes). A user_id change alone does not restart it. Autopilot sends no events through Pulse. Changing the shape or timing of getState() is therefore a contract change (see Cross-Repo Contracts).

Recording pipeline

trackers → Portal → Web Worker (encode; header = schemaVersion + tabId[|frameId]) → gzip
         → POST upload.picsart.com/v2/files                 (CDN)
         → POST analytics.picsart.com/session-replay/web     (replay-receiver, one record per batch)

Batches are at most 300 KB, sent on a 10 s cadence with a 30 s maximum wait, and a session is capped at 30 minutes of recording. What autopilot masks is described in Masking.

Iframes

A standard-mode page posts { type: 'autopilot/set-frame-id', frameState } to its child iframes, 10 times at 1 s intervals. The mini bundle inside the iframe waits for that message before it loads autopilot in iframe mode. The loader's AutopilotFrameState type is a hand-written copy of autopilot's FrameStatePayload, so the two must be changed together.

replay-receiver: replay metadata ingest

Endpoint	Caller	Behaviour
`POST /web`	autopilot	Receives `{ app, platform, version, country_code, user_id, device_id, session_id, url, start, timestamp }`, validates it with Ajv and produces it to Kafka `session_replay`. Returns 200 for an invalid record (counted and dropped) and 500 when the Kafka produce fails
`POST /assets/link`	autopilot	Maps stylesheet URLs to stored copies on the CDN
`POST /assets/lookup`	tools-portal-front	Looks up those stored copies at replay time

Pulse state matters here because Pulse must provide ids before autopilot uploads. An empty session_id fails the pattern ^[^\s]+$, and device_id uses the stricter ^(?![0-]*$)[^\s]+$, which also rejects all-zero ids. A batch whose record fails validation never appears in the portal, even though its data reached the CDN.

Where to look in the code

Concern	Location
Web ingest route	`event-receiver/apps/instance/src/controllers/web.v2.controller.ts`
Event processing	`event-receiver/apps/instance/src/services/event-processor.service.ts`
Definition-based validation	`event-receiver/libs/definitions/src/services/validator-store.service.ts`, `event-receiver/libs/attributes/src/services/attribute-validator-store.service.ts`
Definition loading	`event-receiver/apps/metadata-loader`
Default Kafka topics	`event-receiver/libs/config/src/default-topics.ts`
Event and attribute definitions UI	`tools-portal-front/src/app/main/events`, `tools-portal-front/src/app/main/attributes`
Live debugger UI and handoff	`tools-portal-front/src/app/main/debugger/`, `tools-portal-front/src/app/main/debugger/device-id-handoff.ts`
Live debugger SDK side	`packages/browser/tracker/src/plugins/live-event-debug.plugin.ts`, `packages/browser/tracker/src/services/live-event-debug.service.ts`
Experiment preview link	`tools-portal-front/src/app/main/experiments/experiment-utils.ts`
Autopilot loading gates	`packages/browser/loader/src/bundles/all-in-one.bundle.ts`
Autopilot script injection	`packages/browser/loader/src/bundles/autopilot.loader.ts`
Replay metadata ingest	`replay-receiver/apps/instance/src/controllers/app.controller.ts`, `replay-receiver/apps/instance/src/controllers/assets.controller.ts`

The Pulse Ecosystem

Definition​

The systems​

End-to-end data flow​

event-receiver: where events go​

Processing order​

Validation strips undeclared fields​

Kafka outputs​

Response semantics the SDK relies on​

Where definitions come from​

tools-portal-front: authoring and debugging​

Event and attribute definitions​

Settings, segments and experiments​

Live debugger​

Experiment preview​

Session replay lookup​

autopilot: session recording​

Loading gates​

What autopilot reads from Pulse​

Recording pipeline​

Iframes​

replay-receiver: replay metadata ingest​

Where to look in the code​