All data collections will be done by the Sitecore Tracker. It continuously tracks the actions of the visitors on the website. Tracker is independent from xDB and xConnect.
How is the data flow?
- When a new visitor visits a website or channel on a Content Delivery role, a session is created for that visitor.
- The session contains a Contact and an Interaction.
- The Contact stores the information about the visitor and is mapped to a unique ID which is stored by tracker. A Interaction contains information about the visit, for example which pages the visitor has visited, which goals have been triggered and so on.
- Both contacts and interactions contains a number of facets. This could for example be a customers address, emails, contact preferences and more or the geo location where an interaction is made or information about the device which the interaction is made on.
- The Tracker uses reference metadata to enrich some of the facets in the contact or interaction. This means that the Content Delivery role requests information from the Reference Data service.
- When the session ends, the Contact and Interactions are submitted to the xConnect Collection.
- The xConnect Collection role stores the new Contact and Interaction information in the xDB collection database
The xDB collection database can potentially contain huge amounts of visitor information. To handle this, the database can be scaled using a mechanism called sharding
Indexing and segmentation of data
Sitecore XP continuously indexes contacts and interactions when they are created or updated. This enables you to search across contacts and interactions in the xDB.
Indexing of contacts and interactions
The Sitecore xConnect layer creates a unified API for systems to push and retrieve contacts and interactions
When a client such as the tracker on the CD submits data to xConnect Collection , the xConnect Collection saves the data in xDB Collection database.
The xConnect Indexer is a worker role that continuously detects changes in the xDB Collection database and runs an indexing operation.
When the xConnect Indexer retrieves the new data, it processes the data and stores it in xDB Index.
The xConnect Indexer processes all data, including the default contact and interaction facets and any custom facets. You can control how the index processes and stores data by enabling and disabling settings the following facet settings:
- The PII Sensitive setting controls personally identifiable information. When you enable this setting, the xConnect Indexer will not index the data and you cannot search for the information. This setting ensures other privacy compliance and data subject rights, such as the right to be forgotten.
- The Do not index setting ensures that the xConnect Indexer will not index or store the data.
Querying for contacts and interactions
Sitecore XP uses xConnect to retrieve lists or segments of contacts to use in several features and applications, for example, Marketing Automation, Email Marketing and for segmentation in the List Manager.
Typically, the Content Management server executes these features, and queries the xConnect Collection Search role for information about contacts or interactions. Trusted client can also retrieve lists of contacts or interactions.
The xConnect Collection Search role queries the xDB index and retrieves the IDs of the contacts or interactions and then retrieves the contacts or interactions from the xDB Collection database before returning them to the calling client.
Live processing and aggregation of data
When Sitecore collects customer data and interactions, xDB stores the information. However, as it is being collected, xDB also processes the data just in time to use in aggregation and to trigger marketing automation.
The xDB processes data as it is submitted in real time or on demand. Processing can continually aggregate the collected data and make it available for actionable insights or reporting through the Experience Analytics, or to external business intelligence tools.
When a session on the CD ends, it submits an interaction to the xConnect Collection, a record in the xDB Processing Pools database is saved and relays information to the xDB Processing application role about how to handle the new interaction.
The xDB Processing application role continuously polls the xDB Processing Pools database and will pull the recently added aggregation task and start the aggregation process.
During the processing, the xDB Processing role will pull the new interaction from the xConnect Collection role – and can pull additional data needed in the aggregation from other sources, for example the Reference Data service.
Finally once the aggregation is done, the resulting data is stored in the xDB Reporting database.
The concept of historical aggregation you can find here