Analyzing Nudge Results

Results for nudge goals with attribution window

This guide will help you understand the new results for nudges. We are introducing nudge-level goals and an attribution window for tracking these goal events with this feature release.

📘

Important

Nudge-level goals with an attribution window logic for goal tracking feature is available from the below SDK version onwards. Please ensure to upgrade your SDK:

• Android Netcore CE Nudges SDK v8.5.14 and above
• iOS Netcore CE Nudges SDK v8.5.8 and above

Understanding the Goal Event

A Goal event is essentially a conversion event that you would want to track to measure the impact of the nudge. We measure such impact by directly looking at how users perform the goal events after they’ve seen the nudge.

Each goal event criteria is made up of the following data points:

• Goal event - Here you can select any event as the goal event. E.g. Subscription Success event for a nudge shown near the _Subscribe _button.
• Goal type - Here you have 3 options to choose from:
  • Total events - Select this option if you want to track total goal events performed by users who've seen the nudge.
  • Unique users - Select this option if you want to track the unique users performing the goal event after they've seen the nudge.
  • Sum of event property - Select this option if you want to track the sum value of a property from the selected goal event. E.g. Subscription_Plan from Subscription Success event in our above example.
• Event property conditions: Optionally you can add event properties conditions in the goal criteria. For E.g. Subscribe button click where page is equal to homepage

Configuring the Attribution Window

An attribution window is a configurable time frame for a given app. When a user performs the defined goal event within this time frame after seeing the nudge, we consider such an event as an attributed goal event for that nudge.

You can find this attribution window option in the app settings on the Hansel dashboard.

Adding Nudge-level Goals

🚧

Please do note that going forward, you will no longer be able to set up the journey level goal.

You can now set a goal criteria for each nudge while configuring the nudge. In order to do so, follow the below steps:

• Click on the given nudge to edit it and go to step 4 to add a nudge-level goal.
• Here you can add up to 3 goal event criteria.
• Once done, click on proceed to save the changes and test or deploy the journey on production.

Once results data starts flowing in, below is how you will start seeing the nudge results on the dashboard.

Below is a quick summary of what each data point that is shown on this page represents:

• Overall population in the segment - These are unique users who form the part of the population of users who are eligible to see the nudge.
• Unique users who have seen the nudge - These are the unique users who have seen the nudge after they perform the trigger condition for the given nudge.
• Attributed users - These are the unique users who have seen the nudge and also performed the goal event within the attribution window.
• Total event counts by attributed users - These are the total number of goal events performed by the users who have seen the nudge within the attribution window. This is shown only if your goal type is of "Total Events".
• Sum of event property for attributed users - This is the sum value of a property for all the goal events performed by the users who have seen the nudge within the attribution window. This is shown only if your goal type is of "Sum of event property".

Adding Goals to A/B experiment

When you create and set up the A/B experiment, you will now have the option to set up goals for the experiment. The goal criteria definition remains the same as discussed above. You can find this option to set goals in the Add or Edit A/B experiment node of the nudge journey as shown below.

Once you deploy the journey on production, you will start seeing A/B experiment results as given shown below:

1. A/B Experiment Summary for results

📘

What is Control group?

A Control group is a set of users who get the default experience of the app and such users do not see the nudges even if they do the trigger event.

From results tracking perspective, we simulate the below-mentioned attributed data points for users in the control group.

This section of the results will give you an A/B Experiment level summary results. You will essentially get a comparison for the Control group vs all the variants in the experiment for the below 5 data points:

• Overall population in the variant - These are unique users who fall in the branch of the given variant.
• Users in the variant who have performed the goal - These are the unique users who have performed the goal event in general irrespective of they've seen the nudge or not.
• Total event counts by users in the variant - These are the total number of goal events performed by the users who fall in the branch of the given variant. This is shown only if your goal type is of "Total Events".
• Sum of event property for users in the variant - This is the sum value of a property for all the goal events performed by the users who fall in the branch of the given variant. This is shown only if your goal type is of "Sum of event property".
• Deviation from the control group - This is the percentage difference between the value of 'goal type' (e.g. unique users/total events/sum of event property) for a given variant and the control group.

2. Attribution data for A/B experiment

This section of the results will give you an A/B Experiment level summary results. You will essentially get a comparison for the Control group vs all the variants in the experiment for the below 5 data points:

• Unique users who have seen the nudge - These are the unique users who have seen the nudge after they perform the trigger condition for the given nudge.
• Attributed users - These are the unique users who have performed the goal event within the attribution window after seeing the nudge.
• Total event counts by users in the variant - These are the total number of goal events performed by attributed users within the attribution window. This is shown only if your goal type is of "Total Events".
• Sum of event property for users in the variant - This is the sum value of a property for all the goal events performed by attributed users within the attribution window. This is shown only if your goal type is of "Sum of event property".
• Deviation from the control group - This is the percentage difference between the value of 'goal type' (e.g. unique users/total events/sum of event property) for a given variant and the control group.

3. Ending A/B experiment and choosing the winning variant

Once you analyze the A/B experiment results and you want to end the experiment, you can click on A/B node - here you will see the "End Experiment" button.

On this next screen, select the winning variant. Once you click on Done, this A/B experiment is replaced with the select variant rolled out to 100% of the users by default.

📘

Important

1. As a best practice, we always recommend running an A/B experiment and conclude the experiment outcome by looking at the results.
2. Having another A/B experiment run within an already running A/B experiment is not allowed. In order to do so, you must end the previous A/B experiment.
3. Whenever you extend any variant branch of the A/B experiment with another nudge (either roll out in one go or roll out in phases) - you will lose the attribution data for the variant nudges. However, you will still be able to see the nudge-level goals results.

Analyzing results of nudge not shown cases

In the diagnosis tab of nudge results, we are now showing details of users who could not be nudged along with reasons behind the same. Read more about the same here

Analyzing Feedback nudge results

Net Promoter Score

For the Net Promoter Score (NPS) nudge, you can view the following details (as shown below)

• Total submissions - This is the count of submissions for your NPS nudge
• NPS score
• Percentage of promoters, detractors, and passives

Where, NPS score = Percentage of Promoters - Percentage of Detractors

As shown below, the graph shows the timeline of the number of promoters, detractors, and passives

If a user rates above 8, the user is classified as a Promoter.
If the user rating is between 6-7, the user is classified as a Passive user.
If a user rates 5 and below, he/she is classified as a Detractor.

Rating and Review

The average rating of all the feedback submitted is displayed as the result. (as shown below)

Also, you can see the distribution of users who voted for each point on the rating here (for eg., users who rated 2 or 3 as a percentage of total submissions)

In the graph shown, you can see the average rating for each day of the selected period.

Multichoice Questionnaire

You can view the number of users who selected a choice in your Multichoice Questionnaire here. (as shown below)

You can view the percentage distribution (mentioned against each option) as well. The chart shown plots the total number of times each option was selected.