Analytics

As a colleague of mine once said, “Analytics is simple, but that doesn’t mean it’s easy”, and I have found this to ring very true. What follows is a punch list of what I (DavidF) think are the core concepts that everyone should know about Game Analytics. A good proportion of this riffs off of classic Data Warehousing techniques and concepts, which are their core are still very applicable even in this age of big data.

Telemetry, Logging, Cooked Analytics Data and Reporting are all different things

Telemetry is the data that gets sent from the title and is stored in our cloud databases via Playfab. It is rare that we will report anything at scale directly off of the Telemetry. Telemetry events are ideally carefully designed and strongly typed (see Telemetry Event Design and Implementation Guidelines) and structured such that they can be used to feed into Cooked Analytics Data and drive aggregated reports. Telemetry is aimed less at learning about a singular game instance or match or session, but instead about learning about behaviors at scale, across hundreds or thousands of players and matches.

Logging is what we use for debugging, it is typically less structured and useful on a per-game-instance basis rather than in aggregate. Sometimes we will use logging to understand the severity or breadth of an issue, but generally speaking, generating reports off unstructured logs is a bit of an anti-pattern.

Cooked Analytics Data, aka Facts and Dimensions is the result of an ETL (Extract, Transform, Load) process that aggregates Telemetry data and transforms it into a format consisting of Fact and Dimension Tables. Fact and Dimension Tables are collections of Measures that are sliceable by Dimensions - they are again strongly typed and thoughtfully designed such that they can help answer questions about how players are interacting with our game.

Reporting is what we do off of the Cooked Analytics Data. For example, We might have a report on Daily Active Users (a calculated measure) sliced by Platform (a dimension) where Daily Active Users is calculated as the count of distinct PlayerIds per day per platform. The Report is a visual representation of this number, generally a table or a chart.

Definitions are Crucially Important

Ultimately analytics is about communicating observations about data to stakeholders, be they business, engineering, design, community, or any combination of the above. What this means is that each person receiving the observations will have the propensity to interpret them slightly differently. Given the potentially complex and/or ambiguous nature of these observations, it is crucial to reduce ambiguity as much as possible. Think about how people not being on the same page on the following examples could cause consternation/or and misinterpretation of results (examples chosen from past experience of said consternation):

How do we define Retention and Churn? When is a player considered Churned? 21 days? 30 days? One calendar month?

Who do we consider a Unique Player, do sub-accounts (think XBox live child accounts) count? How about the same player who plays both on XBox and Steam and the same day? Alt accounts? Can we even identify alts even if we wanted to? Does it matter?

Are our Monthly Active Users calculated on the monthly boundary, or are the calculated on the last 30 days of data? What timezone do we use as cutoff to calculate Daily Active Users?

If I’m playing a game starting at 11:59pm and finish at 12:10am the next day, should I be counted in the Daily Active Users for only the first day, or the first and second, or both? This may not sound like a big deal, but if you’re using UTC midnight as your day boundary, bear in mind that is right in the middle of US peak gaming time, so this can substantially skew your numbers.

You need both Behavioral and Attitudinal data to get a complete picture

Behavioral data is what you get from Telemetry, this is the “what” of “what are players doing in our game?”.

Attitudinal data is what you get from observing our Community and through User Research (e.g. surveys). This is the “why” of “why are players doing (or not doing) what they’re doing in our game?” and the perennial “is it fun?” question. That said, it’s equally important to try and quantify our attitudinal data, through measuring community sentiment or survey results in a principled way, otherwise tracking trends and changes becomes squishy at best and impossible at worst.

Combining both Behavioral and Attitudinal data is the Real Ultimate Power(TM) of Analytics, and having both cross-referenceable is incredibly powerful, e.g. “people who were having fun were doing A, B and C, and people who were not having fun were doing D, E and F.”

Analyses are EXPENSIVE, so we must be efficient

There is a reason the roles of “BI Analyst” and “Data Scientist” exist: because detailed analyses WILL take substantial chunks of time.

That said, smart planning, telemetry design and data architecture can substantially improve the efficiency of analyses. Anecdotally on past products, I’ve seen data scientists spend 80-90% of their analysis time massaging poorly designed telemetry and data into a form that allows them to gain insights into whatever question they’re trying to answer, and as a result, only spending 10-20% of their time on actual analysis. With some foresight and thoughtfulness, these numbers can be reversed, and this can yield a substantial improvement in efficiency.

Focus on analyses that are actionable rather than just interesting

Always consider how you are going to use the results of an analysis before embarking on it. If you can’t take any concrete action as a result of a finding, then the analysis is essentially useless, and you’ve largely wasted your time.

If you REALLY want to know what caused something, you need A/B Testing

Causality can only be inferred through controlled experiments, otherwise you’re just guessing.

Be cognizant of the number of samples you will require to measure a particular effect (i.e. know the statistical power requirements of your experiment).

Always limit the number of variables you change, otherwise, even with an experiment, you won’t know what actually caused your changes.