While compliance definition varies greatly in different contexts, it's an essential factor to monitor during the study to ensure collecting high-quality and representative data. The compliance report provided in this section is defined as the number of records each participant has uploaded per hour during the participation period, for each stream type.
This provides an easy way to monitor the amount of data provided by each participant, and quickly spot non-compliant situations which might require an intervention. The purpose of the compliance presented here is to show a bird's-eye view on the quantity of the data received from each participant.
One of the main uses of these graph is to show whether data is available for a given time window or not, and if not, whether data is missing from specific streams, or from all streams. Such information can be interpreted as follow:
- If the data is available for all streams, it means the app was functioning as expected during that time window (though the quality of data should be investigated separately).
- If the data is missing for a set of streams, it shows the app was functioning properly, though an external factor was preventing the app from collecting data for a specific stream. For example, while the app was running, user had manually turned off her GPS.
- If the data is missing from all streams, very likely the app was not operational during that time window. Therefore none of the automatic surveys were triggered either. We will explain the potential reasons for such case in the remaining of this section.
At the same time, the graphs presented here do not represent the quality of the data provided. For example assume your study requires EMA responses which is issued 3 times per day. Participants can answer all questions in the EMA, or they can skip some or all of it. In each case, Ethica app will upload their responses, and counts it as one EMA response. Therefore while the compliance report shows 3 responses were provided from each participant, the quality of the response is different.
As another example assume your study records GPS data. The compliance report can easily show how often the participant provided GPS data and how often they did not. But this report does not distinguish between cases where the particiapnt had forgot to carry their phone and abandoned it on the desk, versus the cases where the participant carried the phone with her at all times.
You can monitor the data quality for each stream separately on its relevant section of the website.
Plotting the report
To plot the compliance report, open Compliance section by selecting it from the left panel. The Compliance page allows you to select a participant, a time period, and one or more streams you are interested in. For example, in the image below we are requesting the compliance report for participant #231, from May 8th to June 7th 2016, for Bluetooth, GPS, and Wi-Fi streams:
Pressing Go will extract the number of records uploaded by participant 231 for each of these streams, per each hour of the specified period, as shown in the image below. You can see that each stream generates different number of records per hour. For example, Wi-Fi has generated between 100 to 1500 records per hour (it can be said user was in proximity to maximum of 1500 Wi-Fi networks during a one hour period), while GPS records are mostly below 600 records per hour, and for Bluetooth it's mainly below 100 records per hour.
It's important to note that the numbers shown for each stream cannot be compared to other streams. For example recording 100 Bluetooth data in 1 hour indicates participant was in proximity of 100 Bluetooth devices, and it cannot be compared in any form to recording 1000 GPS locations in the same time window.
For each graph, if you move your cursor on the line, for each data point you can see the time and the number of records represented by that data point. You also can drag your cursor over a specific period to zoom-in, or double-click on the graph surface to zoom-out, as shown in the following image:
The following image shows the number of GPS records uploaded from a given participant during May 8th.
The curser in this graph points at 7am, which shows there has been 217 GPS records at this hour. It means the participant has uploaded 217 GPS records from May 8th, 2016 7:00am to May 8th, 2016 7:59am, inclusive. The timezone of this values is the participant's local timezone.
This graph also shows starting from 7am, the participant's device has uploaded considerably more GPS data until 4pm, at which point again the device has reported a modest number of GPS records. This is in accordance with GPS data collection logic (as explained in GPS stream description), which tries to monitor participant's mobility, and record GPS data only if necessary.
Therefore, this graph can be interpreted as follow:
The participant has visited many locations during the day (from 7am to 4pm), while after 4pm and before 7am, she has been mainly in the place and has not moved as much.
The following image shows the number of Wi-Fi access points observed by participant's device per hour, during May 8th.
The selected point in the graph shows the participant's device has scanned 456 Wi-Fi access points from 4:00pm to 4:59pm inclusive. Note that these access points are not unique, so it does not mean the participant was in proximity of 456 unique access points. Ethica scans for access points in proximity on average every 5 minutes, and assuming the participant is been stationary for the whole hour, the same access point can be scanned 12 times. Therefore a given access point can be counted on average 12 times per hour.
While the number shown here does not indicate unique access points, the higher number usually indicates the denser Wi-Fi networks in proximity, which is a good indication of a densely populated, mostly commercial, areas.
Ethica guarantees exactly one battery reading per operating cycle, as explained in Battery stream section. We also know Ethica has on average 12 operating cycles per hour. Based on this two points, looking at the battery compliance plot we can observe how long the Ethica app was running on participant's device and how long it was not. We will explain these inferences using the plot below, which shows the hourly battery readings from May 7th to May 8th:
You can see in this plot that for most cases, participant recorded between 10 to 12 battery records per hour, which is consistent with our statement on having 12 operating cycles per hour.
Between 3am to 2pm May 7th there was no battery readings, which indicates the app was not running during this time. This can be either because the device was turned off, it was out of battery, or the app had stopped operation either explicitly by participant or due to other issues.
Also between 2am to 4am May 7th the readings were very low, 2 and 3 readings respectively. This usually happens when the participant does not interact with her device for a long time. In this case Android puts device on deep sleep and lowers the operating cycle from 12 cycles per hour. Usually this drops to 4 or 5 cycles per hour, and dropping to this low is not that common. In most cases such drop is not a point of concern.
Survey responses plot shows the number of responses recorded at each hour. In addition to the number of responses fully, or partially, completed by participant, this count also includes the number of questionnaires cancelled or expired.
Unlike other streams, you don't expect to have some survey responses for each hour. Therefore the plot for survey responses looks a lot more sparse than other plots. The following image shows the number of responses provided by a participant during May 8th 2016:
As you can see for most hours there is no response, and at 8pm there is 3 responses recorded. If you look closely, you notice there is also one survey reported at 5am and another at 6am. Assuming you have made sure no automatic survey is issued during night hours, the most likely scenario is that the questionnaire had been issued previously and it was expired at this time. Therefore Ethica has marked them as expired and uploaded them, which still counts here as one response record. We can confirm this by looking at detailed data in Surveys section.
Selecting Bluetooth in the streams section will plot the number of Bluetooth devices scanned in proximity per hour. Similar to Wi-Fi access points, the number of visited Bluetooth devices does not show unique devices visited, but each device can potentially be scanned on average 12 times per hour, once for every 5-minute interval which the destination device is been in proximity of the participant's device.
The following graph shows the number of devices visited by a participant during May 7th 2016. You can see no device is recorded from 3:00am to 2:00pm. This can be due to multiple reasons:
- There has been no Bluetooth devices in the proximity
- Participant has turned off the Bluetooth on her device to save battery
- The phone or the app has turned off and was not operating at that time.
To distinguish between these scenarios, we can use Battery data. The following graph shows the number of data recorded per hour for both Battery and Bluetooth streams.
You can see here at exactly the same period that no Bluetooth data was recorded, no battery data was recorded either. Considering the consistency of battery data, we can be sure the app was not operational at that time.
Other automated streams
You can plot the number of records uploaded by each participant from each stream in Compliance section. It's important to keep in mind the number of records plotted here is aggregated for each hour, and should be interpreted by the number expected from that stream. Some streams such as accelerometer upload thousands of records per hour, while others such as survey responses upload either none or very few.
The following image shows the number of records uploaded from accelerometer and light sensor during the course of one day from one participant. You can see while both these data are from automatic sensors, they are uploaded in different quantities.
Common causes of non-compliance
Using the plots shown here we can find out how much data is uploaded by each participant at any period of time. Here we describe the main reasons causing a gap in each participant's data.
Turning off specific sensors to save battery
Some sensors can be turned off externally, such as Wi-Fi, Bluetooth, or GPS, which prevents data collection from that sensor. In this case, Ethica shows a relevant notification to the user, letting them know the study is partially interrupted and they need to turn on the sensor to resume the data collection. You can see different notifications participants might receive in Participants section.
While the status of each sensor (whether a given device provides the sensor, and if so, whether it's enabled or not) is recorded in Ethica operational logs, as of now there is no way to access these logs directly. We are working to provide ways to access these data as well.
In the meantime, you can check the status of other streams. If during the same time other streams were providing data, but not the stream you are interested, it's usually an indication that other external factors prevented Ethica from operating that particular stream.
Snoozing the app
In order to ensure participant's privacy, Ethica allows participants to snooze the app. Snoozing the app puts the app to sleep for 1 hour, and participants can put the app to snooze as many times as needed. You can read about this feature in Snoozing the App section.
While the app is snoozed, no data is being collected from any streams. While participants can start any user-triggered survey and respond to it, none of the automated surveys are issued during this time either. While Ethica records explicit information on using snooze behaviour by each participant, as of now there is no automated channel to access this data.
Turning off the sensor permanently from the app
While selecting data streams for a new study, you can choose the stream to be optional or mandatory. If the stream is marked as optional, participants will have the option to opt-out from the stream completely and not provide the requested data. You can read the steps involved for participants to opt-out from a stream here.
If participants opt-out from a stream, no data from that stream is recorded and uploaded, therefore the compliance plot for that stream will be empty.
One of the most common reasons for a participant not to upload the data is if their device has run out of battery and has been off for sometime. This obviously leads to no data being collected or uploaded during that period.
If a device turns off for any reason, none of the data collected prior to it will be lost. Also any surveys in progress will be resumed when the device turns back on again, and participants will be able to continue answering the surveys. When the device turns on, Ethica will start operation automatically and participants don't have to remember to start the app.