The Data Quality App provides views of the integrity of data across your portfolio, helping you to determine overall data completeness, discover spikes and flatlines that are automatically detected by Atrius, and quickly identify data issues or anomalies.
👍This article will help you:
Gain visibility into portfolio-wide data quality
Fill bill point data gaps
To access Data Quality, go to Apps in the side menu at left, then Data Quality.
Overview
The Overview tab provides a portfolio-wide summary of all your organization's data in Atrius, including percent online/current, average completeness, gaps, spikes, flatlines, and statuses.
Use Filter and All time to update the callouts and tables on each page. You can filter by Point, Type (i.e., Resource), Scope, Integration, Status, Building, Building Group, or Utility Provider.
Using the period filter ("All time")
The following fields update when changing the period filter:
Completeness
Gaps
Spikes
Flatlines
The following fields do NOT update when changing the period filter. They always reflect "now":
Percent online/current shows the proportion of all points and bill points that are online/current to those that are offline/outdated or flatlined at this moment. This does not include disabled, historical, calculated, or manual points with no offline threshold.
Status shows if a point or bill point is online/current, offline/outdated, or flatlined at this moment.
Outdated bill points shows how many bill points are not current at this moment.
Downloading table and column sorting limits
When the period is set to "All time," the 'Download current table view' action and the ability to sort by column will only be available if the results are 1000 rows or less.
If a period other than "All time" is selected, the 'Download current table view' action and the ability to sort by column will only be available if the results are 100 rows or less.
To avoid reaching the limits, try filtering first (by point type, integration, building, etc.).
Completeness
The Completeness tab shows how much data exists for each point. Completeness is calculated as (Time Elapsed - Missing Data) / Time Elapsed * 100.
The Data Completeness column shows how much data exists for that point, over the selected period. A higher percentage indicates more complete data; a lower percentage indicates the presence of data gaps.
Select any column title to sort the table by that column's values.
Column | Description |
Building | The building to which the point belongs. |
Point | The name of the point or bill point. |
Type | The point type of the point or bill point. |
Integration | The integration to which the point belongs. |
First Date | If Period = 'All time', this represents the first date that data were received for the point. Select this to view data in Data Manager, starting at this date. |
Last Date | If Period = 'All time', this represents the last date that data were received for the point. Select this to view data in Data Manager, ending at this date. |
Time Elapsed | The length of time between the First Date and Last Date, depending on period selection. |
Missing Data | The total amount of missing data between the First Date and Last Date, depending on period selection. Select this to view missing data gaps in Data Manager. |
Data Completeness | The percentage of data that exists for each point, calculated as (Time Elapsed - Missing Data) / Time Elapsed * 100. |
Some users find it easier to examine a CSV file to troubleshoot certain issues. To export all data quality information—including data completeness, gaps, spikes, and flatlines—for the filtered view and period, go to Actions, then select 'Download current table view'.
Gaps
Gaps are date ranges where no reading exists on a given point or bill point. Note, the gaps table is filtered by 'bill points' only by default. To view gaps on 'points' as well select Filter > Data Type > Point > Apply.
Two different views are available in the Gaps tab:
Points shows all points and bill points with gaps. Each row is a point. The table is sorted by most to least gaps, by default.
Data Manager shows all gaps across all points and bill points. Each row is a gap. If there are multiple gaps for a point, then there will be repeating rows of the same point, with each row representing a unique gap. The table is sorted alphabetically by building name, by default.
Filling gaps
To fill point gaps, add manual readings via the point Data Manager, or use the Batch Point Data Upload tool.
To fill bill point gaps, add manual readings via the bill point Data Manager, or from the Gaps page go to Actions, then select 'Fill bill data gaps'. A modal window will appear. Follow the steps below.
Download CSV template: Use this pre-populated template to see all existing bill point data gaps.
Complete your CSV template: Fill out the downloaded CSV template by adding known values to the Total Cost and Total Consumption columns, or adding gap justifications to the Notes column. You can also break up gap ranges into multiple rows in your CSV, then re-upload the smaller gaps ranges and values.
Upload completed CSV template: Once you are finished, upload your completed CSV template. Press 'Save'.
Notes
Notes can help explain to your team, or serve as a reminder to yourself, about observed data anomalies.
To add a note:
Select the Data Manager view in the Gaps tab.
Select the dropdown menu at the right of the table for each gap row, then select 'Add a note'. A modal window will appear. When finished, press 'Save'.
To edit an existing note, select the gap row to expand the table and show all notes for that gap. Then, select the "edit" (pencil) icon.
Other ways to add notes
For any point or bill point, select the Data Manager tab, then select the 'Source' view.
In Batch Bill Data Upload app, select Upload data, then 'Download CSV'. Here, you may add notes in the Notes column.
Spikes
The Spikes tab displays unusually high or low readings. To adjust spike sensitivity, select a Spike Threshold for each point.
Once detected, spikes are hidden from Atrius visualizations and can be inspected in Data Manager. Spike Threshold can be set on a point's Profile tab.
In the Spikes table, you have the ability to restore selected spikes (if they are valid readings that have been incorrectly identified as spikes), or delete selected spikes (if they are invalid readings that have been correctly identified as spikes that you wish to permanently remove).
To restore or delete a spike, select one or more spikes using the checkboxes on the left side of the table. Go to Actions, then select 'Restore selected spikes' or 'Delete selected spikes'.
A note on spike detection
Spike detection relies on taking a sampling of readings and using that to create a set of stats that determine what normal readings looks like. If a point with a spike threshold enabled does not contain historical data, it may take a few days of collecting readings before the spike detection is trained and able to successfully detect spikes. If spike detection is not working as expected on a point where a spike threshold is enabled, contact customer support.
Flatlines
The Flatlines tab displays repeated values or zero readings over multiple intervals. Flatlines typically occur when the connection between a point and point gateway or building automation system is broken. To adjust flatline sensitivity, select a Flatline Threshold for each point.
Once detected, flatlines are hidden from Atrius visualizations and can be inspected in Data Manager. Flatline Threshold can be set on a point's Profile tab.
In the Flatlines table, you have the ability to restore selected flatlines (e.g., if they are valid readings that have been incorrectly identified as flatlines), or delete selected flatlines (e.g., if they are invalid readings that have been correctly identified as flatlines that you wish to permanently remove).
To restore or delete a flatlines, select one or more flatlines using the checkboxes on the left side of the table. Go to Actions, then select 'Restore selected flatlines' or 'Delete selected flatlines'.
Meter Replacements
Only the following integrations are supported: Arcadia Bills, Capturis.
The Meter Replacements tab displays bill points which may have been swapped out by your utility provider. Swapping out physical meter hardware can result in existing bill points with an "Outdated" status, and new bill points with an "Unconnected" status. Once you have confirmed a match, update the status of your outdated point to 'Historical', update calculated bill points' formulas, and adjust Whole Building scopes accordingly.
Matches are unconnected bill points that have a similar profile to an existing outdated bill point, based on four or more attributes: utility provider, account number, point type, last/first reading dates, point of delivery ID, and address.
Navigating from Bills to Meter Replacements
You can easily go from a bill point to the Meter Replacements table. In Bills > Bill Points, select the 'Troubleshoot' link below any Outdated bill point with an Arcadia or Capturis supported integration.
Video tutorial
Understand the importance of data quality and cleanliness for your energy reduction initiatives, and how to use alerts in Atrius to save time and increase accuracy.