Overview
The Rate Accuracy Report helps you keep track of the AI model's performance. The accuracy of AI predictions will always show some variation, so the goal is to keep within a certain margin of error. You can find this margin of error expressed as a percentage on the Rate Accuracy Report, so you can gauge the AI's accuracy for yourself.
Each time we train a model for you, we take a sampling of your past two weeks of data (or more in certain cases) and we withhold that from the model’s training. This dataset is called the test set. The model then runs predictions on the test set, which we compare to the rates on successful bookings to see how well the model is performing.
You can find the Rate Accuracy Report by clicking Accuracy Report right under the Get Rates button on the Rates tab:
You can always reach out to your CSM or business analyst with any questions about the Rate Accuracy Report.
Margin of Error
The accuracy of AI predictions will always show some variation, so the goal is to keep within a certain margin of error. You can find this margin of error expressed as a percentage on the Rate Accuracy Report so that you can gauge the AI's accuracy for yourself.
Think of a game of darts. We want to be as close to the bullseye as possible, right? In this scenario, the bullseye is the actual booked rate.
Go Live Tab
The Go Live tab focuses on readiness for implementation and initial performance metrics.
The margin of error on rate predictions is the first indicator we use for gauging the accuracy of a model. We break down the margin on the Go Live tab by Confidence Level interval, new vs. traveled lanes, and Confidence Level distribution for a high-level view of model accuracy.
We monitor these stats closely during your first 30 days of using Greenscreens, and we continue to keep track of them after that, because they’re a good quick indicator of how your model is doing. Our goal is to keep the average margin of error under 10%, but you may see a higher percentage there, especially where the Confidence Level is low. Where the margin of error is high, you might want to get a second opinion on loads at that Confidence Level.
Widgets
Margin of Error by Confidence Interval shows how far off the model is on average when we compare predictions to actual outcomes.
New vs. Traveled Lanes compares the model’s performance on new lanes (those you haven’t run predictions for in the past three months) and traveled lanes (those with prediction data from the past three months). This helps in identifying the model's learning curve.
Confidence Distribution shows how well the Confidence Level on predictions correlates with accuracy in our tests.
Filters
The Go Live tab shows aggregated data for a particular time period. The default is the past 30 days. You can select another time period in the drop-down menu at the top right of the page:
Usage Tab
The Usage tab provides analytics on user interaction and model utilization. It’s designed to give you a better idea of how your Reps are using Greenscreens.
Widgets
Predictions by Confidence breaks down your prediction by Confidence Level for a layered view of model certainty. You can view details for each time period by hovering your mouse over the corresponding bar:
Prediction Volume by User: illustrates user engagement by showing prediction counts for individual users. If a user is not logged into a Greenscreens account, their predictions will be counted under Undeclared API Users.
Bid and Quote Volume by User offer insight into user behavior by showing bid and quote counts for individual users.
Filters
You can view data for a different time frame using the drop-down menu at the upper right of the page:
You can filter the Usage tab by:
Transport type:
Pickup and dropoff location:
User:
Exporting Data
To export your usage data to an Excel spreadsheet, click Export to XLSX:
Live Model Performance Tab
The Live Model Performance tab offers a deep dive into the live operational performance of the model. It shows forecast bias and a list of outlier predictions. It includes enhanced filtering so you can gain tailored insights into model performance. You can not only see if your model is shooting a little high or low, you can see exactly where the bias is – are your predictions a bit low for reefer loads, for instance, or a little high for loads leaving a particular market area? It allows pricing managers to dive in and see where they should take a look at the network, for instance, or use the Target Buy Rate to incentivize a group, or make general adjustments.
Widgets
Forecast Bias % shows the high or low relative bias of the model as a percentage.
Clicking Use absolute values will show you how close the predictions are without regard for whether they’re high or low:
Forecast Bias Distribution % shows how bias is distributed across predictions by percentage. Clicking Hide extreme outliers will remove any outliers that might bias the average:
Forecast Bias $ shows the high or low relative bias of the model in dollar amounts:
Forecast Bias Distribution $ shows how bias is distributed across predictions by dollar amount. Clicking Hide extreme outliers will remove any outliers that might bias the average:
Clicking Use absolute values will show you how close the predictions are without regard for whether they’re high or low:
Top Outliers shows predictions that fall outside the general range:
Filters
You can view data for a different time frame using the drop-down menu at the upper right of the page:
You can filter your Live Model Performance data by:
Transport type:
Confidence Level:
Pickup and dropoff location:
Customer or carrier manager:
Matching time and transit time:
Single- or multi-stop (Specialties):
Under Widget Settings you can select whether to compare the Target Buy Rate or the Network Rate, and whether to view rates as flat or per mile:
Exporting Data
To export your Live Model Performance data to an Excel spreadsheet, click Export to XLSX:
To export your top outliers to an Excel spreadsheet, click Export to XLSX in the Top Outliers widget:
Best Practices and Tips
Maximizing Value
Data interpretation: When analyzing the Forecast Bias, consider both percentage and dollar values to understand not just the direction but the magnitude of any discrepancies.
Strategic filtering: Use the advanced filters to segment data for specific scenarios, such as comparing weekday versus weekend performance, or evaluating specific customer and manager regions.
Comparative analysis: Regularly compare your rates with the network to identify competitive advantages or areas for improvement.
Offset your markup: One advanced way to use this data is to potentially offset your markup based on the direction of the model's bias.
Identify potential problem areas in your freight network: This data can help you find potential problem areas in your freight network that need attention.
Troubleshooting
No data showing: If a filtered view isn't displaying data:
Ensure that the date range is correctly set.
Verify that there is data available for the selected criteria.
Data export issues: Should there be any issues exporting data to Excel:
Check for pop-up blockers
Make sure your browser allows downloads from the platform.
Visualization clarity problems: If charts are not displaying correctly:
Try clearing the cache or refreshing the page.
Verify that your web browser is up to date with the latest version.
FAQ
How often is the Rate Accuracy Report updated?
The report is updated daily to ensure you're working with the latest data.
What should I do if I notice a significant margin of error (10-15%+) in my report?
Investigate the underlying factors that may be contributing to the error, such as data quality or market volatility. Adjust your model parameters accordingly and monitor the changes.
Reach out to your analyst and ask him to investigate
Can I customize the date range for the data displayed in the report?
Yes, you can select custom date ranges using the date filter feature on all tabs.
Why was the 'Predictive vs. Booked' tab removed?
It was removed due to low usage and the confusion it caused, as well as its limited value in contributing to rate accuracy insights. It was superseded by the value produced by the updated Live Model Performance view.
Rate Accuracy Report Training
If you would like special training in using the Rate Accuracy Report, ask your analyst and Customer Success Manager to give you some training during a sync. If you would like to go over the report outside of sync time, just shoot your CSM or analyst an email and they'll get you set up.
You can also request training videos on most Greenscreens features from Sync Logistics Training, just by letting your CSM know you'd like to sign up. There's no charge for Sync training for Greenscreens customers.