If you get a notification in the Honeycomb trace viewer that there are missing spans, this document will help you understand why that may be happening. There are several potential causes:
Spans are outside of the time range for the query
Whenever you’re viewing a trace in the Honeycomb UI, the underlying data comes from a standard Honeycomb query that is cached. If you’re looking at a very recent trace, all of the spans may not have reached Honeycomb yet, and pressing the Rerun button may help. Once received by Honeycomb’s API, spans are typically available for query in under 2 seconds. Some other things you can try:
Verify the timestamps on each span
You must set the timestamp to the start of the span and provide the duration of the span as a part of the span record. Within your system, double check that the timestamp is being set correctly (e.g. not set to 1970-01-01T00:00:00 or some other unexpected value).
Ensure the query time window is large enough
Once you have verified the timestamps are set correctly, be aware that we currently only locate spans in the time range of the original query. So if, for example, you do a query of "Last 30 minutes" and then on click a traceId, we only locate the spans in that 30 minute window. If some of the span timestamps are older than that, you won't see them.
Spans made it to Honeycomb, but were not accepted
Were your events Rate Limited or Throttled?
Honeycomb uses Rate Limits to protect our system from unexpectedly large spikes of traffic that may impact our operations and customers, and exceeding these limits can mean some data is dropped. For Free and Pro teams that have gone over their monthly event quota for two consecutive months may be Throttled. For more information see this document.
In case of either situation, your team owners would have received an email notification and you would also see these marked in the graph on your Usage page.
Ensure events were not rejected due to incorrect API tokens or improper formatting
Honeycomb will reject events as errored if they are submitted with an incorrect API token, or if the events don’t conform to either the Honeycomb Events API or the OpenTelemetry OTLP wire format. Check your application console logs for any errors logged by libhoney or OpenTelemetry.
Spans never made it to Honeycomb
Serverless platforms (like AWS Lambda)
Serverless platforms are designed to suspend a task once it has returned all user data, and this may interfere with background tasks such as Honeycomb Beelines or OpenTelemetry which will send telemetry asynchronously. This is particularly a problem for lambdas which are very infrequently invoked, because they don’t get a chance to resume sending telemetry on subsequent invocations. For more information see this document about our AWS Lambda integration.
Events are being dropped by overloaded proxies (OpenTelemetry Collector or Honeycomb Refinery)
Many customers choose to send all of their outbound telemetry through an on-premise proxy service such as the OpenTelemetry Collector or Honeycomb Refinery. Both of these services have configurable memory limiters, or may be limited by hardware (insufficient memory, CPU or network bandwidth).
Look for inconsistent sampling
Inconsistent in-app sampling settings across multiple apps/services (participating in the same trace) can cause missing spans, as can an incorrectly functioning custom sampler. You can check the sample rates of all the spans participating in a trace by running a query in Usage Mode, filtering for a specific trace ID and grouping by Sample Rate and name.
Ideally all of the spans in a trace should have the same sample rate, but if you do mix sample rates ensure that leaf spans of the trace have higher sample rates than parent spans.