Overview

When submitting a Support ticket for a private Server installation of CircleCI, you may be asked to provide a support bundle by the CircleCI Support Engineer working on your ticket. This can be created by following the steps below:

Prerequisites:

A running deployment of CircleCI Server 4.0 (or higher) with access to the cluster/namespace using kubectl

Generating a Support Bundle on Server 4.x

Confirm that you have access to the cluster/namespace by running:
```
kubectl -n <namespace> get pods
```
Make sure that krew is installed
- Krew installation instructions can be found on the official documentation page here
Install the support-bundle (kubectl plugin) by running:
```
kubectl krew install support-bundle
```

When ready, run the support bundle from the current directory and wait for it to finish

kubectl support-bundle https://raw.githubusercontent.com/CircleCI-Public/server-scripts/main/support/support-bundle.yaml

Generating a Support Bundle for Container Runner

For Container Runner, follow the steps above for installing krew and the support bundle plugin but use the following command for the bundle:

kubectl support-bundle https://raw.githubusercontent.com/CircleCI-Public/circleci-support-scripts/refs/heads/main/container-runner-support-bundle/support-bundle.yamlkubectl support-bundle

Additional details can be found in our documentation here.

Important Considerations

Log Line Limitations

⚠️ Important: Support bundles have a default limit of 10,000 lines per pod log. (https://troubleshoot.sh/docs/collect/logs/#limits-optional) This limitation means that depending on when you create the support bundle, critical error messages might not be captured if they've been pushed out by newer log entries.

Recommendation: To ensure all relevant logs are captured:

Generate the support bundle immediately after reproducing the error
If possible, restart the affected pods before reproducing the issue to ensure a clean log slate
Consider capturing pod logs separately using kubectl logs if you need more than 10,000 lines

Troubleshooting: Rate Limiter Error

When generating support bundles, you might encounter the following error:

failed to get log stream: client rate limiter Wait returned an error: context deadline exceeded

This happens when Kubernetes API rate limiting is too restrictive.

Solution

Check your current settings:

kubectl get configmap kube-proxy-config -n kube-system -o yaml

Edit the configuration:

kubectl edit configmap kube-proxy-config -n kube-system

Update the rate limits:
The code expects to max out to 100 so inline with that: https://github.com/replicatedhq/troubleshoot/blob/main/pkg/constants/constants.go#L6-L9
```
clientConnection:
  burst: 100 # Recommended: 100-200
  qps: 100 # Recommended: 100-200
```
Verify the fix:
After applying these changes, run your support bundle operation again. The error should be resolved.

Troubleshooting Self-Hosted Runners (Machine Runner & Container Runner)

Does Container Runner support SSH Debugging?