Resolved -
The repository service has now been fully restored. A new node version has been released that prevents issues upgrading the impacted devices.
Customers may continue to see Repo Connectivity health check warnings on appliance nodes running older versions of the software. These warnings are expected and may take up to 24 hours to clear as systems fully reconcile and refresh connectivity state.
We appreciate your patience during this incident.
Jan 9, 17:03 CST
Update -
We intend to remain in this state with repository access restored but apt package updates disabled until we can fully test and release a version that addresses the underlying issue impacting the affected appliance models.
We expect this state to remain in place for several days while we complete testing and prepare a validated release.
During this period:
- Apt package installs and appliance upgrades will remain unavailable and should not be initiated.
- Appliance nodes will report Repo Connectivity health check failures while apt access is intentionally disabled.
- Other repository-dependent functionality will continue to operate normally.
We will provide additional updates as testing completes and a validated release becomes available. We appreciate your patience while we work to resolve this issue safely.
Jan 7, 09:25 CST
Update -
We are continuing to monitor for any further issues.
Jan 7, 09:25 CST
Monitoring -
We are beginning to restore access to the repository. As part of this process, access to apt packages will remain deliberately disabled to prevent appliance updates.
This approach allows other repository-dependent functionality to recover while we continue to test and validate a fix for the issue impacting a specific subset of appliance models (for example, Lanner NCA-1210 running Ubuntu 22.04 LTS).
Current impact during this phase:
- Appliance nodes will not be able to install or update apt packages.
- Appliance node upgrades will continue to fail and should not be initiated.
Other repository-backed functionality will begin to recover as repository access is restored.
We will keep apt package access disabled until a validated release is available that resolves the issue for the affected models. Further updates will be provided as testing progresses.
This will is expected to be the last status update for the evening.
Jan 6, 20:24 CST
Update -
We are continuing to work on a fix for this issue.
Jan 6, 20:22 CST
Identified -
We have identified an issue that could cause a small subset of node appliances to fail during updates. As a preventative measure, we have temporarily taken the package repository offline.
As a result, the following impacts are expected:
- Appliance node upgrades will fail while the repository is unavailable. Customers should avoid initiating upgrades during this time.
- Appliance nodes may report Repo Connectivity health issues. These are expected to resolve once the repository is restored.
- Container images cannot be pushed or pulled. New containers will not start if they require an image pull. Existing running containers are not impacted and can restart without issue, provided the container version tag is not modified.
- Container exec output files will not upload.
- Debug logs will not be published.
We are actively working to determine the safest path forward and will restore repository access as soon as possible.
Jan 6, 16:28 CST
Update -
We are continuing to investigate this issue.
Jan 6, 16:27 CST
Investigating -
Out of an abundance of caution, we have made the package repository unavailable while verifying some configuration changes.
Jan 6, 13:47 CST
