Remote Troubleshooting


~80% of complaints were remote-access problems	Zero shell access by design	3 tiers of recovery	2 capabilities shipped, publicly dated

SSH was the mechanism, not the goal

During beta, Panorama customers had SSH. When the appliance misbehaved they logged in remotely and fixed it.

At general availability, SSH was gone from the production appliance for security reasons. For a device sitting inside a customer’s operational network processing their camera feeds, a remote shell wasn’t defensible. That was the right call, and it meant that on launch day every customer lost their only diagnostic tool with nothing to replace it.

The asymmetry made it sharper. SSH still existed on the Appliance Developer Kit which was a variant not secured for use on production networks or workloads. Developers building applications could shell in. Operations teams running production fleets could not.

The task wasn’t “add troubleshooting features”, it was to reproduce the capabilities SSH gave people to fix the problem.

SSH High-level design for Panorama remote troubleshooting

Replacing one unbounded tool with several bounded ones

SSH is useful because it has no limits. Anything replacing it has to be a set of narrow capabilities, each individually safe. This turns a design question into a prioritization question: which small number of bounded actions covers the largest share of real failures?

Answering it meant knowing which failures actually happened. I proceeded to differentiate device-related complaints from cloud-related ones. Across the customers we’d heard from, roughly 80% of what people raised were problems a simple device reboot would have solved, therefore the majority of customers simply needed remote reboot as to avoid needing to send someone on site.

That produced a three-tier model, each tier reaching further and costing the customer more:

Tier	Capability	Reaches
1	Remote reboot	The largest class of reported problems
2	Status reporting + CloudWatch deep-links	Diagnosable issues, device online
3	Encrypted log transfer over USB	Device offline or unreachable

AWS Panorama Devices list showing 18 devices with Online, Offline and Awaiting statuses, and the Actions menu open on three entries: Reboot, Update software, and Delete — Reboot sits beside Update software and Delete, on the list where operators already work.

Reboot confirmation dialog for device Security-SJC-018-01, stating that all applications will be stopped and the device power cycled, that the reboot takes about five minutes, and warning that pending application deployments may not resume — The confirmation names the device, what stops, how long it takes, and what may not come back.

Design board for the reboot flow showing seven console states side by side: the Actions dropdown menu, the confirmation modal, a 'cannot proceed' modal for a device that cannot be rebooted, a blue 'reboot in progress' banner, a red 'reboot failed' banner, a green 'reboot succeeded' banner, and the reboot action on the device detail page — One menu item, seven states — including the two ways it fails. For a troubleshooting feature the edge cases are the product.

Resource status lifecycle diagram for Panorama devices, listing steady statuses (Online, Offline, Error, Update needed) and transitional ones (Awaiting credentials, Creation pending, Deletion pending, Loading status, Updating software, Rebooting), then charting the status sequence for add, deletion, device reboot, software update and device lease expiry, each with its success and failure paths — Every status a device can hold, and every path between them. Customers could not act on a device without first knowing what state it was in.

Designing an interface with one bit of output

Encrypted log transfer over USB aka Tier 3 is where this stops resembling normal UX work.

When the appliance can’t reach the cloud, the design surface isn’t a console. It’s a USB port, an LED, and a technician who may have four minutes on site. No screen, no text, no error code, and no way for them to verify their own work, because the log image is encrypted and only AWS can read it.

The LED had to carry a sequence someone could act on: drive recognized, transfer running, complete and safe to remove, failed and retry. Getting it wrong doesn’t confuse a user; it produces a corrupted image and a wasted site visit.

Getting there at all took sustained work. AppSec’s opening position was that extracting any data from the appliance was off the table. I wasn’t in those review sessions, so the persistence was aimed at my PM and engineers, who were. The solution had to satisfy the security bar rather than request an exception to it: an encrypted image the customer can carry but cannot read.

End-to-end flow for retrieving device logs over USB, from the IT Admin persona through four stages: reading the documentation portal, formatting a USB thumbdrive and creating a /panorama/managed_logs folder, inserting the drive and rebooting the appliance while the status LED goes from solid green to blinking blue and back to solid green when the transfer is complete, then plugging the drive into a computer and opening a support ticket so an AWS support engineer can decrypt the logs — Blink patterns mapped to states and to the technician's next action. The whole interface, on a device with no screen.

How I worked

Set the north star — The target was never “what will security permit.” It was, “a customer resolves their own device issue without sending someone to site”.

Align the organization — Framed the problem as tiers of reach so the team could argue about coverage rather than individual tickets, and catalogued every resource status across the console so customers could tell what state the system was in.

Engage the constraints and push them — I partnered early with the engineer who owned the security boundary, not to learn what I was allowed to design, but to learn the techical environment and argue credibly about where the CX line should sit.

Deliver with rigor — Documented states, interactions and edge cases as a deliverable. For a troubleshooting feature the edge cases are the product: a reboot sent to an offline device, a transfer interrupted mid-write, a log group that doesn’t exist yet.

Impact

Remote reboot shipped on 13 October 2022; offline log retrieval 24 August 2022 . Both can be seen in the AWS Panorama release history.

Reboot addressed the largest class of reported problems, eliminating site visits for customers who were rarely near their hardware. For Deloitte’s deployment with the Victorian Department of Transport, Panorama appliances sat across 1,200 cameras and 23,000 km of road network. A site visit isn’t an inconvenience, it’s a truck and a lane closure. Deloitte and TaskWatch, the two largest customers, said the suite addressed their biggest operational problem.

What I’d change

I should have argued for this earlier in the product cycle.

Troubleshooting was treated as an operational concern, something you need once a fleet is running. That framing is wrong. The moment customers most need to diagnose a device is during initial integration, when it’s going into a network nobody has tested it against, by a team seeing it for the first time, often with the engineers who understand it a great distance away. A customer who can’t get past installation never becomes an operational customer at all.

The platform arrived there eventually from the other direction. A year later the Panorama SDK shipped local debugging and off-device testing. Same gap, addressed at build time, where I’d argued it belonged.

AWS ended support for Panorama on 31 May 2026. The work here reflects the service as it shipped.