Cloud native: we're building the world's most advanced distributed system
Also cloud native: Run these 10 agents and sidecars on every node
— Matt Rickard (@mattrickard) May 25, 2021
s
The agent (or sidecar) pattern is everywhere in distributed systems now that we have Kubernetes everywhere. The agent simply is a program that runs on the same machine as your program to provide functionality. One example is the Envoy Proxy which makes the network transparent to applications by providing service discovery on localhost (i.e., a service mesh). The agent pattern has existed forever but is even more prevalent now that we have systems like Kubernetes that support pods – enabling containerized applications to share certain namespaces. An agent might also just run on every node, separate from your application.
But the agent pattern has turned simple deployments into complex ones. Not only are they used for service meshes, but also logging, monitoring, and security. But it's not even deployment that's more complex – it's debugging, management, and resource consumption. eBPF has enabled agents to collect even more granular data about running processes and state.
Not all use cases need agents. Kubernetes and Cloud APIs provide a stable control plane to extract information. Some security startups are experimenting with agentless scanning by analyzing just-in-time snapshots. It's a different kind of observability, but sometimes just as effective.
Of course, there are also startups going the other direction, making agents do even more work than before. In a world where agents might have had a narrow purpose and acted as "dumb pipes", some are becoming smarter and doing more processing at the edge. The benefit of this is that not as much telemetry needs to go to the observability endpoint, making it quicker and cheaper. Instead, you just send aggregated data – counters, averages, and other statistics. While some information is lost, companies are already selectively monitoring different applications at different levels of specificity – for cost and simplicity.
There's no one size fits all, but I imagine the one-size-fits-all type of observability agent will split into more specialized types on the spectrum of agentless, thin client, and thick client.