Today’s digital-centric businesses must keep both eyes open when relying on cloud. As Australian businesses continue to link their enterprise cloud with every process, system, and digital asset, tech professionals are growing concerned with their ability to monitor—and observe—the internal health of their cloud resources. This issue of observability is cause and effect; the more apps and systems bolted onto the cloud, the harder it becomes to track or monitor for incompatibilities, issues, and conflicts. This could lead to instability at best, or exploitable vulnerabilities and security breaches at worst—placing Australian enterprises and their data at serious risk.
Compliance and regulation make up a huge portion of these risks. Without proper methods of observability, the possibility of data breaches occurring within unmonitored cloud connections rises and, when detected, can be too late. Australian businesses will be in danger of being fined for non-compliance to data policies at home or international data regulations—such as the GDPR—when doing business abroad.
There is a great need to increase observability of enterprise clouds, especially as velocity of adoption continues to increase globally, and within Australia. In the 2018 SolarWinds IT Trends Report, over 94% of technology professionals have said cloud and hybrid IT will be among the top five technologies their organisations will invest heavily in—with 51% further listing cloud/hybrid IT as their most indispensable technological investment.
When thinking in terms of observability, however, it’s important to keep in mind that cloud environments differ considerably from on-premises ones. This will not only affect how we approach observability in the cloud, but also how we measure success. As we become more dependent on cloud environments and distributed workloads, tech professionals must go beyond simple performance metrics to achieve the right level of observability in the cloud, leveraging combined metrics, logs, application traces, and best practices for controllability. And further, they must build these capabilities into the core of their organisation’s cloud strategy.
The Five Ways to Master Observability
Working toward new methods of observability requires specific strategies and tactics. To get started, tech professionals can follow these five tips to help gain observability in the cloud:
- Be flexible to adapt to the unknown: It goes without saying that mature monitoring tools and strategies are not the same in the cloud as they were on-premises; an application will behave differently if you lift and shift it to a container, or if you have a VM that’s running on a cloud provider—why would observability be any different? Flexibility to adapt application performance monitoring practices and tools is key: not only which tools to use, but also the way in which you’re monitoring. For example, even though your monitoring platforms will probably support new abilities, e.g., adding custom metrics, you may not have leveraged that type of functionality before. Incorporating data from unknown sources and collecting it in what appears to be novel ways is critical, and flexibility to adapt to the unknown is an important part of regaining that lost observability.
- Aggregate data: In addition to changing the monitoring and managing landscape, cloud has introduced an entirely new set of metrics into the fray—metrics and numbers that can be distilled into business insights and used to meet business goals, rather than basic mechanics of usage from an ROI/cost operations data perspective. Aggregating and collating performance metrics into a dashboard with the business metrics that hinge on the performance of the IT systems underneath, along with feed and speeds, as well as events, deployment, and new metrics like error rate, is key to demonstrating cloud ROI to management, and ensuring uptime and a positive user experience for end-users.
- Be brave: As a tech pro in the era of cloud, you will most definitely need to take the time to explain to management that some of the metrics they relied on previously don’t apply in a cloud environment. But that doesn’t (necessarily) mean something isn’t working, or that certain business-critical applications aren’t being sufficiently monitored. Technology pros who have migrated to the cloud and then come back on-premises, for example, likely suffered an observability problem that didn’t allow them to fix the application.There’s a lot of human engineering that goes along with this implementation (in addition to actual services) and the only way to get through that is to remember, you’re the expert—stand up for yourself and your environment.
- Learn to trust (and verify) monitoring services: We often think that our monitoring systems need to collect and store information on-premises, because that’s the “safest” place to put it. At the same time, relying on a service, or a hosted service, especially for most mission-critical information that we use to debug a system, can be nerve-wracking—what if there’s a VPN outage and these alerts cut out? Whether it’s log activation, custom metric storage, or distributed tracing, there must be an end point for all of these moving pieces.Verifying new technology and learning to trust service-based monitoring means developing skills in new technologies to ensure that this system is going to work together with redundancy you didn’t need before when it was on-prem, and to gain insight into “observability of the observability” (i.e., who’s monitoring the monitoring system), to allow you to start thinking about breadth of monitoring, metrics, reach, and completeness. Fortunately, increased visibility both illuminates the dark corners in need of monitoring, but also can prove that you have enough.
- Extend monitoring: Extend monitoring to include classic cases of on-prem events and logs in the services to determine whether or not connections are being made to cloud resources in violation of policy. In the cloud, the last mile of that observability can be a challenge. Data is often collected and thrown into lakes like CloudWatch, without a mechanism to pull it into a dashboard with visibility. With GDPR implementation, for example, the ability to monitor traffic source outside of policy is incredibly important. This is just one area where custom metrics and events monitoring may be helpful.
Making the Cloud Crystal Clear Again
Just like any ubiquitous piece of technology, the cloud requires tech pros to leverage the skills and knowledge toward strengthening their control—and observability—of their cloud networks. As hybrid IT extends beyond just the usual mix of on-prem and cloud to also include future technologies—such as AI and machine learning—the need to maintain visibility over all moving parts of an organisation will only intensify. Australian businesses must be prepared, both in mind and infrastructure, to meet this intelligent future with both eyes open.
