Logging and monitoring are core components of public and private data centers. Logging can be decisive in finding both security holes in monitoring infrastructure and application anomalies. Azure is no exception to seeing these challenges. Over the past few years, Microsoft has acquired companies such as Bluestripe, Cloudyn, MetricsHub, etc., to boost its presence in the cloud space. And now Azure can generate a variety of logs, such as service logs, system logs, application logs, network logs, etc. Some of this logging is enabled automatically, although it may require custom configurations, as it’s quite tedious to design a compliant logging and monitoring system to connect all the dots. This article is aimed at shedding some light on this Azure’s logging capabilities.
Logs can be broadly broken down into a few distinct categories based on their nature and usage.
Network logs are critical for addressing security incidents, as well for troubleshooting network issues. Azure provides services, such as Network Security Groups (NSG) logging, Network Watcher, Network Performance Monitor (NPM), etc., for visibility into discrete network events. NSG flow and application gateway logging collect five-tuple network traffic to Azure subnets, which can be put into Azure storage, or streamed to Power BI for real-time monitoring. NPM is aimed at monitoring the on-prem to Azure latencies and to watch machine to machine performance. Network Watcher provides a web interface to test next hop, packet inspection, VPN Diagnostics, IP flow verify, etc.
Azure and Azure APIs are accessed through the Azure portal, CLI or the service principal. These audit and sign-in logs can be streamed to Power BI, or obtained through audit APIs and connected to the Azure Security Center.
Azure monitor keeps track of all calls made to Azure APIs, and can be well integrated into Operations Management Suite (OMS) to gain insights on the overall subscription. This data can be used to trigger alerts or runbooks to perform a variety of tasks.
Azure Instance logs
Both Windows and Linux operating systems can be bootstrapped with Microsoft Monitoring Agents (MMAs) to enable VM diagnostic logging. Windows VMs support events performance counters and IIS logging, while Linux VMs support syslogs and performance logs.
Diagnostics logs can be turned on for Azure services, such as storage, Azure SQL database, load balancers, key vault, NSGs, app gateways, Azure Container Service (AKS), etc. Once connected, OMS provides pre-canned management packs, such as NSG analytics, key vault analytics, SQL server analytics, etc. to monitor resource level actions and performance.
Traditional multi-tier or microservices based applications can be monitored using AppInsights. This competes with the application performance management (APM) features of tools such as Dynatrace, AppDynamics and New Relic. AppInsights provides easy integration with developer tools such as Visual Studio or Eclipse, to enable developers to inject custom metrics and log in to monitor the performance of the applications. AppInsights logs can be streamed to Power BI for real-time monitoring.
OMS & Log Analytics
Operations Management Suite helps bring a single pane of glass solution to analyzing and monitoring logs from different sources. OMS provides pre-canned monitoring solutions called intelligence packs, for specific uses. Intelligence packs include Key Vault Analytics, NSG Analytics, AD assessment, Malware assessment and SQL server assessment, to name a few. System Center products, such as SCCM (configuration manager) and SCOM (operations manager), can be integrated into OMS to bring the on-prem VM logs to OMS as well.
Log analytics help with searching any data from any type of log source. Log search queries can be designed to detect anomalies and trigger remediation. For example, a search query can identify new VMs in a subscription, and alert the Azure automation runbook to install anti-malware agents.
From the pricing standpoint, the OMS free plan sets a daily cap of 500 MB per workspace, which is ideal for development environments. For production environments, OMS is offered as standard. Premium plans have no limit on the amount of data that is uploaded, and costs $20 (E1) or $35 (E2) per instance. Standalone Log Analytics workspaces are billed based on the volume of ingested data and cost $35 per GB.
With Microsoft’s latest Cloudyn acquisition, Azure Cost Management now allows tracking cloud usage and expenditures for resources on Azure, as well as other cloud providers, including AWS and Google. Upon activation,Cloudyn enables cost allocation reporting by matching definitions of cost entities (business units, sales regions, departments, etc.), coupled with flexible cost category creation and tagging/auto-tagging functionality for measuring and reporting on the metrics that matter most to business.
Integration with SIEM systems
Some of our customers use SIEM platforms, such as Splunk, SolarWinds, etc., to comply with the requirements of PCI, FISMA, HIPAA or other regulations. Azure Log Integration is a free Windows server add-on that integrates raw logs from your Azure resources into on-premises Security Information and Event Management (SIEM) systems.
With evolving changes in security threat patterns, cloud infrastructure and application architectures, it is vital for enterprises to design comprehensive, systematic logging and monitoring capabilities. With its wide array of tools and services, Azure enables designing end-to-end logging and monitoring systems to meet these challenges.